Commit graph

53 commits

Author SHA1 Message Date
affb702e90 fix(tools): auto-create work directory if missing in production
All checks were successful
BotServer CI / build (push) Successful in 8m12s
Added logic to create botserver-stack/data/system/work directory
if it doesn't exist. This ensures production deployments work
without manual directory setup.

Changes:
- Added fs::create_dir_all() in use_tool.rs
- Added fs::create_dir_all() in tool_context.rs
- Logs when directory is created
- Fixes production deployment where /system/work may not exist

This ensures the tool loading works in fresh production environments
where the work folder hasn't been populated yet.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-22 12:58:58 -03:00
febfa2e708 fix(tools): use relative path for tool .mcp.json files
All checks were successful
BotServer CI / build (push) Successful in 8m37s
Changed tool loading to use relative path from current directory instead
of hardcoded HOME/gb path. This makes the code portable across different
deployment environments.

- Updated use_tool.rs to use std::env::current_dir()
- Updated tool_context.rs to use std::env::current_dir()
- Added PathBuf import to both files
- Tools now load from botserver-stack/data/system/work/

Fixes issue where tools weren't being loaded because .mcp.json files
were in a different location than expected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-22 11:21:07 -03:00
de017241f2 fix: Complete security remediation - RCE and SSRF fixes
All checks were successful
BotServer CI / build (push) Successful in 7m34s
- Fixed RCE vulnerability in trusted_shell_script_arg execution
- Fixed SSRF vulnerability in GET command with internal IP blocking
- Updated SafeCommand to use explicit positional arguments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 01:14:14 +00:00
e143968179 feat: Add JWT secret rotation and health verification
SEC-02: Implement credential rotation security improvements

- Add JWT secret rotation to rotate-secret command
- Generate 64-character HS512-compatible secrets
- Automatic .env backup with timestamp
- Atomic file updates via temp+rename pattern
- Add health verification for rotated credentials
- Route rotate-secret, rotate-secrets, vault commands in CLI
- Add verification attempts for database and JWT endpoints

Security improvements:
- JWT_SECRET now rotatable (previously impossible)
- Automatic rollback via backup files
- Health checks catch configuration errors
- Clear warnings about token invalidation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-19 19:42:41 +00:00
d7211a6c19 fix: Resolve unused import and variable warnings 2026-02-19 11:48:17 +00:00
3b21ab5ef9 fix: prevent duplicate message in chat when tool is executed
- Track tool_was_executed flag in stream_response
- Send empty content in final is_complete message when tool already sent results
- Prevents the LLM's pre-tool text from appearing twice in the chat UI
- DB message saving is unaffected (uses full_response_clone before the check)
2026-02-18 20:31:34 +00:00
b1118f977d fix: Correct parameter names in tool .bas files to match database schema
- Tool 06: Change tipoExibicao to tipoDescricao (matches pedidos_uso_imagem table)
- Tool 07: Change tipoExibicao to categoriaDescricao (matches licenciamentos table)
- Both tools now compile and execute successfully with database inserts

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:51:47 +00:00
c264ad1294 fix: Improve tool call handling and BASIC SELECT/CASE conversion
All checks were successful
BotServer CI / build (push) Successful in 9m34s
- Add tool_call_buffer to accumulate JSON chunks across multiple LLM responses
- Handle incomplete tool call JSON that spans multiple chunks
- Convert SELECT...CASE/END SELECT to Rhai match expressions
- Fix NOT IN operator conversion to !in for IF conditions
2026-02-16 09:30:19 +00:00
4ca7e5da40 fix: Handle tool calls in JSON array format
All checks were successful
BotServer CI / build (push) Successful in 9m25s
Fixed issue where LLM tool calls returned as JSON arrays were not being
detected and were displayed as raw JSON in the chat instead of being executed.

The parse_tool_call method now handles:
- Single tool call objects
- Arrays of tool calls (OpenAI standard format)

This prevents tool call JSON from appearing in the chat window and ensures
tools are executed properly.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-16 00:19:03 +00:00
b92ef7c034 feat: Add detailed logging for session tools loading in WebSocket
All checks were successful
BotServer CI / build (push) Successful in 10m35s
Add error and warning logs to help diagnose why session tools are not
working in production. Logs now show:
- Number of tools loaded successfully
- Detailed error messages when tool loading fails
- Bot name lookup failures

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 23:57:17 +00:00
8257d49967 feat: Load session tools in WebSocket connection
Some checks failed
BotServer CI / build (push) Has been cancelled
This loads and sends the available tools to the client when establishing
a WebSocket connection. Tools are loaded based on the bot configuration
and sent in the initial welcome message.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 23:19:38 +00:00
a9cbbbffa0 fix: Use correct default LLM model name for local DeepSeek server
Changed default model from 'gpt-3.5-turbo' to 'DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_M.gguf'
in bot message handler. This ensures the local llama-server receives the correct model
name and can process requests properly.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 20:23:04 +00:00
8a8008a72c fix: Resolve compilation errors
All checks were successful
BotServer CI / build (push) Successful in 13m3s
- Escape format placeholders in designer_ai.rs ({{botname}})
- Remove undefined 'prefix' filter in drive_monitor
- Fix type mismatch in use_tool.rs (str vs &String)
- Remove unused TextExpressionMethods import

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 22:15:35 +00:00
e34848507d fix: Update multiple modules for i18n and drive monitoring
Some checks failed
BotServer CI / build (push) Failing after 6m22s
- Update auto_task modules (app_generator, designer_ai, intent_classifier)
- Refactor use_tool.rs for better structure
- Update bot core and website crawler
- Improve drive_monitor and local_file_monitor
- Update bootstrap module

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 22:06:57 +00:00
4b44602d39 Update bootstrap and core modules
All checks were successful
BotServer CI / build (push) Successful in 10m27s
2026-02-14 09:54:14 +00:00
5ea171d126 Refactor: Split large files into modular subdirectories
Some checks failed
BotServer CI / build (push) Failing after 1m34s
Split 20+ files over 1000 lines into focused subdirectories for better
maintainability and code organization. All changes maintain backward
compatibility through re-export wrappers.

Major splits:
- attendance/llm_assist.rs (2074→7 modules)
- basic/keywords/face_api.rs → face_api/ (7 modules)
- basic/keywords/file_operations.rs → file_ops/ (8 modules)
- basic/keywords/hear_talk.rs → hearing/ (6 modules)
- channels/wechat.rs → wechat/ (10 modules)
- channels/youtube.rs → youtube/ (5 modules)
- contacts/mod.rs → contacts_api/ (6 modules)
- core/bootstrap/mod.rs → bootstrap/ (5 modules)
- core/shared/admin.rs → admin_*.rs (5 modules)
- designer/canvas.rs → canvas_api/ (6 modules)
- designer/mod.rs → designer_api/ (6 modules)
- docs/handlers.rs → handlers_api/ (11 modules)
- drive/mod.rs → drive_handlers.rs, drive_types.rs
- learn/mod.rs → types.rs
- main.rs → main_module/ (7 modules)
- meet/webinar.rs → webinar_api/ (8 modules)
- paper/mod.rs → (10 modules)
- security/auth.rs → auth_api/ (7 modules)
- security/passkey.rs → (4 modules)
- sources/mod.rs → sources_api/ (5 modules)
- tasks/mod.rs → task_api/ (5 modules)

Stats: 38,040 deletions, 1,315 additions across 318 files

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 21:09:30 +00:00
76abcea5e9 Fix start.bas repeated execution and add tool calling system
- Add Redis-based tracking to prevent start.bas from running repeatedly
  when clicking suggestion buttons. start.bas now executes only once per
  session with a 24-hour expiration on the tracking key.

- Add generic tool executor (ToolExecutor) for parsing and executing
  tool calls from any LLM provider. Works with Claude, OpenAI, and
  other providers that use standard tool calling formats.

- Update both start.bas execution paths (WebSocket handler and LLM
  message handler) to check Redis before executing.

- Fix suggestion duplication by clearing suggestions from Redis after
  fetching them.

- Add rate limiter for LLM API calls.

- Improve error handling and logging throughout.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 13:49:54 +00:00
6215908536 Add ConfigWatcher and fix model routing
New features:
- Add ConfigWatcher for hot-reloading config.csv from ~/data
- Add LocalFileMonitor for watching ~/data/*.gbai directories
- Add GLM LLM provider implementation
- Add tool context for LLM tool calling

Bug fixes:
- Fix model routing to respect session → bot → default hierarchy
- Fix ConfigWatcher to handle local embedded (llm-server=true)
- Skip DriveMonitor for default bot (managed via ConfigWatcher)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-09 15:10:27 +00:00
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
5fb4c889b7 fix(llm-config): Fix ConfigManager fallback logic for LLM configuration
Some checks failed
GBCI / build (push) Failing after 12m26s
- Fix ConfigManager to treat 'none', 'null', 'n/a', and empty values as placeholders
  and fall back to default bot's configuration instead of using these as literal values

- Fix ConfigManager to detect local file paths (e.g., .gguf, .bin, ../) and fall back
  to default bot's model when using remote API, allowing bots to keep local model
  config for local LLM server while automatically using remote model for API calls

- Fix get_default_bot() to return the bot actually named 'default' instead of
  the first active bot by ID, ensuring consistent fallback behavior

- Add comprehensive debug logging to trace LLM configuration from database to API call

This fixes the issue where bots with incomplete or local LLM configuration would
fail with 401/400 errors when trying to use remote API, instead of automatically
falling back to the default bot's configuration from config.csv.

Closes: #llm-config-fallback
2026-02-02 19:20:37 -03:00
26963f2caf Fix bot_id: Use bot_id from URL path instead of client message
- Extract bot_name from WebSocket query parameters
- Look up bot_id from bot_name using database
- Pass bot_id to WebSocket message handler
- Use session's bot_id for LLM configuration instead of client-provided bot_id
- Fixes issue where client sends 'default' bot_id when accessing /edu
2026-01-28 17:18:22 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
6fa52e1dd8 feat: implement feature bundling architecture and fix conditional compilation
- Restructured Cargo.toml with Bundle Pattern for easy feature selection
- Added feature bundles: tasks → automation + drive + monitoring
- Applied conditional compilation guards throughout codebase:
  * AppState fields (drive, cache, task_engine, task_scheduler)
  * main.rs initialization (S3, Redis, Tasks)
  * SessionManager Redis usage
  * bootstrap S3/Drive operations
  * compiler task scheduling
  * shared module Task/NewTask exports
- Eliminated all botserver compilation warnings
- Minimal build now compiles successfully
- Accepted core dependencies: automation (Rhai), drive (S3), cache (Redis)
- Created DEPENDENCY_FIX_PLAN.md with complete documentation

Minimal feature set: chat + automation + drive + cache
Verified: cargo check -p botserver --no-default-features --features minimal 
2026-01-23 13:14:20 -03:00
66abce913f Feature gating refactor: modular compilation with minimal feature set 2026-01-22 19:45:18 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
a4cbf145d2 fix: Multiple compilation fixes per PROMPT.md
Video module:
- Fix state.db -> state.conn field name
- Fix analytics.rs imports placement
- Remove AppState dependency from websocket.rs (use global broadcaster)
- Simplify render.rs broadcaster usage

Other modules:
- Add sha1 crate dependency
- Fix AppState import paths (project, legal)
- Fix db_pool -> conn throughout codebase
- Add missing types: RefundResult, ExternalSyncError, TasksIntegrationError, RecordingError, FallbackAttemptTracker
- Add stub implementations for GoogleContactsClient, MicrosoftPeopleClient
- Fix social/mod.rs format string
- Fix designer/canvas.rs SVG path
- Remove doc comments per PROMPT.md
- Add missing handler implementations in calendar_integration.rs
2026-01-08 15:35:03 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
c743754c6c Add per-bot database creation
- Added database_name field to bots schema
- Bot creation now creates a dedicated PostgreSQL database (bot_{name})
- Updated add_bot.rs to create database and store database_name
- Added create_bot_database() function with safe name validation
- Added dynamic table check to all db_api handlers
2026-01-02 19:34:59 -03:00
a5dee11002 Security audit: Remove all production .unwrap()/.expect(), add SafeCommand, ErrorSanitizer
- Phase 1 Critical: All 115 .unwrap() verified in test code only
- Phase 1 Critical: All runtime .expect() converted to proper error handling
- Phase 2 H1: Antivirus commands now use SafeCommand (added which/where to whitelist)
- Phase 2 H2: db_api.rs error responses use log_and_sanitize()
- Phase 2 H5: Removed duplicate sanitize_identifier (re-exports from sql_guard)

32 files modified for security hardening.
Moon deployment criteria: 10/10 met
2025-12-28 21:26:08 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
dd91fc2d3b refactor: fix TaskEngine feature gate, thread-safe Extensions with Arc<RwLock> 2025-12-20 19:57:57 -03:00
512ddc5f75 Fix LLM 400 error - remove invalid bot_id from messages array
The bot_id object was being pushed into the messages array sent to the LLM,
but LLM APIs expect messages to only contain role/content objects.
2025-12-10 18:29:34 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
d088490f41 - botbook extracted from. 2025-12-03 19:59:27 -03:00
faaf9c18c8 Looking at this diff, I need to summarize the significant documentation
and code changes:

``` Add natural language scheduling, docs, wizard, and branding

- Add SET SCHEDULE natural language parser supporting patterns like
  "every hour", "at 9am", "weekdays at 8am", "business
2025-11-30 12:20:48 -03:00
60706bf1c7 Replace magic numbers with MessageType constants
This commit replaces all hardcoded message type integers (0, 1, 2, 3, 4,
5) with named constants from a new MessageType module, improving code
readability and maintainability across the codebase.
2025-11-28 18:15:09 -03:00
c67ff407b1 Add .env.example with comprehensive configuration template
The commit adds a complete example environment configuration file
documenting all available settings for BotServer, including logging,
database, server, drive, LLM, Redis, email, and feature flags.

Also removes hardcoded environment variable usage throughout the
codebase, replacing them with configuration via config.csv or
appropriate defaults. This includes:

- WhatsApp, Teams, Instagram adapter configurations
- Weather API key handling
- Email and directory service configurations
- Console feature conditionally compiles monitoring code
- Improved logging configuration with library suppression
2025-11-28 13:19:03 -03:00
1789952c5b Remove unused sqlx dependency and related code
The sqlx database library has been removed from the project along with
associated database-specific code that was no longer being used. This
includes removal of various sqlx-related dependencies from Cargo.lock
and cleanup of database connection pool references.
2025-11-28 09:27:29 -03:00
7f2967b04b Add SQLx dependencies for calendar feature 2025-11-27 23:10:43 -03:00
4b5d18db71 Add metadata and refactor BASIC compiler
- Add package metadata (keywords, categories) to Cargo.toml
- Add #[must_use] attributes to constructor methods
- Simplify conditional logic by inverting if-else blocks
- Replace string formatting with interpolation syntax
2025-11-27 15:19:17 -03:00
ac03af0b20 Refactor InstagramAdapter initialization and implement file sending
The InstagramAdapter constructor is simplified to remove unused
parameters, and the send_instagram_file function is fully implemented
with S3 upload and message sending capabilities.
2025-11-27 13:53:16 -03:00
6e48474d55 olithic route configuration
- Add route configuration and handlers to calendar module
- Add route configuration and handlers to task module
- Update main.rs to build router from module configurations
- Fix various compiler warnings (dead code, unused variables)

This improves code organization by keeping routes co-located with their
implementation logic.
2025-11-27 13:53:00 -03:00
db06927e24 Looking at the diff, I can see this commit removes the api_router.rs
file and distributes its functionality to individual modules. The
calendar and task modules now have their own route configuration and API
handlers.

Remove centralized API router in favor of module-based routing

Decentralizes API route configuration by moving route definitions and
handlers to their respective modules. Each module now exports its own
`configure_*_routes()` function that is merged in main.rs.

- Delete api_router.rs with its mon
2025-11-27 09:38:50 -03:00
4424fcf548 Fix compiler warnings and improve code consistency
- Remove unused imports and comment them for potential future use
- Add missing .send() to HTTP request chain
- Fix integer type suffixes for JSON values
- Simplify async execution by using tokio::block_in_place
- Remove unused function parameters to eliminate warnings
- Extract temporary variables to avoid borrowing issues
- Add placeholder methods to SessionManager for analytics
- Implement real database operations for admin endpoints
- Remove duplicate or conflicting type definitions

These changes address all compiler warnings while maintaining the
existing functionality and preparing the codebase for future
enhancements in areas like analytics and session management.
2025-11-27 08:34:24 -03:00