Commit graph

37 commits

Author SHA1 Message Date
3bb115266b feat: Add GUID prefix to Qdrant collection names for KB security isolation
Some checks failed
BotServer CI / build (push) Failing after 1s
2026-03-19 19:51:28 -03:00
7fb73e683f feat: add campaigns, attendance SLA, and marketing modules 2026-03-14 16:35:42 -03:00
859db6b8a0 fix: Lower KB search thresholds and add Cloudflare AI embedding support
All checks were successful
BotServer CI / build (push) Successful in 10m35s
- Lower score_threshold in kb_indexer.rs from 0.5 to 0.3
- Lower website search threshold in kb_context.rs from 0.6 to 0.4
- Lower KB search threshold in kb_context.rs from 0.7 to 0.5
- Add Cloudflare AI (/ai/run/) URL detection in cache.rs
- Add Cloudflare AI request format ({"text": ...}) in cache.rs
- Add Cloudflare AI response parsing (result.data) in cache.rs

This fixes the issue where KB search returned 0 results even with
114 chunks indexed. The high thresholds were filtering out all results.
2026-03-05 00:11:08 -03:00
8500949fcd fix: Lower KB search thresholds and add Cloudflare AI embedding support
- Lower score_threshold in kb_indexer.rs from 0.5 to 0.3
- Lower website search threshold in kb_context.rs from 0.6 to 0.4
- Lower KB search threshold in kb_context.rs from 0.7 to 0.5
- Add Cloudflare AI (/ai/run/) URL detection in cache.rs
- Add Cloudflare AI request format ({"text": ...}) in cache.rs
- Add Cloudflare AI response parsing (result.data) in cache.rs

This fixes the issue where KB search returned 0 results even with
114 chunks indexed. The high thresholds were filtering out all results.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-03-05 00:06:17 -03:00
5404e3e7ba feat: Enhance KB context, embedding generator, and website crawler
- Improved kb_context with better context management
- Enhanced embedding_generator with extended functionality (+231 lines)
- Updated kb_indexer with improved indexing logic
- Expanded website_crawler_service capabilities (+230 lines)
- Updated use_website keyword implementation
- Refined bootstrap_manager and utils
- Improved drive monitoring and local file monitor
- Added server enhancements
2026-03-04 15:43:37 -03:00
764f058653 fix: update work directory paths to use botserver-stack/data/system/work
All checks were successful
BotServer CI / build (push) Successful in 7m4s
Updated all hardcoded work/ directory references to use the correct
relative path from the current working directory:
- botserver-stack/data/system/work

This ensures consistent file location resolution regardless of where
botserver is run from (/home/rodriguez/src/gb/ or /opt/gbo/bin/).

Changes:
- local_file_monitor.rs: Use std::env::current_dir() for work_root
- drive_monitor/mod.rs: Use work_root PathBuf for tool compilation
- website_crawler_service.rs: Use std::env::current_dir() for work_path

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-22 16:20:07 -03:00
1856215d05 chore: update dependencies and formatting
All checks were successful
BotServer CI / build (push) Successful in 7m30s
2026-02-22 15:55:39 -03:00
e143968179 feat: Add JWT secret rotation and health verification
SEC-02: Implement credential rotation security improvements

- Add JWT secret rotation to rotate-secret command
- Generate 64-character HS512-compatible secrets
- Automatic .env backup with timestamp
- Atomic file updates via temp+rename pattern
- Add health verification for rotated credentials
- Route rotate-secret, rotate-secrets, vault commands in CLI
- Add verification attempts for database and JWT endpoints

Security improvements:
- JWT_SECRET now rotatable (previously impossible)
- Automatic rollback via backup files
- Health checks catch configuration errors
- Clear warnings about token invalidation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-19 19:42:41 +00:00
e34848507d fix: Update multiple modules for i18n and drive monitoring
Some checks failed
BotServer CI / build (push) Failing after 6m22s
- Update auto_task modules (app_generator, designer_ai, intent_classifier)
- Refactor use_tool.rs for better structure
- Update bot core and website crawler
- Improve drive_monitor and local_file_monitor
- Update bootstrap module

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 22:06:57 +00:00
5ea171d126 Refactor: Split large files into modular subdirectories
Some checks failed
BotServer CI / build (push) Failing after 1m34s
Split 20+ files over 1000 lines into focused subdirectories for better
maintainability and code organization. All changes maintain backward
compatibility through re-export wrappers.

Major splits:
- attendance/llm_assist.rs (2074→7 modules)
- basic/keywords/face_api.rs → face_api/ (7 modules)
- basic/keywords/file_operations.rs → file_ops/ (8 modules)
- basic/keywords/hear_talk.rs → hearing/ (6 modules)
- channels/wechat.rs → wechat/ (10 modules)
- channels/youtube.rs → youtube/ (5 modules)
- contacts/mod.rs → contacts_api/ (6 modules)
- core/bootstrap/mod.rs → bootstrap/ (5 modules)
- core/shared/admin.rs → admin_*.rs (5 modules)
- designer/canvas.rs → canvas_api/ (6 modules)
- designer/mod.rs → designer_api/ (6 modules)
- docs/handlers.rs → handlers_api/ (11 modules)
- drive/mod.rs → drive_handlers.rs, drive_types.rs
- learn/mod.rs → types.rs
- main.rs → main_module/ (7 modules)
- meet/webinar.rs → webinar_api/ (8 modules)
- paper/mod.rs → (10 modules)
- security/auth.rs → auth_api/ (7 modules)
- security/passkey.rs → (4 modules)
- sources/mod.rs → sources_api/ (5 modules)
- tasks/mod.rs → task_api/ (5 modules)

Stats: 38,040 deletions, 1,315 additions across 318 files

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 21:09:30 +00:00
6215908536 Add ConfigWatcher and fix model routing
New features:
- Add ConfigWatcher for hot-reloading config.csv from ~/data
- Add LocalFileMonitor for watching ~/data/*.gbai directories
- Add GLM LLM provider implementation
- Add tool context for LLM tool calling

Bug fixes:
- Fix model routing to respect session → bot → default hierarchy
- Fix ConfigWatcher to handle local embedded (llm-server=true)
- Skip DriveMonitor for default bot (managed via ConfigWatcher)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-09 15:10:27 +00:00
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
d26f0652e5 fix(website-crawler): Add missing crawl_status column to SELECT query
Some checks failed
GBCI / build (push) Failing after 14m19s
Fix the SQL query in check_and_crawl_websites() to also include crawl_status
in the SELECT clause. The WebsiteCrawlRecord struct expects this field but it was
only present in the WHERE clause, causing Diesel to fail with
'Column crawl_status was not present in query'.

This resolves the second website crawler service error.
2026-02-02 20:10:20 -03:00
841b59affd fix(website-crawler): Add missing next_crawl column to SELECT query
Some checks failed
GBCI / build (push) Failing after 12m2s
Fix the SQL query in check_and_crawl_websites() to include next_crawl in the
SELECT clause. The WebsiteCrawlRecord struct expects this field but it was
missing, causing Diesel to fail with 'Column next_crawl was not present in query'.

This resolves the website crawler service error that was preventing websites
from being properly queried and recrawled.
2026-02-02 19:58:11 -03:00
748fceff5d Fix issues: remove unused import, fix ownership error, reduce crawler interval 2026-01-30 12:21:30 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
033bb504b9 Various updates: dependencies, features, and bug fixes 2026-01-16 11:29:22 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
a4cbf145d2 fix: Multiple compilation fixes per PROMPT.md
Video module:
- Fix state.db -> state.conn field name
- Fix analytics.rs imports placement
- Remove AppState dependency from websocket.rs (use global broadcaster)
- Simplify render.rs broadcaster usage

Other modules:
- Add sha1 crate dependency
- Fix AppState import paths (project, legal)
- Fix db_pool -> conn throughout codebase
- Add missing types: RefundResult, ExternalSyncError, TasksIntegrationError, RecordingError, FallbackAttemptTracker
- Add stub implementations for GoogleContactsClient, MicrosoftPeopleClient
- Fix social/mod.rs format string
- Fix designer/canvas.rs SVG path
- Remove doc comments per PROMPT.md
- Add missing handler implementations in calendar_integration.rs
2026-01-08 15:35:03 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
29b80f597c Fix email_accounts -> user_email_accounts table name typo in list_emails_htmx 2026-01-04 08:48:27 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
a5dee11002 Security audit: Remove all production .unwrap()/.expect(), add SafeCommand, ErrorSanitizer
- Phase 1 Critical: All 115 .unwrap() verified in test code only
- Phase 1 Critical: All runtime .expect() converted to proper error handling
- Phase 2 H1: Antivirus commands now use SafeCommand (added which/where to whitelist)
- Phase 2 H2: db_api.rs error responses use log_and_sanitize()
- Phase 2 H5: Removed duplicate sanitize_identifier (re-exports from sql_guard)

32 files modified for security hardening.
Moon deployment criteria: 10/10 met
2025-12-28 21:26:08 -03:00
aeb4e8af0f Fix all clippy warnings - 0 warnings 2025-12-28 14:27:52 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
6bc6a35948 fix: resolve all warnings - wire up services properly 2025-12-17 17:41:37 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
c67ff407b1 Add .env.example with comprehensive configuration template
The commit adds a complete example environment configuration file
documenting all available settings for BotServer, including logging,
database, server, drive, LLM, Redis, email, and feature flags.

Also removes hardcoded environment variable usage throughout the
codebase, replacing them with configuration via config.csv or
appropriate defaults. This includes:

- WhatsApp, Teams, Instagram adapter configurations
- Weather API key handling
- Email and directory service configurations
- Console feature conditionally compiles monitoring code
- Improved logging configuration with library suppression
2025-11-28 13:19:03 -03:00
1789952c5b Remove unused sqlx dependency and related code
The sqlx database library has been removed from the project along with
associated database-specific code that was no longer being used. This
includes removal of various sqlx-related dependencies from Cargo.lock
and cleanup of database connection pool references.
2025-11-28 09:27:29 -03:00
7f2967b04b Add SQLx dependencies for calendar feature 2025-11-27 23:10:43 -03:00
db06927e24 Looking at the diff, I can see this commit removes the api_router.rs
file and distributes its functionality to individual modules. The
calendar and task modules now have their own route configuration and API
handlers.

Remove centralized API router in favor of module-based routing

Decentralizes API route configuration by moving route definitions and
handlers to their respective modules. Each module now exports its own
`configure_*_routes()` function that is merged in main.rs.

- Delete api_router.rs with its mon
2025-11-27 09:38:50 -03:00
f40cb6c7b4 Fix typos in bot file extensions and keyword names
Changed incorrect references to .vbs files to .bas and corrected
USE_WEBSITE keyword naming. Also added missing fields to API response
structure and clarified that start.bas is optional for bots.
2025-11-26 22:54:22 -03:00