Commit graph

84 commits

Author SHA1 Message Date
9087bb17cd feat: complete General Bots 7.0 (v6.2.0) 2026-01-25 10:29:54 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
6fa52e1dd8 feat: implement feature bundling architecture and fix conditional compilation
- Restructured Cargo.toml with Bundle Pattern for easy feature selection
- Added feature bundles: tasks → automation + drive + monitoring
- Applied conditional compilation guards throughout codebase:
  * AppState fields (drive, cache, task_engine, task_scheduler)
  * main.rs initialization (S3, Redis, Tasks)
  * SessionManager Redis usage
  * bootstrap S3/Drive operations
  * compiler task scheduling
  * shared module Task/NewTask exports
- Eliminated all botserver compilation warnings
- Minimal build now compiles successfully
- Accepted core dependencies: automation (Rhai), drive (S3), cache (Redis)
- Created DEPENDENCY_FIX_PLAN.md with complete documentation

Minimal feature set: chat + automation + drive + cache
Verified: cargo check -p botserver --no-default-features --features minimal 
2026-01-23 13:14:20 -03:00
fc9c07d204 Refactor schema.rs into modular files 2026-01-22 13:57:40 -03:00
033bb504b9 Various updates: dependencies, features, and bug fixes 2026-01-16 11:29:22 -03:00
00acf1c76e fix: Add trusted_shell_script_arg for internal scripts
- shell_script_arg blocks $( and backticks for user input safety
- trusted_shell_script_arg allows these for internal installer scripts
- Internal scripts need shell features like command substitution
- Updated bootstrap, installer, facade, and llm modules
2026-01-09 12:13:35 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
a4cbf145d2 fix: Multiple compilation fixes per PROMPT.md
Video module:
- Fix state.db -> state.conn field name
- Fix analytics.rs imports placement
- Remove AppState dependency from websocket.rs (use global broadcaster)
- Simplify render.rs broadcaster usage

Other modules:
- Add sha1 crate dependency
- Fix AppState import paths (project, legal)
- Fix db_pool -> conn throughout codebase
- Add missing types: RefundResult, ExternalSyncError, TasksIntegrationError, RecordingError, FallbackAttemptTracker
- Add stub implementations for GoogleContactsClient, MicrosoftPeopleClient
- Fix social/mod.rs format string
- Fix designer/canvas.rs SVG path
- Remove doc comments per PROMPT.md
- Add missing handler implementations in calendar_integration.rs
2026-01-08 15:35:03 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
7bb7f8883c Fix Azure Claude to use Anthropic API format (x-api-key, /v1/messages) 2025-12-29 10:40:18 -03:00
38f9abb7db Fix organizations foreign key reference (org_id not id) 2025-12-29 08:07:42 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
3a8510d191 Fix clippy warnings: raw strings, unused imports, format args, must_use attributes 2025-12-24 09:39:23 -03:00
7cbfe43319 Update attendance, keywords, calendar, compliance, console, core, drive, email, llm, msteams, security, and tasks modules 2025-12-24 09:29:27 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
6bc6a35948 fix: resolve all warnings - wire up services properly 2025-12-17 17:41:37 -03:00
4486e31320 Fix llm-server default to true, convert debug eprintln to trace 2025-12-15 16:33:13 -03:00
37c2db21ef feat: configurable stack paths and local installer support
- Add BOTSERVER_INSTALLERS_PATH env var to use local installers without downloading
- Replace hardcoded ./botserver-stack paths with configurable stack_path
- Add stack_dir() and vault_bin() helper methods in BootstrapManager
- Add Port: 8300 to Zitadel config to fix port binding issue
- Start Directory service before setup_directory() call
- Add SKIP_LLM_SERVER env var to skip local LLM in tests
- Update template loading to check ../bottemplates and botserver-templates paths
2025-12-14 15:58:54 -03:00
fda88c0d94 Bootstrap started! 6.1.0
- Add rss and scraper dependencies for web data keywords
- Add SMS keyword with priority support (low, normal, high, urgent)
- Add web_data keywords: RSS, SCRAPE, SCRAPE_ALL, SCRAPE_TABLE, SCRAPE_LINKS, SCRAPE_IMAGES
- Add ai_tools keywords: TRANSLATE, OCR, SENTIMENT, CLASSIFY
- Improve LLM server health check with better diagnostics and increased timeout
- Fix compilation errors and warnings
- Register SMS keywords in BASIC engine
2025-12-10 18:22:02 -03:00
051c8f720c fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
af6b797572 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
501bd77f9c fix(llm): Respect llm-server config flag to skip local server startup
When llm-server=false in bot_configuration, the code now skips
attempting to start local llama-server processes. This prevents
the 60-attempt timeout error when using external LLM endpoints
or when local LLM serving is intentionally disabled.
2025-12-08 23:39:01 -03:00
d9b09e24d4 feat(email): implement email read tracking with pixel support
- Add email-read-pixel config parameter to enable/disable tracking
- Implement tracking pixel injection in HTML emails
- Add sent_email_tracking table with migration
- Create 4 new API endpoints:
  - GET /api/email/tracking/pixel/{id} - serve pixel & record read
  - GET /api/email/tracking/status/{id} - get email read status
  - GET /api/email/tracking/list - list all tracked emails
  - GET /api/email/tracking/stats - get aggregate statistics
- Store tracking data: read_at, read_count, IP, user_agent
- Integrate with send_email() to auto-inject pixel when enabled
2025-12-04 18:15:09 -03:00
340ac8840a - New templates. 2025-12-03 07:15:54 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
fd5a1ee1e1 Add implementation plan and multi-agent features
This commit introduces comprehensive documentation and implementation
for multi-agent orchestration capabilities:

- Add IMPLEMENTATION-PLAN.md with 4-phase roadmap
- Add Kubernetes deployment manifests (deployment.yaml, hpa.yaml)
- Add database migrations for multi-agent tables (6.1.1, 6.1.2)
- Implement A2A protocol for agent-to-agent communication
- Implement user memory keywords for cross-session persistence
- Implement model routing for dynamic L
2025-11-30 19:18:23 -03:00
c67ff407b1 Add .env.example with comprehensive configuration template
The commit adds a complete example environment configuration file
documenting all available settings for BotServer, including logging,
database, server, drive, LLM, Redis, email, and feature flags.

Also removes hardcoded environment variable usage throughout the
codebase, replacing them with configuration via config.csv or
appropriate defaults. This includes:

- WhatsApp, Teams, Instagram adapter configurations
- Weather API key handling
- Email and directory service configurations
- Console feature conditionally compiles monitoring code
- Improved logging configuration with library suppression
2025-11-28 13:19:03 -03:00
1789952c5b Remove unused sqlx dependency and related code
The sqlx database library has been removed from the project along with
associated database-specific code that was no longer being used. This
includes removal of various sqlx-related dependencies from Cargo.lock
and cleanup of database connection pool references.
2025-11-28 09:27:29 -03:00
7f2967b04b Add SQLx dependencies for calendar feature 2025-11-27 23:10:43 -03:00
82e92de011 Remove unused Python scripts and book.toml
The scripts were development utilities for SVG processing and mdBook
configuration that are no longer needed in the project.
2025-11-26 15:27:47 -03:00
53bac1f5ec - Refactor folder as features. 2025-11-22 22:55:35 -03:00
9ea9360231 - Removing warnings. 2025-11-22 22:54:45 -03:00
e013f3f800 - Even more keywords. 2025-11-22 12:26:16 -03:00
a39bfce495 - New stuff, 6.1. 2025-11-21 23:23:53 -03:00
5364a5df56 Migrate HTTP API from Actix to Axum 2025-11-20 13:28:35 -03:00
afe87e1ab5 feat: add actix-files dependency for file serving support
Added actix-files and its dependencies (http-range, mime_guess, unicase, v_htmlescape) to enable static file functionality in the botserver. This will allow serving static assets and files through the web server. The change includes all required transitive dependencies for proper file handling and MIME type detection.
2025-11-15 19:08:26 -03:00
7a3a761a8e feat(llm): remove deprecated args and clean up server startup
Removed commented-out code for deprecated LLM server arguments (n_moe, parallel, cont_batching, etc.) since these are no longer used. Also cleaned up the model arguments string by removing --jinja and --flash-attn flags which were moved to TODO comments for future config implementation. The change simplifies the server startup code while maintaining core functionality.
2025-11-14 14:14:21 -03:00
c767040ea9 feat(llm): add jinja flag to local server args
Added the --jinja flag to the LLM server startup arguments to enable Jinja template support. This allows for more flexible prompt formatting when using the local LLM server. The change maintains all existing functionality while adding the new feature.
2025-11-12 18:37:01 -03:00
d3b66217a4 feat(llm): simplify log file paths and remove redundant verbose flag
Removed the redundant `--verbose` flag from Windows command since it's not needed. Standardized log file names to `llm-stdout.log` and `llmembd-stdout.log` for consistency across platforms. This makes log management simpler and more predictable.
2025-11-12 18:20:50 -03:00
ac3a274109 feat(llm): add flash-attn flag to local server args
Added the `--flash-attn on` flag to the LLM server startup arguments to enable flash attention optimization. This improves performance while maintaining existing parameters (top_p, temp, repeat-penalty). A TODO was added to move these parameters to config for better maintainability.
2025-11-12 18:19:23 -03:00
40e4ce5995 feat(llm): standardize ctx-size parameter naming
Updated the parameter name from 'n-ctx-size' to 'ctx-size' in both config lookup and argument formatting for consistency. This change aligns with the naming convention used elsewhere in the codebase and makes the parameter name more concise while maintaining clarity. The functionality remains unchanged.
2025-11-12 17:27:32 -03:00
780e9d3172 feat(llm): standardize config key naming to use hyphens
Changed the config key 'llm-server-n_ctx_size' to 'llm-server-n-ctx-size' in local.rs to maintain consistent hyphen-separated naming convention across configuration parameters. This improves code readability and aligns with existing naming patterns.
2025-11-12 17:25:08 -03:00