Commit graph

104 commits

Author SHA1 Message Date
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
3a8510d191 Fix clippy warnings: raw strings, unused imports, format args, must_use attributes 2025-12-24 09:39:23 -03:00
7cbfe43319 Update attendance, keywords, calendar, compliance, console, core, drive, email, llm, msteams, security, and tasks modules 2025-12-24 09:29:27 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
5619cf2e16 fix: add DNS configuration to containers before apt-get
- Add Google DNS (8.8.8.8, 8.8.4.4) to /etc/resolv.conf
- Fixes 'Temporary failure resolving' errors in containers
2025-12-19 22:32:57 -03:00
2cff6bf430 fix: make exec_in_container return errors instead of silently warning
- Changed from warn() to error() and return Err()
- Added info logging for each command executed
- Now we can see why downloads/installs fail
2025-12-19 19:59:15 -03:00
27b5184f82 fix: pass VAULT_ADDR inside container via bash -c
- env() on Command only sets host env, not container env
- Use bash -c with VAULT_ADDR=... prefix for init and unseal commands
2025-12-19 13:19:50 -03:00
110ca2f3de fix: store vault-unseal-keys in working directory, not /opt/gbo/secrets 2025-12-19 12:28:22 -03:00
c750e26c3b feat: auto-generate .env and vault-unseal-keys on vault install
- Vault is automatically initialized with 5 keys, threshold 3
- Creates /opt/gbo/secrets/vault-unseal-keys with chmod 600
- Creates or appends to .env with VAULT_ADDR, VAULT_TOKEN, VAULT_UNSEAL_KEYS_FILE
- Vault is automatically unsealed after init
- No manual steps required for initial setup
2025-12-19 12:20:55 -03:00
4062c79cce feat: use secrets file for Vault auto-unseal (more secure)
- VAULT_UNSEAL_KEYS_FILE points to /opt/gbo/secrets/vault-unseal-keys
- Unseal keys stored in separate file with chmod 600
- Not in .env (which might be in git, logs, etc.)
- Output shows step-by-step setup instructions
2025-12-19 11:36:00 -03:00
2a8f884864 fix: improve container IP detection and add VAULT_UNSEAL_KEYS to .env
- Wait for network before getting IP
- Try hostname -I as fallback for IP detection
- Add VAULT_UNSEAL_KEY_1/2/3 to .env output for auto-unseal
- botserver can auto-unseal Vault on startup using these keys
2025-12-19 11:02:27 -03:00
3eb86772fa feat: show connection info after container install
- Only Vault returns .env variables (VAULT_ADDR, VAULT_TOKEN)
- All other components show 'botserver vault put' commands to store credentials
- Added proper vault init/unseal instructions
- CLI now prints InstallResult with IP, ports, and connection info
2025-12-19 10:49:09 -03:00
79e83dd3b8 fix: install base packages (wget, unzip, curl) in containers before download
- Vault download was failing because unzip was not installed
- Now all containers get wget, unzip, curl, ca-certificates as base packages
- Also added vault config.hcl generation in pre-install commands
2025-12-19 09:48:15 -03:00
2c61c816b5 fix: allow container install without Vault configured
- Make create_container_service work without DATABASE_URL from Vault
- Components like vault, vector_db that don't need DB can now install first
- DB_PASSWORD defaults to empty string when Vault is not available
2025-12-19 09:33:56 -03:00
8efeeaed4b fix: make container installation fully non-interactive
- Add automatic lxd init --auto before container creation
- Use DEBIAN_FRONTEND=noninteractive for apt-get
- Add apt-get update -qq before installing packages
2025-12-18 21:20:47 -03:00
3987546ebc feat: add fallback images for LXC container creation and install-dependencies.sh
- Try ubuntu:24.04, ubuntu:22.04, images:debian/12, images:debian/11 in order
- Clean up failed container attempts before trying next image
- Add install-dependencies.sh script for runtime library installation
- Fixes 404 error when LXD image server has outdated cached images
2025-12-18 21:09:26 -03:00
49b2886614 Add rotate-secret and rotate-secrets commands for credential rotation 2025-12-18 11:20:31 -03:00
f4d2172366 Add version --all command 2025-12-18 11:06:19 -03:00
952a822e1d Add vault CLI commands for secret migration 2025-12-18 09:40:16 -03:00
37c1cee805 Fix: Skip starting components that are already running (minio, valkey) 2025-12-16 10:48:42 -03:00
f1f0436fdd Fix vault health check - quote URL to prevent & being interpreted as shell background 2025-12-15 18:04:53 -03:00
562d3cf0e4 Add debug logs to package manager start function 2025-12-15 17:50:03 -03:00
4486e31320 Fix llm-server default to true, convert debug eprintln to trace 2025-12-15 16:33:13 -03:00
02117e2eec Fix Zitadel masterkey to come from Vault
- Add masterkey field to gbo/directory secret in Vault during bootstrap
- Generate random 32-char masterkey if not exists
- Update Zitadel exec_cmd to read masterkey from Vault
- Remove hardcoded masterkey values
2025-12-15 13:56:50 -03:00
37c2db21ef feat: configurable stack paths and local installer support
- Add BOTSERVER_INSTALLERS_PATH env var to use local installers without downloading
- Replace hardcoded ./botserver-stack paths with configurable stack_path
- Add stack_dir() and vault_bin() helper methods in BootstrapManager
- Add Port: 8300 to Zitadel config to fix port binding issue
- Start Directory service before setup_directory() call
- Add SKIP_LLM_SERVER env var to skip local LLM in tests
- Update template loading to check ../bottemplates and botserver-templates paths
2025-12-14 15:58:54 -03:00
7647774339 Move Directory (Zitadel) to port 8300
- Directory/Zitadel: 8080 -> 8300
- BotServer API remains on 8080
- Updated all references in bootstrap, installer, oauth, config
2025-12-11 08:43:28 -03:00
696a044d75 Update llama.cpp to b7345 with platform-specific builds and checksums
- Update 3rdparty.toml: llama.cpp b4547 -> b7345 with SHA256 checksums
- Add config/llm_releases.json with complete checksums for all 24 release assets
- Fix Windows binary naming in installer.rs (win-cpu-x64, win-cpu-arm64)
- Add Vulkan detection for Windows
- Add platform-specific variants: CUDA 12/13, Vulkan, HIP, SYCL, OpenCL
2025-12-10 12:54:52 -03:00
137a5c8012 fix(llm): Restore llama.cpp b6148 which was working 2025-12-10 10:46:54 -03:00
9fefd6c1fa fix(llm): Use pre-built llama.cpp binaries, remove compilation
- Revert to pre-built binaries (b4547) instead of compiling from source
- Remove cmake/build-essential dependencies
- Remove post_install compilation commands
- Fix exec_cmd path back to {{BIN_PATH}}/llama-server
2025-12-10 09:00:05 -03:00
051c8f720c fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
af6b797572 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
8711406865 fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
224ddd4a03 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
185aba3631 fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
a755d4d13e feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00
05fa0e95c5 fix: remove duplicate logger init causing 'cannot set logger' error
Logger is already initialized in main.rs, cli.rs was trying to
initialize it again when CLI commands were run.
2025-12-08 17:59:43 -03:00
559154b923 fix: preserve cached files in botserver-installers after extraction
- Don't delete archive files from cache after tar/zip extraction
- Copy binaries from cache instead of moving them
- Cached files are now preserved for offline installation reuse
2025-12-08 14:19:55 -03:00
eed537ac42 feat: add offline installer cache and health endpoints
- Add /health and /api/health endpoints for botui connectivity
- Create 3rdparty.toml with all download URLs for offline bundles
- Add botserver-installers/ cache directory for downloaded files
- Implement DownloadCache module with:
  - Automatic cache lookup before downloading
  - Support for pre-populated offline bundles
  - SHA256 checksum verification (optional)
  - Cache management utilities (list, clear, size)
- Update download_and_install to use cache system
- Data files (models) also cached for reuse

Cache behavior:
- First run: downloads to botserver-installers/
- Subsequent runs: uses cached files
- Delete botserver-stack/ without losing downloads
- Pre-populate cache for fully offline installation
2025-12-08 14:08:49 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
6ff2b32f2c feat(bootstrap): implement mTLS for Vault access
- Add create_vault_config() function to generate config.hcl with mTLS settings
- Configure Vault to require client certificate verification
- Generate client certificate for botserver in bootstrap
- Update .env to include mTLS paths (VAULT_CACERT, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY)
- Remove unused import in tls.rs
2025-12-07 02:13:28 -03:00
b611f8c535 Add --stack-path CLI option for custom stack path
Allows overriding the default botserver-stack location via:
- --stack-path /path/to/stack CLI argument
- BOTSERVER_STACK_PATH environment variable

Useful for testing botserver bootstrap in isolated temp directories.
2025-12-06 14:55:42 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
48288e8523 ``` Add KB Statistics keywords and infrastructure documentation
- Add KB Statistics keywords for Qdrant vector database monitoring: KB
  STATISTICS, KB COLLECTION STATS, KB DOCUMENTS COUNT, KB DOCUMENTS
  ADDED SINCE, KB LIST COLLECTIONS, KB STORAGE SIZE

- Add comprehensive infrastructure documentation:
  - Scaling and load balancing with LXC containers
  - Infrastructure design with encryption, sharding strategies
  - Observ
2025-11-30 16:25:51 -03:00
c6bbd50946 , dt.month, dt.hour, dt.is_weekend, etc.)
- Add startup wizard module for first-run configuration
- Add white-label branding system with .product file support
- Add bot manager for lifecycle, MinIO buckets, and templates
- Add version tracking registry for component updates
- Create comparison doc: BASIC vs n8n/Zapier/Make/Copilot
- Add WhatsApp-style sample dialogs to template documentation
- Add data traceability SVG diagram ```
2025-11-30 15:07:29 -03:00
ad1cd3ee36 HTMX enters. 2025-11-29 16:29:28 -03:00