Commit graph

53 commits

Author SHA1 Message Date
db267714ca fix: Allow URL-safe characters in SafeCommand arguments
- Allow &, ?, = in URL arguments (http:// or https://)
- Allow // pattern in URLs (needed for protocol)
- These are safe since Command::new().args() doesn't use shell
- Fixes Vault health check with query parameters
- Add debug logging to safe_curl and vault_health_check
2026-01-09 11:56:11 -03:00
cb59ceb60f fix: Smart mTLS for Vault - use client cert if exists, plain TLS during bootstrap
- Add vault_health_check() function that checks if client certs exist
- If certs exist: use mTLS (secure, post-installation)
- If certs don't exist yet: use plain TLS (during initial bootstrap)
- This allows bootstrap to complete while maintaining mTLS security after setup
- No security hole: mTLS is enforced once certs are generated
2026-01-09 11:23:49 -03:00
63aee6f6bc Revert "fix: Disable mTLS for Vault (temporary until new binary compiled)"
This reverts commit efb458ba66588de090fe6df76cdf04515eacce52.
2026-01-09 11:19:36 -03:00
460d39f399 fix: Disable mTLS for Vault (temporary until new binary compiled)
- Remove tls_client_ca_file from vault config templates
- Remove --cert/--key from health checks
- TLS still enabled for encryption, just no client cert required
- TODO: Re-enable mTLS when binary with cert health checks is compiled
2026-01-09 11:18:30 -03:00
c59d456e25 fix: Add mTLS client certificate to all Vault health checks
- Keep mTLS enabled for security (even in dev)
- Add --cert and --key to all curl commands for Vault health checks
- Fix fetch_vault_credentials to use https and mTLS
- Fix Zitadel commands to use https with VAULT_CACERT
- All Vault communications now use proper mutual TLS
2026-01-09 10:48:13 -03:00
95fbb4f129 Revert "fix: Remove mTLS requirement from Vault config to fix health check failures"
This reverts commit 510fadf24b60c61bb39938f59d88add9b4c08168.
2026-01-09 10:45:41 -03:00
6c904f7dc9 fix: Remove mTLS requirement from Vault config to fix health check failures
- Remove tls_client_ca_file from vault config in installer.rs (Linux and macOS)
- Remove tls_client_ca_file from vault config in bootstrap/mod.rs
- TLS encryption still enabled, just no client certificate required
- Health checks now work with simple -sk curl flags
2026-01-09 10:33:11 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
8c24e25965 fix(bootstrap): Make S3/MinIO connection failure non-fatal
- Change from error to warn when bucket creation fails
- Continue bootstrap without drive if MinIO not available
- Prevents startup failure when S3 not configured
2025-12-27 23:15:51 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
3a8510d191 Fix clippy warnings: raw strings, unused imports, format args, must_use attributes 2025-12-24 09:39:23 -03:00
7cbfe43319 Update attendance, keywords, calendar, compliance, console, core, drive, email, llm, msteams, security, and tasks modules 2025-12-24 09:29:27 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
8405f1cfbb Fix warnings: add Debug derive, prefix unused vars 2025-12-17 11:26:58 -03:00
f1f0436fdd Fix vault health check - quote URL to prevent & being interpreted as shell background 2025-12-15 18:04:53 -03:00
66f340e20a Fix Vault retry logic - wait up to 20 seconds for vault to start 2025-12-15 17:37:00 -03:00
54af69e761 Fix Vault startup: detect connection refused and start vault properly 2025-12-15 17:32:15 -03:00
7b6b6491a1 Fix: add trace to log imports 2025-12-15 17:14:39 -03:00
cfd251c6e8 Fix Vault startup - add retry logic and auto-restart zombie vault processes 2025-12-15 17:11:47 -03:00
02117e2eec Fix Zitadel masterkey to come from Vault
- Add masterkey field to gbo/directory secret in Vault during bootstrap
- Generate random 32-char masterkey if not exists
- Update Zitadel exec_cmd to read masterkey from Vault
- Remove hardcoded masterkey values
2025-12-15 13:56:50 -03:00
37c2db21ef feat: configurable stack paths and local installer support
- Add BOTSERVER_INSTALLERS_PATH env var to use local installers without downloading
- Replace hardcoded ./botserver-stack paths with configurable stack_path
- Add stack_dir() and vault_bin() helper methods in BootstrapManager
- Add Port: 8300 to Zitadel config to fix port binding issue
- Start Directory service before setup_directory() call
- Add SKIP_LLM_SERVER env var to skip local LLM in tests
- Update template loading to check ../bottemplates and botserver-templates paths
2025-12-14 15:58:54 -03:00
65693cdee1 feat: Add Auto Task system - Intent Compiler, MCP Client, Safety Layer
- intent_compiler.rs: LLM-to-BASIC translation engine
- auto_task.rs: Auto-executing task data structures
- mcp_client.rs: Model Context Protocol server integration
- safety_layer.rs: Constraints, simulation, audit trail
- autotask_api.rs: HTTP API handlers for Auto Task UI
- Updated mod.rs with new modules and keyword list
2025-12-12 12:33:17 -03:00
7647774339 Move Directory (Zitadel) to port 8300
- Directory/Zitadel: 8080 -> 8300
- BotServer API remains on 8080
- Updated all references in bootstrap, installer, oauth, config
2025-12-11 08:43:28 -03:00
e55cc31673 Fix Vault re-init to preserve other services + simplify shutdown message
- When Vault unseal fails, only restart Vault - NOT full bootstrap
- Preserve PostgreSQL, Redis, MinIO, etc. when Vault needs re-init
- Simplify shutdown message to 3 lines with pragmatismo.com.br
- Never kill all stack processes just for Vault issues
2025-12-10 18:41:45 -03:00
c66fca3117 fix(bootstrap): Kill existing stack processes at bootstrap start
- Call kill_stack_processes() at the beginning of bootstrap()
- Add port-based killing using fuser for ports 8200, 5432, 9000, etc.
- Use -f (pattern match) instead of -x (exact) for process names
- Fix process names: 'vault server', 'llama-server' instead of 'vault', 'ollama'
- Increase wait time to 1 second for processes to terminate

This fixes issues on dev machines where old processes from a deleted
stack are still running and blocking ports.
2025-12-10 08:54:51 -03:00
8711406865 fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
951cadde4c fix(bootstrap): NEVER delete user data, suppress migration output, protect secrets
Critical safety improvements:

1. REMOVED clean_stack_directory() - NEVER delete botserver-stack
   - Contains user data, configs, databases
   - Only reset_vault_only() for Vault re-initialization

2. Single instance check
   - check_single_instance() uses .lock file with PID
   - Prevents multiple botserver processes on same stack

3. Protect existing Vault secrets
   - Check if secret exists before writing
   - Never overwrite customer credentials in distributed environments
   - Especially critical for encryption key

4. Suppress migration output
   - Use MigrationHarness directly instead of HarnessWithOutput
   - Prevents console UI corruption from migration messages
2025-12-09 09:04:56 -03:00
484534a88e fix(bootstrap): start_all() must re-bootstrap when Vault unseal fails
When start_all() detects Vault init.json is missing (unseal fails),
it now:
1. Kills all stack processes
2. Cleans the stack directory
3. Runs full bootstrap()
4. Returns (bootstrap starts all services)

Also made SecretsManager init failure in start_all() a hard error
instead of a warning - if Vault is supposed to be working but
SecretsManager can't connect, that's a fatal error.
2025-12-09 08:56:58 -03:00
b4be4dbe1b fix(bootstrap): Write .env IMMEDIATELY after Vault unseal, re-init SecretsManager
The core issue was that .env was written at the END of setup_vault(),
but other components needed Vault credentials BEFORE that.

Now:
1. Unseal Vault
2. Write .env with VAULT_ADDR and VAULT_TOKEN immediately
3. Re-initialize SecretsManager so it connects to Vault
4. THEN store secrets in Vault

This ensures SecretsManager is properly configured before any
code tries to use create_conn() or other Vault-dependent functions.
2025-12-09 08:52:19 -03:00
c1edec9913 fix(bootstrap): Skip early SecretsManager init if bootstrap incomplete, add logging
- Only initialize SecretsManager early if .env and init.json exist
- Fix error handling for bootstrap() - no longer silently ignores failures
- Add detailed logging to trace bootstrap flow
- Log component installation decisions (installed, needs_install flags)
2025-12-09 08:10:47 -03:00
c8598def62 fix(bootstrap): Initialize SecretsManager after Vault starts in start_all/ensure_services_running
When services are already configured (bootstrap completed), the code
calls start_all() or ensure_services_running() which starts Vault
but didn't initialize SecretsManager. This caused create_conn() to
fail with 'Vault not configured' even though Vault was running.

Now both functions initialize SecretsManager after Vault is unsealed,
ensuring database connections can retrieve credentials from Vault.
2025-12-09 07:58:39 -03:00
224ddd4a03 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
5669515c68 fix(bootstrap): Run migrations with direct password before Vault is set up
The bootstrap order is vault -> tables -> directory. When tables
component was processed, it tried to call establish_pg_connection()
which requires Vault, but Vault setup only happens when processing
the vault component (which is earlier in the loop but the if-block
for tables runs for the tables component, not vault).

Fix: Run migrations directly with the generated password during the
tables component setup, before Vault is configured. This avoids the
dependency on Vault being set up.
2025-12-09 07:49:01 -03:00
a755d4d13e feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00
5058b4a2c2 Fix Vault CLI mTLS issue - unset client cert env vars before CLI commands 2025-12-08 09:14:31 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
acb78b30f9 refactor: standardize component names for certificates and DNS
- Rename service names to component names:
  - postgres -> tables
  - minio -> drive
  - redis -> cache
  - qdrant -> vectordb
  - mail -> email

- Update bootstrap certificate generation with new hostnames
- Update CoreDNS zone file with component-based DNS entries
- Update CA service certificates to use component names
- Update CA directory creation to use component names

All certificates include 127.0.0.1 as SAN for IP-based mTLS access
2025-12-07 10:42:02 -03:00
6ff2b32f2c feat(bootstrap): implement mTLS for Vault access
- Add create_vault_config() function to generate config.hcl with mTLS settings
- Configure Vault to require client certificate verification
- Generate client certificate for botserver in bootstrap
- Update .env to include mTLS paths (VAULT_CACERT, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY)
- Remove unused import in tls.rs
2025-12-07 02:13:28 -03:00
e9dad539af Update dependencies and remove problematic crates
Drop image (with ravif/paste), sqlx, zitadel, and related dependencies
that were causing compilation issues. Replace image processing with
direct png crate usage. Update rcgen to 0.14 with new API changes.
Refactor CA certificate generation to use Issuer pattern.
2025-12-03 16:05:30 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
2bf53ab230 Add toml dependency for Askama custom filters config 2025-11-30 23:48:08 -03:00
11ffccaa94 - More htmx. 2025-11-29 17:27:13 -03:00
ad1cd3ee36 HTMX enters. 2025-11-29 16:29:28 -03:00
6f686565ab Update default features and add quick start guide
Replaces ui-server with console in default features, adds comprehensive
quick start documentation, implements automatic database migrations at
startup, and ensures critical services (PostgreSQL and MinIO) are
started automatically.

Key changes:
- Console UI now enable
2025-11-28 13:50:28 -03:00