Commit graph

59 commits

Author SHA1 Message Date
b1118f977d fix: Correct parameter names in tool .bas files to match database schema
- Tool 06: Change tipoExibicao to tipoDescricao (matches pedidos_uso_imagem table)
- Tool 07: Change tipoExibicao to categoriaDescricao (matches licenciamentos table)
- Both tools now compile and execute successfully with database inserts

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:51:47 +00:00
dec3a086ab fix: Extract Valkey to cache/bin/ directory
All checks were successful
BotServer CI / build (push) Successful in 8m1s
- Binaries at cache/bin/valkey-server (correct production path)
- Use --strip-components=1 for extraction
- Matches /opt/gbo/bin/botserver-stack/bin/cache/bin/

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 01:22:59 +00:00
bfbdf8396f fix: Extract Valkey binaries directly to cache/ directory
Some checks failed
BotServer CI / build (push) Has been cancelled
- Use --strip-components=2 to flatten tarball structure
- Binaries go to cache/valkey-server (not cache/bin/valkey-server)
- Matches production path: /opt/gbo/bin/botserver-stack/bin/cache/

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 01:21:20 +00:00
0af972eda6 fix: Use correct valkey-server binary path
Some checks failed
BotServer CI / build (push) Has been cancelled
- Binary is at cache/bin/bin/valkey-server after extraction
- Update exec_cmd and check_cmd to use bin/ subdirectory
- Create symlinks at parent level for convenience

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 01:20:56 +00:00
14b3a0e6c0 fix: Remove redis symlinks from Valkey installation
Some checks failed
BotServer CI / build (push) Has been cancelled
- Use valkey-server and valkey-cli directly
- No redis compatibility symlinks needed
- Simplifies installation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-15 01:20:08 +00:00
4e4aca0986 fix: Use precompiled Valkey binary from download.valkey.io
All checks were successful
BotServer CI / build (push) Successful in 7m53s
- Update 3rdparty.toml to download valkey-9.0.2-jammy-x86_64.tar.gz
- Remove source compilation approach (no gcc/make required)
- Update installer to handle precompiled binaries directly
- Fix symlinks for redis compatibility

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 23:51:33 +00:00
23c93cbb3d feat: Restore .env file creation for local Vault bootstrap
All checks were successful
BotServer CI / build (push) Successful in 9m6s
- Add TLS certificate generation to Vault post-install commands (CA, server, client, PostgreSQL)
- Add initialize_vault_local() function to handle Vault initialization for local installs
- Add ensure_env_file_exists() function to create .env when Vault already initialized
- Modify start() method to call Vault initialization after successful start (local mode)
- Fix Vault CLI flags: use -tls-skip-verify (not -skip-verify or -skip-tls-verify)

This restores the behavior where .env is automatically created with VAULT_ADDR,
VAULT_TOKEN, and VAULT_CACERT during local bootstrap, matching the LXC container
deployment behavior in facade.rs.

Fixes issue where .env file was only created for LXC deployments but not for
local installations.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 16:35:24 +00:00
5ea171d126 Refactor: Split large files into modular subdirectories
Some checks failed
BotServer CI / build (push) Failing after 1m34s
Split 20+ files over 1000 lines into focused subdirectories for better
maintainability and code organization. All changes maintain backward
compatibility through re-export wrappers.

Major splits:
- attendance/llm_assist.rs (2074→7 modules)
- basic/keywords/face_api.rs → face_api/ (7 modules)
- basic/keywords/file_operations.rs → file_ops/ (8 modules)
- basic/keywords/hear_talk.rs → hearing/ (6 modules)
- channels/wechat.rs → wechat/ (10 modules)
- channels/youtube.rs → youtube/ (5 modules)
- contacts/mod.rs → contacts_api/ (6 modules)
- core/bootstrap/mod.rs → bootstrap/ (5 modules)
- core/shared/admin.rs → admin_*.rs (5 modules)
- designer/canvas.rs → canvas_api/ (6 modules)
- designer/mod.rs → designer_api/ (6 modules)
- docs/handlers.rs → handlers_api/ (11 modules)
- drive/mod.rs → drive_handlers.rs, drive_types.rs
- learn/mod.rs → types.rs
- main.rs → main_module/ (7 modules)
- meet/webinar.rs → webinar_api/ (8 modules)
- paper/mod.rs → (10 modules)
- security/auth.rs → auth_api/ (7 modules)
- security/passkey.rs → (4 modules)
- sources/mod.rs → sources_api/ (5 modules)
- tasks/mod.rs → task_api/ (5 modules)

Stats: 38,040 deletions, 1,315 additions across 318 files

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 21:09:30 +00:00
3566a8c87f Fix batismo tool natural language processing
- Enhanced conversational input handling for batismo tool
- Improved keyword extraction and format recognition
- Fixed field extraction from informal user messages
- Better natural language understanding

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 20:17:49 +00:00
3258f3ef85 Fix PostgreSQL startup failure for non-interactive users
All checks were successful
BotServer CI / build (push) Successful in 11m58s
Changed pg_isready checks from '-U gbuser' to '-d postgres' to properly
detect PostgreSQL readiness during bootstrap. The gbuser database doesn't
exist yet during startup, causing pg_isready to fail and bootstrap to timeout.

This fixes the issue when running botserver as a non-interactive user
(e.g., sudo -u gbuser).
2026-02-05 20:40:52 -03:00
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
b103c07248 Fix migration errors and reorganize migration files
- Fixed 'relation session_kb_associations does not exist' error in core consolidated migration.
- Renamed migration directories from timestamp-based to version-based (6.0.x, 6.1.x, 6.2.x).
- Reorganized migrations into dedicated feature folders (products, dashboards, learn, video).
- Updated migration execution order in core/shared/utils.rs.
- Moves legacy migrations to 6.0.x/6.1.x and workflow to 6.2.0.
2026-01-27 13:45:54 -03:00
3c279f43e5 refactor: embed 3rdparty.toml and remove hardcoded defaults from installer
All checks were successful
GBCI / build (push) Successful in 10m36s
2026-01-26 20:48:57 -03:00
b113267aef Refactor: Genericize default organization to 'system' and update tenant paths 2026-01-26 17:00:21 -03:00
ff4e6c4fe8 fix(lxc): configure lxd-sock proxy for brother mode and update installers
Some checks failed
GBCI / build (push) Failing after 47s
2026-01-26 11:44:18 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
00acf1c76e fix: Add trusted_shell_script_arg for internal scripts
- shell_script_arg blocks $( and backticks for user input safety
- trusted_shell_script_arg allows these for internal installer scripts
- Internal scripts need shell features like command substitution
- Updated bootstrap, installer, facade, and llm modules
2026-01-09 12:13:35 -03:00
cb59ceb60f fix: Smart mTLS for Vault - use client cert if exists, plain TLS during bootstrap
- Add vault_health_check() function that checks if client certs exist
- If certs exist: use mTLS (secure, post-installation)
- If certs don't exist yet: use plain TLS (during initial bootstrap)
- This allows bootstrap to complete while maintaining mTLS security after setup
- No security hole: mTLS is enforced once certs are generated
2026-01-09 11:23:49 -03:00
63aee6f6bc Revert "fix: Disable mTLS for Vault (temporary until new binary compiled)"
This reverts commit efb458ba66588de090fe6df76cdf04515eacce52.
2026-01-09 11:19:36 -03:00
460d39f399 fix: Disable mTLS for Vault (temporary until new binary compiled)
- Remove tls_client_ca_file from vault config templates
- Remove --cert/--key from health checks
- TLS still enabled for encryption, just no client cert required
- TODO: Re-enable mTLS when binary with cert health checks is compiled
2026-01-09 11:18:30 -03:00
c59d456e25 fix: Add mTLS client certificate to all Vault health checks
- Keep mTLS enabled for security (even in dev)
- Add --cert and --key to all curl commands for Vault health checks
- Fix fetch_vault_credentials to use https and mTLS
- Fix Zitadel commands to use https with VAULT_CACERT
- All Vault communications now use proper mutual TLS
2026-01-09 10:48:13 -03:00
95fbb4f129 Revert "fix: Remove mTLS requirement from Vault config to fix health check failures"
This reverts commit 510fadf24b60c61bb39938f59d88add9b4c08168.
2026-01-09 10:45:41 -03:00
6c904f7dc9 fix: Remove mTLS requirement from Vault config to fix health check failures
- Remove tls_client_ca_file from vault config in installer.rs (Linux and macOS)
- Remove tls_client_ca_file from vault config in bootstrap/mod.rs
- TLS encryption still enabled, just no client certificate required
- Health checks now work with simple -sk curl flags
2026-01-09 10:33:11 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
7a7d455414 Change verbose startup logs from INFO to TRACE (credentials, spawn commands) 2025-12-29 10:34:25 -03:00
38f9abb7db Fix organizations foreign key reference (org_id not id) 2025-12-29 08:07:42 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
79e83dd3b8 fix: install base packages (wget, unzip, curl) in containers before download
- Vault download was failing because unzip was not installed
- Now all containers get wget, unzip, curl, ca-certificates as base packages
- Also added vault config.hcl generation in pre-install commands
2025-12-19 09:48:15 -03:00
37c1cee805 Fix: Skip starting components that are already running (minio, valkey) 2025-12-16 10:48:42 -03:00
f1f0436fdd Fix vault health check - quote URL to prevent & being interpreted as shell background 2025-12-15 18:04:53 -03:00
562d3cf0e4 Add debug logs to package manager start function 2025-12-15 17:50:03 -03:00
4486e31320 Fix llm-server default to true, convert debug eprintln to trace 2025-12-15 16:33:13 -03:00
02117e2eec Fix Zitadel masterkey to come from Vault
- Add masterkey field to gbo/directory secret in Vault during bootstrap
- Generate random 32-char masterkey if not exists
- Update Zitadel exec_cmd to read masterkey from Vault
- Remove hardcoded masterkey values
2025-12-15 13:56:50 -03:00
7647774339 Move Directory (Zitadel) to port 8300
- Directory/Zitadel: 8080 -> 8300
- BotServer API remains on 8080
- Updated all references in bootstrap, installer, oauth, config
2025-12-11 08:43:28 -03:00
696a044d75 Update llama.cpp to b7345 with platform-specific builds and checksums
- Update 3rdparty.toml: llama.cpp b4547 -> b7345 with SHA256 checksums
- Add config/llm_releases.json with complete checksums for all 24 release assets
- Fix Windows binary naming in installer.rs (win-cpu-x64, win-cpu-arm64)
- Add Vulkan detection for Windows
- Add platform-specific variants: CUDA 12/13, Vulkan, HIP, SYCL, OpenCL
2025-12-10 12:54:52 -03:00
137a5c8012 fix(llm): Restore llama.cpp b6148 which was working 2025-12-10 10:46:54 -03:00
9fefd6c1fa fix(llm): Use pre-built llama.cpp binaries, remove compilation
- Revert to pre-built binaries (b4547) instead of compiling from source
- Remove cmake/build-essential dependencies
- Remove post_install compilation commands
- Fix exec_cmd path back to {{BIN_PATH}}/llama-server
2025-12-10 09:00:05 -03:00
051c8f720c fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
af6b797572 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
8711406865 fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
185aba3631 fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
a755d4d13e feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00