Commit graph

3629 commits

Author SHA1 Message Date
a1fef6fc37 Add Cargo caching for faster builds
Some checks failed
GBCI Bundle / build-bundle (push) Failing after 26s
GBCI / build (push) Failing after 23s
2025-12-16 16:35:22 -03:00
d380368fee Add Forgejo CI workflows - botserver.yaml and botserver-bundle.yaml with cross-platform builds
Some checks failed
GBCI / build (push) Waiting to run
GBCI Bundle / build-bundle (push) Has been cancelled
2025-12-16 13:25:26 -03:00
e486689644 Fix: Skip starting components that are already running (minio, valkey) 2025-12-16 10:48:42 -03:00
2f9622ab77 Update calendar and email modules 2025-12-15 23:16:08 -03:00
9a309b4bea Fix vault health check - quote URL to prevent & being interpreted as shell background 2025-12-15 18:04:53 -03:00
62915b2164 Add debug logs to package manager start function 2025-12-15 17:50:03 -03:00
3c56b6e088 Fix Vault retry logic - wait up to 20 seconds for vault to start 2025-12-15 17:37:00 -03:00
7ccca11a25 Fix Vault startup: detect connection refused and start vault properly 2025-12-15 17:32:15 -03:00
dfe07d8449 Fix: add trace to log imports 2025-12-15 17:14:39 -03:00
4bae62b736 Fix Vault startup - add retry logic and auto-restart zombie vault processes 2025-12-15 17:11:47 -03:00
2d310a71ac Fix llm-server default to true, convert debug eprintln to trace 2025-12-15 16:33:13 -03:00
e200f47864 Fix Zitadel masterkey to come from Vault
- Add masterkey field to gbo/directory secret in Vault during bootstrap
- Generate random 32-char masterkey if not exists
- Update Zitadel exec_cmd to read masterkey from Vault
- Remove hardcoded masterkey values
2025-12-15 13:56:50 -03:00
17618f692c feat: configurable stack paths and local installer support
- Add BOTSERVER_INSTALLERS_PATH env var to use local installers without downloading
- Replace hardcoded ./botserver-stack paths with configurable stack_path
- Add stack_dir() and vault_bin() helper methods in BootstrapManager
- Add Port: 8300 to Zitadel config to fix port binding issue
- Start Directory service before setup_directory() call
- Add SKIP_LLM_SERVER env var to skip local LLM in tests
- Update template loading to check ../bottemplates and botserver-templates paths
2025-12-14 15:58:54 -03:00
7029e49f80 Add MCP server support via mcp.csv
- New mcp_directory.rs: McpCsvLoader to load MCP servers from mcp.csv
- CSV format: name,type,command,args,description,enabled,auth_type,auth_env
- Support for stdio, http, websocket, tcp connection types
- Support for api_key and bearer authentication
- Updated sources/mod.rs with MCP management API endpoints
- New sources/mcp.rs with helper functions
- MCP tools available to Tasks like BASIC keywords
2025-12-13 00:26:49 -03:00
8b004f1a89 Lowercase botserver/botbook references 2025-12-12 23:20:42 -03:00
9545db65f1 Fix truncated files, implement TODOs, remove all compilation errors and warnings
- Complete truncated auto_task.rs, autotask_api.rs, intent_compiler.rs, mcp_client.rs, safety_layer.rs
- Add missing structs: RiskFactor, ResourceUsage, TaskError, RollbackState, TaskSchedule, HealthStatus
- Add missing IntentCompiler methods: call_llm, assess_risks, estimate_resources, check_ambiguity, store_compiled_intent
- Implement SET ATTENDANT STATUS database storage
- Implement queue position calculation in transfer_to_human
- Add llm_tokens to ResourceEstimate
- Fix all unused imports and variables
- Add proper derives (Copy, PartialOrd) where needed
2025-12-12 17:33:11 -03:00
0d4797738b Update compiler and add goto_transform 2025-12-12 16:39:44 -03:00
59a74fa3ec Add llama.cpp support and embedded config for Orange Pi / Raspberry Pi 2025-12-12 13:46:04 -03:00
fed35e116a Add embedded deployment script for Orange Pi, Raspberry Pi, ARM SBCs 2025-12-12 13:43:40 -03:00
82db06e435 docs: Add Auto Task Architecture documentation 2025-12-12 12:35:40 -03:00
48f3cfb6f3 feat: Add Auto Task system - Intent Compiler, MCP Client, Safety Layer
- intent_compiler.rs: LLM-to-BASIC translation engine
- auto_task.rs: Auto-executing task data structures
- mcp_client.rs: Model Context Protocol server integration
- safety_layer.rs: Constraints, simulation, audit trail
- autotask_api.rs: HTTP API handlers for Auto Task UI
- Updated mod.rs with new modules and keyword list
2025-12-12 12:33:17 -03:00
777a3eae63 Move Directory (Zitadel) to port 8300
- Directory/Zitadel: 8080 -> 8300
- BotServer API remains on 8080
- Updated all references in bootstrap, installer, oauth, config
2025-12-11 08:43:28 -03:00
c986539fad Add missing API endpoints for UI suite screens
- src/meet/mod.rs: Add UI-compatible endpoints:
  - /api/meet/rooms (list_rooms_ui)
  - /api/meet/recent (recent_meetings)
  - /api/meet/participants (all_participants)
  - /api/meet/scheduled (scheduled_meetings)

- src/drive/mod.rs: Add UI-compatible endpoint:
  - /api/drive/list (list_drive_files_ui)

- src/calendar/mod.rs: Add UI-compatible endpoints (from previous session):
  - /api/calendar/list (list_calendars)
  - /api/calendar/upcoming (upcoming_events)

All endpoints return stub JSON responses for UI compatibility.
2025-12-10 23:50:06 -03:00
88d629e310 Revert "Add static file serving for suite UI"
This reverts commit dfd0dded23.
2025-12-10 21:52:43 -03:00
dfd0dded23 Add static file serving for suite UI
- Serve suite UI from botui (dev) or botserver-stack (installed)
- SPA fallback to index.html for client-side routing
- Search paths: ../botui/ui/suite, ./botserver-stack/ui/suite, ./ui/suite
2025-12-10 21:41:11 -03:00
30d433e409 Silence vaultrs and rustify logs polluting console 2025-12-10 19:47:39 -03:00
6307725766 Update directory config 2025-12-10 19:46:42 -03:00
49304a70b5 Fix Vault re-init to preserve other services + simplify shutdown message
- When Vault unseal fails, only restart Vault - NOT full bootstrap
- Preserve PostgreSQL, Redis, MinIO, etc. when Vault needs re-init
- Simplify shutdown message to 3 lines with pragmatismo.com.br
- Never kill all stack processes just for Vault issues
2025-12-10 18:41:45 -03:00
0bbaaf4878 Add beautiful graceful shutdown on CTRL+C
- Display 'Thank you for using General Bots!' message
- Show version, links, and farewell
- Handle both SIGTERM and SIGINT (Ctrl+C)
- Graceful 10 second timeout for in-flight requests
2025-12-10 18:31:58 -03:00
fe301d003b Fix LLM 400 error - remove invalid bot_id from messages array
The bot_id object was being pushed into the messages array sent to the LLM,
but LLM APIs expect messages to only contain role/content objects.
2025-12-10 18:29:34 -03:00
9b124156ad Bootstrap started! 6.1.0
- Add rss and scraper dependencies for web data keywords
- Add SMS keyword with priority support (low, normal, high, urgent)
- Add web_data keywords: RSS, SCRAPE, SCRAPE_ALL, SCRAPE_TABLE, SCRAPE_LINKS, SCRAPE_IMAGES
- Add ai_tools keywords: TRANSLATE, OCR, SENTIMENT, CLASSIFY
- Improve LLM server health check with better diagnostics and increased timeout
- Fix compilation errors and warnings
- Register SMS keywords in BASIC engine
2025-12-10 18:22:02 -03:00
b6d3e0a2d5 Update llama.cpp to b7345 with platform-specific builds and checksums
- Update 3rdparty.toml: llama.cpp b4547 -> b7345 with SHA256 checksums
- Add config/llm_releases.json with complete checksums for all 24 release assets
- Fix Windows binary naming in installer.rs (win-cpu-x64, win-cpu-arm64)
- Add Vulkan detection for Windows
- Add platform-specific variants: CUDA 12/13, Vulkan, HIP, SYCL, OpenCL
2025-12-10 12:54:52 -03:00
a8863d9051 fix(llm): Restore llama.cpp b6148 which was working 2025-12-10 10:46:54 -03:00
b4aeb5e9cc fix(llm): Use pre-built llama.cpp binaries, remove compilation
- Revert to pre-built binaries (b4547) instead of compiling from source
- Remove cmake/build-essential dependencies
- Remove post_install compilation commands
- Fix exec_cmd path back to {{BIN_PATH}}/llama-server
2025-12-10 09:00:05 -03:00
0047ad3937 fix(bootstrap): Kill existing stack processes at bootstrap start
- Call kill_stack_processes() at the beginning of bootstrap()
- Add port-based killing using fuser for ports 8200, 5432, 9000, etc.
- Use -f (pattern match) instead of -x (exact) for process names
- Fix process names: 'vault server', 'llama-server' instead of 'vault', 'ollama'
- Increase wait time to 1 second for processes to terminate

This fixes issues on dev machines where old processes from a deleted
stack are still running and blocking ports.
2025-12-10 08:54:51 -03:00
e443470295 fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
c95b56f257 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
2da5f0ccdf fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
0b9ad6c80d fix(bootstrap): NEVER delete user data, suppress migration output, protect secrets
Critical safety improvements:

1. REMOVED clean_stack_directory() - NEVER delete botserver-stack
   - Contains user data, configs, databases
   - Only reset_vault_only() for Vault re-initialization

2. Single instance check
   - check_single_instance() uses .lock file with PID
   - Prevents multiple botserver processes on same stack

3. Protect existing Vault secrets
   - Check if secret exists before writing
   - Never overwrite customer credentials in distributed environments
   - Especially critical for encryption key

4. Suppress migration output
   - Use MigrationHarness directly instead of HarnessWithOutput
   - Prevents console UI corruption from migration messages
2025-12-09 09:04:56 -03:00
a2b091914f fix(bootstrap): start_all() must re-bootstrap when Vault unseal fails
When start_all() detects Vault init.json is missing (unseal fails),
it now:
1. Kills all stack processes
2. Cleans the stack directory
3. Runs full bootstrap()
4. Returns (bootstrap starts all services)

Also made SecretsManager init failure in start_all() a hard error
instead of a warning - if Vault is supposed to be working but
SecretsManager can't connect, that's a fatal error.
2025-12-09 08:56:58 -03:00
1e98083d13 fix(bootstrap): Write .env IMMEDIATELY after Vault unseal, re-init SecretsManager
The core issue was that .env was written at the END of setup_vault(),
but other components needed Vault credentials BEFORE that.

Now:
1. Unseal Vault
2. Write .env with VAULT_ADDR and VAULT_TOKEN immediately
3. Re-initialize SecretsManager so it connects to Vault
4. THEN store secrets in Vault

This ensures SecretsManager is properly configured before any
code tries to use create_conn() or other Vault-dependent functions.
2025-12-09 08:52:19 -03:00
87f4052b82 debug: Add explicit branch logging to trace bootstrap flow 2025-12-09 08:26:29 -03:00
4ea3b76adc fix(bootstrap): Skip early SecretsManager init if bootstrap incomplete, add logging
- Only initialize SecretsManager early if .env and init.json exist
- Fix error handling for bootstrap() - no longer silently ignores failures
- Add detailed logging to trace bootstrap flow
- Log component installation decisions (installed, needs_install flags)
2025-12-09 08:10:47 -03:00
715a60315e feat(console): Add scrolling support for System Logs and Editor panels
System Logs:
- Add scroll_offset tracking with auto-scroll to bottom on new logs
- Up/Down/j/k keys to scroll line by line
- PageUp/PageDown for page scrolling
- Home/End to jump to top/bottom
- Show scroll indicators in title: [^v], [SCROLL] when not auto-scrolling
- Display log count in title

Editor:
- Fix scroll_offset to follow cursor when moving up/down
- Add PageUp/PageDown for faster navigation
- Add Ctrl+Home/Ctrl+End to jump to start/end of file
- ensure_cursor_visible() keeps cursor in view

Tab Navigation:
- FileTree -> Editor (if open) or Logs -> Chat -> back to start
- Consistent cycling through all panels
2025-12-09 08:06:30 -03:00
824b12365b fix(bootstrap): Initialize SecretsManager after Vault starts in start_all/ensure_services_running
When services are already configured (bootstrap completed), the code
calls start_all() or ensure_services_running() which starts Vault
but didn't initialize SecretsManager. This caused create_conn() to
fail with 'Vault not configured' even though Vault was running.

Now both functions initialize SecretsManager after Vault is unsealed,
ensuring database connections can retrieve credentials from Vault.
2025-12-09 07:58:39 -03:00
583e764bb9 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
b204aebd50 fix(bootstrap): Run migrations with direct password before Vault is set up
The bootstrap order is vault -> tables -> directory. When tables
component was processed, it tried to call establish_pg_connection()
which requires Vault, but Vault setup only happens when processing
the vault component (which is earlier in the loop but the if-block
for tables runs for the tables component, not vault).

Fix: Run migrations directly with the generated password during the
tables component setup, before Vault is configured. This avoids the
dependency on Vault being set up.
2025-12-09 07:49:01 -03:00
49d1d0cf2e fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
ffdc39a6fd config: Enable llm-server by default for clean stack installs 2025-12-09 00:28:46 -03:00
21855fab99 fix(llm): Respect llm-server config flag to skip local server startup
When llm-server=false in bot_configuration, the code now skips
attempting to start local llama-server processes. This prevents
the 60-attempt timeout error when using external LLM endpoints
or when local LLM serving is intentionally disabled.
2025-12-08 23:39:01 -03:00