Commit graph

3613 commits

Author SHA1 Message Date
0d4797738b Update compiler and add goto_transform 2025-12-12 16:39:44 -03:00
59a74fa3ec Add llama.cpp support and embedded config for Orange Pi / Raspberry Pi 2025-12-12 13:46:04 -03:00
fed35e116a Add embedded deployment script for Orange Pi, Raspberry Pi, ARM SBCs 2025-12-12 13:43:40 -03:00
82db06e435 docs: Add Auto Task Architecture documentation 2025-12-12 12:35:40 -03:00
48f3cfb6f3 feat: Add Auto Task system - Intent Compiler, MCP Client, Safety Layer
- intent_compiler.rs: LLM-to-BASIC translation engine
- auto_task.rs: Auto-executing task data structures
- mcp_client.rs: Model Context Protocol server integration
- safety_layer.rs: Constraints, simulation, audit trail
- autotask_api.rs: HTTP API handlers for Auto Task UI
- Updated mod.rs with new modules and keyword list
2025-12-12 12:33:17 -03:00
777a3eae63 Move Directory (Zitadel) to port 8300
- Directory/Zitadel: 8080 -> 8300
- BotServer API remains on 8080
- Updated all references in bootstrap, installer, oauth, config
2025-12-11 08:43:28 -03:00
c986539fad Add missing API endpoints for UI suite screens
- src/meet/mod.rs: Add UI-compatible endpoints:
  - /api/meet/rooms (list_rooms_ui)
  - /api/meet/recent (recent_meetings)
  - /api/meet/participants (all_participants)
  - /api/meet/scheduled (scheduled_meetings)

- src/drive/mod.rs: Add UI-compatible endpoint:
  - /api/drive/list (list_drive_files_ui)

- src/calendar/mod.rs: Add UI-compatible endpoints (from previous session):
  - /api/calendar/list (list_calendars)
  - /api/calendar/upcoming (upcoming_events)

All endpoints return stub JSON responses for UI compatibility.
2025-12-10 23:50:06 -03:00
88d629e310 Revert "Add static file serving for suite UI"
This reverts commit dfd0dded23.
2025-12-10 21:52:43 -03:00
dfd0dded23 Add static file serving for suite UI
- Serve suite UI from botui (dev) or botserver-stack (installed)
- SPA fallback to index.html for client-side routing
- Search paths: ../botui/ui/suite, ./botserver-stack/ui/suite, ./ui/suite
2025-12-10 21:41:11 -03:00
30d433e409 Silence vaultrs and rustify logs polluting console 2025-12-10 19:47:39 -03:00
6307725766 Update directory config 2025-12-10 19:46:42 -03:00
49304a70b5 Fix Vault re-init to preserve other services + simplify shutdown message
- When Vault unseal fails, only restart Vault - NOT full bootstrap
- Preserve PostgreSQL, Redis, MinIO, etc. when Vault needs re-init
- Simplify shutdown message to 3 lines with pragmatismo.com.br
- Never kill all stack processes just for Vault issues
2025-12-10 18:41:45 -03:00
0bbaaf4878 Add beautiful graceful shutdown on CTRL+C
- Display 'Thank you for using General Bots!' message
- Show version, links, and farewell
- Handle both SIGTERM and SIGINT (Ctrl+C)
- Graceful 10 second timeout for in-flight requests
2025-12-10 18:31:58 -03:00
fe301d003b Fix LLM 400 error - remove invalid bot_id from messages array
The bot_id object was being pushed into the messages array sent to the LLM,
but LLM APIs expect messages to only contain role/content objects.
2025-12-10 18:29:34 -03:00
9b124156ad Bootstrap started! 6.1.0
- Add rss and scraper dependencies for web data keywords
- Add SMS keyword with priority support (low, normal, high, urgent)
- Add web_data keywords: RSS, SCRAPE, SCRAPE_ALL, SCRAPE_TABLE, SCRAPE_LINKS, SCRAPE_IMAGES
- Add ai_tools keywords: TRANSLATE, OCR, SENTIMENT, CLASSIFY
- Improve LLM server health check with better diagnostics and increased timeout
- Fix compilation errors and warnings
- Register SMS keywords in BASIC engine
2025-12-10 18:22:02 -03:00
b6d3e0a2d5 Update llama.cpp to b7345 with platform-specific builds and checksums
- Update 3rdparty.toml: llama.cpp b4547 -> b7345 with SHA256 checksums
- Add config/llm_releases.json with complete checksums for all 24 release assets
- Fix Windows binary naming in installer.rs (win-cpu-x64, win-cpu-arm64)
- Add Vulkan detection for Windows
- Add platform-specific variants: CUDA 12/13, Vulkan, HIP, SYCL, OpenCL
2025-12-10 12:54:52 -03:00
a8863d9051 fix(llm): Restore llama.cpp b6148 which was working 2025-12-10 10:46:54 -03:00
b4aeb5e9cc fix(llm): Use pre-built llama.cpp binaries, remove compilation
- Revert to pre-built binaries (b4547) instead of compiling from source
- Remove cmake/build-essential dependencies
- Remove post_install compilation commands
- Fix exec_cmd path back to {{BIN_PATH}}/llama-server
2025-12-10 09:00:05 -03:00
0047ad3937 fix(bootstrap): Kill existing stack processes at bootstrap start
- Call kill_stack_processes() at the beginning of bootstrap()
- Add port-based killing using fuser for ports 8200, 5432, 9000, etc.
- Use -f (pattern match) instead of -x (exact) for process names
- Fix process names: 'vault server', 'llama-server' instead of 'vault', 'ollama'
- Increase wait time to 1 second for processes to terminate

This fixes issues on dev machines where old processes from a deleted
stack are still running and blocking ports.
2025-12-10 08:54:51 -03:00
e443470295 fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
c95b56f257 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
2da5f0ccdf fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
0b9ad6c80d fix(bootstrap): NEVER delete user data, suppress migration output, protect secrets
Critical safety improvements:

1. REMOVED clean_stack_directory() - NEVER delete botserver-stack
   - Contains user data, configs, databases
   - Only reset_vault_only() for Vault re-initialization

2. Single instance check
   - check_single_instance() uses .lock file with PID
   - Prevents multiple botserver processes on same stack

3. Protect existing Vault secrets
   - Check if secret exists before writing
   - Never overwrite customer credentials in distributed environments
   - Especially critical for encryption key

4. Suppress migration output
   - Use MigrationHarness directly instead of HarnessWithOutput
   - Prevents console UI corruption from migration messages
2025-12-09 09:04:56 -03:00
a2b091914f fix(bootstrap): start_all() must re-bootstrap when Vault unseal fails
When start_all() detects Vault init.json is missing (unseal fails),
it now:
1. Kills all stack processes
2. Cleans the stack directory
3. Runs full bootstrap()
4. Returns (bootstrap starts all services)

Also made SecretsManager init failure in start_all() a hard error
instead of a warning - if Vault is supposed to be working but
SecretsManager can't connect, that's a fatal error.
2025-12-09 08:56:58 -03:00
1e98083d13 fix(bootstrap): Write .env IMMEDIATELY after Vault unseal, re-init SecretsManager
The core issue was that .env was written at the END of setup_vault(),
but other components needed Vault credentials BEFORE that.

Now:
1. Unseal Vault
2. Write .env with VAULT_ADDR and VAULT_TOKEN immediately
3. Re-initialize SecretsManager so it connects to Vault
4. THEN store secrets in Vault

This ensures SecretsManager is properly configured before any
code tries to use create_conn() or other Vault-dependent functions.
2025-12-09 08:52:19 -03:00
87f4052b82 debug: Add explicit branch logging to trace bootstrap flow 2025-12-09 08:26:29 -03:00
4ea3b76adc fix(bootstrap): Skip early SecretsManager init if bootstrap incomplete, add logging
- Only initialize SecretsManager early if .env and init.json exist
- Fix error handling for bootstrap() - no longer silently ignores failures
- Add detailed logging to trace bootstrap flow
- Log component installation decisions (installed, needs_install flags)
2025-12-09 08:10:47 -03:00
715a60315e feat(console): Add scrolling support for System Logs and Editor panels
System Logs:
- Add scroll_offset tracking with auto-scroll to bottom on new logs
- Up/Down/j/k keys to scroll line by line
- PageUp/PageDown for page scrolling
- Home/End to jump to top/bottom
- Show scroll indicators in title: [^v], [SCROLL] when not auto-scrolling
- Display log count in title

Editor:
- Fix scroll_offset to follow cursor when moving up/down
- Add PageUp/PageDown for faster navigation
- Add Ctrl+Home/Ctrl+End to jump to start/end of file
- ensure_cursor_visible() keeps cursor in view

Tab Navigation:
- FileTree -> Editor (if open) or Logs -> Chat -> back to start
- Consistent cycling through all panels
2025-12-09 08:06:30 -03:00
824b12365b fix(bootstrap): Initialize SecretsManager after Vault starts in start_all/ensure_services_running
When services are already configured (bootstrap completed), the code
calls start_all() or ensure_services_running() which starts Vault
but didn't initialize SecretsManager. This caused create_conn() to
fail with 'Vault not configured' even though Vault was running.

Now both functions initialize SecretsManager after Vault is unsealed,
ensuring database connections can retrieve credentials from Vault.
2025-12-09 07:58:39 -03:00
583e764bb9 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
b204aebd50 fix(bootstrap): Run migrations with direct password before Vault is set up
The bootstrap order is vault -> tables -> directory. When tables
component was processed, it tried to call establish_pg_connection()
which requires Vault, but Vault setup only happens when processing
the vault component (which is earlier in the loop but the if-block
for tables runs for the tables component, not vault).

Fix: Run migrations directly with the generated password during the
tables component setup, before Vault is configured. This avoids the
dependency on Vault being set up.
2025-12-09 07:49:01 -03:00
49d1d0cf2e fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
ffdc39a6fd config: Enable llm-server by default for clean stack installs 2025-12-09 00:28:46 -03:00
21855fab99 fix(llm): Respect llm-server config flag to skip local server startup
When llm-server=false in bot_configuration, the code now skips
attempting to start local llama-server processes. This prevents
the 60-attempt timeout error when using external LLM endpoints
or when local LLM serving is intentionally disabled.
2025-12-08 23:39:01 -03:00
f3e38d8d8b feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00
5f71614451 fix: remove duplicate logger init causing 'cannot set logger' error
Logger is already initialized in main.rs, cli.rs was trying to
initialize it again when CLI commands were run.
2025-12-08 17:59:43 -03:00
d73d782659 fix: preserve cached files in botserver-installers after extraction
- Don't delete archive files from cache after tar/zip extraction
- Copy binaries from cache instead of moving them
- Cached files are now preserved for offline installation reuse
2025-12-08 14:19:55 -03:00
89e92a4739 feat: add offline installer cache and health endpoints
- Add /health and /api/health endpoints for botui connectivity
- Create 3rdparty.toml with all download URLs for offline bundles
- Add botserver-installers/ cache directory for downloaded files
- Implement DownloadCache module with:
  - Automatic cache lookup before downloading
  - Support for pre-populated offline bundles
  - SHA256 checksum verification (optional)
  - Cache management utilities (list, clear, size)
- Update download_and_install to use cache system
- Data files (models) also cached for reuse

Cache behavior:
- First run: downloads to botserver-installers/
- Subsequent runs: uses cached files
- Delete botserver-stack/ without losing downloads
- Pre-populate cache for fully offline installation
2025-12-08 14:08:49 -03:00
89ff716bef Fix Vault CLI mTLS issue - unset client cert env vars before CLI commands 2025-12-08 09:14:31 -03:00
f7ccc95e60 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
59f3bb8c7e refactor: standardize component names for certificates and DNS
- Rename service names to component names:
  - postgres -> tables
  - minio -> drive
  - redis -> cache
  - qdrant -> vectordb
  - mail -> email

- Update bootstrap certificate generation with new hostnames
- Update CoreDNS zone file with component-based DNS entries
- Update CA service certificates to use component names
- Update CA directory creation to use component names

All certificates include 127.0.0.1 as SAN for IP-based mTLS access
2025-12-07 10:42:02 -03:00
b225d7e83c chore: Update secrets module and remove task files 2025-12-07 09:57:05 -03:00
cfa1a01bc9 feat(bootstrap): implement mTLS for Vault access
- Add create_vault_config() function to generate config.hcl with mTLS settings
- Configure Vault to require client certificate verification
- Generate client certificate for botserver in bootstrap
- Update .env to include mTLS paths (VAULT_CACERT, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY)
- Remove unused import in tls.rs
2025-12-07 02:13:28 -03:00
41f5847f56 Add --stack-path CLI option for custom stack path
Allows overriding the default botserver-stack location via:
- --stack-path /path/to/stack CLI argument
- BOTSERVER_STACK_PATH environment variable

Useful for testing botserver bootstrap in isolated temp directories.
2025-12-06 14:55:42 -03:00
67c971557e Update: General project updates 2025-12-06 11:09:12 -03:00
38cb30276f Add test infrastructure: AppState::default(), mock providers, fix attendance tests
- Add Default impl for AppState with mock services for testing
- Add MockLLMProvider for tests that need LLM without real API
- Add create_mock_auth_service() for Zitadel testing
- Add test_utils.rs with TestAppStateBuilder, MockChannelAdapter
- Enable rhai 'sync' feature to fix Send+Sync for Dynamic types
- Fix attendance.rs tests to use pure functions (no DB required)
- Fix on_error.rs tests to use String instead of i32
- Remove unused imports in attendance.rs

All tests pass, 0 warnings, 0 errors.
2025-12-05 16:43:14 -03:00
1b669d4c11 feat(attendance): Multi-channel attendant config (email, phone, teams, google)
- Add email, teams, google columns to attendant.csv
- Add find_attendant_by_identifier() for multi-channel lookup
- Add find_attendants_by_channel() and find_attendants_by_department()
- Implement handle_status_command with database persistence
- Implement handle_transfer_command with actual transfer logic
- Update AttendantCSV struct with all new fields
2025-12-05 14:54:55 -03:00
bde3244ce9 feat(attendance): Add LLM-assisted attendant features
- Real-time tips when customer messages arrive
- Message polishing with one click
- Smart reply generation (3 contextual suggestions)
- Auto-summary when attendant takes conversation
- LLM-powered sentiment analysis with escalation warnings

WhatsApp Attendant Commands:
- /queue, /take, /status, /transfer, /resolve
- /tips, /polish, /replies, /summary, /help
- Portuguese versions: /fila, /pegar, /dicas, /polir, /respostas, /resumo

Config options (config.csv):
- attendant-llm-tips
- attendant-polish-message
- attendant-smart-replies
- attendant-auto-summary
- attendant-sentiment-analysis

API Endpoints:
- POST /api/attendance/llm/tips
- POST /api/attendance/llm/polish
- POST /api/attendance/llm/smart-replies
- GET /api/attendance/llm/summary/{session_id}
- POST /api/attendance/llm/sentiment
- GET /api/attendance/llm/config/{bot_id}

Uses bot's system prompt for consistency between bot and human-assisted responses.
2025-12-05 13:47:15 -03:00
415c7cce77 refactor: rename redis-cache feature to cache
- Simplified feature name from 'redis-cache' to 'cache'
- Updated Cargo.toml feature definitions
- Updated state.rs cfg attributes
- More concise naming
2025-12-05 12:09:02 -03:00
b1193afda2 feat: unified keywords with spaces, ON ERROR RESUME NEXT, unified DELETE
Keywords now use spaces instead of underscores:
- SEND MAIL (was SEND_MAIL)
- GENERATE PDF (was GENERATE_PDF)
- MERGE PDF (was MERGE_PDF)
- SET HEADER (was SET_HEADER)
- CLEAR HEADERS (was CLEAR_HEADERS)

New ON ERROR RESUME NEXT implementation:
- ON ERROR RESUME NEXT - enable error trapping
- ON ERROR GOTO 0 - disable error trapping
- CLEAR ERROR - clear error state
- ERROR MESSAGE - get last error message
- ERR - get error number

Unified DELETE keyword:
- DELETE url - HTTP DELETE (auto-detected)
- DELETE table, filter - Database DELETE
- DELETE path - File DELETE

Changes:
- errors/on_error.rs: New VB-style error handling
- errors/mod.rs: Include on_error module
- send_mail.rs: SEND MAIL with spaces
- file_operations.rs: GENERATE PDF, MERGE PDF with spaces
- data_operations.rs: Unified DELETE with auto-detection
- http_operations.rs: Cleaned up DELETE HTTP
- compiler/mod.rs: Removed underscore normalization
- mod.rs: Updated command list
2025-12-05 09:55:13 -03:00