Commit graph

112 commits

Author SHA1 Message Date
512ddc5f75 Fix LLM 400 error - remove invalid bot_id from messages array
The bot_id object was being pushed into the messages array sent to the LLM,
but LLM APIs expect messages to only contain role/content objects.
2025-12-10 18:29:34 -03:00
fda88c0d94 Bootstrap started! 6.1.0
- Add rss and scraper dependencies for web data keywords
- Add SMS keyword with priority support (low, normal, high, urgent)
- Add web_data keywords: RSS, SCRAPE, SCRAPE_ALL, SCRAPE_TABLE, SCRAPE_LINKS, SCRAPE_IMAGES
- Add ai_tools keywords: TRANSLATE, OCR, SENTIMENT, CLASSIFY
- Improve LLM server health check with better diagnostics and increased timeout
- Fix compilation errors and warnings
- Register SMS keywords in BASIC engine
2025-12-10 18:22:02 -03:00
696a044d75 Update llama.cpp to b7345 with platform-specific builds and checksums
- Update 3rdparty.toml: llama.cpp b4547 -> b7345 with SHA256 checksums
- Add config/llm_releases.json with complete checksums for all 24 release assets
- Fix Windows binary naming in installer.rs (win-cpu-x64, win-cpu-arm64)
- Add Vulkan detection for Windows
- Add platform-specific variants: CUDA 12/13, Vulkan, HIP, SYCL, OpenCL
2025-12-10 12:54:52 -03:00
137a5c8012 fix(llm): Restore llama.cpp b6148 which was working 2025-12-10 10:46:54 -03:00
9fefd6c1fa fix(llm): Use pre-built llama.cpp binaries, remove compilation
- Revert to pre-built binaries (b4547) instead of compiling from source
- Remove cmake/build-essential dependencies
- Remove post_install compilation commands
- Fix exec_cmd path back to {{BIN_PATH}}/llama-server
2025-12-10 09:00:05 -03:00
c66fca3117 fix(bootstrap): Kill existing stack processes at bootstrap start
- Call kill_stack_processes() at the beginning of bootstrap()
- Add port-based killing using fuser for ports 8200, 5432, 9000, etc.
- Use -f (pattern match) instead of -x (exact) for process names
- Fix process names: 'vault server', 'llama-server' instead of 'vault', 'ollama'
- Increase wait time to 1 second for processes to terminate

This fixes issues on dev machines where old processes from a deleted
stack are still running and blocking ports.
2025-12-10 08:54:51 -03:00
051c8f720c fix(llm): Compile llama.cpp from source for CPU compatibility
Instead of downloading pre-built binaries (which may require AVX2),
compile llama.cpp from source during installation. This ensures:
- Works on older CPUs (Sandy Bridge, Haswell, etc.)
- Uses GGML_NATIVE=ON to optimize for the current CPU
- Binary path updated to build/bin/llama-server

Reverts the AVX2 detection that was incorrectly disabling LLM.
2025-12-10 08:43:27 -03:00
af6b797572 fix(llm): Detect AVX2 support and gracefully disable LLM on incompatible CPUs
- Add cpu_supports_avx2() function to check /proc/cpuinfo for AVX2 flag
- Skip LLM binary download on CPUs without AVX2 (pre-built llama.cpp requires it)
- Add runtime check for llama-server binary compatibility (catches 'Illegal instruction')
- Provide helpful error messages about alternatives (compile from source or use external API)
- Sandy Bridge (2nd gen i7) and older CPUs now gracefully skip local LLM
2025-12-10 08:35:35 -03:00
8711406865 fix(bootstrap): Improve Vault startup diagnostics and error handling
- Fix create_conn/establish_pg_connection to return Result instead of panicking
- Fix AppConfig::from_env to not require database access (circular dependency)
- Add #[cfg(test)] to AppState Default impl to prevent accidental panic
- Add extensive debug logging for Vault startup troubleshooting
- Remove Stdio::null() from start() to allow shell redirections to work
- Add direct vault start test in bootstrap for debugging
- Make Vault setup failure fatal (was silently continuing)
2025-12-10 08:30:49 -03:00
951cadde4c fix(bootstrap): NEVER delete user data, suppress migration output, protect secrets
Critical safety improvements:

1. REMOVED clean_stack_directory() - NEVER delete botserver-stack
   - Contains user data, configs, databases
   - Only reset_vault_only() for Vault re-initialization

2. Single instance check
   - check_single_instance() uses .lock file with PID
   - Prevents multiple botserver processes on same stack

3. Protect existing Vault secrets
   - Check if secret exists before writing
   - Never overwrite customer credentials in distributed environments
   - Especially critical for encryption key

4. Suppress migration output
   - Use MigrationHarness directly instead of HarnessWithOutput
   - Prevents console UI corruption from migration messages
2025-12-09 09:04:56 -03:00
484534a88e fix(bootstrap): start_all() must re-bootstrap when Vault unseal fails
When start_all() detects Vault init.json is missing (unseal fails),
it now:
1. Kills all stack processes
2. Cleans the stack directory
3. Runs full bootstrap()
4. Returns (bootstrap starts all services)

Also made SecretsManager init failure in start_all() a hard error
instead of a warning - if Vault is supposed to be working but
SecretsManager can't connect, that's a fatal error.
2025-12-09 08:56:58 -03:00
b4be4dbe1b fix(bootstrap): Write .env IMMEDIATELY after Vault unseal, re-init SecretsManager
The core issue was that .env was written at the END of setup_vault(),
but other components needed Vault credentials BEFORE that.

Now:
1. Unseal Vault
2. Write .env with VAULT_ADDR and VAULT_TOKEN immediately
3. Re-initialize SecretsManager so it connects to Vault
4. THEN store secrets in Vault

This ensures SecretsManager is properly configured before any
code tries to use create_conn() or other Vault-dependent functions.
2025-12-09 08:52:19 -03:00
c1edec9913 fix(bootstrap): Skip early SecretsManager init if bootstrap incomplete, add logging
- Only initialize SecretsManager early if .env and init.json exist
- Fix error handling for bootstrap() - no longer silently ignores failures
- Add detailed logging to trace bootstrap flow
- Log component installation decisions (installed, needs_install flags)
2025-12-09 08:10:47 -03:00
c8598def62 fix(bootstrap): Initialize SecretsManager after Vault starts in start_all/ensure_services_running
When services are already configured (bootstrap completed), the code
calls start_all() or ensure_services_running() which starts Vault
but didn't initialize SecretsManager. This caused create_conn() to
fail with 'Vault not configured' even though Vault was running.

Now both functions initialize SecretsManager after Vault is unsealed,
ensuring database connections can retrieve credentials from Vault.
2025-12-09 07:58:39 -03:00
224ddd4a03 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
5669515c68 fix(bootstrap): Run migrations with direct password before Vault is set up
The bootstrap order is vault -> tables -> directory. When tables
component was processed, it tried to call establish_pg_connection()
which requires Vault, but Vault setup only happens when processing
the vault component (which is earlier in the loop but the if-block
for tables runs for the tables component, not vault).

Fix: Run migrations directly with the generated password during the
tables component setup, before Vault is configured. This avoids the
dependency on Vault being set up.
2025-12-09 07:49:01 -03:00
185aba3631 fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
a755d4d13e feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00
05fa0e95c5 fix: remove duplicate logger init causing 'cannot set logger' error
Logger is already initialized in main.rs, cli.rs was trying to
initialize it again when CLI commands were run.
2025-12-08 17:59:43 -03:00
559154b923 fix: preserve cached files in botserver-installers after extraction
- Don't delete archive files from cache after tar/zip extraction
- Copy binaries from cache instead of moving them
- Cached files are now preserved for offline installation reuse
2025-12-08 14:19:55 -03:00
eed537ac42 feat: add offline installer cache and health endpoints
- Add /health and /api/health endpoints for botui connectivity
- Create 3rdparty.toml with all download URLs for offline bundles
- Add botserver-installers/ cache directory for downloaded files
- Implement DownloadCache module with:
  - Automatic cache lookup before downloading
  - Support for pre-populated offline bundles
  - SHA256 checksum verification (optional)
  - Cache management utilities (list, clear, size)
- Update download_and_install to use cache system
- Data files (models) also cached for reuse

Cache behavior:
- First run: downloads to botserver-installers/
- Subsequent runs: uses cached files
- Delete botserver-stack/ without losing downloads
- Pre-populate cache for fully offline installation
2025-12-08 14:08:49 -03:00
5058b4a2c2 Fix Vault CLI mTLS issue - unset client cert env vars before CLI commands 2025-12-08 09:14:31 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
acb78b30f9 refactor: standardize component names for certificates and DNS
- Rename service names to component names:
  - postgres -> tables
  - minio -> drive
  - redis -> cache
  - qdrant -> vectordb
  - mail -> email

- Update bootstrap certificate generation with new hostnames
- Update CoreDNS zone file with component-based DNS entries
- Update CA service certificates to use component names
- Update CA directory creation to use component names

All certificates include 127.0.0.1 as SAN for IP-based mTLS access
2025-12-07 10:42:02 -03:00
c3ab7fe1a5 chore: Update secrets module and remove task files 2025-12-07 09:57:05 -03:00
6ff2b32f2c feat(bootstrap): implement mTLS for Vault access
- Add create_vault_config() function to generate config.hcl with mTLS settings
- Configure Vault to require client certificate verification
- Generate client certificate for botserver in bootstrap
- Update .env to include mTLS paths (VAULT_CACERT, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY)
- Remove unused import in tls.rs
2025-12-07 02:13:28 -03:00
b611f8c535 Add --stack-path CLI option for custom stack path
Allows overriding the default botserver-stack location via:
- --stack-path /path/to/stack CLI argument
- BOTSERVER_STACK_PATH environment variable

Useful for testing botserver bootstrap in isolated temp directories.
2025-12-06 14:55:42 -03:00
2abd875aec Update: General project updates 2025-12-06 11:09:12 -03:00
c2aa82733a Add test infrastructure: AppState::default(), mock providers, fix attendance tests
- Add Default impl for AppState with mock services for testing
- Add MockLLMProvider for tests that need LLM without real API
- Add create_mock_auth_service() for Zitadel testing
- Add test_utils.rs with TestAppStateBuilder, MockChannelAdapter
- Enable rhai 'sync' feature to fix Send+Sync for Dynamic types
- Fix attendance.rs tests to use pure functions (no DB required)
- Fix on_error.rs tests to use String instead of i32
- Remove unused imports in attendance.rs

All tests pass, 0 warnings, 0 errors.
2025-12-05 16:43:14 -03:00
5d93505012 feat(attendance): Add LLM-assisted attendant features
- Real-time tips when customer messages arrive
- Message polishing with one click
- Smart reply generation (3 contextual suggestions)
- Auto-summary when attendant takes conversation
- LLM-powered sentiment analysis with escalation warnings

WhatsApp Attendant Commands:
- /queue, /take, /status, /transfer, /resolve
- /tips, /polish, /replies, /summary, /help
- Portuguese versions: /fila, /pegar, /dicas, /polir, /respostas, /resumo

Config options (config.csv):
- attendant-llm-tips
- attendant-polish-message
- attendant-smart-replies
- attendant-auto-summary
- attendant-sentiment-analysis

API Endpoints:
- POST /api/attendance/llm/tips
- POST /api/attendance/llm/polish
- POST /api/attendance/llm/smart-replies
- GET /api/attendance/llm/summary/{session_id}
- POST /api/attendance/llm/sentiment
- GET /api/attendance/llm/config/{bot_id}

Uses bot's system prompt for consistency between bot and human-assisted responses.
2025-12-05 13:47:15 -03:00
6882d52552 refactor: rename redis-cache feature to cache
- Simplified feature name from 'redis-cache' to 'cache'
- Updated Cargo.toml feature definitions
- Updated state.rs cfg attributes
- More concise naming
2025-12-05 12:09:02 -03:00
6c4ecea2fc feat(auth): Add OAuth login for Google, Discord, Reddit, Twitter, Microsoft, Facebook
- Create core/oauth module with OAuthProvider enum and shared types
- Implement providers.rs with auth URLs, token exchange, user info endpoints
- Add routes for /auth/oauth/providers, /auth/oauth/{provider}, and callbacks
- Update login.html with OAuth button grid and dynamic provider loading
- Add OAuth config settings to config.csv with setup documentation and links
- Uses HTMX for login form, minimal JS for OAuth provider visibility
2025-12-04 22:53:40 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
d088490f41 - botbook extracted from. 2025-12-03 19:59:27 -03:00
b2f1ea40e4 Add comprehensive documentation for GB templates and configuration
modules

Add detailed README documentation for 15+ bot templates including:
- Platform analytics, BI reporting, and web crawler templates
- CRM, contacts, and marketing automation templates
- Legal document processing and office productivity templates
- LLM tools, LLM server, and API client integration templates
- Reminder management and meta-template for creating new templates

Add new Rust configuration modules:
- BM25 config for Tantivy-based sparse
2025-12-03 16:05:50 -03:00
e9dad539af Update dependencies and remove problematic crates
Drop image (with ravif/paste), sqlx, zitadel, and related dependencies
that were causing compilation issues. Replace image processing with
direct png crate usage. Update rcgen to 0.14 with new API changes.
Refactor CA certificate generation to use Issuer pattern.
2025-12-03 16:05:30 -03:00
340ac8840a - New templates. 2025-12-03 07:15:54 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
2bf53ab230 Add toml dependency for Askama custom filters config 2025-11-30 23:48:08 -03:00
16a792b5cb Add Suite app documentation, templates, and Askama config
- Add askama.toml for template configuration (ui/ directory)
- Add Suite app documentation with flow diagrams (SVG)
  - App launcher, chat flow, drive flow, tasks flow
  - Individual app docs: chat, drive, tasks, mail, etc.
- Add HTML templates for Suite apps
  - Base template with header and app launcher
  - Auth login page
  - Chat, Drive, Mail, Meet, Tasks templates
  - Partial templates for messages, sessions, notifications
- Add Extensions type to AppState for type-erased storage
- Add mTLS module for service-to-service authentication
- Update web handlers to use new template paths (suite/)
- Fix auth module to avoid axum-extra TypedHeader dependency
2025-11-30 21:00:48 -03:00
48288e8523 ``` Add KB Statistics keywords and infrastructure documentation
- Add KB Statistics keywords for Qdrant vector database monitoring: KB
  STATISTICS, KB COLLECTION STATS, KB DOCUMENTS COUNT, KB DOCUMENTS
  ADDED SINCE, KB LIST COLLECTIONS, KB STORAGE SIZE

- Add comprehensive infrastructure documentation:
  - Scaling and load balancing with LXC containers
  - Infrastructure design with encryption, sharding strategies
  - Observ
2025-11-30 16:25:51 -03:00
c6bbd50946 , dt.month, dt.hour, dt.is_weekend, etc.)
- Add startup wizard module for first-run configuration
- Add white-label branding system with .product file support
- Add bot manager for lifecycle, MinIO buckets, and templates
- Add version tracking registry for component updates
- Create comparison doc: BASIC vs n8n/Zapier/Make/Copilot
- Add WhatsApp-style sample dialogs to template documentation
- Add data traceability SVG diagram ```
2025-11-30 15:07:29 -03:00
faaf9c18c8 Looking at this diff, I need to summarize the significant documentation
and code changes:

``` Add natural language scheduling, docs, wizard, and branding

- Add SET SCHEDULE natural language parser supporting patterns like
  "every hour", "at 9am", "weekdays at 8am", "business
2025-11-30 12:20:48 -03:00
d1e3c25eb8 Add tar/flate2 deps and document new BASIC keywords
Add flate2 and tar dependencies for archive extraction support in file
operations. Update documentation with:

- New BASIC keywords: SWITCH/CASE, WEBHOOK, INSTR, IS_NUMERIC
- HTTP operations: POST, PUT, PATCH, DELETE_HTTP, GRAPHQL, SOAP
- Data operations: SAVE, INSERT, UPDATE, DELETE, MERGE, FILTER, etc.
- File operations: READ, WRITE, COMPRESS, EXTRACT, GENERATE_PDF, etc.

Simplify README and add appendices for external services and environment
variables. Add monitoring dashboard and player UI docs.
2025-11-30 07:53:58 -03:00
11ffccaa94 - More htmx. 2025-11-29 17:27:13 -03:00
ad1cd3ee36 HTMX enters. 2025-11-29 16:29:28 -03:00
31b80ba7f9 Add message type constants and documentation
Introduce a shared enum-based system for categorizing message types
across the Rust backend and JavaScript frontend. This replaces magic
numbers with named constants for improved type safety, readability, and
maintainability.

The implementation includes:
- Rust MessageType enum with serialization support
- JavaScript constants matching the Rust enum values
- Helper
2025-11-29 11:17:01 -03:00
60706bf1c7 Replace magic numbers with MessageType constants
This commit replaces all hardcoded message type integers (0, 1, 2, 3, 4,
5) with named constants from a new MessageType module, improving code
readability and maintainability across the codebase.
2025-11-28 18:15:09 -03:00
87b7d96031 run
- Database migrations run automatically on startup
- New QUICK_START.md with usage examples and troubleshooting
- Better handling of already-running services
2025-11-28 15:06:30 -03:00
6f686565ab Update default features and add quick start guide
Replaces ui-server with console in default features, adds comprehensive
quick start documentation, implements automatic database migrations at
startup, and ensures critical services (PostgreSQL and MinIO) are
started automatically.

Key changes:
- Console UI now enable
2025-11-28 13:50:28 -03:00