Commit graph

149 commits

Author SHA1 Message Date
c8598def62 fix(bootstrap): Initialize SecretsManager after Vault starts in start_all/ensure_services_running
When services are already configured (bootstrap completed), the code
calls start_all() or ensure_services_running() which starts Vault
but didn't initialize SecretsManager. This caused create_conn() to
fail with 'Vault not configured' even though Vault was running.

Now both functions initialize SecretsManager after Vault is unsealed,
ensuring database connections can retrieve credentials from Vault.
2025-12-09 07:58:39 -03:00
224ddd4a03 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
5669515c68 fix(bootstrap): Run migrations with direct password before Vault is set up
The bootstrap order is vault -> tables -> directory. When tables
component was processed, it tried to call establish_pg_connection()
which requires Vault, but Vault setup only happens when processing
the vault component (which is earlier in the loop but the if-block
for tables runs for the tables component, not vault).

Fix: Run migrations directly with the generated password during the
tables component setup, before Vault is configured. This avoids the
dependency on Vault being set up.
2025-12-09 07:49:01 -03:00
185aba3631 fix(bootstrap): Suppress all subprocess output to prevent console UI corruption
- Redirect Vault exec_cmd output to logs/vault.log
- Add stdout/stderr null pipes to component spawn in installer.rs
- Suppress output in run_commands() in facade.rs
- All component output now goes to respective log files in logs/
2025-12-09 01:09:04 -03:00
a755d4d13e feat(console): Show UI immediately with live system logs
- Add state_channel field to XtreeUI for receiving AppState updates
- Add set_state_channel() method to enable async state communication
- Poll for AppState in event loop to initialize panels when ready
- UI now shows loading state instantly, logs stream in real-time
- Transitions to full interactive mode when AppState is received
2025-12-08 23:35:33 -03:00
05fa0e95c5 fix: remove duplicate logger init causing 'cannot set logger' error
Logger is already initialized in main.rs, cli.rs was trying to
initialize it again when CLI commands were run.
2025-12-08 17:59:43 -03:00
559154b923 fix: preserve cached files in botserver-installers after extraction
- Don't delete archive files from cache after tar/zip extraction
- Copy binaries from cache instead of moving them
- Cached files are now preserved for offline installation reuse
2025-12-08 14:19:55 -03:00
eed537ac42 feat: add offline installer cache and health endpoints
- Add /health and /api/health endpoints for botui connectivity
- Create 3rdparty.toml with all download URLs for offline bundles
- Add botserver-installers/ cache directory for downloaded files
- Implement DownloadCache module with:
  - Automatic cache lookup before downloading
  - Support for pre-populated offline bundles
  - SHA256 checksum verification (optional)
  - Cache management utilities (list, clear, size)
- Update download_and_install to use cache system
- Data files (models) also cached for reuse

Cache behavior:
- First run: downloads to botserver-installers/
- Subsequent runs: uses cached files
- Delete botserver-stack/ without losing downloads
- Pre-populate cache for fully offline installation
2025-12-08 14:08:49 -03:00
5058b4a2c2 Fix Vault CLI mTLS issue - unset client cert env vars before CLI commands 2025-12-08 09:14:31 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
acb78b30f9 refactor: standardize component names for certificates and DNS
- Rename service names to component names:
  - postgres -> tables
  - minio -> drive
  - redis -> cache
  - qdrant -> vectordb
  - mail -> email

- Update bootstrap certificate generation with new hostnames
- Update CoreDNS zone file with component-based DNS entries
- Update CA service certificates to use component names
- Update CA directory creation to use component names

All certificates include 127.0.0.1 as SAN for IP-based mTLS access
2025-12-07 10:42:02 -03:00
c3ab7fe1a5 chore: Update secrets module and remove task files 2025-12-07 09:57:05 -03:00
6ff2b32f2c feat(bootstrap): implement mTLS for Vault access
- Add create_vault_config() function to generate config.hcl with mTLS settings
- Configure Vault to require client certificate verification
- Generate client certificate for botserver in bootstrap
- Update .env to include mTLS paths (VAULT_CACERT, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY)
- Remove unused import in tls.rs
2025-12-07 02:13:28 -03:00
b611f8c535 Add --stack-path CLI option for custom stack path
Allows overriding the default botserver-stack location via:
- --stack-path /path/to/stack CLI argument
- BOTSERVER_STACK_PATH environment variable

Useful for testing botserver bootstrap in isolated temp directories.
2025-12-06 14:55:42 -03:00
2abd875aec Update: General project updates 2025-12-06 11:09:12 -03:00
c2aa82733a Add test infrastructure: AppState::default(), mock providers, fix attendance tests
- Add Default impl for AppState with mock services for testing
- Add MockLLMProvider for tests that need LLM without real API
- Add create_mock_auth_service() for Zitadel testing
- Add test_utils.rs with TestAppStateBuilder, MockChannelAdapter
- Enable rhai 'sync' feature to fix Send+Sync for Dynamic types
- Fix attendance.rs tests to use pure functions (no DB required)
- Fix on_error.rs tests to use String instead of i32
- Remove unused imports in attendance.rs

All tests pass, 0 warnings, 0 errors.
2025-12-05 16:43:14 -03:00
5d93505012 feat(attendance): Add LLM-assisted attendant features
- Real-time tips when customer messages arrive
- Message polishing with one click
- Smart reply generation (3 contextual suggestions)
- Auto-summary when attendant takes conversation
- LLM-powered sentiment analysis with escalation warnings

WhatsApp Attendant Commands:
- /queue, /take, /status, /transfer, /resolve
- /tips, /polish, /replies, /summary, /help
- Portuguese versions: /fila, /pegar, /dicas, /polir, /respostas, /resumo

Config options (config.csv):
- attendant-llm-tips
- attendant-polish-message
- attendant-smart-replies
- attendant-auto-summary
- attendant-sentiment-analysis

API Endpoints:
- POST /api/attendance/llm/tips
- POST /api/attendance/llm/polish
- POST /api/attendance/llm/smart-replies
- GET /api/attendance/llm/summary/{session_id}
- POST /api/attendance/llm/sentiment
- GET /api/attendance/llm/config/{bot_id}

Uses bot's system prompt for consistency between bot and human-assisted responses.
2025-12-05 13:47:15 -03:00
6882d52552 refactor: rename redis-cache feature to cache
- Simplified feature name from 'redis-cache' to 'cache'
- Updated Cargo.toml feature definitions
- Updated state.rs cfg attributes
- More concise naming
2025-12-05 12:09:02 -03:00
6c4ecea2fc feat(auth): Add OAuth login for Google, Discord, Reddit, Twitter, Microsoft, Facebook
- Create core/oauth module with OAuthProvider enum and shared types
- Implement providers.rs with auth URLs, token exchange, user info endpoints
- Add routes for /auth/oauth/providers, /auth/oauth/{provider}, and callbacks
- Update login.html with OAuth button grid and dynamic provider loading
- Add OAuth config settings to config.csv with setup documentation and links
- Uses HTMX for login form, minimal JS for OAuth provider visibility
2025-12-04 22:53:40 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
d088490f41 - botbook extracted from. 2025-12-03 19:59:27 -03:00
b2f1ea40e4 Add comprehensive documentation for GB templates and configuration
modules

Add detailed README documentation for 15+ bot templates including:
- Platform analytics, BI reporting, and web crawler templates
- CRM, contacts, and marketing automation templates
- Legal document processing and office productivity templates
- LLM tools, LLM server, and API client integration templates
- Reminder management and meta-template for creating new templates

Add new Rust configuration modules:
- BM25 config for Tantivy-based sparse
2025-12-03 16:05:50 -03:00
e9dad539af Update dependencies and remove problematic crates
Drop image (with ravif/paste), sqlx, zitadel, and related dependencies
that were causing compilation issues. Replace image processing with
direct png crate usage. Update rcgen to 0.14 with new API changes.
Refactor CA certificate generation to use Issuer pattern.
2025-12-03 16:05:30 -03:00
340ac8840a - New templates. 2025-12-03 07:15:54 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
2bf53ab230 Add toml dependency for Askama custom filters config 2025-11-30 23:48:08 -03:00
16a792b5cb Add Suite app documentation, templates, and Askama config
- Add askama.toml for template configuration (ui/ directory)
- Add Suite app documentation with flow diagrams (SVG)
  - App launcher, chat flow, drive flow, tasks flow
  - Individual app docs: chat, drive, tasks, mail, etc.
- Add HTML templates for Suite apps
  - Base template with header and app launcher
  - Auth login page
  - Chat, Drive, Mail, Meet, Tasks templates
  - Partial templates for messages, sessions, notifications
- Add Extensions type to AppState for type-erased storage
- Add mTLS module for service-to-service authentication
- Update web handlers to use new template paths (suite/)
- Fix auth module to avoid axum-extra TypedHeader dependency
2025-11-30 21:00:48 -03:00
48288e8523 ``` Add KB Statistics keywords and infrastructure documentation
- Add KB Statistics keywords for Qdrant vector database monitoring: KB
  STATISTICS, KB COLLECTION STATS, KB DOCUMENTS COUNT, KB DOCUMENTS
  ADDED SINCE, KB LIST COLLECTIONS, KB STORAGE SIZE

- Add comprehensive infrastructure documentation:
  - Scaling and load balancing with LXC containers
  - Infrastructure design with encryption, sharding strategies
  - Observ
2025-11-30 16:25:51 -03:00
c6bbd50946 , dt.month, dt.hour, dt.is_weekend, etc.)
- Add startup wizard module for first-run configuration
- Add white-label branding system with .product file support
- Add bot manager for lifecycle, MinIO buckets, and templates
- Add version tracking registry for component updates
- Create comparison doc: BASIC vs n8n/Zapier/Make/Copilot
- Add WhatsApp-style sample dialogs to template documentation
- Add data traceability SVG diagram ```
2025-11-30 15:07:29 -03:00
faaf9c18c8 Looking at this diff, I need to summarize the significant documentation
and code changes:

``` Add natural language scheduling, docs, wizard, and branding

- Add SET SCHEDULE natural language parser supporting patterns like
  "every hour", "at 9am", "weekdays at 8am", "business
2025-11-30 12:20:48 -03:00
d1e3c25eb8 Add tar/flate2 deps and document new BASIC keywords
Add flate2 and tar dependencies for archive extraction support in file
operations. Update documentation with:

- New BASIC keywords: SWITCH/CASE, WEBHOOK, INSTR, IS_NUMERIC
- HTTP operations: POST, PUT, PATCH, DELETE_HTTP, GRAPHQL, SOAP
- Data operations: SAVE, INSERT, UPDATE, DELETE, MERGE, FILTER, etc.
- File operations: READ, WRITE, COMPRESS, EXTRACT, GENERATE_PDF, etc.

Simplify README and add appendices for external services and environment
variables. Add monitoring dashboard and player UI docs.
2025-11-30 07:53:58 -03:00
11ffccaa94 - More htmx. 2025-11-29 17:27:13 -03:00
ad1cd3ee36 HTMX enters. 2025-11-29 16:29:28 -03:00
31b80ba7f9 Add message type constants and documentation
Introduce a shared enum-based system for categorizing message types
across the Rust backend and JavaScript frontend. This replaces magic
numbers with named constants for improved type safety, readability, and
maintainability.

The implementation includes:
- Rust MessageType enum with serialization support
- JavaScript constants matching the Rust enum values
- Helper
2025-11-29 11:17:01 -03:00
60706bf1c7 Replace magic numbers with MessageType constants
This commit replaces all hardcoded message type integers (0, 1, 2, 3, 4,
5) with named constants from a new MessageType module, improving code
readability and maintainability across the codebase.
2025-11-28 18:15:09 -03:00
87b7d96031 run
- Database migrations run automatically on startup
- New QUICK_START.md with usage examples and troubleshooting
- Better handling of already-running services
2025-11-28 15:06:30 -03:00
6f686565ab Update default features and add quick start guide
Replaces ui-server with console in default features, adds comprehensive
quick start documentation, implements automatic database migrations at
startup, and ensures critical services (PostgreSQL and MinIO) are
started automatically.

Key changes:
- Console UI now enable
2025-11-28 13:50:28 -03:00
c67ff407b1 Add .env.example with comprehensive configuration template
The commit adds a complete example environment configuration file
documenting all available settings for BotServer, including logging,
database, server, drive, LLM, Redis, email, and feature flags.

Also removes hardcoded environment variable usage throughout the
codebase, replacing them with configuration via config.csv or
appropriate defaults. This includes:

- WhatsApp, Teams, Instagram adapter configurations
- Weather API key handling
- Email and directory service configurations
- Console feature conditionally compiles monitoring code
- Improved logging configuration with library suppression
2025-11-28 13:19:03 -03:00
1789952c5b Remove unused sqlx dependency and related code
The sqlx database library has been removed from the project along with
associated database-specific code that was no longer being used. This
includes removal of various sqlx-related dependencies from Cargo.lock
and cleanup of database connection pool references.
2025-11-28 09:27:29 -03:00
7f2967b04b Add SQLx dependencies for calendar feature 2025-11-27 23:10:43 -03:00
4b5d18db71 Add metadata and refactor BASIC compiler
- Add package metadata (keywords, categories) to Cargo.toml
- Add #[must_use] attributes to constructor methods
- Simplify conditional logic by inverting if-else blocks
- Replace string formatting with interpolation syntax
2025-11-27 15:19:17 -03:00
ac03af0b20 Refactor InstagramAdapter initialization and implement file sending
The InstagramAdapter constructor is simplified to remove unused
parameters, and the send_instagram_file function is fully implemented
with S3 upload and message sending capabilities.
2025-11-27 13:53:16 -03:00
6e48474d55 olithic route configuration
- Add route configuration and handlers to calendar module
- Add route configuration and handlers to task module
- Update main.rs to build router from module configurations
- Fix various compiler warnings (dead code, unused variables)

This improves code organization by keeping routes co-located with their
implementation logic.
2025-11-27 13:53:00 -03:00
db06927e24 Looking at the diff, I can see this commit removes the api_router.rs
file and distributes its functionality to individual modules. The
calendar and task modules now have their own route configuration and API
handlers.

Remove centralized API router in favor of module-based routing

Decentralizes API route configuration by moving route definitions and
handlers to their respective modules. Each module now exports its own
`configure_*_routes()` function that is merged in main.rs.

- Delete api_router.rs with its mon
2025-11-27 09:38:50 -03:00
4424fcf548 Fix compiler warnings and improve code consistency
- Remove unused imports and comment them for potential future use
- Add missing .send() to HTTP request chain
- Fix integer type suffixes for JSON values
- Simplify async execution by using tokio::block_in_place
- Remove unused function parameters to eliminate warnings
- Extract temporary variables to avoid borrowing issues
- Add placeholder methods to SessionManager for analytics
- Implement real database operations for admin endpoints
- Remove duplicate or conflicting type definitions

These changes address all compiler warnings while maintaining the
existing functionality and preparing the codebase for future
enhancements in areas like analytics and session management.
2025-11-27 08:34:24 -03:00
f40cb6c7b4 Fix typos in bot file extensions and keyword names
Changed incorrect references to .vbs files to .bas and corrected
USE_WEBSITE keyword naming. Also added missing fields to API response
structure and clarified that start.bas is optional for bots.
2025-11-26 22:54:22 -03:00
82e92de011 Remove unused Python scripts and book.toml
The scripts were development utilities for SVG processing and mdBook
configuration that are no longer needed in the project.
2025-11-26 15:27:47 -03:00
fd97028995 - From 4 to 7. 2025-11-23 20:12:09 -03:00
53bac1f5ec - Refactor folder as features. 2025-11-22 22:55:35 -03:00