Commit graph

47 commits

Author SHA1 Message Date
b2c5895887 Add migration for bot database_name column
- Creates database_name column in bots table
- Creates tenant_id column for multi-tenant support
- Adds indexes for both columns
- Migration runs automatically on server start
2026-01-02 19:56:39 -03:00
86ac5ca8f5 Fix panic: check if item_groups is empty before iterating
Prevents 'index out of bounds: the len is 0 but the index is 0' error
in complete_item_group_range when item_groups is empty.
2026-01-02 12:54:54 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
23868e4c7d Consolidate all migrations into single file (129 tables preserved)
- Merge all 6.0.x through 6.1.x migrations into one file
- All 129 CREATE TABLE statements preserved
- 3661 lines of SQL in consolidated up.sql
- No tables removed, just combined into single migration
2025-12-29 12:41:25 -03:00
7634429896 Consolidate all database migrations into single optimized schema
BREAKING CHANGE: Complete database redesign

Migrations:
- Remove all 6.x.x and 7.0.0 migrations
- Create single consolidated migration 20250101000000_consolidated_schema
- Add Diesel initial setup migration

Schema Optimizations:
- All domain values use SMALLINT instead of VARCHAR/TEXT (2 bytes vs 20+ bytes)
- Sharding infrastructure: shard_config, tenant_shard_map tables
- All tables include tenant_id and shard_id for horizontal scaling
- Proper indexing strategies for high-throughput queries
- Foreign key constraints for data integrity
- Updated_at triggers via diesel_manage_updated_at

Tables (35 total):
- Core: tenants, users, bots, bot_configuration, bot_channels
- Sessions: user_sessions, message_history
- Memory: bot_memories
- Auto Tasks: auto_tasks, execution_plans, task_approvals, task_decisions
- Safety: safety_audit_log, intent_classifications
- Apps: generated_apps, designer_changes, designer_pending_changes
- KB: kb_collections, kb_documents, session_kb_associations, kb_sources
- Tools: tools, system_automations, pending_info
- Analytics: usage_analytics, analytics_events
- Tasks: tasks, task_comments
- Accounts: connected_accounts, session_account_associations
- Communication: whatsapp_numbers, clicks
- Access: table_role_access, context_injections
- Organizations: organizations, user_organizations

Rust Schema:
- Updated schema.rs to match new consolidated schema
- All diesel table definitions with proper types
- Foreign key relationships defined

WARNING: This is a DESTRUCTIVE migration - drop existing database before applying
2025-12-29 12:29:23 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
0e7b4b66ec Add DROP TABLE for knowledge base tables to fix migration 2025-12-29 08:36:50 -03:00
6aceb4bc59 Remove embedding columns from PostgreSQL - Qdrant handles vector storage 2025-12-29 08:29:12 -03:00
84fa9abe0a Make pgvector extension optional - use BYTEA fallback for embeddings 2025-12-29 08:27:57 -03:00
c665189279 Add DROP TABLE CASCADE at start of 6.1.0 migration for clean re-run 2025-12-29 08:18:14 -03:00
6112732e9c Add DROP TABLE before user_organizations CREATE to fix partial migration 2025-12-29 08:12:17 -03:00
38f9abb7db Fix organizations foreign key reference (org_id not id) 2025-12-29 08:07:42 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
a491cc13a6 Update botserver 2025-12-21 23:40:43 -03:00
5d5668fee2 chore: consolidate migrations 6.1.1-6.1.3 into 6.1.0_enterprise_suite
- Integrate config ID type fixes (TEXT → UUID)
- Integrate connected_accounts, session_account_associations, account_sync_items
- Integrate bot hierarchy (parent_bot_id, enabled_tabs_json, inherit_parent_config)
- Integrate email_monitors, folder_monitors, folder_change_events, email_received_events
- Integrate user_organizations table
- Remove separate 6.1.1, 6.1.2, 6.1.3 migration folders
- Maximum version remains 6.1.0 per project standards
2025-12-18 18:52:52 -03:00
57bf5b8754 feat: add ON EMAIL and ON CHANGE keywords for event-driven monitoring
- Add ON EMAIL keyword with FROM/SUBJECT filters
- Add ON CHANGE keyword with account:// syntax (gdrive, onedrive, dropbox, local)
- Add TriggerKind::EmailReceived (5) and FolderChange (6)
- Add migration 6.1.3_bot_hierarchy_monitors with:
  - email_monitors, folder_monitors tables
  - email_received_events, folder_change_events tables
  - user_organizations table
  - Bot hierarchy: parent_bot_id, enabled_tabs_json, inherit_parent_config
- Add 26 unit tests (12 on_email, 12 on_change, 2 trigger_kind)
- Update PROMPT.md with weekly maintenance checklist
- Zero warnings, zero errors
2025-12-18 16:17:58 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
6c4ecea2fc feat(auth): Add OAuth login for Google, Discord, Reddit, Twitter, Microsoft, Facebook
- Create core/oauth module with OAuthProvider enum and shared types
- Implement providers.rs with auth URLs, token exchange, user info endpoints
- Add routes for /auth/oauth/providers, /auth/oauth/{provider}, and callbacks
- Update login.html with OAuth button grid and dynamic provider loading
- Add OAuth config settings to config.csv with setup documentation and links
- Uses HTMX for login form, minimal JS for OAuth provider visibility
2025-12-04 22:53:40 -03:00
d9b09e24d4 feat(email): implement email read tracking with pixel support
- Add email-read-pixel config parameter to enable/disable tracking
- Implement tracking pixel injection in HTML emails
- Add sent_email_tracking table with migration
- Create 4 new API endpoints:
  - GET /api/email/tracking/pixel/{id} - serve pixel & record read
  - GET /api/email/tracking/status/{id} - get email read status
  - GET /api/email/tracking/list - list all tracked emails
  - GET /api/email/tracking/stats - get aggregate statistics
- Store tracking data: read_at, read_count, IP, user_agent
- Integrate with send_email() to auto-inject pixel when enabled
2025-12-04 18:15:09 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
fd5a1ee1e1 Add implementation plan and multi-agent features
This commit introduces comprehensive documentation and implementation
for multi-agent orchestration capabilities:

- Add IMPLEMENTATION-PLAN.md with 4-phase roadmap
- Add Kubernetes deployment manifests (deployment.yaml, hpa.yaml)
- Add database migrations for multi-agent tables (6.1.1, 6.1.2)
- Implement A2A protocol for agent-to-agent communication
- Implement user memory keywords for cross-session persistence
- Implement model routing for dynamic L
2025-11-30 19:18:23 -03:00
c6bbd50946 , dt.month, dt.hour, dt.is_weekend, etc.)
- Add startup wizard module for first-run configuration
- Add white-label branding system with .product file support
- Add bot manager for lifecycle, MinIO buckets, and templates
- Add version tracking registry for component updates
- Create comparison doc: BASIC vs n8n/Zapier/Make/Copilot
- Add WhatsApp-style sample dialogs to template documentation
- Add data traceability SVG diagram ```
2025-11-30 15:07:29 -03:00
f40cb6c7b4 Fix typos in bot file extensions and keyword names
Changed incorrect references to .vbs files to .bas and corrected
USE_WEBSITE keyword naming. Also added missing fields to API response
structure and clarified that start.bas is optional for bots.
2025-11-26 22:54:22 -03:00
e013f3f800 - Even more keywords. 2025-11-22 12:26:16 -03:00
99037d5876 ``` Add comprehensive email account management and user settings
interface

Implements multi-user authentication system with email account
management, profile settings, drive configuration, and security
controls. Includes database migrations for user accounts, email
credentials, preferences, and session management. Frontend provides
intuitive UI for adding IMAP/SMTP accounts with provider presets and
connection testing. Backend supports per-user vector databases for email
and file indexing with Zitadel SSO integration and automatic workspace
initialization. ```
2025-11-21 09:28:35 -03:00
9c5d38b60e feat(automation): increase schedule field size and improve task checking
- Increased schedule field size from bpchar(12) to bpchar(20) in database schema
- Reduced task checking interval from 60s to 5s for more responsive automation
- Improved error handling for schedule parsing and execution
- Added proper error logging for automation failures
- Changed automation execution to use bot_id instead of nil UUID
- Enhanced HEAR keyword functionality (partial diff shown)
2025-11-12 15:04:04 -03:00
e8fc78e638 feat: add diesel_migrations and update dependencies
Added diesel_migrations crate (v2.3.0) to enable database migration functionality. Updated Cargo.toml and Cargo.lock to include the new dependency along with its required sub-dependencies (migrations_internals and migrations_macros). Also made minor cleanups in the codebase:
- Removed unused UI code from platform README
- Cleaned up LLM server initialization code
- Added additional build dependencies in documentation
2025-11-12 12:48:06 -03:00
73d0ab3a1a refactor: simplify UI panels, use pooled DB, add --noui flag
- Removed unused `id` and `app_state` fields from `ChatPanel`; updated constructor to accept but ignore the state, reducing memory footprint.
- Switched database access in `ChatPanel` from a raw `Mutex` lock to a connection pool (`app_state.conn.get()`), improving concurrency and error handling.
- Reordered and cleaned up imports in `status_panel.rs` and formatted struct fields for readability.
- Updated VS Code launch configuration to pass `--noui` argument, enabling headless mode for debugging.
- Bumped several crate versions in `Cargo.lock` (e.g., `bitflags` to 2.10.0, `syn` to 2.0.108, `cookie` to 0.16.2) and added the new `ashpd` dependency, aligning the project with latest library releases.
2025-11-11 09:42:52 -03:00
3626719d7f chore: point LLM client and default configs to local endpoint
Updated the 6.0.4 migration to use `http://localhost:8081/v1` for the default OpenAI model configurations (gpt‑4 and gpt‑3.5‑turbo) and the local embed service. Adjusted `OpenAIClient` to default to the same localhost base URL instead of the production OpenAI API.

Reorganized imports and module ordering in `src/main.rs` (moved `mod llm`, `mod nvidia`, and `BotOrchestrator` import), cleaned up formatting, and removed unused imports. These changes streamline development by directing LLM calls to a local server and improve code readability.
2025-11-07 16:40:19 -03:00
60938bc606 fix(migrations, ui): remove redundant constraint check and logo click handler
Removed `IF NOT EXISTS` from the unique constraint in `system_automations` to ensure proper enforcement, and deleted the unused `floatLogo` click event listener to clean up UI behavior.
2025-11-05 15:29:01 -03:00
5245fc0567 feat(db): add unique constraint to system_automations and fix clicks PK
- Added migration 6.0.6 to enforce a unique constraint on `(bot_id, kind, param)` in `system_automations`, preventing “no unique or exclusion constraint matching the ON CONFLICT specification” errors, and created a supporting index.
- Added migration 6.0.7 to replace the `clicks` table with a correctly defined primary key and a unique `(campaign_id, email)` constraint, satisfying Diesel
2025-11-05 10:19:02 -03:00
3a57c6ac8d feat(schedule): optimize set_schedule with update-first approach
Changed the set_schedule function to first attempt updating existing records before inserting new ones. This improves efficiency by avoiding unnecessary insert conflicts and subsequent updates. The logic now:
1. Tries to update matching existing schedule first
2. Only performs insert if no matching record was found
3. Maintains same functionality but with better performance
2025-11-05 09:05:25 -03:00
b1d4e8dd5a feat(db): add bot_id column, constraints, enforce DATABASE_URL
- Extend `system_automations` with a non‑null `bot_id` UUID column, create an index on it, and add a unique constraint on `(bot_id, kind, param)` to support upserts.
- Add a unique constraint on `bot_configuration.config_key` to prevent duplicate configuration keys.
- Include migration guards to ensure the new constraint is only created once.
- Remove automatic writing of drive configuration to a `.env` file, cleaning up side‑effects during config loading.
- Change database connection handling to require `DATABASE_URL` to be set (no fallback), making the environment initialization explicit.
2025-11-05 08:47:28 -03:00
49d6debc83 feat(automation): add unique constraint and refactor action execution
- Added UNIQUE constraint on system_automations (bot_id, kind, param) to prevent duplicate automations
- Refactored execute_action to accept full Automation struct instead of just param
- Simplified bot name resolution by using automation.bot_id directly
- Improved error handling in action execution with proper error propagation
- Removed redundant bot name lookup logic by leveraging existing bot_id
2025-11-02 20:57:53 -03:00
937dbfcc76 feat: add bot_id to system_automations and enhance schedule handling
- Introduced `bot_id` column in `system_automations` table (migration 6.0.0.sql) and updated the Diesel schema/model to include it.
- Adjusted migrations to remove the hard‑coded “Update Summary” automation and only create an index on the `name` column.
- Extended the `SET_SCHEDULE` keyword:
  - Added a second string argument for the script name.
  - Passed the invoking user's `bot_id` to the database layer.
  - Updated function signature to accept a full `UserSession` instead of discarding it.
- Modified `execute_set_schedule` to store `bot_id`, script name, and activation flag; added conflict handling on `(bot_id, param)` to update schedule and reset trigger state.
- Updated imports and logging to reflect new parameters.

These changes enable per‑bot automation management, allow specifying the script to run, and improve idempotent schedule updates.
2025-11-02 19:32:25 -03:00
a9062b6a9c feat(automation): update logging and script handling
- Update RUST_LOG environment variable in launch.json to exclude more verbose crates
- Change automation kind from 3 to 0 in migrations/6.0.5.sql
- Replace debug with trace logging for schedule evaluation in automation service
- Modify script path resolution to use .ast extension instead of .bas
- Improve error message when script reading fails
- Remove MinIO download fallback for scripts (simplifying script handling)

These changes improve logging verbosity control and simplify script handling by removing the MinIO fallback mechanism. The trace-level
2025-11-02 19:03:52 -03:00
1c80c026d2 feat: refactor database connection handling and add LLM component
- Replace direct database connection establishment with shared `establish_pg_connection` utility
- Add "llm" to required components list in bootstrap manager
- Lower default RUST_LOG level from debug to info in VSCode config
- Clean up imports and connection error messages
- Remove hardcoded database URL strings in favor of centralized connection handling

The changes improve code maintainability by centralizing database connection logic and adding support for the new LLM component in the bootstrap process.
2025-11-01 07:20:04 -03:00
1f10a69644 feat: remove LLM configuration defaults migration
The migration file 6.0.6.sql was deleted as it contained default LLM server configurations that are no longer needed. The file was originally used to set up local LLM server settings including model paths and URLs, but these configurations are now handled differently in the system.
2025-10-30 17:36:15 -03:00
f8e2c3f94b feat(config): remove default server config and update database credentials
Removed the default server configuration section from migrations as these values should now be managed through environment variables. Updated default database username from 'postgres' to 'gbuser' in bootstrap and changed default database name from 'gbuser' to 'botserver' in config to align with the removed server configuration defaults.
2025-10-30 17:32:21 -03:00
930c47f705 - Migration to Open ADAL. 2025-10-28 14:53:45 -03:00
f6385d0218 refactor: update configuration prefix to 'pragmatismo-' and add CLI example format 2025-10-27 18:32:36 -03:00
54c29e01a5 Add LLM configuration defaults 2025-10-21 23:10:28 -03:00
dc3d9b44b1 Enable template bot creation and fix bot schema 2025-10-21 22:43:28 -03:00
9760f57653 - New web assets for 6.0.5. 2025-10-20 23:44:47 -03:00
0a9fddc145 Add new KB and session association tables 2025-10-20 19:49:54 -03:00
09b57aafa0 - Knowledge management. 2025-10-18 18:20:02 -03:00