Commit graph

61 commits

Author SHA1 Message Date
86eb3d2160 Fix migration error - remove invalid session_kb_associations DROP 2026-02-05 14:48:08 -03:00
5d5be5546f Fix migration: Remove orphaned COMMENT statements for session_kb_associations table 2026-02-05 14:28:20 -03:00
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
748fceff5d Fix issues: remove unused import, fix ownership error, reduce crawler interval 2026-01-30 12:21:30 -03:00
5568ef5802 Fix migration 6.0.0: Comment out sent_email_tracking 2026-01-27 14:14:20 -03:00
b103c07248 Fix migration errors and reorganize migration files
- Fixed 'relation session_kb_associations does not exist' error in core consolidated migration.
- Renamed migration directories from timestamp-based to version-based (6.0.x, 6.1.x, 6.2.x).
- Reorganized migrations into dedicated feature folders (products, dashboards, learn, video).
- Updated migration execution order in core/shared/utils.rs.
- Moves legacy migrations to 6.0.x/6.1.x and workflow to 6.2.0.
2026-01-27 13:45:54 -03:00
9087bb17cd feat: complete General Bots 7.0 (v6.2.0) 2026-01-25 10:29:54 -03:00
fc9c07d204 Refactor schema.rs into modular files 2026-01-22 13:57:40 -03:00
446dee26b6 Add extended product fields: dimensions, weight, tax codes, brand, inventory, pricing, SEO 2026-01-14 10:02:55 -03:00
95a7591c39 Add Stone Pagamentos fields: SKU, dimensões, peso, NCM, GTIN, tributos, etc 2026-01-14 09:50:39 -03:00
a2783f9b32 Fix 5 errors and 32 warnings: calendar, compliance, billing_alert_broadcast, unused vars 2026-01-13 22:21:25 -03:00
31777432b4 Implement TODO items: session auth, face API, task logs, intent storage
Learn Module:
- All 9 handlers now use AuthenticatedUser extractor

Security:
- validate_session_sync reads roles from SESSION_CACHE

AutoTask:
- get_task_logs reads from manifest with status logs
- store_compiled_intent saves to cache and database

Face API:
- AWS Rekognition, OpenCV, InsightFace implementations
- Detection, verification, analysis methods

Other fixes:
- Calendar/task integration database queries
- Recording database methods
- Analytics insights trends
- Email/folder monitoring mock data
2026-01-13 14:48:49 -03:00
a886478548 Implement database persistence for dashboards, legal, and compliance modules
- Add PostgreSQL persistence for dashboards module (was returning empty vec![])
  - Tables: dashboards, dashboard_widgets, dashboard_data_sources, dashboard_filters,
    dashboard_widget_data_sources, conversational_queries
  - Full CRUD operations with spawn_blocking pattern

- Add PostgreSQL persistence for legal module (was using in-memory HashMap)
  - Tables: legal_documents, legal_document_versions, cookie_consents, consent_history,
    legal_acceptances, data_deletion_requests, data_export_requests
  - GDPR-compliant consent tracking and document management

- Add PostgreSQL persistence for compliance module (was returning empty results)
  - Tables: compliance_checks, compliance_issues, compliance_audit_log, compliance_evidence,
    compliance_risk_assessments, compliance_risks, compliance_training_records,
    compliance_access_reviews
  - Support for GDPR, SOC2, ISO27001, HIPAA, PCI-DSS frameworks

- Add migration files for all new tables
- Update schema.rs with new table definitions and joinables
- Register new routes in main.rs
- Add recursion_limit = 512 for macro expansion
2026-01-13 00:07:22 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
b2c5895887 Add migration for bot database_name column
- Creates database_name column in bots table
- Creates tenant_id column for multi-tenant support
- Adds indexes for both columns
- Migration runs automatically on server start
2026-01-02 19:56:39 -03:00
86ac5ca8f5 Fix panic: check if item_groups is empty before iterating
Prevents 'index out of bounds: the len is 0 but the index is 0' error
in complete_item_group_range when item_groups is empty.
2026-01-02 12:54:54 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
23868e4c7d Consolidate all migrations into single file (129 tables preserved)
- Merge all 6.0.x through 6.1.x migrations into one file
- All 129 CREATE TABLE statements preserved
- 3661 lines of SQL in consolidated up.sql
- No tables removed, just combined into single migration
2025-12-29 12:41:25 -03:00
7634429896 Consolidate all database migrations into single optimized schema
BREAKING CHANGE: Complete database redesign

Migrations:
- Remove all 6.x.x and 7.0.0 migrations
- Create single consolidated migration 20250101000000_consolidated_schema
- Add Diesel initial setup migration

Schema Optimizations:
- All domain values use SMALLINT instead of VARCHAR/TEXT (2 bytes vs 20+ bytes)
- Sharding infrastructure: shard_config, tenant_shard_map tables
- All tables include tenant_id and shard_id for horizontal scaling
- Proper indexing strategies for high-throughput queries
- Foreign key constraints for data integrity
- Updated_at triggers via diesel_manage_updated_at

Tables (35 total):
- Core: tenants, users, bots, bot_configuration, bot_channels
- Sessions: user_sessions, message_history
- Memory: bot_memories
- Auto Tasks: auto_tasks, execution_plans, task_approvals, task_decisions
- Safety: safety_audit_log, intent_classifications
- Apps: generated_apps, designer_changes, designer_pending_changes
- KB: kb_collections, kb_documents, session_kb_associations, kb_sources
- Tools: tools, system_automations, pending_info
- Analytics: usage_analytics, analytics_events
- Tasks: tasks, task_comments
- Accounts: connected_accounts, session_account_associations
- Communication: whatsapp_numbers, clicks
- Access: table_role_access, context_injections
- Organizations: organizations, user_organizations

Rust Schema:
- Updated schema.rs to match new consolidated schema
- All diesel table definitions with proper types
- Foreign key relationships defined

WARNING: This is a DESTRUCTIVE migration - drop existing database before applying
2025-12-29 12:29:23 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
0e7b4b66ec Add DROP TABLE for knowledge base tables to fix migration 2025-12-29 08:36:50 -03:00
6aceb4bc59 Remove embedding columns from PostgreSQL - Qdrant handles vector storage 2025-12-29 08:29:12 -03:00
84fa9abe0a Make pgvector extension optional - use BYTEA fallback for embeddings 2025-12-29 08:27:57 -03:00
c665189279 Add DROP TABLE CASCADE at start of 6.1.0 migration for clean re-run 2025-12-29 08:18:14 -03:00
6112732e9c Add DROP TABLE before user_organizations CREATE to fix partial migration 2025-12-29 08:12:17 -03:00
38f9abb7db Fix organizations foreign key reference (org_id not id) 2025-12-29 08:07:42 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
a491cc13a6 Update botserver 2025-12-21 23:40:43 -03:00
5d5668fee2 chore: consolidate migrations 6.1.1-6.1.3 into 6.1.0_enterprise_suite
- Integrate config ID type fixes (TEXT → UUID)
- Integrate connected_accounts, session_account_associations, account_sync_items
- Integrate bot hierarchy (parent_bot_id, enabled_tabs_json, inherit_parent_config)
- Integrate email_monitors, folder_monitors, folder_change_events, email_received_events
- Integrate user_organizations table
- Remove separate 6.1.1, 6.1.2, 6.1.3 migration folders
- Maximum version remains 6.1.0 per project standards
2025-12-18 18:52:52 -03:00
57bf5b8754 feat: add ON EMAIL and ON CHANGE keywords for event-driven monitoring
- Add ON EMAIL keyword with FROM/SUBJECT filters
- Add ON CHANGE keyword with account:// syntax (gdrive, onedrive, dropbox, local)
- Add TriggerKind::EmailReceived (5) and FolderChange (6)
- Add migration 6.1.3_bot_hierarchy_monitors with:
  - email_monitors, folder_monitors tables
  - email_received_events, folder_change_events tables
  - user_organizations table
  - Bot hierarchy: parent_bot_id, enabled_tabs_json, inherit_parent_config
- Add 26 unit tests (12 on_email, 12 on_change, 2 trigger_kind)
- Update PROMPT.md with weekly maintenance checklist
- Zero warnings, zero errors
2025-12-18 16:17:58 -03:00
17d5a263f1 Fix config.csv loading on startup
- Disable TLS on Vault for local development (HTTP instead of HTTPS)
- Fix bot_configuration id column type mismatch (TEXT -> UUID)
- Add migration 6.1.1 to convert config table id columns to UUID
- Fix sync_config_csv_to_db to use UUID binding for id column
- Make start_all async with proper Vault startup sequence
- Sync default.gbai config.csv to existing 'Default Bot' from migrations
- Add diagnostic logging for config loading
- Change default LLM/embedding URLs from https to http for local dev
2025-12-08 00:19:29 -03:00
6c4ecea2fc feat(auth): Add OAuth login for Google, Discord, Reddit, Twitter, Microsoft, Facebook
- Create core/oauth module with OAuthProvider enum and shared types
- Implement providers.rs with auth URLs, token exchange, user info endpoints
- Add routes for /auth/oauth/providers, /auth/oauth/{provider}, and callbacks
- Update login.html with OAuth button grid and dynamic provider loading
- Add OAuth config settings to config.csv with setup documentation and links
- Uses HTMX for login form, minimal JS for OAuth provider visibility
2025-12-04 22:53:40 -03:00
d9b09e24d4 feat(email): implement email read tracking with pixel support
- Add email-read-pixel config parameter to enable/disable tracking
- Implement tracking pixel injection in HTML emails
- Add sent_email_tracking table with migration
- Create 4 new API endpoints:
  - GET /api/email/tracking/pixel/{id} - serve pixel & record read
  - GET /api/email/tracking/status/{id} - get email read status
  - GET /api/email/tracking/list - list all tracked emails
  - GET /api/email/tracking/stats - get aggregate statistics
- Store tracking data: read_at, read_count, IP, user_agent
- Integrate with send_email() to auto-inject pixel when enabled
2025-12-04 18:15:09 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00
fd5a1ee1e1 Add implementation plan and multi-agent features
This commit introduces comprehensive documentation and implementation
for multi-agent orchestration capabilities:

- Add IMPLEMENTATION-PLAN.md with 4-phase roadmap
- Add Kubernetes deployment manifests (deployment.yaml, hpa.yaml)
- Add database migrations for multi-agent tables (6.1.1, 6.1.2)
- Implement A2A protocol for agent-to-agent communication
- Implement user memory keywords for cross-session persistence
- Implement model routing for dynamic L
2025-11-30 19:18:23 -03:00
c6bbd50946 , dt.month, dt.hour, dt.is_weekend, etc.)
- Add startup wizard module for first-run configuration
- Add white-label branding system with .product file support
- Add bot manager for lifecycle, MinIO buckets, and templates
- Add version tracking registry for component updates
- Create comparison doc: BASIC vs n8n/Zapier/Make/Copilot
- Add WhatsApp-style sample dialogs to template documentation
- Add data traceability SVG diagram ```
2025-11-30 15:07:29 -03:00
f40cb6c7b4 Fix typos in bot file extensions and keyword names
Changed incorrect references to .vbs files to .bas and corrected
USE_WEBSITE keyword naming. Also added missing fields to API response
structure and clarified that start.bas is optional for bots.
2025-11-26 22:54:22 -03:00
e013f3f800 - Even more keywords. 2025-11-22 12:26:16 -03:00
99037d5876 ``` Add comprehensive email account management and user settings
interface

Implements multi-user authentication system with email account
management, profile settings, drive configuration, and security
controls. Includes database migrations for user accounts, email
credentials, preferences, and session management. Frontend provides
intuitive UI for adding IMAP/SMTP accounts with provider presets and
connection testing. Backend supports per-user vector databases for email
and file indexing with Zitadel SSO integration and automatic workspace
initialization. ```
2025-11-21 09:28:35 -03:00
9c5d38b60e feat(automation): increase schedule field size and improve task checking
- Increased schedule field size from bpchar(12) to bpchar(20) in database schema
- Reduced task checking interval from 60s to 5s for more responsive automation
- Improved error handling for schedule parsing and execution
- Added proper error logging for automation failures
- Changed automation execution to use bot_id instead of nil UUID
- Enhanced HEAR keyword functionality (partial diff shown)
2025-11-12 15:04:04 -03:00
e8fc78e638 feat: add diesel_migrations and update dependencies
Added diesel_migrations crate (v2.3.0) to enable database migration functionality. Updated Cargo.toml and Cargo.lock to include the new dependency along with its required sub-dependencies (migrations_internals and migrations_macros). Also made minor cleanups in the codebase:
- Removed unused UI code from platform README
- Cleaned up LLM server initialization code
- Added additional build dependencies in documentation
2025-11-12 12:48:06 -03:00
73d0ab3a1a refactor: simplify UI panels, use pooled DB, add --noui flag
- Removed unused `id` and `app_state` fields from `ChatPanel`; updated constructor to accept but ignore the state, reducing memory footprint.
- Switched database access in `ChatPanel` from a raw `Mutex` lock to a connection pool (`app_state.conn.get()`), improving concurrency and error handling.
- Reordered and cleaned up imports in `status_panel.rs` and formatted struct fields for readability.
- Updated VS Code launch configuration to pass `--noui` argument, enabling headless mode for debugging.
- Bumped several crate versions in `Cargo.lock` (e.g., `bitflags` to 2.10.0, `syn` to 2.0.108, `cookie` to 0.16.2) and added the new `ashpd` dependency, aligning the project with latest library releases.
2025-11-11 09:42:52 -03:00
3626719d7f chore: point LLM client and default configs to local endpoint
Updated the 6.0.4 migration to use `http://localhost:8081/v1` for the default OpenAI model configurations (gpt‑4 and gpt‑3.5‑turbo) and the local embed service. Adjusted `OpenAIClient` to default to the same localhost base URL instead of the production OpenAI API.

Reorganized imports and module ordering in `src/main.rs` (moved `mod llm`, `mod nvidia`, and `BotOrchestrator` import), cleaned up formatting, and removed unused imports. These changes streamline development by directing LLM calls to a local server and improve code readability.
2025-11-07 16:40:19 -03:00
60938bc606 fix(migrations, ui): remove redundant constraint check and logo click handler
Removed `IF NOT EXISTS` from the unique constraint in `system_automations` to ensure proper enforcement, and deleted the unused `floatLogo` click event listener to clean up UI behavior.
2025-11-05 15:29:01 -03:00
5245fc0567 feat(db): add unique constraint to system_automations and fix clicks PK
- Added migration 6.0.6 to enforce a unique constraint on `(bot_id, kind, param)` in `system_automations`, preventing “no unique or exclusion constraint matching the ON CONFLICT specification” errors, and created a supporting index.
- Added migration 6.0.7 to replace the `clicks` table with a correctly defined primary key and a unique `(campaign_id, email)` constraint, satisfying Diesel
2025-11-05 10:19:02 -03:00
3a57c6ac8d feat(schedule): optimize set_schedule with update-first approach
Changed the set_schedule function to first attempt updating existing records before inserting new ones. This improves efficiency by avoiding unnecessary insert conflicts and subsequent updates. The logic now:
1. Tries to update matching existing schedule first
2. Only performs insert if no matching record was found
3. Maintains same functionality but with better performance
2025-11-05 09:05:25 -03:00
b1d4e8dd5a feat(db): add bot_id column, constraints, enforce DATABASE_URL
- Extend `system_automations` with a non‑null `bot_id` UUID column, create an index on it, and add a unique constraint on `(bot_id, kind, param)` to support upserts.
- Add a unique constraint on `bot_configuration.config_key` to prevent duplicate configuration keys.
- Include migration guards to ensure the new constraint is only created once.
- Remove automatic writing of drive configuration to a `.env` file, cleaning up side‑effects during config loading.
- Change database connection handling to require `DATABASE_URL` to be set (no fallback), making the environment initialization explicit.
2025-11-05 08:47:28 -03:00
49d6debc83 feat(automation): add unique constraint and refactor action execution
- Added UNIQUE constraint on system_automations (bot_id, kind, param) to prevent duplicate automations
- Refactored execute_action to accept full Automation struct instead of just param
- Simplified bot name resolution by using automation.bot_id directly
- Improved error handling in action execution with proper error propagation
- Removed redundant bot name lookup logic by leveraging existing bot_id
2025-11-02 20:57:53 -03:00
937dbfcc76 feat: add bot_id to system_automations and enhance schedule handling
- Introduced `bot_id` column in `system_automations` table (migration 6.0.0.sql) and updated the Diesel schema/model to include it.
- Adjusted migrations to remove the hard‑coded “Update Summary” automation and only create an index on the `name` column.
- Extended the `SET_SCHEDULE` keyword:
  - Added a second string argument for the script name.
  - Passed the invoking user's `bot_id` to the database layer.
  - Updated function signature to accept a full `UserSession` instead of discarding it.
- Modified `execute_set_schedule` to store `bot_id`, script name, and activation flag; added conflict handling on `(bot_id, param)` to update schedule and reset trigger state.
- Updated imports and logging to reflect new parameters.

These changes enable per‑bot automation management, allow specifying the script to run, and improve idempotent schedule updates.
2025-11-02 19:32:25 -03:00