Commit graph

150 commits

Author SHA1 Message Date
3e75bbff97 MS Office 100% Compatibility - Phase 1 Implementation
- Add rust_xlsxwriter for Excel export with formatting support
- Add docx-rs for Word document import/export with HTML conversion
- Add PPTX export support with slides, shapes, and text elements
- Refactor sheet module into 7 files (types, formulas, handlers, etc)
- Refactor docs module into 6 files (types, handlers, storage, etc)
- Refactor slides module into 6 files (types, handlers, storage, etc)
- Fix collaboration modules (borrow issues, rand compatibility)
- Add ooxmlsdk dependency for future Office 2021 features
- Fix type mismatches in slides storage
- Update security protection API router type

Features:
- Excel: Read xlsx/xlsm/xls, write xlsx with styles
- Word: Read/write docx with formatting preservation
- PowerPoint: Write pptx with slides, shapes, text
- Real-time collaboration via WebSocket (already working)
- Theme-aware UI with --sentient-* CSS variables
2026-01-11 09:56:15 -03:00
46695c0f75 feat(security): add BASIC keywords for security protection tools
Add security_protection.rs with 8 new BASIC keywords:
- SECURITY TOOL STATUS - Check if tool is installed/running
- SECURITY RUN SCAN - Execute security scan
- SECURITY GET REPORT - Get latest scan report
- SECURITY UPDATE DEFINITIONS - Update signatures
- SECURITY START SERVICE - Start security service
- SECURITY STOP SERVICE - Stop security service
- SECURITY INSTALL TOOL - Install security tool
- SECURITY HARDENING SCORE - Get Lynis hardening index

Also:
- Registered protection routes in main.rs
- Added Security Protection category to keywords list
- All functions use proper error handling (no unwrap/expect)
2026-01-10 20:32:56 -03:00
f07448d40a fix: add /js/* route to serve suite JS files like i18n.js 2026-01-10 10:13:46 -03:00
e3b3f04206 Normalize API paths: remove unnecessary /v1/ prefix
- Update all internal API routes from /api/v1/* to /api/*
- Protection API: /api/security/protection/*
- Botmodels calls: /api/vision/*, /api/audio/*, /api/speech/*
- Remove /api/v1/health from anonymous paths (keep /api/health)

External APIs (Reddit, Facebook, etc.) keep their original versioned paths
2026-01-10 09:48:43 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
c743754c6c Add per-bot database creation
- Added database_name field to bots schema
- Bot creation now creates a dedicated PostgreSQL database (bot_{name})
- Updated add_bot.rs to create database and store database_name
- Added create_bot_database() function with safe name validation
- Added dynamic table check to all db_api handlers
2026-01-02 19:34:59 -03:00
65b2583add Add dynamic table check to all db_api handlers
- Added is_table_allowed_with_conn check to get_record_handler
- Added is_table_allowed_with_conn check to create_record_handler
- Added is_table_allowed_with_conn check to update_record_handler
- Added is_table_allowed_with_conn check to delete_record_handler
- Returns 404 with clear message if table doesn't exist
2026-01-02 19:18:58 -03:00
a43aea3320 Serve vendor files (htmx) from MinIO instead of local filesystem
- Added serve_vendor_file() to serve from {bot}.gblib/vendor/ in MinIO
- Added /js/vendor/* route to app_server
- Removed local ServeDir for /js/vendor from main.rs
- Added ensure_vendor_files_in_minio() to upload htmx.min.js on startup
- Uses include_bytes! to embed htmx.min.js in binary
2026-01-02 18:26:34 -03:00
bbbb9e190f Allow dynamic tables from app_generator in db_api
- Added table_exists_in_database() to check if table exists in PostgreSQL
- Updated validate_table_name() to allow valid identifiers (not just whitelist)
- Added validate_table_name_with_conn() for full validation with DB check
- Added is_table_allowed_with_conn() for handlers to verify table existence
- Updated list_records_handler and count_records_handler to use dynamic check
- Uses parameterized query for table existence check (SQL injection safe)
2026-01-02 18:20:04 -03:00
762620f7a9 Rewrite CDN URLs to local paths when serving HTML
- Added rewrite_cdn_urls() to replace HTMX CDN URLs with /js/vendor/htmx.min.js
- Applied to both MinIO and filesystem serving paths
- Supports unpkg.com, jsdelivr, cdnjs variants
- Ensures old apps with CDN references work with local files
2026-01-02 18:12:31 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
a5dee11002 Security audit: Remove all production .unwrap()/.expect(), add SafeCommand, ErrorSanitizer
- Phase 1 Critical: All 115 .unwrap() verified in test code only
- Phase 1 Critical: All runtime .expect() converted to proper error handling
- Phase 2 H1: Antivirus commands now use SafeCommand (added which/where to whitelist)
- Phase 2 H2: db_api.rs error responses use log_and_sanitize()
- Phase 2 H5: Removed duplicate sanitize_identifier (re-exports from sql_guard)

32 files modified for security hardening.
Moon deployment criteria: 10/10 met
2025-12-28 21:26:08 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
561264521c Add SECURITY_TASKS.md - security audit checklist, consolidate duplicate utils 2025-12-28 15:32:48 -03:00
aeb4e8af0f Fix all clippy warnings - 0 warnings 2025-12-28 14:27:52 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
36fb7988cb refactor: Move AutoTask system from basic/keywords to auto_task module
- Move app_generator, intent_classifier, intent_compiler, autotask_api, designer_ai, ask_later, auto_task, safety_layer to src/auto_task/
- Create auto_task/mod.rs with exports and route configuration
- Update imports in moved files
- Update main.rs to use auto_task::configure_autotask_routes
- Keep table_definition in keywords (shared utility)
2025-12-27 22:58:43 -03:00
a384678fb8 feat(autotask): Complete AutoTask flow with LLM-based app generation
- Add comprehensive platform capabilities prompt for LLM (all APIs, HTMX, BASIC)
- Add designer.js to all generated pages (dashboard, list, form)
- Add /api/autotask/pending endpoint for ASK LATER items
- Add /api/designer/modify endpoint for AI-powered app modifications
- Wire autotask routes in main.rs
- Create APP_GENERATOR_PROMPT.md with full API reference
- LLM decides everything - no hardcoded templates
2025-12-27 22:38:16 -03:00
06d0bf1f0a feat(autotask): Add missing API endpoints for UI compatibility
- Add execute_task_handler for POST /api/autotask/:task_id/execute
- Add get_task_logs_handler for GET /api/autotask/:task_id/logs
- Add apply_recommendation_handler for POST /api/autotask/recommendations/:rec_id/apply
- Update route configuration in mod.rs
2025-12-27 21:16:12 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
c2588d80d5 Fix conditional parameter usage and core_functions references 2025-12-24 09:50:30 -03:00
3a8510d191 Fix clippy warnings: raw strings, unused imports, format args, must_use attributes 2025-12-24 09:39:23 -03:00
7cbfe43319 Update attendance, keywords, calendar, compliance, console, core, drive, email, llm, msteams, security, and tasks modules 2025-12-24 09:29:27 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
a491cc13a6 Update botserver 2025-12-21 23:40:43 -03:00
dd91fc2d3b refactor: fix TaskEngine feature gate, thread-safe Extensions with Arc<RwLock> 2025-12-20 19:57:57 -03:00
57bf5b8754 feat: add ON EMAIL and ON CHANGE keywords for event-driven monitoring
- Add ON EMAIL keyword with FROM/SUBJECT filters
- Add ON CHANGE keyword with account:// syntax (gdrive, onedrive, dropbox, local)
- Add TriggerKind::EmailReceived (5) and FolderChange (6)
- Add migration 6.1.3_bot_hierarchy_monitors with:
  - email_monitors, folder_monitors tables
  - email_received_events, folder_change_events tables
  - user_organizations table
  - Bot hierarchy: parent_bot_id, enabled_tabs_json, inherit_parent_config
- Add 26 unit tests (12 on_email, 12 on_change, 2 trigger_kind)
- Update PROMPT.md with weekly maintenance checklist
- Zero warnings, zero errors
2025-12-18 16:17:58 -03:00
6bc6a35948 fix: resolve all warnings - wire up services properly 2025-12-17 17:41:37 -03:00
8405f1cfbb Fix warnings: add Debug derive, prefix unused vars 2025-12-17 11:26:58 -03:00
bac139d0b3 Add MCP server support via mcp.csv
- New mcp_directory.rs: McpCsvLoader to load MCP servers from mcp.csv
- CSV format: name,type,command,args,description,enabled,auth_type,auth_env
- Support for stdio, http, websocket, tcp connection types
- Support for api_key and bearer authentication
- Updated sources/mod.rs with MCP management API endpoints
- New sources/mcp.rs with helper functions
- MCP tools available to Tasks like BASIC keywords
2025-12-13 00:26:49 -03:00
bfc2ca817b Fix truncated files, implement TODOs, remove all compilation errors and warnings
- Complete truncated auto_task.rs, autotask_api.rs, intent_compiler.rs, mcp_client.rs, safety_layer.rs
- Add missing structs: RiskFactor, ResourceUsage, TaskError, RollbackState, TaskSchedule, HealthStatus
- Add missing IntentCompiler methods: call_llm, assess_risks, estimate_resources, check_ambiguity, store_compiled_intent
- Implement SET ATTENDANT STATUS database storage
- Implement queue position calculation in transfer_to_human
- Add llm_tokens to ResourceEstimate
- Fix all unused imports and variables
- Add proper derives (Copy, PartialOrd) where needed
2025-12-12 17:33:11 -03:00
585411d0e9 Update compiler and add goto_transform 2025-12-12 16:39:44 -03:00
65693cdee1 feat: Add Auto Task system - Intent Compiler, MCP Client, Safety Layer
- intent_compiler.rs: LLM-to-BASIC translation engine
- auto_task.rs: Auto-executing task data structures
- mcp_client.rs: Model Context Protocol server integration
- safety_layer.rs: Constraints, simulation, audit trail
- autotask_api.rs: HTTP API handlers for Auto Task UI
- Updated mod.rs with new modules and keyword list
2025-12-12 12:33:17 -03:00
fda88c0d94 Bootstrap started! 6.1.0
- Add rss and scraper dependencies for web data keywords
- Add SMS keyword with priority support (low, normal, high, urgent)
- Add web_data keywords: RSS, SCRAPE, SCRAPE_ALL, SCRAPE_TABLE, SCRAPE_LINKS, SCRAPE_IMAGES
- Add ai_tools keywords: TRANSLATE, OCR, SENTIMENT, CLASSIFY
- Improve LLM server health check with better diagnostics and increased timeout
- Fix compilation errors and warnings
- Register SMS keywords in BASIC engine
2025-12-10 18:22:02 -03:00
224ddd4a03 chore: Remove emoji icons from log messages and UI
Replace emoji icons with plain text/ASCII equivalents to prevent
encoding issues and improve console compatibility:
- Replace checkmarks with *
- Replace x marks with x
- Replace status indicators with text [OK], [ERR], etc.
- Remove decorative emojis from info/debug log messages
- Keep functional emojis in user-facing chat/sentiment features
2025-12-09 07:55:11 -03:00
c2aa82733a Add test infrastructure: AppState::default(), mock providers, fix attendance tests
- Add Default impl for AppState with mock services for testing
- Add MockLLMProvider for tests that need LLM without real API
- Add create_mock_auth_service() for Zitadel testing
- Add test_utils.rs with TestAppStateBuilder, MockChannelAdapter
- Enable rhai 'sync' feature to fix Send+Sync for Dynamic types
- Fix attendance.rs tests to use pure functions (no DB required)
- Fix on_error.rs tests to use String instead of i32
- Remove unused imports in attendance.rs

All tests pass, 0 warnings, 0 errors.
2025-12-05 16:43:14 -03:00
0120c904d7 feat(attendance): Multi-channel attendant config (email, phone, teams, google)
- Add email, teams, google columns to attendant.csv
- Add find_attendant_by_identifier() for multi-channel lookup
- Add find_attendants_by_channel() and find_attendants_by_department()
- Implement handle_status_command with database persistence
- Implement handle_transfer_command with actual transfer logic
- Update AttendantCSV struct with all new fields
2025-12-05 14:54:55 -03:00
5d93505012 feat(attendance): Add LLM-assisted attendant features
- Real-time tips when customer messages arrive
- Message polishing with one click
- Smart reply generation (3 contextual suggestions)
- Auto-summary when attendant takes conversation
- LLM-powered sentiment analysis with escalation warnings

WhatsApp Attendant Commands:
- /queue, /take, /status, /transfer, /resolve
- /tips, /polish, /replies, /summary, /help
- Portuguese versions: /fila, /pegar, /dicas, /polir, /respostas, /resumo

Config options (config.csv):
- attendant-llm-tips
- attendant-polish-message
- attendant-smart-replies
- attendant-auto-summary
- attendant-sentiment-analysis

API Endpoints:
- POST /api/attendance/llm/tips
- POST /api/attendance/llm/polish
- POST /api/attendance/llm/smart-replies
- GET /api/attendance/llm/summary/{session_id}
- POST /api/attendance/llm/sentiment
- GET /api/attendance/llm/config/{bot_id}

Uses bot's system prompt for consistency between bot and human-assisted responses.
2025-12-05 13:47:15 -03:00
6da93f6c36 feat: unified keywords with spaces, ON ERROR RESUME NEXT, unified DELETE
Keywords now use spaces instead of underscores:
- SEND MAIL (was SEND_MAIL)
- GENERATE PDF (was GENERATE_PDF)
- MERGE PDF (was MERGE_PDF)
- SET HEADER (was SET_HEADER)
- CLEAR HEADERS (was CLEAR_HEADERS)

New ON ERROR RESUME NEXT implementation:
- ON ERROR RESUME NEXT - enable error trapping
- ON ERROR GOTO 0 - disable error trapping
- CLEAR ERROR - clear error state
- ERROR MESSAGE - get last error message
- ERR - get error number

Unified DELETE keyword:
- DELETE url - HTTP DELETE (auto-detected)
- DELETE table, filter - Database DELETE
- DELETE path - File DELETE

Changes:
- errors/on_error.rs: New VB-style error handling
- errors/mod.rs: Include on_error module
- send_mail.rs: SEND MAIL with spaces
- file_operations.rs: GENERATE PDF, MERGE PDF with spaces
- data_operations.rs: Unified DELETE with auto-detection
- http_operations.rs: Cleaned up DELETE HTTP
- compiler/mod.rs: Removed underscore normalization
- mod.rs: Updated command list
2025-12-05 09:55:13 -03:00
d9b09e24d4 feat(email): implement email read tracking with pixel support
- Add email-read-pixel config parameter to enable/disable tracking
- Implement tracking pixel injection in HTML emails
- Add sent_email_tracking table with migration
- Create 4 new API endpoints:
  - GET /api/email/tracking/pixel/{id} - serve pixel & record read
  - GET /api/email/tracking/status/{id} - get email read status
  - GET /api/email/tracking/list - list all tracked emails
  - GET /api/email/tracking/stats - get aggregate statistics
- Store tracking data: read_at, read_count, IP, user_agent
- Integrate with send_email() to auto-inject pixel when enabled
2025-12-04 18:15:09 -03:00
c76de427e3 Implement real database functions, remove TODOs and placeholders
- CRM Lead Scoring: Implement get_lead_score_from_db and update_lead_score_in_db
  using bot_memories table with diesel queries
- Bot Manager: Implement real org lookup from database and template loading from filesystem
- KB Manager: Implement get_collection_info to query Qdrant for real statistics
- Analytics: Replace placeholder metrics with actual database queries for users,
  sessions, and storage stats
- Email Setup: Implement Stalwart admin account creation via management API
- Add CollectionInfo struct for Qdrant collection metadata

All implementations use diesel for database operations, no sqlx.
2025-12-03 22:23:30 -03:00
e9dad539af Update dependencies and remove problematic crates
Drop image (with ravif/paste), sqlx, zitadel, and related dependencies
that were causing compilation issues. Replace image processing with
direct png crate usage. Update rcgen to 0.14 with new API changes.
Refactor CA certificate generation to use Issuer pattern.
2025-12-03 16:05:30 -03:00
340ac8840a - New templates. 2025-12-03 07:15:54 -03:00
c4c9521dd9 - Split into botui. 2025-12-02 21:09:43 -03:00