Commit graph

163 commits

Author SHA1 Message Date
9087bb17cd feat: complete General Bots 7.0 (v6.2.0) 2026-01-25 10:29:54 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
6fa52e1dd8 feat: implement feature bundling architecture and fix conditional compilation
- Restructured Cargo.toml with Bundle Pattern for easy feature selection
- Added feature bundles: tasks → automation + drive + monitoring
- Applied conditional compilation guards throughout codebase:
  * AppState fields (drive, cache, task_engine, task_scheduler)
  * main.rs initialization (S3, Redis, Tasks)
  * SessionManager Redis usage
  * bootstrap S3/Drive operations
  * compiler task scheduling
  * shared module Task/NewTask exports
- Eliminated all botserver compilation warnings
- Minimal build now compiles successfully
- Accepted core dependencies: automation (Rhai), drive (S3), cache (Redis)
- Created DEPENDENCY_FIX_PLAN.md with complete documentation

Minimal feature set: chat + automation + drive + cache
Verified: cargo check -p botserver --no-default-features --features minimal 
2026-01-23 13:14:20 -03:00
66abce913f Feature gating refactor: modular compilation with minimal feature set 2026-01-22 19:45:18 -03:00
fc9c07d204 Refactor schema.rs into modular files 2026-01-22 13:57:40 -03:00
4f63065da2 Update botserver submodule 2026-01-19 15:43:45 -03:00
5126c648ff Auto-commit: 20260118_195334 2026-01-18 19:53:34 -03:00
033bb504b9 Various updates: dependencies, features, and bug fixes 2026-01-16 11:29:22 -03:00
e4524d0584 Add SCAN BARCODE keyword and BotModelsClient.scan_barcode 2026-01-14 11:43:34 -03:00
ee9341163f Add PRODUCTS, PRODUCT, SEARCH PRODUCTS keywords for ERP integration 2026-01-14 10:20:07 -03:00
1c4cc2f986 Add SEARCH keyword and store tools: classify-product, search-products, calculate-shipping 2026-01-14 10:13:33 -03:00
a2783f9b32 Fix 5 errors and 32 warnings: calendar, compliance, billing_alert_broadcast, unused vars 2026-01-13 22:21:25 -03:00
31777432b4 Implement TODO items: session auth, face API, task logs, intent storage
Learn Module:
- All 9 handlers now use AuthenticatedUser extractor

Security:
- validate_session_sync reads roles from SESSION_CACHE

AutoTask:
- get_task_logs reads from manifest with status logs
- store_compiled_intent saves to cache and database

Face API:
- AWS Rekognition, OpenCV, InsightFace implementations
- Detection, verification, analysis methods

Other fixes:
- Calendar/task integration database queries
- Recording database methods
- Analytics insights trends
- Email/folder monitoring mock data
2026-01-13 14:48:49 -03:00
3e75bbff97 MS Office 100% Compatibility - Phase 1 Implementation
- Add rust_xlsxwriter for Excel export with formatting support
- Add docx-rs for Word document import/export with HTML conversion
- Add PPTX export support with slides, shapes, and text elements
- Refactor sheet module into 7 files (types, formulas, handlers, etc)
- Refactor docs module into 6 files (types, handlers, storage, etc)
- Refactor slides module into 6 files (types, handlers, storage, etc)
- Fix collaboration modules (borrow issues, rand compatibility)
- Add ooxmlsdk dependency for future Office 2021 features
- Fix type mismatches in slides storage
- Update security protection API router type

Features:
- Excel: Read xlsx/xlsm/xls, write xlsx with styles
- Word: Read/write docx with formatting preservation
- PowerPoint: Write pptx with slides, shapes, text
- Real-time collaboration via WebSocket (already working)
- Theme-aware UI with --sentient-* CSS variables
2026-01-11 09:56:15 -03:00
46695c0f75 feat(security): add BASIC keywords for security protection tools
Add security_protection.rs with 8 new BASIC keywords:
- SECURITY TOOL STATUS - Check if tool is installed/running
- SECURITY RUN SCAN - Execute security scan
- SECURITY GET REPORT - Get latest scan report
- SECURITY UPDATE DEFINITIONS - Update signatures
- SECURITY START SERVICE - Start security service
- SECURITY STOP SERVICE - Stop security service
- SECURITY INSTALL TOOL - Install security tool
- SECURITY HARDENING SCORE - Get Lynis hardening index

Also:
- Registered protection routes in main.rs
- Added Security Protection category to keywords list
- All functions use proper error handling (no unwrap/expect)
2026-01-10 20:32:56 -03:00
f07448d40a fix: add /js/* route to serve suite JS files like i18n.js 2026-01-10 10:13:46 -03:00
e3b3f04206 Normalize API paths: remove unnecessary /v1/ prefix
- Update all internal API routes from /api/v1/* to /api/*
- Protection API: /api/security/protection/*
- Botmodels calls: /api/vision/*, /api/audio/*, /api/speech/*
- Remove /api/v1/health from anonymous paths (keep /api/health)

External APIs (Reddit, Facebook, etc.) keep their original versioned paths
2026-01-10 09:48:43 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
c743754c6c Add per-bot database creation
- Added database_name field to bots schema
- Bot creation now creates a dedicated PostgreSQL database (bot_{name})
- Updated add_bot.rs to create database and store database_name
- Added create_bot_database() function with safe name validation
- Added dynamic table check to all db_api handlers
2026-01-02 19:34:59 -03:00
65b2583add Add dynamic table check to all db_api handlers
- Added is_table_allowed_with_conn check to get_record_handler
- Added is_table_allowed_with_conn check to create_record_handler
- Added is_table_allowed_with_conn check to update_record_handler
- Added is_table_allowed_with_conn check to delete_record_handler
- Returns 404 with clear message if table doesn't exist
2026-01-02 19:18:58 -03:00
a43aea3320 Serve vendor files (htmx) from MinIO instead of local filesystem
- Added serve_vendor_file() to serve from {bot}.gblib/vendor/ in MinIO
- Added /js/vendor/* route to app_server
- Removed local ServeDir for /js/vendor from main.rs
- Added ensure_vendor_files_in_minio() to upload htmx.min.js on startup
- Uses include_bytes! to embed htmx.min.js in binary
2026-01-02 18:26:34 -03:00
bbbb9e190f Allow dynamic tables from app_generator in db_api
- Added table_exists_in_database() to check if table exists in PostgreSQL
- Updated validate_table_name() to allow valid identifiers (not just whitelist)
- Added validate_table_name_with_conn() for full validation with DB check
- Added is_table_allowed_with_conn() for handlers to verify table existence
- Updated list_records_handler and count_records_handler to use dynamic check
- Uses parameterized query for table existence check (SQL injection safe)
2026-01-02 18:20:04 -03:00
762620f7a9 Rewrite CDN URLs to local paths when serving HTML
- Added rewrite_cdn_urls() to replace HTMX CDN URLs with /js/vendor/htmx.min.js
- Applied to both MinIO and filesystem serving paths
- Supports unpkg.com, jsdelivr, cdnjs variants
- Ensures old apps with CDN references work with local files
2026-01-02 18:12:31 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
a5dee11002 Security audit: Remove all production .unwrap()/.expect(), add SafeCommand, ErrorSanitizer
- Phase 1 Critical: All 115 .unwrap() verified in test code only
- Phase 1 Critical: All runtime .expect() converted to proper error handling
- Phase 2 H1: Antivirus commands now use SafeCommand (added which/where to whitelist)
- Phase 2 H2: db_api.rs error responses use log_and_sanitize()
- Phase 2 H5: Removed duplicate sanitize_identifier (re-exports from sql_guard)

32 files modified for security hardening.
Moon deployment criteria: 10/10 met
2025-12-28 21:26:08 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
561264521c Add SECURITY_TASKS.md - security audit checklist, consolidate duplicate utils 2025-12-28 15:32:48 -03:00
aeb4e8af0f Fix all clippy warnings - 0 warnings 2025-12-28 14:27:52 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
36fb7988cb refactor: Move AutoTask system from basic/keywords to auto_task module
- Move app_generator, intent_classifier, intent_compiler, autotask_api, designer_ai, ask_later, auto_task, safety_layer to src/auto_task/
- Create auto_task/mod.rs with exports and route configuration
- Update imports in moved files
- Update main.rs to use auto_task::configure_autotask_routes
- Keep table_definition in keywords (shared utility)
2025-12-27 22:58:43 -03:00
a384678fb8 feat(autotask): Complete AutoTask flow with LLM-based app generation
- Add comprehensive platform capabilities prompt for LLM (all APIs, HTMX, BASIC)
- Add designer.js to all generated pages (dashboard, list, form)
- Add /api/autotask/pending endpoint for ASK LATER items
- Add /api/designer/modify endpoint for AI-powered app modifications
- Wire autotask routes in main.rs
- Create APP_GENERATOR_PROMPT.md with full API reference
- LLM decides everything - no hardcoded templates
2025-12-27 22:38:16 -03:00
06d0bf1f0a feat(autotask): Add missing API endpoints for UI compatibility
- Add execute_task_handler for POST /api/autotask/:task_id/execute
- Add get_task_logs_handler for GET /api/autotask/:task_id/logs
- Add apply_recommendation_handler for POST /api/autotask/recommendations/:rec_id/apply
- Update route configuration in mod.rs
2025-12-27 21:16:12 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00
c2588d80d5 Fix conditional parameter usage and core_functions references 2025-12-24 09:50:30 -03:00
3a8510d191 Fix clippy warnings: raw strings, unused imports, format args, must_use attributes 2025-12-24 09:39:23 -03:00
7cbfe43319 Update attendance, keywords, calendar, compliance, console, core, drive, email, llm, msteams, security, and tasks modules 2025-12-24 09:29:27 -03:00
883c6d07e1 Remove all code comments and fix ratatui version 2025-12-23 18:40:58 -03:00
e2a5bf091a Update server components and keywords 2025-12-23 15:52:35 -03:00
a491cc13a6 Update botserver 2025-12-21 23:40:43 -03:00
dd91fc2d3b refactor: fix TaskEngine feature gate, thread-safe Extensions with Arc<RwLock> 2025-12-20 19:57:57 -03:00
57bf5b8754 feat: add ON EMAIL and ON CHANGE keywords for event-driven monitoring
- Add ON EMAIL keyword with FROM/SUBJECT filters
- Add ON CHANGE keyword with account:// syntax (gdrive, onedrive, dropbox, local)
- Add TriggerKind::EmailReceived (5) and FolderChange (6)
- Add migration 6.1.3_bot_hierarchy_monitors with:
  - email_monitors, folder_monitors tables
  - email_received_events, folder_change_events tables
  - user_organizations table
  - Bot hierarchy: parent_bot_id, enabled_tabs_json, inherit_parent_config
- Add 26 unit tests (12 on_email, 12 on_change, 2 trigger_kind)
- Update PROMPT.md with weekly maintenance checklist
- Zero warnings, zero errors
2025-12-18 16:17:58 -03:00
6bc6a35948 fix: resolve all warnings - wire up services properly 2025-12-17 17:41:37 -03:00
8405f1cfbb Fix warnings: add Debug derive, prefix unused vars 2025-12-17 11:26:58 -03:00
bac139d0b3 Add MCP server support via mcp.csv
- New mcp_directory.rs: McpCsvLoader to load MCP servers from mcp.csv
- CSV format: name,type,command,args,description,enabled,auth_type,auth_env
- Support for stdio, http, websocket, tcp connection types
- Support for api_key and bearer authentication
- Updated sources/mod.rs with MCP management API endpoints
- New sources/mcp.rs with helper functions
- MCP tools available to Tasks like BASIC keywords
2025-12-13 00:26:49 -03:00
bfc2ca817b Fix truncated files, implement TODOs, remove all compilation errors and warnings
- Complete truncated auto_task.rs, autotask_api.rs, intent_compiler.rs, mcp_client.rs, safety_layer.rs
- Add missing structs: RiskFactor, ResourceUsage, TaskError, RollbackState, TaskSchedule, HealthStatus
- Add missing IntentCompiler methods: call_llm, assess_risks, estimate_resources, check_ambiguity, store_compiled_intent
- Implement SET ATTENDANT STATUS database storage
- Implement queue position calculation in transfer_to_human
- Add llm_tokens to ResourceEstimate
- Fix all unused imports and variables
- Add proper derives (Copy, PartialOrd) where needed
2025-12-12 17:33:11 -03:00