Commit graph

26 commits

Author SHA1 Message Date
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
6fa52e1dd8 feat: implement feature bundling architecture and fix conditional compilation
- Restructured Cargo.toml with Bundle Pattern for easy feature selection
- Added feature bundles: tasks → automation + drive + monitoring
- Applied conditional compilation guards throughout codebase:
  * AppState fields (drive, cache, task_engine, task_scheduler)
  * main.rs initialization (S3, Redis, Tasks)
  * SessionManager Redis usage
  * bootstrap S3/Drive operations
  * compiler task scheduling
  * shared module Task/NewTask exports
- Eliminated all botserver compilation warnings
- Minimal build now compiles successfully
- Accepted core dependencies: automation (Rhai), drive (S3), cache (Redis)
- Created DEPENDENCY_FIX_PLAN.md with complete documentation

Minimal feature set: chat + automation + drive + cache
Verified: cargo check -p botserver --no-default-features --features minimal 
2026-01-23 13:14:20 -03:00
66abce913f Feature gating refactor: modular compilation with minimal feature set 2026-01-22 19:45:18 -03:00
a2783f9b32 Fix 5 errors and 32 warnings: calendar, compliance, billing_alert_broadcast, unused vars 2026-01-13 22:21:25 -03:00
31777432b4 Implement TODO items: session auth, face API, task logs, intent storage
Learn Module:
- All 9 handlers now use AuthenticatedUser extractor

Security:
- validate_session_sync reads roles from SESSION_CACHE

AutoTask:
- get_task_logs reads from manifest with status logs
- store_compiled_intent saves to cache and database

Face API:
- AWS Rekognition, OpenCV, InsightFace implementations
- Detection, verification, analysis methods

Other fixes:
- Calendar/task integration database queries
- Recording database methods
- Analytics insights trends
- Email/folder monitoring mock data
2026-01-13 14:48:49 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
29b80f597c Fix email_accounts -> user_email_accounts table name typo in list_emails_htmx 2026-01-04 08:48:27 -03:00
56265ee5d2 Clean LLM output for generated files
- Added clean_file_content() to strip markdown fences from LLM responses
- For HTML: extracts content from <!DOCTYPE> to </html>
- For CSS/JS: filters out explanation lines (Here, This, Note:, etc)
- Fixes malformed HTML output from LLM
2026-01-02 19:15:15 -03:00
2f045bffa5 Serve HTMX locally - no CDN dependencies
- Added /js/vendor route to serve local vendor JS files
- Downloaded htmx.min.js v1.9.10 to botserver-stack/static/js/vendor/
- Reverted CSP to strict 'self' only (no external CDN)
- Updated APP_GENERATOR_PROMPT to use /js/vendor/htmx.min.js
- Updated designer prompt to use local HTMX path
2026-01-02 17:54:36 -03:00
7bad8d50f7 Fix designer file writing and CSP for CDN assets
- Designer now uses state.bucket_name (like app_generator) instead of DB lookup
- Fixed local file path to match app_server fallback: {site_path}/{bot}.gbai/{bot}.gbapp/{app}/{file}
- Fixed S3 path to match app_server: {bot}.gbapp/{app}/{file} in bucket {bot}.gbai
- Added S3 bucket creation retry logic (like app_generator)
- Updated CSP to allow unpkg.com, cdnjs.cloudflare.com, cdn.jsdelivr.net for scripts/styles
- Added fonts.googleapis.com and fonts.gstatic.com for web fonts
- Updated APP_GENERATOR_PROMPT to use HTMX CDN instead of non-existent /js/vendor path
- Added designer prompt guidelines for relative asset paths
2026-01-02 17:48:51 -03:00
021080d763 fix: Mark child sections as Completed when all items inside are completed 2026-01-02 14:44:54 -03:00
f4aae20e05 fix: Sequential phase execution (DB→Files), compact header, remove URL duplicate 2026-01-02 13:23:00 -03:00
86ac5ca8f5 Fix panic: check if item_groups is empty before iterating
Prevents 'index out of bounds: the len is 0 but the index is 0' error
in complete_item_group_range when item_groups is empty.
2026-01-02 12:54:54 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
8fbc52b054 Show real file progress and app URL on completion
- Update step_results in DB with real file list during generation
- Show app URL in completion event and notification
- Update task progress/current_step/total_steps as files are written
- Mark task as completed with app_url when done
2025-12-31 12:51:27 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
928f29e888 feat(security): Complete security wiring and log audit
SECURITY WIRING:
- Auth middleware wired to main router with AnonymousPath config
- CORS allowed origins loaded from bot_configuration database (config.csv)
- Zitadel auth config loads from Vault via SecretsManager
- No more env vars for sensitive config (only VAULT_* allowed)

LOG AUDIT:
- Added is_sensitive_config_key() check in ask_later.rs
- Sensitive config values (password, secret, token, key, etc) now logged as [REDACTED]
- Removed potential credential exposure in pending_info logs

CONFIG LOADING ORDER:
1. VAULT_ADDR and VAULT_TOKEN from .env
2. All secrets from Vault (gbo/directory for Zitadel)
3. Bot config from config.csv (cors-allowed-origins, etc)

Auth Config Paths:
- Anonymous: /health, /healthz, /api/health, /ws, /auth
- Public: /static, /favicon.ico
2025-12-28 19:41:33 -03:00
561264521c Add SECURITY_TASKS.md - security audit checklist, consolidate duplicate utils 2025-12-28 15:32:48 -03:00
aeb4e8af0f Fix all clippy warnings - 0 warnings 2025-12-28 14:27:52 -03:00
96cf7b57f8 App generator LLM-only, app logs, knowledge base, web search, designer magic 2025-12-28 11:50:50 -03:00
9dee342817 fix(intent_compiler): Use LLM-generated basic_code instead of hardcoded keyword mapping
- Remove hardcoded keyword switch/case (CREATE_TASK, LLM, RUN_PYTHON, etc.)
- Use step.basic_code directly from LLM generation
- LLM generates BASIC code for tools/schedulers/monitors
- Simple fallback if no code generated
2025-12-27 23:29:21 -03:00
36fb7988cb refactor: Move AutoTask system from basic/keywords to auto_task module
- Move app_generator, intent_classifier, intent_compiler, autotask_api, designer_ai, ask_later, auto_task, safety_layer to src/auto_task/
- Create auto_task/mod.rs with exports and route configuration
- Update imports in moved files
- Update main.rs to use auto_task::configure_autotask_routes
- Keep table_definition in keywords (shared utility)
2025-12-27 22:58:43 -03:00