Commit graph

3892 commits

Author SHA1 Message Date
b4647cd8d2 feat(rbac): implement complete RBAC middleware and route permissions
- Add rbac_middleware_fn for use in middleware layer chain
- Add RBAC middleware to request processing pipeline (after auth)
- Complete route permissions for ALL apps:
  - Anonymous: health, i18n, product, auth/login, chat, websocket
  - Authenticated users: drive, mail, calendar, tasks, docs, paper, sheet,
    slides, meet, research, sources, canvas, video, player, workspaces,
    projects, goals, settings, bots (read), designer, dashboards, crm,
    contacts, billing, products, tickets, learn, social, llm, autotask
  - Admin/SuperAdmin: users, groups, bot management, analytics, monitoring,
    audit, security, admin panel, attendant
  - SuperAdmin only: RBAC management
- Add all /api/ui/** HTMX routes with proper permissions
- Chat remains anonymous for customer support functionality
2026-01-10 11:41:25 -03:00
81b8fd8f2d fix(auth): handle Zitadel session tokens and grant Admin role
- Treat non-JWT bearer tokens as Zitadel session IDs
- Grant Admin role to valid sessions (temporary until proper role lookup)
- Add is_jwt_format helper to distinguish JWTs from session IDs
- Update RBAC to allow authenticated users access to UI monitoring routes
2026-01-10 11:14:33 -03:00
1686bfb454 feat(i18n): add missing navigation keys to TRANSLATION_KEYS
- Add nav-docs, nav-sheet, nav-slides, nav-social, nav-all-apps
- Add nav-people, nav-editor, nav-dashboards, nav-security
- Add nav-designer, nav-project, nav-canvas, nav-goals
- Add nav-player, nav-workspace, nav-video, nav-learn
- Add nav-crm, nav-billing, nav-products, nav-tickets
2026-01-10 10:53:56 -03:00
a15d020556 fix: add /api/i18n to anonymous paths for unauthenticated access 2026-01-10 10:27:01 -03:00
f07448d40a fix: add /js/* route to serve suite JS files like i18n.js 2026-01-10 10:13:46 -03:00
e3b3f04206 Normalize API paths: remove unnecessary /v1/ prefix
- Update all internal API routes from /api/v1/* to /api/*
- Protection API: /api/security/protection/*
- Botmodels calls: /api/vision/*, /api/audio/*, /api/speech/*
- Remove /api/v1/health from anonymous paths (keep /api/health)

External APIs (Reddit, Facebook, etc.) keep their original versioned paths
2026-01-10 09:48:43 -03:00
79ee009983 Fix: Add auth endpoints to anonymous paths list
- /api/auth/login was being blocked by auth middleware
- Add all auth endpoints to allow_anonymous_paths:
  - /api/auth/login
  - /api/auth/logout
  - /api/auth/refresh
  - /api/auth/bootstrap
  - /api/auth/2fa/verify
  - /api/auth/2fa/resend
  - /oauth
  - /auth/callback
2026-01-10 09:44:59 -03:00
faeae250bc Add security protection module with sudo-based privilege escalation
- Create installer.rs for 'botserver install protection' command
- Requires root to install packages and create sudoers config
- Sudoers uses exact commands (no wildcards) for security
- Update all tool files (lynis, rkhunter, chkrootkit, suricata, lmd) to use sudo
- Update manager.rs service management to use sudo
- Add 'sudo' and 'visudo' to command_guard.rs whitelist
- Update CLI with install/remove/status protection commands

Security model:
- Installation requires root (sudo botserver install protection)
- Runtime uses sudoers NOPASSWD for specific commands only
- No wildcards in sudoers - exact command specifications
- Tools run on host system, not in containers
2026-01-10 09:41:12 -03:00
27ecca0899 Fix Router type mismatch in project::configure and remove unused Html import 2026-01-09 19:19:41 -03:00
1c7a5c80b2 fix: Direct login without password change requirement
- Set change_required=false when creating admin password in Zitadel
- Admin can now login directly at /suite/login without forced password change
- Create security reminder file for admin to change password later
- Update console and credential file messages to reflect direct login
- Password change is recommended but not enforced on first login
2026-01-09 13:03:26 -03:00
115b2770cb fix: Add VAULT_CACERT to vault CLI commands in ensure_vault_unsealed
- vault status and unseal commands now use CA cert for TLS verification
- Fixes x509 certificate signed by unknown authority error
2026-01-09 12:29:45 -03:00
00acf1c76e fix: Add trusted_shell_script_arg for internal scripts
- shell_script_arg blocks $( and backticks for user input safety
- trusted_shell_script_arg allows these for internal installer scripts
- Internal scripts need shell features like command substitution
- Updated bootstrap, installer, facade, and llm modules
2026-01-09 12:13:35 -03:00
db267714ca fix: Allow URL-safe characters in SafeCommand arguments
- Allow &, ?, = in URL arguments (http:// or https://)
- Allow // pattern in URLs (needed for protocol)
- These are safe since Command::new().args() doesn't use shell
- Fixes Vault health check with query parameters
- Add debug logging to safe_curl and vault_health_check
2026-01-09 11:56:11 -03:00
cb59ceb60f fix: Smart mTLS for Vault - use client cert if exists, plain TLS during bootstrap
- Add vault_health_check() function that checks if client certs exist
- If certs exist: use mTLS (secure, post-installation)
- If certs don't exist yet: use plain TLS (during initial bootstrap)
- This allows bootstrap to complete while maintaining mTLS security after setup
- No security hole: mTLS is enforced once certs are generated
2026-01-09 11:23:49 -03:00
63aee6f6bc Revert "fix: Disable mTLS for Vault (temporary until new binary compiled)"
This reverts commit efb458ba66588de090fe6df76cdf04515eacce52.
2026-01-09 11:19:36 -03:00
460d39f399 fix: Disable mTLS for Vault (temporary until new binary compiled)
- Remove tls_client_ca_file from vault config templates
- Remove --cert/--key from health checks
- TLS still enabled for encryption, just no client cert required
- TODO: Re-enable mTLS when binary with cert health checks is compiled
2026-01-09 11:18:30 -03:00
703497b00c fix: Remove dead code and unused fields - proper fixes per PROMPT.md
- large_org_optimizer.rs: Remove unused fields (cached_at, roles, organization_id from structs)
  Add partition_manager() getter to use the field
- middleware.rs: Implement Display for AuthError to use InvalidToken message
- organization_rbac.rs: Remove unused user_groups and user_direct_permissions fields
- passkey.rs: Remove unused first_attempt_at field
- opencv.rs: Remove unused format field and ImageFormat enum
- rekognition.rs: Remove unused liveness_sessions field

No #[allow(dead_code)] - code is properly fixed or deleted
2026-01-09 11:00:14 -03:00
c59d456e25 fix: Add mTLS client certificate to all Vault health checks
- Keep mTLS enabled for security (even in dev)
- Add --cert and --key to all curl commands for Vault health checks
- Fix fetch_vault_credentials to use https and mTLS
- Fix Zitadel commands to use https with VAULT_CACERT
- All Vault communications now use proper mutual TLS
2026-01-09 10:48:13 -03:00
95fbb4f129 Revert "fix: Remove mTLS requirement from Vault config to fix health check failures"
This reverts commit 510fadf24b60c61bb39938f59d88add9b4c08168.
2026-01-09 10:45:41 -03:00
6c904f7dc9 fix: Remove mTLS requirement from Vault config to fix health check failures
- Remove tls_client_ca_file from vault config in installer.rs (Linux and macOS)
- Remove tls_client_ca_file from vault config in bootstrap/mod.rs
- TLS encryption still enabled, just no client certificate required
- Health checks now work with simple -sk curl flags
2026-01-09 10:33:11 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
41f9a778d1 fix: Add more missing types and fix duplicate derives
- Add ExportBounds and ExportError in whiteboard_export.rs
- Add RekognitionError in rekognition.rs
- Fix duplicate derive attributes on RefundResult and FallbackAttemptTracker
- Fix Recording -> WebinarRecording type references
2026-01-08 17:25:25 -03:00
3bf47a65d5 deps: Add ciborium crate for CBOR parsing in WebAuthn 2026-01-08 15:42:52 -03:00
a4cbf145d2 fix: Multiple compilation fixes per PROMPT.md
Video module:
- Fix state.db -> state.conn field name
- Fix analytics.rs imports placement
- Remove AppState dependency from websocket.rs (use global broadcaster)
- Simplify render.rs broadcaster usage

Other modules:
- Add sha1 crate dependency
- Fix AppState import paths (project, legal)
- Fix db_pool -> conn throughout codebase
- Add missing types: RefundResult, ExternalSyncError, TasksIntegrationError, RecordingError, FallbackAttemptTracker
- Add stub implementations for GoogleContactsClient, MicrosoftPeopleClient
- Fix social/mod.rs format string
- Fix designer/canvas.rs SVG path
- Remove doc comments per PROMPT.md
- Add missing handler implementations in calendar_integration.rs
2026-01-08 15:35:03 -03:00
998e4c2806 feat(video): Complete video editing module implementation
- Complete engine.rs with all AI-powered video operations
- Complete handlers.rs with 28+ HTTP API endpoints
- Add analytics.rs for video engagement tracking
- Add mcp_tools.rs for AI agent integration (6 tools)
- Add render.rs with FFmpeg worker and .gbdrive storage
- Add websocket.rs for real-time export progress
- Wire up all submodules and routes in mod.rs

AI features: transcription, auto-captions, TTS, scene detection,
auto-reframe, background removal, enhancement, beat sync, waveforms

Follows PROMPT.md: SafeCommand, SafeErrorResponse, no unwrap/comments
2026-01-08 13:44:45 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
29b80f597c Fix email_accounts -> user_email_accounts table name typo in list_emails_htmx 2026-01-04 08:48:27 -03:00
b2c5895887 Add migration for bot database_name column
- Creates database_name column in bots table
- Creates tenant_id column for multi-tenant support
- Adds indexes for both columns
- Migration runs automatically on server start
2026-01-02 19:56:39 -03:00
c743754c6c Add per-bot database creation
- Added database_name field to bots schema
- Bot creation now creates a dedicated PostgreSQL database (bot_{name})
- Updated add_bot.rs to create database and store database_name
- Added create_bot_database() function with safe name validation
- Added dynamic table check to all db_api handlers
2026-01-02 19:34:59 -03:00
65b2583add Add dynamic table check to all db_api handlers
- Added is_table_allowed_with_conn check to get_record_handler
- Added is_table_allowed_with_conn check to create_record_handler
- Added is_table_allowed_with_conn check to update_record_handler
- Added is_table_allowed_with_conn check to delete_record_handler
- Returns 404 with clear message if table doesn't exist
2026-01-02 19:18:58 -03:00
56265ee5d2 Clean LLM output for generated files
- Added clean_file_content() to strip markdown fences from LLM responses
- For HTML: extracts content from <!DOCTYPE> to </html>
- For CSS/JS: filters out explanation lines (Here, This, Note:, etc)
- Fixes malformed HTML output from LLM
2026-01-02 19:15:15 -03:00
a43aea3320 Serve vendor files (htmx) from MinIO instead of local filesystem
- Added serve_vendor_file() to serve from {bot}.gblib/vendor/ in MinIO
- Added /js/vendor/* route to app_server
- Removed local ServeDir for /js/vendor from main.rs
- Added ensure_vendor_files_in_minio() to upload htmx.min.js on startup
- Uses include_bytes! to embed htmx.min.js in binary
2026-01-02 18:26:34 -03:00
bbbb9e190f Allow dynamic tables from app_generator in db_api
- Added table_exists_in_database() to check if table exists in PostgreSQL
- Updated validate_table_name() to allow valid identifiers (not just whitelist)
- Added validate_table_name_with_conn() for full validation with DB check
- Added is_table_allowed_with_conn() for handlers to verify table existence
- Updated list_records_handler and count_records_handler to use dynamic check
- Uses parameterized query for table existence check (SQL injection safe)
2026-01-02 18:20:04 -03:00
762620f7a9 Rewrite CDN URLs to local paths when serving HTML
- Added rewrite_cdn_urls() to replace HTMX CDN URLs with /js/vendor/htmx.min.js
- Applied to both MinIO and filesystem serving paths
- Supports unpkg.com, jsdelivr, cdnjs variants
- Ensures old apps with CDN references work with local files
2026-01-02 18:12:31 -03:00
2f045bffa5 Serve HTMX locally - no CDN dependencies
- Added /js/vendor route to serve local vendor JS files
- Downloaded htmx.min.js v1.9.10 to botserver-stack/static/js/vendor/
- Reverted CSP to strict 'self' only (no external CDN)
- Updated APP_GENERATOR_PROMPT to use /js/vendor/htmx.min.js
- Updated designer prompt to use local HTMX path
2026-01-02 17:54:36 -03:00
7bad8d50f7 Fix designer file writing and CSP for CDN assets
- Designer now uses state.bucket_name (like app_generator) instead of DB lookup
- Fixed local file path to match app_server fallback: {site_path}/{bot}.gbai/{bot}.gbapp/{app}/{file}
- Fixed S3 path to match app_server: {bot}.gbapp/{app}/{file} in bucket {bot}.gbai
- Added S3 bucket creation retry logic (like app_generator)
- Updated CSP to allow unpkg.com, cdnjs.cloudflare.com, cdn.jsdelivr.net for scripts/styles
- Added fonts.googleapis.com and fonts.gstatic.com for web fonts
- Updated APP_GENERATOR_PROMPT to use HTMX CDN instead of non-existent /js/vendor path
- Added designer prompt guidelines for relative asset paths
2026-01-02 17:48:51 -03:00
021080d763 fix: Mark child sections as Completed when all items inside are completed 2026-01-02 14:44:54 -03:00
20b5755c4c fix: Remove duplicate title from STATUS section 2026-01-02 14:23:34 -03:00
87f6589219 fix: Add expanded class to tree sections/children by default 2026-01-02 13:59:51 -03:00
f4aae20e05 fix: Sequential phase execution (DB→Files), compact header, remove URL duplicate 2026-01-02 13:23:00 -03:00
86ac5ca8f5 Fix panic: check if item_groups is empty before iterating
Prevents 'index out of bounds: the len is 0 but the index is 0' error
in complete_item_group_range when item_groups is empty.
2026-01-02 12:54:54 -03:00
938e154c8e Fix progress UI: remove View Details, add section dot on right side 2026-01-01 10:36:42 -03:00
1f9c3fbdf4 Fix progress UI: dots instead of checkboxes, proper durations, status indicators
- Remove checkbox text [x]/[>]/[ ] from HTML, use CSS dots only
- Add View Details links to sections
- Add status-indicator (pulsing dot) and status-gear icon
- Auto-expand running sections/children
- Format runtime and estimated time properly
- Fix decision point display
2026-01-01 10:13:27 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
bad6ebd501 Add /apps to public paths - no auth required for app access 2025-12-31 13:11:16 -03:00
8fbc52b054 Show real file progress and app URL on completion
- Update step_results in DB with real file list during generation
- Show app URL in completion event and notification
- Update task progress/current_step/total_steps as files are written
- Mark task as completed with app_url when done
2025-12-31 12:51:27 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00