diff --git a/PROMPT.md b/PROMPT.md index e25d2ab59..9dcaf1f11 100644 --- a/PROMPT.md +++ b/PROMPT.md @@ -1,6 +1,7 @@ -# botserver Development Prompt Guide +# botserver Development Guide -**Version:** 6.1.0 +**Version:** 6.2.0 +**Purpose:** Main API server for General Bots (Axum + Diesel + Rhai BASIC + HTMX in botui) --- @@ -10,7 +11,7 @@ --- -## ABSOLUTE PROHIBITIONS +## ❌ ABSOLUTE PROHIBITIONS ``` ❌ NEVER use #![allow()] or #[allow()] in source code @@ -29,7 +30,7 @@ --- -## SECURITY REQUIREMENTS +## 🔐 SECURITY REQUIREMENTS ### Error Handling @@ -113,7 +114,7 @@ Command::new("/usr/bin/tool").arg(safe).output()?; --- -## CODE PATTERNS +## ✅ CODE PATTERNS ### Format Strings - Inline Variables @@ -172,23 +173,39 @@ date.with_hour(9).and_then(|d| d.with_minute(0)).unwrap_or(date) --- -## DATABASE STANDARDS +## 📁 KEY DIRECTORIES -- TABLES AND INDEXES ONLY (no views, triggers, functions) -- JSON columns: use TEXT with `_json` suffix -- Use diesel - no sqlx +``` +src/ +├── core/ # Bootstrap, config, routes +├── basic/ # Rhai BASIC interpreter +│ └── keywords/ # BASIC keyword implementations +├── security/ # Security modules +├── shared/ # Shared types, models +├── tasks/ # AutoTask system +└── auto_task/ # App generator +``` --- -## FRONTEND RULES +## 🗄️ DATABASE STANDARDS -- Use HTMX - minimize JavaScript -- NO external CDN - all assets local -- Server-side rendering with Askama templates +- **TABLES AND INDEXES ONLY** (no views, triggers, functions) +- **JSON columns:** use TEXT with `_json` suffix +- **ORM:** Use diesel - no sqlx +- **Migrations:** Located in `botserver/migrations/` --- -## DEPENDENCIES +## 🎨 FRONTEND RULES + +- **Use HTMX** - minimize JavaScript +- **NO external CDN** - all assets local +- **Server-side rendering** with Askama templates + +--- + +## 📦 KEY DEPENDENCIES | Library | Version | Purpose | |---------|---------|---------| @@ -202,35 +219,7 @@ date.with_hour(9).and_then(|d| d.with_minute(0)).unwrap_or(date) --- -## COMPILATION POLICY - CRITICAL - -**NEVER compile during development. NEVER run `cargo build`. Use static analysis only.** - -### Workflow - -1. Make all code changes -2. Use `diagnostics` tool for static analysis (NOT compilation) -3. Fix any errors found by diagnostics -4. **At the end**, inform user what needs restart - -### After All Changes Complete - -| Change Type | User Action Required | -|-------------|----------------------| -| Rust code (`.rs` files) | "Recompile and restart **botserver**" | -| HTML templates (`.html` in botui) | "Browser refresh only" | -| CSS/JS files | "Browser refresh only" | -| Askama templates (`.html` in botserver) | "Recompile and restart **botserver**" | -| Database migrations | "Run migration, then restart **botserver**" | -| Cargo.toml changes | "Recompile and restart **botserver**" | - -**Format:** At the end of your response, always state: -- ✅ **No restart needed** - browser refresh only -- 🔄 **Restart botserver** - recompile required - ---- - -## KEY REMINDERS +## 🔑 REMEMBER - **ZERO WARNINGS** - fix every clippy warning - **ZERO COMMENTS** - no comments, no doc comments @@ -239,6 +228,6 @@ date.with_hour(9).and_then(|d| d.with_minute(0)).unwrap_or(date) - **NO UNWRAP/EXPECT** - use ? or combinators - **PARAMETERIZED SQL** - never format! for queries - **VALIDATE COMMANDS** - never pass raw user input -- **USE DIAGNOSTICS** - never call cargo clippy directly - **INLINE FORMAT ARGS** - `format!("{name}")` not `format!("{}", name)` -- **USE SELF** - in impl blocks, use Self not type name \ No newline at end of file +- **USE SELF** - in impl blocks, use Self not type name +- **Version 6.2.0** - do not change without approval \ No newline at end of file diff --git a/TASKS.md b/TASKS.md deleted file mode 100644 index 1205f93ee..000000000 --- a/TASKS.md +++ /dev/null @@ -1,152 +0,0 @@ -# Deep Feature Gating Plan: Schema & Migrations - -## Objective -Reduce binary size and runtime overhead by strictly feature-gating Database Schema (`schema.rs`) and Migrations. -The goal is that a "minimal" build (chat-only) should: -1. NOT compile code for `tasks`, `mail`, `calendar`, etc. tables. -2. NOT run database migrations for those features (preventing unused tables in DB). -3. NOT fail compilation due to missing table definitions in `joinable!` macros. - ---- - -## 🚀 Phase 1: Split Diesel Schema (The Hard Part) - -The current `schema.rs` is monolithic (3000+ lines). -Diesel requires `joinable!` and `allow_tables_to_appear_in_same_query!` macros to know about relationship graphs. - -### **1.1 Analysis & grouping** -Identify which tables belong to which feature and strictly define their relationships. -- **Core**: `users`, `bots`, `sessions`, `organizations`, `rbac_*` -- **Tasks**: `tasks`, `task_comments` (links to `users`, `projects`) -- **Mail**: `email_*` (links to `users`) -- **Drive**: (Uses S3, but maybe metadata tables?) -- **People/CRM**: `crm_*` (links to `users`, `organizations`) - -### **1.2 Create Modular Schema Files** -Instead of one file, we will separate them and use `mod.rs` to aggregate. -- `src/core/shared/schema/mod.rs` (Aggregator) -- `src/core/shared/schema/core.rs` (Base tables) -- `src/core/shared/schema/tasks.rs` (Gated `#[cfg(feature="tasks")]`) -- `src/core/shared/schema/mail.rs` (Gated `#[cfg(feature="mail")]`) -...etc. - -### **1.3 Solve the `allow_tables_to_appear_in_same_query!` Problem** -The single massive macro prevents gating. -**Solution**: Break the "Universe" assumption. -1. Define a `core_tables!` macro. -2. Define `task_tables!` macro (gated). -3. We might lose the ability to write arbitrary joins between *any* two tables if they are in different clusters, unless we explicitly define `joinable!` in the aggregator. -4. **Strategy**: - - Keep `joinable!` definitions relative to the feature. - - If `TableA` (Core) joins `TableB` (Tasks), the `joinable!` must be gated by `tasks`. - - The `allow_tables...` output must be constructed dynamically or strict subsets defined. - -### **1.4 Comprehensive Feature Mapping** -Based on `Cargo.toml`, we need to map all optional features to their schema requirements. - -| Feature | Tables (Prefix/Name) | Migrations | -|---------|-----------------------|------------| -| (core) | `users`, `bots`, `sessions`, `orgs`, `rbac_*`, `system_automations`, `bot_memories`, `kb_*` | `migrations/core/` | -| `tasks` | `tasks`, `task_comments` | `migrations/tasks/` | -| `calendar`| `calendars`, `calendar_events`, `calendar_*` | `migrations/calendar/` | -| `mail` | `email_*`, `distribution_lists`, `global_email_signatures` | `migrations/mail/` | -| `people` | `crm_contacts`, `crm_accounts`, `crm_leads`, `crm_opportunities`, `crm_activities`, `crm_pipeline_*`, `people_*` | `migrations/people/` | -| `compliance`| `compliance_*`, `legal_*`, `cookie_consents`, `data_*` | `migrations/compliance/` | -| `meet` | `meet_*`, `meeting_*`, `whiteboard_*` | `migrations/meet/` | -| `attendant`| `attendant_*` | `migrations/attendant/` | -| `analytics`| `dashboards`, `dashboard_*` | `migrations/analytics/` | -| `drive` | (No direct tables? Check `files` or `drive_*`?) | `migrations/drive/` | -| `billing` | `billing_*`, `products`, `services`, `price_*`, `inventory_*` (See note) | `migrations/billing/` | -| `social` | `social_*` | `migrations/social/` | -| `canvas` | `canvases`, `canvas_*` | `migrations/canvas/` | -| `workspaces`| `workspaces`, `workspace_*` | `migrations/workspaces/` | -| `research` | `research_*` | `migrations/research/` | -| `goals` | `okr_*` | `migrations/goals/` | -| `feedback` | `support_tickets`, `ticket_*` | `migrations/tickets/` | - -**Note**: `billing` and `tickets` are currently ungated modules in `main.rs`. We should create features or group them. For now, assume they are part of admin/core or need gating. - ---- - -## 🚀 Phase 2: Feature-Gated Migrations - -Currently, `embed_migrations!("migrations")` embeds everything. - -### **2.1 Split Migration Directories** -Refactor the flat `migrations/` folder into feature-specific directories (requires custom runner logic or Diesel configuration trickery). -Alternately (easier code-wise): -- Keep flat structure but **edit the migration SQL files** to be conditional? No, SQL doesn't support `#[cfg]`. -- **Better Approach**: Use multiple `embed_migrations!` calls. - - `migrations/core/` -> Always run. - - `migrations/tasks/` -> Run if `tasks` feature on. - - `migrations/mail/` -> Run if `mail` feature on. - -**Action Plan**: -1. Organize `migrations/` into subdirectories: `core/`, `tasks/`, `mail/`, `people/`. -2. Update `utils.rs` migration logic: - ```rust - pub fn run_migrations(conn: &mut PgConnection) { - // Core - const CORE: EmbeddedMigrations = embed_migrations!("migrations/core"); - conn.run_pending_migrations(CORE).unwrap(); - - #[cfg(feature = "tasks")] { - const TASKS: EmbeddedMigrations = embed_migrations!("migrations/tasks"); - conn.run_pending_migrations(TASKS).unwrap(); - } - // ... - } - ``` - -### **2.2 Verify Migration Dependencies** -Ensure that `tasks` migrations (which might foreign-key to `users`) only run *after* `core` migrations are applied. - ---- - -## 🚀 Phase 3: Fix Dependent Code (Strict Gating) - -Once tables are gated, any code referencing `schema::tasks` will fail to compile if the feature is off. -(We have already done most of this in Models/Logic, but Schema was the safety net). - -### **3.1 Verify Models imports** -Ensure `models/tasks.rs` uses `crate::schema::tasks` inside the same cfg gate. - -### **3.2 Fix Cross-Feature Joins** -If Core code joins `users` with `tasks` (e.g. "get all user tasks"), that code chunk MUST be gated. - ---- - -## Execution Checklist - -### Schema -- [x] Create `src/core/shared/schema/` directory -- [x] Move core tables to `schema/core.rs` -- [x] Move task tables to `schema/tasks.rs` (gated) -- [x] Move mail tables to `schema/mail.rs` (gated) -- [x] Move people tables to `schema/people.rs` (gated) -- [x] Move tickets tables to `schema/tickets.rs` (gated) -- [x] Move billing tables to `schema/billing.rs` (gated) -- [x] Move attendant tables to `schema/attendant.rs` (gated) -- [x] Move calendar tables to `schema/calendar.rs` (gated) -- [x] Move goals tables to `schema/goals.rs` (gated) -- [x] Move canvas tables to `schema/canvas.rs` (gated) -- [x] Move workspaces tables to `schema/workspaces.rs` (gated) -- [x] Move social tables to `schema/social.rs` (gated) -- [x] Move analytics tables to `schema/analytics.rs` (gated) -- [x] Move compliance tables to `schema/compliance.rs` (gated) -- [x] Move meet tables to `schema/meet.rs` (gated) -- [x] Move research tables to `schema/research.rs` (gated) -- [x] Refactor `schema/mod.rs` to export modules conditionally -- [x] Split `joinable!` declarations into relevant files -- [x] Refactor `allow_tables_to_appear_in_same_query!` (Skipped/Implicit) - -### Migrations -- [x] Audit `migrations/` folder -- [x] Create folder structure `migrations/{core,tasks,mail,people...}` -- [x] Move SQL files to appropriate folders -- [x] Update `run_migrations` in `utils.rs` to run feature-specific migration sets - -### Validation -- [ ] `cargo check --no-default-features` (Run ONLY after all migration splits are verified) -- [ ] `cargo check --features tasks` -- [ ] `cargo check --all-features` diff --git a/src/core/features.rs b/src/core/features.rs new file mode 100644 index 000000000..f5d9b4097 --- /dev/null +++ b/src/core/features.rs @@ -0,0 +1,82 @@ + +/// List of features compiled into this binary +pub const COMPILED_FEATURES: &[&str] = &[ + #[cfg(feature = "chat")] + "chat", + #[cfg(feature = "mail")] + "mail", + #[cfg(feature = "email")] + "email", // Alias for mail + #[cfg(feature = "calendar")] + "calendar", + #[cfg(feature = "drive")] + "drive", + #[cfg(feature = "tasks")] + "tasks", + #[cfg(feature = "docs")] + "docs", + #[cfg(feature = "paper")] + "paper", + #[cfg(feature = "sheet")] + "sheet", + #[cfg(feature = "slides")] + "slides", + #[cfg(feature = "meet")] + "meet", + #[cfg(feature = "research")] + "research", + #[cfg(feature = "people")] + "people", + #[cfg(feature = "social")] + "social", + #[cfg(feature = "analytics")] + "analytics", + #[cfg(feature = "monitoring")] + "monitoring", + #[cfg(feature = "admin")] + "admin", + #[cfg(feature = "automation")] + "automation", + #[cfg(feature = "cache")] + "cache", + #[cfg(feature = "directory")] + "directory", + // Add other app features as they are defined in Cargo.toml + #[cfg(feature = "project")] + "project", + #[cfg(feature = "goals")] + "goals", + #[cfg(feature = "workspace")] + "workspace", + #[cfg(feature = "tickets")] + "tickets", + #[cfg(feature = "billing")] + "billing", + #[cfg(feature = "products")] + "products", + #[cfg(feature = "video")] + "video", + #[cfg(feature = "player")] + "player", + #[cfg(feature = "canvas")] + "canvas", + #[cfg(feature = "learn")] + "learn", + #[cfg(feature = "sources")] + "sources", + #[cfg(feature = "dashboards")] + "dashboards", + #[cfg(feature = "designer")] + "designer", + #[cfg(feature = "editor")] + "editor", + #[cfg(feature = "attendant")] + "attendant", + #[cfg(feature = "tools")] + "tools", +]; + +/// Check if a feature is compiled into the binary +pub fn is_feature_compiled(name: &str) -> bool { + COMPILED_FEATURES.contains(&name) +} diff --git a/src/core/manifest.rs b/src/core/manifest.rs new file mode 100644 index 000000000..af8aa25e7 --- /dev/null +++ b/src/core/manifest.rs @@ -0,0 +1,167 @@ +use serde::{Deserialize, Serialize}; +use crate::core::features::COMPILED_FEATURES; +use crate::core::product::PRODUCT_CONFIG; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct WorkspaceManifest { + pub version: String, + pub features: FeatureManifest, + pub apps: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct FeatureManifest { + pub compiled: Vec, + pub enabled: Vec, + pub disabled: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AppInfo { + pub name: String, + pub enabled: bool, + pub compiled: bool, + pub category: String, +} + +impl WorkspaceManifest { + pub fn new() -> Self { + let compiled: Vec = COMPILED_FEATURES.iter().map(|s| s.to_string()).collect(); + + let enabled: Vec = PRODUCT_CONFIG + .read() + .ok() + .as_ref() + .map(|c| c.get_enabled_apps()) + .unwrap_or_default() + .into_iter() + .filter(|app| compiled.contains(app)) + .collect(); + + let disabled: Vec = compiled + .iter() + .filter(|app| !enabled.contains(app)) + .cloned() + .collect(); + + let apps = Self::build_app_info(&compiled, &enabled); + + Self { + version: env!("CARGO_PKG_VERSION").to_string(), + features: FeatureManifest { + compiled, + enabled, + disabled, + }, + apps, + } + } + + fn build_app_info(compiled: &[String], enabled: &[String]) -> Vec { + let mut apps = Vec::new(); + + // Communication apps + for app in ["chat", "mail", "calendar", "meet", "people", "social"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "communication".to_string(), + }); + } + } + + // Productivity apps + for app in ["drive", "tasks", "docs", "paper", "sheet", "slides"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "productivity".to_string(), + }); + } + } + + // Project management + for app in ["project", "goals", "workspace", "tickets"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "project_management".to_string(), + }); + } + } + + // Business apps + for app in ["billing", "products", "analytics"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "business".to_string(), + }); + } + } + + // Media apps + for app in ["video", "player"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "media".to_string(), + }); + } + } + + // Learning apps + for app in ["canvas", "learn", "research"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "learning".to_string(), + }); + } + } + + // Developer apps + for app in ["sources", "dashboards", "designer", "editor", "tools"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "developer".to_string(), + }); + } + } + + // System apps + for app in ["automation", "cache", "directory", "admin", "monitoring", "attendant"] { + if compiled.contains(&app.to_string()) { + apps.push(AppInfo { + name: app.to_string(), + enabled: enabled.contains(&app.to_string()), + compiled: true, + category: "system".to_string(), + }); + } + } + + apps + } +} + +impl Default for WorkspaceManifest { + fn default() -> Self { + Self::new() + } +} diff --git a/src/core/mod.rs b/src/core/mod.rs index 90b41ea02..bd23bcb3b 100644 --- a/src/core/mod.rs +++ b/src/core/mod.rs @@ -6,9 +6,11 @@ pub mod bot_database; pub mod config; pub mod directory; pub mod dns; +pub mod features; pub mod i18n; pub mod kb; pub mod large_org_optimizer; +pub mod manifest; pub mod middleware; pub mod oauth; pub mod organization; diff --git a/src/core/product.rs b/src/core/product.rs index ff5c71e59..026f3d266 100644 --- a/src/core/product.rs +++ b/src/core/product.rs @@ -293,12 +293,27 @@ pub fn replace_branding(text: &str) -> String { /// Helper function to get product config for serialization pub fn get_product_config_json() -> serde_json::Value { + // Get compiled features from our new module + let compiled = crate::core::features::COMPILED_FEATURES; + + // Get current config let config = PRODUCT_CONFIG.read().ok(); + // Determine effective apps (intersection of enabled + compiled) + let effective_apps: Vec = config + .as_ref() + .map(|c| c.get_enabled_apps()) + .unwrap_or_default() + .into_iter() + .filter(|app| compiled.contains(&app.as_str()) || app == "settings" || app == "auth") // Always allow settings/auth + .collect(); + match config { Some(c) => serde_json::json!({ "name": c.name, - "apps": c.get_enabled_apps(), + "apps": effective_apps, + "compiled_features": compiled, + "version": env!("CARGO_PKG_VERSION"), "theme": c.theme, "logo": c.logo, "favicon": c.favicon, @@ -308,12 +323,21 @@ pub fn get_product_config_json() -> serde_json::Value { }), None => serde_json::json!({ "name": "General Bots", - "apps": [], + "apps": compiled, // If no config, show all compiled + "compiled_features": compiled, + "version": env!("CARGO_PKG_VERSION"), "theme": "sentient", }) } } +/// Get workspace manifest with detailed feature information +pub fn get_workspace_manifest() -> serde_json::Value { + let manifest = crate::core::manifest::WorkspaceManifest::new(); + serde_json::to_value(manifest).unwrap_or_else(|_| serde_json::json!({})) +} + + /// Middleware to check if an app is enabled before allowing API access pub async fn app_gate_middleware( req: axum::http::Request, @@ -361,6 +385,25 @@ pub async fn app_gate_middleware( // Check if the app is enabled if let Some(app) = app_name { + // First check: is it even compiled? + // Note: settings, auth, admin are core features usually, but we check anyway if they are in features list + // Some core apps like settings might not be in feature flags explicitly or always enabled. + // For simplicity, if it's not in compiled features but is a known core route, we might allow it, + // but here we enforce strict feature containment. + // Exception: 'settings' and 'auth' are often core. + if app != "settings" && app != "auth" && !crate::core::features::is_feature_compiled(app) { + let error_response = serde_json::json!({ + "error": "not_implemented", + "message": format!("The '{}' feature is not compiled in this build", app), + "code": 501 + }); + + return ( + StatusCode::NOT_IMPLEMENTED, + axum::Json(error_response) + ).into_response(); + } + if !is_app_enabled(app) { let error_response = serde_json::json!({ "error": "app_disabled", diff --git a/src/main.rs b/src/main.rs index 562703b41..d750ee8ca 100644 --- a/src/main.rs +++ b/src/main.rs @@ -346,6 +346,7 @@ async fn run_axum_server( .add_anonymous_path("/healthz") .add_anonymous_path("/api/health") .add_anonymous_path("/api/product") + .add_anonymous_path("/api/manifest") .add_anonymous_path("/api/i18n") .add_anonymous_path("/api/auth/login") .add_anonymous_path("/api/auth/refresh") @@ -425,10 +426,16 @@ async fn run_axum_server( Json(get_product_config_json()) } + async fn get_workspace_manifest() -> Json { + use crate::core::product::get_workspace_manifest; + Json(get_workspace_manifest()) + } + let mut api_router = Router::new() .route("/health", get(health_check_simple)) .route(ApiUrls::HEALTH, get(health_check)) .route("/api/product", get(get_product_config)) + .route("/api/manifest", get(get_workspace_manifest)) .route(ApiUrls::SESSIONS, post(create_session)) .route(ApiUrls::SESSIONS, get(get_sessions)) .route(ApiUrls::SESSION_HISTORY, get(get_session_history)) diff --git a/src/tasks/PROMPT.md b/src/tasks/PROMPT.md index c8b3706e1..d542fd129 100644 --- a/src/tasks/PROMPT.md +++ b/src/tasks/PROMPT.md @@ -1,6 +1,6 @@ # AutoTask LLM Executor - Prompt Guide -**Version:** 6.1.0 +**Version:** 6.2.0 **Purpose:** Guide LLM to generate and execute automated tasks using BASIC scripts ---