botserver/TASKS.md

Deep Feature Gating Plan: Schema & Migrations

Objective

Reduce binary size and runtime overhead by strictly feature-gating Database Schema (schema.rs) and Migrations. The goal is that a "minimal" build (chat-only) should:

1. NOT compile code for tasks, mail, calendar, etc. tables.
2. NOT run database migrations for those features (preventing unused tables in DB).
3. NOT fail compilation due to missing table definitions in joinable! macros.

---

🚀 Phase 1: Split Diesel Schema (The Hard Part)

The current schema.rs is monolithic (3000+ lines). Diesel requires joinable! and allow_tables_to_appear_in_same_query! macros to know about relationship graphs.

1.1 Analysis & grouping

Identify which tables belong to which feature and strictly define their relationships.

• Core: users, bots, sessions, organizations, rbac_*
• Tasks: tasks, task_comments (links to users, projects)
• Mail: email_* (links to users)
• Drive: (Uses S3, but maybe metadata tables?)
• People/CRM: crm_* (links to users, organizations)

1.2 Create Modular Schema Files

Instead of one file, we will separate them and use mod.rs to aggregate.

• src/core/shared/schema/mod.rs (Aggregator)
• src/core/shared/schema/core.rs (Base tables)
• src/core/shared/schema/tasks.rs (Gated #[cfg(feature="tasks")])
• src/core/shared/schema/mail.rs (Gated #[cfg(feature="mail")]) ...etc.

1.3 Solve the allow_tables_to_appear_in_same_query! Problem

The single massive macro prevents gating. Solution: Break the "Universe" assumption.

1. Define a core_tables! macro.
2. Define task_tables! macro (gated).
3. We might lose the ability to write arbitrary joins between any two tables if they are in different clusters, unless we explicitly define joinable! in the aggregator.
4. Strategy:
   • Keep joinable! definitions relative to the feature.
   • If TableA (Core) joins TableB (Tasks), the joinable! must be gated by tasks.
   • The allow_tables... output must be constructed dynamically or strict subsets defined.

1.4 Comprehensive Feature Mapping

Based on Cargo.toml, we need to map all optional features to their schema requirements.

Feature	Tables (Prefix/Name)	Migrations
(core)	users, bots, sessions, orgs, rbac_*, system_automations, bot_memories, kb_*	migrations/core/
tasks	tasks, task_comments	migrations/tasks/
calendar	calendars, calendar_events, calendar_*	migrations/calendar/
mail	email_*, distribution_lists, global_email_signatures	migrations/mail/
people	crm_contacts, crm_accounts, crm_leads, crm_opportunities, crm_activities, crm_pipeline_*, people_*	migrations/people/
compliance	compliance_*, legal_*, cookie_consents, data_*	migrations/compliance/
meet	meet_*, meeting_*, whiteboard_*	migrations/meet/
attendant	attendant_*	migrations/attendant/
analytics	dashboards, dashboard_*	migrations/analytics/
drive	(No direct tables? Check files or drive_*?)	migrations/drive/
billing	billing_*, products, services, price_*, inventory_* (See note)	migrations/billing/
social	social_*	migrations/social/
canvas	canvases, canvas_*	migrations/canvas/
workspaces	workspaces, workspace_*	migrations/workspaces/
research	research_*	migrations/research/
goals	okr_*	migrations/goals/
feedback	support_tickets, ticket_*	migrations/tickets/

Note: billing and tickets are currently ungated modules in main.rs. We should create features or group them. For now, assume they are part of admin/core or need gating.

---

🚀 Phase 2: Feature-Gated Migrations

Currently, embed_migrations!("migrations") embeds everything.

2.1 Split Migration Directories

Refactor the flat migrations/ folder into feature-specific directories (requires custom runner logic or Diesel configuration trickery). Alternately (easier code-wise):

• Keep flat structure but edit the migration SQL files to be conditional? No, SQL doesn't support #[cfg].
• Better Approach: Use multiple embed_migrations! calls.
  • migrations/core/ -> Always run.
  • migrations/tasks/ -> Run if tasks feature on.
  • migrations/mail/ -> Run if mail feature on.

Action Plan:

1. Organize migrations/ into subdirectories: core/, tasks/, mail/, people/.
2. Update utils.rs migration logic:

   pub fn run_migrations(conn: &mut PgConnection) {
       // Core
       const CORE: EmbeddedMigrations = embed_migrations!("migrations/core");
       conn.run_pending_migrations(CORE).unwrap();
   
       #[cfg(feature = "tasks")] {
           const TASKS: EmbeddedMigrations = embed_migrations!("migrations/tasks");
           conn.run_pending_migrations(TASKS).unwrap();
       }
       // ...
   }
   

2.2 Verify Migration Dependencies

Ensure that tasks migrations (which might foreign-key to users) only run after core migrations are applied.

---

🚀 Phase 3: Fix Dependent Code (Strict Gating)

Once tables are gated, any code referencing schema::tasks will fail to compile if the feature is off. (We have already done most of this in Models/Logic, but Schema was the safety net).

3.1 Verify Models imports

Ensure models/tasks.rs uses crate::schema::tasks inside the same cfg gate.

3.2 Fix Cross-Feature Joins

If Core code joins users with tasks (e.g. "get all user tasks"), that code chunk MUST be gated.

---

Execution Checklist

Schema

• Create src/core/shared/schema/ directory
• Move core tables to schema/core.rs
• Move task tables to schema/tasks.rs (gated)
• Move mail tables to schema/mail.rs (gated)
• Move people tables to schema/people.rs (gated)
• Move tickets tables to schema/tickets.rs (gated)
• Move billing tables to schema/billing.rs (gated)
• Move attendant tables to schema/attendant.rs (gated)
• Move calendar tables to schema/calendar.rs (gated)
• Move goals tables to schema/goals.rs (gated)
• Move canvas tables to schema/canvas.rs (gated)
• Move workspaces tables to schema/workspaces.rs (gated)
• Move social tables to schema/social.rs (gated)
• Move analytics tables to schema/analytics.rs (gated)
• Move compliance tables to schema/compliance.rs (gated)
• Move meet tables to schema/meet.rs (gated)
• Move research tables to schema/research.rs (gated)
• Refactor schema/mod.rs to export modules conditionally
• Split joinable! declarations into relevant files
• Refactor allow_tables_to_appear_in_same_query! (Skipped/Implicit)

Migrations

• Audit migrations/ folder
• Create folder structure migrations/{core,tasks,mail,people...}
• Move SQL files to appropriate folders
• Update run_migrations in utils.rs to run feature-specific migration sets

Validation

• cargo check --no-default-features (Run ONLY after all migration splits are verified)
• cargo check --features tasks
• cargo check --all-features