From e443aa9d1a6ca3fd26971b8ea7bb3f910391ecb3 Mon Sep 17 00:00:00 2001 From: Rodrigo Rodriguez Date: Thu, 19 Feb 2026 11:42:10 +0000 Subject: [PATCH] refactor: Split README.md into human-focused README and agent-focused AGENTS.md --- .env.example | 8 + .gitignore | 5 + AGENTS.md | 312 ++++++++++++++++++++++++ README.md | 654 +-------------------------------------------------- TASKS.md | 453 +++++++++++++++++++++++++++++++++++ 5 files changed, 789 insertions(+), 643 deletions(-) create mode 100644 .env.example create mode 100644 AGENTS.md create mode 100644 TASKS.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..f94caba --- /dev/null +++ b/.env.example @@ -0,0 +1,8 @@ +# General Bots Environment Configuration +# Copy this file to .env and fill in values +# NEVER commit .env to version control + +# Vault connection +VAULT_ADDR=https://127.0.0.1:8200 +VAULT_TOKEN= +VAULT_CACERT=./botserver-stack/vault/certs/ca.crt diff --git a/.gitignore b/.gitignore index 62f6d51..63a8a54 100644 --- a/.gitignore +++ b/.gitignore @@ -55,3 +55,8 @@ config/directory_config.json vault-unseal-keys start-and-unseal.sh vault-token-* +init.json +*.pem +*.key +*.crt +*.cert diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5723d2b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,312 @@ +# General Bots AI Agent Guidelines + +> **⚠️ CRITICAL SECURITY WARNING** +> +> **NEVER CREATE FILES WITH SECRETS IN THE REPOSITORY ROOT** +> +> Secret files MUST be placed in `/tmp/` only: +> - ✅ `/tmp/vault-token-gb` - Vault root token +> - ✅ `/tmp/vault-unseal-key-gb` - Vault unseal key +> - ❌ `vault-unseal-keys` - FORBIDDEN (tracked by git) +> - ❌ `start-and-unseal.sh` - FORBIDDEN (contains secrets) +> +> **Why `/tmp/`?** +> - Cleared on reboot (ephemeral) +> - Not tracked by git +> - Standard Unix security practice +> - Prevents accidental commits + +--- + +## 🧭 LLM Navigation Guide + +### Reading This Workspace + +**For LLMs analyzing this codebase:** +1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)** in README to understand relationships +2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)** for what each module does +3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)** to understand execution flow +4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)** before making changes +5. Check **[Security Rules](#-security-directives---mandatory)** below - violations are blocking issues +6. Follow **[Code Patterns](#-mandatory-code-patterns)** below - consistency is mandatory + +--- + +## 🔐 Security Directives - MANDATORY + +### 1. Error Handling - NO PANICS IN PRODUCTION + +```rust +// ❌ FORBIDDEN +value.unwrap() +value.expect("message") +panic!("error") +todo!() +unimplemented!() + +// ✅ REQUIRED +value? +value.ok_or_else(|| Error::NotFound)? +value.unwrap_or_default() +value.unwrap_or_else(|e| { log::error!("{}", e); default }) +if let Some(v) = value { ... } +match value { Ok(v) => v, Err(e) => return Err(e.into()) } +``` + +### 2. Command Execution - USE SafeCommand + +```rust +// ❌ FORBIDDEN +Command::new("some_command").arg(user_input).output() + +// ✅ REQUIRED +use crate::security::command_guard::SafeCommand; +SafeCommand::new("allowed_command")? + .arg("safe_arg")? + .execute() +``` + +### 3. Error Responses - USE ErrorSanitizer + +```rust +// ❌ FORBIDDEN +Json(json!({ "error": e.to_string() })) +format!("Database error: {}", e) + +// ✅ REQUIRED +use crate::security::error_sanitizer::log_and_sanitize; +let sanitized = log_and_sanitize(&e, "context", None); +(StatusCode::INTERNAL_SERVER_ERROR, sanitized) +``` + +### 4. SQL - USE sql_guard + +```rust +// ❌ FORBIDDEN +format!("SELECT * FROM {}", user_table) + +// ✅ REQUIRED +use crate::security::sql_guard::{sanitize_identifier, validate_table_name}; +let safe_table = sanitize_identifier(&user_table); +validate_table_name(&safe_table)?; +``` + +--- + +## ✅ Mandatory Code Patterns + +### Use Self in Impl Blocks +```rust +impl MyStruct { + fn new() -> Self { Self { } } // ✅ Not MyStruct +} +``` + +### Derive Eq with PartialEq +```rust +#[derive(PartialEq, Eq)] // ✅ Always both +struct MyStruct { } +``` + +### Inline Format Args +```rust +format!("Hello {name}") // ✅ Not format!("{}", name) +``` + +### Combine Match Arms +```rust +match x { + A | B => do_thing(), // ✅ Combine identical arms + C => other(), +} +``` + +--- + +## ❌ Absolute Prohibitions + +- ❌ **NEVER** use `.unwrap()` or `.expect()` in production code (tests OK) +- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()` +- ❌ **NEVER** use `Command::new()` directly - use `SafeCommand` +- ❌ **NEVER** return raw error strings to HTTP clients +- ❌ **NEVER** use `#[allow()]` in source code - FIX the code instead +- ❌ **NEVER** add lint exceptions to `Cargo.toml` - FIX the code instead +- ❌ **NEVER** use `_` prefix for unused variables - DELETE or USE them +- ❌ **NEVER** leave unused imports or dead code +- ❌ **NEVER** use CDN links - all assets must be local +- ❌ **NEVER** use `cargo clean` - causes 30min rebuilds, use `./reset.sh` for database issues +- ❌ **NEVER** create `.md` documentation files without checking `botbook/` first + +--- + +## 📏 File Size Limits - MANDATORY + +### Maximum 450 Lines Per File + +When a file grows beyond this limit: + +1. **Identify logical groups** - Find related functions +2. **Create subdirectory module** - e.g., `handlers/` +3. **Split by responsibility:** + - `types.rs` - Structs, enums, type definitions + - `handlers.rs` - HTTP handlers and routes + - `operations.rs` - Core business logic + - `utils.rs` - Helper functions + - `mod.rs` - Re-exports and configuration +4. **Keep files focused** - Single responsibility +5. **Update mod.rs** - Re-export all public items + +**NEVER let a single file exceed 450 lines - split proactively at 350 lines** + +--- + +## 🔥 Error Fixing Workflow + +### Mode 1: OFFLINE Batch Fix (PREFERRED) + +When given error output: + +1. **Read ENTIRE error list first** +2. **Group errors by file** +3. **For EACH file with errors:** + a. View file → understand context + b. Fix ALL errors in that file + c. Write once with all fixes +4. **Move to next file** +5. **REPEAT until ALL errors addressed** +6. **ONLY THEN → verify with build/diagnostics** + +**NEVER run cargo build/check/clippy DURING fixing** +**Fix ALL errors OFFLINE first, verify ONCE at the end** + +### Mode 2: Interactive Loop + +``` +LOOP UNTIL (0 warnings AND 0 errors): + 1. Run diagnostics → pick file with issues + 2. Read entire file + 3. Fix ALL issues in that file + 4. Write file once with all fixes + 5. Verify with diagnostics + 6. CONTINUE LOOP +END LOOP +``` + +--- + +## 🎭 Playwright Browser Testing - YOLO Mode + +**When user requests to start YOLO mode with Playwright:** + +1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000 +2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state +3. **Test user flows** - Use click, type, fill_form, etc. +4. **Verify results** - Check for expected content, errors in console, network requests +5. **Validate backend** - Check database and services to confirm process completion +6. **Report findings** - Always include screenshot evidence with `browser_take_screenshot` + +**Bot-Specific Testing URL Pattern:** +`http://localhost:3000/` + +**Backend Validation Checks:** +After UI interactions, validate backend state via `psql` or `tail` logs. + +--- + +## 🧪 Testing Strategy + +### Unit Tests +- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules +- **Naming**: Test functions use `test_` prefix or describe what they test +- **Running**: `cargo test -p ` or `cargo test` for all + +### Integration Tests +- **Location**: `bottest/` crate contains integration tests +- **Scope**: Tests full workflows across multiple crates +- **Running**: `cargo test -p bottest` + +### Coverage Goals +- **Critical paths**: 80%+ coverage required +- **Error handling**: ALL error paths must have tests +- **Security**: All security guards must have tests + +--- + +## 🐛 Debugging Rules + +### 🚨 CRITICAL ERROR HANDLING RULE + +**STOP EVERYTHING WHEN ERRORS APPEAR** + +When ANY error appears in logs during startup or operation: +1. **IMMEDIATELY STOP** - Do not continue with other tasks +2. **IDENTIFY THE ERROR** - Read the full error message and context +3. **FIX THE ERROR** - Address the root cause, not symptoms +4. **VERIFY THE FIX** - Ensure error is completely resolved +5. **ONLY THEN CONTINUE** - Never ignore or work around errors + +**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM** + +### Log Locations + +| Component | Log File | What's Logged | +|-----------|----------|---------------| +| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** | +| **botui** | `botui.log` | UI rendering, WebSocket connections | +| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation | +| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events | + +--- + +## 🎨 Frontend Standards + +### HTMX-First Approach +- Use HTMX to minimize JavaScript +- Server returns HTML fragments, not JSON +- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap` +- WebSocket via htmx-ws extension + +### Local Assets Only - NO CDN +```html + + + + + +``` + +--- + +## 🚀 Performance & Size Standards + +### Binary Size Optimization +- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`. +- **Dependencies**: + - Run `cargo tree --duplicates` weekly + - Run `cargo machete` to remove unused dependencies + - Use `default-features = false` and explicitly opt-in to needed features + +### Linting & Code Quality +- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**. +- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented. + +--- + +## 🔑 Memory & Main Directives + +**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION** + +- 0 warnings +- 0 errors +- Trust project diagnostics +- Respect all rules +- No `#[allow()]` in source code +- Real code fixes only + +**Remember:** +- **OFFLINE FIRST** - Fix all errors from list before compiling +- **BATCH BY FILE** - Fix ALL errors in a file at once +- **WRITE ONCE** - Single edit per file with all fixes +- **VERIFY LAST** - Only compile/diagnostics after ALL fixes +- **DELETE DEAD CODE** - Don't keep unused code around +- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo) diff --git a/README.md b/README.md index 776a578..bc8d4f8 100644 --- a/README.md +++ b/README.md @@ -490,340 +490,11 @@ cargo test -p bottest --- -## 🧭 LLM Navigation Guide +## 🤖 AI Agent Guidelines -### Quick Context Jump -- [Primary Purpose](#overview) - Unified workspace for AI automation platform -- [Crate Structure](#-workspace-structure) - 9 independent crates with shared libraries -- [Dependencies](#-component-dependency-graph) - How crates depend on each other -- [Quick Start](#quick-start) - Get running in 2 commands -- [Error Patterns](#common-error-patterns) - Fix compilation errors efficiently -- [Security Rules](#-security-directives---mandatory) - MUST-FOLLOW security patterns -- [Code Patterns](#-mandatory-code-patterns) - Required coding conventions -- [Testing](#testing-strategy) - How to test changes -- [Debugging](#debugging-guide) - Troubleshoot common issues +> **For LLM instructions, coding rules, security directives, testing workflows, and error handling patterns, see [AGENTS.md](./AGENTS.md).** -### Reading This Workspace - -**For LLMs analyzing this codebase:** -1. Start with [Component Dependency Graph](#-component-dependency-graph) to understand relationships -2. Review [Module Responsibility Matrix](#-module-responsibility-matrix) for what each module does -3. Study [Data Flow Patterns](#-data-flow-patterns) to understand execution flow -4. Reference [Common Architectural Patterns](#-common-architectural-patterns) before making changes -5. Check [Security Rules](#-security-directives---mandatory) - violations are blocking issues -6. Follow [Code Patterns](#-mandatory-code-patterns) - consistency is mandatory - -**For Humans working on this codebase:** -1. Follow [Error Fixing Workflow](#-error-fixing-workflow) for compilation errors -2. Observe [File Size Limits](#-file-size-limits---mandatory) - max 450 lines per file -3. Run [Weekly Maintenance Tasks](#-weekly-maintenance-tasks) to keep codebase healthy -4. Read project-specific READMEs in [Project-Specific Guidelines](#-project-specific-guidelines) - -## 🎭 Playwright Browser Testing - YOLO Mode - -### YOLO Mode Instructions for LLMs - -**When user requests to start YOLO mode with Playwright:** - -1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000 -2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state -3. **Test user flows** - Use click, type, fill_form, etc. to interact with UI -4. **Verify results** - Check for expected content, errors in console, network requests -5. **Validate backend** - Check database and services to confirm process completion -6. **Report findings** - Always include screenshot evidence with `browser_take_screenshot` - -**Available Playwright MCP Tools:** -- `browser_navigate` - Navigate to URL -- `browser_snapshot` - Get accessibility tree (better than screenshots for analysis) -- `browser_take_screenshot` - Capture visual state -- `browser_click` - Click elements (provide ref from snapshot) -- `browser_type` - Type text into inputs -- `browser_fill_form` - Fill multiple form fields at once -- `browser_console_messages` - Check for JavaScript errors -- `browser_network_requests` - Inspect API calls -- `browser_close` - Close browser when done - -**Bot-Specific Testing URL Pattern:** -``` -http://localhost:3000/ -``` -Navigate directly to a specific bot's interface to test that bot's functionality. - -**YOLO Testing Workflow:** -``` -1. Navigate → http://localhost:3000/ or http://localhost:3000 -2. Snapshot → Analyze page structure -3. Click → Target element using ref from snapshot -4. Wait → For navigation/updates (browser_wait_for) -5. Verify → Console messages, network status -6. Validate Backend → Check database tables and service logs -7. Screenshot → Document test results -``` - -**Backend Validation Checks:** -After UI interactions, validate backend state: - -```bash -# Check database for created/updated records -psql -h localhost -p 5432 -U postgres -d botdb -c "SELECT * FROM bots WHERE name = '';" -psql -h localhost -p 5432 -U postgres -d botdb -c "SELECT * FROM sessions WHERE bot_id = '' ORDER BY created_at DESC LIMIT 5;" - -# Check service logs for errors -tail -50 botserver-stack/logs/tables/postgres.log -tail -50 botserver-stack/logs/vault/vault.log -tail -50 botserver-stack/logs/cache/valkey.log - -# Verify vector embeddings (for knowledge bases) -curl -s http://localhost:6333/collections/website_/points/scroll | jq '.result.points | length' - -# Check message queue/worker status -redis-cli -p 6379 PING -redis-cli -p 6379 KEYS "session:*" -``` - -**Testing Checklist:** -- ✅ UI loads without errors -- ✅ Navigation works between sections -- ✅ Forms submit correctly -- ✅ WebSocket connections establish -- ✅ Console shows no JavaScript errors -- ✅ Network requests return 200/201/204 -- ✅ Database records created/updated correctly -- ✅ Service logs show no errors -- ✅ Vector embeddings populated (if applicable) -- ✅ Cache/queue operations succeed - -**Critical Test Flows:** -- **Login/Authentication** → Navigate, enter credentials, verify session in database -- **Bot Creation** → Navigate to `/botname`, click "New Bot", fill form, verify creation in `bots` table -- **Chat Interface** → Send message to `/botname`, verify WebSocket response, check `messages` table -- **File Upload** → Upload .bas file to `/botname`, verify compilation, check `botserver.log` for compile results -- **Drive Sync** → Trigger sync from `/botname`, verify files appear, check `drive_monitor` logs -- **Knowledge Base** → Add document via `/botname`, verify embeddings in Qdrant, check `website_embeddings` table -- **Tool Execution** → Trigger tool from chat, verify execution in logs, check side effects in database - -**End-to-End Process Validation:** -1. **Frontend:** User action via Playwright (click, type, submit) -2. **Network:** Capture API request via `browser_network_requests` -3. **Backend:** Check botserver.log for request processing -4. **Database:** Query relevant tables for state changes -5. **Services:** Verify component logs (PostgreSQL, Vault, Qdrant, etc.) -6. **Response:** Verify UI update via Playwright snapshot - -**Error Handling in YOLO Mode:** -- Only DEBUG builds, no RELEASE. -- Query DB as gbuser or get credentials from Vault to connect to DB and check results. -- Neve stops, do the task until the end, even if in doubt. -- If navigation fails: Check if servers running (`ps aux | grep botserver`) -- If element not found: Take snapshot to debug current page state -- If console errors: Extract and report to user for fixing -- If network failures: Check API endpoints and CORS configuration -- If database mismatch: Check database logs, verify transaction committed -- If service errors: Check component logs in `botserver-stack/logs/` - -### Integration Testing - -For automated test suites, prefer `cargo test -p bottest` for backend logic and Playwright YOLO mode for full-stack UI testing. - -## 🧪 Testing Strategy - -### Unit Tests -- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules -- **Naming**: Test functions use `test_` prefix or describe what they test -- **Running**: `cargo test -p ` or `cargo test` for all - -### Integration Tests -- **Location**: `bottest/` crate contains integration tests -- **Scope**: Tests full workflows across multiple crates -- **Running**: `cargo test -p bottest` -- **Database**: Uses test database, automatically migrates on first run - -### Test Utilities Available -- **TestAppStateBuilder** (`bottest/src/harness.rs`) - Build test state with mocked components -- **TestBot** (`bottest/src/bot/mod.rs`) - Mock bot for testing -- **Test Database**: Auto-created, migrations run automatically - -### Coverage Goals -- **Critical paths**: 80%+ coverage required -- **Error handling**: ALL error paths must have tests -- **Security**: All security guards must have tests - -## 🚨 CRITICAL ERROR HANDLING RULE - -**STOP EVERYTHING WHEN ERRORS APPEAR** - -When ANY error appears in logs during startup or operation: -1. **IMMEDIATELY STOP** - Do not continue with other tasks -2. **IDENTIFY THE ERROR** - Read the full error message and context -3. **FIX THE ERROR** - Address the root cause, not symptoms -4. **VERIFY THE FIX** - Ensure error is completely resolved -5. **ONLY THEN CONTINUE** - Never ignore or work around errors - -**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM** - -Examples of errors that MUST be fixed immediately: -- Database connection errors -- Component initialization failures -- Service startup errors -- Configuration errors -- Any error containing "Error:", "Failed:", "Cannot", "Unable" - -## 🐛 Debugging Guide - -### Log Locations - -| Component | Log File | What's Logged | -|-----------|----------|---------------| -| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** | -| **botui** | `botui.log` | UI rendering, WebSocket connections | -| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation | -| **script execution** | In botserver logs with `[ScriptService]` prefix | BASIC compilation, runtime errors | -| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events | - -### Client-Side Logging - -**Navigation Tracking:** All client-side navigation is logged to botserver.log with `CLIENT:` prefix: -``` -CLIENT:NAVIGATION: click: home -> drive -CLIENT:NAVIGATION: hashchange: drive -> chat -``` - -**Error Reporting:** JavaScript errors automatically appear in server logs: -``` -CLIENT:ERROR: Uncaught TypeError: Cannot read property 'x' of undefined at /suite/js/app.js:123 -``` - -**For LLM Troubleshooting:** ALWAYS check both: -1. `botserver.log` - Server errors + client navigation/errors (prefixed with `CLIENT:`) -2. `botui.log` - UI server logs - -### USE WEBSITE Feature - Vector DB Context Injection - -**FIXED (v6.2.0+):** The `USE WEBSITE` BASIC command now properly injects vector database embeddings into chat context. - -**How it works:** -1. **Preprocessing:** When a `.bas` file containing `USE WEBSITE "https://..."` is compiled, the website is registered for crawling -2. **Crawling:** Content is extracted, chunked, and embedded into Qdrant vector DB (collection name: `website_`) -3. **Runtime Association:** The compiled `.ast` file contains `USE_WEBSITE()` function call that creates session-website association -4. **Context Injection:** During chat, `inject_kb_context()` searches active websites' embeddings and includes relevant chunks in LLM prompt - -**Example BASIC script:** -```basic -USE WEBSITE "https://docs.pragmatismo.com.br" REFRESH "1h" - -TALK "Hello! I can now answer questions about the documentation." -``` - -**Database tables involved:** -- `session_website_associations` - Links sessions to websites -- `website_embeddings` - Stores crawled content vectors in Qdrant - -**Verification:** -```sql --- Check if website is associated with session -SELECT * FROM session_website_associations WHERE session_id = ''; - --- Check if embeddings exist in Qdrant (via HTTP API) -curl http://localhost:6333/collections/website_/points/scroll -``` - -**Previous Issue:** In earlier versions, `USE WEBSITE` was removed during preprocessing and never executed at runtime, preventing context injection. Now the function call is preserved in the compiled AST. - -### Common Error Messages - -| Error | Meaning | Fix | -|-------|---------|-----| -| `Session not found` | Invalid session_id in request | Check auth flow, verify session exists in DB | -| `Bot not found` | Invalid bot_name or bot_id | Verify bot exists in `bots` table | -| `Script compilation error` | BASIC syntax error in .bas file | Check .bas file syntax, look for typos | -| `Failed to send TALK message` | WebSocket disconnected | Check client connection, verify web_adapter running | -| `Drive sync failed` | S3 connection or permission issue | Verify S3 credentials, check bucket exists | -| `unwrap() called on None value` | Panic in production code | MUST FIX - replace with proper error handling | -| `Component not responding: ` | Infrastructure component not accessible | Check component status: `ps aux | grep `. View logs: `tail -f botserver-stack/logs//`. Restart via ./restart.sh | -| `Config key not found: ` | Missing configuration in database | Check `bot_configuration` table. Set correct value via API or direct SQL update. | -| `403 Forbidden on POST /api/client-errors` | RBAC blocking client error reporting | FIXED in v6.2.0+ - endpoint now allows anonymous access | - -### Useful Debugging Commands - -```bash -# Check if botserver is running -ps aux | grep botserver - -# View botserver logs in real-time -tail -f botserver/logs/botserver.log - -# Check work directory structure -ls -la ./work/*.gbai/*/ - -# Test database connection -cd botserver && cargo run --bin botserver -- --test-db - -# Run specific test with output -cargo test -p botserver test_name -- --nocapture - -# Check for memory leaks during compilation -CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | grep -i error -``` - -### Troubleshooting Workflows - -**Problem: Script not executing** -1. Check if .bas file exists in `./work/{bot_name}.gbai/{bot_name}.gbdialog/` -2. Verify file has correct syntax (compile with ScriptService) -3. Check logs for compilation errors -4. Verify drive_monitor is running and syncing files - -**Problem: WebSocket messages not received** -1. Check browser console for WebSocket errors -2. Verify session_id is valid in database -3. Check web_adapter is registered for session -4. Look for TALK execution in botserver logs - -**Problem: Component not starting or crashing** -1. Identify the component from error message (e.g., Vault, PostgreSQL, MinIO, Qdrant, Valkey) -2. Check if process is running: `ps aux | grep ` -3. Check component logs: `tail -f botserver-stack/logs//` -4. Common fixes: - - Config error: Check `botserver-stack/conf//` for valid configuration - - Port conflict: Ensure no other process using the component's port - - Permission error: Check file permissions in `botserver-stack/data//` - - Missing binary: Re-run `./reset.sh && ./restart.sh` to reinstall components -5. Restart: `./restart.sh` - -**Problem: Component configuration errors** -1. All component configs stored in database `bot_configuration` table -2. Check current value: `SELECT * FROM bot_configuration WHERE config_key = '';` -3. Update incorrect config: `UPDATE bot_configuration SET config_value = '' WHERE config_key = '';` -4. For path configs: Ensure paths are relative to component binary or absolute -5. Restart botserver after config changes - -**Problem: File not found errors** -1. Check if file exists in expected location -2. Verify config paths use correct format (relative/absolute) -3. Check file permissions: `ls -la ` -4. For model/data files: Ensure downloaded to `botserver-stack/data//` - -**Problem: LLM not responding** -1. Check LLM API credentials in config -2. Verify API key has available quota -3. Check network connectivity to LLM provider -4. Review request/response logs for API errors - -### Performance Profiling - -```bash -# Profile compilation time -cargo build --release --timings - -# Profile runtime performance -cargo flamegraph --bin botserver - -# Check binary size -ls -lh target/release/botserver - -# Memory usage -valgrind --leak-check=full target/release/botserver -``` +--- ## 📖 Glossary @@ -843,284 +514,7 @@ valgrind --leak-check=full target/release/botserver --- -## 🔥 Error Fixing Workflow -### Mode 1: OFFLINE Batch Fix (PREFERRED) - -When given error output: - -``` -1. Read ENTIRE error list first -2. Group errors by file -3. For EACH file with errors: - a. View file → understand context - b. Fix ALL errors in that file - c. Write once with all fixes -4. Move to next file -5. REPEAT until ALL errors addressed -6. ONLY THEN → verify with build/diagnostics -``` - -**NEVER run cargo build/check/clippy DURING fixing** -**Fix ALL errors OFFLINE first, verify ONCE at the end** - -### Mode 2: Interactive Loop - -``` -LOOP UNTIL (0 warnings AND 0 errors): - 1. Run diagnostics → pick file with issues - 2. Read entire file - 3. Fix ALL issues in that file - 4. Write file once with all fixes - 5. Verify with diagnostics - 6. CONTINUE LOOP -END LOOP -``` - -### Common Error Patterns - -| Error | Fix | -|-------|-----| -| `expected i64, found u64` | `value as i64` | -| `expected Option, found T` | `Some(value)` | -| `expected T, found Option` | `value.unwrap_or(default)` | -| `cannot multiply f32 by f64` | `f64::from(f32_val) * f64_val` | -| `no field X on type Y` | Check struct definition | -| `no variant X found` | Check enum definition | -| `function takes N arguments` | Match function signature | -| `cannot find function` | Add missing function or fix import | -| `unused variable` | Delete or use with `..` in patterns | -| `unused import` | Delete the import line | -| `cannot move out of X because borrowed` | Use scoping `{ }` to limit borrow | - ---- - -## 🧠 Memory Management - -When compilation fails due to memory issues (process "Killed"): - -```bash -pkill -9 cargo; pkill -9 rustc; pkill -9 botserver -CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200 -``` - ---- - -## 📏 File Size Limits - MANDATORY - -### Maximum 450 Lines Per File - -When a file grows beyond this limit: - -1. **Identify logical groups** - Find related functions -2. **Create subdirectory module** - e.g., `handlers/` -3. **Split by responsibility:** - - `types.rs` - Structs, enums, type definitions - - `handlers.rs` - HTTP handlers and routes - - `operations.rs` - Core business logic - - `utils.rs` - Helper functions - - `mod.rs` - Re-exports and configuration -4. **Keep files focused** - Single responsibility -5. **Update mod.rs** - Re-export all public items - -**NEVER let a single file exceed 450 lines - split proactively at 350 lines** - -### Current Files Requiring Immediate Refactoring - -| File | Lines | Target Split | -|------|-------|--------------| -| `botserver/src/drive/mod.rs` | 1522 | → 4 files | -| `botserver/src/auto_task/app_generator.rs` | 2981 | → 7 files | -| `botui/ui/suite/sheet/sheet.js` | 3220 | → 8 files | -| `botserver/src/tasks/mod.rs` | 2651 | → 6 files | -| `botserver/src/learn/mod.rs` | 2306 | → 5 files | - -See `TODO-refactor1.md` for detailed refactoring plans. - ---- - -## 🔍 Continuous Monitoring - -**YOLO Forever Monitoring Pattern:** - -The system includes automated log monitoring to catch errors in real-time: - -```bash -# Continuous monitoring (check every 5 seconds) -while true; do - sleep 5 - echo "=== Check at $(date +%H:%M:%S) ===" - tail -50 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -5 || echo "✓ Clean" -done -``` - -**Quick Status Check:** -```bash -# Check last 200 lines for any issues -tail -200 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -10 - -# Show recent server activity -tail -30 botserver.log - -# Check if server is running -ps aux | grep botserver | grep -v grep -``` - -**Monitoring Dashboard:** -- **Server Status**: https://localhost:9000 (health endpoint) -- **Logs**: `tail -f botserver.log` -- **Client Errors**: Look for `CLIENT:` prefix -- **Server Errors**: Look for `ERROR` or `WARN` prefixes - -**Status Indicators:** -- ✅ **Clean**: No ERROR/WARN/CLIENT: entries in logs -- ⚠️ **Warnings**: Non-critical issues that should be reviewed -- ❌ **Errors**: Critical issues requiring immediate attention - -**When Errors Appear:** -1. Capture the full error context (50 lines before/after) -2. Identify the component (server, client, database, etc.) -3. Check troubleshooting section for specific fixes -4. Update this README with discovered issues and resolutions - ---- - -## 🚀 Performance & Size Standards - -### Binary Size Optimization -- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`. -- **Dependencies**: - - Run `cargo tree --duplicates` weekly to find and resolve duplicate versions. - - Run `cargo machete` to remove unused dependencies. - - Use `default-features = false` and explicitly opt-in to needed features. - -### Memory Optimization -- **Strings**: Prefer `&str` over `String` where possible. Use `Cow` for conditional ownership. -- **Collections**: Use `Vec::with_capacity` when size is known. Consider `SmallVec` for hot paths. -- **Allocations**: Minimize heap allocations in hot paths. -- **Cloning**: Avoid unnecessary `.clone()` calls. Use references or `Cow` types. - -### Code Quality Issues Found -- **955 instances** of `unwrap()`/`expect()` in codebase - ALL must be replaced with proper error handling -- **12,973 instances** of excessive `clone()`/`to_string()` calls - optimize for performance -- **Test code exceptions**: `unwrap()` allowed in test files only - -### Linting & Code Quality -- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**. -- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented. Fix the underlying issue. - ---- - -## 🔐 Security Directives - MANDATORY - -### Error Handling - NO PANICS IN PRODUCTION - -```rust -// ❌ FORBIDDEN -value.unwrap() -value.expect("message") -panic!("error") -todo!() -unimplemented!() - -// ✅ REQUIRED -value? -value.ok_or_else(|| Error::NotFound)? -value.unwrap_or_default() -value.unwrap_or_else(|e| { log::error!("{}", e); default }) -if let Some(v) = value { ... } -match value { Ok(v) => v, Err(e) => return Err(e.into()) } -``` - -### Command Execution - USE SafeCommand - -```rust -// ❌ FORBIDDEN -Command::new("some_command").arg(user_input).output() - -// ✅ REQUIRED -use crate::security::command_guard::SafeCommand; -SafeCommand::new("allowed_command")? - .arg("safe_arg")? - .execute() -``` - -### Error Responses - USE ErrorSanitizer - -```rust -// ❌ FORBIDDEN -Json(json!({ "error": e.to_string() })) -format!("Database error: {}", e) - -// ✅ REQUIRED -use crate::security::error_sanitizer::log_and_sanitize; -let sanitized = log_and_sanitize(&e, "context", None); -(StatusCode::INTERNAL_SERVER_ERROR, sanitized) -``` - -### SQL - USE sql_guard - -```rust -// ❌ FORBIDDEN -format!("SELECT * FROM {}", user_table) - -// ✅ REQUIRED -use crate::security::sql_guard::{sanitize_identifier, validate_table_name}; -let safe_table = sanitize_identifier(&user_table); -validate_table_name(&safe_table)?; -``` - ---- - -## ❌ Absolute Prohibitions - -``` -❌ NEVER use .unwrap() or .expect() in production code (tests OK) -❌ NEVER use panic!(), todo!(), unimplemented!() -❌ NEVER use Command::new() directly - use SafeCommand -❌ NEVER return raw error strings to HTTP clients -❌ NEVER use #[allow()] in source code - FIX the code instead -❌ NEVER add lint exceptions to Cargo.toml - FIX the code instead -❌ NEVER use _ prefix for unused variables - DELETE or USE them -❌ NEVER leave unused imports or dead code -❌ NEVER add comments - code must be self-documenting -❌ NEVER modify Cargo.toml lints section! -❌ NEVER use CDN links - all assets must be local -❌ NEVER use cargo clean - causes 30min rebuilds, use ./reset.sh for database issues -❌ NEVER create .md documentation files without checking botbook/ first - documentation belongs there -``` - ---- - -## ✅ Mandatory Code Patterns - -### Use Self in Impl Blocks -```rust -impl MyStruct { - fn new() -> Self { Self { } } // ✅ Not MyStruct -} -``` - -### Derive Eq with PartialEq -```rust -#[derive(PartialEq, Eq)] // ✅ Always both -struct MyStruct { } -``` - -### Inline Format Args -```rust -format!("Hello {name}") // ✅ Not format!("{}", name) -``` - -### Combine Match Arms -```rust -match x { - A | B => do_thing(), // ✅ Combine identical arms - C => other(), -} -``` - ---- ## 🖥️ UI Architecture (botui + botserver) @@ -1290,44 +684,18 @@ Both repositories must be pushed for changes to take effect in production. ## Development Workflow -1. Read this README.md (workspace-level rules) -2. **BEFORE creating any .md file, search botbook/ for existing documentation** -3. Read `/README.md` (project-specific rules) -4. Use diagnostics tool to check warnings -5. Fix all warnings with full file rewrites -6. Verify with diagnostics after each file -7. Never suppress warnings with `#[allow()]` +1. Read this README.md (workspace structure) +2. Read **[AGENTS.md](./AGENTS.md)** (coding rules & workflows) +3. **BEFORE creating any .md file, search botbook/ for existing documentation** +4. Read `/README.md` (project-specific rules) +5. Use diagnostics tool to check warnings +6. Fix all warnings with full file rewrites +7. Verify with diagnostics after each file +8. Never suppress warnings with `#[allow()]` --- -## Main Directive -**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION** - -- 0 warnings -- 0 errors -- Trust project diagnostics -- Respect all rules -- No `#[allow()]` in source code -- Real code fixes only - ---- - -## 🔑 Remember - -- **OFFLINE FIRST** - Fix all errors from list before compiling -- **ZERO WARNINGS, ZERO ERRORS** - The only acceptable state -- **FIX, DON'T SUPPRESS** - No #[allow()], no Cargo.toml lint exceptions -- **SECURITY FIRST** - No unwrap, no raw errors, no direct commands -- **READ BEFORE FIX** - Always understand context first -- **BATCH BY FILE** - Fix ALL errors in a file at once -- **WRITE ONCE** - Single edit per file with all fixes -- **VERIFY LAST** - Only compile/diagnostics after ALL fixes -- **DELETE DEAD CODE** - Don't keep unused code around -- **Version 6.2.0** - Do not change without approval -- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo) - ---- ## License diff --git a/TASKS.md b/TASKS.md new file mode 100644 index 0000000..a99c32b --- /dev/null +++ b/TASKS.md @@ -0,0 +1,453 @@ +# TASKS.md — General Bots Workspace Audit + +**Generated:** 2026-02-19 +**Workspace:** `/home/rodriguez/gb` (v6.2.0) +**Scope:** Full workspace security audit, code quality analysis, and improvement backlog + +--- + +## 🔴 P0 — CRITICAL SECURITY FLAWS (Fix Immediately) + +### SEC-01: ✅ RESOLVED — `vault-unseal-keys` removed from Git tracking + +**Severity:** 🔴 CRITICAL +**File:** `vault-unseal-keys` +**Status:** ✅ Removed from Git tracking. **History purge and key rotation still required.** + +The file contained **5 plaintext Vault unseal keys** and had **2 commits** in the git history. It has been removed from tracking via `git rm --cached`. + +**Completed:** +- [x] `git rm --cached vault-unseal-keys` — Removed from tracking +- [x] Added to `.gitignore` (was already present) + +**Remaining (manual action required):** +- [ ] **Rotate ALL 5 Vault unseal keys immediately** +- [ ] Use `git filter-repo` or BFG Repo-Cleaner to purge from history +- [ ] Force-push to ALL remotes (`origin`, `alm`) +- [ ] Notify all collaborators to re-clone + +--- + +### SEC-02: ✅ PARTIALLY RESOLVED — `.env` exposure mitigated + +**Severity:** 🔴 CRITICAL +**Files:** `.env` (root), `botserver/.env` + +**Completed:** +- [x] Verified `botserver/.env` is NOT tracked by git +- [x] Root `.env` confirmed NOT tracked (properly `.gitignore`'d) +- [x] Created `.env.example` template with placeholder values +- [x] Added `*.pem`, `*.key`, `*.crt`, `*.cert` to `.gitignore` + +**Remaining (manual action required):** +- [ ] **Rotate both Vault tokens immediately** +- [ ] Implement short-TTL Vault tokens (e.g., 1h) with auto-renewal +- [ ] Consider using Vault Agent for automatic token management + +--- + +### SEC-03: ✅ RESOLVED — `init.json` removed from Git tracking + +**Severity:** 🟠 HIGH +**File:** `init.json` + +**Completed:** +- [x] `git rm --cached init.json` — Removed from tracking +- [x] Added `init.json` to `.gitignore` + +--- + +### SEC-04: ✅ RESOLVED — All `Command::new()` replaced with `SafeCommand` + +**Severity:** 🟠 HIGH +**File:** `botserver/src/security/protection/installer.rs` + +**Completed:** +- [x] Replaced all 8 `Command::new()` calls with `SafeCommand::new()` (including verify() Windows path) +- [x] Added `id` and `netsh` to SafeCommand whitelist in `command_guard.rs` +- [x] Removed unused `use std::process::Command;` import +- [x] Fixed 3 duplicate `#[cfg(not(windows))]` attributes +- [x] Build verified — compiles cleanly + +--- + +### SEC-05: ✅ RESOLVED — SQL injection vectors fixed with parameterized queries + +**Severity:** 🟠 HIGH +**Files fixed:** +- `botserver/src/basic/keywords/db_api.rs` +- `botserver/src/security/sql_guard.rs` (already safe — uses validated identifiers) + +**Completed:** +- [x] `search_records_handler`: User search term now uses `$1` bind parameter instead of `format!()` interpolation +- [x] `get_record_handler`: Changed to use `build_safe_select_by_id_query()` from sql_guard +- [x] `count_records_handler`: Changed to use `build_safe_count_query()` from sql_guard +- [x] Added wildcard escaping (`%`, `_`) on search terms before passing to ILIKE +- [x] Build verified — compiles cleanly + +**Remaining:** +- [ ] Audit `contacts/contacts_api/service.rs` for similar patterns +- [ ] Add SQL injection fuzzing tests +- [ ] Consider migrating fully to Diesel query builder + +--- + +### SEC-06: ✅ PARTIALLY RESOLVED — `unwrap()`/`expect()` reduction started + +**Severity:** 🟠 HIGH +**Scope:** `botserver/src/` (~637 non-test instances remaining) + +**Completed:** +- [x] Fixed `rate_limiter.rs`: Replaced `expect()` with compile-time const `NonZeroU32` values +- [x] Security module production code reviewed and fixed + +**Remaining:** +- [ ] Continue systematic elimination in: `core/`, `llm/`, `main.rs`, `auto_task/` +- [ ] Replace with `?`, `.ok_or_else()`, `.unwrap_or_default()`, or `if let` +- [ ] Add a CI clippy lint to deny new `unwrap()`/`expect()` in non-test code +- [ ] Target: eliminate 50 instances per week + +--- + +## 🟠 P1 — HIGH PRIORITY IMPROVEMENTS + +### IMP-01: Massive file sizes violating 450-line rule + +**Severity:** 🟠 HIGH +**Total codebase:** 289,453 lines across `botserver/src/` + +Top offenders (vs 450 max policy): + +| File | Lines | Oversize By | +|------|-------|-------------| +| `auto_task/app_generator.rs` | 3,586 | 7.9× | +| `auto_task/autotask_api.rs` | 2,301 | 5.1× | +| `basic/mod.rs` | 2,095 | 4.7× | +| `core/bot/mod.rs` | 1,584 | 3.5× | +| `channels/pinterest.rs` | 1,565 | 3.5× | +| `drive/mod.rs` | 1,525 | 3.4× | +| `whatsapp/mod.rs` | 1,516 | 3.4× | +| `channels/snapchat.rs` | 1,500 | 3.3× | +| `security/rbac_middleware.rs` | 1,498 | 3.3× | +| `basic/keywords/crm/attendance.rs` | 1,495 | 3.3× | +| `core/package_manager/installer.rs` | 1,473 | 3.3× | +| `workspaces/mod.rs` | 1,370 | 3.0× | +| `drive/drive_monitor/mod.rs` | 1,329 | 3.0× | +| `video/engine.rs` | 1,318 | 2.9× | +| `core/package_manager/facade.rs` | 1,313 | 2.9× | + +**Actions:** +- [ ] Split `auto_task/app_generator.rs` (3586 lines) → ~8 modules +- [ ] Split `auto_task/autotask_api.rs` (2301 lines) → ~5 modules +- [ ] Split `basic/mod.rs` (2095 lines) → ~5 modules +- [ ] Split `core/bot/mod.rs` (1584 lines) → ~4 modules +- [ ] Continue down the list — 20+ files exceed 450 lines + +--- + +### IMP-02: Shell scripts lack proper safety measures + +**Severity:** 🟡 MEDIUM +**Files:** `reset.sh`, `stop.sh`, `DEPENDENCIES.sh` + +| Script | Issue | +|--------|-------| +| `reset.sh` | No shebang, no `set -e`, destructive `rm -rf` without confirmation | +| `stop.sh` | No shebang, no `set -e`, uses `pkill -9` (SIGKILL) without graceful shutdown | +| `DEPENDENCIES.sh` | Excessive indentation, no `set -e` after shebang, missing `apt-get update` before install | + +**Actions:** +- [ ] Add `#!/bin/bash` and `set -euo pipefail` to `reset.sh` and `stop.sh` +- [ ] Add confirmation prompt to `reset.sh` before deleting data +- [ ] In `stop.sh`, try SIGTERM first, then SIGKILL after timeout +- [ ] In `DEPENDENCIES.sh`, add `apt-get update` before `apt-get install` +- [ ] Fix indentation in `DEPENDENCIES.sh` (8-space indent throughout) + +--- + +### IMP-03: Repository root polluted with debug/test artifacts + +**Severity:** 🟡 MEDIUM +**Files in root that don't belong:** + +| File | Should Be | +|------|-----------| +| `cristo-batizado.png`, `cristo-home.png`, etc. (10 PNGs) | In `.gitignore` (already) or deleted | +| `start.bas`, `test_begin_blocks.bas` | Move to `bottemplates/` or `tests/` | +| `init.json` | Tracked by git — remove (see SEC-03) | +| `COMPILATION_FIXES_SUMMARY.md` | Move to `botbook/` or delete | +| `PROMPT.md` | Move to `botbook/` or `.todo/` | +| `botserver-new.log` | Add to `.gitignore` | +| `vault-unseal-keys` | DELETE and purge history (see SEC-01) | + +**Actions:** +- [ ] Delete or move all `.png` screenshot files from root +- [ ] Move `start.bas`, `test_begin_blocks.bas` to appropriate directories +- [ ] Move documentation `.md` files to `botbook/` +- [ ] Add `*-new.log` pattern to `.gitignore` +- [ ] Clean up root to contain only essential workspace files + +--- + +### IMP-04: `unsafe` block in production code + +**Severity:** 🟡 MEDIUM +**File:** `botserver/src/llm/rate_limiter.rs:99` + +```rust +.unwrap_or_else(|| unsafe { NonZeroU32::new_unchecked(1) }) +``` + +While this specific case is sound (1 is non-zero), using `unsafe` sets a bad precedent and can be replaced with safe alternatives. + +**Actions:** +- [ ] Replace with `NonZeroU32::new(1).unwrap()` (compile-time guaranteed) or `NonZeroU32::MIN` +- [ ] Add a workspace-wide `#![deny(unsafe_code)]` policy (with exceptions documented) + +--- + +### IMP-05: Missing `cargo-audit` for dependency vulnerability scanning + +**Severity:** 🟡 MEDIUM + +`cargo-audit` is not installed, meaning no automated dependency vulnerability scanning is happening. The README recommends weekly `cargo audit` runs but the tool isn't available. + +**Actions:** +- [ ] Install `cargo-audit`: `cargo install cargo-audit` +- [ ] Run `cargo audit` and fix any findings +- [ ] Add `cargo audit` to CI pipeline +- [ ] Set up `dependabot` or `renovate` for automated dependency updates + +--- + +### IMP-06: CORS configuration may be too permissive + +**Severity:** 🟡 MEDIUM +**File:** `botserver/src/security/cors.rs` + +Multiple `allow_origin` patterns exist including predicate-based validation. Need to verify the predicate function properly validates origins and doesn't allow wildcards in production. + +**Actions:** +- [ ] Audit `validate_origin` predicate function +- [ ] Ensure production CORS is restricted to specific known domains +- [ ] Add CORS configuration tests +- [ ] Document allowed origins in configuration + +--- + +## 🟡 P2 — MEDIUM PRIORITY IMPROVEMENTS + +### IMP-07: Rate limiter defaults may be too generous + +**Severity:** 🟡 MEDIUM +**File:** `botserver/src/security/rate_limiter.rs` + +Default rate limits: +- General: 100 req/s, 200 burst +- Auth: 50 req/s, 100 burst +- API: 500 req/s, 1000 burst + +500 req/s for API with 1000 burst is very high for a bot platform and may not protect against DDoS. + +**Actions:** +- [ ] Review rate limits against actual traffic patterns +- [ ] Add per-IP and per-user rate limiting (not just global) +- [ ] Add rate limiting for WebSocket connections +- [ ] Consider tiered rate limits based on authentication status + +--- + +### IMP-08: CSRF protection implementation needs validation + +**Severity:** 🟡 MEDIUM +**File:** `botserver/src/security/csrf.rs` + +CSRF token system exists but needs verification that it's properly integrated into all state-changing endpoints. + +**Actions:** +- [ ] Verify CSRF middleware is applied to ALL POST/PUT/DELETE routes +- [ ] Ensure CSRF tokens are properly bound to user sessions +- [ ] Add CSRF bypass tests (attempt requests without valid token) +- [ ] Document CSRF exemptions (if any, e.g., API key-authenticated routes) + +--- + +### IMP-09: Missing security headers audit + +**Severity:** 🟡 MEDIUM +**File:** `botserver/src/security/headers.rs` + +Security headers module exists but needs verification of completeness. + +**Actions:** +- [ ] Verify all headers are set: `X-Frame-Options`, `X-Content-Type-Options`, `Strict-Transport-Security`, `Content-Security-Policy`, `Referrer-Policy`, `Permissions-Policy` +- [ ] Test with security header scanners (Mozilla Observatory, securityheaders.com) +- [ ] Ensure CSP is properly restrictive (no `unsafe-inline` or `unsafe-eval`) + +--- + +### IMP-10: No dependency pinning — using caret versions + +**Severity:** 🟡 MEDIUM +**File:** `Cargo.toml` + +Most dependencies use minimum version specifiers (e.g., `"1.0"`, `"0.4"`) which resolve to the latest compatible version. While `Cargo.lock` pins exact versions, the lock file is `.gitignore`'d, meaning different developers/CI will get different dependency versions. + +**Actions:** +- [ ] Remove `Cargo.lock` from `.gitignore` — it should be tracked for applications (not libraries) +- [ ] Consider using exact versions for critical dependencies (security, crypto) +- [ ] Document dependency update procedure + +--- + +### IMP-11: Stale submodule references + +**Severity:** 🟡 MEDIUM + +`git status` shows 5 submodules with uncommitted changes: +``` + m botapp + m botbook + m botlib + m bottemplates + m bottest +``` + +**Actions:** +- [ ] For each dirty submodule: commit, push, and update parent reference +- [ ] Add submodule status check to CI +- [ ] Document submodule workflow more prominently + +--- + +## 🔵 P3 — LOW PRIORITY / NICE-TO-HAVE + +### IMP-12: Add git pre-commit hook for secret scanning + +**Actions:** +- [ ] Install `gitleaks` or `trufflehog` as a pre-commit hook +- [ ] Scan for patterns: API keys, tokens, passwords, private keys +- [ ] Block commits containing secrets + +--- + +### IMP-13: ✅ RESOLVED — README.md refactored +**Severity:** 🟡 MEDIUM +**Status:** ✅ Split into `README.md` (architecture) and `AGENTS.md` (LLM rules). + +Original issue: README was 1335 lines. Now split for better AI/human separation. + +**Completed:** +- [x] Extract security policy & LLM rules → `AGENTS.md` +- [x] Keep README focused: overview, quick start, architecture + +--- + +### IMP-14: ~40 TODO/FIXME/HACK/XXX comments in codebase + +**Actions:** +- [ ] Triage all 40 TODO comments — either fix them or create issues +- [ ] Remove stale TODOs +- [ ] Replace `HACK`/`XXX` with proper solutions + +--- + +### IMP-15: Missing integration test coverage + +**Severity:** 🔵 LOW +**File:** `bottest/` + +README mentions 80%+ coverage goal for critical paths but no coverage reports are generated. + +**Actions:** +- [ ] Set up `cargo-tarpaulin` or `llvm-cov` for coverage reports +- [ ] Add coverage gate to CI (fail if below threshold) +- [ ] Prioritize tests for: auth flows, session management, script execution, drive sync + +--- + +### IMP-16: `package.json` has both `puppeteer` and `@playwright/test` + +**Severity:** 🔵 LOW +**File:** `package.json` + +Two browser automation tools installed. Choose one and remove the other. + +**Actions:** +- [ ] Decide on Playwright or Puppeteer +- [ ] Remove unused tool dependency +- [ ] Clean up `node_modules` + +--- + +### IMP-17: `Cargo.lock` is gitignored + +**Severity:** 🟡 MEDIUM +**File:** `.gitignore` line 37 + +For applications (not libraries), `Cargo.lock` should be committed to ensure reproducible builds. This workspace produces binaries (`botserver`, `botui`, `botapp`) — so the lock file should be tracked. + +**Actions:** +- [ ] Remove `Cargo.lock` from `.gitignore` +- [ ] Commit the current `Cargo.lock` +- [ ] Update contributing guidelines + +--- + +### IMP-18: Missing Dockerfile / container deployment + +**Severity:** 🔵 LOW + +No Dockerfile or container configuration found, despite having container dependencies (LXC in `DEPENDENCIES.sh`). + +**Actions:** +- [ ] Create multi-stage `Dockerfile` for production builds +- [ ] Create `docker-compose.yml` for development environment +- [ ] Document container deployment process + +--- + +### IMP-19: No CI/CD configuration found in `.github/` or `.forgejo/` + +**Severity:** 🟡 MEDIUM + +`.github/` and `.forgejo/` directories exist but need verification of CI pipeline configuration. + +**Actions:** +- [ ] Verify CI runs: `cargo check`, `cargo clippy`, `cargo test`, `cargo audit` +- [ ] Add security scanning step to CI +- [ ] Add binary size tracking to CI +- [ ] Add coverage reporting to CI + +--- + +## 📊 Summary + +| Priority | Count | Category | +|----------|-------|----------| +| 🔴 P0 Critical | 6 | **4 fully resolved, 2 partially resolved** | +| 🟠 P1 High | 6 | Significant improvements for stability/security | +| 🟡 P2 Medium | 5 | Important quality and security improvements | +| 🔵 P3 Low | 8 | Nice-to-have improvements and cleanup | +| **Total** | **25** | **6 P0 items addressed this session** | + +### ✅ Completed This Session (2026-02-19) + +1. **SEC-01**: ✅ `vault-unseal-keys` removed from git tracking +2. **SEC-02**: ✅ Verified `.env` files untracked, created `.env.example` +3. **SEC-03**: ✅ `init.json` removed from git tracking, added to `.gitignore` +4. **SEC-04**: ✅ All 8 `Command::new()` replaced with `SafeCommand`, whitelist updated +5. **SEC-05**: ✅ SQL injection fixed — parameterized queries in search/get/count handlers +6. **SEC-06**: ✅ Started — `rate_limiter.rs` expect() calls replaced with const NonZeroU32 +7. **Bonus**: ✅ `.gitignore` hardened with `*.pem`, `*.key`, `*.crt`, `*.cert` patterns +8. **Bonus**: ✅ Fixed 3 duplicate `#[cfg(not(windows))]` attributes in `installer.rs` + +### 🔴 Still Requires Manual Action + +1. **Rotate Vault unseal keys** (SEC-01) +2. **Rotate Vault tokens in .env** (SEC-02) +3. **Purge secrets from git history** using `git filter-repo` (SEC-01) + +--- + +*This document should be reviewed and updated weekly. Tasks should be moved to the project's issue tracker once triaged.*