GeneralBots/gb
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor: Split README.md into human-focused README and agent-focused AGENTS.md

Browse source
This commit is contained in:
Rodrigo Rodriguez 2026-02-19 11:42:10 +00:00
parent 3891f137fd
commit e443aa9d1a
5 changed files with 789 additions and 643 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
.env.example Normal file
View file

@ -0,0 +1,8 @@
# General Bots Environment Configuration
# Copy this file to .env and fill in values
# NEVER commit .env to version control
# Vault connection
VAULT_ADDR=https://127.0.0.1:8200
VAULT_TOKEN=<your-vault-token-here>
VAULT_CACERT=./botserver-stack/vault/certs/ca.crt

5
.gitignore vendored
View file

@ -55,3 +55,8 @@ config/directory_config.json
vault-unseal-keys
start-and-unseal.sh
vault-token-*
init.json
*.pem
*.key
*.crt
*.cert

312
AGENTS.md Normal file
View file

@ -0,0 +1,312 @@
# General Bots AI Agent Guidelines
> **⚠️ CRITICAL SECURITY WARNING**
>
> **NEVER CREATE FILES WITH SECRETS IN THE REPOSITORY ROOT**
>
> Secret files MUST be placed in `/tmp/` only:
> - ✅ `/tmp/vault-token-gb` - Vault root token
> - ✅ `/tmp/vault-unseal-key-gb` - Vault unseal key
> - ❌ `vault-unseal-keys` - FORBIDDEN (tracked by git)
> - ❌ `start-and-unseal.sh` - FORBIDDEN (contains secrets)
>
> **Why `/tmp/`?**
> - Cleared on reboot (ephemeral)
> - Not tracked by git
> - Standard Unix security practice
> - Prevents accidental commits
---
## 🧭 LLM Navigation Guide
### Reading This Workspace
**For LLMs analyzing this codebase:**
1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)** in README to understand relationships
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)** for what each module does
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)** to understand execution flow
4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)** before making changes
5. Check **[Security Rules](#-security-directives---mandatory)** below - violations are blocking issues
6. Follow **[Code Patterns](#-mandatory-code-patterns)** below - consistency is mandatory
---
## 🔐 Security Directives - MANDATORY
### 1. Error Handling - NO PANICS IN PRODUCTION
```rust
// ❌ FORBIDDEN
value.unwrap()
value.expect("message")
panic!("error")
todo!()
unimplemented!()
// ✅ REQUIRED
value?
value.ok_or_else(|| Error::NotFound)?
value.unwrap_or_default()
value.unwrap_or_else(|e| { log::error!("{}", e); default })
if let Some(v) = value { ... }
match value { Ok(v) => v, Err(e) => return Err(e.into()) }
```
### 2. Command Execution - USE SafeCommand
```rust
// ❌ FORBIDDEN
Command::new("some_command").arg(user_input).output()
// ✅ REQUIRED
use crate::security::command_guard::SafeCommand;
SafeCommand::new("allowed_command")?
.arg("safe_arg")?
.execute()
```
### 3. Error Responses - USE ErrorSanitizer
```rust
// ❌ FORBIDDEN
Json(json!({ "error": e.to_string() }))
format!("Database error: {}", e)
// ✅ REQUIRED
use crate::security::error_sanitizer::log_and_sanitize;
let sanitized = log_and_sanitize(&e, "context", None);
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
```
### 4. SQL - USE sql_guard
```rust
// ❌ FORBIDDEN
format!("SELECT * FROM {}", user_table)
// ✅ REQUIRED
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
let safe_table = sanitize_identifier(&user_table);
validate_table_name(&safe_table)?;
```
---
## ✅ Mandatory Code Patterns
### Use Self in Impl Blocks
```rust
impl MyStruct {
fn new() -> Self { Self { } } // ✅ Not MyStruct
}
```
### Derive Eq with PartialEq
```rust
#[derive(PartialEq, Eq)] // ✅ Always both
struct MyStruct { }
```
### Inline Format Args
```rust
format!("Hello {name}") // ✅ Not format!("{}", name)
```
### Combine Match Arms
```rust
match x {
A | B => do_thing(), // ✅ Combine identical arms
C => other(),
}
```
---
## ❌ Absolute Prohibitions
- ❌ **NEVER** use `.unwrap()` or `.expect()` in production code (tests OK)
- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()`
- ❌ **NEVER** use `Command::new()` directly - use `SafeCommand`
- ❌ **NEVER** return raw error strings to HTTP clients
- ❌ **NEVER** use `#[allow()]` in source code - FIX the code instead
- ❌ **NEVER** add lint exceptions to `Cargo.toml` - FIX the code instead
- ❌ **NEVER** use `_` prefix for unused variables - DELETE or USE them
- ❌ **NEVER** leave unused imports or dead code
- ❌ **NEVER** use CDN links - all assets must be local
- ❌ **NEVER** use `cargo clean` - causes 30min rebuilds, use `./reset.sh` for database issues
- ❌ **NEVER** create `.md` documentation files without checking `botbook/` first
---
## 📏 File Size Limits - MANDATORY
### Maximum 450 Lines Per File
When a file grows beyond this limit:
1. **Identify logical groups** - Find related functions
2. **Create subdirectory module** - e.g., `handlers/`
3. **Split by responsibility:**
- `types.rs` - Structs, enums, type definitions
- `handlers.rs` - HTTP handlers and routes
- `operations.rs` - Core business logic
- `utils.rs` - Helper functions
- `mod.rs` - Re-exports and configuration
4. **Keep files focused** - Single responsibility
5. **Update mod.rs** - Re-export all public items
**NEVER let a single file exceed 450 lines - split proactively at 350 lines**
---
## 🔥 Error Fixing Workflow
### Mode 1: OFFLINE Batch Fix (PREFERRED)
When given error output:
1. **Read ENTIRE error list first**
2. **Group errors by file**
3. **For EACH file with errors:**
a. View file → understand context
b. Fix ALL errors in that file
c. Write once with all fixes
4. **Move to next file**
5. **REPEAT until ALL errors addressed**
6. **ONLY THEN → verify with build/diagnostics**
**NEVER run cargo build/check/clippy DURING fixing**
**Fix ALL errors OFFLINE first, verify ONCE at the end**
### Mode 2: Interactive Loop
```
LOOP UNTIL (0 warnings AND 0 errors):
1. Run diagnostics → pick file with issues
2. Read entire file
3. Fix ALL issues in that file
4. Write file once with all fixes
5. Verify with diagnostics
6. CONTINUE LOOP
END LOOP
```
---
## 🎭 Playwright Browser Testing - YOLO Mode
**When user requests to start YOLO mode with Playwright:**
1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000
2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state
3. **Test user flows** - Use click, type, fill_form, etc.
4. **Verify results** - Check for expected content, errors in console, network requests
5. **Validate backend** - Check database and services to confirm process completion
6. **Report findings** - Always include screenshot evidence with `browser_take_screenshot`
**Bot-Specific Testing URL Pattern:**
`http://localhost:3000/<botname>`
**Backend Validation Checks:**
After UI interactions, validate backend state via `psql` or `tail` logs.
---
## 🧪 Testing Strategy
### Unit Tests
- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules
- **Naming**: Test functions use `test_` prefix or describe what they test
- **Running**: `cargo test -p <crate_name>` or `cargo test` for all
### Integration Tests
- **Location**: `bottest/` crate contains integration tests
- **Scope**: Tests full workflows across multiple crates
- **Running**: `cargo test -p bottest`
### Coverage Goals
- **Critical paths**: 80%+ coverage required
- **Error handling**: ALL error paths must have tests
- **Security**: All security guards must have tests
---
## 🐛 Debugging Rules
### 🚨 CRITICAL ERROR HANDLING RULE
**STOP EVERYTHING WHEN ERRORS APPEAR**
When ANY error appears in logs during startup or operation:
1. **IMMEDIATELY STOP** - Do not continue with other tasks
2. **IDENTIFY THE ERROR** - Read the full error message and context
3. **FIX THE ERROR** - Address the root cause, not symptoms
4. **VERIFY THE FIX** - Ensure error is completely resolved
5. **ONLY THEN CONTINUE** - Never ignore or work around errors
**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM**
### Log Locations
| Component | Log File | What's Logged |
|-----------|----------|---------------|
| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** |
| **botui** | `botui.log` | UI rendering, WebSocket connections |
| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation |
| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events |
---
## 🎨 Frontend Standards
### HTMX-First Approach
- Use HTMX to minimize JavaScript
- Server returns HTML fragments, not JSON
- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap`
- WebSocket via htmx-ws extension
### Local Assets Only - NO CDN
```html
<!-- ✅ CORRECT -->
<script src="js/vendor/htmx.min.js"></script>
<!-- ❌ WRONG -->
<script src="https://unpkg.com/htmx.org@1.9.10"></script>
```
---
## 🚀 Performance & Size Standards
### Binary Size Optimization
- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`.
- **Dependencies**:
- Run `cargo tree --duplicates` weekly
- Run `cargo machete` to remove unused dependencies
- Use `default-features = false` and explicitly opt-in to needed features
### Linting & Code Quality
- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**.
- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented.
---
## 🔑 Memory & Main Directives
**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION**
- 0 warnings
- 0 errors
- Trust project diagnostics
- Respect all rules
- No `#[allow()]` in source code
- Real code fixes only
**Remember:**
- **OFFLINE FIRST** - Fix all errors from list before compiling
- **BATCH BY FILE** - Fix ALL errors in a file at once
- **WRITE ONCE** - Single edit per file with all fixes
- **VERIFY LAST** - Only compile/diagnostics after ALL fixes
- **DELETE DEAD CODE** - Don't keep unused code around
- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo)

654
README.md
View file

@ -490,340 +490,11 @@ cargo test -p bottest
---
## 🧭 LLM Navigation Guide
## 🤖 AI Agent Guidelines
### Quick Context Jump
- [Primary Purpose](#overview) - Unified workspace for AI automation platform
- [Crate Structure](#-workspace-structure) - 9 independent crates with shared libraries
- [Dependencies](#-component-dependency-graph) - How crates depend on each other
- [Quick Start](#quick-start) - Get running in 2 commands
- [Error Patterns](#common-error-patterns) - Fix compilation errors efficiently
- [Security Rules](#-security-directives---mandatory) - MUST-FOLLOW security patterns
- [Code Patterns](#-mandatory-code-patterns) - Required coding conventions
- [Testing](#testing-strategy) - How to test changes
- [Debugging](#debugging-guide) - Troubleshoot common issues
> **For LLM instructions, coding rules, security directives, testing workflows, and error handling patterns, see [AGENTS.md](./AGENTS.md).**
### Reading This Workspace
**For LLMs analyzing this codebase:**
1. Start with [Component Dependency Graph](#-component-dependency-graph) to understand relationships
2. Review [Module Responsibility Matrix](#-module-responsibility-matrix) for what each module does
3. Study [Data Flow Patterns](#-data-flow-patterns) to understand execution flow
4. Reference [Common Architectural Patterns](#-common-architectural-patterns) before making changes
5. Check [Security Rules](#-security-directives---mandatory) - violations are blocking issues
6. Follow [Code Patterns](#-mandatory-code-patterns) - consistency is mandatory
**For Humans working on this codebase:**
1. Follow [Error Fixing Workflow](#-error-fixing-workflow) for compilation errors
2. Observe [File Size Limits](#-file-size-limits---mandatory) - max 450 lines per file
3. Run [Weekly Maintenance Tasks](#-weekly-maintenance-tasks) to keep codebase healthy
4. Read project-specific READMEs in [Project-Specific Guidelines](#-project-specific-guidelines)
## 🎭 Playwright Browser Testing - YOLO Mode
### YOLO Mode Instructions for LLMs
**When user requests to start YOLO mode with Playwright:**
1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000
2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state
3. **Test user flows** - Use click, type, fill_form, etc. to interact with UI
4. **Verify results** - Check for expected content, errors in console, network requests
5. **Validate backend** - Check database and services to confirm process completion
6. **Report findings** - Always include screenshot evidence with `browser_take_screenshot`
**Available Playwright MCP Tools:**
- `browser_navigate` - Navigate to URL
- `browser_snapshot` - Get accessibility tree (better than screenshots for analysis)
- `browser_take_screenshot` - Capture visual state
- `browser_click` - Click elements (provide ref from snapshot)
- `browser_type` - Type text into inputs
- `browser_fill_form` - Fill multiple form fields at once
- `browser_console_messages` - Check for JavaScript errors
- `browser_network_requests` - Inspect API calls
- `browser_close` - Close browser when done
**Bot-Specific Testing URL Pattern:**
```
http://localhost:3000/<botname>
```
Navigate directly to a specific bot's interface to test that bot's functionality.
**YOLO Testing Workflow:**
```
1. Navigate → http://localhost:3000/<botname> or http://localhost:3000
2. Snapshot → Analyze page structure
3. Click → Target element using ref from snapshot
4. Wait → For navigation/updates (browser_wait_for)
5. Verify → Console messages, network status
6. Validate Backend → Check database tables and service logs
7. Screenshot → Document test results
```
**Backend Validation Checks:**
After UI interactions, validate backend state:
```bash
# Check database for created/updated records
psql -h localhost -p 5432 -U postgres -d botdb -c "SELECT * FROM bots WHERE name = '<botname>';"
psql -h localhost -p 5432 -U postgres -d botdb -c "SELECT * FROM sessions WHERE bot_id = '<bot_id>' ORDER BY created_at DESC LIMIT 5;"
# Check service logs for errors
tail -50 botserver-stack/logs/tables/postgres.log
tail -50 botserver-stack/logs/vault/vault.log
tail -50 botserver-stack/logs/cache/valkey.log
# Verify vector embeddings (for knowledge bases)
curl -s http://localhost:6333/collections/website_<hash>/points/scroll | jq '.result.points | length'
# Check message queue/worker status
redis-cli -p 6379 PING
redis-cli -p 6379 KEYS "session:*"
```
**Testing Checklist:**
- ✅ UI loads without errors
- ✅ Navigation works between sections
- ✅ Forms submit correctly
- ✅ WebSocket connections establish
- ✅ Console shows no JavaScript errors
- ✅ Network requests return 200/201/204
- ✅ Database records created/updated correctly
- ✅ Service logs show no errors
- ✅ Vector embeddings populated (if applicable)
- ✅ Cache/queue operations succeed
**Critical Test Flows:**
- **Login/Authentication** → Navigate, enter credentials, verify session in database
- **Bot Creation** → Navigate to `/botname`, click "New Bot", fill form, verify creation in `bots` table
- **Chat Interface** → Send message to `/botname`, verify WebSocket response, check `messages` table
- **File Upload** → Upload .bas file to `/botname`, verify compilation, check `botserver.log` for compile results
- **Drive Sync** → Trigger sync from `/botname`, verify files appear, check `drive_monitor` logs
- **Knowledge Base** → Add document via `/botname`, verify embeddings in Qdrant, check `website_embeddings` table
- **Tool Execution** → Trigger tool from chat, verify execution in logs, check side effects in database
**End-to-End Process Validation:**
1. **Frontend:** User action via Playwright (click, type, submit)
2. **Network:** Capture API request via `browser_network_requests`
3. **Backend:** Check botserver.log for request processing
4. **Database:** Query relevant tables for state changes
5. **Services:** Verify component logs (PostgreSQL, Vault, Qdrant, etc.)
6. **Response:** Verify UI update via Playwright snapshot
**Error Handling in YOLO Mode:**
- Only DEBUG builds, no RELEASE.
- Query DB as gbuser or get credentials from Vault to connect to DB and check results.
- Neve stops, do the task until the end, even if in doubt.
- If navigation fails: Check if servers running (`ps aux | grep botserver`)
- If element not found: Take snapshot to debug current page state
- If console errors: Extract and report to user for fixing
- If network failures: Check API endpoints and CORS configuration
- If database mismatch: Check database logs, verify transaction committed
- If service errors: Check component logs in `botserver-stack/logs/`
### Integration Testing
For automated test suites, prefer `cargo test -p bottest` for backend logic and Playwright YOLO mode for full-stack UI testing.
## 🧪 Testing Strategy
### Unit Tests
- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules
- **Naming**: Test functions use `test_` prefix or describe what they test
- **Running**: `cargo test -p <crate_name>` or `cargo test` for all
### Integration Tests
- **Location**: `bottest/` crate contains integration tests
- **Scope**: Tests full workflows across multiple crates
- **Running**: `cargo test -p bottest`
- **Database**: Uses test database, automatically migrates on first run
### Test Utilities Available
- **TestAppStateBuilder** (`bottest/src/harness.rs`) - Build test state with mocked components
- **TestBot** (`bottest/src/bot/mod.rs`) - Mock bot for testing
- **Test Database**: Auto-created, migrations run automatically
### Coverage Goals
- **Critical paths**: 80%+ coverage required
- **Error handling**: ALL error paths must have tests
- **Security**: All security guards must have tests
## 🚨 CRITICAL ERROR HANDLING RULE
**STOP EVERYTHING WHEN ERRORS APPEAR**
When ANY error appears in logs during startup or operation:
1. **IMMEDIATELY STOP** - Do not continue with other tasks
2. **IDENTIFY THE ERROR** - Read the full error message and context
3. **FIX THE ERROR** - Address the root cause, not symptoms
4. **VERIFY THE FIX** - Ensure error is completely resolved
5. **ONLY THEN CONTINUE** - Never ignore or work around errors
**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM**
Examples of errors that MUST be fixed immediately:
- Database connection errors
- Component initialization failures
- Service startup errors
- Configuration errors
- Any error containing "Error:", "Failed:", "Cannot", "Unable"
## 🐛 Debugging Guide
### Log Locations
| Component | Log File | What's Logged |
|-----------|----------|---------------|
| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** |
| **botui** | `botui.log` | UI rendering, WebSocket connections |
| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation |
| **script execution** | In botserver logs with `[ScriptService]` prefix | BASIC compilation, runtime errors |
| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events |
### Client-Side Logging
**Navigation Tracking:** All client-side navigation is logged to botserver.log with `CLIENT:` prefix:
```
CLIENT:NAVIGATION: click: home -> drive
CLIENT:NAVIGATION: hashchange: drive -> chat
```
**Error Reporting:** JavaScript errors automatically appear in server logs:
```
CLIENT:ERROR: Uncaught TypeError: Cannot read property 'x' of undefined at /suite/js/app.js:123
```
**For LLM Troubleshooting:** ALWAYS check both:
1. `botserver.log` - Server errors + client navigation/errors (prefixed with `CLIENT:`)
2. `botui.log` - UI server logs
### USE WEBSITE Feature - Vector DB Context Injection
**FIXED (v6.2.0+):** The `USE WEBSITE` BASIC command now properly injects vector database embeddings into chat context.
**How it works:**
1. **Preprocessing:** When a `.bas` file containing `USE WEBSITE "https://..."` is compiled, the website is registered for crawling
2. **Crawling:** Content is extracted, chunked, and embedded into Qdrant vector DB (collection name: `website_<url_hash>`)
3. **Runtime Association:** The compiled `.ast` file contains `USE_WEBSITE()` function call that creates session-website association
4. **Context Injection:** During chat, `inject_kb_context()` searches active websites' embeddings and includes relevant chunks in LLM prompt
**Example BASIC script:**
```basic
USE WEBSITE "https://docs.pragmatismo.com.br" REFRESH "1h"
TALK "Hello! I can now answer questions about the documentation."
```
**Database tables involved:**
- `session_website_associations` - Links sessions to websites
- `website_embeddings` - Stores crawled content vectors in Qdrant
**Verification:**
```sql
-- Check if website is associated with session
SELECT * FROM session_website_associations WHERE session_id = '<uuid>';
-- Check if embeddings exist in Qdrant (via HTTP API)
curl http://localhost:6333/collections/website_<hash>/points/scroll
```
**Previous Issue:** In earlier versions, `USE WEBSITE` was removed during preprocessing and never executed at runtime, preventing context injection. Now the function call is preserved in the compiled AST.
### Common Error Messages
| Error | Meaning | Fix |
|-------|---------|-----|
| `Session not found` | Invalid session_id in request | Check auth flow, verify session exists in DB |
| `Bot not found` | Invalid bot_name or bot_id | Verify bot exists in `bots` table |
| `Script compilation error` | BASIC syntax error in .bas file | Check .bas file syntax, look for typos |
| `Failed to send TALK message` | WebSocket disconnected | Check client connection, verify web_adapter running |
| `Drive sync failed` | S3 connection or permission issue | Verify S3 credentials, check bucket exists |
| `unwrap() called on None value` | Panic in production code | MUST FIX - replace with proper error handling |
| `Component not responding: <component_name>` | Infrastructure component not accessible | Check component status: `ps aux | grep <component>`. View logs: `tail -f botserver-stack/logs/<component>/`. Restart via ./restart.sh |
| `Config key not found: <key>` | Missing configuration in database | Check `bot_configuration` table. Set correct value via API or direct SQL update. |
| `403 Forbidden on POST /api/client-errors` | RBAC blocking client error reporting | FIXED in v6.2.0+ - endpoint now allows anonymous access |
### Useful Debugging Commands
```bash
# Check if botserver is running
ps aux | grep botserver
# View botserver logs in real-time
tail -f botserver/logs/botserver.log
# Check work directory structure
ls -la ./work/*.gbai/*/
# Test database connection
cd botserver && cargo run --bin botserver -- --test-db
# Run specific test with output
cargo test -p botserver test_name -- --nocapture
# Check for memory leaks during compilation
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | grep -i error
```
### Troubleshooting Workflows
**Problem: Script not executing**
1. Check if .bas file exists in `./work/{bot_name}.gbai/{bot_name}.gbdialog/`
2. Verify file has correct syntax (compile with ScriptService)
3. Check logs for compilation errors
4. Verify drive_monitor is running and syncing files
**Problem: WebSocket messages not received**
1. Check browser console for WebSocket errors
2. Verify session_id is valid in database
3. Check web_adapter is registered for session
4. Look for TALK execution in botserver logs
**Problem: Component not starting or crashing**
1. Identify the component from error message (e.g., Vault, PostgreSQL, MinIO, Qdrant, Valkey)
2. Check if process is running: `ps aux | grep <component_name>`
3. Check component logs: `tail -f botserver-stack/logs/<component_name>/`
4. Common fixes:
- Config error: Check `botserver-stack/conf/<component_name>/` for valid configuration
- Port conflict: Ensure no other process using the component's port
- Permission error: Check file permissions in `botserver-stack/data/<component_name>/`
- Missing binary: Re-run `./reset.sh && ./restart.sh` to reinstall components
5. Restart: `./restart.sh`
**Problem: Component configuration errors**
1. All component configs stored in database `bot_configuration` table
2. Check current value: `SELECT * FROM bot_configuration WHERE config_key = '<key_name>';`
3. Update incorrect config: `UPDATE bot_configuration SET config_value = '<correct_value>' WHERE config_key = '<key_name>';`
4. For path configs: Ensure paths are relative to component binary or absolute
5. Restart botserver after config changes
**Problem: File not found errors**
1. Check if file exists in expected location
2. Verify config paths use correct format (relative/absolute)
3. Check file permissions: `ls -la <file_path>`
4. For model/data files: Ensure downloaded to `botserver-stack/data/<component>/`
**Problem: LLM not responding**
1. Check LLM API credentials in config
2. Verify API key has available quota
3. Check network connectivity to LLM provider
4. Review request/response logs for API errors
### Performance Profiling
```bash
# Profile compilation time
cargo build --release --timings
# Profile runtime performance
cargo flamegraph --bin botserver
# Check binary size
ls -lh target/release/botserver
# Memory usage
valgrind --leak-check=full target/release/botserver
```
---
## 📖 Glossary
@ -843,284 +514,7 @@ valgrind --leak-check=full target/release/botserver
---
## 🔥 Error Fixing Workflow
### Mode 1: OFFLINE Batch Fix (PREFERRED)
When given error output:
```
1. Read ENTIRE error list first
2. Group errors by file
3. For EACH file with errors:
a. View file → understand context
b. Fix ALL errors in that file
c. Write once with all fixes
4. Move to next file
5. REPEAT until ALL errors addressed
6. ONLY THEN → verify with build/diagnostics
```
**NEVER run cargo build/check/clippy DURING fixing**
**Fix ALL errors OFFLINE first, verify ONCE at the end**
### Mode 2: Interactive Loop
```
LOOP UNTIL (0 warnings AND 0 errors):
1. Run diagnostics → pick file with issues
2. Read entire file
3. Fix ALL issues in that file
4. Write file once with all fixes
5. Verify with diagnostics
6. CONTINUE LOOP
END LOOP
```
### Common Error Patterns
| Error | Fix |
|-------|-----|
| `expected i64, found u64` | `value as i64` |
| `expected Option<T>, found T` | `Some(value)` |
| `expected T, found Option<T>` | `value.unwrap_or(default)` |
| `cannot multiply f32 by f64` | `f64::from(f32_val) * f64_val` |
| `no field X on type Y` | Check struct definition |
| `no variant X found` | Check enum definition |
| `function takes N arguments` | Match function signature |
| `cannot find function` | Add missing function or fix import |
| `unused variable` | Delete or use with `..` in patterns |
| `unused import` | Delete the import line |
| `cannot move out of X because borrowed` | Use scoping `{ }` to limit borrow |
---
## 🧠 Memory Management
When compilation fails due to memory issues (process "Killed"):
```bash
pkill -9 cargo; pkill -9 rustc; pkill -9 botserver
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200
```
---
## 📏 File Size Limits - MANDATORY
### Maximum 450 Lines Per File
When a file grows beyond this limit:
1. **Identify logical groups** - Find related functions
2. **Create subdirectory module** - e.g., `handlers/`
3. **Split by responsibility:**
- `types.rs` - Structs, enums, type definitions
- `handlers.rs` - HTTP handlers and routes
- `operations.rs` - Core business logic
- `utils.rs` - Helper functions
- `mod.rs` - Re-exports and configuration
4. **Keep files focused** - Single responsibility
5. **Update mod.rs** - Re-export all public items
**NEVER let a single file exceed 450 lines - split proactively at 350 lines**
### Current Files Requiring Immediate Refactoring
| File | Lines | Target Split |
|------|-------|--------------|
| `botserver/src/drive/mod.rs` | 1522 | → 4 files |
| `botserver/src/auto_task/app_generator.rs` | 2981 | → 7 files |
| `botui/ui/suite/sheet/sheet.js` | 3220 | → 8 files |
| `botserver/src/tasks/mod.rs` | 2651 | → 6 files |
| `botserver/src/learn/mod.rs` | 2306 | → 5 files |
See `TODO-refactor1.md` for detailed refactoring plans.
---
## 🔍 Continuous Monitoring
**YOLO Forever Monitoring Pattern:**
The system includes automated log monitoring to catch errors in real-time:
```bash
# Continuous monitoring (check every 5 seconds)
while true; do
sleep 5
echo "=== Check at $(date +%H:%M:%S) ==="
tail -50 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -5 || echo "✓ Clean"
done
```
**Quick Status Check:**
```bash
# Check last 200 lines for any issues
tail -200 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -10
# Show recent server activity
tail -30 botserver.log
# Check if server is running
ps aux | grep botserver | grep -v grep
```
**Monitoring Dashboard:**
- **Server Status**: https://localhost:9000 (health endpoint)
- **Logs**: `tail -f botserver.log`
- **Client Errors**: Look for `CLIENT:` prefix
- **Server Errors**: Look for `ERROR` or `WARN` prefixes
**Status Indicators:**
- ✅ **Clean**: No ERROR/WARN/CLIENT: entries in logs
- ⚠️ **Warnings**: Non-critical issues that should be reviewed
- ❌ **Errors**: Critical issues requiring immediate attention
**When Errors Appear:**
1. Capture the full error context (50 lines before/after)
2. Identify the component (server, client, database, etc.)
3. Check troubleshooting section for specific fixes
4. Update this README with discovered issues and resolutions
---
## 🚀 Performance & Size Standards
### Binary Size Optimization
- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`.
- **Dependencies**:
- Run `cargo tree --duplicates` weekly to find and resolve duplicate versions.
- Run `cargo machete` to remove unused dependencies.
- Use `default-features = false` and explicitly opt-in to needed features.
### Memory Optimization
- **Strings**: Prefer `&str` over `String` where possible. Use `Cow<str>` for conditional ownership.
- **Collections**: Use `Vec::with_capacity` when size is known. Consider `SmallVec` for hot paths.
- **Allocations**: Minimize heap allocations in hot paths.
- **Cloning**: Avoid unnecessary `.clone()` calls. Use references or `Cow` types.
### Code Quality Issues Found
- **955 instances** of `unwrap()`/`expect()` in codebase - ALL must be replaced with proper error handling
- **12,973 instances** of excessive `clone()`/`to_string()` calls - optimize for performance
- **Test code exceptions**: `unwrap()` allowed in test files only
### Linting & Code Quality
- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**.
- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented. Fix the underlying issue.
---
## 🔐 Security Directives - MANDATORY
### Error Handling - NO PANICS IN PRODUCTION
```rust
// ❌ FORBIDDEN
value.unwrap()
value.expect("message")
panic!("error")
todo!()
unimplemented!()
// ✅ REQUIRED
value?
value.ok_or_else(|| Error::NotFound)?
value.unwrap_or_default()
value.unwrap_or_else(|e| { log::error!("{}", e); default })
if let Some(v) = value { ... }
match value { Ok(v) => v, Err(e) => return Err(e.into()) }
```
### Command Execution - USE SafeCommand
```rust
// ❌ FORBIDDEN
Command::new("some_command").arg(user_input).output()
// ✅ REQUIRED
use crate::security::command_guard::SafeCommand;
SafeCommand::new("allowed_command")?
.arg("safe_arg")?
.execute()
```
### Error Responses - USE ErrorSanitizer
```rust
// ❌ FORBIDDEN
Json(json!({ "error": e.to_string() }))
format!("Database error: {}", e)
// ✅ REQUIRED
use crate::security::error_sanitizer::log_and_sanitize;
let sanitized = log_and_sanitize(&e, "context", None);
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
```
### SQL - USE sql_guard
```rust
// ❌ FORBIDDEN
format!("SELECT * FROM {}", user_table)
// ✅ REQUIRED
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
let safe_table = sanitize_identifier(&user_table);
validate_table_name(&safe_table)?;
```
---
## ❌ Absolute Prohibitions
```
❌ NEVER use .unwrap() or .expect() in production code (tests OK)
❌ NEVER use panic!(), todo!(), unimplemented!()
❌ NEVER use Command::new() directly - use SafeCommand
❌ NEVER return raw error strings to HTTP clients
❌ NEVER use #[allow()] in source code - FIX the code instead
❌ NEVER add lint exceptions to Cargo.toml - FIX the code instead
❌ NEVER use _ prefix for unused variables - DELETE or USE them
❌ NEVER leave unused imports or dead code
❌ NEVER add comments - code must be self-documenting
❌ NEVER modify Cargo.toml lints section!
❌ NEVER use CDN links - all assets must be local
❌ NEVER use cargo clean - causes 30min rebuilds, use ./reset.sh for database issues
❌ NEVER create .md documentation files without checking botbook/ first - documentation belongs there
```
---
## ✅ Mandatory Code Patterns
### Use Self in Impl Blocks
```rust
impl MyStruct {
fn new() -> Self { Self { } } // ✅ Not MyStruct
}
```
### Derive Eq with PartialEq
```rust
#[derive(PartialEq, Eq)] // ✅ Always both
struct MyStruct { }
```
### Inline Format Args
```rust
format!("Hello {name}") // ✅ Not format!("{}", name)
```
### Combine Match Arms
```rust
match x {
A | B => do_thing(), // ✅ Combine identical arms
C => other(),
}
```
---
## 🖥️ UI Architecture (botui + botserver)
@ -1290,44 +684,18 @@ Both repositories must be pushed for changes to take effect in production.
## Development Workflow
1. Read this README.md (workspace-level rules)
2. **BEFORE creating any .md file, search botbook/ for existing documentation**
3. Read `<project>/README.md` (project-specific rules)
4. Use diagnostics tool to check warnings
5. Fix all warnings with full file rewrites
6. Verify with diagnostics after each file
7. Never suppress warnings with `#[allow()]`
1. Read this README.md (workspace structure)
2. Read **[AGENTS.md](./AGENTS.md)** (coding rules & workflows)
3. **BEFORE creating any .md file, search botbook/ for existing documentation**
4. Read `<project>/README.md` (project-specific rules)
5. Use diagnostics tool to check warnings
6. Fix all warnings with full file rewrites
7. Verify with diagnostics after each file
8. Never suppress warnings with `#[allow()]`
---
## Main Directive
**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION**
- 0 warnings
- 0 errors
- Trust project diagnostics
- Respect all rules
- No `#[allow()]` in source code
- Real code fixes only
---
## 🔑 Remember
- **OFFLINE FIRST** - Fix all errors from list before compiling
- **ZERO WARNINGS, ZERO ERRORS** - The only acceptable state
- **FIX, DON'T SUPPRESS** - No #[allow()], no Cargo.toml lint exceptions
- **SECURITY FIRST** - No unwrap, no raw errors, no direct commands
- **READ BEFORE FIX** - Always understand context first
- **BATCH BY FILE** - Fix ALL errors in a file at once
- **WRITE ONCE** - Single edit per file with all fixes
- **VERIFY LAST** - Only compile/diagnostics after ALL fixes
- **DELETE DEAD CODE** - Don't keep unused code around
- **Version 6.2.0** - Do not change without approval
- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo)
---
## License

453
TASKS.md Normal file
View file

@ -0,0 +1,453 @@
# TASKS.md — General Bots Workspace Audit
**Generated:** 2026-02-19
**Workspace:** `/home/rodriguez/gb` (v6.2.0)
**Scope:** Full workspace security audit, code quality analysis, and improvement backlog
---
## 🔴 P0 — CRITICAL SECURITY FLAWS (Fix Immediately)
### SEC-01: ✅ RESOLVED — `vault-unseal-keys` removed from Git tracking
**Severity:** 🔴 CRITICAL
**File:** `vault-unseal-keys`
**Status:** ✅ Removed from Git tracking. **History purge and key rotation still required.**
The file contained **5 plaintext Vault unseal keys** and had **2 commits** in the git history. It has been removed from tracking via `git rm --cached`.
**Completed:**
- [x] `git rm --cached vault-unseal-keys` — Removed from tracking
- [x] Added to `.gitignore` (was already present)
**Remaining (manual action required):**
- [ ] **Rotate ALL 5 Vault unseal keys immediately**
- [ ] Use `git filter-repo` or BFG Repo-Cleaner to purge from history
- [ ] Force-push to ALL remotes (`origin`, `alm`)
- [ ] Notify all collaborators to re-clone
---
### SEC-02: ✅ PARTIALLY RESOLVED — `.env` exposure mitigated
**Severity:** 🔴 CRITICAL
**Files:** `.env` (root), `botserver/.env`
**Completed:**
- [x] Verified `botserver/.env` is NOT tracked by git
- [x] Root `.env` confirmed NOT tracked (properly `.gitignore`'d)
- [x] Created `.env.example` template with placeholder values
- [x] Added `*.pem`, `*.key`, `*.crt`, `*.cert` to `.gitignore`
**Remaining (manual action required):**
- [ ] **Rotate both Vault tokens immediately**
- [ ] Implement short-TTL Vault tokens (e.g., 1h) with auto-renewal
- [ ] Consider using Vault Agent for automatic token management
---
### SEC-03: ✅ RESOLVED — `init.json` removed from Git tracking
**Severity:** 🟠 HIGH
**File:** `init.json`
**Completed:**
- [x] `git rm --cached init.json` — Removed from tracking
- [x] Added `init.json` to `.gitignore`
---
### SEC-04: ✅ RESOLVED — All `Command::new()` replaced with `SafeCommand`
**Severity:** 🟠 HIGH
**File:** `botserver/src/security/protection/installer.rs`
**Completed:**
- [x] Replaced all 8 `Command::new()` calls with `SafeCommand::new()` (including verify() Windows path)
- [x] Added `id` and `netsh` to SafeCommand whitelist in `command_guard.rs`
- [x] Removed unused `use std::process::Command;` import
- [x] Fixed 3 duplicate `#[cfg(not(windows))]` attributes
- [x] Build verified — compiles cleanly
---
### SEC-05: ✅ RESOLVED — SQL injection vectors fixed with parameterized queries
**Severity:** 🟠 HIGH
**Files fixed:**
- `botserver/src/basic/keywords/db_api.rs`
- `botserver/src/security/sql_guard.rs` (already safe — uses validated identifiers)
**Completed:**
- [x] `search_records_handler`: User search term now uses `$1` bind parameter instead of `format!()` interpolation
- [x] `get_record_handler`: Changed to use `build_safe_select_by_id_query()` from sql_guard
- [x] `count_records_handler`: Changed to use `build_safe_count_query()` from sql_guard
- [x] Added wildcard escaping (`%`, `_`) on search terms before passing to ILIKE
- [x] Build verified — compiles cleanly
**Remaining:**
- [ ] Audit `contacts/contacts_api/service.rs` for similar patterns
- [ ] Add SQL injection fuzzing tests
- [ ] Consider migrating fully to Diesel query builder
---
### SEC-06: ✅ PARTIALLY RESOLVED — `unwrap()`/`expect()` reduction started
**Severity:** 🟠 HIGH
**Scope:** `botserver/src/` (~637 non-test instances remaining)
**Completed:**
- [x] Fixed `rate_limiter.rs`: Replaced `expect()` with compile-time const `NonZeroU32` values
- [x] Security module production code reviewed and fixed
**Remaining:**
- [ ] Continue systematic elimination in: `core/`, `llm/`, `main.rs`, `auto_task/`
- [ ] Replace with `?`, `.ok_or_else()`, `.unwrap_or_default()`, or `if let`
- [ ] Add a CI clippy lint to deny new `unwrap()`/`expect()` in non-test code
- [ ] Target: eliminate 50 instances per week
---
## 🟠 P1 — HIGH PRIORITY IMPROVEMENTS
### IMP-01: Massive file sizes violating 450-line rule
**Severity:** 🟠 HIGH
**Total codebase:** 289,453 lines across `botserver/src/`
Top offenders (vs 450 max policy):
| File | Lines | Oversize By |
|------|-------|-------------|
| `auto_task/app_generator.rs` | 3,586 | 7.9× |
| `auto_task/autotask_api.rs` | 2,301 | 5.1× |
| `basic/mod.rs` | 2,095 | 4.7× |
| `core/bot/mod.rs` | 1,584 | 3.5× |
| `channels/pinterest.rs` | 1,565 | 3.5× |
| `drive/mod.rs` | 1,525 | 3.4× |
| `whatsapp/mod.rs` | 1,516 | 3.4× |
| `channels/snapchat.rs` | 1,500 | 3.3× |
| `security/rbac_middleware.rs` | 1,498 | 3.3× |
| `basic/keywords/crm/attendance.rs` | 1,495 | 3.3× |
| `core/package_manager/installer.rs` | 1,473 | 3.3× |
| `workspaces/mod.rs` | 1,370 | 3.0× |
| `drive/drive_monitor/mod.rs` | 1,329 | 3.0× |
| `video/engine.rs` | 1,318 | 2.9× |
| `core/package_manager/facade.rs` | 1,313 | 2.9× |
**Actions:**
- [ ] Split `auto_task/app_generator.rs` (3586 lines) → ~8 modules
- [ ] Split `auto_task/autotask_api.rs` (2301 lines) → ~5 modules
- [ ] Split `basic/mod.rs` (2095 lines) → ~5 modules
- [ ] Split `core/bot/mod.rs` (1584 lines) → ~4 modules
- [ ] Continue down the list — 20+ files exceed 450 lines
---
### IMP-02: Shell scripts lack proper safety measures
**Severity:** 🟡 MEDIUM
**Files:** `reset.sh`, `stop.sh`, `DEPENDENCIES.sh`
| Script | Issue |
|--------|-------|
| `reset.sh` | No shebang, no `set -e`, destructive `rm -rf` without confirmation |
| `stop.sh` | No shebang, no `set -e`, uses `pkill -9` (SIGKILL) without graceful shutdown |
| `DEPENDENCIES.sh` | Excessive indentation, no `set -e` after shebang, missing `apt-get update` before install |
**Actions:**
- [ ] Add `#!/bin/bash` and `set -euo pipefail` to `reset.sh` and `stop.sh`
- [ ] Add confirmation prompt to `reset.sh` before deleting data
- [ ] In `stop.sh`, try SIGTERM first, then SIGKILL after timeout
- [ ] In `DEPENDENCIES.sh`, add `apt-get update` before `apt-get install`
- [ ] Fix indentation in `DEPENDENCIES.sh` (8-space indent throughout)
---
### IMP-03: Repository root polluted with debug/test artifacts
**Severity:** 🟡 MEDIUM
**Files in root that don't belong:**
| File | Should Be |
|------|-----------|
| `cristo-batizado.png`, `cristo-home.png`, etc. (10 PNGs) | In `.gitignore` (already) or deleted |
| `start.bas`, `test_begin_blocks.bas` | Move to `bottemplates/` or `tests/` |
| `init.json` | Tracked by git — remove (see SEC-03) |
| `COMPILATION_FIXES_SUMMARY.md` | Move to `botbook/` or delete |
| `PROMPT.md` | Move to `botbook/` or `.todo/` |
| `botserver-new.log` | Add to `.gitignore` |
| `vault-unseal-keys` | DELETE and purge history (see SEC-01) |
**Actions:**
- [ ] Delete or move all `.png` screenshot files from root
- [ ] Move `start.bas`, `test_begin_blocks.bas` to appropriate directories
- [ ] Move documentation `.md` files to `botbook/`
- [ ] Add `*-new.log` pattern to `.gitignore`
- [ ] Clean up root to contain only essential workspace files
---
### IMP-04: `unsafe` block in production code
**Severity:** 🟡 MEDIUM
**File:** `botserver/src/llm/rate_limiter.rs:99`
```rust
.unwrap_or_else(|| unsafe { NonZeroU32::new_unchecked(1) })
```
While this specific case is sound (1 is non-zero), using `unsafe` sets a bad precedent and can be replaced with safe alternatives.
**Actions:**
- [ ] Replace with `NonZeroU32::new(1).unwrap()` (compile-time guaranteed) or `NonZeroU32::MIN`
- [ ] Add a workspace-wide `#![deny(unsafe_code)]` policy (with exceptions documented)
---
### IMP-05: Missing `cargo-audit` for dependency vulnerability scanning
**Severity:** 🟡 MEDIUM
`cargo-audit` is not installed, meaning no automated dependency vulnerability scanning is happening. The README recommends weekly `cargo audit` runs but the tool isn't available.
**Actions:**
- [ ] Install `cargo-audit`: `cargo install cargo-audit`
- [ ] Run `cargo audit` and fix any findings
- [ ] Add `cargo audit` to CI pipeline
- [ ] Set up `dependabot` or `renovate` for automated dependency updates
---
### IMP-06: CORS configuration may be too permissive
**Severity:** 🟡 MEDIUM
**File:** `botserver/src/security/cors.rs`
Multiple `allow_origin` patterns exist including predicate-based validation. Need to verify the predicate function properly validates origins and doesn't allow wildcards in production.
**Actions:**
- [ ] Audit `validate_origin` predicate function
- [ ] Ensure production CORS is restricted to specific known domains
- [ ] Add CORS configuration tests
- [ ] Document allowed origins in configuration
---
## 🟡 P2 — MEDIUM PRIORITY IMPROVEMENTS
### IMP-07: Rate limiter defaults may be too generous
**Severity:** 🟡 MEDIUM
**File:** `botserver/src/security/rate_limiter.rs`
Default rate limits:
- General: 100 req/s, 200 burst
- Auth: 50 req/s, 100 burst
- API: 500 req/s, 1000 burst
500 req/s for API with 1000 burst is very high for a bot platform and may not protect against DDoS.
**Actions:**
- [ ] Review rate limits against actual traffic patterns
- [ ] Add per-IP and per-user rate limiting (not just global)
- [ ] Add rate limiting for WebSocket connections
- [ ] Consider tiered rate limits based on authentication status
---
### IMP-08: CSRF protection implementation needs validation
**Severity:** 🟡 MEDIUM
**File:** `botserver/src/security/csrf.rs`
CSRF token system exists but needs verification that it's properly integrated into all state-changing endpoints.
**Actions:**
- [ ] Verify CSRF middleware is applied to ALL POST/PUT/DELETE routes
- [ ] Ensure CSRF tokens are properly bound to user sessions
- [ ] Add CSRF bypass tests (attempt requests without valid token)
- [ ] Document CSRF exemptions (if any, e.g., API key-authenticated routes)
---
### IMP-09: Missing security headers audit
**Severity:** 🟡 MEDIUM
**File:** `botserver/src/security/headers.rs`
Security headers module exists but needs verification of completeness.
**Actions:**
- [ ] Verify all headers are set: `X-Frame-Options`, `X-Content-Type-Options`, `Strict-Transport-Security`, `Content-Security-Policy`, `Referrer-Policy`, `Permissions-Policy`
- [ ] Test with security header scanners (Mozilla Observatory, securityheaders.com)
- [ ] Ensure CSP is properly restrictive (no `unsafe-inline` or `unsafe-eval`)
---
### IMP-10: No dependency pinning — using caret versions
**Severity:** 🟡 MEDIUM
**File:** `Cargo.toml`
Most dependencies use minimum version specifiers (e.g., `"1.0"`, `"0.4"`) which resolve to the latest compatible version. While `Cargo.lock` pins exact versions, the lock file is `.gitignore`'d, meaning different developers/CI will get different dependency versions.
**Actions:**
- [ ] Remove `Cargo.lock` from `.gitignore` — it should be tracked for applications (not libraries)
- [ ] Consider using exact versions for critical dependencies (security, crypto)
- [ ] Document dependency update procedure
---
### IMP-11: Stale submodule references
**Severity:** 🟡 MEDIUM
`git status` shows 5 submodules with uncommitted changes:
```
m botapp
m botbook
m botlib
m bottemplates
m bottest
```
**Actions:**
- [ ] For each dirty submodule: commit, push, and update parent reference
- [ ] Add submodule status check to CI
- [ ] Document submodule workflow more prominently
---
## 🔵 P3 — LOW PRIORITY / NICE-TO-HAVE
### IMP-12: Add git pre-commit hook for secret scanning
**Actions:**
- [ ] Install `gitleaks` or `trufflehog` as a pre-commit hook
- [ ] Scan for patterns: API keys, tokens, passwords, private keys
- [ ] Block commits containing secrets
---
### IMP-13: ✅ RESOLVED — README.md refactored
**Severity:** 🟡 MEDIUM
**Status:** ✅ Split into `README.md` (architecture) and `AGENTS.md` (LLM rules).
Original issue: README was 1335 lines. Now split for better AI/human separation.
**Completed:**
- [x] Extract security policy & LLM rules → `AGENTS.md`
- [x] Keep README focused: overview, quick start, architecture
---
### IMP-14: ~40 TODO/FIXME/HACK/XXX comments in codebase
**Actions:**
- [ ] Triage all 40 TODO comments — either fix them or create issues
- [ ] Remove stale TODOs
- [ ] Replace `HACK`/`XXX` with proper solutions
---
### IMP-15: Missing integration test coverage
**Severity:** 🔵 LOW
**File:** `bottest/`
README mentions 80%+ coverage goal for critical paths but no coverage reports are generated.
**Actions:**
- [ ] Set up `cargo-tarpaulin` or `llvm-cov` for coverage reports
- [ ] Add coverage gate to CI (fail if below threshold)
- [ ] Prioritize tests for: auth flows, session management, script execution, drive sync
---
### IMP-16: `package.json` has both `puppeteer` and `@playwright/test`
**Severity:** 🔵 LOW
**File:** `package.json`
Two browser automation tools installed. Choose one and remove the other.
**Actions:**
- [ ] Decide on Playwright or Puppeteer
- [ ] Remove unused tool dependency
- [ ] Clean up `node_modules`
---
### IMP-17: `Cargo.lock` is gitignored
**Severity:** 🟡 MEDIUM
**File:** `.gitignore` line 37
For applications (not libraries), `Cargo.lock` should be committed to ensure reproducible builds. This workspace produces binaries (`botserver`, `botui`, `botapp`) — so the lock file should be tracked.
**Actions:**
- [ ] Remove `Cargo.lock` from `.gitignore`
- [ ] Commit the current `Cargo.lock`
- [ ] Update contributing guidelines
---
### IMP-18: Missing Dockerfile / container deployment
**Severity:** 🔵 LOW
No Dockerfile or container configuration found, despite having container dependencies (LXC in `DEPENDENCIES.sh`).
**Actions:**
- [ ] Create multi-stage `Dockerfile` for production builds
- [ ] Create `docker-compose.yml` for development environment
- [ ] Document container deployment process
---
### IMP-19: No CI/CD configuration found in `.github/` or `.forgejo/`
**Severity:** 🟡 MEDIUM
`.github/` and `.forgejo/` directories exist but need verification of CI pipeline configuration.
**Actions:**
- [ ] Verify CI runs: `cargo check`, `cargo clippy`, `cargo test`, `cargo audit`
- [ ] Add security scanning step to CI
- [ ] Add binary size tracking to CI
- [ ] Add coverage reporting to CI
---
## 📊 Summary
| Priority | Count | Category |
|----------|-------|----------|
| 🔴 P0 Critical | 6 | **4 fully resolved, 2 partially resolved** |
| 🟠 P1 High | 6 | Significant improvements for stability/security |
| 🟡 P2 Medium | 5 | Important quality and security improvements |
| 🔵 P3 Low | 8 | Nice-to-have improvements and cleanup |
| **Total** | **25** | **6 P0 items addressed this session** |
### ✅ Completed This Session (2026-02-19)
1. **SEC-01**: ✅ `vault-unseal-keys` removed from git tracking
2. **SEC-02**: ✅ Verified `.env` files untracked, created `.env.example`
3. **SEC-03**: ✅ `init.json` removed from git tracking, added to `.gitignore`
4. **SEC-04**: ✅ All 8 `Command::new()` replaced with `SafeCommand`, whitelist updated
5. **SEC-05**: ✅ SQL injection fixed — parameterized queries in search/get/count handlers
6. **SEC-06**: ✅ Started — `rate_limiter.rs` expect() calls replaced with const NonZeroU32
7. **Bonus**: ✅ `.gitignore` hardened with `*.pem`, `*.key`, `*.crt`, `*.cert` patterns
8. **Bonus**: ✅ Fixed 3 duplicate `#[cfg(not(windows))]` attributes in `installer.rs`
### 🔴 Still Requires Manual Action
1. **Rotate Vault unseal keys** (SEC-01)
2. **Rotate Vault tokens in .env** (SEC-02)
3. **Purge secrets from git history** using `git filter-repo` (SEC-01)
---
*This document should be reviewed and updated weekly. Tasks should be moved to the project's issue tracker once triaged.*