Compare commits
17 commits
49d9b193b2
...
282892505a
| Author | SHA1 | Date | |
|---|---|---|---|
| 282892505a | |||
| 8ef5ec7d68 | |||
| 288274b6ae | |||
| 54538c1c27 | |||
| e2d5e4c029 | |||
| 66f54677bf | |||
| 63c2e17818 | |||
| f695cad94f | |||
| e7b5718469 | |||
| a01c229d71 | |||
| be19d9e06b | |||
| 60f2a87955 | |||
| 7d2b4fbab2 | |||
| 8619b1b09d | |||
| 647f3f1c6c | |||
| 41adbc2cca | |||
| c865017561 |
28 changed files with 774 additions and 12391 deletions
|
|
@ -1,8 +0,0 @@
|
|||
# General Bots Environment Configuration
|
||||
# Copy this file to .env and fill in values
|
||||
# NEVER commit .env to version control
|
||||
|
||||
# Vault connection
|
||||
VAULT_ADDR=https://127.0.0.1:8200
|
||||
VAULT_TOKEN=<your-vault-token-here>
|
||||
VAULT_CACERT=./botserver-stack/vault/certs/ca.crt
|
||||
16
.gitignore
vendored
16
.gitignore
vendored
|
|
@ -32,9 +32,9 @@ botserver-installers/*
|
|||
botserver-stack
|
||||
TODO*
|
||||
work
|
||||
.swp
|
||||
# Lock file
|
||||
# Cargo.lock (should be tracked)
|
||||
|
||||
# Lock file (regenerated from Cargo.toml)
|
||||
Cargo.lock
|
||||
.kiro
|
||||
config
|
||||
|
||||
|
|
@ -50,13 +50,3 @@ node_modules/
|
|||
/playwright/.auth/
|
||||
config/directory_config.json
|
||||
# CI cache bust: Fri Feb 13 22:33:51 UTC 2026
|
||||
|
||||
# Secrets - NEVER commit these files
|
||||
vault-unseal-keys
|
||||
start-and-unseal.sh
|
||||
vault-token-*
|
||||
init.json
|
||||
*.pem
|
||||
*.key
|
||||
*.crt
|
||||
*.cert
|
||||
|
|
|
|||
362
AGENTS.md
362
AGENTS.md
|
|
@ -1,362 +0,0 @@
|
|||
# General Bots AI Agent Guidelines
|
||||
|
||||
> **⚠️ CRITICAL SECURITY WARNING**
|
||||
>
|
||||
> **NEVER CREATE FILES WITH SECRETS IN THE REPOSITORY ROOT**
|
||||
>
|
||||
> Secret files MUST be placed in `/tmp/` only:
|
||||
> - ✅ `/tmp/vault-token-gb` - Vault root token
|
||||
> - ✅ `/tmp/vault-unseal-key-gb` - Vault unseal key
|
||||
> - ❌ `vault-unseal-keys` - FORBIDDEN (tracked by git)
|
||||
> - ❌ `start-and-unseal.sh` - FORBIDDEN (contains secrets)
|
||||
>
|
||||
> **Why `/tmp/`?**
|
||||
> - Cleared on reboot (ephemeral)
|
||||
> - Not tracked by git
|
||||
> - Standard Unix security practice
|
||||
> - Prevents accidental commits
|
||||
|
||||
---
|
||||
|
||||
## 🧭 LLM Navigation Guide
|
||||
|
||||
### Reading This Workspace
|
||||
|
||||
**For LLMs analyzing this codebase:**
|
||||
1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)** in README to understand relationships
|
||||
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)** for what each module does
|
||||
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)** to understand execution flow
|
||||
4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)** before making changes
|
||||
5. Check **[Security Rules](#-security-directives---mandatory)** below - violations are blocking issues
|
||||
6. Follow **[Code Patterns](#-mandatory-code-patterns)** below - consistency is mandatory
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security Directives - MANDATORY
|
||||
|
||||
### 1. Error Handling - NO PANICS IN PRODUCTION
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
value.unwrap()
|
||||
value.expect("message")
|
||||
panic!("error")
|
||||
todo!()
|
||||
unimplemented!()
|
||||
|
||||
// ✅ REQUIRED
|
||||
value?
|
||||
value.ok_or_else(|| Error::NotFound)?
|
||||
value.unwrap_or_default()
|
||||
value.unwrap_or_else(|e| { log::error!("{}", e); default })
|
||||
if let Some(v) = value { ... }
|
||||
match value { Ok(v) => v, Err(e) => return Err(e.into()) }
|
||||
```
|
||||
|
||||
### 2. Command Execution - USE SafeCommand
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
Command::new("some_command").arg(user_input).output()
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::command_guard::SafeCommand;
|
||||
SafeCommand::new("allowed_command")?
|
||||
.arg("safe_arg")?
|
||||
.execute()
|
||||
```
|
||||
|
||||
### 3. Error Responses - USE ErrorSanitizer
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
Json(json!({ "error": e.to_string() }))
|
||||
format!("Database error: {}", e)
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::error_sanitizer::log_and_sanitize;
|
||||
let sanitized = log_and_sanitize(&e, "context", None);
|
||||
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
|
||||
```
|
||||
|
||||
### 4. SQL - USE sql_guard
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
format!("SELECT * FROM {}", user_table)
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
|
||||
let safe_table = sanitize_identifier(&user_table);
|
||||
validate_table_name(&safe_table)?;
|
||||
```
|
||||
|
||||
### 5. Rate Limiting Strategy (IMP-07)
|
||||
|
||||
- **Default Limits:**
|
||||
- General: 100 req/s (global)
|
||||
- Auth: 10 req/s (login endpoints)
|
||||
- API: 50 req/s (per token)
|
||||
- **Implementation:**
|
||||
- MUST use `governor` crate
|
||||
- MUST implement per-IP and per-User tracking
|
||||
- WebSocket connections MUST have message rate limits (e.g., 10 msgs/s)
|
||||
|
||||
### 6. CSRF Protection (IMP-08)
|
||||
|
||||
- **Requirement:** ALL state-changing endpoints (POST, PUT, DELETE, PATCH) MUST require a CSRF token.
|
||||
- **Implementation:**
|
||||
- Use `tower_csrf` or similar middleware
|
||||
- Token MUST be bound to user session
|
||||
- Double-Submit Cookie pattern or Header-based token verification
|
||||
- **Exemptions:** API endpoints using Bearer Token authentication (stateless)
|
||||
|
||||
### 7. Security Headers (IMP-09)
|
||||
|
||||
- **Mandatory Headers on ALL Responses:**
|
||||
- `Content-Security-Policy`: "default-src 'self'; script-src 'self'; object-src 'none';"
|
||||
- `Strict-Transport-Security`: "max-age=63072000; includeSubDomains; preload"
|
||||
- `X-Frame-Options`: "DENY" or "SAMEORIGIN"
|
||||
- `X-Content-Type-Options`: "nosniff"
|
||||
- `Referrer-Policy`: "strict-origin-when-cross-origin"
|
||||
- `Permissions-Policy`: "geolocation=(), microphone=(), camera=()"
|
||||
|
||||
### 8. Dependency Management (IMP-10)
|
||||
|
||||
- **Pinning:**
|
||||
- Application crates (`botserver`, `botui`) MUST track `Cargo.lock`
|
||||
- Library crates (`botlib`) MUST NOT track `Cargo.lock`
|
||||
- **Versions:**
|
||||
- Critical dependencies (crypto, security) MUST use exact versions (e.g., `=1.0.1`)
|
||||
- Regular dependencies MAY use caret (e.g., `1.0`)
|
||||
- **Auditing:**
|
||||
- Run `cargo audit` weekly
|
||||
- Update dependencies only via PR with testing
|
||||
|
||||
---
|
||||
|
||||
## ✅ Mandatory Code Patterns
|
||||
|
||||
### Use Self in Impl Blocks
|
||||
```rust
|
||||
impl MyStruct {
|
||||
fn new() -> Self { Self { } } // ✅ Not MyStruct
|
||||
}
|
||||
```
|
||||
|
||||
### Derive Eq with PartialEq
|
||||
```rust
|
||||
#[derive(PartialEq, Eq)] // ✅ Always both
|
||||
struct MyStruct { }
|
||||
```
|
||||
|
||||
### Inline Format Args
|
||||
```rust
|
||||
format!("Hello {name}") // ✅ Not format!("{}", name)
|
||||
```
|
||||
|
||||
### Combine Match Arms
|
||||
```rust
|
||||
match x {
|
||||
A | B => do_thing(), // ✅ Combine identical arms
|
||||
C => other(),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ❌ Absolute Prohibitions
|
||||
|
||||
- ❌ **NEVER** build in release mode - ONLY debug builds allowed
|
||||
- ❌ **NEVER** use `--release` flag on ANY cargo command
|
||||
- ❌ **NEVER** use `--all-targets` with clippy - too slow (1m 44s without vs 10min+ with)
|
||||
- ❌ **NEVER** use `--all-features` unless testing specific feature gates
|
||||
- ❌ **ALWAYS** use: `cargo clippy --workspace` (DEBUG mode, lib + bin only)
|
||||
- ❌ **NEVER** run `cargo build` - use `cargo check` for syntax verification
|
||||
|
||||
**Current Status:** ✅ **0 clippy warnings** (down from 61 - PERFECT SCORE in YOLO mode)
|
||||
- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()`
|
||||
- ❌ **NEVER** use `Command::new()` directly - use `SafeCommand`
|
||||
- ❌ **NEVER** return raw error strings to HTTP clients
|
||||
- ❌ **NEVER** use `#[allow()]` in source code - FIX the code instead
|
||||
- ❌ **NEVER** add lint exceptions to `Cargo.toml` - FIX the code instead
|
||||
- ❌ **NEVER** use `_` prefix for unused variables - DELETE or USE them
|
||||
- ❌ **NEVER** leave unused imports or dead code
|
||||
- ❌ **NEVER** use CDN links - all assets must be local
|
||||
- ❌ **NEVER** use `cargo clean` - causes 30min rebuilds, use `./reset.sh` for database issues
|
||||
- ❌ **NEVER** create `.md` documentation files without checking `botbook/` first
|
||||
- ❌ **NEVER** comment out code - FIX it or DELETE it entirely
|
||||
|
||||
---
|
||||
|
||||
## 📏 File Size Limits - MANDATORY
|
||||
|
||||
### Maximum 450 Lines Per File
|
||||
|
||||
When a file grows beyond this limit:
|
||||
|
||||
1. **Identify logical groups** - Find related functions
|
||||
2. **Create subdirectory module** - e.g., `handlers/`
|
||||
3. **Split by responsibility:**
|
||||
- `types.rs` - Structs, enums, type definitions
|
||||
- `handlers.rs` - HTTP handlers and routes
|
||||
- `operations.rs` - Core business logic
|
||||
- `utils.rs` - Helper functions
|
||||
- `mod.rs` - Re-exports and configuration
|
||||
4. **Keep files focused** - Single responsibility
|
||||
5. **Update mod.rs** - Re-export all public items
|
||||
|
||||
**NEVER let a single file exceed 450 lines - split proactively at 350 lines**
|
||||
|
||||
---
|
||||
|
||||
## 🔥 Error Fixing Workflow
|
||||
|
||||
### Mode 1: OFFLINE Batch Fix (PREFERRED)
|
||||
|
||||
When given error output:
|
||||
|
||||
1. **Read ENTIRE error list first**
|
||||
2. **Group errors by file**
|
||||
3. **For EACH file with errors:**
|
||||
a. View file → understand context
|
||||
b. Fix ALL errors in that file
|
||||
c. Write once with all fixes
|
||||
4. **Move to next file**
|
||||
5. **REPEAT until ALL errors addressed**
|
||||
6. **ONLY THEN → verify with build/diagnostics**
|
||||
|
||||
**NEVER run cargo build/check/clippy DURING fixing**
|
||||
**Fix ALL errors OFFLINE first, verify ONCE at the end**
|
||||
|
||||
### Mode 2: Interactive Loop
|
||||
|
||||
```
|
||||
LOOP UNTIL (0 warnings AND 0 errors):
|
||||
1. Run diagnostics → pick file with issues
|
||||
2. Read entire file
|
||||
3. Fix ALL issues in that file
|
||||
4. Write file once with all fixes
|
||||
5. Verify with diagnostics
|
||||
6. CONTINUE LOOP
|
||||
END LOOP
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎭 Playwright Browser Testing - YOLO Mode
|
||||
|
||||
**When user requests to start YOLO mode with Playwright:**
|
||||
|
||||
1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000
|
||||
2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state
|
||||
3. **Test user flows** - Use click, type, fill_form, etc.
|
||||
4. **Verify results** - Check for expected content, errors in console, network requests
|
||||
5. **Validate backend** - Check database and services to confirm process completion
|
||||
6. **Report findings** - Always include screenshot evidence with `browser_take_screenshot`
|
||||
|
||||
**Bot-Specific Testing URL Pattern:**
|
||||
`http://localhost:3000/<botname>`
|
||||
|
||||
**Backend Validation Checks:**
|
||||
After UI interactions, validate backend state via `psql` or `tail` logs.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules
|
||||
- **Naming**: Test functions use `test_` prefix or describe what they test
|
||||
- **Running**: `cargo test -p <crate_name>` or `cargo test` for all
|
||||
|
||||
### Integration Tests
|
||||
- **Location**: `bottest/` crate contains integration tests
|
||||
- **Scope**: Tests full workflows across multiple crates
|
||||
- **Running**: `cargo test -p bottest`
|
||||
|
||||
### Coverage Goals
|
||||
- **Critical paths**: 80%+ coverage required
|
||||
- **Error handling**: ALL error paths must have tests
|
||||
- **Security**: All security guards must have tests
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Debugging Rules
|
||||
|
||||
### 🚨 CRITICAL ERROR HANDLING RULE
|
||||
|
||||
**STOP EVERYTHING WHEN ERRORS APPEAR**
|
||||
|
||||
When ANY error appears in logs during startup or operation:
|
||||
1. **IMMEDIATELY STOP** - Do not continue with other tasks
|
||||
2. **IDENTIFY THE ERROR** - Read the full error message and context
|
||||
3. **FIX THE ERROR** - Address the root cause, not symptoms
|
||||
4. **VERIFY THE FIX** - Ensure error is completely resolved
|
||||
5. **ONLY THEN CONTINUE** - Never ignore or work around errors
|
||||
|
||||
**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM**
|
||||
|
||||
### Log Locations
|
||||
|
||||
| Component | Log File | What's Logged |
|
||||
|-----------|----------|---------------|
|
||||
| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** |
|
||||
| **botui** | `botui.log` | UI rendering, WebSocket connections |
|
||||
| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation |
|
||||
| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events |
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Frontend Standards
|
||||
|
||||
### HTMX-First Approach
|
||||
- Use HTMX to minimize JavaScript
|
||||
- Server returns HTML fragments, not JSON
|
||||
- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap`
|
||||
- WebSocket via htmx-ws extension
|
||||
|
||||
### Local Assets Only - NO CDN
|
||||
```html
|
||||
<!-- ✅ CORRECT -->
|
||||
<script src="js/vendor/htmx.min.js"></script>
|
||||
|
||||
<!-- ❌ WRONG -->
|
||||
<script src="https://unpkg.com/htmx.org@1.9.10"></script>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Performance & Size Standards
|
||||
|
||||
### Binary Size Optimization
|
||||
- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`.
|
||||
- **Dependencies**:
|
||||
- Run `cargo tree --duplicates` weekly
|
||||
- Run `cargo machete` to remove unused dependencies
|
||||
- Use `default-features = false` and explicitly opt-in to needed features
|
||||
|
||||
### Linting & Code Quality
|
||||
- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**.
|
||||
- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented.
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Memory & Main Directives
|
||||
|
||||
**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION**
|
||||
|
||||
- 0 warnings
|
||||
- 0 errors
|
||||
- Trust project diagnostics
|
||||
- Respect all rules
|
||||
- No `#[allow()]` in source code
|
||||
- Real code fixes only
|
||||
|
||||
**Remember:**
|
||||
- **OFFLINE FIRST** - Fix all errors from list before compiling
|
||||
- **BATCH BY FILE** - Fix ALL errors in a file at once
|
||||
- **WRITE ONCE** - Single edit per file with all fixes
|
||||
- **VERIFY LAST** - Only compile/diagnostics after ALL fixes
|
||||
- **DELETE DEAD CODE** - Don't keep unused code around
|
||||
- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo)
|
||||
|
|
@ -1,167 +0,0 @@
|
|||
# Cristo Redentor Bot - Compilation Fixes Summary
|
||||
|
||||
**Date**: 2026-02-18 14:35 UTC
|
||||
**Status**: ✅ All Compilation Issues Fixed
|
||||
|
||||
---
|
||||
|
||||
## Problems Fixed
|
||||
|
||||
### 1. Expression Complexity Error ✅ FIXED
|
||||
|
||||
**Error**: `Expression exceeds maximum complexity (line 32)`
|
||||
|
||||
**Root Cause**: TALK blocks with many lines exceeded Rhai's expression complexity limit when all chunks were combined into a single expression.
|
||||
|
||||
**Solution**: Implemented hierarchical chunking in `botserver/src/basic/compiler/blocks/talk.rs`:
|
||||
- Primary chunks: 5 lines each
|
||||
- Combined chunks: Groups of 5 primary chunks
|
||||
- Final TALK: Combines only combined chunks (2-3 chunks instead of 8+)
|
||||
|
||||
**Example Output**:
|
||||
```rhai
|
||||
let __talk_chunk_0__ = "Line 1" + "\n" + "Line 2" + ...; // 5 lines
|
||||
...
|
||||
let __talk_chunk_7__ = "Line 36" + "\n" + ...; // 5 lines
|
||||
let __talk_combined_0_0__ = __talk_chunk_0__ + ... + __talk_chunk_4__;
|
||||
let __talk_combined_0_1__ = __talk_chunk_5__ + ... + __talk_chunk_7__;
|
||||
TALK __talk_combined_0_0__ + "\n" + __talk_combined_0_1__; // Only 2 chunks!
|
||||
```
|
||||
|
||||
**Files Modified**:
|
||||
- `botserver/src/basic/compiler/blocks/talk.rs` (lines 150-200)
|
||||
|
||||
### 2. Email Quoting Issue ✅ FIXED
|
||||
|
||||
**Error**: `Syntax error: Unknown operator: '@'`
|
||||
|
||||
**Root Cause**: `BEGIN MAIL "email@example.com"` passed email with quotes to the mail compiler, which then added another set of quotes, resulting in `""email@example.com""`.
|
||||
|
||||
**Solution**: Strip existing quotes before adding new ones in `botserver/src/basic/compiler/blocks/mail.rs`:
|
||||
```rust
|
||||
let recipient_expr = if recipient.contains('@') {
|
||||
let stripped = recipient.trim_matches('"');
|
||||
format!("\"{}\"", stripped)
|
||||
} else {
|
||||
recipient.to_string()
|
||||
};
|
||||
```
|
||||
|
||||
**BASIC Syntax**:
|
||||
```basic
|
||||
BEGIN MAIL "imagem@santuariocristoredentor.com.br"
|
||||
Subject: Nova Solicitacao - ${protocoloNumero}
|
||||
...
|
||||
END MAIL
|
||||
```
|
||||
|
||||
**Rhai Output**:
|
||||
```rhai
|
||||
send_mail("imagem@santuariocristoredentor.com.br", "Nova Solicitacao - ...", body, []);
|
||||
```
|
||||
|
||||
**Files Modified**:
|
||||
- `botserver/src/basic/compiler/blocks/mail.rs` (lines 134-138)
|
||||
|
||||
### 3. Runtime Preprocessing Overhead ✅ FIXED
|
||||
|
||||
**Issue**: All preprocessing (TALK/MAIL conversion, IF/THEN, SELECT/CASE, keyword conversion) happened at runtime for every tool execution.
|
||||
|
||||
**Solution**: Moved all preprocessing to compile-time in `botserver/src/basic/compiler/mod.rs`:
|
||||
- .ast files now contain fully converted Rhai code
|
||||
- No runtime conversion overhead
|
||||
- Tools execute directly from precompiled .ast files
|
||||
|
||||
**Files Modified**:
|
||||
- `botserver/src/basic/compiler/mod.rs` (lines 597-607)
|
||||
|
||||
---
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Compilation Status: ✅ All 10 Tools Compile Successfully
|
||||
|
||||
| Tool | Combined Chunks | Admin Email Quoted | Status |
|
||||
|------|-----------------|-------------------|--------|
|
||||
| 06 - Uso de Imagem | 3 | ✅ `("email@...")` | ✅ Success |
|
||||
| 07 - Licenciamento | 3 | ✅ `("email@...")` | ✅ Success |
|
||||
| 08 - Evento/Iluminação | 3 | ✅ `("email@...")` | ✅ Success |
|
||||
| 09 - Cadastrar Guia | 3 | ✅ `("email@...")` | ✅ Success |
|
||||
| 10 - Fazer Doação | 0 (short) | ✅ `("email@...")` | ✅ Success |
|
||||
|
||||
### Verification Commands
|
||||
|
||||
```bash
|
||||
# Check hierarchical chunking
|
||||
grep -c "__talk_combined_" /home/rodriguez/gb/work/cristo.gbai/cristo.gbdialog/06-uso-imagem.ast
|
||||
# Output: 3
|
||||
|
||||
# Check email quoting
|
||||
grep "send_mail.*santuariocristoredentor.com.br" /home/rodriguez/gb/work/cristo.gbai/cristo.gbdialog/*.ast
|
||||
# Output: send_mail("email@domain", ...) - single quotes, no double quotes
|
||||
|
||||
# Verify compilation
|
||||
tail /home/rodriguez/gb/botserver.log | grep "Successfully compiled"
|
||||
# Output: All tools show "Successfully compiled"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database State
|
||||
|
||||
**Current Records** (as of 2026-02-18 14:35 UTC):
|
||||
|
||||
| Table | Records | Last ID |
|
||||
|-------|---------|---------|
|
||||
| batizados | 1 | BAT-526725-4167 |
|
||||
| casamentos | 1 | SAVE-TEST-001 |
|
||||
| doacoes | 1 | DOA-20260218-9830 |
|
||||
| missas | 0 | - |
|
||||
| peregrinacoes | 0 | - |
|
||||
| pedidos_oracao | 0 | - |
|
||||
| pedidos_uso_imagem | 0 | - |
|
||||
| licenciamentos | 0 | - |
|
||||
| eventos_iluminacao | 0 | - |
|
||||
| guias_turismo | 0 | - |
|
||||
|
||||
**Total**: 3/10 tested, 7 remaining
|
||||
|
||||
---
|
||||
|
||||
## Remaining Work
|
||||
|
||||
### Runtime Testing Blocked by LLM Configuration
|
||||
|
||||
**Issue**: LLM token expired (`token expired or incorrect`)
|
||||
- Bot configured to use GLM API at https://api.z.ai/api/coding/paas/v4/
|
||||
- Local LLM models not available (missing from ./data/llm/)
|
||||
- Chat-based tool invocation requires working LLM
|
||||
|
||||
**Solutions** (choose one):
|
||||
1. Update GLM API token in configuration
|
||||
2. Install local LLM models (DeepSeek-R1, embedding models)
|
||||
3. Switch to different LLM provider
|
||||
|
||||
**Code Status**: ✅ Complete and verified
|
||||
- All compilation fixes applied
|
||||
- All tools compile successfully
|
||||
- .ast files contain correct, ready-to-execute code
|
||||
- No code changes required
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **All compilation issues fixed**
|
||||
✅ **Expression complexity resolved via hierarchical chunking**
|
||||
✅ **Email quoting corrected**
|
||||
✅ **Preprocessing moved to compile-time**
|
||||
✅ **All 10 tools compile with zero errors**
|
||||
|
||||
⏳ **Runtime testing pending** (blocked by LLM configuration, not code)
|
||||
|
||||
**Next Steps** (when LLM is available):
|
||||
1. Test tool execution through chat interface
|
||||
2. Verify database record creation
|
||||
3. Verify email delivery
|
||||
4. Update TEST.md with test results
|
||||
11595
Cargo.lock
generated
11595
Cargo.lock
generated
File diff suppressed because it is too large
Load diff
|
|
@ -175,7 +175,7 @@ indicatif = "0.18.0"
|
|||
|
||||
# ─── MEMORY ALLOCATOR ───
|
||||
tikv-jemallocator = "0.6"
|
||||
tikv-jemalloc-ctl = { version = "0.6", default-features = false, features = ["stats"] }
|
||||
tikv-jemalloc-ctl = { version = "0.6", default-features = false }
|
||||
|
||||
# ─── SECRETS / VAULT ───
|
||||
vaultrs = "0.7"
|
||||
|
|
|
|||
699
README.md
699
README.md
|
|
@ -1,25 +1,6 @@
|
|||
RULE 0: Never call tool_call while thinking. Ex NEVER do this: Let me check if the API call succeeded:<tool_call>terminal<arg_key>command</arg_key><arg_value>tail -50 botserver.log | grep -E "LLM streaming error|error|Error|SUCCESS|200"</arg_value><arg_key>cd</arg_key><arg_value>gb</arg_value></tool_call>. First finish Thinking, then emit a explanation and tool!
|
||||
# General Bots Workspace
|
||||
|
||||
## ⚠️ CRITICAL SECURITY WARNING
|
||||
|
||||
**NEVER CREATE FILES WITH SECRETS IN THE REPOSITORY ROOT**
|
||||
|
||||
Secret files MUST be placed in `/tmp/` only:
|
||||
- ✅ `/tmp/vault-token-gb` - Vault root token
|
||||
- ✅ `/tmp/vault-unseal-key-gb` - Vault unseal key
|
||||
- ❌ `vault-unseal-keys` - FORBIDDEN (tracked by git)
|
||||
- ❌ `start-and-unseal.sh` - FORBIDDEN (contains secrets)
|
||||
|
||||
**Files added to .gitignore:** `vault-unseal-keys`, `start-and-unseal.sh`, `vault-token-*`
|
||||
|
||||
**Why `/tmp/`?**
|
||||
- Cleared on reboot (ephemeral)
|
||||
- Not tracked by git
|
||||
- Standard Unix security practice
|
||||
- Prevents accidental commits
|
||||
|
||||
---
|
||||
|
||||
|
||||
**Version:** 6.2.0
|
||||
**Type:** Rust Workspace (Monorepo with Independent Subproject Repos)
|
||||
|
|
@ -38,7 +19,7 @@ For comprehensive documentation, see **[docs.pragmatismo.com.br](https://docs.pr
|
|||
|
||||
| Crate | Purpose | Port | Tech Stack |
|
||||
|-------|---------|------|------------|
|
||||
| **botserver** | Main API server, business logic | 9000 | Axum, Diesel, Rhai BASIC |
|
||||
| **botserver** | Main API server, business logic | 8088 | Axum, Diesel, Rhai BASIC |
|
||||
| **botui** | Web UI server (dev) + proxy | 3000 | Axum, HTML/HTMX/CSS |
|
||||
| **botapp** | Desktop app wrapper | - | Tauri 2 |
|
||||
| **botlib** | Shared library | - | Core types, errors |
|
||||
|
|
@ -92,36 +73,15 @@ Place local bot packages in `/opt/gbo/data/` for automatic loading and monitorin
|
|||
|
||||
BotServer automatically installs, configures, and manages all infrastructure components on first run. **DO NOT manually start these services** - BotServer handles everything.
|
||||
|
||||
**Automatic Service Lifecycle:**
|
||||
1. **Start**: When botserver starts, it automatically launches all infrastructure components (PostgreSQL, Vault, MinIO, Valkey, Qdrant, etc.)
|
||||
2. **Credentials**: BotServer retrieves all service credentials (passwords, tokens, API keys) from Vault
|
||||
3. **Connection**: BotServer uses these credentials to establish secure connections to each service
|
||||
4. **Query**: All database queries, cache operations, and storage requests are authenticated using Vault-managed credentials
|
||||
|
||||
**Credential Flow:**
|
||||
```
|
||||
botserver starts
|
||||
↓
|
||||
Launch PostgreSQL, MinIO, Valkey, Qdrant
|
||||
↓
|
||||
Connect to Vault
|
||||
↓
|
||||
Retrieve service credentials (from database)
|
||||
↓
|
||||
Authenticate with each service using retrieved credentials
|
||||
↓
|
||||
Ready to handle requests
|
||||
```
|
||||
|
||||
| Component | Purpose | Port | Binary Location | Credentials From |
|
||||
|-----------|---------|------|-----------------|------------------|
|
||||
| **Vault** | Secrets management | 8200 | `botserver-stack/bin/vault/vault` | Auto-unsealed |
|
||||
| **PostgreSQL** | Primary database | 5432 | `botserver-stack/bin/tables/bin/postgres` | Vault → database |
|
||||
| **MinIO** | Object storage (S3-compatible) | 9000/9001 | `botserver-stack/bin/drive/minio` | Vault → database |
|
||||
| **Zitadel** | Identity/Authentication | 8300 | `botserver-stack/bin/directory/zitadel` | Vault → database |
|
||||
| **Qdrant** | Vector database (embeddings) | 6333 | `botserver-stack/bin/vector_db/qdrant` | Vault → database |
|
||||
| **Valkey** | Cache/Queue (Redis-compatible) | 6379 | `botserver-stack/bin/cache/valkey-server` | Vault → database |
|
||||
| **Llama.cpp** | Local LLM server | 8081 | `botserver-stack/bin/llm/build/bin/llama-server` | Vault → database |
|
||||
| Component | Purpose | Port | Binary Location | Managed By |
|
||||
|-----------|---------|------|-----------------|------------|
|
||||
| **Vault** | Secrets management | 8200 | `botserver-stack/bin/vault/vault` | botserver |
|
||||
| **PostgreSQL** | Primary database | 5432 | `botserver-stack/bin/tables/bin/postgres` | botserver |
|
||||
| **MinIO** | Object storage (S3-compatible) | 9000/9001 | `botserver-stack/bin/drive/minio` | botserver |
|
||||
| **Zitadel** | Identity/Authentication | 8300 | `botserver-stack/bin/directory/zitadel` | botserver |
|
||||
| **Qdrant** | Vector database (embeddings) | 6333 | `botserver-stack/bin/vector_db/qdrant` | botserver |
|
||||
| **Valkey** | Cache/Queue (Redis-compatible) | 6379 | `botserver-stack/bin/cache/valkey-server` | botserver |
|
||||
| **Llama.cpp** | Local LLM server | 8081 | `botserver-stack/bin/llm/build/bin/llama-server` | botserver |
|
||||
|
||||
### 📦 Component Installation System
|
||||
|
||||
|
|
@ -340,12 +300,9 @@ cd botserver-stack/bin/cache && ./valkey-cli ping
|
|||
The script handles BOTH servers properly:
|
||||
1. Stop existing processes cleanly
|
||||
2. Build botserver and botui sequentially (no race conditions)
|
||||
3. Start botserver in background → **automatically starts all infrastructure services (PostgreSQL, Vault, MinIO, Valkey, Qdrant)**
|
||||
4. BotServer retrieves credentials from Vault and authenticates with all services
|
||||
5. Start botui in background → proxy to botserver
|
||||
6. Show process IDs and monitoring commands
|
||||
|
||||
**Infrastructure services are fully automated - no manual configuration required!**
|
||||
3. Start botserver in background → auto-bootstrap infrastructure
|
||||
4. Start botui in background → proxy to botserver
|
||||
5. Show process IDs and monitoring commands
|
||||
|
||||
**Monitor startup:**
|
||||
```bash
|
||||
|
|
@ -354,7 +311,7 @@ tail -f botserver.log botui.log
|
|||
|
||||
**Access:**
|
||||
- Web UI: http://localhost:3000
|
||||
- API: http://localhost:9000
|
||||
- API: http://localhost:8088
|
||||
|
||||
### 📊 Monitor & Debug
|
||||
|
||||
|
|
@ -378,7 +335,7 @@ grep -E " E |W |CLIENT:" botserver.log | tail -20
|
|||
|
||||
```bash
|
||||
cd botserver && cargo run -- --noconsole > ../botserver.log 2>&1 &
|
||||
cd botui && BOTSERVER_URL="http://localhost:9000" cargo run > ../botui.log 2>&1 &
|
||||
cd botui && BOTSERVER_URL="http://localhost:8088" cargo run > ../botui.log 2>&1 &
|
||||
```
|
||||
|
||||
### 🛑 Stop Servers
|
||||
|
|
@ -396,7 +353,7 @@ rm -rf botserver-stack/data/vault botserver-stack/conf/vault/init.json && ./rest
|
|||
|
||||
**Port in use?** Find and kill:
|
||||
```bash
|
||||
lsof -ti:9000 | xargs kill -9
|
||||
lsof -ti:8088 | xargs kill -9
|
||||
lsof -ti:3000 | xargs kill -9
|
||||
```
|
||||
|
||||
|
|
@ -407,16 +364,8 @@ All infrastructure services (PostgreSQL, Vault, Redis, Qdrant, MinIO, etc.) are
|
|||
- **Configurations:** `botserver-stack/conf/`
|
||||
- **Data storage:** `botserver-stack/data/`
|
||||
- **Service logs:** `botserver-stack/logs/` (check here for troubleshooting)
|
||||
- **Credentials:** Stored in Vault, retrieved by botserver at startup
|
||||
|
||||
**Do NOT install or reference global PostgreSQL, Redis, or other services.** When botserver starts, it automatically:
|
||||
1. Launches all required stack services
|
||||
2. Connects to Vault
|
||||
3. Retrieves credentials from the `bot_configuration` database table
|
||||
4. Authenticates with each service using retrieved credentials
|
||||
5. Begins handling requests with authenticated connections
|
||||
|
||||
If you encounter service errors, check the individual service logs in `./botserver-stack/logs/[service]/` directories.
|
||||
**Do NOT install or reference global PostgreSQL, Redis, or other services.** When botserver starts, it automatically launches all required stack services. If you encounter service errors, check the individual service logs in `./botserver-stack/logs/[service]/` directories.
|
||||
|
||||
### UI File Deployment - Production Options
|
||||
|
||||
|
|
@ -473,7 +422,7 @@ See `botserver/deploy/README.md` for deployment scripts.
|
|||
cd botserver && cargo run -- --noconsole
|
||||
|
||||
# Terminal 2: botui
|
||||
cd botui && BOTSERVER_URL="http://localhost:9000" cargo run
|
||||
cd botui && BOTSERVER_URL="http://localhost:8088" cargo run
|
||||
```
|
||||
|
||||
### Build Commands
|
||||
|
|
@ -490,11 +439,292 @@ cargo test -p bottest
|
|||
|
||||
---
|
||||
|
||||
## 🤖 AI Agent Guidelines
|
||||
## 🧭 LLM Navigation Guide
|
||||
|
||||
> **For LLM instructions, coding rules, security directives, testing workflows, and error handling patterns, see [AGENTS.md](./AGENTS.md).**
|
||||
### Quick Context Jump
|
||||
- [Primary Purpose](#overview) - Unified workspace for AI automation platform
|
||||
- [Crate Structure](#-workspace-structure) - 9 independent crates with shared libraries
|
||||
- [Dependencies](#-component-dependency-graph) - How crates depend on each other
|
||||
- [Quick Start](#quick-start) - Get running in 2 commands
|
||||
- [Error Patterns](#common-error-patterns) - Fix compilation errors efficiently
|
||||
- [Security Rules](#-security-directives---mandatory) - MUST-FOLLOW security patterns
|
||||
- [Code Patterns](#-mandatory-code-patterns) - Required coding conventions
|
||||
- [Testing](#testing-strategy) - How to test changes
|
||||
- [Debugging](#debugging-guide) - Troubleshoot common issues
|
||||
|
||||
---
|
||||
### Reading This Workspace
|
||||
|
||||
**For LLMs analyzing this codebase:**
|
||||
1. Start with [Component Dependency Graph](#-component-dependency-graph) to understand relationships
|
||||
2. Review [Module Responsibility Matrix](#-module-responsibility-matrix) for what each module does
|
||||
3. Study [Data Flow Patterns](#-data-flow-patterns) to understand execution flow
|
||||
4. Reference [Common Architectural Patterns](#-common-architectural-patterns) before making changes
|
||||
5. Check [Security Rules](#-security-directives---mandatory) - violations are blocking issues
|
||||
6. Follow [Code Patterns](#-mandatory-code-patterns) - consistency is mandatory
|
||||
|
||||
**For Humans working on this codebase:**
|
||||
1. Follow [Error Fixing Workflow](#-error-fixing-workflow) for compilation errors
|
||||
2. Observe [File Size Limits](#-file-size-limits---mandatory) - max 450 lines per file
|
||||
3. Run [Weekly Maintenance Tasks](#-weekly-maintenance-tasks) to keep codebase healthy
|
||||
4. Read project-specific READMEs in [Project-Specific Guidelines](#-project-specific-guidelines)
|
||||
|
||||
## 🎭 Playwright Browser Testing - YOLO Mode
|
||||
|
||||
### YOLO Mode Instructions for LLMs
|
||||
|
||||
**When user requests to start YOLO mode with Playwright:**
|
||||
|
||||
1. **Start the browser** - Use `mcp__playwright__browser_navigate` to open http://localhost:3000
|
||||
2. **Take snapshot** - Use `mcp__playwright__browser_snapshot` to see current page state
|
||||
3. **Test user flows** - Use click, type, fill_form, etc. to interact with UI
|
||||
4. **Verify results** - Check for expected content, errors in console, network requests
|
||||
5. **Report findings** - Always include screenshot evidence with `browser_take_screenshot`
|
||||
|
||||
**Available Playwright MCP Tools:**
|
||||
- `browser_navigate` - Navigate to URL
|
||||
- `browser_snapshot` - Get accessibility tree (better than screenshots for analysis)
|
||||
- `browser_take_screenshot` - Capture visual state
|
||||
- `browser_click` - Click elements (provide ref from snapshot)
|
||||
- `browser_type` - Type text into inputs
|
||||
- `browser_fill_form` - Fill multiple form fields at once
|
||||
- `browser_console_messages` - Check for JavaScript errors
|
||||
- `browser_network_requests` - Inspect API calls
|
||||
- `browser_close` - Close browser when done
|
||||
|
||||
**YOLO Testing Workflow:**
|
||||
```
|
||||
1. Navigate → http://localhost:3000
|
||||
2. Snapshot → Analyze page structure
|
||||
3. Click → Target element using ref from snapshot
|
||||
4. Wait → For navigation/updates (browser_wait_for)
|
||||
5. Verify → Console messages, network status
|
||||
6. Screenshot → Document test results
|
||||
```
|
||||
|
||||
**Testing Checklist:**
|
||||
- ✅ UI loads without errors
|
||||
- ✅ Navigation works between sections
|
||||
- ✅ Forms submit correctly
|
||||
- ✅ WebSocket connections establish
|
||||
- ✅ Console shows no JavaScript errors
|
||||
- ✅ Network requests return 200/201/204
|
||||
|
||||
**Critical Test Flows:**
|
||||
- **Login/Authentication** → Navigate, enter credentials, verify session
|
||||
- **Bot Creation** → Click "New Bot", fill form, verify creation
|
||||
- **Chat Interface** → Send message, verify WebSocket response
|
||||
- **File Upload** → Upload .bas file, verify compilation
|
||||
- **Drive Sync** → Trigger sync, verify files appear
|
||||
|
||||
**Error Handling in YOLO Mode:**
|
||||
- If navigation fails: Check if servers running (`ps aux | grep botserver`)
|
||||
- If element not found: Take snapshot to debug current page state
|
||||
- If console errors: Extract and report to user for fixing
|
||||
- If network failures: Check API endpoints and CORS configuration
|
||||
|
||||
### Integration Testing
|
||||
|
||||
For automated test suites, prefer `cargo test -p bottest` for backend logic and Playwright YOLO mode for full-stack UI testing.
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **Location**: Each crate has `tests/` directory or inline `#[cfg(test)]` modules
|
||||
- **Naming**: Test functions use `test_` prefix or describe what they test
|
||||
- **Running**: `cargo test -p <crate_name>` or `cargo test` for all
|
||||
|
||||
### Integration Tests
|
||||
- **Location**: `bottest/` crate contains integration tests
|
||||
- **Scope**: Tests full workflows across multiple crates
|
||||
- **Running**: `cargo test -p bottest`
|
||||
- **Database**: Uses test database, automatically migrates on first run
|
||||
|
||||
### Test Utilities Available
|
||||
- **TestAppStateBuilder** (`bottest/src/harness.rs`) - Build test state with mocked components
|
||||
- **TestBot** (`bottest/src/bot/mod.rs`) - Mock bot for testing
|
||||
- **Test Database**: Auto-created, migrations run automatically
|
||||
|
||||
### Coverage Goals
|
||||
- **Critical paths**: 80%+ coverage required
|
||||
- **Error handling**: ALL error paths must have tests
|
||||
- **Security**: All security guards must have tests
|
||||
|
||||
## 🚨 CRITICAL ERROR HANDLING RULE
|
||||
|
||||
**STOP EVERYTHING WHEN ERRORS APPEAR**
|
||||
|
||||
When ANY error appears in logs during startup or operation:
|
||||
1. **IMMEDIATELY STOP** - Do not continue with other tasks
|
||||
2. **IDENTIFY THE ERROR** - Read the full error message and context
|
||||
3. **FIX THE ERROR** - Address the root cause, not symptoms
|
||||
4. **VERIFY THE FIX** - Ensure error is completely resolved
|
||||
5. **ONLY THEN CONTINUE** - Never ignore or work around errors
|
||||
|
||||
**NEVER restart servers to "fix" errors - FIX THE ACTUAL PROBLEM**
|
||||
|
||||
Examples of errors that MUST be fixed immediately:
|
||||
- Database connection errors
|
||||
- Component initialization failures
|
||||
- Service startup errors
|
||||
- Configuration errors
|
||||
- Any error containing "Error:", "Failed:", "Cannot", "Unable"
|
||||
|
||||
## 🐛 Debugging Guide
|
||||
|
||||
### Log Locations
|
||||
|
||||
| Component | Log File | What's Logged |
|
||||
|-----------|----------|---------------|
|
||||
| **botserver** | `botserver.log` | API requests, errors, script execution, **client navigation events** |
|
||||
| **botui** | `botui.log` | UI rendering, WebSocket connections |
|
||||
| **drive_monitor** | In botserver logs with `[drive_monitor]` prefix | File sync, compilation |
|
||||
| **script execution** | In botserver logs with `[ScriptService]` prefix | BASIC compilation, runtime errors |
|
||||
| **client errors** | In botserver logs with `CLIENT:` prefix | JavaScript errors, navigation events |
|
||||
|
||||
### Client-Side Logging
|
||||
|
||||
**Navigation Tracking:** All client-side navigation is logged to botserver.log with `CLIENT:` prefix:
|
||||
```
|
||||
CLIENT:NAVIGATION: click: home -> drive
|
||||
CLIENT:NAVIGATION: hashchange: drive -> chat
|
||||
```
|
||||
|
||||
**Error Reporting:** JavaScript errors automatically appear in server logs:
|
||||
```
|
||||
CLIENT:ERROR: Uncaught TypeError: Cannot read property 'x' of undefined at /suite/js/app.js:123
|
||||
```
|
||||
|
||||
**For LLM Troubleshooting:** ALWAYS check both:
|
||||
1. `botserver.log` - Server errors + client navigation/errors (prefixed with `CLIENT:`)
|
||||
2. `botui.log` - UI server logs
|
||||
|
||||
### USE WEBSITE Feature - Vector DB Context Injection
|
||||
|
||||
**FIXED (v6.2.0+):** The `USE WEBSITE` BASIC command now properly injects vector database embeddings into chat context.
|
||||
|
||||
**How it works:**
|
||||
1. **Preprocessing:** When a `.bas` file containing `USE WEBSITE "https://..."` is compiled, the website is registered for crawling
|
||||
2. **Crawling:** Content is extracted, chunked, and embedded into Qdrant vector DB (collection name: `website_<url_hash>`)
|
||||
3. **Runtime Association:** The compiled `.ast` file contains `USE_WEBSITE()` function call that creates session-website association
|
||||
4. **Context Injection:** During chat, `inject_kb_context()` searches active websites' embeddings and includes relevant chunks in LLM prompt
|
||||
|
||||
**Example BASIC script:**
|
||||
```basic
|
||||
USE WEBSITE "https://docs.pragmatismo.com.br" REFRESH "1h"
|
||||
|
||||
TALK "Hello! I can now answer questions about the documentation."
|
||||
```
|
||||
|
||||
**Database tables involved:**
|
||||
- `session_website_associations` - Links sessions to websites
|
||||
- `website_embeddings` - Stores crawled content vectors in Qdrant
|
||||
|
||||
**Verification:**
|
||||
```sql
|
||||
-- Check if website is associated with session
|
||||
SELECT * FROM session_website_associations WHERE session_id = '<uuid>';
|
||||
|
||||
-- Check if embeddings exist in Qdrant (via HTTP API)
|
||||
curl http://localhost:6333/collections/website_<hash>/points/scroll
|
||||
```
|
||||
|
||||
**Previous Issue:** In earlier versions, `USE WEBSITE` was removed during preprocessing and never executed at runtime, preventing context injection. Now the function call is preserved in the compiled AST.
|
||||
|
||||
### Common Error Messages
|
||||
|
||||
| Error | Meaning | Fix |
|
||||
|-------|---------|-----|
|
||||
| `Session not found` | Invalid session_id in request | Check auth flow, verify session exists in DB |
|
||||
| `Bot not found` | Invalid bot_name or bot_id | Verify bot exists in `bots` table |
|
||||
| `Script compilation error` | BASIC syntax error in .bas file | Check .bas file syntax, look for typos |
|
||||
| `Failed to send TALK message` | WebSocket disconnected | Check client connection, verify web_adapter running |
|
||||
| `Drive sync failed` | S3 connection or permission issue | Verify S3 credentials, check bucket exists |
|
||||
| `unwrap() called on None value` | Panic in production code | MUST FIX - replace with proper error handling |
|
||||
| `Component not responding: <component_name>` | Infrastructure component not accessible | Check component status: `ps aux | grep <component>`. View logs: `tail -f botserver-stack/logs/<component>/`. Restart via ./restart.sh |
|
||||
| `Config key not found: <key>` | Missing configuration in database | Check `bot_configuration` table. Set correct value via API or direct SQL update. |
|
||||
| `403 Forbidden on POST /api/client-errors` | RBAC blocking client error reporting | FIXED in v6.2.0+ - endpoint now allows anonymous access |
|
||||
|
||||
### Useful Debugging Commands
|
||||
|
||||
```bash
|
||||
# Check if botserver is running
|
||||
ps aux | grep botserver
|
||||
|
||||
# View botserver logs in real-time
|
||||
tail -f botserver/logs/botserver.log
|
||||
|
||||
# Check work directory structure
|
||||
ls -la ./work/*.gbai/*/
|
||||
|
||||
# Test database connection
|
||||
cd botserver && cargo run --bin botserver -- --test-db
|
||||
|
||||
# Run specific test with output
|
||||
cargo test -p botserver test_name -- --nocapture
|
||||
|
||||
# Check for memory leaks during compilation
|
||||
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | grep -i error
|
||||
```
|
||||
|
||||
### Troubleshooting Workflows
|
||||
|
||||
**Problem: Script not executing**
|
||||
1. Check if .bas file exists in `./work/{bot_name}.gbai/{bot_name}.gbdialog/`
|
||||
2. Verify file has correct syntax (compile with ScriptService)
|
||||
3. Check logs for compilation errors
|
||||
4. Verify drive_monitor is running and syncing files
|
||||
|
||||
**Problem: WebSocket messages not received**
|
||||
1. Check browser console for WebSocket errors
|
||||
2. Verify session_id is valid in database
|
||||
3. Check web_adapter is registered for session
|
||||
4. Look for TALK execution in botserver logs
|
||||
|
||||
**Problem: Component not starting or crashing**
|
||||
1. Identify the component from error message (e.g., Vault, PostgreSQL, MinIO, Qdrant, Valkey)
|
||||
2. Check if process is running: `ps aux | grep <component_name>`
|
||||
3. Check component logs: `tail -f botserver-stack/logs/<component_name>/`
|
||||
4. Common fixes:
|
||||
- Config error: Check `botserver-stack/conf/<component_name>/` for valid configuration
|
||||
- Port conflict: Ensure no other process using the component's port
|
||||
- Permission error: Check file permissions in `botserver-stack/data/<component_name>/`
|
||||
- Missing binary: Re-run `./reset.sh && ./restart.sh` to reinstall components
|
||||
5. Restart: `./restart.sh`
|
||||
|
||||
**Problem: Component configuration errors**
|
||||
1. All component configs stored in database `bot_configuration` table
|
||||
2. Check current value: `SELECT * FROM bot_configuration WHERE config_key = '<key_name>';`
|
||||
3. Update incorrect config: `UPDATE bot_configuration SET config_value = '<correct_value>' WHERE config_key = '<key_name>';`
|
||||
4. For path configs: Ensure paths are relative to component binary or absolute
|
||||
5. Restart botserver after config changes
|
||||
|
||||
**Problem: File not found errors**
|
||||
1. Check if file exists in expected location
|
||||
2. Verify config paths use correct format (relative/absolute)
|
||||
3. Check file permissions: `ls -la <file_path>`
|
||||
4. For model/data files: Ensure downloaded to `botserver-stack/data/<component>/`
|
||||
|
||||
**Problem: LLM not responding**
|
||||
1. Check LLM API credentials in config
|
||||
2. Verify API key has available quota
|
||||
3. Check network connectivity to LLM provider
|
||||
4. Review request/response logs for API errors
|
||||
|
||||
### Performance Profiling
|
||||
|
||||
```bash
|
||||
# Profile compilation time
|
||||
cargo build --release --timings
|
||||
|
||||
# Profile runtime performance
|
||||
cargo flamegraph --bin botserver
|
||||
|
||||
# Check binary size
|
||||
ls -lh target/release/botserver
|
||||
|
||||
# Memory usage
|
||||
valgrind --leak-check=full target/release/botserver
|
||||
```
|
||||
|
||||
## 📖 Glossary
|
||||
|
||||
|
|
@ -514,7 +744,284 @@ cargo test -p bottest
|
|||
|
||||
---
|
||||
|
||||
## 🔥 Error Fixing Workflow
|
||||
|
||||
### Mode 1: OFFLINE Batch Fix (PREFERRED)
|
||||
|
||||
When given error output:
|
||||
|
||||
```
|
||||
1. Read ENTIRE error list first
|
||||
2. Group errors by file
|
||||
3. For EACH file with errors:
|
||||
a. View file → understand context
|
||||
b. Fix ALL errors in that file
|
||||
c. Write once with all fixes
|
||||
4. Move to next file
|
||||
5. REPEAT until ALL errors addressed
|
||||
6. ONLY THEN → verify with build/diagnostics
|
||||
```
|
||||
|
||||
**NEVER run cargo build/check/clippy DURING fixing**
|
||||
**Fix ALL errors OFFLINE first, verify ONCE at the end**
|
||||
|
||||
### Mode 2: Interactive Loop
|
||||
|
||||
```
|
||||
LOOP UNTIL (0 warnings AND 0 errors):
|
||||
1. Run diagnostics → pick file with issues
|
||||
2. Read entire file
|
||||
3. Fix ALL issues in that file
|
||||
4. Write file once with all fixes
|
||||
5. Verify with diagnostics
|
||||
6. CONTINUE LOOP
|
||||
END LOOP
|
||||
```
|
||||
|
||||
### Common Error Patterns
|
||||
|
||||
| Error | Fix |
|
||||
|-------|-----|
|
||||
| `expected i64, found u64` | `value as i64` |
|
||||
| `expected Option<T>, found T` | `Some(value)` |
|
||||
| `expected T, found Option<T>` | `value.unwrap_or(default)` |
|
||||
| `cannot multiply f32 by f64` | `f64::from(f32_val) * f64_val` |
|
||||
| `no field X on type Y` | Check struct definition |
|
||||
| `no variant X found` | Check enum definition |
|
||||
| `function takes N arguments` | Match function signature |
|
||||
| `cannot find function` | Add missing function or fix import |
|
||||
| `unused variable` | Delete or use with `..` in patterns |
|
||||
| `unused import` | Delete the import line |
|
||||
| `cannot move out of X because borrowed` | Use scoping `{ }` to limit borrow |
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Memory Management
|
||||
|
||||
When compilation fails due to memory issues (process "Killed"):
|
||||
|
||||
```bash
|
||||
pkill -9 cargo; pkill -9 rustc; pkill -9 botserver
|
||||
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📏 File Size Limits - MANDATORY
|
||||
|
||||
### Maximum 450 Lines Per File
|
||||
|
||||
When a file grows beyond this limit:
|
||||
|
||||
1. **Identify logical groups** - Find related functions
|
||||
2. **Create subdirectory module** - e.g., `handlers/`
|
||||
3. **Split by responsibility:**
|
||||
- `types.rs` - Structs, enums, type definitions
|
||||
- `handlers.rs` - HTTP handlers and routes
|
||||
- `operations.rs` - Core business logic
|
||||
- `utils.rs` - Helper functions
|
||||
- `mod.rs` - Re-exports and configuration
|
||||
4. **Keep files focused** - Single responsibility
|
||||
5. **Update mod.rs** - Re-export all public items
|
||||
|
||||
**NEVER let a single file exceed 450 lines - split proactively at 350 lines**
|
||||
|
||||
### Current Files Requiring Immediate Refactoring
|
||||
|
||||
| File | Lines | Target Split |
|
||||
|------|-------|--------------|
|
||||
| `botserver/src/drive/mod.rs` | 1522 | → 4 files |
|
||||
| `botserver/src/auto_task/app_generator.rs` | 2981 | → 7 files |
|
||||
| `botui/ui/suite/sheet/sheet.js` | 3220 | → 8 files |
|
||||
| `botserver/src/tasks/mod.rs` | 2651 | → 6 files |
|
||||
| `botserver/src/learn/mod.rs` | 2306 | → 5 files |
|
||||
|
||||
See `TODO-refactor1.md` for detailed refactoring plans.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Continuous Monitoring
|
||||
|
||||
**YOLO Forever Monitoring Pattern:**
|
||||
|
||||
The system includes automated log monitoring to catch errors in real-time:
|
||||
|
||||
```bash
|
||||
# Continuous monitoring (check every 5 seconds)
|
||||
while true; do
|
||||
sleep 5
|
||||
echo "=== Check at $(date +%H:%M:%S) ==="
|
||||
tail -50 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -5 || echo "✓ Clean"
|
||||
done
|
||||
```
|
||||
|
||||
**Quick Status Check:**
|
||||
```bash
|
||||
# Check last 200 lines for any issues
|
||||
tail -200 botserver.log | grep -E "ERROR|WARN|CLIENT:" | tail -10
|
||||
|
||||
# Show recent server activity
|
||||
tail -30 botserver.log
|
||||
|
||||
# Check if server is running
|
||||
ps aux | grep botserver | grep -v grep
|
||||
```
|
||||
|
||||
**Monitoring Dashboard:**
|
||||
- **Server Status**: https://localhost:8088 (health endpoint)
|
||||
- **Logs**: `tail -f botserver.log`
|
||||
- **Client Errors**: Look for `CLIENT:` prefix
|
||||
- **Server Errors**: Look for `ERROR` or `WARN` prefixes
|
||||
|
||||
**Status Indicators:**
|
||||
- ✅ **Clean**: No ERROR/WARN/CLIENT: entries in logs
|
||||
- ⚠️ **Warnings**: Non-critical issues that should be reviewed
|
||||
- ❌ **Errors**: Critical issues requiring immediate attention
|
||||
|
||||
**When Errors Appear:**
|
||||
1. Capture the full error context (50 lines before/after)
|
||||
2. Identify the component (server, client, database, etc.)
|
||||
3. Check troubleshooting section for specific fixes
|
||||
4. Update this README with discovered issues and resolutions
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Performance & Size Standards
|
||||
|
||||
### Binary Size Optimization
|
||||
- **Release Profile**: Always maintain `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`.
|
||||
- **Dependencies**:
|
||||
- Run `cargo tree --duplicates` weekly to find and resolve duplicate versions.
|
||||
- Run `cargo machete` to remove unused dependencies.
|
||||
- Use `default-features = false` and explicitly opt-in to needed features.
|
||||
|
||||
### Memory Optimization
|
||||
- **Strings**: Prefer `&str` over `String` where possible. Use `Cow<str>` for conditional ownership.
|
||||
- **Collections**: Use `Vec::with_capacity` when size is known. Consider `SmallVec` for hot paths.
|
||||
- **Allocations**: Minimize heap allocations in hot paths.
|
||||
- **Cloning**: Avoid unnecessary `.clone()` calls. Use references or `Cow` types.
|
||||
|
||||
### Code Quality Issues Found
|
||||
- **955 instances** of `unwrap()`/`expect()` in codebase - ALL must be replaced with proper error handling
|
||||
- **12,973 instances** of excessive `clone()`/`to_string()` calls - optimize for performance
|
||||
- **Test code exceptions**: `unwrap()` allowed in test files only
|
||||
|
||||
### Linting & Code Quality
|
||||
- **Clippy**: Code MUST pass `cargo clippy --all-targets --all-features` with **0 warnings**.
|
||||
- **No Allow**: Do not use `#[allow(clippy::...)]` unless absolutely necessary and documented. Fix the underlying issue.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security Directives - MANDATORY
|
||||
|
||||
### Error Handling - NO PANICS IN PRODUCTION
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
value.unwrap()
|
||||
value.expect("message")
|
||||
panic!("error")
|
||||
todo!()
|
||||
unimplemented!()
|
||||
|
||||
// ✅ REQUIRED
|
||||
value?
|
||||
value.ok_or_else(|| Error::NotFound)?
|
||||
value.unwrap_or_default()
|
||||
value.unwrap_or_else(|e| { log::error!("{}", e); default })
|
||||
if let Some(v) = value { ... }
|
||||
match value { Ok(v) => v, Err(e) => return Err(e.into()) }
|
||||
```
|
||||
|
||||
### Command Execution - USE SafeCommand
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
Command::new("some_command").arg(user_input).output()
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::command_guard::SafeCommand;
|
||||
SafeCommand::new("allowed_command")?
|
||||
.arg("safe_arg")?
|
||||
.execute()
|
||||
```
|
||||
|
||||
### Error Responses - USE ErrorSanitizer
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
Json(json!({ "error": e.to_string() }))
|
||||
format!("Database error: {}", e)
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::error_sanitizer::log_and_sanitize;
|
||||
let sanitized = log_and_sanitize(&e, "context", None);
|
||||
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
|
||||
```
|
||||
|
||||
### SQL - USE sql_guard
|
||||
|
||||
```rust
|
||||
// ❌ FORBIDDEN
|
||||
format!("SELECT * FROM {}", user_table)
|
||||
|
||||
// ✅ REQUIRED
|
||||
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
|
||||
let safe_table = sanitize_identifier(&user_table);
|
||||
validate_table_name(&safe_table)?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ❌ Absolute Prohibitions
|
||||
|
||||
```
|
||||
❌ NEVER use .unwrap() or .expect() in production code (tests OK)
|
||||
❌ NEVER use panic!(), todo!(), unimplemented!()
|
||||
❌ NEVER use Command::new() directly - use SafeCommand
|
||||
❌ NEVER return raw error strings to HTTP clients
|
||||
❌ NEVER use #[allow()] in source code - FIX the code instead
|
||||
❌ NEVER add lint exceptions to Cargo.toml - FIX the code instead
|
||||
❌ NEVER use _ prefix for unused variables - DELETE or USE them
|
||||
❌ NEVER leave unused imports or dead code
|
||||
❌ NEVER add comments - code must be self-documenting
|
||||
❌ NEVER modify Cargo.toml lints section!
|
||||
❌ NEVER use CDN links - all assets must be local
|
||||
❌ NEVER use cargo clean - causes 30min rebuilds, use ./reset.sh for database issues
|
||||
❌ NEVER create .md documentation files without checking botbook/ first - documentation belongs there
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Mandatory Code Patterns
|
||||
|
||||
### Use Self in Impl Blocks
|
||||
```rust
|
||||
impl MyStruct {
|
||||
fn new() -> Self { Self { } } // ✅ Not MyStruct
|
||||
}
|
||||
```
|
||||
|
||||
### Derive Eq with PartialEq
|
||||
```rust
|
||||
#[derive(PartialEq, Eq)] // ✅ Always both
|
||||
struct MyStruct { }
|
||||
```
|
||||
|
||||
### Inline Format Args
|
||||
```rust
|
||||
format!("Hello {name}") // ✅ Not format!("{}", name)
|
||||
```
|
||||
|
||||
### Combine Match Arms
|
||||
```rust
|
||||
match x {
|
||||
A | B => do_thing(), // ✅ Combine identical arms
|
||||
C => other(),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ UI Architecture (botui + botserver)
|
||||
|
||||
|
|
@ -523,13 +1030,13 @@ cargo test -p bottest
|
|||
| Server | Port | Purpose |
|
||||
|--------|------|---------|
|
||||
| **botui** | 3000 | Serves UI files + proxies API to botserver |
|
||||
| **botserver** | 9000 | Backend API + embedded UI fallback |
|
||||
| **botserver** | 8088 | Backend API + embedded UI fallback |
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
Browser → localhost:3000 → botui (serves HTML/CSS/JS)
|
||||
→ /api/* proxied to botserver:9000
|
||||
→ /api/* proxied to botserver:8088
|
||||
→ /suite/* served from botui/ui/suite/
|
||||
```
|
||||
|
||||
|
|
@ -684,18 +1191,44 @@ Both repositories must be pushed for changes to take effect in production.
|
|||
|
||||
## Development Workflow
|
||||
|
||||
1. Read this README.md (workspace structure)
|
||||
2. Read **[AGENTS.md](./AGENTS.md)** (coding rules & workflows)
|
||||
3. **BEFORE creating any .md file, search botbook/ for existing documentation**
|
||||
4. Read `<project>/README.md` (project-specific rules)
|
||||
5. Use diagnostics tool to check warnings
|
||||
6. Fix all warnings with full file rewrites
|
||||
7. Verify with diagnostics after each file
|
||||
8. Never suppress warnings with `#[allow()]`
|
||||
1. Read this README.md (workspace-level rules)
|
||||
2. **BEFORE creating any .md file, search botbook/ for existing documentation**
|
||||
3. Read `<project>/README.md` (project-specific rules)
|
||||
4. Use diagnostics tool to check warnings
|
||||
5. Fix all warnings with full file rewrites
|
||||
6. Verify with diagnostics after each file
|
||||
7. Never suppress warnings with `#[allow()]`
|
||||
|
||||
---
|
||||
|
||||
## Main Directive
|
||||
|
||||
**LOOP AND COMPACT UNTIL 0 WARNINGS - MAXIMUM PRECISION**
|
||||
|
||||
- 0 warnings
|
||||
- 0 errors
|
||||
- Trust project diagnostics
|
||||
- Respect all rules
|
||||
- No `#[allow()]` in source code
|
||||
- Real code fixes only
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Remember
|
||||
|
||||
- **OFFLINE FIRST** - Fix all errors from list before compiling
|
||||
- **ZERO WARNINGS, ZERO ERRORS** - The only acceptable state
|
||||
- **FIX, DON'T SUPPRESS** - No #[allow()], no Cargo.toml lint exceptions
|
||||
- **SECURITY FIRST** - No unwrap, no raw errors, no direct commands
|
||||
- **READ BEFORE FIX** - Always understand context first
|
||||
- **BATCH BY FILE** - Fix ALL errors in a file at once
|
||||
- **WRITE ONCE** - Single edit per file with all fixes
|
||||
- **VERIFY LAST** - Only compile/diagnostics after ALL fixes
|
||||
- **DELETE DEAD CODE** - Don't keep unused code around
|
||||
- **Version 6.2.0** - Do not change without approval
|
||||
- **GIT WORKFLOW** - ALWAYS push to ALL repositories (github, pragmatismo)
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
|
|
@ -1,71 +0,0 @@
|
|||
# TASKS.md — General Bots Workspace Audit
|
||||
|
||||
**Generated:** 2026-02-19
|
||||
**Workspace:** `/home/rodriguez/gb` (v6.2.0)
|
||||
**Scope:** Security Audit and Improvements Execution
|
||||
|
||||
---
|
||||
|
||||
## 🔴 P0 — CRITICAL SECURITY FLAWS
|
||||
|
||||
### SEC-01: ✅ RESOLVED — History Clean
|
||||
**Status:** ✅ Repositor history rewritten (git-filter-repo).
|
||||
- [x] `vault-unseal-keys`, `init.json` removed
|
||||
- [x] Remote `origin` force-pushed
|
||||
|
||||
### SEC-02: ✅ PARTIALLY RESOLVED — `.env` exposure
|
||||
**Status:** ✅ Mitigated (Untracked). **Rotation needed.**
|
||||
- [ ] **Rotate Vault tokens immediately**
|
||||
|
||||
### SEC-03: ✅ RESOLVED — `init.json` removed
|
||||
**Status:** ✅ Removed from tracking.
|
||||
|
||||
### SEC-04: ✅ RESOLVED — Command Execution Hardened
|
||||
**Status:** ✅ Replaced `Command::new` with `SafeCommand`.
|
||||
|
||||
### SEC-05: ✅ RESOLVED — SQL Injection Hardened
|
||||
**Status:** ✅ Parameterized queries implemented. Build verified.
|
||||
|
||||
### SEC-06: ✅ RESOLVED — `unwrap()`/`expect()` verified
|
||||
**Status:** ✅ Core/LLM production code verified clean.
|
||||
- [x] `botserver/src/core`: Clean (Unwraps confined to tests/stubs)
|
||||
- [x] `botserver/src/llm`: Clean (Unwraps confined to tests)
|
||||
- [x] Fixed `rate_limiter.rs` (unsafe) & `utils.rs` (expect)
|
||||
|
||||
---
|
||||
|
||||
## 🟠 P1 — HIGH PRIORITY IMPROVEMENTS
|
||||
|
||||
### IMP-03: ✅ RESOLVED — Artifact Cleanup
|
||||
- [x] Deleted `.bas`, `PROMPT.md`
|
||||
- [x] Added `Cargo.lock` to tracking
|
||||
|
||||
### IMP-04: ✅ RESOLVED — Unsafe Code Fix
|
||||
- [x] Replaced `unsafe` block in `rate_limiter.rs`
|
||||
|
||||
### IMP-06: ✅ RESOLVED — CORS Configuration
|
||||
- [x] Fixed syntax and logic in `validate_origin`
|
||||
|
||||
### IMP-14: 🟡 IN PROGRESS — Code Cleanup (TODOs)
|
||||
**Status:** Features partially implemented.
|
||||
- [x] Cleaned stale README references
|
||||
- [x] **IMPLEMENTED `drive_handlers.rs`** (S3 Integration Active)
|
||||
- [ ] Implement `admin_invitations.rs` (Stubbed)
|
||||
- [ ] Remaining minor TODOs
|
||||
|
||||
### IMP-15: 🟡 READY — Integration Tests
|
||||
**Status:** Tool installed (`cargo-tarpaulin` available).
|
||||
- [ ] Generate coverage report (Run `cargo tarpaulin --out Html`)
|
||||
|
||||
---
|
||||
|
||||
## 🟡 P2 — POLICIES (Completed)
|
||||
|
||||
### IMP-07 to IMP-10: ✅ RESOLVED — Policies Added
|
||||
- [x] Rate Limiting, CSRF, Headers, Dependency Management documented in `AGENTS.md`.
|
||||
|
||||
### IMP-16: ✅ RESOLVED — Tool Consolidation
|
||||
- [x] Removed Puppeteer.
|
||||
|
||||
### IMP-17: ✅ RESOLVED — Lockfile
|
||||
- [x] Tracked `Cargo.lock`.
|
||||
2
botapp
2
botapp
|
|
@ -1 +1 @@
|
|||
Subproject commit 66ea6cffbc98b4c10449104806a80d368e3460d4
|
||||
Subproject commit b5ee6e061acf1388aef777ddcd9a2bf84bd6ed57
|
||||
2
botbook
2
botbook
|
|
@ -1 +1 @@
|
|||
Subproject commit cb84ad2b5686cff7cf4ff413144c35231c4c6942
|
||||
Subproject commit 6d48dbba1b21f7fdefd4dfa30d0e33e2879980bb
|
||||
2
botlib
2
botlib
|
|
@ -1 +1 @@
|
|||
Subproject commit 48dd1155ba75c5cf1425b38f1da2aad4cb75e74a
|
||||
Subproject commit 5759fdcd5b528c6f3474cf48f067af4c371e9234
|
||||
|
|
@ -1 +1 @@
|
|||
Subproject commit de017241f2cc8b023dc15e4fbfb7b88e7b3b9dd4
|
||||
Subproject commit c264ad1294a4b62db9ad007c5675b66f48165a30
|
||||
|
|
@ -1 +1 @@
|
|||
Subproject commit b01ee95c7b7d5c8ff69dd2d1bdfe6932762a80c8
|
||||
Subproject commit c2a92c32e105b2dae622a97f0abc844b6264caa9
|
||||
2
bottest
2
bottest
|
|
@ -1 +1 @@
|
|||
Subproject commit a35f70ab3dcc9ee3ccebbc13c09254cab4aa524a
|
||||
Subproject commit 68542cd8ffc1fd58f659a16038981e81b95d145b
|
||||
2
botui
2
botui
|
|
@ -1 +1 @@
|
|||
Subproject commit 4987a15858a09566913e1f81fa43b0bce901529e
|
||||
Subproject commit a8bff4e1a7ff8512e1e2eb5368981d4c0904f41f
|
||||
19
console-errors.txt
Normal file
19
console-errors.txt
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
Total messages: 587 (Errors: 16, Warnings: 4)
|
||||
Returning 16 messages for level "error"
|
||||
|
||||
[ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ http://localhost:3000/public/themes/vapordream.css:0
|
||||
[ERROR] ✗ Failed: 💭 Vapor Dream @ http://localhost:3000/cristo/suite/js/theme-manager.js:85
|
||||
[ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ http://localhost:3000/public/themes/saturdaycartoons.css:0
|
||||
[ERROR] ✗ Failed: 📺 Cartoons @ http://localhost:3000/cristo/suite/js/theme-manager.js:85
|
||||
[ERROR] Failed to load resource: the server responded with a status of 404 (Not Found) @ http://localhost:3000/public/themes/cyberpunk.css:0
|
||||
[ERROR] ✗ Failed: 🌃 Cyberpunk @ http://localhost:3000/cristo/suite/js/theme-manager.js:85
|
||||
[ERROR] WebSocket connection to 'ws://localhost:3000/ws/chat?session_id=f48dd5f0-662b-49bb-a9a6-4039f6e143e7&user_id=9f6c12e7-15b7-4443-bb4a-15b8e17ba076&bot_name=cristo' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED @ :819
|
||||
[ERROR] WebSocket error: Event @ :879
|
||||
[ERROR] WebSocket connection to 'ws://localhost:3000/ws/chat?session_id=f48dd5f0-662b-49bb-a9a6-4039f6e143e7&user_id=9f6c12e7-15b7-4443-bb4a-15b8e17ba076&bot_name=cristo' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED @ :819
|
||||
[ERROR] WebSocket error: Event @ :879
|
||||
[ERROR] WebSocket connection to 'ws://localhost:3000/ws/chat?session_id=f48dd5f0-662b-49bb-a9a6-4039f6e143e7&user_id=9f6c12e7-15b7-4443-bb4a-15b8e17ba076&bot_name=cristo' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED @ :819
|
||||
[ERROR] WebSocket error: Event @ :879
|
||||
[ERROR] WebSocket connection to 'ws://localhost:3000/ws/chat?session_id=f48dd5f0-662b-49bb-a9a6-4039f6e143e7&user_id=9f6c12e7-15b7-4443-bb4a-15b8e17ba076&bot_name=cristo' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED @ :819
|
||||
[ERROR] WebSocket error: Event @ :879
|
||||
[ERROR] WebSocket connection to 'ws://localhost:3000/ws/chat?session_id=f48dd5f0-662b-49bb-a9a6-4039f6e143e7&user_id=9f6c12e7-15b7-4443-bb4a-15b8e17ba076&bot_name=cristo' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED @ :819
|
||||
[ERROR] WebSocket error: Event @ :879
|
||||
7
console-i18n.json
Normal file
7
console-i18n.json
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
Total messages: 36 (Errors: 0, Warnings: 4)
|
||||
Returning 4 messages for level "warning"
|
||||
|
||||
[WARNING] [GBSecurity] NO TOKEN - request will be unauthenticated @ https://chat.pragmatismo.com.br/suite/js/security-bootstrap.js?v=20260207b:157
|
||||
[WARNING] i18n: Missing translation key: chat-mention-title @ https://chat.pragmatismo.com.br/suite/js/i18n.js:129
|
||||
[WARNING] i18n: Missing translation key: chat-mention-title @ https://chat.pragmatismo.com.br/suite/js/i18n.js:129
|
||||
[WARNING] i18n: Missing translation key: chat-mention-title @ https://chat.pragmatismo.com.br/suite/js/i18n.js:129
|
||||
33
console-output-after-fix.txt
Normal file
33
console-output-after-fix.txt
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
Total messages: 31 (Errors: 0, Warnings: 1)
|
||||
|
||||
[LOG] [GBSecurity] HTMX interceptor registered @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:172
|
||||
[LOG] [GBSecurity] Fetch interceptor registered @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:224
|
||||
[LOG] [GBSecurity] XHR interceptor registered @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:269
|
||||
[LOG] [GBSecurity] Security bootstrap initialized @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:125
|
||||
[LOG] [GBSecurity] Current token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:126
|
||||
[LOG] [ErrorReporter] Client-side error reporting initialized @ http://localhost:3000/suite/js/error-reporter.js?v=20260207c:102
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/i18n/en token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/product token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/product token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] [NavigationLogger] Navigation tracking initialized @ http://localhost:3000/suite/js/error-reporter.js?v=20260207c:152
|
||||
[LOG] Initializing HTMX application... @ http://localhost:3000/cristo/suite/js/htmx-app.js:560
|
||||
[LOG] ✓ Theme Manager initialized @ http://localhost:3000/cristo/suite/js/theme-manager.js:130
|
||||
[LOG] HTMX application initialized @ http://localhost:3000/cristo/suite/js/htmx-app.js:580
|
||||
[LOG] 🤖 Bot detected from path: cristo @ http://localhost:3000/cristo/suite/js/suite_app.js:722
|
||||
[LOG] 🚀 Initializing General Bots with HTMX... @ http://localhost:3000/cristo/suite/js/suite_app.js:748
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/bot/config?bot_name=cristo token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] No auth token found - user is signed out @ http://localhost:3000/cristo/suite/js/suite_app.js:1296
|
||||
[LOG] ✓ Theme loaded: ☀️ Light @ http://localhost:3000/cristo/suite/js/theme-manager.js:79
|
||||
[LOG] ✅ Bot 'cristo' is public - authentication not required @ http://localhost:3000/cristo/suite/js/suite_app.js:737
|
||||
[LOG] [GBSecurity] htmx:configRequest for: /suite/chat/chat.html token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:146
|
||||
[WARNING] [GBSecurity] NO TOKEN - request will be unauthenticated @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:157
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/bot/config?bot_name=cristo token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] [GBSecurity] fetch intercepted: /api/auth?bot_name=cristo token: NONE @ http://localhost:3000/suite/js/security-bootstrap.js?v=20260207b:184
|
||||
[LOG] Chat module initialized with @ mentions support @ :1048
|
||||
[LOG] Bot config loaded: {color1: #3b82f6, color2: #f5deb3, title: cristo} @ :936
|
||||
[LOG] Auth: {currentUserId: 042f12c8-7151-4bf3-bd7b-5f8cfd560735, currentSessionId: ecc69ce7-2419-4da7-a149-3c62a1a0af90, currentBotId: 11873f09-9251-4e92-92fa-8de8cabaae7a, currentBotName: cristo} @ :961
|
||||
[LOG] WebSocket connected @ :822
|
||||
[LOG] Chat WebSocket received: {bot_id: 11873f09-9251-4e92-92fa-8de8cabaae7a, message: Connected to bot server, session_id: ecc69ce7-2419-4da7-a149-3c62a1a0af90, type: connected, user_id: 042f12c8-7151-4bf3-bd7b-5f8cfd560735} @ :830
|
||||
[LOG] Chat WebSocket received: {bot_id: 11873f09-9251-4e92-92fa-8de8cabaae7a, user_id: 042f12c8-7151-4bf3-bd7b-5f8cfd560735, session_id: ecc69ce7-2419-4da7-a149-3c62a1a0af90, channel: web, content: Olá! Sou o assistente virtual do Santuário Cristo …sobre celebrações, eventos, visitação ou orações?} @ :830
|
||||
[LOG] Processing bot response: {bot_id: 11873f09-9251-4e92-92fa-8de8cabaae7a, user_id: 042f12c8-7151-4bf3-bd7b-5f8cfd560735, session_id: ecc69ce7-2419-4da7-a149-3c62a1a0af90, channel: web, content: Olá! Sou o assistente virtual do Santuário Cristo …sobre celebrações, eventos, visitação ou orações?} @ :858
|
||||
[LOG] Rendering 10 suggestions @ :707
|
||||
|
|
@ -10,6 +10,7 @@
|
|||
},
|
||||
"scripts": {},
|
||||
"dependencies": {
|
||||
"puppeteer": "^24.37.2",
|
||||
"ws": "^8.19.0"
|
||||
}
|
||||
}
|
||||
2
prod-console-test.json
Normal file
2
prod-console-test.json
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Total messages: 36 (Errors: 0, Warnings: 4)
|
||||
Returning 0 messages for level "error"
|
||||
2
prod-console.json
Normal file
2
prod-console.json
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Total messages: 34 (Errors: 0, Warnings: 4)
|
||||
Returning 0 messages for level "error"
|
||||
5
prod-network.json
Normal file
5
prod-network.json
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
[GET] https://chat.pragmatismo.com.br/api/product => [200]
|
||||
[GET] https://chat.pragmatismo.com.br/api/product => [200]
|
||||
[GET] https://chat.pragmatismo.com.br/suite/chat/chat.html => [200]
|
||||
[GET] https://chat.pragmatismo.com.br/api/bot/config?bot_name=default => [200]
|
||||
[GET] https://chat.pragmatismo.com.br/api/auth?bot_name=default => [200]
|
||||
57
restart.sh
57
restart.sh
|
|
@ -1,24 +1,59 @@
|
|||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "Stopping..."
|
||||
pkill -f botserver || true
|
||||
echo "🛑 Stopping existing processes..."
|
||||
pkill -f "botserver --noconsole" || true
|
||||
pkill -f botui || true
|
||||
pkill -f rustc || true
|
||||
# Note: PostgreSQL, Vault, and Valkey are managed by botserver bootstrap, don't kill them
|
||||
|
||||
echo "Cleaning..."
|
||||
echo "🧹 Cleaning logs..."
|
||||
rm -f botserver.log botui.log
|
||||
|
||||
echo "Building..."
|
||||
echo "🔨 Building botserver..."
|
||||
cargo build -p botserver
|
||||
|
||||
echo "🔨 Building botui..."
|
||||
cargo build -p botui
|
||||
|
||||
echo "Starting botserver..."
|
||||
./target/debug/botserver --noconsole > botserver.log 2>&1 &
|
||||
echo " PID: $!"
|
||||
echo "🗄️ Starting PostgreSQL..."
|
||||
./botserver-stack/bin/tables/bin/postgres -D botserver-stack/data/tables/pgdata -c config_file=botserver-stack/conf/postgresql.conf > botserver-stack/logs/tables/postgres.log 2>&1 &
|
||||
echo " PostgreSQL PID: $!"
|
||||
sleep 2
|
||||
|
||||
echo "Starting botui..."
|
||||
BOTSERVER_URL="http://localhost:9000" ./target/debug/botui > botui.log 2>&1 &
|
||||
echo " PID: $!"
|
||||
echo "🔑 Starting Valkey (cache)..."
|
||||
./botserver-stack/bin/cache/valkey-server --daemonize no --dir botserver-stack/data/cache > /dev/null 2>&1 &
|
||||
echo " Valkey started"
|
||||
sleep 2
|
||||
|
||||
echo "Done. Logs: tail -f botserver.log botui.log"
|
||||
echo "🚀 Starting botserver..."
|
||||
export VAULT_ADDR="https://localhost:8200"
|
||||
export VAULT_TOKEN="hvs.JjKHlEzycO2jvKdhhlRAoODu"
|
||||
export VAULT_CACERT="./botserver-stack/conf/system/certificates/ca/ca.crt"
|
||||
export VAULT_CACHE_TTL="300"
|
||||
RUST_LOG=info ./target/debug/botserver --noconsole > botserver.log 2>&1 &
|
||||
BOTSERVER_PID=$!
|
||||
|
||||
echo "⏳ Waiting for Vault to start (unsealing in background)..."
|
||||
(
|
||||
sleep 8
|
||||
echo "🔓 Unsealing Vault..."
|
||||
UNSEAL_KEY=$(python3 -c "import json; print(json.load(open('botserver-stack/conf/vault/init.json'))['unseal_keys_b64'][0])" 2>/dev/null)
|
||||
if [ -n "$UNSEAL_KEY" ]; then
|
||||
curl -s --cacert botserver-stack/conf/system/certificates/ca/ca.crt \
|
||||
-X POST \
|
||||
-H "X-Vault-Token: hvs.JjKHlEzycO2jvKdhhlRAoODu" \
|
||||
-d "{\"key\": \"$UNSEAL_KEY\"}" \
|
||||
https://localhost:8200/v1/sys/unseal 2>/dev/null && echo "✅ Vault unsealed" || echo "⚠️ Unseal failed"
|
||||
else
|
||||
echo "⚠️ Could not extract unseal key"
|
||||
fi
|
||||
) &
|
||||
|
||||
echo "🚀 Starting botui..."
|
||||
BOTSERVER_URL="http://localhost:8080" ./target/debug/botui > botui.log 2>&1 &
|
||||
BOTUI_PID=$!
|
||||
|
||||
echo "✅ Started botserver (PID: $BOTSERVER_PID) and botui (PID: $BOTUI_PID)"
|
||||
echo "📊 Monitor with: tail -f botserver.log botui.log"
|
||||
echo "🌐 Access at: http://localhost:3000"
|
||||
|
|
|
|||
26
start-and-unseal.sh
Executable file
26
start-and-unseal.sh
Executable file
|
|
@ -0,0 +1,26 @@
|
|||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "🔓 Unsealing Vault..."
|
||||
UNSEAL_KEY="$(cat botserver-stack/conf/vault/init.json | grep -o '"unseal_keys_b64":\["[^"]*"' | cut -d'"' -f4)"
|
||||
|
||||
# Wait for Vault to start
|
||||
for i in {1..30}; do
|
||||
if curl -sfk --cacert botserver-stack/conf/system/certificates/ca/ca.crt \
|
||||
https://localhost:8200/v1/sys/health > /dev/null 2>&1; then
|
||||
echo "✅ Vault is running"
|
||||
break
|
||||
fi
|
||||
echo "⏳ Waiting for Vault... ($i/30)"
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Unseal Vault
|
||||
echo "🔓 Unsealing..."
|
||||
curl -s --cacert botserver-stack/conf/system/certificates/ca/ca.crt \
|
||||
-X POST \
|
||||
-H "X-Vault-Token: hvs.JjKHlEzycO2jvKdhhlRAoODu" \
|
||||
-d "{\"key\": \"$UNSEAL_KEY\"}" \
|
||||
https://localhost:8200/v1/sys/unseal
|
||||
|
||||
echo "✅ Vault unsealed"
|
||||
55
tasks.md
55
tasks.md
|
|
@ -1,55 +0,0 @@
|
|||
# Security Audit & Remediation Tasks
|
||||
|
||||
## Executive Summary
|
||||
A security review has been performed on the Rust codebase (server, logic, and core libraries). Earlier concerns regarding the Dynamic Table API acting as an explicit vulnerability have been retracted, as the behavior corresponds to intentional system design relying on role-based access scoping. However, significant vulnerabilities were identified regarding execution isolation and network requests originating from the bot engine.
|
||||
|
||||
**Actions Taken:** Both identified active vulnerabilities have been corrected directly in the codebase.
|
||||
|
||||
---
|
||||
|
||||
## 1. [FIXED] Remote Code Execution (RCE) via `trusted_shell_script_arg` Command Injection
|
||||
**Status:** Remediated in `botserver/src/llm/local.rs`
|
||||
**Severity:** CRITICAL
|
||||
|
||||
**Description:**
|
||||
The codebase has a custom wrapper `SafeCommand` in `botserver/src/security/command_guard.rs`, which is designed to prevent shell injections. However, `trusted_shell_script_arg` acts as a bypass to this safety check, accepting arbitrary shell strings. In `botserver/src/llm/local.rs`, `trusted_shell_script_arg` had been executing commands during Llama.cpp server startup by embedding database configuration values (like `llm_server_path`, `n_moe`) directly inside a `sh -c` shell string. If those configuration variables contained shell control operators (like `;`, `&`, `|`), an attacker modifying those configs could achieve arbitrary remote code execution on the host operating system.
|
||||
|
||||
**Remediation Applied:**
|
||||
- Replaced the vulnerable shell orchestrations with pure, safe bindings using `std::process::Command::spawn()`.
|
||||
- Arguments mapped exclusively via explicit positional `.arg()` blocks rather than shell interpolation, removing the possibility of execution breakout.
|
||||
|
||||
---
|
||||
|
||||
## 2. [FIXED] Server-Side Request Forgery (SSRF) in Rhai Execution `GET` requests
|
||||
**Status:** Remediated in `botserver/src/basic/keywords/get.rs`
|
||||
**Severity:** HIGH
|
||||
|
||||
**Description:**
|
||||
In `botserver/src/basic/keywords/get.rs`, Rhai scripts have a `GET` command that wraps an HTTP client request. It used `is_safe_path` to allegedly prevent abuse. Unfortunately, `is_safe_path` implicitly permitted internal routing by returning `true` whenever the user-provided protocol was `http://` or `https://` without checking the corresponding host IP addresses.
|
||||
If a user supplied `http://169.254.169.254/latest/meta-data/` or `http://10.0.0.1/admin`, the system evaluated it. This was an open SSRF allowing attackers to scan internal corporate networks, pivot to internal APIs, and steal Cloud Provider IAM Metadata / Instance Profiles.
|
||||
|
||||
**Remediation Applied:**
|
||||
- Introduced `url` crate hostname parsing inside the `is_safe_path` check.
|
||||
- Added strict evaluation to block the `localhost`, link-local instances (`169.254.x.x`), internal Subnets (`10.x`, `172.16-31.x`, `192.168.x`), and standard metadata API FQDN lookups (like `metadata.google.internal`).
|
||||
|
||||
---
|
||||
|
||||
## 3. [MEDIUM] Lax CSRF and Cookie Security
|
||||
**Status:** Pending Review
|
||||
|
||||
**Description:**
|
||||
Upon examining headers in `botserver/src/weba/mod.rs` and other API segments, it appears `SameSite` policies may not be strictly enforced for all configurations, and TLS/Cert Pinning is hand-rolled (`botserver/src/security/cert_pinning.rs`).
|
||||
|
||||
**Recommended Action:**
|
||||
- Ensure all session cookies have `HttpOnly`, `Secure`, and `SameSite=Lax` (or `Strict`).
|
||||
- Run `cargo audit` to handle dependency vulnerabilities in any manual implementations of TLS processing.
|
||||
|
||||
---
|
||||
|
||||
## Informational Notes: Dynamic Database Table Operations
|
||||
|
||||
**Observation:**
|
||||
The `is_table_allowed_with_conn` allows requests targeting general bot administration records, rendering a wide subset of internal tables addressable over the `/api/v1/db/{table_name}` system endpoints when `dynamic_table_definitions` rules don't exist.
|
||||
|
||||
**Conclusion:**
|
||||
This behavior operates **by design** to supply a generic API approach where bots expose privilege data based directly on Bot Identity restrictions and internal User Object role bindings across standard API interactions. This was deemed safe and required for generic bot privilege allocations.
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
DESCRIPTION "Test BEGIN TALK and BEGIN MAIL preprocessing"
|
||||
|
||||
' Test BEGIN TALK block
|
||||
BEGIN TALK
|
||||
This is line 1 with ${variable}
|
||||
This is line 2 with ${anotherVariable}
|
||||
END TALK
|
||||
|
||||
' Test BEGIN MAIL block
|
||||
BEGIN MAIL test@example.com
|
||||
Subject: Test Email Subject
|
||||
|
||||
This is the body line 1
|
||||
This is the body line 2 with ${data}
|
||||
END MAIL
|
||||
|
||||
TALK "Test complete"
|
||||
5
vault-unseal-keys
Normal file
5
vault-unseal-keys
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
Unseal Key 1: JDWcTlskyqgKO5dNghRZyVCrdvH9MsJsfmS/nQhpfUyl
|
||||
Unseal Key 2: UuCsxt0MdaKm0z9ki7oHdLxu7CW47yBnUSrQ2/RiafnT
|
||||
Unseal Key 3: 5g9WTB3Yg5iwEodnaFM8fUH7DzqGfKMO7Wj8rO6YuJdO
|
||||
Unseal Key 4: Y5UQzF1PjgeJZ7/3J6fyvTkrNM2W6+hXCyk/5iyOp+0e
|
||||
Unseal Key 5: f6lFx6B4+RJeKdz3B9r/YNKaTdolenU7PuixDwXMXeJ/
|
||||
Loading…
Add table
Reference in a new issue