GeneralBots/botbook
2
2
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Update SUMMARY and add cargo-tools documentation

Browse source
This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2025-12-07 09:57:27 -03:00
parent b54d8e84dd
commit 7123f881c6
3 changed files with 596 additions and 243 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

242
TASKS.md
View file

@ -1,242 +0,0 @@
# Documentation Tasks & Discrepancies Report
**Generated:** 2025-01
**Status:** Active development
> **Summary**: This report tracks discrepancies between documentation and implementation in the General Bots codebase. Key fixes applied: keywords now use spaces (not underscores), ON ERROR RESUME NEXT implemented, DELETE unified.
---
## ✅ COMPLETED
### Stub Files Now Have Content
- [x] `keyword-soap.md` - SOAP web service integration
- [x] `keyword-merge-pdf.md` - PDF merging (now `MERGE PDF` with space)
- [x] `keyword-kb-statistics.md` - Overall KB statistics
- [x] `keyword-kb-collection-stats.md` - Per-collection statistics
- [x] `keyword-kb-documents-count.md` - Document count
- [x] `keyword-kb-documents-added-since.md` - Recent document tracking
- [x] `keyword-kb-list-collections.md` - Collection listing
- [x] `keyword-kb-storage-size.md` - Storage monitoring
### Code Changes Applied
- [x] `SEND MAIL` - Changed from `SEND_MAIL` to space-separated
- [x] `GENERATE PDF` - Changed from `GENERATE_PDF` to space-separated
- [x] `MERGE PDF` - Changed from `MERGE_PDF` to space-separated
- [x] `ON ERROR RESUME NEXT` - Implemented in `errors/on_error.rs`
- [x] `DELETE` - Unified keyword (auto-detects HTTP/DB/File)
- [x] `PROMPT.md` - Updated with keyword naming rules
---
## ✅ COMPLETED: Model Name Updates
Model names updated to current versions (2025-01):
| File | Updated To |
|------|------------|
| `appendix-external-services/README.md` | `claude-sonnet-4.5` |
| `appendix-external-services/hosting-dns.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` |
| `appendix-external-services/llm-providers.md` | `DeepSeek-R3, Claude Sonnet 4.5` |
| `chapter-04-gbui/how-to/create-first-bot.md` | `claude-sonnet-4.5` |
| `chapter-04-gbui/how-to/monitor-sessions.md` | `LLM active` (generic) |
| `chapter-04-gbui/suite-manual.md` | `Claude Sonnet 4.5`, `Claude Opus 4.5`, `Gemini Pro`, `Llama 3.3` |
| `chapter-06-gbdialog/keyword-model-route.md` | `claude-sonnet-4.5`, `gemini-flash`, `claude-opus-4.5` |
| `chapter-06-gbdialog/basic-vs-automation-tools.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` |
| `chapter-07-gbapp/architecture.md` | `GPT-5 and o3`, `Claude Sonnet 4.5 and Opus 4.5` |
| `chapter-08-config/llm-config.md` | `claude-sonnet-4.5` |
| `chapter-08-config/secrets-management.md` | `claude-sonnet-4.5` |
| `chapter-11-features/ai-llm.md` | `GPT-5`, `o3`, `Claude Sonnet 4.5` |
| `chapter-11-features/core-features.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` |
| `executive-vision.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` |
### Naming Convention Applied
- OpenAI: `GPT-5`, `GPT-5 mini`, `o3`
- Anthropic: `Claude Sonnet 4.5`, `Claude Opus 4.5`
- Google: `Gemini Pro`, `Gemini Flash`
- Meta: `Llama 3.3`
- DeepSeek: `DeepSeek-V3`, `DeepSeek-R3`
- Local: `model.gguf`, `local-model`
---
## ✅ RESOLVED: Keyword Syntax Clarifications
### Keywords Now Use Spaces (NOT Underscores)
| ✅ Correct | ❌ Wrong |
|-----------|----------|
| `SEND MAIL` | `SEND_MAIL` |
| `GENERATE PDF` | `GENERATE_PDF` |
| `MERGE PDF` | `MERGE_PDF` |
| `SET HEADER` | `SET_HEADER` |
| `CLEAR HEADERS` | `CLEAR_HEADERS` |
| `ON ERROR RESUME NEXT` | N/A |
| `KB STATISTICS` | `KB_STATISTICS` |
### Keyword Mappings (Documentation Aliases)
| Documented Pattern | Actual Keyword |
|-------------------|----------------|
| `GENERATE FROM TEMPLATE file WITH data` | `FILL data, template` |
| `GENERATE WITH PROMPT prompt` | `LLM prompt` |
| `DELETE HTTP url` | `DELETE url` (unified) |
| `DELETE FILE path` | `DELETE path` (unified) |
| `SEND EMAIL TO addr WITH body` | `SEND MAIL addr, subject, body, []` |
### Unified DELETE Keyword
The `DELETE` keyword now auto-detects context:
```basic
' HTTP DELETE - detects URL
DELETE "https://api.example.com/resource/123"
' Database DELETE - table + filter
DELETE "customers", "status = 'inactive'"
' File DELETE - path without URL prefix
DELETE "temp/report.pdf"
```
---
## ✅ IMPLEMENTED: Error Handling
### ON ERROR RESUME NEXT (BASIC-style)
Now fully implemented in `src/basic/keywords/errors/on_error.rs`:
```basic
ON ERROR RESUME NEXT ' Enable error trapping
result = RISKY_OPERATION()
IF ERROR THEN
TALK "Error: " + ERROR MESSAGE
END IF
ON ERROR GOTO 0 ' Disable error trapping
CLEAR ERROR ' Clear error state
```
### Available Error Keywords
| Keyword | Description |
|---------|-------------|
| `ON ERROR RESUME NEXT` | Enable error trapping |
| `ON ERROR GOTO 0` | Disable error trapping |
| `ERROR` | Returns true if error occurred |
| `ERROR MESSAGE` | Get last error message |
| `ERR` | Get error number |
| `CLEAR ERROR` | Clear error state |
| `THROW "msg"` | Raise an error |
| `ASSERT cond, "msg"` | Assert condition |
---
## 🟡 CONFIG PARAMETERS
### Verified in Source Code ✅
```
llm-key, llm-url, llm-model
llm-cache, llm-cache-ttl, llm-cache-semantic, llm-cache-threshold
llm-server, llm-server-* (all server params)
embedding-url, embedding-model
email-from, email-server, email-port, email-user, email-pass
custom-server, custom-port, custom-database, custom-username, custom-password
image-generator-*, video-generator-*
botmodels-enabled, botmodels-host, botmodels-port
episodic-memory-threshold, episodic-memory-history
oauth-*-enabled, oauth-*-client-id, oauth-*-client-secret
website-max-depth, website-max-pages
```
### In Code But Missing from Default config.csv
| Parameter | Used In |
|-----------|---------|
| `llm-temperature` | console/status_panel.rs |
| `llm-server-reasoning-format` | llm/local.rs |
| `email-read-pixel` | email/mod.rs |
| `server-url` | email/mod.rs |
| `teams-app-id`, `teams-*` | core/bot/channels/teams.rs |
| `whatsapp-api-key`, `whatsapp-*` | core/bot/channels/whatsapp.rs |
| `twilio-*`, `aws-*`, `vonage-*` | basic/keywords/sms.rs |
| `qdrant-url` | vector database connection |
### In config.csv But Status Unknown
| Parameter | Status |
|-----------|--------|
| `website-expires` | ❓ Not found in search |
| `default-generator` | ❓ Not found in search |
---
## 📋 REMAINING TASKS
### Priority 1 - Documentation Updates
- [x] Update all model names to generic/current versions (17 files)
- [x] Update keyword examples to use spaces not underscores
- [x] Fix `SEND EMAIL` examples to use `SEND MAIL` syntax
- [x] Document unified `DELETE` keyword behavior (already documented in `keyword-delete.md`)
- [x] Add `ON ERROR RESUME NEXT` to error handling docs (created `keyword-on-error.md`)
### Priority 2 - New Documentation Needed
- [x] Full config.csv parameter reference (expanded `parameters.md` with all SMS, WhatsApp params)
- [x] SMS provider configuration (created `sms-providers.md` - Twilio, AWS SNS, Vonage, MessageBird)
- [x] Teams/WhatsApp channel setup (created `teams-channel.md` and `whatsapp-channel.md`)
- [x] SOAP authentication configuration (expanded `keyword-soap.md` with WS-Security, OAuth, certs)
### Priority 3 - Structure Cleanup
- [x] Renamed folders with proper numbering (01-introduction, 02-templates, etc.)
- [x] Removed VB/Visual Basic references (now BASIC only)
- [x] Deleted empty `04-ui-legacy` folder
- [x] Deleted `17-appendix-env-vars` (only VAULT_* vars are real, rest was lies)
- [x] Fixed vbs code blocks to use `basic` syntax
- [x] Updated SUMMARY.md with new folder paths
---
## 📊 Quick Reference
### Keyword Implementation Files
| Category | Source File |
|----------|-------------|
| Email | `send_mail.rs` → `SEND MAIL`, `SEND TEMPLATE` |
| PDF | `file_operations.rs` → `GENERATE PDF`, `MERGE PDF` |
| Data | `data_operations.rs` → `DELETE`, `INSERT`, `UPDATE`, `FILL` |
| HTTP | `http_operations.rs` → `POST`, `PUT`, `GRAPHQL`, `SOAP` |
| Errors | `errors/on_error.rs` → `ON ERROR RESUME NEXT` |
| KB Stats | `kb_statistics.rs` → `KB STATISTICS`, etc. |
| LLM | `llm_keyword.rs` → `LLM` |
### Summary Statistics
| Category | Count |
|----------|-------|
| Files with old model names | 0 (fixed) |
| Stub files completed | 8 |
| Keywords fixed (underscore→space) | 10 |
| SEND EMAIL → SEND MAIL | 9 files fixed |
| Error handling keywords added | 8 |
| Error handling doc created | `keyword-on-error.md` |
| Config params documented | 50+ |
| New channel docs created | `teams-channel.md`, `whatsapp-channel.md` |
| New provider docs created | `sms-providers.md` |
| SOAP auth expanded | WS-Security, OAuth, certificates |
| VB references removed | All files now use BASIC only |
| Folders renamed | `01-introduction`, `02-templates`, etc. |
| Deleted trash | `04-ui-legacy`, `17-appendix-env-vars` |
---
## Architecture Notes
- **Language**: Rust with Rhai scripting engine
- **Keywords**: Registered via `register_custom_syntax()` in Rhai
- **Preprocessing**: `basic/compiler/mod.rs` normalizes some syntax
- **Config**: CSV files parsed by `ConfigManager`
- **Vector DB**: Qdrant
- **Relational DB**: PostgreSQL
- **File Storage**: MinIO/SeaweedFS

594
src/07-gbapp/cargo-tools.md Normal file
View file

@ -0,0 +1,594 @@
# Cargo Tools Reference
This chapter documents essential Cargo tools for BotServer development, including code coverage, security auditing, performance profiling, and code quality tools.
## Overview
The Rust ecosystem provides powerful tools through Cargo extensions. These tools help maintain code quality, identify security vulnerabilities, measure test coverage, and optimize performance.
## Code Coverage with cargo-tarpaulin
### Installation
```bash
cargo install cargo-tarpaulin
```
### Basic Usage
Run code coverage analysis:
```bash
cargo tarpaulin
```
This generates a coverage report showing which lines of code are exercised by tests.
### Output Formats
Generate HTML report:
```bash
cargo tarpaulin --out Html
```
Generate multiple formats (coverage report, lcov for CI):
```bash
cargo tarpaulin --out Html --out Lcov --out Json
```
### Coverage with Features
Test with specific features enabled:
```bash
cargo tarpaulin --features vectordb,email
```
Test all features:
```bash
cargo tarpaulin --all-features
```
### Excluding Files
Exclude test files and generated code from coverage:
```bash
cargo tarpaulin --ignore-tests --exclude-files "gen/*" "tests/*"
```
### Coverage Thresholds
Fail if coverage drops below a threshold (useful for CI):
```bash
cargo tarpaulin --fail-under 80
```
### Verbose Output
Show detailed coverage per function:
```bash
cargo tarpaulin --verbose
```
### Integration with CI
Example GitHub Actions workflow:
```yaml
- name: Install tarpaulin
run: cargo install cargo-tarpaulin
- name: Generate coverage
run: cargo tarpaulin --out Xml --fail-under 70
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
files: cobertura.xml
```
### Configuration File
Create `.tarpaulin.toml` for project-wide settings:
```toml
[config]
command = "test"
features = "vectordb"
ignore-tests = true
out = ["Html", "Lcov"]
exclude-files = ["gen/*"]
timeout = "120s"
```
## Security Auditing with cargo-audit
### Installation
```bash
cargo install cargo-audit
```
### Basic Usage
Check for known security vulnerabilities in dependencies:
```bash
cargo audit
```
### Continuous Auditing
Run audit as part of CI pipeline to catch new vulnerabilities:
```bash
cargo audit --deny warnings
```
### Fix Vulnerabilities
Generate a fix for vulnerable dependencies (when possible):
```bash
cargo audit fix
```
### Database Updates
Update the vulnerability database:
```bash
cargo audit fetch
```
### Ignore Known Issues
Create `.cargo/audit.toml` to ignore specific advisories:
```toml
[advisories]
ignore = [
"RUSTSEC-2020-0071", # Reason for ignoring
]
```
### JSON Output for CI
```bash
cargo audit --json > audit-report.json
```
## Dependency Analysis with cargo-deny
### Installation
```bash
cargo install cargo-deny
```
### Configuration
Create `deny.toml`:
```toml
[advisories]
vulnerability = "deny"
unmaintained = "warn"
[licenses]
unlicensed = "deny"
allow = [
"MIT",
"Apache-2.0",
"BSD-3-Clause",
]
[bans]
multiple-versions = "warn"
deny = [
{ name = "openssl" }, # Prefer rustls
]
[sources]
unknown-registry = "deny"
unknown-git = "deny"
```
### Usage
Check all configured rules:
```bash
cargo deny check
```
Check specific categories:
```bash
cargo deny check advisories
cargo deny check licenses
cargo deny check bans
cargo deny check sources
```
## Code Formatting with cargo-fmt
### Usage
Format all code:
```bash
cargo fmt
```
Check formatting without changes:
```bash
cargo fmt --check
```
### Configuration
Create `rustfmt.toml`:
```toml
edition = "2021"
max_width = 100
tab_spaces = 4
use_small_heuristics = "Default"
reorder_imports = true
group_imports = "StdExternalCrate"
```
## Linting with cargo-clippy
### Usage
Run clippy with warnings as errors:
```bash
cargo clippy -- -D warnings
```
Run with all lints:
```bash
cargo clippy -- -W clippy::all -W clippy::pedantic
```
### Fix Suggestions Automatically
```bash
cargo clippy --fix --allow-dirty
```
### Configuration
Add to `Cargo.toml`:
```toml
[lints.clippy]
all = "warn"
pedantic = "warn"
unwrap_used = "deny"
expect_used = "deny"
```
Or create `.clippy.toml`:
```toml
avoid-breaking-exported-api = false
msrv = "1.70"
```
## Documentation with cargo-doc
### Generate Documentation
```bash
cargo doc --open
```
With private items:
```bash
cargo doc --document-private-items --open
```
### Check Documentation
Find broken links and missing docs:
```bash
cargo rustdoc -- -D warnings
```
## Benchmarking with cargo-criterion
### Installation
```bash
cargo install cargo-criterion
```
### Usage
Run benchmarks:
```bash
cargo criterion
```
### Benchmark Example
Create `benches/my_benchmark.rs`:
```rust
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn fibonacci(n: u64) -> u64 {
match n {
0 => 1,
1 => 1,
n => fibonacci(n-1) + fibonacci(n-2),
}
}
fn benchmark(c: &mut Criterion) {
c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}
criterion_group!(benches, benchmark);
criterion_main!(benches);
```
## Dependency Management with cargo-outdated
### Installation
```bash
cargo install cargo-outdated
```
### Usage
Check for outdated dependencies:
```bash
cargo outdated
```
Show only root dependencies:
```bash
cargo outdated --root-deps-only
```
## Binary Size Analysis with cargo-bloat
### Installation
```bash
cargo install cargo-bloat
```
### Usage
Show largest functions:
```bash
cargo bloat --release -n 20
```
Show largest crates:
```bash
cargo bloat --release --crates
```
### Size Comparison
Compare sizes between releases:
```bash
cargo bloat --release --crates > before.txt
# Make changes
cargo bloat --release --crates > after.txt
diff before.txt after.txt
```
## Dependency Tree with cargo-tree
### Usage
View full dependency tree:
```bash
cargo tree
```
Find duplicate dependencies:
```bash
cargo tree --duplicates
```
Find why a dependency is included:
```bash
cargo tree --invert tokio
```
## Watch Mode with cargo-watch
### Installation
```bash
cargo install cargo-watch
```
### Usage
Auto-rebuild on changes:
```bash
cargo watch -x build
```
Auto-test on changes:
```bash
cargo watch -x test
```
Run multiple commands:
```bash
cargo watch -x check -x test -x clippy
```
## Memory Profiling with cargo-valgrind
### Installation (Linux)
```bash
sudo apt install valgrind
cargo install cargo-valgrind
```
### Usage
```bash
cargo valgrind run
```
## LLVM Coverage with cargo-llvm-cov
### Installation
```bash
cargo install cargo-llvm-cov
```
### Usage
More accurate coverage than tarpaulin for some cases:
```bash
cargo llvm-cov
```
Generate HTML report:
```bash
cargo llvm-cov --html
```
## Recommended CI Pipeline
Example complete CI configuration using these tools:
```yaml
name: CI
on: [push, pull_request]
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- name: Format check
run: cargo fmt --check
- name: Clippy
run: cargo clippy -- -D warnings
- name: Build
run: cargo build --release
- name: Test
run: cargo test
coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- name: Install tarpaulin
run: cargo install cargo-tarpaulin
- name: Coverage
run: cargo tarpaulin --out Xml --fail-under 70
- uses: codecov/codecov-action@v3
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- name: Install audit
run: cargo install cargo-audit
- name: Security audit
run: cargo audit --deny warnings
```
## Quick Reference
| Tool | Purpose | Command |
|------|---------|---------|
| cargo-tarpaulin | Code coverage | `cargo tarpaulin` |
| cargo-audit | Security vulnerabilities | `cargo audit` |
| cargo-deny | License/dependency rules | `cargo deny check` |
| cargo-fmt | Code formatting | `cargo fmt` |
| cargo-clippy | Linting | `cargo clippy` |
| cargo-doc | Documentation | `cargo doc --open` |
| cargo-criterion | Benchmarking | `cargo criterion` |
| cargo-outdated | Outdated dependencies | `cargo outdated` |
| cargo-bloat | Binary size analysis | `cargo bloat --release` |
| cargo-tree | Dependency tree | `cargo tree` |
| cargo-watch | Auto-rebuild | `cargo watch -x build` |
| cargo-llvm-cov | LLVM coverage | `cargo llvm-cov` |
## Installation Script
Install all recommended tools at once:
```bash
#!/bin/bash
# install-cargo-tools.sh
cargo install cargo-tarpaulin
cargo install cargo-audit
cargo install cargo-deny
cargo install cargo-outdated
cargo install cargo-bloat
cargo install cargo-watch
cargo install cargo-criterion
cargo install cargo-llvm-cov
```
## Next Steps
After setting up these tools:
1. Run `cargo audit` regularly to catch security issues
2. Add `cargo tarpaulin` to your CI pipeline
3. Use `cargo clippy` before every commit
4. Set up pre-commit hooks for automatic formatting
See [Building from Source](./building.md) for build-specific information.

1
src/SUMMARY.md
View file

@ -218,6 +218,7 @@
- [Chapter 07: gbapp Architecture Reference](./07-gbapp/README.md)
- [Architecture Overview](./07-gbapp/architecture.md)
- [Building from Source](./07-gbapp/building.md)
- [Cargo Tools Reference](./07-gbapp/cargo-tools.md)
- [Container Deployment (LXC)](./07-gbapp/containers.md)
- [Docker Deployment](./07-gbapp/docker-deployment.md)
- [Scaling and Load Balancing](./07-gbapp/scaling.md)