From 7123f881c6679055bbe8797e7dd4b826108fcd00 Mon Sep 17 00:00:00 2001 From: "Rodrigo Rodriguez (Pragmatismo)" Date: Sun, 7 Dec 2025 09:57:27 -0300 Subject: [PATCH] docs: Update SUMMARY and add cargo-tools documentation --- TASKS.md | 242 --------------- src/07-gbapp/cargo-tools.md | 594 ++++++++++++++++++++++++++++++++++++ src/SUMMARY.md | 3 +- 3 files changed, 596 insertions(+), 243 deletions(-) delete mode 100644 TASKS.md create mode 100644 src/07-gbapp/cargo-tools.md diff --git a/TASKS.md b/TASKS.md deleted file mode 100644 index 460cabac..00000000 --- a/TASKS.md +++ /dev/null @@ -1,242 +0,0 @@ -# Documentation Tasks & Discrepancies Report - -**Generated:** 2025-01 -**Status:** Active development - -> **Summary**: This report tracks discrepancies between documentation and implementation in the General Bots codebase. Key fixes applied: keywords now use spaces (not underscores), ON ERROR RESUME NEXT implemented, DELETE unified. - ---- - -## ✅ COMPLETED - -### Stub Files Now Have Content -- [x] `keyword-soap.md` - SOAP web service integration -- [x] `keyword-merge-pdf.md` - PDF merging (now `MERGE PDF` with space) -- [x] `keyword-kb-statistics.md` - Overall KB statistics -- [x] `keyword-kb-collection-stats.md` - Per-collection statistics -- [x] `keyword-kb-documents-count.md` - Document count -- [x] `keyword-kb-documents-added-since.md` - Recent document tracking -- [x] `keyword-kb-list-collections.md` - Collection listing -- [x] `keyword-kb-storage-size.md` - Storage monitoring - -### Code Changes Applied -- [x] `SEND MAIL` - Changed from `SEND_MAIL` to space-separated -- [x] `GENERATE PDF` - Changed from `GENERATE_PDF` to space-separated -- [x] `MERGE PDF` - Changed from `MERGE_PDF` to space-separated -- [x] `ON ERROR RESUME NEXT` - Implemented in `errors/on_error.rs` -- [x] `DELETE` - Unified keyword (auto-detects HTTP/DB/File) -- [x] `PROMPT.md` - Updated with keyword naming rules - ---- - -## ✅ COMPLETED: Model Name Updates - -Model names updated to current versions (2025-01): - -| File | Updated To | -|------|------------| -| `appendix-external-services/README.md` | `claude-sonnet-4.5` | -| `appendix-external-services/hosting-dns.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` | -| `appendix-external-services/llm-providers.md` | `DeepSeek-R3, Claude Sonnet 4.5` | -| `chapter-04-gbui/how-to/create-first-bot.md` | `claude-sonnet-4.5` | -| `chapter-04-gbui/how-to/monitor-sessions.md` | `LLM active` (generic) | -| `chapter-04-gbui/suite-manual.md` | `Claude Sonnet 4.5`, `Claude Opus 4.5`, `Gemini Pro`, `Llama 3.3` | -| `chapter-06-gbdialog/keyword-model-route.md` | `claude-sonnet-4.5`, `gemini-flash`, `claude-opus-4.5` | -| `chapter-06-gbdialog/basic-vs-automation-tools.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` | -| `chapter-07-gbapp/architecture.md` | `GPT-5 and o3`, `Claude Sonnet 4.5 and Opus 4.5` | -| `chapter-08-config/llm-config.md` | `claude-sonnet-4.5` | -| `chapter-08-config/secrets-management.md` | `claude-sonnet-4.5` | -| `chapter-11-features/ai-llm.md` | `GPT-5`, `o3`, `Claude Sonnet 4.5` | -| `chapter-11-features/core-features.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` | -| `executive-vision.md` | `GPT-5, o3`, `Claude Sonnet 4.5, Opus 4.5` | - -### Naming Convention Applied -- OpenAI: `GPT-5`, `GPT-5 mini`, `o3` -- Anthropic: `Claude Sonnet 4.5`, `Claude Opus 4.5` -- Google: `Gemini Pro`, `Gemini Flash` -- Meta: `Llama 3.3` -- DeepSeek: `DeepSeek-V3`, `DeepSeek-R3` -- Local: `model.gguf`, `local-model` - ---- - -## ✅ RESOLVED: Keyword Syntax Clarifications - -### Keywords Now Use Spaces (NOT Underscores) - -| ✅ Correct | ❌ Wrong | -|-----------|----------| -| `SEND MAIL` | `SEND_MAIL` | -| `GENERATE PDF` | `GENERATE_PDF` | -| `MERGE PDF` | `MERGE_PDF` | -| `SET HEADER` | `SET_HEADER` | -| `CLEAR HEADERS` | `CLEAR_HEADERS` | -| `ON ERROR RESUME NEXT` | N/A | -| `KB STATISTICS` | `KB_STATISTICS` | - -### Keyword Mappings (Documentation Aliases) - -| Documented Pattern | Actual Keyword | -|-------------------|----------------| -| `GENERATE FROM TEMPLATE file WITH data` | `FILL data, template` | -| `GENERATE WITH PROMPT prompt` | `LLM prompt` | -| `DELETE HTTP url` | `DELETE url` (unified) | -| `DELETE FILE path` | `DELETE path` (unified) | -| `SEND EMAIL TO addr WITH body` | `SEND MAIL addr, subject, body, []` | - -### Unified DELETE Keyword - -The `DELETE` keyword now auto-detects context: - -```basic -' HTTP DELETE - detects URL -DELETE "https://api.example.com/resource/123" - -' Database DELETE - table + filter -DELETE "customers", "status = 'inactive'" - -' File DELETE - path without URL prefix -DELETE "temp/report.pdf" -``` - ---- - -## ✅ IMPLEMENTED: Error Handling - -### ON ERROR RESUME NEXT (BASIC-style) - -Now fully implemented in `src/basic/keywords/errors/on_error.rs`: - -```basic -ON ERROR RESUME NEXT ' Enable error trapping -result = RISKY_OPERATION() -IF ERROR THEN - TALK "Error: " + ERROR MESSAGE -END IF -ON ERROR GOTO 0 ' Disable error trapping -CLEAR ERROR ' Clear error state -``` - -### Available Error Keywords - -| Keyword | Description | -|---------|-------------| -| `ON ERROR RESUME NEXT` | Enable error trapping | -| `ON ERROR GOTO 0` | Disable error trapping | -| `ERROR` | Returns true if error occurred | -| `ERROR MESSAGE` | Get last error message | -| `ERR` | Get error number | -| `CLEAR ERROR` | Clear error state | -| `THROW "msg"` | Raise an error | -| `ASSERT cond, "msg"` | Assert condition | - ---- - -## 🟡 CONFIG PARAMETERS - -### Verified in Source Code ✅ - -``` -llm-key, llm-url, llm-model -llm-cache, llm-cache-ttl, llm-cache-semantic, llm-cache-threshold -llm-server, llm-server-* (all server params) -embedding-url, embedding-model -email-from, email-server, email-port, email-user, email-pass -custom-server, custom-port, custom-database, custom-username, custom-password -image-generator-*, video-generator-* -botmodels-enabled, botmodels-host, botmodels-port -episodic-memory-threshold, episodic-memory-history -oauth-*-enabled, oauth-*-client-id, oauth-*-client-secret -website-max-depth, website-max-pages -``` - -### In Code But Missing from Default config.csv - -| Parameter | Used In | -|-----------|---------| -| `llm-temperature` | console/status_panel.rs | -| `llm-server-reasoning-format` | llm/local.rs | -| `email-read-pixel` | email/mod.rs | -| `server-url` | email/mod.rs | -| `teams-app-id`, `teams-*` | core/bot/channels/teams.rs | -| `whatsapp-api-key`, `whatsapp-*` | core/bot/channels/whatsapp.rs | -| `twilio-*`, `aws-*`, `vonage-*` | basic/keywords/sms.rs | -| `qdrant-url` | vector database connection | - -### In config.csv But Status Unknown - -| Parameter | Status | -|-----------|--------| -| `website-expires` | ❓ Not found in search | -| `default-generator` | ❓ Not found in search | - ---- - -## 📋 REMAINING TASKS - -### Priority 1 - Documentation Updates -- [x] Update all model names to generic/current versions (17 files) -- [x] Update keyword examples to use spaces not underscores -- [x] Fix `SEND EMAIL` examples to use `SEND MAIL` syntax -- [x] Document unified `DELETE` keyword behavior (already documented in `keyword-delete.md`) -- [x] Add `ON ERROR RESUME NEXT` to error handling docs (created `keyword-on-error.md`) - -### Priority 2 - New Documentation Needed -- [x] Full config.csv parameter reference (expanded `parameters.md` with all SMS, WhatsApp params) -- [x] SMS provider configuration (created `sms-providers.md` - Twilio, AWS SNS, Vonage, MessageBird) -- [x] Teams/WhatsApp channel setup (created `teams-channel.md` and `whatsapp-channel.md`) -- [x] SOAP authentication configuration (expanded `keyword-soap.md` with WS-Security, OAuth, certs) - -### Priority 3 - Structure Cleanup -- [x] Renamed folders with proper numbering (01-introduction, 02-templates, etc.) -- [x] Removed VB/Visual Basic references (now BASIC only) -- [x] Deleted empty `04-ui-legacy` folder -- [x] Deleted `17-appendix-env-vars` (only VAULT_* vars are real, rest was lies) -- [x] Fixed vbs code blocks to use `basic` syntax -- [x] Updated SUMMARY.md with new folder paths - ---- - -## 📊 Quick Reference - -### Keyword Implementation Files - -| Category | Source File | -|----------|-------------| -| Email | `send_mail.rs` → `SEND MAIL`, `SEND TEMPLATE` | -| PDF | `file_operations.rs` → `GENERATE PDF`, `MERGE PDF` | -| Data | `data_operations.rs` → `DELETE`, `INSERT`, `UPDATE`, `FILL` | -| HTTP | `http_operations.rs` → `POST`, `PUT`, `GRAPHQL`, `SOAP` | -| Errors | `errors/on_error.rs` → `ON ERROR RESUME NEXT` | -| KB Stats | `kb_statistics.rs` → `KB STATISTICS`, etc. | -| LLM | `llm_keyword.rs` → `LLM` | - -### Summary Statistics - -| Category | Count | -|----------|-------| -| Files with old model names | 0 (fixed) | -| Stub files completed | 8 | -| Keywords fixed (underscore→space) | 10 | -| SEND EMAIL → SEND MAIL | 9 files fixed | -| Error handling keywords added | 8 | -| Error handling doc created | `keyword-on-error.md` | -| Config params documented | 50+ | -| New channel docs created | `teams-channel.md`, `whatsapp-channel.md` | -| New provider docs created | `sms-providers.md` | -| SOAP auth expanded | WS-Security, OAuth, certificates | -| VB references removed | All files now use BASIC only | -| Folders renamed | `01-introduction`, `02-templates`, etc. | -| Deleted trash | `04-ui-legacy`, `17-appendix-env-vars` | - ---- - -## Architecture Notes - -- **Language**: Rust with Rhai scripting engine -- **Keywords**: Registered via `register_custom_syntax()` in Rhai -- **Preprocessing**: `basic/compiler/mod.rs` normalizes some syntax -- **Config**: CSV files parsed by `ConfigManager` -- **Vector DB**: Qdrant -- **Relational DB**: PostgreSQL -- **File Storage**: MinIO/SeaweedFS \ No newline at end of file diff --git a/src/07-gbapp/cargo-tools.md b/src/07-gbapp/cargo-tools.md new file mode 100644 index 00000000..7fe3f6e4 --- /dev/null +++ b/src/07-gbapp/cargo-tools.md @@ -0,0 +1,594 @@ +# Cargo Tools Reference + +This chapter documents essential Cargo tools for BotServer development, including code coverage, security auditing, performance profiling, and code quality tools. + +## Overview + +The Rust ecosystem provides powerful tools through Cargo extensions. These tools help maintain code quality, identify security vulnerabilities, measure test coverage, and optimize performance. + +## Code Coverage with cargo-tarpaulin + +### Installation + +```bash +cargo install cargo-tarpaulin +``` + +### Basic Usage + +Run code coverage analysis: + +```bash +cargo tarpaulin +``` + +This generates a coverage report showing which lines of code are exercised by tests. + +### Output Formats + +Generate HTML report: + +```bash +cargo tarpaulin --out Html +``` + +Generate multiple formats (coverage report, lcov for CI): + +```bash +cargo tarpaulin --out Html --out Lcov --out Json +``` + +### Coverage with Features + +Test with specific features enabled: + +```bash +cargo tarpaulin --features vectordb,email +``` + +Test all features: + +```bash +cargo tarpaulin --all-features +``` + +### Excluding Files + +Exclude test files and generated code from coverage: + +```bash +cargo tarpaulin --ignore-tests --exclude-files "gen/*" "tests/*" +``` + +### Coverage Thresholds + +Fail if coverage drops below a threshold (useful for CI): + +```bash +cargo tarpaulin --fail-under 80 +``` + +### Verbose Output + +Show detailed coverage per function: + +```bash +cargo tarpaulin --verbose +``` + +### Integration with CI + +Example GitHub Actions workflow: + +```yaml +- name: Install tarpaulin + run: cargo install cargo-tarpaulin + +- name: Generate coverage + run: cargo tarpaulin --out Xml --fail-under 70 + +- name: Upload coverage to Codecov + uses: codecov/codecov-action@v3 + with: + files: cobertura.xml +``` + +### Configuration File + +Create `.tarpaulin.toml` for project-wide settings: + +```toml +[config] +command = "test" +features = "vectordb" +ignore-tests = true +out = ["Html", "Lcov"] +exclude-files = ["gen/*"] +timeout = "120s" +``` + +## Security Auditing with cargo-audit + +### Installation + +```bash +cargo install cargo-audit +``` + +### Basic Usage + +Check for known security vulnerabilities in dependencies: + +```bash +cargo audit +``` + +### Continuous Auditing + +Run audit as part of CI pipeline to catch new vulnerabilities: + +```bash +cargo audit --deny warnings +``` + +### Fix Vulnerabilities + +Generate a fix for vulnerable dependencies (when possible): + +```bash +cargo audit fix +``` + +### Database Updates + +Update the vulnerability database: + +```bash +cargo audit fetch +``` + +### Ignore Known Issues + +Create `.cargo/audit.toml` to ignore specific advisories: + +```toml +[advisories] +ignore = [ + "RUSTSEC-2020-0071", # Reason for ignoring +] +``` + +### JSON Output for CI + +```bash +cargo audit --json > audit-report.json +``` + +## Dependency Analysis with cargo-deny + +### Installation + +```bash +cargo install cargo-deny +``` + +### Configuration + +Create `deny.toml`: + +```toml +[advisories] +vulnerability = "deny" +unmaintained = "warn" + +[licenses] +unlicensed = "deny" +allow = [ + "MIT", + "Apache-2.0", + "BSD-3-Clause", +] + +[bans] +multiple-versions = "warn" +deny = [ + { name = "openssl" }, # Prefer rustls +] + +[sources] +unknown-registry = "deny" +unknown-git = "deny" +``` + +### Usage + +Check all configured rules: + +```bash +cargo deny check +``` + +Check specific categories: + +```bash +cargo deny check advisories +cargo deny check licenses +cargo deny check bans +cargo deny check sources +``` + +## Code Formatting with cargo-fmt + +### Usage + +Format all code: + +```bash +cargo fmt +``` + +Check formatting without changes: + +```bash +cargo fmt --check +``` + +### Configuration + +Create `rustfmt.toml`: + +```toml +edition = "2021" +max_width = 100 +tab_spaces = 4 +use_small_heuristics = "Default" +reorder_imports = true +group_imports = "StdExternalCrate" +``` + +## Linting with cargo-clippy + +### Usage + +Run clippy with warnings as errors: + +```bash +cargo clippy -- -D warnings +``` + +Run with all lints: + +```bash +cargo clippy -- -W clippy::all -W clippy::pedantic +``` + +### Fix Suggestions Automatically + +```bash +cargo clippy --fix --allow-dirty +``` + +### Configuration + +Add to `Cargo.toml`: + +```toml +[lints.clippy] +all = "warn" +pedantic = "warn" +unwrap_used = "deny" +expect_used = "deny" +``` + +Or create `.clippy.toml`: + +```toml +avoid-breaking-exported-api = false +msrv = "1.70" +``` + +## Documentation with cargo-doc + +### Generate Documentation + +```bash +cargo doc --open +``` + +With private items: + +```bash +cargo doc --document-private-items --open +``` + +### Check Documentation + +Find broken links and missing docs: + +```bash +cargo rustdoc -- -D warnings +``` + +## Benchmarking with cargo-criterion + +### Installation + +```bash +cargo install cargo-criterion +``` + +### Usage + +Run benchmarks: + +```bash +cargo criterion +``` + +### Benchmark Example + +Create `benches/my_benchmark.rs`: + +```rust +use criterion::{black_box, criterion_group, criterion_main, Criterion}; + +fn fibonacci(n: u64) -> u64 { + match n { + 0 => 1, + 1 => 1, + n => fibonacci(n-1) + fibonacci(n-2), + } +} + +fn benchmark(c: &mut Criterion) { + c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20)))); +} + +criterion_group!(benches, benchmark); +criterion_main!(benches); +``` + +## Dependency Management with cargo-outdated + +### Installation + +```bash +cargo install cargo-outdated +``` + +### Usage + +Check for outdated dependencies: + +```bash +cargo outdated +``` + +Show only root dependencies: + +```bash +cargo outdated --root-deps-only +``` + +## Binary Size Analysis with cargo-bloat + +### Installation + +```bash +cargo install cargo-bloat +``` + +### Usage + +Show largest functions: + +```bash +cargo bloat --release -n 20 +``` + +Show largest crates: + +```bash +cargo bloat --release --crates +``` + +### Size Comparison + +Compare sizes between releases: + +```bash +cargo bloat --release --crates > before.txt +# Make changes +cargo bloat --release --crates > after.txt +diff before.txt after.txt +``` + +## Dependency Tree with cargo-tree + +### Usage + +View full dependency tree: + +```bash +cargo tree +``` + +Find duplicate dependencies: + +```bash +cargo tree --duplicates +``` + +Find why a dependency is included: + +```bash +cargo tree --invert tokio +``` + +## Watch Mode with cargo-watch + +### Installation + +```bash +cargo install cargo-watch +``` + +### Usage + +Auto-rebuild on changes: + +```bash +cargo watch -x build +``` + +Auto-test on changes: + +```bash +cargo watch -x test +``` + +Run multiple commands: + +```bash +cargo watch -x check -x test -x clippy +``` + +## Memory Profiling with cargo-valgrind + +### Installation (Linux) + +```bash +sudo apt install valgrind +cargo install cargo-valgrind +``` + +### Usage + +```bash +cargo valgrind run +``` + +## LLVM Coverage with cargo-llvm-cov + +### Installation + +```bash +cargo install cargo-llvm-cov +``` + +### Usage + +More accurate coverage than tarpaulin for some cases: + +```bash +cargo llvm-cov +``` + +Generate HTML report: + +```bash +cargo llvm-cov --html +``` + +## Recommended CI Pipeline + +Example complete CI configuration using these tools: + +```yaml +name: CI + +on: [push, pull_request] + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + + - name: Format check + run: cargo fmt --check + + - name: Clippy + run: cargo clippy -- -D warnings + + - name: Build + run: cargo build --release + + - name: Test + run: cargo test + + coverage: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + + - name: Install tarpaulin + run: cargo install cargo-tarpaulin + + - name: Coverage + run: cargo tarpaulin --out Xml --fail-under 70 + + - uses: codecov/codecov-action@v3 + + security: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + + - name: Install audit + run: cargo install cargo-audit + + - name: Security audit + run: cargo audit --deny warnings +``` + +## Quick Reference + +| Tool | Purpose | Command | +|------|---------|---------| +| cargo-tarpaulin | Code coverage | `cargo tarpaulin` | +| cargo-audit | Security vulnerabilities | `cargo audit` | +| cargo-deny | License/dependency rules | `cargo deny check` | +| cargo-fmt | Code formatting | `cargo fmt` | +| cargo-clippy | Linting | `cargo clippy` | +| cargo-doc | Documentation | `cargo doc --open` | +| cargo-criterion | Benchmarking | `cargo criterion` | +| cargo-outdated | Outdated dependencies | `cargo outdated` | +| cargo-bloat | Binary size analysis | `cargo bloat --release` | +| cargo-tree | Dependency tree | `cargo tree` | +| cargo-watch | Auto-rebuild | `cargo watch -x build` | +| cargo-llvm-cov | LLVM coverage | `cargo llvm-cov` | + +## Installation Script + +Install all recommended tools at once: + +```bash +#!/bin/bash +# install-cargo-tools.sh + +cargo install cargo-tarpaulin +cargo install cargo-audit +cargo install cargo-deny +cargo install cargo-outdated +cargo install cargo-bloat +cargo install cargo-watch +cargo install cargo-criterion +cargo install cargo-llvm-cov +``` + +## Next Steps + +After setting up these tools: + +1. Run `cargo audit` regularly to catch security issues +2. Add `cargo tarpaulin` to your CI pipeline +3. Use `cargo clippy` before every commit +4. Set up pre-commit hooks for automatic formatting + +See [Building from Source](./building.md) for build-specific information. diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 89db9654..2bc906b6 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -218,6 +218,7 @@ - [Chapter 07: gbapp Architecture Reference](./07-gbapp/README.md) - [Architecture Overview](./07-gbapp/architecture.md) - [Building from Source](./07-gbapp/building.md) + - [Cargo Tools Reference](./07-gbapp/cargo-tools.md) - [Container Deployment (LXC)](./07-gbapp/containers.md) - [Docker Deployment](./07-gbapp/docker-deployment.md) - [Scaling and Load Balancing](./07-gbapp/scaling.md) @@ -382,4 +383,4 @@ - [SVG and Conversation Standards](./16-appendix-docs-style/svg.md) [Glossary](./glossary.md) -[Contact](./contact/README.md) \ No newline at end of file +[Contact](./contact/README.md)