GeneralBots/botserver
2
1
Fork 0
Code Issues Pull requests Projects Releases 323 Packages Wiki Activity Actions

Merge branch 'main' into main

Browse source
This commit is contained in:
ChristopherCastilho 2025-10-28 07:18:56 -03:00 • committed by GitHub
commit 6c511b4ebb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
230 changed files with 22565 additions and 5930 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.gitignore vendored
View file

@ -1,7 +1,11 @@
target
target*
.env
*.env
work
*.out
bin
botserver-stack
*logfile*
*-log*
docs/book
.rdb

3610
Cargo.lock generated
View file

File diff suppressed because it is too large Load diff

25
Cargo.toml
View file

@ -11,7 +11,6 @@ authors = [
"Arenas.io",
"Atylla L",
"Christopher de Castilho",
"Dario Lima",
"Dario Junior",
"David Lerner",
"Experimentation Garage",
@ -38,13 +37,17 @@ license = "AGPL-3.0"
repository = "https://github.com/GeneralBots/BotServer"
[features]
default = ["vectordb"]
desktop = ["tauri", "tauri-plugin-opener", "tauri-plugin-dialog"]
default = [ "vectordb"]
vectordb = ["qdrant-client"]
email = ["imap"]
web_automation = ["headless_chrome"]
webapp = ["tauri", "tauri-plugin-opener", "tauri-plugin-dialog"]
[dependencies]
actix-cors = "0.7"
aws-config = "0.57.0"
csv = "1.3"
actix-multipart = "0.7"
imap = { version = "3.0.0-alpha.15", optional = true }
actix-web = "4.9"
@ -69,8 +72,6 @@ include_dir = "0.7"
log = "0.4"
mailparse = "0.15"
native-tls = "0.2"
num-format = "0.4"
qdrant-client = { version = "1.12", optional = true }
rhai = { git = "https://github.com/therealprof/rhai.git", branch = "features/use-web-time" }
@ -89,7 +90,7 @@ urlencoding = "2.1"
uuid = { version = "1.11", features = ["serde", "v4"] }
zip = "2.2"
time = "0.3.44"
aws-sdk-s3 = "1.108.0"
aws-sdk-s3 = { version = "1.108.0", features = ["behavior-version-latest"] }
headless_chrome = { version = "1.0.18", optional = true }
rand = "0.9.2"
pdf-extract = "0.10.0"
@ -97,3 +98,17 @@ scraper = "0.20"
sha2 = "0.10.9"
ureq = "3.1.2"
indicatif = "0.18.0"
tauri = { version = "2", features = ["unstable"], optional = true }
tauri-plugin-opener = { version = "2", optional = true }
tauri-plugin-dialog = { version = "2", optional = true }
[build-dependencies]
tauri-build = { version = "2", features = [] }
[profile.release]
lto = true # Enables Link-Time Optimization
opt-level = "z" # Optimizes for size instead of speed
strip = true # Strips debug symbols
panic = "abort" # Reduces size by removing panic unwinding
codegen-units = 1 # More aggressive optimization

32
README.md
View file

@ -136,34 +136,6 @@ DRIVE_ACCESSKEY=minioadmin
DRIVE_SECRET=minioadmin
DRIVE_ORG_PREFIX=botserver-
# Cache (Redis - auto-installed)
CACHE_URL=redis://localhost:6379
# LLM (llama.cpp - auto-installed with models)
LLM_LOCAL=false
LLM_URL=http://localhost:8081/v1
EMBEDDING_URL=http://localhost:8082
```
#### Optional Settings
```bash
# Server Configuration
SERVER_HOST=127.0.0.1
SERVER_PORT=8080
# External AI API (Groq, OpenAI, Azure, etc.)
AI_KEY=your-api-key-here
AI_ENDPOINT=https://api.groq.com/openai/v1/chat/completions
AI_LLM_MODEL=openai/gpt-4
# Email (for notifications)
EMAIL_FROM=bot@example.com
EMAIL_SERVER=smtp.example.com
EMAIL_PORT=587
EMAIL_USER=your-email@example.com
EMAIL_PASS=your-password
```
#### Legacy Mode (Use existing infrastructure)
If you already have PostgreSQL, MinIO, etc. running, set these in `.env`:
```bash
@ -176,10 +148,6 @@ TABLES_PASSWORD=your-password
DRIVE_SERVER=https://your-minio-host
DRIVE_ACCESSKEY=your-access-key
DRIVE_SECRET=your-secret-key
# Existing AI endpoint
AI_ENDPOINT=https://your-llm-endpoint
AI_KEY=your-api-key
```
BotServer will detect existing infrastructure and skip auto-installation.

78
TODO.md Normal file
View file

@ -0,0 +1,78 @@
# Documentation Completion Checklist
- [x] Created Chapter 01 files (README, installation, first-conversation, sessions)
- [ ] Fill Chapter 02 files (README, gbai, gbdialog, gbkb, gbot, gbtheme, gbdrive) – already have content
- [ ] Complete Chapter 03 files
- [ ] README.md
- [ ] vector-collections.md
- [ ] indexing.md
- [ ] qdrant.md
- [ ] semantic-search.md
- [ ] context-compaction.md
- [ ] caching.md (if needed)
- [ ] Complete Chapter 04 files
- [ ] README.md
- [ ] structure.md
- [ ] web-interface.md
- [ ] css.md
- [ ] html.md
- [ ] Complete Chapter 05 files
- [ ] README.md
- [ ] basics.md
- [ ] templates.md
- [ ] template-start.md
- [ ] template-auth.md
- [ ] template-summary.md
- [ ] template-enrollment.md
- [ ] keywords.md
- [ ] All keyword pages (talk, hear, set-user, set-context, llm, get-bot-memory, set-bot-memory, set-kb, add-kb, add-website, add-tool, list-tools, remove-tool, clear-tools, get, find, set, on, set-schedule, create-site, create-draft, website-of, print, wait, format, first, last, for-each, exit-for)
- [ ] Complete Chapter 06 files
- [ ] README.md
- [ ] architecture.md
- [ ] building.md
- [ ] crates.md
- [ ] services.md
- [ ] custom-keywords.md
- [ ] dependencies.md
- [ ] Complete Chapter 07 files
- [ ] README.md
- [ ] config-csv.md
- [ ] parameters.md
- [ ] answer-modes.md
- [ ] llm-config.md
- [ ] context-config.md
- [ ] minio.md
- [ ] Complete Chapter 08 files
- [ ] README.md
- [ ] tool-definition.md
- [ ] param-declaration.md
- [ ] compilation.md
- [ ] mcp-format.md
- [ ] openai-format.md
- [ ] get-integration.md
- [ ] external-apis.md
- [ ] Complete Chapter 09 files
- [ ] README.md
- [ ] core-features.md
- [ ] conversation.md
- [ ] ai-llm.md
- [ ] knowledge-base.md
- [ ] automation.md
- [ ] email.md
- [ ] web-automation.md
- [ ] storage.md
- [ ] channels.md
- [ ] Complete Chapter 10 files
- [ ] README.md
- [ ] setup.md
- [ ] standards.md
- [ ] testing.md
- [ ] pull-requests.md
- [ ] documentation.md
- [ ] Complete Appendix I files
- [ ] README.md
- [ ] schema.md
- [ ] tables.md
- [ ] relationships.md
- [ ] Verify SUMMARY.md links
- [ ] Run mdbook build to ensure no errors

32
add-req.sh
View file

@ -19,26 +19,26 @@ for file in "${prompts[@]}"; do
done
dirs=(
#"auth"
#"automation"
#"basic"
#"bot"
"auth"
"automation"
"basic"
"bot"
"bootstrap"
"package_manager"
#"channels"
"channels"
"config"
#"context"
#"email"
#"file"
#"llm"
#"llm_legacy"
#"org"
#"session"
"context"
"email"
"file"
"llm"
"llm_legacy"
"org"
"session"
"shared"
#"tests"
#"tools"
#"web_automation"
#"whatsapp"
"tests"
"tools"
"web_automation"
"whatsapp"
)
filter_rust_file() {

1
dev-start.sh
View file

@ -1 +0,0 @@
sudo systemctl start valkey-server

417
docs/CHANGELOG_TOOL_MANAGEMENT.md
View file

@ -1,417 +0,0 @@
# Changelog: Multiple Tool Association Feature
## Version: 6.0.4 (Feature Release)
**Date**: 2024
**Type**: Major Feature Addition
---
## 🎉 Summary
Implemented **real database-backed multiple tool association** system allowing users to dynamically manage multiple BASIC tools per conversation session. Replaces SQL placeholder comments with fully functional Diesel ORM code.
---
## ✨ New Features
### 1. Multiple Tools Per Session
- Users can now associate unlimited tools with a single conversation
- Each session maintains its own independent tool list
- Tools are stored persistently in the database
### 2. Four New BASIC Keywords
#### `ADD_TOOL`
- Adds a compiled BASIC tool to the current session
- Validates tool exists and is active
- Prevents duplicate additions
- Example: `ADD_TOOL ".gbdialog/enrollment.bas"`
#### `REMOVE_TOOL`
- Removes a specific tool from the current session
- Does not affect other sessions
- Example: `REMOVE_TOOL ".gbdialog/enrollment.bas"`
#### `LIST_TOOLS`
- Lists all tools currently active in the session
- Shows numbered list with tool names
- Example: `LIST_TOOLS`
#### `CLEAR_TOOLS`
- Removes all tool associations from current session
- Useful for resetting conversation context
- Example: `CLEAR_TOOLS`
### 3. Database Implementation
#### New Table: `session_tool_associations`
```sql
CREATE TABLE IF NOT EXISTS session_tool_associations (
id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
added_at TEXT NOT NULL,
UNIQUE(session_id, tool_name)
);
```
#### Indexes for Performance
- `idx_session_tool_session` - Fast session lookups
- `idx_session_tool_name` - Fast tool name searches
- UNIQUE constraint prevents duplicate associations
### 4. Prompt Processor Integration
- Automatically loads all session tools during prompt processing
- Tools become available to LLM for function calling
- Maintains backward compatibility with legacy `current_tool` field
---
## 🔧 Technical Changes
### New Files Created
1. **`src/basic/keywords/remove_tool.rs`**
- Implements `REMOVE_TOOL` keyword
- Handles tool removal logic
- 138 lines
2. **`src/basic/keywords/clear_tools.rs`**
- Implements `CLEAR_TOOLS` keyword
- Clears all session tool associations
- 103 lines
3. **`src/basic/keywords/list_tools.rs`**
- Implements `LIST_TOOLS` keyword
- Displays active tools in formatted list
- 107 lines
4. **`docs/TOOL_MANAGEMENT.md`**
- Comprehensive documentation (620 lines)
- Covers all features, use cases, and API
- Includes troubleshooting and best practices
5. **`docs/TOOL_MANAGEMENT_QUICK_REF.md`**
- Quick reference guide (176 lines)
- Common patterns and examples
- Fast lookup for developers
6. **`examples/tool_management_example.bas`**
- Working example demonstrating all features
- Shows progressive tool loading
- Demonstrates all four keywords
### Modified Files
1. **`src/basic/keywords/add_tool.rs`**
- Replaced TODO comments with real Diesel queries
- Added validation against `basic_tools` table
- Implemented `INSERT ... ON CONFLICT DO NOTHING`
- Added public API functions:
- `get_session_tools()` - Retrieve all session tools
- `remove_session_tool()` - Remove specific tool
- `clear_session_tools()` - Remove all tools
- Changed from 117 lines to 241 lines
2. **`src/basic/keywords/mod.rs`**
- Added module declarations:
- `pub mod clear_tools;`
- `pub mod list_tools;`
- `pub mod remove_tool;`
3. **`src/basic/mod.rs`**
- Imported new keyword functions
- Registered keywords with Rhai engine:
- `remove_tool_keyword()`
- `clear_tools_keyword()`
- `list_tools_keyword()`
4. **`src/context/prompt_processor.rs`**
- Added import: `use crate::basic::keywords::add_tool::get_session_tools;`
- Modified `get_available_tools()` method
- Queries `session_tool_associations` table
- Loads all tools for current session
- Adds tools to LLM context automatically
- Maintains legacy `current_tool` support
5. **`src/shared/models.rs`**
- Wrapped all `diesel::table!` macros in `pub mod schema {}`
- Re-exported schema at module level: `pub use schema::*;`
- Maintains backward compatibility with existing code
- Enables proper module access for new keywords
---
## 🗄️ Database Schema Changes
### Migration: `6.0.3.sql`
Already included the `session_tool_associations` table definition.
**No new migration required** - existing schema supports this feature.
---
## 🔄 API Changes
### New Public Functions
```rust
// In src/basic/keywords/add_tool.rs
/// Get all tools associated with a session
pub fn get_session_tools(
conn: &mut PgConnection,
session_id: &Uuid,
) -> Result<Vec<String>, diesel::result::Error>
/// Remove a tool association from a session
pub fn remove_session_tool(
conn: &mut PgConnection,
session_id: &Uuid,
tool_name: &str,
) -> Result<usize, diesel::result::Error>
/// Clear all tool associations for a session
pub fn clear_session_tools(
conn: &mut PgConnection,
session_id: &Uuid,
) -> Result<usize, diesel::result::Error>
```
### Modified Function Signatures
Changed from `&PgConnection` to `&mut PgConnection` to match Diesel 2.x requirements.
---
## 🔀 Backward Compatibility
### Fully Backward Compatible
- ✅ Legacy `current_tool` field still works
- ✅ Existing tool loading mechanisms unchanged
- ✅ All existing BASIC scripts continue to work
- ✅ No breaking changes to API or database schema
### Migration Path
Old code using single tool:
```rust
session.current_tool = Some("enrollment".to_string());
```
New code using multiple tools:
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
ADD_TOOL ".gbdialog/payment.bas"
```
Both approaches work simultaneously!
---
## 🎯 Use Cases Enabled
### 1. Progressive Tool Loading
Load tools as conversation progresses based on user needs.
### 2. Context-Aware Tool Management
Different tool sets for different conversation stages.
### 3. Department-Specific Tools
Route users to appropriate toolsets based on department/role.
### 4. A/B Testing
Test different tool combinations for optimization.
### 5. Multi-Phase Conversations
Switch tool sets between greeting, main interaction, and closing phases.
---
## 🚀 Performance Improvements
- **Indexed Lookups**: Fast queries via database indexes
- **Batch Loading**: All tools loaded in single query
- **Session Isolation**: No cross-session interference
- **Efficient Storage**: Only stores references, not code
---
## 🛡️ Security Enhancements
- **Bot ID Validation**: Tools validated against bot ownership
- **SQL Injection Prevention**: All queries use Diesel parameterization
- **Session Isolation**: Users can't access other sessions' tools
- **Input Sanitization**: Tool names extracted and validated
---
## 📝 Documentation Added
1. **Comprehensive Guide**: `TOOL_MANAGEMENT.md`
- Architecture overview
- Complete API reference
- Use cases and patterns
- Troubleshooting guide
- Security considerations
- Performance optimization
2. **Quick Reference**: `TOOL_MANAGEMENT_QUICK_REF.md`
- Fast lookup for common operations
- Code snippets and examples
- Common patterns
- Error reference
3. **Example Script**: `tool_management_example.bas`
- Working demonstration
- All four keywords in action
- Commented for learning
---
## 🧪 Testing
### Manual Testing
- Example script validates all functionality
- Can be run in development environment
- Covers all CRUD operations on tool associations
### Integration Points Tested
- ✅ Diesel ORM queries execute correctly
- ✅ Database locks acquired/released properly
- ✅ Async execution via Tokio runtime
- ✅ Rhai engine integration
- ✅ Prompt processor loads tools correctly
- ✅ LLM receives updated tool list
---
## 🐛 Bug Fixes
### Fixed in This Release
- **SQL Placeholders Removed**: All TODO comments replaced with real code
- **Mutable Reference Handling**: Proper `&mut PgConnection` usage throughout
- **Schema Module Structure**: Proper module organization for Diesel tables
- **Thread Safety**: Correct mutex handling for database connections
---
## ⚠️ Known Limitations
1. **No Auto-Cleanup**: Tool associations persist until manually removed
- Future: Auto-cleanup when session expires
2. **No Tool Priority**: All tools treated equally
- Future: Priority/ordering system
3. **No Tool Groups**: Tools managed individually
- Future: Group operations
---
## 🔮 Future Enhancements
Potential features for future releases:
1. **Tool Priority System**: Specify preferred tool order
2. **Tool Groups**: Manage related tools as a set
3. **Auto-Cleanup**: Remove associations when session ends
4. **Tool Statistics**: Track usage metrics
5. **Conditional Loading**: LLM-driven tool selection
6. **Fine-Grained Permissions**: User-level tool access control
7. **Tool Versioning**: Support multiple versions of same tool
---
## 📊 Impact Analysis
### Lines of Code Changed
- **Added**: ~1,200 lines (new files + modifications)
- **Modified**: ~150 lines (existing files)
- **Total**: ~1,350 lines
### Files Changed
- **New Files**: 6
- **Modified Files**: 5
- **Total Files**: 11
### Modules Affected
- `src/basic/keywords/` (4 files)
- `src/basic/mod.rs` (1 file)
- `src/context/prompt_processor.rs` (1 file)
- `src/shared/models.rs` (1 file)
- `docs/` (3 files)
- `examples/` (1 file)
---
## ✅ Verification Steps
To verify this feature works:
1. **Check Compilation**
```bash
cargo build --release
```
2. **Verify Database**
```sql
SELECT * FROM session_tool_associations;
```
3. **Run Example**
```bash
# Load examples/tool_management_example.bas in bot
```
4. **Test BASIC Keywords**
```basic
ADD_TOOL ".gbdialog/test.bas"
LIST_TOOLS
REMOVE_TOOL ".gbdialog/test.bas"
```
---
## 🤝 Contributors
- Implemented real database code (replacing placeholders)
- Added four new BASIC keywords
- Integrated with prompt processor
- Created comprehensive documentation
- Built working examples
---
## 📄 License
This feature maintains the same license as the parent project.
---
## 🔗 References
- **Issue**: Multiple tools association request
- **Feature Request**: "ADD_TOOL, several calls in start, according to what user can talk"
- **Database Schema**: `migrations/6.0.3.sql`
- **Main Implementation**: `src/basic/keywords/add_tool.rs`
---
## 🎓 Learning Resources
For developers working with this feature:
1. Read `TOOL_MANAGEMENT.md` for comprehensive overview
2. Review `TOOL_MANAGEMENT_QUICK_REF.md` for quick reference
3. Study `examples/tool_management_example.bas` for practical usage
4. Examine `src/basic/keywords/add_tool.rs` for implementation details
---
## 🏁 Conclusion
This release transforms the tool management system from a single-tool, placeholder-based system to a fully functional, database-backed, multi-tool architecture. Users can now dynamically manage multiple tools per session with persistent storage, proper validation, and a clean API.
The implementation uses real Diesel ORM code throughout, with no SQL placeholders or TODOs remaining. All features are production-ready and fully tested.
**Status**: ✅ Complete and Production Ready

623
docs/DEPLOYMENT_CHECKLIST.md
View file

@ -1,623 +0,0 @@
# KB and Tools System - Deployment Checklist
## 🎯 Pre-Deployment Checklist
### Infrastructure Requirements
- [ ] **PostgreSQL 12+** running and accessible
- [ ] **Qdrant** vector database running (port 6333)
- [ ] **MinIO** object storage running (ports 9000, 9001)
- [ ] **LLM Server** for embeddings (port 8081)
- [ ] **Redis** (optional, for caching)
### System Resources
- [ ] **Minimum 4GB RAM** (8GB recommended)
- [ ] **10GB disk space** for documents and embeddings
- [ ] **2+ CPU cores** for parallel processing
- [ ] **Network access** to external APIs (if using ADD_WEBSITE)
---
## 📋 Configuration Steps
### 1. Environment Variables
Create/update `.env` file:
```bash
# Core Settings
DATABASE_URL=postgresql://user:pass@localhost:5432/botserver
QDRANT_URL=http://localhost:6333
LLM_URL=http://localhost:8081
CACHE_URL=redis://127.0.0.1/
# MinIO Configuration
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_USE_SSL=false
MINIO_ORG_PREFIX=org1_
# Server Configuration
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
RUST_LOG=info
```
**Verify:**
- [ ] All URLs are correct and accessible
- [ ] Credentials are set properly
- [ ] Org prefix matches your organization
---
### 2. Database Setup
```bash
# Connect to PostgreSQL
psql -U postgres -d botserver
# Run migration
\i migrations/create_kb_and_tools_tables.sql
# Verify tables created
\dt kb_*
\dt basic_tools
# Check triggers
\df update_updated_at_column
```
**Verify:**
- [ ] Tables `kb_documents`, `kb_collections`, `basic_tools` exist
- [ ] Indexes are created
- [ ] Triggers are active
- [ ] No migration errors
---
### 3. MinIO Bucket Setup
```bash
# Using MinIO CLI (mc)
mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/org1_default.gbai
mc policy set public local/org1_default.gbai
# Or via MinIO Console at http://localhost:9001
```
**Create folder structure:**
```
org1_default.gbai/
├── .gbkb/ # Knowledge Base documents
└── .gbdialog/ # BASIC scripts
```
**Verify:**
- [ ] Bucket created with correct name
- [ ] Folders `.gbkb/` and `.gbdialog/` exist
- [ ] Upload permissions work
- [ ] Download/read permissions work
---
### 4. Qdrant Setup
```bash
# Check Qdrant is running
curl http://localhost:6333/
# Expected response: {"title":"qdrant - vector search engine","version":"..."}
```
**Verify:**
- [ ] Qdrant responds on port 6333
- [ ] API is accessible
- [ ] Dashboard works at http://localhost:6333/dashboard
- [ ] No authentication errors
---
### 5. LLM Server for Embeddings
```bash
# Check LLM server is running
curl http://localhost:8081/v1/models
# Test embeddings endpoint
curl -X POST http://localhost:8081/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"input": ["test"], "model": "text-embedding-ada-002"}'
```
**Verify:**
- [ ] LLM server responds
- [ ] Embeddings endpoint works
- [ ] Vector dimension is 1536 (or update in code)
- [ ] Response time < 5 seconds
---
## 🚀 Deployment
### 1. Build Application
```bash
# Clean build
cargo clean
cargo build --release
# Verify binary
./target/release/botserver --version
```
**Verify:**
- [ ] Compilation succeeds with no errors
- [ ] Binary created in `target/release/`
- [ ] All features enabled correctly
---
### 2. Upload Initial Files
**Upload to MinIO `.gbkb/` folder:**
```bash
# Example: Upload enrollment documents
mc cp enrollment_guide.pdf local/org1_default.gbai/.gbkb/enrollpdfs/
mc cp requirements.pdf local/org1_default.gbai/.gbkb/enrollpdfs/
mc cp faq.pdf local/org1_default.gbai/.gbkb/enrollpdfs/
```
**Upload to MinIO `.gbdialog/` folder:**
```bash
# Upload BASIC tools
mc cp start.bas local/org1_default.gbai/.gbdialog/
mc cp enrollment.bas local/org1_default.gbai/.gbdialog/
mc cp pricing.bas local/org1_default.gbai/.gbdialog/
```
**Verify:**
- [ ] Documents uploaded successfully
- [ ] BASIC scripts uploaded
- [ ] Files are readable via MinIO
- [ ] Correct folder structure maintained
---
### 3. Start Services
```bash
# Start botserver
./target/release/botserver
# Or with systemd
sudo systemctl start botserver
sudo systemctl enable botserver
# Or with Docker
docker-compose up -d botserver
```
**Monitor startup logs:**
```bash
# Check logs
tail -f /var/log/botserver.log
# Or Docker logs
docker logs -f botserver
```
**Look for:**
- [ ] `KB Manager service started`
- [ ] `MinIO Handler service started`
- [ ] `Startup complete!`
- [ ] No errors about missing services
---
### 4. Verify KB Indexing
**Wait 30-60 seconds for initial indexing**
```bash
# Check Qdrant collections
curl http://localhost:6333/collections
# Should see collections like:
# - kb_<bot_id>_enrollpdfs
# - kb_<bot_id>_productdocs
```
**Check logs for indexing:**
```bash
grep "Indexing document" /var/log/botserver.log
grep "Document indexed successfully" /var/log/botserver.log
```
**Verify:**
- [ ] Collections created in Qdrant
- [ ] Documents indexed (check chunk count)
- [ ] No indexing errors in logs
- [ ] File hashes stored in database
---
### 5. Test Tool Compilation
**Check compiled tools:**
```bash
# List work directory
ls -la ./work/*/default.gbdialog/
# Should see:
# - *.ast files (compiled AST)
# - *.mcp.json files (MCP definitions)
# - *.tool.json files (OpenAI definitions)
```
**Verify:**
- [ ] AST files created for each .bas file
- [ ] MCP JSON files generated (if PARAM exists)
- [ ] Tool JSON files generated (if PARAM exists)
- [ ] No compilation errors in logs
---
## 🧪 Testing
### Test 1: KB Search
```bash
# Create test session with answer_mode=2 (documents only)
curl -X POST http://localhost:8080/sessions \
-H "Content-Type: application/json" \
-d '{
"user_id": "test-user",
"bot_id": "default",
"answer_mode": 2
}'
# Send query
curl -X POST http://localhost:8080/chat \
-H "Content-Type: application/json" \
-d '{
"session_id": "<session_id>",
"message": "What documents do I need for enrollment?"
}'
```
**Expected:**
- [ ] Response contains information from indexed PDFs
- [ ] References to source documents
- [ ] Relevant chunks retrieved
---
### Test 2: Tool Calling
```bash
# Call enrollment tool endpoint
curl -X POST http://localhost:8080/default/enrollment \
-H "Content-Type: application/json" \
-d '{
"name": "Test User",
"email": "test@example.com"
}'
```
**Expected:**
- [ ] Tool executes successfully
- [ ] Data saved to CSV
- [ ] Response includes enrollment ID
- [ ] KB activated (if SET_KB in script)
---
### Test 3: Mixed Mode (KB + Tools)
```bash
# Create session with answer_mode=4 (mixed)
curl -X POST http://localhost:8080/sessions \
-H "Content-Type: application/json" \
-d '{
"user_id": "test-user",
"bot_id": "default",
"answer_mode": 4
}'
# Send query that should use both KB and tools
curl -X POST http://localhost:8080/chat \
-H "Content-Type: application/json" \
-d '{
"session_id": "<session_id>",
"message": "I want to enroll. What information do you need?"
}'
```
**Expected:**
- [ ] Bot references both KB documents and available tools
- [ ] Intelligently decides when to use KB vs tools
- [ ] Context includes both document excerpts and tool info
---
### Test 4: Website Indexing
```bash
# In BASIC or via API, test ADD_WEBSITE
# (Requires script with ADD_WEBSITE keyword)
# Check temporary collection created
curl http://localhost:6333/collections | grep temp_website
```
**Expected:**
- [ ] Website crawled successfully
- [ ] Temporary collection created
- [ ] Content indexed
- [ ] Available for current session only
---
## 🔍 Monitoring
### Health Checks
```bash
# Botserver health
curl http://localhost:8080/health
# Qdrant health
curl http://localhost:6333/
# MinIO health
curl http://localhost:9000/minio/health/live
# Database connection
psql -U postgres -d botserver -c "SELECT 1"
```
**Set up alerts for:**
- [ ] Service downtime
- [ ] High memory usage (>80%)
- [ ] Disk space low (<10%)
- [ ] Indexing failures
- [ ] Tool compilation errors
---
### Log Monitoring
**Important log patterns to watch:**
```bash
# Successful indexing
grep "Document indexed successfully" botserver.log
# Indexing errors
grep "ERROR.*Indexing" botserver.log
# Tool compilation
grep "Tool compiled successfully" botserver.log
# KB Manager activity
grep "KB Manager" botserver.log
# MinIO handler activity
grep "MinIO Handler" botserver.log
```
---
### Database Monitoring
```sql
-- Check document count per collection
SELECT collection_name, COUNT(*) as doc_count
FROM kb_documents
GROUP BY collection_name;
-- Check indexing status
SELECT
collection_name,
COUNT(*) as total,
COUNT(indexed_at) as indexed,
COUNT(*) - COUNT(indexed_at) as pending
FROM kb_documents
GROUP BY collection_name;
-- Check compiled tools
SELECT tool_name, compiled_at, is_active
FROM basic_tools
ORDER BY compiled_at DESC;
-- Recent KB activity
SELECT * FROM kb_documents
ORDER BY updated_at DESC
LIMIT 10;
```
---
## 🔒 Security Checklist
- [ ] Change default MinIO credentials
- [ ] Enable SSL/TLS for MinIO
- [ ] Set up firewall rules
- [ ] Enable Qdrant authentication
- [ ] Use secure PostgreSQL connections
- [ ] Validate file uploads (size, type)
- [ ] Implement rate limiting
- [ ] Set up proper CORS policies
- [ ] Use environment variables for secrets
- [ ] Enable request logging
- [ ] Set up backup strategy
---
## 📊 Performance Tuning
### MinIO Handler
```rust
// In src/kb/minio_handler.rs
interval(Duration::from_secs(15)) // Adjust polling interval
```
### KB Manager
```rust
// In src/kb/mod.rs
interval(Duration::from_secs(30)) // Adjust check interval
```
### Embeddings
```rust
// In src/kb/embeddings.rs
const CHUNK_SIZE: usize = 512; // Adjust chunk size
const CHUNK_OVERLAP: usize = 50; // Adjust overlap
```
### Qdrant
```rust
// In src/kb/qdrant_client.rs
let vector_size = 1536; // Match your embedding model
```
**Tune based on:**
- [ ] Document update frequency
- [ ] System resource usage
- [ ] Query performance requirements
- [ ] Embedding model characteristics
---
## 🔄 Backup & Recovery
### Database Backup
```bash
# Daily backup
pg_dump -U postgres botserver > botserver_$(date +%Y%m%d).sql
# Restore
psql -U postgres botserver < botserver_20240101.sql
```
### MinIO Backup
```bash
# Backup bucket
mc mirror local/org1_default.gbai/ ./backups/minio/
# Restore
mc mirror ./backups/minio/ local/org1_default.gbai/
```
### Qdrant Backup
```bash
# Snapshot all collections
curl -X POST http://localhost:6333/collections/{collection_name}/snapshots
# Download snapshot
curl http://localhost:6333/collections/{collection_name}/snapshots/{snapshot_name}
```
**Schedule:**
- [ ] Database: Daily at 2 AM
- [ ] MinIO: Daily at 3 AM
- [ ] Qdrant: Weekly
- [ ] Test restore monthly
---
## 📚 Documentation
- [ ] Update API documentation
- [ ] Document custom BASIC keywords
- [ ] Create user guides for tools
- [ ] Document KB collection structure
- [ ] Create troubleshooting guide
- [ ] Document deployment process
- [ ] Create runbooks for common issues
---
## ✅ Post-Deployment Verification
**Final Checklist:**
- [ ] All services running and healthy
- [ ] Documents indexing automatically
- [ ] Tools compiling on upload
- [ ] KB search working correctly
- [ ] Tool endpoints responding
- [ ] Mixed mode working as expected
- [ ] Logs are being written
- [ ] Monitoring is active
- [ ] Backups scheduled
- [ ] Security measures in place
- [ ] Documentation updated
- [ ] Team trained on system
---
## 🆘 Rollback Plan
**If deployment fails:**
1. **Stop services**
```bash
sudo systemctl stop botserver
```
2. **Restore database**
```bash
psql -U postgres botserver < botserver_backup.sql
```
3. **Restore MinIO**
```bash
mc mirror ./backups/minio/ local/org1_default.gbai/
```
4. **Revert code**
```bash
git checkout <previous-version>
cargo build --release
```
5. **Restart services**
```bash
sudo systemctl start botserver
```
6. **Verify rollback**
- Test basic functionality
- Check logs for errors
- Verify data integrity
---
## 📞 Support Contacts
- **Infrastructure Issues:** DevOps Team
- **Database Issues:** DBA Team
- **Application Issues:** Development Team
- **Security Issues:** Security Team
---
## 📅 Maintenance Schedule
- **Daily:** Check logs, monitor services
- **Weekly:** Review KB indexing stats, check disk space
- **Monthly:** Test backups, review performance metrics
- **Quarterly:** Security audit, update dependencies
---
**Deployment Status:** ⬜ Not Started | 🟡 In Progress | ✅ Complete
**Deployed By:** ________________
**Date:** ________________
**Version:** ________________
**Sign-off:** ________________

542
docs/KB_AND_TOOLS.md
View file

@ -1,542 +0,0 @@
# Knowledge Base (KB) and Tools System
## Overview
This document describes the comprehensive Knowledge Base (KB) and BASIC Tools compilation system integrated into the botserver. This system enables:
1. **Dynamic Knowledge Base Management**: Monitor MinIO buckets for document changes and automatically index them in Qdrant vector database
2. **BASIC Tool Compilation**: Compile BASIC scripts into AST and generate MCP/OpenAI tool definitions
3. **Intelligent Context Processing**: Enhance prompts with relevant KB documents and available tools based on answer mode
4. **Temporary Website Indexing**: Crawl and index web pages for session-specific knowledge
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Bot Server │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ KB Manager │ │ MinIO │ │ Qdrant │ │
│ │ │◄──►│ Handler │◄──►│ Client │ │
│ │ │ │ │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ ▲ │
│ │ │ │
│ ▼ │ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ BASIC │ │ Embeddings │ │
│ │ Compiler │ │ Generator │ │
│ │ │ │ │ │
│ └──────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Prompt Processor │ │
│ │ (Integrates KB + Tools based on Answer Mode) │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## Components
### 1. KB Manager (`src/kb/mod.rs`)
The KB Manager coordinates MinIO monitoring and Qdrant indexing:
- **Watches collections**: Monitors `.gbkb/` folders for document changes
- **Detects changes**: Uses file hashing (SHA256) to detect modified files
- **Indexes documents**: Splits documents into chunks and generates embeddings
- **Stores metadata**: Maintains document information in PostgreSQL
#### Key Functions
```rust
// Add a KB collection to be monitored
kb_manager.add_collection(bot_id, "enrollpdfs").await?;
// Remove a collection
kb_manager.remove_collection("enrollpdfs").await?;
// Start the monitoring service
let kb_handle = kb_manager.spawn();
```
### 2. MinIO Handler (`src/kb/minio_handler.rs`)
Monitors MinIO buckets for file changes:
- **Polling**: Checks for changes every 15 seconds
- **Event detection**: Identifies created, modified, and deleted files
- **State tracking**: Maintains file ETags and sizes for change detection
#### File Change Events
```rust
pub enum FileChangeEvent {
Created { path: String, size: i64, etag: String },
Modified { path: String, size: i64, etag: String },
Deleted { path: String },
}
```
### 3. Qdrant Client (`src/kb/qdrant_client.rs`)
Manages vector database operations:
- **Collection management**: Create, delete, and check collections
- **Point operations**: Upsert and delete vector points
- **Search**: Semantic search using cosine similarity
#### Example Usage
```rust
let client = get_qdrant_client(&state)?;
// Create collection
client.create_collection("kb_bot123_enrollpdfs", 1536).await?;
// Search
let results = client.search("kb_bot123_enrollpdfs", query_vector, 5).await?;
```
### 4. Embeddings Generator (`src/kb/embeddings.rs`)
Handles text embedding and document indexing:
- **Chunking**: Splits documents into 512-character chunks with 50-char overlap
- **Embedding**: Generates vectors using local LLM server
- **Indexing**: Stores chunks with metadata in Qdrant
#### Document Processing
```rust
// Index a document
index_document(&state, "kb_bot_collection", "file.pdf", &content).await?;
// Search for similar documents
let results = search_similar(&state, "kb_bot_collection", "query", 5).await?;
```
### 5. BASIC Compiler (`src/basic/compiler/mod.rs`)
Compiles BASIC scripts and generates tool definitions:
#### Input: BASIC Script with Metadata
```basic
PARAM name AS string LIKE "Abreu Silva" DESCRIPTION "Required full name"
PARAM birthday AS date LIKE "23/09/2001" DESCRIPTION "Birth date in DD/MM/YYYY"
PARAM email AS string LIKE "user@example.com" DESCRIPTION "Email address"
DESCRIPTION "Enrollment process for new users"
// Script logic here
SAVE "enrollments.csv", id, name, birthday, email
TALK "Thanks, you are enrolled!"
SET_KB "enrollpdfs"
```
#### Output: Multiple Files
1. **enrollment.ast**: Compiled Rhai AST
2. **enrollment.mcp.json**: MCP tool definition
3. **enrollment.tool.json**: OpenAI tool definition
#### MCP Tool Format
```json
{
"name": "enrollment",
"description": "Enrollment process for new users",
"input_schema": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Required full name",
"example": "Abreu Silva"
},
"birthday": {
"type": "string",
"description": "Birth date in DD/MM/YYYY",
"example": "23/09/2001"
}
},
"required": ["name", "birthday", "email"]
}
}
```
### 6. Prompt Processor (`src/context/prompt_processor.rs`)
Enhances queries with context based on answer mode:
#### Answer Modes
| Mode | Value | Description |
|------|-------|-------------|
| Direct | 0 | No additional context, direct LLM response |
| WithTools | 1 | Include available tools in prompt |
| DocumentsOnly | 2 | Search KB only, no LLM generation |
| WebSearch | 3 | Include web search results |
| Mixed | 4 | Combine KB documents + tools (context-aware) |
#### Mixed Mode Flow
```
User Query
│
▼
┌─────────────────────────┐
│ Prompt Processor │
│ (Answer Mode: Mixed) │
└─────────────────────────┘
│
├──► Search KB Documents (Qdrant)
│ └─► Returns relevant chunks
│
├──► Get Available Tools (Session Context)
│ └─► Returns tool definitions
│
▼
┌─────────────────────────┐
│ Enhanced Prompt │
│ • System Prompt │
│ • Document Context │
│ • Available Tools │
│ • User Query │
└─────────────────────────┘
```
## BASIC Keywords
### SET_KB
Activates a KB collection for the current session.
```basic
SET_KB "enrollpdfs"
```
- Creates/ensures Qdrant collection exists
- Updates session context with active collection
- Documents in `.gbkb/enrollpdfs/` are indexed
### ADD_KB
Adds an additional KB collection (can have multiple).
```basic
ADD_KB "productbrochurespdfsanddocs"
```
### ADD_TOOL
Compiles and registers a BASIC tool.
```basic
ADD_TOOL "enrollment.bas"
```
Downloads from MinIO (`.gbdialog/enrollment.bas`), compiles to:
- `./work/{bot_id}.gbai/{bot_id}.gbdialog/enrollment.ast`
- `./work/{bot_id}.gbai/{bot_id}.gbdialog/enrollment.mcp.json`
- `./work/{bot_id}.gbai/{bot_id}.gbdialog/enrollment.tool.json`
#### With MCP Endpoint
```basic
ADD_TOOL "enrollment.bas" as MCP
```
Creates an HTTP endpoint at `/default/enrollment` that:
- Accepts JSON matching the tool schema
- Executes the BASIC script
- Returns the result
### ADD_WEBSITE
Crawls and indexes a website for the current session.
```basic
ADD_WEBSITE "https://example.com/docs"
```
- Fetches HTML content
- Extracts readable text (removes scripts, styles)
- Creates temporary Qdrant collection
- Indexes content with embeddings
- Available for remainder of session
## Database Schema
### kb_documents
Stores metadata about indexed documents:
```sql
CREATE TABLE kb_documents (
id UUID PRIMARY KEY,
bot_id UUID NOT NULL,
collection_name TEXT NOT NULL,
file_path TEXT NOT NULL,
file_size BIGINT NOT NULL,
file_hash TEXT NOT NULL,
first_published_at TIMESTAMPTZ NOT NULL,
last_modified_at TIMESTAMPTZ NOT NULL,
indexed_at TIMESTAMPTZ,
metadata JSONB DEFAULT '{}',
UNIQUE(bot_id, collection_name, file_path)
);
```
### kb_collections
Stores KB collection information:
```sql
CREATE TABLE kb_collections (
id UUID PRIMARY KEY,
bot_id UUID NOT NULL,
name TEXT NOT NULL,
folder_path TEXT NOT NULL,
qdrant_collection TEXT NOT NULL,
document_count INTEGER NOT NULL DEFAULT 0,
UNIQUE(bot_id, name)
);
```
### basic_tools
Stores compiled BASIC tools:
```sql
CREATE TABLE basic_tools (
id UUID PRIMARY KEY,
bot_id UUID NOT NULL,
tool_name TEXT NOT NULL,
file_path TEXT NOT NULL,
ast_path TEXT NOT NULL,
mcp_json JSONB,
tool_json JSONB,
compiled_at TIMESTAMPTZ NOT NULL,
is_active BOOLEAN NOT NULL DEFAULT true,
UNIQUE(bot_id, tool_name)
);
```
## Workflow Examples
### Example 1: Enrollment with KB
**File Structure:**
```
bot.gbai/
├── .gbkb/
│ └── enrollpdfs/
│ ├── enrollment_guide.pdf
│ ├── requirements.pdf
│ └── faq.pdf
├── .gbdialog/
│ ├── start.bas
│ └── enrollment.bas
```
**start.bas:**
```basic
ADD_TOOL "enrollment.bas" as MCP
ADD_KB "enrollpdfs"
```
**enrollment.bas:**
```basic
PARAM name AS string LIKE "John Doe" DESCRIPTION "Full name"
PARAM email AS string LIKE "john@example.com" DESCRIPTION "Email"
DESCRIPTION "Enrollment process with KB support"
// Validate input
IF name = "" THEN
TALK "Please provide your name"
EXIT
END IF
// Save to database
SAVE "enrollments.csv", name, email
// Set KB for enrollment docs
SET_KB "enrollpdfs"
TALK "Thanks! You can now ask me about enrollment procedures."
```
**User Interaction:**
1. User: "I want to enroll"
2. Bot calls `enrollment` tool, collects parameters
3. After enrollment, SET_KB activates `enrollpdfs` collection
4. User: "What documents do I need?"
5. Bot searches KB (mode=2 or 4), finds relevant PDFs, responds with info
### Example 2: Product Support with Web Content
**support.bas:**
```basic
PARAM product AS string LIKE "fax" DESCRIPTION "Product name"
DESCRIPTION "Get product information"
// Find in database
price = -1
productRecord = FIND "products.csv", "name = ${product}"
IF productRecord THEN
price = productRecord.price
END IF
// Add product documentation website
ADD_WEBSITE "https://example.com/products/${product}"
// Add product brochures KB
SET_KB "productbrochurespdfsanddocs"
RETURN price
```
**User Flow:**
1. User: "What's the price of a fax machine?"
2. Tool executes, finds price in CSV
3. ADD_WEBSITE indexes product page
4. SET_KB activates brochures collection
5. User: "How do I set it up?"
6. Prompt processor (Mixed mode) searches both:
- Temporary website collection
- Product brochures KB
7. Returns setup instructions from indexed sources
## Configuration
### Environment Variables
```bash
# Qdrant Configuration
QDRANT_URL=http://localhost:6333
# LLM for Embeddings
LLM_URL=http://localhost:8081
# MinIO Configuration (from config)
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_ORG_PREFIX=org1_
# Database
DATABASE_URL=postgresql://user:pass@localhost/botserver
```
### Answer Mode Selection
Set in session's `answer_mode` field:
```rust
// Example: Update session to Mixed mode
session.answer_mode = 4;
```
Or via API when creating session:
```json
POST /sessions
{
"user_id": "...",
"bot_id": "...",
"answer_mode": 4
}
```
## Security Considerations
1. **Path Traversal Protection**: All file paths validated to prevent `..` attacks
2. **Safe Tool Paths**: Tools must be in `.gbdialog/` folder
3. **URL Validation**: ADD_WEBSITE only allows HTTP/HTTPS URLs
4. **Bucket Isolation**: Each organization has separate MinIO bucket
5. **Hash Verification**: File changes detected by SHA256 hash
## Performance Tuning
### KB Manager
- **Poll Interval**: 30 seconds (adjustable in `kb/mod.rs`)
- **Chunk Size**: 512 characters (in `kb/embeddings.rs`)
- **Chunk Overlap**: 50 characters
### MinIO Handler
- **Poll Interval**: 15 seconds (adjustable in `kb/minio_handler.rs`)
- **State Caching**: File states cached in memory
### Qdrant
- **Vector Size**: 1536 (OpenAI ada-002 compatible)
- **Distance Metric**: Cosine similarity
- **Search Limit**: Configurable per query
## Troubleshooting
### Documents Not Indexing
1. Check MinIO handler is watching correct prefix:
```rust
minio_handler.watch_prefix(".gbkb/").await;
```
2. Verify Qdrant connection:
```bash
curl http://localhost:6333/collections
```
3. Check logs for indexing errors:
```
grep "Indexing document" botserver.log
```
### Tools Not Compiling
1. Verify PARAM syntax is correct
2. Check tool file is in `.gbdialog/` folder
3. Ensure work directory exists and is writable
4. Review compilation logs
### KB Search Not Working
1. Verify collection exists in session context
2. Check Qdrant collection created:
```bash
curl http://localhost:6333/collections/{collection_name}
```
3. Ensure embeddings are being generated (check LLM server)
## Future Enhancements
1. **Incremental Indexing**: Only reindex changed chunks
2. **Document Deduplication**: Detect and merge duplicate content
3. **Advanced Crawling**: Follow links, handle JavaScript
4. **Tool Versioning**: Track tool versions and changes
5. **KB Analytics**: Track search queries and document usage
6. **Automatic Tool Discovery**: Scan `.gbdialog/` on startup
7. **Distributed Indexing**: Scale across multiple workers
8. **Real-time Notifications**: WebSocket updates when KB changes
## References
- **Qdrant Documentation**: https://qdrant.tech/documentation/
- **Model Context Protocol**: https://modelcontextprotocol.io/
- **MinIO Documentation**: https://min.io/docs/
- **Rhai Scripting**: https://rhai.rs/book/
## Support
For issues or questions:
- GitHub Issues: https://github.com/GeneralBots/BotServer/issues
- Documentation: https://docs.generalbots.ai/

398
docs/QUICKSTART_KB_TOOLS.md
View file

@ -1,398 +0,0 @@
# Quick Start: KB and Tools System
## 🎯 Overview
O sistema KB (Knowledge Base) e Tools é completamente **automático e dirigido pelo Drive**:
- **Monitora o Drive (MinIO/S3)** automaticamente
- **Compila tools** quando `.bas` é alterado em `.gbdialog/`
- **Indexa documentos** quando arquivos mudam em `.gbkb/`
- **KB por usuário**, não por sessão
- **Tools por sessão**, não compilados no runtime
## 🚀 Quick Setup (5 minutos)
### 1. Install Dependencies
```bash
# Start required services
docker-compose up -d qdrant postgres
# MinIO (or S3-compatible storage)
docker run -p 9000:9000 -p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin \
minio/minio server /data --console-address ":9001"
```
### 2. Configure Environment
```bash
# .env
QDRANT_URL=http://localhost:6333
LLM_URL=http://localhost:8081
DRIVE_ENDPOINT=localhost:9000
DRIVE_ACCESS_KEY=minioadmin
DRIVE_SECRET_KEY=minioadmin
DATABASE_URL=postgresql://user:pass@localhost/botserver
```
**Nota:** Use nomes genéricos como `DRIVE_*` ao invés de `MINIO_*` quando possível.
### 3. Run Database Migration
```sql
-- Run migration (compatível SQLite e Postgres)
sqlite3 botserver.db < migrations/6.0.3.sql
-- ou
psql -d botserver -f migrations/6.0.3.sql
```
### 4. Create Bot Structure in Drive
Create bucket: `org1_default.gbai`
```
org1_default.gbai/
├── .gbkb/ # Knowledge Base folders
│ ├── enrollpdfs/ # Collection 1 (auto-indexed)
│ │ ├── guide.pdf
│ │ └── requirements.pdf
│ └── productdocs/ # Collection 2 (auto-indexed)
│ └── catalog.pdf
└── .gbdialog/ # BASIC scripts (auto-compiled)
├── start.bas
├── enrollment.bas
└── pricing.bas
```
## 📝 Create Your First Tool (2 minutes)
### enrollment.bas
```basic
PARAM name AS string LIKE "John Doe" DESCRIPTION "Full name"
PARAM email AS string LIKE "john@example.com" DESCRIPTION "Email address"
DESCRIPTION "User enrollment process"
SAVE "enrollments.csv", name, email
TALK "Enrolled! You can ask me about enrollment procedures."
RETURN "success"
```
### start.bas
```basic
REM ADD_TOOL apenas ASSOCIA a tool à sessão (não compila!)
REM A compilação acontece automaticamente quando o arquivo muda no Drive
ADD_TOOL "enrollment"
ADD_TOOL "pricing"
REM ADD_KB é por USER, não por sessão
REM Basta existir em .gbkb/ que já está indexado
ADD_KB "enrollpdfs"
TALK "Hi! I can help with enrollment and pricing."
```
## 🔄 How It Works: Drive-First Approach
```
┌─────────────────────────────────────────────────────┐
│ 1. Upload file.pdf to .gbkb/enrollpdfs/ │
│ ↓ │
│ 2. DriveMonitor detecta mudança (30s polling) │
│ ↓ │
│ 3. Automaticamente indexa no Qdrant │
│ ↓ │
│ 4. Metadados salvos no banco (kb_documents) │
│ ↓ │
│ 5. KB está disponível para TODOS os usuários │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ 1. Upload enrollment.bas to .gbdialog/ │
│ ↓ │
│ 2. DriveMonitor detecta mudança (30s polling) │
│ ↓ │
│ 3. Automaticamente compila para .ast │
│ ↓ │
│ 4. Gera .mcp.json e .tool.json (se tem PARAM) │
│ ↓ │
│ 5. Salvo em ./work/default.gbai/default.gbdialog/ │
│ ↓ │
│ 6. Metadados salvos no banco (basic_tools) │
│ ↓ │
│ 7. Tool compilada e pronta para uso │
└─────────────────────────────────────────────────────┘
```
## 🎯 Keywords BASIC
### ADD_TOOL (Associa tool à sessão)
```basic
ADD_TOOL "enrollment" # Apenas o nome, sem .bas
```
**O que faz:**
- Associa a tool **já compilada** com a sessão atual
- NÃO compila (isso é feito automaticamente pelo DriveMonitor)
- Armazena em `session_tool_associations` table
**Importante:** A tool deve existir em `basic_tools` (já compilada).
### ADD_KB (Adiciona KB para o usuário)
```basic
ADD_KB "enrollpdfs"
```
**O que faz:**
- Associa KB com o **usuário** (não sessão!)
- Armazena em `user_kb_associations` table
- KB já deve estar indexado (arquivos em `.gbkb/enrollpdfs/`)
### ADD_WEBSITE (Adiciona website como KB para o usuário)
```basic
ADD_WEBSITE "https://docs.example.com"
```
**O que faz:**
- Faz crawling do website (usa `WebCrawler`)
- Cria KB temporário para o usuário
- Indexa no Qdrant
- Armazena em `user_kb_associations` com `is_website=1`
## 📊 Database Tables (SQLite/Postgres Compatible)
### kb_documents (Metadados de documentos indexados)
```sql
CREATE TABLE kb_documents (
id TEXT PRIMARY KEY,
bot_id TEXT NOT NULL,
user_id TEXT NOT NULL,
collection_name TEXT NOT NULL,
file_path TEXT NOT NULL,
file_size INTEGER NOT NULL,
file_hash TEXT NOT NULL,
indexed_at TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
```
### basic_tools (Tools compiladas)
```sql
CREATE TABLE basic_tools (
id TEXT PRIMARY KEY,
bot_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
file_path TEXT NOT NULL,
ast_path TEXT NOT NULL,
file_hash TEXT NOT NULL,
mcp_json TEXT,
tool_json TEXT,
compiled_at TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1
);
```
### user_kb_associations (KB por usuário)
```sql
CREATE TABLE user_kb_associations (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
bot_id TEXT NOT NULL,
kb_name TEXT NOT NULL,
is_website INTEGER NOT NULL DEFAULT 0,
website_url TEXT,
UNIQUE(user_id, bot_id, kb_name)
);
```
### session_tool_associations (Tools por sessão)
```sql
CREATE TABLE session_tool_associations (
id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
added_at TEXT NOT NULL,
UNIQUE(session_id, tool_name)
);
```
## 🔧 Drive Monitor (Automatic Background Service)
O `DriveMonitor` roda automaticamente ao iniciar o servidor:
```rust
// In main.rs
let bucket_name = format!("{}default.gbai", cfg.org_prefix);
let drive_monitor = Arc::new(DriveMonitor::new(app_state, bucket_name));
let _handle = drive_monitor.spawn();
```
**Monitora:**
- `.gbdialog/*.bas` → Compila automaticamente
- `.gbkb/*/*.{pdf,txt,md}` → Indexa automaticamente
**Intervalo:** 30 segundos (ajustável)
## 📚 Example: Complete Enrollment Flow
### 1. Upload enrollment.bas to Drive
```bash
mc cp enrollment.bas local/org1_default.gbai/.gbdialog/
```
### 2. Wait for Compilation (30s max)
```
[INFO] New BASIC tool detected: .gbdialog/enrollment.bas
[INFO] Tool compiled successfully: enrollment
[INFO] AST: ./work/default.gbai/default.gbdialog/enrollment.ast
[INFO] MCP tool definition generated
```
### 3. Upload KB documents
```bash
mc cp guide.pdf local/org1_default.gbai/.gbkb/enrollpdfs/
mc cp faq.pdf local/org1_default.gbai/.gbkb/enrollpdfs/
```
### 4. Wait for Indexing (30s max)
```
[INFO] New KB document detected: .gbkb/enrollpdfs/guide.pdf
[INFO] Extracted 5420 characters from .gbkb/enrollpdfs/guide.pdf
[INFO] Document indexed successfully: .gbkb/enrollpdfs/guide.pdf
```
### 5. Use in BASIC Script
```basic
REM start.bas
ADD_TOOL "enrollment"
ADD_KB "enrollpdfs"
TALK "Ready to help with enrollment!"
```
### 6. User Interaction
```
User: "I want to enroll"
Bot: [Calls enrollment tool, collects info]
User: "What documents do I need?"
Bot: [Searches enrollpdfs KB, returns relevant info from guide.pdf]
```
## 🎓 Best Practices
### ✅ DO
- Upload files to Drive and let the system auto-compile/index
- Use generic names (Drive, Cache) when possible
- Use `ADD_KB` for persistent user knowledge
- Use `ADD_TOOL` to activate tools in session
- Keep tools in `.gbdialog/`, KB docs in `.gbkb/`
### ❌ DON'T
- Don't try to compile tools in runtime (it's automatic!)
- Don't use session for KB (it's user-based)
- Don't use `SET_KB` and `ADD_KB` together (they do the same)
- Don't expect instant updates (30s polling interval)
## 🔍 Monitoring
### Check Compiled Tools
```bash
ls -la ./work/default.gbai/default.gbdialog/
# Should see:
# - enrollment.ast
# - enrollment.mcp.json
# - enrollment.tool.json
# - pricing.ast
# - pricing.mcp.json
# - pricing.tool.json
```
### Check Indexed Documents
```bash
# Query Qdrant
curl http://localhost:6333/collections
# Should see collections like:
# - kb_default_enrollpdfs
# - kb_default_productdocs
```
### Check Database
```sql
-- Compiled tools
SELECT tool_name, compiled_at, is_active FROM basic_tools;
-- Indexed documents
SELECT file_path, indexed_at FROM kb_documents;
-- User KBs
SELECT user_id, kb_name, is_website FROM user_kb_associations;
-- Session tools
SELECT session_id, tool_name FROM session_tool_associations;
```
## 🐛 Troubleshooting
### Tool not compiling?
1. Check file is in `.gbdialog/` folder
2. File must end with `.bas`
3. Wait 30 seconds for DriveMonitor poll
4. Check logs: `grep "Compiling BASIC tool" botserver.log`
### Document not indexing?
1. Check file is in `.gbkb/collection_name/` folder
2. File must be `.pdf`, `.txt`, or `.md`
3. Wait 30 seconds for DriveMonitor poll
4. Check logs: `grep "Indexing KB document" botserver.log`
### ADD_TOOL fails?
1. Tool must be already compiled (check `basic_tools` table)
2. Use only tool name: `ADD_TOOL "enrollment"` (not `.bas`)
3. Check if `is_active=1` in database
### KB search not working?
1. Use `ADD_KB` in user's script (not session)
2. Check collection exists in Qdrant
3. Verify `user_kb_associations` has entry
4. Check answer_mode (use 2 or 4 for KB)
## 🆘 Support
- Full Docs: `docs/KB_AND_TOOLS.md`
- Examples: `examples/`
- Deployment: `docs/DEPLOYMENT_CHECKLIST.md`
---
**The system is fully automatic and drive-first!** 🚀
Just upload to Drive → DriveMonitor handles the rest.

620
docs/TOOL_MANAGEMENT.md
View file

@ -1,620 +0,0 @@
# Tool Management System
## Overview
The Bot Server now supports **multiple tool associations** per user session. This allows users to dynamically load, manage, and use multiple BASIC tools during a single conversation without needing to restart or change sessions.
## Features
- **Multiple Tools per Session**: Associate multiple compiled BASIC tools with a single conversation
- **Dynamic Management**: Add or remove tools on-the-fly during a conversation
- **Session Isolation**: Each session has its own independent set of active tools
- **Persistent Associations**: Tool associations are stored in the database and survive across requests
- **Real Database Implementation**: No SQL placeholders - fully implemented with Diesel ORM
## Database Schema
### `session_tool_associations` Table
```sql
CREATE TABLE IF NOT EXISTS session_tool_associations (
id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
added_at TEXT NOT NULL,
UNIQUE(session_id, tool_name)
);
```
**Indexes:**
- `idx_session_tool_session` on `session_id`
- `idx_session_tool_name` on `tool_name`
The UNIQUE constraint ensures a tool cannot be added twice to the same session.
## BASIC Keywords
### `ADD_TOOL`
Adds a compiled tool to the current session, making it available for the LLM to call.
**Syntax:**
```basic
ADD_TOOL "<path_to_tool>"
```
**Example:**
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
ADD_TOOL ".gbdialog/payment.bas"
ADD_TOOL ".gbdialog/support.bas"
```
**Behavior:**
- Validates that the tool exists in the `basic_tools` table
- Verifies the tool is active (`is_active = 1`)
- Checks the tool belongs to the current bot
- Inserts into `session_tool_associations` table
- Returns success message or error if tool doesn't exist
- If tool is already associated, reports it's already active
**Returns:**
- Success: `"Tool 'enrollment' is now available in this conversation"`
- Already added: `"Tool 'enrollment' is already available in this conversation"`
- Error: `"Tool 'enrollment' is not available. Make sure the tool file is compiled and active."`
---
### `REMOVE_TOOL`
Removes a tool association from the current session.
**Syntax:**
```basic
REMOVE_TOOL "<path_to_tool>"
```
**Example:**
```basic
REMOVE_TOOL ".gbdialog/support.bas"
```
**Behavior:**
- Removes the tool from `session_tool_associations` for this session
- Does not delete the compiled tool itself
- Only affects the current session
**Returns:**
- Success: `"Tool 'support' has been removed from this conversation"`
- Not found: `"Tool 'support' was not active in this conversation"`
---
### `CLEAR_TOOLS`
Removes all tool associations from the current session.
**Syntax:**
```basic
CLEAR_TOOLS
```
**Example:**
```basic
CLEAR_TOOLS
```
**Behavior:**
- Removes all entries in `session_tool_associations` for this session
- Does not affect other sessions
- Does not delete compiled tools
**Returns:**
- Success: `"All 3 tool(s) have been removed from this conversation"`
- No tools: `"No tools were active in this conversation"`
---
### `LIST_TOOLS`
Lists all tools currently associated with the session.
**Syntax:**
```basic
LIST_TOOLS
```
**Example:**
```basic
LIST_TOOLS
```
**Output:**
```
Active tools in this conversation (3):
1. enrollment
2. payment
3. analytics
```
**Returns:**
- With tools: Lists all active tools with numbering
- No tools: `"No tools are currently active in this conversation"`
---
## How It Works
### Tool Loading Flow
1. **User calls `ADD_TOOL` in BASIC script**
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
```
2. **System validates tool exists**
- Queries `basic_tools` table
- Checks `bot_id` matches current bot
- Verifies `is_active = 1`
3. **Association is created**
- Inserts into `session_tool_associations`
- Uses UNIQUE constraint to prevent duplicates
- Stores session_id, tool_name, and timestamp
4. **LLM requests include tools**
- When processing prompts, system loads all tools from `session_tool_associations`
- Tools are added to the LLM's available function list
- LLM can now call any associated tool
### Integration with Prompt Processor
The `PromptProcessor::get_available_tools()` method now:
1. Loads tool stack from bot configuration (existing behavior)
2. **NEW**: Queries `session_tool_associations` for the current session
3. Adds all associated tools to the available tools list
4. Maintains backward compatibility with legacy `current_tool` field
**Code Example:**
```rust
// From src/context/prompt_processor.rs
if let Ok(mut conn) = self.state.conn.lock() {
match get_session_tools(&mut *conn, &session.id) {
Ok(session_tools) => {
for tool_name in session_tools {
if !tools.iter().any(|t| t.tool_name == tool_name) {
tools.push(ToolContext {
tool_name: tool_name.clone(),
description: format!("Tool: {}", tool_name),
endpoint: format!("/default/{}", tool_name),
});
}
}
}
Err(e) => error!("Failed to load session tools: {}", e),
}
}
```
---
## Rust API
### Public Functions
All functions are in `botserver/src/basic/keywords/add_tool.rs`:
```rust
/// Get all tools associated with a session
pub fn get_session_tools(
conn: &mut PgConnection,
session_id: &Uuid,
) -> Result<Vec<String>, diesel::result::Error>
/// Remove a tool association from a session
pub fn remove_session_tool(
conn: &mut PgConnection,
session_id: &Uuid,
tool_name: &str,
) -> Result<usize, diesel::result::Error>
/// Clear all tool associations for a session
pub fn clear_session_tools(
conn: &mut PgConnection,
session_id: &Uuid,
) -> Result<usize, diesel::result::Error>
```
**Usage Example:**
```rust
use crate::basic::keywords::add_tool::get_session_tools;
let tools = get_session_tools(&mut conn, &session_id)?;
for tool_name in tools {
println!("Active tool: {}", tool_name);
}
```
---
## Use Cases
### 1. Progressive Tool Loading
Start with basic tools and add more as needed:
```basic
REM Start with customer service tool
ADD_TOOL ".gbdialog/customer_service.bas"
REM If user needs technical support, add that tool
IF user_needs_technical_support THEN
ADD_TOOL ".gbdialog/technical_support.bas"
END IF
REM If billing question, add payment tool
IF user_asks_about_billing THEN
ADD_TOOL ".gbdialog/billing.bas"
END IF
```
### 2. Context-Aware Tool Management
Different tools for different conversation stages:
```basic
REM Initial greeting phase
ADD_TOOL ".gbdialog/greeting.bas"
HEAR "start"
REM Main interaction phase
REMOVE_TOOL ".gbdialog/greeting.bas"
ADD_TOOL ".gbdialog/enrollment.bas"
ADD_TOOL ".gbdialog/faq.bas"
HEAR "continue"
REM Closing phase
CLEAR_TOOLS
ADD_TOOL ".gbdialog/feedback.bas"
HEAR "finish"
```
### 3. Department-Specific Tools
Route to different tool sets based on department:
```basic
GET "/api/user/department" AS department
IF department = "sales" THEN
ADD_TOOL ".gbdialog/lead_capture.bas"
ADD_TOOL ".gbdialog/quote_generator.bas"
ADD_TOOL ".gbdialog/crm_integration.bas"
ELSE IF department = "support" THEN
ADD_TOOL ".gbdialog/ticket_system.bas"
ADD_TOOL ".gbdialog/knowledge_base.bas"
ADD_TOOL ".gbdialog/escalation.bas"
END IF
```
### 4. A/B Testing Tools
Test different tool combinations:
```basic
GET "/api/user/experiment_group" AS group
IF group = "A" THEN
ADD_TOOL ".gbdialog/tool_variant_a.bas"
ELSE
ADD_TOOL ".gbdialog/tool_variant_b.bas"
END IF
REM Both groups get common tools
ADD_TOOL ".gbdialog/common_tools.bas"
```
---
## Answer Modes
The system respects the session's `answer_mode`:
- **Mode 0 (Direct)**: No tools used
- **Mode 1 (WithTools)**: Uses associated tools + legacy `current_tool`
- **Mode 2 (DocumentsOnly)**: Only KB documents, no tools
- **Mode 3 (WebSearch)**: Web search enabled
- **Mode 4 (Mixed)**: Tools from `session_tool_associations` + KB documents
Set answer mode via session configuration or dynamically.
---
## Best Practices
### 1. **Validate Before Use**
Always check if a tool is successfully added:
```basic
ADD_TOOL ".gbdialog/payment.bas"
LIST_TOOLS REM Verify it was added
```
### 2. **Clean Up When Done**
Remove tools that are no longer needed to improve LLM performance:
```basic
REMOVE_TOOL ".gbdialog/onboarding.bas"
```
### 3. **Use LIST_TOOLS for Debugging**
When developing, list tools to verify state:
```basic
LIST_TOOLS
PRINT "Current tools listed above"
```
### 4. **Tool Names are Simple**
Tool names are extracted from paths automatically:
- `.gbdialog/enrollment.bas` → `enrollment`
- `payment.bas` → `payment`
### 5. **Session Isolation**
Each session maintains its own tool list. Tools added in one session don't affect others.
### 6. **Compile Before Adding**
Ensure tools are compiled and present in the `basic_tools` table before attempting to add them. The DriveMonitor service handles compilation automatically when `.bas` files are saved.
---
## Migration Guide
### Upgrading from Single Tool (`current_tool`)
**Before (Legacy):**
```rust
// Single tool stored in session.current_tool
session.current_tool = Some("enrollment".to_string());
```
**After (Multi-Tool):**
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
ADD_TOOL ".gbdialog/payment.bas"
ADD_TOOL ".gbdialog/support.bas"
```
**Backward Compatibility:**
The system still supports the legacy `current_tool` field. If set, it will be included in the available tools list alongside tools from `session_tool_associations`.
---
## Technical Implementation Details
### Database Operations
All operations use Diesel ORM with proper error handling:
```rust
// Insert with conflict resolution
diesel::insert_into(session_tool_associations::table)
.values((/* ... */))
.on_conflict((session_id, tool_name))
.do_nothing()
.execute(&mut *conn)
// Delete specific tool
diesel::delete(
session_tool_associations::table
.filter(session_id.eq(&session_id_str))
.filter(tool_name.eq(tool_name))
).execute(&mut *conn)
// Load all tools
session_tool_associations::table
.filter(session_id.eq(&session_id_str))
.select(tool_name)
.load::<String>(&mut *conn)
```
### Thread Safety
All operations use Arc<Mutex<PgConnection>> for thread-safe database access:
```rust
let mut conn = state.conn.lock().map_err(|e| {
error!("Failed to acquire database lock: {}", e);
format!("Database connection error: {}", e)
})?;
```
### Async Execution
Keywords spawn async tasks using Tokio runtime to avoid blocking the Rhai engine:
```rust
std::thread::spawn(move || {
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build();
// ... execute async operation
});
```
---
## Error Handling
### Common Errors
1. **Tool Not Found**
- Message: `"Tool 'xyz' is not available. Make sure the tool file is compiled and active."`
- Cause: Tool doesn't exist in `basic_tools` or is inactive
- Solution: Compile the tool or check bot_id matches
2. **Database Lock Error**
- Message: `"Database connection error: ..."`
- Cause: Failed to acquire database mutex
- Solution: Check database connection health
3. **Timeout**
- Message: `"ADD_TOOL timed out"`
- Cause: Operation took longer than 10 seconds
- Solution: Check database performance
### Error Recovery
All operations are atomic - if they fail, no partial state is committed:
```basic
ADD_TOOL ".gbdialog/nonexistent.bas"
REM Error returned, no changes made
LIST_TOOLS
REM Still shows previous tools only
```
---
## Performance Considerations
### Database Indexes
The following indexes ensure fast lookups:
- `idx_session_tool_session`: Fast retrieval of all tools for a session
- `idx_session_tool_name`: Fast tool name lookups
- UNIQUE constraint on (session_id, tool_name): Prevents duplicates
### Query Optimization
Tools are loaded once per prompt processing:
```rust
// Efficient batch load
let tools = get_session_tools(&mut conn, &session.id)?;
```
### Memory Usage
- Tool associations are lightweight (only stores IDs and names)
- No tool code is duplicated in the database
- Compiled tools are referenced, not copied
---
## Security
### Access Control
- Tools are validated against bot_id
- Users can only add tools belonging to their current bot
- Session isolation prevents cross-session access
### Input Validation
- Tool names are extracted and sanitized
- SQL injection prevented by Diesel parameterization
- Empty tool names are rejected
---
## Testing
### Example Test Script
See `botserver/examples/tool_management_example.bas` for a complete working example.
### Unit Testing
Test the Rust API directly:
```rust
#[test]
fn test_multiple_tool_association() {
let mut conn = establish_connection();
let session_id = Uuid::new_v4();
// Add tools
add_tool(&mut conn, &session_id, "tool1").unwrap();
add_tool(&mut conn, &session_id, "tool2").unwrap();
// Verify
let tools = get_session_tools(&mut conn, &session_id).unwrap();
assert_eq!(tools.len(), 2);
// Remove one
remove_session_tool(&mut conn, &session_id, "tool1").unwrap();
let tools = get_session_tools(&mut conn, &session_id).unwrap();
assert_eq!(tools.len(), 1);
// Clear all
clear_session_tools(&mut conn, &session_id).unwrap();
let tools = get_session_tools(&mut conn, &session_id).unwrap();
assert_eq!(tools.len(), 0);
}
```
---
## Future Enhancements
Potential improvements:
1. **Tool Priority/Ordering**: Specify which tools to try first
2. **Tool Groups**: Add/remove sets of related tools together
3. **Auto-Cleanup**: Remove tool associations when session ends
4. **Tool Statistics**: Track which tools are used most frequently
5. **Conditional Tool Loading**: Load tools based on LLM decisions
6. **Tool Permissions**: Fine-grained control over which users can use which tools
---
## Troubleshooting
### Tools Not Appearing
1. Check compilation:
```sql
SELECT * FROM basic_tools WHERE tool_name = 'enrollment';
```
2. Verify bot_id matches:
```sql
SELECT bot_id FROM basic_tools WHERE tool_name = 'enrollment';
```
3. Check is_active flag:
```sql
SELECT is_active FROM basic_tools WHERE tool_name = 'enrollment';
```
### Tools Not Being Called
1. Verify answer_mode is 1 or 4
2. Check tool is in session associations:
```sql
SELECT * FROM session_tool_associations WHERE session_id = '<your-session-id>';
```
3. Review LLM logs to see if tool was included in prompt
### Database Issues
Check connection:
```bash
psql -h localhost -U your_user -d your_database
\dt session_tool_associations
```
---
## References
- **Schema**: `botserver/migrations/6.0.3.sql`
- **Implementation**: `botserver/src/basic/keywords/add_tool.rs`
- **Prompt Integration**: `botserver/src/context/prompt_processor.rs`
- **Models**: `botserver/src/shared/models.rs`
- **Example**: `botserver/examples/tool_management_example.bas`
---
## License
This feature is part of the Bot Server project. See the main LICENSE file for details.

176
docs/TOOL_MANAGEMENT_QUICK_REF.md
View file

@ -1,176 +0,0 @@
# Tool Management Quick Reference
## 🚀 Quick Start
### Add a Tool
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
```
### Remove a Tool
```basic
REMOVE_TOOL ".gbdialog/enrollment.bas"
```
### List Active Tools
```basic
LIST_TOOLS
```
### Clear All Tools
```basic
CLEAR_TOOLS
```
---
## 📋 Common Patterns
### Multiple Tools in One Session
```basic
ADD_TOOL ".gbdialog/enrollment.bas"
ADD_TOOL ".gbdialog/payment.bas"
ADD_TOOL ".gbdialog/support.bas"
LIST_TOOLS
```
### Progressive Loading
```basic
REM Start with basic tool
ADD_TOOL ".gbdialog/greeting.bas"
REM Add more as needed
IF user_needs_help THEN
ADD_TOOL ".gbdialog/support.bas"
END IF
```
### Tool Rotation
```basic
REM Switch tools for different phases
REMOVE_TOOL ".gbdialog/onboarding.bas"
ADD_TOOL ".gbdialog/main_menu.bas"
```
---
## ⚡ Key Features
- ✅ **Multiple tools per session** - No limit on number of tools
- ✅ **Dynamic management** - Add/remove during conversation
- ✅ **Session isolation** - Each session has independent tool list
- ✅ **Persistent** - Survives across requests
- ✅ **Real database** - Fully implemented with Diesel ORM
---
## 🔍 What Happens Behind the Scenes
1. **ADD_TOOL** → Validates tool exists → Inserts into `session_tool_associations` table
2. **Prompt Processing** → Loads all tools for session → LLM can call them
3. **REMOVE_TOOL** → Deletes association → Tool no longer available
4. **CLEAR_TOOLS** → Removes all associations for session
---
## 📊 Database Table
```sql
CREATE TABLE session_tool_associations (
id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
added_at TEXT NOT NULL,
UNIQUE(session_id, tool_name)
);
```
---
## 🎯 Use Cases
### Customer Service Bot
```basic
ADD_TOOL ".gbdialog/faq.bas"
ADD_TOOL ".gbdialog/ticket_system.bas"
ADD_TOOL ".gbdialog/escalation.bas"
```
### E-commerce Bot
```basic
ADD_TOOL ".gbdialog/product_search.bas"
ADD_TOOL ".gbdialog/cart_management.bas"
ADD_TOOL ".gbdialog/checkout.bas"
ADD_TOOL ".gbdialog/order_tracking.bas"
```
### HR Bot
```basic
ADD_TOOL ".gbdialog/leave_request.bas"
ADD_TOOL ".gbdialog/payroll_info.bas"
ADD_TOOL ".gbdialog/benefits.bas"
```
---
## ⚠️ Important Notes
- Tool must be compiled and in `basic_tools` table
- Tool must have `is_active = 1`
- Tool must belong to current bot (`bot_id` match)
- Path can be with or without `.gbdialog/` prefix
- Tool names auto-extracted: `enrollment.bas` → `enrollment`
---
## 🐛 Common Errors
### "Tool not available"
- **Cause**: Tool not compiled or inactive
- **Fix**: Compile the `.bas` file first
### "Database connection error"
- **Cause**: Can't acquire DB lock
- **Fix**: Check database health
### "Timeout"
- **Cause**: Operation took >10 seconds
- **Fix**: Check database performance
---
## 💡 Pro Tips
1. **Verify additions**: Use `LIST_TOOLS` after adding tools
2. **Clean up**: Remove unused tools to improve LLM performance
3. **Session-specific**: Tools don't carry over to other sessions
4. **Backward compatible**: Legacy `current_tool` still works
---
## 📚 More Information
See `TOOL_MANAGEMENT.md` for comprehensive documentation including:
- Complete API reference
- Security details
- Performance optimization
- Testing strategies
- Troubleshooting guide
---
## 🔗 Related Files
- **Example Script**: `examples/tool_management_example.bas`
- **Implementation**: `src/basic/keywords/add_tool.rs`
- **Schema**: `migrations/6.0.3.sql`
- **Models**: `src/shared/models.rs`
---
## 📞 Support
For issues or questions:
1. Check the full documentation in `TOOL_MANAGEMENT.md`
2. Review the example script in `examples/`
3. Check database with: `SELECT * FROM session_tool_associations WHERE session_id = 'your-id';`

201
docs/basic/keywords/PROMPT.md
View file

@ -1,201 +0,0 @@
# Modelo de Prompt para Aprendizado de BASIC em Markdown
## 🎯 **ESTRUTURA PARA APRENDIZ DE BASIC**
```
**CONCEITO BASIC:**
[Nome do conceito ou comando]
**NÍVEL:**
☐ Iniciante ☐ Intermediário ☐ Avançado
**OBJETIVO DE APRENDIZADO:**
[O que você quer entender ou criar]
**CÓDIGO EXEMPLO:**
```basic
[Seu código ou exemplo aqui]
```
**DÚVIDAS ESPECÍFICAS:**
- [Dúvida 1 sobre o conceito]
- [Dúvida 2 sobre sintaxe]
- [Dúvida 3 sobre aplicação]
**CONTEXTO DO PROJETO:**
[Descrição do que está tentando fazer]
**RESULTADO ESPERADO:**
[O que o código deve fazer]
**PARTES QUE NÃO ENTENDE:**
- [Trecho específico do código]
- [Mensagem de erro]
- [Lógica confusa]
```
---
## 📚 **EXEMPLO PRÁTICO: LOOP FOR**
```
**CONCEITO BASIC:**
LOOP FOR
**NÍVEL:**
☒ Iniciante ☐ Intermediário ☐ Avançado
**OBJETIVO DE APRENDIZADO:**
Entender como criar um contador de 1 a 10
**CÓDIGO EXEMPLO:**
```basic
10 FOR I = 1 TO 10
20 PRINT "Número: "; I
30 NEXT I
```
**DÚVIDAS ESPECÍFICAS:**
- O que significa "NEXT I"?
- Posso usar outras letras além de "I"?
- Como fazer contagem regressiva?
**CONTEXTO DO PROJETO:**
Estou criando um programa que lista números
**RESULTADO ESPERADO:**
Que apareça: Número: 1, Número: 2, etc.
**PARTES QUE NÃO ENTENDE:**
- Por que precisa do número 10 na linha 10?
- O que acontece se esquecer o NEXT?
```
---
## 🛠️ **MODELO PARA RESOLVER ERROS**
```
**ERRO NO BASIC:**
[Mensagem de erro ou comportamento estranho]
**MEU CÓDIGO:**
```basic
[Coloque seu código completo]
```
**LINHA COM PROBLEMA:**
[Linha específica onde ocorre o erro]
**COMPORTAMENTO ESPERADO:**
[O que deveria acontecer]
**COMPORTAMENTO ATUAL:**
[O que está acontecendo de errado]
**O QUE JÁ TENTEI:**
- [Tentativa 1 de correção]
- [Tentativa 2]
- [Tentativa 3]
**VERSÃO DO BASIC:**
[QBASIC, GW-BASIC, FreeBASIC, etc.]
```
---
## 📖 **MODELO PARA EXPLICAR COMANDOS**
```
**COMANDO:**
[Nome do comando - ex: PRINT, INPUT, GOTO]
**SYNTAX:**
[Como escrever corretamente]
**PARÂMETROS:**
- Parâmetro 1: [Função]
- Parâmetro 2: [Função]
**EXEMPLO SIMPLES:**
```basic
[Exemplo mínimo e funcional]
```
**EXEMPLO PRÁTICO:**
```basic
[Exemplo em contexto real]
```
**ERROS COMUNS:**
- [Erro frequente 1]
- [Erro frequente 2]
**DICA PARA INICIANTES:**
[Dica simples para não errar]
**EXERCÍCIO SUGERIDO:**
[Pequeno exercício para praticar]
```
---
## 🎨 **FORMATAÇÃO MARKDOWN PARA BASIC**
### **Como documentar seu código em .md:**
```markdown
# [NOME DO PROGRAMA]
## 🎯 OBJETIVO
[O que o programa faz]
## 📋 COMO USAR
1. [Passo 1]
2. [Passo 2]
## 🧩 CÓDIGO FONTE
```basic
[Seu código aqui]
```
## 🔍 EXPLICAÇÃO
- **Linha X**: [Explicação]
- **Linha Y**: [Explicação]
## 🚀 EXEMPLO DE EXECUÇÃO
```
[Saída do programa]
```
```
---
## 🏆 **MODELO DE PROJETO COMPLETO**
```
# PROJETO BASIC: [NOME]
## 📝 DESCRIÇÃO
[Descrição do que o programa faz]
## 🎨 FUNCIONALIDADES
- [ ] Funcionalidade 1
- [ ] Funcionalidade 2
- [ ] Funcionalidade 3
## 🧩 ESTRUTURA DO CÓDIGO
```basic
[Seu código organizado]
```
## 🎯 APRENDIZADOS
- [Conceito 1 aprendido]
- [Conceito 2 aprendido]
## ❓ DÚVIDAS PARA EVOLUIR
- [Dúvida para melhorar]
- [O que gostaria de fazer depois]
```
gerenerate several examples
for this keyword written in rhai do this only for basic audience:

402
docs/basic/keywords/format.md
View file

@ -1,402 +0,0 @@
# 📚 **BASIC LEARNING EXAMPLES - FORMAT Function**
## 🎯 **EXAMPLE 1: BASIC CONCEPT OF FORMAT FUNCTION**
```
**BASIC CONCEPT:**
FORMAT FUNCTION - Value formatting
**LEVEL:**
☒ Beginner ☐ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Understand how to format numbers, dates, and text
**CODE EXAMPLE:**
```basic
10 NUMBER = 1234.56
20 TEXT$ = "John"
30 DATE$ = "2024-03-15 14:30:00"
40
50 PRINT FORMAT(NUMBER, "n") ' 1234.56
60 PRINT FORMAT(NUMBER, "F") ' 1234.56
70 PRINT FORMAT(TEXT$, "Hello @!") ' Hello John!
80 PRINT FORMAT(DATE$, "dd/MM/yyyy") ' 15/03/2024
```
**SPECIFIC QUESTIONS:**
- What's the difference between "n" and "F"?
- What does "@" mean in text?
- How to format dates in Brazilian format?
**PROJECT CONTEXT:**
I need to display data in a nicer way
**EXPECTED RESULT:**
Values formatted according to the pattern
**PARTS I DON'T UNDERSTAND:**
- When to use each type of formatting
- How it works internally
```
---
## 🛠️ **EXAMPLE 2: NUMERIC FORMATTING**
```
**BASIC CONCEPT:**
NUMBER FORMATTING
**LEVEL:**
☒ Beginner ☐ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Learn to format numbers as currency and with separators
**CODE EXAMPLE:**
```basic
10 VALUE = 1234567.89
20
30 PRINT "Standard: "; FORMAT(VALUE, "n") ' 1234567.89
40 PRINT "Decimal: "; FORMAT(VALUE, "F") ' 1234567.89
45 PRINT "Integer: "; FORMAT(VALUE, "f") ' 1234567
50 PRINT "Percentage: "; FORMAT(0.856, "0%") ' 86%
60
70 ' Formatting with locale
80 PRINT "Dollar: "; FORMAT(VALUE, "C2[en]") ' $1,234,567.89
90 PRINT "Real: "; FORMAT(VALUE, "C2[pt]") ' R$ 1.234.567,89
100 PRINT "Euro: "; FORMAT(VALUE, "C2[fr]") ' €1,234,567.89
```
**SPECIFIC QUESTIONS:**
- What does "C2[pt]" mean?
- How to change decimal places?
- Which locales are available?
**PROJECT CONTEXT:**
Multi-currency financial system
**EXPECTED RESULT:**
Numbers formatted according to regional standards
**PARTS I DON'T UNDERSTAND:**
- Syntax of complex patterns
- Differences between locales
```
---
## 📖 **EXAMPLE 3: EXPLAINING FORMAT COMMAND**
```
**COMMAND:**
FORMAT - Formats values
**SYNTAX:**
```basic
RESULT$ = FORMAT(VALUE, PATTERN$)
```
**PARAMETERS:**
- VALUE: Number, date or text to format
- PATTERN$: String with formatting pattern
**SIMPLE EXAMPLE:**
```basic
10 PRINT FORMAT(123.45, "n") ' 123.45
20 PRINT FORMAT("Mary", "Ms. @") ' Ms. Mary
```
**PRACTICAL EXAMPLE:**
```basic
10 INPUT "Name: "; NAME$
20 INPUT "Salary: "; SALARY
30 INPUT "Birth date: "; BIRTH_DATE$
40
50 PRINT "Record:"
60 PRINT "Name: "; FORMAT(NAME$, "!") ' UPPERCASE
70 PRINT "Salary: "; FORMAT(SALARY, "C2[en]") ' $1,234.56
80 PRINT "Birth: "; FORMAT(BIRTH_DATE$, "MM/dd/yyyy")
```
**COMMON ERRORS:**
- Using wrong pattern for data type
- Forgetting it returns string
- Formatting date without correct format
**BEGINNER TIP:**
Test each pattern separately before using in project
**SUGGESTED EXERCISE:**
Create a bank statement with professional formatting
```
---
## 🎨 **EXAMPLE 4: DATE AND TIME FORMATTING**
```
**BASIC CONCEPT:**
DATE AND TIME FORMATTING
**LEVEL:**
☐ Beginner ☒ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Learn all date formatting patterns
**CODE EXAMPLE:**
```basic
10 DATE$ = "2024-03-15 14:30:25"
20
30 PRINT "Brazilian: "; FORMAT(DATE$, "dd/MM/yyyy") ' 15/03/2024
40 PRINT "Complete: "; FORMAT(DATE$, "dd/MM/yyyy HH:mm") ' 15/03/2024 14:30
50 PRINT "US: "; FORMAT(DATE$, "MM/dd/yyyy") ' 03/15/2024
60 PRINT "International: "; FORMAT(DATE$, "yyyy-MM-dd") ' 2024-03-15
70
80 PRINT "24h Time: "; FORMAT(DATE$, "HH:mm:ss") ' 14:30:25
90 PRINT "12h Time: "; FORMAT(DATE$, "hh:mm:ss tt") ' 02:30:25 PM
100 PRINT "Long date: "; FORMAT(DATE$, "dd 'of' MMMM 'of' yyyy")
```
**SPECIFIC QUESTIONS:**
- What's the difference between HH and hh?
- How to show month name?
- What is "tt"?
**PROJECT CONTEXT:**
Scheduling system and reports
**EXPECTED RESULT:**
Dates formatted according to needs
**PARTS I DON'T UNDERSTAND:**
- All formatting codes
- How milliseconds work
```
---
## 🏆 **EXAMPLE 5: COMPLETE PROJECT - BANK STATEMENT**
```
# BASIC PROJECT: FORMATTED BANK STATEMENT
## 📝 DESCRIPTION
System that generates bank statement with professional formatting
## 🎨 FEATURES
- [x] Currency formatting
- [x] Date formatting
- [x] Value alignment
## 🧩 CODE STRUCTURE
```basic
10 ' Customer data
20 NAME$ = "Carlos Silva"
30 BALANCE = 12567.89
40
50 ' Transactions
60 DIM DATES$(3), DESCRIPTIONS$(3), AMOUNTS(3)
70 DATES$(1) = "2024-03-10 09:15:00" : DESCRIPTIONS$(1) = "Deposit" : AMOUNTS(1) = 2000
80 DATES$(2) = "2024-03-12 14:20:00" : DESCRIPTIONS$(2) = "Withdrawal" : AMOUNTS(2) = -500
90 DATES$(3) = "2024-03-14 11:30:00" : DESCRIPTIONS$(3) = "Transfer" : AMOUNTS(3) = -150.50
100
110 ' Header
120 PRINT FORMAT("BANK STATEMENT", "!")
130 PRINT "Customer: "; FORMAT(NAME$, "&")
140 PRINT "Date: "; FORMAT("2024-03-15 08:00:00", "dd/MM/yyyy HH:mm")
150 PRINT STRING$(40, "-")
160
170 ' Transactions
180 FOR I = 1 TO 3
190 FORMATTED_DATE$ = FORMAT(DATES$(I), "dd/MM HH:mm")
200 FORMATTED_AMOUNT$ = FORMAT(AMOUNTS(I), "C2[en]")
210
220 PRINT FORMATTED_DATE$; " - ";
230 PRINT DESCRIPTIONS$(I);
240 PRINT TAB(30); FORMATTED_AMOUNT$
250 NEXT I
260
270 ' Balance
280 PRINT STRING$(40, "-")
290 PRINT "Balance: "; TAB(30); FORMAT(BALANCE, "C2[en]")
```
## 🎯 LEARNINGS
- Currency formatting with locale
- Date formatting
- Composition of multiple formats
## ❓ QUESTIONS TO EVOLVE
- How to perfectly align columns?
- How to format negative numbers in red?
- How to add more locales?
```
---
## 🛠️ **EXAMPLE 6: TEXT FORMATTING**
```
**BASIC CONCEPT:**
STRING/TEXT FORMATTING
**LEVEL:**
☒ Beginner ☐ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Learn to use placeholders in text
**CODE EXAMPLE:**
```basic
10 NAME$ = "Mary"
20 CITY$ = "são paulo"
21 COUNTRY$ = "BRAZIL"
22 AGE = 25
30
40 PRINT FORMAT(NAME$, "Hello @!") ' Hello Mary!
50 PRINT FORMAT(NAME$, "Welcome, @") ' Welcome, Mary
60 PRINT FORMAT(CITY$, "City: !") ' City: SÃO PAULO
70 PRINT FORMAT(CITY$, "City: &") ' City: são paulo
80 PRINT FORMAT(COUNTRY$, "Country: &") ' Country: brazil
90
100 ' Combining with numbers
110 PRINT FORMAT(NAME$, "@ is ") + FORMAT(AGE, "n") + " years old"
120 ' Mary is 25 years old
```
**SPECIFIC QUESTIONS:**
- What's the difference between @, ! and &?
- Can I use multiple placeholders?
- How to escape special characters?
**PROJECT CONTEXT:**
Personalized report generation
**EXPECTED RESULT:**
Dynamic texts formatted automatically
**PARTS I DON'T UNDERSTAND:**
- Placeholder limitations
- How to mix different types
```
---
## 📚 **EXAMPLE 7: PRACTICAL EXERCISES**
```
# EXERCISES: PRACTICING WITH FORMAT
## 🎯 EXERCISE 1 - BASIC
Create a program that formats product prices.
**SOLUTION:**
```basic
10 DIM PRODUCTS$(3), PRICES(3)
20 PRODUCTS$(1) = "Laptop" : PRICES(1) = 2500.99
30 PRODUCTS$(2) = "Mouse" : PRICES(2) = 45.5
40 PRODUCTS$(3) = "Keyboard" : PRICES(3) = 120.75
50
60 FOR I = 1 TO 3
70 PRINT FORMAT(PRODUCTS$(I), "@: ") + FORMAT(PRICES(I), "C2[en]")
80 NEXT I
```
## 🎯 EXERCISE 2 - INTERMEDIATE
Make a program that shows dates in different formats.
**SOLUTION:**
```basic
10 DATE$ = "2024-12-25 20:00:00"
20
30 PRINT "Christmas: "; FORMAT(DATE$, "dd/MM/yyyy")
40 PRINT "US: "; FORMAT(DATE$, "MM/dd/yyyy")
50 PRINT "Dinner: "; FORMAT(DATE$, "HH'h'mm")
60 PRINT "Formatted: "; FORMAT(DATE$, "dd 'of' MMMM 'of' yyyy 'at' HH:mm")
```
## 🎯 EXERCISE 3 - ADVANCED
Create a school report card system with formatting.
**SOLUTION:**
```basic
10 NAME$ = "ana silva"
20 AVERAGE = 8.75
21 ATTENDANCE = 0.92
30 REPORT_DATE$ = "2024-03-15 10:00:00"
40
50 PRINT FORMAT("SCHOOL REPORT CARD", "!")
60 PRINT "Student: "; FORMAT(NAME$, "&")
70 PRINT "Date: "; FORMAT(REPORT_DATE$, "dd/MM/yyyy")
80 PRINT "Average: "; FORMAT(AVERAGE, "n")
90 PRINT "Attendance: "; FORMAT(ATTENDANCE, "0%")
```
## 💡 TIPS
- Always test patterns before using
- Use PRINT to see each formatting result
- Combine simple formats to create complex ones
```
---
## 🎨 **EXAMPLE 8: COMPLETE REFERENCE GUIDE**
```markdown
# FORMAT FUNCTION - COMPLETE GUIDE
## 🎯 OBJECTIVE
Format numbers, dates and text professionally
## 📋 SYNTAX
```basic
RESULT$ = FORMAT(VALUE, PATTERN$)
```
## 🔢 NUMERIC FORMATTING
| Pattern | Example | Result |
|---------|---------|--------|
| "n" | `FORMAT(1234.5, "n")` | 1234.50 |
| "F" | `FORMAT(1234.5, "F")` | 1234.50 |
| "f" | `FORMAT(1234.5, "f")` | 1234 |
| "0%" | `FORMAT(0.85, "0%")` | 85% |
| "C2[en]" | `FORMAT(1234.5, "C2[en]")` | $1,234.50 |
| "C2[pt]" | `FORMAT(1234.5, "C2[pt]")` | R$ 1.234,50 |
## 📅 DATE FORMATTING
| Code | Meaning | Example |
|------|---------|---------|
| yyyy | 4-digit year | 2024 |
| yy | 2-digit year | 24 |
| MM | 2-digit month | 03 |
| M | 1-2 digit month | 3 |
| dd | 2-digit day | 05 |
| d | 1-2 digit day | 5 |
| HH | 24h hour 2-digit | 14 |
| H | 24h hour 1-2 digit | 14 |
| hh | 12h hour 2-digit | 02 |
| h | 12h hour 1-2 digit | 2 |
| mm | 2-digit minute | 05 |
| m | 1-2 digit minute | 5 |
| ss | 2-digit second | 09 |
| s | 1-2 digit second | 9 |
| tt | AM/PM | PM |
| t | A/P | P |
## 📝 TEXT FORMATTING
| Placeholder | Function | Example |
|-------------|----------|---------|
| @ | Insert original text | `FORMAT("John", "@")` → John |
| ! | Text in UPPERCASE | `FORMAT("John", "!")` → JOHN |
| & | Text in lowercase | `FORMAT("John", "&")` → john |
## ⚠️ LIMITATIONS
- Dates must be in "YYYY-MM-DD HH:MM:SS" format
- Very large numbers may have issues
- Supported locales: en, pt, fr, de, es, it
```
These examples cover from basic to advanced applications of the FORMAT function! 🚀

348
docs/basic/keywords/last.md
View file

@ -1,348 +0,0 @@
# 📚 **BASIC LEARNING EXAMPLES - LAST Function**
## 🎯 **EXAMPLE 1: BASIC CONCEPT OF LAST FUNCTION**
```
**BASIC CONCEPT:**
LAST FUNCTION - Extract last word
**LEVEL:**
☒ Beginner ☐ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Understand how the LAST function extracts the last word from text
**CODE EXAMPLE:**
```basic
10 PALAVRA$ = "The mouse chewed the clothes"
20 ULTIMA$ = LAST(PALAVRA$)
30 PRINT "Last word: "; ULTIMA$
```
**SPECIFIC QUESTIONS:**
- How does the function know where the last word ends?
- What happens if there are extra spaces?
- Can I use it with numeric variables?
**PROJECT CONTEXT:**
I'm creating a program that analyzes sentences
**EXPECTED RESULT:**
Should display: "Last word: clothes"
**PARTS I DON'T UNDERSTAND:**
- Why are parentheses needed?
- How does the function work internally?
```
---
## 🛠️ **EXAMPLE 2: SOLVING ERROR WITH LAST**
```
**BASIC ERROR:**
"Syntax error" when using LAST
**MY CODE:**
```basic
10 TEXTO$ = "Good day world"
20 RESULTADO$ = LAST TEXTO$
30 PRINT RESULTADO$
```
**PROBLEM LINE:**
Line 20
**EXPECTED BEHAVIOR:**
Show "world" on screen
**CURRENT BEHAVIOR:**
Syntax error
**WHAT I'VE TRIED:**
- Tried without parentheses
- Tried with different quotes
- Tried changing variable name
**BASIC VERSION:**
QBASIC with Rhai extension
**CORRECTED SOLUTION:**
```basic
10 TEXTO$ = "Good day world"
20 RESULTADO$ = LAST(TEXTO$)
30 PRINT RESULTADO$
```
```
---
## 📖 **EXAMPLE 3: EXPLAINING LAST COMMAND**
```
**COMMAND:**
LAST - Extracts last word
**SYNTAX:**
```basic
ULTIMA$ = LAST(TEXTO$)
```
**PARAMETERS:**
- TEXTO$: String from which to extract the last word
**SIMPLE EXAMPLE:**
```basic
10 FRASE$ = "The sun is bright"
20 ULTIMA$ = LAST(FRASE$)
30 PRINT ULTIMA$ ' Shows: bright
```
**PRACTICAL EXAMPLE:**
```basic
10 INPUT "Enter your full name: "; NOME$
20 SOBRENOME$ = LAST(NOME$)
30 PRINT "Hello Mr./Mrs. "; SOBRENOME$
```
**COMMON ERRORS:**
- Forgetting parentheses: `LAST TEXTO$` ❌
- Using with numbers: `LAST(123)` ❌
- Forgetting to assign to a variable
**BEGINNER TIP:**
Always use parentheses and ensure content is text
**SUGGESTED EXERCISE:**
Create a program that asks for a sentence and shows the first and last word
```
---
## 🎨 **EXAMPLE 4: COMPLETE PROJECT WITH LAST**
```
# BASIC PROJECT: SENTENCE ANALYZER
## 📝 DESCRIPTION
Program that analyzes sentences and extracts useful information
## 🎨 FEATURES
- [x] Extract last word
- [x] Count words
- [x] Show statistics
## 🧩 CODE STRUCTURE
```basic
10 PRINT "=== SENTENCE ANALYZER ==="
20 INPUT "Enter a sentence: "; FRASE$
30
40 ' Extract last word
50 ULTIMA$ = LAST(FRASE$)
60
70 ' Count words (simplified)
80 PALAVRAS = 1
90 FOR I = 1 TO LEN(FRASE$)
100 IF MID$(FRASE$, I, 1) = " " THEN PALAVRAS = PALAVRAS + 1
110 NEXT I
120
130 PRINT
140 PRINT "Last word: "; ULTIMA$
150 PRINT "Total words: "; PALAVRAS
160 PRINT "Original sentence: "; FRASE$
```
## 🎯 LEARNINGS
- How to use LAST function
- How to count words manually
- String manipulation in BASIC
## ❓ QUESTIONS TO EVOLVE
- How to extract the first word?
- How to handle punctuation?
- How to work with multiple sentences?
```
---
## 🏆 **EXAMPLE 5: SPECIAL CASES AND TESTS**
```
**BASIC CONCEPT:**
SPECIAL CASES OF LAST FUNCTION
**LEVEL:**
☐ Beginner ☒ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Understand how LAST behaves in special situations
**CODE EXAMPLES:**
```basic
' Case 1: Empty string
10 TEXTO$ = ""
20 PRINT LAST(TEXTO$) ' Result: ""
' Case 2: Single word only
30 TEXTO$ = "Sun"
40 PRINT LAST(TEXTO$) ' Result: "Sun"
' Case 3: Multiple spaces
50 TEXTO$ = "Hello World "
60 PRINT LAST(TEXTO$) ' Result: "World"
' Case 4: With tabs and newlines
70 TEXTO$ = "Line1" + CHR$(9) + "Line2" + CHR$(13)
80 PRINT LAST(TEXTO$) ' Result: "Line2"
```
**SPECIFIC QUESTIONS:**
- What happens with empty strings?
- How does it work with special characters?
- Is it case-sensitive?
**PROJECT CONTEXT:**
I need to robustly validate user inputs
**EXPECTED RESULT:**
Consistent behavior in all cases
**PARTS I DON'T UNDERSTAND:**
- How the function handles whitespace?
- What are CHR$(9) and CHR$(13)?
```
---
## 🛠️ **EXAMPLE 6: INTEGRATION WITH OTHER FUNCTIONS**
```
**BASIC CONCEPT:**
COMBINING LAST WITH OTHER FUNCTIONS
**LEVEL:**
☐ Beginner ☒ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Learn to use LAST in more complex expressions
**CODE EXAMPLE:**
```basic
10 ' Example 1: With concatenation
20 PARTE1$ = "Programming"
30 PARTE2$ = " in BASIC"
40 FRASE_COMPLETA$ = PARTE1$ + PARTE2$
50 PRINT LAST(FRASE_COMPLETA$) ' Result: "BASIC"
60 ' Example 2: With string functions
70 NOME_COMPLETO$ = "Maria Silva Santos"
80 SOBRENOME$ = LAST(NOME_COMPLETO$)
90 PRINT "Mr./Mrs. "; SOBRENOME$
100 ' Example 3: In conditional expressions
110 FRASE$ = "The sky is blue"
120 IF LAST(FRASE$) = "blue" THEN PRINT "The last word is blue!"
```
**SPECIFIC QUESTIONS:**
- Can I use LAST directly in IF?
- How to combine with LEFT$, RIGHT$, MID$?
- Is there a size limit for the string?
**PROJECT CONTEXT:**
Creating validations and text processing
**EXPECTED RESULT:**
Use LAST flexibly in different contexts
**PARTS I DON'T UNDERSTAND:**
- Expression evaluation order
- Performance with very large strings
```
---
## 📚 **EXAMPLE 7: PRACTICAL EXERCISES**
```
# EXERCISES: PRACTICING WITH LAST
## 🎯 EXERCISE 1 - BASIC
Create a program that asks for the user's full name and greets using only the last name.
**SOLUTION:**
```basic
10 INPUT "Enter your full name: "; NOME$
20 SOBRENOME$ = LAST(NOME$)
30 PRINT "Hello, Mr./Mrs. "; SOBRENOME$; "!"
```
## 🎯 EXERCISE 2 - INTERMEDIATE
Make a program that analyzes if the last word of a sentence is "end".
**SOLUTION:**
```basic
10 INPUT "Enter a sentence: "; FRASE$
20 IF LAST(FRASE$) = "end" THEN PRINT "Sentence ends with 'end'" ELSE PRINT "Sentence doesn't end with 'end'"
```
## 🎯 EXERCISE 3 - ADVANCED
Create a program that processes multiple sentences and shows statistics.
**SOLUTION:**
```basic
10 DIM FRASES$(3)
20 FRASES$(1) = "The sun shines"
30 FRASES$(2) = "The rain falls"
40 FRASES$(3) = "The wind blows"
50
60 FOR I = 1 TO 3
70 PRINT "Sentence "; I; ": "; FRASES$(I)
80 PRINT "Last word: "; LAST(FRASES$(I))
90 PRINT
100 NEXT I
```
## 💡 TIPS
- Always test with different inputs
- Use PRINT for debugging
- Start with simple examples
```
---
## 🎨 **EXAMPLE 8: MARKDOWN DOCUMENTATION**
```markdown
# LAST FUNCTION - COMPLETE GUIDE
## 🎯 OBJECTIVE
Extract the last word from a string
## 📋 SYNTAX
```basic
RESULTADO$ = LAST(TEXTO$)
```
## 🧩 PARAMETERS
- `TEXTO$`: Input string
## 🔍 BEHAVIOR
- Splits string by spaces
- Returns the last part
- Ignores extra spaces at beginning/end
## 🚀 EXAMPLES
```basic
10 PRINT LAST("hello world") ' Output: world
20 PRINT LAST("one word") ' Output: word
30 PRINT LAST(" spaces ") ' Output: spaces
```
## ⚠️ LIMITATIONS
- Doesn't work with numbers
- Requires parentheses
- Considers only spaces as separators
```
These examples cover from the basic concept to practical applications of the LAST function, always focusing on BASIC beginners! 🚀

78
docs/platform/DEV.md
View file

@ -1,78 +0,0 @@
# LLM
ZED for Windows: https://zed.dev/windows
Zed Assistant: Groq + GPT OSS 120B |
FIX Manual: DeepSeek | ChatGPT 120B | Claude 4.5 Thinking | Mistral
ADD Manual: Claude/DeepSeek -> DeepSeek
# Install
cargo install cargo-audit
cargo install cargo-edit
apt install -y libpq-dev
apt install -y valkey-cli
## Cache
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/valkey.gpg
echo "deb [signed-by=/usr/share/keyrings/valkey.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/valkey.list
sudo apt install valkey-server
## Meet
curl -sSL https://get.livekit.io | bash
livekit-server --dev
# Util
cargo upgrade
cargo audit
valkey-cli -p 6379 monitor
# Prompt add-ons
- Prompt add-ons: Fill the file with info!, trace! and debug! macros.
-
# Zed Agents
```
"language_models": {
"openai_compatible": {
"Groq GPT 120b": {
"api_url": "https://api.groq.com/openai/v1",
"available_models": [
{
"name": "meta-llama/llama-4-scout-17b-16e-instruct",
"max_tokens": 30000,
"capabilities": {
"tools": true,
"images": false,
"parallel_tool_calls": false,
"prompt_cache_key": false
}
},
{
"name": "groq/compound",
"max_tokens": 70000
},
{
"name": "openai/gpt-oss-120b",
"max_tokens": 8000,
"capabilities": {
"tools": true,
"images": false,
"parallel_tool_calls": false,
"prompt_cache_key": false
}
}
]
}
}
},
```

6
docs/platform/GLOSSARY.md
View file

@ -1,6 +0,0 @@
RPM: Requests per minute
RPD: Requests per day
TPM: Tokens per minute
TPD: Tokens per day
ASH: Audio seconds per hour
ASD: Audio seconds per day

90
docs/platform/guide/automation.md
View file

@ -1,90 +0,0 @@
# Automation System Documentation
## Overview
The automation system allows you to execute scripts automatically based on triggers like database changes or scheduled times.
## Database Configuration
### system_automations Table Structure
To create an automation, insert a record into the `system_automations` table:
| Column | Type | Description |
|--------|------|-------------|
| id | UUID | Unique identifier (auto-generated) |
| name | TEXT | Human-readable name |
| kind | INTEGER | Trigger type (see below) |
| target | TEXT | Target table name (for table triggers) |
| param | TEXT | Script filename or path |
| schedule | TEXT | Cron pattern (for scheduled triggers) |
| is_active | BOOLEAN | Whether automation is enabled |
| last_triggered | TIMESTAMP | Last execution time |
### Trigger Types (kind field)
- `0` - TableInsert (triggers on new rows)
- `1` - TableUpdate (triggers on row updates)
- `2` - TableDelete (triggers on row deletions)
- `3` - Scheduled (triggers on cron schedule)
## Configuration Examples
### 1. Scheduled Automation (Daily at 2:30 AM)
```sql
INSERT INTO system_automations (name, kind, target, param, schedule, is_active)
VALUES ('Daily Resume Update', 3, NULL, 'daily_resume.js', '30 2 * * *', true);
```
### 2. Table Change Automation
```sql
-- Trigger when new documents are added to documents table
INSERT INTO system_automations (name, kind, target, param, schedule, is_active)
VALUES ('Process New Documents', 0, 'documents', 'process_document.js', NULL, true);
```
## Cron Pattern Format
Use standard cron syntax: `minute hour day month weekday`
Examples:
- `0 9 * * *` - Daily at 9:00 AM
- `30 14 * * 1-5` - Weekdays at 2:30 PM
- `0 0 1 * *` - First day of every month at midnight
## Sample Script
```BASIC
let text = GET "default.gbdrive/default.pdf"
let resume = LLM "Build table resume with deadlines, dates and actions: " + text
SET BOT MEMORY "resume", resume
```
## Script Capabilities
### Available Commands
- `GET "path"` - Read files from storage
- `LLM "prompt"` - Query language model with prompts
- `SET BOT MEMORY "key", value` - Store data in bot memory
- Database operations (query, insert, update)
- HTTP requests to external APIs
## Best Practices
1. **Keep scripts focused** - Each script should do one thing well
2. **Handle errors gracefully** - Use try/catch blocks
3. **Log important actions** - Use console.log for debugging
4. **Test thoroughly** - Verify scripts work before automating
5. **Monitor execution** - Check logs for any automation errors
## Monitoring
Check application logs to monitor automation execution:
```bash
# Look for automation-related messages
grep "Automation\|Script executed" application.log
```
The system will automatically update `last_triggered` timestamps and log any errors encountered during execution.

0
docs/platform/guide/debugging.md
View file

168
docs/platform/guide/file.md
View file

@ -1,168 +0,0 @@
# File Upload Service with Actix Web and S3/MinIO
## Overview
This service provides a REST API endpoint for uploading files to S3-compatible storage (including MinIO) using Actix Web. It handles multipart form data, temporarily stores files locally, and transfers them to object storage.
## BASIC Keywords Reference
- **UPLOAD**: Handles file uploads via multipart form data
- **CONFIG**: Manages S3/MinIO configuration and client initialization
- **TEMP**: Uses temporary files for processing uploads
- **CLIENT**: Maintains S3 client connection
- **ERROR**: Comprehensive error handling for upload failures
- **BUCKET**: Configures and uses S3 buckets for storage
- **PATH**: Manages folder paths for object organization
## API Reference
### POST `/files/upload/{folder_path}`
Uploads a file to the specified folder in S3/MinIO storage.
**Path Parameters:**
- `folder_path` (string): Target folder path in S3 bucket
**Request:**
- Content-Type: `multipart/form-data`
- Body: File data in multipart format
**Response:**
- `200 OK`: Upload successful
- `500 Internal Server Error`: Upload failed
**Example:**
```bash
curl -X POST \
http://localhost:8080/files/upload/documents \
-F "file=@report.pdf"
```
## Configuration
### DriveConfig Structure
```rust
// Example configuration
let config = DriveConfig {
access_key: "your-access-key".to_string(),
secret_key: "your-secret-key".to_string(),
server: "minio.example.com:9000".to_string(),
s3_bucket: "my-bucket".to_string(),
use_ssl: false,
};
```
### Client Initialization
```rust
use crate::config::DriveConfig;
// Initialize S3 client
let drive_config = DriveConfig {
access_key: "minioadmin".to_string(),
secret_key: "minioadmin".to_string(),
server: "localhost:9000".to_string(),
s3_bucket: "uploads".to_string(),
use_ssl: false,
};
let s3_client = init_drive(&drive_config).await?;
```
## Implementation Guide
### 1. Setting Up AppState
```rust
use crate::shared::state::AppState;
// Configure application state with S3 client
let app_state = web::Data::new(AppState {
s3_client: Some(s3_client),
config: Some(drive_config),
// ... other state fields
});
```
### 2. Error Handling Patterns
The service implements several error handling strategies:
```rust
// Configuration errors
let bucket_name = state.get_ref().config.as_ref()
.ok_or_else(|| actix_web::error::ErrorInternalServerError(
"S3 bucket configuration is missing"
))?;
// Client initialization errors
let s3_client = state.get_ref().s3_client.as_ref()
.ok_or_else(|| actix_web::error::ErrorInternalServerError(
"S3 client is not initialized"
))?;
// File operation errors with cleanup
let mut temp_file = NamedTempFile::new().map_err(|e| {
actix_web::error::ErrorInternalServerError(format!(
"Failed to create temp file: {}", e
))
})?;
```
### 3. File Processing Flow
```rust
// 1. Create temporary file
let mut temp_file = NamedTempFile::new()?;
// 2. Process multipart data
while let Some(mut field) = payload.try_next().await? {
// Extract filename from content disposition
if let Some(disposition) = field.content_disposition() {
file_name = disposition.get_filename().map(|s| s.to_string());
}
// Stream data to temporary file
while let Some(chunk) = field.try_next().await? {
temp_file.write_all(&chunk)?;
}
}
// 3. Upload to S3
upload_to_s3(&s3_client, &bucket_name, &s3_key, &temp_file_path).await?;
// 4. Cleanup temporary file
let _ = std::fs::remove_file(&temp_file_path);
```
## Key Features
### Temporary File Management
- Uses `NamedTempFile` for secure temporary storage
- Automatic cleanup on both success and failure
- Efficient streaming of multipart data
### S3/MinIO Compatibility
- Path-style addressing for MinIO compatibility
- Configurable SSL/TLS
- Custom endpoint support
### Security Considerations
- Temporary files are automatically deleted
- No persistent storage of uploaded files on server
- Secure credential handling
## Error Scenarios
1. **Missing Configuration**: Returns 500 if S3 bucket or client not configured
2. **File System Errors**: Handles temp file creation/write failures
3. **Network Issues**: Manages S3 connection timeouts and errors
4. **Invalid Uploads**: Handles malformed multipart data
## Performance Notes
- Streams data directly from multipart to temporary file
- Uses async operations for I/O-bound tasks
- Minimal memory usage for large file uploads
- Efficient cleanup prevents disk space leaks

348
docs/platform/guide/last.md
View file

@ -1,348 +0,0 @@
# 📚 **BASIC LEARNING EXAMPLES - LAST Function**
## 🎯 **EXAMPLE 1: BASIC CONCEPT OF LAST FUNCTION**
```
**BASIC CONCEPT:**
LAST FUNCTION - Extract last word
**LEVEL:**
☒ Beginner ☐ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Understand how the LAST function extracts the last word from text
**CODE EXAMPLE:**
```basic
10 PALAVRA$ = "The mouse chewed the clothes"
20 ULTIMA$ = LAST(PALAVRA$)
30 PRINT "Last word: "; ULTIMA$
```
**SPECIFIC QUESTIONS:**
- How does the function know where the last word ends?
- What happens if there are extra spaces?
- Can I use it with numeric variables?
**PROJECT CONTEXT:**
I'm creating a program that analyzes sentences
**EXPECTED RESULT:**
Should display: "Last word: clothes"
**PARTS I DON'T UNDERSTAND:**
- Why are parentheses needed?
- How does the function work internally?
```
---
## 🛠️ **EXAMPLE 2: SOLVING ERROR WITH LAST**
```
**BASIC ERROR:**
"Syntax error" when using LAST
**MY CODE:**
```basic
10 TEXTO$ = "Good day world"
20 RESULTADO$ = LAST TEXTO$
30 PRINT RESULTADO$
```
**PROBLEM LINE:**
Line 20
**EXPECTED BEHAVIOR:**
Show "world" on screen
**CURRENT BEHAVIOR:**
Syntax error
**WHAT I'VE TRIED:**
- Tried without parentheses
- Tried with different quotes
- Tried changing variable name
**BASIC VERSION:**
QBASIC with Rhai extension
**CORRECTED SOLUTION:**
```basic
10 TEXTO$ = "Good day world"
20 RESULTADO$ = LAST(TEXTO$)
30 PRINT RESULTADO$
```
```
---
## 📖 **EXAMPLE 3: EXPLAINING LAST COMMAND**
```
**COMMAND:**
LAST - Extracts last word
**SYNTAX:**
```basic
ULTIMA$ = LAST(TEXTO$)
```
**PARAMETERS:**
- TEXTO$: String from which to extract the last word
**SIMPLE EXAMPLE:**
```basic
10 FRASE$ = "The sun is bright"
20 ULTIMA$ = LAST(FRASE$)
30 PRINT ULTIMA$ ' Shows: bright
```
**PRACTICAL EXAMPLE:**
```basic
10 INPUT "Enter your full name: "; NOME$
20 SOBRENOME$ = LAST(NOME$)
30 PRINT "Hello Mr./Mrs. "; SOBRENOME$
```
**COMMON ERRORS:**
- Forgetting parentheses: `LAST TEXTO$` ❌
- Using with numbers: `LAST(123)` ❌
- Forgetting to assign to a variable
**BEGINNER TIP:**
Always use parentheses and ensure content is text
**SUGGESTED EXERCISE:**
Create a program that asks for a sentence and shows the first and last word
```
---
## 🎨 **EXAMPLE 4: COMPLETE PROJECT WITH LAST**
```
# BASIC PROJECT: SENTENCE ANALYZER
## 📝 DESCRIPTION
Program that analyzes sentences and extracts useful information
## 🎨 FEATURES
- [x] Extract last word
- [x] Count words
- [x] Show statistics
## 🧩 CODE STRUCTURE
```basic
10 PRINT "=== SENTENCE ANALYZER ==="
20 INPUT "Enter a sentence: "; FRASE$
30
40 ' Extract last word
50 ULTIMA$ = LAST(FRASE$)
60
70 ' Count words (simplified)
80 PALAVRAS = 1
90 FOR I = 1 TO LEN(FRASE$)
100 IF MID$(FRASE$, I, 1) = " " THEN PALAVRAS = PALAVRAS + 1
110 NEXT I
120
130 PRINT
140 PRINT "Last word: "; ULTIMA$
150 PRINT "Total words: "; PALAVRAS
160 PRINT "Original sentence: "; FRASE$
```
## 🎯 LEARNINGS
- How to use LAST function
- How to count words manually
- String manipulation in BASIC
## ❓ QUESTIONS TO EVOLVE
- How to extract the first word?
- How to handle punctuation?
- How to work with multiple sentences?
```
---
## 🏆 **EXAMPLE 5: SPECIAL CASES AND TESTS**
```
**BASIC CONCEPT:**
SPECIAL CASES OF LAST FUNCTION
**LEVEL:**
☐ Beginner ☒ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Understand how LAST behaves in special situations
**CODE EXAMPLES:**
```basic
' Case 1: Empty string
10 TEXTO$ = ""
20 PRINT LAST(TEXTO$) ' Result: ""
' Case 2: Single word only
30 TEXTO$ = "Sun"
40 PRINT LAST(TEXTO$) ' Result: "Sun"
' Case 3: Multiple spaces
50 TEXTO$ = "Hello World "
60 PRINT LAST(TEXTO$) ' Result: "World"
' Case 4: With tabs and newlines
70 TEXTO$ = "Line1" + CHR$(9) + "Line2" + CHR$(13)
80 PRINT LAST(TEXTO$) ' Result: "Line2"
```
**SPECIFIC QUESTIONS:**
- What happens with empty strings?
- How does it work with special characters?
- Is it case-sensitive?
**PROJECT CONTEXT:**
I need to robustly validate user inputs
**EXPECTED RESULT:**
Consistent behavior in all cases
**PARTS I DON'T UNDERSTAND:**
- How the function handles whitespace?
- What are CHR$(9) and CHR$(13)?
```
---
## 🛠️ **EXAMPLE 6: INTEGRATION WITH OTHER FUNCTIONS**
```
**BASIC CONCEPT:**
COMBINING LAST WITH OTHER FUNCTIONS
**LEVEL:**
☐ Beginner ☒ Intermediate ☐ Advanced
**LEARNING OBJECTIVE:**
Learn to use LAST in more complex expressions
**CODE EXAMPLE:**
```basic
10 ' Example 1: With concatenation
20 PARTE1$ = "Programming"
30 PARTE2$ = " in BASIC"
40 FRASE_COMPLETA$ = PARTE1$ + PARTE2$
50 PRINT LAST(FRASE_COMPLETA$) ' Result: "BASIC"
60 ' Example 2: With string functions
70 NOME_COMPLETO$ = "Maria Silva Santos"
80 SOBRENOME$ = LAST(NOME_COMPLETO$)
90 PRINT "Mr./Mrs. "; SOBRENOME$
100 ' Example 3: In conditional expressions
110 FRASE$ = "The sky is blue"
120 IF LAST(FRASE$) = "blue" THEN PRINT "The last word is blue!"
```
**SPECIFIC QUESTIONS:**
- Can I use LAST directly in IF?
- How to combine with LEFT$, RIGHT$, MID$?
- Is there a size limit for the string?
**PROJECT CONTEXT:**
Creating validations and text processing
**EXPECTED RESULT:**
Use LAST flexibly in different contexts
**PARTS I DON'T UNDERSTAND:**
- Expression evaluation order
- Performance with very large strings
```
---
## 📚 **EXAMPLE 7: PRACTICAL EXERCISES**
```
# EXERCISES: PRACTICING WITH LAST
## 🎯 EXERCISE 1 - BASIC
Create a program that asks for the user's full name and greets using only the last name.
**SOLUTION:**
```basic
10 INPUT "Enter your full name: "; NOME$
20 SOBRENOME$ = LAST(NOME$)
30 PRINT "Hello, Mr./Mrs. "; SOBRENOME$; "!"
```
## 🎯 EXERCISE 2 - INTERMEDIATE
Make a program that analyzes if the last word of a sentence is "end".
**SOLUTION:**
```basic
10 INPUT "Enter a sentence: "; FRASE$
20 IF LAST(FRASE$) = "end" THEN PRINT "Sentence ends with 'end'" ELSE PRINT "Sentence doesn't end with 'end'"
```
## 🎯 EXERCISE 3 - ADVANCED
Create a program that processes multiple sentences and shows statistics.
**SOLUTION:**
```basic
10 DIM FRASES$(3)
20 FRASES$(1) = "The sun shines"
30 FRASES$(2) = "The rain falls"
40 FRASES$(3) = "The wind blows"
50
60 FOR I = 1 TO 3
70 PRINT "Sentence "; I; ": "; FRASES$(I)
80 PRINT "Last word: "; LAST(FRASES$(I))
90 PRINT
100 NEXT I
```
## 💡 TIPS
- Always test with different inputs
- Use PRINT for debugging
- Start with simple examples
```
---
## 🎨 **EXAMPLE 8: MARKDOWN DOCUMENTATION**
```markdown
# LAST FUNCTION - COMPLETE GUIDE
## 🎯 OBJECTIVE
Extract the last word from a string
## 📋 SYNTAX
```basic
RESULTADO$ = LAST(TEXTO$)
```
## 🧩 PARAMETERS
- `TEXTO$`: Input string
## 🔍 BEHAVIOR
- Splits string by spaces
- Returns the last part
- Ignores extra spaces at beginning/end
## 🚀 EXAMPLES
```basic
10 PRINT LAST("hello world") ' Output: world
20 PRINT LAST("one word") ' Output: word
30 PRINT LAST(" spaces ") ' Output: spaces
```
## ⚠️ LIMITATIONS
- Doesn't work with numbers
- Requires parentheses
- Considers only spaces as separators
```
These examples cover from the basic concept to practical applications of the LAST function, always focusing on BASIC beginners! 🚀

0
docs/platform/guide/quickstart.md
View file

27
docs/platform/limits_llm.md
View file

@ -1,27 +0,0 @@
## 🚀 **OPTIMAL RANGE:**
- **10-30 KB** - **SWEET SPOT** for quality Rust analysis
- **Fast responses** + **accurate error fixing**
## ⚡ **PRACTICAL MAXIMUM:**
- **50-70 KB** - **ABSOLUTE WORKING LIMIT**
- Beyond this, quality may degrade
## 🛑 **HARD CUTOFF:**
- **~128 KB** - Technical token limit
- But **quality drops significantly** before this
## 🎯 **MY RECOMMENDATION:**
**Send 20-40 KB chunks** for:
- ✅ **Best error analysis**
- ✅ **Fastest responses**
- ✅ **Most accurate Rust fixes**
- ✅ **Complete code returns**
## 💡 **PRO STRATEGY:**
1. **Extract problematic module** (15-25 KB)
2. **Include error messages**
3. **I'll fix it and return FULL code**
4. **Iterate if needed**
**You don't need 100KB** - 30KB will get you **BETTER RESULTS** with most Rust compiler errors! 🦀

140
docs/src/SUMMARY.md Normal file
View file

@ -0,0 +1,140 @@
# Summary
[Introduction](./introduction.md)
# Part I - Getting Started
- [Chapter 01: Run and Talk](./chapter-01/README.md)
- [Installation](./chapter-01/installation.md)
- [First Conversation](./chapter-01/first-conversation.md)
- [Understanding Sessions](./chapter-01/sessions.md)
# Part II - Package System
- [Chapter 02: About Packages](./chapter-02/README.md)
- [.gbai Architecture](./chapter-02/gbai.md)
- [.gbdialog Dialogs](./chapter-02/gbdialog.md)
- [.gbkb Knowledge Base](./chapter-02/gbkb.md)
- [.gbot Bot Configuration](./chapter-02/gbot.md)
- [.gbtheme UI Theming](./chapter-02/gbtheme.md)
- [.gbdrive File Storage](./chapter-02/gbdrive.md)
# Part III - Knowledge Base
- [Chapter 03: gbkb Reference](./chapter-03/README.md)
- [Vector Collections](./chapter-03/vector-collections.md)
- [Document Indexing](./chapter-03/indexing.md)
- [Qdrant Integration](./chapter-03/qdrant.md)
- [Semantic Search](./chapter-03/semantic-search.md)
- [Context Compaction](./chapter-03/context-compaction.md)
- [Semantic Caching](./chapter-03/caching.md)
# Part IV - Themes and UI
- [Chapter 04: gbtheme Reference](./chapter-04/README.md)
- [Theme Structure](./chapter-04/structure.md)
- [Web Interface](./chapter-04/web-interface.md)
- [CSS Customization](./chapter-04/css.md)
- [HTML Templates](./chapter-04/html.md)
# Part V - BASIC Dialogs
- [Chapter 05: gbdialog Reference](./chapter-05/README.md)
- [Dialog Basics](./chapter-05/basics.md)
- [Template Examples](./chapter-05/templates.md)
- [start.bas](./chapter-05/template-start.md)
- [auth.bas](./chapter-05/template-auth.md)
- [generate-summary.bas](./chapter-05/template-summary.md)
- [enrollment Tool Example](./chapter-05/template-enrollment.md)
- [Keyword Reference](./chapter-05/keywords.md)
- [TALK](./chapter-05/keyword-talk.md)
- [HEAR](./chapter-05/keyword-hear.md)
- [SET_USER](./chapter-05/keyword-set-user.md)
- [SET_CONTEXT](./chapter-05/keyword-set-context.md)
- [LLM](./chapter-05/keyword-llm.md)
- [GET_BOT_MEMORY](./chapter-05/keyword-get-bot-memory.md)
- [SET_BOT_MEMORY](./chapter-05/keyword-set-bot-memory.md)
- [SET_KB](./chapter-05/keyword-set-kb.md)
- [ADD_KB](./chapter-05/keyword-add-kb.md)
- [ADD_WEBSITE](./chapter-05/keyword-add-website.md)
- [ADD_TOOL](./chapter-05/keyword-add-tool.md)
- [LIST_TOOLS](./chapter-05/keyword-list-tools.md)
- [REMOVE_TOOL](./chapter-05/keyword-remove-tool.md)
- [CLEAR_TOOLS](./chapter-05/keyword-clear-tools.md)
- [GET](./chapter-05/keyword-get.md)
- [FIND](./chapter-05/keyword-find.md)
- [SET](./chapter-05/keyword-set.md)
- [ON](./chapter-05/keyword-on.md)
- [SET_SCHEDULE](./chapter-05/keyword-set-schedule.md)
- [CREATE_SITE](./chapter-05/keyword-create-site.md)
- [CREATE_DRAFT](./chapter-05/keyword-create-draft.md)
- [WEBSITE OF](./chapter-05/keyword-website-of.md)
- [PRINT](./chapter-05/keyword-print.md)
- [WAIT](./chapter-05/keyword-wait.md)
- [FORMAT](./chapter-05/keyword-format.md)
- [FIRST](./chapter-05/keyword-first.md)
- [LAST](./chapter-05/keyword-last.md)
- [FOR EACH](./chapter-05/keyword-for-each.md)
- [EXIT FOR](./chapter-05/keyword-exit-for.md)
# Part VI - Extending BotServer
- [Chapter 06: gbapp Reference](./chapter-06/README.md)
- [Rust Architecture](./chapter-06/architecture.md)
- [Building from Source](./chapter-06/building.md)
- [Crate Structure](./chapter-06/crates.md)
- [Service Layer](./chapter-06/services.md)
- [Creating Custom Keywords](./chapter-06/custom-keywords.md)
- [Adding Dependencies](./chapter-06/dependencies.md)
# Part VII - Bot Configuration
- [Chapter 07: gbot Reference](./chapter-07/README.md)
- [config.csv Format](./chapter-07/config-csv.md)
- [Bot Parameters](./chapter-07/parameters.md)
- [Answer Modes](./chapter-07/answer-modes.md)
- [LLM Configuration](./chapter-07/llm-config.md)
- [Context Configuration](./chapter-07/context-config.md)
- [MinIO Drive Integration](./chapter-07/minio.md)
# Part VIII - Tools and Integration
- [Chapter 08: Tooling](./chapter-08/README.md)
- [Tool Definition](./chapter-08/tool-definition.md)
- [PARAM Declaration](./chapter-08/param-declaration.md)
- [Tool Compilation](./chapter-08/compilation.md)
- [MCP Format](./chapter-08/mcp-format.md)
- [OpenAI Tool Format](./chapter-08/openai-format.md)
- [GET Keyword Integration](./chapter-08/get-integration.md)
- [External APIs](./chapter-08/external-apis.md)
# Part IX - Feature Reference
- [Chapter 09: Feature Matrix](./chapter-09/README.md)
- [Core Features](./chapter-09/core-features.md)
- [Conversation Management](./chapter-09/conversation.md)
- [AI and LLM](./chapter-09/ai-llm.md)
- [Knowledge Base](./chapter-09/knowledge-base.md)
- [Automation](./chapter-09/automation.md)
- [Email Integration](./chapter-09/email.md)
- [Web Automation](./chapter-09/web-automation.md)
- [Storage and Data](./chapter-09/storage.md)
- [Multi-Channel Support](./chapter-09/channels.md)
# Part X - Community
- [Chapter 10: Contributing](./chapter-10/README.md)
- [Development Setup](./chapter-10/setup.md)
- [Code Standards](./chapter-10/standards.md)
- [Testing](./chapter-10/testing.md)
- [Pull Requests](./chapter-10/pull-requests.md)
- [Documentation](./chapter-10/documentation.md)
# Appendices
- [Appendix I: Database Model](./appendix-i/README.md)
- [Schema Overview](./appendix-i/schema.md)
- [Tables](./appendix-i/tables.md)
- [Relationships](./appendix-i/relationships.md)
[Glossary](./glossary.md)

49
docs/src/appendix-i/README.md Normal file
View file

@ -0,0 +1,49 @@
## Appendix I – Database Model
The core database schema for GeneralBots is defined in `src/shared/models.rs`. It uses **Diesel** with SQLite (or PostgreSQL) and includes the following primary tables:
| Table | Description |
|-------|-------------|
| `users` | Stores user accounts, authentication tokens, and profile data. |
| `sessions` | Tracks active `BotSession` instances, their start/end timestamps, and associated user. |
| `knowledge_bases` | Metadata for each `.gbkb` collection (name, vector store configuration, creation date). |
| `messages` | Individual chat messages (role = user/assistant, content, timestamp, linked to a session). |
| `tools` | Registered custom tools per session (name, definition JSON, activation status). |
| `files` | References to files managed by the `.gbdrive` package (path, size, MIME type, storage location). |
### Relationships
- **User ↔ Sessions** – One‑to‑many: a user can have many sessions.
- **Session ↔ Messages** – One‑to‑many: each session contains a sequence of messages.
- **Session ↔ KnowledgeBase** – Many‑to‑one: a session uses a single knowledge base at a time.
- **Session ↔ Tools** – One‑to‑many: tools are scoped to the session that registers them.
- **File ↔ KnowledgeBase** – Optional link for documents stored in a knowledge base.
### Key Fields (excerpt)
```rust
pub struct User {
pub id: i32,
pub username: String,
pub email: String,
pub password_hash: String,
pub created_at: NaiveDateTime,
}
pub struct Session {
pub id: i32,
pub user_id: i32,
pub started_at: NaiveDateTime,
pub last_active: NaiveDateTime,
pub knowledge_base_id: i32,
}
pub struct Message {
pub id: i32,
pub session_id: i32,
pub role: String, // "user" or "assistant"
pub content: String,
pub timestamp: NaiveDateTime,
}
```
The schema is automatically migrated by Diesel when the server starts. For custom extensions, add new tables to `models.rs` and run `diesel migration generate <name>`.

1
docs/src/appendix-i/relationships.md Normal file
View file

@ -0,0 +1 @@
# Relationships

1
docs/src/appendix-i/schema.md Normal file
View file

@ -0,0 +1 @@
# Schema Overview

1
docs/src/appendix-i/tables.md Normal file
View file

@ -0,0 +1 @@
# Tables

39
docs/src/chapter-01/README.md Normal file
View file

@ -0,0 +1,39 @@
## Run and Talk
```bas
TALK "Welcome! How can I help you today?"
HEAR user_input
```
*Start the server:* `cargo run --release`
### Installation
```bash
# Clone the repository
git clone https://github.com/GeneralBots/BotServer.git
cd BotServer
# Build the project
cargo build --release
# Run the server
cargo run --release
```
### First Conversation
```bas
TALK "Hello! I'm your GeneralBots assistant."
HEAR user_input
IF user_input CONTAINS "weather" THEN
TALK "Sure, let me check the weather for you."
CALL GET_WEATHER
ELSE
TALK "I can help with many tasks, just ask!"
ENDIF
```
### Understanding Sessions
Each conversation is represented by a **BotSession**. The session stores:
- User identifier
- Conversation history
- Current context (variables, knowledge base references, etc.)
Sessions are persisted in the SQLite database defined in `src/shared/models.rs`.

9
docs/src/chapter-01/first-conversation.md Normal file
View file

@ -0,0 +1,9 @@
# First Conversation
After the server is running, open a web browser at `http://localhost:8080` and start a chat. The default dialog (`start.bas`) greets the user and demonstrates the `TALK` keyword.
```basic
TALK "Welcome to GeneralBots! How can I assist you today?"
```
You can type a question, and the bot will respond using the LLM backend combined with any relevant knowledge‑base entries.

59
docs/src/chapter-01/installation.md Normal file
View file

@ -0,0 +1,59 @@
# Installation
## System Requirements
- **Operating System**: Linux, macOS, or Windows
- **Memory**: 8GB RAM minimum, 16GB recommended
- **Storage**: 10GB free space
- **Dependencies**: Docker (optional), PostgreSQL, Redis
## Installation Methods
### Method 1: Package Manager (Recommended)
```bash
# Install using the built‑in package manager
botserver install tables
botserver install drive
botserver install cache
botserver install llm
```
### Method 2: Manual Installation
1. Download the botserver binary
2. Set environment variables:
```bash
export DATABASE_URL="postgres://gbuser:password@localhost:5432/botserver"
export DRIVE_SERVER="http://localhost:9000"
export DRIVE_ACCESSKEY="minioadmin"
export DRIVE_SECRET="minioadmin"
```
3. Run the server: `./botserver`
## Configuration
Create a `.env` file in your working directory:
```env
BOT_GUID=your-bot-id
DATABASE_URL=postgres://gbuser:password@localhost:5432/botserver
DRIVE_SERVER=http://localhost:9000
DRIVE_ACCESSKEY=minioadmin
DRIVE_SECRET=minioadmin
```
## Verification
After installation, verify everything is working:
1. Access the web interface at `http://localhost:8080`
2. Check that all services are running:
```bash
botserver status tables
botserver status drive
botserver status cache
botserver status llm
```
The system will automatically create necessary database tables and storage buckets on first run.

3
docs/src/chapter-01/sessions.md Normal file
View file

@ -0,0 +1,3 @@
# Understanding Sessions
A **session** groups all messages exchanged between a user and the bot. Sessions are stored in the database and can be resumed later. The `SET_USER` and `SET_CONTEXT` keywords let you manipulate session data programmatically.

9
docs/src/chapter-02/README.md Normal file
View file

@ -0,0 +1,9 @@
## About Packages
| Component | Extension | Role |
|-----------|-----------|------|
| Dialog scripts | `.gbdialog` | BASIC‑style conversational logic |
| Knowledge bases | `.gbkb` | Vector‑DB collections |
| UI themes | `.gbtheme` | CSS/HTML assets |
| Bot config | `.gbbot` | CSV mapping to `UserSession` |
| Application Interface | `.gbai` | Core application architecture |
| File storage | `.gbdrive` | Object storage integration (MinIO) |

56
docs/src/chapter-02/gbai.md Normal file
View file

@ -0,0 +1,56 @@
# .gbai Architecture
The `.gbai` extension defines the overall architecture and structure of a GeneralBots application. It serves as the container for all other package types.
## What is .gbai?
`.gbai` (General Bot Application Interface) is the root package that contains:
- Bot identity and metadata
- Organizational structure
- References to other package types
- Application-level configuration
## .gbai Structure
A typical `.gbai` package contains:
```
my-bot.gbai/
├── manifest.json # Application metadata
├── .gbdialog/ # Dialog scripts
├── .gbkb/ # Knowledge bases
├── .gbot/ # Bot configuration
├── .gbtheme/ # UI themes
└── dependencies.json # External dependencies
```
## Manifest File
The `manifest.json` defines application properties:
```json
{
"name": "Customer Support Bot",
"version": "1.0.0",
"description": "AI-powered customer support assistant",
"author": "Your Name",
"bot_id": "uuid-here",
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z"
}
```
## Application Lifecycle
1. **Initialization**: Bot loads .gbai structure and dependencies
2. **Configuration**: Applies .gbot settings and parameters
3. **Activation**: Loads .gbdialog scripts and .gbkb collections
4. **Execution**: Begins processing user interactions
5. **Termination**: Cleanup and state preservation
## Multi-Bot Environments
A single GeneralBots server can host multiple .gbai applications:
- Each runs in isolation with separate configurations
- Can share common knowledge bases if configured
- Maintain separate user sessions and contexts

67
docs/src/chapter-02/gbdialog.md Normal file
View file

@ -0,0 +1,67 @@
# .gbdialog Dialogs
The `.gbdialog` package contains BASIC scripts that define conversation flows, tool integrations, and bot behavior.
## What is .gbdialog?
`.gbdialog` files are written in a specialized BASIC dialect that controls:
- Conversation flow and logic
- Tool calls and integrations
- User input processing
- Context management
- Response generation
## Basic Structure
A typical `.gbdialog` script contains:
```basic
REM This is a comment
TALK "Hello! How can I help you today?"
HEAR user_input
IF user_input = "help" THEN
TALK "I can help you with various tasks..."
ELSE
LLM user_input
END IF
```
## Key Components
### 1. Control Flow
- `IF/THEN/ELSE/END IF` for conditional logic
- `FOR EACH/IN/NEXT` for loops
- `EXIT FOR` to break loops
### 2. User Interaction
- `HEAR variable` to get user input
- `TALK message` to send responses
- `WAIT seconds` to pause execution
### 3. Data Manipulation
- `SET variable = value` for assignment
- `GET url` to fetch external data
- `FIND table, filter` to query databases
### 4. AI Integration
- `LLM prompt` for AI-generated responses
- `ADD_TOOL tool_name` to enable functionality
- `SET_KB collection` to use knowledge bases
## Script Execution
Dialog scripts run in a sandboxed environment with:
- Access to session context and variables
- Ability to call external tools and APIs
- Integration with knowledge bases
- LLM generation capabilities
## Error Handling
The system provides built-in error handling:
- Syntax errors are caught during compilation
- Runtime errors log details but don't crash the bot
- Timeouts prevent infinite loops
- Resource limits prevent abuse

85
docs/src/chapter-02/gbdrive.md Normal file
View file

@ -0,0 +1,85 @@
# .gbdrive File Storage
The `.gbdrive` system manages file storage and retrieval using MinIO (S3-compatible object storage).
## What is .gbdrive?
`.gbdrive` provides:
- Centralized file storage for all packages
- Versioning and backup capabilities
- Access control and organization
- Integration with knowledge bases and tools
## Storage Structure
Files are organized in a bucket-per-bot structure:
```
org-prefixbot-name.gbai/
├── .gbdialog/
│ └── [script files]
├── .gbkb/
│ └── [collection folders]
├── .gbot/
│ └── config.csv
├── .gbtheme/
│ └── [theme files]
└── user-uploads/
└── [user files]
```
## File Operations
### Uploading Files
```basic
REM Files can be uploaded via API or interface
REM They are stored in the bot's MinIO bucket
```
### Retrieving Files
```basic
REM Get file content as text
GET ".gbdialog/start.bas"
REM Download files via URL
GET "user-uploads/document.pdf"
```
### File Management
- Automatic synchronization on bot start
- Change detection for hot reloading
- Version history maintenance
- Backup and restore capabilities
## Integration Points
### Knowledge Bases
- Documents are stored in .gbkb collections
- Automatic processing and embedding
- Version tracking for updates
### Themes
- Static assets served from .gbtheme
- CSS, JS, and HTML files
- Caching for performance
### Tools
- Tool scripts stored in .gbdialog
- AST and compiled versions
- Dependency management
## Access Control
Files have different access levels:
- **Public**: Accessible without authentication
- **Authenticated**: Requires user login
- **Bot-only**: Internal bot files
- **Admin**: Configuration and sensitive files
## Storage Backends
Supported storage options:
- **MinIO** (default): Self-hosted S3-compatible
- **AWS S3**: Cloud object storage
- **Local filesystem**: Development and testing
- **Hybrid**: Multiple backends with fallback

79
docs/src/chapter-02/gbkb.md Normal file
View file

@ -0,0 +1,79 @@
# .gbkb Knowledge Base
The `.gbkb` package manages knowledge base collections that provide contextual information to the bot during conversations.
## What is .gbkb?
`.gbkb` (General Bot Knowledge Base) collections store:
- Document collections for semantic search
- Vector embeddings for similarity matching
- Metadata and indexing information
- Access control and organization
## Knowledge Base Structure
Each `.gbkb` collection is organized as:
```
collection-name.gbkb/
├── documents/
│ ├── doc1.pdf
│ ├── doc2.txt
│ └── doc3.html
├── embeddings/ # Auto-generated
├── metadata.json # Collection info
└── index.json # Search indexes
```
## Supported Formats
The knowledge base can process:
- **Text files**: .txt, .md, .html
- **Documents**: .pdf, .docx
- **Web content**: URLs and web pages
- **Structured data**: .csv, .json
## Vector Embeddings
Each document is processed into vector embeddings using:
- BGE-small-en-v1.5 model (default)
- Chunking for large documents
- Metadata extraction and indexing
- Semantic similarity scoring
## Collection Management
### Creating Collections
```basic
ADD_KB "company-policies"
ADD_WEBSITE "https://company.com/docs"
```
### Using Collections
```basic
SET_KB "company-policies"
LLM "What is the vacation policy?"
```
### Multiple Collections
```basic
ADD_KB "policies"
ADD_KB "procedures"
ADD_KB "faqs"
REM All active collections contribute to context
```
## Semantic Search
The knowledge base provides:
- **Similarity search**: Find relevant documents
- **Hybrid search**: Combine semantic and keyword
- **Context injection**: Automatically add to LLM prompts
- **Relevance scoring**: Filter by similarity threshold
## Integration with Dialogs
Knowledge bases are automatically used when:
- `SET_KB` or `ADD_KB` is called
- Answer mode is set to use documents
- LLM queries benefit from contextual information

82
docs/src/chapter-02/gbot.md Normal file
View file

@ -0,0 +1,82 @@
# .gbot Bot Configuration
The `.gbot` package contains configuration files that define bot behavior, parameters, and operational settings.
## What is .gbot?
`.gbot` files configure:
- Bot identity and description
- LLM provider settings
- Answer modes and behavior
- Context management
- Integration parameters
## Configuration Structure
The primary configuration file is `config.csv`:
```csv
key,value
bot_name,Customer Support Assistant
bot_description,AI-powered support agent
answer_mode,1
llm_provider,openai
llm_model,gpt-4
temperature,0.7
max_tokens,1000
system_prompt,You are a helpful customer support agent...
```
## Key Configuration Parameters
### Bot Identity
- `bot_name`: Display name for the bot
- `bot_description`: Purpose and capabilities
- `version`: Bot version for tracking
### LLM Configuration
- `llm_provider`: openai, azure, local
- `llm_model`: Model name (gpt-4, claude-3, etc.)
- `temperature`: Creativity control (0.0-1.0)
- `max_tokens`: Response length limit
### Answer Modes
- `0`: Direct LLM responses only
- `1`: LLM with tool calling
- `2`: Knowledge base documents only
- `3`: Include web search results
- `4`: Mixed mode with tools and KB
### Context Management
- `context_window`: Number of messages to retain
- `context_provider`: How context is managed
- `memory_enabled`: Whether to use bot memory
## Configuration Loading
The system loads configuration from:
1. `config.csv` in the .gbot package
2. Environment variables (override)
3. Database settings (persistent)
4. Runtime API calls (temporary)
## Configuration Precedence
Settings are applied in this order (later overrides earlier):
1. Default values
2. .gbot/config.csv
3. Environment variables
4. Database configuration
5. Runtime API updates
## Dynamic Configuration
Some settings can be changed at runtime:
```basic
REM Change answer mode dynamically
SET_BOT_MEMORY "answer_mode", "2"
```
## Bot Memory
The `SET_BOT_MEMORY` and `GET_BOT_MEMORY` keywords allow storing and retrieving bot-specific data that persists across sessions.

95
docs/src/chapter-02/gbtheme.md Normal file
View file

@ -0,0 +1,95 @@
# .gbtheme UI Theming
The `.gbtheme` package contains user interface customization files for web and other frontend interfaces.
## What is .gbtheme?
`.gbtheme` defines the visual appearance and user experience:
- CSS stylesheets for styling
- HTML templates for structure
- JavaScript for interactivity
- Assets like images and fonts
## Theme Structure
A typical theme package contains:
```
theme-name.gbtheme/
├── web/
│ ├── index.html # Main template
│ ├── chat.html # Chat interface
│ └── login.html # Authentication
├── css/
│ ├── main.css # Primary styles
│ ├── components.css # UI components
│ └── responsive.css # Mobile styles
├── js/
│ ├── app.js # Application logic
│ └── websocket.js # Real-time communication
└── assets/
├── images/
├── fonts/
└── icons/
```
## Web Interface
The main web interface consists of:
### HTML Templates
- `index.html`: Primary application shell
- `chat.html`: Conversation interface
- Component templates for reusable UI
### CSS Styling
- Color schemes and typography
- Layout and responsive design
- Animation and transitions
- Dark/light mode support
### JavaScript
- WebSocket communication
- UI state management
- Event handling
- API integration
## Theme Variables
Themes can use CSS custom properties for easy customization:
```css
:root {
--primary-color: #2563eb;
--secondary-color: #64748b;
--background-color: #ffffff;
--text-color: #1e293b;
--border-radius: 8px;
--spacing-unit: 8px;
}
```
## Responsive Design
Themes should support:
- **Desktop**: Full-featured interface
- **Tablet**: Adapted layout and interactions
- **Mobile**: Touch-optimized experience
- **Accessibility**: Screen reader and keyboard support
## Theme Switching
Multiple themes can be provided:
- Light and dark variants
- High contrast for accessibility
- Brand-specific themes
- User-selected preferences
## Customization Points
Key areas for theme customization:
- Color scheme and branding
- Layout and component arrangement
- Typography and spacing
- Animation and micro-interactions
- Iconography and imagery

22
docs/src/chapter-02/summary.md Normal file
View file

@ -0,0 +1,22 @@
# Chapter 02 – Package Documentation Overview
This chapter provides a concise overview of the GeneralBots package types introduced in Chapter 02. Each package type is documented in its own markdown file. Below is a quick reference with brief descriptions and links to the full documentation.
| Package | File | Description |
|---------|------|-------------|
| **.gbai** | [gbai.md](gbai.md) | Defines the overall application architecture, metadata, and package hierarchy. |
| **.gbdialog** | [gbdialog.md](gbdialog.md) | Contains BASIC‑style dialog scripts that drive conversation flow and tool integration. |
| **.gbdrive** | [gbdrive.md](gbdrive.md) | Manages file storage and retrieval via MinIO (or other S3‑compatible backends). |
| **.gbkb** | [gbkb.md](gbkb.md) | Handles knowledge‑base collections, vector embeddings, and semantic search. |
| **.gbot** | [gbot.md](gbot.md) | Stores bot configuration (CSV) for identity, LLM settings, answer modes, and runtime parameters. |
| **.gbtheme** | [gbtheme.md](gbtheme.md) | Provides UI theming assets: CSS, HTML templates, JavaScript, and static resources. |
## How to Use This Overview
- **Navigate**: Click the file links above to read the detailed documentation for each package.
- **Reference**: Use this table as a quick lookup when developing or extending a GeneralBots application.
- **Extend**: When adding new package types, update this table and create a corresponding markdown file.
---
*This summary was added to fill the missing documentation for Chapter 02.*

14
docs/src/chapter-03/README.md Normal file
View file

@ -0,0 +1,14 @@
## gbkb Reference
The knowledge‑base package provides three main commands:
- **ADD_KB** – Create a new vector collection.
- **SET_KB** – Switch the active collection for the current session.
- **ADD_WEBSITE** – Crawl a website and add its pages to the active collection.
**Example:**
```bas
ADD_KB "support_docs"
SET_KB "support_docs"
ADD_WEBSITE "https://docs.generalbots.com"
```
These commands are implemented in the Rust code under `src/kb/` and exposed to BASIC scripts via the engine.

43
docs/src/chapter-03/caching.md Normal file
View file

@ -0,0 +1,43 @@
# Caching (Optional)
Caching can improve response times for frequently accessed knowledge‑base queries.
## In‑Memory Cache
The bot maintains an LRU (least‑recently‑used) cache of the last 100 `FIND` results. This cache is stored in the bot’s process memory and cleared on restart.
## Persistent Cache
For longer‑term caching, the `gbkb` package can write query results to a local SQLite file (`cache.db`). The cache key is a hash of the query string and collection name.
## Configuration
Add the following to `.gbot/config.csv`:
```csv
key,value
cache_enabled,true
cache_max_entries,500
```
## Usage Example
```basic
SET_KB "company-policies"
FIND "vacation policy" INTO RESULT ' first call hits VectorDB
FIND "vacation policy" INTO RESULT ' second call hits cache
TALK RESULT
```
The second call returns instantly from the cache.
## Cache Invalidation
- When a document is added or updated, the cache for that collection is cleared.
- Manual invalidation: `CLEAR_CACHE "company-policies"` (custom keyword provided by the system).
## Benefits
- Reduces latency for hot queries.
- Lowers load on VectorDB.
- Transparent to the script author; caching is automatic.

36
docs/src/chapter-03/context-compaction.md Normal file
View file

@ -0,0 +1,36 @@
# Context Compaction
When a conversation grows long, the bot’s context window can exceed the LLM’s token limit. **Context compaction** reduces the stored history while preserving essential information.
## Strategies
1. **Summarization** – Periodically run `TALK FORMAT` with a summarization prompt and replace older messages with the summary.
2. **Memory Pruning** – Use `SET_BOT_MEMORY` to store only key facts (e.g., user name, preferences) and discard raw chat logs.
3. **Chunk Rotation** – Keep a sliding window of the most recent *N* messages (configurable via `context_window` in `.gbot/config.csv`).
## Implementation Example
```basic
' After 10 exchanges, summarize
IF MESSAGE_COUNT >= 10 THEN
TALK "Summarizing recent conversation..."
SET_BOT_MEMORY "summary" FORMAT(RECENT_MESSAGES, "summarize")
CLEAR_MESSAGES ' removes raw messages
ENDIF
```
## Configuration
- `context_window` (in `.gbot/config.csv`) defines how many recent messages are kept automatically.
- `memory_enabled` toggles whether the bot uses persistent memory.
## Benefits
- Keeps token usage within limits.
- Improves response relevance by focusing on recent context.
- Allows long‑term facts to persist without bloating the prompt.
## Caveats
- Over‑aggressive pruning may lose important details.
- Summaries should be concise (max 200 tokens) to avoid re‑inflating the context.

22
docs/src/chapter-03/indexing.md Normal file
View file

@ -0,0 +1,22 @@
# Document Indexing
When a document is added to a knowledge‑base collection with `ADD_KB` or `ADD_WEBSITE`, the system performs several steps to make it searchable:
1. **Content Extraction** – Files are read and plain‑text is extracted (PDF, DOCX, HTML, etc.).
2. **Chunking** – The text is split into 500‑token chunks to keep embeddings manageable.
3. **Embedding Generation** – Each chunk is sent to the configured LLM embedding model (default **BGE‑small‑en‑v1.5**) to produce a dense vector.
4. **Storage** – Vectors, along with metadata (source file, chunk offset), are stored in VectorDB under the collection’s namespace.
5. **Indexing** – VectorDB builds an IVF‑PQ index for fast approximate nearest‑neighbor search.
## Index Refresh
If a document is updated, the system re‑processes the file and replaces the old vectors. The index is automatically refreshed; no manual action is required.
## Example
```basic
ADD_KB "company-policies"
ADD_WEBSITE "https://example.com/policies"
```
After execution, the `company-policies` collection contains indexed vectors ready for semantic search via the `FIND` keyword.

43
docs/src/chapter-03/qdrant.md Normal file
View file

@ -0,0 +1,43 @@
# VectorDB Integration
GeneralBots uses **VectorDB** as the vector database for storing and searching embeddings. The Rust client for the configured VectorDB is used to communicate with the service.
## Configuration
The connection is configured via environment variables:
```env
VECTORDB_URL=http://localhost:6333
VECTORDB_API_KEY=your-api-key # optional
```
These values are read at startup and passed to the `VectorDBClient`.
## Collection Mapping
Each `.gbkb` collection maps to a VectorDB collection with the same name. For example, a knowledge base named `company-policies` becomes a VectorDB collection `company-policies`.
## Operations
- **Insert** – Performed during indexing (see Chapter 03).
- **Search** – Executed by the `FIND` keyword, which sends a query vector and retrieves the top‑k nearest neighbors.
- **Delete/Update** – When a document is removed or re‑indexed, the corresponding vectors are deleted and replaced.
## Performance Tips
- Keep the number of vectors per collection reasonable (tens of thousands) for optimal latency.
- Adjust VectorDB’s `hnsw` parameters in `VectorDBClient::new` if you need higher recall.
- Use the `FILTER` option to restrict searches by metadata (e.g., source file).
## Example `FIND` Usage
```basic
SET_KB "company-policies"
FIND "vacation policy" INTO RESULT
TALK RESULT
```
The keyword internally:
1. Generates an embedding for the query string.
2. Calls VectorDB’s `search` API.
3. Returns the most relevant chunk as `RESULT`.

36
docs/src/chapter-03/semantic-search.md Normal file
View file

@ -0,0 +1,36 @@
# Semantic Search
Semantic search enables the bot to retrieve information based on meaning rather than exact keyword matches. It leverages the vector embeddings stored in VectorDB.
## How It Works
1. **Query Embedding** – The user’s query string is converted into a dense vector using the same embedding model as the documents.
2. **Nearest‑Neighbor Search** – VectorDB returns the top‑k vectors that are closest to the query vector.
3. **Result Formatting** – The matching document chunks are concatenated and passed to the LLM as context for the final response.
## Using the `FIND` Keyword
```basic
SET_KB "company-policies"
FIND "how many vacation days do I have?" INTO RESULT
TALK RESULT
```
- `SET_KB` selects the collection.
- `FIND` performs the semantic search.
- `RESULT` receives the best matching snippet.
## Parameters
- **k** – Number of results to return (default 3). Can be overridden with `FIND "query" LIMIT 5 INTO RESULT`.
- **filter** – Optional metadata filter, e.g., `FILTER source="policy.pdf"`.
## Best Practices
- Keep the query concise (1‑2 sentences) for optimal embedding quality.
- Use `FORMAT` to clean up the result before sending to the user.
- Combine with `GET_BOT_MEMORY` to store frequently accessed answers.
## Performance
Semantic search latency is typically < 100 ms for collections under 50 k vectors. Larger collections may require tuning VectorDB’s HNSW parameters.

23
docs/src/chapter-03/summary.md Normal file
View file

@ -0,0 +1,23 @@
# Chapter 03 – Knowledge‑Base (VectorDB) Documentation Overview
This chapter explains how GeneralBots manages knowledge‑base collections, indexing, caching, and semantic search. The implementation now references a generic **VectorDB** (instead of a specific Qdrant instance) and highlights the use of the **.gbdrive** package for storage when needed.
| Document | File | Description |
|----------|------|-------------|
| **README** | [README.md](README.md) | High‑level reference for the `.gbkb` package and its core commands (`ADD_KB`, `SET_KB`, `ADD_WEBSITE`). |
| **Caching** | [caching.md](caching.md) | Optional in‑memory and persistent SQLite caching to speed up frequent `FIND` queries. |
| **Context Compaction** | [context-compaction.md](context-compaction.md) | Techniques to keep the LLM context window within limits (summarization, memory pruning, sliding window). |
| **Indexing** | [indexing.md](indexing.md) | Process of extracting, chunking, embedding, and storing document vectors in the VectorDB. |
| **VectorDB Integration** | [qdrant.md](qdrant.md) | (Renamed) Details the VectorDB connection, collection mapping, and operations. References to **Qdrant** have been generalized to **VectorDB**. |
| **Semantic Search** | [semantic-search.md](semantic-search.md) | How the `FIND` keyword performs meaning‑based retrieval using the VectorDB. |
| **Vector Collections** | [vector-collections.md](vector-collections.md) | Definition and management of vector collections, including creation, document addition, and usage in dialogs. |
## How to Use This Overview
- **Navigate**: Click the file links to read the full documentation for each topic.
- **Reference**: Use this table as a quick lookup when developing or extending knowledge‑base functionality.
- **Update**: When the underlying storage or VectorDB implementation changes, edit the corresponding markdown files and keep this summary in sync.
---
*This summary was added to provide a cohesive overview of Chapter 03, aligning terminology with the current architecture (VectorDB, .gbdrive, etc.).*

45
docs/src/chapter-03/vector-collections.md Normal file
View file

@ -0,0 +1,45 @@
# Vector Collections
A **vector collection** is a set of documents that have been transformed into vector embeddings for fast semantic similarity search. Each collection lives under a `.gbkb` folder and is identified by a unique name.
## Creating a Collection
Use the `ADD_KB` keyword in a dialog script:
```basic
ADD_KB "company-policies"
```
This creates a new collection named `company-policies` in the bot’s knowledge base.
## Adding Documents
Documents can be added directly from files or by crawling a website:
```basic
ADD_KB "company-policies" ' adds a new empty collection
ADD_WEBSITE "https://example.com/policies"
```
The system will download the content, split it into chunks, generate embeddings using the default LLM model, and store them in the collection.
## Managing Collections
- `SET_KB "collection-name"` – selects the active collection for subsequent `ADD_KB` or `FIND` calls.
- `LIST_KB` – (not a keyword, but you can query via API) lists all collections.
## Use in Dialogs
When a collection is active, the `FIND` keyword searches across its documents, and the `GET_BOT_MEMORY` keyword can retrieve relevant snippets to inject into LLM prompts.
```basic
SET_KB "company-policies"
FIND "vacation policy" INTO RESULT
TALK RESULT
```
## Technical Details
- Embeddings are generated with the BGE‑small‑en‑v1.5 model.
- Vectors are stored in VectorDB (see Chapter 04).
- Each document is chunked into 500‑token pieces for efficient retrieval.

1
docs/src/chapter-04/README.md Normal file
View file

@ -0,0 +1 @@
# Chapter 04: gbtheme Reference

50
docs/src/chapter-04/css.md Normal file
View file

@ -0,0 +1,50 @@
# CSS Customization
The **gbtheme** CSS files define the visual style of the bot UI. They are split into three layers to make them easy to extend.
## Files
| File | Role |
|------|------|
| `main.css` | Core layout, typography, and global variables. |
| `components.css` | Styles for reusable UI components (buttons, cards, modals). |
| `responsive.css` | Media queries for mobile, tablet, and desktop breakpoints. |
## CSS Variables (in `main.css`)
```css
:root {
--primary-color: #2563eb;
--secondary-color: #64748b;
--background-color: #ffffff;
--text-color: #1e293b;
--border-radius: 8px;
--spacing-unit: 8px;
}
```
Changing a variable updates the entire theme without editing individual rules.
## Extending the Theme
1. **Add a new variable** – Append to `:root` and reference it in any selector.
2. **Override a component** – Duplicate the selector in `components.css` after the original definition; the later rule wins.
3. **Create a dark mode** – Add a `@media (prefers-color-scheme: dark)` block that redefines the variables.
```css
@media (prefers-color-scheme: dark) {
:root {
--primary-color: #3b82f6;
--background-color: #111827;
--text-color: #f9fafb;
}
}
```
## Best Practices
* Keep the file size small – avoid large image data URIs; store images in `assets/`.
* Use `rem` units for font sizes; they scale with the root `font-size`.
* Limit the depth of nesting; flat selectors improve performance.
All CSS files are loaded in `index.html` in the order: `main.css`, `components.css`, `responsive.css`.

71
docs/src/chapter-04/html.md Normal file
View file

@ -0,0 +1,71 @@
# HTML Templates
The **gbtheme** HTML files provide the markup for the bot’s UI. They are deliberately minimal to allow easy customization.
## index.html
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>GeneralBots Chat</title>
<link rel="stylesheet" href="css/main.css">
<link rel="stylesheet" href="css/components.css">
<link rel="stylesheet" href="css/responsive.css">
</head>
<body>
<header class="app-header">
<h1>GeneralBots</h1>
</header>
<main id="chat-container"></main>
<footer class="app-footer">
<p>&copy; 2025 GeneralBots</p>
</footer>
<script src="js/websocket.js"></script>
<script src="js/app.js"></script>
</body>
</html>
```
*Loads the CSS layers and the JavaScript modules.*
## chat.html
```html
<div class="chat-window">
<div id="messages" class="messages"></div>
<div class="input-area">
<input id="user-input" type="text" placeholder="Type a message…" autocomplete="off"/>
<button id="send-btn">Send</button>
</div>
<div id="typing-indicator" class="typing">…</div>
</div>
```
*Used by `app.js` to render the conversation view.*
## login.html
```html
<div class="login-form">
<h2>Login</h2>
<input id="username" type="text" placeholder="Username"/>
<input id="password" type="password" placeholder="Password"/>
<button id="login-btn">Login</button>
</div>
```
*Optional page displayed when the bot requires authentication.*
## Customization Tips
* Replace the `<header>` content with your brand logo.
* Add additional `<meta>` tags (e.g., Open Graph) in `index.html`.
* Insert extra `<script>` tags for analytics or feature flags.
* Use the `assets/` folder to store images referenced via `<img src="assets/images/logo.png">`.
All HTML files are located under the theme’s `web/` directory and are referenced by the server based on the `.gbtheme` configuration in `config.csv`.

38
docs/src/chapter-04/structure.md Normal file
View file

@ -0,0 +1,38 @@
# Theme Structure
The **gbtheme** package follows a conventional layout that separates concerns between markup, styling, scripts, and assets.
```
theme-name.gbtheme/
├── web/
│ ├── index.html # Main application shell
│ ├── chat.html # Conversation UI
│ └── login.html # Authentication page
├── css/
│ ├── main.css # Core styles
│ ├── components.css # UI component styling
│ └── responsive.css # Media‑query breakpoints
├── js/
│ ├── app.js # Front‑end logic, WebSocket handling
│ └── websocket.js # Real‑time communication layer
└── assets/
├── images/
├── fonts/
└── icons/
```
### Design Principles
* **Separation of concerns** – HTML defines structure, CSS defines appearance, JS defines behavior.
* **Custom properties** – `css/variables.css` (included in `main.css`) provides theme colors, spacing, and radius that can be overridden per‑bot.
* **Responsive** – `responsive.css` uses mobile‑first breakpoints (`@media (min-width: 768px)`) to adapt the layout.
* **Asset locality** – All images, fonts, and icons are stored under `assets/` to keep the theme self‑contained and portable.
### Extending a Theme
1. Duplicate an existing theme folder (e.g., `default.gbtheme` → `mybrand.gbtheme`).
2. Edit `css/main.css` to change colors via the `:root` variables.
3. Replace `web/index.html` header/footer with brand‑specific markup.
4. Add new icons to `assets/icons/` and reference them in the HTML.
The system automatically picks up any theme placed under `@/templates/…` when the bot’s configuration (`.gbtheme` entry in `config.csv`) points to the folder name.

32
docs/src/chapter-04/web-interface.md Normal file
View file

@ -0,0 +1,32 @@
# Web Interface
The **gbtheme** web interface provides the front‑end experience for end users. It consists of three core HTML pages and a set of JavaScript modules that handle real‑time communication with the bot server.
## Pages
| Page | Purpose |
|------|---------|
| `index.html` | Application shell, loads the main JavaScript bundle and displays the navigation bar. |
| `chat.html` | Primary conversation view – shows the chat transcript, input box, and typing indicator. |
| `login.html` | Simple authentication screen used when the bot is configured with a login flow. |
## JavaScript Modules
* **app.js** – Initializes the WebSocket connection, routes incoming bot messages to the UI, and sends user input (`TALK`) back to the server.
* **websocket.js** – Low‑level wrapper around the browser’s `WebSocket` API, handling reconnection logic and ping/pong keep‑alive.
## Interaction Flow
1. **Load** – `index.html` loads `app.js`, which creates a `WebSocket` to `ws://<host>/ws`.
2. **Handshake** – The server sends a `HELLO` message containing bot metadata (name, version).
3. **User Input** – When the user presses *Enter* in the chat input, `app.js` sends a `TALK` JSON payload.
4. **Bot Response** – The server streams `MESSAGE` events; `app.js` appends them to the chat window.
5. **Typing Indicator** – While the LLM processes, the server sends a `TYPING` event; the UI shows an animated ellipsis.
## Customization Points
* **CSS Variables** – Override colors, fonts, and spacing in `css/main.css` (`:root { --primary-color: … }`).
* **HTML Layout** – Replace the `<header>` or `<footer>` sections in `index.html` to match branding.
* **JS Hooks** – Add custom event listeners in `app.js` (e.g., analytics on `MESSAGE` receipt).
All files are located under the theme’s `web/` and `js/` directories as described in the [Theme Structure](./structure.md).

9
docs/src/chapter-05/README.md Normal file
View file

@ -0,0 +1,9 @@
# Chapter 05 – gbdialog Reference
This chapter documents the BASIC dialog language used to define conversational flows. It includes:
* **Basics** – Core language concepts and control flow.
* **Templates** – Example `.bas` files (`start.bas`, `auth.bas`, `generate-summary.bas`, enrollment tool).
* **Keywords** – Detailed reference for every keyword defined in `src/basic/keywords/`.
Each keyword page contains syntax, description, parameters, and a real example taken from the official template files.

36
docs/src/chapter-05/basics.md Normal file
View file

@ -0,0 +1,36 @@
# Dialog Basics
BASIC dialogs are plain‑text scripts that the GeneralBots engine compiles into an abstract syntax tree (AST). The language is intentionally simple, using English‑like keywords.
## Core Concepts
* **Lines** – Each line is either a command (`TALK`, `HEAR`, etc.) or a comment (`REM`).
* **Variables** – Declared with `SET name = expression`. Types are inferred at runtime.
* **Control Flow** – `IF … THEN … ENDIF`, `FOR EACH … IN … NEXT`, and `EXIT FOR`.
* **Blocks** – Enclosed in `{ … }` for multi‑line statements (e.g., `TALK { … }`).
## Example Script (`start.bas`)
```basic
REM Simple greeting dialog
SET user_name = "Guest"
TALK "Hello, " + user_name + "! How can I help you today?"
HEAR user_input
IF user_input = "help" THEN
TALK "Sure, I can assist with account info, orders, or support."
ELSE
TALK "Sorry, I didn't understand."
ENDIF
```
## Execution Model
1. **Parse** – The script is tokenized and turned into an AST.
2. **Compile** – Keywords are mapped to Rust functions (see `src/basic/keywords/`).
3. **Run** – The engine walks the AST, executing each node synchronously, while async tasks (e.g., LLM calls) are spawned as needed.
## Best Practices
* Keep scripts short; split complex flows into multiple `.gbdialog` files and `ADD_TOOL` them.
* Use `SET_BOT_MEMORY` for data that must persist across sessions.
* Avoid heavy computation inside the script; offload to LLM or external tools.

28
docs/src/chapter-05/keyword-add-kb.md Normal file
View file

@ -0,0 +1,28 @@
# ADD_KB Keyword
**Syntax**
```
ADD_KB "collection-name"
```
**Parameters**
- `"collection-name"` – Identifier for a new knowledge‑base folder inside the bot’s `.gbkb` directory.
**Description**
`ADD_KB` does not create a physical folder on disk; instead it informs the **context manager** (found in `src/context/`) that a new vector‑DB collection, identified by the given name, should be considered part of the current conversation’s context. The collection is treated as a drive‑based knowledge source that can be queried by the prompt processor.
Multiple `ADD_KB` calls can be issued in a single dialog or by a tool invoked through its trigger phrase. Each call adds the named collection to the session’s **additional_kb_collections** list, which the prompt processor later reads to retrieve relevant documents. This makes it possible for a user to say things like “talk about the latest policy” after a tool has prepared a new collection.
The keyword is typically used at the start of a conversation or inside a tool that changes context (e.g., after uploading a set of files to a drive or after an email attachment is processed). By adding the collection to the context, subsequent `FIND` or `LLM` calls automatically incorporate the newly available knowledge.
**Example**
```basic
ADD_KB "company-policies"
TALK "Knowledge base 'company-policies' is now part of the conversation context."
```
After execution, the `company-policies` collection is registered with the context manager and will be consulted for any future queries in this session.

38
docs/src/chapter-05/keyword-add-tool.md Normal file
View file

@ -0,0 +1,38 @@
# ADD_TOOL Keyword
**Syntax**
```
ADD_TOOL "tool-path.bas"
```
**Parameters**
- `"tool-path.bas"` – Relative path to a `.bas` file inside the `.gbdialog` package (e.g., `enrollment.bas`).
**Description**
`ADD_TOOL` compiles the specified BASIC script and registers it as a tool for the current session. The compiled tool becomes available for use in the same conversation, allowing its keywords to be invoked.
The keyword performs the following steps:
1. Extracts the tool name from the provided path (removing the `.bas` extension and any leading `.gbdialog/` prefix).
2. Validates that the tool name is not empty.
3. Spawns an asynchronous task that:
- Checks that the tool exists and is active for the bot in the `basic_tools` table.
- Inserts a row into `session_tool_associations` linking the tool to the current session (or does nothing if the association already exists).
4. Returns a success message indicating the tool is now available, or an error if the tool cannot be found or the database operation fails.
**Example**
```basic
ADD_TOOL "enrollment.bas"
TALK "Enrollment tool added. You can now use ENROLL command."
```
After execution, the `enrollment.bas` script is compiled and its keywords become callable in the current dialog.
**Implementation Notes**
- The operation runs in a separate thread with its own Tokio runtime to avoid blocking the main engine.
- Errors are logged and propagated as runtime errors in the BASIC script.

26
docs/src/chapter-05/keyword-add-website.md Normal file
View file

@ -0,0 +1,26 @@
# ADD_WEBSITE Keyword
**Syntax**
```
ADD_WEBSITE "https://example.com"
```
**Parameters**
- `"url"` – A valid HTTP or HTTPS URL pointing to a website that should be added to the conversation context.
**Description**
`ADD_WEBSITE` validates the provided URL and, when the `web_automation` feature is enabled, launches a headless browser to crawl the site, extract its textual content, and index it into a vector‑DB collection associated with the current user. The collection name is derived from the URL and the bot’s identifiers. After indexing, the website becomes a knowledge source that can be queried by `FIND` or `LLM` calls.
If the feature is not compiled, the keyword returns an error indicating that web automation is unavailable.
**Example**
```basic
ADD_WEBSITE "https://en.wikipedia.org/wiki/General_Bots"
TALK "Website added. You can now search its content with FIND."
```
After execution, the Wikipedia page is crawled, its text is stored in a KB collection, and subsequent `FIND "website_of" "General Bots"` calls will consider this new source.

26
docs/src/chapter-05/keyword-clear-tools.md Normal file
View file

@ -0,0 +1,26 @@
# CLEAR_TOOLS Keyword
**Syntax**
```
CLEAR_TOOLS
```
**Parameters**
_None_ – This keyword takes no arguments.
**Description**
`CLEAR_TOOLS` removes every tool that has been added to the current conversation session. It clears the list of active tools stored in the session‑tool association table, effectively resetting the tool environment for the dialog. After execution, no previously added tools (via `ADD_TOOL`) remain available.
**Example**
```basic
ADD_TOOL "enrollment.bas"
TALK "Enrollment tool added."
CLEAR_TOOLS
TALK "All tools have been cleared from this conversation."
```
After `CLEAR_TOOLS` runs, the `enrollment.bas` tool is no longer accessible in the same session.

26
docs/src/chapter-05/keyword-create-draft.md Normal file
View file

@ -0,0 +1,26 @@
# CREATE_DRAFT Keyword
**Syntax**
```
CREATE_DRAFT "to-address", "subject", "reply-text"
```
**Parameters**
- `"to-address"` – Email address of the recipient.
- `"subject"` – Subject line for the draft email.
- `"reply-text"` – Body content for the draft. If a previous email exists in the user's mailbox to the same address, its content is appended after a separator.
**Description**
`CREATE_DRAFT` composes an email draft and saves it to the user's mailbox. It first checks whether a prior email has been sent to the same recipient using the `GET_LATEST_SENT_TO` helper. If such an email exists, its body (converted to HTML line breaks) is appended to the new reply text, separated by `<br><hr><br>`. The combined content is then stored as a draft via the email service configured in the application (`save_email_draft`). The keyword returns a success message or an error string.
**Example**
```basic
CREATE_DRAFT "john.doe@example.com", "Project Update", "Here is the latest status..."
TALK "Draft created and saved."
```
If an earlier email to `john.doe@example.com` exists, the draft will contain the new reply followed by the previous email content, allowing the user to continue the conversation seamlessly.

34
docs/src/chapter-05/keyword-create-site.md Normal file
View file

@ -0,0 +1,34 @@
# CREATE_SITE Keyword
**Syntax**
```
CREATE_SITE "alias", "template-dir", "prompt"
```
**Parameters**
- `"alias"` – Name of the new site (used as a folder name under the configured site path).
- `"template-dir"` – Relative path to a directory containing HTML template files that will be combined.
- `"prompt"` – Text prompt sent to the LLM to generate the final site content.
**Description**
`CREATE_SITE` generates a new static website based on existing HTML templates and an LLM‑generated prompt. The keyword performs the following steps:
1. Creates a directory for the new site at `<site_path>/<alias>`.
2. Reads all `.html` files from `<site_path>/<template-dir>` and concatenates their contents, separating each with a clear delimiter.
3. Constructs a prompt that includes the combined template content and the user‑provided `prompt`.
4. Sends the prompt to the configured LLM provider (`utils::call_llm`) and receives generated HTML.
5. Writes the LLM output to `<site_path>/<alias>/index.html`.
The resulting site can be served directly from the `site_path` directory. Errors during directory creation, file reading, or LLM generation are logged and returned as error messages.
**Example**
```basic
CREATE_SITE "my_blog", "templates/blog", "Generate a modern blog homepage for a tech writer."
TALK "Site created at /my_blog. Access it via the web server."
```
After execution, a folder `my_blog` is created with an `index.html` containing the LLM‑generated page, ready to be served.

35
docs/src/chapter-05/keyword-exit-for.md Normal file
View file

@ -0,0 +1,35 @@
# EXIT FOR Keyword
**Syntax**
```
EXIT FOR
```
**Parameters**
_None_ – This keyword takes no arguments.
**Description**
`EXIT FOR` terminates the execution of the nearest enclosing `FOR EACH … IN … NEXT` loop prematurely. When the interpreter encounters `EXIT FOR`, it stops iterating over the collection and continues execution after the `NEXT` statement that matches the loop variable.
**Example**
```basic
FOR EACH item IN my_list
IF item = "stop" THEN
EXIT FOR
ENDIF
TALK item
NEXT item
TALK "Loop ended."
```
In this script, the loop stops as soon as `item` equals `"stop"`, and the subsequent `TALK "Loop ended."` is executed.
**Usage Notes**
- `EXIT FOR` can only be used inside a `FOR EACH … IN … NEXT` block.
- It does not accept any parameters; it simply signals an early exit.
- The keyword is case‑insensitive; `exit for` works the same way.

37
docs/src/chapter-05/keyword-find.md Normal file
View file

@ -0,0 +1,37 @@
# FIND Keyword
**Syntax**
```
FIND "table-name", "filter-expression"
```
**Parameters**
- `"table-name"` – The name of the database table to query.
- `"filter-expression"` – A simple `column=value` expression used to filter rows.
**Description**
`FIND` executes a read‑only query against the configured PostgreSQL database. It builds a SQL statement of the form:
```sql
SELECT * FROM table-name WHERE filter-expression LIMIT 10
```
The keyword returns an array of dynamic objects representing the matching rows. The result can be used directly in BASIC scripts or passed to other keywords (e.g., `TALK`, `FORMAT`). Errors during query execution are logged and returned as runtime errors.
**Example**
```basic
SET results = FIND "customers", "country=US"
TALK "Found " + LENGTH(results) + " US customers."
```
The script retrieves up to ten rows from the `customers` table where the `country` column equals `US` and stores them in `results`. The `LENGTH` function (provided by the BASIC runtime) can then be used to count the rows.
**Implementation Notes**
- The filter expression is parsed by `utils::parse_filter` and bound safely to prevent SQL injection.
- Only a limited subset of SQL is supported (simple equality filters). Complex queries should be performed via custom tools or the `GET` keyword.
- The keyword runs synchronously within the script but performs the database call on a separate thread to avoid blocking the engine.

30
docs/src/chapter-05/keyword-first.md Normal file
View file

@ -0,0 +1,30 @@
# FIRST Keyword
**Syntax**
```
FIRST "text"
```
**Parameters**
- `"text"` – A string expression from which the first word will be extracted.
**Description**
`FIRST` returns the first whitespace‑separated token of the provided string. If the string is empty or contains only whitespace, the result is an empty string. The keyword is useful for extracting a leading command or identifier from user input.
**Example**
```basic
SET command = FIRST user_input
TALK "You entered the command: " + command
```
If `user_input` is `"search books about Rust"`, `FIRST` returns `"search"`.
**Implementation Notes**
- The keyword splits the string on any whitespace (spaces, tabs, newlines) and returns the first element.
- It does not modify the original string.
- Case‑insensitive; the returned word preserves the original casing.

42
docs/src/chapter-05/keyword-for-each.md Normal file
View file

@ -0,0 +1,42 @@
# FOR EACH Keyword
**Syntax**
```
FOR EACH $var IN $collection
// block of statements
NEXT $var
```
**Parameters**
- `$var` – Identifier that will hold each element of the collection during iteration.
- `$collection` – An array or iterable expression whose items will be traversed.
**Description**
`FOR EACH` iterates over every element of the supplied collection, assigning the current element to the loop variable `$var` for the duration of the block. The block is executed once per element. After the loop finishes, execution continues after the matching `NEXT $var` statement.
If the collection is not an array, the keyword raises a runtime error indicating the expected type.
**Example**
```basic
SET numbers = [1, 2, 3, 4, 5]
FOR EACH n IN numbers
TALK "Number: " + n
NEXT n
TALK "All numbers processed."
```
The script outputs each number in the list sequentially and then prints a final message.
**Control Flow**
- `EXIT FOR` can be used inside the block to break out of the loop early.
- Nested `FOR EACH` loops are supported; each must have a distinct loop variable.
**Implementation Notes**
- The keyword evaluates the collection expression once before entering the loop.
- The loop variable is scoped to the block; it does not affect variables outside the loop.

82
docs/src/chapter-05/keyword-format.md Normal file
View file

@ -0,0 +1,82 @@
# FORMAT Keyword
The **FORMAT** keyword formats numbers, dates, and text for display. Use it when you need a quick, readable representation without writing custom code.
## Syntax
```basic
RESULT = FORMAT(VALUE, PATTERN)
```
## BASIC EXAMPLE
```basic
NUMBER = 1234.56
TEXT = "John"
DATE = "2024-03-15 14:30:00"
TALK FORMAT(NUMBER, "n") ' 1234.56
TALK FORMAT(TEXT, "Hello @!") ' Hello John!
TALK FORMAT(DATE, "dd/MM/yyyy") ' 15/03/2024
```
- **VALUE** – any number, date string (`YYYY‑MM‑DD HH:MM:SS`), or text.
- **PATTERN** – a short format string (see tables below).
## Quick Reference
### Numeric Patterns
| Pattern | Example | Output |
|---------|---------|--------|
| `n` | `FORMAT(1234.5, "n")` | `1234.50` |
| `F` | `FORMAT(1234.5, "F")` | `1234.50` |
| `f` | `FORMAT(1234.5, "f")` | `1234` |
| `0%` | `FORMAT(0.85, "0%")` | `85%` |
| `C2[en]` | `FORMAT(1234.5, "C2[en]")` | `$1,234.50` |
| `C2[pt]` | `FORMAT(1234.5, "C2[pt]")` | `R$ 1.234,50` |
### Date Patterns
| Code | Meaning | Example |
|------|---------|---------|
| `yyyy` | 4‑digit year | `2024` |
| `yy` | 2‑digit year | `24` |
| `MM` | month (01‑12) | `03` |
| `M` | month (1‑12) | `3` |
| `dd` | day (01‑31) | `05` |
| `d` | day (1‑31) | `5` |
| `HH` | 24‑hour (00‑23) | `14` |
| `hh` | 12‑hour (01‑12) | `02` |
| `mm` | minutes (00‑59) | `05` |
| `ss` | seconds (00‑59) | `09` |
| `tt` | AM/PM | `PM` |
**Example**
```basic
DATE = "2024-03-15 14:30:25"
TALK FORMAT(DATE, "dd/MM/yyyy HH:mm") ' 15/03/2024 14:30
```
### Text Patterns
| Placeholder | Effect |
|-------------|--------|
| `@` | Insert original text |
| `!` | Upper‑case |
| `&` | Lower‑case |
**Example**
```basic
NAME = "Maria"
TALK FORMAT(NAME, "Hello, !") ' Hello, MARIA
```
## Practical Tips
- **Test each pattern** in isolation before combining.
- **Locale codes** (`en`, `pt`, `fr`, …) go inside `C2[…]` for currency.
- **Dates must follow** `YYYY‑MM‑DD HH:MM:SS`; otherwise formatting fails.
- **Combine patterns** by nesting calls:
```basic
TALK FORMAT(FORMAT(VALUE, "C2[en]"), "!") ' $1,234.50 (uppercase not needed here)
```
## Common Pitfalls
- Using a date pattern on a non‑date string → returns the original string.
- Forgetting locale brackets (`C2[en]`) → defaults to system locale.
- Mixing placeholders (`@`, `!`, `&`) in the same pattern – only the last one applies.
Use **FORMAT** whenever you need a clean, user‑friendly output without extra code. It keeps scripts short and readable.

26
docs/src/chapter-05/keyword-get-bot-memory.md Normal file
View file

@ -0,0 +1,26 @@
# GET_BOT_MEMORY Keyword
**Syntax**
```
GET_BOT_MEMORY "key"
```
**Parameters**
- `"key"` – The memory key to retrieve.
**Description**
`GET_BOT_MEMORY` reads a value stored in the bot’s persistent memory table (`bot_memories`). It returns the stored string or an empty string if the key does not exist.
**Example (from `auth.bas`)**
```basic
SET attempts = GET_BOT_MEMORY "login_attempts"
IF attempts = "" THEN
SET attempts = "0"
ENDIF
```
The script fetches the number of previous login attempts, defaulting to zero if the key is missing.

44
docs/src/chapter-05/keyword-get.md Normal file
View file

@ -0,0 +1,44 @@
# GET Keyword
**Syntax**
```
GET "url-or-path"
```
**Parameters**
- `"url-or-path"` – Either an HTTP/HTTPS URL (e.g., `https://api.example.com/data`) or a relative path to an object stored in the configured MinIO bucket.
**Description**
`GET` fetches the content from the specified location.
- If the argument starts with `http://` or `https://`, the keyword performs an HTTP GET request using a timeout‑protected `reqwest` client. The response must have a successful status code; otherwise a runtime error is raised.
- If the argument does not start with a scheme, it is treated as a path inside the bot’s MinIO bucket. The keyword reads the object, automatically handling PDF extraction when the file ends with `.pdf`. The content is returned as a UTF‑8 string.
The fetched content can be stored in a variable or passed to other keywords such as `TALK` or `FORMAT`.
**Example (HTTP)**
```basic
SET data = GET "https://api.example.com/users"
TALK "Received data: " + data
```
**Example (Bucket file)**
```basic
SET report = GET "reports/summary.txt"
TALK "Report content:\n" + report
```
**Security**
The implementation validates the path to prevent directory traversal (`..`) and other unsafe patterns. Invalid or unsafe paths cause a runtime error.
**Implementation Notes**
- The request runs in a separate thread with its own Tokio runtime to avoid blocking the main engine.
- Network timeouts are set to 30 seconds; connection timeouts to 10 seconds.
- For bucket access, the keyword ensures the bucket exists before attempting to read.

26
docs/src/chapter-05/keyword-hear.md Normal file
View file

@ -0,0 +1,26 @@
# HEAR Keyword
**Syntax**
```
HEAR variable_name
```
**Parameters**
- `variable_name` – Identifier where the user’s next message will be stored.
**Description**
`HEAR` pauses script execution and waits for the next user input. The received text is assigned to the specified variable, which can then be used in subsequent commands.
**Example (from `start.bas`)**
```basic
HEAR user_input
IF user_input = "help" THEN
TALK "Sure, I can assist with account info, orders, or support."
ENDIF
```
The script waits for the user to type a message, stores it in `user_input`, and then evaluates the condition.

30
docs/src/chapter-05/keyword-last.md Normal file
View file

@ -0,0 +1,30 @@
# LAST Keyword
**Syntax**
```
LAST "text"
```
**Parameters**
- `"text"` – A string expression from which the last word will be extracted.
**Description**
`LAST` returns the final whitespace‑separated token of the provided string. If the string is empty or contains only whitespace, the result is an empty string. This keyword is useful for retrieving the trailing part of a user’s input or any delimited text.
**Example**
```basic
SET command = LAST user_input
TALK "You entered the last word: " + command
```
If `user_input` is `"search books about Rust"`, `LAST` returns `"Rust"`.
**Implementation Notes**
- The keyword splits the string on any whitespace (spaces, tabs, newlines) and returns the last element.
- It does not modify the original string.
- Case‑insensitive; the returned word preserves the original casing.

44
docs/src/chapter-05/keyword-list-tools.md Normal file
View file

@ -0,0 +1,44 @@
# LIST_TOOLS Keyword
**Syntax**
```
LIST_TOOLS
```
**Parameters**
_None_ – This keyword takes no arguments.
**Description**
`LIST_TOOLS` returns a formatted string that lists all tools currently associated with the active conversation session. The list includes each tool’s name and its order of addition. If no tools are active, the keyword returns a message indicating that the tool set is empty.
**Example**
```basic
ADD_TOOL "enrollment.bas"
ADD_TOOL "weather.bas"
SET tools = LIST_TOOLS
TALK tools
```
Possible output:
```
Active tools in this conversation (2):
1. enrollment
2. weather
```
If no tools have been added:
```
No tools are currently active in this conversation.
```
**Implementation Notes**
- The keyword queries the `session_tool_associations` table for the current session ID.
- The result is a plain text string; it can be directly passed to `TALK` or stored in a variable.
- Errors during database access are logged and result in a runtime error.

34
docs/src/chapter-05/keyword-llm.md Normal file
View file

@ -0,0 +1,34 @@
# LLM Keyword
**Syntax**
```
LLM "prompt text"
```
**Parameters**
- `"prompt text"` – The text that will be sent to the configured Large Language Model (LLM) provider.
**Description**
`LLM` forwards the supplied prompt to the LLM service defined in the application configuration (`.gbot/config.csv`). The LLM processes the prompt and returns a response string, which is then made available to the script as the result of the keyword.
The keyword runs the LLM call in a background thread with its own Tokio runtime to avoid blocking the main engine. If the LLM provider returns an error or times out, a runtime error is raised.
**Example**
```basic
SET topic = "GeneralBots platform"
TALK "Generating summary for " + topic + "..."
SET summary = LLM "Summarize the following: " + topic
TALK summary
```
The script asks the LLM to summarize the topic and then outputs the generated summary.
**Implementation Notes**
- The prompt is wrapped in a standard instruction that tells the model to act as a BASIC keyword assistant.
- The keyword returns the raw response text; any formatting must be handled by the script (e.g., using `FORMAT`).
- Network errors, timeouts, or provider failures result in a runtime error.

42
docs/src/chapter-05/keyword-on.md Normal file
View file

@ -0,0 +1,42 @@
# ON Keyword
**Syntax**
```
ON trigger-type OF "table-name"
```
**Parameters**
- `trigger-type` – The type of database trigger to listen for. Valid values are:
- `INSERT`
- `UPDATE`
- `DELETE`
- `"table-name"` – The name of the database table to monitor.
**Description**
`ON` registers a database trigger for the current session. When the specified event occurs on the given table, the engine records the trigger in the `system_automations` table, linking it to the session. This enables scripts to react to data changes by executing associated actions (e.g., sending a notification, updating a variable).
The keyword performs the following steps:
1. Validates the `trigger-type` and converts it to the internal `TriggerKind` enum.
2. Constructs a parameter name in the form `<table>_<trigger>.rhai` (e.g., `orders_insert.rhai`).
3. Inserts a row into `system_automations` with the trigger kind, target table, and parameter name.
4. Returns the number of rows affected (normally `1` on success).
If the trigger type is invalid, the keyword raises a runtime error.
**Example**
```basic
ON INSERT OF "orders"
TALK "A new order was added. Processing..."
```
After execution, any new row inserted into the `orders` table will cause the session to be notified, allowing the script to handle the event.
**Implementation Notes**
- The keyword runs synchronously but performs the database insertion on a separate thread to avoid blocking.
- Errors during insertion are logged and returned as runtime errors.

30
docs/src/chapter-05/keyword-print.md Normal file
View file

@ -0,0 +1,30 @@
# PRINT Keyword
**Syntax**
```
PRINT $expression
```
**Parameters**
- `$expression` – Any valid BASIC expression whose evaluated value will be printed to the server log.
**Description**
`PRINT` evaluates the given expression and writes its string representation to the application log (using the `log::info!` macro). It does not send any output back to the user; it is primarily a debugging aid for developers to inspect variable values or intermediate results during script execution.
**Example**
```basic
SET total = 42
PRINT total ; logs "42"
PRINT "User: " + user_name ; logs "User: Alice"
```
When the script runs, the values are written to the server’s log file and can be viewed in the console or log aggregation system.
**Implementation Notes**
- The keyword always returns `UNIT` (no value) to the script.
- It runs synchronously and does not affect the dialog flow.

37
docs/src/chapter-05/keyword-remove-tool.md Normal file
View file

@ -0,0 +1,37 @@
# REMOVE_TOOL Keyword
**Syntax**
```
REMOVE_TOOL "tool-path.bas"
```
**Parameters**
- `"tool-path.bas"` – Relative path to a `.bas` file that was previously added with `ADD_TOOL`.
**Description**
`REMOVE_TOOL` disassociates a previously added tool from the current conversation session. After execution, the tool’s keywords are no longer available for invocation in the same dialog.
The keyword performs the following steps:
1. Extracts the tool name from the provided path (removing the `.bas` extension and any leading `.gbdialog/` prefix).
2. Validates that the tool name is not empty.
3. Spawns an asynchronous task that:
- Deletes the corresponding row from `session_tool_associations` for the current session.
- Returns a message indicating whether the tool was removed or was not active.
**Example**
```basic
REMOVE_TOOL "enrollment.bas"
TALK "Enrollment tool removed from this conversation."
```
If the `enrollment.bas` tool was active, it will be removed; otherwise the keyword reports that the tool was not active.
**Implementation Notes**
- The operation runs in a separate thread with its own Tokio runtime to avoid blocking the main engine.
- Errors during database deletion are logged and propagated as runtime errors.

31
docs/src/chapter-05/keyword-set-bot-memory.md Normal file
View file

@ -0,0 +1,31 @@
# SET_BOT_MEMORY Keyword
**Syntax**
```
SET_BOT_MEMORY "key", "value"
```
**Parameters**
- `"key"` – Identifier for the memory entry to store.
- `"value"` – The string value to associate with the key.
**Description**
`SET_BOT_MEMORY` stores a key‑value pair in the persistent bot memory table (`bot_memories`). The entry is scoped to the bot instance, not to a specific user. If the key already exists, its value is updated; otherwise a new row is inserted. The operation is performed asynchronously in a background task, so the keyword returns immediately.
**Example**
```basic
SET_BOT_MEMORY "last_greeting", "Hello, world!"
TALK "Bot memory updated."
```
After execution, the key `last_greeting` will contain the value `"Hello, world!"` and can be retrieved later with `GET_BOT_MEMORY`.
**Implementation Notes**
- The keyword spawns a Tokio task that acquires a database connection, checks for an existing entry, and either updates or inserts the record.
- Errors are logged but do not interrupt script execution; the keyword always returns `UNIT`.
- Values are stored as plain strings; binary data should be encoded (e.g., Base64) before storage.

24
docs/src/chapter-05/keyword-set-context.md Normal file
View file

@ -0,0 +1,24 @@
# SET_CONTEXT Keyword
**Syntax**
```
SET_CONTEXT "context-string"
```
**Parameters**
- `"context-string"` – Arbitrary text that will be stored as the session’s context.
**Description**
`SET_CONTEXT` saves a string in the session’s Redis cache (if configured). It can be retrieved later with `GET_CONTEXT` (not a separate keyword; the engine reads the stored value automatically). This is useful for persisting short pieces of information across multiple dialog turns.
**Example**
```basic
SET_CONTEXT "order_id=12345"
TALK "Your order ID has been saved."
```
Later in the script you could retrieve it via a custom function or by accessing the Redis key directly.

32
docs/src/chapter-05/keyword-set-kb.md Normal file
View file

@ -0,0 +1,32 @@
# SET_KB Keyword
**Syntax**
```
SET_KB "kb-name"
```
**Parameters**
- `"kb-name"` – Identifier for a knowledge‑base collection to be associated with the current user.
**Description**
`SET_KB` registers a knowledge‑base (KB) with the user’s session. The keyword validates that the name contains only alphanumeric characters, underscores, or hyphens. It then creates (or ensures the existence of) a vector‑DB collection for the KB and links it to the user in the `user_kb_associations` table. After execution, the KB becomes part of the user’s active knowledge sources and can be queried by `FIND` or used by LLM prompts.
If the KB already exists for the user, the keyword simply confirms the association.
**Example**
```basic
SET_KB "company-policies"
TALK "Knowledge base 'company-policies' is now active."
```
After the command, the `company-policies` collection is available for searches within the current conversation.
**Implementation Notes**
- The operation runs asynchronously in a background thread.
- Errors are logged and returned as runtime errors.
- The keyword always returns `UNIT`.

1
docs/src/chapter-05/keyword-set-schedule.md Normal file
View file

@ -0,0 +1 @@
# SET_SCHEDULE

24
docs/src/chapter-05/keyword-set-user.md Normal file
View file

@ -0,0 +1,24 @@
# SET_USER Keyword
**Syntax**
```
SET_USER "user-id"
```
**Parameters**
- `"user-id"` – UUID string identifying the user.
**Description**
`SET_USER` updates the current session’s user identifier. This is useful when a dialog authenticates a user and wants to associate subsequent interactions with that user’s record.
**Example (from `auth.bas`)**
```basic
SET_USER "550e8400-e29b-41d4-a716-446655440000"
TALK "User authenticated."
```
After execution, all future bot messages in this session are linked to the specified user ID.

1
docs/src/chapter-05/keyword-set.md Normal file
View file

@ -0,0 +1 @@
# SET

54
docs/src/chapter-05/keyword-talk.md Normal file
View file

@ -0,0 +1,54 @@
# TALK Keyword
Sends a message to the user through the current channel.
## Syntax
```
TALK message
```
## Parameters
- `message` - The text to send to the user (string expression)
## Description
The `TALK` keyword outputs a message to the user through whatever channel the conversation is happening on (web, voice, WhatsApp, etc.). This is the primary way for the bot to communicate with users.
## Examples
### Basic Usage
```basic
TALK "Hello! Welcome to our service."
```
### With Variables
```basic
SET user_name = "John"
TALK "Hello, " + user_name + "! How can I help you today?"
```
### Multi-line Messages
```basic
TALK "Here are your options:" + CHR(10) + "1. Check balance" + CHR(10) + "2. Make payment"
```
## Usage Notes
- Messages are sent immediately when the TALK command executes
- Multiple TALK commands in sequence will send multiple messages
- The message content can include variables and expressions
- Special characters and emoji are supported
- Message length may be limited by the channel (e.g., SMS character limits)
## Channel Behavior
Different channels may handle TALK messages differently:
- **Web**: Messages appear in the chat interface
- **Voice**: Text is converted to speech
- **WhatsApp**: Sent as text messages
- **Email**: Added to email conversation thread
## Related Keywords
- `HEAR` - Receive user input
- `WAIT` - Pause before sending
- `FORMAT` - Format message content

34
docs/src/chapter-05/keyword-wait.md Normal file
View file

@ -0,0 +1,34 @@
# WAIT Keyword
**Syntax**
```
WAIT seconds
```
**Parameters**
- `seconds` – Number of seconds to pause execution. Can be an integer or floating‑point value.
**Description**
`WAIT` suspends the script for the specified duration. The keyword validates that the argument is a non‑negative number, caps the wait time at 300 seconds (5 minutes) to prevent excessively long pauses, and then sleeps the current thread for the requested period.
During the wait, the engine does not process other commands; the dialog is effectively paused.
**Example**
```basic
TALK "Processing your request..."
WAIT 2
TALK "Done."
```
The script will wait two seconds between the two `TALK` statements.
**Implementation Notes**
- The keyword uses `std::thread::sleep` with a `Duration` derived from the provided seconds.
- Negative values result in a runtime error.
- The maximum allowed wait time is 300 seconds; values above this are truncated to the limit.
- The keyword returns a string indicating the actual wait time (e.g., `"Waited 2 seconds"`).

32
docs/src/chapter-05/keyword-website-of.md Normal file
View file

@ -0,0 +1,32 @@
# WEBSITE_OF Keyword
**Syntax**
```
WEBSITE_OF "search-term"
```
**Parameters**
- `"search-term"` – The term to search for using a headless browser (e.g., a query string).
**Description**
`WEBSITE_OF` performs a web search for the given term using a headless Chromium instance (via the `headless_chrome` crate). It navigates to DuckDuckGo, enters the search term, and extracts the first non‑advertisement result URL. The keyword returns the URL as a string, which can then be used with `ADD_WEBSITE` or other keywords.
**Example**
```basic
SET url = WEBSITE_OF "GeneralBots documentation"
ADD_WEBSITE url
TALK "Added the top result as a knowledge source."
```
The script searches for “GeneralBots documentation”, retrieves the first result URL, adds it as a website KB, and notifies the user.
**Implementation Notes**
- The keyword runs the browser actions in a separate thread with its own Tokio runtime.
- If no results are found, the keyword returns the string `"No results found"`.
- Errors during navigation or extraction are logged and cause a runtime error.
- The search is performed on DuckDuckGo to avoid reliance on proprietary APIs.

42
docs/src/chapter-05/keywords.md Normal file
View file

@ -0,0 +1,42 @@
# Keyword Reference
This section lists every BASIC keyword implemented in the GeneralBots engine. Each keyword page includes:
* **Syntax** – Exact command format.
* **Parameters** – Expected arguments.
* **Description** – What the keyword does.
* **Example** – A short snippet from the official template files.
The source code for each keyword lives in `src/basic/keywords/`. Only the keywords listed here exist in the system.
## Keywords
- [TALK](./keyword-talk.md)
- [HEAR](./keyword-hear.md)
- [SET_USER](./keyword-set-user.md)
- [SET_CONTEXT](./keyword-set-context.md)
- [LLM](./keyword-llm.md)
- [GET_BOT_MEMORY](./keyword-get-bot-memory.md)
- [SET_BOT_MEMORY](./keyword-set-bot-memory.md)
- [SET_KB](./keyword-set-kb.md)
- [ADD_KB](./keyword-add-kb.md)
- [ADD_WEBSITE](./keyword-add-website.md)
- [ADD_TOOL](./keyword-add-tool.md)
- [LIST_TOOLS](./keyword-list-tools.md)
- [REMOVE_TOOL](./keyword-remove-tool.md)
- [CLEAR_TOOLS](./keyword-clear-tools.md)
- [GET](./keyword-get.md)
- [FIND](./keyword-find.md)
- [SET](./keyword-set.md)
- [ON](./keyword-on.md)
- [SET_SCHEDULE](./keyword-set-schedule.md)
- [CREATE_SITE](./keyword-create-site.md)
- [CREATE_DRAFT](./keyword-create-draft.md)
- [WEBSITE_OF](./keyword-website-of.md)
- [PRINT](./keyword-print.md)
- [WAIT](./keyword-wait.md)
- [FORMAT](./keyword-format.md)
- [FIRST](./keyword-first.md)
- [LAST](./keyword-last.md)
- [FOR EACH](./keyword-for-each.md)
- [EXIT FOR](./keyword-exit-for.md)

21
docs/src/chapter-05/template-auth.md Normal file
View file

@ -0,0 +1,21 @@
# auth.bas
```basic
REM Simple authentication flow
SET attempts = 0
LABEL auth_loop
HEAR password
IF password = "secret123" THEN
TALK "Authentication successful."
ELSE
SET attempts = attempts + 1
IF attempts >= 3 THEN
TALK "Too many attempts. Goodbye."
EXIT
ENDIF
TALK "Incorrect password. Try again."
GOTO auth_loop
ENDIF
```
This template demonstrates a basic password check with a limited number of attempts. It uses the `HEAR`, `TALK`, `SET`, `IF`, `ELSE`, `GOTO`, and `EXIT` keywords to manage the dialog flow.

97
docs/src/chapter-05/template-enrollment.md Normal file
View file

@ -0,0 +1,97 @@
# Enrollment Tool Example
This example shows a complete enrollment tool with parameter definitions and data saving.
## Complete Enrollment Script
```basic
PARAM name AS string LIKE "Abreu Silva"
DESCRIPTION "Required full name of the individual."
PARAM birthday AS date LIKE "23/09/2001"
DESCRIPTION "Required birth date of the individual in DD/MM/YYYY format."
PARAM email AS string LIKE "abreu.silva@example.com"
DESCRIPTION "Required email address for contact purposes."
PARAM personalid AS integer LIKE "12345678900"
DESCRIPTION "Required Personal ID number of the individual (only numbers)."
PARAM address AS string LIKE "Rua das Flores, 123 - SP"
DESCRIPTION "Required full address of the individual."
DESCRIPTION "This is the enrollment process, called when the user wants to enrol. Once all information is collected, confirm the details and inform them that their enrollment request has been successfully submitted. Provide a polite and professional tone throughout the interaction."
REM Enrollment Process
TALK "Welcome to the enrollment process! Let's get you registered."
TALK "First, what is your full name?"
HEAR name
TALK "Thank you. What is your birth date? (DD/MM/YYYY)"
HEAR birthday
TALK "What is your email address?"
HEAR email
TALK "Please provide your Personal ID number (numbers only):"
HEAR personalid
TALK "Finally, what is your full address?"
HEAR address
REM Validate and confirm
TALK "Please confirm your details:"
TALK "Name: " + name
TALK "Birth Date: " + birthday
TALK "Email: " + email
TALK "Personal ID: " + personalid
TALK "Address: " + address
TALK "Are these details correct? (yes/no)"
HEAR confirmation
IF confirmation = "yes" THEN
REM Save to CSV file
SAVE "enrollments.csv", name, birthday, email, personalid, address
TALK "Thank you! Your enrollment has been successfully submitted. You will receive a confirmation email shortly."
ELSE
TALK "Let's start over with the correct information."
REM In a real implementation, you might loop back or use a different approach
END IF
```
## Tool Parameters
This tool defines 5 parameters with specific types and validation:
1. **name** (string): Full name with example format
2. **birthday** (date): Birth date in DD/MM/YYYY format
3. **email** (string): Email address for contact
4. **personalid** (integer): Numeric personal ID
5. **address** (string): Complete physical address
## Data Storage
The `SAVE` command writes the collected data to a CSV file:
- Creates "enrollments.csv" if it doesn't exist
- Appends new records with all fields
- Maintains data consistency across sessions
## Usage Flow
1. User initiates enrollment process
2. Bot collects each piece of information sequentially
3. User confirms accuracy of entered data
4. Data is saved to persistent storage
5. Confirmation message is sent
## Error Handling
The script includes:
- Input validation through parameter types
- Confirmation step to prevent errors
- Clear user prompts with format examples
- Graceful handling of correction requests
This example demonstrates a complete, production-ready tool implementation using the BASIC scripting language.

1
docs/src/chapter-05/template-start.md Normal file
View file

@ -0,0 +1 @@
# start.bas

1
docs/src/chapter-05/template-summary.md Normal file
View file

@ -0,0 +1 @@
# generate-summary.bas

57
docs/src/chapter-05/templates.md Normal file
View file

@ -0,0 +1,57 @@
# Template Examples
The `templates` section showcases the official BASIC dialog templates shipped with GeneralBots. They are stored under `templates/` and can be referenced directly in dialogs via `ADD_TOOL`.
## start.bas
```basic
REM Basic greeting and help flow
SET user_name = "Guest"
TALK "Hello, " + user_name + "! How can I help you today?"
HEAR user_input
IF user_input = "help" THEN
TALK "Sure, I can assist with account info, orders, or support."
ELSE
TALK "Sorry, I didn't understand."
ENDIF
```
## auth.bas
```basic
REM Simple authentication flow
SET attempts = 0
LABEL auth_loop
HEAR password
IF password = "secret123" THEN
TALK "Authentication successful."
ELSE
SET attempts = attempts + 1
IF attempts >= 3 THEN
TALK "Too many attempts. Goodbye."
EXIT
ENDIF
TALK "Incorrect password. Try again."
GOTO auth_loop
ENDIF
```
## generate-summary.bas
```basic
REM Generates a summary using the LLM keyword
SET topic = "GeneralBots platform"
TALK "Generating summary for " + topic + "..."
SET summary = LLM "Summarize the following: " + topic
TALK summary
```
## enrollment.bas (Tool Example)
```basic
REM Demonstrates adding a custom tool to the conversation
ADD_TOOL "enrollment.bas"
TALK "Enrollment tool added. You can now use ENROLL command."
```
These templates illustrate common patterns: greeting, authentication, LLM integration, and tool registration. Users can copy and adapt them for their own bots.

31
docs/src/chapter-05/templates/auth.md Normal file
View file

@ -0,0 +1,31 @@
# auth.bas (Template)
This template demonstrates a simple authentication flow using the BASIC dialog language.
```basic
REM Simple authentication flow
SET attempts = 0
LABEL auth_loop
HEAR password
IF password = "secret123" THEN
TALK "Authentication successful."
ELSE
SET attempts = attempts + 1
IF attempts >= 3 THEN
TALK "Too many attempts. Goodbye."
EXIT
ENDIF
TALK "Incorrect password. Try again."
GOTO auth_loop
ENDIF
```
**Purpose**
- Shows how to collect a password with `HEAR`.
- Limits the number of attempts to three.
- Uses `TALK` to give feedback and `EXIT` to end the dialog after too many failures.
**Keywords used:** `SET`, `HEAR`, `IF`, `ELSE`, `GOTO`, `EXIT`, `TALK`.
---

74
docs/src/chapter-05/templates/enrollment.md Normal file
View file

@ -0,0 +1,74 @@
# enrollment.bas (Template)
A comprehensive enrollment dialog that gathers user information, confirms it, and saves it to a CSV file.
```basic
REM Enrollment Tool Example
## Complete Enrollment Script
PARAM name AS string LIKE "Abreu Silva"
DESCRIPTION "Required full name of the individual."
PARAM birthday AS date LIKE "23/09/2001"
DESCRIPTION "Required birth date of the individual in DD/MM/YYYY format."
PARAM email AS string LIKE "abreu.silva@example.com"
DESCRIPTION "Required email address for contact purposes."
PARAM personalid AS integer LIKE "12345678900"
DESCRIPTION "Required Personal ID number of the individual (only numbers)."
PARAM address AS string LIKE "Rua das Flores, 123 - SP"
DESCRIPTION "Required full address of the individual."
DESCRIPTION "This is the enrollment process, called when the user wants to enrol. Once all information is collected, confirm the details and inform them that their enrollment request has been successfully submitted. Provide a polite and professional tone throughout the interaction."
REM Enrollment Process
TALK "Welcome to the enrollment process! Let's get you registered."
TALK "First, what is your full name?"
HEAR name
TALK "Thank you. What is your birth date? (DD/MM/YYYY)"
HEAR birthday
TALK "What is your email address?"
HEAR email
TALK "Please provide your Personal ID number (numbers only):"
HEAR personalid
TALK "Finally, what is your full address?"
HEAR address
REM Validate and confirm
TALK "Please confirm your details:"
TALK "Name: " + name
TALK "Birth Date: " + birthday
TALK "Email: " + email
TALK "Personal ID: " + personalid
TALK "Address: " + address
TALK "Are these details correct? (yes/no)"
HEAR confirmation
IF confirmation = "yes" THEN
REM Save to CSV file
SAVE "enrollments.csv", name, birthday, email, personalid, address
TALK "Thank you! Your enrollment has been successfully submitted. You will receive a confirmation email shortly."
ELSE
TALK "Let's start over with the correct information."
REM In a real implementation, you might loop back or use a different approach
END IF
```
**Purpose**
- Shows how to define parameters with `PARAM` and `DESCRIPTION`.
- Demonstrates a multi‑step data collection flow using `HEAR` and `TALK`.
- Confirms data before persisting it via `SAVE`.
**Keywords used:** `PARAM`, `DESCRIPTION`, `HEAR`, `TALK`, `IF`, `ELSE`, `SAVE`.
---

25
docs/src/chapter-05/templates/start.md Normal file
View file

@ -0,0 +1,25 @@
# start.bas (Template)
A minimal greeting and help flow to get users started.
```basic
REM Basic greeting and help flow
SET user_name = "Guest"
TALK "Hello, " + user_name + "! How can I help you today?"
HEAR user_input
IF user_input = "help" THEN
TALK "Sure, I can assist with account info, orders, or support."
ELSE
TALK "Sorry, I didn't understand."
ENDIF
```
**Purpose**
- Shows how to set a variable with `SET`.
- Uses `TALK` to send a message and `HEAR` to receive user input.
- Demonstrates simple branching with `IF/ELSE`.
**Keywords used:** `SET`, `TALK`, `HEAR`, `IF`, `ELSE`.
---

1
docs/src/chapter-06/README.md Normal file
View file

@ -0,0 +1 @@
# Chapter 06: gbapp Reference

1
docs/src/chapter-06/architecture.md Normal file
View file

@ -0,0 +1 @@
# Rust Architecture

1
docs/src/chapter-06/building.md Normal file
View file

@ -0,0 +1 @@
# Building from Source

1
docs/src/chapter-06/crates.md Normal file
View file

@ -0,0 +1 @@
# Crate Structure

1
docs/src/chapter-06/custom-keywords.md Normal file
View file

@ -0,0 +1 @@
# Creating Custom Keywords

1
docs/src/chapter-06/dependencies.md Normal file
View file

@ -0,0 +1 @@
# Adding Dependencies

Some files were not shown because too many files have changed in this diff Show more