GeneralBots/gb
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gb/README.md

24 KiB
Raw Blame History

General Bots Workspace

⚠️ CRITICAL SECURITY WARNING

NEVER CREATE FILES WITH SECRETS IN THE REPOSITORY ROOT

Secret files MUST be placed in /tmp/ only:

  • ✅ /tmp/vault-token-gb - Vault root token
  • ✅ /tmp/vault-unseal-key-gb - Vault unseal key
  • ❌ vault-unseal-keys - FORBIDDEN (tracked by git)
  • ❌ start-and-unseal.sh - FORBIDDEN (contains secrets)

Files added to .gitignore: vault-unseal-keys, start-and-unseal.sh, vault-token-*

Why /tmp/?

  • Cleared on reboot (ephemeral)
  • Not tracked by git
  • Standard Unix security practice
  • Prevents accidental commits

Version: 6.2.0
Type: Rust Workspace (Monorepo with Independent Subproject Repos)

Overview

General Bots is a comprehensive automation platform built with Rust, providing a unified workspace for building AI-powered bots, web interfaces, desktop applications, and integration tools. The workspace follows a modular architecture with independent subprojects that can be developed and deployed separately while sharing common libraries and standards.

For comprehensive documentation, see docs.pragmatismo.com.br or the BotBook for detailed guides, API references, and tutorials.

📁 Workspace Structure

Crate Purpose Port Tech Stack
botserver Main API server, business logic 9000 Axum, Diesel, Rhai BASIC
botui Web UI server (dev) + proxy 3000 Axum, HTML/HTMX/CSS
botapp Desktop app wrapper - Tauri 2
botlib Shared library - Core types, errors
botbook Documentation - mdBook
bottest Integration tests - tokio-test
botdevice IoT/Device support - Rust
botmodels Data models visualization - -
botplugin Browser extension - JS

Key Paths

  • Binary: target/debug/botserver
  • Run from: botserver/ directory
  • Env file: botserver/.env
  • Stack: botserver-stack/
  • UI Files: botui/ui/suite/
  • Local Bot Data: /opt/gbo/data/ (place .gbai packages here)

Local Bot Data Directory

Place local bot packages in /opt/gbo/data/ for automatic loading and monitoring:

Directory Structure:

/opt/gbo/data/
└── mybot.gbai/
    ├── mybot.gbdialog/
    │   ├── start.bas
    │   └── main.bas
    └── mybot.gbot/
        └── config.csv

Features:

  • Auto-loading: Bots automatically mounted on server startup
  • Auto-compilation: .bas files compiled to .ast on change
  • Auto-creation: New bots automatically added to database
  • Hot-reload: Changes trigger immediate recompilation
  • Monitored by: LocalFileMonitor and ConfigWatcher services

Usage:

  1. Create bot directory structure in /opt/gbo/data/
  2. Add .bas files to <bot_name>.gbai/<bot_name>.gbdialog/
  3. Server automatically detects and loads the bot
  4. Optional: Add config.csv for bot configuration

🏗️ BotServer Component Architecture

🔧 Infrastructure Components (Auto-Managed)

BotServer automatically installs, configures, and manages all infrastructure components on first run. DO NOT manually start these services - BotServer handles everything.

Automatic Service Lifecycle:

  1. Start: When botserver starts, it automatically launches all infrastructure components (PostgreSQL, Vault, MinIO, Valkey, Qdrant, etc.)
  2. Credentials: BotServer retrieves all service credentials (passwords, tokens, API keys) from Vault
  3. Connection: BotServer uses these credentials to establish secure connections to each service
  4. Query: All database queries, cache operations, and storage requests are authenticated using Vault-managed credentials

Credential Flow:

botserver starts
    ↓
Launch PostgreSQL, MinIO, Valkey, Qdrant
    ↓
Connect to Vault
    ↓
Retrieve service credentials (from database)
    ↓
Authenticate with each service using retrieved credentials
    ↓
Ready to handle requests
Component Purpose Port Binary Location Credentials From
Vault Secrets management 8200 botserver-stack/bin/vault/vault Auto-unsealed
PostgreSQL Primary database 5432 botserver-stack/bin/tables/bin/postgres Vault → database
MinIO Object storage (S3-compatible) 9000/9001 botserver-stack/bin/drive/minio Vault → database
Zitadel Identity/Authentication 8300 botserver-stack/bin/directory/zitadel Vault → database
Qdrant Vector database (embeddings) 6333 botserver-stack/bin/vector_db/qdrant Vault → database
Valkey Cache/Queue (Redis-compatible) 6379 botserver-stack/bin/cache/valkey-server Vault → database
Llama.cpp Local LLM server 8081 botserver-stack/bin/llm/build/bin/llama-server Vault → database

📦 Component Installation System

Components are defined in botserver/3rdparty.toml and managed by the PackageManager (botserver/src/core/package_manager/):

[components.cache]
name = "Valkey Cache (Redis-compatible)"
url = "https://github.com/valkey-io/valkey/archive/refs/tags/8.0.2.tar.gz"
filename = "valkey-8.0.2.tar.gz"

[components.llm]
name = "Llama.cpp Server"
url = "https://github.com/ggml-org/llama.cpp/releases/download/b7345/llama-b7345-bin-ubuntu-x64.zip"
filename = "llama-b7345-bin-ubuntu-x64.zip"

Installation Flow:

  1. Download: Components downloaded to botserver-installers/ (cached)
  2. Extract/Build: Binaries placed in botserver-stack/bin/<component>/
  3. Configure: Config files generated in botserver-stack/conf/<component>/
  4. Start: Components started with proper TLS certificates
  5. Monitor: Components monitored and auto-restarted if needed

Bootstrap Process:

  • First run: Full bootstrap (downloads, installs, configures all components)
  • Subsequent runs: Only starts existing components (uses cached binaries)
  • Config stored in: botserver-stack/conf/system/bootstrap.json

🚀 PROPER STARTUP PROCEDURES

❌ FORBIDDEN:

  • NEVER manually start infrastructure components (Vault, PostgreSQL, MinIO, etc.)
  • NEVER run cargo run or cargo build for botserver directly without ./restart.sh
  • NEVER modify botserver-stack/ files manually (use botserver API)

✅ REQUIRED:

Option 1: Development (Recommended)

./restart.sh

This script:

  1. Kills existing processes cleanly
  2. Builds botserver and botui sequentially (no race conditions)
  3. Starts botserver in background with logging to botserver.log
  4. Starts botui in background with logging to botui.log
  5. Shows process IDs and access URLs

Option 2: Production/Release

# Build release binary first
cargo build --release -p botserver

# Start with release binary
RUST_LOG=info ./target/release/botserver --noconsole 2>&1 | tee botserver.log &

Option 3: Using Exec (Systemd/Supervisord)

# In systemd service or similar
ExecStart=/home/rodriguez/src/gb/target/release/botserver --noconsole

🔒 Component Communication

All components communicate through internal networks with mTLS:

  • Vault: mTLS for secrets access
  • PostgreSQL: TLS encrypted connections
  • MinIO: TLS with client certificates
  • Zitadel: mTLS for user authentication

Certificates auto-generated in: botserver-stack/conf/system/certificates/

📊 Component Status

Check component status anytime:

# Check if all components are running
ps aux | grep -E "vault|postgres|minio|zitadel|qdrant|valkey" | grep -v grep

# View component logs
tail -f botserver-stack/logs/vault/vault.log
tail -f botserver-stack/logs/tables/postgres.log
tail -f botserver-stack/logs/drive/minio.log

# Test component connectivity
cd botserver-stack/bin/vault && ./vault status
cd botserver-stack/bin/cache && ./valkey-cli ping

🏗️ Component Dependency Graph

┌─────────────────────────────────────────────────────────────────┐
│                         Client Layer                            │
├─────────────────────────────────────────────────────────────────┤
│  botui (Web UI)    │  botapp (Desktop)   │  botplugin (Ext)   │
│  HTMX + Axum       │  Tauri 2 Wrapper    │  Browser Extension  │
└─────────┬───────────────────┬──────────────────┬─────────────────┘
          │                   │                  │
          └───────────────────┼──────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │   botlib          │
                    │  (Shared Types)   │
                    └─────────┬─────────┘
                              │
          ┌───────────────────┼───────────────────┐
          │                   │                   │
    ┌─────▼─────┐      ┌─────▼─────┐      ┌─────▼─────┐
    │ botserver │      │ bottest   │      │ botdevice  │
    │ API Core  │      │ Tests     │      │ IoT/Device │
    └───────────┘      └───────────┘      └───────────┘

Dependency Rules

Crate Depends On Why
botserver botlib Shared types, error handling, models
botui botlib Common data structures, API client
botapp botlib Shared types, desktop-specific utilities
bottest botserver, botlib Integration testing with real components
botdevice botlib Device types, communication protocols
botplugin - Standalone browser extension (JS)

Key Principle: botlib contains ONLY shared types and utilities. No business logic. All business logic lives in botserver or specialized crates.

📦 Module Responsibility Matrix

botserver/src/ Module Structure

Module Responsibility Key Types Dependencies
core/bot/ WebSocket handling, bot orchestration BotOrchestrator, UserMessage basic, shared
core/session/ Session management, conversation history SessionManager, UserSession shared, database
basic/ Rhai BASIC scripting engine ScriptService, Engine rhai, keywords
basic/keywords/ BASIC keyword implementations (TALK, HEAR, etc.) Keyword functions basic, state
llm/ Multi-vendor LLM API integration LLMClient, ModelConfig reqwest, shared
drive/ S3 file storage and monitoring DriveMonitor, compile_tool s3, basic
security/ Security guards (command, SQL, error) SafeCommand, ErrorSanitizer state
shared/ Database models, schema definitions Bot, Session, Message diesel
tasks/ AutoTask execution system TaskRunner, TaskScheduler core/basic
auto_task/ LLM-powered app generation AppGenerator, template engine llm, tasks
learn/ Knowledge base management KBManager, vector storage database, drive
attendance/ LLM-assisted customer service AttendantManager, queue core/bot

Data Flow Patterns

1. User Request Flow:
   Client → WebSocket → botserver/src/core/bot/mod.rs
                          ↓
                    BotOrchestrator::stream_response()
                          ↓
              ┌───────────┴───────────┐
              │                       │
         LLM API Call            Script Execution
         (llm/mod.rs)            (basic/mod.rs)
              │                       │
              └───────────┬───────────┘
                          ↓
                    Response → WebSocket → Client

2. File Sync Flow:
   S3 Drive → drive_monitor/src/drive_monitor/mod.rs
                          ↓
                    Download .bas files
                          ↓
              compile_file() → Generate .ast
                          ↓
              Store in ./work/{bot_name}.gbai/

3. Script Execution Flow:
   .bas file → ScriptService::compile()
                    ↓
              preprocess_basic_script()
                    ↓
              engine.compile() → AST
                    ↓
              ScriptService::run() → Execute
                    ↓
              TALK commands → WebSocket messages

Common Architectural Patterns

Pattern Where Used Purpose
State via Arc All handlers Shared async state (DB, cache, config)
Extension(state) extractor Axum handlers Inject Arc into route handlers
tokio::spawn_blocking CPU-intensive tasks Offload blocking work from async runtime
WebSocket with split() Real-time comms Separate sender/receiver for WS streams
ErrorSanitizer for responses All HTTP errors Prevent leaking sensitive info in errors
SafeCommand for execution Command running Whitelist-based command validation
Rhai for scripting BASIC interpreter Embeddable scripting language
Diesel ORM Database access Type-safe SQL queries
Redis for cache Session data Fast key-value storage
S3 for storage File system Scalable object storage

Quick Start

🚀 Simple Startup (ALWAYS USE restart.sh)

./restart.sh

⚠️ CRITICAL: ALWAYS use restart.sh - NEVER start servers individually!

The script handles BOTH servers properly:

  1. Stop existing processes cleanly
  2. Build botserver and botui sequentially (no race conditions)
  3. Start botserver in background → automatically starts all infrastructure services (PostgreSQL, Vault, MinIO, Valkey, Qdrant)
  4. BotServer retrieves credentials from Vault and authenticates with all services
  5. Start botui in background → proxy to botserver
  6. Show process IDs and monitoring commands

Infrastructure services are fully automated - no manual configuration required!

Monitor startup:

tail -f botserver.log botui.log

Access:

📊 Monitor & Debug

tail -f botserver.log botui.log

Quick status check:

ps aux | grep -E "botserver|botui" | grep -v grep

Quick error scan:

grep -E " E |W |CLIENT:" botserver.log | tail -20

🔧 Manual Startup (If needed)

⚠️ WARNING: Only use if restart.sh fails. Always prefer restart.sh!

cd botserver && cargo run -- --noconsole > ../botserver.log 2>&1 &
cd botui && BOTSERVER_URL="http://localhost:9000" cargo run > ../botui.log 2>&1 &

🛑 Stop Servers

pkill -f botserver; pkill -f botui

⚠️ Common Issues

Vault init error? Delete stale state:

rm -rf botserver-stack/data/vault botserver-stack/conf/vault/init.json && ./restart.sh

Port in use? Find and kill:

lsof -ti:9000 | xargs kill -9
lsof -ti:3000 | xargs kill -9

⚠️ IMPORTANT: Stack Services Management All infrastructure services (PostgreSQL, Vault, Redis, Qdrant, MinIO, etc.) are automatically started by botserver and managed through botserver-stack/ directory, NOT global system installations. The system uses:

  • Local binaries: botserver-stack/bin/ (PostgreSQL, Vault, Redis, etc.)
  • Configurations: botserver-stack/conf/
  • Data storage: botserver-stack/data/
  • Service logs: botserver-stack/logs/ (check here for troubleshooting)
  • Credentials: Stored in Vault, retrieved by botserver at startup

Do NOT install or reference global PostgreSQL, Redis, or other services. When botserver starts, it automatically:

  1. Launches all required stack services
  2. Connects to Vault
  3. Retrieves credentials from the bot_configuration database table
  4. Authenticates with each service using retrieved credentials
  5. Begins handling requests with authenticated connections

If you encounter service errors, check the individual service logs in ./botserver-stack/logs/[service]/ directories.

UI File Deployment - Production Options

Option 1: Embedded UI (Recommended for Production)

The embed-ui feature compiles UI files directly into the botui binary, eliminating the need for separate file deployment:

# Build with embedded UI files
cargo build --release -p botui --features embed-ui

# The binary now contains all UI files - no additional deployment needed!
# The botui binary is self-contained and production-ready

Benefits of embed-ui:

  • ✅ Single binary deployment (no separate UI files)
  • ✅ Faster startup (no filesystem access)
  • ✅ Smaller attack surface
  • ✅ Simpler deployment process

Option 2: Filesystem Deployment (Development Only)

For development, UI files are served from the filesystem:

# UI files must exist at botui/ui/suite/
# This is automatically available in development builds

Option 3: Manual File Deployment (Legacy)

If you need to deploy UI files separately (not recommended):

# Deploy UI files to production location
./botserver/deploy/deploy-ui.sh /opt/gbo

# Verify deployment
ls -la /opt/gbo/bin/ui/suite/index.html

See botserver/deploy/README.md for deployment scripts.

Start Both Servers (Automated)

# Use restart script (RECOMMENDED)
./restart.sh

Start Both Servers (Manual)

# Terminal 1: botserver
cd botserver && cargo run -- --noconsole

# Terminal 2: botui  
cd botui && BOTSERVER_URL="http://localhost:9000" cargo run

Build Commands

# Check single crate
cargo check -p botserver

# Build workspace
cargo build

# Run tests
cargo test -p bottest

🤖 AI Agent Guidelines

For LLM instructions, coding rules, security directives, testing workflows, and error handling patterns, see AGENTS.md.

📖 Glossary

Term Definition Usage
Bot AI agent with configuration, scripts, and knowledge bases Primary entity in system
Session Single conversation instance between user and bot Stored in sessions table
Dialog Collection of BASIC scripts (.bas files) for bot logic Stored in {bot_name}.gbdialog/
Tool Reusable function callable by LLM Defined in .bas files, compiled to .ast
Knowledge Base (KB) Vector database of documents for semantic search Managed in learn/ module
Scheduler Time-triggered task execution Cron-like scheduling in BASIC scripts
Drive S3-compatible storage for files Abstracted in drive/ module
Rhai Embedded scripting language for BASIC dialect Rhai engine in basic/ module
WebSocket Adapter Component that sends messages to connected clients web_adapter in state
AutoTask LLM-generated task automation system In auto_task/ and tasks/ modules
Orchestrator Coordinates LLM, tools, KBs, and user input BotOrchestrator in core/bot/

🖥️ UI Architecture (botui + botserver)

Two Servers During Development

Server Port Purpose
botui 3000 Serves UI files + proxies API to botserver
botserver 9000 Backend API + embedded UI fallback

How It Works

Browser → localhost:3000 → botui (serves HTML/CSS/JS)
                        → /api/* proxied to botserver:9000
                        → /suite/* served from botui/ui/suite/

Adding New Suite Apps

  1. Create folder: botui/ui/suite/<appname>/
  2. Add to SUITE_DIRS in botui/src/ui_server/mod.rs
  3. Rebuild botui: cargo build -p botui
  4. Add menu entry in botui/ui/suite/index.html

Hot Reload

  • UI files (HTML/CSS/JS): Edit & refresh browser (no restart)
  • botui Rust code: Rebuild + restart botui
  • botserver Rust code: Rebuild + restart botserver

Production (Single Binary)

When botui/ui/suite/ folder not found, botserver uses embedded UI compiled into binary via rust-embed.

🎨 Frontend Standards

HTMX-First Approach

  • Use HTMX to minimize JavaScript
  • Server returns HTML fragments, not JSON
  • Use hx-get, hx-post, hx-target, hx-swap
  • WebSocket via htmx-ws extension

Local Assets Only - NO CDN

<!-- ✅ CORRECT -->
<script src="js/vendor/htmx.min.js"></script>

<!-- ❌ WRONG -->
<script src="https://unpkg.com/htmx.org@1.9.10"></script>

Vendor Libraries Location

ui/suite/js/vendor/
├── htmx.min.js
├── htmx-ws.js
├── marked.min.js
└── gsap.min.js

📋 Project-Specific Guidelines

Each crate has its own README.md with specific guidelines:

Crate README.md Location Focus
botserver botserver/README.md API, security, Rhai BASIC
botui botui/README.md UI, HTMX, CSS design system
botapp botapp/README.md Tauri, desktop features
botlib botlib/README.md Shared types, errors
botbook botbook/README.md Documentation, mdBook
bottest bottest/README.md Test infrastructure

Special Prompts

File Purpose
botserver/src/tasks/README.md AutoTask LLM executor
botserver/src/auto_task/APP_GENERATOR_PROMPT.md App generation

📚 Documentation

For complete documentation, guides, and API references:

🔧 Immediate Technical Debt

Critical Issues to Address

  1. Error Handling Debt: 955 instances of unwrap()/expect() in production code
  2. Performance Debt: 12,973 excessive clone()/to_string() calls
  3. File Size Debt: 7 files exceed 450 lines (largest: 3220 lines)
  4. Test Coverage: Missing integration tests for critical paths
  5. Documentation: Missing inline documentation for complex algorithms

Weekly Maintenance Tasks

# Check for duplicate dependencies
cargo tree --duplicates

# Remove unused dependencies  
cargo machete

# Check binary size
cargo build --release && ls -lh target/release/botserver

# Performance profiling
cargo bench

# Security audit
cargo audit

Git Structure

Note: Each subproject has its own git repository. This root repository only tracks workspace-level files:

  • Cargo.toml - Workspace configuration
  • README.md - This file
  • .gitignore - Ignore patterns
  • ADDITIONAL-SUGGESTIONS.md - Enhancement ideas
  • TODO-*.md - Task tracking files

Subprojects (botapp, botserver, botui, etc.) are independent repositories referenced as git submodules.

⚠️ CRITICAL: Submodule Push Workflow

When making changes to any submodule (botserver, botui, botlib, etc.):

  1. Commit and push changes within the submodule directory:

    cd botserver
git add .
git commit -m "Your changes"
git push pragmatismo main
git push github main

  2. Update the global gb repository submodule reference:

    cd ..  # Back to gb root
git add botserver
git commit -m "Update botserver submodule to latest commit"
git push pragmatismo main
git push github main

Failure to push the global gb repository will cause submodule changes to not trigger CI/CD pipelines.

Both repositories must be pushed for changes to take effect in production.

Development Workflow

  1. Read this README.md (workspace structure)
  2. Read AGENTS.md (coding rules & workflows)
  3. BEFORE creating any .md file, search botbook/ for existing documentation
  4. Read <project>/README.md (project-specific rules)
  5. Use diagnostics tool to check warnings
  6. Fix all warnings with full file rewrites
  7. Verify with diagnostics after each file
  8. Never suppress warnings with #[allow()]

License

See individual project repositories for license information.