+
# General Bots Documentation
Welcome to the **General Bots** documentation. This guide explains how to install, configure, extend, and deploy conversational AI bots using General Bots' template-based package system and BASIC scripting language.
diff --git a/docs/src/appendix-i/README.md b/docs/src/appendix-i/README.md
index 8f0927005..59373cf32 100644
--- a/docs/src/appendix-i/README.md
+++ b/docs/src/appendix-i/README.md
@@ -1,6 +1,6 @@
## Appendix I – Database Model
-
+
The core database schema for GeneralBots is defined in `src/shared/models.rs`. It uses **Diesel** with PostgreSQL and includes the following primary tables:
@@ -44,3 +44,9 @@ The core database schema for GeneralBots is defined in `src/shared/models.rs`. I
- timestamp: Timestamp
The schema is automatically migrated when the server starts.
+
+---
+
+
+
+
diff --git a/docs/src/appendix-i/schema.md b/docs/src/appendix-i/schema.md
index 798561891..75e7e51ed 100644
--- a/docs/src/appendix-i/schema.md
+++ b/docs/src/appendix-i/schema.md
@@ -14,12 +14,12 @@ The database schema follows these design principles:
## Entity Relationship Overview
-
+
### Core Tables Structure
## Detailed Schema
-
+
## Schema Categories
@@ -57,12 +57,12 @@ The database schema follows these design principles:
### Session Management Flow
### Session Flow
-
+
### Knowledge Base Access Pattern
### Knowledge Base Access
-
+
### Primary Relationships
diff --git a/docs/src/chapter-01/README.md b/docs/src/chapter-01/README.md
index d1a470365..8067b635f 100644
--- a/docs/src/chapter-01/README.md
+++ b/docs/src/chapter-01/README.md
@@ -1,833 +1,44 @@
-# Chapter 01: Run and Talk - System Bootstrap and Initial Configuration
+# Chapter 01: Run and Talk
-Welcome to General Bots - a comprehensive, self-hosted artificial intelligence platform designed for enterprise deployment and individual sovereignty. This chapter provides detailed technical guidance on system initialization, bootstrap processes, and fundamental operational concepts that form the foundation of your AI infrastructure.
+Get General Bots running in 5 minutes.
-## Executive Summary
+## What You'll Learn
-General Bots represents a paradigm shift in conversational AI deployment, offering a fully autonomous, self-contained system that eliminates dependencies on external cloud services while maintaining enterprise-grade capabilities. The platform implements a zero-configuration bootstrap methodology that provisions all required infrastructure components automatically, enabling rapid deployment without specialized DevOps expertise.
+- How to start General Bots
+- Your first conversation
+- How sessions work
+- Basic bot interaction
-### Key Architectural Principles
-
-The system adheres to four fundamental design principles that govern its implementation:
-
-1. **Infrastructure Autonomy**: Complete ownership of the technology stack, from persistent storage layers to natural language processing engines. All components operate within your controlled environment, ensuring data sovereignty and regulatory compliance.
-
-2. **Deterministic Bootstrap**: The initialization sequence follows a predictable, idempotent process that guarantees consistent deployment outcomes across heterogeneous environments. Each bootstrap operation validates prerequisites, provisions resources, and verifies operational readiness through comprehensive health checks.
-
-3. **Economic Efficiency**: The platform's architecture optimizes for total cost of ownership (TCO) by leveraging commodity hardware, open-source components, and efficient resource utilization patterns. Operational costs typically achieve 10x reduction compared to equivalent cloud-based solutions at scale.
-
-4. **Privacy by Design**: Data residency, processing, and storage remain exclusively within your infrastructure perimeter. The system implements defense-in-depth security layers, including encryption at rest, transport layer security, and role-based access control (RBAC).
-
-## System Requirements and Prerequisites
-
-### Hardware Requirements
-
-The platform requires the following minimum hardware specifications for production deployment:
-
-| Component | Minimum Specification | Recommended Specification | Enterprise Specification |
-|-----------|----------------------|---------------------------|-------------------------|
-| CPU | 4 cores x86_64 | 8 cores x86_64 | 16+ cores x86_64 |
-| Memory | 8 GB RAM | 16 GB RAM | 32+ GB RAM |
-| Storage | 50 GB SSD | 200 GB NVMe SSD | 1+ TB NVMe RAID |
-| Network | 100 Mbps | 1 Gbps | 10 Gbps |
-| GPU | Not required | NVIDIA GTX 1060 | NVIDIA A100 |
-
-### Operating System Compatibility
-
-The bootstrap system supports the following operating system distributions:
-
-- **Linux Distributions**: Ubuntu 20.04 LTS+, Debian 11+, RHEL 8+, CentOS Stream 8+, Fedora 35+, openSUSE Leap 15.3+
-- **macOS**: macOS 12 Monterey or later (Intel and Apple Silicon architectures)
-- **Windows**: Windows 10 Professional/Enterprise (version 2004+) with WSL2
-- **Container Platforms**: Docker 20.10+, Kubernetes 1.21+, LXC/LXD 4.0+
-
-### Network Configuration
-
-The system requires specific network ports for inter-process communication and external access:
-
-```
-Port Allocation Table:
-├── 8080 : HTTP Web Interface (configurable)
-├── 8081 : LLM Inference Server (internal)
-├── 5432 : PostgreSQL Database (internal)
-├── 6379 : Cache Service (internal)
-├── 9333 : Vector Database HTTP (internal)
-├── 9334 : Vector Database gRPC (internal)
-├── 9090 : Storage Service (internal)
-└── 9091 : Storage Admin Interface (internal)
-```
-
-## The Bootstrap Process - Technical Deep Dive
-
-### Phase 1: Environment Validation
-
-The bootstrap initiator performs comprehensive environment validation before resource provisioning:
+## Quick Start
+Download and run:
```bash
-$ ./botserver
-[Bootstrap] Detecting system architecture...
-[Bootstrap] Platform: linux-x86_64
-[Bootstrap] Available memory: 16384 MB
-[Bootstrap] Available storage: 487 GB
-[Bootstrap] Network interfaces: eth0 (1000 Mbps)
-[Bootstrap] Validating kernel capabilities...
-[Bootstrap] Checking file system permissions...
-[Bootstrap] All prerequisites satisfied ✓
+./botserver
```
-The validation phase examines:
-- Processor architecture and instruction set extensions
-- Available memory and swap configuration
-- Storage capacity and file system types
-- Network interface availability and bandwidth
-- Kernel capabilities and security modules
-- User permissions and system limits
+That's it. Everything installs automatically on first run.
-### Phase 2: Component Provisioning
+## What Happens
-The system provisions infrastructure components in a carefully orchestrated sequence that respects inter-service dependencies:
+1. **Auto-Bootstrap**: Installs PostgreSQL, cache, storage, and LLM
+2. **Load Bots**: Finds `.gbai` folders and loads them
+3. **Start Server**: Opens on `http://localhost:8080`
+4. **Ready**: Talk to your bot
-#### Database Layer Initialization
+## First Conversation
-PostgreSQL deployment includes:
-- Binary distribution download and verification
-- Database cluster initialization with optimal configuration
-- Schema creation and migration execution
-- Index generation and statistics initialization
-- Connection pooling configuration
-- Backup and recovery setup
+Open browser, type a message, get a response. Simple.
-```
-[PostgreSQL] Downloading PostgreSQL 16.2...
-[PostgreSQL] Verifying checksum: SHA256:a7b9c5d...✓
-[PostgreSQL] Initializing database cluster...
-[PostgreSQL] Creating system catalogs...
-[PostgreSQL] Configuring shared buffers: 4096 MB
-[PostgreSQL] Configuring effective cache: 12288 MB
-[PostgreSQL] Creating connection pool (size: 100)
-[PostgreSQL] Applying schema migrations...
-[PostgreSQL] Database ready on port 5432 ✓
-```
+## Topics Covered
-#### Caching Layer Deployment
+- [Overview](./overview.md) - What General Bots does
+- [Quick Start](./quick-start.md) - Get running fast
+- [Installation](./installation.md) - Detailed setup options
+- [First Conversation](./first-conversation.md) - Using your bot
+- [Sessions](./sessions.md) - How conversations are managed
-The high-performance caching subsystem provides:
-- In-memory data structure storage
-- Session state persistence
-- Distributed locking mechanisms
-- Pub/sub messaging channels
-- Time-series data support
+---
-```
-[Cache] Installing Valkey 7.2.5...
-[Cache] Configuring memory limit: 2048 MB
-[Cache] Setting eviction policy: allkeys-lru
-[Cache] Enabling persistence: AOF+RDB
-[Cache] Configuring replication: disabled
-[Cache] Cache service ready on port 6379 ✓
-```
-
-#### Object Storage System
-
-The distributed object storage layer implements:
-- Content-addressable storage (CAS)
-- Automatic replication and erasure coding
-- Multi-tier storage with hot/cold data management
-- S3-compatible API interface
-- Inline deduplication and compression
-
-```
-[Storage] Deploying SeaweedFS 3.59...
-[Storage] Initializing master server...
-[Storage] Starting volume servers (count: 3)...
-[Storage] Configuring replication: 001 (no replication)
-[Storage] Setting up S3 gateway...
-[Storage] Creating default buckets...
-[Storage] Object storage ready on port 9090 ✓
-```
-
-#### Vector Database Engine
-
-The semantic search infrastructure provides:
-- High-dimensional vector indexing
-- Approximate nearest neighbor search
-- Dynamic index updates
-- Filtered search capabilities
-- Horizontal scaling support
-
-```
-[Vectors] Installing Qdrant 1.7.4...
-[Vectors] Configuring storage: ./botserver-stack/qdrant
-[Vectors] Setting vector dimensions: 384
-[Vectors] Creating default collection...
-[Vectors] Building HNSW index (M=16, ef=100)...
-[Vectors] Vector database ready on port 9333 ✓
-```
-
-### Phase 3: AI Model Deployment
-
-The platform provisions machine learning models for various cognitive tasks:
-
-#### Embedding Model Configuration
-
-```
-[Models] Downloading embedding model...
-[Models] Model: all-MiniLM-L6-v2 (384 dimensions)
-[Models] Format: ONNX (optimized for CPU)
-[Models] Quantization: INT8 (4x size reduction)
-[Models] Loading model into memory...
-[Models] Warming up inference engine...
-[Models] Embeddings ready (latency: 12ms) ✓
-```
-
-#### Language Model Setup
-
-For local inference capability (optional):
-
-```
-[LLM] Configuring language model server...
-[LLM] Model path: ./models/llama-7b-q4.gguf
-[LLM] Context size: 4096 tokens
-[LLM] Batch size: 512 tokens
-[LLM] Thread count: 8
-[LLM] GPU acceleration: disabled
-[LLM] Starting inference server on port 8081...
-[LLM] Model loaded (3.9 GB memory) ✓
-```
-
-### Phase 4: Application Initialization
-
-The final bootstrap phase configures the application runtime:
-
-```
-[App] Creating default bot package...
-[App] Loading dialog templates...
-[App] Compiling BASIC interpreter...
-[App] Registering system keywords...
-[App] Initializing REST API endpoints...
-[App] Starting WebSocket server...
-[App] Launching web interface on :8080...
-[App] System operational ✓
-
-Bootstrap completed in 4m 32s
-Web interface: http://localhost:8080
-```
-
-## Creating Your First Bot - Implementation Guide
-
-### Package Structure and Organization
-
-The bot package system implements a convention-over-configuration approach with hierarchical resource organization:
-
-```
-templates/
-└── my-bot.gbai/ # Package root (mandatory .gbai extension)
- ├── my-bot.gbdialog/ # Dialog scripts container
- │ ├── start.bas # Entry point (required)
- │ ├── handlers/ # Event handlers
- │ │ ├── welcome.bas # User onboarding
- │ │ └── error.bas # Error handling
- │ └── tools/ # Custom tools
- │ ├── scheduler.bas # Appointment booking
- │ └── calculator.bas # Calculations
- ├── my-bot.gbkb/ # Knowledge base
- │ ├── policies/ # Document collection
- │ │ ├── hr-manual.pdf # Human resources
- │ │ └── it-security.pdf # IT policies
- │ └── products/ # Product information
- │ ├── catalog.pdf # Product catalog
- │ └── pricing.xlsx # Pricing matrix
- ├── my-bot.gbot/ # Configuration
- │ └── config.csv # Bot parameters
- ├── my-bot.gbtheme/ # Visual customization
- │ └── default.css # Style definitions
- └── my-bot.gbdrive/ # File storage
- └── templates/ # Document templates
-```
-
-### Dialog Script Implementation
-
-BASIC dialog scripts orchestrate conversational flows with minimal complexity:
-
-```basic
-' start.bas - Primary conversation entry point
-' This script executes when users initiate conversation
-
-' Initialize conversation context
-SET CONTEXT "greeting_shown" = FALSE
-
-' Load knowledge resources
-USE KB "policies" ' HR and IT policy documents
-USE KB "products" ' Product catalog and pricing
-
-' Enable tool extensions
-USE TOOL "scheduler" ' Appointment booking capability
-USE TOOL "calculator" ' Mathematical computations
-
-' Conversation state machine
-main_loop:
- IF GET CONTEXT "greeting_shown" = FALSE THEN
- TALK "Welcome! I'm your intelligent assistant."
- TALK "I have access to company policies and product information."
- TALK "I can also schedule appointments and perform calculations."
- SET CONTEXT "greeting_shown" = TRUE
- END IF
-
- TALK "How may I assist you today?"
-
- ' Capture user input with intent classification
- user_input = HEAR
-
- ' The system AI automatically processes the input using:
- ' - Loaded knowledge bases for information retrieval
- ' - Enabled tools for action execution
- ' - Context history for coherent responses
-
- ' Continue conversation loop
- GOTO main_loop
-```
-
-### Knowledge Base Integration
-
-Document collections are automatically indexed for semantic retrieval:
-
-#### Document Processing Pipeline
-
-1. **Ingestion**: Documents are parsed from supported formats (PDF, DOCX, XLSX, TXT, MD)
-2. **Chunking**: Content is segmented into semantic units (typically 256-512 tokens)
-3. **Embedding**: Each chunk is converted to vector representation using the embedding model
-4. **Indexing**: Vectors are inserted into the vector database with metadata
-5. **Retrieval**: Semantic search identifies relevant chunks based on query similarity
-
-#### Supported Document Formats
-
-| Format | Extensions | Processing Method | Metadata Extraction |
-|--------|------------|------------------|-------------------|
-| PDF | .pdf | Apache PDFBox | Title, Author, Creation Date |
-| Word | .docx, .doc | Apache POI | Properties, Styles |
-| Excel | .xlsx, .xls | Apache POI | Sheets, Named Ranges |
-| Text | .txt, .md | Direct | File Properties |
-| HTML | .html, .htm | JSoup | Meta Tags, Structure |
-| CSV | .csv | Native Parser | Headers, Schema |
-
-### Tool Development
-
-Tools extend bot capabilities with parameterized functions:
-
-```basic
-' scheduler.bas - Meeting scheduling tool
-PARAM person AS STRING DESCRIPTION "Person to meet"
-PARAM date AS DATE DESCRIPTION "Meeting date"
-PARAM time AS TIME DESCRIPTION "Meeting time"
-PARAM duration AS INTEGER DEFAULT 30 DESCRIPTION "Duration in minutes"
-PARAM location AS STRING DEFAULT "Conference Room" DESCRIPTION "Meeting location"
-
-DESCRIPTION "Schedule a meeting with specified parameters"
-
-' Validate scheduling constraints
-IF date < TODAY() THEN
- TALK "Cannot schedule meetings in the past."
- EXIT
-END IF
-
-IF time < "08:00" OR time > "18:00" THEN
- TALK "Meetings must be scheduled during business hours (8 AM - 6 PM)."
- EXIT
-END IF
-
-' Check availability (simplified example)
-conflicts = FIND "meetings", "date=" + date + " AND time=" + time
-IF conflicts.COUNT > 0 THEN
- TALK "Time slot unavailable. Suggesting alternatives..."
- ' AI generates alternative suggestions
- EXIT
-END IF
-
-' Create meeting record
-meeting_id = GENERATE_ID()
-SAVE "meetings", meeting_id, person, date, time, duration, location
-
-' Send confirmation
-TALK "Meeting scheduled successfully!"
-TALK "Details: " + person + " on " + FORMAT(date, "MMMM dd, yyyy")
-TALK "Time: " + FORMAT(time, "h:mm a") + " for " + duration + " minutes"
-TALK "Location: " + location
-
-' Optional: Send calendar invitation
-IF GET BOT MEMORY "send_invites" = TRUE THEN
- SEND MAIL person, "Meeting Invitation", "You're invited to a meeting..."
-END IF
-```
-
-## Configuration Management
-
-### Environment Variables
-
-The system uses environment variables for deployment-specific configuration:
-
-```bash
-# Network Configuration
-HTTP_PORT=8080 # Web interface port
-HTTP_HOST=0.0.0.0 # Binding address
-BASE_URL=https://bot.company.com # Public URL
-
-# Resource Limits
-MAX_MEMORY=8192 # Maximum memory (MB)
-MAX_CONNECTIONS=1000 # Connection pool size
-MAX_WORKERS=16 # Worker thread count
-
-# Storage Paths
-DATA_DIR=/var/lib/botserver # Data directory
-TEMP_DIR=/tmp/botserver # Temporary files
-LOG_DIR=/var/log/botserver # Log files
-
-# Feature Flags
-ENABLE_GPU=false # GPU acceleration
-ENABLE_ANALYTICS=true # Usage analytics
-ENABLE_BACKUP=true # Automatic backups
-```
-
-### Configuration File (config.csv)
-
-Bot-specific parameters are defined in CSV format:
-
-```csv
-name,value,description,category
-# LLM Configuration
-llm-provider,local,LLM service provider,llm
-llm-model,./models/llama2-7b.gguf,Model file path,llm
-llm-context,4096,Context window size,llm
-llm-temperature,0.7,Sampling temperature,llm
-llm-max-tokens,2048,Maximum response tokens,llm
-
-# Context Management
-context-compaction,auto,Compaction strategy,context
-context-max-messages,50,Message history limit,context
-context-summary-threshold,1000,Summary trigger tokens,context
-
-# Knowledge Base
-kb-chunk-size,512,Document chunk size,knowledge
-kb-chunk-overlap,50,Chunk overlap tokens,knowledge
-kb-relevance-threshold,0.7,Minimum similarity score,knowledge
-kb-max-results,5,Maximum search results,knowledge
-
-# Session Management
-session-timeout,3600,Session timeout (seconds),session
-session-persistence,true,Persist sessions to disk,session
-session-encryption,true,Encrypt session data,session
-```
-
-## Operational Procedures
-
-### System Monitoring
-
-Monitor system health through built-in endpoints:
-
-```bash
-# Check component status
-$ ./botserver status
-Component Status CPU Memory Uptime
-─────────────────────────────────────────────────
-PostgreSQL Running 2.1% 487 MB 4h 32m
-Cache Running 0.8% 156 MB 4h 32m
-Storage Running 1.4% 234 MB 4h 32m
-Vectors Running 3.2% 892 MB 4h 32m
-LLM Server Running 45.6% 3.9 GB 4h 31m
-Web Server Running 5.3% 312 MB 4h 31m
-
-# View metrics
-$ ./botserver metrics
-Metric Value Change (1h)
-──────────────────────────────────────────────────
-Total Conversations 1,247 +82
-Active Sessions 34 +5
-Messages Processed 15,823 +1,247
-Knowledge Queries 3,421 +234
-Tool Invocations 892 +67
-Average Response Time 342ms -12ms
-Cache Hit Rate 87.3% +2.1%
-Vector Search Latency 23ms -3ms
-```
-
-### Backup and Recovery
-
-The system implements comprehensive backup strategies:
-
-#### Automated Backups
-
-```bash
-# Configure automated backups
-$ ./botserver backup configure
-Backup Schedule: Daily at 02:00 UTC
-Retention Policy: 30 days
-Destination: ./backups/
-Compression: Enabled (zstd)
-Encryption: Enabled (AES-256)
-
-# Manual backup
-$ ./botserver backup create
-Creating backup snapshot...
-[Database] Dumping PostgreSQL... 234 MB
-[Vectors] Exporting collections... 567 MB
-[Storage] Copying objects... 1.2 GB
-[Config] Saving configuration... 12 KB
-Compressing backup... 689 MB (42% ratio)
-Backup created: backup-20240315-143022.tar.zst
-```
-
-#### Disaster Recovery
-
-```bash
-# Restore from backup
-$ ./botserver restore backup-20240315-143022.tar.zst
-Extracting backup archive...
-Stopping services...
-[Database] Restoring PostgreSQL...
-[Vectors] Importing collections...
-[Storage] Restoring objects...
-[Config] Applying configuration...
-Starting services...
-Restoration completed successfully
-```
-
-### Performance Optimization
-
-#### Database Tuning
-
-```sql
--- Analyze query performance
-EXPLAIN ANALYZE
-SELECT * FROM conversations
-WHERE bot_id = $1 AND created_at > $2
-ORDER BY created_at DESC
-LIMIT 100;
-
--- Create optimized indexes
-CREATE INDEX idx_conversations_bot_created
-ON conversations(bot_id, created_at DESC);
-
--- Update statistics
-ANALYZE conversations;
-```
-
-#### Cache Optimization
-
-```bash
-# Monitor cache performance
-$ ./botserver cache stats
-Keyspace hits: 127,834
-Keyspace misses: 14,234
-Hit ratio: 89.97%
-Evicted keys: 3,421
-Memory used: 1.4 GB / 2.0 GB
-```
-
-#### Vector Index Tuning
-
-```python
-# Optimize HNSW parameters
-{
- "index_type": "hnsw",
- "parameters": {
- "m": 32, # Connectivity parameter
- "ef_construction": 200, # Construction time quality
- "ef_runtime": 100, # Search time quality
- "metric": "cosine" # Distance metric
- }
-}
-```
-
-## Container Deployment Architecture
-
-### LXC Container Isolation
-
-The platform supports Linux Container (LXC) deployment for enhanced isolation:
-
-```bash
-$ ./botserver --container
-[LXC] Creating container network...
-[LXC] Network: 10.10.10.0/24
-[LXC] Creating containers...
-├── botserver-postgres (10.10.10.2)
-├── botserver-cache (10.10.10.3)
-├── botserver-storage (10.10.10.4)
-├── botserver-vectors (10.10.10.5)
-└── botserver-app (10.10.10.6)
-[LXC] Configuring inter-container communication...
-[LXC] Setting resource limits...
-[LXC] Starting containers...
-[LXC] All containers operational ✓
-```
-
-### Container Resource Management
-
-```yaml
-# Container resource limits
-postgres:
- cpu: 2
- memory: 4GB
- storage: 100GB
-
-cache:
- cpu: 1
- memory: 2GB
- storage: 10GB
-
-storage:
- cpu: 2
- memory: 2GB
- storage: 500GB
-
-vectors:
- cpu: 2
- memory: 4GB
- storage: 50GB
-
-app:
- cpu: 4
- memory: 8GB
- storage: 20GB
-```
-
-## Security Considerations
-
-### Network Security
-
-The platform implements defense-in-depth network security:
-
-1. **Transport Layer Security**: All inter-service communication uses TLS 1.3
-2. **Network Segmentation**: Services operate in isolated network namespaces
-3. **Firewall Rules**: Automatic iptables/nftables configuration
-4. **Rate Limiting**: Connection and request throttling
-5. **DDoS Protection**: SYN flood mitigation and connection limits
-
-### Data Protection
-
-Comprehensive data protection mechanisms include:
-
-1. **Encryption at Rest**: AES-256 for database and storage
-2. **Key Management**: Hardware security module (HSM) integration
-3. **Access Control**: Role-based permissions with audit logging
-4. **Data Anonymization**: PII detection and masking
-5. **Compliance**: GDPR, HIPAA, SOC 2 support
-
-## Troubleshooting Guide
-
-### Common Issues and Resolutions
-
-#### Port Conflicts
-
-```bash
-# Error: Address already in use
-$ lsof -i :8080
-COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
-nginx 1234 www 6u IPv4 12345 0t0 TCP *:8080
-
-# Resolution
-$ HTTP_PORT=3000 ./botserver
-# Or stop conflicting service
-$ sudo systemctl stop nginx
-```
-
-#### Memory Limitations
-
-```bash
-# Error: Cannot allocate memory
-# Check system limits
-$ ulimit -a
-$ sysctl vm.overcommit_memory
-
-# Increase limits
-$ ulimit -n 65536 # File descriptors
-$ ulimit -u 32768 # Processes
-$ sudo sysctl vm.overcommit_memory=1
-```
-
-#### Database Connection Issues
-
-```bash
-# Error: Connection refused
-# Check PostgreSQL status
-$ ./botserver status postgres
-PostgreSQL: Stopped
-
-# View logs
-$ tail -f botserver-stack/postgres/logs/postgresql.log
-
-# Restart service
-$ ./botserver restart postgres
-```
-
-### Diagnostic Commands
-
-```bash
-# System diagnostics
-$ ./botserver diagnose
-Running system diagnostics...
-✓ CPU architecture compatible
-✓ Memory sufficient (12.4 GB available)
-✓ Storage adequate (234 GB free)
-✓ Network connectivity verified
-✓ DNS resolution working
-✓ Time synchronization accurate
-⚠ Swap disabled (recommended: 8 GB)
-✓ Kernel parameters optimal
-✓ File descriptors adequate (65536)
-
-# Component health checks
-$ ./botserver health
-Service Endpoint Status Latency
-────────────────────────────────────────────────────
-PostgreSQL localhost:5432 Healthy 2ms
-Cache localhost:6379 Healthy <1ms
-Storage localhost:9090 Healthy 5ms
-Vectors localhost:9333 Healthy 3ms
-LLM localhost:8081 Healthy 45ms
-Application localhost:8080 Healthy 12ms
-```
-
-## Performance Metrics and Benchmarks
-
-### System Performance Characteristics
-
-| Operation | Latency (p50) | Latency (p99) | Throughput | Resource Usage |
-|-----------|---------------|---------------|------------|----------------|
-| Message Processing | 42ms | 156ms | 2,400/sec | CPU: 15%, RAM: 1.2GB |
-| Knowledge Query | 78ms | 234ms | 450/sec | CPU: 25%, RAM: 2.1GB |
-| Tool Invocation | 156ms | 512ms | 180/sec | CPU: 35%, RAM: 1.8GB |
-| Document Indexing | 2.3s/doc | 5.6s/doc | 25 docs/min | CPU: 45%, RAM: 3.2GB |
-| Session Creation | 12ms | 34ms | 5,000/sec | CPU: 8%, RAM: 0.5GB |
-
-### Scalability Characteristics
-
-The platform demonstrates linear scalability characteristics:
-
-- **Horizontal Scaling**: Add nodes for increased capacity
-- **Vertical Scaling**: Utilize additional CPU/RAM resources
-- **Load Distribution**: Automatic request balancing
-- **Session Affinity**: Sticky sessions for stateful operations
-- **Cache Coherence**: Distributed cache synchronization
-
-## Integration Capabilities
-
-### API Integration Patterns
-
-The platform supports multiple integration patterns:
-
-#### REST API Integration
-```bash
-# Create conversation
-POST /api/conversations
-{
- "bot_id": "my-bot",
- "user_id": "user-123",
- "message": "Hello, I need help"
-}
-
-# Response
-{
- "conversation_id": "conv-789",
- "session_id": "sess-456",
- "response": "Hello! I'm here to help. What do you need assistance with?",
- "suggested_actions": ["Product Information", "Technical Support", "Billing"]
-}
-```
-
-#### WebSocket Real-time Communication
-```javascript
-// Client connection
-const ws = new WebSocket('ws://localhost:8080/ws');
-
-ws.on('message', (data) => {
- const message = JSON.parse(data);
- console.log('Bot response:', message.text);
-});
-
-ws.send(JSON.stringify({
- type: 'message',
- text: 'What are your business hours?'
-}));
-```
-
-#### Webhook Event Notifications
-```yaml
-webhooks:
- - url: https://api.company.com/bot-events
- events:
- - conversation.started
- - conversation.ended
- - tool.invoked
- - error.occurred
- headers:
- Authorization: Bearer ${WEBHOOK_SECRET}
-```
-
-## Advanced Configuration
-
-### Multi-Tenant Deployment
-
-Configure isolated bot instances for multiple tenants:
-
-```csv
-name,value,category
-tenant-mode,enabled,system
-tenant-isolation,strict,system
-tenant-database-prefix,tenant_,system
-tenant-storage-bucket,tenant-{id},system
-tenant-resource-limits,true,system
-tenant-max-conversations,1000,limits
-tenant-max-storage,10GB,limits
-tenant-max-compute,4CPU,limits
-```
-
-### High Availability Configuration
-
-Implement redundancy for production deployments:
-
-```yaml
-cluster:
- mode: active-passive
- nodes:
- - primary: bot1.company.com
- - standby: bot2.company.com
-
- replication:
- type: streaming
- lag_threshold: 100ms
-
- failover:
- automatic: true
- timeout: 30s
-
- load_balancing:
- algorithm: least_connections
- health_check: /health
- interval: 10s
-```
-
-## Compliance and Governance
-
-### Audit Logging
-
-Comprehensive audit trail for all system operations:
-
-```json
-{
- "timestamp": "2024-03-15T14:30:22.123Z",
- "event_type": "conversation.message",
- "actor": {
- "type": "user",
- "id": "user-123",
- "ip": "192.168.1.100"
- },
- "resource": {
- "type": "conversation",
- "id": "conv-789",
- "bot_id": "my-bot"
- },
- "action": "create",
- "result": "success",
- "metadata": {
- "message_length": 45,
- "tools_used": ["scheduler"],
- "knowledge_bases": ["policies
\ No newline at end of file
+
\ No newline at end of file
diff --git a/docs/src/chapter-02/gbai.md b/docs/src/chapter-02/gbai.md
index 075fa331f..d5c47f1d4 100644
--- a/docs/src/chapter-02/gbai.md
+++ b/docs/src/chapter-02/gbai.md
@@ -17,7 +17,7 @@ That's it. No manifests, no build files, no dependencies. Copy the folder to dep
### Visual Architecture
### Architecture
-
+
## How Bootstrap Finds Bots
@@ -224,7 +224,7 @@ templates/
When BotServer starts:
```
-
+
```
Takes about 5-10 seconds per bot.
diff --git a/docs/src/chapter-03/README.md b/docs/src/chapter-03/README.md
index cecb49603..0a87d5e56 100644
--- a/docs/src/chapter-03/README.md
+++ b/docs/src/chapter-03/README.md
@@ -12,82 +12,11 @@ The knowledge base system transforms unstructured documents into queryable seman
The knowledge base architecture implements a multi-stage pipeline for document processing and retrieval:
-
+
### Technical Specifications
-
+
The core database schema for GeneralBots is defined in `src/shared/models.rs`. It uses **Diesel** with PostgreSQL and includes the following primary tables:
@@ -44,3 +44,9 @@ The core database schema for GeneralBots is defined in `src/shared/models.rs`. I
- timestamp: Timestamp
The schema is automatically migrated when the server starts.
+
+---
+
+
## Schema Categories
@@ -57,12 +57,12 @@ The database schema follows these design principles:
### Session Management Flow
### Session Flow
-
+
### Knowledge Base Access Pattern
### Knowledge Base Access
-
+
### Primary Relationships
diff --git a/docs/src/chapter-01/README.md b/docs/src/chapter-01/README.md
index d1a470365..8067b635f 100644
--- a/docs/src/chapter-01/README.md
+++ b/docs/src/chapter-01/README.md
@@ -1,833 +1,44 @@
-# Chapter 01: Run and Talk - System Bootstrap and Initial Configuration
+# Chapter 01: Run and Talk
-Welcome to General Bots - a comprehensive, self-hosted artificial intelligence platform designed for enterprise deployment and individual sovereignty. This chapter provides detailed technical guidance on system initialization, bootstrap processes, and fundamental operational concepts that form the foundation of your AI infrastructure.
+Get General Bots running in 5 minutes.
-## Executive Summary
+## What You'll Learn
-General Bots represents a paradigm shift in conversational AI deployment, offering a fully autonomous, self-contained system that eliminates dependencies on external cloud services while maintaining enterprise-grade capabilities. The platform implements a zero-configuration bootstrap methodology that provisions all required infrastructure components automatically, enabling rapid deployment without specialized DevOps expertise.
+- How to start General Bots
+- Your first conversation
+- How sessions work
+- Basic bot interaction
-### Key Architectural Principles
-
-The system adheres to four fundamental design principles that govern its implementation:
-
-1. **Infrastructure Autonomy**: Complete ownership of the technology stack, from persistent storage layers to natural language processing engines. All components operate within your controlled environment, ensuring data sovereignty and regulatory compliance.
-
-2. **Deterministic Bootstrap**: The initialization sequence follows a predictable, idempotent process that guarantees consistent deployment outcomes across heterogeneous environments. Each bootstrap operation validates prerequisites, provisions resources, and verifies operational readiness through comprehensive health checks.
-
-3. **Economic Efficiency**: The platform's architecture optimizes for total cost of ownership (TCO) by leveraging commodity hardware, open-source components, and efficient resource utilization patterns. Operational costs typically achieve 10x reduction compared to equivalent cloud-based solutions at scale.
-
-4. **Privacy by Design**: Data residency, processing, and storage remain exclusively within your infrastructure perimeter. The system implements defense-in-depth security layers, including encryption at rest, transport layer security, and role-based access control (RBAC).
-
-## System Requirements and Prerequisites
-
-### Hardware Requirements
-
-The platform requires the following minimum hardware specifications for production deployment:
-
-| Component | Minimum Specification | Recommended Specification | Enterprise Specification |
-|-----------|----------------------|---------------------------|-------------------------|
-| CPU | 4 cores x86_64 | 8 cores x86_64 | 16+ cores x86_64 |
-| Memory | 8 GB RAM | 16 GB RAM | 32+ GB RAM |
-| Storage | 50 GB SSD | 200 GB NVMe SSD | 1+ TB NVMe RAID |
-| Network | 100 Mbps | 1 Gbps | 10 Gbps |
-| GPU | Not required | NVIDIA GTX 1060 | NVIDIA A100 |
-
-### Operating System Compatibility
-
-The bootstrap system supports the following operating system distributions:
-
-- **Linux Distributions**: Ubuntu 20.04 LTS+, Debian 11+, RHEL 8+, CentOS Stream 8+, Fedora 35+, openSUSE Leap 15.3+
-- **macOS**: macOS 12 Monterey or later (Intel and Apple Silicon architectures)
-- **Windows**: Windows 10 Professional/Enterprise (version 2004+) with WSL2
-- **Container Platforms**: Docker 20.10+, Kubernetes 1.21+, LXC/LXD 4.0+
-
-### Network Configuration
-
-The system requires specific network ports for inter-process communication and external access:
-
-```
-Port Allocation Table:
-├── 8080 : HTTP Web Interface (configurable)
-├── 8081 : LLM Inference Server (internal)
-├── 5432 : PostgreSQL Database (internal)
-├── 6379 : Cache Service (internal)
-├── 9333 : Vector Database HTTP (internal)
-├── 9334 : Vector Database gRPC (internal)
-├── 9090 : Storage Service (internal)
-└── 9091 : Storage Admin Interface (internal)
-```
-
-## The Bootstrap Process - Technical Deep Dive
-
-### Phase 1: Environment Validation
-
-The bootstrap initiator performs comprehensive environment validation before resource provisioning:
+## Quick Start
+Download and run:
```bash
-$ ./botserver
-[Bootstrap] Detecting system architecture...
-[Bootstrap] Platform: linux-x86_64
-[Bootstrap] Available memory: 16384 MB
-[Bootstrap] Available storage: 487 GB
-[Bootstrap] Network interfaces: eth0 (1000 Mbps)
-[Bootstrap] Validating kernel capabilities...
-[Bootstrap] Checking file system permissions...
-[Bootstrap] All prerequisites satisfied ✓
+./botserver
```
-The validation phase examines:
-- Processor architecture and instruction set extensions
-- Available memory and swap configuration
-- Storage capacity and file system types
-- Network interface availability and bandwidth
-- Kernel capabilities and security modules
-- User permissions and system limits
+That's it. Everything installs automatically on first run.
-### Phase 2: Component Provisioning
+## What Happens
-The system provisions infrastructure components in a carefully orchestrated sequence that respects inter-service dependencies:
+1. **Auto-Bootstrap**: Installs PostgreSQL, cache, storage, and LLM
+2. **Load Bots**: Finds `.gbai` folders and loads them
+3. **Start Server**: Opens on `http://localhost:8080`
+4. **Ready**: Talk to your bot
-#### Database Layer Initialization
+## First Conversation
-PostgreSQL deployment includes:
-- Binary distribution download and verification
-- Database cluster initialization with optimal configuration
-- Schema creation and migration execution
-- Index generation and statistics initialization
-- Connection pooling configuration
-- Backup and recovery setup
+Open browser, type a message, get a response. Simple.
-```
-[PostgreSQL] Downloading PostgreSQL 16.2...
-[PostgreSQL] Verifying checksum: SHA256:a7b9c5d...✓
-[PostgreSQL] Initializing database cluster...
-[PostgreSQL] Creating system catalogs...
-[PostgreSQL] Configuring shared buffers: 4096 MB
-[PostgreSQL] Configuring effective cache: 12288 MB
-[PostgreSQL] Creating connection pool (size: 100)
-[PostgreSQL] Applying schema migrations...
-[PostgreSQL] Database ready on port 5432 ✓
-```
+## Topics Covered
-#### Caching Layer Deployment
+- [Overview](./overview.md) - What General Bots does
+- [Quick Start](./quick-start.md) - Get running fast
+- [Installation](./installation.md) - Detailed setup options
+- [First Conversation](./first-conversation.md) - Using your bot
+- [Sessions](./sessions.md) - How conversations are managed
-The high-performance caching subsystem provides:
-- In-memory data structure storage
-- Session state persistence
-- Distributed locking mechanisms
-- Pub/sub messaging channels
-- Time-series data support
+---
-```
-[Cache] Installing Valkey 7.2.5...
-[Cache] Configuring memory limit: 2048 MB
-[Cache] Setting eviction policy: allkeys-lru
-[Cache] Enabling persistence: AOF+RDB
-[Cache] Configuring replication: disabled
-[Cache] Cache service ready on port 6379 ✓
-```
-
-#### Object Storage System
-
-The distributed object storage layer implements:
-- Content-addressable storage (CAS)
-- Automatic replication and erasure coding
-- Multi-tier storage with hot/cold data management
-- S3-compatible API interface
-- Inline deduplication and compression
-
-```
-[Storage] Deploying SeaweedFS 3.59...
-[Storage] Initializing master server...
-[Storage] Starting volume servers (count: 3)...
-[Storage] Configuring replication: 001 (no replication)
-[Storage] Setting up S3 gateway...
-[Storage] Creating default buckets...
-[Storage] Object storage ready on port 9090 ✓
-```
-
-#### Vector Database Engine
-
-The semantic search infrastructure provides:
-- High-dimensional vector indexing
-- Approximate nearest neighbor search
-- Dynamic index updates
-- Filtered search capabilities
-- Horizontal scaling support
-
-```
-[Vectors] Installing Qdrant 1.7.4...
-[Vectors] Configuring storage: ./botserver-stack/qdrant
-[Vectors] Setting vector dimensions: 384
-[Vectors] Creating default collection...
-[Vectors] Building HNSW index (M=16, ef=100)...
-[Vectors] Vector database ready on port 9333 ✓
-```
-
-### Phase 3: AI Model Deployment
-
-The platform provisions machine learning models for various cognitive tasks:
-
-#### Embedding Model Configuration
-
-```
-[Models] Downloading embedding model...
-[Models] Model: all-MiniLM-L6-v2 (384 dimensions)
-[Models] Format: ONNX (optimized for CPU)
-[Models] Quantization: INT8 (4x size reduction)
-[Models] Loading model into memory...
-[Models] Warming up inference engine...
-[Models] Embeddings ready (latency: 12ms) ✓
-```
-
-#### Language Model Setup
-
-For local inference capability (optional):
-
-```
-[LLM] Configuring language model server...
-[LLM] Model path: ./models/llama-7b-q4.gguf
-[LLM] Context size: 4096 tokens
-[LLM] Batch size: 512 tokens
-[LLM] Thread count: 8
-[LLM] GPU acceleration: disabled
-[LLM] Starting inference server on port 8081...
-[LLM] Model loaded (3.9 GB memory) ✓
-```
-
-### Phase 4: Application Initialization
-
-The final bootstrap phase configures the application runtime:
-
-```
-[App] Creating default bot package...
-[App] Loading dialog templates...
-[App] Compiling BASIC interpreter...
-[App] Registering system keywords...
-[App] Initializing REST API endpoints...
-[App] Starting WebSocket server...
-[App] Launching web interface on :8080...
-[App] System operational ✓
-
-Bootstrap completed in 4m 32s
-Web interface: http://localhost:8080
-```
-
-## Creating Your First Bot - Implementation Guide
-
-### Package Structure and Organization
-
-The bot package system implements a convention-over-configuration approach with hierarchical resource organization:
-
-```
-templates/
-└── my-bot.gbai/ # Package root (mandatory .gbai extension)
- ├── my-bot.gbdialog/ # Dialog scripts container
- │ ├── start.bas # Entry point (required)
- │ ├── handlers/ # Event handlers
- │ │ ├── welcome.bas # User onboarding
- │ │ └── error.bas # Error handling
- │ └── tools/ # Custom tools
- │ ├── scheduler.bas # Appointment booking
- │ └── calculator.bas # Calculations
- ├── my-bot.gbkb/ # Knowledge base
- │ ├── policies/ # Document collection
- │ │ ├── hr-manual.pdf # Human resources
- │ │ └── it-security.pdf # IT policies
- │ └── products/ # Product information
- │ ├── catalog.pdf # Product catalog
- │ └── pricing.xlsx # Pricing matrix
- ├── my-bot.gbot/ # Configuration
- │ └── config.csv # Bot parameters
- ├── my-bot.gbtheme/ # Visual customization
- │ └── default.css # Style definitions
- └── my-bot.gbdrive/ # File storage
- └── templates/ # Document templates
-```
-
-### Dialog Script Implementation
-
-BASIC dialog scripts orchestrate conversational flows with minimal complexity:
-
-```basic
-' start.bas - Primary conversation entry point
-' This script executes when users initiate conversation
-
-' Initialize conversation context
-SET CONTEXT "greeting_shown" = FALSE
-
-' Load knowledge resources
-USE KB "policies" ' HR and IT policy documents
-USE KB "products" ' Product catalog and pricing
-
-' Enable tool extensions
-USE TOOL "scheduler" ' Appointment booking capability
-USE TOOL "calculator" ' Mathematical computations
-
-' Conversation state machine
-main_loop:
- IF GET CONTEXT "greeting_shown" = FALSE THEN
- TALK "Welcome! I'm your intelligent assistant."
- TALK "I have access to company policies and product information."
- TALK "I can also schedule appointments and perform calculations."
- SET CONTEXT "greeting_shown" = TRUE
- END IF
-
- TALK "How may I assist you today?"
-
- ' Capture user input with intent classification
- user_input = HEAR
-
- ' The system AI automatically processes the input using:
- ' - Loaded knowledge bases for information retrieval
- ' - Enabled tools for action execution
- ' - Context history for coherent responses
-
- ' Continue conversation loop
- GOTO main_loop
-```
-
-### Knowledge Base Integration
-
-Document collections are automatically indexed for semantic retrieval:
-
-#### Document Processing Pipeline
-
-1. **Ingestion**: Documents are parsed from supported formats (PDF, DOCX, XLSX, TXT, MD)
-2. **Chunking**: Content is segmented into semantic units (typically 256-512 tokens)
-3. **Embedding**: Each chunk is converted to vector representation using the embedding model
-4. **Indexing**: Vectors are inserted into the vector database with metadata
-5. **Retrieval**: Semantic search identifies relevant chunks based on query similarity
-
-#### Supported Document Formats
-
-| Format | Extensions | Processing Method | Metadata Extraction |
-|--------|------------|------------------|-------------------|
-| PDF | .pdf | Apache PDFBox | Title, Author, Creation Date |
-| Word | .docx, .doc | Apache POI | Properties, Styles |
-| Excel | .xlsx, .xls | Apache POI | Sheets, Named Ranges |
-| Text | .txt, .md | Direct | File Properties |
-| HTML | .html, .htm | JSoup | Meta Tags, Structure |
-| CSV | .csv | Native Parser | Headers, Schema |
-
-### Tool Development
-
-Tools extend bot capabilities with parameterized functions:
-
-```basic
-' scheduler.bas - Meeting scheduling tool
-PARAM person AS STRING DESCRIPTION "Person to meet"
-PARAM date AS DATE DESCRIPTION "Meeting date"
-PARAM time AS TIME DESCRIPTION "Meeting time"
-PARAM duration AS INTEGER DEFAULT 30 DESCRIPTION "Duration in minutes"
-PARAM location AS STRING DEFAULT "Conference Room" DESCRIPTION "Meeting location"
-
-DESCRIPTION "Schedule a meeting with specified parameters"
-
-' Validate scheduling constraints
-IF date < TODAY() THEN
- TALK "Cannot schedule meetings in the past."
- EXIT
-END IF
-
-IF time < "08:00" OR time > "18:00" THEN
- TALK "Meetings must be scheduled during business hours (8 AM - 6 PM)."
- EXIT
-END IF
-
-' Check availability (simplified example)
-conflicts = FIND "meetings", "date=" + date + " AND time=" + time
-IF conflicts.COUNT > 0 THEN
- TALK "Time slot unavailable. Suggesting alternatives..."
- ' AI generates alternative suggestions
- EXIT
-END IF
-
-' Create meeting record
-meeting_id = GENERATE_ID()
-SAVE "meetings", meeting_id, person, date, time, duration, location
-
-' Send confirmation
-TALK "Meeting scheduled successfully!"
-TALK "Details: " + person + " on " + FORMAT(date, "MMMM dd, yyyy")
-TALK "Time: " + FORMAT(time, "h:mm a") + " for " + duration + " minutes"
-TALK "Location: " + location
-
-' Optional: Send calendar invitation
-IF GET BOT MEMORY "send_invites" = TRUE THEN
- SEND MAIL person, "Meeting Invitation", "You're invited to a meeting..."
-END IF
-```
-
-## Configuration Management
-
-### Environment Variables
-
-The system uses environment variables for deployment-specific configuration:
-
-```bash
-# Network Configuration
-HTTP_PORT=8080 # Web interface port
-HTTP_HOST=0.0.0.0 # Binding address
-BASE_URL=https://bot.company.com # Public URL
-
-# Resource Limits
-MAX_MEMORY=8192 # Maximum memory (MB)
-MAX_CONNECTIONS=1000 # Connection pool size
-MAX_WORKERS=16 # Worker thread count
-
-# Storage Paths
-DATA_DIR=/var/lib/botserver # Data directory
-TEMP_DIR=/tmp/botserver # Temporary files
-LOG_DIR=/var/log/botserver # Log files
-
-# Feature Flags
-ENABLE_GPU=false # GPU acceleration
-ENABLE_ANALYTICS=true # Usage analytics
-ENABLE_BACKUP=true # Automatic backups
-```
-
-### Configuration File (config.csv)
-
-Bot-specific parameters are defined in CSV format:
-
-```csv
-name,value,description,category
-# LLM Configuration
-llm-provider,local,LLM service provider,llm
-llm-model,./models/llama2-7b.gguf,Model file path,llm
-llm-context,4096,Context window size,llm
-llm-temperature,0.7,Sampling temperature,llm
-llm-max-tokens,2048,Maximum response tokens,llm
-
-# Context Management
-context-compaction,auto,Compaction strategy,context
-context-max-messages,50,Message history limit,context
-context-summary-threshold,1000,Summary trigger tokens,context
-
-# Knowledge Base
-kb-chunk-size,512,Document chunk size,knowledge
-kb-chunk-overlap,50,Chunk overlap tokens,knowledge
-kb-relevance-threshold,0.7,Minimum similarity score,knowledge
-kb-max-results,5,Maximum search results,knowledge
-
-# Session Management
-session-timeout,3600,Session timeout (seconds),session
-session-persistence,true,Persist sessions to disk,session
-session-encryption,true,Encrypt session data,session
-```
-
-## Operational Procedures
-
-### System Monitoring
-
-Monitor system health through built-in endpoints:
-
-```bash
-# Check component status
-$ ./botserver status
-Component Status CPU Memory Uptime
-─────────────────────────────────────────────────
-PostgreSQL Running 2.1% 487 MB 4h 32m
-Cache Running 0.8% 156 MB 4h 32m
-Storage Running 1.4% 234 MB 4h 32m
-Vectors Running 3.2% 892 MB 4h 32m
-LLM Server Running 45.6% 3.9 GB 4h 31m
-Web Server Running 5.3% 312 MB 4h 31m
-
-# View metrics
-$ ./botserver metrics
-Metric Value Change (1h)
-──────────────────────────────────────────────────
-Total Conversations 1,247 +82
-Active Sessions 34 +5
-Messages Processed 15,823 +1,247
-Knowledge Queries 3,421 +234
-Tool Invocations 892 +67
-Average Response Time 342ms -12ms
-Cache Hit Rate 87.3% +2.1%
-Vector Search Latency 23ms -3ms
-```
-
-### Backup and Recovery
-
-The system implements comprehensive backup strategies:
-
-#### Automated Backups
-
-```bash
-# Configure automated backups
-$ ./botserver backup configure
-Backup Schedule: Daily at 02:00 UTC
-Retention Policy: 30 days
-Destination: ./backups/
-Compression: Enabled (zstd)
-Encryption: Enabled (AES-256)
-
-# Manual backup
-$ ./botserver backup create
-Creating backup snapshot...
-[Database] Dumping PostgreSQL... 234 MB
-[Vectors] Exporting collections... 567 MB
-[Storage] Copying objects... 1.2 GB
-[Config] Saving configuration... 12 KB
-Compressing backup... 689 MB (42% ratio)
-Backup created: backup-20240315-143022.tar.zst
-```
-
-#### Disaster Recovery
-
-```bash
-# Restore from backup
-$ ./botserver restore backup-20240315-143022.tar.zst
-Extracting backup archive...
-Stopping services...
-[Database] Restoring PostgreSQL...
-[Vectors] Importing collections...
-[Storage] Restoring objects...
-[Config] Applying configuration...
-Starting services...
-Restoration completed successfully
-```
-
-### Performance Optimization
-
-#### Database Tuning
-
-```sql
--- Analyze query performance
-EXPLAIN ANALYZE
-SELECT * FROM conversations
-WHERE bot_id = $1 AND created_at > $2
-ORDER BY created_at DESC
-LIMIT 100;
-
--- Create optimized indexes
-CREATE INDEX idx_conversations_bot_created
-ON conversations(bot_id, created_at DESC);
-
--- Update statistics
-ANALYZE conversations;
-```
-
-#### Cache Optimization
-
-```bash
-# Monitor cache performance
-$ ./botserver cache stats
-Keyspace hits: 127,834
-Keyspace misses: 14,234
-Hit ratio: 89.97%
-Evicted keys: 3,421
-Memory used: 1.4 GB / 2.0 GB
-```
-
-#### Vector Index Tuning
-
-```python
-# Optimize HNSW parameters
-{
- "index_type": "hnsw",
- "parameters": {
- "m": 32, # Connectivity parameter
- "ef_construction": 200, # Construction time quality
- "ef_runtime": 100, # Search time quality
- "metric": "cosine" # Distance metric
- }
-}
-```
-
-## Container Deployment Architecture
-
-### LXC Container Isolation
-
-The platform supports Linux Container (LXC) deployment for enhanced isolation:
-
-```bash
-$ ./botserver --container
-[LXC] Creating container network...
-[LXC] Network: 10.10.10.0/24
-[LXC] Creating containers...
-├── botserver-postgres (10.10.10.2)
-├── botserver-cache (10.10.10.3)
-├── botserver-storage (10.10.10.4)
-├── botserver-vectors (10.10.10.5)
-└── botserver-app (10.10.10.6)
-[LXC] Configuring inter-container communication...
-[LXC] Setting resource limits...
-[LXC] Starting containers...
-[LXC] All containers operational ✓
-```
-
-### Container Resource Management
-
-```yaml
-# Container resource limits
-postgres:
- cpu: 2
- memory: 4GB
- storage: 100GB
-
-cache:
- cpu: 1
- memory: 2GB
- storage: 10GB
-
-storage:
- cpu: 2
- memory: 2GB
- storage: 500GB
-
-vectors:
- cpu: 2
- memory: 4GB
- storage: 50GB
-
-app:
- cpu: 4
- memory: 8GB
- storage: 20GB
-```
-
-## Security Considerations
-
-### Network Security
-
-The platform implements defense-in-depth network security:
-
-1. **Transport Layer Security**: All inter-service communication uses TLS 1.3
-2. **Network Segmentation**: Services operate in isolated network namespaces
-3. **Firewall Rules**: Automatic iptables/nftables configuration
-4. **Rate Limiting**: Connection and request throttling
-5. **DDoS Protection**: SYN flood mitigation and connection limits
-
-### Data Protection
-
-Comprehensive data protection mechanisms include:
-
-1. **Encryption at Rest**: AES-256 for database and storage
-2. **Key Management**: Hardware security module (HSM) integration
-3. **Access Control**: Role-based permissions with audit logging
-4. **Data Anonymization**: PII detection and masking
-5. **Compliance**: GDPR, HIPAA, SOC 2 support
-
-## Troubleshooting Guide
-
-### Common Issues and Resolutions
-
-#### Port Conflicts
-
-```bash
-# Error: Address already in use
-$ lsof -i :8080
-COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
-nginx 1234 www 6u IPv4 12345 0t0 TCP *:8080
-
-# Resolution
-$ HTTP_PORT=3000 ./botserver
-# Or stop conflicting service
-$ sudo systemctl stop nginx
-```
-
-#### Memory Limitations
-
-```bash
-# Error: Cannot allocate memory
-# Check system limits
-$ ulimit -a
-$ sysctl vm.overcommit_memory
-
-# Increase limits
-$ ulimit -n 65536 # File descriptors
-$ ulimit -u 32768 # Processes
-$ sudo sysctl vm.overcommit_memory=1
-```
-
-#### Database Connection Issues
-
-```bash
-# Error: Connection refused
-# Check PostgreSQL status
-$ ./botserver status postgres
-PostgreSQL: Stopped
-
-# View logs
-$ tail -f botserver-stack/postgres/logs/postgresql.log
-
-# Restart service
-$ ./botserver restart postgres
-```
-
-### Diagnostic Commands
-
-```bash
-# System diagnostics
-$ ./botserver diagnose
-Running system diagnostics...
-✓ CPU architecture compatible
-✓ Memory sufficient (12.4 GB available)
-✓ Storage adequate (234 GB free)
-✓ Network connectivity verified
-✓ DNS resolution working
-✓ Time synchronization accurate
-⚠ Swap disabled (recommended: 8 GB)
-✓ Kernel parameters optimal
-✓ File descriptors adequate (65536)
-
-# Component health checks
-$ ./botserver health
-Service Endpoint Status Latency
-────────────────────────────────────────────────────
-PostgreSQL localhost:5432 Healthy 2ms
-Cache localhost:6379 Healthy <1ms
-Storage localhost:9090 Healthy 5ms
-Vectors localhost:9333 Healthy 3ms
-LLM localhost:8081 Healthy 45ms
-Application localhost:8080 Healthy 12ms
-```
-
-## Performance Metrics and Benchmarks
-
-### System Performance Characteristics
-
-| Operation | Latency (p50) | Latency (p99) | Throughput | Resource Usage |
-|-----------|---------------|---------------|------------|----------------|
-| Message Processing | 42ms | 156ms | 2,400/sec | CPU: 15%, RAM: 1.2GB |
-| Knowledge Query | 78ms | 234ms | 450/sec | CPU: 25%, RAM: 2.1GB |
-| Tool Invocation | 156ms | 512ms | 180/sec | CPU: 35%, RAM: 1.8GB |
-| Document Indexing | 2.3s/doc | 5.6s/doc | 25 docs/min | CPU: 45%, RAM: 3.2GB |
-| Session Creation | 12ms | 34ms | 5,000/sec | CPU: 8%, RAM: 0.5GB |
-
-### Scalability Characteristics
-
-The platform demonstrates linear scalability characteristics:
-
-- **Horizontal Scaling**: Add nodes for increased capacity
-- **Vertical Scaling**: Utilize additional CPU/RAM resources
-- **Load Distribution**: Automatic request balancing
-- **Session Affinity**: Sticky sessions for stateful operations
-- **Cache Coherence**: Distributed cache synchronization
-
-## Integration Capabilities
-
-### API Integration Patterns
-
-The platform supports multiple integration patterns:
-
-#### REST API Integration
-```bash
-# Create conversation
-POST /api/conversations
-{
- "bot_id": "my-bot",
- "user_id": "user-123",
- "message": "Hello, I need help"
-}
-
-# Response
-{
- "conversation_id": "conv-789",
- "session_id": "sess-456",
- "response": "Hello! I'm here to help. What do you need assistance with?",
- "suggested_actions": ["Product Information", "Technical Support", "Billing"]
-}
-```
-
-#### WebSocket Real-time Communication
-```javascript
-// Client connection
-const ws = new WebSocket('ws://localhost:8080/ws');
-
-ws.on('message', (data) => {
- const message = JSON.parse(data);
- console.log('Bot response:', message.text);
-});
-
-ws.send(JSON.stringify({
- type: 'message',
- text: 'What are your business hours?'
-}));
-```
-
-#### Webhook Event Notifications
-```yaml
-webhooks:
- - url: https://api.company.com/bot-events
- events:
- - conversation.started
- - conversation.ended
- - tool.invoked
- - error.occurred
- headers:
- Authorization: Bearer ${WEBHOOK_SECRET}
-```
-
-## Advanced Configuration
-
-### Multi-Tenant Deployment
-
-Configure isolated bot instances for multiple tenants:
-
-```csv
-name,value,category
-tenant-mode,enabled,system
-tenant-isolation,strict,system
-tenant-database-prefix,tenant_,system
-tenant-storage-bucket,tenant-{id},system
-tenant-resource-limits,true,system
-tenant-max-conversations,1000,limits
-tenant-max-storage,10GB,limits
-tenant-max-compute,4CPU,limits
-```
-
-### High Availability Configuration
-
-Implement redundancy for production deployments:
-
-```yaml
-cluster:
- mode: active-passive
- nodes:
- - primary: bot1.company.com
- - standby: bot2.company.com
-
- replication:
- type: streaming
- lag_threshold: 100ms
-
- failover:
- automatic: true
- timeout: 30s
-
- load_balancing:
- algorithm: least_connections
- health_check: /health
- interval: 10s
-```
-
-## Compliance and Governance
-
-### Audit Logging
-
-Comprehensive audit trail for all system operations:
-
-```json
-{
- "timestamp": "2024-03-15T14:30:22.123Z",
- "event_type": "conversation.message",
- "actor": {
- "type": "user",
- "id": "user-123",
- "ip": "192.168.1.100"
- },
- "resource": {
- "type": "conversation",
- "id": "conv-789",
- "bot_id": "my-bot"
- },
- "action": "create",
- "result": "success",
- "metadata": {
- "message_length": 45,
- "tools_used": ["scheduler"],
- "knowledge_bases": ["policies
\ No newline at end of file
+
## System Requirements
diff --git a/docs/src/chapter-01/quick-start.md b/docs/src/chapter-01/quick-start.md
index 172a9ac89..c497747c9 100644
--- a/docs/src/chapter-01/quick-start.md
+++ b/docs/src/chapter-01/quick-start.md
@@ -46,7 +46,7 @@ Start chatting with your bot!
### Bootstrap Flow
-
+
The **automatic bootstrap** process:
@@ -79,7 +79,7 @@ DRIVE_SECRET=my-secret-key
### How Tools Work
-
+
Tools are just `.bas` files. Create `enrollment.bas`:
diff --git a/docs/src/chapter-01/sessions.md b/docs/src/chapter-01/sessions.md
index 020305525..9d380ea85 100644
--- a/docs/src/chapter-01/sessions.md
+++ b/docs/src/chapter-01/sessions.md
@@ -40,7 +40,7 @@ curl -H "Authorization: Bearer secret-token" \
## Session Lifecycle
```
-
+
```
### Timeouts
@@ -82,7 +82,7 @@ session:uuid:last_activity
### Data Flow Diagram
```
-
+
```
## Session Variables
diff --git a/docs/src/chapter-02/README.md b/docs/src/chapter-02/README.md
index 5d2c642b4..79ff19121 100644
--- a/docs/src/chapter-02/README.md
+++ b/docs/src/chapter-02/README.md
@@ -1,942 +1,76 @@
-# Chapter 02: Package System Architecture and Component Reference
+# Chapter 02: About Packages
-The General Bots package system implements a modular, template-based architecture for organizing and deploying conversational AI applications. This chapter provides comprehensive technical documentation on package structure, component types, lifecycle management, and best practices for enterprise deployment.
+How bots are organized in General Bots.
-## System Overview and Design Philosophy
+## What You'll Learn
-### Architectural Principles
+- Package structure (`.gbai` folders)
+- Dialog scripts in BASIC
+- Knowledge bases
+- Configuration basics
+- How packages load
-The package system adheres to fundamental design principles that ensure scalability, maintainability, and operational efficiency:
+## Package Structure
-1. **Convention over Configuration**: Packages follow strict naming conventions and directory structures, eliminating configuration overhead while maintaining flexibility for customization.
-
-2. **Component Isolation**: Each package component operates independently with well-defined interfaces, enabling parallel development and testing workflows.
-
-3. **Resource Virtualization**: Package resources are abstracted from physical storage, allowing transparent migration between development, staging, and production environments.
-
-4. **Lazy Loading**: Components load on-demand to optimize memory utilization and reduce startup latency.
-
-5. **Immutable Deployments**: Package deployments are versioned and immutable, ensuring reproducible behavior and simplified rollback procedures.
-
-### Package Architecture Layers
-
-The system implements a multi-layered architecture for package management:
+A bot is just a folder ending in `.gbai`:
```
-┌─────────────────────────────────────────────────────────┐
-│ Application Layer │
-│ (Bot Logic, Conversations, UI) │
-├─────────────────────────────────────────────────────────┤
-│ Package Layer │
-│ (.gbai, .gbdialog, .gbkb, .gbot, .gbtheme) │
-├─────────────────────────────────────────────────────────┤
-│ Resource Manager Layer │
-│ (File System, Object Storage, Vector Database) │
-├─────────────────────────────────────────────────────────┤
-│ Infrastructure Layer │
-│ (Database, Cache, Message Queue, Search) │
-└─────────────────────────────────────────────────────────┘
+my-bot.gbai/
+├── my-bot.gbdialog/ # BASIC scripts
+├── my-bot.gbkb/ # Documents
+├── my-bot.gbot/ # Configuration
+├── my-bot.gbtheme/ # Optional styling
+└── my-bot.gbdrive/ # Optional storage
```
-## Package Component Types - Detailed Specification
+Drop the folder in `templates/`, it loads automatically.
-### Component Type Matrix
+## Key Concepts
-| Component | Extension | Primary Function | Storage Location | Processing Model | Caching Strategy |
-|-----------|-----------|-----------------|------------------|------------------|------------------|
-| Application Interface | `.gbai` | Package container and metadata management | File system + Database | Synchronous initialization | Metadata cached indefinitely |
-| Dialog Scripts | `.gbdialog` | Conversation flow orchestration | Object storage | Interpreted at runtime | Script AST cached per session |
-| Knowledge Bases | `.gbkb` | Document storage and semantic search | Vector DB + Object storage | Asynchronous indexing | Embeddings cached permanently |
-| Bot Configuration | `.gbot` | Runtime parameters and settings | Database | Loaded at startup | Configuration cached until restart |
-| UI Themes | `.gbtheme` | Visual styling and branding | Object storage | Loaded per request | CSS cached with ETags |
-| File Storage | `.gbdrive` | General-purpose file management | Object storage | On-demand access | Files cached based on LRU policy |
+### Dialogs (.gbdialog)
+- BASIC scripts that control conversation
+- Must have `start.bas` as entry point
+- Simple commands like TALK and HEAR
-### Component Specifications
+### Knowledge Base (.gbkb)
+- Put PDFs and documents in folders
+- Automatically becomes searchable
+- Bot can answer questions from documents
-#### .gbai - Application Interface Container
+### Configuration (.gbot)
+- Single `config.csv` file
+- Simple name,value pairs
+- Missing values use defaults
-The `.gbai` directory serves as the root container for all bot resources:
+### Themes (.gbtheme)
+- Optional CSS styling
+- Most bots don't need this
-```
-Technical Specifications:
-├── Format: Directory with .gbai extension
-├── Naming: Lowercase alphanumeric with hyphens
-├── Location: templates/ directory
-├── Discovery: Automatic during bootstrap
-├── Validation: Checked for required subdirectories
-└── Metadata: Stored in database with UUID
-```
+### Storage (.gbdrive)
+- Links to S3-compatible storage
+- For large files and uploads
-**Directory Structure Requirements:**
-```
-bot-name.gbai/ # Root container (required)
-├── bot-name.gbdialog/ # Dialog scripts (required)
-│ └── start.bas # Entry point (required)
-├── bot-name.gbkb/ # Knowledge base (optional)
-│ └── [collection-name]/ # Document collections
-├── bot-name.gbot/ # Configuration (optional)
-│ └── config.csv # Settings file
-├── bot-name.gbtheme/ # Theming (optional)
-│ └── default.css # Style definitions
-└── bot-name.gbdrive/ # File storage (optional)
- └── [arbitrary structure] # User-defined organization
-```
+## How It Works
-#### .gbdialog - BASIC Script Processing Engine
+1. **Discovery**: Finds `.gbai` folders
+2. **Loading**: Reads all components
+3. **Indexing**: Processes documents
+4. **Activation**: Bot is ready
-Dialog components implement conversation logic through BASIC scripts:
+No build process. No compilation. Just folders and files.
-```
-Processing Pipeline:
-1. Script Discovery → File system scan for *.bas files
-2. Syntax Validation → Parser verification of BASIC syntax
-3. AST Generation → Abstract syntax tree construction
-4. Optimization → Dead code elimination, constant folding
-5. Registration → Script registration in execution context
-6. Execution → Runtime interpretation with context injection
-```
+## Topics Covered
-**Script File Specifications:**
-```basic
-' File: example.bas
-' Encoding: UTF-8
-' Line endings: LF (Unix) or CRLF (Windows)
-' Maximum file size: 1 MB
-' Maximum line length: 1024 characters
-' Maximum script complexity: 10,000 AST nodes
+- [.gbai Architecture](./gbai.md) - Package details
+- [.gbdialog Dialogs](./gbdialog.md) - BASIC scripting
+- [.gbkb Knowledge Base](./gbkb.md) - Document management
+- [.gbot Configuration](./gbot.md) - Settings
+- [.gbtheme UI Theming](./gbtheme.md) - Styling
+- [.gbdrive File Storage](./gbdrive.md) - Storage integration
+- [Bot Templates](./templates.md) - Example bots
-' Metadata declarations (optional)
-META AUTHOR "Engineering Team"
-META VERSION "1.0.0"
-META DESCRIPTION "Customer service dialog"
+---
-' Parameter declarations for tools
-PARAM customer_id AS STRING
-PARAM issue_type AS STRING
-
-' Main script body
-MAIN:
- ' Script implementation
- TALK "Processing customer request..."
-```
-
-#### .gbkb - Knowledge Base and Vector Search System
-
-Knowledge base packages implement semantic document search capabilities:
-
-```
-Document Processing Workflow:
-┌──────────────┐ ┌──────────────┐ ┌──────────────┐
-│ Document │────▶│ Chunking │────▶│ Embedding │
-│ Ingestion │ │ Pipeline │ │ Generation │
-└──────────────┘ └──────────────┘ └──────────────┘
- │ │ │
- ▼ ▼ ▼
-┌──────────────┐ ┌──────────────┐ ┌──────────────┐
-│ Metadata │ │ Index │ │ Vector │
-│ Extraction │ │ Building │ │ Storage │
-└──────────────┘ └──────────────┘ └──────────────┘
-```
-
-**Collection Configuration:**
-```yaml
-# Collection metadata (auto-generated)
-collection_name: technical_documentation
-document_count: 1,247
-total_chunks: 15,823
-embedding_model: all-MiniLM-L6-v2
-vector_dimensions: 384
-index_type: HNSW
-distance_metric: cosine
-chunk_size: 512
-chunk_overlap: 50
-```
-
-**Supported Document Formats and Processing:**
-
-| Format | MIME Type | Parser | Max Size | Features |
-|--------|-----------|--------|----------|----------|
-| PDF | application/pdf | Apache PDFBox | 100 MB | Text, images, metadata |
-| Word | application/vnd.openxmlformats | Apache POI | 50 MB | Formatted text, tables |
-| Excel | application/vnd.ms-excel | Apache POI | 25 MB | Sheets, formulas |
-| PowerPoint | application/vnd.ms-powerpoint | Apache POI | 50 MB | Slides, notes |
-| Text | text/plain | Native | 10 MB | Plain text |
-| Markdown | text/markdown | CommonMark | 10 MB | Formatted text |
-| HTML | text/html | JSoup | 10 MB | Structured content |
-| CSV | text/csv | Apache Commons | 100 MB | Tabular data |
-| JSON | application/json | Jackson | 10 MB | Structured data |
-| XML | application/xml | JAXB | 10 MB | Structured data |
-
-#### .gbot - Configuration Management System
-
-The configuration system uses a simple CSV format with name-value pairs:
-
-```csv
-# config.csv - Actual Bot Configuration Format
-# Simple name,value pairs with optional empty rows for visual grouping
-
-name,value
-
-# Server Configuration
-server-host,0.0.0.0
-server-port,8080
-sites-root,/tmp
-
-# LLM Configuration
-llm-key,none
-llm-url,http://localhost:8081
-llm-model,../../../../data/llm/DeepSeek-R1-Distill-Qwen-1.5B-Q3_K_M.gguf
-
-# LLM Cache Settings
-llm-cache,false
-llm-cache-ttl,3600
-llm-cache-semantic,true
-llm-cache-threshold,0.95
-
-# Prompt Configuration
-prompt-compact,4
-
-# MCP Server
-mcp-server,false
-
-# Embedding Configuration
-embedding-url,http://localhost:8082
-embedding-model,../../../../data/llm/bge-small-en-v1.5-f32.gguf
-
-# LLM Server Settings (when running embedded)
-llm-server,false
-llm-server-path,botserver-stack/bin/llm/build/bin
-llm-server-host,0.0.0.0
-llm-server-port,8081
-llm-server-gpu-layers,0
-llm-server-n-moe,0
-llm-server-ctx-size,4096
-llm-server-n-predict,1024
-llm-server-parallel,6
-llm-server-cont-batching,true
-llm-server-mlock,false
-llm-server-no-mmap,false
-
-# Email Configuration
-email-from,from@domain.com
-email-server,mail.domain.com
-email-port,587
-email-user,user@domain.com
-email-pass,
-
-# Custom Database Configuration
-custom-server,localhost
-custom-port,5432
-custom-database,mycustomdb
-custom-username,
-custom-password,
-```
-
-**Key Configuration Parameters:**
-
-| Category | Parameter | Description | Default Value |
-|----------|-----------|-------------|---------------|
-| **Server** | server-host | Binding address for web interface | 0.0.0.0 |
-| | server-port | HTTP port for web interface | 8080 |
-| | sites-root | Directory for generated sites | /tmp |
-| **LLM** | llm-url | LLM server endpoint | http://localhost:8081 |
-| | llm-model | Path to GGUF model file | Relative path to model |
-| | llm-cache | Enable response caching | false |
-| | prompt-compact | Context compaction level (1-5) | 4 |
-| **Embeddings** | embedding-url | Embedding server endpoint | http://localhost:8082 |
-| | embedding-model | Path to embedding model | Relative path to model |
-| **Email** | email-server | SMTP server address | mail.domain.com |
-| | email-port | SMTP port | 587 |
-| | email-from | Sender email address | from@domain.com |
-
-Note: Empty values are acceptable for optional settings. The system uses sensible defaults when values are not provided.
-
-#### .gbtheme - UI Styling and Branding System
-
-Theme packages provide comprehensive visual customization:
-
-```css
-/* default.css - Theme Specification Template */
-
-/* ============================================
- Color System - CSS Custom Properties
- ============================================ */
-:root {
- /* Primary Palette */
- --color-primary: #007bff;
- --color-primary-light: #4da3ff;
- --color-primary-dark: #0056b3;
-
- /* Neutral Palette */
- --color-background: #ffffff;
- --color-surface: #f8f9fa;
- --color-border: #dee2e6;
- --color-text: #212529;
- --color-text-secondary: #6c757d;
-
- /* Semantic Colors */
- --color-success: #28a745;
- --color-warning: #ffc107;
- --color-error: #dc3545;
- --color-info: #17a2b8;
-
- /* Typography System */
- --font-family-base: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto;
- --font-family-mono: "SF Mono", Monaco, "Cascadia Code", monospace;
- --font-size-base: 16px;
- --font-weight-normal: 400;
- --font-weight-bold: 600;
- --line-height-base: 1.5;
-
- /* Spacing System (8px grid) */
- --spacing-xs: 4px;
- --spacing-sm: 8px;
- --spacing-md: 16px;
- --spacing-lg: 24px;
- --spacing-xl: 32px;
- --spacing-xxl: 48px;
-
- /* Layout Breakpoints */
- --breakpoint-sm: 576px;
- --breakpoint-md: 768px;
- --breakpoint-lg: 992px;
- --breakpoint-xl: 1200px;
-
- /* Animation System */
- --transition-base: all 0.2s ease-in-out;
- --animation-fade-in: fadeIn 0.3s ease-in;
- --animation-slide-up: slideUp 0.3s ease-out;
-}
-
-/* ============================================
- Component Styling
- ============================================ */
-
-/* Chat Container */
-.chat-container {
- background: var(--color-background);
- border: 1px solid var(--color-border);
- border-radius: 8px;
- height: calc(100vh - var(--spacing-xxl));
- display: flex;
- flex-direction: column;
- font-family: var(--font-family-base);
-}
-
-/* Message Bubbles */
-.message {
- padding: var(--spacing-md);
- margin: var(--spacing-sm);
- border-radius: 12px;
- max-width: 70%;
- word-wrap: break-word;
- animation: var(--animation-fade-in);
-}
-
-.message-user {
- background: var(--color-primary);
- color: white;
- align-self: flex-end;
- margin-left: auto;
- border-bottom-right-radius: 4px;
-}
-
-.message-bot {
- background: var(--color-surface);
- color: var(--color-text);
- align-self: flex-start;
- border-bottom-left-radius: 4px;
- border: 1px solid var(--color-border);
-}
-
-/* Input Area */
-.input-container {
- display: flex;
- padding: var(--spacing-md);
- border-top: 1px solid var(--color-border);
- background: var(--color-surface);
-}
-
-.chat-input {
- flex: 1;
- padding: var(--spacing-sm) var(--spacing-md);
- border: 1px solid var(--color-border);
- border-radius: 24px;
- font-size: var(--font-size-base);
- transition: var(--transition-base);
-}
-
-.chat-input:focus {
- outline: none;
- border-color: var(--color-primary);
- box-shadow: 0 0 0 3px rgba(0, 123, 255, 0.1);
-}
-
-/* Responsive Design */
-@media (max-width: 768px) {
- .message {
- max-width: 85%;
- }
-
- .chat-container {
- border-radius: 0;
- height: 100vh;
- }
-}
-
-/* Dark Mode Support */
-@media (prefers-color-scheme: dark) {
- :root {
- --color-background: #1a1a1a;
- --color-surface: #2d2d2d;
- --color-border: #404040;
- --color-text: #e0e0e0;
- --color-text-secondary: #a0a0a0;
- }
-}
-
-/* Print Styles */
-@media print {
- .input-container,
- .sidebar,
- .toolbar {
- display: none;
- }
-
- .message {
- break-inside: avoid;
- max-width: 100%;
- }
-}
-```
-
-#### .gbdrive - File Storage and Management System
-
-Drive packages provide structured file storage with versioning:
-
-```
-Storage Architecture:
-┌─────────────────────────────────────────────┐
-│ Application Layer │
-│ (File Operations API) │
-├─────────────────────────────────────────────┤
-│ Abstraction Layer │
-│ (Virtual File System Interface) │
-├─────────────────────────────────────────────┤
-│ Storage Backend │
-│ (S3-Compatible Object Storage) │
-└─────────────────────────────────────────────┘
-```
-
-**File Operations and Metadata:**
-```json
-{
- "file_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "path": "/templates/invoice-template.docx",
- "name": "invoice-template.docx",
- "size": 45678,
- "mime_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
- "created_at": "2024-03-15T10:30:00Z",
- "modified_at": "2024-03-15T14:45:30Z",
- "version": 3,
- "checksum": "sha256:a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3",
- "metadata": {
- "author": "Finance Team",
- "tags": ["template", "invoice", "finance"],
- "permissions": {
- "owner": "user:finance-admin",
- "read": ["group:finance", "group:accounting"],
- "write": ["user:finance-admin", "group:finance-managers"]
- }
- },
- "versions": [
- {
- "version": 1,
- "created_at": "2024-03-01T09:00:00Z",
- "size": 44567,
- "checksum": "sha256:b665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae2"
- },
- {
- "version": 2,
- "created_at": "2024-03-10T11:30:00Z",
- "size": 45123,
- "checksum": "sha256:c665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae4"
- }
- ]
-}
-```
-
-## Package Lifecycle Management
-
-### Lifecycle Phases
-
-The package system implements a comprehensive lifecycle management framework:
-
-```
-┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
-│Development │───▶│ Build │───▶│ Deploy │───▶│ Runtime │
-└────────────┘ └────────────┘ └────────────┘ └────────────┘
- │ │ │ │
- ▼ ▼ ▼ ▼
- ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
- │ Edit │ │ Validate │ │ Upload │ │ Execute │
- │ Test │ │ Package │ │ Index │ │ Monitor │
- │ Debug │ │ Optimize │ │ Cache │ │ Update │
- └──────────┘ └──────────┘ └──────────┘ └──────────┘
-```
-
-### Development Phase
-
-During development, packages undergo iterative refinement:
-
-```bash
-# Development workflow
-$ botserver dev --package my-bot
-[Dev] Starting development server...
-[Dev] Watching: templates/my-bot.gbai/
-[Dev] Hot reload enabled
-[Dev] Debugger listening on port 9229
-[Dev] Development UI: http://localhost:3000
-
-# File change detected
-[Dev] Change detected: my-bot.gbdialog/handler.bas
-[Dev] Reloading dialog scripts...
-[Dev] Validation passed ✓
-[Dev] Scripts reloaded in 127ms
-```
-
-### Build Phase
-
-The build process validates and optimizes package components:
-
-```bash
-# Package build process
-$ botserver build --package my-bot
-[Build] Validating package structure...
-├── Checking required directories... ✓
-├── Validating dialog scripts... ✓
-├── Verifying configuration... ✓
-└── Scanning knowledge base... ✓
-
-[Build] Optimizing resources...
-├── Minifying CSS (saved 2.3 KB)... ✓
-├── Compressing images (saved 156 KB)... ✓
-├── Optimizing documents (saved 1.2 MB)... ✓
-└── Generating manifest... ✓
-
-[Build] Package built successfully
-Output: dist/my-bot.gbai.tar.gz (3.4 MB)
-```
-
-### Deployment Phase
-
-Deployment transfers packages to the runtime environment:
-
-```bash
-# Deployment process
-$ botserver deploy my-bot.gbai.tar.gz
-[Deploy] Uploading package...
-├── Transferring archive (3.4 MB)... ✓
-├── Extracting contents... ✓
-├── Validating integrity... ✓
-└── Updating registry... ✓
-
-[Deploy] Processing components...
-├── Indexing knowledge base (1,247 documents)... ✓
-├── Compiling dialog scripts (23 files)... ✓
-├── Loading configuration... ✓
-├── Applying theme... ✓
-└── Initializing storage... ✓
-
-[Deploy] Creating bot instance...
-├── Bot ID: bot-f47ac10b-58cc
-├── Endpoint: https://bot.company.com/my-bot
-├── Status: Active
-└── Version: 1.2.3
-
-[Deploy] Deployment completed successfully
-```
-
-### Runtime Phase
-
-At runtime, packages are loaded and executed on demand:
-
-```
-Runtime Execution Model:
-┌─────────────────────────────────────────────┐
-│ Request Handler │
-├─────────────────────────────────────────────┤
-│ Package Resolution (by bot_id) │
-├─────────────────────────────────────────────┤
-│ Component Loading │
-│ ┌──────────────┐ ┌──────────────┐ │
-│ │ Scripts │ │ Knowledge │ │
-│ │ (Cached) │ │ (Indexed) │ │
-│ └──────────────┘ └──────────────┘ │
-├─────────────────────────────────────────────┤
-│ Execution Context │
-│ ┌──────────────┐ ┌──────────────┐ │
-│ │ Session │ │ Memory │ │
-│ │ State │ │ Storage │ │
-│ └──────────────┘ └──────────────┘ │
-├─────────────────────────────────────────────┤
-│ Response Generation │
-└─────────────────────────────────────────────┘
-```
-
-## Package Storage Architecture
-
-### Storage Hierarchy
-
-Packages utilize a multi-tier storage architecture:
-
-```
-┌─────────────────────────────────────────────┐
-│ Hot Tier (Memory) │
-│ Active Scripts, Session Data │
-│ Latency: <1ms │
-├─────────────────────────────────────────────┤
-│ Warm Tier (SSD) │
-│ Frequently Accessed Documents │
-│ Latency: 5-10ms │
-├─────────────────────────────────────────────┤
-│ Cold Tier (HDD/Object) │
-│ Archives, Backups, Large Files │
-│ Latency: 50-200ms │
-└─────────────────────────────────────────────┘
-```
-
-### Storage Distribution
-
-Package components are distributed across storage systems:
-
-| Component | Primary Storage | Secondary Storage | Cache Strategy | Retention Policy |
-|-----------|----------------|-------------------|----------------|------------------|
-| Scripts | Object Storage | Memory Cache | LRU with 1h TTL | Versioned, Permanent |
-| Configuration | Database | Memory Cache | Invalidate on Change | Versioned, Permanent |
-| Documents | Vector DB + Object | Disk Cache | LRU with 24h TTL | Permanent |
-| Themes | Object Storage | Browser Cache | ETag Validation | Versioned, Permanent |
-| User Files | Object Storage | None | On-Demand | User-Defined |
-| Session Data | Cache | Database | TTL-Based | Configurable TTL |
-
-## Multi-Bot Architecture
-
-### Tenant Isolation
-
-The system supports complete isolation between bot instances:
-
-```
-Isolation Boundaries:
-┌──────────────────────────────────┐
-│ Bot Instance A │
-│ ┌────────────┬────────────┐ │
-│ │ Database │ Storage │ │
-│ │ Schema A │ Bucket A │ │
-│ └────────────┴────────────┘ │
-│ ┌────────────┬────────────┐ │
-│ │ Cache │ Vectors │ │
-│ │ Namespace A│ Collection A│ │
-│ └────────────┴────────────┘ │
-└──────────────────────────────────┘
-
-┌──────────────────────────────────┐
-│ Bot Instance B │
-│ ┌────────────┬────────────┐ │
-│ │ Database │ Storage │ │
-│ │ Schema B │ Bucket B │ │
-│ └────────────┴────────────┘ │
-│ ┌────────────┬────────────┐ │
-│ │ Cache │ Vectors │ │
-│ │ Namespace B│ Collection B│ │
-│ └────────────┴────────────┘ │
-└──────────────────────────────────┘
-```
-
-### Resource Sharing
-
-While maintaining isolation, bots can share infrastructure:
-
-```yaml
-resource_sharing:
- compute:
- type: shared_pool
- allocation: fair_share
- limits:
- cpu: 4_cores_per_bot
- memory: 8GB_per_bot
-
- storage:
- type: quota_based
- limits:
- database: 10GB_per_bot
- objects: 100GB_per_bot
- vectors: 1M_documents_per_bot
-
- network:
- type: bandwidth_limited
- limits:
- ingress: 100Mbps_per_bot
- egress: 100Mbps_per_bot
-```
-
-## Migration Strategies
-
-### From Legacy Bot Platforms
-
-Migration from traditional bot platforms requires paradigm shift:
-
-#### Traditional Approach (Complex)
-```javascript
-// Legacy: 500+ lines of intent matching
-const intents = {
- 'book_meeting': {
- patterns: [
- 'schedule a meeting',
- 'book an appointment',
- 'set up a call'
- ],
- entities: ['person', 'date', 'time'],
- handler: async (context) => {
- // Complex state machine logic
- if (!context.state.person) {
- return askForPerson(context);
- }
- if (!context.state.date) {
- return askForDate(context);
- }
- // ... 100 more lines
- }
- }
- // ... 50 more intents
-};
-```
-
-#### General Bots Approach (Simple)
-```basic
-' Modern: 10 lines with LLM intelligence
-USE TOOL "scheduler"
-TALK "I can help you schedule meetings."
-
-' The LLM handles:
-' - Intent recognition
-' - Entity extraction
-' - Context management
-' - Error handling
-' - Natural conversation flow
-```
-
-### Migration Checklist
-
-Pre-migration assessment and planning:
-
-```markdown
-## Migration Readiness Assessment
-
-### Data Inventory
-- [ ] Identify all conversation flows
-- [ ] Catalog knowledge documents
-- [ ] List integration points
-- [ ] Document user personas
-- [ ] Map data dependencies
-
-### Technical Evaluation
-- [ ] Review dialog complexity
-- [ ] Assess integration requirements
-- [ ] Evaluate performance needs
-- [ ] Identify security requirements
-- [ ] Plan testing strategy
-
-### Migration Approach
-- [ ] Choose phased or big-bang migration
-- [ ] Define success criteria
-- [ ] Establish rollback plan
-- [ ] Schedule downtime windows
-- [ ] Prepare user communication
-
-### Post-Migration Validation
-- [ ] Verify conversation flows
-- [ ] Test knowledge retrieval
-- [ ] Validate integrations
-- [ ] Measure performance metrics
-- [ ] Conduct user acceptance testing
-```
-
-## Performance Optimization
-
-### Package Loading Optimization
-
-Techniques for optimizing package load times:
-
-```python
-# Lazy loading strategy
-class PackageLoader:
- def __init__(self):
- self.loaded_components = {}
- self.loading_queue = PriorityQueue()
-
- def load_package(self, package_id: str):
- # Load critical components immediately
- self.load_critical_components(package_id)
-
- # Queue non-critical components for background loading
- self.queue_background_loading(package_id)
-
- def load_critical_components(self, package_id: str):
- # Load only what's needed for first response
- components = [
- 'configuration',
- 'start_script',
- 'theme_css'
- ]
- for component in components:
- self.load_component(package_id, component)
-
- def queue_background_loading(self, package_id: str):
- # Load remaining components asynchronously
- components = [
- 'knowledge_base',
- 'additional_scripts',
- 'drive_files'
- ]
- for component in components:
- priority = self.calculate_priority(component)
- self.loading_queue.put((priority, package_id, component))
-```
-
-### Caching Strategies
-
-Multi-level caching for optimal performance:
-
-```yaml
-caching_configuration:
- l1_cache: # CPU Cache
- type: in_process
- size: 100MB
- ttl: 60s
- items:
- - compiled_scripts
- - hot_configuration
-
- l2_cache: # Application Cache
- type: memory
- size: 1GB
- ttl: 3600s
- items:
- - script_ast
- - user_sessions
- - embeddings
-
- l3_cache: # Distributed Cache
- type: redis_cluster
- size: 10GB
- ttl: 86400s
- items:
- - knowledge_chunks
- - document_metadata
- - search_results
-
- l4_cache: # CDN/Edge Cache
- type: cloudflare
- ttl: 604800s
- items:
- - static_assets
- - theme_files
- - public_documents
-```
-
-## Security Considerations
-
-### Package Validation
-
-Comprehensive security validation for packages:
-
-```python
-class PackageValidator:
- def validate(self, package_path: str) -> ValidationResult:
- validations = [
- self.check_structure(),
- self.scan_for_malware(),
- self.validate_scripts(),
- self.check_permissions(),
- self.verify_signatures()
- ]
-
- return ValidationResult(all(validations))
-
- def validate_scripts(self, scripts: List[str]) -> bool:
- """Validate BASIC scripts for security issues"""
- for script in scripts:
- # Check for dangerous operations
- if self.contains_dangerous_operations(script):
- return False
-
- # Validate syntax
- if not self.valid_syntax(script):
- return False
-
- # Check resource limits
- if self.exceeds_complexity_limit(script):
- return False
-
- return True
-```
-
-### Access Control
-
-Role-based access control for package resources:
-
-```yaml
-access_control:
- roles:
- admin:
- permissions:
- - package:create
- - package:read
- - package:update
- - package:delete
- - package:deploy
-
- developer:
- permissions:
- - package:create
- - package:read
- - package:update
- - package:test
-
- operator:
- permissions:
- - package:read
- - package:deploy
- - package:monitor
-
- user:
- permissions:
- - package:use
- - conversation:create
- - file:read
-```
-
-## Monitoring and Diagnostics
-
-### Package Metrics
-
-Real-time monitoring of package performance:
-
-```json
-{
- "package_id": "customer-service-bot",
- "timestamp": "2024-03-15T14:30:00Z",
- "metrics": {
- "performance": {
- "script_execution_time_p50": 45,
- "script_execution_time_p99": 234,
- "knowledge_query_time_p50": 78,
- "knowledge_query_time_p99": 345,
- "response_generation_time_p50": 123,
- "response_generation_time_p99": 567
- },
- "usage": {
- "total_conversations": 1247,
- "active_sessions": 34,
- "messages_processed": 15823,
- "knowledge
\ No newline at end of file
+
## How Bootstrap Finds Bots
@@ -224,7 +224,7 @@ templates/
When BotServer starts:
```
-
+
```
Takes about 5-10 seconds per bot.
diff --git a/docs/src/chapter-03/README.md b/docs/src/chapter-03/README.md
index cecb49603..0a87d5e56 100644
--- a/docs/src/chapter-03/README.md
+++ b/docs/src/chapter-03/README.md
@@ -12,82 +12,11 @@ The knowledge base system transforms unstructured documents into queryable seman
The knowledge base architecture implements a multi-stage pipeline for document processing and retrieval:
-
+
### Technical Specifications
-