diff --git a/README.md b/README.md index 79cc245d..49caa806 100644 --- a/README.md +++ b/README.md @@ -6,30 +6,9 @@ ## Quick Links -- **[Complete Documentation](docs/src/SUMMARY.md)** - Full documentation index -- **[Quick Start Guide](docs/QUICK_START.md)** - Get started in minutes -- **[Changelog](CHANGELOG.md)** - Version history - -## Documentation Structure - -All documentation has been organized into the **[docs/](docs/)** directory: - -### Core Documentation -- **[Introduction & Getting Started](docs/src/chapter-01/README.md)** -- **[Package System](docs/src/chapter-02/README.md)** -- **[Knowledge Base Reference](docs/src/chapter-03/README.md)** -- **[User Interface](docs/src/chapter-04-gbui/README.md)** -- **[BASIC Dialogs](docs/src/chapter-06-gbdialog/README.md)** -- **[Architecture Reference](docs/src/chapter-07-gbapp/README.md)** -- **[Configuration](docs/src/chapter-08-config/README.md)** -- **[REST API Reference](docs/src/chapter-10-api/README.md)** -- **[Security & Authentication](docs/src/chapter-12-auth/README.md)** - -### Technical References -- **[KB & Tools System](docs/src/chapter-03/kb-and-tools.md)** - Core system architecture -- **[Semantic Cache](docs/src/chapter-03/caching.md)** - LLM caching with 70% cost reduction -- **[Universal Messaging](docs/src/chapter-06-gbdialog/universal-messaging.md)** - Multi-channel communication -- **[External Services](docs/src/appendix-external-services/README.md)** - Service integrations +- **[Getting Started](docs/guides/getting-started.md)** - Installation and first bot +- **[API Reference](docs/api/README.md)** - REST and WebSocket endpoints +- **[BASIC Language](docs/reference/basic-language.md)** - Dialog scripting reference ## What is General Bots? @@ -39,45 +18,49 @@ General Bots is a **self-hosted AI automation platform** that provides: - **MCP + LLM Tools Generation** - Instant tool creation from code/functions - **Semantic Caching** - Intelligent response caching (70% cost reduction) - **Web Automation Engine** - Browser automation + AI intelligence -- **External Data APIs** - Integrated services via connectors - **Enterprise Data Connectors** - CRM, ERP, database native integrations - **Git-like Version Control** - Full history with rollback capabilities -- **Contract Analysis** - Legal document review and summary -## Command-Line Options +## Quick Start + +### Prerequisites + +- **Rust** (1.75+) - [Install from rustup.rs](https://rustup.rs/) +- **Git** - [Download from git-scm.com](https://git-scm.com/downloads) + +### Installation ```bash -# Run with default settings (console UI enabled) +git clone https://github.com/GeneralBots/BotServer +cd BotServer cargo run - -# Run without console UI -cargo run -- --noconsole - -# Run in desktop mode (Tauri) -cargo run -- --desktop - -# Run without any UI -cargo run -- --noui - -# Specify tenant -cargo run -- --tenant - -# LXC container mode -cargo run -- --container ``` -### Default Behavior -- **Console UI is enabled by default** - Shows real-time system status, logs, and file browser -- **Minimal UI is served by default** at `http://localhost:8080` - Lightweight, fast-loading interface -- Full suite UI available at `http://localhost:8080/suite` - Complete multi-application interface -- Use `--noconsole` to disable the terminal UI and run as a background service -- The HTTP server always runs on port 8080 unless in desktop mode +On first run, BotServer automatically sets up PostgreSQL, S3 storage, Redis cache, and downloads AI models. +The server will be available at `http://localhost:8080`. + +## Documentation + +``` +docs/ +├── api/ # API documentation +│ ├── README.md # API overview +│ ├── rest-endpoints.md # HTTP endpoints +│ └── websocket.md # Real-time communication +├── guides/ # How-to guides +│ ├── getting-started.md # Quick start +│ ├── deployment.md # Production setup +│ └── templates.md # Using templates +└── reference/ # Technical reference + ├── basic-language.md # BASIC keywords + ├── configuration.md # Config options + └── architecture.md # System design +``` ## Key Features ### 4 Essential Keywords -General Bots provides a minimal, focused system for managing Knowledge Bases and Tools: ```basic USE KB "kb-name" ' Load knowledge base into vector database @@ -86,66 +69,32 @@ USE TOOL "tool-name" ' Make tool available to LLM CLEAR TOOLS ' Remove all tools from session ``` -### Strategic Advantages -- **vs ChatGPT/Claude**: Automates entire business processes, not just chat -- **vs n8n/Make**: Simpler approach with little programming needed -- **vs Microsoft 365**: User control, not locked systems -- **vs Salesforce**: Open-source AI orchestration connecting all systems +### Example Bot -## Quick Start +```basic +' customer-support.bas +USE KB "support-docs" +USE TOOL "create-ticket" +USE TOOL "check-order" -### Prerequisites -- **Rust** (latest stable) - [Install from rustup.rs](https://rustup.rs/) -- **Git** (latest stable) - [Download from git-scm.com](https://git-scm.com/downloads) +SET CONTEXT "support" AS "You are a helpful customer support agent." -### Installation - -```bash -# Clone the repository -git clone https://github.com/GeneralBots/BotServer -cd BotServer - -# Run the server (auto-installs dependencies) -cargo run +TALK "Welcome! How can I help you today?" ``` -On first run, BotServer automatically: -- Installs required components (PostgreSQL, S3-compatible storage, Cache, LLM) -- Sets up database with migrations -- Downloads AI models -- Uploads template bots -- Starts HTTP server at `http://127.0.0.1:8080` - -### Management Commands -```bash -botserver start # Start all components -botserver stop # Stop all components -botserver restart # Restart all components -botserver list # List available components -botserver status # Check component status -``` - -## Current Status - -**Version:** 6.0.8 -**Build Status:** SUCCESS -**Production Ready:** YES -**Compilation:** 0 errors - -## Deployment - -General Bots supports deployment via **LXC containers** for isolated, lightweight virtualization: +## Command-Line Options ```bash -# Deploy with LXC container isolation -cargo run -- --container +cargo run # Default: console UI + web server +cargo run -- --noconsole # Background service mode +cargo run -- --desktop # Desktop application (Tauri) +cargo run -- --tenant # Specify tenant +cargo run -- --container # LXC container mode ``` -See [Container Deployment](docs/src/chapter-07-gbapp/containers.md) for detailed LXC setup instructions. - ## Environment Variables -General Bots uses minimal environment configuration. Only Directory service variables are required: +Only directory service variables are required: | Variable | Purpose | |----------|---------| @@ -153,23 +102,32 @@ General Bots uses minimal environment configuration. Only Directory service vari | `DIRECTORY_CLIENT_ID` | OAuth client ID | | `DIRECTORY_CLIENT_SECRET` | OAuth client secret | -All service credentials (database, storage, cache) are managed automatically by the Directory service. Application configuration is done through `config.csv` files in each bot's `.gbot` folder. +All service credentials are managed automatically. See [Configuration](docs/reference/configuration.md) for details. -See [Environment Variables](docs/src/appendix-env-vars/README.md) for details. +## Current Status + +**Version:** 6.0.8 +**Build Status:** SUCCESS +**Production Ready:** YES + +## Deployment + +See [Deployment Guide](docs/guides/deployment.md) for: + +- Single server setup +- Docker Compose +- LXC containers +- Kubernetes +- Reverse proxy configuration ## Contributing -We welcome contributions! Please read: -- **[Contributing Guidelines](docs/src/chapter-13-community/README.md)** -- **[Development Setup](docs/src/chapter-13-community/setup.md)** -- **[Testing Guide](docs/src/chapter-13-community/testing.md)** +We welcome contributions! Please read our contributing guidelines before submitting PRs. ## Security Security issues should be reported to: **security@pragmatismo.com.br** -See **[Security Policy](docs/src/chapter-12-auth/security-policy.md)** for our security guidelines. - ## License General Bot Copyright (c) pragmatismo.com.br. All rights reserved. @@ -177,37 +135,12 @@ Licensed under the **AGPL-3.0**. According to our dual licensing model, this program can be used either under the terms of the GNU Affero General Public License, version 3, or under a proprietary license. -See [LICENSE](LICENSE) for details. +## Support -## Key Facts - -- LLM Orchestrator AGPL licensed (contribute back for custom-label SaaS) -- True community governance -- No single corporate control -- 5+ years of stability -- Never changed license -- Enterprise-grade -- Hosted locally or multicloud - -## Support & Resources - -- **Documentation:** [docs.pragmatismo.com.br](https://docs.pragmatismo.com.br) -- **GitHub:** [github.com/GeneralBots/BotServer](https://github.com/GeneralBots/BotServer) +- **GitHub Issues:** [github.com/GeneralBots/BotServer/issues](https://github.com/GeneralBots/BotServer/issues) - **Stack Overflow:** Tag questions with `generalbots` - **Video Tutorial:** [7 AI General Bots LLM Templates](https://www.youtube.com/watch?v=KJgvUPXi3Fw) -## Demo - -See conversational data analytics in action: - -```basic -TALK "General Bots Labs presents FISCAL DATA SHOW BY BASIC" -result = GET "https://api.fiscaldata.treasury.gov/services/api/..." -data = SELECT YEAR(record_date) as Yr, SUM(...) AS Amount FROM data -img = CHART "bar", data -SEND FILE img -``` - ## Contributors @@ -216,8 +149,6 @@ SEND FILE img --- -**General Bots Code Name:** [Guaribas](https://en.wikipedia.org/wiki/Guaribas) (a city in Brazil, state of Piauí) +**General Bots Code Name:** [Guaribas](https://en.wikipedia.org/wiki/Guaribas) -> "No one should have to do work that can be done by a machine." - Roberto Mangabeira Unger - -Ask a question | Read the Docs \ No newline at end of file +> "No one should have to do work that can be done by a machine." - Roberto Mangabeira Unger \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..4d2fda9a --- /dev/null +++ b/docs/README.md @@ -0,0 +1,62 @@ +# General Bots Documentation + +Welcome to the General Bots documentation. This guide covers everything you need to build, deploy, and manage AI-powered bots. + +## Quick Navigation + +| Section | Description | +|---------|-------------| +| [Getting Started](guides/getting-started.md) | Installation and first bot | +| [API Reference](api/README.md) | REST endpoints and WebSocket | +| [BASIC Language](reference/basic-language.md) | Dialog scripting reference | +| [Configuration](reference/configuration.md) | Environment and settings | + +## Documentation Structure + +``` +docs/ +├── api/ # API documentation +│ ├── README.md # API overview +│ ├── rest-endpoints.md # HTTP endpoints +│ └── websocket.md # Real-time communication +├── guides/ # How-to guides +│ ├── getting-started.md # Quick start +│ ├── deployment.md # Production setup +│ └── templates.md # Using templates +└── reference/ # Technical reference + ├── basic-language.md # BASIC keywords + ├── configuration.md # Config options + └── architecture.md # System design +``` + +## Core Concepts + +### Knowledge Bases (KB) +Store documents, FAQs, and data that the AI can search and reference: +```basic +USE KB "company-docs" +``` + +### Tools +Functions the AI can call to perform actions: +```basic +USE TOOL "send-email" +USE TOOL "create-ticket" +``` + +### Dialogs +BASIC scripts that define conversation flows and automation: +```basic +TALK "Hello! How can I help?" +answer = HEAR +``` + +## Quick Links + +- **[GitHub Repository](https://github.com/GeneralBots/BotServer)** +- **[Issue Tracker](https://github.com/GeneralBots/BotServer/issues)** +- **[Contributing Guide](../CONTRIBUTING.md)** + +## Version + +This documentation covers **General Bots v6.x**. \ No newline at end of file diff --git a/docs/api/README.md b/docs/api/README.md new file mode 100644 index 00000000..4e76e86d --- /dev/null +++ b/docs/api/README.md @@ -0,0 +1,104 @@ +# API Reference + +General Bots exposes a REST API and WebSocket interface for integration with external systems. + +## Base URL + +``` +http://localhost:8080/api +``` + +## Authentication + +All API requests require authentication via Bearer token: + +```bash +curl -H "Authorization: Bearer " \ + http://localhost:8080/api/sessions +``` + +## Endpoints Overview + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/sessions` | GET | List active sessions | +| `/api/sessions` | POST | Create new session | +| `/api/sessions/:id` | GET | Get session details | +| `/api/sessions/:id/messages` | POST | Send message | +| `/api/drive/*` | * | File storage operations | +| `/api/tasks/*` | * | Task management | +| `/api/email/*` | * | Email operations | +| `/api/calendar/*` | * | Calendar/CalDAV | +| `/api/meet/*` | * | Video meetings | +| `/api/kb/*` | * | Knowledge base search | +| `/api/analytics/*` | * | Analytics dashboard | + +## WebSocket + +Real-time communication via WebSocket: + +``` +ws://localhost:8080/ws +``` + +### Connection + +```javascript +const ws = new WebSocket('ws://localhost:8080/ws'); +ws.onmessage = (event) => { + const data = JSON.parse(event.data); + console.log('Received:', data); +}; +``` + +### Message Format + +```json +{ + "type": "message", + "session_id": "uuid", + "content": "Hello bot", + "timestamp": "2024-01-01T00:00:00Z" +} +``` + +## Response Format + +All responses follow a consistent structure: + +### Success + +```json +{ + "success": true, + "data": { ... } +} +``` + +### Error + +```json +{ + "success": false, + "error": { + "code": "NOT_FOUND", + "message": "Resource not found" + } +} +``` + +## Rate Limiting + +API requests are rate limited per IP: + +| Endpoint Type | Requests/Second | Burst | +|--------------|-----------------|-------| +| Standard API | 100 | 200 | +| Auth endpoints | 10 | 20 | +| LLM endpoints | 5 | 10 | + +## Detailed Documentation + +- [REST Endpoints](rest-endpoints.md) - Complete endpoint reference +- [WebSocket API](websocket.md) - Real-time communication +- [HTMX Integration](htmx.md) - Frontend patterns \ No newline at end of file diff --git a/docs/api/rest-endpoints.md b/docs/api/rest-endpoints.md new file mode 100644 index 00000000..261c7b82 --- /dev/null +++ b/docs/api/rest-endpoints.md @@ -0,0 +1,599 @@ +# REST API Endpoints + +Complete reference for all General Bots REST API endpoints. + +## Sessions + +### List Sessions + +```http +GET /api/sessions +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `limit` | integer | Max results (default: 50) | +| `offset` | integer | Pagination offset | +| `status` | string | Filter by status: `active`, `closed` | + +**Response:** + +```json +{ + "sessions": [ + { + "id": "550e8400-e29b-41d4-a716-446655440000", + "user_id": "user@example.com", + "bot_id": "default", + "status": "active", + "created_at": "2024-01-01T10:00:00Z", + "last_activity": "2024-01-01T10:30:00Z" + } + ], + "total": 1, + "limit": 50, + "offset": 0 +} +``` + +### Create Session + +```http +POST /api/sessions +``` + +**Request Body:** + +```json +{ + "bot_id": "default", + "user_id": "user@example.com", + "metadata": { + "channel": "web", + "language": "en" + } +} +``` + +### Get Session + +```http +GET /api/sessions/:id +``` + +### Send Message + +```http +POST /api/sessions/:id/messages +``` + +**Request Body:** + +```json +{ + "content": "Hello, I need help", + "type": "text" +} +``` + +**Response:** + +```json +{ + "id": "msg-uuid", + "session_id": "session-uuid", + "role": "assistant", + "content": "Hello! How can I assist you today?", + "timestamp": "2024-01-01T10:31:00Z" +} +``` + +--- + +## Drive (File Storage) + +### List Files + +```http +GET /api/drive +GET /api/drive/:path +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `recursive` | boolean | Include subdirectories | +| `type` | string | Filter: `file`, `folder` | + +### Upload File + +```http +POST /api/drive/:path +Content-Type: multipart/form-data +``` + +### Download File + +```http +GET /api/drive/:path/download +``` + +### Delete File + +```http +DELETE /api/drive/:path +``` + +--- + +## Tasks + +### List Tasks + +```http +GET /api/tasks +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `status` | string | `pending`, `completed`, `cancelled` | +| `priority` | string | `low`, `medium`, `high` | +| `due_before` | datetime | Filter by due date | + +### Create Task + +```http +POST /api/tasks +``` + +**Request Body:** + +```json +{ + "title": "Follow up with client", + "description": "Send proposal document", + "due_date": "2024-01-15T17:00:00Z", + "priority": "high", + "assignee": "user@example.com" +} +``` + +### Update Task + +```http +PUT /api/tasks/:id +``` + +### Delete Task + +```http +DELETE /api/tasks/:id +``` + +--- + +## Email + +### List Emails + +```http +GET /api/email +GET /api/email/:folder +``` + +**Folders:** `inbox`, `sent`, `drafts`, `trash` + +### Send Email + +```http +POST /api/email/send +``` + +**Request Body:** + +```json +{ + "to": ["recipient@example.com"], + "cc": [], + "bcc": [], + "subject": "Meeting Tomorrow", + "body": "Hi, let's meet at 3pm.", + "html": false, + "attachments": [] +} +``` + +### Get Email + +```http +GET /api/email/:id +``` + +### Delete Email + +```http +DELETE /api/email/:id +``` + +--- + +## Calendar + +### List Events + +```http +GET /api/calendar/events +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `start` | datetime | Range start | +| `end` | datetime | Range end | +| `calendar_id` | string | Specific calendar | + +### Create Event + +```http +POST /api/calendar/events +``` + +**Request Body:** + +```json +{ + "title": "Team Meeting", + "start_time": "2024-01-15T14:00:00Z", + "end_time": "2024-01-15T15:00:00Z", + "location": "Conference Room A", + "attendees": ["team@example.com"], + "recurrence": null +} +``` + +### Export Calendar (iCal) + +```http +GET /api/calendar/export.ics +``` + +### Import Calendar + +```http +POST /api/calendar/import +Content-Type: text/calendar +``` + +--- + +## Meet (Video Conferencing) + +### Create Room + +```http +POST /api/meet/rooms +``` + +**Request Body:** + +```json +{ + "name": "Team Standup", + "scheduled_start": "2024-01-15T09:00:00Z", + "max_participants": 10 +} +``` + +**Response:** + +```json +{ + "room_id": "room-uuid", + "join_url": "https://meet.example.com/room-uuid", + "token": "participant-token" +} +``` + +### Join Room + +```http +POST /api/meet/rooms/:id/join +``` + +### List Participants + +```http +GET /api/meet/rooms/:id/participants +``` + +--- + +## Knowledge Base + +### Search + +```http +GET /api/kb/search +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `q` | string | Search query (required) | +| `limit` | integer | Max results (default: 10) | +| `threshold` | float | Similarity threshold (0-1) | +| `collection` | string | Specific KB collection | + +**Response:** + +```json +{ + "results": [ + { + "id": "doc-uuid", + "content": "Relevant document content...", + "score": 0.95, + "metadata": { + "source": "company-docs", + "title": "Employee Handbook" + } + } + ], + "query": "vacation policy", + "total": 5 +} +``` + +### List Collections + +```http +GET /api/kb/collections +``` + +### Reindex Collection + +```http +POST /api/kb/reindex +``` + +--- + +## Analytics + +### Dashboard Stats + +```http +GET /api/analytics/stats +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `time_range` | string | `day`, `week`, `month`, `year` | + +**Response:** + +```json +{ + "total_messages": 15420, + "active_sessions": 45, + "avg_response_time_ms": 230, + "error_rate": 0.02, + "top_queries": [ + {"query": "password reset", "count": 120} + ] +} +``` + +### Message Trends + +```http +GET /api/analytics/messages/trend +``` + +--- + +## Paper (Documents) + +### List Documents + +```http +GET /api/paper +``` + +### Create Document + +```http +POST /api/paper +``` + +**Request Body:** + +```json +{ + "title": "Meeting Notes", + "content": "# Meeting Notes\n\n...", + "type": "note" +} +``` + +### Get Document + +```http +GET /api/paper/:id +``` + +### Update Document + +```http +PUT /api/paper/:id +``` + +### Delete Document + +```http +DELETE /api/paper/:id +``` + +--- + +## Designer (Bot Builder) + +### List Dialogs + +```http +GET /api/designer/dialogs +``` + +### Create Dialog + +```http +POST /api/designer/dialogs +``` + +**Request Body:** + +```json +{ + "name": "greeting", + "content": "TALK \"Hello!\"\nanswer = HEAR" +} +``` + +### Validate Dialog + +```http +POST /api/designer/dialogs/:id/validate +``` + +**Response:** + +```json +{ + "valid": true, + "errors": [], + "warnings": ["Line 15: Consider using END IF"] +} +``` + +### Deploy Dialog + +```http +POST /api/designer/dialogs/:id/deploy +``` + +--- + +## Sources (Templates) + +### List Templates + +```http +GET /api/sources/templates +``` + +**Query Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `category` | string | Filter by category | + +### Get Template + +```http +GET /api/sources/templates/:id +``` + +### Use Template + +```http +POST /api/sources/templates/:id/use +``` + +--- + +## Admin + +### System Stats + +```http +GET /api/admin/stats +``` + +**Response:** + +```json +{ + "uptime_seconds": 86400, + "memory_used_mb": 512, + "active_connections": 23, + "database_size_mb": 1024, + "cache_hit_rate": 0.85 +} +``` + +### Health Check + +```http +GET /api/health +``` + +**Response:** + +```json +{ + "status": "healthy", + "components": { + "database": "ok", + "cache": "ok", + "storage": "ok", + "llm": "ok" + } +} +``` + +--- + +## Error Codes + +| Code | HTTP Status | Description | +|------|-------------|-------------| +| `BAD_REQUEST` | 400 | Invalid request parameters | +| `UNAUTHORIZED` | 401 | Missing or invalid auth token | +| `FORBIDDEN` | 403 | Insufficient permissions | +| `NOT_FOUND` | 404 | Resource not found | +| `CONFLICT` | 409 | Resource already exists | +| `RATE_LIMITED` | 429 | Too many requests | +| `INTERNAL_ERROR` | 500 | Server error | + +--- + +## Pagination + +List endpoints support pagination: + +```http +GET /api/tasks?limit=20&offset=40 +``` + +Response includes pagination info: + +```json +{ + "data": [...], + "pagination": { + "total": 150, + "limit": 20, + "offset": 40, + "has_more": true + } +} +``` diff --git a/docs/api/websocket.md b/docs/api/websocket.md new file mode 100644 index 00000000..4f0d7ad9 --- /dev/null +++ b/docs/api/websocket.md @@ -0,0 +1,362 @@ +# WebSocket API + +Real-time bidirectional communication with General Bots. + +## Connection + +### Endpoint + +``` +ws://localhost:8080/ws +wss://your-domain.com/ws (production) +``` + +### Authentication + +Include the auth token as a query parameter or in the first message: + +```javascript +// Option 1: Query parameter +const ws = new WebSocket('ws://localhost:8080/ws?token='); + +// Option 2: First message +ws.onopen = () => { + ws.send(JSON.stringify({ + type: 'auth', + token: '' + })); +}; +``` + +## Message Format + +All messages are JSON objects with a `type` field: + +```json +{ + "type": "message_type", + "payload": { ... }, + "timestamp": "2024-01-01T10:00:00Z" +} +``` + +## Client Messages + +### Send Chat Message + +```json +{ + "type": "message", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "content": "Hello, I need help with my order" +} +``` + +### Start Typing + +```json +{ + "type": "typing_start", + "session_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +### Stop Typing + +```json +{ + "type": "typing_stop", + "session_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +### Subscribe to Session + +```json +{ + "type": "subscribe", + "session_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +### Unsubscribe from Session + +```json +{ + "type": "unsubscribe", + "session_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +### Ping (Keep-Alive) + +```json +{ + "type": "ping" +} +``` + +## Server Messages + +### Chat Response + +```json +{ + "type": "message", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "message_id": "msg-uuid", + "role": "assistant", + "content": "I'd be happy to help! What's your order number?", + "timestamp": "2024-01-01T10:00:01Z" +} +``` + +### Streaming Response + +For LLM responses, content streams in chunks: + +```json +{ + "type": "stream_start", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "message_id": "msg-uuid" +} +``` + +```json +{ + "type": "stream_chunk", + "message_id": "msg-uuid", + "content": "I'd be happy to " +} +``` + +```json +{ + "type": "stream_chunk", + "message_id": "msg-uuid", + "content": "help! What's your " +} +``` + +```json +{ + "type": "stream_end", + "message_id": "msg-uuid", + "content": "I'd be happy to help! What's your order number?" +} +``` + +### Bot Typing Indicator + +```json +{ + "type": "bot_typing", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "is_typing": true +} +``` + +### Tool Execution + +When the bot calls a tool: + +```json +{ + "type": "tool_call", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "tool": "search_orders", + "arguments": {"order_id": "12345"} +} +``` + +```json +{ + "type": "tool_result", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "tool": "search_orders", + "result": {"status": "shipped", "tracking": "1Z999..."} +} +``` + +### Error + +```json +{ + "type": "error", + "code": "SESSION_NOT_FOUND", + "message": "Session does not exist or has expired" +} +``` + +### Pong (Keep-Alive Response) + +```json +{ + "type": "pong", + "timestamp": "2024-01-01T10:00:00Z" +} +``` + +### Session Events + +```json +{ + "type": "session_created", + "session_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +```json +{ + "type": "session_closed", + "session_id": "550e8400-e29b-41d4-a716-446655440000", + "reason": "user_disconnect" +} +``` + +## JavaScript Client Example + +```javascript +class BotClient { + constructor(url, token) { + this.url = url; + this.token = token; + this.ws = null; + this.handlers = {}; + } + + connect() { + this.ws = new WebSocket(`${this.url}?token=${this.token}`); + + this.ws.onopen = () => { + console.log('Connected to bot'); + this.emit('connected'); + }; + + this.ws.onmessage = (event) => { + const data = JSON.parse(event.data); + this.emit(data.type, data); + }; + + this.ws.onclose = () => { + console.log('Disconnected'); + this.emit('disconnected'); + // Auto-reconnect after 3 seconds + setTimeout(() => this.connect(), 3000); + }; + + this.ws.onerror = (error) => { + console.error('WebSocket error:', error); + this.emit('error', error); + }; + } + + send(type, payload) { + if (this.ws?.readyState === WebSocket.OPEN) { + this.ws.send(JSON.stringify({ type, ...payload })); + } + } + + sendMessage(sessionId, content) { + this.send('message', { session_id: sessionId, content }); + } + + subscribe(sessionId) { + this.send('subscribe', { session_id: sessionId }); + } + + on(event, handler) { + this.handlers[event] = this.handlers[event] || []; + this.handlers[event].push(handler); + } + + emit(event, data) { + (this.handlers[event] || []).forEach(h => h(data)); + } + + disconnect() { + this.ws?.close(); + } +} + +// Usage +const client = new BotClient('ws://localhost:8080/ws', 'auth-token'); + +client.on('connected', () => { + client.subscribe('session-uuid'); +}); + +client.on('message', (data) => { + console.log('Bot:', data.content); +}); + +client.on('stream_chunk', (data) => { + process.stdout.write(data.content); +}); + +client.on('error', (data) => { + console.error('Error:', data.message); +}); + +client.connect(); +client.sendMessage('session-uuid', 'Hello!'); +``` + +## Meet WebSocket + +Video conferencing uses a separate WebSocket endpoint: + +``` +ws://localhost:8080/ws/meet +``` + +### Join Room + +```json +{ + "type": "join", + "room_id": "room-uuid", + "participant_name": "John" +} +``` + +### Leave Room + +```json +{ + "type": "leave", + "room_id": "room-uuid" +} +``` + +### Signaling (WebRTC) + +```json +{ + "type": "signal", + "room_id": "room-uuid", + "target_id": "participant-uuid", + "signal": { /* WebRTC signal data */ } +} +``` + +## Connection Limits + +| Limit | Value | +|-------|-------| +| Max connections per IP | 100 | +| Max message size | 64 KB | +| Idle timeout | 5 minutes | +| Ping interval | 30 seconds | + +## Error Codes + +| Code | Description | +|------|-------------| +| `AUTH_FAILED` | Invalid or expired token | +| `SESSION_NOT_FOUND` | Session doesn't exist | +| `RATE_LIMITED` | Too many messages | +| `MESSAGE_TOO_LARGE` | Exceeds 64 KB limit | +| `INVALID_FORMAT` | Malformed JSON | +| `SUBSCRIPTION_FAILED` | Cannot subscribe to session | \ No newline at end of file diff --git a/docs/guides/deployment.md b/docs/guides/deployment.md new file mode 100644 index 00000000..37d4ecdb --- /dev/null +++ b/docs/guides/deployment.md @@ -0,0 +1,465 @@ +# Deployment Guide + +Deploy General Bots in production environments with security, scalability, and reliability. + +## Deployment Options + +| Method | Best For | Complexity | +|--------|----------|------------| +| Single Server | Small teams, development | Low | +| Docker Compose | Medium deployments | Medium | +| LXC Containers | Isolated multi-tenant | Medium | +| Kubernetes | Large scale, high availability | High | + +## Single Server Deployment + +### Requirements + +- **CPU**: 4+ cores +- **RAM**: 16GB minimum +- **Disk**: 100GB SSD +- **OS**: Ubuntu 22.04 LTS / Debian 12 + +### Installation + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source ~/.cargo/env + +# Clone and build +git clone https://github.com/GeneralBots/BotServer +cd BotServer +cargo build --release + +# Run as service +sudo cp target/release/botserver /usr/local/bin/ +sudo cp scripts/botserver.service /etc/systemd/system/ +sudo systemctl enable botserver +sudo systemctl start botserver +``` + +### Systemd Service + +```ini +# /etc/systemd/system/botserver.service +[Unit] +Description=General Bots Server +After=network.target postgresql.service + +[Service] +Type=simple +User=botserver +Group=botserver +WorkingDirectory=/opt/botserver +ExecStart=/usr/local/bin/botserver --noconsole +Restart=always +RestartSec=5 +Environment=RUST_LOG=info + +[Install] +WantedBy=multi-user.target +``` + +## Docker Deployment + +### Docker Compose + +```yaml +# docker-compose.yml +version: '3.8' + +services: + botserver: + image: generalbots/botserver:latest + ports: + - "8080:8080" + environment: + - DATABASE_URL=postgres://bot:password@postgres/botserver + - REDIS_URL=redis://redis:6379 + - S3_ENDPOINT=http://minio:9000 + depends_on: + - postgres + - redis + - minio + volumes: + - ./templates:/app/templates + - ./data:/app/data + restart: unless-stopped + + postgres: + image: postgres:15 + environment: + - POSTGRES_USER=bot + - POSTGRES_PASSWORD=password + - POSTGRES_DB=botserver + volumes: + - postgres_data:/var/lib/postgresql/data + restart: unless-stopped + + redis: + image: redis:7-alpine + volumes: + - redis_data:/data + restart: unless-stopped + + minio: + image: minio/minio + command: server /data --console-address ":9001" + environment: + - MINIO_ROOT_USER=minioadmin + - MINIO_ROOT_PASSWORD=minioadmin + volumes: + - minio_data:/data + ports: + - "9001:9001" + restart: unless-stopped + + qdrant: + image: qdrant/qdrant + volumes: + - qdrant_data:/qdrant/storage + restart: unless-stopped + +volumes: + postgres_data: + redis_data: + minio_data: + qdrant_data: +``` + +### Start Services + +```bash +docker-compose up -d +docker-compose logs -f botserver +``` + +## LXC Container Deployment + +LXC provides lightweight isolation for multi-tenant deployments. + +### Create Container + +```bash +# Create container +lxc launch ubuntu:22.04 botserver + +# Configure resources +lxc config set botserver limits.cpu 4 +lxc config set botserver limits.memory 8GB + +# Enter container +lxc exec botserver -- bash +``` + +### Inside Container + +```bash +# Install dependencies +apt update && apt install -y curl build-essential + +# Install Rust and build +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source ~/.cargo/env +git clone https://github.com/GeneralBots/BotServer +cd BotServer +cargo build --release + +# Run +./target/release/botserver --container +``` + +### Container Networking + +```bash +# Forward port from host +lxc config device add botserver http proxy \ + listen=tcp:0.0.0.0:8080 \ + connect=tcp:127.0.0.1:8080 +``` + +## Reverse Proxy Setup + +### Nginx + +```nginx +# /etc/nginx/sites-available/botserver +upstream botserver { + server 127.0.0.1:8080; + keepalive 32; +} + +server { + listen 80; + server_name bot.example.com; + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl http2; + server_name bot.example.com; + + ssl_certificate /etc/letsencrypt/live/bot.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/bot.example.com/privkey.pem; + + # Security headers + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + + location / { + proxy_pass http://botserver; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + # WebSocket support + location /ws { + proxy_pass http://botserver; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_read_timeout 86400; + } +} +``` + +### Enable Site + +```bash +sudo ln -s /etc/nginx/sites-available/botserver /etc/nginx/sites-enabled/ +sudo nginx -t +sudo systemctl reload nginx +``` + +## SSL Certificates + +### Let's Encrypt + +```bash +sudo apt install certbot python3-certbot-nginx +sudo certbot --nginx -d bot.example.com +``` + +### Auto-Renewal + +```bash +sudo certbot renew --dry-run +``` + +## Environment Configuration + +### Production Environment + +```bash +# /opt/botserver/.env +RUST_LOG=warn,botserver=info + +# Directory Service (required) +DIRECTORY_URL=https://auth.example.com +DIRECTORY_CLIENT_ID=your-client-id +DIRECTORY_CLIENT_SECRET=your-secret + +# Optional overrides +DATABASE_URL=postgres://user:pass@localhost/botserver +REDIS_URL=redis://localhost:6379 +S3_ENDPOINT=https://s3.example.com + +# Rate limiting +RATE_LIMIT_ENABLED=true +RATE_LIMIT_API_RPS=100 +``` + +## Database Setup + +### PostgreSQL Production Config + +```sql +-- Create database and user +CREATE USER botserver WITH PASSWORD 'secure_password'; +CREATE DATABASE botserver OWNER botserver; +GRANT ALL PRIVILEGES ON DATABASE botserver TO botserver; + +-- Enable extensions +\c botserver +CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; +CREATE EXTENSION IF NOT EXISTS "pg_trgm"; +``` + +### Connection Pooling (PgBouncer) + +```ini +# /etc/pgbouncer/pgbouncer.ini +[databases] +botserver = host=localhost dbname=botserver + +[pgbouncer] +listen_addr = 127.0.0.1 +listen_port = 6432 +auth_type = md5 +pool_mode = transaction +max_client_conn = 1000 +default_pool_size = 20 +``` + +## Monitoring + +### Health Check Endpoint + +```bash +curl http://localhost:8080/api/health +``` + +### Prometheus Metrics + +```yaml +# prometheus.yml +scrape_configs: + - job_name: 'botserver' + static_configs: + - targets: ['localhost:8080'] + metrics_path: /metrics +``` + +### Log Aggregation + +```bash +# Stream logs to file +journalctl -u botserver -f >> /var/log/botserver/app.log + +# Logrotate config +# /etc/logrotate.d/botserver +/var/log/botserver/*.log { + daily + rotate 14 + compress + delaycompress + missingok + notifempty +} +``` + +## Backup Strategy + +### Database Backup + +```bash +#!/bin/bash +# /opt/botserver/scripts/backup.sh +DATE=$(date +%Y%m%d_%H%M%S) +BACKUP_DIR=/backups/botserver + +# PostgreSQL +pg_dump -U botserver botserver | gzip > $BACKUP_DIR/db_$DATE.sql.gz + +# File storage +tar -czf $BACKUP_DIR/files_$DATE.tar.gz /opt/botserver/data + +# Retain 30 days +find $BACKUP_DIR -mtime +30 -delete +``` + +### Cron Schedule + +```bash +# Daily backup at 2 AM +0 2 * * * /opt/botserver/scripts/backup.sh +``` + +## Security Checklist + +- [ ] Run as non-root user +- [ ] Enable firewall (only ports 80, 443) +- [ ] Configure SSL/TLS +- [ ] Set secure file permissions +- [ ] Enable rate limiting +- [ ] Configure authentication +- [ ] Regular security updates +- [ ] Audit logging enabled +- [ ] Backup encryption +- [ ] Network isolation for database + +## Scaling + +### Horizontal Scaling + +```yaml +# docker-compose.scale.yml +services: + botserver: + deploy: + replicas: 3 + + nginx: + image: nginx + ports: + - "80:80" + volumes: + - ./nginx-lb.conf:/etc/nginx/nginx.conf +``` + +### Load Balancer Config + +```nginx +upstream botserver_cluster { + least_conn; + server botserver1:8080; + server botserver2:8080; + server botserver3:8080; +} +``` + +## Troubleshooting + +### Check Service Status + +```bash +sudo systemctl status botserver +journalctl -u botserver -n 100 +``` + +### Database Connection Issues + +```bash +psql -U botserver -h localhost -d botserver -c "SELECT 1" +``` + +### Memory Issues + +```bash +# Check memory usage +free -h +cat /proc/meminfo | grep -E "MemTotal|MemFree|Cached" + +# Increase swap if needed +sudo fallocate -l 4G /swapfile +sudo chmod 600 /swapfile +sudo mkswap /swapfile +sudo swapon /swapfile +``` + +## Updates + +### Rolling Update + +```bash +# Build new version +cd /opt/botserver +git pull +cargo build --release + +# Graceful restart +sudo systemctl reload botserver +``` + +### Zero-Downtime with Docker + +```bash +docker-compose pull +docker-compose up -d --no-deps --build botserver +``` diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md new file mode 100644 index 00000000..9a50ce5d --- /dev/null +++ b/docs/guides/getting-started.md @@ -0,0 +1,248 @@ +# Getting Started with General Bots + +This guide will help you install, configure, and run your first General Bots instance. + +## Prerequisites + +- **Rust** (1.75 or later) - [Install from rustup.rs](https://rustup.rs/) +- **Git** - [Download from git-scm.com](https://git-scm.com/downloads) +- **8GB RAM** minimum (16GB recommended) +- **10GB disk space** for dependencies and data + +## Installation + +### 1. Clone the Repository + +```bash +git clone https://github.com/GeneralBots/BotServer +cd BotServer +``` + +### 2. Run the Server + +```bash +cargo run +``` + +On first run, General Bots automatically: + +1. Downloads and compiles dependencies +2. Sets up PostgreSQL database +3. Configures S3-compatible storage (MinIO) +4. Initializes Redis cache +5. Downloads default LLM models +6. Creates template bots +7. Starts the HTTP server + +The server will be available at `http://localhost:8080`. + +## First Steps + +### Access the Web Interface + +Open your browser to: + +- **Minimal UI**: `http://localhost:8080` - Lightweight chat interface +- **Full Suite**: `http://localhost:8080/suite` - Complete application suite + +### Create Your First Bot + +1. Navigate to the `templates/` directory +2. Copy the `template.gbai` folder: + +```bash +cp -r templates/template.gbai templates/mybot.gbai +``` + +3. Edit the configuration in `mybot.gbai/mybot.gbot/config.csv`: + +```csv +name,value +theme-title,My First Bot +theme-color1,#1565C0 +``` + +4. Add knowledge to `mybot.gbai/mybot.gbkb/`: + +```markdown +# Company FAQ + +## What are your hours? +We are open Monday to Friday, 9 AM to 5 PM. + +## How do I contact support? +Email support@example.com or call 1-800-EXAMPLE. +``` + +5. Create a dialog in `mybot.gbai/mybot.gbdialog/start.bas`: + +```basic +' Welcome dialog +USE KB "mybot.gbkb" + +TALK "Welcome to My Bot!" +TALK "How can I help you today?" + +SET CONTEXT "assistant" AS "You are a helpful assistant for My Company." +``` + +6. Restart the server to load your new bot. + +## Command-Line Options + +```bash +# Default: console UI + web server +cargo run + +# Disable console UI (background service) +cargo run -- --noconsole + +# Desktop application mode (Tauri) +cargo run -- --desktop + +# Specify tenant +cargo run -- --tenant mycompany + +# LXC container mode +cargo run -- --container + +# Disable all UI +cargo run -- --noui +``` + +## Project Structure + +``` +mybot.gbai/ +├── mybot.gbot/ # Bot configuration +│ └── config.csv # Theme and settings +├── mybot.gbkb/ # Knowledge base +│ └── faq.md # FAQ documents +├── mybot.gbdialog/ # Dialog scripts +│ └── start.bas # Main dialog +└── mybot.gbdrive/ # File storage + └── templates/ # Document templates +``` + +## Essential BASIC Keywords + +### Knowledge Base + +```basic +USE KB "knowledge-name" ' Load knowledge base +CLEAR KB ' Remove from session +``` + +### Tools + +```basic +USE TOOL "tool-name" ' Make tool available +CLEAR TOOLS ' Remove all tools +``` + +### Conversation + +```basic +TALK "message" ' Send message to user +answer = HEAR ' Wait for user input +WAIT 5 ' Wait 5 seconds +``` + +### Data + +```basic +SAVE "table.csv", field1, field2 ' Save to storage +data = GET "https://api.example.com" ' HTTP request +SEND FILE "document.pdf" ' Send file to user +``` + +## Environment Variables + +General Bots requires minimal configuration. Only directory service variables are needed: + +```bash +export DIRECTORY_URL="https://zitadel.example.com" +export DIRECTORY_CLIENT_ID="your-client-id" +export DIRECTORY_CLIENT_SECRET="your-secret" +``` + +All other services (database, storage, cache) are configured automatically. + +## Testing Your Bot + +### Via Web Interface + +1. Open `http://localhost:8080` +2. Type a message in the chat box +3. The bot responds using your knowledge base and dialogs + +### Via API + +```bash +# Create a session +curl -X POST http://localhost:8080/api/sessions \ + -H "Content-Type: application/json" \ + -d '{"bot_id": "mybot"}' + +# Send a message +curl -X POST http://localhost:8080/api/sessions/{session_id}/messages \ + -H "Content-Type: application/json" \ + -d '{"content": "What are your hours?"}' +``` + +### Via WebSocket + +```javascript +const ws = new WebSocket('ws://localhost:8080/ws'); + +ws.onopen = () => { + ws.send(JSON.stringify({ + type: 'message', + session_id: 'your-session-id', + content: 'Hello!' + })); +}; + +ws.onmessage = (event) => { + const data = JSON.parse(event.data); + console.log('Bot:', data.content); +}; +``` + +## Common Issues + +### Port Already in Use + +```bash +# Find process using port 8080 +lsof -i :8080 + +# Kill the process or use a different port +cargo run -- --port 8081 +``` + +### Database Connection Failed + +Ensure PostgreSQL is running: + +```bash +botserver status postgres +botserver restart postgres +``` + +### LLM Not Responding + +Check your LLM configuration in the admin panel or verify API keys are set. + +## Next Steps + +- **[API Reference](../api/README.md)** - Integrate with external systems +- **[BASIC Language](../reference/basic-language.md)** - Complete keyword reference +- **[Templates](templates.md)** - Pre-built bot templates +- **[Deployment](deployment.md)** - Production setup guide + +## Getting Help + +- **GitHub Issues**: [github.com/GeneralBots/BotServer/issues](https://github.com/GeneralBots/BotServer/issues) +- **Stack Overflow**: Tag questions with `generalbots` +- **Documentation**: [docs.pragmatismo.com.br](https://docs.pragmatismo.com.br) \ No newline at end of file diff --git a/docs/guides/templates.md b/docs/guides/templates.md new file mode 100644 index 00000000..26f9bdd8 --- /dev/null +++ b/docs/guides/templates.md @@ -0,0 +1,308 @@ +# Using Templates + +Templates are pre-built bot configurations that accelerate development. General Bots includes templates for common use cases like CRM, HR, IT support, and more. + +## Template Structure + +Each template follows the `.gbai` package format: + +``` +template-name.gbai/ +├── template-name.gbot/ # Configuration +│ └── config.csv # Bot settings +├── template-name.gbkb/ # Knowledge base +│ └── *.md # Documentation files +├── template-name.gbdialog/ # Dialog scripts +│ ├── start.bas # Entry point +│ └── *.bas # Tool scripts +└── template-name.gbdrive/ # File storage + └── templates/ # Document templates +``` + +## Available Templates + +### Business Operations + +| Template | Description | Path | +|----------|-------------|------| +| CRM Contacts | Contact management | `crm/contacts.gbai` | +| Sales Pipeline | Deal tracking | `crm/sales-pipeline.gbai` | +| HR Employees | Employee management | `hr/employees.gbai` | +| IT Helpdesk | Ticket system | `it/helpdesk.gbai` | + +### Productivity + +| Template | Description | Path | +|----------|-------------|------| +| Office | Document automation | `productivity/office.gbai` | +| Reminder | Task reminders | `productivity/reminder.gbai` | +| Analytics | Data dashboards | `platform/analytics.gbai` | + +### Integration + +| Template | Description | Path | +|----------|-------------|------| +| API Client | External API calls | `integration/api-client.gbai` | +| Public APIs | 70+ free APIs | `integration/public-apis.gbai` | + +### Compliance + +| Template | Description | Path | +|----------|-------------|------| +| HIPAA Medical | Healthcare compliance | `compliance/hipaa-medical.gbai` | +| Privacy | GDPR/LGPD | `compliance/privacy.gbai` | + +## Using a Template + +### 1. Copy the Template + +```bash +cp -r templates/crm/contacts.gbai templates/mycrm.gbai +``` + +### 2. Rename Internal Folders + +```bash +cd templates/mycrm.gbai +mv contacts.gbot mycrm.gbot +mv contacts.gbkb mycrm.gbkb +mv contacts.gbdialog mycrm.gbdialog +mv contacts.gbdrive mycrm.gbdrive +``` + +### 3. Update Configuration + +Edit `mycrm.gbot/config.csv`: + +```csv +name,value +theme-title,My CRM Bot +theme-color1,#2196F3 +theme-color2,#E3F2FD +prompt-history,2 +``` + +### 4. Customize Knowledge Base + +Edit files in `mycrm.gbkb/`: + +```markdown +# My Company CRM Guide + +## Adding Contacts +To add a new contact, say "add contact" and provide: +- Name +- Email +- Phone number +- Company + +## Searching Contacts +Say "find contact [name]" to search. +``` + +### 5. Modify Dialogs + +Edit `mycrm.gbdialog/start.bas`: + +```basic +' My CRM Bot - Start Script + +USE KB "mycrm.gbkb" +USE TOOL "add-contact" +USE TOOL "search-contact" + +SET CONTEXT "crm" AS "You are a CRM assistant for My Company." + +TALK "Welcome to My Company CRM!" +TALK "I can help you manage contacts and deals." + +ADD SUGGESTION "add" AS "Add new contact" +ADD SUGGESTION "search" AS "Find a contact" +ADD SUGGESTION "list" AS "Show all contacts" +``` + +### 6. Restart Server + +```bash +cargo run +``` + +## Creating Custom Templates + +### Step 1: Create Package Structure + +```bash +mkdir -p templates/mytemplate.gbai/{mytemplate.gbot,mytemplate.gbkb,mytemplate.gbdialog,mytemplate.gbdrive} +``` + +### Step 2: Create Configuration + +```csv +# mytemplate.gbot/config.csv +name,value +theme-title,My Template +theme-color1,#1565C0 +theme-color2,#E3F2FD +theme-logo,https://example.com/logo.svg +prompt-history,2 +prompt-compact,4 +``` + +### Step 3: Create Start Dialog + +```basic +' mytemplate.gbdialog/start.bas + +' Load knowledge base +USE KB "mytemplate.gbkb" + +' Register tools +USE TOOL "my-action" + +' Set AI context +SET CONTEXT "assistant" AS "You are a helpful assistant." + +' Welcome message +BEGIN TALK + **Welcome to My Template!** + + I can help you with: + • Feature 1 + • Feature 2 + • Feature 3 +END TALK + +' Add quick suggestions +CLEAR SUGGESTIONS +ADD SUGGESTION "help" AS "Show help" +ADD SUGGESTION "action" AS "Do something" +``` + +### Step 4: Create Tools + +```basic +' mytemplate.gbdialog/my-action.bas + +PARAM item_name AS STRING LIKE "Example" DESCRIPTION "Name of the item" +PARAM quantity AS INTEGER LIKE 1 DESCRIPTION "How many items" + +DESCRIPTION "Performs an action with the specified item." + +' Validate input +IF item_name = "" THEN + TALK "Please provide an item name." + item_name = HEAR +END IF + +' Process +let result = item_name + " x " + quantity + +' Save record +SAVE "items.csv", item_name, quantity, result + +' Respond +TALK "✅ Created: " + result + +RETURN result +``` + +### Step 5: Add Knowledge Base + +```markdown +# mytemplate.gbkb/guide.md + +# My Template Guide + +## Overview +This template helps you accomplish specific tasks. + +## Features + +### Feature 1 +Description of feature 1. + +### Feature 2 +Description of feature 2. + +## FAQ + +### How do I get started? +Just say "help" to see available commands. + +### How do I contact support? +Email support@example.com. +``` + +## Template Best Practices + +### Configuration + +- Use clear, descriptive `theme-title` +- Choose accessible color combinations +- Set appropriate `prompt-history` (2-4 recommended) + +### Knowledge Base + +- Write clear, concise documentation +- Use headers for organization +- Include FAQ section +- Keep files under 50KB each + +### Dialogs + +- Always include `start.bas` +- Use `DESCRIPTION` for all tools +- Validate user input +- Provide helpful error messages +- Add suggestions for common actions + +### File Organization + +``` +mytemplate.gbai/ +├── mytemplate.gbot/ +│ └── config.csv +├── mytemplate.gbkb/ +│ ├── guide.md # Main documentation +│ ├── faq.md # Frequently asked questions +│ └── troubleshooting.md # Common issues +├── mytemplate.gbdialog/ +│ ├── start.bas # Entry point (required) +│ ├── tool-1.bas # First tool +│ ├── tool-2.bas # Second tool +│ └── jobs.bas # Scheduled tasks +└── mytemplate.gbdrive/ + └── templates/ + └── report.docx # Document templates +``` + +## Sharing Templates + +### Export + +```bash +tar -czf mytemplate.tar.gz templates/mytemplate.gbai +``` + +### Import + +```bash +tar -xzf mytemplate.tar.gz -C templates/ +``` + +### Version Control + +Templates work well with Git: + +```bash +cd templates/mytemplate.gbai +git init +git add . +git commit -m "Initial template" +``` + +## Next Steps + +- [BASIC Language Reference](../reference/basic-language.md) - Complete keyword list +- [API Reference](../api/README.md) - Integrate with external systems +- [Deployment Guide](deployment.md) - Production setup \ No newline at end of file diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md new file mode 100644 index 00000000..edd3429e --- /dev/null +++ b/docs/reference/architecture.md @@ -0,0 +1,530 @@ +# Architecture Reference + +System architecture and design overview for General Bots. + +## High-Level Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Clients │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Web │ │ Mobile │ │ API │ │ WhatsApp │ │ +│ │ (HTMX) │ │ App │ │ Clients │ │ Telegram │ │ +│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ +└───────┼─────────────┼─────────────┼─────────────┼───────────────┘ + │ │ │ │ + └─────────────┴──────┬──────┴─────────────┘ + │ + ┌────────▼────────┐ + │ HTTP Server │ + │ (Axum) │ + └────────┬────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ +┌───────▼───────┐ ┌─────────▼─────────┐ ┌──────▼──────┐ +│ REST API │ │ WebSocket │ │ Static │ +│ Handlers │ │ Handlers │ │ Files │ +└───────┬───────┘ └─────────┬─────────┘ └─────────────┘ + │ │ + └────────┬───────────┘ + │ + ┌────────▼────────┐ + │ App State │ + │ (Shared Arc) │ + └────────┬────────┘ + │ + ┌─────────────┼─────────────┬─────────────┐ + │ │ │ │ +┌──▼──┐ ┌────▼────┐ ┌────▼────┐ ┌───▼───┐ +│ DB │ │ Cache │ │ Storage │ │ LLM │ +│(PG) │ │ (Redis) │ │ (S3) │ │(Multi)│ +└─────┘ └─────────┘ └─────────┘ └───────┘ +``` + +## Core Components + +### HTTP Server (Axum) + +The main entry point for all requests. + +```rust +// Main router structure +Router::new() + .nest("/api", api_routes()) + .nest("/ws", websocket_routes()) + .nest_service("/", static_files()) + .layer(middleware_stack()) +``` + +**Responsibilities:** +- Request routing +- Middleware execution +- CORS handling +- Rate limiting +- Authentication + +### App State + +Shared application state passed to all handlers. + +```rust +pub struct AppState { + pub db_pool: Pool>, + pub redis: RedisPool, + pub s3_client: S3Client, + pub llm_client: LlmClient, + pub qdrant: QdrantClient, + pub config: AppConfig, +} +``` + +**Contains:** +- Database connection pool +- Redis cache connection +- S3 storage client +- LLM provider clients +- Vector database client +- Application configuration + +### Request Flow + +``` +Request → Router → Middleware → Handler → Response + │ + ├── Auth Check + ├── Rate Limit + ├── Logging + └── Error Handling +``` + +## Module Structure + +``` +botserver/src/ +├── main.rs # Entry point +├── lib.rs # Library exports +├── core/ # Core infrastructure +│ ├── shared/ # Shared types and state +│ │ ├── state.rs # AppState definition +│ │ ├── models.rs # Common models +│ │ └── schema.rs # Database schema +│ ├── urls.rs # URL constants +│ ├── secrets/ # Vault integration +│ └── rate_limit.rs # Rate limiting +├── basic/ # BASIC language +│ ├── compiler/ # Compiler/parser +│ ├── runtime/ # Execution engine +│ └── keywords/ # Keyword implementations +├── llm/ # LLM integration +│ ├── mod.rs # Provider abstraction +│ ├── openai.rs # OpenAI client +│ ├── anthropic.rs # Anthropic client +│ └── prompt_manager/ # Prompt management +├── multimodal/ # Media processing +│ ├── vision.rs # Image analysis +│ ├── audio.rs # Speech processing +│ └── document.rs # Document parsing +├── security/ # Authentication +│ ├── auth.rs # Auth middleware +│ ├── zitadel.rs # Zitadel client +│ └── jwt.rs # Token handling +├── analytics/ # Analytics module +├── calendar/ # Calendar/CalDAV +├── designer/ # Bot builder +├── drive/ # File storage +├── email/ # Email (IMAP/SMTP) +├── meet/ # Video conferencing +├── paper/ # Document editor +├── research/ # KB search +├── sources/ # Templates +└── tasks/ # Task management +``` + +## Data Flow + +### Chat Message Flow + +``` +User Input + │ + ▼ +┌──────────────┐ +│ WebSocket │ +│ Handler │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ ┌──────────────┐ +│ Session │────▶│ Message │ +│ Manager │ │ History │ +└──────┬───────┘ └──────────────┘ + │ + ▼ +┌──────────────┐ +│ KB Search │◀────── Vector DB (Qdrant) +│ (Context) │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ +│ Tool Check │◀────── Registered Tools +│ │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ +│ LLM Call │◀────── Semantic Cache (Redis) +│ │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ +│ Response │ +│ Streaming │ +└──────┬───────┘ + │ + ▼ +User Response +``` + +### Tool Execution Flow + +``` +LLM Response (tool_call) + │ + ▼ +┌──────────────┐ +│ Tool Router │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ +│BASIC Runtime │ +└──────┬───────┘ + │ + ├──▶ Database Operations + ├──▶ HTTP Requests + ├──▶ File Operations + └──▶ Email/Notifications + │ + ▼ +┌──────────────┐ +│ Tool Result │ +└──────┬───────┘ + │ + ▼ +LLM (continue or respond) +``` + +## Database Schema + +### Core Tables + +```sql +-- User sessions +CREATE TABLE user_sessions ( + id UUID PRIMARY KEY, + user_id VARCHAR(255), + bot_id VARCHAR(100), + status VARCHAR(20), + metadata JSONB, + created_at TIMESTAMPTZ, + updated_at TIMESTAMPTZ +); + +-- Message history +CREATE TABLE message_history ( + id UUID PRIMARY KEY, + session_id UUID REFERENCES user_sessions(id), + role VARCHAR(20), + content TEXT, + tokens_used INTEGER, + created_at TIMESTAMPTZ +); + +-- Bot configurations +CREATE TABLE bot_configs ( + id UUID PRIMARY KEY, + bot_id VARCHAR(100) UNIQUE, + config JSONB, + created_at TIMESTAMPTZ, + updated_at TIMESTAMPTZ +); + +-- Knowledge base documents +CREATE TABLE kb_documents ( + id UUID PRIMARY KEY, + collection VARCHAR(100), + content TEXT, + embedding_id VARCHAR(100), + metadata JSONB, + created_at TIMESTAMPTZ +); +``` + +### Indexes + +```sql +CREATE INDEX idx_sessions_user ON user_sessions(user_id); +CREATE INDEX idx_sessions_bot ON user_sessions(bot_id); +CREATE INDEX idx_messages_session ON message_history(session_id); +CREATE INDEX idx_kb_collection ON kb_documents(collection); +``` + +## Caching Strategy + +### Cache Layers + +``` +┌─────────────────────────────────────────┐ +│ Semantic Cache │ +│ (LLM responses by query similarity) │ +└───────────────────┬─────────────────────┘ + │ +┌───────────────────▼─────────────────────┐ +│ Session Cache │ +│ (Active sessions, user context) │ +└───────────────────┬─────────────────────┘ + │ +┌───────────────────▼─────────────────────┐ +│ Data Cache │ +│ (KB results, config, templates) │ +└─────────────────────────────────────────┘ +``` + +### Cache Keys + +| Pattern | TTL | Description | +|---------|-----|-------------| +| `session:{id}` | 30m | Active session data | +| `semantic:{hash}` | 24h | LLM response cache | +| `kb:{collection}:{hash}` | 1h | KB search results | +| `config:{bot_id}` | 5m | Bot configuration | +| `user:{id}` | 15m | User preferences | + +### Semantic Cache + +```rust +// Query similarity check +let cache_key = compute_embedding_hash(query); +if let Some(cached) = redis.get(&cache_key).await? { + if similarity(query, cached.query) > 0.95 { + return cached.response; + } +} +``` + +## LLM Integration + +### Provider Abstraction + +```rust +pub trait LlmProvider: Send + Sync { + async fn complete(&self, request: CompletionRequest) + -> Result; + + async fn complete_stream(&self, request: CompletionRequest) + -> Result>; + + async fn embed(&self, text: &str) + -> Result>; +} +``` + +### Supported Providers + +| Provider | Models | Features | +|----------|--------|----------| +| OpenAI | GPT-4, GPT-3.5 | Streaming, Functions, Vision | +| Anthropic | Claude 3 | Streaming, Long context | +| Groq | Llama, Mixtral | Fast inference | +| Ollama | Any local | Self-hosted | + +### Request Flow + +```rust +// 1. Build messages with context +let messages = build_messages(history, kb_context, system_prompt); + +// 2. Add tools if registered +let tools = get_registered_tools(session); + +// 3. Check semantic cache +if let Some(cached) = semantic_cache.get(&messages).await? { + return Ok(cached); +} + +// 4. Call LLM +let response = llm.complete(CompletionRequest { + messages, + tools, + temperature: config.temperature, + max_tokens: config.max_tokens, +}).await?; + +// 5. Cache response +semantic_cache.set(&messages, &response).await?; +``` + +## BASIC Runtime + +### Compilation Pipeline + +``` +Source Code (.bas) + │ + ▼ +┌──────────────┐ +│ Lexer │──▶ Tokens +└──────────────┘ + │ + ▼ +┌──────────────┐ +│ Parser │──▶ AST +└──────────────┘ + │ + ▼ +┌──────────────┐ +│ Analyzer │──▶ Validated AST +└──────────────┘ + │ + ▼ +┌──────────────┐ +│ Runtime │──▶ Execution +└──────────────┘ +``` + +### Execution Context + +```rust +pub struct RuntimeContext { + pub session: Session, + pub variables: HashMap, + pub tools: Vec, + pub kb: Vec, + pub state: Arc, +} +``` + +## Security Architecture + +### Authentication Flow + +``` +Client Request + │ + ▼ +┌──────────────┐ +│ Extract │ +│ Token │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ ┌──────────────┐ +│ Validate │────▶│ Zitadel │ +│ JWT │ │ (OIDC) │ +└──────┬───────┘ └──────────────┘ + │ + ▼ +┌──────────────┐ +│ Check │ +│ Permissions │ +└──────┬───────┘ + │ + ▼ +Handler Execution +``` + +### Security Layers + +1. **Transport**: TLS 1.3 (rustls) +2. **Authentication**: JWT/OAuth 2.0 (Zitadel) +3. **Authorization**: Role-based access control +4. **Rate Limiting**: Per-IP token bucket +5. **Input Validation**: Type-safe parameters +6. **Output Sanitization**: HTML escaping + +## Deployment Architecture + +### Single Instance + +``` +┌─────────────────────────────────────┐ +│ Single Server │ +│ ┌─────────────────────────────┐ │ +│ │ BotServer │ │ +│ └─────────────────────────────┘ │ +│ ┌────────┐ ┌────────┐ ┌───────┐ │ +│ │PostgreSQL│ │ Redis │ │ MinIO │ │ +│ └────────┘ └────────┘ └───────┘ │ +└─────────────────────────────────────┘ +``` + +### Clustered + +``` +┌─────────────────────────────────────────────────────┐ +│ Load Balancer │ +└───────────────────────┬─────────────────────────────┘ + │ + ┌───────────────┼───────────────┐ + │ │ │ + ┌────▼────┐ ┌─────▼────┐ ┌────▼────┐ + │BotServer│ │BotServer │ │BotServer│ + │ #1 │ │ #2 │ │ #3 │ + └────┬────┘ └─────┬────┘ └────┬────┘ + │ │ │ + └───────────────┼───────────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ +┌──▼───┐ ┌─────▼────┐ ┌────▼────┐ +│ PG │ │ Redis │ │ S3 │ +│Cluster│ │ Cluster │ │ Cluster │ +└──────┘ └──────────┘ └─────────┘ +``` + +## Performance Characteristics + +| Operation | Latency | Throughput | +|-----------|---------|------------| +| REST API | < 10ms | 10,000 req/s | +| WebSocket message | < 5ms | 50,000 msg/s | +| KB search | < 50ms | 1,000 req/s | +| LLM call (cached) | < 20ms | 5,000 req/s | +| LLM call (uncached) | 500-3000ms | 50 req/s | + +## Monitoring Points + +| Metric | Description | +|--------|-------------| +| `http_requests_total` | Total HTTP requests | +| `http_request_duration` | Request latency | +| `ws_connections` | Active WebSocket connections | +| `llm_requests_total` | LLM API calls | +| `llm_cache_hits` | Semantic cache hit rate | +| `db_pool_size` | Active DB connections | +| `memory_usage` | Process memory | + +## Extension Points + +### Adding a New Module + +1. Create module in `src/` +2. Define routes in `mod.rs` +3. Register in `lib.rs` +4. Add to router in `main.rs` + +### Adding a New LLM Provider + +1. Implement `LlmProvider` trait +2. Add provider enum variant +3. Register in provider factory + +### Adding a New BASIC Keyword + +1. Add keyword to lexer +2. Implement AST node +3. Add runtime handler +4. Update documentation \ No newline at end of file diff --git a/docs/reference/basic-language.md b/docs/reference/basic-language.md new file mode 100644 index 00000000..7bc884a5 --- /dev/null +++ b/docs/reference/basic-language.md @@ -0,0 +1,525 @@ +# BASIC Language Reference + +Complete reference for General Bots BASIC dialog scripting language. + +## Overview + +General Bots BASIC is a domain-specific language for creating conversational AI dialogs. It provides keywords for: + +- User interaction (TALK, HEAR) +- Knowledge base management (USE KB, CLEAR KB) +- Tool registration (USE TOOL, CLEAR TOOLS) +- Data operations (SAVE, GET, POST) +- File handling (SEND FILE, DOWNLOAD) +- Flow control (IF/THEN/ELSE, FOR/NEXT) + +## Conversation Keywords + +### TALK + +Send a message to the user. + +```basic +TALK "Hello, how can I help you?" +TALK "Your order number is: " + ordernumber +``` + +#### Multi-line Messages + +```basic +BEGIN TALK + **Welcome!** + + I can help you with: + • Orders + • Shipping + • Returns +END TALK +``` + +### HEAR + +Wait for and capture user input. + +```basic +answer = HEAR +name = HEAR AS NAME +email = HEAR AS EMAIL +choice = HEAR AS "Option A", "Option B", "Option C" +confirmed = HEAR AS BOOLEAN +``` + +#### Input Types + +| Type | Description | Example | +|------|-------------|---------| +| `STRING` | Free text | `answer = HEAR` | +| `NAME` | Person name | `name = HEAR AS NAME` | +| `EMAIL` | Email address | `email = HEAR AS EMAIL` | +| `PHONE` | Phone number | `phone = HEAR AS PHONE` | +| `INTEGER` | Whole number | `count = HEAR AS INTEGER` | +| `NUMBER` | Decimal number | `amount = HEAR AS NUMBER` | +| `BOOLEAN` | Yes/No | `confirm = HEAR AS BOOLEAN` | +| `DATE` | Date value | `date = HEAR AS DATE` | +| Options | Multiple choice | `choice = HEAR AS "A", "B", "C"` | + +### WAIT + +Pause execution for specified seconds. + +```basic +WAIT 5 +TALK "Processing..." +WAIT 2 +TALK "Done!" +``` + +## Knowledge Base Keywords + +### USE KB + +Load a knowledge base into the current session. + +```basic +USE KB "company-docs" +USE KB "product-catalog.gbkb" +``` + +### CLEAR KB + +Remove knowledge base from session. + +```basic +CLEAR KB "company-docs" +CLEAR KB ' Clear all KBs +``` + +## Tool Keywords + +### USE TOOL + +Register a tool for the AI to call. + +```basic +USE TOOL "create-ticket" +USE TOOL "send-email" +USE TOOL "search-orders" +``` + +### CLEAR TOOLS + +Remove all registered tools. + +```basic +CLEAR TOOLS +``` + +## Context Keywords + +### SET CONTEXT + +Define AI behavior context. + +```basic +SET CONTEXT "assistant" AS "You are a helpful customer service agent for Acme Corp." +``` + +### System Prompt + +Define detailed AI instructions. + +```basic +BEGIN SYSTEM PROMPT + You are a professional assistant. + Always be polite and helpful. + If you don't know something, say so. + Never make up information. +END SYSTEM PROMPT +``` + +## Suggestion Keywords + +### ADD SUGGESTION + +Add a quick-reply button for users. + +```basic +ADD SUGGESTION "help" AS "Show help" +ADD SUGGESTION "order" AS "Track my order" +ADD SUGGESTION "contact" AS "Contact support" +``` + +### CLEAR SUGGESTIONS + +Remove all suggestions. + +```basic +CLEAR SUGGESTIONS +``` + +## Data Keywords + +### SAVE + +Save data to storage. + +```basic +SAVE "contacts.csv", name, email, phone +SAVE "orders.csv", orderid, product, quantity, total +``` + +### GET + +HTTP GET request. + +```basic +data = GET "https://api.example.com/users" +weather = GET "https://api.weather.com/current?city=" + city +``` + +### POST + +HTTP POST request. + +```basic +result = POST "https://api.example.com/orders", orderdata +``` + +### PUT + +HTTP PUT request. + +```basic +result = PUT "https://api.example.com/users/" + userid, userdata +``` + +### DELETE HTTP + +HTTP DELETE request. + +```basic +result = DELETE HTTP "https://api.example.com/users/" + userid +``` + +### SET HEADER + +Set HTTP header for requests. + +```basic +SET HEADER "Authorization" = "Bearer " + token +SET HEADER "Content-Type" = "application/json" +data = GET "https://api.example.com/protected" +``` + +### CLEAR HEADERS + +Remove all custom headers. + +```basic +CLEAR HEADERS +``` + +## File Keywords + +### SEND FILE + +Send a file to the user. + +```basic +SEND FILE "report.pdf" +SEND FILE filepath +``` + +### DOWNLOAD + +Download a file from URL. + +```basic +file = DOWNLOAD "https://example.com/document.pdf" +SEND FILE file +``` + +### DELETE FILE + +Delete a file from storage. + +```basic +DELETE FILE "old-report.pdf" +``` + +## Email Keywords + +### SEND MAIL + +Send an email. + +```basic +SEND MAIL "recipient@example.com", "Subject Line", "Email body text" +SEND MAIL email, subject, body +``` + +## Memory Keywords + +### SET BOT MEMORY + +Store a value in bot memory (persists across sessions). + +```basic +SET BOT MEMORY "last_order", orderid +SET BOT MEMORY "user_preference", preference +``` + +### GET BOT MEMORY + +Retrieve a value from bot memory. + +```basic +lastorder = GET BOT MEMORY("last_order") +pref = GET BOT MEMORY("user_preference") +``` + +## Schedule Keywords + +### SET SCHEDULE + +Define when a job should run (cron format). + +```basic +SET SCHEDULE "0 9 * * *" ' Daily at 9 AM +SET SCHEDULE "0 0 * * 1" ' Weekly on Monday +SET SCHEDULE "0 8 1 * *" ' Monthly on the 1st at 8 AM +``` + +#### Cron Format + +``` +┌───────────── minute (0-59) +│ ┌───────────── hour (0-23) +│ │ ┌───────────── day of month (1-31) +│ │ │ ┌───────────── month (1-12) +│ │ │ │ ┌───────────── day of week (0-6, Sun=0) +│ │ │ │ │ +* * * * * +``` + +## Flow Control + +### IF/THEN/ELSE + +Conditional execution. + +```basic +IF status = "active" THEN + TALK "Your account is active." +ELSE IF status = "pending" THEN + TALK "Your account is pending approval." +ELSE + TALK "Your account is inactive." +END IF +``` + +### FOR/NEXT + +Loop through a range. + +```basic +FOR i = 1 TO 10 + TALK "Item " + i +NEXT +``` + +### FOR EACH + +Loop through a collection. + +```basic +FOR EACH item IN items + TALK item.name + ": $" + item.price +END FOR +``` + +## Variables + +### Declaration + +```basic +let name = "John" +let count = 42 +let price = 19.99 +let active = TRUE +``` + +### String Operations + +```basic +let greeting = "Hello, " + name + "!" +let upper = UCASE(text) +let lower = LCASE(text) +let length = LEN(text) +let part = MID(text, 1, 5) +``` + +### Array Operations + +```basic +let items = SPLIT(text, ",") +let first = items[0] +let count = LEN(items) +``` + +## Tool Definition + +Tools are BASIC files that the AI can call. + +### Structure + +```basic +' tool-name.bas + +PARAM parametername AS TYPE LIKE "example" DESCRIPTION "What this parameter is" +PARAM optionalparam AS STRING DESCRIPTION "Optional parameter" + +DESCRIPTION "What this tool does. Called when user wants to [action]." + +' Implementation +IF parametername = "" THEN + TALK "Please provide the parameter." + parametername = HEAR +END IF + +let result = "processed: " + parametername + +SAVE "records.csv", parametername, result + +TALK "✅ Done: " + result + +RETURN result +``` + +### Parameter Types + +| Type | Description | +|------|-------------| +| `STRING` | Text value | +| `INTEGER` | Whole number | +| `NUMBER` | Decimal number | +| `BOOLEAN` | True/False | +| `DATE` | Date value | +| `EMAIL` | Email address | +| `PHONE` | Phone number | + +## Comments + +```basic +' This is a single-line comment + +REM This is also a comment + +' Multi-line comments use multiple single-line comments +' Line 1 +' Line 2 +``` + +## Built-in Functions + +### String Functions + +| Function | Description | Example | +|----------|-------------|---------| +| `LEN(s)` | String length | `LEN("hello")` → `5` | +| `UCASE(s)` | Uppercase | `UCASE("hello")` → `"HELLO"` | +| `LCASE(s)` | Lowercase | `LCASE("HELLO")` → `"hello"` | +| `TRIM(s)` | Remove whitespace | `TRIM(" hi ")` → `"hi"` | +| `MID(s,start,len)` | Substring | `MID("hello",2,3)` → `"ell"` | +| `LEFT(s,n)` | Left characters | `LEFT("hello",2)` → `"he"` | +| `RIGHT(s,n)` | Right characters | `RIGHT("hello",2)` → `"lo"` | +| `SPLIT(s,delim)` | Split to array | `SPLIT("a,b,c",",")` → `["a","b","c"]` | +| `REPLACE(s,old,new)` | Replace text | `REPLACE("hello","l","x")` → `"hexxo"` | + +### Date Functions + +| Function | Description | Example | +|----------|-------------|---------| +| `NOW()` | Current datetime | `NOW()` | +| `TODAY()` | Current date | `TODAY()` | +| `YEAR(d)` | Extract year | `YEAR(date)` → `2024` | +| `MONTH(d)` | Extract month | `MONTH(date)` → `12` | +| `DAY(d)` | Extract day | `DAY(date)` → `15` | +| `DATEADD(d,n,unit)` | Add to date | `DATEADD(date,7,"days")` | +| `DATEDIFF(d1,d2,unit)` | Date difference | `DATEDIFF(date1,date2,"days")` | + +### Math Functions + +| Function | Description | Example | +|----------|-------------|---------| +| `ABS(n)` | Absolute value | `ABS(-5)` → `5` | +| `ROUND(n,d)` | Round number | `ROUND(3.456,2)` → `3.46` | +| `FLOOR(n)` | Round down | `FLOOR(3.7)` → `3` | +| `CEILING(n)` | Round up | `CEILING(3.2)` → `4` | +| `MIN(a,b)` | Minimum | `MIN(5,3)` → `3` | +| `MAX(a,b)` | Maximum | `MAX(5,3)` → `5` | +| `SUM(arr)` | Sum of array | `SUM(numbers)` | +| `AVG(arr)` | Average | `AVG(numbers)` | + +### Conversion Functions + +| Function | Description | Example | +|----------|-------------|---------| +| `STR(n)` | Number to string | `STR(42)` → `"42"` | +| `VAL(s)` | String to number | `VAL("42")` → `42` | +| `INT(n)` | To integer | `INT(3.7)` → `3` | + +## Complete Example + +```basic +' customer-support.bas - Main support dialog + +' Setup +USE KB "support-docs" +USE TOOL "create-ticket" +USE TOOL "check-order" +USE TOOL "request-refund" + +SET CONTEXT "support" AS "You are a helpful customer support agent for Acme Store." + +' Welcome +BEGIN TALK + **Welcome to Acme Support!** + + I can help you with: + • Order tracking + • Returns and refunds + • Product questions +END TALK + +' Quick actions +CLEAR SUGGESTIONS +ADD SUGGESTION "order" AS "Track my order" +ADD SUGGESTION "return" AS "Request a return" +ADD SUGGESTION "help" AS "Other questions" + +BEGIN SYSTEM PROMPT + Be friendly and professional. + Always verify order numbers before making changes. + For refunds over $100, escalate to human support. + If asked about competitors, politely redirect to our products. +END SYSTEM PROMPT +``` + +## Keyword Quick Reference + +| Category | Keywords | +|----------|----------| +| Conversation | `TALK`, `HEAR`, `WAIT` | +| Knowledge | `USE KB`, `CLEAR KB` | +| Tools | `USE TOOL`, `CLEAR TOOLS` | +| Context | `SET CONTEXT`, `SYSTEM PROMPT` | +| Suggestions | `ADD SUGGESTION`, `CLEAR SUGGESTIONS` | +| Data | `SAVE`, `GET`, `POST`, `PUT`, `DELETE HTTP` | +| HTTP | `SET HEADER`, `CLEAR HEADERS` | +| Files | `SEND FILE`, `DOWNLOAD`, `DELETE FILE` | +| Email | `SEND MAIL` | +| Memory | `SET BOT MEMORY`, `GET BOT MEMORY` | +| Schedule | `SET SCHEDULE` | +| Flow | `IF/THEN/ELSE/END IF`, `FOR/NEXT`, `FOR EACH` | +| Tools | `PARAM`, `DESCRIPTION`, `RETURN` | \ No newline at end of file diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 00000000..53394fb3 --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,342 @@ +# Configuration Reference + +Complete reference for General Bots configuration options. + +## Configuration Files + +### Bot Configuration (`config.csv`) + +Located in each bot's `.gbot` folder: + +``` +mybot.gbai/ +└── mybot.gbot/ + └── config.csv +``` + +Format: CSV with `name,value` columns. + +```csv +name,value +theme-title,My Bot +theme-color1,#1565C0 +theme-color2,#E3F2FD +``` + +## Theme Settings + +| Setting | Description | Default | Example | +|---------|-------------|---------|---------| +| `theme-title` | Bot display name | Bot ID | `My Company Bot` | +| `theme-color1` | Primary color | `#1565C0` | `#2196F3` | +| `theme-color2` | Secondary/background | `#E3F2FD` | `#FFFFFF` | +| `theme-logo` | Logo URL | Default logo | `https://example.com/logo.svg` | +| `theme-favicon` | Favicon URL | Default | `https://example.com/favicon.ico` | + +### Color Examples + +```csv +name,value +theme-color1,#1565C0 +theme-color2,#E3F2FD +``` + +| Scheme | Primary | Secondary | +|--------|---------|-----------| +| Blue | `#1565C0` | `#E3F2FD` | +| Green | `#2E7D32` | `#E8F5E9` | +| Purple | `#7B1FA2` | `#F3E5F5` | +| Orange | `#EF6C00` | `#FFF3E0` | +| Red | `#C62828` | `#FFEBEE` | +| Dark | `#212121` | `#424242` | + +## Prompt Settings + +| Setting | Description | Default | Range | +|---------|-------------|---------|-------| +| `prompt-history` | Messages in context | `2` | 1-10 | +| `prompt-compact` | Compact mode threshold | `4` | 2-20 | +| `prompt-max-tokens` | Max response tokens | `2048` | 256-8192 | +| `prompt-temperature` | Response creativity | `0.7` | 0.0-2.0 | + +```csv +name,value +prompt-history,2 +prompt-compact,4 +prompt-max-tokens,2048 +prompt-temperature,0.7 +``` + +### History Settings + +- `prompt-history=1`: Minimal context, faster responses +- `prompt-history=2`: Balanced (recommended) +- `prompt-history=5`: More context, slower responses + +## LLM Settings + +| Setting | Description | Default | +|---------|-------------|---------| +| `llm-provider` | LLM provider | `openai` | +| `llm-model` | Model name | `gpt-4` | +| `llm-api-key` | API key (or use env) | - | +| `llm-endpoint` | Custom endpoint | Provider default | + +```csv +name,value +llm-provider,openai +llm-model,gpt-4-turbo +``` + +### Supported Providers + +| Provider | Models | +|----------|--------| +| `openai` | `gpt-4`, `gpt-4-turbo`, `gpt-3.5-turbo` | +| `anthropic` | `claude-3-opus`, `claude-3-sonnet` | +| `groq` | `llama-3-70b`, `mixtral-8x7b` | +| `ollama` | Any local model | + +## Feature Flags + +| Setting | Description | Default | +|---------|-------------|---------| +| `feature-voice` | Enable voice input/output | `false` | +| `feature-file-upload` | Allow file uploads | `true` | +| `feature-suggestions` | Show quick replies | `true` | +| `feature-typing` | Show typing indicator | `true` | +| `feature-history` | Show chat history | `true` | + +```csv +name,value +feature-voice,true +feature-file-upload,true +feature-suggestions,true +``` + +## Security Settings + +| Setting | Description | Default | +|---------|-------------|---------| +| `auth-required` | Require authentication | `false` | +| `auth-provider` | Auth provider | `zitadel` | +| `allowed-domains` | Allowed email domains | `*` | +| `rate-limit` | Requests per minute | `60` | + +```csv +name,value +auth-required,true +auth-provider,zitadel +allowed-domains,example.com,company.org +rate-limit,30 +``` + +## Environment Variables + +### Required + +| Variable | Description | +|----------|-------------| +| `DIRECTORY_URL` | Zitadel/Auth instance URL | +| `DIRECTORY_CLIENT_ID` | OAuth client ID | +| `DIRECTORY_CLIENT_SECRET` | OAuth client secret | + +### Optional Overrides + +| Variable | Description | Default | +|----------|-------------|---------| +| `DATABASE_URL` | PostgreSQL connection | Auto-configured | +| `REDIS_URL` | Redis connection | Auto-configured | +| `S3_ENDPOINT` | S3/MinIO endpoint | Auto-configured | +| `S3_ACCESS_KEY` | S3 access key | Auto-configured | +| `S3_SECRET_KEY` | S3 secret key | Auto-configured | +| `QDRANT_URL` | Vector DB URL | Auto-configured | + +### LLM API Keys + +| Variable | Description | +|----------|-------------| +| `OPENAI_API_KEY` | OpenAI API key | +| `ANTHROPIC_API_KEY` | Anthropic API key | +| `GROQ_API_KEY` | Groq API key | + +### Server Settings + +| Variable | Description | Default | +|----------|-------------|---------| +| `PORT` | HTTP server port | `8080` | +| `HOST` | Bind address | `0.0.0.0` | +| `RUST_LOG` | Log level | `info` | +| `WORKERS` | Thread pool size | CPU cores | + +### Example `.env` File + +```bash +# Authentication (required) +DIRECTORY_URL=https://auth.example.com +DIRECTORY_CLIENT_ID=abc123 +DIRECTORY_CLIENT_SECRET=secret + +# LLM Provider +OPENAI_API_KEY=sk-... + +# Optional overrides +DATABASE_URL=postgres://user:pass@localhost/botserver +REDIS_URL=redis://localhost:6379 + +# Server +PORT=8080 +RUST_LOG=info,botserver=debug +``` + +## Rate Limiting + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `RATE_LIMIT_ENABLED` | Enable rate limiting | `true` | +| `RATE_LIMIT_API_RPS` | API requests/second | `100` | +| `RATE_LIMIT_API_BURST` | API burst limit | `200` | +| `RATE_LIMIT_AUTH_RPS` | Auth requests/second | `10` | +| `RATE_LIMIT_AUTH_BURST` | Auth burst limit | `20` | +| `RATE_LIMIT_LLM_RPS` | LLM requests/second | `5` | +| `RATE_LIMIT_LLM_BURST` | LLM burst limit | `10` | + +```bash +RATE_LIMIT_ENABLED=true +RATE_LIMIT_API_RPS=100 +RATE_LIMIT_LLM_RPS=5 +``` + +## Logging + +### Log Levels + +| Level | Description | +|-------|-------------| +| `error` | Errors only | +| `warn` | Warnings and errors | +| `info` | General information (default) | +| `debug` | Detailed debugging | +| `trace` | Very verbose | + +### Module-Specific Logging + +```bash +# General info, debug for botserver +RUST_LOG=info,botserver=debug + +# Quiet except errors, debug for specific module +RUST_LOG=error,botserver::llm=debug + +# Full trace for development +RUST_LOG=trace +``` + +## Database Configuration + +### Connection Pool + +| Setting | Description | Default | +|---------|-------------|---------| +| `DB_POOL_MIN` | Minimum connections | `2` | +| `DB_POOL_MAX` | Maximum connections | `10` | +| `DB_TIMEOUT` | Connection timeout (sec) | `30` | + +### PostgreSQL Tuning + +```sql +-- Recommended settings for production +ALTER SYSTEM SET shared_buffers = '256MB'; +ALTER SYSTEM SET effective_cache_size = '1GB'; +ALTER SYSTEM SET max_connections = 200; +``` + +## Storage Configuration + +### S3/MinIO Settings + +| Variable | Description | Default | +|----------|-------------|---------| +| `S3_ENDPOINT` | Endpoint URL | Auto | +| `S3_REGION` | AWS region | `us-east-1` | +| `S3_BUCKET` | Default bucket | `botserver` | +| `S3_ACCESS_KEY` | Access key | Auto | +| `S3_SECRET_KEY` | Secret key | Auto | + +## Cache Configuration + +### Redis Settings + +| Variable | Description | Default | +|----------|-------------|---------| +| `REDIS_URL` | Connection URL | Auto | +| `CACHE_TTL` | Default TTL (seconds) | `3600` | +| `SEMANTIC_CACHE_ENABLED` | Enable LLM caching | `true` | +| `SEMANTIC_CACHE_THRESHOLD` | Similarity threshold | `0.95` | + +## Complete Example + +### config.csv + +```csv +name,value +theme-title,Acme Support Bot +theme-color1,#1565C0 +theme-color2,#E3F2FD +theme-logo,https://acme.com/logo.svg +prompt-history,2 +prompt-compact,4 +prompt-temperature,0.7 +llm-provider,openai +llm-model,gpt-4-turbo +feature-voice,false +feature-file-upload,true +feature-suggestions,true +auth-required,true +rate-limit,30 +``` + +### .env + +```bash +# Auth +DIRECTORY_URL=https://auth.acme.com +DIRECTORY_CLIENT_ID=bot-client +DIRECTORY_CLIENT_SECRET=supersecret + +# LLM +OPENAI_API_KEY=sk-... + +# Server +PORT=8080 +RUST_LOG=info + +# Rate limiting +RATE_LIMIT_ENABLED=true +RATE_LIMIT_API_RPS=100 +``` + +## Configuration Precedence + +1. **Environment variables** (highest priority) +2. **Bot config.csv** +3. **Default values** (lowest priority) + +Environment variables always override config.csv settings. + +## Validation + +On startup, General Bots validates configuration and logs warnings for: + +- Missing required settings +- Invalid values +- Deprecated options +- Security concerns (e.g., weak rate limits) + +Check logs for configuration issues: + +```bash +RUST_LOG=info cargo run 2>&1 | grep -i config +```