From 198e658bbd839a7906881948eb1a6662f7960dfe Mon Sep 17 00:00:00 2001 From: "Rodrigo Rodriguez (Pragmatismo)" Date: Fri, 21 Nov 2025 12:13:48 -0300 Subject: [PATCH] WPP. --- README.md | 2 +- docs/QUICK_START.md | 324 +++++++++++++++ docs/src/SUMMARY.md | 1 + docs/src/chapter-01/README.md | 175 ++++++--- docs/src/chapter-01/first-conversation.md | 319 +++++++++++++-- docs/src/chapter-01/installation.md | 253 ++++++++++-- docs/src/chapter-06/containers.md | 454 ++++++++++++++++++++++ docs/src/chapter-08/tool-definition.md | 227 +++++++++++ 8 files changed, 1647 insertions(+), 108 deletions(-) create mode 100644 docs/QUICK_START.md create mode 100644 docs/src/chapter-06/containers.md diff --git a/README.md b/README.md index fb1aa172..68b86314 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ | Samples | [BASIC](https://github.com/GeneralBots/BotServer/tree/master/packages/default.gbdialog) or [![TypeScript](https://badges.frapsoft.com/typescript/code/typescript.svg?v=101)](https://github.com/GeneralBots/AzureADPasswordReset.gbapp) | [Docker Image](https://github.com/lpicanco/docker-botserver) ![Docker Pulls](https://img.shields.io/docker/pulls/lpicanco/botserver.svg)
*Provided by [@lpicanco](https://github.com/lpicanco/docker-botserver)* | -# General Bots +# BotServer - Just Run It! ๐Ÿš€ ![General Bot Logo](https://github.com/GeneralBots/BotServer/blob/main/logo.png?raw=true)) diff --git a/docs/QUICK_START.md b/docs/QUICK_START.md new file mode 100644 index 00000000..23c48956 --- /dev/null +++ b/docs/QUICK_START.md @@ -0,0 +1,324 @@ +# BotServer Quick Start + +## Installation in 3 Steps + +### 1. Run BotServer + +```bash +./botserver +``` + +That's it! No configuration needed. + +### 2. Wait for Bootstrap (2-5 minutes) + +You'll see: + +``` +๐Ÿš€ BotServer starting... +๐Ÿ“ฆ Bootstrap: Detecting system... +๐Ÿ“ฆ Installing PostgreSQL... + โœ“ Database created + โœ“ Schema initialized + โœ“ Credentials saved to .env +๐Ÿ“ฆ Installing MinIO... + โœ“ Object storage ready + โœ“ Buckets created +๐Ÿ“ฆ Installing Valkey... + โœ“ Cache server running +๐Ÿค– Creating bots from templates... + โœ“ default.gbai โ†’ Default bot + โœ“ announcements.gbai โ†’ Announcements bot +โœ… BotServer ready at http://localhost:8080 +``` + +### 3. Open Browser + +``` +http://localhost:8080 +``` + +Start chatting with your bot! + +--- + +## What Just Happened? + +The **automatic bootstrap** process: + +1. โœ… Detected your OS (Linux/macOS/Windows) +2. โœ… Installed PostgreSQL database +3. โœ… Installed MinIO object storage +4. โœ… Installed Valkey cache +5. โœ… Generated secure credentials โ†’ `.env` +6. โœ… Created database schema +7. โœ… Created default bots from `templates/` +8. โœ… Started web server on port 8080 + +**Zero manual configuration required!** + +--- + +## Create Your First Tool + +Tools are just `.bas` files. Create `enrollment.bas`: + +```bas +' Student enrollment tool +PARAM name AS string LIKE "John Smith" DESCRIPTION "Student name" +PARAM email AS string LIKE "john@example.com" DESCRIPTION "Email address" +PARAM course AS string LIKE "Computer Science" DESCRIPTION "Course to enroll" + +DESCRIPTION "Processes student enrollment" + +SAVE "enrollments.csv", name, email, course, NOW() +TALK "Welcome to " + course + ", " + name + "!" +``` + +The LLM automatically discovers this tool and knows when to call it! + +--- + +## Add Knowledge Base + +Drop documents in a `.gbkb/` folder: + +``` +mybot.gbai/ + mybot.gbkb/ + docs/ + manual.pdf + faq.docx + guide.txt +``` + +The bot automatically: +- Indexes documents with vector embeddings +- Answers questions from the content +- Updates when files change + +--- + +## Container Mode (LXC) + +BotServer uses **LXC** (Linux Containers) for containerized deployment: + +```bash +# Force container mode +./botserver --container + +# Components run in isolated LXC containers +# - PostgreSQL in {tenant}-tables container +# - MinIO in {tenant}-drive container +# - Valkey in {tenant}-cache container +``` + +**Benefits**: +- โœ… Clean isolation - system-level containers +- โœ… Easy cleanup - `lxc delete {container}` +- โœ… No system pollution - everything in containers +- โœ… Lightweight - more efficient than VMs + +**Requires**: LXC/LXD installed (`sudo snap install lxd`) + +--- + +## Build from Source + +```bash +git clone https://github.com/GeneralBots/BotServer.git +cd BotServer +cargo run +``` + +Same automatic bootstrap process! + +--- + +## Optional Components + +After installation, add more features: + +```bash +./botserver install email # Stalwart email server +./botserver install directory # Zitadel identity provider +./botserver install llm # Local LLM server (offline mode) +./botserver install meeting # LiveKit video conferencing +``` + +--- + +## Example Bot Structure + +``` +mybot.gbai/ +โ”œโ”€โ”€ mybot.gbdialog/ # Dialog scripts +โ”‚ โ”œโ”€โ”€ start.bas # Entry point (required) +โ”‚ โ”œโ”€โ”€ get-weather.bas # Tool (auto-discovered) +โ”‚ โ””โ”€โ”€ send-email.bas # Tool (auto-discovered) +โ”œโ”€โ”€ mybot.gbkb/ # Knowledge base +โ”‚ โ”œโ”€โ”€ docs/ # Document collection +โ”‚ โ””โ”€โ”€ faq/ # FAQ collection +โ”œโ”€โ”€ mybot.gbot/ # Configuration +โ”‚ โ””โ”€โ”€ config.csv # Bot parameters +โ””โ”€โ”€ mybot.gbtheme/ # UI theme (optional) + โ””โ”€โ”€ custom.css +``` + +Save to `templates/mybot.gbai/` and restart - bot created automatically! + +--- + +## How It Really Works + +You DON'T write complex dialog flows. Instead: + +### 1. Add Documents +``` +mybot.gbkb/ + policies/enrollment-policy.pdf + catalog/courses.pdf +``` + +### 2. Create Tools (Optional) +```bas +' enrollment.bas - just define what it does +PARAM name AS string +PARAM course AS string +SAVE "enrollments.csv", name, course +``` + +### 3. Start Chatting! +``` +User: I want to enroll in computer science +Bot: I'll help you enroll! What's your name? +User: John Smith +Bot: [Automatically calls enrollment.bas with collected params] + Welcome to Computer Science, John Smith! +``` + +The LLM handles ALL conversation logic automatically! + +--- + +## Configuration (Optional) + +Bootstrap creates `.env` automatically: + +```env +DATABASE_URL=postgres://gbuser:RANDOM_PASS@localhost:5432/botserver +DRIVE_SERVER=http://localhost:9000 +DRIVE_ACCESSKEY=GENERATED_KEY +DRIVE_SECRET=GENERATED_SECRET +``` + +You can also configure per-bot settings in `config.csv`: + +```csv +name,value +server_port,8080 +llm-url,http://localhost:8081 +prompt-compact,4 +theme-color1,#0d2b55 +``` + +--- + +## Troubleshooting + +### Port 8080 in use? + +Edit `templates/default.gbai/default.gbot/config.csv`: + +```csv +name,value +server_port,3000 +``` + +### Clean install? + +```bash +# Remove everything and start fresh +rm -rf /opt/gbo # Linux/macOS +./botserver +``` + +### Check component status + +```bash +./botserver status tables # PostgreSQL +./botserver status drive # MinIO +./botserver status cache # Valkey +``` + +--- + +## Documentation + +- **[Full Installation Guide](docs/src/chapter-01/installation.md)** - Detailed bootstrap explanation +- **[Tool Definition](docs/src/chapter-08/tool-definition.md)** - Creating tools +- **[BASIC Keywords](docs/src/chapter-05/keywords.md)** - Language reference +- **[Package System](docs/src/chapter-02/README.md)** - Creating bots +- **[Architecture](docs/src/chapter-06/architecture.md)** - How it works + +--- + +## The Magic Formula + +``` +๐Ÿ“š Documents + ๐Ÿ”ง Tools + ๐Ÿค– LLM = โœจ Intelligent Bot +``` + +### What You DON'T Need: +- โŒ IF/THEN logic +- โŒ Intent detection +- โŒ Dialog flow charts +- โŒ State machines +- โŒ Complex routing + +### What You DO: +- โœ… Drop documents in `.gbkb/` +- โœ… Create simple `.bas` tools (optional) +- โœ… Start chatting! + +The LLM understands context, calls tools, searches documents, and maintains conversation naturally. + +--- + +## Philosophy + +1. **Just Run It** - No manual configuration +2. **Simple Scripts** - BASIC-like language anyone can learn +3. **Automatic Discovery** - Tools and KBs auto-detected +4. **Secure by Default** - Credentials auto-generated +5. **Production Ready** - Built for real-world use + +--- + +## Real Example: Education Bot + +1. **Add course materials:** + ``` + edu.gbkb/ + courses/computer-science.pdf + policies/enrollment.pdf + ``` + +2. **Create enrollment tool:** + ```bas + ' enrollment.bas + PARAM name AS string + PARAM course AS string + SAVE "enrollments.csv", name, course + ``` + +3. **Just chat:** + ``` + User: What courses do you offer? + Bot: [Searches PDFs] We offer Computer Science, Data Science... + + User: I want to enroll + Bot: [Calls enrollment.bas] Let me help you enroll... + ``` + +**No programming logic needed - the LLM handles everything!** ๐ŸŽ‰ \ No newline at end of file diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 20ac90d3..3c589b74 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -82,6 +82,7 @@ - [Chapter 06: Rust Architecture Reference](./chapter-06/README.md) - [Architecture Overview](./chapter-06/architecture.md) - [Building from Source](./chapter-06/building.md) + - [Container Deployment (LXC)](./chapter-06/containers.md) - [Module Structure](./chapter-06/crates.md) - [Service Layer](./chapter-06/services.md) - [Creating Custom Keywords](./chapter-06/custom-keywords.md) diff --git a/docs/src/chapter-01/README.md b/docs/src/chapter-01/README.md index 59621c9f..a1ebab43 100644 --- a/docs/src/chapter-01/README.md +++ b/docs/src/chapter-01/README.md @@ -1,56 +1,143 @@ -## Run and Talk -```bas -TALK "Welcome! How can I help you today?" -HEAR user_input -``` -*Start the server:* `cargo run --release` +# Chapter 01: Run and Talk + +Getting started with BotServer is incredibly simple: **just run it!** + +## Quick Start -### Installation ```bash -# Clone the repository -git clone https://github.com/GeneralBots/BotServer.git -cd BotServer +# Download and run +./botserver -# Build the project -cargo build --release - -# Run the server -cargo run --release +# Or build from source +cargo run ``` -### First Conversation +That's it! The bootstrap process handles everything automatically. + +## What You'll Learn + +This chapter covers everything you need to get started: + +1. **[Installation](./installation.md)** - How the automatic bootstrap works +2. **[First Conversation](./first-conversation.md)** - Start chatting with your bot +3. **[Understanding Sessions](./sessions.md)** - How conversations are managed + +## The Bootstrap Magic + +When you first run BotServer, it automatically: + +- โœ… Detects your operating system +- โœ… Installs PostgreSQL database +- โœ… Installs MinIO object storage +- โœ… Installs Valkey cache +- โœ… Generates secure credentials +- โœ… Creates default bots +- โœ… Starts the web server + +**No manual configuration needed!** Everything just works. + +## Your First Bot + +After bootstrap completes (2-5 minutes), open your browser to: + +``` +http://localhost:8080 +``` + +You'll see the default bot ready to chat! Just start talking - the LLM handles everything. + +## The Magic Formula + +``` +๐Ÿ“š Documents (.gbkb/) + ๐Ÿ”ง Tools (.bas) + ๐Ÿค– LLM = โœจ Intelligent Bot +``` + +**No programming required!** Just: +1. Drop documents in `.gbkb/` folders +2. Create simple tools as `.bas` files (optional) +3. Start chatting - the LLM does the rest! + +## Example: Student Enrollment Bot + +### 1. Add Course Documents + +``` +edu.gbai/ + edu.gbkb/ + policies/ + enrollment-policy.pdf + course-catalog.pdf +``` + +### 2. Create Enrollment Tool + +`edu.gbdialog/enrollment.bas`: + ```bas -TALK "Hello! I'm your GeneralBots assistant." -HEAR user_input -IF user_input CONTAINS "weather" THEN - TALK "Sure, let me check the weather for you." - CALL GET_WEATHER -ELSE - TALK "I can help with many tasks, just ask!" -ENDIF +PARAM name AS string LIKE "John Smith" DESCRIPTION "Student name" +PARAM email AS string LIKE "john@example.com" DESCRIPTION "Email" +PARAM course AS string LIKE "Computer Science" DESCRIPTION "Course" + +DESCRIPTION "Processes student enrollment" + +SAVE "enrollments.csv", name, email, course, NOW() +TALK "Welcome to " + course + ", " + name + "!" ``` -### Understanding Sessions -Each conversation is represented by a **BotSession** that persists across multiple interactions. The session manages: +### 3. Just Chat! -- **User identity** (authenticated or anonymous) -- **Conversation history** (full message transcript) -- **Context state** (variables, knowledge base references, active tools) -- **Interaction metrics** (message counts, timing) +``` +User: I want to enroll in computer science +Bot: I'll help you enroll! What's your name? +User: John Smith +Bot: Thanks John! What's your email? +User: john@example.com +Bot: [Executes enrollment.bas] + Welcome to Computer Science, John Smith! +``` -#### Storage Architecture -Sessions use a multi-layer persistence model: -1. **PostgreSQL** - Primary storage for all session data -2. **Redis** - Caching layer for active session state -3. **In-memory** - Hot session data for performance +The LLM automatically: +- โœ… Understands user wants to enroll +- โœ… Calls the enrollment tool +- โœ… Collects required parameters +- โœ… Executes when ready +- โœ… Answers questions from your documents -#### Key API Endpoints -- `POST /api/sessions` - Create new session -- `GET /api/sessions` - List user sessions -- `POST /api/sessions/{id}/start` - Activate session -- `GET /api/sessions/{id}` - Get conversation history +## Key Concepts -#### Advanced Features -- **Context compaction** - Reduces memory usage for long conversations -- **Interaction counting** - Tracks message frequency -- **Multi-device sync** - Shared state across clients +### Tools = Just `.bas` Files + +A **tool** is simply a `.bas` file that the LLM discovers and calls automatically. + +### Knowledge = Just Documents + +Drop PDFs, Word docs, or text files in `.gbkb/` - instant searchable knowledge base! + +### Sessions + +Each conversation is a **session** that persists: +- User identity (authenticated or anonymous) +- Conversation history +- Context and variables +- Active tools and knowledge bases + +Sessions automatically save to PostgreSQL and cache in Redis for performance. + +## Next Steps + +- **[Installation](./installation.md)** - Understand the bootstrap process +- **[First Conversation](./first-conversation.md)** - Try out your bot +- **[Understanding Sessions](./sessions.md)** - Learn about conversation state +- **[About Packages](../chapter-02/README.md)** - Create your own bots + +## Philosophy + +BotServer follows these principles: + +1. **Just Run It** - Bootstrap handles everything +2. **Just Chat** - No complex dialog flows needed +3. **Just Add Content** - Documents + tools = intelligent bot +4. **LLM Does the Work** - No IF/THEN logic required +5. **Production Ready** - Built for real-world use + +Ready to get started? Head to [Installation](./installation.md)! \ No newline at end of file diff --git a/docs/src/chapter-01/first-conversation.md b/docs/src/chapter-01/first-conversation.md index 58c3ee49..949ee006 100644 --- a/docs/src/chapter-01/first-conversation.md +++ b/docs/src/chapter-01/first-conversation.md @@ -1,42 +1,299 @@ # First Conversation -## Starting a Session -After the server is running, open a web browser at `http://localhost:8080` to begin. The system will automatically create a new session and load the default dialog (`start.bas`). +After BotServer starts, you can immediately start chatting with your bot. No programming required! -## Basic Interaction -The default greeting demonstrates the `TALK` keyword: -```basic -TALK "Welcome to GeneralBots! How can I assist you today?" +## Just Start Talking + +Open your browser to: +``` +http://localhost:8080 ``` -## Example Dialog Flow -Here's a more complete example showing question handling: -```basic -TALK "Hello! I'm your GeneralBots assistant." -HEAR user_input +And start chatting: -IF user_input CONTAINS "help" THEN - TALK "I can help with:" - TALK "- Answering questions" - TALK "- Finding information" - TALK "- Running tools" -ELSE IF user_input CONTAINS "time" THEN - CALL GET_CURRENT_TIME - TALK "The current time is " + time_result -ELSE - TALK "I didn't understand. Try asking for 'help'." +``` +You: Hi! +Bot: Hello! How can I help you today? + +You: I want to enroll in a course +Bot: I'll help you with enrollment. Let me collect your information... + [Bot automatically calls enrollment.bas tool] + +You: What documents do you have? +Bot: [Searches .gbkb/ folders and answers from your documents] +``` + +**That's it!** The LLM handles everything automatically. + +## How It Works + +### 1. Drop Documents in `.gbkb/` + +``` +mybot.gbai/ + mybot.gbkb/ + policies/ + enrollment-policy.pdf + course-catalog.pdf + faqs/ + student-faq.docx + payment-guide.txt +``` + +The bot automatically: +- โœ… Indexes all documents +- โœ… Creates vector embeddings +- โœ… Searches when users ask questions +- โœ… Provides accurate answers from your content + +### 2. Create Tools as `.bas` Files + +Create `enrollment.bas`: + +```bas +PARAM name AS string LIKE "John Smith" DESCRIPTION "Student full name" +PARAM email AS string LIKE "john@example.com" DESCRIPTION "Contact email" +PARAM course AS string LIKE "Computer Science" DESCRIPTION "Course to enroll in" + +DESCRIPTION "Enrollment tool - collects student information and processes enrollment" + +SAVE "enrollments.csv", name, email, course, NOW() +TALK "Enrollment complete! Welcome to " + course + ", " + name +``` + +The LLM automatically: +- โœ… Discovers this tool +- โœ… Knows when to call it (when users want to enroll) +- โœ… Collects required parameters through conversation +- โœ… Executes the tool when ready + +### 3. The LLM Does Everything Else + +You **DON'T** need to write: +- โŒ IF/THEN logic +- โŒ Intent detection +- โŒ Dialog flows +- โŒ State machines +- โŒ Complex routing + +The LLM automatically: +- Understands user intent +- Calls appropriate tools +- Searches knowledge bases +- Maintains conversation context +- Handles follow-up questions + +## Real Example Conversation + +With `enrollment.bas` tool and course documents in `.gbkb/`: + +``` +User: Hello, I'd like to know about your computer science program + +Bot: I'd be happy to help you learn about our Computer Science program! + Based on our course catalog, the program offers a comprehensive + 4-year curriculum covering software development, algorithms, + databases, and AI. Would you like to know about specific courses + or the enrollment process? + +User: Yes, I want to enroll + +Bot: Great! I'll help you enroll in the Computer Science program. + Let me collect your information. What's your full name? + +User: John Smith + +Bot: Thank you, John. What email address should we use for contact? + +User: john.smith@gmail.com + +Bot: Perfect! I'm enrolling you in Computer Science now... + [Executes enrollment.bas tool] + Enrollment complete! Welcome to Computer Science, John Smith! + You'll receive a confirmation at john.smith@gmail.com. + +User: What are the payment options? + +Bot: [Searches payment-guide.txt from .gbkb/] + According to our payment guide, we offer three options: + 1. Full payment with 10% discount + 2. Monthly installments over 12 months + 3. Semester-based payments + Which option would you prefer? +``` + +## The Magic Formula + +``` +๐Ÿ“š Documents (.gbkb/) + ๐Ÿ”ง Tools (.bas) + ๐Ÿค– LLM = โœจ Intelligent Bot +``` + +No complex programming needed! + +## Creating Your First Bot + +### Step 1: Create Structure + +```bash +mkdir -p mybot.gbai/mybot.gbdialog +mkdir -p mybot.gbai/mybot.gbkb/docs +mkdir -p mybot.gbai/mybot.gbot +``` + +### Step 2: Add Documents + +Drop any documents in `.gbkb/`: +- PDFs +- Word documents +- Text files +- Markdown files + +### Step 3: Create a Tool (Optional) + +Create `mybot.gbdialog/my-tool.bas`: + +```bas +PARAM user_name AS string +PARAM request AS string + +DESCRIPTION "Handles user requests" + +result = CALL "/api/process", user_name, request +TALK "Done! " + result +``` + +### Step 4: Start Chatting + +Restart BotServer and chat! The LLM will: +- Answer questions from your documents +- Call your tools when appropriate +- Handle the entire conversation naturally + +## No Programming Required + +Traditional chatbots require complex logic: + +```bas +' โŒ OLD WAY - DON'T DO THIS! +IF user_input CONTAINS "enroll" THEN + TALK "What's your name?" + HEAR name + TALK "What's your email?" + HEAR email + ' ... lots more code ... ENDIF ``` -## Key Features to Try -1. **Basic Responses**: Simple question/answer -2. **Conditional Logic**: Different responses based on input -3. **Tool Integration**: Calling external functions (`CALL` keyword) -4. **Variables**: Storing and using conversation state +With BotServer: -## Troubleshooting -- If you don't get a response, check: - - The server is running (`botserver status`) - - The LLM service is available (`botserver status llm`) - - There are no errors in the console +```bas +' โœ… NEW WAY - Just create the tool! +PARAM name AS string +PARAM email AS string +DESCRIPTION "Enrollment tool" +SAVE "enrollments.csv", name, email +``` +The LLM handles all the conversation logic! + +## What Can You Build? + +### Customer Support Bot +- Add product manuals to `.gbkb/` +- Create `create-ticket.bas` tool +- LLM answers questions and creates support tickets + +### HR Assistant +- Add employee handbook to `.gbkb/` +- Create `leave-request.bas` tool +- LLM explains policies and processes leave requests + +### Education Platform +- Add course materials to `.gbkb/` +- Create `enrollment.bas` and `submit-assignment.bas` tools +- LLM teaches content and manages student tasks + +### Sales Assistant +- Add product catalogs to `.gbkb/` +- Create `create-quote.bas` tool +- LLM answers product questions and generates quotes + +## Advanced Features + +### Dynamic Tool Loading + +The LLM can load tools based on context: + +```bas +' In start.bas - minimal setup +ADD_KB "general" ' Load general knowledge base +' Tools are auto-discovered from .gbdialog/ folder +``` + +### Multi-Language Support + +The LLM handles multiple languages automatically: + +``` +User: Olรก, quero me inscrever no curso +Bot: Claro! Vou ajudรก-lo com a inscriรงรฃo... + +User: ๆˆ‘ๆƒณไบ†่งฃ่ฎก็ฎ—ๆœบ็ง‘ๅญฆ่ฏพ็จ‹ +Bot: ๆˆ‘ๅพˆไนๆ„ๅธฎๆ‚จไบ†่งฃ่ฎก็ฎ—ๆœบ็ง‘ๅญฆ่ฏพ็จ‹... +``` + +### Context Awareness + +The LLM maintains conversation context: + +``` +User: I want to enroll +Bot: I'll help you enroll. What's your name? +User: Actually, first tell me about the prerequisites +Bot: Of course! The Computer Science program requires... +``` + +## Tips for Success + +### 1. Organize Documents Clearly + +``` +mybot.gbkb/ + policies/ # Policy documents + products/ # Product information + faqs/ # Frequently asked questions + tutorials/ # How-to guides +``` + +### 2. Name Tools Descriptively + +Good tool names: +- `enrollment.bas` +- `create-ticket.bas` +- `schedule-meeting.bas` + +The LLM understands what each tool does from its name and description. + +### 3. Use Tool Descriptions + +Always add descriptions to tools: + +```bas +DESCRIPTION "This tool processes student enrollment for courses" +``` + +### 4. Let the LLM Work + +Don't try to control every aspect of the conversation. Let the LLM: +- Rephrase responses naturally +- Handle unexpected questions +- Maintain conversation flow + +## Next Steps + +- [Understanding Sessions](./sessions.md) - How conversations persist +- [About Packages](../chapter-02/README.md) - Package structure +- [Tool Definition](../chapter-08/tool-definition.md) - Creating tools +- [Knowledge Base](../chapter-03/README.md) - Document management + +Remember: **Just add documents and tools - the LLM does the rest!** ๐Ÿš€ \ No newline at end of file diff --git a/docs/src/chapter-01/installation.md b/docs/src/chapter-01/installation.md index 840e1730..a2b508e0 100644 --- a/docs/src/chapter-01/installation.md +++ b/docs/src/chapter-01/installation.md @@ -1,59 +1,248 @@ # Installation +## Quick Start + +BotServer automatically installs and configures everything you need. Just run it! + +```bash +# Download and run +./botserver + +# Or build from source +cargo run +``` + +That's it! The bootstrap process handles everything automatically. + +## What Happens Automatically + +When you first run BotServer, the **bootstrap process** automatically: + +1. **Detects your system** - Linux, macOS, or Windows +2. **Creates installation directory** - `/opt/gbo` (Linux/macOS) or `botserver-stack` (Windows) +3. **Installs core components**: + - **PostgreSQL** (database) - Generates secure credentials, creates schema + - **MinIO** (file storage) - Creates buckets, sets up access keys + - **Valkey/Redis** (cache) - Configures and starts service +4. **Writes `.env` file** - All credentials stored securely +5. **Creates default bots** - Scans `templates/` for `.gbai` packages +6. **Starts web server** - Ready at `http://localhost:8080` + +### First Run Output + +You'll see messages like: + +``` +๐Ÿš€ BotServer starting... +๐Ÿ“ฆ Bootstrap: Detecting system... +๐Ÿ“ฆ Installing PostgreSQL... + โœ“ Database created + โœ“ Schema initialized + โœ“ Credentials saved to .env +๐Ÿ“ฆ Installing MinIO... + โœ“ Object storage ready + โœ“ Buckets created +๐Ÿ“ฆ Installing Valkey... + โœ“ Cache server running +๐Ÿค– Creating bots from templates... + โœ“ default.gbai โ†’ Default bot + โœ“ announcements.gbai โ†’ Announcements bot +โœ… BotServer ready at http://localhost:8080 +``` + ## System Requirements -- **Operating System**: Linux, macOS, or Windows - **Memory**: 8GB RAM minimum, 16GB recommended - **Storage**: 10GB free space -- **Dependencies**: Docker (optional), PostgreSQL, Redis +- **OS**: Linux, macOS, or Windows -## Installation Methods +## Installation Modes -### Method 1: Package Manager (Recommended) +BotServer automatically detects and chooses the best installation mode for your system. + +### Local Mode (Default) + +Components install directly on your system: +- PostgreSQL at `localhost:5432` +- MinIO at `localhost:9000` +- Valkey at `localhost:6379` + +**When used**: No container runtime detected, or you prefer native installation. + +### Container Mode (LXC) + +Components run in isolated **LXC** (Linux Containers): +- System-level container isolation +- Lightweight virtualization +- Automatic setup with `lxd init --auto` ```bash -# Install using the builtโ€‘in package manager -botserver install tables -botserver install drive -botserver install cache -botserver install llm +# Force container mode +./botserver --container + +# Force local mode +./botserver --local ``` -### Method 2: Manual Installation +**Benefits**: +- โœ… Isolated environments - no system pollution +- โœ… Easy cleanup - just remove containers +- โœ… Version control - run multiple BotServer instances +- โœ… Security - containers provide isolation + +**What happens**: +1. Bootstrap detects LXC/LXD availability +2. Creates Debian 12 containers for PostgreSQL, MinIO, Valkey +3. Mounts host directories for persistent data +4. Maps container ports to localhost +5. Creates systemd services inside containers +6. Manages container lifecycle automatically + +**Container names**: `{tenant}-tables`, `{tenant}-drive`, `{tenant}-cache` + +### Hybrid Mode + +Mix local and containerized components: -1. Download the botserver binary -2. Set environment variables: ```bash -export DATABASE_URL="postgres://gbuser:password@localhost:5432/botserver" -export DRIVE_SERVER="http://localhost:9000" -export DRIVE_ACCESSKEY="minioadmin" -export DRIVE_SECRET="minioadmin" +# Install PostgreSQL locally, MinIO in container +./botserver install tables --local +./botserver install drive --container + +# Or mix any combination +./botserver install cache --local ``` -3. Run the server: `./botserver` -## Configuration +**Note**: Container mode requires LXC/LXD installed on your system. -Create a `.env` file in your working directory: +## Post-Installation + +### Access the Web Interface + +Open your browser to `http://localhost:8080` and start chatting! + +### Check Component Status + +```bash +./botserver status tables # PostgreSQL +./botserver status drive # MinIO +./botserver status cache # Valkey +``` + +### View Configuration + +The bootstrap process creates `.env` with all credentials: + +```bash +cat .env +``` + +You'll see auto-generated values like: ```env -BOT_GUID=your-bot-id -DATABASE_URL=postgres://gbuser:password@localhost:5432/botserver +DATABASE_URL=postgres://gbuser:SECURE_RANDOM_PASS@localhost:5432/botserver DRIVE_SERVER=http://localhost:9000 -DRIVE_ACCESSKEY=minioadmin -DRIVE_SECRET=minioadmin +DRIVE_ACCESSKEY=GENERATED_KEY +DRIVE_SECRET=GENERATED_SECRET ``` -## Verification +## Optional Components -After installation, verify everything is working: +After installation, you can add more components: -1. Access the web interface at `http://localhost:8080` -2. Check that all services are running: ```bash -botserver status tables -botserver status drive -botserver status cache -botserver status llm +./botserver install email # Stalwart email server +./botserver install directory # Zitadel identity provider +./botserver install llm # Local LLM server +./botserver install meeting # LiveKit video conferencing ``` -The system will automatically create necessary database tables and storage buckets on first run. +## Adding Your Own Bot + +1. Create a `.gbai` folder in `templates/`: + ``` + templates/mybot.gbai/ + โ”œโ”€โ”€ mybot.gbdialog/ + โ”‚ โ””โ”€โ”€ start.bas + โ”œโ”€โ”€ mybot.gbot/ + โ”‚ โ””โ”€โ”€ config.csv + โ””โ”€โ”€ mybot.gbkb/ + โ””โ”€โ”€ docs/ + ``` + +2. Restart BotServer: + ```bash + ./botserver + ``` + +3. Bootstrap automatically detects and creates your bot! + +## Troubleshooting + +### Port Already in Use + +If port 8080 is taken, edit `templates/default.gbai/default.gbot/config.csv`: + +```csv +name,value +server_port,3000 +``` + +### Database Connection Failed + +The bootstrap will retry automatically. If it persists: + +```bash +./botserver install tables --force +``` + +### LXC Not Available + +If you see "LXC not found" and want container mode: + +```bash +# Install LXC/LXD +sudo snap install lxd +sudo lxd init --auto + +# Then run bootstrap +./botserver --container +``` + +### Clean Install + +Remove installation directory and restart: + +```bash +# Linux/macOS +rm -rf /opt/gbo +./botserver + +# Windows +rmdir /s botserver-stack +botserver.exe +``` + +## Advanced: Manual Component Control + +While bootstrap handles everything, you can manually control components: + +```bash +# Install specific component +./botserver install + +# Start/stop components +./botserver start tables +./botserver stop drive + +# Uninstall component +./botserver uninstall cache +``` + +Available components: `tables`, `cache`, `drive`, `llm`, `email`, `directory`, `proxy`, `dns`, `meeting`, `vector_db`, and more. + +## Next Steps + +- [First Conversation](./first-conversation.md) - Start chatting with your bot +- [Understanding Sessions](./sessions.md) - Learn how conversations work +- [About Packages](../chapter-02/README.md) - Create custom bots \ No newline at end of file diff --git a/docs/src/chapter-06/containers.md b/docs/src/chapter-06/containers.md new file mode 100644 index 00000000..1f1029dd --- /dev/null +++ b/docs/src/chapter-06/containers.md @@ -0,0 +1,454 @@ +# Container Deployment (LXC) + +BotServer uses **LXC** (Linux Containers) for isolated component deployment. This provides system-level containerization without the overhead of full virtualization. + +## What is LXC? + +LXC is a lightweight container technology that runs on Linux: + +- **System containers** - Full Linux userspace (like lightweight VMs) +- **Shared kernel** - More efficient than virtual machines +- **Isolation** - Separate process trees, networking, filesystems +- **Resource control** - CPU, memory, and I/O limits + +BotServer uses LXC to run PostgreSQL, MinIO, and Valkey in isolated containers. + +## Automatic Container Setup + +When you run BotServer with `--container` flag: + +```bash +./botserver --container +``` + +The bootstrap process automatically: + +1. **Detects LXC/LXD** - Checks if `lxc` command is available +2. **Initializes LXD** - Runs `lxd init --auto` if needed +3. **Creates containers** - Launches Debian 12 containers for each component +4. **Mounts directories** - Binds host paths for persistent data +5. **Configures networking** - Maps container ports to localhost +6. **Installs components** - Runs installation inside containers +7. **Creates services** - Sets up systemd inside containers +8. **Starts containers** - Everything starts automatically + +## Container Architecture + +### Container Naming + +Each component runs in a dedicated container: + +``` +{tenant}-tables โ†’ PostgreSQL database +{tenant}-drive โ†’ MinIO object storage +{tenant}-cache โ†’ Valkey cache +{tenant}-llm โ†’ LLM server (optional) +{tenant}-email โ†’ Stalwart mail (optional) +``` + +Default tenant is `default`, so containers are named: +- `default-tables` +- `default-drive` +- `default-cache` + +### Directory Mounting + +Host directories are mounted into containers for persistence: + +``` +Host: botserver-stack/tables/data/ + โ†’ Container: /opt/gbo/data/ + +Host: botserver-stack/tables/conf/ + โ†’ Container: /opt/gbo/conf/ + +Host: botserver-stack/tables/logs/ + โ†’ Container: /opt/gbo/logs/ +``` + +Data persists even if containers are deleted! + +### Port Forwarding + +Container ports are mapped to localhost: + +``` +Container: 5432 โ†’ Host: 5432 (PostgreSQL) +Container: 9000 โ†’ Host: 9000 (MinIO API) +Container: 9001 โ†’ Host: 9001 (MinIO Console) +Container: 6379 โ†’ Host: 6379 (Valkey) +``` + +Access services on localhost as if they were running natively! + +## Manual Container Operations + +### List Containers + +```bash +lxc list +``` + +Output: +``` ++----------------+---------+----------------------+ +| NAME | STATE | IPV4 | ++----------------+---------+----------------------+ +| default-tables | RUNNING | 10.x.x.x (eth0) | +| default-drive | RUNNING | 10.x.x.x (eth0) | +| default-cache | RUNNING | 10.x.x.x (eth0) | ++----------------+---------+----------------------+ +``` + +### Execute Commands in Container + +```bash +# PostgreSQL container +lxc exec default-tables -- psql -U gbuser botserver + +# MinIO container +lxc exec default-drive -- mc admin info local + +# Valkey container +lxc exec default-cache -- valkey-cli ping +``` + +### View Container Logs + +```bash +# Container system logs +lxc exec default-tables -- journalctl -u tables + +# Service logs (mounted from host) +tail -f botserver-stack/tables/logs/postgresql.log +``` + +### Stop/Start Containers + +```bash +# Stop a container +lxc stop default-tables + +# Start a container +lxc start default-tables + +# Restart a container +lxc restart default-drive +``` + +### Delete Containers + +```bash +# Stop and delete +lxc stop default-tables +lxc delete default-tables + +# Or force delete +lxc delete default-tables --force +``` + +**Note**: Data in mounted directories persists! + +## Container Configuration + +### Resource Limits + +Limit CPU and memory per container: + +```bash +# Limit to 2 CPU cores +lxc config set default-tables limits.cpu 2 + +# Limit to 4GB RAM +lxc config set default-tables limits.memory 4GB + +# View configuration +lxc config show default-tables +``` + +### Privileged vs Unprivileged + +By default, BotServer creates **privileged containers** for compatibility: + +```bash +lxc launch images:debian/12 default-tables \ + -c security.privileged=true +``` + +For better security, use **unprivileged containers** (may require additional setup): + +```bash +lxc launch images:debian/12 default-tables \ + -c security.privileged=false +``` + +### Storage Backends + +LXC supports multiple storage backends: + +- **dir** - Simple directory (default) +- **btrfs** - Copy-on-write filesystem +- **zfs** - Advanced filesystem with snapshots +- **lvm** - Logical volume management + +Check current backend: + +```bash +lxc storage list +``` + +## Advanced Usage + +### Container Snapshots + +Create snapshots before updates: + +```bash +# Create snapshot +lxc snapshot default-tables backup-2024-01-15 + +# List snapshots +lxc info default-tables + +# Restore snapshot +lxc restore default-tables backup-2024-01-15 + +# Delete snapshot +lxc delete default-tables/backup-2024-01-15 +``` + +### Container Cloning + +Clone containers for testing: + +```bash +# Clone a container +lxc copy default-tables test-tables + +# Start the clone +lxc start test-tables +``` + +### Network Configuration + +View container network: + +```bash +# Show network interfaces +lxc exec default-tables -- ip addr + +# Show network devices +lxc config device show default-tables +``` + +Add additional network interfaces: + +```bash +lxc config device add default-tables eth1 nic \ + nictype=bridged parent=lxdbr0 +``` + +## Troubleshooting + +### LXC Not Installed + +```bash +# Ubuntu/Debian +sudo snap install lxd +sudo lxd init --auto + +# Or APT (older) +sudo apt install lxd lxd-client +sudo lxd init --auto +``` + +### Permission Denied + +```bash +# Add your user to lxd group +sudo usermod -aG lxd $USER + +# Logout and login again +newgrp lxd +``` + +### Container Won't Start + +```bash +# Check container status +lxc info default-tables + +# View container logs +lxc console default-tables --show-log + +# Check for errors +lxc exec default-tables -- dmesg | tail +``` + +### Port Already in Use + +```bash +# Check what's using the port +sudo netstat -tulpn | grep 5432 + +# Change port forwarding +lxc config device remove default-tables port-5432 +lxc config device add default-tables port-5433 proxy \ + listen=tcp:0.0.0.0:5433 connect=tcp:127.0.0.1:5432 +``` + +### Container Can't Access Network + +```bash +# Restart LXD networking +sudo systemctl restart lxd-agent + +# Check firewall rules +sudo iptables -L +``` + +## Monitoring + +### Resource Usage + +```bash +# CPU and memory usage +lxc list --columns ns4t + +# Detailed info +lxc info default-tables +``` + +### Disk Usage + +```bash +# Check container disk usage +lxc exec default-tables -- df -h + +# Check all containers +lxc list --format json | jq -r '.[].name' | while read container; do + echo -n "$container: " + lxc exec $container -- df -h / --output=used | tail -n1 +done +``` + +### Process Monitoring + +```bash +# Show processes in container +lxc exec default-tables -- ps aux + +# Show top processes +lxc exec default-tables -- htop +``` + +## Security Best Practices + +### 1. Use Unprivileged Containers + +When possible, avoid privileged containers: + +```bash +lxc launch images:debian/12 default-tables \ + -c security.privileged=false +``` + +### 2. Isolate Networks + +Create separate networks for different tenants: + +```bash +lxc network create tenant1-net +lxc network attach tenant1-net tenant1-tables eth0 +``` + +### 3. Limit Resources + +Prevent resource exhaustion: + +```bash +lxc config set default-tables limits.cpu 4 +lxc config set default-tables limits.memory 8GB +lxc config set default-tables limits.disk 50GB +``` + +### 4. Regular Updates + +Keep containers updated: + +```bash +lxc exec default-tables -- apt update +lxc exec default-tables -- apt upgrade -y +``` + +### 5. Backup Snapshots + +Regular snapshots for disaster recovery: + +```bash +# Create daily snapshot +lxc snapshot default-tables daily-$(date +%Y%m%d) + +# Automated with cron +0 2 * * * lxc snapshot default-tables daily-$(date +\%Y\%m\%d) +``` + +## Container vs Local Mode + +### When to Use Containers + +โœ… **Use containers when:** +- You want clean isolation +- Running multiple BotServer instances +- Easy cleanup/reinstall is important +- Testing new versions +- Security isolation is critical + +### When to Use Local Mode + +โœ… **Use local when:** +- Maximum performance needed +- LXC not available +- Simple single-instance deployment +- Direct access to services preferred + +## Migration + +### Local to Container + +1. Export data from local installation +2. Install in container mode +3. Import data + +```bash +# Backup local data +pg_dump botserver > backup.sql + +# Switch to container mode +./botserver --container + +# Restore data +lxc exec default-tables -- psql -U gbuser botserver < backup.sql +``` + +### Container to Local + +```bash +# Backup from container +lxc exec default-tables -- pg_dump -U gbuser botserver > backup.sql + +# Uninstall containers +./botserver uninstall tables + +# Install locally +./botserver install tables --local + +# Restore data +psql -U gbuser botserver < backup.sql +``` + +## Next Steps + +- [Architecture Overview](./architecture.md) - Understand the system design +- [Building from Source](./building.md) - Compile with container support +- [Service Layer](./services.md) - Learn about service management \ No newline at end of file diff --git a/docs/src/chapter-08/tool-definition.md b/docs/src/chapter-08/tool-definition.md index 98882a20..3d3cf2ab 100644 --- a/docs/src/chapter-08/tool-definition.md +++ b/docs/src/chapter-08/tool-definition.md @@ -1 +1,228 @@ # Tool Definition + +In BotServer, a **tool** is simply a `.bas` file. That's it! + +## How It Works + +1. **Create a `.bas` file** in your `.gbdialog/` folder +2. **The LLM automatically discovers it** and can call it when needed +3. **No manual registration required** - it just works! + +## Simple Example + +Create `get-weather.bas`: + +```bas +' This tool gets weather information +' The LLM will call this when users ask about weather + +TALK "Let me check the weather for you..." +weather = CALL "/api/weather", "San Francisco" +TALK "The weather is: " + weather +``` + +That's a tool! The LLM now knows it can call this when users ask about weather. + +## Tool with Parameters + +Create `send-email.bas`: + +```bas +' Send an email to someone +PARAM to AS STRING +PARAM subject AS STRING +PARAM body AS STRING + +CALL "/email/send", to, subject, body +TALK "Email sent to " + to +``` + +The `PARAM` declarations tell the LLM what parameters this tool accepts. + +## Making Tools Available + +### Method 1: Automatic Discovery (Default) + +All `.bas` files in your `.gbdialog/` folder are automatically available. + +``` +mybot.gbai/ + mybot.gbdialog/ + start.bas โ† Entry point + get-weather.bas โ† Tool (auto-discovered) + send-email.bas โ† Tool (auto-discovered) + create-task.bas โ† Tool (auto-discovered) +``` + +### Method 2: Manual Registration + +In your `start.bas`, explicitly add tools: + +```bas +' Register tools for this conversation +ADD_TOOL "get-weather" +ADD_TOOL "send-email" +ADD_TOOL "create-task" + +TALK "Hello! I can help with weather, email, and tasks." +``` + +### Method 3: Dynamic Registration + +Add tools based on context: + +```bas +' In start.bas +TALK "What do you need help with?" +HEAR user_input + +IF user_input CONTAINS "weather" THEN + ADD_TOOL "get-weather" + TALK "I've loaded the weather tool." +ENDIF + +IF user_input CONTAINS "email" THEN + ADD_TOOL "send-email" + TALK "I can help with email now." +ENDIF +``` + +## Tool Format Conversion + +BotServer automatically converts your `.bas` tools to: + +- **MCP (Model Context Protocol)** format +- **OpenAI function calling** format +- Other LLM provider formats + +You never write these formats manually - just write `.bas` files! + +## Complete Example + +Here's a real tool from the codebase - `enrollment.bas`: + +```bas +' Student enrollment tool +PARAM student_name AS STRING +PARAM course AS STRING +PARAM email AS STRING + +' Validate email +IF NOT email CONTAINS "@" THEN + TALK "Please provide a valid email address." + RETURN +ENDIF + +' Create enrollment record +enrollment_id = CALL "/database/insert", "enrollments", { + "student_name": student_name, + "course": course, + "email": email, + "enrolled_at": NOW() +} + +' Send confirmation email +CALL "/email/send", email, "Enrollment Confirmed", + "Welcome " + student_name + "! You're enrolled in " + course + +TALK "Enrollment complete! Confirmation sent to " + email +``` + +## That's It! + +To create a tool: +1. โœ… Create a `.bas` file +2. โœ… Add `PARAM` declarations if you need parameters +3. โœ… Write your logic using `TALK`, `HEAR`, `CALL`, etc. +4. โœ… Done! + +The LLM will automatically: +- Discover your tool +- Understand what it does (from comments and code) +- Know when to call it +- Pass the right parameters + +No JSON schemas, no manual registration, no complex configuration. Just write BASIC! + +## Best Practices + +### 1. Add Comments + +The LLM reads your comments to understand the tool: + +```bas +' This tool books a meeting room +' It checks availability and sends calendar invites +PARAM room_name AS STRING +PARAM date AS STRING +PARAM attendees AS ARRAY +``` + +### 2. Validate Parameters + +Always validate input: + +```bas +IF room_name IS NULL THEN + TALK "Please specify which room you want to book." + RETURN +ENDIF +``` + +### 3. Provide Feedback + +Let users know what's happening: + +```bas +TALK "Checking room availability..." +available = CALL "/calendar/check", room_name, date + +IF available THEN + TALK "Great! Booking the room now..." + CALL "/calendar/book", room_name, date, attendees + TALK "Meeting room booked successfully!" +ELSE + TALK "Sorry, that room is not available on " + date +ENDIF +``` + +## Tool Discovery + +The LLM discovers tools by: +1. **Reading `.bas` files** in your `.gbdialog/` folder +2. **Extracting comments** to understand purpose +3. **Parsing PARAM declarations** to understand parameters +4. **Building a function signature** automatically + +Example tool discovery from `send-email.bas`: + +``` +Function: send-email +Description: Send an email to someone +Parameters: + - to: STRING (required) + - subject: STRING (required) + - body: STRING (required) +``` + +This is generated automatically from your `.bas` file! + +## Removing Tools + +```bas +' Remove a specific tool +REMOVE_TOOL "send-email" + +' Clear all tools +CLEAR_TOOLS + +' List active tools +tools = LIST_TOOLS +TALK "Available tools: " + tools +``` + +## Next Steps + +- [PARAM Declaration](./param-declaration.md) - Parameter types and validation +- [GET Keyword Integration](./get-integration.md) - Using GET to call tools +- [External APIs](./external-apis.md) - Calling external services \ No newline at end of file