botserver/docs/src/chapter-10-features/ai-llm.md

AI and LLM

BotServer integrates with Large Language Models (LLMs) to provide intelligent conversational capabilities and natural language understanding.

Overview

The LLM integration in BotServer enables:

• Natural language conversations
• Context-aware responses
• Tool discovery and invocation
• Document understanding
• Text generation and summarization

LLM Providers

OpenAI

Primary LLM provider with support for:

• GPT-3.5 Turbo
• GPT-4
• GPT-4 Turbo
• Custom fine-tuned models

Configuration:

OPENAI_API_KEY=your-api-key
LLM_MODEL=gpt-4

Local Models

Support for self-hosted models:

• Llama.cpp compatible servers
• Custom inference endpoints
• Privacy-preserving deployments

Configuration:

LLM_PROVIDER=local
LLM_ENDPOINT=http://localhost:8081

The LLM Keyword

Basic Usage

Basic Generation (Background Processing)

# For background/scheduled tasks only - not for interactive conversations
let summary = LLM "Explain quantum computing in simple terms"
SET_BOT_MEMORY "quantum_explanation", summary  # Store for all users

With Context

Document Summarization (Background Processing)

# Scheduled task to generate summaries for all users
let document = GET "knowledge/policy.pdf"
let summary = LLM "Summarize this document: " + document
SET_BOT_MEMORY "policy_summary", summary  # Available to all users

Question Answering

Context-Aware Conversations (Interactive)

# For interactive conversations - use SET CONTEXT, not LLM
TALK "What's your question?"
let question = HEAR
let context = GET_BOT_MEMORY("knowledge")
SET CONTEXT "background", context
TALK "Based on our knowledge base, here's what I can tell you..."
# System AI automatically uses the context when responding

LLM Provider Implementation

Located in src/llm/:

• mod.rs - Provider trait and factory
• openai.rs - OpenAI implementation
• local.rs - Local model support

Provider Trait

All LLM providers implement:

• generate() - Text generation
• generate_stream() - Streaming responses
• get_embedding() - Vector embeddings
• count_tokens() - Token counting

Context Management

Context Window

Managing limited context size:

• Automatic truncation
• Context compaction
• Relevance filtering
• History summarization

Context Sources

1. Conversation History - Recent messages
2. Knowledge Base - Relevant documents
3. Bot Memory - Persistent context
4. Tool Definitions - Available functions
5. User Profile - Personalization

Prompt Engineering

System Prompts

Configured in bot memory:

let system_prompt = GET_BOT_MEMORY("system_prompt")
SET_CONTEXT "system" AS system_prompt

Dynamic Prompts

Building prompts programmatically:

# For interactive conversations - use SET CONTEXT
SET CONTEXT "user_name", user_name
SET CONTEXT "current_date", NOW()
# System AI automatically incorporates this context

Streaming Responses

WebSocket Streaming

Real-time token streaming:

1. LLM generates tokens
2. Tokens sent via WebSocket
3. UI updates progressively
4. Complete response assembled

Stream Control

• Start/stop generation
• Cancel long responses
• Timeout protection
• Error recovery

Embeddings

Vector Generation

Creating embeddings for semantic search:

let embedding = llm_provider.get_embedding(text).await?;

Embedding Models

• OpenAI: text-embedding-ada-002
• Local: Sentence transformers
• Custom: Configurable models

Token Management

Token Counting

Estimating usage before calls:

let token_count = llm_provider.count_tokens(prompt)?;

Token Limits

• Model-specific limits
• Context window constraints
• Rate limiting
• Cost management

Error Handling

Common Errors

• API key invalid
• Rate limit exceeded
• Context too long
• Model unavailable
• Network timeout

Fallback Strategies

1. Retry with backoff
2. Switch to backup model
3. Reduce context size
4. Cache responses
5. Return graceful error

Performance Optimization

Caching

• Response caching
• Embedding caching
• Token count caching
• Semantic cache

Batching

• Group similar requests
• Batch embeddings
• Parallel processing
• Queue management

Cost Management

Usage Tracking

• Tokens per request
• Cost per conversation
• Daily/monthly limits
• Per-user quotas

Optimization Strategies

1. Use smaller models when possible
2. Cache frequent queries
3. Compress context
4. Limit conversation length
5. Implement quotas

Security Considerations

API Key Management

• Never hardcode keys
• Use environment variables
• Rotate keys regularly
• Monitor usage

Content Filtering

• Input validation
• Output sanitization
• PII detection
• Inappropriate content blocking

Monitoring

Metrics

• Response time
• Token usage
• Error rate
• Cache hit rate
• Model performance

Logging

• Request/response pairs
• Error details
• Performance metrics
• Usage statistics

Best Practices

1. Choose Appropriate Models: Balance cost and capability
2. Optimize Prompts: Clear, concise instructions
3. Manage Context: Keep relevant information only
4. Handle Errors: Graceful degradation
5. Monitor Usage: Track costs and performance
6. Cache Wisely: Reduce redundant calls
7. Stream When Possible: Better user experience

Summary

The LLM integration is central to BotServer's intelligence, providing natural language understanding and generation capabilities. Through careful prompt engineering, context management, and provider abstraction, bots can deliver sophisticated conversational experiences while managing costs and performance.