bottest/TASKS.md

BotTest - Comprehensive Test Infrastructure

Version: 6.1.0
Status: Production-ready test framework
Architecture: Isolated ephemeral environments with real services

---

Overview

BotTest provides enterprise-grade testing infrastructure for the General Bots ecosystem. Each test run creates a completely isolated environment with real PostgreSQL, MinIO, and Redis instances on dynamic ports, ensuring zero state pollution between tests and enabling full parallel execution.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Test Harness                              │
├─────────────────────────────────────────────────────────────────┤
│  ./tmp/bottest-{uuid}/                                          │
│  ├── postgres/     (data + socket)                              │
│  ├── minio/        (buckets)                                    │
│  ├── redis/        (dump.rdb)                                   │
│  └── logs/         (service logs)                               │
├─────────────────────────────────────────────────────────────────┤
│  Dynamic Port Allocation (49152-65535)                          │
│  ├── PostgreSQL    :random                                      │
│  ├── MinIO API     :random                                      │
│  ├── MinIO Console :random                                      │
│  └── Redis         :random                                      │
├─────────────────────────────────────────────────────────────────┤
│  Mock Servers (wiremock)                                        │
│  ├── LLM API       (OpenAI-compatible)                          │
│  ├── WhatsApp      (Business API)                               │
│  ├── Teams         (Bot Framework)                              │
│  └── Zitadel       (Auth/OIDC)                                  │
└─────────────────────────────────────────────────────────────────┘

---

Core Components

Test Harness

Orchestrates complete test lifecycle with automatic cleanup:

pub struct TestHarness {
    pub id: Uuid,
    pub root_dir: PathBuf,
    pub postgres: PostgresService,
    pub minio: MinioService,
    pub redis: RedisService,
    pub mocks: MockRegistry,
}

impl TestHarness {
    pub async fn new(config: TestConfig) -> Result<Self>;
    pub async fn with_botserver(&self) -> Result<BotServerProcess>;
    pub fn connection_string(&self) -> String;
    pub fn s3_endpoint(&self) -> String;
}

impl Drop for TestHarness {
    fn drop(&mut self) {
        // Graceful shutdown + cleanup ./tmp/bottest-{uuid}/
    }
}

Service Management

Real services via botserver bootstrap (no Docker dependency):

impl PostgresService {
    pub async fn start(port: u16, data_dir: &Path) -> Result<Self>;
    pub async fn run_migrations(&self) -> Result<()>;
    pub async fn create_database(&self, name: &str) -> Result<()>;
    pub async fn execute(&self, sql: &str) -> Result<()>;
    pub fn connection_string(&self) -> String;
}

impl MinioService {
    pub async fn start(api_port: u16, console_port: u16, data_dir: &Path) -> Result<Self>;
    pub async fn create_bucket(&self, name: &str) -> Result<()>;
    pub fn endpoint(&self) -> String;
    pub fn credentials(&self) -> (String, String);
}

impl RedisService {
    pub async fn start(port: u16, data_dir: &Path) -> Result<Self>;
    pub fn connection_string(&self) -> String;
}

Mock Servers

Flexible expectation-based mocking:

impl MockLLM {
    pub async fn start(port: u16) -> Result<Self>;
    pub fn expect_completion(&mut self, prompt_contains: &str, response: &str) -> &mut Self;
    pub fn expect_streaming(&mut self, chunks: Vec<&str>) -> &mut Self;
    pub fn expect_embedding(&mut self, dimensions: usize) -> &mut Self;
    pub fn with_latency(&mut self, ms: u64) -> &mut Self;
    pub fn with_error_rate(&mut self, rate: f32) -> &mut Self;
    pub fn verify(&self) -> Result<()>;
}

impl MockWhatsApp {
    pub async fn start(port: u16) -> Result<Self>;
    pub fn expect_send_message(&mut self, to: &str) -> MessageExpectation;
    pub fn expect_send_template(&mut self, name: &str) -> TemplateExpectation;
    pub fn simulate_incoming(&self, from: &str, text: &str) -> Result<()>;
    pub fn simulate_webhook(&self, event: WebhookEvent) -> Result<()>;
}

impl MockZitadel {
    pub async fn start(port: u16) -> Result<Self>;
    pub fn expect_login(&mut self, user: &str, password: &str) -> TokenResponse;
    pub fn expect_token_refresh(&mut self) -> &mut Self;
    pub fn expect_introspect(&mut self, token: &str, active: bool) -> &mut Self;
    pub fn create_test_user(&mut self, email: &str) -> User;
}

---

Test Categories

Unit Tests

Fast, isolated, no external services:

#[test]
fn test_basic_parser() {
    let ast = parse("TALK \"Hello\"").unwrap();
    assert_eq!(ast.statements.len(), 1);
}

#[test]
fn test_config_csv_parsing() {
    let config = ConfigManager::from_str("name,value\nllm-model,test.gguf");
    assert_eq!(config.get("llm-model"), Some("test.gguf"));
}

Integration Tests

Real services, isolated environment:

#[tokio::test]
async fn test_database_operations() {
    let harness = TestHarness::new(TestConfig::default()).await.unwrap();
    
    harness.postgres.execute("INSERT INTO users (email) VALUES ('test@example.com')").await.unwrap();
    
    let result = harness.postgres.query_one("SELECT email FROM users").await.unwrap();
    assert_eq!(result.get::<_, String>("email"), "test@example.com");
}

#[tokio::test]
async fn test_file_storage() {
    let harness = TestHarness::new(TestConfig::default()).await.unwrap();
    
    harness.minio.create_bucket("test-bucket").await.unwrap();
    harness.minio.put_object("test-bucket", "file.txt", b"content").await.unwrap();
    
    let data = harness.minio.get_object("test-bucket", "file.txt").await.unwrap();
    assert_eq!(data, b"content");
}

Bot Conversation Tests

Simulate full conversation flows:

#[tokio::test]
async fn test_greeting_flow() {
    let harness = TestHarness::new(TestConfig::with_llm_mock()).await.unwrap();
    
    harness.mocks.llm.expect_completion("greeting", "Hello! How can I help?");
    
    let mut conv = ConversationTest::new(&harness, "test-bot").await.unwrap();
    
    conv.user_says("Hi").await;
    conv.assert_response_contains("Hello").await;
    conv.assert_response_contains("help").await;
}

#[tokio::test]
async fn test_knowledge_base_search() {
    let harness = TestHarness::new(TestConfig::with_kb()).await.unwrap();
    
    harness.seed_kb("products", vec![
        ("SKU-001", "Widget Pro - Premium quality widget"),
        ("SKU-002", "Widget Basic - Entry level widget"),
    ]).await.unwrap();
    
    let mut conv = ConversationTest::new(&harness, "kb-bot").await.unwrap();
    
    conv.user_says("Tell me about Widget Pro").await;
    conv.assert_response_contains("Premium quality").await;
}

#[tokio::test]
async fn test_human_handoff() {
    let harness = TestHarness::new(TestConfig::default()).await.unwrap();
    
    let mut conv = ConversationTest::new(&harness, "support-bot").await.unwrap();
    
    conv.user_says("I want to speak to a human").await;
    conv.assert_transferred_to_human().await;
    conv.assert_queue_position(1).await;
}

Attendance Module Tests

Multi-user concurrent scenarios:

#[tokio::test]
async fn test_queue_ordering() {
    let harness = TestHarness::new(TestConfig::default()).await.unwrap();
    
    let customer1 = harness.create_customer("customer1@test.com").await;
    let customer2 = harness.create_customer("customer2@test.com").await;
    let attendant = harness.create_attendant("agent@test.com").await;
    
    harness.enter_queue(&customer1, Priority::Normal).await;
    harness.enter_queue(&customer2, Priority::High).await;
    
    let next = harness.get_next_in_queue(&attendant).await.unwrap();
    assert_eq!(next.customer_id, customer2.id); // High priority first
}

#[tokio::test]
async fn test_concurrent_assignment() {
    let harness = TestHarness::new(TestConfig::default()).await.unwrap();
    
    let customers: Vec<_> = (0..10).map(|i| 
        harness.create_customer(&format!("c{}@test.com", i))
    ).collect();
    
    let attendants: Vec<_> = (0..3).map(|i|
        harness.create_attendant(&format!("a{}@test.com", i))
    ).collect();
    
    // Concurrent assignment - no race conditions
    let assignments = join_all(customers.iter().map(|c| 
        harness.auto_assign(c)
    )).await;
    
    // Verify no double-assignments
    let assigned_attendants: HashSet<_> = assignments.iter()
        .filter_map(|a| a.as_ref().ok())
        .map(|a| a.attendant_id)
        .collect();
    
    assert!(assignments.iter().all(|a| a.is_ok()));
}

E2E Browser Tests

Full stack with real browser:

#[tokio::test]
async fn test_chat_interface() {
    let harness = TestHarness::new(TestConfig::full_stack()).await.unwrap();
    let server = harness.with_botserver().await.unwrap();
    
    let browser = Browser::new_headless().await.unwrap();
    let page = browser.new_page().await.unwrap();
    
    page.goto(&format!("{}/chat/test-bot", server.url())).await.unwrap();
    page.wait_for("#chat-input").await.unwrap();
    
    page.fill("#chat-input", "Hello").await.unwrap();
    page.click("#send-button").await.unwrap();
    
    page.wait_for(".bot-message").await.unwrap();
    let response = page.text(".bot-message").await.unwrap();
    
    assert!(response.contains("Hello"));
}

#[tokio::test]
async fn test_attendant_dashboard() {
    let harness = TestHarness::new(TestConfig::full_stack()).await.unwrap();
    let server = harness.with_botserver().await.unwrap();
    
    let browser = Browser::new_headless().await.unwrap();
    let page = browser.new_page().await.unwrap();
    
    // Login as attendant
    page.goto(&format!("{}/login", server.url())).await.unwrap();
    page.fill("#email", "attendant@test.com").await.unwrap();
    page.fill("#password", "testpass").await.unwrap();
    page.click("#login-button").await.unwrap();
    
    page.wait_for(".queue-panel").await.unwrap();
    
    // Verify queue display
    let queue_count = page.text(".queue-count").await.unwrap();
    assert_eq!(queue_count, "0");
}

---

Fixtures

Data Factories

pub mod fixtures {
    pub fn admin_user() -> User {
        User {
            id: Uuid::new_v4(),
            email: "admin@test.com".into(),
            role: Role::Admin,
            ..Default::default()
        }
    }
    
    pub fn customer(phone: &str) -> Customer {
        Customer {
            id: Uuid::new_v4(),
            phone: phone.into(),
            channel: Channel::WhatsApp,
            ..Default::default()
        }
    }
    
    pub fn bot_with_kb(name: &str) -> Bot {
        Bot {
            id: Uuid::new_v4(),
            name: name.into(),
            kb_enabled: true,
            ..Default::default()
        }
    }
}

BASIC Script Fixtures

fixtures/scripts/
├── greeting.bas          # Simple greeting flow
├── kb_search.bas         # Knowledge base integration
├── attendance.bas        # Human handoff flow
├── error_handling.bas    # ON ERROR RESUME NEXT patterns
├── llm_tools.bas         # LLM with tool calls
├── data_operations.bas   # FIND, SAVE, UPDATE, DELETE
└── http_integration.bas  # POST, GET, GRAPHQL, SOAP

---

CI/CD Integration

GitHub Actions

name: Tests
on: [push, pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo test --lib --workspace

  integration:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo test -p bottest --test integration -- --test-threads=4

  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright install chromium
      - run: cargo test -p bottest --test e2e

  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo llvm-cov --workspace --lcov --output-path lcov.info
      - uses: codecov/codecov-action@v3

---

Usage

# Run all tests
cargo test -p bottest

# Unit tests only (fast, no services)
cargo test -p bottest --lib

# Integration tests (starts real services)
cargo test -p bottest --test integration

# E2E tests (starts browser)
cargo test -p bottest --test e2e

# Specific test
cargo test -p bottest test_queue_ordering

# With visible browser for debugging
HEADED=1 cargo test -p bottest --test e2e

# Parallel execution (default)
cargo test -p bottest -- --test-threads=8

# Keep test environment for inspection
KEEP_ENV=1 cargo test -p bottest test_name

---

Implementation Status

Component	Status	Notes
Test Harness	✅ Complete	Ephemeral environments working
Port Allocation	✅ Complete	Dynamic 49152-65535 range
PostgreSQL Service	✅ Complete	Via botserver bootstrap
MinIO Service	✅ Complete	Via botserver bootstrap
Redis Service	✅ Complete	Via botserver bootstrap
Cleanup	✅ Complete	Drop trait + signal handlers
Mock LLM	✅ Complete	OpenAI-compatible
Mock WhatsApp	✅ Complete	Business API
Mock Zitadel	✅ Complete	OIDC/Auth
Conversation Tests	✅ Complete	Full flow simulation
BASIC Runner	✅ Complete	Direct script execution
Fixtures	✅ Complete	Users, bots, sessions
Browser Automation	✅ Complete	fantoccini/WebDriver
Attendance Tests	✅ Complete	Multi-user scenarios
CI Integration	✅ Complete	GitHub Actions
Coverage Reports	✅ Complete	cargo-llvm-cov

---

Performance

Test Type	Count	Duration	Parallel
Unit	450+	~5s	Yes
Integration	120+	~45s	Yes
E2E	35+	~90s	Limited
Total	605+	< 3 min	-

---

Coverage Targets

Module	Current	Target
botserver/src/basic	82%	85%
botserver/src/attendance	91%	95%
botserver/src/llm	78%	80%
botserver/src/core	75%	80%
Overall	79%	80%