481 lines
No EOL
15 KiB
Markdown
481 lines
No EOL
15 KiB
Markdown
# BotTest - Comprehensive Test Infrastructure
|
|
|
|
**Version:** 6.1.0
|
|
**Status:** Production-ready test framework
|
|
**Architecture:** Isolated ephemeral environments with real services
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
BotTest provides enterprise-grade testing infrastructure for the General Bots ecosystem. Each test run creates a completely isolated environment with real PostgreSQL, MinIO, and Redis instances on dynamic ports, ensuring zero state pollution between tests and enabling full parallel execution.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Test Harness │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ ./tmp/bottest-{uuid}/ │
|
|
│ ├── postgres/ (data + socket) │
|
|
│ ├── minio/ (buckets) │
|
|
│ ├── redis/ (dump.rdb) │
|
|
│ └── logs/ (service logs) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ Dynamic Port Allocation (49152-65535) │
|
|
│ ├── PostgreSQL :random │
|
|
│ ├── MinIO API :random │
|
|
│ ├── MinIO Console :random │
|
|
│ └── Redis :random │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ Mock Servers (wiremock) │
|
|
│ ├── LLM API (OpenAI-compatible) │
|
|
│ ├── WhatsApp (Business API) │
|
|
│ ├── Teams (Bot Framework) │
|
|
│ └── Zitadel (Auth/OIDC) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Core Components
|
|
|
|
### Test Harness
|
|
|
|
Orchestrates complete test lifecycle with automatic cleanup:
|
|
|
|
```rust
|
|
pub struct TestHarness {
|
|
pub id: Uuid,
|
|
pub root_dir: PathBuf,
|
|
pub postgres: PostgresService,
|
|
pub minio: MinioService,
|
|
pub redis: RedisService,
|
|
pub mocks: MockRegistry,
|
|
}
|
|
|
|
impl TestHarness {
|
|
pub async fn new(config: TestConfig) -> Result<Self>;
|
|
pub async fn with_botserver(&self) -> Result<BotServerProcess>;
|
|
pub fn connection_string(&self) -> String;
|
|
pub fn s3_endpoint(&self) -> String;
|
|
}
|
|
|
|
impl Drop for TestHarness {
|
|
fn drop(&mut self) {
|
|
// Graceful shutdown + cleanup ./tmp/bottest-{uuid}/
|
|
}
|
|
}
|
|
```
|
|
|
|
### Service Management
|
|
|
|
Real services via botserver bootstrap (no Docker dependency):
|
|
|
|
```rust
|
|
impl PostgresService {
|
|
pub async fn start(port: u16, data_dir: &Path) -> Result<Self>;
|
|
pub async fn run_migrations(&self) -> Result<()>;
|
|
pub async fn create_database(&self, name: &str) -> Result<()>;
|
|
pub async fn execute(&self, sql: &str) -> Result<()>;
|
|
pub fn connection_string(&self) -> String;
|
|
}
|
|
|
|
impl MinioService {
|
|
pub async fn start(api_port: u16, console_port: u16, data_dir: &Path) -> Result<Self>;
|
|
pub async fn create_bucket(&self, name: &str) -> Result<()>;
|
|
pub fn endpoint(&self) -> String;
|
|
pub fn credentials(&self) -> (String, String);
|
|
}
|
|
|
|
impl RedisService {
|
|
pub async fn start(port: u16, data_dir: &Path) -> Result<Self>;
|
|
pub fn connection_string(&self) -> String;
|
|
}
|
|
```
|
|
|
|
### Mock Servers
|
|
|
|
Flexible expectation-based mocking:
|
|
|
|
```rust
|
|
impl MockLLM {
|
|
pub async fn start(port: u16) -> Result<Self>;
|
|
pub fn expect_completion(&mut self, prompt_contains: &str, response: &str) -> &mut Self;
|
|
pub fn expect_streaming(&mut self, chunks: Vec<&str>) -> &mut Self;
|
|
pub fn expect_embedding(&mut self, dimensions: usize) -> &mut Self;
|
|
pub fn with_latency(&mut self, ms: u64) -> &mut Self;
|
|
pub fn with_error_rate(&mut self, rate: f32) -> &mut Self;
|
|
pub fn verify(&self) -> Result<()>;
|
|
}
|
|
|
|
impl MockWhatsApp {
|
|
pub async fn start(port: u16) -> Result<Self>;
|
|
pub fn expect_send_message(&mut self, to: &str) -> MessageExpectation;
|
|
pub fn expect_send_template(&mut self, name: &str) -> TemplateExpectation;
|
|
pub fn simulate_incoming(&self, from: &str, text: &str) -> Result<()>;
|
|
pub fn simulate_webhook(&self, event: WebhookEvent) -> Result<()>;
|
|
}
|
|
|
|
impl MockZitadel {
|
|
pub async fn start(port: u16) -> Result<Self>;
|
|
pub fn expect_login(&mut self, user: &str, password: &str) -> TokenResponse;
|
|
pub fn expect_token_refresh(&mut self) -> &mut Self;
|
|
pub fn expect_introspect(&mut self, token: &str, active: bool) -> &mut Self;
|
|
pub fn create_test_user(&mut self, email: &str) -> User;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Test Categories
|
|
|
|
### Unit Tests
|
|
|
|
Fast, isolated, no external services:
|
|
|
|
```rust
|
|
#[test]
|
|
fn test_basic_parser() {
|
|
let ast = parse("TALK \"Hello\"").unwrap();
|
|
assert_eq!(ast.statements.len(), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn test_config_csv_parsing() {
|
|
let config = ConfigManager::from_str("name,value\nllm-model,test.gguf");
|
|
assert_eq!(config.get("llm-model"), Some("test.gguf"));
|
|
}
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
Real services, isolated environment:
|
|
|
|
```rust
|
|
#[tokio::test]
|
|
async fn test_database_operations() {
|
|
let harness = TestHarness::new(TestConfig::default()).await.unwrap();
|
|
|
|
harness.postgres.execute("INSERT INTO users (email) VALUES ('test@example.com')").await.unwrap();
|
|
|
|
let result = harness.postgres.query_one("SELECT email FROM users").await.unwrap();
|
|
assert_eq!(result.get::<_, String>("email"), "test@example.com");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_file_storage() {
|
|
let harness = TestHarness::new(TestConfig::default()).await.unwrap();
|
|
|
|
harness.minio.create_bucket("test-bucket").await.unwrap();
|
|
harness.minio.put_object("test-bucket", "file.txt", b"content").await.unwrap();
|
|
|
|
let data = harness.minio.get_object("test-bucket", "file.txt").await.unwrap();
|
|
assert_eq!(data, b"content");
|
|
}
|
|
```
|
|
|
|
### Bot Conversation Tests
|
|
|
|
Simulate full conversation flows:
|
|
|
|
```rust
|
|
#[tokio::test]
|
|
async fn test_greeting_flow() {
|
|
let harness = TestHarness::new(TestConfig::with_llm_mock()).await.unwrap();
|
|
|
|
harness.mocks.llm.expect_completion("greeting", "Hello! How can I help?");
|
|
|
|
let mut conv = ConversationTest::new(&harness, "test-bot").await.unwrap();
|
|
|
|
conv.user_says("Hi").await;
|
|
conv.assert_response_contains("Hello").await;
|
|
conv.assert_response_contains("help").await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_knowledge_base_search() {
|
|
let harness = TestHarness::new(TestConfig::with_kb()).await.unwrap();
|
|
|
|
harness.seed_kb("products", vec![
|
|
("SKU-001", "Widget Pro - Premium quality widget"),
|
|
("SKU-002", "Widget Basic - Entry level widget"),
|
|
]).await.unwrap();
|
|
|
|
let mut conv = ConversationTest::new(&harness, "kb-bot").await.unwrap();
|
|
|
|
conv.user_says("Tell me about Widget Pro").await;
|
|
conv.assert_response_contains("Premium quality").await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_human_handoff() {
|
|
let harness = TestHarness::new(TestConfig::default()).await.unwrap();
|
|
|
|
let mut conv = ConversationTest::new(&harness, "support-bot").await.unwrap();
|
|
|
|
conv.user_says("I want to speak to a human").await;
|
|
conv.assert_transferred_to_human().await;
|
|
conv.assert_queue_position(1).await;
|
|
}
|
|
```
|
|
|
|
### Attendance Module Tests
|
|
|
|
Multi-user concurrent scenarios:
|
|
|
|
```rust
|
|
#[tokio::test]
|
|
async fn test_queue_ordering() {
|
|
let harness = TestHarness::new(TestConfig::default()).await.unwrap();
|
|
|
|
let customer1 = harness.create_customer("customer1@test.com").await;
|
|
let customer2 = harness.create_customer("customer2@test.com").await;
|
|
let attendant = harness.create_attendant("agent@test.com").await;
|
|
|
|
harness.enter_queue(&customer1, Priority::Normal).await;
|
|
harness.enter_queue(&customer2, Priority::High).await;
|
|
|
|
let next = harness.get_next_in_queue(&attendant).await.unwrap();
|
|
assert_eq!(next.customer_id, customer2.id); // High priority first
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_concurrent_assignment() {
|
|
let harness = TestHarness::new(TestConfig::default()).await.unwrap();
|
|
|
|
let customers: Vec<_> = (0..10).map(|i|
|
|
harness.create_customer(&format!("c{}@test.com", i))
|
|
).collect();
|
|
|
|
let attendants: Vec<_> = (0..3).map(|i|
|
|
harness.create_attendant(&format!("a{}@test.com", i))
|
|
).collect();
|
|
|
|
// Concurrent assignment - no race conditions
|
|
let assignments = join_all(customers.iter().map(|c|
|
|
harness.auto_assign(c)
|
|
)).await;
|
|
|
|
// Verify no double-assignments
|
|
let assigned_attendants: HashSet<_> = assignments.iter()
|
|
.filter_map(|a| a.as_ref().ok())
|
|
.map(|a| a.attendant_id)
|
|
.collect();
|
|
|
|
assert!(assignments.iter().all(|a| a.is_ok()));
|
|
}
|
|
```
|
|
|
|
### E2E Browser Tests
|
|
|
|
Full stack with real browser:
|
|
|
|
```rust
|
|
#[tokio::test]
|
|
async fn test_chat_interface() {
|
|
let harness = TestHarness::new(TestConfig::full_stack()).await.unwrap();
|
|
let server = harness.with_botserver().await.unwrap();
|
|
|
|
let browser = Browser::new_headless().await.unwrap();
|
|
let page = browser.new_page().await.unwrap();
|
|
|
|
page.goto(&format!("{}/chat/test-bot", server.url())).await.unwrap();
|
|
page.wait_for("#chat-input").await.unwrap();
|
|
|
|
page.fill("#chat-input", "Hello").await.unwrap();
|
|
page.click("#send-button").await.unwrap();
|
|
|
|
page.wait_for(".bot-message").await.unwrap();
|
|
let response = page.text(".bot-message").await.unwrap();
|
|
|
|
assert!(response.contains("Hello"));
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_attendant_dashboard() {
|
|
let harness = TestHarness::new(TestConfig::full_stack()).await.unwrap();
|
|
let server = harness.with_botserver().await.unwrap();
|
|
|
|
let browser = Browser::new_headless().await.unwrap();
|
|
let page = browser.new_page().await.unwrap();
|
|
|
|
// Login as attendant
|
|
page.goto(&format!("{}/login", server.url())).await.unwrap();
|
|
page.fill("#email", "attendant@test.com").await.unwrap();
|
|
page.fill("#password", "testpass").await.unwrap();
|
|
page.click("#login-button").await.unwrap();
|
|
|
|
page.wait_for(".queue-panel").await.unwrap();
|
|
|
|
// Verify queue display
|
|
let queue_count = page.text(".queue-count").await.unwrap();
|
|
assert_eq!(queue_count, "0");
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Fixtures
|
|
|
|
### Data Factories
|
|
|
|
```rust
|
|
pub mod fixtures {
|
|
pub fn admin_user() -> User {
|
|
User {
|
|
id: Uuid::new_v4(),
|
|
email: "admin@test.com".into(),
|
|
role: Role::Admin,
|
|
..Default::default()
|
|
}
|
|
}
|
|
|
|
pub fn customer(phone: &str) -> Customer {
|
|
Customer {
|
|
id: Uuid::new_v4(),
|
|
phone: phone.into(),
|
|
channel: Channel::WhatsApp,
|
|
..Default::default()
|
|
}
|
|
}
|
|
|
|
pub fn bot_with_kb(name: &str) -> Bot {
|
|
Bot {
|
|
id: Uuid::new_v4(),
|
|
name: name.into(),
|
|
kb_enabled: true,
|
|
..Default::default()
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### BASIC Script Fixtures
|
|
|
|
```
|
|
fixtures/scripts/
|
|
├── greeting.bas # Simple greeting flow
|
|
├── kb_search.bas # Knowledge base integration
|
|
├── attendance.bas # Human handoff flow
|
|
├── error_handling.bas # ON ERROR RESUME NEXT patterns
|
|
├── llm_tools.bas # LLM with tool calls
|
|
├── data_operations.bas # FIND, SAVE, UPDATE, DELETE
|
|
└── http_integration.bas # POST, GET, GRAPHQL, SOAP
|
|
```
|
|
|
|
---
|
|
|
|
## CI/CD Integration
|
|
|
|
### GitHub Actions
|
|
|
|
```yaml
|
|
name: Tests
|
|
on: [push, pull_request]
|
|
|
|
jobs:
|
|
unit:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- run: cargo test --lib --workspace
|
|
|
|
integration:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- run: cargo test -p bottest --test integration -- --test-threads=4
|
|
|
|
e2e:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- run: npx playwright install chromium
|
|
- run: cargo test -p bottest --test e2e
|
|
|
|
coverage:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- run: cargo llvm-cov --workspace --lcov --output-path lcov.info
|
|
- uses: codecov/codecov-action@v3
|
|
```
|
|
|
|
---
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Run all tests
|
|
cargo test -p bottest
|
|
|
|
# Unit tests only (fast, no services)
|
|
cargo test -p bottest --lib
|
|
|
|
# Integration tests (starts real services)
|
|
cargo test -p bottest --test integration
|
|
|
|
# E2E tests (starts browser)
|
|
cargo test -p bottest --test e2e
|
|
|
|
# Specific test
|
|
cargo test -p bottest test_queue_ordering
|
|
|
|
# With visible browser for debugging
|
|
HEADED=1 cargo test -p bottest --test e2e
|
|
|
|
# Parallel execution (default)
|
|
cargo test -p bottest -- --test-threads=8
|
|
|
|
# Keep test environment for inspection
|
|
KEEP_ENV=1 cargo test -p bottest test_name
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation Status
|
|
|
|
| Component | Status | Notes |
|
|
|-----------|--------|-------|
|
|
| Test Harness | ✅ Complete | Ephemeral environments working |
|
|
| Port Allocation | ✅ Complete | Dynamic 49152-65535 range |
|
|
| PostgreSQL Service | ✅ Complete | Via botserver bootstrap |
|
|
| MinIO Service | ✅ Complete | Via botserver bootstrap |
|
|
| Redis Service | ✅ Complete | Via botserver bootstrap |
|
|
| Cleanup | ✅ Complete | Drop trait + signal handlers |
|
|
| Mock LLM | ✅ Complete | OpenAI-compatible |
|
|
| Mock WhatsApp | ✅ Complete | Business API |
|
|
| Mock Zitadel | ✅ Complete | OIDC/Auth |
|
|
| Conversation Tests | ✅ Complete | Full flow simulation |
|
|
| BASIC Runner | ✅ Complete | Direct script execution |
|
|
| Fixtures | ✅ Complete | Users, bots, sessions |
|
|
| Browser Automation | ✅ Complete | fantoccini/WebDriver |
|
|
| Attendance Tests | ✅ Complete | Multi-user scenarios |
|
|
| CI Integration | ✅ Complete | GitHub Actions |
|
|
| Coverage Reports | ✅ Complete | cargo-llvm-cov |
|
|
|
|
---
|
|
|
|
## Performance
|
|
|
|
| Test Type | Count | Duration | Parallel |
|
|
|-----------|-------|----------|----------|
|
|
| Unit | 450+ | ~5s | Yes |
|
|
| Integration | 120+ | ~45s | Yes |
|
|
| E2E | 35+ | ~90s | Limited |
|
|
| **Total** | **605+** | **< 3 min** | - |
|
|
|
|
---
|
|
|
|
## Coverage Targets
|
|
|
|
| Module | Current | Target |
|
|
|--------|---------|--------|
|
|
| botserver/src/basic | 82% | 85% |
|
|
| botserver/src/attendance | 91% | 95% |
|
|
| botserver/src/llm | 78% | 80% |
|
|
| botserver/src/core | 75% | 80% |
|
|
| **Overall** | **79%** | **80%** | |