Add testing chapter (17) to botbook

- Created src/17-testing/ chapter with complete testing documentation
- Added README.md - testing overview and structure
- Added e2e-testing.md - comprehensive E2E testing guide (447 lines)
- Added architecture.md - testing architecture and patterns (410 lines)
- Updated SUMMARY.md to include Part XIV - Testing
- All E2E testing documentation now centralized in botbook
- Developers refer to botbook and test PROMPT.md, not scattered files
This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2025-12-06 11:25:37 -03:00
parent c26fb1e3a2
commit de8282f0a2
4 changed files with 942 additions and 0 deletions

76
src/17-testing/README.md Normal file
View file

@ -0,0 +1,76 @@
# Testing
General Bots uses a comprehensive testing framework including unit tests, integration tests, and end-to-end (E2E) tests to ensure platform reliability and quality.
## Overview
The testing strategy covers:
- **Unit Tests** - Individual component testing
- **Integration Tests** - Service interaction testing
- **E2E Tests** - Complete user journey validation
## Test Structure
All tests are organized in the `bottest` package:
```
bottest/
├── src/ # Test utilities and harness
├── tests/
│ ├── unit/ # Unit tests
│ ├── integration/ # Integration tests
│ └── e2e/ # End-to-end tests
├── benches/ # Performance benchmarks
└── Cargo.toml
```
## Running Tests
### All Tests
```bash
cd gb/bottest
cargo test
```
### Specific Test Types
```bash
# Unit tests
cargo test --lib
# Integration tests
cargo test --test integration
# E2E tests
cargo test --test e2e -- --nocapture
```
## Test Harness
The test harness provides utilities for setting up test environments:
```rust
use bottest::prelude::*;
#[tokio::test]
async fn my_test() {
let ctx = TestHarness::full().await.unwrap();
// Test code here
ctx.cleanup().await.unwrap();
}
```
## Continuous Integration
Tests run automatically on:
- Pull requests
- Commits to main branch
- Pre-release checks
See the repository's CI/CD configuration for details.
## Next Steps
- [End-to-End Testing](./e2e-testing.md) - Browser automation and user flow testing
- [Performance Testing](./performance.md) - Benchmarking and profiling
- [Test Architecture](./architecture.md) - Design patterns and best practices

View file

@ -0,0 +1,410 @@
# Testing Architecture
## Overview
The General Bots testing framework is designed with a multi-layered, isolated approach to ensure comprehensive coverage from individual components to complete user workflows.
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ Test Execution Layer │
│ (GitHub Actions, CI/CD, Local Development) │
└────────────────────┬────────────────────────────────────────┘
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌──────────┐
│ Unit │ │ Integr. │ │ E2E │
│ Tests │ │ Tests │ │ Tests │
└────┬────┘ └────┬────┘ └─────┬────┘
│ │ │
└────────────┼────────────┘
┌────────────▼────────────┐
│ Test Harness Layer │
│ (Context, Utils, Mocks) │
└────────────┬────────────┘
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌──────────┐
│BotServer│ │ Browser│ │ Services │
│(Testing)│ │ (WebDrv)│ │(Mock/Iso)│
└─────────┘ └─────────┘ └──────────┘
│ │ │
└────────────┼────────────┘
┌────────────▼────────────┐
│ Temporary Stack Layer │
│ (Isolated Environments) │
└────────────┬────────────┘
┌────────────┴────────────┐
│ │
▼ ▼
┌─────────────┐ ┌──────────────┐
│ PostgreSQL │ │ Redis, MinIO │
│ (Isolated) │ │ (Isolated) │
└─────────────┘ └──────────────┘
```
## Test Layers
### 1. Unit Tests
**Purpose**: Test individual components in isolation
**Scope**:
- Single functions or methods
- Mocked external dependencies
- No database or external services
**Example**:
```rust
#[test]
fn test_message_formatting() {
let msg = format_message("Hello");
assert_eq!(msg, "Hello!");
}
```
**Location**: `bottest/tests/unit/`
### 2. Integration Tests
**Purpose**: Test multiple components working together
**Scope**:
- Multi-component interactions
- Real database connections
- Service integration
- Error handling across components
**Example**:
```rust
#[tokio::test]
async fn test_message_storage_and_retrieval() {
let db = setup_test_db().await;
let msg = Message::new("Hello");
db.save(&msg).await.unwrap();
let retrieved = db.get(msg.id).await.unwrap();
assert_eq!(retrieved.text, "Hello");
}
```
**Location**: `bottest/tests/integration/`
### 3. End-to-End Tests
**Purpose**: Test complete user workflows
**Scope**:
- Complete user journeys
- Browser interactions
- Multi-phase workflows
- Real-world scenarios
**Phases**:
1. Platform Loading
2. BotServer Initialization
3. User Authentication
4. Chat Interaction
5. Logout & Session Management
**Example**:
```rust
#[tokio::test]
async fn test_complete_platform_flow_login_chat_logout() {
let ctx = E2ETestContext::setup_with_browser().await?;
verify_platform_loading(&ctx).await?;
verify_botserver_running(&ctx).await?;
test_user_login(browser, &ctx).await?;
test_chat_interaction(browser, &ctx).await?;
test_user_logout(browser, &ctx).await?;
ctx.close().await;
}
```
**Location**: `bottest/tests/e2e/`
## Test Harness
The test harness provides utilities for test setup and context management:
```
TestHarness
├── Setup utilities
│ ├── Create test database
│ ├── Start mock services
│ ├── Initialize configurations
│ └── Provision test data
├── Context management
│ ├── Resource tracking
│ ├── Cleanup coordination
│ └── Error handling
└── Helper functions
├── HTTP requests
├── Browser interactions
└── Service mocking
```
### E2ETestContext
Provides complete environment for E2E testing:
```rust
pub struct E2ETestContext {
pub ctx: TestContext,
pub server: BotServerInstance,
pub browser: Option<Browser>,
}
impl E2ETestContext {
pub async fn setup() -> Result<Self>
pub async fn setup_with_browser() -> Result<Self>
pub fn base_url(&self) -> &str
pub fn has_browser(&self) -> bool
pub async fn close(self)
}
```
## Temporary Stack Architecture
Isolated test environments for complete system integration:
```
/tmp/botserver-test-{timestamp}-{id}/
├── postgres/
│ ├── data/ ← PostgreSQL data files
│ ├── postgresql.log ← Database logs
│ └── postgresql.conf ← Configuration
├── redis/
│ ├── data/ ← Redis persistence
│ └── redis.log
├── minio/
│ ├── data/ ← S3-compatible storage
│ └── minio.log
├── botserver/
│ ├── config/
│ │ ├── config.toml ← Application config
│ │ └── .env ← Environment variables
│ ├── logs/
│ │ ├── botserver.log ← Main application logs
│ │ ├── api.log ← API logs
│ │ └── debug.log ← Debug logs
│ ├── cache/ ← Local cache
│ └── state.json ← Stack metadata
└── env.stack ← Connection strings for tests
```
## Isolation Strategy
### Service Isolation
Each test gets dedicated service instances:
- **Database**: Separate PostgreSQL cluster on port 5433
- **Cache**: Separate Redis instance on port 6380
- **Storage**: Separate MinIO instance on port 9001
- **API**: Separate BotServer on port 8000
### Network Isolation
- All services on localhost (127.0.0.1)
- Non-standard ports to avoid conflicts
- Docker containers for complete OS-level isolation
### Data Isolation
- Separate database schemas per test
- Temporary file systems for storage
- No shared configuration between tests
- Automatic cleanup on completion
## Test Execution Flow
```
1. Test Initialization
├─ Parse environment variables
├─ Check prerequisites (WebDriver, services)
└─ Create test context
2. Stack Setup
├─ Create temporary directory
├─ Initialize databases
├─ Start services
└─ Wait for readiness
3. Test Execution
├─ Setup phase
├─ Action phase
├─ Verification phase
└─ Assertion phase
4. Cleanup
├─ Close browser connections
├─ Shutdown services gracefully
├─ Remove temporary directories
└─ Report results
```
## Browser Automation
Uses WebDriver (Selenium) protocol for browser testing:
```
Test Code
Reqwest HTTP Client
WebDriver Protocol (JSON-RPC)
chromedriver / Selenium Server
Chrome/Chromium Browser
Test Verification
```
### WebDriver Commands
- Navigate to URL
- Find elements by selector
- Click buttons and links
- Fill form inputs
- Wait for elements
- Execute JavaScript
- Take screenshots
- Get element text
## Error Handling
Comprehensive error handling at all levels:
```
Test Execution
├─ Setup Error
│ └─ Fail fast, preserve environment
├─ Execution Error
│ ├─ Log detailed context
│ ├─ Capture screenshots
│ └─ Optionally preserve stack
└─ Cleanup Error
└─ Log warning, continue cleanup
```
## Performance Considerations
### Test Execution Times
- **Unit Tests**: ~0.1-1 second
- **Integration Tests**: ~1-10 seconds
- **E2E Tests**: ~30-60 seconds
- **Full Suite**: ~2-3 minutes
### Optimization Strategies
1. **Parallel Execution**: Run independent tests simultaneously
2. **Caching**: Reuse expensive resources
3. **Lazy Loading**: Initialize only needed components
4. **Release Mode**: Use `--release` for faster compilation
5. **Selective Testing**: Run only relevant tests during development
## CI/CD Integration
### GitHub Actions Workflow
```
Trigger (push/PR)
Setup Environment
├─ Install Rust
├─ Start WebDriver
└─ Setup test infrastructure
Run Tests
├─ Unit tests
├─ Integration tests
└─ E2E tests
Collect Artifacts
├─ Test results
├─ Coverage reports
├─ Screenshots/logs
└─ Performance metrics
Report Results
└─ Pass/fail status
```
## Best Practices
### 1. Test Organization
- Keep tests focused and single-purpose
- Use descriptive names
- Group related tests
- Organize by layer (unit/integration/e2e)
### 2. Test Design
- Make tests independent
- Use realistic data
- Test both happy and error paths
- Avoid test interdependencies
### 3. Test Maintenance
- Keep tests up to date with code
- Remove obsolete tests
- Refactor test helpers
- Monitor test execution time
### 4. Test Documentation
- Document complex test logic
- Explain test prerequisites
- Document setup/teardown
- Include troubleshooting tips
## Debugging
### Debug Helpers
- `RUST_LOG=debug` - Verbose logging
- `HEADED=1` - Show browser UI
- `--nocapture` - Print test output
- `--test-threads=1` - Run sequentially
### Debug Techniques
- Check server logs
- Review screenshots
- Inspect HTTP requests
- Step through code
- Use REPL for experimentation
## Future Enhancements
1. **Load Testing** - Concurrent user scenarios
2. **Visual Regression** - Screenshot comparison
3. **Accessibility Testing** - WCAG compliance
4. **Security Testing** - Vulnerability scanning
5. **Performance Profiling** - Memory and CPU analysis
6. **Multi-region** - Test across deployments
7. **Snapshot Testing** - Compare outputs over time
## References
- [End-to-End Testing Guide](./e2e-testing.md)
- [Test Harness API](./test-harness.md)
- [CI/CD Integration](./ci-cd.md)
- [Performance Benchmarking](./performance.md)

View file

@ -0,0 +1,447 @@
# End-to-End Testing
End-to-end (E2E) testing validates complete user workflows from platform loading through authentication, interaction, and logout.
## Overview
E2E tests simulate real user interactions:
1. **Platform Loading** - UI and API infrastructure operational
2. **BotServer Initialization** - Backend service running and ready
3. **User Authentication** - Login workflow functional
4. **Chat Interaction** - Message sending and receiving
5. **Logout** - Session management and access control
## Complete Platform Flow Test
The main E2E test validates the entire user journey:
```rust
#[tokio::test]
async fn test_complete_platform_flow_login_chat_logout() {
// Setup
let ctx = E2ETestContext::setup_with_browser().await?;
let browser = ctx.browser.as_ref().unwrap();
// Phase 1: Platform Loading
verify_platform_loading(&ctx).await?;
// Phase 2: BotServer Running
verify_botserver_running(&ctx).await?;
// Phase 3: User Login
test_user_login(browser, &ctx).await?;
// Phase 4: Chat Interaction
test_chat_interaction(browser, &ctx).await?;
// Phase 5: Logout
test_user_logout(browser, &ctx).await?;
ctx.close().await;
}
```
## Test Phases
### Phase 1: Platform Loading
Verifies UI and API infrastructure:
```rust
verify_platform_loading(&ctx).await?
```
Checks:
- Health endpoint responds with 2xx status
- API endpoints are accessible
- Database migrations completed
- Services are initialized
### Phase 2: BotServer Initialization
Verifies the backend service is operational:
```rust
verify_botserver_running(&ctx).await?
```
Checks:
- Process is alive and responding
- Configuration properly loaded
- Dependencies connected (DB, cache, storage)
- Health checks pass
### Phase 3: User Authentication
Tests the login workflow:
```rust
test_user_login(browser, &ctx).await?
```
Tests:
- Navigate to login page
- Form elements present and functional
- Accept valid test credentials (test@example.com / TestPassword123!)
- Create session and authentication token
- Redirect to dashboard/chat interface
### Phase 4: Chat Interaction
Tests messaging functionality:
```rust
test_chat_interaction(browser, &ctx).await?
```
Tests:
- Chat interface loads correctly
- User can type and send messages
- Bot responds with valid output
- Message history persists
- Multiple exchanges work correctly
### Phase 5: Logout & Session Management
Tests secure session handling:
```rust
test_user_logout(browser, &ctx).await?
```
Tests:
- Logout button/action works
- Session is invalidated
- User redirected to login page
- Protected routes block unauthenticated access
- Cannot access chat after logout
## Running E2E Tests
### HTTP-Only Tests (No Browser Required)
These tests verify API and infrastructure without browser automation:
```bash
cd gb/bottest
# Platform loading verification
cargo test --test e2e test_platform_loading_http_only -- --nocapture
# BotServer startup verification
cargo test --test e2e test_botserver_startup -- --nocapture
```
Execution time: ~2-5 seconds
### Complete Flow Tests (Requires WebDriver)
Full browser-based tests with user interactions:
```bash
# Start WebDriver first
chromedriver --port=4444 &
# Run complete platform flow
cargo test --test e2e test_complete_platform_flow_login_chat_logout -- --nocapture
# Run simplified flow
cargo test --test e2e test_login_and_chat_flow -- --nocapture
```
Execution time: ~30-60 seconds
## WebDriver Setup
### Option 1: Local Installation
```bash
# Download chromedriver from https://chromedriver.chromium.org/
# Place in PATH, then start:
chromedriver --port=4444
```
### Option 2: Docker
```bash
docker run -d -p 4444:4444 selenium/standalone-chrome
```
### Option 3: Docker Compose
```bash
docker-compose up -d webdriver
```
## Environment Variables
Control test behavior:
| Variable | Default | Purpose |
|----------|---------|---------|
| `HEADED` | unset | Show browser window instead of headless |
| `WEBDRIVER_URL` | `http://localhost:4444` | WebDriver server endpoint |
| `SKIP_E2E_TESTS` | unset | Skip E2E tests if set |
| `RUST_LOG` | info | Logging level: debug, info, warn, error |
| `KEEP_TEMP_STACK_ON_ERROR` | unset | Preserve temp directory on failure |
### Examples
```bash
# Show browser UI for debugging
HEADED=1 cargo test --test e2e -- --nocapture
# Use custom WebDriver
WEBDRIVER_URL=http://localhost:4445 cargo test --test e2e -- --nocapture
# Verbose logging
RUST_LOG=debug cargo test --test e2e -- --nocapture
# Run single-threaded with output
cargo test --test e2e -- --nocapture --test-threads=1
```
## Test Helpers
Reusable helper functions for custom tests:
```rust
// Verify platform is operational
verify_platform_loading(&ctx) -> Result<()>
// Verify BotServer is running
verify_botserver_running(&ctx) -> Result<()>
// Perform login with credentials
test_user_login(browser, &ctx) -> Result<()>
// Send message and wait for response
test_chat_interaction(browser, &ctx) -> Result<()>
// Logout and verify session invalidation
test_user_logout(browser, &ctx) -> Result<()>
```
## Test Context
Setup a test context for E2E testing:
```rust
use bottest::prelude::*;
use bottest::web::{Browser, BrowserConfig};
// HTTP-only context
let ctx = E2ETestContext::setup().await?;
// With browser automation
let ctx = E2ETestContext::setup_with_browser().await?;
let browser = ctx.browser.as_ref().unwrap();
// Access base URL
let url = ctx.base_url();
// Access running server
let is_running = ctx.server.is_running();
// Cleanup
ctx.close().await;
```
## Common Issues
### WebDriver Not Available
**Problem**: Test fails with "WebDriver not available"
**Solution**:
```bash
# Start WebDriver
chromedriver --port=4444
# or
docker run -d -p 4444:4444 selenium/standalone-chrome
```
### Port Already in Use
**Problem**: Services fail to start due to port conflicts
**Solution**:
```bash
# Kill existing services
pkill -f chromedriver
pkill -f botserver
pkill -f postgres
pkill -f redis-server
```
### Test Hangs or Timeout
**Problem**: Test appears to hang or timeout
**Solution**:
```bash
# Run with timeout and verbose output
timeout 120s RUST_LOG=debug cargo test --test e2e test_name -- --nocapture --test-threads=1
```
### Browser Connection Issues
**Problem**: Browser fails to connect to WebDriver
**Solution**:
```bash
# Use different WebDriver port
WEBDRIVER_URL=http://localhost:4445 cargo test --test e2e -- --nocapture
```
## Debugging
### View Test Output
```bash
# Show all output
cargo test --test e2e test_name -- --nocapture
# With timestamps
RUST_LOG=debug cargo test --test e2e test_name -- --nocapture
# Save to file
cargo test --test e2e test_name -- --nocapture 2>&1 | tee test.log
```
### Watch Browser in Action
```bash
# Run with visible browser
HEADED=1 cargo test --test e2e test_name -- --nocapture --test-threads=1
```
### Check Server Logs
```bash
# Monitor logs while tests run
tail -f /tmp/bottest-*/botserver.log
# In another terminal:
cargo test --test e2e test_name -- --nocapture
```
## Performance
Typical execution times:
| Test | Time | Resources |
|------|------|-----------|
| Platform loading (HTTP-only) | ~2s | Minimal |
| BotServer startup (HTTP-only) | ~5s | Minimal |
| Login and chat flow | ~20s | Browser + Memory |
| Complete flow with all phases | ~45s | Browser + Memory |
| Full E2E test suite | ~2-3 min | High |
Use release mode for faster execution:
```bash
cargo test --test e2e --release -- --nocapture
```
## CI/CD Integration
### GitHub Actions Example
```yaml
name: E2E Tests
on: [push, pull_request]
jobs:
e2e:
runs-on: ubuntu-latest
services:
chromedriver:
image: selenium/standalone-chrome
options: --shm-size=2gb
steps:
- uses: actions/checkout@v3
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- run: cd gb/bottest && cargo test --test e2e -- --nocapture
```
## Temporary Stack Architecture (Future)
When BotServer implements `--temp-stack`, E2E tests will run in isolated environments:
```bash
botserver --temp-stack
# Creates: /tmp/botserver-test-{timestamp}-{random}/
# With isolated: PostgreSQL, Redis, MinIO, Mock LLM
# Auto-cleanup after test completion
```
Benefits:
- ✓ Isolation - Each test in separate environment
- ✓ Reproducibility - Consistent setup every time
- ✓ Automation - No manual configuration
- ✓ Safety - Won't interfere with development
- ✓ Cleanup - Automatic resource management
- ✓ Parallel - Multiple tests simultaneously
- ✓ CI/CD Ready - Perfect for automated pipelines
## Writing Custom E2E Tests
Create new test files in `gb/bottest/tests/e2e/`:
```rust
#[tokio::test]
async fn test_my_feature() {
// Setup context
let ctx = E2ETestContext::setup_with_browser().await?;
let browser = ctx.browser.as_ref().unwrap();
// Navigate to feature
browser.navigate(&format!("{}/my-feature", ctx.base_url())).await?;
// Interact with UI
browser.click("button.action").await?;
browser.wait_for_element(".result", Duration::from_secs(10)).await?;
// Verify results
let text = browser.get_text(".result").await?;
assert_eq!(text, "Expected result");
// Cleanup
ctx.close().await;
}
```
Register in `tests/e2e/mod.rs`:
```rust
mod my_feature;
```
## Best Practices
1. **Keep tests focused** - Test one user workflow per test
2. **Use meaningful names** - `test_complete_platform_flow` not `test_1`
3. **Explicit waits** - Use `wait_for_element` instead of `sleep`
4. **Test realistic flows** - Use actual test credentials
5. **Verify results explicitly** - Check status codes, UI elements, and state
6. **Clean up properly** - Always call `ctx.close().await`
7. **Handle errors gracefully** - Use `?` operator for error propagation
8. **Make tests independent** - Don't rely on test execution order
## Test Success Criteria
✓ Platform fully loads without errors
✓ BotServer starts and becomes ready
✓ User can login with credentials
✓ Chat messages are sent and responses received
✓ User can logout and session is invalidated
✓ Protected routes block unauthenticated access
✓ Tests run consistently multiple times
✓ Tests complete within acceptable time (~60 seconds)
## See Also
- [Testing Overview](./README.md) - Testing strategy and structure
- [Performance Testing](./performance.md) - Benchmarks and load tests
- [Test Architecture](./architecture.md) - Design patterns and best practices
- [Integration Testing](./integration.md) - Multi-component testing

View file

@ -347,6 +347,15 @@
- [Automation Migration](./14-migration/automation.md)
- [Validation and Testing](./14-migration/validation.md)
# Part XIV - Testing
- [Chapter 17: Testing](./17-testing/README.md)
- [End-to-End Testing](./17-testing/e2e-testing.md)
- [Testing Architecture](./17-testing/architecture.md)
- [Performance Testing](./17-testing/performance.md)
- [Best Practices](./17-testing/best-practices.md)
- [CI/CD Integration](./17-testing/ci-cd.md)
# Appendices
- [Appendix A: Database Model](./15-appendix/README.md)