CI: Rename workflow to force refresh
This commit is contained in:
parent
45cb0c6371
commit
3f5e8eee1c
6 changed files with 899 additions and 13 deletions
|
|
@ -439,7 +439,7 @@ END SWITCH
|
|||
### Reading This Workspace
|
||||
/opt/gbo/data is a place also for bots.
|
||||
**For LLMs analyzing this codebase:**
|
||||
0. Bots are in /opt/gbo/data primary
|
||||
0. Bots are in drive, each bucket is a bot. Respect LOAD_ONLY.
|
||||
1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)** in README to understand relationships
|
||||
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)** for what each module does
|
||||
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)** to understand execution flow
|
||||
|
|
@ -1868,4 +1868,3 @@ AutoTask is an AI-driven task execution system that:
|
|||
| `USER` | Current user object |
|
||||
| `SESSION` | Current session object |
|
||||
| `BOT` | Current bot object |
|
||||
|
||||
|
|
|
|||
132
PROD.md
132
PROD.md
|
|
@ -18,7 +18,7 @@ The host machine is accessed via `ssh user@<hostname>`, running Incus (an LXD fo
|
|||
|
||||
| Container | Service | Technology | Binary Path | Logs Path | Data Path | Notes |
|
||||
|-----------|---------|------------|-------------|-----------|-----------|-------|
|
||||
| **system** | BotServer + BotUI | Rust/Axum | `/opt/gbo/bin/botserver`<br>`/opt/gbo/bin/botui` | `/opt/gbo/logs/out.log`<br>`/opt/gbo/logs/err.log` | `/opt/gbo/data/`<br>`/opt/gbo/work/` | Main API + UI proxy |
|
||||
| **system** | BotServer + BotUI | Rust/Axum | `/opt/gbo/bin/botserver`<br>`/opt/gbo/bin/botui` | `/opt/gbo/logs/out.log`<br>`/opt/gbo/logs/err.log` | `/opt/gbo/work/` | Main API + UI proxy |
|
||||
| **tables** | PostgreSQL | PostgreSQL 15+ | `/usr/lib/postgresql/*/bin/postgres` | `/opt/gbo/logs/postgresql/` | `/opt/gbo/data/pgdata/` | Primary database |
|
||||
| **vault** | HashiCorp Vault | Vault | `/opt/gbo/bin/vault` | `/opt/gbo/logs/vault/` | `/opt/gbo/data/vault/` | Secrets management |
|
||||
| **cache** | Valkey | Valkey (Redis fork) | `/opt/gbo/bin/valkey-server` | `/opt/gbo/logs/valkey/` | `/opt/gbo/data/valkey/` | Distributed cache |
|
||||
|
|
@ -305,7 +305,7 @@ sudo incus exec system -- systemctl start botserver
|
|||
|
||||
## Services Detail
|
||||
|
||||
Botserver runs as user `gbuser`, binary at `/opt/gbo/bin/botserver`, logs at `/opt/gbo/logs/out.log` and `/opt/gbo/logs/err.log`, systemd unit at `/etc/systemd/system/botserver.service`, env loaded from `/opt/gbo/bin/.env`. Bot BASIC scripts live under `/opt/gbo/data/<botname>.gbai/<botname>.gbdialog/*.bas`; compiled AST cache goes to `/opt/gbo/work/`.
|
||||
Botserver runs as user `gbuser`, binary at `/opt/gbo/bin/botserver`, logs at `/opt/gbo/logs/out.log` and `/opt/gbo/logs/err.log`, systemd unit at `/etc/systemd/system/botserver.service`, env loaded from `/opt/gbo/bin/.env`. Bot BASIC scripts are stored in MinIO Drive under `{bot}.gbai/{bot}.gbdialog/*.bas` and are downloaded/compiled by DriveMonitor to `/opt/gbo/work/{bot}.gbai/{bot}.gbdialog/*.ast`.
|
||||
|
||||
The directory service runs Zitadel as user `root`, binary at `/opt/gbo/bin/zitadel`, logs at `/opt/gbo/logs/zitadel.log`, systemd unit at `/etc/systemd/system/directory.service`, and loads environment from the service configuration. Zitadel provides identity management and OAuth2 services for the platform.
|
||||
|
||||
|
|
@ -407,6 +407,36 @@ sudo incus exec system -- /opt/gbo/bin/botserver --version 2>&1 | head -3
|
|||
sudo incus exec system -- tail -5 /opt/gbo/logs/err.log
|
||||
```
|
||||
|
||||
### Monitor CI/CD Build Status
|
||||
|
||||
**Check latest build status:**
|
||||
```bash
|
||||
# View latest 3 builds with status
|
||||
sudo incus exec alm -- bash -c 'cd /opt/gbo/data/GeneralBots/BotServer/actions/runs && for dir in $(ls -t | head -3); do echo "=== Build $dir ==="; cat $dir/jobs/0.json 2>/dev/null | grep -E "\"status\"|\"commit\"|\"workflow\"" | head -5; done'
|
||||
|
||||
# Watch runner logs in real-time
|
||||
sudo incus exec alm-ci -- tail -f /opt/gbo/logs/forgejo-runner.log | grep -E "Clone|Build|Deploy|Success|Failure"
|
||||
```
|
||||
|
||||
**Understand build timing:**
|
||||
- **Rust compilation**: 2-5 minutes (cold build), 30-60 seconds (incremental)
|
||||
- **Dependencies**: First build downloads ~200 dependencies
|
||||
- **Deploy step**: ~5 seconds
|
||||
- **Total CI time**: 2-6 minutes depending on cache
|
||||
|
||||
**Verify binary was updated:**
|
||||
```bash
|
||||
# Check binary timestamp
|
||||
ssh administrator@63.141.255.9 "sudo incus exec system -- stat -c '%y' /opt/gbo/bin/botserver"
|
||||
|
||||
# Check running version
|
||||
ssh administrator@63.141.255.9 "sudo incus exec system -- /opt/gbo/bin/botserver --version"
|
||||
|
||||
# Check health endpoint
|
||||
curl -sf https://chat.pragmatismo.com.br/api/health || echo "Health check failed"
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## DriveMonitor & Bot Configuration
|
||||
|
|
@ -464,19 +494,105 @@ ssh user@<hostname> "sudo incus exec system -- curl -s --cacert /opt/gbo/conf/sy
|
|||
|
||||
---
|
||||
|
||||
## DNS Management
|
||||
|
||||
### Updating DNS Records
|
||||
|
||||
**CRITICAL:** When updating DNS zone files, you MUST:
|
||||
|
||||
1. **Update the serial number** in the SOA record (format: YYYYMMDDNN)
|
||||
2. **Run sync-zones.sh** to propagate changes to secondary nameservers
|
||||
3. **Anonymize IPs and credentials** in all documentation and logs
|
||||
|
||||
**Workflow:**
|
||||
```bash
|
||||
# 1. Edit zone file
|
||||
sudo incus exec dns -- nano /opt/gbo/data/pragmatismo.com.br.zone
|
||||
|
||||
# 2. Update serial (YYYYMMDDNN format)
|
||||
# Example: 2026041801 (April 18, 2026, change #1)
|
||||
sudo incus exec dns -- sed -i 's/2026041801/2026041802/' /opt/gbo/data/pragmatismo.com.br.zone
|
||||
|
||||
# 3. Reload CoreDNS
|
||||
sudo incus exec dns -- pkill -HUP coredns
|
||||
|
||||
# 4. Sync to secondary NS
|
||||
sudo /opt/gbo/bin/sync-zones.sh
|
||||
|
||||
# 5. Verify on secondary
|
||||
ssh -o StrictHostKeyChecking=no -i /home/administrator/.ssh/id_ed25519 admin@<secondary-ip> 'getent hosts <domain>'
|
||||
```
|
||||
|
||||
**Zone File Location:** `/opt/gbo/data/<domain>.zone` in the `dns` container
|
||||
|
||||
**Sync Script:** `/opt/gbo/bin/sync-zones.sh` - copies zone files to secondary NS (3.218.224.38)
|
||||
|
||||
**⚠️ Security Rules:**
|
||||
- NEVER include real IPs in documentation - use `<ip>` or `10.x.x.x`
|
||||
- NEVER include credentials - use `<password>` or `<token>`
|
||||
- NEVER commit zone files with secrets to git
|
||||
|
||||
---
|
||||
|
||||
### Adding New Subdomains (HTTPS with Caddy)
|
||||
|
||||
**CRITICAL:** When adding new subdomains that need HTTPS, follow this order:
|
||||
|
||||
1. **Add DNS record FIRST** (see above workflow)
|
||||
2. **Wait for DNS propagation** (can take up to 1 hour)
|
||||
3. **Add Caddy config** - Caddy will automatically obtain Let's Encrypt certificate
|
||||
|
||||
**Complete Workflow:**
|
||||
```bash
|
||||
# 1. Add DNS record (update serial, sync zones)
|
||||
sudo incus exec dns -- nano /opt/gbo/data/pragmatismo.com.br.zone
|
||||
# Add: news IN A <ip>
|
||||
sudo incus exec dns -- sed -i 's/2026041801/2026041802/' /opt/gbo/data/pragmatismo.com.br.zone
|
||||
sudo incus exec dns -- pkill -HUP coredns
|
||||
sudo /opt/gbo/bin/sync-zones.sh
|
||||
|
||||
# 2. Verify DNS propagation (wait until this works)
|
||||
dig @9.9.9.9 news.pragmatismo.com.br A +short
|
||||
|
||||
# 3. Add Caddy config (AFTER DNS is working)
|
||||
sudo sh -c 'cat >> /opt/gbo/conf/config << EOF
|
||||
|
||||
news.pragmatismo.com.br {
|
||||
import tls_config
|
||||
reverse_proxy http://<container-ip>:<port> {
|
||||
header_up Host {host}
|
||||
header_up X-Real-IP {remote}
|
||||
header_up X-Forwarded-Proto https
|
||||
}
|
||||
}
|
||||
EOF'
|
||||
|
||||
# 4. Restart Caddy
|
||||
sudo incus exec proxy -- systemctl restart proxy
|
||||
|
||||
# 5. Wait for certificate (Caddy will auto-obtain from Let's Encrypt)
|
||||
# Check logs: sudo incus exec proxy -- tail -f /opt/gbo/logs/access.log
|
||||
```
|
||||
|
||||
**⚠️ Important:**
|
||||
- Caddy will fail to obtain certificate if DNS is not propagated
|
||||
- Wait up to 1 hour for DNS propagation before adding Caddy config
|
||||
- Check Caddy logs for "challenge failed" errors - indicates DNS not ready
|
||||
- Certificate is automatically renewed by Caddy
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Quick Reference
|
||||
|
||||
**GLIBC mismatch (`GLIBC_2.39 not found`):** The binary was compiled on the CI runner (glibc 2.41) not inside the system container (glibc 2.36). The CI workflow must SSH into the system container to build. Check `botserver.yaml` to confirm this.
|
||||
|
||||
**botserver won't start:** Run `sudo incus exec system -- ldd /opt/gbo/bin/botserver | grep "not found"` to check for missing libraries. Run `sudo incus exec system -- timeout 10 /opt/gbo/bin/botserver 2>&1` to see startup errors. Confirm `/opt/gbo/data/` exists and is accessible.
|
||||
**botserver won't start:** Run `sudo incus exec system -- ldd /opt/gbo/bin/botserver | grep "not found"` to check for missing libraries. Run `sudo incus exec system -- timeout 10 /opt/gbo/bin/botserver 2>&1` to see startup errors. Confirm `/opt/gbo/work/` exists and is accessible.
|
||||
|
||||
**botui can't reach botserver:** Check that the `ui.service` systemd file has `BOTSERVER_URL=http://localhost:5858` — not the external HTTPS URL. Fix with `sed -i 's|BOTSERVER_URL=.*|BOTSERVER_URL=http://localhost:5858|'` on the service file, then `systemctl daemon-reload` and `systemctl restart ui`.
|
||||
|
||||
**Suggestions not showing:** Confirm bot `.bas` files exist under `/opt/gbo/data/<bot>.gbai/<bot>.gbdialog/`. Check logs for compilation errors. Clear the AST cache in `/opt/gbo/work/` and restart botserver.
|
||||
**Suggestions not showing:** Confirm bot `.bas` files exist in MinIO Drive under `{bot}.gbai/{bot}.gbdialog/`. Check logs for compilation errors. Clear the AST cache in `/opt/gbo/work/` and restart botserver.
|
||||
|
||||
**IPv6 DNS timeouts on external APIs (Groq, Cloudflare):** The container's DNS may return AAAA records without IPv6 connectivity. The container should have `IPV6=no` in its network config and `gai.conf` set appropriately. Check for `RES_OPTIONS=inet4` in `botserver.service` if issues persist.
|
||||
|
||||
**Logs show development paths instead of `/opt/gbo/data/`:** Botserver is using hardcoded dev paths. Check `.env` has `DATA_DIR=/opt/gbo/data/` and `WORK_DIR=/opt/gbo/work/`, verify the systemd unit has `EnvironmentFile=/opt/gbo/bin/.env`, and confirm Vault is reachable so service discovery works. Expected startup log lines include `info watcher:Watching data directory /opt/gbo/data` and `info botserver:BotServer started successfully on port 5858`.
|
||||
**Logs show development paths instead of Drive:** Botserver is using hardcoded dev paths. Check `.env` has `DATA_DIR=/opt/gbo/work/` and `WORK_DIR=/opt/gbo/work/`, verify the systemd unit has `EnvironmentFile=/opt/gbo/bin/.env`, and confirm Vault is reachable so service discovery works. Expected startup log lines include `info watcher:Watching data directory /opt/gbo/work` and `info botserver:BotServer started successfully on port 5858`.
|
||||
|
||||
**Migrations not running after push:** If `stat /opt/gbo/bin/botserver` shows old timestamp and `__diesel_schema_migrations` table has no new entries, CI did not rebuild. Make a trivial code change (e.g., add a comment) in botserver and push again to force rebuild.
|
||||
|
||||
|
|
@ -669,7 +785,7 @@ sudo incus exec alm-ci -- ls -la /opt/gbo/
|
|||
|
||||
# Fix ownership
|
||||
sudo incus exec alm-ci -- chown -R gbuser:gbuser /opt/gbo/bin/
|
||||
sudo incus exec alm-ci -- chown -R gbuser:gbuser /opt/gbo/data/
|
||||
sudo incus exec alm-ci -- chown -R gbuser:gbuser /opt/gbo/work/
|
||||
```
|
||||
|
||||
**CI Runner Down:**
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
Subproject commit 4518bf7bd1c4ae6fafd5c42fa5f4515a383b480a
|
||||
Subproject commit 4083fa2d0f40fa83b367f6d2aa0776c1aec1dd5e
|
||||
|
|
@ -1 +1 @@
|
|||
Subproject commit fed92acaab8a4c9c9a32aa39db5b4a2bf11daeda
|
||||
Subproject commit 2ff68e5a2a24b7b63386efd2e2325d478efbb925
|
||||
2
botui
2
botui
|
|
@ -1 +1 @@
|
|||
Subproject commit d49eeab1c54bc8a51db249814d2476f7eabdd851
|
||||
Subproject commit 54cf998885676143859a48b7cc424bb2c2d9ebd6
|
||||
771
crates.tmp
Normal file
771
crates.tmp
Normal file
|
|
@ -0,0 +1,771 @@
|
|||
# BotServer Microservices Partition Plan
|
||||
|
||||
## 🎯 Objective
|
||||
|
||||
Partition the monolithic `botserver` into multiple microservices as separate Rust crates that share state, making the system easier to compile, maintain, and scale.
|
||||
|
||||
## 📊 Current State Analysis
|
||||
|
||||
### Current Structure
|
||||
- **Single monolithic crate**: `botserver` (242 dependencies in Cargo.toml)
|
||||
- **Feature flags**: 50+ features (chat, mail, tasks, drive, llm, etc.)
|
||||
- **Main entry point**: `botserver/src/main.rs` (436 lines)
|
||||
- **Shared state**: `AppState` in `core::shared::state`
|
||||
- **Modules**: 40+ top-level modules
|
||||
|
||||
### Problems with Current Approach
|
||||
1. **Long compile times**: Full rebuilds take 30+ minutes
|
||||
2. **Memory intensive**: Requires significant RAM during compilation
|
||||
3. **Tight coupling**: All modules compiled together
|
||||
4. **Hard to scale**: Can't deploy individual services independently
|
||||
5. **Feature flag complexity**: Many features, hard to test all combinations
|
||||
|
||||
## 🏗️ Proposed Architecture
|
||||
|
||||
### Shared State Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SHARED STATE LAYER │
|
||||
│ ┌─────────────────────────────────────────────────────────┐│
|
||||
│ │ botstate (NEW) - Shared state types & interfaces ││
|
||||
│ │ - AppState definition ││
|
||||
│ │ - Database connection pool ││
|
||||
│ │ - Redis/Valkey client ││
|
||||
│ │ - Configuration ││
|
||||
│ │ - LLM provider interface ││
|
||||
│ │ - Secret manager interface ││
|
||||
│ └─────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Microservice Crates
|
||||
|
||||
```
|
||||
Workspace Members:
|
||||
├── botlib/ # Existing - Core types, errors, utilities
|
||||
├── botstate/ # NEW - Shared state layer
|
||||
├── botserver-core/ # NEW - Core HTTP server, routing, middleware
|
||||
├── botserver-auth/ # NEW - Authentication & authorization
|
||||
├── botserver-chat/ # NEW - WebSocket chat, LLM integration
|
||||
├── botserver-mail/ # NEW - Email integration (IMAP/SMTP)
|
||||
├── botserver-tasks/ # NEW - Task scheduler, cron jobs
|
||||
├── botserver-drive/ # NEW - File storage (S3/MinIO)
|
||||
├── botserver-llm/ # NEW - LLM providers, embeddings
|
||||
├── botserver-vectordb/ # NEW - Qdrant vector database
|
||||
├── botserver-compliance/# NEW - Legal, audit, compliance
|
||||
├── botserver-video/ # NEW - Video processing, LiveKit
|
||||
├── botserver-contacts/ # NEW - CRM, contacts, calendar
|
||||
├── botserver-auto/ # NEW - Automation, Rhai scripting
|
||||
├── botserver-ui/ # NEW - Embedded UI, TUI (optional)
|
||||
└── botserver/ # EXISTING - Becomes meta-crate that re-exports all
|
||||
```
|
||||
|
||||
## 📦 Detailed Crate Responsibilities
|
||||
|
||||
### 1. **botstate** (NEW - Foundation)
|
||||
**Purpose**: Define shared state types that all services will use
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib` (workspace)
|
||||
- `tokio`, `redis`, `diesel`, `chrono`, `uuid`, `serde`
|
||||
|
||||
**Exports**:
|
||||
```rust
|
||||
pub struct AppState {
|
||||
pub db_pool: PgPool,
|
||||
pub redis_client: Arc<redis::Client>,
|
||||
pub config: Arc<Config>,
|
||||
pub llm_provider: Arc<dyn LLMProvider>,
|
||||
pub secret_manager: Arc<dyn SecretManager>,
|
||||
// ... other shared resources
|
||||
}
|
||||
|
||||
pub trait LLMProvider {
|
||||
async fn generate(&self, prompt: &str) -> Result<String>;
|
||||
}
|
||||
|
||||
pub trait SecretManager {
|
||||
fn get_secret(&self, key: &str) -> Result<String>;
|
||||
}
|
||||
```
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botstate/`
|
||||
|
||||
---
|
||||
|
||||
### 2. **botserver-core** (NEW - HTTP Foundation)
|
||||
**Purpose**: Core HTTP server, routing, middleware, health checks
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`
|
||||
- `axum`, `tower`, `tower-http`, `tokio`
|
||||
|
||||
**Responsibilities**:
|
||||
- HTTP server setup (Axum)
|
||||
- Middleware (CORS, logging, rate limiting)
|
||||
- Health check endpoints
|
||||
- Graceful shutdown
|
||||
- Static file serving
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-core/`
|
||||
|
||||
---
|
||||
|
||||
### 3. **botserver-auth** (NEW - Security)
|
||||
**Purpose**: Authentication, authorization, RBAC
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `jsonwebtoken`, `argon2`, `axum`
|
||||
|
||||
**Responsibilities**:
|
||||
- User authentication (login/logout)
|
||||
- JWT token management
|
||||
- RBAC (role-based access control)
|
||||
- Session management
|
||||
- OAuth integration
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-auth/`
|
||||
|
||||
---
|
||||
|
||||
### 4. **botserver-chat** (NEW - Real-time Communication)
|
||||
**Purpose**: WebSocket chat, message handling, LLM integration
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `tokio-tungstenite`, `axum`, `serde_json`
|
||||
|
||||
**Responsibilities**:
|
||||
- WebSocket connections
|
||||
- Message routing
|
||||
- Chat history
|
||||
- Suggestion buttons
|
||||
- Tool execution framework
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-chat/`
|
||||
|
||||
---
|
||||
|
||||
### 5. **botserver-mail** (NEW - Email)
|
||||
**Purpose**: Email integration (IMAP/SMTP)
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `lettre`, `imap`, `mailparse`
|
||||
|
||||
**Responsibilities**:
|
||||
- IMAP email retrieval
|
||||
- SMTP email sending
|
||||
- Email parsing
|
||||
- Attachments
|
||||
- Email templates
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-mail/`
|
||||
|
||||
---
|
||||
|
||||
### 6. **botserver-tasks** (NEW - Scheduling)
|
||||
**Purpose**: Task scheduler, cron jobs
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `cron`, `tokio`
|
||||
|
||||
**Responsibilities**:
|
||||
- Cron job scheduler
|
||||
- Task queue management
|
||||
- Recurring tasks
|
||||
- Task execution tracking
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-tasks/`
|
||||
|
||||
---
|
||||
|
||||
### 7. **botserver-drive** (NEW - Storage)
|
||||
**Purpose**: File storage (S3/MinIO)
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `aws-sdk-s3`, `aws-config`
|
||||
|
||||
**Responsibilities**:
|
||||
- S3/MinIO client
|
||||
- File upload/download
|
||||
- File synchronization
|
||||
- PDF extraction
|
||||
- Document processing
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-drive/`
|
||||
|
||||
---
|
||||
|
||||
### 8. **botserver-llm** (NEW - Language Models)
|
||||
**Purpose**: LLM providers, embeddings, caching
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`
|
||||
- `reqwest`, `serde_json`
|
||||
|
||||
**Responsibilities**:
|
||||
- LLM provider abstraction (Groq, OpenAI, etc.)
|
||||
- Embedding generation
|
||||
- Response caching
|
||||
- Local LLM management (llama.cpp)
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-llm/`
|
||||
|
||||
---
|
||||
|
||||
### 9. **botserver-vectordb** (NEW - Vector Database)
|
||||
**Purpose**: Qdrant vector database operations
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-llm`
|
||||
- `qdrant-client`
|
||||
|
||||
**Responsibilities**:
|
||||
- Vector embeddings storage
|
||||
- Similarity search
|
||||
- Knowledge base chunks
|
||||
- RAG (retrieval-augmented generation)
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-vectordb/`
|
||||
|
||||
---
|
||||
|
||||
### 10. **botserver-auto** (NEW - Automation)
|
||||
**Purpose**: Automation, Rhai scripting engine
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `rhai`, `tokio`
|
||||
|
||||
**Responsibilities**:
|
||||
- Rhai script engine
|
||||
- BASIC keyword execution
|
||||
- Workflow automation
|
||||
- Event handlers
|
||||
- Script compilation
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-auto/`
|
||||
|
||||
---
|
||||
|
||||
### 11. **botserver-contacts** (NEW - CRM)
|
||||
**Purpose**: Contacts, CRM, calendar integration
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `icalendar`, `chrono`
|
||||
|
||||
**Responsibilities**:
|
||||
- Contact management
|
||||
- CRM operations
|
||||
- Calendar integration
|
||||
- Google/Microsoft sync
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-contacts/`
|
||||
|
||||
---
|
||||
|
||||
### 12. **botserver-compliance** (NEW - Legal)
|
||||
**Purpose**: Legal, audit, compliance
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `csv`
|
||||
|
||||
**Responsibilities**:
|
||||
- Audit logging
|
||||
- Compliance reports
|
||||
- Legal document management
|
||||
- Data retention policies
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-compliance/`
|
||||
|
||||
---
|
||||
|
||||
### 13. **botserver-video** (NEW - Video)
|
||||
**Purpose**: Video processing, LiveKit integration
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`, `botserver-core`
|
||||
- `livekit`
|
||||
|
||||
**Responsibilities**:
|
||||
- LiveKit integration
|
||||
- Video analytics
|
||||
- Meeting management
|
||||
- Media processing
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-video/`
|
||||
|
||||
---
|
||||
|
||||
### 14. **botserver-ui** (OPTIONAL - TUI)
|
||||
**Purpose**: Console/TUI interface
|
||||
|
||||
**Dependencies**:
|
||||
- `botlib`, `botstate`
|
||||
- `ratatui`, `crossterm`
|
||||
|
||||
**Responsibilities**:
|
||||
- Terminal UI
|
||||
- Progress monitoring
|
||||
- Service status dashboard
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver-ui/`
|
||||
|
||||
---
|
||||
|
||||
### 15. **botserver** (EXISTING - Meta-crate)
|
||||
**Purpose**: Re-exports all services, maintains backward compatibility
|
||||
|
||||
**Dependencies**:
|
||||
- All botserver-* crates
|
||||
- Feature flags to enable/disable services
|
||||
|
||||
**Responsibilities**:
|
||||
- Re-export all services
|
||||
- Main binary entry point
|
||||
- Feature flag coordination
|
||||
|
||||
**Location**: `/home/rodriguez/src/gb/botserver/` (existing, modified)
|
||||
|
||||
---
|
||||
|
||||
## 🔄 State Sharing Strategy
|
||||
|
||||
### Approach: Shared AppState via botstate Crate
|
||||
|
||||
```rust
|
||||
// botstate/src/lib.rs
|
||||
use redis::Client as RedisClient;
|
||||
use diesel::r2d2::Pool as DbPool;
|
||||
use std::sync::Arc;
|
||||
|
||||
pub struct AppState {
|
||||
pub db_pool: Arc<DbPool>,
|
||||
pub redis_client: Arc<RedisClient>,
|
||||
pub config: Arc<Config>,
|
||||
pub llm_provider: Arc<dyn LLMProvider + Send + Sync>,
|
||||
pub secret_manager: Arc<dyn SecretManager + Send + Sync>,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
pub fn new(
|
||||
db_pool: DbPool,
|
||||
redis_client: RedisClient,
|
||||
config: Config,
|
||||
llm_provider: Box<dyn LLMProvider + Send + Sync>,
|
||||
secret_manager: Box<dyn SecretManager + Send + Sync>,
|
||||
) -> Self {
|
||||
Self {
|
||||
db_pool: Arc::new(db_pool),
|
||||
redis_client: Arc::new(redis_client),
|
||||
config: Arc::new(config),
|
||||
llm_provider: Arc::from(llm_provider),
|
||||
secret_manager: Arc::from(secret_manager),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Each Service Gets AppState as Parameter
|
||||
|
||||
```rust
|
||||
// botserver-chat/src/lib.rs
|
||||
use botstate::AppState;
|
||||
use axum::Router;
|
||||
|
||||
pub fn create_router(state: Arc<AppState>) -> Router {
|
||||
Router::new()
|
||||
.route("/ws", websocket_handler)
|
||||
.with_state(state)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Migration Steps
|
||||
|
||||
### Phase 1: Foundation (Week 1-2)
|
||||
1. ✅ Create `botstate` crate
|
||||
- Extract `AppState` from `botserver`
|
||||
- Define trait interfaces (LLMProvider, SecretManager)
|
||||
- Export shared types
|
||||
|
||||
2. ✅ Create `botserver-core` crate
|
||||
- Extract HTTP server setup
|
||||
- Extract middleware
|
||||
- Extract health checks
|
||||
|
||||
3. ✅ Update workspace `Cargo.toml`
|
||||
- Add new members
|
||||
- Configure dependencies
|
||||
|
||||
### Phase 2: Extract Services (Week 3-6)
|
||||
4. ✅ Extract `botserver-auth`
|
||||
- Move authentication code
|
||||
- Update imports
|
||||
|
||||
5. ✅ Extract `botserver-chat`
|
||||
- Move WebSocket code
|
||||
- Move message handling
|
||||
|
||||
6. ✅ Extract `botserver-mail`
|
||||
- Move email integration
|
||||
- Move IMAP/SMTP code
|
||||
|
||||
7. ✅ Extract `botserver-drive`
|
||||
- Move S3/MinIO code
|
||||
- Move file sync
|
||||
|
||||
8. ✅ Extract `botserver-auto`
|
||||
- Move Rhai engine
|
||||
- Move BASIC keywords
|
||||
|
||||
### Phase 3: Advanced Services (Week 7-10)
|
||||
9. ✅ Extract `botserver-llm`
|
||||
- Move LLM providers
|
||||
- Move caching
|
||||
|
||||
10. ✅ Extract `botserver-vectordb`
|
||||
- Move Qdrant client
|
||||
- Move embeddings
|
||||
|
||||
11. ✅ Extract `botserver-tasks`
|
||||
- Move scheduler
|
||||
- Move cron jobs
|
||||
|
||||
12. ✅ Extract `botserver-contacts`
|
||||
- Move CRM code
|
||||
- Move calendar sync
|
||||
|
||||
13. ✅ Extract `botserver-compliance`
|
||||
- Move audit logging
|
||||
- Move compliance reports
|
||||
|
||||
14. ✅ Extract `botserver-video`
|
||||
- Move LiveKit integration
|
||||
- Move video analytics
|
||||
|
||||
### Phase 4: Integration & Testing (Week 11-12)
|
||||
15. ✅ Update `botserver` as meta-crate
|
||||
- Re-export all services
|
||||
- Maintain backward compatibility
|
||||
- Feature flags coordination
|
||||
|
||||
16. ✅ Test all services independently
|
||||
- Unit tests
|
||||
- Integration tests
|
||||
- End-to-end tests
|
||||
|
||||
17. ✅ Documentation
|
||||
- Update README
|
||||
- Document APIs
|
||||
- Update AGENTS.md
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Benefits
|
||||
|
||||
### Compile Time Improvements
|
||||
- **Before**: Full rebuild 30+ minutes
|
||||
- **After**:
|
||||
- botstate: ~30 seconds
|
||||
- Individual service: 1-3 minutes
|
||||
- Full workspace: Still 30+ minutes (but cached builds work better)
|
||||
|
||||
### Memory Usage
|
||||
- **Before**: 16GB+ RAM required
|
||||
- **After**: 2-4GB per service
|
||||
|
||||
### Development Workflow
|
||||
- Work on single service without rebuilding everything
|
||||
- Better separation of concerns
|
||||
- Easier to test individual components
|
||||
- Can deploy services independently
|
||||
|
||||
### Scalability
|
||||
- Deploy only needed services
|
||||
- Scale services independently
|
||||
- Different services can use different versions of dependencies
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Challenges & Solutions
|
||||
|
||||
### Challenge 1: Circular Dependencies
|
||||
**Problem**: Services reference each other
|
||||
|
||||
**Solution**:
|
||||
- Use traits in `botstate` for cross-service communication
|
||||
- Dependency injection via `AppState`
|
||||
- Event-driven architecture for loose coupling
|
||||
|
||||
### Challenge 2: Database Migrations
|
||||
**Problem**: Multiple crates need to coordinate migrations
|
||||
|
||||
**Solution**:
|
||||
- Keep all migrations in `botserver` (meta-crate)
|
||||
- Use `diesel_migrations` with centralized migration directory
|
||||
- Each service documents its required migrations
|
||||
|
||||
### Challenge 3: Shared Configuration
|
||||
**Problem**: All services need config values
|
||||
|
||||
**Solution**:
|
||||
- `Config` struct in `botstate`
|
||||
- Loaded once, shared via `Arc<Config>`
|
||||
- Each service reads only what it needs
|
||||
|
||||
### Challenge 4: Feature Flags
|
||||
**Problem**: Complex feature flag interactions
|
||||
|
||||
**Solution**:
|
||||
- Each service has its own feature flags
|
||||
- `botserver` meta-crate coordinates via feature aliases
|
||||
- Default: all services enabled
|
||||
- Can build minimal subset for testing
|
||||
|
||||
---
|
||||
|
||||
## 📊 Dependency Graph
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
botlib[botlib - Core types]
|
||||
botstate[botstate - Shared state]
|
||||
botcore[botserver-core - HTTP server]
|
||||
|
||||
auth[botserver-auth]
|
||||
chat[botserver-chat]
|
||||
mail[botserver-mail]
|
||||
tasks[botserver-tasks]
|
||||
drive[botserver-drive]
|
||||
llm[botserver-llm]
|
||||
vectordb[botserver-vectordb]
|
||||
auto[botserver-auto]
|
||||
contacts[botserver-contacts]
|
||||
compliance[botserver-compliance]
|
||||
video[botserver-video]
|
||||
ui[botserver-ui]
|
||||
|
||||
botserver[botserver - Meta-crate]
|
||||
|
||||
botlib --> botstate
|
||||
botstate --> botcore
|
||||
botcore --> auth
|
||||
botcore --> chat
|
||||
botcore --> mail
|
||||
botcore --> tasks
|
||||
botcore --> drive
|
||||
botcore --> auto
|
||||
botcore --> contacts
|
||||
botcore --> compliance
|
||||
botcore --> video
|
||||
|
||||
llm --> botstate
|
||||
vectordb --> botstate
|
||||
vectordb --> llm
|
||||
|
||||
auth --> botstate
|
||||
chat --> botstate
|
||||
chat --> llm
|
||||
mail --> botstate
|
||||
tasks --> botstate
|
||||
drive --> botstate
|
||||
auto --> botstate
|
||||
contacts --> botstate
|
||||
compliance --> botstate
|
||||
video --> botstate
|
||||
|
||||
botserver --> botcore
|
||||
botserver --> auth
|
||||
botserver --> chat
|
||||
botserver --> mail
|
||||
botserver --> tasks
|
||||
botserver --> drive
|
||||
botserver --> llm
|
||||
botserver --> vectordb
|
||||
botserver --> auto
|
||||
botserver --> contacts
|
||||
botserver --> compliance
|
||||
botserver --> video
|
||||
botserver --> ui
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Structure After Migration
|
||||
|
||||
```
|
||||
gb/
|
||||
├── Cargo.toml (workspace)
|
||||
├── botlib/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
├── botstate/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── state.rs
|
||||
│ ├── traits.rs
|
||||
│ └── config.rs
|
||||
├── botserver-core/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── server.rs
|
||||
│ ├── middleware.rs
|
||||
│ └── health.rs
|
||||
├── botserver-auth/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── jwt.rs
|
||||
│ ├── rbac.rs
|
||||
│ └── session.rs
|
||||
├── botserver-chat/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── websocket.rs
|
||||
│ └── messages.rs
|
||||
├── botserver-mail/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── imap.rs
|
||||
│ └── smtp.rs
|
||||
├── botserver-tasks/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── scheduler.rs
|
||||
│ └── cron.rs
|
||||
├── botserver-drive/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── s3.rs
|
||||
│ └── sync.rs
|
||||
├── botserver-llm/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── provider.rs
|
||||
│ └── cache.rs
|
||||
├── botserver-vectordb/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ └── qdrant.rs
|
||||
├── botserver-auto/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── rhai_engine.rs
|
||||
│ └── keywords.rs
|
||||
├── botserver-contacts/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ └── crm.rs
|
||||
├── botserver-compliance/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ └── audit.rs
|
||||
├── botserver-video/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ └── livekit.rs
|
||||
├── botserver-ui/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ └── tui.rs
|
||||
└── botserver/ (meta-crate)
|
||||
├── Cargo.toml
|
||||
└── src/
|
||||
├── main.rs
|
||||
└── lib.rs (re-exports)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Step 1: Create botstate Crate
|
||||
```bash
|
||||
cd /home/rodriguez/src/gb
|
||||
mkdir -p botstate/src
|
||||
cat > botstate/Cargo.toml << 'EOF'
|
||||
[package]
|
||||
name = "botstate"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
botlib = { workspace = true }
|
||||
tokio = { workspace = true, features = ["full"] }
|
||||
redis = { workspace = true, features = ["tokio-comp"] }
|
||||
diesel = { workspace = true, features = ["postgres", "r2d2"] }
|
||||
chrono = { workspace = true }
|
||||
uuid = { workspace = true, features = ["v4"] }
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
async-trait = { workspace = true }
|
||||
EOF
|
||||
```
|
||||
|
||||
### Step 2: Update Workspace
|
||||
Edit `/home/rodriguez/src/gb/Cargo.toml`:
|
||||
```toml
|
||||
members = [
|
||||
"botapp",
|
||||
"botdevice",
|
||||
"botlib",
|
||||
"botserver",
|
||||
"bottest",
|
||||
"botui",
|
||||
"botstate", # NEW
|
||||
"botserver-core", # NEW
|
||||
"botserver-auth", # NEW
|
||||
"botserver-chat", # NEW
|
||||
# ... etc
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Success Criteria
|
||||
|
||||
1. ✅ **botstate** compiles independently
|
||||
2. ✅ **botserver-core** starts HTTP server without other services
|
||||
3. ✅ Each service can be built independently
|
||||
4. ✅ No circular dependencies between crates
|
||||
5. ✅ All tests pass
|
||||
6. ✅ Compile time reduced by 70% for individual services
|
||||
7. ✅ Memory usage during compilation reduced by 60%
|
||||
8. ✅ Backward compatibility maintained (existing code still works)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- This plan prioritizes **incremental migration** - each step is reversible
|
||||
- Start with `botstate` and `botserver-core` as foundation
|
||||
- Test after each service extraction
|
||||
- Keep the meta-crate `botserver` for backward compatibility
|
||||
- Use feature flags to enable/disable services during transition
|
||||
|
||||
---
|
||||
|
||||
**Created**: 2026-04-18
|
||||
**Status**: Planning Phase
|
||||
**Next Step**: Create `botstate` crate with shared state types
|
||||
Loading…
Add table
Reference in a new issue