botserver/CODE_IMPLEMENTATION_ROADMAP.md

# Code Implementation Roadmap - Closing the Documentation/Code Gap

## Executive Summary

**The Problem**: Documentation describes 14 UI applications in the General Bots Suite, but only 6 have fully implemented backends. The other 5 have complete HTML/CSS/JavaScript frontends that are waiting for Rust backend handlers.

**The Solution**: Use HTMX + Rust to implement minimal backend handlers that render HTML. No additional JavaScript frameworks needed.

**Timeline**: 2-3 weeks to complete all missing implementations

---

## Current State Analysis

### What Exists (✅ Complete)

| Component | Status | Where | Notes |
|-----------|--------|-------|-------|
| **Chat** | ✅ Complete | `/api/sessions`, `/ws` | Real-time messaging, context management |
| **Drive** | ✅ Complete | `/api/drive/*` | S3-based file storage, upload/download |
| **Tasks** | ✅ Complete | `/api/tasks/*` | Task CRUD, assignment, status tracking |
| **Mail** | ✅ Complete | `/api/email/*` | IMAP/SMTP integration, folders, drafts |
| **Calendar** | ✅ Complete | CalDAV, `/api/calendar/*` | Event management, CalDAV protocol |
| **Meet** | ✅ Complete | `/api/meet/*`, `/ws/meet` | LiveKit integration, video calls |
| **Monitoring** | ✅ Complete | `/api/admin/stats` | System metrics, performance data |
| **BASIC Compiler** | ✅ Complete | `/src/basic/compiler/` | Dialog validation, script parsing |
| **Vector DB** | ✅ Complete | Qdrant integration | Knowledge base embeddings, search |
| **LLM Integration** | ✅ Complete | `/src/llm/mod.rs` | Multiple model support, routing |
| **HTMX App** | ✅ Complete | `htmx-app.js` | All HTMX infrastructure ready |

### What's Missing (❌ No Backend)

| App | Frontend | Backend | Effort | Impact |
|-----|----------|---------|--------|--------|
| **Analytics** | ✅ Complete HTML/CSS/JS | ❌ No handlers | 4-6 hrs | High (metrics critical) |
| **Paper** | ✅ Complete editor | ❌ No document API | 2-3 hrs | High (users want docs) |
| **Research** | ✅ Complete UI | 🟡 Partial (JSON only) | 1-2 hrs | Medium (search exists) |
| **Designer** | ✅ Complete builder | ❌ No dialog API | 6-8 hrs | Medium (admin feature) |
| **Sources** | ✅ Complete grid | ❌ No template API | 2-3 hrs | Low (nice-to-have) |
| **Player** | ❌ No HTML | ❌ No handlers | 2-3 hrs | Low (can use Drive) |

### Database/Infrastructure Status

| Component | Status | Ready to Use |
|-----------|--------|--------------|
| PostgreSQL Connection Pool | ✅ | Yes - in AppState |
| S3 Drive Integration | ✅ | Yes - Drive module |
| Redis Cache | ✅ | Yes - feature gated |
| Qdrant Vector DB | ✅ | Yes - VectorDB module |
| LLM Models | ✅ | Yes - LLM module |
| BASIC Compiler | ✅ | Yes - can call directly |
| Askama Templates | ✅ | Yes - multiple examples |
| HTMX Framework | ✅ | Yes - in UI already |

---

## Implementation Strategy

### Core Principle: HTMX-First Backend

**Pattern**: Frontend sends HTMX request → Backend handler → Askama template → HTML response

```
User clicks button with hx-get="/api/resource"
    ↓
Browser sends GET to /api/resource
    ↓
Rust handler executes
    ↓
Handler calls Askama template with data
    ↓
Template renders HTML fragment
    ↓
HTMX replaces target element with HTML
    ↓
Done - no JSON parsing, no JavaScript needed
```

### Why This Approach

✅ **Minimal Code** - Just Rust + HTML templates, no JavaScript frameworks  
✅ **Reuses Everything** - Drive, LLM, Compiler, Database already available  
✅ **Server-Side Rendering** - Better for SEO, accessibility, performance  
✅ **Existing Infrastructure** - HTMX already loaded in all pages  
✅ **Type Safe** - Rust compiler catches errors at build time  
✅ **Fast Development** - Copy patterns from existing modules  

---

## Implementation Details by App

### Priority 1: Analytics Dashboard (CRITICAL)

**Timeline**: 4-6 hours  
**Complexity**: Low (pure SQL queries)  
**Impact**: High (essential metrics)

**What to Create**:

1. **New Module**: `botserver/src/analytics/mod.rs`
   - Handler: `async fn analytics_dashboard()`
   - Handler: `async fn analytics_sessions()`
   - Handler: `async fn analytics_bots()`
   - Handler: `async fn analytics_top_queries()`
   - Handler: `async fn analytics_errors()`

2. **Database Queries**:
   ```sql
   SELECT COUNT(*) FROM message_history WHERE created_at > NOW() - INTERVAL;
   SELECT AVG(response_time) FROM message_history;
   SELECT COUNT(*) FROM sessions WHERE active = true;
   SELECT error_count FROM system_metrics;
   ```

3. **Askama Templates**: `templates/analytics/dashboard.html`
   ```html
   <div class="metrics-grid">
     <div class="metric-card">{{ messages_count }}</div>
     <div class="metric-card">{{ avg_response_time }}ms</div>
   </div>
   ```

4. **URL Routes** (add to `urls.rs`):
   ```rust
   pub const ANALYTICS_DASHBOARD: &'static str = "/api/analytics/dashboard";
   pub const ANALYTICS_SESSIONS: &'static str = "/api/analytics/sessions";
   pub const ANALYTICS_BOTS: &'static str = "/api/analytics/bots";
   ```

5. **Wire in main.rs**:
   ```rust
   api_router = api_router.merge(analytics::configure());
   ```

**Frontend Already Has**:
- ✅ HTML form with time range selector
- ✅ HTMX attributes pointing to `/api/analytics/*`
- ✅ Charts expecting data
- ✅ Metric cards ready for values

---

### Priority 2: Paper Documents (HIGH VALUE)

**Timeline**: 2-3 hours  
**Complexity**: Low-Medium (reuse Drive module)  
**Impact**: High (users want document editor)

**What to Create**:

1. **New Module**: `botserver/src/documents/mod.rs`
   - Handler: `async fn create_document()`
   - Handler: `async fn list_documents()`
   - Handler: `async fn get_document()`
   - Handler: `async fn update_document()`
   - Handler: `async fn delete_document()`

2. **Storage Pattern** (reuse Drive):
   ```rust
   // Store in Drive under .gbdocs/ folder
   let bucket = format!("{}.gbai", bot_name);
   let key = format!(".gbdocs/{}/document.md", doc_id);
   drive_client.put_object(bucket, key, content).await;
   ```

3. **Document Structure**:
   ```rust
   pub struct Document {
       pub id: Uuid,
       pub title: String,
       pub content: String,
       pub doc_type: DocumentType, // draft, note, template
       pub created_at: DateTime<Utc>,
       pub updated_at: DateTime<Utc>,
       pub user_id: Uuid,
   }
   ```

4. **Askama Templates**: `templates/documents/list.html`
   - Grid of document cards
   - Open/edit/delete buttons with HTMX

5. **Wire in main.rs**:
   ```rust
   api_router = api_router.merge(documents::configure());
   ```

**Frontend Already Has**:
- ✅ Rich text editor (complete)
- ✅ Formatting toolbar
- ✅ AI suggestion panel
- ✅ Document list sidebar
- ✅ HTMX ready to send to `/api/documents`

**Can Optionally Add** (later):
- LLM integration for AI rewrite/summarize
- PDF export (using existing pdf-extract dependency)
- Version history (store multiple versions)

---

### Priority 3: Research - HTML Integration (QUICK WIN)

**Timeline**: 1-2 hours  
**Complexity**: Very Low (just change response format)  
**Impact**: Medium (search already works, just needs HTML)

**What to Change**:

1. **Update Handler**: `botserver/src/core/kb/mod.rs`
   ```rust
   // Current: returns Json<SearchResults>
   // Change to: returns Html<String> when format=html
   
   pub async fn search_kb(
       Query(params): Query<SearchQuery>, // add format: Option<String>
   ) -> impl IntoResponse {
       if params.format == Some("html") {
           Html(template.render().unwrap())
       } else {
           Json(results).into_response()
       }
   }
   ```

2. **Create Template**: `templates/kb/search_results.html`
   ```html
   <div class="result-item" hx-get="/api/kb/{id}" hx-trigger="click">
     <h3>{{ title }}</h3>
     <p>{{ snippet }}</p>
     <span class="relevance">{{ score }}</span>
   </div>
   ```

3. **Update Frontend**: Already done - just works when backend returns HTML

**Frontend Already Has**:
- ✅ Search input with HTMX
- ✅ Results container waiting to be filled
- ✅ Filters and limits
- ✅ Stats panels

---

### Priority 4: Sources - Template Manager (MEDIUM)

**Timeline**: 2-3 hours  
**Complexity**: Low-Medium (file enumeration + parsing)  
**Impact**: Low-Medium (nice-to-have feature)

**What to Create**:

1. **New Module**: `botserver/src/sources/mod.rs`
   - Handler: `async fn list_sources()`
   - Handler: `async fn get_source()`
   - Handler: `async fn create_from_template()`

2. **Logic**:
   ```rust
   // List templates from Drive
   let templates = drive.list_objects(".gbai/templates")?;
   
   // Parse metadata from template file
   let content = drive.get_object(".gbai/templates/template.bas")?;
   let metadata = parse_yaml_metadata(&content);
   ```

3. **Source Structure**:
   ```rust
   pub struct Source {
       pub id: String, // filename
       pub name: String,
       pub description: String,
       pub category: String, // templates, prompts, samples
       pub content: String,
       pub tags: Vec<String>,
       pub downloads: i32,
       pub rating: f32,
   }
   ```

4. **Templates**: `templates/sources/grid.html`
   - Grid of source cards
   - Category filter tabs
   - Search capability

5. **Wire in main.rs**

**Frontend Already Has**:
- ✅ Source grid layout
- ✅ Category selector
- ✅ Search box with HTMX
- ✅ Source detail view

---

### Priority 5: Designer - Dialog Configuration (COMPLEX)

**Timeline**: 6-8 hours  
**Complexity**: Medium-High (BASIC compiler integration)  
**Impact**: Medium (admin/developer feature)

**What to Create**:

1. **New Module**: `botserver/src/designer/mod.rs`
   - Handler: `async fn list_dialogs()`
   - Handler: `async fn create_dialog()`
   - Handler: `async fn update_dialog()`
   - Handler: `async fn validate_dialog()`
   - Handler: `async fn deploy_dialog()`

2. **Validation Flow**:
   ```rust
   // Use existing BASIC compiler
   use crate::basic::compiler::BASICCompiler;
   
   let compiler = BASICCompiler::new();
   match compiler.compile(&dialog_content) {
       Ok(_) => { /* valid */ }
       Err(errors) => { /* return errors in HTML */ }
   }
   ```

3. **Storage** (Drive):
   ```rust
   // Store .bas files in Drive
   let bucket = format!("{}.gbai", bot_name);
   let key = format!(".gbdialogs/{}.bas", dialog_name);
   drive.put_object(bucket, key, content).await;
   ```

4. **Dialog Structure**:
   ```rust
   pub struct Dialog {
       pub id: Uuid,
       pub bot_id: Uuid,
       pub name: String,
       pub content: String, // BASIC code
       pub status: DialogStatus, // draft, valid, deployed
       pub version: i32,
       pub created_at: DateTime<Utc>,
       pub updated_at: DateTime<Utc>,
   }
   ```

5. **Templates**:
   - `templates/designer/dialog_list.html` - List of dialogs
   - `templates/designer/dialog_editor.html` - Code editor
   - `templates/designer/validation_results.html` - Error display

6. **Wire in main.rs**

**Frontend Already Has**:
- ✅ Dialog list with HTMX
- ✅ Code editor interface
- ✅ Deploy button
- ✅ Validation result display

**Notes**:
- Can reuse existing BASIC compiler from `botserver/src/basic/compiler/`
- Compiler already available in AppState
- Just need to call it and render results as HTML

---

## Implementation Checklist

### Week 1: Foundation (High-Value, Low-Effort)

- [ ] **Research HTML Integration** (1-2 hrs)
  - [ ] Update `/api/kb/search` to support `?format=html`
  - [ ] Create template for search results
  - [ ] Test with frontend

- [ ] **Paper Documents** (2-3 hrs)
  - [ ] Create `botserver/src/documents/mod.rs`
  - [ ] Implement CRUD handlers
  - [ ] Add Askama templates
  - [ ] Wire routes in main.rs
  - [ ] Test with frontend

- [ ] **Analytics Dashboard** (4-6 hrs)
  - [ ] Create `botserver/src/analytics/mod.rs`
  - [ ] Write SQL aggregation queries
  - [ ] Create Askama templates for metrics
  - [ ] Implement handlers
  - [ ] Wire routes in main.rs
  - [ ] Test with frontend

**Week 1 Result**: 3 apps complete, majority of UI functional

### Week 2: Medium Effort

- [ ] **Sources Template Manager** (2-3 hrs)
  - [ ] Create `botserver/src/sources/mod.rs`
  - [ ] Implement Drive template enumeration
  - [ ] Create template listing
  - [ ] Test with frontend

- [ ] **Designer - Dialog Configuration** (6-8 hrs)
  - [ ] Create `botserver/src/designer/mod.rs`
  - [ ] Implement BASIC validation integration
  - [ ] Create dialog list/editor templates
  - [ ] Implement CRUD handlers
  - [ ] Add deploy functionality
  - [ ] Test with frontend

**Week 2 Result**: All 5 missing apps complete

### Week 3: Polish & Testing

- [ ] Integration testing across all apps
- [ ] Performance optimization
- [ ] Error handling refinement
- [ ] Documentation updates
- [ ] Deployment validation

---

## Technical Patterns to Follow

### Handler Pattern (Copy This Template)

```rust
// botserver/src/analytics/mod.rs

use axum::{
    extract::{Query, State},
    http::StatusCode,
    response::Html,
    routing::get,
    Router,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use askama::Template;

#[derive(Deserialize)]
pub struct AnalyticsQuery {
    pub time_range: Option<String>,
}

#[derive(Serialize)]
pub struct AnalyticsData {
    pub messages_count: i64,
    pub sessions_count: i64,
    pub avg_response_time: f64,
}

#[derive(Template)]
#[template(path = "analytics/dashboard.html")]
struct AnalyticsDashboardTemplate {
    data: AnalyticsData,
}

pub async fn analytics_dashboard(
    Query(params): Query<AnalyticsQuery>,
    State(state): State<Arc<AppState>>,
) -> Result<Html<String>, StatusCode> {
    // Query database
    let mut conn = state.conn.get()
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
    
    let data = AnalyticsData {
        messages_count: get_message_count(&mut conn)?,
        sessions_count: get_session_count(&mut conn)?,
        avg_response_time: get_avg_response_time(&mut conn)?,
    };
    
    // Render template
    let template = AnalyticsDashboardTemplate { data };
    Ok(Html(template.render()
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?))
}

pub fn configure() -> Router<Arc<AppState>> {
    Router::new()
        .route("/api/analytics/dashboard", get(analytics_dashboard))
}
```

### Template Pattern (Copy This Template)

```html
<!-- botserver/templates/analytics/dashboard.html -->
<div class="analytics-dashboard">
  <div class="metrics-grid">
    <div class="metric-card">
      <span class="label">Messages</span>
      <span class="value">{{ data.messages_count }}</span>
    </div>
    <div class="metric-card">
      <span class="label">Sessions</span>
      <span class="value">{{ data.sessions_count }}</span>
    </div>
    <div class="metric-card">
      <span class="label">Avg Response</span>
      <span class="value">{{ data.avg_response_time }}ms</span>
    </div>
  </div>
</div>
```

---

## Dependencies Already Available

### Database Access

```rust
// Get database connection from AppState
let mut conn = state.conn.get()?;

// Use existing queries from botserver/src/schema.rs
use botserver::schema::message_history::dsl::*;
use diesel::prelude::*;

let results = message_history
    .filter(created_at.gt(now - interval))
    .load::<MessageHistory>(&mut conn)?;
```

### S3 Drive Access

```rust
// Get S3 client from AppState
let drive = state.drive.as_ref().ok_or("Drive not configured")?;

// Use existing methods
drive.list_objects("bucket", "path").await?;
drive.get_object("bucket", "key").await?;
drive.put_object("bucket", "key", content).await?;
```

### LLM Integration

```rust
// Get LLM from AppState or instantiate
let llm_client = state.llm_client.clone();

// Call for AI features (Paper app)
let response = llm_client.complete(&prompt).await?;
```

### BASIC Compiler

```rust
// Already available for Designer app
use botserver::basic::compiler::BASICCompiler;

let compiler = BASICCompiler::new();
match compiler.compile(&dialog_code) {
    Ok(ast) => { /* valid */ }
    Err(errors) => { /* return errors */ }
}
```

---

## Testing Strategy

### Unit Tests (Per Module)

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_analytics_dashboard() {
        // Create mock AppState
        // Call analytics_dashboard()
        // Assert response contains HTML
        // Assert metrics are reasonable
    }
}
```

### Integration Tests (HTMX Flow)

```bash
# Test Analytics
curl -X GET "http://localhost:3000/api/analytics/dashboard?timeRange=day" \
  -H "Accept: text/html"

# Verify response is HTML, not JSON
# Verify contains metric divs
# Verify values are numbers
```

### Frontend Tests (Browser)

1. Open `http://localhost:3000`
2. Click on app in menu (e.g., "Analytics")
3. Verify HTMX request goes to `/api/analytics/*`
4. Verify HTML content loads in page
5. Verify styling is correct
6. Test interactive features (filters, buttons)

---

## Deployment Notes

### Build

```bash
cd botserver
cargo build --release --features "analytics,documents,sources,designer"
```

### Feature Flags

Add to `Cargo.toml`:

```toml
[features]
analytics = []
documents = []
designer = []
sources = []
research-html = []
```

### Environment

No new environment variables needed - all modules use existing AppState configuration.

---

## Risk Mitigation

### What Could Go Wrong

| Risk | Mitigation |
|------|-----------|
| SQL injection in queries | Use Diesel ORM (type-safe) |
| Template rendering errors | Test templates with sample data |
| Drive not configured | Check AppState initialization |
| Compiler failures in Designer | Use existing compiler tests |
| HTMX attribute errors | Verify hx-* attributes in HTML |

### Testing Before Deploy

- [ ] All handlers return valid HTML
- [ ] All HTMX attributes point to correct endpoints
- [ ] No 404s in browser console
- [ ] No error messages in backend logs
- [ ] Database queries complete in <100ms
- [ ] Templates render without errors
- [ ] CSS styles load correctly
- [ ] Responsive design works on mobile

---

## Success Criteria

### Definition of Done (Per App)

- ✅ Rust handlers implement all CRUD operations
- ✅ Askama templates render without errors
- ✅ Routes registered in main.rs
- ✅ HTMX attributes in frontend point to correct endpoints
- ✅ HTML responses work with HTMX swapping
- ✅ No JavaScript errors in console
- ✅ All CRUD operations tested manually
- ✅ No database errors in logs
- ✅ Response time <100ms for queries
- ✅ Frontend UI works as designed

### Overall Success

By end of Week 3:
- ✅ All 5 missing apps have backend handlers
- ✅ All app UIs are fully functional
- ✅ No HTMX errors in browser
- ✅ All endpoints tested and working
- ✅ Documentation updated with new APIs

---

## References

### Existing Working Examples

Study these modules to understand patterns:

- `botserver/src/tasks/mod.rs` - Complete CRUD example
- `botserver/src/email/mod.rs` - API handlers pattern
- `botserver/src/drive/mod.rs` - S3 integration pattern
- `botserver/src/calendar/mod.rs` - Complex routes example

### Key Files to Edit

- `botserver/src/main.rs` - Add `.merge(analytics::configure())` etc.
- `botserver/src/core/urls.rs` - Define new URL constants
- `botserver/templates/` - Add new Askama templates
- `botui/ui/suite/*/` - HTML already complete, no changes needed

### Documentation References

- HTMX: https://htmx.org/attributes/hx-get/
- Axum: https://docs.rs/axum/latest/axum/
- Askama: https://docs.rs/askama/latest/askama/
- Diesel: https://docs.rs/diesel/latest/diesel/

---

## Questions & Answers

**Q: Do we need to modify frontend HTML?**  
A: No - all HTML files already have correct HTMX attributes. Just implement the backend endpoints.

**Q: Can we use JSON responses with HTMX?**  
A: Technically yes, but HTML responses are more efficient with HTMX and require no frontend JavaScript.

**Q: What if a database query takes too long?**  
A: Add database indexes on frequently queried columns. Use EXPLAIN to analyze slow queries.

**Q: How do we handle errors in templates?**  
A: Return HTTP error status codes (400, 404, 500) with HTML error messages. HTMX handles swapping them appropriately.

**Q: Can we add new dependencies?**  
A: Prefer using existing dependencies already in Cargo.toml. If needed, add to existing feature flags.

**Q: What about authentication/authorization?**  
A: Use existing auth middleware from Drive/Tasks modules. Copy the pattern.

---

## Next Steps

1. **Start with Priority 1** (Research HTML Integration)
   - Easiest to implement (1-2 hours)
   - Low risk
   - Good way to understand HTMX pattern

2. **Move to Priority 2** (Paper Documents)
   - High user value
   - Medium complexity
   - Reuses Drive module

3. **Tackle Priority 3** (Analytics)
   - Most SQL-heavy
   - Pure data aggregation
   - High impact for users

4. **Complete Priority 4 & 5** (Designer & Sources)
   - More complex features
   - Can be done in parallel
   - Nice-to-have, not critical

**Estimated Total Time**: 2-3 weeks for all 5 apps to be production-ready.

---

## Success Metrics

After implementation:

- **Code Coverage**: 85%+ of new handlers have tests
- **Performance**: All endpoints respond <200ms
- **Reliability**: 99.5%+ uptime for new features
- **User Satisfaction**: All UI apps work as documented
- **Maintainability**: All code follows existing patterns
- **Documentation**: API docs auto-generated from code

---

**Last Updated**: 2024  
**Status**: Ready for Implementation  
**Maintainer**: General Bots Team