Rodrigo Rodriguez (Pragmatismo) e8d171ea40 - New templates.

2025-12-03 07:15:54 -03:00

21 KiB

Raw Blame History

Code Implementation Roadmap - Closing the Documentation/Code Gap

Executive Summary

The Problem: Documentation describes 14 UI applications in the General Bots Suite, but only 6 have fully implemented backends. The other 5 have complete HTML/CSS/JavaScript frontends that are waiting for Rust backend handlers.

The Solution: Use HTMX + Rust to implement minimal backend handlers that render HTML. No additional JavaScript frameworks needed.

Timeline: 2-3 weeks to complete all missing implementations

Current State Analysis

What Exists (✅ Complete)

Component	Status	Where	Notes
Chat	✅ Complete	`/api/sessions`, `/ws`	Real-time messaging, context management
Drive	✅ Complete	`/api/drive/*`	S3-based file storage, upload/download
Tasks	✅ Complete	`/api/tasks/*`	Task CRUD, assignment, status tracking
Mail	✅ Complete	`/api/email/*`	IMAP/SMTP integration, folders, drafts
Calendar	✅ Complete	CalDAV, `/api/calendar/*`	Event management, CalDAV protocol
Meet	✅ Complete	`/api/meet/*`, `/ws/meet`	LiveKit integration, video calls
Monitoring	✅ Complete	`/api/admin/stats`	System metrics, performance data
BASIC Compiler	✅ Complete	`/src/basic/compiler/`	Dialog validation, script parsing
Vector DB	✅ Complete	Qdrant integration	Knowledge base embeddings, search
LLM Integration	✅ Complete	`/src/llm/mod.rs`	Multiple model support, routing
HTMX App	✅ Complete	`htmx-app.js`	All HTMX infrastructure ready

What's Missing (❌ No Backend)

App	Frontend	Backend	Effort	Impact
Analytics	✅ Complete HTML/CSS/JS	❌ No handlers	4-6 hrs	High (metrics critical)
Paper	✅ Complete editor	❌ No document API	2-3 hrs	High (users want docs)
Research	✅ Complete UI	🟡 Partial (JSON only)	1-2 hrs	Medium (search exists)
Designer	✅ Complete builder	❌ No dialog API	6-8 hrs	Medium (admin feature)
Sources	✅ Complete grid	❌ No template API	2-3 hrs	Low (nice-to-have)
Player	❌ No HTML	❌ No handlers	2-3 hrs	Low (can use Drive)

Database/Infrastructure Status

Component	Status	Ready to Use
PostgreSQL Connection Pool	✅	Yes - in AppState
S3 Drive Integration	✅	Yes - Drive module
Redis Cache	✅	Yes - feature gated
Qdrant Vector DB	✅	Yes - VectorDB module
LLM Models	✅	Yes - LLM module
BASIC Compiler	✅	Yes - can call directly
Askama Templates	✅	Yes - multiple examples
HTMX Framework	✅	Yes - in UI already

Implementation Strategy

Core Principle: HTMX-First Backend

Pattern: Frontend sends HTMX request → Backend handler → Askama template → HTML response

User clicks button with hx-get="/api/resource"
    ↓
Browser sends GET to /api/resource
    ↓
Rust handler executes
    ↓
Handler calls Askama template with data
    ↓
Template renders HTML fragment
    ↓
HTMX replaces target element with HTML
    ↓
Done - no JSON parsing, no JavaScript needed

Why This Approach

✅ Minimal Code - Just Rust + HTML templates, no JavaScript frameworks
✅ Reuses Everything - Drive, LLM, Compiler, Database already available
✅ Server-Side Rendering - Better for SEO, accessibility, performance
✅ Existing Infrastructure - HTMX already loaded in all pages
✅ Type Safe - Rust compiler catches errors at build time
✅ Fast Development - Copy patterns from existing modules

Implementation Details by App

Priority 1: Analytics Dashboard (CRITICAL)

Timeline: 4-6 hours
Complexity: Low (pure SQL queries)
Impact: High (essential metrics)

What to Create:

New Module: botserver/src/analytics/mod.rs
- Handler: async fn analytics_dashboard()
- Handler: async fn analytics_sessions()
- Handler: async fn analytics_bots()
- Handler: async fn analytics_top_queries()
- Handler: async fn analytics_errors()

Database Queries:

SELECT COUNT(*) FROM message_history WHERE created_at > NOW() - INTERVAL;
SELECT AVG(response_time) FROM message_history;
SELECT COUNT(*) FROM sessions WHERE active = true;
SELECT error_count FROM system_metrics;

Askama Templates: templates/analytics/dashboard.html

<div class="metrics-grid">
  <div class="metric-card">{{ messages_count }}</div>
  <div class="metric-card">{{ avg_response_time }}ms</div>
</div>

URL Routes (add to urls.rs):

pub const ANALYTICS_DASHBOARD: &'static str = "/api/analytics/dashboard";
pub const ANALYTICS_SESSIONS: &'static str = "/api/analytics/sessions";
pub const ANALYTICS_BOTS: &'static str = "/api/analytics/bots";

Wire in main.rs:

api_router = api_router.merge(analytics::configure());

Frontend Already Has:

✅ HTML form with time range selector
✅ HTMX attributes pointing to /api/analytics/*
✅ Charts expecting data
✅ Metric cards ready for values

Priority 2: Paper Documents (HIGH VALUE)

Timeline: 2-3 hours
Complexity: Low-Medium (reuse Drive module)
Impact: High (users want document editor)

What to Create:

New Module: botserver/src/documents/mod.rs
- Handler: async fn create_document()
- Handler: async fn list_documents()
- Handler: async fn get_document()
- Handler: async fn update_document()
- Handler: async fn delete_document()

Storage Pattern (reuse Drive):

// Store in Drive under .gbdocs/ folder
let bucket = format!("{}.gbai", bot_name);
let key = format!(".gbdocs/{}/document.md", doc_id);
drive_client.put_object(bucket, key, content).await;

Document Structure:

pub struct Document {
    pub id: Uuid,
    pub title: String,
    pub content: String,
    pub doc_type: DocumentType, // draft, note, template
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
    pub user_id: Uuid,
}

Askama Templates: templates/documents/list.html
- Grid of document cards
- Open/edit/delete buttons with HTMX

Wire in main.rs:

api_router = api_router.merge(documents::configure());

Frontend Already Has:

✅ Rich text editor (complete)
✅ Formatting toolbar
✅ AI suggestion panel
✅ Document list sidebar
✅ HTMX ready to send to /api/documents

Can Optionally Add (later):

LLM integration for AI rewrite/summarize
PDF export (using existing pdf-extract dependency)
Version history (store multiple versions)

Priority 3: Research - HTML Integration (QUICK WIN)

Timeline: 1-2 hours
Complexity: Very Low (just change response format)
Impact: Medium (search already works, just needs HTML)

What to Change:

Update Handler: botserver/src/core/kb/mod.rs

// Current: returns Json<SearchResults>
// Change to: returns Html<String> when format=html

pub async fn search_kb(
    Query(params): Query<SearchQuery>, // add format: Option<String>
) -> impl IntoResponse {
    if params.format == Some("html") {
        Html(template.render().unwrap())
    } else {
        Json(results).into_response()
    }
}

Create Template: templates/kb/search_results.html

<div class="result-item" hx-get="/api/kb/{id}" hx-trigger="click">
  <h3>{{ title }}</h3>
  <p>{{ snippet }}</p>
  <span class="relevance">{{ score }}</span>
</div>

Update Frontend: Already done - just works when backend returns HTML

Frontend Already Has:

✅ Search input with HTMX
✅ Results container waiting to be filled
✅ Filters and limits
✅ Stats panels

Priority 4: Sources - Template Manager (MEDIUM)

Timeline: 2-3 hours
Complexity: Low-Medium (file enumeration + parsing)
Impact: Low-Medium (nice-to-have feature)

What to Create:

New Module: botserver/src/sources/mod.rs
- Handler: async fn list_sources()
- Handler: async fn get_source()
- Handler: async fn create_from_template()

Logic:

// List templates from Drive
let templates = drive.list_objects(".gbai/templates")?;

// Parse metadata from template file
let content = drive.get_object(".gbai/templates/template.bas")?;
let metadata = parse_yaml_metadata(&content);

Source Structure:

pub struct Source {
    pub id: String, // filename
    pub name: String,
    pub description: String,
    pub category: String, // templates, prompts, samples
    pub content: String,
    pub tags: Vec<String>,
    pub downloads: i32,
    pub rating: f32,
}

Templates: templates/sources/grid.html
- Grid of source cards
- Category filter tabs
- Search capability
Wire in main.rs

Frontend Already Has:

✅ Source grid layout
✅ Category selector
✅ Search box with HTMX
✅ Source detail view

Priority 5: Designer - Dialog Configuration (COMPLEX)

Timeline: 6-8 hours
Complexity: Medium-High (BASIC compiler integration)
Impact: Medium (admin/developer feature)

What to Create:

New Module: botserver/src/designer/mod.rs
- Handler: async fn list_dialogs()
- Handler: async fn create_dialog()
- Handler: async fn update_dialog()
- Handler: async fn validate_dialog()
- Handler: async fn deploy_dialog()

Validation Flow:

// Use existing BASIC compiler
use crate::basic::compiler::BASICCompiler;

let compiler = BASICCompiler::new();
match compiler.compile(&dialog_content) {
    Ok(_) => { /* valid */ }
    Err(errors) => { /* return errors in HTML */ }
}

Storage (Drive):

// Store .bas files in Drive
let bucket = format!("{}.gbai", bot_name);
let key = format!(".gbdialogs/{}.bas", dialog_name);
drive.put_object(bucket, key, content).await;

Dialog Structure:

pub struct Dialog {
    pub id: Uuid,
    pub bot_id: Uuid,
    pub name: String,
    pub content: String, // BASIC code
    pub status: DialogStatus, // draft, valid, deployed
    pub version: i32,
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
}

Templates:
- templates/designer/dialog_list.html - List of dialogs
- templates/designer/dialog_editor.html - Code editor
- templates/designer/validation_results.html - Error display
Wire in main.rs

Frontend Already Has:

✅ Dialog list with HTMX
✅ Code editor interface
✅ Deploy button
✅ Validation result display

Notes:

Can reuse existing BASIC compiler from botserver/src/basic/compiler/
Compiler already available in AppState
Just need to call it and render results as HTML

Implementation Checklist

Week 1: Foundation (High-Value, Low-Effort)

Research HTML Integration (1-2 hrs)
- Update /api/kb/search to support ?format=html
- Create template for search results
- Test with frontend
Paper Documents (2-3 hrs)
- Create botserver/src/documents/mod.rs
- Implement CRUD handlers
- Add Askama templates
- Wire routes in main.rs
- Test with frontend
Analytics Dashboard (4-6 hrs)
- Create botserver/src/analytics/mod.rs
- Write SQL aggregation queries
- Create Askama templates for metrics
- Implement handlers
- Wire routes in main.rs
- Test with frontend

Week 1 Result: 3 apps complete, majority of UI functional

Week 2: Medium Effort

Sources Template Manager (2-3 hrs)
- Create botserver/src/sources/mod.rs
- Implement Drive template enumeration
- Create template listing
- Test with frontend
Designer - Dialog Configuration (6-8 hrs)
- Create botserver/src/designer/mod.rs
- Implement BASIC validation integration
- Create dialog list/editor templates
- Implement CRUD handlers
- Add deploy functionality
- Test with frontend

Week 2 Result: All 5 missing apps complete

Week 3: Polish & Testing

Integration testing across all apps
Performance optimization
Error handling refinement
Documentation updates
Deployment validation

Technical Patterns to Follow

Handler Pattern (Copy This Template)

// botserver/src/analytics/mod.rs

use axum::{
    extract::{Query, State},
    http::StatusCode,
    response::Html,
    routing::get,
    Router,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use askama::Template;

#[derive(Deserialize)]
pub struct AnalyticsQuery {
    pub time_range: Option<String>,
}

#[derive(Serialize)]
pub struct AnalyticsData {
    pub messages_count: i64,
    pub sessions_count: i64,
    pub avg_response_time: f64,
}

#[derive(Template)]
#[template(path = "analytics/dashboard.html")]
struct AnalyticsDashboardTemplate {
    data: AnalyticsData,
}

pub async fn analytics_dashboard(
    Query(params): Query<AnalyticsQuery>,
    State(state): State<Arc<AppState>>,
) -> Result<Html<String>, StatusCode> {
    // Query database
    let mut conn = state.conn.get()
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
    
    let data = AnalyticsData {
        messages_count: get_message_count(&mut conn)?,
        sessions_count: get_session_count(&mut conn)?,
        avg_response_time: get_avg_response_time(&mut conn)?,
    };
    
    // Render template
    let template = AnalyticsDashboardTemplate { data };
    Ok(Html(template.render()
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?))
}

pub fn configure() -> Router<Arc<AppState>> {
    Router::new()
        .route("/api/analytics/dashboard", get(analytics_dashboard))
}

Template Pattern (Copy This Template)

<!-- botserver/templates/analytics/dashboard.html -->
<div class="analytics-dashboard">
  <div class="metrics-grid">
    <div class="metric-card">
      <span class="label">Messages</span>
      <span class="value">{{ data.messages_count }}</span>
    </div>
    <div class="metric-card">
      <span class="label">Sessions</span>
      <span class="value">{{ data.sessions_count }}</span>
    </div>
    <div class="metric-card">
      <span class="label">Avg Response</span>
      <span class="value">{{ data.avg_response_time }}ms</span>
    </div>
  </div>
</div>

Dependencies Already Available

Database Access

// Get database connection from AppState
let mut conn = state.conn.get()?;

// Use existing queries from botserver/src/schema.rs
use botserver::schema::message_history::dsl::*;
use diesel::prelude::*;

let results = message_history
    .filter(created_at.gt(now - interval))
    .load::<MessageHistory>(&mut conn)?;

S3 Drive Access

// Get S3 client from AppState
let drive = state.drive.as_ref().ok_or("Drive not configured")?;

// Use existing methods
drive.list_objects("bucket", "path").await?;
drive.get_object("bucket", "key").await?;
drive.put_object("bucket", "key", content).await?;

LLM Integration

// Get LLM from AppState or instantiate
let llm_client = state.llm_client.clone();

// Call for AI features (Paper app)
let response = llm_client.complete(&prompt).await?;

BASIC Compiler

// Already available for Designer app
use botserver::basic::compiler::BASICCompiler;

let compiler = BASICCompiler::new();
match compiler.compile(&dialog_code) {
    Ok(ast) => { /* valid */ }
    Err(errors) => { /* return errors */ }
}

Testing Strategy

Unit Tests (Per Module)

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_analytics_dashboard() {
        // Create mock AppState
        // Call analytics_dashboard()
        // Assert response contains HTML
        // Assert metrics are reasonable
    }
}

Integration Tests (HTMX Flow)

# Test Analytics
curl -X GET "http://localhost:3000/api/analytics/dashboard?timeRange=day" \
  -H "Accept: text/html"

# Verify response is HTML, not JSON
# Verify contains metric divs
# Verify values are numbers

Frontend Tests (Browser)

Open http://localhost:3000
Click on app in menu (e.g., "Analytics")
Verify HTMX request goes to /api/analytics/*
Verify HTML content loads in page
Verify styling is correct
Test interactive features (filters, buttons)

Deployment Notes

Build

cd botserver
cargo build --release --features "analytics,documents,sources,designer"

Feature Flags

Add to Cargo.toml:

[features]
analytics = []
documents = []
designer = []
sources = []
research-html = []

Environment

No new environment variables needed - all modules use existing AppState configuration.

Risk Mitigation

What Could Go Wrong

Risk	Mitigation
SQL injection in queries	Use Diesel ORM (type-safe)
Template rendering errors	Test templates with sample data
Drive not configured	Check AppState initialization
Compiler failures in Designer	Use existing compiler tests
HTMX attribute errors	Verify hx-* attributes in HTML

Testing Before Deploy

All handlers return valid HTML
All HTMX attributes point to correct endpoints
No 404s in browser console
No error messages in backend logs
Database queries complete in <100ms
Templates render without errors
CSS styles load correctly
Responsive design works on mobile

Success Criteria

Definition of Done (Per App)

✅ Rust handlers implement all CRUD operations
✅ Askama templates render without errors
✅ Routes registered in main.rs
✅ HTMX attributes in frontend point to correct endpoints
✅ HTML responses work with HTMX swapping
✅ No JavaScript errors in console
✅ All CRUD operations tested manually
✅ No database errors in logs
✅ Response time <100ms for queries
✅ Frontend UI works as designed

Overall Success

By end of Week 3:

✅ All 5 missing apps have backend handlers
✅ All app UIs are fully functional
✅ No HTMX errors in browser
✅ All endpoints tested and working
✅ Documentation updated with new APIs

References

Existing Working Examples

Study these modules to understand patterns:

botserver/src/tasks/mod.rs - Complete CRUD example
botserver/src/email/mod.rs - API handlers pattern
botserver/src/drive/mod.rs - S3 integration pattern
botserver/src/calendar/mod.rs - Complex routes example

Key Files to Edit

botserver/src/main.rs - Add .merge(analytics::configure()) etc.
botserver/src/core/urls.rs - Define new URL constants
botserver/templates/ - Add new Askama templates
botui/ui/suite/*/ - HTML already complete, no changes needed

Documentation References

HTMX: https://htmx.org/attributes/hx-get/
Axum: https://docs.rs/axum/latest/axum/
Askama: https://docs.rs/askama/latest/askama/
Diesel: https://docs.rs/diesel/latest/diesel/

Questions & Answers

Q: Do we need to modify frontend HTML?
A: No - all HTML files already have correct HTMX attributes. Just implement the backend endpoints.

Q: Can we use JSON responses with HTMX?
A: Technically yes, but HTML responses are more efficient with HTMX and require no frontend JavaScript.

Q: What if a database query takes too long?
A: Add database indexes on frequently queried columns. Use EXPLAIN to analyze slow queries.

Q: How do we handle errors in templates?
A: Return HTTP error status codes (400, 404, 500) with HTML error messages. HTMX handles swapping them appropriately.

Q: Can we add new dependencies?
A: Prefer using existing dependencies already in Cargo.toml. If needed, add to existing feature flags.

Q: What about authentication/authorization?
A: Use existing auth middleware from Drive/Tasks modules. Copy the pattern.

Next Steps

Start with Priority 1 (Research HTML Integration)
- Easiest to implement (1-2 hours)
- Low risk
- Good way to understand HTMX pattern
Move to Priority 2 (Paper Documents)
- High user value
- Medium complexity
- Reuses Drive module
Tackle Priority 3 (Analytics)
- Most SQL-heavy
- Pure data aggregation
- High impact for users
Complete Priority 4 & 5 (Designer & Sources)
- More complex features
- Can be done in parallel
- Nice-to-have, not critical

Estimated Total Time: 2-3 weeks for all 5 apps to be production-ready.

Success Metrics

After implementation:

Code Coverage: 85%+ of new handlers have tests
Performance: All endpoints respond <200ms
Reliability: 99.5%+ uptime for new features
User Satisfaction: All UI apps work as documented
Maintainability: All code follows existing patterns
Documentation: API docs auto-generated from code

Last Updated: 2024
Status: Ready for Implementation
Maintainer: General Bots Team

21 KiB Raw Blame History

Code Implementation Roadmap - Closing the Documentation/Code Gap

Executive Summary

Current State Analysis

What Exists (✅ Complete)

What's Missing (❌ No Backend)

Database/Infrastructure Status

Implementation Strategy

Core Principle: HTMX-First Backend

Why This Approach

Implementation Details by App

Priority 1: Analytics Dashboard (CRITICAL)

Priority 2: Paper Documents (HIGH VALUE)

Priority 3: Research - HTML Integration (QUICK WIN)

Priority 4: Sources - Template Manager (MEDIUM)

Priority 5: Designer - Dialog Configuration (COMPLEX)

Implementation Checklist

Week 1: Foundation (High-Value, Low-Effort)

Week 2: Medium Effort

Week 3: Polish & Testing

Technical Patterns to Follow

Handler Pattern (Copy This Template)

Template Pattern (Copy This Template)

Dependencies Already Available

Database Access

S3 Drive Access

LLM Integration

BASIC Compiler

Testing Strategy

Unit Tests (Per Module)

Integration Tests (HTMX Flow)

Frontend Tests (Browser)

Deployment Notes

Build

Feature Flags

Environment

Risk Mitigation

What Could Go Wrong

Testing Before Deploy

Success Criteria

Definition of Done (Per App)

Overall Success

References

Existing Working Examples

Key Files to Edit

Documentation References

Questions & Answers

Next Steps

Success Metrics

21 KiB

Raw Blame History