From ceb57e820a0933f275a9efddb9380ef2f89ade0b Mon Sep 17 00:00:00 2001 From: "Rodrigo Rodriguez (Pragmatismo)" Date: Sun, 28 Dec 2025 11:57:55 -0300 Subject: [PATCH] Update submodules: app generator, logging, knowledge base, web search, designer magic --- PENDING_TASKS.md | 241 +++++++++++++++++++++++++++++++++++++++++++++++ botapp | 2 +- botbook | 2 +- botdevice | 2 +- botlib | 2 +- botserver | 2 +- bottest | 2 +- botui | 2 +- 8 files changed, 248 insertions(+), 7 deletions(-) create mode 100644 PENDING_TASKS.md diff --git a/PENDING_TASKS.md b/PENDING_TASKS.md new file mode 100644 index 0000000..0b395f6 --- /dev/null +++ b/PENDING_TASKS.md @@ -0,0 +1,241 @@ +# General Bots - Pending Tasks for Next Sessions + +**Created:** Session cleanup +**Purpose:** Consolidated list of pending work for LLM continuation + +--- + +## ✅ COMPLETED THIS SESSION + +### 1. Sources Module - Knowledge Base Backend ✅ +**Location:** `botserver/src/sources/knowledge_base.rs` + +**Implemented:** +- `POST /api/sources/kb/upload` - Upload documents for ingestion +- `GET /api/sources/kb/list` - List ingested sources +- `POST /api/sources/kb/query` - Query knowledge base with full-text search +- `GET /api/sources/kb/:id` - Get source details +- `DELETE /api/sources/kb/:id` - Remove source +- `POST /api/sources/kb/reindex` - Re-process sources +- `GET /api/sources/kb/stats` - Get knowledge base statistics + +**Features:** +- Document chunking with configurable size/overlap +- Text extraction for PDF, DOCX, TXT, Markdown, HTML, CSV, XLSX +- Full-text search with PostgreSQL ts_rank +- Status tracking (pending, processing, indexed, failed, reindexing) + +--- + +### 2. Research Module - Web Search Backend ✅ +**Location:** `botserver/src/research/web_search.rs` + +**Implemented:** +- `POST /api/research/web/search` - Web search via DuckDuckGo +- `POST /api/research/web/summarize` - Summarize search results +- `POST /api/research/web/deep` - Deep research with multiple queries +- `GET /api/research/web/history` - Search history +- `GET /api/research/web/instant` - Instant answers from DuckDuckGo API + +**Features:** +- DuckDuckGo HTML scraping (no API key required) +- Result parsing with favicon extraction +- Related query generation +- Citation tracking + +--- + +### 3. App Generator - Full LLM-Based Generation ✅ +**Location:** `botserver/src/auto_task/app_generator.rs` + +**Completely rewritten to:** +- Generate ALL files (HTML, CSS, JS, BAS) via LLM +- Removed ALL hardcoded templates +- Single LLM call generates complete app structure +- Includes tables, pages, tools, schedulers + +--- + +### 4. App Logging System ✅ +**Location:** `botserver/src/auto_task/app_logs.rs` + +**Implemented:** +- Server-side log storage per app +- Client-side JavaScript logger (`/api/app-logs/logger.js`) +- Error context injection into Designer prompts +- Auto-cleanup scheduler (D-1 retention) + +**Endpoints:** +- `POST /api/app-logs/client` - Receive client logs +- `GET /api/app-logs/list` - List logs with filters +- `GET /api/app-logs/stats` - Log statistics +- `POST /api/app-logs/clear/{app_name}` - Clear app logs +- `GET /api/app-logs/logger.js` - Client logger script + +--- + +### 5. Database Migration ✅ +**Location:** `botserver/migrations/6.1.3_knowledge_base_sources/` + +**Created tables:** +- `knowledge_sources` - Uploaded documents metadata +- `knowledge_chunks` - Text chunks for RAG +- `research_search_history` - Search history tracking + +--- + +## 🔴 HIGH PRIORITY + +### 1. Calendar UI Completion + +**Location:** `botui/ui/suite/calendar/` +**Backend exists:** `botserver/src/calendar/` (fully implemented with CalDAV) + +**What's missing:** +- Week view +- Day view +- Drag-and-drop event moving +- Recurring events UI +- Calendar sharing UI + +**Backend is complete** - just needs frontend polish. + +--- + +### 2. Vector Embeddings Integration + +**Location:** `botserver/src/sources/knowledge_base.rs` + +**What's needed:** +- Connect to LLM for embedding generation +- Store embeddings in PostgreSQL pgvector +- Implement semantic search alongside full-text search +- Integrate with existing `drive/vectordb.rs` + +--- + +## 🟡 MEDIUM PRIORITY + +### 3. Meet Module - LiveKit Integration + +**Location:** `botserver/src/meet/` +**UI exists:** `botui/ui/suite/meet/` + +**What's missing:** +- LiveKit server configuration documentation +- Room creation and management +- Participant tracking +- Recording integration + +**Requires external setup:** +- LiveKit server (self-hosted or cloud) +- TURN/STUN servers for WebRTC + +--- + +### 4. Custom Domain - Config.csv Integration + +**Location:** `botserver/src/core/dns/` + +**Current state:** DNS routes exist but config.csv parsing not connected + +**What's needed:** +```csv +# In bot's config.csv +appname-domain,app.customerdomain.com +``` + +- Parse `appname-domain` from config.csv during bot load +- Register with DNS service automatically +- Auto-provision SSL via Let's Encrypt + +--- + +### 5. Designer Magic Button - LLM Integration + +**Location:** +- `botui/ui/suite/designer.html` (dialog designer - DONE) +- `botui/ui/suite/editor.html` (code editor - DONE) +- `botserver/src/designer/mod.rs` (endpoints - DONE) + +**What's missing:** +- Connect `/api/v1/editor/magic` to actual LLM when `feature = "llm"` is enabled +- Currently uses fallback suggestions only +- Need to test with LLM enabled + +--- + +## 🟢 LOW PRIORITY / POLISH + +### 6. SEO Meta Tags Verification + +Verify all HTMX pages have proper SEO: +- `botui/ui/suite/**/*.html` +- Generated apps from `app_generator.rs` + +Required tags: +```html + + + + +``` + +--- + +### 7. Login Flow Documentation + +**Credentials shown during setup:** +- Displayed in terminal with box formatting +- Pauses for user to copy +- NOT saved to file (security) + +**Location of display:** `botserver/src/core/package_manager/setup/directory_setup.rs` + +Consider adding: +- First-login wizard to change password +- Email verification flow +- Password recovery + +--- + +## 📋 Session Continuation Notes + +### Files Modified This Session: +- `botserver/src/auto_task/app_generator.rs` - Complete rewrite for LLM-only generation +- `botserver/src/auto_task/app_logs.rs` - NEW: App logging system +- `botserver/src/auto_task/mod.rs` - Added app_logs exports and routes +- `botserver/src/sources/mod.rs` - Added knowledge_base module +- `botserver/src/sources/knowledge_base.rs` - NEW: KB ingestion backend +- `botserver/src/research/mod.rs` - Added web_search module +- `botserver/src/research/web_search.rs` - NEW: Web search backend +- `botserver/src/designer/mod.rs` - Added error context to prompts +- `botserver/migrations/6.1.3_knowledge_base_sources/` - NEW: DB migration + +### Build Status: +- `cargo check -p botserver` - ✅ 0 errors, 0 warnings + +### How to Continue: +1. Pick a HIGH PRIORITY task +2. Read the relevant source files +3. Implement missing functionality +4. Test with `cargo check` +5. Update this file when complete + +--- + +## 🎯 Quick Start for Next Session + +``` +Start with: +1. "Complete Calendar UI - add week/day views" + OR +2. "Add vector embeddings to knowledge base" + OR +3. "Test app generator with LLM enabled" + +Context files to read first: +- botui/ui/suite/calendar/ +- botserver/src/sources/knowledge_base.rs +- botserver/src/auto_task/app_generator.rs +``` diff --git a/botapp b/botapp index bc875b5..5a82552 160000 --- a/botapp +++ b/botapp @@ -1 +1 @@ -Subproject commit bc875b511f7a9be11b8cb49d8be4f33e2998f015 +Subproject commit 5a82552a64982ef391605c37c09de805f2d97ae0 diff --git a/botbook b/botbook index 2ec5044..d5d147c 160000 --- a/botbook +++ b/botbook @@ -1 +1 @@ -Subproject commit 2ec504448d13f1cb3fed486be9c6a99387258889 +Subproject commit d5d147ce0ce54e9d6519440b87bb4dea5ddcfecc diff --git a/botdevice b/botdevice index f2afa7e..a3add8a 160000 --- a/botdevice +++ b/botdevice @@ -1 +1 @@ -Subproject commit f2afa7e7bebac98272940715f499c50381b30d4f +Subproject commit a3add8ada34a0313bac4285d20441bb643f74962 diff --git a/botlib b/botlib index 94f333f..a50d229 160000 --- a/botlib +++ b/botlib @@ -1 +1 @@ -Subproject commit 94f333f983aa7f4fcfc541b9d6f52f2261d3095a +Subproject commit a50d229346167c539d3fbae2dcf26160df311e41 diff --git a/botserver b/botserver index 9920538..643c101 160000 --- a/botserver +++ b/botserver @@ -1 +1 @@ -Subproject commit 9920538708da467c925953df03960e9c991a3004 +Subproject commit 643c101d7dad4242c76cf16fac93d6af690d9d5f diff --git a/bottest b/bottest index b38574c..1232b2f 160000 --- a/bottest +++ b/bottest @@ -1 +1 @@ -Subproject commit b38574c5885d356b026f5060af51ad02d2fcef21 +Subproject commit 1232b2fc6540595fe683115bb3499ccddf1d7270 diff --git a/botui b/botui index d96c546..3f95c46 160000 --- a/botui +++ b/botui @@ -1 +1 @@ -Subproject commit d96c546c6afadb572c462406668f8f89729fe247 +Subproject commit 3f95c4645d69b68186849e5aac68c433e93bd545