diff --git a/PROMPT.md b/PROMPT.md index 5dd7e58..9ea148c 100644 --- a/PROMPT.md +++ b/PROMPT.md @@ -135,6 +135,35 @@ botbook/ # Documentation - NO custom JavaScript where HTMX can handle it ``` +### JavaScript Usage Guidelines + +**JS is ONLY acceptable when HTMX cannot handle the requirement:** + +| Use Case | Solution | +|----------|----------| +| Data fetching | HTMX `hx-get`, `hx-post` | +| Form submission | HTMX `hx-post`, `hx-put` | +| Real-time updates | HTMX WebSocket extension `hx-ext="ws"` | +| Content swapping | HTMX `hx-target`, `hx-swap` | +| Polling | HTMX `hx-trigger="every 5s"` | +| Loading states | HTMX `hx-indicator` | +| **Modal show/hide** | **JS required** - DOM manipulation | +| **Toast notifications** | **JS required** - dynamic element creation | +| **Clipboard operations** | **JS required** - `navigator.clipboard` API | +| **Keyboard shortcuts** | **JS required** - `keydown` event handling | +| **WebSocket state mgmt** | **JS required** - connection lifecycle | +| **Complex animations** | **JS required** - GSAP or custom | +| **Client-side validation** | **JS required** - before submission UX | + +**When writing JS:** +``` +- Keep it minimal - one function per concern +- No frameworks (React, Vue, etc.) - vanilla JS only +- Use vendor libs sparingly (htmx, marked, gsap, alpine) +- All JS must work with HTMX lifecycle (htmx:afterSwap, etc.) +- Prefer CSS for animations when possible +``` + ### Local Assets Only All external libraries are bundled locally - NEVER use CDN: @@ -346,7 +375,7 @@ grep -r "unpkg.com\|cdnjs\|jsdelivr" ui/ ``` src/main.rs # Entry point, mode detection src/lib.rs # Feature-gated exports -src/http_client.rs # BotServerClient wrapper +src/http_client.rs # botserverClient wrapper src/ui_server/mod.rs # Axum router, static files ui/suite/index.html # Main UI entry ui/suite/base.html # Base template diff --git a/README.md b/README.md index a3c7d93..c920a3c 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # General Bots Desktop -An AI-powered desktop automation tool that records and plays back user interactions useful for legacy systems and common desktop tasks. The BotDesktop automation tool fills a critical gap in the enterprise automation landscape by addressing legacy systems and desktop applications that lack modern APIs or integration capabilities. While BotServer excels at creating conversational bots for modern channels like web, mobile and messaging platforms, many organizations still rely heavily on traditional desktop applications, mainframe systems, and custom internal tools that can only be accessed through their user interface. BotDesktop's ability to record and replay user interactions provides a practical bridge between these legacy systems and modern automation needs. +An AI-powered desktop automation tool that records and plays back user interactions useful for legacy systems and common desktop tasks. The BotDesktop automation tool fills a critical gap in the enterprise automation landscape by addressing legacy systems and desktop applications that lack modern APIs or integration capabilities. While botserver excels at creating conversational bots for modern channels like web, mobile and messaging platforms, many organizations still rely heavily on traditional desktop applications, mainframe systems, and custom internal tools that can only be accessed through their user interface. BotDesktop's ability to record and replay user interactions provides a practical bridge between these legacy systems and modern automation needs.  @@ -10,10 +10,10 @@ An AI-powered desktop automation tool that records and plays back user interacti The tool's AI-powered approach to desktop automation represents a significant advancement over traditional robotic process automation (RPA) tools. By leveraging machine learning to understand screen elements and user interactions, BotDesktop can adapt to minor UI changes and variations that would break conventional scripted automation. This resilience is particularly valuable in enterprise environments where applications receive regular updates or where slight variations exist between different versions or installations of the same software. The AI component also simplifies the creation of automation scripts - instead of requiring complex programming, users can simply demonstrate the desired actions which BotDesktop observes and learns to replicate. -From an integration perspective, BotDesktop complements BotServer by enabling end-to-end automation scenarios that span both modern and legacy systems. For example, a bot created in BotServer could collect information from users through a modern chat interface, then use BotDesktop to input that data into a legacy desktop application that lacks API access. This hybrid approach allows organizations to modernize their user interactions while still leveraging their existing IT investments. Additionally, BotDesktop can automate routine desktop tasks like file management, data entry, and application monitoring that fall outside the scope of conversational bot interactions. +From an integration perspective, BotDesktop complements botserver by enabling end-to-end automation scenarios that span both modern and legacy systems. For example, a bot created in botserver could collect information from users through a modern chat interface, then use BotDesktop to input that data into a legacy desktop application that lacks API access. This hybrid approach allows organizations to modernize their user interactions while still leveraging their existing IT investments. Additionally, BotDesktop can automate routine desktop tasks like file management, data entry, and application monitoring that fall outside the scope of conversational bot interactions. -The combined toolset of BotServer and BotDesktop provides organizations with comprehensive automation capabilities across their entire technology stack. While BotServer handles the modern, API-driven interactions with users across multiple channels, BotDesktop extends automation capabilities to the desktop environment where many critical business processes still reside. This dual approach allows organizations to progressively modernize their systems while maintaining operational efficiency through automation of both new and legacy components. The result is a more flexible and complete automation solution that can adapt to various technical environments and business needs. +The combined toolset of botserver and BotDesktop provides organizations with comprehensive automation capabilities across their entire technology stack. While botserver handles the modern, API-driven interactions with users across multiple channels, BotDesktop extends automation capabilities to the desktop environment where many critical business processes still reside. This dual approach allows organizations to progressively modernize their systems while maintaining operational efficiency through automation of both new and legacy components. The result is a more flexible and complete automation solution that can adapt to various technical environments and business needs. ## Setup 1. Install dependencies: diff --git a/ui/suite/tasks/autotask.js b/ui/suite/tasks/autotask.js index 0fc15b4..8df662f 100644 --- a/ui/suite/tasks/autotask.js +++ b/ui/suite/tasks/autotask.js @@ -8,40 +8,40 @@ // ============================================================================= const AutoTaskState = { - currentFilter: 'all', - tasks: [], - compiledPlan: null, - pendingDecisions: [], - pendingApprovals: [], - refreshInterval: null, - wsConnection: null + currentFilter: "all", + tasks: [], + compiledPlan: null, + pendingDecisions: [], + pendingApprovals: [], + refreshInterval: null, + wsConnection: null, }; // ============================================================================= // INITIALIZATION // ============================================================================= -document.addEventListener('DOMContentLoaded', function() { - initAutoTask(); +document.addEventListener("DOMContentLoaded", function () { + initAutoTask(); }); function initAutoTask() { - // Initialize WebSocket for real-time updates - initWebSocket(); + // Initialize WebSocket for real-time updates + initWebSocket(); - // Start auto-refresh - startAutoRefresh(); + // Start auto-refresh + startAutoRefresh(); - // Setup event listeners - setupEventListeners(); + // Setup event listeners + setupEventListeners(); - // Load initial stats - updateStats(); + // Load initial stats + updateStats(); - // Setup keyboard shortcuts - setupKeyboardShortcuts(); + // Setup keyboard shortcuts + setupKeyboardShortcuts(); - console.log('AutoTask initialized'); + console.log("AutoTask initialized"); } // ============================================================================= @@ -49,57 +49,57 @@ function initAutoTask() { // ============================================================================= function initWebSocket() { - const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'; - const wsUrl = `${protocol}//${window.location.host}/ws/autotask`; + const protocol = window.location.protocol === "https:" ? "wss:" : "ws:"; + const wsUrl = `${protocol}//${window.location.host}/ws/autotask`; - try { - AutoTaskState.wsConnection = new WebSocket(wsUrl); + try { + AutoTaskState.wsConnection = new WebSocket(wsUrl); - AutoTaskState.wsConnection.onopen = function() { - console.log('AutoTask WebSocket connected'); - }; + AutoTaskState.wsConnection.onopen = function () { + console.log("AutoTask WebSocket connected"); + }; - AutoTaskState.wsConnection.onmessage = function(event) { - handleWebSocketMessage(JSON.parse(event.data)); - }; + AutoTaskState.wsConnection.onmessage = function (event) { + handleWebSocketMessage(JSON.parse(event.data)); + }; - AutoTaskState.wsConnection.onclose = function() { - console.log('AutoTask WebSocket disconnected, reconnecting...'); - setTimeout(initWebSocket, 5000); - }; + AutoTaskState.wsConnection.onclose = function () { + console.log("AutoTask WebSocket disconnected, reconnecting..."); + setTimeout(initWebSocket, 5000); + }; - AutoTaskState.wsConnection.onerror = function(error) { - console.error('AutoTask WebSocket error:', error); - }; - } catch (e) { - console.warn('WebSocket not available, using polling'); - } + AutoTaskState.wsConnection.onerror = function (error) { + console.error("AutoTask WebSocket error:", error); + }; + } catch (e) { + console.warn("WebSocket not available, using polling"); + } } function handleWebSocketMessage(data) { - switch (data.type) { - case 'task_update': - updateTaskInList(data.task); - break; - case 'step_progress': - updateStepProgress(data.taskId, data.step, data.progress); - break; - case 'decision_required': - showDecisionNotification(data.decision); - break; - case 'approval_required': - showApprovalNotification(data.approval); - break; - case 'task_completed': - onTaskCompleted(data.task); - break; - case 'task_failed': - onTaskFailed(data.task, data.error); - break; - case 'stats_update': - updateStatsFromData(data.stats); - break; - } + switch (data.type) { + case "task_update": + updateTaskInList(data.task); + break; + case "step_progress": + updateStepProgress(data.taskId, data.step, data.progress); + break; + case "decision_required": + showDecisionNotification(data.decision); + break; + case "approval_required": + showApprovalNotification(data.approval); + break; + case "task_completed": + onTaskCompleted(data.task); + break; + case "task_failed": + onTaskFailed(data.task, data.error); + break; + case "stats_update": + updateStatsFromData(data.stats); + break; + } } // ============================================================================= @@ -107,66 +107,73 @@ function handleWebSocketMessage(data) { // ============================================================================= function setupEventListeners() { - // Intent form submission - const intentForm = document.getElementById('intent-form'); - if (intentForm) { - intentForm.addEventListener('htmx:afterSwap', function(event) { - if (event.detail.target.id === 'compilation-result') { - onCompilationComplete(event); - } - }); - } + // Intent form submission + const intentForm = document.getElementById("intent-form"); + if (intentForm) { + intentForm.addEventListener("htmx:afterSwap", function (event) { + if (event.detail.target.id === "compilation-result") { + onCompilationComplete(event); + } + }); + } - // Task list updates - const taskList = document.getElementById('task-list'); - if (taskList) { - taskList.addEventListener('htmx:afterSwap', function() { - updateStats(); - highlightPendingItems(); - }); - } + // Task list updates + const taskList = document.getElementById("task-list"); + if (taskList) { + taskList.addEventListener("htmx:afterSwap", function () { + updateStats(); + highlightPendingItems(); + }); + } - // Expand log entries on details open - document.addEventListener('toggle', function(event) { - if (event.target.classList.contains('execution-log') && event.target.open) { - const taskId = event.target.closest('.autotask-item')?.dataset.taskId; - if (taskId) { - loadExecutionLogs(taskId); - } + // Expand log entries on details open + document.addEventListener( + "toggle", + function (event) { + if ( + event.target.classList.contains("execution-log") && + event.target.open + ) { + const taskId = event.target.closest(".autotask-item")?.dataset.taskId; + if (taskId) { + loadExecutionLogs(taskId); } - }, true); + } + }, + true, + ); } function setupKeyboardShortcuts() { - document.addEventListener('keydown', function(e) { - // Alt + N: Focus on intent input - if (e.altKey && e.key === 'n') { - e.preventDefault(); - document.getElementById('intent-input')?.focus(); - } + document.addEventListener("keydown", function (e) { + // Alt + N: Focus on intent input + if (e.altKey && e.key === "n") { + e.preventDefault(); + document.getElementById("intent-input")?.focus(); + } - // Alt + R: Refresh tasks - if (e.altKey && e.key === 'r') { - e.preventDefault(); - refreshTasks(); - } + // Alt + R: Refresh tasks + if (e.altKey && e.key === "r") { + e.preventDefault(); + refreshTasks(); + } - // Escape: Close any open modal - if (e.key === 'Escape') { - closeAllModals(); - } + // Escape: Close any open modal + if (e.key === "Escape") { + closeAllModals(); + } - // Alt + 1-4: Switch filters - if (e.altKey && e.key >= '1' && e.key <= '4') { - e.preventDefault(); - const filters = ['all', 'running', 'approval', 'decision']; - const index = parseInt(e.key) - 1; - const tabs = document.querySelectorAll('.filter-tab'); - if (tabs[index]) { - tabs[index].click(); - } - } - }); + // Alt + 1-4: Switch filters + if (e.altKey && e.key >= "1" && e.key <= "4") { + e.preventDefault(); + const filters = ["all", "running", "approval", "decision"]; + const index = parseInt(e.key) - 1; + const tabs = document.querySelectorAll(".filter-tab"); + if (tabs[index]) { + tabs[index].click(); + } + } + }); } // ============================================================================= @@ -174,19 +181,19 @@ function setupKeyboardShortcuts() { // ============================================================================= function startAutoRefresh() { - // Refresh every 5 seconds - AutoTaskState.refreshInterval = setInterval(function() { - if (!document.hidden) { - updateStats(); - } - }, 5000); + // Refresh every 5 seconds + AutoTaskState.refreshInterval = setInterval(function () { + if (!document.hidden) { + updateStats(); + } + }, 5000); } function stopAutoRefresh() { - if (AutoTaskState.refreshInterval) { - clearInterval(AutoTaskState.refreshInterval); - AutoTaskState.refreshInterval = null; - } + if (AutoTaskState.refreshInterval) { + clearInterval(AutoTaskState.refreshInterval); + AutoTaskState.refreshInterval = null; + } } // ============================================================================= @@ -194,36 +201,39 @@ function stopAutoRefresh() { // ============================================================================= function updateStats() { - fetch('/api/autotask/stats') - .then(response => response.json()) - .then(stats => { - updateStatsFromData(stats); - }) - .catch(error => { - console.error('Failed to fetch stats:', error); - }); + fetch("/api/autotask/stats") + .then((response) => response.json()) + .then((stats) => { + updateStatsFromData(stats); + }) + .catch((error) => { + console.error("Failed to fetch stats:", error); + }); } function updateStatsFromData(stats) { - // Header stats - document.getElementById('stat-running').textContent = stats.running || 0; - document.getElementById('stat-pending').textContent = stats.pending || 0; - document.getElementById('stat-completed').textContent = stats.completed || 0; - document.getElementById('stat-approval').textContent = stats.pending_approval || 0; + // Header stats + document.getElementById("stat-running").textContent = stats.running || 0; + document.getElementById("stat-pending").textContent = stats.pending || 0; + document.getElementById("stat-completed").textContent = stats.completed || 0; + document.getElementById("stat-approval").textContent = + stats.pending_approval || 0; - // Filter counts - document.getElementById('count-all').textContent = stats.total || 0; - document.getElementById('count-running').textContent = stats.running || 0; - document.getElementById('count-approval').textContent = stats.pending_approval || 0; - document.getElementById('count-decision').textContent = stats.pending_decision || 0; + // Filter counts + document.getElementById("count-all").textContent = stats.total || 0; + document.getElementById("count-running").textContent = stats.running || 0; + document.getElementById("count-approval").textContent = + stats.pending_approval || 0; + document.getElementById("count-decision").textContent = + stats.pending_decision || 0; - // Highlight if approvals needed - const approvalStat = document.querySelector('.stat-item.highlight'); - if (approvalStat && stats.pending_approval > 0) { - approvalStat.classList.add('attention'); - } else if (approvalStat) { - approvalStat.classList.remove('attention'); - } + // Highlight if approvals needed + const approvalStat = document.querySelector(".stat-item.highlight"); + if (approvalStat && stats.pending_approval > 0) { + approvalStat.classList.add("attention"); + } else if (approvalStat) { + approvalStat.classList.remove("attention"); + } } // ============================================================================= @@ -231,28 +241,28 @@ function updateStatsFromData(stats) { // ============================================================================= function filterTasks(filter, button) { - AutoTaskState.currentFilter = filter; + AutoTaskState.currentFilter = filter; - // Update active tab - document.querySelectorAll('.filter-tab').forEach(tab => { - tab.classList.remove('active'); - }); - button.classList.add('active'); + // Update active tab + document.querySelectorAll(".filter-tab").forEach((tab) => { + tab.classList.remove("active"); + }); + button.classList.add("active"); - // Trigger HTMX request - htmx.ajax('GET', `/api/autotask/list?filter=${filter}`, { - target: '#task-list', - swap: 'innerHTML' - }); + // Trigger HTMX request + htmx.ajax("GET", `/api/autotask/list?filter=${filter}`, { + target: "#task-list", + swap: "innerHTML", + }); } function refreshTasks() { - const filter = AutoTaskState.currentFilter; - htmx.ajax('GET', `/api/autotask/list?filter=${filter}`, { - target: '#task-list', - swap: 'innerHTML' - }); - updateStats(); + const filter = AutoTaskState.currentFilter; + htmx.ajax("GET", `/api/autotask/list?filter=${filter}`, { + target: "#task-list", + swap: "innerHTML", + }); + updateStats(); } // ============================================================================= @@ -260,78 +270,111 @@ function refreshTasks() { // ============================================================================= function onCompilationComplete(event) { - const result = event.detail.target.querySelector('.compiled-plan'); - if (result) { - // Scroll to result - result.scrollIntoView({ behavior: 'smooth', block: 'start' }); + const result = event.detail.target.querySelector(".compiled-plan"); + if (result) { + // Scroll to result + result.scrollIntoView({ behavior: "smooth", block: "start" }); - // Store compiled plan - const planId = result.dataset?.planId; - if (planId) { - AutoTaskState.compiledPlan = planId; - } - - // Syntax highlight the code - highlightBasicCode(); + // Store compiled plan + const planId = result.dataset?.planId; + if (planId) { + AutoTaskState.compiledPlan = planId; } + + // Syntax highlight the code + highlightBasicCode(); + } } function highlightBasicCode() { - const codeBlocks = document.querySelectorAll('.code-preview code'); - codeBlocks.forEach(block => { - // Basic syntax highlighting for BASIC keywords - let html = block.innerHTML; + const codeBlocks = document.querySelectorAll(".code-preview code"); + codeBlocks.forEach((block) => { + // Basic syntax highlighting for BASIC keywords + let html = block.innerHTML; - // Keywords - const keywords = [ - 'PLAN_START', 'PLAN_END', 'STEP', 'SET', 'GET', 'IF', 'THEN', 'ELSE', 'END IF', - 'FOR EACH', 'NEXT', 'WHILE', 'WEND', 'TALK', 'HEAR', 'LLM', 'CREATE_TASK', - 'RUN_PYTHON', 'RUN_JAVASCRIPT', 'RUN_BASH', 'USE_MCP', 'POST', 'GET', 'PUT', - 'PATCH', 'DELETE HTTP', 'REQUIRE_APPROVAL', 'SIMULATE_IMPACT', 'AUDIT_LOG', - 'SEND_MAIL', 'SAVE', 'UPDATE', 'INSERT', 'DELETE', 'FIND' - ]; + // Keywords + const keywords = [ + "PLAN_START", + "PLAN_END", + "STEP", + "SET", + "GET", + "IF", + "THEN", + "ELSE", + "END IF", + "FOR EACH", + "NEXT", + "WHILE", + "WEND", + "TALK", + "HEAR", + "LLM", + "CREATE_TASK", + "RUN_PYTHON", + "RUN_JAVASCRIPT", + "RUN_BASH", + "USE_MCP", + "POST", + "GET", + "PUT", + "PATCH", + "DELETE HTTP", + "REQUIRE_APPROVAL", + "SIMULATE_IMPACT", + "AUDIT_LOG", + "SEND_MAIL", + "SAVE", + "UPDATE", + "INSERT", + "DELETE", + "FIND", + ]; - keywords.forEach(keyword => { - const regex = new RegExp(`\\b${keyword}\\b`, 'g'); - html = html.replace(regex, `${keyword}`); - }); - - // Comments - html = html.replace(/(\'[^\n]*)/g, '$1'); - - // Strings - html = html.replace(/("[^"]*")/g, '$1'); - - // Numbers - html = html.replace(/\b(\d+)\b/g, '$1'); - - block.innerHTML = html; + keywords.forEach((keyword) => { + const regex = new RegExp(`\\b${keyword}\\b`, "g"); + html = html.replace(regex, `${keyword}`); }); + + // Comments + html = html.replace(/(\'[^\n]*)/g, '$1'); + + // Strings + html = html.replace(/("[^"]*")/g, '$1'); + + // Numbers + html = html.replace(/\b(\d+)\b/g, '$1'); + + block.innerHTML = html; + }); } function copyGeneratedCode() { - const code = document.querySelector('.code-preview code')?.textContent; - if (code) { - navigator.clipboard.writeText(code).then(() => { - showToast('Code copied to clipboard', 'success'); - }).catch(() => { - showToast('Failed to copy code', 'error'); - }); - } + const code = document.querySelector(".code-preview code")?.textContent; + if (code) { + navigator.clipboard + .writeText(code) + .then(() => { + showToast("Code copied to clipboard", "success"); + }) + .catch(() => { + showToast("Failed to copy code", "error"); + }); + } } function discardPlan() { - if (confirm('Are you sure you want to discard this plan?')) { - document.getElementById('compilation-result').innerHTML = ''; - AutoTaskState.compiledPlan = null; - document.getElementById('intent-input').value = ''; - document.getElementById('intent-input').focus(); - } + if (confirm("Are you sure you want to discard this plan?")) { + document.getElementById("compilation-result").innerHTML = ""; + AutoTaskState.compiledPlan = null; + document.getElementById("intent-input").value = ""; + document.getElementById("intent-input").focus(); + } } function editPlan() { - // TODO: Implement plan editor - showToast('Plan editor coming soon!', 'info'); + // TODO: Implement plan editor + showToast("Plan editor coming soon!", "info"); } // ============================================================================= @@ -339,17 +382,17 @@ function editPlan() { // ============================================================================= function simulatePlan(planId) { - showSimulationModal(); + showSimulationModal(); - fetch(`/api/autotask/simulate/${planId}`, { - method: 'POST' + fetch(`/api/autotask/simulate/${planId}`, { + method: "POST", + }) + .then((response) => response.json()) + .then((result) => { + renderSimulationResult(result); }) - .then(response => response.json()) - .then(result => { - renderSimulationResult(result); - }) - .catch(error => { - document.getElementById('simulation-content').innerHTML = ` + .catch((error) => { + document.getElementById("simulation-content").innerHTML = `