Add docs, UI pages, code scanner, and Docker deployment guide
- Add CRM contacts template documentation - Add Docker deployment documentation with compose examples - Add BASIC code scanner for security compliance checking - Add visual dialog designer UI (designer.html) - Add drive file manager UI (drive/index.html) - Add sources browser UI (sources/index.html) - Add compliance report tool UI (tools/compliance.html)
This commit is contained in:
parent
3ca3a2c3e3
commit
5edb45133f
9 changed files with 7316 additions and 0 deletions
|
|
@ -22,6 +22,7 @@
|
|||
- [.gbtheme UI Theming](./chapter-02/gbtheme.md)
|
||||
- [.gbdrive File Storage](./chapter-02/gbdrive.md)
|
||||
- [Bot Templates](./chapter-02/templates.md)
|
||||
- [Template: CRM Contacts](./chapter-02/template-crm-contacts.md)
|
||||
|
||||
# Part III - Knowledge Base
|
||||
|
||||
|
|
@ -151,6 +152,7 @@
|
|||
- [Architecture Overview](./chapter-07-gbapp/architecture.md)
|
||||
- [Building from Source](./chapter-07-gbapp/building.md)
|
||||
- [Container Deployment (LXC)](./chapter-07-gbapp/containers.md)
|
||||
- [Docker Deployment](./chapter-07-gbapp/docker-deployment.md)
|
||||
- [Scaling and Load Balancing](./chapter-07-gbapp/scaling.md)
|
||||
- [Infrastructure Design](./chapter-07-gbapp/infrastructure.md)
|
||||
- [Observability](./chapter-07-gbapp/observability.md)
|
||||
|
|
|
|||
441
docs/src/chapter-02/template-crm-contacts.md
Normal file
441
docs/src/chapter-02/template-crm-contacts.md
Normal file
|
|
@ -0,0 +1,441 @@
|
|||
# CRM Contacts Template
|
||||
|
||||
The CRM Contacts template provides a complete contact management solution with natural language interface. Users can add, search, update, and manage contacts through conversational interactions.
|
||||
|
||||
## Overview
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Template ID | `crm/contacts.gbai` |
|
||||
| Category | CRM |
|
||||
| Complexity | Intermediate |
|
||||
| Dependencies | Database, Email (optional) |
|
||||
|
||||
## Features
|
||||
|
||||
- Add new contacts with validation
|
||||
- Search contacts by name, email, or phone
|
||||
- Update contact information
|
||||
- Tag and categorize contacts
|
||||
- Export contacts to CSV
|
||||
- Integration with email campaigns
|
||||
|
||||
## Installation
|
||||
|
||||
```basic
|
||||
DEPLOY TEMPLATE "crm/contacts.gbai"
|
||||
```
|
||||
|
||||
Or copy the template folder to your work directory:
|
||||
|
||||
```bash
|
||||
cp -r templates/crm/contacts.gbai work/
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Add these settings to your `config.csv`:
|
||||
|
||||
| Key | Value | Description |
|
||||
|-----|-------|-------------|
|
||||
| `contacts-table` | `contacts` | Database table name |
|
||||
| `require-email` | `true` | Require email for new contacts |
|
||||
| `allow-duplicates` | `false` | Allow duplicate entries |
|
||||
| `default-tags` | `lead` | Default tags for new contacts |
|
||||
|
||||
## Sample Conversation
|
||||
|
||||
<div class="chat-container">
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">Hi, I need to add a new contact</div>
|
||||
<div class="chat-time">10:32 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">I'd be happy to help you add a new contact! 📇
|
||||
|
||||
What's the contact's full name?</div>
|
||||
<div class="chat-time">10:32 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">Maria Santos</div>
|
||||
<div class="chat-time">10:32 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">Got it! Maria Santos.
|
||||
|
||||
What's Maria's email address?</div>
|
||||
<div class="chat-time">10:32 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">maria.santos@company.com</div>
|
||||
<div class="chat-time">10:33 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">Perfect! And what's the phone number? (You can skip this by typing "skip")</div>
|
||||
<div class="chat-time">10:33 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">+55 11 98765-4321</div>
|
||||
<div class="chat-time">10:33 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">Which company does Maria work for?</div>
|
||||
<div class="chat-time">10:33 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">Tech Solutions Ltd</div>
|
||||
<div class="chat-time">10:34 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">Great! Here's the contact I'm about to save:
|
||||
|
||||
📋 **Contact Details**
|
||||
━━━━━━━━━━━━━━━━━
|
||||
👤 **Name:** Maria Santos
|
||||
📧 **Email:** maria.santos@company.com
|
||||
📱 **Phone:** +55 11 98765-4321
|
||||
🏢 **Company:** Tech Solutions Ltd
|
||||
🏷️ **Tags:** lead
|
||||
|
||||
Should I save this contact?</div>
|
||||
<div class="chat-time">10:34 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">Yes, save it</div>
|
||||
<div class="chat-time">10:34 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">✅ Contact saved successfully!
|
||||
|
||||
Maria Santos has been added to your contacts with ID #1247.
|
||||
|
||||
What would you like to do next?
|
||||
• Add another contact
|
||||
• Search contacts
|
||||
• View recent contacts</div>
|
||||
<div class="chat-time">10:34 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message user">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">You</div>
|
||||
<div class="chat-text">Search for contacts at Tech Solutions</div>
|
||||
<div class="chat-time">10:35 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chat-message bot">
|
||||
<div class="chat-bubble">
|
||||
<div class="chat-sender">CRM Bot</div>
|
||||
<div class="chat-text">🔍 Found 3 contacts at "Tech Solutions":
|
||||
|
||||
1. **Maria Santos** - maria.santos@company.com
|
||||
📱 +55 11 98765-4321 | 🏷️ lead
|
||||
|
||||
2. **João Silva** - joao.silva@techsolutions.com
|
||||
📱 +55 11 91234-5678 | 🏷️ customer
|
||||
|
||||
3. **Ana Costa** - ana.costa@techsolutions.com
|
||||
📱 +55 11 99876-5432 | 🏷️ prospect
|
||||
|
||||
Type a number to see full details, or tell me what else you need.</div>
|
||||
<div class="chat-time">10:35 AM</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
## Dialog Files
|
||||
|
||||
### start.bas
|
||||
|
||||
The main entry point for the CRM contacts bot:
|
||||
|
||||
```basic
|
||||
' CRM Contacts - Main Dialog
|
||||
' Handles contact management through conversation
|
||||
|
||||
SET CONTEXT "You are a helpful CRM assistant that manages contacts.
|
||||
Be professional and efficient. Always confirm before saving data."
|
||||
|
||||
ADD SUGGESTION "add" AS "Add new contact"
|
||||
ADD SUGGESTION "search" AS "Search contacts"
|
||||
ADD SUGGESTION "recent" AS "View recent"
|
||||
ADD SUGGESTION "export" AS "Export to CSV"
|
||||
|
||||
BEGIN TALK
|
||||
Welcome to Contact Manager! 📇
|
||||
|
||||
I can help you:
|
||||
• Add new contacts
|
||||
• Search existing contacts
|
||||
• Update contact information
|
||||
• Export your contact list
|
||||
|
||||
What would you like to do?
|
||||
END TALK
|
||||
```
|
||||
|
||||
### add-contact.bas
|
||||
|
||||
Handles adding new contacts:
|
||||
|
||||
```basic
|
||||
' Add Contact Dialog
|
||||
|
||||
TALK "I'd be happy to help you add a new contact! 📇"
|
||||
TALK "What's the contact's full name?"
|
||||
|
||||
HEAR contact_name AS STRING
|
||||
|
||||
TALK "Got it! " + contact_name + "."
|
||||
TALK "What's " + contact_name + "'s email address?"
|
||||
|
||||
HEAR contact_email AS EMAIL
|
||||
|
||||
TALK "Perfect! And what's the phone number? (You can skip this by typing \"skip\")"
|
||||
|
||||
HEAR contact_phone AS STRING
|
||||
|
||||
IF contact_phone = "skip" THEN
|
||||
contact_phone = ""
|
||||
END IF
|
||||
|
||||
TALK "Which company does " + contact_name + " work for?"
|
||||
|
||||
HEAR contact_company AS STRING
|
||||
|
||||
' Build confirmation message
|
||||
WITH contact_data
|
||||
name = contact_name
|
||||
email = contact_email
|
||||
phone = contact_phone
|
||||
company = contact_company
|
||||
tags = "lead"
|
||||
created_at = NOW()
|
||||
END WITH
|
||||
|
||||
TALK "Great! Here's the contact I'm about to save:"
|
||||
TALK ""
|
||||
TALK "📋 **Contact Details**"
|
||||
TALK "━━━━━━━━━━━━━━━━━"
|
||||
TALK "👤 **Name:** " + contact_name
|
||||
TALK "📧 **Email:** " + contact_email
|
||||
TALK "📱 **Phone:** " + contact_phone
|
||||
TALK "🏢 **Company:** " + contact_company
|
||||
TALK "🏷️ **Tags:** lead"
|
||||
TALK ""
|
||||
TALK "Should I save this contact?"
|
||||
|
||||
HEAR confirmation AS STRING
|
||||
|
||||
IF INSTR(LOWER(confirmation), "yes") > 0 OR INSTR(LOWER(confirmation), "save") > 0 THEN
|
||||
SAVE "contacts", contact_data
|
||||
|
||||
contact_id = LAST("contacts", "id")
|
||||
|
||||
TALK "✅ Contact saved successfully!"
|
||||
TALK ""
|
||||
TALK contact_name + " has been added to your contacts with ID #" + contact_id + "."
|
||||
ELSE
|
||||
TALK "No problem! The contact was not saved."
|
||||
END IF
|
||||
|
||||
TALK ""
|
||||
TALK "What would you like to do next?"
|
||||
TALK "• Add another contact"
|
||||
TALK "• Search contacts"
|
||||
TALK "• View recent contacts"
|
||||
```
|
||||
|
||||
### search-contact.bas
|
||||
|
||||
Handles contact search:
|
||||
|
||||
```basic
|
||||
' Search Contact Dialog
|
||||
|
||||
TALK "🔍 What would you like to search for?"
|
||||
TALK "You can search by name, email, company, or phone number."
|
||||
|
||||
HEAR search_term AS STRING
|
||||
|
||||
' Search across multiple fields
|
||||
results = FIND "contacts" WHERE
|
||||
name LIKE "%" + search_term + "%" OR
|
||||
email LIKE "%" + search_term + "%" OR
|
||||
company LIKE "%" + search_term + "%" OR
|
||||
phone LIKE "%" + search_term + "%"
|
||||
|
||||
result_count = COUNT(results)
|
||||
|
||||
IF result_count = 0 THEN
|
||||
TALK "No contacts found matching \"" + search_term + "\"."
|
||||
TALK ""
|
||||
TALK "Would you like to:"
|
||||
TALK "• Try a different search"
|
||||
TALK "• Add a new contact"
|
||||
ELSE
|
||||
TALK "🔍 Found " + result_count + " contact(s) matching \"" + search_term + "\":"
|
||||
TALK ""
|
||||
|
||||
counter = 1
|
||||
FOR EACH contact IN results
|
||||
TALK counter + ". **" + contact.name + "** - " + contact.email
|
||||
TALK " 📱 " + contact.phone + " | 🏷️ " + contact.tags
|
||||
TALK ""
|
||||
counter = counter + 1
|
||||
NEXT
|
||||
|
||||
TALK "Type a number to see full details, or tell me what else you need."
|
||||
END IF
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
The template creates this table structure:
|
||||
|
||||
```sql
|
||||
CREATE TABLE contacts (
|
||||
id SERIAL PRIMARY KEY,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
email VARCHAR(255) UNIQUE,
|
||||
phone VARCHAR(50),
|
||||
company VARCHAR(255),
|
||||
tags VARCHAR(255) DEFAULT 'lead',
|
||||
notes TEXT,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
created_by UUID REFERENCES users(id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_contacts_email ON contacts(email);
|
||||
CREATE INDEX idx_contacts_company ON contacts(company);
|
||||
CREATE INDEX idx_contacts_tags ON contacts(tags);
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
The template exposes these REST endpoints:
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/api/v1/contacts` | List all contacts |
|
||||
| GET | `/api/v1/contacts/:id` | Get single contact |
|
||||
| POST | `/api/v1/contacts` | Create contact |
|
||||
| PUT | `/api/v1/contacts/:id` | Update contact |
|
||||
| DELETE | `/api/v1/contacts/:id` | Delete contact |
|
||||
| GET | `/api/v1/contacts/search?q=` | Search contacts |
|
||||
| GET | `/api/v1/contacts/export` | Export to CSV |
|
||||
|
||||
## Customization
|
||||
|
||||
### Adding Custom Fields
|
||||
|
||||
Edit `tables.bas` to add custom fields:
|
||||
|
||||
```basic
|
||||
TABLE contacts
|
||||
FIELD id AS INTEGER PRIMARY KEY
|
||||
FIELD name AS STRING(255) REQUIRED
|
||||
FIELD email AS EMAIL UNIQUE
|
||||
FIELD phone AS PHONE
|
||||
FIELD company AS STRING(255)
|
||||
FIELD tags AS STRING(255)
|
||||
FIELD notes AS TEXT
|
||||
' Add your custom fields below
|
||||
FIELD linkedin AS STRING(255)
|
||||
FIELD job_title AS STRING(255)
|
||||
FIELD lead_source AS STRING(100)
|
||||
FIELD lead_score AS INTEGER DEFAULT 0
|
||||
END TABLE
|
||||
```
|
||||
|
||||
### Changing Default Tags
|
||||
|
||||
Update `config.csv`:
|
||||
|
||||
```csv
|
||||
key,value
|
||||
default-tags,"prospect,website"
|
||||
```
|
||||
|
||||
### Adding Validation
|
||||
|
||||
Edit `add-contact.bas` to add custom validation:
|
||||
|
||||
```basic
|
||||
' Validate email domain
|
||||
IF NOT INSTR(contact_email, "@company.com") THEN
|
||||
TALK "⚠️ Warning: This email is not from your company domain."
|
||||
END IF
|
||||
|
||||
' Check for duplicates
|
||||
existing = FIND "contacts" WHERE email = contact_email
|
||||
IF COUNT(existing) > 0 THEN
|
||||
TALK "⚠️ A contact with this email already exists!"
|
||||
TALK "Would you like to update the existing contact instead?"
|
||||
' Handle duplicate logic
|
||||
END IF
|
||||
```
|
||||
|
||||
## Related Templates
|
||||
|
||||
- [Sales Pipeline](./template-sales-pipeline.md) - Track deals and opportunities
|
||||
- [Marketing Campaigns](./template-marketing.md) - Email campaigns and automation
|
||||
- [Customer Support](./template-helpdesk.md) - Support ticket management
|
||||
|
||||
## Support
|
||||
|
||||
For issues with this template:
|
||||
- Check the [troubleshooting guide](../chapter-13-community/README.md)
|
||||
- Open an issue on [GitHub](https://github.com/GeneralBots/BotServer/issues)
|
||||
- Join the [community chat](https://discord.gg/generalbots)
|
||||
505
docs/src/chapter-07-gbapp/docker-deployment.md
Normal file
505
docs/src/chapter-07-gbapp/docker-deployment.md
Normal file
|
|
@ -0,0 +1,505 @@
|
|||
# Docker Deployment
|
||||
|
||||
General Bots supports multiple Docker deployment strategies to fit your infrastructure needs. This guide covers all available options from single-container deployments to full orchestrated environments.
|
||||
|
||||
> **Note**: Docker support is currently **experimental**. While functional, some features may change in future releases.
|
||||
|
||||
## Deployment Options Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ DEPLOYMENT OPTIONS │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Option 1: All-in-One Container │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ botserver container │ │
|
||||
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
||||
│ │ │PostgreSQL│ │ MinIO │ │ Qdrant │ │ Vault │ │BotServer│ │ │
|
||||
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Option 2: Microservices (Separate Containers) │
|
||||
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
|
||||
│ │ PostgreSQL│ │ MinIO │ │ Qdrant │ │ Vault │ │
|
||||
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ └─────────────┴──────┬──────┴─────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────┴──────┐ │
|
||||
│ │ BotServer │ │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Option 1: All-in-One Container
|
||||
|
||||
The simplest deployment option runs everything inside a single container. This is ideal for:
|
||||
- Development environments
|
||||
- Small deployments
|
||||
- Quick testing
|
||||
- Resource-constrained environments
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name botserver \
|
||||
-p 8000:8000 \
|
||||
-p 9000:9000 \
|
||||
-v botserver-data:/opt/gbo/data \
|
||||
-e ADMIN_PASS=your-secure-password \
|
||||
pragmatismo/botserver:latest
|
||||
```
|
||||
|
||||
### Docker Compose (All-in-One)
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
botserver:
|
||||
image: pragmatismo/botserver:latest
|
||||
container_name: botserver
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000" # Main API
|
||||
- "9000:9000" # MinIO/Drive
|
||||
- "9001:9001" # MinIO Console
|
||||
volumes:
|
||||
- botserver-data:/opt/gbo/data
|
||||
- botserver-conf:/opt/gbo/conf
|
||||
- botserver-logs:/opt/gbo/logs
|
||||
- ./work:/opt/gbo/work # Your bot packages
|
||||
environment:
|
||||
- ADMIN_PASS=${ADMIN_PASS:-changeme}
|
||||
- DOMAIN=${DOMAIN:-localhost}
|
||||
- TZ=UTC
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
botserver-data:
|
||||
botserver-conf:
|
||||
botserver-logs:
|
||||
```
|
||||
|
||||
### Resource Requirements (All-in-One)
|
||||
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| CPU | 2 cores | 4+ cores |
|
||||
| RAM | 4GB | 8GB+ |
|
||||
| Storage | 20GB | 50GB+ |
|
||||
|
||||
## Option 2: Microservices Deployment
|
||||
|
||||
For production environments, we recommend running each component as a separate container. This provides:
|
||||
- Independent scaling
|
||||
- Better resource allocation
|
||||
- Easier maintenance and updates
|
||||
- High availability options
|
||||
|
||||
### Docker Compose (Microservices)
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# PostgreSQL - Primary Database
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: gb-postgres
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
environment:
|
||||
POSTGRES_USER: botserver
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
||||
POSTGRES_DB: botserver
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U botserver"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# MinIO - Object Storage / Drive
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: gb-minio
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${DRIVE_ACCESSKEY}
|
||||
MINIO_ROOT_PASSWORD: ${DRIVE_SECRET}
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
|
||||
interval: 30s
|
||||
timeout: 20s
|
||||
retries: 3
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# Qdrant - Vector Database
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: gb-qdrant
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "6333:6333"
|
||||
- "6334:6334"
|
||||
volumes:
|
||||
- qdrant-data:/qdrant/storage
|
||||
environment:
|
||||
QDRANT__SERVICE__GRPC_PORT: 6334
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# Vault - Secrets Management
|
||||
vault:
|
||||
image: hashicorp/vault:latest
|
||||
container_name: gb-vault
|
||||
restart: unless-stopped
|
||||
cap_add:
|
||||
- IPC_LOCK
|
||||
ports:
|
||||
- "8200:8200"
|
||||
volumes:
|
||||
- vault-data:/vault/data
|
||||
- ./vault-config:/vault/config
|
||||
environment:
|
||||
VAULT_ADDR: http://127.0.0.1:8200
|
||||
command: server -config=/vault/config/config.hcl
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# Redis - Caching (Optional but recommended)
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: gb-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# InfluxDB - Time Series (Optional - for analytics)
|
||||
influxdb:
|
||||
image: influxdb:2.7-alpine
|
||||
container_name: gb-influxdb
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8086:8086"
|
||||
volumes:
|
||||
- influxdb-data:/var/lib/influxdb2
|
||||
environment:
|
||||
DOCKER_INFLUXDB_INIT_MODE: setup
|
||||
DOCKER_INFLUXDB_INIT_USERNAME: admin
|
||||
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUX_PASSWORD}
|
||||
DOCKER_INFLUXDB_INIT_ORG: pragmatismo
|
||||
DOCKER_INFLUXDB_INIT_BUCKET: metrics
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
# BotServer - Main Application
|
||||
botserver:
|
||||
image: pragmatismo/botserver:latest
|
||||
container_name: gb-botserver
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
minio:
|
||||
condition: service_healthy
|
||||
qdrant:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./work:/opt/gbo/work
|
||||
- botserver-logs:/opt/gbo/logs
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://botserver:${DB_PASSWORD}@postgres:5432/botserver
|
||||
|
||||
# Drive/Storage
|
||||
DRIVE_URL: http://minio:9000
|
||||
DRIVE_ACCESSKEY: ${DRIVE_ACCESSKEY}
|
||||
DRIVE_SECRET: ${DRIVE_SECRET}
|
||||
|
||||
# Vector DB
|
||||
QDRANT_URL: http://qdrant:6333
|
||||
|
||||
# Vault
|
||||
VAULT_ADDR: http://vault:8200
|
||||
VAULT_TOKEN: ${VAULT_TOKEN}
|
||||
|
||||
# Redis
|
||||
REDIS_URL: redis://redis:6379
|
||||
|
||||
# InfluxDB
|
||||
INFLUX_URL: http://influxdb:8086
|
||||
INFLUX_TOKEN: ${INFLUX_TOKEN}
|
||||
INFLUX_ORG: pragmatismo
|
||||
INFLUX_BUCKET: metrics
|
||||
|
||||
# General
|
||||
ADMIN_PASS: ${ADMIN_PASS}
|
||||
DOMAIN: ${DOMAIN}
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
networks:
|
||||
- gb-network
|
||||
|
||||
networks:
|
||||
gb-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
minio-data:
|
||||
qdrant-data:
|
||||
vault-data:
|
||||
redis-data:
|
||||
influxdb-data:
|
||||
botserver-logs:
|
||||
```
|
||||
|
||||
### Environment File (.env)
|
||||
|
||||
Create a `.env` file with your configuration:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
DB_PASSWORD=your-secure-db-password
|
||||
|
||||
# Drive/MinIO
|
||||
DRIVE_ACCESSKEY=minioadmin
|
||||
DRIVE_SECRET=your-minio-secret
|
||||
|
||||
# Vault
|
||||
VAULT_TOKEN=your-vault-token
|
||||
|
||||
# InfluxDB
|
||||
INFLUX_PASSWORD=your-influx-password
|
||||
INFLUX_TOKEN=your-influx-token
|
||||
|
||||
# General
|
||||
ADMIN_PASS=your-admin-password
|
||||
DOMAIN=your-domain.com
|
||||
```
|
||||
|
||||
## Building Custom Images
|
||||
|
||||
### Dockerfile for BotServer
|
||||
|
||||
```dockerfile
|
||||
FROM rust:1.75-slim-bookworm AS builder
|
||||
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
RUN cargo build --release
|
||||
|
||||
FROM debian:bookworm-slim
|
||||
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ca-certificates \
|
||||
libssl3 \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /opt/gbo
|
||||
|
||||
COPY --from=builder /app/target/release/botserver /opt/gbo/bin/
|
||||
COPY --from=builder /app/templates /opt/gbo/templates/
|
||||
COPY --from=builder /app/ui /opt/gbo/ui/
|
||||
|
||||
ENV PATH="/opt/gbo/bin:${PATH}"
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["botserver"]
|
||||
```
|
||||
|
||||
### Multi-Architecture Build
|
||||
|
||||
```bash
|
||||
# Build for multiple architectures
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
-t pragmatismo/botserver:latest \
|
||||
--push .
|
||||
```
|
||||
|
||||
## Kubernetes Deployment
|
||||
|
||||
For large-scale deployments, use Kubernetes:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: botserver
|
||||
labels:
|
||||
app: botserver
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: botserver
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: botserver
|
||||
spec:
|
||||
containers:
|
||||
- name: botserver
|
||||
image: pragmatismo/botserver:latest
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "250m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "1000m"
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: botserver-secrets
|
||||
key: database-url
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: botserver
|
||||
spec:
|
||||
selector:
|
||||
app: botserver
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 8000
|
||||
type: LoadBalancer
|
||||
```
|
||||
|
||||
## Health Checks and Monitoring
|
||||
|
||||
All containers expose health endpoints:
|
||||
|
||||
| Service | Health Endpoint |
|
||||
|---------|-----------------|
|
||||
| BotServer | `GET /health` |
|
||||
| PostgreSQL | `pg_isready` command |
|
||||
| MinIO | `GET /minio/health/live` |
|
||||
| Qdrant | `GET /health` |
|
||||
| Vault | `GET /v1/sys/health` |
|
||||
| Redis | `redis-cli ping` |
|
||||
| InfluxDB | `GET /health` |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container Won't Start
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs gb-botserver
|
||||
|
||||
# Check if dependencies are running
|
||||
docker ps
|
||||
|
||||
# Verify network connectivity
|
||||
docker network inspect gb-network
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
```bash
|
||||
# Test database connection from botserver container
|
||||
docker exec -it gb-botserver psql $DATABASE_URL -c "SELECT 1"
|
||||
|
||||
# Check PostgreSQL logs
|
||||
docker logs gb-postgres
|
||||
```
|
||||
|
||||
### Storage Issues
|
||||
|
||||
```bash
|
||||
# Check MinIO status
|
||||
docker exec -it gb-minio mc admin info local
|
||||
|
||||
# Check volume mounts
|
||||
docker inspect gb-botserver | jq '.[0].Mounts'
|
||||
```
|
||||
|
||||
### Memory Issues
|
||||
|
||||
If containers are being killed due to OOM:
|
||||
|
||||
```yaml
|
||||
# Increase memory limits in docker-compose.yml
|
||||
services:
|
||||
botserver:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
reservations:
|
||||
memory: 2G
|
||||
```
|
||||
|
||||
## Migration from Non-Docker
|
||||
|
||||
To migrate an existing installation to Docker:
|
||||
|
||||
1. **Backup your data**:
|
||||
```bash
|
||||
pg_dump botserver > backup.sql
|
||||
mc cp --recursive /path/to/drive minio/backup/
|
||||
```
|
||||
|
||||
2. **Start Docker containers**
|
||||
|
||||
3. **Restore data**:
|
||||
```bash
|
||||
docker exec -i gb-postgres psql -U botserver < backup.sql
|
||||
docker exec -it gb-minio mc cp --recursive /backup minio/drive/
|
||||
```
|
||||
|
||||
4. **Copy bot packages** to the `work` volume
|
||||
|
||||
5. **Verify** everything works via the health endpoints
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Scaling and Load Balancing](./scaling.md)
|
||||
- [Infrastructure Design](./infrastructure.md)
|
||||
- [Observability](./observability.md)
|
||||
739
src/compliance/code_scanner.rs
Normal file
739
src/compliance/code_scanner.rs
Normal file
|
|
@ -0,0 +1,739 @@
|
|||
//! Code Scanner for BASIC Files
|
||||
//!
|
||||
//! Scans .bas files for security issues, fragile code patterns, and misconfigurations.
|
||||
//! Used by the /apicompliance endpoint to generate compliance reports.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use regex::Regex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use walkdir::WalkDir;
|
||||
|
||||
/// Issue severity levels
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum IssueSeverity {
|
||||
Info,
|
||||
Low,
|
||||
Medium,
|
||||
High,
|
||||
Critical,
|
||||
}
|
||||
|
||||
impl std::fmt::Display for IssueSeverity {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
IssueSeverity::Info => write!(f, "info"),
|
||||
IssueSeverity::Low => write!(f, "low"),
|
||||
IssueSeverity::Medium => write!(f, "medium"),
|
||||
IssueSeverity::High => write!(f, "high"),
|
||||
IssueSeverity::Critical => write!(f, "critical"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Issue types for categorization
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum IssueType {
|
||||
PasswordInConfig,
|
||||
HardcodedSecret,
|
||||
DeprecatedKeyword,
|
||||
FragileCode,
|
||||
ConfigurationIssue,
|
||||
UnderscoreInKeyword,
|
||||
MissingVault,
|
||||
InsecurePattern,
|
||||
DeprecatedIfInput,
|
||||
}
|
||||
|
||||
impl std::fmt::Display for IssueType {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
IssueType::PasswordInConfig => write!(f, "Password in Config"),
|
||||
IssueType::HardcodedSecret => write!(f, "Hardcoded Secret"),
|
||||
IssueType::DeprecatedKeyword => write!(f, "Deprecated Keyword"),
|
||||
IssueType::FragileCode => write!(f, "Fragile Code"),
|
||||
IssueType::ConfigurationIssue => write!(f, "Configuration Issue"),
|
||||
IssueType::UnderscoreInKeyword => write!(f, "Underscore in Keyword"),
|
||||
IssueType::MissingVault => write!(f, "Missing Vault Config"),
|
||||
IssueType::InsecurePattern => write!(f, "Insecure Pattern"),
|
||||
IssueType::DeprecatedIfInput => write!(f, "Deprecated IF...input Pattern"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A single compliance issue found in the code
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CodeIssue {
|
||||
pub id: String,
|
||||
pub severity: IssueSeverity,
|
||||
pub issue_type: IssueType,
|
||||
pub title: String,
|
||||
pub description: String,
|
||||
pub file_path: String,
|
||||
pub line_number: Option<usize>,
|
||||
pub code_snippet: Option<String>,
|
||||
pub remediation: String,
|
||||
pub category: String,
|
||||
pub detected_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
/// Scan result for a single bot
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct BotScanResult {
|
||||
pub bot_id: String,
|
||||
pub bot_name: String,
|
||||
pub scanned_at: DateTime<Utc>,
|
||||
pub files_scanned: usize,
|
||||
pub issues: Vec<CodeIssue>,
|
||||
pub stats: ScanStats,
|
||||
}
|
||||
|
||||
/// Statistics for a scan
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct ScanStats {
|
||||
pub critical: usize,
|
||||
pub high: usize,
|
||||
pub medium: usize,
|
||||
pub low: usize,
|
||||
pub info: usize,
|
||||
pub total: usize,
|
||||
}
|
||||
|
||||
impl ScanStats {
|
||||
pub fn add_issue(&mut self, severity: &IssueSeverity) {
|
||||
match severity {
|
||||
IssueSeverity::Critical => self.critical += 1,
|
||||
IssueSeverity::High => self.high += 1,
|
||||
IssueSeverity::Medium => self.medium += 1,
|
||||
IssueSeverity::Low => self.low += 1,
|
||||
IssueSeverity::Info => self.info += 1,
|
||||
}
|
||||
self.total += 1;
|
||||
}
|
||||
|
||||
pub fn merge(&mut self, other: &ScanStats) {
|
||||
self.critical += other.critical;
|
||||
self.high += other.high;
|
||||
self.medium += other.medium;
|
||||
self.low += other.low;
|
||||
self.info += other.info;
|
||||
self.total += other.total;
|
||||
}
|
||||
}
|
||||
|
||||
/// Full compliance scan result
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ComplianceScanResult {
|
||||
pub scanned_at: DateTime<Utc>,
|
||||
pub duration_ms: u64,
|
||||
pub bots_scanned: usize,
|
||||
pub total_files: usize,
|
||||
pub stats: ScanStats,
|
||||
pub bot_results: Vec<BotScanResult>,
|
||||
}
|
||||
|
||||
/// Pattern definition for scanning
|
||||
struct ScanPattern {
|
||||
regex: Regex,
|
||||
issue_type: IssueType,
|
||||
severity: IssueSeverity,
|
||||
title: String,
|
||||
description: String,
|
||||
remediation: String,
|
||||
category: String,
|
||||
}
|
||||
|
||||
/// Code scanner for BASIC files
|
||||
pub struct CodeScanner {
|
||||
patterns: Vec<ScanPattern>,
|
||||
base_path: PathBuf,
|
||||
}
|
||||
|
||||
impl CodeScanner {
|
||||
/// Create a new code scanner
|
||||
pub fn new(base_path: impl AsRef<Path>) -> Self {
|
||||
let patterns = Self::build_patterns();
|
||||
Self {
|
||||
patterns,
|
||||
base_path: base_path.as_ref().to_path_buf(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the list of patterns to scan for
|
||||
fn build_patterns() -> Vec<ScanPattern> {
|
||||
let mut patterns = Vec::new();
|
||||
|
||||
// Critical: Password/secret patterns in code
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)password\s*=\s*["'][^"']+["']"#).unwrap(),
|
||||
issue_type: IssueType::PasswordInConfig,
|
||||
severity: IssueSeverity::Critical,
|
||||
title: "Hardcoded Password".to_string(),
|
||||
description: "A password is hardcoded in the source code. This is a critical security risk.".to_string(),
|
||||
remediation: "Move the password to Vault using: vault_password = GET VAULT SECRET \"password_key\"".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)(api[_-]?key|apikey|secret[_-]?key|client[_-]?secret)\s*=\s*["'][^"']{8,}["']"#).unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::Critical,
|
||||
title: "Hardcoded API Key/Secret".to_string(),
|
||||
description: "An API key or secret is hardcoded in the source code.".to_string(),
|
||||
remediation: "Store secrets in Vault and retrieve with GET VAULT SECRET".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)token\s*=\s*["'][a-zA-Z0-9_\-]{20,}["']"#).unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::High,
|
||||
title: "Hardcoded Token".to_string(),
|
||||
description: "A token appears to be hardcoded in the source code.".to_string(),
|
||||
remediation: "Store tokens securely in Vault".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// High: Deprecated IF...input pattern
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)IF\s+.*\binput\b"#).unwrap(),
|
||||
issue_type: IssueType::DeprecatedIfInput,
|
||||
severity: IssueSeverity::Medium,
|
||||
title: "Deprecated IF...input Pattern".to_string(),
|
||||
description:
|
||||
"Using IF with raw input variable. Prefer HEAR AS for type-safe input handling."
|
||||
.to_string(),
|
||||
remediation: "Replace with: HEAR response AS STRING\nIF response = \"value\" THEN"
|
||||
.to_string(),
|
||||
category: "Code Quality".to_string(),
|
||||
});
|
||||
|
||||
// Medium: Underscore in keywords
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)\b(GET_BOT_MEMORY|SET_BOT_MEMORY|GET_USER_MEMORY|SET_USER_MEMORY|USE_KB|USE_TOOL|SEND_MAIL|CREATE_TASK)\b"#).unwrap(),
|
||||
issue_type: IssueType::UnderscoreInKeyword,
|
||||
severity: IssueSeverity::Low,
|
||||
title: "Underscore in Keyword".to_string(),
|
||||
description: "Keywords should use spaces instead of underscores for consistency.".to_string(),
|
||||
remediation: "Use spaces: GET BOT MEMORY, SET BOT MEMORY, etc.".to_string(),
|
||||
category: "Naming Convention".to_string(),
|
||||
});
|
||||
|
||||
// Medium: POST TO INSTAGRAM with inline credentials
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)POST\s+TO\s+INSTAGRAM\s+\w+\s*,\s*\w+"#).unwrap(),
|
||||
issue_type: IssueType::InsecurePattern,
|
||||
severity: IssueSeverity::High,
|
||||
title: "Instagram Credentials in Code".to_string(),
|
||||
description:
|
||||
"Instagram username/password passed directly. Use secure credential storage."
|
||||
.to_string(),
|
||||
remediation: "Store Instagram credentials in Vault and retrieve securely.".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// Low: Direct SQL in BASIC
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)(SELECT|INSERT|UPDATE|DELETE)\s+.*(FROM|INTO|SET)\s+"#)
|
||||
.unwrap(),
|
||||
issue_type: IssueType::FragileCode,
|
||||
severity: IssueSeverity::Medium,
|
||||
title: "Raw SQL Query".to_string(),
|
||||
description: "Raw SQL queries in BASIC code may be vulnerable to injection."
|
||||
.to_string(),
|
||||
remediation:
|
||||
"Use parameterized queries or the built-in data operations (SAVE, GET, etc.)"
|
||||
.to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// Info: Eval or dynamic execution
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)\bEVAL\s*\("#).unwrap(),
|
||||
issue_type: IssueType::FragileCode,
|
||||
severity: IssueSeverity::High,
|
||||
title: "Dynamic Code Execution".to_string(),
|
||||
description: "EVAL can execute arbitrary code and is a security risk.".to_string(),
|
||||
remediation: "Avoid EVAL. Use structured control flow instead.".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// Check for base64 encoded secrets (potential obfuscated credentials)
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(
|
||||
r#"(?i)(password|secret|key|token)\s*=\s*["'][A-Za-z0-9+/=]{40,}["']"#,
|
||||
)
|
||||
.unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::High,
|
||||
title: "Potential Encoded Secret".to_string(),
|
||||
description: "A base64-like string is assigned to a sensitive variable.".to_string(),
|
||||
remediation: "Remove encoded secrets from code. Use Vault for secret management."
|
||||
.to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// AWS credentials pattern
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)(AKIA[0-9A-Z]{16})"#).unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::Critical,
|
||||
title: "AWS Access Key".to_string(),
|
||||
description: "An AWS access key ID is hardcoded in the source code.".to_string(),
|
||||
remediation: "Remove immediately and rotate the key. Use IAM roles or Vault."
|
||||
.to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// Private key patterns
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"-----BEGIN\s+(RSA\s+)?PRIVATE\s+KEY-----"#).unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::Critical,
|
||||
title: "Private Key in Code".to_string(),
|
||||
description: "A private key is embedded in the source code.".to_string(),
|
||||
remediation: "Remove private key immediately. Store in secure key management system."
|
||||
.to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
// Connection strings with credentials
|
||||
patterns.push(ScanPattern {
|
||||
regex: Regex::new(r#"(?i)(postgres|mysql|mongodb|redis)://[^:]+:[^@]+@"#).unwrap(),
|
||||
issue_type: IssueType::HardcodedSecret,
|
||||
severity: IssueSeverity::Critical,
|
||||
title: "Database Credentials in Connection String".to_string(),
|
||||
description: "Database connection string contains embedded credentials.".to_string(),
|
||||
remediation: "Use environment variables or Vault for database credentials.".to_string(),
|
||||
category: "Security".to_string(),
|
||||
});
|
||||
|
||||
patterns
|
||||
}
|
||||
|
||||
/// Scan all bots in the base path
|
||||
pub async fn scan_all(
|
||||
&self,
|
||||
) -> Result<ComplianceScanResult, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let start_time = std::time::Instant::now();
|
||||
let mut bot_results = Vec::new();
|
||||
let mut total_stats = ScanStats::default();
|
||||
let mut total_files = 0;
|
||||
|
||||
// Find all .gbai directories (bot packages)
|
||||
let templates_path = self.base_path.join("templates");
|
||||
let work_path = self.base_path.join("work");
|
||||
|
||||
let mut bot_paths = Vec::new();
|
||||
|
||||
// Scan templates directory
|
||||
if templates_path.exists() {
|
||||
for entry in WalkDir::new(&templates_path).max_depth(3) {
|
||||
if let Ok(entry) = entry {
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
let name = path.file_name().unwrap_or_default().to_string_lossy();
|
||||
if name.ends_with(".gbai") || name.ends_with(".gbdialog") {
|
||||
bot_paths.push(path.to_path_buf());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scan work directory (deployed bots)
|
||||
if work_path.exists() {
|
||||
for entry in WalkDir::new(&work_path).max_depth(3) {
|
||||
if let Ok(entry) = entry {
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
let name = path.file_name().unwrap_or_default().to_string_lossy();
|
||||
if name.ends_with(".gbai") || name.ends_with(".gbdialog") {
|
||||
bot_paths.push(path.to_path_buf());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scan each bot
|
||||
for bot_path in &bot_paths {
|
||||
let result = self.scan_bot(bot_path).await?;
|
||||
total_files += result.files_scanned;
|
||||
total_stats.merge(&result.stats);
|
||||
bot_results.push(result);
|
||||
}
|
||||
|
||||
let duration_ms = start_time.elapsed().as_millis() as u64;
|
||||
|
||||
Ok(ComplianceScanResult {
|
||||
scanned_at: Utc::now(),
|
||||
duration_ms,
|
||||
bots_scanned: bot_results.len(),
|
||||
total_files,
|
||||
stats: total_stats,
|
||||
bot_results,
|
||||
})
|
||||
}
|
||||
|
||||
/// Scan a specific bot directory
|
||||
pub async fn scan_bot(
|
||||
&self,
|
||||
bot_path: &Path,
|
||||
) -> Result<BotScanResult, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let bot_name = bot_path
|
||||
.file_name()
|
||||
.unwrap_or_default()
|
||||
.to_string_lossy()
|
||||
.to_string();
|
||||
|
||||
let bot_id = format!("{:x}", md5::compute(&bot_name));
|
||||
|
||||
let mut issues = Vec::new();
|
||||
let mut stats = ScanStats::default();
|
||||
let mut files_scanned = 0;
|
||||
|
||||
// Find all .bas files in the bot directory
|
||||
for entry in WalkDir::new(bot_path) {
|
||||
if let Ok(entry) = entry {
|
||||
let path = entry.path();
|
||||
if path.is_file() {
|
||||
let extension = path.extension().unwrap_or_default().to_string_lossy();
|
||||
if extension == "bas" || extension == "csv" {
|
||||
files_scanned += 1;
|
||||
let file_issues = self.scan_file(path).await?;
|
||||
for issue in file_issues {
|
||||
stats.add_issue(&issue.severity);
|
||||
issues.push(issue);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check for missing Vault configuration
|
||||
let config_path = bot_path.join("config.csv");
|
||||
if config_path.exists() {
|
||||
let vault_configured = self.check_vault_config(&config_path).await?;
|
||||
if !vault_configured {
|
||||
let issue = CodeIssue {
|
||||
id: uuid::Uuid::new_v4().to_string(),
|
||||
severity: IssueSeverity::Info,
|
||||
issue_type: IssueType::MissingVault,
|
||||
title: "Vault Not Configured".to_string(),
|
||||
description: "This bot is not configured to use Vault for secrets management.".to_string(),
|
||||
file_path: config_path.to_string_lossy().to_string(),
|
||||
line_number: None,
|
||||
code_snippet: None,
|
||||
remediation: "Add VAULT_ADDR and VAULT_TOKEN to configuration for secure secret management.".to_string(),
|
||||
category: "Configuration".to_string(),
|
||||
detected_at: Utc::now(),
|
||||
};
|
||||
stats.add_issue(&issue.severity);
|
||||
issues.push(issue);
|
||||
}
|
||||
}
|
||||
|
||||
// Sort issues by severity (critical first)
|
||||
issues.sort_by(|a, b| b.severity.cmp(&a.severity));
|
||||
|
||||
Ok(BotScanResult {
|
||||
bot_id,
|
||||
bot_name,
|
||||
scanned_at: Utc::now(),
|
||||
files_scanned,
|
||||
issues,
|
||||
stats,
|
||||
})
|
||||
}
|
||||
|
||||
/// Scan a single file for issues
|
||||
async fn scan_file(
|
||||
&self,
|
||||
file_path: &Path,
|
||||
) -> Result<Vec<CodeIssue>, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let content = tokio::fs::read_to_string(file_path).await?;
|
||||
let mut issues = Vec::new();
|
||||
|
||||
let relative_path = file_path
|
||||
.strip_prefix(&self.base_path)
|
||||
.unwrap_or(file_path)
|
||||
.to_string_lossy()
|
||||
.to_string();
|
||||
|
||||
for (line_number, line) in content.lines().enumerate() {
|
||||
let line_num = line_number + 1;
|
||||
|
||||
// Skip comments
|
||||
let trimmed = line.trim();
|
||||
if trimmed.starts_with("REM") || trimmed.starts_with("'") || trimmed.starts_with("//") {
|
||||
continue;
|
||||
}
|
||||
|
||||
for pattern in &self.patterns {
|
||||
if pattern.regex.is_match(line) {
|
||||
// Redact sensitive information in the snippet
|
||||
let snippet = self.redact_sensitive(line);
|
||||
|
||||
let issue = CodeIssue {
|
||||
id: uuid::Uuid::new_v4().to_string(),
|
||||
severity: pattern.severity.clone(),
|
||||
issue_type: pattern.issue_type.clone(),
|
||||
title: pattern.title.clone(),
|
||||
description: pattern.description.clone(),
|
||||
file_path: relative_path.clone(),
|
||||
line_number: Some(line_num),
|
||||
code_snippet: Some(snippet),
|
||||
remediation: pattern.remediation.clone(),
|
||||
category: pattern.category.clone(),
|
||||
detected_at: Utc::now(),
|
||||
};
|
||||
issues.push(issue);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(issues)
|
||||
}
|
||||
|
||||
/// Redact sensitive information in code snippets
|
||||
fn redact_sensitive(&self, line: &str) -> String {
|
||||
let mut result = line.to_string();
|
||||
|
||||
// Redact quoted strings that look like secrets
|
||||
let secret_pattern = Regex::new(r#"(["'])[^"']{8,}(["'])"#).unwrap();
|
||||
result = secret_pattern
|
||||
.replace_all(&result, "$1***REDACTED***$2")
|
||||
.to_string();
|
||||
|
||||
// Redact AWS keys
|
||||
let aws_pattern = Regex::new(r#"AKIA[0-9A-Z]{16}"#).unwrap();
|
||||
result = aws_pattern
|
||||
.replace_all(&result, "AKIA***REDACTED***")
|
||||
.to_string();
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Check if Vault is configured for a bot
|
||||
async fn check_vault_config(
|
||||
&self,
|
||||
config_path: &Path,
|
||||
) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let content = tokio::fs::read_to_string(config_path).await?;
|
||||
|
||||
// Check for Vault-related configuration
|
||||
let has_vault = content.to_lowercase().contains("vault_addr")
|
||||
|| content.to_lowercase().contains("vault_token")
|
||||
|| content.to_lowercase().contains("vault-");
|
||||
|
||||
Ok(has_vault)
|
||||
}
|
||||
|
||||
/// Scan specific bots by ID
|
||||
pub async fn scan_bots(
|
||||
&self,
|
||||
bot_ids: &[String],
|
||||
) -> Result<ComplianceScanResult, Box<dyn std::error::Error + Send + Sync>> {
|
||||
if bot_ids.is_empty() || bot_ids.contains(&"all".to_string()) {
|
||||
return self.scan_all().await;
|
||||
}
|
||||
|
||||
// For specific bots, we'd need to look them up by ID
|
||||
// For now, scan all and filter
|
||||
let mut full_result = self.scan_all().await?;
|
||||
full_result
|
||||
.bot_results
|
||||
.retain(|r| bot_ids.contains(&r.bot_id) || bot_ids.contains(&r.bot_name));
|
||||
|
||||
// Recalculate stats
|
||||
let mut new_stats = ScanStats::default();
|
||||
for bot in &full_result.bot_results {
|
||||
new_stats.merge(&bot.stats);
|
||||
}
|
||||
full_result.stats = new_stats;
|
||||
full_result.bots_scanned = full_result.bot_results.len();
|
||||
|
||||
Ok(full_result)
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate a compliance report in various formats
|
||||
pub struct ComplianceReporter;
|
||||
|
||||
impl ComplianceReporter {
|
||||
/// Generate HTML report
|
||||
pub fn to_html(result: &ComplianceScanResult) -> String {
|
||||
let mut html = String::new();
|
||||
|
||||
html.push_str("<!DOCTYPE html><html><head><title>Compliance Report</title>");
|
||||
html.push_str("<style>body{font-family:system-ui;margin:20px;}table{border-collapse:collapse;width:100%;}th,td{border:1px solid #ddd;padding:8px;text-align:left;}.critical{color:#dc2626;}.high{color:#ea580c;}.medium{color:#d97706;}.low{color:#65a30d;}.info{color:#0891b2;}</style>");
|
||||
html.push_str("</head><body>");
|
||||
|
||||
html.push_str(&format!("<h1>Compliance Scan Report</h1>"));
|
||||
html.push_str(&format!("<p>Scanned at: {}</p>", result.scanned_at));
|
||||
html.push_str(&format!("<p>Duration: {}ms</p>", result.duration_ms));
|
||||
html.push_str(&format!("<p>Bots scanned: {}</p>", result.bots_scanned));
|
||||
html.push_str(&format!("<p>Files scanned: {}</p>", result.total_files));
|
||||
|
||||
html.push_str("<h2>Summary</h2>");
|
||||
html.push_str(&format!(
|
||||
"<p class='critical'>Critical: {}</p>",
|
||||
result.stats.critical
|
||||
));
|
||||
html.push_str(&format!("<p class='high'>High: {}</p>", result.stats.high));
|
||||
html.push_str(&format!(
|
||||
"<p class='medium'>Medium: {}</p>",
|
||||
result.stats.medium
|
||||
));
|
||||
html.push_str(&format!("<p class='low'>Low: {}</p>", result.stats.low));
|
||||
html.push_str(&format!("<p class='info'>Info: {}</p>", result.stats.info));
|
||||
|
||||
html.push_str("<h2>Issues</h2>");
|
||||
html.push_str("<table><tr><th>Severity</th><th>Type</th><th>File</th><th>Line</th><th>Description</th></tr>");
|
||||
|
||||
for bot in &result.bot_results {
|
||||
for issue in &bot.issues {
|
||||
html.push_str(&format!(
|
||||
"<tr><td class='{}'>{}</td><td>{}</td><td>{}</td><td>{}</td><td>{}</td></tr>",
|
||||
issue.severity.to_string(),
|
||||
issue.severity,
|
||||
issue.issue_type,
|
||||
issue.file_path,
|
||||
issue
|
||||
.line_number
|
||||
.map(|n| n.to_string())
|
||||
.unwrap_or("-".to_string()),
|
||||
issue.description
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
html.push_str("</table></body></html>");
|
||||
html
|
||||
}
|
||||
|
||||
/// Generate JSON report
|
||||
pub fn to_json(result: &ComplianceScanResult) -> Result<String, serde_json::Error> {
|
||||
serde_json::to_string_pretty(result)
|
||||
}
|
||||
|
||||
/// Generate CSV report
|
||||
pub fn to_csv(result: &ComplianceScanResult) -> String {
|
||||
let mut csv = String::new();
|
||||
csv.push_str("Severity,Type,Category,File,Line,Title,Description,Remediation\n");
|
||||
|
||||
for bot in &result.bot_results {
|
||||
for issue in &bot.issues {
|
||||
csv.push_str(&format!(
|
||||
"{},{},{},{},{},{},{},{}\n",
|
||||
issue.severity,
|
||||
issue.issue_type,
|
||||
issue.category,
|
||||
issue.file_path,
|
||||
issue
|
||||
.line_number
|
||||
.map(|n| n.to_string())
|
||||
.unwrap_or("-".to_string()),
|
||||
escape_csv(&issue.title),
|
||||
escape_csv(&issue.description),
|
||||
escape_csv(&issue.remediation)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
csv
|
||||
}
|
||||
}
|
||||
|
||||
/// Escape a string for CSV output
|
||||
fn escape_csv(s: &str) -> String {
|
||||
if s.contains(',') || s.contains('"') || s.contains('\n') {
|
||||
format!("\"{}\"", s.replace('"', "\"\""))
|
||||
} else {
|
||||
s.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_pattern_matching() {
|
||||
let scanner = CodeScanner::new("/tmp/test");
|
||||
|
||||
// Test password detection
|
||||
let password_pattern = scanner
|
||||
.patterns
|
||||
.iter()
|
||||
.find(|p| matches!(p.issue_type, IssueType::PasswordInConfig))
|
||||
.unwrap();
|
||||
assert!(password_pattern.regex.is_match(r#"password = "secret123""#));
|
||||
assert!(password_pattern.regex.is_match(r#"PASSWORD = 'mypass'"#));
|
||||
|
||||
// Test underscore keyword detection
|
||||
let underscore_pattern = scanner
|
||||
.patterns
|
||||
.iter()
|
||||
.find(|p| matches!(p.issue_type, IssueType::UnderscoreInKeyword))
|
||||
.unwrap();
|
||||
assert!(underscore_pattern.regex.is_match("GET_BOT_MEMORY"));
|
||||
assert!(underscore_pattern.regex.is_match("SET_USER_MEMORY"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_severity_ordering() {
|
||||
assert!(IssueSeverity::Critical > IssueSeverity::High);
|
||||
assert!(IssueSeverity::High > IssueSeverity::Medium);
|
||||
assert!(IssueSeverity::Medium > IssueSeverity::Low);
|
||||
assert!(IssueSeverity::Low > IssueSeverity::Info);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_stats_merge() {
|
||||
let mut stats1 = ScanStats {
|
||||
critical: 1,
|
||||
high: 2,
|
||||
medium: 3,
|
||||
low: 4,
|
||||
info: 5,
|
||||
total: 15,
|
||||
};
|
||||
|
||||
let stats2 = ScanStats {
|
||||
critical: 1,
|
||||
high: 1,
|
||||
medium: 1,
|
||||
low: 1,
|
||||
info: 1,
|
||||
total: 5,
|
||||
};
|
||||
|
||||
stats1.merge(&stats2);
|
||||
|
||||
assert_eq!(stats1.critical, 2);
|
||||
assert_eq!(stats1.high, 3);
|
||||
assert_eq!(stats1.total, 20);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_csv_escape() {
|
||||
assert_eq!(escape_csv("simple"), "simple");
|
||||
assert_eq!(escape_csv("with,comma"), "\"with,comma\"");
|
||||
assert_eq!(escape_csv("with\"quote"), "\"with\"\"quote\"");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_redact_sensitive() {
|
||||
let scanner = CodeScanner::new("/tmp/test");
|
||||
|
||||
let line = r#"password = "supersecretpassword123""#;
|
||||
let redacted = scanner.redact_sensitive(line);
|
||||
assert!(redacted.contains("***REDACTED***"));
|
||||
assert!(!redacted.contains("supersecretpassword123"));
|
||||
}
|
||||
}
|
||||
|
|
@ -2,6 +2,12 @@
|
|||
//!
|
||||
//! This module provides automated compliance monitoring, audit logging,
|
||||
//! risk assessment, and security policy enforcement capabilities.
|
||||
//!
|
||||
//! Includes code scanning for BASIC files to detect:
|
||||
//! - Hardcoded passwords and secrets
|
||||
//! - Deprecated keywords and patterns
|
||||
//! - Configuration issues
|
||||
//! - Security vulnerabilities
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
|
@ -9,10 +15,17 @@ use std::collections::HashMap;
|
|||
|
||||
pub mod access_review;
|
||||
pub mod audit;
|
||||
pub mod code_scanner;
|
||||
pub mod policy_checker;
|
||||
pub mod risk_assessment;
|
||||
pub mod training_tracker;
|
||||
|
||||
// Re-export commonly used types from code_scanner
|
||||
pub use code_scanner::{
|
||||
CodeIssue, CodeScanner, ComplianceReporter, ComplianceScanResult, IssueSeverity, IssueType,
|
||||
ScanStats,
|
||||
};
|
||||
|
||||
/// Compliance framework types
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub enum ComplianceFramework {
|
||||
|
|
|
|||
1465
ui/suite/designer.html
Normal file
1465
ui/suite/designer.html
Normal file
File diff suppressed because it is too large
Load diff
1460
ui/suite/drive/index.html
Normal file
1460
ui/suite/drive/index.html
Normal file
File diff suppressed because it is too large
Load diff
1653
ui/suite/sources/index.html
Normal file
1653
ui/suite/sources/index.html
Normal file
File diff suppressed because it is too large
Load diff
1038
ui/suite/tools/compliance.html
Normal file
1038
ui/suite/tools/compliance.html
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue