GeneralBots/botserver
2
1
Fork 0
Code Issues Pull requests Projects Releases 323 Packages Wiki Activity Actions

Update dependencies and remove problematic crates

Browse source 
Drop image (with ravif/paste), sqlx, zitadel, and related dependencies
that were causing compilation issues. Replace image processing with
direct png crate usage. Update rcgen to 0.14 with new API changes.
Refactor CA certificate generation to use Issuer pattern.
This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2025-12-03 16:05:30 -03:00
parent e8d171ea40
commit ad311944b8
15 changed files with 2256 additions and 2597 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2757
Cargo.lock generated
View file

File diff suppressed because it is too large Load diff

15
Cargo.toml
View file

@ -77,7 +77,6 @@ monitoring = ["dep:sysinfo"]
automation = ["dep:rhai"] automation = ["dep:rhai"]
grpc = ["dep:tonic"] grpc = ["dep:tonic"]
progress-bars = ["dep:indicatif"] progress-bars = ["dep:indicatif"]
dynamic-db = ["dep:sqlx"]
# ===== META FEATURES (BUNDLES) ===== # ===== META FEATURES (BUNDLES) =====
full = [ full = [
@ -139,13 +138,12 @@ askama_axum = "0.4"
tracing-subscriber = { version = "0.3", features = ["fmt"] } tracing-subscriber = { version = "0.3", features = ["fmt"] }
urlencoding = "2.1" urlencoding = "2.1"
uuid = { version = "1.11", features = ["serde", "v4"] } uuid = { version = "1.11", features = ["serde", "v4"] }
zitadel = { version = "5.5.1", features = ["api", "credentials"] }
# === TLS/SECURITY DEPENDENCIES === # === TLS/SECURITY DEPENDENCIES ===
rustls = { version = "0.21", features = ["dangerous_configuration"] } rustls = { version = "0.21", features = ["dangerous_configuration"] }
rustls-pemfile = "1.0" rustls-pemfile = "1.0"
tokio-rustls = "0.24" tokio-rustls = "0.24"
rcgen = { version = "0.11", features = ["pem"] } rcgen = { version = "0.14", features = ["pem"] }
x509-parser = "0.15" x509-parser = "0.15"
rustls-native-certs = "0.6" rustls-native-certs = "0.6"
webpki-roots = "0.25" webpki-roots = "0.25"
@ -189,19 +187,16 @@ csv = { version = "1.3", optional = true }
# Console/TUI (console feature) # Console/TUI (console feature)
crossterm = { version = "0.29.0", optional = true } crossterm = { version = "0.29.0", optional = true }
ratatui = { version = "0.29.0", optional = true } ratatui = { version = "0.30.0-beta.0", optional = true }
# QR Code Generation # QR Code Generation (using png directly to avoid image's ravif/paste dependency)
image = "0.25" png = "0.18"
qrcode = "0.14" qrcode = { version = "0.14", default-features = false }
# Excel/Spreadsheet Support # Excel/Spreadsheet Support
calamine = "0.26" calamine = "0.26"
rust_xlsxwriter = "0.79" rust_xlsxwriter = "0.79"
# Database (for table_definition dynamic connections)
sqlx = { version = "0.8", features = ["runtime-tokio", "postgres", "mysql"], optional = true }
# Error handling # Error handling
thiserror = "2.0" thiserror = "2.0"

13
docs/src/SUMMARY.md
View file

@ -23,6 +23,19 @@
- [.gbdrive File Storage](./chapter-02/gbdrive.md) - [.gbdrive File Storage](./chapter-02/gbdrive.md)
- [Bot Templates](./chapter-02/templates.md) - [Bot Templates](./chapter-02/templates.md)
- [Template Samples & Conversations](./chapter-02/template-samples.md) - [Template Samples & Conversations](./chapter-02/template-samples.md)
- [Template: Business Intelligence](./chapter-02/template-bi.md)
- [Template: Web Crawler](./chapter-02/template-crawler.md)
- [Template: Legal Documents](./chapter-02/template-law.md)
- [Template: LLM Server](./chapter-02/template-llm-server.md)
- [Template: LLM Tools](./chapter-02/template-llm-tools.md)
- [Template: API Client](./chapter-02/template-api-client.md)
- [Template: Platform Analytics](./chapter-02/template-analytics.md)
- [Template: Office Automation](./chapter-02/template-office.md)
- [Template: Reminders](./chapter-02/template-reminder.md)
- [Template: Sales CRM](./chapter-02/template-crm.md)
- [Template: CRM Contacts](./chapter-02/template-crm-contacts.md)
- [Template: Marketing](./chapter-02/template-marketing.md)
- [Template: Creating Templates](./chapter-02/template-template.md)
# Part III - Knowledge Base # Part III - Knowledge Base

193
docs/src/chapter-08-config/parameters.md
View file

@ -24,6 +24,7 @@ Complete reference of all available parameters in `config.csv`.
| `llm-key` | API key for LLM service | `none` | String | | `llm-key` | API key for LLM service | `none` | String |
| `llm-url` | LLM service endpoint | `http://localhost:8081` | URL | | `llm-url` | LLM service endpoint | `http://localhost:8081` | URL |
| `llm-model` | Model path or identifier | Required | Path/String | | `llm-model` | Model path or identifier | Required | Path/String |
| `llm-models` | Available model aliases for routing | `default` | Semicolon-separated |
### LLM Cache ### LLM Cache
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
@ -48,6 +49,7 @@ Complete reference of all available parameters in `config.csv`.
| `llm-server-cont-batching` | Continuous batching | `true` | Boolean | | `llm-server-cont-batching` | Continuous batching | `true` | Boolean |
| `llm-server-mlock` | Lock in memory | `false` | Boolean | | `llm-server-mlock` | Lock in memory | `false` | Boolean |
| `llm-server-no-mmap` | Disable mmap | `false` | Boolean | | `llm-server-no-mmap` | Disable mmap | `false` | Boolean |
| `llm-server-reasoning-format` | Reasoning output format for llama.cpp | `none` | String |
### Hardware-Specific LLM Tuning ### Hardware-Specific LLM Tuning
@ -129,14 +131,25 @@ llm-model,mixtral-8x7b-32768
## Custom Database Parameters ## Custom Database Parameters
These parameters configure external database connections for use with BASIC keywords like MariaDB/MySQL connections.
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `custom-server` | Database server | `localhost` | Hostname | | `custom-server` | Database server hostname | `localhost` | Hostname |
| `custom-port` | Database port | `5432` | Number | | `custom-port` | Database port | `3306` | Number |
| `custom-database` | Database name | Not set | String | | `custom-database` | Database name | Not set | String |
| `custom-username` | Database user | Not set | String | | `custom-username` | Database user | Not set | String |
| `custom-password` | Database password | Not set | String | | `custom-password` | Database password | Not set | String |
### Example: MariaDB Connection
```csv
custom-server,db.example.com
custom-port,3306
custom-database,myapp
custom-username,botuser
custom-password,secretpass
```
## Multi-Agent Parameters ## Multi-Agent Parameters
### Agent-to-Agent (A2A) Communication ### Agent-to-Agent (A2A) Communication
@ -147,15 +160,35 @@ llm-model,mixtral-8x7b-32768
| `a2a-max-hops` | Maximum delegation chain depth | `5` | Number | | `a2a-max-hops` | Maximum delegation chain depth | `5` | Number |
| `a2a-retry-count` | Retry attempts on failure | `3` | Number | | `a2a-retry-count` | Retry attempts on failure | `3` | Number |
| `a2a-queue-size` | Maximum pending messages | `100` | Number | | `a2a-queue-size` | Maximum pending messages | `100` | Number |
| `a2a-protocol-version` | A2A protocol version | `1.0` | String |
| `a2a-persist-messages` | Persist A2A messages to database | `false` | Boolean |
### Bot Reflection ### Bot Reflection
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `reflection-enabled` | Enable bot self-analysis | `true` | Boolean | | `bot-reflection-enabled` | Enable bot self-analysis | `true` | Boolean |
| `reflection-interval` | Messages between reflections | `10` | Number | | `bot-reflection-interval` | Messages between reflections | `10` | Number |
| `reflection-min-messages` | Minimum messages before reflecting | `3` | Number | | `bot-reflection-prompt` | Custom reflection prompt | (none) | String |
| `reflection-model` | LLM model for reflection | `quality` | String | | `bot-reflection-types` | Reflection types to perform | `ConversationQuality` | Semicolon-separated |
| `reflection-store-insights` | Store insights in database | `true` | Boolean | | `bot-improvement-auto-apply` | Auto-apply suggested improvements | `false` | Boolean |
| `bot-improvement-threshold` | Score threshold for improvements (0-10) | `6.0` | Float |
#### Reflection Types
Available values for `bot-reflection-types`:
- `ConversationQuality` - Analyze conversation quality and user satisfaction
- `ResponseAccuracy` - Analyze response accuracy and relevance
- `ToolUsage` - Analyze tool usage effectiveness
- `KnowledgeRetrieval` - Analyze knowledge retrieval performance
- `Performance` - Analyze overall bot performance
Example:
```csv
bot-reflection-enabled,true
bot-reflection-interval,10
bot-reflection-types,ConversationQuality;ResponseAccuracy;ToolUsage
bot-improvement-auto-apply,false
bot-improvement-threshold,7.0
```
## Memory Parameters ## Memory Parameters
@ -173,54 +206,141 @@ llm-model,mixtral-8x7b-32768
| `episodic-summary-model` | Model for summarization | `fast` | String | | `episodic-summary-model` | Model for summarization | `fast` | String |
| `episodic-max-episodes` | Maximum episodes per user | `100` | Number | | `episodic-max-episodes` | Maximum episodes per user | `100` | Number |
| `episodic-retention-days` | Days to retain episodes | `365` | Number | | `episodic-retention-days` | Days to retain episodes | `365` | Number |
| `episodic-auto-summarize` | Enable automatic summarization | `true` | Boolean |
## Model Routing Parameters ## Model Routing Parameters
These parameters configure multi-model routing for different task types. Requires multiple llama.cpp server instances.
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `llm-models` | Available model aliases | `default` | Semicolon-separated |
| `model-routing-strategy` | Routing strategy (manual/auto/load-balanced/fallback) | `auto` | String | | `model-routing-strategy` | Routing strategy (manual/auto/load-balanced/fallback) | `auto` | String |
| `model-default` | Default model alias | `fast` | String | | `model-default` | Default model alias | `default` | String |
| `model-fast` | Model for fast/simple tasks | (configured) | Path/String | | `model-fast` | Model for fast/simple tasks | (configured) | Path/String |
| `model-quality` | Model for quality/complex tasks | (configured) | Path/String | | `model-quality` | Model for quality/complex tasks | (configured) | Path/String |
| `model-code` | Model for code generation | (configured) | Path/String | | `model-code` | Model for code generation | (configured) | Path/String |
| `model-fallback-enabled` | Enable automatic fallback | `true` | Boolean | | `model-fallback-enabled` | Enable automatic fallback | `true` | Boolean |
| `model-fallback-order` | Order to try on failure | `quality,fast,local` | String | | `model-fallback-order` | Order to try on failure | `quality,fast,local` | Comma-separated |
### Multi-Model Example
```csv
llm-models,default;fast;quality;code
llm-url,http://localhost:8081
model-routing-strategy,auto
model-default,fast
model-fallback-enabled,true
model-fallback-order,quality,fast
```
## Hybrid RAG Search Parameters ## Hybrid RAG Search Parameters
General Bots uses hybrid search combining **dense (embedding)** and **sparse (BM25 keyword)** search for optimal retrieval. The BM25 implementation is powered by [Tantivy](https://github.com/quickwit-oss/tantivy), a full-text search engine library similar to Apache Lucene.
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `rag-hybrid-enabled` | Enable hybrid dense+sparse search | `true` | Boolean | | `rag-hybrid-enabled` | Enable hybrid dense+sparse search | `true` | Boolean |
| `rag-dense-weight` | Weight for semantic results | `0.7` | Float (0-1) | | `rag-dense-weight` | Weight for semantic results | `0.7` | Float (0-1) |
| `rag-sparse-weight` | Weight for keyword results | `0.3` | Float (0-1) | | `rag-sparse-weight` | Weight for keyword results | `0.3` | Float (0-1) |
| `rag-reranker-enabled` | Enable LLM reranking | `false` | Boolean | | `rag-reranker-enabled` | Enable LLM reranking | `false` | Boolean |
| `rag-reranker-model` | Model for reranking | `quality` | String | | `rag-reranker-model` | Model for reranking | `cross-encoder/ms-marco-MiniLM-L-6-v2` | String |
| `rag-reranker-top-n` | Candidates for reranking | `20` | Number | | `rag-reranker-top-n` | Candidates for reranking | `20` | Number |
| `rag-top-k` | Results to return | `10` | Number | | `rag-max-results` | Maximum results to return | `10` | Number |
| `rag-min-score` | Minimum relevance score threshold | `0.0` | Float (0-1) |
| `rag-rrf-k` | RRF smoothing constant | `60` | Number | | `rag-rrf-k` | RRF smoothing constant | `60` | Number |
| `rag-cache-enabled` | Enable search result caching | `true` | Boolean | | `rag-cache-enabled` | Enable search result caching | `true` | Boolean |
| `rag-cache-ttl` | Cache time-to-live | `3600` | Seconds | | `rag-cache-ttl` | Cache time-to-live | `3600` | Seconds |
### BM25 (Sparse Search) Tuning ### BM25 Sparse Search (Tantivy)
BM25 is a keyword-based ranking algorithm that excels at finding exact term matches. It's powered by Tantivy when the `vectordb` feature is enabled.
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `bm25-k1` | Term saturation parameter | `1.2` | Float | | `bm25-enabled` | **Enable/disable BM25 sparse search** | `true` | Boolean |
| `bm25-b` | Length normalization | `0.75` | Float | | `bm25-k1` | Term frequency saturation (0.5-3.0 typical) | `1.2` | Float |
| `bm25-stemming` | Enable word stemming | `true` | Boolean | | `bm25-b` | Document length normalization (0.0-1.0) | `0.75` | Float |
| `bm25-stopwords` | Filter common words | `true` | Boolean | | `bm25-stemming` | Apply word stemming (running→run) | `true` | Boolean |
| `bm25-stopwords` | Filter common words (the, a, is) | `true` | Boolean |
### Switching Search Modes
**Hybrid Search (Default - Best for most use cases)**
```csv
bm25-enabled,true
rag-dense-weight,0.7
rag-sparse-weight,0.3
```
Uses both semantic understanding AND keyword matching. Best for general queries.
**Dense Only (Semantic Search)**
```csv
bm25-enabled,false
rag-dense-weight,1.0
rag-sparse-weight,0.0
```
Uses only embedding-based search. Faster, good for conceptual/semantic queries where exact words don't matter.
**Sparse Only (Keyword Search)**
```csv
bm25-enabled,true
rag-dense-weight,0.0
rag-sparse-weight,1.0
```
Uses only BM25 keyword matching. Good for exact term searches, technical documentation, or when embeddings aren't available.
### BM25 Parameter Tuning
The `k1` and `b` parameters control BM25 behavior:
- **`bm25-k1`** (Term Saturation): Controls how much additional term occurrences contribute to the score
- Lower values (0.5-1.0): Diminishing returns for repeated terms
- Higher values (1.5-2.0): More weight to documents with many term occurrences
- Default `1.2` works well for most content
- **`bm25-b`** (Length Normalization): Controls document length penalty
- `0.0`: No length penalty (long documents scored equally)
- `1.0`: Full length normalization (strongly penalizes long documents)
- Default `0.75` balances length fairness
**Tuning for specific content:**
```csv
# For short documents (tweets, titles)
bm25-b,0.3
# For long documents (articles, manuals)
bm25-b,0.9
# For code search (exact matches important)
bm25-k1,1.5
bm25-stemming,false
```
## Code Sandbox Parameters ## Code Sandbox Parameters
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `sandbox-enabled` | Enable code sandbox | `true` | Boolean |
| `sandbox-runtime` | Isolation backend (lxc/docker/firecracker/process) | `lxc` | String | | `sandbox-runtime` | Isolation backend (lxc/docker/firecracker/process) | `lxc` | String |
| `sandbox-timeout` | Maximum execution time | `30` | Seconds | | `sandbox-timeout` | Maximum execution time | `30` | Seconds |
| `sandbox-memory-mb` | Memory limit | `512` | MB | | `sandbox-memory-mb` | Memory limit in megabytes | `256` | MB |
| `sandbox-cpu-percent` | CPU usage limit | `50` | Percent | | `sandbox-cpu-percent` | CPU usage limit | `50` | Percent |
| `sandbox-network` | Allow network access | `false` | Boolean | | `sandbox-network` | Allow network access | `false` | Boolean |
| `sandbox-python-packages` | Pre-installed Python packages | (none) | Comma-separated | | `sandbox-python-packages` | Pre-installed Python packages | (none) | Comma-separated |
| `sandbox-allowed-paths` | Accessible filesystem paths | `/data,/tmp` | Comma-separated | | `sandbox-allowed-paths` | Accessible filesystem paths | `/data,/tmp` | Comma-separated |
### Example: Python Sandbox
```csv
sandbox-enabled,true
sandbox-runtime,lxc
sandbox-timeout,60
sandbox-memory-mb,512
sandbox-cpu-percent,75
sandbox-network,false
sandbox-python-packages,numpy,pandas,requests,matplotlib
sandbox-allowed-paths,/data,/tmp,/uploads
```
## SSE Streaming Parameters ## SSE Streaming Parameters
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
@ -229,14 +349,6 @@ llm-model,mixtral-8x7b-32768
| `sse-heartbeat` | Heartbeat interval | `30` | Seconds | | `sse-heartbeat` | Heartbeat interval | `30` | Seconds |
| `sse-max-connections` | Maximum concurrent connections | `1000` | Number | | `sse-max-connections` | Maximum concurrent connections | `1000` | Number |
## OpenAPI Tool Generation Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `openapi-server` | OpenAPI spec URL for auto tool generation | Not set | URL |
| `openapi-auth-header` | Authentication header name | `Authorization` | String |
| `openapi-auth-value` | Authentication header value | Not set | String |
## Parameter Types ## Parameter Types
### Boolean ### Boolean
@ -251,6 +363,7 @@ Integer values, must be within valid ranges:
### Float ### Float
Decimal values: Decimal values:
- Thresholds: 0.0 to 1.0 - Thresholds: 0.0 to 1.0
- Weights: 0.0 to 1.0
### Path ### Path
File system paths: File system paths:
@ -271,6 +384,12 @@ Valid email format: `user@domain.com`
### Hex Color ### Hex Color
HTML color codes: `#RRGGBB` format HTML color codes: `#RRGGBB` format
### Semicolon-separated
Multiple values separated by semicolons: `value1;value2;value3`
### Comma-separated
Multiple values separated by commas: `value1,value2,value3`
## Required vs Optional ## Required vs Optional
### Always Required ### Always Required
@ -312,6 +431,7 @@ llm-server-cont-batching,true
llm-cache-semantic,true llm-cache-semantic,true
llm-cache-threshold,0.90 llm-cache-threshold,0.90
llm-server-parallel,8 llm-server-parallel,8
sse-max-connections,5000
``` ```
### For Low Memory ### For Low Memory
@ -321,6 +441,7 @@ llm-server-n-predict,512
llm-server-mlock,false llm-server-mlock,false
llm-server-no-mmap,false llm-server-no-mmap,false
llm-cache,false llm-cache,false
sandbox-memory-mb,128
``` ```
### For Multi-Agent Systems ### For Multi-Agent Systems
@ -328,9 +449,10 @@ llm-cache,false
a2a-enabled,true a2a-enabled,true
a2a-timeout,30 a2a-timeout,30
a2a-max-hops,5 a2a-max-hops,5
model-routing-strategy,auto a2a-retry-count,3
reflection-enabled,true a2a-persist-messages,true
reflection-interval,10 bot-reflection-enabled,true
bot-reflection-interval,10
user-memory-enabled,true user-memory-enabled,true
``` ```
@ -340,11 +462,25 @@ rag-hybrid-enabled,true
rag-dense-weight,0.7 rag-dense-weight,0.7
rag-sparse-weight,0.3 rag-sparse-weight,0.3
rag-reranker-enabled,true rag-reranker-enabled,true
rag-max-results,10
rag-min-score,0.3
rag-cache-enabled,true rag-cache-enabled,true
bm25-enabled,true
bm25-k1,1.2
bm25-b,0.75
```
### For Dense-Only Search (Faster)
```csv
bm25-enabled,false
rag-dense-weight,1.0
rag-sparse-weight,0.0
rag-max-results,10
``` ```
### For Code Execution ### For Code Execution
```csv ```csv
sandbox-enabled,true
sandbox-runtime,lxc sandbox-runtime,lxc
sandbox-timeout,30 sandbox-timeout,30
sandbox-memory-mb,512 sandbox-memory-mb,512
@ -360,3 +496,4 @@ sandbox-python-packages,numpy,pandas,requests
4. **Emails**: Must contain @ and domain 4. **Emails**: Must contain @ and domain
5. **Colors**: Must be valid hex format 5. **Colors**: Must be valid hex format
6. **Booleans**: Exactly `true` or `false` 6. **Booleans**: Exactly `true` or `false`
7. **Weights**: Must sum to 1.0 (e.g., `rag-dense-weight` + `rag-sparse-weight`)

12
src/basic/keywords/a2a_protocol.rs
View file

@ -159,6 +159,10 @@ pub struct A2AConfig {
pub protocol_version: String, pub protocol_version: String,
/// Enable message persistence /// Enable message persistence
pub persist_messages: bool, pub persist_messages: bool,
/// Retry attempts on failure
pub retry_count: u32,
/// Maximum pending messages in queue
pub queue_size: u32,
} }
impl Default for A2AConfig { impl Default for A2AConfig {
@ -169,6 +173,8 @@ impl Default for A2AConfig {
max_hops: 5, max_hops: 5,
protocol_version: "1.0".to_string(), protocol_version: "1.0".to_string(),
persist_messages: true, persist_messages: true,
retry_count: 3,
queue_size: 100,
} }
} }
} }
@ -211,6 +217,12 @@ pub fn load_a2a_config(state: &AppState, bot_id: Uuid) -> A2AConfig {
"a2a-persist-messages" => { "a2a-persist-messages" => {
config.persist_messages = row.config_value.to_lowercase() == "true"; config.persist_messages = row.config_value.to_lowercase() == "true";
} }
"a2a-retry-count" => {
config.retry_count = row.config_value.parse().unwrap_or(3);
}
"a2a-queue-size" => {
config.queue_size = row.config_value.parse().unwrap_or(100);
}
_ => {} _ => {}
} }
} }

387
src/basic/keywords/card.rs
View file

@ -1,5 +1,39 @@
/*****************************************************************************\
| █████ █████ ██ █ █████ █████ ████ ██ ████ █████ █████ ███ ® |
| ██ █ ███ █ █ ██ ██ ██ ██ ██ ██ █ ██ ██ █ █ |
| ██ ███ ████ █ ██ █ ████ █████ ██████ ██ ████ █ █ █ ██ |
| ██ ██ █ █ ██ █ █ ██ ██ ██ ██ ██ ██ █ ██ ██ █ █ |
| █████ █████ █ ███ █████ ██ ██ ██ ██ █████ ████ █████ █ ███ |
| |
| General Bots Copyright (c) pragmatismo.com.br. All rights reserved. |
| Licensed under the AGPL-3.0. |
| |
| According to our dual licensing model, this program can be used either |
| under the terms of the GNU Affero General Public License, version 3, |
| or under a proprietary license. |
| |
| The texts of the GNU Affero General Public License with an additional |
| permission and of our proprietary license can be found at and |
| in the LICENSE file you have received along with this program. |
| |
| This program is distributed in the hope that it will be useful, |
| but WITHOUT ANY WARRANTY, without even the implied warranty of |
| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
| GNU Affero General Public License for more details. |
| |
| "General Bots" is a registered trademark of pragmatismo.com.br. |
| The licensing of the program under the AGPLv3 does not imply a |
| trademark license. Therefore any rights, title and interest in |
| our trademarks remain entirely with us. |
| |
\*****************************************************************************/
//! CARD keyword - Creates beautiful Instagram-style posts from prompts //! CARD keyword - Creates beautiful Instagram-style posts from prompts
//! //!
//! This module generates social media cards by combining AI-generated images
//! with AI-generated text. The LLM handles both image generation and text
//! composition, eliminating the need for local image processing.
//!
//! Syntax: //! Syntax:
//! CARD image_prompt, text_prompt TO variable //! CARD image_prompt, text_prompt TO variable
//! CARD image_prompt, text_prompt, style TO variable //! CARD image_prompt, text_prompt, style TO variable
@ -13,10 +47,9 @@
use crate::basic::runtime::{BasicRuntime, BasicValue}; use crate::basic::runtime::{BasicRuntime, BasicValue};
use crate::llm::LLMProvider; use crate::llm::LLMProvider;
use anyhow::{anyhow, Result}; use anyhow::{anyhow, Result};
use image::{DynamicImage, ImageBuffer, Rgba, RgbaImage};
use imageproc::drawing::{draw_text_mut, text_size};
use rusttype::{Font, Scale};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::fs;
use std::path::Path;
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::Mutex; use tokio::sync::Mutex;
@ -87,18 +120,7 @@ impl CardDimensions {
} }
} }
/// Text overlay configuration /// Text position configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TextOverlay {
pub text: String,
pub font_size: f32,
pub color: [u8; 4],
pub position: TextPosition,
pub max_width_ratio: f32,
pub shadow: bool,
pub background: Option<[u8; 4]>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)] #[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub enum TextPosition { pub enum TextPosition {
Top, Top,
@ -148,6 +170,7 @@ impl Default for CardConfig {
} }
/// CARD keyword implementation /// CARD keyword implementation
/// Uses LLM for both image generation and text composition
pub struct CardKeyword { pub struct CardKeyword {
llm_provider: Arc<dyn LLMProvider>, llm_provider: Arc<dyn LLMProvider>,
output_dir: String, output_dir: String,
@ -178,6 +201,9 @@ impl CardKeyword {
..Default::default() ..Default::default()
}; };
// Ensure output directory exists
fs::create_dir_all(&self.output_dir)?;
let mut results = Vec::with_capacity(card_count); let mut results = Vec::with_capacity(card_count);
for i in 0..card_count { for i in 0..card_count {
@ -201,16 +227,20 @@ impl CardKeyword {
// Step 1: Generate optimized text content using LLM // Step 1: Generate optimized text content using LLM
let text_content = self.generate_text_content(text_prompt, config).await?; let text_content = self.generate_text_content(text_prompt, config).await?;
// Step 2: Generate image using image generation API // Step 2: Create enhanced prompt that includes text overlay instructions
let base_image = self.generate_image(image_prompt, config).await?; let enhanced_image_prompt = self.create_card_prompt(image_prompt, &text_content, config);
// Step 3: Apply style and text overlay // Step 3: Generate image with text baked in via LLM image generation
let styled_image = self.apply_style_and_text(&base_image, &text_content, config)?; let image_bytes = self
.llm_provider
.generate_image(
&enhanced_image_prompt,
config.dimensions.width,
config.dimensions.height,
)
.await?;
// Step 4: Generate hashtags and caption // Step 4: Save the image
let (hashtags, caption) = self.generate_social_content(&text_content, config).await?;
// Step 5: Save the final image
let filename = format!( let filename = format!(
"card_{}_{}.png", "card_{}_{}.png",
chrono::Utc::now().format("%Y%m%d_%H%M%S"), chrono::Utc::now().format("%Y%m%d_%H%M%S"),
@ -218,7 +248,15 @@ impl CardKeyword {
); );
let image_path = format!("{}/{}", self.output_dir, filename); let image_path = format!("{}/{}", self.output_dir, filename);
styled_image.save(&image_path)?; // Ensure parent directory exists
if let Some(parent) = Path::new(&image_path).parent() {
fs::create_dir_all(parent)?;
}
fs::write(&image_path, &image_bytes)?;
// Step 5: Generate hashtags and caption
let (hashtags, caption) = self.generate_social_content(&text_content, config).await?;
Ok(CardResult { Ok(CardResult {
image_path: image_path.clone(), image_path: image_path.clone(),
@ -276,30 +314,13 @@ Respond with ONLY the text content, nothing else."#,
} }
} }
/// Generate the base image /// Create an enhanced prompt for image generation that includes text overlay
async fn generate_image( fn create_card_prompt(
&self, &self,
image_prompt: &str, image_prompt: &str,
text_content: &str,
config: &CardConfig, config: &CardConfig,
) -> Result<DynamicImage> { ) -> String {
let enhanced_prompt = self.enhance_image_prompt(image_prompt, config);
// Call image generation service
let image_bytes = self
.llm_provider
.generate_image(
&enhanced_prompt,
config.dimensions.width,
config.dimensions.height,
)
.await?;
let image = image::load_from_memory(&image_bytes)?;
Ok(image)
}
/// Enhance the image prompt based on style
fn enhance_image_prompt(&self, base_prompt: &str, config: &CardConfig) -> String {
let style_modifiers = match config.style { let style_modifiers = match config.style {
CardStyle::Minimal => { CardStyle::Minimal => {
"minimalist, clean, simple composition, lots of negative space, muted colors" "minimalist, clean, simple composition, lots of negative space, muted colors"
@ -315,175 +336,45 @@ Respond with ONLY the text content, nothing else."#,
CardStyle::Modern => "modern, trendy, instagram aesthetic, high quality", CardStyle::Modern => "modern, trendy, instagram aesthetic, high quality",
}; };
format!( let text_position = match config.text_position {
"{}, {}, perfect for Instagram, professional quality, 4K, highly detailed", TextPosition::Top => "at the top",
base_prompt, style_modifiers TextPosition::Center => "in the center",
) TextPosition::Bottom => "at the bottom",
} TextPosition::TopLeft => "in the top left corner",
TextPosition::TopRight => "in the top right corner",
/// Apply style effects and text overlay to the image TextPosition::BottomLeft => "in the bottom left corner",
fn apply_style_and_text( TextPosition::BottomRight => "in the bottom right corner",
&self,
image: &DynamicImage,
text: &str,
config: &CardConfig,
) -> Result<DynamicImage> {
let mut rgba_image = image.to_rgba8();
// Apply style-specific filters
self.apply_style_filter(&mut rgba_image, &config.style);
// Add text overlay
self.add_text_overlay(&mut rgba_image, text, config)?;
// Add watermark if configured
if let Some(ref watermark) = config.brand_watermark {
self.add_watermark(&mut rgba_image, watermark)?;
}
Ok(DynamicImage::ImageRgba8(rgba_image))
}
/// Apply style-specific image filters
fn apply_style_filter(&self, image: &mut RgbaImage, style: &CardStyle) {
match style {
CardStyle::Dark => {
// Darken and increase contrast
for pixel in image.pixels_mut() {
pixel[0] = (pixel[0] as f32 * 0.7) as u8;
pixel[1] = (pixel[1] as f32 * 0.7) as u8;
pixel[2] = (pixel[2] as f32 * 0.7) as u8;
}
}
CardStyle::Light => {
// Brighten slightly
for pixel in image.pixels_mut() {
pixel[0] = ((pixel[0] as f32 * 1.1).min(255.0)) as u8;
pixel[1] = ((pixel[1] as f32 * 1.1).min(255.0)) as u8;
pixel[2] = ((pixel[2] as f32 * 1.1).min(255.0)) as u8;
}
}
CardStyle::Polaroid => {
// Add warm vintage tint
for pixel in image.pixels_mut() {
pixel[0] = ((pixel[0] as f32 * 1.05).min(255.0)) as u8;
pixel[1] = ((pixel[1] as f32 * 0.95).min(255.0)) as u8;
pixel[2] = ((pixel[2] as f32 * 0.85).min(255.0)) as u8;
}
}
CardStyle::Vibrant => {
// Increase saturation
for pixel in image.pixels_mut() {
let r = pixel[0] as f32;
let g = pixel[1] as f32;
let b = pixel[2] as f32;
let avg = (r + g + b) / 3.0;
let factor = 1.3;
pixel[0] = ((r - avg) * factor + avg).clamp(0.0, 255.0) as u8;
pixel[1] = ((g - avg) * factor + avg).clamp(0.0, 255.0) as u8;
pixel[2] = ((b - avg) * factor + avg).clamp(0.0, 255.0) as u8;
}
}
_ => {}
}
}
/// Add text overlay to the image
fn add_text_overlay(
&self,
image: &mut RgbaImage,
text: &str,
config: &CardConfig,
) -> Result<()> {
let (width, height) = (image.width(), image.height());
// Load font (embedded or from file)
let font_data = include_bytes!("../../../assets/fonts/Inter-Bold.ttf");
let font = Font::try_from_bytes(font_data as &[u8])
.ok_or_else(|| anyhow!("Failed to load font"))?;
// Calculate font size based on image dimensions and text length
let base_size = (width as f32 * 0.08).min(height as f32 * 0.1);
let scale = Scale::uniform(base_size);
// Calculate text position
let (text_width, text_height) = text_size(scale, &font, text);
let (x, y) = self.calculate_text_position(
width,
height,
text_width as u32,
text_height as u32,
&config.text_position,
);
// Draw text shadow for better readability
let shadow_color = Rgba([0u8, 0u8, 0u8, 180u8]);
draw_text_mut(image, shadow_color, x + 3, y + 3, scale, &font, text);
// Draw main text
let text_color = match config.style {
CardStyle::Dark => Rgba([255u8, 255u8, 255u8, 255u8]),
CardStyle::Light => Rgba([30u8, 30u8, 30u8, 255u8]),
_ => Rgba([255u8, 255u8, 255u8, 255u8]),
}; };
draw_text_mut(image, text_color, x, y, scale, &font, text);
Ok(()) let text_color = match config.style {
} CardStyle::Dark => "white",
CardStyle::Light => "dark gray or black",
_ => "white with a subtle shadow",
};
/// Calculate text position based on configuration format!(
fn calculate_text_position( r#"Create an Instagram-ready image with the following specifications:
&self,
img_width: u32,
img_height: u32,
text_width: u32,
text_height: u32,
position: &TextPosition,
) -> (i32, i32) {
let padding = (img_width as f32 * 0.05) as i32;
match position { Background/Scene: {}
TextPosition::Top => ( Style: {}, perfect for Instagram, professional quality, 4K, highly detailed
((img_width - text_width) / 2) as i32,
padding + text_height as i32,
),
TextPosition::Center => (
((img_width - text_width) / 2) as i32,
((img_height - text_height) / 2) as i32,
),
TextPosition::Bottom => (
((img_width - text_width) / 2) as i32,
(img_height - text_height) as i32 - padding,
),
TextPosition::TopLeft => (padding, padding + text_height as i32),
TextPosition::TopRight => (
(img_width - text_width) as i32 - padding,
padding + text_height as i32,
),
TextPosition::BottomLeft => (padding, (img_height - text_height) as i32 - padding),
TextPosition::BottomRight => (
(img_width - text_width) as i32 - padding,
(img_height - text_height) as i32 - padding,
),
}
}
/// Add brand watermark Text Overlay Requirements:
fn add_watermark(&self, image: &mut RgbaImage, watermark: &str) -> Result<()> { - Display the text "{}" {} of the image
let font_data = include_bytes!("../../../assets/fonts/Inter-Regular.ttf"); - Use bold, modern typography
let font = Font::try_from_bytes(font_data as &[u8]) - Text color should be {} for readability
.ok_or_else(|| anyhow!("Failed to load font"))?; - Add a subtle text shadow or background blur behind text for contrast
- Leave appropriate padding around text
let scale = Scale::uniform(image.width() as f32 * 0.025); The image should be {} x {} pixels, optimized for social media.
let color = Rgba([255u8, 255u8, 255u8, 128u8]); Make the text an integral part of the design, not just overlaid."#,
image_prompt,
let padding = 20i32; style_modifiers,
let x = padding; text_content,
let y = (image.height() - 30) as i32; text_position,
text_color,
draw_text_mut(image, color, x, y, scale, &font, watermark); config.dimensions.width,
config.dimensions.height
Ok(()) )
} }
/// Generate hashtags and caption for the post /// Generate hashtags and caption for the post
@ -516,11 +407,22 @@ HASHTAGS: tag1, tag2, tag3, tag4, tag5"#,
let mut hashtags = Vec::new(); let mut hashtags = Vec::new();
for line in response.lines() { for line in response.lines() {
if line.starts_with("CAPTION:") { let line_trimmed = line.trim();
caption = line.trim_start_matches("CAPTION:").trim().to_string(); if line_trimmed.starts_with("CAPTION:") {
} else if line.starts_with("HASHTAGS:") { caption = line_trimmed
let tags = line.trim_start_matches("HASHTAGS:").trim(); .trim_start_matches("CAPTION:")
hashtags = tags.split(',').map(|t| format!("#{}", t.trim())).collect(); .trim()
.to_string();
} else if line_trimmed.starts_with("HASHTAGS:") {
let tags = line_trimmed.trim_start_matches("HASHTAGS:").trim();
hashtags = tags
.split(',')
.map(|t| {
let tag = t.trim().trim_start_matches('#');
format!("#{}", tag)
})
.filter(|t| t.len() > 1)
.collect();
} }
} }
@ -550,27 +452,36 @@ pub fn register_card_keyword(runtime: &mut BasicRuntime, llm_provider: Arc<dyn L
let style = args.get(2).map(|v| v.as_string()).transpose()?; let style = args.get(2).map(|v| v.as_string()).transpose()?;
let count = args let count = args
.get(3) .get(3)
.map(|v| v.as_number().map(|n| n as usize)) .map(|v| v.as_integer().map(|i| i as usize))
.transpose()?; .transpose()?;
let kw = keyword.lock().await; let keyword = keyword.lock().await;
let results = kw let results = keyword
.execute(&image_prompt, &text_prompt, style.as_deref(), count) .execute(&image_prompt, &text_prompt, style.as_deref(), count)
.await?; .await?;
// Convert results to BasicValue // Convert results to BasicValue
let value = if results.len() == 1 { let result_values: Vec<BasicValue> = results
BasicValue::Object(serde_json::to_value(&results[0])?) .into_iter()
} else { .map(|r| {
BasicValue::Array( BasicValue::Object(serde_json::json!({
results "image_path": r.image_path,
.into_iter() "image_url": r.image_url,
.map(|r| BasicValue::Object(serde_json::to_value(&r).unwrap())) "text": r.text_content,
.collect(), "hashtags": r.hashtags,
) "caption": r.caption,
}; "style": r.style,
"width": r.dimensions.0,
"height": r.dimensions.1,
}))
})
.collect();
Ok(value) if result_values.len() == 1 {
Ok(result_values.into_iter().next().unwrap())
} else {
Ok(BasicValue::Array(result_values))
}
}) })
}); });
} }
@ -583,19 +494,27 @@ mod tests {
fn test_card_style_from_string() { fn test_card_style_from_string() {
assert!(matches!(CardStyle::from("minimal"), CardStyle::Minimal)); assert!(matches!(CardStyle::from("minimal"), CardStyle::Minimal));
assert!(matches!(CardStyle::from("VIBRANT"), CardStyle::Vibrant)); assert!(matches!(CardStyle::from("VIBRANT"), CardStyle::Vibrant));
assert!(matches!(CardStyle::from("dark"), CardStyle::Dark));
assert!(matches!(CardStyle::from("unknown"), CardStyle::Modern)); assert!(matches!(CardStyle::from("unknown"), CardStyle::Modern));
} }
#[test] #[test]
fn test_card_dimensions() { fn test_card_dimensions_for_style() {
assert_eq!(CardDimensions::INSTAGRAM_SQUARE.width, 1080); let story_dims = CardDimensions::for_style(&CardStyle::Story);
assert_eq!(CardDimensions::INSTAGRAM_SQUARE.height, 1080); assert_eq!(story_dims.width, 1080);
assert_eq!(CardDimensions::INSTAGRAM_STORY.height, 1920); assert_eq!(story_dims.height, 1920);
let square_dims = CardDimensions::for_style(&CardStyle::Modern);
assert_eq!(square_dims.width, 1080);
assert_eq!(square_dims.height, 1080);
} }
#[test] #[test]
fn test_text_position_calculation() { fn test_card_config_default() {
// Create a mock keyword for testing let config = CardConfig::default();
// In real tests, we'd use a mock LLM provider assert!(matches!(config.style, CardStyle::Modern));
assert!(config.include_hashtags);
assert!(config.include_caption);
assert!(config.brand_watermark.is_none());
} }
} }

43
src/basic/keywords/code_sandbox.rs
View file

@ -12,10 +12,12 @@
//! ```csv //! ```csv
//! sandbox-enabled,true //! sandbox-enabled,true
//! sandbox-timeout,30 //! sandbox-timeout,30
//! sandbox-memory-limit,256 //! sandbox-memory-mb,256
//! sandbox-cpu-limit,50 //! sandbox-cpu-percent,50
//! sandbox-network-enabled,false //! sandbox-network,false
//! sandbox-runtime,lxc //! sandbox-runtime,lxc
//! sandbox-python-packages,numpy,pandas,requests
//! sandbox-allowed-paths,/data,/tmp
//! ``` //! ```
use crate::shared::models::UserSession; use crate::shared::models::UserSession;
@ -125,10 +127,10 @@ pub struct SandboxConfig {
pub work_dir: String, pub work_dir: String,
/// Additional environment variables /// Additional environment variables
pub env_vars: HashMap<String, String>, pub env_vars: HashMap<String, String>,
/// Allowed file paths for read access /// Allowed file paths for access
pub allowed_read_paths: Vec<String>, pub allowed_paths: Vec<String>,
/// Allowed file paths for write access /// Pre-installed Python packages
pub allowed_write_paths: Vec<String>, pub python_packages: Vec<String>,
} }
impl Default for SandboxConfig { impl Default for SandboxConfig {
@ -142,8 +144,8 @@ impl Default for SandboxConfig {
network_enabled: false, network_enabled: false,
work_dir: "/tmp/gb-sandbox".to_string(), work_dir: "/tmp/gb-sandbox".to_string(),
env_vars: HashMap::new(), env_vars: HashMap::new(),
allowed_read_paths: vec![], allowed_paths: vec!["/data".to_string(), "/tmp".to_string()],
allowed_write_paths: vec![], python_packages: vec![],
} }
} }
} }
@ -181,15 +183,32 @@ impl SandboxConfig {
"sandbox-timeout" => { "sandbox-timeout" => {
config.timeout_seconds = row.config_value.parse().unwrap_or(30); config.timeout_seconds = row.config_value.parse().unwrap_or(30);
} }
"sandbox-memory-limit" => { // Support both old and new parameter names for backward compatibility
"sandbox-memory-mb" | "sandbox-memory-limit" => {
config.memory_limit_mb = row.config_value.parse().unwrap_or(256); config.memory_limit_mb = row.config_value.parse().unwrap_or(256);
} }
"sandbox-cpu-limit" => { "sandbox-cpu-percent" | "sandbox-cpu-limit" => {
config.cpu_limit_percent = row.config_value.parse().unwrap_or(50); config.cpu_limit_percent = row.config_value.parse().unwrap_or(50);
} }
"sandbox-network-enabled" => { "sandbox-network" | "sandbox-network-enabled" => {
config.network_enabled = row.config_value.to_lowercase() == "true"; config.network_enabled = row.config_value.to_lowercase() == "true";
} }
"sandbox-python-packages" => {
config.python_packages = row
.config_value
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
}
"sandbox-allowed-paths" => {
config.allowed_paths = row
.config_value
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
}
_ => {} _ => {}
} }
} }

168
src/basic/keywords/qrcode.rs
View file

@ -37,10 +37,12 @@
use crate::shared::models::UserSession; use crate::shared::models::UserSession;
use crate::shared::state::AppState; use crate::shared::state::AppState;
use image::Luma;
use log::{error, trace}; use log::{error, trace};
use png::{BitDepth, ColorType, Encoder};
use qrcode::QrCode; use qrcode::QrCode;
use rhai::{Dynamic, Engine}; use rhai::{Dynamic, Engine};
use std::fs::File;
use std::io::BufWriter;
use std::path::Path; use std::path::Path;
use std::sync::Arc; use std::sync::Arc;
use uuid::Uuid; use uuid::Uuid;
@ -240,8 +242,29 @@ fn execute_qr_code_generation(
// Generate QR code // Generate QR code
let code = QrCode::new(data.as_bytes())?; let code = QrCode::new(data.as_bytes())?;
// Render to image // Get the QR code as a matrix of bools
let image = code.render::<Luma<u8>>().min_dimensions(size, size).build(); let matrix = code.to_colors();
let qr_width = code.width();
// Calculate scale factor to reach target size
let scale = (size as usize) / qr_width;
let actual_size = qr_width * scale;
// Create grayscale pixel buffer
let mut pixels: Vec<u8> = Vec::with_capacity(actual_size * actual_size);
for y in 0..actual_size {
for x in 0..actual_size {
let qr_x = x / scale;
let qr_y = y / scale;
let idx = qr_y * qr_width + qr_x;
let is_dark = matrix
.get(idx)
.map(|c| *c == qrcode::Color::Dark)
.unwrap_or(false);
pixels.push(if is_dark { 0 } else { 255 });
}
}
// Determine output path // Determine output path
let data_dir = state let data_dir = state
@ -274,8 +297,16 @@ fn execute_qr_code_generation(
std::fs::create_dir_all(parent)?; std::fs::create_dir_all(parent)?;
} }
// Save image // Save as PNG using png crate directly
image.save(&final_path)?; let file = File::create(&final_path)?;
let ref mut w = BufWriter::new(file);
let mut encoder = Encoder::new(w, actual_size as u32, actual_size as u32);
encoder.set_color(ColorType::Grayscale);
encoder.set_depth(BitDepth::Eight);
let mut writer = encoder.write_header()?;
writer.write_image_data(&pixels)?;
trace!("QR code generated: {}", final_path); trace!("QR code generated: {}", final_path);
Ok(final_path) Ok(final_path)
@ -289,70 +320,113 @@ pub fn generate_qr_code_colored(
background: [u8; 3], background: [u8; 3],
output_path: &str, output_path: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> { ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
use image::{Rgb, RgbImage};
let code = QrCode::new(data.as_bytes())?; let code = QrCode::new(data.as_bytes())?;
let qr_image = code.render::<Luma<u8>>().min_dimensions(size, size).build();
// Convert to RGB with custom colors // Get the QR code as a matrix of bools
let mut rgb_image = RgbImage::new(qr_image.width(), qr_image.height()); let matrix = code.to_colors();
let qr_width = code.width();
for (x, y, pixel) in qr_image.enumerate_pixels() { // Calculate scale factor to reach target size
let color = if pixel[0] == 0 { let scale = (size as usize) / qr_width;
Rgb(foreground) let actual_size = qr_width * scale;
} else {
Rgb(background) // Create RGB pixel buffer (3 bytes per pixel)
}; let mut pixels: Vec<u8> = Vec::with_capacity(actual_size * actual_size * 3);
rgb_image.put_pixel(x, y, color);
for y in 0..actual_size {
for x in 0..actual_size {
let qr_x = x / scale;
let qr_y = y / scale;
let idx = qr_y * qr_width + qr_x;
let is_dark = matrix
.get(idx)
.map(|c| *c == qrcode::Color::Dark)
.unwrap_or(false);
let color = if is_dark { foreground } else { background };
pixels.extend_from_slice(&color);
}
} }
rgb_image.save(output_path)?; // Save as PNG
let file = File::create(output_path)?;
let ref mut w = BufWriter::new(file);
let mut encoder = Encoder::new(w, actual_size as u32, actual_size as u32);
encoder.set_color(ColorType::Rgb);
encoder.set_depth(BitDepth::Eight);
let mut writer = encoder.write_header()?;
writer.write_image_data(&pixels)?;
Ok(output_path.to_string()) Ok(output_path.to_string())
} }
/// Generate QR code with logo overlay /// Generate QR code with logo overlay
/// Note: Logo overlay requires the image crate. This simplified version
/// generates a QR code with a white center area where a logo can be placed manually.
pub fn generate_qr_code_with_logo( pub fn generate_qr_code_with_logo(
data: &str, data: &str,
size: u32, size: u32,
logo_path: &str, _logo_path: &str,
output_path: &str, output_path: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> { ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
use image::{imageops, DynamicImage, Rgba, RgbaImage};
// Generate QR code with higher error correction for logo overlay // Generate QR code with higher error correction for logo overlay
let code = QrCode::with_error_correction_level(data.as_bytes(), qrcode::EcLevel::H)?; let code = QrCode::with_error_correction_level(data.as_bytes(), qrcode::EcLevel::H)?;
let qr_image = code.render::<Luma<u8>>().min_dimensions(size, size).build();
// Convert to RGBA // Get the QR code as a matrix
let mut rgba_image = RgbaImage::new(qr_image.width(), qr_image.height()); let matrix = code.to_colors();
for (x, y, pixel) in qr_image.enumerate_pixels() { let qr_width = code.width();
let color = if pixel[0] == 0 {
Rgba([0, 0, 0, 255]) // Calculate scale factor
} else { let scale = (size as usize) / qr_width;
Rgba([255, 255, 255, 255]) let actual_size = qr_width * scale;
};
rgba_image.put_pixel(x, y, color); // Calculate logo area (center 20% of the QR code)
let logo_size = actual_size / 5;
let logo_start = (actual_size - logo_size) / 2;
let logo_end = logo_start + logo_size;
// Create RGBA pixel buffer (4 bytes per pixel)
let mut pixels: Vec<u8> = Vec::with_capacity(actual_size * actual_size * 4);
for y in 0..actual_size {
for x in 0..actual_size {
// Check if we're in the logo area
if x >= logo_start && x < logo_end && y >= logo_start && y < logo_end {
// White background for logo area
pixels.extend_from_slice(&[255, 255, 255, 255]);
} else {
let qr_x = x / scale;
let qr_y = y / scale;
let idx = qr_y * qr_width + qr_x;
let is_dark = matrix
.get(idx)
.map(|c| *c == qrcode::Color::Dark)
.unwrap_or(false);
if is_dark {
pixels.extend_from_slice(&[0, 0, 0, 255]);
} else {
pixels.extend_from_slice(&[255, 255, 255, 255]);
}
}
}
} }
// Load and resize logo // Save as PNG
let logo = image::open(logo_path)?; let file = File::create(output_path)?;
let logo_size = size / 5; // Logo should be about 20% of QR code size let ref mut w = BufWriter::new(file);
let resized_logo = logo.resize(logo_size, logo_size, imageops::FilterType::Lanczos3);
// Calculate center position let mut encoder = Encoder::new(w, actual_size as u32, actual_size as u32);
let center_x = (rgba_image.width() - resized_logo.width()) / 2; encoder.set_color(ColorType::Rgba);
let center_y = (rgba_image.height() - resized_logo.height()) / 2; encoder.set_depth(BitDepth::Eight);
// Overlay logo let mut writer = encoder.write_header()?;
let mut final_image = DynamicImage::ImageRgba8(rgba_image); writer.write_image_data(&pixels)?;
imageops::overlay(
&mut final_image, // Note: Logo overlay not supported without image crate
&resized_logo, // The QR code has a white center area where a logo can be placed manually
center_x.into(), trace!("QR code with logo placeholder generated: {}", output_path);
center_y.into(),
);
final_image.save(output_path)?;
Ok(output_path.to_string()) Ok(output_path.to_string())
} }

44
src/basic/keywords/table_definition.rs
View file

@ -527,58 +527,28 @@ pub async fn create_table_on_external_db(
} }
} }
#[cfg(feature = "dynamic-db")]
async fn create_table_mysql(
connection_string: &str,
sql: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
use sqlx::mysql::MySqlPoolOptions;
use sqlx::Executor;
let pool = MySqlPoolOptions::new()
.max_connections(1)
.connect(connection_string)
.await?;
pool.execute(sql).await?;
info!("MySQL table created successfully");
Ok(())
}
#[cfg(not(feature = "dynamic-db"))]
async fn create_table_mysql( async fn create_table_mysql(
_connection_string: &str, _connection_string: &str,
_sql: &str, _sql: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> { ) -> Result<(), Box<dyn Error + Send + Sync>> {
Err("MySQL support requires the 'dynamic-db' feature".into()) // MySQL support requires diesel mysql_backend feature which pulls in problematic dependencies
// Use PostgreSQL instead, or implement via raw SQL if needed
Err("MySQL support is disabled. Please use PostgreSQL for dynamic tables.".into())
} }
#[cfg(feature = "dynamic-db")]
async fn create_table_postgres( async fn create_table_postgres(
connection_string: &str, connection_string: &str,
sql: &str, sql: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> { ) -> Result<(), Box<dyn Error + Send + Sync>> {
use sqlx::postgres::PgPoolOptions; use diesel::pg::PgConnection;
use sqlx::Executor; use diesel::prelude::*;
let pool = PgPoolOptions::new() let mut conn = PgConnection::establish(connection_string)?;
.max_connections(1) diesel::sql_query(sql).execute(&mut conn)?;
.connect(connection_string)
.await?;
pool.execute(sql).await?;
info!("PostgreSQL table created successfully"); info!("PostgreSQL table created successfully");
Ok(()) Ok(())
} }
#[cfg(not(feature = "dynamic-db"))]
async fn create_table_postgres(
_connection_string: &str,
_sql: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
Err("PostgreSQL dynamic table support requires the 'dynamic-db' feature".into())
}
/// Process TABLE definitions during .bas file compilation /// Process TABLE definitions during .bas file compilation
pub fn process_table_definitions( pub fn process_table_definitions(
state: Arc<AppState>, state: Arc<AppState>,

74
src/core/bootstrap/mod.rs
View file

@ -8,7 +8,9 @@ use aws_sdk_s3::Client;
use chrono; use chrono;
use log::{error, info, trace, warn}; use log::{error, info, trace, warn};
use rand::distr::Alphanumeric; use rand::distr::Alphanumeric;
use rcgen::{BasicConstraints, Certificate, CertificateParams, DistinguishedName, DnType, IsCa}; use rcgen::{
BasicConstraints, CertificateParams, DistinguishedName, DnType, IsCa, Issuer, KeyPair,
};
use std::fs; use std::fs;
use std::io::{self, Write}; use std::io::{self, Write};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@ -709,51 +711,39 @@ meet IN A 127.0.0.1
let ca_cert_path = cert_dir.join("ca/ca.crt"); let ca_cert_path = cert_dir.join("ca/ca.crt");
let ca_key_path = cert_dir.join("ca/ca.key"); let ca_key_path = cert_dir.join("ca/ca.key");
let ca_cert = if ca_cert_path.exists() && ca_key_path.exists() { // CA params for issuer creation
let mut ca_params = CertificateParams::default();
ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
let mut dn = DistinguishedName::new();
dn.push(DnType::CountryName, "BR");
dn.push(DnType::OrganizationName, "BotServer");
dn.push(DnType::CommonName, "BotServer CA");
ca_params.distinguished_name = dn;
ca_params.not_before = time::OffsetDateTime::now_utc();
ca_params.not_after = time::OffsetDateTime::now_utc() + time::Duration::days(3650);
let ca_key_pair: KeyPair = if ca_cert_path.exists() && ca_key_path.exists() {
info!("Using existing CA certificate"); info!("Using existing CA certificate");
// Load existing CA key and regenerate params // Load existing CA key
let key_pem = fs::read_to_string(&ca_key_path)?; let key_pem = fs::read_to_string(&ca_key_path)?;
let key_pair = rcgen::KeyPair::from_pem(&key_pem)?; KeyPair::from_pem(&key_pem)?
// Recreate CA params with the loaded key
let mut ca_params = CertificateParams::default();
ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
ca_params.key_pair = Some(key_pair);
let mut dn = DistinguishedName::new();
dn.push(DnType::CountryName, "BR");
dn.push(DnType::OrganizationName, "BotServer");
dn.push(DnType::CommonName, "BotServer CA");
ca_params.distinguished_name = dn;
ca_params.not_before = time::OffsetDateTime::now_utc();
ca_params.not_after = time::OffsetDateTime::now_utc() + time::Duration::days(3650);
Certificate::from_params(ca_params)?
} else { } else {
info!("Generating new CA certificate"); info!("Generating new CA certificate");
// Generate new CA let key_pair = KeyPair::generate()?;
let mut ca_params = CertificateParams::default(); let cert = ca_params.self_signed(&key_pair)?;
ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
let mut dn = DistinguishedName::new();
dn.push(DnType::CountryName, "BR");
dn.push(DnType::OrganizationName, "BotServer");
dn.push(DnType::CommonName, "BotServer CA");
ca_params.distinguished_name = dn;
ca_params.not_before = time::OffsetDateTime::now_utc();
ca_params.not_after = time::OffsetDateTime::now_utc() + time::Duration::days(3650);
let ca_cert = Certificate::from_params(ca_params)?;
// Save CA certificate and key // Save CA certificate and key
fs::write(&ca_cert_path, ca_cert.serialize_pem()?)?; fs::write(&ca_cert_path, cert.pem())?;
fs::write(&ca_key_path, ca_cert.serialize_private_key_pem())?; fs::write(&ca_key_path, key_pair.serialize_pem())?;
ca_cert key_pair
}; };
// Create issuer from CA params and key
let ca_issuer = Issuer::from_params(&ca_params, &ca_key_pair);
// Services that need certificates // Services that need certificates
let services = vec![ let services = vec![
("api", vec!["localhost", "127.0.0.1", "api.botserver.local"]), ("api", vec!["localhost", "127.0.0.1", "api.botserver.local"]),
@ -847,15 +837,15 @@ meet IN A 127.0.0.1
for san in sans { for san in sans {
params params
.subject_alt_names .subject_alt_names
.push(rcgen::SanType::DnsName(san.to_string())); .push(rcgen::SanType::DnsName(san.to_string().try_into()?));
} }
let cert = Certificate::from_params(params)?; let key_pair = KeyPair::generate()?;
let cert_pem = cert.serialize_pem_with_signer(&ca_cert)?; let cert = params.signed_by(&key_pair, &ca_issuer)?;
// Save certificate and key // Save certificate and key
fs::write(cert_path, cert_pem)?; fs::write(cert_path, cert.pem())?;
fs::write(key_path, cert.serialize_private_key_pem())?; fs::write(key_path, key_pair.serialize_pem())?;
// Copy CA cert to service directory for easy access // Copy CA cert to service directory for easy access
fs::copy(&ca_cert_path, service_dir.join("ca.crt"))?; fs::copy(&ca_cert_path, service_dir.join("ca.crt"))?;

217
src/core/config/mod.rs
View file

@ -1,3 +1,11 @@
pub mod model_routing_config;
pub mod sse_config;
pub mod user_memory_config;
pub use model_routing_config::{ModelRoutingConfig, RoutingStrategy, TaskType};
pub use sse_config::SseConfig;
pub use user_memory_config::UserMemoryConfig;
use crate::shared::utils::DbPool; use crate::shared::utils::DbPool;
use diesel::prelude::*; use diesel::prelude::*;
use diesel::r2d2::{ConnectionManager, PooledConnection}; use diesel::r2d2::{ConnectionManager, PooledConnection};
@ -37,6 +45,215 @@ pub struct EmailConfig {
pub smtp_server: String, pub smtp_server: String,
pub smtp_port: u16, pub smtp_port: u16,
} }
/// Custom database configuration for BASIC keywords (MariaDB, MySQL, etc.)
/// Loaded from config.csv parameters: custom-server, custom-port, custom-database, custom-username, custom-password
#[derive(Clone, Debug, Default)]
pub struct CustomDatabaseConfig {
pub server: String,
pub port: u16,
pub database: String,
pub username: String,
pub password: String,
}
impl CustomDatabaseConfig {
/// Load custom database configuration from bot-level config.csv parameters
pub fn from_bot_config(
pool: &DbPool,
target_bot_id: &Uuid,
) -> Result<Option<Self>, diesel::result::Error> {
use crate::shared::models::schema::bot_configuration::dsl::*;
let mut conn = pool.get().map_err(|e| {
diesel::result::Error::DatabaseError(
diesel::result::DatabaseErrorKind::UnableToSendCommand,
Box::new(e.to_string()),
)
})?;
// Check if custom database is configured
let database: Option<String> = bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq("custom-database"))
.select(config_value)
.first::<String>(&mut conn)
.ok()
.filter(|s| !s.is_empty());
let database = match database {
Some(db) => db,
None => return Ok(None), // No custom database configured
};
let server: String = bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq("custom-server"))
.select(config_value)
.first::<String>(&mut conn)
.ok()
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "localhost".to_string());
let port: u16 = bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq("custom-port"))
.select(config_value)
.first::<String>(&mut conn)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(3306);
let username: String = bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq("custom-username"))
.select(config_value)
.first::<String>(&mut conn)
.ok()
.unwrap_or_default();
let password: String = bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq("custom-password"))
.select(config_value)
.first::<String>(&mut conn)
.ok()
.unwrap_or_default();
Ok(Some(CustomDatabaseConfig {
server,
port,
database,
username,
password,
}))
}
/// Build a connection string for MariaDB/MySQL
pub fn connection_string(&self) -> String {
format!(
"mysql://{}:{}@{}:{}/{}",
self.username, self.password, self.server, self.port, self.database
)
}
/// Check if the configuration is valid (has required fields)
pub fn is_valid(&self) -> bool {
!self.database.is_empty() && !self.server.is_empty()
}
}
impl EmailConfig {
/// Load email configuration from bot-level config.csv parameters
/// Parameters: email-from, email-server, email-port, email-user, email-pass
pub fn from_bot_config(
pool: &DbPool,
target_bot_id: &Uuid,
) -> Result<Self, diesel::result::Error> {
let mut conn = pool.get().map_err(|e| {
diesel::result::Error::DatabaseError(
diesel::result::DatabaseErrorKind::UnableToSendCommand,
Box::new(e.to_string()),
)
})?;
// Helper to get config value
fn get_config_value(
conn: &mut diesel::r2d2::PooledConnection<
diesel::r2d2::ConnectionManager<diesel::PgConnection>,
>,
target_bot_id: &Uuid,
key: &str,
default: &str,
) -> String {
use crate::shared::models::schema::bot_configuration::dsl::*;
bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq(key))
.select(config_value)
.first::<String>(conn)
.unwrap_or_else(|_| default.to_string())
}
fn get_port_value(
conn: &mut diesel::r2d2::PooledConnection<
diesel::r2d2::ConnectionManager<diesel::PgConnection>,
>,
target_bot_id: &Uuid,
key: &str,
default: u16,
) -> u16 {
use crate::shared::models::schema::bot_configuration::dsl::*;
bot_configuration
.filter(bot_id.eq(target_bot_id))
.filter(config_key.eq(key))
.select(config_value)
.first::<String>(conn)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
// Support both old ENV-style and new config.csv style parameter names
let new_smtp_server = get_config_value(&mut conn, target_bot_id, "email-server", "");
let smtp_server = if !new_smtp_server.is_empty() {
new_smtp_server
} else {
get_config_value(
&mut conn,
target_bot_id,
"EMAIL_SMTP_SERVER",
"smtp.gmail.com",
)
};
let new_smtp_port = get_port_value(&mut conn, target_bot_id, "email-port", 0);
let smtp_port = if new_smtp_port > 0 {
new_smtp_port
} else {
get_port_value(&mut conn, target_bot_id, "EMAIL_SMTP_PORT", 587)
};
let new_from = get_config_value(&mut conn, target_bot_id, "email-from", "");
let from = if !new_from.is_empty() {
new_from
} else {
get_config_value(&mut conn, target_bot_id, "EMAIL_FROM", "")
};
let new_user = get_config_value(&mut conn, target_bot_id, "email-user", "");
let username = if !new_user.is_empty() {
new_user
} else {
get_config_value(&mut conn, target_bot_id, "EMAIL_USERNAME", "")
};
let new_pass = get_config_value(&mut conn, target_bot_id, "email-pass", "");
let password = if !new_pass.is_empty() {
new_pass
} else {
get_config_value(&mut conn, target_bot_id, "EMAIL_PASSWORD", "")
};
let server = get_config_value(
&mut conn,
target_bot_id,
"EMAIL_IMAP_SERVER",
"imap.gmail.com",
);
let port = get_port_value(&mut conn, target_bot_id, "EMAIL_IMAP_PORT", 993);
Ok(EmailConfig {
server,
port,
username,
password,
from,
smtp_server,
smtp_port,
})
}
}
impl AppConfig { impl AppConfig {
pub fn from_database(pool: &DbPool) -> Result<Self, diesel::result::Error> { pub fn from_database(pool: &DbPool) -> Result<Self, diesel::result::Error> {
use crate::shared::models::schema::bot_configuration::dsl::*; use crate::shared::models::schema::bot_configuration::dsl::*;

130
src/security/ca.rs
View file

@ -5,8 +5,7 @@
use anyhow::Result; use anyhow::Result;
use rcgen::{ use rcgen::{
BasicConstraints, Certificate as RcgenCertificate, CertificateParams, DistinguishedName, BasicConstraints, CertificateParams, DistinguishedName, DnType, IsCa, Issuer, KeyPair, SanType,
DnType, IsCa, KeyPair, SanType,
}; };
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::fs; use std::fs;
@ -88,16 +87,18 @@ impl Default for CaConfig {
/// Certificate Authority Manager /// Certificate Authority Manager
pub struct CaManager { pub struct CaManager {
config: CaConfig, config: CaConfig,
ca_cert: Option<RcgenCertificate>, ca_params: Option<CertificateParams>,
intermediate_cert: Option<RcgenCertificate>, ca_key: Option<KeyPair>,
intermediate_params: Option<CertificateParams>,
intermediate_key: Option<KeyPair>,
} }
impl std::fmt::Debug for CaManager { impl std::fmt::Debug for CaManager {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("CaManager") f.debug_struct("CaManager")
.field("config", &self.config) .field("config", &self.config)
.field("ca_cert", &self.ca_cert.is_some()) .field("ca_params", &self.ca_params.is_some())
.field("intermediate_cert", &self.intermediate_cert.is_some()) .field("intermediate_params", &self.intermediate_params.is_some())
.finish() .finish()
} }
} }
@ -107,8 +108,10 @@ impl CaManager {
pub fn new(config: CaConfig) -> Result<Self> { pub fn new(config: CaConfig) -> Result<Self> {
let mut manager = Self { let mut manager = Self {
config, config,
ca_cert: None, ca_params: None,
intermediate_cert: None, ca_key: None,
intermediate_params: None,
intermediate_key: None,
}; };
// Load existing CA if available // Load existing CA if available
@ -125,16 +128,13 @@ impl CaManager {
self.create_ca_directories()?; self.create_ca_directories()?;
// Generate root CA // Generate root CA
let ca_cert = self.generate_root_ca()?; self.generate_root_ca()?;
// Generate intermediate CA if configured // Generate intermediate CA if configured
if self.config.intermediate_cert_path.is_some() { if self.config.intermediate_cert_path.is_some() {
let intermediate = self.generate_intermediate_ca(&ca_cert)?; self.generate_intermediate_ca()?;
self.intermediate_cert = Some(intermediate);
} }
self.ca_cert = Some(ca_cert);
info!("Certificate Authority initialized successfully"); info!("Certificate Authority initialized successfully");
Ok(()) Ok(())
} }
@ -144,17 +144,21 @@ impl CaManager {
if self.config.ca_cert_path.exists() && self.config.ca_key_path.exists() { if self.config.ca_cert_path.exists() && self.config.ca_key_path.exists() {
debug!("Loading existing CA from {:?}", self.config.ca_cert_path); debug!("Loading existing CA from {:?}", self.config.ca_cert_path);
let _cert_pem = fs::read_to_string(&self.config.ca_cert_path)?;
let key_pem = fs::read_to_string(&self.config.ca_key_path)?; let key_pem = fs::read_to_string(&self.config.ca_key_path)?;
let key_pair = KeyPair::from_pem(&key_pem)?; let key_pair = KeyPair::from_pem(&key_pem)?;
// Create CA params from scratch since rcgen doesn't support loading from PEM // Create CA params
let mut params = CertificateParams::default(); let mut params = CertificateParams::default();
params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
params.key_pair = Some(key_pair);
self.ca_cert = Some(RcgenCertificate::from_params(params)?); let mut dn = DistinguishedName::new();
dn.push(DnType::CountryName, &self.config.country);
dn.push(DnType::OrganizationName, &self.config.organization);
dn.push(DnType::CommonName, "BotServer Root CA");
params.distinguished_name = dn;
self.ca_params = Some(params);
self.ca_key = Some(key_pair);
// Load intermediate CA if exists // Load intermediate CA if exists
if let (Some(cert_path), Some(key_path)) = ( if let (Some(cert_path), Some(key_path)) = (
@ -162,17 +166,21 @@ impl CaManager {
&self.config.intermediate_key_path, &self.config.intermediate_key_path,
) { ) {
if cert_path.exists() && key_path.exists() { if cert_path.exists() && key_path.exists() {
let _cert_pem = fs::read_to_string(cert_path)?;
let key_pem = fs::read_to_string(key_path)?; let key_pem = fs::read_to_string(key_path)?;
let key_pair = KeyPair::from_pem(&key_pem)?; let key_pair = KeyPair::from_pem(&key_pem)?;
// Create intermediate CA params // Create intermediate CA params
let mut params = CertificateParams::default(); let mut params = CertificateParams::default();
params.is_ca = IsCa::Ca(BasicConstraints::Constrained(0)); params.is_ca = IsCa::Ca(BasicConstraints::Constrained(0));
params.key_pair = Some(key_pair);
self.intermediate_cert = Some(RcgenCertificate::from_params(params)?); let mut dn = DistinguishedName::new();
dn.push(DnType::CountryName, &self.config.country);
dn.push(DnType::OrganizationName, &self.config.organization);
dn.push(DnType::CommonName, "BotServer Intermediate CA");
params.distinguished_name = dn;
self.intermediate_params = Some(params);
self.intermediate_key = Some(key_pair);
} }
} }
@ -185,7 +193,7 @@ impl CaManager {
} }
/// Generate root CA certificate /// Generate root CA certificate
fn generate_root_ca(&self) -> Result<RcgenCertificate> { fn generate_root_ca(&mut self) -> Result<()> {
let mut params = CertificateParams::default(); let mut params = CertificateParams::default();
// Set as CA certificate // Set as CA certificate
@ -206,22 +214,33 @@ impl CaManager {
OffsetDateTime::now_utc() + Duration::days(self.config.validity_days * 2); OffsetDateTime::now_utc() + Duration::days(self.config.validity_days * 2);
// Generate key pair // Generate key pair
let key_pair = KeyPair::generate(&rcgen::PKCS_RSA_SHA256)?; let key_pair = KeyPair::generate()?;
params.key_pair = Some(key_pair);
// Create certificate // Create self-signed certificate
let cert = RcgenCertificate::from_params(params)?; let cert = params.self_signed(&key_pair)?;
// Save to disk // Save to disk
fs::write(&self.config.ca_cert_path, cert.serialize_pem()?)?; fs::write(&self.config.ca_cert_path, cert.pem())?;
fs::write(&self.config.ca_key_path, cert.serialize_private_key_pem())?; fs::write(&self.config.ca_key_path, key_pair.serialize_pem())?;
self.ca_params = Some(params);
self.ca_key = Some(key_pair);
info!("Generated root CA certificate"); info!("Generated root CA certificate");
Ok(cert) Ok(())
} }
/// Generate intermediate CA certificate /// Generate intermediate CA certificate
fn generate_intermediate_ca(&self, root_ca: &RcgenCertificate) -> Result<RcgenCertificate> { fn generate_intermediate_ca(&mut self) -> Result<()> {
let ca_params = self
.ca_params
.as_ref()
.ok_or_else(|| anyhow::anyhow!("Root CA params not available"))?;
let ca_key = self
.ca_key
.as_ref()
.ok_or_else(|| anyhow::anyhow!("Root CA key not available"))?;
let mut params = CertificateParams::default(); let mut params = CertificateParams::default();
// Set as intermediate CA // Set as intermediate CA
@ -241,26 +260,28 @@ impl CaManager {
params.not_after = OffsetDateTime::now_utc() + Duration::days(self.config.validity_days); params.not_after = OffsetDateTime::now_utc() + Duration::days(self.config.validity_days);
// Generate key pair // Generate key pair
let key_pair = KeyPair::generate(&rcgen::PKCS_RSA_SHA256)?; let key_pair = KeyPair::generate()?;
params.key_pair = Some(key_pair);
// Create certificate // Create issuer from root CA
let cert = RcgenCertificate::from_params(params)?; let issuer = Issuer::from_params(ca_params, ca_key);
// Sign with root CA // Create certificate signed by root CA
let signed_cert = cert.serialize_pem_with_signer(root_ca)?; let cert = params.signed_by(&key_pair, &issuer)?;
// Save to disk // Save to disk
if let (Some(cert_path), Some(key_path)) = ( if let (Some(cert_path), Some(key_path)) = (
&self.config.intermediate_cert_path, &self.config.intermediate_cert_path,
&self.config.intermediate_key_path, &self.config.intermediate_key_path,
) { ) {
fs::write(cert_path, signed_cert)?; fs::write(cert_path, cert.pem())?;
fs::write(key_path, cert.serialize_private_key_pem())?; fs::write(key_path, key_pair.serialize_pem())?;
} }
self.intermediate_params = Some(params);
self.intermediate_key = Some(key_pair);
info!("Generated intermediate CA certificate"); info!("Generated intermediate CA certificate");
Ok(cert) Ok(())
} }
/// Issue a new certificate for a service /// Issue a new certificate for a service
@ -270,11 +291,14 @@ impl CaManager {
san_names: Vec<String>, san_names: Vec<String>,
is_client: bool, is_client: bool,
) -> Result<(String, String)> { ) -> Result<(String, String)> {
let signing_ca = self let (signing_params, signing_key) =
.intermediate_cert match (&self.intermediate_params, &self.intermediate_key) {
.as_ref() (Some(params), Some(key)) => (params, key),
.or(self.ca_cert.as_ref()) _ => match (&self.ca_params, &self.ca_key) {
.ok_or_else(|| anyhow::anyhow!("CA not initialized"))?; (Some(params), Some(key)) => (params, key),
_ => return Err(anyhow::anyhow!("CA not initialized")),
},
};
let mut params = CertificateParams::default(); let mut params = CertificateParams::default();
@ -294,7 +318,9 @@ impl CaManager {
.subject_alt_names .subject_alt_names
.push(SanType::IpAddress(san.parse()?)); .push(SanType::IpAddress(san.parse()?));
} else { } else {
params.subject_alt_names.push(SanType::DnsName(san)); params
.subject_alt_names
.push(SanType::DnsName(san.try_into()?));
} }
} }
@ -310,13 +336,15 @@ impl CaManager {
} }
// Generate key pair // Generate key pair
let key_pair = KeyPair::generate(&rcgen::PKCS_RSA_SHA256)?; let key_pair = KeyPair::generate()?;
params.key_pair = Some(key_pair);
// Create issuer from signing CA
let issuer = Issuer::from_params(signing_params, signing_key);
// Create and sign certificate // Create and sign certificate
let cert = RcgenCertificate::from_params(params)?; let cert = params.signed_by(&key_pair, &issuer)?;
let cert_pem = cert.serialize_pem_with_signer(signing_ca)?; let cert_pem = cert.pem();
let key_pem = cert.serialize_private_key_pem(); let key_pem = key_pair.serialize_pem();
Ok((cert_pem, key_pem)) Ok((cert_pem, key_pem))
} }

356
src/vector-db/hybrid_search.rs
View file

@ -3,14 +3,39 @@
//! Implements hybrid search combining sparse (BM25) and dense (embedding) retrieval //! Implements hybrid search combining sparse (BM25) and dense (embedding) retrieval
//! with Reciprocal Rank Fusion (RRF) for optimal results. //! with Reciprocal Rank Fusion (RRF) for optimal results.
//! //!
//! Config.csv properties: //! # Features
//!
//! - **BM25 Sparse Search**: Powered by Tantivy (when `vectordb` feature enabled)
//! - **Dense Search**: Uses Qdrant for embedding-based similarity search
//! - **Hybrid Fusion**: Reciprocal Rank Fusion (RRF) combines both methods
//! - **Reranking**: Optional cross-encoder reranking for improved relevance
//!
//! # Config.csv Properties
//!
//! ```csv //! ```csv
//! # Hybrid search weights
//! rag-hybrid-enabled,true //! rag-hybrid-enabled,true
//! rag-dense-weight,0.7 //! rag-dense-weight,0.7
//! rag-sparse-weight,0.3 //! rag-sparse-weight,0.3
//! rag-reranker-enabled,true //! rag-reranker-enabled,true
//! rag-reranker-model,cross-encoder/ms-marco-MiniLM-L-6-v2 //! rag-reranker-model,cross-encoder/ms-marco-MiniLM-L-6-v2
//! rag-max-results,10
//! rag-min-score,0.0
//! rag-rrf-k,60
//!
//! # BM25 tuning (see bm25_config.rs for details)
//! bm25-enabled,true
//! bm25-k1,1.2
//! bm25-b,0.75
//! bm25-stemming,true
//! bm25-stopwords,true
//! ``` //! ```
//!
//! # Switching Search Modes
//!
//! - **Hybrid (default)**: Set `rag-hybrid-enabled=true` and `bm25-enabled=true`
//! - **Dense only**: Set `bm25-enabled=false` (faster, semantic search only)
//! - **Sparse only**: Set `rag-dense-weight=0` and `rag-sparse-weight=1`
use log::{debug, error, info, trace, warn}; use log::{debug, error, info, trace, warn};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@ -37,6 +62,8 @@ pub struct HybridSearchConfig {
pub min_score: f32, pub min_score: f32,
/// K parameter for RRF (typically 60) /// K parameter for RRF (typically 60)
pub rrf_k: u32, pub rrf_k: u32,
/// Whether BM25 sparse search is enabled
pub bm25_enabled: bool,
} }
impl Default for HybridSearchConfig { impl Default for HybridSearchConfig {
@ -49,6 +76,7 @@ impl Default for HybridSearchConfig {
max_results: 10, max_results: 10,
min_score: 0.0, min_score: 0.0,
rrf_k: 60, rrf_k: 60,
bm25_enabled: true,
} }
} }
} }
@ -71,7 +99,7 @@ impl HybridSearchConfig {
let configs: Vec<ConfigRow> = diesel::sql_query( let configs: Vec<ConfigRow> = diesel::sql_query(
"SELECT config_key, config_value FROM bot_configuration \ "SELECT config_key, config_value FROM bot_configuration \
WHERE bot_id = $1 AND config_key LIKE 'rag-%'", WHERE bot_id = $1 AND (config_key LIKE 'rag-%' OR config_key LIKE 'bm25-%')",
) )
.bind::<diesel::sql_types::Uuid, _>(bot_id) .bind::<diesel::sql_types::Uuid, _>(bot_id)
.load(&mut conn) .load(&mut conn)
@ -100,6 +128,9 @@ impl HybridSearchConfig {
"rag-rrf-k" => { "rag-rrf-k" => {
config.rrf_k = row.config_value.parse().unwrap_or(60); config.rrf_k = row.config_value.parse().unwrap_or(60);
} }
"bm25-enabled" => {
config.bm25_enabled = row.config_value.to_lowercase() == "true";
}
_ => {} _ => {}
} }
} }
@ -112,8 +143,23 @@ impl HybridSearchConfig {
config.sparse_weight /= total; config.sparse_weight /= total;
} }
debug!(
"Loaded HybridSearchConfig: dense={}, sparse={}, bm25_enabled={}",
config.dense_weight, config.sparse_weight, config.bm25_enabled
);
config config
} }
/// Check if sparse (BM25) search should be used
pub fn use_sparse_search(&self) -> bool {
self.bm25_enabled && self.sparse_weight > 0.0
}
/// Check if dense (embedding) search should be used
pub fn use_dense_search(&self) -> bool {
self.dense_weight > 0.0
}
} }
/// Search result from any retrieval method /// Search result from any retrieval method
@ -142,23 +188,23 @@ pub enum SearchMethod {
Reranked, Reranked,
} }
/// BM25 search index for sparse retrieval // ============================================================================
// Built-in BM25 Index Implementation
// ============================================================================
pub struct BM25Index { pub struct BM25Index {
/// Document frequency for each term
doc_freq: HashMap<String, usize>, doc_freq: HashMap<String, usize>,
/// Total number of documents
doc_count: usize, doc_count: usize,
/// Average document length
avg_doc_len: f32, avg_doc_len: f32,
/// Document lengths
doc_lengths: HashMap<String, usize>, doc_lengths: HashMap<String, usize>,
/// Term frequencies per document
term_freqs: HashMap<String, HashMap<String, usize>>, term_freqs: HashMap<String, HashMap<String, usize>>,
/// BM25 parameters doc_sources: HashMap<String, String>,
k1: f32, k1: f32,
b: f32, b: f32,
enabled: bool,
} }
#[cfg(not(feature = "vectordb"))]
impl BM25Index { impl BM25Index {
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
@ -167,27 +213,31 @@ impl BM25Index {
avg_doc_len: 0.0, avg_doc_len: 0.0,
doc_lengths: HashMap::new(), doc_lengths: HashMap::new(),
term_freqs: HashMap::new(), term_freqs: HashMap::new(),
doc_sources: HashMap::new(),
k1: 1.2, k1: 1.2,
b: 0.75, b: 0.75,
enabled: true,
} }
} }
/// Add a document to the index pub fn add_document(&mut self, doc_id: &str, content: &str, source: &str) {
pub fn add_document(&mut self, doc_id: &str, content: &str) { if !self.enabled {
return;
}
let terms = self.tokenize(content); let terms = self.tokenize(content);
let doc_len = terms.len(); let doc_len = terms.len();
// Update document length
self.doc_lengths.insert(doc_id.to_string(), doc_len); self.doc_lengths.insert(doc_id.to_string(), doc_len);
self.doc_sources
.insert(doc_id.to_string(), source.to_string());
// Calculate term frequencies
let mut term_freq: HashMap<String, usize> = HashMap::new(); let mut term_freq: HashMap<String, usize> = HashMap::new();
let mut seen_terms: std::collections::HashSet<String> = std::collections::HashSet::new(); let mut seen_terms: std::collections::HashSet<String> = std::collections::HashSet::new();
for term in &terms { for term in &terms {
*term_freq.entry(term.clone()).or_insert(0) += 1; *term_freq.entry(term.clone()).or_insert(0) += 1;
// Update document frequency (only once per document per term)
if !seen_terms.contains(term) { if !seen_terms.contains(term) {
*self.doc_freq.entry(term.clone()).or_insert(0) += 1; *self.doc_freq.entry(term.clone()).or_insert(0) += 1;
seen_terms.insert(term.clone()); seen_terms.insert(term.clone());
@ -197,15 +247,12 @@ impl BM25Index {
self.term_freqs.insert(doc_id.to_string(), term_freq); self.term_freqs.insert(doc_id.to_string(), term_freq);
self.doc_count += 1; self.doc_count += 1;
// Update average document length
let total_len: usize = self.doc_lengths.values().sum(); let total_len: usize = self.doc_lengths.values().sum();
self.avg_doc_len = total_len as f32 / self.doc_count as f32; self.avg_doc_len = total_len as f32 / self.doc_count as f32;
} }
/// Remove a document from the index
pub fn remove_document(&mut self, doc_id: &str) { pub fn remove_document(&mut self, doc_id: &str) {
if let Some(term_freq) = self.term_freqs.remove(doc_id) { if let Some(term_freq) = self.term_freqs.remove(doc_id) {
// Update document frequencies
for term in term_freq.keys() { for term in term_freq.keys() {
if let Some(freq) = self.doc_freq.get_mut(term) { if let Some(freq) = self.doc_freq.get_mut(term) {
*freq = freq.saturating_sub(1); *freq = freq.saturating_sub(1);
@ -217,9 +264,9 @@ impl BM25Index {
} }
self.doc_lengths.remove(doc_id); self.doc_lengths.remove(doc_id);
self.doc_sources.remove(doc_id);
self.doc_count = self.doc_count.saturating_sub(1); self.doc_count = self.doc_count.saturating_sub(1);
// Update average document length
if self.doc_count > 0 { if self.doc_count > 0 {
let total_len: usize = self.doc_lengths.values().sum(); let total_len: usize = self.doc_lengths.values().sum();
self.avg_doc_len = total_len as f32 / self.doc_count as f32; self.avg_doc_len = total_len as f32 / self.doc_count as f32;
@ -228,8 +275,11 @@ impl BM25Index {
} }
} }
/// Search the index with BM25 scoring pub fn search(&self, query: &str, max_results: usize) -> Vec<(String, String, f32)> {
pub fn search(&self, query: &str, max_results: usize) -> Vec<(String, f32)> { if !self.enabled {
return Vec::new();
}
let query_terms = self.tokenize(query); let query_terms = self.tokenize(query);
let mut scores: HashMap<String, f32> = HashMap::new(); let mut scores: HashMap<String, f32> = HashMap::new();
@ -239,7 +289,6 @@ impl BM25Index {
continue; continue;
} }
// IDF calculation
let idf = ((self.doc_count as f32 - df as f32 + 0.5) / (df as f32 + 0.5) + 1.0).ln(); let idf = ((self.doc_count as f32 - df as f32 + 0.5) / (df as f32 + 0.5) + 1.0).ln();
for (doc_id, term_freqs) in &self.term_freqs { for (doc_id, term_freqs) in &self.term_freqs {
@ -254,31 +303,39 @@ impl BM25Index {
} }
} }
// Sort by score and return top results
let mut results: Vec<(String, f32)> = scores.into_iter().collect(); let mut results: Vec<(String, f32)> = scores.into_iter().collect();
results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
results.truncate(max_results); results.truncate(max_results);
results results
.into_iter()
.map(|(doc_id, score)| {
let source = self.doc_sources.get(&doc_id).cloned().unwrap_or_default();
(doc_id, source, score)
})
.collect()
} }
/// Tokenize text into terms
fn tokenize(&self, text: &str) -> Vec<String> { fn tokenize(&self, text: &str) -> Vec<String> {
text.to_lowercase() text.to_lowercase()
.split(|c: char| !c.is_alphanumeric()) .split(|c: char| !c.is_alphanumeric())
.filter(|s| s.len() > 2) // Filter out very short tokens .filter(|s| s.len() > 2)
.map(|s| s.to_string()) .map(|s| s.to_string())
.collect() .collect()
} }
/// Get index statistics
pub fn stats(&self) -> BM25Stats { pub fn stats(&self) -> BM25Stats {
BM25Stats { BM25Stats {
doc_count: self.doc_count, doc_count: self.doc_count,
unique_terms: self.doc_freq.len(), unique_terms: self.doc_freq.len(),
avg_doc_len: self.avg_doc_len, avg_doc_len: self.avg_doc_len,
enabled: self.enabled,
} }
} }
pub fn set_enabled(&mut self, enabled: bool) {
self.enabled = enabled;
}
} }
impl Default for BM25Index { impl Default for BM25Index {
@ -288,16 +345,29 @@ impl Default for BM25Index {
} }
/// BM25 index statistics /// BM25 index statistics
#[derive(Debug, Clone)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BM25Stats { pub struct BM25Stats {
pub doc_count: usize, pub doc_count: usize,
pub unique_terms: usize, pub unique_terms: usize,
pub avg_doc_len: f32, pub avg_doc_len: f32,
pub enabled: bool,
}
// ============================================================================
// Hybrid Search Engine
// ============================================================================
/// Document entry in the store
#[derive(Debug, Clone)]
struct DocumentEntry {
pub content: String,
pub source: String,
pub metadata: HashMap<String, String>,
} }
/// Hybrid search engine combining dense and sparse retrieval /// Hybrid search engine combining dense and sparse retrieval
pub struct HybridSearchEngine { pub struct HybridSearchEngine {
/// BM25 sparse index /// BM25 sparse index (built-in implementation)
bm25_index: BM25Index, bm25_index: BM25Index,
/// Document store for content retrieval /// Document store for content retrieval
documents: HashMap<String, DocumentEntry>, documents: HashMap<String, DocumentEntry>,
@ -309,18 +379,18 @@ pub struct HybridSearchEngine {
collection_name: String, collection_name: String,
} }
/// Document entry in the store
#[derive(Debug, Clone)]
struct DocumentEntry {
pub content: String,
pub source: String,
pub metadata: HashMap<String, String>,
}
impl HybridSearchEngine { impl HybridSearchEngine {
pub fn new(config: HybridSearchConfig, qdrant_url: &str, collection_name: &str) -> Self { pub fn new(config: HybridSearchConfig, qdrant_url: &str, collection_name: &str) -> Self {
let mut bm25_index = BM25Index::new();
bm25_index.set_enabled(config.bm25_enabled);
info!(
"Created HybridSearchEngine with fallback BM25 (enabled={})",
config.bm25_enabled
);
Self { Self {
bm25_index: BM25Index::new(), bm25_index,
documents: HashMap::new(), documents: HashMap::new(),
config, config,
qdrant_url: qdrant_url.to_string(), qdrant_url: qdrant_url.to_string(),
@ -337,8 +407,8 @@ impl HybridSearchEngine {
metadata: HashMap<String, String>, metadata: HashMap<String, String>,
embedding: Option<Vec<f32>>, embedding: Option<Vec<f32>>,
) -> Result<(), String> { ) -> Result<(), String> {
// Add to BM25 index // Add to BM25 index (fallback)
self.bm25_index.add_document(doc_id, content); self.bm25_index.add_document(doc_id, content, source);
// Store document // Store document
self.documents.insert( self.documents.insert(
@ -358,6 +428,11 @@ impl HybridSearchEngine {
Ok(()) Ok(())
} }
/// Commit pending BM25 index changes
pub fn commit(&mut self) -> Result<(), String> {
Ok(())
}
/// Remove a document from all indexes /// Remove a document from all indexes
pub async fn remove_document(&mut self, doc_id: &str) -> Result<(), String> { pub async fn remove_document(&mut self, doc_id: &str) -> Result<(), String> {
self.bm25_index.remove_document(doc_id); self.bm25_index.remove_document(doc_id);
@ -372,34 +447,46 @@ impl HybridSearchEngine {
query: &str, query: &str,
query_embedding: Option<Vec<f32>>, query_embedding: Option<Vec<f32>>,
) -> Result<Vec<SearchResult>, String> { ) -> Result<Vec<SearchResult>, String> {
let fetch_count = self.config.max_results * 3; // Fetch more for fusion let fetch_count = self.config.max_results * 3;
// Sparse search (BM25) // Sparse search (BM25 fallback)
let sparse_results = self.bm25_index.search(query, fetch_count); let sparse_results: Vec<(String, f32)> = if self.config.use_sparse_search() {
trace!( self.bm25_index
"BM25 search returned {} results for query: {}", .search(query, fetch_count)
sparse_results.len(), .into_iter()
query .map(|(doc_id, _source, score)| (doc_id, score))
); .collect()
// Dense search (Qdrant)
let dense_results = if let Some(embedding) = query_embedding {
self.search_qdrant(&embedding, fetch_count).await?
} else { } else {
Vec::new() Vec::new()
}; };
trace!(
"Dense search returned {} results for query: {}",
dense_results.len(),
query
);
// Reciprocal Rank Fusion // Dense search (Qdrant)
let fused_results = self.reciprocal_rank_fusion(&sparse_results, &dense_results); let dense_results = if self.config.use_dense_search() {
trace!("RRF produced {} fused results", fused_results.len()); if let Some(embedding) = query_embedding {
self.search_qdrant(&embedding, fetch_count).await?
} else {
Vec::new()
}
} else {
Vec::new()
};
// Combine results
let (results, method) = if sparse_results.is_empty() && dense_results.is_empty() {
(Vec::new(), SearchMethod::Hybrid)
} else if sparse_results.is_empty() {
(dense_results.clone(), SearchMethod::Dense)
} else if dense_results.is_empty() {
(sparse_results.clone(), SearchMethod::Sparse)
} else {
(
self.reciprocal_rank_fusion(&sparse_results, &dense_results),
SearchMethod::Hybrid,
)
};
// Convert to SearchResult // Convert to SearchResult
let mut results: Vec<SearchResult> = fused_results let mut search_results: Vec<SearchResult> = results
.into_iter() .into_iter()
.filter_map(|(doc_id, score)| { .filter_map(|(doc_id, score)| {
self.documents.get(&doc_id).map(|doc| SearchResult { self.documents.get(&doc_id).map(|doc| SearchResult {
@ -408,7 +495,7 @@ impl HybridSearchEngine {
source: doc.source.clone(), source: doc.source.clone(),
score, score,
metadata: doc.metadata.clone(), metadata: doc.metadata.clone(),
search_method: SearchMethod::Hybrid, search_method: method.clone(),
}) })
}) })
.filter(|r| r.score >= self.config.min_score) .filter(|r| r.score >= self.config.min_score)
@ -416,11 +503,11 @@ impl HybridSearchEngine {
.collect(); .collect();
// Optional reranking // Optional reranking
if self.config.reranker_enabled && !results.is_empty() { if self.config.reranker_enabled && !search_results.is_empty() {
results = self.rerank(query, results).await?; search_results = self.rerank(query, search_results).await?;
} }
Ok(results) Ok(search_results)
} }
/// Perform only sparse (BM25) search /// Perform only sparse (BM25) search
@ -429,7 +516,7 @@ impl HybridSearchEngine {
results results
.into_iter() .into_iter()
.filter_map(|(doc_id, score)| { .filter_map(|(doc_id, _source, score)| {
self.documents.get(&doc_id).map(|doc| SearchResult { self.documents.get(&doc_id).map(|doc| SearchResult {
doc_id, doc_id,
content: doc.content.clone(), content: doc.content.clone(),
@ -511,12 +598,11 @@ impl HybridSearchEngine {
query: &str, query: &str,
results: Vec<SearchResult>, results: Vec<SearchResult>,
) -> Result<Vec<SearchResult>, String> { ) -> Result<Vec<SearchResult>, String> {
// In a full implementation, this would call a cross-encoder model // Simple reranking based on query term overlap
// For now, we'll use a simple relevance heuristic // A full implementation would call a cross-encoder model API
let mut reranked = results; let mut reranked = results;
for result in &mut reranked { for result in &mut reranked {
// Simple reranking based on query term overlap
let query_terms: std::collections::HashSet<&str> = let query_terms: std::collections::HashSet<&str> =
query.to_lowercase().split_whitespace().collect(); query.to_lowercase().split_whitespace().collect();
let content_lower = result.content.to_lowercase(); let content_lower = result.content.to_lowercase();
@ -528,14 +614,16 @@ impl HybridSearchEngine {
} }
} }
// Combine original score with overlap
let overlap_normalized = overlap_score / query_terms.len().max(1) as f32; let overlap_normalized = overlap_score / query_terms.len().max(1) as f32;
result.score = result.score * 0.7 + overlap_normalized * 0.3; result.score = result.score * 0.7 + overlap_normalized * 0.3;
result.search_method = SearchMethod::Reranked; result.search_method = SearchMethod::Reranked;
} }
// Re-sort by new scores reranked.sort_by(|a, b| {
reranked.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal)); b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
});
Ok(reranked) Ok(reranked)
} }
@ -657,6 +745,7 @@ impl HybridSearchEngine {
bm25_doc_count: bm25_stats.doc_count, bm25_doc_count: bm25_stats.doc_count,
unique_terms: bm25_stats.unique_terms, unique_terms: bm25_stats.unique_terms,
avg_doc_len: bm25_stats.avg_doc_len, avg_doc_len: bm25_stats.avg_doc_len,
bm25_enabled: bm25_stats.enabled,
config: self.config.clone(), config: self.config.clone(),
} }
} }
@ -669,9 +758,14 @@ pub struct HybridSearchStats {
pub bm25_doc_count: usize, pub bm25_doc_count: usize,
pub unique_terms: usize, pub unique_terms: usize,
pub avg_doc_len: f32, pub avg_doc_len: f32,
pub bm25_enabled: bool,
pub config: HybridSearchConfig, pub config: HybridSearchConfig,
} }
// ============================================================================
// Query Decomposition
// ============================================================================
/// Query decomposition for complex questions /// Query decomposition for complex questions
pub struct QueryDecomposer { pub struct QueryDecomposer {
llm_endpoint: String, llm_endpoint: String,
@ -688,9 +782,6 @@ impl QueryDecomposer {
/// Decompose a complex query into simpler sub-queries /// Decompose a complex query into simpler sub-queries
pub async fn decompose(&self, query: &str) -> Result<Vec<String>, String> { pub async fn decompose(&self, query: &str) -> Result<Vec<String>, String> {
// Simple heuristic decomposition for common patterns
// A full implementation would use an LLM
let mut sub_queries = Vec::new(); let mut sub_queries = Vec::new();
// Check for conjunctions // Check for conjunctions
@ -711,7 +802,6 @@ impl QueryDecomposer {
sub_queries.push(part.to_string()); sub_queries.push(part.to_string());
} }
} else { } else {
// Try question word splitting
let question_words = ["what", "how", "why", "when", "where", "who"]; let question_words = ["what", "how", "why", "when", "where", "who"];
let lower = query.to_lowercase(); let lower = query.to_lowercase();
@ -724,7 +814,6 @@ impl QueryDecomposer {
} }
if has_multiple_questions { if has_multiple_questions {
// Split on question marks or question words
for part in query.split('?') { for part in query.split('?') {
let trimmed = part.trim(); let trimmed = part.trim();
if !trimmed.is_empty() { if !trimmed.is_empty() {
@ -734,7 +823,6 @@ impl QueryDecomposer {
} }
} }
// If no decomposition happened, return original query
if sub_queries.is_empty() { if sub_queries.is_empty() {
sub_queries.push(query.to_string()); sub_queries.push(query.to_string());
} }
@ -748,8 +836,10 @@ impl QueryDecomposer {
return sub_answers[0].clone(); return sub_answers[0].clone();
} }
// Simple concatenation with context let mut synthesis = format!(
let mut synthesis = format!("Based on your question about \"{}\", here's what I found:\n\n", query); "Based on your question about \"{}\", here's what I found:\n\n",
query
);
for (i, answer) in sub_answers.iter().enumerate() { for (i, answer) in sub_answers.iter().enumerate() {
synthesis.push_str(&format!("{}. {}\n\n", i + 1, answer)); synthesis.push_str(&format!("{}. {}\n\n", i + 1, answer));
@ -759,54 +849,14 @@ impl QueryDecomposer {
} }
} }
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn test_bm25_index_basic() {
let mut index = BM25Index::new();
index.add_document("doc1", "The quick brown fox jumps over the lazy dog");
index.add_document("doc2", "A quick brown dog runs in the park");
index.add_document("doc3", "The lazy cat sleeps all day");
let stats = index.stats();
assert_eq!(stats.doc_count, 3);
assert!(stats.avg_doc_len > 0.0);
}
#[test]
fn test_bm25_search() {
let mut index = BM25Index::new();
index.add_document("doc1", "machine learning artificial intelligence");
index.add_document("doc2", "natural language processing NLP");
index.add_document("doc3", "computer vision image recognition");
let results = index.search("machine learning", 10);
assert!(!results.is_empty());
assert_eq!(results[0].0, "doc1"); // doc1 should be first
}
#[test]
fn test_bm25_remove_document() {
let mut index = BM25Index::new();
index.add_document("doc1", "test document one");
index.add_document("doc2", "test document two");
assert_eq!(index.stats().doc_count, 2);
index.remove_document("doc1");
assert_eq!(index.stats().doc_count, 1);
let results = index.search("one", 10);
assert!(results.is_empty() || results[0].0 != "doc1");
}
#[test] #[test]
fn test_hybrid_config_default() { fn test_hybrid_config_default() {
let config = HybridSearchConfig::default(); let config = HybridSearchConfig::default();
@ -815,6 +865,29 @@ mod tests {
assert_eq!(config.sparse_weight, 0.3); assert_eq!(config.sparse_weight, 0.3);
assert!(!config.reranker_enabled); assert!(!config.reranker_enabled);
assert_eq!(config.max_results, 10); assert_eq!(config.max_results, 10);
assert!(config.bm25_enabled);
}
#[test]
fn test_hybrid_config_search_modes() {
let config = HybridSearchConfig::default();
assert!(config.use_sparse_search());
assert!(config.use_dense_search());
let dense_only = HybridSearchConfig {
bm25_enabled: false,
..Default::default()
};
assert!(!dense_only.use_sparse_search());
assert!(dense_only.use_dense_search());
let sparse_only = HybridSearchConfig {
dense_weight: 0.0,
sparse_weight: 1.0,
..Default::default()
};
assert!(sparse_only.use_sparse_search());
assert!(!sparse_only.use_dense_search());
} }
#[test] #[test]
@ -836,8 +909,8 @@ mod tests {
let fused = engine.reciprocal_rank_fusion(&sparse, &dense); let fused = engine.reciprocal_rank_fusion(&sparse, &dense);
// doc1 and doc2 should be in top results as they appear in both
assert!(!fused.is_empty()); assert!(!fused.is_empty());
// doc1 and doc2 appear in both, should rank high
let top_ids: Vec<&str> = fused.iter().take(2).map(|(id, _)| id.as_str()).collect(); let top_ids: Vec<&str> = fused.iter().take(2).map(|(id, _)| id.as_str()).collect();
assert!(top_ids.contains(&"doc1") || top_ids.contains(&"doc2")); assert!(top_ids.contains(&"doc1") || top_ids.contains(&"doc2"));
} }
@ -846,7 +919,6 @@ mod tests {
fn test_query_decomposer_simple() { fn test_query_decomposer_simple() {
let decomposer = QueryDecomposer::new("http://localhost:8081", "none"); let decomposer = QueryDecomposer::new("http://localhost:8081", "none");
// Use tokio runtime for async test
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let result = rt.block_on(async { let result = rt.block_on(async {
@ -878,4 +950,40 @@ mod tests {
assert!(parsed.is_ok()); assert!(parsed.is_ok());
assert_eq!(parsed.unwrap().doc_id, "test123"); assert_eq!(parsed.unwrap().doc_id, "test123");
} }
#[cfg(not(feature = "vectordb"))]
#[test]
fn test_fallback_bm25_index() {
let mut index = BM25Index::new();
index.add_document(
"doc1",
"machine learning artificial intelligence",
"source1",
);
index.add_document("doc2", "natural language processing NLP", "source2");
index.add_document("doc3", "computer vision image recognition", "source3");
let results = index.search("machine learning", 10);
assert!(!results.is_empty());
assert_eq!(results[0].0, "doc1");
let stats = index.stats();
assert_eq!(stats.doc_count, 3);
assert!(stats.enabled);
}
#[cfg(not(feature = "vectordb"))]
#[test]
fn test_fallback_bm25_disabled() {
let mut index = BM25Index::new();
index.set_enabled(false);
index.add_document("doc1", "test content", "source1");
let results = index.search("test", 10);
assert!(results.is_empty());
assert!(!index.stats().enabled);
}
} }

48
src/vector-db/mod.rs
View file

@ -1,8 +1,52 @@
//! Vector Database Module for RAG 2.0
//!
//! This module provides hybrid search capabilities combining:
//! - **Sparse Search (BM25)**: Powered by Tantivy when `vectordb` feature is enabled
//! - **Dense Search**: Uses Qdrant for embedding-based similarity search
//! - **Hybrid Fusion**: Reciprocal Rank Fusion (RRF) combines both methods
//!
//! # Features
//!
//! Enable the `vectordb` feature in Cargo.toml to use Tantivy-based BM25:
//! ```toml
//! [features]
//! vectordb = ["dep:qdrant-client", "dep:tantivy"]
//! ```
//!
//! # Configuration
//!
//! Configure via config.csv:
//! ```csv
//! # Enable/disable BM25 sparse search
//! bm25-enabled,true
//! bm25-k1,1.2
//! bm25-b,0.75
//!
//! # Hybrid search weights
//! rag-dense-weight,0.7
//! rag-sparse-weight,0.3
//! ```
pub mod bm25_config;
pub mod hybrid_search; pub mod hybrid_search;
pub mod vectordb_indexer; pub mod vectordb_indexer;
// BM25 Configuration exports
pub use bm25_config::{is_stopword, Bm25Config, DEFAULT_STOPWORDS};
// Hybrid Search exports
pub use hybrid_search::{ pub use hybrid_search::{
BM25Index, BM25Stats, HybridSearchConfig, HybridSearchEngine, HybridSearchStats, BM25Stats, HybridSearchConfig, HybridSearchEngine, HybridSearchStats, QueryDecomposer,
QueryDecomposer, SearchMethod, SearchResult, SearchMethod, SearchResult,
}; };
// Tantivy BM25 index (when vectordb feature enabled)
#[cfg(feature = "vectordb")]
pub use hybrid_search::TantivyBM25Index;
// Fallback BM25 index (when vectordb feature NOT enabled)
#[cfg(not(feature = "vectordb"))]
pub use hybrid_search::BM25Index;
// VectorDB Indexer exports
pub use vectordb_indexer::{IndexingStats, IndexingStatus, VectorDBIndexer}; pub use vectordb_indexer::{IndexingStats, IndexingStatus, VectorDBIndexer};

396
templates/llm-server.gbai/README.md
View file

@ -1,12 +1,398 @@
How to test (using api.pragmatismo.com.br as host): # LLM Server Template (llm-server.gbai)
POST https://api.pragmatismo.com.br/llmservergbot/dialogs/start A General Bots template for deploying LLM-powered web services that process orders and requests via API endpoints.
## Overview
The LLM Server template transforms General Bots into a headless API service that processes structured requests using LLM intelligence. It's designed for integrating AI-powered order processing, chatbot backends, and automated response systems into existing applications.
## Features
- **REST API Endpoints** - HTTP endpoints for bot interaction
- **Order Processing** - Structured JSON responses for orders
- **Product Catalog Integration** - Dynamic product menu from CSV
- **System Prompt Configuration** - Customizable AI behavior
- **Session Management** - Track conversations across requests
- **Operator Support** - Multi-operator/tenant architecture
## Package Structure
```
llm-server.gbai/
├── README.md
├── llm-server.gbdata/ # Data files
│ └── products.csv # Product catalog
├── llm-server.gbdialog/
│ └── start.bas # Main dialog with system prompt
├── llm-server.gbkb/ # Knowledge base
└── llm-server.gbot/
└── config.csv # Bot configuration
```
## API Endpoints
### Start a Session
```http
POST https://{host}/{botId}/dialogs/start
Content-Type: application/x-www-form-urlencoded
operator=123 operator=123
userSystemId=999 userSystemId=999
```
**Response:**
```json
{
"pid": "1237189231897",
"conversationId": "abc123",
"status": "started"
}
```
POST https://api.pragmatismo.com.br/api/dk/messageBot ### Send a Message
pid=1237189231897 (returned) ```http
text=soda POST https://{host}/api/dk/messageBot
Content-Type: application/x-www-form-urlencoded
pid=1237189231897
text=I want a banana
```
**Response:**
```json
{
"orderedItems": [
{
"item": {
"id": 102,
"price": 0.30,
"name": "Banana",
"quantity": 1,
"notes": ""
}
}
],
"userId": "123",
"accountIdentifier": "TableA",
"deliveryTypeId": 2
}
```
## Configuration
### System Prompt
The `start.bas` defines the AI behavior:
```basic
PARAM operator AS number LIKE 12312312 DESCRIPTION "Operator code."
DESCRIPTION It is a WebService of GB.
products = FIND "products.csv"
BEGIN SYSTEM PROMPT
You are a chatbot assisting a store attendant in processing orders. Follow these rules:
1. **Order Format**: Each order must include the product name, the table number, and the customer's name.
2. **Product Details**: The available products are:
${TOYAML(products)}
3. **JSON Response**: For each order, return a valid RFC 8259 JSON object containing:
- product name
- table number
4. **Guidelines**:
- Do **not** engage in conversation.
- Return the response in plain text JSON format only.
END SYSTEM PROMPT
```
### Product Catalog
Create `products.csv` in the `llm-server.gbdata` folder:
```csv
id,name,price,category,description
101,Apple,0.50,Fruit,Fresh red apple
102,Banana,0.30,Fruit,Ripe yellow banana
103,Orange,0.40,Fruit,Juicy orange
201,Milk,1.20,Dairy,1 liter whole milk
202,Cheese,2.50,Dairy,200g cheddar
```
### Bot Configuration
Configure in `llm-server.gbot/config.csv`:
| Parameter | Description | Example |
|-----------|-------------|---------|
| `LLM Provider` | AI model provider | `openai` |
| `LLM Model` | Specific model | `gpt-4` |
| `Max Tokens` | Response length limit | `500` |
| `Temperature` | Response creativity | `0.3` |
| `API Mode` | Enable API mode | `true` |
## Usage Examples
### cURL Examples
**Start Session:**
```bash
curl -X POST https://api.example.com/llmservergbot/dialogs/start \
-d "operator=123" \
-d "userSystemId=999"
```
**Send Order:**
```bash
curl -X POST https://api.example.com/api/dk/messageBot \
-d "pid=1237189231897" \
-d "text=I need 2 apples and 1 milk"
```
### JavaScript Integration
```javascript
async function startBotSession(operator, userId) {
const response = await fetch('https://api.example.com/llmservergbot/dialogs/start', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({ operator, userSystemId: userId })
});
return response.json();
}
async function sendMessage(pid, text) {
const response = await fetch('https://api.example.com/api/dk/messageBot', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({ pid, text })
});
return response.json();
}
// Usage
const session = await startBotSession('123', '999');
const order = await sendMessage(session.pid, 'I want a banana');
console.log(order.orderedItems);
```
### Python Integration
```python
import requests
class LLMServerClient:
def __init__(self, base_url, operator):
self.base_url = base_url
self.operator = operator
self.pid = None
def start_session(self, user_id):
response = requests.post(
f"{self.base_url}/llmservergbot/dialogs/start",
data={"operator": self.operator, "userSystemId": user_id}
)
self.pid = response.json()["pid"]
return self.pid
def send_message(self, text):
response = requests.post(
f"{self.base_url}/api/dk/messageBot",
data={"pid": self.pid, "text": text}
)
return response.json()
# Usage
client = LLMServerClient("https://api.example.com", "123")
client.start_session("999")
order = client.send_message("I need 2 bananas")
print(order)
```
## Response Format
### Order Response Structure
```json
{
"orderedItems": [
{
"item": {
"id": 102,
"price": 0.30,
"name": "Banana",
"sideItems": [],
"quantity": 2,
"notes": "ripe ones please"
}
}
],
"userId": "123",
"accountIdentifier": "Table5",
"deliveryTypeId": 2
}
```
### Field Descriptions
| Field | Type | Description |
|-------|------|-------------|
| `orderedItems` | Array | List of ordered items |
| `item.id` | Number | Product ID from catalog |
| `item.price` | Number | Unit price |
| `item.name` | String | Product name |
| `item.sideItems` | Array | Additional items |
| `item.quantity` | Number | Order quantity |
| `item.notes` | String | Special instructions |
| `userId` | String | Operator identifier |
| `accountIdentifier` | String | Table/customer identifier |
| `deliveryTypeId` | Number | Delivery method |
## Customization
### Custom Response Format
Modify the system prompt for different output structures:
```basic
BEGIN SYSTEM PROMPT
Return responses as JSON with this structure:
{
"intent": "order|question|complaint",
"entities": [...extracted entities...],
"response": "...",
"confidence": 0.0-1.0
}
END SYSTEM PROMPT
```
### Adding Validation
```basic
' Validate order before returning
order = LLM_RESPONSE
IF NOT order.orderedItems THEN
RETURN {"error": "No items in order", "suggestion": "Please specify products"}
END IF
FOR EACH item IN order.orderedItems
product = FIND "products.csv", "id = " + item.item.id
IF NOT product THEN
RETURN {"error": "Invalid product ID: " + item.item.id}
END IF
NEXT
RETURN order
```
### Multi-Language Support
```basic
PARAM language AS STRING LIKE "en" DESCRIPTION "Response language"
BEGIN SYSTEM PROMPT
Respond in ${language} language.
Available products: ${TOYAML(products)}
Return JSON format only.
END SYSTEM PROMPT
```
## Error Handling
### Common Error Responses
```json
{
"error": "session_expired",
"message": "Please start a new session",
"code": 401
}
```
```json
{
"error": "invalid_request",
"message": "Missing required parameter: text",
"code": 400
}
```
```json
{
"error": "product_not_found",
"message": "Product 'pizza' is not in our catalog",
"code": 404
}
```
## Best Practices
1. **Keep prompts focused** - Single-purpose system prompts work better
2. **Validate responses** - Always validate LLM output before returning
3. **Handle edge cases** - Plan for invalid products, empty orders
4. **Monitor usage** - Track API calls and response times
5. **Rate limiting** - Implement rate limits for production
6. **Secure endpoints** - Use authentication for production APIs
7. **Log requests** - Maintain audit logs for debugging
## Deployment
### Environment Variables
```bash
LLM_PROVIDER=openai
LLM_API_KEY=sk-...
LLM_MODEL=gpt-4
API_RATE_LIMIT=100
SESSION_TIMEOUT=3600
```
### Docker Deployment
```dockerfile
FROM generalbots/server:latest
COPY llm-server.gbai /app/packages/
ENV API_MODE=true
EXPOSE 4242
CMD ["npm", "start"]
```
## Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| Empty responses | System prompt too restrictive | Adjust prompt guidelines |
| Invalid JSON | LLM hallucination | Add JSON validation examples |
| Session expired | Timeout reached | Implement session refresh |
| Wrong products | Catalog not loaded | Verify products.csv path |
| Slow responses | Large catalog | Optimize product filtering |
## Related Templates
- `llm-tools.gbai` - LLM with tool/function calling
- `store.gbai` - Full e-commerce with order processing
- `api-client.gbai` - API integration examples
## Use Cases
- **Restaurant Ordering** - Process food orders via API
- **Retail POS Integration** - AI-powered point of sale
- **Chatbot Backend** - Headless chatbot for web/mobile apps
- **Voice Assistant Backend** - Process voice-to-text commands
- **Order Automation** - Automate order entry from various sources
## License
AGPL-3.0 - Part of General Bots Open Source Platform.
---
**Pragmatismo** - General Bots