GeneralBots/botbook
2
2
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: add email-read-pixel parameter and tracking API documentation

Browse source 
- Document email-read-pixel in parameters.md
- Add Email Read Tracking section to email-api.md
- Include all 4 tracking endpoints with examples
- Add database schema for sent_email_tracking
- Include privacy considerations (GDPR/LGPD)
This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2025-12-04 18:15:18 -03:00
parent 4328bec2f9
commit fde61592ab
26 changed files with 832 additions and 210 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

66
PROMPT.md
View file

@ -5,6 +5,72 @@
--- ---
## Official Icons - MANDATORY
**NEVER generate icons with LLM. ALWAYS use official SVG icons from `src/assets/icons/`.**
### Available Icons
| Icon | File | Usage |
|------|------|-------|
| Logo | `gb-logo.svg` | Main GB branding |
| Bot | `gb-bot.svg` | Bot/assistant representation |
| Analytics | `gb-analytics.svg` | Charts, metrics, dashboards |
| Calendar | `gb-calendar.svg` | Scheduling, events |
| Chat | `gb-chat.svg` | Conversations, messaging |
| Compliance | `gb-compliance.svg` | Security, auditing |
| Designer | `gb-designer.svg` | Workflow automation |
| Drive | `gb-drive.svg` | File storage, documents |
| Mail | `gb-mail.svg` | Email functionality |
| Meet | `gb-meet.svg` | Video conferencing |
| Paper | `gb-paper.svg` | Document editing |
| Research | `gb-research.svg` | Search, investigation |
| Sources | `gb-sources.svg` | Knowledge bases |
| Tasks | `gb-tasks.svg` | Task management |
| Chart | `gb-chart.svg` | Data visualization |
| Check | `gb-check.svg` | Success, completion |
| Database | `gb-database.svg` | Data storage |
| Folder | `gb-folder.svg` | File organization |
| Gear | `gb-gear.svg` | Settings |
| Globe | `gb-globe.svg` | Web, internet |
| Lightbulb | `gb-lightbulb.svg` | Ideas, tips |
| Lock | `gb-lock.svg` | Security |
| Note | `gb-note.svg` | Notes, comments |
| Package | `gb-package.svg` | Packages, modules |
| Palette | `gb-palette.svg` | Themes, styling |
| Rocket | `gb-rocket.svg` | Launch, deploy |
| Search | `gb-search.svg` | Search functionality |
| Signal | `gb-signal.svg` | Connectivity |
| Target | `gb-target.svg` | Goals, objectives |
| Tree | `gb-tree.svg` | Hierarchy, structure |
| Warning | `gb-warning.svg` | Alerts, cautions |
### Usage in Documentation
```markdown
<!-- Reference icons in docs -->
![Chat](../assets/icons/gb-chat.svg)
<!-- Or with HTML for sizing -->
<img src="../assets/icons/gb-analytics.svg" alt="Analytics" width="24">
```
### Icon Guidelines
- All icons use `stroke="currentColor"` for CSS theming
- ViewBox: `0 0 24 24`
- Stroke width: `1.5`
- Rounded line caps and joins
- Consistent with GB brand identity
**DO NOT:**
- Generate new icons with AI/LLM
- Use emoji or unicode symbols as icons
- Use external icon libraries (FontAwesome, etc.)
- Create inline SVG diagrams when an official icon exists
---
## Project Overview ## Project Overview
BotBook is the **mdBook-based documentation** for the General Bots platform. It provides comprehensive guides, tutorials, and API references for users and developers. BotBook is the **mdBook-based documentation** for the General Bots platform. It provides comprehensive guides, tutorials, and API references for users and developers.

11
src/appendix-external-services/llm-providers.md
View file

@ -388,14 +388,17 @@ llm-batch-size,10
### Context Management ### Context Management
Optimize context window usage: Optimize context window usage with episodic memory:
```csv ```csv
llm-context-compaction,true episodic-memory-enabled,true
llm-max-history-turns,10 episodic-memory-threshold,4
llm-summarize-long-contexts,true episodic-memory-history,2
episodic-memory-auto-summarize,true
``` ```
See [Episodic Memory](../chapter-03/episodic-memory.md) for details.
## Troubleshooting ## Troubleshooting
### Common Issues ### Common Issues

12
src/assets/icons/gb-analytics.svg Normal file
View file

@ -0,0 +1,12 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Chart bars - analytics visualization -->
<rect x="3" y="14" width="4" height="8" rx="1"/>
<rect x="10" y="8" width="4" height="14" rx="1"/>
<rect x="17" y="4" width="4" height="18" rx="1"/>
<!-- Trend line -->
<path d="M3 10 L10 6 L17 2" stroke-dasharray="2 2"/>
<!-- Data points -->
<circle cx="3" cy="10" r="1.5" fill="currentColor" stroke="none"/>
<circle cx="10" cy="6" r="1.5" fill="currentColor" stroke="none"/>
<circle cx="17" cy="2" r="1.5" fill="currentColor" stroke="none"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 678 B

16
src/assets/icons/gb-calendar.svg Normal file
View file

@ -0,0 +1,16 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Calendar body - rounded rectangle inspired by GB logo -->
<rect x="3" y="4" width="18" height="18" rx="3" ry="3"/>
<!-- Calendar header binding rings -->
<line x1="8" y1="2" x2="8" y2="6"/>
<line x1="16" y1="2" x2="16" y2="6"/>
<!-- Header separator -->
<line x1="3" y1="10" x2="21" y2="10"/>
<!-- Calendar grid dots - representing dates -->
<circle cx="8" cy="14" r="1" fill="currentColor" stroke="none"/>
<circle cx="12" cy="14" r="1" fill="currentColor" stroke="none"/>
<circle cx="16" cy="14" r="1" fill="currentColor" stroke="none"/>
<circle cx="8" cy="18" r="1" fill="currentColor" stroke="none"/>
<circle cx="12" cy="18" r="1" fill="currentColor" stroke="none"/>
<circle cx="16" cy="18" r="1" fill="currentColor" stroke="none"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 937 B

8
src/assets/icons/gb-chat.svg Normal file
View file

@ -0,0 +1,8 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Chat bubble - main -->
<path d="M21 12c0 4.418-4.029 8-9 8-1.107 0-2.17-.169-3.156-.476L4 21l1.338-3.346C4.493 16.267 4 14.686 4 13c0-4.418 4.029-8 9-8s8 3.582 8 7z"/>
<!-- Three dots indicating typing/conversation -->
<circle cx="9" cy="12" r="1" fill="currentColor" stroke="none"/>
<circle cx="12" cy="12" r="1" fill="currentColor" stroke="none"/>
<circle cx="15" cy="12" r="1" fill="currentColor" stroke="none"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 601 B

8
src/assets/icons/gb-compliance.svg Normal file
View file

@ -0,0 +1,8 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Shield shape - security/compliance -->
<path d="M12 2 L20 6 L20 12 C20 16.418 16.418 20 12 22 C7.582 20 4 16.418 4 12 L4 6 Z"/>
<!-- Checkmark inside shield -->
<polyline points="8 12 11 15 16 9"/>
<!-- Document lines - representing compliance docs -->
<line x1="9" y1="18" x2="15" y2="18" opacity="0.5"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 491 B

17
src/assets/icons/gb-designer.svg Normal file
View file

@ -0,0 +1,17 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Workflow nodes -->
<circle cx="5" cy="5" r="3"/>
<circle cx="19" cy="5" r="3"/>
<circle cx="5" cy="19" r="3"/>
<circle cx="19" cy="19" r="3"/>
<!-- Center hub -->
<rect x="9" y="9" width="6" height="6" rx="1"/>
<!-- Connection lines -->
<line x1="8" y1="5" x2="9" y2="10"/>
<line x1="16" y1="5" x2="15" y2="10"/>
<line x1="8" y1="19" x2="9" y2="14"/>
<line x1="16" y1="19" x2="15" y2="14"/>
<!-- Flow arrows -->
<path d="M12 6 L12 9" stroke-dasharray="1 1"/>
<path d="M12 15 L12 18" stroke-dasharray="1 1"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 711 B

9
src/assets/icons/gb-drive.svg Normal file
View file

@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Cloud storage shape -->
<path d="M18 10h-1.26A8 8 0 1 0 9 20h9a5 5 0 0 0 0-10z"/>
<!-- Upload arrow -->
<polyline points="12 13 12 17"/>
<polyline points="10 15 12 13 14 15"/>
<!-- Folder tab -->
<path d="M5 8h4l2 2h6"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 408 B

16
src/assets/icons/gb-logo.svg Normal file
View file

@ -0,0 +1,16 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- GB Bot head - rounded square inspired by brand -->
<rect x="4" y="4" width="16" height="14" rx="3" ry="3"/>
<!-- Eyes - circular like GB branding -->
<circle cx="9" cy="10" r="1.5" fill="currentColor" stroke="none"/>
<circle cx="15" cy="10" r="1.5" fill="currentColor" stroke="none"/>
<!-- Smile -->
<path d="M9 14 Q12 16 15 14" fill="none"/>
<!-- Antenna -->
<line x1="12" y1="4" x2="12" y2="1"/>
<circle cx="12" cy="1" r="1" fill="currentColor" stroke="none"/>
<!-- Base/body connection -->
<path d="M8 18 L8 21 L16 21 L16 18"/>
<line x1="10" y1="21" x2="10" y2="23"/>
<line x1="14" y1="21" x2="14" y2="23"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 812 B

9
src/assets/icons/gb-mail.svg Normal file
View file

@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Envelope body -->
<rect x="2" y="4" width="20" height="16" rx="2" ry="2"/>
<!-- Envelope flap lines -->
<polyline points="2 4 12 13 22 4"/>
<!-- Bottom fold accent -->
<line x1="2" y1="20" x2="8" y2="14"/>
<line x1="22" y1="20" x2="16" y2="14"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 433 B

11
src/assets/icons/gb-meet.svg Normal file
View file

@ -0,0 +1,11 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Video camera body -->
<rect x="2" y="5" width="14" height="14" rx="2" ry="2"/>
<!-- Camera lens/record indicator -->
<circle cx="9" cy="12" r="3" fill="currentColor" stroke="none" opacity="0.3"/>
<circle cx="9" cy="12" r="1.5" fill="currentColor" stroke="none"/>
<!-- Viewfinder triangle -->
<path d="M16 9.5 L22 6 L22 18 L16 14.5"/>
<!-- Signal waves -->
<path d="M5 2 Q9 0 13 2" stroke-dasharray="1 1" opacity="0.5"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 611 B

11
src/assets/icons/gb-paper.svg Normal file
View file

@ -0,0 +1,11 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Document body with folded corner -->
<path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/>
<!-- Folded corner -->
<polyline points="14 2 14 8 20 8"/>
<!-- Text lines -->
<line x1="8" y1="13" x2="16" y2="13"/>
<line x1="8" y1="17" x2="14" y2="17"/>
<!-- Pen/edit indicator -->
<circle cx="8" cy="9" r="1" fill="currentColor" stroke="none"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 548 B

10
src/assets/icons/gb-research.svg Normal file
View file

@ -0,0 +1,10 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Magnifying glass -->
<circle cx="10" cy="10" r="7"/>
<line x1="15" y1="15" x2="21" y2="21"/>
<!-- Document/research lines inside -->
<line x1="7" y1="8" x2="13" y2="8"/>
<line x1="7" y1="11" x2="11" y2="11"/>
<!-- Spark/insight indicator -->
<path d="M18 4 L19 2 L20 4 L22 5 L20 6 L19 8 L18 6 L16 5 Z" fill="currentColor" stroke="none"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 527 B

13
src/assets/icons/gb-sources.svg Normal file
View file

@ -0,0 +1,13 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Book/document stack representing knowledge sources -->
<path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/>
<path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"/>
<!-- Knowledge connection nodes -->
<circle cx="10" cy="8" r="1.5" fill="currentColor" stroke="none"/>
<circle cx="15" cy="8" r="1.5" fill="currentColor" stroke="none"/>
<circle cx="12.5" cy="12" r="1.5" fill="currentColor" stroke="none"/>
<!-- Connection lines between nodes -->
<line x1="10" y1="8" x2="15" y2="8" opacity="0.5"/>
<line x1="10" y1="8" x2="12.5" y2="12" opacity="0.5"/>
<line x1="15" y1="8" x2="12.5" y2="12" opacity="0.5"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 811 B

18
src/assets/icons/gb-tasks.svg Normal file
View file

@ -0,0 +1,18 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round">
<!-- Task list container -->
<rect x="3" y="3" width="18" height="18" rx="2" ry="2"/>
<!-- Checkbox 1 - checked -->
<rect x="6" y="6" width="4" height="4" rx="1"/>
<polyline points="7 8 8 9 10 7"/>
<!-- Task line 1 -->
<line x1="13" y1="8" x2="18" y2="8"/>
<!-- Checkbox 2 - checked -->
<rect x="6" y="10" width="4" height="4" rx="1"/>
<polyline points="7 12 8 13 10 11"/>
<!-- Task line 2 -->
<line x1="13" y1="12" x2="18" y2="12"/>
<!-- Checkbox 3 - unchecked -->
<rect x="6" y="14" width="4" height="4" rx="1"/>
<!-- Task line 3 -->
<line x1="13" y1="16" x2="18" y2="16"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 776 B

2
src/chapter-03/README.md
View file

@ -70,7 +70,7 @@ rag-top-k,10
- [Vector Collections](./vector-collections.md) - Collection management - [Vector Collections](./vector-collections.md) - Collection management
- [Document Indexing](./indexing.md) - Processing pipeline - [Document Indexing](./indexing.md) - Processing pipeline
- [Semantic Search](./semantic-search.md) - Search mechanics - [Semantic Search](./semantic-search.md) - Search mechanics
- [Context Compaction](./context-compaction.md) - Token management - [Episodic Memory](./episodic-memory.md) - Conversation history and context management
- [Semantic Caching](./caching.md) - Performance optimization - [Semantic Caching](./caching.md) - Performance optimization
## See Also ## See Also

279
src/chapter-03/episodic-memory.md
View file

@ -1,160 +1,187 @@
# Episodic Memory # Episodic Memory
Episodic memory automatically manages conversation history to stay within token limits while preserving important information through intelligent summarization. Episodic memory automatically manages conversation history to stay within LLM token limits while preserving important information through intelligent summarization. This system handles context compaction transparently, ensuring conversations remain coherent without manual intervention.
## Overview
Large Language Models have fixed context windows (e.g., 8K, 32K, 128K tokens). Long conversations can exceed these limits, causing truncation or errors. Episodic memory solves this by:
1. Monitoring conversation length
2. Summarizing older exchanges when thresholds are reached
3. Keeping recent messages in full detail
4. Storing summaries as "episodic memory" for continuity
## Configuration
Episodic memory is controlled by parameters in `config.csv`:
```csv
name,value
episodic-memory-enabled,true
episodic-memory-threshold,4
episodic-memory-history,2
episodic-memory-model,fast
episodic-memory-max-episodes,100
episodic-memory-retention-days,365
episodic-memory-auto-summarize,true
```
## Parameter Reference
| Parameter | Default | Type | Description |
|-----------|---------|------|-------------|
| `episodic-memory-enabled` | `true` | Boolean | Enable/disable episodic memory system |
| `episodic-memory-threshold` | `4` | Integer | Number of exchanges before compaction triggers |
| `episodic-memory-history` | `2` | Integer | Recent exchanges to keep in full detail |
| `episodic-memory-model` | `fast` | String | Model for generating summaries (`fast`, `quality`, or model name) |
| `episodic-memory-max-episodes` | `100` | Integer | Maximum episode summaries per user |
| `episodic-memory-retention-days` | `365` | Integer | Days to retain episode summaries |
| `episodic-memory-auto-summarize` | `true` | Boolean | Automatically summarize when threshold reached |
## How It Works ## How It Works
Episodic memory is controlled by two parameters in `config.csv`: ### Context Compaction Process
```csv 1. **Monitor**: System tracks message count since last summary
episodic-memory-history,2 # Keep last 2 message exchanges 2. **Trigger**: When count reaches `episodic-memory-threshold`, compaction starts
episodic-memory-threshold,4 # Compact after 4 total exchanges 3. **Summarize**: Older messages are summarized using the configured LLM
``` 4. **Preserve**: Last `episodic-memory-history` exchanges remain in full
5. **Store**: Summary saved with role "episodic" for future context
## Configuration Parameters ### Example Timeline
### episodic-memory-history With defaults (`episodic-memory-threshold=4`, `episodic-memory-history=2`):
Determines how many previous exchanges to include in the LLM context:
- Default: `2` (keeps last 2 user messages and 2 bot responses)
- Range: 1-10 depending on your token budget
- Higher values = more context but more tokens used
### episodic-memory-threshold | Exchange | Action | Context State |
Triggers compaction after N exchanges: |----------|--------|---------------|
- Default: `4` (compacts conversation after 4 back-and-forth exchanges) | 1-2 | Normal | Messages 1-2 in full |
- When reached, older messages are summarized or removed | 3-4 | Normal | Messages 1-4 in full |
- Helps manage long conversations efficiently | 5 | **Compaction** | Summary of 1-2 + Messages 3-5 in full |
| 6-7 | Normal | Summary + Messages 3-7 in full |
| 8 | **Compaction** | Summary of 1-5 + Messages 6-8 in full |
## Automatic Behavior ### Automatic Behavior
The system automatically: The system automatically:
1. Tracks conversation length 1. Tracks conversation length
2. When exchanges exceed `episodic-memory-threshold` value 2. Triggers compaction when exchanges exceed `episodic-memory-threshold`
3. Summarizes older messages using LLM 3. Summarizes older messages using the configured LLM
4. Keeps only the last `episodic-memory-history` exchanges in full 4. Keeps only the last `episodic-memory-history` exchanges in full
5. Stores summary as an "episodic memory" for context 5. Stores the summary as an "episodic memory" for future context
## Example Flow The scheduler runs every 60 seconds, checking all active sessions and processing those that exceed the threshold.
With default settings (`episodic-memory-history=2`, `episodic-memory-threshold=4`): ## Tuning Guidelines
``` ### High-Context Conversations
Exchange 1: User asks, bot responds
Exchange 2: User asks, bot responds For complex discussions requiring more history:
Exchange 3: User asks, bot responds
Exchange 4: User asks, bot responds ```csv
Exchange 5: Episodic memory created - exchanges 1-2 summarized, 3-4 kept in full name,value
Exchange 6: Context = [episodic summary] + exchanges 4-5 episodic-memory-history,5
episodic-memory-threshold,10
``` ```
### Visual Flow Diagram ### Token-Constrained Environments
<svg width="600" height="400" viewBox="0 0 600 400" xmlns="http://www.w3.org/2000/svg" style="background: transparent;"> For smaller context windows or cost optimization:
<!-- Title -->
<text x="300" y="25" text-anchor="middle" font-family="Arial, sans-serif" font-size="16" font-weight="bold" fill="currentColor">Episodic Memory Flow</text> ```csv
name,value
<!-- Conversation History --> episodic-memory-history,1
<rect x="200" y="50" width="200" height="40" fill="none" stroke="currentColor" stroke-width="2" rx="5"/> episodic-memory-threshold,2
<text x="300" y="75" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">Conversation History</text> ```
<!-- Arrow down --> ### Disable Compaction
<path d="M 300 90 L 300 110" stroke="currentColor" stroke-width="2" fill="none" marker-end="url(#arrowhead)" opacity="0.7"/>
Set threshold to 0 to disable automatic compaction:
<!-- Check Exchange Count -->
<rect x="200" y="110" width="200" height="40" fill="none" stroke="currentColor" stroke-width="2" rx="5"/> ```csv
<text x="300" y="135" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">Check Exchange Count</text> name,value
episodic-memory-threshold,0
<!-- Arrow down --> ```
<path d="M 300 150 L 300 170" stroke="currentColor" stroke-width="2" fill="none" marker-end="url(#arrowhead)" opacity="0.7"/>
### Extended Retention
<!-- Decision Diamond -->
<path d="M 300 170 L 380 210 L 300 250 L 220 210 Z" fill="none" stroke="currentColor" stroke-width="2"/> For long-term memory across sessions:
<text x="300" y="205" text-anchor="middle" font-family="Arial, sans-serif" font-size="11" fill="currentColor">Exceeds</text>
<text x="300" y="220" text-anchor="middle" font-family="Arial, sans-serif" font-size="11" fill="currentColor">episodic-memory-threshold?</text> ```csv
name,value
<!-- No branch (left) --> episodic-memory-max-episodes,500
<path d="M 220 210 L 150 210 L 150 280" stroke="currentColor" stroke-width="2" fill="none" marker-end="url(#arrowhead)" opacity="0.7"/> episodic-memory-retention-days,730
<text x="135" y="205" text-anchor="end" font-family="Arial, sans-serif" font-size="10" fill="currentColor" opacity="0.7">No</text> ```
<!-- Keep All --> ## Use Case Recommendations
<rect x="75" y="280" width="150" height="40" fill="none" stroke="currentColor" stroke-width="2" rx="5"/>
<text x="150" y="305" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">Keep All Messages</text> | Use Case | History | Threshold | Rationale |
|----------|---------|-----------|-----------|
<!-- Yes branch (right) --> | FAQ Bot | 1 | 2 | Questions are independent |
<path d="M 380 210 L 450 210 L 450 280" stroke="currentColor" stroke-width="2" fill="none" marker-end="url(#arrowhead)" opacity="0.7"/> | Customer Support | 2 | 4 | Some context needed |
<text x="465" y="205" text-anchor="start" font-family="Arial, sans-serif" font-size="10" fill="currentColor" opacity="0.7">Yes</text> | Technical Discussion | 4 | 8 | Complex topics require history |
| Therapy/Coaching | 5 | 10 | Continuity is critical |
<!-- Summarize + Keep Last N --> | Long-term Assistant | 3 | 6 | Balance memory and context |
<rect x="350" y="280" width="200" height="40" fill="none" stroke="currentColor" stroke-width="2" rx="5"/>
<text x="450" y="295" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">Summarize Old + Keep Last</text> ## Token Savings
<text x="450" y="310" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">episodic-memory-history</text>
Compaction significantly reduces token usage:
<!-- Converge paths -->
<path d="M 150 320 L 150 340 L 300 340" stroke="currentColor" stroke-width="2" fill="none" opacity="0.7"/> | Scenario | Without Compaction | With Compaction | Savings |
<path d="M 450 320 L 450 340 L 300 340" stroke="currentColor" stroke-width="2" fill="none" opacity="0.7"/> |----------|-------------------|-----------------|---------|
<path d="M 300 340 L 300 360" stroke="currentColor" stroke-width="2" fill="none" marker-end="url(#arrowhead)" opacity="0.7"/> | 10 exchanges | ~5,000 tokens | ~2,000 tokens | 60% |
| 20 exchanges | ~10,000 tokens | ~3,000 tokens | 70% |
<!-- Continue Conversation --> | 50 exchanges | ~25,000 tokens | ~5,000 tokens | 80% |
<rect x="200" y="360" width="200" height="40" fill="none" stroke="currentColor" stroke-width="2" rx="5"/>
<text x="300" y="385" text-anchor="middle" font-family="Arial, sans-serif" font-size="12" fill="currentColor">Continue Conversation</text> *Actual savings depend on message length and summary quality.*
<!-- Arrow marker definition --> ## Summary Storage
<defs>
<marker id="arrowhead" markerWidth="10" markerHeight="10" refX="5" refY="5" orient="auto"> Summaries are stored with special role identifiers:
<polygon points="0 0, 10 5, 0 10" fill="currentColor"/>
</marker> - Role `episodic` or `compact` marks summary messages
</defs> - Summaries include key points from compacted exchanges
</svg> - Original messages are not deleted, just excluded from active context
- Episodes are searchable for context retrieval across sessions
## Benefits ## Benefits
- **Automatic management** - No manual intervention needed - **Automatic management** - No manual intervention needed
- **Token efficiency** - Stay within model limits - **Token efficiency** - Stay within model context limits
- **Context preservation** - Important info kept via summaries - **Context preservation** - Important information kept via summaries
- **Relevant context** - Keeps recent exchanges in full - **Relevant context** - Recent exchanges kept in full detail
- **Cost savings** - Fewer tokens = lower API costs - **Cost savings** - Fewer tokens = lower API costs
- **Long-term memory** - Episode storage enables recall across sessions
## Adjusting Settings ## Interaction with Caching
### For longer context: Episodic memory works alongside semantic caching:
```csv
episodic-memory-history,5 # Keep more history in full
episodic-memory-threshold,10 # Summarize less frequently
```
### For minimal context: - **Caching**: Reuses responses for similar queries (see [Semantic Caching](./caching.md))
```csv - **Episodic Memory**: Manages conversation length over time
episodic-memory-history,1 # Only last exchange in full
episodic-memory-threshold,2 # Summarize aggressively
```
## Use Cases Both features reduce costs and improve performance independently.
### Customer Support
- Lower values work well (customers ask independent questions)
- `episodic-memory-history,1` and `episodic-memory-threshold,2`
### Complex Discussions
- Higher values needed (maintain conversation flow)
- `episodic-memory-history,4` and `episodic-memory-threshold,8`
### FAQ Bots
- Minimal context needed (each question is standalone)
- `episodic-memory-history,1` and `episodic-memory-threshold,2`
## Important Notes
- Episodic memory is automatic based on config.csv
- Summaries are created using the configured LLM
- Settings apply to all conversations
- Changes require bot restart
- Summaries are stored with role "episodic" in message history
## Best Practices ## Best Practices
1. **Start with defaults** - Work well for most use cases 1. **Start with defaults** - Work well for most use cases
2. **Monitor token usage** - Adjust if hitting limits 2. **Monitor token usage** - Adjust if hitting context limits
3. **Consider conversation type** - Support vs discussion 3. **Consider conversation type** - Support vs complex discussion
4. **Test different values** - Find optimal balance 4. **Test different values** - Find optimal balance for your users
5. **Set retention appropriately** - Balance memory vs privacy requirements
The system handles all compaction automatically - just configure the values that work for your use case! ## Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| Context too long | Threshold too high | Lower `episodic-memory-threshold` |
| Lost context | History too low | Increase `episodic-memory-history` |
| Summaries missing info | Model limitations | Use `quality` instead of `fast` |
| No compaction occurring | Threshold is 0 or disabled | Set positive threshold, enable feature |
| Old episodes not deleted | Retention too long | Lower `episodic-memory-retention-days` |
## See Also
- [Semantic Caching](./caching.md) - Response caching system
- [Configuration Parameters](../chapter-08-config/parameters.md) - Full parameter reference
- [LLM Configuration](../chapter-08-config/llm-config.md) - Model settings

58
src/chapter-03/summary.md
View file

@ -1,23 +1,57 @@
# Chapter 03 – Knowledge‑Base (VectorDB) Documentation Overview # Chapter 03 – Knowledge Base System Overview
This chapter explains how GeneralBots manages knowledge‑base collections, indexing, caching, and semantic search. The implementation uses vector databases for semantic search and highlights the use of the **.gbdrive** package for storage when needed. This chapter explains how BotServer manages knowledge base collections, indexing, caching, semantic search, and conversation memory. The implementation uses vector databases for semantic search and intelligent memory management for context optimization.
| Document | File | Description | | Document | File | Description |
|----------|------|-------------| |----------|------|-------------|
| **README** | [README.md](README.md) | High‑level reference for the `.gbkb` package and its core commands (`USE KB`, `CLEAR KB`, `USE WEBSITE`). | | **README** | [README.md](README.md) | High-level reference for the `.gbkb` package and its core commands (`USE KB`, `CLEAR KB`, `USE WEBSITE`). |
| **Caching** | [caching.md](caching.md) | Optional in‑memory and persistent SQLite caching to speed up frequent `FIND` queries. | | **KB and Tools** | [kb-and-tools.md](kb-and-tools.md) | Integration patterns for knowledge bases and tool systems. |
| **Context Compaction** | [context-compaction.md](context-compaction.md) | Techniques to keep the LLM context window within limits (summarization, memory pruning, sliding window). |
| **Indexing** | [indexing.md](indexing.md) | Process of extracting, chunking, embedding, and storing document vectors in the VectorDB. |
| **Semantic Caching** | [caching.md](caching.md) | Intelligent caching for LLM responses, including semantic similarity matching. |
| **Semantic Search** | [semantic-search.md](semantic-search.md) | How the `FIND` keyword performs meaning‑based retrieval using the VectorDB. |
| **Vector Collections** | [vector-collections.md](vector-collections.md) | Definition and management of vector collections, including creation, document addition, and usage in dialogs. | | **Vector Collections** | [vector-collections.md](vector-collections.md) | Definition and management of vector collections, including creation, document addition, and usage in dialogs. |
| **Document Indexing** | [indexing.md](indexing.md) | Process of extracting, chunking, embedding, and storing document vectors in the VectorDB. |
| **Semantic Search** | [semantic-search.md](semantic-search.md) | How semantic search performs meaning-based retrieval using vector embeddings. |
| **Episodic Memory** | [episodic-memory.md](episodic-memory.md) | Automatic conversation history management, context compaction, and intelligent summarization. |
| **Semantic Caching** | [caching.md](caching.md) | Intelligent caching for LLM responses, including semantic similarity matching. |
## Key Configuration Parameters
### Knowledge Base
| Parameter | Default | Description |
|-----------|---------|-------------|
| `embedding-url` | `http://localhost:8082` | Embedding service endpoint |
| `embedding-model` | `bge-small-en-v1.5` | Model for vector embeddings |
| `rag-hybrid-enabled` | `true` | Enable hybrid search |
| `rag-top-k` | `10` | Number of results to retrieve |
### Episodic Memory
| Parameter | Default | Description |
|-----------|---------|-------------|
| `episodic-memory-enabled` | `true` | Enable/disable episodic memory |
| `episodic-memory-threshold` | `4` | Exchanges before compaction |
| `episodic-memory-history` | `2` | Recent exchanges to keep |
| `episodic-memory-model` | `fast` | Model for summarization |
| `episodic-memory-max-episodes` | `100` | Max episodes per user |
| `episodic-memory-retention-days` | `365` | Days to keep episodes |
| `episodic-memory-auto-summarize` | `true` | Auto-summarize conversations |
### LLM Cache
| Parameter | Default | Description |
|-----------|---------|-------------|
| `llm-cache` | `false` | Enable/disable response caching |
| `llm-cache-ttl` | `3600` | Cache time-to-live in seconds |
| `llm-cache-semantic` | `true` | Use semantic similarity matching |
| `llm-cache-threshold` | `0.95` | Similarity threshold for cache hits |
## How to Use This Overview ## How to Use This Overview
- **Navigate**: Click the file links to read the full documentation for each topic. - **Navigate**: Click the file links to read the full documentation for each topic.
- **Reference**: Use this table as a quick lookup when developing or extending knowledge‑base functionality. - **Reference**: Use the parameter tables for quick configuration lookup.
- **Update**: When the underlying storage or VectorDB implementation changes, edit the corresponding markdown files and keep this summary in sync. - **Update**: When the underlying implementation changes, edit the corresponding markdown files and keep this summary in sync.
--- ## See Also
*This summary was added to provide a cohesive overview of Chapter 03, aligning terminology with the current architecture (VectorDB, .gbdrive, etc.).* - [.gbkb Package](../chapter-02/gbkb.md) - Folder structure for knowledge bases
- [LLM Configuration](../chapter-08-config/llm-config.md) - Model and provider settings
- [Hybrid Search](../chapter-11-features/hybrid-search.md) - Advanced RAG techniques

10
src/chapter-04-gbui/apps/README.md
View file

@ -42,6 +42,16 @@ Each application in the Suite has its own dedicated documentation with:
| 📚 **Sources** | Prompts, templates, and models | [sources.md](./sources.md) | | 📚 **Sources** | Prompts, templates, and models | [sources.md](./sources.md) |
| 🛡️ **Compliance** | Security scanner | [compliance.md](./compliance.md) | | 🛡️ **Compliance** | Security scanner | [compliance.md](./compliance.md) |
## System Components
| Component | Description | Location |
|-----------|-------------|----------|
| 🔐 **Auth** | Authentication views | `ui/suite/auth/` |
| 👤 **Attendant** | Attendant interface | `ui/suite/attendant/` |
| 🧩 **Partials** | Reusable HTML fragments | `ui/suite/partials/` |
| 🔧 **Tools** | Developer utilities | `ui/suite/tools/` |
| 📈 **Monitoring** | System monitoring dashboard | `ui/suite/monitoring/` |
--- ---
## App Launcher ## App Launcher

13
src/chapter-04-gbui/apps/chat.md
View file

@ -110,11 +110,18 @@ ws://your-server:8080/ws
| Endpoint | Method | Description | | Endpoint | Method | Description |
|----------|--------|-------------| |----------|--------|-------------|
| `/api/sessions/current/message` | POST | Send a message | | `/api/sessions` | GET | List sessions |
| `/api/sessions/current/history` | GET | Get chat history | | `/api/sessions/current/message` | POST | Send message (current session) |
| `/api/sessions/current/history` | GET | Get chat history (current session) |
| `/api/sessions/:id` | GET | Get session details |
| `/api/sessions/:id/history` | GET | Get chat history by ID |
| `/api/sessions/:id/start` | POST | Start session |
| `/api/sessions/:id/end` | POST | End session |
| `/api/voice/start` | POST | Start voice recording | | `/api/voice/start` | POST | Start voice recording |
| `/api/voice/stop` | POST | Stop voice recording | | `/api/voice/stop` | POST | Stop voice recording |
| `/api/suggestions` | GET | Get suggestion chips | | `/api/voice/status` | GET | Get voice status |
> **Note:** The frontend uses `/api/sessions/current/*` endpoints which resolve to the active session automatically.
--- ---

14
src/chapter-04-gbui/apps/drive.md
View file

@ -179,14 +179,14 @@ Drive is your personal cloud storage within General Bots Suite. Upload, organize
| Endpoint | Method | Description | | Endpoint | Method | Description |
|----------|--------|-------------| |----------|--------|-------------|
| `/api/drive/files` | GET | List files | | `/api/drive/list` | GET | List files |
| `/api/drive/files` | POST | Upload file | | `/api/drive/upload` | POST | Upload file |
| `/api/drive/files/:path` | GET | Download file | | `/api/drive/download/:path` | GET | Download file |
| `/api/drive/files/:path` | DELETE | Delete file | | `/api/drive/delete/:path` | DELETE | Delete file |
| `/api/drive/files/:path` | PATCH | Rename/move file | | `/api/drive/move` | POST | Move/rename file |
| `/api/drive/folders` | POST | Create folder | | `/api/drive/copy` | POST | Copy file |
| `/api/drive/mkdir` | POST | Create folder |
| `/api/drive/share` | POST | Share file | | `/api/drive/share` | POST | Share file |
| `/api/drive/search` | GET | Search files |
### Query Parameters ### Query Parameters

54
src/chapter-04-gbui/htmx-architecture.md
View file

@ -77,44 +77,40 @@ HTMX extends HTML with attributes that define:
``` ```
ui/suite/ ui/suite/
├── index.html # Main entry point with navigation ├── index.html # Main entry point with navigation
├── base.html # Base template
├── home.html # Home page
├── default.gbui # Full desktop layout ├── default.gbui # Full desktop layout
├── single.gbui # Simple chat layout ├── single.gbui # Simple chat layout
├── designer.html # Visual dialog designer ├── designer.html # Visual dialog designer
├── editor.html # Code editor ├── editor.html # Code editor
├── settings.html # User settings ├── settings.html # User settings
├── css/ ├── css/
│ └── app.css # Global styles │ ├── app.css # Application styles
│ ├── apps-extended.css # Extended app styles
│ ├── components.css # UI components
│ └── global.css # Global styles
├── js/ ├── js/
│ ├── layout.js # Layout management │ ├── htmx-app.js # HTMX application logic
│ └── theme-manager.js # Theme switching │ ├── theme-manager.js # Theme switching
│ └── vendor/ # Third-party libraries
├── partials/ # Reusable HTML fragments
├── auth/ # Authentication views
├── attendant/ # Attendant interface
├── chat/ ├── chat/
│ ├── chat.html # Chat component │ ├── chat.html # Chat component
│ └── chat.css # Chat styles │ ├── chat.css # Chat styles
├── drive/ │ └── projector.html # Projector view
│ └── index.html # File manager ├── drive/ # File manager
├── tasks/ ├── tasks/ # Task manager
│ ├── tasks.html # Task manager ├── mail/ # Email client
│ └── tasks.css # Task styles ├── calendar/ # Calendar view
├── mail/ ├── meet/ # Video meetings
│ ├── mail.html # Email client ├── paper/ # Document editor
│ └── mail.css # Email styles ├── research/ # AI search
├── calendar/ ├── analytics/ # Dashboards
│ └── calendar.html # Calendar view ├── sources/ # Prompts & templates
├── meet/ ├── tools/ # Developer tools
│ ├── meet.html # Video meetings └── monitoring/ # System monitoring
│ └── meet.css # Meeting styles
├── paper/
│ └── paper.html # Document editor
├── research/
│ └── research.html # AI search
├── analytics/
│ └── analytics.html # Dashboards
├── sources/
│ └── index.html # Prompts & templates
├── tools/
│ └── compliance.html # Security scanner
└── monitoring/
└── ... # System monitoring
``` ```
### Loading Pattern ### Loading Pattern

51
src/chapter-04-gbui/ui-structure.md
View file

@ -10,20 +10,45 @@ The BotServer UI system provides two interface implementations designed for diff
ui/ ui/
├── suite/ # Full-featured interface ├── suite/ # Full-featured interface
│ ├── index.html │ ├── index.html
│ ├── base.html
│ ├── home.html
│ ├── default.gbui
│ ├── single.gbui
│ ├── designer.html
│ ├── editor.html
│ ├── settings.html
│ ├── js/ │ ├── js/
│ │ ├── htmx-app.js
│ │ ├── theme-manager.js
│ │ └── vendor/
│ ├── css/ │ ├── css/
│ │ ├── app.css
│ │ ├── apps-extended.css
│ │ ├── components.css
│ │ └── global.css
│ ├── public/ │ ├── public/
│ ├── drive/ │ ├── partials/
│ ├── auth/
│ ├── attendant/
│ ├── chat/ │ ├── chat/
│ │ ├── chat.html
│ │ ├── chat.css
│ │ └── projector.html
│ ├── drive/
│ ├── mail/ │ ├── mail/
│ ├── tasks/ │ ├── tasks/
│ ├── default.gbui │ ├── calendar/
│ └── single.gbui │ ├── meet/
│ ├── paper/
│ ├── research/
│ ├── analytics/
│ ├── sources/
│ ├── tools/
│ └── monitoring/
│ │
└── minimal/ # Lightweight interface └── minimal/ # Lightweight interface
├── index.html ├── index.html
├── styles.css └── js/
└── app.js
``` ```
## Suite Interface ## Suite Interface
@ -55,7 +80,7 @@ UI paths are configured in several locations throughout the codebase.
The main server configuration in `src/main.rs` sets the static path: The main server configuration in `src/main.rs` sets the static path:
```rust ```rust
let static_path = std::path::Path::new("./web/suite"); let static_path = std::path::Path::new("./ui/suite");
``` ```
The UI server module at `src/core/ui_server/mod.rs` defines its own path: The UI server module at `src/core/ui_server/mod.rs` defines its own path:
@ -96,11 +121,17 @@ The Minimal UI implements full compliance with the Bot Core API. Both interfaces
| `/ws` | WebSocket | Real-time messaging | | `/ws` | WebSocket | Real-time messaging |
| `/api/auth` | GET | Authentication | | `/api/auth` | GET | Authentication |
| `/api/sessions` | GET/POST | Session management | | `/api/sessions` | GET/POST | Session management |
| `/api/sessions/{id}` | GET | Session details | | `/api/sessions/current/message` | POST | Send message (current session) |
| `/api/sessions/{id}/history` | GET | Message history | | `/api/sessions/current/history` | GET | Message history (current session) |
| `/api/sessions/{id}/start` | POST | Start session | | `/api/sessions/:id` | GET | Session details |
| `/api/sessions/:id/history` | GET | Message history by ID |
| `/api/sessions/:id/start` | POST | Start session |
| `/api/sessions/:id/end` | POST | End session |
| `/api/voice/start` | POST | Voice input start | | `/api/voice/start` | POST | Voice input start |
| `/api/voice/stop` | POST | Voice input stop | | `/api/voice/stop` | POST | Voice input stop |
| `/api/voice/status` | GET | Voice status |
> **Note:** The frontend uses `/api/sessions/current/*` endpoints which resolve to the active session automatically.
Both interfaces use the same WebSocket message types for communication. TEXT (1) handles regular text messages, VOICE (2) handles voice messages, CONTINUE (3) continues interrupted responses, CONTEXT (4) manages context changes, and SYSTEM (5) delivers system messages. Both interfaces use the same WebSocket message types for communication. TEXT (1) handles regular text messages, VOICE (2) handles voice messages, CONTINUE (3) continues interrupted responses, CONTEXT (4) manages context changes, and SYSTEM (5) delivers system messages.
@ -169,7 +200,7 @@ curl http://localhost:8080/
curl http://localhost:8080/suite/ curl http://localhost:8080/suite/
# Check static file serving # Check static file serving
curl http://localhost:8080/js/app.js curl http://localhost:8080/js/htmx-app.js
``` ```
## Customization ## Customization

115
src/chapter-08-config/parameters.md
View file

@ -103,13 +103,6 @@ llm-model,mixtral-8x7b-32768
| `embedding-url` | Embedding service endpoint | `http://localhost:8082` | URL | | `embedding-url` | Embedding service endpoint | `http://localhost:8082` | URL |
| `embedding-model` | Embedding model path | Required for KB | Path | | `embedding-model` | Embedding model path | Required for KB | Path |
## Prompt Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `episodic-memory-threshold` | Context compaction level | `4` | Number |
| `episodic-memory-history` | Messages in history | Not set | Number |
## Email Parameters ## Email Parameters
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
@ -119,6 +112,34 @@ llm-model,mixtral-8x7b-32768
| `email-port` | SMTP port | `587` | Number | | `email-port` | SMTP port | `587` | Number |
| `email-user` | SMTP username | Required for email | String | | `email-user` | SMTP username | Required for email | String |
| `email-pass` | SMTP password | Required for email | String | | `email-pass` | SMTP password | Required for email | String |
| `email-read-pixel` | Enable read tracking pixel in HTML emails | `false` | Boolean |
### Email Read Tracking
When `email-read-pixel` is enabled, a 1x1 transparent tracking pixel is automatically injected into HTML emails sent via the API. This allows you to:
- Track when emails are opened
- See how many times an email was opened
- Get the approximate location (IP) and device (user agent) of the reader
**API Endpoints for tracking:**
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/email/tracking/pixel/{tracking_id}` | GET | Serves the tracking pixel (called by email client) |
| `/api/email/tracking/status/{tracking_id}` | GET | Get read status for a specific email |
| `/api/email/tracking/list` | GET | List all sent emails with tracking status |
| `/api/email/tracking/stats` | GET | Get overall tracking statistics |
**Example configuration:**
```csv
email-read-pixel,true
server-url,https://yourdomain.com
```
**Note:** The `server-url` parameter is used to generate the tracking pixel URL. Make sure it's accessible from the recipient's email client.
**Privacy considerations:** Email tracking should be used responsibly. Consider disclosing tracking in your email footer for transparency.
## Theme Parameters ## Theme Parameters
@ -128,6 +149,8 @@ llm-model,mixtral-8x7b-32768
| `theme-color2` | Secondary color | Not set | Hex color | | `theme-color2` | Secondary color | Not set | Hex color |
| `theme-logo` | Logo URL | Not set | URL | | `theme-logo` | Logo URL | Not set | URL |
| `theme-title` | Bot display title | Not set | String | | `theme-title` | Bot display title | Not set | String |
| `bot-name` | Bot display name | Not set | String |
| `welcome-message` | Initial greeting message | Not set | String |
## Custom Database Parameters ## Custom Database Parameters
@ -136,11 +159,71 @@ These parameters configure external database connections for use with BASIC keyw
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `custom-server` | Database server hostname | `localhost` | Hostname | | `custom-server` | Database server hostname | `localhost` | Hostname |
| `custom-port` | Database port | `3306` | Number | | `custom-port` | Database port | `5432` | Number |
| `custom-database` | Database name | Not set | String | | `custom-database` | Database name | Not set | String |
| `custom-username` | Database user | Not set | String | | `custom-username` | Database user | Not set | String |
| `custom-password` | Database password | Not set | String | | `custom-password` | Database password | Not set | String |
## Website Crawling Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `website-expires` | Cache expiration for crawled content | `1d` | Duration |
| `website-max-depth` | Maximum crawl depth | `3` | Number |
| `website-max-pages` | Maximum pages to crawl | `100` | Number |
## Image Generator Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `image-generator-model` | Diffusion model path | Not set | Path |
| `image-generator-steps` | Inference steps | `4` | Number |
| `image-generator-width` | Output width | `512` | Pixels |
| `image-generator-height` | Output height | `512` | Pixels |
| `image-generator-gpu-layers` | GPU offload layers | `20` | Number |
| `image-generator-batch-size` | Batch size | `1` | Number |
## Video Generator Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `video-generator-model` | Video model path | Not set | Path |
| `video-generator-frames` | Frames to generate | `24` | Number |
| `video-generator-fps` | Frames per second | `8` | Number |
| `video-generator-width` | Output width | `320` | Pixels |
| `video-generator-height` | Output height | `576` | Pixels |
| `video-generator-gpu-layers` | GPU offload layers | `15` | Number |
| `video-generator-batch-size` | Batch size | `1` | Number |
## BotModels Service Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `botmodels-enabled` | Enable BotModels service | `true` | Boolean |
| `botmodels-host` | BotModels bind address | `0.0.0.0` | IP address |
| `botmodels-port` | BotModels port | `8085` | Number |
## Generator Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `default-generator` | Default content generator | `all` | String |
## Teams Channel Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `teams-app-id` | Microsoft Teams App ID | Not set | String |
| `teams-app-password` | Microsoft Teams App Password | Not set | String |
| `teams-tenant-id` | Microsoft Teams Tenant ID | Not set | String |
| `teams-bot-id` | Microsoft Teams Bot ID | Not set | String |
## SMS Parameters
| Parameter | Description | Default | Type |
|-----------|-------------|---------|------|
| `sms-provider` | SMS provider name | Not set | String |
### Example: MariaDB Connection ### Example: MariaDB Connection
```csv ```csv
custom-server,db.example.com custom-server,db.example.com
@ -199,14 +282,18 @@ bot-improvement-threshold,7.0
| `user-memory-max-keys` | Maximum keys per user | `1000` | Number | | `user-memory-max-keys` | Maximum keys per user | `1000` | Number |
| `user-memory-default-ttl` | Default time-to-live (0=no expiry) | `0` | Seconds | | `user-memory-default-ttl` | Default time-to-live (0=no expiry) | `0` | Seconds |
### Episodic Memory ### Episodic Memory (Context Compaction)
| Parameter | Description | Default | Type | | Parameter | Description | Default | Type |
|-----------|-------------|---------|------| |-----------|-------------|---------|------|
| `episodic-memory-enabled` | Enable conversation summaries | `true` | Boolean | | `episodic-memory-enabled` | Enable episodic memory system | `true` | Boolean |
| `episodic-summary-model` | Model for summarization | `fast` | String | | `episodic-memory-threshold` | Exchanges before compaction triggers | `4` | Number |
| `episodic-max-episodes` | Maximum episodes per user | `100` | Number | | `episodic-memory-history` | Recent exchanges to keep in full | `2` | Number |
| `episodic-retention-days` | Days to retain episodes | `365` | Number | | `episodic-memory-model` | Model for summarization | `fast` | String |
| `episodic-auto-summarize` | Enable automatic summarization | `true` | Boolean | | `episodic-memory-max-episodes` | Maximum episodes per user | `100` | Number |
| `episodic-memory-retention-days` | Days to retain episodes | `365` | Number |
| `episodic-memory-auto-summarize` | Enable automatic summarization | `true` | Boolean |
Episodic memory automatically manages conversation context to stay within LLM token limits. When conversation exchanges exceed `episodic-memory-threshold`, older messages are summarized and only the last `episodic-memory-history` exchanges are kept in full. See [Chapter 03 - Episodic Memory](../chapter-03/episodic-memory.md) for details.
## Model Routing Parameters ## Model Routing Parameters

202
src/chapter-10-api/email-api.md
View file

@ -358,6 +358,156 @@ stalwart-admin-password,secure-password
| Inbox | 300/hour per user | | Inbox | 300/hour per user |
| Attachments | 50/hour per user | | Attachments | 50/hour per user |
## Email Read Tracking
General Bots supports email read tracking via an invisible 1x1 pixel embedded in HTML emails. When enabled, you can track when recipients open your emails.
### Configuration
Enable tracking in `config.csv`:
```csv
name,value
email-read-pixel,true
server-url,https://yourdomain.com
```
### How It Works
1. When sending an HTML email, a tracking pixel is automatically injected
2. When the recipient opens the email, their email client loads the pixel
3. The server records the open event with timestamp and metadata
4. You can query the tracking status via API or view in the Suite UI
### Tracking Endpoints
#### Serve Tracking Pixel
**GET** `/api/email/tracking/pixel/:tracking_id`
This endpoint is called automatically by email clients when loading the tracking pixel. It returns a 1x1 transparent GIF and records the read event.
**Response:** Binary GIF image (1x1 pixel)
**Headers Set:**
- `Content-Type: image/gif`
- `Cache-Control: no-store, no-cache, must-revalidate, max-age=0`
#### Get Tracking Status
**GET** `/api/email/tracking/status/:tracking_id`
Get the read status for a specific sent email.
**Response:**
```json
{
"success": true,
"data": {
"tracking_id": "550e8400-e29b-41d4-a716-446655440000",
"to_email": "recipient@example.com",
"subject": "Meeting Tomorrow",
"sent_at": "2024-01-15T10:30:00Z",
"is_read": true,
"read_at": "2024-01-15T14:22:00Z",
"read_count": 3
}
}
```
#### List Tracked Emails
**GET** `/api/email/tracking/list`
List all sent emails with their tracking status.
**Query Parameters:**
- `account_id` - Filter by email account (optional)
- `limit` - Number of results (default: 50)
- `offset` - Pagination offset (default: 0)
- `filter` - Filter by status: `all`, `read`, `unread` (default: all)
**Response:**
```json
{
"success": true,
"data": [
{
"tracking_id": "550e8400-e29b-41d4-a716-446655440000",
"to_email": "recipient@example.com",
"subject": "Meeting Tomorrow",
"sent_at": "2024-01-15T10:30:00Z",
"is_read": true,
"read_at": "2024-01-15T14:22:00Z",
"read_count": 3
},
{
"tracking_id": "661e8400-e29b-41d4-a716-446655440001",
"to_email": "another@example.com",
"subject": "Project Update",
"sent_at": "2024-01-15T11:00:00Z",
"is_read": false,
"read_at": null,
"read_count": 0
}
]
}
```
#### Get Tracking Statistics
**GET** `/api/email/tracking/stats`
Get aggregate statistics for email tracking.
**Response:**
```json
{
"success": true,
"data": {
"total_sent": 150,
"total_read": 98,
"read_rate": 65.33,
"avg_time_to_read_hours": 4.5
}
}
```
### Tracking Data Stored
For each tracked email, the following data is recorded:
| Field | Description |
|-------|-------------|
| `tracking_id` | Unique ID embedded in the pixel URL |
| `to_email` | Recipient email address |
| `subject` | Email subject line |
| `sent_at` | Timestamp when email was sent |
| `is_read` | Whether email has been opened |
| `read_at` | Timestamp of first open |
| `read_count` | Number of times opened |
| `first_read_ip` | IP address of first open |
| `last_read_ip` | IP address of most recent open |
| `user_agent` | Browser/client user agent string |
### Privacy Considerations
- Email tracking should be used responsibly
- Consider disclosing tracking in your email footer
- Some email clients block tracking pixels by default
- Users may have images disabled, preventing tracking
- GDPR/LGPD may require consent for tracking
### Suite UI Integration
The Suite email interface shows tracking status:
- **📊 Tracking** folder shows all tracked emails
- Green checkmarks (✓✓) indicate opened emails
- Gray checkmarks indicate sent but unread
- Hover over emails to see open timestamp
- Statistics panel shows open rates
## Security Notes ## Security Notes
1. **Never hardcode credentials** - Use config.csv 1. **Never hardcode credentials** - Use config.csv
@ -392,8 +542,58 @@ CREATE TABLE email_drafts (
); );
``` ```
## Database Schema
```sql
-- user_email_accounts
CREATE TABLE user_email_accounts (
id UUID PRIMARY KEY,
user_id UUID REFERENCES users(id),
email TEXT NOT NULL,
imap_server TEXT,
smtp_server TEXT,
encrypted_password TEXT,
created_at TIMESTAMPTZ
);
-- email_drafts
CREATE TABLE email_drafts (
id UUID PRIMARY KEY,
user_id UUID REFERENCES users(id),
recipients JSONB,
subject TEXT,
body TEXT,
attachments JSONB,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);
-- sent_email_tracking (for read receipts)
CREATE TABLE sent_email_tracking (
id UUID PRIMARY KEY,
tracking_id UUID NOT NULL UNIQUE,
bot_id UUID NOT NULL,
account_id UUID NOT NULL,
from_email VARCHAR(255) NOT NULL,
to_email VARCHAR(255) NOT NULL,
cc TEXT,
bcc TEXT,
subject TEXT NOT NULL,
sent_at TIMESTAMPTZ NOT NULL,
is_read BOOLEAN NOT NULL DEFAULT FALSE,
read_at TIMESTAMPTZ,
read_count INTEGER NOT NULL DEFAULT 0,
first_read_ip VARCHAR(45),
last_read_ip VARCHAR(45),
user_agent TEXT,
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
);
```
## See Also ## See Also
- [SEND MAIL Keyword](../chapter-06-gbdialog/keyword-send-mail.md) - BASIC email - [SEND MAIL Keyword](../chapter-06-gbdialog/keyword-send-mail.md) - BASIC email
- [CREATE DRAFT Keyword](../chapter-06-gbdialog/keyword-create-draft.md) - Draft creation - [CREATE DRAFT Keyword](../chapter-06-gbdialog/keyword-create-draft.md) - Draft creation
- [External Services](../appendix-external-services/README.md) - Service configuration - [External Services](../appendix-external-services/README.md) - Service configuration
- [Configuration Parameters](../chapter-08-config/parameters.md) - email-read-pixel setting

9
src/chapter-11-features/memory-management.md
View file

@ -269,9 +269,12 @@ SET CONTEXT "previous_interactions" AS episodes
```csv ```csv
name,value name,value
episodic-memory-enabled,true episodic-memory-enabled,true
episodic-summary-model,fast episodic-memory-threshold,4
episodic-max-episodes,100 episodic-memory-history,2
episodic-retention-days,365 episodic-memory-model,fast
episodic-memory-max-episodes,100
episodic-memory-retention-days,365
episodic-memory-auto-summarize,true
``` ```
## Memory Patterns ## Memory Patterns