- Add tokio timeout to SSE stream reads in OpenAI client (60s)
- Prevents indefinite hang when Kimi/Nvidia stops responding
- Add scanning AtomicBool to prevent concurrent check_gbkb_changes calls
- Skip GBKB scan entirely when all KBs already indexed in Qdrant
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Log full LLM response preview (500 chars) with has_html detection
- Log WebSocket send with message type, completeness, and content preview
- Use clone() for chunk in BotResponse to ensure accurate logging
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Add BOTSERVER_BUILD_DATE env var to /api/health response
- Set build date during CI compilation via environment variable
- Enables checking deployed binary age without SSH access
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Use last_modified timestamp instead of ETag for change detection
- Skip re-queueing KBs that are already indexed in Qdrant
- Preserve indexed status across scans when content unchanged
- Add normalize_etag helper for consistent ETag comparison
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
When a tool button like Cartas activates a KB via USE KB, instead of
returning just the tool result (empty/label), the handler now checks
if session has active KBs. If so and result is empty/trivial,
falls through to the full LLM pipeline which injects KB context.
- Add kb_indexed_folders set to track successfully indexed KB folders
- Skip re-queuing KB for indexing if already indexed and files unchanged
- Remove kb_key from indexed set when files change (forces re-index)
- Clear indexed set on KB folder deletion
- Fix hardcoded salesianos in drive_monitor prompt key (from previous commit)
- drive_monitor: replace hardcoded salesianos.gbot with dynamic bot_name
- llm/mod.rs: stop falling back to reasoning_content as content
- llm/claude.rs: same fix for Claude handler
- deepseek_r3: export strip_think_tags for reuse
- gpt_oss_20b: use strip_think_tags so all models strip tags
- gpt_oss_120b: use strip_think_tags so all models strip tags
ETag in MinIO is an MD5 content hash, so re-uploading the same content
preserves the ETag. Add last_modified comparison so config.csv changes
that don't alter content hash still get synced. Also fixes EmbeddingConfig
fallback from previous commit.
When a bot lacks embedding-url in its own config, from_bot_config now
falls back to the default bot's config via ConfigManager::get_config.
Previously it returned empty string, causing embedding server connection
failures for bots without explicit embedding configuration.
Root cause: file_states.write().await was held while trying to acquire
file_states.read().await for KB backoff check. Tokio RwLock is not
reentrant - this caused permanent deadlock.
Fix: Removed the file_states.read() backoff check. KB processor now
just checks files_being_indexed set and queues to pending_kb_index.
Backoff is handled by the KB processor itself based on fail_count.
This fixes salesianos DriveMonitor hanging for 5+ minutes every cycle.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Added debug logging at key points in check_gbkb_changes:
- ENTER with bot ID and prefix
- Object listing results
- File states lock acquisition
- New/modified file detection
- PDF detection
- File download batches
- Final remaining files download
- EXIT confirmation
This will help identify exactly where the 5-minute timeout occurs.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Added stub start_kb_processor() for non-llm builds
- Added _pending_kb_index field for non-llm builds
- Extracted KB processor logic into start_kb_processor_inner()
- Removed unused is_embedding_server_ready import
This ensures DriveMonitor compiles and runs correctly in production
where CI builds without --features llm.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Added start_kb_processor() method: long-running background task per bot
- check_gbkb_changes now queues KB folders to pending_kb_index (non-blocking)
- KB processor polls pending_kb_index and processes one at a time per bot
- Removed inline tokio::spawn from check_gbkb_changes that was causing 5min timeouts
- Added pending_kb_index field to DriveMonitor struct
This fixes salesianos DriveMonitor timeout - check_for_changes now completes
in seconds instead of hanging on KB embedding/indexing.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Added prefix filter to list_objects_v2 call: only scans {bot}.gbot/
- Removed scanning of .gbkb and .gbdialog paths which caused 5min timeouts
- This fixes salesianos DriveMonitor timeout and embed/index failure
Also fixed header detection for name,value CSV format.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Header detection was only checking for key,value but the actual
CSV uses name,value as header row. Now both are detected and skipped.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Removed unconditional .skip(1) that was skipping first config line
- Added header detection: skips first line only if it looks like 'key,value' header
- Added validation to skip empty keys
- Also fixed indentation in drive_monitor gbkb file processing
This fixes the issue where config.csv changes on Drive weren't being
synced to bot_configuration database table for salesianos bot.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- Add ETag tracking for config.csv files in DriveMonitor
- Only download and sync config.csv when ETag changes
- Prevents unnecessary database updates on every check
- Uses __config__ prefix for config.csv state keys
When a .gbkb file is deleted from the bucket, DriveMonitor now:
- Deletes the downloaded file from work directory
- When entire KB folder is empty, removes the folder too
- Prevents disk accumulation of orphaned knowledge base files
When a .bas file is deleted from the bucket, DriveMonitor now:
- Deletes the corresponding .ast compiled file
- Deletes .bas, .mcp.json, .tool.json files from work directory
- Removes the path from file_states tracking
This prevents stale compiled files from accumulating in production.
Fixed bug where DriveMonitor would overwrite indexed=true status after
successful compilation, causing files to be recompiled on every cycle.
Changes:
- Track successful compilations in HashSet before acquiring write lock
- Set indexed=true for successfully compiled files in merge loop
- Preserve indexed status for unchanged files
- Handle compilation failures with proper fail_count tracking
This ensures new .bas files are compiled to .ast once and the indexed
status is preserved, preventing unnecessary recompilation.
- Do not mark .bas files as indexed unconditionally
- Only set indexed=true when compile_tool() completes successfully
- Reset fail_count and last_failed_at on successful compilation
- Retry failed compilations automatically on next cycle
- Fixes permanent compilation failure state for salesianos start.bas
- Replace ADD SUGGESTION TOOL with ADD_SUGG_TOOL (single token)
- Replace ADD SUGGESTION TEXT with ADD_SUGG_TEXT
- Replace ADD SUGGESTION with ADD_SUGG
- Keep ADD_SUGGESTION_TOOL as legacy alias for backward compat
- Preprocessor converts ADD SUGGESTION TOOL -> ADD_SUGG_TOOL automatically
- Eliminates collision with ADD BOT, ADD MEMBER in Rhai parser
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Previously, ensure_llama_servers_running() would return early
when both LLM and embedding servers were already running, without
calling set_embedding_server_ready(true). This caused DriveMonitor
to skip KB indexing with 'Embedding server not yet marked ready'.
Fix: call set_embedding_server_ready(true) before returning early
when servers are already running.
- Filter states by kb_folder_pattern (e.g. 'cartas/', 'proc/')
- Only apply backoff based on files in that specific KB folder
- Each KB folder has independent retry timing
The bge-small-en-v1.5-f32.gguf model has n_ctx_train=512. With batch_size=16
and ~300+ tokens per chunk, total tokens exceed 512 causing GGML_ASSERT crash.
Now with batch_size=2, embeddings are processed safely.
- Skip re-indexing files that failed 3+ times within 1 hour
- Update file_states on indexing success (indexed=true, fail_count=0)
- Update file_states on indexing failure (fail_count++, last_failed_at=now)
- Don't skip KB indexing when embedding server not marked ready yet
- Embedding server health will be detected via wait_for_server() in kb_indexer
- Remove drive_monitor bypass of embedding check - let kb_indexer handle it
- Remove hardcoded URL list for remote API detection
- Try /health first, then probe with HEAD if 404/405
- Re-enable embedding server ready check in drive_monitor
- No more embedding_key hack that skipped health checks entirely
- Bug 1: check_gbkb_changes now preserves indexed=true from previous
state when etag matches, preventing redundant re-indexing every cycle
- Bug 2: USE KB fallback uses bot_id_short (8 chars) instead of random
UUID, matching the collection name convention used by DriveMonitor
- Bug 3: handle_gbkb_change now upserts into kb_collections table after
successful indexing, so USE KB can find the collection at runtime
- Changed ON CONFLICT DO NOTHING to DO UPDATE for kb_collections inserts
- Changed process_gbkb_folder return type to Result<IndexingResult>
File states were stored under /opt/gbo/work/{UUID}/file_states.json
but should be under /opt/gbo/work/{bucket_name}/file_states.json
like other bot data (e.g. /opt/gbo/work/salesianos.gbai/)
Also fixed file_states_static signature to use bucket_name consistently.
