'Audiozusammenfassung' and 'Textzusammenfassung' now correctly trigger
the audio/text summary flows instead of falling through to regular LLM
which says it can't create audio files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When text bot captures a frame during active call and gets 8x8 garbage
(E2EE not yet decrypted), retry once after 2s to allow key propagation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The re-fetch check was placed after the 5s cooldown return, so it never
executed. Now it triggers after 3+ DEC_FAILED regardless of cooldown.
Also relaxed stale key age filter from 60s to 300s to handle key
rotation during ongoing calls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call rotates E2EE keys by re-sending index 0 with a new value
when screen share starts. The LiveKit frame cryptor caches derived AES
keys per index, so overwriting index 0 does not force re-derivation.
Fix: detect when index 0 value changes and map to incrementing internal
index so the frame cryptor gets a fresh key slot. Sets all accumulated
keys on late arrival so cryptor can try both during transition.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Timeline key fetch now filters by sent_ts (max 60s age) to avoid
using keys from a previous call session
- After 3+ consecutive DEC_FAILED events, automatically re-fetches
key from timeline in case rotation happened
- Tracks DEC_FAILED count per participant, resets on OK
This should fix the issue where the bot picks up stale encryption keys
from previous calls and can't decrypt the current caller's audio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Plays immediate spoken feedback so the user knows the bot is processing
their screen share / camera before the vision API responds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The on_e2ee_state callback crashed with NameError on time.monotonic()
when video tracks (screen share) arrived, preventing E2EE key re-derivation
and causing the bot to miss screen-share related questions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Render scanned/image-based PDF pages to PNG at 200 DPI and send to AI
model as image content when text extraction returns empty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: aggressive video re-keying (set_key at 0.3/0.8/2/5s intervals)
briefly cleared encryption_key between SetKey and HKDF callback, causing
DEC_FAILED oscillation. Single set_key per track subscription is sufficient.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PR #904 callback-based HKDF hack only fired for the first frame cryptor
(audio), leaving video frame cryptors with PBKDF2 - DEC_FAILED oscillation.
PR #921 integrates HKDF natively at the WebRTC C++ level, applying uniformly
to all frame cryptors (audio + video).
Also removes aggressive video re-keying workaround and adds 5s cooldown
to DEC_FAILED re-keying handler to prevent tight loops.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Video frame cryptors may not be fully initialized when set_key() is
first called during on_track_subscribed. Audio works immediately but
video oscillates OK↔DEC_FAILED with the same key.
Add staggered re-keying at 0.3s, 0.8s, 2s, 5s after video track
subscription to ensure the key is applied after the frame cryptor
is fully ready.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
KDF_PBKDF2=0 does NOT mean raw mode — libwebrtc applies its built-in
PBKDF2 on top of pre-derived keys, causing DEC_FAILED for audio too.
Revert to KDF_HKDF=1 (Rust applies HKDF, we pass raw base keys).
Keep diagnostic improvements:
- _derive_and_set_key() wrapper with logging
- Per-track type logging (audio vs video) in on_track_subscribed
- Frame size check in look_at_screen (detect E2EE failure)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch from Rust-side HKDF (KDF_HKDF=1) to Python-side HKDF derivation
with raw key mode (KDF_RAW=0). This eliminates potential HKDF implementation
mismatches between Rust FFI and Element Call JS that caused video frame
decryption failures (audio worked, video showed 8x8 garbage frames).
Changes:
- Add _derive_and_set_key() helper that pre-derives HKDF then calls set_key()
- Set key_derivation_function=KDF_RAW (proto 0 = no Rust-side derivation)
- Replace all direct set_key() calls with _derive_and_set_key()
- Add per-track diagnostic logging (audio vs video)
- Add frame size check in look_at_screen (detect E2EE failure early)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
8x8 frames are encrypted garbage from E2EE video decryption failure.
Skip frames < 64x64 to avoid sending black/noise images to the LLM.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Text bot can now capture video frames from active call when user
types vision-related queries ("siehst du meinen bildschirm", etc.)
2. Voice transcript injected into text bot context during active calls
3. Text messages injected into voice transcript with [typed in chat] prefix
4. Bot text replies injected back into voice transcript
This enables seamless context sharing between voice calls and text chat.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Video tracks (camera + screen share) were never getting E2EE keys set
via set_key() because the condition on track_subscribed only matched
audio tracks (kind==1). This caused DEC_FAILED for all video frames,
making look_at_screen return encrypted garbage or fail entirely.
Also added track source logging to distinguish camera vs screen share.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add HTTPS instruction to system prompt so LLM never generates http:// links.
Fix bare matrixhost.eu/settings references to use full https:// URLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add scheduled messages/reminders system:
- New scheduled_messages table in memory-service with CRUD endpoints
- schedule_message, list_reminders, cancel_reminder tools for the bot
- Background scheduler loop (30s) sends due reminders automatically
- Supports one-time, daily, weekly, weekdays, monthly repeat patterns
Make article URL handling non-blocking:
- Show 3 options (discuss, text summary, audio) instead of forcing audio wizard
- Default to passing article context to AI if user just keeps chatting
- New AWAITING_LANGUAGE state for cleaner audio flow FSM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates _ensure_cross_signing() into Bot.start() flow. On first run, generates
and uploads cross-signing keys, then signs the bot device. On subsequent restarts,
detects existing cross-signatures and skips. Seeds persisted for device recovery.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows the bot to paginate back up to 500 messages in a room
to find specific content, beyond the default 10-message context window.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Route ~90% of simple chat to claude-haiku (4x cheaper), escalate to
claude-sonnet for code blocks, long messages, technical keywords,
multimodal, and explicit requests. Sentry tags track model_used,
escalation_reason, and token usage breadcrumbs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make user_id required on all request models with field validators
- Always include user_id in WHERE clause for chunk queries (prevents cross-user data leak)
- Add bearer token auth on all endpoints except /health
- Add composite index on (user_id, room_id) for conversation_chunks
- Bot: guard query_chunks with sender check, pass room_id, send auth token
- Docker: pass MEMORY_SERVICE_TOKEN to both bot and memory-service
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Per-user Fernet encryption for fact/chunk_text/summary fields
- Postgres RLS with memory_app restricted role
- SSL for memory-db connections
- Data migration script (migrate_encrypt.py)
- DB migration (migrate_rls.sql)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scanned passport PDFs have completely garbled OCR text that makes
the LLM think they're not passports, even though the AI-generated
title and summary correctly identify them. Added explicit instruction
to trust title/summary fields.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When doc_context is available, limit history to just 4 messages (2 exchanges)
to prevent stale answer patterns from overriding fresh document search results.
Without RAG results, keep 10 messages for normal conversation context.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The AI reply often contains full document content (passport details, etc.)
which the memory extraction LLM incorrectly stores as user facts. Limiting
to 200 chars avoids including document content while keeping the gist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
30 messages of "only one passport" history overwhelmed fresh RAG results.
Reducing to 10 messages (5 exchanges) provides enough conversation context
without letting stale patterns dominate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two changes:
1. Reorder messages: doc_context now placed RIGHT BEFORE the user message
(after chat history), so fresh search results override historical patterns
where the bot repeatedly said "only one passport"
2. Strengthen doc_context instructions: explicitly tell LLM that fresh search
results override chat history, and to list ALL matching documents
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The memory extraction prompt was extracting facts from RAG search results
(e.g., passport holder names) and storing them as if they were facts about
the user. Added explicit instruction to only extract facts the user directly
states about themselves.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
