- Query user memories at call start and inject into agent system prompt
- Extract new facts after each exchange using claude-haiku via LiteLLM
- Add Brave Search tool (@function_tool) for current data queries
- Pass memory client and caller_user_id through VoiceSession constructor
- Pre-compute 8 HMAC-ratcheted EC keys for reliable E2EE decryption
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bot.py: track active callers per room; only stop session when last
caller leaves (fixes premature cancellation when Playwright browser
hangs up while real app is still in call)
- voice.py: pre-compute 8 HMAC-ratcheted keys from EC's base key so
decryption works immediately without waiting ~30s for Matrix to
deliver EC's key-rotation event (root cause of user→bot silence)
- voice.py: fix set_key() argument order (identity, key, index) at all
call sites — was (identity, index, key) causing TypeError
- voice.py: add audio frame monitor (AUDIO_FLOW) and mute/unmute event
handlers for diagnostics
- voice.py: update livekit-agents 1.4.2 event names: user_state_changed,
user_input_transcribed, conversation_item_added
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, not shared key mode.
Bot now generates its own key, publishes it, and sets both
keys via key_provider.set_key() after connecting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys via MatrixKeyProvider.onSetEncryptionKey(),
not shared key mode. This was causing silence with E2EE enabled.
- Set bot's own key and caller's key separately via e2ee_manager.key_provider.set_key()
- Live-update caller key when received after connect
- Fallback to set_shared_key if per-participant API unavailable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bot now publishes the same key as the caller so both sides can decrypt.
Falls back to no-encryption if no caller key received.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip bot own encryption_keys events in on_unknown handler
- Always pass valid RoomOptions to AgentSession.start()
- Wait up to 10s for remote participant to connect before starting pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call distributes encryption keys as timeline events, not room
state events. Changed bot to publish keys via room_send and fetch from
/messages endpoint instead of /state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Reorder: send call member event BEFORE creating VoiceSession
- Store VoiceSession BEFORE start so sync handler can forward keys
- Increase E2EE key wait from 3s to 10s
- Add INFO-level logging for key lookup + room state scan via HTTP API
- Tighten voice system prompt to prevent long rambling greetings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix state_key format: try @user:domain:DEVICE_ID (Element Call format),
then @user:domain, then scan all room state as fallback
- Publish bot E2EE key to room so Element shows encrypted status
- Extract caller device_id from call member event content
- Also fix pipecat-poc pipeline with context aggregators (CF-1579)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stop VoiceSession when call leave event received
- Copy libstdc++ from rust build stage to fix CXXABI_1.3.15 mismatch
- Read caller encryption key from room state before starting VoiceSession
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses HKDF-SHA256 + AES-128-GCM for frame encryption,
while the LiveKit Rust SDK defaults to PBKDF2 + AES-256-GCM.
- Multi-stage Dockerfile builds patched Rust FFI from EC-compat fork
- Generates Python protobuf bindings with new fields
- patch_sdk.py modifies installed livekit-rtc for new proto fields
- agent.py passes E2EE options with HKDF to ctx.connect()
- bot.py exchanges encryption keys via Matrix state events
- Separate Dockerfile.bot for bot service (no Rust build needed)
Ref: livekit/rust-sdks#904, livekit/python-sdks#570
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses SHA256(room_id + "|m.call#ROOM") encoded as unpadded
base64 for LiveKit room names (via lk-jwt-service). The bot was using
the raw Matrix room ID, causing agent and user to join different rooms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add memory-service (FastAPI + pgvector) for semantic memory storage.
Bot now queries relevant memories per conversation instead of dumping all 50.
Includes migration script for existing JSON files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace separate bot-crypto/bot-memories volumes with single bot-data:/data
volume so user_keys.json and language_prefs.json persist across restarts
- Remove redundant language_prefs.json infrastructure (constant, load/save,
dict) — language preference now read from memories (last match wins)
- Add robust JSON extraction in _extract_memories (regex fallback for
markdown fences, embedded arrays, non-array responses)
- Add info-level logging throughout memory extraction pipeline
- Add asyncio.wait_for timeout (15s) on memory extraction to prevent hangs
- Add !ai memory <fact> command for explicit, reliable memory storage
- Update _get_preferred_language to return last match (most recent wins)
- Update !ai forget to clear in-memory caches (pending translate/reply)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Upgrade memory/translation debug logs from debug to warning level
- Auto-detect language preference from extracted memory facts
- Persist language prefs to separate JSON file for reliability
- Add translation detection logging
- Use single linebreaks in translation menu
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect when a DM message is in a foreign language and offer an
interactive menu: translate, compose reply in that language, or
respond normally. Supports forwarded WhatsApp messages via Element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract and store memorable facts (name, language, preferences) per user
- Inject memories into system prompt for personalized responses
- LLM-based extraction after each response, deduplication against existing
- JSON files on Docker volume (/data/memories), capped at 50 per user
- System prompt updated: respond in users language, use memories
- Commands: !ai memories (view), !ai forget (delete all)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add RoomEncryptedImage callback with decrypt_attachment for E2E rooms
- Cache recent images per room (60s TTL) so follow-up text messages
like "was ist das" get the image context instead of hallucinating
- Treat filenames (containing dots) as no-caption, default to
"What's in this image?"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Upload with encrypt=True and filesize param. Handle UploadError
gracefully. Use m.file encrypted format when encryption keys returned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Register RoomMessageFile callback, filter for application/pdf
- Extract text from PDFs using pymupdf (fitz)
- Send extracted text as context to LLM for summarization/Q&A
- Truncate at 50k chars to avoid token limits
- Add pymupdf to requirements.txt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Register RoomMessageImage callback to handle incoming images
- Download and base64-encode images, send as multimodal content to LLM
- Add LLM tool calling with generate_image tool for natural image generation
- Upload generated images back to Matrix via m.image events
- Update system prompt to inform LLM about vision and image gen capabilities
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
!ai connect (no args) now starts a browser-based device authorization
flow instead of requiring a raw API key. Direct key input preserved
as fallback. Bot polls WildFiles for approval with 5s interval.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- !ai connect <key>: validates key against WildFiles, stores per-user mapping, redacts message
- !ai disconnect: removes stored key
- RAG searches use per-user API key when available, fall back to WILDFILES_ORG
- Keys stored in /data/user_keys.json (Docker volume)
Implements WF-90
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_md_to_html now converts [text](url) to <a> tags and auto-links bare URLs.
Also instructs LLM to use markdown links instead of raw URLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
format_context() was only passing metadata (title, date, link) to the LLM,
so the bot could not answer content questions. Now passes full OCR text.
Also removes auto-rename for DMs (Element reuses single DM room per user pair).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Matrix reuses a single DM room per user pair, so 'new' DMs jump
back to the old thread. Now the bot re-renames the room if >5min
has passed since the last rename, reflecting the new topic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embedding generation on CPU (bge-m3) takes ~3s, plus network latency
can exceed 5s causing silent failures and empty results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DM rooms (1:1 with bot) now auto-rename after first response
without needing !ai auto-rename on
- Group rooms still require explicit opt-in
- Skip rename if room already has a meaningful name
- Improved prompt: emoji prefix, 3-5 words, same language as chat
(inspired by Open WebUI title generation)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a user asks "Wer ist Mieter in diesem Haus?" after discussing
a specific house, the raw message lacks context for RAG search.
Now uses a quick LLM call to resolve pronouns/references before
searching WildFiles (e.g. "diesem Haus" -> "Mieter Haus Coburg").
Also moved history fetch before RAG search so context is available.
Refs: WF-90
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
source_url is now a top-level field on DocumentChunk, not nested
in metadata. Fall back to metadata for backwards compatibility.
Refs: WF-90
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add !ai auto-rename on/off command to auto-name rooms based on conversation topic
- Persist auto-rename setting via room state event (ai.agiliton.auto_rename)
- Generate short title via LLM after first AI response, set as m.room.name
- Load persisted model and auto-rename settings lazily from room state
- Strengthen system prompt: prohibit asking about document storage, file locations
- Fix bot suggesting !ai commands and admin contact to users
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace room.timeline (non-existent in nio) with client.room_messages() API
- Add markdown-to-HTML conversion for formatted Matrix messages
- Route in-room verification events from both UnknownEvent and RoomMessageUnknown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element withholds megolm keys from unverified devices. Implements
the full in-room m.key.verification.* protocol so Element Verify works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extends bot.py with text message handling:
- RoomMessageText callback with @mention detection
- LLM responses via LiteLLM (OpenAI-compatible)
- WildFiles document search (DocumentRAG class)
- Per-room model selection via room state events
- Commands: !ai help/models/set-model/search
- Typing indicators during AI response generation
- 30s staleness check to avoid replaying history
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for voice agent not responding to speech:
1. Agent name: add --agent-name matrix-ai to CLI (was empty, dispatch couldnt match)
2. Move dispatch from on_invite to on_unknown call handler (dispatch when call starts, not on room join)
3. Use LiveKit room name from foci_preferred instead of raw Matrix room ID
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Agent disconnects custom room when all real participants leave
(prevents zombie participants blocking auto-dispatch)
- Bot sends m.call.member state event on call detection
(Element Call shows bot as joined)
- Use RoomInputOptions(participant_identity=...) to target real user
audio input (framework agent-AJ_xxx participant was confusing RoomIO)
- Removed incorrect bot dispatch (Matrix room ID != LiveKit room name)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Save user_id/device_id/access_token to crypto store on first login
- restore_login() on subsequent starts (no new device each restart)
- Enables proper Olm session persistence across restarts
CF-1147
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bot now trusts all room member devices on each sync, enabling
Megolm key exchange. Logs undecryptable events for debugging.
CF-1147
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- matrix-nio[e2e] with libolm for Megolm encryption
- Persistent crypto store volume for key persistence
- Auto-accept key verification (SAS)
- Upload device keys on first login
CF-1147
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
