Voice bot can now see the users camera or screen share when asked.
Captures a single frame, encodes as JPEG, sends to Sonnet vision
with full context (transcript + document). Triggered by phrases like
schau mal, siehst du das, can you see this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generalize PDF-only voice context to support all document types:
- Rename _room_pdf_context → _room_document_context (list-based, 5 cap)
- Handle .docx (python-docx), .txt, .md, .csv, .json, .xml, .html, .yaml, .log
- Store AI image descriptions for voice context
- Multi-document context building with type labels and per-type truncation
- _respond_with_ai now returns reply text for caller use
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Register RoomMessageFile callback, filter for application/pdf
- Extract text from PDFs using pymupdf (fitz)
- Send extracted text as context to LLM for summarization/Q&A
- Truncate at 50k chars to avoid token limits
- Add pymupdf to requirements.txt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extends bot.py with text message handling:
- RoomMessageText callback with @mention detection
- LLM responses via LiteLLM (OpenAI-compatible)
- WildFiles document search (DocumentRAG class)
- Per-room model selection via room state events
- Commands: !ai help/models/set-model/search
- Typing indicators during AI response generation
- 30s staleness check to avoid replaying history
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- matrix-nio[e2e] with libolm for Megolm encryption
- Persistent crypto store volume for key persistence
- Auto-accept key verification (SAS)
- Upload device keys on first login
CF-1147
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- livekit-plugins-silero 1.4.x (not 0.25)
- livekit/livekit-api 1.x (not 0.x)
- Use livekit-plugins-elevenlabs directly instead of routing through LiteLLM
CF-1147
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
