io.element.call.encryption_keys events are Megolm-encrypted in this room
(appear as m.room.encrypted). The HTTP fetch cannot decrypt them — only
the nio sync client can via Olm/Megolm decryption.
Change the post-connect rotation poll to check self._caller_all_keys
directly (updated by on_encryption_key() via nio sync) instead of calling
_fetch_encryption_key_http() which always returns nothing in encrypted rooms.
Also extends wait to 10s and adds progress logging every 2s.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Element Call rotates its encryption key when a new participant joins the
LiveKit room. Previously the bot fetched only the pre-join key and set it
at index 0, while EC was already encrypting with the rotated key (index 1).
Changes:
- After connecting to LiveKit, poll the Matrix timeline up to 5s (10×0.5s)
to detect the post-join key rotation
- Set ALL known caller key indices (not just 0) so the Rust FFI cryptor
has the correct key regardless of which index EC is currently using
- Also set via caller_identity (belt+suspenders) if different from LK identity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shared-key mode uses HKDF with empty info, but Element Call JS uses
participant identity as HKDF info. Per-participant mode (set_key with
identity) matches EC's derivation.
Previous per-participant attempt (b65d043) failed because key rotation
(index 0→1 when bot joins) wasn't handled. Now on_encryption_key calls
set_key(caller_id, key, index) on rotation, so the bot stays in sync.
Changes:
- _build_e2ee_options(): remove caller_key param, shared_key=b"" (per-participant mode)
- _run(): set_key(remote_identity, caller_key, 0) for incoming decryption
- on_encryption_key: only set_key() on rotation (no set_shared_key)
Add export_shared_key() verification after connect to confirm key
is stored. Also set per-participant key for caller (belt+suspenders)
so both shared-key and per-participant decryption paths are active.
When Element Call sees the bot join, it rotates its encryption key
(index 0 → 1). The on_encryption_key callback was calling set_key()
(per-participant) which has no effect in shared-key mode. Switch to
set_shared_key() so the shared-key decryption path stays current when
the caller rotates keys.
Per-participant set_key() for remote identities doesn't work for
incoming decryption in this Rust FFI build (set_shared_key() after
connect is also ignored in per-participant mode).
Solution: initialize with caller_key as shared_key (true shared-key
mode) so the Rust FFI uses it for incoming decryption. Then override
outgoing encryption via set_key(bot_identity, bot_key) after connect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rust FFI may not use per-participant key for remote participant
decryption in all code paths. Set the caller key as both per-participant
AND shared key so either path works for incoming frame decryption.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call may rotate encryption keys to index > 0. Previously we
always called set_key(identity, key, 0) regardless of the actual index,
causing decryption to fail when the active key was at a non-zero index.
- _fetch_encryption_key_http: collect all {index->key} pairs from event
- _run: set each caller key at its correct index
- on_encryption_key: handle multiple indices, remove first-key-only gate
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NoneType causes TypeError in patched room.py proto assignment.
Empty bytes is falsy so shared_key is not set in proto,
initializing key provider in per-participant mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shared_key locks provider in shared-key mode, making set_key()
ineffective for per-participant decryption. Remove shared_key so
SDK initializes in per-participant mode. Also: failure_tolerance=-1
to prevent premature track closure on decrypt failures,
ratchet_window_size=16 to match Element Call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
E2EE key setup may briefly appear as participant disconnect.
Keep session alive to allow audio to flow once keys are settled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, not shared key mode.
Bot now generates its own key, publishes it, and sets both
keys via key_provider.set_key() after connecting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call now rejects unencrypted audio. Use caller's key
as shared_key so both sides encrypt/decrypt with the same key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, LiveKit Python SDK shared key mode
cannot properly decrypt. Reverting to working state (no LiveKit E2EE).
Bot still publishes keys so Element Call shows encryption indicator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both bot and caller must use the same key in shared key mode.
Bot now reuses caller's key and publishes it back, instead of
generating a separate bot key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Per-participant set_key alone with empty shared_key caused silent incoming audio.
Now connects with caller key as shared_key, then overlays per-participant keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bot tells which model it uses when asked
- Injects current UTC datetime into prompt
- Responds in users language instead of always German
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys via MatrixKeyProvider.onSetEncryptionKey(),
not shared key mode. This was causing silence with E2EE enabled.
- Set bot's own key and caller's key separately via e2ee_manager.key_provider.set_key()
- Live-update caller key when received after connect
- Fallback to set_shared_key if per-participant API unavailable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Audio pipeline confirmed working without E2EE. E2EE key derivation
mismatch with Element Call needs separate investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bot now publishes the same key as the caller so both sides can decrypt.
Falls back to no-encryption if no caller key received.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip bot own encryption_keys events in on_unknown handler
- Always pass valid RoomOptions to AgentSession.start()
- Wait up to 10s for remote participant to connect before starting pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call distributes encryption keys as timeline events, not room
state events. Changed bot to publish keys via room_send and fetch from
/messages endpoint instead of /state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All participants must use the SAME shared key. Bot was generating
its own key which couldn't decrypt user's audio. Now:
1. Fetch caller's key from room state via HTTP API
2. Fall back to waiting for key via sync handler
3. Publish the SAME key back (not a new one)
4. Only connect with E2EE if key available
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call encrypts media by default. Bot must:
1. Generate its own 32-byte E2EE key
2. Publish it to room state (io.element.call.encryption_keys)
3. Connect to LiveKit with HKDF E2EE enabled
4. Use caller's key when received, own key as fallback
This fixes: Nicht verschlüsselt warning, silent audio (encrypted
frames couldn't be decoded by VAD/STT)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pass participant_identity via RoomOptions so AgentSession knows
which audio track to consume (was silently ignoring user audio)
- Add USER_SPEECH and AGENT_SPEECH event handlers for debugging
- Simplify greeting to exact text to prevent hallucination
- Use httpx for room state scan (nio API was unreliable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Reorder: send call member event BEFORE creating VoiceSession
- Store VoiceSession BEFORE start so sync handler can forward keys
- Increase E2EE key wait from 3s to 10s
- Add INFO-level logging for key lookup + room state scan via HTTP API
- Tighten voice system prompt to prevent long rambling greetings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix state_key format: try @user:domain:DEVICE_ID (Element Call format),
then @user:domain, then scan all room state as fallback
- Publish bot E2EE key to room so Element shows encrypted status
- Extract caller device_id from call member event content
- Also fix pipecat-poc pipeline with context aggregators (CF-1579)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rust:latest produces FFI needing CXXABI_1.3.15 (GCC 14 libstdc++).
GCC 14 libstdc++ needs GLIBC 2.38. Bookworm only has 2.36.
Trixie has GLIBC 2.38+ — fixes the CXXABI_1.3.15 runtime error.
Also reverts to rust:latest since bookworm GCC 12 cant compile webrtc C++20.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rust:latest links against GLIBC_2.38 libstdc++ which is incompatible with bookworm.
rust:bookworm (1.93.1) produces FFI binary compatible with bookworm libstdc++.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stop VoiceSession when call leave event received
- Copy libstdc++ from rust build stage to fix CXXABI_1.3.15 mismatch
- Read caller encryption key from room state before starting VoiceSession
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
voice.py runs in bot container, not agent container.
- Wait 3s for encryption key before connecting
- Build E2EE options with HKDF when key received
- Bot container now uses patched Dockerfile (needs FFI)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
