- Add VOICE_TIMEZONE env var (default: Europe/Berlin) for local time
- Bot knows exact date/time at call start via _build_voice_prompt()
- Respond in user language (DE or EN) instead of always German
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove setLevel(DEBUG) for livekit.agents/plugins (added for diagnostics)
- Remove periodic E2EE cryptor/participant state poll loop (no longer needed)
- Remove pipecat-poc/pipeline.py (POC never deployed, LiveKit approach confirmed)
E2EE bidirectional voice confirmed working in MAT-36.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: caller track subscribed during 2s rotation wait creates a
frame cryptor with no key → DEC_FAILED state → all incoming frames dropped.
Setting the key after the wait doesn't recover the cryptor.
Fix: set bot + caller keys immediately after lk_room.connect(), using
the Matrix-provided caller identity. The post-rotation and post-find-remote
key updates remain as belt+suspenders.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds REMOTE_PARTICIPANT log every 10s to confirm caller is present
and tracks are subscribed during E2EE decryption diagnosis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 2 diagnostics: caller audio arrives immediately; setting the key
earlier (2s vs 10s) avoids dropping initial frames. E2EE_CRYPTOR log
now fires every 10s (was 30s) to confirm decryption state for incoming
caller audio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LiveKit JS SDK deriveKeys(): info=new ArrayBuffer(128) (128 zero bytes, NOT identity), output=16 bytes AES-128.
Previous code used identity as info and 32-byte output - both wrong, caused silence in both directions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rust FFI's KDF_HKDF path for incoming decryption may use wrong parameters.
Pre-derive HKDF(base_key, salt="LKFrameEncryptionKey", info=identity) in Python
and pass derived key with KDF_NONE so Rust FFI uses it directly as frame key.
Matches EC's MatrixKeyProvider: ratchetWindowSize=10, keyringSize=256.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
io.element.call.encryption_keys events are Megolm-encrypted in this room
(appear as m.room.encrypted). The HTTP fetch cannot decrypt them — only
the nio sync client can via Olm/Megolm decryption.
Change the post-connect rotation poll to check self._caller_all_keys
directly (updated by on_encryption_key() via nio sync) instead of calling
_fetch_encryption_key_http() which always returns nothing in encrypted rooms.
Also extends wait to 10s and adds progress logging every 2s.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Element Call rotates its encryption key when a new participant joins the
LiveKit room. Previously the bot fetched only the pre-join key and set it
at index 0, while EC was already encrypting with the rotated key (index 1).
Changes:
- After connecting to LiveKit, poll the Matrix timeline up to 5s (10×0.5s)
to detect the post-join key rotation
- Set ALL known caller key indices (not just 0) so the Rust FFI cryptor
has the correct key regardless of which index EC is currently using
- Also set via caller_identity (belt+suspenders) if different from LK identity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shared-key mode uses HKDF with empty info, but Element Call JS uses
participant identity as HKDF info. Per-participant mode (set_key with
identity) matches EC's derivation.
Previous per-participant attempt (b65d043) failed because key rotation
(index 0→1 when bot joins) wasn't handled. Now on_encryption_key calls
set_key(caller_id, key, index) on rotation, so the bot stays in sync.
Changes:
- _build_e2ee_options(): remove caller_key param, shared_key=b"" (per-participant mode)
- _run(): set_key(remote_identity, caller_key, 0) for incoming decryption
- on_encryption_key: only set_key() on rotation (no set_shared_key)
Add export_shared_key() verification after connect to confirm key
is stored. Also set per-participant key for caller (belt+suspenders)
so both shared-key and per-participant decryption paths are active.
When Element Call sees the bot join, it rotates its encryption key
(index 0 → 1). The on_encryption_key callback was calling set_key()
(per-participant) which has no effect in shared-key mode. Switch to
set_shared_key() so the shared-key decryption path stays current when
the caller rotates keys.
Per-participant set_key() for remote identities doesn't work for
incoming decryption in this Rust FFI build (set_shared_key() after
connect is also ignored in per-participant mode).
Solution: initialize with caller_key as shared_key (true shared-key
mode) so the Rust FFI uses it for incoming decryption. Then override
outgoing encryption via set_key(bot_identity, bot_key) after connect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rust FFI may not use per-participant key for remote participant
decryption in all code paths. Set the caller key as both per-participant
AND shared key so either path works for incoming frame decryption.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call may rotate encryption keys to index > 0. Previously we
always called set_key(identity, key, 0) regardless of the actual index,
causing decryption to fail when the active key was at a non-zero index.
- _fetch_encryption_key_http: collect all {index->key} pairs from event
- _run: set each caller key at its correct index
- on_encryption_key: handle multiple indices, remove first-key-only gate
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NoneType causes TypeError in patched room.py proto assignment.
Empty bytes is falsy so shared_key is not set in proto,
initializing key provider in per-participant mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shared_key locks provider in shared-key mode, making set_key()
ineffective for per-participant decryption. Remove shared_key so
SDK initializes in per-participant mode. Also: failure_tolerance=-1
to prevent premature track closure on decrypt failures,
ratchet_window_size=16 to match Element Call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
E2EE key setup may briefly appear as participant disconnect.
Keep session alive to allow audio to flow once keys are settled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, not shared key mode.
Bot now generates its own key, publishes it, and sets both
keys via key_provider.set_key() after connecting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call now rejects unencrypted audio. Use caller's key
as shared_key so both sides encrypt/decrypt with the same key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, LiveKit Python SDK shared key mode
cannot properly decrypt. Reverting to working state (no LiveKit E2EE).
Bot still publishes keys so Element Call shows encryption indicator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both bot and caller must use the same key in shared key mode.
Bot now reuses caller's key and publishes it back, instead of
generating a separate bot key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Per-participant set_key alone with empty shared_key caused silent incoming audio.
Now connects with caller key as shared_key, then overlays per-participant keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bot tells which model it uses when asked
- Injects current UTC datetime into prompt
- Responds in users language instead of always German
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys via MatrixKeyProvider.onSetEncryptionKey(),
not shared key mode. This was causing silence with E2EE enabled.
- Set bot's own key and caller's key separately via e2ee_manager.key_provider.set_key()
- Live-update caller key when received after connect
- Fallback to set_shared_key if per-participant API unavailable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Audio pipeline confirmed working without E2EE. E2EE key derivation
mismatch with Element Call needs separate investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bot now publishes the same key as the caller so both sides can decrypt.
Falls back to no-encryption if no caller key received.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip bot own encryption_keys events in on_unknown handler
- Always pass valid RoomOptions to AgentSession.start()
- Wait up to 10s for remote participant to connect before starting pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call distributes encryption keys as timeline events, not room
state events. Changed bot to publish keys via room_send and fetch from
/messages endpoint instead of /state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All participants must use the SAME shared key. Bot was generating
its own key which couldn't decrypt user's audio. Now:
1. Fetch caller's key from room state via HTTP API
2. Fall back to waiting for key via sync handler
3. Publish the SAME key back (not a new one)
4. Only connect with E2EE if key available
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call encrypts media by default. Bot must:
1. Generate its own 32-byte E2EE key
2. Publish it to room state (io.element.call.encryption_keys)
3. Connect to LiveKit with HKDF E2EE enabled
4. Use caller's key when received, own key as fallback
This fixes: Nicht verschlüsselt warning, silent audio (encrypted
frames couldn't be decoded by VAD/STT)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pass participant_identity via RoomOptions so AgentSession knows
which audio track to consume (was silently ignoring user audio)
- Add USER_SPEECH and AGENT_SPEECH event handlers for debugging
- Simplify greeting to exact text to prevent hallucination
- Use httpx for room state scan (nio API was unreliable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Reorder: send call member event BEFORE creating VoiceSession
- Store VoiceSession BEFORE start so sync handler can forward keys
- Increase E2EE key wait from 3s to 10s
- Add INFO-level logging for key lookup + room state scan via HTTP API
- Tighten voice system prompt to prevent long rambling greetings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix state_key format: try @user:domain:DEVICE_ID (Element Call format),
then @user:domain, then scan all room state as fallback
- Publish bot E2EE key to room so Element shows encrypted status
- Extract caller device_id from call member event content
- Also fix pipecat-poc pipeline with context aggregators (CF-1579)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
