- Add LLM prompt rule to ignore *Störgeräusche* etc. annotations
instead of overriding stt_node (which broke VAD pipeline)
- Switch voice to vmVmHDKBkkCgbLVIOJRb per user preference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Threshold 0.60 too strict, user speech consistently not detected.
Back to default 0.50 with min_speech_duration=0.2 as noise guard.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace George (British EN) with Jack Marlowe (Gng1FdSGZlhs6jKgzAxL),
the only native German voice in the library. Fixes garbled number/date
pronunciation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
George (British) voice mangles German digit strings. Force LLM to
write all numbers as German words so TTS pronounces them correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
0.75 too strict, user voice not detected. 0.60 with min_speech_duration=0.2
should balance noise rejection vs speech detection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pcm_24000 caused silent playback through livekit. Reverting to
plugin default encoding which is known working.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch from mp3_22050_32 (default) to lossless PCM 24kHz for cleaner
voice output. Add language=de for German text normalization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scribe_v2_realtime annotates background audio as *Störgeräusche*,
*Fernsehgeräusche* etc. Override stt_node to drop these so the LLM
only receives actual speech transcripts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Plugin requires explicit aiohttp session; livekit http_context not available
in this job setup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Query user memories at call start and inject into agent system prompt
- Extract new facts after each exchange using claude-haiku via LiteLLM
- Add Brave Search tool (@function_tool) for current data queries
- Pass memory client and caller_user_id through VoiceSession constructor
- Pre-compute 8 HMAC-ratcheted EC keys for reliable E2EE decryption
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bot.py: track active callers per room; only stop session when last
caller leaves (fixes premature cancellation when Playwright browser
hangs up while real app is still in call)
- voice.py: pre-compute 8 HMAC-ratcheted keys from EC's base key so
decryption works immediately without waiting ~30s for Matrix to
deliver EC's key-rotation event (root cause of user→bot silence)
- voice.py: fix set_key() argument order (identity, key, index) at all
call sites — was (identity, index, key) causing TypeError
- voice.py: add audio frame monitor (AUDIO_FLOW) and mute/unmute event
handlers for diagnostics
- voice.py: update livekit-agents 1.4.2 event names: user_state_changed,
user_input_transcribed, conversation_item_added
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: C++ set_key() only applies HKDF when impl_->GetKey(pid) returns a valid
handler, which requires the frame cryptor for that participant to be initialized.
Frame cryptors are created at track subscription time, not at connect time.
Calling set_key(caller_identity, key) immediately after connect() skips HKDF
derivation (impl_->GetKey returns null) → raw key stored → DEC_FAILED.
Fix: move caller key setting to on_track_subscribed where frame cryptor definitely exists.
Also update on_encryption_key to use set_key() for key rotation updates.
If VAD triggers → EC audio reaches pipeline without decryption (plaintext or format issue).
If VAD silent → E2EE encryption on EC side but key/format mismatch on our side.
Note: bot greeting will be unencrypted so EC may not hear it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diagnostic: if Rust FFI HKDF produces different result than EC JS HKDF,
set_key(caller) would always fail (DEC_FAILED). Test: pre-derive AES key
in Python matching livekit-client-sdk-js params (SHA-256, salt=LKFrameEncryptionKey,
info=128-zeros, 16-byte output), pass to set_shared_key() which stores raw (no KDF).
If user→bot decryption now works, root cause = Rust HKDF mismatch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The confirmed-working Feb 21 commit (e3ede3f) used:
- kp.set_shared_key(caller_key) as fallback for incoming audio decryption
- failure_tolerance=10 (not -1) so DEC_FAILED state changes are visible
Per-participant kp.set_key() alone is insufficient — the patched Rust FFI
appears to fall back to shared_key for incoming track decryption.
failure_tolerance=-1 was masking the DEC_FAILED state making diagnosis hard.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two race conditions when bot joins first (remote=0):
1. Key arrives before participant joins LK → on_participant_connected now applies stored keys
2. Key arrives after session start → on_encryption_key now sets key for all remote_participants by LK identity
Fixes identity mismatch between Matrix device_id (from key event) and LK participant identity.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sharing one session between ElevenLabs STT (WebSocket) and TTS (HTTP)
can cause connection conflicts. Use dedicated sessions for each.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add VOICE_TIMEZONE env var (default: Europe/Berlin) for local time
- Bot knows exact date/time at call start via _build_voice_prompt()
- Respond in user language (DE or EN) instead of always German
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove setLevel(DEBUG) for livekit.agents/plugins (added for diagnostics)
- Remove periodic E2EE cryptor/participant state poll loop (no longer needed)
- Remove pipecat-poc/pipeline.py (POC never deployed, LiveKit approach confirmed)
E2EE bidirectional voice confirmed working in MAT-36.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: caller track subscribed during 2s rotation wait creates a
frame cryptor with no key → DEC_FAILED state → all incoming frames dropped.
Setting the key after the wait doesn't recover the cryptor.
Fix: set bot + caller keys immediately after lk_room.connect(), using
the Matrix-provided caller identity. The post-rotation and post-find-remote
key updates remain as belt+suspenders.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds REMOTE_PARTICIPANT log every 10s to confirm caller is present
and tracks are subscribed during E2EE decryption diagnosis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 2 diagnostics: caller audio arrives immediately; setting the key
earlier (2s vs 10s) avoids dropping initial frames. E2EE_CRYPTOR log
now fires every 10s (was 30s) to confirm decryption state for incoming
caller audio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LiveKit JS SDK deriveKeys(): info=new ArrayBuffer(128) (128 zero bytes, NOT identity), output=16 bytes AES-128.
Previous code used identity as info and 32-byte output - both wrong, caused silence in both directions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rust FFI's KDF_HKDF path for incoming decryption may use wrong parameters.
Pre-derive HKDF(base_key, salt="LKFrameEncryptionKey", info=identity) in Python
and pass derived key with KDF_NONE so Rust FFI uses it directly as frame key.
Matches EC's MatrixKeyProvider: ratchetWindowSize=10, keyringSize=256.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
io.element.call.encryption_keys events are Megolm-encrypted in this room
(appear as m.room.encrypted). The HTTP fetch cannot decrypt them — only
the nio sync client can via Olm/Megolm decryption.
Change the post-connect rotation poll to check self._caller_all_keys
directly (updated by on_encryption_key() via nio sync) instead of calling
_fetch_encryption_key_http() which always returns nothing in encrypted rooms.
Also extends wait to 10s and adds progress logging every 2s.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Element Call rotates its encryption key when a new participant joins the
LiveKit room. Previously the bot fetched only the pre-join key and set it
at index 0, while EC was already encrypting with the rotated key (index 1).
Changes:
- After connecting to LiveKit, poll the Matrix timeline up to 5s (10×0.5s)
to detect the post-join key rotation
- Set ALL known caller key indices (not just 0) so the Rust FFI cryptor
has the correct key regardless of which index EC is currently using
- Also set via caller_identity (belt+suspenders) if different from LK identity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shared-key mode uses HKDF with empty info, but Element Call JS uses
participant identity as HKDF info. Per-participant mode (set_key with
identity) matches EC's derivation.
Previous per-participant attempt (b65d043) failed because key rotation
(index 0→1 when bot joins) wasn't handled. Now on_encryption_key calls
set_key(caller_id, key, index) on rotation, so the bot stays in sync.
Changes:
- _build_e2ee_options(): remove caller_key param, shared_key=b"" (per-participant mode)
- _run(): set_key(remote_identity, caller_key, 0) for incoming decryption
- on_encryption_key: only set_key() on rotation (no set_shared_key)
Add export_shared_key() verification after connect to confirm key
is stored. Also set per-participant key for caller (belt+suspenders)
so both shared-key and per-participant decryption paths are active.
When Element Call sees the bot join, it rotates its encryption key
(index 0 → 1). The on_encryption_key callback was calling set_key()
(per-participant) which has no effect in shared-key mode. Switch to
set_shared_key() so the shared-key decryption path stays current when
the caller rotates keys.
Per-participant set_key() for remote identities doesn't work for
incoming decryption in this Rust FFI build (set_shared_key() after
connect is also ignored in per-participant mode).
Solution: initialize with caller_key as shared_key (true shared-key
mode) so the Rust FFI uses it for incoming decryption. Then override
outgoing encryption via set_key(bot_identity, bot_key) after connect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rust FFI may not use per-participant key for remote participant
decryption in all code paths. Set the caller key as both per-participant
AND shared key so either path works for incoming frame decryption.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call may rotate encryption keys to index > 0. Previously we
always called set_key(identity, key, 0) regardless of the actual index,
causing decryption to fail when the active key was at a non-zero index.
- _fetch_encryption_key_http: collect all {index->key} pairs from event
- _run: set each caller key at its correct index
- on_encryption_key: handle multiple indices, remove first-key-only gate
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NoneType causes TypeError in patched room.py proto assignment.
Empty bytes is falsy so shared_key is not set in proto,
initializing key provider in per-participant mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shared_key locks provider in shared-key mode, making set_key()
ineffective for per-participant decryption. Remove shared_key so
SDK initializes in per-participant mode. Also: failure_tolerance=-1
to prevent premature track closure on decrypt failures,
ratchet_window_size=16 to match Element Call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
E2EE key setup may briefly appear as participant disconnect.
Keep session alive to allow audio to flow once keys are settled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call uses per-participant keys, not shared key mode.
Bot now generates its own key, publishes it, and sets both
keys via key_provider.set_key() after connecting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Element Call now rejects unencrypted audio. Use caller's key
as shared_key so both sides encrypt/decrypt with the same key.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
