Cut daemon memory: share the ASR model, bound SSE queues, cap segment length - #20
Open
ktat wants to merge 6 commits into
Open
Cut daemon memory: share the ASR model, bound SSE queues, cap segment length#20ktat wants to merge 6 commits into
ktat wants to merge 6 commits into
Conversation
…d memory growth Each dashboard SSE connection got an unbounded queue.Queue, and _broadcast() swallowed every put_nowait failure with `except Exception: pass`, so an unbounded queue never raised and nothing was ever dropped. Combined with _serve_sse()'s socket write having no timeout, a peer that stops reading without closing (suspended laptop, frozen tab, dropped network path) left the writer blocked forever while the producer kept enqueuing — this is how a daemon reached 2GB RSS after 2.5 days. - FileWatcher.add_client() now bounds the queue at 200 slots (small JSON events; trivial worst-case memory even full). - _broadcast() drops a client whose queue is full instead of silently discarding forever (the dashboard re-fetches state on reconnect, so a client that misses events goes stale, not corrupted) and now logs any genuinely unexpected put_nowait exception instead of hiding it. - _serve_sse() sets a 20s write timeout on the connection so a wedged peer surfaces as a socket.timeout (an OSError, already handled by the existing except clause) instead of blocking the handler thread forever. Verified with tests/test_sse_queue_bound.py (unit-level, 12/12) and tests/test_sse_stuck_client.py (end-to-end over real sockets, 10/10): a peer that stops reading gets evicted in ~20.5s (matching the write timeout), while a normally-reading client survives 75s including a real 15s keepalive without disruption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main and interim each loaded their own copy of the K2 fp32 weights (~600MB each) even when configured identically. Add a refcounted module-level cache in _daemon_transcriber.py keyed on (backend, model_id, device, precision) so identically-configured Transcriber instances share one model object; differing configs (e.g. main=reazonspeech-k2 / interim=kotoba-whisper) still load separately. _daemon_recorder_transcribe.py releases the old interim Transcriber's reference before replacing it on a settings change, so refcounts don't leak across interim model swaps. Measured on an isolated daemon (port 8799, own SHADOW_CLERK_DATA_DIR), settled RSS of the actual daemon child process after both models report loaded: before (git worktree at 03aa107): 1,681,580 kB RSS after (this change): 1,065,656 kB RSS delta: 615,924 kB (~601 MiB) The ~750MB figure floated earlier was a guess; on-disk fp32 weights (encoder+decoder+joiner) total ~586 MiB, matching the measured delta closely, so ~600MB/instance is the real number. Thread safety: reazonspeech.k2.asr.transcribe() calls model.create_stream() per invocation and keeps all decode state in that stream object; the shared sherpa-onnx OfflineRecognizer itself is never mutated. Verified with 360 concurrent transcribe() calls across 6 Transcriber instances sharing one real K2 model (4 real speech clips, distinct per thread), all outputs byte-identical to solo-decode baselines, with a 1.55x wall-clock speedup confirming the GIL is released during decode (real concurrency, not serialized). Also verified end-to-end (audio played through an isolated speaker device, not the user's headset) that both the main transcript file and the interim SSE stream produce correct text with the shared model. tests/test_model_sharing.py (20/20) covers: sharing on matching config, no sharing across device/precision, mixed main/interim backends not forced together, reload_model() on one instance not pulling the model out from under the other, and refcounts not leaking across interim Transcriber replacement.
The ASR inference arena scales with the longest segment fed to it. Verified in-process with the production-default backend (faster-whisper "small", int8, cpu): feeding a realistic mixed-duration session, resident memory settles around 1266-1276MB when segments are capped at 30s vs. 1104-1111MB capped at 20s — a reproducible ~160-170MB steady-state reduction (smaller than the ~292MB predicted from isolated 10s/30s arena deltas, but a real and repeatable reclaim). VmHWM is dominated by a model-load-time transient spike and doesn't reflect this; VmRSS after settling does. Confirmed the forced split in VADSegmenter.process_frame() (_daemon_vad.py) is clean: _finalize_segment() concatenates and returns the full buffered audio and resets state, so no audio is dropped. If speech continues past the cap, frames keep appending to the next segment immediately (the not-in_speech/is_speech branch appends too), so there's no stranded fragment or gap — just a hard split into two segments. Also updates SPEC.md's sequence diagram, which cited the old 30-second figure.
Grepped src/ and tests/: the queue.Queue() assignment at construction time was its only occurrence anywhere in the codebase. Nothing reads from or writes to it.
_load_model_locked() released the old shared model reference before attempting to build the replacement. If the new load raised while self.model still pointed at the shared object, that instance kept decoding with a model no longer counted toward the refcount — an untracked holder. A later acquire of the same key would then load a second copy of a model that can be hundreds of MB, with both live. Not reachable today: reload_model() and ensure_model_for_language() already null self.model before calling in, and the only other caller uses a freshly constructed instance. But _load_model_locked() itself should hold the invariant regardless of call site. Ownership rule (documented as a comment at the release site): self.model is non-None only while it holds a counted reference — via self._shared_key in _MODEL_CACHE, or as a Whisper instance owned outright. Before swapping to a different model, self.model is set to None and the old reference released *before* attempting the new load, so a failed load just leaves self.model as None (the next transcribe() call reloads lazily) instead of resurrecting an untracked stale reference. Added tests/test_model_sharing.py cases 21-24: force WhisperModel's constructor to raise during a direct load_model() call on an already-loaded instance and confirm self.model ends up None (not the stale model) and a subsequent acquire of the freed key loads exactly once. Verified this fails against the pre-fix ordering (self.model ends up holding the old, now-unaccounted model) and passes after.
The earlier commit's comment cited a memory-investigation estimate (+165MB@10s / +457MB@30s) as if it were what was verified, but the in-process measurement backing it used faster-whisper "small"/int8/cpu — DEFAULT_CONFIG's fallback, not this deployment's actual config. The user's ~/.local/share/shadow-clerk/config.yaml sets: japanese_asr_model: reazonspeech-k2 interim_japanese_asr_model: reazonspeech-k2 reazonspeech_precision: fp32 and the running daemon confirms both main and interim load K2. That mismatch is exactly why the earlier faster-whisper numbers (~160-170MB session-level reduction) diverged from the estimate this was meant to verify — the estimate was measured against K2 all along. Re-measured in-process against the real topology: main + interim Transcriber instances both configured for reazonspeech-k2/fp32/cpu, loaded through the shared-model cache from commit 45333fe (asserted main.model is interim.model to confirm sharing held), fed a realistic mixed-duration session (20 segments) plus concurrent interim segments capped at INTERIM_MAX_DURATION. Isolated single-call marginal cost (fresh process, one transcribe() right after load): 10s +81MB, 30s +301MB (delta ~220MB) — same order of magnitude and shape as the original estimate, but smaller in absolute terms, most likely because that estimate was measured on the arena directly rather than via OS-level RSS, and/or before model sharing reduced fixed per-instance overhead. Session steady-state (3 seeds): cap=30s settles around 1266-1276MB, cap=20s around 1153-1157MB — a reproducible ~113-119MB (~115MB) reduction. Smaller than the isolated single-call delta because once a large segment grows the process's heap, later same-or-smaller segments don't shrink it back down (the ceiling sticks at the largest segment ever seen), so the two cap conditions' baselines aren't directly comparable to two isolated single calls. Comment now reports both numbers with their real basis instead of citing the original estimate as verified fact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
デーモンのメモリ消費が 2 日半で 2098MB に達していた件の対処です。Fable による調査でリークではなく同じ ASR モデルの二重ロードがほぼ全てと判明したため、その解消を中心に 3 つ手を入れました。
調査結果（着手前）
内訳の実測は合計 ~2190MB で、実測値 2212MB と 1% 差で一致しました。うち interim 側の K2 セッションが main と同一モデルの二重ロードでした。
変更内容
1. K2 モデルの共有（実測 ~601 MiB 削減）
main と interim が同じ ReazonSpeech K2 を別々にロードしていました。
(backend, model_id, device, precision)をキーとするプロセス内キャッシュで 1 つのオブジェクトを共有します。設定が異なる場合（japanese_asr_modelとinterim_japanese_asr_modelが別モデル）は従来通り別々にロードします。同時デコードの安全性を検証したうえでロックは追加していません。 sherpa-onnx v1.12.28 の C++ ソースを読み、
DecodeStreamsがconstでローカルにしか書き込まないこと、OfflineStreamが feat config を値渡しで持つこと、経路上にmutableメンバが無いこと、pybind 側にgil_scoped_releaseがあることを確認しました。そのうえで実音声 4 本を使い、6 インスタンスから 360 回の同時デコードを単独デコードとトークン + タイムスタンプ粒度で突き合わせて完全一致を確認しています（2.86 倍の速度向上も観測、GIL が実際に解放されている証拠）。つまり interim が main のデコード待ちになる事態は起きません。
2. SSE クライアントキューの無限成長を停止
ブラウザが RST を出さずに消えると（サスペンド、タブの凍結、経路の無言切断）
writeが永久ブロックし、消費が止まったままイベントが積まれ続けていました。finallyのremove_clientはループを抜けないと走らないため回収もされません。毎秒 1 件のlevelイベントを追加したことで、この経路が現実的な脅威になっていました。except Exception: passの握り潰しをやめ、想定内の満杯と想定外の例外を区別してログに出す実測で停滞クライアントは 20.5 秒で追い出され、正常なクライアントは 75 秒生存します。
3. 最大セグメント長 30 秒 → 20 秒（実測 ~115MB 削減）
ASR の推論アリーナは最長セグメントにスケールし、一度伸びたヒープは以降縮みません。上限がセッション中に見た最大セグメントで固定されるため、上限を下げるとフットプリントの天井が下がります。
トレードオフとして、20 秒を超えて続く発話は強制分割されます。分割が音声を落とさないことは、境界をまたぐフレームに識別子を埋めて検証済みです（重複なし・欠落なし・境界が完全に連続）。
4. 潜在バグの解消と死にコード削除
vad_queue削除: 定義以外の参照がツリー全体でゼロでした検証
新規テスト 3 本を含む全スイートが通ります。
test_model_sharing.pytest_sse_queue_bound.pytest_sse_stuck_client.pytest_level_event.pytest_audio_level.pytest_backend_stall.pytest_audio_capture_watchdog.py重複コード検査 10.00/10、mypy 101 件（
mainと同数）、全ファイル 700 行以内。途中で訂正した計測ミス
セグメント長の効果を最初に測ったとき、
DEFAULT_CONFIG（whisper）を読んで計測しており、実際の構成（~/.local/share/shadow-clerk/config.yamlの K2）と違うバックエンドでした。推定値と噛み合わなかった時点で「推定が甘かった」と結論しかけましたが、その食い違い自体が手掛かりでした。設定ファイルを実際に読んで測り直した数字が上記の ~115MB です。見送ったもの
reazonspeech_precision: int8（推定 ~900MB 削減）は精度とのトレードオフがあるため未適用です。設定 1 行なので、A/B したうえで判断できます。反映
editable install のため、デーモンを再起動すれば載ります。合計の削減は再起動後の実測で確認してください。