C-393: Speech-to-Text Backend Service (sherpa-onnx streaming + whisper.cpp batch)

apps/backend/local-stack/.env.example

@@ -11,12 +11,15 @@
 #   text   — llama.cpp server (OpenAI-compatible /v1) on port 11434
 #   image  — stable-diffusion.cpp sd-server on port 8188
 #   voice  — sherpa-onnx Kokoro TTS on port 8089
-#   stt    — sherpa-onnx Moonshine STT on port 8087 (same voice container)
+#   stt    — sherpa-onnx Moonshine STT + whisper.cpp on port 8087 (same voice container)
 #   web    — opt-in web client container on port 5274
 #   ollama / comfyui — advanced alternatives on the same ports (advanced)
 # The model fetcher is profile-scoped: it downloads only the models for the
 # modalities you enable.
-COMPOSE_PROFILES=text,image,voice,stt
+#
+# STT is OFF by default (C-393 AC-7): a microphone-adjacent service must be
+# opt-in. Add `stt` here and set ENABLE_STT=true below to enable it.
+COMPOSE_PROFILES=text,image,voice
 
 # ── Hardware backend ──────────────────────────────────────────────────────
 # One of: cpu, cuda, rocm, vulkan, intel, musa.
@@ -30,6 +33,22 @@ COMPOSE_FILE=compose.yaml:compose.cpu.yaml
 TEXT_MODEL=qwen2.5-1.5b-instruct-q4_k_m.gguf
 IMAGE_MODEL=flux1-schnell-q4_k.gguf
 
+# ── STT model tiers (C-393) ───────────────────────────────────────────────
+# Manifest targetPaths under the models volume; the fetcher downloads ONLY
+# the selected entries plus the Silero VAD model. Tiers:
+#   minimal  (shipped default): stt/sherpa-onnx-moonshine-tiny-en-int8 + stt/whisper-tiny/ggml-tiny.bin
+#   default: stt/sherpa-onnx-moonshine-base-en-int8 + stt/whisper-base/ggml-base.bin
+#   accuracy: stt/sherpa-onnx-moonshine-base-en-int8 + stt/whisper-small/ggml-small.bin
+# STT_STREAM_MODEL=stt/sherpa-onnx-moonshine-tiny-en-int8
+# STT_BATCH_MODEL=stt/whisper-tiny/ggml-tiny.bin
+# STT_STREAM_ENGINE=moonshine
+# STT_BATCH_ENGINE=whisper-cpp
+# STT_ALLOWED_ORIGINS=http://localhost:5274,http://127.0.0.1:5274,tauri://localhost,http://tauri.localhost,https://tauri.localhost
+# STT_VAD_THRESHOLD=0.5
+# STT_VAD_MIN_SPEECH_MS=250
+# STT_VAD_MIN_SILENCE_MS=500
+# STT_VAD_MAX_SPEECH_MS=30000
+
 # ── Licences ─────────────────────────────────────────────────────────────
 # SD 1.5 is CreativeML OpenRAIL-M (use-restricted). Leave empty to skip it;
 # the fetcher prints the licence and skips the download. Accept with the
@@ -45,9 +64,15 @@ AIKAMI_ACCEPT_LICENSES=
 # WEB_PORT=5274
 
 # ── Voice extras ─────────────────────────────────────────────────────────
-# Set true to also start the STT websocket server inside the voice container
-# (required when you enable the `stt` profile).
-ENABLE_STT=true
+# Enable the C-393 STT service inside the voice container (WS streaming on
+# 8087, OpenAI-compatible batch proxy, capabilities + health). Requires ALL
+# THREE of: the `stt` profile above, ENABLE_STT=true, and compose.stt.yaml in
+# COMPOSE_FILE (the base file never publishes the STT port — AC-7).
+#
+#   COMPOSE_PROFILES=text,image,voice,stt
+#   COMPOSE_FILE=compose.yaml:compose.cpu.yaml:compose.stt.yaml
+#   ENABLE_STT=true
+ENABLE_STT=false
 
 # ── Existing model trees (AC-13) ─────────────────────────────────────────
 # Point at a pre-existing models/ directory from the old stack to bind-mount

apps/backend/local-stack/README.md

@@ -129,7 +129,7 @@ edit `.env` directly.
 | `text` | llama.cpp server (`/v1`) | Qwen2.5-1.5B Q4_K_M (CPU default) |
 | `image` | sd-server | FLUX.1-schnell Q4_K |
 | `voice` | sherpa-onnx Kokoro TTS | Kokoro-82M |
-| `stt` | sherpa-onnx Moonshine STT (same container) | Moonshine tiny |
+| `stt` | sherpa-onnx Moonshine streaming + whisper.cpp batch (same container) | Moonshine tiny + whisper tiny + Silero VAD (minimal tier) |
 | `web` | the web client container | — |
 
 The model fetcher is **profile-scoped**: `COMPOSE_PROFILES=text` downloads
@@ -140,6 +140,72 @@ only the text model. Enable what you need:
 COMPOSE_PROFILES=text,image,voice,stt,web
 ```
 
+> ⚠️ **STT is opt-in (C-393 AC-7).** The shipped defaults do **not** start
+> the STT service: `.env.example` lists `COMPOSE_PROFILES=text,image,voice`
+> and `ENABLE_STT=false`. To enable speech-to-text, add `stt` to
+> `COMPOSE_PROFILES`, set `ENABLE_STT=true`, **and** include
+> `compose.stt.yaml` in `COMPOSE_FILE` (the base file never publishes the
+> STT port, so without the override port 8087 stays unbound). A
+> microphone-adjacent service that starts unasked would be a privacy
+> problem.
+
+### Speech-to-text (STT)
+
+The `stt` profile runs two engines inside the same voice container:
+
+| Protocol | Endpoint | Engine | Use case |
+|---|---|---|---|
+| Streaming | `WS 127.0.0.1:8087/v1/stream` | sherpa-onnx Moonshine + Silero VAD | Push-to-talk, hands-free; partial hypotheses while speaking |
+| Batch | `POST 127.0.0.1:8087/v1/audio/transcriptions` | whisper.cpp | Recorded clips, imported audio; OpenAI-compatible |
+| Introspection | `GET 127.0.0.1:8087/v1/capabilities` | — | Engines, models, languages, VAD, `wordTimestamps` |
+| Readiness | `GET 127.0.0.1:8087/health` | — | 200 healthy / 503 naming the missing model file |
+
+The streaming protocol is defined in
+`packages/shared/schemas/src/lib/local_ai/stt.ts` (the wire contract C-359
+codes against):
+
+- **Audio format is fixed**: 16 kHz mono 16-bit PCM (`pcm_s16le`), 32000
+  bytes/sec. Resampling is the client's job; the server rejects anything
+  else with `error: bad-audio-format`.
+- Client → server: `{"type":"start","protocolVersion":1,"audio":{...}}`
+  (JSON text frame — `audio` declares the fixed 16 kHz mono 16-bit PCM
+  format), then binary frames of raw PCM, then `{"type":"stop"}`.
+- Server → client: `ready`, `speech-start`, `partial`*, `final`,
+  `speech-end`, `error`. VAD runs **server-side** — the client never infers
+  endpointing.
+- **Moonshine is English-only.** Requesting another language returns
+  `error: unsupported-language` pointing at the batch endpoint; the service
+  never transcribes non-English audio as garbled English. whisper.cpp
+  covers ~99 languages for batch.
+- The engine behind each protocol is env-selected
+  (`STT_STREAM_ENGINE`, `STT_BATCH_ENGINE`) — the seam for a future
+  licensed CrisperWhisper provider with word-level timestamps.
+
+**Privacy posture**: audio is processed in memory only — it is **never
+written to disk, never logged, and never leaves the machine**. The service
+binds `127.0.0.1` on the host, and websocket connections are rejected when
+the `Origin` header is not on the allowlist (`STT_ALLOWED_ORIGINS`) so a
+random web page cannot open a socket to your local transcription service.
+There is no debug audio dump, not even behind a flag.
+
+**Model tiers** (C-393): the fetcher downloads exactly the selected tier
+plus the Silero VAD model — not every entry of the modality:
+
+| Tier | `STT_STREAM_MODEL` | `STT_BATCH_MODEL` |
+|---|---|---|
+| minimal (shipped default) | `stt/sherpa-onnx-moonshine-tiny-en-int8` | `stt/whisper-tiny/ggml-tiny.bin` |
+| default | `stt/sherpa-onnx-moonshine-base-en-int8` | `stt/whisper-base/ggml-base.bin` |
+| accuracy | `stt/sherpa-onnx-moonshine-base-en-int8` | `stt/whisper-small/ggml-small.bin` |
+
+Set the envs in `.env` to select a tier. The minimal tier is shipped as the
+default because it is the only one that reliably meets the 300 ms
+first-partial latency budget on CPU.
+
+VAD tuning (all optional): `STT_VAD_THRESHOLD` (default 0.5),
+`STT_VAD_MIN_SPEECH_MS` (250), `STT_VAD_MIN_SILENCE_MS` (500),
+`STT_VAD_MAX_SPEECH_MS` (30000 — a too-long utterance is capped with a
+`final`, then a new segment starts).
+
 ### Advanced: Ollama and ComfyUI
 
 Both are available as drop-in alternatives on the **same ports**:
@@ -239,10 +305,19 @@ engine on a Mac is CPU-only and slow. On Darwin the supported setup is:
    ```bash
    ./bin/run-native-llm.sh   # llama-server on 11434 (or shimmy if present)
    ./bin/run-native-tts.sh   # sherpa-onnx Kokoro TTS on 8089
-   ./bin/run-native-stt.sh   # sherpa-onnx Moonshine STT on 8087
+   ./bin/run-native-stt.sh   # C-393 STT service on 8087 (Moonshine streaming + whisper.cpp batch)
    ```
-   Each downloads its default model on first run. The native path and the
-   containerised path expose **identical endpoints** (same ports).
+   The LLM and TTS launchers download their default models on first run;
+   `run-native-stt.sh` does **not** — STT models are provisioned by the
+   model fetcher (like the containerised path), so run the fetcher first or
+   the script exits with a fetch hint. The native path and the
+   containerised path expose **identical endpoints** (same ports, same
+   protocol). `run-native-stt.sh` needs `pip install sherpa-onnx` on the
+   host; batch transcription additionally needs the whisper.cpp
+   `whisper-server` binary (`brew` provides `whisper-cli`, but the server
+   is a source build with `-DWHISPER_BUILD_SERVER=ON` — see the script
+   header). Without it the streaming service still runs and reports batch
+   unavailable via `/v1/capabilities`.
 2. Only the optional web client is containerised:
    ```bash
    COMPOSE_PROFILES=web docker compose up -d
@@ -262,7 +337,12 @@ provides no Metal passthrough for the engines.
   other's health. `docker compose ps` shows meaningful per-service state.
 - **Missing model**: a service whose model file is absent starts and then
   fails its own health check — the health message names the missing file, and
-  the other engines are unaffected.
+  the other engines are unaffected. With the `stt` profile enabled, the voice
+  health check covers BOTH the TTS port (8089) and the STT port (8087) — a
+  dead batch process cannot hide behind a healthy TTS.
+- **STT observability**: logs cover connection lifecycle, model load,
+  language, and decode duration. **Transcript text is never logged** — it is
+  user speech content (AC-8).
 - **Offline**: once images and models are cached, the whole stack starts with
   networking disabled. A missing model disables only its own service.
 - **Warm start**: an already-provisioned stack reaches all-healthy in well
@@ -293,6 +373,11 @@ bun moon run local-stack:lint
 | Ports match `development_ports.ts`, loopback binds, no 8080 | AC-11 |
 | Native launchers present, executable, port-defaulted (explicit Darwin branch) | AC-12 |
 | `MODELS_PATH` bind mount render + health | AC-13 |
+| STT off by default (`.env.example` defaults, no STT port render) | C-393 AC-7 |
+| STT manifest tiers + no weights COPYed into the image | C-393 AC-11 |
+| STT live wire contract (`stt_service.test.ts`: partials+final, VAD, batch, capabilities, language, format, origin) | C-393 AC-1..AC-9 (live) |
+| Audio never persisted / transcript never logged (container fs + log grep) | C-393 AC-8 (live) |
+| Missing STT model → unhealthy naming the file (throwaway container) | C-393 AC-10 (live) |
 
 ## Container security
 

apps/backend/local-stack/bin/run-native-stt.sh

@@ -1,42 +1,104 @@
 #!/usr/bin/env bash
 # apps/backend/local-stack/bin/run-native-stt.sh
-# Native host launcher for local speech-to-text (STT) without Docker.
+# Native host launcher for local speech-to-text (STT) without Docker — the
+# macOS path (Docker Desktop has no Metal passthrough; this is a
+# latency-sensitive service, C-393 AC-12).
 #
-# Runs the sherpa-onnx C++ offline websocket STT server with a Moonshine
-# int8-quantized ONNX model. whisper.cpp users can swap the binary below for
-# `whisper-server` (whisper.cpp example server) — the websocket protocol the
-# client speaks is what matters.
+# Starts the SAME service the container runs, on the same port and protocol:
+#   - python3 docker/voice/stt_server.py on $STT_PORT (8087) — the C-393
+#     streaming websocket (WS /v1/stream, Moonshine + Silero VAD), plus
+#     GET /v1/capabilities, GET /health, and the OpenAI-compatible batch
+#     proxy.
+#   - whisper-server (whisper.cpp) on the internal WHISPER_PORT when the
+#     binary is present — batch transcription (POST /v1/audio/transcriptions).
+#
+# Host requirements:
+#   - python3 (any modern 3.x) with the sherpa-onnx wheel:
+#       pip install sherpa-onnx
+#     (and a C compiler + cmake if you build sherpa-onnx from source)
+#   - whisper.cpp server for batch (optional but recommended):
+#       brew install whisper-cpp        # provides whisper-cli
+#       # whisper-server needs a source build with WHISPER_BUILD_SERVER=ON:
+#       #   git clone https://github.com/ggml-org/whisper.cpp
+#       #   cmake -B build -DWHISPER_BUILD_SERVER=ON && cmake --build build --target whisper-server
+#       #   ln -s "$PWD/build/bin/whisper-server" /usr/local/bin/
+#
+# Models live in ./models/stt and are provisioned by the stack/model
+# fetcher — this script never downloads them; it verifies the files exist
+# and exits with a fetch hint when a model is missing. Model selection
+# mirrors the container:
+#   STT_STREAM_MODEL / STT_BATCH_MODEL / STT_VAD_MODEL (manifest targetPaths).
 set -euo pipefail
 
-MODEL_DIR="$(pwd)/models/stt"
-MODEL_NAME="sherpa-onnx-moonshine-tiny-en-int8"
-MODEL_PATH="$MODEL_DIR/$MODEL_NAME"
-# Port from packages/shared/constants development_ports.ts (C-390 AC-11).
+# Ports from packages/shared/constants development_ports.ts (C-390 AC-11).
 PORT="${STT_PORT:-8087}"
+# Export so stt_server.py's batch proxy reads the SAME internal port the
+# whisper-server was launched on (it defaults to 8091 on its own).
+export WHISPER_PORT="${WHISPER_PORT:-8091}"
+BIND="${STT_BIND_ADDRESS:-127.0.0.1}"
+
+# C-393 model selection (manifest targetPaths, mirror the container defaults).
+STT_STREAM_MODEL="${STT_STREAM_MODEL:-stt/sherpa-onnx-moonshine-tiny-en-int8}"
+STT_BATCH_MODEL="${STT_BATCH_MODEL:-stt/whisper-tiny/ggml-tiny.bin}"
+STT_VAD_MODEL="${STT_VAD_MODEL:-stt/silero_vad.onnx}"
+
+MODELS_DIR="$(cd "$(dirname "$0")/.." && pwd)/models"
+STREAM_DIR="$MODELS_DIR/$STT_STREAM_MODEL"
+BATCH_FILE="$MODELS_DIR/$STT_BATCH_MODEL"
+VAD_FILE="$MODELS_DIR/$STT_VAD_MODEL"
 
-# Verify the sherpa-onnx binary is installed on the host BEFORE downloading
-# any model — don't pull gigabytes of weights for a server that can't run.
-if ! command -v sherpa-onnx-offline-websocket-server >/dev/null 2>&1; then
-    echo "❌ sherpa-onnx is not installed on the host."
-    echo "   Install it with:  pip install sherpa-onnx"
-    echo "   or download the prebuilt C++ binaries from the k2-fsa GitHub releases."
+# Verify the streaming model files exist BEFORE fetching anything the server
+# cannot run — the service must not claim readiness without its model.
+if [ ! -d "$STREAM_DIR" ] || [ ! -f "$STREAM_DIR/encode.int8.onnx" ]; then
+    echo "❌ Moonshine STT model missing in $MODELS_DIR/$STT_STREAM_MODEL."
+    echo "   Run the model fetcher:  bun stack/fetch_models.ts --entry stt-moonshine-tiny-en-int8 --entry stt-whisper-tiny"
+    echo "   (or download the tarball from the k2-fsa sherpa-onnx releases and"
+    echo "   extract it to $STREAM_DIR)"
+    exit 1
+fi
+if [ ! -f "$VAD_FILE" ]; then
+    echo "❌ Silero VAD model missing at $VAD_FILE — fetch it with the model fetcher."
     exit 1
 fi
 
-if [ ! -d "$MODEL_PATH" ]; then
-    echo "Moonshine STT model missing in $MODEL_DIR. Downloading..."
-    mkdir -p "$MODEL_DIR"
-    curl -fSL -o "$MODEL_DIR/moonshine.tar.bz2" \
-        "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-moonshine-tiny-en-int8.tar.bz2"
-    tar xjf "$MODEL_DIR/moonshine.tar.bz2" -C "$MODEL_DIR"
-    rm -f "$MODEL_DIR/moonshine.tar.bz2"
+# Export the model paths for stt_server.py (it resolves defaults itself, but
+# the explicit exports keep this script the single source of truth).
+export MODELS_DIR STT_STREAM_MODEL STT_BATCH_MODEL STT_VAD_MODEL STT_BIND_ADDRESS="$BIND"
+
+# Batch engine (optional on the host): whisper-server must be installed
+# separately; without it the service still streams and reports batch
+# unavailable via /v1/capabilities.
+if command -v whisper-server >/dev/null 2>&1; then
+    if [ ! -f "$BATCH_FILE" ]; then
+        echo "⚠ whisper batch model missing at $BATCH_FILE — batch will be unavailable"
+        echo "  (fetch it with: bun stack/fetch_models.ts --entry stt-whisper-tiny)"
+    else
+        echo "Starting whisper.cpp batch server on 127.0.0.1:$WHISPER_PORT ..."
+        WHISPER_LOG="$(mktemp "${TMPDIR:-/tmp}/whisper-server.XXXXXX.log")"
+        whisper-server \
+            --host 127.0.0.1 \
+            --port "$WHISPER_PORT" \
+            --model "$BATCH_FILE" \
+            --threads "${STT_WHISPER_THREADS:-4}" \
+            --no-gpu \
+            > "$WHISPER_LOG" 2>&1 &
+        WHISPER_PID=$!
+    fi
+else
+    echo "⚠ whisper-server not found on the host — batch endpoint unavailable (streaming still works)"
 fi
 
-echo "Starting native sherpa-onnx WebSocket STT server on port $PORT..."
-exec sherpa-onnx-offline-websocket-server \
-    --port="$PORT" \
-    --moonshine-preprocessor="$MODEL_PATH/preprocess.onnx" \
-    --moonshine-encoder="$MODEL_PATH/encode.int8.onnx" \
-    --moonshine-uncached-decoder="$MODEL_PATH/uncached_decode.int8.onnx" \
-    --moonshine-cached-decoder="$MODEL_PATH/cached_decode.int8.onnx" \
-    --tokens="$MODEL_PATH/tokens.txt"
+# Keep the background whisper-server alive while stt_server.py runs and
+# clean it up (plus its unique temp log) when the script exits — an exec
+# handoff would orphan the batch process once the STT server stopped.
+cleanup() {
+    if [ -n "${WHISPER_PID:-}" ]; then
+        kill "$WHISPER_PID" 2>/dev/null || true
+        wait "$WHISPER_PID" 2>/dev/null || true
+    fi
+    rm -f "${WHISPER_LOG:-}"
+}
+trap cleanup EXIT
+
+echo "Starting native STT server on $BIND:$PORT ..."
+python3 "$(dirname "$0")/../docker/voice/stt_server.py" "$PORT"

apps/backend/local-stack/compose.stt.yaml

@@ -0,0 +1,23 @@
+# apps/backend/local-stack/compose.stt.yaml
+#
+# C-393 STT port override — adds the STT port publish to the voice service.
+#
+# The base compose.yaml deliberately does NOT publish the STT port: AC-7
+# requires that the default stack binds no STT port. Docker Compose cannot
+# conditionally publish a port per-profile, so enabling STT means adding this
+# override to COMPOSE_FILE (documented in .env.example):
+#
+#   COMPOSE_FILE=compose.yaml:compose.cpu.yaml:compose.stt.yaml
+#
+# Everything else about the STT service (the streaming server, whisper.cpp
+# batch engine, health check) is defined in the base file and activated by
+# the `stt` profile + ENABLE_STT=true.
+#
+# Port: 8087 (emulator) / 8086 (staging) / 8090 (production) — from
+# packages/shared/constants development_ports.ts (C-390 AC-11). The host
+# publish is loopback-only, like every engine.
+
+services:
+  voice:
+    ports:
+      - "127.0.0.1:${STT_PORT:-8087}:8087"