jadai — Offline Voice Agent

A local, deterministic voice agent for Windows. Captures a wake word with a small Vosk model, opens a bounded command window backed by a larger Vosk model, parses utterances through a declarative scoring engine, and executes only allowlisted actions — system controls, browser operations, and Spotify Web API integration. No cloud STT, no LLM in the dispatch path, no direct subprocess execution in the application layer.

Architecture

                        Mic (sounddevice RawInputStream)
                                       │
                                       ▼
                              Bounded audio queue
                              (8s cap, drop-on-full)
                                       │
                ┌──────────────────────┼──────────────────────┐
                ▼                                             ▼
        Wake recognizer                            Command recognizer
        (small Vosk model)                         (large Vosk model)
                │ "jadai"                                     │
                └──────────────► Activity window ◄────────────┘
                                       │
                                       ▼
                              normalize_text()
                       (filler strip, canonical map,
                        phonetic merge for Vosk variants)
                                       │
                                       ▼
                              match_intent_trace()
                       (declarative rule scan, scoring,
                        priority + alpha tie-break)
                                       │
                                       ▼
                            validate_allowed_action()
                                  (registry gate)
                                       │
                                       ▼
                                  handler()
                  webbrowser │ pycaw │ sbc │ spotipy │ pyautogui

Data flow: Audio is consumed by a single owner. Wake detection runs on a fast model; once triggered, a separate command recognizer consumes the same queue inside an activity window. Every emitted utterance is normalized, scored against every rule, and dispatched only if it survives the allowlist gate.

Features

Core

• Two-stage Vosk pipeline — small wake model (~40 MB) for low-latency trigger, large command model (~1.8 GB) for accurate recognition
• Declarative intent rules — every command is an IntentRule dataclass; no if-cascade dispatch
• Scoring engine with deterministic tie-break — (score, priority, action) ordering means RULES insertion order is never a tie-breaker
• Activity-based command window — 12s deadline resetting on every meaningful utterance, with early-exit on SILENCE_AFTER_SPEECH
• Bounded audio queue — 8s cap with drop-on-full, prevents stale-audio replay after slow actions

Recognition Hardening

• Canonical map — known Vosk mishearings (you do → youtube, spot if i → spotify) corrected pre-parse with word-boundary regex
• Phonetic merge — (you|your) + <tube-sound> collapsed when phrase is short or contains an action verb
• Time-based partial stability — Vosk partials must stay identical for 0.7s before firing, blocking mid-sentence misfires on long queries
• Two-layer dedup — text-level cooldown on normalized utterances + intent-level cooldown on (action, param) tuples
• Bare-noun guard — youtube / spotify alone only fires OPEN_X when no conflicting verb (search, play, pause) is present

Action Catalog

• Browser — open YouTube, open Spotify, YouTube search, Google search
• System — set volume [0,100], volume up/down, set brightness [0,100]
• Spotify — play, pause, next, previous, play specific track, search
• Spotify resilience — API-first with classified fallback: 403 (Premium) and 404 (no device) surface real errors; transient failures fall back to media keys

Spotify Integration

• Lazy OAuth client — credentials read once from .env; init failures swallow exceptions to prevent traceback leakage
• Token cache — refresh token persisted to .cache-spotify (gitignored)
• Track-qualified search — q="track:<query>" first to bias matches toward titles, free-text fallback for STT-mangled queries
• Karaoke/cover filter — top 5 candidates pulled; entries with karaoke, tribute, or cover version in name/artist are skipped
• transfer_playback(force_play=True) — reliable resume even when no playback context exists
• HTTP timeout — every Spotipy call bounded to 10 seconds; prevents network stalls from freezing the orchestrator

Observability

• Rotating structured log — logs/app.log, 1 MB × 3, UTF-8
• State transitions logged — IDLE → LISTENING → EXECUTING → IDLE with timestamps
• Full intent trace — every utterance produces INTENT_DECISION (winner score, runners-up, skipped rules) when DEBUG=True
• Spotify audit trail — SPOTIFY_CANDIDATES (all returned tracks) and SPOTIFY_TRACK_SELECTED (final choice with URI) for debugging wrong-song complaints

System Guarantees

• Local STT — speech recognition runs entirely locally via Vosk; raw audio is never uploaded
• Streaming-only audio — processed as an in-memory stream; nothing written to disk
• Allowlist-gated dispatch — validate_allowed_action runs before every handler call
• Fail-closed on parse — unknown utterances drop silently and log PARSE_FAILED
• Deterministic intent resolution after normalization — once a normalized utterance is produced, rule selection is fully reproducible

Failure Behavior

• Parse failure → PARSE_FAILED log → continue listening
• Handler raises → EXECUTION_FAILED log → orchestrator resets to LISTENING
• Spotify 403 / 404 → user-facing message, no silent fallback
• Spotify network error → media-key fallback (where applicable)
• Network hang → 10-second hard timeout on every Spotipy call
• Orchestrator exception → 1-second backoff → state reset to IDLE

Known Limitations

• Windows-only — pycaw, screen_brightness_control, and pywin32 are Windows-specific
• Vosk wake-word variance — "jadai" is not an English dictionary word; the small wake model may transcribe it inconsistently across speakers
• Single-track search precision — limit=5 with a noise filter is a heuristic, not semantic search
• No speaker verification — anyone within microphone range can issue commands
• No multi-language — English-only models

Requirements

• Windows 10 / 11
• Python 3.10+ (PEP 604 union syntax used throughout); tested on Python 3.11
• Working microphone
• Vosk models: vosk-model-small-en-us-0.15 (wake) and vosk-model-en-us-0.22 (command) — free from https://alphacephei.com/vosk/models. The command model is large (~1.8 GB); expect non-trivial disk usage and memory pressure during recognition.
• Optional: Spotify developer credentials for API-driven playback

Why Vosk? Chosen for low-latency streaming inference and fully local execution. Alternatives like Whisper are more accurate on long-form audio but introduce higher per-chunk latency and (in their cloud-API form) violate the local-only constraint.

Quick Start

1. Clone

git clone https://github.com/<you>/jadai.git
cd jadai

2. Create virtual environment

python -m venv venv
venv\Scripts\pip install -r requirements.txt

3. Install Vosk models

Download both models from https://alphacephei.com/vosk/models and unzip into models/:

models/
├── vosk-model-small-en-us-0.15/    # ~40 MB — wake word
└── vosk-model-en-us-0.22/          # ~1.8 GB — commands

Paths are pinned in app/config.py.

4. (Optional) Configure Spotify

Without this step, Spotify commands degrade to media keys and browser-based search. For API-driven playback (Premium required):

copy .env.example .env

Create an app at https://developer.spotify.com/dashboard, set the redirect URI to http://localhost:8888/callback in the app settings, and fill .env:

SPOTIFY_CLIENT_ID=...
SPOTIFY_CLIENT_SECRET=...
SPOTIFY_REDIRECT_URI=http://localhost:8888/callback

OAuth consent runs once on first Spotify command and caches the refresh token to .cache-spotify.

5. Run

venv\Scripts\python -m app.main

Wait for Loading wake model..., say jadai, then issue a command.

Commands

Trigger phrase	Action	Notes
open youtube	Browser to youtube.com
open spotify	Launch Spotify (or web fallback)
search youtube <query>	YouTube search page	URL-encoded
search <query> on youtube	YouTube search page	Either phrasing works
google <query> / search <query>	Google search
set volume to <n>	System volume [0,100]	Digits or word-form numbers
volume up / volume down	±10%	Also: louder, quieter, raise, lower
set brightness to <n>	Laptop backlight [0,100]
play spotify	Resume playback	API → media key fallback
pause	Pause Spotify	API → media key fallback
next / previous / go back	Skip track	API → media key fallback
play <song> on spotify	Search + start playback	Premium required for API; browser fallback otherwise
search <query> on spotify	Spotify search page	No auth required

Multiple commands in one utterance are split on " and ":

jadai open youtube and play billie jean on spotify

Extending

Adding a new command

1. Implement the handler in app/executor/actions.py and self-register:

def lock_screen(_=None):
    ctypes.windll.user32.LockWorkStation()

registry.register("LOCK_SCREEN", lock_screen)

2. Add an IntentRule in app/intent/rules.py:

IntentRule(
    action="LOCK_SCREEN",
    required=frozenset({"lock", "screen"}),
    base_score=10,
),

That's the full surface — no dispatcher edits, no parser changes. The engine picks it up.

For commands with parameters, write an extractor in app/intent/extractors.py returning either the param value or the SKIP sentinel; existing extractors in that file are short and copy-friendly.

Configuration

All tunable constants live in app/config.py.

Parameter	Default	Description
WAKE_WORD	"jadai"	Trigger word — must be first or last in an utterance
WAKE_MODEL_PATH	models/vosk-model-small-en-us-0.15	Path to small wake model
COMMAND_MODEL_PATH	models/vosk-model-en-us-0.22	Path to large command model
SAMPLE_RATE	16000	Hz (Vosk requirement)
BLOCK_SIZE	2000	Samples per chunk (~125 ms)
ACTIVITY_TIMEOUT	12	Seconds before idle command window closes
SILENCE_AFTER_SPEECH	3	Seconds of silence after a command before window closes early
PARTIAL_STABILITY_SECONDS	0.7	How long a Vosk partial must stay identical before firing
COMMAND_COOLDOWN	1.5	Blocks duplicate intent firing within this window
DEBUG	True	[raw]/[normalized] stdout + INTENT_DECISION log lines

Observability

Log Format

Rotating handler at logs/app.log (1 MB × 3, UTF-8). Each command produces a complete trace:

COMMAND_CAPTURED: 'play billie jean on spotify'
INTENT_DECISION: SPOTIFY_PLAY_SONG=19 | runners: - | skipped: SPOTIFY_PLAY
INTENT: SPOTIFY_PLAY_SONG | PARAM: 'billie jean'
STATE: EXECUTING | action=SPOTIFY_PLAY_SONG param='billie jean'
SPOTIFY_CANDIDATES: query='billie jean' -> ['Billie Jean' by 'Michael Jackson'; ...]
SPOTIFY_TRACK_SELECTED: query='billie jean' selected='Billie Jean' by 'Michael Jackson' uri=spotify:track:5ChkMS8OtdzJeqyybCc9R5
SPOTIFY_PLAYBACK_STARTED: device='LIVING-ROOM' track='Billie Jean'
EXECUTION_SUCCESS: SPOTIFY_PLAY_SONG

Diagnostic Log Lines

Line	Emitted when	Severity
STATE: <name>	Every state transition	INFO
COMMAND_CAPTURED	Every utterance accepted from STT	INFO
INTENT_DECISION	Every parse attempt (when DEBUG=True)	INFO
INTENT	Every successful parse	INFO
PARSE_FAILED	Utterance had no matching rule	WARNING
COOLDOWN_SKIP / INTENT_COOLDOWN_SKIP	Duplicate command suppressed	WARNING
EXECUTION_SUCCESS / EXECUTION_FAILED	Handler completed / raised	INFO / ERROR
SPOTIFY_API_<code>	Spotipy returned non-2xx	WARNING
WINDOW_CLOSED	Command window closed on timeout	INFO

Security Model

This section documents the system's trust assumptions, the threats it is designed to mitigate, and the threats explicitly out of scope.

Trust Model

This system protects against:

• Cloud-side speech data leakage — STT runs entirely locally via Vosk
• Arbitrary command execution — only registry-registered handlers can run
• URL injection — every browser-opened URL goes through urllib.parse.quote(_plus)
• Out-of-range numeric inputs — volume/brightness clamped to [0,100]
• Oversized queries — handler-side caps at 100 / 200 chars depending on backend
• Token leakage via logs or tracebacks — Spotify init exceptions caught and discarded; nothing in .env or .cache-spotify is ever logged
• Network stalls — every Spotipy call bounded by requests_timeout=10
• Unbounded memory growth — audio queue capped at 8 seconds with drop-on-full
• Mid-sentence command firing — PARTIAL_STABILITY_SECONDS time-gate plus bare-noun guards

This system does NOT protect against:

• Anyone within microphone range — no speaker verification
• Compromised Spotify Web API responses — track URIs are trusted as returned
• A modified actions.py — registry is the only allowlist; tampering it adds whatever handlers you write
• Future feature creep — adding subprocess execution, filesystem mutation, LLM in the dispatch path, or browser automation invalidates most of the above

Credential Handling

• .env and .cache-spotify are gitignored
• .env.example ships with empty placeholders only
• Credentials are read only by app/integrations/spotify_client.py; the rest of the app receives a Spotipy client object or None
• Spotify OAuth flow uses standard local-callback (http://localhost:8888/callback) — never a public redirect

Dependency Hygiene

All dependencies are version-pinned in requirements.txt. Audit before pushing changes:

venv\Scripts\pip-audit

Project Structure

jadai/
├── app/
│   ├── main.py              # Entry point — calls orchestrator.run()
│   ├── config.py            # All tunable constants
│   ├── stt/
│   │   └── wake_listener.py # Sole mic owner: wake detection + command window
│   ├── intent/
│   │   ├── parser.py        # normalize_text() + parse_command()
│   │   ├── models.py        # IntentRule, IntentMatch dataclasses + SKIP sentinel
│   │   ├── extractors.py    # Param extractors, guards, strip-sets, word-form numbers
│   │   ├── rules.py         # Declarative rule registry — source of truth
│   │   └── engine.py        # match_intent() / match_intent_trace() — scoring + selection
│   ├── executor/
│   │   └── actions.py       # All action handlers; self-registers on import
│   ├── integrations/
│   │   └── spotify_client.py # Lazy Spotipy wrapper; sole SPOTIFY_* env consumer
│   ├── core/
│   │   ├── registry.py      # CommandRegistry
│   │   ├── orchestrator.py  # Control loop with two-layer dedup + decision logging
│   │   └── state.py         # AgentState enum
│   └── utils/
│       ├── logger.py        # RotatingFileHandler
│       └── security.py      # validate_allowed_action()
├── .env.example             # Spotify credential template
├── requirements.txt         # Pinned dependencies
└── CLAUDE.md                # Full architecture & design rationale

Failure Testing

Verify graceful degradation:

# Without Spotify credentials — media keys should still work
ren .env .env.bak
venv\Scripts\python -m app.main
# Say: jadai pause   → should still pause via media key fallback
ren .env.bak .env

# Inspect intent decisions
type logs\app.log | findstr INTENT_DECISION

# Inspect Spotify track selection
type logs\app.log | findstr SPOTIFY_CANDIDATES

Troubleshooting

Wake word never triggers. Run with DEBUG=True and watch the [wake] stdout lines for whatever Vosk transcribed. "jadai" may land as jay die, jaday, or similar — broaden the matcher in app/stt/wake_listener.py:start_wake_listener to accept a set of acceptable spellings.

'AudioDevice' object has no attribute 'Activate'. pycaw < 20251023 — upgrade with venv\Scripts\pip install -U pycaw.

play <song> on spotify plays the wrong track. Check the SPOTIFY_CANDIDATES log line. If the canonical track isn't in the candidate list, the query is too ambiguous for Spotify's search. If it's there but a different one was selected, the noise filter rejected it — adjust _TRACK_REJECT_TOKENS in app/executor/actions.py.

Commands fire mid-sentence. Increase PARTIAL_STABILITY_SECONDS in app/config.py (default 0.7; try 1.0–1.5 for slow speakers).

Spotify OAuth prompts on every run. .cache-spotify must be writable in the project root. If it can't be persisted, the refresh token is discarded and full consent re-runs.

License

MIT