mlx_lm.server: generation hangs at 0% CPU right after prompt processing on ~22-26k-token streaming requests (gemma-4-26b-a4b, 0.31.3)

[TadeuszWolfGang]

Summary

mlx_lm.server hangs (decode never starts, process at 0% CPU) immediately after prompt processing completes, on long (~22–26k token) streaming chat-completion requests from a real client (Obsidian Copilot). Reproduced 2/2 times with real client traffic; synthetic requests of the same size — including concurrent + streaming — complete fine, so the trigger seems to be a specific parameter/content combination rather than prompt length alone.

Environment

• mlx-lm 0.31.3, mlx/mlx-metal 0.31.2 (installed via uv tool install mlx-lm, transformers==5.12.1)
• macOS 26.5 (Darwin 25.5.0), Apple M5 Max, 128 GB
• Python 3.12.13
• Model: mlx-community/gemma-4-26b-a4b-it-8bit (model_type gemma4), pinned at startup
• Server: mlx_lm.server --model /path/to/gemma-4-26b-a4b-it-8bit --host 127.0.0.1 --port 8082 --decode-concurrency 32 --prompt-concurrency 8
• (Environment note: a local one-line models/gemma4_unified.py alias exists for a different model, mirroring PR Add gemma4_unified model alias for Gemma 4 12B support #1386; it is not imported by the affected instance, which serves a stock gemma4 checkpoint.)

Failing requests (2/2 hangs)

Client: Obsidian Copilot plugin ("3rd party (openai-format)" provider). Request shape: stream: true, temperature: 0.1, max_tokens: 16000, system prompt + long mixed Polish/English markdown context.

• Hang 1: prompt 22,258 tokens — prefill completes in ~8 s, then nothing.
• Hang 2 (after server restart): prompt 25,660 tokens — prefill completes in ~12 s, then nothing.

In both cases the log shows prompt processing reaching N/N and then goes silent; no traceback, no completion line. The server process stays alive but at 0.0% CPU (checked minutes later — 5–18 min). Client eventually gives up.

2026-07-05 08:19:39,913 - INFO - Prompt processing progress: 23032/25660
2026-07-05 08:19:40,840 - INFO - Prompt processing progress: 25080/25660
2026-07-05 08:19:41,170 - INFO - Prompt processing progress: 25659/25660
2026-07-05 08:19:41,208 - INFO - Prompt processing progress: 25660/25660
(no further log lines; process at 0% CPU)

Right before the big request the client issued two small chat completions (~60 and ~300 prompt tokens), which completed normally.

What does NOT reproduce it (all pass)

Same server instance, same model:

1. Non-streaming 22,225-token prompt, max_tokens 60 → OK in 7.3 s.
2. Non-streaming 10,525-token prompt → OK in 9.5 s.
3. Concurrent mix mimicking the client: 2 small non-streaming + 1 streaming 22k-token request (max_tokens 200) fired simultaneously → all OK in ≤12 s.

So plain long prompts, streaming, and concurrency by themselves are fine. The failing requests differ mainly in max_tokens: 16000, a system prompt, temperature: 0.1, and real markdown content; also the small preliminary requests populate the prompt cache right before the big one (the log prints "Prompt Cache: N sequences" before each request), so a prompt-cache/trim interaction is a plausible suspect.

Expected behavior

Either generate tokens after prefill, or fail loudly. A silent hang with the worker at 0% CPU makes the failure invisible to KeepAlive-style supervision.

Happy to provide full logs or run patched builds to help narrow this down.

[rajanshxrma]

Your generation thread has likely died from an unhandled exception — the server process stays alive and keeps answering other endpoints, but every request in flight at that moment (and every new one afterward) hangs forever behind it, because the client-facing wait has no timeout. Worth checking your server's stderr/logs from right around the hang time for anything unusual, even something that looks unrelated to caching.

I couldn't reproduce your exact scenario (different model/hardware — I only have an M1/16GB), but I hit this exact failure shape from a different trigger, which is what points at the mechanism above: running the concurrent server (--decode-concurrency 32 --prompt-concurrency 8) with 40+ concurrent requests including one large cache-primed prompt, I got a GPU OOM (RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory) inside the background generation thread at server.py:891 (r.logprobs[r.token].item()). That thread crashed and was never restarted. Every request mid-stream at that point hung forever — process alive, 0% CPU on real work, no error ever surfaced to the client. Confirmed even a brand-new trivial request 15 minutes later still hung, so the whole server was permanently dead, not just the batch in flight. Root cause: response_queue.get() at server.py:1037/1048 has no timeout, and the one background generation thread has no supervision or restart — so any unhandled exception there (OOM here, maybe something else for you) looks identical from outside.

My trigger was GPU OOM under concurrent load (unsurprising on 16GB vs. your 128GB), so probably not literally your cause — but if stderr shows anything from around the hang, that'd confirm whether this is the same mechanism.

[TadeuszWolfGang]

@rajanshxrma Thanks — I checked the complete launchd-captured logs.

Your proposed mechanism is definitely present in a different mlx_lm.server instance from the same diagnostic set.

In mlx-server-daily.log — the Qwen instance running on port 8080 — immediately after processing a 15,297-token prompt, Thread-1 (_generate) died at approximately 07:07:40 with:

Exception in thread Thread-1 (_generate):
Traceback (most recent call last):
  ...
  File ".../mlx_lm/generate.py", line 1369, in _step
    mx.async_eval(self._next_tokens, self._next_logprobs, token_context)
RuntimeError: [metal::malloc] Resource limit (499000) exceeded.

The HTTP server process remained alive after that. At 08:25:32 the same instance still returned:

GET /v1/models HTTP/1.1" 200

So the dead-generation-thread / zombie-server failure mode you described is confirmed: the inference worker dies, while the HTTP server and non-generation endpoints remain available.

However, the two Gemma 4 A4B hangs reported in this issue appear slightly different.

The affected instance on port 8082 is launched by launchd with both StandardOutPath and StandardErrorPath pointing to the same mlx-reviewer.log, so Python thread exceptions should have been captured.

For the two failures, the log ends at:

Prompt processing progress: 22258/22258

and:

Prompt processing progress: 25660/25660

After each line there is no further output until the server is restarted. There is no:

Exception in thread Thread-1 (_generate)

and no Python traceback.

Since stderr capture demonstrably worked for the Qwen failure, my current best hypothesis is that the Gemma generation thread did not exit because of a normal Python exception. Instead, it may have remained alive but become blocked inside native MLX/Metal during the prefill-to-decode transition, probably around the first decode step or synchronization.

This means #1505 / #1513 should address the confirmed dead-thread zombie case, but it may not detect this second failure mode if _generation_thread.is_alive() remains true while the thread is blocked indefinitely.

On the next occurrence I can capture:

• a native process sample using sample <pid> or spindump,
• Python thread stacks using faulthandler,
• whether a new trivial completion also hangs,
• whether /v1/models continues returning HTTP 200,
• the generation thread state immediately after the last prefill progress line.

Please let me know which diagnostic would be most useful, or whether there is a patched build you would like me to test.

[rajanshxrma]

Thanks for pulling the full logs — that Qwen traceback is a clean confirmation of the zombie mechanism, and it's actually a second distinct crash site: my repro died at server.py:891 (r.logprobs[r.token].item()), yours at generate.py:1369 (mx.async_eval in _step, under [metal::malloc] Resource limit). Two different uncaught exceptions, same outcome, which supports the "supervise the worker, don't patch individual crash sites" direction in #1513.

On the Gemma hangs — I think you're right that it's a different mode, and the code shape is consistent with your blocked-in-native hypothesis. With --decode-concurrency set, all inference (prefill and decode, across all requests) runs in one batch loop that repeatedly calls batch_generator.next(); the HTTP handler threads only consume per-request queues fed by that loop, and those get() calls have no timeout. So if next() blocks in native MLX/Metal at the prefill→decode transition, the entire engine stalls with the thread still alive: no exception for stderr to capture, 0% CPU, every in-flight and subsequent completion hangs, /v1/models still 200 (it never touches the engine). Your log capture working for the Qwen crash but showing nothing here is exactly what this mode would produce.

Most useful diagnostic on next occurrence, by a wide margin: a stack dump of the live process at hang time —

uv tool install py-spy   # or pipx install py-spy
py-spy dump --pid <pid> --native

That one artifact shows the exact Python line the batch loop is blocked on plus the native frames below it, and it discriminates between the two candidate buckets:

1. blocked in the prompt-cache fetch/trim path (Python frames in server.py's cache handling) — plausible given your failing requests are immediately preceded by small cache-priming completions and gemma4 has sliding-window layers, which would put this near the windowed-cache-trim territory of Prompt-cache reuse can return KV state that doesn't match the keyed prefix (windowed-cache trim contract) #1494;
2. blocked under mx.eval/mx.async_eval in a Metal wait that never completes — a deeper MLX-side issue, in which case a sample <pid> 5 / spindump alongside it is worth capturing for the native side.

Your trivial-completion probe is also worth doing (it confirms the stall is engine-wide rather than per-request), and I'd expect /v1/models to stay 200 in both buckets, so that one carries less signal.

And yes — agreed on the #1513 gap: as written it catches a dead worker, not a blocked-alive one. If your dump confirms a stalled batch loop, the natural extension is a liveness heartbeat (the loop timestamps each iteration; the supervisor treats "requests in flight + no heartbeat for N seconds" as a stall and fails the in-flight requests with real errors instead of silence). Happy to put that together as a follow-up once there's a stack showing where it actually sticks — better to aim it at the confirmed block point than guess.

[nesquikm]

Same problem here on an M5 Max, gemma-4-26b-a4b (4bit MLX) through LM Studio. Long multi-turn chats freeze at 0% CPU right after prompt processing, short/fresh ones are fine. Can't reproduce it with synthetic prompts either.

[IzD680]

Same symptom class on mlx-lm 0.31.3 / mlx 0.32.0, larger model: GLM-5.2 4-bit (~390 GB) on Mac Studio M3 Ultra 512 GB, mlx_lm.server, non-streaming batched requests (6-22k token prompts).

Twice in 24h on two different machines: server process stays alive, /v1/models answers instantly, but generation never starts — 0% CPU, request hangs indefinitely (observed 7h in one case). No exception, nothing in the server log after the hang. kill + restart fully restores throughput.

No reliable trigger identified; in both cases it happened after hours of continuous batch serving. One confounder we hit while restarting: an mlx_lm from a different (older) venv fails first request with ValueError: Missing 627 parameters (self_attn.indexer.*) while HTTP stays up, which mimics the same wedge — worth ruling out before diagnosing a hang.

Workaround deployed: external watchdog issuing a 4-token generation probe every 5 min and hard-restarting the server on timeout.

[rajanshxrma]

@nesquikm @IzD680 thanks — both reports widen the picture usefully. GLM-5.2 non-streaming batched inside mlx_lm.server plus gemma-4 streaming shows this isn't tied to one model family or request mode; the common thread so far is long-context work after sustained serving, wedging right at the end of prompt processing. And if the LM Studio repro is the same underlying stall, that would point below the server layer entirely (LM Studio runs its own engine on top of MLX rather than mlx_lm.server) — worth keeping in mind, though I'd treat it as unconfirmed until we see a stack.

One correction to the capture recipe from my earlier comment, before anyone spends the next occurrence on it: py-spy's --native flag isn't available on macOS (see the platform support table in the py-spy README), and py-spy on macOS needs root. So on Apple silicon the pair to grab from the hung process is:

sudo py-spy dump --pid <pid>          # Python frames — works even if the stuck thread is inside native code
sample <pid> 5 -f mlx-hang-native.txt # native frames for the same window

Together those cover both candidate buckets (Python-side cache path vs. a Metal wait under mx.eval — on current main the decode step's blocking sync is the mx.eval at the end of GenerationBatch._step, right after the mx.async_eval for the next step, which is exactly where an alive-but-wedged thread would sit at 0% CPU with nothing for stderr to capture).

@IzD680 — good catch on the confounder. The missing-parameters ValueError case should be distinguishable from the silent hang by its traceback in the server log; the mode discussed here leaves nothing after the last Prompt processing progress line. And your external 4-token-probe watchdog is effectively a production version of the liveness heartbeat discussed above as the follow-up to #1513 (which as written catches a dead worker, not a blocked-alive one) — the fact that it's holding up in production is a useful validation of that direction. A stack dump from whoever hits it next tells us where to aim the built-in version.

[TadeuszWolfGang]

Caught it live with full stack captures — and the result is more interesting than either predicted bucket: the generation thread is not blocked at all; it's a livelock. The batch loop keeps iterating _step indefinitely while no output is ever delivered to any consumer.

Reproduction (now 3/3 for this trigger)

Same real-client trigger as the two original hangs: Obsidian Copilot vault-QA request against the pinned gemma-4-26b-a4b-it-8bit instance (--max-tokens 32768 --decode-concurrency 32 --prompt-concurrency 8, mlx-lm 0.31.3) — two small cache-priming completions immediately followed by one streaming request with system prompt + ~25k tokens of real mixed Polish/English markdown, temperature 0.1. Prompt processing completed normally; then the wedge. Notably, a synthetic loop mimicking this shape (2 small + 1 big streaming, 21–28k tokens of real markdown, cache churn) ran 12+ cycles without triggering — the real-client request wedges it reliably, the imitation doesn't (differences left: Copilot's <source>-tagged RAG template, max_tokens 16000, exact conversation state).

Stack captures (3 snapshots over 6 minutes, same PID)

T+0 and T+60s (sudo py-spy dump, identical at both instants) — generation thread alive, inside the async-eval sync:

Thread 0x16FF87000 (active): "Thread-1 (_generate)"
    _step (mlx_lm/generate.py:1369)      # mx.async_eval(...)
    next (mlx_lm/generate.py:1415)
    _next (mlx_lm/generate.py:1775)
    next (mlx_lm/generate.py:1855)
    _generate (mlx_lm/server.py:853)

Native frames for that window (sample <pid> 5, dominant path of the thread):

mlx::core::async_eval(std::vector<mlx::core::array>)
  mlx::core::eval_impl(std::vector<mlx::core::array>, bool)
    std::condition_variable::wait(...)
      __psynch_cvwait

T+6min (independent watchdog capture, same hang episode) — the thread has moved; now mid-forward-pass:

Thread 0x16FF87000 (active+gil): "Thread-1 (_generate)"
    __call__ (mlx_lm/models/gemma4_text.py:173)
    __call__ (mlx_lm/models/gemma4_text.py:360)
    __call__ (mlx_lm/models/gemma4_text.py:564)
    __call__ (mlx_lm/models/gemma4_text.py:596)
    __call__ (mlx_lm/models/gemma4.py:48)
    _step (mlx_lm/generate.py:1332)      # model forward, earlier in the same function
    ...same parents...

So across 6 minutes the loop alternates between the forward call (generate.py:1332) and the eval sync (generate.py:1369) — i.e. GenerationBatch keeps stepping. The sample output also shows ongoing Metal encoder activity (AGXMetalG17X compute-encoding frames) and the process shows real CPU time — consistent with continuous compute, not a stall.

Meanwhile, for the entire episode:

• the original streaming request delivered zero chunks;
• fresh trivial completions (new connections) hung past 180 s (response_queue.get() with no timeout — request threads parked at server.py:1037/1048, visible in all three dumps);
• GET /v1/models returned 200 throughout;
• state persisted until an external launchctl kickstart restored the instance; restart fully recovers.

Why this matters for the supervision work

This failure mode defeats both detection strategies discussed so far:

• _generation_thread.is_alive() (Keep the server generation loop alive when a batched request fails #1513 as written) — true the whole time;
• a naive per-iteration heartbeat (the follow-up idea) — would also tick, because iterations are happening.

The liveness signal has to be defined at the delivery level: requests in flight + no tokens delivered to any consumer queue for N seconds = stalled engine. (An external 4-token generation probe, as @IzD680 deployed, measures exactly this from outside; ours detects and auto-captures within ≤5 min.)

Where the loop might be stuck

Not established from the stacks alone — two readings fit:

1. the batch keeps generating but completions/admissions never finalize (results never dispatched to per-request queues, new requests never admitted into the batch);
2. generation itself never terminates for the wedged batch (e.g. per-request position/cache state corrupted after the cache-primed >22k-token prompt on gemma4's sliding-window layers — Prompt-cache reuse can return KV state that doesn't match the keyed prefix (windowed-cache trim contract) #1494 territory — so stop conditions never fire).

Given the trigger requires prompt-cache priming plus a >22k real prompt (solo same-size prompts pass), a cache/position interaction on sliding-window models looks plausible, but I'll leave that to someone who knows the batch internals.

Full capture bundles (3× py-spy, 2× 5s sample, server logs, probe transcripts, process stats) are archived; happy to share them, run instrumented/patched builds, or re-trigger on demand — the reproducer is armed and this machine hits it reliably.

[rajanshxrma]

Nice find — livelock is a real third failure mode, and it's a clean piece of evidence: the thread staying "alive" while making zero forward progress on delivery is exactly the gap between what #1513 checks and what actually needs checking.

Confirms why _generation_thread.is_alive() won't catch this: is_alive() only tells you the OS thread hasn't exited, which is true for a livelocked batch loop that's still calling _step() and burning real CPU/GPU cycles — nothing about that state is distinguishable from healthy work at the thread-liveness level. A naive heartbeat that ticks per loop iteration has the identical blind spot, just one layer down: your stacks show the loop genuinely iterating (alternating between the forward pass at generate.py:1332 and the eval sync at generate.py:1369) for the full 6 minutes with zero chunks delivered, so an iteration-counter heartbeat would tick the entire time and never trip.

The signal that actually distinguishes this state is delivery, not iteration: track wall-clock time since a token was last actually handed to a consumer (a per-request response queue, not the loop's own bookkeeping), and treat "requests in flight, no delivery for N seconds" as a stall — reset or fail the batch the same way #1513 already fails in-flight requests on an exception. That's a small extension of #1513's existing failure path, not a new mechanism: same fail-in-flight / log / drop-and-reset-the-batch-generator / keep-serving shape, just gated on a delivery-staleness timer instead of (or in addition to) is_alive().

Happy to build this — I've got direct working history in this exact file from #1513 (the admission/serving exception handling there is basically the sibling of what a delivery-liveness check would extend). Since #1513 is already open and unreviewed, is the preference to fold this in as an addition to that PR, or split it out as its own PR against main once #1513 lands? Either works on my end.

[TadeuszWolfGang]

Thanks — that approach makes sense and matches what the captured stacks show. I’d be happy to test the implementation against the reliable Gemma reproducer.

From my perspective, a separate follow-up PR based on #1513 would be cleaner: #1513 could remain focused on exceptions and dead-worker recovery, while delivery-stall detection could be reviewed independently. It could be opened as a stacked draft now and rebased onto main once #1513 lands. Of course, I’d defer to the maintainers if they prefer folding it into #1513.

I can validate against the real-world trigger that the livelock is detected, in-flight requests receive an explicit error, the server recovers, and subsequent requests complete without a process restart.

[rajanshxrma]

Opened the stacked draft as discussed: #1598. (GitHub won't let a cross-fork PR base on a branch that only exists on my fork, so it's opened against main — the diff will include #1513's changes until #1513 merges, noted in the PR description.)

It's a delivery-staleness watchdog — tracks time since anything was last put on a request's response queue, and if requests are in flight with nothing delivered for --generation-stall-timeout seconds (default 60), fails the in-flight requests and resets the batch generator using #1513's existing recovery path, no restart needed. Test simulates the livelock by mocking BatchGenerator.next() to spin without ever returning results.

@TadeuszWolfGang — it's armed for you to validate against your reliable Gemma reproducer whenever convenient. Let me know what you see.

[TadeuszWolfGang]

Validation update for #1598 at exact head
6e3f367c4b4b6693ef1006d2b9fc23dc4695842b.

One correction to my earlier shorthand: the saved July 17 Copilot
conversations confirm that two attempts in the first chat completed and the
fresh-chat attempt hung. The earlier "3/3" meant the two July 5 incidents plus
the July 17 incident, not three consecutive failures that day.

I replayed the July 17 workflow with the same Obsidian Copilot 3.3.3 Vault QA
client and vault/index:

1. new chat, same query;
2. same query again in that chat;
3. same query in a fresh chat;
4. two additional fresh-chat attempts.

All five turns completed. The three fresh-chat main prefills were 29,571,
29,915, and 26,565 tokens and completed in <=73.7 s, 105.3 s, and 155.2 s.
There was no generation stalled warning, traceback, or PID change, and two
subsequent control completions returned HTTP 200.

This was not a byte-identical replay: Vault QA injects dynamic source context.
The cache-priming shape was present, but the fresh replay prompts were generally
larger than the captured 22,258, 25,660, and 23,404-token incidents.

The real-client failure therefore did not reproduce. This is a
no-regression result, not validation of recovery, because the watchdog never
fired.

I also need to correct one conclusion from my initial local review. The July 17
captures moved from mx.async_eval to a later invocation's forward pass and
the server emitted intervening prompt-progress updates, so
BatchGenerator.next() did return during the captured livelock. A separate
blocking-next() probe identifies an additional boundary in #1598, but it does
not characterize the historical #1493 incident. I have put the detailed test
results and implementation questions on #1598.

If a longer soak would help, I can run the PR build on the reviewer instance
with --generation-stall-timeout 60 and report the next real occurrence. I
also retained the original capture bundle and can provide a redacted subset.