server: any uncaught exception in _generate leaves HTTP threads serving while every completion hangs forever

[pengying]

Environment: mlx-lm 0.31.3, mlx 0.32.0, macOS 26.5.2 (M3 Max, 36GB), Python 3.13.11, mlx_lm.server --prompt-concurrency 2, model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit.

Symptom (design issue): when Thread-1 (_generate) dies from any uncaught exception (we have observed several distinct ones in production under sustained batched load: TypeError: 'NoneType' object is not iterable from batch-merged logits_processors, RuntimeError: [metal::malloc] Resource limit (499000) exceeded per #1332, ValueError: Slice indices must be 32-bit integers, and a mask-vs-empty-KV broadcast_shapes error), mlx_lm.server keeps accepting connections:

• GET /v1/models returns 200
• every POST /v1/chat/completions hangs indefinitely
• nothing is logged beyond the single traceback; no 5xx, no exit

Health checks pass while the server is unrecoverable — a zombie. Before we added an external watchdog we measured real outages of multiple hours per occurrence; with a probing watchdog it is still ~6 minutes of silent hang per event.

Suggested fix (any of):

1. Exit the process when the generation thread dies — supervisors (launchd/systemd) restart it. This is what we do today via a threading.excepthook wrapper; recovery drops to ~40s.
2. Restart the generation thread and fail all pending requests with 503.
3. At minimum, make /v1/models (or a health endpoint) fail once the generation thread is gone, so probes can detect the state.

This issue multiplies the severity of every generation-side bug: with it fixed, the crashes above would be brief, detectable blips instead of silent outages. Happy to provide full tracebacks for each of the observed death modes or test a patch.

[rajanshxrma]

Verified this end-to-end on current main (M1, 16GB, Llama-3.2-1B-Instruct-4bit, --prompt-concurrency 2) — and it's worse than "several distinct exceptions": one of them is triggerable with just two plain HTTP requests. A request without sampling penalties entering the prompt batch alone, followed ~1s later by one with repetition_penalty, reproduces your TypeError: 'NoneType' object is not iterable deterministically when the two co-finish prefill: PromptProcessingBatch.extend plants None placeholders for logits_processors (generate.py:1149-1156), and GenerationBatch._step iterates them (generate.py:1428).

After the thread dies I see exactly the zombie state you describe: both requests hang, every subsequent request blocks forever in ResponseGenerator.generate() on a queue nothing services, HTTP threads pile up, and /health keeps returning {"status": "ok"}.

Fix is up in #1513: empty-list placeholders in extend (root cause of this specific crash), plus the design fix — per-request exception handling in the batch admission path, fail-in-flight-and-reset on a serving failure so the generation loop survives (which also covers your [metal::malloc] case), and a fail-fast in generate() when the thread is gone so a dead worker can never again turn into infinite client hangs. Regression tests for both included.

[TadeuszWolfGang]

Independent validation cross-reference: I reproduced the deterministic
mixed-logits_processors failure on pre-fix commit 15b522f
(TypeError: 'NoneType' object is not iterable) and confirmed that the stacked
#1513 code completes both request UIDs. I posted the full A/B method and scope
on #1513.

This was an API-level BatchGenerator probe, not a two-request HTTP replay, and
I did not inject a live [metal::malloc] Resource limit failure.

[TadeuszWolfGang]

Independent confirmation of this failure mode on different hardware, in case a
second data point is useful.

Same exception you list, verbatim including the number:
RuntimeError: [metal::malloc] Resource limit (499000) exceeded, raised from
mx.async_eval in GenerationBatch._step and escaping _generate at
server.py:853. Apple M5 Max / 128 GB, macOS 26.5.2, Python 3.12.13, stock
mlx-lm 0.31.3 / mlx 0.31.2, mlx-community/Qwen3.6-27B-8bit,
--decode-concurrency 64 --prompt-concurrency 16, ordinary chat traffic, no
provocation.

Measured on the still-zombied process before restarting it:

• GET /v1/models → 200 in 0.021 s
• 4-token POST /v1/chat/completions → no response in 60 s
• py-spy dump → no Thread-1 (_generate) thread in the process at all;
  MainThread still in serve_forever; request threads parked forever in
  Queue.get()

Four occurrences on this host since 2026-07-05, all four with byte-identical
tracebacks:

Died after	Restarted	Outage	Served while zombied
2026-07-05 07:07:40	2026-07-07 11:42:03	~52 h 34 min	5× /v1/models 200, 1× /health 200
2026-07-21 11:12:44	2026-07-21 17:14:42	~6 h 01 min	7× /v1/models 200
2026-07-27 08:00:05	2026-07-28 10:41:33	~26 h 41 min	1× /v1/models 200
2026-07-30 ~16:40	2026-07-30 22:15:30	~5 h 35 min	2× /v1/models 200

That matches your "multiple hours per occurrence". These instances run under
launchd with KeepAlive enabled, which cannot help: the process never exits,
only the thread does. Every recovery above was a manual or scheduled process
restart. The GET /health 200 during the first window is consistent with your
point 3 — the control endpoints stay green while the server is unrecoverable.

One detail that may help anyone diagnosing this from logs, and which explains
your "nothing is logged beyond the single traceback": a completion submitted
after the death blocks in ResponseGenerator.generate() at server.py:1048,
which handle_completion reaches at line 1419, before the header send at
1431–1437. BaseHTTPRequestHandler.log_request therefore never runs for it and
no POST /v1/chat/completions line is written at all. The absence of POST lines
in a zombie window understates how many requests hung rather than showing there
were none. Requests admitted before the death park one frame deeper, at
server.py:1037, and those do have a logged POST … 200 line from when their
stream headers went out.

Full capture — py-spy dump, log excerpts, probe transcripts, code proof — is
written up on #1513, since that is the PR addressing this. Happy to share a
redacted subset or run a patched build here.

[Avicennasis]

Another independent reproduction, plus a downstream workaround that has been running in production for two weeks in case it helps anyone until one of the PRs lands.

Environment: mlx-lm 0.31.3 (git cf10f962), mlx 0.32.0, macOS 26.6.2, M1 Max 64 GB, mlx_lm.server defaults (--decode-concurrency 32 --prompt-concurrency 8), Qwen3.8-27B-8bit and Qwen3.6-27B-8bit.

Same failure, measured cost: Thread-1 (_generate) dies with RuntimeError: [metal::malloc] Resource limit (499000) exceeded. from generate.py:1451 (mx.async_eval in _step), the process stays up, /v1/models keeps returning 200, and every completion blocks at the untimed response_queue.get() (server.py:997). One box served in that state for ~26 hours before anyone noticed, because the only health probe it had was /v1/models. Three occurrences across two machines before we intervened.

What triggers it for us, quantified: the crash itself is #1845 / #1662 — ArraysCache.advance() pinning one Metal buffer per linear-attention layer per decode step through BatchGenerator. With the tiny-model repro from #1845 on this hardware: cf10f962 dies at 14,216 generated tokens, current main (32bb4e68, post-#1632) at 14,217, and main + #1845's proposed advance() deferral runs flat to 20,000. For a 48-linear-layer 27B that is ~10,617 tokens in one decode, which our uncapped (--max-tokens 32768) thinking-mode requests reach routinely; a real 27B decode died at ~10,554 on the same box today.

Workaround (until the worker is guarded upstream): a sitecustomize.py on PYTHONPATH that installs a threading.excepthook, recognises the _generate worker's death (frame in mlx_lm/server.py named _generate, or the default Thread-N (_generate) name), chains to the previous hook so the traceback still reaches the log, and calls os._exit(70). Under launchd KeepAlive (or any supervisor) the silent permanent wedge becomes a ~30 s restart. It has a kill switch and a negative-control test (a benign thread death must not exit). Not a fix — the decode is still lost — but it turns "false-healthy for a day" into "restarted before the next request." We now also carry #1845's deferral the same way, which removes the trigger.

Happy to provide any of the measurements above in more detail.