linalg: thread long-k GEMVs below the MMM panel gate

[czoli1976]

Problem. The rayon MMM gate counts panels only. A decode GEMV uses the wide 64x1 kernel, so every 2048-row projection of Llama-3.2-1B (q, o and down, k up to 8192) is 32 panels, under the gate of 64, and runs on one thread. Multithreaded decode scaled about 1.8x on the M1 Max nightly.

Fix. A GEMV also threads once its multiply-accumulates reach set_threading_mac_threshold (default 2^21, off on wasm, where dispatch costs ~10x more). Matrix-matrix products keep the panel gate alone, and anything threaded today still is.

Perf. Llama-3.2-1B-Instruct q40ef16, llm-bench, tok/s, 2 interleaved rounds, main 2275e7c9c:

device	threads	TG128 main	TG128 PR	PP512 main	PP512 PR
M4	4	47.4	63.2 (+33%)	336	336
M4	all (10)	51.3	70.2 (+37%)	432	426 (noise)
M1 Pro	6	37.5	51.6 (+38%)	362	370
M1 Pro	all (8)	37.1	50.5 (+36%)	398	383 (noise)

No regression on CI's multithreaded nets (mobilenet_v2, inceptionv3) or an 11-model noise-suppression panel (FastEnhancer t/b/s/m/l, GTCRN, DTLN 1/2, DeepFilterNet3 x3) at 4 threads and all cores: every row within ±1.3% on M4 (3 rounds, medians), ±3% on M1. Single-thread is untouched.

🤖 Generated with Claude Code

🍍

[czoli1976]

@kali lets see if we can land this one next

[github-actions]

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 4 secondary

Reference: 2026-10-13 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +6.3%	trunet evaltime · pulse1_f16	cortex-a7	80.2 ms/pulse 8.02 RTF → 85.2 ms/pulse 8.52 RTF

Improvements

Δ	metric	device	main → PR
🟢 -27.0%	llama_3_2_1B_instruct_q40ef16_541 decode · cpu_mt	apple-m1-max	25.4 ms/tok 39.4 tok/s → 18.5 ms/tok 53.97 tok/s
🟢 -8.4%	en_tdnn_15M evaltime · pulse_240ms	orangepi-rv2	72 ms/pulse 0.3 RTF → 65.9 ms/pulse 0.275 RTF
🟢 -7.6%	en_tdnn_15M evaltime · 2600ms	cortex-a55	627 ms → 580 ms

⚠️ 4 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +18.3%	hey_snips_v31 load · 400ms	cortex-a7	334 ms → 395 ms
⚠️ +17.7%	hey_snips_v31 load+optimize · 400ms	cortex-a7	384 ms → 452 ms
⚠️ +9.5%	hey_snips_v1 load · 400ms	cortex-a7	63 ms → 69 ms
⚠️ +6.1%	llama_3_2_1B_q40ef32_516 load+optimize · cpu	apple-m1-max	4.56 s → 4.84 s

---

Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.