HRX: dense on HRX0 with experts on another device fails for Qwen MoE models (fused router)

Found while measuring the MoE three-way split (docs/moe-streaming.md), llama.cpp pin 96f6b89 / 5556bf2, Strix Halo:

llama-bench -dev HRX0/Vulkan0 -ts 1/0 -ot exps=Vulkan0 (dense on HRX0, routed experts on Vulkan0):

model	result
Qwen3-Coder-30B-A3B Q4_K_M	decode fails: `binding route_ids ... range=[0, 1024) is outside runtime binding length 544`
Qwen3.6-35B-A3B UD-Q5_K_XL	decode fails
GLM-4.7-Flash Q4_K_M (sigmoid router, no Qwen fusion)	runs, 25.2 tok/s

Binding length. The Qwen router dispatch (dispatch-moe-router.cpp, two places) binds token_count * route_stride route ids. When the experts are on another device, the ids view becomes a graph output whose buffer ends at (token_count - 1) * route_stride + route_count ids. Binding that length removes the error. This is the same bound HRX: chat on Qwen3-Coder-30B-A3B and GLM-4.7-Flash fails with 500 "Compute error." #95's MUL_MAT_ID fix uses.
Wrong output. With (1) fixed, the answer is wrong ("Paris 8T inalytics oneer..."). With GGML_HRX_DISABLE_DISPATCH=moe_router, it goes back to "Compute error.", because fused_context_claim claims the router chain only for the fused dispatch. The fused Qwen router seems to assume its consumers (MUL_MAT_ID and the weighted sum) are on HRX0, so the tensors the other device reads are not all materialized.

Not blocking. All-Vulkan beats every split measured (Coder 92.4 tok/s vs 27.0 with the experts on HRX0), so nothing in the engine uses this split today.

🤖 Generated with Claude Code

Bug 1 (route_ids binding length crash) fixed: 1bit-MONSTER/llama.cpp#7, mirrors the same bound #95 uses for MUL_MAT_ID (dispatch-mul-mat-id-common.h). Not execution-tested yet (device was busy; no built tree in a fresh worktree) - review/CI pending.

Bug 2 (wrong output once decode stops crashing) is still open and not root-caused. The fused router chain (fused-context-claim.h) has no device-awareness at all - it is a pure per-op shape/pattern predicate deciding whether HRX claims a node, with no notion of which backend the consumers live on. Once bug 1 lands, the next step is live multi-device debugging to find which specific intermediate tensor(s) the fused dispatch does not materialize for a cross-device MUL_MAT_ID/weighted-sum consumer - not something to guess at without hardware validation.

🤖 Generated with Claude Code

Update, live-tested with the binding-length fix (#7) applied, HRX+Vulkan build from 1bit/hrx-vulkan-patched @ 83ea7be3e:

The wrong-output bug (bug 2) is not specific to the cross-device split. HRX0 alone (no split, no Vulkan, -dev HRX0 -ngl 99) already miscomputes Qwen3-Coder-30B-A3B-Instruct-Q4_K_M, greedy/temp 0:

config	"What is the capital of France?"	"Name the largest planet..."
Vulkan0 alone	`The capital of France is Paris.` (correct)	`Here's a simple Python function...` (correct, different prompt)
HRX0 alone	`The capital of France. is one short sentence. Answer.............................`	`Jupiter is largest planet in our solar system.` then trails into `...............`
HRX0/Vulkan0 split (`-ot exps=Vulkan0`)	`The the{(r CapSF Andres self,ORG 아주 the books_hiN Varihil from a回Fx...)` (word salad)	not retested this round

HRX0-alone gets the right fact (Paris, Jupiter) but drops words ("the"), truncates the natural sentence, and degenerates into a repetition loop - a real, reproducible, early-position deviation, not a coincidental greedy-decode artifact. The split is qualitatively worse (outright word salad), consistent with an additional, separate materialization problem on top of this one when experts move to another device - but this HRX0-alone result means there are (at least) two distinct problems, not one.

Isolated to the Qwen top-8 fused router specifically. GLM-4.7-Flash-Q4_K_M (sigmoid router, no Qwen fusion) on HRX0 alone produces clean, coherent, correctly-structured chain-of-thought text (cut off by max_tokens mid-reasoning, not garbled) - matching the original report's "GLM-4.7-Flash ... runs, 25.2 tok/s" note. So this is not a general HRX correctness problem: it is specific to the match_moe_router_top8_dispatch / match_moe_router_projection_top8_fused_decode_dispatch fused chain (dispatch-moe-router.cpp) or its consumers, and reproduces with no device split at all.

This changes where to look next: the router's own kernel math (SOFT_MAX -> ARGSORT -> GET_ROWS -> SUM_ROWS -> CLAMP, or the top-8 selection/normalization it feeds into MUL_MAT_ID) needs auditing against the reference CPU path for Qwen3's specific route_count/expert_count/route_stride values, independent of any cross-device work. The cross-device materialization question from the original report is still open too, but is now known to be layered on top of this more fundamental bug, not the sole cause.

Not yet done: bisecting which of the five fused ops (or which specific value: expert ids, routing weights, or their normalization) is wrong; a numeric comparison against the CPU reference for the same prompt/positions (I only compared decoded text, not raw router outputs).

🤖 Generated with Claude Code

Both bugs root-caused and fixed - PR 1bit-MONSTER/llama.cpp#7 (targets 1bit/hrx-vulkan-patched, 2 commits, not merged).

Bug 2's actual cause was not what my last comment guessed: it's not (only) a cross-device materialization gap. qwen3_moe_router_projection_top8_fused_decode_f32's launch-config hardcodes experts_per_wave = 4, but the projection template it dispatches to resolves to a 2-experts-per-workgroup variant on gfx1151 specifically (our exact target). expert_count/4 workgroups x 2 experts each covers only half the experts - the top half of every decode step's router logits row is stale/uninitialized on this hardware, independent of any device split.

Verified end to end (Strix Halo, Qwen3-Coder-30B-A3B-Instruct-Q4_K_M, greedy decode): -dev HRX0 alone now matches Vulkan0 exactly, and the original -dev HRX0/Vulkan0 -ts 1/0 -ot exps=Vulkan0 repro now gives correct answers on 3 tested prompts (previously outright word salad). Full write-up and the bisection method that found it are in PR #7's description.

I noticed ~/1bit-engine running a parallel qwen-allhrx-nofused investigation of the same issue while I was working this - if that's still in flight, the fix above should make it unnecessary, but flagging in case findings diverge.

🤖 Generated with Claude Code

Verified on the second model the report names, Qwen3.6-35B-A3B-Q8_0, with PR #7's fix (both commits):

llama-bench -dev HRX0/Vulkan0 -ts 1/0 -ot exps=Vulkan0 -p 512 -n 8: no crash, pp512 422.18 tok/s, tg8 28.67 tok/s.

llama-server with the same split, two prompts (greedy, cache_prompt: false, enough max_tokens for this model's own thinking length - it's a reasoning model, needs 300-600 tokens to close its <think> block on a short answer): both correct.

"What is the capital of France?" -> content The capital of France is Paris., coherent reasoning.
"Name the largest planet..." -> content Jupiter is the largest planet in our solar system., coherent reasoning.

Confirms the launch-geometry root cause is general (it's about gfx1151's launch geometry, not this model's specific data) rather than something specific to Qwen3-Coder-30B-A3B.

🤖 Generated with Claude Code

Long-context and multi-turn verification (PR #7's fix, Qwen3-Coder-30B-A3B-Instruct-Q4_K_M, -dev HRX0,Vulkan0 -ts 1,0 -ot exps=Vulkan0):

Long-context retrieval (1470 tokens): a synthetic ~1200-word document with a code word ("XYLOPHONE-7") buried in the middle, followed by "what is the secret code word?" - answered exactly correctly.

Multi-turn (3 turns, split config): told the name "Alex" and a cat "Whiskers" (turn 1), added a dog "Rex" and asked for both pet names (turn 2, correct), then asked for the name plus a one-sentence story using both pet names (turn 3) - got Your name is Alex. Alex came home from work to find Whiskers the cat perched on the windowsill and Rex the dog excitedly running toward the door.... Fully correct, coherent, synthesized all three turns' facts.

Found a separate, pre-existing, unrelated bug while pushing context length further - flagging it here since it surfaced in the same testing, but it is NOT the router bug and NOT introduced by this PR:

At key_value_token_count around 4864 (a ~3000-word/~3800-token document), decode fails with Compute error.:

HRX Loom JIT Loom compilation failed: select-templates cannot resolve template.apply
against template family <ggml.flash_attention.decode_split.reduce_fused>: all_rejected
compile ... ggml_flash_attention_decode_split_f32_f16_wmma_next_q8 ... key_value_token_count=4864 ...

Reproduces identically with no split at all (-dev HRX0 alone) - so it's a general HRX flash-attention "decode split" (long-KV-cache) template-resolution gap, not related to the MoE router, the cross-device split, or Qwen specifically. Confirmed working at 1470 tokens, confirmed broken somewhere between ~1470 and ~3800 tokens for this exact model/config. Might be worth its own issue if it isn't tracked already - not investigated further here, out of scope for this one.

🤖 Generated with Claude Code

Navigation Menu

Repository navigation

HRX: dense on HRX0 with experts on another device fails for Qwen MoE models (fused router) #108

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: dense on HRX0 with experts on another device fails for Qwen MoE models (fused router) #108

Description

Activity

bong-water-water-bong commented on Sep 26, 2026

bong-water-water-bong commented on Sep 26, 2026

bong-water-water-bong commented on Sep 26, 2026

bong-water-water-bong commented on Sep 26, 2026

bong-water-water-bong commented on Sep 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions