[bong-water-water-bong]

What

tools/q4nx_to_gguf.py writes a GGUF that holds exactly the weights in a model.q4nx, so a GPU decoding the GGUF and the NPU running the Q4NX compute the same model. It is the inverse of tools/gguf_to_q4nx.py.

• write <model.q4nx> <source.gguf> <out.gguf>: q4_1 tiles become GGML Q4_1 blocks. They keep the same 4-bit codes, and d/m are the tile's bf16 scale and zero converted to f16. The tool stops if any value would change in that conversion. Norms are written as F32. The embedding is F16 when every value is exact in f16, otherwise F32. lm_head is written as output.weight (gguf_to_q4nx.py quantizes a tied head, so it is no longer the embedding). Metadata and tokenizer are copied from the source GGUF, and general.file_type is set to MOSTLY_Q4_1.
• verify <model.q4nx> <out.gguf> [--roundtrip <source.gguf>]: checks dequant(out.gguf) == dequant(model.q4nx) on every tensor, and with --roundtrip checks that gguf_to_q4nx.py pack of the new GGUF is byte-identical to the original model.q4nx.

Why

gguf_to_q4nx.py requantizes every non-Q4 type to 4-bit q4_1. In Qwen3-0.6B-Q4_K_M that covers 28 tensors: attn_v and ffn_down in layers 0, 1, 2, 5, 8, 11, 14, 17, 20, 23, 24, 25, 26 and 27 are Q6_K. Hybrid prefill (NPU) feeding HRX decode on the source GGUF therefore mixes two different models.

Measured (Qwen3-0.6B-Q4_K_M.gguf -> model.q4nx -> this tool)

• bf16 -> f16: 0 of 192,831,488 scale, zero and embedding values change.
• 311/311 tensors dequantize identically (max |diff| 0).
• The round trip is byte-identical to the original model.q4nx.
• Gate numbers (hybrid KV gate on the new GGUF): see the comments below.

Draft: the HRX gate and the Q4_1-vs-Q4_K_M decode speed on HRX are still to be measured.

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit c1b93d6)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect tensor name mapping The logic for mapping tensor names during the write operation incorrectly assumes that when name is "lm_head.weight" and t.name is "token_embd.weight", the resulting gname should be "output.weight". However, this condition is unreachable because name is derived from the header keys, which are already in the correct form. This could lead to incorrect tensor naming and potentially break compatibility with downstream tools expecting specific tensor names. gname = t.name if name != "lm_head.weight" or t.name != "token_embd.weight" else "output.weight" Potential precision loss in bf16 to f16 conversion The function bf16_to_f16_exact checks for exact conversion from bf16 to f16, but the check only verifies if the conversion results in a different float32 value. It does not account for the fact that some values may be representable in f16 but not in bf16, which could lead to subtle differences in model behavior. This is particularly concerning for the NPU inference engine where precision matters. def bf16_to_f16_exact(bits, what): f = g2q.from_bf16(bits) h = f.astype(np.float16) bad = h.astype(np.float32) != f return h, int(bad.sum()), (float((np.abs(h.astype(np.float32) - f)[bad] / np.abs(f[bad])).max()) if bad.any() else 0.0) Possible incorrect tensor shape handling In the verify function, when handling tensors with dtype "I8", the code reshapes the reference tensor to (N, K) but does not ensure that the tensor's actual shape matches this expectation. This could lead to incorrect comparisons if the tensor dimensions do not align as expected, especially in cases where the tensor is not a 2D matrix. K, N = int(t.shape[0]), int(t.shape[1]) ref = g2q.untile(raw(name), N, K).astype(np.float32) got = got.reshape(N, K)

[bong-water-water-bong]

Measured on strixhalo with Qwen3-0.6B (hybrid KV gate: 10 prompts x {128, 512} tokens, 64 teacher-forced steps, 1280 positions, KLD(native || arm)). The NPU KV is the unchanged state files from the hybrid proof. The native side is the GGUF written by this tool, decoding along its own greedy continuation.

native decoder	NPU KV: KLD mean	p99	top-1	first token
HRX, Q4NX GGUF (this tool)	0.0002	0.0020	99.4%	100%
CPU llama-server, Q4NX GGUF	0.0012	0.012	99.1%	100%
HRX, source Q4_K_M (before)	0.0234	0.176	94.7%	95%

HRX decode speed (package 1bit-hrx-3092596, 2 x 6 runs of 128 tokens after a 512-token prompt, shared box):

model	decode best / median	prompt best
Q4_K_M	321.8 / 301.5 tok/s	19,244 tok/s
Q4NX GGUF (Q4_1)	269.5 / 250.4 tok/s	13,567 tok/s

Q4_1 runs on the GPU (it is not falling back to the CPU), but it is about 17% slower than Q4_K_M at decode and about 30% slower at prefill. ggml-hrx serves Q4_1 only through its generic quantized mul_mat, and the gpu-kernels add-on has no Q4_1 matcher.

🤖 Generated with Claude Code

[github-actions]

Persistent review updated to latest commit 0ce30d8

[github-actions]

Persistent review updated to latest commit c1b93d6