Skip to content

Commit 2b305c2

Browse files
committed
skill: the lm-head requant is implemented; prefill hang is the open blocker
1 parent e728f88 commit 2b305c2

1 file changed

Lines changed: 37 additions & 31 deletions

File tree

‎.agents/skills/open-gguf-direct/SKILL.md‎

Lines changed: 37 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -465,7 +465,7 @@ a warm rebuild was `cmp`-identical. Root cause and fix above.
465465
completes without core-body cooperation. Stage isolation by stream
466466
truncation does NOT work here.
467467

468-
### The lm-head container gap (found + gated 2026-10-22)
468+
### The lm-head container gap (closed 2026-10-22)
469469

470470
With the sampler fixed, decode ran — and produced ~4.8 tok/s of garbage: ids
471471
248068/271/248069/271 (the think template, correct) then a fixed cycling
@@ -475,36 +475,42 @@ rows are NaN**, all within the first 191,736 ids, ~25% of them, no row
475475
periodicity (mod 2/3/4/5/8 flat) — a garbage scale read, not a per-group
476476
bug. Everything past file coverage recovers the memset zeros.
477477

478-
Root cause: pool pack for the hybrid family. The pool's projection tensors
479-
go through `std_perm_gguf` (which dequantizes/requantizes Q4_K/Q6_K/etc.
480-
into the pool law), but the head op is `lmhead_q8` (`src/open_qwen36/
481-
pools.cpp` ~773), a **plain memcpy in the kernel's supertile order** — it
482-
assumes `lm_head.weight` (= gguf `output.weight`) already holds the engine's
483-
q8 chunk raster: 32-row × 256-column chunks, 256 bf16 scales then 8192 int8
484-
codes, at pool chunk k ← file `(4*(k/per_band) + k%4)*nk + k%per_band/4`,
485-
`per_band = 4*nk` (nk = in_dim/256). Only a q4nx-build conversion writes
486-
that (Q8NX layout). A raw llama.cpp GGUF stores the head row-major in its
487-
own quant; Ornith-1.5's `output.weight` is **Q6_K** (4096 × 248320),
488-
`Qwen3.8-9B-Distill-GGUF`'s is whole-file **Q8_0** (byte count even lands
489-
exactly on the chunk grid — 1,080,688,640 = 124,160 × 8,704 — but row-major
490-
raster + fp16 scales, still wrong). The op's `bounds()` silently passed a
491-
short count and filled the pool with misread bytes.
492-
493-
Gate (now in `lmhead_q8`): a `GgufFile` source is refused by name + type
494-
("convert with q4nx-build"); also `n % CH8 != 0` refuses non-chunk-aligned
495-
survivors. Verified: ornith (Q6_K) and qwen3.8-distill (Q8_0) both fail
496-
cleanly at load; the q4nx path (`qwen35-npu2:0.8b` → full coherent
497-
"**Paris**." turn) is untouched. `gguf_pack_test` + the registered ctest
498-
(GGUF-PACK, OPEN-MANIFEST) pass.
499-
500-
**Still missing**: the raw-GGUF q8 head requant — the analogue of
501-
std_perm_gguf's kq path for the q8 law. Q8_0 needs no arithmetic at all (its
502-
per-32-block fp16 d IS the group scale — remap into the bf16 scale plane
503-
with RNE and apply the supertile permutation); Q6_K/Q4_K need
504-
dequant-row → per-32-group requant of int8 code + bf16 scale, matching
505-
`bf16_rne` semantics used elsewhere in this file. The head requant is our
506-
open item on the hybrid family's GGUF path; while it is open, the packer
507-
refuses and a hybrid GGUF needs a q4nx-build-converted container.
478+
Root cause: the head op `lmhead_q8` (`src/open_qwen36/pools.cpp` ~773) was a
479+
plain memcpy in the kernel's supertile order — it assumed `lm_head.weight`
480+
(= gguf `output.weight`) already held the engine's q8 chunk raster: 32-row ×
481+
256-column chunks, 256 bf16 scales then 8192 int8 codes, at pool chunk
482+
k ← file `(4*(k/per_band) + k%4)*nk + k%per_band/4`, `per_band = 4*nk`
483+
(nk = in_dim/256). Only a q4nx-build conversion writes that (Q8NX layout).
484+
A raw llama.cpp GGUF stores the head row-major in its own quant; Ornith-1.5's
485+
`output.weight` is **Q6_K** (4096 × 248320), `Qwen3.8-9B-Distill-GGUF`'s
486+
is whole-file **Q8_0** (byte count even lands exactly on the chunk grid —
487+
1,080,688,640 = 124,160 × 8,704 — but row-major raster + fp16 scales,
488+
still wrong). The op's `bounds()` silently passed a short count and filled
489+
the pool with misread bytes.
490+
491+
Implemented (`lmhead_q8`, GGUF branch): the same transcode the pool ops do.
492+
- Q8_0: pure remap — the source already carries per-32-block int8 codes
493+
and an fp16 scale, so widen fp16→bf16 (RNE, matching the converter's
494+
cast) and write into the chunk raster at the plain (row-major) index
495+
`(row/32)*ncol + kt` — a raw GGUF is row-major, so no supertile
496+
permutation (the memcpy path permutes because the q4nx container stores
497+
the head in supertile order). Bit-identical to a q4nx conversion of the
498+
same Q8_0 tensor.
499+
- anything else (Q6_K, Q4_K, Q4_0, ...): dequant each row to f32 (the
500+
same llama.cpp-faithful path `std_perm_gguf` uses for the pool tensors)
501+
and requant to int8 code + bf16 scale per 32-value group
502+
(`bf16_rne(mx/127)`, clamp to [-127,127]), at the same plain index.
503+
Verified: a fresh CLI run on ornith-gguf:9b-i1 now reports **0 NaN logit
504+
rows** (was 47,695) with a real argmax; `gguf_pack_test` + the registered
505+
ctest (GGUF-PACK, OPEN-MANIFEST) pass; the q4nx path is untouched
506+
(`qwen35-npu2:0.8b` still answers a full coherent "**Paris**." turn).
507+
508+
Open: the 9B hybrid GGUF path **hangs at prefill layer 30 (full_attention)**
509+
on every run — it reproduces with both the f32 and the q4_1 law, so it is
510+
not the head and not the law; it is a per-layer slice issue on the 9B hybrid
511+
path (the 0.8B native and the dense GGUFs run cleanly) that needs its own
512+
investigation. Until that is resolved, hybrid GGUF-direct does not complete
513+
a turn on this box.
508514

509515
## Still open (Phase-MoE / later)
510516

0 commit comments

Comments
 (0)