Repository navigation
Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI, exact on 18 models - #19
Merged
Merged
Conversation
bong-water-water-bong
force-pushed
the
hf-tokenizers
branch
2 times, most recently
from
September 23, 2026 20:45
f991cbd to
d7b2d8d
Compare
…e-exact on 18 models third_party/tokenizers = huggingface/tokenizers v0.23.2 (88a4498a), the release the tokenizers npm/PyPI packages ship. hf_tokenizers/ is our C ABI (from_json/encode/decode/vocab_size/last_error) built as a static lib (onig only), with Rust pinned (rust-toolchain.toml 1.98.1) and Cargo.lock (--locked). -DONEBIT_HF_TOKENIZERS=ON builds it; hf_tokenizer_test runs golden cases made by the same Python release for 18 models under ONEBIT_TOKENIZER_ROOT. bump-tokenizers.yml keeps it current. Verified: 18/18 models x 17/17 cases identical ids and decoded text (vocab 64k-262k: Qwen3/3.5/3.6/VL, Llama 3.1/3.2, Gemma3/4, Phi4-mini, LFM2, Nanbeige); the engine's Qwen golden 14/14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…exempt) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
force-pushed
the
hf-tokenizers
branch
from
September 23, 2026 20:55
d7b2d8d to
1ef449d
Compare
This was referenced Sep 26, 2026
bong-water-water-bong
pushed a commit
that referenced
this pull request
Sep 26, 2026
Bumps third_party/llama.cpp-vulkan to 57c22a3 (fork PR #19, 1bit/moe-stream). ExpertCache takes caller-owned slot memory (CacheOptions::region), as the fork's copy does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Sep 26, 2026
* serve: --moe-slots streams MoE experts from the model file (Vulkan) Bumps third_party/llama.cpp-vulkan to 57c22a3 (fork PR #19, 1bit/moe-stream). ExpertCache takes caller-owned slot memory (CacheOptions::region), as the fork's copy does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: --moe-slots in serve, streaming in the inference path measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 21, 2026
…k skipping (Qwen3-4B pp4096 +45%) (#491) gpu-kernels 08bb2241 adds, since e85b7547: - #19: W4A4 prompt prefill on the iu4 WMMA (Q4_0, off by default). - #20: fa_prompt_f16_d128, a WMMA prompt flash attention for f16 KV caches, head 128, that skips KV blocks whose mask is -inf for a whole query tile; no 2048-row limit. Measured on strixhalo gfx1151, performance mode, private build (llama.cpp 5dcf6623 + hrx-system 82804197 + the add-on): Qwen3-4B Q5_K_M 4096-token prompt: attention 916.5 -> 296.0 ms at ub 2048; llama-bench pp4096 ub2048 2103.0 -> 3043.8 (+44.7%), pp4096 ub512 +6.3%, pp2048 ub2048 +7.5%. Qwen3.8-27B unchanged (interleaved x3, within +-0.3%). Correctness: fa-prompt-bench vs a double reference PASS on 7+ shapes; KLD vs the Loom kernel 4B 0.000002, Coder-30B-A3B 0.000912, ZAYA1-8B 0.007729 (top-1 routing amplifies it; against a CPU reference the new kernel is as close as Loom: ZAYA ub128 0.0412 vs 0.0417, Qwen3-1.7B 0.007708 vs 0.007716). scripts/hw-gate.sh --bin (ops,kat,kld,needle,placement) on that build: PASS on every row except kat zaya1-8b varied-1620w, which diverged at token 5 - the near-tie noted in #487 (the answer is unchanged, 7341). tests/golden/hw-gate/kat-zaya1-8b.json re-recorded at this pin (scripts/hw-gate.sh --phases kat --models zaya1-8b --record --bin <that build>). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 21, 2026
bong-water-water-bong
pushed a commit
that referenced
this pull request
Oct 30, 2026
…tree Owner decision on this PR: the probe microkernel and the perf records are kernel material, so they live with the rest of the ELF tree (npu-kernels) rather than in the public engine. Moved to npu-kernels (PR #19, branch arena/leaderboard-and-probe): * `npu/probe/mm_dmabw.cc` -- the pass-through AIE microkernel the ceiling probe links. * `docs/arena/LEADERBOARD.md`, `docs/arena/HARVEST.md` -- the per-phase records and the techniques/findings behind them. What stays here is host tooling and documentation only: * the arena's harness generator, runner, oracle, task/probe builders and ceiling driver, which reference the private tree by path (`NPU_KERNELS`/`PROBE_SRC`/`GENERATORS`) and carry no kernel source; * the README, now describing the method instead of our measured tables; * `q4nx_weight_layout.h` (the layout contract, mechanism only -- the numbers moved to the records) and its CI test. Also applies tools/copyright.py --fix to the new files (the build's copyright check was failing on them).
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 30, 2026
…for the layout contract (#668) * tools/npu-arena: the kernel arena, its DMA-ceiling probe, and a test for the layout contract The arena that produced the last two PRs was a private directory. It belongs in the repo: it is the harness every future kernel change is scored with, and the numbers it produces only mean something if anyone can rebuild it. tools/npu-arena/ is that harness, self-contained and box-agnostic (paths resolve from the environment, or from a git-ignored TOOLCHAIN.lock; TOOLCHAIN.lock.example documents the pins): * `harness/gen.py` freezes a per-phase harness: A (activations) int8, B int8 in the engine's packed layout, the expected int32 = the exact integer matmul, per-column scales for provenance, and SHA256SUMS. Three seeds, ternary with the container's real per-32-row-group magnitude structure, so a lossless 2-bit packing stays a legitimate entry. * `runner/runner.cpp` is the on-device scorer: one full ELF, the frozen harness, a byte-exact int32 check (two consecutive clean reads, because the runtime's readback is occasionally short -- engine issue #666), median of N launches. It can also score a submission's own weight layout (`--w`) and skip the check for a probe (`--no-check`). * `make_task.sh` + `oracle.sh` are the competition: a task bundle per phase, and an oracle that rebuilds a submission from source with the pinned toolchain, refuses a shared device, replays the leader, and prints the leaderboard row on ACCEPT. * `make_probe.sh` + `probe/mm_dmabw.cc` + `ceiling.sh` are phase 1 of the 2026-10-30 14:58 plan: the phase's own objectfifos, taps, blocked layout, tile geometry and column-group loop with the matrix unit replaced by a pass-through microkernel, so a launch measures the data path alone. Two variants per phase (8-core M=16 / 32-core M=64 with 4 core rows) and a CEILING.md report, so every entry can be judged as a percentage of what this box can actually stream. * `LEADERBOARD.md` and `HARVEST.md` carry the records: the blocked-B layout is entry #2 on every phase (2.4-2.6x, since merged as #667), and what the numbers still owe (the 32-core N-partition, the 2-bit ternary stream). Also moves the blocked layout's index math out of the runtime header into npu/forward/include/q4nx_weight_layout.h, and tests it: tests/npu_blocked_layout_test.cpp proves the mapping is a bijection over six geometries (including the 27B's 5120x17408 and 17408x5120), that each 64x128 tile occupies one contiguous 8192-byte window with the tiles k-fastest, that element (i,j) inside a tile lands where the int8 microkernel reads it (b_in[((kb*16+nb)*8+r)*8+c], an independent restatement of its "B LAYOUT silicon-verified" note), that a geometry which is not a whole number of tiles is refused, and that the row-major layout every earlier container uses has not moved. Pure index math, no device and no runtime, so it runs in CI. No kernel binaries: the ELFs stay private and are emitted from source with the pinned toolchain (the same rule as the rest of the NPU artifacts). * tools/npu-arena: split the kernel material out to the private kernel tree Owner decision on this PR: the probe microkernel and the perf records are kernel material, so they live with the rest of the ELF tree (npu-kernels) rather than in the public engine. Moved to npu-kernels (PR #19, branch arena/leaderboard-and-probe): * `npu/probe/mm_dmabw.cc` -- the pass-through AIE microkernel the ceiling probe links. * `docs/arena/LEADERBOARD.md`, `docs/arena/HARVEST.md` -- the per-phase records and the techniques/findings behind them. What stays here is host tooling and documentation only: * the arena's harness generator, runner, oracle, task/probe builders and ceiling driver, which reference the private tree by path (`NPU_KERNELS`/`PROBE_SRC`/`GENERATORS`) and carry no kernel source; * the README, now describing the method instead of our measured tables; * `q4nx_weight_layout.h` (the layout contract, mechanism only -- the numbers moved to the records) and its CI test. Also applies tools/copyright.py --fix to the new files (the build's copyright check was failing on them). * npu: make the layout test compile itself, so CI actually runs it CI builds an explicit target list and then runs `ctest`, so a registered test whose binary is not on that list reports "Not Run" and never executes -- which is exactly what happened to the layout test (test 41, 46 tests, 1 failed, exit 8). Editing that workflow list needs a token scope this lane does not have, and a test of pure index math does not need a build target at all: `tests/npu_blocked_layout_test.sh` compiles tests/npu_blocked_layout_test.cpp with the build's own compiler at test time and runs it. add_test() therefore passes the compiler, the include dir and the source. It uses nothing else: no runtime, no device, no NPU toolchain, so it passes on the hosted runner. Verified with the runner's own configure (cmake -G Ninja, gcc-16/g++-16, no NPU options): `ctest -R npu_blocked_layout` -> Passed 0.21s. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Any model's
tokenizer.json, through Hugging Facetokenizersitself. This is the Rust library behind thetokenizersnpm and PyPI packages, at the same release (v0.23.2).third_party/tokenizers=huggingface/tokenizersv0.23.2 (88a4498a). There's no official C binding upstream, and mlc-ai/tokenizers-cpp is stuck on 0.21.2.hf_tokenizers/is our own C ABI over it (about 100 lines of Rust), plus a small C++ owner,onebit::HfTokenizer. It builds as a static library with only theonigfeature.rust-toolchain.toml(1.98.1) and the crate graph inCargo.lock(--locked).-DONEBIT_HF_TOKENIZERS=ONbuilds it. With-DONEBIT_TOKENIZER_ROOT=<model dirs>, ctest runs one golden test per model.tools/make_tokenizer_golden.pymakes the goldens with Pythontokenizers==0.23.2. The corpus covers eight scripts, code, numbers, whitespace runs, emoji with ZWJ and chat markup.bump-tokenizers.ymlmoves the pin to each new release and refreshes the lock.Verified
18/18 models, 17/17 cases each: identical ids and identical decoded text. The models are Qwen3 0.6B/1.7B/4B/8B, Qwen3-VL-4B, Qwen3.5-4B, Qwen3.6-35B-A3B, Llama 3.1-8B, 3.2-1B and 3.2-3B, Gemma3 1B/4B, Gemma4 E2B/E4B, Phi4-mini, LFM2 1.2B/2.6B and Nanbeige4.1-3B, with vocabularies from 64k to 262k. The engine's own Qwen golden also matches, 14/14. That ran through the real CMake build:
ctest -R hf_tokenizerpasses 18/18.🤖 Generated with Claude Code