Skip to content

docs: the 21 Oct narrative (HRX + our HIP kernels + the NPU engine, inside Lemonade) #488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bong-water-water-bong merged 1 commit into from
Oct 21, 2026
Merged

docs: the 21 Oct narrative (HRX + our HIP kernels + the NPU engine, inside Lemonade) #488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 70 additions & 80 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,83 +20,73 @@ limitations under the License.

**Site:** [github.com/1bit-MONSTER/site](https://github.com/1bit-MONSTER/site) (builds 1bit.gg from this repository's README.md and docs/)

**The 1bit engine runs inside [Lemonade](https://github.com/lemonade-sdk/lemonade).** Lemonade stays
the server you talk to (its catalog, downloads, router and UI), and for the models it hands to 1bit,
Lemonade runs the engine as one of its backends, the same way it runs `llama-server`. The engine
serves each model behind an OpenAI-compatible API (`1bit serve`), whatever device runs it:

- the XDNA 2 NPU engine, on our own NPU runtime (no XRT; [docs/npu.md](docs/npu.md#the-npu-runtime)); in progress: the release carries the runtime, and the NPU kernels built from source are not merged yet
- HRX on the Radeon iGPU (AMD's ggml-hrx with our Loom kernels), the engine's one GPU route ([docs/hrx.md](docs/hrx.md)): the whole graph runs on HRX, or the model is refused with the op that would not (strict by default, [docs/serve.md](docs/serve.md#off-hrx-placement-strict-by-default)); `--device cpu` runs the same llama.cpp build on the CPU
- DwarfStar, for DeepSeek V4 Flash, GLM 5.x and Qwen3.8-Flash-Next in its own GGUFs, with SSD expert streaming
- ONNX Runtime GenAI models (Lemonade's ONNX format) on the CPU ([docs/onnx.md](docs/onnx.md))
- Laya, which decides where each request runs
- every Hugging Face model architecture, kept current by a daily census

A tuned setting `1bit serve` gives a backend is a recipe with its measurement attached
([docs/recipes.md](docs/recipes.md); the built-in set is empty until there are HRX measurements), and the serve numbers in these docs come from
`tools/bench.py`, which A/B-measures configurations against a baseline in the same run
([docs/bench.md](docs/bench.md)). HRX pin bumps and PRs labelled `hw-gate` are checked on the
Strix Halo GPU before merge by the hardware gate: kernel tests, known answers on three models,
KLD against the release pin and a 4.7K-token needle ([docs/hw-gate.md](docs/hw-gate.md)).

On the GPU the engine is pure HRX with HIP kernels; no Vulkan, ROCm backend, Metal, ZINC or CUDA.
Every other backend and platform (Vulkan, ROCm, CUDA, Apple's MLX and Metal, Windows) is Lemonade's,
through its own backends and builds.

**Install:** through Lemonade. The release is HRX with our HIP kernels and the NPU engine, all inside
Lemonade. The weekly release run (`scripts/weekly.sh`, on Sunday) rebuilds the packages at that week's
upstream pins; the first release is not published yet. The packages:
Lemonade with the onebit recipe and the engine (unpack it and run `lemond-onebit`), and 1bit OS, a
Lemonade appliance with the engine on a bootable USB stick ([docs/releases.md](docs/releases.md)).
The engine has no standalone package of its own.

> **Direction:** the engine is HRX (AMD's ggml-hrx, kernels in Loom or HIP) plus the NPU.[^geramyl] HRX met the decode gates in
> [RFC #213](https://github.com/1bit-MONSTER/engine/discussions/213), and its stage 3 removed the
> engine's Vulkan and ROCm llama.cpp builds, with `--lean`, `--adaptive`, `--long-model` and
> `--prefill-device`. What HRX does not run, Lemonade serves with its own llamacpp backends.
>
> **Status:** the engine runs inside Lemonade through `1bit serve` ([docs/lemonade.md](docs/lemonade.md),
> [docs/serve.md](docs/serve.md)); the NPU and HRX each pass its end-to-end test on
> Strix Halo (the NPU route is still in progress for the release, above), and the Lemonade recipe that runs it (`onebit`, in our fork
> [1bit-MONSTER/lemonade](https://github.com/1bit-MONSTER/lemonade)) passes Lemonade's own LLM test
> suite on HRX. Following a review,[^geramyl] the engine no longer
> vendors Lemonade: Lemonade is the host, 1bit is the engine inside it. Ported so far: HRX on AMD's live ggml-hrx,
> on AMD's `e3e6a976` hrx-system line since 2026-10-18, now hrx-system `82804197` with llama.cpp `00f2a68de` (`5dcf66234` since 2026-10-21: a stall-aware `/health` and an exit on a lost GPU for [one hop from Lemonade](docs/serve.md#one-hop-from-lemonade---print-backend), plus the census-gap fixes G4-G6; `8792923f8` since 2026-10-20: the SET_ROWS hook the private kernels' q8_0 KV cache needs, census G1; `dcf8c1ec7`: gated delta net with up to 8 sequences (census G9) and a narrower GDN claim)
> (2026-10-19, #458, #461; our four fixes replayed plus our long-context live-out fix for ROCm/hrx-system#1535;
> measured when the line moved on 2026-10-18 with llama.cpp `8d9ce2c1a`, #441, against the old pins on Strix Halo:
> test-backend-ops 1141/1141, long-prompt needles 20/20 + 5/5 + 5/5, KLD 0 and identical output on two
> models, pp512 +4.4% / +7.8%, tg128 +2.5% / +2.8%)
> ([docs/hrx.md](docs/hrx.md); its decode-split race, #123/#140, is fixed and the kernel is on by default; Q2_K, IQ2 and IQ3_XXS GGUFs run on HRX instead of the CPU, and `--mtp` on Qwen3.8-27B runs NaN-free since #257), the NPU engine on full ELFs and on our own
> NPU runtime, XRT-free ([docs/npu.md](docs/npu.md); its layer kernel and lm-head now decode from
> source, ELF to ELF, with no xclbin at any point, verified on Qwen3-0.6B/1.7B/4B against an fp64
> reference — the kernel sources stay private and are not yet merged here, docs/npu.md Step 3d).
> The MLX route on Apple Silicon is removed (Lemonade serves MLX). ZAYA1-8B (Zyphra)
> runs from our llama.cpp on HRX, matching transformers, at about 90 tok/s decode in Q4_K_M
> ([docs/hrx.md](docs/hrx.md)). The rest of Zyphra's family (Zamba, Zamba2, BlackMamba) and
> Qwen3.8-Flash-Next ran only on the removed Vulkan build; they are to be ported to HRX. Experimental,
> and closed source: Qwen3.6-35B-A3B on the NPU through a private add-on, parity against fp64 passes,
> 16.3-16.5 tok/s decode ([docs/npu.md](docs/npu.md#private-routes)). GGUFs of six architectures
> (Qwen2.5, Qwen3 MoE, Qwen3.6-35B-A3B, MiniCPM4/5, GLM-4.7-Flash) also answer on the NPU through
> `1bit serve --device npu`: correct, not yet fast (0.006-0.17 tok/s;
> [docs/npu.md](docs/npu.md#from-a-gguf)). DwarfStar reads 1BP packages from our fork, pinned since
> 2026-10-17 to `5b013dc1bb75` (1BP v5: reader, converter, checker); checked on hardware at that pin,
> DeepSeek V4 Flash Q2 gives identical answers and token counts from its GGUF and from its 1BP package,
> and `1bit serve -m <model>.1bp` routes a package to ds4 by itself
> ([docs/dwarfstar.md](docs/dwarfstar.md)). Step 4, the Laya router, has landed as an opt-in:
> `1bit serve --laya` classifies each conversation (code, prose, short, long document; 95.5% on
> 200 labelled requests) and a measured policy picks the device; the scorer runs on HRX at
> 15-16 ms a decision ([docs/laya.md](docs/laya.md)). ZAYA1-8B is the engine's Project Manager:
> `1bit serve --pm` (Lemonade entry `ZAYA1-8B-PM-1bit`) answers general chat itself and delegates
> code, long reasoning and agentic tasks to expert models Lemonade loads beside it; `--pm-router laya`
> has the engine's Laya router pick the expert before the PM's first round, and `--pm-spawn-url` adds a
> `spawn_worker` tool that asks a local executor for a long-running agent worker instead of a single
> expert answer ([docs/pm.md](docs/pm.md)). Step 5, the model registry, has landed: of 338,650
> HF text-generation models with an architecture (the 2026-10-17 census), 94.62% are mapped to a
> backend and 63.9% have an architecture checked end to end on Strix Halo; a daily census keeps the
> counts current ([docs/registry.md](docs/registry.md)). The working engine is being ported from 1bit-MONSTER,
> our private development repository; see [docs/PORTING.md](docs/PORTING.md).
> Measured results are on the [wiki](https://github.com/1bit-MONSTER/engine/wiki). This repository
> holds the verified code without the development history.
**The 1bit engine is AMD's HRX runtime, our own HIP kernels on the Strix Halo GPU, and our NPU
engine, all inside [Lemonade](https://github.com/lemonade-sdk/lemonade).** On a Ryzen AI Max+ 395,
every graph node of every model and mode we test runs on the GPU: 43 of 43 configurations in the
census, with 0 nodes on the CPU ([#476](https://github.com/1bit-MONSTER/engine/pull/476),
[#480](https://github.com/1bit-MONSTER/engine/pull/480),
[#481](https://github.com/1bit-MONSTER/engine/pull/481)). Qwen3.8-27B (UD-Q4_K_XL) with MTP
speculative decoding decodes at 36.0 / 26.8 / 29.9 tok/s on code / prose / short chats
([#487](https://github.com/1bit-MONSTER/engine/pull/487)).

Lemonade stays the server you talk to (its catalog, downloads, router and UI). For the models it
hands to 1bit, it runs the engine as one of its backends, behind an OpenAI-compatible API:

- **HRX on the Radeon iGPU** ([docs/hrx.md](docs/hrx.md)): AMD's ggml-hrx with its Loom kernels
and our HIP kernels, the engine's one GPU route. For a GGUF, Lemonade starts the engine's
`llama-server` directly, one HTTP hop
([#475](https://github.com/1bit-MONSTER/engine/pull/475),
[docs/serve.md](docs/serve.md#one-hop-from-lemonade---print-backend)). Placement is strict by
default: the whole graph runs on HRX, or the load stops with the node that would not
([#482](https://github.com/1bit-MONSTER/engine/pull/482),
[docs/serve.md](docs/serve.md#off-hrx-placement-strict-by-default)). `--device cpu` runs the
same build on the CPU.
- **The NPU engine** on the XDNA 2 NPU, in progress: correctness first, speed next
([docs/npu.md](docs/npu.md)).
- **ONNX Runtime GenAI** (Lemonade's ONNX format) on the CPU, and on the NPU through the user's
own install of AMD's Ryzen AI Software, on Linux
([#386](https://github.com/1bit-MONSTER/engine/pull/386), [docs/onnx.md](docs/onnx.md)).
- **DwarfStar**, for DeepSeek V4 Flash, GLM 5.x and Qwen3.8-Flash-Next in its own GGUFs, with SSD
expert streaming ([docs/dwarfstar.md](docs/dwarfstar.md)).
- **Laya**, opt-in, which picks where each request runs ([docs/laya.md](docs/laya.md)), and
**the Project Manager**, ZAYA1-8B delegating to expert models ([docs/pm.md](docs/pm.md)).
- **Every Hugging Face model architecture**, mapped to a backend by a registry and kept current
by a daily census ([docs/registry.md](docs/registry.md)).

Vulkan, ROCm, CUDA, ZINC, ComfyUI, Metal/MLX and Windows were cut from the engine's core
([#306](https://github.com/1bit-MONSTER/engine/pull/306)) and are not routes of the engine.
Lemonade serves them with its own backends and builds ([docs/lemonade.md](docs/lemonade.md)).

**Checked on the hardware.** Every HRX pin change and every PR labelled `hw-gate` passes the
hardware gate on Strix Halo before merge: it builds our HIP kernels into the server as the release
does, runs kernel tests, known answers, KLD against the release pin and long-prompt needles, and
checks that every graph of every gate model is on the GPU
([#478](https://github.com/1bit-MONSTER/engine/pull/478),
[#484](https://github.com/1bit-MONSTER/engine/pull/484), [docs/hw-gate.md](docs/hw-gate.md)). The
runtime keeps one memory budget for every buffer, and after a GPU fault the server refuses new work
and exits cleanly, so Lemonade can restart the model
([#486](https://github.com/1bit-MONSTER/engine/pull/486)). Serve numbers in these docs come from
`tools/bench.py`, which A/B-measures configurations in the same run ([docs/bench.md](docs/bench.md)).

**Install:** through Lemonade. The packages are Lemonade with the `onebit` recipe and the engine
(unpack it and run `lemond-onebit`), and 1bit OS, a Lemonade appliance on a bootable USB image.
On Ubuntu 26.04 the stock `lemond` from the Lemonade PPA runs the engine through a systemd drop-in,
with no patch to Lemonade ([#483](https://github.com/1bit-MONSTER/engine/pull/483),
[docs/lemonade.md](docs/lemonade.md#ubuntu-2604-the-ppas-lemond)). The engine has no standalone
package of its own. The first release goes out when every item on the release checklist is done
([docs/releases.md](docs/releases.md)).

> **Direction:** HRX with our HIP kernels, plus the NPU engine, embedded into Lemonade rather than
> Lemonade embedded into the engine.[^geramyl] HRX met the decode gates in
> [RFC #213](https://github.com/1bit-MONSTER/engine/discussions/213), which removed the engine's
> Vulkan and ROCm builds. HRX is pinned to AMD's hrx-system and ggml-hrx with our commits on top,
> kept current by `bump-hrx` ([docs/hrx.md](docs/hrx.md#pinned-sources-kept-current)). ZAYA1-8B
> (Zyphra) runs from our llama.cpp on HRX at about 90 tok/s decode in Q4_K_M, matching
> transformers. Zyphra's Zamba, Zamba2 and BlackMamba ran only on the removed Vulkan build and are
> to be ported to HRX. The working engine is being ported from our private development repository;
> see [docs/PORTING.md](docs/PORTING.md). Measured results are on the
> [wiki](https://github.com/1bit-MONSTER/engine/wiki). This repository holds the verified code
> without the development history.

## License

Expand All @@ -107,7 +97,7 @@ compiled in: the 1bit NPU runtime and the 1bit HIP kernels for HRX. They are not
repository, their source is not published, and they are not under Apache-2.0: they ship under the
[1bit Binary License](LICENSE-BINARY) (free to use for any purpose and to pass on unmodified;
third-party parts in [NOTICE-BINARY](NOTICE-BINARY)). A build from this
repository without them has no NPU lane and runs HRX on the public kernels only
repository without them has no NPU engine and runs HRX on the public kernels only
([NOTICE](NOTICE), [docs/releases.md](docs/releases.md#private-components)).

## Thank you
Expand All @@ -124,8 +114,8 @@ The repositories this engine is built on, in order of importance:

| # | Repository | What it gives this engine | License |
|---|---|---|---|
| 1 | [amd/xdna-driver](https://github.com/amd/xdna-driver) | The XDNA 2 NPU kernel driver (`amdxdna`) the NPU lane talks to | GPL-2.0 (driver) |
| 2 | [Xilinx/XRT](https://github.com/Xilinx/XRT) | The NPU runtime API (full ELFs, hardware contexts, buffers, runlists) our own runtime keeps the names of; the NPU lane ran on XRT until October 2026 and no longer builds or ships it | Apache-2.0 (userspace) |
| 1 | [amd/xdna-driver](https://github.com/amd/xdna-driver) | The XDNA 2 NPU kernel driver (`amdxdna`) the NPU engine runs on | GPL-2.0 (driver) |
| 2 | [Xilinx/XRT](https://github.com/Xilinx/XRT) | AMD's NPU runtime and its API, where the engine's NPU work started | Apache-2.0 (userspace) |
| 3 | [Xilinx/mlir-aie](https://github.com/Xilinx/mlir-aie) | IRON and aiecc: how our own NPU kernels are written and compiled | Apache-2.0 WITH LLVM-exception |
| 4 | [Xilinx/llvm-aie](https://github.com/Xilinx/llvm-aie) | Peano, the C++ compiler for the NPU's AI Engine cores | Apache-2.0 WITH LLVM-exception |
| 5 | [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | GGUF inference on the Radeon iGPU through HRX (AMD's tested pair), and on the CPU | MIT |
Expand Down
Loading
Loading