[bong-water-water-bong]

• 1bit serve --moe-slots N (with --device vulkan): keeps routed experts in the file and holds N of them in GPU memory. It sets ONEBIT_MOE_FILE/ONEBIT_MOE_SLOTS for llama-server and loads the experts memory-mapped (-ot exps=CPU).
• Bumps third_party/llama.cpp-vulkan to 57c22a3 (fork PR Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI, exact on 18 models #19, 1bit/moe-stream).
• moe/ExpertCache: caller-owned slot memory (CacheOptions::region), matching the fork's copy.
• Docs: serve.md section, moe-streaming.md "Streaming in the inference path" (speed table, bit-exact perplexity), README line.

Verified on Strix Halo, Qwen3-Coder-30B-A3B Q4_K_M through 1bit serve: 4,608 slots gave 18.9 tok/s cold, then 28.3 and 27.1; 1,536 slots gave 7.1/6.6/6.0. The answers were the same.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit 4f6b65e

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 19 - Partially compliant Compliant requirements: Build and integrate Hugging Face tokenizers v0.23.2 as a static library with only the onig feature Ensure that the tokenizer integration works with 18 models, with identical tokenization results (ids and decoded text) The Rust release is pinned in rust-toolchain.toml (1.98.1) and the crate graph in Cargo.lock (--locked) The C ABI wrapper (hf_tokenizers/) is implemented with about 100 lines of Rust The tools/make_tokenizer_golden.py script is used to generate golden tests with Python tokenizers==0.23.2 The corpus covers eight scripts, code, numbers, whitespace runs, emoji with ZWJ and chat markup The bump-tokenizers.yml workflow moves the pin to each new release and refreshes the lock Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review MoE slots validation The validation for --moe-slots in Launch launch_for(const Options& o, const std::string& device, int child_port) only checks that the device is "vulkan" and that split and fork_arch are false. It does not validate that the model file path is valid or that the model actually supports MoE layers. This could lead to runtime errors or incorrect behavior if the model is not properly configured for MoE streaming. if (o.moe_slots > 0) { // routed experts stay in the file (mmap-ed, never read whole); N of them are cached in // Vulkan buffers and streamed in as the router asks (docs/moe-streaming.md) if (device != "vulkan" || split || fork_arch) throw std::runtime_error("--moe-slots works with --device vulkan, without --prefill-device"); env.push_back("ONEBIT_MOE_FILE=" + std::filesystem::absolute(o.model).string()); env.push_back("ONEBIT_MOE_SLOTS=" + std::to_string(o.moe_slots)); argv.insert(argv.end(), {"-ot", "exps=CPU"}); } Memory allocation error handling In ExpertCache::ExpertCache, when allocating memory with ::mmap, if MAP_FAILED is returned, the error message includes the size in MiB. However, if opt_.region is provided, the error handling for memory allocation is skipped entirely, which could lead to silent failures if the caller-provided memory region is insufficient or invalid. if (opt_.region) { // caller-owned memory, one region per list for (size_t li = 0; li < layouts_.size(); ++li) { layouts_[li].base = opt_.region((int) li, layouts_[li].region_bytes); if (!layouts_[li].base) throw std::runtime_error("no memory for expert slot list " + std::to_string(li)); } } else { void* p = ::mmap(nullptr, mem_bytes_, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (p == MAP_FAILED) throw std::runtime_error("cannot map " + std::to_string(mem_bytes_ >> 20) + " MiB of expert slots"); mem_ = static_cast<uint8_t*>(p); ::madvise(mem_, mem_bytes_, MADV_HUGEPAGE); if (opt_.pin && ::mlock(mem_, mem_bytes_) != 0) throw std::runtime_error(std::string("mlock of the expert slots failed: ") + std::strerror(errno)); for (auto& lay : layouts_) lay.base = mem_ + lay.region_off; }