Repository navigation
Description
Proposal
Serve DeepSeek-V4.1-Flash and its -1017 refresh (DeepseekV41ForCausalLM, image and text in, text out) on one 8-GPU node, built on the GLM-5.2/5.3 stack (#386 to #406). Follow-up to this comment on #492. Four phases: text correctness first, then the sparse attention and FP4 KV that make it cheap, then vision, then DSpark and 1M context. I have the compute for the 8-GPU runs and benchmarks below.
What the checkpoint is
Read from config.json, the safetensors index, the 48 shard headers (range requests, no weights downloaded), inference/model.py and the tech report, at HF commits 314113f8a7 (base) and 1bae0e1088 (-1017). The two repos have byte-identical config.json, safetensors index, inference/model.py and encoding/encoding.py. The 48 shards have the same names and sizes but different hashes, so one port serves both.
| Piece | Config | Bytes on disk |
|---|---|---|
| Backbone | 40 layers, hidden 5120, 64 heads, one 512-d latent per token (K = V, last 64 dims RoPE), q LoRA 1280, grouped output LoRA (8 x 1024), attention sink | attention 5.2 GB (FP8) |
| MoE | 384 routed (top 6) + 1 shared, inter 2304, sqrtsoftplus + noaux_tc bias, separate routing bias for image tokens, route scale 1.5, SwiGLU clamp 10 |
routed 288.8 GB (FP4 packed as I8, E8M0 scale per 32), shared 1.4 GB (FP8), router 0.16 GB (BF16) |
| CED | layers 0-19 encoder, 20-39 decoder; layer 20 projects the decoder's global KV from the last encoder hidden state | |
| CSA2 | every layer: 128-token SWA. Layers 2-19 add ratio-2 compressed KV (sources 2, 8, 14), layers 20-39 ratio 1 (source 20). Indexer on 2, 8, 14, 20, 24, 28, 32, 36, top-512; layer 20 also picks 2048 candidate blocks of 8 for 24-36 | in attention |
| mHC | hc_mult 4, Sinkhorn 20 iterations |
0.2 GB (F32) |
| Engram | layers 1 and 14, two tables of 384M rows x 256, n-grams of 2 to 4 tokens hashed over a 99,092-token compressed vocab | 203.1 GB (FP8, E8M0 scales) |
| DSpark | 3 blocks (SWA only, 128 experts top 3), draft block of 5, Markov head, confidence head, reads layers 37-39 | 7.9 GB |
| Vision | DeepSeek-ViT, 32 layers x 1024, patch 14, 2D RoPE, 3x3 unshuffle + 2-layer MLP, up to 1024 tokens per image | 1.0 GB (BF16) |
| Embed, head | vocab 129,280, untied | 2.6 GB (BF16) |
| Total | 763.2B params in the HF count; the README's 552B backbone leaves out Engram (196.6B) and the DSpark experts | 510.29 GB (475.2 GiB), 48 shards |
Things that change the port:
- Dense linears are FP8 with 32x32 blocks and power-of-two (UE8M0) scales; the stack's fast paths are tuned for 128x128 fp32 scales.
- The reference keeps KV in BF16 buffers but rounds the values in place to the FP8 grid (SWA) and the FP4 grid (latents and index keys). BF16 represents the rounded values exactly, so real FP4/FP8 storage dequantizes to the same bits and can be tested against the rounded path exactly. Attention outputs from a different kernel still differ in summation order.
- The reference only does a prefill from position 0 and one-token decode, and has no speculative accept/reject loop (
generate.pydecodes one token at a time; onlymodel.py's self-test callsforward_spec). Chunked prefill, prefix reuse and DSpark verification go beyond it. - Bounded Replay (prefill runs only the encoder for all but the last 128 tokens, tech report 3.2.2) is approximate by design. The reference runs the full forward.
- No Jinja template in either repo's
tokenizer_config.json. The base repo has a communitychat_template.jinja(HF #68), -1017 doesn't, and the README points toencoding/encoding.py(DSML tool calls, thinking mode, reasoning effort 1-100).
Does it fit on one node
TP8 attention, EP8 experts and Engram split by rows (as in the reference's convert.py). That's 63.3 GB sharded plus 3.7 GB replicated per GPU. The KV latent has one head, so every TP rank keeps the same copy (as with MLA in GLM-5.2): per-GPU capacity is node capacity. The 20 GB reserve for activations, graphs and NCCL is a guess to replace with a measurement in phase 1.
| Per GPU | Weights | H200 141 GB: KV room | B200 180 GB: KV room |
|---|---|---|---|
| FP4 experts as shipped | 67.0 GB | 54 GB, 60M tokens at 890 B | 93 GB, 104M tokens at 890 B |
| Phase 1: experts upcast to FP8, dense linears in BF16 | 102.8 GB | 18 GB, 5.7M tokens at 3,200 B | 57 GB, 17.9M tokens at 3,200 B |
- 890 B/token: three ratio-2 FP4 latents (3 x 288 / 2) + the ratio-1 latent (288) + FP4 index keys (3 x 68 / 2 + 68). It matches the model card.
- Phase 1 keeps the same caches in BF16: 3,200 B/token, so a 1M-token sequence takes 3.2 GB instead of 0.89 GB.
- SWA is a fixed 2.9 MB per request (43 layers x 128 x 528 B in FP8), plus draft headroom in phase 4.
- Both GPU types fit with plenty of KV room. KV is not the limit for this model; weights and prefill activations are.
- Disk: 510 GB of weights plus about the same again for the reference's converted TP8 copy used as the oracle.
Mapping onto M*
S, M and L are rough difficulty.
| V4.1 piece | Reuse | New | Where | Size |
|---|---|---|---|---|
| Config, loader | nested text_config parsing (main's qwen3_5/config.py, #390's glm5_next/config.py); raise-on-unfilled loader and a block dequantizer that already takes 32x32 E8M0 blocks (#387, glm52/weight_loader.py, glm52/quantization.py); pread loader with E8M0 reads (#389, model/loader/iterators.py) |
FP4 unpacking and upcast, Engram row split | mstar/model/deepseek_v41/ |
M |
| Attention core | FlashInfer MLA wrapper on main with kpe 0 and the whole 512-d latent as ckv, so K = V comes out right (engine/resources/attn/flashinfer_mla.py) |
sink folded in through the LSE, inverse RoPE on the output, grouped output LoRA | model | M |
| SWA window | recurrent pool on main, one fixed slot per request (engine/resources/recurrent/pool.py) |
43-layer 128-token ring and the ratio-2 compressor's partial group in that slot | model + resource declaration | M |
| Compressed KV, CED, Full/Reindex/Reuse | paged KV with an MLA-style 512-wide layout (engine/resources/kv/); #391 keeps a pooled key in its group's last-token row |
4 source caches shared by their consumer layers; strided storage (one row per 2 tokens) in phase 2 | KV resource config | M |
| Indexer, hierarchical candidates | select_topk_causal (#387, glm52/components/indexer.py), paged index keys as a second KV resource with chunked scores (#392, glm52/dsa_paged.py), sparse MLA over one-token pages (#391, engine/resources/attn/sparse_mla.py) |
the candidate-block level (layer 20 picks, 24-36 score inside it), top-512, ratio-aware visibility | model + #391's resource | M-L |
| FP4 KV, FP8 SWA | packed E2M1 + E4M3-per-16 latents, E2M1 + E8M0-per-32 index keys; #391's sparse MLA takes BF16 only | new KV dtype + kernels | L | |
| Routed MoE | fused_experts_fp8 on main takes a block shape and swiglu_limit (utils/fused_moe/runner.py); FP4 to FP8 at load as in convert.py --expert-dtype fp8 |
sqrtsoftplus router, image-token bias. For parity, three numeric options in main's kernel: the routing weight applied before the down projection's FP8 quantization (main applies it after the down GEMM), SiLU times up kept in FP32 (main rounds SiLU to BF16 first), and power-of-two activation scales (utils/quant_fp8.py). FP4 W4A8 grouped GEMM in phase 2 |
model/components/moe.py, utils/fused_moe/ |
M, L for FP4 |
| Dense FP8 linears | #388's fp8_linear.py / dense_fp8.py take a block size |
32x32 fast path. Phase 1 dequantizes weights to BF16 at load and keeps the reference's FP8 rounding of each input activation (power-of-two scale per 32) | model | S then M |
| mHC | #390's Glm5NextHyperConnection and Sinkhorn kernel (glm5_next/mhc.py, sinkhorn_kernel.py), same math |
single-pass wiring (each sublayer's collapse weights come from the sublayer before), exit collapse by the last weights instead of a mean | move to model/components/ |
S |
| Engram | n-gram hash from the request's last 3 tokens (image tokens break n-grams), row-split lookup + all-reduce | model, history from the request's token ids | M | |
| Bounded Replay | encoder-only prefill for all but the last 128 tokens; decoder SWA rebuilt from them | model + step declaration | M | |
| DSpark | MTP and speculative sampling (#388, #398, #405, #406), sampler resource | block drafter, Markov head, per-request verify length from the confidence head | model + engine/resources/sampler/ |
L |
| Vision | Qwen3.5 ViT and data-worker image path (model/qwen3_5/components/vision.py, model/multimodal.py), ragged attention (engine/resources/attn/ragged/) |
DeepSeek-ViT + aligner, the reference's resize rules, image-span embeddings | model | M |
| Long context | #391/#392 long-context paths and configs | YaRN 16x from 64k, compressed-layer RoPE theta 160k; a committed passkey and multi-key harness (#391 reports results but doesn't include one) | config, benchmark/ |
M |
| Chat | #397's glm_chat.py pattern and serving_chat.py hooks |
port of encoding.py (MIT) and its completion parser |
api_server/openai/ |
S-M |
| TP comms | symmetric all-reduce (#386, distributed/all_reduce.py), fused all-reduce norm (#388) |
none expected | S |
Plan
Each phase is one PR or a short stack. Every run is on the same random 1024-in/512-out client sweep (c1/c8/c32/c64, greedy) and GSM8K-200 as #388, so the numbers line up with GLM-5.2. H200 is the default; B200 only for the native FP4 expert path. 13 eight-GPU runs in total.
Phase 1: text-only correctness on 8 GPUs (4 runs, H200)
- TP8 attention, EP8 experts, Engram split by rows, vocab-parallel embedding and head.
- Experts upcast to FP8 on main's fused MoE, with the three parity options above.
- The expert upcast is lossless only while the 32 row scales sharing an FP8 block stay within about 2^13 of each other. A sample of 60 real expert scale tensors (691k blocks) peaks at 2^8. The loader checks every block and refuses otherwise.
- Dense FP8 weights dequantized to BF16, with each input activation rounded to FP8 first, as the reference does.
- KV in BF16, rounded to the reference's FP8/FP4 grids.
- Indexer and candidate selection in torch, exact.
- Full forward on every token, eager, context held to 64k.
- DSpark weights loaded and unused. Chat, reasoning and tool calls through the
encoding.pyport. - CPU tests:
- a reduced-size model built from the real config;
state_dictagainst the index by name and shape onmeta;- single-pass mHC, compressor, indexer and Engram hashes against the reference's torch code;
- chunked prefill equal to one-shot;
- the 5 shipped
encoding/testscases; - the loader refusing a missing tensor.
- GPU:
- per-layer hidden states and greedy output against the reference oracle at 64, 512, 4k and 32k tokens;
- a CJK and a malformed request;
- 8 concurrent requests.
- Benchmarks: the sweep and GSM8K-200, measured headroom in place of the 20 GB guess, load-to-ready. This is a baseline, not a target.
Phase 2: sparse attention and the FP4 KV cache (3 runs: 2 H200, 1 B200)
- FP4 latents and index keys, FP8 SWA, strided ratio-2 caches.
- A fused kernel over window + top-512 with the sink; indexer kernels on Extend GLM-5.3-Flash to 1M-token contexts #391/Extend GLM-5.2 to 256k-token contexts #392's approach, with layers 24-36 scoring only candidate blocks.
- 32x32 FP8 dense linears; FP4 W4A8 expert GEMM. On Hopper it casts to FP8 in the kernel, as the reference does; B200 gets the native path.
- Decode CUDA graphs and chunked prefill. Bounded Replay behind a flag, off by default.
- Tests:
- dequantized FP4/FP8 cache contents bit-exact against phase 1's rounded BF16 cache;
- each kernel against the torch path;
- captured decode equal to eager;
- KV bytes per token from the allocator equal to 890.
- Benchmarks: the sweep and GSM8K-200 against phase 1, on H200 and separately on B200; prefill at 8k and 32k with replay on and off; GSM8K-200 with replay on.
Phase 3: vision (2 runs, H200)
- ViT and aligner, the image processor in the data worker, the image routing bias, Engram masking on image spans.
- Tests: image token counts against the reference's processor (CPU); ViT and aligner parity on the two example images; images inside a chunked prefill and next to a cached prefix.
- Benchmarks: 200 DocVQA questions against the reference's answers; image-to-text throughput at c1/c8/c32.
Phase 4: DSpark and long context (4 runs, H200)
- DSpark with a fixed verify length first, then the confidence-scheduled length.
- SWA ring headroom for the draft length, so a rejected draft never overwrites a live entry.
- 1M context; prefix cache on GLM-5.2/GLM-5.3 PR 10: engine: prefix cache at TP > 1, recurrent snapshots, speculative sampling #405/GLM-5.2/GLM-5.3 PR 11: prefix cache and MTP with temperature #406 with exact SWA snapshots, and Bounded Replay as the opt-in cheaper path.
- Tests:
- drafter forward against
forward_spec; - greedy output with DSpark equal to greedy without;
- a temperature request next to a greedy one;
- a rejected draft at the ring boundary;
- a prefix hit equal to a fresh prefill on the exact path.
- drafter forward against
- Benchmarks:
- the sweep with and without DSpark, plus tokens per verify step;
- passkey and multi-key to about 960k tokens, the checks Extend GLM-5.3-Flash to 1M-token contexts #391 reports, with the harness committed this time;
- 4 x 32k concurrent TTFT/TPOT;
- GSM8K-200 and a multi-turn agent session with replay on.
- Last run: the full table again on the final head.
Open questions
- Build on the stack head (GLM-5.2/GLM-5.3 PR 11: prefix cache and MTP with temperature #406) now, or wait for GLM-5.2/GLM-5.3 PR 2: GLM-5.2 with MLA, fp8 MoE and DSA #387 to Add GLM-5.3-Flash on 8 GPUs, plain and with MTP #390 to merge? Moving mHC,
select_topk_causaland sparse MLA into shared components as they land would let V4.1 depend on main. - Is FP8-upcast experts on main's fused MoE fine for phase 1? It costs 33 GB per GPU and needs three small numeric options in the existing kernel rather than a new one.
- Engram tables on GPU (25.3 GB per GPU) or in pinned host memory? Lookups depend only on token ids, so they can be prefetched.
- Bounded Replay is what the model was post-trained with but is not exact. Off by default, or on for prefill once GSM8K and the agent session show no change?
- Default checkpoint: -1017, with the base repo selectable? They share the code and config.
- The DSpark scheduler uses throughput curves profiled per engine. Is a fixed verify length acceptable as the first merge?
Related
- GLM-5.2/GLM-5.3 PR 1: engine: MLA attention, fp8 fused MoE, symmetric all-reduce #386 (merged): MLA attention, FP8 fused MoE, symmetric all-reduce.
- GLM-5.2/GLM-5.3 PR 2: GLM-5.2 with MLA, fp8 MoE and DSA #387 to GLM-5.2/GLM-5.3 PR 11: prefix cache and MTP with temperature #406: the GLM-5.2/5.3 stack this builds on (DSA, MTP, KDA and pread loader, mHC in Add GLM-5.3-Flash on 8 GPUs, plain and with MTP #390, 1M and 256k context, chat, prefix cache at TP, MTP with temperature).
@vasilevklart @merceod @NSagan271 FYI, feel free to assign this to me.
Activity
arun-elr commentedon Oct 23, 2026
I'm planning to work on this.