Skip to content

[RFC] DeepSeek-V4.1-Flash support #811

Open

Description

@arun-elr

Proposal

Serve DeepSeek-V4.1-Flash and its -1017 refresh (DeepseekV41ForCausalLM, image and text in, text out) on one 8-GPU node, built on the GLM-5.2/5.3 stack (#386 to #406). Follow-up to this comment on #492. Four phases: text correctness first, then the sparse attention and FP4 KV that make it cheap, then vision, then DSpark and 1M context. I have the compute for the 8-GPU runs and benchmarks below.

What the checkpoint is

Read from config.json, the safetensors index, the 48 shard headers (range requests, no weights downloaded), inference/model.py and the tech report, at HF commits 314113f8a7 (base) and 1bae0e1088 (-1017). The two repos have byte-identical config.json, safetensors index, inference/model.py and encoding/encoding.py. The 48 shards have the same names and sizes but different hashes, so one port serves both.

Piece Config Bytes on disk
Backbone 40 layers, hidden 5120, 64 heads, one 512-d latent per token (K = V, last 64 dims RoPE), q LoRA 1280, grouped output LoRA (8 x 1024), attention sink attention 5.2 GB (FP8)
MoE 384 routed (top 6) + 1 shared, inter 2304, sqrtsoftplus + noaux_tc bias, separate routing bias for image tokens, route scale 1.5, SwiGLU clamp 10 routed 288.8 GB (FP4 packed as I8, E8M0 scale per 32), shared 1.4 GB (FP8), router 0.16 GB (BF16)
CED layers 0-19 encoder, 20-39 decoder; layer 20 projects the decoder's global KV from the last encoder hidden state
CSA2 every layer: 128-token SWA. Layers 2-19 add ratio-2 compressed KV (sources 2, 8, 14), layers 20-39 ratio 1 (source 20). Indexer on 2, 8, 14, 20, 24, 28, 32, 36, top-512; layer 20 also picks 2048 candidate blocks of 8 for 24-36 in attention
mHC hc_mult 4, Sinkhorn 20 iterations 0.2 GB (F32)
Engram layers 1 and 14, two tables of 384M rows x 256, n-grams of 2 to 4 tokens hashed over a 99,092-token compressed vocab 203.1 GB (FP8, E8M0 scales)
DSpark 3 blocks (SWA only, 128 experts top 3), draft block of 5, Markov head, confidence head, reads layers 37-39 7.9 GB
Vision DeepSeek-ViT, 32 layers x 1024, patch 14, 2D RoPE, 3x3 unshuffle + 2-layer MLP, up to 1024 tokens per image 1.0 GB (BF16)
Embed, head vocab 129,280, untied 2.6 GB (BF16)
Total 763.2B params in the HF count; the README's 552B backbone leaves out Engram (196.6B) and the DSpark experts 510.29 GB (475.2 GiB), 48 shards

Things that change the port:

  • Dense linears are FP8 with 32x32 blocks and power-of-two (UE8M0) scales; the stack's fast paths are tuned for 128x128 fp32 scales.
  • The reference keeps KV in BF16 buffers but rounds the values in place to the FP8 grid (SWA) and the FP4 grid (latents and index keys). BF16 represents the rounded values exactly, so real FP4/FP8 storage dequantizes to the same bits and can be tested against the rounded path exactly. Attention outputs from a different kernel still differ in summation order.
  • The reference only does a prefill from position 0 and one-token decode, and has no speculative accept/reject loop (generate.py decodes one token at a time; only model.py's self-test calls forward_spec). Chunked prefill, prefix reuse and DSpark verification go beyond it.
  • Bounded Replay (prefill runs only the encoder for all but the last 128 tokens, tech report 3.2.2) is approximate by design. The reference runs the full forward.
  • No Jinja template in either repo's tokenizer_config.json. The base repo has a community chat_template.jinja (HF #68), -1017 doesn't, and the README points to encoding/encoding.py (DSML tool calls, thinking mode, reasoning effort 1-100).

Does it fit on one node

TP8 attention, EP8 experts and Engram split by rows (as in the reference's convert.py). That's 63.3 GB sharded plus 3.7 GB replicated per GPU. The KV latent has one head, so every TP rank keeps the same copy (as with MLA in GLM-5.2): per-GPU capacity is node capacity. The 20 GB reserve for activations, graphs and NCCL is a guess to replace with a measurement in phase 1.

Per GPU Weights H200 141 GB: KV room B200 180 GB: KV room
FP4 experts as shipped 67.0 GB 54 GB, 60M tokens at 890 B 93 GB, 104M tokens at 890 B
Phase 1: experts upcast to FP8, dense linears in BF16 102.8 GB 18 GB, 5.7M tokens at 3,200 B 57 GB, 17.9M tokens at 3,200 B
  • 890 B/token: three ratio-2 FP4 latents (3 x 288 / 2) + the ratio-1 latent (288) + FP4 index keys (3 x 68 / 2 + 68). It matches the model card.
  • Phase 1 keeps the same caches in BF16: 3,200 B/token, so a 1M-token sequence takes 3.2 GB instead of 0.89 GB.
  • SWA is a fixed 2.9 MB per request (43 layers x 128 x 528 B in FP8), plus draft headroom in phase 4.
  • Both GPU types fit with plenty of KV room. KV is not the limit for this model; weights and prefill activations are.
  • Disk: 510 GB of weights plus about the same again for the reference's converted TP8 copy used as the oracle.

Mapping onto M*

S, M and L are rough difficulty.

V4.1 piece Reuse New Where Size
Config, loader nested text_config parsing (main's qwen3_5/config.py, #390's glm5_next/config.py); raise-on-unfilled loader and a block dequantizer that already takes 32x32 E8M0 blocks (#387, glm52/weight_loader.py, glm52/quantization.py); pread loader with E8M0 reads (#389, model/loader/iterators.py) FP4 unpacking and upcast, Engram row split mstar/model/deepseek_v41/ M
Attention core FlashInfer MLA wrapper on main with kpe 0 and the whole 512-d latent as ckv, so K = V comes out right (engine/resources/attn/flashinfer_mla.py) sink folded in through the LSE, inverse RoPE on the output, grouped output LoRA model M
SWA window recurrent pool on main, one fixed slot per request (engine/resources/recurrent/pool.py) 43-layer 128-token ring and the ratio-2 compressor's partial group in that slot model + resource declaration M
Compressed KV, CED, Full/Reindex/Reuse paged KV with an MLA-style 512-wide layout (engine/resources/kv/); #391 keeps a pooled key in its group's last-token row 4 source caches shared by their consumer layers; strided storage (one row per 2 tokens) in phase 2 KV resource config M
Indexer, hierarchical candidates select_topk_causal (#387, glm52/components/indexer.py), paged index keys as a second KV resource with chunked scores (#392, glm52/dsa_paged.py), sparse MLA over one-token pages (#391, engine/resources/attn/sparse_mla.py) the candidate-block level (layer 20 picks, 24-36 score inside it), top-512, ratio-aware visibility model + #391's resource M-L
FP4 KV, FP8 SWA packed E2M1 + E4M3-per-16 latents, E2M1 + E8M0-per-32 index keys; #391's sparse MLA takes BF16 only new KV dtype + kernels L
Routed MoE fused_experts_fp8 on main takes a block shape and swiglu_limit (utils/fused_moe/runner.py); FP4 to FP8 at load as in convert.py --expert-dtype fp8 sqrtsoftplus router, image-token bias. For parity, three numeric options in main's kernel: the routing weight applied before the down projection's FP8 quantization (main applies it after the down GEMM), SiLU times up kept in FP32 (main rounds SiLU to BF16 first), and power-of-two activation scales (utils/quant_fp8.py). FP4 W4A8 grouped GEMM in phase 2 model/components/moe.py, utils/fused_moe/ M, L for FP4
Dense FP8 linears #388's fp8_linear.py / dense_fp8.py take a block size 32x32 fast path. Phase 1 dequantizes weights to BF16 at load and keeps the reference's FP8 rounding of each input activation (power-of-two scale per 32) model S then M
mHC #390's Glm5NextHyperConnection and Sinkhorn kernel (glm5_next/mhc.py, sinkhorn_kernel.py), same math single-pass wiring (each sublayer's collapse weights come from the sublayer before), exit collapse by the last weights instead of a mean move to model/components/ S
Engram n-gram hash from the request's last 3 tokens (image tokens break n-grams), row-split lookup + all-reduce model, history from the request's token ids M
Bounded Replay encoder-only prefill for all but the last 128 tokens; decoder SWA rebuilt from them model + step declaration M
DSpark MTP and speculative sampling (#388, #398, #405, #406), sampler resource block drafter, Markov head, per-request verify length from the confidence head model + engine/resources/sampler/ L
Vision Qwen3.5 ViT and data-worker image path (model/qwen3_5/components/vision.py, model/multimodal.py), ragged attention (engine/resources/attn/ragged/) DeepSeek-ViT + aligner, the reference's resize rules, image-span embeddings model M
Long context #391/#392 long-context paths and configs YaRN 16x from 64k, compressed-layer RoPE theta 160k; a committed passkey and multi-key harness (#391 reports results but doesn't include one) config, benchmark/ M
Chat #397's glm_chat.py pattern and serving_chat.py hooks port of encoding.py (MIT) and its completion parser api_server/openai/ S-M
TP comms symmetric all-reduce (#386, distributed/all_reduce.py), fused all-reduce norm (#388) none expected S

Plan

Each phase is one PR or a short stack. Every run is on the same random 1024-in/512-out client sweep (c1/c8/c32/c64, greedy) and GSM8K-200 as #388, so the numbers line up with GLM-5.2. H200 is the default; B200 only for the native FP4 expert path. 13 eight-GPU runs in total.

Phase 1: text-only correctness on 8 GPUs (4 runs, H200)

  • TP8 attention, EP8 experts, Engram split by rows, vocab-parallel embedding and head.
  • Experts upcast to FP8 on main's fused MoE, with the three parity options above.
  • The expert upcast is lossless only while the 32 row scales sharing an FP8 block stay within about 2^13 of each other. A sample of 60 real expert scale tensors (691k blocks) peaks at 2^8. The loader checks every block and refuses otherwise.
  • Dense FP8 weights dequantized to BF16, with each input activation rounded to FP8 first, as the reference does.
  • KV in BF16, rounded to the reference's FP8/FP4 grids.
  • Indexer and candidate selection in torch, exact.
  • Full forward on every token, eager, context held to 64k.
  • DSpark weights loaded and unused. Chat, reasoning and tool calls through the encoding.py port.
  • CPU tests:
    • a reduced-size model built from the real config;
    • state_dict against the index by name and shape on meta;
    • single-pass mHC, compressor, indexer and Engram hashes against the reference's torch code;
    • chunked prefill equal to one-shot;
    • the 5 shipped encoding/tests cases;
    • the loader refusing a missing tensor.
  • GPU:
    • per-layer hidden states and greedy output against the reference oracle at 64, 512, 4k and 32k tokens;
    • a CJK and a malformed request;
    • 8 concurrent requests.
  • Benchmarks: the sweep and GSM8K-200, measured headroom in place of the 20 GB guess, load-to-ready. This is a baseline, not a target.

Phase 2: sparse attention and the FP4 KV cache (3 runs: 2 H200, 1 B200)

  • FP4 latents and index keys, FP8 SWA, strided ratio-2 caches.
  • A fused kernel over window + top-512 with the sink; indexer kernels on Extend GLM-5.3-Flash to 1M-token contexts #391/Extend GLM-5.2 to 256k-token contexts #392's approach, with layers 24-36 scoring only candidate blocks.
  • 32x32 FP8 dense linears; FP4 W4A8 expert GEMM. On Hopper it casts to FP8 in the kernel, as the reference does; B200 gets the native path.
  • Decode CUDA graphs and chunked prefill. Bounded Replay behind a flag, off by default.
  • Tests:
    • dequantized FP4/FP8 cache contents bit-exact against phase 1's rounded BF16 cache;
    • each kernel against the torch path;
    • captured decode equal to eager;
    • KV bytes per token from the allocator equal to 890.
  • Benchmarks: the sweep and GSM8K-200 against phase 1, on H200 and separately on B200; prefill at 8k and 32k with replay on and off; GSM8K-200 with replay on.

Phase 3: vision (2 runs, H200)

  • ViT and aligner, the image processor in the data worker, the image routing bias, Engram masking on image spans.
  • Tests: image token counts against the reference's processor (CPU); ViT and aligner parity on the two example images; images inside a chunked prefill and next to a cached prefix.
  • Benchmarks: 200 DocVQA questions against the reference's answers; image-to-text throughput at c1/c8/c32.

Phase 4: DSpark and long context (4 runs, H200)

Open questions

  • Build on the stack head (GLM-5.2/GLM-5.3 PR 11: prefix cache and MTP with temperature #406) now, or wait for GLM-5.2/GLM-5.3 PR 2: GLM-5.2 with MLA, fp8 MoE and DSA #387 to Add GLM-5.3-Flash on 8 GPUs, plain and with MTP #390 to merge? Moving mHC, select_topk_causal and sparse MLA into shared components as they land would let V4.1 depend on main.
  • Is FP8-upcast experts on main's fused MoE fine for phase 1? It costs 33 GB per GPU and needs three small numeric options in the existing kernel rather than a new one.
  • Engram tables on GPU (25.3 GB per GPU) or in pinned host memory? Lookups depend only on token ids, so they can be prefetched.
  • Bounded Replay is what the model was post-trained with but is not exact. Off by default, or on for prefill once GSM8K and the agent session show no change?
  • Default checkpoint: -1017, with the base repo selectable? They share the code and config.
  • The DSpark scheduler uses throughput curves profiled per engine. Is a fixed verify length acceptable as the first merge?

Related


@vasilevklart @merceod @NSagan271 FYI, feel free to assign this to me.

Activity

  1. arun-elr commented on Oct 23, 2026

    @arun-elr
    ContributorAuthor

    I'm planning to work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions