Skip to content

[RFC] Speculative decoding in the resource layer #1186

Open

Description

@nick-rui

Speculative decoding in the resource layer

Every path:line is at main 9a50fdf or at the stack top, cbad78a, whose tree equals 75fd5be975's. engine.py is mstar/engine/engine.py, and other short paths are under mstar/engine/resources/.

The implementation is one linear stack of seven PRs, titled "(PR n of 7 in the stack)":

#1323, Qwen3.5 with its multi-token prediction (MTP) head, is held for the State Contract, because test_state_lint refuses code in mstar/model/qwen3_5 that drives resources directly, and mstar.engine.state has no drafting API. It leaves the numbering as a GitHub draft, and #1324 sits directly on #1322. Its generic pieces moved down, PrefillInputs and the scaffold's prefill call to #1321 and the chain helpers and AttentionManager.prompt_last_rows to #1325, and the GDN rerun stays in #1322. Its MTP head, node and configs land later on mstar.engine.state with a chunk-boundary rule that hands MTP the next prompt token, DraftInputs.positions and ObserveInputs.positions. #1319 to #1325, except #1323, are open and not drafts. Each PR's description gives its files, tests and results.

Summary

This RFC adds speculative decoding to M* as one engine-level layer. A drafter inside the target node's forward proposes k draft tokens, the target scores them in one verify step, and the sampler publishes a verify result with each request's accepted count and tokens, from which each resource corrects its own state. The GPU holds each request's exact KV length, and the host counts all k + 1 tokens at commit and subtracts the rejected ones once at a later admit, in the take-back. A node runs in asynchronous mode, planning step M+1 while step M runs, when the GPU can correct every resource on it, and in synchronous mode otherwise. A drafter declares its resources as a model does, and its KV joins the target's pool as a KV layer set.

The stack's single verify path is in the shared decoder scaffold that main already has, MultimodalDecoderModel and SampledTokenDecoder, and runs only on a drafting node, a node whose config has a draft block. A sampler's loop-back masters are its device tensors with one row per request, which main reallocates as requests register. Without a drafter, the one intended behaviour change is #1322's fixed_loopback_masters, which on a one-rank node instead allocates them once, at the rows #938's in-graph scatter reserves, 4,096 by default. The other shared changes default to main's behaviour: #1322's UPDATE_STATE constexpr on the shared causal-conv kernel, #1321's node_token_stop and node_device_loopback classmethods, which build a node's TokenStop and DeviceLoopback so a declaration can read them before the submodule exists, and #1321's accelerator_graph node attribute, default true.

Terms

A verify step runs the target over k + 1 rows per request, its verify block. The first row is the last emitted token, which has no KV entry yet, and the other k rows are the drafts. Drafts before the first rejection are accepted, and the target adds a bonus token after them. The kept rows are the first row and the accepted drafts. The emitted tokens are the accepted drafts and the bonus token, less any that a stop or the token budget cuts. The acceptance length τ is the mean of accepted + 1.

  • Resident planning, from Resident decode planning: decode rows on device tables #1127, keeps each request's block table, KV length, recurrent slot and position offset in a GPU resident row that captured decode steps read and update without a host plan.
  • The GPU length is the exact KV length in the resident row.
  • The verified length counts tokens whose verify results are applied. It is exact and at most two of the row's own committed verify steps behind the GPU length. The take-back ledger, VerifyLedger, records each verify step's results and numbers steps by the node's commits.
  • A returning row is one an offload moved to the host that has not been admitted back.
  • The reserved length is stored_len, the verified length plus k + 1 per committed step whose result is unapplied.
  • The KV slack is the 2(k + 1) slots each request needs beyond its limit, because at most two committed steps can have unapplied results.
  • A lagged drafter, such as a draft model, runs over the target's verify block shifted back one row, because its KV trails the target's by one token.
  • The four drafter arms of the real-weights tests are DSpark, EAGLE-3, and the draft model at k = 3 and at k = 4.
  • An attention binding is one attention resource, with its own wrapper and plan, that a group of layers reads by attn_key.

Motivation

main has no engine-level speculative decoding. Its model-local MTP from the GLM stack, #388, #389, #390 and #1117, serves GLM-5.3-Flash, from #390, and GLM-5.2, from #388, both at TP8 and greedy only. GLM-5.3-Flash relies on #389, where Kimi Delta Attention (KDA) corrects its own recurrent state. This layer uses none of them, and moving the GLM stack onto it would revisit the multi-head latent attention and KDA non-goals. The open efforts are #257 for Kimi K3 with DSpark, #1090 and its stack for Qwen3.5 with MTP on ROCm, and #1133, the only generic layer, for Qwen3-8B with DSpark. Each rebuilds the draft loop, verification, KV length correction, recurrent undo and stop handling differently. This RFC starts from #1133.

Under invariant 1 of AGENTS.md this belongs in the engine. A verify step's effect on KV, positions and recurrent state is unknown when the step is planned, so the feature crosses pre-plan, capture, eviction, stops and TP agreement, where mistakes corrupt data silently. Model-level versions opt out of engine mechanisms, as #1090 does by turning prefix keys off and running on one GPU only. The gain is largest at low concurrency, where weight reads bound decode. For a pruned Kimi K3 at TP8 on 8 x H100, #257 reports 2.3x vLLM's plain throughput at concurrency 1, 1.14x at 32 and 1.19x at 64. Against its own plain decoding that is 2.15x at 1 and 0.94x at 64. #1090 reports 1.47x its plain decoding at 1 and 1.10x at 64 for Qwen3.5-9B on MI325X.

Design

Scope and configuration

A node may use speculative decoding only if all of the following hold:

  • Each step samples one token id per position through SamplerResource.
  • The node stops through TokenStop without withhold.
  • Its verify walk declares DeviceLoopback with that sampler, because the next step reads the bonus token from the request's loop-back slot, its row in the loop-back masters.
  • Its KV pool serves no other node, and no other instance reads it through KV publication or transfer.
  • It is not on the worker's side lane, which side_stream: true selects, and has no outgoing streaming edge, whose consumers expect one item per step.
  • On the scaffold, its checkpoint's generation_config leaves repetition_penalty unset or at 1, because verify refuses any other default.

validate_deployment checks these and every resource's capabilities from config alone and names each refusal's reason.

DSpark, #1321's drafter, is a small checkpoint that drafts all k tokens in one pass from taps, copies of the residual stream entering chosen target layers. #1325 adds EAGLE-3 and draft models of the target's family, such as Qwen3-0.6B. EAGLE-3 is a one-layer drafter that reads three layers' taps and chains k one-token passes through chain_drafts, which #1325 takes from the held MTP head with kept_rows, at_last_rows and AttentionManager.prompt_last_rows. EAGLE-3 drafts its first token from its carry, a per-request hidden vector that crosses steps, here its output before the final norm.

model: "qwen3_8b"
max_seq_len: 40960
model_kwargs:
  draft:
    method: dspark                              # eagle3, draft_model
    model: deepseek-ai/dspark_qwen3_8b_block7
    num_draft_tokens: 7
    mode: auto                                  # auto | synchronous | asynchronous
  accelerator_graph: true
resources:
  kv_cache:
    max_num_pages: 1024
    cpu_offload_pages: 512
node_groups:
  - node_names: [LLM]
    ranks: [0]

The engine owns the draft block's schema, parses it in validate_deployment and refuses unknown keys. Without the block, speculative decoding is off. The methods are dspark, eagle3 and draft_model, in DRAFT_METHODS at draft/config.py:42-49, and mtp returns with Qwen3.5 MTP.

Every shipped drafting config sets resources.kv_cache.cpu_offload_pages. Preemption never drops a request on a drafting node, so without a host pool admission must reserve each request's whole footprint up front. _warn_hostless, at engine.py:254, warns once per pool without a host pool. DSpark and EAGLE-3 set 512 pages, about 10.25 and 9.3 GiB pinned, and the draft model 256, about 8 GiB, as defaults to tune.

The stack adds no environment variable and two settings. The attention resource's verify_backend pins the verify kernel, and model_kwargs.accelerator_graph, which #1321 adds to Qwen3, makes a drafting node capture nothing and verify synchronously when false.

max_num_pages still counts pages, and startup logs each set's bytes per token. drafter_resources raises each paged pool's max_seq_len by the slack through KVSpec.bind_draft_slack, so KVSpec.request_limit is the pool's max_seq_len minus draft_slack, and a resources.<kv>.max_seq_len override on a drafting node's pool stays the client's limit. Nothing on main reads the required top-level max_seq_len, as docs/serving.rst:241-244 says, but a drafting node refuses a value above the request limit, and slack_problems, at draft/startup.py:320, names the request limit and the two fixes, raising the override or lowering the serving max_seq_len.

k is fixed per node, because it sizes the verify capture, KV slack, random-number stride and take-back amounts, on which ranks must agree. DSpark configs must set num_draft_tokens. EAGLE-3's k defaults to 3 and may be at most 7, with a warning above its checkpoint's trained depth of 3. A draft model's k defaults to 3 and may be at most 4. On one H100 at e0f7d1e78, over 1 to 64 rows, benchmark/draft_model_cost/step_cost.py puts its one-row passes at 0.45 to 0.72x of Qwen3-8B's verify step at k = 3, 0.60 to 0.85x at k = 4 and 1.04 to 1.41x at k = 7, so 4 is the largest measured k whose drafting costs less than a verify step at every row count, to re-check at 96 and 128 rows. Passes over the whole block under one plan cost 0.78 to 1.76x, so they are not built, though one-row passes make k - 1 more plan calls a step at 0.057 to 0.283 ms each.

Worker metrics gain mstar_spec_* counters for drafts proposed and accepted and verify rows, and a histogram of accepted drafts per verify row. Each node logs its method, k, mode and why at startup, logs τ every 60 seconds, and warns once if under 2% of its first 4,096 drafts were accepted. Responses gain no field.

Draft, then verify

The drafter runs inside the target's forward, so one captured graph per bucket covers the step. A drafting node builds a verifying sampler, a SamplerResource with SamplerSpec.verifies=True. A request enters a verify step with GPU length L and the previous bonus token in its loop-back slot. The step runs in this order:

  1. The drafter proposes d1 .. dk, and draft_verify_step broadcasts rank 0's drafts through SamplerResource.agree.
  2. The target runs over [input, d1 .. dk] at positions L .. L + k and writes KV for all k + 1 rows.
  3. The sampler verifies and broadcasts rank 0's result.
  4. Each resource that corrects its own state, such as GDN, overrides Resource.correct_verify_state(result), which draft_verify_step calls right after verify, in the runner's order, from DrafterShare.correcting_keys.
  5. A drafter with state observes the verified block.
  6. The engine adds accepted + 1 to the GPU length and stores the new carry.

Only engine-owned state crosses steps, meaning the bonus token, the drafter's KV, a declared carry, and for a lagged draft model DRAFT_KEEP_FIRST, a per-row flag the engine sets when the row accepted all k drafts.

Drafter and DrafterDecl are protocols with abstract methods. A Drafter declares needs, a DrafterNeeds, and its propose(inputs) returns [n, k] drafts. The target callable that draft_verify_step receives returns TargetOutput(logits, hidden=None), and a bare tensor raises TypeError while the step is traced.

SampledTokenDecoder._verify, which _declare_verify and verify_layer_steps declare, makes the one call, self.draft_verify_step(engine_inputs, input_ids.view(n), target), with the DrafterShare the engine bound at load through NodeSubmodule.bind_draft(share). Attention layers declare kv_layer_set, and load refuses a model in which the number of layers naming a set differs from the set's size. A model package holds only the drafter factory, drafter_decl, llm_node_class, verify_node_fields, tap exposure through decoder_kwargs, refusals and the drafter's weights, and no cross-step state, undo, verification, token cutting or prompt hook.

Drafter resources

The model builds a drafter's declaration from config and passes its resource specs through drafter_resources, which adds the drafter's resources, joins its layer set to the target's pool, adds its carry to the loop-back masters, sets the KV slack and makes the sampler verifying. On every prompt and verify step, captured ones included, the engine adds the drafter's resource steps and loop-back writes, plus a draft model's lag on verify steps, and refuses a model step that declares these itself.

Each drafter resource joins the target's or is its own:

  • KV joins the target's pool, keeping one capacity and one prefix decision.
  • Positions join the target's position resource and rotary base. Startup refuses a different rope_theta, and a draft model reads positions shifted back one row on verify steps.
  • The carry joins the loop-back masters as DRAFT_CARRY, which the sampler purges with the request's row, so it survives slot eviction and KV offload.
  • Taps are a general resource, TapSpec and TapResource in taps/, one per node. run_layers rewrites them every step, so no take-back applies. A drafter names its tap resource in DrafterNeeds.tap_key.
  • EAGLE-3 shares the target's query heads and mask, so it attends through the target's binding. DSpark has its own mask and a draft model its own query heads, so each declares its own binding. Every binding reads the pool's KV plan and makes its own wrapper plan call.

A lagged drafter declares DrafterNeeds.lag, how many rows its block trails and how its passes split it. The engine copies it into the verify walk's attention, KV and position steps and checks it at capture and on every verify step. A lagged binding gets one FA3PagedWrapper per pass on FlashAttention-3 (FA3) and keeps host plans on FlashInfer, where resolve_mode refuses asynchronous mode.

EAGLE-3. RedHatAI/Qwen3-8B-speculator.eagle3 is one decoder layer with a QKV input twice the hidden size, its own final norm and a 32,000-token head, reading taps at layers 2, 18 and 33.

Draft model. Qwen3-0.6B, #1133's pair, lags the target by one token. Its verify block is [prev, bonus, d1 .. d(k-1)], where bonus is the target's first row and prev the token before it, at positions L - 1 .. L + k - 1, causal over a KV extent of L + k, so one stored_len serves both sets, prefill writes every prompt row, and prompt pages are complete. It drafts with a pass over [prev, bonus] and then one-row passes, and its carry holds prev. DRAFT_KEEP_FIRST keeps row 0's write, which otherwise goes to the sink page every KVManager keeps for discarded writes.

KV layer sets. A KV layer set is a contiguous run of a pool's layers with one storage geometry, meaning KV heads, head size and per-rank split. The target's layers are the set "target". All sets in a pool share one page table, page size, allocator, admission, eviction and offload decision, prefix key, stored_len, take-back, KV slack and KV dtype, and page p holds the same token slots in every set. Allocating in pages, not bytes, gives every rank the same page ids. A set with the pool's geometry adds layers to the pool's tensor, as DSpark (36 + 5 layers, 164 KiB a token), EAGLE-3 (36 + 1, 148 KiB) and Qwen3-0.6B (36 + 28, 256 KiB) do. In #1324 a set of another geometry gets its own layer-major KVStorage over the same page ids, which only a toy draft model with 4 KV heads of 64 exercises, and startup refuses it unless a binding plans with it. A one-set pool is unchanged. A binding plans with its set's KV heads and head size and its own query heads, through AttentionConfig(layer_set=..., num_qo_heads=...), on FlashInfer and FA3's verify kernel, and any other backend is refused at build. Query heads are not storage geometry, so Qwen3-0.6B's 16 query heads against Qwen3-8B's 32 change only its binding. Under TP each set splits its KV heads across ranks, replicated when it has fewer heads than ranks.

On the scaffold, the engine and the base classes make every other drafting change. A drafting node has no combined walks or MIXED capture, and SampledTokenDecoder skips #1295's media-mixed capture with it. Each subclass drops its own MIXED ChunkingPolicy, because Model.chunking_policy_of validates a subclass's policies before a base-class filter could run, and Qwen3 declares none. MultimodalDecoderModel adds the KV slack to the layout as scratch and refuses the repetition_penalty default above, and the conductor raises a drafting node's pool scratch to at least the slack in ResourceReqConfig.apply_spec. After a prompt step samples, the scaffold calls drafter.prefill(PrefillInputs(...)), where DSpark writes its prompt KV from the taps, and from #1322 stores a carrying drafter's first carry through store_draft_carry, refusing a None return.

Synchronous and asynchronous mode

Asynchronous mode is the performance target, because the Oct 27 design doc reports the plan thread on GLM-5.3's long-context path at 0.55 ms per step against 7.2 ms when it waited for the verify result. The doc gives no hardware, and this stack has not re-measured them. Synchronous mode is the fallback and test oracle. Both modes run the same take-back, the same token cut, which trims a row's emitted tokens at a stop or the token budget, and the same captured graph where both capture. They differ in async scheduling, which the worker code calls speculation, and in how far the take-back lags.

mode: auto picks asynchronous mode when all of the following hold:

  • Async scheduling is on for the node, which under TP means MSTAR_TP_ASYNC_SCHED=1, the default.
  • Resident planning is on, and the verify walk is in resident_walks(). Qwen3 declares resident_walks = {decode} only when accelerator_graph is on.
  • Every resource's capabilities allow GPU correction, and none refuses through resident_refusal or is shared with another node.
  • Verify attention reads lengths from the GPU with the drafter's mask.

Otherwise auto picks synchronous mode and logs why, and mode: asynchronous fails startup. Engine._agreed_mode, at engine.py:1098, folds every rank's refusals into one MIN reduction, a single joint.agree call with a two-element vector, and the mode decision adds no other startup collective. Each rank raises a rank-local refusal only after that call, so every rank reaches the same mode or refusal. The one earlier raise, for specs and built resources that disagree, follows from config and fires on every rank alike. A verify_backend: fa3 pin where FA3 cannot run fails startup on every rank. The mode is fixed at load, Speculator.can_speculate keeps async scheduling off a synchronous node, and under TP such a node runs on the serial path.

FA3's paged kernel takes KV lengths as a device tensor and serves asynchronous verify on H100 and H200, while FlashInfer plans on the host from upper bounds and runs synchronous mode only. main's fa3-fwd wheel lacks paged KV, so asynchronous verify needs the separately installed flash_attn_3 wheel, about 400 MB, which docs/installation.rst documents. verify_backend: auto takes FA3 where it runs, fa3 refuses to start without it, and flashinfer keeps the node synchronous. FA3PagedWrapper, in attn/fa3_paged.py, has plans_in_graph = True, so a resident step fills its page table and lengths in the graph through main's gather_resident. A verify step that plans a second KV label is refused.

Timing. The stack adds no thread and no wait. Commit counts k + 1 tokens per request on the GPU thread right after launch. In asynchronous mode, postprocess of step M-1 copies its tokens to the host after the plan thread has pre-admitted step M, so admit of step M can apply results only up to step M-2, and the plan reads KV lengths from the GPU. In synchronous mode, admit of step M follows postprocess of step M-1, so every host reader on main stays exact. Every rank applies results by step number, and the take-back never delays the token cut.

Reserved length and the take-back

stored_len stays the reserved length, so main's commit and arena are unchanged, and the verified length derives from it. Admit, layout and the host copy read the reserved length. Attention, write slots and positions read the GPU length. Eviction and the client-visible limit read the verified length.

No new reservation term. Commit, at engine.py:2730, already counts k + 1, so admit, its sized refusal and the arena fast path admit_rows, at kv/arena.py:295 on main, cover every slot the next step can write. #1133 and #1090 add a second reservation number instead. Each drafter declares how much KV it writes past the verify block, and startup refuses any drafter that declares more than zero.

Length, never pages. A take-back lowers stored_len and never frees or moves a page, and the next step overwrites rejected slots. A length change commutes with offload, reload and teardown, and moving pages outside admit would need journaling across ranks.

When. Postprocess records each result in the take-back ledger. A result is due at step M when its step is at or before M-2 in asynchronous mode, or M-1 in synchronous mode. At the head of pre_admit or admit of step M, whichever comes first, the take-back pass applies the due results of the step's own rows before any resource's admit, because applying in postprocess would change stored_len under the plan thread and let TP ranks admit from different lengths. A row outside the step keeps its results until it runs or is removed, and an offload applies nothing. Engine.offload_request marks a moved row returning through _mark_offloaded, at engine.py:3923, and releases its resident slot, and a follower rank, which replays rank 0's journal, marks it in every OFFLOAD replay branch. VerifyLedger.take_due, at draft/ledger.py:146, makes every result a returning row holds due at its first admit back, on every rank, and raises on an unrecorded entry, with no new journal entry. The pass calls take_back(rids, n) on each resource that defines it for rows where n, the span commit counted minus accepted + 1, is non-zero. The counted span differs from k + 1 when commit drops the span of a stream offloaded mid-step, at kv/manager.py:3122-3130 on main, and stamps its rows as KVStep.dropped, which committed_spans reads.

Exactly once. A step can reach the pass twice, through pre-admit then admit, a dropped pre-plan or a refused admit. Lockstep ranks commit a node's steps in the same order, so each rank's own commit count numbers them alike. At engine.py:1202-1203 the engine passes the ledger's pending and unrecorded queries to main's resident planner. Applying a result pops it, main's clear_preplan unwind leaves an applied take-back in place, and admit raises if a due result was never recorded. The engine voids a step that failed between commit and postprocess, and handle release purges the ledger's per-request state, _returning included, as mstar/worker/rid_table.py lists.

KV's take_back, at kv/manager.py:5702, is a columnar KVManager method under the manager's lock that raises instead of clamping, on no stream, a stream that no copy holds, more than one group, a step in flight, a negative length, or a host copy below its protected prefix, and VerifyLedger._take raises on n outside [0, counted span]. It lowers stored_len and host_len and bumps generation and reset_generation as commit does at kv/manager.py:3154 on main. For a row in _readonly_rows it calls _drop_readonly_plans() but never _touch_plan_cache, which would rewrite the resident row from the host. It stays Python, because worker_phases on one H100 puts the verify bookkeeping under 0.5 ms a step on DSpark at concurrency 32 and 64, as Benchmarking gives.

Lengths on the GPU

Resident planning on main refuses any span but 1, at resident.py:627. On verify walks the resident graph instead adds accepted + 1 to a row's KV length cell, and host_len catches up at take-back, so no host write touches the cell while results are unapplied. The layout footprint and every resident size check include the KV slack. Verify walks refuse a kv_layout without the slack, KV labels that the node publishes to other nodes, KV forks, retention, attention sinks and multi-group pools.

Slot eviction. main rewrites an evicted resident row from stored_len, too high while results are unapplied, so the rewrite keeps the device length cell. A row with an unrecorded result keeps its slot, and one whose results are all recorded may lose it, because its next admit applies them first. In asynchronous mode the returning mark keeps ResidentPlanner from raising "unapplied verify results but no slot" for an offloaded row that is reloaded before the node's next commit.

Verify batches stay within captured sizes. In asynchronous mode every verify step must be captured and resident-planned, because a host-planned step plans from stored_len, at kv/arena.py:343 on main, and would attend to rejected slots. A verify-walk attention declared without AttentionStep(verify=True) would fall back to such a plan, so _refuse_uncaptured_verify, at engine.py:1031, refuses at startup an asynchronous node whose capture left no verify bucket or planned one on the host, naming AttentionStep(verify=True). It also refuses one whose sampler keeps no loop-back slots, as with MSTAR_DEVICE_LOOPBACK off.

The verify walk's batch cap is the largest captured bucket, which ranks agree on even when memory planning drops one. Extra requests wait a step, and the engine raises on a verify step with no resident capture, except that synchronous mode allows eager verify steps. Qwen3-8B's ladder is MSTAR_DECODE_BUCKETS, default 1,2,4,8,16,32,48,64,96,128, so verify captures reach 128 x (k + 1) tokens, 1,024 for DSpark at k = 7.

Each resource corrects itself

Every resource with per-token state fixes it from the verify result and writes no other's plans. A position is KV length plus a per-request offset, so the position resource lowers its host counter at take-back by KV's amounts, and the offset, which also absorbs rotary jumps after an image, never changes.

Each resource spec returns one ResourceCapabilities from config through NodeResourceSpec.resource_capabilities(ctx). Its verify_refusal says whether the resource can verify and its async_verify_refusal whether the device can correct it, each None or a reason. The base class refuses both, so an undeclared resource is never assumed safe. Resource.capabilities() narrows the answer per rank, for instance by whether FA3 runs on its GPU. A verify_refusal refuses drafting on the node, and an async_verify_refusal makes auto pick synchronous mode. Resource also gains these methods:

  • bind_verify_walk(walk, block, slack) sets up the verify walk, chooses its kernel, or refuses the node.
  • committed_spans(step, rids, label, span) gives the span the last commit counted.
  • take_back, for a resource with per-token state, and correct_verify_state, for one that corrects its state in the graph.

uncorrected_problems, at draft/startup.py:355, refuses a recurrent pool on the node that no linear-attention resource reruns after a verify step.

Verification in the sampler

SamplerResource gets verify beside sample, building on its TP broadcast and #938's in-graph scatter. It returns tokens of shape [n, k+1] and accepted of shape [n] and writes the bonus token to the loop-back slot. engine_inputs.verify_output puts the block under TokenStop.buffer, and the block and accepted counts into the stop-check buffers, which reach the host in the stop-check copy.

Acceptance rule. A greedy request accepts drafts while each equals the target's top token, then emits the top token. A sampled request accepts draft d_j with probability p_j(d_j) under the target's filtered distribution p_j. At the first rejection it draws the bonus token from p_j without d_j, renormalised, and after k acceptances from p_{k+1}. Every drafter here drafts by argmax, for which this is standard speculative sampling, min(1, p(d) / q(d)), so the API takes no draft probabilities. With random numbers independent of the drafter, the output follows the target's distribution exactly, with plain sampling's filters in the same order.

One graph per bucket. Greedy and sampled rows share one graph per bucket and read their own resident settings. Since #1328 a sampler may take a greedy twin capture, an argmax-only capture of each batched bucket for all-greedy batches. Verify never reads the greedy flag, so a verifying sampler takes no greedy twin, and SamplerResource.lean_captures returns False once bind_verify has run at load. Prompt steps on a drafting node keep Sampler.sample's eager greedy short-cut, which only the served real-weights tests exercise, and nothing tests it in isolation.

Random numbers. verify draws from a counter hash of the request's seed, salted apart from other streams, and the shared offset. Each step the graph adds a fixed stride to the offset, the larger of k + 1 and plain sampling's, on every rank and in both modes, so no bonus draw overlaps the next step's. A request draws the same numbers in any batch and on any device, which plain sampling's FlashInfer path does not guarantee.

Cutting tokens at stops and the budget

One engine-owned, columnar token cut, engine.cut_verify_step at engine.py:4362, runs in StepExecutor postprocess after HostCopies.prematerialize_for_check_stop and before engine.step_finished, which extends prefix keys. The cut computes counts = min(accepted + 1, budget left), cuts after the first stop id and adds a stopped column. Corrections read accepted, never the emitted count, so with k = 7, 2 tokens of budget left and 5 drafts accepted, the GPU keeps 6 KV slots and the cut emits 2.

TokenStop stays the only stop declaration and gains verify_walks, which the engine sets at load. On those walks the cut reads stop_ids and ignore_eos. withhold and the Rust runtime's runtime_check decide from one value per row, so the engine refuses withhold and turns off a model-declared runtime_check on verify walks with a log line. In the Python stop check, the per-request check_stop raises on a verify walk, and check_stop_batched reads the cut's VERIFY_STOPPED column when present, otherwise the stop buffer's first value.

Every iteration emits at least one token, so #604's iteration cap cannot fire before the cut, which enforces max_output_tokens exactly. Inline ResultTokens frames hold one integer per request, so verify walks send per-request result tensors. #1359's usage and finish_reason count what the cut emits, which test_chat_usage.py checks on the CPU through the real cut, inline postprocess and API server, failing without the cut.

Drafter state, weights and inputs

Taps. An optional hook in run_layers, at mstar/model/components/decoder_layer.py:102, copies declared taps into TapResource's buffers. It copies as soon as forward_mlp_residual returns, because on one GPU add_rms_norm updates the residual in place, at mstar/model/components/norm.py:40-56.

Weights. Each drafter's weights come from its own checkpoint, and a drafter without an embedding or LM head shares the target's. Startup refuses a vocabulary or weight dtype mismatch, taps outside the target's layers, and a hidden-size mismatch for a drafter that reads taps or target hidden states. EAGLE-3's 32,000-token vocabulary, against Qwen3-8B's 151,936, goes through a draft-to-target TokenMap, and the drafter remaps each argmax inside propose, before a chained pass re-embeds it, so the engine only sees target ids.

Masks. Each backend implements or refuses every declared drafter mask. DSpark drafts over a k + 1-row block in the verify block's KV slots, attending non-causally to the context and the block's first k rows but not its last. It declares this hidden-last-row mask with AttentionStep.hide_last, which #1133 has on FlashInfer only. #1320 ports that FlashInfer version for synchronous mode, and on FA3, which lacks it, runs DSpark's block as a non-causal call over a per-request KV length one row short.

Media and chunked prompts. Startup refuses media on a drafting node unless its drafter declares that input kind, and the stack is text-only, since Qwen3.5 image prompts left with #1323. Verify walks never prefill in chunks, and each drafter declares its chunk-boundary rule or is refused. DSpark needs none, because its KV at p depends only on the taps at p, and a draft model needs none either, because its KV at p depends only on tokens up to p. EAGLE-3 declares no rule, so chunking_problem refuses it on a target that chunks prompts.

Prefix caching and eviction

The prefix index seals every full keyed page at commit, before a verify result is known, so keying reply tokens would seal pages that hold rejected drafts. Drafting nodes key prompt pages only, at the cost of prefilling each turn's previous reply again, and the engine turns off a declared growth=True with a log line, as main does above one rank at kv/manager.py:895-898. The take-back thus reaches only unkeyed pages and never uses main's page-moving correct_len or _drop_sealed_tail, which #388 uses. A prompt page also holds the drafter's KV, so a cached prefix needs the same target and drafter checkpoint. The real-weights "repeated prompt hits the prefix cache" test passed for all four drafter arms on one H100 at cbd655c34.

An offload files no prefix keys for reply pages, because the first verify commit drops the stream's chain, and a removed request drops its unapplied results with its pages.

Preemption

In main's preemption series, #871 and #872 move admission into the runner, with allotments for growth and for guaranteed and preemptible requests, and #887 lets a request pass the queue head. KVManager.preempt and make_room move victims to the host through offload_request when a pool runs short, and above one rank followers replay a journal of drops and moves from #873. The stack follows five rules on drafting nodes, which docs/design/admission_and_preemption.md §1, §6 and §7 describe, and each of ten mutations that break one of them fails a test.

  1. The take-back stays at the head of the runner's admit and pre_admit.
  2. A row outside the step keeps its results, and a row moved to the host is marked returning.
  3. Rank 0 never chooses a victim with an unrecorded result, whose true length exists only on the GPU. At the top of the allocation-failure branch, StepPipeline._refuse_unrecorded_victims, at mstar/worker/step_pipeline.py:735, raises, as an assertion, if any row of a drafting node holds an unrecorded step. It runs before preempt, Admission.relieve and the fallback eviction choose anything, which also covers the victims KVManager.preempt and make_room choose inside the pool, so Admission._evict carries no filter. make_room picks only stalled rows. Under MSTAR_KV_DEBUG_ASSERTS, Preemption._check_verify_victims, at mstar/worker/preemption.py:180-191, checks that no moved or dropped victim holds an unrecorded step, and that no victim the pool asks again for its allotment holds a verify result unless it is returning.
  4. The engine turns rebuild off for drafting nodes through rebuild_policies_of(model), at engine.py:275, and _make_rebuilds refuses a drafting node's policy and warns when rebuild_host_mb is set for nothing. Through main's Rebuilds.can_rebuild, KVReqConfig.rebuildable is then False, so no request on a drafting node is ever dropped.
  5. A guaranteed footprint covers the KV slack. In test_spec_guaranteed.py a guaranteed request reserves prompt plus cap plus scratch 2(k + 1) and every grant fits, and with scratch 0 the last steps are refused.

The returning mark is the only mechanism keeping stored lengths equal across ranks through a host move, and no check compares them.

On one H100 at 967165cbc the KV-pressure case passed for all four drafter arms, each preempting 172 to 231 times with no drops, against plain decoding's 143, and the fp32 screen's ratios against plain decoding were 1.00 to 1.14, every p-value at least 0.59. On 2 x H100, asynchronous with MSTAR_KV_DEBUG_ASSERTS=1, the TP2 pressure case passed for DSpark at ce0b27d and for DSpark and EAGLE-3 at 75fd5be975, with no drops and screen ratios of 1.00 to 1.26, every p-value at least 0.58.

Tensor parallelism

Broadcasting rank 0's drafts, and rank 0's verify result in place of the plain token broadcast, makes ranks write KV for the same tokens and agree on GPU lengths. The token cut and the take-back are deterministic over replicated state, issue no collectives and cannot fail on one rank alone, so no per-step vote is needed.

Above one rank the sampler does not use the in-graph scatter, whose fixed rows would need admission agreed across ranks, so its loop-back masters still grow as requests register. Since a captured graph holds fixed addresses, #1324's carry staging copies each row's carry into the rows the graph reads before every captured verify step's forward, on the same stream after the previous commit. Each rank computes its carry from the agreed drafts and result, so the copy needs no collective, and a carry that differs across ranks changes only rank 0's drafts and acceptance, never the emitted tokens. Under MSTAR_KV_DEBUG_ASSERTS above one rank, the sampler counts carry digest differences without raising. A layer set with fewer KV heads than ranks is checked at config level only. This RFC does no TP performance work.

Recurrent state

GDN layers update per-request state in place, so the resource must undo a rejected draft's update. #1322 adds that undo, tested on a hybrid toy, and no shipped model uses it until Qwen3.5 MTP moves onto mstar.engine.state.

The verify forward leaves each GDN slot and short-convolution window unchanged, with FlashInfer's state update disabled and UPDATE_STATE off. Once accepted counts are known on the GPU, correct_verify_state reruns each layer's recurrence over the step's k + 1 inputs from the slot with gates closed on rejected tokens and rebuilds the window from the kept inputs, so each state ends as if only the kept rows had run, with one capture per bucket, no host length read and no second copy. CLOSED_GATE is a finite -1e4, because -inf gave NaN at 16 or more rows on H100, which by hypothesis comes from the kernel's wider-batch path. Decode-time recurrent prefix snapshots are off, because main saves them from a count that includes rejected tokens, at recurrent/pool.py:735-738. GDN verify is refused off CUDA, by current_platform, without FlashInfer's gated_delta_rule_mtp, above one rank, on a state pool that isn't float32, and on a pool with disable_sink_slot or more than one label.

On H100, test_gdn_verify_device passed 49 tests at 606ebda and 75fd5be975, including finite states at 16 to 64 rows and a returning row with a pending result. For invariant 4, main's test_gdn_cuda_graph and test_qwen3_5_device_loopback passed at main and at #1322 and #1325 heads 3e7878063 and 967165cbc, and the shared conv kernel is unchanged since.

Request options and feature conflicts

The scaffold forwards only its SAMPLING_KEYS to the sampler, and the conductor reads seed and max_output_tokens itself. On Qwen3, temperature, top_k, top_p, seed, ignore_eos and max_output_tokens are honoured, and min_p only where the checkpoint defaults it, through enable_min_p, with an HTTP 400 otherwise, as without drafting. #1319 lists the keys Qwen3's request_kwargs exclude. Request defaults come from the same generation_config and serving config as plain sampling.

A verifying sampler's SamplingReqConfig.validate refuses repetition_penalty other than 1, return_logprobs, penalize_prompt false, and repetition_window or min_tokens above 0, which the conductor turns into a 400. repetition_window and min_tokens depend on history the take-back does not restore.

As main does for sliding-window attention, at attn/config.py:175-180, the engine turns off, with a log line, a default or model-declared feature, such as reply-token prefix keys and TokenStop.runtime_check, and startup refuses one the operator asked for. Startup also refuses LoRA, a prefill and decode split, sliding-window, bounded or ring KV, multi-head latent attention, DeepSeek sparse attention, a sampler built with log-probs or a windowed repetition penalty, and any split or combined walk that holds the verify walk, because mixed prompt-plus-verify steps are not supported yet.

Correctness

  • Plain parity. feat(qwen3): add plain dense Qwen3-8B on the shared decoder scaffold (PR 2 of 7 in the stack) #1319's gate, in mstar/testing_utils/logprob_gate.py, compares served greedy replies with transformers in fp32 and bf16 over 8 short prompts and 3 long texts. The ratio R of served to bf16 log-prob error against fp32, each trimmed at its 0.98 quantile, must be at most 1.5, and served flips, tokens whose greedy choice differs from fp32's, at most bf16's plus 3. The gate must reject a rope_theta x1.1 control, and an rms_norm_eps 1e-5 control was dropped because its served log-probs match the real model's at every length. On 1 x H100 at 4d22dcb the real model passed, at R 0.911 and 0.962, and the gate rejected the control on both long checks, at R 2.374, but not on the short set, at 1.038.
  • Speculative against plain, greedy. Where a speculative reply first differs from the plain one, the fp32 reference's top-two log-prob gap must be at most MARGIN = 0.25.
  • The fp32 screen. For the KV-pressure and TP runs, test/integration/fp32_screen.py teacher-forces every distinct served reply through a float32 reference, and a departure is a served token that is not its top token. An arm fails if a one-sided conditional binomial test finds its departure rate above plain decoding's under the same pages and arrival orders, if a hypergeometric test finds too many departures deeper than plain's 99th-percentile log-prob deficit, or on a tripwire, a served token at a high fp32 rank or deficit, a burst of departures after a reload, or a reply cut short where its plain pair goes on.
  • Exact cases. Byte equality is the bar on fp32 toy models, and between the two modes at compile level 0, because the default level tunes kernels per process. An always-reject drafter equals plain decoding, and an oracle drafter has every draft accepted.
  • Sampled. A sampled reply must follow plain sampling's distribution, not reproduce its samples. The check runs on the CPU only, in test_spec_verify_sampling.py, and a served distribution test and the random-number overlap tests that Speculative sampling under top-k and top-p, with one filter kernel for the candidates #1103 asks for are planned and do not exist yet.
  • Lengths. Every verify test asserts that the verified length is at most the GPU length, which is at most the reserved length.
  • Negative controls. Each mechanism a PR adds has a test that fails when that mechanism alone is removed. All 12 mutations of the chain helpers fail test_chained_draft.py or EAGLE-3's first-carry test. The debug victim check, the Speculator.can_speculate guard, and the real-Preemption path in test_spec_worker have negative controls.
  • No cost elsewhere. A node that does not verify reads no draft or verify state. Spec and step dumps for Qwen3.5 and Command A+ are identical on main and at the stack top, a CPU test checks over the 11 scaffold nodes that each declared loop-back and token stop is the node's own, and code review, not a test, checked that drafter-only branches stay off. worker_phases on one H100 at -O1, M*'s compile level 1, alternating arms with a fresh Inductor cache per boot, shows no regression. In two rounds at concurrency 32 and 64, the stack's iter_total p50 on Qwen3.5-0.8B was 0.5 to 2% below main's, and on plain Qwen3-8B within 0.2% of feat(qwen3): add plain dense Qwen3-8B on the shared decoder scaffold (PR 2 of 7 in the stack) #1319's. A plain Qwen3-0.6B sweep for invariant 6 has not run.

CPU tests run toy models, including a GDN and attention hybrid, with scripted drafters and a real Worker, in synchronous mode end to end and with resident tables updated as a captured replay does, and cover exactly-once take-back, eviction, offload, reload, handle reuse, TP divergence, take-backs at every page offset, and every refusal. On every head, full CPU Core fails only main's own two tests, the lockstep simulation, test_tp_async_lockstep_sim.py, passes 11 tests at #1319 and 16 at #1320 and later heads, and ruff is clean.

The GPU gates ran on H100 with torch 2.12.1+cu132, FlashInfer 0.7.2, FA3 3.0.1 and Triton 3.7.1:

  • FA3 against FlashInfer on DSpark's mask passed in test_spec_attention's CUDA cases, 27 at 51211940f, an earlier feat(engine): add speculative decoding with verify steps and KV take-back (PR 3 of 7 in the stack) #1320 head, run with test_h2d_to_device, 10 passed and 6 skipped, and test_worker_thread_device, whose 2 tests skip on one GPU. It passed in the real-weights test_flashinfer_and_fa3_paged_kernels_agree for all four arms.
  • Greedy and capture parity on real weights passed in both modes, as did the served sampled test, which checks for non-empty replies and τ above 1.5 at checkpoint defaults.
  • TP2 on 2 x H100 over NVLink passed for every drafter in both modes, and at cbd655c34 test_tp2_verifies_as_one_gpu_does and test_tp2_drafts_as_well_while_the_loopback_masters_grow, which covers carry staging, passed for all four drafter arms.
  • The KV-pressure case, 128 pages of 16 tokens and 4 rounds of 32 requests, asserts preempted above 0, drops at 0 and reloads above 0 from /metrics and the engine's reload lines, since main no longer logs "offloaded victim=", and passed as Preemption gives.
  • The τ gate, which requires τ within 5% of vLLM's with the same drafter, covers EAGLE-3 only, at 2.12 and 2.14 against vLLM's 2.08 to 2.26.

At the final heads (#1322 at 606ebda, #1325 at cbad78a) the device modules, the single-GPU real-weights tests for all four drafter arms (24 passed) and the TP2 tests (8 passed) pass; the single-GPU KV-pressure rounds ran at 967165cbc, before the last review fixes. #1319 has had no GPU run of mstar serve with the shipped config through the OpenAI route, no eager GPU run and no TP run. That #1321's write_kv on the unfused attention path writes what the fused path writes is checked on the CPU only, in test_attention_write_kv.py, and on CUDA it is a hypothesis. A served Qwen3.5 MTP run from #1323 set the thresholds in test_qwen3_dspark_real_weights.py.

Benchmarking

benchmark/worker_phases runs first, then the end-to-end client. The planned arms are plain decoding, each drafter, both modes, FA3 and FlashInfer verify, and vLLM with the same drafter. The headline metrics are τ, p50 TPOT, and the concurrency where speedup falls below 1, judged against an A/A run.

Because benchmark.runner's client counts one token per streamed chunk, undercounting by a factor of about τ, speculative arms use #1321's benchmark/qwen3/bench_openai.py, which re-tokenizes replies and reads τ from the mstar_spec_* counters, and vLLM's τ comes from its SpecDecoding log lines. Prompts come from main 9a50fdf's tree through --root, so every arm sent the same 64 coding chats of 596 to 2,327 prompt tokens. Each run was greedy with ignore_eos and 256 output tokens, one warmup wave and then at least 5 waves, 16 requests at concurrency 1 and 160 at 32, on one H100 SXM 80GB with the other GPU idle, the software Correctness lists and M* at -O2, with vLLM 0.33.0 on torch 2.14.0+cu130. The heads were 77462e224 on #1324 and e0f7d1e78 on #1325, and later commits change only tests, docs, comments and refusals.

At temperature 0 on Qwen3-8B, in output tokens per second, with p50 time per output token (TPOT) and time to first token (TTFT):

Arm c1 tok/s c1 TPOT p50 c1 TTFT p50 c32 tok/s c32 TPOT p50 c32 TTFT p50 / p90 τ at c1 / c32
vLLM plain 145.5 6.68 ms 55 ms 3,227 9.30 ms 101 / 209 ms
M* plain 162.1 6.00 ms 56 ms 3,292 9.17 ms 49 / 183 ms
vLLM EAGLE-3, k = 3 260.3 3.71 ms 38 ms 5,602 5.10 ms 35 / 195 ms 2.08 to 2.26 over log intervals
M* EAGLE-3, k = 3 262.0 3.63 ms 54 ms 4,790 5.98 ms 45 / 461 ms 2.12 / 2.14
M* DSpark, k = 7 314.0 3.06 ms 55 ms 4,871 5.91 ms 56 / 437 ms 2.92 / 2.89

No request came back short. DSpark has no vLLM arm in this run. vLLM 0.33.0 does serve DSpark: the 2026-11-08 round, on an older tree of this stack, measured vLLM's DSpark at k = 7 at 320 / 3,944 / 7,870 tok/s at concurrency 1 / 16 / 64 against M*'s 325 / 3,321 / 6,091, with τ 2.90 against 2.91. The draft model was not re-measured on this client, and the KV-pressure workload on one H100 at 967165cbc gave it τ 3.086 at k = 3 and 3.599 at k = 4 on other prompts.

  • EAGLE-3 is on par with vLLM at concurrency 1 and 15% below it at 32, where its TTFT p90 is 2.4x vLLM's. τ matches, so acceptance is not the cause. In worker_phases at -O1 at concurrency 32 and 64 the GPU stream is busy for the whole step, so host work is not on the critical path. A read of the phase logs and both engines' code, with no new GPU run, points at two causes, both hypotheses until profiled: a drafting node runs each arriving prompt as its own prefill step, about 9-11% of its GPU time, where vLLM mixes arrivals into its decode step, and M*'s verify step itself is longer than vLLM's. This RFC records the gap without tuning it.
  • DSpark's event_sync is 76 to 78% of the step at concurrency 32 and 64, so it is GPU-bound. In worker_phases at -O1 its iter_total p50 is 14.94 and 25.00 ms there, and EAGLE-3's 11.44 and 16.26 ms, with engine.verify.cut at 0.29 to 0.43 ms.
  • At concurrency 32, DSpark is 1.48x plain decoding and EAGLE-3 1.46x. Separate -O1 runs at 64 gave a rough 1.20x and 1.36x. Before the stack moved onto the shared decoder scaffold, which this RFC calls the port, DSpark was 1.23x and EAGLE-3 1.41x at 64, so the scaffold may shrink speedups at 64 slightly, a hypothesis while the effect of mixed steps and greedy twin captures is not isolated.

If FA3 verify loses to synchronous FlashInfer at any concurrency, the attention choice gets revisited. Before the port, in H100 throughput, FA3 beat it at concurrency 1, 4, 16 and 64, by 25 to 84% on EAGLE-3, with τ within 0.7%. FA3 now gathers through #1143's resident_gather and runs to 128 rows.

#1322 switches to replay if the GDN rerun's state differs from a from-scratch reference by more than fp32 tolerance, or the rerun adds over 10% to verify step time at concurrency 16 or below. Before the port, on Qwen3.5 MTP in H100 wall time, it took 2.3 to 6.7% of the verify step on Qwen3.5-9B at 1 to 16 rows, 7.2% on 2B at 16 rows, and about 14% on both at 64 rows. The GDN code has changed since. No served model runs the rerun until MTP, so benchmark/gdn_verify's served share, and whether the 10% rule should cover batches above 64 rows, wait for it.

Still unmeasured are the planned arms the table lacks, startup time, capture graph memory and dropped buckets at 128 decode rows for plain and drafting configs, where running out of memory or dropping buckets caps the drafting ladder at 64, concurrency 16, 64 and 128 against vLLM, sampling at temperature 1, MT-Bench, HumanEval, GSM8K with #1321's gsm8k.py, Spec-Bench, ShareGPT, 8k to 32k-token prompts, an A/A run, the cost of the stack's torch.compiler.disable boundaries through a TORCH_LOGS=graph_breaks,recompiles count against main's, and the EAGLE-3 nsys comparison.

Non-goals

  • Tree drafts, CPU-side drafters, and adaptive or per-batch-size k.
  • For EAGLE-3 and draft models, draft-sampled drafts and draft probabilities, hybrid or recurrent draft models, a draft model with a different tokenizer or vocabulary, a drafter whose TP degree differs from the target's, a separate KV dtype per layer set, EAGLE-3 or draft models on Qwen3.5, and draft models outside the Qwen3 dense family.
  • Sliding-window drafters, or a sliding-window target with any drafter.
  • KV publication or remote transfer from a drafting node.
  • Prefix keys over reply tokens.
  • AMD, and asynchronous mode off Hopper.
  • GLM-5.2 MTP at TP8, Kimi Delta Attention and Mamba-2 in this stack.

Alternatives considered

Option Used by Why not
A separate drafter KV pool #1133, #257 Two pools split memory at startup, so one can refuse requests while the other has room, and their prefix indexes can disagree. #1133's draft-model pool, one token shorter, adds a second length and incomplete prompt pages.
Drafter layers as a KVGroup none main's KVGroup, at kv/config.py:794, splits the page table, which verify, take-back, resident tables, forks and remote transfer refuse.
A separate drafter node none It adds a hop every step.
One resource rewriting others' plans, or patching a stale pre-plan on the host #1090, #1133's correct_preplan Plans are opaque to all but their owner, and a host patch adds a third mode that GPU lengths make unnecessary.
Verify then draft, or a seed pass, an extra target pass for the drafter's first input #1090, the GLM stack Either keeps more state across steps, and a seed pass errs at chunk boundaries.
Per-request correct_len, commit_adjust, or no take-back correct_len in #257 and the GLM stack, commit_adjust in #1090, none in GLM-5.3 dense The first two are per-request model code and lose the correction when commit drops an offloaded span. Without a take-back, lengths and KV grow wrong.
Cutting through -1 padding, a GPU-thread wait, or a model hook #1090, the GLM stack Every consumer must filter padding, the wait cost 1.8 ms of a 10.2 ms step at concurrency 1 in a commit on #1117's branch, and a model hook runs after prefix keying.
k + 1 recurrent state copies, scratch states copied back in the forward, or host snapshots #1090, #1133's contract For #1090's copies at 27B TP4 that is about 200 MiB per request and an estimated 12.8 GiB at batch 64, with an AMD-only kernel. On Qwen3.5-9B a state is about 48 MB, so at k = 3 the k + 1 = 4 states take about 192 MB a row and about 12 GB at 64 rows. Host snapshots are too slow for asynchronous mode.
Replaying the accepted inputs on the next step the GLM stack, #257 On one H100 at f62111f42, an earlier tree, benchmark/gdn_verify found replay slower than the rerun in 4 of 8 configurations and at most 6% faster in the rest. FlashInfer's chunked GDN kernel checkpoints state only every 64 tokens on H100, so a valid replay costs two chunk passes, like the rerun.
Fixed loop-back rows for the carry, reserved at admission under TP none It needs a cross-rank admission decision and a journal of reserves and releases.
The carry as a per-request block in RecurrentStatePool none restores_prefix is True, so a carry-only pool shuts its node's prefix cache, and it is refused next to GDN.
Applying results at the offload, with a SETTLE journal entry the stack before the rebase The returning mark gives TP agreement by construction, with no new journal entry.
Qwen3.5 MTP on the resource path with lint exceptions, or extending the State Contract in this stack none The first is a parallel path, and the second adds permanent contract API that needs the contract owners.

Open questions

  • Stage order. This RFC argues from the main loop that postprocess of step M-2 precedes admit of step M on every worker path, and that every rank commits a node's steps in the same order. Under that order the lockstep simulation keeps the ledger in agreement across 2 and 3 ranks in both modes. Its 16 tests include one that a follower never replays an offload of a row it hasn't recorded, and one that the offload replay rests on the serial path draining. Real-Worker tests run through StepPipeline, covering _refuse_unrecorded_victims and the Speculator guard, and TP2 passes on GPU. No test covers the micro-scheduler running other walks between a node's verify steps, or the hypothesis that Worker: the next step's (node, walk) as a scheduling policy, with model priorities #1311's Priority policy, which no config uses, can make a run of verify steps yield to other walks before its first step.
  • The scaffold's verify path on the State Contract. mstar.engine.state has no drafting API, meaning no verify read or step on an Array, KV layer set, taps, verify sampling, carry, or speculative GDN update with its correction. Its API only grows after release, so additions need the contract owners' agreement. Qwen3.5 MTP needs it, and so does any scaffold model moving onto the contract, Qwen3 included.
  • Thresholds. With feat(qwen3): add plain dense Qwen3-8B on the shared decoder scaffold (PR 2 of 7 in the stack) #1319's gate decided, still open are MARGIN = 0.25 for the speculative gates, the τ gate, the Benchmarking thresholds, and the Fp32Precision scope. Importing mstar.engine sets matmul precision "high", at mstar/engine/__init__.py:91, and only test_qwen3_real_weights.py:184-187 scopes Fp32Precision, so that fp32_screen.py and test_qwen3_dspark_real_weights.py use TF32 matmuls is a hypothesis, unchecked on a GPU.
  • Stop strings. Whether the API server, which matches stop strings after detokenizing, truncates inside a multi-token chunk is unchecked. test_chat_usage covers stop ids and finish_reason only.
  • Host pools. What drafting costs on a KV pool without a host pool, where every request is guaranteed its whole footprint, is unmeasured.
  • Take-back and page checks. Whether take_back's host-only branch is reachable, which by hypothesis only a hand-built test does, and whether startup should enforce k + 1 <= page_size, which the shipped configs meet at 8 against 128.
  • EAGLE-3 drafter rows at a prefix hit. A published page's last drafter row depends on the next token, which the prefix key omits, so a hit followed by another token reuses a stale row and lowers acceptance by an unmeasured amount. Whether to rewrite that row on a hit is open.
  • Untested paths. Nothing tests a prefix host-tier write and fetch with a drafter storage on CUDA, or a replicated-head layer set end to end.

Next

Mixed prompt-plus-verify steps come first, because before the port prompt-only steps took 18.5% of GPU time on the held Qwen3.5-9B MTP node at concurrency 16 on one H100, against 14.2% for plain decoding's mixed steps. After that come the scaffold's verify path on mstar.engine.state and Qwen3.5 MTP on it, then:

  • drop-and-rebuild on drafting nodes, which needs the held results applied before the drop, Rebuilds keeping every token a verify step emits, supports_drop on the tap resource, and a parity check after a rebuild;
  • a debug digest of admitted rows' stored lengths across ranks in page_trail;
  • a recurrent pool that opts out of prefix restore, so the carry can leave the sampler;
  • GLM-5.3-Flash's KDA onto correct_verify_state, an issue still to be opened;
  • a capabilities field for default kernels, a tap as a step output, and decode on FA3PagedWrapper.

Activity

  1. stephen-dwq commented on Nov 5, 2026

    @stephen-dwq
    Collaborator

    Since accepted lives only on the device and the host only has bounds on L, any host write to the length cell kv_len[slot] before take-back is wrong. Beyond slot eviction, three paths on main still copy host stored_len into that cell:

    • KVResidentTables.row, on whole-row rewrites (kv/manager.py:4715)
    • refresh_len, for rows a host-planned step moved (:4723)
    • stage_step's whole-row rewrite when a row needs 2+ new block-table columns (kv/resident.py:102), which k + 1 steps hit routinely

    Verify walks need all three to skip that cell.

  2. nick-rui commented on Nov 6, 2026

    @nick-rui
    CollaboratorAuthor

    Update (Nov 6): the body above is revised.

    1. Drafters declare their own resources. Each declaration joins a resource the target declares or is the drafter's own. KV joins the target's pool as a KV layer set: it shares the page table, allocation, eviction, prefix keys and take-back, and keeps storage shaped to its own geometry. A separate pool would split memory at startup and could give prefix hits that disagree. Milestones 1 and 2 change only the declaration; their storage stays bit-identical.
    2. Milestone 3: EAGLE-3 and separate draft models (Qwen3-0.6B) on Qwen3-8B. Three measurements come first: a draft model's cost per step, the planning cost of an extra attention binding, and time to first token.
    3. From implementing milestones 1 and 2 on H100:
      • GDN undo runs FlashInfer's multi-token kernel with the state update disabled, so the slot is its own checkpoint and state memory isn't doubled. It costs at most 7.2% of a verify step up to 16 rows, and about 14% at 64. Replay was measured and rejected.
      • The drafter carry survives eviction, and positions keep main's counter.
      • Video prompts are refused on verifying nodes, and Qwen3.5 with MTP is tested on one GPU.

    Both milestones pass their gates locally. An async-verify bug on Qwen3.5-9B is being diagnosed before the PRs open.

  3. nick-rui commented on Nov 12, 2026

    @nick-rui
    CollaboratorAuthor

    Update, Nov 11. I revised the body above. These are the changes since the Nov 8 version.

    Design changes after review on #1320:

    • An offload no longer applies a row's held verify results. The engine marks the row returning, meaning it is on the host and not yet admitted back, and all its held results come due at its first admit back, on every rank.
    • The drafter's carry stays on the sampler, since a carry in RecurrentStatePool would turn off the node's prefix cache.
    • Verifying nodes never drop and rebuild a request. Every drafting config sets cpu_offload_pages, and startup warns once per pool without a host pool. Drop-and-rebuild is a follow-up.
    • One ResourceCapabilities replaces the verify refusal hooks, and startup reduces every rank's refusals in one all-reduce.
    • Taps are a general resource, and the FA3 wrapper is FA3PagedWrapper.
    • Startup refuses an asynchronous node that plans a verify bucket on the host.

    #1319's parity gate compares served log-probs with transformers in fp32. Per prompt set, the served error over the bf16 error, each trimmed at its 0.98 quantile, must be at most 1.5, and served greedy flips at most bf16's plus 3. On one H100 at 4d22dcb the real model gave ratios of 0.911 on the short prompt set and 0.962 on the long one. On the long set the gate rejects a rope_theta x1.1 control on both checks, at a ratio of 2.374 and 8 flips against 3. On the short set it accepts the control.

    Testing:

    Benchmarks on Qwen3-8B, one H100 SXM, bench_openai.py with 64 coding chats, M* at -O2 against vLLM 0.33.0:

    • Plain M* is 1.11x vLLM at concurrency 1 and 1.02x at concurrency 32.
    • EAGLE-3 is on par at concurrency 1 and 15% below vLLM at concurrency 32, with TTFT p90 461 ms against 195 ms. τ matches, so acceptance is not the cause. The gap is recorded, and its cause is a hypothesis until an nsys comparison.
    • DSpark is 1.48x plain M* at concurrency 32, with no vLLM arm.
    • Concurrency above 32 against vLLM, sampling at temperature 1 and draft-model throughput are unmeasured. The body lists the rest.

    On one H100 at -O1, worker_phases puts verify bookkeeping under 0.5 ms a step at concurrency 32 and 64, and shows no regression on plain Qwen3.5-0.8B or Qwen3-8B.

    On one H100 with benchmark/draft_model_cost/step_cost.py, Qwen3-0.6B's passes cost 0.60 to 0.85x of a verify step at k = 4 and 1.04 to 1.41x at k = 7, over 1 to 64 rows, so k is capped at 4. The cap is unchecked at 96 and 128 rows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions