Skip to content

feat(engine): add speculative decoding with verify steps and KV take-back (PR 3 of 7 in the stack) - #1320

Open
nick-rui wants to merge 65 commits into
users/nick/sd-prQfrom
users/nick/sd-pr1
Open

nick-rui wants to merge 65 commits into
users/nick/sd-prQfrom
users/nick/sd-pr1

Conversation

@nick-rui

@nick-rui nick-rui commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

PR 3 of 7 in the stack, stacked on #1319 (to keep the stack linear; nothing here uses it). Head 4d8b6e8, base main 2c8ffde.

The engine and resource layer for speculative decoding (RFC #1186): a drafter in a target node's forward proposes k tokens, the target scores them in one verify step of k + 1 rows per request, the sampler publishes how many were accepted, and each resource takes back what the rejected drafts wrote.

No shipped model drafts here; it is tested on toy models. End-to-end numbers are in #1321 (DSpark) and #1325 (EAGLE-3, draft model).

What it does

  • Declaration. The model_kwargs.draft block (DraftConfig); DrafterNeeds, DrafterDecl, VerifyNode and draft_verify_step (draft/drafter.py). A model on the state contract declares a Draft in declare_state; the adapter turns it into the drafter's specs through drafter_resources, which joins the drafter's KV layers to the target's pool as a layer set, adds 2(k + 1) tokens of KV slack and marks the sampler as verifying. Taps (taps/) give a drafter the target's residual stream.
  • Take-back. Commit counts all k + 1 tokens; VerifyLedger numbers steps by commit, so every lockstep rank applies the same result at the same step with no collective, lowering KV and position lengths at a later admit. NodeDraft.cut cuts each run at its stop id and budget.
  • Offload and preemption. An offload marks the row; everything it holds comes due at its first admit back, on every rank. Preemption raises if a victim still holds an unrecorded verify step, which the step order rules out. A verifying node never drops a request, so a pool without a host pool warns at startup. Under main's page lending ([Preemption] Let Preemptible Requests Borrow What Guaranteed Requests Haven't Written #1415), a guaranteed verifying request lends its unwritten reservation and reclaims it by moving a borrower, apart from the verify take-back.
  • Resident planning at accepted + 1: the captured graph advances each row's device length, and no host write touches it until the result is applied. LongRoPE table ids follow the verify span.
  • Sampler. Greedy rows accept while drafts equal the argmax; sampled rows use speculative sampling with per-row counter-hash randomness. A verifying node's prompt-step token is drawn as a mixed step's prompt row draws it (first_draw: the request's own stream at the bonus draw's number), so a seeded reply doesn't depend on which its prompt rode; a node that doesn't draft keeps main's sample. An all-greedy step verifies by the argmax alone (greedy_verify, captured as a lean twin), skipping 1-2.8 ms of full-vocabulary work a call at 128-512 rows (H100, the verify sampler replayed in a CUDA graph). Requests a verifying sampler can't honour are a 400.
  • Attention. FA3's paged kernel reads device lengths for asynchronous verify; FlashInfer is the host-planned fallback (verify_backend pins one).
  • Mixed steps. A verifying node may run arrivals inside its verify steps (VerifyNode.mixed_walk): verify rows lead, prompt rows follow, only verify rows enter the ledger, on FA3's ragged wrapper; the drafter's own attentions on the mixed walk run on the verify kernel too.
  • The asynchronous mixed envelope. Asynchronously a mixed step runs only on a capture, so warmup caps the mixed walk's rows and token budget (a node-level max_batch_tokens override included) at the widest captured bucket that holds its worst step: verify blocks plus a min_chunk_tokens prompt chunk (Engine._fit_mixed_steps_to_captures). A cap that lowers either is logged.
  • Its startup refusals. An asynchronous mixed walk is refused with no token budget, with a prompt walk whose chunking the serving config turned off, or with no captured bucket that holds the worst step. The last message gives the floor no kept bucket holds (min_chunk_tokens capped at the budget and rounded up to the chunk alignment) and the remedies that apply: a lower min_chunk_tokens, memory for a left-out bucket that would hold the step, mode: synchronous.
  • Other mixed-step guards. A mixed walk over a pool the model writes under more than one KV label is refused; a mixed capture that finds no ragged wrapper raises. A captured mixed step zeroes its padding rows in place with main's zero_unplanned_rows (DSpark +1.25% at c64 in a shared-cache A/B; about −0.75 ms per mixed graph in nsys). Over pages larger than 32 tokens, FA3 can read past a row's length in its last page; a one-time warning names resources.kv_cache.page_size (FA3_TAIL_SAFE_PAGE_SIZE).
  • Mode and startup. mode: auto runs asynchronously where every resource can correct itself on the device, else synchronously; one MIN reduction across the lockstep instance makes every rank agree. validate_deployment lists every problem before a worker spawns: geometry, slack, capabilities, a drafter declaring plain RoPE over scaled positions, sliding windows, sequence or context parallelism. Engine.repeat_refusal refuses repeated decode on the verify walk once main's Run several decode steps per launch (step_repeats) #1424 calls it.
  • Observability and docs. mstar_spec_* metrics, three worker_phases verify timers, a bounded-memory offload fingerprint under MSTAR_KV_DEBUG_ASSERTS; docs in .rst and .mdx and docs/design/admission_and_preemption.md.
  • Other models. Every path except the debug fingerprint (under MSTAR_KV_DEBUG_ASSERTS, for every model) switches on only for a node with a VerifyNode. Qwen3.5-0.8B's worker_phases throughput at c64 measured −0.2% / −1.5% against two main arms 1.3% apart at d7b935363f (an earlier local tree of the stack), and its e2e outputs equal main's at aa18f98 (the pushed feat(qwen3): add EAGLE-3 and Qwen3-0.6B draft-model drafting (PR 7 of 7 in the stack) #1325).
  • The plain path at this head. That it is unchanged is a hypothesis from reading the code since those runs, not a measurement: what this PR adds to the sampler runs behind SamplerResource.verify_draft_tokens, and the mixed and envelope code only for a node with a VerifyNode.

One new optional dependency: flash_attn_3 (3.0.1), FA3's paged kernel, which asynchronous verify needs on Hopper; it is installed by no extra (docs/installation.rst), and without it a verifying node runs synchronously on FlashInfer. No new environment variables.

Since the last push (09f2a1e): review fixes, each with its tests and docs.

  • The asynchronous mixed envelope and its refusals (no budget, chunking off, no kept bucket that holds the worst step), its message (the fitted floor, the left-out buckets, memory only for a bucket that would hold the step), and two buckets of one size.
  • The drafter's own attentions on a mixed walk run on the verify kernel.
  • A mixed walk over a pool written under two KV labels is refused.
  • first_draw: a verifying node's prompt token is drawn as a mixed step's prompt row draws it; _VerifyParams types its filter order and slots.

Tests

On an 8 x H100 80GB host, torch 2.12.1+cu132, FlashInfer 0.7.2, FA3 3.0.1. Commits are named per row: 09f2a1e is the pushed head, and 5a74a57 is it less its last four commits (page-size warning and test, plain-RoPE refusal, padding change).

d9e344ac71 is an unpushed earlier state of this head; it differs from it only in test/modular/test_spec_mixed.py. 0023eff82b and dbc627e8f7 are unpushed earlier states of #1325's head, which differ from it only in test/modular/ files.

Command Commit Result
CPU Core (AGENTS.md paths, CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) + test_tp_async_lockstep_sim.py (-n 3) 09f2a1e 17300 passed, 1968 skipped, 2 failed (main's own two, by test id); sim 16 passed
ruff check . (0.17.5, 0.16.0) this head clean
CPU Core, Linux arm64 Docker (torch 2.10.0+cpu, real Triton 3.6.0, mstar_rust; -n 6, OMP_NUM_THREADS=2), failing tests by id against main's on the same host d9e344ac71 13 failed, 17259 passed, 1887 skipped (+40 passed, 5 skipped for test_token2wav.py in one process); the 13 main also fails on this host (arm64 environment failures), no others
test_spec_*.py in the same Docker image, -n 3 this head 561 passed, 64 skipped
test_state_lint.py, test_validate_deployment.py 5a74a57 517 passed
CUDA modules at this layer (spec, draft, drafter, GDN, H2D, resident, Qwen3 LM), 2 x H100 5a74a57 961 passed, 7 skipped
the same this head (2026-11-18) 1070 passed, 7 skipped
The same set across the whole stack, including test_spec_mixed_attention.py and test_spec_page_sizes.py aa18f98 except config comments and two harness lines 1422 passed, 19 skipped
the same dbc627e8f7 (2026-11-18) 1537 passed, 19 skipped
test_spec_verify_sampling.py, test_spec_mixed.py on CUDA dbc627e8f7 (2026-11-18) 109 passed

The real-weights tests of this code are in #1321 and #1325.

New tests for the review fixes, each failing before its fix:

  • test_spec_engine.py: the envelope cap, the min_chunk_tokens refusal, two buckets of one size, the two-label refusal at load.
  • test_state_draft.py, test_spec_mixed_attention.py, test_spec_startup.py.
  • test_spec_mixed.py::test_a_seeded_reply_is_the_same_whether_its_prompt_rode_a_mixed_step: 8 seeds, two drafters, T=20; unfiltered, top-k 20 / top-p 0.7, min-p 0.5, and top-p before top-k.
  • test_spec_verify_sampling.py::test_a_verifying_samplers_prompt_token_is_a_refused_verify_rows_token.

Tests of existing code, checked by mutation:

  • The filtered seeded cases fail when the prompt draw drops the request's top-k and top-p, its min-p (the 16 min-p cases), or its filter order (the 16 top-p-first cases). top_p_first is CUDA-only, so that case stands in for the CUDA filter with torch.
  • test_top_p_first_changes_some_seeded_reply_on_a_verifying_node checks, on both paths, that some seed's reply with top-p first differs from the same request's with top-k first. With the order dropped from every verify draw's parameters (_verify_params), the 64 seeded cases still pass and it fails.
  • test_an_async_mixed_walk_whose_prompt_walk_turns_chunking_off_is_refused fails when the chunking-off clause goes.
  • test_an_async_mixed_refusal_names_the_floor_no_kept_bucket_holds_not_the_configured_one (130 at an alignment of 8 prints 136; 2000 over a 1024-token budget prints 1024) fails when the message prints the configured value.
  • test_an_async_mixed_refusal_names_the_buckets_capture_left_out_and_a_floor_a_kept_bucket_holds fails when the left-out remedy is dropped, when the min_chunk_tokens advice ignores the alignment, or when it is given with less than one aligned chunk of room.
  • test_an_async_mixed_refusal_offers_no_memory_where_no_bucket_capture_left_out_would_hold_the_step, and a too-narrow left-out bucket in the case above, fail when the free-memory remedy names every left-out bucket.

New modules include test_spec_engine.py (the real Engine, a toy target: greedy output equals plain greedy whatever the drafts), test_spec_tp.py, test_spec_worker.py, test_spec_mixed*.py, test_spec_greedy_verify.py, test_spec_page_sizes.py and test_state_draft.py.

GPU runs (2026-11-18)

  • The CUDA mixed modules under the envelope pass at this head (above), with no found no resident capture or planned no FA3 ragged wrapper line; the four mixed steps capped at lines come from test_spec_engine.py's narrowed-capture tests.
  • At 0023eff82b, five asynchronous captured cases of feat(qwen3): add DSpark drafting on the decoder scaffold's verify path (PR 4 of 7 in the stack) #1321's test_qwen3_dspark_device.py failed with this PR's refusal, because that test's mixed ladder held no worst step. The ladder is fixed in feat(qwen3): add DSpark drafting on the decoder scaffold's verify path (PR 4 of 7 in the stack) #1321, and the stack's CUDA set passes at dbc627e8f7.
  • At 0023eff82b, the four shipped drafting configs (DSpark, EAGLE-3, draft model, DSpark TP2) boot past the two-label refusal and serve c16-c64 with no plans one KV label, no-resident or traceback line.
  • A served seeded probe at 0023eff82b (8 seeds at T 0.8, each alone twice and under c32 load) was inconclusive: DSpark matched 3/8 under load against an alone-vs-alone 4/8, and plain Qwen3-8B, the control, 0/8 against 6/8. Hypothesis, unchecked: the second alone run hits the prefix cache, so its prompt runs in another shape.

Open

Not yet run on a GPU

Follow-ups

  • Keeping row_views through the cut; a tap that leaves the step as an output (Qwen3-Omni); decode on FA3PagedWrapper; a capabilities field for a resource's default kernels.
  • Drop-and-rebuild on a verifying node, and a cross-rank digest of stored lengths.
  • Moving FA3's forward probe into one module (draft refactor(attn): move the fa3-fwd availability probe into attn/fa3.py #1437).
  • Main: a tiled _qk_norm_rope_kv_kernel (6.25 against 21.7 µs a launch at 512 tokens, microbenchmark), and zeroing a page's tail at hand-over.

AI assistance was used (Claude Code); the tests run and their results are listed above.

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Nov 8, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
m-star 🟡 Building – Nov 8, 2026, 3:03 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@merceod merceod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static pass on 4199f31, no GPU yet. I read the engine, resource, worker and sampler changes with their callers, against the RFC, plus the tests.

The plain path looks unchanged to me. Every new branch a non-verifying node goes through tests an empty map or set, the three resident kernels reduce to main's arithmetic at span 1, and the sampler and offload helpers are straight extractions. On the verify path I couldn't break the ledger's due rule on leader or follower, the slot pinning for unrecorded rows, the settle before an offload, or the accept and residual math in _chain_verify. One gap is inline.

Two questions:

  1. The slack. The docs say a model sets DraftConfig.kv_slack by hand in three places (KVSpec.draft_slack, the pool's max_seq_len, the layout's scratch), and startup checks each one. drafter_resources already edits the pool for the join. Could it set draft_slack and max_seq_len there too, so a model only declares its drafter? I get that scratch comes from process_prompt, outside the engine.
  2. Verify-step host cost. NodeDraft.cut, cut_verify_step (a slice per row, then row_views = None, which sends the step's outputs per rid), refuse_step and the ledger calls are per-row Python on every verify step. Is there a worker_phases run on a verifying node at c32/c64 yet, or is that waiting on #1325 as the RFC says?

sd-prQ was force-pushed onto main after #1318 merged, so this needs a restack, and GitHub shows it conflicting. Against current main the only conflicts are the import line and load_model's new kwargs from #1184 in engine.py and engine_manager.py, both keep-both. CI hasn't run on this head because the base isn't main, and none of the CUDA rows in the description ran on it. Once it's restacked we'll run the CUDA modules on H200 with and without flash_attn_3, the plain-path A/B on Qwen3.5-0.8B, and plain TP2 under KV pressure.

Comment thread mstar/engine/engine.py
# its rows would commit k + 1 tokens no ledger entry ever takes back
raise RuntimeError(f"{batch.node_name}: verify walk {draft.walk!r} rows inside a combined step")
if draft is not None and ctx.graph_walk == draft.walk:
if draft.asynchronous and ctx.resident is None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In asynchronous mode this only checks that a resident capture exists. If an attention on the verify walk is declared without verify=True, FlashInferManager.resident_gather marks the step host_planned, the bucket never joins host_plan_free, and every replay plans that attention (and the KV host plan) from stored_len, the reserved length, while the write slots come from the device. The row then attends up to k stale slots past its true length and nothing raises. #1321 sets the flag, so nothing breaks today, but the next model that leaves it off gets wrong tokens quietly. Raising here when ctx.slot_lease.bucket not in ctx.resident.host_plan_free, or having refuse_step require verify=True on every AttentionStep of the verify walk, would make that an error on the first step.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed this is a real gap, and it is wider than the flag: any resource that host-plans a captured verify bucket gives the same stale read. Since that is fixed at capture, I'll check it once at startup instead of on the first step. After capture, an asynchronous verifying node is refused if any resource host-planned a captured bucket of its verify walk, and the error names the resource keys. Every input is agreed across the lockstep instance, so all ranks raise alike, with no extra collective. I'm not adding a verify=True requirement in refuse_step, because it would miss a resource that host-plans without the flag. A toy test with a FlashInfer attention declared without verify=True shows the refusal.

@NSagan271 NSagan271 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall shape of the implementation looks correct; I left some comments on design, mainly generalizing some of the added components beyond spec dec

return None


class FA3VerifyWrapper:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this wrapper necessarily specific to spec dec verification? Apart from hide_last and the verify_block marker that FlashInferManager.resident_gather checks with getattr, it reads as a general paged FA3 wrapper over rows of one fixed span, with lengths staged from the host or gathered off the resident tables. At span=1 it's already a decode wrapper. Could it get a generic name (e.g. FA3PagedWrapper, with fa3_verify_unavailable_reason renamed to match), with hide_last as a verify-only option on it? Decode on it can be a follow-up. Ragged prefill would need more, since the graph-mode qo_indptr, max_seqlen_q and the gather_resident length all assume a fixed span.

Related: attn/dense.py already has _fa3_unavailable_reason for fa3-fwd. Now that there are two FA3 wheels, it'd be nice to have both probes in one module (say attn/fa3.py).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It becomes FA3PagedWrapper in attn/fa3_paged.py, with the probe renamed fa3_paged_unavailable_reason. I'd keep hide_last off it entirely: it is kv_extent = span - hide_last with causal=False, AttentionStep already enforces that rule, and #1324's lagged passes need kv_extent anyway. So the wrapper takes span, kv_extent and causal, and the manager translates the step. The verify_block marker goes too. The wrapper sets plans_in_graph and sizes its page table to the pool's block-table width, with the same formula as KVManager.bind_resident, so it uses the existing in-graph gather branch with main's signature. Decode on it stays a follow-up with an A/B. For the probes, I'd move fa3-fwd's out of dense.py in a small separate PR off main, since it touches main's backend-selection tests and the AITER fallback, and keep this stack to the paged probe. Does that split work for you?

Comment thread mstar/engine/resources/base.py Outdated
stream, which a take-back may give back at most."""
return [span] * len(rids)

def verify_hardware_refusal(self) -> str | None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be generalized to not be spec dec-specific, i.e. one load-time hook that answers for both the resource's default kernels and its verify kernel? Two things would need to change for that:

  1. It does three jobs right now: it picks the kernel (FlashInferManager sets _verify_fa3), raises for a pinned verify_backend: fa3 that can't run, and returns a soft blocker that drops the node to synchronous mode. A general hook needs to return the severity (fatal vs. degrades a feature) instead of encoding it as raise-vs-return, since a default-backend problem is fatal (today check_flashinfer_head_dim raises in __init__).
  2. The kernel choice would move into bind_verify_walk, which runs after _agreed_mode, so that the query has no side effect.

This ties into my comment on spec.py: the load-time hook could return the same capabilities type as the config-time one, narrowed by what this worker's hardware can do.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both points. The load-time hook becomes Resource.capabilities(). It returns this worker's refusals in the same ResourceCapabilities type as the config hook, the engine narrows the config answer with it, and it has no side effects. Severity comes from which field is refused: a pinned verify_backend: fa3 that can't run refuses verify, which is fatal, and auto without FA3 refuses asynchronous verify, which drops auto to synchronous. The kernel choice moves into bind_verify_walk, after _agreed_mode, and bind never refuses. KV's remote-transfer check and GDN's state checks move out of bind into capabilities. For default kernels, the type can take another field later. But answering it means asking the hook for every node at load, not only verifying ones, which changes every model's startup. I'd do that, with check_flashinfer_head_dim, in a follow-up off main. OK with you?

Comment thread mstar/engine/resources/spec.py Outdated
Comment on lines +146 to +159
def verify_refusal(self, draft: "DraftConfig") -> str | None:
"""Why a node that verifies drafts (speculative decoding) can't use
this resource, or None. A verify step writes ``k + 1`` tokens a row
and keeps ``accepted + 1``: a resource that keeps per-token state
corrects it from the result (`Resource.take_back`), and one that
keeps none says so. Refused by default, so a resource nobody checked
is never assumed safe. From config alone, before workers spawn."""
return f"{type(self).__name__} declares no speculative decoding support"

def async_verify_refusal(self, draft: "DraftConfig") -> str | None:
"""Why this resource can't run asynchronous verify steps, planned
from upper-bound lengths while the step before runs and corrected on
the device, or None. Asked only of a resource that verifies at all."""
return f"{type(self).__name__} corrects nothing on the device"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design: thoughts on folding these two into one resource_capabilities(ctx: ResourceCapabilitiesCtx) -> ResourceCapabilities, the way get_chunking_policy replaced several hooks for chunked prefill? None of the overrides reads draft today, and a single hook would show a resource author every option in one place; the next capability (e.g. #1322's recurrent undo) adds a field rather than another method. Something like

@dataclass(frozen=True)
class ResourceCapabilities:
    verify_refusal: str | None          # None: supports verify
    async_verify_refusal: str | None
    ...

with refusal strings only (a supports_* bool beside each could disagree with it), and the base implementation still refusing by default with the class name in the message. The prefix-caching properties (keeps_prefix_cache, restores_prefix, writes_prefix_snapshots) could move in later in a separate PR, with world_size and checkpoint_named in the ctx.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. One resource_capabilities(ctx) -> ResourceCapabilities, refusal strings only, refused by default with the class name, and draft leaves the signature. The ctx holds the node and that node's specs. Overrides replace only the fields they checked, and the fields have no defaults, so a field added later stays refused until someone checks it. Validation and the engine call one helper, so they can't disagree. The hook must be a pure function of the specs, because every rank raises on it before any collective. Env knobs and hardware stay in the agreed mode reduction. #1322's recurrent undo adds no field: its refusals are overrides of this hook, and its post-bind verify_problem becomes a config-time startup check. I'll leave the prefix-caching properties, and main's supports_drop and resident_refusal, for a later PR, as you suggest.

Comment thread mstar/utils/ipc_format.py Outdated
Comment on lines +198 to +202
# parallel lists: rids[i] was offloaded if is_offload[i], else reloaded; or, if
# settled_only[i], only had its verify results applied by an offload that moved nothing
rids: list[str] = field(default_factory=list)
is_offload: list[bool] = field(default_factory=list)
settled_only: list[bool] = field(default_factory=list)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: settled_only makes an entry's kind two bools, where is_offload=True, settled_only=True means nothing, and peek_left() still returns (rid, is_offload), so apply_resident_delta peeks settled_only[0] to tell a settle from a reload. That's handled correctly today, but the next reader of peek_left would take a settle for a reload. Could this be one kind column (offload / reload / settle) that peek_left and pop_left return? It stays a dataclass of lists, so it still encodes typed on the wire.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The settle is gone, so this PR no longer touches ipc_format.py. A row moved to the host keeps its recorded verify results held, and they come due at its first admit back (VerifyLedger.take_due). The engine marks the row returning wherever an offload moves it (Engine._mark_offloaded), and a follower marks it when it replays that OFFLOAD, in every branch of the replay, so every rank applies the results at the same step. The mark needs no move kind of its own, so settled_only, the settle path and its journal entry went, and main's delta, with its kinds column of MoveKind, is unchanged.

Comment thread mstar/engine/resources/kv/manager.py Outdated
Comment on lines +4608 to +4616
def committed_spans(self, rids: Sequence[int], label: str, span: int) -> list[int]:
"""The span the last commit counted for each of ``rids``' ``label``
stream: ``span``, or 0 where the stream was offloaded mid-step and
commit dropped it. Read right after that commit, on its thread."""
with self._lock:
dropped, self._dropped_spans = self._dropped_spans, set()
if not dropped:
return [span] * len(rids)
return [0 if (rid, label) in dropped else span for rid in rids]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

committed_spans reads what the previous commit stashed in _dropped_spans, so it's only right when it's called right after that commit, on the same thread. That stash is also why _dropped_spans needs its own exception in rid_table.py. Could commit return (or record on the step) the rows it dropped, so NodeDraft.committed gets them directly and there's nothing to reset?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll record it on the step. KVStep gets a manager-set dropped field next to commits, and committed_spans reads only the step it is given. _dropped_spans, its reset and the rid_table exception all go. I'd rather not change Resource.commit's return across all its overrides for one consumer.

return None


class TapResource(Resource):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing here is specific to drafting: it's "the residual stream at these layers, captured per slot", and run_layers(taps=) is general too. Qwen3-Omni hand-rolls the same thing twice: the thinker keeps layer_0_embed / layer_n_hidden (accept_hidden_layer) for the talker, and the vision tower keeps its deepstack taps. Could this live outside draft/ (e.g. resources/taps/) as a general resource? The one thing Qwen3-Omni would need on top is a tap that leaves the step as an output, since the thinker's is streamed to the talker, where this one is only read inside the same forward. That part can be a follow-up, but the placement and naming are worth settling now.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on placement. It moves to resources/taps/, with docstrings written for any reader of the residual stream (a drafter is named only as an example), and run_layers types its taps argument. The drafter names its tap resource by key in DrafterNeeds, which becomes the one place the key lives. For now I'm keeping the rule that a verifying node has exactly one TapSpec. run_layers writes one tap resource, and a second one would be read unwritten under capture without an error. I'd leave Qwen3-Omni's vision deepstack taps as a plain return, since they feed mergers inside the tower's own forward. The thinker's taps fit once the tap-as-output follow-up exists.

Comment thread mstar/model/submodule_base.py Outdated
Comment on lines +948 to +949
# The node's `Drafter`, on a node that verifies drafts (`Model.verify_nodes`).
self.drafter: Any = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The engine wires the drafter in through loose attributes on the submodule: it reads drafter (typed Any), sets drafter_share, which engine_step reads back with getattr, and replaces token_stop with a copy that has verify_walks added. Then the base submodule gets k from node_resources[...].verify_draft_tokens (uniform_row_inputs), and _verifying_sampler scans the resources with getattr for a sampler whose key the engine already has (NodeDraft.sampler_key). Could this be one typed hook, e.g. NodeSubmodule.bind_draft(share) with the share carrying the block size and sampler key, and drafter: Drafter | None?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The engine binds one typed DrafterShare once at load through NodeSubmodule.bind_draft(share), the same on every rank: the verify walk, the sampler key, k, the prompt walks, and the steps the drafter's own resources take on prompt and verify steps. A second call raises. It sits on NodeSubmodule rather than ARNodeSubmodule, because engine_step reads the share to add the drafter's steps to every step the node declares (drafter_step), so a target declares none of them. TokenStop.verify_walks goes, the hooks ask is_verify_walk, the verify helpers take the sampler from the share's key, and SamplerResource.verify_buffer goes. drafter is a Drafter | None set in __init__, and NodeSubmodule.__init_subclass__ refuses a class that sets drafter as a class attribute, because a class value hides an nn.Module drafter registered in _modules. The drafter stays a constructor argument, since the model loads its weights, and drafters read k from DraftInputs.num_draft_tokens, so k has one source.

Comment on lines +107 to +109
# The node verifies drafts (speculative decoding): set by the model when the
# deployment's ``model_kwargs.draft`` is on, so ingest refuses what verify can't honour.
verifies: bool = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding to Atindra's slack question: verifies is a fourth field a model sets by hand when model_kwargs.draft is on, which startup then checks (shape_problems). Like draft_slack and the pool's max_seq_len, it follows from the draft config alone, so drafter_resources (or whatever ends up setting the slack) could set it too, and a model would only declare its drafter.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. drafter_resources will set verifies along with draft_slack and the pool's max_seq_len, and turn off the repetition penalty on that sampler. A model then declares only its drafter.

nick-rui and others added 26 commits November 16, 2026 18:28
…ged paged wrapper

FA3's paged verify wrapper takes rows of k + 1 queries, so a mixed step (its captures' too) planned its verify rows with its prompt rows on FlashInfer's prefill wrapper, a host plan per step. Where FA3 runs the verify walk, a mixed step's attention now plans on FA3RaggedWrapper, which takes rows of any span from the KV plan's arrays as FlashInfer's wrapper does. Each verify row's last keys are hidden in those arrays (hide_tail_views) as before, so FA3 and FlashInfer read the same lengths. It is bottom-right causal for the target, or non-causal for a drafter's binding. Under a captured graph its page table, lengths and query offsets are the bucket's static buffers, filled by the host plan; the kernel's query bound is the bucket's tokens, and the queries past the step's rows (a replay's padding) come out zero. It pins FA3's split count and head packing, which its heuristics would take from max_seqlen_q, so an eager step and a captured replay reduce each row alike. FlashInfer plans the step where FA3 doesn't run. Mixed steps stay synchronous.

Tests: on the CPU, the host plan's lengths, query offsets and page tables for verify, prompt and padding rows, hidden and not; a captured plan fills the bucket's buffers and refuses other rows or more tokens; eager and captured runs pass the pinned schedule; the manager plans a mixed step on the ragged wrapper with FA3 bound, causal and DSpark's mask, and on FlashInfer without it. On CUDA, the ragged wrapper against FlashInfer's prefill wrapper and a dense reference at the toy and Qwen3-8B shapes, and a captured replay at a padded bucket is the eager plan's bit for bit, with zeros past its rows, for two plans of one bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s prompt rows on the host plan

An asynchronous mixed step's verify rows hold lengths only the device knows until their results are recorded, as a verify step's do, while its prompt rows' are the host's. Both now go into one captured step, with every branch below reachable only for a walk in the planner's mixed_walks, which the engine sets on an asynchronous verify node with a mixed walk and which is empty elsewhere:

- Planner (resident.py): a mixed step's leading verify rows hold slots, and the engine orders them ahead of the prompt rows; a step where they aren't is refused. The prompt rows sit on the trash row as padding does, so no owner stages, rewrites or advances them. The bucket's buffers say the step is mixed (ResidentBuffers.mixed), and gather_column takes skip_trash.
- KV: the plan stages the whole step's host addressing, and the graph's write-slot gather skips the trash rows (SKIP_TRASH), so the verify rows take the tables' slots and the prompt rows keep the host's.
- Positions: the same, the host staging every row and the resident gather skipping trash rows. LongRoPE's table ids ride that gather.
- Attention: FA3's ragged wrapper of a captured mixed step reads its verify rows' block tables and device lengths plus k + 1 - hide_last off the tables, in the graph (FA3RaggedWrapper.gather_resident). The host plan's prompt rows and query offsets stand. The FlashInfer manager keeps each bucket's ragged wrapper and the verify block.
- Sampler: a mixed step's rows stage on the host, as for a step the planner doesn't take. Every row's settings, offset and loop-back slot are host-known, and its resident gather records nothing for such a bucket.
- Draft: draft_mixed_step writes the planner's advance buffer (accepted + 1) as draft_verify_step does, so the graph advances each verify row's length cell by what it keeps.
- Mode: resolve_mode no longer holds a mixed node to synchronous verify. The engine refuses an asynchronous mixed step only where it found no resident capture, as it does an asynchronous verify step.

Tests:
- A harness on the real KV and position resources and the real planner, with the KV debug checks on (test_spec_mixed_resident.py). Through random acceptance and asynchronous recording, the verify rows write and read at their true lengths and the prompt rows at their own tokens from zero, and once everything is applied the host holds every length. A row with an unrecorded result keeps its slot and device length through a mixed step. FA3's ragged wrapper reads the verify rows' pages and lengths off the tables, for the target's mask and DSpark's, while the prompt and padding rows keep the host plan's.
- A mixed node resolves to asynchronous as a verify node does, and to synchronous without async scheduling.
- A mixed step with a resident advance buffer fills it with each row's accepted + 1 (dropping only the write fails it).
- Main's resident, KV-offload and position modules show no new failure, with the KV debug checks on and off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lending, and its verify take-back is apart

Main's page lending has a guaranteed request lend what it hasn't written of its reservation, and take it back by moving a borrower when its grant runs short. On a verifying node that reservation includes the verify slack. Where the pool moves nothing (no host pool, since a verifying node has no rebuild policy) nothing is lent; with a host pool the slack is lent like the rest. The verify take-back of rejected tokens lowers lengths only, while lending counts pages held, so the two don't interact. The admission design and adding_models say so, and tell the two take-backs apart.

Tests: the guaranteed-request test sets the pool to move nothing, as a pool with no host pool does. A twin on a pool with a host pool has a borrower take what the guaranteed verifying request lends. When its grant runs short, the refused admit has already lowered its length by the rejected tokens, the grant moves the borrower to the host, and the retry applies nothing again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n, and the tests stub what main now reads

Under MSTAR_KV_DEBUG_ASSERTS, the preemption victim check asked every node's engine for its verify state, on every pool. On a pool no node verifies on there is nothing to check, and main's preemption tests, whose stub engines answer no verify queries, failed there. The check now runs only where the pool reserves a verify slack (KVManager.verify_slack, set by bind_verify_walk).

Tests: the guaranteed-request tests bind their pool's verify walk once the plain requests beside them are in, so main's debug bound on committed slots counts the scratch as it does on a verifying pool. The loop-back threading test gives its bare Speculator the runtime that main's drop warning now logs request ids through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model on the state contract declares speculative decoding with a Draft in declare_state, a contract value of plain fields: the node, the paged KV array the drafter's layers join, the sampler that verifies, the verify walk and mixed walk, the drafter by its draft-block key, checkpoint and resolved checkpoint directory, num_draft_tokens and mode, and the target's depth, residual width and weight dtype (HF's name). It hashes and digests as every declaration does, and contract.py imports nothing new from the resources.

- Adapter: draft_config checks the settings as a serving config's draft block (DraftConfig.parse). node_resources passes the declared specs through drafter_resources, the same rewrite a model on resources runs on its own, with the drafter's declaration from Model.drafter(draft): the model builds it from the Draft with its family's code (its config's refusals, its tokenizer, its taps), which the engine doesn't import. verify_node builds the VerifyNode, the target's ModelGeometry from the Draft and the array's entry and the sampler's vocabulary. Model.declared_verify_nodes builds one per Draft, with the node's token_stop and device_loopback from Model.draft_node_fields; a model on the contract returns them from its verify_nodes when its draft block is given, so the default stays the empty list and a model that doesn't draft never reads its declarations for it. validate_deployment checks speculative decoding only once the declarations built their specs, so a declaration that fails is reported once.
- DrafterDecl gains prefill_steps and verify_steps, empty unless a drafter's own resources step.
- Analysis: a Draft must name a paged array and a sampler of its node and a walk it steps on. node_handles builds no handle for one.
- engine_step expands a node's declared state before it adds the drafter's share, so the share sees the resource steps a node on the contract declares as state; on resources there is no state to expand, so the order changes nothing there. drafter_step keeps the expanded step's handles, which `replace` leaves behind (an init=False field).
- drafter_step: a step with verify rows runs every attention on its verify kernel, and a verify step's sampler takes one row a request, which a node on the contract has no step field to declare. A node on resources declares the same, so for it nothing changes.

The surface snapshot gains Draft, and the contract's hash and equality test covers it. Tests (test_state_draft.py, a declared toy node): its specs are drafter_resources over the arrays' and differ from a plain node's; its VerifyNode equals one written by hand, the target's depth the declared one; a Draft hashes and digests as a value, its settings are checked as a draft block, and a malformed one is refused; the analysis names a wrong array, sampler or walk; drafter_step marks verify and mixed steps, leaves prompt steps alone, and keeps what a node on resources declares. The default verify_nodes reads no declaration. Mutations caught: the verify marking dropped, the spec rewrite dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s length

On pages of more than 32 tokens, FA3's paged kernel (flash_attn_3 3.0.1) reads
a row's last page past its length once the step's longest row has about 64
queries, and masks those scores to zero probability. A stale NaN or Inf an
earlier request's broken forward left in a reused page then still reaches the
row, as 0 times NaN. Decode and verify rows never take that path; a mixed
step's prompt row can. The attention manager warns once when it first builds a
mixed step's FA3 wrapper over such pages; pages of 32 tokens or fewer avoid
it. A CUDA test pins the observed behaviour, so a wheel that changes it shows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ModelGeometry.plain_rope: a drafter that rotates with plain RoPE through the
node's positions (EAGLE-3's) is refused at startup over a position resource
that scales (rope_scale other than 1, Llama 3.1 factors, LongRoPE), which
would rotate the target one way and the drafter another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verify rows (spans 4 and 8, a hidden last key) on FA3PagedWrapper and mixed
rows (prompt rows whose new tokens are no multiple of the page) on
FA3RaggedWrapper, eager and replayed with padding, over a pool whose sink page
holds NaN, as do the slots past each row's length at 16 and 32: real rows are
finite and match a dense fp32 reference, and a replay's padding comes out zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FA3RaggedWrapper.run masked a replay's padding queries with a token-id
compare and an out-of-place masked_fill in every layer, two kernels and a
copy of the attention output each time. It now calls main's
zero_unplanned_rows, which zeroes the rows from the planned total in place
on the device: one kernel, the same rows, and the real rows untouched. In a
Qwen3-8B mixed graph that is 41 kernels fewer and about 0.75 ms less.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at holds it

An asynchronous verifying node runs a mixed prompt-plus-verify step only on
a capture whose verify rows read the resident tables, and a step that finds
none raises and fails every row in it. The mixed walk was capped at its
ladder's widest configured bucket and its token budget at the model's, so a
capture that left the widest mixed buckets out (memory short) let the
scheduler build steps no kept bucket held.

At warmup the engine now picks the widest kept, non-lean mixed bucket that
holds the worst step under its own caps (verify rows of a block each, and a
prompt row at the budget's min_chunk_tokens floor), caps the walk's rows
there (get_max_batch_size) and its budget, a serving override included, at
its token capacity (ChunkingManagement.cap_budget). The runtime raise stays
for overruns the bound doesn't cover. New startup refusals on an
asynchronous mixed node: no token budget on the mixed walk (or its prompts'
chunking off), and no kept bucket that holds the worst step.

The instance's leader warns when the caps are lowered (rows below the
ladder's top, or tokens below the budget), naming both and any buckets
capture left out.

Every new branch runs only for an asynchronous verifying node with a mixed
walk; elsewhere the caps are as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l too

drafter_step forced verify=True on the node's own attentions on a mixed
walk, but added the drafter's own steps (prompt and verify alike) after
that, untouched. A drafter attention declared for prompt steps alone would
then plan a mixed step's verify rows on FlashInfer's prefill wrapper from
the host, over their reserved lengths in asynchronous mode, and nothing
caught it. Every shipped drafter declares its attention verify=True in its
verify steps, so this was latent. The force now covers the drafter's own
attentions on the mixed walk, at the declaration's source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…abels is refused

A mixed step plans each KV label apart, and FA3's ragged wrapper that a
resident gather fills off the tables holds one label's rows while the
gather covers every row of the bucket. A mixed step over two labels would
fail on shapes, or under combined labels gather the rows in the wrong
order. Nothing reaches it today (every shipped drafter's node writes one
label), so it is refused rather than supported:

- at startup, before workers spawn and again at load (mixed_label_problem):
  a verifying node with a mixed walk whose KV pool the model writes under
  more than one label (Model.kv_labels);
- where its attention plans, as a verify step's already is: a mixed step
  that plans two labels on FA3 raises;
- in a mixed capture's resident gather, a bucket that registered no ragged
  wrapper raises instead of leaving its verify rows on the host's reserved
  lengths, which drafter_step's verify force now rules out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t doesn't fit

A serving min_chunk_tokens past what the widest kept mixed bucket leaves
beside its verify rows refused the node with a message about memory and a
knob that doesn't exist. The refusal now names the floor and the most it can
be, names the buckets capture left out only when there are some, and drops
the knob. The envelope's docstring says it reads (bs, tokens) alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng their keys

BucketKey has no order, so max() over (bs, tokens, key) raised TypeError when
two kept mixed buckets tied on size and differed in their capture key. Rank
by (bs, tokens) alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ompt row draws it

A seeded reply on a verifying node depended on whether its prompt ran a prompt
step of its own (main's sampler, FlashInfer's batch-position draw) or rode a
mixed step (a refused verify row, drawn from the request's own stream at the
bonus draw's number). sample() on a verifying sampler now draws through
first_draw, which reads the row's filtered distribution and the same number a
refused row reads, and lands the token and offsets as verify() does, so every
token of a drafting node's reply comes from the per-row stream. verify() is
split into _verify_params and _land for that; fused_draw_args is None on a
verifying node, so no forward-side draw bypasses it. A node that doesn't draft
keeps main's sample() unchanged (verify_draft_tokens is 0 there).

The toy target samples its prompt token through the node's sampler, which the
CPU can now do; the seeded-reply test runs both paths at T=20 over 8 seeds and
two drafters, and a sampler test checks sample() against a fully refused
verify() step after step. serving.rst (and its mintlify copy) says a seed fixes
a drafting node's reply whatever the batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
serving.rst's draft block (and its mintlify copy) now says an asynchronous
mixed walk is capped at the widest captured bucket that holds its worst step,
that the cap covers prompt-only steps under the combined walk and a node-level
max_batch_tokens override, and which mixed walks startup refuses: no token
budget, no captured bucket that holds one, and a pool written under two KV
labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et_up_draft

ruff --preview's E302 and E301, which the stack added and main's lint
config doesn't select.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
order is the batch's FilterOrder or None and slots the eager step's slot
tensor or None; both were typed object. Two blank lines on either side of the
class, as ruff --preview's E302 and E303 ask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r its prompt rode a mixed step

The seeded test ran with top_k 0 and top_p 1 only, so a prompt draw that
dropped the request's filters passed it. It is parametrized over a filtered
case (top_k 20, top_p 0.7), and the toy driver takes both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an aligned floor only where one fits

The refusal's other branches: a kept bucket short of the floor beside a
mixed bucket capture left out, on a node with no chunk alignment and on one
aligned to 8 tokens. The left-out mixed bucket is named and another walk's
isn't; the min_chunk_tokens advice is a multiple of the alignment and absent
where a kept bucket leaves less than one beside its verify rows; and the
value it names starts the node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…f, and name the floor the message gives

serving.rst's draft block (and its mintlify copy) left out the refusal for a
mixed walk whose member prompt walk has its chunking turned off, and said the
message names the configured min_chunk_tokens; it gives the floor it fit,
capped at the budget and rounded to the chunk alignment, and the remedies
that apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et that would hold the step

The refusal advised freeing memory for the mixed buckets capture left out
whenever there were any, though none of them might hold the worst step
either: with that bucket captured the node is refused the same. The advice
now names only the left-out buckets that pass the fit the kept ones get,
bounded by their key's token count, since a left-out bucket may not be
registered with the runner. The docstring lists the chunking-off refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…used, and the refusal prints the fitted floor

Two refusal paths no test reached. A serving override of null
max_batch_tokens on a prompt walk inside a budgeted mixed walk is refused
(and with the override at 1024 the node starts), so dropping the
chunking-off clause fails. And the floor the no-bucket refusal prints is
min_chunk_tokens as the fit used it, not as configured: 130 at an alignment
of 8 prints 136, and 2000 over a 1024-token budget prints 1024.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he same whether its prompt rode a mixed step

The filtered case covered top-k and top-p only, so a prompt draw that
dropped the request's min_p or its filter order passed. The seeded test
adds a min_p 0.5 case and a top_p_first case, and the toy's sampler takes
both when a test asks. top_p_first is CUDA-only, so its case lifts the
CPU refusal and stands in for the CUDA filter with torch.

The seeded case compares the two paths with each other, so a verifying
node that dropped the order on both passes it. A second test checks, on
each path, that some seed's reply with top-p first differs from the same
request's with top-k first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd two, not four, before the timing test

ruff's preview E302 and E303 in the stack's own test module; default ruff
doesn't select them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd memory only for a bucket that would hold the step

"The floor it fit" read as if the floor fit; it is the one no kept bucket
holds, rounded up to the chunk alignment. The free-memory remedy names only
a left-out bucket that would hold the step, as the engine now offers it. And
the KV-label refusal says it applies in either mode, so it doesn't read as
one of the asynchronous refusals before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
staging - docs/mintlify — 4d8b6e83 Deployed Nov 18, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants