Repository navigation
feat(engine): add speculative decoding with verify steps and KV take-back (PR 3 of 7 in the stack) - #1320
feat(engine): add speculative decoding with verify steps and KV take-back (PR 3 of 7 in the stack)#1320nick-rui wants to merge 65 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
merceod
left a comment
There was a problem hiding this comment.
Static pass on 4199f31, no GPU yet. I read the engine, resource, worker and sampler changes with their callers, against the RFC, plus the tests.
The plain path looks unchanged to me. Every new branch a non-verifying node goes through tests an empty map or set, the three resident kernels reduce to main's arithmetic at span 1, and the sampler and offload helpers are straight extractions. On the verify path I couldn't break the ledger's due rule on leader or follower, the slot pinning for unrecorded rows, the settle before an offload, or the accept and residual math in _chain_verify. One gap is inline.
Two questions:
- The slack. The docs say a model sets
DraftConfig.kv_slackby hand in three places (KVSpec.draft_slack, the pool'smax_seq_len, the layout'sscratch), and startup checks each one.drafter_resourcesalready edits the pool for the join. Could it setdraft_slackandmax_seq_lenthere too, so a model only declares its drafter? I get thatscratchcomes fromprocess_prompt, outside the engine. - Verify-step host cost.
NodeDraft.cut,cut_verify_step(a slice per row, thenrow_views = None, which sends the step's outputs per rid),refuse_stepand the ledger calls are per-row Python on every verify step. Is there aworker_phasesrun on a verifying node at c32/c64 yet, or is that waiting on #1325 as the RFC says?
sd-prQ was force-pushed onto main after #1318 merged, so this needs a restack, and GitHub shows it conflicting. Against current main the only conflicts are the import line and load_model's new kwargs from #1184 in engine.py and engine_manager.py, both keep-both. CI hasn't run on this head because the base isn't main, and none of the CUDA rows in the description ran on it. Once it's restacked we'll run the CUDA modules on H200 with and without flash_attn_3, the plain-path A/B on Qwen3.5-0.8B, and plain TP2 under KV pressure.
| # its rows would commit k + 1 tokens no ledger entry ever takes back | ||
| raise RuntimeError(f"{batch.node_name}: verify walk {draft.walk!r} rows inside a combined step") | ||
| if draft is not None and ctx.graph_walk == draft.walk: | ||
| if draft.asynchronous and ctx.resident is None: |
There was a problem hiding this comment.
In asynchronous mode this only checks that a resident capture exists. If an attention on the verify walk is declared without verify=True, FlashInferManager.resident_gather marks the step host_planned, the bucket never joins host_plan_free, and every replay plans that attention (and the KV host plan) from stored_len, the reserved length, while the write slots come from the device. The row then attends up to k stale slots past its true length and nothing raises. #1321 sets the flag, so nothing breaks today, but the next model that leaves it off gets wrong tokens quietly. Raising here when ctx.slot_lease.bucket not in ctx.resident.host_plan_free, or having refuse_step require verify=True on every AttentionStep of the verify walk, would make that an error on the first step.
There was a problem hiding this comment.
Agreed this is a real gap, and it is wider than the flag: any resource that host-plans a captured verify bucket gives the same stale read. Since that is fixed at capture, I'll check it once at startup instead of on the first step. After capture, an asynchronous verifying node is refused if any resource host-planned a captured bucket of its verify walk, and the error names the resource keys. Every input is agreed across the lockstep instance, so all ranks raise alike, with no extra collective. I'm not adding a verify=True requirement in refuse_step, because it would miss a resource that host-plans without the flag. A toy test with a FlashInfer attention declared without verify=True shows the refusal.
NSagan271
left a comment
There was a problem hiding this comment.
Overall shape of the implementation looks correct; I left some comments on design, mainly generalizing some of the added components beyond spec dec
| return None | ||
|
|
||
|
|
||
| class FA3VerifyWrapper: |
There was a problem hiding this comment.
Is this wrapper necessarily specific to spec dec verification? Apart from hide_last and the verify_block marker that FlashInferManager.resident_gather checks with getattr, it reads as a general paged FA3 wrapper over rows of one fixed span, with lengths staged from the host or gathered off the resident tables. At span=1 it's already a decode wrapper. Could it get a generic name (e.g. FA3PagedWrapper, with fa3_verify_unavailable_reason renamed to match), with hide_last as a verify-only option on it? Decode on it can be a follow-up. Ragged prefill would need more, since the graph-mode qo_indptr, max_seqlen_q and the gather_resident length all assume a fixed span.
Related: attn/dense.py already has _fa3_unavailable_reason for fa3-fwd. Now that there are two FA3 wheels, it'd be nice to have both probes in one module (say attn/fa3.py).
There was a problem hiding this comment.
Yes. It becomes FA3PagedWrapper in attn/fa3_paged.py, with the probe renamed fa3_paged_unavailable_reason. I'd keep hide_last off it entirely: it is kv_extent = span - hide_last with causal=False, AttentionStep already enforces that rule, and #1324's lagged passes need kv_extent anyway. So the wrapper takes span, kv_extent and causal, and the manager translates the step. The verify_block marker goes too. The wrapper sets plans_in_graph and sizes its page table to the pool's block-table width, with the same formula as KVManager.bind_resident, so it uses the existing in-graph gather branch with main's signature. Decode on it stays a follow-up with an A/B. For the probes, I'd move fa3-fwd's out of dense.py in a small separate PR off main, since it touches main's backend-selection tests and the AITER fallback, and keep this stack to the paged probe. Does that split work for you?
| stream, which a take-back may give back at most.""" | ||
| return [span] * len(rids) | ||
|
|
||
| def verify_hardware_refusal(self) -> str | None: |
There was a problem hiding this comment.
Can this be generalized to not be spec dec-specific, i.e. one load-time hook that answers for both the resource's default kernels and its verify kernel? Two things would need to change for that:
- It does three jobs right now: it picks the kernel (
FlashInferManagersets_verify_fa3), raises for a pinnedverify_backend: fa3that can't run, and returns a soft blocker that drops the node to synchronous mode. A general hook needs to return the severity (fatal vs. degrades a feature) instead of encoding it as raise-vs-return, since a default-backend problem is fatal (todaycheck_flashinfer_head_dimraises in__init__). - The kernel choice would move into
bind_verify_walk, which runs after_agreed_mode, so that the query has no side effect.
This ties into my comment on spec.py: the load-time hook could return the same capabilities type as the config-time one, narrowed by what this worker's hardware can do.
There was a problem hiding this comment.
Agreed on both points. The load-time hook becomes Resource.capabilities(). It returns this worker's refusals in the same ResourceCapabilities type as the config hook, the engine narrows the config answer with it, and it has no side effects. Severity comes from which field is refused: a pinned verify_backend: fa3 that can't run refuses verify, which is fatal, and auto without FA3 refuses asynchronous verify, which drops auto to synchronous. The kernel choice moves into bind_verify_walk, after _agreed_mode, and bind never refuses. KV's remote-transfer check and GDN's state checks move out of bind into capabilities. For default kernels, the type can take another field later. But answering it means asking the hook for every node at load, not only verifying ones, which changes every model's startup. I'd do that, with check_flashinfer_head_dim, in a follow-up off main. OK with you?
| def verify_refusal(self, draft: "DraftConfig") -> str | None: | ||
| """Why a node that verifies drafts (speculative decoding) can't use | ||
| this resource, or None. A verify step writes ``k + 1`` tokens a row | ||
| and keeps ``accepted + 1``: a resource that keeps per-token state | ||
| corrects it from the result (`Resource.take_back`), and one that | ||
| keeps none says so. Refused by default, so a resource nobody checked | ||
| is never assumed safe. From config alone, before workers spawn.""" | ||
| return f"{type(self).__name__} declares no speculative decoding support" | ||
|
|
||
| def async_verify_refusal(self, draft: "DraftConfig") -> str | None: | ||
| """Why this resource can't run asynchronous verify steps, planned | ||
| from upper-bound lengths while the step before runs and corrected on | ||
| the device, or None. Asked only of a resource that verifies at all.""" | ||
| return f"{type(self).__name__} corrects nothing on the device" |
There was a problem hiding this comment.
Design: thoughts on folding these two into one resource_capabilities(ctx: ResourceCapabilitiesCtx) -> ResourceCapabilities, the way get_chunking_policy replaced several hooks for chunked prefill? None of the overrides reads draft today, and a single hook would show a resource author every option in one place; the next capability (e.g. #1322's recurrent undo) adds a field rather than another method. Something like
@dataclass(frozen=True)
class ResourceCapabilities:
verify_refusal: str | None # None: supports verify
async_verify_refusal: str | None
...with refusal strings only (a supports_* bool beside each could disagree with it), and the base implementation still refusing by default with the class name in the message. The prefix-caching properties (keeps_prefix_cache, restores_prefix, writes_prefix_snapshots) could move in later in a separate PR, with world_size and checkpoint_named in the ctx.
There was a problem hiding this comment.
Yes. One resource_capabilities(ctx) -> ResourceCapabilities, refusal strings only, refused by default with the class name, and draft leaves the signature. The ctx holds the node and that node's specs. Overrides replace only the fields they checked, and the fields have no defaults, so a field added later stays refused until someone checks it. Validation and the engine call one helper, so they can't disagree. The hook must be a pure function of the specs, because every rank raises on it before any collective. Env knobs and hardware stay in the agreed mode reduction. #1322's recurrent undo adds no field: its refusals are overrides of this hook, and its post-bind verify_problem becomes a config-time startup check. I'll leave the prefix-caching properties, and main's supports_drop and resident_refusal, for a later PR, as you suggest.
| # parallel lists: rids[i] was offloaded if is_offload[i], else reloaded; or, if | ||
| # settled_only[i], only had its verify results applied by an offload that moved nothing | ||
| rids: list[str] = field(default_factory=list) | ||
| is_offload: list[bool] = field(default_factory=list) | ||
| settled_only: list[bool] = field(default_factory=list) |
There was a problem hiding this comment.
nit: settled_only makes an entry's kind two bools, where is_offload=True, settled_only=True means nothing, and peek_left() still returns (rid, is_offload), so apply_resident_delta peeks settled_only[0] to tell a settle from a reload. That's handled correctly today, but the next reader of peek_left would take a settle for a reload. Could this be one kind column (offload / reload / settle) that peek_left and pop_left return? It stays a dataclass of lists, so it still encodes typed on the wire.
There was a problem hiding this comment.
The settle is gone, so this PR no longer touches ipc_format.py. A row moved to the host keeps its recorded verify results held, and they come due at its first admit back (VerifyLedger.take_due). The engine marks the row returning wherever an offload moves it (Engine._mark_offloaded), and a follower marks it when it replays that OFFLOAD, in every branch of the replay, so every rank applies the results at the same step. The mark needs no move kind of its own, so settled_only, the settle path and its journal entry went, and main's delta, with its kinds column of MoveKind, is unchanged.
| def committed_spans(self, rids: Sequence[int], label: str, span: int) -> list[int]: | ||
| """The span the last commit counted for each of ``rids``' ``label`` | ||
| stream: ``span``, or 0 where the stream was offloaded mid-step and | ||
| commit dropped it. Read right after that commit, on its thread.""" | ||
| with self._lock: | ||
| dropped, self._dropped_spans = self._dropped_spans, set() | ||
| if not dropped: | ||
| return [span] * len(rids) | ||
| return [0 if (rid, label) in dropped else span for rid in rids] |
There was a problem hiding this comment.
committed_spans reads what the previous commit stashed in _dropped_spans, so it's only right when it's called right after that commit, on the same thread. That stash is also why _dropped_spans needs its own exception in rid_table.py. Could commit return (or record on the step) the rows it dropped, so NodeDraft.committed gets them directly and there's nothing to reset?
There was a problem hiding this comment.
I'll record it on the step. KVStep gets a manager-set dropped field next to commits, and committed_spans reads only the step it is given. _dropped_spans, its reset and the rid_table exception all go. I'd rather not change Resource.commit's return across all its overrides for one consumer.
| return None | ||
|
|
||
|
|
||
| class TapResource(Resource): |
There was a problem hiding this comment.
Nothing here is specific to drafting: it's "the residual stream at these layers, captured per slot", and run_layers(taps=) is general too. Qwen3-Omni hand-rolls the same thing twice: the thinker keeps layer_0_embed / layer_n_hidden (accept_hidden_layer) for the talker, and the vision tower keeps its deepstack taps. Could this live outside draft/ (e.g. resources/taps/) as a general resource? The one thing Qwen3-Omni would need on top is a tap that leaves the step as an output, since the thinker's is streamed to the talker, where this one is only read inside the same forward. That part can be a follow-up, but the placement and naming are worth settling now.
There was a problem hiding this comment.
Agreed on placement. It moves to resources/taps/, with docstrings written for any reader of the residual stream (a drafter is named only as an example), and run_layers types its taps argument. The drafter names its tap resource by key in DrafterNeeds, which becomes the one place the key lives. For now I'm keeping the rule that a verifying node has exactly one TapSpec. run_layers writes one tap resource, and a second one would be read unwritten under capture without an error. I'd leave Qwen3-Omni's vision deepstack taps as a plain return, since they feed mergers inside the tower's own forward. The thinker's taps fit once the tap-as-output follow-up exists.
| # The node's `Drafter`, on a node that verifies drafts (`Model.verify_nodes`). | ||
| self.drafter: Any = None |
There was a problem hiding this comment.
The engine wires the drafter in through loose attributes on the submodule: it reads drafter (typed Any), sets drafter_share, which engine_step reads back with getattr, and replaces token_stop with a copy that has verify_walks added. Then the base submodule gets k from node_resources[...].verify_draft_tokens (uniform_row_inputs), and _verifying_sampler scans the resources with getattr for a sampler whose key the engine already has (NodeDraft.sampler_key). Could this be one typed hook, e.g. NodeSubmodule.bind_draft(share) with the share carrying the block size and sampler key, and drafter: Drafter | None?
There was a problem hiding this comment.
Yes. The engine binds one typed DrafterShare once at load through NodeSubmodule.bind_draft(share), the same on every rank: the verify walk, the sampler key, k, the prompt walks, and the steps the drafter's own resources take on prompt and verify steps. A second call raises. It sits on NodeSubmodule rather than ARNodeSubmodule, because engine_step reads the share to add the drafter's steps to every step the node declares (drafter_step), so a target declares none of them. TokenStop.verify_walks goes, the hooks ask is_verify_walk, the verify helpers take the sampler from the share's key, and SamplerResource.verify_buffer goes. drafter is a Drafter | None set in __init__, and NodeSubmodule.__init_subclass__ refuses a class that sets drafter as a class attribute, because a class value hides an nn.Module drafter registered in _modules. The drafter stays a constructor argument, since the model loads its weights, and drafters read k from DraftInputs.num_draft_tokens, so k has one source.
| # The node verifies drafts (speculative decoding): set by the model when the | ||
| # deployment's ``model_kwargs.draft`` is on, so ingest refuses what verify can't honour. | ||
| verifies: bool = False |
There was a problem hiding this comment.
Adding to Atindra's slack question: verifies is a fourth field a model sets by hand when model_kwargs.draft is on, which startup then checks (shape_problems). Like draft_slack and the pool's max_seq_len, it follows from the draft config alone, so drafter_resources (or whatever ends up setting the slack) could set it too, and a model would only declare its drafter.
There was a problem hiding this comment.
Yes. drafter_resources will set verifies along with draft_slack and the pool's max_seq_len, and turn off the repetition penalty on that sampler. A model then declares only its drafter.
…ged paged wrapper FA3's paged verify wrapper takes rows of k + 1 queries, so a mixed step (its captures' too) planned its verify rows with its prompt rows on FlashInfer's prefill wrapper, a host plan per step. Where FA3 runs the verify walk, a mixed step's attention now plans on FA3RaggedWrapper, which takes rows of any span from the KV plan's arrays as FlashInfer's wrapper does. Each verify row's last keys are hidden in those arrays (hide_tail_views) as before, so FA3 and FlashInfer read the same lengths. It is bottom-right causal for the target, or non-causal for a drafter's binding. Under a captured graph its page table, lengths and query offsets are the bucket's static buffers, filled by the host plan; the kernel's query bound is the bucket's tokens, and the queries past the step's rows (a replay's padding) come out zero. It pins FA3's split count and head packing, which its heuristics would take from max_seqlen_q, so an eager step and a captured replay reduce each row alike. FlashInfer plans the step where FA3 doesn't run. Mixed steps stay synchronous. Tests: on the CPU, the host plan's lengths, query offsets and page tables for verify, prompt and padding rows, hidden and not; a captured plan fills the bucket's buffers and refuses other rows or more tokens; eager and captured runs pass the pinned schedule; the manager plans a mixed step on the ragged wrapper with FA3 bound, causal and DSpark's mask, and on FlashInfer without it. On CUDA, the ragged wrapper against FlashInfer's prefill wrapper and a dense reference at the toy and Qwen3-8B shapes, and a captured replay at a padded bucket is the eager plan's bit for bit, with zeros past its rows, for two plans of one bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s prompt rows on the host plan An asynchronous mixed step's verify rows hold lengths only the device knows until their results are recorded, as a verify step's do, while its prompt rows' are the host's. Both now go into one captured step, with every branch below reachable only for a walk in the planner's mixed_walks, which the engine sets on an asynchronous verify node with a mixed walk and which is empty elsewhere: - Planner (resident.py): a mixed step's leading verify rows hold slots, and the engine orders them ahead of the prompt rows; a step where they aren't is refused. The prompt rows sit on the trash row as padding does, so no owner stages, rewrites or advances them. The bucket's buffers say the step is mixed (ResidentBuffers.mixed), and gather_column takes skip_trash. - KV: the plan stages the whole step's host addressing, and the graph's write-slot gather skips the trash rows (SKIP_TRASH), so the verify rows take the tables' slots and the prompt rows keep the host's. - Positions: the same, the host staging every row and the resident gather skipping trash rows. LongRoPE's table ids ride that gather. - Attention: FA3's ragged wrapper of a captured mixed step reads its verify rows' block tables and device lengths plus k + 1 - hide_last off the tables, in the graph (FA3RaggedWrapper.gather_resident). The host plan's prompt rows and query offsets stand. The FlashInfer manager keeps each bucket's ragged wrapper and the verify block. - Sampler: a mixed step's rows stage on the host, as for a step the planner doesn't take. Every row's settings, offset and loop-back slot are host-known, and its resident gather records nothing for such a bucket. - Draft: draft_mixed_step writes the planner's advance buffer (accepted + 1) as draft_verify_step does, so the graph advances each verify row's length cell by what it keeps. - Mode: resolve_mode no longer holds a mixed node to synchronous verify. The engine refuses an asynchronous mixed step only where it found no resident capture, as it does an asynchronous verify step. Tests: - A harness on the real KV and position resources and the real planner, with the KV debug checks on (test_spec_mixed_resident.py). Through random acceptance and asynchronous recording, the verify rows write and read at their true lengths and the prompt rows at their own tokens from zero, and once everything is applied the host holds every length. A row with an unrecorded result keeps its slot and device length through a mixed step. FA3's ragged wrapper reads the verify rows' pages and lengths off the tables, for the target's mask and DSpark's, while the prompt and padding rows keep the host plan's. - A mixed node resolves to asynchronous as a verify node does, and to synchronous without async scheduling. - A mixed step with a resident advance buffer fills it with each row's accepted + 1 (dropping only the write fails it). - Main's resident, KV-offload and position modules show no new failure, with the KV debug checks on and off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lending, and its verify take-back is apart Main's page lending has a guaranteed request lend what it hasn't written of its reservation, and take it back by moving a borrower when its grant runs short. On a verifying node that reservation includes the verify slack. Where the pool moves nothing (no host pool, since a verifying node has no rebuild policy) nothing is lent; with a host pool the slack is lent like the rest. The verify take-back of rejected tokens lowers lengths only, while lending counts pages held, so the two don't interact. The admission design and adding_models say so, and tell the two take-backs apart. Tests: the guaranteed-request test sets the pool to move nothing, as a pool with no host pool does. A twin on a pool with a host pool has a borrower take what the guaranteed verifying request lends. When its grant runs short, the refused admit has already lowered its length by the rejected tokens, the grant moves the borrower to the host, and the retry applies nothing again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n, and the tests stub what main now reads Under MSTAR_KV_DEBUG_ASSERTS, the preemption victim check asked every node's engine for its verify state, on every pool. On a pool no node verifies on there is nothing to check, and main's preemption tests, whose stub engines answer no verify queries, failed there. The check now runs only where the pool reserves a verify slack (KVManager.verify_slack, set by bind_verify_walk). Tests: the guaranteed-request tests bind their pool's verify walk once the plain requests beside them are in, so main's debug bound on committed slots counts the scratch as it does on a verifying pool. The loop-back threading test gives its bare Speculator the runtime that main's drop warning now logs request ids through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model on the state contract declares speculative decoding with a Draft in declare_state, a contract value of plain fields: the node, the paged KV array the drafter's layers join, the sampler that verifies, the verify walk and mixed walk, the drafter by its draft-block key, checkpoint and resolved checkpoint directory, num_draft_tokens and mode, and the target's depth, residual width and weight dtype (HF's name). It hashes and digests as every declaration does, and contract.py imports nothing new from the resources. - Adapter: draft_config checks the settings as a serving config's draft block (DraftConfig.parse). node_resources passes the declared specs through drafter_resources, the same rewrite a model on resources runs on its own, with the drafter's declaration from Model.drafter(draft): the model builds it from the Draft with its family's code (its config's refusals, its tokenizer, its taps), which the engine doesn't import. verify_node builds the VerifyNode, the target's ModelGeometry from the Draft and the array's entry and the sampler's vocabulary. Model.declared_verify_nodes builds one per Draft, with the node's token_stop and device_loopback from Model.draft_node_fields; a model on the contract returns them from its verify_nodes when its draft block is given, so the default stays the empty list and a model that doesn't draft never reads its declarations for it. validate_deployment checks speculative decoding only once the declarations built their specs, so a declaration that fails is reported once. - DrafterDecl gains prefill_steps and verify_steps, empty unless a drafter's own resources step. - Analysis: a Draft must name a paged array and a sampler of its node and a walk it steps on. node_handles builds no handle for one. - engine_step expands a node's declared state before it adds the drafter's share, so the share sees the resource steps a node on the contract declares as state; on resources there is no state to expand, so the order changes nothing there. drafter_step keeps the expanded step's handles, which `replace` leaves behind (an init=False field). - drafter_step: a step with verify rows runs every attention on its verify kernel, and a verify step's sampler takes one row a request, which a node on the contract has no step field to declare. A node on resources declares the same, so for it nothing changes. The surface snapshot gains Draft, and the contract's hash and equality test covers it. Tests (test_state_draft.py, a declared toy node): its specs are drafter_resources over the arrays' and differ from a plain node's; its VerifyNode equals one written by hand, the target's depth the declared one; a Draft hashes and digests as a value, its settings are checked as a draft block, and a malformed one is refused; the analysis names a wrong array, sampler or walk; drafter_step marks verify and mixed steps, leaves prompt steps alone, and keeps what a node on resources declares. The default verify_nodes reads no declaration. Mutations caught: the verify marking dropped, the spec rewrite dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s length On pages of more than 32 tokens, FA3's paged kernel (flash_attn_3 3.0.1) reads a row's last page past its length once the step's longest row has about 64 queries, and masks those scores to zero probability. A stale NaN or Inf an earlier request's broken forward left in a reused page then still reaches the row, as 0 times NaN. Decode and verify rows never take that path; a mixed step's prompt row can. The attention manager warns once when it first builds a mixed step's FA3 wrapper over such pages; pages of 32 tokens or fewer avoid it. A CUDA test pins the observed behaviour, so a wheel that changes it shows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ModelGeometry.plain_rope: a drafter that rotates with plain RoPE through the node's positions (EAGLE-3's) is refused at startup over a position resource that scales (rope_scale other than 1, Llama 3.1 factors, LongRoPE), which would rotate the target one way and the drafter another. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verify rows (spans 4 and 8, a hidden last key) on FA3PagedWrapper and mixed rows (prompt rows whose new tokens are no multiple of the page) on FA3RaggedWrapper, eager and replayed with padding, over a pool whose sink page holds NaN, as do the slots past each row's length at 16 and 32: real rows are finite and match a dense fp32 reference, and a replay's padding comes out zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FA3RaggedWrapper.run masked a replay's padding queries with a token-id compare and an out-of-place masked_fill in every layer, two kernels and a copy of the attention output each time. It now calls main's zero_unplanned_rows, which zeroes the rows from the planned total in place on the device: one kernel, the same rows, and the real rows untouched. In a Qwen3-8B mixed graph that is 41 kernels fewer and about 0.75 ms less. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at holds it An asynchronous verifying node runs a mixed prompt-plus-verify step only on a capture whose verify rows read the resident tables, and a step that finds none raises and fails every row in it. The mixed walk was capped at its ladder's widest configured bucket and its token budget at the model's, so a capture that left the widest mixed buckets out (memory short) let the scheduler build steps no kept bucket held. At warmup the engine now picks the widest kept, non-lean mixed bucket that holds the worst step under its own caps (verify rows of a block each, and a prompt row at the budget's min_chunk_tokens floor), caps the walk's rows there (get_max_batch_size) and its budget, a serving override included, at its token capacity (ChunkingManagement.cap_budget). The runtime raise stays for overruns the bound doesn't cover. New startup refusals on an asynchronous mixed node: no token budget on the mixed walk (or its prompts' chunking off), and no kept bucket that holds the worst step. The instance's leader warns when the caps are lowered (rows below the ladder's top, or tokens below the budget), naming both and any buckets capture left out. Every new branch runs only for an asynchronous verifying node with a mixed walk; elsewhere the caps are as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l too drafter_step forced verify=True on the node's own attentions on a mixed walk, but added the drafter's own steps (prompt and verify alike) after that, untouched. A drafter attention declared for prompt steps alone would then plan a mixed step's verify rows on FlashInfer's prefill wrapper from the host, over their reserved lengths in asynchronous mode, and nothing caught it. Every shipped drafter declares its attention verify=True in its verify steps, so this was latent. The force now covers the drafter's own attentions on the mixed walk, at the declaration's source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…abels is refused A mixed step plans each KV label apart, and FA3's ragged wrapper that a resident gather fills off the tables holds one label's rows while the gather covers every row of the bucket. A mixed step over two labels would fail on shapes, or under combined labels gather the rows in the wrong order. Nothing reaches it today (every shipped drafter's node writes one label), so it is refused rather than supported: - at startup, before workers spawn and again at load (mixed_label_problem): a verifying node with a mixed walk whose KV pool the model writes under more than one label (Model.kv_labels); - where its attention plans, as a verify step's already is: a mixed step that plans two labels on FA3 raises; - in a mixed capture's resident gather, a bucket that registered no ragged wrapper raises instead of leaving its verify rows on the host's reserved lengths, which drafter_step's verify force now rules out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t doesn't fit A serving min_chunk_tokens past what the widest kept mixed bucket leaves beside its verify rows refused the node with a message about memory and a knob that doesn't exist. The refusal now names the floor and the most it can be, names the buckets capture left out only when there are some, and drops the knob. The envelope's docstring says it reads (bs, tokens) alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng their keys BucketKey has no order, so max() over (bs, tokens, key) raised TypeError when two kept mixed buckets tied on size and differed in their capture key. Rank by (bs, tokens) alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ompt row draws it A seeded reply on a verifying node depended on whether its prompt ran a prompt step of its own (main's sampler, FlashInfer's batch-position draw) or rode a mixed step (a refused verify row, drawn from the request's own stream at the bonus draw's number). sample() on a verifying sampler now draws through first_draw, which reads the row's filtered distribution and the same number a refused row reads, and lands the token and offsets as verify() does, so every token of a drafting node's reply comes from the per-row stream. verify() is split into _verify_params and _land for that; fused_draw_args is None on a verifying node, so no forward-side draw bypasses it. A node that doesn't draft keeps main's sample() unchanged (verify_draft_tokens is 0 there). The toy target samples its prompt token through the node's sampler, which the CPU can now do; the seeded-reply test runs both paths at T=20 over 8 seeds and two drafters, and a sampler test checks sample() against a fully refused verify() step after step. serving.rst (and its mintlify copy) says a seed fixes a drafting node's reply whatever the batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
serving.rst's draft block (and its mintlify copy) now says an asynchronous mixed walk is capped at the widest captured bucket that holds its worst step, that the cap covers prompt-only steps under the combined walk and a node-level max_batch_tokens override, and which mixed walks startup refuses: no token budget, no captured bucket that holds one, and a pool written under two KV labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et_up_draft ruff --preview's E302 and E301, which the stack added and main's lint config doesn't select. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
order is the batch's FilterOrder or None and slots the eager step's slot tensor or None; both were typed object. Two blank lines on either side of the class, as ruff --preview's E302 and E303 ask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r its prompt rode a mixed step The seeded test ran with top_k 0 and top_p 1 only, so a prompt draw that dropped the request's filters passed it. It is parametrized over a filtered case (top_k 20, top_p 0.7), and the toy driver takes both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an aligned floor only where one fits The refusal's other branches: a kept bucket short of the floor beside a mixed bucket capture left out, on a node with no chunk alignment and on one aligned to 8 tokens. The left-out mixed bucket is named and another walk's isn't; the min_chunk_tokens advice is a multiple of the alignment and absent where a kept bucket leaves less than one beside its verify rows; and the value it names starts the node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…f, and name the floor the message gives serving.rst's draft block (and its mintlify copy) left out the refusal for a mixed walk whose member prompt walk has its chunking turned off, and said the message names the configured min_chunk_tokens; it gives the floor it fit, capped at the budget and rounded to the chunk alignment, and the remedies that apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et that would hold the step The refusal advised freeing memory for the mixed buckets capture left out whenever there were any, though none of them might hold the worst step either: with that bucket captured the node is refused the same. The advice now names only the left-out buckets that pass the fit the kept ones get, bounded by their key's token count, since a left-out bucket may not be registered with the runner. The docstring lists the chunking-off refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…used, and the refusal prints the fitted floor Two refusal paths no test reached. A serving override of null max_batch_tokens on a prompt walk inside a budgeted mixed walk is refused (and with the override at 1024 the node starts), so dropping the chunking-off clause fails. And the floor the no-bucket refusal prints is min_chunk_tokens as the fit used it, not as configured: 130 at an alignment of 8 prints 136, and 2000 over a 1024-token budget prints 1024. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he same whether its prompt rode a mixed step The filtered case covered top-k and top-p only, so a prompt draw that dropped the request's min_p or its filter order passed. The seeded test adds a min_p 0.5 case and a top_p_first case, and the toy's sampler takes both when a test asks. top_p_first is CUDA-only, so its case lifts the CPU refusal and stands in for the CUDA filter with torch. The seeded case compares the two paths with each other, so a verifying node that dropped the order on both passes it. A second test checks, on each path, that some seed's reply with top-p first differs from the same request's with top-k first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd two, not four, before the timing test ruff's preview E302 and E303 in the stack's own test module; default ruff doesn't select them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd memory only for a bucket that would hold the step "The floor it fit" read as if the floor fit; it is the one no kept bucket holds, rounded up to the chunk alignment. The free-memory remedy names only a left-out bucket that would hold the step, as the engine now offers it. And the KV-label refusal says it applies in either mode, so it doesn't read as one of the asynchronous refusals before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
09f2a1e to
4d8b6e8
Compare
PR 3 of 7 in the stack, stacked on #1319 (to keep the stack linear; nothing here uses it). Head 4d8b6e8, base main 2c8ffde.
The engine and resource layer for speculative decoding (RFC #1186): a drafter in a target node's forward proposes
ktokens, the target scores them in one verify step ofk + 1rows per request, the sampler publishes how many were accepted, and each resource takes back what the rejected drafts wrote.No shipped model drafts here; it is tested on toy models. End-to-end numbers are in #1321 (DSpark) and #1325 (EAGLE-3, draft model).
What it does
model_kwargs.draftblock (DraftConfig);DrafterNeeds,DrafterDecl,VerifyNodeanddraft_verify_step(draft/drafter.py). A model on the state contract declares aDraftindeclare_state; the adapter turns it into the drafter's specs throughdrafter_resources, which joins the drafter's KV layers to the target's pool as a layer set, adds2(k + 1)tokens of KV slack and marks the sampler as verifying. Taps (taps/) give a drafter the target's residual stream.k + 1tokens;VerifyLedgernumbers steps by commit, so every lockstep rank applies the same result at the same step with no collective, lowering KV and position lengths at a later admit.NodeDraft.cutcuts each run at its stop id and budget.accepted + 1: the captured graph advances each row's device length, and no host write touches it until the result is applied. LongRoPE table ids follow the verify span.first_draw: the request's own stream at the bonus draw's number), so a seeded reply doesn't depend on which its prompt rode; a node that doesn't draft keeps main'ssample. An all-greedy step verifies by the argmax alone (greedy_verify, captured as a lean twin), skipping 1-2.8 ms of full-vocabulary work a call at 128-512 rows (H100, the verify sampler replayed in a CUDA graph). Requests a verifying sampler can't honour are a 400.verify_backendpins one).VerifyNode.mixed_walk): verify rows lead, prompt rows follow, only verify rows enter the ledger, on FA3's ragged wrapper; the drafter's own attentions on the mixed walk run on the verify kernel too.max_batch_tokensoverride included) at the widest captured bucket that holds its worst step: verify blocks plus amin_chunk_tokensprompt chunk (Engine._fit_mixed_steps_to_captures). A cap that lowers either is logged.min_chunk_tokenscapped at the budget and rounded up to the chunk alignment) and the remedies that apply: a lowermin_chunk_tokens, memory for a left-out bucket that would hold the step,mode: synchronous.zero_unplanned_rows(DSpark +1.25% at c64 in a shared-cache A/B; about −0.75 ms per mixed graph in nsys). Over pages larger than 32 tokens, FA3 can read past a row's length in its last page; a one-time warning namesresources.kv_cache.page_size(FA3_TAIL_SAFE_PAGE_SIZE).mode: autoruns asynchronously where every resource can correct itself on the device, else synchronously; one MIN reduction across the lockstep instance makes every rank agree.validate_deploymentlists every problem before a worker spawns: geometry, slack, capabilities, a drafter declaring plain RoPE over scaled positions, sliding windows, sequence or context parallelism.Engine.repeat_refusalrefuses repeated decode on the verify walk once main's Run several decode steps per launch (step_repeats) #1424 calls it.mstar_spec_*metrics, threeworker_phasesverify timers, a bounded-memory offload fingerprint underMSTAR_KV_DEBUG_ASSERTS; docs in.rstand.mdxanddocs/design/admission_and_preemption.md.MSTAR_KV_DEBUG_ASSERTS, for every model) switches on only for a node with aVerifyNode. Qwen3.5-0.8B'sworker_phasesthroughput at c64 measured −0.2% / −1.5% against two main arms 1.3% apart at d7b935363f (an earlier local tree of the stack), and its e2e outputs equal main's at aa18f98 (the pushed feat(qwen3): add EAGLE-3 and Qwen3-0.6B draft-model drafting (PR 7 of 7 in the stack) #1325).SamplerResource.verify_draft_tokens, and the mixed and envelope code only for a node with aVerifyNode.One new optional dependency:
flash_attn_3(3.0.1), FA3's paged kernel, which asynchronous verify needs on Hopper; it is installed by no extra (docs/installation.rst), and without it a verifying node runs synchronously on FlashInfer. No new environment variables.Since the last push (09f2a1e): review fixes, each with its tests and docs.
first_draw: a verifying node's prompt token is drawn as a mixed step's prompt row draws it;_VerifyParamstypes its filter order and slots.Tests
On an 8 x H100 80GB host, torch 2.12.1+cu132, FlashInfer 0.7.2, FA3 3.0.1. Commits are named per row: 09f2a1e is the pushed head, and 5a74a57 is it less its last four commits (page-size warning and test, plain-RoPE refusal, padding change).
d9e344ac71 is an unpushed earlier state of this head; it differs from it only in
test/modular/test_spec_mixed.py. 0023eff82b and dbc627e8f7 are unpushed earlier states of #1325's head, which differ from it only intest/modular/files.CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) +test_tp_async_lockstep_sim.py(-n 3)ruff check .(0.17.5, 0.16.0)mstar_rust;-n 6,OMP_NUM_THREADS=2), failing tests by id against main's on the same hosttest_token2wav.pyin one process); the 13 main also fails on this host (arm64 environment failures), no otherstest_spec_*.pyin the same Docker image,-n 3test_state_lint.py,test_validate_deployment.pytest_spec_mixed_attention.pyandtest_spec_page_sizes.pytest_spec_verify_sampling.py,test_spec_mixed.pyon CUDAThe real-weights tests of this code are in #1321 and #1325.
New tests for the review fixes, each failing before its fix:
test_spec_engine.py: the envelope cap, themin_chunk_tokensrefusal, two buckets of one size, the two-label refusal at load.test_state_draft.py,test_spec_mixed_attention.py,test_spec_startup.py.test_spec_mixed.py::test_a_seeded_reply_is_the_same_whether_its_prompt_rode_a_mixed_step: 8 seeds, two drafters, T=20; unfiltered, top-k 20 / top-p 0.7, min-p 0.5, and top-p before top-k.test_spec_verify_sampling.py::test_a_verifying_samplers_prompt_token_is_a_refused_verify_rows_token.Tests of existing code, checked by mutation:
top_p_firstis CUDA-only, so that case stands in for the CUDA filter with torch.test_top_p_first_changes_some_seeded_reply_on_a_verifying_nodechecks, on both paths, that some seed's reply with top-p first differs from the same request's with top-k first. With the order dropped from every verify draw's parameters (_verify_params), the 64 seeded cases still pass and it fails.test_an_async_mixed_walk_whose_prompt_walk_turns_chunking_off_is_refusedfails when the chunking-off clause goes.test_an_async_mixed_refusal_names_the_floor_no_kept_bucket_holds_not_the_configured_one(130 at an alignment of 8 prints 136; 2000 over a 1024-token budget prints 1024) fails when the message prints the configured value.test_an_async_mixed_refusal_names_the_buckets_capture_left_out_and_a_floor_a_kept_bucket_holdsfails when the left-out remedy is dropped, when themin_chunk_tokensadvice ignores the alignment, or when it is given with less than one aligned chunk of room.test_an_async_mixed_refusal_offers_no_memory_where_no_bucket_capture_left_out_would_hold_the_step, and a too-narrow left-out bucket in the case above, fail when the free-memory remedy names every left-out bucket.New modules include
test_spec_engine.py(the realEngine, a toy target: greedy output equals plain greedy whatever the drafts),test_spec_tp.py,test_spec_worker.py,test_spec_mixed*.py,test_spec_greedy_verify.py,test_spec_page_sizes.pyandtest_state_draft.py.GPU runs (2026-11-18)
found no resident captureorplanned no FA3 ragged wrapperline; the fourmixed steps capped atlines come fromtest_spec_engine.py's narrowed-capture tests.test_qwen3_dspark_device.pyfailed with this PR's refusal, because that test's mixed ladder held no worst step. The ladder is fixed in feat(qwen3): add DSpark drafting on the decoder scaffold's verify path (PR 4 of 7 in the stack) #1321, and the stack's CUDA set passes at dbc627e8f7.plans one KV label, no-resident or traceback line.Open
wait_for_workwith requests running (4 of 7 runs at 64 pages, 2 of 2 at 32, 3 of 3 at 16).Engine.repeat_refusalis called by nothing until Run several decode steps per launch (step_repeats) #1424 lands.Not yet run on a GPU
Follow-ups
row_viewsthrough the cut; a tap that leaves the step as an output (Qwen3-Omni); decode onFA3PagedWrapper; a capabilities field for a resource's default kernels._qk_norm_rope_kv_kernel(6.25 against 21.7 µs a launch at 512 tokens, microbenchmark), and zeroing a page's tail at hand-over.AI assistance was used (Claude Code); the tests run and their results are listed above.
🤖 Generated with Claude Code