Repository navigation
Conversation
NSagan271
left a comment
There was a problem hiding this comment.
Overall looks good, some relatively minor design comments
| any `verify_problem`; raise to refuse the node.""" | ||
| return | ||
|
|
||
| def verify_problem(self) -> str | None: |
There was a problem hiding this comment.
This adds a fifth way for a resource to refuse a verifying node, beside verify_refusal / async_verify_refusal on the spec, verify_hardware_refusal, and raising from bind_verify_walk. It exists for one check, a recurrent pool that nothing reruns, and needs a back-pointer for it (GDNManager.bind_verify_walk sets self._pool.verify_settler = self). The two new spec overrides in this PR (LinearAttnSpec, RecurrentStateSpec) don't read draft either.
If #1320 moves to a single capabilities object, this check could be made at the spec level, where all of the node's specs can be seen together ("a recurrent pool on a verifying node needs a GDN linear-attention spec on it"), with no hook or back-pointer.
There was a problem hiding this comment.
Agreed. With the single capabilities hook from #1320, verify_problem and its pass after bind go away. It becomes a startup check over the node's specs: every recurrent pool on a verifying node has to be the recurrent_state of a GDN LinearAttnSpec on the same node. The recurrent and GDN overrides become overrides of that hook, and GDN's dtype and missing-pool raises move out of bind_verify_walk. The back-pointer goes too. The check is the spec-level uncorrected_problems in node_problems, which validation and the engine's load both run, before any resource is built; nothing at bind checks the pools.
| or None.""" | ||
| return None | ||
|
|
||
| def settle_verify_step(self, result) -> None: |
There was a problem hiding this comment.
The docstring says this is "named apart from KDA's own settle_verify, a different contract." On main, GLM5-Next already does MTP speculative decoding through KDA, driven by the model: glm5_next/submodules.py:535 calls set_prefix_len(accepted), and KDA replays from a two-sided state. This PR adds an engine-driven GDN undo by rerun, and the engine path refuses KDA.
Is GLM5-Next meant to move onto draft_verify_step, with KDA's replay behind settle_verify_step? If so, a note or an issue would help. Either way, the near-identical names will confuse whoever touches KDA next.
There was a problem hiding this comment.
Not in this stack. Moving GLM5-Next's model-local MTP onto the engine's verify step is an open question in the RFC. It would also need MLA verify attention and its own TP broadcast replaced, so I'll open an issue once this design settles. KDA's replay fits the same contract: after the sampler's verify, inside the step's graph, leave the state as if only the kept tokens ran. For the name, I'll rename ours to correct_verify_state and write that contract in its docstring. I'm avoiding undo_rejected because take_back already gives back the rejected drafts on the host side.
| _observe(drafter, block, result) | ||
| from mstar.engine.resources.base import Resource | ||
|
|
||
| for resource in resources.values(): |
There was a problem hiding this comment.
This walks every resource on every eager verify step and calls settle_verify_step where type(resource).settle_verify_step is not Resource.settle_verify_step. The engine already computes take_back_keys once at bind. A settle_keys computed the same way, on NodeDraft or DrafterShare, would make the set and its order explicit, skip the per-step check, and drop the "only an override is called" rule.
There was a problem hiding this comment.
Agreed. The engine computes the list once at bind, in the same dependency order as the take-back keys, and puts it on the drafter binding, named correcting_keys after the hook. The verify step calls exactly those resources in that order. Which resources are on it is still decided by whether they override correct_verify_state, but that test now runs once at bind instead of every step. The list only fixes which resources run the hook and in what order; the pool check from the other thread is a separate spec-level check.
| return sampler.loopback_state(name, request_ids)[:n] | ||
|
|
||
|
|
||
| def _observe( |
There was a problem hiding this comment.
With needs.target_hidden, the target function returns (logits, hidden) instead of logits, and observe takes (block, result, hidden) instead of (block, result). Its return value is the carry only when needs.carry is set. propose already takes a DraftInputs NamedTuple. An ObserveInputs(block, result, hidden=None) and a TargetOutput(logits, hidden=None) would give one signature each, and observe could then go into the Drafter protocol (see my comment on #1321 about the protocols).
There was a problem hiding this comment.
Agreed, and done. Every target function returns TargetOutput with the logits and an optional hidden state, and observe takes one ObserveInputs and returns the new carry or None. ObserveInputs starts in #1320, where observe first appears, and this PR adds the hidden field. observe stays on Drafter with a default, since the drafters now subclass it explicitly, so a second protocol isn't needed. target_hidden stays as the declaration that makes the target return the hidden state, and the engine refuses a drafter that declares a carry and returns none.
| kept = {} | ||
| if needs.carry is not None: | ||
| width, dtype = needs.carry | ||
| kept[DRAFT_CARRY] = LoopbackState(width, dtype, (0.0,) * width) |
There was a problem hiding this comment.
Question: was a carry resource declared through drafter_resources (like TapSpec) considered, instead of the sampler's loop-back state? Reusing the sampler's per-request rows is pragmatic, but it ties a drafter's hidden-vector carry to the sampler's master allocation. That's where the startup refusal above one rank or with the in-graph scatter off comes from (lifted in #1324), and the warning when the recurrent pool holds more requests than the sampler has rows.
There was a problem hiding this comment.
The carry stays on the sampler. I tried giving it its own per-request state, as a block in a recurrent state pool, but a pool that holds only the carry has no prefix snapshots to restore, so it shuts prefix caching on its node (the Qwen3 DSpark node loses its KV prefix cache), and on a node with a GDN pool the startup check refuses a second pool that no linear-attention layer reruns. A pool that opts out of prefix restore, because every step overwrites its state whole, would remove that obstacle; I'll leave that as a follow-up rather than add it to this stack.
| prompt_walks: frozenset[str] | ||
| sampler_key: str | ||
| # the sampler loop-back state the drafter's carry is stored through (`carry_writes`) | ||
| prompt_writes: frozenset[str] |
There was a problem hiding this comment.
nit: prompt_writes and verify_writes always hold the same value in this PR, and _read_carry's name has no caller. The description says they diverge in #1324, so they could land there.
There was a problem hiding this comment.
Agreed. This PR will keep one field for both walks and drop the unused name. #1324 adds both when the lag flag first needs them.
| ) | ||
| rows = sampler_slots() | ||
| for key, resource in mgmt.resources.items(): | ||
| usable = getattr(getattr(resource, "config", None), "usable_slots", None) |
There was a problem hiding this comment.
nit: getattr(getattr(resource, "config", None), "usable_slots", None) finds the recurrent pools by duck typing. An isinstance check, or a property on Resource, would make it clearer what's being matched.
There was a problem hiding this comment.
Agreed. It will check isinstance against RecurrentStatePool and read config.usable_slots directly.
4e8e071 to
606ebda
Compare
606ebda to
fab212e
Compare
merceod
left a comment
There was a problem hiding this comment.
Read it on top of #1321 and #1320, with NSagan271's seven threads. They're all in the code: correct_verify_state with correcting_keys built once at bind in the runner's order, the spec-level uncorrected_problems in place of the bind-time hook and the back-pointer, TargetOutput and ObserveInputs, one loopback_writes field, and the isinstance check. CI is green on this head.
The closed gate is right for the delta rule. With a = b = -1e4, softplus(a + dt_bias) underflows to 0 in fp32, so the decay is exp(0) = 1, and sigmoid(-1e4) = 1/(1 + inf) = 0. A rejected token then leaves the state as it was, bit for bit, on both of FlashInfer's MTP kernels, and since rejected tokens come after the kept ones the slot ends after the kept tokens. The NaN with -inf has a known cause too (inline on the wide-batch test). That inline is the one thing I'd want fixed before approving, since the test meant to guard it runs on the other kernel.
Plain path, Qwen3.5 on main included. The conv kernel at UPDATE_STATE=True runs the same loads and stores (the window update moved under a constexpr if, x_base hoisted). GDNManager adds a dict lookup per plan and a None check per layer. The pool adds a frozenset test per commit, and a span check per resident gather that only a node with a verify block can trip. The sampler masters change only for loop-back state, which on main is Whisper's small timestamp-rule row, now allocated at 4097 rows from the start. I expect bit-identical output, and we'll check it on an H200.
Two more:
- The rerun keys layers by the resource cursor, which main's GDN layer never sets (inline). #1323 lands on the state contract, so this path should work there too, or it gets redone.
- Merge with current main conflicts in
gdn.pywith #1147 (SP-split GDN:_split_rows, and main'sresident_refusalnow refuses resident decode on an SP-split node) and inrecurrent/pool.pywith #1423 (span * repeats). I think the resolution needs a refusal for verify on an SP-split GDN node and forstep_repeatson a verify walk.
What we'll run on an H200, restacked and merged with main: test_gdn_verify_device.py with the wide case past B x HV = 128, main's GDN capture, device loop-back, SP and prefix-root modules, Qwen3.5-0.8B greedy ids and worker_phases at c32/c64 against main, a Whisper timestamp run against main, and _warm_verify's log line at a Qwen3.5 geometry.
|
|
||
|
|
||
| @pytest.mark.parametrize("device", ["cpu", "cuda"]) | ||
| @pytest.mark.parametrize("rows", [16, 32, 64]) |
There was a problem hiding this comment.
This case doesn't reach the kernel the NaN came from. In FlashInfer 0.7.2, gated_delta_rule_mtp runs the inline kernel when B x HV <= 128 and the warp-specialized gdn_verify_kernel_mtp above that (run_mtp_decode: use_inline_kernel = (B * HV) <= 128). The warp kernel computes softplus without a branch, use_softplus * softplus_val + (1 - use_softplus) * x, so a = -inf gives 0 x -inf = NaN. The inline kernel branches. The toy here has 2 value heads, so 16, 32 and 64 rows are B x HV of 32, 64 and 128, all inline. That also fits what you saw if it was on a geometry with 16 value heads (8 rows inline, 16 rows warp). No device test in the PR runs the warp kernel, and it's the one a real Qwen3.5 batch takes past 128 / HV rows. Running this case at 80 or 128 rows, or with 16 value heads, would cover it. A direct check that all-closed gates leave the state bit-identical at B x HV > 128 would pin the -1e4 choice, and the CLOSED_GATE comment can name the cause instead of calling it a hypothesis.
| if self._verify_block is None: | ||
| return plan.run(q, k, v, a, b, state_layer, a_log, dt_bias) | ||
| # the forward leaves the slots as they were, which is what the rerun starts from | ||
| rerun = self._kept.setdefault(self._default_layer_idx, VerifyRerun()) |
There was a problem hiding this comment.
_kept is keyed by _default_layer_idx, which only the resource-path cursor sets (the toy's _GDNMix.set_layer_idx). Main's GatedDeltaNet reaches run and run_conv through mstar.engine.state.calls.gated_delta_net and causal_conv, which pass the layer's views but never set the cursor, and plan resets it to None. So on Qwen3.5 with a drafter, every layer's VerifyRerun would land on _kept[None], and correct_verify_state would raise in pool.block("state", None). The toy works because it overrides GatedDeltaNet.forward with a copy of its body on the resource path (the description says it runs main's forward directly). Could run and run_conv take the layer from the caller (the handle has h._layer_id(layer)), and the rerun use the views it was given? Then the toy can run main's forward unchanged, and #1323 doesn't have to rework this.
| start = time.perf_counter() | ||
| state = torch.zeros(rows + 1, g.num_v_heads, g.head_v_dim, g.head_k_dim, device=device) | ||
| a_log = torch.zeros(g.num_v_heads, device=device) | ||
| for n in range(1, rows + 1): |
There was a problem hiding this comment.
FlashInfer's MTP compile cache (_get_compiled_mtp_kernel and its inline twin) is keyed on the tile config, the kernel kind, disable_state_update, the head dims, dtypes and layouts, not on B. So the variants are the five configs get_mtp_config picks across B x HV, times frozen or not, and the other row counts here are only launches. One row count per config would do. The scratch state is (usable_slots + 1) x HV x V x K fp32, about 0.5 GiB at 256 slots and 32 value heads. Not blocking. The startup time at a Qwen3.5 geometry is the number I'd want in the description.
fab212e to
f0b9493
Compare
… and the docs name the carry store that exists The scaffold calls a drafter's prefill(PrefillInputs) after a prompt step samples; a drafter that declares a carry returns the prompt's first one, and the scaffold now stores it through store_draft_carry, which exists from this PR on. Both copies of adding_models told a model to call engine_inputs.store_draft_carry(carry), which this PR's restack moved onto ARNodeSubmodule as store_draft_carry(engine_inputs, carry); they now say a scaffold drafter returns the carry from prefill and any other node calls self.store_draft_carry(engine_inputs, carry). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The target was a function to logits, or to a (logits, hidden) pair when the drafter declared target_hidden, so draft_verify_step branched on the declaration to unpack it. Every target now returns TargetOutput(logits, hidden=None); target_hidden stays the declaration that adds the hidden states, and a drafter that declares it and gets none is refused at the verify step's trace. The scaffold's target sets hidden only when its drafter declares it; the toys' targets and the docs (both copies) follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he carry by itself DrafterShare had prompt_writes and verify_writes, always the same set (the drafter's carry_writes); it has one loopback_writes, which drafter_step adds to the sampler step of both the prompt and the verify walks. _read_carry loses its name argument, which only ever was DRAFT_CARRY. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resource with usable_slots The check that a carrying node's recurrent pool holds no more requests than the sampler has carry rows found the pool by duck-typing a usable_slots on any resource's config; it asks isinstance(resource, RecurrentStatePool). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… refuses an unrerun pool The pool kept a verify_settler, which GDN set at bind, only so its take-back could raise if no linear-attention resource reran its state. Startup already refuses that from the specs (unsettled_problems: no LinearAttnSpec of the node names the pool as its recurrent_state), and the resources that rerun are bound once at load (settle_keys). The attribute, GDN's assignment and the take-back's guard go, with the test of the guard and the engine test that unset it by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ettle settle_keys becomes correcting_keys and unsettled_problems becomes uncorrected_problems, after the hook they serve (correct_verify_state), so they no longer read as siblings of KDA's settle_verify, a different contract. Both copies of adding_models now say the engine calls only the resources that override the hook, as bound once at load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A target written to the old API returns a bare logits tensor, which the step unpacked along its rows: a confusing ValueError for most row counts, and with one request and k = 1 its two rows became logits and hidden states. draft_verify_step now raises TypeError, once, while the step is traced. spec_toy's wrapper that turned a bare tensor into a TargetOutput goes, so the toys' targets return one; the hidden-state and TP tests that patch the toy's target follow. Every doc and docstring that still said a target returns logits, or (logits, hidden), now says TargetOutput, and Drafter.prefill's says it returns the rows' first carry for a drafter that declares one. Tests: the TypeError; the target_hidden raise for a target that returns no hidden states, which nothing tested; and the warning for a recurrent pool holding more requests than the sampler's carry rows, also untested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is refused The scaffold stored the carry only when prefill returned one, so a drafter that declares a carry and forgot to return it drafted its first verify step from the loop-back default with no error, though draft_verify_step refuses the same mistake in observe. It now raises the same way, while the step is traced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both walk branches assigned the same writes; only the steps differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne run it Since the take-back guard went, this spec check is the only thing that refuses a verifying node with a recurrent pool no linear-attention resource reruns, and its only test called uncorrected_problems itself. A second pool on the hybrid toy's node is now refused by check_deployment and by the engine's load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous PR typed it None, since nothing stored a carry there. Now the scaffold stores what a carrying drafter's prefill returns, so both protocols return a Tensor or None, and the hosted one's docstring says what it returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It still called verify_unrecorded with the old one-argument signature, a TypeError on CUDA, and asserted the offload applied row 2's held result, which was the deleted settle's behaviour. The offload now leaves the result held and marks the row returning; the test checks the mark after the offload and the reload, that the row's first admit back clears it, and that the run stays plain greedy. It fails without _mark_offloaded. Not run here: the module needs CUDA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TP's Qwen3.5's MTP head is held out of this stack, so CarrySpec's docstring and both adding-models guides describe the carry without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efill that returns none With the Qwen3.5 MTP head held out of the stack, no scaffold drafter in this PR declares a carry, so the scaffold's store after prefill and its refusal of a carrying drafter whose prefill returns none had no test until EAGLE-3. On the CPU Qwen3 harness, DSpark is made to declare a 4-wide carry and its prefill to return a known one: the prompt step leaves it in the sampler's DRAFT_CARRY rows, and a prefill that returns none raises. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s is refused GDNManager.capabilities checked only the pool and the state dtype, and bind_verify_walk warmed the multi-token kernel whenever the device type was "cuda", which a ROCm build reports too. A worker off CUDA, or whose FlashInfer lacks gated_delta_rule_mtp, would then fail at the warmup's import (a hypothesis, not run on ROCm) instead of refusing at load. The platform now comes from current_platform, the import is tried in capabilities, and binding warms only where both pass. The rerun has no test on a lockstep instance, so a node of more than one rank is refused until one exists; the instance's size is the same on every rank, so they all refuse alike. No shipped model verifies on GDN yet, so the refusals cost nothing today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GDNManager's verify walks, planned block and kept reruns were set in __init__, so a manager built without it (main's GDN prefill and Qwen3.5 mixed-step tests build one with __new__ or a subclass that skips super().__init__) hit an AttributeError on a step that verifies nothing, and two of main's tests were edited just to set them. They are now immutable class defaults that an instance rebinds (bind_verify_walk builds a new mapping rather than writing into the shared one), as the decoder scaffold does with _drafting, and both of main's tests are back to main's text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLOSED_GATE is finite because -inf gave NaN at 16 rows and more, where the multi-token kernel runs its wider-batch path, but no test reran a verify step that wide: the device rerun test runs one row, so a switch back to -inf would pass. The new case reruns 16, 32 and 64 rows, each accepting a different count, and checks every row's state is finite and that of its prompt and kept tokens against the float64 reference. Its CPU case runs the stand-ins and checks the test itself; the CUDA case needs a GPU and FlashInfer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…correcting resources refuse it The mixed prompt-plus-verify step catches up with the drafter interface this series grew: - the target returns a TargetOutput (its hidden states over the step's tokens, for a drafter that reads them); - a carrying drafter reads its rows' carry before propose, and its observe returns every row's carry, a prompt row's being its prompt's first, which the sampler keeps from its last chunk, as a prompt step's. A resource that corrects a verify step's state in its graph (GDN's) reads every row as a verify block, which a mixed step's prompt rows aren't. A node with one and a mixed walk now refuses at load, from the resources' types, so the refusal is the same on every rank. Tests: the carrying toy drafter in a mixed step drafts off its carry alone (accepted in full there and on the late prompt's first verify step, whose carry the mixed step stored), and decodes plain greedy; and a node with a correcting resource refuses a mixed walk and loads without one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with no layer id GDNManager kept each layer's rerun under the resource's default layer cursor, which only the resource path sets: main's GatedDeltaNet on the state contract passes each layer's pool views and never sets it, so a drafting Qwen3.5 would have recorded every layer under None, and the correction would have indexed the pool by it and raised. The verify run and conv now record the views they were given, the manager keeps the two in the order its layers ran, and the correction writes each back into its own view. A step that ran the conv and the delta rule on different numbers of layers is refused. The hybrid toy and the GDN verify tests no longer set the cursor, so they run the path main's layer takes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…before anything binds The refusal ran after the resources bound the verify walk (GDN's binding warms its kernels) and suggested verifying asynchronously, though it fires in either mode. It now runs right after the node's other startup checks, before the mode agreement and any binding, and names only the way out that exists: run the node's prompts in steps of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every GDN plan rebound _kept_conv and _kept_delta to fresh lists, on nodes that never verify too, which main's plan doesn't do. The rebind now runs only when the step planned is a verify step (its block, or the preplanned one's), so a node with no drafter plans as on main; the class comment says which defaults are rebound and which a verify step replaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VerifyRerun held both roles' fields, half of them unset per record, and the kept lists were annotated as tuples while a verify step holds lists. ConvRerun (run_conv's pool view, input and slots) and DeltaRerun (run's pool view, block, slots, output and settings) each hold one role; the annotations say list on a verify step, the empty tuple otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ach layer it ran The handle path (calls.causal_conv, calls.gated_delta_net) on model layers 1 and 3, which the pool holds at 0 and 1, with no layer cursor: each layer's conv window and state end at the kept tokens. It fails before the rerun kept the pool views its layers ran on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…that never verifies too Every plan that isn't a resident replay writes _verify_block (None on a node with no verify walk), so a node that never verifies doesn't keep the class default; only the rerun lists stay class defaults off a verify step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed ahead or when it runs, on the CPU The test covered only the plan made when the step runs, so reverting the gate on the async worker's preplanned branch passed it. It now runs both branches, through test_gdn_verify's CPU harness (whose step helper can plan ahead), and moves there from the device module, where it was the one case that needs no CUDA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed ahead or when it runs Every plan that wasn't a resident replay looked the step's walk up among the verify walks and wrote _verify_block, None on a node with no verify walk, so a node that never verifies ran code main doesn't. The lookup and the writes, of the block and of the preplan's cached block, now run only on a manager bound a verify walk (at startup, before any plan), and the class comment says so. The gate test checks that a node that never verifies keeps neither block as instance state, whichever branch planned its step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The preplanned gate case took its plan right after making it, so a preplan that wrote the verify block or the rerun lists as it planned passed every CPU test. The new case plans the next step ahead where the async worker's plan thread does, once the step before has committed (StepExecutor .arm_preplan waits on that and nothing else), both ways round (a verify step then a prompt step, and the reverse), on a verifying node and on one that never verifies. The verify block and the lists are still the step before's after the preplan, and the next step then keeps and corrects its reruns as its own plan says. The step helper is split into plan, take and run. Planning ahead before the running step's layers would be a different order: the worker never produces it, and main's own wrappers (_current) don't survive it either, so the case doesn't model it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0d50223 to
5946a09
Compare
PR 5 of 7 in the stack, stacked on #1321. Head 5946a09, base main 2c8ffde.
#1320 refuses a verifying node with recurrent state, because GDN layers write their state in place and nothing undoes a rejected draft. This PR adds that undo for GDN on CUDA, the per-request state a drafter keeps across steps (the carry), and a typed target output.
No shipped model uses the GDN path yet; its first user is Qwen3.5 MTP (#1323, held). It is tested on the hybrid toy, whose GDN layer keeps main's weights and projections and overrides
forwardto run on the linear-attention resource. The carry's first model user is EAGLE-3 (#1325).What it does
Resource.correct_verify_state(a new hook, no-op by default) runs right after verify:GDNManagerreruns the delta rule over the block with rejected tokens' gates closed, andsettle_convrebuilds each conv window in one Triton kernel. Both are captured in the verify graph and replay from gathered slots when asynchronous.ConvRerun,DeltaRerun), so it works through the state contract's handle.CLOSED_GATE = -1e4, because-infgave NaN at the 16 and 64 rows measured on a Qwen3.5-2B geometry on H100.take_backcorrects the token count; a verify step saves no decode snapshot; main'severy_row_has_state([State Contract 8/N] Multi-Label Steps, with Bagel, Chatterbox and Whisper #1387) is kept.disable_sink_slotor several labels, a non-fp32 state, off CUDA, withoutgated_delta_rule_mtp, more than one rank, and a recurrent pool no linear-attention spec names.DrafterNeeds.carry = CarrySpec(width, dtype)is kept as the sampler loop-back stateDRAFT_CARRY, one row per request. The engine adds the writes to prompt, verify and mixed steps;draft_verify_stepreads it beforeproposeand stores whatobservereturns; the scaffold stores whatprefillreturns.TargetOutput(logits, hidden)for a verify step's target; a drafter declaringtarget_hiddengets the hidden states.adding_models(GDN verify, the carry,TargetOutput) andMSTAR_SAMPLER_INGRAPH_SCATTER's row inenvironment_variables(.rstand.mdx).A plain GDN node pays a few lookups per step and compiles the conv's plain path under
UPDATE_STATE=True. The plain-node measurements are #1320's, at earlier trees; that a plain node is unchanged at this head is a hypothesis from the code (the plan writes verify state only under the bound verify walks,_verify_walks). End-to-end drafting numbers are in #1321 and #1325.No new dependencies or environment variables.
Since the last push (0d50223): review fixes, each with its tests and docs.
Tests
On an 8 x H100 80GB host, torch 2.12.1+cu132, FlashInfer 0.7.2, FA3 3.0.1; commits are named per row. 0d50223 is the pushed head.
57600142e3 and f729f5c227 are unpushed earlier states of this head, which differ from it only in
test/modular/files outside the GDN modules. 0023eff82b and dbc627e8f7 are unpushed earlier states of #1325's head, which differ from it only intest/modular/files.CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) +test_tp_async_lockstep_sim.py(-n 3)test_sequence_parallelcase that hit the 120 s timeout under load; the module passed run alone at this layer's earlier state-contract head); sim 16 passedruff check .(0.17.5, 0.16.0)mstar_rust;-n 6,OMP_NUM_THREADS=2), failing tests by id against main's on the same hosttest_token2wav.pyin one process); the 13 main also fails on this host (arm64 environment failures), no otherstest_spec_*.py,test_qwen3_mixed.pyandtest_qwen3_dspark_device.py's CPU cases, same Docker image,-n 3test_gdn_*, spec, draft, DSpark, ...), 2 x H100test_gdn_verify_{device,resident,engine}.py,test_gdn_cuda_graph.py, three runs eachThe plan gate and the rerun's views:
test_gdn_verify_handle.py(CPU): a verify step through the state contract's handle on model layers 1 and 3 (pool indices 0 and 1), no layer cursor, settles each layer at the kept tokens; it fails before the rerun kept its layers' views.test_gdn_verify.py::test_a_step_that_doesnt_verify_keeps_nothing_for_a_rerun(CPU) checks the plan gate on both branches, a plan made when the step runs and one made ahead on the async worker's plan thread. Reverting either branch's gate fails it, and it checks that a node that never verifies keeps neither verify block as instance state.test_gdn_verify.py::test_a_step_planned_ahead_writes_no_verify_state_until_it_runsplans the next step ahead where the plan thread does, after the step before commits. The verify block and rerun lists stay the step before's until the next step takes its plan, so a preplan that wrote them as it planned fails it.The correction itself:
test_gdn_verify.py,test_gdn_verify_engine.py,test_gdn_verify_resident.py(CPU): after the correction the state and conv window equal a sequential reference over the kept tokens, for every accepted count; without it they don't; greedy output equals plain greedy on the real engine.test_gdn_verify_device.py(CUDA): eager, captured and asynchronous; prefix hits; the rerun at 16, 32 and 64 rows with finite states;settle_convbit for bit; an offload with a pending result underMSTAR_KV_DEBUG_ASSERTS.test_spec_engine.py,test_spec_startup.py,test_spec_mixed.pyandtest_scaffold_carry.py(stored from prompt,observeand mixed steps; survives recycled handles and offload; the refusals).Open review items
CLOSED_GATEmutated to -inf it still passes (at 0023eff82b). A case above 128 and theCLOSED_GATEcomment change together._warm_verifycompiles every row count, not one per FlashInfer tile config. At the Qwen3.5-0.8B and 9B geometries (256 slots, block 4, at 0023eff82b) it takes 30.7 s and 29.7 s into fresh cache dirs and 0.1-0.2 s warm, ending with 4 compile-cache entries each, at +271 and +535 MiB peak.Not yet run on a GPU
Follow-ups
correct_verify_state.AI assistance was used (Claude Code); the tests run and their results are listed above.
🤖 Generated with Claude Code