Repository navigation
Conversation
513bf0f to
cbad78a
Compare
cbad78a to
b5fc61b
Compare
merceod
left a comment
There was a problem hiding this comment.
Static pass. I read eagle3.py against vLLM's EAGLE-3 path and it matches on everything I checked: embeddings before the hidden in the layer input, norm_before_residual and its default per format, the pre-norm sum as the carry, target = draft + d2t[draft], the default taps as layer inputs at (2, L/2, L-3), and the positions of the chained passes. observe rewriting the layer's K/V over the block from the target's taps is what keeps later steps reading target-derived K/V, same as vLLM. The draft model's lag, observe and keep flag read right too.
One bug, inline on eagle3.py: a reused prefix page can hold EAGLE-3 K/V built from another request's next token. Small impact, but it's the same thing that made you refuse chunked prompts.
Other asks:
- Nothing checks the EAGLE-3 head against a reference. The served tests gate on tau > 2.0, and the closest evidence is the benchmark's tau next to vLLM's at e0f7d1e78. A one-off comparison of the first drafts against a reference EAGLE-3 forward on the same taps would settle it, with the agreement rate stated. We can run that on our H200 if you'd rather.
- The draft model ships a config but has no throughput row at all, and your numbers hint k = 4 may be the faster setting (inline on the config).
- The c32 gap. Your worker_phases row has the verify batch at about 27 at c32, so on average about 5 of 32 requests aren't in the step. If vLLM's batch is near 32, that alone is roughly the 15%, and it's the same wait that makes TTFT p90 long. So the prompt side looks like at least as big a lever as the verify kernels. Does a prompt step on a drafting node take every waiting prompt at once (up to the token budget), or one at a time? If it's one, batching them looks like a cheap fix inside this stack.
- The perf table is from e0f7d1e78 on main 9a50fdf, and the stack is now 22 commits behind main with 29 conflicting files, none of them new in this PR. Fine for now, but the numbers need a rerun at the merge head.
What we'd run on our H200 before approving: the real-weights module for both drafters (one GPU and TP2), the EAGLE-3 reference comparison above, a repro of the prefix case, and a throughput table (plain, EAGLE-3 k = 3, draft model k = 3 and 4, against vLLM) with the c32 split by walk.
| taps=self.taps, tap_key=self.tap_key, | ||
| kv=geometry.layer_set(self.kv_key, LAYER_SET), | ||
| carry=CarrySpec(cfg.hidden_size, CARRY_DTYPE), | ||
| # no chunk_boundary (UNDECLARED): the prefill reads the token sampled at each prompt's last row, so a |
There was a problem hiding this comment.
The reason chunked prompts are refused also applies to a reused prefix. A row's K/V at position p is built from the embedding of the token at p + 1, so the K/V in the last slot of a cached page depends on the token after the page, and the page's prefix key doesn't cover that token. Say request A cached pages 0..j, and request B shares those tokens but differs at the first token after page j. Or B extends A's prompt and A's length was a page multiple, so A's last row read A's own sampled token. B reuses page j, its prompt step starts at the next page, and every EAGLE-3 query of B attends a K/V built from A's next token. Output stays right and it needs the split to land on a page boundary, so it's rare for arbitrary prompts. But a shared system prompt whose length lands on one makes it systematic, and Qwen3's template drops <think> from history, so follow-up turns diverge right at the old prompt's end. vLLM drops the last matched block when EAGLE is on. Trimming a reused prefix by one page on a node whose drafter reads the next token (a DrafterNeeds field the KV manager reads) would match that. The repeated-prompt test in the real-weights module can't catch this, since an identical prompt recomputes its last page anyway.
|
|
||
| LAYER_SET = "eagle3" | ||
| LABEL = "main" | ||
| CARRY_DTYPE = torch.float32 |
There was a problem hiding this comment.
As far as I can tell _head returns the weight dtype, and propose casts the carry back to it, so the carry is a bf16 value stored as fp32. With the in-graph scatter the master is fixed at 4096 rows, so that's 64 MiB where bf16 would be 32, plus the per-slot rows. Is there a reason to keep it fp32?
| model: Qwen/Qwen3-0.6B | ||
| # one-row passes stay under the target's verify step to k = 4 (0.60-0.85x on one H100, | ||
| # benchmark/draft_model_cost/step_cost.py) | ||
| num_draft_tokens: 3 |
There was a problem hiding this comment.
Why k = 3 here? Going by this PR's own numbers k = 4 may be faster: tau 3.08 at k = 3 and 3.60 at k = 4 in the KV pressure rounds, against draft passes of 0.45-0.72x and 0.60-0.85x of the verify step. Ignoring the target's extra row, that's roughly 1.8-2.1 against 1.95-2.25 tokens per verify-step time. A c1/c32 pair at both values would pick it.
b5fc61b to
7da2d55
Compare
merceod
left a comment
There was a problem hiding this comment.
Read the new commit. EAGLE-3's chunked prompts read right: DRAFT_FOLLOWING is built from the whole prompt before any cut, cut with the ids at a chunk and at a cached prefix, and a prompt's last row still takes the sampled token.
The prefix-page bug from my last review is still there, and the new prefix test can't see it (inline). Now that ChunkBoundary.FOLLOWING says which drafters read the next token, it looks like the field the KV manager could read to trim a reused prefix by one page.
The description now contradicts the code in two places: "It declares no chunk boundary, so a target that chunks prompts is refused" and "Chunked prompts on EAGLE-3" under what this PR doesn't do. EAGLE-3's chunking also has no GPU run yet. The CUDA test the commit adds is the draft model's, and the TP2 pressure rounds ran before it.
Still open from last time: a check of the EAGLE-3 head against a reference forward, the draft model's throughput row and its k (inline), the fp32 carry (inline), and the perf table from e0f7d1e78.
| return prefill(self, inputs) | ||
|
|
||
| monkeypatch.setattr(Eagle3Drafter, "prefill", recorded) | ||
| prompt = list(range(1, 15)) |
There was a problem hiding this comment.
Both requests send this same prompt, and an identical prompt recomputes its last page, so this can't show the case from the eagle3.py:153 thread. That needs B to share A's cached pages and differ at the token right after the last shared page: that page's last slot then holds EAGLE-3 K/V built from A's next token. With this harness's 4-token pages, A = 1..14 and B = 1..8 then four other tokens would do it: B reuses pages 0 and 1, and position 7 holds K/V built from A's token 9. Then compare B's drafter K/V at position 7, or its drafts, against a cold run of B.
Two more drafters for Qwen3-8B. EAGLE-3 (RedHatAI/Qwen3-8B-speculator.eagle3, k = 3) is one decoder layer over the streams entering three target layers, a carrying drafter that chains its passes over a 32,000-token vocabulary its d2t table maps back to the target's; its K/V is one more layer of the pool. The Qwen3-0.6B draft model runs a token behind the target, as a lagged drafter; it has the target's KV geometry, so its 28 layers join the pool's tensor. The configs qwen3_8b_eagle3.yaml and qwen3_8b_draft_model.yaml serve them. Brought in here, where EAGLE-3 first uses them, from the Qwen3.5 MTP work held out of this stack: `mstar/model/components/chained_draft.py` (`chain_drafts`, `kept_rows`, `at_last_rows`) and `AttentionManager.prompt_last_rows`. Against main: the MTP cases of the drafter-share test and the MTP real-weights test stay out with the MTP work, the share test's pre-plan key case runs on DSpark, and the docs keep both sides with no Qwen3.5 MTP drafter. These earlier commits of the PR are folded in here, since their changes touch lines this restack changed: the hosted protocols, ObserveInputs and the first carry for EAGLE-3 and the draft model, and the review's fixes and bound-drafter tests. Qwen3 declares each drafter as it declares DSpark's, a Draft on the state contract; test_qwen3_draft_port.py compares the specs, verify node and steps for all three drafters against the hand-written ones. The step-share test compares the engine's step against the target's declared state expanded, the verify walk marked as the hand-written target declared it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eads max_seq_len, and the cost benchmark's stand-in drafter takes the stack's interfaces The two configs' "read by nothing" comment on max_seq_len said what this stack made untrue: a verifying deployment is refused at startup if its KV pool holds a request to less. The draft-model cost benchmark's stand-in drafter subclasses HostedDrafter (the node now checks a drafter by class), takes prefill(PrefillInputs), drafts DraftInputs.num_draft_tokens, and drops kv_layers_unbound, which the stack removed; its declaration subclasses HostedDrafterDecl and drops a tap key nothing reads (the node reads needs.tap_key, which it leaves unset). Checked on a CPU-only host: the stand-in builds, is a Drafter by class, and drafts each row's anchor. The benchmark itself needs a GPU and was not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wen3 node stores its first carry On the real CPU engine with the Qwen3 target and a config-only EAGLE-3 checkpoint. A prompt step writes the drafter's K/V at the layer the pool placed its layer set at, past the target's layers, and every target layer holds what the same target writes without a drafter; the test fails if the attention callable drops the set's offset. A second test records what the drafter's prefill returns and reads the sampler's DRAFT_CARRY after the step: it holds that carry for each request, and the test fails if the scaffold drops the store. The EAGLE-3 CPU checkpoint is built by one helper the node test shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does Both kept self.k from the declaration, where the benchmark stand-in read DraftInputs.num_draft_tokens, the engine's to bind. The draft model's passes are its declaration's lag, built from the same k, which startup checks against the verify block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dtype keys of both levels were merged, so a top-level torch_dtype and a transformer_layer_config dtype that disagreed were refused though each level agreed with itself (a hypothesis for real checkpoints, which none here shows). The top level's keys are read if it names one, else the inner level's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EAGLE-3 declares a carry; with its prefill made to return nothing, the Qwen3 node's prompt step raises, as the scaffold now does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EAGLE-3 chains its passes with chain_drafts, and the serving guides' draft block lists dspark, eagle3 and draft_model: Qwen3.5's MTP head is held out of this stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As for DSpark: a verifying node never drops a request, so with no host pool every request it serves is guaranteed. EAGLE-3's config sets 512 host pages (about 9.3 GiB pinned at 148 KiB a token) and the draft model's 256 (about 8 GiB at 256 KiB a token). The sizes are defaults to tune per deployment, not measured choices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chain_drafts, kept_rows, at_last_rows and prompt_last_rows had no test that failed when they were broken: EAGLE-3 is their only user, and its verify path runs only in the GPU real-weights test. A new CPU module checks each against a plain reference. The chain is compared with a sequential per-row loop for k = 1..4, including the rows each pass keeps and the outputs and drafts it reads. kept_rows is checked for every accepted count, and prompt_last_rows on a padded packed step (padding rows write to the sink), on one-token rows and with no query offsets. EAGLE-3's prompt step had only a check that its carry is non-zero and stored, which holds whichever row it reads. The new case packs three prompts of different lengths and compares the carry with the head run over every row in the same step, taken at each prompt's last row, and the tokens it reads with the next prompt tokens and the sampled ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DSpark stores no carry, so on its prompt walk drafter_step leaves the sampler step as declared, and the test no longer covered the step it rebuilds with the drafter's loop-back writes. EAGLE-3 stores one, so running the test on it checks that the rebuilt sampler step keeps the step's segments, which the pre-plan key compares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EAGLE-3's notes and Qwen3's dtype rule pointed at the held Qwen3.5 MTP code, which this tree doesn't have; they now say what they mean. The fp32 screen's gates say where they were calibrated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EAGLE-3 carries state across steps and drafts over a token map, so its rows moving to the host and back under TP2 is a second case for the returning mark and the carry's staging, beside DSpark's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EAGLE-3's state at a prompt row reads the prompt's next token, which a chunk's last row doesn't hold, so it declared no chunk-boundary rule and a drafting Qwen3 node prefilled each prompt whole under it. It now declares the rule ChunkBoundary.FOLLOWING, and the node chunks its prompts. The scaffold gives such a drafter's prompt step each row's following token (DRAFT_FOLLOWING, a [tokens] input; -1 at a prompt's last row, which follows with the token the step samples). The input is cut with the ids, by a cached prefix as by a chunk (cuts_tensor_inputs keeps the prefix probe), and staged in the prompt step's captures. EAGLE-3's prefill reads it in place of rolling the step's own ids. A draft model's state at a row depends on the rows up to it, as the target's does; it already declared NONE, and its chunked prompts are tested on the GPU. Tests: a prompt prefilled in chunks of 1, 3 and 7 tokens leaves EAGLE-3's layer the same K/V at every prompt position, stores the same first carry, and gives the same drafts and the same reply. Each chunk's rows read the prompt's next tokens. A prefix hit still cuts an EAGLE-3 prompt and stores the cold request's carry. On CUDA, a draft model's chunked prompts give plain replies and its own dense chain's drafts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y steps EAGLE-3 now declares mixed passes (DrafterNeeds.mixed), so a synchronous Qwen3 node drafting with it runs mixed prompt-plus-verify steps, as a DSpark node does. - Propose: its chain runs over every token of the step (chain_drafts' mixed form). Each verify row's block places read its carry and draft as a verify step's do, and each pass keeps its row's block place. A prompt row's places are unread, and their writes are rewritten after the target. - After the target, one pass of its layer over the step's tokens is both a verify row's observe and a prompt row's prefill. A block place reads the token the verify gave. A prompt token reads its following one (DRAFT_FOLLOWING), and its prompt's last the token the row sampled. Each row's carry is the output at its last kept place, a prompt row's at its last token, which the sampler keeps from its last chunk. - A replay's padding row reads the last real token's place (MixedRows.at), so no write goes by a prompt or padding row's place: the chain's buffers and the observe's next ids send them to a sink (mixed_next_ids), and a prompt row's last place keeps its own sampled token. - The scaffold's mixed inputs and capture carry DRAFT_FOLLOWING for a drafter that reads it, and observe sees the step's tensors (ObserveInputs.tensors). The draft model lags its target (DrafterNeeds.lag) and declares no mixed passes, so its node keeps its prompts in steps of their own. Tests (test_qwen3_mixed.py), with the running rows on their first verify step and looped back. EAGLE-3 in a mixed step, against the late prompts' own prompt step, gives: - the same drafts for the running rows; - the same layer K/V and first carry for the late prompts; - the same carry for the running rows after their verify; - plain greedy replies. The draft model's node declares no mixed walk. And a padding row never writes the last prompt row's next token. Mutations caught: a padding row's sampled token written at its read place, the chain kept at the wrong place, the following ids dropped, a prompt row's carry kept at its first token, and the verify tokens left out of the observe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ronous and asynchronous, on one GPU and TP2 Qwen3-8B with each drafter. Its prompts are sent a few verify steps apart, so each lands while the earlier replies decode, then sent again as prefix-cache hits. DSpark and EAGLE-3 run them inside their verify steps, counted by the LLM node's mixed walk (mstar_step_seconds). The draft model lags its target and declares no mixed walk, so it has no case. - **Synchronous:** the replies hold to plain greedy under the margin rule, and the acceptance stays within 10% of the asynchronous node's. Under a mixed-step budget that leaves a prompt a page a step past the verify rows' blocks (the combined walk's, which its prompt walk's steps run under too), every prompt is chunked across steps, mixed ones among them. - **Asynchronous:** a mixed step's verify rows run on the resident tables, and the worker speculates prompt rows into a verify chain. The replies hold to plain greedy, mixed steps ran, and the acceptance holds to the node's at once. - **TP2, asynchronous, DSpark and EAGLE-3:** the replies pass the fp32 screen against plain, mixed steps ran, and acceptance is within 10% of one GPU's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ke them back The pressure rounds' host pool holds four times the device's pages, so no request turns guaranteed and the pool's page lending never runs. A variant gives the host 16 pages, far below what the rounds set aside at their peak. The host fills, requests turn guaranteed, and they lend what they haven't written of their reservations, the verify slack included. It asserts that guaranteed requests took back pages they lent (mstar_kv_take_backs_total, each a move of a request that borrowed them) and prints the most pages lent at once (the mstar_kv_pages_lent gauge, which reports each second and so can miss a short borrow). It also asserts that nothing was dropped, and that the replies pass the fp32 screen against plain under the same pressure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its attention passed the layer's neutral Llama 3.1 factors, which select FlashInfer's Llama 3.1 launcher; that launcher has no head-parallel grid, so a step's few tokens ran on a handful of blocks (16.7 us a launch at 128 rows, 4.6 us on the head-parallel grid). The drafter now passes none, so the plain launcher runs, with the same frequencies at these factors. The config refuses a checkpoint whose RoPE scales, and the drafter declares plain RoPE (ModelGeometry.plain_rope), so startup refuses positions that scale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An arrival that hits the prefix cache recomputes at most 31 tokens past its last full page instead of 127, so a mixed step's prompt rows fit its +32 rung, and FA3's paged kernel never reads past a row's length at this size. The token capacity is the same as 1024 pages of 128. The max_seq_len comment now says what the value is and what startup refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t page's key EAGLE-3's K/V at position p is built from the token at p + 1, so the K/V in a page's last slot depends on the next page's first token, which the page's own prefix key doesn't cover. A request that shared a cached page and differed right after it read another request's last-slot K/V. Trimming the last matched page isn't enough: the index keeps a key's first writer and files later pages under it, so a multi-turn prefix can still hand an interior page filed for another next token. A drafter declaring ChunkBoundary.FOLLOWING now sets the pool's derived PagedKVConfig.names_ahead to 1, and the index names page i by key i + 1, which chains every token through the next page's first. One helper names pages at every site that pairs a page with a key (the lease, a follower's adopt, host copy-in, P/D read-in, the admission and lending estimates, the offload's keys and the reload estimate), one helper caps a lease, and the routing mirror follows. A page is filed once its successor's key covers a whole page. chain_key and chain_keys stay unshifted for recurrent pools, and a FOLLOWING drafter beside a recurrent pool is refused, since its snapshots are keyed by page. A non-page-multiple exact repeat now reuses one page fewer under EAGLE-3. With names_ahead 0, every path is as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…selector It named the KV, positions and attention its passes select and bound them itself; DraftInputs.select_pass, which the engine binds from the lag's resources, does that for every lagged drafter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amed ahead The mirror sliced every request's key list at names_ahead, copying it at 0 too. It slices only where pages are named by a later key, as KVManager._names does, so a pool with no EAGLE-3 drafter keeps main's routing work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ache or the mirror An image end's page is filed under the key of the page before it, unshifted, while a pool an EAGLE-3 drafter joins names its pages a page ahead; the mirror shifted them. EAGLE-3 is text-only, so no prompt reached that, but nothing said so. KVManager._root keeps no item ends on such a pool, the filing pass skips one, and the mirror counts none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and a prefix reused from the host, on the CPU A CPU Core module over the CPU harness, with CPUPagePool on unpinned memory copying synchronously; the EAGLE-3 pool has one storage, so what names_ahead changes is host-side. A is offloaded right after its prompt step, since its prefix chain goes at its first verify step, and then reloaded with reload_request: - A's drafter K/V once back is the bytes it had on a run that stayed on the device, and its reply the same; the offload files A's two whole pages. - B, which shares A's first two pages, arrives while A is on the host with the device index emptied. Its readiness is checked until its host copy lands, and it reuses page 0 alone, copied from the host, as it does from the device. Its drafter K/V, and A's, match the run where A stayed. With _offload_keys or _fetch_from_host reverted to unshifted names, the second case fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de's resources The identity test hands ARNodeSubmodule a DrafterShare it built with the selector set, so dropping the engine's select_pass binding passed every CPU module. On the CPU harness's real engine load, the draft model's share carries a selector that selects the pass on the KV, positions and attention the lag names among the node's resources; DSpark and EAGLE-3 get none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
models.rst's EAGLE-3 paragraph (and its mintlify copy) now says its arrivals run inside its verify steps, as DSpark's do, so its prompts chunk at the mixed budget: 128 verify blocks plus 512 tokens, 1,024 at k = 3, capped asynchronously at the widest captured mixed bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e device, and a reload from the device index The offload module emptied the device index before every lookup and reload, so two names_ahead paths ran in no test: a host copy that continues a prefix still on the device (its parent is the shifted name of the last device page), and a reload that takes a request's keyed pages back from the device index. It also left out the host write-backs of cached pages, which file them under the device index's own names with no offload at all. Three cases: - A offloaded and its page 1 alone evicted: B reuses page 0 from the device and page 1 from the host, as it reuses both while A stays, with the same drafter K/V and reply. - A offloaded and reloaded with its pages still indexed: the offload's keys find those pages, and the reload takes them back. - No offload: A finishes, its cached pages are written back through after_output_copy and landed, the device index is emptied, and B reuses page 0 from the host as from the device. The runner's result names B's reuse as one tuple, which the cases share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the CPU attention needs raising=False The binding test checked what the bound selector does, so an engine that bound a callable building a new selector on every call passed it. It now records what pass_selector returns while the engine loads and checks that the drafter share holds that function, and that a drafter with no lag builds none. The monkeypatch of select_pass says why it doesn't raise: the CPU harness's attention defines none of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aa18f98 to
0b0f1b6
Compare
PR 7 of 7 in the stack, the top, stacked on #1324. Head 0b0f1b6, base main 2c8ffde.
Adds two drafters for Qwen3-8B: EAGLE-3, a carrying drafter over a reduced vocabulary, and Qwen3-0.6B as a separate draft model one token behind the target.
What it does
mstar/model/components/eagle3.py,configs/qwen3_8b_eagle3.yaml,RedHatAI/Qwen3-8B-speculator.eagle3,k = 3): one decoder layer over the streams entering target layers 2, 18 and 33, chainingkdrafts over a 32,000-token vocabulary mapped back throughd2t, with its pre-norm output as the carry. One layer joins the target's pool as layer set"eagle3". Both checkpoint formats parse; unsupported options are refused.k = 3).iin the prefix index by the key of pagei + 1(PagedKVConfig.names_ahead, derived fromChunkBoundary.FOLLOWING). That holds at every site that leases, files, offloads, reloads or reads in a page, and in the conductor's routing mirror, so a reused page always holds the drafter K/V of its own following token.Engine.step_finishedincluded; that the worker's step loop does the same is read from the code, not run).HostWrites, queued when the index becomes a page's one owner and launched after a step's output copy), which file each page under the index's own name (arena.pyqueuesindex.key_of(page)), so they are named alike. Such a pool caches no image ends (EAGLE-3 is text-only), and FOLLOWING beside a recurrent pool is refused.mstar/model/components/draft_model.py,configs/qwen3_8b_draft_model.yaml,Qwen/Qwen3-0.6B,k = 3): its own 28 layers and attention binding as layer set"draft_model"of the target's pool, lagged one row behind the target in one-row passes (feat(engine): support drafters with their own KV geometry, a lag behind the target and a token map (PR 6 of 7 in the stack) #1324), selecting each pass through the selector the engine binds (DraftInputs.select_pass). It runs no mixed steps. Refused: other families, vocabularies or tokenizers, andkabove 4.ChunkBoundary.FOLLOWINGand theDRAFT_FOLLOWINGprompt input (draft/drafter.py,ObserveInputs.tensors), whichSampledTokenDecoderbuilds and cuts with the ids;ParallelAttention.output_hidden_sizefor EAGLE-3's 2H-wide layer.chained_draft.pyandAttentionManager.prompt_last_rows;Qwen3Modeltakesdspark,eagle3anddraft_model;benchmark/draft_model_cost/(step_cost.pyset the draft model'skcap); docs inmodels,servingandadding_models(.rstand.mdx).No new dependencies or environment variables.
Since the last push (aa18f98): review fixes, each with its tests and docs.
test_qwen3_draft_model_device.py: the self-drafting case's full-block check counts only blocks that fit in the reply, and the plain check requires a reply as long as the plain one (Tests).Performance
Measured on 0023eff82b, an unpushed earlier state of this head that differs from it only in three test files (
test_spec_mixed.py,test_qwen3_dspark_device.py,test_qwen3_draft_model_device.py).Method: one H100 80GB per server;
benchmark/qwen3/bench_openai.py(64 coding chats, greedy,ignore_eos, 256 output tokens, 5 waves); M* at-O2on torch 2.12.1+cu132, asynchronous, a fresh Inductor cache per boot; vLLM 0.33.0 on torch 2.14.0+cu130 with the same checkpoints andk, its τ read off its spec-decode counters by M*'s definition; driver 580.173.02; 3 rounds, medians, arms rotated across GPUs (2026-11-18).Noise: the host was shared with other GPU jobs (load average 2.6-16.4), so differences of a few percent are within it.
Page size (EAGLE-3), at 0023eff82b, 3 rounds: 32-token pages against 128 measured +1.2% / +6.7% / +12.7% at c16 / c32 / c64, τ unchanged. It was measured with names_ahead on, on the benchmark's coding chats; the gain is from prefix hits, as on DSpark (#1321).
Against aa18f98 (the pushed head, before names_ahead), both on 32-token pages, it is level at c16 and 2.2% / 1.5% lower at c32 / c64, the rounds not overlapping. That is a cross-process A/B on a shared host, so a lead, not a measured regression; recorded, not tuned.
At aa18f98, with no prefix reuse, 32 against 128 was −0.8% to +0.8% at c1-c64. The draft model runs no mixed steps and stays on 128.
Tests
On an 8 x H100 80GB host (two GPUs for TP2), torch 2.12.1+cu132, FlashInfer 0.7.2, FA3 3.0.1; commits are named per row. aa18f98 is the pushed head.
0023eff82b and dbc627e8f7 are unpushed earlier states of this head: dbc627e8f7 differs from it in
test_qwen3_draft_model_device.py's plain check and comments oftest_qwen3_dspark_device.py, and 0023eff82b also intest_spec_mixed.pyand both device modules' fixes.CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) +test_tp_async_lockstep_sim.py(-n 3)ruff check .(0.17.5, 0.16.0)mstar_rust;-n 6,OMP_NUM_THREADS=2), failing tests by id against main's on the same hosttest_token2wav.pyin one process); the 13 main also fails on this host (arm64 environment failures), plustest_tp_async_lockstep_sim::test_b2_void_holds_all_scenarios_2_and_3_ranksat the 120 s limit under load (its module alone: 16 passed, that case in 21 s)-n 3test_qwen3_dspark.py::test_the_shipped_configs_name_the_rfc_drafter,test_qwen3_draft_port.py,test_state_lint.py), CPU onlytest_qwen3_draft_model_device.py,test_draft_lag.py, 1 x H100test_qwen3_dspark_real_weights.py, one GPU:[eagle3]/[draft_model]/[draft_model_k4][eagle3]/[draft_model][eagle3]/[draft_model]and[draft_model_k4][draft_model]and[draft_model_k4][dspark][eagle3]on a test-only copy at 32-token pages (64 pressure pages, 8 lending host pages)The real-weights module:
The prefix naming:
test_qwen3_chunked_drafters.py,test_drafter_layer_sets.pyandtest_conductor_replica_assignment.pyfail before it.test_qwen3_eagle3_offload.py(CPU Core;CPUPagePoolon unpinned memory, copying synchronously, since the EAGLE-3 pool has one storage and what names_ahead changes is host-side):_offload_keysor_fetch_from_hostreverted to unshifted names._offload_keysunshifted._fetch_from_hostunshifted.test_qwen3_draft_model_device.py(CUDA) checks a toy draft model of another KV geometry against a dense float64 chain.kcap dropped (a scratch copy), atk= 6 the last block can't hold 6 drafts, which failed it (12 of 16) though every draft was exact; it now passes atk= 6 and 5.k= 3, where every block fits, that check fails with the draft model's own weights (0 of 84 full blocks).NodeDraft.cutstopping each reply one token short (a scratch mutant), the module passed before this check (10 passed at dbc627e8f7) and fails all 10 cases after it.Other modules:
test_drafter_share.pychecks the engine binds the draft model's pass selector over the node's resources at load, and that what it binds is the functionpass_selectorbuilt then (a callable that builds one per call fails it). CPU modules includetest_eagle3_config.py(with a CUDA case: plain and neutral Llama 3.1 RoPE rotate bit for bit),test_draft_model_method.py,test_chained_draft.py,test_qwen3_mixed.pyandtest_qwen3_chunked_drafters.py.Open review items
Coverage across the stack
[dspark]one GPU and TP2,[eagle3]and the draft model's cases on one GPU, the draft model's at TP2. EAGLE-3 at TP2 last ran at aa18f98.gsm8k.py).Follow-ups
KVManagerunder a retention policy (it neither models the manager's front release nor caps only on a step-declared policy); no keyed label carries retention today, so it is latent. A follow-up against main.prefill_cost.py(it ships unrun).AI assistance was used (Claude Code); the tests run and their results are listed above.
🤖 Generated with Claude Code