Repository navigation
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
merceod
left a comment
There was a problem hiding this comment.
The model is clean against invariant 1. It declares its four resources, captures through the engine, and reuses
Qwen3DenseLM and TokenStepNode, so the decoder itself stays in one place. Startup refusals, stop ids and the tied
head path all check out.
The main question is design. Why not MultimodalDecoderModel plus SampledTokenDecoder with no media? Both are
written to take a text-only model. media defaults to (), SampledTokenDecoder.__init__ says a text-only node
passes no encoders, and its docstring names Qwen3DenseLM as a decoder it drives. Here get_graph_walk_graphs is
media_walks(max_iters, ()) with prefill_text renamed, and the KV label, loop caps, forward-pass args, generation
defaults and sampler spec repeat the scaffold. What the scaffold has that this node doesn't: the mixed prefill and
decode step, a rebuild policy for a dropped request, and decode capture up to 128 rows. The stack then teaches both
nodes to verify (#1321 changes qwen3/submodules.py, #1323 changes sampled_token_decoder.py). If DSpark or EAGLE-3
taps need something the scaffold can't give, please say what in the description. Otherwise moving this onto the
scaffold now is much cheaper than after six PRs build on it.
The parity gate uses a chosen 0.25 margin with no scale next to it. What does transformers bf16 against fp32 give
under the same rule on these 8 prompts? And one control that has to fail (say rope_theta off by 10%) would show the
rule can tell a wrong model from a right one.
No competitor numbers is fine for a stack base if the description says so. vLLM and SGLang both serve Qwen3-8B,
though, so one c1/c32 row would be cheap.
| ids = ids.reshape(-1).to(torch.long) | ||
| n, budget = ids.numel(), self.get_max_output_tokens(**kwargs) | ||
| # in the data worker, a ValueError reaches the client as a 400 | ||
| if n == 0 or n + budget > self.max_seq_len: |
There was a problem hiding this comment.
With no max_output_tokens in the request, budget is the 2048 default, so a prompt within 2048 tokens of max_seq_len (30721 to 32767 on the shipped config) is a 400 here. The conductor would serve it. With no length kwarg it runs _fit_loop_caps and shrinks the reply to the room left, and the KV config's validate already refuses a request that really doesn't fit. The OpenAI adapter sends no default max_tokens, so this is the normal path for a client that leaves it out. I'd keep the empty and vocabulary checks and drop the length one.
There was a problem hiding this comment.
Agreed. The check goes: _fit_loop_caps shrinks the reply to the room left, and KVReqConfig.validate refuses a prompt that can't fit. The scaffold has no such check, so this goes with the port. I'll add an intake test: a prompt four tokens short of max_seq_len with no max_tokens is served with its reply cut to the room left, and one past max_seq_len gets a 400 for its length.
| # deployment can't honour (a 400) rather than it being dropped here | ||
| params = {key: kw[key] for key in _REQUEST_KNOBS if key in kw} | ||
| # the checkpoint's generation_config.json supplies what the request leaves out | ||
| defaults = {"temperature": 0.0} if self.generation.get("do_sample") is False else self.generation |
There was a problem hiding this comment.
load_generation_defaults in mstar/model/utils.py already reads generation_config.json, including do_sample: false as greedy, max_new_tokens and min_p. Can this use it? As written, do_sample: false drops every other default the file sets.
There was a problem hiding this comment.
Yes. On the scaffold the defaults come from load_generation_defaults, min_p included, with enable_min_p following it. This goes with the port.
| return {"input_ids": torch.cat([inp.input_ids for inp in inputs])} | ||
|
|
||
| # Keyed by request id, so outside dynamo (the sampler runs eager either way). | ||
| @torch.compiler.disable |
There was a problem hiding this comment.
SamplerResource.sample is already @torch.compiler.disable, so this wrapper adds nothing. Calling the resource directly is enough.
There was a problem hiding this comment.
Right, it's redundant. The scaffold calls the resource directly, so this goes with the port.
| ) | ||
| # the node's capabilities for the generation-aware knobs, as its ingest checks them | ||
| # (`SamplerResource._generation_controls_problem`), refused here, before dispatch | ||
| if (self.repetition_window or 0) > spec.max_repetition_window: |
There was a problem hiding this comment.
These three repeat SamplerResource._generation_controls_problem. One helper both call would keep the two from drifting. Also, since these knobs skip normalize_generation_kwargs, a string from /generate ("min_tokens": "4") raises TypeError here, which the conductor turns into a 500 rather than a 400. I know the type checks are your listed follow-up, just flagging the status code.
There was a problem hiding this comment.
Agreed on both. With min_tokens gone and Qwen3 on the scaffold, nothing in this PR forwards min_tokens, repetition_window or the top_p_* knobs any more, so the checks this PR added here, and the 500 a string could trigger, go away with the port rather than becoming a shared helper.
merceod
left a comment
There was a problem hiding this comment.
The new commits are the restack onto #1318's new head and test_plain_model.py reading GpuStepBody, so the points from last round still stand as written. We ran it on one H200 this time (torch 2.10.0, FlashInfer 0.7.2, Triton 3.6.0), at this head and merged with main c12ba47.
Tests: test_qwen3_real_weights.py 2 passed with both checkpoints cached, nothing skipped. mstar/model/qwen3/tests 20 passed, and the startup check with an empty HF cache passed.
Parity, 20 prompts (your 8, 8 more, a ~1.1k-token document, 3 with thinking on), 256 greedy tokens, first divergence against transformers fp32 under your margin rule:
| Run | Identical to fp32 | Diverge | Fail the 0.25 rule | Largest margin at a divergence |
|---|---|---|---|---|
configs/qwen3_8b.yaml |
7 | 13 | 0 | 0.172 |
same, accelerator_graph: false |
9 | 11 | 0 | 0.192 |
| merged with main | 7 | 13 | 0 | 0.172 (ids equal to the head's) |
| transformers bf16 | 8 | 12 | 0 | 0.215 |
rope_theta x1.1, same weights |
7 | 13 | 2 | 0.331 |
So the served model sits inside the spread transformers bf16 has against its own fp32, which answers the scale question. The negative control is the weak spot. A 10% rope_theta error passes 18 of 20 prompts and is token-identical to fp32 on 7. On your test's own 8 prompts at 128 tokens it trips on one prompt (token 88, margin 0.33), so the gate can fail, but only just. Comparing against the bf16 spread, or a teacher-forced check of the served tokens' log-probs against fp32, would catch an error like that with room to spare. Not blocking.
Throughput, bench_openai.py from #1321, 256 tokens ignore_eos, 3 boots, output tok/s median (spread under 1% across boots): c1 212.7, c16 2318, c32 4380, c64 6743. Your H100 rows are c1 163.2, c16 2009, c64 4975, so this is in line for the bandwidth step. A warm boot of the shipped config is about 40 s.
Functional on the shipped config: max_tokens 1, 2 and 3 give 1, 2 and 3 tokens, two identical requests give identical ids, an aborted stream frees its slot (the next request served in 0.15 s), min_tokens: 4 is a 400 and serves with serve_min_tokens: true, an 8 x 8k-token burst serves on the captured and eager configs, and a multi-turn chat works. Two rows confirm last round's comments. A 31,321-token prompt with no max_tokens gets the 400 from qwen3_model.py:245 ("prompt (31321 tokens) + max_output_tokens (2048) must fit max_seq_len (32768)") and serves with max_tokens: 512. "min_tokens": "4" on /generate is a 500 (TypeError: '>' not supported between instances of 'str' and 'int').
The design question from last round is still the one that decides where the rest of the stack attaches. I'd like an answer on it before approving.
46aaed8 to
b9b8452
Compare
|
/review |
| from mstar.model.base import ( | ||
| ForwardPassArgs, | ||
| KVLabel, | ||
| LabelLayout, | ||
| Model, | ||
| PrefixKeys, | ||
| ProcessPromptOutput, | ||
| Span, | ||
| ) |
There was a problem hiding this comment.
[P0] The model is written against a Model API that main has since replaced, so merging it breaks mstar.model.qwen3 at import.
On current main, mstar.model.base no longer exports ForwardPassArgs (it lives in mstar/conductor/conductor.py), and CurrentForwardConductorMetadata (imported at R14) doesn't exist at all. Model.walk_plan is now abstract (mstar/model/base.py:972) and replaces get_initial_forward_pass_args / get_partition_forward_pass_args (R199-225), which nothing calls any more. The conductor also calls get_request_resource_configs(input_modalities, output_modalities, model_kwargs) (conductor.py:766), while R161 takes (partition_fwd_args, model_kwargs). None of these hunks conflicts textually, so the merge looks clean. Afterwards, import mstar.model.qwen3.qwen3_model raises ImportError. That fails CPU Core's test_registration.py, test_plain_model.py and test_startup_graph_validation.py[qwen3_8b], and mstar serve qwen3_8b won't start. Fix: rebase, then implement walk_plan (prefill, then WalkStep(DECODE_WALK, bind={"text_inputs": "new_token"})) and the new get_request_resource_configs signature. MultimodalDecoderModel already has both (see the design question). How to confirm: on main 156d2fc with this diff applied, python -c 'import mstar.model.qwen3.qwen3_model'.
There was a problem hiding this comment.
Moot after the rework. Qwen3Model subclasses MultimodalDecoderModel and inherits walk_plan and get_request_resource_configs; nothing in mstar/model/qwen3 imports ForwardPassArgs or the removed forward-pass hooks. The head is on main 9a50fdf, and a merge-tree against current main conflicts only in test/modular/test_startup_graph_validation.py, which the next rebase resolves.
| accelerator_graph: bool = True, | ||
| prefill_token_buckets: list[int] | None = None, | ||
| ): | ||
| super().__init__(prefill_token_buckets) |
There was a problem hiding this comment.
[P0] super().__init__(prefill_token_buckets) raises TypeError every time the node is built.
The MRO is Qwen3LMSubmodule → TokenStepNode → ARNodeSubmodule → NodeSubmodule. Neither of the first two defines __init__, so this calls NodeSubmodule.__init__(self) with an extra positional argument, even when the value is None. Both make_submodule on any worker that has weights (every served qwen3_8b) and the CPU Core tests test_every_output_of_a_decode_step_stays_on_the_device and test_an_eager_decode_step_reads_its_routed_tokens fail. CI installs triton, so importorskip doesn't skip them. Even when fixed, the value never reaches the ladder, because R95 reads the class's PREFILL_TOKEN_BUCKETS. The engine already replaces a packed capture's ladder from the serving config's prefill_token_buckets ("a model plumbs nothing", docs/adding_models.rst:2429). So drop the prefill_token_buckets kwarg from Qwen3Model and from this class, and correct the docs rows that advertise model_kwargs.prefill_token_buckets. How to confirm: Qwen3LMSubmodule(None, [96]) raises TypeError: NodeSubmodule.__init__() takes 1 positional argument but 2 were given.
| super().__init__(prefill_token_buckets) | |
| super().__init__() |
There was a problem hiding this comment.
Moot after the rework. The node is a SampledTokenDecoder subclass with no constructor of its own, and the prefill_token_buckets model kwarg and its docs rows are gone. The ladder is the scaffold's, and a deployment sets it through the serving config's prefill_token_buckets.
| thinking = kwargs.get("enable_thinking", (kwargs.get("chat_template_kwargs") or {}).get("enable_thinking")) | ||
| extra = {} if thinking is None else {"enable_thinking": bool(thinking)} |
There was a problem hiding this comment.
[P3] enable_thinking sent under chat_template_kwargs is read by truthiness, so {"chat_template_kwargs": {"enable_thinking": "false"}} turns thinking on.
normalize_generation_kwargs checks only a flat enable_thinking as a bool (its comment says why: "false" is truthy in a template). TextChatAdapter passes chat_template_kwargs through unflattened, so the nested value reaches bool(thinking) unchecked. Refuse a non-bool here with check_bool_knob("enable_thinking", thinking). Alternatively, flatten the template kwargs at the adapter, as Qwen3_5Adapter / Rust template_kwargs_chat do, so the existing check covers them.
There was a problem hiding this comment.
Fixed in the rework. enable_thinking, flat or under chat_template_kwargs, must be a boolean or the request is a 400. A chat_template_kwargs that isn't an object is a 400 too. Tests cover both.
| single_request_inputs=ARNodeInputs( | ||
| input_ids=torch.zeros(1, dtype=torch.long, device=device), input_seq_len=1, | ||
| ), | ||
| capture_batch_sizes=self.DECODE_CAPTURE_BATCH_SIZES, |
There was a problem hiding this comment.
[P3] perf Decode inherits TokenStepNode.DECODE_CAPTURE_BATCH_SIZES (up to 64). The engine caps a decode step at its largest capture, so above 64 concurrent requests every decode pass splits in two. The shared SampledTokenDecoder captures up to 128.
Est.: Qwen3-8B is about 16.4 GB of bf16 weights, about 4.9 ms per step at 3.35 TB/s on H100. Decode at these widths is weight-bandwidth-bound (GPU-bound, from the arithmetic, not measured). At c=128, two 64-row steps re-read the weights where one 128-row step would read them once, so up to ~1.6-1.8x decode tok/s. Probability medium (~50%): it doesn't pay if per-step CPU overhead or the 1024-page KV pool (131k tokens) binds first. Expected: gains only at concurrency above 64, none below. Difficulty S: set DECODE_CAPTURE_BATCH_SIZES = [1, 2, 4, 8, 16, 32, 48, 64, 96, 128]. Confirm with benchmark/qwen3/bench_openai.py at c=128 and c=256, A/B on this ladder.
There was a problem hiding this comment.
Moot after the rework. The node inherits SampledTokenDecoder's decode ladder, which goes to 128 rows. I haven't measured the effect above c=64, so the description makes no claim about it.
| body = GpuStepBody.__new__(GpuStepBody) | ||
| body._graph_runtime = SimpleNamespace(supports_inline_emit=True) | ||
| body._device_loopback = {} |
There was a problem hiding this comment.
[P3] Nit: the test builds GpuStepBody with __new__ and sets its privates _graph_runtime and _device_loopback by name. A rename of either breaks the test with no grep hit at the definition. Meanwhile the real constructor can change what all_on_device reads without this test noticing. AGENTS.md's Testing section asks for the smallest real object instead: make_worker in mstar/testing_utils/worker.py builds a real step executor over the collaborators the test names.
There was a problem hiding this comment.
Agreed. I'll build the step body through make_worker, the way test_worker_device_loopback.py does, so the test goes through the real constructor.
This comment has been minimized.
This comment has been minimized.
|
No live test yet: fix what the review left open first (open P0–P2 findings: mstar/model/qwen3/qwen3_model.py:30 (P0), mstar/model/qwen3/submodules.py:70 (P0), mstar/model/qwen3/submodules.py:91 (P2)), then comment |
|
Re the reviews of Nov 8 and Nov 9: Thanks for both rounds and for the H200 runs. Design. You're right, and nothing in the stack needs a separate node. The hidden-state taps for DSpark and EAGLE-3 go through So Qwen3 is now on #1323, Qwen3.5 MTP, is held. Main's State Contract (#1227, #1232) moved Qwen3.5 onto declared state reached through handles, and #1323 was written against the old resource path, so Qwen3.5 MTP lands later on While a node verifies, the scaffold turns off combined walks, rebuild and the mixed split, because the engine refuses a verifying node in a combined walk for now. On the scaffold, plain Qwen3 should pick up the mixed prefill/decode step, the rebuild policy and larger decode captures, so I'll re-measure throughput and capture memory for both plain and drafted configs, not only parity. Current head. b9b8452 fails CPU Core, and it no longer builds against current main: #1184 removed the Parity gate. Thanks for the bf16 spread and the
On one H100 at this head, the real model scores R 0.911 with 0 flips against bf16's 2 on the short set, and R 0.962 with 5 flips against 7 on a long set of three prompts of 1.3k to 3.4k tokens, so it passes both checks on both sets.
Competitors. The description now has a c1/c32 vLLM row for Qwen3-8B. |
|
Re the automated review of Nov 10: Outside the diff, Description, the stale head. Right, the description was stale. The rework onto the shared decoder scaffold is pushed as 4d22dcb, and the description names that head and only what ran on it. Description, the CPU Core claim. The constructor bug is gone in the rework. CPU Core on that head is in the description. Description, Description, the sampler knobs. Moot after the rework. The SamplingReqConfig change is no longer in this PR, and the model forwards only the scaffold's sampling keys. The description will drop that section. Design question, why not the scaffold. Agreed, and done in the rework. Qwen3Model subclasses MultimodalDecoderModel and the node subclasses SampledTokenDecoder, declaring only its attention key, layer steps and resident walks. The walk plan, mixed steps, rebuild policy, TP fusion init and 128-row decode ladder come from the scaffold. PR-wide question, where speculation lands. On the shared one. In the reworked stack, #1321 adds the verify path to SampledTokenDecoder, and the Qwen3 node only connects its drafter. The drafter is bound through Description, the provenance history. Agreed. The next description leads with what ran on the pushed head, GPU included or explicitly not, and drops the history of earlier heads. |
|
/review |
| checkpoint's ``generation_config.json``. A request is bounded by the checkpoint's | ||
| context: one that asks no length decodes to the room its prompt leaves, and a prompt | ||
| past it is a 400. Rope scaling, sliding windows and attention biases are refused at |
There was a problem hiding this comment.
[P2] The docs say a request that asks no length "decodes to the room its prompt leaves". It actually stops at 2048 tokens.
Neither Qwen3Model nor MultimodalDecoderModel overrides get_max_output_tokens, so a request with no max_output_tokens gets Model's MAX_OUTPUT_TOKENS = 2048 (mstar/model/base.py:150). Conductor._fit_loop_caps only shrinks that cap when the context has less room. It never raises it. A 100-token prompt with thinking on and no max_tokens therefore ends at 2048 tokens, not near 40k. sampling_defaults drops a generation_config max_new_tokens, and Qwen3-8B's file sets none anyway. The same sentence is in docs/mintlify/models.mdx:178 (invariant 9 wants both fixed). Either describe the 2048 default, as below, or override get_max_output_tokens to return the room the context leaves when no length is given, as GLM-5.2 does with context_limit(). The second fits Qwen3's long thinking replies, but it changes the default every client gets. How to confirm: send /v1/chat/completions with a short prompt, thinking on and no max_tokens. The reply ends with finish_reason: length at 2048 completion tokens.
| checkpoint's ``generation_config.json``. A request is bounded by the checkpoint's | |
| context: one that asks no length decodes to the room its prompt leaves, and a prompt | |
| past it is a 400. Rope scaling, sliding windows and attention biases are refused at | |
| checkpoint's ``generation_config.json``. A request that asks no length decodes at most | |
| 2048 tokens, fewer when its prompt leaves less room in the checkpoint's context, and a prompt | |
| past it is a 400. Rope scaling, sliding windows and attention biases are refused at |
Automated review🟡 1 P2 on Both P0s (the import break on main's Earlier findings: 5 fixed, 0 fixed above, 0 open, 0 disputed, 1 moot. Description
Design
What the review checked
Live test: full (correctness, performance). No live test has passed on this PR, and the model was rewritten onto the shared scaffold, so the whole served path (prefill, mixed and decode captures, eager fallback, both frontends) is new at this head. The live-test job posts its outcome in a separate comment. What's left for you
Approve if you're OK with these. The live-test comment adds its own items. Any approval is posted by the live-test job, by the rule in |
|
No live test yet: fix what the review left open first (open P0–P2 findings: docs/models.rst:642 (P2)), then comment |
4d22dcb to
8b10155
Compare
merceod
left a comment
There was a problem hiding this comment.
Served the shipped configs/qwen3_8b.yaml at this head, merged with current main, on one H200, through the Python frontend and through --rust-frontend. Everything I tried works.
| Check | Python frontend | --rust-frontend |
|---|---|---|
| boot | 250 s cold, CapturePlan 98 of 98 buckets | 64 s (warm compile cache) |
your 8 prompts, greedy chat, max_tokens 256 |
8 x 200, usage and finish_reason on each (7 stop, 1 length at 256) |
same, text byte-equal to the Python frontend's |
max_tokens 1 / 2 / 3 |
1 / 2 / 3 tokens, length |
same |
one streamed chat with include_usage |
67 chunks, text and usage equal to the non-streamed reply, stop, [DONE] |
same |
| 16 chats at once | 16 x 200, usage and finish_reason on all 16 |
same |
40,912-token prompt, no max_tokens (48 positions left) |
200, 49 tokens, length |
same |
| 40,976-token prompt | 400, "needs 40977 positions of 'main', past the model's context of 40960" | same |
| client closes a stream after 5 chunks, then a chat | next chat 200 in 0.15 s | same |
Against transformers' bf16 Qwen3-8B greedy on the same 8 rendered prompts (256 tokens): 5 of 8 are token-identical. The other 3 split at near-ties, with fp32 top-2 margins of 0.026, 0.048 and 0.114 nats, and bf16 has the two tokens tied at all three spots. fp32 sides with the served token on 2 of the 3. Both frontends return the same ids, and every chat text equals its decoded ids.
The docs line from the bot's last pass is still the one thing open. A short prompt with no length and ignore_eos stops at 2048 tokens here, so "decodes to the room its prompt leaves" only holds when that room is under 2048. Fixing the sentence is enough.
8b10155 to
12b7516
Compare
12b7516 to
02d4ae4
Compare
Qwen3Model is a MultimodalDecoderModel with no media and Qwen3LMSubmodule a SampledTokenDecoder, as Ripple-65B is. It is registered as qwen3_8b and served with mstar serve and the OpenAI route. The scaffold owns the walks (prefill_text, the mixed step, decode), the walk plan, the decode loop's cap, rebuild, the sampler's request config and the stop check. The model keeps its config refusals, the union of every EOS the checkpoint names, its resources, and the chat template's rendering: enable_thinking flat or in chat_template_kwargs (checked to be a boolean; a chat_template_kwargs that isn't an object is a 400), a message's text parts joined by newlines, rendered to text and encoded through encode_ids as Ripple and Qwen3.5 do. chat_template.jinja is fetched with the tokenizer's files. Decode reads its rows off the resident tables. The KV pool holds the checkpoint's context; the configs say the top-level max_seq_len is required and unread. Tests: registration; request_kwargs' contract; a conductor intake test on a 4096-token context (four decode steps and five tokens for a prompt four short of it, the context's 400 for one past it); sampling defaults from generation_config.json with min_p turning its filter on; the startup-validation factory builds the model through its own __init__ with the config's model_kwargs; the decode-step test builds the worker's step body through make_worker; the intake test sends the prompt signal the data worker sends; real-weights parity against transformers. Docs: the model's entry and notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… transformers' bf16 spread Each served greedy reply is teacher-forced through transformers in float32; its largest deficit below the fp32 argmax must stay within twice transformers' own bf16-to-fp32 log-prob spread over the same tokens, where a bf16 implementation can break a near-tie. The fixed 0.25 margin goes. A checkpoint served with rope_theta scaled by 1.1 must fail the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on greedy flips The old gate compared the largest fp32 deficit (the margin given up where a greedy token flipped) with transformers' bf16 spread. A deficit is seen only at a flip and only bounds a bug from below, so on GPU both negative controls passed the main gate and none reached CONTROL_FACTOR times it: their flips were near-ties of 0.07-0.33 nats against a 2.86 bar, and two unrelated bugs flipped the same one. Nothing measured the served model's error. The server now returns each reply token's own log-prob (the sampler's raw log-softmax at T = 1: resources.sampler.enable_logprobs and the request's return_logprobs), and every reply position is compared, teacher-forced on the served prefix: e_srv = |served - fp32| and e_bf = |bf16 - fp32| at the served token, with fp32 under the highest-precision scope. The gate is the ratio of pooled means, R = mean(e_srv) / mean(e_bf): the real model needs R <= R_MAX, each control R >= R_CONTROL_MIN. Both (2 and 4) are placeholders for a GPU calibration run. The report prints R over positions where fp32's p(s_t) is in [0.05, 0.95], mean and p90 of both errors per prompt and pooled, and the old deficit and spread. CONTROL_FACTOR, SPREAD_CEILING and the transformers-greedy run that set the tolerance go. Plain Qwen3's log-prob path is traced, not run here: the scaffold forwards return_logprobs, the sampler spec takes enable_logprobs from resources:, and each token's value reaches the stream as metadata["logprobs"] beside token_ids. The arithmetic is checked on CPU with a stand-in reference (R 1.0 for served equal to bf16, 5.0 for five times its error). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… alone at R_MAX 1.5, with rope x1.1 its control Calibrated on an H100 at the previous commit, pooled R was: the real model 0.91 short and 0.92 long, rope_theta x1.1 1.07 and 2.58, rope_theta x2 3.18 and 3.86, rms_norm_eps 1e-5 1.07 and 0.93. The short prompts' replies are too short for a RoPE error to show, so: - A long set: 3.4k rendered tokens of fixed public-domain prose, the first chapter and a half of Alice's Adventures in Wonderland (transcribed), as test/integration/data/alice_chapters_1_2.txt, with a question about it. - R per prompt set, never pooled across sets: the short replies would dilute the control on the long set to about 1.6. - One threshold, R_MAX = 1.5: the real model must stay under it on every set (a 1.6x margin), and rope_theta x1.1 must fail it on the long set (1.7x). R_CONTROL_MIN goes. - rms_norm_eps 1e-5 goes as a control: its served log-probs are the real model's at every length (0.93 against 0.92), so it is no bug this gate has to catch. Checked on CPU with a stand-in reference (per-set R 1.0 and 3.0 as constructed) and the long prompt's rendering with Qwen3-8B's tokenizer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g prompts At the previous commit the real model failed its own GPU run, R 1.806 on the Alice prompt: the reply stopped at 92 tokens, and one served greedy flip (fp32 margin 0.60, e_srv 0.46 against a bf16 maximum of 0.094 there) took the plain mean ratio from 1.11 to 1.8. A mean over ~100 positions is too fragile. R per set is now trim(e_srv).mean() / trim(e_bf).mean(), each error's positions under its own 0.98 quantile. On earlier GPU dumps of a different long set (two of its three prompts were repo documents, not the texts committed here) that gave, pooled, the real model 0.93 and rope_theta x1.1 2.42 at 128 reply tokens. R_MAX stays 1.5. The numbers on the committed prompts come from this commit's own GPU run and are in the module docstring from the next commit. The long set is three prompts of fixed public-domain prose, committed and transcribed, each with a question that asks for a full reply: Alice (3.4k rendered tokens), Poe's The Cask of Amontillado (3.2k), and Pride and Prejudice chapter I into II (1.3k). Each prompt and set prints R, the plain-mean and p90 ratios, the served and bf16 greedy-flip counts and the largest deficit; the mid-probability ratio and the spread go. Checked on CPU with a stand-in reference: one 0.46 spike moves the plain mean ratio to 1.19 and the trimmed R to 1.004; a 2.5x error on the long set gives R 2.5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted prompts The calibration figures came from GPU dumps of a different long set, two of whose prompts were repo documents. The gate's run on the committed prompts (H100) gave R 0.911 short and 0.962 long for the real model, and 1.038 short and 2.374 long for rope_theta x1.1. Only the long set separates the control. Per prompt the two overlap (the real model reaches 1.65 on one short prompt, the control 1.42 on one long one), which the docstring now says is why R is pooled per set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lus 3 Trimmed R measures drift spread over the replies, and its 0.98 trim drops errors confined to a few positions: +2 nats at 8 of 384 positions, each a greedy flip, keeps R near 1.2. Per prompt set the served greedy flips must now also be at most the bf16 flips plus 3, and a set passes only with both checks. On an H100 the real model is at 0 vs 2 (short) and 5 vs 7 (long); rope_theta x1.1 at 3 vs 3 and 8 vs 3, so its long set fails this check too. The control test asserts only that the combined gate fails, and the report logs both checks per set. The gate's math moves to mstar/testing_utils/logprob_gate.py, so a CPU test (test_logprob_gate) can show the localized spikes passing trimmed R and failing the gate, spread drift failing R alone, and the real model's flip counts passing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Qwen3 declares its state rather than writing its specs: one paged KV array (a KVEntry, a causal Attend read, RoPE positions) and the scaffold's sampler, as Qwen3-ASR's text decoder does, with the mixed split as a kernel setting. Its LLM node is a DeclaredTokenDecoder, and the decoder attends through the array (Qwen3DenseLM(array=...)). The package imports nothing from mstar.engine.resources, and goes on the state lint's PORTED list. The adapter names the built resources as the hand-written specs did (kv_cache, attn, rope, sampler), so the engine builds the same objects: test_contract_port.py compares the specs, on the default and published configs, and the expanded prefill, decode and mixed steps, field by field, with the hand-written ones as reference. Mutations caught: a different RoPE base, and the mixed split left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
02d4ae4 to
50e6a0e
Compare
The models page said such a request decodes to the room its prompt leaves. It is capped at the server's default output cap (MAX_OUTPUT_TOKENS, 2048), and fitted to the context only below that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 2 of 7 in the stack, stacked directly on main (#1318, PR 1, is merged). Head ecf8981, base main 2c8ffde. Adds
qwen3_8b(Qwen/Qwen3-8Btext chat) throughmstar serve,/generateand/v1/chat/completions. Nothing here drafts; #1320 and the PRs above build speculative decoding (RFC #1186) on this model.What it does
Qwen3Modelis aDeclaredDecoderModelwhose oneLLMnode is the scaffold'sDeclaredTokenDecoder, so it reuses the scaffold's walks, captures, sampler and stop check. feat(qwen3): add DSpark drafting on the decoder scaffold's verify path (PR 4 of 7 in the stack) #1321 adds its verify path to the sharedSampledTokenDecoder, so Qwen3 only declares and connects its drafter there.declare_state) and imports nothing frommstar.engine.resources(the state lint'sPORTEDlist);test_contract_port.pychecks the built specs and steps against hand-written ones.model_type, RoPE scaling, sliding windows,attention_bias, non-silu); stop ids as the union of every EOS the checkpoint names; chat-template rendering withenable_thinkingflat or inchat_template_kwargs(malformed values are a 400); safetensors loading into main'sQwen3DenseLM(which takes a tied head from the embedding).MODEL_REGISTRY,HF_MODELS, the CLI defaults andADAPTER_REGISTRY(TextChatAdapter).configs/qwen3_8b.yaml: one GPU, 1024 KV pages of 128 tokens; a request is bounded by the checkpoint's 40960-token context. Docs indocs/models.rstand its mintlify page; they say a request with nomax_tokensdecodes at most 2048 tokens (the server's default output cap), fewer where its prompt leaves less room.No new dependencies or environment variables.
Since the last push (50e6a0e): one docs commit, the default output cap above.
Tests
On an 8 x H100 80GB host, torch 2.12.1+cu132, FlashInfer 0.7.2, at 50e6a0e. This head adds one docs commit since (the default output cap), which ran nothing on a GPU.
CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) +test_tp_async_lockstep_sim.py(-n 3)ruff check .(0.17.5, 0.16.0)mstar_rust;-n 6,OMP_NUM_THREADS=2), failing tests by id against main's on the same host, at this headtest_token2wav.pyin one process). 13 are main's own on this host (arm64 environment failures:test_api_completion_guardx5,test_cosyvoice3_modelx3,test_kokoro_modelx5); the other,test_kv_context_parallel::test_two_ranks_split_the_thinkers_steps_like_they_run_copied[by_head-ulysses], passes with its module run alone (78 passed, 26 skipped)test_state_lint.py,test_validate_deployment.pymstar/model/qwen3/tests, Qwen3 LM, resident planning, H2D, GDN, and main'stest_spec_{chain_key,geometry_refusals,rows_parts}), 2 x H100test/integration/test_qwen3_real_weights.py(1 x H100)Main's two failures (on every layer of the stack):
test_ar_fusion_fallback::test_a_shared_group_keeps_a_workspace_per_width_and_dtype,test_chatterbox_model::test_undecodable_reference_is_a_client_error.Parity gate (
test_qwen3_real_weights.py): served 128-token greedy replies teacher-forced through transformers in fp32 and bf16; a set passes if its trimmed log-prob error is at most 1.5x bf16's and its greedy flips at most bf16's + 3. At 50e6a0e (this head adds only docs): R 0.970 / 0.951 (short / long), flips 0 vs 2 and 6 vs 7; therope_thetax1.1 control fails the long set (R 2.418, flips 12 vs 3).Performance
Plain Qwen3-8B against vLLM, with no draft block, measured on #1325's tree 0023eff82b (unpushed; it differs from #1325's head only in three test files), not at this head.
That the plain path performs the same at this head is a hypothesis: the layers above change shared engine code, gated on a drafting node (
SamplerResource.verify_draft_tokens, GDN's bound verify walks), which is read from the code, not measured here.Method: one H100 80GB per server;
benchmark/qwen3/bench_openai.py(64 coding chats, greedy,ignore_eos, 256 output tokens, 5 waves); M* at-O2on torch 2.12.1+cu132, a fresh Inductor cache per boot; vLLM 0.33.0 on torch 2.14.0+cu130; driver 580.173.02; 3 rounds, medians, arms rotated across GPUs (2026-11-18).Noise: the host was shared with other GPU jobs (load average 2.6-16.4), so differences of a few percent are within it.
Open review items
Follow-ups
accelerator_graph.AI assistance was used (Claude Code); the tests run and their results are listed above.
🤖 Generated with Claude Code