Skip to content

feat(qwen3): add plain dense Qwen3-8B on the shared decoder scaffold (PR 2 of 7 in the stack) - #1319

Open
nick-rui wants to merge 9 commits into
mainfrom
users/nick/sd-prQ
Open

nick-rui wants to merge 9 commits into
mainfrom
users/nick/sd-prQ

Conversation

@nick-rui

@nick-rui nick-rui commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

PR 2 of 7 in the stack, stacked directly on main (#1318, PR 1, is merged). Head ecf8981, base main 2c8ffde. Adds qwen3_8b (Qwen/Qwen3-8B text chat) through mstar serve, /generate and /v1/chat/completions. Nothing here drafts; #1320 and the PRs above build speculative decoding (RFC #1186) on this model.

What it does

  • Model. Qwen3Model is a DeclaredDecoderModel whose one LLM node is the scaffold's DeclaredTokenDecoder, so it reuses the scaffold's walks, captures, sampler and stop check. feat(qwen3): add DSpark drafting on the decoder scaffold's verify path (PR 4 of 7 in the stack) #1321 adds its verify path to the shared SampledTokenDecoder, so Qwen3 only declares and connects its drafter there.
  • State contract. It declares one paged KV array and the scaffold's sampler (declare_state) and imports nothing from mstar.engine.resources (the state lint's PORTED list); test_contract_port.py checks the built specs and steps against hand-written ones.
  • What Qwen3 owns: config refusals (model_type, RoPE scaling, sliding windows, attention_bias, non-silu); stop ids as the union of every EOS the checkpoint names; chat-template rendering with enable_thinking flat or in chat_template_kwargs (malformed values are a 400); safetensors loading into main's Qwen3DenseLM (which takes a tied head from the embedding).
  • Serving. Registered in MODEL_REGISTRY, HF_MODELS, the CLI defaults and ADAPTER_REGISTRY (TextChatAdapter). configs/qwen3_8b.yaml: one GPU, 1024 KV pages of 128 tokens; a request is bounded by the checkpoint's 40960-token context. Docs in docs/models.rst and its mintlify page; they say a request with no max_tokens decodes at most 2048 tokens (the server's default output cap), fewer where its prompt leaves less room.

No new dependencies or environment variables.

Since the last push (50e6a0e): one docs commit, the default output cap above.

Tests

On an 8 x H100 80GB host, torch 2.12.1+cu132, FlashInfer 0.7.2, at 50e6a0e. This head adds one docs commit since (the default output cap), which ran nothing on a GPU.

Command Result
CPU Core (AGENTS.md paths, CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=2 -n 8 --timeout=120) + test_tp_async_lockstep_sim.py (-n 3) 16824 passed, 1898 skipped, 2 failed (main's own two, by test id); sim 11 passed
ruff check . (0.17.5, 0.16.0) clean
CPU Core, Linux arm64 Docker (torch 2.10.0+cpu, real Triton 3.6.0, mstar_rust; -n 6, OMP_NUM_THREADS=2), failing tests by id against main's on the same host, at this head 14 failed, 16699 passed, 1817 skipped (+40 passed, 5 skipped for test_token2wav.py in one process). 13 are main's own on this host (arm64 environment failures: test_api_completion_guard x5, test_cosyvoice3_model x3, test_kokoro_model x5); the other, test_kv_context_parallel::test_two_ranks_split_the_thinkers_steps_like_they_run_copied[by_head-ulysses], passes with its module run alone (78 passed, 26 skipped)
test_state_lint.py, test_validate_deployment.py 517 passed
CUDA modules at this layer (mstar/model/qwen3/tests, Qwen3 LM, resident planning, H2D, GDN, and main's test_spec_{chain_key,geometry_refusals,rows_parts}), 2 x H100 512 passed, 7 skipped (XPU only)
test/integration/test_qwen3_real_weights.py (1 x H100) 3 passed

Main's two failures (on every layer of the stack): test_ar_fusion_fallback::test_a_shared_group_keeps_a_workspace_per_width_and_dtype, test_chatterbox_model::test_undecodable_reference_is_a_client_error.

Parity gate (test_qwen3_real_weights.py): served 128-token greedy replies teacher-forced through transformers in fp32 and bf16; a set passes if its trimmed log-prob error is at most 1.5x bf16's and its greedy flips at most bf16's + 3. At 50e6a0e (this head adds only docs): R 0.970 / 0.951 (short / long), flips 0 vs 2 and 6 vs 7; the rope_theta x1.1 control fails the long set (R 2.418, flips 12 vs 3).

Performance

Plain Qwen3-8B against vLLM, with no draft block, measured on #1325's tree 0023eff82b (unpushed; it differs from #1325's head only in three test files), not at this head.

That the plain path performs the same at this head is a hypothesis: the layers above change shared engine code, gated on a drafting node (SamplerResource.verify_draft_tokens, GDN's bound verify walks), which is read from the code, not measured here.

Method: one H100 80GB per server; benchmark/qwen3/bench_openai.py (64 coding chats, greedy, ignore_eos, 256 output tokens, 5 waves); M* at -O2 on torch 2.12.1+cu132, a fresh Inductor cache per boot; vLLM 0.33.0 on torch 2.14.0+cu130; driver 580.173.02; 3 rounds, medians, arms rotated across GPUs (2026-11-18).

Noise: the host was shared with other GPU jobs (load average 2.6-16.4), so differences of a few percent are within it.

c1 c16 c32 c64
M* tok/s 165 1,916 3,544 5,365
vLLM tok/s 148 1,821 3,488 5,486
M* / vLLM 1.11 1.05 1.02 0.98
TTFT p50, M* / vLLM 54 / 39 ms 52 / 55 ms 56 / 50 ms 91 / 78 ms

Open review items

  • The config keeps the scaffold's prefix caching; no test here exercises a prefix-cache hit on plain Qwen3.

Follow-ups

AI assistance was used (Claude Code); the tests run and their results are listed above.

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
m-star 🟢 Ready View Preview Nov 18, 2026, 11:24 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@nick-rui
nick-rui changed the base branch from main to users/nick/sd-prR November 8, 2026 18:52
@nick-rui
nick-rui added this pull request to stack #1330 November 8, 2026 19:10
@nick-rui nick-rui changed the title Add plain dense Qwen3-8B feat(qwen3): add plain dense Qwen3-8B (PR 2 of 8 in the stack) Nov 8, 2026
@nick-rui
nick-rui marked this pull request as ready for review November 8, 2026 20:36
@merceod
merceod self-requested a review November 8, 2026 20:52

@merceod merceod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The model is clean against invariant 1. It declares its four resources, captures through the engine, and reuses
Qwen3DenseLM and TokenStepNode, so the decoder itself stays in one place. Startup refusals, stop ids and the tied
head path all check out.

The main question is design. Why not MultimodalDecoderModel plus SampledTokenDecoder with no media? Both are
written to take a text-only model. media defaults to (), SampledTokenDecoder.__init__ says a text-only node
passes no encoders, and its docstring names Qwen3DenseLM as a decoder it drives. Here get_graph_walk_graphs is
media_walks(max_iters, ()) with prefill_text renamed, and the KV label, loop caps, forward-pass args, generation
defaults and sampler spec repeat the scaffold. What the scaffold has that this node doesn't: the mixed prefill and
decode step, a rebuild policy for a dropped request, and decode capture up to 128 rows. The stack then teaches both
nodes to verify (#1321 changes qwen3/submodules.py, #1323 changes sampled_token_decoder.py). If DSpark or EAGLE-3
taps need something the scaffold can't give, please say what in the description. Otherwise moving this onto the
scaffold now is much cheaper than after six PRs build on it.

The parity gate uses a chosen 0.25 margin with no scale next to it. What does transformers bf16 against fp32 give
under the same rule on these 8 prompts? And one control that has to fail (say rope_theta off by 10%) would show the
rule can tell a wrong model from a right one.

No competitor numbers is fine for a stack base if the description says so. vLLM and SGLang both serve Qwen3-8B,
though, so one c1/c32 row would be cheap.

Comment thread mstar/model/qwen3/qwen3_model.py Outdated
ids = ids.reshape(-1).to(torch.long)
n, budget = ids.numel(), self.get_max_output_tokens(**kwargs)
# in the data worker, a ValueError reaches the client as a 400
if n == 0 or n + budget > self.max_seq_len:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With no max_output_tokens in the request, budget is the 2048 default, so a prompt within 2048 tokens of max_seq_len (30721 to 32767 on the shipped config) is a 400 here. The conductor would serve it. With no length kwarg it runs _fit_loop_caps and shrinks the reply to the room left, and the KV config's validate already refuses a request that really doesn't fit. The OpenAI adapter sends no default max_tokens, so this is the normal path for a client that leaves it out. I'd keep the empty and vocabulary checks and drop the length one.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The check goes: _fit_loop_caps shrinks the reply to the room left, and KVReqConfig.validate refuses a prompt that can't fit. The scaffold has no such check, so this goes with the port. I'll add an intake test: a prompt four tokens short of max_seq_len with no max_tokens is served with its reply cut to the room left, and one past max_seq_len gets a 400 for its length.

Comment thread mstar/model/qwen3/qwen3_model.py Outdated
# deployment can't honour (a 400) rather than it being dropped here
params = {key: kw[key] for key in _REQUEST_KNOBS if key in kw}
# the checkpoint's generation_config.json supplies what the request leaves out
defaults = {"temperature": 0.0} if self.generation.get("do_sample") is False else self.generation

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

load_generation_defaults in mstar/model/utils.py already reads generation_config.json, including do_sample: false as greedy, max_new_tokens and min_p. Can this use it? As written, do_sample: false drops every other default the file sets.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. On the scaffold the defaults come from load_generation_defaults, min_p included, with enable_min_p following it. This goes with the port.

Comment thread mstar/model/qwen3/submodules.py Outdated
return {"input_ids": torch.cat([inp.input_ids for inp in inputs])}

# Keyed by request id, so outside dynamo (the sampler runs eager either way).
@torch.compiler.disable

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SamplerResource.sample is already @torch.compiler.disable, so this wrapper adds nothing. Calling the resource directly is enough.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it's redundant. The scaffold calls the resource directly, so this goes with the port.

)
# the node's capabilities for the generation-aware knobs, as its ingest checks them
# (`SamplerResource._generation_controls_problem`), refused here, before dispatch
if (self.repetition_window or 0) > spec.max_repetition_window:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These three repeat SamplerResource._generation_controls_problem. One helper both call would keep the two from drifting. Also, since these knobs skip normalize_generation_kwargs, a string from /generate ("min_tokens": "4") raises TypeError here, which the conductor turns into a 500 rather than a 400. I know the type checks are your listed follow-up, just flagging the status code.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both. With min_tokens gone and Qwen3 on the scaffold, nothing in this PR forwards min_tokens, repetition_window or the top_p_* knobs any more, so the checks this PR added here, and the 500 a string could trigger, go away with the port rather than becoming a shared helper.

@merceod merceod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commits are the restack onto #1318's new head and test_plain_model.py reading GpuStepBody, so the points from last round still stand as written. We ran it on one H200 this time (torch 2.10.0, FlashInfer 0.7.2, Triton 3.6.0), at this head and merged with main c12ba47.

Tests: test_qwen3_real_weights.py 2 passed with both checkpoints cached, nothing skipped. mstar/model/qwen3/tests 20 passed, and the startup check with an empty HF cache passed.

Parity, 20 prompts (your 8, 8 more, a ~1.1k-token document, 3 with thinking on), 256 greedy tokens, first divergence against transformers fp32 under your margin rule:

Run Identical to fp32 Diverge Fail the 0.25 rule Largest margin at a divergence
configs/qwen3_8b.yaml 7 13 0 0.172
same, accelerator_graph: false 9 11 0 0.192
merged with main 7 13 0 0.172 (ids equal to the head's)
transformers bf16 8 12 0 0.215
rope_theta x1.1, same weights 7 13 2 0.331

So the served model sits inside the spread transformers bf16 has against its own fp32, which answers the scale question. The negative control is the weak spot. A 10% rope_theta error passes 18 of 20 prompts and is token-identical to fp32 on 7. On your test's own 8 prompts at 128 tokens it trips on one prompt (token 88, margin 0.33), so the gate can fail, but only just. Comparing against the bf16 spread, or a teacher-forced check of the served tokens' log-probs against fp32, would catch an error like that with room to spare. Not blocking.

Throughput, bench_openai.py from #1321, 256 tokens ignore_eos, 3 boots, output tok/s median (spread under 1% across boots): c1 212.7, c16 2318, c32 4380, c64 6743. Your H100 rows are c1 163.2, c16 2009, c64 4975, so this is in line for the bandwidth step. A warm boot of the shipped config is about 40 s.

Functional on the shipped config: max_tokens 1, 2 and 3 give 1, 2 and 3 tokens, two identical requests give identical ids, an aborted stream frees its slot (the next request served in 0.15 s), min_tokens: 4 is a 400 and serves with serve_min_tokens: true, an 8 x 8k-token burst serves on the captured and eager configs, and a multi-turn chat works. Two rows confirm last round's comments. A 31,321-token prompt with no max_tokens gets the 400 from qwen3_model.py:245 ("prompt (31321 tokens) + max_output_tokens (2048) must fit max_seq_len (32768)") and serves with max_tokens: 512. "min_tokens": "4" on /generate is a 500 (TypeError: '>' not supported between instances of 'str' and 'int').

The design question from last round is still the one that decides where the rest of the stack attaches. I'd like an answer on it before approving.

@NSagan271

Copy link
Copy Markdown
Collaborator

/review

Comment thread mstar/model/qwen3/qwen3_model.py Outdated
Comment on lines +22 to +30
from mstar.model.base import (
ForwardPassArgs,
KVLabel,
LabelLayout,
Model,
PrefixKeys,
ProcessPromptOutput,
Span,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P0] The model is written against a Model API that main has since replaced, so merging it breaks mstar.model.qwen3 at import.

On current main, mstar.model.base no longer exports ForwardPassArgs (it lives in mstar/conductor/conductor.py), and CurrentForwardConductorMetadata (imported at R14) doesn't exist at all. Model.walk_plan is now abstract (mstar/model/base.py:972) and replaces get_initial_forward_pass_args / get_partition_forward_pass_args (R199-225), which nothing calls any more. The conductor also calls get_request_resource_configs(input_modalities, output_modalities, model_kwargs) (conductor.py:766), while R161 takes (partition_fwd_args, model_kwargs). None of these hunks conflicts textually, so the merge looks clean. Afterwards, import mstar.model.qwen3.qwen3_model raises ImportError. That fails CPU Core's test_registration.py, test_plain_model.py and test_startup_graph_validation.py[qwen3_8b], and mstar serve qwen3_8b won't start. Fix: rebase, then implement walk_plan (prefill, then WalkStep(DECODE_WALK, bind={"text_inputs": "new_token"})) and the new get_request_resource_configs signature. MultimodalDecoderModel already has both (see the design question). How to confirm: on main 156d2fc with this diff applied, python -c 'import mstar.model.qwen3.qwen3_model'.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moot after the rework. Qwen3Model subclasses MultimodalDecoderModel and inherits walk_plan and get_request_resource_configs; nothing in mstar/model/qwen3 imports ForwardPassArgs or the removed forward-pass hooks. The head is on main 9a50fdf, and a merge-tree against current main conflicts only in test/modular/test_startup_graph_validation.py, which the next rebase resolves.

Comment thread mstar/model/qwen3/submodules.py Outdated
accelerator_graph: bool = True,
prefill_token_buckets: list[int] | None = None,
):
super().__init__(prefill_token_buckets)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P0] super().__init__(prefill_token_buckets) raises TypeError every time the node is built.

The MRO is Qwen3LMSubmodule → TokenStepNode → ARNodeSubmodule → NodeSubmodule. Neither of the first two defines __init__, so this calls NodeSubmodule.__init__(self) with an extra positional argument, even when the value is None. Both make_submodule on any worker that has weights (every served qwen3_8b) and the CPU Core tests test_every_output_of_a_decode_step_stays_on_the_device and test_an_eager_decode_step_reads_its_routed_tokens fail. CI installs triton, so importorskip doesn't skip them. Even when fixed, the value never reaches the ladder, because R95 reads the class's PREFILL_TOKEN_BUCKETS. The engine already replaces a packed capture's ladder from the serving config's prefill_token_buckets ("a model plumbs nothing", docs/adding_models.rst:2429). So drop the prefill_token_buckets kwarg from Qwen3Model and from this class, and correct the docs rows that advertise model_kwargs.prefill_token_buckets. How to confirm: Qwen3LMSubmodule(None, [96]) raises TypeError: NodeSubmodule.__init__() takes 1 positional argument but 2 were given.

Suggested change
super().__init__(prefill_token_buckets)
super().__init__()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moot after the rework. The node is a SampledTokenDecoder subclass with no constructor of its own, and the prefill_token_buckets model kwarg and its docs rows are gone. The ladder is the scaffold's, and a deployment sets it through the serving config's prefill_token_buckets.

Comment thread mstar/model/qwen3/qwen3_model.py Outdated
Comment on lines +267 to +268
thinking = kwargs.get("enable_thinking", (kwargs.get("chat_template_kwargs") or {}).get("enable_thinking"))
extra = {} if thinking is None else {"enable_thinking": bool(thinking)}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] enable_thinking sent under chat_template_kwargs is read by truthiness, so {"chat_template_kwargs": {"enable_thinking": "false"}} turns thinking on.

normalize_generation_kwargs checks only a flat enable_thinking as a bool (its comment says why: "false" is truthy in a template). TextChatAdapter passes chat_template_kwargs through unflattened, so the nested value reaches bool(thinking) unchecked. Refuse a non-bool here with check_bool_knob("enable_thinking", thinking). Alternatively, flatten the template kwargs at the adapter, as Qwen3_5Adapter / Rust template_kwargs_chat do, so the existing check covers them.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the rework. enable_thinking, flat or under chat_template_kwargs, must be a boolean or the request is a 400. A chat_template_kwargs that isn't an object is a 400 too. Tests cover both.

Comment thread mstar/model/qwen3/submodules.py Outdated
single_request_inputs=ARNodeInputs(
input_ids=torch.zeros(1, dtype=torch.long, device=device), input_seq_len=1,
),
capture_batch_sizes=self.DECODE_CAPTURE_BATCH_SIZES,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] perf Decode inherits TokenStepNode.DECODE_CAPTURE_BATCH_SIZES (up to 64). The engine caps a decode step at its largest capture, so above 64 concurrent requests every decode pass splits in two. The shared SampledTokenDecoder captures up to 128.

Est.: Qwen3-8B is about 16.4 GB of bf16 weights, about 4.9 ms per step at 3.35 TB/s on H100. Decode at these widths is weight-bandwidth-bound (GPU-bound, from the arithmetic, not measured). At c=128, two 64-row steps re-read the weights where one 128-row step would read them once, so up to ~1.6-1.8x decode tok/s. Probability medium (~50%): it doesn't pay if per-step CPU overhead or the 1024-page KV pool (131k tokens) binds first. Expected: gains only at concurrency above 64, none below. Difficulty S: set DECODE_CAPTURE_BATCH_SIZES = [1, 2, 4, 8, 16, 32, 48, 64, 96, 128]. Confirm with benchmark/qwen3/bench_openai.py at c=128 and c=256, A/B on this ladder.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moot after the rework. The node inherits SampledTokenDecoder's decode ladder, which goes to 128 rows. I haven't measured the effect above c=64, so the description makes no claim about it.

Comment on lines +112 to +114
body = GpuStepBody.__new__(GpuStepBody)
body._graph_runtime = SimpleNamespace(supports_inline_emit=True)
body._device_loopback = {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Nit: the test builds GpuStepBody with __new__ and sets its privates _graph_runtime and _device_loopback by name. A rename of either breaks the test with no grep hit at the definition. Meanwhile the real constructor can change what all_on_device reads without this test noticing. AGENTS.md's Testing section asks for the smallest real object instead: make_worker in mstar/testing_utils/worker.py builds a real step executor over the collaborators the test names.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'll build the step body through make_worker, the way test_worker_device_loopback.py does, so the test goes through the real constructor.

@mstar-review

This comment has been minimized.

@mstar-review

mstar-review Bot commented Nov 10, 2026

Copy link
Copy Markdown

No live test yet: fix what the review left open first (open P0–P2 findings: mstar/model/qwen3/qwen3_model.py:30 (P0), mstar/model/qwen3/submodules.py:70 (P0), mstar/model/qwen3/submodules.py:91 (P2)), then comment /review again.

@nick-rui
nick-rui removed this pull request from stack #1330 November 12, 2026 01:21
@nick-rui
nick-rui added this pull request to stack #1454 November 12, 2026 01:22
@nick-rui nick-rui changed the title feat(qwen3): add plain dense Qwen3-8B (PR 2 of 8 in the stack) feat(qwen3): add plain dense Qwen3-8B on the shared decoder scaffold (PR 2 of 7 in the stack) Nov 12, 2026
@nick-rui

nick-rui commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Re the reviews of Nov 8 and Nov 9:

Thanks for both rounds and for the H200 runs.

Design. You're right, and nothing in the stack needs a separate node. The hidden-state taps for DSpark and EAGLE-3 go through SampledTokenDecoder.decoder_kwargs into Qwen3DenseLM.forward(taps=...), which both nodes already drive. The drafter's pass over the prompt is its prefill(PrefillInputs), which the scaffold calls after a prompt step samples. The Qwen3-0.6B draft model builds its own Qwen3DenseLM and never touches the target node.

So Qwen3 is now on MultimodalDecoderModel + SampledTokenDecoder with media=(), following Ripple-65B (#1344). The stack has one verify path, in the scaffold, landing in #1321. It only switches on for a node with a drafter, so the other scaffold models are unchanged; #1321's tests check the declared loop-back and token stop across all eleven scaffold adopters.

#1323, Qwen3.5 MTP, is held. Main's State Contract (#1227, #1232) moved Qwen3.5 onto declared state reached through handles, and #1323 was written against the old resource path, so Qwen3.5 MTP lands later on mstar.engine.state. Its generic pieces moved down into #1321, #1322 and #1325, and nothing in the stack touches mstar/model/qwen3_5. The stack is now seven PRs, numbered from #1318.

While a node verifies, the scaffold turns off combined walks, rebuild and the mixed split, because the engine refuses a verifying node in a combined walk for now.

On the scaffold, plain Qwen3 should pick up the mixed prefill/decode step, the rebuild policy and larger decode captures, so I'll re-measure throughput and capture memory for both plain and drafted configs, not only parity.

Current head. b9b8452 fails CPU Core, and it no longer builds against current main: #1184 removed the prefill_token_buckets constructor argument, and #1134 removed the model API the separate Qwen3Model is written against. Rather than patch that, this head (4d22dcb) is the port itself, on main 9a50fdf.

Parity gate. Thanks for the bf16 spread and the rope_theta control. The fixed 0.25 margin is gone. The test teacher-forces each served reply through transformers in fp32 and in bf16, and compares the served log-probs' error from fp32 with bf16's own. A prompt set passes only if both of these hold (mstar/testing_utils/logprob_gate.py, a pure function, tested on CPU on synthetic errors in test/modular/test_logprob_gate.py):

  • Trimmed R, the served error's mean over bf16's with each cut at its own 98th percentile so one greedy flip can't carry it, is at most 1.5. This catches drift spread over the replies, such as a wrong rope base.
  • The served greedy flips are at most bf16's plus 3. Errors confined to a few positions sit past the percentile, so the trim drops them; where they flip the greedy token, this catches them.

On one H100 at this head, the real model scores R 0.911 with 0 flips against bf16's 2 on the short set, and R 0.962 with 5 flips against 7 on a long set of three prompts of 1.3k to 3.4k tokens, so it passes both checks on both sets. rope_theta x1.1 fails the long set on both checks: R 2.374, and 8 flips against bf16's 3, where the cap is 6. It passes the short set, R 1.038 and 3 flips against 3, because those replies are too short for a position error to build up; that is why the long set exists. The control test asserts that the gate fails on at least one set.

min_tokens and pre-tokenized input. The scaffold forwards neither, and verify refuses min_tokens, so both leave this PR.

Competitors. The description now has a c1/c32 vLLM row for Qwen3-8B.

@nick-rui

nick-rui commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Re the automated review of Nov 10:

Outside the diff, dynamo/bridges.py:90. Moot after the rework. The model no longer serves min_tokens, and the serve_min_tokens kwarg and the conductor-side checks are gone. Its request kwargs are the scaffold's, which leave min_tokens out, so the Dynamo strip matches.

Description, the stale head. Right, the description was stale. The rework onto the shared decoder scaffold is pushed as 4d22dcb, and the description names that head and only what ran on it.

Description, the CPU Core claim. The constructor bug is gone in the rework. CPU Core on that head is in the description.

Description, model_kwargs.prefill_token_buckets. The kwarg is gone in the rework. The prefill ladder is the scaffold's, and the serving config's prefill_token_buckets overrides it.

Description, the sampler knobs. Moot after the rework. The SamplingReqConfig change is no longer in this PR, and the model forwards only the scaffold's sampling keys. The description will drop that section.

Design question, why not the scaffold. Agreed, and done in the rework. Qwen3Model subclasses MultimodalDecoderModel and the node subclasses SampledTokenDecoder, declaring only its attention key, layer steps and resident walks. The walk plan, mixed steps, rebuild policy, TP fusion init and 128-row decode ladder come from the scaffold.

PR-wide question, where speculation lands. On the shared one. In the reworked stack, #1321 adds the verify path to SampledTokenDecoder, and the Qwen3 node only connects its drafter. The drafter is bound through NodeSubmodule.bind_draft in #1320, per its review.

Description, the provenance history. Agreed. The next description leads with what ran on the pushed head, GPU included or explicitly not, and drops the history of earlier heads.

@NSagan271

Copy link
Copy Markdown
Collaborator

/review

Comment thread docs/models.rst Outdated
Comment on lines +640 to +642
checkpoint's ``generation_config.json``. A request is bounded by the checkpoint's
context: one that asks no length decodes to the room its prompt leaves, and a prompt
past it is a 400. Rope scaling, sliding windows and attention biases are refused at

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] The docs say a request that asks no length "decodes to the room its prompt leaves". It actually stops at 2048 tokens.

Neither Qwen3Model nor MultimodalDecoderModel overrides get_max_output_tokens, so a request with no max_output_tokens gets Model's MAX_OUTPUT_TOKENS = 2048 (mstar/model/base.py:150). Conductor._fit_loop_caps only shrinks that cap when the context has less room. It never raises it. A 100-token prompt with thinking on and no max_tokens therefore ends at 2048 tokens, not near 40k. sampling_defaults drops a generation_config max_new_tokens, and Qwen3-8B's file sets none anyway. The same sentence is in docs/mintlify/models.mdx:178 (invariant 9 wants both fixed). Either describe the 2048 default, as below, or override get_max_output_tokens to return the room the context leaves when no length is given, as GLM-5.2 does with context_limit(). The second fits Qwen3's long thinking replies, but it changes the default every client gets. How to confirm: send /v1/chat/completions with a short prompt, thinking on and no max_tokens. The reply ends with finish_reason: length at 2048 completion tokens.

Suggested change
checkpoint's ``generation_config.json``. A request is bounded by the checkpoint's
context: one that asks no length decodes to the room its prompt leaves, and a prompt
past it is a 400. Rope scaling, sliding windows and attention biases are refused at
checkpoint's ``generation_config.json``. A request that asks no length decodes at most
2048 tokens, fewer when its prompt leaves less room in the checkpoint's context, and a prompt
past it is a 400. Rope scaling, sliding windows and attention biases are refused at

@mstar-review

mstar-review Bot commented Nov 12, 2026

Copy link
Copy Markdown

Automated review

🟡 1 P2 on 4d22dcb, full review · design risk medium: Kept at the earlier level until a maintainer other than the author confirms. The model-local design call is now resolved in code (Qwen3 is on the shared scaffold), and what remains is confirming that the stack's speculation lands on SampledTokenDecoder.

Both P0s (the import break on main's walk_plan API and the constructor TypeError) and all three P3s are fixed by the port onto MultimodalDecoderModel / SampledTokenDecoder. The Dynamo min_tokens item is moot because min_tokens support is gone. One new P2: the docs overstate how long a request with no length can decode.

Earlier findings: 5 fixed, 0 fixed above, 0 open, 0 disputed, 1 moot.

Description

  • Missing: a request with no max_tokens is capped at 2048 output tokens (the base MAX_OUTPUT_TOKENS) and then fitted to the context. The description says only that a request is bounded by the checkpoint's context (40960), and with thinking on, Qwen3 replies often run past 2048.
  • The vLLM comparison was measured on a head above this one and in one server session per arm (one warmup, then 5 waves per level). The benchmarking skill asks for rounds across restarts with the arm order rotated. That matters most for the c32 'on par' claim, where the two arms are 2% apart.
  • Missing: the shipped config inherits the scaffold's prefix caching (the PrefixKeys on text_inputs), and neither the description nor the parity run says whether a prefix-cache hit was exercised.

Design

  • Phase A ideal: MultimodalDecoderModel with media = (), plus a SampledTokenDecoder subclass declaring KV and attention layer_steps, with resources declared and sampling defaults from load_generation_defaults, so adoption costs declarations only.
  • The PR now matches the derivation row for row: about 160 lines of model code, the same shape as Ripple-65B. It keeps the Qwen3-specific config refusals, the stop-id union and template rendering.
  • North star: one shared dense-decoder node that every text LLM and VLM subclasses, with speculation added there once. This PR is on it, and per the author #​1321's verify path lands in SampledTokenDecoder.
  • The one remaining gap is the default output length: whether a Qwen3 request with no length should take the shared 2048 cap (question 3, P2 docs finding).
What the review checked
  • Prior P0 (import on main): Qwen3Model now subclasses MultimodalDecoderModel, which supplies walk_plan, get_request_resource_configs(input_modalities, output_modalities, model_kwargs) and the walks. No removed API is imported.
  • Prior P0 (constructor): Qwen3LMSubmodule has no __init__. SampledTokenDecoder.__init__(model, config) builds TokenStop from config.stop_token_ids, which Qwen3Model.__init__ sets.
  • Prior P3 (enable_thinking): _prompt_ids R97-106 refuses a non-dict chat_template_kwargs and a non-bool enable_thinking, flat or nested, with ValueError (a 400). The tests cover both cases.
  • Prior P3 (decode ladder): DECODE_CAPTURE_BATCH_SIZES is inherited from SampledTokenDecoder and goes up to 128.
  • Prior P3 (test privates): the test builds the step body through make_worker(...).executor._body, the same pattern as test_worker_device_loopback.py.
  • Dropped hunks: the SamplingReqConfig.validate checks, serve_min_tokens, the prefill_token_buckets kwarg and the model-side length check are gone. The conductor's _fit_loop_caps and KVReqConfig.validate now cover length (test_qwen3_intake.py), and load_generation_defaults supplies sampling defaults.
  • Invariant 15: TextChatAdapter is unchanged. Both frontends pass chat_template_kwargs through unflattened (Python _passthrough, Rust text_only_chat's req.extra), and request_kwargs now lists it.
  • Invariant 1: the node declares only layer_steps and ATTN_KEY. KV, attention, rope and sampler are declared in get_node_resources, and nothing is allocated, captured or synchronised in model code.
  • resident_walks returns {decode}, as Phi-4-MM, Qwen3.5 and Cosmos3 do. Decode's preprocess reads only the loopback tokens, Qwen3Attention rotates through the position resource inside the forward, and the engine refuses if a resource can't do resident rows.
  • build_lm on meta + to_empty: this is the pattern MiniCPM-o already uses for Qwen3DenseLM on main. The model has no init-time buffers, and qkv_proj_weight is registered after load.
  • Stop ids: the union of the EOS ids in config.json, generation_config.json and the tokenizer. The tokenizer is loaded only when tokenizer files exist, so the CPU tests need no tokenizer.
  • Config refusals: model_type, rope_scaling, scaled rope_parameters, sliding window, attention_bias and hidden_act. rope_parameters.rope_theta overrides the top-level value, and the same cfg.rope_theta feeds both PositionConfig and Qwen3Attention.
  • Behaviour lens 1 (inputs): empty /generate text renders as an empty user turn, and the adapter refuses empty chats. A prompt past the context gets a 400 at the conductor. A request with no length is capped at 2048 (the P2 docs finding).
  • Behaviour lens 2 (dependants): the shared test_kv_layout_intake._conduct only gains initial_signals, defaulting to {}, so its existing callers are unchanged.
  • Behaviour lens 3 (pins): rebased onto main 9a50fdf. Every scaffold hook used here (_sampler_spec, MIXED_SPLIT, _resolve_weights, chat_turns, encode_ids) exists on main.
  • Behaviour lens 4 (shared state): no per-request or handle-keyed state in the model. TP fusion init comes from the scaffold's get_submodule.
  • Behaviour lens 5 (silent fallbacks): keys in chat_template_kwargs other than enable_thinking are ignored, as on the other text-chat models. No other fallbacks.
  • Behaviour lens 6 (captured paths): no new ops. Every forward path is the scaffold's.
  • Behaviour lens 7 (tests under CI): the new modules guard with importorskip('triton'). test_logprob_gate uses seeded CPU generators. The integration test skips without CUDA or cached weights. The sys.path.insert import of test_kv_layout_intake follows an existing pattern.
  • Behaviour lens 8 (claimed gains): the plain-vs-vLLM numbers come from a head above, whose plain path the description says is this PR's. Methodology is listed under description problems.
  • Abstraction fit: model scaffold -> MultimodalDecoderModel: fits.
  • Abstraction fit: decoder node -> SampledTokenDecoder: fits.
  • Abstraction fit: parity gate logic -> mstar/testing_utils/logprob_gate.py, a pure function with CPU tests: fits.
  • Minimality: 3 concepts added (Qwen3Config.stop_token_ids, qwen3_lm_config, the logprob gate), none deletable.
  • Docs: the model row and notes are in both docs/models.rst and docs/mintlify/models.mdx. No env vars or dependencies are added.

Live test: full (correctness, performance). No live test has passed on this PR, and the model was rewritten onto the shared scaffold, so the whole served path (prefill, mixed and decode captures, eager fallback, both frontends) is new at this head. The live-test job posts its outcome in a separate comment.

What's left for you

  • mstar/model/qwen3/submodules.py:10: The PR now does what Phase A derived: Qwen3LMSubmodule(SampledTokenDecoder) and Qwen3Model(MultimodalDecoderModel) with no media. The earlier alternative, a hand-built TokenStepNode node, is gone. What this does well: walk_plan, the mixed step, rebuild, TP fusion and the 128-row ladder come from the scaffold, and #​1321's verify path can serve every scaffold model. To decide: confirm that speculation (#​1321) should extend the shared SampledTokenDecoder rather than a Qwen3 node. Changing that later costs porting the verify path.
  • mstar/model/qwen3/qwen3_model.py:56: Why a new Model + NodeSubmodule pair rather than MultimodalDecoderModel with media = () and a SampledTokenDecoder subclass declaring KV/attention layer_steps? That scaffold already drives Qwen3DenseLM for Qwen3-ASR and Higgs Audio. It owns walk_plan, the mixed prefill+decode step, rebuild policies, the TP all-reduce fusion init, load_generation_defaults and a 128-row decode ladder. If nothing Qwen3-specific blocks it, most of qwen3_model.py and submodules.py go away, and the rebase onto walk_plan comes for free.
  • (PR-wide): #​1321 builds speculative decoding on this node. Should that land on SampledTokenDecoder, so every dense decoder (Qwen3.5, Qwen3-ASR, Higgs) can opt in, or only on Qwen3LMSubmodule? If the shared one, this PR's class is the wrong base for the stack.
  • mstar/model/qwen3/qwen3_model.py:27: Should a Qwen3 request with no length default to the room its context leaves, rather than the shared 2048, given that thinking-mode replies routinely run past 2048 and Qwen's model card suggests far longer outputs? The answer decides whether the docs finding is fixed in the docs or in a get_max_output_tokens override.
  • [P2] docs/models.rst:642: The docs say a request that asks no length "decodes to the room its prompt leaves". It actually stops at 2048 tokens.

Approve if you're OK with these. The live-test comment adds its own items.

Any approval is posted by the live-test job, by the rule in gpu_review.py. Rules: AGENTS.md. Reply to discuss.

@mstar-review

mstar-review Bot commented Nov 12, 2026

Copy link
Copy Markdown

No live test yet: fix what the review left open first (open P0–P2 findings: docs/models.rst:642 (P2)), then comment /review again.

@merceod merceod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Served the shipped configs/qwen3_8b.yaml at this head, merged with current main, on one H200, through the Python frontend and through --rust-frontend. Everything I tried works.

Check Python frontend --rust-frontend
boot 250 s cold, CapturePlan 98 of 98 buckets 64 s (warm compile cache)
your 8 prompts, greedy chat, max_tokens 256 8 x 200, usage and finish_reason on each (7 stop, 1 length at 256) same, text byte-equal to the Python frontend's
max_tokens 1 / 2 / 3 1 / 2 / 3 tokens, length same
one streamed chat with include_usage 67 chunks, text and usage equal to the non-streamed reply, stop, [DONE] same
16 chats at once 16 x 200, usage and finish_reason on all 16 same
40,912-token prompt, no max_tokens (48 positions left) 200, 49 tokens, length same
40,976-token prompt 400, "needs 40977 positions of 'main', past the model's context of 40960" same
client closes a stream after 5 chunks, then a chat next chat 200 in 0.15 s same

Against transformers' bf16 Qwen3-8B greedy on the same 8 rendered prompts (256 tokens): 5 of 8 are token-identical. The other 3 split at near-ties, with fp32 top-2 margins of 0.026, 0.048 and 0.114 nats, and bf16 has the two tokens tied at all three spots. fp32 sides with the served token on 2 of the 3. Both frontends return the same ids, and every chat text equals its decoded ids.

The docs line from the bot's last pass is still the one thing open. A short prompt with no length and ignore_eos stops at 2048 tokens here, so "decodes to the room its prompt leaves" only holds when that room is under 2048. Fixing the sentence is enough.

nick-rui and others added 8 commits November 15, 2026 15:54
Qwen3Model is a MultimodalDecoderModel with no media and Qwen3LMSubmodule a
SampledTokenDecoder, as Ripple-65B is. It is registered as qwen3_8b and
served with mstar serve and the OpenAI route. The scaffold owns the walks
(prefill_text, the mixed step, decode), the walk plan, the decode loop's
cap, rebuild, the sampler's request config and the stop check. The model
keeps its config refusals, the union of every EOS the checkpoint names, its
resources, and the chat template's rendering: enable_thinking flat or in
chat_template_kwargs (checked to be a boolean; a chat_template_kwargs that
isn't an object is a 400), a message's text parts joined by newlines,
rendered to text and encoded through encode_ids as Ripple and Qwen3.5 do.
chat_template.jinja is fetched with the tokenizer's files. Decode reads its
rows off the resident tables. The KV pool holds the checkpoint's context;
the configs say the top-level max_seq_len is required and unread.

Tests: registration; request_kwargs' contract; a conductor intake test on a
4096-token context (four decode steps and five tokens for a prompt four
short of it, the context's 400 for one past it); sampling defaults from
generation_config.json with min_p turning its filter on; the
startup-validation factory builds the model through its own __init__ with
the config's model_kwargs; the decode-step test builds the worker's step
body through make_worker; the intake test sends the prompt signal the data
worker sends; real-weights parity against transformers. Docs: the model's entry and
notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… transformers' bf16 spread

Each served greedy reply is teacher-forced through transformers in float32; its
largest deficit below the fp32 argmax must stay within twice transformers' own
bf16-to-fp32 log-prob spread over the same tokens, where a bf16 implementation
can break a near-tie. The fixed 0.25 margin goes. A checkpoint served with
rope_theta scaled by 1.1 must fail the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on greedy flips

The old gate compared the largest fp32 deficit (the margin given up where a
greedy token flipped) with transformers' bf16 spread. A deficit is seen only
at a flip and only bounds a bug from below, so on GPU both negative controls
passed the main gate and none reached CONTROL_FACTOR times it: their flips
were near-ties of 0.07-0.33 nats against a 2.86 bar, and two unrelated bugs
flipped the same one. Nothing measured the served model's error.

The server now returns each reply token's own log-prob (the sampler's raw
log-softmax at T = 1: resources.sampler.enable_logprobs and the request's
return_logprobs), and every reply position is compared, teacher-forced on
the served prefix: e_srv = |served - fp32| and e_bf = |bf16 - fp32| at the
served token, with fp32 under the highest-precision scope. The gate is the
ratio of pooled means, R = mean(e_srv) / mean(e_bf): the real model needs
R <= R_MAX, each control R >= R_CONTROL_MIN. Both (2 and 4) are placeholders
for a GPU calibration run. The report prints R over positions where fp32's
p(s_t) is in [0.05, 0.95], mean and p90 of both errors per prompt and pooled,
and the old deficit and spread. CONTROL_FACTOR, SPREAD_CEILING and the
transformers-greedy run that set the tolerance go.

Plain Qwen3's log-prob path is traced, not run here: the scaffold forwards
return_logprobs, the sampler spec takes enable_logprobs from resources:,
and each token's value reaches the stream as metadata["logprobs"] beside
token_ids. The arithmetic is checked on CPU with a stand-in reference
(R 1.0 for served equal to bf16, 5.0 for five times its error).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… alone at R_MAX 1.5, with rope x1.1 its control

Calibrated on an H100 at the previous commit, pooled R was: the real model
0.91 short and 0.92 long, rope_theta x1.1 1.07 and 2.58, rope_theta x2
3.18 and 3.86, rms_norm_eps 1e-5 1.07 and 0.93. The short prompts' replies
are too short for a RoPE error to show, so:

- A long set: 3.4k rendered tokens of fixed public-domain prose, the first
  chapter and a half of Alice's Adventures in Wonderland (transcribed), as
  test/integration/data/alice_chapters_1_2.txt, with a question about it.
- R per prompt set, never pooled across sets: the short replies would
  dilute the control on the long set to about 1.6.
- One threshold, R_MAX = 1.5: the real model must stay under it on every
  set (a 1.6x margin), and rope_theta x1.1 must fail it on the long set
  (1.7x). R_CONTROL_MIN goes.
- rms_norm_eps 1e-5 goes as a control: its served log-probs are the real
  model's at every length (0.93 against 0.92), so it is no bug this gate
  has to catch.

Checked on CPU with a stand-in reference (per-set R 1.0 and 3.0 as
constructed) and the long prompt's rendering with Qwen3-8B's tokenizer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g prompts

At the previous commit the real model failed its own GPU run, R 1.806 on
the Alice prompt: the reply stopped at 92 tokens, and one served greedy
flip (fp32 margin 0.60, e_srv 0.46 against a bf16 maximum of 0.094 there)
took the plain mean ratio from 1.11 to 1.8. A mean over ~100 positions is
too fragile.

R per set is now trim(e_srv).mean() / trim(e_bf).mean(), each error's
positions under its own 0.98 quantile. On earlier GPU dumps of a
different long set (two of its three prompts were repo documents, not the
texts committed here) that gave, pooled, the real model 0.93 and
rope_theta x1.1 2.42 at 128 reply tokens. R_MAX stays 1.5. The numbers on
the committed prompts come from this commit's own GPU run and are in the
module docstring from the next commit.

The long set is three prompts of fixed public-domain prose, committed and
transcribed, each with a question that asks for a full reply: Alice (3.4k
rendered tokens), Poe's The Cask of Amontillado (3.2k), and Pride and
Prejudice chapter I into II (1.3k). Each prompt and set prints R, the
plain-mean and p90 ratios, the served and bf16 greedy-flip counts and the
largest deficit; the mid-probability ratio and the spread go.

Checked on CPU with a stand-in reference: one 0.46 spike moves the plain
mean ratio to 1.19 and the trimmed R to 1.004; a 2.5x error on the long
set gives R 2.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted prompts

The calibration figures came from GPU dumps of a different long set, two
of whose prompts were repo documents. The gate's run on the committed
prompts (H100) gave R 0.911 short and 0.962 long for the real model, and
1.038 short and 2.374 long for rope_theta x1.1. Only the long set
separates the control. Per prompt the two overlap (the real model reaches
1.65 on one short prompt, the control 1.42 on one long one), which the
docstring now says is why R is pooled per set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lus 3

Trimmed R measures drift spread over the replies, and its 0.98 trim drops
errors confined to a few positions: +2 nats at 8 of 384 positions, each a
greedy flip, keeps R near 1.2. Per prompt set the served greedy flips must
now also be at most the bf16 flips plus 3, and a set passes only with both
checks. On an H100 the real model is at 0 vs 2 (short) and 5 vs 7 (long);
rope_theta x1.1 at 3 vs 3 and 8 vs 3, so its long set fails this check
too. The control test asserts only that the combined gate fails, and the
report logs both checks per set.

The gate's math moves to mstar/testing_utils/logprob_gate.py, so a CPU test
(test_logprob_gate) can show the localized spikes passing trimmed R and
failing the gate, spread drift failing R alone, and the real model's flip
counts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Qwen3 declares its state rather than writing its specs: one paged KV array (a KVEntry, a causal Attend read, RoPE positions) and the scaffold's sampler, as Qwen3-ASR's text decoder does, with the mixed split as a kernel setting. Its LLM node is a DeclaredTokenDecoder, and the decoder attends through the array (Qwen3DenseLM(array=...)). The package imports nothing from mstar.engine.resources, and goes on the state lint's PORTED list.

The adapter names the built resources as the hand-written specs did (kv_cache, attn, rope, sampler), so the engine builds the same objects: test_contract_port.py compares the specs, on the default and published configs, and the expanded prefill, decode and mixed steps, field by field, with the hand-written ones as reference. Mutations caught: a different RoPE base, and the mixed split left out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The models page said such a request decodes to the room its prompt leaves.
It is capped at the server's default output cap (MAX_OUTPUT_TOKENS, 2048),
and fitted to the context only below that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
staging - docs/mintlify — ecf8981a Deployed Nov 18, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants