This page describes everything you must implement to add a new model to mstar. When
you finish, the conductor can schedule your model, workers can execute it on GPU, and you
can launch it with mstar-serve.
A model in mstar has several separate responsibilities:
- The
Modelclass (mstar/model/base.py) is the interface that the rest of the system calls. It tokenizes prompts, declares the computation graph, declares the resources each node needs, builds forward-pass arguments, and post-processes outputs. It contains no GPU compute. - Submodules (
NodeSubmoduleinmstar/model/submodule_base.py) are thetorch.nn.Modules that perform the compute. Each graph node maps to one submodule. - State (
mstar.engine.state) is what a node keeps between steps: a KV cache, a recurrent value, positions and a sampler. The model declares it indeclare_state(), and its layers reach it through handles. The engine builds the resources (mstar/engine/resources/) that hold each declaration and binds a handle into the submodule and its layers. A node that declares no state was called a "stateless" node in earlier versions. Per-request LoRA adapters stand outside it: they are weights that no request writes, and a model declares them inlora_roots()(see Per-request LoRA adapters). - The engine (
mstar/engine/engine.py) is a single class that runs every node. It compiles forwards, captures CUDA graphs, batches requests, and runs each step's resource lifecycle (admit, plan, forward, commit). You never write an engine. Earlier versions required you to choose an engine type per node. That is no longer true. A node's capabilities now follow from the state it declares. - The graph (
mstar/graph/base.py) declares what runs in what order: nodes, edges between them, and loops. All of a model's work forms one large computation graph. Each named "graph walk" (for exampleprefillordecode) is one path through that graph. In code, however, you declare each walk as a separate standalone graph. A node that appears in several walks is referenced by name in each walk. The submodule behind that name, and its resources, are shared across all of them. - The config YAML (
configs/) maps graph nodes to physical GPU ranks throughnode_groups. Disaggregation is configured here. The same model code runs on one GPU or on many, and only the config changes.
One vocabulary note for this page. A tensor bundle routed between nodes is a
NameToTensorList, which is dict[str, list[torch.Tensor]]
(mstar/communication/tensors.py). It maps an edge name to a list of tensors. The list
usually has length 1.
The diagram below shows the request flow at the conductor and model level. Its granularity is one graph walk. The model decides at ingest which walks a request runs; the conductor runs them in order, moving on as each walk completes. Everything that happens inside a walk is described in later steps and is omitted here:
process_prompt() # text/media -> initial tensors
│
▼
walk_plan() # per partition: the walks to run (e.g. prefill, decode)
│
▼ (conductor sends each walk its inputs, the engine runs each node)
postprocess() # model output tensor -> bytes for the client
A typical model lives in its own package under mstar/model/<your_model>/:
mstar/model/<your_model>/
├── __init__.py
├── config.py # a @dataclass with architecture + generation params, and declared names
├── <your_model>_model.py # the Model subclass (the contract)
├── submodules.py # NodeSubmodule subclasses (the compute wrappers)
└── components/ # the actual nn.Modules (attention, decoder, etc.)
Plus a handful of things outside that package:
- an entry in
mstar/model/registry.pyso the model is discoverable, - a config YAML in
configs/mapping nodes to ranks, - an entry in
DEFAULT_CONFIGS(mstar/cli/main.py) somstar serveknows the name, - a row in
docs/models.rst, and - an OpenAI adapter, only if the model needs OpenAI-compatible routes.
Step 1 covers all of them.
Open mstar/model/registry.py and add your model to MODEL_REGISTRY (and, if it
loads weights from Hugging Face, to HF_MODELS). The dict key is the string you put
under model: in a config YAML.
The value is a (module_path, class_name) tuple, not an imported class: the module is
imported lazily through import_module when the model is actually used, which keeps
mstar.model.registry cheap to import. A direct class import here would pull every
model's dependencies into every process.
MODEL_REGISTRY: dict[str, tuple[str, str]] = {
# ...
"your_model": ("mstar.model.your_model.your_model_model", "YourModel"),
}
HF_MODELS: dict[str, dict] = {
# ...
"your_model": {"model_path_hf": "org/your-model-id"},
}There is no plugin scan, so the registry is the only place the class is discovered — but it is not the only wiring step. A model also needs:
DEFAULT_CONFIGSinmstar/cli/main.py, mapping the name to its default config file. Without it,mstar serve <your_model>exits witherror: unknown modelbefore anything loads.- a row in
docs/models.rst, which is the model list users read. - an
OpenAIAdaptersubclass inADAPTER_REGISTRY(mstar/api_server/openai/adapters.py), only if the model maps onto OpenAI semantics such aschat.completions,audio.speechorimages.generate. A model reached only throughPOST /generateneeds none. The adapter is ported to the Rust frontend in the same change (a type implementingOpenAIAdapterinrust/server/src/adapters/and its row inADAPTERS, tested bytest/rust/test_rust_openai_adapters.py):mstar serveserves HTTP frommstar-server, the Python HTTP frontend is deprecated, and a surface only it serves is unfinished. - the client-example tuples further down
mstar/cli/main.py, somstar serveprints a usage snippet. Cosmetic, but user-visible.
test/modular/test_model_registration.py enforces the registry, CLI and docs entries, so
CI fails rather than shipping a model nobody can reach.
Subclass :class:`mstar.model.base.Model` and implement its abstract methods. The
constructor receives model_path_hf (from HF_MODELS) and any **kwargs. It
normally loads the tokenizer and stores a config dataclass. Do not load weights in the
constructor. Load them in get_submodule, so that the conductor process never
allocates GPU memory.
You must implement these abstract methods:
get_graph_walk_graphs(self) -> dict[str, GraphSection]- Return
{walk_name: graph}. See Step 3 — Declare the computation graph. process_prompt(self, prompt, input_modalities, output_modalities, tensors=None, prompt_parts=None, **kwargs) -> NameToTensorListTokenize the prompt and produce the initial request tensors, for example
{"text_inputs": [token_ids]}. This method runs in the API-server data worker, after raw media tensors are loaded. It can therefore readtensors(for exampleimage_inputs,audio_inputsorvideo_inputs) to compute derived tensors such aspixel_values. The returned dict is merged into the request's tensors. See Layouts for what it returns about the request's state.input_modalitiesis the request layout: one entry per prompt element in the order written, so["image", "text", "image"]is a distinct request from["image", "image", "text"].prompt_partscarries the same sequence with the text attached, and isNonefor entrypoints that submit files and text separately. :mod:`mstar.model.multimodal` turns either into the prefill plan thatprocess_promptand the schedule builder both work from.walk_plan(self, partition_name, input_modalities, output_modalities, input_signals, model_kwargs) -> list[WalkStep]- The walks a partition runs for this request, in order, as
:class:`mstar.model.base.WalkStep` s;
[]when the request doesn't use the partition. A prefill-decode model returns[WalkStep("prefill"), WalkStep("decode", bind={"text_inputs": "new_token"})]. The conductor derives each step's input edges from the walk's graph: every external and loop-back input that no stream feeds gets the persisted signal of its own name, the onebindnames, or an empty edge (a denoise loop's seed, a chunk loop's tick). A persisted tensor a step reads is released when the step is sent, unless a later step of some partition reads it with no walk of that partition persisting the name in between; anything else stays held until the request ends.step_metadatacarries request constants to every node of the step's walk. The step that samples the reply's first token, the last prefill before decode, setsFINAL_PREFILL(frommstar.conductor.request_info) in its step metadata;media_prefill_stepdoes this for aMultimodalDecoderModel. A step whose walk a stream feeds and that runs no Loop but chunk loops runs once per chunk until the worker reports the stream's final chunk, its inputs' tensors sent with the first chunk only; any other step advances when its walk reports, and the partition is done after its last step. AValueErrorhere refuses the request with a 400, and so does a plan of[]for every partition (nothing would ever run). A plan the conductor can't run is a model bug, refused with a 500: a step whose walk is not one of its partition's, abindkey that is not an input of the step's walk, a step after one that runs once per chunk (only the stream's end, which ends the partition, moves past it), or a later step whose walk takes no input the conductor sends (it moves a worker onto a step only by sending it inputs). In a partition a stream moves (Connection.consumer_walk), the stream decides how many passes each step runs, but the conductor moves past a step on its first report when it is the last step (the partition ends) or the next step takes inputs it sends, so such a step must run once (aLoop, say). Such a partition's plan must list every walk the stream can move it to: before the partition's last pass, a report of a walk the plan doesn't run fails the request. Qwen3.5 (throughMultimodalDecoderModel), Z-Image, Orpheus and Chatterbox are references. get_submodule(self, node_name, device="cpu", tp_group=None, autocast_dtype=None, moe_parallel=None, cfg_group=None) -> NodeSubmodule | NoneBuild and return the
NodeSubmodulefornode_name, and cache the result. Load weights here, ondevice. ReturnNonefor dummy mode.tp_groupis the node's tensor-parallel communicator when the node is sharded. Pass it to the parallel-linear constructors. See :ref:`Step 6 <tensor-parallelism>`. A node on sequence-parallel ranks gets no communicator: its resources run the exchanges (see Sequence and context parallelism under Tuning resources per deployment).cfg_groupis passed only to a node whose group setscfg_size > 1; see Guidance (CFG) parallelism.moe_parallelis the node group'sep_config, parsed into aMoeParallelConfig. It is passed only to an EP-enabled node whose group sets one, so a model without MoE layers never receives it. See Expert parallelism (MoE).autocast_dtypeis the dtype in which the node's parameters must be allocated. If you build the module on themetadevice, cast it to this dtype before callingto_empty(device). Casting afterto_emptyallocates every parameter in float32 first, which doubles the peak VRAM during loading.
Note
model_kwargs reaches your model from clients through the OpenAI routes'
extra_body passthrough. The Dynamo bridge
(mstar/integrations/dynamo/bridges.py) strips OpenAI-standard fields no
model consumes (_STRIP_KEYS) before that passthrough runs. If your model
starts reading such a field from model_kwargs — the way ignore_eos is
read — also delete the key from _STRIP_KEYS, or requests arriving through
the Dynamo frontend will silently lose it.
Note
model_kwargs cross the control plane to the conductor as plain data, so a
value that process_prompt moves onto a tensor edge must not travel there
too. List those kwargs in PREPROCESS_ONLY_KWARGS: the data worker keeps the
originals for postprocess and leaves them out of the conductor's copy. Pi0.5
lists robot_state, and Cosmos3 lists action and robot_state (it passes
only their sizes on, as action_shape and robot_state_dim metadata). A tensor
left in model_kwargs fails at send
with an error that names the kwarg.
Note
The metadata process_prompt returns is laid over the request's own
model_kwargs, so a key the model reads back there (a prompt layout) must be
set in every mode and listed in INTERNAL_KWARGS: the data worker drops a
client's value for it before process_prompt. Qwen3-TTS lists its prompt
layout, speaker, language and clip digest.
Note
A stream that starts with context-only items (a voice clone's reference frames)
says so in the metadata as "stream_lead_items": {edge: n}: edge names a
streaming Connection, and n, at most that edge's policy's left context,
leads the first window as context and is never output. Leave the key out when
n is 0. The data worker drops a client's value, and the consumer partition's
workers prime the stream with it (Qwen3-TTS, VoxCPM2).
The following methods have defaults and are optional to override:
text_output, postprocess and postprocess_inline (described next),
get_request_resource_configs (described below), get_sampling_config,
get_max_output_tokens, get_autocast_dtype, load_image, load_audio,
load_video, load_decoded_image, and the partition methods described at the end
of this page. The default load_audio decodes an upload to mono at the class attribute
AUDIO_INPUT_RATE (16 kHz) and refuses one longer than MAX_AUDIO_SECONDS (30
minutes); set those rather than overriding it. load_decoded_image takes an image the
API server already holds decoded (RGB uint8 [3, H, W], from the openpi WebSocket) and
must return what load_image returns for the same pixels in a file. The default refuses,
so a model takes images only as files until it overrides this. pi0.5's load_image calls
it, so the two paths cannot drift apart. A robot policy also overrides robot_io,
returning a RobotIO: the model_kwargs key its robot state goes under, and the
[action_horizon, action_dim] shape of the float32 actions its postprocess returns.
The openpi WebSocket serves a model by these two hooks alone, so a new policy needs no
change to the API server.
text_output(self) -> TextOutput | None- Declare how text output (token ids) becomes bytes: a detokenizer, and the ids that
end a reply, which are dropped unless the request sets
ignore_eos.postprocessandpostprocess_inlinederive their text paths from it. A text chunk may end inside a character, since the API server joins a request's text chunks before decoding them, so declare a detokenizer frommstar.model.utilsthat emits each token's raw bytes (TextChunkDetokenizerorByteLevelDetokenizerfor byte-level BPE,SentencePieceDetokenizerfor a SentencePiece vocabulary) rather thantokenizer.decode([id]), which turns a split character into U+FFFD. postprocess(self, output, modality, request_kwargs=None) -> bytes- Encode a finished output tensor to bytes for the client:
utf-8for text, PNG for images, and so on. The default encodes text throughtext_outputand mono audio as headerless 16-bit PCM (a float waveform in [-1, 1] is scaled and truncated; integer samples are cast), and raisesValueErrorfor any other modality. Override it for those, or for audio with more than one channel, and hand the rest tosuper().postprocess. postprocess_inline(self, values, modality, request_kwargs) -> list[bytes]postprocessfor one decode step's inline values (one sampled token id per request), called once per step. Withtext_outputdeclared, each id maps to its bytes (the detokenizer'stoken_bytes); otherwise each value goes throughpostprocessas a one-element tensor.
Model.nodes is a read-only property. It returns the sorted set of node names that
appear in any graph walk.
A declaration is fixed at load time and is shared by every request. Parameters that differ per request, such as sampling parameters, come from a separate method:
get_request_resource_configs(
self, input_modalities: list[str], output_modalities: list[str], model_kwargs: dict,
) -> dict[str, SampleSettings]
The returned dict is keyed by the sampler's declared name. The conductor calls this method
once per request, with the request's modalities and kwargs as walk_plan gets them, and
sets the seed, derived per sampler so each sampler of a request draws its own stream. A name
that no sampler declares fails the request.
SampleSettings holds a request's knobs:
temperature,top_k,top_p,repetition_penalty,ignore_eosandmin_p.min_pfollows the HF processor order (after the penalty and temperature, before top-k/top-p) and needsSampler(min_p=True), which adds the filter to that node's captured sampler only.penalize_promptsays whether the penalty also counts the prompt's tokens.min_tokens: the sampler'seos_idsare barred untilmin_tokenstokens exist, applied after top-k/top-p (CUDA only).return_logprobsandlogprobs_mode.return_logprobsreports each sampled token's log-prob undersoftmax(logits / T), before the penalty and filters, with T = 1 for greedy requests; it reaches the client as{"logprobs": [...]}in the chunk's metadata. The model passes it into itsSampleSettings(the conductor refuses the request otherwise), its forward samples once per step and hands over row-addressed stop buffers (BatchedModelOutput.check_stop_buffers): the log-probs ride the same copy to the host. The captured sampler scores every step once it is on, so it is off by default and a deployment turns it on withresources: {<sampler name>: {enable_logprobs: true}}.logprobs_mode: "processed"(enable_processed_logprobs, CUDA) scores the distribution the token was drawn from instead: after the penalty, temperature, min-p, top-k/top-p and min-tokens floor. A greedy token scores 0 there, so a trainer sampling greedily wants"raw". It renorms every step's distribution again, about 0.15 ms at batch 1 and 0.5 ms at batch 128 on a GB200 with a 248k vocabulary, so it is a separate switch.repetition_window,top_p_firstandtop_p_min_keepstay at their defaults until a port needs them: they ask the node's sampler for a capability aSamplerdeclaration doesn't name yet.
A step whose every row samples greedily (temperature 0, no repetition penalty, no
min_tokens and no log-probs) takes each row's argmax instead of the full sampler's draw,
eagerly or in a twin of each batched capture (see Lean captures below); any other step
runs the full sampler. By default only a sampler whose vocab_size is past the one-kernel
draw (4096) does this, since a narrower one draws in one kernel already. A deployment's
resources: {<sampler name>: {greedy_capture: false}} turns it off, and with it the twins'
capture memory and time, for a deployment that never samples greedily, and true turns it
on whatever the vocabulary. The model declares nothing for it.
A model can pass through what the request set:
def get_request_resource_configs(self, input_modalities, output_modalities, model_kwargs):
keys = ["temperature", "top_p", "top_k", "repetition_penalty", "ignore_eos",
"return_logprobs", "logprobs_mode"]
return {SAMPLER: SampleSettings(**{k: model_kwargs[k] for k in keys if k in model_kwargs})}Sampling knobs mean the same thing on every model (:ref:`generation-parameters`), and the server range-checks them before your model sees them. A model's part of that contract:
- Default each knob the request leaves unset from the checkpoint's own
generation_config.json(mstar.model.utils.load_generation_defaults), falling back to your config only when the checkpoint ships none. - Raise
ValueError(a 400) for an input you cannot serve; never clamp, coerce, or fall back silently. ASamplerdeclared withoutpenaltyormin_prefuses those knobs at admission, so do not default a penalty onto it. - Declare every
model_kwargskey you read inrequest_kwargs(); any other key a client sends is logged as ignored by the server.test_model_contracts.pyfails for a registered model that does not override it. - Bound a request's length, or a denoise loop's step count, through
get_loop_caps: the conductor refuses a cap past the loop'smax_itersinstead of cutting the request short, so do not clamp it, or check it against a typed-in limit, yourself. - Draw any model-local randomness from the request's seed (
random_seedon the forward-pass info), never the global RNG.
A spec can also build its own request's config from model_kwargs
(NodeResourceSpec.request_config), which the conductor adds beside the model's. That
is how a resource the deployment declares, such as a LoRA pool, takes per-request
settings without model code. A row with no config under a resource's key may let that
resource run lean for the step; see Lean captures under the resources table.
get_sampling_config(node_name, model_kwargs) still exists as a helper for assembling
sampling parameters. The engine no longer reads it directly. Pass its result into a
SampleSettings here.
Warning
Never pass sampling parameters into a captured forward as Python scalars. Their values
are recorded into the captured kernel launch, and every replay then reuses the values
from capture time. No error is raised. Parameters that reach the sampler through
SampleSettings are stored in buffers whose addresses do not change across replays,
so they stay per-request and remain safe under CUDA graphs.
A model with an append-only array states, for every request, what its walks write into each
label of the array. The engine turns that layout into the request's footprint, the most
pages each label can hold, before the request runs. An array declares its labels
(Array(labels=...), {"main"} by default), and each request's layout comes from
process_prompt.
kv_labels(self) -> dict[str, dict[str, KVLabel]]Return
{array: {label: KVLabel}}for every label of every append-only array, naming only declared labels. This hook is temporary, until prefix keys come from layouts. A :class:`mstar.model.base.KVLabel` has two fields:growth:(walk, loop)pairs, each a walk that keeps writing after admission, bounded by the named loop. Only a named loop can bound growth, since an unnamed one is renamed in every process.keys: a :class:`mstar.model.base.PrefixKeys`, orNone. See Cross-request prefix reuse below.
A label written only by spans and scratch, such as a guidance branch, is
KVLabel().
process_prompt returns, under state_layout in its metadata, a Layout for every
label the request opens, keyed by array and then label: the labels it can name on any node
and walk, main when it names none. On an append-only array the request opens the labels
the layout names there, so the model returns no KVReqConfig for them. On one the layout
doesn't name, it opens the labels kv_labels() lists for it, or main where that lists
none, and a paged array needs a layout for each label it opens, so leaving one out fails the
request: lay out every paged array. A layout may be empty where an opened label is never
written. A Value's layout is Layout(), because a Value takes one slot per label. A layout
has four fields:
spans: the committed writes known at intake, in walk order. ASpanhas akind:"ids"for token ids from the tensorsourcenames,"digest"for KV made from an uploaded file (sourceis(modality, index)), and"rows"for KV made from anything else, such as another node's outputs. A request whose spans are not all"ids"can be rebuilt at one rank when the node'sRebuildPolicynames their walks inwalks, whatever their kind: the worker keeps host copies of what the node reads in each prefill step, and a rebuild row runs each step again from them, in the order the layout's steps say (opens_step). The policy'scopy_widthgives, for each walk where the node reads more than the ids, how many values those copies hold per KV slot, in the node's dtype, and startup raises without it. The last span must be an"ids"span, and a label the request keys must be keyed over every span. Otherwise, in a pool that decides admission, the request can only move to the host, into room the host sets aside for the pages it holds, and reserves its whole footprint when the host can't. A rebuild row gives its walk what the step read, under the step's metadata, and the last row's ids tensor gets the samples appended, so the walk must read its prompt in that shape.lengthis the slots the write fills,advancehow far it moves the position counter. A span'sscratchis the most one non-committing step writes before the span commits into those slots, as a window's denoise steps do.growth:(loop, entries)pairs,entriesper iteration ofloop.scratch: the most entries one non-committing step writes past the kept length.max_growth: the most entries the growth reserves, below what the loops' caps give, for a label whose loops stop sooner than their caps say. None takes the caps.
process_prompt may also return keyed_spans, keyed by array and then label: how many
leading spans the label's keys cover, by default every span before the first rows span.
A prompt on a walk the keys don't name sets 0, because the data worker refuses a keyed span
from any other walk. This key is temporary, with kv_labels.
A name either key doesn't declare fails the request.
The footprint adds the spans, the growth at each loop's iterations (up to the layout's
max_growth) and the scratch, and rounds up to whole pages per label. A span's own scratch counts only with the spans before
it, so a window's denoise steps and its commit count once. Under a RetentionPolicy it
takes the window's peak instead. KVReqConfig.validate refuses a request whose footprint
is bigger than its pool with a 400 at intake, and KVReqConfig.loop_room cuts the default
length of a request that asks for none until it fits. With every loop at 0 iterations, the
same sum is the prompt a request reserves when its pool admits it.
prompt_spans(plan, texts, text_walk, item_spans) builds the spans of a prompt whose
text and attachments, in prefill_plan order, are each written by their own walk, as
BAGEL lays out its prompt:
layout = Layout(
spans=tuple(prompt_spans(prefill_plan(parts), segments, walk, image)),
growth=(("decode_loop", 1),),
)
return ProcessPromptOutput(result, {
"state_layout": {KV_CACHE: {"main": layout}, GDN_STATE: {"main": Layout()}},
})get_loop_caps(self, **model_kwargs) -> dict[str, int]- The iterations each named loop may run for this request. The conductor refuses a request
whose cap is below 1 or past its loop's graph
max_iters, or whose layout at its caps holds more positions than its pool'smax_seq_len, with a 400. A request that sets no length has the defaultmax_output_tokenscut to the context its prompt leaves, andrequest_info.max_tokenscarries the cut. A model that reads a length from another key lists it inLENGTH_KWARGS. Speculation never runs past a cap, so the cap is the loop's budget. A model with an AR decode loop names it in the class attributeDECODE_LOOP, and the defaultget_loop_capscaps it atmax_output_tokens - 1, the prefill having sampled the first token; its stop check counts nothing. A model with other loops whose count each request knows up front (denoise steps, rollout frames) overridesget_loop_capsto add them, callingsuper()for the decode loop, and its stop check counts nothing either. A model whose own stop check still ends a loop at a count returns that count, so the two end the loop on the same iteration;test/modular/test_model_kv_layouts.pychecks caps per model. Growth on an uncapped loop is counted to its graphmax_iters.
Where process_prompt computes a length that a walk computes another way, such as a
media item's rows, the walk asserts the two agree: BAGEL's ViT walk against
vit_patches, Qwen3-Omni's vision encoder against the processor's grid.
A model that hasn't moved to declarations returns today's LabelLayout under
kv_layout, which adds the RetentionPolicy its committing steps declare, and
max_growth: the most slots growth writes in all, for loops that end before their
caps, as Nemotron's do. Loading fails in three cases, and a request fails in two:
- A paged pool has no labels. The message is
<Model> declares no labels for its paged pool '<key>'; add them to `kv_labels`, and return each request's layout for them from `process_prompt`. A ring pool needs none. - A label's growth or keys name a walk the model never runs. The message is
<Model> declares '<key>'/'<label>' over ['<walk>'], which it never runs; its walks are [...]. - A label grows on a loop its walk doesn't run. The message is
<Model> declares '<key>'/'<label>' growing on '<loop>', which '<walk>' doesn't run; its loops are [...]. - A request's layout leaves out a label the request opens, or lays out one the model
doesn't declare. The request fails with a 500,
<Model> returned no kv_layout for '<key>'/'<label>', which the request opens, or<Model> returned a kv_layout for '<key>'/'<label>', which it does not declare.
A request that takes a page past its footprint logs once per label, with the layout that
under-counted. A request is held only to the labels it opens, and to no pages on a label
it doesn't open. A page there logs the same line, reported apart from a real overrun on
that label so neither hides the other. Under MSTAR_KV_DEBUG_ASSERTS=1 each commit is also
checked against the layout: the walk that wrote each span, growth past its bound, scratch
past its declaration and the retention a step declares. At removal, a label left a page
or more under its footprint logs the most pages it held.
Two requests that begin with the same tokens fill the same KV pages, and the second can
attend the pages the first filled instead of recomputing them. A model opts a label in by
giving it keys, and names its checkpoint. A model with no keyed label is never cached.
The deployment switch and the per-request opt-out are described under Resource
overrides in :doc:`serving`.
PrefixKeys(tensor, walks, growth=False)tensornames the request tensor whose entries the layout's ids spans are keyed by, asprocess_promptreturns it.walks: only a write from one of these walks keeps the keys going. A write from another walk ends them, and the rest of the label is computed, never served.growth: the ids the label's growth walk samples extend the keys, so a later request that re-sends the reply matches it too.
The keys cover the label's first
keyed_spansspans. An ids span is keyed by its token ids, a digest span by its file. Each span is its own step unless it setsopens_step=False, which puts it in one row with the span before it, of the same walk (LabelLayout.row()fails a layout where it isn't): a match cuts the row, and the row keeps placing its own positions, so its walk declares the advance of the whole row whatever the cut left (Qwen3-Omni's prefill of text around an attachment).BAGEL keys its text and ViT walks and the decode that follows them:
return {"kv": {"main": KVLabel( growth=((DECODE, "decode_loop"),), keys=PrefixKeys("text_inputs", ("prefill_text", "prefill_vit"), growth=True), ), ...}}
Orpheus keys its prompt walk:
return {KV_CACHE: {"main": KVLabel( growth=(("decode", "decode_loop"),), keys=PrefixKeys("text_inputs", ("prefill",)), )}}
checkpoint_path(self) -> str | None- Return the local directory that holds the weights and
config.json. The cache hashes the checkpoint's safetensors index and config into the root of every key, so naming the checkpoint is what keeps two builds off each other's pages. The default returnsNone, which leaves the weights out of the root. A model without a checkpoint therefore needsprefix_cache_saltset to use the cache at all. Without one, the engine keeps the cache shut and logs why.
A recurrent layer can't skip tokens: its state after a token is all it keeps of the tokens
before it. On a node with a recurrent state pool, a hit is cut to the longest snapshot of
that state the pool holds under it. Name the KV cache in the pool's config
(RecurrentStateConfig(kv_cache=...)), and give the deployment prefix_slots under
the pool's resources: key. Each snapshot slot costs as much as a request slot.
- A prefill keeps the state at its last page boundary before its end, since a resend's KV hit always leaves the last page to recompute, and at its KV hit's last page boundary when that is past what it restored. A shared prefix is then served from the third request that shares it. A step longer than the largest captured bucket checkpoints less often, so that its checkpoints fit the captured steps' buffer, and keeps its boundaries on that interval.
- A prompt the step budget cuts into chunks keeps each chunk's last boundary (a chunk with
more of its prompt to come keeps its own end), so a resend restores past its first chunk,
and a later chunk's snapshot makes the earlier one the first to evict, as a reply's is.
Checkpoints fall a page apart from where a chunk starts, so the pool declares a page as its
chunk_alignmentand the scheduler cuts its chunks whole pages long. - A decode step that ends a page keeps the state there, so a later turn that re-sends the reply restores it. It replaces the request's previous decode snapshot once the KV chain keys its page; a step whose token never reaches the chain, such as the one speculated past the reply's end, leaves the previous one in place.
- A snapshot is keyed by the KV chain's key at its page. A full set of slots evicts the least recently used snapshot that no request holds, a decode snapshot before a prompt's, so the slots hold about one prompt each. A decode snapshot that a later prompt restores is kept like a prompt's from then on.
Snapshots stay off without prefix_slots, and also above one rank, without a named KV
cache, when that cache isn't on every node of the pool, or when its prefix_cache is
off or can't open at the pool's world size; in those cases the pool allocates no snapshot
slots. Neither does a pool whose cache the engine keeps shut for another reason, such as a
model that names no checkpoint and sets no salt. Every cache on the node of a pool without
snapshots stays shut, and the engine logs which pool kept it shut.
opening_prefix_caches in mstar/engine/resources/kv/config.py is that rule over the
specs. The engine evaluates it once, before it builds any resource, and opens exactly the
caches it returns of those it holds; the conductor's routing and the API server's prompt
hashing take, for each rank, the caches the rank holds and its engine opens
(prefix_caches_by_rank). Each spec answers for itself:
NodeResourceSpec.keeps_prefix_cache says whether the resource keeps a prefix cache of
its own, and restores_prefix marks one whose state a hit must restore (the pool, and
the resource over it), so another resource takes part through its spec alone.
A model with a keyed label is checked at load time. Loading fails in two more cases:
- A position resource over a keyed cache uses a scheme other than sequential. A page is
reusable only where its tokens land at the positions they were written at. The message
is
resource '<key>' places positions by <scheme>, but it sits over ['<kv key>'], which the model keyed for prefix reuse; either drop the stream or give the resource a sequential scheme. - A resource on a keyed node does not set
prefix_skip_safe, and is neither a recurrent state pool nor a resource over pools alone. A hit takes the front of the request away from every resource on the node, including resources that never saw those tokens. The message isresource '<key>' cannot serve a request whose prefix came from the cache, but it sits on ['<node>'], which the model keyed for prefix reuse; either drop the stream or give the resource the prefix hooks.
Some requests stay uncached even when the model keys a label. The engine checks each
request's inputs on the keyed walk and cuts a prefix only from inputs that
split_inputs can slice. A request whose inputs carry kwargs or
resource_step_info is never probed; BAGEL's guided requests carry requires_cfg
there. The default split_inputs slices only the sequence, so a request whose inputs
carry tensor_inputs is probed only when the submodule sets cuts_tensor_inputs
and overrides split_inputs to cut those tensors. For a label keyed over several
walks, where a keyed node streams into a partition other than its own, a request that
runs that partition is not cached either, since a hit would skip the rows the node
streams for the tokens the cache holds. BAGEL
keys an understanding prompt's text and images, and a generation's first span when it
is text: an edit's images are resized before their walks see them, and a file's digest
doesn't cover the resize. A label under a step's retention (RetentionPolicy) files
pages only up to its protected prefix, which the retention never releases.
A request on a LoRA adapter fills different KV for the same tokens, so its keys chain from
a root that names the adapter's weights: a SHA-256 over every module's A, B (with dtype
and shape) and alpha, read once when the adapter is registered (adapter_prefix_root
in mstar/engine/resources/lora/identity.py). The API server takes the root when it
selects a request's adapter (ServedModels.select) and the preprocess worker keys the
prompt from it. The root travels with the request as KVReqConfig.prefix_root so that a page
generation fills is keyed under it too. The same tokens under two adapters, or under an
adapter and the base model, never share pages; two names for identical weights do; and a
name registered again with other weights misses everything the old weights filled; a
request selected under the old weights carries their digest (LoraReqConfig.digest),
and a pool serving the new ones refuses it at ingest. A base-model request chains from
the empty root, so its keys are what they were before LoRA. Only a registered adapter
can be selected, so every request on one has its root; one that arrives without is not
cached.
Nothing on the model side is needed for this.
A model that picks a request's adapter itself (Phi-4-multimodal runs an image prompt on
its vision adapter and an audio prompt on speech) names the adapters it may pick
in Model.prompt_lora_adapters and returns its pick as ProcessPromptOutput.lora_adapter
(None for the base model; left at KEEP_ADAPTER, the client's choice stands). The
API server refuses at startup a deployment whose resources: lora: adapters: block does
not serve them all. The data worker then gives the request what ServedModels.select
gives a client that names the adapter: its digest and its root, from the API server's
ServedModels, so runtime loads and unloads apply. An adapter unloaded since startup,
or one other than the adapter the client named (None included), refuses the request
with a 400; a pick outside prompt_lora_adapters is a 500.
A resource whose KV differs per request for the same tokens (a LongRoPE table picked by
prompt length) says so in NodeResourceSpec.prefix_variant(model_kwargs): opaque
bytes, from the same kwargs its request_config reads, or None. The data worker folds
every declared spec's variant into the root after any adapter's, in declaration order and
under the spec's key (variant_root in mstar/engine/resources/kv/keys.py), keys the
prompt and sends that root on, so generated pages are keyed under it too. Requests share
pages only under the same variants, and a request whose resources name none keys under the
root it had.
declare_state(self) -> list[Array | Sampler | SeqShard] says what each node keeps between
steps, and whether its steps may split over SP ranks. The declaration says only what the model's
math needs: the shape of each entry, its dtype, which earlier entries each layer reads, and what
its kernels update. How the state is stored, such as a cache's page size and layout, depends on
the deployment, so the engine decides it. mstar.engine.state is the whole surface a model
imports, and its __all__ only grows. SubmoduleStep imports from there too, and stays out
of __all__, since it is the engine's own dataclass and its fields will change.
Every process calls declare_state and kernel_settings, so each must be a pure function
of the model's config and the deployment's settings, with no device and no process state.
Declarations are values: they hash, and changing a dict after passing it changes nothing. A
declaration that follows the deployment reads it in get_worker_graphs, as BAGEL's nodes
follow its CFG placement. Every worker and the API server check the digest of each declaration
and of kernel_settings() against the conductor's at startup, and a mismatch stops the
server, naming what differs.
An array holds, per request, per label and per declared layer, entries at log positions
0 to n - 1, with n <= max_len, or with no bound when max_len is None. An entry's
index is its log position, never its RoPE position or a ring slot.
def declare_state(self):
c = self.config
return [
Array(
KV_CACHE, nodes={"LLM"}, layers=frozenset(range(c.num_hidden_layers)),
entry=KVEntry(heads=c.num_key_value_heads, head_dim=c.head_dim),
max_len=c.max_position_embeddings,
positions=Positions(rope=Rope(theta=c.rope_theta)),
reads={ATTN: Read(mask=Causal(), op=Attend(q_heads=c.num_attention_heads))},
),
Sampler(SAMPLER, nodes={"LLM"}, vocab_size=c.vocab_size, penalty=True),
]The names you pass (the array's, each read's and update's, the sampler's, the sequence shard's)
share one namespace. Existing models define them as module-level constants in config.py. A
deployment YAML tunes them under resources: (see :ref:`Step 6 <config-yaml>`).
Array(name, nodes=, layers=, entry=, max_len=, reads=, updates=, rewritable=, labels=, positions=).nodeslists who may touch the array, andlayersholds the model's own layer ids. Layer sets of different arrays may overlap.labelsnames the streams a request may open,{"main"}by default.Entry({field: TensorType(shape, dtype=, shard=)}): what one entry holds. Shapes are global, before sharding.dtype=Noneis the engine's compute dtype, andshardis the axis split across TP ranks,Nonewhen every GPU holds all of it.KVEntry(heads=, head_dim=)builds an entry ofkandvsharded on their heads.Read(mask=, op=, layers=, walks=): how a layer reads the array. Each row reads through the one read declared for its layer and walk, and loading refuses two reads over one layer and walk.Causal()sees every entry up to the query's own,Window(size)the lastsizeof them, its own included, andBidirectional()every entry the row keeps and every entry this step adds for it. On an array that keeps entries, reads of one attention config share one resource, unless a walk reads through two of them with different masks, and each step planscausalfrom the read its walk resolves to: BAGEL readskvthroughattn(Causal()) onprefill_textanddecode, and throughattn_image(Bidirectional()) on its image and denoise walks. Loading refuses reads of two configs over one layer, since a layer reads through one resource on every walk. It also refuses a combined walk whose member walks read a layer with two masks, and a capture whose replay walks do, since a plan runs one mask. A piecewise region's step, which has no walk, can't list an array whose reads name walks, and a cross read takes nowalks. A walk can't read throughBidirectional()orBlock()on a node where a chunking policy splits it, a label'sPrefixKeyskey it, or a rebuild policy runs it again, since each regroups a row's adds across steps and would change what its tokens see. A keyed walk whose node doesn't takePrefixKeys.tensor, as BAGEL'sLLMonprefill_vit, is checked per request instead: an ids span, or a digest span that shares its step, on such a walk fails the request.Attend(q_heads=)is attention with the model's query heads, a multiple of the entry's heads.Value(name, nodes=, layers=, entry=, updates=): a rewritable array of one entry, such as a recurrent state.updatesnames the kernels that read and write one of its fields in place:CausalConv()needs a fieldconvof shape(channels, kernel - 1),GatedDeltaNet()a fieldstateof shape(h_v, d_k, d_v), andMamba2()a fieldssmof shape(heads, head_dim, state_size).max_len=0: the array keeps nothing between steps, for attention within this step only. Its one read usesBlock(), which sees only the entries of the token's own segment. See Cacheless attention (encoder towers).max_len=None: the log has no bound, and the engine keeps what the reads can still reach. Today such an array takes one label and oneWindowread throughAttend, withpacked=Falseand as many query heads as the entry, over every layer. A bounded KV holds it, one slot per request (Bounded (sink + window) caches). Its read takesdistance_bias,(rows, q_heads, T, size), whose entrydadds to the score of the key at distanced, andPositions()is each request's count of committed entries.Positions(rope=): a position counter per request and label, on an append-only array that keeps entries.rope=Noneis a counter only.Ropecarriestheta,rotary_dim,interleave,scale, the Llama 3.1 factors (all three or none), the rotation'sdtypeand atable_dtypefor a cos/sin table built once.longropetakes aLongRoPE's short and long factor sets, and a request rotates with the long set when the prompt lengthprocess_promptsets underMAX_POSITION_KWARGpasses the original context (Phi-4-multimodal). On the contract this picks the kernel, and a shared layer'srope_*arguments have no effect, so a port declares what its layer passed before. The counter is separate from log positions: a kept step advances it by each row's advance, a fork copies it, and a prefix hit advances it over the skipped spans.Sampler(name, nodes=, vocab_size=, penalty=, min_p=, eos_ids=, draws=).penaltyandmin_pdeclare capabilities, not intentions: each selects which kernel variant is recorded into the captured graph.drawsis how many draws each step that lists the sampler makes, bysampleordrew. Each is a fresh draw that advances each kept row's sampling state. Log-probs and a device loop-back take one token per step, so loading refuses both above one draw.SeqShard(name, nodes=, labels=, walks=, exchange=): the node's steps may split over its SP ranks. It extends the state contract of #1232 to sequence parallelism, and the engine builds aSeqShardSpecfrom it undername(see Sequence and context parallelism under Tuning resources per deployment). Per step the engine decides whether the tokens oflabels({"main"}by default) split, from the step alone, so it has no step of its own. It cuts the label's tensors as the node's reads take queries on it: rows of one span (packed=False, Wan2.2 and LTX-2.5's DiTs) each row alike, packed tokens or a segmented row (Qwen3.5, Qwen3-Omni's Thinker, MiniMax-H3's DiT) contiguously; loading refuses reads that take both.walksare rows' walks, as a read's are, and a combined walk splits where a walk it combines is listed;Noneis every walk. List only walks with steps of many tokens a row, so that decode's compiled frame traces the bracket straight through.exchangeis the author's default for how a split step reaches the node's paged attentions (auto,none,ulysses,all_gatherorring), which a deployment'sresources:block overrides undername. The layer stack callsscatterbefore it andgatherafter it on the shard's handle (Reaching your state), which also gives the rank's range (split_range, Wan2.2 and MiniMax-H3) and an axis other than the shard's (dim=); the reads and updates of the node's arrays exchange what a split step needs by themselves. Loading checkswalksagainst the model's walks and the shard's nodes, and refuses a label no array on those nodes opens, and a shard of more than one label, since its handle brackets one. A node with no arrays can't declare aSeqShard, since its label has none to match: it declares aSeqShardSpecdirectly, as MiniMax-H3's video decoder does.
Qwen3.5's declaration shows both kinds of state, its sequence shard and its ViT:
def declare_state(self):
c, v = self.config, self.vision_config
# gdn is an Entry of "state" and "conv"
state = [
Array(KV_CACHE, nodes={"LLM"}, layers=c.full_layer_indices,
entry=KVEntry(heads=c.num_key_value_heads, head_dim=c.head_dim),
max_len=c.max_position_embeddings, positions=Positions(),
reads={ATTN: Read(mask=Causal(), op=Attend(q_heads=c.num_attention_heads))}),
Value(GDN_STATE, nodes={"LLM"}, layers=c.linear_layer_indices, entry=gdn,
updates={CONV: CausalConv(), LINEAR_ATTN: GatedDeltaNet(qk_l2norm=True)}),
Sampler(SAMPLER, nodes={"LLM"}, vocab_size=c.vocab_size, penalty=True),
# prefills may split, never decode; Ulysses, all_gather and ring are the deployment's to opt into
SeqShard(LLM_SHARD, nodes={"LLM"}, walks={PREFILL_TEXT, *media_walk_names(self.encoders)},
exchange="none"),
]
if v is not None:
state.append(Array(VISION_TOKENS, nodes={"vision_encoder"}, layers=frozenset(range(v.depth)),
entry=KVEntry(heads=v.num_heads, head_dim=v.head_dim), max_len=0,
reads={VISION_ATTN: Read(mask=Block(), op=Attend(q_heads=v.num_heads))}))
return stateA declaration today's engine can't hold fails at load, with no layout yet and what is
missing. Every refusal names the declaration and the field.
A read with queries=ARRAY is cross-attention: its queries are laid out by that array's
segments in this step, and each request's k-th segment on one pairs with its k-th on the other.
It takes a mask that uses no query position: Block() on an array with max_len=0, and
Bidirectional() on one that keeps entries, whose context is added on the walk that carries it
and listed at span 0 afterwards. A cross read on an array that keeps entries is that array's one
read, over one label, and its queries come from an append-only array that keeps entries in pages.
It is refused at load on a node that splits its steps over SP ranks, and where its pool or the
queried array's pool is stored by position (context_parallel). A max_len=0 cross read's
queries come from an array of one label, and queries never names the
read's own array. Separate token sets are separate arrays, and arrays of one kind and geometry on one node share one resource,
each under its own label.
Under sequence parallelism an array declares nothing: the node's sequence shard
(the SeqShard in declare_state) lists the labels whose tokens split, and an array
is split on a step when the shard lists its label and splits that step. Segments always cover
the whole sequence, so declare_step is the same on every rank, and the read's resource picks
the exchange per step: Ulysses for a self read over split tokens; for a cross read, none with
only the queries split and a merge of the ranks' partial attentions with only the keys. A cross
read with both sides split, and a read on SDPA (backend: sdpa) over split keys, are
refused at load.
A read takes its queries packed, (tokens, heads, head_dim) over every row. A read whose layers
pass rows of one length, (rows, len, heads, head_dim), as a DiT's do, declares
packed=False, and a packed call to it is refused. A read on an array with a max_len above 0
takes packed queries, since the paged kernels take no others. A read on an unbounded array takes
rows, each row's span first and the rest of its length padding.
A read whose request's segments lie end to end in one row, as a DiT's packed rows and their pad
do, declares segmented=True. It takes Block(), packed=False and no queries, and it
runs one request a step. Under sequence parallelism its shard splits the whole row (axis=0),
not each segment.
A Value may hold only host fields, small values a step changes, as
Entry({}, host={"eos_sent": bool}). Each starts at its type's zero value. host(h, request,
label, layer) reads them, and ArrayStep(host={(row, label, layer): {field: value}}) writes
them when the step commits, so the next step's prepare_inputs sees the write and a step that
never ran changes nothing.
A Value with tensor fields and no updates holds state the model reads and writes whole
per row, Value(name, nodes=, layers={0}, entry=Entry({"tail": TensorType((8, 3), dtype=torch.float32),
"count": TensorType((), dtype=torch.int64)})), where a shape of () is one scalar per entry.
h.read(layer).fields[f] is [rows, *shape] in the step's row order, padding rows included, in
tensors the caller owns, so a captured forward may write them in place. A row with nothing written
yet, at span 0 or that pads the step reads its start: h.set_start({f: tensor}) in bind_state
takes tensors of one entry's shape, made at load, and zeros stand in for a field it doesn't name.
h.write(layer, **fields) writes the rows' entries in place where the forward runs it, in an eager
step or a replay, so a step refused after a write keeps it in the slots its rows already held. In a
step, a layer's reads come before its write, a field takes one write, and each row at span 1 or more
gets every field from the step's own write or from a piecewise region whose own step lists it. Such a Value takes no keep=False and sits on no
node with prefix keys. Parakeet's transducer state and both speech codecs are the examples.
Chatterbox's S3Gen keeps its vocoder tail and reference in s3gen_stream's tensor fields and its
generator state in a host field: prepare_inputs draws the chunk's noise from the state host
returns, and the step declares the next one in ArrayStep(host=), so a step prepared again after
a refused admit draws the same noise.
kernel_settings(self) is a temporary hook for what the engine doesn't pick itself yet:
splits, decode_fmha and decode_tensor_cores on a read of a paged array,
splits on a GatedDeltaNet, and max_segments_per_request, max_segments_per_step,
compile_op, workspace_mb and backend on a Block read. Under an array's name it sets the default size of the array's pool, max_num_pages
on a paged array and max_slots on a Value with tensor fields or an unbounded array, which a deployment's
resources: block overrides. A cross read on an array that keeps entries takes
decode_tensor_cores alone, and two reads that share a resource can't set one field to two
values. It refuses anything else.
The engine builds these specs from declare_state, and a deployment tunes them under the
declared names. A model that hasn't moved to declarations still writes them by hand in
get_node_resources, which returns a flat list of NodeResourceSpec objects and is
[] by default. A new model package declares its state through the contract and is listed
in PORTED in test/modular/test_state_lint.py, which fails a package that is on
neither PORTED nor its frozen UNPORTED list. get_node_resources remains for the
packages on UNPORTED. A ported file defines it only for a spec the contract can't
declare, such as the sequence shard on MiniMax-H3's video decoder, a node with no arrays, and
the lint lists that function with its reason. When the contract can't declare a model's state
yet, the mask or op and its binding in adapters.py land first, in their own PR. Every spec
has these common members:
resource_keyis the name of this resource. Three places use this name, and all three must agree: the layers bind to it,declare_stepuses it as the key of its per-resource steps, and a deployment YAML tunes it underresources:.nodesis the set of graph-node names that share this resource. When two nodes name the same resource, they share one object. For example, BAGEL'sLLMnode and its CFG branch nodes share one KV pool. A node that no spec names receives no resources. Such a node was called "stateless" in earlier versions.depends_on()returns the keys of other specs that this spec is built against.AttentionSpec,PositionSpecandDSASpecdepend on thekv_cachekey they name. If a spec names a dependency that the model does not declare, loading fails immediately, rather than during a forward pass.RaggedAttentionSpecis the one attention spec that depends on nothing: it names no cache, because it has none.combined_splits()returns aCombinedSplitfor each combined walk this spec runs on split kernels (see :ref:`combined walks <combined-walks>`); the default is none. The engine reads only this hook to order a split walk's rows, so a spec whose config carriessplitsmust return them here.AttentionSpecandLinearAttnSpecreturn their config'ssplits.
Two lockstep-parallel nodes on one worker (TP, SP or CFG nodes placed on the same ranks) must not
share a resource whose state rank 0 decides for the other ranks outside the step that uses it:
a resource implementing supports_eviction, lease_cached_prefix or page_trail, which
the paged KV cache does with or without host offload. The other ranks replay each decision
with the next step of the node it was recorded under, so two nodes on one such resource could
admit out of step. The conductor refuses that placement (Resource.journals_resident_moves
says which resources it covers); give each node a spec of its own, or place the nodes on
different workers.
The spec types are:
| Spec | What it builds |
|---|---|
KVSpec(config=PagedKVConfig(...)) |
A paged KV cache. KVConfig is the abstract base for common model geometry;
PagedKVConfig adds max_seq_len and five fields that
a deployment can tune: max_num_pages, page_size, cpu_offload_pages
(the number of pinned host pages used for offload; 0 disables offload),
prefix_cache and prefix_cache_salt. layout=KVLayout.MLA stores one
Multi-head Latent Attention latent per token in place of K and V. It needs
kv_lora_rank and qk_rope_head_dim, and sets num_kv_heads to 1 and
head_dim to their sum. Every rank holds the whole latent; only
num_qo_heads shards, and MLA attention needs it set to the model's query heads.
A pool keeps only a window of the layers a windowed attention over it runs
(AttentionConfig.sliding_window, below); its page_size, max_num_pages
and cpu_offload_pages then work as described there.
A spec whose nodes can run on several workers, as Bagel's LLM nodes do under CFG
parallel, sets leader to the node whose pool admits each request. Left unset, each
worker's pool admits requests on its own. A declared array on several nodes takes as
leader the node that runs every walk any of its nodes runs, LLM for BAGEL under CFG
parallel, and none where no node or several do.
correct_len(rid, label, delta) moves a stream's length between steps, never
between a step's admit and commit. A speculative verify step commits its whole block,
and the next declare_step takes its rejected tail back (delta < 0) before
admit. The caller also calls the position resource's correct_len, which doesn't
move by itself. A cache with KV groups refuses it. GLM-5.2's MTP is the user, and
this is interim until draft-verify is an engine resource (#1195). |
KVSpec(config=RingKVConfig(...)) |
A fixed-capacity frame ring. It adds tokens_per_frame, one
RingKVLayerConfig per layer, and the deployment-tunable num_sessions.
Ring storage is currently paired with FlexAttention. |
BoundedKVSpec(config=BoundedKVConfig(...)) |
Fixed-size per-request KV with sink + window retention, and its attention. A
request's stream is an optional read-only source (e.g. a voice prompt's keys,
shared by the requests that use it) followed by what it writes; retention maps
the source length to the SinkWindow kept (the first sink entries and the
last window). Each step declares its rows' StreamPosition and writes
(BoundedKVStep), so rows at different positions batch, and nothing is
reserved or released per step. attend takes an optional per-distance score
term (rel_bias) for relative-position attention. Float32 by default (Triton
kernels). See Bounded (sink + window) caches. |
RollingKVSpec(config=RollingKVConfig(...)) |
Per-request sink + window K/V kept compacted in stream order, in regions of a pool
reserved at startup (pool_tokens), with no attention of its own: a step writes its
tokens with write and hands window (one contiguous range) to whatever attention
the model runs, a ragged one say. For long chunks where that kernel's speed matters (a
causal video DiT). See Rolling (compacted) caches. |
AttentionSpec(config=AttentionConfig(kv_cache=...)) |
Self-attention planned over the named cache. backend selects
AttnBackend.AUTO (the default, which lets the engine pick the platform's
paged backend, FlashInfer for now), AttnBackend.FLASHINFER,
AttnBackend.DENSE, AttnBackend.FLEX or AttnBackend.FLASHINFER_MLA.
Leave it at AUTO unless the model needs a specific kernel. FlashInfer, dense
and MLA attention require a PagedKVConfig; FlexAttention requires a
RingKVConfig. DENSE runs one varlen pass over each stream's frozen
prefix (gathered from the pages once) and the step's fresh K/V, eager only:
FA3 (AITER on ROCm), or FlashInfer's ragged prefill where FA3 has no kernel for
the GPU, the same choice the ragged resources make (dense_or_flashinfer), with
flashinfer_backend picking its kernel. The engine plans and runs a tiny layout on
that kernel when it builds the resource, so a backend the GPU or FlashInfer build
cannot run stops the server with a ValueError naming it. Its wrappers share one
workspace, taken at build.
flashinfer_backend selects a kernel generation: "auto", "fa2" or
"fa3" when the FlashInfer backend is selected. decode_tensor_cores
(default True) runs one-token decode rows on FlashInfer's tensor-core
kernel; False picks the CUDA-core decode kernel, which can be faster for
MHA and narrow GQA (few query heads per KV head). Measure before changing it:
Whisper, Qwen3-ASR and Qwen3-TTS set False in kernel_settings. decode_fmha (default False) runs
captured one-token decode rows on FlashInfer's trtllm-gen fmha kernel on Blackwell,
with no FlashInfer plan per step; a geometry trtllm-gen has no kernel for keeps
the paged kernel, with a warning. Eager steps and prompt rows stay on the paged
kernels. Qwen3.5 declares True. sm_scale (default
1/sqrt(head_dim)) sets the softmax scale; FlashInfer and FLASHINFER_MLA
read it, and the other backends refuse it. FLASHINFER_MLA is
Multi-head Latent Attention over a KVLayout.MLA cache. It needs sm_scale,
the softmax scale of the model's original qk head dim. Its run takes the
absorbed query ([tokens, heads, kv_lora_rank])
with the rope part as q_pe, and returns the output in the latent space, so
the layer applies W_UV. FlashInfer's kernel covers a 512-wide latent with a
64-wide rope part, or none from FlashInfer 0.6.18; any other shape runs a torch
fallback, which cannot be captured in a CUDA graph. The layer is MLAAttention
(mstar/model/components/mla.py); its config picks rotated decoupled dims (GLM-5.2)
or none (GLM-5.3).
sliding_window=N lets each query attend to the last N positions, its own
included, and the cache keeps only about the last N tokens of this
attention's layers per request, returning the rest to the pool, so long requests
take far fewer pages. It needs the FlashInfer backend, causal attention, a paged
NHD cache and layers=(...), the KV layer indices this attention runs (the ones
set_layer_idx takes). The cache's other layers keep every token, and the
attentions over it that declare no window run them (Command A+); each attention
refuses a layer it doesn't run. The cache works this out from its attentions
before anything is built (derive_kv_groups); a window at least the cache's
max_seq_len keeps every token, so it needs no layers and costs nothing.
Startup refuses a window over another kind of cache, a window without layers,
a layer declared twice, two different windows over one cache (one per cache for
now), a window over every layer of a cache, other layers that no attention
without a window reads, cross attention over such a cache, and a backend other
than FlashInfer; its kernel must be auto or fa2, or fa3 on SM90, and
from SM100 auto runs fa2. The window's pages hold page_size × full
layers / sliding layers tokens, which must be whole. Left unset, the page
(tokens_per_page) is the smallest whole one whose pages of either group hold at least 64 tokens
(192 for 13 full and 39 sliding layers, 184 for 16 and 46); a page_size in the
YAML must be a multiple of sliding layers / gcd(full, sliding). A window is
kept in whole pages of its layers plus up to one more, so with few sliding layers
among many full ones (1984-token sliding pages at 31 full and 1 sliding) a short
window saves little. max_num_pages and cpu_offload_pages count pages of
every layer at the declared page_size, or at 128 tokens when it is unset, so
the pool's bytes don't move with the derived page. Such a cache doesn't prefix
cache yet, so its prefix_cache goes off (logged at info), and a serving config's
prefix_cache: true is refused. With context_parallel the window's layers stay
whole on every partner (Context parallelism, below). It doesn't fork, serve a node on another instance
(replicas included) or hand out
write_slots yet either; each of these is refused. |
DSASpec(config=DSAConfig(kv_cache=..., topk=...)) |
DeepSeek sparse attention (DSA) over the named KVLayout.MLA cache. A step
declares DSAStep(), and the resource takes each request's rows from the KV plan.
The model's indexer layers store their index keys (store_keys) and pick each
query's top topk positions by the indexer's score (select); a layer that
reuses an earlier layer's pick reads it with shared_selection. attend runs
absorbed MLA over just the picked latents. The keys sit in per-request buffers
outside the paged pool and the selection runs per request on the host, so the
resource is eager only, a prompt must fit in topk and past topk a request
decodes one row per step. Once one row of a batch passes topk, every row of it
attends its pick. GLM-5.2 declares it under dsa_long_context. Planned: the keys
move onto the MLA cache's pages under its page table, so the allocator, eviction and
prefix caching cover them, and selection gets fixed-shape buffers so decode captures. |
CrossAttentionSpec(config=CrossAttentionConfig(...)) |
Attention over a paged context that is written once and never extended.
backend defaults to AttnBackend.AUTO. Only FlashInfer is implemented.
A step whose rows each query one token runs FlashInfer's decode kernel, and
decode_tensor_cores picks its variant as for AttentionConfig.
See Cross-attention (encoder-decoder models). |
RaggedAttentionSpec(config=RaggedAttentionConfig(...)) |
Cacheless (ragged) varlen self-attention over the segments packed into one
forward. Nothing is paged, and nothing carries to the next step.
RaggedAttentionConfig holds num_qo_heads, num_kv_heads and
head_dim (all pre-sharding, as in KVConfig), an sm_scale that
defaults to head_dim ** -0.5, backend (RaggedAttnBackend.AUTO, the
default, RaggedAttnBackend.FLASHINFER, RaggedAttnBackend.DENSE, or
RaggedAttnBackend.CUDNN), flashinfer_backend, the
two CUDA-graph ceilings max_segments_per_request (max_segments_per_step
for a one_row_per_step region) and
max_tokens_per_request, compile_op and workspace_mb. CUDNN runs torch's
cuDNN SDPA over equal-length segments (see Cacheless attention (encoder towers)).
DENSE runs FA3
flash_attn_varlen (AITER on ROCm), or FA3's fixed-length kernel for an eager
plan of one non-causal segment on Hopper (and, at head dim 128, once per segment
for a plan whose segments are all at least 1536 rows), and falls back to FlashInfer
if that kernel is missing. The heads are split across the node's TP ranks (TP × SP over
a label the node's shard lists, on a node whose shard splits, below).
workspace_mb is the floor of each label's FlashInfer workspace per slot (default
MSTAR_WORKSPACE_BUFFER_MB, 512, which doesn't apply once it is declared): a plan that
asks for more at startup grows it. A node with many labels can declare less, since a
ragged prefill only uses it for split-KV partials. The spec's labels are
the labels its plans run under that may split (a within-step array's read fills
them from its label): where the node's sequence shard lists one, the resource
attends it by Ulysses, splitting the heads over TP × SP on a node whose shard
splits (SeqShardSpec.splits; see Sequence and context parallelism).
compile_op, workspace_mb and labels are the model's declaration; the
serving config's resources: overrides take only the backends and the ceilings.
See Cacheless attention (encoder towers) for compile_op. |
RaggedCrossAttentionSpec(config=RaggedAttentionConfig(...)) |
The cross-attention counterpart: one span of each request attends another
(RaggedCrossAttentionStep(pairs=...)). Same config. See Cross-attention
between spans. |
RaggedBlockCausalAttentionSpec(config=RaggedAttentionConfig(...), block_size=...) |
Block-causal self-attention within each span (AttentionStep(causal=False)): causal
between block_size-token blocks, bidirectional within one. Same config. See
Block-causal attention within spans. |
RaggedKeyPrefixAttentionSpec(config=RaggedAttentionConfig(...)) |
Self-attention within each span over a prefix of its keys
(KeyPrefixAttentionStep(key_lens=...)): every token queries, only the first
key_lens[i] are keys. Same config. See Key padding within spans. |
BlockCausalAttentionSpec(config=AttentionConfig(kv_cache=...), block_size=...) |
The same mask over a paged KV cache: a row's new tokens attend its cached blocks and their own, and a step runs only new tokens. FlashInfer only. See Block-causal attention over a KV cache. |
PositionSpec(config=PositionConfig(kv_cache=...)) |
Position tracking and RoPE. scheme is PosScheme.SEQUENTIAL or
PosScheme.BLOCK. The RoPE parameters are set here: rope_theta,
rope_scale, and the Llama-3.1 parameters low_freq_factor,
high_freq_factor and old_context_len. Layers rotate with
apply_qk(q, k, label). pos_ids= rotates at positions the model passes
instead of the label's plan, and k=None rotates one tensor. interleave
selects GPT-J pairing, rotary_dim a partial rotation, and rope_table_dtype
a cos/sin table built once (GLM-5.2 uses float32); cos_sin_table hands it to a
kernel that rotates inside a larger fusion, and interleave says which pairs that
kernel must turn. A model with learned position
embeddings also declares this spec, because it needs the position counter. Such a
model never applies RoPE.
longrope=LongRoPE.from_hf(config) (Phi-3, Phi-4-mini) builds a short and a long
table, each with its per-dimension factors and attention scale (LongRopeManager).
A request whose MAX_POSITION_KWARG (set by process_prompt; the data worker
drops a client's value) passes
original_max_position_embeddings rotates with the long one for its whole life,
others with the short one. Set it to the length the model's reference picks its table
by: the prompt's for Phi-3 and Phi-4, not prompt plus max_tokens. The spec builds the request's config from that kwarg and
names the long table in its prefix keys (PositionSpec.prefix_variant), so the two
cannot disagree. Explicit pos_ids are refused under it. On resident tables a
row keeps its table id's offset beside its position's, both fixed between host
steps. A cos/sin table on CUDA needs a rotary_dim that is a multiple of 32 with NeoX
pairing, of 16 with GPT-J pairing (interleave). |
SamplerSpec(vocab_size=..., enable_repetition_penalty=...) |
A sampler with its own per-request parameter buffers, philox stream, and optional
seen-token mask. enable_repetition_penalty declares a capability, not an
intention. It selects which kernel variant is recorded into the captured graph.
Whether the penalty runs on a given step is decided from the
repetition_penalty values of the resident requests.
enable_min_p, max_repetition_window, min_tokens_stop_ids and
enable_top_p_first are capabilities of the same kind. See
SampleSettings above. |
RecurrentStateSpec(config=RecurrentStateConfig(...)) |
Fixed-size per-request state that carries between steps (an LSTM's h and
c, a cursor, a conv tail, a Mamba or gated-DeltaNet layer's conv window and SSM
state, a streaming encoder's caches): one slot per request and label, labels
naming every label a request's steps address or fork into (("main",) by default)
so that admission reserves a slot for each, blocks naming a shape and dtype for
each piece (dtype None: the engine's compute dtype,
what its KV caches store), num_layers copies of each, max_slots slots
(deployment-tunable) with slot 0 the sink that padding rows address. A recurrent
kernel reads the step's addressing. declare_step gives each row a
RecurrentStep segment with span >= 1. The forward reads its rows with
pool.gather(name, start=...) and writes them back with
pool.scatter_(name, value); both follow the step's plan, so there is no
addressing to fetch, and eager and captured steps reach the same slots. A row with
no saved state (a request's first step, or a padding row) reads start, zeros
when none is given. gather / scatter_ need the sink, so leave
disable_sink_slot unset. A model on the contract declares a Value instead. |
SeqShardSpec(labels=..., axis=..., walks=None) |
Whether a step's tokens split over the node's sequence-parallel ranks, and each
listed label's per-rank ranges. Its step is the node's segments (declared by the
model, as for every resource), and the model brackets its layer stack with the resource's
scatter / gather. A model on the state contract declares a SeqShard
instead, from which the engine builds this spec (Step 2a — Declare your state). See
Sequence and context parallelism under Tuning resources per deployment. |
LinearAttnSpec(config=LinearAttnConfig(recurrent_state=..., variant=...)) |
Linear attention planned against a RecurrentStateSpec pool that holds its
state: gated DeltaNet (LinearAttnVariant.GDN), Kimi delta attention (KDA:
fused Triton kernels on CUDA, a torch reference elsewhere) or Mamba-2. A layer
calls gated DeltaNet through causal_conv and gated_delta_net, Mamba-2
through causal_conv and mamba2, and KDA through KDACallable
(mstar/engine/resources/convenience.py), which hands the kernels the layer's
blocks of the pool, so the layer never names one. A KDA pool built with
DeltaNetGeometry.to_blocks(speculative_tokens=k) also runs verify steps
(LinearAttnStep(speculative=True)), and KDACallable.set_prefix_len
settles each one's accepted drafts. The layers are shared:
ParallelGatedDeltaNet and ParallelKimiDeltaAttention
(mstar/model/components/distributed/, over KDAConfig). A KDA layer tells
the resource its gates as it binds: with Kimi K3's full-rank output gate
(full_rank_output_gate) the resource runs the torch reference. Where the fused
kernels run, activations other than bf16 or a forget-gate bound below -5 are a
ValueError at load. The torch reference's prefill can't be captured
(KDAManager.prefill_capturable is False), so the resource refuses a prefill
bucket on it and the model leaves those buckets out. |
LoraSpec (built from lora_roots) |
Per-request LoRA adapters on the node's LoRA-capable linears: a pool of device
slots, a pinned host tier, and the per-step addressing a LoRA kernel reads.
Opt-in (NodeResourceSpec.opt_in): built from the model's lora_roots()
rather than this list, and only when the serving YAML has a resources: lora:
block. One spec spans every node adapters reach, with a pool per node. See
Per-request LoRA adapters. |
Lean captures. A resource whose kernels can be left out of a step whose rows all
allow it sets Resource.lean_capture, answers lean_for(request_ids) from state
every rank of a lockstep instance sets alike (its ingests), and says which kinds of capture
get its twin (lean_captures). Beside each bucket whose step drives such resources, the
engine captures a twin for each set of them, keyed by that set (BucketKey.lean). A step
replays the twin of every resource its rows let run lean, else one with fewer of them
(largest set first, ties in declaration order), else the plain bucket; the engine names the
set in StepContext.lean, eager steps included, and each resource checks its rows again
at plan. A resource whose callers read its plan inside the forward also overrides
Resource.outside_step, which the engine enters around compile warm-ups outside any step
(the LoRA pool's runs them lean). Twins are sized, planned and captured after every plain
bucket, and a twin left out (under the capture budget, say) is logged at INFO and is not a
dropped bucket: its steps replay the plain bucket. Lean resources: the LoRA pool, for
steps none of whose rows names an adapter, in every kind of capture; and a sampler (see
SamplingReqConfig), for steps whose rows all sample greedily, in batched captures. A
model declares nothing for these.
Orpheus's declarations build four specs for its one autoregressive node, a KVSpec, an
AttentionSpec, a PositionSpec and a SamplerSpec. Its snac_decoder node appears
in no declaration, so it receives no state:
# mstar/model/orpheus/config.py: the keys its SampledTokenDecoder reads, and its attention's
KV_CACHE, SAMPLER, ROPE = keys.KV_CACHE, keys.SAMPLER, keys.ROPE
ATTN = "attn"
# mstar/model/orpheus/orpheus_model.py
def declare_state(self) -> list[Array | Sampler]:
c, scaling = self.config, self.config.rope_scaling
return [
Array(
KV_CACHE, nodes={"LLM"}, layers=frozenset(range(c.num_hidden_layers)),
entry=KVEntry(heads=c.num_key_value_heads, head_dim=c.head_dim), max_len=c.max_position_embeddings,
positions=Positions(rope=Rope(
theta=c.rope_theta, scale=scaling["factor"], low_freq_factor=scaling["low_freq_factor"],
high_freq_factor=scaling["high_freq_factor"],
old_context_len=scaling["original_max_position_embeddings"],
)),
reads={ATTN: Read(mask=Causal(), op=Attend(q_heads=c.num_attention_heads))},
),
Sampler(SAMPLER, nodes={"LLM"}, vocab_size=c.vocab_size, penalty=True),
]Which state does a node need?
The rows below name specs where a declaration doesn't reach them yet.
| Node | Declare |
|---|---|
| Self-attending LLM (text decode, or an LLM used as a denoiser in a flow loop) | An Array with positions and an Attend read. Add a Sampler if the
node samples. |
| Decoder of an encoder-decoder model | The row above, plus a second Array for the encoder context with a
Bidirectional() read whose queries names the first; see Cross-attention
(encoder-decoder models). |
| An LLM with DeepSeek sparse attention (GLM-5.2) | The specs above over an MLA cache (KVLayout.MLA, AttnBackend.FLASHINFER_MLA),
plus a DSASpec over the same cache. |
| A node that carries per-request state between steps of a loop (an RNN-T decoder, recurrent attention) | A Value with tensor fields, read and written whole, and for a linear-attention
layer stack a Value with a GatedDeltaNet or a Mamba2. |
| A node that samples with two different sets of parameters | Two SamplerSpec objects that both name the node, under different keys. |
| An encoder tower that attends within the segments of one packed forward | An Array with max_len=0 and a Block read, and nothing else: there is no
cache to page and no counter to advance. Needed only if the tower's attention must
run inside a CUDA graph; see Cacheless attention (encoder towers). |
| ViT, VAE or audio encoder, codec decoder, projection stage, combine stage | Usually nothing. Declare nothing that names the node, unless the node needs the row above. |
If a node's forward samples with different parameters, declare one Sampler per parameter
set. Both name the same node. Qwen3-Omni's Talker node needs this. The Talker LLM samples
codec group 0. The CodePredictor samples groups 1 to N-1, using its own vocabulary size and
no repetition penalty, draws times per step:
Sampler(TALKER_SAMPLER, nodes={"Talker"}, vocab_size=c.talker_text.vocab_size, penalty=True),
Sampler(CODE_PRED_SAMPLER, nodes={"Talker"}, vocab_size=c.code_predictor.vocab_size,
penalty=False, draws=c.num_code_groups - 1),The forward samples through each one's handle, and the rows come from the step:
layer0 = sample(self.talker_sampler, logits) code = sample(self.cp_sampler, logits)
Each sampler owns its own buffers. The two parameter sets are therefore independent, per-request, and safe under CUDA graphs.
A decoder that attends to a fixed encoder context declares two arrays that keep entries: its own self-attention array, and a second array that holds the context.
Use two separate arrays, not two labels on one. The self-attention resource plans one wrapper per label of the cache it names. If the context shared that cache, the resource would re-plan the context on every decode step. The context is large and never changes, so this work would be wasted.
Whisper (mstar/model/whisper/) is the reference implementation. The context array's one
read is Bidirectional(), and queries=KV_CACHE lays its queries out by the self-attention
array's segments:
Array(KV_CACHE, nodes={DECODER_NODE}, layers=layers, entry=kv, max_len=c.max_target_positions,
positions=Positions(), reads={ATTN: Read(mask=Causal(), op=attend)}),
Array(CROSS_KV_CACHE, nodes={DECODER_NODE}, layers=layers, entry=kv, max_len=c.max_source_positions,
labels={CONTEXT_LABEL},
reads={CROSS_ATTN: Read(mask=Bidirectional(), op=attend, queries=KV_CACHE)}),There is no special API for writing the context. You express it in the step declaration.
The prefill step declares a non-zero span on the context array's ArrayStep. Every later
step declares a span of 0. A span of 0 reads the stream without extending it.
The encoder K and V are added once per request, in preprocess under
ArrayStep(eager_add=True), so the captured prefill is sized by its prompt alone. Each
layer's ParallelCrossAttention(context_kv_key=CROSS_KV_CACHE, layer=i) reads the context
through its handle, and its add_context adds the layer's K and V. Whisper's decoder calls it
for each layer:
for layer in self.layers:
layer.encoder_attn.add_context(encoder_states)
A model with several context sources, such as "audio" and "image", declares one context
array per source, and each cross-attention layer names its source's array in
context_kv_key.
A ViT or audio encoder attends within its own sequence and keeps nothing afterwards. Several images are packed into one forward, and each must attend only within its own span. That is varlen, or "ragged", attention: the whole layout belongs to this step, and nothing carries to the next.
Such a tower needs no state at all if it calls a varlen kernel directly, passing
cu_seqlens as an argument. Declare an array with max_len=0 and a Block read when
the tower must run inside a CUDA graph. The engine builds a RaggedAttentionSpec from it.
The plan is what makes that possible: the engine plans the layout
outside the graph, into buffers whose addresses do not change, and the captured region
attends through them. A kernel that reads cu_seqlens as an argument, or builds its
mask from it, cannot be captured this way.
RaggedAttnBackend.AUTO (and an explicit FLASHINFER) plan through FlashInfer's
ragged prefill wrapper. For a self-attention whose config names neither a backend nor a
flashinfer_backend (None, the default; an explicit auto is FlashInfer's own choice), AUTO first reads the ragged_attention kernel table
(mstar/kernels/tables/ragged_attention/, see Tuned kernel configs) for the rank's head
geometry on the GPU. An eager plan runs the row its layout's longest query segment falls in, so
one slot can run a different kernel from one layout to the next; a captured bucket runs the row of
the layout that first plans it. On sm_103 the table names torch's cuDNN SDPA (RaggedAttnBackend.CUDNN) for
klein 4B / 9B and Z-Image's geometries (24, 32 and 30 heads of 128; Wan2.2-TI2V-5B's 24 heads read
klein 4B's entry, LTX-2.5's 32-head video self-attention klein 9B's): under a CUDA graph FlashInfer's
auto stays on FA2 there (its cutlass plan can't be replanned in graph mode, and its cuDNN path needs
cuDNN 9.24), which is 3.5-5x slower from 1k to 64k tokens. An entry whose larger sizes no served model
has run bounds them with a numeric last row and a "*" row naming flashinfer, the resource's own
default. A GPU of the architecture reads the file
whichever model it is, and the build-time log names the GPU the row was swept on. The cuDNN wrapper runs a layout of equal-length
segments, a DiT batch's, as one dense [segments, S, H, D] call. A captured one replays only
the layout it was planned with, so a captured bucket takes it only when the step that plans it
declares AttentionStep(fixed_layout=True) (ArrayStep(fixed_layout=True) on a declared
array), and stays on FlashInfer otherwise. The DiT scaffold's denoise step declares it on each
of its arrays: every row of a denoise bucket, padding rows included, spans the bucket's tokens.
An eager layout of unequal
segments runs FlashInfer. It is self-attention only. A config that names backend: cudnn
fails at build where cuDNN can't run it: on a cross-attention or block-causal resource, under
GQA, in a dtype other than bf16 / fp16, or at a head dim FlashInfer pads. Where no table has the
geometry, AUTO is FlashInfer with its own auto choice. RaggedAttnBackend.DENSE plans the same layouts into static
cu_seqlens buffers and runs FA3 flash_attn_varlen (AITER on ROCm) instead, with
no page scatter. If that kernel is not importable, or the spec is a
RaggedBlockCausalAttentionSpec (whose overlapping prefixes varlen cannot express),
the factory logs a warning and builds FlashInfer. The fallback does not cover a GPU or
dtype the kernel lacks: fa3-fwd 0.0.3 is built for Hopper (sm_90a) and runs bf16
but not fp16.
Qwen3.5's tower is the reference on the contract:
Array(VISION_TOKENS, nodes={"vision_encoder"}, layers=frozenset(range(v.depth)),
entry=KVEntry(heads=v.num_heads, head_dim=v.head_dim), max_len=0,
reads={VISION_ATTN: Read(mask=Block(), op=Attend(q_heads=v.num_heads))})A layer adds its k and v and reads with its q, and the handle hands them to the kernel
together. The add keeps nothing, so keep=True raises, and a read takes the add before it:
self.state.add(self.layer, k=k, v=v)
out = self.state.read(self.layer, q)On an array with max_len=0 the calls add no break of their own. With compile_op set through
kernel_settings(), the read stays in a compiled caller's graph, so the region keeps one graph
across the attention. Without it, the read breaks at the kernel. A step declares state={VISION_TOKENS: ArrayStep()} over segments that are the
entire layout, and a captured region names the array (Piecewise CUDA graphs (capturing an
inner loop)).
BAGEL's SigLIP2 tower is declared the same way, over VIT_TOKENS, and sets
max_segments_per_request: 1 in kernel_settings(), since a request has one image. A
tower that hasn't moved writes its RaggedAttentionSpec by hand, over the encoder node only,
naming no cache.
On that path a layer binds the key as it binds any other resource, then attends through the resource's
run(q, k, v, label=None). There is no KV write to pair it with, so no
AttentionCallable is involved. A compiled caller breaks at run; with
RaggedAttentionConfig(compile_op=True), run traces as the mstar::ragged_attention
custom op instead, so the region keeps one graph across the attention (worth it for a
compiled, uncaptured region; it costs startup time and capture memory where the region
is also captured). An eager call goes straight to the kernel.
The step is an ordinary AttentionStep, and its segments are the entire layout.
There is no cache to read the layout off, so a label that carries no segment in the
declaration cannot be attended: the forward raises, rather than silently reusing the
previous step's plan. Because the tower is normally captured as a piecewise region, that
declaration lives in the region's declare_step, and names no KVStep. See Piecewise
CUDA graphs (capturing an inner loop).
A diffusion transformer whose streams attend each other (video queries over text keys, or
audio over video) has several spans per request and no cache either. Declare each span as
its own label, and the cross-attentions as a RaggedCrossAttentionSpec beside the
RaggedAttentionSpec for the self-attention: a resource is one kind of attention. Its
step names the (q_label, kv_label) pairs:
RaggedCrossAttentionStep(
segments=segments, # one Segment per (request, label) the pairs use
pairs=(("video", "text"), ("audio", "video")),
)Each pair is planned as its own layout, pairing each request's q_label span with that
request's kv_label span; cross-attention is never causal. A layer attends through
resource.run(q, k, v, label=cross_label(q_label, kv_label)), passing the key
span's tokens as k and v. One resource has one head geometry, so a model whose
attentions differ in head count or head dim declares one spec per (kind, geometry).
LTX-2.5 is the reference (mstar/model/ltx2_5/submodules.py).
Called with return_lse=True, a ragged or ragged cross-attention resource's run on the
FlashInfer backend also returns each query row's log-sum-exp ([total_tokens, H], fp32,
natural log) from the same plan, which is what merging attentions over disjoint key sets
needs; a compiled caller breaks there even with compile_op. A label exchanged over SP ranks
on its step can't return one: on SP ranks of a node whose shard splits, that is every label of a
Ulysses resource (one over a label its shard lists; copied steps included), and a cross-attention
pair whose keys split on the step. Where keys split, the resource merges the partials itself
(LTX-2.5's audio queries over each rank's share of the video tokens, merged by gather_merge).
A Whisper-style encoder trained for streaming masks attention by chunk: a frame in chunk
b sees every frame of chunks 0..b and nothing later. Declare it as a
RaggedBlockCausalAttentionSpec with the chunk length as block_size, and step it like
any ragged self-attention, one segment per independently attending span, with
AttentionStep(causal=False). Blocks are counted from each span's start, so a span need
not be a multiple of block_size.
The prefixes overlap, which a ragged layout cannot express, so this kind runs FlashInfer's paged prefill over the packed keys viewed as one-token pages, each query block listing its prefix; nothing is copied. Under a captured graph the block count and page list are sized for the worst layout the bucket's token count allows.
Note
The eager varlen path is sometimes good to have. The BAGEL ViT has a head dimension of
72, which the FlashInfer kernel — it supports only a few fixed head dimensions — forces
us to zero-pad to 128. So that tower attends through VIT_TOKENS only when it is
replaying a captured graph, and runs flash-attn varlen otherwise.
An encoder run over padded clips can carry a key padding mask: a span's last frames are
padding as keys but still compute outputs, which can reach what the model keeps (Step-Audio2's
Whisper encoder masks one or two frames a clip). Declare a RaggedKeyPrefixAttentionSpec
and step it with KeyPrefixAttentionStep, one segment per span and key_lens parallel to
the segments, each in 1..span. The resource gathers each span's key prefix into a key layout
of its own and runs the cross-attention kernel against it; under a captured graph the gather
index is a static buffer the plan restages, padded to the bucket's tokens.
Under a block-causal mask a frame's output never depends on a later block, so what a stream
has already computed stays valid and need not be run again. Declare a paged KVSpec and a
BlockCausalAttentionSpec on it, with the block length as block_size, and step only the
new tokens: KVStep(commit=True) and AttentionStep(causal=False), the layer writing K/V
as usual. A step's new tokens must start on a block, and may run several blocks: the KV plan's
rows expand into one paged request per block, each reading its row's prefix pages. CosyVoice3's
flow solves its hops this way, one KV layer per DiT block per Euler step.
A streaming decoder that keeps a fixed amount of context per request (a prompt or voice
prefix plus the last N steps, as StreamingLLM-style sinks or a streaming vocoder's chunk
caches do) declares a BoundedKVSpec, or on the state contract an Array with
max_len=None. A request takes a slot when it first runs and
keeps it; the window is a ring overwritten in place, so a step reserves and releases
nothing. A step's layout is at most five key ranges and three write ranges per row,
computed on the host from the position the step declares, which is also what lets rows at
different positions share a batch and a captured graph.
The model calls attend(layer_idx, q, k, v, source, plan): each row attends its
retained keys in stream order and the step's keys it writes (all of them for a row that
writes none), unmasked, and its writes land after. Pass
source ([sources, rows, H, L, 2 * head_dim], keys then values) when requests start
from shared prefixes that the slots should not copy; each row reads the one its
StreamPosition.source names, so rows of different prefixes share a batch and a graph. rel_bias adds a term that depends only on
the query-key distance, for Transformer-XL-style relative positions. A streaming vocoder
like MiniCPM-o's token2wav fits three of them: its two conformer caches and its DiT's.
Two flags cover retention rules that are not quite a token FIFO.
SinkWindow(window_includes_step=True) counts the step's own tokens toward the window.
BoundedKVConfig(reverse_step_order=True) appends each step's tokens last-first, which
only makes sense for attention that ignores key order; it decides which of the oldest
step's tokens leave first.
By default the window is per step: every query of a step sees the same keys, the retained
ones and the step's own. A streaming encoder whose frames each see a fixed left
context (a conformer's att_context_size=[left, 0]) sets
BoundedKVConfig(retention=SinkWindow(0, left), query_window=left) instead: each query
attends the keys at distances 0..left from it, so a step of several frames is causal
within itself and matches the offline mask. rel_bias is then indexed by that distance,
[rows, H, T, left + 1], and passed unscaled (the kernel scales the sum by
head_dim ** -0.5). A per-query window takes no sink, and its right context is 0.
retention may be a single SinkWindow when it does not depend on the source length.
BoundedKVConfig(dtype=None) stores the cache in the engine's KV dtype, for layers that
compute in it. The streaming FastConformer (Nemotron VoiceChat's ENCODER_KV, an Array
with max_len=None and a Window read, which the state contract holds in such a cache) is
the per-query example.
A causal video DiT that denoises chunk after chunk against a sink and a sliding window of
earlier chunks (Self-Forcing's rolling cache; LingBot-World's DIT_KV) declares a
RollingKVSpec. Its chunks are thousands of queries over tens of thousands of keys, so
the attention kernel dominates the step, and bounded KV's Triton kernel (built for short
streaming steps) is several times slower there than the ragged FlashInfer one. The rolling
store therefore runs no attention: it keeps the stream's first sink tokens and its last
window (the step's included) in stream order, shifting the window left behind the sink
when a step would overflow, so a step's keys are always one contiguous slice.
Each step declares its rows' RollingPosition(written, sink, window) in tokens
(RollingKVStep), so the retention can follow a request's own geometry. plan applies
the shift, once per step even when planned twice; the model then calls write(layer, k, v)
(as often as its denoise loop rewrites the chunk; the last write stays) and attends
window(layer) with its own kernel.
The K/V live in one pool reserved at startup, pool_tokens a layer. The model declares
each request's size by returning RollingKVReqConfig(tokens=sink + window) from
get_request_resource_configs; a request it declares nothing for fails at its first step.
A request waits in the engine's admission queue, in arrival order, until the pool has its
room, which is then set aside for it, and one larger than the whole pool is a 400 at the
conductor. Its first step takes a contiguous region of exactly that size (first fit), given
back when the request ends, so requests of different sizes share the pool instead of each
holding the largest. When no free gap holds a new request but the free tokens do, the
resident regions slide left to join the gaps, a copy of what they hold, between steps; one
that slides across a short gap is copied through a staging block of 512 tokens a layer,
reserved with the pool (90 MiB at LingBot-World's geometry), rather than in many small
copies. How many requests run at once is the deployment's max_concurrent_requests, not
the store's: the pool counts tokens, so it admits many small requests, and any state a
model keeps per request outside the store is bounded only by that limit (LingBot-World
requires it to be 1 for this reason). A step that declares this store runs eager (no
capture yet), and it does not shard across ranks yet.
A LoraSpec serves LoRA adapters per request over frozen base weights: each target
linear adds scaling * (x @ A^T) @ B^T for the adapter of the token's request, so rows of
one batch can run different adapters, or none. The engine owns the adapter weights. A
model declares which linears adapters may touch; it never indexes the pool itself.
A model declares LoRA from its own modules: the shared linears add the delta
themselves once bound, and the bridge builds the spec by walking each node's weights
root. LoRA stands outside the state contract, as weights do: no request writes an
adapter, a request picks one by its kwargs, and each step's rows come from its segments,
so nothing per request is left for declare_state to hold. A model declares a
LoraRoot for each node adapters reach in lora_roots(), and declared_lora_spec
builds the one spec from them. The spec is opt-in (NodeResourceSpec.opt_in), built
only when the serving YAML has a resources: lora: block. Orpheus is the reference
(OrpheusModel.lora_roots):
from mstar.model.components.lora.modules import PER_SEGMENT, LoraRoot
def lora_roots(self) -> tuple[LoraRoot, ...]:
return (LoraRoot(
node="LLM",
# builds the module the node's weight loader fills, at one rank, on the
# meta device: only once the spec is read, which a deployment without
# a `lora:` block never does
build=lambda group: OrpheusForCausalLM(self.config, comm_group=group),
adapter_roots=[("", "model.")], # adapters saved on the bare LlamaModel
rows={"lm_head": PER_SEGMENT}, # rows that are not the step's tokens
embeddings=True, # every embedding call is in the forward
),)root_path is the root's attribute path in the node submodule ("model" by
default), and adapters names the adapters a checkpoint ships for the root
(Phi-4-multimodal's vision and speech). No model builds a LoraSpec itself.
The state lint (test/modular/test_state_lint.py) refuses get_node_resources in a
ported model, and any import in a ported model from mstar.model.components.lora or
below but LoraRoot, AdapterRoot and PER_SEGMENT. It also refuses, in any model
file outside the bridge (mstar/model/base.py and
mstar/model/components/lora/modules.py, with tests folders skipped), a call to
declared_lora_spec, module_lora_spec or join_lora_specs and a built
LoraSpec.
The weights root states how the checkpoint lands on it once
(checkpoint_rules() -> CheckpointRules: a prefix, a remap, the stacked-shard
rules); its weight loader and module_lora_spec both use them, so adapters are
placed exactly as weights are.
- What takes a delta.
ColumnParallelLinear(andMergedColumnParallelLinear,QKVParallelLinear,KVColumnParallelLinear),RowParallelLinear,FusedColumnLinearandVocabParallelEmbeddingdescribe their own target from their own shard layout (slices, heads replicated past their count, packs, per-rank padding:LoraCapable.lora_target). Binding swaps a module to its family's twin class, which adds the delta after the base GEMM and before the module's own gather or all-reduce (after it, for the embedding) throughapply_lora(delta, x, y) -> y, the one call site. A plainnn.Linear, aConv1d/2d/3dwhose kernel is its stride, and subclasses of either that keep their base's forward are swapped alike, and are whole on every rank. An unbound module keeps its class, so a deployment without alora:block runs, traces and guards exactly as before (tests hold the aten ops, the dynamo graph and its guards). An MoE layer'sFusedExpertsis a target per parameter (LoraCapable.lora_targets):gate_up_projanddown_proj, each with an A and B per expert, namedexperts_gate_up_proj/experts_down_projand placed per expert by the loader's"gate:N"/"up:N"/"down:N"shard ids; its twin adds the deltas inside the dispatch (FusedExperts._dispatch), after each grouped GEMM (before the activation, and before the top-k sum and the all-reduce), so a subclass with a_dispatchof its own is refused as one with its own forward is; so is an adapter of PEFT's fused 3-D form (one A and B on the experts container), read per expert only. Anything else holding weight matrices (a router, a depthwise conv,nn.Embedding, a forward overridden on an instance or hooked) takes none: the walk records why, and an adapter touching it is refused with that reason. So does a child its parent runs through the weight itself rather than its forward (folded at load, anF.linearover it), which the parent names inlora_unservable(a path below it -> reason; a path that is not one of its submodules is aValueError): MLA's absorbed projections, SigLIP's patch embedding, Command-A+'s fused decoder-layer projections, the Qwen3 talkers' code-predictor heads. Nothing else can tell such a child from one whose forward runs: an undeclared one is bound, and its delta silently never added, so code that reads a linear's weight directly declares it there. - Targets. One per role in a layer stack (
<stack>.N.<role>, the firstModuleListorSequentialin the path, whatever it is named;layersset where a role is on some layers only), named by the role's last component (namesoverrides), plus one per other module: one outside the stacks, or at a role whose entries differ in shape (UNet levels, vision stages), which is named by its path but keeps its role forrows. - What the model states. Only what modules cannot know.
rows: a target that sees one row per segment (PER_SEGMENT: a head after the last-token select) or one perkfolded tokens.exclude(with a reason): modules called outside the planned step, a text projection or a conditioning encoder run inprepare_inputs.embeddings=Trueonly when every embedding call is in the step.adapter_roots/aliases/adapter_stacked_params: the prefixes, names and split projections adapters write, rewritten to the checkpoint's before the root's rules place them.ignore: modules an adapter may carry that never run. - Several nodes. A model whose adapters reach several nodes (an LLM and its vision
encoder, which a TP config may put on different ranks) declares a root on each, and
the bridge joins their specs into one (
join_lora_specs): its targets name their nodes, a checkpoint module belongs to the node that places it (the one whoseadapter_rootsclaim it most specifically, when several do), so one adapter file serves both, and each node gets a pool of its own (LoraSpec.per_node, keyedlora@<node>). The onelora:block sizes them all. - Binding and rows.
NodeSubmodule.bind_node_resourcesbinds every module placed on its node underroot_path, after checking it is the declared target and that the pool's shard is its rank-local shape (a module built whole on a TP node fails here, not in a forward). The pool's step is the node's own segments (Resource.steps_from_segments): the engine (engine_step) adds aResourceStepof them under the pool's key, the rows its linears see, in order (a request may have several: Chatterbox's cond and uncond copies; Qwen3.5's vision encoder: one segment per frame, the merger's rows folding them). A step that lays its rows out otherwise declares its own step under the pool's key, which the engine keeps. A step that isNoneor declares no segments gets none, and an adapted module it runs refuses, so a node serving adapters returns a step with its segments wherever one runs. - Only inside the step. The pool holds a step's addressing from its plan to its
commit (or rollback), for the thread that planned it, so a delta called outside a step
(
prepare_inputs, which the async worker runs on the main thread while the GPU thread's step is planned; an output cached from an earlier step) refuses, eager or compiled, instead of adding another step's adapters to these rows. The engine's own warm-ups outside any step (compile_regions,warmup_compiles) run as a step whose rows name no adapter: a compile there traces no delta, and the first step with an adapter row compiles its own. A piecewise region's step takes its rows as the node's does, from its own segments (anattn_keyregion's: one per row), planned per replay and handing the enclosing step's plan back at its commit. A region replayed as one row per step (one_row_per_step) addresses every row to the first request's adapter, so the engine captures it with the pool lean (BucketKey.leannames the pool) and leases it only for a step none of whose rows runs an adapter; a step with one runs the region eagerly under the node's step.
declared_lora_spec produces an ordinary LoraSpec, whose parts follow.
- Targets.
LoraTargetgives the unsharded linear's shape and how the base layer is split under tensor parallelism. For a column-parallel linear, A is replicated and B is split by output rows, per slice (a slice with fewer heads than ranks is replicated, asQKVParallelLineardoes with K/V). For a row-parallel linear, A is split along the input and B is replicated; the delta is a partial sum, added to the partial base output before the all-reduce. LoRA adds no collective to the forward.layersrestricts a target to some layers of a hybrid stack. A checkpoint tensor that packs several of the target's slices, each split across ranks on its own, is named inslice_packs(Qwen3.5's delta-netin_proj_qkv: one A for the three, B cut by their widths), and a base linear that pads a slice per rank for alignment says so inslice_align.parallel="none"is a target whole on every rank, its delta added to an output that is already whole.experts=Eis one stacked parameter of an MoE layer's routed experts: each (token, top-k) row runs its token's slot's A and B for the expert the router picked, addressed by one combined id(slot - 1) * E + expert(the zero adapter's are not stored), on the fused-MoE grouped GEMM (LoraBackend.apply_experts_; one align per layer over the combined ids, then a shrink and an accumulating expand per slice). Experts targets are opt-in, since every slot holds a copy of each expert's factors:module_lora_specdeclares them inLoraSpec.dropped_targets, and the YAML'stargets:serves them by naming them. The align op groups fewer than 1024 ids once padded to its warp of 32, so a spec serves at most992 // Eslots (7 at 128 experts) and refuses more where the YAML sizes it. Experts split by expert parallelism are refused at deployment validation and at bind (expert_split); serve them under TP. - Target kinds and rows.
kind="embedding"is a token embedding:in_featuresis the vocabulary, the layer passes token ids asx, and the shrink is a gather of A's column per token (mstar::lora_gather_shrink), PEFT'slora_embedding_A/_B; an id outside the vocabulary (an image position's-1) adds nothing.per_segmentis one row per segment andtoken_fold=kone row perkconsecutive tokens of a segment. The pool plans an addressing per grouping its targets use; the call checks the rows it gets. - Checkpoint layouts. Adapters are validated from the same read that pins their
digest: modules map to targets, shapes fit them (an adapter for another model size
is a 400, not an admit failure), and a conv adapter whose kernel is its stride
flattens to the linear over the patch. DoRA (PEFT's
lora_magnitude_vector, Kohya'sdora_scale),modules_to_save, trainable tokens and bias LoRA are refused by name. A YAMLtargets: [...]keeps a subset of the model's targets (dropping vocabulary-sized ones saves their slots); an adapter touching a dropped target is refused, and its modules stay unbound. Without the list, a deployment withruntime_updatesoff keeps the targets its configured adapters touch (a node left with none gets no pool): the API server reads them once at startup (resolve_lora_adapters) and the conductor, handed that config, spawns every worker with the list, so every process builds the same pools; an adapter it cannot read fails the deployment. - Requests. Nothing per model.
LoraSpec.request_configopens a request that names an adapter in thelora_adapterkwarg, which the OpenAImodelfield maps onto (or which the model'sprocess_promptsets; see Cross-request prefix reuse), with aLoraReqConfig, which each node's pool takes; a base-model request carries none. The API server refuses, with a 400, an adapter it does not serve (ServedModels), and the conductor, with a 409, one a runtime load or unload is in flight for; an adapter is checked when it is registered (rank withinmax_rank, only declared targets, shapes that fit), not per request. A request on an adapter is prefix-cached in its adapter's own namespace (see Cross-request prefix reuse). - Capture and compile. Nothing per model: a node serving adapters captures the same
buckets and compiles the same as one that does not. A capture plans every row on the
zero adapter; a replay's plan writes each row's slot into the buffers that capture
recorded, so a captured graph serves any mix of adapters without recapture. The
kernels are the custom ops
mstar::lora_shrinkandmstar::lora_expand_(the latter declares theyit writes), and an experts target's grouped GEMMs are one more,mstar::lora_experts_apply_, sotorch.compiletakes them as single nodes without a graph break. Under Triton the launch grid depends on the token count and the slot count alone. The pool is a lean resource (Resource.lean_capture; see Lean captures under the resources table), so the engine captures every bucket whose step drives it a second time with no LoRA kernel in it: a step none of whose rows names an adapter replays that twin (or runs its deltas asyuntouched, eagerly), and the LoRA capture when the twin was not captured. The choice reads only which adapter each request named, the same on every rank, and the pool checks it again at plan;cg_key_infoanddeclare_stepstill state only the node's own key. - dtype. The slot weights take the nodes' compute dtype (the serving config's
autocast_dtype, e.g.float16). A node that runs outside autocast setsLoraSpec.dtypeto its weights' dtype.
A request takes its adapter's slot on its first admit and holds it until it is removed.
Slot 0 is a zero adapter (rank 0) that base-model rows and capture padding rows address,
and the kernels skip it. A miss loads the adapter from the host tier into a free slot, or
into the unheld slot released longest ago. A request whose adapter cannot get a slot (every
slot held by running requests, or reserved for requests admitted ahead of it) waits at the
runner's admission (LoraAdapterPool.ask / reserve), in arrival order with every
other resource that asks, so it holds no KV pages meanwhile and the rows already running
keep stepping; the admit's AllocationFailed stays only as a backstop. Only the first
waiting request is admitted, so a cold adapter is not starved by steady traffic on resident
ones: requests behind it wait too, and the held slots drain. On one GPU, where KV admits
every request as well, that holds back base-model requests too: a request waiting for a
slot, or for its adapter's host copy, stops every newer admission on its node until a slot
drains or the read lands. Under tensor parallelism KV does not ask, so base requests go on
past it. The slot wait does not arise while the adapters in rotation fit in max_slots,
and the copy wait is at most one read, which starts at the request's ingest. A slot is
reserved for a request only once its adapter's host copy is in (even for an adapter
resident in a slot, which a miss admitted ahead may evict), so its admit does not wait on a
read there; a request whose adapter failed to read, and is in no slot, is let through at
once, and its admit fails it before handing out any slot, so it takes none that a counted
request needs. A reservation counts slots and pins none: which slot an admit takes follows
from the slot table alone. An
adapter with nothing for a pool's targets (a text-only adapter on a vision tower's pool)
takes no slot there and runs no delta (its steps can replay the lean capture; see Lean captures): the
pool knows it from the file's header at startup, or from a runtime load's read, and its
rows address the zero adapter. Only the
rank that chooses batches admits: on a lockstep instance the leader, whose choice the
followers run as sent, so its reservations need no agreement across ranks.
The plan writes each step's addressing (for each row grouping its targets read, per token,
per segment or per fold, the rows grouped by slot: the only thing the kernels read) into
one fixed buffer per capture slot, sized to its bucket, so a captured replay
reads stable addresses; a replay's padding rows address the zero adapter and its tail
tokens no slot. The pool admits and plans on the GPU thread only (no pre-plan), in step
order, and every write it makes to the device (an adapter load, the addressing) goes on
the stream the forwards run on, so a load into a slot lands after every step queued
before it. backend picks the kernels a pool registers behind the custom ops, which
every device runs: "triton" (slot-grouped shrink and expand, the default on CUDA) or
"torch", which registers none and leaves the reference (no data-dependent shapes, the
default elsewhere). Registration is process-wide: once anything in the process has
registered the Triton kernels, every pool on CUDA runs them.
Under tensor parallelism each rank of the node's lockstep instance keeps its own pool and
makes its own slot decisions, with nothing journalled from rank 0, and they come out the
same on every rank: admits run in the order rank 0 broadcasts, and the worker lands each
teardown between the same two admits on every rank (the after_tp_seq stamp on a
forwarded remove, honoured on rank 0 by waiting for the admit of the step it is stamped
behind). That is why the LRU clock counts admits rather than commits, which may land on
either side of a teardown. Which adapters are resident, which slot a miss takes and
whether an admit refuses then follow from the same state on every rank. Reading an
adapter from disk is the one step that can fail on one rank only, so the ranks agree where
each one's copy stands (in, still being read, or failed) before any waits for one, and
every rank installs it or refuses the request: one gloo all-reduce per admit with an
adapter to install (all of its misses at once), a second when a rank was still reading,
none in steady state. Files are read on the pool's
own reader thread: every rank starts reading an adapter into the host tier when it ingests
a request that names it, the leader reserves a slot for the request only once its copy is
in, and an admit never reads a file itself. A follower whose read is still in flight
waits for it, unless another rank's read has failed. That wait is short when the
follower's read started at the same ingest as the leader's, the usual case, but it can be
a whole read: a tier keeps what the ingests and removes on its rank leave it, a follower
takes removes from rank 0 and ingests from the conductor, and a remove that races an
ingest can leave a follower without a copy rank 0 kept. Neither the wait nor the
agreement holds the pool's lock, so the main thread's ingests and removes go on
meanwhile.
A node that splits its steps over its SP ranks (a SeqShardSpec, SP > 1) or runs CFG
parallelism (cfg_size > 1) cannot serve LoRA yet: the pool lays rows out per request over the
whole batch, while each SP rank holds a shard of the tokens and each CFG rank one guidance
branch's rows, so the deployment is refused before spawn (SP) or when the pool is built (CFG).
That includes a node whose KV pools place positions over its SP ranks and whose sequence shard
splits its steps (exchange all_gather or ring). One whose steps run copied (positions
placed, exchange auto or none) runs every row on every SP rank, so it serves LoRA.
Adapters loaded and unloaded at runtime (/v1/load_lora_adapter) reach each worker as
a control message from the conductor, which each rank applies whenever it arrives rather
than at a step the instance agrees on. LoraAdapterPool.load_adapter_async only reads the
adapter into the host tier, on the reader thread (the worker acknowledges the load from its
main loop once every pool has it), and unload_adapter only stops new requests from naming
it; once no live request names it (immediately, or at the last one's teardown) the pool
forgets its host copy and its name's slot entry, and leaves the slot's weights and LRU
standing as they were. Neither touches what an admit decides from, and the next request
that names the adapter is ingested after a reload, which reaches every rank after the
unload did. Nothing per model is needed for either.
A resource whose capacity can run out before a request starts, and that the request then holds
until it is removed (KV pages, recurrent-state slots, sampler rows), answers for that capacity
through the admission hooks on Resource. The runner's Admission decides who gets it and
when: one arrival order across resources, and a request takes everything it asks for or nothing.
Capacity a single step takes and gives back goes through admit and rollback_admit
instead.
ask(rid, node_name, graph_walk)returnsASKSwhile the request needs this resource's capacity to start on the node, None when it needs none, or a failing outcome when it can never fit.reserve(rid, node_name, graph_walk, published)takes what the request asked for and returns None, or returns a mark of what it didn't fit against.unreserve(rid)gives back whatreservetook, when another resource the request asks didn't fit.grew_since(mark)says whether a request that didn't fit againstmarkmay fit now. The runner tries a blocked queue again only once it is True, so a pass that changed nothing costs one call.
A request is admitted once it is first in every queue it waits in. A resource may also let a later
request pass a first request that can't start, by overriding
reserve_past(rid, node_name, graph_walk, published). It takes the passer's capacity while
leaving the first request what it needs, so passing can't starve that request, or returns a mark,
and the runner tries a passing request again only once grew_since that mark. The runner calls
it only while the first request can't start: a resource that lets requests pass refused it and
hasn't grown since, or the oldest request holds back that resource's queue while it waits to return,
and the passing request passes, or takes nothing, at that resource too. A resource that doesn't
override it keeps strict order, and a request waiting in its queue can't pass anywhere. A resource
that overrides it asks nothing more of a waiting request whose capacity it set aside on its own, as
the KV cache does when it moves other requests for it, and the request may then start wherever it
stands in that resource's queue.
get_graph_walk_graphs returns one graph per walk. The primitives are defined in
mstar/graph/base.py:
GraphNode(name, input_names, outputs)is one unit of compute.nameis the node name thatget_submodulereceives, and the name that resource specs put in theirnodesset.Model.nodesreturns all such names across all walks.input_nameslists the tensor names that must be present before the node can run.outputsis a list ofGraphEdge.GraphEdge(next_node, name, ...)routes an output tensor namednametonext_node. Two flags are important.persist=Truekeeps the tensor available for later steps and walks, which is how a generated token is carried fromprefillinto thedecodeloop.output_modality, combined withnext_node=EMIT_TO_CLIENT, streams the tensor to the client. Its value is one of"text","image","audio","video"or"action". The special destinationsEMIT_TO_CLIENTandEMPTY_DESTINATIONare defined inmstar/graph/special_destinations.py. Note that no edge flag stops adecodeloop. A loop stops when a submodule'scheck_stopregisters a stop signal against thatLoop, for example on EOS. See Step 4.Sequential([...])andParallel([...])compose subgraphs in order or concurrently.Loop(name, section, max_iters, outputs)is a subgraph that iterates. Its body feeds its own outputs back as the inputs of the next iteration. It runs at mostmax_iterstimes, and can stop earlier. A decode loop'smax_itersis the model's context length, and each request's own length is its cap fromget_loop_caps. Give the loop anameso that a submodule'scheck_stopcan register a stop signal against it. This is the usualdecodeloop.
A minimal text generator has two walks: a prefill node that runs once, and a
decode Loop whose body feeds its own output back as the next input.
def get_graph_walk_graphs(self) -> dict[str, GraphSection]:
prefill = GraphNode(
name="LLM",
input_names=["text_inputs"],
# the generated token persists so the decode loop can pick it up
outputs=[GraphEdge(next_node=EMPTY_DESTINATION, name="new_token",
persist=True)],
)
decode = Loop(
name="decode_loop",
section=GraphNode(
name="LLM",
input_names=["text_inputs"],
outputs=[GraphEdge(next_node="LLM", name="text_inputs")], # loop-back
),
max_iters=self.config.max_position_embeddings,
outputs=[],
)
return dict(prefill=prefill, decode=decode)Each node name maps to a :class:`mstar.model.submodule_base.NodeSubmodule`, which is a
torch.nn.Module. Autoregressive nodes use the ARNodeSubmodule subclass. The
methods are:
prepare_inputs(self, graph_walk, fwd_info, inputs, **kwargs) -> NodeInputsConvert the routed
NameToTensorListinto a typedNodeInputs, or into anARNodeInputswithinput_idsorinput_embeds. This method runs once per request. Do only cheap host-side work here: shape and length bookkeeping, building position metadata, slicing token id lists. Do not launch GPU compute here. GPU compute belongs inforward. The engine may callprepare_inputson a different thread from the GPU thread, and much earlier than execution.Always set
input_seq_len. This field is on the baseNodeInputsclass and has two important uses. The engine sums it across the batch to select a CUDA-graph capture bucket and to compute padding sizes, anddeclare_stepnormally computes its spans from it. Leave it at 0 only when the submodule's inputs are not sequence-shaped. Ifdeclare_stepneeds a value thatforwarddoes not use, put that value inresource_step_info.get_input_sequence_len(self, graph_walk, fwd_info, inputs, **kwargs) -> InputSeqLenInfo | None(optional)- Return the length of the row's whole input, before any cached prefix is cut --
the engine subtracts what the prefix cache already holds. Return
Nonefor a walk that declares no length, or when the inputs it would measure are absent. Read off input shapes alone: this runs beforeprepare_inputs. speculative_input_seq_len(self, graph_walk, fwd_info, prev_seq_len) -> int | None(optional)- Estimate the length of a row continuing from the step in flight, before that step's
outputs reach it;
prev_seq_lenis its length in that step, or -1 if it ran on another node. The default repeatsprev_seq_len. Only an estimate: it sizes the token budget left for the rows a speculation merges in beside it. A walk with a fixed length (one token a decode step) should return it. declare_step(self, graph_walk, request_ids, inputs, slot_lease=None, piecewise_leases=None, per_request_info=None, **kwargs) -> SubmoduleStep | None- Declare what this batch's step does to the node's resources. See Step 4a — Declare the step below.
preprocess(self, graph_walk, engine_inputs, inputs) -> dict- Collate a list of
NodeInputsinto the keyword arguments thatforwardexpects. The default implementation handles batch size 1. Override it to support batching.ARNodeSubmoduledeclares this method abstract, because autoregressive submodules normally support continuous batching. You can still disable batching for one node, or for specific graph walks. See :ref:`Step 5 <step-5>`. forward(self, graph_walk, engine_inputs, **kwargs) -> NameToTensorListThe tensor-to-tensor computation. The keys of the returned dict are the edge names that the graph routes downstream. Reach declared state through handles. See Reaching your state.
The engine applies
torch.compileto bothforwardandforward_batched, for every submodule. It also captures CUDA graphs for them when you declare capture configs (see :ref:`Step 5 <step-5>`). To disable compilation for a submodule, set the class attributedisable_torch_compile = True. An encoder whose output feeds a prefix cache layout walk compiles with dynamic shapes instead. One that marks every dimension that varies withtorch._dynamo.mark_dynamicinprepare_inputssetsmarks_dynamic_dims = Trueand compiles as any other submodule. The engine raises the recompile limit around a node's compiled calls, pastMSTAR_RECOMPILE_LIMIT, to hold an entry per bucket its compiled captures take (each batch size and token count); a submodule whose frames keep several entries per bucket (GLM-5.2's decoder layers are traced once per layer kind) declares how many ascompiled_entries_per_shape. Keep the compiled paths compile-friendly. If a helper must not be traced, because it uses data-dependent Python control flow or forces a host synchronization, exclude it explicitly with@torch.compiler.disable. A submodule that disables the whole-forward compile but wants parts of it compiled returns them fromcompile_regions(): attribute paths (a module, or a method) mapped to example arguments to warm each with at start-up. The engine compiles them, with dynamic shapes and before capture, on an accelerator only; prefer it to callingtorch.compilein model code. Its options are fixed (dynamic shapes, torch.compile's default mode), so a part that needs static shapes, a mode orfullgraphstill compiles in model code. Qwen3.5's vision encoder and Omnilingual ASR are examples.Some submodules must run in their own parameter dtype. One example is an fp32 vocoder that is numerically sensitive. Such a submodule sets
disable_autocast = True. The engine then does not cast its parameters to the autocast dtype, does not wrap its forward in autocast, and explicitly disables any enclosing autocast. These two class attributes replace the removedget_stateless_flavormethod.The engine runs float32 matmuls as TF32 in every process (
torch.set_float32_matmul_precision("high")). A node that needs full fp32, for example a flow solve that amplifies TF32 rounding, setsfloat32_matmul_precision = "highest", andcudnn_allow_tf32 = Falsefor cuDNN convolutions and RNNs. The engine applies these around that node's forward, its compile and its CUDA-graph capture, and restores the default afterwards, so other nodes keep TF32.None(the default) keeps the process setting. Don't fliptorch.backendsflags in model code: they are process-wide, and a CUDA graph keeps the precision it was captured with. The setting coversforwardandforward_batchedonly, notpreprocessorpostprocess. It can also be set per instance in__init__, for example only when the model loads in fp32.A node whose captured convolutions are faster with cuDNN's timed algorithm choice sets
capture_cudnn_benchmark = True: the engine turnstorch.backends.cudnn.benchmarkon around that node's capture sizing only, so each captured shape is searched once, and eager steps keep the heuristic pick (Kokoro is an example). The search tries workspaces as large as the GPU has free, so on a deployment that setsgpu_memory_fractionthe engine holds the allocator to that share while it sizes the node's captures, and the boot fails if the node's largest eager step does not fit in it beside what the worker already holds.postprocess(...)(optional)- Metadata-only fixups that run on the GPU thread. This method must not read tensor
values. Do not call
.item(),.cpu()or.tolist()here. This is a performance requirement, not a correctness requirement. Reading a value forces a host synchronization, which stalls the GPU thread and loses the worker's asynchronous scheduling overlap. Use this method only to rename outputs for routing. Put decisions that depend on tensor values incheck_stop. check_stop(...) -> set[str] | StopCheck(optional)- Runs off the GPU thread, and may read tensor values. Return the names of the
Loopobjects to stop, for example after seeing the EOS token. This is how a decode loop terminates. An AR submodule whose loop ends on a stop token setsself.token_stop = TokenStop(...)in__init__instead of writing one; that also gives itcheck_stop_batched, which checks a whole step off one host copy of its tokens. Qwen3.5'sLLMSubmoduleis an example. The token budget is the loop's cap fromget_loop_caps, not a stop check. A decode walk that feeds each row its last sampled token can keep that token on the device instead of routing it, with aDeviceLoopback(see Device loop-back). A prefill samples the first token before its decode loop exists, so a stop there is returned asEND_PARTITION(frommstar.model.submodule_base) instead: the worker reports the partition done with that pass, and the conductor ends it rather than start decode. Return it only from a row outside any loop; from inside one the request fails.TokenStopdoes this for the walks in itsprefill_walks, whose token counts toward the budget, soget_loop_capsgives the loopmax(1, max_output_tokens - 1). Itslast_prefill_flagnames thestep_metadataflag of the pass whose token is the reply's first, so a stop id sampled on an earlier pass ends nothing.SampledTokenDecodernamesFINAL_PREFILL, which the model's walk plan sets on that step. It suits a partition with nothing left to run once generation stops:END_PARTITIONends the partition whatever its plan has left, so a model that runs another walk after its loop (analignwalk, or image generation after thinking) or publishes final-only state at the loop's stop must not return it there. Keepcheck_stop_batched, if you write one, in agreement. To also keep some of this step's outputs from being routed, returnStopCheck(stops, withheld=(names...)), for example a frame streamed alongside the EOS token: its consumer then never receives it, instead of finding and dropping it with a device read.check_stop_batchedreturns aBatchedStopCheckfor the same. The Qwen3-Omni Talker and Fish S2 Pro's AR keep their EOS frame out of the codec stream this way; aTokenStopnames such outputs inwithhold, left unrouted on a row whose token is a stop id (the Qwen3-TTS Talker). Only an output that leaves by a streaming edge, as a tensor or by value, can be withheld (any other consumer would wait for an item that never comes), and not one that a node this worker speculates into reads, nor the loop-back signal. Startup refuses aTokenStop.withholdthat breaks this; aStopCheckfromcheck_stopis not checked. on_request_added(self, request_id, request_info)(optional)- Called on the worker's main thread when the request reaches the worker, before any of
its steps, with the request's first
CurrentForwardPassInfo. For host work worth starting early on a thread of the submodule's own; it must not touch the device. The DiT scaffold draws a request's seeded noise here (PREFETCH_SEEDS). get_stop_policies(self, graph_walk) -> dict[str, StopPolicy] | None(optional)- Declare what
check_stopamounts to on a walk, per loop it can stop, when it is only the common checks; the graph runtime then runs them over the step's token column without callingcheck_stop. That takes the batched output carrying the policies'token_signalas a stop buffer (BatchedModelOutput(check_stop_buffers={...}), as Qwen3.5 and Orpheus return); a step without one, or one that mixes walks, still callscheck_stop.StopPolicy(StopKind.AR, token_signal=..., eos_tokens=..., sampler=...)stops its loop on an EOS token (unless thesamplerresource's request config setsignore_eos) or when the loop's iteration count plus one reachesmax_iters(info.max_tokensunless given).StopKind.ITERchecks only the count, andStopKind.EOSonly the token, for a loop whose cap is its budget.ignores_eos=replacessamplerfor a node with no such resource: a callable on the forward-pass info, which it reads for every partition of the request on the worker. The limits are computed from the request's forward-pass info whenever the worker receives it, not per step. Return{}for a walk on whichcheck_stopnever stops a loop, andNone(the default) to keep callingcheck_stop. The policies must agree withcheck_stopfor every row;test/modular/test_stop_policies.pychecks the models that declare them. ATokenStopwithruntime_check=Truedeclares them for you: its loop's stop ids, asStopKind.EOS, onwalk; itsprefill_walkskeepcheck_stop. cleanup_request(self, request_id)(optional)- Free per-request state held inside the submodule when a request finishes: buffers,
per-request caches, counters. See Qwen3-Omni's
Code2WavSubmodulefor an example. filter_batched_output(...)andunpack_packed_outputs(...)(optional)- Output fixups that run after the forward. A captured forward always emits the same set
of keys, because the graph shape is fixed. Use
filter_batched_outputto drop keys that a particular request must not receive. Useunpack_packed_outputsto slice a packed(total_tokens, ...)tensor into per-request entries, when the slice boundaries depend on the real sequence lengths and therefore cannot be computed inside the captured region. Both methods are defined onNodeSubmodule, so any node can use them. spanned_outputs(graph_walk) -> frozenset[str](optional)- Outputs whose valid length per row is known only on the device, such as audio with
silence to trim.
forward_batchedreturns aBatchedModelOutputwhoseoutput_spansmaps the output's name to a[bs, 2]int64 tensor on the device: rowiholds the first and one-past-last index of rowi's valid samples along the output's last dimension. A step that returns spans returns one for every output declared here, since per-request forwards are merged by name. The forward keeps a fixed width and does no host read. The worker cuts each row to its span after the step's completion event, before the stop check, transport or a streaming edge reads it. The cut is a view, not a copy. Engine-sidepostprocessstill sees the uncut output. A row whose span is empty is left out of the store after the stop check, as a withheld output is (StopCheck): a streaming edge gets no item for it. A spanned output must not feed a node that the same worker speculates into.
Two more methods control batching and CUDA graphs: can_batch with
forward_batched, and get_cuda_graph_configs. They are described in Step 5.
Chunked prefill (optional). A walk with long prompts can run each across several steps, at most a token budget per step, so the decode rows batched beside it keep running. Each chunk is one iteration of a Loop around the node, which counts the tokens its chunks ran and ends at the prompt's length. To opt a walk in:
Model.get_chunking_policies()returns{node: (ChunkingPolicy, ...)}, each policy naming thewalksit covers. It is declared on the model, not the submodule, because the graph build reads it:Modelwraps each chunkable node in its Loop whereverget_graph_walk_graphsis read, so the model returns its walks as usual. The fields:chunkable=Truelets the walk's rows be cut.max_batch_tokens: the most tokens a step runs. A combined walk that carries chunked rows sets one too, since its steps run under it. Its step may pass it beside a speculation chain's decode rows, by at most those rows' tokens, and the engine trims a serving config'sprefill_token_bucketsassuming each such row runs one token. A walk whose decode rows run more (an MTP verify row's k+1) passes that bound, and its steps past a trimmed ladder's top run eagerly.accumulated_outputs: output edges a consumer needs every chunk of. They arrive as one item per chunk, all when the prompt finishes.per_chunk_outputs: streaming edges sent on every chunk as it runs, for a consumer that can start on a prefix. Any edge in neither set carries the final chunk's output alone.reuses_cached_prefix: a callable saying whether a row may start past its cached prefix;None(the default) is always. Set it where a row may have to write its prefix again.reserve_whole_prefill(default on): a row's first chunk reserves the state its whole prefill needs, so a part-done row cannot wedge on a full pool. Turn it off for a walk whose state grows by something other than the tokens a step runs.
A resource that keeps state only on a grid counted from a chunk's start declares
Resource.chunk_alignment, and a cut chunk of the node's rows is then a multiple of every alignment its resources declare (Engine.chunk_alignment). The chunk floor rounds up to it, and a budget under it fails loading.A streaming consumer of an accumulated edge whose
ChunkPolicyexpects one item (FixedChunkPolicy(1), say) setsConnection.join_dim, so itsStreamBufferjoins the pieces into that item. A consumer that can take the pieces as they come says so withChunkPolicy.allow_partial_input(FixedChunkPolicy(allow_partial=True)), and the producer then sends the edge per chunk, as Qwen3-Omni's Thinker does into its Talker. A chunked stream its consumer would misread, each piece taken for the whole input, is refused at startup.get_input_sequence_len(graph_walk, fwd_info, inputs)gives a row's length from its input shapes, beforeprepare_inputsruns. The scheduler cuts the row by it, and a row whose prepared input turns out another length fails.split_inputs(graph_walk, fwd_info, inputs, start, end)cuts the whole prepared input to tokens[start, end).ARNodeSubmodule's default cuts the token ids or embeddings and any custom position ids, and refuses inputs it cannot see into (tensor_inputs,kwargs,resource_step_info).
Each chunk is a full forward, so it samples a token, but only the final chunk's token is
real (inputs.is_final_chunk). The model need not handle that: the sampler commits no
RNG offset or seen token for a row mid-prefill (it reads StepContext.chunk_tokens_after),
the Loop carries only the final chunk's outputs, and the worker takes no stop or prefix
key from a non-final chunk. test/modular/test_chunked_prefill.py and
test/modular/test_bagel_chunked_prefill.py show how to test a chunking model. A
deployment can change the budgets (see Chunked prefill budgets in :doc:`serving`).
declare_step tells the engine what one batched step does to the node's resources:
which cache streams it touches, how much each stream grows, what the attention plan is
based on, which streams fork, and what is committed after the forward completes.
The runner calls the declaration and the resource lifecycle around your forward:
declare_step() → admit → plan → preprocess → forward → commit
The runner owns this lifecycle. A submodule that declares a step must therefore contain no plan calls and no advance calls of its own.
Declaring the work also lets the worker handle a cache that is full. If a step cannot be
admitted, the engine returns an admit failure. The scheduler can then apply backpressure
and evict other requests. Without a declaration, the same situation raises a
RuntimeError inside the forward.
Returning None means that the submodule manages its own resources. This is the legacy
path, kept for submodules that have not been migrated.
A walk named in resident_walks() runs its captured steps off per-request tables on the
device instead of a plan rebuilt per step (mstar/engine/resources/resident.py). The
walk must declare, for each resource, one stream that takes one token per row, or one
it only reads at span 0 (a cross-attention context), with no forks, retention or sink,
and its preprocess must read no planned position or KV location, because the
captured graph fills those in first. Each KV pool of the node keeps its own block table
and lengths; positions read the pool their PositionConfig.kv_cache names. Capture raises on a declaration that breaks this. The engine
turns it on only if every resource of the node can (Resource.resident_refusal) and
the deployment left knobs: resident_planning at auto; the worker log says why
otherwise. A resource the node shares with other nodes is fine while they plan on the host (a
sharer's host step marks the rows it moved, and an offload it causes frees the slot on
every planner bound to the resource); it is refused if another sharer names resident
walks too, or runs in lockstep. A GDN layer on a node whose steps can split refuses it (resident
decode beside split prefills is untested); a sequence shard itself doesn't. Steps of other walks,
and eager steps, keep the host plan and leave the row
stale, so the first resident step after them writes it. The sampler takes part on its
own: its per-request scalars (temperature, top-k, top-p, seed, penalties) become cells
of the row, so a resident step stages none of them, while the RNG offset, the last token
and the repetition-penalty mask stay by the sampler's own row of each request.
A SubmoduleStep contains a list of Segment objects and, under state, what the
step does to each array and sampler it touches. The engine expands state into one
ResourceStep per resource, so nothing downstream changes:
Segment(request_id, label, span)is one request's contribution to one label in this step.spanis how far the row advances on that label: an append-only array adds that many entries, and a Value folds that many positions into its one entry. Aspanof 0 reads the array without extending it. Undermax_len=0each segment is one block.SubmoduleStep(combined_labels={("main", "cfg_img"): "cfg_batched"})packs several labels into a single plan. Batched classifier-free guidance uses this to run two branches through one attention call. EveryResourceStepgets the mapping, so the KV plan and a sequence shard read the same grouping; positions take their packing from the KV plan, andPositionStepkeys itspos_idsby the combined label.ArrayStep(segments=, keep=, forks=, placeholders=, eager_add=, fixed_layout=, advance=, position_ids=). A step lists every array it touches, and a call on one it doesn't list raises.segmentsgives one array its own spans,Nonebeing the step's.keep=Falsemakes every add this step scratch.advancemoves each row's position counter by something other than its span, for a prompt whose grid covers fewer positions than tokens, andposition_idspasses explicit ids.SampleStep(apply_penalty=, apply_filters=, prompt_tokens=, kept_rows=).prompt_tokensinitializes the repetition-penalty mask with the prompt tokens. Only the prefill step passes them, because the sampler tracks every token it samples afterwards.kept_rowsnames the rows whose draw counts,Nonebeing every row. A dropped row still draws, but its RNG offset and seen-token mask don't advance.apply_filters=Falsesamples a step with top-k/top-p off (e.g. a TTS's first code); declare it the same way on every step of a walk.- Reads and updates need nothing in the step, because the declaration says which read each layer uses in each walk.
Command A+ is the simplest form on the contract. It uses one label and computes spans directly from the prepared inputs:
def declare_step(self, graph_walk, request_ids, inputs, **kwargs):
return SubmoduleStep(
segments=[
Segment(request_id=rid, label="main", span=inp.input_seq_len)
for rid, inp in zip(request_ids, inputs, strict=True)
],
state={
KV_CACHE: ArrayStep(),
SAMPLER: SampleStep(prompt_tokens={
rid: inp.input_ids for rid, inp in zip(request_ids, inputs, strict=True)
} if graph_walk in PREFILL_WALKS else {}),
},
)A model that hasn't moved declares today's ResourceStep objects under steps, keyed by
resource: KVStep(commit=, eager_write=, pre_forks=, post_forks=),
AttentionStep(causal=, context_only=) (context_only=True has the queries attend to the
stored context alone; only the MLA backend reads it), PositionStep(pos_ids=, advance=) and
SamplerStep(apply_penalty=, prefill_tracked_tokens=).
Padding rows. Under a captured graph, the batch is padded to the shape of the capture
bucket, and request_ids also contains the ids of the padding rows. Declare segments
for those rows in the same way as for real rows. The zip(..., strict=True) in the
example above already does this.
The two leases. slot_lease is the CUDA-graph slot on which this step will replay.
It is None for an eager step. If a submodule's declaration differs between the
captured case and the eager case, it must check the lease, not its own capture key. The
capture key only says that the batch could be captured. The lease says that the batch was
captured. For example, Cosmos3 packs both guidance branches into a single plan for the
captured shape, and uses the dense backend otherwise.
piecewise_leases names the inner regions of this node that hold their own slot for
this step. Such a region declares, plans and commits its own work. Any resource that the
region owns must therefore be excluded from the outer declaration. See
Piecewise CUDA graphs (capturing an inner loop).
Advanced ArrayStep fields. These fields cover cases that the engine previously
handled as special cases. BAGEL and Whisper use them, and a model that hasn't moved writes the
form in parentheses:
keep=False(KVStep(commit=False)): the step reads and plans, but its writes do not become resident yet.keep=Falseon a Value with updates is refused for now, because its kernels write in place.forks(pre_forksandpost_forks):Fork(src="main", dst="cfg_text", at=...)forks one label from another.at="start"forks before any planning or writing, for a branch that must keep the context from before this step.at="end"forks at commit time, for a branch that must include this step's writes.rowslimits the fork to some rows, and today a step's forks must all name the same rows. A fork shares the prefix of an append-only array and copies a Value's entry.placeholders(sink):(row, label)pairs that hold a place but read and keep nothing.eager_add=True(eager_write): the step's adds run outside any captured graph, for example inpreprocessbefore the replay. Their addressing stays an ordinary plan rather than the slot's static buffers, which are sized for the bucket's in-graph tokens. Whisper's prefill uses it to write its 1500-token encoder context while replaying a graph sized by its few prompt tokens. The reads of such a step attend the pages alone, so one that needs the step's own k and v (an attention through an exchange, a context-parallel pool on a multi-token step, the dense backend) is refused at the first add.fixed_layout=True(fixed_layout): every replay of the step's capture bucket lays its rows out as the capture did, so a ragged self-attention may capture a kernel that bakes the layout in (cuDNN). A resource's step takes it only if every read on the resource declares it.
combined_labels is declared once, on the SubmoduleStep, and only a model that hasn't
moved writes it. On the contract, a step's segments may name several of an array's labels,
label-major: every row's first label, then every row's second. The engine plans them as one
plan, named by joining the labels with +, which a declared label can't hold. Every call
covers every segment in segment order: at_last_token(h, x) gives each (row, label)'s last
token in that order, and at_last_token(h, x, label=L) label L's rows in row order. On
an array that keeps entries a row has one segment per label a step. A Value takes one label a
step, and a cross read's queries come from one label. The one plan is for every array on KV
pages, so a step that packs labels on one paged array lists the node's other paged arrays on
the same labels, and the engine refuses a step that lists one on other labels. Chatterbox's
guided T3 steps and BAGEL's guided text steps and denoise are the examples: Chatterbox
combines its guidance from every (row, label)'s last token, and BAGEL samples from
label="main".
Writing K/V yourself. A layer that hasn't moved and writes its K/V inside a fused kernel
of its own, rather than through write_kv, asks the KV resource where the step's tokens go:
kv.write_target(layer_idx, label) returns (layer cache, page of each token, slot in
page of each token, number of tokens to write) for the current plan (NHD layout only).
Tokens past that number are the plan's padding and must not be written. A layer on the
contract gets the same through a @state_op's target, as qk_norm_rope_attend in
mstar/model/components/qwen3_lm.py does.
Testing. declare_step performs only host-side bookkeeping, so you can unit-test it
on CPU without model weights. test/modular/test_state_ports.py compares both ports' steps
against hand-written ones, and test/modular/test_resource_runner.py tests the runner's
lifecycle directly.
The engine binds one handle per declared array, sampler and sequence shard.
bind_state(self, handles) receives them by declared name, on the submodule and on every layer
that defines it:
class LLMSubmodule(ARNodeSubmodule):
def bind_state(self, handles):
self.kv = handles[KV_CACHE]
self.sampler = handles[SAMPLER]An array's handle has three calls. layer is always the model's layer id as declared,
and each row's label comes from the step's segments, so there are no cursors:
st.kv.add(layer, k=k, v=v) # kept
st.kv.add(layer, k=k, v=v, keep=False) # scratch
out = st.kv.read(layer, q)
st.mem.write(layer, state=x) # rewritable arrays onlyaddappends each row's span at its next log positions. A kept add happens at most once per layer per step, and it is the layer's last add. Scratch adds can repeat, and last until the next add or the end of the step.keep=Nonetakes the step'skeep. The resources keep or drop a step's adds as a whole, so a step that keeps needs each layer to end on a kept add, unless every row it lists has span 0, is a placeholder or got its adds from a piecewise region whose own step lists it, and a step withkeep=Falserefuseskeep=True.readsees kept entries from earlier steps, plus this layer's latest add, at the positions its mask reaches. It runs the op the declaration names for the layer.writesets the step's rows' entries in place where the forward runs it, not at the step's commit.
These raise, naming the array and the cause: a call on an array the step doesn't list or on
a layer it doesn't declare, an add after a kept add, a kept add after a read in the same
layer, and write on an append-only array. The checks run in eager steps and at capture, so
graph replays skip them. On an array with max_len=0 a compiled block skips them too, since a
check there would guard on the layer.
The rest are free functions in mstar.engine.state:
out = add_read(self.kv, layer, q, k=k, v=v) # add, then read, behind one break
qkv = causal_conv(self.gdn, layer, qkv, weight=w, bias=b) # reads and writes `conv`
core = gated_delta_net(self.gdn, layer, q, k, v, a, b, A_log, dt_bias) # reads and writes `state`
y = mamba2(self.ssm, layer, x, dt, b, c, A_log, D, dt_bias) # reads and writes `ssm`
p = position(self.kv, fwd_info.rid_handle, label) # a request's next position, on the host
ids = positions(self.kv) # this step's ids, on the device
q, k = rope(self.kv, q, k) # rotate at this step's ids with the declared Rope
tokens = sample(self.sampler, logits) # one token per row, each a fresh draw
last = at_last_token(self.kv, hidden) # each row's last token in this step
ids = last_sampled(self.sampler) # each row's last sampled token, for a DeviceLoopback
x, cos, sin = scatter(self.shard, x, cos, sin) # this rank's share where the step splits, else as they are
hidden = gather(self.shard, hidden) # every rank's share, back whole
mine = split_range(self.shard) # this rank's range (start, count, counts), None where copied
ranks = self.shard.ranks # the SP ranks a layout must tile overThe shard's handle is opened on every step of its node and takes the label it brackets from the step's segments and the walk from the step, so the calls name neither. Both are the identity on a copied step, at one SP rank and outside a step.
A layer with nothing traced between its add and its read calls add_read, since two breaks
compile a frame for the gap on every shape. It runs both calls' checks in their order, and refuses an
array with max_len=0, whose calls a compiled block traces. Call it from the frame the layer
compiles: a break inside a method that frame calls compiles that method as a frame of its own.
On an array that keeps entries, every call that touches state breaks the compiled graph and runs
eagerly, so model code adds no torch.compiler.disable of its own. A layer stack calls
run_layers(..., no_cursor), which runs its loop eagerly and compiles each layer's frame once,
and the forward around it takes @eager_frame when it does no tensor work of its own. So that one
frame serves every layer, a compiled layer passes its layer id as a 0-d int64 CPU tensor, held as a
plain attribute, not a buffer. An eager caller can pass the int.
kernel_setting(h, read, field) reads a kernel_settings field in effect, YAML
included. It is temporary, with that hook.
A kernel that writes state itself, in one launch with the math before it, is a state op. It names what it touches, the formats its kernel can address, and a reference over the contract's calls, which any other format runs:
@state_op(adds=("kv",), reads=("kv",), mutates=("q",), formats=(PagedNHD(devices={"cuda"}),),
reference=_reference)
def attend(kv, layer, q, k, v, *, target: PagedNHDTarget):
... # write k and v at target, and change q in place
return kv.read(layer, q)
out = attend(self.kv, self.layer, q, k, v) # the layer's add, then its read, behind one breakreads may name only the array in adds, and the call refuses an array with max_len=0,
as add_read does. The engine picks the kernel or the reference at bind, and a step whose reads
attend its own k and v (a context-parallel pool's multi-token step) runs the reference.
target says where this step's entries go on the layer. A conformance test holds each kernel to
its reference on each listed format, what it returns included, and MSTAR_KV_DEBUG_ASSERTS=1
checks that an eager call writes nothing outside the step's slots.
Between steps, a model may keep a memo of a pure function of request inputs, whose loss costs time and changes no output, and scratch buffers that it makes at load and that no step reads before writing. Anything else it keeps between steps is declared state or an edge. This rests on the engine running a node's forwards one at a time, in stream order. The diffusion loop's schedule and prefetched seeds are memos, its latents and solver state are loop-back edges, and a code predictor's frame-local KV is scratch.
A model on the contract reads no node_resources, engine_inputs.resources,
request_state or per_request_states, imports the engine only through
mstar.engine.state and defines no bind_resources; test/modular/test_state_lint.py
holds each ported package to that.
Qwen3-TTS's batch_postprocess keeps the opt-in watermark's per-stream state in
request_state under one named lint exception, until it moves to a watermark node's plain
Value (issue #1447).
ModelInputsFromEngine carries six other useful fields:
stepis this step'sSubmoduleStep. A forward that must match its own declaration reads it here instead of computing the same information again.capturedis true when this forward runs under a CUDA-graph capture or replay. This field replaces the oldcache_manager.is_captured. It is useful whenpreprocesspacks its inputs differently for the fixed capture shape.syntheticis true when every row is a dummy request, during CUDA-graph capture or its warmup. Use it to keep statistics such as routing counts free of dummy traffic.real_seq_lenis the token count of the real requests. Their rows come first; any rows after them are padding that a replay adds to reach the bucket shape. It isNonewhen nothing was padded, so every row is real. Read it instead of assuming anything about the padding rows' own inputs.per_request_statesis aMappingthat is resolved on first read.piecewise_runnersholds the piecewise CUDA-graph runners, keyed by region name.
The shared layers take their layer id in the constructor. ParallelGatedDeltaNet takes a
handle, and Attention, ParallelAttention and VisionAttention pick their path at
bind from what they receive: a handle under their key, or the resources.
A model that hasn't moved reads engine_inputs.resources in the forward, keyed by
resource_key. In a layer, resources are resolved once at load time. The engine calls
submodule.bind_node_resources(resources). That method stores the resources, then
iterates over self.modules() and calls bind_resources(resources) on every module
that defines it. A layer names the keys it needs in its constructor and resolves them in
bind_resources. If a layer names a key that its node does not have, the error occurs
at bind time rather than during a forward pass:
ParallelAttention(..., attn_key=ATTN, kv_key=KV_CACHE, pos_key=ROPE)Inside the layer, AttentionCallable (mstar/engine/resources/convenience.py) wraps
the KV write and the attention call together. The label and the layer index are cursors
stored on the resources. They are not passed as arguments on each call. The code that
drives the layer stack binds the label once, then advances the index for each layer:
def forward(self, query_sequence: torch.Tensor, *, label: str) -> torch.Tensor:
self.layers[0].self_attn.attend.bind_step(label)
for layer_idx, decoder_layer in enumerate(self.layers):
decoder_layer.self_attn.attend.set_layer_idx(layer_idx)
query_sequence = decoder_layer(hidden_states=query_sequence)
return self.norm(query_sequence) # the runner advances the cache, from the stepCursors are used instead of arguments because passing the layer index as an argument makes inductor specialize on that integer. The frame is then retraced once per layer.
Device loop-back. A decode loop normally routes its sampled token back to itself as a
tensor per request. An AR node that sets self.device_loopback = DeviceLoopback(walk,
signal, sampler, buffer) instead reads each row's last sampled token off the sampler on
the device (MSTAR_DEVICE_LOOPBACK): the base class's device_loopback_signals names
the signal, so the worker routes it without a tensor and sends the client's copy inline,
prepare_inputs returns self.uniform_row_inputs(...) for such a row, and
preprocess reads the tokens with self.loopback_inputs(engine_inputs, inputs). A
row whose last step did not keep its value on the device (its first decode step, after
the prefill) still brings it routed, and that read puts it over the slot's.
SampledTokenDecoder declares it for Qwen3.5, Muse Glimmer, Higgs Audio, Qwen3-ASR and the Cosmos3 reasoner,
and a node on it that reads the routed token sets device_loopback = None; Whisper and
Qwen3-Omni's Thinker declare their own. A node on DeclaredTokenDecoder and Whisper's decoder
read the tokens with last_sampled, and put a row's routed token over its slot's with
splice_loopback_ids.
Its walk may be a frozenset of walks when several decode walks read the token back; a
combined step's rows each follow their own walk, and naming a combined walk is refused at
startup.
A token that also streams to another node can go down the stream by value, when the
stream's Connection sets by_value=True: its consumer then takes a window of the
stream as one CPU int64 tensor of its items' values. A step whose node names the signal in
inline_signals(walk) sends each row's value in the step's EDGE_ITEMS frame instead
of a tensor, with nothing to stage, read or acknowledge. A step carries one inline column:
the first of those signals it outputs, and every other it outputs that reads the same stop
buffer, so a client token and a stream of the same token both ride inline; a signal read
off another buffer keeps its tensors. A row whose stop check withheld one of them
(StopCheck.withheld) sends nothing for it, and its stream counts no item. The step
keeps its rows on the device when every output is one of device_loopback_signals or
rides the column on every edge (a client emit or a by_value stream, neither persisted,
no non-streamed edge counted toward the conductor's output tokens, and not collected by
an enclosing loop):
MiniCPM-o's TTS decode loops its code back on the device and streams the same code to the
vocoder by value. A rows-only step builds no per-request outputs and skips
postprocess, so whatever a submodule's postprocess does must be about the outputs
a stored step routes. A lockstep node's streams keep their tensors, and a by_value
connection cannot join its items, drive its consumer's walk or come from the client.
On such a node a captured step writes each row's token and RNG offset back to the request's
row inside the graph (knobs: sampler_ingraph_scatter, on by default). Those rows are
allocated once, knobs: sampler_slots of them, so a request takes its row when the runner
admits it and waits while none is free, as it does for a recurrent state slot. A row the
step drops (SampleStep(kept_rows=...)) writes the trash row, so it keeps its offset.
State that a decode loop carries from step to step beside the token is a Value with tensor
fields, which the forward reads and writes each step, captured or not. Whisper's
decoder_state holds its timestamp rule row and its transcript.
A kernel in the forward can draw its rows' tokens itself, as Qwen3-TTS's depth-loop kernel
does for fifteen draws a step. fused_draw_args(s) returns the step's per-row temperature,
top-k, top-p, seed and offset, in the step's row order, padding rows included. The kernel draws
row r's i-th token with draw_row(..., seed[r], offset[r] + i, ..., LAY=draw_row_layout()),
imported from mstar.engine.state, which picks the token sample would and consumes one draw
from the row's random stream, as one sample call does. After the launch, drew(s, n) counts
n draws toward the sampler's draws and moves the offsets. fused_draw_args returns None
when sample would do more than that draw (an eager step, the penalty, min-p, logprobs, a kept
last token, a TP, SP or CFG node) or the declared vocabulary is above 2048, where fused_sample
leaves the layout draw_row assumes; the model then calls sample. These calls and
FusedDrawArgs, the tuple fused_draw_args returns, stay out of mstar.engine.state.__all__,
so they can change until a second model draws in place.
get_submodule loads a node's parameters. Weight loading is standardized through
mstar/model/loader/. Use it instead of a custom load_state_dict call. Only then
can the same code load both a single-GPU checkpoint and a tensor-parallel shard. See
:ref:`Step 6 <tensor-parallelism>`. There are three layers:
- In
get_submodule, build thenn.Moduleon themetadevice, materialize it withto_empty(device=...), then callload_weights(module, source, device=...)frommstar.model.loader. This function selects the correct safetensors iterator, for a single file or for a sharded Hugging Face directory, and then callsmodule.load_weights(weights). - Your module implements
load_weights(self, weights)and delegates toload_hf_weights(self, weights, stacked_params=..., name_remapper=...). That function streams the(name, tensor)pairs and dispatches each pair to theweight_loaderof the matching parameter. stacked_paramsis a list ofStackedParamRule. Each rule routes several checkpoint keys into one fused parameter, for example the Hugging Faceq_proj,k_projandv_projkeys into a singleqkv_projparameter.LLAMA_STACKED_PARAMSis a predefined rule set for Llama models.name_remapperrewrites or drops checkpoint keys that do not match your parameter paths.
# in the Model: build on meta, materialize, hand off to the driver
def _create_llm_submodule(self, device, tp_group=None):
from mstar.model.loader import load_weights
with torch.device("meta"):
language_model = OrpheusForCausalLM(self.config, comm_group=tp_group)
language_model.to_empty(device=device)
load_weights(language_model, local_dir, device=device) # → module.load_weights(...)
...
# in the nn.Module: declare the fused-shard routing and delegate
def load_weights(self, weights):
from mstar.model.loader import LLAMA_STACKED_PARAMS, load_hf_weights
return load_hf_weights(self, weights, stacked_params=LLAMA_STACKED_PARAMS)Each parameter's weight_loader also performs tensor-parallel sharding. When the module
is built with a comm_group and tp_world_size > 1, the loader slices the incoming
tensor along that parameter's shard dimension before copying it. For this reason, one
load_weights path serves both single-GPU runs and tensor-parallel runs without change.
A quantized checkpoint is refused at load unless the model reads its format itself. Without this, a model that knows only unquantized weights would load fp8 values unscaled and drop the scales and packed weights, and then serve garbage. There are two checks:
- The safetensors iterators refuse a checkpoint that declares a quantization:
quantization_configinconfig.json(HF fp8, AWQ, GPTQ, bitsandbytes, mxfp4, compressed-tensors, ModelOpt), ModelOpt'shf_quant_config.json, or AutoGPTQ's or AutoAWQ's config file. load_weights_intorefuses a quantized weight's part, such asweight_scale_inv,weight_scale,weight_packedorqweight, when no parameter takes it.
A model that loads a format itself names its quant_method. GLM-5.2 and GLM-5.3-Flash, for
example, pass quant_methods=("fp8",) to load_weights and to the iterators.
Continuous batching and CUDA graphs are the two main throughput optimizations. Both are optional. For an autoregressive node, you normally want both. These two mechanisms have the most detailed rules in the submodule interface, so they are described here separately.
Continuous batching. The worker's micro-scheduler groups compatible in-flight requests into one GPU call. A submodule controls this behavior with these methods:
can_batch(self, batch: ExecutingBatch, model_inputs) -> boolreturns whether these requests can share one forward pass. The default isFalse, which disables batching. Override it to accept batches, for example when the requests use the same graph walk and have compatible shapes.forward_batched(self, graph_walk, engine_inputs, **kwargs) -> dict[str, NameToTensorList]performs the batched compute. It returns per-request outputs, keyed byrequest_id. When a batch runs, the engine calls this method instead of the single-requestforward.max_batch_size(self, graph_walk)sets an optional upper limit.step_order(self, graph_walk) -> StepOrdersays who runs first when more rows are ready thanmax_batch_sizeallows. The default,StepOrder.ARRIVAL, is first come, first served to completion: the requests holding the step keep it, then the request that arrived first.StepOrder.TURNSis longest-waiting first: a request that just ran gives way to one that waited, so a node capped at one row advances its requests a step each in rotation. Declare it on a walk whose sessions have to keep moving while others run, as LingBot's rollout and Cosmos3's chained windows do. A combined walk takes turns if any of its walks does. The order applies to the rows a speculation chain continues, the backlog and the queued rows alike; when every ready row fits, it doesn't matter. An admit that runs out of KV pages offloads the row last in that same line.
Walk priorities. Which (node, walk) a worker's next step goes to is a
SchedulingPolicy (mstar/worker/scheduling_policy.py). By default it is round robin:
the least recently scheduled key first, and a speculation chain keeps its first step before
it gives way to other ready work. A model that returns WalkPriority entries from
Model.get_walk_priorities gets Priority instead. An entry names a node (all its
walks), a walk (all its nodes) or one (node, walk), and where entries overlap the most
specific decides: a pair over a walk over a node. Keys with ready rows go in priority order,
round robin among equals, and a chain gives way at once when a key above its own has a row
the engine would take. That trades the other keys' steps for the listed one's latency, so it
suits a walk whose steps are short, such as a TTS model's prefill, whose first audio waits on
it while decode streams have time to spare: list the walk, and every node on it is covered.
A key that stays ready holds lower ones back for as long as it does, unless they have a
max_wait: a ready key that has gone that many of the worker's chain steps (every step on
its main chain, side-lane steps not counted) since it last ran goes
first, longest waiting first, and a chain gives it its next step. A priority on one walk of a
combined walk lifts the whole combined key in that order, its other walks' rows included, and
a backlogged key, split over several steps, still finishes first whatever its priority. A
deployment replaces the entries, sets a default max_wait or turns priorities off with the
serving config's scheduling (see :doc:`serving`). An entry naming nothing the model runs,
a negative priority, or two entries for the same target refuse the worker's start.
Combined walks. By default a step holds one walk. A model lets several walks of a node
share a step by returning {node: {combined_walk: {walks}}} from
get_combined_graph_walks, for example decode rows beside a prompt's prefill. The
combined walk is a name of its own, which max_batch_size and the CUDA-graph configs
can key on. A step of one walk keeps that walk's cap. A step that mixes walks takes the
combined walk's max_batch_size when it has one, else the smallest cap of the walks it
holds, so a small prefill cap never limits a decode-only step.
max_rows_in_combined(self, graph_walk) also bounds how many rows of one walk a mixed
step may hold, for kernels sized to a few prompt rows (see CombinedSplit). To run a
combined walk on such split kernels, pass a CombinedSplit in the splits of each
resource config that supports one (AttentionConfig on the FLASHINFER backend, for
captured steps only, and LinearAttnConfig with the GDN variant; other backends ignore
splits and run the walk unsplit); their specs return it from combined_splits(),
which is what the engine reads, and resources of one node must agree on a walk's
prefill_walks. The engine then orders that walk's steps with the prefill_walks
rows last, so the kernels read the decode rows as the step's first tokens, in place. Every
other walk of the combined walk must be one token per row.
Steps beside the chain. A worker's main loop runs one chain of steps, and any step on it
costs a pass of the loop and holds the next one behind its kernels. A node that shares a
worker with a heavier one can set side_stream_walks on its submodule to the walks that
may run beside the chain instead, and a deployment turns that on with side_stream: true
on the node's group (see :doc:`serving`). Such a step neither takes a turn on the chain nor
makes it yield, runs on a second stream, and the windows that come ready while it runs
batch into the next one (see StepPipeline._launch_side in mstar/worker/step_pipeline.py). Its graphs capture on a stream
of their own, since graphs captured on one stream must not overlap. It suits a stream-fed
walk that nothing on the chain reads, such as a codec. Where the config asks,
EngineManager.build refuses what the lane can't serve, since a side step is neither
speculated nor planned ahead and lands between two chain steps: a TP, SP or CFG node, a node
alone on its worker, a device without streams, a node with chain walks or resources, and a
side walk that is not the node's alone, sits in a combined walk, or has output spans or a
stop check. A config without the field keeps the node on the chain, whatever it declares.
ARNodeSubmodule declares preprocess abstract because autoregressive nodes are
expected to collate a batch. You can still disable batching for one node, or for specific
walks, by returning False from can_batch in those cases.
CUDA graphs. A submodule declares the shapes it can capture in
get_cuda_graph_configs(self, device, tp_world_size=1) -> list[CudaGraphConfig]. The
default is an empty list, which means eager execution. For each config, the engine first
runs torch.compile, controlled by the config's compile flag, which defaults to
True. It then records a CUDA graph and replays it. Two config types are defined in
mstar/engine/cuda_graph_config.py. They differ in which stage of the submodule
pipeline they freeze:
| Config type | Use and captured stage |
|---|---|
BatchedCudaGraphConfig |
Decode-style forward passes, in which every request in the batch has the same
length. That length is usually one token. Pass single_request_inputs, which is
a NodeInputs for one request. The runner clones it to build each captured
batch. This config fixes the output of prepare_inputs, and preprocess runs
on both capture and replay. |
PackedCudaGraphConfig |
Prefill-style forward passes that operate on packed, variable-length sequences.
Pass capture_token_lengths, which lists the token-count buckets to record, and
make_node_input(n), a factory that builds a NodeInputs for one request of
length n. The runner partitions each bucket across the batch. Attention is
planned at capture time from the step declaration, and planned again into the
captured buffers on each replay. |
Both types share the base CudaGraphConfig fields:
capture_graph_walkis the walk to capture.replay_graph_walkslists the walks that may replay this capture. One capture can therefore serve several walks, for exampleprefill_audioreusingprefill_text.capture_batch_sizeslists the batch sizes to record.Noneuses the runner default;[]records none.additional_key_infois an extra hashable component of the capture bucket key. Use it when one walk must be captured more than once because the batch shape alone does not identify the capture. Its value must equal both the value returned by the submodule'scg_key_info(graph_walk, per_request_info)for a batch and thecg_key_infovalue that the batch'sdeclare_stepsets on its step. A mismatch raises no error. The batch simply does not match any capture and runs eagerly. Compute all three values from one place. This field replaces the removedlabelsandrequires_cfgfields.capture_forward_methodnames the method to capture. The default is"forward_batched".caps_eager_batch_sizecontrols whether this config's captured sizes also limit the engine's eager batch size for the walk. The default isTrue, so the engine never batches beyond a captured size.compilerunstorch.compilebefore capture. The default isTrue.compile_modeis thattorch.compile's mode. The default,None, takes the deployment's compile level (mstar serve -O,MSTAR_COMPILE_LEVEL); naming a mode takes the config out of that level.defaultkeeps Inductor's fusion but takes cuBLAS for every GEMM and skips autotuning.deterministicis the engine's own: Inductor's defaults in its deterministic mode, which picks kernel configs by rule rather than timing them on the device, so every process compiles the same kernels (a torch without the mode logs a warning and compiles with the defaults). A compile outside the runner takes the same options fromaccelerator_graph_runner.compile_kwargs("deterministic"), ormstar.platforms.compile.deterministic_inductor_options()where it builds its own; the Wan VAE decode, the DiT scaffold'scompile_blocksand Cosmos3's denoise step do. Building the config refuses an unknown mode, and one that turns on Inductor's own CUDA graphs (reduce-overhead,max-autotune), which fail inside the runner's capture.inductor_optionsare Inductor config entries the compile runs under, over the mode's or the compile level's own, e.g.{"max_autotune_gemm_backends": "ATEN"}to keep max-autotune's fusion choices but cuBLAS for the GEMMs (autotune's timings of decode-width GEMMs can vary between processes). Building the config refuses an entry Inductor doesn't have, andtriton.cudagraphs.requiredmakes every bucket in the config mandatory. If capture fails locally or on another participating rank, warmup raises after rank-wide agreement instead of dropping the bucket and falling back to eager execution. The default isFalse.
BatchedCudaGraphConfig also accepts total_tokens_multiplier. Use it when one
request's step commits KV across several labels that are combined into a single plan, as
in batched guidance that packs the conditional and unconditional sequences together. The
static buffer must hold all of them, and this field scales the buffer independently of the
per-label span.
For example, a token decoder that captures a batched decode graph and a packed
prefill graph declares the following (SampledTokenDecoder builds these, and a
mixed one, for its subclasses):
PREFILL_TOKEN_BUCKETS = [32, 64, 128, 256, 512, 1024]
PREFILL_CAPTURE_BATCH_SIZES = [1, 2, 4, 8, 16]
def get_cuda_graph_configs(self, device, tp_world_size=1):
return [
BatchedCudaGraphConfig(
capture_graph_walk="decode",
single_request_inputs=ARNodeInputs(
input_ids=torch.zeros(1, dtype=torch.long, device=device),
input_seq_len=1,
),
),
PackedCudaGraphConfig(
capture_graph_walk="prefill",
capture_token_lengths=self.PREFILL_TOKEN_BUCKETS,
make_node_input=lambda n: ARNodeInputs(
input_ids=torch.zeros((n,), dtype=torch.long, device=device),
input_seq_len=n,
),
capture_batch_sizes=self.PREFILL_CAPTURE_BATCH_SIZES,
),
]A deployment replaces a packed capture's capture_token_lengths with the serving
config's prefill_token_buckets (node -> capture walk -> ladder, see :doc:`serving`).
The engine applies it to the configs get_accelerator_graph_configs returns, so a
model plumbs nothing. A PiecewisePackedConfig region's total_tokens takes the
ladder its label names the same way.
BAGEL shows the case of two captures for one walk. It captures decode twice, once with
additional_key_info=False and once with additional_key_info=True. A decode step
with guidance enabled lists each row on main and cfg_img, so its capture sets
total_tokens_multiplier=2, and a decode step with guidance disabled lists it on main
alone. BAGEL's cg_key_info reports which of the two a batch is.
The configs above capture the whole forward_batched method of a submodule. The engine
then drives the replay, including sampling and output remapping.
Sometimes you want to capture only one inner region of a forward, such as a transformer block loop, and keep the surrounding code in eager Python. The code before the region computes embeddings and assembles the sequence. The code after it applies the final norm and projection. A piecewise CUDA graph supports this.
A submodule enables piecewise capture by returning one or more configs from:
get_piecewise_cuda_graph_configs(self, device, autocast_dtype, tp_world_size=1)
-> dict[str, PiecewiseCudaGraphConfig]
The dict key is a region name, which is any string that identifies the captured region.
Several keys capture several independent graphs. At warmup, the engine builds one
PiecewiseCudaGraphRunner per key and puts them in
engine_inputs.piecewise_runners. Your forward looks up the runner by key and calls it.
Nothing is stored on the submodule.
Two config types are defined in mstar/engine/cuda_graph_config.py. They correspond to
the two whole-forward types:
| Config type | Use and captured shape |
|---|---|
PiecewiseBatchedConfig |
Batched inputs of equal length. The captured region sees [bs, seq_len, D], and
every request has the same seq_len. Pass seq_len, the number of tokens per
request. One graph is captured per batch size. |
PiecewisePackedConfig |
Packed, variable-length inputs. The captured region sees [total_tokens, D],
with several sequences of different lengths packed together. Pass total_tokens,
a list of token-count buckets. One graph is captured per pair of batch size and
bucket. |
Both types share the base PiecewiseCudaGraphConfig fields:
capture_fnis the callable to capture. Its interface is described below.make_static_inputsis a factory with signature(shape: PiecewiseCaptureShape) -> dict[str, Tensor]. It returns the persistent buffers that the captured region reads. The runner owns these buffers. Before each replay, it copies your real tensors into them by name.shapeprovidesbs,seq_lensandtotal_tokensfor the bucket being built. Allocate each buffer in the dtype of the tensor you pass for it, which for a hidden state is usuallyautocast_dtype. The copy before replay casts a value into a wider dtype of its kind (int32 token ids into an int64 buffer, a bf16 table into an fp32 one). A value the buffer may round (a narrower dtype, such as an fp32 hidden state into a bf16 buffer, or an integer into a floating-point or complex buffer) is refused with aValueErrornaming the buffer.forward_kwargsholds static keyword arguments that are passed tocapture_fnon every call, for example a layer count or anis_causalflag.requiredrefuses to start the worker when any of the region's shapes is left out of capture, for memory or by a failure, rather than serving those shapes eagerly. Set it on a region with no eager path, such as one whose per-stream state only its captures read and write. The default isFalse.declare_stephas signature(request_ids, seq_lens) -> SubmoduleStep | None. It declares the region's own resource work over the padded batch. It is separate from the submodule'sdeclare_stepbecause the region has its own shape. The runner admits, plans and commits this step on each replay. This field replaces the removeduses_kv_cache,plan_fn,advance_seq_lensandcache_labelsfields. A region that reads a KV cache declares aKVStepand anAttentionStephere. A cacheless region that uses ragged attention declares only anAttentionStep, against itsRaggedAttentionSpeckey.lease_before_stepmakes the runner take this region's CUDA-graph slot before the outerdeclare_stepruns, and report the slot in that call'spiecewise_leasesargument. The region still declares, plans and commits its own work. The lease only tells the outer declaration which resources are already taken, so that it can exclude them.can_leasehas signature(inputs: list[NodeInputs]) -> bool. It applies only withlease_before_step, and lets the region refuse a batch for a limit that the capture bucket key doesn't express, such as Qwen3.5's segment count per step. The engine then gives the region no lease for that step. Your forward must replay alease_before_stepregion exactly when its label is inengine_inputs.piecewise_leases, and not wheneverrunner.can_runis true: a refused batch can still fit a bucket, and replaying it would declare the region's resources twice.one_row_per_stepmakes the runner lease, check (can_run) and replay the region as one row, whatever the step's batch size, under the step's first request. It is for a packed region that runs every request's segments as one ragged batch, such as a vision tower, and needslease_before_stepandcapture_batch_sizes=[1]. The region'sdeclare_stepthen receives[first_rid]with every request'sseq_lens, so the whole step's segments count against one replay's plan: size its attention resource withmax_segments_per_step, and pair it with acan_leasethat refuses a step with more. On a node serving LoRA adapters it is leased only for a step whose rows run none, since its replay would address every request's tokens to the first request's adapter.capture_batch_sizeslists the batch sizes to record.Noneuses the runner default;[]records none, for an eager-only region (below).eager_fallbackruns a shape with no graph eagerly throughruninstead of raising: the runner declares, plans and commits the region's step over the real rows (as a replay declares it, without padding), so the caller has one path for captured and uncaptured shapes. Off, the caller checkscan_runfirst. MiniCPM-o's vocoder uses it, with eager-only regions for the shapes it never captures.compilerunstorch.compileoncapture_fnbefore capture. The default isFalse.compile_modeis thattorch.compile's mode, as onAcceleratorGraphConfig.requiredmakes every declared shape mandatory. If any participating rank cannot capture one, warmup raises instead of leaving that shape on the eager path. The default isFalse.
Splitting the declaration. When a region leases its own slot, exactly one of the two
declarations must own each resource. The common pattern is for the outer declare_step
to return None when the region holds a lease, and to declare the resources itself
otherwise, which covers the eager fallback path:
def declare_step(self, graph_walk, request_ids, inputs,
slot_lease=None, piecewise_leases=None, **kwargs):
if (piecewise_leases or {}).get(BLOCK_LOOP_REGION):
return None # the region declares its own KV + attention
return SubmoduleStep( # eager path: nobody else reserves these pages
segments=[
Segment(request_id=rid, label="main", span=inp.input_seq_len)
for rid, inp in zip(request_ids, inputs, strict=True)
],
state={KV_CACHE: ArrayStep()},
)The capture_fn interface. The captured callable takes one PiecewiseCallInputs
argument and returns a dict:
capture_fn(inp: PiecewiseCallInputs) -> dict[str, Tensor]
PiecewiseCallInputs has four fields. static_inputs holds the buffers owned by the
runner. engine_inputs is the region's view of the batch: request ids and per-request
info padded to the capture bucket, plus the node's resources. kwargs is the config's
forward_kwargs, unchanged. resources is a shortcut for
engine_inputs.resources. Capture and replay both use this same type, so a region
cannot read a field that exists on only one of the two paths.
Warning
Read tensors from inp.static_inputs, and never assign to its entries. The runner
passes the same dict object at capture time, and updates those buffers in place before
each replay. An assignment such as static_inputs["x"] = ... replaces the buffer,
and the region then no longer uses the memory address that the graph recorded.
A capture_fn may also return a single Tensor. The runner wraps it as
{"x": ...}.
Calling the runner. Look up the runner by region name and pass your real inputs to it.
It returns a PiecewiseOutput, which behaves like a dict. Indexing and .get return
a clone that you own and can keep. .get_view returns a view without copying, which is
only valid until the next call to run. See mstar/engine/cuda_graph_runner.py. The
runner handles input padding, admission and planning of the region's declared step,
replay, commit, and output slicing.
The V-JEPA2 AC predictor is the reference implementation. See
mstar/model/vjepa2/submodules.py, region "block_loop". It has an eager section
before the region, a captured block loop that reads the KV cache over a fixed per-step
sequence, and an eager section after the region.
from mstar.engine.cuda_graph_config import (
PiecewiseBatchedConfig, PiecewiseCallInputs, PiecewiseCaptureShape,
)
# --- the captured inner region ---
def _block_loop_capture(self, inp: PiecewiseCallInputs) -> dict[str, torch.Tensor]:
# READ out of inp.static_inputs; never reassign its entries
cond_tokens = inp.kwargs.get("cond_tokens")
fn = self.predictor.make_block_loop_fn(cached=True, static_pos_bufs=inp.static_inputs, cond_tokens=cond_tokens)
return {"x": fn(inp.static_inputs["x"])}
# --- declare the region ---
def get_piecewise_cuda_graph_configs(self, device, autocast_dtype, tp_world_size=1, **kwargs):
def make_static_inputs(shape: PiecewiseCaptureShape) -> dict[str, torch.Tensor]:
# hidden state in autocast_dtype so the replay copy_ is a same-dtype memcpy;
# position buffers stay float32 (RoPE frequency precision matters more)
return {
"x": torch.zeros(shape.bs, capture_seq_len, embed_dim,
dtype=autocast_dtype, device=device),
"d_pos": torch.zeros(N * N, dtype=torch.float32, device=device),
...
}
def declare_step(request_ids: list[str], seq_lens: list[int]) -> SubmoduleStep:
return SubmoduleStep(
segments=[
Segment(request_id=rid, label="main", span=seq_len)
for rid, seq_len in zip(request_ids, seq_lens, strict=True)
],
state={KV_CACHE: ArrayStep()},
)
return {
BLOCK_LOOP_REGION: PiecewiseBatchedConfig(
capture_fn=self._block_loop_capture,
make_static_inputs=make_static_inputs,
declare_step=declare_step,
# take the slot before the outer declaration, so it can leave
# this region's KV to the runner
lease_before_step=True,
seq_len=capture_seq_len,
forward_kwargs={"cond_tokens": cond_tokens},
capture_batch_sizes=[1, 2, 4, 8],
)
}
# --- invoke it inside the forward ---
runner = engine_inputs.piecewise_runners.get(BLOCK_LOOP_REGION)
if runner is not None and runner.can_run(x.size(0)):
out = runner.run( # admits + plans + replays + commits
static_inputs={"x": x, "d_pos": d_pos, ...},
request_ids=engine_inputs.request_ids,
)
x = out["x"] # owned cloneThree points in this example are worth noting. First, positions are computed eagerly and
passed in through static_inputs, so the captured region does not compute them again.
Second, the same block loop is used on both the captured path and the eager path, so the
code exists in one place only. Third, the region's declare_step covers only the
captured path. The eager path is covered by the submodule's own declare_step, as shown
under Splitting the declaration above.
BAGEL's ViT tower is the packed, cacheless counterpart. See
mstar/model/bagel/submodules.py, region "vit_block_loop": a
PiecewisePackedConfig over VIT_TOKENS, with patch
embedding and the RoPE gathers left eager because they are data-dependent indexing.
A packed region on the state contract whose rows each attend within themselves passes
array, the declared array with max_len=0, and the region declares its step over it. One
that hasn't moved passes attn_key (the
attention resource) instead of a declare_step: the config then declares one
non-causal segment per row, or under one_row_per_step one per length of its row.
A config maps nodes to GPU ranks. The value under model: is your registry key. Each
node_groups entry assigns one or more node_names to ranks. An entry can also
name specific graph_walks, which is how prefill-decode disaggregation is expressed.
Every node of a walk must be placed by exactly one entry that applies to that walk; the
conductor refuses a config that places it twice or not at all.
model: "your_model"
max_seq_len: 2048
node_groups:
- node_names: ["LLM"]
ranks: [0]Run it with:
mstar-serve --config configs/your_model.yaml --host 0.0.0.0 --port 8000The model declares the shapes that the model requires. A deployment then tunes the values
that suit the machine it runs on. Use a resources: block for this, with one sub-block
per resource_key. A model with two caches of the same kind, such as Whisper's decoder
cache and its encoder context, can therefore tune each one separately:
model: "whisper_large"
max_seq_len: 448
resources:
kv_cache:
cpu_offload_pages: 128
cross_kv_cache:
cpu_offload_pages: 128Each spec declares which keys it accepts. An unknown key raises an error at load time, so that a misspelled setting is never silently ignored:
| Spec | Accepts |
|---|---|
KVSpec with PagedKVConfig |
max_num_pages, page_size, max_seq_len, cpu_offload_pages,
prefix_cache, prefix_cache_salt, context_parallel (see below) |
KVSpec with RingKVConfig |
num_sessions |
BoundedKVSpec |
max_slots |
RollingKVSpec |
pool_tokens |
SamplerSpec |
enable_logprobs, enable_processed_logprobs (see SamplingReqConfig),
greedy_capture (true / false; by default a sampler past the one-kernel
draw takes each row's argmax in a step whose rows all sample greedily, in a twin of
each batched capture) |
RecurrentStateSpec |
max_slots, state_dtype (on a Value, only with a GatedDeltaNet),
prefix_slots (see Cross-request prefix reuse).
max_slots counts the sink unless disable_sink_slot; a value that leaves no
slot for a request is refused at startup. |
AttentionSpec |
backend (auto / flashinfer / dense / flex / xpu_paged),
flashinfer_backend (auto / fa2 / fa3), decode_tensor_cores
(true / false), decode_fmha (true / false) |
CrossAttentionSpec |
backend (auto / flashinfer), flashinfer_backend
(auto / fa2 / fa3), decode_tensor_cores (true / false) |
RaggedAttentionSpec |
backend (auto / flashinfer / dense / cudnn), flashinfer_backend
(unset by default: a self-attention reads the ragged_attention kernel table; any name,
auto / fa2 / fa3 / cudnn / cutlass, runs that FlashInfer backend and no
table), max_segments_per_request,
max_segments_per_step, max_tokens_per_request. The ceilings size CUDA-graph
buckets, which is a memory-against-coverage trade the deployment makes, not the model.
A resource sized per step refuses max_segments_per_request, which it would ignore. |
SeqShardSpec |
min_tokens, exchange (auto / none / ulysses / all_gather /
ring; see Sequence and context parallelism below) |
LoraSpec |
max_slots (adapters resident on the device at once, beside the zero adapter),
max_rank (the largest rank served; slots are sized to it), cpu_capacity
(adapters held in pinned host memory), adapters (adapter name to safetensors
path: the names requests can ask for), targets (the subset of the model's
targets to allocate; by default, with runtime_updates off, the targets the
configured adapters touch, and every target with it on, except experts targets,
which are allocated only when the list names them; a list chooses among all the
declared targets, so a later one may name a target an earlier one left out; a node
left with none gets no pool, while a YAML list keeping none at all is an error), runtime_updates (default false: serve
/v1/load_lora_adapter and /v1/unload_lora_adapter),
runtime_update_timeout_s (default 120: how long the conductor waits for every
worker to answer one before giving up on it). The one lora: block sizes the
pool of every node the model's spec reaches |
Tune the cache shape on the KV resource, not on the attention resource that reads it. For
example, configs/qwen3tts.yaml selects FA2 under talker_attn,
configs/cosmos3_nano.yaml sets the page count under its paged KV key, and
configs/waypoint.yaml sets the resident world count under its ring KV key.
Sequence and context parallelism. A node's sp_size ranks share each of its steps, along
two axes:
- Per step, the tokens are split or copied. A
SeqShardSpecon the node decides, every step: split, each rank runs the token-local layers (norms, MLPs, projections) on its share of each listed label (contiguous, or on a ring node striped by KV position); or copied, every rank runs them all. Without one, every step is copied. - Per paged KV pool, stored K/V is split by head or by position. By head (the default) over the
node's
tp * spranks; by position withcontext_parallel: true(below), heads over TP.
Each attention resource derives what it does from the two and the shard's exchange (below);
the model writes no exchange (a ragged attention's Ulysses follows from the labels it plans):
| Tokens / storage | By head | By position (context_parallel) |
Cacheless (ragged) |
|---|---|---|---|
| Copied | head slice: this rank's heads of q, k and v, write and attend, gather the heads | each rank attends its slots, partials merged (below) | heads over TP; over a label the shard lists on a splitting node, a head slice |
| Split | Ulysses: one all-to-all from split tokens to split heads, write and attend, one back | by the shard's exchange (below): all_gather, one gather of every rank's q, k and
v, attend as a copied step, keep this rank's rows; or ring. With auto, the node's
steps run copied |
Ulysses over a label the spec's labels list; any other split label refused at its first split step (hand-written spec) |
| Cross, split queries | n/a | n/a | each rank's queries over every key |
| Cross, split keys | n/a | n/a | each rank's partial over its keys, merged by log-sum-exp (gather_merge) |
| Cross, both split | n/a | n/a | refused at load for array reads; a hand-written spec, at its first split step |
Splitting is declared in four places:
SeqShardSpec(resource_key, nodes, labels, axis, walks=None)names the labels whose tokens split (on the state contract, theSeqShardthe model declares, on its reads' axis).axis=0cuts a label's packed tokens into contiguous ranges;axis=1cuts each row of a[B, S, ...]layout alike (every row of the label one length).walkslimits splitting to those walks; the others plan copied. A step splits only when it isn't one token a row, its listed tokens reachmax(sp_size, min_tokens), and every listed label has at leastsp_sizetokens (a row, on axis 1).min_tokens(default 0) is aresources:override, a deployment's trade between the exchanges' cost and the split layers' saving, as isexchange(below), how a split step reaches the node's paged attentions. A captured step decides from its bucket's size, not its prompt's, so capture and replay split alike; where a resource's split steps run eagerly only (NodeResourceSpec.splits_eagerly), the shard sets its own floor at load, one past each walk's largest captured bucket, so a captured bucket runs copied and only a bigger, eager step splits (min_tokensonly raises it). Declare the spec over every node of the attention resources that depend on it: a resource spanning nodes the shard doesn't cover is refused, as is a KV pool whose attentions depend on two specs.exchanges=Falsedeclares that no op on the node mixes the label's units, so nothing exchanges them: the node needs no attention, and a rank may get none of them (fewer units than ranks still splits). MiniMax-H3's video decoder splits each request's clips this way, readingshard.split_range(label)for its range and gathering the decoded clips withshard.gather(mstar/model/minimax_h3/submodules.py); such a shard is refused beside an attention that takes split steps.- The model declares no step for the shard: its step is the node's segments and
combined_labels(SubmoduleStep, asResource.steps_from_segmentssays), added at any SP size (at one rank its plan is a constant), so a combined plan label gets a range too. A step that declares no segments has no shard plan, and the attention resources refuse it, so it fails on one GPU. The diffusion scaffold's step holds each label's segments once, however many arrays attend them (LTX-2.5's video tokens are the keys of two). - The forward brackets its layer stack (on the state contract, with
scatter(shard, *xs)andgather(shard, x)on the shard's handle, as every ported model does; a model off the contract calls the submodule'ssequence_shard, bound with its resources, asshard.scatter(label, *xs, dim=None)andshard.gather(label, x)):scatterbefore it (this rank's share of each tensor,dimthe spec's axis by default;Nonepasses through),gatherafter it.split_range(shard)is this rank's range of the label (start,count, every rank'scounts) where the step splits it, elseNone, for what must follow the cut (Wan2.2's timestep runs, MiniMax-H3's rope rows). Both are the identity on a copied step, at one SP rank, and outside a step (a warmup compile); at one rank they trace straight through a compiled frame, and on SP ranks each breaks the graph once. Everything between them runs on this rank's tokens. LTX-2.5 brackets the DiT in its denoise step (mstar/model/ltx2_5/submodules.py), Wan2.2 and MiniMax-H3 inside their DiT's forward; Cosmos3 brackets the generation layer stack after its whole-sequence prologue (mstar/model/cosmos3/components/transformer.py); Qwen3.5 brackets its LLM's stack, gated delta rule layers included (Qwen3_5ForCausalLM.forward). - The attention is reached through the resource: a paged attention through
AttentionCallable(attend.write(k, v)writes K/V alone; a no-op where the attention writes K/V inside its own run,requires_kv_writeFalse), a ragged one through itsrun. A layer that writes throughwrite_targetor callsattn.runitself raises on a split label, since it would treat this rank's share as the whole step, and on any step over KV stored by head, since it would pass the TP heads where the pool holds this rank's slice. A state op that fuses the K/V write into its norm/RoPE kernel (Qwen3-Omni'sqk_norm_rope_table_attend) runs its reference, throughaddand the exchange, wherever the step's read exchanges (AttentionManager.step_exchanges(label)): a label the step splits, and every step over KV stored by head, copied steps too (each rank's head slice). A cacheless self-attention over a label the shard lists runs Ulysses: an array declares nothing for it (its read's resource lists its label), and a hand-writtenRaggedAttentionSpeclists the label inlabels. Both are no-ops on a node with noSeqShardSpec, or one whose sequence shard never splits (SeqShardSpec.splits), so one declaration serves every deployment shape. A read bound to SDPA (backend: sdpa) runs outside the resources and is refused over keys the shard splits.
Recurrent and sliding-window layers. They take split steps through their own resources
too, and the model writes nothing more than the bracket. Each resource's state is stored one of
three ways over the SP ranks (NodeResourceSpec.storage): by head, by position
(context_parallel), or copied (heads over TP alone, every SP rank holding all of it). A copied
resource takes its own exchange wherever the node's shard splits, whatever the shard's
exchange:
- A recurrent pool (GDN, Mamba-2) is copied, so decode reads it with no collective, and a split
step leaves the same bytes in every replica. On a split step the conv runs on each rank's tokens
from the earlier ranks' last inputs (
conv_halo, one all-gather a layer), and every row's final window is written on every rank. A combined walk's leading decode rows run copied. The prompt rows take the exchange the linear resource picks from the model's head counts:ulysseswhere each TP rank's heads splitsp_sizeways (one all-to-all to a block of heads, the chunked kernel over that block of the state, the block's final states gathered, one all-to-all back),state_passingwhere they don't (each rank runs its pieces, hands its state on in one fixed-size gather, and runs a row continuing into it from the state it gets). Mamba-2's scan is a loop over tokens, soulyssessaves memory there, not time.state_passingand Mamba-2's split are tested on CPU only and no in-tree model adopts them yet. Both exchanges run eagerly, so a node with a recurrent layer never splits a captured step: the shard's floor sits above its largest captured bucket, so a bigger step runs eagerly and splits, and every other step keeps its graph. Prefix snapshots stay off above one rank. - A sliding-window attention over a pool stored by position keeps its window layers whole on
every SP rank (
PagedKVConfig.placed_groups) while the full layers' slots stripe: its copied steps (decode included) attend as on one rank, with no collective, and its split steps gather the step's keys (kv_gather), eagerly. Its pages hold1/sp_sizethe tokens they would unplaced, so each window page must splitsp_sizeways. Its split, copied and decode steps are tested on CPU only, over FlashInfer's reference.
Every resource of a splitting node must support split steps (Resource.supports_split_steps:
the KV pool, paged FlashInfer and dense attention, ragged self and cross attention, positions,
the sampler, the recurrent pool and its GDN and Mamba-2 resources); MLA, DSA, KDA, flex and
XPU attention don't, so a splitting node can't own them. The conductor refuses,
before a worker starts (and the engine again at load): a node on SP ranks that neither splits its steps (a
SeqShardSpec whose exchange isn't none, with a resource that takes split steps or
exchanges=False) nor places its pools by position; a shard on axis 1 over a pool stored by
position; a head-stored pool whose query and KV heads per TP rank don't divide by sp_size; a
ragged Ulysses attention whose heads don't divide by tp * sp, or beside a ring; an SDPA read over
keys the shard splits; a ragged cross-attention under a shard on axis 0 or with exchanges=False
(either can leave a rank none of a row); a linear layer (not Mamba-2) whose heads per TP rank don't
divide by sp_size (so it takes state_passing) and whose key and value head dims differ, at
sp_size 3 or more; a linear layer under a shard on axis 1 or a ring's; a kv_gather attention
under a ring's (it plans contiguous pieces); a window page that doesn't split sp_size ways; expert
parallelism. The engine alone refuses a resource without supports_split_steps at load, and, at
warmup (after the weights load and the compiles), piecewise captures, a capture replayed under listed
and unlisted walks, and a packed capture of a listed walk unless the shard lists one label on axis 0
(its padding joins that label). Each attention resource of a splitting node logs at load how it takes
split steps.
Choosing the exchange. exchange on the node's SeqShardSpec resources: block picks
how its split steps' tokens reach its paged attentions, from where their KV is stored. auto
(the default) runs Ulysses over KV stored by head, and over KV stored by position runs the node's
steps copied, so a node whose pools store KV by position splits its steps only where the
deployment opts in with all_gather or ring. A value the node's storage can't take fails at
load, at any SP size. It only picks the split exchange: copied steps keep the storage's own path.
A ragged attention runs Ulysses over a label it plans that the shard lists (above), and a copied
resource takes its own exchange (the list above), on a shard that splits. Each row assumes the node
declares a SeqShardSpec and runs on sp_size above 1:
exchange |
KV stored | Split steps | Copied steps | Pick it when |
|---|---|---|---|---|
auto / ulysses |
by head | Ulysses all-to-all | head slice | q and KV heads per TP rank divide by sp_size (Cosmos3; LTX-2.5, Wan2.2 and MiniMax-H3's
DiT, which have no paged pool, by their ragged attentions over the labels their shards list) |
auto / none |
by position (context_parallel) |
never: the node runs copied, as one without a shard | partials merged | KV capacity alone (DCP), no token split |
all_gather |
by position | gather q, k and v, attend as copied, keep own rows | partials merged | long prompts where one KV head per TP rank can't split by head (Qwen3-Omni's Thinker on TP4 x SP2) |
ring |
by position | striped pass-Q ring, eager only | partials merged | attention is the step's peak memory: past a per-model crossover (below); not beside a
recurrent layer or a kv_gather attention |
none |
by head | refused at load: the SP ranks would only share heads | n/a | never; use TP |
And for the copied resources, under any exchange that splits:
| Resource | Split steps | Copied steps | Pick it when |
|---|---|---|---|
| sliding-window attention over KV stored by position | kv_gather, eager only (CPU-tested only) |
as on one rank, no collective | always; its window pages must split sp_size ways |
recurrent layer, ulysses |
conv halo; all-to-all to heads; eager only | as on one rank | chosen where the heads (and Mamba-2's B/C groups) per TP rank divide by sp_size (Qwen3.5) |
recurrent layer, state_passing |
conv halo; one state hand-off; eager only | as on one rank | chosen where the heads per TP rank don't divide; a hand-off's bytes don't grow with the prompt (at P = 2 it
waits on the previous rank, so it isn't faster). CPU-tested only |
ulysses over KV stored by position, and all_gather or ring over KV stored by head, are
refused. A shard that never splits (none, or auto over KV stored by position) makes the
node plan every step copied, from the config alone, so it takes what a node without a shard takes
(expert parallelism, say).
Ring (exchange: ring) stripes each split step's tokens over the SP ranks by KV position: a row whose pool
already stores c tokens gives rank r its tokens at positions x with x % P == r,
exactly the ones whose K/V it stores, so it writes its own K/V and nothing moves. Each layer
then runs P rounds: rank r sends its queries to rank r + i, attends rank r - i's
over its own slots (stored context and the step's tokens together, one call, causal or not as the
step declares), sends the partial back, and merges what it gets in fp32. Its hops are
synchronous for now, and only the flashinfer backend plans it. Another is refused when the
engine builds the attention, at any SP size like the storage checks, but after the validation that
runs before a worker starts. Ring is eager only, so a ring node doesn't capture the buckets a
step could split in: those steps split eagerly, and the rest keep their graphs. A step that combines listed labels into one plan
(combined_labels, Cosmos3's batched CFG) is refused when it plans on a ring node, not at load
(a step declares its combined labels): the stripe reads each listed label's rows from the pool's
plan, which holds the combined label's instead. A step's peak memory is its resident tensors plus the
larger of its attention and its MLP (or MoE) phase. All-gather's attention phase holds the
whole step's q, k and v and P partials per query; ring's holds this rank's tokens alone, and
two fp32 copies of their result while a round merges. So ring lowers the peak only once attention
outgrows the MLP phase. Estimated from allocation sites: for the Thinker's geometry from about
P = 4 (at P = 2 the MoE phase is larger and the peaks equal), and sooner for dense
models, whose MLP phase lacks the top-k expansion (not measured). The Thinker at 128K tokens on
8 H100s, measured before this code's last merge of main (a rerun is pending), peaked at 58.8 GB per
GPU under ring and 57.9 GB under all-gather at TP4 x SP2, and at 66.8 GB and 70.5 GB at
TP2 x SP4; ring prefilled faster in both (2.52 against 2.79 s, and 2.69 against 3.26 s). Ring
merges P bf16 partials per query, and on 2 H200s at TP1 x SP2 changed 15 of 48 long greedy
summaries against the copied path, where all-gather changed none.
Chunked prefill stays off on SP ranks either way, so a long prompt prefills in one step.
Context parallelism. context_parallel: true on a paged KV pool stripes each page's slots
over the ranks that would otherwise store the same KV heads: slot s lives on partner
s % P, page ids stay the same on every rank, and each rank holds page_size / P slots a
page. Attention runs on each rank's slots and merges the partials across the partners exactly
(mstar/engine/resources/context_parallel.py). The model declares nothing.
- TP mode (no SP ranks). The partners are the TP ranks sharing a KV head,
P = tp / num_kv_heads, so TP must be a larger multiple of the KV heads. They are the contiguous blockskv_head_replicas(mstar/distributed/utils.py) gives, asQKVParallelLinearloads them; a model that shards K and V itself must match. Each placed layer exchanges queries and partials over the TP group, except for a plan label of an eager step that isn't one token a row and has no stored context on any row, which attends its own tokens alone. - SP mode. The partners are the node's SP ranks,
P = sp_size. Heads shard over TP alone. Without aSeqShardSpecevery layer but attention runs copied across SP; prefer TP mode where it applies. With one whoseexchangeisall_gatherorring, long steps split their tokens too (above). Every paged pool of the node must set the key. Each placed layer all-gathers the partials of a step copied or split byall_gather. Under the same exception such a step attends no slots and gathers its own tokens' shares alone; a ring step never does. Chunked prefill stays off on SP ranks (_unchunkable_nodes), so SP mode prefills a prompt in one step; chunking an SP-placed pool's prompts is not supported yet. TP mode chunks prompts as without placement.
A page's tokens (page_size, 128 when it is unset) must be divisible by P; the conductor
checks these rules against node_groups before it starts a worker. max_num_pages counts
logical pages, so raise it by P to use the memory placement frees (the pool logs its bytes per
rank). Every resource over the pool must support it, or loading fails naming the one that doesn't.
FlashInfer attention (flashinfer) with flashinfer_backend auto, fa2 or fa3
(auto runs as fa2 from SM100; after an unexplained one-off lockstep hang in an fa2
prefill over an SP-placed pool (#1363), an SP-placed pool that prefills on fa2 warns at
startup: an explicit fa2 on any architecture, and auto where FlashInfer has no fa3
(below SM90 and from SM100), while auto on SM90 prefills on fa3), RoPE positions and the
recurrent state pool (GDN, KDA; the same state on every SP partner) do; dense, flex, XPU,
block-causal and MLA (flashinfer_mla) attention, cross-attention and DSA don't, and a ring or
bounded KV pool can't set the key. A sliding_window kept in its own group of layers stays whole
on every partner, in pages of 1/P its unplaced tokens (which must be whole), while the full
layers stripe; a window at least max_seq_len reads the placed layers and is refused there. The
fa2 warning above holds for the window's attention too. A decode/prefill splits
and decode_fmha are ignored over a placed pool, and a node over one plans on the host rather
than resident (resident_planning). A node that captures a step combining labels into one plan
(SubmoduleStep.combined_labels, as classifier-free guidance does) at a batch size above 1 is
refused at load: a replay's padding rows would sit between its real ones.
A step of one token a row attends the pages after its KV write (in TP mode on tensor cores
whatever decode_tensor_cores says). Any other step attends its stored context on the pages
and its own tokens from the k and v the layer passes run, so a layer that calls
attn.run itself passes them (AttentionCallable and MLAAttention without
mla_absorb do). A fused K/V write that leaves k unrotated asks reads_step_kv() first
and rotates it when the step attends it (AttentionCallable.norm_rope_attend); a state op's fused
kernel doesn't run on such a step, so its reference's add hands the handle k and v.
In SP mode, rank r attends only the step's
own tokens in zigzag chunks r and 2P - 1 - r of each row's 2P (zigzag_rows),
which shares the causal work about evenly on rows of 2P tokens or more; it merges them into
its context partial before the gather.
Note
A top-level kv_cache: block is no longer read. It raises an error at load time,
with a message describing the migration. Move the block under resources:, keyed by
the resource name.
When a node group has several replicas, the conductor sends each new request to the replica where it is cheapest (see "Data-parallel routing" in :doc:`serving`). A resource whose state makes a request cheaper on one replica than another takes part through its spec, so models declare nothing for it.
The hook. NodeResourceSpec.routing_mirror(ctx) returns a RoutingMirror or
None, the default. The router asks once per set of node groups with replicas that
holds one of the spec's nodes, and ctx (a RoutingContext) gives the replica count,
the walks of the worker graphs there that hold the spec's nodes, the resource's keyed
labels with the walks that key each (empty for a resource that is not a KV cache), and
whether every replica's engine opens its prefix cache, which the router derives with
prefix_caches_by_rank.
A mirror is conductor-side bookkeeping that follows the resource's own rules as far as
the conductor's inputs allow. The module that defines one must stay free of kernels,
since the conductor builds it.
The mirror's methods. Each is given the request's ResourceReqConfig for the
resource, which the conductor builds and validates before the pick.
facts(config)reports the request'sRoutingFactson each replica: named counts, such aswork(prompt positions) andkv_hit(of those, the ones predicted cached). The router adds every mirror's row on a replica field by field, then prices the sum once withrouting_costinmstar/conductor/routing.py, one term perrouting_weightskey, each 0 when its facts are absent.add_request(config, replica)records the placement once the request is dispatched, and returns a handle.remove_request(handle, ran)releases it when the request leaves.ransays whether a pass came back for one of the request's partitions with a walk on the resource's node groups.
Adding a resource. A new resource adds its fields to RoutingFacts, its mirror,
and its term to routing_cost and RoutingWeights. ReplicaRouter changes only
if its fields are not counts that add, or if its mirror reads a model declaration other
than the KV ones, which then becomes a RoutingContext field that ReplicaRouter
fills. KVSpec with a PagedKVConfig is the one implementation. Its mirror runs the
worker's own PrefixIndex and PageArena per replica, with no storage, takes each
request's lease, pages and index entries at dispatch, an item's end inside a page among
them, and exists only where a walk keying the cache runs and opening_prefix_caches
says the workers open it.
The API server runs load_image / process_prompt and output encoding with torch's
intra-op pool. Torch sizes it at one thread per host core, which in a container can be
several times the CPUs the cgroup quota lets run, and an op waits on any of its threads
the scheduler has parked. The API server therefore defaults to a quarter of the CPUs it
may use (its affinity, capped by the cgroup cpu.max quota), at most 4. A model sets
its own size with PREPROCESS_TORCH_THREADS on the class (Qwen3.5 uses 4). A
deployment overrides either at the top level of its config; null keeps torch's
default:
model: "qwen3_5_4b"
preprocess_torch_threads: 4The setting applies only to the API server process. The conductor and workers keep
torch's default. It does not size video decoding: each torchcodec decode that asks
media_limits.video_decode_threads (Model.load_video does) runs FFmpeg with up to 8
threads, no more than the CPUs the process's CPU plan leaves unpinned (CpuPlan.left).
The data worker prepares one request at a time on its own thread, so a request's media
decode holds up every request behind it and the output chunks of the ones running. A model
whose load_image / load_audio / load_video and process_prompt keep no shared
state sets PREPROCESS_PARALLEL = True, and requests then prepare on a pool sized to the
CPUs left (CpuPlan.prepare_workers: at most 8, and the pool times the process's intra-op
thread count stays within the CPUs left, so a higher preprocess_torch_threads means fewer
workers).
Tensors are still stored and requests still reach the conductor from the data worker's thread,
in arrival order. A deployment sets the pool at the top level of its config; 1 turns it off:
model: "qwen3_omni"
preprocess_workers: 2The pool only helps work that releases the GIL, so measure each stage first. Qwen3-Omni,
V-JEPA2, Cosmos3, Parakeet, Nemotron ASR and Omnilingual ASR declare it (configs/vjepa2.yaml and configs/vjepa2_ac.yaml set 8
and 4, the sizes measured for their video decode; Cosmos3's DROID policy configs set 1, since
a robot sends one request at a time). A model with client streams always prepares inline,
since an append has to follow its request.
A model that takes several images per request and whose load_image is mostly a decode can
also set PREPROCESS_PARALLEL_IMAGES = True, which decodes one request's images at once on a
second pool the size of the prepare pool, unless preprocess_workers is 1. Pi0.5 declares it
for its camera frames.
A pool that small is slow for a large copy, and a 121-frame 832x480 video is 145 MB. A
video decoder should therefore emit its clip in the layout the encoder takes, transposed
on the GPU inside its forward, so postprocess hands the tensor it reads straight to the
encoder: [T, C, H, W] for torchcodec's VideoEncoder (Cosmos3) and [T, H, W, 3]
rgb24 frames for PyAV (Wan2.2, LTX-2.5) or a raw video_frame stream (LingBot-World).
A permute in postprocess costs a host copy of the whole clip, or a slower copy of each
frame inside PyAV. A PyAV mp4 should be written with
mstar.model.components.video_encode.encode_mp4, which takes the model's x264 options,
rate and padding, and converts the frames to the encoder's pixel format on a thread a few
frames ahead of the encoder, so x264 gets the frames the plain stream.encode loop
would hand it without waiting on the conversion between them. A torchcodec mp4 of a long
clip can be written with video_encode.encode_mp4_chunked, which encodes it in parallel
pieces on the CPU plan's threads and joins them into one stream (a keyframe per piece). The
SHM arena's CPU read of an
output slices a large copy over threads of its own
(mstar.utils.host_copy.HostCopier), one encode's share of the CPUs.
To shard a node across several GPUs, add tp_size to its node_groups entry and list
tp_size ranks. The runtime splits the group's ranks into TP groups of that size and
builds one comm_group per shard.
A node is sharded only if its components are built from the tensor-parallel modules in
mstar/model/components/distributed: ParallelAttention, ParallelCrossAttention, ParallelGatedMLP,
ColumnParallelLinear, RowParallelLinear, VocabParallelEmbedding and others.
The weight_loader of each such parameter slices it automatically. See Loading
weights. A node whose components do not use these modules is replicated on every rank
instead. For example, a tensor-parallel Qwen3-Omni Talker keeps its code predictor
replicated.
Once a component is built from these modules, moving from one GPU to tensor parallelism
requires only the YAML change and a small sharding declaration. No model code changes. The
example below runs the Orpheus LLM with tensor parallelism across two GPUs. See
configs/orpheus_tp2.yaml:
model: "orpheus"
node_groups:
- node_names: [LLM]
ranks: [0, 1]
tp_size: 2
graph_walks: [prefill, decode]
- node_names: [snac_decoder]
ranks: [0]
graph_walks: [snac_chunk]For a node to be eligible for tp_size > 1 it must be declared TP-enabled by the model.
Override get_default_sharding_config to return a ShardingConfig that names the
shardable nodes, and any non-default shard dimensions:
def get_default_sharding_config(self):
from mstar.distributed.base import ShardingConfig
return ShardingConfig(groups=[], tp_enabled_nodes={"LLM"}, shard_dim={})Two separate mechanisms split work across the TP group. Keep them distinct:
- Weights are sharded inside the components, by each parameter's
weight_loader. Column-parallel and row-parallel linear layers built with thecomm_groupdo this automatically, once the module is constructed for tensor parallelism. The config needs nothing more. - Activations that cross a node boundary are handled by
shard_dimin theShardingConfig. It maps an inter-node edge or signal name to the dimension along which that tensor is split across the group. If a name is absent, or maps toNone, the tensor is replicated to every rank. You need an entry only for edges where the producer and the consumer both keep the data sharded. The common case is replicated activations, which needs no entry. You can also setshard_dimper run, under asharding_configblock in the YAML. A sharded signal is split over the node's whole lockstep instance, so placement refuses one on a node group withsp_sizeorcfg_sizeabove 1.
If a node group has tp_size > 1 and names a node that is not in tp_enabled_nodes,
loading fails. See configs/qwen3omni_thinker_tp2.yaml for an example with several node
types.
A row-parallel projection followed by an RMSNorm can leave its all-reduce to the norm, which
runs both in one fused kernel. Set the projection's reduce_results = False, so it returns
its partial sum, and hand the norm the comm group: RMSNorm.forward_residual(x, residual,
comm_group) reduces, adds and norms, and RMSNorm.forward(x, comm_group) reduces and
norms (a post-norm before the residual add). The two go together, since a norm called
without the group norms an unreduced partial sum, which is wrong only under TP. Keep both in
the module whose forward calls the norms, as Qwen3_5LanguageModel,
MuseGlimmerTextModel and RippleTextModel do. MultimodalDecoderModel builds the
kernel's workspace for its TP node.
A diffusion node that runs its classifier-free-guidance branches as one batch can split
them over a node group's cfg_size ranks, a mesh axis of its own beside tp_size
and sp_size (see :doc:`serving`). Declare the node and the most branches it runs in
cfg_enabled_nodes; placement refuses a cfg_size that doesn't divide that count:
def get_default_sharding_config(self):
return ShardingConfig(
groups=[], tp_enabled_nodes=set(), shard_dim={},
cfg_enabled_nodes={"dit": 2},
)The submodule receives cfg_group from get_submodule, takes its rows of the branch
batch with my_rows(cfg_group, branches)
(mstar/distributed/communication.py), and after the forward
reassembles them with cfg_group.all_gather(out, dim=0). Every rank then runs the same
guidance combine and solver step on replicated latents. CFG ranks keep every attention
head; only TP and SP ranks split them. The ranks run in lockstep on the same requests, so
any per-request resource the node declares must stay the same shape on every rank, and a
node with cfg_size > 1 can't hold a paged KV cache (placement refuses one before
any worker starts): branches that hold different KV are separate nodes in a graph Parallel
instead, on a node group with cfg_size: 1, as BAGEL's LLM_cfg_text and LLM_cfg_img are.
Wan22DitSubmodule is the reference.
A sharded MoE node can split its routed experts in two ways. Under TP, the default, every
rank holds a slice of every expert. Under EP, each rank holds whole experts. A deployment
picks one per node group with an ep_config block; see :doc:`serving` for the keys. The
shared pieces are in mstar/model/components/moe_parallel.py. The Qwen3-Omni Thinker
(mstar/model/qwen3_omni/components/thinker.py) is the reference user.
To support it in a model:
Declare the node EP-enabled. Add it to
ep_enabled_nodesinget_default_sharding_config. Anep_configon any other node fails at start-up.return ShardingConfig( groups=[], tp_enabled_nodes={"LLM"}, shard_dim={}, ep_enabled_nodes={"LLM"}, )
Build the MoE layers with
instantiate_moe_block. Acceptmoe_parallelinget_submoduleand pass it, with the node'stp_group, down to each MoE layer. The helper returns aParallelSparseMoeBlock, withexpert_parallel=Trueunder EP. It keeps its experts in aFusedExperts(block.experts), which owns thegate_up_proj/down_projparameters, the weight loaders that keep only the rank's slice or its own experts, and the dispatch.moe_parallel=Nonegives TP.self.mlp = instantiate_moe_block( hidden_size=cfg.hidden_size, moe_intermediate_size=cfg.moe_intermediate_size, num_experts=cfg.num_experts, num_experts_per_tok=cfg.num_experts_per_tok, norm_topk_prob=cfg.norm_topk_prob, comm_group=tp_group, moe_parallel=moe_parallel, )
Optionally, mix in
MoeParallelMixinon the model's top-level module, and callself.init_moe_parallel(moe_parallel)at the end of its__init__. It provides:maybe_log_expert_load(synthetic), which implementslog_expert_load_every. Call it from eager code before each forward, usually frompreprocess, withsynthetic=engine_inputs.syntheticso capture and warmup routing are discarded.moe_token_valid(hidden_states, num_valid_tokens), which implementsmask_padding. It returns thetoken_validmask for the MoE blocks, orNonewhen masking is off. Pass the mask only to MoE layers, not dense MLPs.num_valid_tokensmust be a(1,)device tensor thatpreprocessreturns as a static input, so a replay can update it; fill it fromengine_inputs.real_seq_len.ThinkerSubmodule.preprocessshows the wiring.
instantiate_moe_block builds no shared-expert block. A node with one
(ParallelSparseMoeBlock(shared_expert=...), the Qwen3-Omni Talker's) builds it directly and
stays TP-only.
Launch configs that depend on the GPU and the shapes, such as the fused MoE's tiles, are data,
not code. Each kernel op (mstar/kernels) names the dimensions its configs are keyed by, and
its tuned configs live in one JSON table per GPU under mstar/kernels/tables/<op>/. A model
on a shared component gets them without code of its own: the component looks its shape up once
per process and keeps the default config where no table has it, and the server logs the sweep
command for that shape (kernel table: no moe_gemm bfloat16 E=128,N=1536,K=2048,top_k=8 ...).
A block-fp8 dense linear (fp8_linear, mstar/utils/dense_fp8.py) takes its kernel per token
count the same way: the w8a16 Triton kernel, FlashInfer's or cuBLAS's block-scale GEMM, where the
GPU's row names one that runs there. A cacheless self-attention (ragged_attention) takes its
kernel per head geometry and longest query segment: torch's cuDNN SDPA or one of FlashInfer's
ragged backends. A table can also be named for an architecture (sm_103.json) rather than one
GPU model, for rows that hold on every GPU of it; the sweep's --arch writes that file.
python -m benchmark.kernels.sweep times the candidates at those shapes on one GPU, checks
each against a reference output, and writes the winners with where they were measured.
--out DIR with MSTAR_KERNEL_TABLES=DIR serves a sweep before it is committed;
benchmark/kernels/README.md has the workflow and how to add an op.
Orpheus (mstar/model/orpheus/) is a small and complete reference. It is a TTS model. A
Llama 3.2 3B LLM emits audio tokens, and a SNAC decoder converts them into 24 kHz PCM.
The two nodes have different needs. The LLM declares a paged array and a sampler, shown under
Resource specs. No declaration names snac_decoder, so that node receives no state. The
SNAC decoder instead declares how it must be run, using two class attributes on the submodule:
class SNACDecoderSubmodule(NodeSubmodule):
disable_torch_compile = True # runs in fp32, without compilation
disable_autocast = TrueThere are three graph walks: prefill and a decode Loop on the LLM, plus a
snac_chunk node that emits audio to the client:
snac_chunk = GraphNode(
name="snac_decoder",
input_names=["new_token"],
outputs=[GraphEdge(next_node=EMIT_TO_CLIENT, name="audio_chunk",
output_modality="audio")],
)process_prompt formats the string "{voice}: {text}", tokenizes it, wraps the ids in
the model's start and end tokens, and returns {"text_inputs": [ids]}, with its layout under
state_layout.
get_submodule builds either the Llama LLM submodule, a DeclaredTokenDecoder that
declares little beyond its walk names and capture buckets, or the SNAC
decoder submodule, and caches the result. The SNAC submodule emits int16
PCM, which the default postprocess passes through as raw bytes.
Orpheus also demonstrates the async partition API, described in the next section. The LLM and the SNAC decoder run as two partitions connected by a streaming edge. Audio is therefore decoded in a sliding window while the LLM is still generating.
Orpheus is a single pipeline. BAGEL (mstar/model/bagel/) is much more complex, and it
shows why the graph abstraction is useful. BAGEL is a unified model. It performs image
understanding, which maps an image to text, and image generation, which maps text to an
image. Both use the same Qwen2 LLM. That LLM is also the denoiser for rectified-flow image
generation. The steps below follow the same order as the rest of this page.
Step 1 — Register. This is already done in registry.py, with the entry
"bagel": BagelModel and an HF_MODELS entry that points to
ByteDance-Seed/BAGEL-7B-MoT.
Step 2 and 2a — Nodes and resources. The model has four core nodes: a ViT encoder
(SigLIP2, used for understanding), a VAE encoder (FLUX, used for editing and generation),
the LLM (Qwen2, which contains the embedding, the transformer, the lm_head and the CFG
logic), and a VAE decoder. It has four more nodes for the CFG-parallel image-generation
path described below: init_latents, the two branch nodes LLM_cfg_text and
LLM_cfg_img, and combine_cfg.
Under CFG parallel the three LLM nodes share one set of state: the kv array and the
sampler name all three in nodes. Otherwise they name LLM alone. The ViT encoder
declares an array of its own with max_len=0 and a Block() read, so that its block loop
can be CUDA-graph captured:
def declare_state(self) -> list[Array | Sampler]:
c, vit = self.config, self.config.vit_config
llm = self._LLM_NODES if self._has_cfg_parallel else frozenset({"LLM"})
attend = Attend(q_heads=c.num_attention_heads)
text = frozenset({"prefill_text", DECODE})
image = frozenset(LLM_GRAPH_WALKS[self._has_cfg_parallel]) - text
return [
Array(VIT_TOKENS, nodes={"vit_encoder"}, ..., max_len=0,
reads={VIT_ATTN: Read(mask=Block(), op=Attend(q_heads=vit.num_attention_heads))}),
Array("kv", nodes=llm, ..., labels=frozenset(NODE_TO_CFG_LABEL.values()),
positions=Positions(rope=Rope(theta=c.rope_theta)),
reads={ATTN: Read(mask=Causal(), op=attend, walks=text),
ATTN_IMAGE: Read(mask=Bidirectional(), op=attend, walks=image)}),
Sampler(SAMPLER, nodes=llm, vocab_size=c.vocab_size, penalty=True),
]No declaration names the VAE encoder, init_latents, combine_cfg or the VAE decoder, so
those nodes receive no state. The CFG branch nodes are used only when the config places them,
in CFG-parallel mode, described under Step 6, and get_worker_graphs reads that into
_has_cfg_parallel. A single-GPU config never routes requests to them.
Per-request resources. Whether a request uses classifier-free guidance determines
which cache labels it reads. This is a property of the request, not of the deployment.
BAGEL's get_request_resource_configs therefore returns a KVReqConfig that names
the active labels per node and walk, and those it publishes under CFG parallel and P/D,
together with its SampleSettings. It settles guidance from the request's modalities and
kwargs as its walk_plan does (an edit, image in and image out, defaults to image guidance).
It keeps that config, which other models leave to their layouts (Layouts), until labels per
node come from steps. A request with guidance disabled names only "main", so a
PD-disaggregated transfer copies only that stream.
The LLM node is deliberately coarse. It contains the text embedding, the lm_head and
the flow projection. These always run on the same GPU, so splitting them into separate
graph nodes would only add IPC overhead. This is a general modeling rule: make a node as
coarse as the colocation boundary allows.
Step 3 — Graph walks. Understanding and generation are different pipelines, so BAGEL
returns five walks from get_graph_walk_graphs instead of two:
| Graph walk | What it does |
|---|---|
prefill_text |
Embed text tokens and prefill the LLM. Attention is causal. |
prefill_vit |
vit_encoder, then the LLM. Encodes an input image for understanding. Attention
is bidirectional. |
prefill_vae |
vae_encoder, then the LLM. Encodes an image for editing or generation. |
decode |
Autoregressive text generation. This is a Loop, the same as in Orpheus. |
image_gen |
The flow-matching denoising Loop. The LLM applies CFG and one Euler step per
iteration. vae_decoder then converts the final latents into pixels. |
The two encoder walks are Sequential chains of two nodes. image_gen is a Loop
followed by the decoder. In the loop body, latents and time_index are routed back
to the same node, and the loop's outputs pass the final latents to vae_decoder:
prefill_vit = Sequential([
GraphNode(name="vit_encoder", input_names=["image_inputs"],
outputs=[GraphEdge(next_node="LLM", name="img_emb")]),
GraphNode(name="LLM", input_names=["img_emb"],
outputs=[GraphEdge(next_node=EMIT_TO_CLIENT, name="new_token",
output_modality="text", persist=True)]),
])
image_gen = Sequential([
Loop(
section=GraphNode(
name="LLM",
input_names=["latents", "time_index"],
outputs=[GraphEdge(next_node="LLM", name="latents"),
GraphEdge(next_node="LLM", name="time_index")],
),
max_iters=self.config.num_timesteps - 1, # one Euler step per interval
outputs=[GraphEdge(next_node="vae_decoder", name="latents")],
),
GraphNode(
name="vae_decoder",
input_names=["latents"],
outputs=[GraphEdge(next_node=EMIT_TO_CLIENT, name="image_output",
output_modality="image")],
),
])Declared outputs are conditional. A node's outputs list is the set of edges that
the node can emit. What it emits on a given step depends on what its submodule produces.
new_token above is the clearest example. The LLM samples a token only when the request
requires text output, which is the case on the understanding path and on every decode
step. On the image-generation and editing paths, the same node still runs and writes the
KV cache, but samples no token, so it does not produce new_token. The edge is present
in the graph because understanding requests need it. Treat declared edges as the possible
outputs, and let the submodule decide which of them are produced on each step.
Choosing the walks per request. BAGEL's plan follows a schedule, unlike Orpheus's two
fixed steps. The output modality is known in advance from the request's
output_modalities, so walk_plan builds the prefill schedule by iterating over the
interleaved text and image inputs, one walk per text span or image, and then adds
decode for text output or image_gen for image output. In think_mode the model
first decodes a reasoning trace, so the plan runs decode and then image_gen.
Each prefill walk reads one span, so process_prompt hands every span over as a signal
of its own with :func:`mstar.model.multimodal.split_spans` (text_inputs.0,
image_inputs.0, ...), and each step binds its walk's input to one:
WalkStep("prefill_text", bind={"text_inputs": "text_inputs.1"}). An ids span keys the
entry its source names, so split_spans renames the KV layout's ids-span sources to
match. This is the same method Orpheus implements; BAGEL's plan is only longer.
Step 4 — Submodules. Each node maps to a NodeSubmodule in bagel/submodules.py:
ViTEncoderSubmodule, VAEEncoderSubmodule, LLMSubmodule and
VAEDecoderSubmodule. get_submodule builds them on demand, so a worker that runs
only vit_encoder never allocates the 7B LLM. process_prompt tokenizes the prompt,
and also a system prompt in think_mode. postprocess selects an encoding by
modality: utf-8 text for decode, and PNG bytes for image.
Step 4a — Step declarations. BAGEL's LLMSubmodule.declare_step is the most complex
declaration in the tree. Read it to see every advanced ArrayStep field in context:
forks: guidance requires acfg_textstream forked frommain, and the fork time differs by walk. Inprefill_text, the branch must keep the context from before the text, so it forks before any planning or writing. Inprefill_vitandprefill_vae, the branch must include the image, so it forks at commit time, after this step's writes are applied.- Several labels: a guided
prefill_text,mixed_textordecodestep lists every row onmain, then oncfg_img, and runs one forward over both. A guided denoise step on one node listsmainand the guidance labels its walk runs. Each is one plan, andposition_idsconcatenates the ids in the same label-major order.placeholdershold an unguided row's place oncfg_imgin a guided text step. keep=False: the denoise step plans and reads, but keeps nothing.cg_key_info: the step records whether guidance is enabled. This value matches theadditional_key_infoof the twodecodecapture configs, so each replay uses its own bucket. The submodule'scg_key_info()method reports the same value from the batch, because the engine leases a slot before the step is declared.
Step 6 — Config and disaggregation. This is the main benefit of the graph abstraction. The same model code runs on one GPU:
model: "bagel"
max_seq_len: 32768
node_groups:
- {node_names: [vit_encoder], ranks: [0]}
- {node_names: [vae_encoder, vae_decoder], ranks: [0]}
- {node_names: [LLM], ranks: [0]}The same code also runs disaggregated across GPUs. Assign the same LLM node to
different ranks per graph walk: prefill on GPU 0, decode on GPU 1, and image generation on
GPU 2:
node_groups:
- {node_names: [LLM], ranks: [0], graph_walks: [prefill_text, prefill_vit, prefill_vae]}
- {node_names: [LLM], ranks: [1], graph_walks: [decode]}
- {node_names: [LLM], ranks: [2], graph_walks: [image_gen, image_gen_head, image_gen_guided2, image_gen_guided3]}A guided request's denoising runs as a sequence of walks. Its guided steps form one window,
known when the request arrives, which runs as image_gen_guided2 (text guidance) or
image_gen_guided3 (text and image guidance), every branch in one combined forward on
LLM. The steps before the window run as image_gen_head, and the steps after it run
as image_gen, which also decodes; a window that reaches the last step is followed by the
decode alone (image_decode). Each walk's loop is capped per request at its share of the
steps (get_loop_caps), the conductor moves the request from one walk to the next, and
the latents cross between walks as persisted edges. Each walk's branch count is fixed when
the request arrives, so no node reads a loop counter to decide it. A request without
guidance runs image_gen alone. A config that places LLM per walk, as above, lists
all four of its denoise walks.
BAGEL also supports a CFG-parallel mode. When the config names the extra
LLM_cfg_text and LLM_cfg_img nodes, as in configs/bagel_cfg_parallel.yaml, the
window runs as image_gen_cfg2 or image_gen_cfg3 instead. The loop body of that walk
is a Parallel of the guidance branches, each on its own GPU, feeding a combine_cfg
node, and a request without guidance never reaches the branch GPUs. The model code detects
this mode only from the node names present in the config, so the extra parallelism is
enabled in YAML and requires no code change. One model can therefore run in many physical
layouts.
Whisper (mstar/model/whisper/) is the reference for encoder-decoder models. It is the
smallest complete example of cross-attention over a fixed context.
It has two nodes. audio_encoder holds no state. decoder declares four: its own array
kv_cache, the context array cross_kv_cache with a Bidirectional() read and
queries=, the decoder_state Value, and a sampler. kv_cache declares
Positions() with no Rope, only for the position counter. Whisper uses learned position
embeddings, so positions(self.kv) indexes an embed_positions table instead of driving
RoPE. The sampler declares penalty and min_p only where the checkpoint's sampling
defaults use them (sampler_capabilities), since ASR transcription decodes greedily.
decoder_state holds each request's timestamp rule row, ts_rules, and its
transcript, each sampled token at the position it is fed back at, which the align walk
reads for word timestamps.
The page counts show how to size a cache for a specific model instead of using the
default. Both caches use 128-token pages and hold WhisperModel.MAX_CONCURRENT_REQUESTS
(64) requests at full length, plus the sink page that every KVManager keeps out of
circulation. A sequence is at most max_target_positions (448) tokens, which is 4 pages
per request, so the self-attention cache has 64 * 4 + 1 = 257 pages, about 5 GiB for
large-v3 in bf16. The 2048-page default would use about 40 GiB. The fixed 30-second context
window is max_source_positions (1500) tokens, which is 12 pages per request held from
prefill to the last token, so the context cache has 64 * 12 + 1 = 769 pages, about 15 GiB.
kernel_settings sets these as max_num_pages on both arrays, through
mstar.model.utils.kv_pages, and decode_tensor_cores: False on both reads, since the
CUDA-core decode kernel is faster for Whisper's MHA.
Sizing the caches is only half of it: configs/whisper_large.yaml also sets
max_concurrent_requests: 64, so the conductor queues further requests instead of the
worker offloading decoding requests to admit them. WhisperModel.validate_config_yaml
refuses at startup a cap that either cache cannot hold after the YAML's resources:
overrides, through mstar.model.utils.check_kv_pools_fit, which any model with KV caches
can call the same way. A model on the contract can size a pool by the request it is sized for,
request_layouts={array: {label: Layout}} with its loop_caps, as Chatterbox passes its
T3 layout on both labels of a guided request with the decode loop at its limit.
The decoder node's declare_step shows how a write-once context is expressed with spans.
Canary's comes from EncoderDecoderNode (mstar/model/components/encoder_decoder.py),
a SampledTokenDecoder that adds only the cross-attention part: the context, an array with a
Bidirectional() read and queries=, and the encoder output on the prefill row, which
preprocess adds to the context (ArrayStep(eager_add=True)) so the prefill replays a
graph sized by its prompt. Whisper's declares its own and adds its context the same way. The
context segments have a non-zero span in the prefill step that adds them, and a span of 0 in
every later step. The prefill step's commit converts the reservation into resident pages that the
later steps read.
The forward reads the step's ids with positions(self.kv), a prefill's last token per row
with at_last_token, and draws with sample. A decode step takes its input tokens from
last_sampled, with a routed row spliced over its slot's by splice_loopback_ids. Each
step's forward reads decoder_state and writes back the next rule row and the sampled token.
The align walk reads the transcript once on the host to build its teacher-forced sequence.
Flow-matching image and video transformers (FLUX-family DiTs, Z-Image, Wan-style
video) share one graph shape, so M* ships it as a scaffold in
mstar/model/components/diffusion/; a model adds only its exact component
ports, its checkpoint remap and a few hooks.
Graph and Walks:
text_encoder ──text_embeds──▶ Loop("denoise_loop", dit) ──latents──▶ vae_decoder ──▶ EMIT image
vae_encoder ──ref_latents──▶ (image_edit only)
encode_text : text_inputs -> text_embeds (persist)
encode_image : image_inputs -> ref_latents (persist)
image_gen : Sequential[Loop(dit) -> vae_decoder]
image_edit : image_gen with ref_latents as an extra Loop input
latents is the loop-back edge; every other input is re-injected each
iteration. The step index is the engine's loop counter
(fwd_info.dynamic_loop_iter_counts), so there is no host sync per step.
The dit node subclasses DenoiseLoopSubmodule (denoise_loop.py), which owns
the engine contract — seeded initial noise, memoized schedules, equal-key request
batching (can_batch), stacked forward_batched, the declare_step over the arrays
it attends through and one CUDA-graph bucket per bucket_key with the Euler update inside
the graph — and asks the model for the hooks below. The loop ends on its iteration count:
a model whose request picks its step count returns it from get_loop_caps
({"denoise_loop": steps}), resolved exactly as schedule_for resolves it, and a
stage that always runs its graph max_iters declares nothing. The scaffold refuses an
iteration past the schedule. The hooks:
class MyDenoise(DenoiseLoopSubmodule):
def bucket_key_for(self, fwd_info): ... # hashable: latent grid, text length, ...
def schedule_for(self, walk, settings, key): ... # FlowMatchSchedule (flow_match.py), memoized
# optional: what the request sets of its schedule; its step count by default
def schedule_settings(self, fwd_info): ...
def seed_latents(self, key, gen): ... # [L, C] initial noise, seeded
def request_inputs(self, fwd_info, inputs, key): ... # per-row conditioning tensors
def num_tokens(self, key): ... # image tokens for the attention step
def capture_request_inputs(self, key, device): ... # dummy rows for graph capture
def denoise(self, engine_inputs, key, latents, timestep, sigma, sigma_next, **cond): ...
# optional: extra attention spans (a refiner over the image tokens alone, ...)
def attention_segments(self, key): return {TOKENS: (("main", self.num_tokens(key)),)}bucket_key is whatever has to match for two rows to share a forward, and it is also
the CUDA-graph bucket key — shape, but equally a knob like whether CFG is on.
A multi-step scheduler that carries solver state between iterations names it in
SOLVER_STATE; those become loop-back edges alongside latents, seeded by
seed_loop_back and returned from denoise as a {name: tensor} mapping. UniPC
comes with the scaffold: a subclass of UniPCDenoiseLoopSubmodule implements
model_output in place of denoise, and the base carries
unipc_model_outputs and unipc_last_sample and steps each row by the loop counter
from its own tables, with no host read, so rows at different steps share a forward
(wan22 is the example). A scheduler needing another per-step
scalar adds a STEP_SCALARS entry naming the attribute of its FlowMatchSchedule
to slice it from — the base stages that tensor on the device and slices it per step, so
nothing in the batching path has to change.
A model whose seed_loop_back depends on nothing but the bucket key and the request's
seed sets PREFETCH_SEEDS = True (klein and Z-Image do): the base then draws the noise
when the request reaches the worker, on a thread of its own, and iteration 0 only copies
it to the device. Otherwise the CPU draw runs on the GPU thread at iteration 0, once per
joining row.
Every span a layer attends over is a segment of a declared array with max_len 0, listed
in attention_segments per request. A layer calls attend(self.state, self.layer, q, k, v)
from attention.py, which adds and reads through its array, and runs SDPA on a layer no
engine bound a handle to. Arrays of one geometry share one resource, each under its own label.
Shared pieces to build the components from: flow_match.py (shift schedules and the
exact Euler step), rope.py (multi-axis rotary embeddings), attention.py (joint
attention over packed tokens: SDPA, or the ragged FlashInfer resource through a read),
text_encoder.py (native Qwen3 hidden-state taps), autoencoder_kl.py (the KL VAE
family), image_io.py (latent packing, PNG) and lora.py (static LoRA merge at load,
also into fused q/k/v shards via FusedColumnLinear.shard_slice; the checkpoint
parsing it uses is shared, in mstar/engine/resources/lora/checkpoint.py).
Two pieces swap a one-pass kernel in for what Inductor would emit, inside a compiled
transformer only; eager calls run the plain PyTorch formulation, which is what a port's
bit-exact tests hold. modulation.ada_layer_norm is a block's gated residual add,
LayerNorm and shift/scale modulation, with the modulation given per group of tokens
([B, G, D]: one group per sample, or per latent frame as in Wan2.2); Inductor reads a
LayerNorm row wider than 1024 twice. qk_norm_rope.rms_norm_rope is the q/k RMSNorm
across heads followed by the interleaved-pair RoPE, q and k in one launch. Both agree
with eager to bf16 rounding rather than bitwise. Wan2.2 is the first user;
test/modular/test_diffusion_dit_kernels.py holds them to the eager form.
On the API side DiffusionImageAdapter (mstar/api_server/openai/adapters.py) serves
/v1/images/generations and /v1/images/edits for any such model (size becomes
width/height, seed and other knobs pass through); register it under the
model's registry key. The SDK exposes generate_image and edit_image.
A decoder LLM with a ViT in front of it (Qwen3.5, Muse Glimmer) shares one graph and
two node shapes, so M* ships them as a scaffold; a model adds its components, its
resource declarations and a few overrides. The older span-split path, a walk per text
span and attachment (prefill_plan, find_media_spans, split_around_spans,
prompt_spans; BAGEL and Qwen3-Omni), is legacy: a new model builds on this scaffold.
BAGEL stays on it because a whole-prompt row would lose what its spans carry: an image to
generate from is written twice, as VAE latents and as ViT patches, by walks of their own,
its classifier-free-guidance branches open KV labels over different subsets of the
prompt's spans, and its leading text span is keyed for prefix reuse. Moving Qwen3-Omni
and BAGEL onto the scaffold is tracked as a follow-up.
Graph and walks (mstar/model/multimodal.py), from the model's MediaEncoder list:
prefill_text : LLM(text_inputs) a text-only prompt
prefill_vision : vision_encoder(pixel_values) each run's grid off MEDIA_LAYOUT
-> LLM(text_inputs, vision_embeds) the whole prompt, one row
prefill_<a>_<b>: Parallel(encoder a, encoder b) -> LLM one walk per subset of encoders
decode : Loop("decode_loop", LLM)
mixed : prefill_text and decode rows in one step (a combined walk)
The Model subclasses MultimodalDecoderModel (mstar/model/multimodal_decoder.py),
which declares the walks (media_walks(max_iters, encoders) and
text_combined_walks()), plans the one prefill walk media_prefill_step picks
and then decode (walk_plan), decodes text output, and owns
media intake (process_prompt), the decode loop's cap (get_loop_caps, the
request's output budget less the prefill's token), the KV label and its layout (kv_labels,
kv_layout), the sampler's per-request config (get_request_resource_configs:
the request's SAMPLING_KEYS over sampling_defaults, which __init__ reads
from the checkpoint's generation_config.json), the LLM's SamplerSpec
(_sampler_spec, whose min-p filter is on only where the checkpoint defaults
min_p, so a request's min_p > 0 is otherwise refused at admission), sharding, the
submodule cache and where the files come from. A model sets NAME and
METADATA_FILES and writes __init__ (setting
config, tokenizer and media), its resources (the sampler from
_sampler_spec; MIXED_SPLIT is the attention resources' split of a mixed step),
_prompt_ids and _create_submodule. A model on the state contract subclasses
DeclaredDecoderModel, which returns the layout as state_layout and the request's
settings as SampleSettings. The scaffold's
rebuild_policies lets the KV cache drop a request and reprefill its ids on its prefill
walk, a media prompt's with host copies of its encoders' outputs. A model whose LLM node
holds a resource that can't drop returns {}, or the engine refuses to start; Muse
Glimmer and Ripple-65B return {} until a drop has run through a sliding-window pool,
and Qwen3-ASR until one has run through its captured prefill. A model without text prompts
(TEXT_PROMPTS = False) is rebuilt on its media walks alone. media holds a
MediaIntake per attachment modality: its MediaEncoder (modalities may share one),
its placeholder pad id (intakes may share one: the nth placeholder of a pad goes to the
nth attachment of that pad in the order of the prompt's parts, which is where _prompt_ids
renders each), the sentinels a template leaves to its processor, how many
placeholders an attachment fills (runs(attachment, metadata), one by default; a Qwen3.5 video fills one
per temporal group), and prepare, which turns an attachment into the encoder's input
tensors and each placeholder's tokens (a PreparedMedia). _prompt_ids renders and
tokenizes the whole prompt once. intake then refuses a prompt whose placeholders don't
match its attachments, prepares each attachment in prompt order, expands each placeholder
to its run (a MediaRun) with expand_media_pads (one pad per token, read from the run's encoder output,
with its grid), and process_prompt returns the pads' layout as MEDIA_LAYOUT
metadata and "kv_layout": self.kv_layout(layout, length).
The layout's runs say which encoders the prompt runs (media_encoders), for the prefill
schedule's walk and the KV layout's alike. kv_layout lays a media prompt's text out as
ids spans and each run of pads as a rows span, which moves the position counter by
media_advance, the run's tokens by default. Under Qwen-style MRoPE it is the positions'
extent instead, an image spanning max(h', w') positions rather than its tokens, which
Qwen3.5's override computes from the run's grid. process_prompt also empties each sent
attachment's raw <modality>_inputs, spent once prepared. The encoders' inputs are
lists in prompt order, as their outputs fill the pads. Each run's patch grid (the
grids of its PreparedMedia) rides MEDIA_LAYOUT rather than a tensor, so a
tower's grid maths in prepare_inputs reads host values (PackedBlockLoopEncoder.grids)
and never syncs on the device.
The LLM node subclasses SampledTokenDecoder
(mstar/model/components/sampled_token_decoder.py), which owns the capture configs
(decode, packed text prefill, mixed, and one packed capture every media walk replays,
which also caps their batches), device loop-back of the sampled token, the splice of
the encoders' outputs into the prompt's embeddings inside the forward
(where(mask, media[cumsum(mask) - 1], embed(ids)), the media in prompt order and the mask
of their tokens laid out from MEDIA_LAYOUT, not found by pad id, so media may share a
placeholder token), the step declaration around the
sampler and the position resource, the forward through the sampler, and the stop: a
TokenStop over the prefill walks and decode. A stop id on a decode row ends the
loop; a stop id on the prefill row (the step the walk plan marks FINAL_PREFILL), or
a one-token budget, ends the partition before decode runs. The loop's cap,
max(1, max_output_tokens - 1), is the only budget. Each encoder's rows match the tokens of
its runs in the layout, checked in prepare_inputs. A prompt's ids, pads and
sentinels included, feed the repetition penalty, as in transformers and vLLM. A model
writes:
class LLMSubmodule(SampledTokenDecoder):
ATTN_KEY = ATTN_FULL # the attention resource that picks each prefill row's last token
def layer_steps(self): # this step's work on the layers' resources
return {KV_CACHE: KVStep(), ATTN_FULL: AttentionStep(causal=True)}
# optional, for positions the resource cannot place (MRoPE):
def prompt_positions(self, start, layout, num_tokens): ... # -> (custom_pos_ids, advance)
def decoder_kwargs(self, graph_walk, engine_inputs, num_tokens, **tensors): ...Its decoder module provides embed(ids), forward(input_embeds, *, label, **kw)
(the stack through its final norm), logits(hidden) and embed_dtype, as
Qwen3_5ForCausalLM and Qwen3DenseLM do. The sampler is declared under
mstar.model.keys' SAMPLER, and the positions under the node's POS_KEY
(ROPE unless the model's layers name another pos_key). A model on the state contract
subclasses DeclaredTokenDecoder instead and sets none of ATTN_KEY, POS_KEY or
layer_steps, since its positions are the array's: its node_step, next_position,
loopback_tokens, last_token_hidden and sample_tokens make the handle calls, and a model
overrides node_step only when it steps a second array, as Qwen3.5 does. MSTAR_DECODE_BUCKETS sets the
default decode ladder, MIXED_MAX_PREFILL_ROWS caps a mixed step's prompt rows, capture_compile_mode(walk) names a capture's compile_mode (None, the default,
takes the deployment's compile level), and CAPTURE_PREFILL = False captures decode
only. A lone encoder's MediaEncoder.walk names its media walk when the model's graph
calls it something other than prefill_<name> (Qwen3-ASR's and Canary-Qwen's
prefill). A graph that names its other walks otherwise sets
PREFILL_WALK, DECODE_WALK and DECODE_LOOP.
It reports a text prompt's length and a decode row's one token
(get_input_sequence_len, speculative_input_seq_len), so a model chunks its text
prefill by declaring get_chunking_policies alone, as Qwen3.5 does.
The vision_encoder node subclasses PackedBlockLoopEncoder
(mstar/model/components/block_loop_encoder.py): up to MAX_BATCH_SIZE requests per
step, every image packed into one call, and the tower's block loop replayed as a piecewise
region per patch bucket (BLOCK_LOOP_PATCH_BUCKETS), the step as one row. Its prepare_inputs returns the
tower's tensors in tensor_inputs and, in resource_step_info, each of
SEGMENT_LABELS' attending spans (frames, windows) over the same patches. The tower
provides embed(**tensors) -> (hidden, cos, sin), encode(hidden, cos, sin) and
project(encoded, **tensors). The replay's input buffers are in the node's autocast
dtype unless BLOCK_LOOP_DTYPE names what embed returns (Muse Glimmer's ends on a
LayerNorm, which autocast returns in fp32). The runner refuses an embed output its
buffers may round, and widens a narrower one, so a declaration wider than embed's
output replays the loop in more precision than the eager loop runs. A step whose spans
under some label exceed the ragged resource's max_segments_per_step takes no lease and
runs eagerly. On a node serving LoRA adapters a leased step still declares its spans, with
no attention step, so the engine adds the pool's step of them for the bound modules around
the loop (engine_step). Qwen3-Omni's ViT node is one too, off the scaffold: its region also returns
the deepstack taps (its _capture_block_loop), which its own forward_batched reads
off the replay.
A tower builds on the pieces in mstar/model/components/qwen3_vit.py:
bilinear_table_taps (the learned position table resampled onto a patch grid),
VisionRotaryEmbedding (axial 2D rope; keep_fp32 keeps its frequencies fp32
through the node's dtype cast) and VisionBlock, whose attention runs over the packed
patches through the ragged resource under the block's label (Qwen3.5's frames;
Muse Glimmer's windows and frames), q, k and v as one projection a split checkpoint
loads by stacked-param rules. An encoder of another medium need only send its MediaEncoder.output,
a row per pad in prompt order: Higgs-Audio's audio tower and Qwen3-ASR's packed AuT
encoder are plain NodeSubmodule s on the audio walk. That walk is prefill_audio
unless the lone encoder's MediaEncoder.walk names it: Qwen3-ASR's keeps prefill,
the name its configs and metrics read. A model whose graph is not
MultimodalDecoderModel's (Canary-Qwen's LLM node is llm) writes its own walks and
hands its LLM node its MediaEncoder s and text_prompts itself, with
MEDIA_LAYOUT on the prefill step.
Models with a single partition can skip this section. The defaults in Model provide
one partition, named "default", that contains all walks.
Use several partitions when one stage must run asynchronously while another continues to produce data, for example an LLM feeding a vocoder, or a thinker feeding a talker. Override these methods:
get_partition_topology()declares the partitions and the streamingConnectionobjects between them, including achunk_policy_factorysuch asSlidingWindowChunkPolicy(window=..., stride=...).get_partitions()declares eachPartitionDefinition: its name and its walks. Every walk belongs to exactly one partition.- Route cross-partition tensors with
StreamingGraphEdge(next_node=..., name=..., target_partition=...)instead of a plainGraphEdge.
A consuming partition's plan needs nothing about the stream. A step whose walk a stream
feeds, and that runs no Loop but chunk loops, runs once per chunk the stream brings,
and the partition ends when the worker reports the pass that consumed the stream's final
chunk. For a stream from another partition, the conductor tells the consumer
producer_done once the producer partition is done; the data worker ends a stream from
the client (below) itself.
The consumer receives each chunk as its items stacked along a new leading dimension, or as the item itself when the chunk holds one. Don't write into a streamed input in place: a one-item chunk is the stored tensor, which the producer may also have routed to other destinations, and a sliding window's overlap items reappear in the next chunk.
Keep a walk that consumes a stream to one node or one Loop on one node group, and move
later work into a downstream partition fed by its own StreamingGraphEdge. A consumer
whose every input is streamed runs again as soon as the next chunk arrives, and the
runtime and conductor count passes per worker graph, so the conductor refuses at startup
such a consumer sharing its worker graph with another top-level node or Loop, and a
stream-consuming walk split over worker graphs, unless every consumer also waits on a
non-streamed input and all the walk's worker graphs run on the same workers. That input
must arrive once per pass (from the conductor, say); startup cannot check this.
A consumer whose walk should follow the producer's progress, rather than a count fixed at
ingest, is producer-triggered: the producer decides the consumer's walk. Set the same
consumer_walk function on every connection into the consumer, all from one producer
partition. The producer worker calls it for each row of a pass that emits on those edges,
with a ProducerWalkCtx: the walk that row ran (its own, even in a step that mixed
walks), the node that just completed, the iteration each enclosing dynamic loop is on,
and the producer's full CurrentForwardPassInfo (step_metadata and all; the hook
runs on the producer). The row's streamed items run under the walk it returns. The
context is reused across calls, so a hook must not keep it.
- Each streamed item carries its assigned walk, and a chunk never spans two walks, so
these connections need a
FixedChunkPolicy. A client stream cannot setconsumer_walk. - The consumer's worker switches walk before popping a chunk, and only once the partition
has no pass under way. The conductor learns each pass's walk from the worker's
WORKER_GRAPHS_DONE. - A walk the consumer enters on its own, with conductor inputs, takes effect when they arrive, so send them only once the stream has moved the consumer through the walk before it.
Qwen3-Omni's Talker (_talker_walk) is the reference: Thinker prefill passes run
talker_prefill, the first decode iteration runs talker_last_prefill, and later
ones run talker_decode. Its plan lists the three walks. The conductor follows the walk
each pass reports and enters the next step only to send it something, so it feeds
talker_decode its first input once talker_last_prefill has run.
A model that listens while it runs (a full-duplex speech model, say) takes its live input
as a stream the client feeds. Declare it like any other connection, with the reserved
producer CLIENT:
from mstar.streaming.topology import CLIENT, Connection
Connection(from_partition=CLIENT, to_partition="Encoder", edge_name="audio_in",
chunk_policy_factory=lambda: FixedChunkPolicy(chunk_size=1))CLIENT is not a partition: it has no walks and no worker, and it must not appear in
partitions. The node that lists audio_in in its input_names (with
consumes_stream=True) consumes it through a StreamBuffer and the connection's
chunk policy, exactly as it would a stream from another partition. Its partition
self-triggers from the stream, and the conductor sends its walk none of the streamed
input. That node must be
the only one, in any walk, that takes the edge; the conductor checks this at startup, and
that the model implements process_stream_chunk unless each client stream declares
ClientAudio (below). A partition may take several client
streams (audio and video, say): each has its own buffer and its own end.
process_stream_chunk(chunk)turns each append the client sends into stream items, keyed by client edge name.chunkis aClientChunk(mstar.model.base):text;tensorsandinput_metadata, the append's files loaded as for a request (audio_inputsand so on);streams, pertensorslist the client stream each file names (Nonewhere the client named none, so the model decides);model_kwargs, the append's own; andrequest_kwargs, the opening message's. RaiseValueErrorfor an append the model doesn't take (the client gets a 400). An append of text alone, or ofmodel_kwargsalone that ends no stream, that yields no item is refused rather than dropped, but the data worker can't tell which items text sent with files yielded: refuse such text yourself if the model takes none. Keep it stateless; the consuming node keeps its own state, in its resources orrequest_state.- Tensors
process_promptreturns under a client edge's name are the stream's first chunk, so a one-shot request (one uploaded file, then nothing) is a stream of one chunk.process_promptgetsopen_streams, the client streams the opening leaves open; a model whose request settings can't take appends on one (Qwen3-TTS text in a layout that streams none) raisesValueErrorthere, so the live opening gets a 400. - A client stream may declare limits,
Connection(..., limits=ClientStreamLimits(...))(mstar.streaming.topology), which the client-stream layer enforces before any of the model's code runs:max_items, what the stream may carry over a call's life, each item countingitem_size(item)(1 by default; a stream of variable-length chunks counts their frames or samples, named byunit). A call pastmax_itemsends with a 400 that names the limit, so the consumer need not check a session cap of its own. Declare the most the consumer can take: a deployment may lower a declared bound underclient_streams:(see :doc:`serving`), but not raise it. - An audio client stream may declare what its audio is,
Connection(..., audio=ClientAudio(sample_rate, frame_samples)): mono audio atsample_rate, whose consumer computes on frames offrame_samples./v1/realtimethen serves the model as a conversation (:doc:`clients`), when only one of its client streams declares it and its audio output is mono: the route decodes the caller's codec, resamples tosample_rateand turns the model's output back into the caller's format fromget_output_sample_rate. On every route the data worker, the stream's producer, sends it in whole frames: each whole frame of an append goes as an item (an opening clip's frames as one) and the rest waits for the next append, or goes with the stream's end. A model whose client streams all declareClientAudioneeds noprocess_stream_chunk: the data worker takes appended audio files atsample_rate(one at another rate is refused: resampling an append alone leaves a seam), refuses non-finite samples and clamps to [-1, 1]. Nemotron VoiceChat declaresClientAudio(16000, 1280): its encoder's 80 ms frame, taken from the encoder's own hop and subsampling. - A model that declares
ClientAudiodecides when to speak (full duplex). To serve/v1/realtimeconversations it also declaresturn_tokens = (start_id, end_id)on the model (a property when the ids come from its config): the text tokens that open and close each reply, and, if its replies may get no end mark,turn_idle_steps: a reply then also ends after that many steps with no text and no audio aboveturn_silence_rms(the RMS of samples in [-1, 1], 0.005 unless the model sets it), and text outside a reply opens one. Without them/generate/wsstill serves it, and a conversation is refused (an omittedintentthen means transcription, if the model's adapter serves it). The data worker marks the text chunks they arrive in withmetadata["turn"], their marks ("start"/"end") in token order, on every route, beforepostprocess(which may decode them to nothing), and stamps each text and audio chunk withmetadata["step"], the iteration of the loop that emitted it./v1/realtimeframes one response per reply from the marks and puts stepk's audio in the reply whose text steps holdk, so the text and the audio node must each emit from a loop that steps once per input item. Startup refuses one that emits outside a loop, or from a partition not stepped one item at a time on every connection from theClientAudiostream;FixedChunkPolicy(chunk_size=1)is the only policy the check takes as one. Turns as a pair of token ids fit a model that brackets each reply (Nemotron's BOS / EOS); one that marks speech another way, such as Moshi-style PAD / EPAD runs, can't declare them yet, and serves/generate/wsonly. - A stream ends when the client ends it:
end_input: trueends all of the call's streams, a list of names only those. The consumer getsproducer_donefor that edge and a final chunk, and a loop fed by the stream ends as any stream-terminated loop does. Over/generate/wsa client that opens withend_input: falseappends and ends input explicitly (see :doc:`clients`), and a/v1/realtimeconversation keeps it open for the call's life; every other request ends the input as soon as it is submitted. Both frontends serve/generate/wsand/v1/realtime. - The API server's data worker is the stream's producer. The conductor tells it the
consuming node and its ranks once per call, when it dispatches the request, and the data
worker sends each append's items to every one of those ranks itself, and then the
stream's end with how many items it carried. Every rank gets each item whole, so a client
stream can't take a
sharding_config.shard_dimentry (startup refuses one). Once the consuming partition ends, later appends for its streams are answered as late. - A consumer with nothing to do on the step that ends its last stream
(
is_final_stream_chunk; empty when the last item was taken before the end arrived) returns None fromprepare_inputs. The row still completes its node, so its loop ends and its partition reports done. None on any other chunk drops the row from the step. Every rank sees the same chunk, so all decline alike.
[ ] mstar/model/<your_model>/config.py — config dataclass + declared names
[ ] mstar/model/<your_model>/components/ — the nn.Modules + weight loading
[ ] layers take their layer id and call their handles (bind_state)
[ ] mstar/model/<your_model>/submodules.py — NodeSubmodule per node
[ ] prepare_inputs (set input_seq_len)
[ ] declare_step
[ ] preprocess / forward (+ forward_batched, can_batch)
[ ] mstar/model/<your_model>/<your_model>_model.py — Model subclass:
[ ] declare_state
[ ] get_graph_walk_graphs
[ ] process_prompt
[ ] walk_plan
[ ] postprocess
[ ] get_submodule
[ ] (optional) get_request_resource_configs
[ ] kv_labels, state_layout from process_prompt (+ get_loop_caps) — with an append-only array
[ ] (optional) keys on a KVLabel + checkpoint_path — cross-request prefix reuse
[ ] mstar/model/registry.py — (module_path, class_name) in MODEL_REGISTRY
[ ] HF_MODELS too, if weights come from Hugging Face
[ ] configs/<your_model>.yaml — node_groups → ranks (+ resources: overrides)
[ ] mstar/cli/main.py DEFAULT_CONFIGS — else `mstar serve <name>` says "unknown model"
[ ] the client-example tuples lower down (cosmetic, user-visible)
[ ] docs/models.rst — one row in the registry-key table
[ ] (optional) mstar/api_server/openai/adapters.py — only for OpenAI-compatible routes
[ ] and rust/server/src/adapters/ — the same adapter in the Rust frontend
[ ] (optional) pyproject.toml extra — if it needs deps the base install lacks
[ ] (optional) async partitions if pipelined
Validate the graph and the worker integration before you use real weights. The modular
tests run on CPU and exercise models in dummy mode, where get_submodule returns
None:
ruff check .
pytest test/modular/ # CPU graph/worker tests
pytest test/integration/ # requires GPU + weights
mstar-serve --config configs/your_model.yaml --port 8000These modular tests are useful while writing a new model. They cover the parts that are easy to get wrong, and they run on CPU:
| Test | Covers |
|---|---|
test/modular/test_resource_runner.py |
The admit, plan and commit lifecycle driven by declare_step. |
test/modular/vjepa2/fake_resources.py |
A stub array handle for running a submodule's layers without a GPU. |
test/modular/test_cuda_graph_capture.py |
Capture-bucket keys, including cg_key_info and additional_key_info. |
test/modular/test_micro_scheduler.py |
Batch admission, and the paths for failed and backpressured requests. |
test/modular/test_admit_failure_handling.py |
Behavior when a resource cannot admit a step. |
test/modular/test_kv_offload.py |
Offload and reload of a request's cache state. |
Then send a POST /generate request and check the streamed output.
The fastest way to add a new model is to base it on the existing model that is closest to it. Use Orpheus for a streaming LLM with a codec, Whisper for encoder-decoder cross-attention, BAGEL for a unified understanding and generation model, and Qwen3-Omni for a full omni-modal model.