Skip to content

Latest commit

 

History

History
428 lines (403 loc) · 20.1 KB

environment_variables.rst

File metadata and controls

428 lines (403 loc) · 20.1 KB

Environment variables

Runtime knobs M* reads from the environment. New variables should be documented here as they are introduced.

Communication

Variable Default Meaning
MSTAR_RUST_ZMQ AUTO Transport selection for the ZeroMQ control mesh (see :func:`mstar.communication.communicator.make_communicator`). AUTO: the Rust-backed RustZMQCommunicator when the vendored rust/ extension imports successfully, pyzmq otherwise. 1: the Rust communicator, raising if the extension is missing. 0: always pyzmq. The two transports are wire-compatible, so this can be set per-process while the rest of the mesh stays on pyzmq. Under MSTAR_RUST_GRAPH=AUTO, 0 also keeps that worker's graph runtime on Python -- the Rust one cannot share a pyzmq transport.
MSTAR_RUST_GRAPH AUTO Which graph runtime a worker builds (see :func:`mstar.graph.runtime.utils.resolve_graph_runtime_type`). AUTO: Rust where the vendored rust/ extension imports and the Rust transport is in use, Python otherwise. 1: the Rust runtime, failing at startup if the transport or the tensor bookkeeper is a Python one -- it holds a share of both, so there is no degraded mode to fall back to. 0: always the Python runtime. Resolved once per process, because it also selects the tensor bookkeeper the runtime and the tensor store must share.
MSTAR_WIRE_CODEC msgpack How messages are encoded on every communicator edge (see :func:`mstar.communication.codec.default_codec`). msgpack: the typed encoding in :mod:`mstar.communication.wire`, language-neutral so an edge can terminate in a Rust process. pickle is no longer supported and fails at startup. Loosely typed metadata uses data-only extensions instead of pickle. Upgrade all processes in a deployment together: older processes cannot decode the new extensions, and updated processes reject legacy pickle messages and metadata.
MSTAR_ZMQ_TRANSPORT constructor's protocol Overrides the communicator protocol (IPC or TCP) for a process, e.g. to run entities on separate hosts.
MSTAR_ZMQ_TCP_HOST 127.0.0.1 Host used to build peer endpoints when the protocol is TCP.
MSTAR_ZMQ_TCP_BASE_PORT 19000 Base of the deterministic entity-id → TCP port map (api_server = base, conductor = base+1, worker_<rank> = base+100+rank).
MSTAR_REQUIRE_CUDA_GRAPHS 0 1 makes a worker fail at startup when any CUDA graph bucket could not be captured, instead of serving that bucket eagerly at 10-20x the latency. Off by default: a failed capture is logged at ERROR with a per-runner summary and the rest keeps running.
MSTAR_SHM_ARENA 0 SHM tensor-transport implementation. 0: per-uuid files. 1: the Rust shared-memory arena (requires the rust/ extension; raises if missing). AUTO: the arena when the extension imports, files otherwise. Must match across the deployment — arena locations ride in the tensor descriptors.
MSTAR_KV_SHM_DIR /dev/shm (or /tmp if unavailable) Base directory for host-staged KV snapshots when remote KV transfer uses SHM. Each deployment creates a private child directory here; request cleanup removes snapshots, and startup removes directories left by dead deployments. All workers sharing KV snapshots must see the same filesystem.
MSTAR_SHM_ARENA_SEGMENT_MB 256 Size of each arena segment. The arena grows segment by segment; existing segments never move (registrations stay valid).
MSTAR_SHM_ARENA_MAX_SEGMENTS 32 Growth cap PER ENTITY. Every entity (workers + the api-server data worker) creates its own arena, so node-wide /dev/shm demand can reach MAX_SEGMENTS x SEGMENT_MB x num_entities — size against df -h /dev/shm (tmpfs defaults to ~50% of RAM). Construction fails fast if one entity's ceiling exceeds /dev/shm. At the cap, sends spill (see MSTAR_SHM_ARENA_SPILL).
MSTAR_SHM_ARENA_FULL_TIMEOUT_S 30 Strict mode only (MSTAR_SHM_ARENA_SPILL=0): how long a send backpressures on a full arena before failing.
MSTAR_SHM_ARENA_SPILL 1 Degrade gracefully at the segment cap: stage the tensor through the per-uuid file protocol instead — slower, never fails, matching the file transport's saturation behavior. 0 restores strict backpressure + timeout — only meaningful where ANOTHER thread drains consumer ACKs (the threaded api-server); on a worker the ACKs arrive on the very thread that would be waiting.
MSTAR_SHM_ARENA_SPILL_AFTER_S 0 Optional grace before spilling, for deployments where another thread frees slots concurrently. Default 0: spill immediately (a worker cannot receive ACKs while it waits).
MSTAR_SHM_ARENA_PIN 1 cudaHostRegister each mapped segment (both sides) so D2H/H2D copies through the side streams run at page-locked bandwidth and stay asynchronous. 0 disables (pageable copies).
MSTAR_SHM_ARENA_PIN_MAX_MB 4096 Budget for TOTAL pinned host memory PER PROCESS, distinct from the segment cap (pinned pages come out of the OS's pageable pool system-wide). Node-wide pinned demand is approx PIN_MAX_MB x num_entities — a consumer pins peer segments too, so one process can pin more than its own arena holds. Segments past the budget stay unpinned: copies work, without async overlap.
MSTAR_SHM_ARENA_SLOT_TTL_S 0 TTL backstop for abort-orphaned slots (a request aborted after staging but before all consumer ACKs defers reclaim forever). A slot older than the request timeout cannot have a legitimate reader, so a bound safely above it (recommend >= 2x the request timeout) cannot race a real consumer. 0 disables (default, pending review discussion); reclaims run under capacity pressure and with the periodic stats sweep, logging loudly.
MSTAR_SHM_ARENA_STATS_INTERVAL_S 60 Under --log-stats: how often the arena logs its occupancy / fragmentation snapshot (segments, free bytes, largest contiguous free block, pinned bytes).

Models

Per-model knobs, read when the submodule is built. They are scoped to one model, so they are named for it.

Variable Default Meaning
MSTAR_VIT_BATCHING 0 BAGEL: batch several requests through the ViT encoder in one forward. Off by default because flash-attn's varlen reductions across packed images produce small bf16 drift, which at greedy temperature=0 can flip a downstream LLM argmax. When off, prefill_vit runs one request at a time.
MSTAR_VIT_CUDA_GRAPH 0 BAGEL: CUDA-graph capture of the ViT block loop, over the node's ragged attention resource (see :doc:`adding_models`). Off by default: capture costs one graph per (batch size, token bucket) and the eager flash-attn path is already fast. The win is removing per-layer launch overhead on small images.
MSTAR_VIT_CG_TOKEN_BUCKETS 512,1024,2048,4096,4900 Comma-separated token-count buckets to capture, when MSTAR_VIT_CUDA_GRAPH=1. A batch longer than the largest bucket runs eagerly. 4900 is 70*70, the exact length the vllm preprocess option emits.
MSTAR_VIT_CG_BATCH_SIZES 1,2,4 Comma-separated batch sizes to capture. Read only when MSTAR_VIT_BATCHING=1; otherwise only batch size 1 is captured.
MSTAR_OMNIVOICE_MODEL_PATH unset OmniVoice: load weights from this local directory instead of the checkpoint the registry names. For serving a fine-tune of the same architecture off a mounted volume, where the registry entry would otherwise pull the public weights from the Hub.

Serving (Python frontend)

Read by mstar-serve / mstar.api_server.entrypoint.

Variable Default Meaning
MSTAR_ROOT_PATH "" Mount the app under a URL sub-path. Set it when an ingress routes the deployment at a prefix and does not strip that prefix before the request reaches the pod (Run:AI serves a workload at /<project>/<job-name>/, for instance); without it FastAPI 404s every route. Empty by default, so a direct deployment is unaffected.

Serving (Rust frontend)

Read by the mstar-server binary and its bridge (mstar-serve --rust-frontend; see :doc:`installation`).

Variable Default Meaning
MSTAR_SERVER_BIN unset Path to the mstar-server binary. Fallback order: --rust-frontend-bin, this variable, $PATH, then the in-repo rust/server/target/release build.
MSTAR_REQUEST_TIMEOUT_S 600 Per-request budget in the Rust frontend; on expiry the client gets an error and the request is aborted in the backend.
MSTAR_SAMPLE_RATE 24000 Sample rate stamped on /v1/audio/speech WAV output.
MSTAR_ALLOW_REMOTE 0 Allow http(s) media URLs in requests (fetched server-side, 30 s timeout). Off by default.
MSTAR_MAX_CONCURRENT_REQUESTS 256 Admission cap on in-flight generation requests; beyond it clients get an immediate 503 instead of queueing into the request timeout. /health and /v1/models bypass the cap.
MSTAR_MAX_BODY_MB 128 Request body limit (multipart uploads included).
MSTAR_TOKENIZE_PARALLEL 1 When a long prompt is encoded as parallel chunks, set TOKENIZERS_PARALLELISM=true for the process first: the Rust tokenizer only uses its thread pool while that variable reads true, and serving environments often export it false. 0 leaves the variable as it is (the chunked encode then runs serially).
MSTAR_TOKENIZER unset Path to a HuggingFace tokenizer.json enabling frontend tokenization. Leave unset with the Python backend — its preprocess worker owns tokenization, and the bridge rejects pre-tokenized ingest.

Rust frontend limitations

The Rust frontend carries no audio encoder or media muxer (the Python frontend uses soundfile / ffmpeg), so two surfaces degrade:

  • /v1/audio/speech produces only wav and pcm. Any other response_format (mp3, opus, flac, …) is rejected with a 400 rather than silently returned as WAV. Run the Python frontend for compressed containers.
  • /v1/videos/generations (Cosmos3) returns the video-only mp4. When a request sets generate_sound, Cosmos3 generates a separate audio track that mstar muxes into the mp4 as an AAC track; the Rust frontend cannot mux it, so the audio is dropped and a warning is logged. Do not set generate_sound on the Rust frontend (it spends compute on a track the client won't receive), or run the Python frontend for sound video.

Worker scheduling

Variable Default Meaning
MSTAR_TP_ASYNC_SCHED 0 Async scheduling for lockstep-parallel (TP / SP) nodes. 1: the instance leader speculates step N+1 of the parallel node during forward N (the existing single-worker speculation machinery, gate opened) and broadcasts it at once as a speculative ScheduleTPNode; followers rebuild the identical batch during their own forward N and every rank submits N+1 the moment N completes. Removes the per-step serial build/plan from the group's critical path. Voids (allocation failure, per-rid failure, a continuing request without loop-back output) are derived by each rank from replicated state, never signalled. A comma-separated list of node names (thinker,talker) enables it for those parallel nodes only. 0: the serial path — leader schedules after N, followers rebuild after the broadcast. Set it identically on every rank of an instance: the workers compare it at startup and refuse to start on a mismatch. Leave MSTAR_ENGINE_STEP_SYNC at 0 with it: that throttle holds the GPU thread until step N drains, which serialises the very overlap this flag buys.
MSTAR_PRE_PLAN_SPEC 1 Pre-plan the speculative batch's attention on a dedicated thread while the previous replay runs. 0 plans inline on the GPU thread.
MSTAR_DECODE_BUCKETS 1,2,4,8,16,32,48,64,96,128 The decode batch sizes a model captures as CUDA graphs (Qwen3.5 reads it; a model with its own ladder reads it the same way). A decode step runs at the smallest bucket that holds its rows, so the ladder bounds both the rows per step and the graph memory; narrow it (for example 1,2,4,8,16,32) on a memory-tight GPU or for an A/B against the former ladder. MSTAR_QWEN35_DECODE_BUCKETS overrides it for Qwen3.5 alone.
MSTAR_LAZY_PUBLISH 1 Publish resource state (KV pages, positions) only for the requests whose frame or completion carries it: finalize_batch keeps a cheap per-request snapshot and the worker finishes it on demand. 0 builds the full publication for every request on every step, the old behaviour; it was a quarter of the GPU thread's host time at batch 32 in a decode loop that never reads it.
MSTAR_MAX_CONSECUTIVE_SPEC_STEPS 1024 Cap on back-to-back speculative steps before the leader yields to other ready work.
MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS 32,64,128,256,512,1024,2048,4096,8192 Total tokens of a packed Qwen3.5 text-prefill step that get a captured graph (one per bucket and batch size). A prompt past the top bucket prefills eagerly, which is launch-bound: an eager 8k prefill on 0.8B spends 78 ms of wall time on 36 ms of kernels. The default now reaches 8192; narrow it to save graph memory and capture time, or extend it.
MSTAR_KV_CHAIN_LAZY_STEPS 1 How many decode steps of sampled ids the KV manager holds back before it extends the requests' prefix chains (the per-page keys the prefix cache is indexed by), one extend per request per batch of steps instead of one per step. The per-request Python of that extension was 0.3 to 0.4 ms a step at 128 rows. A page key then appears up to that many steps late and a later commit indexes it, as the single-request path already allows; a request that finishes in between may leave its last page unindexed. The default 1 extends on every step: 16 measured as noise at c64/c128 on 0.8B (the extension is a small part of the stop check's tail).
MSTAR_KV_PINNED_INDPTRS 0 The KV plan writes FlashInfer's index arrays (qo_indptr, kv_indptr, kv_indices, last_page_len) into a ring of page-locked host buffers and the decode wrapper copies the indices straight into its captured device buffer from there, so the copies FlashInfer's plan issues are really asynchronous. From pageable memory the driver stages each one and FlashInfer makes the indices copy blocking. Measured neutral to slightly negative at c32-c128 on 0.8B (the event bookkeeping per take costs about what it saves), so it is off by default.
MSTAR_KV_PLAN_CACHE 0 The KV manager plans a captured decode step off the previous step's plan when the batch is the same rows in the same order and nothing but that step's commit touched the streams: every length is one more, pages change only where a row crossed a page boundary, and the five FlashInfer index arrays come from numpy instead of a per-row Python pass. The first version measured a loss at c64/c128 on 0.8B (batches change every step or two there, and a hit still rebuilt the views); the reworked path keeps the views lazy and is off by default until measured. 0 plans every step from the streams. 2 runs both and logs a difference, for checking a run.
MSTAR_DEVICE_LOOPBACK 1 Whether a decode node may keep its loop-back token on the device: the sampler writes each request's last token to a slot-addressed master and the next step reads its input ids from it, so the worker routes the signal with no tensor (no per-request store, hold or cleanup per step). Only nodes that declare the read (reads_device_loopback: Qwen3.5 decode) and only when the sampler has graph buffers; the sampler keeps the masters for those nodes alone, so a node that takes its tokens on the host (Whisper's decoder) no longer pays the eager-step master write, a small synchronous host-to-device copy per step. 0 keeps every node on the per-request tensor path.
MSTAR_INGRAPH_DECODE_ROPE 0 A captured decode step builds its rotary tables inside the graph from the position resource's planned positions instead of staging them. Experiment knob, measured and gated on its own.
MSTAR_SAMPLER_INGRAPH_SCATTER 0 The captured sampler gathers its RNG offsets from the slot masters and scatters the advanced offsets and the sampled tokens back inside the graph, so a replayed decode step issues none of those launches on the gpu thread afterwards (each one is a GIL release the main thread's postprocess wins). Padding rows then address a trash master row and the masters are allocated once for MSTAR_SAMPLER_SLOTS concurrent requests (default 4096, a hard cap with this on). Only with the device loop-back on. An experiment knob; measure before turning it on.
MSTAR_RUST_SEND_TIMING unset 1 makes the Rust runtime print, every 500 calls, how long the worker's per-step send spends in its own work against the whole call (which includes taking the GIL back when the send had to release it). A diagnostic for the host-overhead work; off by default.
MSTAR_SPEC_PEEK_FOR_FAIRNESS 1 Yield the speculation chain only when another (node, walk) is actually ready right now; 0 uses the consecutive-step cap alone.
MSTAR_PHASE_TIMING 0 N > 0: every N iterations log per-phase p50/p95/mean of the worker main loop (speculate, await_gpu, submit_spec, ...).
MSTAR_KV_DEBUG_ASSERTS 0 1: after every admit, commit, reset_request and remove_request, check the KV page bookkeeping (free list, owner counts, seals) against the streams holding the pages. Walks every live stream; tests and debugging only.

Compilation

Variable Default Meaning
MSTAR_RECOMPILE_LIMIT 84 Dynamo's recompile limit, clamped to [8, 256]. Raise it if the logs show hit config.recompile_limit.