Summary
A public MoE router precision audit compared AutoModel's effective router behavior with the pinned Hugging Face or checkpoint-owned reference implementations across 188 training, finetune, and benchmark YAMLs.
The audit checks four independent stages:
- Param: stored router projection parameter dtype after checkpoint loading.
- Proj: router linear/matmul arithmetic dtype.
- Score: sigmoid/softmax, correction bias, normalization, and top-k arithmetic dtype.
- Out: selected routing-weight dtype passed to expert computation.
At the audited head, 9 router families matched all four stages, 23 were mismatched or recipe-dependent, and Inkling could not be independently verified against public reference code. These are correctness-sensitive differences: projection or score precision can change top-k expert identity, while parameter/output dtype mismatches can change checkpoint fidelity or expert accumulation.
This issue tracks model-owned fixes and validation. It should not be solved by applying one global FP32 policy: several reference routers intentionally use BF16 for projection, scoring, or returned weights.
Reference and load-convention policy
Checkpoint tensor dtype, reference model code, and observed runtime behavior answer different questions and must be recorded separately. A Hugging Face model loaded with an explicit model-wide BF16 dtype can legitimately downcast an F32 checkpoint outlier; that loaded model is evidence for that named harness convention, not proof that the checkpoint intended the parameter to be BF16.
Production serving implementations such as vLLM, SGLang, and TensorRT-LLM are useful independent references when they have a model-specific implementation, but they are not universal precision authorities either. Record the exact runtime/version, load options, and observed Param / Proj / Score / Out chain instead of inferring per-parameter behavior from a global dtype=auto setting.
For MiniMax M2 specifically, vLLM's native implementation at 6a9c69f explicitly allocates the gate parameter in FP32, requests FP32 gate output/router logits, and keeps the correction bias in FP32. This independently corroborates the released checkpoint's F32 gate storage and the checkpoint-faithful FP32 router target. This is a model-specific override: vLLM's general dtype=auto policy otherwise selects a model-wide weight/activation dtype.
Active work
Any shared sigmoid-scoring default changed by the MiniMax work must be rechecked against BF16-score references such as the DeepSeek V4 learned router. Prefer a model-owned default or an explicit typed control when router families intentionally differ.
Remaining families
Likely small model-owned policy changes using existing router controls
F32 checkpoint router storage
The existing strict-FP32 parameter/buffer retention machinery should make these focused changes, but each still needs a checkpoint-load and model-cast test proving the gate remains F32.
Model-specific BF16/de-promotion or routing-semantic work
These are not simple “turn on FP32” fixes. AutoModel currently over-promotes at least one stage or has a router variant whose arithmetic differs from the reference. They may require a model-owned score/projection policy or a narrowly typed shared control.
Reference required
Acceptance criteria per family
- Identify an immutable reference implementation and, when relevant, checkpoint tensor dtype evidence.
- State every reference load convention explicitly; do not treat a framework loaded with a model-wide dtype override as a checkpoint-storage oracle.
- Where a model-specific production implementation exists, cross-check a pinned serving runtime and record its actual loaded parameter and routing arithmetic dtypes.
- Match Param / Proj / Score / Out independently; do not infer full correctness from FP32 logits alone.
- Keep model-specific defaults in the owning model package. Preserve explicit caller overrides where supported.
- Add focused tests for stored gate dtype, projection/scoring arithmetic dtype, selected-weight dtype, and default/explicit-override behavior.
- Run a tiny numerical router/component comparison against the reference, followed by the relevant end-to-end checkpoint/parity test.
- Update this checklist and link the fixing PR plus exact-head validation evidence.
Already exact in the audit
No action is currently required for Qwen2 MoE fallback, ERNIE 4.5 MoE, Qwen3 MoE, Qwen3 Next, Qwen3.5-family MoE, Qwen3 Omni MoE, Qwen3 VL MoE, Kimi K3, or Kimi Linear. Re-audit them if shared router behavior changes.
Summary
A public MoE router precision audit compared AutoModel's effective router behavior with the pinned Hugging Face or checkpoint-owned reference implementations across 188 training, finetune, and benchmark YAMLs.
The audit checks four independent stages:
At the audited head, 9 router families matched all four stages, 23 were mismatched or recipe-dependent, and Inkling could not be independently verified against public reference code. These are correctness-sensitive differences: projection or score precision can change top-k expert identity, while parameter/output dtype mismatches can change checkpoint fidelity or expert accumulation.
This issue tracks model-owned fixes and validation. It should not be solved by applying one global FP32 policy: several reference routers intentionally use BF16 for projection, scoring, or returned weights.
Reference and load-convention policy
Checkpoint tensor dtype, reference model code, and observed runtime behavior answer different questions and must be recorded separately. A Hugging Face model loaded with an explicit model-wide BF16 dtype can legitimately downcast an F32 checkpoint outlier; that loaded model is evidence for that named harness convention, not proof that the checkpoint intended the parameter to be BF16.
Production serving implementations such as vLLM, SGLang, and TensorRT-LLM are useful independent references when they have a model-specific implementation, but they are not universal precision authorities either. Record the exact runtime/version, load options, and observed Param / Proj / Score / Out chain instead of inferring per-parameter behavior from a global
dtype=autosetting.For MiniMax M2 specifically, vLLM's native implementation at
6a9c69fexplicitly allocates the gate parameter in FP32, requests FP32 gate output/router logits, and keeps the correction bias in FP32. This independently corroborates the released checkpoint's F32 gate storage and the checkpoint-faithful FP32 router target. This is a model-specific override: vLLM's generaldtype=autopolicy otherwise selects a model-wide weight/activation dtype.Active work
ForCausalLM/recipe construction.Any shared sigmoid-scoring default changed by the MiniMax work must be rechecked against BF16-score references such as the DeepSeek V4 learned router. Prefer a model-owned default or an explicit typed control when router families intentionally differ.
Remaining families
Likely small model-owned policy changes using existing router controls
ForCausalLMconstructor, so its normal construction path must also install the model-owned FP32 projection/scoring default while preserving an explicit caller override.need_fp32_gate=true.F32 checkpoint router storage
The existing strict-FP32 parameter/buffer retention machinery should make these focused changes, but each still needs a checkpoint-load and model-cast test proving the gate remains F32.
Model-specific BF16/de-promotion or routing-semantic work
These are not simple “turn on FP32” fixes. AutoModel currently over-promotes at least one stage or has a router variant whose arithmetic differs from the reference. They may require a model-owned score/projection policy or a narrowly typed shared control.
Reference required
Acceptance criteria per family
Already exact in the audit
No action is currently required for Qwen2 MoE fallback, ERNIE 4.5 MoE, Qwen3 MoE, Qwen3 Next, Qwen3.5-family MoE, Qwen3 Omni MoE, Qwen3 VL MoE, Kimi K3, or Kimi Linear. Re-audit them if shared router behavior changes.