[MakazhanAlpamys]

Summary

main has been red since #1250 merged: test (windows-latest, 3.10) and test (windows-latest, 3.12)
in run 36251313755 (head 0334b975) die with Windows fatal exception: code 0xc000001d
(STATUS_ILLEGAL_INSTRUCTION) in the first test of tests/test_issue1235_prm_fp32_master_weights.py
that runs a real CPU bf16-autocast step. The crash kills the interpreter, so each cell reports nothing
about the 22,433 of its 25,318 selected tests it had not reached.

This keeps the tests running on every cell instead of skipping them. A new conftest fixture,
aten_half_matmuls, moves the step's half-precision matrix multiplies off the two libraries that pick
their instruction set at run time:

• torch.backends.mkldnn.enabled = False, so bf16/fp16 matmuls run in ATen's own precompiled gemm
  instead of oneDNN's JIT kernels;
• SDPA on its math backend, which upcasts q/k/v to fp32, so attention no longer calls MKL's
  cblas_gemm_bf16bf16f32 / cblas_gemm_f16f16f32.

The dtypes #1235 pins (fp32 master weights, the reward head taking the forward's dtype, the steps moving
the norms) do not depend on which library multiplies the matrices, and autocast stays on.

This cannot be proven before merge. Which runner a Windows job lands on is a lottery: #1250's own
run passed on all three Windows cells, so a green run on this PR proves nothing either way. What
supports the fix is the elimination below. If 0xc000001d comes back in these tests with the fixture
on, the fallback is the #382 skip (skip_on_windows_ci) on these tests, with
tests/test_issue382_windows_ci_guard_is_shared.py updated to match.

Test-only change: no changelog fragment, and no src/ change.

Root cause: evidence

The crash. Both cells die in
TestBf16MixedPrecisionTrains::test_one_step_runs_and_moves_the_norms_and_projections[docs-lr], right
after the eight TestTheLoadDtype tests (setup only, no forward) pass. The faulthandler frame is the
first bf16 matmul of the step:

torch/nn/modules/linear.py:134 in forward
transformers/models/llama/modeling_llama.py:254 in forward   # self.q_proj(hidden_states), layer 0
...
torch/amp/autocast_mode.py:44 in decorate_autocast
accelerate/utils/operations.py:943 in forward               # ConvertOutputsToFp32
soup_cli/trainer/prm.py:67 in compute_loss
transformers/trainer.py:2014 in training_step
soup_cli/trainer/prm.py:371 in train
tests/test_issue1235_prm_fp32_master_weights.py:276

No bitsandbytes and no Soup native code are on the stack (Soup has none). Stack: torch 2.14.0 (the CPU
wheel from PyPI), transformers 5.17.0, accelerate 1.15.0, image windows-2025-vs2026 20260922.246.2.

Which hosts. The fault depends on the host, not the code or the interpreter:

run	tree	windows 3.10	windows 3.11	windows 3.12
36051967682	#1250 at dc042a4e	ran the steps, no fault*	ran the steps, no fault*	pass
36103050921	#1250 at ca14f096	pass	pass	pass
36251313755	main 0334b975	0xc000001d	pass	0xc000001d

* Both failed an == 0.0 assertion that #1250 then relaxed to <= 0.01: 1 of 320 norm elements
unmoved. Both logged the same loss and grad-norm sequence, and it matches neither ATen path on the dev
box (AVX2: 0.5018 / 47.63 at step 2; forced DEFAULT: 0.9504 / 69.96 at step 4, against their
0.5025 / 47.65 and 0.9517 / 70.08). So those hosts ran run-time-dispatched half-precision kernels,
without a fault.

The ubuntu and macos cells have never crashed on these tests, or on the other CPU bf16 steps below.

Which code. On x86, torch 2.14 routes a CPU bf16 matmul as follows (source at v2.14.0):

• cpublas::gemm (bf16 in, bf16 out) calls mkldnn_bf16_gemm first
  (aten/src/ATen/native/CPUBlas.cpp). mkldnn_gemm takes it when
  userEnabledMkldnn() && mkldnn_bf16_device_check() and m*n*k > 16**3
  (aten/src/ATen/native/mkldnn/Matmul.cpp). The device check is ideep::has_bf16_type_support(),
  which is true on AVX-512 and AVX2-VNNI-2 hosts. Everything else falls to ATen's gemm_stub
  (native/cpu/BlasKernel.cpp: templated C++, no BF16 intrinsics, compiled per ATen capability).
• The fused CPU SDPA kernel multiplies bf16/fp16 blocks through the mixed-output cpublas::gemm, which
  goes to MKL's cblas_gemm_bf16bf16f32 / cblas_gemm_f16f16f32 when MKL has them. The oneDNN flag
  does not gate that route.

oneDNN and MKL both pick the instructions they execute at run time from their own CPU probe. The
precompiled ATen kernels are selected once, through cpuinfo, from the OS-enabled feature set. So the
question is which of these the crashing hosts could not run:

• fp32 training runs there. On the two 2026-09-17 cells that died the same way (runs 35228388681 and
  35254266989), test_issue341_seed_and_fullft.py::TestFullFinetuneTrainer (a real trainer.train(),
  and Soup's CPU flags are bf16=False, fp16=False) passed at 12%. On today's cells the real fp32
  GRPO steps of test_issue1232_grpo_variant_kl.py::TestTheRealTrainer passed right before the crash.
  So MKL's SGEMM runs there.
• bf16 element-wise ATen kernels run there. On the 2026-09-17 cells,
  test_issue722_chunked_distill.py's bfloat16 cases (log-softmax/KL, no matmul) passed at 20%.
• The first CPU bf16 matmul dies. On 2026-09-17 that was
  test_rewind_hf.py::test_recorded_rows_match_sampler_order at 41%, the first trainer.train() under
  trl's SFTConfig. trl 0.29.1 defaults bf16=True even with use_cpu=True; this was checked, and
  GRPOConfig does the same, which covers the 2026-08-31 GRPO crash in test_v05311.py as well.
  Today it is PRM on CUDA trains bf16 weights with no fp32 master copy: the first bf16 step fails with a dtype mismatch, and with that patched the base updates round to zero #1235's first bf16 step.

What is left is the run-time-dispatched half-precision matmul code. Which instruction faults cannot be
read off these logs: the job logs do not name the CPU model, and a 0xc000001d dump carries no
registers. oneDNN 3.12 (bundled with torch 2.14) does gate AMX on Windows by XCR0[18:17]. A oneDNN
AMX matmul has produced exactly this signature before (pytorch/pytorch#99509). An unrelated project hit
the same thing on 2026-09-25: nn.Linear under CPU bf16 autocast, 0xC000001D, on some windows-latest
hosts, gone on a rerun (maged15/beamgrad#12). The fix below does not depend on which instruction it is.

Why this cannot be reproduced on the dev box, and what the fix removes. The dev box is an
i9-14900HX (AVX2 + AVX-VNNI, no AVX-512). There torch.ops.mkldnn._is_mkldnn_bf16_supported() is
False, so bf16 matmuls never reach oneDNN. ONEDNN_VERBOSE=1 logs 0 primitives across the six
bf16/fp16 tests. The positive control works: an fp32 conv2d logs primitives, and they stop inside
mkldnn.flags(enabled=False). MKL_VERBOSE=1 across the same six tests:

	GEMM_BF16BF16F32	GEMM_F16F16F32	SGEMM	SGEMM_BATCH	oneDNN primitives
main	768	384	333	0	0 (no AVX-512 here)
this PR	0	0	109	264	0

With the fixture, the step's half-precision matmuls run in ATen's gemm_stub, and attention runs as
fp32 SGEMM, the MKL path the crashing hosts already run without fault.

Why this option

• Not the CI: windows-latest cells die with 0xc000001d in the NF4 training-step test — three occurrences, now on main #382 skip. skip_on_windows_ci would stop the crash for certain, but it would drop these
  five cases on every Windows host, including the ones that run them fine (windows 3.11 on the same
  main run, and all three on fix(prm): train PRM on fp32 master weights on every device (#1235) #1250's run).
• Not an ISA cap in the workflow. ONEDNN_MAX_CPU_ISA and MKL_ENABLE_INSTRUCTIONS are
  process-wide and read once when the library initialises, so they cannot be scoped to one test. They
  would change every test on the Windows cells, and choosing the cap needs the faulting ISA, which the
  logs do not show.
• The fixture is scoped to the tests that request it, on every OS, so the kernel choice for these
  tests is the same on all nine cells. It uses monkeypatch plus the sdpa_kernel context manager, so
  both switches are restored after each test.

Changes

• tests/conftest.py: the aten_half_matmuls fixture, with the evidence above in its docstring.
• tests/test_issue1235_prm_fp32_master_weights.py:
  • precision requests the fixture, which covers the four bf16 cases and the fp16-flag case.
  • The fp16 autocast GradScaler test requests it too. That test was never reached on a crashing host,
    and fp16 goes through the same two routes.
  • Each of these tests asserts that both switches were in effect during the step.
  • New TestTheAtenMatmulSwitchStaysScoped, placed last in the file, asserts that both switches are
    back on outside the fixture.
• tests/_windows_ci.py: the module docstring points at the fixture as the first thing to try when
  the crash frame is a CPU bf16/fp16 matmul.

Verification (dev box, PYTHONPATH=<worktree>/src, CUDA_VISIBLE_DEVICES=-1)

• tests/test_issue1235_prm_fp32_master_weights.py: 25 passed (was 23; +2 scoping tests), and
  25 passed again with CI=true, so nothing is skipped on a Windows runner.
• tests/test_issue382_windows_ci_guard_is_shared.py,
  tests/test_v07202.py::TestTheWindowsCiSkipStaysNarrow, tests/test_issue833_gpu_marker.py:
  60 passed.
• ruff check src/soup_cli/ tests/ scripts/ benchmarks/: clean.
• Mutation checks. Each was applied to the committed tree and then reverted, and each makes the named
  tests fail with the intended message:
  • precision without the fixture: 5 failed.
  • The fp16 test without the fixture: 1 failed.
  • The fixture with oneDNN off but fused SDPA left on: the scoping test and every step test failed.
  • A leaking fixture that sets both switches and never restores them: the scoping test failed.

What this cannot prove before merge. The fault is a host lottery. #1250's own run passed on all
three Windows cells, so a green run on this PR is not evidence either way. The evidence is the
elimination above. If this signature comes back in these tests with the fixture on, the faulting code
is not oneDNN's or MKL's half-precision GEMM. The fallback is then skip_on_windows_ci, with the
guard-sharing ratchet updated.

Open PRs that reach the same kernels

• fix(mole): train the router gate on an fp32 master weight on every device (#1266) #1312 (fix(mole), task: moe_lora_routing on CUDA trains the router gate in bf16 with no fp32 master copy: at the default lr, 82-91% of the gate's weights never move #1266): tests/test_issue1266_mole_gate_fp32_master.py runs real CPU
  wrapper.train() steps with a bf16 frozen base (the device="cuda" branch loads the base in bf16, so
  the forward runs bf16 matmuls even without autocast), under cpu_run(bf16=True), and in a CPU fp16
  autocast GradScaler step, at hidden sizes 64 and 576. That is the same class of kernel. It should
  request aten_half_matmuls in cpu_run, in the fp16 test, and in the other tests that train on the
  bf16 base.
• feat(peft): fused Fast-LoRA SwiGLU MLP autograd (#837) #1107 / feat(peft): shared-X Fast-LoRA QKV autograd (#838) #1108 (fast-LoRA MLP/QKV): CPU bf16 autocast on 8x12 layers with a (2, 5, 8) input. That
  is m*n*k <= 960, under oneDNN's 4096 threshold, and there is no SDPA. Low risk.
• feat(stream): prepare real preference VRAM probes (#840) #1141 (Layer streaming: extend training.stream_vram_probe to dpo/orpo/simpo/kto by measuring the loss's own step, not a causal-LM stand-in #840 probe): Linear(8, 8) under bf16 autocast on a (2, 8) input, m*n*k = 128. Low risk.
• fix(grpo): skip non-finite optimizer steps via gradient watchdog (Refs #342) #1084 (bf16 element-wise only) and feat(stream): make NF4 fused forward checkpoint-visible (#842) #1143 (CUDA-only, gpu marker): not affected. feat(trainer): robustly handle multi-turn, ShareGPT, and Alpaca formats in GRPO dataset preparation #1153's diff is too
  large for the API (>20k lines); its delta against main adds no CPU half-precision test code.

Follow-ups (not in this PR)

• test_rewind_hf.py and test_v05311.py::test_real_trl_grpotrainer_end_to_end_step skip on Windows
  CI under CI: windows-latest cells die with 0xc000001d in the NF4 training-step test — three occurrences, now on main #382. Their crash sites are this same CPU bf16 matmul class, because trl's configs default to
  bf16. They could request aten_half_matmuls and drop the skip, which would restore their Windows
  coverage.
• tests/_windows_ci.py still attributes the whole signature to bitsandbytes. That fits only the
  original NF4 site; the 2026-09-18 comment on CI: windows-latest cells die with 0xc000001d in the NF4 training-step test — three occurrences, now on main #382 already noted this for the rewind crashes.

🤖 Generated with Claude Code