test — Agent Guide

For deterministic DSA indexer backward, cover large TopK with more than twice as many query rows as producer planes, so persistent CTAs consume at least three rows. Restore score inputs and poison workspace/outputs before each run; use assert_bitwise_runs to compare raw bytes, including signed zero, and retain an independent gradient reference. The L1 test_DSA_indexer_backward_deterministic_topk2048_boundary covers both backends at 193 rows, 96 planes, and TopK 896/2048.

Recurrent FP64 autograd references can retain a K x V state for every token and head. When a production stress case scales heads with the GPU SM count, bound the reference by independent head chunks while keeping the full kernel workload and tolerances. Compare every chunked gradient with the original whole-head reference on smaller cases, including ragged/empty sequences and partial chunks; preserve grouped-head reductions unless explicitly validated. An OOM in the reference after kernel comparisons passed is not evidence of a kernel allocation failure. Keep that attribution explicit in CI triage.

An asynchronous CUDA failure in an SDPA reference is not proof of an attention kernel defect. Check the execution order and reproduce the reference alone, including its batch sizes, strides and concurrent processes per GPU. On SM107, varied-shape FP32 strided-batched cuBLAS calls can fail or hang even when a fixed shape loop passes. The shared reference uses ordinary GEMMs per broadcast batch; keep each dot product and the separate GQA reduction intact. Bounding only output rows leaves the large-K dK/dV contractions on the failing batched path. sdpa/graph/test_mhas_v2.py::test_sdpa_reference_matmul checks the replacement's values and gradients against exact CPU FP64 results, including broadcast layouts.

Two suites: test/cpp (Catch2, C++ graph API) and test/python (pytest). Both need an NVIDIA GPU and a cuDNN 9.x backend at runtime. Build/install commands: ../AGENTS.md.

C++ tests (`test/cpp`)

Catch2 v3 binary, target tests, built by the default CMake build (CUDNN_FRONTEND_BUILD_TESTS=ON) into build/bin/tests.
Run all: ./build/bin/tests. List: --list-tests. One case: ./build/bin/tests "Validate conv node". Filter by tag: ./build/bin/tests "[serialize]".

Plan failure diagnostics

Match each rejected plan's name and position in the same diagnostic fragment. Independent searches can pass when the message associates a reason with the wrong plan. test/cpp/plans.cpp excludes the actual offered engine names to exercise this without depending on a backend-specific failure string.

Autotune failure contracts

Warmup success alone is not a measured candidate. Check all-failed tuning against real invalid bindings, then correct the bindings and execute the preserved plan; test/cpp/autotune.cpp also covers unbuilt slots and retry. For timed-only failure investigations, inject a backend error after successful warmup and check that no winner is published. A CUDA event failure must not become a zero-time winner. After successful tuning, compare each surviving plan object's engine/knob identity, name and notes against its pre-tune snapshot. Do not assert a timing rank. Include a rejected prefix and an unbuilt suffix so pruning deterministically moves a valid plan to index zero; then replay its identity, append/build another config, and tune a serialized/reloaded plan whose engine-config list is absent.

Python tests (`test/python`)

When moving Python modules, build the validation wheel with fresh Python build output. Reusing build/lib.* can ship removed modules even when the source tree and its import tests are clean. Inspect the wheel's module paths, then run the API inventory against that wheel. QSA's package-move regression in core/test_import_boundaries.py also checks that removed imports do not resolve.

Run from test/python so pytest.ini and conftest.py apply:

cd test/python
pytest                       # pytest.ini addopts default to -m L0 (smoke) --tb=short --no-header
pytest -m L1                 # levels L0..L4; higher = larger sweeps
pytest -n 4                  # pytest-xdist; mind marker gpu_exclusive for tests that need the GPU alone
pytest -m smoke sdpa/frost   # the SMOKE tier: one cell per code path, marked from tiers/smoke_<arch>.txt (tiers/README.md)
pytest -m "L0 and not nightly_only" sdpa/frost   # the FULL local tier: L0 minus tiers/nightly_only.txt; CI stays -m L0
pytest conv/graph/test_conv_fprop.py  # one file — note the default -m L0 filter still applies
pytest gemm/cutedsl/                  # CuTe DSL kernel tests

Read pytest's own summary line; do not post-process the output. ... | grep -c "passed" counts a collection error as a pass, and cmd | tail reports tail's exit status, so a failed build or a failed suite behind a pipe looks like success. Both have produced confidently wrong "all green" reports here. Requirements: pip install -e . plus pytest pytest-xdist looseversion. The cutedsl/ directories additionally require an SM90/SM100-class GPU; tests skip (or should skip) on unsupported arch/dtype/backend-version combos rather than fail.

conftest.py landmines — read before editing

Before releasing Torch's cache, synchronize every device with reserved memory. Expandable-segment cleanup can synchronize a default-stream handle on the wrong current device, then unmap memory still used by another GPU. The later CuTe launch can report that fault. test_empty_cache_waits_for_other_devices_default_stream checks completion before the unsafe release, so its failing control stays safe.

For a timed subprocess test, arm a child-side faulthandler dump before the parent deadline and surface TimeoutExpired.stdout / .stderr in the failure. The parent's watchdog shows only subprocess.communicate(), and pytest's ordinary timeout traceback does not display those captured child streams. prepared_bwd_cache_utils.run_artifact_child uses this for cold artifact checks. Validate with a deliberately blocked child that emits both streams; retain the original deadline and do not classify a compilation stall as a GPU hang without child-stage evidence.

Source-package subprocess probes must exclude __pycache__ and bytecode when copying a tree shared by xdist workers. Python atomically renames its temporary .pyc files, so copytree can enumerate a file that disappears before it is copied (the causal-conv1d import-contract probe hit this in CI).
PYTORCH_CUDA_ALLOC_CONF is set at the very top, before any torch import (torch reads it once at CUDA-allocator init). Don't move it, and don't import torch in a plugin that loads earlier.
import transformer_engine happens (in try/except) before import cudnn — TE and cuDNN conflict if loaded in the other order. Preserve this ordering.
Crash isolation (# Crash isolation block in conftest.py): pytest_cmdline_main injects -n1 --max-worker-restart=100000, so a segfault, a poisoned CUDA context, or a hang kills only one xdist worker, which the controller replaces before continuing. After every test pytest_runtest_logfinish probes the context with torch.cuda.synchronize(); a per-test faulthandler.dump_traceback_later(exit=True) deadline (CUDNN_TEST_TIMEOUT, default 1500 s, 0 disables) covers the probe too. It is faulthandler's C watchdog, not a Python thread or SIGALRM, because a hung CUDA driver call holds the GIL and parks the main thread. Not injected under -n<N>, -s, --pdb, --collect-only, or CUDNN_TEST_NO_ISOLATION=1; without a worker to restart, a dead context stops the run via pytest.exit and a hang still hard-exits. Killing a worker does not stop a kernel it left running -- the driver keeps that context until the kernel ends, and the next worker can block behind it.
Several pytest processes on one tree or one GPU. Two things in conftest.py make that safe. (1) The FROST routing tallies (# FROST routing summary block) are written to a PER-RUN directory, test/python/.frost_routing_<host>_<run id>: the controller (or the single process) mints the id in pytest_configure and exports it as CUDNN_TEST_RUN_ID, the xdist workers it spawns inherit it (a worker without it falls back to xdist's PYTEST_XDIST_TESTRUNUID), a child pytest spawned by a test is a run of its own (a real worker is marked by config.workerinput; a session that merely inherited a worker's PYTEST_XDIST_* / CUDNN_TEST_RUN_ID drops them in pytest_cmdline_main), and pytest_sessionstart sweeps a directory whose controller pid is gone only when its name carries THIS host (_sweep_stale_routing_dirs): the tree may be shared storage holding another machine's live run, and os.kill(pid, 0) answers for this machine's pids only, so another host's directory -- and one of the previous layout, .frost_routing_<pid>_<uid>, with no host in its name -- is left alone until it is a day old (_ROUTING_LEFTOVER_TTL_S); a directory of this host whose pid is alive is never removed, whatever its age, and a pid os.kill cannot represent counts as dead; the owner is the bare hostname, so two machines that report the same name and share a tree judge each other's pids as before. With ONE shared directory a second process's session start deleted the first one's worker files and a worker that lost the makedirs / open race died with FileNotFoundError at session finish -- a red run with every test green. The banner prints Test run id: <id>. (2) The GPU memory gate (# GPU memory gate block: wait for a floor of free device memory before each test, torch.cuda.empty_cache() after it, one retry on torch.OutOfMemoryError; CUDNN_TEST_MEM_GATE_FRACTION, default 0.2 of the device, and CUDNN_TEST_MEM_GATE_TIMEOUT, default 30 s) arms itself only under several xdist workers -- never in the injected -n1 of a plain pytest <module>. K independent processes on one GPU hold their allocator high-water marks against each other in the same way, so set CUDNN_TEST_SHARED_GPU=1 in each of them: the gate then arms ([mem-gate] armed by CUDNN_TEST_SHARED_GPU=1 on stderr once per test process) and nothing else changes. Pins: test_conftest_parallel.py. (3) A record a PASSING test wants in the CI job log goes through frost_routing.measured(key, text): the controller prints it under measured next to the routing tally, aggregated across workers the same way (<worker>.measured.json in the per-run directory). A passed test's captured stdout never reaches that log -- the lanes run without -rA -- so a measurement that qualifies a constant in the package (the cc 10.7 MXFP8 backend planning guard's version bound, test_mhas_v2.py's guard-currency detector) was invisible there until it used the channel. Pin: test_conftest_parallel.py::test_a_measurement_reaches_the_controllers_summary. (4) A host-only test of version-gated behaviour pins cudnn.backend_version (a monkeypatch fixture, e.g. crashing_backend in sdpa/frost/test_sdpa_backend_guard.py): the login box and the board run cuDNN 9.26 / 9.27 while the CI lanes run 9.28, so a test that inherits the installed version passes locally and fails -- or passes vacuously -- in CI the moment the gate's threshold sits between them.
A side-stream test must order input production before consumption: call stream.wait_stream(torch.cuda.current_stream()) after creating inputs on the current stream and before switching streams. Synchronizing the consumer afterward cannot repair a missing producer dependency; the GAT/GATv2 current-stream probes exposed this under concurrent CI load.
Raw pygraph.execute launches on the cuDNN handle's stream. For CUDA-graph capture, bind the handle to the capture stream and pass it to execute. Treat the empty-CUDA-graph warning as an error so a replay test cannot capture no work (test_forward_state_alias_after_graph_warmup). A shared handle may already name stream zero: create a non-default capture stream and restore the handle afterward. Side-stream dependencies must also follow output initialization (including sentinel fills), not just input creation. Detectors: test_cuda_graph_retention_option with a default-stream handle, and QSA's test_entry_points_select_the_tensors_device_with_another_device_current with delayed producer-stream fills.
A session-scoped autouse cudnn_handle fixture creates one handle bound to a dedicated stream; use it instead of creating handles per-test. Tests that rebind it must save cudnn.get_stream(cudnn_handle) before setup and restore that exact stream in finally, including setup failures/skips. Restoring torch.cuda.current_stream() instead leaks a different stream into later tests.
pytest_configure asserts torch.cuda.is_available() — there is no CPU-only mode.
Many custom CLI options exist (--dryrun, --repro, --seed, --perf, per-op dimension overrides like --b/--s_q, --nsa-*, --dsa-*); check pytest_addoption before adding new ones.
Keep sdpa/frost/ paths contiguous on the command line. sdpa/frost/conftest.py sets CUDNN_FRONTEND_ENABLE_FROST_ENGINES through an autouse fixture, because the FROST manifest rows are opt-in. Interleaving a top-level path between two sdpa/frost/ paths — pytest sdpa/frost/test_a.py test_dispatch.py sdpa/frost/test_b.py — silently drops that fixture for everything after the top-level file: _frost_opt_in is absent from item.fixturenames and no FROST row is offered. Any top-level file does it, not just test_dispatch.py, and both contiguous orders are fine. Tests that pin an engine then fail loudly, but a test that pytest.skips when it finds no python plan goes falsely green. Detector, as its own plugin so no test file changes:
```
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
    print(item.name, os.environ.get("CUDNN_FRONTEND_ENABLE_FROST_ENGINES"), "_frost_opt_in" in item.fixturenames)
    yield
```

Native flavor routing assertions

PTX hash records also depend on imported helpers and the trace probe's fake tensor geometry. A scheduler or shared decoder change can invalidate a record even when the kernel file itself is unchanged. Before refreshing a record, render its historical source/probe and the current source/probe with the same recorded compiler distribution in separate processes, compare the PTX changes, and retain the existing SASS checks. Check distribution metadata as well as the import path: an internal build may print the public version in cutlass.__version__. test_fork_rendering_ptx_md5_is_recorded covers the SM107 D512 backward record; its dense scheduler and shared THD work-prefix changes require different attribution. A store-order optimization qualified for one width must not silently move the other widths of a shared body. Preserve their established lowering, and compare both the historical small-width cubins and the optimized width before landing. The SM107 bias/off/streamed records detect D32/D64 epilogue changes that D128 functional coverage alone misses; keep their hashes and SASS assertions intact.

When a native head-dimension flavor replaces an envelope, update the exact template expectation in existing end-to-end routing tests as well as adding new flavor tests. Keep the routing assertion, numerical comparisons and shape unchanged. The SM107 GPT-OSS D64 case in test_sdpa_fwd_cc107_decode_tile_envelope_pinned_L0 exercises native D64 prefill; a short query alone does not imply the D128 decode template.

Layout

test/python/<operation>/<backend>/ — tests grouped by operation (sdpa, gemm, conv, norm, rope, linear_attention, ...), then by the backend or framework they exercise: graph (native cuDNN through cudnn.pygraph), frost (FROST engines), cutedsl (CuTe DSL frontend-only APIs), torch and jax (framework integrations), and engine names such as linear_attention/cake. Tests that span backends sit at the operation root (e.g. linear_attention/test_la.py).
test/python/core/<backend>/ — infrastructure tests not tied to one operation (dispatch, import boundaries, graph execution, FROST buffers, ...).
test/python/test_utils.py holds shared helpers; shared SDPA references are in test/python/sdpa/.
test/python/sdpa/ is a mixed directory and the test_ prefix is load-bearing. fp16.py, helpers.py, random_config.py are harness modules the tests import; sdpa/test_*.py (and sdpa/frost/test_*.py) are collected tests. pytest.ini sets no python_files override, so a test file dropped there without the prefix is silently treated as a helper — it is never collected, and the suite stays green while asserting nothing. After moving or adding a test, confirm it is picked up by the default sweep, not just when named directly:
```
pytest --collect-only -q | grep -c sdpa/torch/test_torch_ops.py
```

Conventions for new tests

Host-only compile-cache tests that replace cute.compile must also isolate architecture-dependent compiler options evaluated before that call. Mock compile_options() when no compilation is intended, so an otherwise valid test does not query the real GPU and fail on supported non-DSA CI hosts. Detector: test_DSA_sparse_attention_backward_sm100_d576_compile_cache_separates_address_width on A100 with CuTe DSL 4.6.2.
Source-routing regexes must handle long multiline non-matches without exponential backtracking. A negated character class already includes newlines; do not repeat an overlapping newline alternative under *. gemm/cutedsl/test_sm110_amax_reduction.py::test_direct_redux_scan_handles_multiline_call covers matching and non-matching reduction calls with long argument lists.
MXFP8 input does not imply three scale-factor buffers or one-byte V. The existing PV-BF16 specialization consumes only SF_Q/SF_K, keeps BF16 V/O, and binds sf_v_ptr=None with a zero third SF tile count. Preserve the fixed role positions while rejecting a supplied SF_V before launch. test_sdpa_native_pv_bf16_binding.py compares actual host frames and checks BF16 V stride/product addresses beyond 2**32 against an independent oracle.
Scope sync-debug to the tested launch inside the capture context. Enter with torch.cuda.graph(...) before enabling set_sync_debug_mode("error"), and restore the previous mode in finally before leaving that context. Torch capture setup can synchronize and some builds reject it under the guard before the API under test runs. Keep the launch guarded and reset each graph in cleanup; test_native_d576_compact_thd_rebind_and_capture exercises this with rebinding, poisoned replay and empty inputs. A pass on a Torch build that also accepts the old scope does not reproduce the CI-specific failure.
Mark with a level (@pytest.mark.L0 ... L4): L0 must stay fast (default CI smoke); big parameter sweeps go to higher levels.
Test tiers are lists, not markers in test files. conftest.py marks cells smoke / nightly_only at collection from the committed node-id lists in tiers/ (smoke_<arch>.txt per compute capability -- CUDNN_TEST_TIER_ARCH=cc107 applies another arch's list on any host -- and nightly_only.txt); the rules a cell must satisfy, the kept twin of every demoted cell and the run lines are in tiers/README.md. CI keeps -m L0; the tiers are opt-in selections (-m smoke; -m "L0 and not nightly_only"). test_tiers.py asserts every listed id still collects and that the markers select exactly the listed cells -- when you rename or re-parametrize a listed test, edit the list in the same commit.
Default L0 coverage is not sufficient if the CI target excludes the provider. Check the actual CI path and -k filters. The general Python target excludes FROST cases, so representative shared-API FROST tests also need collection under sdpa/frost/; test_sdpa_ordered_bindings.py reuses the shared ordered-binding smoke logic. Verify both target collection and execution on a supported GPU.
Check for a module-level pytestmark before adding per-test markers. Many files apply a level or capability marker file-wide (pytestmark = ... near the top); duplicating it on each test is noise, and suggesting it in review wastes a round-trip (recurred on PRs #814, #811, #797).
Gate on capability, don't assume it: skip via check_support() failures, cudnn.backend_version(), and torch.cuda.get_device_capability().
If an optional import's except ImportError block ends before the test body, use return pytest.skip(...) to make the exit explicit. CodeQL can otherwise report the imported name as potentially uninitialized even though pytest raises a skip exception (PR #2042). Check the separate CodeQL alert result as well as the successful analysis workflow.
Test feature combinations through the support contract, not intersections of capability tables. The same head dimension can support gate and split on different paths while rejecting their combination. Assert acceptance/decline for concrete graph facts and requested knobs; avoid heuristic winner/order or timing goldens.
Shape-override windows and strict xfails measured on one native plan must also gate on the selected plan configuration. Negative geometry probes must allocate storage for every overridden row and use matching physical strides; an xfail does not contain an out-of-bounds GPU access.
Large physical-stride tests must handle allocation-time memory pressure: another xdist worker can consume free memory after mem_get_info(). The gpu_exclusive marker alone does not serialize ordinary xdist scheduling (the convention is the marker plus @pytest.mark.xdist_group(name="gpu_exclusive"), which --dist loadgroup runs on one worker; a test that deliberately wedges the GPU and the detector it is the control for must both carry it -- see the time-slicing lesson in python/cudnn/sdpa/AGENTS.md). Catch torch.OutOfMemoryError only around the large test-storage allocation and skip for unavailable resources; never catch the launch or numerical assertions. Retain a successful physical run with sufficient memory.
Compare against a reference implementation (see existing *_ref.py / *_reference.py patterns) with dtype-appropriate tolerances.
Check reference GEMM rounding before relaxing a chained-kernel tolerance. Save the failing inputs and compare the projection in FP64 before its BF16 cast. A small FP32 accumulation error can cross that cast's midpoint, then grow through normalization. The serving pool tests use the existing acc_dtype=torch.float64 oracle option while retaining all intermediate dtype casts and tolerances; pipeline 74800620 exposed this at token 350, head 0, column 109 on a 212-SM Rubin GPU.
Reference cache (opt-in). sdpa/ref_cache.py::cached_reference(name, key, compute) stores an oracle's outputs on disk, keyed by the reference sources' content (_REF_SOURCES, ref_cache.py itself included; the list is closed under its own relative / sdpa. imports -- sdpa/test_ref_cache.py::test_the_reference_sources_are_closed_under_their_own_imports fails when a listed module imports an unlisted one), the WHOLE content of the recipe's files -- the file compute is defined in and every file on the call stack at the call that is not the interpreter's own: the recipe module, a shared wrapper that built the oracle closure, the test module (frames of pytest / pluggy / torch / the standard library are skipped, not stopped at); the draw order, the quantization call sites and the oracle wrapper are hashed as files, so an edit anywhere in them is a miss, never a stale hit -- the key names the inputs' recipe literally; a helper the recipe imports and has already returned from when the oracle is called is covered through _REF_SOURCES only, so list it there --, the device class (name, compute capability, SM count -- one seed is a different dataset on a different SM count), torch's version and the caller's key -- which names what the files' content does not pin because it arrives as a parameter: shape, seed, scale, mask, dtype; off unless CUDNN_TEST_REF_CACHE=<node-local dir> names a directory (never a cross-site network file system). Files are hashed as they are on disk at the call, so an edit made WHILE a cache-enabled run is in progress keys that run's later entries by the new content though the old code computed them: after such an edit run once with VERIFY on, or use a fresh directory. A hit returns a previous run's values bitwise, so tolerances are untouched; the point is CUDNN_TEST_REF_CACHE_VERIFY=1 (0 / false / no / off = off), which recomputes on every hit and fails the test if the reference drifted from its own past. Entries are read with torch's weights-only unpickler -- tensors and plain Python containers / scalars, all the cache ever stores -- so a pickle payload or a foreign file at an entry path is refused as a miss, never executed; still, keep the directory writable by the test account / CI job only, since whoever writes it decides what a run compares against. The fp8 d=256 backward suite (sdpa/frost/test_sdpa_bwd_fp8_sm107.py::_run_fp8) also memoizes its oracle in-process across the parametrizations of one case (the gradient dtype only casts the fp32 gradients), bounded by bytes; under VERIFY the memo stands aside so every repeat is verified too.
Scale the tolerance to the tensor, not to the dtype alone. A fixed absolute bound quietly becomes wrong when magnitudes grow: GQA dK/dV sum over h_q/h_kv query heads, so at a group size of 4 the relative error stays ~0.5% while |dv| peaks near 9.6 and blows a bound that passed at h_kv == h_q. Compare against TOL * max(|ref|.max(), 1.0), or the next GQA ratio someone adds will look like a correctness regression.
Shape-override tests must cover a backend lowering decline as well as a lowered graph. Ragged-offset tensors are backend-only operands and can be absent from the Python-only layout; filter those auxiliary overrides against _variant_pack_uids() while requiring every Q/K/V/O, Stats and length operand. test_thd_cache_shape_grid_tracks_runtime_capacity exercises both layouts without weakening capture, launch-bound or replay checks.
Plan selection must respect current filters. When testing build-all after an explicit selection, also exclude that selection before rebuilding. Retain a previous candidate only if it is still accepted; a stale candidate must not turn an all-rejected build into success. The Python walk must also restore the retained C++ candidate: compare selected behavior-note and handle-taking workspace queries with indexed queries after ALL (test_build_all_preserves_backend_selection_queries). test/cpp/plan_selection.cpp covers this alongside initially unselected and explicitly selected plans. An OSS support check can select a sentinel before compilation: inject a failing engine build and verify native fallback, failure with no alternative, and a successfully built OSS control. Support alone must not count as a built plan.
Heuristic tests must survive legitimate tuning changes. Do not pin a particular workload's winning scheduler, packing or split count, candidate order/exact set, or a performance threshold. Do not turn the current measured/unmeasured shape boundary into a correctness contract: another independent optimization may legitimately choose a different plan for the same control shape. Test explicit knob admission and rejection, validity of proposed candidates, and transport of a mocked chooser's result; see test_propose_preserves_recommendations_and_places_one_marker. Check geometry against an independent oracle, rather than spying on private arithmetic helper calls. Kernel tests explicitly select supported knobs and verify O/LSE, changed-input capture/replay, and storage bounds. Exact expectations belong to semantic/API contracts, with the invariant stated in the test. Performance rankings and tuning boundaries belong in reproducible offline benchmarks with source/hardware attribution, not CI golden assertions.
Benchmark labels must describe the executed backend layout. Apply route-specific page-size constraints before deduplicating cases, and check the subprocess arguments against result metadata. The CPU-only tests in benchmark/attention_inference/tests/test_runner.py cover b12x's distinct paged, absorbed-MLA and contiguous routes without launching kernels.
Standalone benchmark summaries must reject non-finite or nonpositive timing medians and non-finite derived measurements before replacing existing output. Use explicit validation rather than assert, which Python -O removes. Serialize with allow_nan=False. Check all expected numerical errors against their tolerance instead of trusting a saved pass flag. Benchmark producers must also use explicit checks for scheduler activation, finite tensors, profiler activity, and numerical acceptance (benchmark/linear_attention/steep_decay/).
Memo-key regression tests should observe the actual lookup and the selected plan, rather than relying on positional offsets in private key tuples. Appending an independent cache axis must not break an unrelated test; keep the miss count, selected configuration, and A-B-A identity checks. test_block_scaled_memo_respects_overlap_margin records the lookup key through its test memo.
Workspace device checks compare storage with the declared launch device, not a fixed CUDA ordinal or the ambient current device. Match the diagnostic's actual expected/observed ordinals; the wrong-device message need not contain the word "device". core/cutedsl/test_workspace_device.py covers the diagnostic with one GPU and both storage-device directions when two are visible, including DLPack-only and CUDA-array-interface-only buffers.
A dispatch fact that reads the device's SM count is asserted against the live resolver, never as a constant. The small-M projection route, the sparse decode split rule and the combine form all read multi_processor_count, so a host test that pins their outcome for one shape (not capture_safe, split == 16, a tile name) encodes the part it was written on: the N 5760 band slab's "graph route" expectation held on 212 and 108 SMs and failed six cells on a 68-SM device, where the slab's 90 64-row tiles overflow a wave and the route names its 128-row tile (review on PR #1754). Compute the expectation from the resolver the API itself runs (proj_gemm._auto_tile_name, sparse_decode_split_choice(..., sm_count=sm)) and assert the pinned branch only under the live SM count that selects it; a declaration-time flag derived from such a resolver is re-read where it is consumed (the block re-reads capture_safe at compile()), so the two cannot disagree across devices.
A regression test must be seen RED. Before trusting one, run it against the unfixed code — restore the old line, confirm it fails, restore the fix. test_dsl_sm100_thd_interleaved_kv_views and test_varlen_backward_does_not_sync were both checked this way, and both were genuinely red beforehand; a test written for a bug and never seen to fail is asserting an unknown.
A harness mirror of an engine's admission rule includes the engine's DSL target gate. sdpa/mxfp8.py::block_scaled_o_draw admits a block-scaled O draw where the FROST MXFP8 row would serve it, and an admitted draw the row then declines FAILS the test (the sf_o output has no backend lowering), so the mirror must agree with the row on every gate. When the cc 10.7 row became a default candidate (PR #1478) the mirror asked the manifest but not cutedsl_arch_requirement_error: on a cc 10.7 host with the public CuTe DSL 4.7.0 (no sm_107a) the row declines through that gate while the mirror admitted the draw, so every flag-less d128 block-scaled draw of test_sdpa_mxfp8_fwd_L0 would have failed where develop ran the backend on the folded draw (verified host-side with the target modelled absent). Model the target on both sides: sdpa/test_mxfp8_block_scaled_fold.py::test_cc107_default_admission_needs_the_sm_107a_dsl_target pins the fold with buffers._cutedsl_has_sm107 patched absent and present.
A version-bounded backend workaround ships with the detector that re-measures it. A constant such as sdpa/fwd/backend_guard.py::SQ1_MXFP8_PLANNING_CRASH_FIXED_IN is a measurement, and nothing in an ordinary test run would ever notice the backend getting fixed (the guard keeps it from being asked). sdpa/graph/test_mhas_v2.py::test_sdpa_mxfp8_cc107_backend_planning_crash_guard_is_current_L0 runs the trigger matrix one interpreter per contract with the guard disabled in the child (the constant set to 0 there; a crash is the child's rc 139, never the runner's) plus a control contract the backend is known to plan, and fails with the version to record once the installed backend plans the matrix cleanly -- or, above a recorded version, once it crashes again. Spawn the children with the test module's own directory and test/python on their sys.path; they inherit neither pytest's path setup nor its fixtures.
Native binder migrations must also run the existing prepared-launch tests. An invalid override can violate both a fixed split shape and Q/O agreement: preserve the primary declared-shape diagnostic, not just rejection. test_sdpa_prepared_thd.py::test_split_prepared_strided_output_rebind_and_capture checks this after numerical and replay validation.
Poison unused attention storage. Use independent indices; poison unused KV with NaN, infinities, and large finite values. Require unchanged valid gradients and zero unused gradients in eager execution and graph replay. Check +inf sinks against a finite dominant-sink control.
Pair very negative LSE with large finite dO. Exponent clamps can still overflow in dS. Use an analytic reference and confirm the test rejects masking after the product.
FP8 attention references must mirror intermediate probability rounding. Full-precision softmax times V is a different function from the kernel's FP8 P product and can exceed a fixed bound, especially with an amplified V tail. Use the shared sdpa.fp8_ref.compute_ref with the selected kernel's P scale and rescale threshold; keep output tolerances unchanged. test_paged_adapter_fp8_compile_key_canonicalizes_the_logical_kv_maximum exercises this while checking 96/128/96 artifact reuse.
Every declared quantized output is asserted -- Amax_O included. The quantized harnesses bound Amax_O and only printed it: the dense MXFP8 compare checked max|O| against the reference's max|O| (never the Amax_O tensor) and the per-tensor FP8 compare checked nothing, so an engine that wrote 0 on a non-zero O, reported a value above the O it stored, or reduced dead query rows into it passed for months (the cuDNN 9.26.0.51 backend does all three: dense MXFP8 fprop with fp16/bf16 O, and paged FP8 with seq_len_q = [0]; found 2026-10-07 while the FROST engines were suspected). sdpa.fp8.assert_amax_o pins the output two ways: against the reference's fp32 pre-cast, pre-scale amax within one P code step of itself (a midpoint flip on the dominant key of the largest element; compute_ref(..., output_type=torch.float32) / the FP8 reference's o_amax), and against the kernel's own stored O within half a code spacing of O's dtype (an amax not taken from the stored values fails even inside the reference bound; per-tensor FP8 drops only the upper bound when the stored O saturated at the top code). Prefill the output with NaN, never zero, so "never written" and "wrote 0" stay distinguishable. A backend-served graph (graph.selected_engine is None) xfails that check with BACKEND_AMAX_O_ISSUE; a FROST-served graph fails. sdpa/test_assert_amax_o.py pins each rejection and the one accepted deviation.
An empty live O still requires a finite, exactly zero Amax_O. Avoid reducing an empty tensor, but keep the numerical assertions and poison-tail checks. sdpa/test_assert_amax_o.py accepts empty outputs and rejects nonzero or nonfinite Amax values, including small values below the ordinary absolute tolerance and the saturated-output path. The all-Q-empty case in sdpa/graph/suites/context/test_context_mxfp8.py::test_context_mxfp8_thd checks the native launch; do not skip it to work around a checker failure.
Low-precision quantization needs exact midpoint tests. Approximate reciprocal multiplication can move an exact E2M1 tie across its rounding boundary even when the native conversion uses round-to-nearest-even. Include signed midpoint values with non-power-of-two block scales, and compare the quantization stage itself before diagnosing amplified attention-gradient differences.
Build the reference in fp64 when the bound is tighter than ~1e-3. The DLFW CI containers run fp32 matmul in TF32 (TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1; recent torch also defaults fp32_precision to tf32 on Blackwell+), a ~3e-4 relative error per Q @ K^T logit. A 1024-column log-sum-exp averages it down to ~2e-5, a causal row with ONE valid column keeps it whole: test_fp8_stats_is_the_exact_softmax_lse[causal] read max|dLSE| 1.3e-4 against its 1e-4 bound on the sm107 lane (2026-09-15) with an exact kernel, and passed on a 208-SM node whose draws happened to be kinder. sdpa/fp8_ref.compute_ref(dtype=torch.float64) is the existing knob; in a hand-rolled reference .double() the operands before the matmul, not just the logsumexp.
An fp8 midpoint flip is PROVED from the reference's intermediates, never inferred from the output's shape. One flipped P/dS code moves one output d-row by (c_alt - c_ref) x descale x operand_row (Q for dK, K for dQ, dO for dV, V for O) -- but a power-of-two multiple of an operand row is not evidence of one: a masked key, another batch's or another head's row, two identical rows each one flip away, or a step no adjacent pair of codes has (8192 in e4m3) all fit that description (review on PR #1075). assert_close_fp8_grad(..., operand=, flip_unit=, intermediates=, fp8_dtype=) lifts the 4 * atol row cap only when, at a VALID position of the row's own reduction (same batch, a q head of the same GQA group, unmasked), the reference's scaled fp32 intermediate -- compute_ref(..., return_intermediates=) / compute_ref_backward(...), re-run on the bad rows only -- sits within 1/32 of a code spacing of the midpoint between its fp8 code and the adjacent one, and that single flip reproduces the row three ways: within the ordinary tolerance, per element within the output's own rounding (atol plus half an output code spacing from each side -- both sides are dequantized output codes; pass out_dtype=torch_otype), and as a whole (the least-squares number of flips fitted to the row is 1 within 1/2 -- two flips fit at 2, and on a row of large gradients rtol * |expected| is itself two flips wide, so the ordinary tolerance alone would take three flips for one). It prints the position, the two codes, the step and the three fits. Packed (ragged) outputs keep the plain cap, as does h_k != h_v. sdpa/test_fp8_flip_budget.py pins the accepted case and each of those rejections on a problem the reference itself built. The negative-score q rows have amplitude 8 at d192, so a legitimate flip moves a dK row by 0.5 -- the fixed cap (0.32) alone would call that a defect (sm107 212-SM lane, test310, 2026-09-15).
Decode Stats must be tested independently of training. The random SDPA harness uses generate_stats=cfg.is_train, so its s_q == 1 inference sweep checks O without checking LSE. For ragged GQA decode, request Stats explicitly, initialize every head to NaN, and compare every head against the reference. Include padded-Stats, MHA, and s_q > 1 controls; pin a backend plan when testing native codegen so FROST cannot mask it. test_mhas_v2.py::test_sdpa_ragged_decode_stats is the detector (NVBug 6783545); run with --runxfail when checking an affected older backend.
Plan-specific xfails must check the selected plan. A heuristic can start ranking a working engine first without fixing another engine's compiler bug. An architecture-only xfail then turns correct outputs into strict-XPASS failures. Inspect get_engine_and_knobs_at_index() for the selected plan and keep strict=True plus the expected exception type for the affected engine. test_sdpa_ragged_decode_stats distinguishes the affected native 10X/107 engines (10/18) from working engine 8.
Match the softmax precision before budgeting FP8 rounding. HALF rounds both the exponent input and its approximate result before the FP8 P cast; a FLOAT-only reference can report data-dependent midpoint flips. Use the documented exponent error propagated through P and V, plus an identical-input unpacked control for packing changes. For split-KV, derive P-rounding bounds at each partition's own maximum and combine them using independently computed softmax weights (including the sink). An unsplit allowance is not a split reference. Keep independent Stats/Amax and replay checks. Do not fit expected failures to one random tensor coordinate.
Seed before you allocate. torch.manual_seed() after constructing the inputs seeds nothing that matters. Two runs meant to be compared then differ by data, and the assertion fails (or worse, passes) for a reason unrelated to what is under test — if two runs must be comparable, build the inputs once and reuse them.
Bound dense attention references independently of kernel memory. A full [B, H, S_q, S_kv] FP32 score tensor and GQA repeat_interleave can exhaust a small GPU when xdist workers share it, even after the tested kernel succeeds. Compute independent heads separately, mapping each query head to its KV head, while preserving the full workload, masks, precision and tolerances. Validate the refactor against the dense oracle on small MHA/GQA/MQA cases and run the original large kernel test under the same allocator cap before and after the fix. test_sdpa_fwd_split_kv_sm120.py covers this pattern.
Every randomized SDPA input uses the per-test generator. A seeded Q/K/V tuple is not a reproducible case if its block mask comes from the process-global CUDA RNG. Pass generator=rng_data_gen to auxiliary draws too; test_block_mask_uses_the_per_test_data_generator perturbs global RNG while holding the case seed fixed. Before attributing an order-dependent failure to an earlier engine, compare the actual masks as well as Q/K/V.
Compiled DSL call arity excludes compile-time parameters. A cutlass.Constexpr argument belongs to the compilation signature and disappears from the compiled runtime call. When checking positional launch sites against _host, exclude these annotations as well as the stream keyword; do not add a runtime argument to satisfy an unfiltered Python signature count. test_every_combine_call_site_matches_the_compiled_arity is the detector.
A new architecture reuses the shared forward harness through _ARCH, not by copying it. sdpa/frost/test_sdpa_fwd_dsl_sm100.py selects its engine architecture through a module-level _ARCH string fed to engine_name(arch=...). A new row reuses its runners and assertions from a file carrying its own device gate:
```
@pytest.fixture
def sm100(monkeypatch):
    import test_sdpa_fwd_dsl_sm100 as module
    monkeypatch.setattr(module, "_ARCH", "sm90")
    return module
```
Two traps. (1) That module's pytestmark = requires_blackwell skips all of its cases on a non-Blackwell card, so the delegating file must carry its own gate and call the runners as functions — widening the shared gate instead puts three other architectures at risk, and its parametrized head dims are the SM100 line's. (2) test_sdpa_compiled_cache_gpu.py keeps _ARCH inside its _CHILD subprocess source string, where monkeypatch cannot reach; parameterize the string. Detector for the pin itself: delete the arch's row from ENGINE_SPECS and every strict select_engine pin must fail with no plan for engine ... — if it doesn't, the pin wasn't strict.
A compiled-helper test does not cover AOT backward. torch.compiler.is_compiling() can be false while AOT traces a custom op's backward with FakeTensor/FunctionalTensor inputs. Keep raw-pointer helpers behind a registered custom op with a fake implementation even in that context. Run the enclosing op's torch.library.opcheck, including dynamic AOT dispatch; sdpa/torch/test_torch_ops.py::TestOpContract::test_opcheck detects this for packed Stats preparation.
Pointer-ABI stride fakes must preserve Int64, including page tables. Annotating a host stride as Int64 is insufficient if its compile-time fake uses a plain Python 0, which can infer Int32. A singleton axis can legally have a stride above 2**31 without requiring a large allocation; use that layout to catch narrowing at binding time. test_graph_decode_prepared_keeps_int64_page_table_batch_stride is the decode detector.
A native binder must keep observed storage separate from effective geometry. Graph declarations and overrides can enlarge logical shapes without enlarging the caller's allocation. Derive ragged capacity from the producer's observed byte span in the effective element width; for fixed-size length/Stats reads validate known observed spans as well as logical numel. Keep the bare-pointer unknown-span contract explicit. test_sdpa_native_thd_binding.py checks these rules against the Python binder, including misaligned int32 lengths and overrides that claim more storage than the producer owns.
Paged overrides retain the producer's storage bound. Validate each pool's effective TMA byte strides and observed span, and page-table element alignment, before launch. A larger override does not enlarge the allocation. Use host-only malformed-fact probes instead of launching an invalid tensor; test_sdpa_paged_binding.py covers short pools, misaligned strides/tables, and valid wide-stride or unknown-span bindings.
Wide host strides must stay wide through device setup arguments. An Int64 pointer host can still truncate a stride while launching a descriptor-setup kernel. Check casts at the launch site and the setup parameter, not just the host signature. Exercise two live sequences with a physical row stride above 2**32 and poisoned output; a singleton-axis or binder-only probe never steps the truncated address. test_thd_output_row_stride_above_int32_reaches_device_descriptors checks numerical output and replay. Cover every served arch/flavor: a pre-Rubin-only marker hid narrowing in all four SM107 half hosts. test_mhas_v2.py::test_sdpa_thd_output_stride_int64 is collected by both CI arch selections and checks native and Python binding independently, including D192/D128.
Unchanged device-function ASTs do not imply unchanged generated code. Replacing static layout constants with runtime strides can change device address calculations; compare GPU time for the affected cases. A unit-stride fast path must also exercise nonunit strides through the same compiled host; test_d256_paged_host_rebinds_table_column_stride checks this contract.
When you remove a fallback, invert its counter assertion — do not delete it. Tests that asserted calls["bwd_cpp"] incremented had to become "calls["bwd"] increments and bwd_cpp does not", so a silent regression to the old path fails the suite instead of passing it.
A standalone FROST adapter is compiled at plan time, so every test driver calls compile() before execute(). The sparse d256 adapter refuses to compile on the execute path (plan-time keys only), and the helpers that still relied on an execute-time compile failed every GPU cell of two modules while the host tier, where those cells skip, stayed green — so a change to an adapter's execute contract re-runs the GPU tier of every suite that drives it, and the detector is an .execute( with no preceding .compile( in the same driver (sdpa/frost/test_sdpa_fwd_sparse_sm107.py, sdpa/frost/sparse_item_sequences.py).

Binding introspection

Binding migrations can change introspection without changing the public API. Python Enum hides instance properties such as name and value from dir(enum_class); the API-index scanner must inspect their descriptors without evaluating them (api_index/test_api_index.py covers Enum, IntEnum and aliases). Signature tests should accept both object and object | None for an argument defaulting to None, while retaining argument order and default checks.

Confirm you are testing the code you edited

Bind tensor views in the graph's declared axis order. A BSHD allocation and a BHSD declaration can have identical shapes when H == S; shape equality then preserves the producer's strides and cannot infer the intended transpose. Use a metadata-only transpose at the test binding boundary, keeping the reference's allocation unchanged. test_sdpa_fp8_paged_equal_head_and_query_axes covers this collision through the actual FP8 harness and prepared executor.

pip install -e . does not put the package on sys.path. It installs a sys.meta_path finder (__editable___nvidia_cudnn_frontend_*_finder.py) whose MAPPING hard-codes an absolute path to the checkout it was installed from. Meta-path finders run before sys.path, so PYTHONPATH cannot shadow it — if you edit a different clone or a git worktree, your changes are silently not under test. Symptoms are indistinguishable from a real result: a probe that should change the output leaves it bit-identical, and edits appear to do nothing.

Check first, every time you work outside the installed checkout:

python -c "import cudnn; print(cudnn.__file__)"   # must be YOUR tree

conftest.py prints the same path in its banner (cuDNN Frontend Path:) — read it rather than assuming. To point the editable install at another tree for one run, patch the finder's MAPPING from a sitecustomize.py on PYTHONPATH (site imports it after processing .pth files, so the finder already exists):

# sitecustomize.py -- the finder module name embeds the installed version, so
# discover it rather than hard-coding it (it changes when __version__ bumps).
import importlib, pkgutil

name = next(
    m.name
    for m in pkgutil.iter_modules()
    if m.name.startswith("__editable___nvidia_cudnn_frontend_") and m.name.endswith("_finder")
)
importlib.import_module(name).MAPPING["cudnn"] = "/path/to/your/worktree/python/cudnn"

The same trap hides inside a run: python/cudnn/frost/template_loader.py loads kernel templates by absolute path via spec_from_file_location, so the template that serves a config may come from elsewhere too. To find out which template a test actually compiles, log path at the top of load_template — do not infer it from _pick_flavor by reading the source.

Nested conftests can also change package lookup after the initial banner. linear_attention/conftest.py inserts its own checkout's python/cudnn into cudnn.__path__. Running a candidate LA test by absolute path with a baseline package can therefore load candidate implementations and produce a false GREEN baseline. For RED/GREEN checks, copy only the new test changes into an isolated baseline worktree and collect there; verify the concrete operation module's __file__ after conftest setup, not only cudnn.__file__.

Read repository documentation relative to the test checkout, not cudnn.__file__. CI can load a wheel's package initializer while conftests overlay source modules; the wheel may omit Markdown files. The GEMM descriptor-order checks in gemm/frost/test_block_scale_smem_layout_sm107.py exercise this boundary.

Import-regression subprocesses are separate interpreters: in-process sys.path or editable-finder changes do not automatically propagate. Carry the selected package/source setup into each child and verify its loaded path. Children that import test helpers need both the helper directory and test/python on their own sys.path; pytest's parent-process path setup is not inherited. Run fresh-process cache probes from their temporary directory so launching pytest from test/python cannot hide missing child imports.

Pending-consumer lifetime probes

A CUDA stream wait on an event that has never been recorded is a no-op; recording it later does not retroactively block the consumer. Do not use an unrecorded event as a host-released latch. A bounded delayed-consumer probe must assert that its completion event is still pending after the producer/churn work. If the delay expires, fail the setup instead of accepting a lifetime result without overlap. Use a bounded byte consumer for recycled-storage negative controls, never a real kernel on a deliberately invalidated workspace.

Caller workspace alias regressions

Check the carved scratch byte range against operand byte spans, including packed uint8 FP4 storage, strided views, optional outputs, and device pointer tables. Intercept the compiled consumer for negative tests so intentional aliasing never reaches a kernel; prove RED before the fix. Also exercise disjoint slices of one allocation so rejecting shared ownership does not substitute for checking overlap. For per-operand negative cases, keep other operand ranges clear of the tested workspace interval and match the rejection to that operand. Otherwise one overlapping fixture can hide missing checks for every other role. Device pointer-table contents remain a caller contract, not a reason for a D2H read.

fp32 operands and TF32 references

Blackwell MMAs have no IEEE fp32 mode: fp32 operands run as TF32. Compute their torch reference with torch.backends.cuda.matmul.fp32_precision = "tf32" set explicitly in the test. Torch's default depends on the environment: NGC containers (CI) set TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1, while pip torch uses IEEE fp32. A test that relies on the default passes in CI and fails locally, where test_gemm_swiglu.py had 64 such failures.

Clamp gradients at the boundary

torch 2.14 changed clamp's backward on the boundary itself: x.clamp(-7, 7) at x = [-7, 0, 7] gives [0, 1, 0], while 2.11 / 2.13 give [1, 1, 1]. An autograd oracle that puts inputs exactly on a clamp limit therefore changes with the image's torch. State the kernel's convention with an explicit torch.where(inside, x, x.detach().clamp(lo, hi)) instead (test_grouped_dgeglu_autograd.py::clamp_closed, #1522).

Sanitizer diagnostic controls

Keep kernel-launch dumps separate from the pass/fail sanitizer invocation. Some Compute Sanitizer builds count each --dump-kernel-launches record in ERROR SUMMARY and apply --error-exitcode even to a valid kernel. Check a valid-memory control with and without that option before diagnosing a kernel error from the count alone. Use a known-bad memory/race canary to verify the detector, and retain a normal run of the real capture test without launch dumps.

CUDA Graph test lifetimes

Explicitly reset a test-owned CUDA Graph after replay verification, using a finally block or context manager. Python reference cycles can defer its CUDA executable destruction until GC runs inside a later test's capture, invalidating that capture. The SM120 FP8 prepared suite reproduced this when the D128 strided-input graph was collected during the D384x320 case; tracing CUDAGraph.__del__ identified both tests. Keep capture error mode unchanged and fix ownership instead of disabling GC or treating a retry as validation. conftest.py enforces this. After a test that leaves a captured, never-reset graph alive, it runs gc.collect() between tests and fails that test in teardown if the collector freed one. Whether a graph lands in a cycle can depend on test order, so a test that passes alone can still trip it in a suite.

Architecture test re-exports

When an architecture test module re-exports shared checks, those checks must not import fixtures or reference math back from that architecture module, even inside a function. Move shared helpers into a non-test_ module and import them from both sides. CodeQL's cyclic-import check catches the return edge; a successful import or collection alone may miss it because the edge is lazy. Preserve reference helper ASTs when moving them.

Prepared quantized launch probes

Native dense SDPA data ports use aligned pitches and must not acquire hidden layout copies. test_native_rebind_stream_capture in test_sdpa_bwd_native_layouts_sm100.py guards padding and forbids legacy tensor compilation/DLPack while checking changed allocations and replay. Unaligned outer pitches are negative admission cases, not a request for staging.

Optional gradients copied from accumulators after a prepared launch still need presence, dtype and extent checks before any staging write. They can be absent from the pointer ABI, so the common binder cannot validate them. The detector is test_native_auxiliary_outputs_validate_before_writes (SM80 backward): forbid copy/zero/launch and pass malformed or uncompiled dBias/dSink outputs.

Test both graph prepared-plan admission and the standalone adapter's compiler selection when retaining a tensor fallback. Declining the graph attachment alone can still compile a prepared artifact inside the adapter and fail at execution. test_fp8_paged_prepared_table_stride_admission checks both decisions for distinct K/V page-table strides; test_fp8_paged_distinct_table_strides_prepared checks both routes numerically with the tensor compiler forbidden, changed allocations and CUDA Graph replay.

For descriptor stride products, inspect the traced multiplication intermediates, not just the final cast or Python annotation. MXFP8 V scales use a separate plane stride: test_mxfp8_v_scale_plane_stride_multiplies_in_int64 checks the real host expression before descriptor encoding and must fail before widening.

The SM107 CI lane selects test_sdpa_fp8_sm107.py explicitly. Keep its prepared FP8 cases in TestPreparedSm107Fp8 there, or update the lane selector together with a move; a new sibling file alone is not exercised by that lane. An imported test function does not inherit its source module's pytestmark. When re-exporting L0 checks into another architecture file, mark the wrapper class L0 explicitly and verify collection with the lane's marker expression. test_sdpa_fwd_mxfp8_sm100.py::TestStagedMxfp8 covers this boundary.

Rebind scale buffers with different values, not only cloned storage: identical values let a stale pointer pass. Poison and rebind amax too, then change scales in place after CUDA Graph capture and check outputs after replay. test_prepared_fp8_rebind_scales_and_buffers and test_prepared_fp8_capture_replay_reads_current_scales cover both lifecycles.

Prepared SM120 dense launches

SM120 masks the rightmost KV tile for every mask mode, unlike SM100's padding-dependent tail rule. A shared binder must preserve that distinction. test_sm120_prepared_bounded_geometry_override shrinks an unpadded plan to S_kv=113, reuses the artifact, and checks O and Stats. Keep head counts and dimensions as compile-time constants: the same binder fixes those per plan.

Sparse score-recompute resource boundaries

Exercise non-power-of-two TMEM slot counts and partial ring wraps across persistent query tiles; a full-ring single-tile case misses slot aliasing and per-slot barrier-phase drift. Keep the focused regressions in deepseek_sparse_attention/cutedsl/test_DSA_sparse_score_recompute.py. For SM100 SMEM planning, test a boundary that distinguishes the usable 227 KiB from the nominal 228 KiB; test_DSA_sparse_attention_score_recompute_uses_launchable_smem_budget must fail against the old planner before accepting the fix.

Sparse metadata and cross-warp reduction scratch also need explicit reader completion before reuse. Run racecheck on both indexer and attention cases: ordering Q/K MMA alone does not publish every metadata lane's stores, and a max-to-sum reduction can overwrite shared scratch before all warps read it.

SM107 metadata/heuristic tests run on non-SM107 CI lanes too. Model compiler target availability explicitly when testing a synthetic Rubin capability row; do not require the worker's DSL wheel to expose sm_107a. Keep real installed compiler-target decline tests separate, and retain the live target check for actual SM107 kernel execution. The release-DSL SM80 lane exposed this split.

A capability row's sm_lo names its lower bound, not the actual device. Prepared admission must respect the adapter's exact device support even when the row spans later compute capabilities. Include future-cc rejection controls alongside the supported device in test_prepared_fp8_override_capability_envelope.

Concurrent prepared frames versus SDK initialization

CuTe DSL 4.7.0/4.7.1 can leak the runtime's process-global initialization lock when two threads first call the same cold artifact; a later test then hangs in cuda_dialect_init_library_once before its kernel launches (see the independent runtime reproduction). For a test of concurrent per-call bindings, initialize the artifact serially, retain that warm call's owners, then use distinct buffers, poisoned outputs, fresh workspace and independent streams for the concurrent calls. test_native_thd_concurrent_streams_use_independent_frames follows this recipe. This tests frame independence; it does not establish that cold concurrent SDK initialization is fixed. Keep that runtime reproduction and its result separate.

MXFP8 prepared scale-factor bindings

Dense V scale factors for D > 128 are plane-major; THD factors are packed per-sequence tiles per head. Reuse _quantize_seq when constructing THD test inputs; reshaping a dense buffer does not produce the THD contract. Rebind E8M0 exponent values as well as pointers and validate after graph replay. The MXFP8 split combine has no per-tensor output scalar: specialize its scale pointer to None and remove Amax unscaling, rather than handing a NULL runtime address to the per-tensor unscale kernel. test_prepared_mxfp8_capture_reads_current_scales covers every native flavor with requested Amax. Physical SF stride units are 16 bytes; widen tile-count products before multiplication. The L1 test_prepared_mxfp8_sf_head_stride_above_int32_units steps a physical 64-GiB head stride and checks numerical output, with resource-only OOM skips.

Linear SF repacks need the same width audit: an Int32 block index can wrap before the byte address is formed even without a wide declared stride. test_sf_repack_steps_past_int32_with_allocated_guards crosses the signed Int32 boundary for both backward SF layouts. Its source and destination prefixes keep the old negative offsets inside allocated storage, so the old implementation fails numerically rather than through an invalid access.

Packed SM80 forward launch migration

After moving a wrapper to a prepared host, forbid the old tensor launcher after warmup and check fresh bindings plus changed-input graph replay. Keep physical stride and product-overflow probes on every native flavor, with wrapped addresses poisoned inside allocated guard storage. test_sdpa_sm80_thd_forward_prepared.py exercises both properties. Dense padded widths retain native strides; unsupported RoPE declarations must decline before compilation.

SM80 native pointer launches

Aligned dense and packed layouts execute through native pointer chains. Before removing a staging fallback, make its entry points raise and run the SM80 suites. Retain gradient/output references, current-pointer rebinding, launch-count checks and fresh-process artifacts. RoPE declarations decline during preparation. test_dense_native_replay_or_rope_decline checks this boundary.

Trailing-dimension and contiguity checks do not prove that a broadcast input has a live batch slice: (0, H, SQ, SKV) passes both for an empty bias. Reject it before workspace writes or pointer binding; the bias_tensor-empty_batch case in test_dense_native_rejects_invalid_operands_before_launch is the RED-then-green detector. A compiler-memo test must also rebuild the wrapper on its second call: clear the wrapper cache while retaining the compiler memo, then forbid JIT. Reusing the same adapter cannot test compiler reuse.

SM80's standalone packed wrapper preserves cumulative tensors as row offsets into the supplied storage. Do not substitute graph API cumulative-length normalization: that changes which Q/K/V rows the standalone call addresses. test_thd_wrapper_preserves_packed_row_origins checks different Q/KV origins, empty sequences and changed origins after capture, including zeroed output capacity outside those origins. Keep baseline compatibility evidence.

Prepared THD artifact reuse must survive a disabled persistent cache and an unknown environment manifest. Change packed capacities and launch bounds with JIT forbidden, then check numerical output and graph replay. Keep workspace sizing per plan and exclude tensors, pointers and streams from the artifact memo. test_wrapper_capacity_reuses_artifact_without_disk_cache is the native SM80 detector. When asserting disk-artifact hits, clear the process memo before both cache population and reload: otherwise an earlier test can prevent the temporary cache from being populated, or a memo hit can bypass the disk counter. A test that SPIES a trace-time call (a monkeypatch wrapper around a host-side helper that runs only while the kernel body is traced) must switch the in-process compile memo off as well as the on-disk cache: set CUDNN_FRONTEND_COMPILED_CACHE_INPROCESS_MEMO=0 next to CUDNN_FRONTEND_DISABLE_COMPILED_CACHE=1 (both documented in python/cudnn/frost/compiled_cache.py). With the memo on, a plan this process has already built for the same (device, key, symbol, options) is handed out without a new trace, so the spy sees 0 calls instead of the 1 per trace it asserts; the plan need not be the test's own: when the arm's cell is the suite's default shape, an earlier test's plan serves it. _bs_dq_arm in sdpa/frost/test_sdpa_bwd_mxfp8_sm107.py sets both and is the detector.

A None compile sample may still occupy a positional TVM-FFI argument slot. When extending a prepared host with a staged-only operand, retain the native entry signature and delegate internally; do not assume the absent operand is removed from the exported call ABI. Exercise both native and staged routes, including a fresh-process artifact reload. For staged THD output padding, test the bounded cast/fold with physical wide output strides as well as the input staging; test_native_packed_physical_stride covers both stride and index-product overflow with allocated guard storage.

Prepared host migrations must preserve persistent compiled artifacts as well as warm execution. A dataclass passed as a Constexpr compile argument can prevent artifact export even though its runtime slots disappear. Carry only the immutable primitive configuration facts the host needs, or read the template module's configuration internally. test_block_output_artifact_reloads_in_fresh_process builds in one process, forbids JIT in another, and checks O/SF/Stats/Amax after execution and poisoned-output CUDA Graph replay. A same-process cache hit alone does not establish cross-process reuse.

Compute template_key(globals(), locals(), ...) before local imports or other local assignments. Including a helper function in locals() makes the key uncacheable even when the graph's specialization is fully static; all eight SM100/SM107 MXFP8 prepared entries must obey this ordering.

Prepared gate pointer regressions

Auxiliary TMA inputs need the same physical Int64 stride coverage as Q/O. test_quantized_gate_batch_stride_above_int32 steps two live BF16 gate batches across an 8-GiB gap. A valid decoy island at the truncated offset makes its Int32 negative control fail numerically without launching an invalid access. After graph capture, swap the gates between sigmoid saturation at zero and one: O must change, while Stats and requested Amax still describe ungated SDPA.

Optional-output flags and sample descriptors must agree at every host boundary. Cover an explicit disabled flag with a retained sample descriptor as well as an absent descriptor: compilation, argument validation, scratch initialization and tensor-operand elision must use the same effective presence decision. test_pv_bf16_no_amax_flag_with_sample_descriptor checks prepared and retained tensor entries; inconsistent decisions caused a D192 None.iterator compile failure and an output that was simultaneously required and forbidden.

For fixed-geometry staged pointer hosts, validate Q/K/V/O shape, dtype and device, and all auxiliary bindings before workspace carving or any copy/fill. A tensor compiler previously checked some of these at dispatch; raw addresses cannot. The SM80 detector test_dense_native_rejects_invalid_operands_before_launch replaces the launch function with a tripwire to check invalid operands before launch. test_sm80_revalidates_every_current_operand[short] separately checks that a short backing-storage span is rejected.

For a GEMM+GLU failure, compare the final output with both the stored GEMM intermediate and an independent dot product before attributing it to GEMM. SwiGLU pairs alternate 32-column input/gate blocks; the two operands are not halves of the N dimension. test_swiglu_failure_diagnostics.py checks that mapping, bounded failure output, and preservation of the original assertion.

Count TMA store stages in committed groups, not individual output subtiles. If one group reads two AB12 slots and one C slot, four AB12 slots and two C slots permit only two outstanding groups. Rotate each output ring by its own consumed-slot count across persistent tiles. A replay that repeatedly overwrites one output can hide an earlier corrupted store; retain distinct outputs and check every launch. test_gemm_swiglu_retained_outputs_replay covers multiple groups within a tile and persistent tile transitions, including one-group tiles that must advance the C ring at every tile boundary.

For bulk MoE COMBINE, establish ownership from the per-CTA MMA slice, not the cluster-wide instruction shape: the normal 128-row slice gives each epilogue thread one complete feature row. Stage every register drain before submitting that row, and wait for async shared-memory reads before reusing it. Reserve staging ahead of the AB rings in both the SMEM budget and descriptor-root model. Bulk and atomic fallback predicates must agree for partial feature tiles and unaligned output rows. gemm/frost/test_moe_combine_bulk.py varies routed rows, features and scores independently, checks output padding, then changes scores under graph replay and rebinds the same artifact to a smaller token count. Its segmented-scale replay cases also change group boundaries and scale values in place. Normal 2-CTA configurations retain atomics pending qualification: the SM100 NVFP4-to-BF16 case at S/N/K=1026/512/1024 reported invalid shared writes under memcheck despite passing ordinary numerics. Keep the admission regression and run test_two_cta_atomic_fallback under compute-sanitizer on SM100 before widening that gate. SM107 swapped bulk uses a separate predicate and has its own native cases; passing those does not qualify normal 2-CTA bulk. The template barrier-parity detector permits the swapped bulk specialization's guard around the exact register-decrease prologue only. MMA operations and barrier calls, predicates and ordering still compare without that exception.

Independent page tables need independent observed-span checks and Int64 stride slots in the prepared host. Test distinct K/V page values and layouts, then rebind allocations and mutate table values under capture replay. Preserve the shared-stride constraint for a host whose ABI still has only one stride pair.

A shared kernel imported as an ordinary module has no template-loader digest. Calling template_key there otherwise returns None and silently bypasses persistent caching. Give the module a source identity and require fresh-process reload of the whole chain, including split combine and both pointer/tensor calling conventions; forbidding JIT only around the attention kernel misses it.

For staged pointer launches, validate aliases against the original caller operands before replacing them with workspace views. The core binder only sees gathered buffers and otherwise misses an Amax scalar or block-scale SF output aliasing the original Q or O. test_staged_amax_alias_checks_original_operand and test_staged_sf_alias_checks_original_operand intercept copies to detect these aliases before any kernel is launched. Reuse the SF binder on original facts so its full atom span and output/input alias rules remain consistent.

For staged copies on multiple GPUs, resolve an omitted launch stream on Q's device and keep that device context active through gather, launch and scatter. A nondefault stream on Q's device must remain authoritative even when another CUDA device is current; restore the caller's device after execution. In multi-GPU tests, check the operand device's architecture before allocating or launching on it. A module-level marker only checks the initially current device; it cannot admit a second target on a heterogeneous machine. Allow a different architecture on the caller's current device when testing context restoration.

When retiring a fake-tensor builder, move negative guards to the live cute.runtime.make_fake_tensor and make_fake_compact_tensor constructors. Patching a deleted helper with raising=False proves nothing. Keep direct SASS and split-partial tests on the production pointer entry, including partial-output inspection before combine.

When retiring a compiler entry, audit its standalone _main() as well as adapter and test callers. A leftover unqualified compile(...) silently resolves to Python's builtin after the definition is deleted. Execute the actual CLI with its replacement compiler intercepted and assert that the prepared entry is called; import-only checks cannot catch this failure.

Wrapper coverage after workspace migrations

After a kernel consolidation, trace each public wrapper to the kernel it actually compiles before porting a correctness fix. A conflict-free change to an old kernel can leave the public route unfixed. Run the regression through the public wrapper and cover the shared kernel's other callers as well.

AMAX reset probes must keep allocation synchronization from hiding a stream race. test_swiglu_wrapper_outputs_and_amax_use_launch_stream preallocates a live poisoned AMAX buffer, retains the real fill kernel, and delays the ambient stream before launching a warmed plan on a different stream. A misplaced late reset overwrites the actual reduction with -inf. Check cold/warm calls, side and default launch streams, output allocation streams, and caller restoration. Include workspace allocated during a cold compilation: moving output allocation to the launch stream alone can leave that new allocation on the ambient stream.

When a prepared adapter starts requiring caller workspace for an existing layout, test every public convenience wrapper that constructs it. Adapter checks with explicit workspace cannot detect a wrapper that still omits it. Exercise BHSD-contiguous and padded conversion inputs plus the compact control, including plan-cache reuse and a non-default current stream. The allocating wrapper must obtain scratch_workspace_bytes() and pass per-call scratch on the input device and actual launch stream; a zero-workspace plan should keep its allocation-free path. Test an explicit stream different from the ambient stream: a non-default current stream alone cannot expose premature scratch reuse. test_wrapper_scratch_survives_explicit_stream_consumer warms the real SM80 wrapper, intercepts execution with a bounded byte write, and checks live ambient- stream allocations while that consumer is pending. Prewarm the churn allocator pool too: a fresh cudaMalloc can synchronize away the intended overlap. The SM120 detector is TestStagedSm120Wrapper in test/python/sdpa/frost/test_sdpa_fwd_dsl_sm120.py.

A fixed-layout wrapper cache must include every input stride used by the prepared plan. Same-shape calls can alternate compact, padded and permuted storage; a shape-only cache reuses an incompatible native plan. test_wrapper_cache_distinguishes_current_input_strides exercises three SM80 layouts and returns to the first one to verify both separation and reuse. When padding Q/K to a vector width, retain the original attention scale and check non-multiple-of-eight widths against an independent reference.

When a test is re-exported from another module, the source module's pytestmark does not follow it. A subprocess-based GPU test must check architecture in the parent before spawning; a pytest.skip in a plain Python child exits nonzero. Keep genuine child failures failing on supported devices. The half SDPA artifact reload detector is re-exported through TestStagedHalf and covers this boundary.

Cross-architecture compile probes must select their target explicitly, before the DSL is imported in a fresh subprocess, and override an inherited CUTE_DSL_ARCH. Otherwise a Blackwell/Rubin probe can lower for the runner's SM80 GPU and fail in NVVM despite a valid kernel. Disable the compiled cache so the probe really compiles. test_rubin_profile_traces_or_names_the_missing_dsl_intrinsic can be run with CUTE_DSL_ARCH=sm_80 to check target isolation; only its named missing-intrinsic error is a skip, while other compiler failures remain failures.

Gate layout admission must share the adapter's TMA predicate, including batch stride alignment for B > 1. A valid head/sequence pitch cannot compensate for an unaligned batch pitch. test_gate_batch_stride_alignment covers FP8, half and float element widths and the non-stepped singleton-batch control.

Fusing standalone RoPE table preprocessing must preserve the FP32 conversion before sin/cos and full-range trigonometry. Approximate PTX sin/cos is not a replacement for Torch's large-angle range reduction. The exact-output detector test_rope_table_large_angles_and_special_values includes large/small finite angles, signed zero and nonfinite inputs. Keep real Int64 stride and product overflow checks on the angle input, with wrapped addresses inside allocated guard storage, plus changed-angle replay and fresh-process artifact reload.

A CuTeDSL cutlass.Array scalar index is a flat element offset; even a one-element tuple takes that path. For non-contiguous rank-one prefixes, form the element offset explicitly in Int64 before indexing. A contiguous-only metadata test misses this. sdpa/torch/test_varlen_metadata.py checks strided prefixes, changed prefixes under replay, physical prefix strides above 2**32, and smaller strides whose index product overflows. Its provider tests pin real backend and FROST graph plans; SM80 standalone THD support is not evidence of SM80 THD graph eligibility.

Packed wrapper initialization belongs in the existing compiled host: a separate Python launch can erase the host savings from fusing device clears. Preserve the wrapper's zeroed capacity holes/tails separately from direct graph outputs and MHA dK/dV, whose unwritten storage must remain untouched. Check runtime word counts above 2**32, including half-tensor extent-to-byte conversion, with physical guard storage and poisoned tail probes. test_sdpa_sm80_packed_init.py covers these cases. A stream_context(None, device) is deliberately a no-op; it does not select the operand device. Packed wrappers must guard that device for both compilation and pointer launch and restore the caller. Test a foreign current device on both operand GPUs, with default and explicit streams; validate outputs after the call.

Same-dtype to(dtype) preserves a sliced input's strides, and reshape can preserve them too. When a wrapper declares compact lengths or sinks to the graph, explicitly normalize both layout and pointer alignment before binding. sdpa/torch/test_aux_metadata.py pins backend and FROST plans with strided native metadata and checks changed-input replay; the old backend silently read gap values while FROST rejected the inconsistent declaration. Performance comparisons must use a numerically valid baseline, such as dtype-converting inputs or an explicit compact-copy control, rather than time the wrong result.

Metadata fusion also needs a single-input timing control. A compact half sink alone already requires one Torch cast; general prepared dispatch can improve GPU time while increasing CPU enqueue cost. Pin backend and FROST separately and measure that case alongside mixed conversions and native views. test_forward_metadata_compact_sink_needs_no_compiler guards the cheap cast, including unaligned source offsets and changed-input replay.

A saved Stats tensor can have requires_grad=True inside an ordinary provider backward where grad mode is disabled. Route assertions must exercise that caller; only an active differentiable helper call needs the Torch autograd fallback. test_varlen_backward_uses_prepared_stats guards the real provider route. Selecting an operand CUDA context does not necessarily change CuTe DSL's default compiler target on heterogeneous hosts. Pass the operand architecture explicitly to the compiler and include it in the artifact key; the metadata target detector checks this alongside execution. For packed-to-padded conversion, include highly uneven lengths and heavy padding in correctness and performance comparisons; measure multiple calls per captured graph so host replay submission cannot hide a device regression.

An explicit compiler target still does not guarantee a runnable execution entry. Some DSL versions compare it with device zero's runtime architecture and leave the entry absent on heterogeneous hosts. Optional shared producers must preserve their Torch fallback in that case. Exercise the actual compiler-to-entry boundary with a missing entry, repeat with changed inputs and graph replay, and verify that ordinary compiler/launch errors still propagate rather than broadly catching them.

Preserving packed metadata storage requires both a native element type and a wide address product. Test Int32/Int64 prefixes and FP16/BF16/FP32 sinks with changed strides, broadcast views and fresh buffers; widen before multiplying the sequence/head index by the element stride. A metadata value may retain an Int32 sequence-length contract while its storage address requires Int64. test_sdpa_sm80_packed_metadata.py includes physically wide strides/products, changed-value replay and zero-copy token-major backward Stats.

Ragged offset divisors inherit the width of the declared token stride. An Int64 offset tensor alone does not check the host's scalar divisor ABI. Exercise physical Q/O token strides above 2^32 with two live sequences, and validate both partial and combine hosts; test_sdpa_native_ragged_decode_binding.py covers this boundary along with wide live page-table rows.

Prepared SM90 migration

A shared pointer binder must preserve the joining kernel's layout and scheduler contract. SM90 D512 serves covering dense permutations beyond the SM100 token-major predicate, but embeds its batch and dense extents in scheduler coordinates. Share the exact layout predicate between graph admission and runtime binding; reject a shrinking THD batch before a fixed-batch kernel reads its lengths. Keep the 128-byte alignment of the embedded Hopper tensor maps. test_sdpa_prepared_sm90.py checks those guards, forbids the old tensor/JIT path, rebinds fresh buffers, captures and replays changed inputs, reloads exported artifacts in a fresh interpreter, and writes physical rows beyond an Int32 stride. Singleton stride spies must inspect the prepared pointer frame, not a tensor launcher that the graph no longer calls.

Standalone THD Stats declarations can use BHS rank three while the shared packed binder also recognizes rank-three TH1. Preserve the declared axis order in buffer facts before binding; changing a tensor view during execute violates the prepared contract. test_standalone_thd_token_major_stats covers declared BHS and packed TH1/TH/flat storage with tensor-conversion methods forbidden. Include S=1: BHS and TH1 can have identical shapes, so disambiguation must also inspect their head/token strides.

A multi-kernel binder validates every final output before its first launch. Poison the partial workspace as well as O/Stats, pass a short final output, and verify that all sentinels survive rejection. Checking only final O can miss a partial kernel launched before validation failed. test_sdpa_native_split_binding.py exercises this after warmup and with fresh workspace/replay.

Square packed Stats binding

A rank-2 (H, H) Stats buffer has identical shape/strides under token-major and head-major interpretations. Preserve the plan's declared packing in both Python and native binders instead of guessing from the runtime shape. The detector is test_ragged_square_stats_follow_declared_packing: both declarations, both binders, numerical Stats and untouched-row canaries after rebinding and replay.

Asymmetric native attention bindings

A native binder must carry separate QK and V widths through Q/K versus V/O geometry, paged pools, partial workspace sizing and combine arguments. Derive host frames from each actual template signature and compare them with Python binding. test_sdpa_native_prefill_binding.py covers D192/V128 alongside the equal-width flavors, fresh storage, invalid current spans and physical wide output strides through both split and unsplit execution.

For unequal-width dK/dV reductions, a vector rejected by the device row limit still belongs to its original output interval. Do not let a masked dK vector fall through to dV with a negative index. The compact cases in test_sdpa_bwd_thd_d192_sm100.py check live gradients and untouched capacity tails after changing lengths under capture, including an all-empty replay.

A launch recorder must reconstruct the plan's actual native binder type after intercepting its entry. Hard-coding THD fails once dense prefill becomes native. Keep the existing KV-tail and fixed-shape rejection diagnostics, as exercised by test_sdpa_prepared_thd.py, when changing the binder implementation.

Physical THD offsets are element addresses for each port. A fused K/V slab or wide O token stride requires matching offset values, with an independent Int64 O offset when it exceeds Int32. Keep these fixtures consistent in test_sdpa_prepared_thd.py and test_mhas_v2.py. Compare graph execution with an independently prepared standalone API; clearing a graph's private prepared plan does not reproduce the standalone compact-prefix contract. Retain poisoned outputs and replay after rebinding for both public entry points.

Keep stream last in private CuTe host signatures and preserve the _positional_order guard. Matching named binder slots and a Python wrapper signature does not qualify a reordered entry; exercise a real compiled launch after ABI changes before considering a change to that convention.

Kernel host imports can also be dynamic module exports: prepared execution reads _host_thd_split through the loaded template module. Before removing an apparently unused import, inspect attribute consumers and compile a plan using that host. test_native_d576_compact_thd_rebind_and_capture[split2_unpacked] catches a missing D576 split-host export before launch; a source-only import check or binder fixture with a synthetic module does not exercise this boundary.

AST-based host frame fixtures must retain unannotated runtime arguments; ast.unparse(None) fails before binding. Raw positional CuTe entries require every runtime slot, including optional None pointers and scalar defaults: Python signature defaults do not shorten the exported FFI frame. Keep the packed combine direct-call and fresh-process artifact tests synchronized with that frame. A graph-lowered adapter carrying physical origins is not a public standalone plan; construct the standalone API independently and retain its own workspace through capture replay, as in test_paged_thd_split_capture_lengths_and_stats.

Native host ABI differences across architectures

Do not infer a dense host signature from another architecture. SM107 D256 has no page-table or partial-output arguments. Native binding may omit those slots only for a non-paged, unsplit plan; other required arguments must still reject at construction. test_sdpa_native_arch_binding.py compares actual SM107 host frames with Python binding, rejects incomplete signatures, and exercises real SM103/SM107 storage rebinding, overrides, capture replay and physical output strides above 2**32. A host-only frame test cannot substitute for those device addressing checks.

SM120 split forward writes half partials through o_ptr, while SM100 writes FP32 partials through a separate o_partial_ptr. Native binders must derive workspace byte offsets from the actual partial dtype and require that extra host slot only for FP32 partials. Compare both main and combine frames with the actual host signature, then run output/Stats numerics and changed-input replay for FP16 and BF16; test_sdpa_native_sm120_binding.py is the detector.

Native backward binding

Exercise the actual standalone adapter as well as the prepared binder. A lower-level frame test can miss an incompatible metadata container built by the adapter. test_shared_standalone_adapter_uses_current_native_storage calls that boundary with the native and Python binders, optional external delta, changed storage and invalid current inputs; the native geometry transport must accept the adapter without falling back to Python.

Derive the runtime frame from each actual host signature: SM80 has a partial that binds four launch bounds, SM107 dense and THD have different optional slots, and some chains use one scale while others use two. Compare native and Python frames, including absent slots and graph length form, before GPU checks. test_sdpa_native_bwd_binding.py covers these contracts and rejects changed storage after warmup. Keep producer shape separate from explicit overrides; only overrides must equal the fixed graph geometry. Run the existing prepared backward replay, auxiliary-output, artifact-cache and physical wide-address suites as well as the new native-route tests.

For a fixed backward host, gradient overlap uses the declared address sets and current pointers, including when a graph carrier is a noncontiguous view. The host does not consume that carrier's strides. Reinterpreting its layout only in the overlap guard changes the graph contract and can reject legal interleaved outputs. test_gradient_overlap_uses_declared_graph_addresses checks both acceptance and rejection without overrides; explicit mismatched geometry overrides must still fail before launch.

SM90 native binding retains the host's natural-unit scale slot, including zero and negative specializations. Its dense layout predicate covers permutations; THD preserves fixed batch and 128-byte tensor-map workspace alignment. Derive frames from the actual host, and keep standalone BHS Stats disambiguation at S=1 in metadata. test_sdpa_native_sm90_binding.py checks these distinctions plus fresh-storage validation, graph/standalone routes, changed-input replay and physical Int64 strides and products across input, output and Stats ports. For THD head-major Stats, widen the head stride while retaining token stride one; use enough heads that a narrow stride product also overflows. A dense Stats probe or a widened THD token stride does not cover this address path. When a recorder replaces a prepared spec's callable, reconstruct any native binder that retained the original entry; otherwise the spy never sees the launch.

For direct split SDPA workspace checks, intercept allocation at the public API.execute boundary as well as the prepared binder; a lower-layer-only counter misses allocations in the adapter. Keep a missing-workspace rejection with unchanged output sentinels, plus caller-workspace correctness and replay. test_standalone_split_requires_caller_workspace_without_allocating covers both Python and native binding and the public Torch wrapper's allocation ownership.

FP8 native binders retain the per-operand element width: FP8 Q/K/V can write half or FP8 O, while split attention writes FP32 partials. Compare the complete main/combine frame and revalidate scalar dtype, alignment, storage span and aliasing on every call. Omitted identity scales live in caller scratch and are initialized on the launch stream after validation. Keep existing rejection messages as well as the no-launch assertion; the native FP8 and prepared paged regressions check both. D512 split is served by the standalone API but is still outside the graph split capability row; do not infer graph admission from a binder-only frame check.

Fixed-layout native forward binding must preserve caller-specific carrier rules. SM80 graph operands are raw storage under graph declarations; standalone Stats may be flat, and bias uses the first contiguous [H,SQ,SKV] plane. Keep these contracts distinct from explicit geometry overrides. The actual-host frame and GPU detectors are test_sdpa_native_sm80_binding.py; the existing prepared SM80 suite checks physical Int64 addressing and RoPE declarations decline before compilation.

Per-tensor FP8 split binders must use the selected host's partial dtype, not the final output dtype: SM120 writes half partial slabs even when final O is FP8; SM100 and the split-capable SM107 flavors write FP32 partials. Check the actual host ABI, byte offsets and the final combine scale together. test_sdpa_native_fp8_arch_binding.py covers both contracts against Python binding, rejects overlapping scalar scratch, and replays changed device scales.

Prepared packed Stats head strides

lse_ext is a head stride for packed head-major Stats, not only a sequence length. Keep its host annotation and compile-time fake Int64 on every half, FP8 and MXFP8 entry, including decode and split hosts. A later cast cannot repair rejection at the FFI boundary. Physical tests must pass a carrier that exposes the complete padded head slabs; increasing its backing allocation alone does not enlarge the producer-observed span. Poison addresses reached by signed-32-bit narrowing and test both a wide stride and a narrow stride whose head product overflows. test_sdpa_prepared_stats_int64.py checks real stores and changed-input capture replay. Use full multidimensional indexing for these global stores: slicing an Array with an Int32 head index can narrow an Int64 stride inside the DSL subview helper before the final scalar store.

Native THD FP8 must validate each operand using its own element width and reject current scalar/workspace aliases before identity initialization, including empty Q. An empty Q still clears the current Amax_O word. Frame recorders must rebuild type(spec.native) after replacing spec.fn, since THD and dense retain different entries. test_sdpa_native_thd_fp8_binding.py checks these contracts against the actual host signature. When patching a newly allocated pybind tuple, move its unique ownership; an additional owning cast makes PyTuple_SetItem reject it. When an ordered graph test substitutes the independent Python binding oracle, also clear the plan's retained native ordered entry. Replacing spec.native alone does not change that entry and would silently compare native execution against itself. The measured-empty-workspace test exercises both transports. Keep launches and recorder consumption such as frames.pop() outside assertions; assert on their saved results so optimized Python cannot remove test state changes.

MXFP8 split partials follow their own host ABI: SM100 D512 and SM107 MXFP8 write half partials, even though other FP8 flavors on those architectures use FP32. Exercise both partial widths with complete main/combine frame comparison; test_sdpa_native_mxfp8_binding.py checks this along with opaque SF byte storage. Compute SF size and address products in checked Int64 before binding pointers.

A measured zero-byte workspace is not an unknown-capacity raw address. Use None for unknown capacity (C++ std::optional), never a numeric sentinel, and reject every observed capacity below required scratch, including zero, before metadata, identity or Amax writes. Test nonnull zero-extent exchange and fallback views with safely oversized backing storage, both mapping and ordered execution, and valid-buffer/raw-pointer recovery. The FP8 THD native suite exercises this with both binders and empty Q. Changing this shared contract also needs non-SDPA consumers: GEMM and linear attention share Workspace.over, including Python views and the native carver. core/frost/test_workspace_capacity.py covers raw-address recovery, measured zero rejection, exact bounds, nested tails and both execution adapters.

Padded THD Stats binding is metadata-only in both implementations. Execution initializes the declared Stats region after every binding and scalar check, even when Q has zero capacity. Cache the fill geometry at prepare time; retain current pointer, span, device and stream validation on each call. Cover the actual FlashInfer-style BSH layout, non-self-inverse permutations, gaps and physical wide stride/products with changed-input replay. test_sdpa_native_padded_stats_binding.py exercises these boundaries and proves that invalid metadata causes no initialization or attention launch.

Automatic handle caches

Indexed execution must preserve the graph's selected plan throughout binding, not just restore it after returning. Coordinate two prebuilt indexed calls at stream resolution and assert which plan each actually executes, plus the retained selection after both finish. core/test_dispatch.py::test_indexed_execution_keeps_each_callers_plan is a CPU-only detector; shared backend/FROST numerical and capture checks remain in core/graph/test_execute_overload.py.

Automatic cuDNN handle caches must isolate both device and calling thread; re-streaming a process-global handle races otherwise. Check A→B→A device reuse, coordinated threads with distinct streams, and destroy/recreate without exiting the process. core/graph/test_wrapper_graph.py covers the fluent wrapper's cache.

For caches spanning devices, check cleanup under a different current device and verify that a failed release still permits other handles to be released and the failed one to be retried. Detectors: test_auto_handle_cleanup_uses_creation_device and test_auto_handle_cleanup_retries_failed_handle.

Native block-scaled SDPA output

NVFP4 producers already expose byte-slot geometry after observation. Normalize only the packed dtype at the native binding boundary; do not halve shape or strides again. Output SF storage must cover the compiled atom padding, even when logical token-major dimensions omit it. Validate its device, byte dtype, physical layout, observed capacity and aliases before any initialization. Check both a physical SF plane stride above 2**32 and a smaller stride whose plane-index product exceeds that bound, with poisoned wrapped addresses. test_sdpa_native_block_output_binding.py compares actual host frames and exercises current storage, ordered/mapping calls and changed-input replay.

Standalone THD length counts

Graph THD binding can accept a smaller effective batch than the prepared maximum; standalone quantized declarations require the exact Q and KV carrier counts. Exercise independent length/prefix forms, shorter and longer carriers on each side and both sides, and rejection before output/workspace writes. Restore valid carriers and replay the same plan to verify rejection does not corrupt it.

Native SDPA output gates

The fused sigmoid gate changes O, while Stats and Amax belong to the preceding SDPA node. Compare both gate extremes after capture, and verify those ungated outputs stay unchanged. Rebind gate storage and physical layout independently of Q/O; cover actual >32-bit strides and batch-index products, with opposite gate values at the wrapped address so narrowing fails numerically.

Standalone prepared device contexts

A valid default-stream handle (including zero) does not identify the operand GPU. Direct standalone adapters must select the plan device and establish its CUDA context before entering a prepared host, then restore the caller even on validation failure. Exercise both the default sentinel and a side stream with a different GPU current; include changed-input capture replay. The detector is test_direct_standalone_with_another_device_current in the SM100 backward staging suite; it exercises both standalone adapters with a real host. A test that intercepts entry before launch provides a safe RED control for this bug.

FROST regression filenames must match the CI -k frost selection. Verify collection with that filter.

When pinning append-only records, assert the historical field segment or a frozen positional prefix, written explicitly rather than derived from the current dataclass under test. A last-N-fields check over the entire current record rejects legal future appends. Source-call pins must distinguish positional operands from appended keyword arguments (use the AST); a final keyword does not move the last positional operand. The SDPA field-order and FP8 THD fold pins exercise these cases.

Native SM80 packed wrapper widths

The covering MMA flavor and the logical QK/V widths are distinct. Keep each logical load bound when removing wrapper padding; marking a partial Q/K width as even reads poisoned head slack. test_sdpa_sm80_thd_native_widths.py checks independent widths, gapped strides, original input pointers, native kernel counts and changed-input/prefix replay. Its graph cases include QK<V in dense and THD. Compact head-major Stats pitch is the dynamic packed capacity; preserve that symbolic pitch when keying a compiled artifact. Fixed inter-head padding is a separate stride, not additional logical rows or required final-head storage. The wrapper capacity-reuse tests forbid compilation after capacities change; test_sdpa_sm80_native_wrappers.py retains fresh-process artifact checks while forbidding normalization and rejecting unsupported inner strides before launch. Its singleton-Stats case uses T=H=1 with zero, compact and padded head pitches: both packings classify the same element, but the token-major binder requires exact strides, so the wrapper must retain the head-major ABI for unused pitches.

FilesExpand file tree

AGENTS.md

Latest commit

History