Sync with Microsoft ONNX Runtime - 07082026 - #1250
Open
ai-fw-intg wants to merge 15 commits into
Open
Conversation
…icrosoft#31635) ## Description Extends the CUDA plugin EP packaging pipeline with Windows ARM64 support, replaces the shared x64/ARM64 architecture lists with per-platform lists that reflect the GPUs each package actually targets, and adds two build-time knobs for controlling binary size. The plugin `.so`/`.dll` is currently very large (~720 MB uncompressed on the CUDA 12.8 Linux leg, ~90% of which is `.nv_fatbin`), and the architecture list is the single biggest lever on that number, so it needs to be tuned per platform rather than shared. ## Summary of Changes ### Windows ARM64 packaging | File | Change | |------|--------| | `plugin-cuda-pipeline.yml` | Add `build_windows_arm64`; rename `invalidAArch64Config` to `invalidArm64Config` and extend it to cover Windows ARM64, since NVIDIA only ships Windows-on-ARM CUDA for 13.x | | `plugin-cuda-packaging-stage.yml` | Add `build_windows_arm64` and `arm64_cuda_version` (13.1), wire the ARM64 stage and its artifacts into NuGet and Foundry Local zip packaging | | `plugin-win-cuda-stage.yml` | Add an `arm64` arch path: ARM64 agent pool, `win-arm64/` CUDA SDK blob prefix, separate cuDNN folder, native ARM64 toolset | ### Per-platform CUDA architecture lists `cmake_x64_cuda_archs` / `cmake_arm64_cuda_archs` are split into four independent lists, since Windows x64, Linux x64, Windows ARM64, and Linux aarch64 serve very different GPU populations: | Parameter | CUDA 12.8 | CUDA 13.x | |------|------|------| | `cmake_windows_x64_cuda_archs` | `61,75,86,89,120` | `75,80,86,89,120` | | `cmake_windows_arm64_cuda_archs` | n/a | `120,121` | | `cmake_linux_x64_cuda_archs` | `75,80,86,89,90,120` | `75,80,86,89,90,120` | | `cmake_linux_aarch64_cuda_archs` | n/a | `89,90,100,103,120,121` | Notable decisions: - **`120-virtual` dropped everywhere.** The `compute_120` PTX measured 176 MB, 27% of the entire `.nv_fatbin` — by far the most expensive single entry. It also cannot carry the NVFP4 kernels, which are only valid as real `sm_120a` (`cuobjdump -ptx | grep -c e2m1x2` returns 0), so it was paying full price for partial coverage. - **Linux aarch64 targets the platforms that actually exist on ARM**: GH200 (`90`), GB200 (`100`), GB300 (`103`), DGX Spark GB10 (`121`), plus discrete cards in ARM chassis (`89`, `120`). `103` is required alongside `100` because ORT normalizes `100` to `100a-real`, and `a` targets are locked to their exact SM. - **`75` dropped from Linux aarch64** — Turing was never paired with an ARM host in practice. ### CUDA architecture normalization | File | Change | |------|--------| | `cmake/external/cuda_configuration.cmake` | `ARCHITECTURES_WITH_ACCEL`: add `103` and `121`, drop `101` (removed by NVIDIA after CUDA 12.9). Without this, `103` and `121` would be built as plain targets and would silently lose the CUTLASS block-scaled/TMA kernels, which are gated on `__CUDA_ARCH_FEAT_SM1xx_ALL`. | ### Build size controls Two new pipeline parameters, both plumbed through the packaging stage to all four platform stages: | Parameter | Default | Effect | |------|------|------| | `enable_cuda_fatbin_size_compression` | `false` | Sets the new `onnxruntime_CUDA_FATBIN_COMPRESS_SIZE` cmake option, forcing `-Xfatbin=-compress-all -compress-mode=size` on the CUDA 12.8 leg. CUDA >= 13.0 already does this unconditionally, so the parameter only changes 12.8. | | `enable_fpa_intb_gemm` | `true` | Sets the existing `onnxruntime_USE_FPA_INTB_GEMM` cmake option. fpA_intB GEMV/GEMM is ~141 MB of device code (22% of `.nv_fatbin`), second only to flash attention. | `onnxruntime_CUDA_FATBIN_COMPRESS_SIZE` fails configuration on CUDA < 12.8 rather than silently passing an unsupported flag to nvcc. Windows composes these via `FatbinCompressOption` / `FpaIntBGemmOption` job variables appended to the `build.py` invocations, mirroring the existing `$(TelemetryOption)` pattern. Linux composes them into `EXTRA_CMAKE_DEFINES`, which `build_cuda_plugin_package.sh` already forwards. ### Packaged binary hardening and verification | File | Change | |------|--------| | `cmake/onnxruntime_providers_cuda_plugin.cmake` | Compile `onnxruntime_providers_cuda.rc` into the plugin DLL on Windows so the packaged binary carries version info; set `SKIP_BUILD_RPATH` on Linux so the build machine's CUDA path is not embedded in a binary that ships as-is | | `plugin-linux-cuda-stage.yml` | Fail the build if the plugin `.so` has an empty `RPATH`/`RUNPATH` component or a hard-coded CUDA path | | `plugin-win-cuda-stage.yml` | Fail the build if the plugin DLL is missing required version-info properties | ## Testing - Pipeline changes are validated by running the CUDA plugin packaging pipeline. Both new parameters default to current behavior (`enable_cuda_fatbin_size_compression: false`, `enable_fpa_intb_gemm: true`), so a default run produces the same build flags as before this PR aside from the architecture list changes. - The cmake `-compress-mode` selection logic was verified in isolation across four combinations: | Toolkit | Option | Result | |---|---|---| | 12.8 | OFF | `-Xfatbin=-compress-all` | | 12.8 | ON | `-Xfatbin=-compress-all -compress-mode=size` | | 13.1 | OFF | `-Xfatbin=-compress-all -compress-mode=size` | | 12.6 | ON | configure-time fatal error, as designed | - The new RPATH and DLL version-info checks are self-verifying: they fail the packaging stage rather than publishing a bad artifact. ## Motivation and Context The primary consumer is Foundry Local (vision, audio, and mostly LLM models), which ships this plugin to end-user machines, so download size matters directly. Trade-offs worth flagging for reviewers: - **`-compress-mode=size` raises the minimum driver** to the CUDA 12.4 level (Linux >= 550.54.14, Windows >= 551.61); older drivers cannot decompress the fatbin at all. It also increases module load time (measured ~0.8 ms to ~4.3 ms for a ~6.5 MB SASS module) and adds a few percent to nvcc time. This is why the parameter defaults to `false` and is opt-in per run. - **The architecture lists are all `-real` with no virtual entry**, so any GPU whose compute capability is not explicitly listed gets `cudaErrorNoKernelImageForDevice` (209) instead of falling back to JIT. This is deliberate given the PTX cost, but it means new architectures must be added explicitly. - **`enable_fpa_intb_gemm: false` is not yet validated end to end.** The fpA_intB path is opt-in at run time via `ORT_FPA_INTB_GEMM` / `ep.cuda.fpa_intb_gemm`, but `matmul_nbits.cc` forces it on whenever weights are prepacked, independent of that flag. The fallback path should be exercised before shipping a package built with this off. ## Checklist - [x] No breaking changes to default pipeline behavior (both new parameters default to existing behavior) - [x] cmake option gated on toolkit version with an explicit error rather than a silent no-op - [ ] Tests added/updated — not applicable; changes are build/packaging configuration
…nd head sink (microsoft#29912) ### Description `PagedAttention` is ORT's continuous-batching attention operator, but on `main` it only supports FP16/BF16 caches with RoPE and softcap, has no paged decode kernel, and forces a device→host synchronization on every node on every step (which makes it uncapturable by CUDA graphs). This PR brings it to feature parity with `GroupQueryAttention` for the popular LLM families and adds the paging and latent-cache primitives that serving frameworks need, **additively** — every model valid under the shipped `com.microsoft::PagedAttention` opset-1 schema keeps working unchanged. The design rationale, the compatibility invariant, and the alternatives that were considered and rejected are written up in the new design document [`docs/contrib_ops/cuda/paged_attention.md`](docs/contrib_ops/cuda/paged_attention.md); the section numbers referenced below point into it. ### Summary of Changes #### Schema (`bert_defs.cc`, `docs/ContribOperators.md`) All additions are trailing optional inputs, new attributes whose defaults reproduce current behavior, or widened type constraints (§4). | New input | Idx | Purpose | |---|---|---| | `slot_mapping` | 10 | Explicit per-token cache slot, so the scheduler owns placement instead of the kernel re-deriving it (§5) | | `head_sink` | 11 | Attention sink / smooth softmax, matching GQA (§6) | | `q_norm_weight` / `k_norm_weight` | 12, 13 | Fused QK-RMSNorm (Qwen3, gpt-oss) (§7) | | `k_scale` / `v_scale` | 14, 15 | Per-tensor or per-channel dequantization scales for a quantized cache (§8) | | `attention_metadata` | 16 | Optional CPU input carrying *replay-wide upper bounds* `[max_query_len, max_kv_len]`, which removes the per-node per-step D→H sync (§4.7) | | New attribute | Default | Purpose | |---|---|---| | `qk_norm_epsilon` | `1e-6` | Epsilon for the fused QK-Norm | | `k_quant_type` / `v_quant_type` | `NONE` | `NONE` \| `PER_TENSOR` \| `PER_CHANNEL` | | `k_cache_dtype` / `v_cache_dtype` | `""` | Logical cache element type, named after the ONNX type it denotes | | `kv_cache_layout` | `SEPARATE` | `SEPARATE` \| `LATENT` (absorbed MLA: one cache, no `value`/`value_cache`) | | `v_head_size` | `0` | Narrower V head, `LATENT` only (DeepSeek-V3 uses 576/512) | | `rotary_offset` | `0` | Applies RoPE to `[rotary_offset, rotary_offset + rotary_dim)` so MLA can rotate only the positional suffix | `key_cache` / `value_cache` move from `T` to a new `T_CACHE` constraint (`float16`, `bfloat16`, `int8`, `float8e4m3fn`), and `value_cache` / `value_cache_out` become optional so a `LATENT` node can omit them. Shape inference now takes the cache element type from inputs 3/4 rather than from `query`, which was wrong for a quantized cache. #### CUDA kernels (`paged_attention_impl.cu`, `paged_attention.cc/.h`, `paged_attention_helper.h`) - **Paged decode kernel** (`LaunchPagedDecodeAttention`) — split-KV, block-table-aware decode with native head-sink, softcap, sliding-window and on-the-fly cache dequantization. - **XQA paged decode** (`onnxruntime/contrib_ops/cuda/bert/xqa/`) — TensorRT-LLM's XQA kernels extended to the paged block layout: 8 new translation units `xqa_paged_{fp16,bf16}_{int8,fp8}_{64,128}.cu` plus a shared paged loader. Selected for quantized-cache decode. - **Quantized paged cache** — `ReshapeAndCache` quantizes on write; all read paths dequantize with `k_scale`/`v_scale` under `PER_TENSOR` or `PER_CHANNEL` granularity. - **`ApplyHeadSink`** — exact post-hoc LSE rescale (`1/(1+exp(s_h − lse))`) applied *after* the quantized/unquantized branch, so no backend can silently drop the sink (§6). - **`QkNormRotaryTNH`** — fuses QK-RMSNorm, RoPE (with `rotary_offset`) and the packed-QKV unpack into one pass. - **Absorbed MLA** (`PagedLatentAttentionKernel` / `LatentAttention`) — single latent cache, V read as the leading `v_head_size` channels of the same row that supplies K (§12). - **CUDA-graph safety** — backend dispatch, grid sizing and workspace extents now come from static shapes and the `block_table.shape[1] * block_size` capacity bound; per-step quantities are read on device. The unconditional `cudaStreamSynchronize` is gone from the capturable path (§4.7). - **`int8`→`fp16` conversion fast path** (`xqa/utils.cuh`, `cvtS8x4ToF16x4`) — replaces a scalar `I2F` loop with a `prmt` + `sub.f16x2` sequence (5 full-rate instructions per 4 elements, bit identical). Shared with the non-paged GQA loader. - Kernel registration is now `<T, T_CACHE>`-typed; FP8 combinations are behind `USE_FP8_KV_CACHE && !DISABLE_FLOAT8_TYPES`. #### GQA bug fix (`flash_api.{h,cc}`, `group_query_attention_impl.cu`) `mha_fwd` had `constexpr void* head_sink = nullptr;` hardcoded inside it, and `FlashAttentionAndQuantizeKV` — the *only* GQA prompt path taken when the KV cache is quantized — called it. So for gpt-oss with an INT8/FP8 KV cache, the attention sinks were silently dropped for the entire prompt on every layer while decode stayed correct. `mha_fwd` now takes `head_sink` and GQA forwards it. Op-level prefill error drops **0.074829 → 0.000122**. On gpt-oss-20b (int4 body, INT8 per-channel KV), MMLU-Pro-800 goes **0.6175 (494/800) → 0.7200 (576/800)**. Existing CI missed this because `atol["int8_fp16"] = 1e-1` in `test_gqa.py` is ~800× wider than the post-fix error. #### Tooling and docs - `symbolic_shape_infer.py`: correct output width for packed-QKV and `LATENT` nodes, and cache outputs typed from the cache inputs. - New `docs/contrib_ops/cuda/paged_attention.md` design document; regenerated `ContribOperators.md` and `OperatorKernels.md`. ### Testing `test_paged_attention_cuda.py` grows from a smoke test to ~2k lines / 198 cases, with new suites for features (`slot_mapping`, head sink, QK-Norm), quantized cache (int8/fp8 × per-tensor/per-channel), the paged decode kernel, the XQA decode path, `attention_metadata`, and MLA — each against a PyTorch reference. ```bash python onnxruntime/test/python/transformers/test_paged_attention_cuda.py # 198 passed python onnxruntime/test/python/transformers/test_gqa.py -k xqa # 714 passed ``` The GQA suite is included because the int8 conversion fast path is shared with the non-paged loader. **Backward compatibility.** A node with none of the new inputs/attributes takes exactly the code path it does today: `T_CACHE == T`, `value_cache` present, `kv_cache_layout == SEPARATE`, all quantization `NONE`. The compatibility invariant is stated normatively in §4.2. ### Experimental Results Measured on gpt-oss-20b, H200. E2E numbers are driven through onnxruntime-genai; the CUDA-graph and engine-side plumbing they depend on is **not** part of this PR — they are included to show what the operator-side changes enable, not as a claim about this diff alone. #### Paged decode kernel (isolated, `nh=64 / kvh=8 / hs=64 / block=256`) XQA on/off at `b=8, ctx=4096`, per decode call: | cache | before | after | |---|---|---| | int8 `PER_TENSOR` | 2315 µs | **122 µs** | | int8 `PER_CHANNEL` | 2339 µs | **128 µs** | | fp8 `PER_TENSOR` | 1633 µs | **57 µs** | | fp8 `PER_CHANNEL` | 1566 µs | **57 µs** | Before XQA the quantized paths were ~2.5× *slower* than fp16 — the generic kernel was the bottleneck, not the KV bytes. The `cvtS8x4ToF16x4` conversion path then closes the residual int8-vs-fp8 gap (nsys median, SASS goes from 3928 to 3592 instructions with 192 → 0 `I2F`): | ctx | batch | int8 before | int8 after | gain | fp8 | |---|---|---|---|---|---| | 1024 | 32 | 19.91 µs | 13.60 µs | −31.7% | 12.64 µs | | 4096 | 8 | 24.32 µs | 17.41 µs | −28.4% | 16.48 µs | | 4096 | 32 | 66.62 µs | 42.66 µs | −36.0% | 41.98 µs | | 16384 | 8 | 78.11 µs | 52.58 µs | −32.7% | 49.18 µs | | 16384 | 32 | 244.71 µs | 162.27 µs | −33.7% | 176.70 µs | The int8/fp8 gap goes from up to +59% down to ≤ 7.6% (int8 is faster at the largest config), so the two cache formats can now be chosen on accuracy grounds. #### End-to-end decode throughput (mxfp4 body, INT8 KV, prompt 128 / new 256) | batch | baseline | + `attention_metadata` (no sync) | + CUDA graphs | total | |---|---|---|---|---| | 1 | 242.7 tok/s | 263.8 | **305.7** | **+26.0%** | | 8 | 796.1 | 811.5 | **870.5** | +9.3% | | 32 | 2635.7 | 2682.0 | **2879.2** | +9.2% | Including XQA, batch-1 int8 decode goes **199.2 → 305.7 tok/s (+53.5%)**. After this work attention is no longer the bottleneck at b=1 — the MoE GEMMs and 49 `MatMulNBits` nodes dominate the step, with XQA at 24 × 8.6 µs. #### PagedAttention vs GroupQueryAttention, matched models Two models built from the identical recipe (int4 body, INT8 per-channel KV, identical `num_heads`/`kv_num_heads`/`scale`/window/rotary, byte-identical weight file), differing only in the attention operator. Greedy generation on this stack is bit-reproducible (0/198 discordance across replicates), so there is no sampling noise to subtract. | | GQA | PagedAttention | read as | |---|---|---|---| | MMLU-Pro-800 | 0.7200 (576/800) | 0.7163 (573/800) | +0.4 pp, 3 questions | | GPQA-diamond | 0.6061 (120/198) | 0.6212 (123/198) | −1.5 pp, 3 questions | The two benchmarks disagree in direction and both deltas are 3 questions: **equivalent within noise.** (An apparent +8 pp advantage for paged in earlier runs turned out to be the GQA sink bug fixed above, seen from the other side.) | config | GQA tok/s | paged tok/s | delta | |---|---|---|---| | b=1, p=128, n=256 | 374.4 | **376.3** | +0.5% | | b=2, p=4096, n=256 | 684.3 | **686.6** | +0.3% | | b=8, p=128, n=256 | 1792.7 | 1683.8 | −6.1% | | b=32, p=128, n=256 | 5285.9 | 4109.5 | −22.3% | Peak device memory at matched KV capacity agrees to within 24 MiB (0.16%) from 16k to 128k `max_length` — paged costs nothing extra, and its advantage is structural (a shared pool sized to aggregate demand rather than `batch × max_length`). The b=32 gap was profiled with `nsys --cuda-graph-trace=node`: the captured graph body is at parity with GQA's eager model pass (6.220 ms vs ~6.2 ms) and the entire regression is a 2.384 ms search/sampling tail, which the onnxruntime-genai `Engine` runs once **per request** rather than once per batch. It is not attributable to this operator, and a partial engine-side fix already recovers b=32 to 4517 tok/s. ### Follow-ups (not in this PR) - `attention_bias` and `output_qk` (§10, §11) — schema slots reserved, kernels deferred. - Sub-byte (`int4` / `float4e2m1`) packed caches — attribute vocabulary reserved and rejected at validation until a backend exists (§21.4). - `.Alias(3, 1).Alias(4, 2)` on the kernel def, so a non-aliasing allocation plan fails at partition time instead of run time (§4.4). - Re-tightening `atol["int8_fp16"]` in `test_gqa.py` now that the sink bug is fixed.
### Description <!-- Describe your changes. --> Add an `ort-release-notes` skill with preset configuration, scoped path support, and a concise workflow for metadata generation and draft output. Also include a lightweight docs entry point and WebGPU EP scoped paths file. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Make it easier for AI to generate release note drafts. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Copilot-Session: 3611670d-4e92-4183-8e0e-b2cb659ab624
…t#31632) This change adds a build option for excluding the prebuilt TensorRT fused multi-head attention cubins from the CUDA Execution Provider. The option is enabled by default to preserve existing behavior; disabling it removes approximately 14 MB of embedded cubin data and leaves attention selection to the other available kernels and unfused fallback paths. ## Key Changes - Add `onnxruntime_USE_TRT_FUSED_ATTENTION`, dependent on CUDA and enabled by default. - Propagate the option to CUDA provider compilation and guard TensorRT fused-attention cubin declarations, metadata, and lookup paths. - Filter the prebuilt TensorRT fused-attention cubin sources from both the regular CUDA provider and CUDA plugin provider when the option is disabled, while retaining the shared driver wrapper needed by sparse attention. - Add Windows plugin DLL version metadata and suppress build-machine RPATH embedding for packaged Linux plugin binaries. - Exercise `onnxruntime_USE_TRT_FUSED_ATTENTION=OFF` in the Windows CUDA no-cuDNN plugin build. ## Testing Notes - `git diff --check origin/main...HEAD` passes. - The Windows CUDA no-cuDNN workflow now builds the CUDA plugin with TensorRT fused attention disabled, providing CI coverage for the opt-out configuration. - No local CUDA/Windows build was run in this environment.
This pull request adds input validation checks to prevent integer overflow issues during CUDA kernel indexing in the LayerNorm and RMSNorm CUDA operators. The main goal is to ensure that the product of `num_rows` and `norm_size` does not exceed `INT_MAX`, which could lead to incorrect behavior or crashes. Input validation for CUDA kernel indexing: * Added a check in `LayerNorm::ComputeInternal` (in `layer_norm.cc`) to return an error if `num_rows * norm_size` exceeds `INT_MAX`, preventing integer overflow during CUDA kernel indexing. * Added a similar check in `RMSNorm::ComputeInternal` (in `rms_norm.cc`) to ensure the input size does not exceed CUDA kernel indexing limits. Code maintenance: * Included the `<limits>` header in both `layer_norm.cc` and `rms_norm.cc` to support the new input validation logic. [[1]](diffhunk://#diff-ebda3d3b7054f5d14c679ebe8e6520a2c84a5a558d8fdcc0798c08e7032345feR9) [[2]](diffhunk://#diff-adebea99f15800767eb7b84d40be4873e81ca4df99ffe9dc7f849eabe0a3c563R9) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#31634) ### Description [CPU] Tighten cache_indirection shape contract in MultiHeadAttention ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…fy (microsoft#31159) ### Description Four changes to the fused NVFP4 QMoE decode GEMV. Stacked on microsoft#31154 — **review only the top four commits**; the base branch is that PR. 1. **Packed E2M1 dequantize** (`Fp4I2FConverter::decode_quad`) — decode a whole 32-bit weight word (eight codes) per step instead of one code at a time. 2. **`ORT_FP4_GEMV_DEFAULT_TILING`** — env switch to bypass the autotuner and take the default tiling, for A/B and for avoiding autotune cost in short runs. 3. **Cut memory and ALU traffic in the decode GEMV.** 4. **`kMaxProfiledExpandedRows` 8 -> 64** so MTP verify steps stay on the GEMV path. ### Motivation and Context Prior profiling established that this kernel is **ALU-pipeline bound**, not memory- or tiling-bound. On the actual Qwen3.6 decode shapes (`hidden=2048`, `inter=512`, `E=256`, `top_k=8`, bf16, SwiGLU), ncu reported for the FC1 SwiGLU-fused GEMV: > ALU 78.9%, DRAM 7.3%, occupancy 21% (register-limited) That is why the levers here are instruction-count levers. Two things were measured and explicitly **dropped** because of it: smaller `CtaN` tiling (the autotuner still picks `threads64`/`CtaN=8`; `CtaN=4` never wins because the kernel is compute-bound, not occupancy-bound), and halving scale bandwidth by storing combined scales as 1-byte e4m3 (DRAM is only ~7%, so it cannot move the needle). All numbers below: 1x H200 SXM (SM90, 132 SM, ~4.8 TB/s HBM), CUDA 13.0, Qwen3.6-35B-A3B-NVFP4 + MTP `N=3` (verify batch `M=4`). ### 1. Packed E2M1 dequantize `prmt` selects four bytes per instruction, so a 4-element magnitude lookup costs one instruction instead of four. Bit-identical to the per-element path (same magnitude tables, same sign handling). The FP4 GEMV kernel SASS shrinks ~30%, and the two QMoE GEMVs drop: | kernel | before | after | |---|---:|---:| | fc1 (SwiGLU-fused) | 33.2 µs | **26.2 µs** | | fc2 | 30.2 µs | **22.2 µs** | ### 2. Cut memory and ALU traffic — −0.46 ms/step (−5.1%) The scales of the `CtaN` columns a block owns sit `Interleave` elements apart, so for the non-interleaved ColumnMajor layout (`Interleave == 1`) the whole `CtaN`-wide scale vector is contiguous and can be fetched with one wide access instead of `CtaN` scalar ones. This matters far more than the byte count suggests: with a groupwise scale (NVFP4 `GroupSize = 16`) and `StepK = 8`, a warp's 32 lanes cover 16 distinct scale rows that are `n` elements apart, so *every* scale load touches 16 different sectors — `CtaN * 16` sectors, using 2 bytes out of each 32-byte sector. Per-kernel (graph OFF, 40 launches/step each): | kernel | before | after | |---|---:|---:| | `moe_gemv_interleaved_swiglu_kernel` | 0.956 ms/step | **0.678 ms/step** | | `moe_gemv_kernel` | 0.700 ms/step | **0.494 ms/step** | | **family total** | **1.657 ms/step** | **1.174 ms/step** | End-to-end (4 interleaved `.so`-swap reps per arm): * before: 8.973 / 9.009 / 8.992 / 9.017 * after: 8.567 / 8.508 / 8.498 / 8.583 **8.998 -> 8.539 ms/step.** No overlap between the two sets. ### 3. `kMaxProfiledExpandedRows` 8 -> 64 The fused GEMV rejects `expanded_num_rows > kMaxProfiledExpandedRows`. Qwen3.6 is top-8, so single-token decode expands to 8 rows (accepted), but an MTP verify does not: an `(N+1)`-token verify for `num_speculative_tokens = N` expands to `(N+1) * 8` rows, i.e. **up to 64 for N=7**. Those steps fell out of the window and back onto the dequantize + CUTLASS grouped-GEMM path, which re-dequantizes all 256 experts per token. The impact of that fallback is large: with the limit at 8, the 2-token verify (expanded 16) dropped MTP to **~2.4 tok/s**; raising the limit put it at **~30–55 tok/s (12–23x)**. 64 covers the `N=3` shape used today with headroom to `N=7`. ### Tests * `onnxruntime_provider_test` FP4/FP8/QMoE: 18/18 pass. * `onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py`: 22/22 pass, including new multi-token GEMV cases and a `gemv_mode="0"` dequant-fallback companion on the identical shape, so both must match the same exact dequantized reference. ### Methodology note End-to-end deltas are quoted as **ms/step** from a fixed-step measurement, never tok/s: any numerics change alters the generated sequence and therefore the MTP acceptance rate, which swamps the speed delta. Per-kernel durations are taken with CUDA graphs **off** — `nsys --cuda-graph-trace=node` inflates durations ~35% globally and up to 3.8x for large-grid kernels. > [!IMPORTANT] > `Fp4I2FConverter::convert()` gained a `PairInterleaved` template parameter in microsoft#31154. The > packed path added here assumes the **plain** nibble order (nibble `j` of the word is logical > element `j`), which is what its `prmt` selectors encode, so it is nested inside > `if constexpr (!PairInterleaved)`. Please check that guard carefully during review — applied > without it, the pair-interleaved SM80 layout would silently decode to the wrong values.
…oft#29611) ### Description Accumulate directly in output_element_t (e.g., f16 for f16 models) instead of hardcoding an f32 accumulator in the MatMulNBits wide-tile shader. **Intel Panther Lake** | | Prefill Length | Default Prefill TPS | Optimized Prefill TPS | Improvement | | :--- | ---: | ---: | ---: | ---: | | gpt-oss-20b-ONNX | 128 | 305.70 | 344.49 | 113% | | gpt-oss-20b-ONNX | 1024 | 396.50 | 429.80 | 108% | | Phi-4-mini-instruct-ONNX | 128 | 515.90 | 592.36 | 115% | | Phi-4-mini-instruct-ONNX | 1024 | 615.39 | 753.40 | 122% | [1] https://huggingface.co/onnx-community/gpt-oss-20b-ONNX [2] https://huggingface.co/onnx-community/Phi-4-mini-instruct-ONNX ### Motivation and Context See above.
…ch (microsoft#31480) ### Description When GroupQueryAttention runs with a quantized KV cache, the K cache and the V cache were dequantized by two separate kernel launches. They have identical shapes and identical per-head scale layouts, so the second launch adds a full grid setup and a second pass over the same index arithmetic for no reason. This change dequantizes both caches in a single launch. The kernel is moved into a new `group_query_attention_qdq.cuh` header and given a `2 *` grid in the cache dimension, so one block range covers K and the other covers V; the buffer pointers and scale pointers are selected from the block index. Output is bit-identical to the two-launch path -- the per-element arithmetic is unchanged, only the launch geometry differs. ### Motivation and Context This is on the decode path of speculative-decoding (MTP) workloads, where the launch is issued every layer, every step, and the per-launch fixed cost is a meaningful fraction of a short kernel. Halving the number of launches removes that fixed cost without changing numerics. Measured on H200 (SM90) with a Qwen3.6-35B-A3B MTP configuration.
Add cuda plugin ep support
Adds odd-N support to WebGPU subgroup-matrix MatMul by padding constant FP16 weights to an even row stride and caching the result.
…icrosoft#31685) ### Description <!-- Describe your changes. --> Add onnxruntime/test/providers/webgpu to plugin-ep-webgpu/paths.txt. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Missed a test path.
…ft#31683) ### Description <!-- Describe your changes. --> Announce JSEP deprecation to developers via docs in the repo. The main doc is `docs/JSEP_Deprecation.md`. Also update the JSEP to WebGPU EP migration design doc to include this initial step as well as some other clarifications. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> JSEP deprecation. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Add more paths related to cuda plugin ep to address release note skills comment microsoft#31669 (comment).
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 6, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.