Skip to content

Update CUDA to version 13.3.0#10617

Merged
smuzaffar merged 6 commits into
cms-sw:IB/CMSSW_20_1_X/masterfrom
fwyzard:IB/CMSSW_20_1_X/master_CUDA_13_3_0
Jun 15, 2026
Merged

Update CUDA to version 13.3.0#10617
smuzaffar merged 6 commits into
cms-sw:IB/CMSSW_20_1_X/masterfrom
fwyzard:IB/CMSSW_20_1_X/master_CUDA_13_3_0

Conversation

@fwyzard

@fwyzard fwyzard commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

@fwyzard

fwyzard commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

enable gpu

@fwyzard

fwyzard commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

cmsbuild commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @fwyzard for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

@cmsbuild

cmsbuild commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

cmsbuild commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53787/summary.html
COMMIT: 8eabad3
CMSSW: CMSSW_20_1_X_2026-06-09-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10617/53787/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

CMSBUILD_PACKAGE=onnxruntime;  export _CMSBUILD_BUILD_ENV_=1; export TMPDIR=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/cmsdist-tmp;  source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/rpm-env.sh; source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/pkgtools/build-system-env.sh 16;  rpmbuild  --noclean --buildroot /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/3d403218bee3a6bbf94b3a90e3dc8271 -bc --short-circuit --define '_smp_build_nthreads 16' --define '_topdir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir' --define "compiling_processes 16"  --define '_builddir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271' --define "cmsdist_directory /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cmsdist" --define "compilerv 1340" --define "cmscompilerv 13" --define "cmsos el9_amd64" --define "__bootstrap ~bootstrap" --define "almalinux_ver 9" --define "almalinux 9" --define "centos_ver 9" --define "centos 9" --define "rhel 9" --define "dist %{!?distprefix0:%{?distprefix}}%{expand:%{lua:for i=0,9999 do print("%{?distprefix" .. i .."}") end}}.el9%{?with_bootstrap:~bootstrap}" --define "el9 1" --define "dist_vendor         AlmaLinux OS Foundation" --define "dist_name           AlmaLinux" --define "dist_home_url       https://almalinux.org/" --define "dist_bug_report_url https://bugs.almalinux.org/" --define "package_vectorization x86-64-v2" --define "cmsswdata_version_link 1" --define "archfirst yes" --define "cmsBuild_bootstrap 1"   /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/SPECS/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/spec >>/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/log 2>&1
build-build-external+onnxruntime+1.26.0-3d403218bee3a6bbf94b3a90e3dc8271 failed.
Failed to build onnxruntime. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/log. Final lines of the log file:
171 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, const &, &&, false,
|                                                                                                                                                                                                                     ^
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/build/_deps/abseil_cpp-src/absl/container/internal/raw_hash_map.h:180:200: error: using template type parameter 'absl::lts_20250814::container_internal::IfRRef::AddPtr' after 'typename'
180 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, &&, const &, false,
|                                                                                                                                                                                                        ^         
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-3d403218bee3a6bbf94b3a90e3dc8271/build/_deps/abseil_cpp-src/absl/container/internal/raw_hash_map.h:180:211: error: template argument 6 is invalid
180 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, &&, const &, false,
|                                                                                                                                                                                                                   ^


@fwyzard
fwyzard force-pushed the IB/CMSSW_20_1_X/master_CUDA_13_3_0 branch from 8eabad3 to 4e5e2bb Compare June 9, 2026 23:55
@fwyzard

fwyzard commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

cmsbuild commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Pull request #10617 was updated.

@fwyzard

fwyzard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53809/summary.html
COMMIT: 4e5e2bb
CMSSW: CMSSW_20_1_X_2026-06-09-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10617/53809/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/py3-torch-cluster-cuda/1.6.3-256da426cb7b958a8355fcf01af22be7/cmsdist-pip-src/torch_cluster-1.6.3
Building wheel for torch_cluster (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch_cluster
Failed to build torch_cluster
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.ZWcyEt (%build)

RPM build warnings:
Macro expanded in comment on line 631: %{pkginstroot}/bin/*

Macro expanded in comment on line 636: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}


@fwyzard
fwyzard force-pushed the IB/CMSSW_20_1_X/master_CUDA_13_3_0 branch from 4e5e2bb to b10b055 Compare June 10, 2026 07:49
@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10617 was updated.

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10617 was updated.

@fwyzard

fwyzard commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53818/summary.html
COMMIT: 88e32d9
CMSSW: CMSSW_20_1_X_2026-06-09-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10617/53818/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

install-cms+cms-git-tools+251202.0-c4424a3c103e232b58c2aed3dbff41ee done
build-build-external+onnxruntime+1.26.0-92eddeb1b3f2986c95279595476fcca1 failed.
Failed to build onnxruntime. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-92eddeb1b3f2986c95279595476fcca1/log. Final lines of the log file:
171 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, const &, &&, false,
|                                                                                                                                                                                                                     ^
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-92eddeb1b3f2986c95279595476fcca1/build/_deps/abseil_cpp-src/absl/container/internal/raw_hash_map.h:180:200: error: using template type parameter 'absl::lts_20250814::container_internal::IfRRef::AddPtr' after 'typename'
180 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, &&, const &, false,
|                                                                                                                                                                                                        ^         
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/onnxruntime/1.26.0-92eddeb1b3f2986c95279595476fcca1/build/_deps/abseil_cpp-src/absl/container/internal/raw_hash_map.h:180:211: error: template argument 6 is invalid
180 |   ABSL_INTERNAL_X(insert_or_assign, insert_or_assign_impl, &&, const &, false,
|                                                                                                                                                                                                                   ^


@fwyzard

fwyzard commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

please test

@fwyzard

fwyzard commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

please abort

fwyzard added 6 commits June 14, 2026 20:08
Fix invalid C++ syntax in cub/device/device_transform.cuh in CUDA 13.
Backport NVIDIA/cccl#8771 from the main branch.
Drop the Volta (sm 7.0) architecture, that is not supported by CUDA 13.
Major updates:
  - support for DMA-BUF mmap backend with CUDA 13.3
  - support for AVX2, AVX-512, MOVDIR64B, NEON CPU instructions
  - support for Linux kernel 6.15+

The updates, fixes and changes in v2.5.2 can be found at
https://github.com/NVIDIA/gdrcopy/releases/tag/v2.5.2 .

The updates, fixes and changes in v2.6 can be found at
https://github.com/NVIDIA/gdrcopy/releases/tag/v2.6 .
Update CMake to the latest 3.x bugfix version, v3.31.12.
Backport cmake!12162 CUDA: Add support for cuda_std_23 for nvcc 13.3+.
Backport microsoft/onnxruntime#28736
Patch Abseil to workaround NVIDIA bug #6302392
Do not include <ciso646> in c++20/c++23 mode
@fwyzard
fwyzard force-pushed the IB/CMSSW_20_1_X/master_CUDA_13_3_0 branch from f4eda55 to 7463906 Compare June 14, 2026 18:08
@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10617 was updated.

@fwyzard

fwyzard commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

please test

@fwyzard

fwyzard commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@smuzaffar @mandrenguyen @ftenchini some non-gpu tests are still running, but all NVIDIA gpu tests passed, finally this looks like it's ready to be merged.

@smuzaffar

Copy link
Copy Markdown
Contributor

please test for CMSSW_20_1_CPP23_X

lets see if this works for gcc14/c++23

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53940/summary.html
COMMIT: 7463906
CMSSW: CMSSW_20_1_X_2026-06-14-0000/el9_amd64_gcc13
Additional Tests: GPU,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10617/53940/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53940/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-33cbca/53940/git-merge-result

Comparison Summary

Summary:

  • You potentially removed 170 lines from the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3267025
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3267007
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
  • Checked 191 log files, 159 edm output root files, 44 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

  • You potentially removed 25 lines from the logs
  • Reco comparison results: 159 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 19681
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 147325
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

  • You potentially removed 25 lines from the logs
  • Reco comparison results: 140 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 21715
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 145291
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_T4 Comparison Summary

Summary:

  • You potentially removed 27 lines from the logs
  • Reco comparison results: 164 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 21517
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 145489
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 37.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 37.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step4 max memory diff 35.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 37.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 37.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 37.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstOnGPUIters01TrackingOnly step2 max memory diff 35.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstOnGPUIters01TrackingOnly step3 max memory diff 34.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 36.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 34.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff -230.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 35.2 exceeds +/- 30.0 MiB

@smuzaffar

Copy link
Copy Markdown
Contributor

+externals

lets get this in IBs

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_20_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@smuzaffar

Copy link
Copy Markdown
Contributor

@fwyzard , with cuda 13.3.0 , we do get a lot of warnings (from alpaka, I think) like below
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/el8_amd64_gcc13/CMSSW_20_1_X_2026-06-16-2300/DataFormats/BeamSpot

In file included from /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/core/Cuda.hpp:7,
                 from /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/idx/bt/IdxBtUniformCudaHipBuiltIn.hpp:9,
                 from /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/acc/AccGpuUniformCudaHipRt.hpp:14,
                 from /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/acc/AccGpuCudaRt.hpp:7,
                 from /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/alpaka.hpp:22,
                 from src/DataFormats/Portable/interface/PortableHostObject.h:8,
                 from src/DataFormats/BeamSpot/interface/BeamSpotHost.h:5,
                 from src/DataFormats/BeamSpot/interface/alpaka/BeamSpotDevice.h:4,
                 from ./src/DataFormats/BeamSpot/src/alpaka/classes_cuda.h:2,
                 from tmp/el8_amd64_gcc13/src/DataFormats/BeamSpot/src/alpaka/DataFormatsBeamSpotCudaAsync/lcgdict/DataFormatsBeamSpotCudaAsync_xr.cc:38:
  /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/external/alpaka/2.1.1-3caaac8d71f39d400ab2511b2403675a/include/alpaka/core/CudaHipCommon.hpp:62:26: warning: 'double4' is deprecated: use double4_16a or double4_32a [-Wdeprecated-declarations]
    62 |             tuple<char4, double4, float4, int4, long4, longlong4, short4, uchar4, uint4, ulong4, ulonglong4, ushort4>;

currently as these are not directly generated by cmssw code so we ignore these. Is there any thing we can do to avoid these ... may be new version of alpaka?

@fwyzard

fwyzard commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@smuzaffar a quick fix would be to add -Wno-deprecated-declarations to the GCC compiler flags.

Support for CUDA 13.3.0 is still being merged in alpaka, I can look into fixing or silencing the warnings in the coming few days.

@fwyzard

fwyzard commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Actually the HEAD of the alpaka development already has a fix, we just don't have a release yet.
I can make a PR based on the current HEAD of the develop branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants