[Do not merge]Enable building with cuda for el9_amd64_gcc15 by smuzaffar · Pull Request #2785 · cms-sw/cms-bot

smuzaffar · 2026-06-09T15:43:43Z

No description provided.

cmsbuild · 2026-06-09T15:44:16Z

A new Pull Request was created by @smuzaffar for branch master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-06-09T15:44:17Z

cms-bot internal usage

smuzaffar · 2026-06-09T15:47:32Z

please test with cms-sw/cmsdist#10613 for CMSSW_20_1_X/el9_amd64_gcc15

lets see if devel branch PR works

cmsbuild · 2026-06-09T19:19:57Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/53784/summary.html
COMMIT: 900ea85
CMSSW: CMSSW_20_1_X_2026-06-09-1100/el9_amd64_gcc15
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2785/53784/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc15/external/py3-torch-cuda/2.11.0-11fd4a3df8667dc745e2a828865a9885/external_py3-torch-cuda_2.11.0-11fd4a3df8667dc745e2a828865a9885-1-build/cmsdist-pip-src/py3-torch-cuda-2.11.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.ZlYxMk (%build)

RPM build warnings:
Macro expanded in comment on line 608: %{pkginstroot}/bin/*

Macro expanded in comment on line 613: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

cmsbuild · 2026-06-13T18:40:01Z

Pull request #2785 was updated.

smuzaffar · 2026-06-16T08:01:08Z

please test for el9_amd64_gcc15

smuzaffar · 2026-06-16T08:21:30Z

please test with cms-sw/cmsdist#10637

cmsbuild · 2026-06-16T11:13:15Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/53971/summary.html
COMMIT: c5535e3
CMSSW: CMSSW_20_1_X_2026-06-13-1100/el9_amd64_gcc15
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2785/53971/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

./usr/lib/.build-id/e8
./usr/lib/.build-id/e8/a15e345c889841a4bfd473fb9fadbdbe79aab3
1209 blocks
+ cp /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/SOURCES/external/rocm/7.2.4-57ab736cd3fc6438ce9e1761b1f8cf53/c++config.h opt/rocm-7.2.4/llvm/lib/clang/20/include/cuda_wrappers/bits
cp: cannot create regular file 'opt/rocm-7.2.4/llvm/lib/clang/20/include/cuda_wrappers/bits': No such file or directory
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.OAUQIr (%build)

RPM build warnings:
Macro expanded in comment on line 492: %{pkginstroot}

smuzaffar · 2026-06-16T11:16:29Z

please test with cms-sw/cmsdist#10637, cms-sw/cmsdist#10639

smuzaffar · 2026-06-17T18:23:30Z

please test with cms-sw/cmsdist#10637 using full cmssw for el9_amd64_gcc15

fwyzard · 2026-06-17T21:15:09Z

please test for el9_amd64_gcc14

The CMSSW archive is no longer available.

smuzaffar · 2026-06-17T21:57:17Z

The CMSSW archive is no longer available.

when externals do not build then we do not have cmssw archive file. As gcc14 fails during the external build phase that is why the archive is missing.

By the way, by default ( i.e. simple please test) cms-bot PRs do not build externals . you need to add a cmsdist PR so that bot builds externals

fwyzard · 2026-06-18T03:38:25Z

Ah, ok.

I'll try to reproduce it locally, then.

cmsbuild · 2026-06-18T06:21:18Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54049/summary.html
COMMIT: cd46f4a
CMSSW: CMSSW_20_1_X_2026-06-17-1100/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2785/54049/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 191 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 82961 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 430451
DQMHistoTests: Total nulls: 347
DQMHistoTests: Total successes: 2983661
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 5.561 KiB( 44 files compared)
DQMHistoSizes: changed ( 2022.0010001 ): -0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001 ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0050001 ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0060001 ): -0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0070001,... ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2025.0010001 ): 0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 7.3 ): 5.585 KiB SiStrip/MechanicalView
Checked 195 log files, 163 edm output root files, 45 DQM output files

cmsbuild · 2026-06-18T06:53:24Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54046/summary.html
COMMIT: cd46f4a
CMSSW: CMSSW_20_1_X_2026-06-16-2300/el9_amd64_gcc15
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2785/54046/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54046/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54046/git-merge-result

Comparison Summary

Summary:

You potentially added 181 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 81595 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 438569
DQMHistoTests: Total nulls: 343
DQMHistoTests: Total successes: 2975547
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 5.861 KiB( 44 files compared)
DQMHistoSizes: changed ( 2022.0010001 ): -0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2023.0020001,... ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0000001 ): -0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001 ): -0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0040001,... ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0050001 ): -0.027 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0060001 ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 7.3 ): 5.960 KiB SiStrip/MechanicalView
Checked 195 log files, 163 edm output root files, 45 DQM output files

smuzaffar · 2026-06-18T21:53:06Z

please test with cms-sw/cmsdist#10659 for CMSSW_20_1_CPP23_X

cmsbuild · 2026-06-19T01:23:35Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54093/summary.html
COMMIT: cd46f4a
CMSSW: CMSSW_20_1_CPP23_X_2026-06-18-1100/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2785/54093/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc14/external/py3-torch-cuda/2.12.0-02c7dfc0bd56aaa7873dd0228b685415/cmsdist-pip-src/py3-torch-cuda-2.12.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.gm87Ax (%build)

RPM build warnings:
Macro expanded in comment on line 607: %{pkginstroot}/bin/*

Macro expanded in comment on line 612: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2026-06-20T08:59:24Z

please test with cms-sw/cmsdist#10659 using full cmssw for el9_amd64_gcc15

cmsbuild · 2026-06-20T19:44:25Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54138/summary.html
COMMIT: cd46f4a
CMSSW: CMSSW_20_1_X_2026-06-18-2300/el9_amd64_gcc15
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2785/54138/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw pyg-lib: Do not build for native arch cmsdist#10664
@ghyls Prevent the MPI modules from throwing upon race conditions or hanging at the end of a run. cmssw#50889
@Dr15Jones Make CalibPPS ESSources concurrent capable cmssw#51220
@suchandradutta Bug fix in Phase2 Tracker Digitizer in cross talk calculation cmssw#51224
@Dr15Jones Move to EventSetupRecordInfiniteIntervalFinder cmssw#51231
@raoatifshad [WarningFix]: Alignment/OfflineValidation- Fix static thread-safety w… cmssw#51254
@slava77 LST updates to handle "pixel tracks" with OT hits cmssw#51261
@smorovic [DAQ] update DTH tests and drop file locking cmssw#51262
@bsunanda Phase2-hgx367K Update the testing code for neighour searches of HGCal silicon cells cmssw#51264
@smuzaffar [TrackerGeometry]Fix nvcc warings: type qualifier on return type is meaningless cmssw#51266

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54138/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-48f289/54138/git-merge-result

Comparison Summary

Summary:

You potentially added 196 lines to the logs
Reco comparison results: 81761 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 440958
DQMHistoTests: Total nulls: 358
DQMHistoTests: Total successes: 2973143
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 5.861 KiB( 44 files compared)
DQMHistoSizes: changed ( 2022.0010001 ): -0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2023.0020001,... ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0000001 ): -0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0030001 ): -0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0040001,... ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0050001 ): -0.027 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 2024.0060001 ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 7.3 ): 5.960 KiB SiStrip/MechanicalView
Checked 195 log files, 163 edm output root files, 45 DQM output files

smuzaffar · 2026-07-17T14:31:25Z

please test with cms-sw/cmsdist#10637 for CMSSW_20_1_CPP23_X

Enable building with cuda for el9_amd64_gcc15

900ea85

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jun 9, 2026

smuzaffar mentioned this pull request Jun 9, 2026

[devel] Update to CUDA 13 cms-sw/cmsdist#10613

Closed

cmsbuild added tests-started requires-external and removed tests-pending labels Jun 9, 2026

fwyzard mentioned this pull request Jun 9, 2026

[gcc15] Update CUDA to version 13.3.0 cms-sw/cmsdist#10616

Closed

cmsbuild added tests-rejected and removed tests-started labels Jun 9, 2026

Merge branch 'master' into smuzaffar-patch-5

c5535e3

cmsbuild removed the tests-rejected label Jun 13, 2026

cmsbuild added tests-pending and removed requires-external labels Jun 13, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 16, 2026

cmsbuild added the requires-external label Jun 16, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 16, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 17, 2026

cmsbuild added tests-started and removed tests-rejected labels Jun 17, 2026

cmsbuild removed the requires-external label Jun 17, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 18, 2026

cmsbuild added requires-external tests-started and removed tests-rejected labels Jun 18, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 19, 2026

smuzaffar mentioned this pull request Jun 19, 2026

PyTorch 2.12.1 cms-sw/cmsdist#10659

Merged

cmsbuild added tests-started and removed tests-rejected labels Jun 20, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 20, 2026

cmsbuild added tests-started and removed tests-rejected labels Jul 17, 2026

Uh oh!

Conversation

smuzaffar commented Jun 9, 2026

Uh oh!

cmsbuild commented Jun 9, 2026

Uh oh!

cmsbuild commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jun 9, 2026

Uh oh!

cmsbuild commented Jun 9, 2026

Failed External Build

Uh oh!

cmsbuild commented Jun 13, 2026

Uh oh!

smuzaffar commented Jun 16, 2026

Uh oh!

smuzaffar commented Jun 16, 2026

Uh oh!

cmsbuild commented Jun 16, 2026

Failed External Build

Uh oh!

smuzaffar commented Jun 16, 2026

Uh oh!

smuzaffar commented Jun 17, 2026

Uh oh!

fwyzard commented Jun 17, 2026

Uh oh!

smuzaffar commented Jun 17, 2026

Uh oh!

fwyzard commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 18, 2026

Comparison Summary

Uh oh!

cmsbuild commented Jun 18, 2026

Comparison Summary

Uh oh!

smuzaffar commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 19, 2026

Failed External Build

Uh oh!

smuzaffar commented Jun 20, 2026

Uh oh!

cmsbuild commented Jun 20, 2026

Comparison Summary

Uh oh!

smuzaffar commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmsbuild commented Jun 9, 2026 •

edited

Loading