Implement CUDA 13 flags to parallelise device-code compilation by Parsifal-2045 · Pull Request #10735 · cms-sw/cmsdist

Parsifal-2045 · 2026-07-20T14:55:46Z

This PR implements a few new nvcc flags to speed-up device code compilation. In particular:

--split-compile number to try and automatically divide heavy TUs into smaller chunks
--threads to allow parallel compilation of device-code TUs
--jobserver to make it so both the split and the threads rely on the same jobs spawned by Make

I have performed some local measurements on a node equipped with 2 AMD EPYC 9534 using #51085 to pull about 25% of CMSSW and especially the RecoTracker/PixelSeeding package as one of the slowest to compile for device. I have also disabled rocm compilation to isolate the effects on the CUDA side. I attach here the results from time for each compilation step (starting from scram b clean before every pass):

Default compilation time:
real    13m16.469s
user    569m49.105s
sys     254m48.762s
Default RecoTracker/PixelSeeding compilation time:
real    9m39.174s
user    33m22.067s
sys     7m16.893s
New flags compilation time:
real    8m25.389s
user    510m24.473s
sys     107m21.013s
New flags RecoTracker/PixelSeeding compilation time:
real    2m32.060s
user    34m28.999s
sys     7m36.785s

Note: Requires cms-sw/cmssw-config#120 to be able to correctly parse the --jobserver flag

- split-compile=0 tries to automatically divide heavy TUs into multiple parallel compilation units - threads=0 uses all available CPU cores for parallel compilation - jobserver allows parallel compilation to be integrated with make's jobserver, so that the number of parallel jobs is limited to the number of jobs make is allowed to run in parallel

cmsbuild · 2026-07-20T14:56:08Z

A new Pull Request was created by @Parsifal-2045 for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-07-20T14:56:09Z

cms-bot internal usage

akritkbehera · 2026-07-20T15:01:03Z

please test with cms-sw/cmssw-config#120 using full cmssw

Parsifal-2045 · 2026-07-20T15:39:25Z

type ngt

cmsbuild · 2026-07-21T00:16:26Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c386a5/54829/summary.html
COMMIT: e8e12b8
CMSSW: CMSSW_20_1_X_2026-07-20-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10735/54829/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/py3-torch-rocm/2.13.0-88987b092c5a3e143ba94a098698df9b/cmsdist-pip-src/torch-2.13.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.VE8OrC (%build)

RPM build warnings:
Macro expanded in comment on line 651: %{pkginstroot}/bin/*

Macro expanded in comment on line 656: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

akritkbehera · 2026-07-21T06:35:41Z

please test with cms-sw/cmssw-config#120 using full cmssw

akritkbehera · 2026-07-21T06:40:28Z

Let's rebuild and see

cmsbuild · 2026-07-21T07:30:31Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c386a5/54836/summary.html
COMMIT: e8e12b8
CMSSW: CMSSW_20_1_X_2026-07-20-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10735/54836/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/py3-torch-rocm/2.13.0-88987b092c5a3e143ba94a098698df9b/cmsdist-pip-src/torch-2.13.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.g3AYMe (%build)

RPM build warnings:
Macro expanded in comment on line 651: %{pkginstroot}/bin/*

Macro expanded in comment on line 656: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

Parsifal-2045 · 2026-07-21T08:16:20Z

I am a bit puzzled by the failure, it looks like nothing here should be compiled using nvcc (or CUDA_FLAGS)

akritkbehera · 2026-07-21T10:28:47Z

Yes it's not. It's puzzling I am taking a look!

smuzaffar · 2026-07-21T13:40:58Z

@akritkbehera , the probklem is the way we build externals locally. For externals which are already installed on cvmfs we create symlinks in the build area and then torch fails when it tries to copy the aotriton libs (https://github.com/pytorch/pytorch/blob/aa034647b537aee3afaec28fd3054c566762d940/cmake/External/aotriton.cmake#L279-L284) . I would suggest to patch torch cmake file and avoid the install step e.g.

 if(DEFINED ENV{AOTRITON_INSTALLED_PREFIX})
-  install(DIRECTORY
-          $ENV{AOTRITON_INSTALLED_PREFIX}/lib
-          $ENV{AOTRITON_INSTALLED_PREFIX}/include
-          DESTINATION ${__AOTRITON_INSTALL_DIR})
   set(__AOTRITON_INSTALL_DIR "$ENV{AOTRITON_INSTALLED_PREFIX}")
   message(STATUS "Using Preinstalled AOTriton at ${__AOTRITON_INSTALL_DIR}")

the install command here copies the symlink $ENV{AOTRITON_INSTALLED_PREFIX}/lib instead of contents of $ENV{AOTRITON_INSTALLED_PREFIX}/lib . See if removing the install command works . If torch really need aotriton libs/include in the local ${__AOTRITON_INSTALL_DIR} then may be just first try to read the symlink and then use the expanded paths in the install command

akritkbehera · 2026-07-22T11:07:00Z

I have a PR open to fix py-torch-rocm.

…g present. cms-sw#10735 (comment) Fix wheel building issue

cmsbuild · 2026-07-22T16:42:07Z

REMINDER @mandrenguyen, @sextonkennedy, @ftenchini: This PR was tested with #10738, please check if they should be merged together

akritkbehera · 2026-07-22T16:42:44Z

please test with cms-sw/cmssw-config#120

akritkbehera · 2026-07-22T18:18:40Z

yay it's building CMSSW!

cmsbuild · 2026-07-22T21:03:10Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c386a5/54878/summary.html
COMMIT: e8e12b8
CMSSW: CMSSW_20_1_X_2026-07-22-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10735/54878/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c386a5/54878/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c386a5/54878/git-merge-result

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testHltDiff had ERRORS

Comparison Summary

Summary:

You potentially removed 169 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3548294
DQMHistoTests: Total failures: 41
DQMHistoTests: Total nulls: 144
DQMHistoTests: Total successes: 3548091
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2381.8030000000003 KiB( 44 files compared)
DQMHistoSizes: changed ( 34434.75,... ): 249.128 KiB HLT/HGCAL
DQMHistoSizes: changed ( 34434.0,... ): 177.407 KiB HGCAL/HGCalValidator
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: found differences in 1 / 44 workflows

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Jul 20, 2026

cmsbuild added tests-started requires-external and removed tests-pending labels Jul 20, 2026

cmsbuild added the ngt label Jul 20, 2026

cmsbuild added tests-rejected and removed tests-started labels Jul 21, 2026

cmsbuild added tests-started and removed tests-rejected labels Jul 21, 2026

cmsbuild added tests-rejected and removed tests-started labels Jul 21, 2026

akritkbehera added a commit to akritkbehera/cmsdist that referenced this pull request Jul 22, 2026

Fix Issues oy-torch-rocm CI Failures related to AOTRITON already bein…

56775db

…g present. cms-sw#10735 (comment) Fix wheel building issue

akritkbehera mentioned this pull request Jul 22, 2026

Fix Issues py-torch-rocm CI Failures related to AOTRITON #10738

Merged

cmsbuild added tests-started and removed tests-rejected labels Jul 22, 2026

cmsbuild added tests-rejected and removed tests-started labels Jul 22, 2026

Uh oh!

Conversation

Parsifal-2045 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jul 20, 2026

Uh oh!

cmsbuild commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akritkbehera commented Jul 20, 2026

Uh oh!

Parsifal-2045 commented Jul 20, 2026

Uh oh!

cmsbuild commented Jul 21, 2026

Failed External Build

Uh oh!

akritkbehera commented Jul 21, 2026

Uh oh!

akritkbehera commented Jul 21, 2026

Uh oh!

cmsbuild commented Jul 21, 2026

Failed External Build

Uh oh!

Parsifal-2045 commented Jul 21, 2026

Uh oh!

akritkbehera commented Jul 21, 2026

Uh oh!

smuzaffar commented Jul 21, 2026

Uh oh!

akritkbehera commented Jul 22, 2026

Uh oh!

cmsbuild commented Jul 22, 2026

Uh oh!

akritkbehera commented Jul 22, 2026

Uh oh!

akritkbehera commented Jul 22, 2026

Uh oh!

cmsbuild commented Jul 22, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Parsifal-2045 commented Jul 20, 2026 •

edited

Loading

cmsbuild commented Jul 20, 2026 •

edited

Loading