Skip to content

PyTorch 2.12.1#10659

Merged
smuzaffar merged 4 commits into
IB/CMSSW_20_1_X/masterfrom
pytorch-2.12.0
Jun 20, 2026
Merged

PyTorch 2.12.1#10659
smuzaffar merged 4 commits into
IB/CMSSW_20_1_X/masterfrom
pytorch-2.12.0

Conversation

@smuzaffar

Copy link
Copy Markdown
Contributor

No description provided.

@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

@cmsbuild

cmsbuild commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54045/summary.html
COMMIT: 9d3ccf9
CMSSW: CMSSW_20_1_X_2026-06-17-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54045/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

patching file torch/utils/cpp_extension.py
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file torch/utils/cpp_extension.py.rej
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.UbkmbA (%build)

RPM build warnings:
Macro expanded in comment on line 545: %{pkginstroot}/bin/*

Macro expanded in comment on line 550: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}


Removed a specific patch source URL from the file.
@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10659 was updated.

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54047/summary.html
COMMIT: ef2ed1c
CMSSW: CMSSW_20_1_X_2026-06-17-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54047/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

+ patch -p1
patching file setup.py
Hunk #1 succeeded at 97 with fuzz 2 (offset 32 lines).
Hunk #2 FAILED at 142.
1 out of 2 hunks FAILED -- saving rejects to file setup.py.rej
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Kw6yXo (%build)

RPM build warnings:
Macro expanded in comment on line 582: %{pkginstroot}/bin/*

Macro expanded in comment on line 587: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}


@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10659 was updated.

@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54053/summary.html
COMMIT: 0fb583e
CMSSW: CMSSW_20_1_X_2026-06-17-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54053/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54053/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54053/git-merge-result

Comparison Summary

Summary:

  • You potentially added 5 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3414477
  • DQMHistoTests: Total failures: 5
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3414454
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
  • Checked 195 log files, 163 edm output root files, 45 DQM output files

@smuzaffar

Copy link
Copy Markdown
Contributor Author

enable gpu

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54096/summary.html
COMMIT: 0fb583e
CMSSW: CMSSW_20_1_X_2026-06-18-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54096/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54096/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54096/git-merge-result

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 3980 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3414477
  • DQMHistoTests: Total failures: 3661
  • DQMHistoTests: Total nulls: 8
  • DQMHistoTests: Total successes: 3410790
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
  • Checked 195 log files, 163 edm output root files, 45 DQM output files

AMD_MI300X Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 234 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 20161
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 146845
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

AMD_W7900 Comparison Summary

There are some workflows for which there are errors in the baseline:
34434.402 step 2
34434.751 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially removed 54 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 219 differences found in the comparisons
  • DQMHistoTests: Total files compared: 5
  • DQMHistoTests: Total histograms compared: 115849
  • DQMHistoTests: Total failures: 14717
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 101132
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 4 files compared)
  • Checked 22 log files, 18 edm output root files, 5 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 266 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 21957
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 145049
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 251 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 21056
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 145950
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 268 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 17955
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 149051
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test for el9_amd64_gcc15

@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test for el9_aarch64_gcc13

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: nvidia_h100UnitTests nvidia_l40sUnitTests nvidia_t4UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54103/summary.html
COMMIT: 0fb583e
CMSSW: CMSSW_20_1_X_2026-06-18-2300/el9_amd64_gcc15
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54103/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54103/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54103/git-merge-result

Comparison Summary

Summary:

  • You potentially added 351 lines to the logs
  • Reco comparison results: 81596 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3414477
  • DQMHistoTests: Total failures: 438582
  • DQMHistoTests: Total nulls: 343
  • DQMHistoTests: Total successes: 2975534
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 5.861 KiB( 44 files compared)
  • DQMHistoSizes: changed ( 2022.0010001 ): -0.008 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2023.0020001,... ): -0.012 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0000001 ): -0.020 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0030001 ): -0.016 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0040001,... ): -0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0050001 ): -0.027 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0060001 ): 0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 7.3 ): 5.960 KiB SiStrip/MechanicalView
  • Checked 195 log files, 163 edm output root files, 45 DQM output files

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54108/summary.html
COMMIT: 0fb583e
CMSSW: CMSSW_20_1_X_2026-06-18-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54108/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54108/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54108/git-merge-result

@fwyzard

fwyzard commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

@smuzaffar PyTorch 2.12.1 was released a couple of days ago to fix a couple of bugs and regressions: https://github.com/pytorch/pytorch/releases/tag/v2.12.1 .

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10659 was updated.

@smuzaffar

Copy link
Copy Markdown
Contributor Author

please test

@smuzaffar smuzaffar changed the title PyTorch 2.12.0 PyTorch 2.12.1 Jun 20, 2026
@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals-AMD_MI300X
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f092e/54137/summary.html
COMMIT: 64a2512
CMSSW: CMSSW_20_1_X_2026-06-19-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10659/54137/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_MI300X

  • 34434.40234434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka/step2_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka.log

Comparison Summary

Summary:

  • You potentially added 8 lines to the logs
  • Reco comparison results: 1 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3414477
  • DQMHistoTests: Total failures: 14931
  • DQMHistoTests: Total nulls: 10
  • DQMHistoTests: Total successes: 3399518
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
  • Checked 195 log files, 163 edm output root files, 45 DQM output files

AMD_W7900 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 121 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 14543
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 152464
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 80 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 15209
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 151798
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 45 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 13665
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 153342
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 70 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 16422
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 150585
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 200.6 exceeds +/- 30.0 MiB

@smuzaffar

Copy link
Copy Markdown
Contributor Author

+externals

@smuzaffar
smuzaffar merged commit 7793fe3 into IB/CMSSW_20_1_X/master Jun 20, 2026
21 of 22 checks passed
@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_20_1_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants