Update TICL & HGCal simulation truth validation to use new simulation truth collections, and simplify validation to use only SimCluster dataformat by tcuisset · Pull Request #51047 · cms-sw/cmssw

tcuisset · 2026-05-27T09:57:40Z

PR description:

This is a follow-up on #50578, which had updated CaloTruthAccumulator to produce multiple SimCluster collections, with the intent to use only one dataformat for calorimeter truth information (instead of CaloParticle / SimCluster dataformat dichotomy).

This PR updates the TICL/HGCal validation to use the new SimCluster collections produced in CaloTruthAccumulator.
Use of CaloParticle dataformat is replaced nearly everywhere with the use of caloparticles in SimCluster dataformat (with just some remaining usage related to simTime).

Summary of the setup after this PR:

Documentation
Slides at TICL meeting
Slide 7 of HGCAL presentation at NGT MC truth workshop

Details:

Associators with CaloParticle are removed (SimCluster associators are used for the "CaloParticle" SimCluster collection). They were mostly duplicated code
SimTrackstersProducer is split into two producers for clarity : SimTrackstersProducers (only for simtracksters) and SimTICLCandidateProducer (produces SimTICLCandidates)
SimTrackstersProducer is now configurable to produce an arbitrary number of simTrackster collections (instead of CP/SC only). Each collection has also a RefVector to the SimCluster that was used to produce the simTrackster.
SimTICLCandidateProducer is also configurable wrt the SimTrackster collections used
all the code designed to deal with the rare special cases where a SimTrackster is made from SimClusters that do not have a SimTrack crossing boundary is removed, since this corner case cannot happen with "boundary" SimClusters
validation configs for HLT and TICLbarrel are updated accordingly

PR validation:

Tested with various Phase 2 workflows:

29634.0 (TTbar), 29634.758 (+HLT timing ticl_barrel), 29634.773 (+NGT scouting Nano val), 29634.7591 (+HLTNanoValid),
34553.209 (CloseByPGun_Barrel_Front+Run4D121_ticl_barrel)

There should not be any validation differences in reco objects. Very minor differences in validation could be caused by the switch from "legacy" to "boundary" SimCluster (which could change slightly validation results for status=1 photons that do not convert before reaching the calorimeter)

FYI @felicepantaleo @waredjeb

cmsbuild · 2026-05-27T09:58:25Z

cms-bot internal usage

cmsbuild · 2026-05-27T10:29:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51047/49491

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/HGCalReco/interface/Trackster.h modified in PR(s): Add directories for TICL-EGM code development #48319, Move stored types in io_v1 namespace, reset class versions to 3 #50968
- File RecoHGCal/TICL/plugins/SimTrackstersProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, TICL-barrel: 3D Pattern recognition in the barrel #50551
- File RecoHGCal/TICL/plugins/TICLDumper.cc modified in PR(s): Add directories for TICL-EGM code development #48319, TICL-barrel: 3D Pattern recognition in the barrel #50551
- File RecoHGCal/TICL/python/SimTracksters_cff.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File RecoHGCal/TICL/python/ticlDumper_cff.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/plugins/AllTracksterToSimTracksterAssociatorsByHitsProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.h modified in PR(s): Add directories for TICL-EGM code development #48319, TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.h modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorEDProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/python/HitToTracksterAssociation_cfi.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/python/LCToSimTSAssociation_cfi.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/python/LCToTSAssociator_cfi.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/python/TSToSimTSAssociationByHits_cfi.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalAssociatorProducers/python/TSToSimTSAssociation_cfi.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File SimCalorimetry/HGCalSimProducers/python/hgcHitAssociation_cfi.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimDataFormats/Associations/interface/LayerClusterToSimTracksterAssociatorBaseImpl.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/Configuration/python/barrelSimValid_cff.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File Validation/Configuration/python/hgcalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltBarrelSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltHGCalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/interface/HGCalValidator.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/interface/HGVHistoProducerAlgo.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/plugins/HGCalValidator.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/python/HLTBarrelValidator_cff.py modified in PR(s): TICL-barrel: 3D Pattern recognition in the barrel #50551
- File Validation/HGCalValidation/python/HLTHGCalValidator_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/src/TICLCandidateValidator.cc modified in PR(s): Add directories for TICL-EGM code development #48319

cmsbuild · 2026-05-27T10:30:14Z

A new Pull Request was created by @tcuisset for master.

It involves the following packages:

DataFormats/HGCalReco (reconstruction)
HLTrigger/NGTScouting (hlt)
RecoHGCal/TICL (reconstruction)
SimCalorimetry/HGCalAssociatorProducers (simulation)
SimCalorimetry/HGCalSimProducers (simulation)
SimDataFormats/Associations (simulation)
Validation/Configuration (dqm, simulation)
Validation/HGCalValidation (dqm)

@Martin-Grunewald, @Moanwar, @civanch, @cmsbuild, @ctarricone, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @rseidita, @srimanob can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @apsallid, @bsunanda, @cseez, @denizsun, @fabiocos, @felicepantaleo, @forthommel, @hatakeyamak, @lecriste, @lgray, @martinamalberti, @missirol, @mmusich, @pfs, @rovere, @salimcerci, @sameasy, @sethzenz, @sobhatta, @vandreev11 this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

felicepantaleo · 2026-05-27T10:30:51Z

@cmsbuild please test

mmusich · 2026-05-27T10:45:50Z

@cmsbuild, please abort

mmusich · 2026-05-27T10:46:16Z

test parameters:

workflows = ph2_hlt

mmusich · 2026-05-27T10:46:26Z

@cmsbuild, please test

cmsbuild · 2026-05-27T12:14:36Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 136KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-becf79/53501/summary.html
COMMIT: a66a777
CMSSW: CMSSW_17_0_X_2026-05-26-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51047/53501/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34634.999

----- Begin Fatal Exception 27-May-2026 13:43:14 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'validation_step14'
   [2] Prefetching for module HGCalValidator/'hltHgcalValidator'
   [3] Calling method for module SimTrackstersProducer/'hltTiclSimTracksters'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<SimCluster>
Looking for module label: mixData
Looking for productInstanceName: MergedCaloTruthBoundaryTrackSimCluster

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

cmsbuild · 2026-07-07T15:58:14Z

Pull request #51047 was updated. @Martin-Grunewald, @Moanwar, @battibass, @civanch, @cmsbuild, @ctarricone, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @rseidita, @srimanob can you please check and sign again.

jfernan2 · 2026-07-09T09:44:38Z

please test

cmsbuild · 2026-07-09T10:02:14Z

-1

Failed Tests: Build
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-becf79/54632/summary.html
COMMIT: e04caed
CMSSW: CMSSW_20_1_X_2026-07-08-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51047/54632/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Build

I found compilation warning when building: See details on the summary page.

tcuisset · 2026-07-09T10:14:14Z

I had accidentally introduced a spurious semicolon in the last commit, now the build should be fixed

cmsbuild · 2026-07-09T10:14:56Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51047/50137

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/HGCalReco/interface/Trackster.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/plugins/SimTrackstersProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, move HGCAL configuration parameters from double to float #51413
- File RecoHGCal/TICL/plugins/TICLDumper.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): move HGCAL configuration parameters from double to float #51413
- File SimCalorimetry/HGCalAssociatorProducers/plugins/AllTracksterToSimTracksterAssociatorsByHitsProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.h modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorEDProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalSimProducers/python/hgcHitAssociation_cfi.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimDataFormats/Associations/interface/LayerClusterToSimTracksterAssociatorBaseImpl.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimGeneral/CaloAnalysis/plugins/PreMixingCaloParticleWorker.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hgcalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltBarrelSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltHGCalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/interface/HGCalValidator.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/interface/HGVHistoProducerAlgo.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/plugins/HGCalValidator.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/python/HLTHGCalValidator_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/src/TICLCandidateValidator.cc modified in PR(s): Add directories for TICL-EGM code development #48319

cmsbuild · 2026-07-09T10:15:20Z

Pull request #51047 was updated. @Martin-Grunewald, @Moanwar, @battibass, @civanch, @cmsbuild, @ctarricone, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @rseidita, @srimanob can you please check and sign again.

jfernan2 · 2026-07-09T10:16:07Z

please test

cmsbuild · 2026-07-09T12:00:49Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-becf79/54635/summary.html
COMMIT: 60694ab
CMSSW: CMSSW_20_1_X_2026-07-08-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51047/54635/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34434.75

----- Begin Fatal Exception 09-Jul-2026 13:50:46 CEST-----------------------
An exception of category 'LogicError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'validation_step'
   [2] Calling method for module HGCalValidator/'hltHgcalValidator'
Exception Message:
Mismatch between SimTICLCandidate SimToReco map and SimTICLCandidate collection
----- End Fatal Exception -------------------------------------------------

34500.0

----- Begin Fatal Exception 09-Jul-2026 13:21:29 CEST-----------------------
An exception of category 'LogicError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'validation_step14'
   [2] Calling method for module HGCalValidator/'hltHgcalValidator'
Exception Message:
Mismatch between SimTICLCandidate SimToReco map and SimTICLCandidate collection
----- End Fatal Exception -------------------------------------------------

34496.0

----- Begin Fatal Exception 09-Jul-2026 13:21:32 CEST-----------------------
An exception of category 'LogicError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'validation_step14'
   [2] Calling method for module HGCalValidator/'hltHgcalValidator'
Exception Message:
Mismatch between SimTICLCandidate SimToReco map and SimTICLCandidate collection
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

cmsbuild · 2026-07-09T15:10:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51047/50144

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/HGCalReco/interface/Trackster.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/plugins/SimTrackstersProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, move HGCAL configuration parameters from double to float #51413
- File RecoHGCal/TICL/plugins/TICLDumper.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): move HGCAL configuration parameters from double to float #51413
- File SimCalorimetry/HGCalAssociatorProducers/plugins/AllTracksterToSimTracksterAssociatorsByHitsProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.h modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorEDProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalSimProducers/python/hgcHitAssociation_cfi.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimDataFormats/Associations/interface/LayerClusterToSimTracksterAssociatorBaseImpl.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimGeneral/CaloAnalysis/plugins/PreMixingCaloParticleWorker.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hgcalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltBarrelSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltHGCalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/interface/HGCalValidator.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/interface/HGVHistoProducerAlgo.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/plugins/HGCalValidator.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/python/HLTHGCalValidator_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/src/TICLCandidateValidator.cc modified in PR(s): Add directories for TICL-EGM code development #48319

cmsbuild · 2026-07-09T15:10:30Z

Pull request #51047 was updated. @Martin-Grunewald, @Moanwar, @battibass, @civanch, @cmsbuild, @ctarricone, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @rseidita, @srimanob can you please check and sign again.

jfernan2 · 2026-07-13T09:46:02Z

please test

cmsbuild · 2026-07-13T12:10:58Z

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-becf79/54716/summary.html
COMMIT: 65c86e3
CMSSW: CMSSW_20_1_X_2026-07-12-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51047/54716/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 5 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 65 differences found in the comparisons
DQMHistoTests: Total files compared: 59
DQMHistoTests: Total histograms compared: 3842662
DQMHistoTests: Total failures: 37533
DQMHistoTests: Total nulls: 3
DQMHistoTests: Total successes: 3805108
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2664.352999999999 KiB( 58 files compared)
DQMHistoSizes: changed ( 34434.0,... ): 779.092 KiB HGCAL/HGCalValidator
DQMHistoSizes: changed ( 34434.0,... ): -64.793 KiB HLT/HGCAL
DQMHistoSizes: changed ( 34434.0,... ): -0.004 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 34434.0,... ): -0.004 KiB MessageLogger/Warnings
Checked 249 log files, 203 edm output root files, 59 DQM output files
TriggerResults: found differences in 19 / 58 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 102.8 exceeds +/- 30.0 MiB

Moanwar · 2026-07-13T17:50:21Z

Hi @tcuisset, Thanks a lot for fixing and updating the PR! Could you please have a look at the memory increase? It seems to be ~70 MB above the threshold. Is that expected?

…om eventContent to reduce the memory usage when running on pileup

tcuisset · 2026-07-15T13:41:51Z

Hi @tcuisset, Thanks a lot for fixing and updating the PR! Could you please have a look at the memory increase? It seems to be ~70 MB above the threshold. Is that expected?

Hi @Moanwar , I've reduced the number of associators that run by default in the config, so it should reduce the memory usage when running with pileup now.

cmsbuild · 2026-07-15T13:43:10Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51047/50208

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/HGCalReco/interface/Trackster.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/plugins/SimTrackstersProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319, move HGCAL configuration parameters from double to float #51413
- File RecoHGCal/TICL/plugins/TICLDumper.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): move HGCAL configuration parameters from double to float #51413
- File SimCalorimetry/HGCalAssociatorProducers/plugins/AllTracksterToSimTracksterAssociatorsByHitsProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.cc modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/HitToSimClusterCaloParticleAssociatorProducer.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreImpl.h modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorEDProducer.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimCalorimetry/HGCalSimProducers/python/hgcHitAssociation_cfi.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File SimDataFormats/Associations/interface/LayerClusterToSimTracksterAssociatorBaseImpl.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File SimGeneral/CaloAnalysis/plugins/PreMixingCaloParticleWorker.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hgcalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltBarrelSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/Configuration/python/hltHGCalSimValid_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/interface/HGCalValidator.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/interface/HGVHistoProducerAlgo.h modified in PR(s): Add directories for TICL-EGM code development #48319
- File Validation/HGCalValidation/plugins/HGCalValidator.cc modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/python/HLTHGCalValidator_cff.py modified in PR(s): Particle Flow cluster validation in ECAL #50292
- File Validation/HGCalValidation/src/TICLCandidateValidator.cc modified in PR(s): Add directories for TICL-EGM code development #48319

Moanwar · 2026-07-15T13:43:17Z

Hi @tcuisset, Thanks a lot for fixing and updating the PR! Could you please have a look at the memory increase? It seems to be ~70 MB above the threshold. Is that expected?

Hi @Moanwar , I've reduced the number of associators that run by default in the config, so it should reduce the memory usage when running with pileup now.

Thanks a lot @tcuisset , sounds good , lets test again

Moanwar · 2026-07-15T13:43:25Z

@cmsbuild please test

cmsbuild · 2026-07-15T13:43:38Z

Pull request #51047 was updated. @Martin-Grunewald, @Moanwar, @battibass, @civanch, @ctarricone, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @rseidita, @srimanob can you please check and sign again.

cmsbuild · 2026-07-15T16:00:33Z

+1

Size: This PR adds an extra 36KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-becf79/54772/summary.html
COMMIT: c885ee3
CMSSW: CMSSW_20_1_X_2026-07-15-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51047/54772/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 5 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 69 differences found in the comparisons
DQMHistoTests: Total files compared: 59
DQMHistoTests: Total histograms compared: 3842950
DQMHistoTests: Total failures: 35849
DQMHistoTests: Total nulls: 3
DQMHistoTests: Total successes: 3807080
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -1533.7920000000008 KiB( 58 files compared)
DQMHistoSizes: changed ( 34434.0,... ): -64.793 KiB HLT/HGCAL
DQMHistoSizes: changed ( 34434.0,... ): -60.537 KiB HGCAL/HGCalValidator
DQMHistoSizes: changed ( 34434.0,... ): -0.004 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 34434.0,... ): -0.004 KiB MessageLogger/Warnings
Checked 249 log files, 203 edm output root files, 59 DQM output files
TriggerResults: found differences in 20 / 58 workflows

Moanwar · 2026-07-16T08:58:57Z

Hi @tcuisset, Thanks a lot for fixing and updating the PR! Could you please have a look at the memory increase? It seems to be ~70 MB above the threshold. Is that expected?

Hi @Moanwar , I've reduced the number of associators that run by default in the config, so it should reduce the memory usage when running with pileup now.

Thanks @tcuisset, it seems to be working.
Just for my understanding, after this change, is what we have in the description still valid? And is the diagram still correct as well?

tcuisset · 2026-07-16T09:12:04Z

Hi @tcuisset, Thanks a lot for fixing and updating the PR! Could you please have a look at the memory increase? It seems to be ~70 MB above the threshold. Is that expected?

Hi @Moanwar , I've reduced the number of associators that run by default in the config, so it should reduce the memory usage when running with pileup now.

Thanks @tcuisset, it seems to be working. Just for my understanding, after this change, is what we have in the description still valid? And is the diagram still correct as well?

Mostly the diagram is still accurate. For the left part (SimCluster part), there is no change. For the right part, the SimTracksters for "merged" and "legacy" are disabled by default (but the "CaloParticle" and "Boundary-track" are still enabled). But you just need to uncomment a few lines of python config to enable them back. The related associators are also disabled (they were not shown on the diagram).
All this can be re-recoed if needed.

Moanwar · 2026-07-16T09:14:34Z

+1

civanch · 2026-07-17T01:57:27Z

+simulation

gabrielmscampos · 2026-07-20T09:08:41Z

+dqm

cmsbuild added this to the CMSSW_17_0_X milestone May 27, 2026

cmsbuild added reconstruction-pending simulation-pending dqm-pending hlt-pending pending-signatures tests-pending orp-pending code-checks-pending labels May 27, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels May 27, 2026

cmsbuild added tests-started and removed tests-pending labels May 27, 2026

cmsbuild added tests-pending and removed tests-started labels May 27, 2026

cmsbuild added tests-started and removed tests-pending labels May 27, 2026

cmsbuild added tests-rejected tests-pending and removed tests-started tests-rejected code-checks-approved labels May 27, 2026

brusale mentioned this pull request Jul 8, 2026

Unfiy TICL-barrel and HGCAL validation #51448

Open

CHange asserts into exceptions

65c86e3

cmsbuild mentioned this pull request Jul 12, 2026

Guard unguarded hitMap lookups in HGVHistoProducerAlgo #51467

Merged

Remove "merged" and "legacy" SimClusters from most associators and fr…

c885ee3

…om eventContent to reduce the memory usage when running on pileup

cmsbuild mentioned this pull request Jul 16, 2026

HGCal displaced gun validations #51510

Merged

cmsbuild mentioned this pull request Jul 20, 2026

Build RecHitTools per event to fix cross-stream double-free #51529

Merged

Uh oh!

Conversation

tcuisset commented May 27, 2026

PR description:

PR validation:

Uh oh!

cmsbuild commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented May 27, 2026

Uh oh!

cmsbuild commented May 27, 2026

Uh oh!

felicepantaleo commented May 27, 2026

Uh oh!

mmusich commented May 27, 2026

Uh oh!

mmusich commented May 27, 2026

Uh oh!

mmusich commented May 27, 2026

Uh oh!

cmsbuild commented May 27, 2026

Failed RelVals

Uh oh!

cmsbuild commented Jul 7, 2026

Uh oh!

jfernan2 commented Jul 9, 2026

Uh oh!

cmsbuild commented Jul 9, 2026

Failed Build

Uh oh!

tcuisset commented Jul 9, 2026

Uh oh!

cmsbuild commented Jul 9, 2026

Uh oh!

cmsbuild commented Jul 9, 2026

Uh oh!

jfernan2 commented Jul 9, 2026

Uh oh!

cmsbuild commented Jul 9, 2026

Failed RelVals

Uh oh!

cmsbuild commented Jul 9, 2026

Uh oh!

cmsbuild commented Jul 9, 2026

Uh oh!

jfernan2 commented Jul 13, 2026

Uh oh!

cmsbuild commented Jul 13, 2026

Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

Moanwar commented Jul 13, 2026

Uh oh!

tcuisset commented Jul 15, 2026

Uh oh!

cmsbuild commented Jul 15, 2026

Uh oh!

Moanwar commented Jul 15, 2026

Uh oh!

Moanwar commented Jul 15, 2026

Uh oh!

cmsbuild commented Jul 15, 2026

Uh oh!

cmsbuild commented Jul 15, 2026

Comparison Summary

Uh oh!

Moanwar commented Jul 16, 2026

Uh oh!

tcuisset commented Jul 16, 2026

Uh oh!

Moanwar commented Jul 16, 2026

Uh oh!

civanch commented Jul 17, 2026

Uh oh!

gabrielmscampos commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

cmsbuild commented May 27, 2026 •

edited

Loading