Skip to content

LST: add LSTGeometry package and associated ESProducer#50679

Merged
cmsbuild merged 5 commits into
cms-sw:masterfrom
SegmentLinking:ariostas/lst_geometry
Jun 18, 2026
Merged

LST: add LSTGeometry package and associated ESProducer#50679
cmsbuild merged 5 commits into
cms-sw:masterfrom
SegmentLinking:ariostas/lst_geometry

Conversation

@ariostas

@ariostas ariostas commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

This PR adds a new RecoTracker/LSTGeometry package containing the module map computation used by the LST algorithm. Currently, the maps are pre-computed by the code in https://github.com/SegmentLinking/LSTGeometry and they are stored in https://github.com/cms-data/RecoTracker-LSTCore. This PR allows for the on-the-fly computation of these maps via an ESProducer, ensuring that they stay consistent with the tracker geometry being used.

This is the last major task in #46746.

c.c. @slava77

@cmsbuild

cmsbuild commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

cmsbuild commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50679/48907

@cmsbuild

cmsbuild commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @ariostas for master.

It involves the following packages:

  • HLTrigger/Configuration (hlt)
  • RecoTracker/IterativeTracking (reconstruction)
  • RecoTracker/LST (reconstruction)
  • RecoTracker/LSTCore (reconstruction)
  • RecoTracker/LSTGeometry (****)

The following packages do not have a category, yet:

RecoTracker/LSTGeometry
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@Martin-Grunewald, @Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @mmusich, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @SohamBhattacharya, @VinInn, @VourMa, @dgulhan, @elusian, @felicepantaleo, @gpetruc, @missirol, @mmasciov, @mmusich, @mtosi, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@mmusich

mmusich commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • enable = hlt_p2_integration, hlt_p2_timing
  • workflows = ph2_hlt

@mmusich

mmusich commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

@cmsbuild

cmsbuild commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

-1

Failed Tests: UnitTests HLTP2Timing
Size: This PR adds an extra 104KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7657dc/52513/summary.html
COMMIT: e612f24
CMSSW: CMSSW_17_0_X_2026-04-07-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50679/52513/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS

Comparison Summary

Summary:

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 17 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.7501_TTbar_14TeV+Run4D121_HLT75e33TrackingOnly step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.7502_TTbar_14TeV+Run4D121_HLT75e33TrackingNtuple step2 max memory diff 191.9 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.752_TTbar_14TeV+Run4D121_HLT75e33TimingTiclV5 step2 max memory diff 189.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.7521_TTbar_14TeV+Run4D121_HLT75e33TimingTiclV5TrackLinkGNN step2 max memory diff 166.0 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.755_TTbar_14TeV+Run4D121_HLT75e33TimingLST step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.756_TTbar_14TeV+Run4D121_HLT75e33TimingTrimmedTracking step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.757_TTbar_14TeV+Run4D121_HLT75e33TimingMkFitFit step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.758_TTbar_14TeV+Run4D121_HLT75e33TimingTiclBarrel step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.759_TTbar_14TeV+Run4D121_HLTPhase2WithNano step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.775_TTbar_14TeV+Run4D121_NGTScoutingCAExtensionMergeT5 step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 191.8 exceeds +/- 90.0 MiB
  • Error: Workflow 34634.999_TTbar_14TeV+Run4D121PU_PMXS1S2PR step3 max memory diff 191.8 exceeds +/- 90.0 MiB

@makortel

makortel commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

Is ~190 MB increase in memory usage expected?

Comment thread RecoTracker/LSTGeometry/test/dumpLSTGeometry.py Outdated
@ariostas

ariostas commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

Is ~190 MB increase in memory usage expected?

That seems a bit high, but it's likely. I'll double-check. Either way, it is only temporarily. Most of it is freed once the maps are constructed.

@makortel

makortel commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

Is ~190 MB increase in memory usage expected?

That seems a bit high, but it's likely. I'll double-check. Either way, it is only temporarily. Most of it is freed once the maps are constructed.

According to the monitoring the peak memory usage would increase by ~190 MB, and thus freeing it afterwards doesn't help much if the job was killed because of going over the limit.

@makortel

makortel commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • workflows_profiling = 34434.0
  • enable_tests = profiling

@makortel

makortel commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

Maybe one round of profiling tests would be worth it.

@Dr15Jones

Copy link
Copy Markdown
Contributor

@fwyzard said

For an EDProducer I don't understand why "all streams" would end up waiting for it.

it could happen if there is no other work that can be done by all the Events (i.e. insufficient concurrency within an Event at that point in the schedule)

@ariostas

Copy link
Copy Markdown
Contributor Author

@fwyzard yeah, it's an ESProducer. And yeah, it only happens once per job, but it causes a big vram spike at the beginning of the job. In #50679 (comment) you can see the spikes at the beginning of the jobs. That is with the caching allocator disabled. Keeping the caching allocator enabled causes the excessive allocation to persist (as expected)

@fwyzard

fwyzard commented May 30, 2026

Copy link
Copy Markdown
Contributor

@ariostas ah, thanks for the clarification.

Is the spike is caused by the ESProducer itself, or by all EDProducers running at the same time as soon as the payload is available ?

@ariostas

Copy link
Copy Markdown
Contributor Author

@fwyzard the spike is caused by the EDProducers running at the same time. I tested it by changing the ESProducer to a dummy one that just waits for 5 seconds.

@fwyzard

fwyzard commented May 31, 2026

Copy link
Copy Markdown
Contributor

OK, thanks for confirming it.
So I guess we need a better way to handle this high memory usage due to concurrency.

@cmsbuild

cmsbuild commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Milestone for this pull request has been moved to CMSSW_20_0_X. Please open a backport if it should also go in to CMSSW_17_0_X.

@makortel

makortel commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

(I assume this PR does not need a backport to 17_0_X (Run 3 legacy))

@cmsbuild

cmsbuild commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Milestone for this pull request has been moved to CMSSW_20_1_X. Please open a backport if it should also go in to CMSSW_20_0_X.

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50679/49752

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #50679 was updated. @Martin-Grunewald, @Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @mmusich, @srimanob can you please check and sign again.

@ariostas

Copy link
Copy Markdown
Contributor Author

I added a new commit with some AI-assisted optimizations that bring down the time of this producer under 1 second. Hopefully this is now quick enough that it will solve the issue with kernel scheduling.

@jfernan2

Copy link
Copy Markdown
Contributor

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: HLTP2Timing
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7657dc/53972/summary.html
COMMIT: d6e94f0
CMSSW: CMSSW_20_1_X_2026-06-15-2300/el9_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50679/53972/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4105866
  • DQMHistoTests: Total failures: 8791
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 4097056
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 60 files compared)
  • Checked 257 log files, 213 edm output root files, 61 DQM output files

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 19 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 47.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7501_TTbar_14TeV+Run4D121_HLT75e33TrackingOnly step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7502_TTbar_14TeV+Run4D121_HLT75e33TrackingNtuple step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7521_TTbar_14TeV+Run4D121_HLT75e33TimingTiclV5TrackLinkGNN step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7522_TTbar_14TeV+Run4D121_HLT75e33TimingMTDatHLT step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.755_TTbar_14TeV+Run4D121_HLT75e33TimingLST step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.756_TTbar_14TeV+Run4D121_HLT75e33TimingTrimmedTracking step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.757_TTbar_14TeV+Run4D121_HLT75e33TimingMkFitFit step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.758_TTbar_14TeV+Run4D121_HLT75e33TimingTiclBarrel step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.759_TTbar_14TeV+Run4D121_HLTPhase2WithNano step2 max memory diff 47.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7591_TTbar_14TeV+Run4D121_HLTPhase2WithNanoValid step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.775_TTbar_14TeV+Run4D121_NGTScoutingCAExtensionMergeT5 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 47.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 47.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 47.2 exceeds +/- 30.0 MiB

@slava77

slava77 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Failed Tests: HLTP2Timing

@mmusich
do you know what happened?
The log file in the bot summary is empty.
(I'm not sure if the tests were started before the HLT timing setup after #51216 merge needed some more time? or is there an unresolved issue still)

@mmusich

mmusich commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

(I'm not sure if the tests were started before the HLT timing setup after #51216 merge needed some more time? or is there an unresolved issue still)

I think tests were sent prematurely here (tagging @jfernan2), let's wait to have a working IB including #51216 to have the baseline measurement and then repeat.

@mmusich

mmusich commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

  • now timing tests should compete

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7657dc/54013/summary.html
COMMIT: d6e94f0
CMSSW: CMSSW_20_1_X_2026-06-16-2300/el9_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50679/54013/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • Reco comparison results: 5 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4105866
  • DQMHistoTests: Total failures: 8803
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 4097044
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 60 files compared)
  • Checked 257 log files, 213 edm output root files, 61 DQM output files

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 19 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7501_TTbar_14TeV+Run4D121_HLT75e33TrackingOnly step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7502_TTbar_14TeV+Run4D121_HLT75e33TrackingNtuple step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 47.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7521_TTbar_14TeV+Run4D121_HLT75e33TimingTiclV5TrackLinkGNN step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7522_TTbar_14TeV+Run4D121_HLT75e33TimingMTDatHLT step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.755_TTbar_14TeV+Run4D121_HLT75e33TimingLST step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.756_TTbar_14TeV+Run4D121_HLT75e33TimingTrimmedTracking step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.757_TTbar_14TeV+Run4D121_HLT75e33TimingMkFitFit step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.758_TTbar_14TeV+Run4D121_HLT75e33TimingTiclBarrel step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.759_TTbar_14TeV+Run4D121_HLTPhase2WithNano step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7591_TTbar_14TeV+Run4D121_HLTPhase2WithNanoValid step2 max memory diff 47.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.775_TTbar_14TeV+Run4D121_NGTScoutingCAExtensionMergeT5 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 47.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 47.2 exceeds +/- 30.0 MiB

@mmusich

mmusich commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

I added a new commit with some AI-assisted optimizations that bring down the time of this producer under 1 second. Hopefully this is now quick enough that it will solve the issue with kernel scheduling.

indeed the resource usage looks better in the last round of tests: link.

This PR (GPU memory) SegmentLinking@d6e94f0 This PR (CPU memory) SegmentLinking@d6e94f0
image image

overall change in timing is negligible link

image

@mmusich

mmusich commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

+hlt

@mmusich

mmusich commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Just for the record, this PR seems to cause tiny widespread changes in HLT tracking and related objects link for a more comprehensive summary.

@jfernan2

Copy link
Copy Markdown
Contributor

+1

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@slava77

slava77 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Just for the record, this PR seems to cause tiny widespread changes in HLT tracking and related objects link for a more comprehensive summary.

small changes were seen in the LST integration tests. The old bin/text file for the LST geometry was derived from the tkLayout csv file; at least some rounding effects would contribute from this.
@ariostas please clarify if some more significant differences were added in the derivation of the maps (beyond just the single/double precision or exact ways to compute [hyperbolic] trig functions).

@ariostas

Copy link
Copy Markdown
Contributor Author

Yeah, the small differences are expected, as Slava mentioned. There were a couple of minor changes with respect to the old maps. The old computation artificially made the modules a bit larger than they actually were and also the center of the modules was displaced a little bit by mistake. So most of the diffences are due to those changes, and the rest are just from floating point errors. I'm currently on vacation typing this on my phone, but I can give a more detailed changelog when I get back, if you prefer.

@mandrenguyen

Copy link
Copy Markdown
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants