Skip to content

Use spatial correlation also in impact jobs - #11695

Draft
ptormene wants to merge 1 commit into
masterfrom
use_spatial_corr
Draft

Use spatial correlation also in impact jobs#11695
ptormene wants to merge 1 commit into
masterfrom
use_spatial_corr

Conversation

@ptormene

@ptormene ptormene commented Aug 7, 2026

Copy link
Copy Markdown
Member

So far, when calculating the gmfs in impact calculations, we have been using the 'basic' (faster) approach without correlation. We need to use 'Silva&Horspool' instead. This will make the computation heavier and slower. Let's see how this will affect the computational performance.

I am also running tests here: https://github.com/gem/oq-engine/actions/runs/31259846195
and here: https://github.com/gem/oq-engine/actions/runs/31259857636

I tried on the staging machine using the impact_report branch plus the spatial correlation, and I got the following error after attempting to run an impact job using the ShakeMap with ID us6000t7zp:

  File "/mnt/impact/oq-engine/openquake/engine/engine.py", line 222, in run_calc
    calc.run(shutdown=True)
  File "/mnt/impact/oq-engine/openquake/calculators/base.py", line 344, in run
    self.pre_execute()
  File "/mnt/impact/oq-engine/openquake/calculators/event_based_risk.py", line 412, in pre_execute
    super().pre_execute()
  File "/mnt/impact/oq-engine/openquake/calculators/base.py", line 743, in pre_execute
    calc.run(remove=False)
  File "/mnt/impact/oq-engine/openquake/calculators/base.py", line 348, in run
    self.result = self.execute()
                  ^^^^^^^^^^^^^^
  File "/mnt/impact/oq-engine/openquake/calculators/event_based.py", line 921, in execute
    base.store_gmfs_from_shakemap(self, self.sitecol, self.assetcol)
  File "/mnt/impact/oq-engine/openquake/calculators/base.py", line 1809, in store_gmfs_from_shakemap
    store_gmfs(calc, sitecol, shakemap, gmf_dict)
  File "/mnt/impact/oq-engine/openquake/calculators/base.py", line 1727, in store_gmfs
    imts, gmfs = to_gmfs(shakemap, gmf_dict, vs30,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/impact/oq-engine/openquake/hazardlib/shakemap/gmfs.py", line 319, in to_gmfs
    gmfs = calculate_gmfs(gmf_dict.pop('kind'), **gmf_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/impact/oq-engine/openquake/baselib/general.py", line 806, in __call__
    return self[key](obj, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/impact/oq-engine/openquake/hazardlib/shakemap/gmfs.py", line 209, in calculate_gmfs_sh
    raise ValueError(CORRELATION_MATRIX_TOO_LARGE % (
ValueError: You have a correlation matrix which is too large: M=5 x N=30608 > 10000.
To avoid that, set a proper `region_grid_spacing` so that your exposure
involves less sites.

Using the small (sampled) exposure, I completed the same calculation on my laptop, but the full exposure makes the computation too expensive. For now I will roll back the staging machine to work without spatial correlation, then we will need to further discuss about this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant