Add ability to run each of a calculation's base sources sequentially instead

[CB-quakemodel]

If sequential_source_models=true in the config, then both the preclassical and the classical stages are ran sequentially (all the preclassical batches first, then all the classical batches), one base source model at a time.

This is important for very large logic trees like the BC Hydro NVA model - in both the classical and the preclassical, it is seemingly tricky to avoid an OOM issue (or similar bottlenecking type problems) without this sequential approach, even with sampling.

• For example, with 16,000 samples we can get through the preclassical stage on the hazard workstation without the sequential approach, but at 20,000 samples we OOM without this sequential approach given the preclassical fails.

• One-to-one comparison on the hazard workstation is therefore seemingly limited to 16,000 samples, where the calculation takes about 9 hours for sequential in only classical, 11 hours for sequential in both preclassical and classical and about 9 hours for regular approaches - with 20,000 samples it is possible with only the sequential approach given it can complete the preclassical stage without OOM.

A new QA test is added (logictree/case_24) which ensures it obtains identical results in both full enumeration and sampling to the "regular" approach. An existing QA test (logictree/case_83) which uses extendModel is expanded to check identical results are obtained when using full enumeration and sampling here versus the "regular" approach too.

NOTE: This batching only works if the source groups are not shared amongst the base sources - an explicit error is raised if this is not true to guard against it.

NOTE: ExtendModel is supported. This is the only case that shared source groups amongst the base sources are permitted when sequential_source_models=true - this is because a final batch is ran on the shared source groups (shared over the different base sources) separately in this case (with the potential limitation that if the logic tree heavily relies on extend model the memory footprint might be similar (or equal) the footprint of the "regular" approach.

NOTE: Currently it only works in classical and disagg given it's (probably) most useful for large logic trees used in site-specific PSHAs (i.e., when using the classical calculator). An error is raised to prevent use in other calculator types. Expanding to other calculators is probably best done so in additional PRs.

NOTE: Some dtype widening is included from #11709 so I can run the model on the hazard workstation using this branch (the PRs are separate for clarity).

[micheles]

I don't like the idea of changing the source model logic tree file. The logic tree should stay unchanged; the calculation logic can be controlled by the job.ini instead. For instance there could be a flag sequential_source_models=true (you are not spawning subcalculations, just making the Starmap sequential by source model). What about the case of a single source belonging to multiple source model? I guess you will lose the performance improvement of the current approach, that could make the New Zealand model much slower (if I remember correctly it is New Zealand, but one would have to re-run all of the mosaic). Also, what about the extendModel feature? This requires a lot of thinking and measurements.