Repository navigation
feat!: region-aware automatic FDSN provider selection - #21
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Provider lists were hardcoded and US-centric (
download_boxesdefaulted to("EARTHSCOPE",),run_pipelineto("EARTHSCOPE", "SCEDC", "NCEDC")). A corridor over Italy queried three American archives and found one station where 52 exist, with nothing in the output indicating a relevant provider had never been asked.Querying everything instead is not viable: ObsPy runs availability queries sequentially per client, so a box's wall time is the sum of its providers' response times. Measured per-provider response for one box ranges from 0.11 s to 4.9 s, and several endpoints hang or serve no station metadata at all.
What this does
Adds a built-in map of where each FDSN archive actually holds stations, on a 5 degree grid. Each box floor-divides its own corners to get the cells it covers, looks those up in an inverted index, and queries only the archives present there, ordered by how many stations each holds inside that box, EarthScope last. No network call, so selection is deterministic for a release and reproducible offline.
providers="auto"is now the default onrun_pipelineanddownload_boxes. An explicit sequence still works unchanged and bypasses the map.providers=Nonenow means auto rather than silently downloading nothing.extra_providers, unioned on top, never region-filtered, ranked last. This is how Raspberry Shake is layered on without replacing automatic selection.SY) and test (XX) networks are never downloaded, in any mode. EarthScope serves 6323SYstations at real-looking coordinates and nothing previously stopped one entering a detection run.KAGSRandUSPpublish station metadata but 404 on dataselect; a station only they report is no longer claimed, since the claim could never be honoured while still blocking every other box from trying. Reported asdiscovered_not_obtainable.MassDownloaderas instances rather than names, removing one service-discovery round trip per provider per box.providers_queriedandproviders_skipped_by_regionper box, so a quiet result is distinguishable from an unasked question.Measured effect
Real network, sampled boxes:
On the 18 of 29 Falcon-9 boxes where auto brings in a provider the old default lacked, stations found go from 19 to 127.
box_015over the Spain/France border goes from 11 to 52, the corridor segment containing Peña-Asensio et al. (2021)'s own optical detection.So the two events show both halves honestly: where the old default was already right (US), auto is much faster for identical data; where it was wrong (Europe), it costs about 22% more time for 6.7x the stations.
Client construction, previously paid per box and now once per run:
INGV8.45 s,EPOSFR2.04 s,GEOFON1.66 s.Breaking change
Auto is not a strict superset of the old defaults:
SCEDCandNCEDCare no longer queried for boxes outside the areas where they hold stations. That is the intent, but it means the "results only grow" guarantee holds only insofar as the map is accurate. On the Shenzhou-15 golden geometry the resolved union still contains all three previous providers.Rollback is passing an explicit provider list, which is unchanged behaviour and needs no code revert.
How the map is built
One EarthScope fedcatalog request, which routes by actual holdings. A provider's own station service is not equivalent: SCEDC republishes metadata for networks it does not serve, which gave it phantom coverage across Alaska. No time filter is applied, so decommissioned stations still occupy cells; filtering to currently-operating stations would drop 54% of GEOFON's footprint and be wrong for reprocessing older events.
tools/build_provider_regions.pyregenerates it. Kept in the repo so the map is auditable rather than asserted, excluded from the wheel. It refuses to write a map in which any inventory provider has zero cells, since such a provider would look supported while never matching a box.Testing
202 passing (42 new). Notable coverage:
lon_min > lon_max), using real geometry fromtrack_to_box_windows. A naive floor range is empty there and would silently resolve to zero providers.PYTHONHASHSEEDvalues, sincebox_cellsreturns a set and ordering ties break on provider name.URL_MAPPINGSkey is classified, so ObsPy adding a provider fails CI rather than silently narrowing what gets queried.Existing fixtures were renamed from network code
"XX"to"ZZ", becauseXXis the reserved test code this change refuses to download.