Skip to content

feat!: region-aware automatic FDSN provider selection - #21

Merged
jsteinberg34 merged 8 commits into
mainfrom
add-auto-provider-selection
Sep 14, 2026
Merged

jsteinberg34 merged 8 commits into
mainfrom
add-auto-provider-selection

Conversation

@jsteinberg34

Copy link
Copy Markdown
Owner

Why

Provider lists were hardcoded and US-centric (download_boxes defaulted to ("EARTHSCOPE",), run_pipeline to ("EARTHSCOPE", "SCEDC", "NCEDC")). A corridor over Italy queried three American archives and found one station where 52 exist, with nothing in the output indicating a relevant provider had never been asked.

Querying everything instead is not viable: ObsPy runs availability queries sequentially per client, so a box's wall time is the sum of its providers' response times. Measured per-provider response for one box ranges from 0.11 s to 4.9 s, and several endpoints hang or serve no station metadata at all.

What this does

Adds a built-in map of where each FDSN archive actually holds stations, on a 5 degree grid. Each box floor-divides its own corners to get the cells it covers, looks those up in an inverted index, and queries only the archives present there, ordered by how many stations each holds inside that box, EarthScope last. No network call, so selection is deterministic for a release and reproducible offline.

  • providers="auto" is now the default on run_pipeline and download_boxes. An explicit sequence still works unchanged and bypasses the map. providers=None now means auto rather than silently downloading nothing.
  • New extra_providers, unioned on top, never region-filtered, ranked last. This is how Raspberry Shake is layered on without replacing automatic selection.
  • Synthetic (SY) and test (XX) networks are never downloaded, in any mode. EarthScope serves 6323 SY stations at real-looking coordinates and nothing previously stopped one entering a detection run.
  • Provider capability is tracked. KAGSR and USP publish station metadata but 404 on dataselect; a station only they report is no longer claimed, since the claim could never be honoured while still blocking every other box from trying. Reported as discovered_not_obtainable.
  • Clients are built once per run and passed to MassDownloader as instances rather than names, removing one service-discovery round trip per provider per box.
  • Manifests record providers_queried and providers_skipped_by_region per box, so a quiet result is distinguishable from an unasked question.

Measured effect

Real network, sampled boxes:

providers/box discovery per box stations
Shenzhou-15 3 to 2.42 3.55 s to 1.74 s (-51%) 135 to 135 (identical)
Falcon-9 3 to 2.83 3.24 s to 1.47 s (-55%) see below

On the 18 of 29 Falcon-9 boxes where auto brings in a provider the old default lacked, stations found go from 19 to 127. box_015 over the Spain/France border goes from 11 to 52, the corridor segment containing Peña-Asensio et al. (2021)'s own optical detection.

So the two events show both halves honestly: where the old default was already right (US), auto is much faster for identical data; where it was wrong (Europe), it costs about 22% more time for 6.7x the stations.

Client construction, previously paid per box and now once per run: INGV 8.45 s, EPOSFR 2.04 s, GEOFON 1.66 s.

Breaking change

Auto is not a strict superset of the old defaults: SCEDC and NCEDC are no longer queried for boxes outside the areas where they hold stations. That is the intent, but it means the "results only grow" guarantee holds only insofar as the map is accurate. On the Shenzhou-15 golden geometry the resolved union still contains all three previous providers.

Rollback is passing an explicit provider list, which is unchanged behaviour and needs no code revert.

How the map is built

One EarthScope fedcatalog request, which routes by actual holdings. A provider's own station service is not equivalent: SCEDC republishes metadata for networks it does not serve, which gave it phantom coverage across Alaska. No time filter is applied, so decommissioned stations still occupy cells; filtering to currently-operating stations would drop 54% of GEOFON's footprint and be wrong for reprocessing older events.

tools/build_provider_regions.py regenerates it. Kept in the repo so the map is auditable rather than asserted, excluded from the wheel. It refuses to write a map in which any inventory provider has zero cells, since such a provider would look supported while never matching a box.

Testing

202 passing (42 new). Notable coverage:

  • Antimeridian boxes (lon_min > lon_max), using real geometry from track_to_box_windows. A naive floor range is empty there and would silently resolve to zero providers.
  • Determinism across PYTHONHASHSEED values, since box_cells returns a set and ordering ties break on provider name.
  • A drift test asserting every URL_MAPPINGS key is classified, so ObsPy adding a provider fails CI rather than silently narrowing what gets queried.
  • A test asserting the documented provider table matches the registry.

Existing fixtures were renamed from network code "XX" to "ZZ", because XX is the reserved test code this change refuses to download.

@jsteinberg34 jsteinberg34 self-assigned this Sep 13, 2026
@codecov

codecov Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.70552% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/groundtrack/download.py 90.66% 3 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

@jsteinberg34
jsteinberg34 merged commit fa28b54 into main Sep 14, 2026
5 checks passed
@jsteinberg34
jsteinberg34 deleted the add-auto-provider-selection branch September 14, 2026 00:11
@jsteinberg34 jsteinberg34 changed the title Region-aware automatic FDSN provider selection feat!: Region-aware automatic FDSN provider selection Sep 14, 2026
@jsteinberg34 jsteinberg34 changed the title feat!: Region-aware automatic FDSN provider selection feat!: region-aware automatic FDSN provider selection Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant