feat!: region-aware automatic FDSN provider selection

CITATION.cff

@@ -43,8 +43,8 @@ keywords:
   - FDSN
   - SGP4
 license: MIT
-version: 0.2.3
-date-released: '2026-08-26'
+version: 0.4.0
+date-released: '2026-09-11'
 references:
   - type: article
     title: "Reentry and disintegration dynamics of space debris tracked using seismic data"

README.md

@@ -190,6 +190,84 @@ results = run_pipeline(
 )
 ```
 
+### Choosing Data Providers
+
+By default (`providers="auto"`) the library picks providers per box from a
+built-in map of where each FDSN archive actually holds stations. A corridor over
+Italy queries INGV, ORFEUS and GEOFON; one over Texas queries TEXNET. Selection
+is offline and deterministic for a given release, so it adds no latency and
+reproduces exactly.
+
+```python
+# Auto (default) - region-selected, plus EarthScope everywhere
+run_pipeline(..., providers="auto")
+
+# Explicit - exactly these, in this order, no region filtering
+run_pipeline(..., providers=("SCEDC", "NCEDC", "EARTHSCOPE"))
+
+# Auto plus one more, always queried regardless of region
+run_pipeline(..., extra_providers=("RASPISHAKE",))
+```
+
+Providers are ordered by how many stations they hold inside each box, with
+EarthScope last, because a regional archive usually has fresher metadata for its
+own network. `extra_providers` rank last of all. Each box's manifest records
+`providers_queried` and `providers_skipped_by_region`, so a box that found
+nothing can be told apart from a box where the relevant provider was never
+asked.
+
+**Supported with `providers="auto"`** (26 archives). `EARTHSCOPE` is queried for
+every box; the rest are selected by region:
+
+| | | |
+|---|---|---|
+| `EARTHSCOPE` *(always)* | `IESDMC` Taiwan | `NIEP` Romania |
+| `AUSPASS` Australia | `IGN` Spain | `NOA` Greece |
+| `BGR` Germany | `INGV` Italy | `NRCAN` Canada |
+| `BGS` United Kingdom | `IPGP` France, overseas | `ORFEUS` Europe (ODC) |
+| `EPOSFR` France | `KAGSR` Kamchatka \* | `SCEDC` S. California |
+| `ETH` Switzerland | `KNMI` Netherlands | `TEXNET` Texas |
+| `GEOFON` Germany, global | `KOERI` Turkey | `UIB-NORSAR` Norway |
+| `GEONET` New Zealand | `LMU` Germany | `USP` Brazil \* |
+| `ICGC` Catalonia | `NCEDC` N. California | |
+
+\* `KAGSR` and `USP` publish station metadata but serve no waveforms. They are
+still queried, since knowing instruments exist near a corridor is a real result,
+but nothing depends on them for data.
+
+Region labels say where an archive is based, not where its coverage stops:
+`GEOFON` holds instruments across Africa, Asia and Antarctica, and `SCEDC`
+reaches well beyond Southern California. The selection map is measured from real
+holdings rather than drawn from these labels.
+
+Not selected automatically, but available by name: `RASPISHAKE` (citizen
+science), `IRISPH5` (nodal), `USGS`/`EMSC`/`ISC` (event catalogues only), and
+`EIDA` (a router). `IRIS`, `EARTHSCOPE+USGS`, `GFZ`, `ODC` and `RESIF` are alternative
+names for archives already listed.
+
+#### Querying Raspberry Shake
+
+Adding the provider is not enough on its own. Raspberry Shakes are short-period
+instruments whose channels are `EHZ` and `SHZ`; they have never carried `HHZ` or
+`BHZ`, so the default `channel_priorities` filters them out at both discovery
+and download and you would silently get nothing. Widen the channels too:
+
+```python
+run_pipeline(
+    ...,
+    extra_providers=("RASPISHAKE",),
+    channel_priorities=("HHZ", "BHZ", "EHZ", "SHZ"),
+)
+```
+
+Appending rather than replacing matters: `channel_priorities` is a priority
+list, so professional stations still match `HHZ` first and only the shakes fall
+through to `EHZ`. (`ENZ` is deliberately omitted — every Shake with an
+accelerometer also has `EHZ`, so it would never be selected.)
+
+Note that Raspberry Shake data is high volume and variable quality, and the
+default gap rejection discards a meaningful fraction of it.
+
 ### Two-Step Workflow
 
 Download first, process later with different parameters:

docs/api/download.rst

@@ -3,7 +3,7 @@ download
 
 The ``download`` module handles waveform acquisition from FDSN seismic data providers. It runs in three phases for each box:
 
-**Phase 1** — inventory query. The library queries each configured provider for the list of stations that exist within the box's geographic bounds. This is a fast, metadata-only request.
+**Phase 1** — inventory query. The library queries the providers selected for that box (see :doc:`providers`) for the list of stations that exist within its geographic bounds. This is a fast, metadata-only request. Clients are constructed once per run and reused, rather than rebuilt per box.
 
 **Phase 2** — distance filtering. Each candidate station is checked against the corridor distance threshold, keeping only stations within ``corridor_km`` of the actual ground track.
 

docs/api/index.rst

@@ -7,6 +7,7 @@ API Reference
    pipeline
    track
    tiling
+   providers
    download
    processing
    stations

docs/api/pipeline.rst

@@ -39,7 +39,8 @@ Parameters Reference
 
 **Download**
 
-- ``providers`` (default: ``("EARTHSCOPE", "SCEDC", "NCEDC")``) — FDSN data providers to query
+- ``providers`` (default: ``"auto"``) — FDSN data providers to query. ``"auto"`` selects them per box from where each archive actually holds stations (see :doc:`providers`); an explicit sequence replaces that entirely and is used in the order given. ``None`` also means ``"auto"``
+- ``extra_providers`` (default: ``()``) — queried for every box in addition to the above, never region-filtered, ranked last. Use this to layer a citizen-science network onto automatic selection
 - ``channel_priorities`` (default: ``("HHZ", "BHZ")``) — channel preference order
 - ``location_priorities`` (default: ``("", "00", "10", "20")``) — location code preference order
 - ``overwrite_existing`` (default: ``False``) — if ``False``, existing waveforms are skipped

docs/api/providers.rst

@@ -0,0 +1,94 @@
+providers
+=========
+
+The ``providers`` module decides which FDSN data providers are worth querying for each download box.
+
+Why this is not simply "query everything": ObsPy's ``MassDownloader`` runs availability queries **sequentially** per client, so a box's wall time is the *sum* of its providers' response times rather than the maximum. Measured per-provider response for a single box ranges from 0.11 s to nearly 5 s, and several endpoints in ``URL_MAPPINGS`` hang or serve no station metadata at all. Across a 30-box event, querying every provider for every box is not affordable for a re-entry-triggered run.
+
+Selection is offline. The module ships a map of the grid cells where each provider actually holds stations, inverted once at import into a cell-to-provider index, so resolving a box is a couple of dictionary lookups and never iterates the provider list. Cost is proportional to the number of cells a box covers, not to how many providers exist, and no network request is involved. Provider choice is therefore deterministic for a given release and reproducible offline.
+
+Supported providers
+-------------------
+
+With ``providers="auto"`` these 26 archives are supported:
+
+================  =====================================
+Provider          Region
+================  =====================================
+``EARTHSCOPE``    Global, **queried for every box**
+----------------  -------------------------------------
+``AUSPASS``       Australia
+``BGR``           Germany
+``BGS``           United Kingdom
+``EPOSFR``        France, mainland
+``ETH``           Switzerland
+``GEOFON``        Germany, global deployments
+``GEONET``        New Zealand
+``ICGC``          Catalonia
+``IESDMC``        Taiwan
+``IGN``           Spain
+``INGV``          Italy
+``IPGP``          France, overseas and volcano networks
+``KAGSR``         Kamchatka, Russia (station metadata only)
+``KNMI``          Netherlands
+``KOERI``         Turkey
+``LMU``           Germany
+``NCEDC``         Northern California
+``NIEP``          Romania
+``NOA``           Greece
+``NRCAN``         Canada
+``ORFEUS``        European aggregator (ODC)
+``SCEDC``         Southern California
+``TEXNET``        Texas
+``UIB-NORSAR``    Norway
+``USP``           Brazil (station metadata only)
+================  =====================================
+
+The region label is where the archive is based, not a boundary. Actual coverage is the measured cell footprint and is often much wider, which is why the map is measured rather than hand-drawn.
+
+``KAGSR`` and ``USP`` publish station metadata but return 404 for waveforms. They are still queried, since knowing instruments exist near a corridor is a meaningful result, but nothing depends on them for data.
+
+Not selected automatically
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- ``RASPISHAKE`` — citizen-science network, available via ``extra_providers``. Its ``EHZ``/``SHZ`` channels also need ``channel_priorities`` widened to be reachable at all.
+- ``IRISPH5`` — nodal experiment data, which can match enormous requests.
+- ``USGS``, ``EMSC``, ``ISC`` — event catalogues, no station metadata.
+- ``EIDA`` — a router holding no data of its own, routing to members already listed.
+- ``IRIS``, ``IRISDMC``, ``EARTHSCOPE+USGS``, ``GFZ``, ``ODC``, ``RESIF`` — alternative names for archives already listed. Passing one is fine; it will not be queried twice. (``EARTHSCOPE+USGS`` is EarthScope with only its *event* service redirected to USGS, which this library does not use.)
+
+Any of these can still be requested by name through ``providers`` or ``extra_providers``.
+
+Selection modes
+---------------
+
+``providers="auto"`` (the default, and what ``None`` now means) selects by region from the built-in inventory and always appends EarthScope. An explicit sequence replaces automatic selection entirely and is used in the caller's own order. ``extra_providers`` is unioned on top of whichever of those applies, is never region-filtered, and ranks last.
+
+Ordering
+--------
+
+The resolved list is ordered by how many stations each provider holds *inside that box*, then EarthScope, then any extras. Order is priority for ``MassDownloader``: the first provider holding a station wins. Sorting by in-box station count is what puts the genuinely local archive first — for a Southern California box, alphabetical order would rank LMU (3 stations there) above SCEDC (over 3000).
+
+.. note::
+   Not every provider serves waveforms. ``KAGSR`` and ``USP`` offer station metadata but return 404 on dataselect. They are still queried, because "instruments exist near this corridor" is a real result, but a station only they report is **not claimed**: ``MassDownloader`` requires both a station and a dataselect service and would drop the provider, so the claim could never be honoured while still preventing any other box from trying. Those stations are reported per box as ``discovered_not_obtainable``, distinct from ``claimed_not_downloaded``.
+
+.. note::
+   Stations in the reserved ``SY`` (synthetic) and ``XX`` (test) networks are never downloaded, in any selection mode, including when the caller names their host provider explicitly. EarthScope serves thousands of ``SY`` stations at real-looking coordinates and nothing else in the pipeline would stop one being treated as a candidate detection.
+
+Regenerating the map
+--------------------
+
+The map is a source constant, generated by ``tools/build_provider_regions.py`` and committed. That script is kept in the repository so the map is auditable and reproducible rather than asserted, but it is excluded from the installed package: nothing at runtime should reach the network.
+
+.. code-block:: console
+
+   python tools/build_provider_regions.py
+
+It takes its data from a single EarthScope fedcatalog request, which routes by *actual holdings*. A provider's own station service is not equivalent — SCEDC republishes metadata for networks it does not serve, which produced phantom coverage across Alaska in an earlier draft. Inventory providers absent from the fedcatalog response fall back to their own station service, and that set is computed each run rather than hardcoded. The generator fails rather than emitting a map in which any inventory provider has no cells, since such a provider would look supported while never matching a box.
+
+Run it when ObsPy adds a provider (a test fails when that happens), when a provider's coverage is known to have changed materially, or periodically. Review the diff before committing: it is a versioned artifact, and ``provider_regions_generated_utc`` in each run manifest records which build produced that run's selection.
+
+.. automodule:: groundtrack.providers
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/changelog.rst

@@ -1,6 +1,23 @@
 Changelog
 =========
 
+0.4.0 (2026-09-11)
+------------------
+
+- **Provider selection is now automatic and region-aware, and this changes default behaviour.** ``providers`` defaults to ``"auto"`` on both ``run_pipeline`` and ``download_boxes``, replacing the hardcoded ``("EARTHSCOPE", "SCEDC", "NCEDC")`` and ``("EARTHSCOPE",)``. Each box queries providers chosen from a built-in map of where each archive actually holds stations. A corridor over Italy previously queried three US archives and found one station; it now queries INGV, ORFEUS and GEOFON. Passing an explicit ``providers=[...]`` sequence keeps the old behaviour exactly and bypasses the map.
+- Auto selection is **not** a strict superset of the old defaults: SCEDC and NCEDC are no longer queried for boxes outside the areas where they hold stations. On the Shenzhou-15 reference geometry the resolved union still contains all three previous providers, and the mean is 2.0 providers per box against 3 before, so it is fewer queries per box while covering more of the world.
+- Added ``extra_providers``, unioned on top of whatever ``providers`` resolves to, never region-filtered, and ranked last. This is how a citizen-science network is layered onto automatic selection without replacing it.
+- ``providers=None`` now means ``"auto"``. It previously produced an empty provider list, so a run would silently download nothing.
+- Resolved providers are ordered by how many stations each holds inside that box, with EarthScope last, because ``MassDownloader`` treats the list as a priority order and a regional archive usually has fresher metadata for its own network.
+- Stations in the reserved ``SY`` (synthetic) and ``XX`` (test) networks are never downloaded, in any selection mode. EarthScope serves thousands of ``SY`` stations at real-looking coordinates and nothing previously stopped them entering a run as candidate detections.
+- Provider capability is tracked: ``KAGSR`` and ``USP`` serve station metadata but no waveforms. They are still queried for discovery, but a station only such a provider reports is no longer claimed, since the claim could never be honoured and would prevent any other box from trying. These are reported per box as ``discovered_not_obtainable``, distinct from ``claimed_not_downloaded``.
+- FDSN clients are now built once per run and passed to ``MassDownloader`` as instances rather than names, removing one service-discovery round trip per provider per box.
+- Manifests gain ``provider_selection_mode``, ``extra_providers`` and ``provider_regions_generated_utc`` at run level, and ``providers_queried`` and ``providers_skipped_by_region`` per box, so a box that found nothing can be distinguished from a box where the relevant provider was never asked. The existing run-level ``providers`` key is retained as the union across boxes.
+- Added ``tools/build_provider_regions.py`` to regenerate the region map from a single EarthScope fedcatalog request. Kept in the repository for auditability, excluded from the wheel.
+
+.. note::
+   Querying Raspberry Shake requires widening ``channel_priorities`` as well as adding the provider. Shakes are short-period instruments carrying ``EHZ``/``SHZ`` and have never carried ``HHZ`` or ``BHZ``, so the default channel priorities filter them out at both discovery and download. Use ``channel_priorities=("HHZ", "BHZ", "EHZ", "SHZ")`` alongside ``extra_providers=("RASPISHAKE",)``.
+
 0.3.0 (2026-08-27)
 ------------------
 

docs/concepts.rst

@@ -31,9 +31,18 @@ Before passing boxes to the download stage, groundtrack automatically drops any
 Two-Phase Download
 ------------------
 
-The download stage runs in two phases for each box:
+Choosing Which Providers to Ask
+-------------------------------
 
-**Phase 1** queries FDSN providers for the station inventory within the box's geographic bounds. This is a cheap metadata-only request that tells us which stations exist in the area.
+Before a box can be queried at all, something has to decide *which* of the roughly thirty FDSN archives to ask. Asking a fixed few is fast but wrong outside their region: a corridor over Italy queried against three American archives finds almost nothing. Asking all of them is correct but slow, because ObsPy issues those availability queries one after another rather than in parallel, so a box's wall-clock time is the sum of its providers' response times. Individual providers range from about 0.2 to 5 seconds, and a re-entry event can have thirty or more boxes.
+
+Groundtrack resolves this with a built-in map of where each archive actually holds stations, recorded on a 5 degree grid. A box floor-divides its own corners to get the handful of grid cells it covers, looks those up, and queries only the archives present in them, ordered by how many stations each holds inside that box. EarthScope is always queried and always last, so a box never resolves to nothing.
+
+The lookup is arithmetic and a couple of dictionary reads, with no network request, so provider choice is deterministic for a given release and reproducible offline. See :doc:`api/providers` for how the map is built and regenerated.
+
+The download stage then runs in two phases for each box:
+
+**Phase 1** queries the selected providers for the station inventory within the box's geographic bounds. This is a cheap metadata-only request that tells us which stations exist in the area.
 
 **Phase 2** filters that inventory by the actual cross-track distance from each station to the ground track. Only stations within the corridor threshold are kept for waveform download.
 

docs/quickstart.rst

@@ -23,6 +23,8 @@ The simplest call downloads waveform data for a re-entry event without any proce
 
 This runs the full pipeline — fetching the TLE, propagating the ground track, tiling it into download boxes, querying FDSN providers for nearby stations, and downloading the raw waveforms. Everything gets written to ``data/outputs/shenzhou15_reentry/``.
 
+Providers are chosen automatically per box, from a built-in map of where each archive actually holds stations, so a corridor anywhere in the world queries the archives relevant to it without you naming them. Pass ``providers=(...)`` to override that, or ``extra_providers=(...)`` to add to it.
+
 The ``results`` dictionary contains:
 
 - ``results["track"]`` — the propagated ground track and TLE metadata

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "groundtrack"
-version = "0.3.0"
+version = "0.4.0"
 description = "Seismic waveform pipeline for orbital reentry events"
 authors = [{name = "Joseph Steinberg", email = "jstein58@jh.edu"}]
 requires-python = ">=3.10"
@@ -26,6 +26,18 @@ test = [
     "pytest-cov",
 ]
 
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+# Only the package under src/ ships. tools/ holds build_provider_regions.py,
+# which regenerates the provider region map from live FDSN services. It is kept
+# in the repo so the map is auditable and reproducible rather than asserted, but
+# it must not be installed: the library resolves providers entirely offline and
+# nothing at runtime should reach the network.
+[tool.setuptools.packages.find]
+where = ["src"]
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 addopts = "-ra"

src/groundtrack/__init__.py

@@ -5,11 +5,12 @@
 using orbital ground tracks and seismic station data.
 """
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 # Allow user to import below functions directly from the package
 from .download import download_boxes 
 from .stations import load_and_filter_stations
+from .providers import resolve_providers
 from .tiling import track_to_box_windows, box_windows_to_download_requests, filter_ocean_boxes
 from .processing import process_boxes, process_box, process_stream
 from .pipeline import run_pipeline