Skip to content

feat: available + compliant criteria evaluation and seal status logic update - #1819

Open
cka-y wants to merge 6 commits into
mainfrom
feat/1784-available+compliant
Open

feat: available + compliant criteria evaluation and seal status logic update#1819
cka-y wants to merge 6 commits into
mainfrom
feat/1784-available+compliant

Conversation

@cka-y

@cka-y cka-y commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary:

Completes the Seal of Reliability criteria set for #1784 by adding the last two evaluators, Available and Compliant, and makes the feed-level outcome say why a feed does not hold the seal. Builds on #1817 (Official / Stable / Fresh) and follows the algorithm in #1761. Fresh / continuous coverage remains out of scope and is tracked by #1782.

Three changes:

  1. Available (available.py) — the latest gtfs_feed_availability_check in a rolling 24h window has success = TRUE. A window with no check at all is unknown, not a failure: it means we did not look, not that the feed was down. 14-day grace period, serves probation.

  2. Compliant (compliant.py) — total_error = 0 on the latest validation report of the feed's latest dataset. Scoped to the latest dataset rather than to the feed: a verdict on a superseded dataset does not describe what we are serving today. A latest dataset with no report yet (unvalidated, or validation lagging publication) is unknown, which freezes the criterion at its last confirmed verdict instead of failing it. 30-day grace period, serves probation.

  3. Feed-level seal_statushas_seal is a boolean and cannot distinguish judged and did not qualify from cannot be judged yet. roll_up_seal_status in shared/common/seal_criteria.py now derives a four-value status: granted, not_granted, unknown (at least one criterion has never produced a verdict), never_evaluated (none ever has). It is derived, not stored — a pure function of confirmed_status plus probation on rows both the job and the API already hold — so there is no schema change in this PR. Exposed on GET /v1/gtfs_feeds/{id}/reliability as a new optional field.

Every criterion evaluation also now emits a log line (feed, criterion, observed, confirmed, previous confirmed, phase, reason), plus one per feed for the roll-up. The run report gains seal_status per feed and a seal_status_counts summary.

Expected behavior:

  • A feed meeting every criterion is granted and holds the seal, as before.
  • A feed whose criteria were all judged but does not qualify is not_granted, has_seal: false.
  • A feed with any criterion that has never produced a verdict is unknown, has_seal: false — it is not denied the seal, it just cannot be decided yet. On a fresh environment most feeds sit here until availability checks and validation reports exist.
  • A transient upstream outage does not move the seal: a criterion that already has a stored verdict keeps it through an unknown run, so one bad night for the availability or validation job cannot blank the seal across the catalogue.
  • GET /v1/gtfs_feeds/{id}/reliability returns the new seal_status alongside the existing has_seal and the six criteria.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title Feat/1784 available+compliant feat: available + compliant criteria evaluation and seal status logic update Aug 27, 2026
@cka-y
cka-y marked this pull request as ready for review August 27, 2026 20:32
@cka-y cka-y linked an issue Aug 27, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seal of reliability: Implement other criteria evaluation

1 participant