Skip to content

feat: evaluate fresh + stable criteria - #1817

Merged
cka-y merged 3 commits into
mainfrom
feat/1784-stable+fresh
Aug 27, 2026
Merged

feat: evaluate fresh + stable criteria#1817
cka-y merged 3 commits into
mainfrom
feat/1784-stable+fresh

Conversation

@cka-y

@cka-y cka-y commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary:

Implements two of the four criteria from #1784Stable and Fresh (future coverage) — following the algorithm spike in [#1761](#1761 (comment)). They join Official (#1783) in the evaluator registry the nightly job iterates, so the has_seal roll-up now weighs three criteria instead of one. Available and Compliant remain open under #1784; Fresh / continuous coverage stays with #1782.

Two deliberate divergences from the spike, both worth reviewer attention:

  • Stable reads feed.created_at, not feed_reliability_seal.created_at. The question is whether we hold six months of history for the feed, so the clock is the feed's own age. Reading the seal row would have meant no feed could qualify until 180 days after the job's first run, and would have made Stable unreplayable, since that row's created_at is stamped by the database clock rather than the run's now.
  • "Latest dataset" is resolved as of the run's now, via downloaded_at, rather than from the gtfsfeed.latest_dataset_id pointer. That pointer always names the newest dataset that exists today, so a replay reading it would judge a past day against data published after it.

This PR also centralizes the seal's policy surface, which was duplicated between the job and the read API. api/src/shared/common/seal_criteria.py is now the only place any policy value is defined — the criterion names, CriterionStatus, CriterionPhase, the grace and probation maps, and the two check-internal windows (TRACKING_PERIOD, FUTURE_COVERAGE_HORIZON). Evaluators no longer declare their own windows: CriterionEvaluator resolves both from self.name via grace_period_for() / probation_period_for(), so a criterion cannot debounce one way for the job and another for the API. The API side drops its local STATUS_PASS/STATUS_FAIL string constants for the shared enum, and tasks/seal_of_reliability/criteria.py is deleted rather than left as a re-export shim.

No schema change: seal_criterion_name already declared all six values. No change to any public endpoint or response shape.

Expected behavior:

Stable — passes when feed.created_at <= now - 180 days and is_producer_url_unstable IS NOT TRUE. No grace period and no probation, so it is a point-in-time check like Official: it fails the day the unstable flag is set and clears the day it is cleared. It never returns unknown — both inputs are columns on the feed row. A producer URL change creates a new feed with its own created_at, so the six months restart with it. Feeds already in the catalog for six months qualify on the very first run.

Fresh (future coverage) — passes when the latest dataset as of now has service_date_range_end >= now + 7 days. 14-day grace period, 180-day probation. Returns not_applicable for a seasonal feed, which withdraws the criterion from the roll-up rather than failing it. Returns unknown in two distinct cases, reported separately: the feed had no dataset at all as of the run, or it had one whose service_date_range_end was never extracted. Neither is read as a failure — an upstream gap must not deny seals across the catalog.

A criterion that has never produced a verdict gets no grace period, so a feed whose first Fresh verdict is a failure is a confirmed failure that same day rather than passing for two weeks on evidence we do not have.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y
cka-y marked this pull request as ready for review August 26, 2026 15:12
@cka-y cka-y self-assigned this Aug 26, 2026
@cka-y
cka-y requested review from davidgamez and jcpitre August 27, 2026 15:23
Comment thread api/src/shared/common/seal_criteria.py Outdated


# Stable: how long we must have been tracking a feed - measured from its
# `feed_reliability_seal.created_at` - before it can be called stable.

@jcpitre jcpitre Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using feed.created_at, no?

Suggested change
# `feed_reliability_seal.created_at` - before it can be called stable.
# `feed.created_at` - before it can be called stable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes you're right. fixed ✅

Comment thread api/src/shared/db_models/reliability_criterion_impl.py
@cka-y
cka-y requested a review from jcpitre August 27, 2026 17:14
return self in (CriterionStatus.PASS, CriterionStatus.FAIL)


class CriterionPhase(str, Enum):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@cka-y
cka-y merged commit acb93ec into main Aug 27, 2026
24 checks passed
@cka-y
cka-y deleted the feat/1784-stable+fresh branch August 27, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants