feat: evaluate fresh + stable criteria - #1817
Merged
Merged
Conversation
jcpitre
reviewed
Aug 27, 2026
|
|
||
|
|
||
| # Stable: how long we must have been tracking a feed - measured from its | ||
| # `feed_reliability_seal.created_at` - before it can be called stable. |
Collaborator
There was a problem hiding this comment.
We're using feed.created_at, no?
Suggested change
| # `feed_reliability_seal.created_at` - before it can be called stable. | |
| # `feed.created_at` - before it can be called stable. |
Contributor
Author
There was a problem hiding this comment.
yes you're right. fixed ✅
jcpitre
reviewed
Aug 27, 2026
davidgamez
approved these changes
Aug 27, 2026
| return self in (CriterionStatus.PASS, CriterionStatus.FAIL) | ||
|
|
||
|
|
||
| class CriterionPhase(str, Enum): |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Implements two of the four criteria from #1784 — Stable and Fresh (future coverage) — following the algorithm spike in [#1761](#1761 (comment)). They join
Official(#1783) in the evaluator registry the nightly job iterates, so thehas_sealroll-up now weighs three criteria instead of one. Available and Compliant remain open under #1784; Fresh / continuous coverage stays with #1782.Two deliberate divergences from the spike, both worth reviewer attention:
feed.created_at, notfeed_reliability_seal.created_at. The question is whether we hold six months of history for the feed, so the clock is the feed's own age. Reading the seal row would have meant no feed could qualify until 180 days after the job's first run, and would have made Stable unreplayable, since that row'screated_atis stamped by the database clock rather than the run'snow.now, viadownloaded_at, rather than from thegtfsfeed.latest_dataset_idpointer. That pointer always names the newest dataset that exists today, so a replay reading it would judge a past day against data published after it.This PR also centralizes the seal's policy surface, which was duplicated between the job and the read API.
api/src/shared/common/seal_criteria.pyis now the only place any policy value is defined — the criterion names,CriterionStatus,CriterionPhase, the grace and probation maps, and the two check-internal windows (TRACKING_PERIOD,FUTURE_COVERAGE_HORIZON). Evaluators no longer declare their own windows:CriterionEvaluatorresolves both fromself.nameviagrace_period_for()/probation_period_for(), so a criterion cannot debounce one way for the job and another for the API. The API side drops its localSTATUS_PASS/STATUS_FAILstring constants for the shared enum, andtasks/seal_of_reliability/criteria.pyis deleted rather than left as a re-export shim.No schema change:
seal_criterion_namealready declared all six values. No change to any public endpoint or response shape.Expected behavior:
Stable — passes when
feed.created_at <= now - 180 daysandis_producer_url_unstable IS NOT TRUE. No grace period and no probation, so it is a point-in-time check like Official: it fails the day the unstable flag is set and clears the day it is cleared. It never returnsunknown— both inputs are columns on the feed row. A producer URL change creates a new feed with its owncreated_at, so the six months restart with it. Feeds already in the catalog for six months qualify on the very first run.Fresh (future coverage) — passes when the latest dataset as of
nowhasservice_date_range_end >= now + 7 days. 14-day grace period, 180-day probation. Returnsnot_applicablefor a seasonal feed, which withdraws the criterion from the roll-up rather than failing it. Returnsunknownin two distinct cases, reported separately: the feed had no dataset at all as of the run, or it had one whoseservice_date_range_endwas never extracted. Neither is read as a failure — an upstream gap must not deny seals across the catalog.A criterion that has never produced a verdict gets no grace period, so a feed whose first Fresh verdict is a failure is a confirmed failure that same day rather than passing for two weeks on evidence we do not have.
Please make sure these boxes are checked before submitting your pull request - thanks!
./scripts/api-tests.shto make sure you didn't break anything