Repository navigation
tools: census guard must cover the GUARANTEES breakdown, not only the README (fixes #4454) - #4455
Open
jorge123255 wants to merge 1 commit into
Open
jorge123255 wants to merge 1 commit into
jorge123255 wants to merge 1 commit into
Conversation
… README
`check_breakdown()` only policed the verdict breakdown in
`tests/checker-corpus/README.md`. The GUARANTEES.md evidence inventory
restates the same breakdown in prose ("Current census: **67 REJECT / 32
ACCEPT / 2 ACCEPT-MANAGED**") and nothing checked it, so it had already
drifted from the corpus on main: the true census is 92 REJECT / 45 ACCEPT /
2 ACCEPT-MANAGED, and the stale numbers do not even add up to the 139
programs stated in the same paragraph.
The gate reproduced the whole bug: on `b1627e8` it printed
"OK: all 7 corpus census surfaces agree with the corpus (139 programs:
ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92)" while the doc next to it said
67/32/2 — the tool knew the right numbers and passed anyway.
`README_BREAKDOWN` becomes `BREAKDOWN_SURFACES`, a list of
(path, regex, group order) so the README sentence and the GUARANTEES
sentence (different word order, no total in the same phrase) are both
checked by one code path. The OK line now counts every surface. The
GUARANTEES numbers are corrected, and the suite pins both new failures: a
stale GUARANTEES breakdown, and a breakdown that does not add up to its own
total.
Same class as limitlessprojects#3724 (later contradictory README breakdown ignored), other
surface.
Co-authored-by: OpenBot <335604065+openbotgit[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4454.
check_breakdown()policed the corpus verdict breakdown in one file only (tests/checker-corpus/README.md). The GUARANTEES.md evidence inventory restates the same breakdown and nothing checked it, so it had drifted onmain— 67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED against a true 92/45/2, a breakdown that does not even sum to the 139 programs stated in the same paragraph. The gate printed the correct numbers and exited 0 on the same run.Change
README_BREAKDOWN→BREAKDOWN_SURFACES: a list of(path, regex, group order), so the README sentence and the GUARANTEES sentence (different word order, and no total in the same phrase) are both checked by one code path. A per-surface group order keeps the comparison honest instead of assuming one sentence shape.Evidence
Against the unfixed doc the extended guard exits 1:
With the fix:
tools/check_guarantees.py(table, registry, corpus, doc agree),tools/check_claims_links.pyandtools/check_test_registration.pystay green. No corpus fixture and no compiler code touched, so the Rust suites are unaffected.Same class as #3724 — later contradictory breakdown ignored — different surface.
Co-authored-by: OpenBot <335604065+openbotgit[bot]@users.noreply.github.com>