Skip to content

tools: census guard must cover the GUARANTEES breakdown, not only the README (fixes #4454) - #4455

Open
jorge123255 wants to merge 1 commit into
limitlessprojects:mainfrom
jorge123255:census-guard-guarantees-breakdown
Open

jorge123255 wants to merge 1 commit into
limitlessprojects:mainfrom
jorge123255:census-guard-guarantees-breakdown

Conversation

@jorge123255

Copy link
Copy Markdown

Fixes #4454.

check_breakdown() policed the corpus verdict breakdown in one file only (tests/checker-corpus/README.md). The GUARANTEES.md evidence inventory restates the same breakdown and nothing checked it, so it had drifted on main — 67 REJECT / 32 ACCEPT / 2 ACCEPT-MANAGED against a true 92/45/2, a breakdown that does not even sum to the 139 programs stated in the same paragraph. The gate printed the correct numbers and exited 0 on the same run.

Change

  • README_BREAKDOWN → BREAKDOWN_SURFACES: a list of (path, regex, group order), so the README sentence and the GUARANTEES sentence (different word order, and no total in the same phrase) are both checked by one code path. A per-surface group order keeps the comparison honest instead of assuming one sentence shape.
  • The success line counts every surface (now 8, not a hardcoded total + 1).
  • GUARANTEES.md's census numbers corrected to 92/45/2.
  • Two new cases: a stale GUARANTEES breakdown, and a breakdown that does not add up to its own total. The existing stale-total case now also carries a correct breakdown line so it keeps asserting exactly one problem.

Evidence

Against the unfixed doc the extended guard exits 1:

ERROR: corpus census drift — a claim disagrees with the corpus:
  - GUARANTEES.md: line 72: breakdown says ACCEPT 32 · REJECT 67, corpus holds ACCEPT 45 · REJECT 92

The corpus holds 139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92.
Update each surface above to that number.

With the fix:

$ python3 tools/check_corpus_census.py
OK: all 8 corpus census surfaces agree with the corpus (139 programs: ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92).

$ python3 tools/test_check_corpus_census.py
Ran 23 tests in 0.067s
OK

tools/check_guarantees.py (table, registry, corpus, doc agree), tools/check_claims_links.py and tools/check_test_registration.py stay green. No corpus fixture and no compiler code touched, so the Rust suites are unaffected.

Same class as #3724 — later contradictory breakdown ignored — different surface.

Co-authored-by: OpenBot <335604065+openbotgit[bot]@users.noreply.github.com>

… README

`check_breakdown()` only policed the verdict breakdown in
`tests/checker-corpus/README.md`. The GUARANTEES.md evidence inventory
restates the same breakdown in prose ("Current census: **67 REJECT / 32
ACCEPT / 2 ACCEPT-MANAGED**") and nothing checked it, so it had already
drifted from the corpus on main: the true census is 92 REJECT / 45 ACCEPT /
2 ACCEPT-MANAGED, and the stale numbers do not even add up to the 139
programs stated in the same paragraph.

The gate reproduced the whole bug: on `b1627e8` it printed
"OK: all 7 corpus census surfaces agree with the corpus (139 programs:
ACCEPT 45 · ACCEPT-MANAGED 2 · REJECT 92)" while the doc next to it said
67/32/2 — the tool knew the right numbers and passed anyway.

`README_BREAKDOWN` becomes `BREAKDOWN_SURFACES`, a list of
(path, regex, group order) so the README sentence and the GUARANTEES
sentence (different word order, no total in the same phrase) are both
checked by one code path. The OK line now counts every surface. The
GUARANTEES numbers are corrected, and the suite pins both new failures: a
stale GUARANTEES breakdown, and a breakdown that does not add up to its own
total.

Same class as limitlessprojects#3724 (later contradictory README breakdown ignored), other
surface.

Co-authored-by: OpenBot <335604065+openbotgit[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tools: census guard never checks the GUARANTEES verdict breakdown (doc already drifted 67/32/2 vs 92/45/2)

1 participant