Skip to content

fix(ca): the coverage gate could not run, in any tree (#507)#512

Closed
SUaDtL wants to merge 1 commit into
chore/vitest-4-ca-toolsfrom
feat/507-coverage-tooling
Closed

fix(ca): the coverage gate could not run, in any tree (#507)#512
SUaDtL wants to merge 1 commit into
chore/vitest-4-ca-toolsfrom
feat/507-coverage-tooling

Conversation

@SUaDtL

@SUaDtL SUaDtL commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Closes #507. Step 3 of the chain; stacked on #510 (base is chore/vitest-4-ca-tools, so the diff stays clean — GitHub retargets to main when #510 merges).

The defect

tdd Phase 5 and refactor Phase 2/6 instruct "run the coverage command from tech-stack.md", and both carry a hard rule forbidding guessing one. No coverage command existed — no provider in any of the three TypeScript trees, no script, and tech-stack.md contained the word "coverage" zero times.

So every run reached the phase, found nothing to run, and passed through on a gap. A BLOCK gate that cannot execute is worse than an absent one — it reads as satisfied in every lane. Same defect class as #501 (five suites running on nothing) and #506 (an assertion that restated the bug).

refactor was the worse casualty. Phase 2 is "Behavioral parity coverage proof" — the gate that justifies the entire lane by proving pre-existing tests can detect a behaviour change before production code is touched. It rested wholly on a command that did not exist, leaving the lane running on Phase 5's "zero pre-existing tests modified" alone.

What lands

Tooling@vitest/coverage-v8@4.1.9 in all three trees, npm run coverage in each, scope/provider/reporters in each vitest.config.ts so the invocation is argument-free and platform-identical. ca/tools gets its first vitest.config.ts; coverage output is gitignored.

The metric, named. The threshold table said only "minimum coverage". That omission was load-bearing, not cosmetic — a report gives four numbers that disagree:

tree lines branches statements functions
plugins/ca/tools 67.22% 59.46% 63.91% 56.77%
plugins/ca-pi/tools 85.37% 78.73% 80.12% 85.31%
plugins/ca-sandbox/tools 86.13% 79.96% 85.49% 81.59%

ca-pi is compliant at maturity 3 on lines and non-compliant on branches. "≥ 85%" with no column named is not a threshold anyone can be held to.

Now lines and branches, both binding. Lines catches code no test reaches — which is exactly what #504 turned out to be, a catch with zero executions inside a 750-test suite. Branches catches the untaken half of a condition a test does reach, which lines alone reports as covered. Statements duplicates lines; functions is noisiest at small counts.

A guard against recurrence. GateCommandTest asserts that any command a gate reads from tech-stack.md is actually defined there. Verified in both directions rather than assumed — run against the pre-fix file it fires, and it correctly identifies both demanders (tdd, refactor).

Deliberate non-choices

  • Threshold not encoded in the vitest configs. The skills apply it from CONTEXT.md stage: against maturity-coverage.md. Copying the number into three package configs forks the source of truth, and the copies drift the first time the stage moves.
  • No CI job enforces coverage. ca/tools is below the stage-2 floor, so wiring it into required CI would block every merge in the repo on an unrelated backfill. It is an orchestrator gate, which is what the skills already assume.
  • stage: stays at 2. Raising to 3 today blocks all three trees — ca-pi fails branches at 78.73, ca-sandbox at 79.96. A gate nothing can pass gets overridden, and an overridden gate teaches readers it is noise.
  • The ca/tools shortfall is plugins/ca/tools is below the stage-2 coverage floor on lines and branches #511, not this PR. Folding it in would turn tdd Phase 5 cannot run: no coverage command exists for any plugin #507 into "write several hundred statements of tests before anything ships".
  • Python hooks stay unmeasured. Both skills now say so explicitly, so the phase is recorded as having no numeric floor rather than silently skipped.

Versioning

The surface renders into three hosts, so this is a payload change for ca, ca-codex and ca-pi — not just a tools/ edit.

  • ca-pi 0.1.34 → 0.1.35 + changelog + regenerated root manifest. Its guard demands a strict advance on any payload change regardless of tag: Pi payload, version, changelog, and root metadata advanced together: 0.1.34 -> 0.1.35.
  • ca (2.9.1) and ca-codex (0.3.0) are unpublished (no v2.9.1 / ca-codex-v0.3.0 tag), and their gates permit a payload change on an unpublished version. Entry added under root ## [Unreleased].

Verification

  • npm run coverage green in all three trees; ca/tools still discovers all 324 tests after gaining a config
  • 23 python suites + unittest discover over plugins/ca/hooks/tests + check-plugin-refs.py
  • sync-core.py --check, build-surface.py --check, build-host-packages.py --check — all in sync
  • test_pi_package.py's exact workspace contract updated for the new script and dependency
  • tsc --noEmit clean in all three trees

build-surface --check caught me editing rendered files instead of core/surface/ templates; the edits were moved to source and re-rendered.

https://claude.ai/code/session_01JgdLb6n8mUdkFiKDTA37ML

`tdd` Phase 5 and `refactor` Phase 2/6 instruct "run the coverage command from
`tech-stack.md`", and both forbid guessing one. No coverage command existed -
no provider installed in any of the three TypeScript trees, no script, and the
word "coverage" appeared in tech-stack.md zero times. Every run reached the
phase, found nothing to run, and passed through on a gap.

A BLOCK gate that cannot execute is worse than an absent one: it reads as
satisfied in every lane. Same defect class as #501's five suites that ran on
nothing and #506's assertion that restated the bug.

`refactor` was the worse casualty. Phase 2 is "Behavioral parity coverage
proof" - the gate that justifies the entire lane by showing the pre-existing
tests can detect a behavior change BEFORE production code is touched - and it
rested wholly on a command that did not exist. The lane was running on Phase
5's "zero pre-existing tests modified" alone.

The threshold table never named a metric, and that omission was load-bearing
rather than cosmetic: a report gives four numbers that disagree, so "at least
70%" with no column named is not something anyone can be held to. ca-pi
measured 85.37% lines against 78.73% branches - compliant at maturity 3, or
not, depending purely on which column the reader picked. It is now lines AND
branches, both binding. Lines catches code no test reaches at all, which is
what #504 turned out to be: a `catch` with zero executions inside a 750-test
suite. Branches catches the untaken half of a condition a test does reach.

The threshold is deliberately NOT encoded in the three vitest configs. The
skills apply it from CONTEXT.md `stage:` against maturity-coverage.md; copying
the number into three package configs would fork the source of truth and the
copies would drift the first time the stage moves.

No CI job enforces coverage, also deliberately. `plugins/ca/tools` is below the
stage-2 floor (67.22% lines, 59.46% branches), so wiring it into required CI
would block every merge in the repo on an unrelated backfill. That backfill is
tracked as its own issue rather than absorbed here, which would have turned
#507 into "write several hundred statements of tests".

A contract test now asserts that any command a gate reads from tech-stack.md is
actually defined there. Verified against the pre-fix file rather than assumed:
it fires.

CHANGELOG: tdd Phase 5 and refactor Phase 2/6 can run their coverage gate; the
maturity threshold now specifies lines and branches, both binding.
Closes #507
Refs #511
Claude-Session: https://claude.ai/code/session_01JgdLb6n8mUdkFiKDTA37ML
@SUaDtL
SUaDtL deleted the branch chore/vitest-4-ca-tools July 26, 2026 19:55
@SUaDtL SUaDtL closed this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant