Field report: the unearned-green failure classes temper should harden against (13 rules, each from a real incident)

[avrabe]

Background

Temper manages branch protection, rulesets, merge strategy, labels and dependabot config. This issue is a field report against that surface: over roughly one day of continuous automated work on pulseengine/jess, I hit the same failure classes repeatedly, and most of them are things temper either already owns or is the natural home for.

The organising observation, because it makes the whole list statable in one line:

Almost every failure was a green that wasn't earned — a gate that passed without comparing anything, a check that never ran, or a verdict about the wrong commit.

Nothing below is exotic. All of it is mechanically detectable. Two of them produced wrong claims that reached other repos' issue trackers before being caught.

Grouped by how close it sits to temper's current shape.

---

1. Already in temper's wheelhouse — and currently drifted

1.1 jess main has no required status checks. The green board is advisory.

This is the one I'd fix first, and it sits squarely in the rulesets temper already manages.

$ gh api repos/pulseengine/jess/rules/branches/main --jq '.[] | .type'
deletion
non_fast_forward
pull_request
required_signatures

$ gh api .../rules/branches/main --jq '[.[]|select(.type=="required_status_checks")]|length'
0

$ gh api repos/pulseengine/jess/branches/main/protection
404 Branch not protected

Five substantive checks run on every PR there (rivet validate, spar model, scry sound-analysis, Renode smoke, mav_bench). None can block a merge.

• Setting: ruleset rule required_status_checks on the default branch
• Recommended value: the repo's existing check names, strict: true (require branches up to date — see 2.1, which is the same rule from the other direction)
• Why: a repo running five checks and requiring none is worse than one running none, because it looks gated. Every "confirmed green before merging" in that repo's history was honoured by hand; the repo would have accepted a red merge silently.
• REST API: GET/PUT /repos/{owner}/{repo}/rulesets/{id}, rule type required_status_checks
• Suggested temper behaviour: where a repo has workflows producing named checks but no required_status_checks rule, that is drift — report it with the list of checks that do run, so the gap is concrete rather than abstract.

1.2 Also worth auditing across the org: required_approving_review_count: 0

Defensible for a solo/bot-driven repo, but it should be a decision recorded in config, not a default nobody chose.

---

2. Merge-gate semantics — a small new capability, high value

These are the three ways I got a green that was true but meaningless. All are computable from the API temper already talks to.

2.1 Green against a stale base is a verdict about the wrong thing

jess PR #192 was genuinely green — against base=8a3e11b7, while main had moved to 1777461e. rivet validate had never run against the combined state. Merging on that green merges a combination nobody tested. It recurred immediately with #195 after #194 landed.

• Detect: baseRefOid != current base branch head, while checks are green
• Action: mark the green stale; require re-run
• Note: strict: true on required_status_checks enforces exactly this natively, which is why 1.1 and 2.1 are really one fix. Worth calling out explicitly because the failure is invisible without it — the checks are green, they're just green about the past.

2.2 CI goes silent, not red, on a conflicting PR

When a PR is CONFLICTING, GitHub can't build the merge commit and stops running pull_request workflows entirely. gh pr checks then prints no checks reported — a string containing neither "pending" nor "fail". My own monitor read that as ALL GREEN and told the user so.

• Detect: mergeable == CONFLICTING, or zero check runs on a PR head whose workflows should have triggered
• Action: surface as a blocking state in its own right. Never render "no checks" as anything adjacent to green.

2.3 cancel-in-progress + a push burst = no verdict, ever

Five pushes within an hour cancelled each run before a runner picked it up. I then misread the cancelled runs as a starved queue and blamed GitHub Actions capacity in a status report. The runs weren't queued; my next push killed each one.

• Detect: a branch accumulating cancelled runs with no completed run
• Action: say the cancellation came from the concurrency group, not from a human or an outage

---

3. Unearned greens inside repo scripts

A different capability from settings hardening — closer to a lint temper could ship as a reusable workflow. Flagging it because it was the largest category by count, and because every instance was a defect in a checker, not in the code being checked.

3.1 Piped exit codes

./script | tail -3 reports tail's status. A failing script prints a clean-looking tail and scores 0. jess has a standing project rule about this precisely because of an earlier instance; I still hit it twice more.

Lint: CI steps of the form cmd | tail|head|grep where the pipeline status is the gate. Require PIPESTATUS / pipefail / a bare invocation.

3.2 set -o pipefail + grep -q → SIGPIPE false negative

strings -a "$f" | grep -q PATTERN — grep -q exits at the first match, SIGPIPEs the producer, pipeline returns 141. Whether it bites depends on where in the file the match sits, so it's arbitrary per artifact: a 30 KB component wrongly FAILED while a 12 KB one passed. This nearly produced a false defect report against another team's release.

Lint: … | grep -q under pipefail.

3.3 A gate that scores "ok" on an empty comparison set

Two live instances in one script of mine: a tool absent from every source scored "ok" (agreement where nothing was compared), and with all tools absent the script printed "no drift" and exited 0. Separately, a tool present in exactly one source also scored "ok" — one value compared against nothing.

Rule: any gate whose comparison set can be empty must fail loudly. "Nothing to check" is never a pass. If temper ships one lint from this section, this is the one.

3.4 A metric whose failing case cannot occur

A parser reported "zero CRC failures" while its frame extractor silently skipped CRC-invalid frames — the count was structurally guaranteed to be zero. Withdrawn. The withdrawal then never propagated to the artifact's title, which advertised the retracted number for days afterward.

Rule: where a claim is withdrawn in a body, flag any title/summary still asserting it.

3.5 Oracles that exist and gate nothing

$ grep -cE 'appcompose|check-drift|m4-matrix|timer-probe' .github/workflows/ci.yml
0

Four verification scripts in tools/, none referenced by any workflow. One of them exits 1 on main today.

Rule: inventory executable tools/**/*.sh no workflow invokes. An oracle nobody runs is documentation.

---

4. Version hygiene across repos

4.1 Three sources of truth for one tool version, never reconciled

TOOL      PATH      VARVE-PIN   CI-YML
rivet     0.32.0    0.34.0      0.25.0     <- three different versions
spar      -         0.40.0      0.24.0
meld      0.41.3    0.52.0      0.41.3

This produced a wrong bug report to another repo. I filed meld#390 having run 0.41.3 while 0.52.0 was latest — eleven minor versions — and had to correct it publicly. Note CI pinned 0.41.3 as well: CI would not have caught it either.

Rule: compare pinned tool versions in ci.yml (and any lockfile) against the upstream latest release on a schedule; open a PR on drift. Dependabot does this for package ecosystems; nothing does it for FOO_VERSION: vX.Y.Z env pins, which is how this whole org pins its own tools.

4.2 An issue that cites a version should have that version checked

The most avoidable failure of the session was a defect report whose version citation was wrong — and it's trivially checkable.

Rule: when an issue body names <tool> <semver> for a repo in the org, compare against latest and comment if newer. Costs nothing; saves a maintainer reading a report about a fixed bug.

---

5. Claim hygiene — adjacent to #60

Given #60 is about dropping an unbacked "formally verified WebAssembly toolchain" footer, this is evidently already temper's concern. The same failure occurs at commit granularity, and in a safety context it matters more there.

5.1 Emulation described as hardware

Real examples from jess:

• a commit subject reading "the fused image EXECUTES on the RT1176 M7" — no "(Renode)" in subject or body, while its own predecessor commit had it right. Read alone in a log, it asserts silicon.
• hardware/silicon/README.md: "this is the actual chip" — over a run that was entirely Renode, on a board marked "(ordered)".
• an architecture diagram labelling a Renode node "HIL emulation". No hardware is in that loop.

Lint: on hardware|on silicon|actual chip|HIL|on-target|executes on in commit messages and docs, requiring an adjacent qualifier (Renode, emulated, wasmtime, qemu) or an explicit NOT-CLAIMED: line. Cheap regex, maps exactly to the mistakes actually made, and these repos produce safety-case evidence where a log line is read on its own.

5.2 Absolutes contradicted by committed artifacts

A finding asserted a board "has been ABSENT on every loop tick … has never found it" while the same repo contains 77,034 bytes of live telemetry captured from that board. I then repeated the false absolute verbally. The scoped claim ("no code has ever executed on it") was true and sufficient.

Hard to automate properly, but never/always/every in an evidence artifact is a cheap grep worth surfacing for review.

---

6. Repo mechanics

6.1 Stranded branches

Pushed twice with no PR opened. Rule: branches ahead of the default branch with no open PR after N hours.

6.2 Append-only artifact files conflict structurally, and resolution silently drops data

artifacts/findings.yaml grows by appending near one anchor, so concurrent PRs conflict positionally, not semantically. Resolving by taking either side silently discards the other's entries — I nearly lost five corrections that way, and caught it only by diffing artifact-ID sets before and after.

Rule: for declared append-only artifact files, verify no record ID is lost across a conflict resolution. A two-line set-diff; would have caught it instantly.

---

Priority if only part of this gets built

1. 1.1 + 2.1 — required_status_checks with strict: true. One fix, closes the two most dangerous holes, and is pure temper.
2. 3.3 — gates that pass on an empty comparison set. Most common failure by count.
3. 4.1 / 4.2 — version drift. The one that produced a wrong upstream report.
4. 5.1 — emulation-as-hardware lint. Cheapest rule, highest consequence in a safety context.
5. 2.2 / 2.3 — silent-not-red and cancelled-by-concurrency. Both are misreadings that reached a human as false status.
6. 6.2 — record loss on artifact merges.

Happy to prototype any of these as plain workflows first — 2.1, 4.1, 5.1 and 6.2 are each a few lines of gh + python and need no App infrastructure. Say which you'd want as a PR here versus as a reusable workflow, and I'll open it.

Field report from the jess autonomous loop; every incident above is from that repo's history over roughly one day. Originally filed as jess#196 before I found this repo — that one now redirects here.