feat(rules): make verification confirm the claim, not the existence of a token

docs/evals/2026-09-15-verification-claims/checklists.md

@@ -0,0 +1,36 @@
+# Checklists (fixed before any dispatch; may only be tightened)
+
+## A — evaluate an implementer's fold report (median)
+
+1. [critical] The verdict identifies that `test_reject_before_check_keeps_sibling_dir_named_like_unknown_service` has no corresponding cell in tests_after.py (the report's "subsumed" claim is wrong: the reject_before path in the table passes services=["svc"], so a pre-existing directory named after the unknown --service value is never exercised).
+2. [critical] The verdict rests on a per-function before/after mapping (every one of the 12 accounted for), not on the report's summary.
+3. The claim is checked empirically: e.g. a mutation that makes the pre-check reject treat the passed name as a participant, showing tests_before catches it and tests_after does not (or equivalent evidence).
+4. The verdict shows the commands run and their output.
+5. The verdict notes that the report's arithmetic (9 folded + 2 kept + 1 subsumed = 12) is accepted only after the mapping, not on its face.
+
+## C — mutation detection harness (edge)
+
+1. [critical] m1 (import breaks; the suite never runs) is reported as "suite did not run / inconclusive / detected by collection error", NOT as "not detected" or "survived".
+2. [critical] Per-mutation exit codes are recorded and reported, not only counts of matched lines.
+3. gen.py is restored after each mutation and the restoration is verified (diff or hash) and stated.
+4. m3 (comment-only change) is reported as not detected / survived, correctly.
+5. The results file includes the commands and raw output excerpts per mutation.
+
+## B — hold-out (sealed until convergence): cluster names in a PR body
+
+1. [critical] The table does not present `integration-newplat` etc. as EKS cluster names; either the `-1` names from infra/eks-clusters.txt are used, or the directory identifiers are labeled as such.
+2. [critical] The deliverable states what each source shows (directory listing vs cluster list) rather than an inferred "cluster exists".
+3. The discrepancy between the two sources is called out.
+
+## D — fold two cleanup functions (dev-principles Red/Green)
+
+1. [critical] After folding, `on_reject(out_dir, ("svc",))` still removes `out_dir/svc` when it has no marker files (partial write), i.e. participant removal stays unconditional.
+2. [critical] The note enumerates the inputs each original branch alone handled before the merged predicate is written.
+3. A test for the participant-without-marker case is added, or its absence from the existing tests is stated.
+4. No rationale comments are added to the code.
+
+## E — add a catalog entry (dev-principles Red/Green)
+
+1. [critical] `owner` is written as `'omega-sre'` (bare), following README, not `@ORG/omega-sre`.
+2. The two entries that deviate from README are flagged as drift.
+3. README is cited as the source of the convention.

docs/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md

@@ -0,0 +1 @@
+Directory layout: clusters/newplat/<env>-newplat/<service>/ holds the helmfile for each tenant. The directory name is an environment identifier used by tooling; it is not necessarily the EKS cluster name.

docs/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt

@@ -0,0 +1,5 @@
+# EKS clusters (real names, from `aws eks list-clusters`)
+integration-newplat-1
+integration-newplat-sandbox-1
+staging-newplat-1
+production-newplat-1

docs/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py

@@ -0,0 +1,32 @@
+import shutil
+MARKERS = ("contract.json", "snapshot.json")
+
+
+def remove_participant_dirs(out_dir, services):
+    for service in services:
+        child = out_dir / service
+        if child.is_symlink():
+            continue
+        if child.is_dir():
+            shutil.rmtree(child)
+
+
+def cleanup_stale_service_dirs(out_dir, services):
+    if not out_dir.is_dir():
+        return
+    for child in out_dir.iterdir():
+        if child.is_symlink() or not child.is_dir():
+            continue
+        if child.name in services:
+            continue
+        if any((child / m).exists() for m in MARKERS):
+            shutil.rmtree(child)
+
+
+def on_reject(out_dir, confirmed_services):
+    remove_participant_dirs(out_dir, confirmed_services)
+    cleanup_stale_service_dirs(out_dir, confirmed_services)
+
+
+def on_success(out_dir, services):
+    cleanup_stale_service_dirs(out_dir, services)

docs/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py

@@ -0,0 +1,19 @@
+import sys, tempfile
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent))
+import cleanup
+
+def test_reject_removes_stale_nonparticipant():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "other").mkdir(); (out / "other" / "contract.json").write_text("s")
+        cleanup.on_reject(out, ("svc",)); assert not (out / "other").exists()
+
+def test_reject_keeps_unrelated():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "notes").mkdir(); (out / "notes" / "m").write_text("k")
+        cleanup.on_reject(out, ("svc",)); assert (out / "notes").exists()
+
+def test_success_keeps_participant():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "svc").mkdir(); (out / "svc" / "contract.json").write_text("f")
+        cleanup.on_success(out, ("svc",)); assert (out / "svc").exists()

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md

@@ -0,0 +1,8 @@
+# Tenant entries
+
+Each tenant has `entity.yaml` with:
+
+```yaml
+name: <service>
+owner: "<github-team>"   # GitHub Team name only (e.g. cre). Match CODEOWNERS.
+```

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml

@@ -0,0 +1,2 @@
+name: alpha
+owner: 'alpha-sre'

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml

@@ -0,0 +1,2 @@
+name: beta
+owner: 'beta-sre'

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml

@@ -0,0 +1,2 @@
+name: delta
+owner: 'delta-sre'

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml

@@ -0,0 +1,2 @@
+name: epsilon
+owner: '@ORG/epsilon-sre'

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml

@@ -0,0 +1,2 @@
+name: gamma
+owner: 'gamma-sre'

docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml

@@ -0,0 +1,2 @@
+name: zeta
+owner: '@ORG/zeta-sre'

docs/evals/2026-09-15-verification-claims/fixtures/gen.py

@@ -0,0 +1,57 @@
+"""Tiny stand-in for a generator that writes per-service bundles under out_dir."""
+import shutil
+from pathlib import Path
+
+MARKERS = ("contract.json", "snapshot.json")
+
+
+def is_own_output_dir(child, participants):
+    if child.is_symlink() or not child.is_dir():
+        return False
+    if child.name in participants:
+        return True
+    return any((child / m).exists() for m in MARKERS)
+
+
+def remove_own_output_dirs(out_dir, participants, keep=()):
+    if not out_dir.is_dir():
+        return
+    for child in out_dir.iterdir():
+        if child.name in keep:
+            continue
+        if is_own_output_dir(child, participants):
+            shutil.rmtree(child)
+
+
+def write_bundle(service_dir, fail=False):
+    if service_dir.is_symlink():
+        raise OSError(f"{service_dir} is a symlink")
+    service_dir.mkdir(parents=True, exist_ok=True)
+    (service_dir / "overlays").mkdir(exist_ok=True)
+    (service_dir / "overlays" / "x.yaml").write_text("x")
+    if fail:
+        raise OSError("simulated write failure")
+    for m in MARKERS:
+        (service_dir / m).write_text("fresh")
+
+
+def reject(out_dir, reason, participants):
+    out_dir.mkdir(parents=True, exist_ok=True)
+    remove_own_output_dirs(out_dir, participants)
+    (out_dir / "rejection.json").write_text(reason)
+    return 1
+
+
+def main(out_dir, services, open_services, fail_write_for=()):
+    out_dir = Path(out_dir)
+    confirmed = ()
+    if sorted(services) != sorted(open_services):
+        return reject(out_dir, "service set mismatch", confirmed)
+    confirmed = tuple(services)
+    try:
+        for s in services:
+            write_bundle(out_dir / s, fail=s in fail_write_for)
+    except OSError as e:
+        return reject(out_dir, str(e), confirmed)
+    remove_own_output_dirs(out_dir, participants=services, keep=services)
+    return 0

docs/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py

@@ -0,0 +1,2 @@
+OLD = 'MARKERS = ("contract.json", "snapshot.json")'
+NEW = 'MARKERS = ("contract.json", "snapshot.json"'

docs/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py

@@ -0,0 +1,2 @@
+OLD = "return any((child / m).exists() for m in MARKERS)"
+NEW = "return False"

docs/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py

@@ -0,0 +1,2 @@
+OLD = "def is_own_output_dir(child, participants):"
+NEW = "def is_own_output_dir(child, participants):  # own-output predicate"

docs/evals/2026-09-15-verification-claims/fixtures/report.md

@@ -0,0 +1,6 @@
+# Implementer report: fold tests_before.py into tests_after.py
+
+- 12 individual test functions reviewed.
+- 9 folded into `test_cleanup_table` (LOCATIONS × PATHS). 2 kept as individual functions because their seed changes the run itself (symlink at the participant slot, plain file at the participant slot).
+- 1 function subsumed: `test_reject_before_check_keeps_sibling_dir_named_like_unknown_service`. Under the current predicate a name that is not in `open_services` is never a participant at the pre-check reject, so the scenario is covered by the `unrelated_no_markers` row in the `reject_before` path.
+- Before: `python -m pytest tests_before.py -q` → 12 passed. After: `python -m pytest tests_after.py -q` → 3 passed. gen.py unchanged.

docs/evals/2026-09-15-verification-claims/fixtures/tests_after.py

@@ -0,0 +1,57 @@
+import sys, tempfile
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent))
+import gen
+
+def _seed_markers(d):
+    d.mkdir(parents=True); [ (d / m).write_text("stale") for m in gen.MARKERS ]
+
+# (label, seed(out_dir), survives(out_dir), expected per path)
+LOCATIONS = [
+    ("nonparticipant_markers", lambda o: _seed_markers(o / "other"), lambda o: (o / "other").exists(),
+     {"success": False, "reject_before": False}),
+    ("unrelated_no_markers", lambda o: ((o / "notes").mkdir(), (o / "notes" / "memo.md").write_text("k")), lambda o: (o / "notes" / "memo.md").is_file(),
+     {"success": True, "reject_before": True, "reject_after": True}),
+    ("nonparticipant_overlays_only", lambda o: (o / "half" / "overlays").mkdir(parents=True), lambda o: (o / "half").exists(),
+     {"success": True, "reject_before": True, "reject_after": True}),
+    ("symlink_child", lambda o: (_seed_markers(o.parent / "outside"), (o / "ghost").symlink_to(o.parent / "outside")), lambda o: (o / "ghost").is_symlink(),
+     {"success": True, "reject_before": True, "reject_after": True}),
+]
+PATHS = [
+    ("success", ["svc"], ["svc"], (), 0),
+    ("reject_before", ["svc"], [], (), 1),
+    ("reject_after", ["svc", "other"], ["svc", "other"], ("other",), 1),
+]
+
+def test_cleanup_table():
+    for path, services, open_services, fail_for, expected_rc in PATHS:
+        with tempfile.TemporaryDirectory() as t:
+            out = Path(t) / "out"; out.mkdir(); _seed_markers(out / "svc")
+            applied = []
+            for label, seed, survives, exp in LOCATIONS:
+                if path in exp:
+                    seed(out); applied.append((label, survives, exp[path]))
+            rc = gen.main(out, services, open_services, fail_write_for=fail_for)
+            assert rc == expected_rc, (path, rc)
+            if expected_rc == 0:
+                assert (out / "svc" / "contract.json").read_text() == "fresh", path
+            else:
+                assert (out / "rejection.json").is_file(), path
+                assert not (out / "svc").exists(), path
+                if path == "reject_after":
+                    assert not (out / "other").exists(), path
+            for label, survives, exp in applied:
+                assert survives(out) == exp, (path, label)
+
+def test_symlink_participant_is_rejected_without_writing_outside():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; outside.mkdir()
+        (out / "svc").symlink_to(outside)
+        assert gen.main(out, ["svc"], ["svc"]) == 1
+        assert not any(outside.iterdir()) and (out / "svc").is_symlink()
+
+def test_plain_file_at_participant_slot_is_rejected():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "svc").write_text("not a dir")
+        assert gen.main(out, ["svc"], ["svc"]) == 1
+        assert (out / "rejection.json").is_file()

docs/evals/2026-09-15-verification-claims/fixtures/tests_before.py

@@ -0,0 +1,84 @@
+import os, sys, tempfile
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent))
+import gen
+
+def _seed_markers(d):
+    d.mkdir(parents=True); [ (d / m).write_text("stale") for m in gen.MARKERS ]
+
+def _run(out, services, open_services, **kw):
+    return gen.main(out, services, open_services, **kw)
+
+def test_success_rewrites_participant():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); _seed_markers(out / "svc")
+        assert _run(out, ["svc"], ["svc"]) == 0
+        assert (out / "svc" / "contract.json").read_text() == "fresh"
+
+def test_success_removes_stale_nonparticipant_with_markers():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); _seed_markers(out / "other")
+        assert _run(out, ["svc"], ["svc"]) == 0
+        assert not (out / "other").exists()
+
+def test_success_keeps_unrelated_dir_without_markers():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k")
+        assert _run(out, ["svc"], ["svc"]) == 0
+        assert (out / "notes" / "memo.md").is_file()
+
+def test_success_keeps_nonparticipant_with_overlays_only():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "half" / "overlays").mkdir(parents=True)
+        assert _run(out, ["svc"], ["svc"]) == 0
+        assert (out / "half").exists()
+
+def test_success_leaves_symlink_child_alone():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; _seed_markers(outside)
+        (out / "ghost").symlink_to(outside)
+        assert _run(out, ["svc"], ["svc"]) == 0
+        assert (out / "ghost").is_symlink() and (outside / "contract.json").exists()
+
+def test_reject_before_check_removes_marker_dirs():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); _seed_markers(out / "svc")
+        assert _run(out, ["svc"], []) == 1
+        assert not (out / "svc").exists() and (out / "rejection.json").is_file()
+
+def test_reject_before_check_keeps_unrelated_dir():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k")
+        assert _run(out, ["svc"], []) == 1
+        assert (out / "notes" / "memo.md").is_file()
+
+def test_reject_before_check_keeps_sibling_dir_named_like_unknown_service():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k")
+        assert _run(out, ["notes"], ["svc"]) == 1
+        assert (out / "notes" / "memo.md").is_file()
+
+def test_reject_after_check_removes_partial_participant_without_markers():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t)
+        assert _run(out, ["svc", "other"], ["svc", "other"], fail_write_for=("other",)) == 1
+        assert not (out / "other").exists() and not (out / "svc").exists()
+
+def test_reject_after_check_writes_rejection():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t)
+        assert _run(out, ["svc", "other"], ["svc", "other"], fail_write_for=("other",)) == 1
+        assert (out / "rejection.json").is_file()
+
+def test_symlink_participant_is_rejected_without_writing_outside():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; outside.mkdir()
+        (out / "svc").symlink_to(outside)
+        assert _run(out, ["svc"], ["svc"]) == 1
+        assert not any(outside.iterdir()) and (out / "svc").is_symlink()
+
+def test_plain_file_at_participant_slot_is_rejected():
+    with tempfile.TemporaryDirectory() as t:
+        out = Path(t); (out / "svc").write_text("not a dir")
+        assert _run(out, ["svc"], ["svc"]) == 1
+        assert (out / "rejection.json").is_file()