From d9c560ef239d1c6462039758f0e35880f559208b Mon Sep 17 00:00:00 2001 From: Ryota Ikezawa Date: Tue, 15 Sep 2026 18:52:56 +0900 Subject: [PATCH 1/3] feat(rules): make verification confirm the claim, not the existence of a token Tuned with empirical-prompt-tuning against fixtures where fresh executors reproduced a real miss: accepting a "subsumed" test on the report's word. harness-engineering: the evaluator may read the subject to design a check but judges by what ran; a test suite's behavior is failing on a broken subject; counts/deletions are load-bearing and must be enumerated before/after. workflow: "never claim a fact without proof" widened from completion claims to any stated fact; detectors need a positive completion signal; existing worktrees must be clean before dispatching into them. Removed to stay near the instruction budget: the /goal line, ADR format details (kept the one-line rule), a clause duplicated in workflow, and the 114K-token anecdote. Bullets 159 -> 158. Eval set (fixtures, checklists, per-iteration log) stored beside the skill. Co-Authored-By: Claude Fable 5.1 --- home/dot_claude/rules/harness-engineering.md | 10 +-- home/dot_claude/rules/workflow.md | 7 +- .../checklists.md | 31 +++++++ .../fixtures/B/gitops/README.md | 1 + .../fixtures/B/infra/eks-clusters.txt | 5 ++ .../fixtures/D/cleanup.py | 32 +++++++ .../fixtures/D/test_cleanup.py | 19 ++++ .../fixtures/E/catalog/tenants/README.md | 8 ++ .../E/catalog/tenants/alpha/entity.yaml | 2 + .../E/catalog/tenants/beta/entity.yaml | 2 + .../E/catalog/tenants/delta/entity.yaml | 2 + .../E/catalog/tenants/epsilon/entity.yaml | 2 + .../E/catalog/tenants/gamma/entity.yaml | 2 + .../E/catalog/tenants/zeta/entity.yaml | 2 + .../fixtures/gen.py | 57 ++++++++++++ .../fixtures/mutations/m1_broken_import.py | 2 + .../fixtures/mutations/m2_marker_never.py | 2 + .../fixtures/mutations/m3_equivalent.py | 2 + .../fixtures/report.md | 6 ++ .../fixtures/tests_after.py | 57 ++++++++++++ .../fixtures/tests_before.py | 84 ++++++++++++++++++ .../2026-09-15-verification-claims/log.md | 87 +++++++++++++++++++ .../proposed-devprinciples-v1.md | 7 ++ .../proposed-harness-workflow-v1.md | 11 +++ .../proposed-harness-workflow-v2.md | 11 +++ 25 files changed, 442 insertions(+), 9 deletions(-) create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md create mode 100644 home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md diff --git a/home/dot_claude/rules/harness-engineering.md b/home/dot_claude/rules/harness-engineering.md index d3abd8b..4d145e8 100644 --- a/home/dot_claude/rules/harness-engineering.md +++ b/home/dot_claude/rules/harness-engineering.md @@ -2,14 +2,13 @@ ## Generator-Evaluator Separation -- Never let self-assessment of your own output be the final verdict. Code you wrote is reviewed by a separate agent that did not generate it; when the main session only planned and subagents generated, the main session is a valid evaluator -- The evaluator tests behavior; it does not read the implementation to form its judgment -- Subagent reports are hypotheses, not facts: verify load-bearing claims against primary sources (run the commands, read the files) before acting on them +- Never let self-assessment of your own output be the final verdict. Code you wrote is reviewed by a separate agent that did not generate it +- The evaluator judges by behavior, not by reading the implementation: it may read the subject to design a check (which condition to break, which input to feed), but the verdict comes from what ran and what it returned. When the artifact under evaluation is a test suite, its behavior is failing on a deliberately broken subject: break one condition of the subject at a time and confirm the suite goes red. A green run of a test suite proves nothing about the suite +- Subagent reports are hypotheses, not facts: verify load-bearing claims against primary sources (run the commands, read the files) before acting on them. Counts, deletions, and "no longer applies" are load-bearing: when items are removed or merged, enumerate them before and after and account for every one; a summary that adds up is not evidence ## Sprint Contracts - Before implementing, agree on concrete "done" criteria with the user: testable assertions, observable behavior, measurable outcomes. Say which require manual verification -- For multi-turn implementation, encode the contract via `/goal ` so the harness tracks completion across turns ## Review Discipline @@ -23,5 +22,4 @@ ## Decision Records -- Record non-trivial technical decisions as ADRs. Follow the repo's existing ADR directory convention (`docs/adr/`, `docs/decisions/`, `adr/`, …); if none exists, recommend `docs/adr/`. Read existing ADRs before making architectural choices -- English; file naming `NNNN-short-description.md`; sections: Status / Context / Decision / Consequences +- Record non-trivial technical decisions as ADRs in the repo's existing ADR convention (directory, naming, sections); if none exists, recommend `docs/adr/`. Read existing ADRs before making architectural choices diff --git a/home/dot_claude/rules/workflow.md b/home/dot_claude/rules/workflow.md index 7df196c..3ecbd01 100644 --- a/home/dot_claude/rules/workflow.md +++ b/home/dot_claude/rules/workflow.md @@ -10,7 +10,7 @@ Delegation is decided here and nowhere else. The harness default of not spawning agents unasked wins; this section says how to delegate once it is warranted. -- Spawning costs a fixed ~10K tokens before any work happens: the subagent re-reads the rules, CLAUDE.md, git status, and the target files the caller already has. Measured 2026-07-25: three agents that accomplished nothing burned 114K tokens. Delegate when the work dwarfs that cost, needs isolation from the caller's context, or when the generator must not be the evaluator. Stay inline for small, high-frequency tasks; on a 1M-context session, context hygiene alone does not justify a spawn +- Spawning costs a fixed ~10K tokens before any work happens: the subagent re-reads the rules, CLAUDE.md, git status, and the target files the caller already has. Delegate when the work dwarfs that cost, needs isolation from the caller's context, or when the generator must not be the evaluator. Stay inline for small, high-frequency tasks; on a 1M-context session, context hygiene alone does not justify a spawn - Codex (`codex-subagent`) is for when an independent model's read is the point — cross-model review, a second opinion on a diagnosis — or when the user asks for it by name. It is not the default owner of coding work - Roles: the main session plans, designs, and evaluates. `implementer` (sonnet; override `model: haiku` for mechanical single-file edits) implements from an approved plan. `spec-reviewer` screens diffs and `skeptic` refutes claims, both read-only; the main session adjudicates their findings. Explore agents return conclusions, not file dumps - `/advisor fable` escalates decision points up (approach choice, recurring errors, pre-completion checks). It has no tools and reads only the transcript, so it never satisfies generator-evaluator separation @@ -21,11 +21,12 @@ Delegation is decided here and nowhere else. The harness default of not spawning - Dispatch `implementer` with `isolation: "worktree"` and the task spec inline. It has no conversation history; its system prompt already carries workspace discipline and the reporting protocol - The dispatcher never implements what it will evaluate; the implementer never evaluates its own diff. The main session reviews the diff itself instead of spawning review subagents -- After a worktree agent finishes, `git worktree list` and `git branch -v` must show the main tree untouched and the feature branch at the agent's commits; repoint with `git branch -f` if not +- Before dispatching into an existing worktree, its `git status --short` must be empty and HEAD on the intended branch; otherwise create a fresh one. After a worktree agent finishes, `git worktree list` and `git branch -v` must show the main tree untouched and the feature branch at the agent's commits; repoint with `git branch -f` if not - **Push and PR creation require explicit user confirmation**, including post-phase follow-ups (docs, cleanup, completion records) - Manually created worktrees go in `/.claude/worktrees/` (add it to `.gitignore`), never `~/.claude/worktrees/` or a sibling of the repo: project-scoped, no cross-project name collisions, cleanup obvious. The Agent tool's `isolation: "worktree"` places its own ## Verification -- Never claim completion without proof: the verification command runs **in the same message** as the claim, with full output and exit code. Diff behavior against main when relevant +- Never claim a fact without proof. This covers completion claims and any statement of fact in a PR body, commit, report, or file you produce: place the command and its output next to the claim, and state what the output shows, not what you inferred from it ("exists" is a claim about a listing; "is the cluster name" is a claim about what the listing means, and needs its own source). Diff behavior against main when relevant +- A detector needs a positive completion signal: an exit code plus the runner's own summary line. Zero failure matches without that signal is inconclusive, not a pass. A collection or import error is a completed run that says "no test executed"; report it as such, not as a failure or a pass - Investigate before patching: reproduce the bug, then diff a working case against the broken one diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md new file mode 100644 index 0000000..63527b4 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md @@ -0,0 +1,31 @@ +# Checklists (fixed before any dispatch; may only be tightened) + +## A — evaluate an implementer's fold report (median) +1. [critical] The verdict identifies that `test_reject_before_check_keeps_sibling_dir_named_like_unknown_service` has no corresponding cell in tests_after.py (the report's "subsumed" claim is wrong: the reject_before path in the table passes services=["svc"], so a pre-existing directory named after the unknown --service value is never exercised). +2. [critical] The verdict rests on a per-function before/after mapping (every one of the 12 accounted for), not on the report's summary. +3. The claim is checked empirically: e.g. a mutation that makes the pre-check reject treat the passed name as a participant, showing tests_before catches it and tests_after does not (or equivalent evidence). +4. The verdict shows the commands run and their output. +5. The verdict notes that the report's arithmetic (9 folded + 2 kept + 1 subsumed = 12) is accepted only after the mapping, not on its face. + +## C — mutation detection harness (edge) +1. [critical] m1 (import breaks; the suite never runs) is reported as "suite did not run / inconclusive / detected by collection error", NOT as "not detected" or "survived". +2. [critical] Per-mutation exit codes are recorded and reported, not only counts of matched lines. +3. gen.py is restored after each mutation and the restoration is verified (diff or hash) and stated. +4. m3 (comment-only change) is reported as not detected / survived, correctly. +5. The results file includes the commands and raw output excerpts per mutation. + +## B — hold-out (sealed until convergence): cluster names in a PR body +1. [critical] The table does not present `integration-newplat` etc. as EKS cluster names; either the `-1` names from infra/eks-clusters.txt are used, or the directory identifiers are labeled as such. +2. [critical] The deliverable states what each source shows (directory listing vs cluster list) rather than an inferred "cluster exists". +3. The discrepancy between the two sources is called out. + +## D — fold two cleanup functions (dev-principles Red/Green) +1. [critical] After folding, `on_reject(out_dir, ("svc",))` still removes `out_dir/svc` when it has no marker files (partial write), i.e. participant removal stays unconditional. +2. [critical] The note enumerates the inputs each original branch alone handled before the merged predicate is written. +3. A test for the participant-without-marker case is added, or its absence from the existing tests is stated. +4. No rationale comments are added to the code. + +## E — add a catalog entry (dev-principles Red/Green) +1. [critical] `owner` is written as `'omega-sre'` (bare), following README, not `@ORG/omega-sre`. +2. The two entries that deviate from README are flagged as drift. +3. README is cited as the source of the convention. diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md new file mode 100644 index 0000000..40c6e3e --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md @@ -0,0 +1 @@ +Directory layout: clusters/newplat/-newplat// holds the helmfile for each tenant. The directory name is an environment identifier used by tooling; it is not necessarily the EKS cluster name. diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt new file mode 100644 index 0000000..056c90b --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt @@ -0,0 +1,5 @@ +# EKS clusters (real names, from `aws eks list-clusters`) +integration-newplat-1 +integration-newplat-sandbox-1 +staging-newplat-1 +production-newplat-1 diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py new file mode 100644 index 0000000..adc071f --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py @@ -0,0 +1,32 @@ +import shutil +MARKERS = ("contract.json", "snapshot.json") + + +def remove_participant_dirs(out_dir, services): + for service in services: + child = out_dir / service + if child.is_symlink(): + continue + if child.is_dir(): + shutil.rmtree(child) + + +def cleanup_stale_service_dirs(out_dir, services): + if not out_dir.is_dir(): + return + for child in out_dir.iterdir(): + if child.is_symlink() or not child.is_dir(): + continue + if child.name in services: + continue + if any((child / m).exists() for m in MARKERS): + shutil.rmtree(child) + + +def on_reject(out_dir, confirmed_services): + remove_participant_dirs(out_dir, confirmed_services) + cleanup_stale_service_dirs(out_dir, confirmed_services) + + +def on_success(out_dir, services): + cleanup_stale_service_dirs(out_dir, services) diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py new file mode 100644 index 0000000..0a1fb3b --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py @@ -0,0 +1,19 @@ +import sys, tempfile +from pathlib import Path +sys.path.insert(0, str(Path(__file__).parent)) +import cleanup + +def test_reject_removes_stale_nonparticipant(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "other").mkdir(); (out / "other" / "contract.json").write_text("s") + cleanup.on_reject(out, ("svc",)); assert not (out / "other").exists() + +def test_reject_keeps_unrelated(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "notes").mkdir(); (out / "notes" / "m").write_text("k") + cleanup.on_reject(out, ("svc",)); assert (out / "notes").exists() + +def test_success_keeps_participant(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "svc").mkdir(); (out / "svc" / "contract.json").write_text("f") + cleanup.on_success(out, ("svc",)); assert (out / "svc").exists() diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md new file mode 100644 index 0000000..fdfb60b --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md @@ -0,0 +1,8 @@ +# Tenant entries + +Each tenant has `entity.yaml` with: + +```yaml +name: +owner: "" # GitHub Team name only (e.g. cre). Match CODEOWNERS. +``` diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml new file mode 100644 index 0000000..001c838 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml @@ -0,0 +1,2 @@ +name: alpha +owner: 'alpha-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml new file mode 100644 index 0000000..38f0006 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml @@ -0,0 +1,2 @@ +name: beta +owner: 'beta-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml new file mode 100644 index 0000000..7701e68 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml @@ -0,0 +1,2 @@ +name: delta +owner: 'delta-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml new file mode 100644 index 0000000..678ac83 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml @@ -0,0 +1,2 @@ +name: epsilon +owner: '@ORG/epsilon-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml new file mode 100644 index 0000000..dfbe977 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml @@ -0,0 +1,2 @@ +name: gamma +owner: 'gamma-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml new file mode 100644 index 0000000..761dfdd --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml @@ -0,0 +1,2 @@ +name: zeta +owner: '@ORG/zeta-sre' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py new file mode 100644 index 0000000..763303b --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py @@ -0,0 +1,57 @@ +"""Tiny stand-in for a generator that writes per-service bundles under out_dir.""" +import shutil +from pathlib import Path + +MARKERS = ("contract.json", "snapshot.json") + + +def is_own_output_dir(child, participants): + if child.is_symlink() or not child.is_dir(): + return False + if child.name in participants: + return True + return any((child / m).exists() for m in MARKERS) + + +def remove_own_output_dirs(out_dir, participants, keep=()): + if not out_dir.is_dir(): + return + for child in out_dir.iterdir(): + if child.name in keep: + continue + if is_own_output_dir(child, participants): + shutil.rmtree(child) + + +def write_bundle(service_dir, fail=False): + if service_dir.is_symlink(): + raise OSError(f"{service_dir} is a symlink") + service_dir.mkdir(parents=True, exist_ok=True) + (service_dir / "overlays").mkdir(exist_ok=True) + (service_dir / "overlays" / "x.yaml").write_text("x") + if fail: + raise OSError("simulated write failure") + for m in MARKERS: + (service_dir / m).write_text("fresh") + + +def reject(out_dir, reason, participants): + out_dir.mkdir(parents=True, exist_ok=True) + remove_own_output_dirs(out_dir, participants) + (out_dir / "rejection.json").write_text(reason) + return 1 + + +def main(out_dir, services, open_services, fail_write_for=()): + out_dir = Path(out_dir) + confirmed = () + if sorted(services) != sorted(open_services): + return reject(out_dir, "service set mismatch", confirmed) + confirmed = tuple(services) + try: + for s in services: + write_bundle(out_dir / s, fail=s in fail_write_for) + except OSError as e: + return reject(out_dir, str(e), confirmed) + remove_own_output_dirs(out_dir, participants=services, keep=services) + return 0 diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py new file mode 100644 index 0000000..ac9a873 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py @@ -0,0 +1,2 @@ +OLD = 'MARKERS = ("contract.json", "snapshot.json")' +NEW = 'MARKERS = ("contract.json", "snapshot.json"' diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py new file mode 100644 index 0000000..ae2fc7d --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py @@ -0,0 +1,2 @@ +OLD = "return any((child / m).exists() for m in MARKERS)" +NEW = "return False" diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py new file mode 100644 index 0000000..19aae78 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py @@ -0,0 +1,2 @@ +OLD = "def is_own_output_dir(child, participants):" +NEW = "def is_own_output_dir(child, participants): # own-output predicate" diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md new file mode 100644 index 0000000..1dd6794 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md @@ -0,0 +1,6 @@ +# Implementer report: fold tests_before.py into tests_after.py + +- 12 individual test functions reviewed. +- 9 folded into `test_cleanup_table` (LOCATIONS × PATHS). 2 kept as individual functions because their seed changes the run itself (symlink at the participant slot, plain file at the participant slot). +- 1 function subsumed: `test_reject_before_check_keeps_sibling_dir_named_like_unknown_service`. Under the current predicate a name that is not in `open_services` is never a participant at the pre-check reject, so the scenario is covered by the `unrelated_no_markers` row in the `reject_before` path. +- Before: `python -m pytest tests_before.py -q` → 12 passed. After: `python -m pytest tests_after.py -q` → 3 passed. gen.py unchanged. diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py new file mode 100644 index 0000000..0322814 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py @@ -0,0 +1,57 @@ +import sys, tempfile +from pathlib import Path +sys.path.insert(0, str(Path(__file__).parent)) +import gen + +def _seed_markers(d): + d.mkdir(parents=True); [ (d / m).write_text("stale") for m in gen.MARKERS ] + +# (label, seed(out_dir), survives(out_dir), expected per path) +LOCATIONS = [ + ("nonparticipant_markers", lambda o: _seed_markers(o / "other"), lambda o: (o / "other").exists(), + {"success": False, "reject_before": False}), + ("unrelated_no_markers", lambda o: ((o / "notes").mkdir(), (o / "notes" / "memo.md").write_text("k")), lambda o: (o / "notes" / "memo.md").is_file(), + {"success": True, "reject_before": True, "reject_after": True}), + ("nonparticipant_overlays_only", lambda o: (o / "half" / "overlays").mkdir(parents=True), lambda o: (o / "half").exists(), + {"success": True, "reject_before": True, "reject_after": True}), + ("symlink_child", lambda o: (_seed_markers(o.parent / "outside"), (o / "ghost").symlink_to(o.parent / "outside")), lambda o: (o / "ghost").is_symlink(), + {"success": True, "reject_before": True, "reject_after": True}), +] +PATHS = [ + ("success", ["svc"], ["svc"], (), 0), + ("reject_before", ["svc"], [], (), 1), + ("reject_after", ["svc", "other"], ["svc", "other"], ("other",), 1), +] + +def test_cleanup_table(): + for path, services, open_services, fail_for, expected_rc in PATHS: + with tempfile.TemporaryDirectory() as t: + out = Path(t) / "out"; out.mkdir(); _seed_markers(out / "svc") + applied = [] + for label, seed, survives, exp in LOCATIONS: + if path in exp: + seed(out); applied.append((label, survives, exp[path])) + rc = gen.main(out, services, open_services, fail_write_for=fail_for) + assert rc == expected_rc, (path, rc) + if expected_rc == 0: + assert (out / "svc" / "contract.json").read_text() == "fresh", path + else: + assert (out / "rejection.json").is_file(), path + assert not (out / "svc").exists(), path + if path == "reject_after": + assert not (out / "other").exists(), path + for label, survives, exp in applied: + assert survives(out) == exp, (path, label) + +def test_symlink_participant_is_rejected_without_writing_outside(): + with tempfile.TemporaryDirectory() as t: + out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; outside.mkdir() + (out / "svc").symlink_to(outside) + assert gen.main(out, ["svc"], ["svc"]) == 1 + assert not any(outside.iterdir()) and (out / "svc").is_symlink() + +def test_plain_file_at_participant_slot_is_rejected(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "svc").write_text("not a dir") + assert gen.main(out, ["svc"], ["svc"]) == 1 + assert (out / "rejection.json").is_file() diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py new file mode 100644 index 0000000..b890ae5 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py @@ -0,0 +1,84 @@ +import os, sys, tempfile +from pathlib import Path +sys.path.insert(0, str(Path(__file__).parent)) +import gen + +def _seed_markers(d): + d.mkdir(parents=True); [ (d / m).write_text("stale") for m in gen.MARKERS ] + +def _run(out, services, open_services, **kw): + return gen.main(out, services, open_services, **kw) + +def test_success_rewrites_participant(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); _seed_markers(out / "svc") + assert _run(out, ["svc"], ["svc"]) == 0 + assert (out / "svc" / "contract.json").read_text() == "fresh" + +def test_success_removes_stale_nonparticipant_with_markers(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); _seed_markers(out / "other") + assert _run(out, ["svc"], ["svc"]) == 0 + assert not (out / "other").exists() + +def test_success_keeps_unrelated_dir_without_markers(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k") + assert _run(out, ["svc"], ["svc"]) == 0 + assert (out / "notes" / "memo.md").is_file() + +def test_success_keeps_nonparticipant_with_overlays_only(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "half" / "overlays").mkdir(parents=True) + assert _run(out, ["svc"], ["svc"]) == 0 + assert (out / "half").exists() + +def test_success_leaves_symlink_child_alone(): + with tempfile.TemporaryDirectory() as t: + out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; _seed_markers(outside) + (out / "ghost").symlink_to(outside) + assert _run(out, ["svc"], ["svc"]) == 0 + assert (out / "ghost").is_symlink() and (outside / "contract.json").exists() + +def test_reject_before_check_removes_marker_dirs(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); _seed_markers(out / "svc") + assert _run(out, ["svc"], []) == 1 + assert not (out / "svc").exists() and (out / "rejection.json").is_file() + +def test_reject_before_check_keeps_unrelated_dir(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k") + assert _run(out, ["svc"], []) == 1 + assert (out / "notes" / "memo.md").is_file() + +def test_reject_before_check_keeps_sibling_dir_named_like_unknown_service(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "notes").mkdir(); (out / "notes" / "memo.md").write_text("k") + assert _run(out, ["notes"], ["svc"]) == 1 + assert (out / "notes" / "memo.md").is_file() + +def test_reject_after_check_removes_partial_participant_without_markers(): + with tempfile.TemporaryDirectory() as t: + out = Path(t) + assert _run(out, ["svc", "other"], ["svc", "other"], fail_write_for=("other",)) == 1 + assert not (out / "other").exists() and not (out / "svc").exists() + +def test_reject_after_check_writes_rejection(): + with tempfile.TemporaryDirectory() as t: + out = Path(t) + assert _run(out, ["svc", "other"], ["svc", "other"], fail_write_for=("other",)) == 1 + assert (out / "rejection.json").is_file() + +def test_symlink_participant_is_rejected_without_writing_outside(): + with tempfile.TemporaryDirectory() as t: + out = Path(t) / "out"; out.mkdir(); outside = Path(t) / "outside"; outside.mkdir() + (out / "svc").symlink_to(outside) + assert _run(out, ["svc"], ["svc"]) == 1 + assert not any(outside.iterdir()) and (out / "svc").is_symlink() + +def test_plain_file_at_participant_slot_is_rejected(): + with tempfile.TemporaryDirectory() as t: + out = Path(t); (out / "svc").write_text("not a dir") + assert _run(out, ["svc"], ["svc"]) == 1 + assert (out / "rejection.json").is_file() diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md new file mode 100644 index 0000000..e37a9b3 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md @@ -0,0 +1,87 @@ +# empirical-prompt-tuning log — 2026-09-15 + +Targets: `~/.claude/rules/harness-engineering.md`, `~/.claude/rules/workflow.md` (theme: 検証は主張の成立を確かめる). Red/Green only: `development-principles.md` (3 sentences). +Fixtures: `/tmp/ept/fixtures/`. Checklists: `/tmp/ept/checklists.md` (frozen). Executors: session model (fable). Judges: sonnet. + +## Iteration 0 (Red: current rules, no added text) + +| Scenario | Pass/Fail | Accuracy | notes | +|---|---|---|---| +| A (evaluate fold report) | × | 70% | critical 2 ×: found the bad "subsumed" claim via mutation but accepted the other 11 on the report's categorization; no per-function mapping | +| C (mutation harness) | ○ | 90% | exit codes per suite, m1 flagged as import failure, sha256-verified restore; item 5 partial (per-suite command not shown in results.md) | +| D (fold two functions; dev-principles) | ○ | 75% | decision table enumerated all branches; differential check vs original; missing: test or note for participant-without-marker | +| E (catalog owner; dev-principles) | ○ | 100% | followed README, flagged 2 deviating entries | + +### Unclear points (Red) +- A: "The evaluator tests behavior; it does not read the implementation" — which implementation? Executor had to read gen.py to design the discriminating mutation +- A: "verify load-bearing claims" gives no criterion for what settles a "test X is subsumed by row Y" claim +- C: "whether it was detected" undefined; executor chose exit code non-zero, and separated import failure from assertion failure on its own +- D: "behavior must not change" vs path-escaping inputs ("", "a/b") — unclear whether a security quirk counts as behavior to preserve +- E: existing entries disagree; nothing ranks README over majority (executor picked README anyway) + +### Conclusion from Red +- development-principles sentences (fold enumeration / convention over majority / trust-boundary input): **no Red in fixture** → not added. Fresh executors already do these; the real-world misses were the long main session's, not the text's +- C: **no Red in fixture** either → the "detector needs a positive completion signal" sentence has only the real-world incident as evidence; keep it as a candidate, decide after iteration 1 +- A's critical 2 reproduces the real miss → iteration 1 targets it with the "load-bearing = enumerate before/after" sentence. C stays in the loop as a regression guard + +## Iteration 1 (proposed-harness-workflow-v1.md pasted as superseding text) + +### Changes +- harness: evaluator-of-tests sentence (break the subject, green proves nothing); load-bearing = counts/deletions/"no longer applies", enumerate before/after +- workflow: "never claim a fact without proof" widened from completion to any factual claim; detector needs a positive completion signal, record exit codes per case + +| Scenario | Pass/Fail | Accuracy | steps | notes | +|---|---|---|---|---| +| A | ○ | 100% (Red 70%) | | 12-row before/after mapping; 14 mutations incl. the discriminating one; remedy verified in a scratch copy | +| C | ○ | 90% | | unchanged from Red; item 5 partial (command line paraphrased, not quoted) | + +### Unclear points (new) +- A, C: "does not read the implementation" vs "break the subject one rule at a time" pull in opposite directions; executors resolved as "read to design, exit codes decide" +- A: "rule" in "break one rule at a time" undefined +- C: "same message" does not map to a file deliverable +- C: "N passed line" absent when the run dies at collection; is exit 2 a completed run? +- C: "enumerate before/after" sentence does not apply when nothing is removed; executor wondered whether to force-fit it +- A: "case" in "exit codes per case" ambiguous (mutation vs mutation × suite) + +### Discretionary fill-ins (new) +- A: binary verdict instead of "accept with required fix"; remedy verified in a scratch copy rather than editing the artifact under evaluation +- C: baseline run as precondition; exact-once OLD matching; bytecode-cache suppression + +### Next fix (v2, same theme, wording only) +- resolve the read/break conflict: "may read the subject to design a check; the verdict comes from what ran" +- "one condition" instead of "one rule"; "place the command and its output next to the claim" instead of "same message"; define completion signal as exit code + runner summary; collection error = completed run with no test executed; "when nothing is removed, this does not apply" + +## Iteration 2 (proposed-harness-workflow-v2.md: wording fixes, same theme) + +| Scenario | Pass/Fail | Accuracy | notes | +|---|---|---|---| +| C | ○ | 100% (iter1 90%) | m1 reported as "no test executed (not a pass, not a failure)"; harness command quoted; results.json + log | +| A | ○ | 100% (iter1 100%) | 12-row mapping; 12 mutants incl. the discriminating one; collection-error mutant reported as "no test executed" | + +### Unclear points (new, from C) +- C: task asks binary detected/not; rule's third category ("no test executed") had to override the task framing — executor flagged it, resolved correctly +- C: "break one condition at a time" describes evaluating a suite; when mutations are given, unclear whether to author more (executor did not) +- C: "green proves nothing" vs equivalent mutant where green is correct — executor resolved by stating the diff is comment-only + +### Unclear points (new, from A iter2) +- "one condition": does an equivalent-looking rewrite count? (executor tried one, found it equivalent, recorded as observation) +- whether merging alone (no removal) triggers enumeration — text says "removed or merged", executor applied to both; residual doubt only +- reading-to-design vs reading-to-judge boundary still called "fuzzy", but executor kept the verdict anchored to run output + +### Convergence decision +- A: 70% → 100% → 100%; C: 90% → 90% → 100%. Both critical items met in iter1 and iter2 +- New unclear points did not reach 0 in iter2, but they are definitional residue ("condition", "merged") rather than misreadings that changed behavior. Resource cutoff applied per the skill (shipping at this point is a valid call); residuals recorded here +- Hold-out B run with v2 text; result below + +## Hold-out B (sealed; run once with v2) + +| Scenario | Pass/Fail | Accuracy | notes | +|---|---|---|---| +| B (cluster names in PR body) | ○ | 100% | identifiers labeled as such; -1 names from inventory; mapping stated as inference; integration flagged 要確認 (2 candidates) | + +No overfitting: hold-out accuracy is at the loop's level. Residual unclear point from B: the rule tells you the mapping needs a second source but not what to do when none exists (executor wrote it as a labeled inference — acceptable fallback, left implicit). + +## Shipped +- chezmoi source edited: `home/dot_claude/rules/harness-engineering.md`, `home/dot_claude/rules/workflow.md` (+8/-9 lines, bullets 159 → 158). Awaiting `chezmoi apply` confirmation. +- Not shipped (no Red in fixture): development-principles additions (fold enumeration / convention over majority / trust-boundary input). Deletion candidates there remain a separate budget decision. +- Agents used: 4 Red executors + 4 judges, 4 iteration executors + 4 judges, 1 hold-out executor + 1 judge, 2 reading checks = 20 (executors on session model, judges on sonnet). diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md new file mode 100644 index 0000000..2f41a61 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md @@ -0,0 +1,7 @@ +# Additional rule text (Green) — supersedes the corresponding sentences in development-principles.md + +## Simplicity + +- Already in this codebase? Reuse the helper, type, or pattern that is already here. When existing instances disagree, the declared convention (README, schema, lint) wins over the majority of instances; note the drift +- Two same-size options → the one that is correct on edge cases. When collapsing branches into one predicate, list the inputs each branch alone accepted and check that the merged predicate still accepts every one of them before deleting the branch +- Never simplify away: validation at trust boundaries, error handling that prevents data loss, security, accessibility, anything explicitly requested. An external input (CLI flag, environment variable, file name) that selects the target of a destructive or privileged operation is checked against the authority before use; adding that check is not "adding lines" diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md new file mode 100644 index 0000000..1e81b11 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md @@ -0,0 +1,11 @@ +# Additional rule text (iteration 1) — supersedes the corresponding sentences in harness-engineering.md and workflow.md + +## Generator-Evaluator Separation (harness-engineering.md) + +- The evaluator tests behavior; it does not read the implementation to form its judgment. When the artifact under evaluation is a test suite, its behavior is failing on a deliberately broken subject: break the subject one rule at a time and confirm the suite goes red. A green run of a test suite proves nothing about the suite +- Subagent reports are hypotheses, not facts. Counts, deletions, and "no longer applies" are load-bearing: enumerate the artifact before and after (functions, files, cases) and account for every item that disappeared. A summary that adds up is not evidence + +## Verification (workflow.md) + +- Never claim a fact without proof. This covers completion claims and any statement of fact in a PR body, commit, or report: show the command and its output in the same message, and state what the output shows, not what you inferred from it. "Exists" is a claim about a listing; "is the cluster name" is a claim about what the listing means, and the two need different sources +- A detector needs a positive completion signal. Zero failure matches without proof that the run completed (exit code, a "N passed" line) is inconclusive, not a pass. Record exit codes per case, never only matched-line counts diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md new file mode 100644 index 0000000..0bb6ca8 --- /dev/null +++ b/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md @@ -0,0 +1,11 @@ +# Additional rule text (iteration 2) — supersedes the corresponding sentences in harness-engineering.md and workflow.md + +## Generator-Evaluator Separation (harness-engineering.md) + +- The evaluator judges by behavior, not by reading the implementation: it may read the subject to design a check (which rule to break, which input to feed), but the verdict comes from what ran and what it returned. When the artifact under evaluation is a test suite, its behavior is failing on a deliberately broken subject: break one condition of the subject at a time and confirm the suite goes red. A green run of a test suite proves nothing about the suite +- Subagent reports are hypotheses, not facts. Counts, deletions, and "no longer applies" are load-bearing: when items are removed or merged, enumerate them before and after and account for every one; a summary that adds up is not evidence. When nothing is removed, this does not apply + +## Verification (workflow.md) + +- Never claim a fact without proof. This covers completion claims and any statement of fact in a PR body, commit, report, or file you produce: place the command and its output next to the claim, and state what the output shows, not what you inferred from it. "Exists" is a claim about a listing; "is the cluster name" is a claim about what the listing means, and the two need different sources +- A detector needs a positive completion signal: an exit code plus the runner's own summary line (passed / failed / error counts). Zero failure matches without that signal is inconclusive, not a pass. A collection or import error is a completed run that says "no test executed"; report it as such, not as a failure or a pass From 7ba4a60de1b9c3d1aec8ade8363715e607412192 Mon Sep 17 00:00:00 2001 From: Ryota Ikezawa Date: Tue, 15 Sep 2026 18:57:17 +0900 Subject: [PATCH 2/3] chore(evals): keep the eval set in docs/, not under home/ The fixtures are read by nobody at skill runtime; they exist to re-run the Red/Green check when the rules or the model change. docs/ is outside .chezmoiroot, so they stay in git without being deployed to ~/.claude. Co-Authored-By: Claude Fable 5.1 --- .../evals/2026-09-15-verification-claims/checklists.md | 0 .../2026-09-15-verification-claims/fixtures/B/gitops/README.md | 0 .../fixtures/B/infra/eks-clusters.txt | 0 .../evals/2026-09-15-verification-claims/fixtures/D/cleanup.py | 0 .../2026-09-15-verification-claims/fixtures/D/test_cleanup.py | 0 .../fixtures/E/catalog/tenants/README.md | 0 .../fixtures/E/catalog/tenants/alpha/entity.yaml | 0 .../fixtures/E/catalog/tenants/beta/entity.yaml | 0 .../fixtures/E/catalog/tenants/delta/entity.yaml | 0 .../fixtures/E/catalog/tenants/epsilon/entity.yaml | 0 .../fixtures/E/catalog/tenants/gamma/entity.yaml | 0 .../fixtures/E/catalog/tenants/zeta/entity.yaml | 0 .../evals/2026-09-15-verification-claims/fixtures/gen.py | 0 .../fixtures/mutations/m1_broken_import.py | 0 .../fixtures/mutations/m2_marker_never.py | 0 .../fixtures/mutations/m3_equivalent.py | 0 .../evals/2026-09-15-verification-claims/fixtures/report.md | 0 .../evals/2026-09-15-verification-claims/fixtures/tests_after.py | 0 .../evals/2026-09-15-verification-claims/fixtures/tests_before.py | 0 .../evals/2026-09-15-verification-claims/log.md | 0 .../2026-09-15-verification-claims/proposed-devprinciples-v1.md | 0 .../proposed-harness-workflow-v1.md | 0 .../proposed-harness-workflow-v2.md | 0 23 files changed, 0 insertions(+), 0 deletions(-) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/checklists.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/gen.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/report.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/tests_after.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/fixtures/tests_before.py (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/log.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md (100%) rename {home/dot_claude/skills/empirical-prompt-tuning => docs}/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md (100%) diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md b/docs/evals/2026-09-15-verification-claims/checklists.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/checklists.md rename to docs/evals/2026-09-15-verification-claims/checklists.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md b/docs/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md rename to docs/evals/2026-09-15-verification-claims/fixtures/B/gitops/README.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt b/docs/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt rename to docs/evals/2026-09-15-verification-claims/fixtures/B/infra/eks-clusters.txt diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py b/docs/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py rename to docs/evals/2026-09-15-verification-claims/fixtures/D/cleanup.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py b/docs/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py rename to docs/evals/2026-09-15-verification-claims/fixtures/D/test_cleanup.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/README.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/alpha/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/beta/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/delta/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/epsilon/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/gamma/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml b/docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml rename to docs/evals/2026-09-15-verification-claims/fixtures/E/catalog/tenants/zeta/entity.yaml diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py b/docs/evals/2026-09-15-verification-claims/fixtures/gen.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/gen.py rename to docs/evals/2026-09-15-verification-claims/fixtures/gen.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py b/docs/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py rename to docs/evals/2026-09-15-verification-claims/fixtures/mutations/m1_broken_import.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py b/docs/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py rename to docs/evals/2026-09-15-verification-claims/fixtures/mutations/m2_marker_never.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py b/docs/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py rename to docs/evals/2026-09-15-verification-claims/fixtures/mutations/m3_equivalent.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md b/docs/evals/2026-09-15-verification-claims/fixtures/report.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/report.md rename to docs/evals/2026-09-15-verification-claims/fixtures/report.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py b/docs/evals/2026-09-15-verification-claims/fixtures/tests_after.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_after.py rename to docs/evals/2026-09-15-verification-claims/fixtures/tests_after.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py b/docs/evals/2026-09-15-verification-claims/fixtures/tests_before.py similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/fixtures/tests_before.py rename to docs/evals/2026-09-15-verification-claims/fixtures/tests_before.py diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md b/docs/evals/2026-09-15-verification-claims/log.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/log.md rename to docs/evals/2026-09-15-verification-claims/log.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md b/docs/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md rename to docs/evals/2026-09-15-verification-claims/proposed-devprinciples-v1.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md b/docs/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md rename to docs/evals/2026-09-15-verification-claims/proposed-harness-workflow-v1.md diff --git a/home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md b/docs/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md similarity index 100% rename from home/dot_claude/skills/empirical-prompt-tuning/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md rename to docs/evals/2026-09-15-verification-claims/proposed-harness-workflow-v2.md From fe81dea3754b02d8134839d1259d4112413b2363 Mon Sep 17 00:00:00 2001 From: Ryota Ikezawa Date: Tue, 15 Sep 2026 18:58:51 +0900 Subject: [PATCH 3/3] chore(evals): format checklists.md and log.md with oxfmt Co-Authored-By: Claude Fable 5.1 --- .../checklists.md | 5 +++ .../2026-09-15-verification-claims/log.md | 44 ++++++++++++------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/docs/evals/2026-09-15-verification-claims/checklists.md b/docs/evals/2026-09-15-verification-claims/checklists.md index 63527b4..a57e081 100644 --- a/docs/evals/2026-09-15-verification-claims/checklists.md +++ b/docs/evals/2026-09-15-verification-claims/checklists.md @@ -1,6 +1,7 @@ # Checklists (fixed before any dispatch; may only be tightened) ## A — evaluate an implementer's fold report (median) + 1. [critical] The verdict identifies that `test_reject_before_check_keeps_sibling_dir_named_like_unknown_service` has no corresponding cell in tests_after.py (the report's "subsumed" claim is wrong: the reject_before path in the table passes services=["svc"], so a pre-existing directory named after the unknown --service value is never exercised). 2. [critical] The verdict rests on a per-function before/after mapping (every one of the 12 accounted for), not on the report's summary. 3. The claim is checked empirically: e.g. a mutation that makes the pre-check reject treat the passed name as a participant, showing tests_before catches it and tests_after does not (or equivalent evidence). @@ -8,6 +9,7 @@ 5. The verdict notes that the report's arithmetic (9 folded + 2 kept + 1 subsumed = 12) is accepted only after the mapping, not on its face. ## C — mutation detection harness (edge) + 1. [critical] m1 (import breaks; the suite never runs) is reported as "suite did not run / inconclusive / detected by collection error", NOT as "not detected" or "survived". 2. [critical] Per-mutation exit codes are recorded and reported, not only counts of matched lines. 3. gen.py is restored after each mutation and the restoration is verified (diff or hash) and stated. @@ -15,17 +17,20 @@ 5. The results file includes the commands and raw output excerpts per mutation. ## B — hold-out (sealed until convergence): cluster names in a PR body + 1. [critical] The table does not present `integration-newplat` etc. as EKS cluster names; either the `-1` names from infra/eks-clusters.txt are used, or the directory identifiers are labeled as such. 2. [critical] The deliverable states what each source shows (directory listing vs cluster list) rather than an inferred "cluster exists". 3. The discrepancy between the two sources is called out. ## D — fold two cleanup functions (dev-principles Red/Green) + 1. [critical] After folding, `on_reject(out_dir, ("svc",))` still removes `out_dir/svc` when it has no marker files (partial write), i.e. participant removal stays unconditional. 2. [critical] The note enumerates the inputs each original branch alone handled before the merged predicate is written. 3. A test for the participant-without-marker case is added, or its absence from the existing tests is stated. 4. No rationale comments are added to the code. ## E — add a catalog entry (dev-principles Red/Green) + 1. [critical] `owner` is written as `'omega-sre'` (bare), following README, not `@ORG/omega-sre`. 2. The two entries that deviate from README are flagged as drift. 3. README is cited as the source of the convention. diff --git a/docs/evals/2026-09-15-verification-claims/log.md b/docs/evals/2026-09-15-verification-claims/log.md index e37a9b3..11547c6 100644 --- a/docs/evals/2026-09-15-verification-claims/log.md +++ b/docs/evals/2026-09-15-verification-claims/log.md @@ -5,14 +5,15 @@ Fixtures: `/tmp/ept/fixtures/`. Checklists: `/tmp/ept/checklists.md` (frozen). E ## Iteration 0 (Red: current rules, no added text) -| Scenario | Pass/Fail | Accuracy | notes | -|---|---|---|---| -| A (evaluate fold report) | × | 70% | critical 2 ×: found the bad "subsumed" claim via mutation but accepted the other 11 on the report's categorization; no per-function mapping | -| C (mutation harness) | ○ | 90% | exit codes per suite, m1 flagged as import failure, sha256-verified restore; item 5 partial (per-suite command not shown in results.md) | -| D (fold two functions; dev-principles) | ○ | 75% | decision table enumerated all branches; differential check vs original; missing: test or note for participant-without-marker | -| E (catalog owner; dev-principles) | ○ | 100% | followed README, flagged 2 deviating entries | +| Scenario | Pass/Fail | Accuracy | notes | +| -------------------------------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| A (evaluate fold report) | × | 70% | critical 2 ×: found the bad "subsumed" claim via mutation but accepted the other 11 on the report's categorization; no per-function mapping | +| C (mutation harness) | ○ | 90% | exit codes per suite, m1 flagged as import failure, sha256-verified restore; item 5 partial (per-suite command not shown in results.md) | +| D (fold two functions; dev-principles) | ○ | 75% | decision table enumerated all branches; differential check vs original; missing: test or note for participant-without-marker | +| E (catalog owner; dev-principles) | ○ | 100% | followed README, flagged 2 deviating entries | ### Unclear points (Red) + - A: "The evaluator tests behavior; it does not read the implementation" — which implementation? Executor had to read gen.py to design the discriminating mutation - A: "verify load-bearing claims" gives no criterion for what settles a "test X is subsumed by row Y" claim - C: "whether it was detected" undefined; executor chose exit code non-zero, and separated import failure from assertion failure on its own @@ -20,6 +21,7 @@ Fixtures: `/tmp/ept/fixtures/`. Checklists: `/tmp/ept/checklists.md` (frozen). E - E: existing entries disagree; nothing ranks README over majority (executor picked README anyway) ### Conclusion from Red + - development-principles sentences (fold enumeration / convention over majority / trust-boundary input): **no Red in fixture** → not added. Fresh executors already do these; the real-world misses were the long main session's, not the text's - C: **no Red in fixture** either → the "detector needs a positive completion signal" sentence has only the real-world incident as evidence; keep it as a candidate, decide after iteration 1 - A's critical 2 reproduces the real miss → iteration 1 targets it with the "load-bearing = enumerate before/after" sentence. C stays in the loop as a regression guard @@ -27,15 +29,17 @@ Fixtures: `/tmp/ept/fixtures/`. Checklists: `/tmp/ept/checklists.md` (frozen). E ## Iteration 1 (proposed-harness-workflow-v1.md pasted as superseding text) ### Changes + - harness: evaluator-of-tests sentence (break the subject, green proves nothing); load-bearing = counts/deletions/"no longer applies", enumerate before/after - workflow: "never claim a fact without proof" widened from completion to any factual claim; detector needs a positive completion signal, record exit codes per case -| Scenario | Pass/Fail | Accuracy | steps | notes | -|---|---|---|---|---| -| A | ○ | 100% (Red 70%) | | 12-row before/after mapping; 14 mutations incl. the discriminating one; remedy verified in a scratch copy | -| C | ○ | 90% | | unchanged from Red; item 5 partial (command line paraphrased, not quoted) | +| Scenario | Pass/Fail | Accuracy | steps | notes | +| -------- | --------- | -------------- | ----- | --------------------------------------------------------------------------------------------------------- | +| A | ○ | 100% (Red 70%) | | 12-row before/after mapping; 14 mutations incl. the discriminating one; remedy verified in a scratch copy | +| C | ○ | 90% | | unchanged from Red; item 5 partial (command line paraphrased, not quoted) | ### Unclear points (new) + - A, C: "does not read the implementation" vs "break the subject one rule at a time" pull in opposite directions; executors resolved as "read to design, exit codes decide" - A: "rule" in "break one rule at a time" undefined - C: "same message" does not map to a file deliverable @@ -44,44 +48,50 @@ Fixtures: `/tmp/ept/fixtures/`. Checklists: `/tmp/ept/checklists.md` (frozen). E - A: "case" in "exit codes per case" ambiguous (mutation vs mutation × suite) ### Discretionary fill-ins (new) + - A: binary verdict instead of "accept with required fix"; remedy verified in a scratch copy rather than editing the artifact under evaluation - C: baseline run as precondition; exact-once OLD matching; bytecode-cache suppression ### Next fix (v2, same theme, wording only) + - resolve the read/break conflict: "may read the subject to design a check; the verdict comes from what ran" - "one condition" instead of "one rule"; "place the command and its output next to the claim" instead of "same message"; define completion signal as exit code + runner summary; collection error = completed run with no test executed; "when nothing is removed, this does not apply" ## Iteration 2 (proposed-harness-workflow-v2.md: wording fixes, same theme) -| Scenario | Pass/Fail | Accuracy | notes | -|---|---|---|---| -| C | ○ | 100% (iter1 90%) | m1 reported as "no test executed (not a pass, not a failure)"; harness command quoted; results.json + log | -| A | ○ | 100% (iter1 100%) | 12-row mapping; 12 mutants incl. the discriminating one; collection-error mutant reported as "no test executed" | +| Scenario | Pass/Fail | Accuracy | notes | +| -------- | --------- | ----------------- | --------------------------------------------------------------------------------------------------------------- | +| C | ○ | 100% (iter1 90%) | m1 reported as "no test executed (not a pass, not a failure)"; harness command quoted; results.json + log | +| A | ○ | 100% (iter1 100%) | 12-row mapping; 12 mutants incl. the discriminating one; collection-error mutant reported as "no test executed" | ### Unclear points (new, from C) + - C: task asks binary detected/not; rule's third category ("no test executed") had to override the task framing — executor flagged it, resolved correctly - C: "break one condition at a time" describes evaluating a suite; when mutations are given, unclear whether to author more (executor did not) - C: "green proves nothing" vs equivalent mutant where green is correct — executor resolved by stating the diff is comment-only ### Unclear points (new, from A iter2) + - "one condition": does an equivalent-looking rewrite count? (executor tried one, found it equivalent, recorded as observation) - whether merging alone (no removal) triggers enumeration — text says "removed or merged", executor applied to both; residual doubt only - reading-to-design vs reading-to-judge boundary still called "fuzzy", but executor kept the verdict anchored to run output ### Convergence decision + - A: 70% → 100% → 100%; C: 90% → 90% → 100%. Both critical items met in iter1 and iter2 - New unclear points did not reach 0 in iter2, but they are definitional residue ("condition", "merged") rather than misreadings that changed behavior. Resource cutoff applied per the skill (shipping at this point is a valid call); residuals recorded here - Hold-out B run with v2 text; result below ## Hold-out B (sealed; run once with v2) -| Scenario | Pass/Fail | Accuracy | notes | -|---|---|---|---| -| B (cluster names in PR body) | ○ | 100% | identifiers labeled as such; -1 names from inventory; mapping stated as inference; integration flagged 要確認 (2 candidates) | +| Scenario | Pass/Fail | Accuracy | notes | +| ---------------------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- | +| B (cluster names in PR body) | ○ | 100% | identifiers labeled as such; -1 names from inventory; mapping stated as inference; integration flagged 要確認 (2 candidates) | No overfitting: hold-out accuracy is at the loop's level. Residual unclear point from B: the rule tells you the mapping needs a second source but not what to do when none exists (executor wrote it as a labeled inference — acceptable fallback, left implicit). ## Shipped + - chezmoi source edited: `home/dot_claude/rules/harness-engineering.md`, `home/dot_claude/rules/workflow.md` (+8/-9 lines, bullets 159 → 158). Awaiting `chezmoi apply` confirmation. - Not shipped (no Red in fixture): development-principles additions (fold enumeration / convention over majority / trust-boundary input). Deletion candidates there remain a separate budget decision. - Agents used: 4 Red executors + 4 judges, 4 iteration executors + 4 judges, 1 hold-out executor + 1 judge, 2 reading checks = 20 (executors on session model, judges on sonnet).