EvalAudit is a small, deterministic toolkit for asking a claim-relative question about benchmark results:
If the fixed systems and observations stay the same, which conclusions survive the declared evaluator meanings?
An audit.yaml binds the benchmark and substrate, states a typed ranking claim, records at least two evidence-backed semantic specifications, and supplies the score vector under each one. The evalaudit CLI separates provenance (origin) from review status (admission), computes a primary identified set from admitted specifications, and reports an admitted-plus-disputed sensitivity envelope. It emits winner, complete-order, pairwise, and separating-witness results; exact-score sets are available only for explicitly commensurate scalar endpoints.
This is an alpha research tool. It audits a declared finite family; it does not certify that the family is complete, that a benchmark is valid, or that a result is scientifically true.
git clone https://github.com/918154429/evalspec-audit.git
cd evalspec-audit
python -m venv .venvActivate the environment (.venv\Scripts\activate on Windows or source .venv/bin/activate on POSIX), then install and run a packaged demo:
python -m pip install -e .
evalaudit demo list
evalaudit demo run beir-scifact --output-dir outputs/beir-scifact-r1
evalaudit verify outputs/beir-scifact-r1 --jsonEvery run is no-clobber: an existing output directory is refused. A successful bundle contains the source declaration, normalized JSON, machine and Markdown reports, and a checksum-closed SHA256SUMS.txt.
The public demos use real, version-bound benchmark evidence and frozen reconstructed score tables. They test the deterministic audit layer; the CLI does not download upstream data or rerun proprietary models.
| Demo | Admitted / disputed | Exact-score set | Winner admitted / envelope | Complete order admitted / envelope | Stable pairs admitted / envelope |
|---|---|---|---|---|---|
| AgentDojo v0.1.35 | 2 / 3 | N/A (ordinal) | yes / yes | yes / no | 3/3 / 2/3 |
| AutoML Benchmark | 3 / 2 | N/A (ordinal) | yes / yes | yes / no | 10/10 / 9/10 |
| BEIR SciFact | 5 / 0 | N/A (ordinal) | yes / yes | no / no | 2/3 / 2/3 |
Disputed specifications do not control the primary headline. Exact-score identification is not claimed for these heterogeneous ordinal families. Stable winners are informative outcomes; the demos were not selected to manufacture winner flips. Full provenance, licenses, reconstruction recipes, and numerical boundaries are in demo provenance. For SciFact, a separate official evaluator-path receipt shows 15/15 five-decimal agreements over the three frozen runs; MRR is explicitly identified as BEIR's custom evaluation path.
Start from a packaged declaration:
evalaudit demo show agentdojo > audit.yaml
evalaudit validate audit.yaml --json
evalaudit run audit.yaml --output-dir outputs/my-audit-r1 --jsonThe contract is evalaudit/v0.3. It is intentionally breaking relative to v0.2: endpoints are explicitly ordinal or commensurate-scalar, and each specification declares separate origin and admission fields plus a family rationale. The authoritative examples are under src/evalaudit/demos, and the machine-readable schema is audit.schema.json. See the protocol for field semantics and migration notes.
EvalAudit includes an append-only, hash-chained registry for contesting specification membership, evidence bindings, claim queries, score reconstructions, or fixed substrates:
evalaudit registry init challenges.jsonl
evalaudit registry add examples/challenge.yaml --registry challenges.jsonl
evalaudit registry verify challenges.jsonl --json
evalaudit registry list challenges.jsonl --jsonThe canonical repository registry starts empty at registry/challenges.jsonl. Challenges are technical, evidence-bearing requests—not misconduct allegations. See challenge registry for lifecycle and trust semantics.
- Claim first: winner and complete-order claims are evaluated separately.
- Typed endpoint: ordinal families report only ordinal claims; commensurate scalar families additionally bind estimand, units, and scale before exact-score sets are allowed.
- Fixed substrate: evaluator meaning varies; systems and underlying observations do not silently drift.
- Evidence-bound family: every member separates evidence origin from admission status and supplies a semantic target, variation axis, family rationale, evidence pointer, confidence, and review state.
- Layered claims: only admitted specifications determine the primary result; disputed candidates appear in a separate sensitivity envelope; excluded candidates determine neither.
- Deterministic identified sets: no model chooses thresholds, scores, family members, or completeness.
- Semantic uncertainty stays separate from sampling or stochastic uncertainty.
- Fail closed: malformed bindings, missing evidence, method-registry drift, non-finite scores, output overwrite, and checksum tampering are rejected.
- Protocol and
audit.yaml - Demo provenance
- Challenge registry
- Validation boundaries
- Author-check materials
- Validity and taxonomy challenge cards
- Contributing
- Security policy
EvalAudit is MIT-licensed. Upstream benchmarks and datasets retain their own licenses; this repository redistributes declarations and links, not the AgentDojo run trees, AutoML source table, or SciFact corpus.