Mutation Testing of Task-Scoped State Oracles

This repository is the key-free replication artifact for:

Mutation Testing of Task-Scoped State Oracles in Software-Agent Benchmarks: A Cross-Benchmark Empirical Study

The study treats executable evaluators in stateful software-agent benchmarks as software test oracles. It injects deterministic harmful state mutations and schema-certified benign transformations into known-valid final states, then measures whether the official evaluators respond correctly.

The manuscript was submitted to Software Quality Journal on 11 August 2026. The repository contains the frozen protocol, archived evaluator outputs, analysis code, derived tables and figures, aggregate blinded-audit decisions, regression tests, and manuscript source needed to reproduce the reported confirmatory results without a model API.

Main confirmatory results

• 60 frozen task templates across Tau2-Bench, ToolSandbox, and AppWorld.
• 145 applicable mutations from 540 attempted task-by-mutation cells.
• Official evaluators rejected 100/110 harmful mutations (90.9%).
• Official evaluators preserved 35/35 applicable benign JSON-key-order variants.
• Ten reachable collateral state changes across six ToolSandbox templates retained full official credit.
• The confirmatory and replication analyses use zero model API calls.

These results do not estimate how often real agents cause harmful side effects, and the sparse AppWorld strata do not support a broad benchmark ranking. See the manuscript and analysis/confirmatory_analysis.md for the full scope, uncertainty intervals, negative results, and limitations.

One-command verification on Windows

The default verification path reads only archived benchmark-evaluator records. It does not contact a model provider, read an API key, or execute the stopped exploratory model arm.

git clone https://github.com/shushuyang231/side-effect-calibration-study.git
Set-Location '.\side-effect-calibration-study'
python -m venv .venv
& '.\.venv\Scripts\python.exe' -m pip install -r '.\requirements-replication.txt'
& '.\scripts\run_public_artifact_verification.ps1' -PythonPath '.\.venv\Scripts\python.exe'

A successful run verifies the six frozen preflight gates and scientific fingerprints, regenerates the statistical tables and five figures, re-analyzes the confirmatory arm, runs nine state-differencing and allowed-delta regression tests, resolves manuscript citations, and scans the deliverables for credential-like values and local identity paths.

Expected final line:

PASS: public artifact verification completed. API calls: 0. Paid cost: CNY 0.

Repository map

• artifacts/formal-full/: append-only confirmatory evaluator records and frozen validation snapshot.
• analysis/: analysis scripts, derived summaries, figures, tables, and the public-tool diagnostic.
• audit-aggregate/: aggregate blinded mutation-audit results; private audit keys and individual reviewer material are intentionally excluded.
• regression/: reference allowed-state-delta oracle and regression tests.
• manuscript/: LaTeX manuscript and bibliography source.
• protocol-amendment-*.md: dated protocol amendments and deviation records.
• research.md: frozen research questions, hypotheses, gates, and experiment history.

Reproduction boundary

The key-free command reproduces the paper's analyses from archived official evaluator outputs. Full clean re-execution of upstream benchmark evaluators also requires the pinned benchmark environments and packages identified in the protocol. Upstream benchmark material is included only where redistribution is permitted; otherwise the artifact provides source identifiers, scripts, and hashes.

Raw provider transcripts, API credentials, private audit keys, synthetic account records, and incomplete exploratory-service logs are not distributed. The stopped exploratory model arm is retained only as an aggregate summary and is excluded from all confirmatory claims.

Citation

Citation metadata is available in CITATION.cff. Until a persistent manuscript DOI is available, please cite the repository and accompanying manuscript by title and author:

Sun, Shengyao. Mutation Testing of Task-Scoped State Oracles in Software-Agent
Benchmarks: A Cross-Benchmark Empirical Study. 2026.

ORCID: 0009-0008-9175-8226