Status: exploratory calibration; not a quality result
Suite: Multi-SWE-bench Rust v0.1, 30 tasks
Date: 2026-08-03
SACP-0.2 needs a preregistered evidence-sufficiency threshold before a paid model run. We replayed deterministic adaptive traces at six query-term coverage thresholds while holding the following values fixed:
- minimum evidence excerpts: 2;
- token ladder: 1,000 through 8,000 in 1,000-token steps;
- node ladder: 2 through 16 in two-node steps;
- BM25 candidate paths followed by verified semantic expansion; and
- no model, embedding, or judge calls.
For each threshold, the selected packet was the first passing round or the final hard-limit round. Gold fix paths were used only after retrieval to score the recorded rounds; they were never supplied to the compiler.
| Coverage threshold | Estimated stops | Mean prompt tokens | Mean token savings | Evidence recall | Complete-evidence tasks |
|---|---|---|---|---|---|
| 30% | 25/30 | 2,442 | 98.64% | 47.31% | 9/30 |
| 40% | 17/30 | 3,445 | 98.12% | 49.37% | 9/30 |
| 50% | 12/30 | 4,344 | 97.81% | 51.85% | 9/30 |
| 60% | 5/30 | 4,957 | 97.65% | 51.85% | 9/30 |
| 70% | 0/30 | 5,138 | 97.59% | 51.85% | 9/30 |
| 75% | 0/30 | 5,138 | 97.59% | 51.85% | 9/30 |
The fixed compiled arm also achieved 51.85% evidence recall. The 50% threshold
was therefore the lowest tested threshold that retained that retrieval metric
while selecting less context. It became the provisional default for the
prospective model run reported in
eceb-adaptive-study.md.
- The original 75% threshold never passed and produced 30/30 hard-limit stops.
- Nine tasks contained all gold paths at an earlier round, averaging 1,533 prompt tokens, but the 75% estimator missed every one.
- At 50%, ten estimated-sufficient tasks still lacked complete gold evidence. The estimator therefore cannot be interpreted as evidence completeness.
- Twenty-one tasks never contained every gold path at any recorded round. No stopping policy can recover evidence that the underlying candidate generator never retrieved.
This calibration reuses the same 30 tasks that exposed the original retrieval gap. It is parameter selection, not independent validation. It does not measure answer correctness, patch success, or model-quality noninferiority.
The 50% threshold was frozen before the follow-up model observations were collected. A general quality claim still requires a held-out task suite without retuning this threshold, followed by replication across models and corpora.