feat(enablement): turn baseline accuracy failure into a self-healing enablement trigger

[ZhengGong-amd]

Hyperloom only entered the enablement path when the model failed to boot. But on ROCm there is a more common — and more dangerous — class of failure:

The model boots fine and throughput looks perfectly healthy, but the output is garbage.

That failure was silent. Take Kimi-Linear-48B-A3B-Instruct: vLLM 0.20.0's KDA path corrupts recurrent state under batched concurrency, so baseline throughput sits at a completely healthy ~6600 tok/s while gsm8k scores 0.0038. The old logic had two possible outcomes, and neither was right:

1. Accuracy missing → stop_reason=baseline_accuracy_failed, the whole run halts, no repair is attempted.
2. Accuracy low but non-zero → accepted as a legitimate baseline and anchored. Every later optimization then chases throughput on a model that emits garbage, and cumulative_gain is fiction.

This PR makes a third outcome the default: treat accuracy failure as an enablement trigger, dispatch a specialist to localize and fix it, then re-verify with a genuine baseline and only accept the fix once that passes.

What changed

1. Accuracy failure becomes an enablement trigger

• The baseline executor recognizes and classifies three eval failure kinds: eval_runtime_failure / accuracy_unavailable / accuracy_below_floor.
• The trigger context (observed accuracy, task/metric, floor, probe config, evidence text) is persisted into SharedState and threaded down the enablement chain to the specialist, so it knows which eval contract it has to reproduce.
• Switch: INFERENCE_OPTIMIZER_ENABLEMENT_ON_EVAL_FAIL (on by default).

2. An eval-failed baseline may not anchor

This is the safety floor of the PR. A baseline that failed eval is stopped by _is_promotable_result and never writes baseline_tput / baseline_accuracy / baseline_config_path, so nothing can "chase gains against garbage output".

The bypass routes are closed too: disable_run_eval and an explicit RUN_EVAL=false can no longer launder "accuracy was never measured" into "accuracy is acceptable".

3. Accuracy floor raised from 0.0 to 0.05

ENABLEMENT_ACCURACY_FLOOR was 0.0 while the KEEP gate is accuracy > floor — which reduces to "anything non-zero counts as correct". A real run once KEPT a candidate at gsm8k 0.00076 (0.08% of a 0.906 baseline) as correct.

The new 0.05 default is only a backstop: it rejects the collapsed regime where the model answers essentially nothing, and makes no quality judgement (that belongs to the operator's INFERENCE_OPTIMIZER_ENABLEMENT_ACCURACY_FLOOR override). The same floor is shared by the trigger side and the KEEP side, so the two cannot drift apart.

4. An eval-origin KEEP requires genuine-baseline revalidation

Even when an eval-origin candidate produces a passing accuracy inside the gate, that is still only the candidate's own single measurement. A new enablement_validation_pending window holds the KEEP: it re-runs a real baseline with the accepted config, and enablement_succeeded is set only once that promotes. A failed revalidation counts toward the stall cap, so it cannot loop forever.

5. "Did the eval actually run?" is now judged from the candidate's own task/metric

An earlier version used an eval-contract fingerprint as a drift veto — but RUN_EVAL is itself one of the hashed contract fields, so a single re-baseline that skips the eval poisons the stored digest and then vetoes every subsequent candidate without ever looking at its accuracy. That is precisely how a correct fix was thrown away in a real run.

The check now reads the task + metric that the candidate's own run recorded (stamped right next to its accuracy value): a score with no task/metric did not come from a real eval, so it fails closed. The fingerprint chain is retained but demoted to diagnostic only — it gates nothing.

6. An already-applied patch counts as satisfied, not as an apply failure

Specialists routinely write both a superset patch and the subset it contains (e.g. "FLA layout revert" and "FLA layout revert + config fix"). Whichever lands first turns all of the other's hunks into no-ops — and POSIX patch and git apply --check return exactly the same code for "already applied" as for "does not apply".

The old logic called that a hard failure, reverted the entire combo, and discarded a fix that had been completely and correctly applied. On Kimi-Linear this destroyed a 7-hour run: the specialist had already localized two defects and taken gsm8k from 0.0008 to 0.875, only for the fix to be thrown away at apply time. That round drove the stall streak to its cap and the run ended at enablement_stalled with baseline_tput=0.

The fix disambiguates with a reverse dry-run (both channels changed): a reverse apply only passes cleanly when every hunk's target state is already present — which is exactly the state a forward apply would produce. Fully satisfied → success, and no backups are recorded (backups belong to the patch that actually made the edits, so a revert does not double-restore). Partially overlapping and genuinely non-applying patches still fail, with reauthor feedback, so this no-op cannot swallow a real change.

Also closed stdin on the patch invocations — it prompts Assume -R? [n] on an already-applied hunk and could otherwise block until the 60s timeout.

7. KB records the real accuracy delta

Nothing ever wrote the gate_evidence["accuracy"] key (_bench_patch graded the eval into a bool and dropped the score), so accuracy_delta_pct was permanently 0.0 and the fingerprint dedup gate had nothing to match on. The raw score is now parsed out alongside the grade; the baseline-side fallback is read unconditionally too.

Supporting changes: reverted and accuracy-losing candidates no longer count as "gained"; config-lever deliverables are deduped by content fingerprint.

8. Independent fixes carried along

• Sibling accuracy salvage: the cold-start guard splits a baseline into warmup (the only round that measures accuracy) and measure (hot throughput only), but the decision happens on the measure round — which by construction has no accuracy. This used to be misread as "accuracy missing", sending a specialist to chase a quality regression that never existed. It now salvages from a sibling attempt in the same session first.
• install.sh single-gateway credential derivation: .env.template ships SAFE_API_KEY + OPENAI_BASE_URL as the default (the ANTHROPIC_* block is commented out), but install.sh's preflight_validate_credentials only recognized ANTHROPIC_*, so an operator following the documented setup could not pass IR-2. It now derives them the same way the CLI's _resolve_llm_endpoints() does; explicit values still win.
• Skip commit-on-KEEP for non-git framework roots: when the framework is pip-installed into site-packages there is no .git, and the old logic ran git add unconditionally and failed with rc=128, logging a bogus commit-on-KEEP failed warning on every enablement KEEP. Durability for such roots is already provided by snapshot_source_layer.
• Microbench timeout capture: a timeout raises subprocess.TimeoutExpired, not RuntimeError; the old code caught only the latter, so the whole trace_analyze died instead of taking its documented fallback.
• Model gate: recognize the HF tokenizer's model_max_length.
• CLI: --framework-discover-timeout-sec read the wrong argparse dest.

9. Reverted: checkpoint-scaled server-ready timeout

dd2691728 scaled the server boot budget by checkpoint bytes (to stop a 1.4 TiB MXFP4 MoE being killed under the 2700s default). That failure scenario no longer exists, so the whole thing is reverted (5238a8693).

The logic only ever engaged above 200 GiB; checkpoints below the threshold always took the original 2700s default, so for the vast majority of models it was a no-op by construction. Verified on the live Kimi-Linear-48B run (92 GiB): resolve_server_ready_timeout() returns 2700 and the materialized lifecycle YAML also carries 2700 — the removed code never fired.

Scope

• Logic: 32 files, +1444 / −147 (net of the 5238a8693 revert)
• Tests: 12 files, +1216 / −12
• New env vars: INFERENCE_OPTIMIZER_ENABLEMENT_ON_EVAL_FAIL, INFERENCE_OPTIMIZER_ENABLEMENT_ACCURACY_FLOOR (both documented in docs/reference/environment-variables.md)
• No new CLI flags; the launcher contract is unchanged
• SharedState gains 14 enablement_* fields; the enablement section of session_breakdown.json is extended to match (schema updated)
• Merged up to date with origin/main (including the forge/install FORGE_PATH consolidation). install.sh had one overlapping region; the auto-merge result was reviewed by hand.

Behavioral changes (please focus review here)

1. Default behavior changes: a baseline below the accuracy floor used to either anchor or halt the run; it now enters the enablement self-healing path. Set INFERENCE_OPTIMIZER_ENABLEMENT_ON_EVAL_FAIL=0 for the old behavior.
2. Accuracy floor moves from 0 to 0.05: in principle this could reject a legitimate model/task combination whose gsm8k is genuinely very low. Such cases should set INFERENCE_OPTIMIZER_ENABLEMENT_ACCURACY_FLOOR explicitly.
3. correctness_ok semantics fork by origin: when accuracy is None, eval-origin fails closed (the trigger was an accuracy failure, so producing no score does not count as fixed) while boot-origin stays provisional (it only ever promised to make the model boot, and must not block eval-less runs).

Verification

• The enablement / framework / integrate_patch test subsets pass on this branch (493 passed / 1 skipped after the merge; 729 passed / 1 skipped across the baseline + lifecycle + magpie subset after the revert). The full suite is left to CI.
• Three known failures are pre-existing and environment-dependent, identical before and after this branch's changes and unrelated to this PR: test_artifact_install_failed_restores_user_stash, test_isolated_venv_discovers_plan, test_bench_specialist_without_explicit_needs_gpu_is_gated. Additionally test_build_backends_kernel_codex also fails standalone on the merged-in ccda68204.
• End-to-end: a live Kimi-Linear-48B-A3B-Instruct run (vllm / MI355X / TP2) exercised the full chain — baseline measured gsm8k 0.0038 → classified accuracy_below_floor → anchoring refused → enablement specialist dispatched, with the trigger evidence carrying a complete task/metric.

Follow-ups / open items

• c20347403 and dd2691728 still carry [Temp] / [temp] prefixes. Their contents have been reviewed hunk by hunk: the parts that went stale are reverted by 5238a8693, and the rest (single-gateway credential derivation, non-git commit skip, microbench timeout capture) is still live logic. Only the titles need changing — recommend renaming before merge.
• A partially overlapping patch currently fails closed and is handed back to the specialist to reauthor. patch -N could apply only the missing hunks, but that requires parsing reject files and carries more risk; deferred to a separate evaluation.

[github-actions]

CI E2E report — ❌ Failed

item	value
result	❌ Failed
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	feat/zgong/enable-accuracy
commit	6214d7d4638b452f3397e7f7ac01649fa97c3b98
session_id	8b74b9ea-de34-4a54-a0a2-ef00dfd0a8ec
queue → dispatch	0s
run time	154m 41s
total	154m 41s
reason	platform reported Failed: container hyperloom exited with code 1 (Error): pruned_families : []
detail	`platform reported Failed: container hyperloom exited with code 1 (Error): pruned_families : []

details