feat(enablement): turn baseline accuracy failure into a self-healing enablement trigger

.env.template

@@ -66,6 +66,19 @@ OPENAI_BASE_URL=https://<your-gateway-host>/api/v1/llm-proxy/v1
 # Populated automatically by src/hyperloom/inference_optimizer/assets/install.sh probe.
 # INFERENCE_OPTIMIZER_FRAMEWORK_SOURCE_ROOTS=
 
+# (Optional) When a first baseline boots but fails its accuracy eval, route it
+# into enablement instead of halting the run. Default on (single-node only).
+# INFERENCE_OPTIMIZER_ENABLEMENT_ON_EVAL_FAIL=1
+# (Optional) Shared accuracy floor for the eval-failure trigger AND the
+# enablement KEEP gate (finite, in [0,1]). Default 0.05.
+# The default is a collapse guard, not a quality bar: it rejects a model that
+# answers essentially nothing correctly (a real run once KEEPed a candidate at
+# gsm8k 0.00076 under the old 0.0 default, i.e. 0.08% of a 0.906 baseline).
+# Raise it if you want a genuine quality gate.
+# Note: score=0.0 always fails regardless of floor (strictly positive required).
+# score>=floor passes (score==floor is a pass). Single-node only.
+# INFERENCE_OPTIMIZER_ENABLEMENT_ACCURACY_FLOOR=0.05
+
 # Writable artifact root: hosts every session dir, optimizer_runs/, and the
 # runtime/ tree generated by install.sh (GEAK e2e checkout,
 # kernel-agent.env.sh, etc.). The default /workspace/hyperloom matches the in-code

docs/conceptual/optimization-loop.md

@@ -120,8 +120,9 @@ The LLM doesn't own a separate framework role in the current runtime.
 
 ### Enablement escalation ladder
 
-When a `(model, backend)` combination cannot launch, enablement repairs it
-along two axes. **Diagnosis (once):** work out which capability layer is
+When a `(model, backend)` combination cannot launch — or it launches but fails
+its accuracy eval (`accuracy_below_floor` / `eval_runtime_failure`) — enablement
+repairs it along two axes. **Diagnosis (once):** work out which capability layer is
 missing — read the failure signature, the model's `config.json` architecture,
 the framework's supported-architecture registry and installed version, and
 upstream (whether the capability already exists and in which version/PR). That
@@ -171,8 +172,11 @@ A verified build does not KEEP on artifact verification alone. After a
 build's artifacts verify, the Coordinator runs a launch probe: it boots the
 actual model with the built runtime through the same runnable-decision gate
 the authored-patch lane uses. Only a runtime that actually launches — and
-passes minimal correctness — earns KEEP. Otherwise the build reverts, or, if
-the boot advanced past the original failure to a new or deeper gap, the loop
+passes minimal correctness — earns KEEP. For an eval-origin trigger the gate
+additionally re-runs the accuracy eval against the captured contract and a
+KEEP requires the accuracy to meet the floor; the KEEP is finalized only after
+a genuine baseline revalidates it. Otherwise the build reverts, or, if the
+boot advanced past the original failure to a new or deeper gap, the loop
 advances to the next round to repair that gap.
 
 ### Discovery-driven build refs

docs/reference/environment-variables.md

@@ -200,6 +200,20 @@ do not publish it unchanged in support bundles.
 
 ---
 
+## Enablement accuracy trigger
+
+When a first baseline boots and measures throughput but its accuracy eval fails
+(crashes, produces no result, or scores below the floor), enablement can repair
+the model instead of halting the run. Single-node only; multi-node keeps the
+existing behavior.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `INFERENCE_OPTIMIZER_ENABLEMENT_ON_EVAL_FAIL` | `1` (on) | Route a baseline accuracy-eval failure into enablement instead of halting. Set `0`/`false`/`no`/`off` to keep the legacy salvage/stop behavior. Reader: `_accuracy_gate.enablement_on_eval_fail_enabled`. |
+| `INFERENCE_OPTIMIZER_ENABLEMENT_ACCURACY_FLOOR` | `0.05` | Shared accuracy floor used by BOTH the baseline eval-failure trigger and the enablement KEEP gate. A collapse guard rather than a quality bar — raise it for a genuine quality gate. A score of exactly `0.0` always fails regardless of the floor (strictly positive is required); otherwise `score >= floor` passes. Accepts a finite value in `[0, 1]`; out-of-range values are ignored with a warning. Reader: `_accuracy_gate.enablement_accuracy_floor`. |
+
+---
+
 ## Framework / source-tree discovery
 
 The following variables configure framework source discovery and path overrides.

src/hyperloom/agents/critic/SKILL.md

@@ -122,11 +122,13 @@ the absence of priors as *unknown*, not as *no contradicting prior*:
 - For **`enablement_landing`** proposals (enablement / framework-agent
   authoring `integrate_patch`): treat like `evidence_producer` — an absent
   KB prior is the default cold-start state, not a blocker. Do **not** block
-  on a missing before/after benchmark, accuracy gate, or a restated
-  rollback plan: the model cannot boot yet (so that evidence cannot
-  exist), and rollback is guaranteed by the enablement integrate executor
-  + runnable-decision gate. Approve unless a *contradicting* KB prior or a
-  packet-local defect (e.g. a malformed patch) is present.
+  on a missing throughput before/after or a restated rollback plan: rollback
+  is guaranteed by the enablement integrate executor + runnable-decision gate.
+  Boot-origin has no baseline yet; eval-origin booted but missed the accuracy
+  floor and the downstream gate re-runs the accuracy eval, so the KEEP evidence
+  is that re-run, not a throughput before/after. Approve unless a
+  *contradicting* KB prior or a packet-local defect (e.g. a malformed patch) is
+  present.
 - For **`evidence_producer`** proposals (`explore` / `specialist` /
   `profile` / `kernel_opt` / ...): an absent KB prior is the **default
   cold-start state**, not a blocker. Approve unless a *contradicting*
@@ -148,7 +150,7 @@ Every proposal in `judge_bundle.proposals` is classified into one of:
 | Class | Actions | Approve bar |
 |---|---|---|
 | `patch_landing` | `integrate`, `integrate_patch`, `apply_patch` (production promotion) | Strict — comparable before/after benchmark + accuracy gate + active-path proof + rollback. Critic is the last gate before `optimization_stack` / `framework_source_roots` mutates. |
-| `enablement_landing` | `integrate` / `integrate_patch` / `apply_patch` tagged `params.enablement` or `params.framework_agent_authoring` | Structural — same bar as `evidence_producer` (provenance + in-phase + no contradicting KB prior). The patch only makes the model **boot at all** (runnability, not throughput): it is dispatched *before* any usable baseline, so a before/after benchmark + accuracy gate are impossible by construction, and rollback is guaranteed by the enablement integrate executor (`git apply` + `git reset --hard` on REVERT) plus the downstream runnable-decision gate. **Default approve when KB priors are silent.** |
+| `enablement_landing` | `integrate` / `integrate_patch` / `apply_patch` tagged `params.enablement` or `params.framework_agent_authoring` | Structural — same bar as `evidence_producer` (provenance + in-phase + no contradicting KB prior). The patch makes the model **run correctly** (runnability, or the accuracy floor for eval-origin — not throughput): boot-origin is dispatched *before* any usable baseline, and eval-origin booted but missed the accuracy floor. A throughput before/after is impossible/irrelevant by construction; rollback is guaranteed by the enablement integrate executor (`git apply` + `git reset --hard` on REVERT) plus the downstream runnable-decision gate (which additionally re-runs the accuracy eval for eval-origin). **Default approve when KB priors are silent.** |
 | `evidence_producer` | `explore`, `specialist`, `sweep`, `profile`, `roofline`, `kernel_opt`, `deep_kernel_analysis`, `operator_tuning`, `vendor_kernel_config` | Structural — provenance non-empty (specialist or default_grid), action in current phase's allowed set, no contradicting KB prior. **Default approve when KB priors are silent.** |
 | `framework_op` | `baseline`, `target_analysis`, `recover`, `report`, `session_breakdown`, `framework_pr` | None — approve by default; Critic is not a useful gatekeeper here. (`framework_pr` = FRAMEWORK_PR candidate pre-screen; landing is re-reviewed strictly as `integrate_patch`.) |
 
@@ -204,10 +206,11 @@ Return `approve` only when all blocker risks are cleared:
 - Robustness findings and known failure patterns do not contradict the
   decision.
 
-### `enablement_landing` proposals — structural-only (pre-boot patch)
+### `enablement_landing` proposals — structural-only
 
 Enablement / framework-agent-authoring `integrate_patch` proposals whose
-purpose is to make the model **boot at all**. Review them with the
+purpose is to make the model **run correctly** — boot-origin (boot at all)
+or eval-origin (boot but meet the accuracy floor). Review them with the
 `evidence_producer` structural bar, **not** the strict `patch_landing`
 bar. Return `approve` when:
 
@@ -216,14 +219,16 @@ bar. Return `approve` when:
 - No KB prior actively contradicts the patch, and the packet shows no
   self-evident defect (e.g. a patch that fails `git apply --check`).
 
-Do **not** require a comparable before/after benchmark or accuracy gate —
-there is no bootable baseline yet, so that evidence cannot exist. Do
-**not** block solely because the proposal does not restate a rollback
-plan: the enablement integrate executor reverts with `git reset --hard`
-and the runnable-decision gate REVERTs any patch that does not boot. The
-runnable gate — not the Critic — is the real filter for these patches.
-Use `needs_review` only when the packet shows an actual defect that the
-runnable gate would not catch.
+Do **not** require a comparable throughput before/after: boot-origin has no
+bootable baseline yet, and eval-origin's KEEP evidence is the downstream
+accuracy re-run, not a throughput delta. Do **not** block solely because the
+proposal does not restate a rollback plan: the enablement integrate executor
+reverts with `git reset --hard`, the runnable-decision gate REVERTs any patch
+that does not boot, and for eval-origin it additionally re-runs the accuracy
+eval and REVERTs a patch that still misses the floor. The runnable/accuracy
+gate — not the Critic — is the real filter for these patches. Use
+`needs_review` only when the packet shows an actual defect that the gate would
+not catch.
 
 ### `evidence_producer` proposals — structural-only
 

src/hyperloom/agents/critic/actions/review_coordinator_inbox.md

@@ -67,11 +67,13 @@ Default behavior summary:
 - `patch_landing` proposal without comparable benchmark + accuracy
   gate → `needs_review` (or `reject` if the packet itself shows a
   regression).
-- `enablement_landing` proposal (pre-boot enablement / framework-agent
+- `enablement_landing` proposal (enablement / framework-agent
   `integrate_patch`) → **approve** on the structural bar; do NOT block on
-  a missing before/after benchmark, accuracy gate, or restated rollback
-  plan (the model cannot boot yet and rollback is automatic). The
-  downstream runnable-decision gate REVERTs any patch that fails to boot.
+  a missing throughput before/after or a restated rollback plan (rollback is
+  automatic). Boot-origin has no baseline yet; eval-origin booted but missed the
+  accuracy floor. Either way the downstream runnable-decision gate REVERTs any
+  patch that fails to boot, and for eval-origin additionally re-runs the accuracy
+  eval and REVERTs a patch that still misses the floor.
 - `framework_op` proposal → approve by default; only block when
   structurally malformed.
 

src/hyperloom/agents/framework/AGENTS.md

@@ -67,12 +67,13 @@ The package serves two distinct objectives, gated differently:
   only if throughput improves. Patch authoring for perf is owned by the
   Hyperloom `specialist → integrate_patch` path; this package does the
   discovery + static audit + git-apply/bench execution.
-- **Enablement** (opt-in) — make a currently **non-runnable**
-  `(model, backend)` combo *run at all*. This path DOES author bridging
+- **Enablement** (opt-in) — make a `(model, backend)` combo that is
+  **non-runnable**, or that boots but **fails its accuracy eval**, *run
+  correctly*. This path DOES author bridging
   patches (via the `enablement_specialist` domain / `SpecialistRunner`
   worktree authoring) and is gated on **runnability** (server boots + minimal
-  correctness), not throughput. Pure, GPU-free building blocks live in this
-  package:
+  correctness) or, for eval-origin, the accuracy floor; not throughput. Pure,
+  GPU-free building blocks live in this package:
   - `hyperloom.agents.framework.enablement` — failure-signature classifier
     (`classify_failure`) + `EnablementRequest` + the `runnable_decision` gate.
   - `hyperloom.agents.framework.enablement_ops` — discovery + authoring:

src/hyperloom/agents/framework/README.md

@@ -10,25 +10,27 @@ integration uses the FRAMEWORK discovery path:
 - **Standalone PR exploration** (`fa candidates` / `fa explore`) —
   ad-hoc tooling outside the `inference_optimizer` runtime path.
 
-- **Enablement** (opt-in) — make a currently **non-runnable**
-  `(model, backend)` combo *run at all* by authoring a bridging patch. Unlike
-  the perf path (gated on throughput), enablement is gated on **runnability**
-  (server boots + minimal correctness). See
+- **Enablement** (opt-in) — make a `(model, backend)` combo that is
+  **non-runnable**, or that boots but **fails its accuracy eval**, *run
+  correctly* by authoring a bridging patch. Unlike the perf path (gated on
+  throughput), enablement is gated on **runnability** (server boots + minimal
+  correctness) or, for an eval-origin trigger, meeting the accuracy floor. See
   [Enablement path](#enablement-path-non-runnable-model--backend-combos).
 
 See [`SKILL.md`](./SKILL.md) for the full architectural overview.
 
 ## Enablement path (non-runnable model + backend combos)
 
-When a `(model, backend)` combo will not start, the enablement building
-blocks turn the failure into an authored bridging patch, gated on *does it
-run* rather than *is it faster*:
+When a `(model, backend)` combo will not start, or it starts but fails its
+accuracy eval, the enablement building blocks turn the failure into an authored
+bridging patch, gated on *does it run correctly* rather than *is it faster*:
 
 1. **Classify** — `hyperloom.agents.framework.enablement.classify_failure(log)` parses a
-   launch/import/build log into a `FailureSignature`
+   launch/import/build/eval log into a `FailureSignature`
    (`missing_model_arch` / `unsupported_dtype` / `hip_kernel_missing` /
    `import_error` / `shape_mismatch` / `not_implemented` /
-   `capability_disabled`) with the offending file/symbol and a `bridge_layer`.
+   `capability_disabled` / `accuracy_below_floor` / `eval_runtime_failure`)
+   with the offending file/symbol and a `bridge_layer`.
 2. **Discover** — `hyperloom.agents.framework.enablement_ops.build_search_plan(...)`
    picks the repos to scout (the framework repo, plus ROCm/HIP/aiter via
    `repo_map.bridge_repo_urls` for the failure's bridge layer) and ranks
@@ -40,7 +42,9 @@ run* rather than *is it faster*:
    `SpecialistRunner`, which writes the patch into an isolated worktree.
 4. **Verify** — `hyperloom.agents.framework.enablement.runnable_decision(...)` is the
    KEEP/REVERT gate: the launch probe must exit 0 (no timeout) and any minimal
-   correctness check must pass; the same failure re-appearing is a reject.
+   correctness check must pass; the same failure re-appearing is a reject. For an
+   eval-origin trigger the gate additionally re-runs the accuracy eval and
+   REVERTs a patch that boots but still misses the accuracy floor.
 
 Editing ROCm/HIP source (`/opt/rocm`) is a **default-on** part of the
 enablement path — the IO-side allowlist always surfaces those roots (alongside

src/hyperloom/agents/framework/SKILL.md

@@ -104,8 +104,9 @@ A discovered candidate reference now drives the enablement targeted build
 
 ## Enablement ladder (methodology)
 
-When a candidate is for enablement (making a non-runnable `(model, backend)`
-combo boot at all, not perf), the repair follows a tiered ladder — diagnose the
+When a candidate is for enablement (making a `(model, backend)` combo that is
+non-runnable, or boots but fails its accuracy eval, run correctly — not perf),
+the repair follows a tiered ladder — diagnose the
 missing capability layer once, then climb only as far as needed: Rung 0
 diagnose, 1 serve-flag/config wire-up, 2 in-tree source patch, 3 attempt-scoped
 runtime, 4 source localization, 5 off-loop compiled build. A supported-but-un-wired

src/hyperloom/agents/framework/decision.py

@@ -115,8 +115,24 @@ def prior_score(
     weights = [score for score, _ in associated]
     weight_sum = sum(weights) or 1.0
     avg_association = sum(weights) / len(weights)
-    gain = sum(to_float(rec.get("tps_delta_pct"), default=0.0) * w for w, rec in associated) / weight_sum
+
+    def _realized_gain(rec: dict[str, Any]) -> float:
+        """Throughput a record actually delivered, else 0.
+
+        A reverted candidate's throughput was never kept, and an accuracy
+        regression disqualifies it outright, so neither may raise the prior.
+        """
+        if str(rec.get("outcome") or "") != "integrated":
+            return 0.0
+        if to_float(rec.get("accuracy_delta_pct"), default=0.0) < 0.0:
+            return 0.0
+        return to_float(rec.get("tps_delta_pct"), default=0.0)
+
+    gain = sum(_realized_gain(rec) * w for w, rec in associated) / weight_sum
     gain_score = max(0.0, min(1.0, gain / 20.0))
+    accuracy_penalty = (
+        sum(w for w, rec in associated if to_float(rec.get("accuracy_delta_pct"), default=0.0) < 0.0) / weight_sum
+    )
     param_score = 0.0
     if candidate_params:
         param_hits = 0.0
@@ -138,7 +154,7 @@ def _success_value(outcome: str) -> float:
 
     apply_score = sum(_success_value(str(rec.get("outcome") or "")) * w for w, rec in associated) / weight_sum
     quality = 0.45 * apply_score + 0.35 * gain_score + 0.20 * param_score
-    score = min(1.0, avg_association) * quality
+    score = min(1.0, avg_association) * quality * (1.0 - accuracy_penalty)
     return round(max(0.0, min(1.0, score)), 4)
 
 

src/hyperloom/agents/framework/enablement.py

@@ -32,11 +32,16 @@
 SERVE_FLAG = "serve_flag"
 # Resource constraints (OOM, TP/GPU count) are NOT code acquisition targets.
 RESOURCE_CONSTRAINT = "resource_constraint"
+# Accuracy-eval triggers (values match _accuracy_gate EVAL_KIND_*): a booting
+# baseline whose accuracy is below the floor, and a crashed eval run.
+ACCURACY_BELOW_FLOOR = "accuracy_below_floor"
+EVAL_RUNTIME_FAILURE = "eval_runtime_failure"
 UNKNOWN = "unknown"
 
 # Ordered most-specific to least-specific.
 FAILURE_KINDS: tuple[str, ...] = (
     MISSING_MODEL_ARCH,
+    ACCURACY_BELOW_FLOOR,
     RESOURCE_CONSTRAINT,
     HIP_KERNEL_MISSING,
     UNSUPPORTED_DTYPE,
@@ -47,6 +52,7 @@
     TOKENIZER_ERROR,
     SERVE_FLAG,
     IMPORT_ERROR,
+    EVAL_RUNTIME_FAILURE,
     UNKNOWN,
 )
 
@@ -186,6 +192,17 @@ def _grp(match: re.Match[str]) -> str:
         confidence=0.95,
         symbol_from=_grp,
     ),
+    _Rule(
+        # A booting baseline whose accuracy is below the floor. Not a bridge-repo
+        # target — the fix is correctness of the model's real output.
+        kind=ACCURACY_BELOW_FLOOR,
+        bridge_layer="",
+        patterns=(
+            re.compile(r"accuracy\s+.*?(?:did not meet|below)\s+.*?floor"),
+            re.compile(r"baseline[_ ]accuracy[_ ]below[_ ]floor"),
+        ),
+        confidence=0.9,
+    ),
     _Rule(
         kind=HIP_KERNEL_MISSING,
         bridge_layer="rocm_hip",
@@ -316,6 +333,19 @@ def _grp(match: re.Match[str]) -> str:
         confidence=0.7,
         symbol_from=_grp,
     ),
+    _Rule(
+        # LAST rule: a generic eval-run crash. Kept lowest-priority so a
+        # co-occurring import/serve-flag/tokenizer/HIP signature in the same log
+        # wins as the primary root cause and this only surfaces as secondary.
+        kind=EVAL_RUNTIME_FAILURE,
+        bridge_layer="",
+        patterns=(
+            re.compile(r"run_eval failed with exit code"),
+            re.compile(r"ERROR: run_eval failed"),
+            re.compile(r"accuracy eval (?:failed|crashed|did not run)"),
+        ),
+        confidence=0.6,
+    ),
 )
 
 
@@ -682,7 +712,9 @@ def is_targeted_build_candidate(
 
 
 __all__ = [
+    "ACCURACY_BELOW_FLOOR",
     "CAPABILITY_DISABLED",
+    "EVAL_RUNTIME_FAILURE",
     "FAILURE_KINDS",
     "HIP_KERNEL_MISSING",
     "IMPORT_ERROR",