feat(engine): two-tier mental-model refresh - a deterministic fast path that serves most refreshes in one LLM call or none, with the agentic loop as escape hatch

[JoshFunnell]

Problem

Mental-model refresh was this deployment's single largest LLM consumer —
~25.1M input tokens/day (the preserved pre-trial snapshots' figure; an earlier
full-ledger measurement read 27.6M/day but its rows were pruned by the 1-day
trace-retention default before close, so we cite the reproducible number), more than every other engine operation combined —
and most of that spend produced nothing: over a third of refreshes ran the
full multi-call agentic loop to completion and applied zero edits. This
PR adds a two-tier fast path that answers those refreshes in at most one LLM
call, with the existing loop unchanged underneath as the escape hatch, and
carries 22.7 hours of production before/after evidence (the operator called the trial at ~22.7h; every rate below is normalized per day).

Baseline configuration: the baseline already ran delta mode. All 33 models
carried trigger.mode: "delta" throughout the baseline window, and the ledger
confirms it fired on essentially every refresh (39 mental_model_delta_ops
calls against 38 reflect traces over the last 6 hours of the era, the window
where every llm_requests row survived the 1-day retention default). Delta
mode does not address this cost, because it sits downstream of it: use_delta
narrows retrieval via created_after, then adds one more LLM call after the
loop returns, taking the loop's output as its input
(candidate_markdown=reflect_result.text). Over that same window the split was
176 loop calls carrying 5.25M input tokens (93.1%) against 39 delta-ops calls
carrying 388k (6.9%). Delta mode is the write-side protection and it works,
applying 372 operations across 210 completed baseline refreshes. The read-side
cost it sits behind is what this PR targets.

The baseline, measured on a production bank (33 mental models in the
baseline window, event-driven refresh after consolidation, single-tenant PG
deployment; the trial that follows ran on a consolidated roster of 29):

• the population median refresh read ~101k input tokens for a median of
  ~373 output tokens (per-refresh medians over the baseline window's
  llm_requests ledger) — a ~270:1 read/write ratio at the median; the
  heaviest observed refresh read 372k input tokens across 13 tool-loop
  calls (single instance, same window);
• 36.8% of refreshes applied zero edits (window: 24h pre-intervention,
  ledger query over llm_requests joined to refresh operations) — the full
  loop ran to completion and changed nothing;
• refresh cost grows with context, not with change: each loop iteration
  resends the whole accumulated conversation.

The pieces needed to skip the loop already exist in the codebase; they were
sequenced behind it instead of in front of it.

Mechanism

The reflect loop's cost is structural: each tool-loop iteration resends the
accumulated context (append-only assembly), so cost grows with iteration
count regardless of how little the refresh changes. The delta window read and
the structured-delta prompt — both already present — can answer most
refreshes in at most one LLM call.

Change (tier lattice)

refresh request (delta mode, baseline present)
  │
  ├─ tier 0: delta window EMPTY → preserve document, advance watermark.
  │          Zero LLM calls. (Designed to absorb the zero-edit share; in the
  │          event-driven trial the windows were never empty and cheap tier 1
  │          absorbed it instead — see Evidence.)
  │
  ├─ tier 1: window non-empty → ONE structured-delta call producing edit
  │          operations applied to the existing document. By construction it
  │          can only edit or preserve — it never writes a synthesised
  │          candidate as the whole document.
  │
  └─ tier 2: anything less than clearly safe hands back to the agentic loop
             unchanged (no baseline, model requests full context, op call or
             parse failure, all ops rejected). Outcomes reachable today stay
             reachable; the fast path changes what a refresh COSTS, not what
             it can produce.

Evidence (22.7h production trial, same bank, same models, same workload)

• Tier distribution (272 completed refreshes over the 22.7h trial era;
  sampler-verified, deduped by model+last_refreshed_at): tier 0: 0 —
  tier 1: 261 (96.0%) — agentic hand-back: 11 (4.0%), every hand-back
  carrying a fast_path_fallback_reason (the escape hatch fired and was
  never dominant). Cost: tier-1 refreshes ran at the era's 12.6k-token
  median (ledger); the 11 hand-backs rode the full agentic path at the
  old cost class. Tier 0 firing zero times is itself a finding: under
  event-driven scope-matched triggering a refresh window is almost never
  empty, so the old zero-edit class lands on cheap tier 1 instead of
  tier 0 (see the zero-edit line below).
• Input tokens per refresh, median and p90: 102,086 / 205,810 before →
  12,600 / 16,526 after (~8.1x cheaper at median). Configuration note:
  the after side is the full trial era from the live ledger; the before
  side was measured at the +12h checkpoint from the then-remaining ledger
  window and preserved as a text snapshot — the deployment ran the
  1-day HINDSIGHT_API_LLM_TRACE_RETENTION_DAYS default, which pruned
  the baseline era's rows before trial close (a deployment lesson this
  PR's evidence protocol now carries: snapshot before you rely on the
  trace ledger).
• Total refresh-lane tokens/day: ~25.1M before → ~5.4M (5.09M over the
  22.7h era; ~4.7x reduction), same bank, heavier-than-usual workload
  (several hundred facts retained during the trial day).
• Zero-edit refreshes: the design expected tier 0 to absorb them; the
  trial corrected the mechanism. Tier 0 fired 0 times (windows are almost
  never empty under event-driven triggers); instead the zero-edit CLASS
  shrank — 48 of 272 completed refreshes (17.6%) vs 36.8% baseline
  (24h pre-trial window) / 48.1% (full baseline era) — and each one now
  costs a single ~12.6k tier-1 call instead of a ~100k agentic run.
• Retain→fresh-model latency: median 458s → 110s, p90 1,115s → 869s,
  same instrument on both sides (each completed retain paired with the
  first refresh completing after it, capped at 120 min — do not compare
  against other definitions; before side snapshot-sourced, above).
• Quality: per-change identifier retention probe (regex identifier classes,
  lost identifiers logged verbatim) ran across the trial: 276 change
  events, 14 loss events, 99.8% identifier retention (16,451 kept / 26
  lost / 500 gained). Disposition: the largest loss cluster was the
  block-index mis-target this PR discloses and fixes (two destroyed
  blocks, both restored from history; zero mis-target losses after the
  anchor fix went live mid-trial); the remainder audited as legitimate
  compression or supersession. Streamed calls do not report
  cached_tokens; latency, not that column, is the caching instrument.
  Output-token figures in this body understate true generation on
  reasoning models: thoughts_tokens is billed at the output rate but
  excluded from output_tokens in the ledger.

Relationship to in-tree work

• Builds on fix(mental-models): never overwrite a document with a delta-window candidate #3182 (fixing Delta refresh fallback overwrites the whole document with a new-facts-only synthesis #3112, ba5a481): that fix stops a delta-window
  candidate replacing the WHOLE document when its operations fail to land.
  Disclosure 1 below extends the same protection one level down — a
  wrong-but-in-range BLOCK index could still silently overwrite an unrelated
  block inside a "successful" apply. Same defect family (delta refreshes
  destroying content grounded in older memories), complementary layer.
• Complements feat(mental-models): dry-run refresh and keep_trace for troubleshooting #3119 (dry-run refresh + keep_trace): those tools
  troubleshoot a refresh you can re-run; disclosure 2's serving_tier in the
  outcome metadata makes tier distribution recoverable for refreshes that
  already happened — the after-the-fact half of the same observability story.

Safety

• Strictly additive: tier 2 is the unmodified existing path; tiers 0/1 are a
  pre-gate in front of it.
• Config kill switches: global default plus per-model trigger override
  (delta_fast_path); either disables the pre-gate entirely.
• The fast path never deletes content it did not target: block operations are
  anchor-guarded (see disclosure 1) and fall back to the loop when rejected.

Disclosures (defects this trial surfaced in the fast path, with fixes in-branch)

1. Block-op index mis-targeting (Fixes Delta block ops can silently overwrite the wrong block: bare index targeting, range-only validation #3273; commit 75fe395). Delta block operations
   addressed paragraphs by a bare LLM-computed integer index into an
   unannotated serialization, and apply-side validation checked RANGE only —
   a wrong-but-in-range index silently overwrote an unrelated block and the
   fast path committed it as success. Observed once in production (one tier-1
   refresh, two off-by-one ops, both destructive). Fix: ops carry a verbatim
   content anchor compared before mutating (mismatch → logged skip), and the
   serialization annotates each block with its index. Regression tests
   included.
2. Serving tier was unobservable after the fact (Fixes Refresh serving path is unobservable after the fact: reflect_response is overwritten, result_metadata carries no tier #3274; commit 84462bf). The
   tier lived only on each model's latest reflect_response and was
   overwritten by the next refresh, so tier distribution — the PR's own
   headline evidence — was unrecoverable from the database and had to be
   reconstructed by an external sampler. RefreshMentalModelOutcomeMetadata
   now records serving_tier and fast_path_fallback_reason per operation.
   A two-tier system whose tier cannot be read back cannot be operated or
   measured; we consider this part of the feature, not an extra.
3. Scope grammar gained an entity leaf (4caca0b, separable). Trial
   observation: on a bank whose tag vocabulary is a handful of broad topics,
   tag scopes cannot isolate a subject (best tag scope for one subject: 25%
   precision against a 15.5% base rate; the entity association on the same
   corpus: 94.2% precision / 86.2% recall). TagGroupEntityLeaf
   ({entities: [names], match: any|all}) joins the existing recursive
   tag-group grammar, riding every surface it already reaches — refresh
   scope, the staleness gate, retrieval filtering. Association is
   inheritance-aware (direct postings OR through source_memory_ids, the
   lane observations use by design). Entity leaves are barred under not at
   the API edge because two surfaces evaluate them permissively (rationale in
   code). 22 tests. Happy to split into its own PR if preferred.

Open questions for maintainers

• Should the fast path default ON (mental_model_delta_fast_path = true) or
  ship opt-in?
• needs_full_context as the hand-back reason name — better ideas welcome.
• The now-dead DELTA_SYSTEM_PROMPT constant: remove in this PR or leave for
  a cleanup pass?
• Disclosure 3 in-PR or split?

[nicoloboschi]

are you using delta mode? that should fix it already

[JoshFunnell]

Yes. Delta mode was on for every model across the entire baseline, and the ledger shows it running on essentially every refresh.

Configuration: the pre-trial trigger snapshot has trigger.mode: "delta" on all 33 models in the baseline window.

Ledger rows: taking the last 6 hours of the baseline era (2026-08-08 03:02Z to 09:06Z), the window where every llm_requests row survived the retention default:

scope	calls	input tokens
reflect	38	1,753,314
reflect_tool_call	138	3,499,106
mental_model_delta_ops	39	387,555

39 delta-ops calls against 38 reflect traces, so delta fired on every refresh. Over the full baseline era it applied 372 operations across 210 completed refreshes. Delta mode was working.

It cannot fix the cost, because it sits downstream of it. In _execute_mental_model_refresh, use_delta does two things: it sets created_after on reflect_kwargs, narrowing retrieval to memories newer than the last refresh, and after the loop returns it adds one more LLM call to convert the result into edit operations. The loop itself is unconditional:

reflect_result = await self.reflect_async(**reflect_kwargs)   # runs in both modes
...
if use_delta:
    user_prompt = build_structured_delta_prompt(
        current_document_json=current_doc.model_dump_json(),
        candidate_markdown=reflect_result.text,   # delta consumes the loop's output
        ...
    )

candidate_markdown=reflect_result.text is the constraint. The delta step's input is the agentic loop's output, so delta runs after the loop and never in place of it. In the table above that is 176 loop calls carrying 5.25M input tokens (93.1%) against 39 delta calls carrying 388k (6.9%). Delta mode adds a call, and the loop calls it would need to remove are upstream of it.

The window narrowing was already active as well. created_after was set on every one of those refreshes and the median refresh still read 102,086 input tokens, because the cost is the loop's append-only context assembly across roughly 4.6 calls rather than the size of the window it retrieves. 48.1% of baseline refreshes ran that loop and applied zero edits.

That is what the fast path targets, and it reuses the delta machinery rather than replacing it. Tier 1 makes the same build_structured_delta_prompt call with candidate_markdown="", since the synthesis call is precisely the one being skipped, so the delta prompt already in the tree answers the refresh from the window directly instead of from a loop-produced candidate. Same bank, same models, same workload: median input per refresh 102,086 to 12,600, calls per trace 4.39 to 1.23, with the loop retained as the escape hatch for the 4% of refreshes the fast path declines.

The PR body should have stated this outright. It assumes delta mode in the tier lattice but never says the baseline was already 100% delta. I will add the baseline configuration explicitly under Problem.