feat(consolidation): warn when a run dead-letters a large share of its work

[JoshFunnell]

The observability half of #2973, following the close of #3309.

#3309 was closed because failure-classification on the consolidation path was too brittle and the recovery path already exists. That was right, and this PR does not reintroduce classification. It implements what was suggested instead:

the missing piece is surfacing it (alerting / a visible warning when a run stamps a large fraction of its pending set) so the endpoint gets used the same day rather than weeks later. That's observability, not classification, and it doesn't need to know a single provider's error vocabulary.

One deliberate divergence, stated up front: you wrote "pending set"; this measures a large fraction of the run's attempted memories — memories_failed / (memories_processed + memories_failed) — not bank-wide pending. So a wiped round still warns even when the bank's backlog is far larger than max_memories_per_round, while a slow bleed of a couple of rows per large round does not. That is the mass-failure shape recover is for, and it needs no opinion about why rows failed. Bank-wide pending would need an extra query and would still not change what this run just stamped.

The gap

Memories that exhaust consolidation retries are stamped consolidation_failed_at and are then skipped by every later run. That default is correct — the engine cannot know whether the provider refused this content or was merely unavailable — but the run itself is quiet. failed_consolidation stays queryable in bank stats; what is missing is a signal at the moment a run mass-stamps failures. POST /v1/default/banks/{bank_id}/consolidation/recover already fixes it in one call, but only for someone who knows to look.

The change

At the end of a consolidation run: if memories_failed >= 3 and memories_failed / (memories_processed + memories_failed) >= HINDSIGHT_API_CONSOLIDATION_DEAD_LETTER_WARN_FRACTION (default 0.5, 0 disables), emit one WARNING. No exception types, no status codes, no message substrings — the decision reads two counters the job already keeps. memories_failed counts only rows stamped consolidation_failed_at in this run; memories_processed is its success counterpart.

[CONSOLIDATION] bank=acme dead-lettered 40/40 memories (100%) in this run. They are
stamped consolidation_failed_at and will NOT be retried by later runs. If this was a
provider outage or quota window rather than unusable content, clear the stamps with:
POST /v1/default/banks/acme/consolidation/recover (CLI: hindsight bank
consolidation-recover acme). Set HINDSIGHT_API_CONSOLIDATION_DEAD_LETTER_WARN_FRACTION=0
to silence this.

Read from the bank-resolved config like the sibling consolidation knobs, so a per-bank value takes effect rather than only round-tripping through bank templates.

The floor of 3 is a constant rather than a second dial: a bank consolidating one or two memories at a time hits 100% routinely, and a warning that fires constantly is one operators stop reading.

Non-goals

No metrics, no alerting integration, no automatic recovery — this assumes WARNING-level logs are somewhere an operator sees, which is the minimum surface that makes recover reachable the same day. Deciding to re-run rows a provider rejected stays with the operator.

Surfaces deliberately not updated

The repo is wider than the engine, so rather than leave these silent:

Surface	State
hindsight-api-slim engine + config + BankTemplateConfig	Updated
openapi.json, bank-template-schema.json, docs skill	Updated, regenerated
Go / Python / TypeScript clients	Updated, regenerated
docs/developer/configuration.md	Updated — new row for the env var
hindsight-control-plane — bank-config-view.tsx	Not updated. Bank-config fields there are hand-written FieldRows with per-locale label/description keys, and the view already omits several per-bank consolidation knobs (consolidation_max_memories_per_round, consolidation_llm_parallelism), so this is consistent with current practice rather than a new gap. Say the word and I'll add the row plus locale strings.
hindsight-cli	Not updated. No new flag: per-setting bank config rides bank set-config, and the coverage manifest already skips update_bank_config.updates wholesale as "flattened into per-setting flags". No coverage-manifest entry is required by this change.
monitoring/ dashboards	Not updated. This adds no metric — see Non-goals; the signal is a log line.
Helm / docker	Not applicable — the setting is an ordinary env var with a default.

Tests

22 tests, no database and no LLM — the decision is a pure str | None function, so the caller logs only on a real signal. Weighted toward the shapes that must stay quiet (clean run, empty run, below fraction, below floor, disabled, negative counters), with both inclusive boundaries pinned: fraction == threshold and failed == 3. Two cover the call site rather than the function — that the fraction comes from the bank-resolved config, and that a non-None decision is actually logged.