Skip to content

fix(stats): drop permanently failed memories from pending_consolidation - #3397

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/pending-consolidation-excludes-failed
Aug 11, 2026
Merged

fix(stats): drop permanently failed memories from pending_consolidation#3397
nicoloboschi merged 1 commit into
mainfrom
fix/pending-consolidation-excludes-failed

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #3362.

The bug

pending_consolidation counted every fact with consolidated_at IS NULL, including the ones stamped consolidation_failed_at that the consolidator's own candidate query (reads.find_unconsolidated) skips on purpose. The gauge had a floor no amount of work could clear — it sat above ?consolidation_state=pending by exactly failed_consolidation, indefinitely — so an operator could not tell a real backlog from an abandoned residue.

pending now carries the consolidator's predicate. The two buckets are disjoint: a fact the LLM could not handle is counted once, under failed, and leaves that bucket only via POST /consolidation/recover.

The same predicate was missing in five more places

  • get_bank_stats keeps a second copy of the freshness SQL for the writes_memory_rows_in_sql path — fixing only counts.py would have left the default Postgres path wrong.
  • hindsight.consolidation.backlog had the same floor, which made "backlog > 0 for N minutes" unalertable on any bank holding a residue.
  • reflect's tool_search_observations derives is_stale / freshness from this count (stale at ≥10), so a residue told the model the observations were stale on every reflect call, for ever. Fixed transitively via get_bank_freshness.
  • The control plane's consolidation card computed done = total - pending, which would have counted the failed rows as done once pending got strict; it now subtracts both, so done / pending / failed no longer overlap.
  • The benchmark runner waits for this count to reach 0, so one permanently failed fact burned the full 3000s timeout.

Everything else that answers "what is left to consolidate" already excluded them — the memories list filter, count_unconsolidated, and the banks_needing_consolidation() maintenance routine — so scheduling was never spinning on the residue.

The issue also names components/data-view.tsx; that surface no longer exists on main, so only the stats card needed the render fix.

Notes

  • pending_consolidation's OpenAPI description said "not yet processed into observations", which is what invited the superset reading. It now states the exclusion; specs and clients are regenerated.
  • The extra consolidation_failed_at IS NULL term is not in idx_memory_units_unconsolidated's predicate, so it is a recheck on rows the index already returned. The failed set is tiny by construction, and the metrics query keeps its SET LOCAL enable_seqscan = off nudge.

Testing

  • New regression test asserts pending_consolidation == GET /memories/list?consolidation_state=pending → total — the exact equality the issue reported as broken — plus failed_consolidation counted separately.
  • test_bank_stats_reports_failed_consolidation and test_get_bank_freshness_... now assert the strict counts instead of >=.
  • test_backlog_metrics.py asserts the gauge SQL carries the exclusion.
  • Local: those three stats tests, test_consolidation_failure_recovery.py + test_maintenance_routines.py (31 passed), and test_backlog_metrics.py (7 passed) all pass; lint clean.

…on (#3362)

`pending_consolidation` counted every fact with `consolidated_at IS NULL`,
including the ones stamped `consolidation_failed_at` that the consolidator's
own candidate query (`reads.find_unconsolidated`) excludes on purpose. The
gauge therefore had a floor no amount of work could clear: it sat above
`?consolidation_state=pending` by exactly `failed_consolidation`, and an
operator could not tell a real backlog from an abandoned residue.

`pending` now carries the consolidator's predicate, so the two buckets are
disjoint and a bank with no live backlog reaches zero. The same predicate was
missing in five more places:

- `get_bank_stats` keeps a second copy of the freshness SQL for the
  `writes_memory_rows_in_sql` path — fixing only `counts.py` would have left
  the default Postgres path wrong.
- `hindsight.consolidation.backlog` had the same floor, which made
  "backlog > 0 for N minutes" unalertable on any bank holding a residue.
- reflect's `tool_search_observations` derives `is_stale` / `freshness` from
  this count, so a residue told the model the observations were stale on every
  call, for ever (fixed transitively via `get_bank_freshness`).
- the control plane's consolidation card computed `done = total - pending`,
  which would have counted the failed rows as done once pending got strict.
- the benchmark runner waits for this count to reach 0, so one permanently
  failed fact burned the full 3000s timeout.

Everything else that answers "what is left to consolidate" already excluded
them: the memories list filter, `count_unconsolidated`, and the
`banks_needing_consolidation()` maintenance routine — so scheduling was never
spinning on the residue.
@nicoloboschi
nicoloboschi merged commit dbe0ffb into main Aug 11, 2026
107 checks passed
@nicoloboschi
nicoloboschi deleted the fix/pending-consolidation-excludes-failed branch August 11, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observations "Pending" counts facts the consolidator has permanently given up on

1 participant