Skip to content

Fix derived label retries, weekly floors and read budgets - #574

Draft
samrusani wants to merge 103 commits into
cursor/derived-labels-docs-a34efrom
codex/derived-label-fix-round
Draft

samrusani wants to merge 103 commits into
cursor/derived-labels-docs-a34efrom
codex/derived-label-fix-round

Conversation

@samrusani

@samrusani samrusani commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Open-loop extraction could fail after a source moved A → B → A, weekly candidates could be stored below their report inputs, and two readers omitted the caller's full fence. This correction recovers the existing per-user digest after a conflict, inserts weekly candidates with the artifact input floor, and fences source GET and enabled legacy review lists, including saved quotes.

The draft remains against the final branch of the existing stack, #571. Round two adds ordinary commits to #574 only; This work does not modify #565–#573. The owner accepted the recorded earlier force-pushes on the condition that there are no further rewrites. This round contains none. The tower must recheck the new head before merge.

The combined candidate retains the earlier UUID normalization, producer and scheduler identity, all-of selection/refill, parser-recognized source-reference withholding, retryable label-lock refusal, SQLite rollback, PostgreSQL repair CAS and source-review writeback fixes. Round two adds:

  • Digest-only conflict recovery on both stores, while retaining the initial scoped lookup and the existing labels.
  • The weekly artifact's sources, memories, loops and reports in the candidate's insert floor, in the same transaction.
  • Source GET's effective domain, sensitivity and locked-project fence, with the same 404 as a missing source. Legacy review items apply effective admission before counts and limits, refill past hidden rows, and recheck original/imported candidates' saved quotes against current sources.
  • Request-local dependency and pure-parser reuse, plus verified single-original-source copy reuse. Missing, aliased, recursive and aggregate parents retain canonical settlement. Parent labels, scope, floor, tenant and structural errors remain in the cache key; admission is checked separately for each caller. Label writes and rollback invalidate the request state.
  • SQL counting of definitely original rows and kernel counting of the complete derived partition. Conservative marker classification and SQLite duplicate-key/JSON decoding preserve the canonical interpretation.
  • Indexed guarded UUID comparison for PostgreSQL event parents, conservative SQLite direct-parent filtering before the event limit, and bounded batched parent/event reads.
  • Request-local native connector input census and source-event reuse, preserving cursor-kind selection, event fallback and caller-owned copies.
  • Broader budget fixtures and guard-removal tests. Nonvacuous reimport/order controls, empty-project producer tests and HTTP/MCP/CLI redact/source-review lock cases close the reported coverage gaps.

Source GET and native MCP budgets use real minted keys. Existing registry policy still disables keybound legacy aliases, even with legacy opt-in. Enabled keyless legacy review tests exercise declared trusted/admin/read-only identities and resolved locked runtime context, plus the owner; this change does not enable the disabled aliases.

The required web audit also identified GHSA-wq5f-xc86-pv6w in Sharp 0.35.4. Sharp and its binary entries now pin 0.35.5, the patched release listed in the GitHub advisory. The existing source-map-js 1.2.2 and tinypool 2.1.2 pins remain. Framework versions are unchanged.

Validation at final head 1754f8e: GitHub CI passed. The full unit shards passed 4,527 + 5,688 + 3,778 = 13,993 tests, with 20 skipped. Combined coverage is 86%; the required 50% total and 45% combined floor across 14 protected paths passed. Full role-separated PostgreSQL integration passed 746 tests, with 1 skipped, followed by 6 passing SQLite varied-parent budget tests. Model-free LongMemEval passed 216 tests, plus two configured-vector contract controls; the semantic release gate intentionally fails closed without a provider.

Final-head web validation passed 244 tests with coverage, types, production build, accessibility and browser budgets. Production audit checked 94 packages with zero matching advisories. Full audit checked 550 packages with two moderate Vitest notices and the existing high braces exception until 2026-11-03; there are zero unexcepted advisories at or above high. Final-head lint, the 279-file mypy command, distribution/install contracts, release truth, Bandit baseline, CodeQL, secrets and ops/configuration checks passed.

Local final-head retry, weekly-floor, reader, saved-quote and held-lock controls passed 38 tests, and dependency/cache controls passed 27. At runtime-identical 0a6f6ad, 251 focused label/connector/doctor tests and both full local store budget matrices passed. Raw logs and source bindings distinguish these results from older diagnostic runs.

Read budgets: paired main 48873b0 and candidate runtime 0a6f6ad, on the same analyzed synthetic data. Final head 1754f8e adds only a prefetched-alias regression-test call; API, web, performance, integration, scripts and eval trees are byte-identical to the measured revision, verified by tree hashes.

All 9 PostgreSQL and 7 SQLite tests passed. Five samples after warmup; each cell is main wall/CPU → head wall/CPU, minimum seconds. Pack and recall satisfy 2 × main + 0.1 s for both wall and process CPU. PostgreSQL native-view minimum wall time is below 1 s. Complete SQLite counts satisfy 1 s for both clocks (maximum of the fixture minima: 0.5440 wall / 0.5438 CPU), with exact admitted populations of 0, 2,500 or 5,000. Repaired fixtures assert zero stored/effective label gaps before measuring. CPU covers the measured Python process, not PostgreSQL's server CPU.

Store / fixture / key Context pack Recall Workspace Dogfooding
postgres / varied / trusted 0.1284/0.0194 → 0.1670/0.0329 0.1317/0.0294 → 0.1457/0.0329 0.1677/0.0518 → 0.6188/0.4912 0.0782/0.0307 → 0.5336/0.4349
postgres / varied / admin 0.1620/0.0353 → 0.1802/0.0448 0.1467/0.0322 → 0.1529/0.0392 0.1737/0.0540 → 0.6167/0.4880 0.0802/0.0313 → 0.1410/0.0865
postgres / repaired / trusted 0.1330/0.0229 → 0.1441/0.0291 0.1318/0.0295 → 0.1338/0.0271 0.1657/0.0515 → 0.5998/0.4912 0.0774/0.0306 → 0.5611/0.4604
postgres / repaired / admin 0.1447/0.0276 → 0.1539/0.0372 0.1368/0.0312 → 0.1062/0.0199 0.1683/0.0527 → 0.5929/0.4901 0.0767/0.0303 → 0.1164/0.0627
postgres / plain / trusted 0.0908/0.0116 → 0.0857/0.0101 0.0820/0.0073 → 0.0755/0.0064 0.1918/0.0676 → 0.2454/0.1550 0.0781/0.0298 → 0.1495/0.1042
postgres / plain / admin 0.0815/0.0095 → 0.0884/0.0112 0.0755/0.0064 → 0.0880/0.0172 0.1861/0.0699 → 0.2330/0.1450 0.0799/0.0302 → 0.0659/0.0294
postgres / many-hidden / trusted 0.0431/0.0093 → 0.0416/0.0058 0.0227/0.0037 → 0.0183/0.0040 0.2545/0.1122 → 0.6432/0.5099 0.0841/0.0308 → 0.5317/0.4380
postgres / many-hidden / admin 0.0435/0.0093 → 0.0409/0.0054 0.0259/0.0043 → 0.0172/0.0039 0.2586/0.1161 → 0.6438/0.5085 0.0879/0.0331 → 0.1170/0.0605
postgres / many-hidden-repaired / trusted 0.1908/0.0814 → 0.0221/0.0053 0.0090/0.0018 → 0.0110/0.0024 0.2510/0.1126 → 0.6447/0.5093 0.0823/0.0311 → 0.5481/0.4424
postgres / many-hidden-repaired / admin 0.1806/0.0820 → 0.0156/0.0034 0.0089/0.0017 → 0.0101/0.0022 0.2681/0.1179 → 0.6467/0.5099 0.0887/0.0326 → 0.3254/0.0577
postgres / one-hidden / trusted 0.1628/0.0190 → 0.2065/0.0094 0.0884/0.0115 → 0.0960/0.0107 0.7317/0.0838 → 0.6493/0.2577 0.2594/0.0319 → 0.4801/0.2012
postgres / one-hidden / admin 0.1737/0.0173 → 0.1947/0.0103 0.1406/0.0162 → 0.1031/0.0083 0.6252/0.0585 → 0.6443/0.2436 0.3121/0.0431 → 0.2604/0.0520
postgres / identical / trusted 0.1489/0.0200 → 0.1429/0.0074 0.1074/0.0103 → 0.0599/0.0085 0.5831/0.0729 → 0.6583/0.2465 0.2495/0.0305 → 0.4730/0.2010
postgres / identical / admin 0.1771/0.0176 → 0.1856/0.0050 0.1014/0.0045 → 0.0569/0.0078 0.6961/0.0751 → 0.6653/0.2580 0.2056/0.0269 → 0.3453/0.0600
sqlite / mixed / trusted 0.1540/0.1532 → 0.1588/0.1582 0.2122/0.2116 → 0.2219/0.2210 Unsupported Unsupported
sqlite / mixed / admin 0.1592/0.1583 → 0.1631/0.1621 0.2198/0.2189 → 0.2307/0.2297 Unsupported Unsupported
sqlite / mixed-repaired / trusted 0.1516/0.1504 → 0.1520/0.1510 0.2063/0.2055 → 0.2125/0.2118 Unsupported Unsupported
sqlite / mixed-repaired / admin 0.1486/0.1469 → 0.1482/0.1471 0.2068/0.2056 → 0.2207/0.2191 Unsupported Unsupported
sqlite / many-hidden / trusted 0.0799/0.0790 → 0.1115/0.1102 0.0966/0.0959 → 0.1084/0.1074 Unsupported Unsupported
sqlite / many-hidden / admin 0.0785/0.0776 → 0.1092/0.1082 0.1024/0.1006 → 0.1031/0.1026 Unsupported Unsupported
sqlite / many-hidden-repaired / trusted 0.1583/0.1566 → 0.0721/0.0715 0.0827/0.0820 → 0.0825/0.0819 Unsupported Unsupported
sqlite / many-hidden-repaired / admin 0.1438/0.1433 → 0.0714/0.0708 0.0828/0.0823 → 0.0841/0.0834 Unsupported Unsupported
sqlite / one-hidden / trusted 0.0746/0.0739 → 0.0993/0.0983 0.0986/0.0966 → 0.0951/0.0944 Unsupported Unsupported
sqlite / one-hidden / admin 0.0746/0.0741 → 0.0956/0.0947 0.0938/0.0932 → 0.0970/0.0964 Unsupported Unsupported
sqlite / identical / trusted 0.0571/0.0564 → 0.0799/0.0792 0.0697/0.0690 → 0.0722/0.0715 Unsupported Unsupported
sqlite / identical / admin 0.0575/0.0569 → 0.0789/0.0783 0.0693/0.0686 → 0.0695/0.0689 Unsupported Unsupported

The new identical case has identical metadata and one parent, while text still identifies each row. The original byte-identical fixtures remain unchanged and passed separately. Their median main → head seconds:

Original control Pack Recall Workspace Dogfooding
SQLite restricted copies 0.0368 → 0.0586 0.0526 → 0.0502 Unsupported Unsupported
PostgreSQL visible copies 0.1123 → 0.1949 0.3335 → 0.1250 0.6533 → 0.6405 0.3281 → 0.4357
PostgreSQL hidden copies 0.1582 → 0.1722 0.1697 → 0.0717 1.1397 → 0.6570 0.4006 → 0.6016

The host was loaded and individual samples exceeded minima. Raw five-sample arrays, earlier failures and profiling attempts are retained; no threshold was relaxed or user process stopped. The pass establishes this fixture matrix, not an SLA under arbitrary contention.

Final-head mutation proofs: 20 of 20 selected guard removals fail the intended regression. They cover both digest retries; the complete weekly artifact-input rule; source GET; legacy effective admission and saved quotes; ancestry and original-row partitioning; bounded redact/source-review locks; plain reimport; reverse ordering; three empty-project producers; indexed PostgreSQL parent lookup; pure-parser/classifier top-level fields; source-copy closure, single-parent ambiguity and parent labels; and connector cursor-kind selection. The indexed-parent mutant fails the plain context-pack timing gate, not collection or a native-view assertion.

The first final-source attempt was 19/20: its alias test did not prime the prefetched-parent path. The final commit adds that call, ordinary controls still pass, and both alias guards are killed. This test-only correction and the failed attempt are retained. Redundant implicit weekly-parent and earlier planner removals still have no independently observable effect; they are disclosed separately and are not included as successful proofs.

Minor responses and release limits:

  • Graph-edge explanations remain outside the caller ceiling; this predates the stack. Doctor aggregate derived-label counts are new in this stack, absent from v0.20.0 and main 48873b0, and also remain outside the ceiling. The alpha limitations, dated release correction and draft security note now distinguish them correctly.
  • Restricted-to-restricted domain diagnostics can remain: generation preserves an explicitly restricted payload domain while the kernel can select another restricted input domain. Labels repair heals the mismatch. Changing the precedence is a separate product decision; this round preserves the insert contract and exact diagnostic.
  • Bounded retrieval, consolidation clustering (default 2,000) and staleness sweeps (default 500) can underfill when hidden/shared rows consume the candidate window. Counts cover the complete population. Refill/ranking remains a separate follow-up; passing timing does not establish retrieval completeness.
  • Rollup identity retains its optional internal default; its only caller, consolidation, passes the resolved identity. The full PostgreSQL consolidation producer with free-text source_refs is not added here. Existing store insert/GET/recall and kernel controls remain; the external builder probe is separate evidence.
  • Expanded open-loop parsing is source-only. Imported or hand-edited irregular/non-ASCII MEMORY-prefixed ids and unnamed SOURCE whitespace forms can remain. At 20,000 loops the external canonical reverse-lookup fixture cost about 10× the earlier scan; SQL text prefiltering is deferred until it preserves canonical escapes and spellings.
  • Repair CAS's sensitivity race is pinned. Optional domain, metadata and project_id update races remain follow-ups. Redundant producer, planner and implicit weekly-parent guards remain; some individual removals survive because another guard enforces the same outcome.
  • UUID-shaped absent project ids still need an existing project to populate project_id. SQLite retirement follows the saved-quote reader's last duplicate-key value and omitted quote subtrees; the product does not write those two forms.
  • The dead append-only helper was removed. The sources_first token-estimate increase of 5 is now stated in the changelog. Capture's small headroom under 15% and the earlier roughly 2.4 ms/dependant relabel result need a quiet-machine recheck before tagging.
  • Historical restricted-owner migration 0067 remains the separately recorded baseline issue. The new 0096 check still runs as a NOSUPERUSER NOBYPASSRLS table owner with FORCE RLS restored.
  • SQLite's on-ramp does not expose native workspace/dogfooding routes. Those cells are unsupported; SQLite validation covers its actual MCP tools and complete counts.

Upgrade Overview

Protected Areas

  • Memory schema
  • Continuity APIs

Compatibility Impact

Public request fields remain compatible. Source GET uses the declared route-local read policy: read-only identities can read admitted sources, while the public-health operator gate still refuses that profile. Source GET returns 404 outside the caller fence, including raw titles/text. Enabled legacy review responses omit hidden memories, quotes and their counts. Owner and admitted admin reads remain available. The earlier explicit producer identity and three-second retryable contention contract remain. Free-text source_refs remain usable outside UUID casts. Web CI remains on Node 22.

Migration / Rollout

Round two adds no migration beyond 0096 in the existing stack. That migration must run as the restricted table owner before requests are served. SQLite open repair, explicit repair and restore keep their atomicity boundary. Preserve a backup and run label checks on a restored copy under the release runbook.

Operator Action

The tower should independently recheck this exact new head before merge, review the disclosed graph/doctor limits, and repeat capture/relabel measurements before tagging. This draft does not establish production or provider-backed semantic evaluation.

Validation

Revision-bound results and paired same-data measurements are above. The focused reproductions preserve owner/admin controls, admitted sources, quotes, refs and connector state. The three CI unit shards, combined coverage, role-separated PostgreSQL integration, eval contracts, distributions, types, web build/accessibility/budgets and security audits are required.

Rollback

Use the owner-controlled code/database backup runbook. Failed repair leaves rows, audit events and the SQLite completion stamp unchanged; PostgreSQL repair refuses concurrent changes. Reverting code alone must not lower raised labels or reverse audit history.

A free-form project name stays in metadata_json.project_scope. v0.20.0 wrote that name into open_loops.project_id and the brief failed on Postgres.
SQLite delete, prune, and markdown replacement now use one pass over the user's loops. v0.20.0 kept the loop text for every spelling other than the stored id or source:<id>.
# Conflicts:
#	CHANGELOG.md
#	apps/api/src/alicebot_api/vnext_stores/sqlite/source_retirement.py
#	docs/alpha/known-limitations.md
#	tests/unit/test_known_limitations_page_shape.py
#	tests/unit/test_store_graph_open_loops_split.py
@samrusani samrusani changed the title Correct derived label boundaries and review evidence Fix derived label retries, weekly floors and read budgets Oct 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant