Move standard.site documents from the Fly proxy into D1 - #424
Open
disnet wants to merge 7 commits into
Open
Conversation
Documents were the last content type where Fly held authoritative state and sat on the read path: a persistent Bun WebSocket spliced `site.standard.document` records into a per-author JSON blob in the proxy's SQLite, and every read traversed Worker → Fly → blob. That meant two Jetstream consumers in two runtimes with two cursor stores, a `wantedDids` filter derived from read traffic rather than from the subscription table, and documents going down whenever one Fly machine did. This applies the move feeds already made. The JetstreamPoller DO drains `site.standard.document` plus its `app.standard-reader.collection` sidecar straight into D1; `listRecords` backfill (the firehose never replays history) runs at subscribe time and from an hourly reconcile; reads are plain queries. The filter that cost the proxy ~550 lines of options_update reconciliation is connect-time state here, because the socket is rebuilt every 60s cycle. Spike handling is layered, per the plan's rev-2 review finding: a server-side DID filter (URL params up to 150 authors, one options_update frame beyond, every DID validated because Jetstream rejects a malformed frame wholesale), a per-event membership re-check, a per-cycle apply cap that carries its cursor so a burst drains losslessly across cycles, the proxy's 100-row per-author cap (the only layer that helps when a *subscribed* author floods), and a cap-saturation counter plus a `documents_ingest_enabled` kill switch. The wire contract is unchanged — same statuses, same `complete` semantics, same digest algorithm over the same sorted (recordUri, recordCid) pairs, with parity pinned by test — so the frontend does not change and `documents_v2_enabled` chooses the source per request. Reads stay on the proxy until an operator flips it after a clean shadow-compare; flipping it back is the rollback. Not included, and gated on that soak: the linkblog-site cutover, decommissioning the proxy's document path, and the Jetstream v2 move. See docs/plans/DOCUMENTS_TO_D1.md and RUNBOOK §4e. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
Addresses the review of the standard.site → D1 move. - Shadow-compare walks the whole subscribed-author set: the endpoint pages by DID with a `cursor`/`remaining`, and an author the proxy returned no entry for now counts as drift. Comparing the same first 25 authors said nothing about author 26, so a clean verdict could admit a lossy cutover. - The reconcile can no longer be starved by an unlistable author: a failed list holds that author out of the queue for a doubling backoff (1h → 7d) and sorts them by that failure rather than by their still-NULL `last_listed_at`. Without it, three deleted accounts monopolised the only self-heal there is — and the migration's `remaining` could never reach 0. - All four paths that create an `atproto.documents` subscription now pull the author's back catalogue, through a shared `ensureAuthorDocuments` that skips a fresh listing or a backing-off author. Only the API path did, so a subscribe from the Atmosphere button, a PDS sync, or another device served `status:'error'` until the hourly reconcile reached that author. - `complete` is recomputed per serve from the author's stored row count, as the proxy did. The stored flag was set at the last list and never cleared by cap eviction, so an author who published past the cap kept claiming completeness. - Backfill prunes reader-collection sidecars the repo no longer has, but only against a listing that both succeeded and was exhaustive — `listAuthorCollections` now reports which, since a failed fetch and an empty repo looked identical. - `documents_v2.canonical_url` is read as the fallback when a publication won't resolve, instead of serving a bare relative path for the negative cache's TTL. - One `DB.batch` per applied event instead of three `run`s, plus a `MAX_DOCUMENT_APPLY_CAP` (900) on the tunable cap: at three statements an event, a cap of 500 issued ~1500 subrequests against a 1000 ceiling, and that failure is silent — the drain counts an error and advances past it. - RUNBOOK: the paging compare loop, why `remaining` terminates, and the bound on how long ingest can stay paused before a held cursor outruns Jetstream's replay buffer. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
The drain's apply cap was sized in events while the cost that matters is statements: `DB.batch([upsert, trim, bookkeeping])` is three D1 queries, so a 900-event cap asked for ~2700 against a 1000 ceiling and started throwing around the 334th event — caught per event, counted as an error, cursor walked past it. The case the cap exists to survive was the one most likely to drop events. An applied event is now one statement. The per-author cap eviction and the ingest bookkeeping are settled once per author at the end of the cycle, and publication metadata resolved for one event is reused by the rest, so a burst from one author costs about one query apiece. On top of that the drain counts what it spends and stops at DOCUMENT_DRAIN_QUERY_BUDGET while it can still afford the flush its writes have earned, whatever the cap says; MAX_DOCUMENT_APPLY_CAP is that budget, since no cap can buy events the cycle cannot fund. The same ceiling applies to fan-out. The two sync paths schedule their back-catalogue walks into the request's own invocation, so a user with fifteen followed publications turning on Atmospheric sync fired fifteen of them at ~110 queries each; both now warm MAX_SYNC_BACKFILLS authors and leave the rest to the reconcile. The PDS-to-local subscription pull was the fifth path creating `atproto.documents` rows with no backfill at all — the restore-from-the-Atmosphere case, where every restored linkblog polled `status:'error'` until the hourly tick reached it — and now warms them the same bounded way. To make "the reconcile has it" true rather than hopeful, the every-minute cron re-lists one author a tick alongside the hourly three; never-listed authors already sort first. Also: `documents_ingest_enabled` now stops the poller's pending backfills and the cron reconcile, not just the drain. It was advertised as "pause writes", and the poller kept writing up to a hundred rows an author a minute while it was off, because the subscriptions stream keeps enqueuing DIDs. The operator backfill endpoint stays exempt — that one is a deliberate repair, and its batch is resized to fit an invocation. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
`BACKFILL_QUERY_COST = MAX_DOCUMENTS_PER_AUTHOR + 12` was not a worst case. It omitted the per-row prune (up to 100 DELETEs, each its own query), the SELECT per sidecar, and the cross-PDS fetches that come out of the same per-invocation ceiling as the D1 statements — so a prune-heavy author cost ~3x the constant its callers sized against, and five of them on the operator endpoint, or the cron's minute-plus-hourly pair, could cross 1,000 and fail partway through a walk that had already written rows. Three changes make the number true and the fan-outs honest: - Count subrequests, not just D1 statements. Cloudflare documents the 1,000-queries limit under read subrequests, so a cold publication resolve is charged all five (two statements, three fetches) and the drain's lookahead matches. A cycle of cold publications now stops where it says it does. - Shrink what the walk actually spends. The document prune is one `updated_at`-scoped DELETE (every kept row was just upserted, so "untouched by this walk" is the stale set — and a row the drain added mid-listing now survives). The sidecars are one read plus one batch. What is still per-row is capped: `MAX_SITE_RESOLVES_PER_BACKFILL`, `MAX_COLLECTION_WRITES_PER_BACKFILL`, both converging on the next reconcile rather than being dropped. - Budget per invocation, not per loop. A `QueryLedger` is created once per invocation and shared: the cron's two reconcile passes, the operator endpoint's batch, the poller cycle's walks, and both halves of `/api/sync` including the walks they schedule into `waitUntil`. Each fan-out asks `canAffordBackfill` before starting an author and leaves the rest in the reconcile queue — with nothing stamped, so a deferred author keeps its place at the front rather than entering a retry backoff. The endpoint reports `deferred` so a driving loop can see it stopped short. Also from the review: an applied delete now stamps `document_authors.last_event_at` like a write does, so the field means "the last event we applied"; the PDS pull's insert batch (the one unbounded term left) charges the ledger and logs when it is large, which is what keeps walks from being scheduled on top of an invocation that has already spent its budget; and the RUNBOOK/plan/CLAUDE.md now name the subscribe-time walks as deliberately exempt from `documents_ingest_enabled` instead of claiming only the operator endpoint writes. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
The per-author bound was one subrequest short of the walk it describes: both listings resolved the author's DID independently and `resolvePdsUrl` is an uncached plc.directory fetch, so a worst-case walk spent 151 against a promised 150 and the operator endpoint refused the fifth author its ledger was sized for. The two listings now share a per-walk `PdsMemo`, so the constant is unchanged and true, and a paging mock counts what the walk actually spends by kind rather than trusting the charge. The flat sidecar cap converged by the week, not by the next reconcile: a listed author leaves the queue for seven days, so a newly subscribed author's curated back catalogue rendered without its item lists for a month. A walk now spends what is left of its own reservation on sidecars — exactly the old 20 when every other term is at its cap, pinned in the budget test — and records what it still cannot afford as `collections_pending`, which keeps the author in the reconcile queue, sorted behind everything genuinely stale. Two smaller edges in the same accounting: a publication starved during one author's walk is no longer memoised as empty for the rest of a fan-out, and a loose https:// site no longer spends a resolve slot on the short-circuit it takes before any I/O. And `/api/sync`'s reserve now charges the Atmosphere reconcile's ops and graph listing instead of hiding them inside a flat 100 they could account for on their own. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves standard.site document ingest and reads from the Fly proxy to Cloudflare's Jetstream Durable Object and D1 store, behind rollout and ingest gates.
This continuation merges the latest
mainchanges and resolves the overlapping subscription-sync work. Main's structured limit notices and pending-write repair remain intact, while document imports still schedule bounded D1 backfills against the shared invocation query ledger.Radial artifact