feat(node): mark certify events persisted immediately and recover blobs from the pending-recovery table - #3668
Open
halfprice wants to merge 1 commit into
Conversation
…bs from the pending-recovery table A certify event that needs a blob sync no longer holds its event handle inside the sync task, which pinned the persisted event cursor on one slow or stuck recovery and forced a full event replay after a restart. Instead, the event durably records the blob in the pending-recovery table, is marked complete immediately, and a background executor on BlobSyncHandler drains the table: records are synced in event order, bounded by a config knob, and deleted only when the sync succeeds or the blob is retired. A sync that cannot finish occupies one concurrency slot without blocking other records. Retirement events delete the record before cancelling the sync so the executor cannot resurrect data. The persisted cursor now tracks the event tip, restarts resume recovery from the table instead of replaying events, and the pending-record count is exposed through metrics and the health endpoint.
Contributor
|
Warning: This PR modifies one of the OpenAPI files. Please consider the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Second PR of the stack on top of #3667.
A certify event that needs a blob sync no longer hands its event handle into the sync task. Holding the handle pinned the persisted event cursor on one slow or stuck recovery: the node kept processing events but could not persist progress, a restart replayed the whole tail (hours under recent production load), and a stall across two epochs escalated into a full recovery-mode scan.
With this change:
BlobSyncHandlerdrains the table: records are processed in event order, syncs are bounded bymax_concurrent_pending_recoveries, and a record is deleted only when its sync succeeds or the blob is retired. A sync that cannot finish occupies one concurrency slot without blocking other records.InvalidBlobID, andBlobDeletedwhen the blob is no longer certified) delete the record before cancelling the in-flight sync, so the executor cannot resurrect data. Expired blobs are cleaned up lazily by the executor's certification check.As a result, the persisted cursor tracks the event tip, restarts resume recovery from the table instead of replaying events, and the pending-record count is visible in metrics and the health endpoint (
pendingRecoverBlobCount).Test plan
certified_event_persisted_while_blob_recovery_pending: the certify event is persisted while an unrecoverable blob's sync runs; invalidation deletes the record and cancels the sync.blob_deleted_event_deletes_pending_recovery_record.advances_cursor_past_incomplete_blobs_with_pending_record(rewritten fromdoes_not_advance_cursor_past_incomplete_blobs, which asserted the now-inverted invariant); also exercises that a stuck sync does not block later records.test_pending_recovery_survives_node_restart: records survive two crash/restart cycles with the cursor at the tip, then drain once recovery is unblocked.walrus-servicesuite (684 tests),test_lagging_node_recovery, workspace clippy.