fix(postgres): restore must not report ACTIVE before the data copy co… - #246
Merged
Conversation
…mpletes DescribeTable's contract is that the first ACTIVE observation on a restore target implies the restored data is fully present. The Postgres restore path created the target through the normal create-table flow, which schedules a CREATING->ACTIVE transition on the creation-delay timer. The row-by-row copy of a large backup outlasts that timer, so a client that waits-for-ACTIVE can observe an empty or partial table (reproduced live: ACTIVE at 515/40000 items on baseline). Fix: create_table_impl gains a defer_active flag. The restore path passes true, writing the row CREATING with no scheduled transition; the existing explicit ACTIVE update at the end of the copy is now the only flip. The public TableEngine::create_table path passes false and is unchanged. Verification (live, Postgres 16 at 127.0.0.1:5433): - New test restore_active_completeness: observer spawns before restore, polls DescribeTable, counts items via paginated Select=COUNT scan at first ACTIVE. Transient describe errors are retried, and scan errors fail the test rather than truncating the count (avoids the oracle bug that makes the equivalent upstream test flaky). - Negative control on baseline: FAILED with ACTIVE at 515/40000 (112s). - With fix: single test PASS (129s); full integration suite --test-threads=1: 418/421 passed, 0 filtered. The 3 reds are capacity_throttling tests requiring 'settings set throttling_enabled true' which the ad-hoc harness omitted; CI sets it (integration.yml).
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn and
yesyayen
as code owners
August 7, 2026 15:10
3 tasks
amrith
approved these changes
Aug 10, 2026
Collaborator
|
approved and merging when ready |
diegotoledano95
added a commit
to diegotoledano95/extenddb
that referenced
this pull request
Aug 10, 2026
Pull in ExtendDB#239 (GSI pagination tiebreaker), ExtendDB#245 (SQLite stale GSI delay), and ExtendDB#246 (Postgres restore race) — the three pre-existing backend bugs the dual-target suite surfaced. Resolve the add/add conflict on restore_active_completeness.rs in favor of this branch version, which carries the reviewer-requested retry-on-scan-error oracle fix and a bounded observer; main branch Postgres restore fix lands via its own source change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR body: fix/restore-active-implies-copy-complete
What
Fixes a race in the Postgres restore path where a restored table could report
ACTIVEinDescribeTablebefore its data copy finished.create_table_implgains adefer_activeflag: the restore path passestrue, writing the table rowCREATINGwith no scheduled transition, so the explicitACTIVEupdate at the end of the row copy is now the only status flip. The publicTableEngine::create_tablepath passesfalseand is unchanged.Why
DynamoDB's contract is that the first
ACTIVEobservation on a restore target implies the restored data is fully present. Our restore created the target through the normal create-table flow, which schedules theCREATINGtoACTIVEtransition on the creation-delay timer. Copying a large backup outlasts that timer, so a client that waits-for-ACTIVE can observe an empty or partial table. Reproduced live on baseline:ACTIVEat 515/40000 items. This is the same defect class the MongoDB backend in #207 fixed on its side, and it is what makes that PR's dual-target restore test red on the Postgres runner.Closes # (no standing issue; surfaced by the restore-completeness test in #207's CI)
Testing done
Live against Postgres 16, TLS, SigV4 auth:
restore_active_completeness::restored_table_has_all_items_when_first_active: an observer task spawns before the restore, pollsDescribeTable, and counts items via a paginatedSelect=COUNTscan at the firstACTIVEobservation. Transient describe errors are retried; scan errors fail the test rather than truncating the count.ACTIVEat 515/40000 items (112s), confirming the test discriminates.--test-threads=1is 418/421 passed, 0 filtered out. The 3 failures arecapacity_throttlingtests that requiresettings set throttling_enabled true, which my ad-hoc harness omitted; CI sets it inintegration.yml.cargo fmt --checkexit 0,cargo clippyclean on the touched crate.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a (internal bug fix;
create_table_implispub(crate), no contract surface changed)By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.