Copy passthrough during snapshot mode if no transformers are configured - #1129
Open
kvch wants to merge 3 commits into
Open
Copy passthrough during snapshot mode if no transformers are configured#1129kvch wants to merge 3 commits into
kvch wants to merge 3 commits into
Conversation
CoverageTotal: 60.3% (+0.1% vs Coverage in packages changed by this PR:
|
tsg
approved these changes
Sep 3, 2026
| if cfg.RetryPolicy.DisableRetries { | ||
| targetConn, err = pglib.NewConnPool(ctx, cfg.TargetURL, poolOpts...) | ||
| } else { | ||
| targetConn, err = pglibretrier.NewQuerier(ctx, cfg.RetryPolicy, func(ctx context.Context) (pglib.Querier, error) { |
Member
There was a problem hiding this comment.
Do we need retries in this case? From my agent:
The target pool is wrapped in the retrier, which re-runs the ExecInTx closure on a retriable error. The pipe reader is already partially consumed by then, so the retry sends a truncated COPY and fails with the row count mismatch, then retries again until the backoff gives up. Nothing corrupt lands because the target tx rolls back, but the retry loop burns time and logs misleading warnings. The passthrough should probably disable retries on the target conn and let the existing page range failure abort the table, or retry the whole range from the source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for snapshotting a table by piping the source's
COPY ... TO STDOUTstraight into the target'sCOPY ... FROM STDINwhen the rows do not need to be parsed. When a transformer, filter, injector or sanitizer is configured, we fall back to the previous behaviour.Previously, we read read rows, decoded every value into Go values via pgx, wrapped them in
wal.Eventstructs, passed them through the processor chain to the bulk ingest writer, which re-encoded them into aCOPY. Both ends were alreadyCOPY; only the round trip through Go in the middle is removed. I measured improvement on ~17% on a text heavy table. It is a workload-specific improvement, since the saving is largest where decoding allocates most. So it can be higher for some cases.Furthermore, #801 (cube) and #1035 (enum) exist because pgx cannot produce those types' binary encodings, which is why
textOnlyCopyTypesandneedsTextCopyexist. Server-to-server COPY never touches a pgx codec, so that family of bugs cannot occur on this path.Builds on the raw COPY stream primitives from #1126.
Please review commit by commit.
feat: stream snapshot rows without decoding them: the new logicrefactor: make the page range reader a strategy:pg_snapshot_range_snapshotter.golooks like a new file but isprocessTableRangerelocated: three receiver renames and one inlined variableType of Change
Changes Made
Commit 1: the passthrough
io.Pipe, so memory is bounded by the pipe rather than the page range, and each side closes its end with its error so neither can block on the otherCommit 2: the strategy extraction
newSnapshotGenerator, which takes the reader as an argument, so no path can leave it unset.Testing
Checklist