fix(worker): size replication targets from source block metadata#1293
Open
gongxun0928 wants to merge 2 commits into
Open
fix(worker): size replication targets from source block metadata#1293gongxun0928 wants to merge 2 commits into
gongxun0928 wants to merge 2 commits into
Conversation
Derive replication copy length and target capacity from the finalized source BlockMeta. Preserve fixed SPDK extents and add a regression test for per-file block sizes larger than the worker client default.
gongxun0928
marked this pull request as ready for review
July 24, 2026 08:04
szbr9486
reviewed
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix worker block replication when the source file uses a larger per-file block size than the replication worker's client default.
Replication now derives both the bytes to copy and the capacity to reserve from the source
BlockMeta, instead of using the unrelatedFsContext::block_size(). This also preserves the source allocation extent for SPDK-backed blocks.Issue Describe / Design
Reproduction configuration
The regression test uses deliberately different block-size settings:
FsContext::block_size()available insideWorkerReplicationManagerThe production equivalent is a file created by a load job or mount with, for example, a 64MB block size while the worker client default remains 4MB.
When the problem occurs
The problem is triggered only when all of the following are true:
Normal reads are unaffected because readers use the block metadata and read only the finalized block length. The failure occurs specifically while creating the replication target.
Block-size meanings
The values involved have different responsibilities:
CreateFileOpts::block_size. It determines how the file is split into blocks and is recorded as file metadata.FsContext::block_size()is the client default used when a caller does not provide a per-file override. It is not authoritative metadata for an existing source block.BlockMeta::lenis the finalized logical byte length of the source block and is the number of bytes replication must copy.BlockMeta::actual_lenis the source block's physical allocation size. For SPDK storage, this represents the fixed extent that must be preserved on the target.block_sizeargument passed when openingBlockWriterRemoteis the target block capacity. It is not the size of an individual RPC payload.replicate_chunk_sizechunks, so the target capacity can be larger than the RPC message size.Root cause
WorkerReplicationManager::replicate_blockcorrectly loaded the source block metadata:However, it ignored that metadata when opening the target writer and used:
That value belongs to the worker's internal client configuration. It may differ from the block size chosen by the client, load job, or mount that originally created the file.
For example:
Replication attempted to copy all 64MB into a target block allocated for 4MB. Once the copied offset exceeded the target capacity, the target write or completion path rejected the block, leaving it under-replicated.
Fix approach
Introduce
BlockMeta::replication_capacity()as the single source of truth for target allocation:BlockMeta::len. Regular files can grow, so the committed logical length is sufficient.max(BlockMeta::len, BlockMeta::actual_len). SPDK blocks use fixed extents, so replication must preserve the physical allocation even when the finalized logical block is partial.The replication manager now:
block_meta.lenas the number of bytes to copy.block_meta.replication_capacity()as the target writer capacity.This is intentionally scoped to the worker replication write path. It does not require protocol changes, additional master lookups, or reconstruction of the source file's
FileStatus.Changes
curvine-server/src/worker/block/block_meta.rsreplication_capacity()with file-backed and SPDK-specific allocation semanticscurvine-server/src/worker/replication/worker_replication_manager.rsFsContext::block_size()curvine-tests/tests/replication_test.rsTest verified
cargo test -p curvine-tests --test replication_test test_replication_honors_source_block_capacitycargo test -p curvine-tests --test replication_testcargo fmt --allgit diff --checkmake formatcurvine-fuse/src/fs/state/node_state.rsfail Clippy with-D warnings