Skip to content

fix(worker): size replication targets from source block metadata#1293

Open
gongxun0928 wants to merge 2 commits into
CurvineIO:mainfrom
gongxun0928:codex/fix-replication-block-capacity
Open

fix(worker): size replication targets from source block metadata#1293
gongxun0928 wants to merge 2 commits into
CurvineIO:mainfrom
gongxun0928:codex/fix-replication-block-capacity

Conversation

@gongxun0928

Copy link
Copy Markdown
Contributor

Summary

Fix worker block replication when the source file uses a larger per-file block size than the replication worker's client default.

Replication now derives both the bytes to copy and the capacity to reserve from the source BlockMeta, instead of using the unrelated FsContext::block_size(). This also preserves the source allocation extent for SPDK-backed blocks.

Issue Describe / Design

Reproduction configuration

The regression test uses deliberately different block-size settings:

Setting Value Meaning
Worker/client default block size 32KB The FsContext::block_size() available inside WorkerReplicationManager
File block size 64KB The per-file block size selected when the file is created
File size 96KB Produces one 64KB block and one 32KB partial block
Initial replicas 2 Allows the test to verify that replication adds another location

The production equivalent is a file created by a load job or mount with, for example, a 64MB block size while the worker client default remains 4MB.

When the problem occurs

The problem is triggered only when all of the following are true:

  1. A file is created with a per-file block size larger than the replication worker's client default.
  2. At least one finalized source block is larger than that worker default.
  3. The block becomes under-replicated and the master schedules a replication job.
  4. The source worker opens a remote writer on the target worker.

Normal reads are unaffected because readers use the block metadata and read only the finalized block length. The failure occurs specifically while creating the replication target.

Block-size meanings

The values involved have different responsibilities:

  • The file block size is selected when the file is created through CreateFileOpts::block_size. It determines how the file is split into blocks and is recorded as file metadata.
  • FsContext::block_size() is the client default used when a caller does not provide a per-file override. It is not authoritative metadata for an existing source block.
  • BlockMeta::len is the finalized logical byte length of the source block and is the number of bytes replication must copy.
  • BlockMeta::actual_len is the source block's physical allocation size. For SPDK storage, this represents the fixed extent that must be preserved on the target.
  • The block_size argument passed when opening BlockWriterRemote is the target block capacity. It is not the size of an individual RPC payload.
  • Replication still transfers data in replicate_chunk_size chunks, so the target capacity can be larger than the RPC message size.

Root cause

WorkerReplicationManager::replicate_block correctly loaded the source block metadata:

let block_meta = self.block_store.get_block(job.block_id)?;

However, it ignored that metadata when opening the target writer and used:

self.fs_client_context.block_size()

That value belongs to the worker's internal client configuration. It may differ from the block size chosen by the client, load job, or mount that originally created the file.

For example:

  • source block logical length: 64MB
  • worker client default: 4MB
  • old target capacity: 4MB

Replication attempted to copy all 64MB into a target block allocated for 4MB. Once the copied offset exceeded the target capacity, the target write or completion path rejected the block, leaving it under-replicated.

Fix approach

Introduce BlockMeta::replication_capacity() as the single source of truth for target allocation:

  • File-backed blocks reserve BlockMeta::len. Regular files can grow, so the committed logical length is sufficient.
  • SPDK-backed blocks reserve max(BlockMeta::len, BlockMeta::actual_len). SPDK blocks use fixed extents, so replication must preserve the physical allocation even when the finalized logical block is partial.

The replication manager now:

  1. Uses block_meta.len as the number of bytes to copy.
  2. Uses block_meta.replication_capacity() as the target writer capacity.
  3. Logs both values to make allocation mismatches observable.
  4. Continues transferring data in bounded replication chunks.

This is intentionally scoped to the worker replication write path. It does not require protocol changes, additional master lookups, or reconstruction of the source file's FileStatus.

Changes

Module / File Change Impact on existing behavior
curvine-server/src/worker/block/block_meta.rs Add replication_capacity() with file-backed and SPDK-specific allocation semantics Centralizes the target-capacity decision
curvine-server/src/worker/replication/worker_replication_manager.rs Allocate the target from source block metadata instead of FsContext::block_size() Replication succeeds when per-file and worker default block sizes differ
curvine-tests/tests/replication_test.rs Add a 32KB worker / 64KB file regression test with replica polling and integrity verification Prevents regression without fixed sleeps

Test verified

Test case Result Notes
cargo test -p curvine-tests --test replication_test test_replication_honors_source_block_capacity PASS Confirmed a 64KB source block replicated while the worker default remained 32KB
cargo test -p curvine-tests --test replication_test PASS 3 passed, 0 failed
Curvine example-cluster SPDK replication smoke test PASS Verified SPDK target allocation and replicated data
cargo fmt --all PASS
git diff --check PASS
make format BLOCKED Existing unrelated unused imports in curvine-fuse/src/fs/state/node_state.rs fail Clippy with -D warnings

Derive replication copy length and target capacity from the finalized source BlockMeta. Preserve fixed SPDK extents and add a regression test for per-file block sizes larger than the worker client default.
@gongxun0928
gongxun0928 marked this pull request as ready for review July 24, 2026 08:04
Comment thread curvine-tests/tests/replication_test.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants