Skip to content

fix(server): prevent partial parallel loads from becoming valid cache#1281

Merged
szbr9486 merged 2 commits into
CurvineIO:mainfrom
gongxun0928:codex/fix-parallel-load-cache-validity
Jul 25, 2026
Merged

fix(server): prevent partial parallel loads from becoming valid cache#1281
szbr9486 merged 2 commits into
CurvineIO:mainfrom
gongxun0928:codex/fix-parallel-load-cache-validity

Conversation

@gongxun0928

@gongxun0928 gongxun0928 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Prevent a cancelled or incomplete parallel UFS-to-Curvine load from stamping ufs_mtime on a pre-resized target and becoming a false cache hit.

Issue Describe / Design

Root cause

run_parallel pre-creates the target and calls resize(src_len) before copying. This is required for disjoint multi-writer ranges, but it also means the target metadata reports the full source length before every range contains data.

Each join-loop iteration checks task.is_cancel() before awaiting a stream handle. A race remains for the final handle:

  1. The parent checks cancellation before awaiting the final stream and sees false.
  2. The task is cancelled while the parent is awaiting that stream.
  3. The stream observes cancellation, cancels its writer, and returns Ok(0) as a soft exit.
  4. The join loop accepts Ok(0) and ends; there is no next iteration to recheck cancellation.
  5. The runner stamps the source ufs_mtime on the target even though the final range is incomplete.
  6. FileStatus::cv_valid can then accept the target because its pre-resized length and stamped mtime match the UFS source.

The result is a false cache hit: later load or read paths may skip reloading an incomplete file.

Trigger conditions

The race requires all of the following:

  • a UFS-to-Curvine load using parallel fan-out (effective_streams > 1);
  • cancellation while the parent is awaiting the final stream;
  • the final stream observing cancellation before completing its assigned range.

Cancellation can come from a user cancel request, a duplicate task superseding the current runner, or master cleanup after a partial dispatch failure.

Fix approach

  • Recheck task.is_cancel() immediately after all stream handles join and before stamping cache validity. Cancellation remains a soft Ok(()) exit, preserving existing task-state semantics.
  • Require written == src_len for the non-cancelled path. Any mismatch is reported as a data-integrity error and refuses to stamp ufs_mtime.
  • Include the task ID in the incomplete-load error for consistency with sibling errors.
  • Use fault-injection points to deterministically reproduce cancellation after the final pre-await check.
  • Stamp ufs_mtime only after both integrity gates pass.

Changes

Module / File Change Impact on existing behavior
curvine-server/src/worker/task/load_task_runner.rs Add final cancellation and copied-length gates, task context in the error, and test-only fault points Cancelled or incomplete parallel loads can no longer become valid cache entries
curvine-server/tests/load_task_runner_fault_test.rs Add a deterministic MiniCluster regression test for final-stream cancellation Regression coverage only
curvine-tests/regression/daily_regression_test.sh Enable curvine-server/fault-injection for Nextest discovery and specific server tests Daily regression now executes feature-gated fault tests

Test verified

Test case Result Notes
cargo fmt --all --check PASS
cargo clippy -p curvine-server --all-targets --features fault-injection -- -D warnings --allow clippy::uninlined-format-args PASS
cargo test -p curvine-server --lib worker::task::load_task_runner PASS 8 passed
cargo nextest run --workspace --profile ci-no-ufs --features curvine-server/fault-injection -E 'test(canceled_final_parallel_stream_does_not_mark_cache_valid)' PASS 1 passed; 15.217s
daily_regression_test.sh ... curvine-server load_task_runner_fault_test canceled_final_parallel_stream_does_not_mark_cache_valid PASS Structured result reports 1/1 passed
Negative control with both final integrity gates temporarily removed EXPECTED FAIL Test observed the invalid cache-validity stamp
bash -n curvine-tests/regression/daily_regression_test.sh PASS
make format BASELINE FAILURE Existing unused imports in curvine-fuse/src/fs/state/node_state.rs

Dependencies

  • Related PRs: None
  • Related issues: None
  • Blocks / blocked by: None

Comment thread curvine-server/src/worker/task/load_task_runner.rs
Comment thread curvine-server/src/worker/task/load_task_runner.rs Outdated
@gongxun0928
gongxun0928 requested a review from szbr9486 July 25, 2026 06:52
@szbr9486
szbr9486 merged commit 03e18a9 into CurvineIO:main Jul 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants