[server] Preserve resolved historical write probes - #4178
Draft
luoyuxia wants to merge 8 commits into
Draft
Conversation
Route writes for expired partitions through internal historical targets while preserving original partition metadata across PUT_KV and PRODUCE_LOG. Tier historical KV and log records back to their original Paimon partitions, fail writes to confirmed missing targets, and safely clean fully tiered historical KV overlays with leader-epoch and offset guards. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 1469/1469 AI-Contributed/UT: 2215/2215
Simplify historical write routing, request handling, and Paimon tiering integration while removing redundant tests. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 446/446 AI-Contributed/UT: 576/576
Bind historical write state to the active KV overlay and defer idle cleanup until its deadline after lake progress catches up. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 234/234 AI-Contributed/UT: 129/129
Return a retriable KV storage error while local historical KV state is being initialized or rebuilt. Clarify cleanup-state naming and terminology, and extend the Paimon integration test through post-recovery writes and restarted tiering. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 244/244 AI-Contributed/UT: 239/239
Refresh registered Paimon partition-bucket file sets when the required lake snapshot changes, allowing unchanged local lookup files to remain reusable. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 161/215 AI-Contributed/UT: 95/125
Exclude the plugin-facing LakeTableLookuper interface from the aggregate per-class coverage rule. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 3/3 AI-Contributed/UT: 0/0
Remove the coupled historical KV cleanup mechanism, defer lake lookup refresh I/O until lookup initialization, and simplify historical write version and routing checks. Reroute missing original targets when no request remains in flight, abort only the affected target when handoff is unsafe, and document the remaining retirement ambiguity. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 56/56 AI-Contributed/UT: 2/2
Cache the previous values resolved by the first local probe and lake fallback so compaction between probe and apply cannot change historical write semantics. Switch later same-key records to staged local state only after an actual mutation. Co-Authored-By: OpenAI Codex <codex@openai.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 344/344 AI-Contributed/UT: 129/129
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Related to #4159.
Depends on #4120. This is a stacked PR; until #4120 merges, review commit
4ebf96b06only.Incremental compaction can remove a historical local value or tombstone after the first local probe but before the write is applied. Re-probing local state during apply would then change the resolved previous value and produce incorrect merge, delete, or changelog semantics.
Brief change log
Tests
./mvnw -pl fluss-server -am -DskipITs -Dtest=HistoricalPartitionManagerTest -Dsurefire.failIfNoSpecifiedTests=false test(8 tests passed)./mvnw -pl fluss-server spotless:checkgit diff --checkfluss-serververify was also attempted. Its unrelated failures came from the local filesystem being 95.15% full, which triggered the RocksDB 85% disk guard, and filesystem-allocation-size assertions; the historical tests passed.API and Format
No public API or storage format changes.
Documentation
No user-facing feature is introduced by this correctness prerequisite.
Generative AI disclosure