[v26.1.x] [CORE-13340] kafka: fix OffsetForLeaderEpoch for term outside cloud coverage#31074
Open
vbotbuildovich wants to merge 3 commits into
Open
Conversation
A getter for the highest term present in cloud storage (whether that be in the most recent segment, or spillover). (cherry picked from commit 26ba0f2)
`get_leader_epoch_last_offset_unbounded()` consults cloud storage when the requested term falls below the first local term. `get_cloud_term_last_offset()` returns `std::nullopt` both when: 1. The term is older than the oldest term in cloud storage 2. The term is higher than the highest term in cloud storage The fallback path assumes only case 1. is possible and always returns the cloud start offset. For case 2, when a term's data lives only in the local log but the local start offset has advanced ahead of the cloud upload watermark (e.g. a raft snapshot taken during partition movement), the term is newer than everything in cloud storage. Returning the cloud start offset then reports an offset below the consumer's position, which the consumer interprets as log truncation and resets to the beginning of the log, re-consuming all data. For an example of this case: 1. A partition whose cloud manifest holds only term 1 2. The local start offset advances to term 3 3. Term 2 is present in neither cloud storage nor the local log. Presently, querying the last offset for the term 2 leader would return the cloud start offset, when it should return the first local offset (the start of the next highest term, i.e. term 3). Fall back to the first local offset (the start of the next-highest term) in that case, mirroring the non-cloud branch. Read replicas keep returning the cloud start offset since they have no local log to fall back on. (cherry picked from commit f620e15)
…offset` `get_leader_epoch_last_offset_unbounded()` branched on `is_read_replica` five times. A read replica serves all reads from cloud storage and never consults the local log, so handle it in a single early-return block at the top and drop the `is_read_replica` qualifiers from the rest of the function. No behavior change. (cherry picked from commit 619d0fc)
Collaborator
Author
CI test resultstest results on build#86986
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #31060