[CORE-16844] kafka/client: recover from offset_out_of_range in consumer group fetch#31064
Open
bartoszpiekny-redpanda wants to merge 2 commits into
Conversation
bartoszpiekny-redpanda
force-pushed
the
CORE-16844-fix-offset_out_of_range-after-retention
branch
2 times, most recently
from
9ca82a8 to
2293be2
Compare
bartoszpiekny-redpanda
force-pushed
the
CORE-16844-fix-offset_out_of_range-after-retention
branch
2 times, most recently
from
940d6dd to
eb5dd14
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes Pandaproxy consumer-group fetch getting stuck returning offset_out_of_range after log retention/prefix-truncation advances a partition’s log start offset beyond the consumer’s initial fetch offset.
Changes:
- Update
fetch_session::apply()to seed the tracked fetch offset fromlog_start_offsetwhen a partition returnsoffset_out_of_range. - Trigger a retry from the Kafka client consumer fetch path so Pandaproxy re-fetches using the corrected offset.
- Add unit and ducktape regression tests covering prefix-trim recovery.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/rptest/tests/pandaproxy_test.py | Adds a ducktape regression test that trims a topic prefix and verifies a fresh consumer-group fetch still succeeds. |
| src/v/kafka/client/test/fetch_session.cc | Adds a unit test ensuring fetch_session::apply() updates tracked offsets on offset_out_of_range. |
| src/v/kafka/client/fetch_session.cc | Implements offset correction on offset_out_of_range using log_start_offset. |
| src/v/kafka/client/consumer.cc | Throws on offset_out_of_range after applying the response to force a retry before returning to Pandaproxy. |
Collaborator
Retry command for Build#86958please wait until all jobs are finished before running the slash command |
Collaborator
CI test resultstest results on build#86958test results on build#87059
test results on build#87065
|
bartoszpiekny-redpanda
force-pushed
the
CORE-16844-fix-offset_out_of_range-after-retention
branch
2 times, most recently
from
3a87023 to
6d5f4d6
Compare
Pandaproxy's consumer group fetch started every fresh assignment at
offset 0 and never advanced once retention moved the log start offset
past it, so every fetch returned offset_out_of_range forever despite
auto.offset.reset=earliest -- the only reset policy pandaproxy accepts.
fetch_session now exposes three named operations instead of one
overloaded apply(): apply() advances offsets from a delivered response,
discard() advances only the session epoch, and reseed() sets a
partition's offset directly. consumer::fetch() collects every broker's
response, then:
- reseeds out-of-range partitions to the broker-reported
log_start_offset. earliest is exactly the log start, and the broker
already returns it in the fetch response, so no separate ListOffsets
is needed.
- delivers the healthy partitions immediately and strips the
out-of-range ones from the response (the pandaproxy serializer
rejects any partition error); those partitions resume from the
corrected offset on the client's next poll. No offset advances past
undelivered records, so nothing is silently skipped.
- only on a dispatch failure -- a whole broker's response missing,
where the topology may be stale -- discards the round and throws, so
the retry in client::consumer_fetch() re-fetches and refreshes
metadata.
Recovering per partition rather than discarding and re-fetching the
whole round avoids re-reading the healthy partitions on every
retention/trim edge. Adds fetch_session unit tests for the split API.
Trim a topic's log prefix past offset 0 and assert a fresh consumer group polls its way to the records at the new log start offset, producing one record per call so each lands in its own batch and the trim offset falls on a batch boundary, as real retention does. A second test trims only one of two partitions and asserts the healthy sibling's records are delivered while the out-of-range partition recovers, covering the per-partition recovery path.
bartoszpiekny-redpanda
force-pushed
the
CORE-16844-fix-offset_out_of_range-after-retention
branch
from
6d5f4d6 to
a037491
Compare
Contributor
Author
|
Force-pushed: squashed the per-partition recovery into the main fix commit so the PR shows the final design directly (no intermediate coarse-recovery version). Behaviour and net diff are unchanged from the previous push — only the commit layout. Diff since the previous push (6d5f4d6 → a037491): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pandaproxy's consumer group fetch always started a fresh partition assignment at offset 0 and never advanced once retention moved the log start offset past it, regardless of auto.offset.reset=earliest requested at consumer creation. Every fetch after that point returned offset_out_of_range forever, because the client tracked its own fetch offset and had no path to correct it — and the REST client cannot control the fetch offset itself.
fetch_session is split into three named operations: apply() advances the tracked offset from a delivered response, discard() advances only the session epoch, and reseed() sets an offset directly. On offset_out_of_range, consumer::fetch() reseeds the affected partitions to the log_start_offset the broker reports alongside the error — earliest is exactly the log start, and the broker already returns it in the fetch response, so no separate ListOffsets is needed since pandaproxy only ever allows the earliest reset policy. The healthy partitions are delivered immediately and the out-of-range ones are stripped from the response (the serializer rejects any partition error), so they resume from the corrected offset on the client's next poll — no retry, and no re-reading of the partitions that already returned records. Only a dispatch failure, where a whole broker's response is missing and the topology may be stale, discards the round and throws so the existing gated_retry_with_mitigation in client::consumer_fetch() re-fetches and refreshes metadata.
Adds fetch_session unit tests covering the split apply/discard/reseed API, and pandaproxy ducktape regression tests: one trims a topic's log prefix and asserts a fresh consumer group polls its way to the records at the new log start offset, and one trims a single partition of a two-partition topic and asserts the healthy sibling's records are delivered while the out-of-range partition recovers.
Design notes
The HTTP Proxy accepts a single reset policy. The consumer-create handler rejects anything else with a 400 before a consumer is built:
if (req_data.auto_offset_reset != "earliest") {
throw parse::error(
parse::error_code::invalid_param, "auto.offset must be earliest");
}
src/v/pandaproxy/rest/handlers.cc (create_consumer).
Because earliest is the only reachable policy, the value is validated and logged but never forwarded to the internal kafka::client — the client doesn't need to know it. This is what keeps recovery cheap: earliest resolves to the partition's log_start_offset, which the broker already returns in every fetch response, so the client reseeds straight from that field. No separate ListOffsets is issued to resolve the reset policy. (If the proxy ever allowed latest/timestamp, that assumption breaks and the value would have to be threaded through and resolved via ListOffsets.)
A fetch that hits offset_out_of_range on some partitions is not retried. consumer::fetch():
This matches how franz-go behaves for a polling consumer: it also delivers the partitions that succeeded and recovers the out-of-range one out-of-band (franz-go reloads just that partition via a background ListOffsets). The observable contract is the same — healthy records now, the recovered partition's records on a later poll, no error surfaced to the client, and no data re-read. The only difference is the recovery mechanism: franz-go issues a ListOffsets, we reseed from the field already present in the fetch response — valid precisely because of assumption (1).
The one case that is retried is a dispatch failure — a whole broker's response missing, where the topology may be stale. There we discard the round and throw so the retry in client::consumer_fetch() runs the update_metadata refresh a stale leader needs.
Fixes: CORE-16844
Backports Required
Release Notes
Bug Fixes