RATIS-2605. Avoid advancing matchIndex from heartbeat AppendEntries success#1519
Merged
Conversation
A follower returns its local nextIndex for successful heartbeat AppendEntries replies. This value describes the follower's local log tail, but it does not prove that all entries up to nextIndex - 1 match the leader's log. If a follower has a divergent uncommitted tail from an old leader, trusting the heartbeat reply nextIndex may incorrectly advance the leader-side matchIndex. Later INCONSISTENCY handling uses matchIndex + 1 as a lower bound, so the polluted matchIndex can prevent nextIndex from backing off to the real common prefix. Track the AppendEntries request metadata together with the reply. For successful heartbeat replies, advance only to request.previousLog.index + 1. For successful append replies, continue using reply.nextIndex since appended entries prove the returned match point. Add unit tests for heartbeat success with and without previousLog, and verify that append success still uses reply.nextIndex.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch fixes leader-side AppendEntries success handling for heartbeat requests.
A follower returns its local
nextIndexfor successful heartbeat AppendEntries replies. That value describes the follower's local log tail, but it does not prove that all entriesup to
nextIndex - 1match the leader's log. If the follower has a divergent uncommitted tail from an old leader, trusting the heartbeat replynextIndexmay incorrectly advancethe leader-side
matchIndex.Once
matchIndexis polluted, laterINCONSISTENCYhandling can usematchIndex + 1as a lower bound and preventnextIndexfrom backing off to the real common prefix.This patch preserves the AppendEntries request metadata together with the reply. For successful heartbeat replies, the leader advances only to
request.previousLog.index + 1. Forsuccessful append replies, the existing behavior is preserved and
reply.nextIndexis still used.Unit tests are added for:
SUCCESSwithpreviousLogadvances only topreviousLog.index + 1.SUCCESSwithoutpreviousLogdoes not advancematchIndex.SUCCESSstill advances usingreply.nextIndex.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-2605
How was this patch tested?
./mvnw -pl ratis-server -am -Dtest=TestLogAppenderDefault test