drbd 9.2.19-flant.11: fix multi-source resync deadlock (cancel-on-pause)#13
Open
astef wants to merge 1 commit into
Open
drbd 9.2.19-flant.11: fix multi-source resync deadlock (cancel-on-pause)#13astef wants to merge 1 commit into
astef wants to merge 1 commit into
Conversation
When a device is resynced from two peers at once, drbd_select_sync_target() keeps one active L_SYNC_TARGET and forces the other to L_PAUSED_SYNC_T. The paused source goes resync-suspended:peer and never delivers replies, so a resync request already sent to it stays "ready-to-send" forever, and via the conflict rule in drbd_should_defer_to_interval() it permanently blocks the ACTIVE source's resync write for the same block. drbd_release_conflicts() only re-drives a parked write when a conflicting interval completes, which never happens -> resync freezes at done:X% (~5% of replicas after a reboot). Fix: drbd_cancel_paused_resync_requests(), called from w_after_state_change() when a peer enters L_PAUSED_SYNC_T, drops that peer's dangling (sent, not yet received) resync requests and releases the writes parked behind them. The blocks stay out-of-sync in the bitmap and are resynced by the active source, or by this peer once it resumes.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the stuck-resync-after-reboot bug: ~5% of a rebooted node's replicas froze mid-resync at
done:X%and never completed.Root cause (multi-source resync conflict deadlock): when a device is resynced from two peers at once,
drbd_select_sync_target()keeps only one activeL_SYNC_TARGETand forces the other toL_PAUSED_SYNC_T. The paused source goesresync-suspended:peerand never delivers replies, so a resync request already sent to it staysREADY_TO_SENDforever. Via the conflict rule indrbd_should_defer_to_interval()it permanently blocks the active source's resync write for the same block, anddrbd_release_conflicts()only re-drives a parked write when a conflicting interval completes — which never happens. The resync deadlocks atdone:X%.This is stock LINBIT interval-tree resync machinery (present in 9.2.x and 9.3), not a flant regression.
Fix
New
drbd_cancel_paused_resync_requests()(drbd/drbd_req.c), called fromw_after_state_change()(drbd/drbd_state.c) when a peer entersL_PAUSED_SYNC_T. Underinterval_lockit drops that peer's dangling (sent, not-yet-received) resync requests and callsdrbd_release_conflicts()first so the active source's parked write is re-driven. Affected blocks stay out-of-sync in the bitmap and are resynced by the active source, or by the peer once it resumes.drbd/drbd_req.c—drbd_cancel_paused_resync_requests()(+68)drbd/drbd_state.c— call onL_PAUSED_SYNC_Tentry (+8)drbd/drbd_int.h— declaration (+1)9.2.19-flant.11(config/spec/changelogs/dockerfiles)Test plan
reboot -fof a node with 76 UpToDate replicas → all 76 resynced with 0done:X%freezes (baseline flant.10 was ~4–5%).SyncTarget+PausedSyncT(resync-suspended:dependency)shape progressed toUpToDateinstead of freezing.