gazctl shards prune: retry transient fragment removal errors - #493
Merged
Conversation
Object deletes fail intermittently with errors which resolve on their own: 5xx responses, rate limits, and dropped connections. The GCS client does not retry a delete having no preconditions, so a single such error counts as a failed removal and fails the whole prune with exit 1, alerting on cleanup which would have succeeded on a second attempt. Retry a failed removal up to three times, a second apart. Authorization failures are returned as before, without retry, and remain counted as a non-fatal permissionDenied.
dgreer-dev
approved these changes
Aug 24, 2026
dgreer-dev
left a comment
dgreer-dev
left a comment
There was a problem hiding this comment.
LGTM. Is a second too short for the retry?
Contributor
Author
I'm assuming these are very transient errors where a retry would immediately resolve. It's possible that 1 second would be too short though if it's something other than that. If we see the failures continue I'll tune it; hopefully it isn't an issue though. In the somewhat near future I'm planning on making larger changes to our recovery log / pruning architecture in Flow so just trying to get by until then. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Object deletes fail intermittently with errors which resolve on their own: 5xx responses, rate limits, and dropped connections. The GCS client does not retry a delete having no preconditions, so a single such error counts as a failed removal and fails the whole prune with exit 1, alerting on cleanup which would have succeeded on a second attempt.
Retry a failed removal up to three times, a second apart. Authorization failures are returned as before, without retry, and remain counted as a non-fatal permissionDenied.
There's a known gap with this implementation where on GCS a deletion might actually succeed but the success response gets lost coming back to us, and then we retry and get a 404 and fail. I haven't seen that actually happen yet conclusively, so I'll worry about that later.