fix: auto-recover from CHANGE MASTER errno 1201 via RESET SLAVE ALL retry - #2
Merged
Merged
Conversation
…etry
Field incident on moodle-education-stg-db (2026-05-14): a previous failed
switchover left pod-1 (the old primary) with a 0-byte master.info file. On
every subsequent reconcile the switchover state machine reached phase 6
(changePrimaryToReplica), issued CHANGE MASTER TO with the default channel
name to demote pod-1 to a replica of pod-0, and got rejected with:
Error 1201 (HY000): Could not initialize master info structure for '';
more error messages can be found in the MariaDB error log
The error path returned without recovery, the reconcile failed, and the
loop repeated indefinitely. Manual recovery required executing
`STOP SLAVE; RESET SLAVE ALL;` on pod-1 between reconciles to clear the
residual slave-channel state, after which the operator's next CHANGE
MASTER TO succeeded and the switchover completed in seconds.
The condition is structural, not transient: any switchover that completes
phase 4 (which runs RESET MASTER / RESET SLAVE ALL on the new primary)
but fails partway through phase 5 or 6 can leave the old primary's data
dir with truncated master.info / relay-log.info. On the next retry,
CHANGE MASTER cannot rebind the in-memory master_info structure from the
inconsistent on-disk state.
Recovery is mechanical and safe: RESET SLAVE ALL clears precisely the
slave-channel state that CHANGE MASTER is about to overwrite anyway, so
running it as a one-shot retry before the second CHANGE MASTER costs
nothing in the healthy case and unblocks the wedge in the broken case.
Add an error-code classifier in pkg/sql (sql.IsMySQLErrorCode, plus a
named constant sql.ErrCodeMasterInfo = 1201) and wrap the CHANGE MASTER
call in pkg/controller/replication/config.go:changeMaster so that:
- on err == nil, return immediately (no behaviour change)
- on err matching errno 1201, call ResetAllSlaves once and retry
CHANGE MASTER, returning the retry result
- on any other err, return the wrapped error unchanged
The classifier uses errors.As so wrapped errors anywhere in the chain are
detected, matching the standard library idiom for typed errors.
Tests:
- pkg/sql/sql_test.go: TestIsMySQLErrorCode covers nil, non-driver,
wrapped/unwrapped driver errors with matching and non-matching codes,
and the %v-vs-%w wrapping distinction (eight table-driven cases).
- pkg/controller/replication/switchover_test.go: documented design record
for the end-to-end recovery path, following this file's existing
convention for paths that need a live SQL connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a previous failed switchover leaves a pod with truncated
master.info/relay-log.infofiles (a 0-byte file is the field-observed signature), thenext
CHANGE MASTER TOagainst the default replication channel errors with:The operator's switchover state machine loops on this error every reconcile,
forever, because nothing in the existing path tries to repair the residual
slave-channel state.
Add a one-shot recovery: detect
errno 1201fromCHANGE MASTER, callRESET SLAVE ALLto clear the corrupt on-disk state, and retry once. Thisis safe because the residual state is exactly what the in-flight
ConfigureReplicais about to overwrite — clearing it removes thecorruption barrier without losing any meaningful information.
Field incident
moodle-education-stg-db, 2026-05-14: a previous switchover left pod-1
with
master.infotruncated to 0 bytes. The operator looped onchangePrimaryToReplica → CHANGE MASTER TO → errno 1201every ~10 seconds.Cluster was wedged with primary on the wrong pod until manual
STOP SLAVE; RESET SLAVE ALL;cleared the state on pod-1, after which thenext operator-issued
CHANGE MASTER TOsucceeded and the switchovercompleted in seconds.
What this PR does
pkg/sql/sql.goErrCodeMasterInfo = 1201and helperIsMySQLErrorCode(err, code)usingerrors.Asfor typed-error detection through wrapping.pkg/controller/replication/config.goCHANGE MASTERinchangeMaster(): onerrno 1201, callResetAllSlavesand retry once. All other errors propagate unchanged.Failure modes the new path handles
master.infofrom prior wedgeerrno 1201returned foreverTest plan
pkg/sql/sql_test.go:TestIsMySQLErrorCode— 8 table-driven cases covering nil, non-driver, wrapped (%w / %v), single vs nested wrapping, matching/non-matching codes. All pass with-race.pkg/controller/replication/switchover_test.go: documented design record for the end-to-end recovery path, matching the file's existing integration-test-only convention for live-SQL paths.go test ./... -race,go vet ./...,gofmtall clean.go build ./...clean.Non-goals
This does not change the upstream cause of the
master.infotruncation(a switchover failing between phases 4-6). That root-cause fix belongs to
a different PR; this PR only ensures the cluster can self-heal on the
next reconcile instead of wedging indefinitely.
It also does not blanket-enable RESET SLAVE ALL on every CHANGE MASTER —
only the narrowly-targeted
errno 1201path triggers it, so a CHANGEMASTER that fails for unrelated reasons (auth, network, syntax) still
surfaces the original error cleanly.