feat(replication): self-heal replicas stuck on repl auth error 1045 (opt-in) - #4
Merged
Merged
Conversation
Opt-in operator self-heal for replicas stuck on repl error 1045: when spec.replication.replica.recovery.reconnectOnAuthError is set and a replica reports a sustained 1045 IO error, re-issue CHANGE MASTER with the current secret. Closes the role-gate gap in reconcileReplicationInPod that leaves credential desyncs stuck indefinitely. Independent of the backup-rebuild recovery path. Claude-Session: https://claude.ai/code/session_01BCHp8v87idhtUwTJ5XJ5qu
Task-by-task TDD plan: API field + helper + predicate (unit-tested), reconcile-path hook, CRD/deepcopy/docs regeneration, plus post-merge rollout. Claude-Session: https://claude.ai/code/session_01BCHp8v87idhtUwTJ5XJ5qu
Regenerates config/crd/bases, the mariadb-operator-crds Helm chart CRDs, deploy/crds/crds.yaml, and docs/api_reference.md to include the spec.replication.replica.recovery.reconnectOnAuthError field added in prior commits. zz_generated.deepcopy.go is unchanged: ReplicaRecovery's DeepCopyInto already copies the whole struct by value (*out = *in), so a plain bool field needs no new deepcopy logic.
… note
Whole-branch review found the example YAML would be rejected at apply time:
ReplicaRecovery.Enabled is a required CRD field, so enabling auth-resync needs
`recovery: { enabled: false, reconnectOnAuthError: true }`. Also corrected the
churn risk note (fires per-reconcile, not per-threshold, when the secret is wrong).
Claude-Session: https://claude.ai/code/session_01BCHp8v87idhtUwTJ5XJ5qu
The operator chart's default image tag is the chart appVersion (26.6.0+ubc.1), an invalid OCI reference (Docker tags can't contain '+') that is also not the image built/loaded in CI. `ct install` therefore rendered the operator, webhook, and cert-controller pods with an unparseable image -> InvalidImageName -> cert-controller never provisioned the webhook cert -> webhook FailedMount -> install timed out. Override image/webhook/certController repo+tag with the locally-built IMG_NAME:VERSION (derived from the Makefile) so the install test exercises the just-built image. Pre-existing since the 26.6.0+ubc.1 bump; surfaced now because this is the first chart change to run ct install. Claude-Session: https://claude.ai/code/session_01BCHp8v87idhtUwTJ5XJ5qu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in operator self-heal for replication replicas stuck on repl auth error 1045 (
Access denied for user 'repl').When
spec.replication.replica.recovery.reconnectOnAuthError: trueand a replica's IO thread reports a sustainedLast_IO_Errno == 1045(persisted pastrecovery.errorDurationThreshold, default 5m), the operator re-issuesCHANGE MASTER— viaConfigureReplica(..., WithResetMaster(false))— to re-sync the repl password from the secret. Once the replica reconnects (Last_IO_Errno → 0), it stops firing.Why
Replica configuration is role-gated, not health-gated: in
ReconcileReplicationInPod, a pod already in theReplicarole early-returns without re-issuingCHANGE MASTER. So if the repl password diverges between the master's grant and the replica'smaster.info(e.g. a secret edit / manualALTER USER/ partial reconcile), the replica is stuck on1045forever — the operator never re-applies its credentials. This closes that gap.Field incident: 7
hotcrp-*DBs on our prod cluster wentStatefulSetNotReadyon1045and had to be recovered by hand (STOP SLAVE; CHANGE MASTER TO MASTER_PASSWORD=…; START SLAVE).Design notes
recovery.enabled— nobootstrapFrom, no backup rebuild. (Note:recovery.enabledis a required CRD field, so the enabling manifest must setenabled: falsealongside the flag.)recoverableIOErrorCodes(that backup-rebuild path is for data-divergence errors like 1236).WithResetMaster(false)— a minimal, non-destructive re-CHANGE-MASTER (noRESET MASTER, GTID position preserved), mirroring the existing switchover reconnect path.Changes
api/v1alpha1/mariadb_replication_types.go—ReconnectOnAuthErrorfield onReplicaRecovery+IsReplicaReconnectOnAuthErrorEnabled()helper.pkg/controller/replication/auth_resync.go(new) —replicaAuthResyncQualifiespredicate +podNeedsAuthResyncwrapper +auth_resync_test.go(table-driven unit tests).pkg/controller/replication/controller.go— the reconcile-path hook in the role-gate.api_reference.md.docs/superpowers/{specs,plans}/…— design spec + implementation plan.Test plan
go build ./...,go vet ./...cleango test ./pkg/controller/replication/— predicate/wrapper unit tests + existing suite passmake manifests code helm-crds docs-api manifests-crds→ no further diff)make gen"Check diff") greenRollout (after merge)
ghcr.io/ubc/mariadb-operator:<sha>+ fork charts.helm upgradethe appcloud operator (expect the fleet-wide agent roll).recovery: { enabled: false, reconnectOnAuthError: true }on the 7 hotcrp DBs ink8s-config.https://claude.ai/code/session_01BCHp8v87idhtUwTJ5XJ5qu