PMM-15227 Remove stale HA replicas from Inventory - #5738
Conversation
Render
Destroyed
Render
Destroyed
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5738 +/- ##
==========================================
+ Coverage 43.59% 45.07% +1.48%
==========================================
Files 415 218 -197
Lines 43134 27916 -15218
==========================================
- Hits 18804 12584 -6220
+ Misses 22454 13977 -8477
+ Partials 1876 1355 -521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughHA setup now removes eligible stale PMM Server nodes when trusted peer data identifies scaled-down replicas. It preserves active replicas and nodes with monitored services. Tests cover cleanup and invalid peer data. Documentation describes the resulting Inventory behavior. ChangesHA stale-node cleanup
Sequence Diagram(s)sequenceDiagram
participant HASetup as setupPMMServerHAAgents
participant Cleanup as RemoveStaleHANodes
participant Nodes as PMM node records
participant Agents as PMM agent records
HASetup->>Cleanup: Pass HA node ID and configured peers
Cleanup->>Nodes: Find obsolete PMM Server nodes
Cleanup->>Agents: Check monitored services
Cleanup->>Nodes: Remove eligible stale nodes
Cleanup-->>HASetup: Return cleanup result
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@documentation/docs/install-pmm/install-HA-clustered.md`:
- Line 832: Revise the statement about removed replica Nodes in the HA cluster
cleanup documentation to say that only eligible stale Nodes disappear after
remaining pods restart. Mention that removal is skipped when peer data is
untrusted or the Node still monitors Services, and instruct operators to move
those Services to a running replica before removal.
In `@managed/models/node_helpers_test.go`:
- Around line 272-275: Replace the live PostgreSQL setup in the
RemoveStaleHANodes unit test with a go-sqlmock database, configuring only the
SQL expectations needed by RemoveStaleHANodes. Keep testdb.Open out of this unit
test; move the coverage to integration tests only if the behavior cannot be
validated with mocked database interactions.
In `@managed/models/node_helpers.go`:
- Around line 375-394: Exclude the legacy PMM Server node from stale-replica
removal by adding a check for node.NodeID == PMMServerNodeID in the loop before
removeNode is called. Preserve the existing handling for eligible HA replicas
and retain the legacy node without attempting removal.
- Around line 411-420: Update the peer parsing helper around the existing
strings.TrimSpace/strings.Cut logic to call net.ParseIP on the full trimmed peer
entry before splitting at “:”, rejecting unbracketed IPv6 addresses instead of
treating their first segment as a node label. Preserve the existing host and
label validation for non-IP entries, and add an unbracketed IPv6 case to the
untrusted-peer tests.
- Around line 359-399: Replace the package-level logrus.Warnf and logrus.Infof
calls in the stale HA cleanup flow with structured *logrus.Entry logging. Attach
relevant fields such as node_id, node_name, peer, and ha_node_id to each event
while preserving the existing messages and control flow.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f0769d83-7fa0-483d-b09c-be3789de1b70
📒 Files selected for processing (4)
documentation/docs/install-pmm/install-HA-clustered.mdmanaged/models/database.gomanaged/models/node_helpers.gomanaged/models/node_helpers_test.go
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
percona/pmm-qa(manual)percona/pmm(manual)
| - HAProxy continues routing to available pods during rollout | ||
| - No data loss (distributed storage) | ||
| - Rolling update strategy minimizes downtime | ||
| - The Nodes of removed replicas disappear from **Inventory > Nodes** once the remaining pods restart |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
State the retention condition.
This sentence guarantees Node removal after restart. Cleanup retains a stale Node when it still monitors Services. Cleanup also skips removal when peer data is not trusted.
State that only eligible stale replica Nodes disappear. Explain that operators must move monitored Services to a running replica before removal.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@documentation/docs/install-pmm/install-HA-clustered.md` at line 832, Revise
the statement about removed replica Nodes in the HA cluster cleanup
documentation to say that only eligible stale Nodes disappear after remaining
pods restart. Mention that removal is skipped when peer data is untrusted or the
Node still monitors Services, and instruct operators to move those Services to a
running replica before removal.
| - HAProxy continues routing to available pods during rollout | ||
| - No data loss (distributed storage) | ||
| - Rolling update strategy minimizes downtime | ||
| - The Nodes of removed replicas disappear from **Inventory > Nodes** once the remaining pods restart |
There was a problem hiding this comment.
| - The Nodes of removed replicas disappear from **Inventory > Nodes** once the remaining pods restart | |
| - The Nodes of removed replicas get removed from **Inventory > Nodes** once the remaining pods restart |
disappear sounded more like an unwanted consequence rather than a necessary cleanup action.
ademidoff
left a comment
ademidoff
left a comment
There was a problem hiding this comment.
Review of the HA stale-node cleanup, against b2cbb72. Findings are inline; the first four are the ones I'd act on before merge, the rest are hardening, efficiency and test/doc notes.
The two headline items: the pre-HA pmm-server Node is classified as stale (either a permanent unactionable warning or, in the tail case, a PermissionDenied that aborts the migration and crash-loops the replica), and the cleanup runs at migration altitude where any failure keeps the server from booting.
Verified locally: make test-models RUN=TestRemoveStaleHANodes passes, golangci-lint is clean on the changed files, and the PermissionDenied path was reproduced with a throwaway test (since deleted). Everything else is reasoned from the code and labelled as such.
Nice fix on the unbracketed IPv6 peer in haPeerNodeName — that one is closed.
| for _, node := range nodes { | ||
| // Set by HA replicas, and by the PMM Server Node of a non-HA deployment; every other | ||
| // Node is one the user monitors. | ||
| if !node.IsPMMServerNode { |
There was a problem hiding this comment.
The legacy pmm-server Node is classified as stale.
The comment right above acknowledges that a non-HA deployment's PMM Server Node also carries the flag, but nothing skips it. On a DB that was first set up non-HA, setupPMMServerAgents (managed/models/database.go:1618) created a Node with NodeID = NodeName = "pmm-server" and IsPMMServerNode: true. That name is never in expected, so the loop treats it as a scaled-down replica. Two outcomes:
- Usually its pmm-agent still owns the
pmm-server-postgresqlexporter, solen(monitored) != 0and every replica logsKeeping stale HA node ... Re-add them from a running replica and remove the node from Inventoryon every single boot. That advice can't be followed: both public delete paths (managed/services/inventory/nodes.go:340,managed/services/management/node.go:181) go throughmodels.RemoveNode->removeNode(..., allowPMMServerNode=false)->PermissionDenied: PMM Server node can't be removed. - If that service is gone,
monitoredis empty andremoveNode(q, "pmm-server", RemoveCascade, true)hits theid == PMMServerNodeIDguard on line 269, returnsPermissionDenied, falls into thedefault:branch on line 405 and aborts the migration transaction.managed/cmd/pmm-managed/main.go:637then retries for 5 minutes and callsl.Fatalf("Could not migrate DB: timeout")— the replica crash-loops.
Reproduced locally: with the fixture's pmm-server-postgresql service deleted, RemoveStaleHANodes returns failed to remove stale HA node "pmm-server": rpc error: code = PermissionDenied desc = PMM Server node can't be removed.
Suggested fix: if node.NodeID == PMMServerNodeID { continue } at the top of the loop, and tolerate PermissionDenied in the switch below.
| logrus.Infof("Setting up PMM Server agents in HA mode, Node ID: %s", params.HANodeID) | ||
|
|
||
| // Before the "agent already exists" early return, so restarted replicas still clean up. | ||
| err := RemoveStaleHANodes(q, params.HANodeID, params.HAPeers) |
There was a problem hiding this comment.
Altitude: a cosmetic cleanup can stop the server from booting.
RemoveStaleHANodes runs inside the schema-migration transaction and returns hard errors, so anything it trips over aborts db.InTransaction in migrateDB, and managed/cmd/pmm-managed/main.go:640 eventually calls l.Fatalf("Could not migrate DB: timeout").
Candidates that reach that path today: PermissionDenied on the pre-HA pmm-server Node (see the comment on RemoveStaleHANodes), a NotFound out of the haNodeMonitoredServices pre-check while another replica commits, a row-lock/serialization failure when three replicas restart at once.
Tidying stale rows out of Inventory should never keep a PMM Server replica from starting. Suggest logging the error and continuing instead of returning it — nothing up the stack can recover from it.
| // haNodeMonitoredServices returns the IDs of Services whose exporters run under a replica's pmm-agent. | ||
| // Remote instances bind theirs to the replica that added them (see management.RDSService), so removing | ||
| // that replica's Node takes them with it. | ||
| func haNodeMonitoredServices(q *reform.Querier, nodeID string) ([]string, error) { |
There was a problem hiding this comment.
The safety check misses Services attached to the Node itself.
This only walks agents whose pmm_agent_id is one of the replica's pmm-agents, so a Service bound to the stale Node via services.node_id is invisible here. That shape exists in the codebase — managed/services/checks/checks.go:1377 special-cases service.NodeID == models.PMMServerNodeID, and under HA PMMServerNodeID is the local replica's Node.
If such a Service's exporter runs under a different replica's pmm-agent, monitored comes back empty, the guard passes, and removeNode(..., RemoveCascade, true) reaches managed/models/node_helpers.go:318 and calls RemoveService(..., RemoveCascade) on it. The Service and its agents are deleted silently — no warning, which is exactly what the guard exists to prevent.
Worth also counting SELECT ... FROM services WHERE node_id = $1 (and agents bound via node_id) before deciding a Node is safe to reap.
|
|
||
| monitored, err := haNodeMonitoredServices(q, node.NodeID) | ||
| if err != nil { | ||
| return err |
There was a problem hiding this comment.
The pre-check doesn't get the race tolerance removeNode was given.
Four lines down, the switch deliberately tolerates reform.ErrNoRows / codes.NotFound because another replica may be reaping the same Node concurrently. This return err bypasses that.
haNodeMonitoredServices -> FindAgents performs a FindAgentByID(filters.PMMAgentID) existence probe (managed/models/agent_helpers.go:241). If replica A commits the deletion of ha-node-2's pmm-agent while replica B sits between its FindNodes snapshot and this call, B gets codes.NotFound here and aborts the whole migration.
Same errors.Is(err, reform.ErrNoRows) || status.Code(err) == codes.NotFound -> continue treatment would close it.
| } | ||
|
|
||
| if n.IsPMMServerNode || id == PMMServerNodeID { | ||
| if id == PMMServerNodeID || (!allowPMMServerNode && n.IsPMMServerNode) { |
There was a problem hiding this comment.
With allowPMMServerNode = true, the only remaining protection is a mutable package variable.
PMMServerNodeID starts as "pmm-server" (managed/models/node_helpers.go / node_model.go:44) but setupPMMServerHAAgents reassigns it to this replica's UUID at managed/models/database.go:1561 and :1610 — after the cleanup call at :1532.
migrateDB retries SetupDB in a loop (managed/cmd/pmm-managed/main.go:637), so on a second attempt in the same process (attempt 1 reached :1610, then the COMMIT failed and rolled back) RemoveStaleHANodes runs with PMMServerNodeID pointing at a Node that no longer exists. The pre-HA pmm-server Node is then unguarded and, with the flag lifted, gets deleted outright.
RemoveAgent's id == PMMServerAgentID guard (managed/models/agent_helpers.go:1450) has the same order dependence. A guard that this very code path mutates is a fragile place to hang "can't delete the server's own node" on — better keyed off the caller's known NodeID.
| // e.g. after a scale-down. Peers are the source of truth because they are regenerated from the replica | ||
| // count and restart every replica, while a missing memberlist member may just be restarting. | ||
| func RemoveStaleHANodes(q *reform.Querier, haNodeID string, haPeers []string) error { | ||
| if len(haPeers) == 0 { |
There was a problem hiding this comment.
Does the headline scenario — scaling down to one replica — actually reach the cleanup?
If the chart emits an empty PMM_HA_PEERS for a single-replica cluster (nothing to gossip with), this early return fires. If it emits a list that doesn't include the surviving pod, the expected[haNodeID] guard below fires. Either way the two departed replicas' Nodes stay in Inventory forever, which is the case the PR is meant to fix.
Nothing here pins down what the chart produces for replicas: 1, and there's no test for it. Worth confirming against the chart before merge — and if the list can legitimately be empty, that state needs its own handling.
|
|
||
| // neither monitored nodes nor the pre-HA pmm-server Node are touched | ||
| assertNodeExists(t, q, "monitored-node") | ||
| assertNodeExists(t, q, models.PMMServerNodeID) |
There was a problem hiding this comment.
This assertion passes by accident, and the interesting path is untested.
The comment above says the pre-HA pmm-server Node isn't touched, but it survives only because SetupFixtures attaches pmm-server-postgresql to it, so the run takes the len(monitored) != 0 branch. The test log shows it:
Keeping stale HA node "pmm-server" (pmm-server): it still monitors services [5bf3df6d-... 5bf3df6d-...]
Nothing here exercises protection for PMM Server Nodes. Drop that one fixture service and RemoveStaleHANodes returns PermissionDenied, failing this subtest at the require.NoError on line 343.
Worth an explicit case for a PMM-Server-flagged Node with no monitored services. Also assertNodeExists uses assert.NoError (line 335) — require would stop the subtest at the real failure instead of cascading.
|
|
||
| var serviceIDs []string | ||
| for _, pmmAgent := range pmmAgents { | ||
| agents, err := FindAgents(q, AgentFilters{PMMAgentID: pmmAgent.AgentID}) |
There was a problem hiding this comment.
N+1 queries, credential decryption, and duplicate IDs, all to read ServiceID.
FindAgentsruns aFindAgentByIDexistence probe (managed/models/agent_helpers.go:241) before its real query, once per pmm-agent.FindPMMAgentsRunningOnNodecallsDecryptAgenton every row (managed/models/agent_helpers.go:423), pulling encrypted passwords through the encryption layer inside the migration transaction just to testagent.ServiceID != nil.- The result carries duplicates straight into the operator-facing warning — the test run logged
it still monitors services [5bf3df6d-361d-42dd-9dd2-3d0b3573ab24 5bf3df6d-361d-42dd-9dd2-3d0b3573ab24]because two exporters point at the same service.
One query replaces the helper:
SELECT DISTINCT service_id FROM agents
WHERE pmm_agent_id IN (SELECT agent_id FROM agents WHERE runs_on_node_id = $1 AND agent_type = 'pmm-agent')
AND service_id IS NOT NULL| return nil | ||
| } | ||
|
|
||
| nodes, err := FindNodes(q, NodeFilters{}) |
There was a problem hiding this comment.
Loads the whole nodes table to keep a handful of rows.
Every HA replica boot materializes every monitored Node inside the migration transaction, only for the loop below to discard everything without IsPMMServerNode. On an install with thousands of Nodes that's a full scan plus full row hydration per replica start.
NodeFilters (managed/models/node_helpers.go:92) has no IsPMMServerNode field; adding one — or issuing q.SelectAllFrom(NodeTable, "WHERE is_pmm_server_node") directly here — makes the cost proportional to the replica count instead of the inventory size.
| - HAProxy continues routing to available pods during rollout | ||
| - No data loss (distributed storage) | ||
| - Rolling update strategy minimizes downtime | ||
| - The Nodes of removed replicas disappear from **Inventory > Nodes** once the remaining pods restart |
There was a problem hiding this comment.
This is stated unconditionally, but the cleanup is skipped in several configurations: PMM_HA_PEERS built from bare IPs (10.244.1.7:9761,...), an empty peer list, a list that doesn't include the local pod, or any entry the parser can't read a name from (a trailing comma is enough). In those cases the Nodes stay in Inventory and an operator will read this bullet as a broken promise.
Suggest qualifying it — the Nodes are removed when PMM_HA_PEERS carries resolvable node names — and mentioning that a WARN is logged when the cleanup is skipped, so there's something to grep for.
3 participants
PMM-15227
Percona-Lab/pmm-submodules#4513
Problem
Every PMM Server replica in an HA deployment registers its own inventory Node at first boot,
and nothing ever removed it. After scaling the deployment down, the departed replica kept
appearing under Inventory > Nodes with status Unknown.
Scope
HA deployments only, and inventory rows only. No API or schema change; metrics already
written to VictoriaMetrics are left alone.
Solution
At startup, remove PMM Server Nodes whose name is absent from the configured
PMM_HA_PEERS.The peer list is the signal because the chart regenerates it from the replica count and
recreates every pod when it changes, whereas a member missing from the memberlist cluster may
simply be restarting.
Cleanup is skipped when the peer list cannot be trusted (an entry that carries no node name,
or a list that omits this replica), and a departing replica that still monitors Services is
kept with a warning instead of having those exporters cascade-deleted.