Skip to content

fix(core): enforce chart uniqueness via lease claim#54

Open
drey wants to merge 3 commits into
mainfrom
fix/webhhook-race
Open

fix(core): enforce chart uniqueness via lease claim#54
drey wants to merge 3 commits into
mainfrom
fix/webhhook-race

Conversation

@drey

@drey drey commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Makes the "at most one HelmClusterAddon per repository/chart pair" rule actually hold under load. The rule was enforced only by a validating webhook, which let duplicates slip through when many addons were created in a burst. Uniqueness is now guaranteed in the reconciler via an atomic Lease-based claim, and the webhook is demoted to a best-effort fast-reject on CREATE.

Why

The validating webhook cannot reliably enforce cross-object uniqueness:

  • Admission requests are served concurrently and before the object is persisted, so two simultaneous CREATEs for the same chart both list the current state, neither sees the other, and both are admitted.
  • The check listed existing addons through the controller cache, which lags the API server, widening the window further.

The result: creating many HelmClusterAddons quickly produced several objects using the same chart/repo — exactly what the webhook was meant to prevent.

The only atomic cross-object primitive Kubernetes offers is object-name uniqueness in etcd. This PR maps each repo/chart pair to a single Lease name and lets the API server serialize the winners.

Key changes

New ChartClaimServiceimages/operator-helm-controller/internal/services/chart_claim_service.go

  • Acquire creates a Lease named deterministically from the repo/chart pair (utils.GetChartClaimLeaseName, a pure hash → always a valid DNS name) with holderIdentity = addon.Name. First creator wins; losers observe a foreign holder.
  • Reads go through mgr.GetAPIReader() (uncached) so an acquisition decision is never made against stale cache data.
  • Self-healing: a claim whose holder no longer exists or has repointed at another chart is taken over via an optimistic Update.
  • Release deletes the Lease only if this addon still owns it, guarded by a ResourceVersion precondition so a taken-over claim is never clobbered.

Reconciler wiringinternal/reconcile/helmclusteraddon/reconciler.go

  • Claims the pair before adding the finalizer: a duplicate that loses the race surfaces a ChartClaimConflict condition and requeues (every 30s) instead of accruing a finalizer it would have to clean up. It recovers on its own once the owner releases the pair.
  • Releases the claim last in reconcileDelete, only after every downstream resource is gone, so no other addon can start reconciling the same chart mid-teardown.

Webhook demoted to CREATE-onlyinternal/webhook/helmclusteraddon/webhook.go, templates/operator-helm-controller/validation-webhook.yaml

  • ValidateUpdate no longer checks uniqueness. With duplicates transiently present, validating updates would reject the very finalizer the reconciler needs to resolve the conflict.
  • UPDATE dropped from the webhook manifest's operations (now ["CREATE", "DELETE"]), since both validators are now no-ops on update — removing needless webhook round-trips and a failure point under failurePolicy: Fail.

Supporting bits

  • api/v1alpha1/conditions.go: new ReasonChartClaimConflict reason.
  • internal/utils/name.go: GetChartClaimLeaseName helper.
  • No RBAC change needed — the controller already has full leases verbs in d8-operator-helm (from leader election).

Review focus / risks

  • Correctness of Acquire (chart_claim_service.go) — the race-safety hinges on: Create returning AlreadyExists → re-read → ownership check, and the takeover path using optimistic Update (conflict → report as claimed + requeue). Verify no branch can wrongly return acquired = true for a foreign-held claim.
  • Ordering: acquire before finalizer — confirm the winner can still add its finalizer after acquiring (it can, now that ValidateUpdate is a no-op), and that a loser correctly gets no finalizer.
  • Claim leak on crash — if the winner acquires but crashes before the finalizer is added and is then deleted, the Lease lingers. It self-heals via takeover (next contender sees the holder gone). Confirm this is acceptable; it is the one known non-cleaned edge.
  • Stale-holder detection (isHolderStale) — a holder that changed its chart leaves its old Lease orphaned until another addon contends for that chart. Verify the "holder's current spec maps to a different lease name ⇒ stale" logic.
  • Lease name hashingGetChartClaimLeaseName uses the shared 12-char hash (utils.GetHash); a collision would be a false conflict. Consistent with existing name helpers, but worth a sanity check for expected addon counts.
  • Webhook behavior change — updates are no longer validated for uniqueness at all; spec-change collisions (repointing an addon at an already-claimed chart) are now caught asynchronously via ChartClaimConflict rather than synchronously at kubectl apply. Confirm this UX trade-off is intended.
section: core
type: fix
summary: enforce HelmClusterAddon chart/repo uniqueness via an atomic Lease claim in the reconciler, fixing duplicates that raced past the validating webhook under load
---
section: api
type: fix
summary: add ChartClaimConflict condition reason for HelmClusterAddon

The validating webhook could not reliably prevent two HelmClusterAddon
objects from using the same repository/chart pair: admission requests are
served concurrently and before the object is persisted, and the check read
from the controller cache, so bursts of creates raced past it.

Move the guarantee into the reconciler and delegate it to the API server's
only atomic cross-object primitive, object-name uniqueness: each repo/chart
pair maps to a single Lease name, claimed via ChartClaimService before any
downstream resource is created. The first creator wins; a duplicate that
loses surfaces a ChartClaimConflict condition and requeues, recovering once
the owner releases the pair. Stale claims (deleted or repointed owner) are
taken over. The claim is acquired before the finalizer so a loser accrues
no finalizer, and released only after all downstream resources are gone.

Drop uniqueness validation from the webhook's UPDATE path (and UPDATE from
the manifest): with duplicates transiently present, validating updates would
reject the very finalizer the reconciler needs to resolve the conflict. The
webhook now only fast-rejects the obvious duplicate on CREATE.

Signed-off-by: Ilya Drey <ilya.drey@flant.com>
@drey
drey force-pushed the fix/webhhook-race branch 3 times, most recently from b3f69d9 to ef34f8a Compare July 22, 2026 08:16
Signed-off-by: Ilya Drey <ilya.drey@flant.com>
@drey
drey force-pushed the fix/webhhook-race branch from ef34f8a to b3fec1a Compare July 22, 2026 09:22
Signed-off-by: Ilya Drey <ilya.drey@flant.com>
@drey
drey force-pushed the fix/webhhook-race branch from 8825300 to ecb7e98 Compare July 22, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant