Skip to content

CNTRLPLANE-3532: migrate CPO status patches to statuspatching helpers - #8966

Open
vsolanki12 wants to merge 1 commit into
openshift:mainfrom
vsolanki12:vsolanki/cntrlplane-3532-tier2-optimistic-lock
Open

CNTRLPLANE-3532: migrate CPO status patches to statuspatching helpers#8966
vsolanki12 wants to merge 1 commit into
openshift:mainfrom
vsolanki12:vsolanki/cntrlplane-3532-tier2-optimistic-lock

Conversation

@vsolanki12

@vsolanki12 vsolanki12 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Migrates 9 status patch call sites in the CPO to use statuspatching.PatchStatus / PatchStatusCondition, adding retry-on-conflict and consistent optimistic locking.

  • hostedcontrolplane_controller.go: 7 sites migrated (reconcileDeletion, update, reconcileValidIDPConfigurationCondition, removeCloudResources, reconcileDefaultSecurityGroup)
  • reencryption.go: 1 site migrated (HCCO re-encryption controller)
  • 2 batch-patch sites deferred — they accumulate status across the full reconcile loop and require restructuring

Part of the broader CNTRLPLANE-3532 migration. Depends on PR #8782 (merged).

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3532

Special notes for your reviewer:

  • 2 batch-patch sites (lines 686, 869 in hostedcontrolplane_controller.go) are intentionally skipped — they accumulate status changes from the entire reconcile loop and migrating them requires restructuring the reconcile flow.
  • The originalHostedControlPlane parameter was removed from reconcileDeletion since the migrated helpers handle re-fetching internally.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
    • Improved HostedControlPlane condition/lifecycle updates for deletion, validation, and cloud resource teardown using more atomic status handling.
    • Enhanced reencryption reconciliation so EtcdDataEncryptionUpToDate tracks the correct desired encryption state.
    • Hosted control plane startup now waits for etcd DNS availability when using managed etcd.
    • Router reconciliation on ARO HCPs now gates on required routes being present/ready; Azure public IP provisioning retries on specific conflict errors.
  • Chores
    • Reduced unnecessary status updates by patching only changed fields.
  • Tests
    • Updated HostedControlPlane deletion tests and expanded router/encryption-related test coverage.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 9, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 9, 2026

Copy link
Copy Markdown

@vsolanki12: This pull request references CNTRLPLANE-3532 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Migrates 9 status patch call sites in the CPO to use statuspatching.PatchStatus / PatchStatusCondition, adding retry-on-conflict and consistent optimistic locking.

  • hostedcontrolplane_controller.go: 7 sites migrated (reconcileDeletion, update, reconcileValidIDPConfigurationCondition, removeCloudResources, reconcileDefaultSecurityGroup)
  • reencryption.go: 1 site migrated (HCCO re-encryption controller)
  • 2 batch-patch sites deferred — they accumulate status across the full reconcile loop and require restructuring

Part of the broader CNTRLPLANE-3532 migration. Depends on PR #8782 (merged).

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3532

Special notes for your reviewer:

  • 2 batch-patch sites (lines 686, 869 in hostedcontrolplane_controller.go) are intentionally skipped — they accumulate status changes from the entire reconcile loop and migrating them requires restructuring the reconcile flow.
  • The originalHostedControlPlane parameter was removed from reconcileDeletion since the migrated helpers handle re-fetching internally.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 9, 2026
@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds stream-aware NodePool image resolution and OS stream status inference, targeted HostedControlPlane status patching, encryption-config hash convergence checks, cloud-provider retry and cleanup paths, GCP PSC network-scoped subnet discovery, and HyperShift AWS resource tagging. It also updates router and deployment ordering, secret cleanup labels, resource override handling, E2E utilities, logging, tests, and build tooling.

Sequence Diagram(s)

sequenceDiagram
  participant NodePoolReconciler
  participant ReleaseImage
  participant ProviderImageResolver
  participant MachineTemplate
  NodePoolReconciler->>ReleaseImage: Resolve selected RHEL stream
  NodePoolReconciler->>ProviderImageResolver: Request stream-specific image
  ProviderImageResolver->>MachineTemplate: Build provider machine template
  MachineTemplate-->>NodePoolReconciler: Return image and rollout identity
Loading

Possibly related PRs

Suggested reviewers: enxebre, devguyio

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release and removed do-not-merge/needs-area labels Jul 9, 2026
@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-tier2-optimistic-lock branch from fb8a23c to d73cc0d Compare July 9, 2026 13:57
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.78873% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.10%. Comparing base (5b60ff2) to head (4237039).

⚠️ Current head 4237039 differs from pull request most recent head 9a6b2ba

Please upload reports for the commit 9a6b2ba to get more accurate results.

Files with missing lines Patch % Lines
...ostedcontrolplane/hostedcontrolplane_controller.go 48.83% 17 Missing and 5 partials ⚠️
...goperator/controllers/reencryption/reencryption.go 89.28% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8966      +/-   ##
==========================================
- Coverage   44.61%   44.10%   -0.52%     
==========================================
  Files         774      772       -2     
  Lines       97098    96167     -931     
==========================================
- Hits        43317    42410     -907     
- Misses      50785    50816      +31     
+ Partials     2996     2941      -55     
Files with missing lines Coverage Δ
...goperator/controllers/reencryption/reencryption.go 70.46% <89.28%> (+1.11%) ⬆️
...ostedcontrolplane/hostedcontrolplane_controller.go 46.02% <48.83%> (-0.01%) ⬇️

... and 59 files with indirect coverage changes

Flag Coverage Δ
cmd-support 38.24% <ø> (-0.16%) ⬇️
cpo-hostedcontrolplane 46.05% <48.83%> (-1.17%) ⬇️
cpo-other 45.20% <89.28%> (-0.05%) ⬇️
hypershift-operator 54.09% <ø> (-0.46%) ⬇️
other 32.12% <ø> (-1.19%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go`:
- Around line 1117-1139: The status patch in hostedcontrolplane_controller.go is
using a stale copy of HostedControlPlane and will overwrite earlier updates made
in update() and reconcileCPOV2. Fix the PatchStatus call to patch the current
in-memory hostedControlPlane state, or explicitly merge the existing status
fields back before setting ValidReleaseInfo. Keep the existing status mutations
such as Ready, KubeConfig, KubeadminPassword, ControlPlaneVersion, Initialized,
and prior conditions intact when applying the patch.

In
`@control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go`:
- Around line 76-89: The `desiredCondition` in `reconcile` is a pointer into
`hcp.Status.Conditions`, so `statuspatching.PatchStatus` may re-fetch and
overwrite the backing slice before the callback uses it. Capture the condition
by value before calling `PatchStatus` (for example, copy the result of
`meta.FindStatusCondition` into a standalone variable) and then use that copied
value inside the patch callback when setting `hcp.Status.Conditions`.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ddb9d783-3baf-45e9-a217-5ed2a8755bb9

📥 Commits

Reviewing files that changed from the base of the PR and between 1f2811f and fb8a23c.

📒 Files selected for processing (3)
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-tier2-optimistic-lock branch from d73cc0d to 7848ce8 Compare July 9, 2026 14:54
@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Both failures are now fully analyzed. Let me produce the final report.

Test Failure Analysis Complete

Job Information


  • Konflux Job: Red Hat Konflux / control-plane-operator-main-on-pull-request
  • Check Run ID: 86150298660
  • PipelineRun: control-plane-operator-main-on-pull-request-gkspc

Test Failure Analysis

Error

# Job 1: verify / Verify — make deps failure
golang.org/x/net@v0.53.0: read "https://proxy.golang.org/golang.org/x/net/@v/v0.53.0.zip": stream error: stream ID 773; INTERNAL_ERROR; received from peer
github.com/alecthomas/chroma/v2@v2.23.1: read "https://proxy.golang.org/github.com/alecthomas/chroma/v2/@v/v2.23.1.zip": stream error: stream ID 787; INTERNAL_ERROR; received from peer
make: *** [Makefile:552: deps] Error 1

# Job 2: Red Hat Konflux / control-plane-operator — rpms-signature-scan image pull failure
the step "rpms-signature-scan" in TaskRun failed to pull the image "quay.io/konflux-ci/tools@sha256:cdab2b0f6e01ed7852ad4c5376fd58bdbfe0db1379dd5acf42aad722a0f1e168":
Back-off pulling image: ErrImagePull: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable.

Summary

Both failures are transient infrastructure flakes completely unrelated to the code changes in PR #8966. The verify / Verify job failed because the Go module proxy (proxy.golang.org) returned HTTP/2 INTERNAL_ERROR stream errors while downloading dependency zip files during make deps. The Konflux / control-plane-operator job failed because Quay.io returned 503 Service Unavailable when the rpms-signature-scan Tekton task attempted to pull its tooling image. All 16 other Konflux pipeline tasks (including build, security scans, and SBOM generation) succeeded. Retrying both jobs should resolve these failures.

Root Cause

Job 1 — verify / Verify:
The make generate update target invokes make deps, which builds tools from hack/tools. During go build -tags=tools, the Go module proxy at proxy.golang.org returned HTTP/2 stream-level INTERNAL_ERROR responses for two module zip downloads:

  1. golang.org/x/net@v0.53.0 — cascading across 7+ import chains (html/charset, proxy, http2, grpc, etc.)
  2. github.com/alecthomas/chroma/v2@v2.23.1 — via golangci-lint → goformat → chroma/quick

These are server-side errors from the Go module proxy CDN, not local network issues or code problems.

Job 2 — Red Hat Konflux / control-plane-operator-main-on-pull-request:
The pipeline completed 16 of 17 tasks successfully. The sole failing task was rpms-signature-scan, which has status TaskRunImagePullFailed. The Tekton pod could not pull quay.io/konflux-ci/tools@sha256:cdab2b... because Quay.io returned HTTP 503 during blob fetch. This also caused the downstream enterprise-contract check to be cancelled (it never ran since it depends on this pipeline).

Neither failure is related to the PR's code changes (migrating CPO status patches to statuspatching helpers).

Recommendations
  1. Rerun both jobs — these are transient infrastructure failures that will resolve on retry.
  2. For the Verify job: Use the GitHub Actions "Re-run failed jobs" button or push an empty commit to re-trigger.
  3. For the Konflux job: Re-trigger via the Konflux UI or add a /retest comment if supported. The downstream enterprise-contract check will automatically run once the pipeline succeeds.
  4. No code changes are needed in PR CNTRLPLANE-3532: migrate CPO status patches to statuspatching helpers #8966.
Evidence
Evidence Detail
Verify failed step Step 4 — "Run make generate update" at Makefile:552 (deps target)
Go proxy error 1 golang.org/x/net@v0.53.0 — HTTP/2 stream ID 773 INTERNAL_ERROR from proxy.golang.org
Go proxy error 2 github.com/alecthomas/chroma/v2@v2.23.1 — HTTP/2 stream ID 787 INTERNAL_ERROR from proxy.golang.org
Konflux failed task rpms-signature-scan — status TaskRunImagePullFailed (8s duration)
Konflux image pull error quay.io/konflux-ci/tools@sha256:cdab2b... — HTTP 503 Service Unavailable from Quay.io
Konflux passing tasks 16/17 tasks passed (init, clone, build-images, clair-scan, sast-snyk, etc.)
Konflux cancelled downstream enterprise-contract check cancelled due to pipeline failure
Other Konflux pipelines hypershift-operator-main-on-pull-request ✅ passed
PR HEAD SHA 7848ce80b2b3931be0f1cf2fecb7582cb036fc0c

@vsolanki12
vsolanki12 marked this pull request as ready for review July 13, 2026 09:27
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 13, 2026
@openshift-ci
openshift-ci Bot requested review from devguyio and enxebre July 13, 2026 09:28
@vsolanki12 vsolanki12 closed this Jul 13, 2026
@vsolanki12 vsolanki12 reopened this Jul 13, 2026

@cblecker cblecker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migration to statuspatching helpers looks well-executed across all call sites. F1 (metrics regression in reencryption.go) is the main concern — the rest are suggestions.

}

// Record metrics when encryption status changed.
if !equality.Semantic.DeepEqual(previousEncryption, desiredEncryption) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recordMigrationState is now gated behind the DeepEqual check, but the old code called it unconditionally on every reconcile. After an HCCO pod restart in steady state (no encryption change), all hypershift_encryption_migration_state gauges stay at zero indefinitely — the "idle" gauge is never re-set to 1. This could confuse dashboards/alerts until the next key rotation, which may be weeks away.

Consider moving recordMigrationState outside the if !equality.Semantic.DeepEqual(...) block so it runs unconditionally, matching the old behavior. recordMigrationDuration should stay inside the guard since it should only fire on actual transitions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Moved recordMigrationState outside the DeepEqual guard so it runs unconditionally on every reconcile, matching the old behavior. recordMigrationDuration stays inside the guard since it should only fire on actual transitions.


AI-assisted response via Claude Code


func (r *HostedControlPlaneReconciler) reconcileDeletion(ctx context.Context, hostedControlPlane *hyperv1.HostedControlPlane, originalHostedControlPlane *hyperv1.HostedControlPlane) (ctrl.Result, error) {
func (r *HostedControlPlaneReconciler) reconcileDeletion(ctx context.Context, hostedControlPlane *hyperv1.HostedControlPlane) (ctrl.Result, error) {
condition := &metav1.Condition{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: condition is declared as a pointer (&metav1.Condition{...}) and then dereferenced (*condition) when passed to PatchStatusCondition. The other migrated sites (reconcileDefaultSecurityGroup, removeCloudResources) use value types or inline literals. Switching to a value type here would be more consistent and avoids the unnecessary indirection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Changed condition from *metav1.Condition to metav1.Condition — consistent with the other migrated sites now.


AI-assisted response via Claude Code

if err := statuspatching.PatchStatus(ctx, r.Client, hcp, func() error {
meta.SetStatusCondition(&hcp.Status.Conditions, condition)
if creationErr == nil {
hcp.Status.Platform = &hyperv1.PlatformStatus{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing, but worth noting since the PatchStatus migration touches this: the callback replaces the entire hcp.Status.Platform struct with a new one containing only the security group ID. If PlatformStatus gains additional fields in the future, they'd be silently cleared on every reconcile. With PatchStatus retrying on conflict, the fresh struct also discards whatever the server has at retry time.

Consider initializing hcp.Status.Platform / hcp.Status.Platform.AWS if nil instead of replacing, then setting only DefaultWorkerSecurityGroupID.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Changed to init-if-nil pattern — hcp.Status.Platform and hcp.Status.Platform.AWS are now initialized only if nil, then only DefaultWorkerSecurityGroupID is set. This preserves any other fields that may be added to PlatformStatus in the future.


AI-assisted response via Claude Code

Status: metav1.ConditionTrue,
Reason: hyperv1.AsExpectedReason,
Message: hyperv1.AllIsWellMessage,
ObservedGeneration: hostedControlPlane.Generation,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a regression (old code also re-fetched before referencing Generation), but ObservedGeneration inside the PatchStatus closure will reflect the re-fetched HCP's generation, which may be newer than the generation used to compute missingImages. If the spec changed between the original read and the re-fetch, the condition content won't match what that generation actually means. A follow-up reconcile self-corrects, so this is minor — just flagging in case you want to snapshot the generation before the PatchStatus call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. This is pre-existing — the old code also re-fetched before referencing Generation. A follow-up reconcile self-corrects, so leaving as-is for now.


AI-assisted response via Claude Code

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-tier2-optimistic-lock branch from 7848ce8 to 4237039 Compare July 15, 2026 04:08
// Capture desired status changes computed by reconcile().
// Copy by value — PatchStatus re-fetches hcp, which replaces the backing slice.
desiredEncryption := *hcp.Status.SecretEncryption.DeepCopy()
var desiredCondition metav1.Condition

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential semantic narrowing: The original MergeFrom(originalHCP) patch captured all status mutations made by reconcile(). This new code snapshots only SecretEncryption and the EtcdDataEncryptionUpToDate condition, then replays just those two fields inside the PatchStatus closure.

If reconcile() (or any of its sub-functions like handleInitialBootstrap, startNewRotation, handleMigratingPhase, etc.) sets other status fields or conditions beyond these two, those changes are silently dropped after PatchStatus re-fetches the object.

Is EtcdDataEncryptionUpToDate the only condition reconcile() touches? If so this is fine — but worth a comment saying so. If not, the other conditions need to be captured and replayed too.

@vsolanki12 vsolanki12 Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Confirmed — reconcile() only mutates SecretEncryption and the EtcdDataEncryptionUpToDate condition. No other status fields or conditions. Added a comment on the snapshot block stating this explicitly.

@openshift-ci openshift-ci Bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. area/api Indicates the PR includes changes for the API area/ci-tooling Indicates the PR includes changes for CI or tooling area/cli Indicates the PR includes changes for CLI area/documentation Indicates the PR includes changes for documentation area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release labels Jul 28, 2026
@openshift-ci openshift-ci Bot added area/platform/aws PR/issue for AWS (AWSPlatform) platform area/platform/azure PR/issue for Azure (AzurePlatform) platform area/platform/gcp PR/issue for GCP (GCPPlatform) platform area/platform/kubevirt PR/issue for KubeVirt (KubevirtPlatform) platform area/platform/openstack PR/issue for OpenStack (OpenStackPlatform) platform area/platform/powervs PR/issue for PowerVS (PowerVSPlatform) platform area/testing Indicates the PR includes changes for e2e testing labels Jul 28, 2026
…ng helpers

Migrate 9 status patch call sites across hostedcontrolplane_controller.go
and reencryption.go to use the shared statuspatching package, adding
retry-on-conflict and consistent optimistic locking.

- hostedcontrolplane_controller.go: 7 sites migrated to PatchStatus /
  PatchStatusCondition (reconcileDeletion, update, reconcileValidIDP,
  removeCloudResources, reconcileDefaultSecurityGroup)
- reencryption.go: 1 site migrated to PatchStatus
- 2 batch-patch sites (lines 686, 869) deferred — they accumulate
  status changes across the full reconcile loop and need restructuring

Signed-off-by: Vimal Solanki <vsolanki@redhat.com>
@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-tier2-optimistic-lock branch from 50d87f2 to 6aa69ee Compare July 28, 2026 12:56
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2026
@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vsolanki12
Once this PR has been reviewed and has the lgtm label, please ask for approval from cblecker. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-tier2-optimistic-lock branch from 6aa69ee to 9a6b2ba Compare July 28, 2026 13:02

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

🧹 Nitpick comments (9)
support/controlplane-component/status.go (1)

39-40: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add regression coverage for the new dependency exclusions.

Please add or verify focused tests in support/controlplane-component/status_test.go proving that both ignition-server and ignition-server-proxy omit the kube-apiserver dependency while existing components retain it. A typo or accidental removal would otherwise silently change reconciliation ordering.

As per coding guidelines, Go code changes and additions must be unit tested.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/controlplane-component/status.go` around lines 39 - 40, Add focused
unit tests in the status test suite covering the dependency exclusions for both
“ignition-server” and “ignition-server-proxy”, asserting they omit
kube-apiserver while existing components still include it. Reuse the existing
status/dependency test helpers and preserve current reconciliation behavior.

Source: Coding guidelines

hack/tools/crd-schema-check/main.go (1)

58-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for common-directory repositories.

Please add or verify a test in hack/tools/crd-schema-check/main_test.go that exercises a .git file pointing to a common Git directory. This option changes repository-opening behavior and should not rely only on the standard repository path.

As per coding guidelines, Go code changes and additions must be unit tested.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/crd-schema-check/main.go` around lines 58 - 60, Add regression
coverage in main_test.go for the repository-opening flow around
gogit.PlainOpenWithOptions, using a repository whose .git is a file pointing to
a common Git directory. Verify the repository opens and the CRD schema check
behavior succeeds, while preserving existing standard-repository test coverage.

Source: Coding guidelines

control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go (1)

105-124: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Route names are duplicated string literals.

ignition-server, oauth-internal, etc. already exist as constants/manifest helpers elsewhere (e.g. hypershift-operator/controllers/manifests/ignitionserver exposes the ignition route name). Referencing those keeps this gate in sync if a route is ever renamed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go`
around lines 105 - 124, Replace the duplicated route-name literals in
aroBaseHCPRouterRouteNames and aroExpectedHCPRouterRouteNames with the existing
route-name constants or manifest helpers, including the ignition, OAuth, and
metrics-proxy routes. Reuse the canonical symbols from the relevant packages so
route validation stays synchronized with any future renames.
cmd/cluster/core/create_test.go (1)

1814-1822: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider making the backoff injectable to avoid real sleeps in unit tests.

validateClusterExistenceWithClient hardcodes a 1s/2x/5-step backoff, so the persistent-timeout and service-unavailable cases sleep for real (~15s and ~3s respectively), noticeably slowing this package's tests. Passing the wait.Backoff as a parameter (or a package-level var overridable in tests) would keep coverage while dropping runtime to near zero.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/cluster/core/create_test.go` around lines 1814 - 1822, The backoff used
by validateClusterExistenceWithClient is hardcoded, causing persistent-error
tests to sleep in real time. Make its wait.Backoff configurable—prefer passing
it into validateClusterExistenceWithClient or using an overridable package-level
value—and configure the relevant tests with a near-zero backoff while preserving
production defaults and retry behavior.
hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller.go (1)

313-333: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

isSubnetInUse re-lists all service attachments for every candidate subnet.

Each loop iteration issues a fresh ListServiceAttachments call (with its own timeout). For a project with many PSC subnets this is O(N) identical API calls and increases the chance of hitting quota/rate limits. Listing once before the loop and matching in-memory would be equivalent.

♻️ Sketch
+	usedSubnets, err := r.usedNATSubnets(ctx)
+	if err != nil {
+		return "", fmt.Errorf("failed to list service attachments: %w", err)
+	}
+
 	// Find the first available PSC subnet in the MC's VPC not already in use by another Service Attachment.
-	var checkErrors int
 	for _, subnet := range subnets {
-		inUse, err := r.isSubnetInUse(ctx, subnet.Name)
-		if err != nil {
-			log.Error(err, "Failed to check subnet usage", "subnet", subnet.Name)
-			checkErrors++
-			continue
-		}
-
-		if !inUse {
+		if !usedSubnets.Has(subnet.Name) {
 			log.Info("Found available PSC subnet", "subnet", subnet.Name)
 			return subnet.Name, nil
 		}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller.go`
around lines 313 - 333, Update the subnet-selection flow around isSubnetInUse to
list the project’s Service Attachments once before iterating over subnets, then
determine each candidate’s usage from that in-memory result. Remove the
per-candidate isSubnetInUse API calls while preserving the existing
available-subnet selection, error logging, and no-available-subnet behavior.
hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller_test.go (1)

387-435: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider consolidating these into table-driven subtests.

TestLookupForwardingRule_* and TestDiscoverNATSubnet_* are near-identical single-scenario functions differing only in fake inputs and expectations; the filter assertion is copy-pasted four times. Folding each group into one table-driven test with When ... it should ... case names would match the repo's test conventions.

As per coding guidelines: "Use table-driven tests where possible" and "Always use 'When ... it should ...' format for describing test cases when creating unit tests".

Also applies to: 529-600

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller_test.go`
around lines 387 - 435, Consolidate the TestLookupForwardingRule_* cases and the
TestDiscoverNATSubnet_* cases into one table-driven test per function, using
subtest names in “When ... it should ...” format. Preserve each scenario’s fake
inputs, expected result, error handling, and forwarding-rule filter assertions
while removing duplicated test setup and checks.

Source: Coding guidelines

hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go (1)

560-566: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

expectedError is never exercised.

No test case sets expectedError: true, so the error branch is dead. Either add a case (e.g. an injected List/Update failure via an interceptor client) or drop the field.

Also applies to: 704-708

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go`
around lines 560 - 566, Update the test cases around the expectedError field so
the error path is actually exercised: add a case that injects a List or Update
failure through the interceptor client and asserts the expected error, or remove
expectedError and its related assertions if error coverage is not intended.
Apply the same change to the additional test table occurrence.
control-plane-operator/hostedclusterconfigoperator/controllers/machine/machine.go (1)

138-146: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Flatten the nested conditionals.

The per-family guards can be combined into single conditions, which keeps this loop body flatter. As per coding guidelines, "Do not over-nest control flow (e.g., nested if or for blocks)."

♻️ Proposed refactor
-			if parsedAddr.Is4() {
-				if len(ipv4MachineAddresses) == 0 {
-					ipv4MachineAddresses = append(ipv4MachineAddresses, machineAddress.Address)
-				}
-			} else {
-				if len(ipv6MachineAddresses) == 0 {
-					ipv6MachineAddresses = append(ipv6MachineAddresses, machineAddress.Address)
-				}
-			}
+			if parsedAddr.Is4() && len(ipv4MachineAddresses) == 0 {
+				ipv4MachineAddresses = append(ipv4MachineAddresses, machineAddress.Address)
+			} else if !parsedAddr.Is4() && len(ipv6MachineAddresses) == 0 {
+				ipv6MachineAddresses = append(ipv6MachineAddresses, machineAddress.Address)
+			}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/hostedclusterconfigoperator/controllers/machine/machine.go`
around lines 138 - 146, Flatten the nested conditionals in the machine address
loop: combine each `parsedAddr.Is4()`/`len(...) == 0` guard into one condition,
and likewise combine the IPv6 guard, while preserving the existing append
behavior and family-specific slices.

Source: Coding guidelines

hypershift-operator/controllers/nodepool/version_test.go (1)

283-291: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Type the fixtures as []*v1beta1.Machine and drop the conversion loop.

Now that the helpers take []*capiv1.Machine directly, the []client.Object field plus unchecked obj.(*v1beta1.Machine) assertion in both tests is unnecessary indirection (and would panic on a mistyped fixture).

Also applies to: 581-588

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/nodepool/version_test.go` around lines 283 -
291, Update both affected tests to declare their machine fixtures as
[]*v1beta1.Machine (or the helper’s expected machine type) instead of
[]client.Object, then pass those fixtures directly to setNodesInfoStatus. Remove
the conversion loops and unchecked obj.(*v1beta1.Machine) assertions while
preserving the existing test cases and expectations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/infra/aws/create_iam.go`:
- Around line 173-178: Update the InfraID handling in the IAM tag construction
flow to find and replace any existing additionalIAMTags entry with key
supportawsutil.HypershiftInfraIDTagKey using the enforced o.InfraID value. Only
append a new tag when no matching key exists, preserving unique tag keys for
CreateRole and TagRole.

In `@cmd/infra/aws/route53.go`:
- Around line 118-131: The TestCreatePrivateZone mock expectation for
ChangeTagsForResource must assert the complete Route53 tag set instead of
accepting gomock.Any(). Match the expected HyperShift and additional EC2 tag
keys and values, preserving the production tags assembled in r53Tags, and add or
update unit-test coverage for this assertion.

In `@cmd/infra/azure/networking.go`:
- Around line 315-321: The retry budget comments overstate the maximum wait
because retry.OnError sleeps only between attempts. In
cmd/infra/azure/networking.go lines 315-321, update the backoff comment to
approximately 75 seconds; in cmd/cluster/core/create.go lines 814-820, update
the corresponding comment to approximately 15 seconds. Keep the existing Steps
values unchanged.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml`:
- Around line 129-145: Add a resources.requests block to the wait-for-etcd init
container, using the matching small request values cpu: 5m and memory: 10Mi
consistent with audit-logs. Do not alter its existing command, environment,
image, or other settings.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go`:
- Around line 126-132: Update metricsProxyRouteRequired and the
aroExpectedHCPRouterRouteNames/ensureHCPRouterRoutesExist call chain to return
and propagate the endpointresolverv2.Predicate error, allowing routerPredicate
to return it and trigger reconciliation rather than treating failure as not
required. Pass the real cpContext into component.WorkloadContext instead of
constructing one with nil Client and Context, while preserving the existing
route requirement checks.

In
`@control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go`:
- Around line 337-340: Update the decode error handling after
secretencryption.DecodeEncryptionConfiguration in the reencryption function to
wrap the returned error with context identifying the encryption configuration
that failed to decode, while preserving the existing TargetKeyAbsent and
configHash return values.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go`:
- Around line 713-723: Update the expectedFinalizersRemoved branch in the Azure
machine test to assert directly on machine-1 rather than relying on iterating
azureMachineList.Items. Treat the expected outcome as either machine-1 being
absent from the list or, if present, having empty finalizers; leave the
non-removal assertion unchanged.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go`:
- Around line 371-375: Update Azure.DeleteOrphanedMachines to return nil when
hc.Spec.Platform.Azure is nil before accessing
AzureAuthenticationConfig.ManagedIdentities, while preserving the existing
managed-identities guard. Add a test covering cleanup with a nil Azure platform
spec and verify it completes without error or panic.

In `@hypershift-operator/controllers/nodepool/version.go`:
- Around line 96-118: Update rhcosOSImageRe and rhcosStreamFromOSImage to
recognize RHEL-based OSImage versions in both 9.x and 10.x forms, such as
“CoreOS 9.6...” and “CoreOS 10.6...”, while preserving existing 4xx-to-RHEL9 and
5xx-to-RHEL10 mappings; return StreamRHEL9 or StreamRHEL10 for the corresponding
formats and keep unrecognized versions returning an empty string.

In `@test/e2e/nodepool_spot_termination_handler_test.go`:
- Around line 150-155: Add the SQS-specific red-hat=true authorization tag to
sqsTags before the CreateQueue call in the node pool termination handler test,
preserving the existing environment, infrastructure, and cluster tags so
NodePoolSQSActions can authorize queue access.

---

Nitpick comments:
In `@cmd/cluster/core/create_test.go`:
- Around line 1814-1822: The backoff used by validateClusterExistenceWithClient
is hardcoded, causing persistent-error tests to sleep in real time. Make its
wait.Backoff configurable—prefer passing it into
validateClusterExistenceWithClient or using an overridable package-level
value—and configure the relevant tests with a near-zero backoff while preserving
production defaults and retry behavior.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go`:
- Around line 105-124: Replace the duplicated route-name literals in
aroBaseHCPRouterRouteNames and aroExpectedHCPRouterRouteNames with the existing
route-name constants or manifest helpers, including the ignition, OAuth, and
metrics-proxy routes. Reuse the canonical symbols from the relevant packages so
route validation stays synchronized with any future renames.

In
`@control-plane-operator/hostedclusterconfigoperator/controllers/machine/machine.go`:
- Around line 138-146: Flatten the nested conditionals in the machine address
loop: combine each `parsedAddr.Is4()`/`len(...) == 0` guard into one condition,
and likewise combine the IPv6 guard, while preserving the existing append
behavior and family-specific slices.

In `@hack/tools/crd-schema-check/main.go`:
- Around line 58-60: Add regression coverage in main_test.go for the
repository-opening flow around gogit.PlainOpenWithOptions, using a repository
whose .git is a file pointing to a common Git directory. Verify the repository
opens and the CRD schema check behavior succeeds, while preserving existing
standard-repository test coverage.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go`:
- Around line 560-566: Update the test cases around the expectedError field so
the error path is actually exercised: add a case that injects a List or Update
failure through the interceptor client and asserts the expected error, or remove
expectedError and its related assertions if error coverage is not intended.
Apply the same change to the additional test table occurrence.

In `@hypershift-operator/controllers/nodepool/version_test.go`:
- Around line 283-291: Update both affected tests to declare their machine
fixtures as []*v1beta1.Machine (or the helper’s expected machine type) instead
of []client.Object, then pass those fixtures directly to setNodesInfoStatus.
Remove the conversion loops and unchecked obj.(*v1beta1.Machine) assertions
while preserving the existing test cases and expectations.

In
`@hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller_test.go`:
- Around line 387-435: Consolidate the TestLookupForwardingRule_* cases and the
TestDiscoverNATSubnet_* cases into one table-driven test per function, using
subtest names in “When ... it should ...” format. Preserve each scenario’s fake
inputs, expected result, error handling, and forwarding-rule filter assertions
while removing duplicated test setup and checks.

In
`@hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller.go`:
- Around line 313-333: Update the subnet-selection flow around isSubnetInUse to
list the project’s Service Attachments once before iterating over subnets, then
determine each candidate’s usage from that in-memory result. Remove the
per-candidate isSubnetInUse API calls while preserving the existing
available-subnet selection, error logging, and no-available-subnet behavior.

In `@support/controlplane-component/status.go`:
- Around line 39-40: Add focused unit tests in the status test suite covering
the dependency exclusions for both “ignition-server” and
“ignition-server-proxy”, asserting they omit kube-apiserver while existing
components still include it. Reuse the existing status/dependency test helpers
and preserve current reconciliation behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: b334061b-bbd9-49a5-905b-60b2092e6526

📥 Commits

Reviewing files that changed from the base of the PR and between 4237039 and 50d87f2.

⛔ Files ignored due to path filters (26)
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_When_endpoint_access_is_Private_with_endpoint_access_private_flags_it_should_render_HostedCluster_with_Private_endpoint_access.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_When_oauth_publishing_strategy_is_LoadBalancer_with_workload_identities_it_should_render_HostedCluster_with_OAuth_LoadBalancer.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_complicated_invocation_from_bryan.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_minimal_flags_necessary_to_render.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_with_KubeAPIServerDNSName.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_with_availability_zones.yaml is excluded by !**/testdata/**
  • cmd/cluster/azure/testdata/zz_fixture_TestCreateCluster_with_disabled_capabilities.yaml is excluded by !**/testdata/**
  • cmd/infra/aws/delegating_client.go is excluded by !cmd/infra/aws/delegating_client.go
  • control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/AROSwift/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/GCP/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/IBMCloud/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/cluster-version-operator/zz_fixture_TestControlPlaneComponents_cluster_version_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/AROSwift/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/AROSwift/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/GCP/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/IBMCloud/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/openshift-oauth-apiserver/zz_fixture_TestControlPlaneComponents_openshift_oauth_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/router/AROSwift/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/router/GCP/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/router/IBMCloud/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/router/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/router/zz_fixture_TestControlPlaneComponents_router_controlplanecomponent.yaml is excluded by !**/testdata/**
  • docs/content/reference/aggregated-docs.md is excluded by !docs/content/reference/aggregated-docs.md
  • vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/nodepool_types.go is excluded by !vendor/**, !**/vendor/**
📒 Files selected for processing (100)
  • .pre-commit-config.yaml
  • Makefile
  • OWNERS_ALIASES
  • api/hypershift/v1beta1/nodepool_types.go
  • cmd/bastion/aws/create.go
  • cmd/cluster/aws/create.go
  • cmd/cluster/aws/create_test.go
  • cmd/cluster/azure/create_test.go
  • cmd/cluster/azure/manifests.go
  • cmd/cluster/core/create.go
  • cmd/cluster/core/create_test.go
  • cmd/cluster/gcp/create.go
  • cmd/cluster/gcp/create_test.go
  • cmd/fix/dr_oidc_iam.go
  • cmd/infra/aws/create.go
  • cmd/infra/aws/create_cli_role.go
  • cmd/infra/aws/create_iam.go
  • cmd/infra/aws/delegatingclientgenerator/main.go
  • cmd/infra/aws/ec2.go
  • cmd/infra/aws/iam.go
  • cmd/infra/aws/iam_test.go
  • cmd/infra/aws/route53.go
  • cmd/infra/aws/route53_test.go
  • cmd/infra/azure/networking.go
  • cmd/infra/azure/networking_test.go
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
  • control-plane-operator/controllers/hostedcontrolplane/manifests/cno.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/cno/component.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/cvo/deployment.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/cvo/deployment_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/deployment.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/secretencryption.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/oauth_apiserver/deployment.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/router/component_test.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/machine/machine.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/machine/machine_test.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption_test.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/resources/ingress/params.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/resources/ingress/params_test.go
  • control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources.go
  • docs/content/reference/infrastructure/aws.md
  • hack/tools/crd-schema-check/main.go
  • hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go
  • hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go
  • hypershift-operator/controllers/nodepool/aws.go
  • hypershift-operator/controllers/nodepool/aws_test.go
  • hypershift-operator/controllers/nodepool/azure.go
  • hypershift-operator/controllers/nodepool/azure_test.go
  • hypershift-operator/controllers/nodepool/conditions.go
  • hypershift-operator/controllers/nodepool/config.go
  • hypershift-operator/controllers/nodepool/config_test.go
  • hypershift-operator/controllers/nodepool/gcp.go
  • hypershift-operator/controllers/nodepool/gcp_test.go
  • hypershift-operator/controllers/nodepool/kubevirt.go
  • hypershift-operator/controllers/nodepool/kubevirt/kubevirt.go
  • hypershift-operator/controllers/nodepool/kubevirt/kubevirt_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go
  • hypershift-operator/controllers/nodepool/nodepool_controller_test.go
  • hypershift-operator/controllers/nodepool/openstack.go
  • hypershift-operator/controllers/nodepool/openstack/openstack.go
  • hypershift-operator/controllers/nodepool/openstack/openstack_test.go
  • hypershift-operator/controllers/nodepool/osstream.go
  • hypershift-operator/controllers/nodepool/osstream_test.go
  • hypershift-operator/controllers/nodepool/powervs.go
  • hypershift-operator/controllers/nodepool/powervs_test.go
  • hypershift-operator/controllers/nodepool/stream.go
  • hypershift-operator/controllers/nodepool/stream_test.go
  • hypershift-operator/controllers/nodepool/stream_test_helpers_test.go
  • hypershift-operator/controllers/nodepool/token.go
  • hypershift-operator/controllers/nodepool/token_test.go
  • hypershift-operator/controllers/nodepool/version.go
  • hypershift-operator/controllers/nodepool/version_test.go
  • hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller.go
  • hypershift-operator/controllers/platform/gcp/privateserviceconnect_controller_test.go
  • product-cli/main.go
  • product-cli/main_test.go
  • support/awsapi/elasticloadbalancingv2.go
  • support/awsapi/route53.go
  • support/awsutil/tags.go
  • support/controlplane-component/defaults.go
  • support/controlplane-component/defaults_test.go
  • support/controlplane-component/status.go
  • support/releaseinfo/releaseinfo.go
  • support/releaseinfo/releaseinfo_test.go
  • support/secretencryption/encryptionconfig.go
  • support/secretencryption/encryptionconfig_test.go
  • test/e2e/external_oidc_test.go
  • test/e2e/karpenter_control_plane_upgrade_test.go
  • test/e2e/karpenter_test.go
  • test/e2e/nodepool_spot_termination_handler_test.go
  • test/e2e/util/aws.go
  • test/e2e/util/fixture.go
  • test/e2e/util/globalps.go
  • test/e2e/util/util_metrics_proxy.go
  • test/e2e/v2/cmd/create-guests/main.go
  • test/e2e/v2/tests/hosted_cluster_psc_test.go
💤 Files with no reviewable changes (2)
  • control-plane-operator/controllers/hostedcontrolplane/v2/cno/component.go
  • control-plane-operator/controllers/hostedcontrolplane/manifests/cno.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

🛑 Comments failed to post (10)
cmd/infra/aws/create_iam.go (1)

173-178: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file exists and relevant lines =="
if [ -f cmd/infra/aws/create_iam.go ]; then
  nl -ba cmd/infra/aws/create_iam.go | sed -n '130,205p'
else
  echo "missing cmd/infra/aws/create_iam.go"
  git ls-files | rg '(^|/)create_iam\.go$' || true
fi

echo
echo "== search for additionalIAMTags and additional tags parsing =="
rg -n "additionalIAMTags|additional-tags|HyperShift|HypershiftInfraIDTagKey|infra-id|Tags:" cmd/infra/aws . -g '*.go' | head -n 200

echo
echo "== AST outline if available =="
ast-grep outline cmd/infra/aws/create_iam.go --view compact || true

Repository: openshift/hypershift

Length of output: 235


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file exists and relevant lines =="
if [ -f cmd/infra/aws/create_iam.go ]; then
  sed -n '1,260p' cmd/infra/aws/create_iam.go
else
  echo "missing cmd/infra/aws/create_iam.go"
  git ls-files | grep -E '(^|/)create_iam\.go$' || true
fi

echo
echo "== search for additionalIAMTags, additional-tags, InfraID tag references =="
grep -RInE "additionalIAMTags|additional-tags|HypershiftInfraIDTagKey|Hyper.*(Infra|INFRA|infra).*Tag|infra-id|Tags:" --include='*.go' --include='*.sh' . | head -n 240

echo
echo "== repository tooling/dependencies hints =="
ls -la | sed -n '1,80p'

Repository: openshift/hypershift

Length of output: 40089


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== remaining create_iam.go from parse method through IAM tag usage =="
sed -n '240,800p' cmd/infra/aws/create_iam.go

echo
echo "== cmd/util ParseAWSTags implementation/candidates =="
grep -RInE "func ParseAWSTags|ParseAWSTags|AdditionalTags|Tag:" --include='*.go' cmd util support | head -n 240

echo
echo "== behavioral probe: deterministic ordering of parsed entries and duplicate append possibility =="
cat cmd/infra/aws/create_iam.go > /tmp/create_iam.go
grep -n "ParseAWSTags\\|additionalIAMTags\\|HypershiftInfraIDTagKey\\|CreateKarpenterRoleARN\\|Create.*Role\\|Tag\\[|Tags" /tmp/create_iam.go

Repository: openshift/hypershift

Length of output: 7437


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== create_cli_role IAM tag helper =="
sed -n '220,280p' cmd/infra/aws/create_cli_role.go

echo
echo "== operator roles additional tag parsing and InfraID handling =="
sed -n '100,150p' cmd/infra/aws/create_operator_roles.go
sed -n '300,345p' cmd/infra/aws/create_operator_roles.go
grep -n "HypershiftInfraIDTagKey|additionalIAMTags|Tags:" cmd/infra/aws/create_operator_roles.go

echo
echo "== remaining create_iam.go IAM tag references =="
grep -n "additionalIAMTags|Tags\\s*:|TagSpecifications|PutRole|CreateRole|CreateKarpenterRole\\|CreateWorkerInstanceProfile" cmd/infra/aws/create_iam.go

echo
echo "== awsutil constants =="
grep -RInE "HypershiftInfraIDTagKey\\s*=|HypershiftSourceTagKey\\s*=" --include='*.go' support cmd/hypershift support | head -n 50

echo
echo "== read AWS SDK Go v2 docs from repository vendored package if present =="
python3 - <<'PY'
from pathlib import Path
roots = [
    Path('go.mod'),
    Path('cmd/util/client.go'),
    Path('cmd/infra/aws/create_cli_role.go'),
    Path('cmd/infra/aws/create_iam.go'),
]
for p in roots:
    if p.exists():
        print(f"exists: {p}")
PY

Repository: openshift/hypershift

Length of output: 4219


🌐 Web query:

AWS IAM CreateRole requires tags array duplicate tag keys behavior

💡 Result:

When using the AWS IAM CreateRole API, providing duplicate tag keys in the Tags array will cause the request to fail [1]. Specifically, IAM enforces uniqueness for tag keys in a single request, and the presence of duplicate keys will typically result in an InvalidInput error [1]. It is important to note that IAM tag keys are case-insensitive [1][2]. Therefore, if you attempt to provide two tags that differ only by case (e.g., "Department" and "department"), the API will treat them as duplicates, and the request will be rejected [1][2]. To ensure successful creation of the role, verify that all tag keys provided in the request are unique, regardless of their casing [1]. If any tag in the request is invalid or violates these constraints, the entire request will fail, and no resource will be created [3][4][5].

Citations:


Avoid duplicate enforced IAM tag keys.

A --additional-tags hypershift.openshift.io/infra-id=... entry is parsed into additionalIAMTags first, then the enforced InfraID tag is appended again. IAM does not accept duplicate tag keys in CreateRole/TagRole; replace any already-present InfraID tag with the enforced value before appends instead of appending it unconditionally.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/infra/aws/create_iam.go` around lines 173 - 178, Update the InfraID
handling in the IAM tag construction flow to find and replace any existing
additionalIAMTags entry with key supportawsutil.HypershiftInfraIDTagKey using
the enforced o.InfraID value. Only append a new tag when no matching key exists,
preserving unique tag keys for CreateRole and TagRole.
cmd/infra/aws/route53.go (1)

118-131: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert the Route53 tag set.

TestCreatePrivateZone accepts ChangeTagsForResource with gomock.Any(), so wrong or missing HyperShift tags would pass. Match the expected tag keys and values.

As per coding guidelines, “Unit test any code changes and additions.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/infra/aws/route53.go` around lines 118 - 131, The TestCreatePrivateZone
mock expectation for ChangeTagsForResource must assert the complete Route53 tag
set instead of accepting gomock.Any(). Match the expected HyperShift and
additional EC2 tag keys and values, preserving the production tags assembled in
r53Tags, and add or update unit-test coverage for this assertion.

Source: Coding guidelines

cmd/infra/azure/networking.go (1)

315-321: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Stated retry budgets overstate the actual wait. With Steps: 5, retry.OnError makes 5 attempts and therefore sleeps only 4 times, so the last term in each comment's sum never occurs.

  • cmd/infra/azure/networking.go#L315-L321: max wait is ~75s (5+10+20+40), not ~155s — update the comment or bump Steps to 6 if 155s was intended.
  • cmd/cluster/core/create.go#L814-L820: max wait is ~15s (1+2+4+8), not ~31s — same correction applies.
📍 Affects 2 files
  • cmd/infra/azure/networking.go#L315-L321 (this comment)
  • cmd/cluster/core/create.go#L814-L820
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/infra/azure/networking.go` around lines 315 - 321, The retry budget
comments overstate the maximum wait because retry.OnError sleeps only between
attempts. In cmd/infra/azure/networking.go lines 315-321, update the backoff
comment to approximately 75 seconds; in cmd/cluster/core/create.go lines
814-820, update the corresponding comment to approximately 15 seconds. Keep the
existing Steps values unchanged.
control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml (1)

129-145: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Compare with the KAS asset's wait-for-etcd init container and check whether the framework defaults init-container resources.
fd -t f 'deployment.yaml' control-plane-operator/controllers/hostedcontrolplane/v2/assets --exec rg -l 'wait-for-etcd' {} \
  | xargs -r rg -n -A20 'name: wait-for-etcd|- args:' 
rg -nP -C5 'InitContainers' support/controlplane-component/defaults.go

Repository: openshift/hypershift

Length of output: 30965


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== defaults.go relevant section =="
rg -C 12 'NeedDefaultInitContainerResourcesAnnotation|InitContainerResources|apply.*Resource|InitContainers|requestLimit' support/controlplane-component/defaults.go || true

echo
echo "== v2 deployment resource blocks =="
python3 - <<'PY'
from pathlib import Path
for path in [
    Path('control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml'),
    Path('control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-apiserver/deployment.yaml'),
]:
    print(f'\n--- {path} ---')
    text = path.read_text()
    lines = text.splitlines()
    in_first = False
    for idx,l in enumerate(lines, 1):
        if 'name:' in l and ('audit-logs:' in l or 'wait-for-etcd:' in l or 'openshift-oauth:' in l or 'kube-apiserver:' in l or 'konnectivity:' in l):
            start=max(1, idx-3); end=min(len(lines), idx+45)
            print(f'\ncontainer around line {idx}:')
            for no in range(start,end+1):
                if 'resources:' in lines[no-1]:
                    print('--- resource block ---')
                print(f'{no}: {lines[no-1]}')
PY

Repository: openshift/hypershift

Length of output: 5799


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== defaults.go key definitions and assignments =="
rg -n -C 12 'ResourceRequestOverrideAnnotationPrefix|NeedDefaultInitContainerResourcesAnnotation|NeedDefaultResourcesAnnotation|InitContainers\[' support/controlplane-component/defaults.go

echo
echo "== openshift-oauth-apiserver deployment relevant lines =="
sed -n '1,170p' control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml

echo
echo "== kube-apiserver deployment relevant lines =="
sed -n '220,340p' control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-apiserver/deployment.yaml

Repository: openshift/hypershift

Length of output: 11454


Add resource requests to wait-for-etcd.

openshift-oauth-apiserver already declares requests on its explicit containers, but this new init container has no resources block. Resource requests affect the pod’s effective request calculation, so add a small matching request set, e.g. cpu: 5m / memory: 10Mi like audit-logs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/assets/openshift-oauth-apiserver/deployment.yaml`
around lines 129 - 145, Add a resources.requests block to the wait-for-etcd init
container, using the matching small request values cpu: 5m and memory: 10Mi
consistent with audit-logs. Do not alter its existing command, environment,
image, or other settings.
control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go (1)

126-132: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Don't swallow the predicate error, and be wary of the synthetic WorkloadContext.

Two concerns here:

  1. err from endpointresolverv2.Predicate is discarded and treated as "not required". A transient failure then silently drops metrics-proxy from the expected route set, so the router can be admitted before its metrics-proxy route exists — the exact condition this gate is meant to prevent. Propagate the error up through aroExpectedHCPRouterRouteNames/ensureHCPRouterRoutesExist so routerPredicate returns it and the reconcile is retried.
  2. component.WorkloadContext{HCP: hcp} has a nil Client and nil Context. This works only as long as endpointresolverv2.Predicate touches nothing but HCP; any future read of Client becomes a nil dereference here. Passing the real cpContext through would be safer.

As per coding guidelines: "Always check errors — don't ignore them."

🛠️ Sketch of error propagation
-func metricsProxyRouteRequired(hcp *hyperv1.HostedControlPlane) bool {
-	enabled, err := endpointresolverv2.Predicate(component.WorkloadContext{HCP: hcp})
-	if err != nil || !enabled {
-		return false
-	}
-	return netutil.LabelHCPRoutes(hcp) || netutil.IsPrivateHCP(hcp)
-}
+func metricsProxyRouteRequired(cpContext component.WorkloadContext) (bool, error) {
+	enabled, err := endpointresolverv2.Predicate(cpContext)
+	if err != nil {
+		return false, fmt.Errorf("failed to evaluate endpoint resolver predicate: %w", err)
+	}
+	if !enabled {
+		return false, nil
+	}
+	hcp := cpContext.HCP
+	return netutil.LabelHCPRoutes(hcp) || netutil.IsPrivateHCP(hcp), nil
+}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@control-plane-operator/controllers/hostedcontrolplane/v2/router/component.go`
around lines 126 - 132, Update metricsProxyRouteRequired and the
aroExpectedHCPRouterRouteNames/ensureHCPRouterRoutesExist call chain to return
and propagate the endpointresolverv2.Predicate error, allowing routerPredicate
to return it and trigger reconciliation rather than treating failure as not
required. Pass the real cpContext into component.WorkloadContext instead of
constructing one with nil Client and Context, while preserving the existing
route requirement checks.

Source: Coding guidelines

control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go (1)

337-340: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Wrap the decode error with context.

Every other error return in this function is wrapped; the decode path returns the raw error, so callers lose the "which config failed to decode" context. As per coding guidelines, "Wrap errors with context when rethrowing."

🛠️ Proposed fix
 	currentConfig, err := secretencryption.DecodeEncryptionConfiguration(configBytes)
 	if err != nil {
-		return secretencryption.TargetKeyAbsent, configHash, err
+		return secretencryption.TargetKeyAbsent, configHash, fmt.Errorf("failed to decode encryption config secret %s/%s: %w", secretKey.Namespace, secretKey.Name, err)
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

	currentConfig, err := secretencryption.DecodeEncryptionConfiguration(configBytes)
	if err != nil {
		return secretencryption.TargetKeyAbsent, configHash, fmt.Errorf("failed to decode encryption config secret %s/%s: %w", secretKey.Namespace, secretKey.Name, err)
	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/hostedclusterconfigoperator/controllers/reencryption/reencryption.go`
around lines 337 - 340, Update the decode error handling after
secretencryption.DecodeEncryptionConfiguration in the reencryption function to
wrap the returned error with context identifying the encryption configuration
that failed to decode, while preserving the existing TargetKeyAbsent and
configHash return values.

Source: Coding guidelines

hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go (1)

713-723: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm fake-client behavior: does removing the last finalizer on a deleting object delete it?
fd -t f 'client.go' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake --exec rg -n -C8 'deleteObject|Finalizers) == 0|DeletionTimestamp' {}

Repository: openshift/hypershift

Length of output: 315


🏁 Script executed:

#!/bin/bash
set -e

printf 'Files matching azure_test.go:\n'
fd -t f 'azure_test\.go$' . | sed -n '1,20p'

printf '\nRelevant section:\n'
f=$(fd -t f 'azure_test\.go$' . | grep 'controllers/hostedcluster/internal/platform/azure/azure_test.go' | head -n1)
if [ -n "$f" ]; then
  sed -n '630,735p' "$f" | nl -ba -v630
fi

printf '\nSearch fake client handling in vendored dependency/path metadata:\n'
rg -n 'controller-runtime|sigs.k8s.io/controller-runtime' go.mod go.sum 2>/dev/null | sed -n '1,40p' || true

Repository: openshift/hypershift

Length of output: 641


🏁 Script executed:

#!/bin/bash
set -e
# Try deterministic behavioral evidence from source code: find relevant fake client implementation
# without assuming vendor layout.
printf 'Candidate controller-runtime fake client files:\n'
find . -path '*/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go' -o -path '*/client/fake/client.go' | sed -n '1,20p'

for f in $(find . -path '*/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go' -o -path '*/client/fake/client.go' 2>/dev/null | sed -n '1,5p'); do
  echo "--- $f"
  rg -n -C8 'Finalizers) == 0|DeletionTimestamp|deleteObject|Delete\(' "$f" | sed -n '1,160p'
done

Repository: openshift/hypershift

Length of output: 517


🏁 Script executed:

#!/bin/bash
set -e

printf 'Relevant test section:\n'
f='hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go'
awk 'NR>=630 && NR<=735 { printf "%6d\t%s\n", NR, $0 }' "$f"

printf '\nController-runtime fake client relevant implementation:\n'
rg -n -C8 'Finalizers) == 0|DeletionTimestamp|delet|deleteObject|Delete\(' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go | sed -n '1,220p'

Repository: openshift/hypershift

Length of output: 3927


🏁 Script executed:

#!/bin/bash
set -e

printf 'Test cases around AzureMachine list construction and expectedFinalizersRemoved cases:\n'
f='hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go'
awk 'NR>=540 && NR<=685 { printf "%6d\t%s\n", NR, $0 }' "$f"

printf '\nController-runtime fake client Delete implementation:\n'
sed -n '1,260p' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go | awk '/func \(.*Delete\(|Finalizers|deletion|deleteObject|Delete\(/ { p=1 } p { print } /func \(.*Update\(/ { if (p && NR>first) exit; p=0 } /^func \(.*Update\(/ { first=NR }' || true

printf '\nAll Finalizers / DeletionTimestamp handling in fake client:\n'
grep -n -E 'Finalizers|DeletionTimestamp|deleteObject|Delete\(' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go | sed -n '1,120p'

Repository: openshift/hypershift

Length of output: 7131


🏁 Script executed:

#!/bin/bash
set -e

awk 'NR>=570 && NR<=665 { printf "%6d\t%s\n", NR, $0 }' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go
printf '\n--- Update behavior around deletion timestamp finalizer block ---\n'
awk 'NR>=870 && NR<=930 { printf "%6d\t%s\n", NR, $0 }' vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go

Repository: openshift/hypershift

Length of output: 6364


Make the finalizer-removal assertion non-vacuous.

In the stale-deletion path, the fake client deletes the object once its finalizers are removed while it still has a DeletionTimestamp, so List may return no machines and this for loop can pass without checking machine-1’s finalizer state. Assert on machine-1 directly: either the object is absent or it has empty finalizers.

Suggested tightening
 			if tc.expectedFinalizersRemoved {
-				for _, machine := range azureMachineList.Items {
-					if !machine.DeletionTimestamp.IsZero() {
-						g.Expect(machine.Finalizers).to(BeEmpty(), "finalizers should be removed for machines with DeletionFailed condition")
-					}
-				}
+				machine := &capiazure.AzureMachine{}
+				err := fakeClient.Get(ctx, client.ObjectKey{Namespace: controlPlaneNamespace, Name: "machine-1"}, machine)
+				if err == nil {
+					g.Expect(machine.Finalizers).To(BeEmpty(), "finalizers should be removed for machines with DeletionFailed condition")
+				} else {
+					g.Expect(apierrors.IsNotFound(err)).To(BeTrue(), "machine should be gone once its last finalizer is removed")
+				}
 			} else {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

			if tc.expectedFinalizersRemoved {
				machine := &capiazure.AzureMachine{}
				err := fakeClient.Get(ctx, client.ObjectKey{Namespace: controlPlaneNamespace, Name: "machine-1"}, machine)
				if err == nil {
					g.Expect(machine.Finalizers).To(BeEmpty(), "finalizers should be removed for machines with DeletionFailed condition")
				} else {
					g.Expect(apierrors.IsNotFound(err)).To(BeTrue(), "machine should be gone once its last finalizer is removed")
				}
			} else {
				for _, machine := range azureMachineList.Items {
					g.Expect(machine.Finalizers).To(Equal(tc.azureMachines[0].Finalizers), "finalizers should not be modified")
				}
			}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure_test.go`
around lines 713 - 723, Update the expectedFinalizersRemoved branch in the Azure
machine test to assert directly on machine-1 rather than relying on iterating
azureMachineList.Items. Treat the expected outcome as either machine-1 being
absent from the list or, if present, having empty finalizers; leave the
non-removal assertion unchanged.
hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go (1)

371-375: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find callers of DeleteOrphanedMachines and check whether they guarantee a non-nil Azure platform spec.
rg -nP -C6 '\bDeleteOrphanedMachines\s*\(' --type=go

Repository: openshift/hypershift

Length of output: 158


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate azure.go =="
fd -a 'azure\.go$' . | sed 's#^\./##'

echo "== locate references =="
rg -n "DeleteOrphanedMachines|AzureAuthenticationConfig|Platform\.Azure|Spec\.Platform|Azure" --type=go -g '*.go' | head -n 200

echo "== target file outline =="
file="$(fd 'azure\.go$' . | grep 'platform/azure/azure.go' | head -n 1)"
[ -n "$file" ] && ast-grep outline "$file" || true

echo "== target lines =="
[ -n "$file" ] && sed -n '340,405p' "$file"

Repository: openshift/hypershift

Length of output: 1785


🏁 Script executed:

#!/bin/bash
set -euo pipefail
file=$(fd 'azure\.go$' . | grep 'hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go' | head -n 1)
echo "== target file: $file"
echo "== target lines 340-405 =="
sed -n '340,405p' "$file"
echo "== all exact references in checked-out tree =="
rg -n "DeleteOrphanedMachines|AzureAuthenticationConfig|Spec\.Platform\.Azure\.AzureAuthenticationConfig|Platform\.Azure" -g '*.go'

Repository: openshift/hypershift

Length of output: 3215


🌐 Web query:

hypershift-operator controllers hostedcluster internal platform azure DeleteOrphanedMachines calls

💡 Result:

The DeleteOrphanedMachines function within the HyperShift operator is a mechanism designed to perform emergency cleanup of machine resources (such as AWSMachine or AzureMachine objects) that become stuck or orphaned when the cluster's cloud identity provider or credentials are determined to be invalid [1][2][3]. When the HostedCluster controller identifies that the cloud provider credentials or identity configurations are invalid (based on status conditions like ValidAWSIdentityProvider or similar checks for Azure), it invokes DeleteOrphanedMachines to prevent these orphaned resources from blocking further reconciliation or deletion processes [1][3]. For the Azure platform, the DeleteOrphanedMachines function performs the following operations [3]: 1. Credential Validation: It first checks if the cluster credentials/managed identities are valid. If they are valid, it returns early and takes no action [3]. 2. Resource Listing: If credentials are invalid, it lists all capiav2 AzureMachine resources located within the HostedCluster's control plane namespace [3]. 3. Cleanup: For any machine found with a non-zero DeletionTimestamp (indicating it is already in a state of deletion but stuck), the function removes the finalizers associated with that resource to allow the Kubernetes garbage collector to finalize the deletion [3]. 4. Error Aggregation: It aggregates and returns any errors encountered during the update of these machine resources [3]. Similar logic exists for the AWS platform, where the function scans for AWSMachine objects and clears finalizers if the AWS identity provider is deemed invalid [1][2][4]. This process ensures that infrastructure resources that cannot be properly deleted via standard controller reconciliation—due to the inability to communicate with the cloud provider—are cleaned up from the control plane [1][3].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "== git status/diff stat =="
git status --short
git diff --stat
echo "== tracked files matching azure.go =="
git ls-files | grep -E 'hypershift-operator/controllers/hostedcluster/internal/platform/.*/azure\.go|hypershift-operator/controllers/hostedcluster/.*/azure\.go' || true
echo "== exact strings in tracked Go files =="
rg -n "DeleteOrphanedMachines|AzureAuthenticationConfig|Spec\.Platform\.Azure" --glob '*.go' . || true
echo "== azure.go around relevant export/calls =="
rg -n "DeleteCredentials|Reconcile.*Azure|NewAzure|func .*\b[A-Z]" hypershift-operator/controllers/hostedcluster -g '*.go' || true

Repository: openshift/hypershift

Length of output: 50377


🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "== azure.go azureutil nil guard around Azure spec =="
sed -n '190,235p' hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go
echo "== hostedcluster controller relevant guard before DeleteOrphanedMachines =="
sed -n '3765,3810p' hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
echo "== azureutil IsSelfManagedAzureWithWorkloadIdentity implementation =="
sed -n '1,90p' support/azureutil/azureutil.go

Repository: openshift/hypershift

Length of output: 6841


🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "== platform GetPlatform implementation =="
rg -n "func GetPlatform|type OrphanDeleter|type.*Azure|func .*OrphanDeleter" hypershift-operator/controllers/hostedcluster --glob '*.go' -C 4
echo "== all IsSelfManagedAzureWithWorkloadIdentity callers within platform/azure =="
rg -n "func .*Azure|IsSelfManagedAzureWithWorkloadIdentity|AzureAuthenticationConfig\.ManagedIdentities|Platform\.Azure" hypershift-operator/controllers/hostedcluster/internal/platform/azure --glob '*.go' -C 3
echo "== exact implementation of IsSelfManagedAzureWithWorkloadIdentity in support/azureutil =="
rg -n "func IsSelfManagedAzureWithWorkloadIdentity" support/azureutil/azureutil.go -C 15

Repository: openshift/hypershift

Length of output: 30041


Guard against a nil Azure platform spec before dereferencing it.

DeleteOrphanedMachines is reached from the deletion path after platform.GetPlatform returns the Azure implementation, but the nil check is not enforced there and the tests only populate Azure. If an Azure HostedCluster reaches cleanup without a platform spec, hc.Spec.Platform.Azure.AzureAuthenticationConfig.ManagedIdentities panics. Add the nil guard at the top and add a test case for a nil Azure spec.

🛡️ Proposed guard
-	// This orphaning behavior is intended for managed-identity cleanup flow.
-	if hc.Spec.Platform.Azure.AzureAuthenticationConfig.ManagedIdentities == nil {
+	// This orphaning behavior is intended for managed-identity cleanup flow.
+	if hc.Spec.Platform.Azure == nil || hc.Spec.Platform.Azure.AzureAuthenticationConfig.ManagedIdentities == nil {
 		return nil
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

func (Azure) DeleteOrphanedMachines(ctx context.Context, c client.Client, hc *hyperv1.HostedCluster, controlPlaneNamespace string) error {
	// This orphaning behavior is intended for managed-identity cleanup flow.
	if hc.Spec.Platform.Azure == nil || hc.Spec.Platform.Azure.AzureAuthenticationConfig.ManagedIdentities == nil {
		return nil
	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@hypershift-operator/controllers/hostedcluster/internal/platform/azure/azure.go`
around lines 371 - 375, Update Azure.DeleteOrphanedMachines to return nil when
hc.Spec.Platform.Azure is nil before accessing
AzureAuthenticationConfig.ManagedIdentities, while preserving the existing
managed-identities guard. Add a test covering cleanup with a nil Azure platform
spec and verify it completes without error or panic.
hypershift-operator/controllers/nodepool/version.go (1)

96-118: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Look for other places in the repo that parse RHCOS OSImage strings / known samples
rg -nP --type=go -C3 'Red Hat Enterprise Linux CoreOS' | head -50

Repository: openshift/hypershift

Length of output: 158


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file excerpt =="
wc -l hypershift-operator/controllers/nodepool/version.go
sed -n '1,180p' hypershift-operator/controllers/nodepool/version.go

echo
echo "== relevant usages/tests =="
rg -n -C3 'rhcosOSImageRe|rhcosStreamFromOSImage|OSImage|StreamRHEL|RHCOS|RHEL9|RHEL10' hypershift-operator -g '*.go' || true

Repository: openshift/hypershift

Length of output: 50376


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import re

rhcosOSImageRe = re.compile(r'Red Hat Enterprise Linux CoreOS (\d)\d{2}\.')
samples = [
    "Red Hat Enterprise Linux CoreOS 419.97.202505081234-0 (Plow)",
    "Red Hat Enterprise Linux CoreOS 510.97.202506011200-0 (Plow)",
    "Red Hat Enterprise Linux CoreOS 9.6.20250523-0",
    "Red Hat Enterprise Linux CoreOS 10.6.20250523-0",
    "Red Hat Enterprise Linux CoreOS 910.5.20250523-0",
    "Ubuntu 22.04 LTS",
]
for s in samples:
    matches = rhcosOSImageRe.findall(s)
    print(f"{s!r}: {matches!r}")
PY

Repository: openshift/hypershift

Length of output: 486


🌐 Web query:

Red Hat Enterprise Linux CoreOS 9.6 NodeInfo OSImage

💡 Result:

In Red Hat Enterprise Linux CoreOS (RHCOS), the.status.nodeInfo.osImage field is a standard Kubernetes Node API attribute that reports the operating system image information as defined in the node's /etc/os-release file [1][2]. For RHCOS nodes based on RHEL 9.6, the osImage string is typically represented in the format "Red Hat Enterprise Linux CoreOS 9.6.YYYYMMDD-0" [3][4]. Key characteristics regarding this field include: Identification: This field is the primary method for confirming the RHEL base version (e.g., 9.6, 9.8) and the specific RHCOS build/snapshot version currently running on a node [1]. Data Source: The string is derived from the /etc/os-release file, which is maintained and updated by the Machine Config Operator (MCO) during cluster updates [3][2]. Context in OpenShift: RHCOS versions are tightly coupled with OpenShift Container Platform (OCP) release versions [5]. For example, OCP 4.20 and 4.21 clusters have utilized RHEL 9.6-based RHCOS images [5]. Command to check: You can view this information for all nodes in a cluster using the following command: oc get node -o custom-columns=NAME:.metadata.name,OSIMAGE:.status.nodeInfo.osImage [6] The specific version string, such as 9.6.20260512-0, represents the RHEL minor release (9.6) followed by a timestamped build identifier [4]. As OpenShift clusters are updated, the MCO manages the transition of this image version across the cluster nodes [1][5].

Citations:


Handle RHEL-based NodeInfo.OSImage formats.

The current regex only supports three-digit RHCOS versions like 419.x/510.x, so RHEL-based strings such as Red Hat Enterprise Linux CoreOS 9.6.20250523-0 or 10.6.20250523-0 are ignored and stream inference may silently fail. Cover these two-component RHEL forms or otherwise define the supported OSImage formats.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/nodepool/version.go` around lines 96 - 118,
Update rhcosOSImageRe and rhcosStreamFromOSImage to recognize RHEL-based OSImage
versions in both 9.x and 10.x forms, such as “CoreOS 9.6...” and “CoreOS
10.6...”, while preserving existing 4xx-to-RHEL9 and 5xx-to-RHEL10 mappings;
return StreamRHEL9 or StreamRHEL10 for the corresponding formats and keep
unrecognized versions returning an empty string.
test/e2e/nodepool_spot_termination_handler_test.go (1)

150-155: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Restore the SQS authorization tag.

These provenance tags omit red-hat=true; the NodePool SQS policy conditions access on aws:ResourceTag/red-hat, so the handler can be denied queue access.

Proposed fix
 sqsTags := e2eutil.E2ETagsFromEnvironment()
+sqsTags["red-hat"] = "true"
 sqsTags[supportawsutil.HypershiftInfraIDTagKey] = s.hostedCluster.Spec.InfraID

Based on learnings, red-hat=true is intentionally SQS-specific and is required by NodePoolSQSActions.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

		sqsTags := e2eutil.E2ETagsFromEnvironment()
		sqsTags["red-hat"] = "true"
		sqsTags[supportawsutil.HypershiftInfraIDTagKey] = s.hostedCluster.Spec.InfraID
		sqsTags[supportawsutil.HypershiftClusterNameTagKey] = s.hostedCluster.Name
		createQueueResult, err := sqsClient.CreateQueue(s.ctx, &sqs.CreateQueueInput{
			QueueName: aws.String(sqsQueueName),
			Tags:      sqsTags,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/nodepool_spot_termination_handler_test.go` around lines 150 - 155,
Add the SQS-specific red-hat=true authorization tag to sqsTags before the
CreateQueue call in the node pool termination handler test, preserving the
existing environment, infrastructure, and cluster tags so NodePoolSQSActions can
authorize queue access.

Source: Learnings

@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@vsolanki12: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Indicates the PR includes changes for the API area/ci-tooling Indicates the PR includes changes for CI or tooling area/cli Indicates the PR includes changes for CLI area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/documentation Indicates the PR includes changes for documentation area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/platform/aws PR/issue for AWS (AWSPlatform) platform area/platform/azure PR/issue for Azure (AzurePlatform) platform area/platform/gcp PR/issue for GCP (GCPPlatform) platform area/platform/kubevirt PR/issue for KubeVirt (KubevirtPlatform) platform area/platform/openstack PR/issue for OpenStack (OpenStackPlatform) platform area/platform/powervs PR/issue for PowerVS (PowerVSPlatform) platform area/testing Indicates the PR includes changes for e2e testing jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants