Skip to content

OCPBUGS-99759: fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero - #9110

Open
judexzhu wants to merge 1 commit into
openshift:mainfrom
judexzhu:fix/scale-from-zero-annotation-keys
Open

OCPBUGS-99759: fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero#9110
judexzhu wants to merge 1 commit into
openshift:mainfrom
judexzhu:fix/scale-from-zero-annotation-keys

Conversation

@judexzhu

@judexzhu judexzhu commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi, which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys
  • The scale-from-zero workaround only wrote machine.openshift.io/* keys (Machine API provider namespace), which the CAPI provider silently ignores
  • Adds correct CAPI annotation keys alongside existing legacy keys so scale-from-zero capacity detection works

Design Decision: Co-existing Keys

Legacy machine.openshift.io/* keys are preserved alongside new capacity.cluster-autoscaler.kubernetes.io/* keys rather than replaced. Rationale:

  1. No breakage — if anything reads legacy keys (monitoring, dashboards, other controllers), it continues to work
  2. Smaller blast radius — additive change, no migration/cleanup logic needed
  3. Rollback safe — reverting this PR just removes the new keys, legacy keys remain functional (for Machine API provider consumers)

Future deprecation: Legacy machine.openshift.io/* keys should be removed in a follow-up once we confirm no consumers depend on them. The constants are already grouped and commented as "Legacy" to make this straightforward.

Related

  • OCPBUGS-99760: CAPA AWSMachineTemplate.Status.Capacity never populated (separate root cause — missing ownerReferences). Together with this bug, scale-from-zero had no working capacity signal.
  • Fixing either bug enables scale-from-zero. This PR fixes the annotation path; OCPBUGS-99760 tracks the native CAPI path.

Changes

File What
scale_from_zero.go Add capiCPUKey, capiMemoryKey, capiGPUCountKey constants; set both key sets; memory uses MiB suffix for CAPI; cleanup covers both
scale_from_zero_test.go Assert both legacy and CAPI keys in all test cases
autoscaling_test.go e2e verifies both CAPI and legacy annotation keys present

Test plan

  • Unit tests pass (870 tests, go test ./hypershift-operator/controllers/nodepool/...)
  • Lint clean (golangci-lint run)
  • e2e-aws-scale-from-zero (CI)

Bug: https://issues.redhat.com/browse/OCPBUGS-99759

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved scale-from-zero autoscaling by supporting additional Cluster Autoscaler capacity annotations, including CPU, memory, and GPU count/type.
    • Updated annotation handling so autoscaler-provided capacity correctly clears both legacy and newer capacity metadata.
    • Ensures accurate CPU/memory (and GPU details when applicable) annotations for CPU-only and GPU configurations across common cloud templates, including taint scenarios.
  • Tests
    • Expanded unit and end-to-end autoscaling coverage to validate the newer capacity annotation keys and expected behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: This pull request references Jira Issue OCPBUGS-99759, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi, which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys
  • The scale-from-zero workaround only wrote machine.openshift.io/* keys (Machine API provider namespace), which the CAPI provider silently ignores
  • Adds correct CAPI annotation keys alongside existing legacy keys so scale-from-zero capacity detection works

Design Decision: Co-existing Keys

Legacy machine.openshift.io/* keys are preserved alongside new capacity.cluster-autoscaler.kubernetes.io/* keys rather than replaced. Rationale:

  1. No breakage — if anything reads legacy keys (monitoring, dashboards, other controllers), it continues to work
  2. Smaller blast radius — additive change, no migration/cleanup logic needed
  3. Rollback safe — reverting this PR just removes the new keys, legacy keys remain functional (for Machine API provider consumers)

Future deprecation: Legacy machine.openshift.io/* keys should be removed in a follow-up once we confirm no consumers depend on them. The constants are already grouped and commented as "Legacy" to make this straightforward.

Related

  • OCPBUGS-99760: CAPA AWSMachineTemplate.Status.Capacity never populated (separate root cause — missing ownerReferences). Together with this bug, scale-from-zero had no working capacity signal.
  • Fixing either bug enables scale-from-zero. This PR fixes the annotation path; OCPBUGS-99760 tracks the native CAPI path.

Changes

File What
scale_from_zero.go Add capiCPUKey, capiMemoryKey, capiGPUCountKey constants; set both key sets; memory uses MiB suffix for CAPI; cleanup covers both
scale_from_zero_test.go Assert both legacy and CAPI keys in all test cases
autoscaling_test.go e2e verifies both CAPI and legacy annotation keys present

Test plan

  • Unit tests pass (870 tests, go test ./hypershift-operator/controllers/nodepool/...)
  • Lint clean (golangci-lint run)
  • e2e-aws-scale-from-zero (CI)

Bug: https://issues.redhat.com/browse/OCPBUGS-99759

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a02c9045-adc9-4fca-916a-b5e59ad1e92e

📥 Commits

Reviewing files that changed from the base of the PR and between 17c88e9 and 5a6f6f0.

📒 Files selected for processing (3)
  • hypershift-operator/controllers/nodepool/scale_from_zero.go
  • hypershift-operator/controllers/nodepool/scale_from_zero_test.go
  • test/e2e/autoscaling_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • hypershift-operator/controllers/nodepool/scale_from_zero_test.go
  • hypershift-operator/controllers/nodepool/scale_from_zero.go

📝 Walkthrough

Walkthrough

The scale-from-zero workaround now manages Cluster Autoscaler CPU, memory, GPU-count, and GPU-type annotations alongside legacy machine annotations. It removes both formats when native capacity exists and populates or clears both when capacity is inferred. Unit and end-to-end tests cover AWS and Azure CPU-only and GPU scenarios.

Suggested reviewers: enxebre, nirshal, jhjaggars

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding CAPI autoscaler capacity annotations for nodepool scale-from-zero.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No test titles use dynamic data; all added/modified names are static strings or table entries with fixed text.
Test Structure And Quality ✅ Passed Tests use table-driven subtests, explicit timeouted Eventually helpers, and deferred workload cleanup; no unbounded cluster waits or leaked resources found.
Topology-Aware Scheduling Compatibility ✅ Passed Only autoscaler capacity annotations/tests changed; no nodeSelector, affinity, PDB, replica, or topology-aware scheduling constraints were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The changed e2e path only checks annotations on cluster objects; no IPv4 literals, IP parsing, host/port concatenation, or public endpoints were added.
No-Weak-Crypto ✅ Passed Touched files only add/verify autoscaler annotations; no crypto primitives, weak ciphers, or secret/token comparisons appear in the diff.
Container-Privileges ✅ Passed Only Go logic/test files changed; no container/K8s manifests or privileged settings were added, and keyword scan found none.
No-Sensitive-Data-In-Logs ✅ Passed Changed logs only print nodepool names and capacity annotations (CPU/memory/GPU/labels); no passwords, tokens, PII, or internal hostnames are logged.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from Nirshal and enxebre July 24, 2026 21:57
@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: judexzhu
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/testing Indicates the PR includes changes for e2e testing and removed do-not-merge/needs-area labels Jul 24, 2026
@judexzhu
judexzhu force-pushed the fix/scale-from-zero-annotation-keys branch from 1950f62 to 17c88e9 Compare July 24, 2026 22:00
@judexzhu

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jul 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: This pull request references Jira Issue OCPBUGS-99759, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jul 24, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
test/e2e/autoscaling_test.go (1)

804-818: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Validate annotation values, not only key presence.

An empty or malformed CPU/memory annotation currently satisfies this predicate, even though the CAPI provider later parses these values as resource quantities and cannot build usable capacity. Parse the CAPI values, require non-zero quantities, and verify the legacy values remain consistent. (raw.githubusercontent.com)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/autoscaling_test.go` around lines 804 - 818, Update the annotation
fallback predicate around the CAPI and legacy capacity keys to validate values
rather than only checking key presence. Parse the CAPI CPU and memory
annotations as resource quantities, reject missing, malformed, or zero values,
and verify the legacy vCPU and memoryMb annotations represent consistent
capacity before returning success; preserve the existing failure messages or
equivalent context.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hypershift-operator/controllers/nodepool/scale_from_zero.go`:
- Around line 134-140: Add the CAPI GPU type annotation alongside the count
annotation in the GPU handling within the scale-from-zero logic, using the
instance GPU resource name such as nvidia.com/gpu. Remove this annotation in
both cleanup paths, and extend the related tests to verify it is present for GPU
instances and absent for non-GPU instances.

---

Nitpick comments:
In `@test/e2e/autoscaling_test.go`:
- Around line 804-818: Update the annotation fallback predicate around the CAPI
and legacy capacity keys to validate values rather than only checking key
presence. Parse the CAPI CPU and memory annotations as resource quantities,
reject missing, malformed, or zero values, and verify the legacy vCPU and
memoryMb annotations represent consistent capacity before returning success;
preserve the existing failure messages or equivalent context.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7434075a-e927-4bcb-b795-57c680689d29

📥 Commits

Reviewing files that changed from the base of the PR and between 144cca9 and 1950f62.

📒 Files selected for processing (3)
  • hypershift-operator/controllers/nodepool/scale_from_zero.go
  • hypershift-operator/controllers/nodepool/scale_from_zero_test.go
  • test/e2e/autoscaling_test.go

Comment thread hypershift-operator/controllers/nodepool/scale_from_zero.go
@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: This pull request references Jira Issue OCPBUGS-99759, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

  • HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi, which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys
  • The scale-from-zero workaround only wrote machine.openshift.io/* keys (Machine API provider namespace), which the CAPI provider silently ignores
  • Adds correct CAPI annotation keys alongside existing legacy keys so scale-from-zero capacity detection works

Design Decision: Co-existing Keys

Legacy machine.openshift.io/* keys are preserved alongside new capacity.cluster-autoscaler.kubernetes.io/* keys rather than replaced. Rationale:

  1. No breakage — if anything reads legacy keys (monitoring, dashboards, other controllers), it continues to work
  2. Smaller blast radius — additive change, no migration/cleanup logic needed
  3. Rollback safe — reverting this PR just removes the new keys, legacy keys remain functional (for Machine API provider consumers)

Future deprecation: Legacy machine.openshift.io/* keys should be removed in a follow-up once we confirm no consumers depend on them. The constants are already grouped and commented as "Legacy" to make this straightforward.

Related

  • OCPBUGS-99760: CAPA AWSMachineTemplate.Status.Capacity never populated (separate root cause — missing ownerReferences). Together with this bug, scale-from-zero had no working capacity signal.
  • Fixing either bug enables scale-from-zero. This PR fixes the annotation path; OCPBUGS-99760 tracks the native CAPI path.

Changes

File What
scale_from_zero.go Add capiCPUKey, capiMemoryKey, capiGPUCountKey constants; set both key sets; memory uses MiB suffix for CAPI; cleanup covers both
scale_from_zero_test.go Assert both legacy and CAPI keys in all test cases
autoscaling_test.go e2e verifies both CAPI and legacy annotation keys present

Test plan

  • Unit tests pass (870 tests, go test ./hypershift-operator/controllers/nodepool/...)
  • Lint clean (golangci-lint run)
  • e2e-aws-scale-from-zero (CI)

Bug: https://issues.redhat.com/browse/OCPBUGS-99759

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
  • Improved scale-from-zero autoscaling by supporting additional Cluster Autoscaler capacity annotation keys (CPU, memory, and GPU count).
  • Updated annotation handling so capacity provided by the autoscaler properly clears any legacy and newer capacity metadata.
  • Ensures correct CPU/memory (and GPU when applicable) annotations for both CPU-only and GPU configurations across common cloud templates, including taint scenarios.
  • Tests
  • Expanded unit and end-to-end autoscaling coverage to validate the newer capacity annotation keys and behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@judexzhu judexzhu changed the title OCPBUGS-99759: fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero Jul 24, 2026
@openshift-ci-robot openshift-ci-robot removed jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Jul 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

Summary

  • HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi, which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys
  • The scale-from-zero workaround only wrote machine.openshift.io/* keys (Machine API provider namespace), which the CAPI provider silently ignores
  • Adds correct CAPI annotation keys alongside existing legacy keys so scale-from-zero capacity detection works

Design Decision: Co-existing Keys

Legacy machine.openshift.io/* keys are preserved alongside new capacity.cluster-autoscaler.kubernetes.io/* keys rather than replaced. Rationale:

  1. No breakage — if anything reads legacy keys (monitoring, dashboards, other controllers), it continues to work
  2. Smaller blast radius — additive change, no migration/cleanup logic needed
  3. Rollback safe — reverting this PR just removes the new keys, legacy keys remain functional (for Machine API provider consumers)

Future deprecation: Legacy machine.openshift.io/* keys should be removed in a follow-up once we confirm no consumers depend on them. The constants are already grouped and commented as "Legacy" to make this straightforward.

Related

  • OCPBUGS-99760: CAPA AWSMachineTemplate.Status.Capacity never populated (separate root cause — missing ownerReferences). Together with this bug, scale-from-zero had no working capacity signal.
  • Fixing either bug enables scale-from-zero. This PR fixes the annotation path; OCPBUGS-99760 tracks the native CAPI path.

Changes

File What
scale_from_zero.go Add capiCPUKey, capiMemoryKey, capiGPUCountKey constants; set both key sets; memory uses MiB suffix for CAPI; cleanup covers both
scale_from_zero_test.go Assert both legacy and CAPI keys in all test cases
autoscaling_test.go e2e verifies both CAPI and legacy annotation keys present

Test plan

  • Unit tests pass (870 tests, go test ./hypershift-operator/controllers/nodepool/...)
  • Lint clean (golangci-lint run)
  • e2e-aws-scale-from-zero (CI)

Bug: https://issues.redhat.com/browse/OCPBUGS-99759

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
  • Improved scale-from-zero autoscaling by supporting additional Cluster Autoscaler capacity annotation keys (CPU, memory, and GPU count).
  • Updated annotation handling so capacity provided by the autoscaler properly clears any legacy and newer capacity metadata.
  • Ensures correct CPU/memory (and GPU when applicable) annotations for both CPU-only and GPU configurations across common cloud templates, including taint scenarios.
  • Tests
  • Expanded unit and end-to-end autoscaling coverage to validate the newer capacity annotation keys and behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jul 24, 2026
@judexzhu judexzhu changed the title fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero OCPBUGS-99759 fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero Jul 24, 2026
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.56%. Comparing base (144cca9) to head (5a6f6f0).
⚠️ Report is 30 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9110      +/-   ##
==========================================
+ Coverage   44.51%   44.56%   +0.05%     
==========================================
  Files         774      774              
  Lines       96997    97009      +12     
==========================================
+ Hits        43179    43234      +55     
+ Misses      50830    50783      -47     
- Partials     2988     2992       +4     
Files with missing lines Coverage Δ
...t-operator/controllers/nodepool/scale_from_zero.go 100.00% <100.00%> (ø)

... and 6 files with indirect coverage changes

Flag Coverage Δ
cmd-support 38.39% <ø> (ø)
cpo-hostedcontrolplane 47.22% <ø> (ø)
cpo-other 45.25% <ø> (ø)
hypershift-operator 54.47% <100.00%> (+0.02%) ⬆️
other 33.07% <ø> (+0.43%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@judexzhu

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@judexzhu judexzhu changed the title OCPBUGS-99759 fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero OCPBUGS-99759: fix(nodepool): add CAPI autoscaler capacity annotations for scale-from-zero Jul 24, 2026
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jul 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: This pull request references Jira Issue OCPBUGS-99759, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

  • HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi, which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys
  • The scale-from-zero workaround only wrote machine.openshift.io/* keys (Machine API provider namespace), which the CAPI provider silently ignores
  • Adds correct CAPI annotation keys alongside existing legacy keys so scale-from-zero capacity detection works

Design Decision: Co-existing Keys

Legacy machine.openshift.io/* keys are preserved alongside new capacity.cluster-autoscaler.kubernetes.io/* keys rather than replaced. Rationale:

  1. No breakage — if anything reads legacy keys (monitoring, dashboards, other controllers), it continues to work
  2. Smaller blast radius — additive change, no migration/cleanup logic needed
  3. Rollback safe — reverting this PR just removes the new keys, legacy keys remain functional (for Machine API provider consumers)

Future deprecation: Legacy machine.openshift.io/* keys should be removed in a follow-up once we confirm no consumers depend on them. The constants are already grouped and commented as "Legacy" to make this straightforward.

Related

  • OCPBUGS-99760: CAPA AWSMachineTemplate.Status.Capacity never populated (separate root cause — missing ownerReferences). Together with this bug, scale-from-zero had no working capacity signal.
  • Fixing either bug enables scale-from-zero. This PR fixes the annotation path; OCPBUGS-99760 tracks the native CAPI path.

Changes

File What
scale_from_zero.go Add capiCPUKey, capiMemoryKey, capiGPUCountKey constants; set both key sets; memory uses MiB suffix for CAPI; cleanup covers both
scale_from_zero_test.go Assert both legacy and CAPI keys in all test cases
autoscaling_test.go e2e verifies both CAPI and legacy annotation keys present

Test plan

  • Unit tests pass (870 tests, go test ./hypershift-operator/controllers/nodepool/...)
  • Lint clean (golangci-lint run)
  • e2e-aws-scale-from-zero (CI)

Bug: https://issues.redhat.com/browse/OCPBUGS-99759

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
  • Improved scale-from-zero autoscaling by supporting additional Cluster Autoscaler capacity annotation keys (CPU, memory, and GPU count).
  • Updated annotation handling so capacity provided by the autoscaler properly clears any legacy and newer capacity metadata.
  • Ensures correct CPU/memory (and GPU when applicable) annotations for both CPU-only and GPU configurations across common cloud templates, including taint scenarios.
  • Tests
  • Expanded unit and end-to-end autoscaling coverage to validate the newer capacity annotation keys and behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown

@judexzhu: This pull request references Jira Issue OCPBUGS-99759, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…m-zero

HyperShift deploys cluster-autoscaler with --cloud-provider=clusterapi,
which reads capacity.cluster-autoscaler.kubernetes.io/* annotation keys.
The scale-from-zero workaround only wrote machine.openshift.io/* keys
(Machine API provider namespace), which the CAPI provider ignores.

Add correct CAPI annotation keys alongside existing legacy keys:
- capacity.cluster-autoscaler.kubernetes.io/cpu
- capacity.cluster-autoscaler.kubernetes.io/memory (with MiB suffix)
- capacity.cluster-autoscaler.kubernetes.io/gpu-count

Legacy machine.openshift.io/* keys are preserved for backward
compatibility and will be deprecated in a follow-up.

Bug: https://issues.redhat.com/browse/OCPBUGS-99759
@judexzhu
judexzhu force-pushed the fix/scale-from-zero-annotation-keys branch from 17c88e9 to 5a6f6f0 Compare July 27, 2026 15:47

@elmiko elmiko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not overly familiar with this code, but using both annotations seems like a good update to me. the cluster-autoscaler at version <=4.22 should recognize both annotations, but after 4.23/5.0 it will not.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 28, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke
/test unit
/test verify

@cwbotbot

cwbotbot commented Jul 28, 2026

Copy link
Copy Markdown

Test Results

e2e-aws

e2e-aks

@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@judexzhu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-v2-gke 5a6f6f0 link true /test e2e-v2-gke
ci/prow/e2e-aws-4-22 5a6f6f0 link true /test e2e-aws-4-22
ci/prow/verify 5a6f6f0 link true /test verify

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/testing Indicates the PR includes changes for e2e testing jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants