fleetshift: harden image mirror CI against build-farm flakes#82029
Conversation
Collapse single-arch and multiarch variants, publish sha+latest from one postsubmit, and add retried periodics so :latest can recover after capacity failures.
openshift-ci
Bot
added
the
approved
|
Warning Review limit reached
Next review available in: 22 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughFleetShift image configurations now build arm64 images and define local-image workflows. Mirror jobs support main and local images, merge workflows also tag ChangesFleetShift image mirroring
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant MergeJob
participant ImageMirrorStep
participant SourceRegistry
participant DestinationRegistry
MergeJob->>ImageMirrorStep: Set IMAGE_TAG and optional ALSO_TAG_LATEST
ImageMirrorStep->>SourceRegistry: Read source image
ImageMirrorStep->>DestinationRegistry: Mirror primary tag with retries
ImageMirrorStep->>DestinationRegistry: Mirror latest when enabled
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/pj-rehearse more |
|
@mshort55: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/fleetshift/ci/image-mirror/fleetshift-ci-image-mirror-commands.sh`:
- Around line 59-66: Update the mirroring flow around oc image mirror to track
its exit status and return success only when the command completes successfully;
do not use mirror_log contents as the success condition. Preserve the existing
error log for failed mirrors and ensure failed commands cannot report the
requested tag as published.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 5c0dd48c-fdd3-433a-a29f-9184193ad7fa
⛔ Files ignored due to path filters (6)
ci-operator/jobs/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main-postsubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main-postsubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (6)
ci-operator/config/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main.yamlci-operator/config/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main__multiarch.yamlci-operator/config/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main.yamlci-operator/config/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main__multiarch.yamlci-operator/step-registry/fleetshift/ci/image-mirror/fleetshift-ci-image-mirror-commands.shci-operator/step-registry/fleetshift/ci/image-mirror/fleetshift-ci-image-mirror-ref.yaml
💤 Files with no reviewable changes (2)
- ci-operator/config/fleetshift/fleetshift-user-interface/fleetshift-fleetshift-user-interface-main__multiarch.yaml
- ci-operator/config/fleetshift/fleetshift-poc/fleetshift-fleetshift-poc-main__multiarch.yaml
Avoid logging pullspecs/secret paths, and treat mirror success as exit 0 plus non-empty stdout.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/fleetshift/ci/image-mirror/fleetshift-ci-image-mirror-commands.sh`:
- Around line 21-24: Ensure the `$HOME/.docker` directory exists before the
`base64` redirection writes `config.json`. Add a `mkdir -p "$HOME/.docker"` step
immediately before the decode command, preserving the existing decode error
handling.
- Around line 44-68: Update the mirroring retry loop to capture stderr in
mirror_log instead of discarding it, suppressing intermediate failures but
logging the final command output before returning failure. Replace the repeated
rm -f cleanup in the mirroring function with a trap that removes mirror_log on
every exit, and remove the unreachable cleanup after the loop.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 3fc83427-4937-4e3c-9f84-2f16cf3fc18b
📒 Files selected for processing (1)
ci-operator/step-registry/fleetshift/ci/image-mirror/fleetshift-ci-image-mirror-commands.sh
Capture stderr, dump the final attempt output into artifacts/logs, and clean up the temp log with a RETURN trap.
Avoid copying full mirror output into public artifacts; keep enough context to debug failures without exposing chatty auth detail.
|
@mshort55, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
CI registry load rejects trap without grace_period; clean up temp files explicitly instead.
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alechenninger, mshort55 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse skip |
|
@mshort55: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
@mshort55: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary
https://redhat.atlassian.net/browse/OME-204
__multiarchvariants) so each push runs one build/mirror path.fleetshift-ci-image-mirrorto optionally tag:latestin the same job as the SHA tag (ALSO_TAG_LATEST), so postsubmits publish both atomically.:latest(UI every 6h; POC staggered) so capacity-related postsubmit failures can recover without another merge.Motivation
Recent
fleetshift-user-interfaceimage-mirror failures were build-farm capacity issues (Insufficient cpu/ node group maxed, long Pending), not Dockerfile/app errors. Presubmit aborts from superseded PRs also looked like red builds. This reduces duplicate multiarch load and adds a durable:latestrecovery path.Test plan
make ci-operator-config/make jobs/make registry-metadata(ormake update) on a supported host:latestfor UI and POCretryafter a capacity flakeSummary by CodeRabbit
ci-operator/to better survive build-farm capacity failures by reshaping periodic republish workflows and improving mirror error handling.*-main.yamlfiles by addingarm64to the image build matrices and removing the now-redundant*__multiarch.yamlconfigs.capabilities: [arm64]where appropriate.ALSO_TAG_LATEST: "true"for merge flows so:latestand commit-SHA tags come from the same build.:latestfor UI (every six hours) and for POC on a staggered schedule, enabling recovery from capacity flakes without another merge.fleetshift-ci-image-mirrorimplementation by:IMAGE_TAGfromJOB_TYPEwhen unset.mirror_image(tag)helper with retries and per-attempt logging.oc image mirrorstderr/output into artifacts (including the final failed attempt inartifacts/logs) and cleaning up temporary logs via aRETURNtrap.fleetshift-ci-image-mirrorstep documentation to clarifyIMAGE_TAGdefaults for postsubmit vs periodic jobs and the purpose ofALSO_TAG_LATEST.