[publish images] bootstrap: rebake CI images to refresh prefetched deps - #30095
[publish images] bootstrap: rebake CI images to refresh prefetched deps#30095Jarred-Sumner wants to merge 2 commits into
Conversation
|
Updated 4:19 AM PT - May 4th, 2026
❌ @Jarred-Sumner, your commit 09d5408 has 1 failures in
🧪 To try this PR locally: bunx bun-pr 30095That installs a local version of the PR into your bun-30095 --bun |
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
WalkthroughGCC 13 installation in the Docker build and bootstrap shell script was changed from adding the ubuntu-toolchain-r/test PPA and apt-installing packages to downloading architecture-specific tarballs of prebuilt gcc-13 .deb packages and installing them. Header version comments in ChangesGCC prebuilt .deb install (toolchain change)
Bootstrap script version headers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Review rate limit: 0/5 reviews remaining, refill in 9 minutes and 45 seconds. Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — version-comment-only bump to trigger CI image rebake.
Extended reasoning...
Overview
This PR changes exactly two lines: the # Version: comment in scripts/bootstrap.sh (34→35) and scripts/bootstrap.ps1 (19→20). Per the inline comments in both files, bumping this version is the established mechanism for signaling that a fresh CI image should be built rather than retroactively updating the existing one. No script logic, dependency versions, or behavior changes.
Security risks
None. The diff touches only comment text. The scripts themselves already exist and run in CI; this PR does not alter what they do, only triggers them to run during a fresh image bake.
Level of scrutiny
Minimal. This is a purely mechanical, comment-only change following an explicitly documented pattern ("increment the version comment to indicate that a new image should be built"). The actual effect — rebaking CI images — is an infrastructure/CI operation that BuildKite will validate, not a code-correctness concern.
Other factors
The PR description clearly explains the motivation (refresh the baked prefetch cache from #29568 to reduce GitHub 503 flakes) and correctly notes "No script content changes; version bump only," which the diff confirms. No bugs were found by the bug-hunting system, and there are no outstanding reviewer comments.
There was a problem hiding this comment.
LGTM — CI image infra change (version bumps + gcc-13 source swap to org-mirrored .debs), self-validated by the [build images] pipeline.
Extended reasoning...
Overview
This PR touches three CI-infrastructure files: .buildkite/Dockerfile, scripts/bootstrap.sh, and scripts/bootstrap.ps1. There are two logical changes across two commits:
- Version-comment bumps (
bootstrap.sh34→35,bootstrap.ps119→20) to force a fresh image bake so the prefetched-deps cache is refreshed. install_gcc()inbootstrap.shand the corresponding Dockerfile RUN step now fetch gcc-13 from a tarball of mirrored .debs hosted atoven-sh/WebKit/releases/gcc-13-focal-debsinstead ofppa:ubuntu-toolchain-r/test, to avoid Launchpad 503/timeouts during image builds. Arch mapping is correct (DockerTARGETARCHisamd64/arm64; bootstrap.sh mapsx64→amd64,aarch64→arm64).
Note: the PR description says "version bump only" — that was accurate for the first commit but a second commit added the gcc-13 source swap.
Security risks
None meaningful. The new download source is a release asset under the same GitHub org (oven-sh), so this is not a supply-chain downgrade — if anything it's an improvement over a third-party PPA. The .debs are installed via apt-get install which still verifies package integrity. This code only runs during CI image construction, never in shipped Bun binaries or on end-user machines.
Level of scrutiny
Low. This is build-image bootstrap tooling, gated behind --gcc-13 / Ubuntu focal, and is fully self-validating: the HEAD commit carries [build images] so BuildKite actually builds the new images on this PR. If the tarball URL, arch mapping, or .deb set were wrong, the image build would fail visibly.
Other factors
No bugs flagged by the bug-hunting system, no outstanding reviewer comments, and CodeRabbit had no actionable findings. The change follows the existing pattern in this file (download tarball → extract → install) used for cmake, node, buildkite-agent, etc.
232e0a1 to
fe7f92b
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.buildkite/Dockerfile:
- Around line 35-37: The Dockerfile currently pipes the remote
gcc-13-focal-${TARGETARCH}.tar.gz straight into tar and installs .deb files from
/tmp/gcc13 without verifying integrity; change the steps to download the tarball
to a file (e.g., /tmp/gcc-13-focal-${TARGETARCH}.tar.gz), compute its SHA-256
checksum with sha256sum, compare it against the pinned checksums for amd64 and
arm64 (use TARGETARCH to select the expected hash), fail the build if the
checksum does not match, then extract to /tmp/gcc13 and run apt-get install -y
--no-install-recommends /tmp/gcc13/*.deb and cleanup; ensure failures return
non-zero so the build aborts on mismatch.
In `@scripts/bootstrap.sh`:
- Around line 1194-1205: The script currently fetches and extracts the GCC .deb
tarball without integrity checks; replace the fetch/pipe sequence by using the
existing download_and_verify_file helper (same pattern used for the age tarball)
to download the URL held in gcc_deb_url into a temporary path (created via
create_tmp_directory or a temp file), verify it against the appropriate SHA256
checksum (use the release's gcc-13-focal-amd64.tar.gz /
gcc-13-focal-arm64.tar.gz checksums), then extract the verified tarball into
gcc_deb_dir and proceed with execute_sudo apt-get install; ensure you still
remove gcc_deb_dir on cleanup. Use the same variables/functions (gcc_deb_url,
gcc_deb_dir, create_tmp_directory, download_and_verify_file, execute,
execute_sudo) so the replacement is local and preserves install/cleanup
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ae262d35-b91c-4e88-87de-213efa124101
📒 Files selected for processing (3)
.buildkite/Dockerfilescripts/bootstrap.ps1scripts/bootstrap.sh
| && wget -qO- "https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-${TARGETARCH}.tar.gz" \ | ||
| | tar -xz -C /tmp/gcc13 \ | ||
| && apt-get install -y --no-install-recommends /tmp/gcc13/*.deb \ |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Verify release assets expose checksum material you can pin in Docker args.
curl -fsSL "https://api.github.com/repos/oven-sh/WebKit/releases/tags/gcc-13-focal-debs" \
| jq -r '.assets[].name'Repository: oven-sh/bun
Length of output: 107
🏁 Script executed:
cat -n .buildkite/Dockerfile | sed -n '25,45p'Repository: oven-sh/bun
Length of output: 1192
🏁 Script executed:
#!/bin/bash
# Query GitHub API for release details including assets with checksums
curl -fsSL "https://api.github.com/repos/oven-sh/WebKit/releases/tags/gcc-13-focal-debs" \
| jq '.assets[] | {name, browser_download_url, size, id}'Repository: oven-sh/bun
Length of output: 469
🏁 Script executed:
#!/bin/bash
# Check if release page includes checksums or hashes in description/body
curl -fsSL "https://api.github.com/repos/oven-sh/WebKit/releases/tags/gcc-13-focal-debs" \
| jq -r '.body'Repository: oven-sh/bun
Length of output: 774
Verify GCC tarball integrity before installing .deb packages.
Lines 35-37 install externally downloaded packages without checksum verification. Add SHA-256 verification before extraction/install to reduce supply-chain risk.
The release notes for gcc-13-focal-debs already expose checksums that can be pinned:
- amd64:
a2b3b6e10b175bbaaefeb3e9e703ca26a97ed6c1f19ca842d3e0b0c8f941e65b - arm64:
be19db90d94c52c6061280bbadcaad9b09db1e9f2e77a12f8c18c5425d2eb056
Proposed hardening diff
+ARG GCC13_FOCAL_AMD64_SHA256="a2b3b6e10b175bbaaefeb3e9e703ca26a97ed6c1f19ca842d3e0b0c8f941e65b"
+ARG GCC13_FOCAL_ARM64_SHA256="be19db90d94c52c6061280bbadcaad9b09db1e9f2e77a12f8c18c5425d2eb056"
RUN apt-get update && apt-get install -y --no-install-recommends \
wget curl git python3 python3-pip ninja-build \
software-properties-common apt-transport-https \
ca-certificates gnupg lsb-release unzip xz-utils \
libxml2-dev ruby ruby-dev bison gawk perl make golang ccache qemu-user-static \
libc6-dev \
# gcc-13 on focal: Launchpad ppa:ubuntu-toolchain-r/test times out / 503s
# often enough to break image builds, so install from a mirrored snapshot
# of the PPA's .debs instead. See oven-sh/WebKit release gcc-13-focal-debs.
&& mkdir /tmp/gcc13 \
- && wget -qO- "https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-${TARGETARCH}.tar.gz" \
- | tar -xz -C /tmp/gcc13 \
+ && wget -qO /tmp/gcc13.tar.gz "https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-${TARGETARCH}.tar.gz" \
+ && GCC13_SHA256=$([ "$TARGETARCH" = "arm64" ] && echo "$GCC13_FOCAL_ARM64_SHA256" || echo "$GCC13_FOCAL_AMD64_SHA256") \
+ && echo "${GCC13_SHA256} /tmp/gcc13.tar.gz" | sha256sum -c - \
+ && tar -xzf /tmp/gcc13.tar.gz -C /tmp/gcc13 \
&& apt-get install -y --no-install-recommends /tmp/gcc13/*.deb \
- && rm -rf /tmp/gcc13 \
+ && rm -rf /tmp/gcc13 /tmp/gcc13.tar.gz \🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.buildkite/Dockerfile around lines 35 - 37, The Dockerfile currently pipes
the remote gcc-13-focal-${TARGETARCH}.tar.gz straight into tar and installs .deb
files from /tmp/gcc13 without verifying integrity; change the steps to download
the tarball to a file (e.g., /tmp/gcc-13-focal-${TARGETARCH}.tar.gz), compute
its SHA-256 checksum with sha256sum, compare it against the pinned checksums for
amd64 and arm64 (use TARGETARCH to select the expected hash), fail the build if
the checksum does not match, then extract to /tmp/gcc13 and run apt-get install
-y --no-install-recommends /tmp/gcc13/*.deb and cleanup; ensure failures return
non-zero so the build aborts on mismatch.
| # gcc-13 on focal: Launchpad ppa:ubuntu-toolchain-r/test times out / 503s | ||
| # often enough to break image builds, so install from a mirrored snapshot | ||
| # of the PPA's .debs instead. See oven-sh/WebKit release gcc-13-focal-debs. | ||
| case "$arch" in | ||
| x64) gcc_deb_arch="amd64" ;; | ||
| aarch64) gcc_deb_arch="arm64" ;; | ||
| esac | ||
| gcc_deb_url="https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-$gcc_deb_arch.tar.gz" | ||
| gcc_deb_dir="$(create_tmp_directory)" | ||
| fetch "$gcc_deb_url" | execute tar -xz -C "$gcc_deb_dir" | ||
| execute_sudo apt-get install -y --no-install-recommends "$gcc_deb_dir"/*.deb libc6-dev | ||
| execute rm -rf "$gcc_deb_dir" |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# List release assets to source per-arch checksums you can pin in bootstrap.sh
curl -fsSL "https://api.github.com/repos/oven-sh/WebKit/releases/tags/gcc-13-focal-debs" \
| jq -r '.assets[].name'Repository: oven-sh/bun
Length of output: 107
🏁 Script executed:
rg "download_and_verify_file" scripts/bootstrap.shRepository: oven-sh/bun
Length of output: 291
🏁 Script executed:
sed -n '1194,1205p' scripts/bootstrap.shRepository: oven-sh/bun
Length of output: 704
🏁 Script executed:
# Check if there are SHA256 checksums documented for these assets
rg "gcc-13-focal" scripts/bootstrap.sh -A 5 -B 5Repository: oven-sh/bun
Length of output: 719
Use download_and_verify_file for the GCC .deb tarball to add integrity verification.
External .deb artifacts are downloaded and extracted without verification. The download_and_verify_file helper already exists in this script (used for age tarball) and supports this pattern.
Implementation approach
case "$arch" in
x64) gcc_deb_arch="amd64" ;;
aarch64) gcc_deb_arch="arm64" ;;
esac
gcc_deb_url="https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-$gcc_deb_arch.tar.gz"
+ case "$gcc_deb_arch" in
+ amd64) gcc_deb_sha256="<fill-from-release>" ;;
+ arm64) gcc_deb_sha256="<fill-from-release>" ;;
+ esac
gcc_deb_dir="$(create_tmp_directory)"
- fetch "$gcc_deb_url" | execute tar -xz -C "$gcc_deb_dir"
+ gcc_deb_tar="$(download_and_verify_file "$gcc_deb_url" "$gcc_deb_sha256")"
+ execute tar -xzf "$gcc_deb_tar" -C "$gcc_deb_dir"
execute_sudo apt-get install -y --no-install-recommends "$gcc_deb_dir"/*.deb libc6-devObtain the SHA256 checksums from the gcc-13-focal-debs release assets (confirmed: gcc-13-focal-amd64.tar.gz and gcc-13-focal-arm64.tar.gz exist).
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # gcc-13 on focal: Launchpad ppa:ubuntu-toolchain-r/test times out / 503s | |
| # often enough to break image builds, so install from a mirrored snapshot | |
| # of the PPA's .debs instead. See oven-sh/WebKit release gcc-13-focal-debs. | |
| case "$arch" in | |
| x64) gcc_deb_arch="amd64" ;; | |
| aarch64) gcc_deb_arch="arm64" ;; | |
| esac | |
| gcc_deb_url="https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-$gcc_deb_arch.tar.gz" | |
| gcc_deb_dir="$(create_tmp_directory)" | |
| fetch "$gcc_deb_url" | execute tar -xz -C "$gcc_deb_dir" | |
| execute_sudo apt-get install -y --no-install-recommends "$gcc_deb_dir"/*.deb libc6-dev | |
| execute rm -rf "$gcc_deb_dir" | |
| # gcc-13 on focal: Launchpad ppa:ubuntu-toolchain-r/test times out / 503s | |
| # often enough to break image builds, so install from a mirrored snapshot | |
| # of the PPA's .debs instead. See oven-sh/WebKit release gcc-13-focal-debs. | |
| case "$arch" in | |
| x64) gcc_deb_arch="amd64" ;; | |
| aarch64) gcc_deb_arch="arm64" ;; | |
| esac | |
| gcc_deb_url="https://github.com/oven-sh/WebKit/releases/download/gcc-13-focal-debs/gcc-13-focal-$gcc_deb_arch.tar.gz" | |
| case "$gcc_deb_arch" in | |
| amd64) gcc_deb_sha256="<fill-from-release>" ;; | |
| arm64) gcc_deb_sha256="<fill-from-release>" ;; | |
| esac | |
| gcc_deb_dir="$(create_tmp_directory)" | |
| gcc_deb_tar="$(download_and_verify_file "$gcc_deb_url" "$gcc_deb_sha256")" | |
| execute tar -xzf "$gcc_deb_tar" -C "$gcc_deb_dir" | |
| execute_sudo apt-get install -y --no-install-recommends "$gcc_deb_dir"/*.deb libc6-dev | |
| execute rm -rf "$gcc_deb_dir" |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scripts/bootstrap.sh` around lines 1194 - 1205, The script currently fetches
and extracts the GCC .deb tarball without integrity checks; replace the
fetch/pipe sequence by using the existing download_and_verify_file helper (same
pattern used for the age tarball) to download the URL held in gcc_deb_url into a
temporary path (created via create_tmp_directory or a temp file), verify it
against the appropriate SHA256 checksum (use the release's
gcc-13-focal-amd64.tar.gz / gcc-13-focal-arm64.tar.gz checksums), then extract
the verified tarball into gcc_deb_dir and proceed with execute_sudo apt-get
install; ensure you still remove gcc_deb_dir on cleanup. Use the same
variables/functions (gcc_deb_url, gcc_deb_dir, create_tmp_directory,
download_and_verify_file, execute, execute_sudo) so the replacement is local and
preserves install/cleanup behavior.
There was a problem hiding this comment.
LGTM — CI image bootstrap changes only (version bumps + gcc-13 mirror swap), no shipped-code impact.
Extended reasoning...
Overview
Touches three CI-infrastructure files: .buildkite/Dockerfile, scripts/bootstrap.sh, and scripts/bootstrap.ps1. The .ps1 change is purely a version-comment bump (19→20). The .sh and Dockerfile changes bump the version comment (34→35) and swap the gcc-13 install source from the flaky Launchpad ppa:ubuntu-toolchain-r/test to a mirrored tarball of the same .debs hosted on the org's own oven-sh/WebKit GitHub release, installed via apt-get install ./*.deb. Note: the PR description says "version bump only" but the diff also carries the gcc-13 mirror change — both are already reflected on main as separate commits.
Security risks
None meaningful. The new download source is a GitHub release on a repo controlled by the same org (oven-sh), so the trust boundary is unchanged vs. the PPA. Packages are still installed through apt's dependency resolver. These scripts run only at CI-image bake time, not in the shipped bun binary or on end-user machines.
Level of scrutiny
Low. This is build-image plumbing: if the new install path is wrong, the Docker image build / Packer bake fails loudly and nothing ships. The fetch | tar -xz and create_tmp_directory helpers are reused exactly as elsewhere in bootstrap.sh, and the Dockerfile change mirrors the same pattern. The arch mapping (x64→amd64, aarch64→arm64) matches Docker's TARGETARCH values used in the Dockerfile.
Other factors
The CI test failures reported by robobun (require-cache, no-orphans, dev-and-prod) are macOS/Windows runtime test flakes unrelated to Linux image-bake scripts. CodeRabbit found nothing actionable. No outstanding reviewer comments.
|
Build #50192: 292/296 passed, all 10 image builds green (incl. aws-2023 with the gcc-13 mirror), zero Remaining:
Ready to squash-merge — |
GitHub release tarball downloads occasionally 503 (~1/10000). The build images bake a dep prefetch cache (since #29568); bumping the version forces a rebake so runtime CI hits the cache instead of github.com.
…ad PPA add-apt-repository ppa:ubuntu-toolchain-r/test has been timing out (553s, then "'~ubuntu-toolchain-r' user or team does not exist") often enough to break the aws-2023 image builds. Install gcc-13 + runtime libs from a snapshot of the PPA's focal .debs hosted at oven-sh/WebKit:gcc-13-focal-debs instead.
fe7f92b to
09d5408
Compare
There was a problem hiding this comment.
LGTM — CI image-build infra only; swaps the flaky Launchpad PPA for a mirrored .deb tarball on oven-sh's own release, matching the existing download patterns in this Dockerfile, and image builds are green.
Extended reasoning...
Overview
This PR touches three CI image-build files: .buildkite/Dockerfile, scripts/bootstrap.sh, and scripts/bootstrap.ps1. The substantive change replaces the gcc-13 install on Ubuntu focal (which previously added ppa:ubuntu-toolchain-r/test and apt-installed packages) with a download of a mirrored tarball of the same PPA's .debs hosted at oven-sh/WebKit/releases/gcc-13-focal-debs, then apt-get installs those .debs. bootstrap.ps1 gets only a version-comment bump (19→20), and bootstrap.sh gets the gcc change plus a version bump (34→35) to force a rebake of the CI images so the prefetched-deps cache is refreshed.
Note: the PR description says "version bump only" but a second commit (09d5408) added the gcc-13 mirror change on top of the original bump commit; the HEAD commit message correctly describes it.
Security risks
CodeRabbit flagged that the new tarball download lacks SHA-256 verification. That's a fair hardening suggestion, but it is not a regression: the same Dockerfile/bootstrap already download cmake, node, the Android NDK, the FreeBSD sysroot, buildkite-agent, llvm.sh, rustup, and bun itself with the same fetch-over-HTTPS-no-checksum pattern. The new source is the org's own oven-sh/WebKit GitHub release — the same trust boundary as the other oven-sh downloads in the file. So the security posture is unchanged. None of this touches the shipped product; it only affects how internal CI build images are baked.
Level of scrutiny
Low–medium. This is internal CI infrastructure (image-bake scripts), not runtime/product code. The change is mechanical (same packages, different delivery mechanism), follows established patterns elsewhere in the same files, and the author reports all 10 image builds green plus 292/296 tests passing with only pre-existing/known flakes remaining.
Other factors
- No CODEOWNERS cover these paths.
- The
install_gcccodepath inbootstrap.shis gated ondistro = ubuntuand--gcc-13, so the focal-specific .deb tarball is only used where it applies. update-alternativesand the libstdc++/include path setup that follow are unchanged, so the resulting toolchain layout is identical to before.- Bug-hunter found no issues.
…ailure (#37915) ### Problem - Build lanes die in a dep fetch: `error: Failed to download after 5 attempts: https://github.com/oven-sh/lol-html/archive/725ce499....tar.gz`, `cause: fetch failed`. Build 93382 lost linux x64-asan on lolhtml. Build 93392 (a one-file PR) lost four lanes: x64-musl and x64-android on cares, mimalloc and the WebKit tarball, freebsd aarch64 on lolhtml, windows aarch64 on cares, libuv, mimalloc and WebKit. - Those are the downloads that miss the image's prefetch cache (everything else in the same logs says `using prefetch cache`): the deps whose pins moved after the images were baked in #34782 (lolhtml #36733, mimalloc #36431, c-ares #34007, libuv #36839, WebKit several times a week), plus WebKit on every lane other than linux arm64, because `prefetch-deps.ts` only enumerates the bake host's own target (handed off separately). Each build makes on the order of a hundred live github.com downloads. - `downloadWithRetry` (`scripts/build/download.ts:156`) made 5 attempts with 2+4+8+16s of backoff, about 30s in total. The logs show agent-wide outages longer than that: on the x64-musl lane three downloads that started together failed on all five attempts over roughly two minutes, after lolhtml had succeeded on the same agent seconds earlier; on the freebsd lane lolhtml failed five times in a row while cares and mimalloc went through and WebKit succeeded on its second try. - `BuildError.format()` (`scripts/build/error.ts:33`) prints one level of cause. node's fetch throws `TypeError: fetch failed` and keeps the real error (DNS, connect timeout, reset) in `.cause`, so every one of these logs says only `cause: fetch failed`, and the retry lines say nothing about what failed. ### Fix - `downloadRetry`: 10 attempts, backoff doubling from 2s and capped at 30s, 180s of backoff in total instead of 30s. Exported as a `RetryPolicy` (optional last parameter of `downloadWithRetry`) so the test can run the production attempt count with the backoff zeroed. - 408 and 429 are retried along with 5xx and network errors. Other 4xx still fail on the first attempt and are still thrown unwrapped, which `prefetch-deps.ts` relies on to tell a 404 (variant not published) from a transient failure. - Each retry line names the failure it is retrying, e.g. `retry 2/10 in 2000ms (fetch failed: other side closed)`, and `format()` prints the whole cause chain through the new `describeError()`, so the next one of these says what the network did. - Why here: the prefetch cache goes stale by design as soon as a pin moves, and WebKit moves faster than images get rebaked (the images were rebaked on July 21 and this came back within two weeks; #30095 was an earlier rebake for the same symptom), so the live path is permanently on every build's critical path and has to outlast the outages CI actually sees. The wider window only costs time while github.com is actually down, when the lane would otherwise have failed; build-bun's step timeout is 60 minutes. - Verified with `test/internal/build-download-retry.test.ts` (`bun bd test`, 7 pass). It drives the real `downloadWithRetry` against a local server that drops connections or returns scripted statuses, checks the retry lines and `format()` output, and pins the schedule's total backoff at two minutes or more. Against the previous `download.ts`/`error.ts` the file fails at import (`downloadRetry` and `describeError` did not exist); each behavioral case is something the old loop did not do (10 attempts, 429 retried, reason on the retry line, cause chain in `format()`). - Also ran the loop under node, the runtime CI builds with, against a dropping server: retry lines read `(fetch failed: other side closed)` and the final report prints `cause: fetch failed: other side closed`. `bunx tsc -p scripts/build/tsconfig.json` reports nothing for these files. ### Background - Dep fetching: configure emits one ninja `dep_fetch` edge per vendored dep, which runs `scripts/build/fetch-cli.ts`; that calls `downloadWithRetry` on `https://github.com/<repo>/archive/<commit>.tar.gz`, and `fetchPrebuilt` uses the same function for release tarballs such as WebKit. Both URL kinds start with a 302 from github.com (to codeload.github.com and objects.githubusercontent.com respectively), which is why one github.com problem takes out both kinds at once. - Prefetch cache: CI images run `scripts/prefetch-deps.ts` at bake time, storing each tarball under `/opt/bun-prefetch/by-url/<sha256(url)>`. `downloadWithRetry` looks there before touching the network, so a dep is served from the image only while its pinned URL is the one that was current at bake time; anything bumped later downloads live until the next `[publish images]` rebake. - `BuildError` is the build system's error type; `fetch-cli.ts` and `build.ts` print failures through its `format()`, which produces the `error:` / `hint:` / `cause:` lines seen in the build log.
2 participants
Bumps
bootstrap.shv34→35 andbootstrap.ps1v19→20 to force a fresh image build.Why: GitHub release-tarball downloads intermittently 503 (~1/10000 runs). Since #29568 the build images bake a dep prefetch cache, so a rebake means runtime CI reads from the image cache instead of hitting github.com — fewer flakes.
No script content changes; version bump only.
[build images]so BuildKite builds the new images on this PR.[publish images]is in the PR title for the squash-merge commit.