ci: record which instance type each job actually ran on (agent tag, log header, failure annotations) - #38042
ci: record which instance type each job actually ran on (agent tag, log header, failure annotations)#38042robobun wants to merge 4 commits into
Conversation
…ine header CI requests one instance type per lane (.buildkite/ci.mjs), but under capacity pressure a job can land on a different, slower type, and the job log had nothing that identified the machine class. The CPU line comes from os.cpus(); the instance type is read from the EC2 or Azure metadata service, selected by the agent's cloud tag, so off-cloud agents skip the request.
|
Warning Review limit reached
Next review available in: 15 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
Comment |
|
Updated 5:41 PM PT - Aug 13th, 2026
❌ @robobun, your commit 931510e has 1 failures in 🧪 To try this PR locally: bunx bun-pr 38042That installs a local version of the PR into your bun-38042 --bun |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — CI-only diagnostics addition to the job log header.
What was reviewed:
getCpuDescription()andgetCloudInstanceType()follow the same shape as the neighboringgetPublicIp()/getTailscaleIp()helpers and degrade toundefinedon any failure (missing env var, curl not found, IMDS 401/timeout).- Confirmed
scripts/agent.mjssets thecloudagent tag that surfaces asBUILDKITE_AGENT_META_DATA_CLOUD, so the gate is live on the AWS/Azure fleets and a no-op elsewhere. Instance Type:is printed only underisCI, and the curl call is bounded by--connect-timeout 1 --max-time 3, so worst case adds ~3s to the header on a misconfigured agent.
Extended reasoning...
Overview
This PR adds two diagnostic lines to the Machine group that printEnvironment() prints at the top of every CI job log: CPU: (from os.cpus()[0].model + os.availableParallelism()) on all platforms, and Instance Type: (from the EC2 or Azure instance metadata service) on cloud CI agents. The only file touched is scripts/utils.mjs; two small helper functions are added and two console.log lines are inserted into printEnvironment(). No src/, no tests, no user-facing runtime code.
Security risks
None. The metadata request goes to the well-known link-local IMDS address (169.254.169.254) and only runs when BUILDKITE_AGENT_META_DATA_CLOUD is set to aws or azure — the same env var and endpoints already used by getCloudMetadata() elsewhere in this file. Output is written to the CI log only. No secrets, no auth, no user input.
Level of scrutiny
Low. This is pure CI observability tooling in a script file, not shipped in the Bun binary. The change is additive (extra log lines), and every failure path returns undefined, which the surrounding code already prints as-is (matching the existing Tailscale IP: and Public IP: lines). If IMDSv2-only is enforced on an EC2 agent the request 401s and prints undefined — consistent with how getCloudMetadata() already behaves in this file, and not a regression.
Other factors
- Verified the
cloudtag is set inscripts/agent.mjs:375, which Buildkite exposes asBUILDKITE_AGENT_META_DATA_CLOUD, so the gate is real. - The curl invocation mirrors the
getPublicIp()pattern immediately above it and uses the localspawnSyncwrapper, which catches spawn errors (e.g., curl missing) and surfaces them via theerrorfield rather than throwing. getCpuDescription()handlescpus()returning an empty array or a model-less entry via optional chaining.- No CODEOWNERS entry covers
scripts/. No outstanding reviewer comments.
There was a problem hiding this comment.
LGTM — CI-log diagnostics only, no runtime changes.
Checked that scripts/agent.mjs sets the cloud agent tag (line 375), which BuildKite exposes as BUILDKITE_AGENT_META_DATA_CLOUD, so the gate is live and non-cloud agents make no request. getCloudInstanceType() mirrors the adjacent getPublicIp() pattern (spawnSync curl, return undefined on error) with --connect-timeout 1 --max-time 3, so a dead metadata endpoint costs at most a few seconds. getCpuDescription() guards cpus()[0]?.model for empty results. The single CI failure (setInterval.test.js on x64-asan) is one of the hardware-sensitive flakes this PR's description tabulates and is unrelated — no src/ or test files changed.
Extended reasoning...
Overview
Adds two diagnostic lines to the Machine header that printEnvironment() prints at the top of every CI job log: CPU: (from os.cpus()[0].model + os.availableParallelism()) and, on cloud agents only, Instance Type: (from the EC2/Azure instance metadata service via a bounded curl). Two new helpers in scripts/utils.mjs and two console.log lines in printEnvironment(); nothing else touched.
Security risks
None. The metadata request targets the link-local IMDS address (169.254.169.254) that getCloudMetadata() in the same file already queries, asks only for the instance-type/vmSize field (no credentials), and only fires when BUILDKITE_AGENT_META_DATA_CLOUD is set — i.e., on agents this repo launches itself. Output goes to the job log, which already prints hostname, public IP, and the full environment.
Level of scrutiny
Low. This is CI tooling (scripts/), not shipped runtime code, and it is additive log output with graceful failure. The new code follows the exact shape of the neighbouring getPublicIp() / getTailscaleIp() helpers: spawnSync a curl, return stdout.trim() when !error, otherwise undefined. Worst-case failure mode is an undefined line in the log or a ~3s stall bounded by --max-time 3.
Other factors
Verified the cloud tag is actually set at scripts/agent.mjs:375, so the BUILDKITE_AGENT_META_DATA_CLOUD gate is real and darwin/GHA runners skip the request as claimed. cpus() returning an empty array (which can happen under some Android/container configs) is handled via optional chaining and an early return. The lone CI failure in build #94115 is test/js/web/timers/setInterval.test.js on the x64-asan lane — precisely one of the timing-sensitive tests the PR body identifies as flaking on fallback instance types, and this change modifies no source or test files.
|
Status
|
The failure annotations are what `bun run ci:errors` and the triage tooling read, so a timeout on fallback hardware now reads "on :debian: 13 x64-asan (r6i.2xlarge)" there without opening the job log. The metadata lookup is memoized since it now runs once per failure as well as once for the header.
robobun
changed the title
There was a problem hiding this comment.
LGTM — CI-only diagnostics addition.
What was reviewed:
getCpuDescription()handles emptycpus()via optional chaining;getCloudInstanceType()is memoized and only fires the curl when the agent'scloudtag isaws/azure, timing out at 3s and degrading toundefinedlike the neighboringgetTailscaleIp()/getPublicIp()lines.- Confirmed
scripts/agent.mjssets thecloudtag (exposed asBUILDKITE_AGENT_META_DATA_CLOUD), so the gate is live on the lanes described. getTestLabel()is only used in annotation HTML (flaky-in-parallel line andformatTestToMarkdown), so appending(instanceType)just changes displayed text; the earlyreturnwhengetBuildLabel()is undefined preserves prior behavior.
Extended reasoning...
Overview
Two CI-script files: scripts/utils.mjs gains getCpuDescription() (from os.cpus() + availableParallelism()) and getCloudInstanceType() (a memoized, best-effort curl to the EC2/Azure IMDS gated on BUILDKITE_AGENT_META_DATA_CLOUD), and printEnvironment() prints both in the Machine header. scripts/runner.node.mjs extends getTestLabel() to append the instance type in parentheses and swaps one getBuildLabel() call in the flaky-in-parallel annotation to getTestLabel() for consistency. No src/ or test changes.
Security risks
None. The only network call is to the link-local 169.254.169.254 IMDS endpoint, read-only, with --connect-timeout 1 --max-time 3, and only when the agent already advertises a cloud tag. The result is a short string printed to the job log. No secrets, no auth, no user input.
Level of scrutiny
Low. This is pure CI observability tooling that does not affect the runtime, bundler, tests, or any shipped artifact. The worst possible failure mode is an extra undefined line in a job log or a few seconds of curl timeout on a mis-tagged agent — both matching the existing behavior of the adjacent Tailscale IP: / Public IP: lines. The PR description shows it was verified live in build 94115 across all lanes.
Other factors
- Follows the file's established patterns exactly: module-level memoization (
priviledgedCommand,detectedCloud),spawnSync(["curl", ...])shape fromgetPublicIp(),getEnv(name, false)for optional env vars, and|| undefinedon empty trimmed output. - Verified
scripts/agent.mjs:375sets thecloudtag on agents, soBUILDKITE_AGENT_META_DATA_CLOUDis populated as claimed; darwin/GitHub Actions have no tag and skip the request. getTestLabel()callers (the flaky annotation andformatTestToMarkdown) both interpolate it into annotation HTML as display text, so the added(type)suffix is cosmetic and matches the commit message intent.- No prior human or bot review comments to address; no CODEOWNERS on
scripts/.
The launched type is a boot-time fact of the machine like kernel or abi-version, so agent.mjs records it as a launched-instance-type tag via the existing metadata helpers. That puts it in the Buildkite API for every job, and the runner reads it back from the tag's environment variable; images baked before the tag existed fall through to the direct metadata request. The key is not instance-type, which is the type the job asked for in its agent query.
robobun
changed the title
There was a problem hiding this comment.
LGTM — CI diagnostics only, no runtime or test changes.
Checked: getCloudInstanceType() memoization and the BUILDKITE_AGENT_META_DATA_CLOUD gate (darwin/GHA make no metadata request); the curl fallback is bounded (--connect-timeout 1 --max-time 3) and a failure degrades to undefined like the neighboring Tailscale IP:/Public IP: lines. getTestLabel()'s two callers (the flaky annotation and formatTestToMarkdown) both pick up the suffix. agent.mjs filters undefined tags, so a missing launched-instance-type is dropped from the tag string. getCloudLaunchedInstanceType mirrors the existing getCloudMetadataTag shape for the Azure JSON path.
Extended reasoning...
Overview
Three CI script files: scripts/utils.mjs gains getCpuDescription(), getCloudInstanceType() (memoized, env-first with a bounded curl fallback), and getCloudLaunchedInstanceType() (async, reuses getCloudMetadata); printEnvironment() prints two new lines. scripts/runner.node.mjs appends (instanceType) to getTestLabel() so failure annotations name the machine, and switches the parallel-batch flaky annotation from getBuildLabel() to getTestLabel() for consistency. scripts/agent.mjs tags each agent with launched-instance-type at start so jobs read it back from BUILDKITE_AGENT_META_DATA_LAUNCHED_INSTANCE_TYPE without a metadata call.
Security risks
None. The only new outbound request is to the link-local IMDS address (169.254.169.254), gated on the agent's own cloud tag, with --noproxy '*' and short timeouts — the same pattern already used by getPublicIp() and getCloudMetadata(). No user input reaches these paths; output is a log line and an annotation suffix.
Level of scrutiny
Low. This is CI observability tooling: it changes what appears in job log headers and BuildKite failure annotations, not what is built, run, or asserted. Worst-case failure is a 3-second stall printing the header or an undefined in a log line. The PR was verified live in this pipeline (builds 94115, 94502) across every lane and both clouds, which is the appropriate test for a change like this.
Other factors
The new helpers follow the file's existing conventions closely (spawnSync curl wrapper, getEnv(name, false), module-level memoization matching detectedCloud/priviledgedCommand, the Azure JSON-parse shape from getCloudMetadataTag). The agent-tag filter (.filter(([, value]) => value !== undefined && ...)) already drops an undefined launched-instance-type. getBuildLabel remains imported in runner.node.mjs because getTestLabel still calls it. No CODEOWNERS on scripts/. No outstanding reviewer comments.
1 participant
Problem
test/js/node/fs/fs.test.tswent red on the debian 13 x64-asan lane in build 93859:readdirSync(path, {recursive: true, withFileTypes: true} should work x 100timed out at 11.0s / 10.98s / 11.14s / 11.05s on its four attempts (10s budget). The branch under test only touchedbun:ffi.fs.test.tsas a whole: 64.7s against 39s to 42s). The agent was a slower machine than ther7i.2xlargethat.buildkite/ci.mjsasks for; under capacity pressure the launcher falls back to other types.free -mtotal in the Memory section, so finding the slow agents meant downloading and diffing the logs of every shard. The requested type is in the pipeline; the type that was actually launched was recorded nowhere.test/js/nodebecome 8). This PR is the missing diagnostics, so the next time a burst lands agents on fallback hardware it is readable off the first screen of the log.Fix
scripts/agent.mjstags each cloud agent withlaunched-instance-type, read at agent start through the existinggetCloudMetadata()path (EC2instance-type, Azurecompute.vmSize), next to thekernel/abi-version/cloudtags it already sets. Buildkite then shows the launched type per job in the API and UI (agent.meta_data), so requested (agent_query_rules, from.buildkite/ci.mjs) against launched is one API call per build, and hands it to the job asBUILDKITE_AGENT_META_DATA_LAUNCHED_INSTANCE_TYPE. It is a separate key frominstance-typeon purpose: that key is what the job asks for, and this machine may not be it.printEnvironment()(theMachinegroup at the top of every build and test job log) printsCPU: <model> x<count>fromos.cpus()/os.availableParallelism()on every platform andInstance Type:on cloud agents. The test runner puts the same type into the failure annotations, which is whatbun run ci:errorsand the triage tooling read:... - code 1 on :debian: 13 x64-asan (r6i.2xlarge), and the parallel-batch flaky variant likewise.getCloudInstanceType()falls through to asking the metadata service itself: one memoizedcurl -sf --noproxy '*' --connect-timeout 1 --max-time 3, in the same shape as thegetPublicIp()next to it, selected by thecloudtag. Agents without acloudtag (the darwin fleet, GitHub Actions) are not machines whose type CI chose; they make no request and gain only the CPU line. The fallback can go once the images have been republished.CPU: Apple M2 Ultra (Virtual) x8and no type;getCloudLaunchedInstanceType()returnsStandard_D16ds_v6on a live Azure VM both with the cloud passed in and with it detected, and the job side prefers the tag's variable when set;agent.mjsstill bundles with esbuild the way the image bake builds it;prettier --checkpasses. Nosrc/or test changes.Header and annotation output from builds 94115 and 94502
Build 94115 header lines: x64-asan shards
Instance Type: r7i.2xlarge/CPU: Intel(R) Xeon(R) Platinum 8488C x8, Windows 2019Standard_D4ds_v6/INTEL(R) XEON(R) PLATINUM 8573C x4, Windows 11 aarch64Standard_D4pds_v6/Cobalt 100 x4, alpine aarch64m8g.xlarge/Neoverse-V2 x4, the build hostr8g.4xlarge/Neoverse-V2 x16, debian x64c7i.xlargeon all 20 shards. 5 of that build's 20 asan shards were not on the requested type (4xr7a.2xlarge, 1xr6i.2xlarge), and its one failed shard (setInterval.test.jsleak test, 4 of 4 attempts) was anr7a.2xlarge.Build 94502, as printed by
bun run ci:errors 94502without any change tofind-build.ts:bun-patch.test.ts - code 1 on :windows: 11 aarch64 (Standard_D4pds_v6) (2 retries),in-process-cron.test.ts - code 1 on :ubuntu: 25.04 aarch64 (c8g.xlarge) (1 retry),inspect-error-leak.test.js - ... (in the parallel batch on :debian: 13 x64-asan (r7i.2xlarge); passed alone). The memoized in-job lookup measured 16ms for the first call and 0.002ms for the second against a live metadata service.Background
.buildkite/ci.mjsnames one instance type per lane (r7i.2xlargefor the x64 asan test lane). The launcher that creates the instances, and its list of fallback types, live outside this repo.BUILDKITE_AGENT_META_DATA_<KEY>.scripts/agent.mjsis bundled into the CI machine images, so a change there reaches the fleet at the next image publish (bump of# Version:inscripts/bootstrap.shplus a[publish images]run), not on merge.fs.test.ts, test(timers): speed up setInterval.test.js and tighten its assertions #35750setInterval.test.js, test(require-cache): measure allocator-live bytes instead of RSS in the source code leak fixtures #37586require-cache.test.ts, test(HTMLRewriter): measure kernel peak RSS, not a point sample that straddles a segment decommit #34786html-rewriter-leak, ci: pin in-process RSS leak tests against parallel-allowlist regen; re-exclude streams-leak #36867inspect-error-leak, test: make sourcetextmodule-leak actually detect leaks and run 5x faster #35917sourcetextmodule-leak)./latest/meta-data/instance-type, Azure returns the VM size at/metadata/instance/compute/vmSizewhen asked with aMetadata: trueheader.Evidence from the 2026-08-13 builds
Agents grouped by the
free -mtotal they report (the memory map differs per instance family), with the median slowdown of each shard's sequential test files against the same files in builds 93850, 93854, 93922 and 93934 (79 of those 80 asan agents were the 63270 class, the other a 62932, and nothing failed). The instance types in the first column are the ones build 94115's new header has confirmed so far; the other three classes will name themselves the next time a burst runs with this header.free -mtotalr7i.2xlarge(requested; 20/20 agents in 93922, 93934, 93960, 93986)r7a.2xlarge(AMD EPYC 9R14)timers/setInterval.test.jsleak test (6 sightings here, 8 more on the three slow classes below, none onr7i)r6i.2xlarge(Xeon 8375C)run/require-cache.test.ts,html-rewriter-leak.test.tsrequire-cache,inspect-error-leak,setIntervalfs/fs.test.ts(7 of 7 sightings on this class),html-rewriter-leak,inspect-error-leak,sourcetextmodule-leak,setInterval,require-cacheinspect-error-leak,sourcetextmodule-leak,setInterval,require-cache, elysia stream testsBuild 93859's four slow shards (0, 1, 3, 15) were all 63510; shard 15 carried
fs.test.ts(64.7s; every test file on it 1.3x to 2.1x slower, median 1.41x). In the 03:28 to 03:39 UTC burst (builds 94018 to 94063) 44 of the 46 failed asan shards were on a non-63270 class, and builds in that burst had only 2 to 4 of their 20 asan agents on 63270 (94024: 3, 94044: 4, 94059: 2).fs.test.tsfailed the same way in 94019, 94021, 94024, 94025, 94031 and 94058, each time on a 63510 agent. With #37828 applied the file takes 20.6s on the lane (build 93509) and no test in it is near a timeout.