Skip to content

ci: kill in-flight test subprocesses on runner SIGINT/SIGTERM - #29317

Open
alii wants to merge 4 commits into
claude/tart-mjs-fixesfrom
claude/runner-pgroup-kill
Open

ci: kill in-flight test subprocesses on runner SIGINT/SIGTERM#29317
alii wants to merge 4 commits into
claude/tart-mjs-fixesfrom
claude/runner-pgroup-kill

Conversation

@alii

@alii alii commented Apr 14, 2026

Copy link
Copy Markdown
Member

Summary

Track spawnSafe's in-flight subprocesses in a module-level Set, and on SIGINT/SIGTERM/SIGHUP kill them (taskkill /T on Windows, proc.kill(9) on POSIX) before exiting. Currently onExit() just calls process.exit() without killing anything, so a Buildkite job-cancel (or local Ctrl+C) orphans the running test process.

This is the conservative subset of the original process-group-kill attempt — no spawn-option changes (no detached, no kill(-pid)), so it won't reap POSIX grandchildren on the timeout path. The full process-group story can follow separately once verified on Linux CI.

The activeSubprocesses Set is declared above the first top-level await spawnSafe(...) (the Linux-only --coredump-upload sysctl probe at ~line 223). spawnSafe is a hoisted function declaration; with the const next to spawnSafe lower in the file, Linux CI hit ReferenceError: Cannot access 'activeSubprocesses' before initialization before any tests ran.

Part 3 of the macOS-CI Phase 0 stack (depends on #29315#29314), though this change is platform-agnostic.

Test plan

  • node --check scripts/runner.node.mjs
  • Local smoke: node scripts/runner.node.mjs --exec-path=$(which bun) --include="which.test" runs and passes
  • --coredump-upload path reaches the sysctl call (no TDZ)
  • Buildkite: Linux test-bun lanes run for real (~5min, not 10s)

@robobun

robobun commented Apr 14, 2026

Copy link
Copy Markdown
Collaborator
Updated 7:23 PM PT - Apr 16th, 2026

@robobun, your commit 537efef has 3 failures in Build #45950 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 29317

That installs a local version of the PR into your bun-29317 executable, so you can run:

bun-29317 --bun

Comment thread scripts/runner.node.mjs Outdated
Comment thread scripts/runner.node.mjs Outdated

@Jarred-Sumner Jarred-Sumner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The fallback will always fail or (worse) kill an unrelated process
  • On Windows, this isn't necessary because processes by default killall their children on exit

Comment thread scripts/runner.node.mjs
@alii
alii force-pushed the claude/tart-mjs-fixes branch from ab88002 to 9d417a6 Compare April 15, 2026 00:20
@alii
alii force-pushed the claude/runner-pgroup-kill branch from 483a2be to af75f88 Compare April 15, 2026 00:20
@alii
alii force-pushed the claude/tart-mjs-fixes branch from 9d417a6 to 9340cb3 Compare April 15, 2026 00:26
@alii
alii force-pushed the claude/runner-pgroup-kill branch from 86708d1 to cd80459 Compare April 15, 2026 00:26
Comment thread scripts/runner.node.mjs Outdated
@alii
alii requested a review from Jarred-Sumner April 15, 2026 00:42
Comment thread scripts/runner.node.mjs
Comment thread scripts/runner.node.mjs Outdated
Comment thread scripts/runner.node.mjs
Comment thread scripts/runner.node.mjs
Comment thread scripts/runner.node.mjs Outdated
Comment thread scripts/runner.node.mjs Outdated
@alii alii changed the title ci: kill the whole process group when a test times out ci: kill in-flight test subprocesses on runner SIGINT/SIGTERM Apr 15, 2026
Comment thread scripts/runner.node.mjs Outdated
@alii
alii marked this pull request as draft April 15, 2026 07:09
@alii
alii force-pushed the claude/runner-pgroup-kill branch from abbeb33 to cde4098 Compare April 15, 2026 18:38
@alii
alii marked this pull request as ready for review April 15, 2026 18:39
@alii
alii force-pushed the claude/runner-pgroup-kill branch from cde4098 to 597882c Compare April 15, 2026 18:39
Comment thread scripts/runner.node.mjs Outdated
Comment thread scripts/runner.node.mjs
Comment thread scripts/runner.node.mjs Outdated
@alii
alii force-pushed the claude/tart-mjs-fixes branch from 3e59d40 to d29f055 Compare April 15, 2026 19:52
@alii
alii force-pushed the claude/runner-pgroup-kill branch from 3bd443e to 6fe0ef8 Compare April 15, 2026 19:53

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — all critical issues from prior rounds are addressed. The remaining sequential-taskkill nit in parallel mode is bounded and only pathological.

Extended reasoning...

Overview

The PR adds a module-level activeSubprocesses Set, populated on the async 'spawn' event and cleared in done(), and iterates it in onExit() to kill in-flight test subprocesses when the runner receives SIGINT/SIGTERM/SIGHUP. Single file changed: scripts/runner.node.mjs (CI tooling only, not production Bun code).

Security risks

None. This is CI runner cleanup logic with no auth, crypto, or data-exposure surface.

Level of scrutiny

Low-to-medium. The change is CI infrastructure (runner.node.mjs), not the Bun runtime itself. The logic is straightforward signal-handler cleanup. The PR description is transparent about the POSIX grandchild limitation being deliberate.

Other factors

All 🔴 issues from prior review rounds are resolved in the committed code: the TDZ placement is correct (line 225, before the first top-level await spawnSafe), a Set is used instead of a single variable, and the taskkill call has timeout: 5000. The one open 🟡 nit (sequential spawnSync calls in --parallel mode) is minor: worst-case exit delay is bounded at N×5 s, only possible under pathological AV-hook conditions, and --parallel defaults to false. The conservative design (no detached, no process.kill(-pid)) is intentional and documented.

alii added 3 commits April 15, 2026 13:11
Track spawnSafe's in-flight subprocesses in a module-level Set, and
on SIGINT/SIGTERM/SIGHUP kill them (taskkill /T on Windows, kill(9)
on POSIX) before exiting. Currently onExit() just calls process.exit()
without killing anything, so a Buildkite job-cancel orphans the
running test process.

This is the conservative subset of the original process-group-kill
attempt — no spawn-option changes (no `detached`, no `kill(-pid)`),
so it won't reap POSIX grandchildren on the timeout path; it only
handles the cancel path. The full process-group story can follow in
a separate PR.

The activeSubprocesses Set is declared above the first top-level
`await spawnSafe(...)` (the Linux-only coredump sysctl probe at
~line 223). spawnSafe is a hoisted function declaration; with the
const placed next to spawnSafe lower in the file, Linux CI hit a TDZ
ReferenceError before any tests ran.
spawnSync serialized N×5s worst-case under --parallel (8 on Windows).
spawn() with detached+unref launches each taskkill as an independent
process that outlives our process.exit() — CreateProcess is
synchronous inside spawn() so the child exists before it returns.
@alii
alii force-pushed the claude/runner-pgroup-kill branch from 6fe0ef8 to 8c0034a Compare April 15, 2026 20:13
@alii

alii commented Apr 16, 2026

Copy link
Copy Markdown
Member Author

@robobun adopt

@robobun

robobun commented Apr 16, 2026

Copy link
Copy Markdown
Collaborator

Adopted. Pushed 537efef addressing the review:

  • Windows taskkill removed — children spawned without detached are placed in the runner's Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, so process.exit() already reaps the whole tree. No by-PID kill on Windows at all now, so no reused-PID risk.
  • No by-PID fallback — POSIX path is just ChildProcess.kill(9) on the live handle, which Node guards against stale PIDs internally. (The process.kill(-pid) + fallback from the original revision was already dropped in the earlier rewrite.)

Verified locally: with the base-branch runner, SIGTERM to the runner leaves the in-flight bun test process alive; with this branch it's reaped.


CI on 537efef (#45950) — all three hard failures are unrelated to this runner signal-handler change:

test platform cause
bun-install-lifecycle-scripts.test.ts win11-aarch64 load-sensitive cpuTime.total < 750_000 threshold (got 937500)
webview-chrome.test.ts darwin-14-aarch64 Chrome not installed on runner — same failure on base #45945
bun-add.test.ts darwin-13-aarch64 GitHub API 504 fetching dylan-conway/install-test-3 tarball

This PR only touches onExit(), which fires on SIGINT/SIGTERM/SIGHUP — never during a normal test run — and on Windows now does strictly less than before. The stack's base branches (#45951, #45945) are independently red. Needs a retry (I lack write_builds).

On Windows, children spawned without `detached` are placed in the
runner's Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, so
process.exit() already terminates the whole tree. The taskkill by PID
was redundant and, in the race where the child exits first, could hit
an unrelated reused PID.

POSIX keeps the explicit proc.kill(9) via the ChildProcess handle
(safe against PID reuse) since Buildkite cancel signals only the
runner PID, not the process group.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants