flaky: live Git Bash background job resolves with no handle on windows-latest

[SUaDtL]

The flake

test/background-jobs.test.ts > session-local background job state > launches and disposes a real Git Bash background job fails intermittently on windows-latest:

AssertionError: expected undefined to be defined
  at background-jobs.test.ts:603

Observed on PR #503, Pi 0.80.10 / windows-latest. A re-run of the identical commit passed, so it is intermittent rather than a regression. The PR that surfaced it changed only Markdown, CHANGELOGs and version strings — no executable code — so nothing in the diff can reach a test that spawns a real Git Bash process.

Why this is NOT the same bug as #495

Worth separating, because the obvious reading is wrong and would lead to the wrong fix.

#495 was a deadline problem: a 5s vitest timeout could not cover a cold CPython launch, and the symptom was Test timed out in 5000ms.

This one is not a timeout at all. The assertion that fails is:

const job = await awaitLiveCondition(
  runtime.launch({ ... }),
  "the live Git Bash launch",
  () => "no job handle - the contained launch never returned",
);
expect(job).toBeDefined();          // <- line 603, this is what fails

awaitLiveCondition rejects with a descriptive did not settle within Nms error when it times out. That error was not raised. So the helper resolved — with undefined — comfortably inside its 60s WINDOWS_LIVE_TEST_TIMEOUT_MS budget.

In other words: runtime.launch(...) returned no job handle, promptly and without erroring. Raising the timeout would change nothing.

What that points at

A launch that resolves to undefined rather than throwing is a spawn that did not produce a tracked job. Candidates worth checking, in rough order:

1. shellPath resolution. The test resolves a real Git Bash. If that lookup returns a path that is momentarily unusable on the runner image, a launch could fail soft.
2. createBackgroundJobRuntime(...) returning a runtime whose launch can resolve empty — note the ! on its construction; if the failure is upstream of launch, the handle is simply never registered.
3. A race between process start and job-table registration — the handle is read before the tracker has recorded it, so the read is legitimately empty rather than late.

(3) would be the most concerning of the three, because it means the runtime can report success while owning nothing — and this is the module that tracks background jobs for cleanup, so an untracked job is a leaked process.

Why it matters beyond a re-run

[GATE ] | [REPO] | Merge readiness aggregates this check, and the release preflight requires that gate green for the exact commit being tagged (#385). A flake here blocks the release lane for that SHA, and it trains readers to re-run a red gate without reading it — the habit the gate exists to prevent. It cost one re-run cycle on #503.

Acceptance criteria

• AC-1: the root cause is identified — specifically, whether launch can resolve undefined on a successful spawn (a tracking race) or only on a failed one.
• AC-2: if a launch cannot produce a handle, it fails loudly rather than resolving empty; expect(job).toBeDefined() should never be the thing that discovers it.
• AC-3: if it is a registration race, the job table is proven to contain the job before launch resolves — an untracked background job is a leaked process, which is what this module exists to prevent.
• AC-4: the test passes across repeated windows-latest runs without a re-run.

Relevant sources

• plugins/ca-pi/tools/test/background-jobs.test.ts:588-606 (runLiveGitBashJob, awaitLiveCondition)
• the createBackgroundJobRuntime / launch implementation under plugins/ca-pi/tools/src/
• Related but distinct: flaky: pi security-evidence test times out on windows-latest (5s deadline covers a cold Python launch) #495 (a genuine deadline, already fixed)

[SUaDtL]

AC-1 answered: it is a designed refusal, not a tracking race

I filed this with three candidate causes and flagged (3) — a registration race, where the runtime reports success while owning nothing — as the concerning one, since an untracked background job is a leaked process.

That branch is ruled out. Reading plugins/ca-pi/tools/src/background-jobs.ts:

async launch(input: BackgroundJobLaunchInput): Promise<Readonly<BackgroundJobSnapshot> | undefined>

undefined is a declared outcome, not an accident. There are five refusal paths, and every one of them transitions the job to a terminal state before returning:

#	condition	disposition
1	runtime disposed / stopping / unhealthy	no job created
2	unparseable input, or a stale authorization lease	no job created
3	manager.createJob refused	no job created
4	authorization went stale after publish	job → cancelled, published
5	#openTree threw	job → failed, published, ownership released

Path 5 is the one that fired on CI: the Git Bash spawn threw, launch refused correctly, released ownership, and marked the job failed. The runtime is fail-closed and behaving as designed — it never owns an untracked process. AC-3 as originally written does not apply.

What is actually wrong

The refusal is correct; the diagnosis is impossible.

} catch {
  releaseOwnership();
  this.#pendingOwnership.delete(ownership);
  const terminal = this.#manager.transitionJob({ id: job.id, state: "failed" });
  if (terminal !== undefined) this.#publish(terminal, "completed");
  return undefined;
}

The spawn error is discarded — not bound, not logged, not carried on the terminal snapshot. So a real environment failure (a Git Bash path that was momentarily unusable, a transient EBUSY, a permissions blip) is indistinguishable from every other refusal, and reaches the operator as:

AssertionError: expected undefined to be defined

The test's own diagnostic compounds it — "no job handle - the contained launch never returned" — which is false. The launch did return; it returned a refusal. So the one message a reader gets actively misdescribes what happened.

There are 13 bare catch { blocks in this module. Several are legitimately "producer behaviour is authoritative", but this one sits on the path where a real spawn failure is the most likely cause and the least visible.

Revised acceptance criteria

• AC-1 answered: designed refusal on spawn failure (path 5); no tracking race.
• AC-3 withdrawn: the job table is never left holding an untracked process — every refusal transitions to a terminal state first.
• AC-2 (revised, and now the substance of this issue): a launch that fails to spawn must carry why. The activity event shape is {kind, id, label, state} with no detail field, so this needs a deliberate call about the contract — extend the event, add a diagnostics channel, or reject rather than resolve on path 5 specifically. Path 5 is distinguishable from paths 1-4: those are policy refusals the caller can predict, this one is an environment failure the caller cannot.
• AC-4 (unchanged): the test passes across repeated windows-latest runs. Note that fixing the diagnostics does not by itself fix the flake — it makes the next occurrence explain itself instead of asserting undefined.

Not fixed here

Surfacing the reason means changing a published contract (the activity event) or the refusal semantics of one path. Both deserve a decision rather than being slipped in behind a flake fix, so this is diagnosis only.

[SUaDtL]

AC-2 addressed in #508; AC-4 remains open and this issue stays open on it.

• AC-1 answered previously: designed refusal on spawn failure, no tracking race.
• AC-2 done. The spawn path now reports a per-launch diagnostic drawn from a closed vocabulary — a recognized Windows containment reason via Windows ca-pi adapter suites flake on hard 5s timing budgets #428's windowsRefusalReasonFromMessage(), or an errno shape — and the background bash tool surfaces it where it already asked health().diagnostic and got nothing. Per-launch rather than runtime-scoped because MAX_ACTIVE_JOBS is 4 and a shared slot would report one launch's environment failure as another's.
• AC-3 withdrawn previously.
• AC-4 open. Diagnostics do not stop the flake. The next windows-latest occurrence will name its cause instead of asserting on undefined; until a real run does that, or the flake stops recurring, AC-4 is unproven.

Rejected along the way: extending the activity event (display-only by contract, control-stripped, 128-code-point cap), and rejecting on path 5 (splits the return contract for the sole caller and ~20 assertions). A bounded raw error.message was the richest option but the channel is open by construction — realpathSync throws already embed the shell path and cwd.