fix(reliability): stop Windows mid-run deaths (keep-awake + real --detach) - #2063
Conversation
Unattended Windows hosts drop into Modern Standby when the screen turns off. A standby-frozen executor thaws into bash spawns that exit 66 with no output, collapsing the DAG tail; other runs become zombie 'running' rows. Hold a per-thread SetThreadExecutionState(ES_CONTINUOUS | ES_SYSTEM_REQUIRED) request for exactly the executing window so the system stays awake while at least one run is active. Refcounted so concurrent runs in one server process share a single request until the last finishes. Best-effort: no-op off Windows or on FFI failure, since a keep-awake failure must never block a run and the OS clears the state on process exit. Acquire sits directly above the run's try so early-return paths can't leak an unpaired acquire; release is the first statement of that finally so it pairs on every exit path.
Reference dossier cataloging the Windows Modern Standby failures (exit-66 bash-tail collapse, zombie 'running' runs) that motivate the keep-awake change. Captures the evidence, incidences, and confirmed-vs-hypothesized causes in one place.
spawnDetachedWorkflowRun used Bun.spawn(...) + child.unref(). unref()
stops the parent waiting, but does NOT put the child in its own process
group, so when the launching shell/console tore down (right after the
parent returned) Windows killed the "detached" child with it — the run
died ~1s in at worktree_creating and forced a manual finish. Evidence:
detach logs stop dead mid-line at worktree_creating with no error.
Switch to node:child_process spawn(cmd[0], cmd.slice(1), { detached:
true, windowsHide: true }). detached makes the child a new process-group
leader so it survives the parent's exit; mirrors setup.ts's proven
spawn(..., { detached: true }) pattern already in this codebase. Only the
spawn mechanism changes — buildDetachedRunCmd (the unit-tested argv
builder) and the log-fd handling are untouched.
Docs: 2026-07-07-archon-detach-fix.md records the root cause, the live
verification path (a --detach run that survives past worktree_creating),
and the start /b Job-Object fallback if detached alone proves insufficient.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughChangesWorkflow keep-awake lifecycle
Detached workflow spawning
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)Keep-awake lifecyclesequenceDiagram
participant workflowRun as executeWorkflow
participant keepAwake
participant kernel32
workflowRun->>keepAwake: acquire()
keepAwake->>kernel32: SetThreadExecutionState(acquire flags)
workflowRun->>workflowRun: execute workflow
workflowRun->>keepAwake: release() in finally
keepAwake->>kernel32: SetThreadExecutionState(clear flags)
Detached workflow launchsequenceDiagram
participant command as workflowRunCommand
participant child as Node child process
participant log as conversation log
command->>child: spawn command with cwd, env, and detached options
child->>log: redirect stdout and stderr
child-->>command: pid or spawn error
command-->>command: success acknowledgment or startup failure
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/2026-07-07-archon-detach-fix.md`:
- Around line 118-131: The verification step is parsing CLI output with brittle
text matching, which can break if formatting changes. Update the Task 2 commands
to use a JSON-aware parser for the run ID and status checks instead of grep/cut,
and keep the logic anchored around the workflow runs and workflow get commands
so the checks remain deterministic even if output formatting shifts.
In `@packages/cli/src/commands/workflow.ts`:
- Around line 195-208: The detached child created in workflow.ts via spawn() can
still emit an asynchronous error before or after unref(), which may crash the
parent CLI if unhandled. Add an error listener on the ChildProcess immediately
after creation and before calling child.unref(), and log the spawn failure with
enough context for ENOENT or invalid cwd cases instead of allowing an uncaught
exception.
In `@packages/workflows/src/utils/keep-awake.ts`:
- Around line 67-101: Catch FFI exceptions in createKeepAwake so errors from
native(ACQUIRE_FLAGS) and native(RELEASE_FLAGS) cannot escape into executor.ts
and abort a workflow run. Update the acquire() and release() methods to wrap the
native calls in try/catch, keep the refcount behavior intact, and log the
failure through getLog() with clear symbols like createKeepAwake, acquire, and
release.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7184d01c-5e21-4bf2-a724-8ddb3705fb1e
📒 Files selected for processing (6)
docs/2026-07-05-archon-mid-run-death-problem-record.mddocs/2026-07-07-archon-detach-fix.mdpackages/cli/src/commands/workflow.tspackages/workflows/src/executor.tspackages/workflows/src/utils/keep-awake.test.tspackages/workflows/src/utils/keep-awake.ts
| - [ ] **Step 2: Wait ~90s, then confirm the child SURVIVED past the death point.** Tail the newest detach log: | ||
|
|
||
| ```bash | ||
| newest=$(ls -t ~/.archon/logs/detached-run-*.log | head -1); echo "$newest"; tail -20 "$newest" | ||
| ``` | ||
|
|
||
| **PASS** = the log continues well past `worktree_creating` (you'll see `worktree_created`, `bootstrap`, migrations, etc.). **FAIL** = it still stops dead at `worktree_creating` → go to Task 3. | ||
|
|
||
| - [ ] **Step 3: Corroborate via the run row.** | ||
|
|
||
| ```bash | ||
| rid=$(archon workflow runs --json --limit 1 | grep -oE '"id":"[a-f0-9]+"' | head -1 | cut -d'"' -f4) | ||
| archon workflow get "$rid" --json | grep -E '"status"|"last_activity_at"' | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Make the verification step parse JSON structurally.
The current ls | grep | cut workflow is brittle and can break if the CLI output formatting changes. Please use a JSON-aware parser for the run ID and status checks so Task 2 stays deterministic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/2026-07-07-archon-detach-fix.md` around lines 118 - 131, The
verification step is parsing CLI output with brittle text matching, which can
break if formatting changes. Update the Task 2 commands to use a JSON-aware
parser for the run ID and status checks instead of grep/cut, and keep the logic
anchored around the workflow runs and workflow get commands so the checks remain
deterministic even if output formatting shifts.
Wirasm
left a comment
There was a problem hiding this comment.
pr: 2063
title: "fix(reliability): stop Windows mid-run deaths (keep-awake + real --detach)"
author: "buun-dev"
reviewed: 2026-07-13
recommendation: request-changes
PR Review: #2063 - fix(reliability): stop Windows mid-run deaths (keep-awake + real --detach)
Author: @buun-dev
Branch: feat/keep-awake-during-runs -> dev
Files Changed: 6 (+722/-3)
Summary
Two complementary Windows-reliability fixes: (1) a refcounted keep-awake controller (SetThreadExecutionState via bun:ffi) held for exactly the executing window of executeWorkflow, and (2) replacing Bun.spawn + unref() with Node's spawn(..., { detached: true, windowsHide: true }) so --detach children survive launcher teardown. The code changes are small, well-commented, well-tested, and correctly scoped. Two of CodeRabbit's three findings from the 2026-07-07 review are real and remain unaddressed on the current head (2800552b), and the two ~450-line "dossier" docs are private-environment artifacts that shouldn't land in docs/ verbatim.
Implementation Context
| Artifact | Path |
|---|---|
| Implementation Report | Not found (external contribution, not via prp-implement) |
| Original Plan | The PR itself ships its plan as docs/2026-07-07-archon-detach-fix.md |
| Documented Deviations | PR body explicitly flags the two dossiers' differing root causes as intentional |
The PR body is unusually thorough (root-cause evidence, rollback plan, explicit "not verified" section). The author openly states the live --detach survival smoketest and a real Modern-Standby cycle were not verified — both require an unattended Windows host.
Changes Overview
| File | Changes | Assessment |
|---|---|---|
packages/workflows/src/utils/keep-awake.ts |
+137 | WARN — unguarded native calls contradict stated contract |
packages/workflows/src/utils/keep-awake.test.ts |
+110 | PASS — good DI-based coverage, runs in the src/utils/ batch |
packages/workflows/src/executor.ts |
+14/-1 | PASS — correct placement (post-validation acquire, release first in finally) |
packages/cli/src/commands/workflow.ts |
+8/-2 | WARN — missing 'error' listener on the detached child |
docs/2026-07-05-…problem-record.md |
+244 | FAIL — private-environment dossier, dangling references |
docs/2026-07-07-archon-detach-fix.md |
+207 | FAIL — task prompt for the author's own agent, private paths |
Issues Found
Critical
No critical issues found.
High Priority
packages/cli/src/commands/workflow.ts:201-208— No'error'listener on the spawned child (unaddressed CodeRabbit finding).- Why: Node's
spawn()reports ENOENT / EACCES / invalid-cwdasynchronously via an'error'event. Unhandled, that becomes an uncaught exception that crashes the CLI — after it has already printedStarted '<workflow>' in the background.(orok: trueJSON atworkflow.ts:672-695). The replacedBun.spawnthrew synchronously, so spawn failures previously surfaced through the CLI's normal error path before any success output. This is a real error-handling regression, and it hits--jsonconsumers hardest (success line emitted, then a stack trace + non-zero exit). - Fix: Attach
child.on('error', err => getLog().error({ err, cmd: cmd[0] }, 'cli.detached_run_spawn_failed'))immediately afterspawn(), beforeunref(). Printing success-before-confirmation is inherent to async detach, but the crash and the silent log gap are not.
- Why: Node's
Medium Priority
-
packages/workflows/src/utils/keep-awake.ts:79-97—acquire()/release()call the FFI function unguarded (unaddressed CodeRabbit finding).- Why: The module header promises "a keep-awake failure must NEVER block or fail a workflow run," and the PR body claims failures are "wrapped in try/catch" — but only
dlopen(load time) and the0return value are handled. A native call that throws inacquire()escapesexecuteWorkflowatexecutor.ts:724(before the try), leaving the already-created run row stuckrunning— the exact zombie class this PR fights. A throw inrelease()atexecutor.ts:993(first statement of the finally) skips the zombie backstop and masks the in-flight return value. Abun:ffiu32 call throwing is unlikely in practice, but the stated invariant should be enforced, not assumed. - Fix: Wrap both
native(...)calls in try/catch, log viagetLog().warn(...), keep refcount behavior intact. ~6 lines; the existing tests still pass and a "native throws" test becomes trivial to add.
- Why: The module header promises "a keep-awake failure must NEVER block or fail a workflow run," and the PR body claims failures are "wrapped in try/catch" — but only
-
docs/2026-07-05-…problem-record.md+docs/2026-07-07-archon-detach-fix.md— Private-environment dossiers committed as project docs.- Why: Both files reference paths that don't exist in this repo (
.agents/plans/…,docs/retros/worked-failures.md,.claude/scripts/heartbeat.py,docs/superpowers/specs/…,MarphobBrain/Memory/…) and hardcode the author's machine layout (D:\Project\Archon-template\…). The detach doc is explicitly a task prompt for the author's own agent ("For the human executor… You said you'll run this yourself"). Additionally, this PR creates the rootdocs/directory — which is Archon's default$DOCS_DIR, injected into workflow prompts — so these dossiers become ambient "project documentation" for every workflow run that references$DOCS_DIR. - Fix: The root-cause evidence is genuinely valuable — preserve it, but not verbatim in
docs/. Preferred: move both dossiers into a GitHub issue (the PR body already offers to file one) and, if desired, keep a short distilled root-cause note in the repo. At minimum, strip the private paths/references and the executor-task framing.
- Why: Both files reference paths that don't exist in this repo (
Suggestions
packages/workflows/src/utils/keep-awake.ts:52-56— The "no module-load side effects" claim on the lazy logger is slightly undercut byloadNative()'s catch path callinggetLog()at import time (Windows FFI-failure case only). Cosmetic; fix the comment or accept it.packages/workflows/src/executor.test.ts— Consider one assertion thatkeepAwake.activeCount()returns to 0 afterexecuteWorkflowcompletes (success and failure paths), since the singleton is importable. Cheap insurance against a future early-return between acquire and try.- Verification debt (author-acknowledged): the live Windows
--detachsurvival smoketest (detach doc, Task 2) and a real Modern-Standby cycle are still pending. I verified locally that Bun'snode:child_processshim honorsdetached: trueon POSIX (child becomes its own process-group leader), so the option is plumbed through — but the WindowsDETACHED_PROCESSbehavior should be confirmed on target hardware before this is trusted for unattended use. Thestart /bJob-Object fallback is already documented if it isn't enough.
Validation Results
Run on macOS (Darwin) in a clean worktree at 2800552b:
| Check | Status | Details |
|---|---|---|
| Type Check | PASS | all 10 packages, exit 0 |
| Lint | PASS | eslint clean (--max-warnings 0) |
| Format | PASS | prettier clean |
Tests (@archon/workflows) |
PASS | all batches green, incl. 7 keep-awake tests |
Tests (@archon/cli) |
PASS | 43 pass / 0 fail |
| Generated files | PASS | check:bundled, check:bundled-schema, check:pi-vendor-map all up to date |
The two symlink EPERM failures the author reported are Windows-host-specific (Developer Mode) and do not reproduce here — everything is green on macOS, confirming they're environmental, not regressions.
Pattern Compliance
- Follows existing code structure (utils placement, DI factory + singleton, logger conventions)
- Type safety maintained (no
any, explicit return types, SDK/FFI types viabun:ffi) - Naming conventions followed (
keepawake.acquire_failed,_completed/_failedpairing) - Tests added for new code (keep-awake; picked up by the existing
src/utils/test batch — no mock.module, no isolation concerns) - Documentation — the two dossiers need relocation/trimming (see Medium)
- Fail-fast exception documented per CLAUDE.md ("intentional, documented fallback") — but see the Medium finding: the code doesn't yet fully enforce it
What's Good
- Exemplary root-cause discipline: exit-66 correlated to Kernel-Power events; the detach failure evidenced by a log frozen mid-line; the two investigations honestly kept separate instead of merged into a false single narrative — and the PR body flags that tension up front.
- keep-awake design is clean: dependency-injected native fn + platform (testable without Windows), refcounting for concurrent runs,
ES_SYSTEM_REQUIREDwithoutES_DISPLAY_REQUIRED, correct>>> 0uint32 coercion with an explanation, OS-clears-on-exit as the crash backstop, and the single-thread constraint documented. - Executor wiring is exactly right: acquire placed after all early-return validation paths so it can't leak, release as the first statement of the finally with the reasoning written down.
- Surgical scope: the unit-tested
buildDetachedRunCmdargv builder untouched; no schema/config/env changes; honest "what was NOT verified" section and a concrete rollback plan.
Recommendation
REQUEST CHANGES
Three items before merge, all small:
- Add the
'error'listener on the detached child (High — real error-handling regression, ~4 lines). - Wrap the FFI calls in
acquire()/release()in try/catch to actually enforce the stated never-fail contract (Medium — ~6 lines; also makes the PR body's claim true). - Move the two dossiers out of
docs/(into a tracking issue, ideally with a distilled root-cause note left behind) and strip the private-machine paths/references.
Items 1 and 2 were already raised by CodeRabbit on this same head commit and are still unaddressed. The core approach is sound and the code quality is high — once these land, this is a valuable reliability fix. A maintainer-side Windows smoketest of --detach survival before release would close the remaining verification gap.
Reviewed by Claude
Report: .claude/PRPs/reviews/pr-2063-review.md
PR Review SummaryMulti-agent review (7 specialist agents: code-reviewer, silent-failure-hunter, pr-test-analyzer, comment-analyzer, type-design-analyzer, docs-impact, code-simplifier) of head Critical Issues (2 found)
Important Issues (7 found)
Suggestions (8 found)
Strengths
Documentation Issues
VerdictNEEDS FIXES Recommended Actions
Multi-agent review dispatched via prp-review --agents; all agents advisory, no files modified. |
Review follow-ups on coleam00#2063 (CodeRabbit + multi-agent review): - keep-awake: wrap both native SetThreadExecutionState calls in try/catch so a throwing FFI call can never escape into executeWorkflow (acquire sits before the run's try — a throw left a zombie 'running' row; release is first in its finally — a throw replaced the run's return value and skipped the zombie backstop) - keep-awake: check release()'s return value — a failed clear pins ES_SYSTEM_REQUIRED until process exit; log keepawake.release_failed instead of unconditionally logging success - keep-awake: include a stack in the release_unbalanced warn so an unbalanced call site is findable - cli --detach: add an 'error' listener on the spawned child (Node spawn reports ENOENT/bad-cwd asynchronously — unhandled it crashed the CLI after the success ack) and fail fast on undefined child.pid (same check as setup.ts trySpawn) so a spawn failure can't print a false "Started" - cli --detach: warn in human mode when no log file could be opened (previously the degraded no-trail state was invisible) - comments: align overclaims with reality (the "never fail a run" contract is now enforced, not asserted; lazy-logger side-effect caveat; the single-thread claim scoped to this codebase, not Bun; "proven pattern" softened with the Job-Object caveat); best-effort rationale stated once and referenced instead of restated three times - tests: assert detached/windowsHide on the spawn options (the actual fix was previously unasserted), add the spawn-failure path, a throwing native-fn keep-awake test, and executor acquire/release pairing tests (success, DAG throw, blocked early return) - docs: remove the two investigation dossiers from docs/ (they referenced the author's private environment; evidence preserved in coleam00#2080 and in this PR's file history at 2800552) and document the keep-awake behavior on the Windows deployment page Deferred with rationale: - acquire() boolean return for executor-side run-id logging: ambiguous semantics on the disabled/no-op paths; keep-awake already logs failures - mock.module('node:child_process') test refactor: the Bun.spawn spy exercises the real shim path and now asserts the detach options - makeKeepAwake() test helper and a CLAUDE.md spawn-guidance bullet: below rule-of-three; maintainer's call Related coleam00#2080
|
Maintainer follow-up: I've pushed Addressed:
Deferred (with rationale):
Validation (macOS): type-check all packages, eslint ( Still open before merge: the live Windows smoketests (detach survival past |
Conflicts: executor.test.ts import hunks (spyOn/keepAwake vs the folder-projects join/resolveProjectPaths imports — union of both). Semantic fix: the folder-project detach test added by coleam00#2055 spawns with a fake lacking `pid`, which now trips the fail-fast undefined-pid check — gave its fake a pid like the other detach tests.
Summary
Describe this PR in 2-5 bullets:
bashchild the executor spawns exits 66 (EX_NOINPUT) with no stdout/stderr, collapsing the whole CI/ledger tail and leavingfailed/zombierunningrows; (2)archon workflow run --detachdoesn't actually detach — the "background" child dies ~1s in atworktree_creating, forcing a manual finish. Historically ~44% of runs never completed (209 completed / 82 cancelled / 81 failed at one capture).paused,archon workflow approve, or manual-finish from the worktree).SetThreadExecutionState(ES_CONTINUOUS | ES_SYSTEM_REQUIRED)request for exactly the executing window so the system stays awake while ≥1 run is active (refcounted across concurrent runs; best-effort no-op off Windows / on FFI failure). (2)--detachfix — replaceBun.spawn(...) + unref()with Node'sspawn(..., { detached: true, windowsHide: true })so the detached child becomes a new process-group leader and survives its launcher's teardown.buildDetachedRunCmd(the pure, unit-tested argv builder) is untouched — only the spawn mechanism changes. No schema, DB, config, or env changes. keep-awake never blocks a run: acquire/release wrap the run'stry/finally, failures are swallowed, and the OS clears the execution state on process exit.UX Journey
Before
After
Architecture Diagram
Before
After
Connection inventory:
cli/workflow.ts:spawnDetachedWorkflowRunBun.spawncli/workflow.ts:spawnDetachedWorkflowRunnode:child_process spawndetached: true, windowsHide: truecli/workflow.ts:spawnDetachedWorkflowRunbuildDetachedRunCmdworkflows/executor.ts:executeWorkflowutils/keep-awake.tstry, release first infinallyworkflows/utils/keep-awake.tskernel32!SetThreadExecutionState(bun:ffi)Label Snapshot
risk: medium— changes the process-spawn lifecycle for every--detachrun on all platforms and wraps everyexecuteWorkflowwith acquire/release; mitigated by mirroring an in-repo proven pattern and best-effort no-op semantics.size: M(~270 lines code across 3 files; ~450 lines of root-cause docs)workflows,cli,docsworkflows:executor,workflows:keep-awake,cli:workflowChange Metadata
bug(primary: stop mid-run deaths) — includes one new capability (keep-awake) +docsmultiLinked Issue
Validation Evidence (required)
Targeted checks for the touched packages (
@archon/cli,@archon/workflows):Full-suite
bun run validate:Touched-package suites, run in isolation, are fully green:
--detachroot-cause is evidenced by a real detach log frozen mid-line atworktree_creating(seedocs/2026-07-07-archon-detach-fix.md); the standby root-cause is evidenced by 7/7 exit-66 runs correlated to Kernel-Power 506/507 events (see the problem-record addendum).bun run validatetest failures are unrelated Windows symlink-privilege (EPERM) failures in untouched packages, not regressions from this PR — they reproduce independent of these commits and require OS Developer Mode to create symlinks. CI (Linux) runs these symlink tests normally.Security Impact (required)
kernel32!SetThreadExecutionStateviabun:ffi(Windows) to inhibit sleep. This is an OS power-state hint only; it grants no new access and is a best-effort no-op on failure or off-Windows. The--detachchange swapsBun.spawnfornode:child_process spawnrunning the same argv — no new capability.Yes, describe risk and mitigation: SetThreadExecutionState only signals "keep the system awake while a run is active"; it cannot read/write data or escalate privilege. Wrapped in try/catch; OS auto-clears the state on process exit.Compatibility / Migration
Human Verification (required)
What was personally validated beyond CI:
keep-awake.test.ts;buildDetachedRunCmdargv builder unchanged and still passing.--detachsurvival smoketest (a real detached run continuing pastworktree_creating— this is OS process-detachment behavior, not unit-testable; procedure is documented in2026-07-07-archon-detach-fix.mdTask 2) and a real Modern-Standby cycle with keep-awake held. These require an unattended Windows host and are best confirmed by a maintainer on target hardware.Side Effects / Blast Radius (required)
--detachspawn path, every platform); everyexecuteWorkflowinvocation (keep-awake acquire/release wrapper — a no-op on non-Windows).worktree_creating; keep-awake acquire/release is refcounted with release as the first statement of thefinally, so no path can leak an unpaired acquire; the OS clears execution state on process exit as a final backstop.Rollback Plan (required)
git revert 2800552b ae233ea7(detach fix + keep-awake). To revert only the detach fix:git checkout <base> -- packages/cli/src/commands/workflow.ts. The edits run from source, so a revert is effective on the nextarchoninvocation with no rebuild.worktree_creating(detach regression); host sleeping mid-run / exit-66 bash-tail collapse (keep-awake not holding).Risks and Mitigations
--detachon all platforms.setup.ts's already-provenspawn(..., { detached: true })pattern;buildDetachedRunCmd(argv) is unchanged and unit-tested; astart /bJob-Object fallback is documented in the detach doc (Task 3) ifdetachedalone proves insufficient on some shells.finally; never blocks or fails a run.Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests