Skip to content

Fork: a 'stalled' task event for autopilot and automations - #5

Merged
jbutton merged 14 commits into
mainfrom
work/18-fork-a-stalled-task-event-for-
Nov 12, 2026
Merged

jbutton merged 14 commits into
mainfrom
work/18-fork-a-stalled-task-event-for-

Conversation

@jbutton

@jbutton jbutton commented Nov 12, 2026

Copy link
Copy Markdown
Owner

The full check now passes on the current head (b725141): 1297 tests, in 645 s. The result is recorded for this exact code, so the board's run can reuse it, and nothing in the folder has changed since.

The board's earlier run didn't fail a test; it hit its 15-minute limit. Seven other tasks' full checks were sharing this 12-core machine, so each got about two cores and a full run took 750–900 s instead of about 112 s. I made two changes so the check finishes reliably on a busy machine:

  • ec3d928, my stall tests: they were starting a real agent (worktree, agent, finish) for every task an automation made, which waits in real time and isn't what they test. Only the autopilot test now runs jobs, and only the coordinator's; the others mark a made task done and drop its job. The stall tests now take about 6 s instead of 22 s.
  • b725141, one flaky existing test: tests/web_board.rs compared two "Ns ago" times read from the real clock. On the loaded machine a second passed between the two reads ("1s ago" against "0s ago"), and it failed two of my full runs. My change doesn't touch that code. The fix is one line that freezes that test's clock once its tasks are made. It's outside this task's scope, but the required check couldn't pass reliably without it.

The stalled event and both review fixes are unchanged. Between my commits the board merged the base branch into this one (4d5cf52), and the passing run includes it. The closing note is updated.

Also noticed:

  • tests/main.rs is at its 300-line cap, so a new top-level test file can't be added.
  • docs/autopilot.md uses "Autopilot stalled" for the coordinator itself, which now shares the word with the task event.
  • There are existing compiler warnings in tests/coordinator.rs:35, tests/coordinator_chat.rs:32, tests/coordinator_chat_more.rs:37 and tests/board_list/pinned.rs:230.
  • The board stops a full check after 15 minutes even when many checks are sharing the machine.

Task: PRJ-18 (herdr-projects/18)

jbutton and others added 14 commits November 12, 2026 20:49
A working task whose agent's screen shows nothing new for 20 minutes
fires no automation (--when stalled is refused) and wakes no
coordinator on autopilot (autopilot.when stalled is refused). These
tests fail until the stalled event exists. Spec: upstream cf20779b
(Restore failed and stalled task event producers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A working task whose agent shows nothing new for stall.after (20m by
default, a setting; none turns it off), or that runs past the Watcher's
estimate by as much again (at least stall.after), now stalls. Each stall
fires automations that wait for it (--when stalled) and wakes a
coordinator on autopilot that looks after the task (autopilot.when
stalled, on by default), once. New output, or a moved guess, re-arms it.

New output is read by a light look at each working agent's screen about
once a minute (watcher/output.rs), with the Watcher's own idea of a
change (spinners, timers and numbers are none), no model call, and the
Watcher off too. The task keeps when its words last changed and when it
was last looked at, so the keeper and an open board judge alike; a stale
look says nothing and a gap in looking (a sleeping machine) starts the
count again. herdr's own readings are refreshed by housekeeping every
two seconds, so they are no sign of life. The Watcher keeps when its
guess last moved (watch.movedAt): the estimate a task runs past.

Rewrite of upstream cf20779b (Restore failed and stalled task event
producers): quiet for 20 minutes or past its estimate with the same
allowance (left + max(left, 20m)), once per stall, re-armed by new
activity or a fresh estimate. MIT, as the fork's LICENSE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on the stalled event:

- A stalled task now needs looking after (autopilot/care.rs needing), so
  it wakes a coordinator on autopilot whose autopilot.tasksAtOnce are
  full, as a failed one does. autopilot show lists it (has stalled), and
  its row no longer says at its limit.
- A stall seen is remembered until something re-arms it (new output, a
  new run, a moved guess), also while the task does not stall for a
  moment: a brief blocked status no longer drops the mark and fires the
  same stall again. Stalls are told apart by kind and since, so a change
  of stall.after is no new stall either. This is upstream cf20779b's
  quietLife/overReport kept in the seen state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review's direct new_events probe (a blocked status between two looks
fires the same stall again) is now a light test: the stall seen is kept
while it does not stall, and only new output makes the next one new.
The stall.after 45m check asserted at the very minute the stall fell
due, so one tick that moved the clock failed it once in a full run on a
loaded machine; it now looks two minutes later, as the other stall
tests do, and says what the task kept when it fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The automation stall tests ran every job the keeper's tick queued,
also the agent put on each task an automation made: a worktree, an
agent and its finish, in real time, for what is not tested here.
They now run only the coordinator's jobs in the autopilot test, and
mark a made task done and drop its job: the stall tests take about 6s
instead of 22s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test read the list's "Ns ago" and the task panel's in two calls on
the real clock; on a busy machine a second passed between them ("1s
ago" against "0s ago") and it failed the full check twice. Its clock
now stands still once the tasks are made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/web_board.rs: main already freezes the clock at the same place;
its version is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's archive.doneAfter and this branch's stall.after each add a group
to the settings page (a row and a blank line), and each raised the two
tests' screens by two; the merge kept one raise of the two. Two more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In a full run something else in the test program held the keeper's
tick lock for twenty of the stall test's ticks in a row: tick_once
skipped them all, as it skips whenever another ticker is at it, so no
look and no automation ran and no stall fired (1 failure in 4 runs of
that process's tests). The test's tick is now tried again until it ran,
as the keeper's next second would be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/automations_open.rs, tests/board_parts.rs: main's review.prComment
row (+1) and this branch's stall.after group (+2) both grow the settings
page; the screens are 68 and 69 lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbutton
jbutton marked this pull request as ready for review November 12, 2026 23:47
@jbutton
jbutton merged commit 345c9ee into main Nov 12, 2026
0 of 4 checks passed
@jbutton
jbutton deleted the work/18-fork-a-stalled-task-event-for- branch November 12, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant