Repository navigation
Fork: a 'stalled' task event for autopilot and automations - #5
Merged
Merged
Conversation
A working task whose agent's screen shows nothing new for 20 minutes fires no automation (--when stalled is refused) and wakes no coordinator on autopilot (autopilot.when stalled is refused). These tests fail until the stalled event exists. Spec: upstream cf20779b (Restore failed and stalled task event producers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A working task whose agent shows nothing new for stall.after (20m by default, a setting; none turns it off), or that runs past the Watcher's estimate by as much again (at least stall.after), now stalls. Each stall fires automations that wait for it (--when stalled) and wakes a coordinator on autopilot that looks after the task (autopilot.when stalled, on by default), once. New output, or a moved guess, re-arms it. New output is read by a light look at each working agent's screen about once a minute (watcher/output.rs), with the Watcher's own idea of a change (spinners, timers and numbers are none), no model call, and the Watcher off too. The task keeps when its words last changed and when it was last looked at, so the keeper and an open board judge alike; a stale look says nothing and a gap in looking (a sleeping machine) starts the count again. herdr's own readings are refreshed by housekeeping every two seconds, so they are no sign of life. The Watcher keeps when its guess last moved (watch.movedAt): the estimate a task runs past. Rewrite of upstream cf20779b (Restore failed and stalled task event producers): quiet for 20 minutes or past its estimate with the same allowance (left + max(left, 20m)), once per stall, re-armed by new activity or a fresh estimate. MIT, as the fork's LICENSE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on the stalled event: - A stalled task now needs looking after (autopilot/care.rs needing), so it wakes a coordinator on autopilot whose autopilot.tasksAtOnce are full, as a failed one does. autopilot show lists it (has stalled), and its row no longer says at its limit. - A stall seen is remembered until something re-arms it (new output, a new run, a moved guess), also while the task does not stall for a moment: a brief blocked status no longer drops the mark and fires the same stall again. Stalls are told apart by kind and since, so a change of stall.after is no new stall either. This is upstream cf20779b's quietLife/overReport kept in the seen state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review's direct new_events probe (a blocked status between two looks fires the same stall again) is now a light test: the stall seen is kept while it does not stall, and only new output makes the next one new. The stall.after 45m check asserted at the very minute the stall fell due, so one tick that moved the clock failed it once in a full run on a loaded machine; it now looks two minutes later, as the other stall tests do, and says what the task kept when it fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fork-a-stalled-task-event-for-
The automation stall tests ran every job the keeper's tick queued, also the agent put on each task an automation made: a worktree, an agent and its finish, in real time, for what is not tested here. They now run only the coordinator's jobs in the autopilot test, and mark a made task done and drop its job: the stall tests take about 6s instead of 22s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test read the list's "Ns ago" and the task panel's in two calls on
the real clock; on a busy machine a second passed between them ("1s
ago" against "0s ago") and it failed the full check twice. Its clock
now stands still once the tasks are made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fork-a-stalled-task-event-for-
tests/web_board.rs: main already freezes the clock at the same place; its version is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fork-a-stalled-task-event-for-
main's archive.doneAfter and this branch's stall.after each add a group to the settings page (a row and a blank line), and each raised the two tests' screens by two; the merge kept one raise of the two. Two more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fork-a-stalled-task-event-for-
In a full run something else in the test program held the keeper's tick lock for twenty of the stall test's ticks in a row: tick_once skipped them all, as it skips whenever another ticker is at it, so no look and no automation ran and no stall fired (1 failure in 4 runs of that process's tests). The test's tick is now tried again until it ran, as the keeper's next second would be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/automations_open.rs, tests/board_parts.rs: main's review.prComment row (+1) and this branch's stall.after group (+2) both grow the settings page; the screens are 68 and 69 lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The full check now passes on the current head (b725141): 1297 tests, in 645 s. The result is recorded for this exact code, so the board's run can reuse it, and nothing in the folder has changed since.
The board's earlier run didn't fail a test; it hit its 15-minute limit. Seven other tasks' full checks were sharing this 12-core machine, so each got about two cores and a full run took 750–900 s instead of about 112 s. I made two changes so the check finishes reliably on a busy machine:
tests/web_board.rscompared two "Ns ago" times read from the real clock. On the loaded machine a second passed between the two reads ("1s ago" against "0s ago"), and it failed two of my full runs. My change doesn't touch that code. The fix is one line that freezes that test's clock once its tasks are made. It's outside this task's scope, but the required check couldn't pass reliably without it.The stalled event and both review fixes are unchanged. Between my commits the board merged the base branch into this one (4d5cf52), and the passing run includes it. The closing note is updated.
Also noticed:
tests/main.rsis at its 300-line cap, so a new top-level test file can't be added.docs/autopilot.mduses "Autopilot stalled" for the coordinator itself, which now shares the word with the task event.tests/coordinator.rs:35,tests/coordinator_chat.rs:32,tests/coordinator_chat_more.rs:37andtests/board_list/pinned.rs:230.Task: PRJ-18 (herdr-projects/18)