feat(bin): corroborate Herdr agent liveness, sandbox crewmate launches, and harvest fleet usage - #3518
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Launch Claude with --permission-mode auto and Codex with -s workspace-write -a never instead of the full-bypass flags, so a crewmate runs under its harness's own approval and sandbox controls rather than with them switched off.
Launch Cursor with --auto-review --sandbox enabled instead of --yolo, so it runs under its own sandbox and review controls rather than with them switched off.
The mkdir that ensures data/ exists ran after the tmp-file write that already needs data/ to exist, so under set -e it could never protect a first harvest on a home whose data/ dir was absent. Move it ahead of the write.
…e spawn launch comments
…rocesses A task whose Pi worker had exited cleanly to its shell kept reading `alive`, so two supported `fm-control.sh <task> relaunch` attempts typed the harness's exit command into a zsh prompt, waited for a stop that could never come, and refused to launch a replacement. Nothing in the task's records could ever clear that state. Herdr's agent registry is written by whatever reports into it, and a report is not withdrawn when the process that made it goes away, so a registration can outlive the agent it describes. The classifier trusted that registration alone. The tmux classifier never had this hole: it settles negative verdicts from the foreground process group, and a group that is nothing but shells is agent-free. Express the same rule through Herdr's own `pane process-info` inventory. When a registration is reported, the existing idle-shell proof runs one strict instantaneous sample; a lone bare idle shell means the registration is stale and the pane classifies agent-free, which makes the endpoint positively eligible for its replacement. The corroboration is positive-only, so a live harness process, an extra foreground process, a shell with a child, an unreadable inventory, and an inventory answering about a different pane all leave the verdict alive. Only the herdr adapter changes. tmux already reads the process group; zellij, orca, and cmux have no recovery-grade classifier and already refuse the stop verbs. The proof reads a vendor-controlled process name, so a new opt-in guard launches every installed harness for real and fails naming the harness and version if a running one ever proves a bare idle shell. Three real-Herdr cases registered an agent on a shell-only pane to mean "genuinely live". That setup is now the husk shape, so they run a real foreground process first through one shared helper, which also removes an existing flake where Herdr dropped the bare registration mid-test.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (3): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| wall_secs:$wall, turns:$turns, | ||
| input_tokens:$it, cached_input_tokens:$ct, output_tokens:$ot, | ||
| reasoning_tokens:$rt, source:$source}' >> "$DATA/usage-ledger.jsonl.tmp.$$" | ||
| cat "$DATA/usage-ledger.jsonl.tmp.$$" >> "$LEDGER" && rm -f -- "$DATA/usage-ledger.jsonl.tmp.$$" |
There was a problem hiding this comment.
When the ledger append fails, set -e exits before the conditional rm runs, while harvest_cleanup does not remove the staging path. Each failed best-effort teardown harvest therefore leaves another usage-ledger.jsonl.tmp.<pid> file in the fleet data directory.
…ch write contract
0c8b0c6 to
cc2b6a6
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c8b0c6139
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cat > "$logdir/session-future.jsonl" <<'JSON' | ||
| {"type":"assistant","message":{"id":"msgX","model":"claude-test","usage":{"input_tokens":999,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":999}}} | ||
| JSON | ||
| touch -t "$(date -r $(( $(file_mtime_epoch "$state/$id.status") + 7200 )) +%Y%m%d%H%M.%S)" \ |
There was a problem hiding this comment.
Captain, in the required Ubuntu tests-portable-serial-3of4 lane, GNU date interprets -r <epoch> as a reference-file argument, so this command substitution is empty and touch fails. Because the test continues, session-future.jsonl retains its current mtime, its 999 tokens are harvested, and the first assertion expects 17 but receives 1016; running bash tests/fm-usage-harvest.test.sh reproduced this failure. Use a BSD/GNU-compatible timestamp helper for this and the other new date -r fixture calls.
Useful? React with 👍 / 👎.
| m=$(file_mtime_epoch "$f") || continue | ||
| if [ "$m" -ge "$START_EPOCH" ] && [ "$m" -le "$END_EPOCH" ]; then | ||
| printf '%s\n' "$f" |
There was a problem hiding this comment.
When a worker appends done: during its final tool call, the harness normally writes the corresponding assistant usage event afterward, making the session file's final mtime later than the status-file mtime used as END_EPOCH. This filter then rejects the entire matching session and records source:"unavailable" with null tokens; the test hides the production ordering by explicitly touching the status file to the log's mtime. Base discovery on teardown time or filter timestamped log records rather than excluding a session by its final file mtime.
Useful? React with 👍 / 👎.
| } | ||
| trap harvest_cleanup EXIT | ||
| epoch_to_touch() { # <epoch> | ||
| date -r "$1" +%Y%m%d%H%M.%S 2>/dev/null || date -u -d "@$1" +%Y%m%d%H%M.%S |
There was a problem hiding this comment.
On GNU/Linux hosts with a non-UTC timezone, the fallback formats the epoch in UTC but touch -t parses that value as local time. Both reference files are consequently shifted by the host's UTC offset before find -newer runs; for example, under TZ=America/Toronto the resulting reference epoch is four hours late, causing valid Claude or Codex logs to be excluded and usage to be recorded as unavailable. Drop -u from this fallback so it matches the BSD branch and touch's local-time interpretation.
Useful? React with 👍 / 👎.
| reasoning_tokens:$rt, source:$source}' >> "$DATA/usage-ledger.jsonl.tmp.$$" | ||
| cat "$DATA/usage-ledger.jsonl.tmp.$$" >> "$LEDGER" && rm -f -- "$DATA/usage-ledger.jsonl.tmp.$$" |
There was a problem hiding this comment.
If the ledger append fails, such as when the path is temporarily unwritable or is a directory, the && rm branch never executes and the EXIT cleanup only removes REFDIR and the lock. Teardown deliberately treats harvest failures as non-fatal, so every such failure leaves another usage-ledger.jsonl.tmp.<pid> file in the persistent data directory. Register the staged path with harvest_cleanup so it is removed on both success and failure.
Useful? React with 👍 / 👎.
| else . end) | ||
| | [.m, .it, .ct, .ot, .rt] | @tsv' "$f" 2>/dev/null || true) | ||
| [ -n "$row" ] || continue | ||
| IFS=$'\t' read -r m it ct ot rt <<<"$row" |
There was a problem hiding this comment.
When a valid Claude message has usage but no .message.model, or a Codex session lacks a turn_context model, @tsv emits an empty model column, but Bash treats tab as whitespace and collapses that empty field during read. Every subsequent value shifts left: for example, Claude usage [null,10,20,30,4] becomes model 10, input 20, cached 30, output 4, and reasoning 0, rather than using the metadata model fallback. Use a non-whitespace delimiter or consume the result as JSON so null columns retain their positions; the identical parsing pattern at line 242 needs the same correction.
Useful? React with 👍 / 👎.
| row=$(jq -rn ' | ||
| reduce inputs as $l ({seen:{},m:null,it:0,ct:0,ot:0,rt:0}; |
There was a problem hiding this comment.
If a session log contains one malformed or truncated line, which can happen when a worker is terminated while writing its log, jq aborts the entire reduce inputs operation rather than skipping that record. The suppressed failure yields no row, so all valid usage in that file is lost; Claude can then record an available source with zero tokens, while Codex records the source as unavailable. Parse each raw line with a tolerant fromjson? step so the documented best-effort behavior preserves valid records around the damaged line.
Useful? React with 👍 / 👎.
| case "$HARNESS" in | ||
| claude) |
There was a problem hiding this comment.
A supported fm-spawn.sh <id> --relaunch --harness ... can replace a task with a different harness or model, and spawn overwrites the task's metadata with that latest incarnation while retaining the same status history and worktree. At teardown this branch scans only the final HARNESS, so a Claude-to-Codex relaunch silently omits all Claude usage; a same-harness model change instead combines both models' tokens under whichever model is encountered first. Preserve per-incarnation harness/model history or harvest the outgoing incarnation before overwriting it so the task ledger does not undercount or misattribute relaunched work.
Useful? React with 👍 / 👎.
| claude) printf '%s' "CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false $2 --permission-mode auto" ;; | ||
| codex) printf '%s' "$2 -s workspace-write -a never --add-dir $4 --add-dir $5" ;; | ||
| cursor) printf '%s' "env -u CLAUDECODE -u PI_CODING_AGENT -u GROK_AGENT -u FM_PI_HARNESS -u CURSOR_INVOKED_AS $2 --trust --auto-review --sandbox enabled --add-dir $4 --add-dir $5 --workspace $3" ;; |
There was a problem hiding this comment.
This guard manually reimplements the three launch commands instead of invoking fm-spawn.sh or a shared launch composer. Because edits to bin/fm-spawn.sh merely select this test without synchronizing these copied strings, a production regression such as dropping __ADDDIRS__, misquoting a granted path, or emitting the wrong sandbox flag can leave the guard launching its independent correct command and passing. Drive the lab launch through the public spawn path, or extract one shared command builder, so the claimed live verification actually covers the command shipped to workers.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0abe9210ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (map(.wall_secs // 0) | add | tostring) ] | @tsv' "$LEDGER" | | ||
| while IFS=$'\t' read -r model tasks it ct ot rt wall; do |
There was a problem hiding this comment.
Captain, if the ledger contains a malformed or truncated row, jq exits nonzero, but this pipeline's final while command returns success because the script does not enable pipefail; the script then prints empty aggregate totals, may print only the valid prefix of per-task rows, and exits 0 while still counting the corrupt line in the header. Validate the ledger once or explicitly propagate both jq statuses so callers cannot mistake an incomplete report for a successful one.
Useful? React with 👍 / 👎.
| printf '%s' 'codex __MODELFLAG____EFFORTFLAG__-s workspace-write -a never __ADDDIRS__"$(__OPINPUT__ encode launch-brief < __BRIEF__)"' | ||
| else | ||
| printf '%s' 'codex __MODELFLAG____EFFORTFLAG__--dangerously-bypass-approvals-and-sandbox -c "notify=[\"bash\",\"-c\",\"touch __TURNEND__\"]" "$(__OPINPUT__ encode launch-brief < __BRIEF__)"' | ||
| printf '%s' 'codex __MODELFLAG____EFFORTFLAG__-s workspace-write -a never __ADDDIRS__-c "notify=[\"bash\",\"-c\",\"touch __TURNEND__\"]" "$(__OPINPUT__ encode launch-brief < __BRIEF__)"' |
There was a problem hiding this comment.
Captain, on hosts without an ambient [sandbox_workspace_write] network_access = true setting, these new workspace-write launches deny outbound network while -a never removes any escalation path. Codex's installed sandbox guidance explicitly states that approval policy never does not enable network and that workspace-write networking depends on that configuration; consequently direct-PR workers cannot perform the required push or gh-axi PR creation in bin/fm-dod-lib.sh, and no-mistakes workers cannot ship either. Enable sandboxed network access explicitly in the launch rather than depending on operator-global configuration.
Useful? React with 👍 / 👎.
| assert_contains "$launch" "--trust --auto-review --sandbox enabled --model 'cursor-grok-4.5-high' --workspace '$WT_DIR'" \ | ||
| "cursor launch did not carry trust, autonomy, model, and exact workspace flags" |
There was a problem hiding this comment.
Captain, the new Cursor template always inserts --add-dir '<state>' between --sandbox enabled and --model, but this changed assertion still requires those tokens to be contiguous. Running bash tests/fm-spawn-dispatch-profile.test.sh therefore deterministically fails at this assertion even though the emitted production command contains the intended flags; update the expectation to include or tolerate the state-directory grant while still pinning its exact path.
Useful? React with 👍 / 👎.
| FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" | ||
| STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" | ||
| DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" | ||
| CLAUDE_DIR="${FM_USAGE_CLAUDE_DIR:-${HOME:-}/.claude/projects}" |
There was a problem hiding this comment.
Captain, when CLAUDE_CONFIG_DIR is set for a work/personal subscription split, fm-spawn.sh explicitly launches the Claude worker with that store, so its project session logs live under $CLAUDE_CONFIG_DIR/projects; this hard-coded default still scans $HOME/.claude/projects. Teardown therefore writes an idempotent source:"unavailable" row with null tokens even though the matching log exists, and a later retry using the correct path is skipped because the task is already present in the ledger. Resolve the log root from the same configured store used at launch, ideally persisting it in task metadata for teardown.
Useful? React with 👍 / 👎.
|
Speaking as Kun's firstmate: full-thread + full-diff review of #3518 (npayette84). Author is not on the blocked list. Attestation: MISMATCH. Body carries First-time fork CI this pass: after safe diff review (zero Contract-class: new-default. Three always-on unconfigured surfaces change together:
Not restore (main promised registry-alone live + bypass launch flags + no fleet ledger). Not opt-in. No auto-merge. Firstmate escalate only when otherwise ready (CLEAN + MATCH + green CI/NM + safe) — not this pass. VISION.md per-rule (inspected
Codex P1s waiting on author (blocking readiness, not security-flag):
Also note non-blocking harvest P2s (tmp leak on append fail, GNU Security: clean for this pass — no workflow/RCE/credential surface; spawn change narrows autonomy; classifier is positive-only fail-closed toward Waiting on author for attestation rebind + Codex P1s + green CI/NM. Not escalating; not merging. |
2 participants
Intent
Fix the safe-relaunch defect reproduced on the task fm-pr-lifecycle-followthrough-monitor: its Pi worker exited cleanly to a zsh shell, and Herdr's process inventory showed no live Pi agent, but two supported
bin/fm-control.sh <task> relaunchattempts classified the agent as alive, typed/quitinto zsh, and failed without launching a replacement. The preserved PR branch and existing work had to remain untouched.Required approach, all of it deliberate:
Root cause established empirically, not assumed. Herdr's agent registry is written by whatever reports into it, and a report is not withdrawn when the process that made it goes away, so a registration can outlive the agent it describes. bin/backends/herdr.sh's fm_backend_herdr_pane_agent_state derived
livefromherdr agent getalone, with no corroboration, so the endpoint readaliveforever and every lifecycle verb typed the harness exit command at a shell. Verified on the installed herdr 0.8.2 in an isolated fm-lab- session: apane report-agentfrom a source herdr does not itself own stays registered on a pane running nothing but zsh (agent get keeps answering idle) whilepane process-inforeports one bare zsh; herdr does validate a report claiming a source it owns, so an otherwise identical --source herdr:pi report on the same pane was rejected. The tmux classifier never had this hole because it settles negative verdicts from the foreground process group.Chosen design and the tradeoffs behind it:
aliveread in poll loops for no benefit, while a single sample keeps the hot path to one extra herdr call plus ps. The corroboration is positive-only, so a transient prompt helper simply keepslivefor that sample and the next poll settles it; every caller that acts on the negative verdict already polls.livetowardno-agent, never the other way. A live harness process, an extra foreground process, a shell with a child, an unreadable inventory, and an inventory answering about a different pane all leavelivestanding, which is what satisfies the refuse-genuinely-live/ambiguous/unreadable/conflicting requirement.dead, so do_exit becomes idempotent (already-stopped) and the replacement launch proceeds, with no new lifecycle path.Per-backend and per-harness review, with evidence rather than assumption: only the herdr adapter changes. tmux already reads the foreground process group; zellij, orca, and cmux have no recovery-grade classifier, report
unverified, and already refuse exit and relaunch. Because the proof reads a process name and argv0 the harness vendor controls, a new opt-in real-harness drift guard launches every INSTALLED harness for real and fails naming the harness and version if a running one ever proves a bare idle shell. Measured on herdr 0.8.2: claude 2.1.252, codex 0.150.1, opencode 1.17.11, pi 0.84.4, and cursor 2026.08.31 all own the pane foreground under their own argv0 and stay alive; pi-signed, grok, kimi, and muse are not installed and are reported explicitly, and the guard refuses a pass that checked nothing.Test changes that are deliberate consequences, not incidental churn: three existing real-Herdr cases (fm-backend-herdr-smoke, fm-backend-herdr-respawn-idem-e2e, fm-backend-herdr-presentation-e2e) registered an agent on a shell-only pane to mean 'a genuinely live duplicate'. Under the corrected contract that setup is the husk shape, so each now gives the pane a real foreground process first through one shared helper added to tests/herdr-test-safety.sh. That also removes a pre-existing flake in which herdr dropped the bare registration mid-test. tests/fm-control-herdr-smoke.test.sh previously encoded the buggy behavior as expected and now runs the identical registered reading twice, once over a plain shell and once over a real foreground process.
Verification actually run: bin/fm-lint.sh clean, bin/fm-doc-audience-check.sh clean, bin/fm-test-run.sh --check-coverage clean, and fm-backend-herdr, fm-control, fm-control-relaunch, fm-control-herdr-smoke, fm-test-run, fm-documentation-audiences, fm-backend, and fm-crew-state all green. Ten suites fail on this machine (seven real-Herdr end-to-end suites plus composer-lib, muse-harness, and spawn-dispatch-profile); every one of those failures was reproduced on the unmodified baseline across three runs with the pre-change file swapped back in, so they are pre-existing herdr 0.8.2 and local-toolchain drift, not regressions from this change.
Repo conventions that apply: one sentence per line in tracked Markdown, plain dash never an em dash, no agent co-author on commits, bin/*.sh shellcheck-clean, tests colocated in tests/ as .test.sh extending existing runners, tests must exercise public or executable interfaces and never assert implementation-source bytes, and maintainer-verification records under docs/verification/ carry dated versions, exact commands, and exact output for current guarantees only.
What Changed
fm_backend_herdr_pane_agent_stateinbin/backends/herdr.shno longer trustsherdr agent getalone: when a registration is reported, it takes one strict instantaneousfm_backend_herdr_pane_idle_shell_sample, and a pane positively proven to hold a lone bare idle shell now classifies asno-agentinstead oflive. The downgrade is positive-only, so a live harness process, an extra foreground process, a shell with a child, an unreadable inventory, or an inventory answering about a different pane all keeplive.fm-control.shandfm-spawn.share unchanged and pick this up through the classifier they already gate on, which makes exit idempotent over an exited shell and lets the same-task replacement launch. New coverage lands intests/fm-backend-herdr.test.sh,tests/fm-control-relaunch.test.sh,tests/fm-control-herdr-smoke.test.sh, and the opt-intests/fm-herdr-agent-free-proof-live-e2e.test.shdrift guard; three existing real-Herdr suites now give their pane a real foreground process through a shared helper added totests/herdr-test-safety.sh.bin/fm-spawn.shreplaces the blanket-bypass launch flags with sandboxed autonomy per harness: claude runs--permission-mode auto, codex runs-s workspace-write -a never, and cursor runs--auto-review --sandbox enabledinstead of--yolo. A new__ADDDIRS__placeholder grants--add-dirfor exactly the two paths the brief permits outside the worktree,state/for every kind plusdata/<task-id>/for a scout, so a sandboxed crewmate can still append the status file it reports through.tests/fm-crewmate-autonomy-live-e2e.test.shis the opt-in real-harness guard, andbin/fm-test-run.shre-selects the live family on a spawn-template edit.bin/fm-usage-harvest.shappends one JSON row per finished task to the gitignoreddata/usage-ledger.jsonl, summing per-request token usage from the worker's own claude or codex session logs and falling back to a null-tokenunavailablerow for cursor, remote, or unmatched tasks; it is idempotent per task id. Newbin/fm-usage-report.shreads that ledger into per-model and per-task plain text, andbin/fm-teardown.shcalls the harvester best-effort on both teardown paths while the state files still exist, warning rather than failing. Docs and the maintainer verification records underdocs/, plus.agents/skills/harness-adapters/SKILL.md, are updated for the new classifier contract, the measured autonomy and write posture per harness, and the two new scripts.Risk Assessment
✅ Low: The reviewed lane is well-bounded and positive-only (the classifier can only move live toward no-agent, every ambiguous/unreadable/foreign-pane input still reads live), the fix round's three changes each verify correct against the code they target, and the sole substantiated defect is a graceful-degradation portability issue in adjacent harvester code that yields null token fields rather than wrong data.
Testing
I read the classifier, the control plane, the Herdr adapter, and the existing behavioral tests, then reproduced the reported defect and its fix end-to-end through the real operator interface rather than only through unit tests. In a private throwawayfm-lab-*Herdr session against the installed herdr 0.8.2, I stood up a task whose pane holds nothing but zsh with a stale agent registration and a real worktree carrying both a commit and uncommitted work, and ranbin/fm-control.sh <task> relaunchtwice: against a scratch copy with the pre-fixbin/backends/herdr.shit classifiedalive, typed/quitat the shell, and failed with no replacement; on this branch the identical fixture classifiesdead, recordsalready-stopped, types no/quit, launches the replacement into the pane foreground, and leaves the branch, HEAD, uncommitted file, and progress note untouched. On top of that I ran the targeted automated set: the portable classifier and relaunch suites are green including all new cases, the real-Herdr control-plane smoke suite is green 9/9, and the opt-in per-harness drift guard launched all five installed harnesses for real and confirmed none can be mistaken for a bare idle shell. The one failing suite, tests/fm-backend-herdr-smoke.test.sh, fails identically on the unmodified baseline, so it is pre-existing local herdr drift and not a regression; the working tree is clean and no lab sessions were left behind.Evidence: Evidence index: before/after relaunch, drift guard, smoke, behavioral cases
Source: Evidence index: before/after relaunch, drift guard, smoke, behavioral cases
Evidence: CLI transcript - pre-fix classifier reproduces the defect
Source: CLI transcript - pre-fix classifier reproduces the defect
=== the reported disagreement (herdr 0.8.2) === $ herdr agent get w1:p2 -> agent_status: idle $ herdr pane process-info -> foreground: [{"name":"zsh","argv0":"zsh"}] firstmate's own classifier -> fm_backend_agent_state herdr = alive === operator runs the supported control verb === $ bin/fm-control.sh demo relaunch --note 'prior Pi worker exited to a shell after the PR went green' error: exit-delivered demo interrupt=not-needed exit-command=delivered agent-state=alive exit=unconfirmed; the agent did not stop within 3s error: relaunch of demo failed while stopping the old agent, which is still running; its original instructions were restored exit status: 1 === what reached the pane === pane received /quit: YES <- the harness exit command was typed at a zsh prompt pane foreground now: [{"name":"zsh","argv0":"zsh"}]Evidence: CLI transcript - this branch relaunches successfully and preserves the work
Source: CLI transcript - this branch relaunches successfully and preserves the work
=== the reported disagreement (herdr 0.8.2) === $ herdr agent get w1:p2 -> agent_status: idle $ herdr pane process-info -> foreground: [{"name":"zsh","argv0":"zsh"}] firstmate's own classifier -> fm_backend_agent_state herdr = dead === operator runs the supported control verb === $ bin/fm-control.sh demo relaunch --note 'prior Pi worker exited to a shell after the PR went green' relaunched demo harness=pi from=pi model=default effort=default backend=herdr endpoint=fm-lab-relaunch-demo-after:w1:p2 worktree=.../wt exit status: 0 === what reached the pane === pane received /quit: no pane foreground now: [{"name":"sleep","argv0":"sleep"}] endpoint still exists: yes === preserved work === branch: task-demo HEAD unchanged: yes uncommitted file: work the previous worker had not committed committed file: committed on the preserved branch journal exit_result: exit_result=already-stopped progress note in the replacement's instructions: yesEvidence: Reproduction script used for the before/after transcripts
Source: Reproduction script used for the before/after transcripts
Evidence: Installed-harness agent-free drift guard (real harnesses in Herdr panes)
Source: Installed-harness agent-free drift guard (real harnesses in Herdr panes)
# herdr: herdr 0.8.2 # claude 2.1.257 (Claude Code): foreground=[2.1.257/claude] state=alive # codex codex-cli 0.150.1: foreground=[codex/codex] state=alive # opencode 1.17.11: foreground=[opencode/opencode] state=alive # pi 0.84.4: foreground=[node/pi] state=alive # cursor 2026.08.31-4057e58: foreground=[node/cursor-agent] state=alive # unverified on this machine (not installed): pi-signed grok kimi muse # checked 5 installed harness(es) on herdr 0.8.2 in workspace w1Evidence: Real-Herdr control plane smoke suite
Source: Real-Herdr control plane smoke suite
ok - real herdr: exit on a pane with no registered agent is idempotent success ok - real herdr: interrupt refuses when herdr's own agent registry reports no agent ok - real herdr: a reported registration the pane's own process inventory contradicts reads agent-free ok - real herdr: an exited worker's pane is positively eligible for its replacement instead of being typed into ok - real herdr: interrupt still refuses on a pane whose registration outlived its agent ok - real herdr: the same registration over a running foreground process stays alive ok - real herdr: interrupt delivers the harness's key and proves the agent survived it ok - real herdr: an agent that does not stop fails closed instead of being reported as stopped ok - real herdr: no control verb removed the endpoint, the task's local copy, or its branchEvidence: New deterministic classifier and relaunch cases
Source: New deterministic classifier and relaunch cases
ok - herdr agent state: a reported registration contradicted by a lone bare idle shell reads agent-free, while the identical registration over a live process stays alive ok - herdr agent state: an ambiguous process inventory leaves a registered agent alive rather than licensing recovery ok - herdr agent state: a shell running a child of its own never proves an agent-free pane ok - herdr agent state: an unreadable process inventory leaves a registered agent alive ok - herdr agent state: a process inventory answering about a different pane never downgrades this pane's registration ok - herdr agent state: an unregistered pane is agent-free without reading the process inventory at all ok - fm_backend_herdr_create_task: a same-labeled tab with a live (even idle) registered agent still refuses exactly as before ok - fm_backend_herdr_create_task: a same-labeled tab whose registration outlived its agent is a husk on the same terms as a restored shell ok - fm-control relaunch: an endpoint whose agent already exited is replaced without an interrupt, without the exit command, and without touching the branch or its uncommitted work ok - fm-control relaunch: a replacement that never comes up still preserves the branch, the uncommitted work, and the progress notePipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
.agents/skills/afk/SKILL.md- branch carries 10 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (272 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
bin/fm-spawn.sh:1117- The codex launch template swapped--dangerously-bypass-approvals-and-sandboxfor-s workspace-write -a never, which confines model-generated writes to the task worktree (cwd/TMPDIR). But bin/fm-brief.sh:318 tells every worker "the only files you may write outside it are the report and the status file below", and bin/fm-brief.sh:179 puts that status file at $FM_HOME/state/<id>.status - a different repo entirely from the worktree. Failure: a codex ship crewmate runsecho "done: PR <url>" >> $FM_HOME/state/<id>.status(bin/fm-brief.sh:320-321), the sandbox denies the write, and-a nevermeans it cannot ask for approval, so the completion signal the watcher, classifier, and teardown gate all read never lands and the task hangs as unreported. Nothing in the branch addswritable_rootsor any codex sandbox config (no match for writable_roots/sandbox_workspace_write/.codex/config anywhere in bin/, docs/, .agents/). Ship tasks additionally push and open PRs, which workspace-write's default network posture also restricts. bin/fm-spawn.sh:1152 makes the same class of change for cursor (--yolo->--auto-review --sandbox enabled), and :1114 for claude (--dangerously-skip-permissions->--permission-mode auto). Either scope the sandbox to include the home state dir, or route the status write through a path the sandbox permits, before this posture ships.tests/fm-spawn-dispatch-profile.test.sh:168- The launch templates in bin/fm-spawn.sh changed but three suites still assert the removed flags, so they fail deterministically. tests/fm-spawn-dispatch-profile.test.sh:168 compares the captured launch for exact equality against a string containingclaude --dangerously-skip-permissions, while the template now emitsclaude --permission-mode auto; :432 assertsclaude --dangerously-skip-permissions --model 'sonnet' --effort 'high'; :382, :449 and :465 assertcodex ... --dangerously-bypass-approvals-and-sandbox, now-s workspace-write -a never. tests/fm-secondmate-harness.test.sh:770, :792 and :857 assert the same removed strings, and tests/fm-backend-orca.test.sh:523 assertsCLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false claude --dangerously-skip-permissionsin the Orca send log. Note this contradicts the change description's claim that spawn-dispatch-profile's failure is pre-existing drift reproduced on an unmodified baseline: the baseline used only restored bin/backends/herdr.sh, so it still carried this branch's bin/fm-spawn.sh. Update the expected launch strings to the new flags (and add fm-secondmate-harness and fm-backend-orca to the suites run before shipping)..agents/skills/harness-adapters/SKILL.md:373- The cursor adapter contract rows still document the pre-change launch: line 373 says the launch carries--trust,--yolo,--modeland--workspace; line 380's Autonomy row says--yolo, the documented alias for--force, whose TUI footer readsRun Everything; line 381 says--yolodoes NOT suppress the trust dialog. bin/fm-spawn.sh:1152 now launches--trust --auto-review --sandbox enabledwith no--yoloat all. This skill is the knowledge half of each adapter that bin/fm-spawn.sh:1099-1100 points at as the owner of autonomy semantics, so the two now contradict each other and the next reader will reason from the wrong autonomy posture.tests/fm-backend-herdr.test.sh:881- test_registration_over_a_shell_with_a_child_stays_alive spins in an unboundedwhile [ -z "$(ps -axo ppid= -o pid= ...)" ]loop waiting for the backgrounded bash to fork itssleepchild. bin/fm-test-run.sh applies no per-test timeout, so if thatpsinvocation ever returns nothing usable (a ps variant that rejects the two -o forms, a platform where the child is reaped before the first sample) the suite hangs until the 15- or 75-minute GitHub job timeout instead of failing. Every other loop added in this change is bounded (attempt < 200 in tests/herdr-test-safety.sh, seq 1 300 in the live guard); bound this one the same way and fail with a clear message.tests/fm-control-herdr-smoke.test.sh:160- The guard provingexitdid not type/quitinto a plain shell reads the pane through pane_text() (line 112), which isherdr pane read "$PANE_ID" --session "$SESSION" 2>/dev/null || true. Unlike fm_backend_herdr_capture (bin/backends/herdr.sh:2636) it passes neither --source nor --lines, and any failure is swallowed into an empty string, which makes thecase ... *"/quit"*check at line 160 silently pass. If a future herdr changespane read's required arguments, the one assertion that proves the reported defect is fixed stops testing anything without going red. Fail explicitly when the read does not succeed.bin/fm-bootstrap.sh:705- Noting the blast radius, not a defect. Because fm_backend_herdr_agent_state maps the new no-agent verdict todead, the session-start secondmate sweep now kills the endpoint and respawns for a herdr pane whose registration outlived its agent, where it previously readaliveand left it alone. That is the intended recovery semantics and matches what a restored husk already got, and the sample's shell-with-a-child rule protects an operator running anything in that pane - but an operator sitting at a bare prompt in a secondmate's pane will now have it closed and replaced at session start. Same widening applies to fm_backend_herdr_create_task's duplicate-tab husk replacement and to fm-herdr-session-cleanup.sh, both of which are already scoped to tasks with no live meta record.🔧 Fix: fix stale launch-flag assertions and vacuous test guards
2 issues (1 warning, 1 info) still open:
bin/fm-usage-harvest.sh:149- epoch_to_touch's GNU fallback renders the timestamp in UTC (date -u -d "@$1" +%Y%m%d%H%M.%S) but the value is consumed bytouch -t(lines 154-155), which parses its argument as LOCAL time. On macOS the BSDdate -r <epoch>branch wins and renders local time, so the pair is consistent; on GNU/Linuxdate -rtreats its argument as a FILE, fails, and the UTC-rendered fallback is used, shifting both window ref files by the host's UTC offset. Failure: on Linux with TZ=America/Toronto, harvesting a task whose window is [T, T+10min] sets $REFDIR/start to T+4h-1s and $REFDIR/end to T+4h+1s, sofind ... -newer "$REFDIR/start"(line 172) excludes every real in-window session log; matched_files returns empty, and the ledger row is written with source "unavailable" and null input/cached/output/reasoning tokens for a task that has full claude or codex logs on disk. East-of-UTC zones shift the refs earlier and silently truncate the tail of the window instead. The script otherwise carries deliberate GNU fallbacks (stat -c %Y/%W at lines 110/116), so this is portability code that does not hold. Fix: drop-ufrom the fallback so both branches emit local time (date -d "@$1" +%Y%m%d%H%M.%S); iso_from_epoch's-uat line 125 is correct as-is because it wants UTC output.bin/fm-usage-harvest.sh:316- The row is staged in "$DATA/usage-ledger.jsonl.tmp.$$" and removed only in the success branch ofcat ... >> "$LEDGER" && rm -f -- .... harvest_cleanup (lines 141-146) removes $REFDIR and releases the lock but never this file. Failure: whenever the ledger append fails (the exact condition tests/fm-usage-harvest.test.sh:400 forces by making $data/usage-ledger.jsonl a directory, and which teardown deliberately tolerates at bin/fm-teardown.sh:2546), set -eu aborts after the failedcatand ausage-ledger.jsonl.tmp.<pid>file is left permanently in the fleet's runtime data dir, accumulating one per failed teardown harvest. Register the temp path with harvest_cleanup so the trap removes it on every exit path.tests/fm-backend-herdr-smoke.test.sh- tests/fm-backend-herdr-smoke.test.sh fails non-deterministically on this machine, but it is pre-existing environment drift rather than a regression from this change. Post-change runs failed 3/3 ("expected exactly 1 tab (the seeded default pruned)" x1, "the secondmate-shaped home's list_live did not see its own task" x1, and one run at the case this change modified, "create_task should refuse a duplicate label whose pane hosts a genuinely live registered agent"). Running the pre-change suite together with the pre-change bin/backends/herdr.sh from a scratch copy failed 2/2 with the same two signatures, so the suite is already red on the unmodified baseline. The single failure at the newly modified live-duplicate case did not reproduce: I re-ran that exact fixture (create_task -> herdr_pane_run_foreground -> pane report-agent -> classify) five consecutive times against real herdr and got 'live' 5/5. Nothing here needs a code change; it is recorded so a red real-Herdr lane in CI is not misread as caused by this fix.bin/fm-test-run.sh tests/fm-backend-herdr.test.sh tests/fm-control-relaunch.test.sh- green (both suites, including the 6 new classifier cases, the stale-registration husk replacement case, and both new exited-shell relaunch cases)bash tests/fm-control-herdr-smoke.test.sh- green against real herdr 0.8.2 (9/9: agent-free classification, idempotent exit with no/quittyped, interrupt refusal, live-process registration stays alive, fail-closed exit, endpoint/worktree/branch preserved)FM_HERDR_AGENT_FREE_PROOF=1 bash tests/fm-herdr-agent-free-proof-live-e2e.test.sh- green; launched claude 2.1.257, codex 0.150.1, opencode 1.17.11, pi 0.84.4, cursor 2026.08.31 for real in Herdr panes, none proved a bare idle shell; pi-signed/grok/kimi/muse reported as not installedManual end-to-end reproduction, this branch:bin/fm-control.sh demo relaunch --note '...'on a real Herdr lab pane holding bare zsh with a staleherdr pane report-agentregistration, over a real git worktree with committed + uncommitted work (evidence dirrelaunch-e2e-demo.sh, transcriptrelaunch-e2e-after-fixed-classifier.txt)Manual end-to-end reproduction, pre-fix baseline: the identical scenario against a scratch copy of the repo withgit show 12f6613:bin/backends/herdr.shrestored - reproducesalive,/quittyped at the zsh prompt, and a failed relaunch (transcriptrelaunch-e2e-before-prefix-classifier.txt)bash tests/fm-backend-herdr-smoke.test.shx3 post-change and x2 on the pre-change suite + pre-change classifier - fails in both, same signatures, so its red state is pre-existing local herdr 0.8.2 driftFlakiness check: the modified live-duplicate fixture (create_task ->herdr_pane_run_foreground->pane report-agent->fm_backend_herdr_pane_agent_state) driven 5 consecutive times against real herdr -live5/5🔧 **Document** - 1 issue found → auto-fixed (3) ✅
docs/verification/runtime-backends.md:909- This branch changed the crewmate launch autonomy posture (claude --permission-mode auto, codex -s workspace-write -a never, cursor --auto-review --sandbox enabled) but no maintainer-verification record measures the new posture. The only autonomy evidence in the repo is Cursor's dated --yolo measurement, which now documents a flag firstmate no longer passes; claude and codex never had autonomy rows at all. I scoped the Cursor record so it can no longer be read as current, but I cannot close the evidence gap by writing prose: whether a crewmate still runs fully unattended, and whether Cursor's enabled sandbox still permits the git and network work a crewmate needs, requires a real measurement. Recommend a follow-up live run per harness that refreshes the Cursor 'Launch, lifecycle, and skills' rows and adds an equivalent record for claude and codex.🔧 Fix: add live crewmate autonomy guard; measure launch write contract
5 issues (2 errors, 2 warnings, 1 info) still open:
bin/fm-spawn.sh:1117- MEASURED DEFECT, deliberately not fixed here per the task's hard boundary. Under the launch posture this branch landed (codex -s workspace-write -a never), a codex crewmate cannot perform the one write bin/fm-brief.sh gives every worker to report status. Driven for real through the harness in an isolated fm-lab- Herdr session on 2026-09-02, codex-cli 0.152.1, macOS aarch64, with the status file at a lab FM_HOME under $HOME (never TMPDIR, which workspace-write grants by default and which would have made the measurement vacuous), codex answered: 'FMREPLY...zsh:1: operation not permitted: <labhome>/state/codex-<id>.status'. The sandbox denies the append and -a never removes any escalation path, so the model sees an execution failure it cannot resolve. bin/fm-brief.sh:179 and :318-321 make that append the ONLY way a worker reports done, blocked, needs-decision, paused, or failed, so every codex crewmate launched under this posture is mute to the control plane. Claude under --permission-mode auto performs the same append successfully ('Allowed by auto mode classifier'), so this is specific to the codex sandbox, not to the contract. Resolving it is an authority and security decision (widen codex's writable roots, relocate the status file, or change the approval policy) that belongs to the captain, not to this step.docs/verification/runtime-backends.md:209- The dated maintainer-verification record this task asked for is NOT yet written. I added the guard, registered it, and produced real measurements for claude and codex, and I pointed the Cursor 'Launch, lifecycle, and skills' rows at a '#crewmate-autonomy-and-the-status-file-write-contract' anchor, but the section that anchor names does not exist yet because the cursor leg of the final run had not completed when this step had to report. Two consequences: that intra-document link is currently dangling (bin/fm-doc-audience-check.sh passed before the pointer edit, so it has not been revalidated against the dangling anchor), and the guard's own header references the same unwritten section. The measured content that belongs in it is: claude 2.1.258 reaches its composer only after the first-launch workspace-trust prompt (measured identically under the old --dangerously-skip-permissions and today's --permission-mode auto, so not a regression from this branch) and DOES append its status file; codex 0.152.1 reaches its composer only after a directory-trust prompt and a 'Hooks need review' prompt and does NOT append its status file; cursor 2026.08.31-4057e58 reaches an empty composer with no key sent under --trust --auto-review --sandbox enabled, and its status-write result is unmeasured.tests/fm-crewmate-autonomy-live-e2e.test.sh:1- An earlier iteration of this guard changed state outside the worktree on this machine, and you should know about it. Codex 0.150.1 rendered a classifiable empty composer and raised its update-available modal a moment later; the guard trusted that earlier read, submitted its prompt, and the Enter landed on the modal's preselected 'Update now' row, which installed codex-cli 0.152.1 (~/.codex/packages/standalone/releases/0.152.1-aarch64-apple-darwin). 0.150.1 is still on disk if you want to pin back. The shipped guard is hardened against a repeat: every key it sends is now gated on composer_settled, an empty composer verdict that must hold for a full 10s window, and update offers are declined with Escape (never Enter, which runs the upgrade). The codex denial finding above was measured on 0.152.1 after the upgrade..agents/skills/harness-adapters/SKILL.md:193- Measured, and partially fixed in place. Claude Code 2.1.258 shows its workspace-trust prompt on every worktree path it has not seen, which is every firstmate task, and the launch flags do not change that: I measured the identical prompt under --dangerously-skip-permissions and under today's --permission-mode auto in two fresh directories, so it is not a regression from this branch. What WAS wrong in the docs is the recovery step: the prompt preselects 'No, exit', so the documented 'bin/fm-send.sh <window> --key Enter' DECLINES and quits Claude. I corrected that paragraph to say Down then Enter. Flagging it because it means no claude crewmate spawn is genuinely unattended today and the captain must still clear the prompt by hand; whether to solve that (a persisted trust entry, a stable worktree root) is a product decision I did not take.tests/fm-herdr-submit-confirm-live-e2e.test.sh:86- tests/fm-herdr-submit-confirm-live-e2e.test.sh still launches Claude with --dangerously-skip-permissions, a flag bin/fm-spawn.sh no longer passes. It does not affect what that guard measures (submit confirmation is independent of permission mode), and changing it is a test-behavior edit outside this documentation step's authority, so I left it. Worth aligning with --permission-mode auto next time that file is touched, so every live guard exercises the posture firstmate actually ships.🔧 Fix: Document crewmate sandbox verification status
1 error still open:
docs/verification/runtime-backends.md:932- The corrected Codex and Cursor writable-root postures remain unmeasured. Run the opt-in guard to prove both permitted writes, denial outside the grant, Cursor workspace binding, and Codex 0.152.1 submit compatibility before treating the new sandbox contract as verified.🔧 Fix: record measured crewmate autonomy and sandbox write contract
✅ Re-checked - no issues remain.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.