diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index 07428f7b8fd..d8469c82613 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -242,7 +242,7 @@ SSH exit 255 preserves the route and local records because remote completion is When safe, teardown kills the direct endpoint, removes the `data/secondmates.md` route, clears the main home metadata, and removes the retired secondmate home. Removing a leased home releases its durable treehouse lease via `treehouse return`, so the pool slot is freed for reuse rather than left leased forever. A plain-clone home with no pool slot is simply removed. -If `treehouse return` fails for a leased home, teardown stops with state intact rather than raw-removing the directory and hiding a held lease. +If `treehouse return` fails for a leased home, teardown stops rather than raw-removing the directory and hiding a held lease: the home and its records stay, but that home's worktree claim retirement is parked, so a rerun refuses until the manual-recovery drill that refusal prints has been worked through. Before either return or direct removal, teardown asks the target home's process-event runner to retire its registrations and physically owned machine-wide claims through the safe generation-bound path. It refuses retirement while that cleanup is uncertain or unavailable, preserving the home and retirement records for a later retry. Raw deletion is unsupported because a blocking process-event child can outlive its home. diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index 64d809c798d..dc894913d4b 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -35,6 +35,19 @@ Use `treehouse status` for treehouse-backed tmux, herdr, zellij, or cmux tasks, Do not sweep another home's endpoints or infer ownership from a matching window label. Before relaunch, prove that no live agent still owns the recorded task and that the existing worktree remains available. +`bin/fm-control.sh relaunch` and its `bin/fm-spawn.sh --relaunch` delegate enforce the shared ownership proof in `bin/fm-worktree-ownership-lib.sh`; a worktree whose `.fm-task-owner` marker names another task or another spawn generation, a conflicting task claim, a contradicting provider binding, a foreign task branch, or a mismatched secondmate home marker each refuse before either path acts on the recorded worktree. +A record with no `worktree=` line has already had that claim retired, so relaunching it refuses rather than adopting a path the provider may have taken back. +Only an interrupted retirement can leave a recoverable copy of the claim, and the refusal prints one manual-recovery drill naming that copy together with the slot's retired `.fm-task-owner` copy whenever that half was stashed too; put both halves back together or neither, and only after confirming with the provider that the path was never released, because a released slot may already belong to another task and a claim restored without its marker can never prove ownership again. +A slot whose `.fm-task-owner` the record cannot prove is its own - one naming another task or another spawn generation, an unreadable or incomplete one, or an entry that is not a regular file - refuses before any retirement begins: the record keeps its `worktree=` line, no claim or marker copy is written, no drill is printed, and the entry is left byte-for-byte as it is. +Nothing is parked there, so attribute that entry by the rules below and rerun once it is resolved, instead of looking for a preserved copy that was never made. +Every retirement that has already begun and then stops short of a confirmed provider release parks that way instead: nothing is recorded as final on its own, no runtime path restores either half, and the drill stops rather than creating or overwriting another owner's marker. +Until an operator works through that drill, the parked record keeps refusing its own teardown and relaunch, so reconcile it deliberately instead of expecting a rerun to finish the remaining cleanup. +A pool slot refused because its `.fm-task-owner` marker names a task with no record was taken by a spawn that was killed before it published one; the refusal reports whether that spawn's ownership record still stands beside the task records in `state/` and names both files, and clearing them by hand after confirming the slot is idle is the only recovery, because no teardown exists for a task that was never recorded. +A fresh spawn of that same task id refuses while any such record still names a slot that carries that marker, or that the spawn cannot read well enough to tell, naming each unresolved spawn generation and the worktree it took, so resolve the stranded slot first and then spawn the id again. +A record whose recorded worktree no longer carries that marker strands nothing: the spawn reports it as leftover paperwork that is safe to delete and proceeds, so it never wedges the id for good. +The same removal is the recovery when the refusal reports that the marked task's own record has moved on to another generation and another worktree, because that task's teardown retires its marker elsewhere and will never clear this slot. +When the refusal instead reports that the marked task's record does not say clearly enough who owns the slot, ownership is unknown: remove nothing, repair that record or establish with the crew which task is working there first. +A relaunch interrupted between restamping the worktree's marker and advancing its record leaves the marker one generation ahead; that is recorded as a generation handoff and needs no repair, so relaunch or tear the task down normally rather than editing either half by hand. Preserve its uncommitted changes and commits, keep the same task identity, and resume or relaunch the recorded harness in that existing worktree with the same brief plus a concise progress note. Do not use a fresh generic spawn while the recorded worktree is unaccounted for, because allocating another worktree can split one task across two copies. If the worktree or ownership cannot be reconciled safely, leave all state intact and report the task failed or blocked with the conflicting evidence. diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index 2882f4a6af2..6792dc2cb32 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -383,8 +383,43 @@ fm_backend_endpoint_atom_valid() { # esac } -fm_backend_validate_task_endpoint() { # - local meta=$1 id=$2 backend_count backend window worktree project binding_count binding +# A worktree-claim retirement (bin/fm-worktree-ownership-lib.sh) is the one way +# a live record loses its worktree identity, and its surviving copy is the only +# remaining record of the path, so name it wherever that loss surfaces. Whether +# the retirement is RECORDED decides the story, not which namespace the copy +# happens to sit in: a recorded retirement - by receipt or by quarantined +# released evidence - means this record holds no path at all and every surviving +# copy is evidence only. An unrecorded retirement parks instead: no runtime path +# restores it, and the same manual drill printed by ownership proof names the +# preserved state and deliberate reconciliation steps. +fm_backend_report_worktree_claim_backup() { # + local meta=$1 backup evidence retired=1 + if declare -F fm_worktree_retirement_receipt_present >/dev/null 2>&1 \ + && fm_worktree_retirement_receipt_present "$meta" >/dev/null 2>&1; then + retired=0 + fi + if declare -F fm_worktree_released_evidence_hint >/dev/null 2>&1 \ + && evidence=$(fm_worktree_released_evidence_hint "$meta" 2>/dev/null) \ + && [ -n "$evidence" ]; then + echo "This record's worktree was already retired and that path may already belong to another task, so a copy of the record it held was quarantined at $evidence. It is evidence of that retirement, never authority over the path, and must never be restored over the record." >&2 + return 0 + fi + declare -F fm_worktree_claim_backup_hint >/dev/null 2>&1 || return 0 + backup=$(fm_worktree_claim_backup_hint "$meta" 2>/dev/null) || return 0 + [ -n "$backup" ] || return 0 + if [ "$retired" -eq 0 ]; then + echo "This record's retirement is recorded, so it holds no worktree and the superseded copy at $backup names a path it no longer owns; it is evidence only and must never be restored over the record." >&2 + return 0 + fi + if declare -F fm_worktree_interrupted_retirement_manual_drill >/dev/null 2>&1; then + echo "An interrupted worktree retirement is parked.$(fm_worktree_interrupted_retirement_manual_drill "$meta")" >&2 + else + echo "An interrupted worktree retirement is parked at $backup; automatic restoration is disabled, so preserve the copy and reconcile the provider outcome manually before any lifecycle action." >&2 + fi +} + +fm_backend_validate_task_endpoint() { # [allow-retired] + local meta=$1 id=$2 allow_retired=${3:-} backend_count backend window worktree project binding_count binding local session pane recorded_session workspace tab terminal worktree_id surface FM_BACKEND_VALIDATED_BACKEND= FM_BACKEND_VALIDATED_TARGET= @@ -401,8 +436,17 @@ fm_backend_validate_task_endpoint() { # return 1 } worktree=$(fm_backend_meta_exact_value "$meta" worktree) || { - echo "REFUSED: task $id has a missing, empty, or ambiguous worktree identity; preserving task state." >&2 - return 1 + # A retired record legitimately identifies no worktree: its provider step + # already released the path. Only cleanup asks to accept that, and it gets + # no path to act on either way. + if [ "$allow_retired" != allow-retired ] \ + || ! declare -F fm_worktree_retirement_receipt_present >/dev/null 2>&1 \ + || ! fm_worktree_retirement_receipt_present "$meta" >/dev/null 2>&1; then + echo "REFUSED: task $id has a missing, empty, or ambiguous worktree identity; preserving task state." >&2 + fm_backend_report_worktree_claim_backup "$meta" + return 1 + fi + worktree= } project=$(fm_backend_meta_exact_value "$meta" project) || { echo "REFUSED: task $id has a missing, empty, or ambiguous project identity; preserving task state." >&2 diff --git a/bin/fm-control.sh b/bin/fm-control.sh index 12387b0602d..5cabdacb360 100755 --- a/bin/fm-control.sh +++ b/bin/fm-control.sh @@ -44,6 +44,9 @@ # inherits the local copy but none of the conversation; a # secondmate reconciles its own home's records at startup, so its # standing charter is never rewritten. +# Before any checkpoint, stop, or worktree mutation, the shared +# ownership resolver proves that no other task claims the path and +# that its provider binding plus task branch or home marker agree. # Records a durable checkpoint and that note, exits the old agent, # then delegates the launch to its single owner, # bin/fm-spawn.sh --relaunch. A failure before publication keeps @@ -126,6 +129,8 @@ DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" # shellcheck source=bin/fm-backend.sh . "$SCRIPT_DIR/fm-backend.sh" +# shellcheck source=bin/fm-worktree-ownership-lib.sh +. "$SCRIPT_DIR/fm-worktree-ownership-lib.sh" # shellcheck source=bin/fm-busy-lib.sh . "$SCRIPT_DIR/fm-busy-lib.sh" # shellcheck source=bin/fm-control-lib.sh @@ -785,6 +790,7 @@ do_relaunch() { local exit_result state note_line local -a spawn_args + fm_worktree_ownership_prove "$STATE" "$ID" "$META" || return 1 require_state_verified_backend relaunch resolve_relaunch_profile diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 9158fce64df..df5b0a61ced 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -30,8 +30,10 @@ # ordinary relaunch. It refuses unless the recorded endpoint is positively # agent-free on a backend with a recovery-grade agent-state classifier (tmux # or herdr), refuses unless the endpoint's shell is sitting in the recorded -# worktree, and clears the previous harness's per-task wiring before arming -# the new incarnation. +# worktree, proves that path still belongs only to this task through +# bin/fm-worktree-ownership-lib.sh, restamps the worktree's .fm-task-owner +# marker, and clears the previous harness's per-task wiring before arming the +# new incarnation. # --harness is the explicit per-spawn harness/profile adapter. The old # positional harness arg still works for back-compat. # --model and --effort are concrete profile @@ -134,6 +136,17 @@ # default-branch commit when safe; skipped syncs warn and launch unchanged. # Ship/scout spawns refuse to launch unless the resolved task path is a real # git worktree root distinct from the primary project checkout. +# A ship or scout spawn then stamps its task id and this spawn generation into +# the worktree's .fm-task-owner marker, excluded from git like the other +# per-task worktree files, so a recycled pool slot can never host two live +# workers. A fresh spawn refuses EVERY marker already there - another task's, +# one it cannot attribute, and one carrying this same task id - because task +# identity alone cannot prove that the marked generation stopped working. +# Only --relaunch, which has the recorded metadata that proves a same-task +# handoff, may restamp a marker this task already owns. +# bin/fm-worktree-ownership-lib.sh owns what that marker proves for every +# later lifecycle verb, and bin/fm-teardown.sh retires it when the slot is +# released. # Before a fresh ship or scout worker starts, its clean task worktree fetches # origin, resolves the current remote default branch, and resets to its tip. # An unreachable origin, unresolved default branch, or non-clean worktree @@ -289,6 +302,8 @@ fm_backlog_directory_present "$STATE" "state directory" || { . "$SCRIPT_DIR/fm-config-inherit-lib.sh" # shellcheck source=bin/fm-backend.sh . "$SCRIPT_DIR/fm-backend.sh" +# shellcheck source=bin/fm-worktree-ownership-lib.sh +. "$SCRIPT_DIR/fm-worktree-ownership-lib.sh" # shellcheck source=bin/fm-control-lib.sh . "$SCRIPT_DIR/fm-control-lib.sh" # shellcheck source=bin/fm-gate-refuse-lib.sh @@ -699,6 +714,9 @@ spawn_remote_secondmate() { } BACKEND= +SPAWN_TASK_OWNER_STAMPED=0 +SPAWN_TASK_OWNER_BACKUP= +SPAWN_TASK_OWNER_PENDING=0 ORCA_ABORT_CLEANUP=0 ORCA_WORKTREE_ID= ORCA_TERMINAL= @@ -757,6 +775,10 @@ parse_orca_worktree_result() { spawn_abort_cleanup() { local status=$? + if [ "$SPAWN_TASK_OWNER_STAMPED" = 1 ] || [ "$SPAWN_TASK_OWNER_PENDING" = 1 ]; then + clear_aborted_task_worktree_owner_stamp \ + || echo "warning: task $ID's worktree owner marker could not be cleared after an aborted spawn" >&2 + fi if [ "$RELAUNCH_REPLACEMENT_PENDING" = 1 ] \ && [ "$SPAWN_META_PUBLISH_STARTED" = 1 ] \ && [ -n "$SPAWN_META_TMP" ] \ @@ -830,6 +852,13 @@ spawn_abort_cleanup() { echo "effort=${EFFORT:-default}" echo "backend=orca" echo "orca_worktree_id=$ORCA_WORKTREE_ID" + # The awareness bit is only ever emitted with the generation it + # binds: a record promising a marker but naming no generation can + # never prove which stamp it means. + if [ -n "${SPAWN_GEN:-}" ]; then + echo "spawn_gen=$SPAWN_GEN" + [ "$KIND" = secondmate ] || echo "task_owner_marker=1" + fi [ -z "${ORCA_TERMINAL:-}" ] || echo "terminal=$ORCA_TERMINAL" } > "$SPAWN_META_TMP" 2>/dev/null \ && fm_backlog_atomic_transition publish "$SPAWN_META_TMP" "$STATE/$ID.meta" "task record" "$STATE" \ @@ -868,6 +897,68 @@ spawn_abort_cleanup() { } trap spawn_abort_cleanup EXIT +# Whether an ownership record left behind by an interrupted spawn is still +# holding a pool slot; refuse_unresolved_task_owner_pending_claims below owns +# why that matters. +# 0 when the worktree an ownership record names still carries this exact task +# and generation's marker, so a slot really is stranded behind it; 1 when the +# path provably strands nothing (it is gone, carries no marker, or carries one +# that is some other task's or generation's); 2 when that cannot be established +# and the claim must be treated as unresolved. +task_owner_pending_claim_strands_a_slot() { # + local pending=$1 worktree generation marker + generation=$(fm_worktree_meta_exact_value "$pending" spawn_gen 2>/dev/null || true) + worktree=$(fm_worktree_meta_exact_value "$pending" worktree 2>/dev/null || true) + [ -n "$generation" ] && [ -n "$worktree" ] || return 2 + [ -d "$worktree" ] || return 1 + marker="$worktree/$FM_WORKTREE_TASK_OWNER_MARKER" + [ -e "$marker" ] || [ -L "$marker" ] || return 1 + [ -f "$marker" ] && [ ! -L "$marker" ] || return 2 + [ "$(fm_worktree_meta_exact_value "$marker" task_id 2>/dev/null || true)" = "$ID" ] || return 1 + [ "$(fm_worktree_meta_exact_value "$marker" spawn_gen 2>/dev/null || true)" = "$generation" ] || return 1 + return 0 +} + +# An ownership record that outlived the spawn that published it means some pool +# slot may still carry this task id's owner marker with no task record to +# attribute it. Recovery is told to keep the same task identity, so the very +# next thing that happens is a fresh spawn of this id - and handing it a second +# slot would strand the first one behind a marker nothing can name. Refuse +# instead, naming every unresolved generation and the path it took, so the +# operator resolves those before this id takes another worktree. +# A record whose slot provably strands nothing is not a reason to refuse: it is +# leftover paperwork, so it is reported as safe to delete and stepped over +# rather than wedging the id for good. Nothing here removes it - this runs +# under this task's spawn lock, but the record is still evidence somebody else +# may be reading. +refuse_unresolved_task_owner_pending_claims() { # + local state=$1 id=$2 pending generation worktree strands + local -a claims=() resolved=() + while IFS= read -r pending; do + [ -n "$pending" ] || continue + strands=0 + task_owner_pending_claim_strands_a_slot "$pending" || strands=$? + if [ "$strands" -eq 1 ]; then + resolved+=("$pending") + else + claims+=("$pending") + fi + done < <(fm_worktree_owner_pending_list "$state" "$id") + for pending in ${resolved[@]+"${resolved[@]}"}; do + worktree=$(fm_worktree_meta_exact_value "$pending" worktree 2>/dev/null || true) + echo "warning: task $id's ownership record at $pending is already resolved - ${worktree:-the worktree it names} carries no $FM_WORKTREE_TASK_OWNER_MARKER for it - so it is not blocking this spawn and is safe to delete" >&2 + done + [ "${#claims[@]}" -gt 0 ] || return 0 + echo "error: task $id has ${#claims[@]} unresolved worktree ownership record(s) from an earlier spawn that never published a task record; refusing to allocate another worktree for task $id while they are unresolved" >&2 + for pending in "${claims[@]}"; do + generation=$(fm_worktree_meta_exact_value "$pending" spawn_gen 2>/dev/null || true) + worktree=$(fm_worktree_meta_exact_value "$pending" worktree 2>/dev/null || true) + echo " spawn generation ${generation:-unreadable} took ${worktree:-an unrecorded worktree}; its ownership record is $pending" >&2 + done + echo "Confirm no agent is still working in each recorded worktree, remove its $FM_WORKTREE_TASK_OWNER_MARKER marker there if it still names task $id, then delete the ownership record and spawn again." >&2 + return 1 +} + # One bounded lock per live Herdr session/socket, shared across all homes. # is required so secondmate and primary spawns serialize against the # same session without writing any other home's state directory. @@ -1079,6 +1170,9 @@ if ! fm_lock_try_acquire "$SPAWN_TASK_LOCK"; then exit 1 fi SPAWN_TASK_LOCK_HELD=1 +if [ "$RELAUNCH" -eq 0 ]; then + refuse_unresolved_task_owner_pending_claims "$STATE" "$ID" || exit 1 +fi PROJ= ARG3= FIRSTMATE_HOME= @@ -1111,6 +1205,7 @@ if [ "$RELAUNCH" -eq 1 ]; then exit 1 } fm_backend_validate_task_endpoint "$RELAUNCH_META" "$ID" || exit 1 + fm_worktree_ownership_prove "$STATE" "$ID" "$RELAUNCH_META" || exit 1 BACKEND=$FM_BACKEND_VALIDATED_BACKEND RELAUNCH_TARGET=$FM_BACKEND_VALIDATED_TARGET fm_backend_validate_spawn "$BACKEND" || exit 1 @@ -2100,10 +2195,10 @@ case "$BACKEND" in # it stands up a DIFFERENT home's own workspace by design - so it asks for # the per-home container instead of inheriting this launcher's. HERDR_LABEL_HOME=$FM_HOME - HERDR_LAUNCHER_RELATIONSHIP=launcher-home + HERDR_LAUNCHER_RELATIONSHIP='launcher-home' if [ "$KIND" = secondmate ]; then HERDR_LABEL_HOME=$PROJ_ABS - HERDR_LAUNCHER_RELATIONSHIP=other-home + HERDR_LAUNCHER_RELATIONSHIP='other-home' fi HERDR_PRESENTATION_JOURNAL=$(fm_backend_herdr_projection_journal_path "$STATE" "$ID") HERDR_PROJECTED=0 @@ -2403,6 +2498,247 @@ kimi_spawn_fail() { # echo "error: $1; inspect window $T" >&2 } +exclude_path() { + local rel=$1 EXCL + EXCL=$(git -C "$WT" rev-parse --git-path info/exclude 2>/dev/null || true) + [ -n "$EXCL" ] || return 0 + mkdir -p "$(dirname "$EXCL")" + grep -qxF "$rel" "$EXCL" 2>/dev/null || echo "$rel" >> "$EXCL" +} + +# Prints the task id an existing owner marker names, or the reason it cannot be +# attributed at all. Silent and 0 only when there is no marker, or when a +# relaunch finds this task's own marker after the metadata-backed ownership +# proof above accepted that exact handoff. A fresh spawn refuses every existing +# marker, including one with the same task id: without relaunch metadata, task +# identity alone cannot prove that the marked generation stopped working. +task_worktree_owner_marker_holder() { # + local marker=$1 owner + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + if [ ! -f "$marker" ] || [ -L "$marker" ]; then + printf '%s' 'an unreadable marker that is not a regular file' + return 1 + fi + owner=$(fm_worktree_meta_exact_value "$marker" task_id 2>/dev/null || true) + if [ -z "$owner" ]; then + printf '%s' 'a marker with no readable task identity' + return 1 + fi + if [ "$RELAUNCH" -eq 1 ] && [ "$owner" = "$ID" ]; then + return 0 + fi + printf 'task %s' "$owner" + return 1 +} + +# 0 when the slot's marker is still exactly the one this run stamped - same task +# and same spawn generation - 2 when no marker is there at all, and 1 when it +# now carries anything else. A rollback may only undo a marker that is still +# byte-for-byte this run's transition; anything else has moved on and is not +# this run's to put back or remove. +task_worktree_owner_marker_is_this_stamp() { # + local marker=$1 + [ -e "$marker" ] || [ -L "$marker" ] || return 2 + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + [ "$(fm_worktree_meta_exact_value "$marker" task_id 2>/dev/null || true)" = "$ID" ] || return 1 + [ "$(fm_worktree_meta_exact_value "$marker" spawn_gen 2>/dev/null || true)" = "$SPAWN_GEN" ] || return 1 +} + +# Names the recovery an operator can actually perform for the marker that just +# refused this slot. A teardown only exists while the marked task still has a +# record; an interrupted spawn leaves its ownership record instead, and a marker +# nothing attributes leaves only itself. Nothing here reclaims, overwrites, or +# removes the foreign marker - the slot stays refused either way. +report_foreign_owner_marker_remedy() { # + local marker=$1 attribution owner generation pending + attribution=$(fm_worktree_owner_marker_attribution "$STATE" "$marker" 2>/dev/null) || attribution= + owner=$(fm_worktree_meta_exact_value "$marker" task_id 2>/dev/null || true) + generation=$(fm_worktree_meta_exact_value "$marker" spawn_gen 2>/dev/null || true) + case "$attribution" in + record) + echo "Its owner marker is $marker; release that slot through task $owner's own teardown before reusing it." >&2 + ;; + pending) + pending=$(fm_worktree_owner_pending_path "$STATE" "$owner" "$generation") + echo "Its owner marker is $marker. Task $owner has no task record: its spawn was interrupted after it took this slot, and its ownership record at $pending names this same slot and generation." >&2 + echo "No teardown exists for task $owner. Confirm no agent is still working in $WT, then remove $marker and $pending to release the slot." >&2 + ;; + stale) + echo "Its owner marker is $marker, but task $owner's own record names a different spawn generation and a different worktree, so task $owner's teardown will never retire this marker." >&2 + echo "Confirm no agent is still working in $WT, then remove $marker to release the slot." >&2 + ;; + unreadable) + echo "Its owner marker is $marker, and task $owner's record in $STATE does not say clearly enough who owns this slot: its spawn generation or worktree line is missing, duplicated, or not a usable path." >&2 + echo "Ownership is unknown, so nothing here may be removed - this marker may be protecting a live worker. Repair task $owner's record, or establish with the crew which task is working in $WT, before reusing this slot." >&2 + ;; + orphan) + echo "Its owner marker is $marker, but no task record and no spawn ownership record in $STATE attributes it to any task." >&2 + echo "No teardown exists for it. Confirm no agent is still working in $WT, then remove $marker to release the slot." >&2 + ;; + *) + echo "Its owner marker is $marker, and it cannot be read well enough to attribute to any task." >&2 + echo "Confirm no agent is still working in $WT, then repair or remove $marker to release the slot." >&2 + ;; + esac +} + +# Stamp this task's identity and this exact spawn generation into the worktree +# after the in-pane treehouse subshell has settled there. The generation keeps +# a recycled slot distinguishable even when a later task reuses the same id. +# bin/fm-worktree-ownership-lib.sh compares both values with metadata, and +# bin/fm-teardown.sh removes the marker before the slot is released. Exclude it +# from git first so it can never read as uncommitted work. +stamp_task_worktree_owner() { + local marker tmp holder prior + [ -n "${WT:-}" ] && [ -d "$WT" ] || return 0 + marker="$WT/$FM_WORKTREE_TASK_OWNER_MARKER" + if ! holder=$(task_worktree_owner_marker_holder "$marker"); then + if [ "$RELAUNCH" -eq 0 ]; then + echo "error: worktree $WT already belongs to $holder; its marker is not task $ID's fresh-spawn authority, because a fresh spawn refuses any existing owner marker and only relaunch has metadata that can prove a same-task handoff" >&2 + else + echo "error: worktree $WT already belongs to $holder, not task $ID; refusing to relaunch task $ID into another task's workspace" >&2 + fi + report_foreign_owner_marker_remedy "$marker" + return 1 + fi + exclude_path "$FM_WORKTREE_TASK_OWNER_MARKER" + # Rewriting a marker this task already owns is one exact generation + # transition, not two independent writes, so the record published below names + # the generation being replaced. The record is what proves, after a crash + # between the rewrite and the metadata advance, that the marker one generation + # ahead is still this task's. Anchored on what the record currently publishes, + # because that is the value every later ownership proof compares against. + prior= + if [ -f "$marker" ] && [ ! -L "$marker" ]; then + prior=$(fm_worktree_meta_exact_value "$STATE/$ID.meta" spawn_gen 2>/dev/null || true) + [ -n "$prior" ] || prior=$(fm_worktree_meta_exact_value "$marker" spawn_gen 2>/dev/null || true) + fi + # The durable half of the binding goes down first. Everything from here to + # the record's publication - the base freshen's network fetch and the whole + # harness launch - is a window a SIGKILL or a reboot can end without running + # this script's abort trap, and a marker stamped without this record would + # survive that as an orphan no operator could attribute or safely clear. + if ! fm_worktree_owner_pending_write "$STATE" "$ID" "$SPAWN_GEN" "$WT" "$prior"; then + echo "error: could not publish task $ID's ownership record for $WT in $STATE; refusing to stamp an owner marker nothing could attribute after an interrupted spawn" >&2 + return 1 + fi + SPAWN_TASK_OWNER_PENDING=1 + # The prior marker remains authoritative until the final mv below. Take its + # rollback copy only after the durable handoff exists, so a failure to publish + # that handoff cannot leak a backup the abort path never learned to clean. + if [ "$RELAUNCH" -eq 1 ] && [ -f "$marker" ] && [ ! -L "$marker" ]; then + SPAWN_TASK_OWNER_BACKUP=$(umask 077; mktemp "$STATE/.$ID.task-owner-prior.XXXXXX") || return 1 + if ! cp -p -- "$marker" "$SPAWN_TASK_OWNER_BACKUP"; then + rm -f -- "$SPAWN_TASK_OWNER_BACKUP" + SPAWN_TASK_OWNER_BACKUP= + return 1 + fi + fi + tmp="$marker.next.${BASHPID:-$$}" + if ! (umask 077; { + printf '%s\n' 'schema=fm-task-owner.v1' + printf 'task_id=%s\n' "$ID" + printf 'spawn_gen=%s\n' "$SPAWN_GEN" + } > "$tmp") \ + || ! mv -f -- "$tmp" "$marker"; then + rm -f -- "$tmp" + echo "error: could not stamp task $ID's ownership marker into $WT; refusing to launch into a worktree nothing can attribute" >&2 + return 1 + fi + SPAWN_TASK_OWNER_STAMPED=1 +} + +# The marker and the task record are two halves of one generation binding, so +# the rollback decision is read from what the record itself publishes rather +# than from how far this run got. Once state/.meta names this spawn +# generation and explicitly promises an owner marker, restoring the superseded +# marker would contradict the live record and wedge every later ownership +# proof, so the stamp is committed instead. +task_record_publishes_owner_binding() { + local meta="$STATE/$ID.meta" recorded awareness + [ -n "${SPAWN_GEN:-}" ] || return 1 + [ -f "$meta" ] && [ ! -L "$meta" ] || return 1 + recorded=$(fm_worktree_meta_exact_value "$meta" spawn_gen 2>/dev/null || true) + awareness=$(fm_worktree_meta_exact_value "$meta" task_owner_marker 2>/dev/null || true) + [ -n "$recorded" ] && [ "$recorded" = "$SPAWN_GEN" ] && [ "$awareness" = 1 ] +} + +# Retracts only the ownership record this exact generation published, so an +# abort can never withdraw a concurrent or later incarnation's claim on the +# slot. Its absence is success: nothing was published, or it is already gone. +clear_task_worktree_owner_pending() { + [ "$SPAWN_TASK_OWNER_PENDING" = 1 ] || return 0 + fm_worktree_owner_pending_clear "$STATE" "$ID" "$SPAWN_GEN" "${WT:-}" || { + echo "warning: task $ID's superseded ownership record remains at $(fm_worktree_owner_pending_path "$STATE" "$ID" "$SPAWN_GEN")" >&2 + return 1 + } + SPAWN_TASK_OWNER_PENDING=0 +} + +clear_aborted_task_worktree_owner_stamp() { + local marker="${WT:-}/$FM_WORKTREE_TASK_OWNER_MARKER" stamp_rc + if [ "$SPAWN_TASK_OWNER_STAMPED" != 1 ]; then + # The ownership record went down first, but the prior marker still occupies + # the slot until the atomic marker mv succeeds. Retract the pending record + # and discard the duplicate rollback copy; neither is authority once the + # original marker was never replaced. + stamp_rc=0 + clear_task_worktree_owner_pending || stamp_rc=$? + if [ -n "$SPAWN_TASK_OWNER_BACKUP" ]; then + if rm -f -- "$SPAWN_TASK_OWNER_BACKUP"; then + SPAWN_TASK_OWNER_BACKUP= + else + echo "warning: task $ID's unused owner-marker backup remains at $SPAWN_TASK_OWNER_BACKUP" >&2 + stamp_rc=1 + fi + fi + return "$stamp_rc" + fi + if task_record_publishes_owner_binding; then + commit_task_worktree_owner_stamp + return $? + fi + stamp_rc=0 + task_worktree_owner_marker_is_this_stamp "$marker" || stamp_rc=$? + if [ "$stamp_rc" -eq 1 ]; then + echo "error: $marker no longer carries task $ID generation $SPAWN_GEN, so task $ID's aborted spawn left it untouched" >&2 + [ -z "$SPAWN_TASK_OWNER_BACKUP" ] \ + || echo "task $ID's superseded owner marker remains recoverable at $SPAWN_TASK_OWNER_BACKUP" >&2 + SPAWN_TASK_OWNER_STAMPED=0 + clear_task_worktree_owner_pending || true + return 1 + fi + if [ "$stamp_rc" -eq 0 ]; then + if [ -n "$SPAWN_TASK_OWNER_BACKUP" ]; then + mv -f -- "$SPAWN_TASK_OWNER_BACKUP" "$marker" || return 1 + else + rm -f -- "$marker" || return 1 + fi + elif [ -n "$SPAWN_TASK_OWNER_BACKUP" ]; then + # The slot carries no marker at all, so there is no transition of this + # run's left to undo and nothing here may put one back onto a path that may + # already have been released. + echo "warning: $marker is gone, so task $ID's aborted spawn did not restore it; its superseded owner marker remains at $SPAWN_TASK_OWNER_BACKUP" >&2 + fi + SPAWN_TASK_OWNER_BACKUP= + SPAWN_TASK_OWNER_STAMPED=0 + clear_task_worktree_owner_pending +} + +commit_task_worktree_owner_stamp() { + [ "$SPAWN_TASK_OWNER_STAMPED" = 1 ] || return 0 + if [ -n "$SPAWN_TASK_OWNER_BACKUP" ] && ! rm -f -- "$SPAWN_TASK_OWNER_BACKUP"; then + echo "warning: task $ID's superseded owner-marker backup remains at $SPAWN_TASK_OWNER_BACKUP" >&2 + fi + SPAWN_TASK_OWNER_BACKUP= + SPAWN_TASK_OWNER_STAMPED=0 + # The record now publishes this generation itself, so it supersedes the + # ownership record that carried the binding until it existed. + clear_task_worktree_owner_pending || true +} + +SPAWN_GEN="s$(date +%s).${BASHPID:-$$}.$RANDOM" + if [ "$RELAUNCH" -eq 1 ]; then # No worktree is acquired: the recorded one is reused as-is. What must be # proven instead is that the adopted endpoint's shell is actually sitting in @@ -2469,6 +2805,9 @@ elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then validate_spawn_worktree "treehouse get" "$T" fi +if [ "$KIND" != secondmate ]; then + stamp_task_worktree_owner || exit 1 +fi if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ]; then freshen_spawn_worktree_base "$WT" || exit 1 fi @@ -2488,13 +2827,6 @@ mkdir -p "$TASK_TMP/gotmp" mkdir -p "$STATE" STATE_REAL=$(cd "$STATE" && pwd -P) TURNEND="$STATE_REAL/$ID.turn-ended" -exclude_path() { - local rel=$1 EXCL - EXCL=$(git -C "$WT" rev-parse --git-path info/exclude 2>/dev/null || true) - [ -n "$EXCL" ] || return 0 - mkdir -p "$(dirname "$EXCL")" - grep -qxF "$rel" "$EXCL" 2>/dev/null || echo "$rel" >> "$EXCL" -} if [ "$RELAUNCH" -eq 1 ]; then # Retire the previous incarnation's per-task harness wiring before arming the # new one. Without this, a harness switch would leave the old adapter's hook @@ -2829,7 +3161,6 @@ fi META_WINDOW=$T [ "$BACKEND" = orca ] && META_WINDOW=$W -SPAWN_GEN="s$(date +%s).${BASHPID:-$$}.$RANDOM" SPAWN_META_PATH="$STATE/$ID.meta" if [ "$SPAWN_META_LOCK_HELD" != 1 ]; then SPAWN_META_LOCK=$(fm_meta_lock_path "$STATE/$ID.meta") || exit 1 @@ -2846,7 +3177,7 @@ SPAWN_META_PATH=$SPAWN_META_TMP preserve_relaunch_meta() { awk -F= ' BEGIN { - split("window endpoint_task_id worktree project harness kind mode yolo tasktmp model effort busy_gen spawn_gen traceparent backend herdr_session herdr_workspace_id herdr_tab_id herdr_pane_id zellij_session zellij_tab_id zellij_pane_id orca_worktree_id terminal cmux_workspace_id cmux_surface_id home projects control_relaunch_tx", keys, " ") + split("window endpoint_task_id worktree project harness kind mode yolo tasktmp model effort busy_gen spawn_gen task_owner_marker traceparent backend herdr_session herdr_workspace_id herdr_tab_id herdr_pane_id zellij_session zellij_tab_id zellij_pane_id orca_worktree_id terminal cmux_workspace_id cmux_surface_id home projects control_relaunch_tx", keys, " ") for (i in keys) owned[keys[i]] = 1 } !($1 in owned) @@ -2866,6 +3197,10 @@ preserve_relaunch_meta() { echo "effort=${EFFORT:-default}" [ -z "${BUSY_GEN:-}" ] || echo "busy_gen=$BUSY_GEN" echo "spawn_gen=$SPAWN_GEN" + # This explicit bit, not spawn_gen (which predates owner markers), makes a + # missing marker authoritative for new ordinary task claims. Existing records + # are never migrated and keep their legacy ownership fallback. + [ "$KIND" = secondmate ] || echo "task_owner_marker=1" # Default-off writes no traceparent= line. # backend= is written only for a non-default (non-tmux) backend, so the # default path's meta stays byte-identical (absent backend= means tmux; @@ -2947,6 +3282,8 @@ if [ "$SPAWN_TASK_SET_LOCK_HELD" = 1 ]; then fi "$SCRIPT_DIR/fm-home-summary-refresh.sh" --best-effort || true [ "$BACKEND" = orca ] && ORCA_ABORT_CLEANUP=0 +# Metadata and the owner marker now publish the same spawn generation. +commit_task_worktree_owner_stamp || exit 1 sq_brief=$(shell_quote "$BRIEF") sq_turnend=$(shell_quote "$TURNEND") diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index ad9e042ba11..8b5e20a1b5a 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -19,6 +19,24 @@ # home with a backlog but no compatible tasks-axi refuses before cleanup. # None of this loosens the landed-work gates below: the transition runs only on # the paths that already proceed to remove the record. +# REFUSES before any worktree-based mutation unless the shared ownership +# resolver proves the recorded path still belongs only to this task. A provider +# return or removal clears worktree= before the slot can become reusable, and +# the worktree's .fm-task-owner marker retires atomically with that claim, so a +# released slot never carries a live claim from the task that just left it. +# That retirement begins only once the marker at the slot is proved absent or +# exactly this record's own: any other entry there - another task's, malformed, +# unreadable, or not a regular file - refuses before the claim is touched, with +# the record's claim, that entry, and the slot exactly as they were and nothing +# parked, so resolving that entry and rerunning is an ordinary recovery. +# Only a confirmed provider release completes a retirement that did begin. Every +# later failure or interruption parks it instead: the claim and owner-marker +# copies are preserved untouched, the record claims no path, every further +# destructive lifecycle action refuses, and one deliberate manual-recovery drill +# is printed. No runtime path restores either half, and none of it ever creates +# or overwrites another owner's marker. bin/fm-worktree-ownership-lib.sh owns +# that recovery contract. +# A force-authorized discard never bypasses that independent ownership proof. # REFUSES if the worktree holds work that has not LANDED, because cleanup # hard-resets/removes the worktree and kills its processes. Work has landed when it is # reachable from any remote-tracking branch (a fork counts as a remote, so @@ -40,6 +58,13 @@ # local-only projects additionally accept work merged into the local default # branch (firstmate performs that merge after configured approval) as a fallback # for the common case where there is no remote at all. +# Every one of those checks inspects the WORKTREE's HEAD. The fm/ ref +# is a separate thing that a detached HEAD, a paused rebase or bisect, or any +# other checked-out branch leaves uninspected, so teardown deletes that ref only +# when the ref itself proves every commit on it is on a remote-tracking branch +# or in the landed target this run already accepted. Otherwise the branch is +# retained and reported, in every mode - retention is never a refusal, and +# --force does not turn it into a deletion. # Scout tasks (kind=scout in meta) carve out of that check: their worktree is # declared scratch and the report at data//report.md is the work # product. Teardown proceeds only once the report exists and the shared @@ -71,8 +96,12 @@ # before preserving the route for retry. Teardown then discards child work, kills # child runtime endpoints, and removes the retired home. Removing a leased home # releases its durable treehouse lease so the pool slot is freed, -# never left leased forever. If the treehouse return fails, teardown leaves the -# leased home and state in place instead of hiding a still-held lease. +# never left leased forever. A return that does not confirm the release leaves +# the leased home and its records in place instead of hiding a still-held lease, +# and parks that home's claim retirement like every other unconfirmed outcome +# above, so a rerun refuses until the manual drill reconciles it. An unconfirmed +# child worktree return stops the forced cleanup the same way, with that child's +# records retained and no second destructive path allowed to retry its removal. # Usage: fm-teardown.sh [--force] # --force skips ordinary-task dirty and landed-work checks, skips scout report # checks, and discards secondmate child work for kind=secondmate. Only use it @@ -169,6 +198,8 @@ SUB_HOME_PARENT_MARKER=".fm-secondmate-parent" . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" # shellcheck source=bin/fm-backend.sh . "$SCRIPT_DIR/fm-backend.sh" +# shellcheck source=bin/fm-worktree-ownership-lib.sh +. "$SCRIPT_DIR/fm-worktree-ownership-lib.sh" # shellcheck source=bin/fm-control-lib.sh . "$SCRIPT_DIR/fm-control-lib.sh" # shellcheck source=bin/fm-lock-lib.sh @@ -226,6 +257,7 @@ DESCENDANT_TASK_KINDS=() DESCENDANT_TASK_HOMES=() teardown_release_locks() { local status=$? i + fm_worktree_claim_retire_abandon || true if declare -F teardown_release_herdr_locks >/dev/null 2>&1; then teardown_release_herdr_locks || true fi @@ -720,7 +752,13 @@ fi # This is the first cleanup authorization check. It is metadata-only and must # complete before fm-guard, a backend command, file removal, branch deletion, # worktree return, registry change, or process termination can run. -fm_backend_validate_task_endpoint "$META" "$ID" || exit 1 +fm_backend_validate_task_endpoint "$META" "$ID" allow-retired || exit 1 +WORKTREE_RETIRED=0 +if fm_worktree_retirement_receipt_present "$META" >/dev/null 2>&1; then + WORKTREE_RETIRED=1 +fi +# Set only once THIS run has taken the recorded path back from its provider. +TASK_WORKTREE_RELEASED=0 BACKEND=$FM_BACKEND_VALIDATED_BACKEND T=$FM_BACKEND_VALIDATED_TARGET WT=$(fm_meta_get "$META" worktree) @@ -1270,6 +1308,10 @@ if ! retry_wait_secs_is_valid "$TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS"; then fi # Compatibility alias used by the safety-check wait path and older call sites. STALE_WORKTREE_LOCK_RETRY_WAIT_SECS=$TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS +# Only the pre-return safety-check path reports this. A provider return that +# ends on a live lock is an ordinary failure: it has already retired the claim, +# so it must park like every other unconfirmed outcome rather than offer a code +# some caller could read as permission for a different destructive fallback. TEARDOWN_TREEHOUSE_LOCK_REFUSED=2 TEARDOWN_WORKTREE_SAFETY_LOCK_BLOCKED=3 TEARDOWN_PROCEVENT_RESTORE_FAILED=4 @@ -1335,13 +1377,21 @@ cleanup_stale_lock_for_safety_check() { # Return a worktree/home via `treehouse return --force`, tolerating a transient or # stale git index.lock left by a killed crew process. See the script header. -teardown_treehouse_return() { +# The wrapper proves ownership again at the provider boundary and clears the +# claim before Treehouse can recycle the slot. Any unconfirmed return parks the +# retirement for deliberate manual recovery; nothing restores automatically. +treehouse_return_once() { # + local dir=$1 cd_dir=$2 + (CDPATH='' cd -- "$cd_dir" && treehouse return --force "$dir") +} + +teardown_treehouse_return_raw() { #