Skip to content
Closed
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ data/
scratchpad*
.no-mistakes/
.lavish/
.squish/
.fm-secondmate-home
.fm-secondmate-parent
.DS_Store
Expand Down
7 changes: 7 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ state/ runtime records and signals; gitignored
.<id>.open-decisions-cursor per-task byte cursor and folded open-decision set bounding the OPEN DECISIONS scan's cost to new status-log appends; written only by fm-classify-lib.sh's status_open_decisions_incremental, removed by teardown, safe to delete (forces one full re-fold)
.status-presentation-cursor .status-presentation-lock fleet-wide per-task status identity plus independent annotation and outcome-backstop byte offsets, with a serialization lock preventing already-presented lines from replaying while preserving delayed signal annotations; owned by fm-classify-lib.sh, with each task's row retired by teardown
.afk durable away-mode flag; present = sub-supervisor may inject escalations (set by /afk, cleared on user return)
.resgate-cap-work .resgate-cap-home durable manual resource-cap override markers forcing the 50% weekly-schedule cap immediately on the work or home PC regardless of the clock window; set and cleared by firstmate on the section 8 "Kappung" trigger (docs/configuration.md "Fleet resource governance")
.watch.lock .wake-queue.lock watcher singleton and queue serialization locks
.claude-autoarm.lock .claude-autoarm-epoch .claude-autoarm-failure-notified .claude-autoarm-failure-alarmed .turnend-claude-blocks .turnend-claude-blocks.lock Claude Stop auto-arm single-flight, epoch, failure-episode, attended-alarm, guard-budget, and budget-lock records; never touch
.cursor-park-owner .cursor-park-owner.lock .turnend-cursor-blocks Cursor stop-hook owner record, publication and commit lock, and bounded repair-nag budget; never touch
Expand Down Expand Up @@ -443,6 +444,12 @@ The skill owns the daemon procedure; these safety facts remain inline:
- Away mode never expands approval authority for merges, ask-user findings, destructive actions, irreversible actions, or security-sensitive choices.
- Bias ambiguous input toward exit because a present captain takes precedence.

### Resource-cap trigger

Arm `state/.resgate-cap-<role>` the moment the captain says "Kappung" in chat, matched case-insensitively: bare "Kappung" arms both roles, `Kappung <hostname or role>` arms only that one.
"Kappung auf" releases the same way, bare for both markers and `Kappung auf <hostname or role>` for that one alone.
docs/configuration.md "Fleet resource governance" owns the marker paths, the release-form-first and longest-hostname-first matching precedence, and the rest of the mechanism.

### Stuck-worker trigger

Load `stuck-crewmate-recovery` after a stale wake, looping or confused pane, answered-by-brief question, unresponsive worker, or failed steer.
Expand Down
557 changes: 557 additions & 0 deletions bin/fm-resgate-lib.sh

Large diffs are not rendered by default.

134 changes: 134 additions & 0 deletions bin/fm-resgate.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
#!/usr/bin/env bash
# Fleet resource governance for the captain's work PC and home PC: weekly
# clock-window percentage caps, a manual override, and GPU exclusivity
# between Qwen and the JARVIS voice worker on the home PC.
# Usage:
# fm-resgate.sh schedule <work|home>
# fm-resgate.sh cap <work|home>
# fm-resgate.sh override set <work|home|both>
# fm-resgate.sh override clear <work|home|both>
# fm-resgate.sh override status <work|home|both>
# fm-resgate.sh gpu status
# fm-resgate.sh gpu allow <qwen|voice>
#
# `schedule` prints the clock-window verdict alone (uncapped/capped/blocked),
# ignoring any manual override. `cap` prints the EFFECTIVE percentage (100,
# 50, or 0), folding in an active override. `override set/clear` arm or
# release the durable state/.resgate-cap-<role> marker documented in
# docs/configuration.md "Fleet resource governance"; `both` touches both
# roles. `gpu status` prints the freshly probed home-PC GPU owner; `gpu allow`
# exits 0 when <workload> may start or keep running on that GPU right now, 1
# otherwise, printing the reason either way. The script header of
# bin/fm-resgate-lib.sh owns the schedule windows, the fail-closed rules, and
# the GPU-detection signals.
set -eu

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}"
FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}"
STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}"

# shellcheck source=bin/fm-resgate-lib.sh
. "$SCRIPT_DIR/fm-resgate-lib.sh"

usage() {
sed -n '2,23{s/^# \{0,1\}//;p;}' "$0"
}

die() {
printf 'error: resgate: %s\n' "$1" >&2
exit 1
}

roles_for() { # <work|home|both>
case "$1" in
work | home) printf '%s\n' "$1" ;;
both) printf 'work\nhome\n' ;;
*) die "role must be work, home, or both: $1" ;;
esac
}

cmd_schedule() {
local role=${1:-}
fm_resgate_role_ok "$role" || die "role must be work or home: $role"
fm_resgate_schedule_state "$role"
printf 'state=%s\n' "$FM_RESGATE_SCHEDULE_STATE"
printf 'reason=%s\n' "$FM_RESGATE_SCHEDULE_REASON"
}

cmd_cap() {
local role=${1:-}
fm_resgate_role_ok "$role" || die "role must be work or home: $role"
fm_resgate_capacity_pct "$STATE" "$role"
printf 'pct=%s\n' "$FM_RESGATE_PCT"
printf 'state=%s\n' "$FM_RESGATE_STATE"
printf 'reason=%s\n' "$FM_RESGATE_REASON"
}

cmd_override() {
local action=${1:-} role_arg=${2:-} role roles rc=0
case "$action" in set | clear | status) ;; *)
die "override subcommand must be set, clear, or status: $action" ;;
esac
# Validated (and, on a bad role, exited) here, in this shell - never inside
# a process-substitution subshell, whose exit status the caller cannot see.
roles=$(roles_for "$role_arg")
for role in $roles; do
Comment thread
greptile-apps[bot] marked this conversation as resolved.
case "$action" in
set)
fm_resgate_override_set "$STATE" "$role" Kappung \
|| { printf 'error: resgate: could not arm override for %s\n' "$role" >&2; rc=1; }
;;
clear)
fm_resgate_override_clear "$STATE" "$role" \
|| { printf 'error: resgate: could not clear override for %s\n' "$role" >&2; rc=1; }
;;
status)
if fm_resgate_override_active "$STATE" "$role"; then
printf '%s=armed\n' "$role"
else
printf '%s=clear\n' "$role"
fi
;;
esac
done
return "$rc"
}

cmd_gpu() {
local action=${1:-} want=${2:-}
case "$action" in
status)
fm_resgate_home_gpu_owner || true
printf 'owner=%s\n' "$FM_RESGATE_GPU_OWNER"
printf 'voice_port=%s\n' "${FM_RESGATE_GPU_VOICE:-unknown}"
printf 'process(%s)=%s\n' "$FM_RESGATE_GPU_PROCESS_NAME" "${FM_RESGATE_GPU_PROCESS:-unknown}"
printf 'gpu_used_mb=%s\n' "${FM_RESGATE_GPU_USED_MB:-unknown}"
printf 'gpu_busy_threshold_mb=%s\n' "$(fm_resgate_gpu_busy_mb)"
;;
allow)
case "$want" in qwen | voice) ;; *) die "gpu allow needs qwen or voice: $want" ;; esac
if fm_resgate_gpu_available_for "$want"; then
printf 'allow=yes\n'
printf 'reason=%s\n' "$FM_RESGATE_GPU_REASON"
else
printf 'allow=no\n'
printf 'reason=%s\n' "$FM_RESGATE_GPU_REASON"
return 1
fi
;;
*) die "gpu subcommand must be status or allow: $action" ;;
esac
}

[ "$#" -ge 1 ] || { usage >&2; exit 2; }
CMD=$1
shift
case "$CMD" in
schedule) cmd_schedule "$@" ;;
cap) cmd_cap "$@" ;;
override) cmd_override "$@" ;;
gpu) cmd_gpu "$@" ;;
-h | --help) usage ;;
*) die "unknown command: $CMD" ;;
esac
18 changes: 9 additions & 9 deletions bin/fm-test-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -657,7 +657,7 @@ portable_serial_unhinted() {
tmp=$(mktemp -d "${TMPDIR:-/tmp}/fm-test-unhinted.XXXXXX") || return 1
portable_serial_weight_hints | awk 'NF { print $1 }' | LC_ALL=C sort -u >"$tmp/hinted"
list_portable_serial | LC_ALL=C sort -u >"$tmp/serial"
comm -23 "$tmp/serial" "$tmp/hinted"
LC_ALL=C comm -23 "$tmp/serial" "$tmp/hinted"
rm -rf "$tmp"
}

Expand Down Expand Up @@ -805,8 +805,8 @@ run_coverage_guard() {
return 1
fi
cat "$tmp/s1" "$tmp/s2" | LC_ALL=C sort -u >"$tmp/shards_union"
missing=$(comm -23 "$tmp/proven" "$tmp/shards_union" || true)
extra=$(comm -13 "$tmp/proven" "$tmp/shards_union" || true)
missing=$(LC_ALL=C comm -23 "$tmp/proven" "$tmp/shards_union" || true)
extra=$(LC_ALL=C comm -13 "$tmp/proven" "$tmp/shards_union" || true)
if [ -n "$missing" ] || [ -n "$extra" ]; then
log "coverage guard: portable shards must equal the proven-isolated set"
[ -z "$missing" ] || { log "missing from shards:"; printf '%s\n' "$missing" >&2; }
Expand Down Expand Up @@ -850,8 +850,8 @@ run_coverage_guard() {
return 1
fi
LC_ALL=C sort -u "$tmp/serial_shards_raw" >"$tmp/serial_shards"
missing=$(comm -23 "$tmp/serial" "$tmp/serial_shards" || true)
extra=$(comm -13 "$tmp/serial" "$tmp/serial_shards" || true)
missing=$(LC_ALL=C comm -23 "$tmp/serial" "$tmp/serial_shards" || true)
extra=$(LC_ALL=C comm -13 "$tmp/serial" "$tmp/serial_shards" || true)
if [ -n "$missing" ] || [ -n "$extra" ]; then
log "coverage guard: portable serial shards must equal the portable serial lane"
[ -z "$missing" ] || { log "missing from serial shards:"; printf '%s\n' "$missing" >&2; }
Expand All @@ -863,7 +863,7 @@ run_coverage_guard() {
for pair in "shards_union:serial" "shards_union:herdr" "serial:herdr"; do
a=${pair%%:*}
b=${pair#*:}
comm -12 "$tmp/$a" "$tmp/$b" >"$tmp/overlap"
LC_ALL=C comm -12 "$tmp/$a" "$tmp/$b" >"$tmp/overlap"
if [ -s "$tmp/overlap" ]; then
log "coverage guard: overlap between $a and $b:"
cat "$tmp/overlap" >&2
Expand All @@ -881,8 +881,8 @@ run_coverage_guard() {
return 1
fi
LC_ALL=C sort -u "$tmp/union_raw" >"$tmp/union"
missing=$(comm -23 "$tmp/all" "$tmp/union" || true)
extra=$(comm -13 "$tmp/all" "$tmp/union" || true)
missing=$(LC_ALL=C comm -23 "$tmp/all" "$tmp/union" || true)
extra=$(LC_ALL=C comm -13 "$tmp/all" "$tmp/union" || true)
if [ -n "$missing" ] || [ -n "$extra" ]; then
log "coverage guard: union of portable shards + portable serial + Herdr must equal tests/*.test.sh"
[ -z "$missing" ] || { log "missing from union:"; printf '%s\n' "$missing" >&2; }
Expand Down Expand Up @@ -912,7 +912,7 @@ run_coverage_guard() {
"$ROOT/bin/fm-test-isolation-proof.sh" --list | LC_ALL=C sort -u >"$tmp/proof_list"
if ! cmp -s "$tmp/proven" "$tmp/proof_list"; then
log "coverage guard: embedded proven-isolated set diverges from bin/fm-test-isolation-proof.sh --list"
comm -3 "$tmp/proven" "$tmp/proof_list" >&2 || true
LC_ALL=C comm -3 "$tmp/proven" "$tmp/proof_list" >&2 || true
rm -rf "$tmp"
return 1
fi
Expand Down
33 changes: 33 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,39 @@ The bound is required rather than cosmetic because churn and pane staleness read
The flag is a home-local supervision-noise preference and is not inherited by secondmate homes, which run their own crew mix.
[`architecture.md`](architecture.md) owns the triage contract and `bin/fm-watch.sh`'s `signal_turnend_panes_churned` owns the exact evidence and fail-closed boundaries.

## Fleet resource governance (state/.resgate-cap-work, state/.resgate-cap-home)

`bin/fm-resgate.sh` (CLI) and `bin/fm-resgate-lib.sh` (the primitives it wraps) protect the captain's own use of the work PC (`Valentino-Arbeit`) and home PC (`Valentino`, RTX 4080 Super) from the fleet, and keep Qwen and the JARVIS voice worker off the home PC's GPU at the same time.
The exact schedule windows, the percentage-cap arithmetic, the fail-closed rules, and the GPU-detection signals are owned by `bin/fm-resgate-lib.sh`'s header; this section covers only where the surface lives and how the captain controls it.

Two fixed roles, `work` and `home`, evaluated once against THIS host's own clock forced into `Europe/Berlin` - never a remote host's clock, which could be wrong or drifted and must never be able to loosen or defeat the gate:

```sh
bin/fm-resgate.sh schedule work # clock-window verdict alone (uncapped/capped/blocked)
bin/fm-resgate.sh cap home # effective percentage (100/50/0), folding in a manual override
bin/fm-resgate.sh gpu status # freshly probed home-PC GPU owner: none/qwen/voice/unknown
bin/fm-resgate.sh gpu allow qwen # exit 0/1: may Qwen start or keep running on the GPU right now
```

Manual override: `state/.resgate-cap-<work|home>` is a plain presence-based marker, written atomically like `state/.afk`.
While it exists, that role reads capped (50%) immediately, regardless of which window the clock lands in - but the marker can only ever tighten the gate, never loosen it: an unreadable clock stays `blocked` at 0% even with the marker armed, because a control whose purpose is to restrict must not hand out capacity no measurement supports.
Firstmate matches the captain's chat wording case-insensitively, and checks the release form FIRST: "Kappung auf" always means release, never an arming instruction, because bare `Kappung` is a substring of it and would otherwise be misread as "arm both".
Bare "Kappung auf" releases both markers, and "Kappung auf <hostname or role>" releases only that one, resolved by the same hostname matching as the arming form below.
Only when the message is not the release form does firstmate arm: bare `Kappung` arms both roles, `Kappung <hostname or role>` arms only that one, matching `Valentino-Arbeit`/`work`/`Arbeits-PC` to `work` and `Valentino`/`home`/`Heim-PC` to `home` (`auf` is not a hostname and never resolves to a role).
The hostname tokens are matched longest-first for the same substring reason: `Valentino` is a substring of `Valentino-Arbeit`, so `Valentino-Arbeit` and `Arbeits-PC` must both be ruled out before bare `Valentino` is considered, or "Kappung Valentino-Arbeit" caps the home PC and leaves the work PC - the machine the captain actually asked to protect - uncapped through his working hours.
This is a plain state file firstmate touches directly (`bin/fm-resgate.sh override set|clear <work|home|both>`, or an equivalent direct write following `fm_resgate_override_set`/`fm_resgate_override_clear`'s header contract in `bin/fm-resgate-lib.sh`), not a skill, and not wired into spawn or dispatch plumbing beyond this marker.
Arming and releasing can both fail (an unwritable `state/`), and both report it: the CLI prints `could not arm/clear override for <role>` and exits non-zero, so a release that did not actually happen is never mistaken for a lifted cap.

Fail-closed discipline: every measurement this surface cannot read - the authoritative clock, an SSH probe, the voice port, the GPU reading - yields the most restrictive answer, never a guess.
"Cannot read" includes a clock that answers confidently with the wrong zone: `date` does not fail when `Europe/Berlin` is unresolvable (missing tzdata on a slim image), it silently falls back to UTC, so the read is accepted only when the zone abbreviation and UTC offset it returns are a matching Europe/Berlin pair, and is otherwise treated exactly like an unreadable clock.
A schedule read that cannot happen at all reports 0% (`blocked`), stricter than the ordinary 50% cap and stricter than an armed override, and a GPU reading that cannot be completed reports `unknown`, which refuses both Qwen and JARVIS voice rather than picking a side.
An out-of-range `FM_RESGATE_VOICE_PORT`, a non-numeric `FM_RESGATE_GPU_BUSY_MB`, or a `FM_RESGATE_SSH_TIMEOUT` that is not a positive integer is treated the same way: the GPU reading reports `unknown` instead of probing the wrong port, comparing against an unusable threshold, or running the probe with a bound of zero seconds, which disables the deadline rather than bounding it.

GPU exclusivity: JARVIS voice is detected by its gateway port (currently `7414`, see `data/learnings.md`), never by process name, because process-name detection has broken this fleet's integration before.
That port reading is authoritative and is decided first: a listening gateway means `owner=voice` and the Qwen signals are never consulted, because aggregate card memory cannot say whose memory it is - an idle-but-resident Qwen model plus the voice worker's own VRAM would otherwise misread as contention and refuse the genuinely-running voice worker the card it already holds.
Qwen is detected by a named-process check (`ollama`, the currently live identity) corroborated by aggregate GPU memory clearing a threshold, not by `nvidia-smi --query-compute-apps` per-process attribution: live-tested against the real home host, that query lists every ordinary desktop GPU context (window compositor, open browsers) with no per-process memory field left to filter the noise by, so it cannot isolate a genuine workload there.
Both signals for the process-plus-memory check, and the port check, come from one bounded SSH round trip to the home host.

## Gate defaults (.no-mistakes.yaml)

The tracked `.no-mistakes.yaml` sets `test.evidence.store_in_repo: true` and pins `commands.lint` to `bin/fm-lint.sh` so local lint matches CI.
Expand Down
2 changes: 2 additions & 0 deletions docs/scripts.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize
| `fm-remote-readiness-lib.sh` | Shared remote second-mate readiness gate: check and, when needed, repair then re-check through `fm-remote-doctor.sh` |
| [`fm-project-origin-lib.sh`](../bin/fm-project-origin-lib.sh) | Accepted origin-form owner shared by both remote provisioning boundaries |
| `fm-spawn.sh` | Spawn crewmates, scouts, `id=repo` batches, and secondmates on the resolved harness and runtime backend |
| `fm-resgate.sh` | CLI for the weekly clock-window resource cap, manual override marker, and home-PC GPU exclusivity |
| `fm-resgate-lib.sh` | Schedule-window, percentage-cap, override-marker, and GPU-exclusivity primitives fm-resgate.sh wraps |
| `fm-backend.sh` | Runtime-backend selection, meta helpers, selector resolution, and operation dispatch |
| `fm-backend-hometag-lib.sh` | Shared per-installation home-tag derivation for zellij tab and cmux workspace titles |
| `fm-composer-lib.sh` | Single fleet-wide owner of composer shapes, capability-aware screen classification, and verdicts |
Expand Down
Loading