fix(daemon): restore instance-lock recovery from a wedged daemon - #165
Closed
neumie wants to merge 1 commit into
Closed
fix(daemon): restore instance-lock recovery from a wedged daemon#165neumie wants to merge 1 commit into
neumie wants to merge 1 commit into
Conversation
Starting Okena while a UI-owned daemon held the instance lock without
advertising ended in a hard lockout the user had to clear by killing the
process by hand:
WARN Instance lock owner pid 95630 neither re-advertised nor exited
within 8s; reaping it
WARN Refusing to reap pid 95630: process is not a verified UI-owned
Okena daemon
The guard verified ownership by looking for `--ui-owned` in the owner's
argv, read via sysinfo after `System::refresh_processes`. That call
refreshes only a minimal set of fields and leaves `cmd()` empty — `name`
and `exe` are populated, argv is not. The flag was therefore never found,
no process ever verified, and the whole reap path was unreachable. Ask
for the command line explicitly.
That made the post-reap check reachable for the first time, and it was
wrong: it failed on "is the lock held by anyone", not "is it still held
by the process we just killed". The likeliest reason a daemon wedges
un-advertised is a restart in flight — and in that state its replacement
is already running, blocked on `--await-pid <old>`, waiting for exactly
this kill to acquire the lock. So a successful recovery reported itself
as the same lockout, now naming a pid that no longer existed. Wait for
the reaped pid to actually exit, fail only if that pid still holds the
lock, and give a successor its own window to advertise.
Process inspection moves behind `read_process_identity` + a pure
`is_ui_owned_okena` predicate so both are testable; the unix test spawns
a child and asserts its argv is readable, which fails on the old call
with `args: []`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Aq3X2s6ftafXzmqaN3A8
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Starting Okena while a UI-owned daemon held the instance lock without advertising ended in a hard lockout — the user had to kill the process by hand:
The process genuinely was Okena.
Root cause 1 — the guard could never succeed
kill_pid_if_ui_owned_okenaverified ownership by looking for--ui-ownedin the owner's argv, read via sysinfo:System::refresh_processesrefreshes only a minimal set of fields and leavescmd()empty —nameandexeare populated, argv is not. Verified with a probe against sysinfo 0.33 on macOS:So
ui_ownedwas alwaysfalse, the function could never returntrue, and the entire reap path was unreachable. Every un-advertised lock holder became a lockout. The command line has to be requested explicitly.Root cause 2 — exposed by fixing the first
Fixing that made the post-reap check reachable for the first time, and it was wrong:
It failed on "is the lock held by anyone", not "is it still held by the process we just killed". The likeliest reason a daemon wedges un-advertised is a restart in flight — and in that state its replacement is already running, blocked in
wait_for_pid_exit(old_pid, 10s), waiting for exactly this kill to acquire the lock. It takes over well inside the 200ms window, so a successful recovery reported itself as the same lockout, now naming a pid that no longer existed.Now: wait for the reaped pid to actually exit (that is what releases the flock) instead of sleeping a fixed guess; fail only if that pid still holds the lock; and give a successor its own settle window. Bounded to 2 reaps so a pathological chain of wedged daemons still terminates rather than reaping forever.
Structure
Process inspection moves behind
read_process_identity+ a pureis_ui_owned_okenapredicate, so both are testable.UI_OWNED_FLAGreplaces the bare string literal at the spawn site and the check.Tests
args: []— I verified by temporarily reverting the call.Full build clean, clippy clean workspace-wide, 1367 tests pass.
Notes
Stacked on #157 — base is
refactorx/full-headless.Follow-up, tracked separately:
is_process_aliveis a barekill(pid, 0), which reports zombies as alive. The GUI holds its daemon'sChildwithout ever reaping it, so on every clean quitwait_for_pid_exitburns its full 3s on a corpse and then logs a false "did not exit gracefully"; it also makes a crashed daemon look live, sending startup down the attach branch into a hard error instead of respawning.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y4Aq3X2s6ftafXzmqaN3A8