Skip to content

mngr_lima: report limactl runtime failures as provider-unavailable instead of leaking ProcessError#2412

Merged
gnguralnick merged 11 commits into
mainfrom
gabriel/roaring-cobra
Jul 14, 2026
Merged

mngr_lima: report limactl runtime failures as provider-unavailable instead of leaking ProcessError#2412
gnguralnick merged 11 commits into
mainfrom
gabriel/roaring-cobra

Conversation

@gnguralnick

Copy link
Copy Markdown
Contributor

Problem

When limactl is installed and correctly versioned but an invocation fails at runtime (e.g. it crashes at startup), Lima discovery surfaced an opaque, unclassified error and left the workspace stuck offline. This is what a real user hit: on their Mac, limactl list --json panicked at package init (panic: user: unknown userid 501 — a transient macOS Directory Services / getpwuid fault), so every Lima discovery poll failed and their local workspace was unreachable with no actionable reason.

Root cause, in mngr_lima:

  • The limactl_* wrapper helpers called cg.run_process_to_completion(...) with the default is_checked_after=True, which raises a raw ProcessError on a non-zero exit — making the if result.returncode != 0: raise LimaCommandError(...) line below each call dead code.
  • ProcessError (a ConcurrencyGroupError) is neither LimaCommandError, OSError, nor ProviderUnavailableError, so it slipped past discover_hosts's handlers and the intended graceful degradation, bubbling up as a generic discovery error.
  • Separately, mngr only modeled Lima as "unavailable" for a missing (LimaNotInstalledError) or too-old (LimaVersionError) limactl — a binary that is present but crashes at runtime was unmodeled.

Fix

  1. Correctness: the limactl_* helpers (list, stop, delete, disk create, disk delete, show-ssh, shell, and get_lima_version) now pass is_checked_after=False, so a non-zero exit raises the intended domain LimaCommandError instead of a raw ProcessError. Side benefits: limactl_disk_delete's tolerate-absent-disk path and limactl_shell's documented non-zero return (inspected by its sole caller _wait_for_cloud_init) work as intended again.

  2. Semantics: new LimaCommandUnavailableError(ProviderUnavailableError). In discover_hosts, a present-but-crashing limactl's LimaCommandError/OSError is converted to that error and propagated, so the provider is reported unavailable (Provider 'lima' is not available: ...) with the standard retry/backoff — matching how other providers report an unreachable backend — instead of the old opaque error. A genuinely absent or too-old limactl still degrades gracefully to a local all-offline host view.

Tests

  • limactl_test.py: limactl_list raises LimaCommandError (not ProcessError) against a crashing fake limactl — the regression guard for the is_checked_after=False fix.
  • instance_test.py: discover_hosts raises LimaCommandUnavailableError/ProviderUnavailableError when a present, correctly-versioned limactl crashes on list; and still degrades to [] for a too-old limactl. Both use a shared install_fake_limactl helper (in testing.py) that writes a fake limactl and prepends its dir to PATH.
  • mngr_lima package: 134 passed, 0 failed (just test-quick); uv run ty check clean; ratchets clean. The full monorepo suite runs in CI.

Notes

  • Two commits: the fix, then a DRY consolidation of the fake-limactl test helper into testing.py (from the autofix gate).
  • This is the mngr_lima half. The complementary minds-side improvement — detect the limactl panic signature, retry discovery a few times before flipping to host_offline, and show an actionable message instead of a bare "offline" — is a separate follow-up, and upstream Lima's fail-hard Must(user.Current()) package init is the deeper cause.

Gabriel Guralnick and others added 3 commits July 9, 2026 12:09
…Error

The limactl wrapper helpers (list, stop, delete, disk create, disk delete,
show-ssh, shell, --version) called run_process_to_completion with the default
is_checked_after=True, so a non-zero exit raised a raw ProcessError and the
manual `if returncode != 0: raise LimaCommandError` below it was dead code.
Because ProcessError is neither LimaCommandError nor OSError, it slipped past
callers -- notably discover_hosts -- and surfaced as an opaque, unclassified
discovery error (this is how a limactl startup crash, e.g. the macOS getpwuid
"unknown userid" init panic, left a Lima workspace stuck offline with no clear
reason).

Two fixes:
- Correctness: run those helpers with is_checked_after=False so they raise the
  intended LimaCommandError on non-zero exit (get_lima_version now checks the
  return code explicitly too). limactl_disk_delete's tolerate-absent-disk path
  is reachable again as a result.
- Semantics: when limactl is installed and correctly versioned but an
  invocation fails at runtime, Lima discovery now raises
  LimaCommandUnavailableError (a ProviderUnavailableError), so the provider is
  reported unavailable with the standard message + retry/backoff, matching how
  other providers report an unreachable backend. A genuinely absent or too-old
  limactl still degrades gracefully to a local all-offline host view.

Co-authored-by: Sculptor <sculptor@imbue.com>
Problem: instance_test.py and limactl_test.py each defined an identical
_write_fake_limactl helper (duplicated test setup), and the shared PATH-prepend
monkeypatch line was repeated three times. Repo convention puts shared non-fixture
test helpers in testing.py. The duplicated helper's docstring also claimed it
prepends the directory to PATH, which it did not.

Fix: add install_fake_limactl to mngr_lima/testing.py that writes the executable
fake limactl and prepends its directory to PATH (making the docstring accurate),
import it in both test files, and drop the duplicated helpers and setenv lines.
The test-helper consolidation left `ProviderUnavailableError` imported but
unused (the crash test asserts on the more specific `LimaCommandUnavailableError`,
which is itself a `ProviderUnavailableError`), tripping the root `test_no_ruff_errors`
meta-check (F401) in CI. Per-package `test-quick` and `ty` don't run that ruff gate,
so it only surfaced on offload.

Co-authored-by: Sculptor <sculptor@imbue.com>
Gabriel Guralnick and others added 2 commits July 9, 2026 15:32
…+ changelog

Address the architecture review's two cosmetic notes:
- LimaCommandUnavailableError previously inherited the base ProviderUnavailableError
  help text ("Ensure the provider backend is running (e.g. start Docker)"), which
  names the wrong tool for a Lima crash. It now supplies Lima-appropriate
  user_help_text / short_remediation (retry; reinstall Lima if it persists).
- Changelog now notes the is_checked_after=False flip also repairs non-discovery
  paths (limactl_shell's non-zero return; the stop/delete/cleanup LimaCommandError
  catch sites), not just discovery.

Co-authored-by: Sculptor <sculptor@imbue.com>
Problem: the mngr_lima changelog's first sentence listed `shell` among the
limactl wrapper helpers that now "surface a non-zero exit as the intended
LimaCommandError", but limactl_shell returns its returncode to the caller and
never raises LimaCommandError. This contradicted the same paragraph's later
sentence noting shell "can once again return a non-zero exit code ... (rather
than raising)".
Fix: removed `shell` from the first sentence's helper list; shell's distinct
return-non-zero behavior remains accurately described by the following sentence.
@gnguralnick gnguralnick marked this pull request as ready for review July 10, 2026 00:14
Comment on lines 232 to 234
result = cg.run_process_to_completion(cmd, timeout=timeout, is_checked_after=False)
if result.returncode != 0:
raise LimaCommandError("stop", result.returncode, result.stderr)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems weird and bad -- why not:

try:
    result = cg.run_process_to_completion(cmd, timeout=timeout, is_checked_after=False)
except SomeTypeOfError as e:
    raise LimaCommandError("stop", e.returncode, e.stderr, e.stdout) from e

Then all of the information gets preserved? And we then no longer can forget to check the exit code

It's weird to be disabling the check, then checking it?

cmd = ["limactl", "shell", instance_name, "--", "sh", "-c", command]
result = cg.run_process_to_completion(cmd, timeout=timeout)
result = cg.run_process_to_completion(cmd, timeout=timeout, is_checked_after=False)
return result.returncode, result.stdout, result.stderr

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't check this one... what callers exist?

can they properly handle this raising for a non-zero exit?

it'd be nice to make it all consistent so that we cannot have silent failures

@joshalbrecht

Copy link
Copy Markdown
Contributor

the overall work here seems good, I just have the above stylistic complaints

@joshalbrecht

Copy link
Copy Markdown
Contributor

lgtm after they are addressed

# Conflicts:
#	libs/mngr_lima/imbue/mngr_lima/limactl_test.py
Gabriel Guralnick and others added 2 commits July 13, 2026 10:49
…actl_shell raises

Addresses review feedback on the disable-check-then-recheck pattern: every
limactl helper now runs through _run_limactl, which lets the ConcurrencyGroup's
default checked run raise and translates the resulting ProcessError into
LimaCommandError (preserving stdout too). This removes the per-call-site
is_checked_after=False + manual return-code check, so a new helper cannot
reintroduce the ProcessError leak. limactl_shell now raises on failure like the
other helpers instead of silently returning a non-zero exit code that its
cloud-init-wait caller ignored.

Co-authored-by: Sculptor <sculptor@imbue.com>
@gnguralnick gnguralnick merged commit 789fb11 into main Jul 14, 2026
14 checks passed
@gnguralnick gnguralnick deleted the gabriel/roaring-cobra branch July 14, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants