Skip to content

Escalate cold-loaded workspaces on an errored provider to the recovery page#2430

Draft
gnguralnick wants to merge 5 commits into
mainfrom
gabriel/escalate-to-recovery
Draft

Escalate cold-loaded workspaces on an errored provider to the recovery page#2430
gnguralnick wants to merge 5 commits into
mainfrom
gabriel/escalate-to-recovery

Conversation

@gnguralnick

Copy link
Copy Markdown
Contributor

Problem

Opening a minds workspace that never finished loading this session, whose compute provider is unreachable (e.g. lima whose limactl crashes so discovery errors), left the app on the "Loading workspace" proxy loader forever. It never escalated to the recovery page ("Can't connect to " with Retry + auto-recovery).

Root cause: a routeless agent makes the forward emit system_interface_backend_failure envelopes with reason UNRESOLVED. should_enroll_suspect_for_backend_failure returned False for UNRESOLVED outright, on the assumption it is always a self-resolving warm-up (or a gone agent). That holds once a workspace has loaded (a present-but-unreachable outage surfaces as CONNECT_ERROR/5xx, which enrolls). But on a cold load that never resolved, the forward emits UNRESOLVED forever → never enrolled → probe loop never runs it → never STUCK → chrome never navigates to recovery.

Fix

Disambiguate UNRESOLVED with the discovery provider-error signal — the exact thing that separates a doomed cold load (provider errored, route will never resolve) from a healthy warm-up (provider fine, route still coming):

  • should_enroll_suspect_for_backend_failure takes a new is_provider_errored: bool; for UNRESOLVED it returns that flag (all other reasons unchanged).
  • New MngrCliBackendResolver.is_agent_provider_errored(agent_id) (interface default False): a single lock-guarded join of the agent's provider_name (from the live discovery snapshot) against the per-provider error map.
  • The run.py backend-failure callback supplies the flag.

Reuses the existing pipeline: enroll → probe loop confirms the forward's 5xx → STUCK after the threshold → chrome redirects to recovery → _classify_dispatch_tier renders BACKEND_UNREACHABLE from the same provider-error map → healthy-poll auto-returns the user when the provider recovers. Self-debouncing: a probe 200 within the stuck threshold → HEALTHY, so a transient provider blip never yields a false STUCK. mngr_forward is untouched (policy stays in the minds consumer).

Scope note

Escalation requires the failing agent to be present in the live discovery snapshot (so its provider is attributable). It covers "provider enumerated the workspace this session, then errored." It does not cover an agent never enumerated this session (provider errored on its first poll, or opened via window-restore while offline) — the provider is then unattributable, the same limitation the recovery page already has. Covering that needs provider attribution from the persisted last-good topology + recovery-classification changes; left as a follow-up.

Tests

  • Extended should_enroll_suspect_for_backend_failure unit test: UNRESOLVED + provider-errored → True, UNRESOLVED + not-errored → False, non-UNRESOLVED unchanged (independent of the flag).
  • New is_agent_provider_errored resolver tests: errored provider → True, healthy provider → False, unknown agent → False, static-resolver default → False.
  • Tracker-level self-heal (enroll → probe success before threshold → HEALTHY, no STUCK) is already covered by existing tracker tests.

Local runs (green): ruff check . and ty check clean; system_interface_health_test.py + backend_resolver_test.py = 123 passed; test_ratchets.py = 57 passed; forward_cli_test.py + workspace_recovery_test.py = 50 passed. Full offload suite runs in CI.

Gabriel Guralnick and others added 2 commits July 10, 2026 11:35
…page

A workspace opened for the first time this session whose compute provider is
unreachable emits UNRESOLVED backend-failure envelopes forever (no route ever
resolves). minds ignored UNRESOLVED outright, so such a workspace was never
enrolled as a probe suspect, never went STUCK, and never navigated to the
recovery page -- it sat on the "Loading workspace" spinner indefinitely.

Gate UNRESOLVED enrollment on whether the agent's provider is currently errored
in discovery: a doomed cold load (provider errored) now enrolls and escalates to
the "Can't connect to <provider>" recovery screen, while a healthy but slow
warm-up (provider not errored) is still left alone so it isn't misreported as
stuck. The escalation reuses the existing probe loop and is self-debouncing: if
the provider error was a blip and the route resolves within the stuck threshold,
the probe succeeds and no STUCK fires.

Co-authored-by: Sculptor <sculptor@imbue.com>
The escalation fires when discovery reports the workspace's provider as errored;
reword so it does not imply coverage of a workspace never enumerated by
discovery this session (still-open follow-up).

Co-authored-by: Sculptor <sculptor@imbue.com>
The is_agent_provider_errored generator expression fits on one line; ruff
format (the root test_no_ruff_errors meta-ratchet, not run by test-quick)
requires it collapsed. No behavior change.

Co-authored-by: Sculptor <sculptor@imbue.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant