Escalate cold-loaded workspaces on an errored provider to the recovery page#2430
Draft
gnguralnick wants to merge 5 commits into
Draft
Escalate cold-loaded workspaces on an errored provider to the recovery page#2430gnguralnick wants to merge 5 commits into
gnguralnick wants to merge 5 commits into
Conversation
…page A workspace opened for the first time this session whose compute provider is unreachable emits UNRESOLVED backend-failure envelopes forever (no route ever resolves). minds ignored UNRESOLVED outright, so such a workspace was never enrolled as a probe suspect, never went STUCK, and never navigated to the recovery page -- it sat on the "Loading workspace" spinner indefinitely. Gate UNRESOLVED enrollment on whether the agent's provider is currently errored in discovery: a doomed cold load (provider errored) now enrolls and escalates to the "Can't connect to <provider>" recovery screen, while a healthy but slow warm-up (provider not errored) is still left alone so it isn't misreported as stuck. The escalation reuses the existing probe loop and is self-debouncing: if the provider error was a blip and the route resolves within the stuck threshold, the probe succeeds and no STUCK fires. Co-authored-by: Sculptor <sculptor@imbue.com>
The escalation fires when discovery reports the workspace's provider as errored; reword so it does not imply coverage of a workspace never enumerated by discovery this session (still-open follow-up). Co-authored-by: Sculptor <sculptor@imbue.com>
The is_agent_provider_errored generator expression fits on one line; ruff format (the root test_no_ruff_errors meta-ratchet, not run by test-quick) requires it collapsed. No behavior change. Co-authored-by: Sculptor <sculptor@imbue.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening a minds workspace that never finished loading this session, whose compute provider is unreachable (e.g. lima whose
limactlcrashes so discovery errors), left the app on the "Loading workspace" proxy loader forever. It never escalated to the recovery page ("Can't connect to " with Retry + auto-recovery).Root cause: a routeless agent makes the forward emit
system_interface_backend_failureenvelopes with reasonUNRESOLVED.should_enroll_suspect_for_backend_failurereturnedFalseforUNRESOLVEDoutright, on the assumption it is always a self-resolving warm-up (or a gone agent). That holds once a workspace has loaded (a present-but-unreachable outage surfaces asCONNECT_ERROR/5xx, which enrolls). But on a cold load that never resolved, the forward emitsUNRESOLVEDforever → never enrolled → probe loop never runs it → never STUCK → chrome never navigates to recovery.Fix
Disambiguate
UNRESOLVEDwith the discovery provider-error signal — the exact thing that separates a doomed cold load (provider errored, route will never resolve) from a healthy warm-up (provider fine, route still coming):should_enroll_suspect_for_backend_failuretakes a newis_provider_errored: bool; forUNRESOLVEDit returns that flag (all other reasons unchanged).MngrCliBackendResolver.is_agent_provider_errored(agent_id)(interface defaultFalse): a single lock-guarded join of the agent'sprovider_name(from the live discovery snapshot) against the per-provider error map.run.pybackend-failure callback supplies the flag.Reuses the existing pipeline: enroll → probe loop confirms the forward's 5xx → STUCK after the threshold → chrome redirects to recovery →
_classify_dispatch_tierrenders BACKEND_UNREACHABLE from the same provider-error map → healthy-poll auto-returns the user when the provider recovers. Self-debouncing: a probe 200 within the stuck threshold → HEALTHY, so a transient provider blip never yields a false STUCK.mngr_forwardis untouched (policy stays in the minds consumer).Scope note
Escalation requires the failing agent to be present in the live discovery snapshot (so its provider is attributable). It covers "provider enumerated the workspace this session, then errored." It does not cover an agent never enumerated this session (provider errored on its first poll, or opened via window-restore while offline) — the provider is then unattributable, the same limitation the recovery page already has. Covering that needs provider attribution from the persisted last-good topology + recovery-classification changes; left as a follow-up.
Tests
should_enroll_suspect_for_backend_failureunit test: UNRESOLVED + provider-errored → True, UNRESOLVED + not-errored → False, non-UNRESOLVED unchanged (independent of the flag).is_agent_provider_erroredresolver tests: errored provider → True, healthy provider → False, unknown agent → False, static-resolver default → False.Local runs (green):
ruff check .andty checkclean;system_interface_health_test.py+backend_resolver_test.py= 123 passed;test_ratchets.py= 57 passed;forward_cli_test.py+workspace_recovery_test.py= 50 passed. Full offload suite runs in CI.