Skip to content

fix: iterate a snapshot of in_progress when reporting a dead runner - #2311

Closed
mlpy0 wants to merge 1 commit into
exo-explore:mainfrom
mlpy0:fix-supervisor-in-progress-iteration
Closed

mlpy0 wants to merge 1 commit into
exo-explore:mainfrom
mlpy0:fix-supervisor-in-progress-iteration

Conversation

@mlpy0

@mlpy0 mlpy0 commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Motivation

If a runner dies while tasks are in flight (in my case an MLX runner aborting with signal 6 on a Metal out-of-memory error), RunnerSupervisor._check_runner sends an ErrorChunk for each task in in_progress. Each send is awaited, and in the meantime _forward_events can pop a task whose TaskStatusUpdated Complete just arrived. The loop then raises RuntimeError: dictionary changed size during iteration, the task group fails, and the whole process exits instead of just reporting the failed runner.

Related to #2176. #2177 also snapshots the dict, but would still send an ErrorChunk for a task that already completed.

Changes

_check_runner iterates a snapshot of in_progress and skips any task that completed while an earlier chunk was being sent.

Why It Works

Pops and inserts during the awaited sends can no longer invalidate the iteration, and the membership check avoids emitting an error for a command whose completion was already forwarded.

Test Plan

Manual Testing

Hit the crash after a runner was killed by a Metal OOM abort with requests in flight; the process exited with "EXO terminated due to unhandled exception".

Automated Testing

  • Added test_check_runner_survives_task_completing_during_error_fanout: two text generation tasks in flight, the second completes during the first one's error send. It fails on main with the RuntimeError and passes with this change.

@mlpy0 mlpy0 closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant