[jaredpalmer]

--device auto on a machine with no CUDA already asks its probe child to try MPS. If that child fails or times out, select launches a second child with the same mps argument. The CPU fallback then waits out two probe timeouts, 60 seconds with the default KEV_DEVICE_PROBE_TIMEOUT when the child hangs, and the warning names the failure twice. A timeout shorter than import torch does the same thing, because the child is killed before it can print anything. kev.train, kev.evaluate, kev.benchmark, and kev.experiment all call select. Explicit --device mps was already one child. Reported by @luc-pimentel in #255.

The child prints kev-probing auto once import torch has returned, before torch.cuda.is_available(). It prints kev-probing mps, cuda, or cpu again before the allocation, and flushes both lines. The parent starts a second child only when it sees auto or cuda. mps means that allocation is already the one that failed, so it is not started again. No kev-probing line means the child died during import, and another process would import the same torch.

The result line is still kev-device <device>. kev-probing does not start with that prefix, so the announcement cannot make a failed probe look successful. A ROCm banner is ignored unless a whole line is exactly one of these markers. TimeoutExpired.output stays bytes when text=True, because communicate() decodes the stream only after it returns. The parent decodes those bytes, then reads the lines. A signal or a Python exception comes back as text on CalledProcessError. The warning names the device the child reached, so a failed MPS attempt shows up once, as mps.

A hang inside torch.cuda.is_available() still tries MPS, because kev-probing auto is already on stdout. A hang inside import torch is the same work the next child would do, so that child is not started. There is no core file if torch crashes while loading, the probe timeout is still KEV_DEVICE_PROBE_TIMEOUT, and auto still tries the Apple GPU before CPU when the CUDA child got that far.

This does not skip the second child when a CUDA probe reports that this torch has no MPS backend. That child can only fail, and it adds a clause to the warning. It also does not read MPS availability in the parent, the process a driver crash must not kill, and it does not change the MLX serve path, which never runs this probe.

The kev-probing line before the allocation, and the rule that a child which already said mps is not probed again, are @gezerui888's, from #256. This branch keeps that commit. The import-timeout case is the one KEV_DEVICE_PROBE_TIMEOUT=0.3 hits, because the child dies during import, before that line exists. On this machine the torch import takes about 1.0s. Before the follow-up, select("auto", timeout=0.3) took 0.61s and the warning named both timeouts.

Test plan

• pytest tests/test_device.py tests/test_conventions.py: 100 passed. Covers a real child that sleeps inside import torch, a real child whose CUDA check sleeps, bytes timeouts with a ROCm banner, a banner line that only contains the marker text, and the signal and dtype-error stand-ins.
• select("auto", timeout=0.3) on this machine returned cpu in 0.31s with one warning, auto: ... timed out after 0.3 seconds, and no second mps clause. A healthy select("auto") returned cpu with no warning.
• CI unit list, before test_auto_names_cuda_and_mps_failures_once was added: 606 passed, 26 skipped. test_resume_is_bit_identical[1-gate0] and test_snapshots_are_checkpoints_kept_and_completed_on_resume[1] failed once and passed on rerun. Both train with --device cpu, which returns before a probe.

Fixes #255.