Skip to content

fix(cli): keep default MPS runtime within socket limits - #2092

Merged
TATP-233 merged 1 commit into
mainfrom
fix/issue-2091-cuda-mps-default-path
Oct 8, 2026
Merged

TATP-233 merged 1 commit into
mainfrom
fix/issue-2091-cuda-mps-default-path

Conversation

@TATP-233

@TATP-233 TATP-233 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Driving issue

Closes #2091.

Problem

The zero-argument uni-cumps workflow could strand itself after a failed or exited NVIDIA control daemon:

  • the old default pipe path under a normal home directory could exceed NVIDIA's effective Unix-socket boundary;
  • NVIDIA returns exit code 1 with no stderr when the control path is too long;
  • failed launches leave control_lock and a pipe/log FIFO;
  • exited control daemons leave control and control_privileged sockets;
  • the next start then reported “Refusing to attach to an existing control path,” while env/stop had no daemon record to use.

Measured on CUDA MPS 13.0 / driver 580:

  • 95-byte control paths succeed;
  • 96-byte and longer paths fail with no stderr.

Fix

Default runtime paths are now short and private:

/tmp/uni-cumps/<daemon-name>/pipe
/tmp/uni-cumps/<daemon-name>/log

All default runtime directories are created with mode 0700 and owned by the invoking user.

The default daemon name now uses the complete canonical GPU UUID:

gpu-<full-uuid>

rather than a shortened prefix, avoiding daemon-name collisions.

Explicit --pipe-dir values are checked before launch. A resulting control path over 95 bytes fails with an actionable message recommending a short path such as /tmp/<short-name>/pipe.

Failed-launch recovery is narrow:

  • known inert artifacts (control_lock and the pipe/log FIFO) are removed after a failed control launch;
  • in the UniLab-owned default runtime only, orphan control / control_privileged sockets are removed when no PID file exists;
  • explicit user-supplied directories are never modified;
  • live sockets with daemon PID evidence are never removed.

A nonzero control exit now appends the control-log tail, making silent NVIDIA failures diagnosable.

Final workflow

Verified on the available Linux/NVIDIA host:

uv run uni-cumps start
eval "$(uv run uni-cumps env)"
uv run uni-cumps doctor
uv run uni-cumps stop

All four completed successfully; doctor exited 0. A second start while live correctly reported the existing live daemon. Restarting after a dead control process also succeeded after safe orphan cleanup.

The immediate start → doctor pair was additionally repeated 20 times in one shell; all 20 doctors returned valid=True and exit 0. The orphan check now validates PID liveness and the /proc/<pid>/stat process name rather than treating the mere presence of a stale PID file as live ownership.

Validation

Commands run on final local head a1084ce8:

make check
make test-all

Results:

  • ruff/mypy/pyright passed;
  • non-slow coverage run: 1845 passed, 43 skipped, 488 deselected;
  • benchmark smoke: module-mode 34/34, script-mode 35/35.

Focused checks:

uv run pytest tests/training/test_cuda_mps_cli.py tests/scripts/test_check_docs.py -q

Result: 57 passed.

New coverage includes:

  • full-UUID default names;
  • explicit socket-path length rejection;
  • removal of inert failed-launch artifacts while preserving unrelated files and live sockets;
  • orphan default-runtime socket cleanup, including stale PID files and procfs process-name matching;
  • control-log diagnostics on nonzero exit.

Scope boundaries

  • No trainer lease/selection behavior is introduced.
  • No multi-GPU, task-per-GPU, MIG, SM-percentage, memory-limit, affinity, or multi-node support is claimed.
  • Explicit pipe directories are never cleaned by UniLab.

@TATP-233
TATP-233 force-pushed the fix/issue-2091-cuda-mps-default-path branch from 7748cb9 to a1084ce Compare October 8, 2026 14:20
@TATP-233
TATP-233 merged commit 902c040 into main Oct 8, 2026
8 checks passed
@TATP-233
TATP-233 deleted the fix/issue-2091-cuda-mps-default-path branch October 8, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(cli): default uni-cumps daemon path exceeds Linux MPS socket limit

1 participant