Skip to content

feat(cli): manage user-owned CUDA MPS daemons - #2085

Merged
TATP-233 merged 1 commit into
mainfrom
feat/issue-2079-cuda-mps-cli
Oct 8, 2026
Merged

TATP-233 merged 1 commit into
mainfrom
feat/issue-2079-cuda-mps-cli

Conversation

@TATP-233

@TATP-233 TATP-233 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Driving issue

Closes #2079.

Final UX

The MPS lifecycle is a short, flat command. On the supported single-GPU host, no arguments are required:

uv run uni-cumps start
eval "$(uv run uni-cumps env)"
uv run uni-cumps doctor
uv run uni-cumps stop

Read-only host inspection:

uv run uni-cumps status

Explicit overrides remain for multi-GPU selection, stable labels, and service-owned paths:

uv run uni-cumps doctor --gpus <gpu-index-or-uuid>
uv run uni-cumps start \
  --gpus <gpu-index-or-uuid> \
  --name <name> \
  --pipe-dir /absolute/path/mps/pipe \
  --log-dir /absolute/path/mps/log
uv run uni-cumps env --name <name>
uv run uni-cumps stop --name <name>
uv run uni-cumps stop --all

Defaults

Setting Default
GPU the sole visible physical GPU, resolved to its canonical UUID
daemon name gpu-<uuid-prefix>
pipe directory ~/.cache/unilab/cuda-mps/<name>/pipe
log directory ~/.cache/unilab/cuda-mps/<name>/log
env / stop target the sole live UniLab-recorded daemon
execution mode detached control daemon

Ambiguity does not silently select a target:

  • more than one visible GPU requires --gpus;
  • more than one live daemon requires --name;
  • no live daemon is an error, not an unmanaged-pipe fallback.

Delivered behavior

  • Adds the uni-cumps package entrypoint, owned by src/unilab/training/cuda_mps_cli.py.
  • Implements status, doctor, start, stop, stop --all, and env.
  • Resolves selectors to canonical physical GPU UUIDs and rejects MIG UUIDs.
  • Models single_gpu, single_task_multi_gpu, and task_per_gpu.
  • Implements only single_gpu; multi-GPU/all daemon execution fails closed and points to CUDA MPS under multi-GPU DP: evidence gaps and design constraints #2063/roadmap: multi-GPU tensor runtime scaling (M10) #1672.
  • Stores records under ~/.cache/unilab/cuda-mps/<name>/daemon.json.
  • Scopes records by (UID, hostname, name).
  • Persists GPU UUIDs, absolute pipe/log paths, control PID, and Linux /proc/<pid>/stat start-time.
  • Parses procfs PID/start-time after the parenthesized process name, so names such as (nvidia-cuda-mps) do not break stop authority.
  • Preserves the exact nvidia-smi UUID spelling, including hyphens and case, for CUDA visibility masks.
  • Permits stop only when UID, host, record, PID, and start-time all match.
  • Quarantines stale records; never attaches to or stops unmanaged pipes.
  • doctor selects the sole matching live managed daemon and probes its pipe/log environment.
  • Prints launcher environment without mutating the parent process.
  • Adds flat subcommand/flag shell completion.
  • Adds ADR-0014 and narrows the ADR-0013 supersession to daemon lifecycle ownership.
  • Updates English and Chinese CLI/production documentation.
  • Keeps trainer semantics unchanged: training.cuda_process_sharing=mps remains explicit, with no auto mode and no implicit daemon selection.

Impact

  • Training runtime semantics: unchanged.
  • Backend scope: doctor reuses the existing MJWarp/single-rank fail-closed probe contract.
  • Platform scope: daemon lifecycle targets Linux/NVIDIA CUDA; status remains useful and fail-closed elsewhere.
  • ROCm profile retains the new entrypoint through a scripts parity test.

Validation

Commands run on final local head 52220875:

make check
make test-all

Results:

  • formatting and lint passed;
  • mypy: 133 source files, no issues;
  • pyright: 0 errors, 0 warnings;
  • non-slow coverage run: 1847 passed, 42 skipped, 488 deselected;
  • benchmark smoke: module-mode 34/34, script-mode 35/35.

Focused checks:

uv run pytest tests/training/test_cuda_mps_cli.py tests/test_completion.py \
  tests/scripts/test_rocm_dependency_profile.py tests/scripts/test_check_docs.py -q

Result: 69 passed.

Sphinx strict build was run successfully with no warning introduced by this change.

Real Linux/NVIDIA lifecycle smoke

Completed on the available host with a real nvidia-cuda-mps-control:

uv run uni-cumps start --name finale2e \
  --pipe-dir /tmp/final-e2e/pipe \
  --log-dir /tmp/final-e2e/log
uv run uni-cumps doctor
uv run uni-cumps status
uv run uni-cumps env --name finale2e
uv run uni-cumps stop --name finale2e
uv run uni-cumps status

Observed:

  • start wrote a record containing PID 2578762, procfs start-time 131179025, canonical UUID, UID, host, and absolute pipe/log paths.
  • doctor selected the live managed daemon and reported valid=True.
  • status reported the daemon live.
  • env emitted the correct CUDA_VISIBLE_DEVICES, pipe, and log exports.
  • stop returned 0 and removed the record.
  • final status reported managed_daemons=none; the control process was no longer present.

Regression coverage added from this real run:

  • parenthesized procfs comm parsing;
  • PID mismatch rejection;
  • canonical UUID spelling with hyphens/case preserved;
  • doctor selection of the sole live managed daemon and its pipe/log environment.

Scope boundaries / limitations

  • single_task_multi_gpu and task_per_gpu are contract names only; execution remains blocked pending CUDA MPS under multi-GPU DP: evidence gaps and design constraints #2063/roadmap: multi-GPU tensor runtime scaling (M10) #1672 and a lease/reference model.
  • Doctor reports the first blocking prerequisite through the existing probe contract, not every possible subsequent error.
  • Trainer does not automatically select or lease a daemon.
  • No multi-node, MIG, SM percentage, memory-limit, or affinity support is introduced.
  • Long-duration daemon soak is not part of this PR, but the real start/status/env/doctor/stop lifecycle has now been exercised once on Linux/NVIDIA.

@TATP-233
TATP-233 force-pushed the feat/issue-2079-cuda-mps-cli branch 2 times, most recently from c2d1d8a to b9b0ad5 Compare October 8, 2026 13:20
@TATP-233
TATP-233 force-pushed the feat/issue-2079-cuda-mps-cli branch from b9b0ad5 to 5222087 Compare October 8, 2026 13:38
@TATP-233
TATP-233 merged commit 3798969 into main Oct 8, 2026
8 checks passed
@TATP-233
TATP-233 deleted the feat/issue-2079-cuda-mps-cli branch October 8, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Work: add a future-proof unilab cuda mps management CLI (initial single-GPU implementation)

1 participant