feat(training): select recorded CUDA MPS daemon

[TATP-233]

Summary

After uv run uni-cumps start, single-GPU MJWarp SAC/FlashSAC training with training.cuda_process_sharing=mps can select the sole live UniLab-recorded daemon without importing its environment into the shell. An explicit CUDA_MPS_PIPE_DIRECTORY takes precedence. Selection matches the physical GPU UUID and fails before environment/learner/collector construction when no live matching daemon exists or the choice is ambiguous.

Startup also waits for a successful control query before recording the daemon. Both the readiness check and the doctor/training probe send get_server_list\n through stdin to nvidia-cuda-mps-control. Passing the command as an argument left the client waiting for stdin on CUDA 13, causing a timeout even though the daemon had accepted the connection. A successful empty server list is accepted because MPS servers start lazily. Failed readiness triggers cleanup of the newly launched control process and reports its log tail.

The trainer only selects an existing daemon; lifecycle management remains CLI-owned.

uv run uni-cumps start
uv run --extra mjwarp train --algo flashsac \
  --task g1_motion_tracking --sim mjwarp \
  training.cuda_process_sharing=mps
uv run uni-cumps stop

eval "$(uv run uni-cumps env)" remains available for explicit shell integration.

Linked Work

• Closes feat(training): let trainer use the sole live uni-cumps daemon without shell eval #2093; follows Work: add a future-proof unilab cuda mps management CLI (initial single-GPU implementation) #2079 and fix(cli): default uni-cumps daemon path exceeds Linux MPS socket limit #2091.
• Governing decisions: ADR-0013 and ADR-0014.
• Base branch: main.

Validation

Final local head: b41fdac5.

• Focused regression tests passed: 58 tests, including stdin query delivery, readiness retry/cleanup, daemon selection, UUID matching, and explicit-environment precedence.
• make test-all passed on the final local head before updating this PR.

Commands run against this tree:

uv run pytest tests/training/test_cuda_mps_cli.py tests/training/test_cuda_process_sharing.py -q
git diff --check
make check
make test
make test-all
uv run uni-cumps start
uv run uni-cumps doctor
uv run uni-cumps stop
uv run uni-cumps start
uv run uni-cumps env
uv run uni-cumps doctor

make check passed. make test and the full make test-all gate passed with 1,853 passed, 42 skipped, and 488 deselected. Coverage: 70%. Benchmark entrypoint smoke: module mode 34/34 and script mode 35/35.

On Linux / NVIDIA RTX 4090 / CUDA MPS control version 13000, an isolated real-daemon reproduction showed the argument-based query timing out and the stdin query returning immediately. With the fix, start, stop, and restart succeeded; doctor reported valid=True. The managed daemon was left running. Full training was not rerun for this correction.

Remote checks for b41fdac52d2b50d1c46db3231cbd1fd8785695b2: CI and Docs both passed. Ruff lint/format, mypy, pyright, benchmark smoke, coverage tests, and Sphinx build succeeded. Documentation deployment was skipped as expected for a PR.

Impact and Limitations

• Platform: Linux / NVIDIA CUDA MPS.
• Backend/training: MJWarp SAC/FlashSAC daemon selection and validation; no MuJoCo or Motrix backend implementation changes.
• Parent shell remains unchanged. Trainer does not start, stop, repair, or lease a daemon.
• Multi-GPU daemon selection remains deferred to CUDA MPS under multi-GPU DP: evidence gaps and design constraints #2063 / roadmap: multi-GPU tensor runtime scaling (M10) #1672; no new support claim or public manifest field.
• No lease/refcount prevents another process from stopping the daemon during training.

Checklist

• Tests cover the affected behavior.
• English and Chinese production guides document automatic daemon selection.
• Driving issue and governing ADRs linked.
• Follow-up scope and limitations noted.