feat(training): let trainer use the sole live uni-cumps daemon without shell eval #2093

Problem

uni-cumps now owns the explicit daemon lifecycle, but the trainer still requires the user to export the daemon environment into the shell:

uv run uni-cumps start
uv run --extra mjwarp train --algo flashsac \
  --task g1_motion_tracking \
  --sim mjwarp \
  training.cuda_process_sharing=mps

With only start completed, the trainer probes /tmp/nvidia-mps/control and fails:

ValueError: training.cuda_process_sharing='mps' could not reach the control daemon
through /tmp/nvidia-mps/control.

The working command currently requires this extra shell integration:

eval "$(uv run uni-cumps env)"

Issue #2079 explicitly deferred this launcher integration: Phase 1 only improved diagnostics and taught users the explicit daemon command. That phase has now landed, so the follow-up can make the normal workflow start-and-train.

Deliverable

For the supported single-host, single-rank MJWarp off-policy path, let the trainer resolve a UniLab-recorded daemon without mutating the parent shell.

When training.cuda_process_sharing=mps and CUDA_MPS_PIPE_DIRECTORY is absent:

1. enumerate current-user/current-host UniLab daemon records;
2. select the sole live daemon whose canonical GPU UUID matches the rank-local learner/collector GPU;
3. set CUDA_MPS_PIPE_DIRECTORY and CUDA_MPS_LOG_DIRECTORY only in the trainer process environment before the fail-closed probe and before environment/learner/collector construction;
4. keep using an explicit caller-provided CUDA_MPS_PIPE_DIRECTORY when present;
5. fail closed when there is no live matching daemon, multiple matching live daemons, or host/UID mismatch;
6. include host-level daemon identity additively in the existing producer diagnostics without making it a stable public scalar contract.

The trainer must not start, stop, repair, or lease-refcount a daemon.

Scope and delivery boundaries

In this work item:

• trainer-side selection of one already-running, already-recorded daemon;
• single-GPU MJWarp SAC/FlashSAC path;
• tests and production-guide updates;
• improved errors that distinguish “no live daemon” from “daemon exists but failed validation”.

Separate outcomes:

• starting/stopping or leasing a daemon from the trainer;
• multi-GPU DP daemon selection and cross-rank aggregation;
• task-per-GPU scheduling/refcounts;
• multi-node daemon discovery;
• training.cuda_process_sharing=auto.

Affected owner layers and contracts

• Training assembly: src/unilab/scripts/train_offpolicy.py
• Daemon record owner: src/unilab/training/cuda_mps_cli.py
• Existing probe owner: src/unilab/training/cuda_process_sharing.py
• Tests: focused off-policy builder tests and MPS CLI tests
• Docs: English and Chinese tensor-runtime production guides
• Governing decisions: ADR-0013, ADR-0014, issue Work: add a future-proof unilab cuda mps management CLI (initial single-GPU implementation) #2079 follow-up boundary

Definition of done

• After uv run uni-cumps start, the documented training command succeeds without eval "$(uv run uni-cumps env)".
• Existing explicit CUDA_MPS_PIPE_DIRECTORY behavior remains unchanged.
• Trainer does not create or stop a daemon.
• No matching live daemon produces an error naming uni-cumps start and the required GPU.
• Multiple matching live daemons fail closed and request an explicit daemon selection mechanism.
• GPU matching uses canonical UUIDs, not ordinals.
• Validation still occurs before env factory, learner construction, and collector spawn.
• make check, make test, and make test-all pass on the final head.

Dependencies and blockers

• PR fix(cli): keep default MPS runtime within socket limits #2092 must land first so the default daemon runtime/record lifecycle is reliable.
• A design choice is needed for selecting among multiple matching daemons if a stable training.cuda_mps_daemon owner key is preferred over requiring explicit environment in that case.

Proposed owner

UniLab training runtime maintainers

Validation plan

1. Unit tests with fake daemon records and fake process identity.
2. Builder test proving selected daemon environment reaches the existing fail-closed probe.
3. Explicit-environment precedence test.
4. No-daemon and ambiguous-daemon fail-closed tests before env construction.
5. Linux/NVIDIA integration: uni-cumps start, then the documented trainer command without shell eval.
6. Full repository gate.