Skip to content

Bug: CUDA MPS CLI tests fail on macOS #2510

Closed

Description

@TATP-233

Summary

tests/training/test_cuda_mps_cli.py is not reliably runnable on macOS. With the normal pytest temporary directory, make test-all reaches 2673 passed / 133 skipped and then fails 13 tests in this file. Shortening the temporary root with TMPDIR=/tmp removes the AF_UNIX path failures but still leaves 4 failures.

Area

CI/tooling

Reproduction

Normal macOS pytest temporary root:

make test-all

Focused run with the longer default Darwin pytest temporary root:

uv run pytest -m 'not slow' tests/training/test_cuda_mps_cli.py -q

Focused run with a shorter temporary root:

TMPDIR=/tmp uv run pytest -m 'not slow' tests/training/test_cuda_mps_cli.py -q

Expected behavior

The MPS CLI test module either passes on supported test platforms or explicitly skips macOS so the repository test gate can complete locally. Any test that intentionally fakes a Linux/NVIDIA host should exercise its intended validation branch independently of the physical host platform.

Actual behavior

Default Darwin pytest paths fail before the intended assertions because Unix-domain socket paths exceed the platform limit. With TMPDIR=/tmp, four tests still fail because platform faking is missing from the direct start_daemon() test paths, so production _require_linux_nvidia() fails first.

Default-path failures (13):

test_start_rejects_task_per_gpu_until_lease_contract_lands
test_status_reports_managed_control_pipe_and_diagnostic
test_start_daemon_uses_explicit_directories_and_records_identity
test_start_daemon_waits_until_control_answers_query
test_start_daemon_cleans_up_unrecorded_control_after_readiness_failure
test_start_daemon_recovers_only_inert_failed_control_artifacts
test_start_daemon_failure_reports_control_log_and_cleans_artifacts
test_default_runtime_recovers_orphan_control_sockets
test_start_daemon_refuses_existing_unmanaged_control_path
test_start_daemon_quarantines_stale_record
test_doctor_uses_sole_live_managed_daemon_pipe
test_doctor_reports_missing_control_daemon_and_exits_invalid
test_regular_file_control_pipe_is_rejected_by_probe

Short-path failures (4):

test_start_rejects_task_per_gpu_until_lease_contract_lands
test_start_daemon_uses_explicit_directories_and_records_identity
test_start_daemon_waits_until_control_answers_query
test_start_daemon_cleans_up_unrecorded_control_after_readiness_failure

Platform guard mismatch example:

CudaMpsError: CUDA MPS management requires Linux and NVIDIA CUDA; current platform is 'Darwin'.
Expected regex: 'current support is single_gpu'

AF_UNIX/path-length examples:

OSError: AF_UNIX path too long
CudaMpsError: MPS control socket path is too long (99 bytes; maximum 95): /private/tmp/pytest-of-yves/.../control

Environment

  • OS: macOS (Darwin)
  • Python: 3.13.12
  • Commit: ed9ff2f9 base (also observed with b1a623c0, which does not touch this module)
  • Backend: MuJoCo environment installed; tests invoke the CUDA MPS CLI with fake subprocesses

Logs and artifacts

  • Full local run discussed in PR fix(rl): initialize off-policy MPS inference timer #2509: 13 failed, 2673 passed, 133 skipped.
  • Focused short-root run: 4 failed, 39 passed.
  • Earlier AF_UNIX fix: closed issue macOS MPS tests exceed AF_UNIX path limit #2089 and commit a1084ce8 (default runtime path), but the current test fixture and several direct tests still derive paths from pytest tmp_path.
  • Direct start_daemon() tests lack the linux_host fixture's platform.system() fake where they call helper logic directly instead of through the fixture path.

Proposed fix direction

  • Keep AF_UNIX-producing tests on a short shared test-only temporary root, or make the module skip when the effective socket path cannot fit the platform limit.
  • Apply the fake Linux platform consistently to direct start_daemon() tests that intentionally simulate a Linux/NVIDIA deployment.
  • Keep production _require_linux_nvidia() behavior unchanged; this is a test-environment issue only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions