Skip to content

fix(training): stabilize APPO and MJWarp tests - #2098

Merged
TATP-233 merged 2 commits into
mainfrom
fix/apppo-and-mjwarp-flaky-tests
Oct 9, 2026
Merged

TATP-233 merged 2 commits into
mainfrom
fix/apppo-and-mjwarp-flaky-tests

Conversation

@TATP-233

@TATP-233 TATP-233 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Fixed APPO HOST_BRIDGE device routing so only an explicit CUDA collector request or explicit runner/config device selects CUDA Manager Torch carriers. Device auto-detection no longer silently puts the CPU-authoritative MuJoCo Manager runtime on CUDA.
  • Made the APPO G1 motion-tracking config-load smoke deterministic with an explicit CPU learner device, independent of whether the CI/host machine exposes CUDA.
  • Stabilized the MJWarp low-friction DR regression by moving slide friction from the numerically pathological 0.01 boundary to 0.1; the test still verifies that materially lower friction changes kick response while same-condition worlds remain pairwise consistent.

Linked Work

Validation

  • make test-all passed on the final local head before this PR was created or updated
  • Additional task-specific validation listed below

Commands actually run:

# final rebased head
uv run --no-sync pytest \
  tests/base/test_mjwarp_dr.py::test_low_friction_world_changes_kick_response \
  "tests/scripts/test_train_script_configs.py::test_appo_task_configs_load[g1_motion_tracking/mujoco]" \
  tests/scripts/test_train_scripts.py::test_appo_host_bridge_carriers_require_explicit_cuda \
  -q --no-header -m slow
# 3 passed

make test-all
# ruff format: 397 files unchanged
# ruff check: all checks passed
# mypy: no issues in 133 source files
# pyright: 0 errors, 0 warnings, 0 informations
# focused tests lint check: passed
# non-slow coverage suite: 1885 passed, 21 skipped
# benchmark smoke: module-mode 34/34; script-mode 35/35

Flake evidence and focused stability:

# Original friction value: NaN reproduced once in 10 runs (world 3 became non-finite).
# Updated value: 20 consecutive focused runs passed, followed by the final reruns above.

Remote CI route:

  • Base main: current-head CI will be recorded on this PR and must pass before merge.

Impact

  • Backend impact: mujoco (APPO smoke), mjwarp (test-only friction value)
  • Platform impact: Linux validated; APPO fix also prevents CUDA-dependent behavior on hosts without CUDA
  • Training effect expected: yes — restores deterministic APPO G1 startup for the CPU-carrier owner without changing explicit CUDA collector behavior

Artifacts

  • W&B: N/A
  • benchmark result: N/A
  • video / screenshot: N/A
  • ONNX / checkpoint: N/A

Checklist

  • Added or updated tests where needed
  • Updated docs if behavior or workflow changed (internal routing/test behavior only; no user-facing workflow change)
  • Linked the driving issue
  • Noted any follow-up work explicitly

@TATP-233
TATP-233 requested a review from caozx1110 as a code owner October 8, 2026 16:13
@TATP-233 TATP-233 added bug Something isn't working area:mujoco MuJoCo related work labels Oct 8, 2026
@TATP-233

TATP-233 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Current-head remote CI: https://github.com/Motphys/UniLab/actions/runs/37807084690 — lint, type, docs, and benchmark jobs passed. The job was still reported as in progress when the run rollup completed; I’m tracking it and will update this PR before merge.

@TATP-233
TATP-233 merged commit dde162d into main Oct 9, 2026
8 checks passed
@TATP-233
TATP-233 deleted the fix/apppo-and-mjwarp-flaky-tests branch October 9, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:mujoco MuJoCo related work bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ipc): bound APPO rollout tensor views to payload size on macOS

1 participant