[Feature] Add reliable VLA-GRPO LIBERO recipe

[vmoens]

Summary

• Fix VLA chunk reward propagation so early terminal success inside a MultiAction(stack_rewards=False) chunk is preserved as the outer transition reward.
• Keep GRPO advantages on ("next", "reward") with summed trajectory returns and mixed-group filtering for useful updates.
• Add the LIBERO OpenVLA-OFT token recipe using the Haozhan72/Openvla-oft-SFT-libero-spatial-trajall checkpoint, candidate oversampling, process inference-server collection, bounded video eval, and SOTA recipe/smoke coverage.
• Reduce LIBERO startup overhead by using homogeneous ParallelEnv factories with per-worker kwargs, closing temporary metadata envs, and adding optional worker-originated metadata collection.

Validation

• pytest -q sota-implementations/vla_grpo/test_openvla.py (46 passed, 1 skipped)
• pytest -q test/transforms/test_action_transforms.py -k stack_rewards_false_done_mid_chunk_keeps_terminal_reward
• pytest -q test/envs/test_parallel.py -k 'get_env_metadata_closes_callable_env or metadata_from_workers'
• pytest -q .github/unittest/linux_sota/scripts/test_sota.py -k recipe
• python -m py_compile sota-implementations/vla_grpo/vla-grpo.py sota-implementations/vla_grpo/utils.py sota-implementations/vla_grpo/smoke.py

Current scale-run status

A full 8-GPU LIBERO-Spatial run is active with async 500-episode scalar eval plus bounded video eval. The PR is draft until the scale run produces collection/update curves and the final command/log evidence is attached.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4024

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7239789 with merge base d7c3fb5 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Cold-start collector update pushed in 7239789344:

• both VLA-GRPO LIBERO ParallelEnv paths use one callable plus worker-specific create_env_kwargs;
• worker task/instruction, group id, init-state, seed, worker offset, and render-GPU assignment are covered;
• direct construction instrumentation confirms 64 metadata envs -> 1 per subcollector, or 320 -> 5 across five 64-env subcollectors, while 320 real workers are unchanged (640 -> 325 total constructions);
• validation: 6 relevant test/envs/test_parallel.py tests passed; sota-implementations/vla_grpo/test_openvla.py passed 48 tests with 1 real-checkpoint test skipped.