feat(nemo-gym): create one gym actor per shard on its own node

[ananthsub]

What does this PR do ?

Builds the actors. Given the shard plan parsed in PR 3 of this stack, this PR creates one NemoGym actor per shard instance, each on its own node, each holding a complete and self-contained Gym stack.

build_nemo_gym_actors(...) in nemo_rl/environments/nemo_gym.py:

1. Expand. Turn the plan into one instance per replica, so a shard with replicas: 3 becomes three actors. The total instance count is the sum of replicas across shards.
2. Place. Request a placement group with one STRICT_SPREAD bundle per instance and wait for it to be ready. STRICT_SPREAD is what makes each actor land on a distinct node, which is the point of sharding — without it the shards would share a node's CPUs and RAM and nothing would be isolated. If the allocation has fewer eligible nodes than instances, this fails here, before any actor is created, with a message naming the shortfall.
3. Merge once per shard. Each shard's Gym config is assembled once and reused across its replicas, so replicas are identical by construction rather than by coincidence.
4. Create and spin up concurrently. One actor per instance, pinned to its bundle, then _spinup on all of them at once so wall-clock time is roughly the slowest shard rather than their sum. Config faults surface inside the actor — a server reference pointing at an entry that is not in this shard's slice raises Gym's ServerRefNotFoundError there. Gym names the entry and field but has no concept of a shard, so the shard and replica are attached here.
5. Discover and validate. Ask one replica per shard for its list_entries(), build the agent-to-shard map, and reject any entry name hosted by more than one shard unless it is in allowed_duplicate_entries. That check is what catches a judge model accidentally defined in two shards, which would claim its GPUs twice.

Any failure or timeout tears down every handle and removes the placement group. This matters more than it looks: ray.get(..., timeout=) does not cancel the actor-side work it stopped waiting for, so without an explicit teardown a half-started stack keeps running with nothing left to stop it.

NemoGymShardSet holds the resulting handles keyed by shard name, the placement group, and the agent-to-shard map, and owns teardown of all three.

The unsharded path is preserved: with no shards key the plan is a single implicit shard with one replica, and spinup_nemo_gym_actor returns that one handle as before.

Issues

None.

Usage

No user-facing change yet — nothing calls build_nemo_gym_actors until PR 7 of the stack.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• Each shard must be reference-closed: every server reference in a shard's config has to resolve inside that shard. A shard is a whole Gym stack, not a fragment of one, so a resources server that names a model server needs that model server in the same shard. This is enforced by Gym itself at spinup rather than by a NeMo-RL pre-check.
• Two shards cannot be colocated on one node with the same port range. Gym picks a port at random within the range and deduplicates only within a stack, so two stacks drawing from one range can collide. port_range_low/port_range_high per shard (PR 3) exists for that case.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.