[Doc] Clarify PPO/Critic docs after #1856

docs/en/advanced/megatron-config.md

@@ -84,14 +84,12 @@ python train.py \
   --expert-tensor-parallel-size 1 \
   --actor-num-nodes 1 \
   --actor-num-gpus-per-node 8 \
-  --critic-num-nodes 1 \
-  --critic-num-gpus-per-node 8 \
   ...
 ```
 
 In this setup:
 
-- CLI defines the shared topology and resource layout.
+- CLI defines the shared topology and resource layout; in current PPO, critic training resources follow the actor configuration.
 - YAML defines the role-specific differences, such as `lr`, `load`, `save`, or optimizer / scheduler parameters.
 
 ### Overriding Only One Role
@@ -114,6 +112,7 @@ In this case the actor keeps the shared CLI arguments unchanged.
 
 - **PPO only for now.** `--megatron-config-path` is currently intended for PPO actor / critic role configuration. It is not the recommended interface for GRPO, REINFORCE++, and other critic-free workflows.
 - **Actor and critic must use the same Megatron parallel topology in current PPO.** In particular, topology-related settings such as `tensor_model_parallel_size`, `pipeline_model_parallel_size`, `context_parallel_size`, `expert_model_parallel_size`, `expert_tensor_parallel_size`, and `sequence_parallel` should not differ between actor and critic.
+- **Actor and critic share the same train placement group in current PPO.** The critic node count and GPUs per node are derived from the actor configuration and cannot be used as an independent resource scale.
 - **Keep topology-related settings on CLI.** The safest current pattern is to keep parallelism and resource arguments in the shared CLI configuration, and only put role-specific differences in YAML, such as `lr`, `load`, `save`, warmup, and optimizer / scheduler settings.
 
 If you configure different parallel topologies for actor and critic, the behavior is currently unsupported and may fail during initialization or training.
@@ -126,6 +125,6 @@ If you configure different parallel topologies for actor and critic, the behavio
 
 Yes. Missing roles automatically inherit the shared CLI arguments, so you do not need to duplicate everything in YAML.
 
-### Q: Can I move `--actor-num-nodes` or `--critic-num-gpus-per-node` into YAML?
+### Q: Can I move resource settings into YAML?
 
-No. Resource allocation and placement groups are still controlled by CLI arguments, and the corresponding YAML fields are ignored.
+No. Resource allocation and placement groups are still controlled by CLI arguments, and the corresponding YAML fields are ignored. `--actor-num-nodes` / `--actor-num-gpus-per-node` determine the PPO train resource scale; the critic node count and GPUs per node follow the actor configuration and cannot be configured independently.

docs/en/get_started/usage.md

@@ -238,35 +238,17 @@ To use PPO, set:
 --advantage-estimator ppo
 ```
 
-**Note: In PPO, the Critic and Actor request GPUs in parallel**, which should be considered when allocating resources. Specifically:
+**Note: In PPO, the critic and actor share the same training GPU group.** You do not need to reserve a separate set of GPUs for the critic. Specifically:
 
-- The critic model occupies a separate set of GPUs, independent from the actor's GPU resources.
-- You can configure critic resources using `--critic-num-nodes` and `--critic-num-gpus-per-node`.
-- If critic resource parameters are not configured, the same resource configuration as the actor will be used by default.
+- PPO creates separate actor and critic training process groups, but places them on the same train placement group.
+- The critic training scale follows the actor configuration, and the actor / critic Megatron parallel topology must currently stay identical.
+- PPO forces train-side offload so that actor and critic can wake up and release memory on the same GPUs in turn.
+- There are currently no separate CLI arguments for configuring critic training resources; the critic node count and GPUs per node are derived from the actor configuration.
 
-Cluster resource allocation example:
-
-```bash
-# Actor uses 1 node, 4 GPUs
---actor-num-nodes 1
---actor-num-gpus-per-node 4
-
-# Critic uses 1 node, 4 GPUs (parallel to Actor)
---critic-num-nodes 1
---critic-num-gpus-per-node 4
-
-# Rollout uses 8 GPUs
---rollout-num-gpus 8
-```
-
-With the above configuration, a total of `4 (actor) + 4 (critic) + 8 (rollout) = 16` GPUs are required.
 
 PPO-related parameters:
 
-- `--critic-load`: Checkpoint path for the critic model.
-- `--critic-save`: Save path for the critic model.
-- `--critic-lr`: Learning rate for the critic model.
-- `--critic-lr-warmup-iters`: Number of warmup steps for the critic model.
+- `--megatron-config-path`: YAML config for role-specific Megatron overrides, such as setting critic-specific `load`, `save`, `lr`, or warmup parameters.
 - `--num-critic-only-steps`: Number of steps to train only the critic at the beginning of training.
 - `--eps-clip`: PPO clip range.
 - `--value-clip`: Clip range for value loss.

docs/zh/advanced/megatron-config.md

@@ -84,14 +84,12 @@ python train.py \
   --expert-tensor-parallel-size 1 \
   --actor-num-nodes 1 \
   --actor-num-gpus-per-node 8 \
-  --critic-num-nodes 1 \
-  --critic-num-gpus-per-node 8 \
   ...
 ```
 
 在这个模式下：
 
-- CLI 负责共享的并行策略和资源配置；
+- CLI 负责共享的并行策略和资源配置；当前 PPO 下 critic 的训练资源会跟随 actor 配置；
 - YAML 负责 actor / critic 的差异项，比如 `lr`、`load`、`save`、optimizer 或 scheduler 相关参数。
 
 ### 只覆盖一个角色
@@ -114,6 +112,7 @@ megatron:
 
 - **目前只支持 PPO。** `--megatron-config-path` 当前主要用于 PPO 工作流中的 actor / critic 角色配置。对于 GRPO、REINFORCE++ 等不依赖 critic 的流程，目前不建议使用这套角色配置。
 - **当前 PPO 下，actor 和 critic 的 Megatron 并行配置必须一致。** 特别是 `tensor_model_parallel_size`、`pipeline_model_parallel_size`、`context_parallel_size`、`expert_model_parallel_size`、`expert_tensor_parallel_size`、`sequence_parallel` 等拓扑相关参数，不应在 actor 和 critic 之间配置成不同的值。
+- **当前 PPO 下，actor 和 critic 共享同一组 train placement group。** critic 的节点数和每节点 GPU 数由 actor 配置派生，不能作为独立资源规模配置。
 - **推荐把并行相关参数继续放在 CLI 中。** 当前最稳妥的用法是：并行与资源参数写在公共 CLI 中，只在 YAML 中覆盖角色差异项，例如 `lr`、`load`、`save`、warmup、optimizer / scheduler 参数等。
 
 如果你在 actor 和 critic 之间写入不同的并行拓扑，当前行为不受支持，可能导致初始化或训练过程出错。
@@ -126,6 +125,6 @@ megatron:
 
 可以。缺失角色会自动继承公共 CLI 参数，不需要把所有参数都重复写一遍。
 
-### Q: 可以把 `--actor-num-nodes` 或 `--critic-num-gpus-per-node` 写进 YAML 吗？
+### Q: 可以把资源配置写进 YAML 吗？
 
-不可以。当前资源分配和 placement group 仍由 CLI 参数控制，YAML 中对应字段会被忽略。
+不可以。当前资源分配和 placement group 仍由 CLI 参数控制，YAML 中对应字段会被忽略。其中 `--actor-num-nodes` / `--actor-num-gpus-per-node` 决定 PPO 的 train 资源规模；critic 的节点数和每节点 GPU 数会跟随 actor 配置，不能独立配置。

docs/zh/get_started/usage.md

@@ -242,35 +242,17 @@ PPO（Proximal Policy Optimization）是经典的 RL 算法，使用 critic 模
 --advantage-estimator ppo
 ```
 
-**注意：PPO 的 Critic 和 Actor 是并列申请 GPU 的**，在资源分配时需要考虑这一点。具体来说：
+**注意：当前 PPO 下 Critic 和 Actor 共享同一组训练 GPU**，资源分配时不需要为 critic 额外预留一组独立 GPU。具体来说：
 
-- Critic 模型会独立占用一组 GPU，与 Actor 的 GPU 资源分开；
-- 可以通过 `--critic-num-nodes` 和 `--critic-num-gpus-per-node` 来配置 critic 使用的资源；
-- 如果不配置 critic 的资源参数，默认会使用与 actor 相同的资源配置。
+- PPO 会创建 actor 和 critic 两套训练进程组，但它们会被放到同一组 train placement group 上；
+- critic 的训练规模跟随 actor 配置，当前 actor / critic 的 Megatron 并行拓扑必须保持一致；
+- PPO 会强制开启 train 侧 offload，使 actor 和 critic 在同一批 GPU 上轮流唤醒和释放显存；
+- 当前没有单独配置 critic 训练资源的 CLI 参数，critic 的节点数和每节点 GPU 数会由 actor 配置派生。
 
-集群资源分配示例：
-
-```bash
-# Actor 使用 1 个节点，4 张 GPU
---actor-num-nodes 1
---actor-num-gpus-per-node 4
-
-# Critic 使用 1 个节点，4 张 GPU（与 Actor 并列）
---critic-num-nodes 1
---critic-num-gpus-per-node 4
-
-# Rollout 使用 8 张 GPU
---rollout-num-gpus 8
-```
-
-在上述配置下，总共需要 `4 (actor) + 4 (critic) + 8 (rollout) = 16` 张 GPU。
 
 PPO 相关参数：
 
-- `--critic-load`：critic 模型的 checkpoint 路径；
-- `--critic-save`：critic 模型的保存路径；
-- `--critic-lr`：critic 模型的学习率；
-- `--critic-lr-warmup-iters`：critic 模型的 warmup 步数；
+- `--megatron-config-path`：通过 YAML 对 actor / critic 分别覆盖 Megatron 参数，例如为 critic 单独设置 `load`、`save`、`lr` 或 warmup 参数；
 - `--num-critic-only-steps`：训练开始时只训练 critic 的步数；
 - `--eps-clip`：PPO clip 范围；
 - `--value-clip`：value loss 的 clip 范围；