From 7b7687b1761d9a4dc4526a4b7b7eb423fd77aa62 Mon Sep 17 00:00:00 2001 From: Chenhe Gu Date: Thu, 9 Jul 2026 11:07:33 +0800 Subject: [PATCH] fix ppo docs --- docs/en/advanced/megatron-config.md | 9 ++++----- docs/en/get_started/usage.md | 30 ++++++----------------------- docs/zh/advanced/megatron-config.md | 9 ++++----- docs/zh/get_started/usage.md | 30 ++++++----------------------- 4 files changed, 20 insertions(+), 58 deletions(-) diff --git a/docs/en/advanced/megatron-config.md b/docs/en/advanced/megatron-config.md index 973363ca81..7faeff1b75 100644 --- a/docs/en/advanced/megatron-config.md +++ b/docs/en/advanced/megatron-config.md @@ -84,14 +84,12 @@ python train.py \ --expert-tensor-parallel-size 1 \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 8 \ - --critic-num-nodes 1 \ - --critic-num-gpus-per-node 8 \ ... ``` In this setup: -- CLI defines the shared topology and resource layout. +- CLI defines the shared topology and resource layout; in current PPO, critic training resources follow the actor configuration. - YAML defines the role-specific differences, such as `lr`, `load`, `save`, or optimizer / scheduler parameters. ### Overriding Only One Role @@ -114,6 +112,7 @@ In this case the actor keeps the shared CLI arguments unchanged. - **PPO only for now.** `--megatron-config-path` is currently intended for PPO actor / critic role configuration. It is not the recommended interface for GRPO, REINFORCE++, and other critic-free workflows. - **Actor and critic must use the same Megatron parallel topology in current PPO.** In particular, topology-related settings such as `tensor_model_parallel_size`, `pipeline_model_parallel_size`, `context_parallel_size`, `expert_model_parallel_size`, `expert_tensor_parallel_size`, and `sequence_parallel` should not differ between actor and critic. +- **Actor and critic share the same train placement group in current PPO.** The critic node count and GPUs per node are derived from the actor configuration and cannot be used as an independent resource scale. - **Keep topology-related settings on CLI.** The safest current pattern is to keep parallelism and resource arguments in the shared CLI configuration, and only put role-specific differences in YAML, such as `lr`, `load`, `save`, warmup, and optimizer / scheduler settings. If you configure different parallel topologies for actor and critic, the behavior is currently unsupported and may fail during initialization or training. @@ -126,6 +125,6 @@ If you configure different parallel topologies for actor and critic, the behavio Yes. Missing roles automatically inherit the shared CLI arguments, so you do not need to duplicate everything in YAML. -### Q: Can I move `--actor-num-nodes` or `--critic-num-gpus-per-node` into YAML? +### Q: Can I move resource settings into YAML? -No. Resource allocation and placement groups are still controlled by CLI arguments, and the corresponding YAML fields are ignored. \ No newline at end of file +No. Resource allocation and placement groups are still controlled by CLI arguments, and the corresponding YAML fields are ignored. `--actor-num-nodes` / `--actor-num-gpus-per-node` determine the PPO train resource scale; the critic node count and GPUs per node follow the actor configuration and cannot be configured independently. diff --git a/docs/en/get_started/usage.md b/docs/en/get_started/usage.md index 6e9eaac40d..2ecebaba65 100644 --- a/docs/en/get_started/usage.md +++ b/docs/en/get_started/usage.md @@ -238,35 +238,17 @@ To use PPO, set: --advantage-estimator ppo ``` -**Note: In PPO, the Critic and Actor request GPUs in parallel**, which should be considered when allocating resources. Specifically: +**Note: In PPO, the critic and actor share the same training GPU group.** You do not need to reserve a separate set of GPUs for the critic. Specifically: -- The critic model occupies a separate set of GPUs, independent from the actor's GPU resources. -- You can configure critic resources using `--critic-num-nodes` and `--critic-num-gpus-per-node`. -- If critic resource parameters are not configured, the same resource configuration as the actor will be used by default. +- PPO creates separate actor and critic training process groups, but places them on the same train placement group. +- The critic training scale follows the actor configuration, and the actor / critic Megatron parallel topology must currently stay identical. +- PPO forces train-side offload so that actor and critic can wake up and release memory on the same GPUs in turn. +- There are currently no separate CLI arguments for configuring critic training resources; the critic node count and GPUs per node are derived from the actor configuration. -Cluster resource allocation example: - -```bash -# Actor uses 1 node, 4 GPUs ---actor-num-nodes 1 ---actor-num-gpus-per-node 4 - -# Critic uses 1 node, 4 GPUs (parallel to Actor) ---critic-num-nodes 1 ---critic-num-gpus-per-node 4 - -# Rollout uses 8 GPUs ---rollout-num-gpus 8 -``` - -With the above configuration, a total of `4 (actor) + 4 (critic) + 8 (rollout) = 16` GPUs are required. PPO-related parameters: -- `--critic-load`: Checkpoint path for the critic model. -- `--critic-save`: Save path for the critic model. -- `--critic-lr`: Learning rate for the critic model. -- `--critic-lr-warmup-iters`: Number of warmup steps for the critic model. +- `--megatron-config-path`: YAML config for role-specific Megatron overrides, such as setting critic-specific `load`, `save`, `lr`, or warmup parameters. - `--num-critic-only-steps`: Number of steps to train only the critic at the beginning of training. - `--eps-clip`: PPO clip range. - `--value-clip`: Clip range for value loss. diff --git a/docs/zh/advanced/megatron-config.md b/docs/zh/advanced/megatron-config.md index 1616c9f830..ab504a8417 100644 --- a/docs/zh/advanced/megatron-config.md +++ b/docs/zh/advanced/megatron-config.md @@ -84,14 +84,12 @@ python train.py \ --expert-tensor-parallel-size 1 \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 8 \ - --critic-num-nodes 1 \ - --critic-num-gpus-per-node 8 \ ... ``` 在这个模式下： -- CLI 负责共享的并行策略和资源配置； +- CLI 负责共享的并行策略和资源配置；当前 PPO 下 critic 的训练资源会跟随 actor 配置； - YAML 负责 actor / critic 的差异项，比如 `lr`、`load`、`save`、optimizer 或 scheduler 相关参数。 ### 只覆盖一个角色 @@ -114,6 +112,7 @@ megatron: - **目前只支持 PPO。** `--megatron-config-path` 当前主要用于 PPO 工作流中的 actor / critic 角色配置。对于 GRPO、REINFORCE++ 等不依赖 critic 的流程，目前不建议使用这套角色配置。 - **当前 PPO 下，actor 和 critic 的 Megatron 并行配置必须一致。** 特别是 `tensor_model_parallel_size`、`pipeline_model_parallel_size`、`context_parallel_size`、`expert_model_parallel_size`、`expert_tensor_parallel_size`、`sequence_parallel` 等拓扑相关参数，不应在 actor 和 critic 之间配置成不同的值。 +- **当前 PPO 下，actor 和 critic 共享同一组 train placement group。** critic 的节点数和每节点 GPU 数由 actor 配置派生，不能作为独立资源规模配置。 - **推荐把并行相关参数继续放在 CLI 中。** 当前最稳妥的用法是：并行与资源参数写在公共 CLI 中，只在 YAML 中覆盖角色差异项，例如 `lr`、`load`、`save`、warmup、optimizer / scheduler 参数等。 如果你在 actor 和 critic 之间写入不同的并行拓扑，当前行为不受支持，可能导致初始化或训练过程出错。 @@ -126,6 +125,6 @@ megatron: 可以。缺失角色会自动继承公共 CLI 参数，不需要把所有参数都重复写一遍。 -### Q: 可以把 `--actor-num-nodes` 或 `--critic-num-gpus-per-node` 写进 YAML 吗？ +### Q: 可以把资源配置写进 YAML 吗？ -不可以。当前资源分配和 placement group 仍由 CLI 参数控制，YAML 中对应字段会被忽略。 \ No newline at end of file +不可以。当前资源分配和 placement group 仍由 CLI 参数控制，YAML 中对应字段会被忽略。其中 `--actor-num-nodes` / `--actor-num-gpus-per-node` 决定 PPO 的 train 资源规模；critic 的节点数和每节点 GPU 数会跟随 actor 配置，不能独立配置。 diff --git a/docs/zh/get_started/usage.md b/docs/zh/get_started/usage.md index ae9bcbb154..533729c40a 100644 --- a/docs/zh/get_started/usage.md +++ b/docs/zh/get_started/usage.md @@ -242,35 +242,17 @@ PPO（Proximal Policy Optimization）是经典的 RL 算法，使用 critic 模 --advantage-estimator ppo ``` -**注意：PPO 的 Critic 和 Actor 是并列申请 GPU 的**，在资源分配时需要考虑这一点。具体来说： +**注意：当前 PPO 下 Critic 和 Actor 共享同一组训练 GPU**，资源分配时不需要为 critic 额外预留一组独立 GPU。具体来说： -- Critic 模型会独立占用一组 GPU，与 Actor 的 GPU 资源分开； -- 可以通过 `--critic-num-nodes` 和 `--critic-num-gpus-per-node` 来配置 critic 使用的资源； -- 如果不配置 critic 的资源参数，默认会使用与 actor 相同的资源配置。 +- PPO 会创建 actor 和 critic 两套训练进程组，但它们会被放到同一组 train placement group 上； +- critic 的训练规模跟随 actor 配置，当前 actor / critic 的 Megatron 并行拓扑必须保持一致； +- PPO 会强制开启 train 侧 offload，使 actor 和 critic 在同一批 GPU 上轮流唤醒和释放显存； +- 当前没有单独配置 critic 训练资源的 CLI 参数，critic 的节点数和每节点 GPU 数会由 actor 配置派生。 -集群资源分配示例： - -```bash -# Actor 使用 1 个节点，4 张 GPU ---actor-num-nodes 1 ---actor-num-gpus-per-node 4 - -# Critic 使用 1 个节点，4 张 GPU（与 Actor 并列） ---critic-num-nodes 1 ---critic-num-gpus-per-node 4 - -# Rollout 使用 8 张 GPU ---rollout-num-gpus 8 -``` - -在上述配置下，总共需要 `4 (actor) + 4 (critic) + 8 (rollout) = 16` 张 GPU。 PPO 相关参数： -- `--critic-load`：critic 模型的 checkpoint 路径； -- `--critic-save`：critic 模型的保存路径； -- `--critic-lr`：critic 模型的学习率； -- `--critic-lr-warmup-iters`：critic 模型的 warmup 步数； +- `--megatron-config-path`：通过 YAML 对 actor / critic 分别覆盖 Megatron 参数，例如为 critic 单独设置 `load`、`save`、`lr` 或 warmup 参数； - `--num-critic-only-steps`：训练开始时只训练 critic 的步数； - `--eps-clip`：PPO clip 范围； - `--value-clip`：value loss 的 clip 范围；