Skip to content

Fix partial rollout off-policy loss mask alignment#2207

Open
chengcuiping wants to merge 1 commit into
THUDM:mainfrom
chengcuiping:fix/partial-rollout-loss-mask
Open

Fix partial rollout off-policy loss mask alignment#2207
chengcuiping wants to merge 1 commit into
THUDM:mainfrom
chengcuiping:fix/partial-rollout-loss-mask

Conversation

@chengcuiping

Copy link
Copy Markdown

Summary

  • Add a small Sample.mask_response_tokens() helper that preserves response-side loss_mask length invariants.
  • Use it when partial rollout masks previous off-policy response tokens.
  • Add CPU regression coverage for partial resume appending new trainable tokens after masking old tokens.

Motivation

When partial rollout is combined with off-policy masking, old generated tokens are masked to zero before resumed generation appends new response tokens. If response_length grows without loss_mask staying aligned, training conversion can fail with len(sample.loss_mask) == sample.response_length, as reported in #1440.

This is a small stability step for the partial rollout / off-policy trajectory work discussed in #1800.

Behavior

  • Old partial response tokens remain masked to 0.
  • Newly resumed response tokens append trainable 1s through append_response_tokens().
  • len(loss_mask) == response_length is preserved.
  • Non-partial rollout behavior is unchanged.
  • No RL algorithm, KV cache, routing, or schema changes.

Testing

  • CUDA_VISIBLE_DEVICES='' python -m pytest tests/test_partial_rollout_loss_mask.py tests/test_sample.py -m unit
  • CUDA_VISIBLE_DEVICES='' pre-commit run --files slime/utils/types.py slime/rollout/sglang_rollout.py tests/test_partial_rollout_loss_mask.py
  • git diff --check

All passed.

tests/test_rollout_metrics.py could not be collected in this environment because the installed sglang package is missing GPU_MEMORY_TYPE_CUDA_GRAPH from sglang.srt.constants.

Refs #1440, #1800.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants