Repository navigation
[CODE] Make the binary reward a true 0/1 success indicator - #542
natolambert merged 2 commits into
Conversation
compute_rewards multiplied correctness by the format check, so partial-credit scores from Reasoning Gym (e.g. spell_backward near misses) leaked into rewards["binary"]. MaxRL then gave well-formatted wrong answers a positive advantage and avg_binary overstated the success rate. Threshold the raw scores instead and add regression tests. Fixes natolambert#541
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57cae0a6a0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # Reasoning Gym scorers can give partial credit (e.g. spell_backward returns the fraction | ||
| # of matching characters), so ``binary`` thresholds correctness instead of multiplying by | ||
| # it. Compare the raw scores: float32 would round a 0.9999999999999999 near miss to 1.0. | ||
| binary = [float(c == 1.0 and f == 1.0) for c, f in zip(correctness, format, strict=True)] |
There was a problem hiding this comment.
Correct the stale MaxRL reward formula
When a completion receives partial correctness, this line now deliberately emits zero, but MaxRLLoss still documents the reward as r = correctness * format in policy_gradients/loss.py:195-196, which says the same completion receives a fractional reward. Update that formula to show the equality/indicator threshold so the algorithm's in-repo description matches the new training semantics.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in b4b5f42: the MaxRLLoss docstring now says r = 1 if correctness == 1 and format == 1 else 0.
|
Updated the |
|
This looks super reasonable to me, thanks, merging. |
Fixes #541
What's wrong
compute_rewardsbuildsbinaryascorrectness * (format == 1.0). TheExperiencedocstring says it is "1.0 iff correctness and format are both maximal, else 0.0", but the product keeps the environment's fractional score. Reasoning Gym'sspell_backwardgives partial credit for wrong answers (fraction of matching characters), so a well-formatted near miss getsbinary ≈ 0.83.Two consequences:
loss: maxrlcomputes advantages frombinary, so near misses get a positive advantage and are reinforced.avg_binary(logged for every loss, and what the README and Chapter 6 exercises tell readers to watch) overstates the success rate.Repro
The script in #541, on
mainand on this branch (CPU only, no model):The fix
binaryis nowfloat(c == 1.0 and f == 1.0)computed on the raw Python scores, before the float32 cast. The raw comparison matters: a correct reversal with trailing junk (gnorwa.) scores0.9999999999999999for 6-letter words, which float32 rounds to exactly1.0.total,correctness,format,penalty, the key order of therewardsTensorDict, and every non-MaxRL advantage are unchanged.Verification
tests/test_binary_reward.py(3 tests, no model download): all fail onmain, pass here.uv run pytest tests: 52 passed.loss: maxrlandavg_binary.Pre-Submit PR Report
Automated Checks
uvx ruff@0.14.5 check .uvx ruff@0.14.5 format --check .(40 files already formatted)## Unreleasedincode/CHANGELOG.mdCode Review
A review pass on the first draft raised two points, both addressed here: threshold before the float32 cast (the
0.9999999999999999case above, now covered by a test), and don't reuse the DAPO-specificaccuracy_max_rewardknob to define success.Verdict: READY FOR PR
@natolambert would you mind taking a look when you have a moment? Happy to adjust, for example if you would rather keep a fractional signal under a separate key.