Skip to content

[CODE] Make the binary reward a true 0/1 success indicator - #542

Merged
natolambert merged 2 commits into
natolambert:mainfrom
albertlieyingadrian:code/fix-binary-reward-partial-credit
Nov 15, 2026
Merged

natolambert merged 2 commits into
natolambert:mainfrom
albertlieyingadrian:code/fix-binary-reward-partial-credit

Conversation

@albertlieyingadrian

Copy link
Copy Markdown
Contributor

Fixes #541

What's wrong

compute_rewards builds binary as correctness * (format == 1.0). The Experience docstring says it is "1.0 iff correctness and format are both maximal, else 0.0", but the product keeps the environment's fractional score. Reasoning Gym's spell_backward gives partial credit for wrong answers (fraction of matching characters), so a well-formatted near miss gets binary ≈ 0.83.

Two consequences:

  • loss: maxrl computes advantages from binary, so near misses get a positive advantage and are reinforced.
  • avg_binary (logged for every loss, and what the README and Chapter 6 exercises tell readers to watch) overstates the success rate.

Repro

The script in #541, on main and on this branch (CPU only, no model):

# main
answer: gnorwa | near miss: gnorwx
correctness: [1.0, 0.8333333134651184, 0.0, 0.8333333134651184]
binary:      [1.0, 0.8333333134651184, 0.0, 0.8333333134651184]
maxrl adv:   [0.5000001192092896, 0.2500000596046448, -1.0, 0.2500000596046448]

# this branch
correctness: [1.0, 0.8333333134651184, 0.0, 0.8333333134651184]
binary:      [1.0, 0.0, 0.0, 0.0]
maxrl adv:   [3.0, -1.0, -1.0, -1.0]

The fix

binary is now float(c == 1.0 and f == 1.0) computed on the raw Python scores, before the float32 cast. The raw comparison matters: a correct reversal with trailing junk (gnorwa.) scores 0.9999999999999999 for 6-letter words, which float32 rounds to exactly 1.0.

total, correctness, format, penalty, the key order of the rewards TensorDict, and every non-MaxRL advantage are unchanged.

Verification

  • New tests/test_binary_reward.py (3 tests, no model download): all fail on main, pass here.
  • uv run pytest tests: 52 passed.
  • I have not re-run a full MaxRL training run (no CUDA box at hand), so I can't say how much the reference curve moves. The changelog entry flags the comparability change for loss: maxrl and avg_binary.

Pre-Submit PR Report

Automated Checks

Check Status Details
Ruff lint PASS uvx ruff@0.14.5 check .
Ruff format PASS uvx ruff@0.14.5 format --check . (40 files already formatted)
Changelog PASS entry under ## Unreleased in code/CHANGELOG.md

Code Review

A review pass on the first draft raised two points, both addressed here: threshold before the float32 cast (the 0.9999999999999999 case above, now covered by a test), and don't reuse the DAPO-specific accuracy_max_reward knob to define success.

Verdict: READY FOR PR

@natolambert would you mind taking a look when you have a moment? Happy to adjust, for example if you would rather keep a fractional signal under a separate key.

compute_rewards multiplied correctness by the format check, so partial-credit
scores from Reasoning Gym (e.g. spell_backward near misses) leaked into
rewards["binary"]. MaxRL then gave well-formatted wrong answers a positive
advantage and avg_binary overstated the success rate.

Threshold the raw scores instead and add regression tests.

Fixes natolambert#541
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T17:47:38.840981Z 57cae0a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 57cae0a6a0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

# Reasoning Gym scorers can give partial credit (e.g. spell_backward returns the fraction
# of matching characters), so ``binary`` thresholds correctness instead of multiplying by
# it. Compare the raw scores: float32 would round a 0.9999999999999999 near miss to 1.0.
binary = [float(c == 1.0 and f == 1.0) for c, f in zip(correctness, format, strict=True)]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the stale MaxRL reward formula

When a completion receives partial correctness, this line now deliberately emits zero, but MaxRLLoss still documents the reward as r = correctness * format in policy_gradients/loss.py:195-196, which says the same completion receives a fractional reward. Update that formula to show the equality/indicator threshold so the algorithm's in-repo description matches the new training semantics.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b4b5f42: the MaxRLLoss docstring now says r = 1 if correctness == 1 and format == 1 else 0.

@albertlieyingadrian

Copy link
Copy Markdown
Contributor Author

Updated the MaxRLLoss docstring in loss.py to match the thresholded reward (r = 1 if correctness == 1 and format == 1 else 0).

@natolambert

Copy link
Copy Markdown
Owner

This looks super reasonable to me, thanks, merging.

@natolambert
natolambert merged commit 0e2cac1 into natolambert:main Nov 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CODE] binary reward is fractional under partial-credit scoring, so MaxRL reinforces near misses

2 participants