Skip to content

Proposal: Cornell CS 4789 guest lecture — policy gradients at the frontier - #535

Merged
natolambert merged 6 commits into
mainfrom
cornell-cs4789-guest-lecture
Nov 11, 2026
Merged

natolambert merged 6 commits into
mainfrom
cornell-cs4789-guest-lecture

Conversation

@natolambert

Copy link
Copy Markdown
Owner

Summary

Adds the outline for Lecture 24 (Wed 11/18) of Sarah Dean's CS 4789 Intro to RL. This PR has the outline only, no slides yet: teach/Cornell-CS4789-2026/README.md.

Arc

  1. The students already know every piece. The course manuscript covers REINFORCE, baselines, off-policy importance weights and PPO-clip, and its Exercise 3 is GRPO.
  2. Here is how those pieces become post-training. The slot fits well: SFT is imitation learning (the previous lecture, L23) and reward models are inverse RL (the next one, L25).
  3. A frontier failure: RL for LLMs is off-policy by accident, because sampler ≠ learner even with the same weights.
  4. Score centering fixes it by applying a baseline to the score instead of the reward.
  5. A rare confirmation: Tinker's StableReinforce, reverse-engineered from gradients, turns out to be REINFORCE + score centering.

Required sources

Notes

  • The outline uses the course's notation ($x, a, \pi_\theta, J(\theta), b(x)$).
  • Every reused slide is tagged with its source deck and exact slide title. All titles and asset filenames were checked to exist.

Open questions (also listed at the end of the file)

  • Slot length.
  • Whether to keep the reward-model teaser.
  • Whether to do a live demo.
  • Permission for the thread figures.
  • I couldn't find the "extremely stable training" quote Marek attributes to the Tinker docs on the StableReinforce docs page.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T20:14:12.933301Z 65f6556 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

natolambert and others added 2 commits November 5, 2026 09:59
…score centering

Draft deck covering policy-gradient code for LLMs, the math of
training-inference mismatch, the score-centering derivation, StableReinforce
vs GRPO, Martin Marek's gradient reverse-engineering thread, and Ripple-65B's
use of score centering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52e81690f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread teach/Cornell-CS4789-2026/talk.md Outdated

# advantages: (B*G, 1), broadcast to every token
per_token_loss = -advantages * (learner_logprobs - centering)
loss = ((per_token_loss * mask).sum(1) / mask.sum(1)).mean()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the sequence-sum reduction when adding score centering

For variable-length completions, dividing by mask.sum(1) changes the estimator from the sequence-summed REINFORCE update derived earlier to an inverse-length-weighted update. Thus, even when sampler and learner agree and the centering correction has zero gradient, this code no longer reproduces the earlier loss: a completion with twice as many tokens receives half the per-token weight. Remove the length division to demonstrate score centering as the advertised single additional term, or explicitly explain and consistently derive the different weighting.

Useful? React with 👍 / 👎.

Comment on lines +219 to +221
# 4. Update, then sync the new weights back to the sampler
(-(advantages * seq_logprobs).mean()).backward()
optimizer.step()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear gradients in the end-to-end training step

When this end-to-end step is repeated, backward() accumulates into the gradients left by the preceding iteration because optimizer.step() does not clear them. Consequently, later updates include previous batches' gradients rather than just the current REINFORCE estimate, unintentionally changing the training algorithm. Include optimizer.zero_grad() before the backward pass or after the optimizer step.

Useful? React with 👍 / 👎.

- Open with RL-vs-pretraining compute (SemiAnalysis chart; DeepSeek R1,
  Beam, Inkling 2 Small GPU-hours with parameter counts)
- Reuse lec3/lec4 slides for policy gradients in code and async RL; split
  the sync/async figure into three step slides plus the comparison
- Add the surrogate-loss step (loss whose gradient is the PG estimate) and
  an autodiff slide; add an agentic-RL MDP slide
- Add score-centering training curves from Marek & Ryabinin (30B
  quantization; quantization vs. staleness) and a TIS-in-code slide
- Notation: x for prompt, xi for score, with a book/lectures column
- Thank-you slide with course thumbnails and links
- Bump colloquium pin for show_animations (natolambert/colloquium#76);
  deck sets show_animations: false while editing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…definition

- RL review + RLVR + agentic RL diagram slides around the MDP tables
- Long-tail generation (static vs. continuous batching, Olmo 3 numbers)
  and a PipelineRL pointer slide for inflight updates
- Surrogate-loss and autodiff slides; drop undefined g-hat
- Score definition slide before the zero-mean recall
- Move the per-token/sequence KL mismatch slides to extra slides
- Book and lecture links for loss aggregation; sync/async text column

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05c6b43f33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread teach/Cornell-CS4789-2026/talk.md Outdated

<!-- step -->

Substitute the score from earlier, $\boldsymbol{\xi}_v = \nabla_\theta f_v - \sum_u p_u \nabla_\theta f_u$:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Derive the logit-space score before substituting it

The slide refers to a score formula “from earlier,” but the preceding slides only define the score as the gradient of a log-probability; they never derive its expansion in terms of logits. This leaves out the softmax differentiation needed to explain both the probability-gap formula here and the subsequent three-token example. Show the intermediate log-softmax expression and its derivative before this substitution. This also violates teach/AGENTS.md's explicit requirement not to skip algebraic steps in teaching derivations.

AGENTS.md reference: teach/AGENTS.md:L31-L31

Useful? React with 👍 / 👎.

…tion

- Add R3 (rollout routing replay) slide with Figure 1, and the IS-correction
  methods table from the score centering paper's appendix
- Simplify the score centering section to match Ch. 6: recall slide defining
  the score, shorthand notes for p_v/q_v, drift described as in the book,
  centered gradient tied back to the policy gradient
- Replace the GRPO vs. StableReinforce table with GRPO/RLOO advantage math
- Move the worked example, drift-compounding, group-baseline, loss-derivation
  and top-k slides to static extra slides
- Link blog posts, docs and X threads inline instead of via BibTeX
- Add teach/scripts/check_slide_overflow.py to flag slides whose content
  overflows the footer, overlaps the citation, or has broken images
- Turn animations back on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@natolambert
natolambert merged commit 1b40790 into main Nov 11, 2026
1 check passed

This branch was successfully deployed

1 active deployment
preview — 65f65568 Deployed Nov 9, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant