Repository navigation
Proposal: Cornell CS 4789 guest lecture — policy gradients at the frontier - #535
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…score centering Draft deck covering policy-gradient code for LLMs, the math of training-inference mismatch, the score-centering derivation, StableReinforce vs GRPO, Martin Marek's gradient reverse-engineering thread, and Ripple-65B's use of score centering. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52e81690f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # advantages: (B*G, 1), broadcast to every token | ||
| per_token_loss = -advantages * (learner_logprobs - centering) | ||
| loss = ((per_token_loss * mask).sum(1) / mask.sum(1)).mean() |
There was a problem hiding this comment.
Preserve the sequence-sum reduction when adding score centering
For variable-length completions, dividing by mask.sum(1) changes the estimator from the sequence-summed REINFORCE update derived earlier to an inverse-length-weighted update. Thus, even when sampler and learner agree and the centering correction has zero gradient, this code no longer reproduces the earlier loss: a completion with twice as many tokens receives half the per-token weight. Remove the length division to demonstrate score centering as the advertised single additional term, or explicitly explain and consistently derive the different weighting.
Useful? React with 👍 / 👎.
| # 4. Update, then sync the new weights back to the sampler | ||
| (-(advantages * seq_logprobs).mean()).backward() | ||
| optimizer.step() |
There was a problem hiding this comment.
Clear gradients in the end-to-end training step
When this end-to-end step is repeated, backward() accumulates into the gradients left by the preceding iteration because optimizer.step() does not clear them. Consequently, later updates include previous batches' gradients rather than just the current REINFORCE estimate, unintentionally changing the training algorithm. Include optimizer.zero_grad() before the backward pass or after the optimizer step.
Useful? React with 👍 / 👎.
- Open with RL-vs-pretraining compute (SemiAnalysis chart; DeepSeek R1, Beam, Inkling 2 Small GPU-hours with parameter counts) - Reuse lec3/lec4 slides for policy gradients in code and async RL; split the sync/async figure into three step slides plus the comparison - Add the surrogate-loss step (loss whose gradient is the PG estimate) and an autodiff slide; add an agentic-RL MDP slide - Add score-centering training curves from Marek & Ryabinin (30B quantization; quantization vs. staleness) and a TIS-in-code slide - Notation: x for prompt, xi for score, with a book/lectures column - Thank-you slide with course thumbnails and links - Bump colloquium pin for show_animations (natolambert/colloquium#76); deck sets show_animations: false while editing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…definition - RL review + RLVR + agentic RL diagram slides around the MDP tables - Long-tail generation (static vs. continuous batching, Olmo 3 numbers) and a PipelineRL pointer slide for inflight updates - Surrogate-loss and autodiff slides; drop undefined g-hat - Score definition slide before the zero-mean recall - Move the per-token/sequence KL mismatch slides to extra slides - Book and lecture links for loss aggregation; sync/async text column Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05c6b43f33
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| <!-- step --> | ||
|
|
||
| Substitute the score from earlier, $\boldsymbol{\xi}_v = \nabla_\theta f_v - \sum_u p_u \nabla_\theta f_u$: |
There was a problem hiding this comment.
Derive the logit-space score before substituting it
The slide refers to a score formula “from earlier,” but the preceding slides only define the score as the gradient of a log-probability; they never derive its expansion in terms of logits. This leaves out the softmax differentiation needed to explain both the probability-gap formula here and the subsequent three-token example. Show the intermediate log-softmax expression and its derivative before this substitution. This also violates teach/AGENTS.md's explicit requirement not to skip algebraic steps in teaching derivations.
AGENTS.md reference: teach/AGENTS.md:L31-L31
Useful? React with 👍 / 👎.
…tion - Add R3 (rollout routing replay) slide with Figure 1, and the IS-correction methods table from the score centering paper's appendix - Simplify the score centering section to match Ch. 6: recall slide defining the score, shorthand notes for p_v/q_v, drift described as in the book, centered gradient tied back to the policy gradient - Replace the GRPO vs. StableReinforce table with GRPO/RLOO advantage math - Move the worked example, drift-compounding, group-baseline, loss-derivation and top-k slides to static extra slides - Link blog posts, docs and X threads inline instead of via BibTeX - Add teach/scripts/check_slide_overflow.py to flag slides whose content overflows the footer, overlaps the citation, or has broken images - Turn animations back on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Adds the outline for Lecture 24 (Wed 11/18) of Sarah Dean's CS 4789 Intro to RL. This PR has the outline only, no slides yet:
teach/Cornell-CS4789-2026/README.md.Arc
Required sources
Notes
Open questions (also listed at the end of the file)
🤖 Generated with Claude Code