diff --git a/pyproject.toml b/pyproject.toml index b84a4031..332d19a0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,6 +20,6 @@ skills = [ teach = [ # Pinned to a commit: uv.lock is gitignored, so tracking colloquium main # let an upstream change break unrelated deploys. This is colloquium main - # after the footnote-citation fix (natolambert/colloquium#70); bump deliberately. - "colloquium @ git+https://github.com/natolambert/colloquium.git@c8d4b1c9e41b3f6a14df074b91ae3568be2c4e94", + # with the show_animations flag (natolambert/colloquium#76); bump deliberately. + "colloquium @ git+https://github.com/natolambert/colloquium.git@440b85e43348bbe3a422745e2294139d3658c449", ] diff --git a/teach/Cornell-CS4789-2026/README.md b/teach/Cornell-CS4789-2026/README.md new file mode 100644 index 00000000..bbcebb42 --- /dev/null +++ b/teach/Cornell-CS4789-2026/README.md @@ -0,0 +1,153 @@ +# Cornell CS 4789/5789 — Guest Lecture (2026) + +**Course:** [CS 4789/5789: Introduction to Reinforcement Learning](https://sdean.website/cs4789.html), Sarah Dean, Cornell +**Slot:** Lecture 24, Wednesday 18 November 2026 (listed as "Guest Lecture") +**Neighbors:** L23 Imitation Learning (11/16) → **L24 (this lecture)** → L25 Inverse RL (11/23) → L26 Specification and Implications → L27 Case Study +**Format:** ~60–70 minutes of slides + questions (slot length TBC) +**Audience:** undergraduate / MEng intro-RL students who have derived REINFORCE, baselines, off-policy importance weighting, KL trust regions, and PPO-clip from the course manuscript +**Status:** slides in progress in `talk.md` (LLM policy-gradient code, training-inference mismatch, score centering, StableReinforce). The rest of the outline below is not built yet. + +## Talk: Policy gradients at the frontier — RL for language models + +*From REINFORCE to post-training, and the bugs that only show up at scale.* + +**Abstract.** Every piece of the RL recipe used to post-train today's frontier language models is something this class has already derived: a contextual-bandit policy, the REINFORCE estimator, a baseline, importance weights, and a KL trust region. This lecture assembles those pieces into the modern post-training pipeline — supervised finetuning (imitation learning, last lecture), reward models learned from preferences (inverse RL, next lecture), and RL with verifiable rewards — and then follows them to a live frontier problem: RL for LLMs is *off-policy by accident*, because the inference engine that samples and the trainer that computes gradients disagree numerically even with identical weights. The fix proposed in *Score Centering Stabilizes Off-policy Reinforcement Learning* (Marek & Ryabinin, Together AI, 2026) is the baseline idea from this course applied to the score instead of the reward. Five weeks after the paper appeared, Thinking Machines shipped a "StableReinforce" loss in Tinker, and the paper's first author reverse-engineered it from gradients alone and showed it is REINFORCE + score centering — a rare public confirmation of a piece of frontier RL science. + +### Why this slot works + +The post-training pipeline lines up with the course's own ordering: + +| CS 4789 | Post-training analogue | +|---|---| +| L19–20 Contextual bandits | An LLM is a contextual-bandit policy: $x$ = prompt, $a$ = completion | +| L3–5, L16–17 Policy gradients, trust regions | REINFORCE / RLOO / GRPO / PPO, KL to a reference model | +| L23 Imitation learning | Supervised finetuning (instruction tuning) | +| **L24 (this lecture)** | RL on language models, and what breaks at scale | +| L25 Inverse RL | Reward models learned from human preferences | + +### Notation + +Use the course manuscript's notation (`rl-manuscript-policyopt.pdf`), not the book's: context $x \sim D_x$, action $a \sim \pi_\theta(\cdot \mid x)$, reward $r \sim R(x, a)$, objective $J(\theta)$, baseline $b(x)$, importance weight $w = \pi_\theta(a \mid x) / \pi_{\theta'}(a \mid x)$. For language, $a = (a_1, \dots, a_T)$ and $\pi_\theta(a \mid x) = \prod_t \pi_\theta(a_t \mid x, a_{ ""` means lift with light edits from `teach/course/`; `adapt` means rework into the course notation; `new` means write from scratch. + +### Open (3) + +1. **Title** — "Policy gradients at the frontier: RL for language models" / "Guest lecture, CS 4789 — Lecture 24". `new` (frontmatter cloned from `teach/SALA-2026/talk.md`, eyebrow "Cornell CS 4789"). +2. **Framing question** (section-break) — *You've derived REINFORCE for a movie recommender. What changes when the policy is a 30B-parameter language model?* `new` +3. **Where today sits** — the L23 → L24 → L25 strip from the table above: SFT ↔ imitation learning, reward model ↔ inverse RL, and today is the RL in the middle. `new` + +### Part 1 — The LLM as a contextual-bandit policy (~8) + +4. Section break. +5. **The mapping** — context $x$ = prompt, action $a$ = full completion, $\pi_\theta(a \mid x) = \prod_t \pi_\theta(a_t \mid x, a_{` frames. **Plant this identity — Part 3 hinges on it.** `adapt` from lec3 "Baselines keep the estimator unbiased" (3 slides → 1 stepped slide). +15. **RLOO and GRPO are choices of $b(x)$** — leave-one-out mean vs. group mean + std normalization; GRPO = the manuscript's Exercise 3 estimator $\hat\nabla J_{\text{GRPO}}$. Answer Exercise 3's question: the std division makes it biased. `reuse: lec3 "RLOO: Leave-one-out baseline"`, `lec3 "GRPO advantage"`. +16. **RLOO worked example** — four completions, rewards, baselines, advantages. `reuse: lec3 "RLOO worked example"` +17. **The manuscript's PPO margin note → the real clipped objective** — "discard datapoints once $w_i > 1+\epsilon$ for positive advantage …" becomes $\min(w A, \mathrm{clip}(w, 1\pm\epsilon) A)$. `reuse: lec3 "PPO core idea 2: Importance sampling"`, `lec3 "The PPO clipped objective (put pieces together)"`. +18. **Clipping visualized** — `reuse: lec3 "PPO clipping visualized"` (`ppo-viz-4x.png`). +19. **Two KLs, don't confuse them** — the manuscript's trust region is KL between successive iterates ($\pi_{\theta'}$ vs. $\pi_\theta$); RLHF adds a KL to a frozen *reference* model. `reuse: lec3 "Two "old" policies — don't confuse them"` + `lec10 "A KL penalty to the reference model controls reward"`. +20. **Why no value function** — PPO needs a critic the size of the policy (four models in memory); the group baseline replaced it for reasoning RL. `adapt` from lec3 "GRPO vs PPO". +21. **Loss aggregation** — per-sequence vs. per-token normalization changes the effective weighting of long completions. `reuse: lec4 "Loss aggregation strategies"` (one slide, keep light). +22. **Part 2 recap** — every algorithm is $\nabla \log \pi_\theta \times$ (some centered reward), optionally importance-weighted and clipped. `new` + +### Part 3 — Off-policy by accident (~12) + +23. Section break: "Off-policy by accident". +24. **The systems picture** — generation runs in an inference engine (vLLM / SGLang), gradients in a training framework (FSDP etc.), often asynchronously on separate GPUs. `reuse: lec4 "Synchronous vs. asynchronous training"` (`async_v_synch_rl.png`) + `lec4 "On-policy vs. off-policy"`. +25. **Same weights, different distribution** — different kernels, precision, batch composition, and quantized KV caches mean the sampler $q$ and learner $p_\theta$ disagree even at identical $\theta$. MiniMax-M1's train-vs-inference token-probability plot. `reuse: lec5 "MiniMax-M1 (Jun. 16, 2025): A fun RL paper!"` (`minimax-m1-logprob-mismatch.png`); cite Yao et al. on off-policy RL frameworks and the Thinking Machines batch-invariance post (making the engines agree exactly is possible but costs throughput). +26. **Your off-policy fix: importance weights** — manuscript §1.3.2: $\hat\nabla J = \frac{1}{n}\sum_i w_i \nabla \log \pi_\theta(a_i \mid x_i)\, r_i$; unbiased but high-variance, and over a 1,000-token completion the sequence ratio explodes. `new` +27. **What the field does: truncate, mask, clip** — TIS caps $w$ at $C$ (one-sided, e.g. $C=2$); MIS / IcePop masks tokens outside a ratio band; PPO/DAPO clip; DPPO masks by an estimated TV divergence. All trade bias for variance. `new` (TIS from `book/chapters/06-policy-gradients.md` §"Truncated Importance Sampling") + `reuse: lec10 "A trust region on the sampling distribution"` (`dppo-ratio-vs-tv.png`). +28. **Drift from the sampler is now the binding failure** — in our TMax terminal-agent work, instabilities came from drift relative to the *sampler* (long multi-turn rollouts, async, engine mismatch), not drift from init. `reuse: lec10 "2026: The trust region / KL pen. is moving on"` (trim to the TMax bullet). +29. **Back to the identity** — on-policy, at each prefix $\mathbb{E}_{v \sim p}[s_v] = 0$. Off-policy, the expected score under the sampler is $\bar s = \sum_v q_v\, s_v \neq 0$. "The fact that made your baselines free is gone." `new` +30. **The decomposition** (``, one manipulation per frame) — start from the per-token update $\mathbb{E}_q[R\, s_{a_t}]$; apply $\mathbb{E}[AB] = \mathbb{E}[A]\,\mathbb{E}[B] + \mathrm{Cov}(A,B)$; substitute $\mathbb{E}_q[s_{a_t}] = \bar s$; label the two terms: + $$\mathbb{E}_q[R\, s_{a_t}] = \underbrace{\mathbb{E}_q[R]\,\bar s}_{\text{drift}} + \underbrace{\mathrm{Cov}_q(R,\, s_{a_t})}_{\text{signal}}$$ + Only the covariance term carries information about which tokens earned reward. `new` +31. **What drift does** — $\bar s = \sum_v q_v \nabla \log p_v$ is the gradient of $-\mathrm{CE}(q, p_\theta)$, so the drift term is *cross-entropy distillation toward the sampler*, scaled by the mean reward at that prefix. The sampler is then refreshed from the learner, so the bias compounds every step rather than converging (unlike offline distillation from a fixed teacher). `new` +32. **The surprising consequence** — online, all-positive $\{+1, 0\}$ rewards are the *least* stable setting (no negative rewards to cancel drift); group-centered rewards (your GRPO baseline) are the most stable. Offline the ordering flips. Paper Figure 2 (Qwen3-1.7B, Countdown). `new` +33. **Score centering: a baseline for the score** — replace $s_{a_t}$ with $\tilde s_{a_t} = s_{a_t} - \bar s$: + $$\boxed{\mathbb{E}_q[R\, \tilde s_{a_t}] = \mathrm{Cov}_q(R,\, s_{a_t})}$$ + Drift cancels exactly, with no ratios, masks, or clipping; with no mismatch the correction is zero. Side-by-side with slide 14: course baselines center $r$; score centering centers $\nabla \log p$. `new` +34. **Making it cheap** — computing $\bar s$ needs the sampler's full next-token distribution; keep the sampler's top-$k$ ($k = 128$) probabilities and model the tail with the learner. As a loss: $L = -R\,\big(\log p_{a_t} - \sum_{v \in H} \mathrm{sg}[q_v - \rho\, p_v] \log p_v\big)$. <1% wall-clock overhead; composes additively with TIS/MIS (and vanilla, un-truncated IS already cancels drift — which is why truncation brings it back). `new` +35. **Results** — paper Figure 1 (Qwen3-30B-A3B-Base, INTELLECT-2 math, increasingly aggressive sampler quantization): with FP4 KV cache, uncorrected PG collapses within 200 steps while score centering reaches 52% (TIS 51%); with an INT8 sampler + INT4 KV cache, score centering reaches 30%, TIS 12%, everything else < 5% training accuracy. Optional second figure: Figure 3 weight-noise sweep (Qwen3-0.6B). `new`, `cite-right` the paper. + +### Part 4 — A rare confirmation of frontier RL science (~6) + +36. Section break: "Did a frontier lab use this?" +37. **Why we usually can't know** — frontier RL losses and infrastructure are unpublished; public papers rarely get confirmed or refuted by what labs actually run. `new` +38. **Tinker ships StableReinforce (21 Oct 2026)** — Thinking Machines' post-training API adds a REINFORCE variant "that keeps your RL runs stable, even when learning drifts from sampling." The docs expose the interface but not the loss: sampling must record extra "stability info" (`record_stability_info=True`), and there is an optional sampler–learner KL term with $k_1$ / $k_3$ estimators (callback to the course's KL estimators; `reuse: qa-02 "Reading: How to *estimate* KL in practice"` as a pointer). `new` (screenshot of the announcement + docs snippet). +39. **Reverse-engineering a loss from its gradients** (``) — Martin Marek's thread (25 Oct 2026): (1) Tinker has no SGD, so use Adam with $\epsilon = 10^4$, making the update $\approx -\text{lr}\cdot g/\epsilon$ and exposing the raw gradient; (2) train a rank-1 LoRA on the unembedding so the adapter gradient equals the logit gradient up to scale; (3) for a single-token update, only **8 logits** have nonzero gradient — plain REINFORCE touches all of them. `new` (thread figures, cited). +40. **The match** — hypothesis: REINFORCE + score centering with $\bar s$ estimated from 8 randomly sampled tokens (unbiased, higher variance — vs. the paper's deterministic, slightly biased top-$k$). Implemented as a custom Tinker loss, the gradients match StableReinforce within numeric precision across all layers. `new` (thread's gradient-match figure). +41. **The honest caveat** — this is a reverse-engineering by the paper's own first author, not a statement from Thinking Machines; the evidence is a gradient match, which is strong but not an admission. Still, it is the closest thing we get to seeing a public RL idea land inside a frontier system within weeks. `new` +42. **The lesson for this class** — the frontier problem lived entirely in intro-course primitives: the zero-mean score, baselines, importance weights, KL. `new` + +### Close (3) + +43. **Takeaways** — (1) an LLM is a contextual-bandit policy and post-training is IL → IRL → RL; (2) GRPO is your Exercise 3; (3) at scale every run is off-policy because sampler ≠ learner; (4) score centering restores $\mathbb{E}[\text{score}] = 0$, the property baselines rely on; (5) sometimes you can verify frontier practice from the outside. `new` +44. **Next time: inverse RL** — in LLMs, that's the reward model: learn $r_\phi$ from pairwise preferences with Bradley–Terry. One teaser figure. `reuse:` figure `pref_rm_training.png` (from lec2's reward-model section); no derivation. +45. **Resources + thank you** — rlhfbook.com (book + free course videos), the score-centering paper and code, Tinker docs, Marek's thread. `new` + +## Reuse inventory + +**From `teach/course/` decks** (titles verified against `##` headings): + +- lec1: "InstructGPT's 3-step RLHF recipe", "Comparing classical RL vs. LLM RLHF and RLVR", "o1: Training-time scaling (with reinforcement learning!)", "DeepSeek-R1-Zero: RL scaling" +- lec3: "Language models as episodic MDPs", "Baselines keep the estimator unbiased", "REINFORCE with baseline", "RLOO: Leave-one-out baseline", "RLOO worked example", "PPO core idea 2: Importance sampling", "The PPO clipped objective (put pieces together)", "PPO clipping visualized", "Two "old" policies — don't confuse them", "GRPO advantage", "GRPO vs PPO" +- lec4: "Bandit-style vs MDP-style RLHF", "Loss aggregation strategies", "On-policy vs. off-policy", "Synchronous vs. asynchronous training" +- lec5: "MiniMax-M1 (Jun. 16, 2025): A fun RL paper!" +- lec10: "A KL penalty to the reference model controls reward", "2026: The trust region / KL pen. is moving on", "A trust region on the sampling distribution" +- qa-02: "Reading: How to *estimate* KL in practice" + +**Figures to copy** from `teach/course/assets/` into `teach/Cornell-CS4789-2026/assets/`: `instructgpt.jpg`, `rlvr-system.png`, `o1-train-time.png`, `deepseek-r1-zero-figure1-training.png`, `olmo31-extended-rl.jpeg`, `ppo-viz-4x.png`, `async_v_synch_rl.png`, `minimax-m1-logprob-mismatch.png`, `dppo-ratio-vs-tv.png`, `pref_rm_training.png`. + +## New assets + +- Paper title page for arXiv 2609.20807 via the `paper-page-figure` skill (for slide 33 or 35). +- Score-centering paper Figures 1 and 2 (optionally 3), cropped from the arXiv HTML. +- Tinker announcement + StableReinforce docs snippet (slide 38). +- Marek thread figures: sparse logit gradient (slide 39) and gradient match (slide 40). Cite and link the thread; ask Martin before reusing figures. +- Optional: one TikZ diagram of the sampler/learner loop ($p_\theta$ trains → weights sync → $q$ samples → back to $p_\theta$), in `diagrams/tikz/`, following the repo's arrow and spacing conventions. + +## Bibliography + +Local `teach/Cornell-CS4789-2026/refs.bib`, copying existing keys from `teach/course/refs.bib` or `book/chapters/bib.bib`: + +- Already in `teach/course/refs.bib`: `williams1992simple`, `schulman2017proximal`, `shao2024deepseekmath`, `guo2025deepseek`, `minimax2025minimax_m1`, `noukhovitch2024asynchronous`, `ivison2026tmax`, `qi2026dppo`, `glm5team2026glm5`, `schulman2020klapprox` +- From `book/chapters/bib.bib`: `ionides2008truncated`, `yao2025offpolicy` +- New: `marek2026scorecentering` (arXiv 2609.20807), Thinking Machines batch-invariant inference post, `@misc` for the Tinker StableReinforce docs, `@misc` for Marek's thread (x.com/mrtnm/status/2114145574987702525) + +## Open questions + +1. Slot length (50 vs. 75 minutes) — at 50 minutes, cut Part 2 to slides 13–17 + 19 and drop slide 35's second figure. +2. Should the reward-model teaser (slide 44) stay, or leave preferences entirely to L25? +3. Live demo? Options: a tiny score-centering vs. vanilla REINFORCE run from `code/`, or a Tinker notebook. +4. Ask Martin Marek for permission to use the thread figures, and whether Thinking Machines has said anything publicly since. +5. ~~Verify the "extremely stable training" phrase Marek quotes from the Tinker docs.~~ It is on the StableReinforce docs page as of 5 Nov 2026. + +## Sources + +- Course page and manuscript: https://sdean.website/cs4789.html, https://sdean.website/rl-manuscript-policyopt.pdf +- Marek & Ryabinin, *Score Centering Stabilizes Off-policy Reinforcement Learning*, arXiv:2609.20807 (17 Sep 2026); code: https://github.com/martin-marek/score-centering +- Tinker StableReinforce docs: https://tinker-docs.thinkingmachines.ai/tinker/losses/stable-reinforce/ +- Martin Marek, reverse-engineering thread (25 Oct 2026): https://x.com/mrtnm/status/2114145574987702525 diff --git a/teach/Cornell-CS4789-2026/assets/async_v_synch_rl.png b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl.png new file mode 100644 index 00000000..8578516a Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl.png differ diff --git a/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_1.png b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_1.png new file mode 100644 index 00000000..4780eee0 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_1.png differ diff --git a/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_2.png b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_2.png new file mode 100644 index 00000000..d6d001f0 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_2.png differ diff --git a/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_3.png b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_3.png new file mode 100644 index 00000000..7b4b5e1d Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/async_v_synch_rl_3.png differ diff --git a/teach/Cornell-CS4789-2026/assets/distributed-rl.png b/teach/Cornell-CS4789-2026/assets/distributed-rl.png new file mode 100644 index 00000000..3e3afb18 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/distributed-rl.png differ diff --git a/teach/Cornell-CS4789-2026/assets/marek-thread-gradient-match.png b/teach/Cornell-CS4789-2026/assets/marek-thread-gradient-match.png new file mode 100644 index 00000000..36a5e6ed Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/marek-thread-gradient-match.png differ diff --git a/teach/Cornell-CS4789-2026/assets/marek-thread-logit-gradients.jpeg b/teach/Cornell-CS4789-2026/assets/marek-thread-logit-gradients.jpeg new file mode 100644 index 00000000..e87ad745 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/marek-thread-logit-gradients.jpeg differ diff --git a/teach/Cornell-CS4789-2026/assets/marek-thread-lora.png b/teach/Cornell-CS4789-2026/assets/marek-thread-lora.png new file mode 100644 index 00000000..8e846bcb Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/marek-thread-lora.png differ diff --git a/teach/Cornell-CS4789-2026/assets/minimax-m1-logprob-mismatch.png b/teach/Cornell-CS4789-2026/assets/minimax-m1-logprob-mismatch.png new file mode 100644 index 00000000..f804c7c9 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/minimax-m1-logprob-mismatch.png differ diff --git a/teach/Cornell-CS4789-2026/assets/olmo3-continuous-batching.png b/teach/Cornell-CS4789-2026/assets/olmo3-continuous-batching.png new file mode 100644 index 00000000..6c515102 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/olmo3-continuous-batching.png differ diff --git a/teach/Cornell-CS4789-2026/assets/olmo3-static-batching.png b/teach/Cornell-CS4789-2026/assets/olmo3-static-batching.png new file mode 100644 index 00000000..4e59c554 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/olmo3-static-batching.png differ diff --git a/teach/Cornell-CS4789-2026/assets/pipelinerl-paper-2025.png b/teach/Cornell-CS4789-2026/assets/pipelinerl-paper-2025.png new file mode 100644 index 00000000..e68b0c5b Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/pipelinerl-paper-2025.png differ diff --git a/teach/Cornell-CS4789-2026/assets/post-training-course-thumbnails.jpeg b/teach/Cornell-CS4789-2026/assets/post-training-course-thumbnails.jpeg new file mode 100644 index 00000000..66bc5280 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/post-training-course-thumbnails.jpeg differ diff --git a/teach/Cornell-CS4789-2026/assets/r3-routing-replay.png b/teach/Cornell-CS4789-2026/assets/r3-routing-replay.png new file mode 100644 index 00000000..d9db7de6 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/r3-routing-replay.png differ diff --git a/teach/Cornell-CS4789-2026/assets/rl.png b/teach/Cornell-CS4789-2026/assets/rl.png new file mode 100644 index 00000000..500ea466 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/rl.png differ diff --git a/teach/Cornell-CS4789-2026/assets/rlvr-system.png b/teach/Cornell-CS4789-2026/assets/rlvr-system.png new file mode 100644 index 00000000..46796501 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/rlvr-system.png differ diff --git a/teach/Cornell-CS4789-2026/assets/score-centering-30b-quantization.png b/teach/Cornell-CS4789-2026/assets/score-centering-30b-quantization.png new file mode 100644 index 00000000..ce2bb9fa Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/score-centering-30b-quantization.png differ diff --git a/teach/Cornell-CS4789-2026/assets/score-centering-is-methods-table.png b/teach/Cornell-CS4789-2026/assets/score-centering-is-methods-table.png new file mode 100644 index 00000000..66814962 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/score-centering-is-methods-table.png differ diff --git a/teach/Cornell-CS4789-2026/assets/score-centering-quant-stale.png b/teach/Cornell-CS4789-2026/assets/score-centering-quant-stale.png new file mode 100644 index 00000000..b3e5feee Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/score-centering-quant-stale.png differ diff --git a/teach/Cornell-CS4789-2026/assets/semianalysis-compute-share.png b/teach/Cornell-CS4789-2026/assets/semianalysis-compute-share.png new file mode 100644 index 00000000..244b7d0e Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/semianalysis-compute-share.png differ diff --git a/teach/Cornell-CS4789-2026/assets/tool_use_rl_loop.png b/teach/Cornell-CS4789-2026/assets/tool_use_rl_loop.png new file mode 100644 index 00000000..d72f0824 Binary files /dev/null and b/teach/Cornell-CS4789-2026/assets/tool_use_rl_loop.png differ diff --git a/teach/Cornell-CS4789-2026/refs.bib b/teach/Cornell-CS4789-2026/refs.bib new file mode 100644 index 00000000..53eab4fe --- /dev/null +++ b/teach/Cornell-CS4789-2026/refs.bib @@ -0,0 +1,115 @@ +@article{ionides2008truncated, + title={Truncated importance sampling}, + author={Ionides, Edward L}, + journal={Journal of Computational and Graphical Statistics}, + volume={17}, + number={2}, + pages={295--311}, + year={2008} +} + +@misc{marek2026score, + title = {Score Centering Stabilizes Off-policy Reinforcement Learning}, + author = {Marek, Martin and Ryabinin, Max}, + year = {2026}, + eprint = {2609.20807}, + archivePrefix = {arXiv}, + primaryClass = {cs.LG}, + doi = {10.48550/arXiv.2609.20807}, + url = {https://arxiv.org/abs/2609.20807} +} + +@misc{minimax2025minimax_m1, + title={MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention}, + author={MiniMax}, + year={2025}, + eprint={2506.13585}, + archivePrefix={arXiv}, + primaryClass={cs.CL}, + doi={10.48550/arXiv.2506.13585}, + url={https://arxiv.org/abs/2506.13585}, +} + +@inproceedings{noukhovitch2024asynchronous, + title={Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models}, + booktitle = {International Conference on Learning Representations (ICLR)}, + author={Noukhovitch, Michael and Huang, Shengyi and Xhonneux, Sophie and Hosseini, Arian and Agarwal, Rishabh and Courville, Aaron}, + year = {2025} +} + +@article{shao2024deepseekmath, + title={Deepseekmath: Pushing the limits of mathematical reasoning in open language models}, + author={Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, YK and Wu, Y and others}, + journal={arXiv preprint arXiv:2402.03300}, + year={2024} +} + +@article{williams1992simple, + title={Simple statistical gradient-following algorithms for connectionist reinforcement learning}, + author={Williams, Ronald J}, + journal={Machine learning}, + volume={8}, + pages={229--256}, + year={1992}, + publisher={Springer} +} + +@article{guo2025deepseeknature, + title = {{DeepSeek-R1} incentivizes reasoning in {LLMs} through reinforcement learning}, + author = {Guo, Daya and Yang, Dejian and Zhang, Haowei and others}, + journal = {Nature}, + volume = {645}, + pages = {633--638}, + doi = {10.1038/s41586-025-09422-z}, + year = {2025} +} + +@misc{deepseekai2025deepseekv3technicalreport, + title = {{DeepSeek-V3} Technical Report}, + author = {{DeepSeek-AI}}, + url = {https://arxiv.org/abs/2412.19437}, + year = {2025} +} + +@inproceedings{Vaswani2017AttentionIA, + title = {Attention Is All You Need}, + author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, {\L}ukasz and Polosukhin, Illia}, + booktitle = {Advances in Neural Information Processing Systems 30 (NeurIPS 2017)}, + year = {2017}, + url = {https://papers.nips.cc/paper/7181-attention-is-all-you-need} +} + +@article{ivison2026tmax, + title = {{TMax}: A Simple Recipe for Terminal Agents}, + author = {Ivison, Hamish and Yin, Junjie Oscar and Shao, Rulin and Xiao, Teng and Lambert, Nathan and Hajishirzi, Hannaneh}, + journal = {arXiv preprint arXiv:2606.23321}, + year = {2026}, + eprint = {2606.23321}, + url = {https://arxiv.org/abs/2606.23321} +} + +@misc{teamolmo2025olmo3, + title = {Olmo 3}, + author = {Team Olmo and Allyson Ettinger and Amanda Bertsch and Bailey Kuehl and David Graham and David Heineman and Dirk Groeneveld and Faeze Brahman and Finbarr Timbers and Hamish Ivison and Jacob Morrison and Jake Poznanski and Kyle Lo and Luca Soldaini and Matt Jordan and Mayee Chen and Michael Noukhovitch and Nathan Lambert and Pete Walsh and Pradeep Dasigi and Robert Berry and Saumya Malik and Saurabh Shah and Scott Geng and Shane Arora and Shashank Gupta and Taira Anderson and Teng Xiao and Tyler Murray and Tyler Romero and Victoria Graf and Akari Asai and Akshita Bhagia and Alexander Wettig and Alisa Liu and Aman Rangapur and Chloe Anastasiades and Costa Huang and Dustin Schwenk and Harsh Trivedi and Ian Magnusson and Jaron Lochner and Jiacheng Liu and Lester James V. Miranda and Maarten Sap and Malia Morgan and Michael Schmitz and Michal Guerquin and Michael Wilson and Regan Huff and Ronan Le Bras and Rui Xin and Rulin Shao and Sam Skjonsberg and Shannon Zejiang Shen and Shuyue Stella Li and Tucker Wilde and Valentina Pyatkin and Will Merrill and Yapei Chang and Yuling Gu and Zhiyuan Zeng and Ashish Sabharwal and Luke Zettlemoyer and Pang Wei Koh and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi}, + year = {2025}, + eprint = {2512.13961}, + archiveprefix = {arXiv}, + primaryclass = {cs.CL}, + url = {https://arxiv.org/abs/2512.13961} +} + +@article{piche2025pipelinerl, + title = {{PipelineRL}: Faster On-policy Reinforcement Learning for Long Sequence Generation}, + author = {Piché, Alexandre and Kamalloo, Ehsan and Pardinas, Rafael and Chen, Xiaoyin and Bahdanau, Dzmitry}, + journal = {arXiv preprint arXiv:2509.19128}, + year = {2025}, + url = {https://arxiv.org/abs/2509.19128} +} + +@article{ma2025r3, + title = {Stabilizing {MoE} Reinforcement Learning by Aligning Training and Inference Routers}, + author = {Ma, Wenhan and Zhang, Hailin and Zhao, Liang and Song, Yifan and Wang, Yudong and Sui, Zhifang and Luo, Fuli}, + journal = {arXiv preprint arXiv:2510.11370}, + year = {2025}, + url = {https://arxiv.org/abs/2510.11370} +} diff --git a/teach/Cornell-CS4789-2026/talk.md b/teach/Cornell-CS4789-2026/talk.md new file mode 100644 index 00000000..ff952b43 --- /dev/null +++ b/teach/Cornell-CS4789-2026/talk.md @@ -0,0 +1,1189 @@ +--- +title: "From Classical RL to Frontier LLMs" +author: "Nathan Lambert" +date: "November 2026" +fonts: + heading: "Rubik" + body: "Poppins" +bibliography: refs.bib +figure_captions: true +show_animations: true +footer: + left: "rlhfbook.com" + center: "Cornell CS 4789" + right: "Lambert {n}/{N}" +custom_css: | + .slide--section-break { background: #F28482; } + :root { + --colloquium-progress-fill: #F28482; + } + .slide--title-sidebar h1 { + font-size: 2.5em; + letter-spacing: 0; + } + /* colloquium's .slide-content has flex: 1, so valign alone can't move it */ + .slide--content.valign-center > .slide-content { flex: 0 1 auto; } + .compute-table td { vertical-align: top; padding-top: 0.75em; padding-bottom: 0.75em; } + .compute-note { font-size: 0.7em; color: #6b7280; } + .shorthand-note { font-size: 0.7em; line-height: 1.2; color: #6b7280; margin-bottom: 0.2em; } + .col-text-sm { font-size: 0.85em; } + .diagram-md img { max-height: 430px; width: auto; } + .slide.diagram-sm .slide-content img { max-height: 400px; width: auto; } + .slide.diagram-xs .slide-content img { max-height: 330px; width: auto; } + .notation-table { font-size: 0.95em; } + .notation-table td, .notation-table th { padding-top: 0.35em; padding-bottom: 0.35em; } + .notation-table td:first-child, .notation-table th { white-space: nowrap; } + .mdp-table { font-size: 0.85em; } + .mdp-table td, .mdp-table th { padding-top: 0.3em; padding-bottom: 0.3em; } + .compute-table td strong { font-size: 1.15em; } + .compute-table td small { display: block; margin-top: 0.2em; font-size: 0.68em; line-height: 1.35; color: #6b7280; } + .compute-table td:last-child strong { font-size: 1.4em; color: #E8676A; } +--- + + + + +# From Classical RL to Frontier LLMs + +
18 November 2026
+ +
+

Nathan Lambert

+
+--- + +## Post-training (RL) now takes most of a frontier lab's compute + + + + + +![Share of combined Anthropic and OpenAI compute by workload, 1Q24 to 4Q26 (later quarters are estimates). Pretraining falls from 67% to 7%, while post-training and RL rise from 5% to 55%.](assets/semianalysis-compute-share.png) + +--- + +## How the times have changed! + +RL compute relative to pretraining compute for the same model, in GPU-hours: + +
+ +| Model | Pretraining | RL | RL / pretraining | +|---|---|---|---| +| **DeepSeek R1** Jan 2025
671B total, 37B active
| **2.66M** H800 GPU-hours [@deepseekai2025deepseekv3technicalreport] | **0.14M** H800 GPU-hours [@guo2025deepseeknature] | **~5%** | +| [**Beam**](https://reflection.ai/blog/introducing-beam) Oct 2026
501B total, 23B active
| **< 4.1M** 6,144 GB300s for under 4 weeks | **~7.1M** 10.5K GB300s for 4 weeks | **> 1.7×** | +| [**Inkling 2 Small**](https://thinkingmachines.ai/news/inkling-2-small) Nov 2026
594B total, 17B active
| **~0.11M\*** ~1.5K GPUs for under 3 days | **~0.44M** 2,048 GPUs for 9 days | **~4×\*** | + +
+ +
*Estimated: Thinking Machines report the pretraining duration but not the GPU count. GPU-hours ignore utilization. RL spends much of its time sampling, so its share of FLOPs is smaller than its share of GPU-hours.
+ +--- + + +## What this lecture covers + +How RL is approached with LLMs and what the challenges of RL are for frontier models. + +RL is mostly an engineering, stability, and throughput problem at the frontier. The learning signal is important, and mostly comes through data engineering (not the focus of this lecture). +--- + +## What this lecture covers + +How RL is approached with LLMs and what the challenges of RL are for frontier models. + +Some examples we include: + +1. **Policy gradients in LLM code.** How LLM log-probs, logits, and a surrogate loss turn REINFORCE into a few lines of PyTorch. +2. **Asynchronous RL systems.** How RL training is usually done for large LLMs. +3. **Training-inference mismatch.** A common challenge that many of the latest RL algorithms address. + +--- + +## Notation in this lecture + +
+ +| Symbol | Meaning | Book / lectures | +|--------|---------|-----------------| +| $x$ | Prompt | $s$ | +| $a = (a_1, \dots, a_T)$ | Completion, one token at a time | $a$ | +| $v \in \mathcal{V}$ | Any token in the vocabulary | $v$ | +| $p_\theta$ | **Learner**: the training framework's (e.g., FSDP) next-token distribution | $\pi_\theta^{\text{learner}}$ | +| $q$ | **Sampler**: the inference engine's (e.g., vLLM) next-token distribution (no gradient updates) | $\pi_\theta^{\text{sampler}}$ | +| $f_\theta$ | Logits, so $p_\theta = \text{softmax}(f_\theta)$ | | +| $\boldsymbol{\xi}_v = \nabla_\theta \log p_v$ | Gradient "**score**" of token $v$ | $\boldsymbol{\xi}_v$ | +| $A$ | Advantage for the whole completion, e.g., $r - b(x)$ | $A$ | + +
+ +$p_v$ and $q_v$ are shorthand for the probability of token $v$ at one position, given $x$ and the prefix $a_{ + +## Implementing RL basics with an LLM + +--- + + + + +## RL review: The agent–environment loop + +**Classical RL**: agent in an MDP $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$ + +- Reward is a known function from the environment per step +- Optimize cumulative return: $J(\pi) = \mathbb{E}_{\tau \sim p_\pi}\!\left[\sum_{t} \gamma^t r(s_t, a_t)\right]$ + +||| + +![](assets/rl.png) + +--- + +## Language models as episodic MDPs + +| MDP concept | Language model | +|-------------|---------------| +| State $s_t$ | Prompt + tokens generated so far: $(x, a_{ + +Two implementation-level views of this MDP: **token-level** (each token is a separate action, used in PPO with GAE) vs. **sequence-level** (entire completion is one action, used in REINFORCE/GRPO). + +--- + +## Agentic RL complicates the MDP + +
+ +| MDP concept | RLVR | Agentic RL | +|-------------|------|------------| +| State $s_t$ | Prompt + tokens so far | Context (prompt + tokens + tool outputs), plus hidden environment state (files, services, other users) | +| Action $a_t$ | Next token | Next token (many are tool calls) | +| Transition | Deterministic-ish: append token | More stochastic: the environment runs the call and appends its output | +| Episode | One prompt → completion | Many turns; [Inkling 2 Small](https://thinkingmachines.ai/news/inkling-2-small)'s p99 rollout takes 4 hours | +| Terminal reward | Verifier on the final answer | Tests on the final state of the environment | +| Loss | Every completion token | Tool-output tokens are masked: the model must not learn to predict the environment | + +
+ +Math RL needed prompts and answer checkers. Agentic RL needs **environments**: containers, real file systems, services, verification tests [@ivison2026tmax]. **Building worlds for LLMs is hard.** + +--- + + + +## RL for LLMs: RLVR + + + +![](assets/rlvr-system.png) + +--- + + +## RL for LLMs: Agentic RL + + + +![](assets/tool_use_rl_loop.png) + +--- + +## Recall: The policy gradient + +The objective and its gradient: + +$$J(\theta) = \mathbb{E}_{\tau \sim p_\theta}[R(\tau)]$$ + +$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta}\!\left[\sum_{t=0}^{T} \Psi_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\right]$$ + +The gradient says: for each sampled token, compute the direction in parameter space that makes it more likely ($\nabla \log \pi$), then scale by how much better than expected it was ($\Psi_t$). + +--- + +## Recall: Why losses are sums of log-probs + +The policy gradient derivation gives: + +$$\nabla_\theta \log p_\theta(\tau) = \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t)$$ + +So we write a loss whose negative gradient is the policy gradient, treating each advantage $A_i$ as a constant across the trajectory: + +$$ +\begin{aligned} +L(\theta) &= -\frac{1}{N} \sum_{i=1}^{N} A_i \sum_{t} \log \pi_\theta(a_{i,t} \mid s_{i,t}) \\ +-\nabla_\theta L(\theta) &= \frac{1}{N} \sum_{i=1}^{N} A_i \sum_{t} \nabla_\theta \log \pi_\theta(a_{i,t} \mid s_{i,t}) +\end{aligned} +$$ + +Gradient descent on $L$ is gradient ascent on $J$. + +--- + +## From the loss to autodiff + +In code, we compute the **log-probs** and let autodiff handle the gradient: + +```python +seq_log_probs = (token_log_probs * completion_mask).sum(dim=-1) +loss = -(seq_log_probs * advantages.detach()).mean() +loss.backward() # fills each parameter's .grad with ∇L +optimizer.step() # θ ← θ - η ∇L +``` + +`loss.backward()` runs backpropagation: it computes $\partial L / \partial \theta$ for every parameter. + +`optimizer.step()` applies the update. + +The loss values themselves are often weird in RL (e.g. zero-centered distributions, rather than nice decreasing accuracy metrics of pretraining) + +Most RL algorithms used today are variations on this computation. + +--- + +## Computing log-probabilities (expanded) + +The fundamental building block: per-token log-probabilities from the policy. + +```python +# Forward pass through the model +logits = model(input_ids).logits # (B, L, vocab_size) + +# Autoregressive shift: logit at position t predicts token at t+1 +logits = logits[:, :-1, :] # (B, L-1, vocab_size) +labels = input_ids[:, 1:] # (B, L-1) +completion_mask = completion_mask[:, 1:] # (B, L-1) + +# Per-token log-probs +log_probs = logits.log_softmax(dim=-1) +token_log_probs = log_probs.gather(dim=-1, index=labels.unsqueeze(-1)).squeeze(-1) # (B, L-1) + +# Per-sequence log-prob: sum over completion tokens +seq_log_probs = (token_log_probs * completion_mask).sum(dim=-1) +``` + +Everything in log-space to avoid numerical underflow from multiplying many small probabilities. + +--- + +## The REINFORCE loss in code + +The simplest policy gradient loss: + +```python +# rewards: (B,) — one reward per sequence +# seq_log_probs: (B,) — sum of log-probs over completion tokens + +# Baseline: average reward in the batch +baseline = rewards.mean() +advantages = rewards - baseline + +# REINFORCE loss (negative because we minimize) +loss = -(advantages * seq_log_probs).mean() +``` + +TPositive advantage → increase probability. Negative → decrease. + +(Aside: How we reduce per-token losses to a scalar — `.mean()` here — turns out to matter more than you'd expect. More in the [book](https://rlhfbook.com/c/06-policy-gradients#loss-aggregation-tradeoffs) and [lecture video](https://www.youtube.com/watch?v=i-AIMpZHgeg&t=2704s).) + +--- + + + +## Challenges of distributed RL training + +--- + +## On-policy vs. off-policy + +Policy-gradient theory is built on the assumption of on-policy data -- from the current model weights in the training framework of the LLM. + +As models get bigger, this is hard because: sampling is expensive, sampling is dominated by long-trail generations, most people use separate systems for inference v training, moving large model weights around is slow, inference weights can be further quantized for speed relative to training weights, multi-cluster deployments, etc. + +--- + +## Synchronous: Share generation and training + + + +The default implementation for policy-gradient algorithms is **on-policy** execution: the generations from the language model are scored before updating the model. + + + +![](assets/async_v_synch_rl_1.png) + +--- + +## Intermediate step: Separate GPUs, still on-policy + + + +Moving generation to an inference engine on its own GPUs makes it fast. To stay on-policy, each side waits for the other. + + + +![](assets/async_v_synch_rl_2.png) + +--- + +## Asynchronous: Off-policy training + + + +The common solution is to constantly run inference and training on separate GPU nodes. The model used for generation may be 1–N steps behind the training model. + + + +![](assets/async_v_synch_rl_3.png) + +This turns out to be mostly "fine" -- but it can be taken too far. + +--- + + +## Synchronous vs. asynchronous training + + + +
+ +**Ideal / Theory (on-policy)**: generate → update → generate → update + +Each batch of completions is scored and used for a short update window (one or a few epochs), then discarded. + +**Reality (async)**: generation and training overlap on different GPU groups for better throughput. The model used for generation may be 1–N steps behind the training model. + +**Tradeoff**: perfect on-policy is slow (GPUs idle during generation or training). Slight staleness (how off-policy the data is) is usually fine. + +
+ +||| + +![Synchronous training idles GPUs; separating generation and training helps, but on-policy requires waiting. Async (off-policy) overlaps both for full utilization.](assets/async_v_synch_rl.png) + +--- + +## Asynchronous RL training + + + +Modern RL for LLMs splits compute into two groups: + +- **Actors** (inference GPUs): generate completions using vLLM or similar +- **Learners** (training GPUs): compute policy gradient updates + +A process management library (e.g., Ray) coordinates data flow between them. Model weights are synced periodically from learner → actor. + +||| + +![Distributed RL system with prompt and result queues between learner and actor GPUs.](assets/distributed-rl.png) + +--- + + + + +## The long-tail generations cause idle GPUs + + + +With **static batching**, each actor generates a whole batch and returns it only when the longest sequence finishes. + +- Wasted compute is (max length − mean length) / max length. +- Olmo 3 at a 32K limit: mean generation length 14,628, so up to **54%** of compute would be wasted. +- Agentic RL is worse: one trajectory makes 2 tool calls, another makes 200. Tool calls add additional lag and complexity. + +||| + +![Static batching. Pink: prefill, green: decoding, dark green: EOS, grey: idle until the batch finishes.](assets/olmo3-static-batching.png) + +--- + + + + +## Solution: Update inference model weights on the fly + + + +
+ +| Olmo 3 RL infrastructure | Tokens/s | +|---|---:| +| Olmo 2 baseline | 881 | +| + continuous batching | 975 | +| + better threading | 1,358 | +| + inflight updates | **2,949** | + +**Continuous batching:** start a new prompt as soon as a sequence finishes. + +**Inflight updates:** load new weights without pausing generation or dropping the KV cache. The biggest speedup, but tokens in one completion now come from different weights. + +
+ +||| + +![Continuous batching backfills finished rows immediately (S5–S7), so no slot sits idle.](assets/olmo3-continuous-batching.png) + +--- + + + + +## Inflight updates: PipelineRL + +The canonical reference for inflight weight updates is **PipelineRL** [@piche2025pipelinerl]. + +[arxiv.org/abs/2509.19128](https://arxiv.org/abs/2509.19128) + +||| + +![](assets/pipelinerl-paper-2025.png) + +--- + +## One training step, end to end + +```python +# 1. Sample completions on the inference engine (pseudocode) +input_ids, completion_mask = sampler.generate(prompts, n=G) + +# 2. Score them, then center within each group +rewards = verifier(prompts, input_ids) # (B*G,) +advantages = group_center(rewards, G) # (B*G,) + +# 3. Recompute log-probs on the trainer, this time with gradients +logits = model(input_ids).logits[:, :-1, :] +token_logprobs = logits.log_softmax(-1).gather( + -1, input_ids[:, 1:].unsqueeze(-1)).squeeze(-1) +seq_logprobs = (token_logprobs * completion_mask[:, 1:]).sum(-1) + +# 4. Update, then sync the new weights back to the sampler +(-(advantages * seq_logprobs).mean()).backward() +optimizer.step() +sampler.load_weights(model) +``` + +Steps 1 and 3 ask two different systems about the same tokens. Do they agree? + +--- + +## Defining the mismatch + +At one position, with prompt $x$ and prefix $a_{ + +**Training-inference mismatch** (TIM) is when $q \neq p$. There are two independent sources: + +- **Staleness:** $\tilde{\theta} \neq \theta$, because the weights have moved since the last syn (on the trainer side). +- **Numerics:** $\tilde{f} \neq f$, because of different kernels, precision, parallelism, or a quantized KV cache (the cached attention keys and values for the prefix, stored in fewer bits). See [Yao et al.](https://fengyao.notion.site/off-policy-rl) and [@marek2026score]. + + +--- + +## Same tokens, different probabilities + + + + + +![Training vs. inference probability for the same tokens in MiniMax-M1, before and after a fix. Agreement improves, but the two systems still differ.](assets/minimax-m1-logprob-mismatch.png) + +--- + +## What training actually estimates + +The gradient we want samples from the learner: + +$$\nabla_\theta J(\theta) = \mathbb{E}_{a \sim p_\theta}\left[ A \sum_{t=1}^{T} \nabla_\theta \log p_\theta(a_t \mid x, a_{ + +The gradient we compute samples from the sampler, but differentiates through the learner: + +$$\hat{g} = \mathbb{E}_{a \sim q}\left[ A \sum_{t=1}^{T} \nabla_\theta \log p_\theta(a_t \mid x, a_{ + +These differ whenever $q \neq p$, and the gap has two parts: the tokens are drawn from the wrong distribution, and so are the prefixes they follow. + +--- + +## The classical fix: Importance sampling + +Reweighting by the sequence ratio recovers the gradient exactly [@marek2026score]: + +$$\nabla_\theta J(\theta) = \mathbb{E}_{a \sim q}\left[ w(a) \, A \sum_{t=1}^{T} \boldsymbol{\xi}_{a_t} \right], \qquad w(a) = \frac{p_\theta(a \mid x)}{q(a \mid x)}$$ + + + +But $w$ can take very large values on rare tokens, which inflates the variance of the gradient. + +Truncated importance sampling [@ionides2008truncated] caps the per-token ratio at a constant $C$, trading a small bias for bounded variance in the policy gradient: + +$$w_t = \frac{p(a_t \mid x, a_{ +## Removing a source of mismatch: Routing replay + + + +In mixture-of-experts (MoE) models, the trainer's router can pick different experts than the sampler's for the same token. Rollout Routing Replay (R3) records the experts chosen at inference and reuses them in the training forward passes. + + + +![](assets/r3-routing-replay.png) + +--- + + + +## Score centering: managing train-inference mismatch + +--- + +## Train-Inference-Mismatch (TIM) can collapse training + + + + + +![Qwen3-30B-A3B-Base trained with RL on math, with the sampler quantized more and more heavily from left to right. W/A is weights and activations; KV is the KV cache. With an FP8 sampler, plain policy gradient (PG) trains stably. With an FP4 KV cache, PG collapses within 200 steps. With INT8 weights and an INT4 KV cache, score centering (SC) reaches 30% accuracy, TIS 12%, and every other method ends below 5%.](assets/score-centering-30b-quantization.png) + +--- + + +## Recall: The policy gradient is a sum of scores + +The policy gradient for one completion: + +$$\nabla_\theta J(\theta) = \mathbb{E}_{a \sim p_\theta}\left[ A \sum_{t=1}^{T} \nabla_\theta \log p_\theta(a_t \mid x, a_{ + +Call each term the [**score**](https://en.wikipedia.org/wiki/Informant_function) of the token, $\boldsymbol{\xi}_v = \nabla_\theta \log p_\theta(v \mid x, a_{ + +At one position, the expected score under the learner, written as a sum: + +$$\mathbb{E}_{v \sim p}[\boldsymbol{\xi}_v] = \sum_{v \in \mathcal{V}} p_v \, \nabla_\theta \log p_v$$ + +
+ +Shorthand: $p_v = p_\theta(v \mid x, a_{\lt t})$, the learner's probability of token $v$ at this position ($\theta$ dropped for brevity); $\mathcal{V}$ is the vocabulary. + +
+ + + +Apply the log-derivative identity, $p_v \nabla_\theta \log p_v = \nabla_\theta p_v$: + +$$= \sum_{v \in \mathcal{V}} \nabla_\theta p_v$$ + + + +Pull the gradient out of the sum. Probabilities sum to one: + +$$= \nabla_\theta \sum_{v \in \mathcal{V}} p_v = \nabla_\theta 1 = \mathbf{0}$$ + + + +This is why baselines are free: $\mathbb{E}[b(x) \, \boldsymbol{\xi}_{a_t}] = b(x) \cdot \mathbf{0}$. It relies on the expectation and the score using **the same distribution**. + +--- + +## Off-policy, the gradient scores are not zero mean + +Tokens now come from $q$, but the score is still computed through $p$. Define the expected score under the sampler: + +$$\bar{\boldsymbol{\xi}} = \mathbb{E}_{v \sim q}[\boldsymbol{\xi}_v] = \sum_{v \in \mathcal{V}} q_v \, \boldsymbol{\xi}_v$$ + +
+ +Shorthand: $q_v = q(v \mid x, a_{\lt t})$ for the sampler and $p_v = p_\theta(v \mid x, a_{\lt t})$ for the learner ($\theta$ dropped for brevity). + +
+ + + +Apply the same log-derivative identity as before: + +$$\bar{\boldsymbol{\xi}} = \sum_{v \in \mathcal{V}} \frac{q_v}{p_v} \, \nabla_\theta p_v$$ + + + +The weights are now $q_v / p_v$ instead of $1$, so the sum no longer collapses to $\nabla_\theta 1 = \mathbf{0}$. The expected score is zero when $q = p$, but not in general. + +--- + +## Computing the gradient of one token + +Start from the estimated gradient, and move the sum outside the expectation: + +$$\hat{g} = \mathbb{E}_{a \sim q}\left[ \sum_{t} A \, \boldsymbol{\xi}_{a_t} \right] = \sum_{t} \mathbb{E}_{a \sim q}\left[ A \, \boldsymbol{\xi}_{a_t} \right]$$ + + + +Sequences are sampled one token at a time, so split each term into the prefix, then the token and everything after it: + +$$\hat{g} = \sum_{t} \mathbb{E}_{a_{ + +From here on, fix the prefix (context) and study the inner term. +We write it as $\mathbb{E}_q[A \, \boldsymbol{\xi}_{a_t}]$. +With the prefix fixed, the token $a_t$ and the advantage $A$ are both still random variables. + +--- + +## Which pieces of the gradient contribute to learning + +Use the covariance identity, $\mathbb{E}[XY] = \mathbb{E}[X] \, \mathbb{E}[Y] + \text{Cov}(X, Y)$, starting on-policy, the expectation of the gradient: + +$$ +\begin{aligned} +\mathbb{E}_p[A \, \boldsymbol{\xi}_{a_t}] &= \mathbb{E}_p[A] \, \mathbb{E}_p[\boldsymbol{\xi}_{a_t}] + \text{Cov}_p(A, \boldsymbol{\xi}_{a_t}) \\ +&= \mathbb{E}_p[A] \cdot \mathbf{0} + \text{Cov}_p(A, \boldsymbol{\xi}_{a_t}) \\ +&= \text{Cov}_p(A, \boldsymbol{\xi}_{a_t}) +\end{aligned} +$$ + + + +Now the same identity under the sampler, where $\mathbb{E}_q[\boldsymbol{\xi}_{a_t}] = \bar{\boldsymbol{\xi}}$: + +$$ +\mathbb{E}_q[A \, \boldsymbol{\xi}_{a_t}] = \mathbb{E}_q[A] \, \mathbb{E}_q[\boldsymbol{\xi}_{a_t}] + \text{Cov}_q(A, \boldsymbol{\xi}_{a_t}) = \underbrace{\mathbb{E}_q[A] \, \bar{\boldsymbol{\xi}}}_{\text{drift}} + \underbrace{\text{Cov}_q(A, \boldsymbol{\xi}_{a_t})}_{\text{signal}} +$$ + + + +Only the covariance sees which tokens led to which outcomes. The drift depends on the advantages only through their mean -- not per-token attribution. + +--- + +## What the drift does + + + +The drift, $\mathbb{E}_q[A] \, \bar{\boldsymbol{\xi}}$, depends on the advantages only through their mean, so it moves the weights the same way regardless of which token was sampled. + +We don't want that: in practice, it can pull the learner's weights toward a stale and/or quantized sampler, causing potentially compounding instabilities. + +--- + +## Score centering: A baseline for the score + +Baselines center the reward. Score centering centers the per-token log-prob gradients (called "score" in statistics), subtracting its expected value under the sampler: + +$$\tilde{\boldsymbol{\xi}}_{a_t} = \boldsymbol{\xi}_{a_t} - \bar{\boldsymbol{\xi}}$$ + +
+ +Recall: the score of token $v$ is $\boldsymbol{\xi}_v = \nabla_\theta \log p_\theta(v \mid x, a_{\lt t})$, the gradient of its log-probability under the learner. + +
+ + + +$\bar{\boldsymbol{\xi}}$ is a sum over the whole vocabulary at a fixed prefix, so it is a constant with respect to the sampled token: + +$$\mathbb{E}_q[\tilde{\boldsymbol{\xi}}_{a_t}] = \mathbb{E}_q[\boldsymbol{\xi}_{a_t} - \bar{\boldsymbol{\xi}}] = \mathbb{E}_q[\boldsymbol{\xi}_{a_t}] - \bar{\boldsymbol{\xi}} = \bar{\boldsymbol{\xi}} - \bar{\boldsymbol{\xi}} = \mathbf{0}$$ + + + +The centered score has zero mean under the sampler! which restores the property that made on-policy learning attractive. When $q = p$ (on-policy), $\bar{\boldsymbol{\xi}} = \mathbf{0}$ and the correction does nothing. + +--- + +## The corrected, per-token gradient + +With the centered score, the policy-gradient estimate becomes: + +$$\hat{g}_{\text{centered}} = \mathbb{E}_{a \sim q}\left[ A \sum_{t=1}^{T} \tilde{\boldsymbol{\xi}}_{a_t} \right] = \sum_{t} \mathbb{E}_{a_{ + +The last step uses that a constant does not change a covariance: $\text{Cov}(X, Y - c) = \text{Cov}(X, Y)$. + +--- + +## Score centering complements other interventions like importance sampling + + + +- **Score centering** removes the drift, the pull toward the sampler that carries no learning signal. +- It does not make training on-policy: the signal is still $\text{Cov}_q(A, \boldsymbol{\xi}_{a_t})$, measured under the sampler, not the learner. +- **Importance sampling** (e.g., TIS) reweights samples toward the learner, but clipping the ratios lets some drift back in. +- They fix different problems and are composable. The score-centering paper recommends using both! + +--- + + +## Quantization vs. staleness + + + +- **Quantization:** The sampler has the learner's weights, just lower precision. Score centering alone is enough. +- **Staleness:** The learner moves away from the sampler, so score centering needs importance sampling too. Taking several gradient steps per batch causes the same kind of staleness. + + + +![Qwen3-0.6B on Countdown with an INT8 sampler (left) and a sampler updated only every 64 steps (right).](assets/score-centering-quant-stale.png) + +--- + + +## Recent algorithms differ mostly in how they clip the log-ratio + + + + + +![Importance-sampling corrections as a grid: per-token or per-sequence ratio, the clipping range (Low to High), what happens outside it, and which advantage signs are corrected.](assets/score-centering-is-methods-table.png) + +--- + + + +## Score-centering in the wild (aka industry) + +--- + +## StableReinforce and GRPO + +(Supposedly) Many labs are supposedly using [**StableReinforce**](https://tinker-docs.thinkingmachines.ai/tinker/losses/stable-reinforce/), the loss Tinker added, rather than GRPO. This is unconfirmed, but the two are related. + +The group advantages are nearly the same. GRPO normalizes each reward by the group mean and standard deviation: + +$$A_i^{\text{GRPO}} = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}$$ + + + +REINFORCE with a group baseline is usually RLOO, which leaves each sample out of its own baseline: + +$$A_i^{\text{RLOO}} = r_i - \frac{1}{G-1} \sum_{j \neq i} r_j = \frac{G}{G-1} \big( r_i - \text{mean}(r_1, \dots, r_G) \big)$$ + + + +Without the std, the two differ only by a constant scale. The bigger difference is the loss: GRPO usually wraps the advantage in PPO's clipped ratio, while REINFORCE uses the plain log-prob loss. + +--- + +## Is StableReinforce score centering? + + + +Tinker's docs describe the behavior, but not the main loss: + +> "In our internal post-training, this loss function has led to extremely stable training, even under highly off-policy async RL and low precision numerics." + + + +They do reveal a clue: sampling must record extra information, through `record_stability_info=True`. + + + +Frontier losses are rarely published, so public ideas are rarely confirmed. Here, Martin Marek, the score centering paper's first author, tried to recover the loss from its gradients alone. + +--- + +## Reverse-engineering a loss from its gradients + + + + + +Tinker does not expose the loss or the "stability info," but it does let you take a step and download the weights. + +**Step 1: Recover the raw gradient.** Tinker has no SGD, so use Adam with $\epsilon = 10{,}000$. This dominates Adam's second moment: + +$$\Delta w \approx -\frac{\text{lr} \cdot g}{\epsilon}$$ + + + +**Step 2: Read it in logit space.** Train a rank-1 LoRA adapter on the unembedding of Qwen3.5-4B. The gradient with respect to $B$ equals the gradient with respect to the logits, up to scale. + +||| + +![A rank-1 LoRA adapter added to the unembedding matrix, so the gradient on B is proportional to the logit gradient. From Marek's thread.](assets/marek-thread-lora.png) + +--- + +## Only 8 logits move + + + + + +For a single-token update: + +- **Reinforce** puts a nonzero gradient on all logits: 248,077 of them move. +- **StableReinforce** moves only 8. + + + +The bars come in multiples of $1/8$, and the third row, Reinforce with score centering, is identical. + +||| + +![Logit gradients for a single-token update under Reinforce, StableReinforce, and Reinforce with score centering. From Marek's thread.](assets/marek-thread-logit-gradients.jpeg) + +--- + +## Why a sparse gradient is the clue + + + +The plot shows gradients with respect to the logits $z$. For a softmax, the gradient of $\log p_v$ is the one-hot vector $e_v$ minus the probabilities: + +$$\boldsymbol{\xi}_v = \nabla_z \log p_v = e_v - p$$ + + + +Estimate $\bar{\boldsymbol{\xi}}$ by drawing 8 tokens from the sampler, $v_1, \dots, v_8 \sim q$: + +$$\bar{\boldsymbol{\xi}} \approx \frac{1}{8} \sum_{i=1}^{8} \boldsymbol{\xi}_{v_i} = \frac{1}{8} \sum_{i=1}^{8} (e_{v_i} - p) = \frac{1}{8} \sum_{i=1}^{8} e_{v_i} - p$$ + + + +Subtract it from the score of the sampled token. The dense $-p$ cancels exactly: + +$$\boldsymbol{\xi}_{a_t} - \bar{\boldsymbol{\xi}} \approx (e_{a_t} - p) - \left( \frac{1}{8} \sum_{i=1}^{8} e_{v_i} - p \right) = e_{a_t} - \frac{1}{8} \sum_{i=1}^{8} e_{v_i}$$ + + + +Flip the signs for a loss and this is the figure: $-1$ on the sampled token, $+1/8$ per draw. + +--- + +## The match + + + +**Hypothesis:** StableReinforce is REINFORCE plus score centering, with the expected score computed over 8 randomly sampled tokens. + + + +**Test:** Implement that as a custom Tinker loss, take an optimizer step, and compare. The gradients are within numeric precision across all layers. + + + +![The StableReinforce gradient minus the Reinforce with score centering gradient is zero across layers. From Marek's thread.](assets/marek-thread-gradient-match.png) + +--- + +## Score centering in a recent model + + + +Ripple-65B Preview, a 65B open-weight model from Ricursive (built on a lot of Olmo 3 infra!), names score centering in its post-training: + +> "Our mixed RL recipe uses truncated importance sampling, score centering, and linear length penalty with high-quality long-horizon synthetic environments." + +--- + + +## Wrapping up + +--- + +## Takeaways + + +--- + + + +## Thank you + +I have a lot of content expanding on this stuff! + +- **Course:** 14 lectures, Q&As, and guest episodes at [rlhfbook.com/course](https://rlhfbook.com/course) +- **Book:** [rlhfbook.com](https://rlhfbook.com), also in print from [Manning](https://www.manning.com/books/reinforcement-learning-from-human-feedback) +- **Newsletter:** [interconnects.ai](https://www.interconnects.ai/) + +nathan@natolambert.com + +||| + +![](assets/post-training-course-thumbnails.jpeg) + +--- + + + + + +## Extra slides + +--- + + + +## Measuring the mismatch at one token + +The ratio between the two systems for a sampled token, and its log: + +$$w_t = \frac{p(a_t \mid x, a_{ +## Mismatch adds up over a completion + +The ratio for a whole completion is the product of the token ratios: + +$$w(a) = \frac{p_\theta(a \mid x)}{q(a \mid x)} = \prod_{t=1}^{T} w_t, \qquad \log w(a) = \sum_{t=1}^{T} \delta_t$$ + +Take the expectation under $q$, and apply the previous slide at each position: + +$$\mathbb{E}_{a \sim q}[\log w(a)] = \sum_{t=1}^{T} \mathbb{E}_{a_{ +## What is the score? + +In statistics, the [**score**](https://en.wikipedia.org/wiki/Informant_function) is the gradient of the log-likelihood with respect to the parameters. For one token $v$ at one position: + +$$\boldsymbol{\xi}_v = \nabla_\theta \log p_\theta(v \mid x, a_{ +## Why the drift compounds + + + +Distilling toward a **fixed** teacher is harmless. It converges to the best fit of that teacher. + +In online RL, the teacher moves with the student: + +- The sampler is a biased copy of the learner, e.g., quantized or stale. +- Each step pulls the learner toward the sampler's errors. +- The new weights are synced back to the sampler, carrying those errors with them. +- The next sampler is biased again, so the error compounds instead of converging. +- The bias does not average out with more samples, since $\bar{\boldsymbol{\xi}}$ is fixed at each prefix. It needs a direct correction. + +--- + + + +## Do group baselines already fix this? + + + +Not fully. Group centering makes advantages sum to zero **over the rollouts of a prompt**: + +$$\frac{1}{G} \sum_{i=1}^{G} (r_i - \bar{r}) = 0$$ + +But drift arises **at each prefix**, and $\mathbb{E}_q[A \mid x, a_{ +## From the update to a loss + + + +We want a scalar whose gradient is $A \, (\boldsymbol{\xi}_{a_t} - \bar{\boldsymbol{\xi}})$. + +The first piece is already a gradient, $\boldsymbol{\xi}_{a_t} = \nabla_\theta \log p_{a_t}$. For the second: + +$$\bar{\boldsymbol{\xi}} = \sum_{v \in \mathcal{V}} q_v \, \nabla_\theta \log p_v$$ + +The sampler's probabilities are logged values, so we mark them as constants with a stop-gradient and pull the gradient out: + +$$\bar{\boldsymbol{\xi}} = \nabla_\theta \sum_{v \in \mathcal{V}} \text{sg}[q_v] \log p_v$$ + +Combine the two pieces, and multiply by $-A$: + +$$L = -A \left( \log p_{a_t} - \sum_{v \in \mathcal{V}} \text{sg}[q_v] \log p_v \right), \qquad -\nabla_\theta L = A \, (\boldsymbol{\xi}_{a_t} - \bar{\boldsymbol{\xi}}) = A \, \tilde{\boldsymbol{\xi}}_{a_t}$$ + +--- + + + +## Score centering in code + +The correction is one extra term in the REINFORCE loss from earlier: + +```python +# Learner: log-probs over the whole vocabulary, with gradients +learner_vocab_logprobs = logits.log_softmax(-1) # (B*G, L, V) +learner_logprobs = learner_vocab_logprobs.gather( + -1, tokens.unsqueeze(-1)).squeeze(-1) # (B*G, L) + +# Sampler: logged at generation time, so it has no gradient path +sampler_probs = sampler_vocab_logprobs.exp() # (B*G, L, V) + +# The gradient of `centering` is the expected score, s-bar +centering = (sampler_probs * learner_vocab_logprobs).sum(-1) # (B*G, L) + +# advantages: (B*G, 1), broadcast to every token +per_token_loss = -advantages * (learner_logprobs - centering) +loss = ((per_token_loss * mask).sum(1) / mask.sum(1)).mean() +``` + +No ratios, no masks, and no clipping. On-policy, the gradient of `centering` is zero. + +--- + + + +## The full distribution is too expensive + + + +Computing $\bar{\boldsymbol{\xi}}$ exactly needs the sampler's next-token distribution at every position: + +- Qwen3's vocabulary has 152K tokens, so one position in fp32 takes about 608KB. +- A batch of 1024 rollouts with 32K tokens each would take about 20TB. + +Instead, log only the sampler's **head** $H$, its $k$ most likely tokens at that position, with $k = 128$. Model the **tail** with the learner, rescaled to carry the sampler's missing mass: + +$$ +\hat{q}_v = +\begin{cases} +q_v & v \in H \\ +\rho \, p_v & v \notin H +\end{cases} +\qquad +\rho = \frac{1 - \sum_{v \in H} q_v}{1 - \sum_{v \in H} p_v} +$$ + +--- + + + +## The tail cancels + + + +Substitute $\hat{q}$ into the expected score, and split the sum into head and tail: + +$$\bar{\boldsymbol{\xi}} \approx \sum_{v} \hat{q}_v \, \boldsymbol{\xi}_v = \sum_{v \in H} q_v \, \boldsymbol{\xi}_v + \rho \sum_{v \notin H} p_v \, \boldsymbol{\xi}_v$$ + +The zero-mean identity splits the same way, which rewrites the tail in terms of the head: + +$$\mathbf{0} = \sum_{v} p_v \, \boldsymbol{\xi}_v = \sum_{v \in H} p_v \, \boldsymbol{\xi}_v + \sum_{v \notin H} p_v \, \boldsymbol{\xi}_v \quad \Rightarrow \quad \sum_{v \notin H} p_v \, \boldsymbol{\xi}_v = -\sum_{v \in H} p_v \, \boldsymbol{\xi}_v$$ + +Substitute, and collect the terms. Only the $k$ head tokens remain: + +$$\bar{\boldsymbol{\xi}} \approx \sum_{v \in H} q_v \, \boldsymbol{\xi}_v - \rho \sum_{v \in H} p_v \, \boldsymbol{\xi}_v = \sum_{v \in H} (q_v - \rho \, p_v) \, \boldsymbol{\xi}_v$$ + +The paper reports that $k = 128$ and $k = 32$ both perform on par with the full vocabulary, at under 1% wall-clock overhead for $k = 128$. + +--- + + + +## Top-k score centering in code + + + +```python +# Logged during generation: sampler_topk_logprobs, sampler_topk_ids (B*G, L, k) +learner_logprobs = learner_vocab_logprobs.gather(-1, tokens[..., None])[..., 0] +head_logprobs = learner_vocab_logprobs.gather(-1, sampler_topk_ids) # (B*G, L, k) +learner_head_probs = head_logprobs.exp() +sampler_head_probs = sampler_topk_logprobs.exp() + +# rho: the ratio of the sampler's tail mass to the learner's tail mass +sampler_tail = (1 - sampler_head_probs.sum(-1)).clamp_min(1e-6) +learner_tail = (1 - learner_head_probs.sum(-1)).clamp_min(1e-6) +rho = sampler_tail / learner_tail # (B*G, L) + +# detach() is required: coef contains the learner's own probabilities +coef = (sampler_head_probs - rho[..., None] * learner_head_probs).detach() +centering = (coef * head_logprobs).sum(-1) # (B*G, L) + +per_token_loss = -advantages * (learner_logprobs - centering) +``` diff --git a/teach/scripts/check_slide_overflow.py b/teach/scripts/check_slide_overflow.py new file mode 100644 index 00000000..24ab4bdf --- /dev/null +++ b/teach/scripts/check_slide_overflow.py @@ -0,0 +1,131 @@ +"""Flag slides whose content overflows the slide, collides with the footer, or has broken images. + +Loads a built colloquium deck in headless Chrome, activates each slide in turn +(the way the presenter view shows it), and measures every element in the slide +body against the slide and footer bounds. + +Usage: + uv run python teach/scripts/check_slide_overflow.py build/html/teach/Cornell-CS4789-2026/index.html + +Exits 1 and prints one line per problem slide if anything overflows. +""" + +import json +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +CHROME_PATHS = [ + "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", + "/Applications/Chromium.app/Contents/MacOS/Chromium", + "google-chrome", + "chromium", +] + +TOLERANCE_PX = 2 + +PROBE = """ + +""" + + +def find_chrome() -> str: + for path in CHROME_PATHS: + if Path(path).exists() or subprocess.run(["which", path], capture_output=True).returncode == 0: + return path + sys.exit("No Chrome/Chromium found") + + +def main() -> int: + if len(sys.argv) != 2: + sys.exit(__doc__) + deck = Path(sys.argv[1]).resolve() + html = deck.read_text() + probed = html.replace("", PROBE % {"tol": TOLERANCE_PX} + "", 1) + + # Write next to the original so relative asset paths still resolve. + with tempfile.NamedTemporaryFile("w", suffix=".html", dir=deck.parent, delete=False) as f: + f.write(probed) + tmp = Path(f.name) + try: + out = subprocess.run( + [ + find_chrome(), + "--headless=new", + "--disable-gpu", + "--window-size=1280,720", + "--virtual-time-budget=30000", + "--dump-dom", + tmp.as_uri(), + ], + capture_output=True, + text=True, + timeout=120, + ).stdout + finally: + tmp.unlink() + + m = re.search(r'
(.*?)
', out, re.S) + if not m: + sys.exit("Probe did not report (page failed to load?)") + problems = json.loads(m.group(1).replace(""", '"').replace("&", "&").replace("<", "<").replace(">", ">")) + if not problems: + print(f"OK: no overflow in {deck}") + return 0 + for p in problems: + print(f"slide {p['slide']} ({p['title']}): " + "; ".join(p["issues"])) + return 1 + + +if __name__ == "__main__": + sys.exit(main())