From 21350ec5a676f3bd240afbdf8a8d8258903c6ed9 Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Thu, 5 Nov 2026 10:04:49 -0500 Subject: [PATCH] Note that score centering composes with corrections like TIS Co-Authored-By: Claude --- book/chapters/06-policy-gradients.md | 1 + 1 file changed, 1 insertion(+) diff --git a/book/chapters/06-policy-gradients.md b/book/chapters/06-policy-gradients.md index 80565ab0..db8c04d5 100644 --- a/book/chapters/06-policy-gradients.md +++ b/book/chapters/06-policy-gradients.md @@ -1067,6 +1067,7 @@ TIS for the learner–sampler ratio has been adopted across major open-source RL Building on the goal of importance-sampling interventions, modern developments in RL have continued to search for interventions that mitigate training-inference mismatch in distributed RL for agentic tasks. Another intervention, called score-centering, comes from the realization that there is a persistent bias in the policy-gradient formulation, especially in off-policy settings [@marek2026score]. +It can be used in addition to importance-sampling corrections like TIS, rather than replacing them. In this section, we derive the bias and highlight the authors' solution. To start, recall the policy gradient formulation with a per-sequence advantage, $A$: