On-Policy Self-Distillation: Continual Learning from Production Feedback

Goyal, Kartik

Introduction

We train a model, ship it, and then we freeze it. Whatever it knows about your domain it learned before it ever met a real user, and from its first day in production it stops getting better. Continual learning is the problem of undoing that bargain: letting a deployed agent keep learning from the experience it accumulates in the field.

This report is a practical recipe for one such approach: learning from textual feedback in a multi-turn agent handling knowledge-work tasks, the setting real deployments actually face. We break the approach into the decisions that matter in practice and measure each one: which kind of feedback to use, how to place that feedback in the agent’s context, which training objective to use, and what the training generalizes to if you keep learning new domains.

The cost of getting better

Every standard way to improve an agent ends at the same requirement: someone has to produce new, curated data. Supervised fine-tuning needs an expert to hand-write correct trajectories. Reinforcement learning, the option most agent teams reach for, needs an environment with a grader and many rollouts per task to surface the good samples. Distilling from gold solutions needs the gold solutions. All three are expensive and labor-intensive, and none of the work carries over: what you spent on the retail agent does nothing for the telecom one. The cost of improvement scales with the number of domains you serve.

The feedback you’re already throwing away

A deployed agent produces a second kind of data continuously, for free, in exactly the domain you care about. Users correct it, clarify what they meant, Aligning Language Models from User Interactions (Kleine Buening et al, 2026) and explain how the task was supposed to go. The environment contributes too: a tool call returns an error, an argument gets rejected, a required step is skipped. There is an enormous amount of it.

This matters beyond cost. Most of what an organization knows is not in any document; it sits with experienced people, in how a deal actually gets structured, which exceptions are really allowed, which client always gets a manual review. That is exactly the knowledge a custom agent needs and never has. Every time they correct an answer, rephrase a request, or explain why the model got it wrong, they hand over a piece of expertise that would otherwise never be written down.

So why does all of it drain away unused? Because this data breaks every assumption the standard playbook makes. It’s text, not labels. It’s messy: sometimes wrong, often terse. It arrives one trajectory at a time, with no second attempt: you can’t re-run a real user’s afternoon to collect a cleaner sample, and you can’t roll the agent out again in the exact environment where it slipped. And there’s no reward attached: nothing tells you, in a number, whether the conversation went well.

That is the setting this report works in. It answers four questions:

Can an agent learn from production feedback, and how close does that get to RLVR?
What kind of feedback, and where does it go in the context?
Which training objective stays stable?
Can it learn a new domain without forgetting the last one?

τ²-bench and the GRPO baseline

Answering those questions takes a setting that looks like a deployment but can still be scored. We use tau2-bench τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al, 2025) retail: a customer-service agent working multi-turn tasks against real tools, with a simulated user on the other side. The methods never see the reward; we need it only to measure them.

We adopt the retail domain’s official train and test split, and we cap episodes at 30 turns and 512 new tokens per turn. We drive the user with Gemini 3.1 Pro at temperature 0, so the user side contributes no sampling noise of its own.

We take the benchmark’s reward as given. An episode scores 1 only if two checks both pass: the database ends in the state the task requires, and the agent tells the user what the task requires it to say. We single out the first check because it governs how the later results read. Only state-changing tool calls can satisfy it; read-only lookups cannot move it.

We report eval-set Pass@1 throughout. At every training step we evaluate the eval set with 4 independent runs per task. We also evaluate the untrained checkpoint before training begins, so each curve is anchored to where the model started.

We train Qwen3-14B Qwen3 Technical Report (Qwen Team, 2025) with LoRA adapters LoRA: Low-Rank Adaptation of Large Language Models (Hu et al, 2021) at rank 64 and thinking disabled, at batch size 16.

The GRPO baseline

We post-trained the same model on the same tasks with GRPO, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al, 2024) the standard reward-RL recipe: eight rollouts per task sampled at temperature 1.0, the group’s mean reward as the advantage baseline, and a KL anchor to the reference policy.

The run does two jobs for us. It shows the benchmark is learnable at all, so a flat line from any later method is a real result rather than a dead environment. And it sets the bar: eval-set Pass@1 climbs from the untrained model’s ~0.24 to a plateau around 0.45–0.46 over roughly 4K training rollouts. Every cheaper method in this report is measured against that plateau.

Eval-set Pass@1 for the GRPO baseline over training rollouts on τ²-bench retail, climbing from about 0.24 to a plateau around 0.45–0.46. — **Figure 1.** GRPO baseline on τ²-bench retail: eval-set Pass@1 against cumulative training rollouts. G = 8 samples per task, rollout temperature 1.0.

On-policy self-distillation

We started from GRPO because it works. But it only works under two conditions. It draws eight rollouts per task, and it draws them hot, at temperature 1.0, so the group spreads far enough apart to produce a gradient. A deployed agent gives you neither. It produces one trajectory per task, at the low temperature you actually ship at. That is the problem the bar leaves behind: how do you learn from a single trajectory and a paragraph of feedback, on rollouts sampled the way production samples them?

Text in place of the reward

The tool is on-policy self-distillation. Take the response the agent already produced, its own on-policy sample. Show the model that same response again, but this time with the feedback in its context, and watch how its own sense of the response shifts: which of its moves it now finds more likely, which mistake it now finds less. That shift is a target, and we distill it back into the model.

On-policy distillation On-Policy Distillation (Lu, 2025) trains a student on its own generated samples On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al, 2023) rather than on text a teacher wrote. Self-distillation makes teacher and student one network, separated only by context Learning by Distilling Context (Snell et al, 2022): the teacher reads privileged information Learning Using Privileged Information: Similarity Control and Knowledge Transfer (Vapnik and Izmailov, 2015) $c$ that the student will not have at inference time. Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (Zhao et al, 2026) take $c$ to be a verified reference solution; Hübotter et al. Reinforcement Learning via Self-Distillation (Hübotter et al, 2026) take it to be feedback written about a trajectory the agent has already produced, which is the setting here. Writing $p_\theta$ for the model and $x$ for the task,

p_T(\,\cdot \mid x, c) \;\triangleq\; p_\theta(\,\cdot \mid x, c), \qquad p_S(\,\cdot \mid x) \;\triangleq\; p_\theta(\,\cdot \mid x)

The student samples a trajectory $y$ from its own policy and both distributions are evaluated at every token of it. The objective is the divergence between them, averaged along the trajectory and taken in expectation over tasks and over the student’s own rollouts:

\mathcal{L}(\theta) \;=\; \mathbb{E}_{(x,\,c)} \;\mathbb{E}_{y \,\sim\, p_S(\cdot \mid x)} \bigg[\, \frac{1}{|y|} \sum_{t=1}^{|y|} D\big(p_S(\,\cdot \mid x,\, y_{<t}) \;\big\|\; p_T(\,\cdot \mid x,\, c,\, y_{<t})\big) \bigg]

That inner expectation is the on-policy part: the trajectories being scored are the ones the model actually produced, not ones written for it.

What remains is the choice of $D$ . Reverse KL puts the student in the first argument, so the sum over the vocabulary $\mathcal{V}$ is weighted by the student’s own probabilities:

D_{\mathrm{KL}}(p_S \,\|\, p_T) \;=\; \sum_{v \in \mathcal{V}} p_S(v) \, \log \frac{p_S(v)}{p_T(v)}

Forward KL is the same expression with the two arguments swapped, weighting by the teacher instead.

Our setup

We distil with reverse KL throughout; a later section puts it head to head against forward KL and generalized JSD. The teacher is the frozen base at $\theta_0$ , so the target holds still while the student trains. There is no reward term and no KL anchor to the reference policy.

The divergence is taken over the full next-token distribution at every position, not just over the probability the two models assign to the token the agent actually emitted. Zhao et al. ablate full-vocabulary logit distillation against the sampled-token alternative, which uses only the teacher’s log-probabilities at the tokens the student emitted, and report the full-vocabulary version stronger, at the cost of holding vocabulary-sized logits at every position. Hübotter et al. show that a top- $K$ approximation at $K = 100$ captures most of that information at virtually no memory overhead, and that is what we use: the student’s top 100 tokens plus a single atom holding the remaining mass.

Illustration

Figure 2. One τ²-bench retail trajectory re-scored with its feedback in context. Shading is the per-token KL between the model reading the feedback and the same model without it. The dark spans are the hallucinated authentication email, the repeated tool call, and the premature ask.

The case for G = 1

Rollout count is the first of the two constraints. GRPO draws eight scored samples per task. We cut that to four, then to one for OPSD, and ask whether the run still learns.

Eval-set Pass@1 against training rollouts on τ²-bench retail for GRPO at G = 8 and OPSD at G = 4 and G = 1. Both OPSD curves reach the GRPO plateau in far fewer rollouts. — **Figure 3.** Eval-set Pass@1 against cumulative training rollouts, GRPO at G = 8 against OPSD at G = 4 and G = 1. Faint lines are raw evaluations, bold are Gaussian-smoothed (σ = 1.5).

As we can see in the graph, both OPSD runs reach the GRPO bar without a reward function. The one that matters for production logs is G = 1: a single trajectory per task, matching GRPO’s best eval-set accuracy at roughly a tenth of its rollouts.

The temperature you’d actually deploy at

The comparison above rolled out at temperature 1.0 to keep OPSD and GRPO on the same footing. However, no one rolls out at this temperature. A deployed tool-calling agent samples at much lower temperatures, because you want reliable actions rather than creative ones. Reward-RL learns from traffic it will never see.

GRPO generally needs hot rollouts because it requires a correct rollout to be present in the group. Self-distillation should not: its target is a per-token shift on the trajectory it already has, so it should be able to learn even when there is no positive rollout, as long as the feedback is present and of high quality. To test this hypothesis, we roll out at 0.6 and 0.3, everything else fixed.

Rollout temperature	Pass@1	Std. dev.
Base model, untrained	0.23	± 0.02
1.0	0.46	± 0.03
0.6	0.48	± 0.02
0.3	0.45	± 0.03

The peaks are flat across all three temperatures with heavily overlapping error bars, so cooling the rollouts to the range you would actually deploy at costs nothing in eval-set Pass@1.

The design choices that matter

Put the last two sections together and we have a recipe that can learn from production logs: data with no reward function and no environment behind it, sampled at the low temperatures agents are actually deployed at, one rollout per task. From here we go into how to implement it: the practical questions that come up once you point this at real traffic, taken one at a time, with the ablations and analysis behind each answer.

Where do you put the feedback?

Self-distillation has so far been a largely single-turn technique. Hübotter et al. evaluate on chemistry questions and competitive programming; the on-policy distillation recipe of Lu and Thinking Machines Lab has the same shape: one prompt, one completion, one thing that went wrong. In that setting placement isn’t a decision at all: there’s exactly one prompt, so the privileged signal goes in a block at the top of it and the whole completion sits downstream of it.

A multi-turn tool-calling agent is where the question becomes real: where in the context does the feedback go? A τ²-bench trajectory is a dozen-odd turns of tool calls, observations and user replies, with several distinct mistakes scattered through it. We start where the prior work does, aggregating all the feedback for the entire rollout into one block at the top, and compare that against localizing the feedback at every turn.

Top-level feedback. Let the trajectory run over turns $k = 1, \dots, T$ , of which the reviewer flags a subset $\mathcal{F}$ of size $N$ and writes feedback $c_k$ on each. The $N$ pieces are concatenated into one block appended to the task description at the top of the prompt,

c \;=\; \textstyle\bigoplus_{k \in \mathcal{F}} c_k

and the divergence is taken across the whole trajectory that follows, under the rollout’s own loss mask $m_t \in \{0, 1\}$ , which is 1 on tokens the agent generated and 0 on tool outputs and user replies:

\mathcal{L}_{\text{top-level}} \;=\; \frac{1}{\sum_t m_t} \sum_{t=1}^{|y|} m_t \, D\big(p_S(\,\cdot \mid x,\, y_{<t}) \;\big\|\; p_T(\,\cdot \mid x,\, c,\, y_{<t})\big)

Every turn contributes, flagged or not, and every scored token is conditioned on all $N$ corrections rather than on the one written about its own turn.

Per-turn feedback. Each flagged turn becomes its own training example, so a rollout with $N$ of them yields $N$ rows rather than one. For turn $k$ the scored tokens are $y_k$ , the agent’s output at that turn alone. The student reads the true causal history $H_k = (x,\, y_{<k})$ ; the teacher reads the same history with $c_k$ attached, and nothing about the other $N - 1$ turns:

\mathcal{L}_{\text{per-turn}} \;=\; \frac{1}{\sum_{k \in \mathcal{F}} |y_k|} \sum_{k \in \mathcal{F}} \; \sum_{t=1}^{|y_k|} D\big(p_S(\,\cdot \mid H_k,\, y_{k,<t}) \;\big\|\; p_T(\,\cdot \mid H_k,\, c_k,\, y_{k,<t})\big)

No mask appears because $y_k$ is a single agent turn, so every token in it is scored.

Otherwise the two arms are the same run twice: same reviewer, same feedback, one config field apart.

Placement	Pass@1	Std. dev.
Base model, untrained	0.23	± 0.02
Per-turn	0.48	± 0.02
Top-level	0.03	± 0.01

Per-turn climbs from the untrained model’s 0.23 to 0.48 and holds it. Top-level never gets above the untrained model at any point in the run: it drops below base on the first update and decays monotonically to 0.01. The rest of this section investigates the root causes of that collapse The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (Zhu et al, 2026) from several angles.

The feedback never reaches past the front of the episode. We started with the loss term itself. For every assistant turn we measured, token by token, how much the feedback-conditioned teacher disagrees with the student (that disagreement is the training signal), and plotted it against the turn’s position in the episode.

We find that at turn 1, right after the injected block, the two placements carry a comparable per-token signal. Top-level then falls away within a turn and sits near zero for the rest of the episode, while per-turn holds its level at every position with no positional trend. The appendix shows this token by token on a single turn scored under both placements.

Per-token reverse KL between student and feedback-conditioned teacher, averaged per assistant turn and plotted against turn position. Top-level spikes to 0.62 nats at turn 1 then sits near zero for the rest of the episode; per-turn stays between 0.15 and 1.4 nats at every position. — **Figure 4.** Exact full-vocab per-token reverse KL D(student‖teacher), the OPSD loss term itself, averaged per assistant turn against turn position, with both arms measured on the same step-1 rollouts.

Signal that doesn’t arrive doesn’t leave the turn alone; it makes it worse. We then measured behaviour by turn position rather than in aggregate. We ran the reviewer over each trained policy’s own rollouts, asked at every position how often it marks that turn wrong, and differenced it against the untrained model, so the comparator is a flat line at zero.

Where the signal reaches, top-level works. Through the opening turns it is flagged far less often than the untrained model, by a wider margin than per-turn: it has essentially perfected the prologue of greet, authenticate, first lookup. Past the reach of its one injected block it crosses zero and spends the rest of the episode worse than the model that was never trained at all. Per-turn sits above zero at essentially every position.

We expected the unreached turns to come out neutral. Falling below zero means the objective degrades them rather than leaving them alone. One hypothesis is the coupling in top-level’s definition: it distils over the whole trajectory, including the turns the reviewer marked correct, and every turn is conditioned on the full aggregated block, most of which is unrelated to it.

Change in reviewer flag rate relative to the untrained model, by turn position. Top-level is +58 to +43 points better than base for turns 1 to 3, then crosses below zero at turn 4 and runs 20 to 35 points worse for the rest of the episode. Per-turn stays positive at nearly every position. — **Figure 5.** Reviewer flag rate by turn position, differenced against the untrained model; positive means fewer turns marked wrong than base. Rates are conditioned on failed rollouts, since the reviewer only runs on failures.

Where the collapse actually comes from. We then measured eval-set tool use over training, split two ways: read-only lookups against state-changing writes, and the volume of each against its error rate. The first outcome is that top-level does learn. Its read-call error rate falls steeply over training, further than per-turn’s, and its read volume triples.

The second outcome is the collapse. Over the same steps its write calls, generally made at the later turns in a rollout, fall to nearly zero. Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents (Ding et al, 2026). They report the same failure under naive self-distillation on τ³-airline. τ²-retail scores an episode on the conjunction of two checks, whether the database ended in the expected state and whether the agent told the user what the task required, and only writes can satisfy the first.

Four panels comparing per-turn and top-level tool use over training. Top-level's read-only calls per episode rise from about 6 to 20 while its state-changing calls fall from 1.2 to nearly zero; its read error rate drops to about 1 percent and its write error rate line ends at step 21 for lack of write calls. Per-turn holds about 7 reads and 2 writes per episode throughout. — **Figure 6.** Eval-set tool use over training, by call class (columns) and volume above error rate (rows). Top-level's write-error line ends at step 21 (open dot): past that there are fewer than 20 write calls in the whole 160-episode eval set. Note the per-panel y-scales.

What kind of feedback do we use?

Placement settles where the signal goes; the other half of the question is what goes in that slot. In production, feedback ranges from what is already sitting in the trajectory up to the actual answer, which you would have to pay an expert for. Two rungs matter, each a different bet on how much to spend turning a raw log into a training signal.

Reflection on raw feedback. The cheap rung is written from what is already in the transcript: the tool errors the agent hit and the user’s own text, corrections, complaints, “no, I meant the other account.” Rather than pass that raw text through, a reviewer reads the trajectory Self-Refine: Iterative Refinement with Self-Feedback (Madaan et al, 2023) up to that point together with the tool error or user message that follows it, and answers a single question: given everything that had happened, why is this the moment that counts? Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al, 2023) The reviewer has privileged information the agent never had in the moment: it can see the whole trajectory, including how things turned out, and it can relate what went wrong back to the domain policy the agent is graded against. It isn’t inventing new facts so much as rewriting the raw signal into words that carry more of its value, the same feedback, made legible. Our reflection and feedback prompts follow Liu et al., HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation (Liu et al, 2026) who introduce hindsight reflection as the privileged context.

Why is reflection needed? In knowledge work the rules live in a policy document, and a turn’s mistake is rarely self-contained, so the feedback for one turn is entangled with turns several steps earlier and cannot be isolated and passed through verbatim. In coding, a stack trace is already verbose and local enough to act as the signal on its own, and reflection may buy little there.

That reflection has to be written by something. A human could do it, but we want an automatic loop, so a model does. We use an off-the-shelf Gemini 3.1 Pro for this. Using a model keeps the step automatic enough to run over logged sessions in production, which is what makes this rung practical at scale. We believe that if the student model is big enough, the model itself can be used as the reviewer to generate the feedback.

Gold, if you’ll pay for it. At the top sits the actual answer. Spend the extra time and money on human graders and they’ll read a full trajectory and hand back the gold result: what the agent should have done. Worth being precise about what that means here: not a golden trajectory or a successful rollout to imitate, but the final actions that satisfy the task’s reward conditions, the reference tool calls, with their exact argument values, that leave the database in the state the grader checks for.

Feedback (per-turn, G = 1)	Pass@1	Std. dev.
Base model, untrained	0.23	± 0.02
Self-reflection	0.48	± 0.02
Gold (oracle)	0.05	± 0.02

Gold collapses because it trains the agent to act and never ask. Conditioned on the exact tool calls that satisfy the reward, the teacher never has a reason to ask the user anything: at every turn its correction amounts to execute the next action now. The student learns the executable half of that and stops reasoning about what it doesn’t know yet: it quits asking the user for information and just fires tool calls at the database. The signature is how much of the conversation is still conversation: user-facing turns fall from the base model’s ~27% of assistant turns to ~15% over twelve steps, while the reflection arm climbs to ~38%.

User-facing turns as a share of all assistant turns over training. The gold arm falls from about 27 percent to 15 percent by step 12, while the reflection arm rises to about 38 percent. — **Figure 7.** User-facing turns as a share of all assistant turns. Both arms start from the untrained model's ~27%.

Figure 8 shows that happening at a single decision. The reference path for this task is two exchange_delivered_order_items calls and nothing else, so at turn 1 the gold teacher demands the exchange, the final action of the task, before the user has been identified at all. The reflection teacher at the same position has no answer sheet, leaves the agent’s lookup alone, and rejects opening with a tool call at all, putting its mass on the first word of a question back to the user.

Figure 8. The agent's first tool call in one failed rollout, where it invented a placeholder email, re-scored under both teachers. Row 1 is the token emitted, red where that teacher disagreed; the three rows beneath are its top-3 there.

Which objective do you distill with?

What the feedback says and where it sits settle what the model sees; the loss settles in which direction it gets pulled toward the feedback-shifted distribution. That is the usual mode-covering-versus-mode-seeking tradeoff, MiniLLM: On-Policy Distillation of Large Language Models (Gu et al, 2023) and it bites differently on agentic rollouts. Three choices.

Forward KL, mode covering. The teacher weights the sum over the vocabulary, so the student is pushed to reproduce the whole feedback-conditioned distribution, tails included.

D_{\mathrm{KL}}(p_T \,\|\, p_S) \;=\; \sum_{v \in \mathcal{V}} p_T(v) \, \log \frac{p_T(v)}{p_S(v)}

Reverse KL, mode seeking. The student weights the sum, so it concentrates on the subset of corrected behaviours it already places mass on rather than spreading across all of them.

D_{\mathrm{KL}}(p_S \,\|\, p_T) \;=\; \sum_{v \in \mathcal{V}} p_S(v) \, \log \frac{p_S(v)}{p_T(v)}

Generalized Jensen–Shannon, the symmetric middle. Both distributions are measured against a mixture of the two, with $\alpha$ dialling between them. Writing $m_\alpha = (1 - \alpha)\, p_S + \alpha\, p_T$ ,

D^{\alpha}_{\mathrm{JSD}}(p_S,\, p_T) \;=\; (1 - \alpha)\, D_{\mathrm{KL}}(p_S \,\|\, m_\alpha) \;+\; \alpha\, D_{\mathrm{KL}}(p_T \,\|\, m_\alpha)

with $\alpha \to 0$ recovering forward KL and $\alpha \to 1$ reverse KL. See Agarwal et al. for the generalized JSD family this interpolation comes from.

Divergence	Pass@1	Std. dev.
Base model, untrained	0.23	± 0.02
Reverse KL (α = 1)	0.48	± 0.02
Forward KL (α = 0)	0.45	± 0.02
Jensen–Shannon (α = 0.5)	0.41	± 0.03

Reverse KL reaches the highest peak and holds it. Forward KL is close on the peak alone, but it crests early and then decays below the untrained model, so the peak understates the gap. Jensen–Shannon settles between the two and stays flat.

Eval-set Pass@1 over 24 training steps for one OPSD reflection·per-turn run at temperature 0.6, distilled three ways: reverse KL rises to about 0.47 and holds, forward KL peaks near 0.45 at step 9 then falls back to about 0.22, and Jensen–Shannon plateaus around 0.40. — **Figure 9.** One OPSD reflection·per-turn run distilled three ways toward the feedback-conditioned teacher. Eval-set Pass@1, Gaussian-smoothed (σ = 1.5).

Forward KL prices the deciding token at almost nothing. We took one scored turn where the next token chooses between opening a tool call and asking the user, and computed each objective’s exact per-token divergence at that position. The right answer there is the user-facing token, not a tool call: two fabricated authentication lookups have already failed, so the teacher puts almost all its mass on asking and almost none on acting again. Reverse KL is zero-forcing: its terms carry the log-ratio $\log(p_S/p_T)$ , which blows up wherever the student holds mass the teacher rules out, and deleting a wrong action is exactly that case. Forward KL is zero-avoiding instead. It is unbounded when the student misses teacher mass, but the term for mass the teacher doesn’t want is scaled by the teacher’s own probability, which here is near zero.

Figure 10. One scored turn where the next token chooses between a tool call and asking the user; the teacher puts 0.962 on I and 2.7e-5 on <tool_call>. Rows are the exact per-token divergence, log-shaded. Forward KL's one large term lands on transfer, a token reached only if the decision already went wrong.

So the two objectives can both be satisfied, by different routes, and probing the same decision on every checkpoint shows which route each one takes. Probabilities sum to one, so raising the probability of asking the user to the teacher’s 0.962 mechanically squeezes the tool call down to about 0.04. Forward KL bottoms out at 0.046 and goes no further: that is the floor mode-covering implies, reached as a side effect rather than as a correction, and it has almost no pressure left to close the remaining three orders of magnitude. Nothing holds the branch down either, and by step 22 it is back above 0.6. Reverse KL drives the same branch to zero within a few steps and it stays there.

Probability of emitting a tool call at the deciding token, probed on every checkpoint of the reverse-KL and forward-KL runs. Both fall from 0.83; reverse reaches 0.000 by step 10 and stays there, while forward bottoms out at 0.046 and climbs back to 0.63 by step 22. — **Figure 11.** P(the agent opens a tool call) at that same decision, probed on every exported checkpoint of both arms. The teacher's own value there is 2.7e-5.

Continual learning without forgetting

One of the quiet reasons to train on-policy at all is that it tends not to wreck what the model already knew. Off-policy SFT drags the model onto a distribution that isn’t its own and can overwrite unrelated skills, the classic catastrophic-forgetting Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting (Chen et al, 2025) failure, whereas on-policy methods keep training on the model’s own samples, which is far gentler RL’s Razor: Why Online Reinforcement Learning Forgets Less (Shenfeld et al, 2025) on everything you weren’t trying to change.

We set the test up as two sequential phases. In phase 1 we train OPSD on τ²-bench retail. In phase 2 we take that checkpoint, optimizer state and all, and continue training on telecom alone, while still evaluating retail at every step. The two domains are far apart: different tools, different rules, and none of telecom’s appear anywhere in retail. Telecom also has tools the agent cannot call at all, because they belong to the user’s own device, so it has to walk the user through operating them, a mode retail never exercises.

If distillation were quietly overwriting what the model knew, retail is where it would show: it receives no gradient at all through phase 2 while telecom training pushes the weights around. If on-policy training is doing its job retail should hold, and to the extent the skill OPSD sharpens is domain-general, it may even tick up.

Gain in eval-set Pass@1 for retail and telecom across 512 training rollouts, split at 256 by a dashed divider. Retail rises to about +0.21 by the divider and then stays near +0.22 for the rest of training. Telecom, trained only after the divider, rises from zero to about +0.24. — **Figure 12.** Each curve is that domain's gain in eval-set Pass@1 over its own score when its training began (retail 0.213, telecom 0.056), so both start at zero; Gaussian σ = 1.5 over faded raw draws.

Retail does not budge. It holds its phase-1 gain through all of phase 2, flat and pointedly not decaying, while telecom picks up a gain of its own of about the same size from a far lower base. Telecom here is trained, not unseen, so this is evidence about forgetting rather than zero-shot transfer, and on that question the answer is clean: sequential specialization accumulates Self-Distillation Enables Continual Learning (Shenfeld et al, 2026) instead of trading off.

Citation

Please cite this work as:

Goyal, Kartik, "On-Policy Self-Distillation: Continual Learning from Production Feedback",
Kartik Goyal: Research, Aug 2026.

Or use the BibTeX citation:

@article{goyal2026onpolicyselfdistillation,
  author = {Kartik Goyal},
  title = {On-Policy Self-Distillation: Continual Learning from Production Feedback},
  journal = {Kartik Goyal: Research},
  year = {2026},
  note = {https://kartikgoyal.ai/research/on-policy-self-distillation},
}

Appendix

The same turn, scored under two placements

Each row below is a single turn from one τ²-bench retail rollout, and the two columns score the identical token sequence, the model’s own output at that turn, against a teacher that read the same reviewer feedback. The only thing that differs is where that feedback sits in the teacher’s context: immediately before the turn on the left, and aggregated into the first user message on the right. Shading is the exact full-vocabulary reverse KL between student and teacher at each token, which is the OPSD loss term itself, so a darker token is one the feedback moved and a pale one is a token the teacher had nothing to say about.

Reading down the rows, the left column stays dark at every position while the right column fades to almost nothing past the opening turns. That is the reach failure from Figure 4 seen one token at a time.

Figure 13. Three turns from one τ²-bench retail rollout, each scored twice: feedback immediately before the turn (left) and the same feedback in the first user message (right). Per-turn is 2× stronger at turn 4, 248× at turn 10 and 32× at turn 12.

On-Policy Self-Distillation:Continual Learning from Production Feedback