A teacher who can see the standard answer may not always be able to give a useful next-step hint. This paper examines on-policy distillation: the student model receives only the problem and writes out its reasoning; a privileged teacher derived from the same checkpoint additionally receives the verified solution, then gives the distribution of the next token for the prefix the student has already generated. Earlier, OPSD kept this teacher fixed. The authors ask: after the student learns during training to look back and correct mistakes, can the teacher from the starting point still provide suitable supervision for these new trajectories?
The paper proposes Dynamic Co-Evolution (DCE): after each round of updates, the new checkpoint continues to serve as the student while, with gradients stopped, becoming the teacher for the next round that can see the standard answer. The paper also pairs this with Self-Refined Concise Learning (SRCL): the model rewrites its own on-policy solution into a shorter version; only rewrites that end naturally, are shorter than the original answer, are structurally valid, and have a correct answer enter training. The former changes the problem that the teacher remains stuck at its initial ability as the student trains, while the latter addresses the cost that repeated checking may lengthen output.
[Figure 1: A frozen teacher is more inclined to terminate than to reflect.] The paper gives fixed incorrect Qwen3-8B trajectory prefixes to both the evolving student (solid line) and the frozen teacher that can see the standard answer (dashed line) for scoring; the blue line is the probability of the end-of-sequence token, and the orange line is the probability of a reflection marker at positions where reflection has already appeared.
A single incorrect trajectory plot illustrates what the authors want to replace
Figure 1 is not an overall accuracy plot, nor is it a statistic from a newly generated sample. It uses fixed incorrect trajectories on AIME 2024, AIME 2025, and AIME 2026. At the endpoints of these incorrect answers, the average probability the frozen teacher assigns to the end-of-sequence token is 94.5%; at reflection positions that already appear in the trajectory, such as Wait, its average probability for a reflection marker is 22.0%. On this basis, the paper offers an interpretation: although the teacher can see the standard answer, it has limited local signal for how the current incorrect reasoning should continue to be corrected; after training, the student reflects more often, and the two distributions gradually diverge.
This is a probe of the frozen teacher's signal; it is not equivalent to saying that 94.5% of answers will be wrong or that 22.0% is the reflection success rate. The concrete method of DCE is also not to train another, larger teacher: after each round of parameter updates is completed, the new checkpoint simultaneously initializes the next round's student and privileged teacher, and the teacher branch does not backpropagate gradients. The standard answer is used only in the condition of the privileged teacher; in SRCL's rewrite generation stage, the model is not given the standard answer, and the standard answer is used to verify whether the rewrite is correct.
[Table 1: Qwen3-8B main results.] Table 1 in the paper reports side by side, under non-thinking, 32K, four mathematics tasks, Average@12, the accuracy and average generated tokens of Base, SFT, GRPO, OPSD, DCE, and DCE+SRCL; the thinking-mode base is listed separately in gray as a cross-mode reference.
65.97% belongs to the Average@12 setting of four competition mathematics tasks, non-thinking, 32K
The main table evaluates AIME 2024, AIME 2025, AIME 2026, and HMMT February 2025, sampling each problem 12 times; each of the four sets has 30 problems, decoding is in non-thinking mode, and the single-generation cap is 32K tokens. Here, Average@12 is the average accuracy across the four evaluations; it cannot be read as single-attempt accuracy, nor can it be directly extrapolated to general question answering or coding tasks.
Under this protocol, Qwen3-8B's base is 19.51%, GRPO is 20.28%, and the frozen-teacher OPSD is 30.35%. DCE reaches 65.76%, and DCE plus SRCL reaches 65.97%, 35.62 percentage points higher than OPSD. The paper also gives a control that only lengthens inference: OPSD-TTS forced to 16K tokens reaches 30.76%, still far below DCE's 65.76%. This supports the experimental conclusion that generating more tokens alone is not enough to reproduce this result; it does not prove that any longer reasoning is ineffective under other training schemes.
The effect of SRCL is mainly seen in the length–accuracy trade-off for 8B
From DCE to DCE+SRCL, the average generation length for 8B drops from 19,046 to 17,561 tokens, a reduction of 7.80%, while average accuracy rises slightly from 65.76% to 65.97%. Changes by individual task are not all increases: AIME 2024 drops from 74.72% to 73.89%, HMMT 2025 drops from 46.94% to 46.11%; AIME 2025 rises from 68.61% to 71.94%, and AIME 2026 drops from 72.78% to 71.94%. Therefore, what this table supports is that the average metric is approximately flat while output is shorter, not that SRCL improves accuracy on every problem set.
Results at smaller scales also indicate conditions of applicability. Qwen3-4B's DCE+SRCL is 61.88%, higher than OPSD's 22.85%, with average length falling from DCE's 19,360 to 17,265 tokens; Qwen3-1.7B, however, goes from DCE's 23.33% to 26.88%, while average length instead rises from 12,600 to 19,504 tokens. In the table, the 1.7B thinking-mode base is 36.88%, higher than this non-thinking result; the 8B thinking-mode base is 62.08%, lower than 65.97%. These gray rows are marked by the paper as cross-decoding-mode references, excluded from the main ranking, and cannot be used as a direct win-loss judgment under the same setting.
The conclusions this paper offers are bounded within the training and evaluation protocol
The paper trains on the 14,717 OpenThoughts mathematical reasoning problems used by OPSD and reports results at Qwen3 scales including 1.7B, 4B, and 8B, as well as a Gemma-4-12B-IT transfer experiment. The main results show that, under this set of mathematical training data, non-thinking decoding, and the specified sampling budget, DCE, which refreshes the privileged teacher round by round, together with verified short rewrites, can improve the accuracy–length combination for 8B and 4B.
What it has not yet answered includes: whether this teacher refresh applies to open-ended writing, real tool calls, or different training corpora; why the length of 1.7B rises; and whether the optimal weight is stable across models and budgets. The paper's own sensitivity experiments also show only local, non-monotonic changes within the tested interval. Describing 65.97% as the general reasoning ability of Qwen3-8B, or treating the probe numbers in Figure 1 as final performance, goes beyond the evidence this paper provides.
📄 Original title
Recursive Self-Improvement via On-Policy Distillation for Reasoning
🔗 Original link