Welcome to follow our latest work on asynchronous Agentic RL. This article focuses on a very fundamental but often overlooked problem in large-scale LLM reinforcement learning systems: when rollout and training are decoupled asynchronously, the old logits that PPO-type off-policy correction truly requires have often been lost.

Paper: Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
Link: https://arxiv.org/abs/2605.12070
Code: https://github.com/millioniron/async_fixedic

This might look like an engineering detail, but it directly affects two correction processes within the PPO objective that should be independent of each other: training-inference discrepancy correction and policy staleness correction. Once the old logits are lost, these two processes become entangled, rendering the originally semantically clear decoupled correction ineffective.

Centered on this problem, we systematically analyze several off-policy correction paradigms in existing asynchronous Agentic RL, discuss three paths for precisely recovering old logits, and propose a lower-cost, more scale-friendly approximation scheme: PPO-EWMA.

Our experiments show: on both Dense and MoE LLMs, the corrected PPO-EWMA achieves a better balance between performance and system overhead; while the idealized Snapshot scheme helps us validate a key judgment: the core of the problem is not "whether to do correction," but "whether the reference policy you are using for correction is semantically correct."

TLDR

In modern Agentic RL training systems, rollouts are often handled by inference engines like vLLM and SGLang, while training is handled by training engines like Megatron-LM and FSDP. Meanwhile, to boost throughput, training pipelines frequently adopt mechanisms such as asynchronous rollout, delayed updates, and partial rollouts.

This brings two distinct problems:

1. Training-Inference Discrepancy: Even if the rollout side and training side nominally use the same model version, the token probabilities may still differ due to discrepancies in kernels, precision, quantization, parallelism strategies, MoE routing, etc.

2. Policy Staleness: By the time the data is actually sent to the actor for an update, the behavior policy that generated these samples is often lagging behind the current training policy.

Theoretically, these two problems should be handled separately. Ideally, the total importance ratio should be decomposed into two parts: one for repairing training-inference discrepancy, and another for constraining policy staleness.

But in a real asynchronous system, this decomposition relies on a key prerequisite: you must be able to obtain the training-side old logits corresponding to when these tokens were generated.

Our core finding is: in asynchronous Agentic RL, this prerequisite often does not hold. Missing old logits will conflate discrepancy repair and staleness correction, destroying the original semantics of decoupled correction.

To address this problem, we did two things: analyze three system-level paths for precise old-logit recovery from a system perspective; and propose a revised PPO-EWMA approximate reference policy better suited for asynchronous windows from an algorithmic perspective.

On Qwen3-4B and Qwen3-30B-A3B, we found PPO-EWMA can stably outperform conventional Decoupled PPO and Linear_prox, and approaches the upper bound of the idealized Snapshot scheme on many tasks.

Research Motivation: Why does old logits become a core issue in asynchronous RL?

An important reason why PPO-type methods are stable is: its importance ratio and clipping mechanism default to knowing "which old policy these samples were actually generated by."

In synchronous or near-synchronous training, this is usually not hard to achieve. But in today's large-scale Agentic RL systems, the training paradigm has changed:

• • Rollout and training are physically decoupled
• • The rollout queue may span multiple versions
• • A trajectory might be a partial rollout, even spanning multiple parameter versions
• • When the actor updates, the training-side logits from earlier versions may already be unrecoverable

Thus a naturally intuitive decomposition loses its semantic prerequisite:

• • r_d: repairs training-inference discrepancy
• • r_s: repairs policy staleness

Ostensibly you are still doing decoupled correction, but in reality, the reference policy you are using is no longer the "true old policy." This brings up a critical question:

A Unified View: There Are Actually Two Different Types of Constraints in Off-Policy Correction

A core viewpoint of this paper is that existing PPO-style off-policy correction can be understood under a unified perspective: they are all trying to constrain two things, but the semantics of these two things are different.

The first type is the Discrepancy constraint. It focuses on how much the rollout-side distribution differs from the training-side distribution, used to filter out tokens that are "numerically unreliable" due to training/inference inconsistency. Intuitively, this constraint should be symmetric around 1, because it is unrelated to the sign of the advantage.

The second type is the Staleness constraint. It focuses on the distance between the current policy and the historical behavior policy, used to implement the trust-region constraint in the PPO sense, thus naturally carrying a clipping structure sensitive to the sign of the advantage.

In other words, although both types of constraints are written on the ratio, they are essentially not the same thing: one is fixing a "training/inference misalignment," and the other is controlling "policy update magnitude." They should not be conflated into the same proxy ratio for interpretation.

A Key Conclusion: Discrepancy Repair Cannot Substitute for Staleness Correction

Some approaches tend to assume that as long as the training-inference mismatch is well-controlled, many off-policy issues can be mitigated along the way. But our analysis shows this idea does not hold.

The reason is: the discrepancy mask is generally symmetric, whereas PPO clipping is typically an asymmetric constraint related to the sign of the advantage. Even if the two numerically look like "filtering abnormal ratios," they are not the same type of constraint and cannot substitute for each other.

Worse, once the old logits are missing, you are forced to use some proxy policy and simultaneously impose these two different objectives onto this proxy. The end result is not a cleaner correction, but two mechanisms interfering with each other: the mask changes the effective working range of the clip, and the clip also affects the distribution of token that are retained.

So from this perspective, the real problem in asynchronous Agentic RL is not "off-policy correction is not strong enough," but rather:

How to Precisely Recover Old Logits? Three System Paths

If the root of the problem lies in the loss of old logits, a natural thought is to retrieve them. We discuss three viable paths in the paper.

Snapshot-based version tracking is the most direct method: save historical parameter snapshots, and reload the corresponding version during training to explicitly restore the training-side old logits at the time the trajectory was generated.

This scheme has the cleanest semantics and is closest to the ideal state. However, it requires extra CPU/host memory and introduces frequent version switching; in partial rollout scenarios, a single sample may span multiple versions, further amplifying I/O and scheduling overhead.

A dedicated old-logit model is the second path: separately maintain a model for computing old logits, allowing the main actor to continue training while isolating the old-logit computation. It can overlap somewhat with the training phase, reducing blocking on the main training path, but requires extra model resources and more complex resource allocation.

Synchronization via partial rollout interruption is the third path: before a certain version actually disappears, temporarily interrupt the rollout, reclaim the partial trajectory, and use the old version still resident to compute the exact logits. This scheme avoids long-term storage of historical weights, but introduces synchronous stalls, disrupts rollout parallelism, and brings resource switching and scheduling perturbations.

All three schemes can solve the problem more precisely, but they point to the same fact: precise old-logit recovery is feasible, but the system cost is not low.

Low-Cost Approximation: Why Did We Choose Revised PPO-EWMA?

Since precise recovery is too expensive, a more realistic question becomes: can we find an approximate reference policy that, while not equal to the real old policy, is at least more reasonable than existing proxies?

Our answer is to use PPO-EWMA.

The core idea here is not "pretending to restore old logits," but constructing a smoother reference policy pi_prox that is closer to the center of the asynchronous version window, and using it concurrently as the reference for both staleness correction and discrepancy repair.

Compared to simple Linear_prox, we emphasize two points more.

First, choose the EWMA decay coefficient according to the staleness window. We do not fix a very large empirical decay, but align beta_prox with the actual asynchronous window length in the system, making this proxy policy closer to the "middle position" of the version interval from which the rollout data originates.

Second, add an automatic reset mechanism. While EWMA is smooth, it can also accumulate and become increasingly outdated. After too many historical versions accumulate, pi_prox / mu_old will drift further and further, causing a large number of points to be dropped by the Train-Infer Mask, ultimately crashing the training.

Therefore, we continuously monitor the proportion of tokens retained by the Train-Infer Mask; when this proportion falls below a threshold, we directly reset the EWMA reference to the current actor.

The essence of this design is: let EWMA provide a smoother reference normally, but once the reference drifts too far away, promptly return the center of gravity. This allows PPO-EWMA to retain low-cost advantages while avoiding long-term distortion of the proxy reference.

Experimental Setup: How Did We Validate This Problem?

We conducted experiments on two representative backbones:

• • Dense model: Qwen3-4B
• • MoE model: Qwen3-30B-A3B

Evaluation tasks cover multiple Agentic benchmarks, including retail/airline/telecom from tau^2-Bench, and in-store/delivery from VitaBench.

Meanwhile, we explicitly control the maximum version gap in asynchronous training, enabling the experiments to more centrally observe what problems old-logit missing brings, and how great the difference is between exact recovery and approximate reference.

The primary methods we compared include Decoupled PPO, Linear_prox, PPO-EWMA, and the idealized precise old-logit recovery scheme Snapshot.

Main Results: PPO-EWMA Achieves a Better Balance Between Performance and Cost

The overall experimental results are very clear.

On Qwen3-4B, PPO-EWMA stably outperforms Decoupled PPO and Linear_prox on multiple metrics: best pass@4 on retail, best pass@2 on VitaBench in-store, and tied for best on telecom.

On Qwen3-30B-A3B, PPO-EWMA still maintains its advantage: strongest performance on the airline split, best on retail pass@4, and best among practical methods on telecom avg@2 and pass@2.

More importantly, it has begun to approach the performance of the idealized Snapshot on multiple tasks. This indicates:

The table below summarizes the main results on dense Qwen3-4B and MoE Qwen3-30B-A3B. Snapshot* represents the idealized precise old-logit recovery; bolded items represent the best results among practical methods on the same backbone.

A Very Interesting Result: Precise Old Logits Are Indeed Better, but Really More Expensive

We also specifically measured system overhead, and the results are quite representative.

For Snapshot, although the correction is the most accurate, the system cost significantly increases: 4B model CPU storage is about 40 GB, 30B MoE model CPU storage is about 76.4 GB, and extra time overhead is about 25s and 150s respectively.

In contrast, the resource cost for PPO-EWMA is much lighter: about 7.9 GB for 4B, about 15.2 GB for 30B, and extra time is about 8s and 34s respectively.

In other words, Snapshot is more like an "ideal upper bound" and analytical tool, rather than the default scheme best suited for large-scale production deployment.

The table below shows the system cost difference between precise old-logit recovery and PPO-EWMA. Snapshot has cleaner semantics, but significantly higher storage and recovery time; PPO-EWMA provides a more realistic performance-cost trade-off.

The resource partitioning results for the dedicated old-logit model are as follows:

While maintaining a separate old-logit model can partially overlap computation, the benefit is highly dependent on the resource partitioning ratio and itself requires more system support, thus it is not always cost-effective. This is also why we ultimately emphasize the practical value of PPO-EWMA.

Threshold Analysis: Why Do Mask and Clip Interact?

Using the precise old logits recovered by Snapshot, we further analyzed the interaction between the discrepancy threshold and the stale-policy threshold.

Experiments show: a looser discrepancy threshold retains more tokens, leading to faster initial learning, but it also admits more highly biased off-policy tokens, making the later stages more prone to instability; a stricter threshold leads to slower initial progress, but the retained-token dynamics are more stable.

At the same time, the mask and clip are not two independent switches separate from each other. Under the same discrepancy threshold, a looser stale-policy threshold allows more problematic tokens to survive in the early stage, which in turn pulls down the Train-Infer Mask; after the clip becomes stronger later, it acts as a suppression on remaining updates, allowing the mask to recover.

This further indicates: when the reference policy is inaccurate, the coupling between the mask and clip becomes even more complex.

PPO-EWMA Ablation: Why Is Automatic Reset Crucial?

We also specifically analyzed the training dynamics of PPO-EWMA, focusing on three signals: task success, Train-Infer Mask, and PPO-CLIP ratio.

The results show that if only EWMA is used without reset, the proxy reference may gradually drift too far, ultimately causing the Train-Infer Mask to collapse.

After adding the automatic reset, just a small number of reset events are needed to pull the reference back to a reasonable region, while preserving the optimization gains brought by the EWMA in the early stage.

This result aligns very well with our intuition: using purely the current policy is not stable enough, purely historical smoothing may be too outdated; the reasonable approach is not to pick one, but to "smooth routinely, reset when too outdated."

The core viewpoint of this work

If summarized in one sentence:

Centered on this problem, we did three things:

1. Provide a unified view, clearly distinguishing between training-inference discrepancy correction and policy staleness correction

2. Point out that the missing-old-logit problem in asynchronous systems will destroy this decomposition

3. From the two directions of precise recovery and low-cost approximation, present a more systematic solution approach

Summary and Future Outlook

This article focuses on a very low-level but critical problem in asynchronous Agentic RL: missing old logits.

Our analysis shows that this problem directly destroys the semantic foundation of decoupled off-policy correction. Based on this, on one hand we discussed three system paths for precise old-logit recovery, and on the other hand propose a revised PPO-EWMA scheme that is more suitable for practical large-scale training.

Experimental results indicate: precisely recovering old logits indeed improves correction fidelity; but from a performance-cost trade-off perspective, PPO-EWMA is a more realistic and easier-to-deploy scheme.

There remain many directions worth advancing in the future, including construction of reference policies under even more extreme asynchronous windows, token-level version alignment in more complex partial rollout scenarios, joint modeling of MoE routing replay and old-logit correction, and a lighter-weight old-logit infrastructure for real-world industrial training stacks.

If you are also focusing on asynchronous LLM RL, Agentic RL, MoE RL, or Off-Policy Correction, discussions are welcome.