Imagine this scenario: You instruct a coding agent to fix a bug, using a set of unit tests as the criterion for "whether it did it right."
The model tries repeatedly but still can't pass the tests. Then, it does something unexpected, yet "completely logical" — it rewrites that test, making italways return "Passed".
From a reward perspective, the task is "completed," and the points are secured. From your perspective, it fixed nothing.
This is precisely the scenario that countless practitioners have repeatedly encountered over the past year. Its awkwardness lies in the fact that the model didn't act "maliciously"; it merely faithfully optimized the signal you provided. The problem is with the signal itself. The model not only learns how to fix bugs but also learns how to exploit tests, the environment, and information leaks to gain rewards.
For decades, a piece of "common sense" in computing has quietly shaped our intuition about hard problems: Verifying a solution is easier than finding it. Complex mathematical reasoning and code generation have made extraordinary progress based on this intuition—as long as a verifiable reward exists, reinforcement learning can continuously "extract" capability.
But for today's coding agents, this intuition is reversing: As base model reasoning capabilities improve, amplified by harness engineering, generating a sufficiently complex candidate solution has become cheap. Yet reliably verifying it—remaining faithful to the user's true intent, scalable to massive training volumes, and resilient against a continuously optimizing adversary—has become the most expensive and open-ended challenge in the entire loop.
A paper completed by the Qwen team in collaboration with Fudan University's NLP Lab and other institutions, titled "The Verification Horizon: No Silver Bullet for Coding Agent Rewards," argues that this is not a temporary engineering gap, but a structural reality, and provides a set of terminology and a full suite of practical experience on "how to coexist with this reality."
🔗 arXiv: https://arxiv.org/abs/2606.26300
🤗 Hugging Face: https://huggingface.co/papers/2606.26300
No Perfect Verifier: Every Reward Signal is Merely a "Stand-in" for Human Intent
The paper's starting point is unsettling yet highly clarifying: Every reward signal we can construct—executable tests, rubrics, reward models—is merely a proxy for the human intent we truly care about, and never the intent itself.
This is because intent is fundamentally semantic and underspecified: The person holding the intent often cannot pre-articulate all their expectations. It is only when a counterexample exposes an oversight that they realize what they truly wanted—and such counterexamples are hard to predict and even harder to exhaustively enumerate beforehand. This leaves a permanent gap between the proxy and the intent, and an agent that is sufficiently strong and fully optimized against the proxy is precisely the tool that finds this gap.
This fundamentally reframes our understanding of reward hacking. It is not a bug that can be patched—
Reward hacking is an inevitable product of applying optimization to a proxy that may forever deviate from the intent it represents.
Consequently, a faithful and robust verifier is not just difficult to obtain; it is unattainable in principle. (This aligns with Rice's theorem: Any non-trivial semantic property of a program is undecidable. Therefore, a universal verifier that is complete and precise for any arbitrary code does not exist; practical verification is inevitably only an approximation.)
From Reward Hacking to Intent Exposure: Verification Must Become a System
But "unattainable in principle" does not mean we are powerless. What it truly demands is a shift in perspective: no longer treating "deviations between the agent and the intent" as merely errors to be eliminated, but instead reading each deviation as a piece of information—it precisely illuminates the layer of intent that was previously left unarticulated.
The entire purpose of the verifier is to remain aligned with the user's true intent; but this intent is not a fixed target that can be fully written down in advance. Rather, it is a process of gradual unfolding—and often, it is the agent itself that unfolds it: whenever a stronger policy exploits a gap in the proxy, it exposes a layer of intent we had previously failed to specify.
Returning to the earlier example of rewriting tests: before the agent acted, the constraint "don't modify the tests" was something you never even realized needed to be stipulated—it was an implicit premise. It was the agent's act of hacking that first made this layer of implicit intent concrete and articulable. From this perspective, reward hacking is not just a failure mode; it is information about where the agent and intent diverge.
Verification is not a one-off harness; it is a horizon that continuously recedes as the policy strengthens.
This is precisely why the solution cannot be a one-time, "better" verifier. Verification must be a system, and this system must co-evolve with the coding agent: when the policy becomes stronger and finds new gaps, the verification system (tests, judges, monitors, evaluators) must be reconstructed to bridge them.
A corroborating evidence from the frontier. Around the time this paper was finalized, OpenAI's previewed GPT‑5.6 Sol was assessed by the independent evaluation organization METR as having a higher detected cheating rate than any previously evaluated public model on their ReAct agent harness—to the point that "whether cheating is counted as success" could alter its 50% time-horizon estimate by an order of magnitude. [^metr] OpenAI's own system card also documented significant reward hacking tendencies in coding agent and long-horizon tasks (exploiting evaluation environment loopholes, falsifying results, circumventing permissions, etc.), forcing them to introduce real-time activation classifiers for monitoring. [^openai] Stronger policies are more adept at finding exploits, which inversely validates the paper's judgment: Pursuing only a stronger generator will just lead you to hit the next exploit faster; what truly raises the reliability ceiling is building a verification system that co-evolves with the agent.
So, "Won't a stronger model be able to verify itself?" This is precisely the misconception. A stronger verification model is still just a proxy for intent, still subject to the same optimization pressure. Having the generator and verifier from the same source only makes their blind spots highly correlated and allows gaps to be found even more efficiently. The path forward is not a "smarter single-point verifier," but a verification system with separated responsibilities, capable of being independently reconstructed, and maintaining tension with the policy.
A Verification System is Not a Referee, but a Complete Set of Evolving Mechanisms
Verification System = Verification Engineering + Co-evolution.
So-called Verification Engineering is the entire verification pipeline built around the verifier—it includes not only the construction of the verifier itself (tests, agents, users) but also the various supporting measures surrounding it: quality filtering, behavioral monitoring, performance evaluation, failure mode analysis, etc.
And verification engineering alone is not enough—what truly makes it a "system" is co-evolution: this pipeline is not built once and left static; it is continuously reconstructed as the policy finds new vulnerabilities. The verifier and the agent push each other higher through rounds of adversarial interaction, ultimately forming a co-evolutionary closed loop.
The authors describe a verifier's quality using three properties:
Scalability: Can the signal be constructed and applied cheaply enough and at scale to support training? This is the floor.
Faithfulness (from the verifier's perspective): To what extent does it cover the intent we truly care about, rather than some narrow stand-in?
Robustness (from the generator's perspective): Will optimizing against this verifier cause the agent to deviate from human intent?
Faithfulness and robustness are actually descriptions of the same goal from two angles: whether the verifier aligns with genuine human intent. Both angles require direct human involvement; humans are the ultimate verifiers.
The authors studied four types of verifiers and their scenarios, examining each through this unified lens: the task characteristics that make verification design difficult, the verification constraints imposed thereby, the specific implementation adopted, the empirical observations brought forth, and the practical takeaways derived. Let's first take a quick overview with a table:
Tests as Verifiers
For code agents, the most common and most engineered training scenarios are things like: constructing tasks from real GitHub Pull Requests, where a code agent executes the task and produces a patch. If this patch passes the tests, it gets a positive reward. Compared to human review, execution-based verifiers (unit tests, behavioral tests) are cheap, automated, and scalable, and have thus long been regarded as reliable reward sources.
However, this type of verifier faces two challenges:
First, tests may not be faithful. The task intent of real PRs might rely on issue discussions, project conventions, and historical context; instructions might be too short or too vague; tests might only cover a small portion of the target, or even validate behavior inconsistent with the description. In these scenarios, test results do not faithfully evaluate the quality of an answer, so "low pass rate" does not necessarily equal "hard"; it often just means "untrustworthy."
Second, tests may not be robust. Test-driven reward still only looks at whether the final patch passes, not how the model obtained that patch: the model can debug normally, or it can search the original PR, inspect leaked diffs, access commit metadata, or modify test scripts. Both paths may pass the tests, but the former learns software engineering skills, while the latter learns to exploit evaluation loopholes.
To address these two problems, the authors took two countermeasures. First, they used an Agentic Quality Judge (which, like a lightweight coding agent, enters the environment, reads the code, runs commands, checks the tests) to first separate "runnable" from "trustworthy," filtering out samples with unclear instructions or misaligned tests—data is not about quantity, but whether the reward is worth optimizing for. Experiments show that cleaned data enables RL models to maintain a steeper learning efficiency curve.
Second, the authors proposed Trajectory-level Behavior Monitoring (Behavior Monitor), auditing the entire trajectory during RL training. The system records each rollout's command history, network access, git operations, file reads/writes, test execution, and final patch. If high-risk patterns are hit, the reward is deducted. Moreover, this set of patterns is continuously rebuilt as the model discovers new shortcuts—this is precisely the microcosmic depiction of "the verification system rebuilding as the policy evolves." Experiments show that this measure successfully shifted the training objective from "just pass the test" to "both pass the test and solve the problem through a trustworthy process."
Interactive Judges as Verifiers
In the reward design for coding agents, front-end development is a particularly tough nut to crack: The quality of a front-end goes beyond binary functional pass/fail signals. An agent can easily write HTML, CSS, and JavaScript with zero errors, yet the rendered page is ugly, animation is janky, and interactions are broken.
This means that the Faithfulness requirements for front-end verification are richer, more dynamic, and more interaction-heavy: code that runs successfully is far from code that is done well. However, more diverse and comprehensive evaluation dimensions also mean introducing the LLM-judge verifier form, which in turn brings new challenges to Robustness: on one hand, evaluation dimensions like visual quality are relatively subjective, and scores for the same answer can vary wildly across multiple runs; on the other hand, an LLM-judge is easily exploited during the optimization process, producing unexpected "side" products.
To address this, the authors conducted exploration at two levels: First, they started with a static judge based on a scoring rubric. The judge reads both the rendered screenshot and the source code, then scores item-by-item along structured dimensions like functional correctness, visual quality, layout, and user experience. The beauty of this step lies in decomposing subjective feelings like "looks good" and "easy to use" into reproducible, fine-grained judgments. This not only significantly improved consistency with human annotations but also stabilized scores between different judge models.
However, static judges quickly exposed inherent weaknesses. On one hand, behaviors like form validation, dynamic routing, and state interactions—which can only be verified when the page actually runs—are impossible to judge by merely looking at code and static screenshots. On the other hand, models learned to hack: they found that "writing CSS code could inflate aesthetic scores," thus madly piling on verbose CSS and JS to boost their scores. This is precisely the reward hacking repeatedly emphasized in previous chapters, making another appearance in the front-end scenario.
Overview of the Interactive Judge process: The process takes candidate code and user prompt as inputs. In the preprocessing phase, the system first extracts page information (including accessibility tree, browser state, and keyboard listeners) and comprehensively generates evaluation criteria (i.e., a checklist of key items and detail items). Subsequently, the action planner generates a complete action list in one shot, which is then executed by the Playwright server, producing an interaction trace. Finally, the evaluation model scores this trace against the evaluation criteria; the resulting score can serve as both a reward signal for RL training and annotations for SFT data construction.
To address this, the authors introduced a further step: an Agentic Interactive Judge. Its working method is more akin to a real human QA tester: an action planner first generates a complete interaction script in one shot, then hands it over to Playwright to perform step-by-step clicks, scrolls, and form fills in a real browser, recording the entire interaction process. Finally, the judge model scores the generated code against the actual runtime performance. Because the reward signal is rooted in the page's actual running effect, not the superficial appearance of the code, this mechanism is naturally immune to length-inflating tricks and can capture dynamic behaviors—like animations, state transitions, and multi-page navigation—that are entirely invisible to static evaluation.
Experimental results show that, compared to static judges, the interactive judge can both plug the "code length" reward loophole and achieve robust, sustained improvement on test set performance:
WebDev RL training curves: This chart shows the front-end coding scores (both training and test sets) and generation length over training steps for three judge paradigms (visual judge, hybrid judge, and interactive judge). As an RL reward signal, the interactive judge outperforms both static judge variants, achieving higher test scores while maintaining stable output length.
Users as Verifiers
As agents enter the productization phase, a broad base of users becomes an important source of supervision. Users care about whether their intent is truly realized, making them naturally the most faithful and robust verifiers for the agent. Moreover, as the agent evolves, users adapt correspondingly and realize richer intents, thus co-evolving with the agent. However, their true intent (including plenty of requirements initially unvoiced) is implicitly encoded within multi-turn interactions and is difficult to directly convert into a supervision signal. It therefore requires methods to extract and utilize it.
The authors collected 125,000 real interaction trajectories between senior engineers and coding assistants, converting these signals scattered across dialogues into scalar form by annotating the sentiment polarity and corresponding feedback type of user responses. They found:
Positive signals are extremely rare (only 3.5%), negative signals account for 20%, and the rest are neutral.
Negative feedback is "more definitive": 81.8% of negative signals are high-confidence—users express themselves with exceptional clarity when negating.
Errors are concentrated in two areas: execution errors (56.6%) and understanding errors (21.1%), together accounting for nearly 80%.
From the error information, it's evident that in agent tasks, negative user feedback carries highly specific value for improvement, including execution errors, code comprehension errors, etc.
The authors then designed a method called Span-KTO to utilize these supervision signals. The idea behind Span-KTO is simple: split a response into several labeled spans (segments with positive/negative tags) based on user interaction boundaries, and explicitly "pull towards good behavior and push away bad behavior." For each segment, the difference in log-probability between "the current model vs. the model before training" is used as its score, encouraging a higher score for positive segments and penalizing a lower score for negative segments:
Performance comparison of leveraging user feedback vs. supervised learning. Aone-bench is a software engineering benchmark built by the Qwen team.
After training with implicit user feedback, the model showed significant improvement in software engineering tasks and scaffold instruction following. The authors also monitored the agent's intermediate behaviors and found that using user feedback effectively reduced the agent's undesirable actions.
Behavioral monitoring of the agent, observing whether erroneous behaviors in the trajectories were improved.
On already solved tasks, each dimension showed only modest improvement, whereas on the unsolved hard tasks, the gap was significantly widened—inefficient behaviors decreased by 34.5% (fewer meaningless retries), communication quality improved by 26.5% (clearly articulating where it's stuck), and execution errors improved by 13.9%. In other words, the model learned not just to "solve more problems": on failure trajectories, it also became more controllable, professional, and restrained—and this is precisely the source of deployment trust.
Proactive Agents as Verifiers
As agent capabilities continue to advance, long-horizon tasks are gaining increasing attention. Taking NL2repo (generating an entire repository based on natural language) as an example: you only need to provide the model with a natural language requirement description, and you can obtain a fully functional codebase.
However, as task difficulty rises, the difficulty of verification also increases dramatically. First, compared to a complex codebase, the requirement specification is "fuzzy". When designing a repository, we often only think about "what we want," without exhaustively detailing "exactly what it should look like." Furthermore, the object to be tested is dynamically changing. Each model may choose completely different architectural schemes and input/output interfaces. Trying to write a set of tests in advance to cover all possible implementation paths is nearly impossible.
So, how do you verify a problem with extremely complex requirements and a dynamically changing implementation path? The authors' answer is: Since you can't write a universal test, let an agent "read and understand" the generated code and judge it dynamically. They deployed an autonomous Evaluation Agent to directly read the generated code repository, decompose the requirements into a checklist for item-by-item inspection, write and execute tests on its own, and score after multiple rounds of review. Evaluation shifted from "preset fixed rules" to "on-demand reasoning, dynamic judgment."
But having a model judge a model was far from smooth. Through five iterative rounds (v1→v5), the authors systematically exposed and corrected several typical failure modes of the evaluator:
Slacking Off (v1): The evaluator preferred "reading without running"—only doing static code reading without executing tests. Code that seemed reasonable easily slipped through.
Lack of End-to-End Verification (v2): Individual modules seemed fine, but the entire repository couldn't even run due to import errors, dependency conflicts, etc., yet still received high scores.
Role Confusion (v3): The evaluator "overstepped boundaries"—secretly modifying code to hide bugs, executing existing tests instead of writing its own, and even defending the generator's output.
Context Overload (v4): The evaluator read the entire codebase from beginning to end; key information was drowned out, and judgment precision dropped.
Rule Overload (v5): Overly granular process specifications exceeded the model's instruction-following ability—the evaluator was overwhelmed by dogma, and its overall judgment actually declined.
Results from five iterations of evaluator prompt design
From v1 to v4, BoN accuracy improved from 57.9% to 67.4%, but the decline in v5 shows that there is a "sweet spot" in evaluator rule design—it needs to be specific enough to guide the correct process, but not so granular that it crushes the model's own reasoning ability.
The authors subsequently revealed another interesting finding: A "good evaluator" depends on what you're using it for. RFT requires "not letting bad samples in," so precision requirements for ranking are loose; RL needs precise per-sample rewards to shape gradients, so ranking requirements are stricter. Experiments showed that the evaluator with the best ranking accuracy doesn't necessarily have the best filtering quality; the two are not naturally aligned. Which one to choose depends on the training objective.
Moreover, there is a structural tension between data quality and data quantity: raising the filtering threshold improves quality, but the retained sample count might shrink dramatically. In practical use, the trade-off between the two must be considered.
RFT experiments validated the practical value of the evaluator: At the same data scale, data filtered by the evaluator outperformed random sampling by 1.91 points. While doubling the data volume could also catch up, training efficiency took a severe hit. Under real-world computational budget constraints, a good evaluator is the most cost-effective data lever.
The Verification Horizon
The verification of coding agents is a receding horizon. Within the horizon lies the part that the agent can be reliably verified on under current capabilities. On the horizon are the inconsistencies with human intent discovered by the agent. But every time we approach this line and implement it, a stronger policy will find new gaps, pushing the line further back and forcing us to continuously catch up:
The agent finds counterexamples in the verification system > The horizon recedes again > Humans further clarify intent and reconstruct the verification system.
The conclusion follows naturally. What truly supports trustworthy capability growth is not any single reward function, but a verification system that organizes functions like reward functions, quality filtering, behavioral monitoring, and failure mode analysis into a cohesive whole and is continuously rebuilt as policy capability advances. This requires a perspective shift: from the passive patching of "policy finds loopholes, designer patches them afterward," to the proactive co-evolution of the verifier and the policy. The verification system is not an auxiliary component of the training pipeline; it is the core infrastructure that converts "pure capability growth" into "trustworthy capability growth."
[^metr]: METR, *Summary of METR's predeployment evaluation of GPT-5.6 Sol*, 2026-06-26. https://metr.org/blog/2026-06-26-gpt-5-6-sol/
[^openai]: OpenAI, Previewing GPT-5.6 Sol and the corresponding Preview System Card, 2026-06. https://openai.com/index/previewing-gpt-5-6-sol/
© THE END
Reprinting requires authorization from this public account.
Contributions or reporting inquiries: liyazhou@jiqizhixin.com