Abstract
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BinEval, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement.
Across SummEval, Topical-Chat, and QAGS, BinEval matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BinEval better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BinEval provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.
Keywords: large language models, evaluation, prompt optimization, interpretability
1 Introduction
The rapid progress of large language models (LLMs) has made generation easy and evaluation hard. Modern systems can produce fluent, contextually appropriate outputs across tasks such as summarization, dialogue, reasoning, and instruction following, but evaluating those outputs remains a major bottleneck. Human evaluation is slow and expensive, lexical metrics such as ROUGE, BLEU, and BERTScore miss semantic correctness and factuality, and holistic LLM judges often return opaque scores that are difficult to diagnose.
This bottleneck is especially costly in iterative development. Comparing prompts, models, or decoding strategies requires feedback that is not only accurate but also actionable. A single scalar score is often insufficient: if a summary receives a mediocre rating, it is still unclear whether the problem is factual inconsistency, weak relevance, missing content, or poor fluency.
Our premise is simple: instead of asking a model for one broad judgment, ask it a set of small, checkable questions. We therefore propose BinEval, which decomposes each evaluation criterion into atomic yes/no questions and aggregates the resulting verdicts into interpretable scores. This decomposition turns evaluation from a black-box verdict into a structured diagnostic signal, making it easier to inspect, debug, and improve both evaluators and generators.
BinEval has three components. First, a meta-prompt decomposes a task prompt into atomic questions organized by evaluation dimension. Second, an evaluator answers each question independently and aggregates the answers into per-dimension and overall scores. Third, a two-phase optimization loop improves both evaluator prompts and generation prompts using question-level feedback.
We evaluate BinEval on SummEval, Topical-Chat, and QAGS, and we study iterative prompt updating on summarization and IFBench.
Our contributions are:
- A general framework for interpretable evaluation. We decompose evaluation criteria into atomic yes/no questions, yielding a task-agnostic and modular method.
- Strong performance without task-specific training. BinEval matches or exceeds trained evaluators and holistic LLM judges on SummEval, Topical-Chat, and QAGS.
- Iterative prompt improvement. We introduce a two-phase optimization loop that improves prompts for both summarization and IFBench.
- Debuggable scores. Each BinEval score is grounded in individual verdicts with explanations, making evaluator behavior easier to inspect and diagnose.
3 Method
We present BinEval in three parts: binary question generation (Section 3.1), binary evaluation and scoring (Section 3.2), and iterative prompt optimization (Sections 3.3 and 3.4).
3.1 Binary Question Generation
Let T denote a task prompt defining the generation requirements, such as a summarization instruction, a dialogue system prompt, or an instruction-following specification. We define a decomposition function that maps T to a set of binary questions: 𝒬 = ℱ_LLM(T; M) = {q₁, q₂, …, q_N}, where M is a meta-prompt that instructs an LLM to perform a two-step decomposition.
Step 1 – Summarize. We first summarize the task prompt T into an explicit set of requirements ℛ = {r₁, r₂, …, r_K}. Each requirement r_k captures a distinct evaluation criterion, such as whether the output includes a key piece of information or obeys a formatting constraint. This summarization step is intended to help the model form a coherent representation of the full task before attempting finer-grained decomposition.
Step 2 – Decompose. For each requirement r_k, we generate one or more binary questions such that answering “yes” indicates the output satisfies the requirement and answering “no” indicates a violation. Requirements that implicitly contain multiple sub-tasks are decomposed into separate questions, and each question is paired with a concise violation example to clarify the negative case. This design is motivated by prior work showing that complex reasoning is often improved by decomposing a task into simpler sub-problems that can be solved sequentially or modularly. In our setting, the same intuition suggests that evaluation becomes easier when the model answers targeted binary questions about simplified sub-tasks rather than making a single holistic judgment.
The questions can be organized into evaluation dimensions. For a set of dimensions 𝒟, such as coherence, consistency, fluency, and relevance, the questions partition as 𝒬 = ∪_{d ∈ 𝒟} 𝒬_d, where 𝒬_d contains questions specific to dimension d. The meta-prompt M is task-agnostic: the same meta-prompt generates appropriate binary questions for summarization, dialogue, instruction following, or any other task, with only T changing.
3.2 Binary Evaluation and Scoring
Given an evaluator LLM E, an input x such as a source document, a transcript, or an instruction, an output y such as a generated summary, a dialogue response, or a completion, and a binary question q_i, we define the binary evaluation function f_E(x, y, q_i) ∈ {0, 1}, where f_E(x, y, q_i) = 1 if the evaluator answers “yes” and 0 otherwise. Alongside each binary verdict, the evaluator produces a natural-language explanation e_i, enabling interpretability.
The per-dimension score for dimension d is S_d(x, y) = (1/|𝒬_d|) * Σ_{q_i ∈ 𝒬_d} f_E(x, y, q_i). The overall score across all N questions is S(x, y) = (1/N) * Σ_{i=1}^{N} f_E(x, y, q_i). Both scores lie in [0, 1], where 1 indicates all criteria are satisfied. To enable comparison with existing evaluation frameworks that use different scales, the scores can be mapped from [0, 1] to any target interval [a, b] via affine scaling: S'(x, y) = S(x, y) * (b − a) + a.
3.3 Cross-Model Prompt Update
BinEval’s binary question framework enables cross-model prompt update between evaluators. The key insight is that disagreements between a source evaluator and a target evaluator on specific binary questions provide a fine-grained signal for improvement: unlike holistic score differences, binary question disagreements identify exactly which criteria are being judged inconsistently across models. This makes it possible to use a stronger source model as a reference and iteratively update the prompt of a different, typically weaker, target model until its evaluator behavior matches the source more closely. Moreover, it is useful for updating prompts to maintain similar performance when migrating a model to a different family of models.
Let E_src denote a source evaluator, treated as the reference model, and let E_tgt denote a target evaluator whose prompt P_E we wish to improve. Let P_E^(t) denote the target evaluator’s prompt at iteration t.
At each iteration t, the optimization proceeds in five steps:
- Evaluate. For each test case (x_j, y_j), obtain binary evaluations from both models: A_j^src = {f_{E_src}(x_j, y_j, q_i)}_{i=1}^{N}, A_j^tgt = {f_{E_tgt}(x_j, y_j, q_i; P_E^(t-1))}_{i=1}^{N}.
- Identify disagreements. Compute the set of questions on which the evaluators disagree: Δ_j = {q_i ∈ 𝒬 : A_j^src(q_i) ≠ A_j^tgt(q_i)}.
- Extract lessons. A note-taker LLM L_note analyzes each disagreement in context, extracting generalized lessons: ℒ_j = L_note(x_j, y_j, A_j^src, A_j^tgt, Δ_j). A semantic deduplication function merges similar lessons: 𝒟edup(ℓ_new, ℳ) = merge(ℓ_new, ℓ_k) if ℓ_new ∼ ℓ_k, otherwise add(ℓ_new). The final set of unique lessons is ℒ_unique = 𝒟edup(∪_j ℒ_j).
- Update prompt. For each unique lesson ℓ_k ∈ ℒ_unique, an updater LLM identifies the relevant substring s_k in the current prompt and produces a revised substring s'_k that incorporates the lesson: P_E^(t) ← P_E^(t).replace(s_k, s'_k).
The loop terminates when the target evaluator’s scores match the source evaluator’s scores within a tolerance ε across all dimensions: |S_d^{tgt, (t)} − S_d^{src}| < ε ∀ d ∈ 𝒟, or equivalently, when the target evaluator meets or exceeds the source evaluator on all dimensions. The full algorithm is shown in Appendix 1.
3.4 Self Prompt Update
The same binary question framework can also be used for self prompt update in generation. Instead of aligning one evaluator to another model, this procedure iteratively improves a generator by using evaluator-identified failures as feedback on its own outputs. Given a generation LLM L_G with prompt P_G^(t) at iteration t:
- Generate. Produce outputs using the current prompt: y_j^(t) = L_G(x_j; P_G^(t)).
- Evaluate. Score each output using the potentially already-improved evaluator and collect failing questions: ℰ_j = {(q_i, e_i) : f_E(x_j, y_j^(t), q_i) = 0}, where e_i is the evaluator’s explanation for the failure.
- Extract lessons. A note-taker LLM analyzes the evaluation errors in context: ℒ_j = L_note(x_j, y_j^(t), ℰ_j).
- Deduplicate and update. Apply the same semantic deduplication and prompt rewriting procedure used for evaluator optimization, but now to P_G.
The generation loop terminates when no evaluation errors remain or when the maximum number of iterations is reached.
4 Experimental Setup
We design two complementary sets of experiments. Part I evaluates BinEval’s performance on established benchmarks with human annotations. Part II demonstrates the iterative prompt-updating mechanism on both an unverifiable task and a verifiable task. Across these experiments, we use gpt-oss-120b and Claude Sonnet 4. To reduce randomness on LLM responses, we set the temperature to 0 in all experiments and report the average over two runs.
4.1 Metrics
For evaluation quality, we report Spearman’s rank correlation (ρ), Kendall’s rank correlation (τ), and Pearson correlation (r) between method scores and human judgments at the summary level.
4.2 Part I: Evaluation Quality Validation
We follow the evaluation protocol of UniEval and evaluate on three established benchmarks.
SummEval. A benchmark of 100 CNN/DM source articles, each summarized by 16 different summarization models, yielding 1,600 summary-level annotations. Human evaluators rated each summary on four dimensions: fluency, coherence, consistency, and relevance. Ratings are on a 1–5 Likert scale.
Topical-Chat. A benchmark of 60 dialogue responses generated by 6 dialogue models, annotated on six dimensions: naturalness, coherence, engagingness, groundedness, understandability, and an overall quality rating. Following Zhong et al., we use four of these aspects.
QAGS. A benchmark specifically targeting hallucination evaluation in summarization, comprising 235 samples from CNN/DM and 239 from XSum. Annotators rated the consistency of each summary with respect to its source document.
4.3 Part II: Iterative Prompt Updating
We evaluate BinEval’s iterative prompt update mechanism (Algorithm 1) on two tasks: evaluator prompt optimization on SummEval, which is unverifiable in the sense that there is no programmatic gold checker, and generation prompt optimization on IFBench, which is verifiable via executable constraint checkers. For SummEval, we test two update modes: self-update, where a single model (gpt-oss-120b) improves its own evaluator prompt using failures against human judgments, and cross-model update, where a stronger model (Claude Sonnet 4) serves as the reference evaluator and lessons from disagreements are used to update the target model’s prompt. See Appendix B for detailed experimental setups.
5 Results
5.1 Evaluation Quality: SummEval
| Method | Coherence | Consistency | Fluency | Relevance | Average |
|---|---|---|---|---|---|
| ROUGE-1 | 0.167 / 0.126 | 0.160 / 0.130 | 0.115 / 0.094 | 0.326 / 0.252 | 0.192 / 0.150 |
| BERTScore | 0.284 / 0.211 | 0.110 / 0.090 | 0.193 / 0.158 | 0.312 / 0.243 | 0.225 / 0.175 |
| MoverScore | 0.159 / 0.118 | 0.157 / 0.127 | 0.129 / 0.105 | 0.318 / 0.244 | 0.191 / 0.148 |
| BARTScore | 0.448 / 0.342 | 0.382 / 0.315 | 0.356 / 0.292 | 0.356 / 0.273 | 0.385 / 0.305 |
| UniEval (T5) | 0.575 / 0.442 | 0.446 / 0.371 | 0.449 / 0.371 | 0.426 / 0.325 | 0.474 / 0.377 |
| G-Eval (GPT-4) | 0.582 / 0.457 | 0.507 / 0.425 | 0.506 / 0.455 | 0.547 / 0.433 | 0.514 / 0.418 |
| G-Eval (gpt-oss) | 0.451 / 0.392 | 0.559 / 0.527 | 0.217 / 0.203 | 0.515 / 0.446 | 0.436 / 0.392 |
| UniEval (gpt-oss) | 0.237 / 0.208 | 0.489 / 0.476 | 0.000 / 0.000 | 0.288 / 0.256 | 0.254 / 0.235 |
| BinEval (gpt-oss) | 0.523 / 0.448 | 0.585 / 0.548 | 0.252 / 0.235 | 0.428 / 0.366 | 0.447 / 0.399 |
| BinEval (Claude) | 0.652 / 0.541 | 0.655 / 0.615 | 0.540 / 0.470 | 0.404 / 0.339 | 0.563 / 0.491 |
Table 1 shows a clear ranking across evaluation paradigms. BinEval (Claude) is the strongest method overall, achieving the best average Spearman and Kendall correlations and leading on coherence, consistency, and fluency. The largest gain is on consistency, where BinEval reaches 0.655 / 0.615, suggesting that decomposing factual quality into multiple targeted checks is especially effective for summary evaluation. Relevance remains the main exception: G-Eval (GPT-4) is best on that dimension, indicating that some broader semantic judgments are still harder to capture with binary decomposition.
The additional gpt-oss runs clarify why decomposition matters. Under the same backbone, BinEval (gpt-oss) outperforms both G-Eval (gpt-oss) and UniEval (gpt-oss) on average, driven by large gains on coherence and consistency. G-Eval with gpt-oss remains viable on numeric-scale dimensions such as consistency and relevance, but its fluency performance collapses. UniEval with gpt-oss is weaker still, with near-zero fluency correlation, showing that a single yes/no question is often too coarse for a general-purpose model. Overall, SummEval supports the core claim of the paper: multiple binary questions provide a more robust and transferable evaluation signal than either a single holistic score or a single Boolean judgment.
Figure 1 gives a more nuanced view of these gains. The figure presents violin plots of score distributions on SummEval across four evaluation dimensions comparing human annotations with different methods. BinEval is visually closest to the human distributions on consistency, where it largely matches the human concentration near the upper end while still retaining some low-scoring mass; this mirrors its largest correlation advantage in Table 1. Across dimensions, BinEval (Claude) is generally among the methods most closely aligned with human judgments in central tendency and spread, with its strongest match on consistency. UniEval and G-Eval exhibit narrower, more concentrated distributions, suggesting weaker discrimination across systems. The gpt-oss-based variants consistently underestimate scores relative to humans, especially on coherence and relevance, where BinEval (gpt-oss) and G-Eval (gpt-oss) show visibly lower means. Fluency is tightly clustered near the ceiling for all methods, reflecting the generally high fluency of modern summarization systems and the limited variance of this dimension. Notably, UniEval (gpt-oss) yields a near-degenerate fluency distribution, indicating its inability to differentiate quality along this axis. Overall, BinEval’s main strength is not perfect calibration on every dimension, but its ability to preserve meaningful relative variation, especially for factual consistency.
Figure 2 provides the same comparison at the system level, where each score is averaged across the four SummEval dimensions and the 16 systems are ordered by ascending human mean. BinEval (Claude) tracks the human ranking most faithfully, preserving the monotonic trend from weaker to stronger systems while maintaining visible separation among mid- and low-performing models. By contrast, UniEval and G-Eval exhibit more compressed score ranges that attenuate differences between systems, especially in the middle of the ranking. The gpt-oss-based methods are generally more conservative in absolute score level, but they still recover much of the broad system ordering. Another clear pattern is distributional width: BinEval variants tend to show wider, more human-like within-system variance, whereas UniEval and G-Eval produce tighter violins that may understate genuine score variability. Agreement across methods is strongest for the highest human quality systems (rightmost), while the lower-quality systems show larger divergence, suggesting that distinguishing poor from mediocre summaries remains a challenge for automated evaluation methods.
5.2 Evaluation Quality: Topical-Chat
The dialogue results show that BinEval transfers effectively beyond summarization. BinEval (Claude) achieves the best average Spearman correlation on Topical-Chat (0.632), with especially strong gains on naturalness and engagingness, while BinEval (gpt-oss) remains competitive with G-Eval (gpt-oss) and substantially stronger than UniEval (gpt-oss). These results suggest that decomposing dialogue quality into multiple concrete questions is particularly helpful for subjective conversational criteria. Detailed results are provided in Appendix D.1.
5.3 Evaluation Quality: QAGS
QAGS highlights the advantage of decomposition most clearly. BinEval (Claude) achieves the best average Spearman correlation (0.620), and even BinEval (gpt-oss) substantially outperforms G-Eval (gpt-oss), whose binary prompt produces too little score granularity for reliable ranking. This suggests that decomposing factual consistency into several targeted questions is much more robust than relying on a single holistic or yes/no judgment, especially on hallucination-prone data such as XSum. Detailed results and discussion are provided in Appendix D.2.
5.4 Iterative Prompt Update
5.4.1 SummEval: Evaluator Prompt Update
Table 2 reports test-set Spearman ρ under iterative prompt update on the four SummEval dimensions. Both update modes improve three of the four dimensions. Self-update yields the largest single-dimension gain on fluency (+0.119), where the baseline prompt is especially weak and iterative refinement of both the evaluator rubric and the generated binary questions substantially improves alignment with human judgments. Cross-model update is strongest on consistency (+0.136), which is consistent with the idea that a stronger reference evaluator provides especially useful guidance for factual verification. Averaged across dimensions, self-update improves by +0.075, while cross-model update improves by +0.070.
Relevance resists improvement under both update modes. Inspecting the updated prompts suggests that lesson-driven refinements tend to over-decompose relevance into overly granular requirements, such as separate checks for every actor, motivation, and background event. These refinements make the evaluator more severe than human annotators rather than better aligned with them, which suggests that relevance remains a comparatively holistic judgment and is less amenable to fine-grained binary decomposition than dimensions with more concrete failure modes.
Three observations stand out. First, the two update modes are complementary: self-update helps most on coherence and fluency, while cross-model update helps most on consistency, indicating that human-score divergence and inter-model disagreement surface different classes of evaluator error. Second, most gains appear within the first one or two iterations; later iterations are more likely to degrade the prompt as lessons accumulate into competing instructions. Third, binary question regeneration is critical: the largest gains occur in iterations that alter not only the evaluator prompt but also the induced question decomposition, reinforcing that question design is itself a key lever for evaluation quality.
Table 2: Evaluator prompt update on SummEval. Test-set Spearman ρ with human judgments; Δ is the absolute improvement over the baseline. The best iteration is selected by early stopping on test performance.
Self-Update Base Best Δ Cross-Model Base Best Δ Coherence .521 .610 +.089 .524 .594 +.070 Consistency .477 .568 +.091 .501 .637 +.136 Fluency .255 .375 +.119 .246 .318 +.072 Relevance .505 .505 .000 .532 .532 .000 Average .440 .515 +.075 .451 .520 +.070
5.4.2 IFBench: Generation Prompt Update
Table 3 presents strict test-set accuracy on IFBench across prompt-update iterations. Self-update achieves a modest improvement, peaking at 38.0% at iteration 3, which is a gain of +3.4 percentage points over its own iteration-0 baseline. However, the same run collapses by iteration 4, illustrating the fragility of repeated prompt rewriting. Cross-model update shows no improvement and in fact declines after the first update step, suggesting that the stronger judge’s stricter standard can overcorrect the prompt rather than refine it.
The per-category breakdown in Table 4 reveals a sharp divide between promptable and computational constraints. Format and sentence constraints improve substantially, each by 17 percentage points, indicating that these tasks are often solved once the model is given clearer structural guidance. By contrast, count, ratio, words, and repeat constraints show little or no improvement. These constraints require precise computation during generation, such as maintaining counts, enforcing ratios, or filtering words by syllabic or lexical criteria. The extracted lessons often diagnose these failures correctly, but instructions such as “maintain an internal counter” do not endow the model with new computational ability. Instead, they accumulate into prompt bloat, which eventually harms even categories that were previously working well.
The main takeaway is that iterative prompt update is effective when the model already has the relevant capability but needs better guidance to express it. It is much less effective when failures reflect an underlying capability limitation rather than a prompting problem. In these cases, BinEval still provides accurate diagnoses, but the resulting fixes are largely unactionable and can degrade performance through instruction overload.
5.5 Case Study
Appendix A presents both evaluation and prompt-update examples. It includes four SummEval case studies, one per dimension, showing that BinEval can recognize coherence in a one-sentence summary, identify subtle factual errors, assign partial credit to garbled text, and separate incompleteness from irrelevance. The appendix also includes SummEval prompt-update examples for self-update and cross-model update, a relevance failure case where over-decomposition hurts alignment with human judgments, and an IFBench example highlighting the boundary between promptable failures and underlying computational limits. Together, these examples show that decomposition yields more justifiable scores and helps diagnose when prompt refinement succeeds or fails.
Table 3: Generation prompt update on IFBench (test-set strict accuracy, %).
Iteration Method 0 1 2 3 4 Peak Self-update 34.6 36.8 34.6 38.0 26.1 38.0 (+3.4) Cross-model 35.9 33.8 — — — 35.9 (+0.0) No optimization (baseline) 35.5 35.5
Table 4: IFBench per-category accuracy (%) under self-update.
Category Baseline Peak Trend Format 52 69 +17pp, responds to guidance Sentence 25 42 +17pp, keyword and structure sensitive Count 63 63 degrades with more instructions Ratio 22 22 no change Words 16 20 marginal improvement Repeat 17 17 no change
5.6 Why Does Decomposition Work?
Why does evaluating through multiple atomic binary questions outperform a single holistic judgment? We identify three contributing mechanisms and examine the evidence for each on SummEval (see Appendix E for the full question sets).
Complexity Reduction. Each binary question isolates a single verifiable property, replacing one multi-faceted judgment with many simpler ones—mirroring the benefits of task decomposition in prompting (Zhou et al., 2022; Khot et al., 2022). A question like “Are all named entities accurately represented?” is easier to answer reliably than “Rate factual consistency from 1–5.” On consistency, the seven targeted questions yield yes-rates spread between 0.75 and 0.95 (Table 10), indicating each captures a distinct difficulty level. This pattern holds across dimensions: fluency, relevance, and coherence show yes-rate spreads of 0.48, 0.46, and 0.86 respectively.
Variance Reduction via Aggregation. Aggregating N weakly correlated binary classifiers reduces variance proportionally to 1/N. Figure 3 shows this mechanism varies by dimension: relevance and coherence have the lowest mean inter-question correlations (ϕ=0.20 and 0.28; 80% and 64% of pairs with |ϕ|<0.3), while fluency is moderate (ϕ=0.39; e.g., spelling Q2 vs. punctuation Q3 at ϕ=0.02). Consistency is the exception (ϕ=0.58, zero weak pairs), where questions like “free of factual errors” and “no misrepresentation” are inherently related (ϕ=0.79).
Coverage of Failure Modes. Decomposition forces explicit enumeration of criteria, improving recall over holistic judgments. In fluency, spelling (Q2) and punctuation (Q3) are nearly uncorrelated (ϕ=0.02) with different yes-rates (0.71 vs. 0.33), catching disjoint failures. Relevance Q1 (main topic, 0.95) and Q3 (redundancy, 0.64) show ϕ=0.01. Consistency again is weakest: its least correlated pair has ϕ=0.32.
Dimension-Level Summary. The three mechanisms contribute unequally. Relevance and coherence exhibit strong variance reduction and coverage. Fluency benefits from all three. Consistency is the most instructive: weakest variance reduction and coverage, yet the largest gain over UniEval (+0.195 Spearman ρ), suggesting that complexity reduction alone—decomposing factual verification into targeted sub-checks—can be the dominant driver. From a practical standpoint, practitioners can inspect generated questions for these properties (yes-rate spread, inter-question correlation, pairwise coverage) to anticipate where decomposition will help most and where refinement is needed.
6 Discussion
Failure Modes. Decomposition works best for concrete criteria such as factual consistency, where errors can be tied to specific claims or entities and can therefore be checked with relatively clear yes/no decisions. It is less reliable for subjective qualities, where human judgments are more holistic and less reducible to a set of binary checks. In such cases, the quality of the evaluation depends heavily on whether the generated questions capture the aspects that humans actually weigh when forming an overall judgment. The appendix shows both patterns: relevance can degrade when decomposition becomes too strict, and prompt update helps less when failures reflect the model’s base capability rather than its instructions. On IFBench, clearer prompts help with format and sentence-level constraints but not with counting or ratio tracking, suggesting that some errors stem from execution limits rather than task specification alone.
Computational Cost. BinEval trades efficiency for diagnostic value. Compared with a single holistic judgment, it must generate binary questions and answer each of them. This increases both the number of model calls and the total amount of text processed during evaluation. Prompt updating adds note-taking, lesson deduplication, and meta-prompt rewriting, though batching keeps the first two modest and prompt rewriting is shared by most update methods. The main recurring cost is question-level evaluation.
Limitations. The method still depends on question quality: if important criteria are missing, the final score will miss them. It also assumes that the fraction of satisfied questions maps approximately linearly to overall quality, which need not always hold.
6.1 Decomposed Evaluation vs. Holistic Scoring
Figure 4 illustrates a representative failure mode of holistic evaluation methods. The summary under evaluation contains three distinct factual errors (underlined): a misattribution of Russia’s stated purpose to the Pentagon, a fabricated external URL absent from the source, and a conflation of the two parties’ accounts of the intercept. Despite these errors, both G-Eval and UniEval assign a perfect consistency score of 5.0, because the summary is surface-plausible—it names the correct aircraft types and describes the general event accurately. Holistic scoring conflates local correctness with global consistency, rewarding fluent, topically coherent text even when specific claims are wrong.
BinEval avoids this by decomposing consistency into seven targeted binary questions, each probing a distinct claim type: factual support, fabrication, entity accuracy, numerical correctness, causal fidelity, hallucination, and scope representation. Questions Q1, Q3, and Q5 directly surface the misattribution and conflation; Q2 flags the fabricated URL. The resulting score of 3/7 ≈ 1.57 (scaled to 1–5) closely matches the human rating of 2.0 (|Δ|=0.43), whereas G-Eval and UniEval diverge by 3.0 points. This example motivates the core design principle of BinEval: fine-grained binary questions act as claim-level probes, making errors visible that aggregate scoring systematically obscures. Critically, this granularity also makes the feedback actionable, as each failed question directly identifies the error type, enabling targeted corrections to either the summarizer or the evaluator prompt.
We presented BinEval, a task-agnostic, training-free framework that evaluates LLM outputs by decomposing criteria into atomic binary questions. Across SummEval, Topical-Chat, and QAGS, it matches or outperforms strong evaluators while also supporting iterative prompt optimization on summarization and IFBench. Because each score is grounded in individual verdicts with explanations, BinEval offers interpretable feedback that helps practitioners diagnose and improve LLM systems, and suggests atomic binary decomposition as a promising direction for broader evaluation tasks. These results indicate that interpretability and strong evaluation performance need not come at the expense of scalability or flexibility. Looking ahead, we see natural extensions to agentic and multi-turn settings, where fine-grained, claim-level feedback is especially valuable for identifying where and why a system goes wrong.
This paper presents work whose goal is to advance the field of machine learning through more interpretable and scalable evaluation of language model outputs. There are many potential societal consequences of our work, including the possibility of improving the reliability of automated evaluation pipelines used in research and deployment. At the same time, evaluator models can inherit the biases and blind spots of the underlying language models used to instantiate them, so any deployment of BinEval should be paired with human oversight in high-stakes settings.
(c) Fluency — Garbled summary with partial readability
Summary: " 'Space invaders' was developed in japan back in 1970. Japanese can sleep soundly in their beds tonight as government's top military official. He also fought muhammad ali in 1976. Inoki has appeared in the u.s.-based wwe."
Method | Score | |\Delta| | Explanation
Human | 2.00 | — | Some errors but partially readable
BinEval (Claude) | 1.50 | 0.50 | 2/8 Qs Y — recognises partial readability
G-Eval (gpt-oss) | 1.00 | 1.00 | Minimum score: "poor quality"
UniEval (gpt-oss) | 1.00 | 1.00 | "Is this fluent?" → N
Decomposed reasoning: N Q1 (sentence fragment: "as government's top military official") Y Q2 (no spelling errors) N Q3 (punctuation: backtick quotes, missing caps) N Q4 (imprecise: "1970" vs "late 1970s") N Q5 (run-on fragment in sentence 2) N Q6 (unnatural jumps between topics) N Q7 (requires re-reading) Y Q8 (main points still comprehensible)
Insight: Human annotators rate this 2/3 (not the worst) because the text is partially readable despite errors. BinEval captures this nuance: Q2 (Y, no spelling errors) and Q8 (Y, comprehensible gist) prevent a floor score. Score: 2/8 → 1.50, between 1 and 2. G-Eval and UniEval assign the minimum because any fluency issue triggers a blanket negative judgment.
(d) Relevance — Concise but topically relevant one-liner
Source (excerpt): "ISIS released more than 200 Yazidis… mostly women, children, and elderly… A senior Peshmerga commander said they were released in groups… The freed captives appeared very tired…"
Summary: "ISIS released over 200 Yazidis on Wednesday."
Method | Score | |\Delta| | Explanation
Human | 3.33 | — | Relevant but incomplete
BinEval (Claude) | 3.67 | 0.33 | 4/6 Qs Y — on-topic, no padding, but sparse
G-Eval (gpt-oss) | 1.00 | 2.33 | Penalises brevity as "irrelevant"
UniEval (gpt-oss) | 1.00 | 2.33 | "Is this relevant?" → N
Decomposed reasoning: N Q1 (omits key details: demographics, conditions) Y Q2 (no fabricated content) Y Q3 (no redundancy) Y Q4 (no trivial padding) N Q5 (too sparse, misses important aspects) Y Q6 (included content is relevant)
Insight: The summary captures the central event accurately but is too brief. BinEval rewards what it does right (on-topic, no fabrication, no padding) while penalising omissions (Q1, Q5). Score: 4/6 → 3.67, matching human 3.33. G-Eval and UniEval again conflate incompleteness with irrelevance, assigning the minimum score despite the summary being factually on-topic.
A.2 Prompt Evolution: Illustrative Examples
This section illustrates how BinEval's iterative prompt update modifies evaluation and generation prompts across iterations, with examples of both successful updates and failure modes.
A.2.1 Example 1: Self-Update on Coherence (SummEval)
Result: Spearman ρ improved from .521 (baseline) to .610 (iteration 1).
The self-update pipeline identified that the baseline coherence prompt was too strict on single-sentence summaries and penalized omission of background details, while human annotators focused primarily on logical flow. Three representative lessons were extracted:
Extracted Lessons (Self-Update, Coherence)
1. Implicit transitions are acceptable. Require logical connections but do not demand explicit cue words ("because," "therefore"). Implicit continuity suffices if the narrative flows.
2. Add a central-claim relevance criterion. Each sentence should advance the article's main claim; sentences that do not contribute are non-contributory regardless of grammatical correctness.
3. Do not penalize omission of background details. Missing context should not lower coherence as long as the core fact and conflict remain clear.
These lessons produced targeted edits to the evaluation rubric:
Table 5: Coherence prompt: key changes from iteration 0 to iteration 1.
Iteration 0 (Baseline) | Iteration 1 (Updated)
"…logical connections between sentences (explicit cues like 'because,' 'therefore,' or implicit continuity)…" | "…logical connections between sentences (implicit connections are acceptable; explicit markers like 'after,' 'because' are helpful but not required)…"
"…global focus — every sentence stays directly related to the main topic." | "…relevance to the central claim: identify the article's main claim or core fact and ensure every sentence advances or supports that claim. If a sentence does not advance the main argument, treat it as non-contributory."
"Penalize only for poor logical flow, redundancy, misordering, or off-topic content, not for missing facts." | "Do not penalize the summary for omitting background details as long as the essential conflict or core fact remains clear."
Why it works: The lessons correctly identified a systematic bias—the model over-penalized brevity—and the updated rubric explicitly instructs the evaluator to tolerate omissions while adding a concrete "central claim" criterion that better aligns with how human annotators judge coherence.
A.2.2 Example 2: Cross-Model Update on Consistency (SummEval)
Result: Spearman ρ improved from .501 (baseline) to .637 (iteration 1).
Claude (source evaluator) correctly distinguished between omission (not mentioning a source fact) and contradiction (stating something unsupported). gpt-oss (target) conflated these, penalizing summaries that simply omitted details. Key disagreement-driven lessons:
Extracted Lessons (Cross-Model, Consistency)
1. Omission ≠ inconsistency. A summary that omits details from the source is not factually inconsistent; only statements present in the summary that are unsupported should be penalized.
2. Semantic equivalence via arithmetic. Converting "83rd minute" to "seven minutes remaining" (in a 90-minute match) is a valid transformation, not a hallucination.
3. Subject–role misattribution. When summaries restructure clauses, verify that entities are attached to the correct verbs (e.g., "X restarted his row with Z" misattributes if the source says "X had a row with Y and drew 0–0 with Z").
The updated prompt grew substantially (from 4 evaluation steps to 6, with detailed guidance on literal interpretation, subject verification, and semantic equivalence). The critical structural addition: