One of the hardest risks to manage in large language models is their tendency to give highly confident answers even when they lack certainty. This high-confidence hallucination causes users to misjudge the reliability of answers: the model may not only fail to recognize its own knowledge boundaries, but may also express its internal uncertainty as overly confident. In this paper from Yale University and Google Research, the authors first unpack a key conceptual distinction: most calibration methods focus on factual calibration—making the probabilities a model outputs match external accuracy—but that is not the same as faithful calibration, which requires the uncertainty the model actually expresses to closely track its own internal confidence.
To address this gap, the work proposes RLMF (Reinforcement Learning with Metacognitive Feedback). Beyond rewarding the model for answering well, RLMF also incorporates a training signal for "how accurately the model can evaluate its own performance." In experiments, RLMF improved faithful calibration by up to 63% over standard reinforcement learning, and achieved an average human-evaluation win rate of 96% for natural-language uncertainty expression. These numbers have clear boundaries: they are limited to uncertainty expression and faithful calibration tasks, and should not be interpreted as overall capability gains.
[Figure 1: Overview of RLMF, combining metacognitive data selection with targeted rewriting to faithfully calibrate LLM uncertainty expression in both numerical and linguistic forms.] The two-stage framework first calibrates numerical confidence with RLMF and metacognitive data selection, then converts it into natural-language uncertainty expression through targeted rewriting.
From "Factual Calibration" to "Faithful Calibration": An Overlooked Trust Gap
To understand the value of RLMF, it is essential to disentangle two concepts that are easily confused. Factual calibration aims to ensure that the confidence a model claims matches the probability that it ultimately answers correctly. If a model says "I am 80% confident," and it is correct about 80% of the time on such predictions, it is considered well calibrated. This is the standard yardstick in academia, but it does not ask whether the uncertainty the model expresses is close to its internal confidence.
Faithful calibration requires the uncertainty a model expresses to align with its internal uncertainty. A model may perform reasonably well on factual calibration, yet still answer in a categorical tone when its confidence is low, or unnecessarily retreat when its confidence is high. This paper focuses precisely on this mismatch between expression and internal signal: models generate incorrect answers with high confidence, fail to recognize knowledge boundaries, and misrepresent their own uncertainty. This is more troublesome than simply answering incorrectly, because users have difficulty judging whether to trust the model.
The paper designed 10 benchmark datasets spanning more than 6 content domains to test this capability, from open-domain question answering to mathematical reasoning, from natural language understanding to hallucination detection, covering different formats and difficulty levels.
RLMF's Mechanism: Teaching the Model to "Grade Its Own Work"
RLMF builds on GRPO (Group Relative Policy Optimization). In standard reinforcement learning, the model generates a batch of candidate answers to the same question, then rewards are computed based on each answer's performance on accuracy, factual calibration, formatting, and other dimensions to decide which answers should be "reinforced."
The key change is the introduction of an additional evaluation dimension: metacognitive advantage scaling. Think of it as "first look at answer quality, then look at self-assessment quality": if a candidate answer's task performance is already above the group average, and the model's self-assessment of that performance is also more accurate, it receives a stronger training signal. Specifically, after GRPO computes the raw advantage value for each candidate answer, RLMF introduces a coefficient Zg to scale it.
Zg measures the accuracy of the model's self-assessment: it compares the model's "predicted" level of faithful calibration on the task against its actually demonstrated level of faithful calibration. The smaller the gap, the closer Zg is to 1, reflecting better metacognitive performance. This scaling is applied only to candidate answers whose task performance already exceeds the group average, avoiding the pitfall of incorrectly reinforcing low-quality answers merely because their "self-assessment looks good."
[Figure 2: Overview of the proposed RLMF method] During the GRPO policy update process, RLMF introduces a metacognitive feedback mechanism: the accuracy of the model's self-judgment is used to scale advantage values, thereby optimizing the target task while also reinforcing the model's ability to evaluate its own performance.
Figure 2
Metacognitive Data Selection: Why Look at Both "High-Scoring" and "Low-Scoring" Samples
In addition to injecting metacognitive signals during training, the work also proposes metacognitive data selection as a companion strategy for filtering training samples. Unlike conventional "active learning" approaches that only pick the samples where the model currently performs worst, the scheme here first has the model self-assess each sample: without access to external ground-truth labels, it gives its own answer a score of "how good I think I am."
Then, it simultaneously selects two portions of samples: the half with the highest self-assessment scores and the half with the lowest scores, forming the final training set. The paper provides experimental evidence that this "bipolar selection" strategy outperforms naive active learning that only picks the lowest-scoring samples. The logic is that highest-scoring and lowest-scoring samples provide complementary learning signals: the model learns from high-scoring samples "what a good state of accurate self-assessment looks like," and from low-scoring samples "how to recognize and correct cognitive biases." Experimental results show that this metacognitive-signal-based data filtering method outperforms random sampling and active learning baselines on the faithful calibration metric cMFG*, while maintaining accuracy and factual calibration levels.
From Numbers to Language: A Decoupled Design Lets AI Say "I'm Not Sure"
Uncertainty expression in large language models must ultimately be conveyed in natural language, not just as a number between 0 and 1. The paper's two-stage approach cleanly decouples this process.
In stage one, RLMF and metacognitive data selection are used to train the model to output sentence-level numerical confidence that more closely tracks internal confidence. This means every sentence in the model's answer carries a corresponding numerical uncertainty label.
In stage two, these faithfully calibrated numbers are mapped to natural, context-appropriate hedge expressions. The paper constructs a mapping table from confidence values to hedge expressions—such as "I'm fairly certain," "my preliminary judgment is," "I have very low confidence on this point"—based on human annotations of the perceived confidence of a large collection of hedges. Then, a relatively low-cost LLM (the experiments used Gemini-2.5-Flash-Lite or GPT-5-Mini) uses this mapping table to rewrite the original answer into a version with appropriate hedging.
This decoupled design offers a practical benefit: stage one depends on expensive reinforcement learning training, but only needs to be run once; the linguistic style, tone, and audience adaptation in stage two can be flexibly adjusted without retraining the model. For example, more precise graded wording can be retained for researchers, while more accessible uncertainty expressions can be used for non-expert users.
Effects and Boundaries: What 63% Improvement and 96% Win Rate Tell Us
The paper conducted systematic evaluations on multiple models, including Qwen3 (1.7B, 4B, 8B) and Llama3.1-Instruct (8B). Several key numbers are worth highlighting:
On faithful calibration tasks, RLMF improved by up to 63% over standard reinforcement learning. This improvement was obtained after training on PopQA and then evaluating across multiple tasks; it does not equate to overall capability gains, nor to accuracy gains. More importantly, this improvement did not come at the cost of task accuracy or factual calibration. The paper also compared several closed-source frontier models with specialized prompting, showing that small models trained with RLMF are competitive on faithful calibration metrics.
[Table 1: Faithful calibration (FC) results compared with baselines, evaluated via cMFG] Blue rows show numerical calibration and linguistic calibration results; yellow rows show a pure-RL ablation with the metacognitive advantage scaling removed. RLMF achieved cMFG scores ≥ 0.80 on all datasets without hurting accuracy. Dataset abbreviations: PQA (PopQA), SA (SelfAware), SQA (SimpleQA), HE (HaluEval), MMLU, SQ (SciQ), MT (MATH), UM (UMWP), AC (ARC-Challenge), SG (SuperGLUE).
For the quality of natural-language expression, the paper carried out systematic human evaluation. Evaluators compared answers generated by the RLMF framework with those from the strongest baseline method (FUT), scoring them on four dimensions: diversity, naturalness, helpfulness, and contextual appropriateness. The results show that the RLMF framework achieved an average 96% win rate on these dimensions, with inter-annotator agreement of 0.93. By contrast, the baseline method showed obvious pattern rigidity, especially in long texts where hedge expressions were monotonous and repetitive; the mapping-and-rewriting scheme mitigated this problem. This also explains why the authors decoupled "numerical faithful calibration" and "linguistic rewriting" into two stages.
[Figure 3: Reliability diagram of expressed confidence vs. internal confidence, evaluated on PopQA] Each bin has a width of 0.1 in internal confidence. FUT and the original model generally fail at low internal confidence, while RLMF shows consistent high faithfulness across the full confidence range.
Several robustness findings are worth noting: although RLMF was trained only on PopQA, it showed generalization across very different tasks including math, science, and natural language understanding; the choice of training task (including math reasoning or hallucination detection tasks) had relatively stable effects on faithful calibration results across different evaluation tasks; and as RLMF training progressed, the accuracy of the model's self-assessment (measured by Zg) also continued to improve. Again, this does not mean the model acquired self-awareness; it only means it became better at the operational task of "evaluating its own performance."
Of course, this method also has clear application boundaries and computational costs. Estimating internal confidence requires sampling 20 answers to the same question and computing agreement, which substantially increases computational overhead. The paper also acknowledges that training was completed on a single dataset; although generalization was good, robustness in more diverse scenarios remains to be verified. Furthermore, RLMF relies on stage-two rewriting to address the diversity of linguistic expression, a pragmatic acknowledgment of the limitations of reinforcement learning itself.
The cMFG metric itself also deserves attention. It fixes a problem in prior faithful calibration evaluations: when a model's internal confidence distribution is concentrated in a narrow range, the traditional cMFG metric systematically underestimates faithfulness. cMFG uses equal-sample-count binning and width-weighted integration to make comparisons across models fairer. Overall, the focus of this work is calibrating expression: when a model is confident or unconfident, the confidence it states should be closer to its actual internal state.
📄 Original title
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
🔗 Original link