GptGet

Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks

TL;DR

Introducing reasoning overconfidence in LLMs; Long-CoT reduces it via iterative reflection, based on the cognitive rigidity hypothesis.

cs.CL 🔴 Advanced 2025-12-01 50 views
Jiannan Guan Qiguang Chen Libo Qin Dengyun Peng Jinhao Liu Liangyu Huo Jian Xie Wanxiang Che
AI Reader Arxiv Page Download PDF
LLMs multi-solution reasoning overconfidence chain-of-thought cognitive rigidity

Key Findings

Methodology

The authors developed MuSoBench, a benchmark with TimeTabling and SubsetSum tasks, to evaluate models on multi-solution problems. They compare Short-CoT and Long-CoT prompting strategies using calibration error (ECE) and recall metrics. Behavioral diagnostics and attention entropy analyses support the cognitive rigidity hypothesis, showing early convergence causes overconfidence. Reflection steps and parallel exploration mitigate this, with models demonstrating improved solution diversity and calibration. The methodology integrates systematic dataset construction, multi-metric evaluation, and internal state analysis to understand and address overconfidence.

Key Results

  • On MuSoBench, Short-CoT models show high ECE (>78%) and low recall, indicating overconfidence and incomplete solutions. Long-CoT reduces ECE by over 18 percentage points, boosting recall to 49.55%. In TimeTabling, Long-CoT achieves CSR of 53.13% and Error Correction of 81.83%, outperforming Short-CoT’s 43.15% and 65.22%. In SubsetSum, recall improves from 9.99% to 21.24%. Multi-path voting further enhances recall and calibration, confirming exploration benefits. Attention entropy analysis reveals lower entropy in core layers for models with better exploration, supporting the cognitive rigidity hypothesis.
  • Results demonstrate that longer reasoning chains and reflection steps lead to better calibration and solution coverage. Increasing exploration breadth reduces overconfidence, especially in complex tasks. The combination of multi-path strategies and internal attention analysis confirms that early convergence underpins overconfidence, and iterative reflection effectively counters this. These findings highlight the importance of exploration for reliable multi-solution reasoning.
  • The experiments validate that models with more extensive reasoning and reflection can better balance confidence and performance. The use of attention entropy as an internal measure supports the cognitive rigidity hypothesis, linking internal model states to exploration capacity. The results suggest that improving exploration strategies is key to mitigating overconfidence and enhancing model reliability in multi-solution tasks.
  • Overall, the study advances understanding of the internal mechanisms behind overconfidence, providing practical methods (reflection, multi-path voting) to improve calibration and solution diversity. These insights are crucial for deploying trustworthy LLMs in real-world multi-answer scenarios, such as planning, scheduling, and decision-making.

Significance

This work uncovers a fundamental flaw—overconfidence—in large language models when handling multi-solution tasks. By systematically analyzing the phenomenon through MuSoBench and behavioral diagnostics, it challenges the prevailing focus on single-answer accuracy. The findings emphasize the importance of exploration and internal model states, advocating for more sophisticated reasoning paradigms. The research bridges theoretical insights with engineering solutions, offering practical strategies like iterative reflection and multi-path exploration to enhance model reliability. These advances are vital for deploying trustworthy AI in complex, real-world applications such as autonomous planning, resource allocation, and multi-objective optimization, where understanding the full solution space is critical. The work also opens avenues for future research into internal model dynamics and calibration methods, aiming to build models that are both accurate and self-aware of their limitations.

Technical Contribution

The paper introduces the concept of reasoning overconfidence (ROC) and develops MuSoBench, a benchmark for evaluating multi-solution reasoning. It proposes a cognitive-rigidity hypothesis, linking early convergence to overconfidence via attention entropy analysis. The authors design reflection steps and multi-path voting strategies to mitigate ROC, demonstrating significant improvements in calibration and recall. The internal mechanism analysis reveals that models with better exploration maintain higher attention entropy in core layers, supporting the hypothesis. These innovations extend the understanding of model calibration, internal states, and exploration strategies, providing a comprehensive framework for enhancing multi-solution reasoning in LLMs.

Novelty

This study is the first to systematically define and quantify reasoning overconfidence in LLMs, linking it to internal cognitive rigidity. It combines behavioral diagnostics, attention entropy analysis, and multi-path exploration to address early convergence issues. Unlike prior work focusing solely on final answer accuracy, this research emphasizes the reasoning process's completeness and diversity. The introduction of MuSoBench provides a controlled environment for empirical evaluation, and the proposed strategies—reflection and voting—offer practical solutions. Overall, it advances the theoretical understanding and engineering of exploration mechanisms in multi-solution reasoning, representing a significant step forward in trustworthy AI development.

Limitations

  • Despite improvements, open-source models still exhibit significant overconfidence, especially in highly complex tasks, indicating room for further calibration enhancements.
  • Multi-path exploration strategies increase computational costs, limiting real-time deployment in resource-constrained environments.
  • The internal mechanism analysis relies on attention entropy, which may not fully capture all aspects of internal model states, necessitating further investigation.

Future Work

Future research will focus on developing more efficient exploration algorithms, possibly integrating reinforcement learning or adaptive reflection mechanisms. Extending calibration techniques to diverse tasks and domains will be prioritized, aiming for models that can self-assess and communicate their confidence more reliably. Additionally, exploring the internal dynamics of models with different architectures and training regimes could deepen understanding of cognitive rigidity and exploration capacity. Ultimately, the goal is to create models capable of comprehensive, trustworthy multi-solution reasoning applicable across complex real-world scenarios.

AI Executive Summary

Deep Dive

Plain Language Accessible to non-experts

想象你在厨房准备一顿大餐。你有很多食材，但不知道用哪些组合能做出不同的菜。你可能会先选一种菜，然后觉得自己已经想到了所有可能的菜肴，但其实还可以尝试其他组合。模型就像厨师，试图找到所有可能的菜肴，但有时候它会太自信，只认定自己想到的那几样菜，忽略了其他可能性。长时间的思考和反复试验，就像厨师不断尝试不同的搭配，最终能找到更多的菜肴。这个研究告诉我们，模型如果太快停止探索，就会错过很多答案，就像厨师太早放弃尝试，错过了更好的菜谱。

ELI14 Explained like you're 14

想象你在学校玩一个游戏，你要列出所有你能想到的朋友的名字，但你只记得一部分。有人告诉你，快点列完所有朋友的名字，但你觉得自己已经说完了，实际上还遗漏了很多。这个就像模型在回答问题时，只想到几个答案，就觉得自己已经找完了所有可能的答案，但其实还可以继续想。你可以多次回头检查，或者试试不同的方法，这样就能找到更多的朋友名字，也能更清楚自己都想到了哪些。研究发现，如果模型太早停止探索，就会错过很多答案，就像只玩一遍游戏就觉得已经赢了。相反，反复思考和尝试不同的路径，就像长链推理，能帮模型找到更多正确答案，不会太自信地认为自己已经全部想到了。

Abstract

Large Language Models (LLMs) excel in reasoning tasks requiring a single correct answer, but they perform poorly in multi-solution tasks that require generating comprehensive and diverse answers. We attribute this limitation to \textbf{reasoning overconfidence}: a tendency to express undue certainty in an incomplete solution set. To examine the effect, we introduce \textit{MuSoBench}, a benchmark of multi-solution problems. Experiments show that the conventional short chain-of-thought (Short-CoT) prompting paradigm exhibits pronounced overconfidence, whereas the emerging long chain-of-thought (Long-CoT) approach mitigates it through iterative exploration and self-reflection. We further characterise observable behaviours and influential factors. To probe the underlying cause, we propose the \textbf{cognitive-rigidity hypothesis}, which posits that overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths. An attention-entropy analysis offers preliminary support for this view. These findings provide tools for assessing the completeness of LLM reasoning and highlight the need to move evaluation beyond single-answer accuracy toward comprehensive exploration.

cs.CL