Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Recent advances in chain-of-thought prompting have significantly improved the reasoning capabilities of large language models, especially in mathematical and logical tasks. Wei et al. (2022) demonstrated that generating intermediate reasoning steps enhances performance, and subsequent reinforcement learning approaches (Shao et al., 2024; Guo et al., 2025) further refine these models by rewarding correct reasoning chains. Despite these developments, the interpretability of reasoning traces remains contentious. Studies by Turpin et al. (2023) and Chen et al. (2025) highlight that many generated reasoning steps are performative or superficial, raising questions about their fidelity. Existing evaluation methods largely focus on overall response correctness or perturbation-based faithfulness, which lack granularity at the step level. This gap motivates the need for a more precise, causal measure of step importance that can distinguish truly influential reasoning steps from superficial ones.

The core challenge lies in quantifying the true importance of individual reasoning steps within a chain, beyond superficial textual features. Traditional methods rely on human annotations or perturbation tests, which are either subjective or computationally expensive. Without an objective, scalable metric, it is difficult to assess whether the generated reasoning steps genuinely contribute to the final answer, especially in complex or ambiguous tasks. This ambiguity hampers the development of models that can generate trustworthy, faithful explanations. Furthermore, the inability to decode step importance solely from text limits the utility of automated interpretability tools, which are crucial for deploying AI in sensitive domains like healthcare or legal decision-making.

The paper introduces the concept of advantage as a formal, quantitative measure of step importance, grounded in reinforcement learning principles. By estimating the change in expected reward caused by each step through Monte Carlo sampling, the authors provide a causal, task-specific metric. They integrate change point detection (PELT algorithm) to automatically identify steps with significant advantage shifts, effectively pinpointing critical reasoning junctures. Additionally, they develop fine-tuned discriminators that predict advantage directly from textual features, bridging the gap between causal importance and textual decodability. This approach surpasses prior methods that relied solely on superficial cues or heuristic importance measures, offering a rigorous, scalable framework for analyzing reasoning processes.

• �� Define reasoning trace as a sequence of steps generated by the model, each represented as a string.
• �� Use Monte Carlo sampling (N=50) at each step to estimate the probability that including that step leads to the same final answer, thus computing the step’s value.
• �� Calculate advantage as the difference between the Q-value (expected reward after executing the step) and the value (expected reward from the current state).
• �� Apply the PELT change point detection algorithm to the advantage trajectory to identify steps where the advantage significantly shifts.
• �� Label steps as consequential if the advantage jump exceeds a threshold δ=0.1 with high confidence.
• �� Fine-tune classifiers to predict advantage from textual features, evaluating their ability to decode importance in both correct and incorrect responses.
• �� Conduct large-scale experiments on datasets like AIME, GSM8K, and Math500, analyzing the distribution and decodability of advantage across different model sizes and reasoning modes.

The experimental setup involves generating multiple responses per prompt from models like Qwen3-1.7B, 4B, and 8B, with and without thinking mode. Each response is segmented into steps, and Monte Carlo rollouts estimate the advantage of each step. The datasets include six math reasoning benchmarks, with a total of 1,800 responses analyzed. The key metrics include the proportion of steps with high advantage, the accuracy of advantage prediction models, and the correlation between advantage and final correctness. The experiments compare model scales, reasoning modes, and the effectiveness of fine-tuning discriminators, providing comprehensive insights into the decodability and importance of reasoning steps.

Results show that high-capacity models outperform baseline judges in identifying consequential steps, but overall PR-AUC remains low (~0.2), indicating difficulty in fully decoding importance from text. Fine-tuned critics improve detection accuracy, especially in incorrect responses, yet still fall short of the theoretical maximum. Advantage patterns vary with task difficulty; in simpler datasets like GSM8K, fewer steps exhibit high advantage, while complex tasks like AIME show more critical steps. The analysis reveals that model improvements are driven more by prior biases than genuine reasoning enhancements, emphasizing the gap between readability and true interpretability.

This methodology enables detailed causal analysis of model reasoning, facilitating the development of more trustworthy AI systems. It can be used to improve training objectives by emphasizing key steps, enhance interpretability in educational tools, and support debugging in complex decision-making applications. The approach is adaptable to various domains, including legal reasoning, medical diagnosis, and multimodal tasks, where understanding the contribution of each reasoning component is crucial.

The reliance on Monte Carlo sampling incurs high computational costs, limiting scalability. Textual cues alone are insufficient to fully decode step importance, especially in correct responses, restricting interpretability. The current focus on mathematical reasoning tasks may limit generalization to other reasoning types. Further research is needed to optimize estimation efficiency and extend applicability to broader domains.