Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations
This study evaluates whether LLM self-explanations improve users' ability to predict model behavior via counterfactual perturbations, showing consistent gains especially with pragmatic strategies.
Key Findings
Methodology
Using StrategyQA, the study integrates chain-of-thought (CoT) and post-hoc explanations, applying diverse pragmatic perturbations—such as presupposition flips, lexical substitutions, scalar adjustments, and contextual enrichments—to generate counterfactual questions. Human and LLM judges predict model answers under these perturbations, with the impact measured by accuracy improvements (acc_improvement). The framework involves two phases: initial prediction without explanations and subsequent prediction with explanations, enabling quantification of explanation utility. Multiple models (GPT-3.5, GPT-4, Llama-3.70B) are employed for generating counterfactuals and explanations, ensuring robustness. The evaluation assesses how different perturbation strategies influence the ability of judges to simulate model behavior, revealing the stability and effectiveness of pragmatic-based methods.
Key Results
- Self-explanations significantly improve prediction accuracy for both human and LLM judges, with average gains of 0.02-0.04. Pragmatic perturbations such as lexical substitution and presupposition flip yield more stable and consistent improvements, especially for weaker judges. Chain-of-thought explanations do not outperform post-hoc rationales, indicating that targeted explanations are sufficient. The results demonstrate that explanations help reduce bias and anchoring effects, particularly when the model's initial answer is incorrect. Human confidence increases from 3.21 to 4.38, validating the interpretability benefits.
- Different perturbation strategies exhibit varying stability; pragmatic, pragmatics-based methods outperform purely generative ones in most settings. The effectiveness diminishes as judge strength increases, with stronger judges already close to the true model behavior, leaving less room for improvement. Qualitative analysis confirms that explanations assist users in identifying the model’s reasoning cues, reducing reliance on initial answers and bias. The LLM judges show high agreement with human predictions, supporting the validity of automated evaluation.
- Overall, the study underscores the value of incorporating linguistic pragmatics into explanation design, providing a systematic approach to enhance model interpretability and trustworthiness. The findings suggest that well-crafted explanations, especially when combined with targeted counterfactual perturbations, can substantially improve users’ mental models of AI systems, paving the way for more transparent and reliable AI deployment in critical applications.
Significance
This work advances the understanding of explanation utility in AI, demonstrating that explanations—particularly those grounded in pragmatic principles—can meaningfully enhance users’ predictive accuracy of model responses under counterfactual scenarios. It bridges cognitive science and NLP by applying Gricean maxims to generate more meaningful perturbations, thereby addressing a key challenge in explainable AI: how to design explanations that are both faithful and practically useful. The insights gained have implications for developing more robust, transparent AI systems capable of supporting human decision-making in high-stakes environments, such as healthcare, legal, and autonomous systems. By systematically comparing different perturbation strategies, the study offers a blueprint for future research aiming to optimize explanation design and evaluation.
Technical Contribution
The paper introduces a novel framework integrating pragmatic linguistics with counterfactual reasoning to evaluate explanation usefulness. It innovatively applies Grice’s maxims to generate diverse, contextually relevant perturbations, enabling a more nuanced assessment of explanation fidelity and utility. The two-phase simulation setup—comparing predictions before and after explanations—provides a rigorous quantitative measure of explanation impact. Additionally, the study employs multiple models and human validation, establishing a comprehensive evaluation pipeline. These contributions push the frontier of explainability research by combining linguistic theory with empirical validation, offering new tools for assessing and improving model interpretability.
Novelty
This research is the first to systematically incorporate Gricean pragmatic principles into the design of counterfactual perturbations for evaluating LLM explanations. Unlike prior work relying solely on generative models, it emphasizes the importance of pragmatic relevance and informativeness, resulting in more stable and meaningful perturbations. The dual evaluation with human and LLM judges, along with the focus on stability across different models and explanation types, distinguishes it from existing approaches. It provides a new perspective on how linguistic pragmatics can inform the creation of more effective and trustworthy explanations in AI systems.
Limitations
- The experimental setup is primarily based on StrategyQA, limiting direct generalization to other domains or complex real-world tasks. The perturbation strategies, while diverse, may not capture all nuances of pragmatic language use, especially in multi-turn dialogues or multimodal contexts. The reliance on pre-trained models for judgment introduces biases and performance variability, which could affect the robustness of conclusions. Further research is needed to adapt these methods to more complex, dynamic environments and to incorporate causal reasoning for deeper interpretability.
Future Work
Future directions include extending the framework to multi-modal tasks, integrating causal inference for deeper explanation fidelity, and exploring adaptive perturbation strategies that dynamically tailor to context. Developing interactive systems that leverage pragmatic perturbations for real-time explanation refinement is also promising. Additionally, investigating the impact of explanations on long-term user trust and decision-making in real-world applications will be crucial. Cross-lingual and cross-cultural studies could further enhance the universality of pragmatic-based explanation design.
AI Executive Summary
The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing, yet understanding their decision-making remains a challenge. While LLMs can generate verbal explanations for their answers, questions about the faithfulness and utility of these rationales persist. This study tackles this issue by systematically evaluating whether self-explanations genuinely help users predict model behavior, especially under counterfactual scenarios.
Using the StrategyQA dataset, the researchers designed a comprehensive framework that incorporates diverse pragmatic perturbations—such as presupposition flips, lexical substitutions, scalar adjustments, and contextual enrichments—to generate counterfactual questions. These perturbations are grounded in Gricean principles, ensuring relevance and informativeness. The core idea is to compare prediction accuracy in two phases: before and after providing the model’s explanations. This setup allows quantification of how much explanations improve the ability to simulate the model’s responses.
Experiments involved both human participants and LLM judges, with models like GPT-3.5, GPT-4, and Llama-3.70B. Results consistently showed that explanations enhance prediction accuracy, with average improvements of 0.02-0.04 points. Pragmatic-based perturbations, especially lexical and presupposition manipulations, proved more stable and effective than purely generative methods. Notably, the benefit was more pronounced for weaker judges, indicating that explanations help bridge understanding gaps. Qualitative analysis revealed that explanations assist users in identifying the reasoning cues, reducing biases and anchoring effects, particularly when initial answers are wrong.
This research underscores the importance of linguistically grounded perturbations in explanation evaluation, offering a new pathway for developing more trustworthy AI systems. It highlights that well-crafted explanations, combined with strategic counterfactuals, can significantly improve human-AI interaction, fostering greater transparency and confidence. Despite some limitations—such as dataset scope and model biases—the findings pave the way for future work in multi-modal, causal, and interactive explanation frameworks, ultimately aiming to build AI that is both powerful and interpretable.
Deep Dive
Abstract
Large Language Models (LLMs) can produce verbalized self-explanations, yet prior studies suggest that such rationales may not reliably reflect the model's true decision process. We ask whether these explanations nevertheless help users predict model behavior, operationalized as counterfactual simulatability. Using StrategyQA, we evaluate how well humans and LLM judges can predict a model's answers to counterfactual follow-up questions, with and without access to the model's chain-of-thought or post-hoc explanations. We compare LLM-generated counterfactuals with pragmatics-based perturbations as alternative ways to construct test cases for assessing the potential usefulness of explanations. Our results show that self-explanations consistently improve simulation accuracy for both LLM judges and humans, but the degree and stability of gains depend strongly on the perturbation strategy and judge strength. We also conduct a qualitative analysis of free-text justifications written by human users when predicting the model's behavior, which provides evidence that access to explanations helps humans form more accurate predictions on the perturbed questions.