Algorithmic Blindness in Large Language Models: A Calibration Study of Performance Prediction
This study evaluates 8 state-of-the-art LLMs' calibration in predicting causal discovery algorithm performance, revealing widespread 'algorithmic blindness'.
Key Findings
Methodology
Using causal discovery as a testbed, the study combines 13 datasets and 4 algorithms (PC, FCI, LiNGAM, NOTEARS). Each algorithm is run 100 times to establish ground truth performance metrics (precision, recall, F1, SHD). Eight leading LLMs (e.g., Claude, GPT-5) are prompted with three different question formulations to generate performance ranges. The predicted intervals are evaluated for calibration by measuring the fraction that contain the true mean. The approach ensures robustness through multiple prompts and baseline comparisons (random, heuristic). Synthetic datasets with varying network sizes assess generalization and memorization effects.
Key Results
- Across 1,664 comparisons, the average calibrated coverage of all models is only 15.9%, far below the 36.5% of random guessing, indicating poor calibration. Claude performs best at 39.4%, but still far from reliable. Seven models perform worse than random, revealing a systemic failure in performance prediction. Predicted ranges are often 8-27 times wider than actual confidence intervals, yet still miss the true mean, illustrating severe miscalibration. On synthetic data, performance degrades with network size, from 20.3% at 12 nodes to 6.2% at 60 nodes, showing poor generalization to complex structures. Model agreement across different models is low, especially on synthetic data, indicating reliance on memorized statistics rather than structured reasoning.
- These results demonstrate that current LLMs lack the ability to reliably predict algorithm performance based solely on problem structure. The widespread failure suggests that models are primarily retrieving benchmark statistics rather than understanding the underlying algorithms. This 'algorithmic blindness' limits their utility in automated algorithm selection, especially in complex or unseen scenarios. The findings highlight the need for new approaches integrating structured reasoning and calibration techniques to improve performance prediction accuracy.
- The study's implications extend to AI deployment in scientific and engineering tasks, emphasizing caution when relying on LLMs for performance estimation. Improving calibration and reasoning capabilities could enable models to better generalize and provide trustworthy guidance for algorithm choice, ultimately advancing autonomous AI systems. The observed degradation with network size underscores the importance of developing models that can truly understand and reason about complex structures, rather than memorize superficial statistics.
Significance
This research exposes a fundamental limitation of current large language models in structured reasoning tasks: their inability to produce calibrated, reliable performance predictions for algorithms based solely on problem structure. The concept of 'algorithmic blindness' underscores that, despite their vast knowledge, models lack genuine understanding of algorithmic behavior, especially in complex, unseen scenarios. This gap hampers their deployment in critical decision-making processes such as automated algorithm selection, scientific modeling, and engineering optimization. By systematically quantifying this deficiency, the study provides a clear benchmark for future improvements, emphasizing the need for models that integrate structured reasoning, calibration, and domain-specific knowledge. Addressing these challenges could unlock new levels of AI reliability and autonomy, transforming how algorithms are chosen and evaluated in practice.
Technical Contribution
The paper introduces a rigorous evaluation framework combining multiple datasets, algorithms, and metrics to assess LLM calibration in performance prediction. It formalizes the concept of 'algorithmic blindness'—the failure to translate declarative algorithmic knowledge into calibrated, structure-conditioned performance estimates. The methodology involves extensive bootstrap-based ground truth computation, multi-prompt querying, and synthetic data testing to distinguish memorization from reasoning. The study quantifies calibration via coverage metrics, revealing systemic biases and poor generalization, especially as network complexity increases. These insights provide a foundation for developing models with improved structured reasoning and calibration, advancing the state-of-the-art in AI interpretability and reliability.
Novelty
This work is the first comprehensive study to systematically evaluate large language models' ability to predict algorithmic performance in a structured, calibration-focused manner. Unlike prior research on factual knowledge or reasoning benchmarks, it emphasizes calibration metrics and synthetic generalization tests, revealing a pervasive 'algorithmic blindness.' The concept of measuring model reliance on memorized benchmark statistics versus genuine reasoning is novel, providing a new lens to assess AI understanding. The integration of multiple datasets, algorithms, and prompting strategies offers a holistic view of current limitations, setting a benchmark for future research in model calibration and structured inference.
Limitations
- The evaluation focuses solely on causal discovery algorithms; other domains may exhibit different behaviors. The generality of findings across AI tasks remains to be validated.
- The ground truth relies on empirical bootstrap means, which may be unstable for high-variance algorithms. The study's scope is limited to specific datasets and metrics.
- Prompt design and model training strategies influence results; exploring diverse prompting and fine-tuning could improve calibration. Computational costs for extensive experiments are high, limiting scalability.
Future Work
Future research should explore integrating explicit reasoning modules, knowledge bases, and calibration techniques to enhance performance prediction. Developing multi-modal prompts and interactive systems could improve understanding of complex structures. Extending evaluations across broader AI domains and tasks will validate generality. Additionally, advancing training paradigms to reduce reliance on memorization and improve structured reasoning will be crucial for deploying trustworthy AI in automated algorithm selection and scientific discovery.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个厨房里做饭。每次做菜，你都需要根据食材的特性选择合适的烹饪方法。有些厨师（模型）只记住了以前做过的菜谱（记忆），遇到新食材或新菜谱时，他们就不知道该用什么方法，结果做出来的菜可能不合口味。这就像大模型在预测算法表现时，只是依赖记忆中的统计信息，没有真正理解算法的结构和原理。研究发现，这些模型的预测范围非常宽，很多时候都不能准确覆盖真实的表现，说明它们没有真正理解问题，只是在模仿过去的经验。未来，我们希望模型能像有经验的厨师一样，理解食材和菜谱的关系，才能做出更好、更可靠的菜。这就像让模型学会真正理解算法背后的逻辑，而不是死记硬背。这样，它们才能在面对新问题时，做出更准确的判断和预测。
ELI14 Explained like you're 14
想象你在玩一个迷宫游戏，你的朋友（模型）试图猜出迷宫的最佳出口路线。他平时只记住了以前遇到的迷宫路径（记忆），但没有真正理解迷宫的结构。当遇到一个全新的复杂迷宫时，他的猜测就变得很糟糕，甚至比随便走走还差。这就像大模型在预测算法表现时，只是依赖以前看到的统计信息，没有真正理解算法背后的原理。研究发现，这些模型的预测范围很宽，很多时候都不能包含真实的表现结果，说明它们没有真正理解问题，只是在模仿过去的经验。就像那个猜迷宫的朋友，他需要学习迷宫的基本结构和逻辑，才能更好地找到出口。未来，我们希望模型像聪明的探险家一样，真正理解迷宫的设计，而不是死记硬背路径。这样，它们才能在面对新迷宫时，做出更准确的判断和预测。
Abstract
Large language models (LLMs) demonstrate remarkable breadth of knowledge, yet their ability to reason about computational processes remains poorly understood. Closing this gap matters for practitioners who rely on LLMs to guide algorithm selection and deployment. We address this limitation using causal discovery as a testbed and evaluate eight frontier LLMs against ground truth derived from algorithm executions. We find systematic, near-total failure across models. The predicted ranges are far wider than true confidence intervals yet still fail to contain the true algorithmic mean in most cases. Most models perform worse than random guessing. The best model's marginal improvement points to benchmark memorization rather than principled reasoning. We term this failure algorithmic blindness and argue it reflects a fundamental gap between declarative knowledge about algorithms and calibrated procedural prediction.