Do Language Models Update their Forecasts with New Information?
Introduces EvolveCast framework to evaluate LLMs' forecast updates; finds models are overly conservative and inconsistent in belief revision.
Key Findings
Methodology
EvolveCast combines community forecast data from Metaculus with news retrieval via semantic similarity to simulate information updates. It compares model belief shifts (∆p) with human reference shifts (∆h) using metrics like Mean Directional Accuracy (MDA), Mean Squared Error (MSE), Symmetric MSE (SMSPE), and ∆Brier. Multiple models (Qwen, LLaMA) across sizes (1.5B-8B) are evaluated under scenarios where new evidence arrives after training cutoff, assessing their ability to revise forecasts appropriately.
Key Results
- Models show limited belief adjustment; for example, Qwen-7B's confidence increases by only +5% after news, whereas human reference shifts by +15%. Directional agreement (MDA) averages around 0.25, far below the ideal 0.5, indicating weak alignment. Confidence calibration worsens post-update, with positive ∆Brier scores. Across architectures and sizes, models remain conservative, often underreacting to new evidence, highlighting a systemic challenge in belief revision.
Significance
This work exposes fundamental weaknesses in current LLMs' ability to dynamically update beliefs, crucial for applications like policy forecasting, scientific reasoning, and decision support. By benchmarking against human community forecasts, it provides a novel evaluation paradigm emphasizing reasoning over static accuracy. The findings motivate the development of mechanisms that enable models to treat new information as evidence, not just context, thus improving their adaptability and trustworthiness in real-world uncertain environments.
Technical Contribution
The paper introduces EvolveCast, a comprehensive framework integrating external news retrieval and community forecast data to quantify belief update quality. It employs metrics capturing directional consistency, magnitude accuracy, and calibration, offering a multi-faceted view of belief dynamics. This approach advances beyond traditional static evaluation, emphasizing the importance of evidence-based belief revision, and highlights the gap between current model behavior and rational Bayesian updating.
Novelty
First systematic assessment of large language models' belief revision capabilities in dynamic environments, using human community forecasts as a reference. It innovates by combining semantic news retrieval with probabilistic metrics, shifting focus from static accuracy to reasoning under uncertainty. This represents a significant step forward in evaluating models' real-world reasoning abilities.
Limitations
- Models remain heavily anchored to pretraining knowledge, limiting their responsiveness to external evidence. News retrieval based on semantic similarity can introduce irrelevant information, affecting evaluation accuracy. The current framework does not fully capture multi-step belief updates over longer periods, and computational costs for large-scale evaluation are high.
Future Work
Future research should explore integrating probabilistic inference methods like Bayesian updating or particle filtering into LLM architectures. Enhancing multi-turn belief revision, multi-modal information integration, and reinforcement learning strategies could further improve models' responsiveness. Additionally, developing training objectives that explicitly reward rational belief updating remains an open challenge.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language understanding, yet their ability to adapt beliefs dynamically remains limited. Traditional evaluations focus on static accuracy, overlooking the crucial aspect of belief revision when new evidence emerges. This gap hampers their deployment in real-world scenarios like policy forecasting, scientific prediction, and decision-making under uncertainty. To address this, the authors introduce EvolveCast, a novel framework that assesses how well models revise their forecasts in response to external information.
EvolveCast combines community-based forecast data from Metaculus with news retrieval techniques based on semantic similarity, creating a realistic simulation of information flow. The framework evaluates models' belief updates using metrics such as Mean Directional Accuracy (MDA), Mean Squared Error (MSE), Symmetric MSE (SMSPE), and ∆Brier, comparing model belief shifts (∆p) with human reference shifts (∆h). Experiments across multiple architectures, sizes, and scenarios reveal a consistent pattern: models tend to underreact to new evidence, exhibiting conservative belief updates that are often misaligned with human reasoning.
For example, a Qwen-7B model's confidence might only increase by +5% after a credible news report, while humans adjust by +15%. The average directional agreement (MDA) hovers around 0.25, indicating weak alignment. Calibration metrics show that models often become less calibrated after updates, with positive ∆Brier scores. These findings underscore the systemic challenge in enabling models to treat new information as evidence rather than context, a core requirement for rational belief revision.
The implications are significant: improving belief dynamics in LLMs could vastly enhance their reliability and utility in uncertain environments. The authors suggest future directions including incorporating Bayesian inference, reinforcement learning, and multi-modal data fusion to foster more responsive and calibrated models. Overall, this work provides a foundational step towards more adaptive AI systems capable of reasoning under evolving evidence, bridging the gap between static knowledge and dynamic decision-making.
Deep Analysis
Background
Recent advances in large language models like GPT, LLaMA, and Qwen have demonstrated impressive capabilities in knowledge recall and reasoning tasks. However, these models are primarily evaluated on static benchmarks, which do not reflect their ability to update beliefs in response to new evidence. Prior works such as Forecast-Bench and OpenForecast have begun exploring probabilistic forecasting, but mainly focus on accuracy metrics, neglecting the dynamic aspect of belief revision. In real-world decision-making, especially in domains like policy, science, and technology, the ability to adapt predictions as new information arrives is crucial. Despite progress, models still lack mechanisms to treat external evidence as rational evidence, often defaulting to pretraining knowledge or underreacting to updates, highlighting a significant gap in their reasoning capabilities.
Core Problem
The core challenge lies in the models’ conservative response to new evidence, which hampers their effectiveness in dynamic environments. Unlike Bayesian agents that update beliefs proportionally to evidence, current LLMs process new information as mere tokens without proper probabilistic reasoning. This results in under-adjustment of forecast confidence, miscalibration, and potential misguidance in critical applications. The problem is compounded by the models’ reliance on pretraining knowledge, which acts as a strong anchor, preventing significant belief shifts. Addressing this requires developing mechanisms that enable models to interpret new evidence as genuine probabilistic signals, rather than just contextual cues.
Innovation
The paper introduces EvolveCast, a comprehensive framework for evaluating belief updates in LLMs. Key innovations include: 1) leveraging community forecast data as a reference standard for rational belief shifts; 2) integrating news retrieval via semantic similarity to simulate real-time evidence; 3) employing multi-metric evaluation capturing directional consistency, magnitude accuracy, and calibration. Unlike traditional static benchmarks, EvolveCast emphasizes the reasoning process, assessing whether models treat new information as evidence that should proportionally influence their beliefs. It also explores the effect of accumulated versus single news updates, providing insights into the models’ capacity for multi-step belief revision.
Methodology
- �� Select binary forecasting questions from Metaculus with rich community forecast data. • Retrieve relevant news articles within a one-week window using Google Search API, ranked by semantic similarity. • Generate model predictions at two time points: T0 (before news) and T1 (after news), with prompts constrained to information available up to each timestamp. • Compute belief change (∆p) and compare with human reference belief change (∆h). • Measure directional agreement via MDA, magnitude error via MSE and SMSPE, and calibration change via ∆Brier. • Analyze multiple models (Qwen, LLaMA) across sizes, assessing their responsiveness and calibration. • Conduct ablation studies on accumulated versus single news updates to evaluate the impact of temporal context.
Experiments
The evaluation involves models from DeepSeek-R1 series, including Qwen-1.5B, 7B, and LLaMA-8B, 3.1B. Questions are selected post-October 2023 to avoid knowledge leakage. Each question is paired with a relevant news update, retrieved via semantic similarity, and model predictions are obtained at T0 and T1. The models are prompted with instructions to only use information available up to each timestamp. Metrics include directional agreement (MDA), magnitude errors (MSE, SMSPE), and calibration change (∆Brier). The experiments also compare accumulated news context versus only the latest update, analyzing their effects on belief revision. Human community forecasts serve as a reference standard, providing a rational benchmark for evaluation.
Results
Models exhibit limited belief adjustment; for instance, Qwen-7B’s confidence increases by only +5% after news, compared to +15% in human forecasts. Directional agreement (MDA) averages around 0.25, indicating weak alignment. Calibration metrics (∆Brier) often worsen after updates, showing underconfidence. Across architectures and sizes, models tend to underreact, with no significant improvement from accumulated news context. These results highlight the systemic conservatism and calibration issues in current models’ belief revision mechanisms, emphasizing the need for more rational evidence integration strategies.
Applications
This framework can be used to evaluate and improve AI systems in fields requiring real-time belief updates, such as financial forecasting, scientific modeling, and policy analysis. By enabling models to treat new information as genuine evidence, future systems could become more adaptive and trustworthy. The methodology also guides the development of training objectives that reward rational belief revision, fostering AI that better mimics human reasoning in uncertain environments.
Limitations & Outlook
Models remain heavily anchored to pretraining knowledge, limiting their responsiveness. News retrieval based on semantic similarity may introduce irrelevant information, affecting evaluation accuracy. The current framework does not fully address multi-step belief updates over extended periods, and computational costs are high for large-scale assessments. Future work should focus on integrating probabilistic inference mechanisms and multi-modal data to overcome these limitations.
Plain Language Accessible to non-experts
Imagine you’re in a classroom, and your teacher gives you a guess about tomorrow’s weather based on what you know today. If later, you hear a weather forecast saying it might rain, you should probably change your guess — maybe bring an umbrella. But sometimes, you stick to your original idea, thinking, ‘Nah, I don’t believe it,’ even if the forecast is credible. Large language models are like students in this classroom. They usually make predictions based on what they’ve learned, but when new information comes in, they often don’t change their minds enough. This study is like testing whether these models can update their guesses properly when they get new clues, like a weather forecast. The researchers found that models tend to be too cautious, only slightly changing their predictions, unlike humans who adjust more confidently. The goal is to teach models to be more flexible and rational, so they can better handle changing information — just like a smart student who updates their opinion when they hear new evidence. This way, AI systems can become more reliable and useful in real life, where facts and situations are always evolving.
ELI14 Explained like you're 14
Imagine you’re playing a game where you have to guess if your friend will come to your party tomorrow. At first, you might think, ‘Probably not,’ because you don’t have much info. But then, your friend texts you saying they might come if they finish their homework early. Now, you should probably change your guess to ‘Maybe yes,’ right? Well, big AI models are kind of like that. They usually make predictions based on what they’ve learned before, but when they hear new news, they’re supposed to update their guesses. The problem is, they often don’t change their minds enough — they stay too confident in their original answer. Researchers wanted to see if these models could be smarter and update their guesses more like humans do. They used a special test called EvolveCast, which checks if models change their predictions properly after new info. Turns out, most models are too cautious and only make tiny adjustments, unlike humans who are more flexible. The scientists hope that one day, AI can learn to be more like us — adjusting beliefs quickly and reasonably when new evidence appears. That way, AI can make better decisions in real life, where facts are always changing and we need to stay updated.
Abstract
Prior work has largely treated forecasting as a static task, failing to consider how forecasts and the confidence in them should evolve as new evidence emerges. To address this gap, we introduce EvolveCast, a framework for evaluating whether large language models revise their forecasts appropriately in response to new information. In particular, EvolveCast assesses whether LLMs update their forecasts when presented with information released after their training cutoff. We use human forecasters as a comparative reference to assess forecast updates and confidence calibration under new information. While LLMs demonstrate some responsiveness to new information, their updates are often inconsistent or overly conservative. We further find that both verbalized and logits-based confidence estimates remain far from the human reference standard. Across settings with a variety of LLMs, models tend to be conservative in updating their forecasts. These findings suggest that current approaches (e.g., RAG-based methods) for updating model knowledge are insufficient for probabilistic reasoning; models treat new information as retrieval context rather than evidence that shifts posterior probability. EvolveCast thus underscores the need for more robust mechanisms to incorporate external knowledge into belief dynamics.