AIA Forecaster: Technical Report
AIA Forecaster combines agentic search, reconciliation, and calibration to achieve expert-level forecasting performance.
Key Findings
Methodology
AIA Forecaster employs a multi-agent architecture integrating agentic search, supervised reconciliation, and statistical calibration. Multiple independent agents perform adaptive, iterative searches over high-quality news sources, then generate probabilistic forecasts based on retrieved evidence. A supervisor agent analyzes disagreements among forecasts, queries additional information if needed, and reconciles predictions. Final probabilities are calibrated via Platt scaling or extremization techniques to correct biases. This architecture leverages reinforcement learning for search optimization, ensemble averaging for stability, and probabilistic calibration for accuracy, achieving human-level performance on ForecastBench and outperforming benchmarks in liquid markets when combined with market consensus.
Key Results
- On ForecastBench, the AIA Forecaster's Brier score matches that of human superforecasters, surpassing all prior LLM baselines, demonstrating its expert-level predictive ability across diverse questions.
- In the liquid market benchmark, the single model underperforms market consensus slightly; however, an ensemble combining the model with market predictions significantly outperforms either alone, indicating the model provides additive, complementary information.
- Repeated sampling and ensemble strategies greatly improve forecast stability, while statistical calibration techniques like Platt scaling effectively mitigate conservative biases, leading to substantial performance gains.
Significance
This work advances AI's capability in judgmental forecasting, addressing longstanding challenges of information integration, bias correction, and prediction stability. It demonstrates that large language models, when equipped with adaptive search, reconciliation, and calibration, can reach or exceed human expert performance at scale. Such systems have transformative potential in policy analysis, financial markets, and scientific prediction, enabling faster, more reliable decision-making. The methodology sets a new standard for AI-based forecasting, bridging the gap between human expertise and machine intelligence, and opens avenues for scalable, transparent, and verifiable predictive systems.
Technical Contribution
The paper introduces a novel multi-agent framework that combines autonomous, adaptive search with supervised reconciliation and probabilistic calibration. Key innovations include reinforcement learning-guided search strategies, a supervisor agent that analyzes forecast divergences to generate targeted queries, and the mathematical demonstration of the equivalence between Platt scaling and forecast extremization. These contributions enable the system to produce stable, calibrated, and expert-level predictions across large question sets, representing a significant leap over existing single-model or static approaches. The integration of these techniques offers a flexible, scalable architecture applicable to various complex forecasting tasks.
Novelty
This research is the first to realize large-scale, verifiably expert-level forecasting by integrating agentic search, dynamic reconciliation, and statistical calibration within a multi-agent system. Unlike prior work limited to static models or simple ensemble methods, this approach actively seeks information, intelligently resolves forecast disagreements, and rigorously corrects biases, achieving superior performance on diverse benchmarks. Its ability to combine multiple advanced techniques into a cohesive framework marks a new milestone in AI forecasting, setting a foundation for future research in scalable, transparent, and high-accuracy predictive systems.
Limitations
- The model underperforms in liquid markets relative to market consensus, indicating room for improvement in real-time information integration and handling of high-noise environments.
- Search relies heavily on the quality and coverage of news sources, which may limit performance in domains with sparse or delayed information flow.
- Bias correction methods like Platt scaling, while effective, may falter in extreme or highly uncertain scenarios, suggesting the need for more sophisticated, adaptive calibration techniques.
Future Work
Future research will explore multi-modal data integration, including images and videos, to enhance understanding of complex events. Improving autonomous search strategies' robustness and efficiency, especially under limited or noisy data, is also a priority. Additionally, developing dynamic, context-aware calibration methods and enhancing model interpretability will be crucial. Extending the framework to multi-class or continuous prediction tasks, and deploying in real-world decision systems, will further validate and refine the approach. These directions aim to make AI forecasting more accurate, reliable, and applicable across diverse domains.
AI Executive Summary
AIA Forecaster marks a significant breakthrough in AI-driven judgmental forecasting. Traditional methods often struggle with integrating unstructured information, correcting biases, and maintaining stability across large question sets. This system innovatively combines multiple advanced techniques: autonomous, adaptive search agents that gather relevant evidence from high-quality news sources; a supervisory agent that analyzes forecast disagreements and generates targeted queries; and statistical calibration methods like Platt scaling to correct conservative biases. These components work synergistically within a multi-agent architecture, enabling the system to produce predictions that rival human experts.
In rigorous evaluations on the ForecastBench benchmark, the AIA Forecaster achieved performance statistically indistinguishable from that of superforecasters, surpassing all prior large language model baselines. Its ability to process diverse questions across domains such as politics, economics, and technology demonstrates its versatility. When tested on a more challenging liquid market-based benchmark, the single model's predictions lagged slightly behind market consensus. However, an ensemble combining the model with market predictions significantly outperformed either source alone, illustrating the model's capacity to provide valuable, additive information.
The core innovations lie in the agentic search strategy, which allows the system to dynamically seek out relevant information, and the supervised reconciliation process, which intelligently resolves forecast disagreements. The application of statistical calibration further refines probability estimates, addressing the inherent biases of large language models. These technical advances collectively establish a new state of the art in AI forecasting, with broad implications for policy, finance, and scientific prediction.
Despite these successes, limitations remain. The model's performance in highly noisy or rapidly evolving environments needs enhancement, and its reliance on news sources may restrict applicability in data-sparse domains. Future work aims to incorporate multi-modal data, improve autonomous search robustness, and develop more adaptive calibration techniques. Overall, this research paves the way for scalable, transparent, and expert-level AI forecasting systems, promising a new era of intelligent decision support.
Deep Dive
Abstract
This technical report describes the AIA Forecaster, a Large Language Model (LLM)-based system for judgmental forecasting using unstructured data. The AIA Forecaster approach combines three core elements: agentic search over high-quality news sources, a supervisor agent that reconciles disparate forecasts for the same event, and a set of statistical calibration techniques to counter behavioral biases in large language models. On the ForecastBench benchmark (Karger et al., 2024), the AIA Forecaster achieves performance equal to human superforecasters, surpassing prior LLM baselines. In addition to reporting on ForecastBench, we also introduce a more challenging forecasting benchmark sourced from liquid prediction markets. While the AIA Forecaster underperforms market consensus on this benchmark, an ensemble combining AIA Forecaster with market consensus outperforms consensus alone, demonstrating that our forecaster provides additive information. Our work establishes a new state of the art in AI forecasting and provides practical, transferable recommendations for future research. To the best of our knowledge, this is the first work that verifiably achieves expert-level forecasting at scale.