Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry

Recent developments in large language models (LLMs) like GPT-4 and PaLM have demonstrated remarkable capabilities in natural language understanding and reasoning. Early efforts in AI forecasting focused on single-model pipelines, utilizing retrieval-augmented generation (Lewis et al., 2020) and chain-of-thought prompting (Wei et al., 2022). However, these systems often face calibration issues and limited accuracy in complex tasks. Inspired by human collective decision mechanisms such as expert panels and Delphi methods, researchers have explored multi-agent systems to leverage distributed knowledge. Yet, most existing approaches treat all models as receiving identical evidence, leading to correlated errors and limited gains, as highlighted by empirical studies showing high error correlation across models (Kim et al., 2025). The challenge remains to design evidence distribution and communication strategies that foster diversity and effective collaboration.

The core problem is that when multiple models or agents are provided with the same evidence, their errors tend to be highly correlated, which diminishes the benefit of ensemble or collaborative reasoning. This phenomenon, known as Herding, prevents the system from achieving the theoretical advantages of diversity. Existing multi-agent debate frameworks, such as Wang et al. (2024), show limited improvements because they do not sufficiently break error correlation. The fundamental question is how to distribute evidence and facilitate communication so that models can hold diverse, complementary signals, thereby enabling genuine belief revision and improved collective accuracy.

This work introduces the principle of designed information asymmetry, operationalized through Evidence Routing, Rationale Sharing, and Confidence-weighted Aggregation. Evidence Routing uses BM25 relevance ranking to partition evidence into shared and private subsets, ensuring each model has unique signals. Rationale Sharing enables models to exchange structured reasoning snippets over multiple rounds, breaking the Markov chain limitations of mere numeric exchange. Confidence-weighted aggregation emphasizes predictions from models with stronger private signals, further boosting accuracy. The combination of these innovations theoretically reduces error correlation and empirically improves forecasting performance, surpassing existing homogeneous-input methods.

• �� Evidence Routing: Using BM25 relevance scores, evidence is split into a shared public pool and disjoint private subsets, ensuring diversity and common ground.
• �� Iterative Deliberation: Models generate structured forecasts and rationales over multiple rounds, exchanging reasoning snippets to propagate private signals.
• �� Rationale Sharing: During each round, models incorporate reasoning excerpts from others, transforming the communication from a closed to an open system.
• �� Confidence-weighted Aggregation: Final predictions are combined in logit space, weighted by prediction extremity, to emphasize the most informed models.
• �� Theoretical Analysis: Formal proofs demonstrate that evidence partitioning reduces inter-model error correlation, supported by covariance and error decomposition analyses.

The evaluation uses PolyGym, a controlled benchmark with 375 real-world Polymarket questions, fixed evidence pools, and standardized retrieval. Models include gpt-4o, DeepSeek-V3.2, and Llama-4-Scout, tested under different configurations: with/without evidence partitioning, multiple rounds, and rationale sharing. Metrics include Brier score and accuracy. Ablation studies systematically remove components to assess their impact. Cross-model tests verify generalization across architectures. Hyperparameters such as ρ=0.5 (public/private split), R=2 rounds, and relevance-based routing are used to optimize performance.

InfoDelphi achieves a Brier score of 0.178, outperforming all baselines by 12-18%, and accuracy reaches 77.9%, surpassing single models and homogeneous multi-agent methods. Evidence partitioning alone improves performance, and multi-round deliberation with rationale sharing yields further gains. Ablation results confirm that removing evidence asymmetry or rationale sharing significantly degrades results, validating the importance of the proposed design. Cross-model experiments demonstrate robustness and scalability across different LLM architectures.

This framework can be applied to real-time financial forecasting, policy analysis, and crisis prediction, especially in environments with dispersed and asymmetric information. Its ability to leverage diverse signals and facilitate effective communication among models makes it suitable for complex decision-making tasks requiring high accuracy and calibration. Additionally, it provides a foundation for developing interpretable and robust multi-agent AI systems in various domains.

The approach depends on the quality of evidence retrieval; poor retrieval reduces effectiveness. Multi-round deliberation increases computational costs and may lead to convergence issues in highly unbalanced information settings. Error correlation, though reduced, is not eliminated entirely, and the method's scalability to larger, more complex tasks remains to be validated. Future work should address these challenges to enhance robustness and efficiency.