Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration
Corex employs multi-model collaboration (Discuss, Review, Retrieve) to significantly improve complex reasoning, outperforming existing baselines with detailed experimental validation.
Key Findings
Methodology
Corex integrates three interaction paradigms among multiple LLM agents: • Discuss mode facilitates iterative exchanges to refine reasoning chains and diversify perspectives. • Review mode involves agents scrutinizing and validating reasoning steps or code, ensuring correctness. • Retrieve mode evaluates the faithfulness of reasoning chains and answers by scoring candidate solutions. These modes operate in a multi-round, dynamic interaction framework, leveraging role-specific prompts and confidence assessments. The system combines collaborative reasoning, code synthesis, and answer verification, effectively reducing hallucinations and increasing reliability across diverse tasks.
Key Results
- On mathematical reasoning datasets like GSM8K and GSM-Hard, Corex-Retrieve achieved accuracy rates of 86.3%, surpassing traditional methods such as CoT-SC (84.6%) and demonstrating robustness on challenging problems.
- In commonsense reasoning tasks (StrategyQA, CSQA), Corex improved average accuracy by approximately 1.2 percentage points, reaching up to 77.6%, indicating strong generalization across task types.
- For symbolic and semi-structured tasks (Big-Bench, FinQA), the framework's code synthesis and multi-round review enabled accuracy improvements over 85%, showing effectiveness in logical and information integration challenges.
Significance
This work advances the state-of-the-art in AI reasoning by systematically harnessing multi-model collaboration, addressing hallucination, bias, and error propagation issues. It offers a task-agnostic, cost-effective solution that enhances the factuality, robustness, and interpretability of LLM-based systems, with broad implications for scientific computing, decision-making, and AI safety. The framework's generality and scalability mark a significant step toward reliable autonomous AI agents capable of complex reasoning.
Technical Contribution
Corex introduces a novel multi-model collaboration paradigm, combining discussion, review, and retrieval modes into a unified framework. This design enables dynamic, role-specific interactions, model validation, and answer verification, providing theoretical guarantees of improved factuality and robustness. It extends prior work on chain-of-thought prompting and ensemble methods by embedding interactive verification and confidence scoring, thus offering a scalable, task-agnostic architecture for complex reasoning tasks.
Novelty
This is the first comprehensive systematization of multi-model collaboration employing three distinct interaction modes—discussion, review, and retrieval—in complex reasoning contexts. Unlike prior approaches limited to static prompts or voting, Corex emphasizes dynamic, role-based exchanges and validation, setting a new standard for collaborative AI systems. Its task-agnostic design and multi-round interaction mechanism represent a fundamental innovation in the field.
Limitations
- The framework's reliance on large-scale LLMs incurs high computational costs, limiting real-time deployment in resource-constrained environments.
- Model interactions may propagate biases or reinforce errors if not carefully managed, especially in adversarial or domain-specific tasks.
- In extremely complex or specialized fields, the current collaboration mechanisms might require further adaptation with domain knowledge integration.
Future Work
Future research will focus on optimizing interaction efficiency, reducing computational overhead, and incorporating external knowledge bases. Exploring adaptive role assignment, extending to multi-modal data, and developing theoretical guarantees for collaboration stability are promising directions. Additionally, scaling the framework for real-world deployment in safety-critical applications remains a key goal.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language processing, yet their capacity for complex reasoning remains constrained by internal representation limitations. Traditional approaches like chain-of-thought prompting and self-consistency decoding have improved performance but still suffer from hallucinations, bias, and error propagation. To address these issues, this paper introduces Corex, a novel framework inspired by human social behaviors, which leverages multi-model collaboration through three paradigms: Discuss, Review, and Retrieve.
Corex transforms multiple autonomous LLM agents into a collaborative reasoning system. In the Discuss mode, models engage in iterative exchanges to refine their reasoning chains, promoting diversity and factuality. The Review mode involves agents scrutinizing each other's outputs, verifying correctness, and suggesting improvements. The Retrieve mode assesses the faithfulness of reasoning chains and answers by scoring candidate solutions, selecting the most aligned response. These modes operate in a multi-round, dynamic interaction, enabling models to collectively overcome hallucinations and biases.
Extensive experiments across four reasoning task categories—mathematical, commonsense, symbolic, and semi-structured—demonstrate Corex’s superior performance. On GSM8K and GSM-Hard, accuracy exceeds 86%, outperforming baseline methods like CoT-SC. In commonsense tasks such as StrategyQA, accuracy improves by over 1.2 points, reaching 77.6%. For symbolic reasoning, the framework achieves over 85% accuracy, especially in code synthesis and logical tasks. These results highlight the framework’s robustness, task adaptability, and cost-effectiveness.
This work significantly advances AI reasoning by systematically integrating collaborative mechanisms, providing a scalable, task-agnostic approach that enhances factuality, reliability, and interpretability. Its potential applications span scientific computing, decision support, and AI safety. Future directions include optimizing interaction efficiency, incorporating external knowledge, and extending to multi-modal data, aiming to realize autonomous, trustworthy AI systems capable of complex reasoning in real-world scenarios.
Deep Dive
Plain Language Accessible to non-experts
想象一个团队在解决一个复杂的难题，比如修建一座大房子。每个人都负责不同的部分，有的负责设计，有的负责检查，有的负责采购材料。大家不断讨论，提出不同的方案，然后互相检查，确保每一步都正确。最后，大家一起确认最好的方案，确保房子既漂亮又结实。Corex的想法也是这样，让多个“聪明的机器人”合作，互相帮助，确保每个步骤都准确无误，最终完成一项复杂任务。这种合作比单打独斗更可靠，也更聪明。
ELI14 Explained like you're 14
想象你和你的朋友们在一起解决一道难题，比如做一道数学题。每个人都提出自己的想法，有的用公式，有的画图。你们不断讨论，互相检查答案，确保没有错误。有时候有人会发现别人的错误，然后改正。最后，你们一起决定哪个答案最靠谱。这就像Corex一样，让很多“聪明的机器人”合作，互相检查、讨论，确保答案既快又准。这样一来，解决复杂问题就变得更容易，也更有信心。
Abstract
Large Language Models (LLMs) are evolving at an unprecedented pace and have exhibited considerable capability in the realm of natural language processing (NLP) with world knowledge. Benefiting from ultra-large-scale training corpora, a single LLM can manage typical NLP tasks competently. However, its performance in executing reasoning tasks is still confined by the limitations of its internal representations. To push this boundary further, we introduce Corex in this paper, a suite of novel general-purpose strategies that transform LLMs into autonomous agents pioneering multi-model collaborations for complex task-solving. Inspired by human behaviors, Corex is constituted by diverse collaboration paradigms including Debate, Review, and Retrieve modes, which collectively work towards enhancing the factuality, faithfulness, and reliability of the reasoning process. These paradigms foster task-agnostic approaches that enable LLMs to ''think outside the box,'' thereby overcoming hallucinations and providing better solutions. Through extensive experiments across four different types of reasoning tasks, we demonstrate that orchestrating multiple LLMs to work in concert yields substantially better performance compared to existing methods. Further results and in-depth analysis demonstrate the cost-effectiveness of our method, facilitating collaboration among different LLMs and promoting annotation efficiency.