Debating with More Persuasive LLMs Leads to More Truthful Answers
Proposes debate framework to improve weak model evaluation of strong models, achieving 76% accuracy with non-expert judges.
Key Findings
Methodology
This study employs an adversarial debate protocol where two expert models (strong models) argue for opposing answers over multiple rounds, while a non-expert judge (weaker model) evaluates the debate to select the correct answer. The framework leverages unsupervised metrics such as win rate and Elo ratings to assess model capabilities without ground truth labels. Different debate protocols, including adversarial debate, interactive debate, and consultancy, are tested on the QuALITY dataset, involving models like GPT-4-Turbo and Claude 2.1. The approach also incorporates optimizing expert model persuasiveness via inference-time methods like best-of-N sampling and critique-refinement, aiming to enhance the judge’s ability to identify truth.
Key Results
- On the QuALITY reading comprehension task, non-expert models guided by debate achieved 76% accuracy, compared to 48% baseline, while human judges reached 88%. Optimization of expert debaters' persuasiveness further improved non-expert judgment accuracy, demonstrating the effectiveness of the debate mechanism in truth extraction.
- Model strength, measured by Elo ratings, correlates positively with debate persuasiveness and judge accuracy. Stronger models are better at arguing for correct answers, and optimizing for win rate enhances their truthfulness. Different judges (including humans) show consistent improvements, validating the robustness of the approach.
- The experiments reveal that debate-based evaluation surpasses traditional supervised methods, especially in zero-label settings. The findings suggest that training models to win debates can lead to more truthful outputs, with implications for scalable AI oversight and alignment.
Significance
This work introduces a paradigm shift in model evaluation, moving away from reliance on labeled data toward unsupervised, adversarial methods. By demonstrating that weaker models can reliably assess stronger models through debate, it addresses the scalability and reliability challenges in AI alignment. The approach offers a promising pathway for deploying large models safely in real-world applications, where ground truth may be unavailable or unreliable. It also opens avenues for integrating human and machine judgment in a scalable, interpretable framework, thereby advancing the field of trustworthy AI.
Technical Contribution
The paper develops a novel unsupervised evaluation framework based on adversarial debate, integrating win rate and Elo rating metrics to quantify model persuasiveness. It innovates by optimizing expert debaters' ability to argue convincingly, thereby indirectly calibrating the judgment process. The study also systematically compares multiple debate protocols and models, providing empirical evidence for the effectiveness of debate in truth discovery. These contributions extend current model evaluation paradigms, enabling scalable, label-free oversight.
Novelty
This is the first comprehensive validation of weak models supervising strong models via adversarial debate in a zero-label setting. Unlike prior work relying on supervised fine-tuning or explicit ground-truth labels, this approach leverages unsupervised metrics to evaluate model truthfulness. The integration of Elo ratings and win rate as proxies for model strength and persuasiveness represents a significant innovation, setting a new standard for scalable AI oversight.
Limitations
- The method relies heavily on the assumption that truthful arguments are more persuasive, which may be exploited by deceptive models. The risk of manipulation remains a concern.
- Current experiments are limited to reading comprehension tasks with textual data; applicability to more complex, multi-modal, or ambiguous tasks needs further validation.
- Computational costs associated with multiple sampling and debate rounds are high, potentially limiting real-time deployment in large-scale systems.
Future Work
Future research will explore multi-turn, dynamic debate systems with human-in-the-loop feedback to improve robustness and transparency. Extending the framework to multi-modal data and more complex tasks, such as reasoning and decision-making, is also planned. Additionally, developing methods to detect and mitigate deceptive strategies within debate protocols will be crucial for ensuring reliability in real-world AI applications.
AI Executive Summary
As large language models (LLMs) become increasingly capable, ensuring their outputs are truthful and reliable remains a critical challenge. Traditional supervised alignment approaches depend heavily on labeled data, which becomes impractical as models surpass human expertise. This study introduces an innovative adversarial debate framework, where two expert models argue for opposing answers, and a non-expert judge evaluates the debate to select the most convincing argument. This setup simulates a scenario where weaker models oversee stronger ones without ground-truth labels, addressing the scalability issue in AI alignment.
The core idea is that models can be trained or optimized to be persuasive, and that persuasion correlates with truthfulness. The researchers employ metrics like win rate and Elo rating to quantify model strength and persuasiveness, enabling unsupervised evaluation. Extensive experiments on the QuALITY dataset demonstrate that non-expert models guided by debate achieve 76% accuracy, significantly outperforming baseline methods. When human judges participate, accuracy reaches 88%, confirming the method's practical effectiveness.
A key finding is that optimizing expert models for persuasiveness enhances their ability to argue for correct answers, which in turn improves the judgment accuracy of non-experts. The results also show that stronger models are better at convincing judges when they argue truthfully, and that this advantage increases with model capability. These insights suggest that debate can serve as a scalable, unsupervised mechanism for aligning models with truth, especially as models grow more sophisticated.
Despite promising results, the approach has limitations, including potential manipulation by deceptive models and high computational costs. Future work aims to incorporate multi-turn, interactive debates with human feedback, extend to multi-modal tasks, and develop safeguards against deception. Overall, this research paves the way for more reliable, scalable AI oversight, crucial for deploying powerful models safely in real-world scenarios.
Deep Dive
Abstract
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.