Agreement Before Diversity: Verification-First Complementarity for Heterogeneous Language-Model Coordination
ABD method achieves 59.43% accuracy on LiveCodeBench-v6, outperforming Single9 and HAC.
Key Findings
Methodology
The paper introduces a novel method for heterogeneous language model coordination called Agreement-Before-Diversity (ABD). This method uses a frozen, label-free decision rule to retain an anchor answer if two additional trusted samples corroborate it under a fixed equivalence relation; otherwise, it is replaced by heterogeneous synthesis. ABD does not assume independence or calibrated confidence, with expected inference cost approximately eight minus five times the coverage in number of calls.
Key Results
- On LiveCodeBench-v6, the ABD method achieved 59.43% accuracy, compared to 52.57% for Single9 and 52.00% for HAC. On the untouched GPQA-Diamond split, ABD reached 75.00% accuracy, with controls at 72.78%.
- Experiments show ABD method has significant anchor advantage on the protected subset with a coverage of 73.33%, contributing 2.78 percentage points.
- In heterogeneous synthesis experiments, ABD method improved by 6.67 percentage points over single model synthesis on GPQA-Diamond.
Significance
The ABD method provides replacement authority through verification structure, addressing the lack of principled replacement criteria in heterogeneous language models after expanding candidate answer space. This method not only improves model accuracy but also offers an auditable decision framework, impacting both academia and industry.
Technical Contribution
ABD method fundamentally differs from existing SOTA methods by offering new theoretical guarantees and engineering possibilities, especially in coordinating heterogeneous language models. Through verification structure, ABD achieves precise decision-making without relying on independence or calibrated confidence.
Novelty
ABD method first introduces verification structure as replacement authority, offering a new decision principle compared to existing model coordination methods. It decides replacement through frozen relational events, avoiding error coverage issues in traditional methods.
Limitations
- On LiveCodeBench-v6, ABD method's coverage is only 1.71%, indicating limited role of verification structure in code generation tasks.
- In some cases, heterogeneous synthesis may lead to incorrect replacement of correct answers.
- The method may increase computational cost under high coverage scenarios.
Future Work
Future research directions include optimizing ABD method's verification structure to enhance coverage and reduce computational cost. Additionally, exploring its application across different tasks and model combinations to verify its universality.
AI Executive Summary
Coordinating heterogeneous language models is a crucial topic in modern AI research. Traditional methods, when expanding the candidate answer space, lack principled replacement criteria, potentially leading to incorrect answers overwriting correct ones. This paper proposes a novel method called Agreement-Before-Diversity (ABD), using a frozen, label-free decision rule to retain an anchor answer if two additional trusted samples corroborate it; otherwise, it is replaced by heterogeneous synthesis. ABD does not assume independence or calibrated confidence, with expected inference cost approximately eight minus five times the coverage in number of calls. Experimental results show significant accuracy improvements on LiveCodeBench-v6 and GPQA-Diamond. This method not only enhances model accuracy but also provides an auditable decision framework, impacting both academia and industry. Future research directions include optimizing ABD method's verification structure to enhance coverage and reduce computational cost.
Deep Analysis
Background
The evolution of language models is a significant research direction in AI. Early models primarily relied on single generation strategies, leading to high error coverage. Recently, coordinating heterogeneous language models has become key to solving this issue. However, existing methods, when expanding candidate answer space, lack principled replacement criteria, potentially leading to incorrect answers overwriting correct ones.
Core Problem
Heterogeneous language models, after expanding candidate answer space, lack principled replacement criteria, potentially leading to incorrect answers overwriting correct ones. This issue not only affects model accuracy but also increases computational cost.
Innovation
ABD method uses a frozen, label-free decision rule to retain an anchor answer if two additional trusted samples corroborate it; otherwise, it is replaced by heterogeneous synthesis. Compared to traditional methods, ABD offers an auditable decision framework, avoiding error coverage issues.
Methodology
- �� ABD method decides replacement through frozen relational events.
- �� Retains anchor answer if two additional trusted samples corroborate it.
- �� Otherwise replaced by heterogeneous synthesis.
- �� Does not assume independence or calibrated confidence.
Experiments
Experimental design includes validation on LiveCodeBench-v6 and GPQA-Diamond. Using a fixed evaluation protocol ensures all candidate answers are generated before decision-making. Results show ABD method has significant anchor advantage on the protected subset.
Results
On LiveCodeBench-v6, the ABD method achieved 59.43% accuracy, compared to 52.57% for Single9 and 52.00% for HAC. On the untouched GPQA-Diamond split, ABD reached 75.00% accuracy, with controls at 72.78%.
Applications
ABD method can be used to improve language model accuracy, especially in coordinating heterogeneous models. It offers an auditable decision framework suitable for tasks requiring high accuracy.
Limitations & Outlook
ABD method may increase computational cost in some scenarios, especially under high coverage. Additionally, verification structure's role is limited in code generation tasks.
Plain Language Accessible to non-experts
Imagine a large library with many books. Every time you need to find a book, three librarians help you. If they all find the same book, you know it's the right one. If they find different books, a more experienced librarian decides which book is better. The ABD method is like this library management system, ensuring you always find the best book.
ELI14 Explained like you're 14
Imagine you're playing a game with many levels. Every time you pass a level, three friends help you choose the best route. If they all choose the same path, you know it's the right one. If they choose different paths, a better player decides which path is best. The ABD method is like this game strategy, ensuring you win every match!
Glossary
Agreement-Before-Diversity (ABD)
A method for coordinating heterogeneous language models, ensuring anchor answer retention with two additional trusted samples.
Used to decide whether to allow replacement of existing answers.
Heterogeneous synthesis
Combining answers generated by different models to improve accuracy.
Used to replace unsupported anchor answers.
Anchor answer
The initial generated answer retained under verification structure support.
Serves as the benchmark for candidate answers.
Verification structure
Label-free event used to decide whether to allow replacement of existing answers.
Foundation for replacement authority.
Coverage
Proportion of answers supported by verification structure.
Used to measure the effectiveness of verification structure.
Open Questions Unanswered questions from this research
- 1 How to optimize verification structure to enhance coverage?
- 2 What is the universality of ABD method across different tasks?
- 3 How to reduce computational cost under high coverage scenarios?
Applications
Immediate Applications
Language Model Optimization
ABD method can be used to improve language model accuracy, especially in coordinating heterogeneous models.
Long-term Vision
Intelligent Decision Systems
ABD method can be used to develop intelligent decision systems, providing an auditable decision framework.
Abstract
Heterogeneous language-model ensembles expand the space of candidate responses, yet they lack a principled criterion for when a newly generated answer should supersede an already supported one. We decouple candidate headroom from replacement authority, rendering the latter as an explicit, auditable object. Our proposed method, Agreement-Before-Diversity (ABD), is a frozen, label-free decision rule: an anchor answer is retained if two additional trusted samples corroborate it under a fixed equivalence relation; otherwise, it is replaced by a heterogeneous synthesis. For this gating mechanism, we prove two exact identities. The first shows that the accuracy gap relative to unconditional synthesis is determined jointly by the agreement coverage and the anchor's advantage on the protected subset. The second shows that the gap relative to never synthesizing reflects a contrast between authorized recovery and authorized destruction. Neither identity assumes independence or calibrated confidence, and the expected inference cost is approximately eight minus five times the coverage in number of calls. Under blind, exact-ID evaluation, ABD achieves 59.43% on the complete LiveCodeBench-v6 (vs. 52.57% for Single9 and 52.00% for HAC; n = 175) and 75.00% on an untouched GPQA-Diamond split (both controls at 72.78%; n = 180). Furthermore, these identities localize every aggregate difference to an enumerable protected stratum: no discordant items occur among the 3 protected cases on LiveCodeBench, where coverage bounds the gate's contribution to 1.71 points a priori; 13 versus 8 discordant cases among 132 on GPQA-Diamond; and 12 versus 0 among 71 under a frozen anchor perturbation. Diversity supplies potential; verification structure supplies authority.