When Outputs Disperse, Does Epistemic Revision Follow? A Black-Box Diagnostic for Machine Collectives
Proposes black-box dispersion–revision diagnostic using CI to verify if output diversity correlates with genuine epistemic revision.
Key Findings
Methodology
This paper develops a black-box diagnostic framework combining the Coherence Index (CI), derived from an external sentence encoder, with the Meta-Predictive Clarity System (MPCS) and Re-Differentiation Protocol (RDP). The approach operates solely on generated text, measuring whether interventions that increase output dispersion in embedding space lead to true epistemic revision. Using false-premise injection tasks, five-agent collectives based on GPT-4o-mini and Gemini-2.5-flash are evaluated. The framework assesses output dispersion changes via CI and independently measures stance shifts through per-turn annotations, identifying the presence of strong or weak coupling between output diversity and epistemic change. Experiments demonstrate that in GPT-4o-mini, conditional dissent significantly improves false-premise correction (+17.7 percentage points, p<1e-6), whereas static persona diversity hampers it. Conversely, in Gemini-2.5-flash, dispersion decreases without improving correction, highlighting configuration dependence of the coupling phenomenon.
Key Results
- In GPT-4o-mini, conditional dissent increased false-premise correction from 26.1% to 43.8%, a 17.7-point improvement (p<1e-6). Static diversity reduced correction by 8.1 points (p=0.007).
- In Gemini-2.5-flash, despite a verified drop in output dispersion (CI decrease of ~15%), correction rate remained unchanged (26.1% vs 27.1%, p=0.84), indicating weaker coupling.
- Mechanism tagging revealed 94% of Gemini responses post-RDP reformulated the false premise without true revision, compared to 24% on GPT, showing internal mechanism differences influence coupling.
Significance
This work advances the evaluation of collective intelligence by providing a quantitative, behavior-based diagnostic that distinguishes between superficial output diversity and genuine epistemic revision. It addresses a critical challenge in multi-agent LLM systems—ensuring that diversity translates into meaningful error correction—thus improving reliability and interpretability. The framework's ability to detect configuration-dependent differences offers valuable insights for designing more robust models. Its practical utility extends to AI safety, factuality, and collaborative reasoning, marking a significant step toward trustworthy AI systems.
Technical Contribution
The paper introduces a novel black-box diagnostic combining the external Coherence Index (CI) with the MPCS/RDP intervention protocol to measure the dispersion–revision relationship. This approach requires no internal model access, relying solely on generated text and external encodings. It operationalizes the concept of strong versus weak coupling through independent stance annotations and premise-preservation metrics, providing a standardized, scalable method for evaluating epistemic dynamics in multi-agent systems. The framework's modularity allows adaptation to various interventions and models, facilitating broad applicability.
Novelty
This study is the first to operationalize the dispersion–revision coupling in multi-agent LLMs using a black-box, output-only approach. It uniquely combines external embedding-based dispersion metrics with intervention protocols to quantify the behavioral link between output diversity and epistemic change. Unlike prior work focusing solely on surface metrics or internal representations, this method emphasizes observable behavior, enabling practical evaluation without model internals. Its configuration-dependent findings reveal nuanced dynamics previously unquantified, marking a significant innovation in collective intelligence assessment.
Limitations
- The reliance on external sentence encoders may introduce bias or sensitivity to linguistic features, affecting dispersion measurements.
- The experimental validation is limited to two model configurations, requiring broader testing across architectures and tasks for generalization.
- The causal pathway linking CI drops to actual epistemic revision remains indirect; internal mechanisms are not directly analyzed, necessitating future white-box studies.
Future Work
Future research will integrate internal activation and representational analyses to elucidate the mechanisms underlying observed behavioral coupling. Expanding experiments to diverse models and more complex tasks will validate the framework's robustness and universality. Additionally, developing more sophisticated dispersion metrics and exploring different intervention strategies could enhance sensitivity and applicability, ultimately advancing trustworthy multi-agent AI systems.
AI Executive Summary
This study introduces a novel black-box diagnostic framework for evaluating whether output diversity in multi-agent large language models (LLMs) correlates with genuine epistemic revision. Traditional metrics often conflate superficial output variation with meaningful cognitive change, risking overestimating a model’s correction capabilities. To address this, the authors develop the Coherence Index (CI), an external embedding-based measure of output dispersion, combined with the Meta-Predictive Clarity System (MPCS) and Re-Differentiation Protocol (RDP). The framework operates solely on generated text, making it broadly applicable without requiring internal model access.
In the experimental setup, five-agent collectives based on GPT-4o-mini and Gemini-2.5-flash are evaluated on false-premise injection tasks across multiple domains. The intervention involves triggering dissent via RDP when outputs over-converge, verified by a drop in CI. The core question is whether this dispersion increase leads to true epistemic revision, measured through independent stance annotations. Results show that in GPT-4o-mini, conditional dissent significantly improves false-premise correction, with a 17.7-point increase in success rate, demonstrating strong coupling. Conversely, in Gemini-2.5-flash, despite verified dispersion reduction, correction rates do not improve, indicating weak coupling.
Mechanism tagging analysis reveals that Gemini models tend to preserve false premises internally, reformulating responses without genuine revision, whereas GPT models more often revise the premise. These findings highlight the importance of behavioral coupling in collective intelligence and suggest that output diversity alone is insufficient for reliable error correction.
The proposed diagnostic offers a practical, scalable tool for evaluating epistemic dynamics in multi-agent systems, with broad implications for AI safety, factuality, and collaborative reasoning. Future work will focus on integrating internal representational analysis and extending the framework to more diverse models and complex tasks, aiming to deepen understanding of the cognitive mechanisms underlying collective intelligence.
Deep Analysis
Background
集体智能研究起源于群体决策的优势，强调多样性带来的错误互补效应。随着大规模语言模型（LLM）的崛起，研究逐渐关注多智能体协作中的行为表现与认知机制。早期工作如Liu等（2022）提出多智能体合作能提升事实性，Shin等（2023）关注角色扮演中的一致性，但普遍发现输出多样性未必对应认知修正。传统评估多依赖表面指标（如BLEU、准确率），忽略行为背后的认知动态。本文在此背景下，提出输出散布与认知修正的耦合关系，为理解多智能体协作提供新视角。
Core Problem
现有方法难以区分输出多样性是否反映真实认知变化。多智能体模型可能在表面表现出多样性（如不同措辞、角色），但内部认知未发生实质性修正。这导致评估结果偏向乐观，无法准确反映模型的认知能力。核心问题在于如何在不访问模型内部激活的情况下，验证输出变化是否伴随认知修正，从而提升多智能体系统的可信度与实用性。
Innovation
本研究创新点在于提出基于外部句子编码器的散布指标（CI）结合干预协议（MPCS/RDP），实现行为层面上的认知修正验证。区别于传统只关注输出多样性的指标，强调行为背后的认知变化。首次在多智能体LLM中引入黑箱诊断，结合立场标注，区分表面变化与深层认知修正。该方法简洁高效，适用范围广，突破了模型内部机制不可见的限制，为多智能体认知评估提供新工具。
Methodology
- �� 设计虚假前提注入任务，模拟虚假信息的表达与纠正。• 利用外部句子编码器（text-embedding-3-small）计算每轮回应的语义散布（CI），衡量输出的语义一致性。• 采用Meta-Predictive Clarity System（MPCS）监测输出的收敛程度，自动在过度收敛时插入Re-Differentiation Protocol（RDP），促使模型识别并指出潜在缺陷。• 通过人工标注每轮回应的立场（-3到+3），分析是否实现认知修正。• 比较干预前后CI指标变化和立场变化，验证散布变化是否伴随认知修正，定义强耦合与弱耦合状态。
Experiments
- �� 采用31个跨领域模板（如生物、历史、物理等）设计虚假前提，模拟多轮对话。• 选用gpt-4o-mini和gemini-2.5-flash两种模型配置，设置温度0.7，最大轮次10，注入真相在第4轮。• 施加不同干预（条件性异议、静态人格多样性），测量输出散布（CI）变化与修正率。• 通过人工立场标注，评估认知修正的真实性。• 统计不同模型配置的效果差异，验证耦合依赖性。
Results
- �� 在gpt-4o-mini中，条件性异议将虚假前提修正率从26.1%提升至43.8%，提升17.7点（p<1e-6），显示散布变化伴随修正。• 静态人格多样性导致修正率下降（-8.1点，p=0.007），表明持续多样性可能阻碍修正。• 在gemini-2.5-flash中，散布下降（CI减少约15%）未改善修正率（26.1% vs 27.1%，p=0.84），显示模型配置影响耦合强度。• 机制标记显示gemini模型中94%的响应通过内部机制保持虚假前提，未实现修正，而GPT仅24%的响应如此，反映模型机制差异。
Applications
- �� 该诊断工具适用于评估多智能体系统在事实校正、辩论和合作任务中的认知动态，有助于设计更具鲁棒性的集体决策模型。• 在实际中，可作为模型调优和策略选择的指标，确保输出多样性真正伴随认知修正，提升系统可信度。• 长远来看，该方法推动多智能体系统向更具解释性和鲁棒性的方向发展，应用于自动决策、知识管理等领域。
Limitations & Outlook
- �� 目前仅在两种模型配置下验证，泛化到更复杂或不同架构模型仍需验证。• 依赖外部句子编码器，可能受编码偏差影响，未能深入模型内部机制。• 仅在虚假前提任务中测试，实际场景中复杂性更高，效果需进一步验证。
Abstract
Collective intelligence research treats disagreement as evidence of epistemic diversity: if agents express different views, the group should retain capacity to revise. In LLM collectives this proxy can break: agents can produce diverse-looking arguments while preserving the same conclusion. We operationalize dispersion-revision coupling: the degree to which an intervention that verifiably increases the dispersion of a collective's outputs in embedding space is accompanied by genuine revision of its epistemic stance rather than premise-preserving reformulation. The diagnostic is black-box: it operates on generated text alone and makes no claims about the internal representations of the generating models. Two channels are measured independently: an output channel, the Coherence Index (CI), verifies that the intervention changed output dispersion; an epistemic channel, per-turn stance annotation, measures whether the collective revised. We propose CI with the Meta-Predictive Clarity System (MPCS), which inserts a Re-Differentiation Protocol (RDP) when outputs over-converge, as a reusable method for estimating this coupling regime. We evaluate five-agent collectives from two configurations (gpt-4o-mini and gemini-2.5-flash; 310 paired episodes per condition). On gpt-4o-mini, conditional dissent improves false-premise recovery by +17.7 points (p<1e-6) while static persona diversity harms recovery (-8.1, p=.007). On gemini-2.5-flash, the same intervention at a comparable budget yields no gain (26.1% vs 27.1%, p=.84) despite a verified dispersion drop; the two treatment effects differ from each other (z=3.79, p<.001). Mechanism tagging shows Gemini preserves the false premise via intra-framework dissent: 94% of tagged post-RDP responses reformulate rather than concede (vs 24% on GPT). We recommend reporting per-intervention stance shift and premise-preservation rate alongside accuracy.