Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology
Introducing Social Chain of Thought (SCoT), a multi-agent framework that improves differential diagnosis recall by 4-12% through multi-round specialist interactions.
Key Findings
Methodology
SCoT employs a multi-round, persona-conditioned multi-agent architecture that simulates multidisciplinary clinical consultations. It involves generating a panel of five specialists, each with distinct roles, who iteratively propose diagnoses, debate, vote, and perform safety checks. The system leverages diverse perspectives to mitigate endogenous biases inherent in single models, enhancing transparency and recall. The process includes seven structured rounds: specialist generation, relevance assessment, initial differentials, list merging, debate, voting, and critical safety review. Experiments on the Open-XDDx benchmark with models like Qwen-2.5, Gemma, and Claude validate the approach, demonstrating significant improvements especially in complex cases.
Key Results
- Across 570 cases, SCoT improved overall recall by 4-12 percentage points, with the highest gains (up to 15.1%) in challenging cases. Smaller models (e.g., Qwen-2.5-3B) benefited most, with an 11.8-point recall increase. Ablation studies showed that multi-agent collaboration outperforms simple multi-round sampling and single-model approaches, especially in difficult scenarios. The results confirm that structured multi-agent debate enhances diagnostic coverage and robustness.
- Single-agent multi-round inference, without heterogeneity, often reduced recall despite increased precision, indicating the importance of diverse perspectives. Best-of-n sampling failed to match SCoT's gains, emphasizing the value of structured collaboration. The benefits were most pronounced in cases where baseline models performed poorly, boosting F1 scores by 71.4%. These findings highlight the critical role of social interaction in complex diagnostic reasoning.
- The experimental evidence demonstrates that multi-agent, multi-round collaboration effectively addresses biases and enhances recall in complex and edge cases, providing a promising pathway for clinical AI systems that require high diagnostic coverage and interpretability.
Significance
This work advances the field of AI-assisted medical diagnosis by demonstrating that structured multi-agent collaboration significantly outperforms traditional monolithic models, especially in complex cases. It mimics real-world multidisciplinary consultations, offering a transparent, explainable, and scalable approach to improve diagnostic accuracy. The architecture addresses core challenges such as bias, information integration, and interpretability, paving the way for more reliable AI tools in clinical settings. Its ability to recover ground-truth diagnoses in difficult cases underscores its potential to support clinicians in critical decision-making, ultimately enhancing patient safety and healthcare quality.
Technical Contribution
The key technical innovation is the integration of persona-conditioned, multi-round deliberation within a multi-agent framework, combined with voting and safety mechanisms. This approach systematically introduces diversity and heterogeneity, reducing endogenous bias and improving recall. The architecture is modular, open-source, and adaptable across models and datasets, enabling reproducibility and extension. It also provides a formalized process for structured debate, akin to clinical Delphi methods, enhancing interpretability and transparency. The experimental validation across multiple models and ablation studies establishes the robustness and scalability of the approach.
Novelty
This study is the first to embed a sociology-inspired, multi-round, multi-agent deliberative process into differential diagnosis tasks, leveraging persona-conditioned agents to simulate multidisciplinary clinical reasoning. Unlike prior multi-agent systems focused on information aggregation or dialogue, SCoT emphasizes structured debate, voting, and safety checks, which significantly boost recall, especially in complex cases. Its architecture bridges social science principles with cutting-edge NLP, representing a novel paradigm in AI medical reasoning.
Limitations
- The approach depends on the underlying LLM capabilities; biases and knowledge gaps in base models can still influence outcomes. Future work should incorporate real clinical data for fine-tuning.
- Multi-round interactions increase computational costs, posing challenges for real-time deployment in clinical settings. Optimization and efficiency improvements are needed.
- The expert personas are simulated and lack real-world clinical data integration, which limits ecological validity. Further validation with real patient data is essential.
Future Work
Future directions include integrating real clinical datasets for fine-tuning, expanding multi-modal inputs such as imaging and lab results, and developing reinforcement learning strategies to adapt expert debate dynamically. Additionally, efforts will focus on optimizing computational efficiency and deploying the system in real-world clinical workflows to evaluate its practical utility and impact.
AI Executive Summary
The rapid advancement of AI in healthcare has highlighted both opportunities and limitations of current diagnostic models. Traditional monolithic large language models, while powerful, struggle with complex, multi-disciplinary cases due to inherent biases and limited transparency. This gap underscores the need for architectures that can emulate the collaborative nature of clinical diagnosis.
In response, this study introduces Social Chain of Thought (SCoT), a novel multi-agent framework inspired by social science principles. SCoT constructs a panel of specialized, persona-conditioned AI agents that engage in iterative diagnosis, debate, and voting across seven structured rounds. This process mimics multidisciplinary clinical consultations, allowing diverse perspectives to converge on a more accurate and comprehensive differential diagnosis.
Experimental validation on the Open-XDDx benchmark involving 570 cases demonstrates that SCoT consistently outperforms single-model and simple multi-round approaches, with improvements in recall ranging from 4% to over 15% in challenging cases. The architecture shows particular strength in difficult scenarios, where baseline models often fail, recovering ground-truth diagnoses that otherwise would be missed. Notably, smaller models benefit most from this collaborative approach, indicating that social heterogeneity compensates for limited individual reasoning capacity.
These findings have profound implications for AI-assisted diagnostics, suggesting that structured multi-agent collaboration can significantly enhance model robustness, transparency, and clinical utility. While computational costs and ecological validity remain challenges, the modular, open-source nature of SCoT provides a scalable foundation for future research. Moving forward, integrating real clinical data, multi-modal inputs, and adaptive learning strategies will be crucial to translating this promising architecture into real-world clinical practice, ultimately improving patient outcomes and healthcare efficiency.
Deep Dive
Abstract
Medical diagnostic reasoning is a high-impact use case for LLMs that carries significant implications for the health and wellbeing of users. When OpenAI (2026) reports that more than 5% of ChatGPT messages globally are healthcare-related, the transparency of these systems becomes a serious design concern. This is especially true for complex cases, where differential diagnosis often requires integrating multiple forms of specialist reasoning. Existing work has proposed multi-agent approaches to medical diagnosis, but it remains unclear when such systems are needed, why they help, and where they outperform monolithic inference. We introduce Social Chain of Thought (SCoT),a multi-round pipeline for medical differential diagnosis that structures multi-agent interaction as a deliberative framework for collabora. tive LLM reasoning. Evaluating SCoT against single-agent baselines, one-agent pipeline ablations, and best-of-n scaling, we show that its recall advantage is not reproduced by monolithic inference alone. SCoT is most successful in the hardest diagnostic cases, where multiple rounds of specialist conversation help recover ground-truth diagnoses and converge on a higher-recall differential.