Do LLM Debates Repeat Arguments Differently Across Languages?
Study finds Chinese debates show higher semantic repetition than English, introducing 'Prior-Argument Similarity' metric.
Key Findings
Methodology
Introduced 'Prior-Argument Similarity' metric, leveraging multilingual embedding models (e.g., BGE-M3, LaBSE) to quantify semantic overlap in debates. Experiments span 71 motions, six languages, and four models.
Key Results
- Chinese debates show +0.047 higher prior-argument similarity than English (BGE-M3), significantly above other languages.
- Multilingual embedding models consistently highlight higher repetition in Chinese, validated by manual calibration.
- Diversity prompts reduced repetition but failed to significantly narrow the Chinese-English gap.
Significance
This study reveals semantic repetition dynamics in multilingual LLM debates, particularly the distinct behavior in Chinese. It provides critical insights for designing multilingual interaction systems and highlights cross-language semantic differences.
Technical Contribution
Introduced 'Prior-Argument Similarity' as a scalable diagnostic for debate dynamics, validated through multilingual embeddings and manual calibration.
Novelty
First systematic analysis of semantic repetition in multilingual LLM debates, proposing new metrics and experimental frameworks.
Limitations
- Metric cannot distinguish meaningful repetition (e.g., quoting or rebuttal).
- Experiments limited to six high-resource languages, excluding low-resource ones.
- Model selection may affect generalizability.
Future Work
Future studies could expand to more languages, explore repetition's impact on debate quality, and refine prompt strategies for semantic development.
AI Executive Summary
This study investigates semantic repetition in multilingual LLM debates, introducing the 'Prior-Argument Similarity' metric to quantify debate dynamics. Experiments across 71 motions, six languages, and four models reveal that Chinese debates exhibit significantly higher semantic repetition compared to English. This finding is consistent across embedding models and manual calibration.
The study shows that Chinese debates tend to revisit prior arguments rather than develop new ones. This insight is crucial for designing multilingual interaction systems, emphasizing the need to account for cross-language semantic dynamics.
While diversity prompts reduced repetition, they failed to significantly narrow the Chinese-English gap. Future work could explore more languages, assess repetition's impact on debate quality, and optimize prompts for fostering semantic progression.
Deep Analysis
Background
LLM debates have gained attention as a multi-agent interaction paradigm capable of surfacing diverse perspectives and improving reasoning. However, existing research focuses on final answers, neglecting semantic dynamics during debates. Multilingual settings further complicate this issue, as language-specific traits may amplify semantic repetition.
Core Problem
The core problem is whether later debate turns develop new arguments or merely rephrase prior ones. Semantic repetition can hinder debate diversity and quality, especially in multilingual interactions where language-specific traits may exacerbate the issue.
Innovation
Introduced 'Prior-Argument Similarity' metric to quantify semantic overlap in debates. Experiments span six languages and four models, ensuring fair language comparisons. Validated the metric through manual calibration and diversity prompts to test its applicability.
Methodology
- �� Used embedding models like BGE-M3 and LaBSE to compute semantic similarity.
- �� Extracted argument units from each debate turn and compared them to prior arguments.
- �� Validated the metric through manual calibration and diversity prompts.
- �� Applied regression analysis and cross-encoder rescoring for robustness.
Experiments
Experiments covered 71 motions, six languages (including Chinese and English), and four models (e.g., GPT-5.5, Claude Opus 4.7). Each debate consisted of eight turns with strict control over argument length and structure. Diversity prompts tested intervention effects on repetition.
Results
Chinese debates showed significantly higher prior-argument similarity than English (+0.047, BGE-M3). Multilingual embedding models consistently highlighted higher repetition in Chinese, validated by manual calibration. Diversity prompts reduced repetition but failed to narrow the Chinese-English gap.
Applications
Findings can optimize multilingual interaction systems, particularly for debate and collaboration tasks. They highlight semantic dynamics across languages, guiding adaptive prompt strategies.
Limitations & Outlook
Metric cannot distinguish types of repetition, such as quoting or rebuttal. Experiments are limited to six high-resource languages, excluding low-resource ones. Model selection may affect generalizability.
Plain Language Accessible to non-experts
Imagine debates as a cooking competition. Each turn is like a chef preparing a dish. Some chefs reuse old recipes (semantic repetition), while others create new ones (semantic development). The study finds Chinese chefs tend to reuse recipes more often than English ones, possibly due to cultural or linguistic traits.
ELI14 Explained like you're 14
Think of debating like a game with friends. Each turn, you need to come up with new ideas, but sometimes you repeat what was said before. The study found that Chinese debates repeat ideas more often, while English debates try new ones. It's like some players using the same moves, while others keep experimenting with new strategies!
Glossary
Prior-Argument Similarity
Measures semantic overlap between current and prior arguments in debates.
Used to analyze semantic repetition dynamics.
Embedding Model
Transforms text into vectors for semantic similarity computation.
Models like BGE-M3 and LaBSE were used in the study.
Diversity Prompt
A strategy encouraging new arguments instead of repetition.
Tested for its impact on semantic repetition.
Semantic Repetition
Occurs when later arguments semantically revisit prior ones.
The core focus of this study.
Manual Calibration
Human annotation to validate automated metrics.
Used to assess the reliability of 'Prior-Argument Similarity'.
Open Questions Unanswered questions from this research
- 1 How to distinguish meaningful repetition, e.g., quoting vs rebuttal?
- 2 Do low-resource languages exhibit similar semantic repetition patterns?
Applications
Immediate Applications
Multilingual Debate Optimization
Refine prompt strategies to reduce semantic repetition and enhance cross-language interaction.
Educational Use
Train debate skills in language learning, emphasizing argument diversity.
Long-term Vision
Cross-Cultural Collaboration
Develop adaptive multilingual systems to foster global collaboration.
Abstract
LLM debate is usually evaluated by final answers, yet transcripts reveal whether later turns develop new arguments or return to earlier claims in new wording. We study this process with \textit{prior-argument similarity}, which compares extracted argument units with earlier units in the same debate. In controlled eight-turn debates over 71 motions, six languages, and four model agents, Chinese is the only tested language with a consistently positive gap relative to English across three multilingual embedding models. The gap persists across agents, turn positions, regression adjustment, metric variants, extraction-length controls, a second-extractor subset, and cross-encoder tail rescoring. Manual calibration shows weak item-level alignment but a high-similarity tail enriched for substantive repetition. A diversity-aware prompt lowers \textit{prior-argument similarity} across languages, yet does not significantly narrow the Chinese--English gap. Multilingual debate evaluation should therefore measure argumentative development over time and report both average and gap terms.