Argument Collapse: LLMs Flatten Long-Form Public Debate
LLMs generate debate essays with only 3.4% unique main arguments, far below humans' 65.3%.
Key Findings
Methodology
The study compares 1,039 human essays from 195 NYT debates with 23,381 essays generated by five frontier LLMs, analyzing convergence in main arguments, sub-arguments, and paragraph structures. Experiments include vanilla, diversified, and position-guided generation setups.
Key Results
- Result 1: In the NYT dataset, humans' main arguments are 65.3% unique, while LLMs' are only 3.4%.
- Result 2: Diversified generation recovers about 50-55% of unique human arguments but fails to cover all human argument clusters.
- Result 3: Sub-arguments are 41.0% unique for humans but only 9.1% for LLMs, which favor generalized and hedged reasoning.
Significance
The study highlights the risk of reduced diversity in public debates due to LLM-generated content, which may narrow perspectives and influence societal decision-making. This has implications for designing generative AI systems.
Technical Contribution
Introduces the concept of 'argument collapse' and provides a quantitative framework to analyze diversity in LLM-generated content. Offers insights into how generative AI impacts public discourse.
Novelty
First systematic study of argument collapse in LLM-generated public debates, proposing diversified and position-guided generation as potential solutions.
Limitations
- Limitation 1: The study uses only two datasets (NYT and BR), which may not represent all debate scenarios.
- Limitation 2: It does not analyze the impact of training data on generated content.
- Limitation 3: Long-term effects on reader perspectives are not explored.
Future Work
Future research could explore more debate scenarios and languages, optimize models to reduce argument collapse, and study the long-term societal impact of generated content.
AI Executive Summary
As large language models (LLMs) are increasingly used to draft public-facing essays, concerns arise about their impact on diversity in public debates. This study introduces the concept of 'argument collapse,' where LLM-generated essays converge on a narrow range of main arguments, sub-arguments, and structures.
By comparing 1,039 human essays from NYT debates with 23,381 essays generated by five LLMs, the study finds that humans' main arguments are 65.3% unique, while LLMs' are only 3.4%. Even with diversified prompts, LLMs recover only 50-55% of unique human arguments. Sub-arguments generated by LLMs are more generalized and hedged, contrasting with humans' concrete and topic-specific reasoning.
This research underscores the need to optimize LLMs for greater diversity in generated content, especially in public debates, policymaking, and societal decision-making. Future work should explore broader scenarios, refine models, and assess the long-term impact of generative AI on public discourse.
Deep Analysis
Background
LLMs are widely used to generate public debate essays, but concerns about reduced diversity in their output persist. Previous studies focused on quality and acceptability but rarely compared human and LLM-generated content diversity directly.
Core Problem
The core issue is whether LLM-generated essays exhibit less diversity in main arguments, sub-arguments, and structures compared to human essays. This 'argument collapse' could narrow public discourse and affect societal decisions.
Innovation
The study introduces 'argument collapse' and designs three generation setups—vanilla, diversified, and position-guided—to systematically analyze diversity in LLM-generated content.
Methodology
- �� Collected NYT and BR debate datasets with 1,039 and 448 human essays, respectively.
- �� Generated 23,381 essays using five frontier LLMs (GPT, Claude, Gemini, DeepSeek, Minimax).
- �� Compared uniqueness in main arguments, sub-arguments, and structures.
- �� Tested diversified and position-guided generation setups to enhance diversity.
Experiments
Experiments used NYT and BR datasets to test vanilla, diversified, and position-guided generation. Diversity in main and sub-arguments was analyzed by comparing human and LLM-generated content.
Results
Results show that under vanilla settings, LLMs' main arguments are only 3.4% unique compared to humans' 65.3%. Diversified generation recovers 50-55% of unique human arguments. Sub-arguments are 41.0% unique for humans but only 9.1% for LLMs.
Applications
The findings are applicable to optimizing LLMs for diversity in public debates, policymaking, and opinion formation.
Limitations & Outlook
The study is limited to two datasets and does not analyze training data effects or long-term societal impacts.
Plain Language Accessible to non-experts
Imagine you're in a debate club discussing 'Are Americans too obsessed with cleanliness?' Human debaters bring up unique points like cultural factors or advertising's role. But if a robot writes, it might repeat 'Maintain hygiene but avoid extremes.' It's like a machine that writes smoothly but sticks to a few fixed ideas, lacking creativity.
ELI14 Explained like you're 14
Hey, imagine you're chatting with friends about 'Are Americans too clean?' Humans might say cool stuff like 'Ads make us feel dirty is uncool!' But if a robot writes, it keeps saying 'Stay clean but not too much.' Robots are smooth talkers but kinda boring, repeating the same stuff. What do you think? Fun or meh?
Glossary
LLM
A deep learning model for generating text content.
Used to generate debate essays in this study.
Argument Collapse
The tendency of generated content to converge on similar arguments.
Describes LLMs' lack of diversity in public debates.
Diversified Generation
A method to increase content diversity through optimized prompts.
One of the generation setups tested in the study.
Sub-Arguments
Supporting claims or evidence for a main argument.
Analyzed to measure content diversity.
Position-Guided Generation
Generation based on human authors' perspectives and styles.
A tested setup to enhance diversity.
Open Questions Unanswered questions from this research
- 1 How can LLMs be optimized to reduce argument collapse?
- 2 What are the long-term effects of generated content on public opinion?
Applications
Immediate Applications
Debate Assistance Tools
Help users generate diverse essays to improve discussion quality.
Policy Suggestion Generation
Provide policymakers with diverse suggestions to avoid narrow perspectives.
Long-term Vision
Optimizing Public Discourse
Enhance diversity in public debates to improve societal decision-making.
Abstract
As LLMs are increasingly used to draft publicfacing arguments, they may flatten public debate by repeatedly introducing the same polished, plausible arguments. We study argument collapse, the tendency of essays generated by different LLMs to converge to a smaller set of main arguments, sub-arguments, and paragraph-level structures. We compare 1,039 human responses from 195 New York Times (NYT) debates, 448 human responses from 61 longer-form Boston Review (BR) forums, and 23,381 LLM-generated essays. In the NYT corpus, 65.3% of human main arguments are unique within a debate, compared to 3.4% of LLM main arguments. Asking LLMs to generate diverse answers adds variation, but a typical model recovers only about half of the distinct human main arguments, with much of the added variation falling outside the observed human argument space. Collapse also appears in sub-arguments, where among essays with the same main argument, 41.0% of human subarguments are unique versus 9.1% from LLM responses. Qualitatively, LLMs often reuse generalized and hedged sub-arguments, while humans prefer more concrete and topic-specific ones. Structure-wise, LLM-generated essays tend to follow a more fixed arc, often opening with a direct claim and moving quickly toward proposals. The same patterns hold in longer BR essays, suggesting that argument collapse extends beyond short-form responses. Finally, human-preference evaluators favor both common arguments and LLM essays, which could reinforce argument collapse.