How Much Does Persuasion Strategy Matter? LLM-Annotated Evidence from Charitable Donation Dialogues
Using Qwen3:30b and others to annotate PersuasionForGood, guilt induction reduces donation rates by 23 percentage points.
Key Findings
Methodology
The study employed Qwen3:30b, Mistral-Small-3.2, and Phi-4 to annotate 10,600 persuader turns with 41 fine-grained strategies. Statistical tests with corrections were applied to analyze strategy-donation associations, alongside sentiment and interest annotations for targets.
Key Results
- Result 1: Guilt induction significantly reduces donation rates (32.7% vs. 56.0%, ∆ = -23 pp, p<0.001). Donation amounts are also 3.5× lower ($0.67 vs. $2.34).
- Result 2: Reciprocity increases donation rates (72.2% vs. 52.2%, ∆ = +20.1 pp, p=0.034). Positive sentiment responses mediate this effect.
- Result 3: Target sentiment and interest strongly predict donation occurrence (sentiment mean: 0.44 vs. 0.27, p<0.001), but not donation amount.
Significance
This study highlights the limited predictive power of strategy identification alone and reveals the counterproductive effect of guilt induction in prosocial contexts. The findings can inform the design of effective dialogue systems and evidence-based training for fundraisers.
Technical Contribution
Contributions include: introducing a 41-strategy taxonomy for persuasion; leveraging three open-source LLMs for robust annotation; releasing a fully annotated corpus for future research.
Novelty
This is the first study to comprehensively analyze strategy-donation associations across a full corpus using fine-grained taxonomy and multi-model annotation, uncovering the negative impact of guilt induction.
Limitations
- Limitation 1: Moderate inter-model agreement on fine-grained labels (κ=0.38-0.54) may introduce annotation noise.
- Limitation 2: Sentiment and interest are dynamic variables measured during conversations, limiting causal inference.
- Limitation 3: Single-label annotation may miss co-occurring strategies within turns.
Future Work
Future work could explore strategy sequences and combinations, conduct causal mediation analysis, and improve LLM annotation consistency.
AI Executive Summary
Persuasion strategies in charitable donation dialogues have long been studied, but existing research often relies on small datasets or coarse taxonomies, leaving key questions unanswered.
This study uses three open-source LLMs (Qwen3:30b, Mistral-Small-3.2, Phi-4) to annotate 10,600 persuader turns in the PersuasionForGood corpus with a 41-strategy taxonomy. Statistical tests reveal that guilt induction significantly reduces donation rates (-23 pp), while reciprocity increases them (+20.1 pp). Target sentiment and interest emerge as the strongest predictors of donation occurrence.
These findings challenge the assumption that strategy identification alone explains persuasion effectiveness and highlight the counterproductive nature of guilt-based appeals. The annotated dataset is released as a resource for academia and industry, paving the way for more effective prosocial dialogue systems and fundraising practices.
Deep Analysis
Background
Persuasion strategy research is crucial in NLP and psychology. Early work, such as Wang et al. (2019), introduced a 10-strategy taxonomy but lacked comprehensive analysis. Later efforts improved classifiers (e.g., Saha et al., 2021) but remained limited to small samples.
Core Problem
The core question is: which persuasion strategies are statistically associated with donation outcomes? Existing studies fail to analyze large-scale corpora with fine-grained strategies or apply robust statistical corrections.
Innovation
Innovations include: 1) defining a 41-strategy taxonomy; 2) using three open-source LLMs for robust annotation; 3) conducting multivariate analysis with corrections across the full corpus.
Methodology
- �� Qwen3:30b served as the primary annotator, with Mistral-Small-3.2 and Phi-4 for robustness checks.
- �� Two-step annotation: first select category, then specific strategy.
- �� Statistical tests (chi-square, logistic regression) assessed strategy-donation associations.
- �� Sentiment and interest annotations captured target responses as covariates.
Experiments
Experiments used the PersuasionForGood dataset (1,017 dialogues, 10,600 persuader turns). Baselines included dialogues without strategies. Metrics: donation rates and amounts. Multiple comparison corrections applied to 41 strategies.
Results
Guilt induction reduces donation rates (-23 pp), reciprocity increases them (+20.1 pp). Sentiment and interest strongly predict donation occurrence but not amount.
Applications
Findings can guide prosocial dialogue system design, optimize fundraising strategies, and support psychological and sociological research.
Limitations & Outlook
Moderate inter-model agreement introduces annotation noise. Dynamic variables limit causal inference. Single-label annotation may miss co-occurring strategies.
Plain Language Accessible to non-experts
Imagine you're organizing a school fundraiser. You ask classmates to donate, and you notice some approaches work better. Saying “I'll match your donation” (reciprocity) gets good results. But saying “If you don't donate, kids will starve” (guilt induction) makes people uncomfortable. This study analyzed thousands of conversations to find which strategies help and which hurt.
ELI14 Explained like you're 14
Hey, imagine you're playing a game and need teammates to donate coins for a mission. If you say “I'll donate some too,” they’re more likely to help (reciprocity). But if you say “If you don’t donate, we’ll fail,” they might get annoyed (guilt induction). This study used AI to analyze tons of chats to figure out what works best!
Glossary
Guilt Induction
A strategy that uses moral obligation or guilt to prompt action.
Found to reduce donation rates in dialogues.
Reciprocity
A strategy offering mutual exchange to encourage action.
Significantly increases donation rates in dialogues.
Target Sentiment
The emotional tone (positive, neutral, negative) expressed by the target.
Strongly predicts donation occurrence.
PersuasionForGood Corpus
A dataset of 1,017 charitable donation dialogues.
Used to analyze strategy-donation associations.
Multiple Comparison Correction
A statistical method to reduce false positives in multiple tests.
Applied in strategy-donation analysis.
Open Questions Unanswered questions from this research
- 1 The impact of strategy combinations on donation outcomes remains unclear.
- 2 Causal links between strategies and target sentiment need further study.
- 3 Improving LLM annotation consistency is an open challenge.
Applications
Immediate Applications
Fundraising Optimization
Provide evidence-based strategy recommendations to avoid guilt induction.
Prosocial Dialogue Systems
Design AI systems to promote charitable donations or other prosocial behaviors.
Long-term Vision
Psychology and Sociology Research
Enable large-scale studies of human persuasion behavior with annotated data.
Abstract
Which persuasion strategies, if any, are associated with donation compliance? Answering this requires fine-grained strategy labels across a full corpus and statistical tests corrected for multiple comparisons. We annotate all 10,600 persuader turns in the 1,017-dialogue PersuasionForGood corpus (Wang et al., 2019), where donation outcomes are directly observable, with a taxonomy of 41 strategies in 11 categories, using three open-source large language models (LLMs; Qwen3:30b, Mistral-Small-3.2, Phi-4). Strategy categories alone explain little variance in donation outcome (pseudo $R^2 \approx 0.015$, consistent across all three annotators). Guilt Induction is the only strategy significantly associated with lower donation rates ($Δ\approx -23$ percentage points), an effect that replicates across all three models despite only moderate inter-model agreement. Reciprocity is the most robust positive correlate. Target sentiment and interest predict whether a donation occurs but show at most a weak correlation with donation amount. These findings suggest that strategy identification alone is insufficient to explain persuasion effectiveness, and that guilt-based appeals may be counterproductive in prosocial settings. We release the fully annotated corpus as a public resource.