A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing
Introduced PERSUASIONTRACE, leveraging Bayesian networks to model multi-turn belief updates, scoring near humans (81 vs 80).
Key Findings
Methodology
PERSUASIONTRACE combines multi-turn belief tracking with rhetorical annotations (logos/pathos/ethos) and introduces a Bayesian network simulator to capture human belief dynamics. Experiments were conducted on a web-based platform across text and audio modalities.
Key Results
- Result 1: Bayesian network simulator achieved near-human similarity scores (81 vs 80), significantly outperforming baseline LLM simulators (64).
- Result 2: Human belief updates clustered into low-shift (mean 0.039) and high-shift (mean 0.437) groups, with the latter more sensitive to emotional rhetoric.
- Result 3: LLMs demonstrated strong persuasion across personalized topics and audio modalities, significantly outperforming control groups.
Significance
This study pioneers process-level analysis of human belief dynamics, moving beyond endpoint-only measurements. It provides precise tools for understanding persuasion mechanisms in human-LLM interactions and lays the groundwork for safer persuasive systems.
Technical Contribution
Contributions include: 1) a multi-turn belief tracking framework; 2) a Bayesian network-based cognitive simulator; 3) quantitative tools for rhetorical analysis; 4) evidence that simulator choice significantly impacts persuasion quality assessment.
Novelty
First to apply Bayesian networks for multi-turn belief dynamics modeling, combined with rhetorical analysis, significantly improving alignment with human behavior.
Limitations
- Limitation 1: Bayesian network structures rely on predefined belief graphs, limiting generalization to complex belief systems.
- Limitation 2: Experiments were conducted in English, leaving cross-linguistic applicability untested.
- Limitation 3: Simulator robustness to extreme rhetorical strategies remains underexplored.
Future Work
Future directions include: 1) extending to multilingual and cross-cultural settings; 2) optimizing Bayesian network structures for complex belief systems; 3) integrating with other cognitive models.
AI Executive Summary
PERSUASIONTRACE introduces a novel framework for studying human-LLM persuasion dynamics by tracking multi-turn belief updates and annotating rhetorical strategies. The study reveals that LLMs are highly persuasive across various topics and modalities, while a Bayesian network simulator closely mimics human belief dynamics (81 vs 80).
Traditional persuasion studies rely on pre/post belief change, missing dynamic shifts during dialogue. PERSUASIONTRACE captures these dynamics, uncovering heterogeneity in human belief updates and sensitivity to rhetorical styles like logos, pathos, and ethos.
This work not only advances the scientific understanding of persuasion but also provides a foundation for designing safer, more effective persuasive systems. Future research will explore cross-cultural applicability and more complex belief systems.
Deep Analysis
Background
Persuasion research is critical in domains like politics, education, and business. While LLMs have shown strong persuasive abilities, existing studies focus on endpoint belief changes, failing to capture dynamic shifts during dialogue.
Core Problem
Traditional methods cannot track belief dynamics within dialogues, limiting understanding of persuasion mechanisms and the evaluation of system safety and effectiveness.
Innovation
Key innovations include: 1) multi-turn belief tracking; 2) rhetorical dimension analysis (logos/pathos/ethos); 3) a Bayesian network simulator that significantly improves alignment with human belief dynamics.
Methodology
- �� Developed a web-based experimental platform for multi-turn belief tracking and rhetorical annotations.
- �� Used gpt-5 as the persuader in interactions with human participants.
- �� Introduced a Bayesian network simulator to dynamically update belief states.
- �� Evaluated human-likeness and persuasion effectiveness using LLM-based metrics.
Experiments
Experiments involved: 1) 255 human-participant rounds; 2) topics from DebateGPT and personalized themes; 3) comparisons between Bayesian network simulators and baseline LLM simulators on human-likeness and belief dynamics.
Results
Results showed: 1) Bayesian network simulator scored near humans (81 vs 80); 2) human belief updates clustered into low-shift and high-shift groups; 3) LLMs outperformed controls in personalized and audio settings.
Applications
Applications include: 1) designing safer persuasive systems; 2) enhancing educational and training outcomes through personalized persuasion; 3) improving virtual assistant interactions.
Limitations & Outlook
Limitations include: 1) limited generalization of Bayesian networks to complex belief systems; 2) untested cross-linguistic applicability; 3) insufficient robustness to extreme rhetorical strategies.
Plain Language Accessible to non-experts
Imagine you're debating with a friend about whether social media makes people less intelligent. Each time your friend makes a point, your opinion shifts slightly. PERSUASIONTRACE acts like a 'belief tracker,' recording these shifts and analyzing whether your friend used logic, emotion, or authority to persuade you. Researchers also built a 'virtual you' using a Bayesian network to predict how your beliefs might change. This helps us understand persuasion better!
ELI14 Explained like you're 14
Imagine chatting with an AI about whether social media makes people dumb. After each message, you rate how much you agree. Researchers used a tool called PERSUASIONTRACE to track these ratings and figure out if the AI used logic, emotions, or authority to convince you. They even made a 'virtual you' with math to predict your next move! Cool, right?
Glossary
PERSUASIONTRACE
A framework for studying persuasion in human-LLM interactions, tracking multi-turn belief updates and rhetorical strategies.
Used for experimental tracking and analysis of belief dynamics.
Bayesian Network
A probabilistic graphical model representing conditional dependencies between variables.
Core tool for modeling human belief dynamics.
Logos
A rhetorical strategy based on logic and facts.
Used to analyze persuader techniques.
Pathos
A rhetorical strategy appealing to emotions.
Analyzed for human sensitivity to emotional persuasion.
Ethos
A rhetorical strategy based on authority or credibility.
Used in rhetorical annotations and simulator design.
Open Questions Unanswered questions from this research
- 1 How can Bayesian networks generalize to more complex belief systems?
- 2 What is the framework's applicability in multilingual and cross-cultural contexts?
- 3 How can simulator robustness to extreme rhetorical strategies be improved?
Applications
Immediate Applications
Education and Training
Enhance learning outcomes through personalized persuasion strategies, helping students better understand and accept new knowledge.
Virtual Assistant Optimization
Improve virtual assistant interactions, making them more persuasive and human-like.
Long-term Vision
Societal Decision Support
Aid in designing more inclusive and transparent decision-making processes in politics and public policy.
Abstract
Large language models can shift human beliefs across high-stakes domains, but most persuasion studies rely on pre/post belief change. These endpoint measures identify whether persuasion occurred, yet miss where and how beliefs moved within a dialogue. We present PERSUASIONTRACE, a framework for studying persuasion in human-LLM interaction. Built on a web-based experimental platform, PERSUASIONTRACE contributes a tool for multi-turn persuasion studies and a process-level evaluation protocol: it records multi-turn belief reports from human or simulated targets of persuasion, annotates persuader turns with rhetorical dimensions (logos/pathos/ethos), and evaluates simulators by fidelity to real human belief dynamics. Using this framework, we find that human targets group into two clusters of multi-turn belief updates and exhibit susceptibility to rhetorical strategies, and that LLMs are persuasive across generic and personalized topics, text and audio modalities, and multi-turn interactions. Prior work has chiefly used vanilla-prompted LLMs to simulate human targets, but we show that these simulators fail to replicate human belief dynamics. We introduce a Bayesian-network simulated target that maintains an explicit latent belief state over time so each persuader message yields cognitively realistic belief updates. In human-likeness evaluation, our Bayesian target scores near a human reference (81 vs 80), while baseline LLM targets score substantially lower (64). PERSUASIONTRACE reframes persuasion evaluation from endpoint movement alone to process fidelity, providing a stronger basis for scientific analysis and safer optimization of persuasive systems.