DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
DiscoTrace structures answers as discourse acts paired with question interpretations, revealing human-AI strategy differences.
Key Findings
Methodology
DiscoTrace integrates RST parsing to segment answers into discourse units, then employs GPT-4 for labeling acts (e.g., background, clarification). It generates multiple question interpretations via prompting, matching them with answer acts to analyze response strategies across nine Reddit communities. The approach captures community-specific answer preferences and compares them with LLM behaviors, revealing that humans exhibit diverse, deep strategies, whereas LLMs tend to produce shallow, broad coverage answers lacking community nuance.
Key Results
- Analysis shows significant community-driven differences in answer structure and depth, with DiscoTrace effectively capturing these variations.
- LLMs, even when prompted, display limited strategic diversity, favoring broad, surface-level coverage over deep, community-specific responses, with answers about 30% longer than human answers.
- Across domains like economics, history, linguistics, DiscoTrace highlights systematic biases in models’ answer strategies, guiding future improvements.
Significance
This work advances understanding of answer strategy diversity, moving beyond content correctness to structural and pragmatic aspects. It enables nuanced evaluation and development of personalized, strategy-aware AI systems, crucial for complex, ambiguous, or multi-interpretation questions. By revealing how different communities and humans approach answering, DiscoTrace provides a foundation for designing models that adapt their responses to user context, improving relevance and user satisfaction.
Technical Contribution
The paper introduces a novel framework combining RST-based parsing with discourse act classification, creating a structured, interpretable representation of answers. It incorporates multi-interpretation generation and matching, facilitating detailed analysis of answer strategies. The methodology enables systematic comparison between human and machine responses, revealing key biases and limitations in current LLMs, and offers a scalable approach for strategy-aware answer generation.
Novelty
This is the first comprehensive application of RST and discourse act ontology to analyze large-scale answer strategies across communities and models. Unlike prior content-focused evaluations, DiscoTrace emphasizes the structural and pragmatic dimensions of answers, providing a new lens to understand and improve AI answer behavior. Its multi-interpretation pairing mechanism is a key innovation, enabling detailed strategy analysis.
Limitations
- Dependence on GPT-4 for labeling introduces potential biases and limits scalability. The accuracy of discourse act recognition may vary across domains.
- The approach mainly relies on Reddit data, which may not generalize well to other platforms or languages.
- Handling highly ambiguous or complex questions remains challenging; future work should incorporate richer interpretative frameworks and multi-modal data.
Future Work
Future directions include integrating multi-modal cues (images, videos) to refine discourse act detection, expanding to multilingual datasets, and developing adaptive models that dynamically adjust strategies based on user feedback. Further, enhancing interpretative diversity and depth, especially for complex questions, will be prioritized. The goal is to create more flexible, context-aware dialogue systems capable of mimicking human-like strategic answering across diverse scenarios.
AI Executive Summary
DiscoTrace offers a groundbreaking approach to dissecting answer strategies in information-seeking questions. Traditional evaluation methods focus primarily on correctness or relevance, often overlooking how answers are constructed and the underlying pragmatic choices. This research introduces a structured framework that combines rhetorical structure theory (RST) with discourse act classification, enabling detailed analysis of answer responses. By parsing answers into discourse segments and labeling each with specific acts—such as providing background, asserting answers, or seeking clarification—DiscoTrace reveals the strategic behaviors embedded within answers.
Applying this method to answers from nine distinct Reddit communities, the authors uncover significant diversity in human answering strategies, reflecting community-specific objectives and cultural norms. In contrast, large language models like GPT-4, even when prompted to mimic community guidelines, tend to produce answers with limited strategic variation, favoring broad coverage over depth. These findings highlight a systemic bias in current models, which tend to generate lengthier, less nuanced responses lacking community-specific tailoring.
The implications are profound: understanding and modeling answer strategies can lead to more pragmatic, context-sensitive AI systems. Such systems could adapt their answering behavior—being more empathetic, skeptical, or evidence-based—according to user needs and community standards. This work not only advances theoretical insights into discourse and pragmatics but also provides practical tools for developing more human-like, strategy-aware dialogue agents. Future work aims to incorporate multi-modal data, expand across languages, and refine interpretative mechanisms, paving the way for truly adaptive AI conversationalists.
Deep Analysis
Background
The evolution of NLP has shifted from fact-based retrieval to complex, multi-faceted question answering. Early datasets like SQuAD and Natural Questions prioritized answer accuracy, but overlooked the strategic and pragmatic aspects of responses. Recent studies, such as Xu et al. (2022), introduced discourse analysis to understand answer organization, emphasizing community influence and answer diversity. Despite these advances, current models remain limited in their ability to produce strategically varied answers, often defaulting to generic, surface-level responses. This gap underscores the need for tools that can analyze and compare answer strategies at a structural level, capturing community-specific behaviors and deep reasoning patterns. DiscoTrace addresses this by integrating rhetorical structure theory with discourse act classification, providing a comprehensive framework for analyzing answer strategies across diverse contexts.
Core Problem
Existing models tend to generate answers that lack strategic diversity and community-specific tailoring, especially in complex, ambiguous, or multi-interpretation questions. This limits their usefulness in real-world applications where nuanced, context-aware responses are essential. Moreover, there is no systematic way to quantify or compare answer strategies, making it difficult to improve models or understand human-answering behaviors across different communities. The core challenge is to develop a method that can reliably dissect answer structures, identify strategic choices, and compare them across human and machine responses, thereby enabling more adaptive and pragmatic AI systems.
Innovation
DiscoTrace's main innovation lies in its integration of RST parsing with a comprehensive discourse act ontology, enabling structured, interpretable representations of answers. It introduces a multi-interpretation generation mechanism, allowing the pairing of answer strategies with diverse question readings. This approach captures the depth and breadth of human answer strategies, contrasting sharply with the more homogeneous responses of current LLMs. The framework's ability to analyze community-specific preferences and systematically compare human and AI answers marks a significant step forward in pragmatic NLP, offering new avenues for strategy-aware model development.
Methodology
- �� Input: question Q and answer A. • Use RST parser to segment answer into discourse units, identifying hierarchical relations. • Label each segment with one of 21 discourse acts (e.g., provide background, assert answer, seek clarification) via GPT-4. • Generate multiple interpretations of Q using prompts with GPT-4 and external models, forming a set SI. • Match each interpretation with relevant discourse acts, pairing them to reveal answer strategies. • Validate annotations through human annotators, ensuring high agreement. • Output a structured sequence of discourse act-interpretation pairs, representing the answer strategy.
Experiments
The dataset comprises nearly 19,000 answers from nine Reddit communities, covering diverse topics like economics, history, and linguistics. Answers are parsed with RST, labeled with discourse acts, and paired with generated interpretations. Baseline models include GPT-4 and external annotators, with evaluation metrics such as Cohen’s Kappa (0.75 for act recognition) and interpretative agreement (74%). Cross-community analysis reveals distinct strategic patterns, with human answers showing deeper, community-specific behaviors, while models tend to produce lengthier, less nuanced responses. Ablation studies confirm the importance of multi-interpretation pairing and discourse act accuracy in strategy analysis.
Results
DiscoTrace successfully captures community-specific answer styles, revealing that humans employ diverse, deep strategies aligned with community norms. Conversely, GPT-4 answers show limited variation, favoring broad coverage and length. Quantitative analysis indicates that model answers are approximately 30% longer than human answers, with less community tailoring. The framework also uncovers systematic biases in models’ answer strategies across domains, providing a foundation for targeted improvements. These results demonstrate the effectiveness of discourse-based analysis in understanding and enhancing AI answer behaviors.
Applications
DiscoTrace can be employed in developing adaptive, community-aware dialogue systems, improving answer relevance and user satisfaction. It supports personalized AI responses by modeling community-specific strategies, useful in customer service, education, and social media moderation. Additionally, the framework aids researchers in analyzing answer behaviors, guiding the design of more nuanced, pragmatic models that can handle complex, ambiguous questions with depth and context sensitivity.
Limitations & Outlook
The approach relies heavily on GPT-4 for labeling, which may introduce biases and limit scalability. Its effectiveness across languages, platforms, and highly complex questions remains to be validated. The current dataset is predominantly Reddit-based, which may not generalize to other contexts. Handling highly ambiguous or multi-interpretation questions still poses challenges, requiring richer interpretative frameworks and multi-modal data integration. Future work should address these limitations to enhance robustness and applicability.
Plain Language Accessible to non-experts
想象你在学校的图书馆，有不同的学生（代表不同社区）喜欢不同的学习方法。有的学生喜欢快速浏览，有的喜欢深入研究。每个学生的学习方式反映了他们的兴趣和目标。现在，假设有个聪明的图书管理员（模型），它可以回答你的问题，但它总是用一种方式回答，缺少变化。DiscoTrace就像一个能分析学生学习策略的工具，它能看出每个学生用的学习方法和偏好，帮你理解他们为什么这么做。通过这个工具，我们可以发现不同学生（社区）喜欢不同的学习策略，也能看到机器（模型）虽然能提供多样答案，却缺少个性和深度。这个方法让我们更好地理解人类和AI在“学习”和“回答问题”时的不同策略，从而帮助AI变得更聪明、更贴合每个人的需求。
ELI14 Explained like you're 14
想象你在学校的食堂点餐，不同的同学喜欢不同的菜。有的喜欢快餐，有的喜欢复杂的菜肴。每个人点的菜都反映了他们的喜好和想法。现在，假设有个超级厨师（模型），它可以做很多菜，但它总是用一样的方式做，缺少变化。DiscoTrace就像一个厨艺分析师，它能看出每个厨师用的材料和做法，帮我们理解他们为什么这么做。这个分析师还能告诉我们，不同的厨师喜欢不同的做法，有的喜欢简单快，有的喜欢复杂深。通过这个工具，我们可以让AI学会根据不同人的喜好，做出更贴心、更丰富的菜肴。这样，未来的AI厨师就能像真正的厨师一样，懂得变换策略，做出符合每个人口味的美味佳肴。
Abstract
We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.