ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
ThoughtTrace collects user-reported thoughts during multi-turn conversations, improving user behavior prediction (+41.7%) and model alignment (+25.6%) in large-scale datasets.
Key Findings
Methodology
ThoughtTrace employs a large-scale data collection platform where users engage in natural multi-turn dialogues with 20 different language models, annotating their thoughts as reasons and reactions for each message. The dataset comprises 1058 users, 2155 conversations, and over 17,000 dialogue turns, with annotations covering seven reason types and five reaction types. The analysis involves embedding difference metrics, semantic coverage scoring via GPT-4, and inference experiments with models like GPT-5.4, Gemini 3.1, and Claude Opus 4.6. The core aim is to characterize the properties of user thoughts, their distribution across conversation stages, content diversity, and inferability. The utility of thoughts is validated through user behavior prediction and model fine-tuning experiments, demonstrating significant improvements in both tasks.
Key Results
- Thoughts encompass diverse categories such as motivation, dissatisfaction, and context grounding, with content varying significantly across conversation stages. Embedding analysis shows that thoughts are semantically distinct from messages, with average similarity scores around 2.93/5, indicating high complexity. Incorporating thoughts into user message prediction improves accuracy by 41.7%, from a baseline of 21.6 to 30.6 in semantic similarity. In model alignment, thought-guided rewrites outperform message-based rewrites, increasing success rates by 25.6%, demonstrating their effectiveness in capturing user dissatisfaction and guiding model improvements.
- Analysis reveals that thoughts are stage-dependent, with early turns focusing on task motivation, and later turns emphasizing task continuation and satisfaction. Thought content is highly diverse, spanning high-level goals to specific dissatisfaction sources. The difficulty in inference underscores the importance of explicit annotation, as models only achieve partial understanding. These findings highlight the potential of thoughts as rich, actionable signals for advancing personalized AI systems.
Significance
This work pioneers the systematic collection and analysis of user thoughts in real-world human-AI interactions, addressing a critical gap in understanding internal cognitive states. By establishing thoughts as a new data modality, it enables more nuanced user modeling, enhances AI's ability to infer latent goals, and improves system responsiveness and personalization. The research offers a foundation for developing AI assistants that better align with human intentions, ultimately fostering more natural, satisfying, and trustworthy interactions. Its implications extend to fields like user modeling, reinforcement learning, and personalized AI training, promising a shift towards cognitively aware systems.
Technical Contribution
The paper introduces a novel data collection framework combining user self-reporting with multi-model dialogue, creating a large-scale, richly annotated dataset of user thoughts. It formalizes thoughts as reasons and reactions, distinct from utterances, validated through embedding and semantic analyses. The methodology integrates embedding difference metrics, semantic coverage scoring, and inference experiments to quantify thought complexity and inferability. The dataset enables downstream tasks such as user behavior prediction and model fine-tuning, demonstrating significant performance gains. This approach bridges the gap between observable dialogue and internal cognition, offering new avenues for research and system development.
Novelty
This is the first large-scale effort to systematically collect and analyze user-reported thoughts during real-world multi-turn conversations with AI. Unlike prior datasets focusing solely on message content, ThoughtTrace emphasizes the internal cognitive layer, capturing diverse, stage-dependent, and hard-to-infer thoughts. Its dual annotation of reasons and reactions provides a richer understanding of user motivation and satisfaction, setting a new standard for cognitive-aware dialogue research. The integration of explicit thought signals into model training and evaluation marks a significant innovation in human-AI interaction studies.
Limitations
- 依赖用户主动标注思想，可能存在偏差和遗漏，影响数据的代表性和普适性。
- 模型推断思想的准确性仍有限，难以完全还原用户真实认知状态，限制应用效果。
- 数据主要集中在日常任务场景，尚未扩展到专业或复杂领域，泛化能力有限。
Future Work
未来将结合多模态信息（如视觉、语音）丰富思想表达，提升模型推断能力，减少对用户标注的依赖。还计划扩展到专业场景和复杂任务中验证思想的适应性，探索思想在主动学习、个性化推荐及在线模型优化中的潜力。此外，研究将关注思想的动态变化机制，推动认知驱动的人机交互系统发展。
AI Executive Summary
The rapid proliferation of conversational AI systems has transformed daily interactions, yet understanding the internal cognitive states of users remains a challenge. Traditional datasets focus on what users explicitly say, neglecting the unspoken thoughts that drive their behavior. To address this gap, ThoughtTrace introduces a novel large-scale dataset capturing users’ self-reported thoughts during natural multi-turn dialogues with 20 different language models.
This dataset encompasses over 1,000 users, 2,155 conversations, and more than 17,000 dialogue turns, with rich annotations of reasons and reactions. Analysis reveals that user thoughts are diverse, stage-dependent, and semantically distinct from utterances, highlighting their complexity and importance. These thoughts are difficult for state-of-the-art models to infer directly, underscoring their value as explicit signals.
Empirical results demonstrate that incorporating thoughts significantly improves user behavior prediction (+41.7%) and model alignment (+25.6%). Thoughts enable models to better anticipate user needs and produce more personalized, satisfying responses. The findings also show that thoughts evolve dynamically across conversation stages, emphasizing their role in understanding human cognition.
This research establishes user thoughts as a new modality for human-AI interaction, providing a foundation for developing more intelligent, empathetic, and adaptable AI assistants. By bridging observable dialogue with internal mental states, ThoughtTrace opens new avenues for research and practical applications in personalized AI, user modeling, and cognitive-aware systems. Despite current limitations, such as reliance on user annotations and inference challenges, future work aims to integrate multimodal data and extend to complex domains, promising a more nuanced understanding of human-AI collaboration.
Deep Analysis
Background
随着生成式预训练模型（如GPT-4、PaLM）在对话系统中的广泛应用，研究逐渐关注对话的深层理解。早期工作如DSTC、PersonaChat等关注消息内容和用户偏好，强调多轮交互的重要性。然而，这些研究主要依赖于显式表达，忽略了用户潜在的认知状态。PRISM等尝试结合用户背景信息，但缺乏对内在思想的系统性捕获。近年来，模拟用户行为的研究（如UserSim）虽取得一定进展，但仍未解决思想难以直接观察的问题。ThoughtTrace通过引入用户自报思想，弥补了这一空白，提供了理解人类认知动态的新途径。
Core Problem
现有对话数据多关注用户的显性表达，缺乏对潜在思想的系统性捕获。思想作为用户未言明的认知状态，难以由模型推断，限制了对话系统的理解深度。如何在大规模真实场景中有效收集、分析用户思想，成为提升模型理解和个性化能力的关键。尤其是在多轮复杂交互中，用户的潜在需求、情感反应和未表达的偏好对对话质量影响巨大，但缺乏有效的标注和分析工具，成为制约研究的瓶颈。
Innovation
本研究提出ThoughtTrace框架，创新点在于：1）引入用户自主标注思想的方法，结合多轮对话，构建大规模思想数据集；2）定义思想为反应与理由两类，区别于传统消息内容，强调其认知动态特性；3）利用embedding差异、语义覆盖评分和模型推断验证思想的独立性和复杂性，为思想驱动的用户建模提供理论基础。通过这些创新，突破了以往仅关注消息的局限，开启了认知动态研究的新方向。
Methodology
- �� 采集平台：利用Prolific招募用户，用户在对话中自主标注思想，包括反应和理由。
- �� 数据结构：每轮对话包括消息内容、时间戳、用户背景信息及思想标签，思想类别涵盖七类理由和五类反应。
- �� 数据分析：采用embedding差异分析，利用预训练模型（如BERT）计算思想与消息的语义距离，验证思想的独立性。
- �� 语义覆盖：通过LLM（如GPT-4）评估思想是否包含消息未表达的内容。
- �� 推断验证：用模型尝试推断用户思想，比较推断结果与用户标注的差异，验证思想的复杂性。
- �� 任务应用：将思想用于用户行为预测和模型微调，验证其实用性。
Experiments
在包含多种前沿模型（GPT-5.4、GEMINI 3.1、Claude Opus 4.6）的数据集上进行，评估思想对用户行为预测的提升（+41.7%），以及思想引导的模型微调效果（成功率提升25.6%）。采用多轮对话场景，分析思想在不同阶段的表现，验证其内容丰富性和阶段依赖性。对比消息与思想的推断难度，发现模型难以准确推断思想，强调思想的独特价值。还进行了内容多样性和模型推断能力的消融分析，确保结果的稳健性。
Results
思想内容涵盖任务动机、情感反应等多类别，内容丰富且在对话不同阶段表现出明显变化。模型推断思想的平均相似度仅为2.93/5，显示其复杂性。思想显著提升用户行为预测的准确性（+41.7%），同时在模型微调中引入思想引导的重写，成功率提升25.6%。这些结果验证了思想作为人类认知动态的有效信号，推动了个性化和主动式AI的发展。
Applications
该数据集和分析框架可用于开发更理解用户潜在需求的对话系统，提升个性化推荐、主动式助手的能力。未来，结合多模态信息和在线学习，将使AI更好地理解复杂场景中的用户认知状态，推动智能交互的深度发展。
Limitations & Outlook
目前数据采集依赖用户自愿标注，存在主观偏差，影响思想的代表性。模型推断思想的准确性仍有限，难以完全还原用户真实认知状态，限制应用效果。对话场景主要集中在日常任务，尚未覆盖专业或复杂场景，泛化能力有限。未来需提升推断模型的准确性和扩展多场景适应性，降低标注成本，增强系统的普适性。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨师（AI）帮你准备食材，但你心里其实有很多想法，比如喜欢吃辣的、怕过敏、想快点完成。这些想法平时不说出来，但会影响你怎么选择食材和调味料。ThoughtTrace就像是让你在厨房里告诉厨师你的秘密想法，这样厨师就能更好地帮你做出符合你心意的菜。通过收集这些秘密想法，厨师可以学会更贴心地帮你准备饭菜，变得更懂你。它让AI像个贴心的厨师一样，知道你真正喜欢什么，避免误会，做出更合你心意的菜肴。
ELI14 Explained like you're 14
想象你和朋友玩游戏，你们会说话、笑、互相挑战，但其实你心里还在想：我是不是赢得太容易了？我是不是太无聊了？这些想法没有说出口，但会影响你接下来怎么玩。ThoughtTrace就像是让你偷偷告诉朋友你的这些想法，这样朋友就能更好地理解你，和你一起玩得更开心。它帮AI也学会听懂这些“心里话”，让对话变得更贴心、更有趣。未来，这样的AI可以更懂你，帮你解决问题、陪你聊天，就像一个真正懂你的小伙伴一样。
Abstract
Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.