Cognitive World Model for Progressive BDI/E Trajectory Evaluation of Conversational Agents
CogWM tracks user BDI/E states for cognitive evaluation of conversational agents, trained on 150K samples, outperforming baselines.
Key Findings
Methodology
CogWM jointly models users' Belief, Desire, Intention, and Emotion (BDI/E) states and responses, enabling explicit cognitive trajectory tracking. It employs the Summarize-and-Allocate (SaA) annotation pipeline to ensure temporal consistency.
Key Results
- CogWM achieves 77.6% emotion accuracy, significantly outperforming GPT-5.5's 36.4%. Its BDI/E state prediction scores are 0.773, 0.705, and 0.458, respectively.
- For response generation, CogWM scores 0.417 ROUGE-L and 0.150 BLEU-4, with better semantic consistency compared to baselines.
- Through 3,600 multi-turn interactions, CogWM reveals distinct cognitive trajectory patterns across agents, quantified via the CTS metric.
Significance
CogWM introduces a novel perspective for evaluating conversational agents by explicitly modeling user BDI/E states. Its hierarchical evaluation framework links conversational strategies to cognitive evolution, offering diagnostic tools for optimizing interaction strategies.
Technical Contribution
CogWM is the first framework to explicitly model user BDI/E states alongside responses, providing traceable cognitive trajectories. The Summarize-and-Allocate annotation pipeline ensures structured and temporally coherent cognitive supervision.
Novelty
CogWM is the first to jointly model explicit BDI/E states and responses, addressing gaps in existing methods that focus solely on surface-level responses or implicit states.
Limitations
- Intention prediction remains challenging due to sparse annotations and its future-oriented nature.
- High annotation costs limit scalability to larger datasets.
- Performance in highly complex tasks requires further validation.
Future Work
Future work includes improving intention prediction, scaling to more complex tasks, and reducing annotation costs for broader applicability.
AI Executive Summary
With the growing deployment of LLM-based conversational agents in open-ended scenarios, evaluating their effectiveness solely through task completion is insufficient. The evolution of users' internal states, including Belief, Desire, Intention, and Emotion (BDI/E), provides a deeper signal of how conversational strategies influence users during multi-turn interactions. However, existing evaluation methods focus on surface-level responses or final outcomes, failing to capture these cognitive dynamics.
To address this, the Cognitive World Model (CogWM) is introduced as an LLM-based cognitive user model that explicitly tracks BDI/E trajectories. CogWM employs a Summarize-and-Allocate (SaA) pipeline to extract temporally coherent annotations from multi-turn dialogues, enabling structured cognitive supervision. Trained on 150K user-turn samples, CogWM outperforms existing baselines in both response generation and cognitive state prediction.
By interacting with six state-of-the-art conversational agents across 3,600 dialogues, CogWM reveals distinct cognitive trajectory patterns and evaluates agents using a hierarchical framework. This framework links turn-level cognitive responses, trajectory dynamics, and task outcomes, providing actionable insights for optimizing conversational strategies. Future work aims to enhance intention modeling, extend to more complex tasks, and reduce annotation costs for broader adoption.
Deep Analysis
Background
Recent advances in LLM-based conversational agents have enabled their deployment in open-ended, interaction-intensive scenarios. However, traditional evaluation metrics like BLEU and ROUGE focus on surface-level response quality, while newer frameworks like MT-Bench and AlpacaEval aggregate multi-turn interactions into final scores. These approaches fail to capture the cognitive dynamics that drive user behavior, particularly in scenarios like emotional support and persuasion, where understanding user beliefs, desires, intentions, and emotions is critical.
Core Problem
Existing evaluation paradigms lack the ability to track users' evolving cognitive states during multi-turn interactions. This limitation hinders diagnosing why agents succeed or fail and prevents the optimization of conversational strategies, especially in cognitively demanding tasks such as emotional support and persuasive dialogues.
Innovation
CogWM introduces explicit modeling of users' BDI/E states, a first in the field. It employs the Summarize-and-Allocate (SaA) annotation pipeline to extract global cognitive evolution from dialogues and allocate turn-level BDI/E labels. Unlike prior methods, CogWM provides interpretable cognitive trajectories, bridging the gap between user behavior and interaction outcomes.
Methodology
- �� CogWM is based on the Qwen3-14B model, fine-tuned using LoRA.
- �� The Summarize-and-Allocate (SaA) pipeline extracts global BDI/E trajectories from four datasets (DailyDialog, ESConv, P4G, DuRecDial) and assigns turn-level annotations.
- �� Defines a structured BDI/E state space with semantic descriptions and task-aware scores.
- �� Jointly optimizes for cognitive state prediction and response generation with the objective p(Ut, St|D<t, P, C).
Experiments
Experiments include: 1) Joint state-response generation to evaluate emotion accuracy, BDI/E prediction, and response quality; 2) Cognitive state understanding with oracle user utterances; 3) Multi-turn interactions with six conversational agents to analyze cognitive trajectory patterns.
Results
CogWM achieves 77.6% emotion accuracy, outperforming baselines. Its BDI/E prediction scores are 0.773, 0.705, and 0.458. In multi-turn interactions, CogWM quantifies cognitive evolution quality using the CTS metric, revealing distinct agent patterns.
Applications
CogWM can be applied to emotional support, persuasive dialogue, and recommendation systems, aiding in optimizing conversational strategies. Its hierarchical evaluation framework also supports academic and industrial dialogue system assessments.
Limitations & Outlook
CogWM struggles with intention prediction due to sparse annotations. Annotation costs limit scalability to larger datasets. Further validation is needed for highly complex tasks.
Plain Language Accessible to non-experts
Imagine you're talking to a friend who's feeling anxious. Through a series of conversations, you help them feel better. Their changing emotions, thoughts, and plans reflect what's happening inside their mind. CogWM acts like a 'mind tracker,' recording and analyzing these internal changes to better understand how effective the conversation is.
ELI14 Explained like you're 14
Think of playing a game where you chat with a virtual character who's sad, and your goal is to make them happy. You choose different dialogue options to influence their thoughts, emotions, and plans. CogWM is like a super tool that shows you how each choice affects the character's feelings and decisions, making it easier to win the game!
Glossary
BDI/E States
Belief, Desire, Intention, and Emotion states that reflect users' internal cognitive processes.
Used to track user evolution during multi-turn dialogues.
CogWM
An LLM-based cognitive user model that explicitly tracks user BDI/E states and responses.
Evaluates cognitive trajectories of conversational agents.
SaA Pipeline
Summarize-and-Allocate pipeline extracts global cognitive evolution and assigns turn-level annotations.
Ensures temporal consistency in BDI/E supervision.
CTS
Composite Trajectory Score quantifies cognitive trajectory quality.
Measures agents' influence on user cognitive states.
Emotional Support Dialogue
A scenario where the goal is to alleviate users' emotional stress.
CogWM evaluates agents' ability to regulate emotions in this context.
Open Questions Unanswered questions from this research
- 1 How can intention prediction be improved for more complex scenarios?
- 2 What methods can reduce annotation costs for large-scale cognitive modeling?
Applications
Immediate Applications
Emotional Support Systems
Helps users alleviate emotional stress by generating empathetic dialogues.
Dialogue Strategy Optimization
Improves multi-turn interaction strategies through cognitive trajectory analysis.
Long-term Vision
Cognition-Driven Agents
Develop next-gen dialogue systems that actively influence user cognitive states.
Abstract
As LLM-based conversational agents advance toward increasingly open-ended and interaction-intensive scenarios, task completion alone provides an incomplete assessment of their effectiveness. The evolution of users' internal states, including beliefs, desires, intentions, and emotions (BDI/E), serves as an intermediate signal connecting agent behaviors with interaction outcomes and reflects how conversational strategies shape users during multi-turn interactions. However, existing evaluation paradigms primarily focus on surface-level responses or final outcomes, providing limited insight into the underlying cognitive processes. This limitation makes it difficult to diagnose why agents succeed or fail and to optimize their interaction strategies. To address this challenge, we propose Cognitive World Model (CogWM), an LLM-based cognitive user model that jointly models users' BDI/E states and corresponding responses, enabling explicit cognitive trajectory tracking. Trained on 150K user-turn samples with Qwen3-14B, CogWM achieves superior performance over existing user simulation baselines in both response fidelity and cognitive state understanding. Interactions with six state-of-the-art LLMs demonstrate that CogWM enables progressive comparison of agents through cognitive trajectories, revealing distinct agent patterns and complementary relationships between cognitive evolution and behavioral outcomes.