Evaluating Human-Language Model Interaction
HALIE framework evaluates human-AI interaction, revealing that superior non-interactive metrics do not guarantee better user experience.
Key Findings
Methodology
This study develops the HALIE framework, integrating five diverse tasks—dialogue, QA, crossword, summarization, metaphor generation—by defining system states, user actions, and interaction traces. It employs multi-dimensional metrics—targets, perspectives, criteria—to evaluate models' interactive performance. Using four state-of-the-art models (GPT-3 variants and Jurassic-1), large-scale interaction data were collected. Analysis reveals significant divergence between traditional non-interactive metrics and actual user-centered interaction performance, emphasizing the need for comprehensive evaluation approaches.
Key Results
- Models like GPT-3 Davinci achieved 92% accuracy in question answering but did not necessarily produce the highest user satisfaction scores, illustrating the gap between static metrics and interactive quality. Conversely, models such as Jumbo excelled in user satisfaction during crossword tasks despite moderate traditional scores. In creative tasks like metaphor generation, the number of user edits correlated with model scores, but effort metrics (time, query count) showed different model rankings, indicating that multiple evaluation angles are essential for true performance assessment.
- The experiments demonstrate that optimizing for traditional benchmarks alone can mislead model selection for real-world interaction. Models with lower static scores sometimes outperform high scorers in user-centered metrics, highlighting the importance of multi-faceted evaluation. The findings advocate for shifting focus from solely output quality to interaction quality, including user enjoyment and task engagement.
- Overall, the results underscore the importance of designing evaluation frameworks that incorporate process, perspective, and preference, enabling more accurate assessment of models in practical, human-centric scenarios. This approach guides future model development towards more natural and satisfying interactions.
Significance
This research advances AI evaluation by moving beyond static output metrics, emphasizing the importance of interaction dynamics and user experience. The HALIE framework provides a comprehensive, multidimensional approach, aligning model assessment with real-world application needs. It addresses a critical gap in current benchmarks, which often overlook the interactive and subjective aspects of AI deployment. The findings influence both academia and industry, encouraging the development of models that are not only accurate but also engaging, user-friendly, and adaptable. This paradigm shift supports the creation of AI systems that truly serve human needs, fostering trust and wider adoption in areas like virtual assistants, education, and content creation.
Technical Contribution
The core innovation lies in the formalization of the HALIE framework, which models human-AI systems as states, actions, and interaction traces, enabling multidimensional evaluation. It introduces the targets-perspectives-criteria schema, allowing simultaneous assessment of output quality, user experience, and process efficiency. The framework is compatible with various tasks and models, facilitating large-scale, systematic analysis of interactive performance. It also incorporates novel metrics such as user satisfaction, effort, and preference, which are rarely considered in traditional benchmarks. This comprehensive approach bridges the gap between static evaluation and real-world usability, offering new tools for model development and benchmarking.
Novelty
This is the first systematic framework explicitly designed to evaluate human-AI interaction across multiple dimensions, integrating process, perspective, and preference. Unlike prior work focusing solely on output quality, HALIE emphasizes the interaction trajectory and subjective user experience. Its multi-task validation demonstrates broad applicability, setting a new standard for interactive AI evaluation. This holistic approach addresses the limitations of existing benchmarks, which often neglect the dynamic, user-centered aspects of AI deployment, marking a significant advancement in the field.
Limitations
- The current implementation primarily relies on English interaction data, limiting cross-lingual applicability. Future work should include multilingual datasets to enhance generalizability.
- UI design choices, such as prompt presentation and interaction flow, significantly impact user experience but were standardized here, leaving room for further optimization.
- Models evaluated are static; incorporating online learning or adaptation mechanisms could better reflect real-world deployment scenarios, which remain to be explored.
Future Work
Future research will extend the framework to multilingual and multimodal settings, integrating physiological and behavioral data for richer evaluation. Incorporating adaptive, online learning models will enable personalized interactions. Additionally, exploring UI/UX design impacts and developing more sophisticated metrics for subjective experience will further refine the evaluation process. The goal is to create a comprehensive, scalable system that guides the development of truly human-centric AI systems capable of natural, engaging, and trustworthy interactions.
AI Executive Summary
The rapid evolution of language models (LMs) has revolutionized natural language processing, enabling applications from chatbots to content creation. However, traditional evaluation methods focus narrowly on static output metrics like BLEU, ROUGE, or accuracy, which inadequately reflect the quality of human-AI interactions in real-world scenarios. Recognizing this gap, the present study introduces the HALIE framework—Human-AI Language-based Interaction Evaluation—a comprehensive, multi-dimensional system designed to assess the full spectrum of human-LM interactions.
HALIE conceptualizes the interaction as a dynamic process involving states, actions, and trajectories, capturing not only the final outputs but also the entire interaction flow. It evaluates models along three orthogonal dimensions: targets (what is being evaluated), perspectives (whose view is considered), and criteria (what aspects are measured). This approach allows for a nuanced understanding of how models perform in practical settings, considering user satisfaction, effort, and preferences.
To validate the framework, five diverse tasks were designed—social dialogue, question answering, crossword puzzles, summarization, and metaphor generation—covering goal-oriented and open-ended interactions. Four state-of-the-art models, including GPT-3 variants and Jurassic-1, were tested extensively. Results revealed that models excelling in traditional benchmarks did not always deliver superior interactive experiences. For example, GPT-3 Davinci achieved high accuracy in question answering but did not outperform others in user satisfaction. Conversely, some models with moderate static scores excelled in user engagement and enjoyment.
These findings underscore the importance of multidimensional evaluation, emphasizing that optimizing for static metrics alone is insufficient for real-world deployment. The study advocates for a shift towards user-centered assessment, integrating subjective preferences and interaction quality. The implications extend to AI system design, encouraging developers to prioritize natural, engaging, and adaptable interactions.
Despite its strengths, the framework has limitations, such as reliance on English data and static models. Future work aims to incorporate multilingual, multimodal, and adaptive learning capabilities, broadening its applicability. Overall, HALIE offers a vital step towards more human-centric AI evaluation, fostering systems that are not only accurate but also enjoyable and trustworthy partners in everyday life.
Deep Dive
Glossary
Interaction Trace (交互轨迹)
A record of system states and user actions during an interaction, used for performance analysis. In this paper, it captures the full human-LM exchange process.
Targets (目标)
The specific elements evaluated in an interaction, such as final outputs or process steps, beyond just the end result.
Perspectives (视角)
The viewpoint from which evaluation occurs, either from the user’s first-person experience or third-party assessment.
Criteria (标准)
The metrics used to judge performance, including objective quality and subjective preferences.
System Logic (系统逻辑)
The formal structure of states, actions, and transitions that define how an interactive system functions.
Open Questions Unanswered questions from this research
- 1 如何在多语种、多文化背景下扩展HALIE框架的适用性仍待探索，尤其是跨文化交互中的偏好和体验差异。
- 2 UI设计对用户体验的影响机制尚未充分理解，未来应结合用户行为和生理数据进行多模态评估。
- 3 模型的动态学习和适应能力在交互中的作用需要进一步研究，以实现更个性化和持续优化的系统。
Applications
Immediate Applications
智能助手优化
利用多维评估体系改善虚拟助手的交互体验，使其更贴合用户偏好，提升满意度和效率。
内容创作工具
通过评估模型在创意写作中的表现，打造更具人性化的写作辅助系统，满足用户个性化需求。
Long-term Vision
人机协作平台
构建多模态、多用户的协作环境，实现更自然、更高效的团队合作与知识共享。
Abstract
Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.