UserBench: An Interactive Gym Environment for User-Centric Agents
UserBench evaluates user interaction capabilities, finding models align with user intent only 20% of the time.
Key Findings
Methodology
UserBench is built on the Gymnasium framework, simulating user preference revelation in travel planning. The environment offers a modular interface supporting tool-augmented search, requiring agents to make decisions under incomplete and indirect information.
Key Results
- Result 1: Models align with user intent only 20% of the time, revealing fewer than 30% of preferences.
- Result 2: Scores drop by 40% on average in single-choice settings.
- Result 3: Preference elicitation rates are low, especially through active querying.
Significance
UserBench fills a gap in existing evaluation environments by emphasizing user interaction capabilities, highlighting the challenges of building true collaborative agents. It provides a scalable platform for future research, advancing user-centric agents.
Technical Contribution
UserBench offers a systematic evaluation environment for agents' user interaction capabilities, emphasizing the combination of preference revelation and tool use, challenging current models' performance under uncertain and dynamic user needs.
Novelty
UserBench is the first to systematically evaluate agents' ability to reveal user preferences in multi-turn dialogues, contrasting with existing tool-use evaluations.
Limitations
- Limitation 1: Models perform poorly in preference revelation, especially during active querying.
- Limitation 2: Simulated user preferences may not fully reflect real-world complexity.
Future Work
Future research could focus on improving models' ability to reveal user preferences, exploring more complex user simulations and diverse interaction scenarios.
AI Executive Summary
UserBench is a novel user-centric evaluation environment designed to test large language models' user interaction capabilities in multi-turn dialogues. While existing models excel in tool use, they fall short in understanding and adapting to user needs.
UserBench simulates user preference revelation in travel planning, providing a modular and scalable testing platform. Experimental results show that models' scores drop by 40% on average in single-choice settings, aligning with user intent only 20% of the time.
This research highlights the challenges of building true collaborative agents, advancing user-centric agents. Future directions include improving models' user preference revelation capabilities and exploring more complex user simulations.
Deep Analysis
Background
In recent years, large language models have made significant progress in reasoning and tool use. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored.
Core Problem
Existing evaluation environments often overlook the role of the user, leading to models that perform well in task execution but poorly in meeting real user needs.
Innovation
UserBench provides a systematic evaluation environment for agents' user interaction capabilities by simulating user preference revelation in travel planning. It emphasizes the combination of preference revelation and tool use, challenging current models' performance under uncertain and dynamic user needs.
Methodology
- �� UserBench is built on the Gymnasium framework, simulating user preference revelation in travel planning.
- �� Offers a modular interface supporting tool-augmented search.
- �� Requires agents to make decisions under incomplete and indirect information.
Experiments
The experimental design includes evaluating several leading open- and closed-source models, using single-choice and multi-choice settings to test models' performance in preference revelation and tool use.
Results
Experimental results show that models' scores drop by 40% on average in single-choice settings, aligning with user intent only 20% of the time, with low preference elicitation rates, especially through active querying.
Applications
UserBench provides a scalable platform for future research, advancing user-centric agents.
Limitations & Outlook
Models perform poorly in preference revelation, especially during active querying. Simulated user preferences may not fully reflect real-world complexity.
Plain Language Accessible to non-experts
Imagine planning a trip but you're not sure what you want. UserBench is like a smart assistant that learns your preferences through conversation, like whether you prefer direct flights or cheaper options. This assistant needs to guess your true intentions when you don't clearly express them and help you make the best choices.
ELI14 Explained like you're 14
Imagine you're playing a game where you're a travel planner. The client tells you where they want to go but not their specific preferences. You need to ask questions to find out what they like, like what kind of hotel or restaurant they prefer. UserBench helps AI understand and assist users better in these situations!
Glossary
Gymnasium Framework
A standard framework for building and evaluating reinforcement learning environments.
UserBench is built on the Gymnasium framework.
Preference Revelation
The process of gradually uncovering a user's true needs and preferences through interaction.
UserBench requires agents to reveal user preferences through multi-turn dialogue.
Tool-Augmented Search
Using external tools to enhance information retrieval and decision-making capabilities.
UserBench supports tool-augmented search functionality.
Multi-Turn Dialogue
A form of conversation where information is gradually revealed and goals are achieved over multiple interactions.
UserBench simulates user preference revelation in multi-turn dialogue.
User-Centric Agent
An intelligent agent capable of understanding and adapting to user needs.
UserBench aims to evaluate the capabilities of user-centric agents.
Open Questions Unanswered questions from this research
- 1 How to improve models' performance in preference revelation?
- 2 How to simulate more complex user interaction scenarios?
Applications
Immediate Applications
Travel Planning Assistant
Helps users plan trips with uncertain needs, gradually learning preferences through multi-turn dialogue.
Long-term Vision
Intelligent Customer Service Systems
Applied in various service scenarios to help businesses better understand and meet customer needs.
Abstract
Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.