Estimating the Empowerment of Language Model Agents
Proposes EELMA, an information-theoretic method to estimate language model agents' empowerment, strongly correlating with task performance.
Key Findings
Methodology
This paper introduces EELMA, an algorithm combining pretrained language model embeddings and contrastive InfoNCE loss to estimate an agent's influence over future states from multi-turn text interactions. It maps observations and actions into a low-dimensional embedding space, then maximizes a variational lower bound of mutual information between actions and future states. This approach addresses high-dimensional, redundant textual data, providing a scalable, goal-agnostic influence measure. Experiments validate EELMA's estimates against ground-truth in structured environments and demonstrate strong correlation with task rewards in web and tool-use scenarios, confirming its utility as a general capability metric.
Key Results
- In Gridworld and Tower of Hanoi environments, EELMA's influence estimates closely match direct calculations, with correlation coefficients exceeding 0.8 and minimal error margins.
- In real web environments like WebArena and τ-bench, influence scores from EELMA significantly correlate with average task rewards (r > 0.7), confirming robustness in complex, natural language settings.
- Ablation studies show that removing Chain-of-Thought prompting or reducing memory length decreases estimated influence, indicating sensitivity to reasoning and memory capacity, aligning with performance drops.
Significance
This work advances AI evaluation by providing a goal-agnostic, scalable influence measure that captures the agent's ability to explore and control in open-ended text environments. It addresses the limitations of reward-based metrics, offering a new perspective on model autonomy, safety, and general intelligence. The influence metric can serve as a complementary tool for understanding model capabilities beyond task-specific success, fostering safer and more interpretable autonomous AI systems.
Technical Contribution
The paper develops EELMA, a novel influence estimation framework leveraging pretrained language embeddings and contrastive mutual information bounds. It introduces a joint training procedure for encoders that encode current and future observations, enabling efficient, robust influence estimation in high-dimensional, variable text spaces. The method extends influence measures from symbolic states to natural language, providing theoretical guarantees and practical scalability, setting a foundation for future multi-modal, multi-task evaluations.
Novelty
This is the first work to adapt influence estimation to natural language environments using deep embedding and contrastive learning, overcoming the challenges of high-dimensional, redundant textual data. Unlike prior symbolic or low-dimensional approaches, EELMA provides a scalable, unsupervised influence measure directly from raw text trajectories, representing a significant innovation in AI capability assessment.
Limitations
- The accuracy of influence estimates depends on the quality of pretrained embeddings; biases or errors in language models can affect results. In highly novel or ambiguous environments, the embeddings may not fully capture the state space.
- Computational costs remain high for large models or long trajectories, requiring further optimization for real-time or large-scale deployment.
- While influence correlates with task reward, it may not fully capture goal-specific performance or safety concerns, necessitating combined metrics for comprehensive evaluation.
Future Work
Future research will extend influence estimation to multi-modal data, integrating visual and auditory inputs for richer environment understanding. Efforts will focus on improving computational efficiency, enabling real-time influence monitoring. Additionally, exploring the relationship between influence and safety, bias detection, and robustness will be key to deploying autonomous systems in real-world applications. The framework could also be adapted for multi-agent interactions, fostering cooperative and competitive AI research.
AI Executive Summary
The rapid advancement of large language models (LLMs) has unlocked unprecedented capabilities in autonomous decision-making and interaction. However, evaluating these models' true potential remains a challenge. Traditional benchmarks rely on goal-specific tasks, which are labor-intensive and limited in scope, often failing to capture the full spectrum of a model’s abilities. Recognizing this gap, the authors propose a novel, goal-agnostic metric—empowerment—derived from information theory, to quantify an agent’s influence over future states through its actions.
Building on this concept, the paper introduces EELMA, an innovative algorithm that estimates empowerment from multi-turn text interactions. EELMA employs pretrained language model embeddings to represent observations and actions, then applies contrastive learning via InfoNCE to approximate the mutual information between actions and future states. This approach effectively handles the high-dimensional, redundant, and variable nature of textual data, providing a scalable and robust influence measure.
Extensive experiments demonstrate that EELMA accurately estimates influence in structured environments like Gridworld and Tower of Hanoi, with correlations exceeding 0.8 compared to direct calculations. In real-world web and tool-use environments, influence scores strongly correlate with task rewards, validating its practical relevance. The method also reveals how model size, prompting strategies, and memory affect influence, offering insights into model capabilities.
Overall, this work offers a powerful new tool for evaluating AI agents beyond task success, emphasizing their intrinsic exploratory and control abilities. While promising, challenges remain in scaling to larger environments and integrating multi-modal data. Future work aims to refine the algorithm, extend its scope, and deepen understanding of influence in autonomous AI systems, ultimately contributing to safer, more capable, and interpretable models.
Deep Dive
Abstract
As language model (LM) agents become increasingly capable and adopted in real-world applications, there is a growing need for scalable evaluation frameworks beyond costly, manually designed benchmarks. We propose information-theoretic evaluation based on empowerment, an information-theoretic measure of an agent's influence on future states through its actions. To handle the unique challenges of text-based environments, we introduce EELMA (Estimating Empowerment of Language Model Agents), an algorithm for approximating effective empowerment from multi-turn text interactions. We demonstrate EELMA on textual games and realistic web and tool-use environments, showing that empowerment strongly correlates with average task performance. We further analyze how empowerment varies across models, environment complexity, and agent configurations, and show that high-empowerment states and actions often mark pivotal moments for general capabilities. These results establish empowerment as a goal-agnostic metric that complements task-success measures for LM-agent evaluation.