Reward Prediction with Factorized World States

Recent advances in reinforcement learning (RL) and planning have emphasized the importance of reward signals for guiding agent behavior. Early methods relied on handcrafted rewards, which limited scalability. Deep learning introduced learned reward models, exemplified by works like VLWM and LLM-based critics, but these often overfit to training domains, hindering zero-shot generalization. To overcome this, semantic representations and hierarchical structures have been explored, aiming to capture task progress more abstractly. Despite progress, existing methods struggle with cross-domain transfer, especially in environments with sparse or noisy signals. The need for robust, interpretable, and generalizable reward prediction mechanisms remains urgent, motivating the development of models like StateFactory that leverage language models for semantic decomposition and dynamic goal interpretation.

The core challenge addressed is how to design reward prediction models that generalize across diverse, unseen environments without retraining. Existing supervised models tend to overfit, losing effectiveness in new tasks. Implicit methods, while flexible, lack explicit semantic grounding, leading to noisy or volatile reward signals. Additionally, evaluating reward quality step-by-step remains difficult due to the absence of standardized benchmarks. These issues hinder the deployment of autonomous agents capable of robust, zero-shot planning in real-world scenarios, where environments are dynamic and goals vary widely. Developing a unified, semantic-based reward prediction framework capable of cross-domain generalization is thus a critical open problem.

The paper introduces several key innovations: 1) StateFactory, which decomposes observations into hierarchical object-attribute structures using language models, providing rich semantic representations; 2) a dynamic goal interpretation module that updates goal states based on interaction history, ensuring contextual relevance; 3) a hierarchical routing mechanism that matches objects and attributes between current and goal states via semantic similarity, enabling fine-grained reward estimation; 4) a new benchmark, RewardPrediction, designed to evaluate step-wise reward accuracy across multiple domains with ground-truth rewards and EPIC distance metrics. These innovations collectively enable zero-shot reward prediction with high accuracy and cross-domain robustness, addressing fundamental limitations of prior models.

• �� State extraction: Use pre-trained language models to parse raw observations into structured object-attribute states, maintaining semantic richness.
• �� Goal interpretation: Employ a recurrent module to dynamically update goal states based on ongoing interactions, capturing environmental changes.
• �� Hierarchical routing: For each goal object, identify corresponding objects in current state by maximizing identity and attribute similarity, using semantic embeddings.
• �� Reward computation: Aggregate local object-level similarities into a global reward signal via normalized averaging, reflecting overall task progress.
• �� Training and evaluation: Use the RewardPrediction benchmark, measuring performance with EPIC distance, and conduct ablation studies to validate each component’s contribution.

The evaluation spans five environments—AlfWorld, ScienceWorld, TextWorld, WebShop, BlocksWorld—covering robotics, scientific reasoning, text puzzles, web navigation, and spatial planning. Each environment provides multiple trajectories with step-wise actions, observations, and ground-truth rewards. The models are compared against supervised reward models, VLWM-critic, and LLM-as-a-Judge, using EPIC distance as the primary metric. Ablation studies analyze the impact of hierarchical structures, goal interpretation, and semantic matching. Results show that StateFactory achieves an average EPIC distance of 0.297, outperforming baselines, with success rate improvements of over 21% in AlfWorld and 12% in ScienceWorld. The experiments demonstrate strong zero-shot generalization and robustness across tasks.

StateFactory significantly outperforms supervised and implicit methods, reducing average EPIC distance to 0.297 across all domains. In unseen tasks, it approaches the upper bound of models trained on all data, with a 60% reduction compared to VLWM-critic. Success rates on AlfWorld and ScienceWorld improve by 21.64% and 12.40%, respectively. Ablation confirms hierarchical object-attribute structures and dynamic goal interpretation as critical for performance. The results establish the method’s strong cross-domain generalization, validating its potential for scalable reward prediction in complex environments.

This approach can be directly applied to robotics, virtual assistants, and complex simulation environments requiring robust reward signals. It enables agents to adapt to new tasks without retraining, reducing development costs and increasing flexibility. The semantic hierarchical framework also supports explainability, aiding debugging and trust. Long-term, it can facilitate autonomous systems capable of continuous learning and adaptation in dynamic real-world settings, such as autonomous vehicles and intelligent manufacturing.

The method relies heavily on pre-trained language models, which may not perform well in highly specialized or low-resource domains. Its computational complexity, especially semantic matching and hierarchical routing, poses challenges for real-time deployment. The framework’s robustness in highly noisy or adversarial environments remains to be tested. Future work should focus on improving efficiency, scalability, and robustness, as well as extending to multi-modal data and more complex tasks.