WorldReward: Reward Modeling for Camera-Conditioned World Models
Introduces WorldReward, a VLM-based pairwise preference reward model combining action consistency and visual quality, validated on a new benchmark.
Key Findings
Methodology
WorldReward decomposes long videos into action-aligned chunks, using structured visual evidence for each. It employs a VLM to perform pairwise preferences at the chunk level, judging both action correctness and visual quality simultaneously. Chunk decisions are aggregated via voting into global preferences. Training data is generated through large-scale reasoning with a frontier VLM, multi-turn tool-based auditing, and human calibration, ensuring high-quality annotations. The model's effectiveness is validated on the newly introduced WorldReward-Bench, covering diverse scenarios. When integrated into RL post-training of HY-WorldPlay 1.5, it significantly improves both action accuracy and visual fidelity across various horizons.
Key Results
- On WorldReward-Bench, WorldReward surpasses GPT-5.5 by 3.42% in action preference, 1.45% in appearance, and 3.56% in motion quality, demonstrating superior alignment with human judgments.
- In RL fine-tuning, it consistently enhances action execution and visual realism, outperforming baseline reward models across short- and long-term horizons.
- Ablation studies confirm the importance of structured chunk reasoning, source image, and detailed panels, with performance drops when these are removed.
Significance
This work advances reward modeling for camera-conditioned world models by unifying action and visual assessments within a single VLM-based framework. It addresses the challenge of evaluating long, multimodal videos with localized errors, providing a scalable solution that improves the fidelity and controllability of generated content. The approach bridges the gap between geometric trajectory accuracy and perceptual quality, enabling more natural and coherent scene synthesis. Its impact extends to virtual reality, automated content creation, and AI-driven scene understanding, offering a robust tool for optimizing complex generative models.
Technical Contribution
The paper introduces a novel chunk-level, structured reasoning paradigm within a VLM-based preference model, enabling localized evaluation of action and visual quality. It innovatively combines pairwise preference learning with multi-modal structured inputs, leveraging large-scale reasoning-augmented datasets refined through tool-based and human feedback. The voting aggregation mechanism ensures robustness against local failures, while the large dataset supports generalization across diverse scenarios. This approach surpasses existing geometric or image-based reward models, offering a unified, scalable, and interpretable framework for video reward modeling.
Novelty
This is the first work to integrate VLMs into a unified, chunk-based preference reward for camera-conditioned long videos, explicitly addressing the local-to-global attribution problem. Unlike prior geometric or perceptual-only models, it combines action alignment and visual quality judgments within a single reasoning process, enabling more accurate and holistic evaluation. Its structured, multi-scale approach and large-scale annotated dataset set new standards in multimodal reward modeling for video generation.
Limitations
- Dependence on high-quality, diverse preference data; biases in annotations may affect model performance, especially in stylized or rare scenarios.
- Computational cost of chunk-level reasoning and large dataset training limits real-time applicability.
- Current focus on camera-controlled videos; generalization to other modalities or uncontrolled environments remains to be validated.
Future Work
Future efforts will focus on automating preference data collection, expanding to more diverse and complex scenes, and integrating more advanced VLM architectures. Enhancing real-time inference efficiency and extending the framework to other modalities like audio-visual content are also promising directions. Additionally, combining this reward model with autonomous learning algorithms could further improve scene realism and controllability in interactive applications.
AI Executive Summary
In recent years, the development of camera-conditioned world models has enabled highly interactive and controllable video generation, with applications spanning virtual reality, gaming, and content creation. However, evaluating the quality of these generated videos remains a significant challenge. Traditional reward models focus either on geometric trajectory accuracy or perceptual image quality, but rarely both simultaneously. This gap limits the ability to generate scenes that are both geometrically correct and visually appealing.
To address this, the authors introduce WorldReward, a novel reward modeling framework based on vision-language models (VLMs). Unlike prior methods, WorldReward decomposes long videos into short, action-aligned chunks, allowing the model to analyze localized scene transitions. Each chunk is evaluated for both action correctness and visual quality using structured visual evidence, including source images, frame grids, and action panels. The model then aggregates these chunk-level preferences via voting, producing comprehensive global assessments.
Training data for WorldReward is meticulously constructed through a multi-stage process: large-scale reasoning with a frontier VLM, multi-turn tool-based auditing, and human calibration. This ensures high-quality, diverse annotations covering various scenarios, styles, and control types. To validate its effectiveness, the authors introduce WorldReward-Bench, a human-annotated benchmark with 760 paired videos across different dimensions. Results show that WorldReward achieves superior agreement with human preferences, outperforming proprietary models like GPT-5.5.
When integrated into reinforcement learning for the HY-WorldPlay 1.5 system, WorldReward significantly enhances both action fidelity and visual realism, demonstrating its practical value. The approach sets a new standard for multimodal video reward modeling, offering a scalable, interpretable, and robust framework that bridges the gap between geometric and perceptual evaluation. Future work will explore data automation, model efficiency, and broader applicability, aiming to push the boundaries of autonomous, high-quality scene synthesis.
Deep Dive
Abstract
Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess these requirements separately: geometry-based rewards estimate trajectory execution but cannot judge the visual quality of the executed motion, whereas image-based rewards measure frame quality without capturing action execution or temporal dynamics. We posit that a vision-language model (VLM) offers a shared reasoning space for relating actions to their visual outcomes. However, judging a complete long video against its full action sequence creates a lengthy, noisy context in which short-lived local action evidence can be missed or diluted. We present WorldReward, a VLM-based pairwise preference reward model that unifies action-consistency and visual-quality evaluation for camera-conditioned world models. WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences. To train it, we construct a large-scale reasoning-augmented preference dataset using structured judgments generated by a frontier VLM and refined through tool-based agent auditing and targeted human review. We further introduce WorldReward-Bench, a human-annotated benchmark measuring reward-model agreement with human preferences across action consistency, appearance quality, and motion quality. WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively. When used for RL post-training of HY-WorldPlay 1.5, it consistently improves both action execution and visual quality across short- to long-term horizons.