ViLBench: A Suite for Vision-Language Process Reward Modeling
Introduces VILBench and 73.6K vision-language process reward data, enabling training of a 3B PRM that outperforms baselines by 3.3%.
Key Findings
Methodology
This work systematically benchmarks seven VLLMs as reward models (ORMs and PRMs) across five vision-language tasks, using a Best-of-N strategy with GPT-4o. By filtering 600 high-difficulty samples from six open-source VLLMs, the authors construct VILBench emphasizing step-wise reward signals. They collect 73.6K process reward samples via an enhanced tree search algorithm, then train a 3B-scale vision-language PRM (ViLPRM) with these data. The approach integrates multi-step reward signals and evaluates performance improvements, revealing key differences between ORM and PRM, and the weak correlation between model strength and reward ability.
Key Results
- GPT-4o Chain-of-Thought (CoT) achieves only 27.3% accuracy on VILBench, indicating high task difficulty. The trained ViLPRM improves average reward prediction by 3.3%, with up to 2.5% gains over untrained models. PRMs outperform ORMs in fine-grained reward tasks, especially in reasoning and math domains. Using last n-step rewards further enhances performance, demonstrating the benefit of multi-step reward fusion.
- Evaluation shows that stronger VLLMs do not necessarily yield better reward models; PRMs trained on large-scale process reward data outperform baseline models. The data-driven approach and tree search sampling significantly boost reward accuracy, validating the importance of detailed reward signals in complex multimodal tasks.
Significance
This research addresses a critical gap in multimodal AI by establishing a benchmark and dataset for fine-grained process rewards, which are essential for training models with better reasoning and interpretability. The large-scale reward data and the trained PRM set new standards for reward modeling, facilitating more robust alignment and reinforcement learning in vision-language systems. The findings highlight that reward quality, rather than model size alone, determines performance, guiding future development of generalizable reward models for complex tasks.
Technical Contribution
The paper introduces VILBench, a novel benchmark emphasizing step-wise reward evaluation, and a large-scale dataset of 73.6K process reward samples. It proposes a tree search-based sampling method to collect high-quality reward data, and trains a 3B-scale PRM (ViLPRM) that surpasses existing models. The integration of multi-step reward fusion strategies and the extensive benchmarking of VLLMs as reward models constitute significant technical advances, enabling more precise and scalable reward learning in multimodal AI.
Novelty
This is the first comprehensive evaluation of VLLMs as reward models in multimodal tasks, emphasizing process supervision. The creation of VILBench and the large-scale process reward dataset represent pioneering steps, shifting focus from final output correctness to detailed reasoning steps. The use of enhanced tree search for reward data collection and the training of a dedicated PRM further distinguish this work from prior single-step or output-only reward approaches.
Limitations
- The dataset, while large, still covers limited task diversity and scenarios, which may restrict generalization. Expanding to more complex, real-world applications remains necessary.
- Reward data collection relies on computationally intensive tree search algorithms, incurring high costs and potential biases from sampling strategies.
- Current models struggle with highly ambiguous or out-of-distribution inputs, indicating the need for more robust reward signals and training methods.
Future Work
Future directions include scaling up data diversity, integrating self-supervised signals, and exploring reinforcement learning with learned reward models. Enhancing reward model robustness, extending to more complex real-world scenarios, and improving efficiency of reward data collection are key goals. Additionally, combining visual and linguistic cues more effectively can further improve model reasoning and interpretability.
AI Executive Summary
In recent years, the integration of vision and language has driven significant advances in AI, enabling systems to understand and reason across multiple modalities. However, guiding these models to produce more accurate and interpretable outputs remains a challenge. Reward models (RMs), which align model responses with human preferences, have traditionally focused on final output correctness. Yet, in complex multimodal tasks, understanding and evaluating the reasoning process itself is crucial. This paper introduces VILBench, a dedicated benchmark designed to evaluate the step-wise reward capabilities of vision-language models. By curating a dataset of over 73,600 process reward samples, the authors demonstrate that detailed, fine-grained rewards significantly improve model performance.
The core innovation lies in leveraging an enhanced tree search algorithm to collect high-quality reward data, which is then used to train a 3B-scale vision-language process reward model (ViLPRM). This model surpasses baseline reward models, achieving an average improvement of 3.3% in reward prediction accuracy. The evaluation across multiple tasks reveals that reward models trained on detailed process signals outperform traditional output-based reward models, especially in reasoning-intensive scenarios.
Furthermore, the study highlights that stronger general VLLMs do not necessarily translate into better reward capabilities, emphasizing the importance of specialized reward training. The multi-step reward fusion strategy further enhances performance, demonstrating that balancing rewards from different steps yields optimal results. Overall, this work advances the understanding of multimodal reward learning, providing a scalable framework and dataset for future research. Despite current limitations in data diversity and computational costs, the findings pave the way for more robust, interpretable, and aligned multimodal AI systems, with broad implications for applications in education, healthcare, and autonomous reasoning.
Deep Dive
Abstract
Process-supervised reward models serve as a fine-grained function that provides detailed step-wise feedback to model responses, facilitating effective selection of reasoning trajectories for complex tasks. Despite its advantages, evaluation on PRMs remains less explored, especially in the multimodal domain. To address this gap, this paper first benchmarks current vision large language models (VLLMs) as two types of reward models: output reward models (ORMs) and process reward models (PRMs) on multiple vision-language benchmarks, which reveal that neither ORM nor PRM consistently outperforms across all tasks, and superior VLLMs do not necessarily yield better rewarding performance. To further advance evaluation, we introduce ViLBench, a vision-language benchmark designed to require intensive process reward signals. Notably, OpenAI's GPT-4o with Chain-of-Thought (CoT) achieves only 27.3% accuracy, indicating the benchmark's challenge for current VLLMs. Lastly, we preliminarily showcase a promising pathway towards bridging the gap between general VLLMs and reward models -- by collecting 73.6K vision-language process reward data using an enhanced tree-search algorithm, our 3B model is able to achieve an average improvement of 3.3% over standard CoT and up to 2.5% compared to its untrained counterpart on ViLBench by selecting OpenAI o1's generations. We release the implementations at https://ucsc-vlaa.github.io/ViLBench with our code, model, and data.