GptGet

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

TL;DR

InternLM-XComposer2.5-Reward enhances LVLMs' generation quality with a multi-modal reward model, achieving 70% accuracy.

cs.CV 🔴 Advanced 2025-01-22 41 views
Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Ziyu Liu Shengyuan Ding Shenxi Wu Yubo Ma Haodong Duan Wenwei Zhang Kai Chen Dahua Lin Jiaqi Wang
AI Reader Arxiv Page Download PDF
multi-modal reward model reinforcement learning vision-language models open-source

Key Findings

Methodology

IXC-2.5-Reward trains a reward model using a multi-modal preference dataset, integrating Proximal Policy Optimization (PPO) to enhance LVLMs' performance on text, image, and video inputs. The model employs a vision encoder and MLP projector, using GPT-4o for preference judgments.

Key Results

  • IXC-2.5-Reward achieved 70.0% macro accuracy on VL-RewardBench, surpassing all open-source models.
  • On text reward model benchmarks, IXC-2.5-Reward achieved an average score of 88.6%, showing strong performance.
  • PPO-trained IXC-2.5-Chat excelled in multi-modal dialogue and instruction-following tasks.

Significance

This research fills the gap in multi-modal reward models for LVLMs, promoting academic and industrial research in enhancing multi-modal generation quality through open-source means. It addresses the current scarcity of multi-modal data, providing an effective training and evaluation method.

Technical Contribution

IXC-2.5-Reward unifies reward scoring across multi-modal inputs, offering new theoretical guarantees and engineering possibilities. It outperforms existing methods in multi-modal tasks, especially in handling complex reasoning tasks.

Novelty

This is the first publicly available multi-modal reward model capable of unified reward scoring across text, image, and video inputs, showing significant innovation compared to existing uni-modal methods.

Limitations

  • The model may underperform with very long text inputs.
  • Data scarcity in specific domains may affect the model's generalization.
  • Further optimization is needed to reduce computational costs.

Future Work

Future research directions include expanding the scale and diversity of multi-modal datasets and optimizing computational efficiency. Exploring more application scenarios and improving model robustness are also important research directions.

AI Executive Summary

Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they sometimes generate inaccurate outputs. Existing reward models (RMs) can improve generation quality through reinforcement learning or test-time scaling strategies, but the scarcity of multi-modal RMs limits their application. InternLM-XComposer2.5-Reward (IXC-2.5-Reward) addresses this issue by constructing a high-quality multi-modal preference dataset. The model performs excellently on the latest multi-modal reward model benchmarks and shows competitive performance on text reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: providing a supervisory signal for RL training, selecting the best response, and cleaning existing image and video instruction-tuning training data.

IXC-2.5-Reward, combined with Proximal Policy Optimization (PPO), generates IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue. The model achieves 70.0% macro accuracy on the multi-modal VL-RewardBench, surpassing all previous generative RMs, including Gemini-1.5-Pro and GPT-4o. Even on uni-modal (text) RM benchmarks, IXC-2.5-Reward demonstrates good results, with an average score of 88.6%.

By open-sourcing all model weights and training recipes, IXC-2.5-Reward provides a foundation for reproducibility and further research in multi-modal reward models. Future research directions include expanding the scale and diversity of multi-modal datasets and optimizing computational efficiency.

Deep Dive

Abstract

Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward

cs.CV cs.CL