SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
SoliReward enhances video generation reward models' robustness using single-item binary annotations and Hierarchical Progressive Query Attention.
Key Findings
Methodology
SoliReward employs single-item binary annotations and cross-prompt pairing strategy, combined with Hierarchical Progressive Query Attention to enhance feature aggregation. It introduces a modified BT loss that considers win-tie scenarios, regularizing score distributions for positive samples.
Key Results
- SoliReward demonstrated superior performance in benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, significantly improving post-training efficacy of video generation models.
- Compared to existing baselines, SoliReward achieved 78.48% accuracy on the physical plausibility task, outperforming VideoAlign's 71.60%.
- The BT-WT loss effectively mitigated reward hacking, enhancing the model's generalization capability.
Significance
This research addresses annotation noise and reward hacking issues by constructing more robust video reward models, enhancing video generation models' performance. It provides new insights for post-training alignment in the video generation field, with significant academic and industrial value.
Technical Contribution
SoliReward innovates in data annotation, training strategy, and model architecture. The proposed BT-WT loss and HPQA architecture offer new theoretical guarantees and open up new engineering possibilities.
Novelty
SoliReward is the first to combine single-item binary annotations with cross-prompt pairing strategy, significantly reducing annotation noise. The HPQA architecture offers unique advantages in feature aggregation.
Limitations
- In complex scenarios, single-item binary annotations may not capture all nuances, affecting model precision.
- The cross-prompt pairing strategy may perform poorly on specific datasets.
Future Work
Future research directions include exploring more complex annotation mechanisms and further optimizing the HPQA architecture to enhance model adaptability and robustness across different scenarios.
AI Executive Summary
Post-training alignment of video generation models is crucial for enhancing performance. However, existing methods face challenges with data annotation noise and reward hacking. SoliReward enhances the robustness of video generation reward models through single-item binary annotations and cross-prompt pairing strategy, combined with Hierarchical Progressive Query Attention. Experimental results show that SoliReward excels in benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, significantly improving post-training efficacy of video generation models. Despite these achievements, the method has limitations in certain complex scenarios. Future research will continue to optimize annotation mechanisms and model architecture to further enhance model adaptability and robustness.
Deep Analysis
Background
Recent advancements in video generation models, such as Sora 2, Veo 3, and Seedance, have shown remarkable capabilities in synthesizing high-fidelity, temporally coherent visual content. However, these models require post-training alignment techniques to enhance performance. Existing methods like DanceGRPO and LongChat-Video have made breakthroughs, but their core reward models face challenges with data annotation noise and reward hacking.
Core Problem
Building reward models that accurately capture complex video qualities faces challenges with data annotation noise and reward hacking. Existing pairwise preference learning methods suffer from significant noise due to annotation ambiguity, affecting reward model accuracy. Additionally, reward models are susceptible to reward hacking during post-training, causing generative models to deviate from human preferences.
Innovation
SoliReward significantly reduces annotation noise through single-item binary annotations and cross-prompt pairing strategy. The introduced BT-WT loss effectively mitigates reward hacking by using win-tie pairs. The Hierarchical Progressive Query Attention mechanism offers unique advantages in feature aggregation, enhancing reward signal robustness.
Methodology
- �� Use single-item binary annotations to reduce annotation noise.
- �� Implement cross-prompt pairing strategy to generate large-scale preference datasets.
- �� Introduce BT-WT loss to regularize score distributions for positive samples.
- �� Propose HPQA architecture to enhance feature aggregation.
Experiments
Experiments were conducted on benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, using video datasets generated by multiple SOTA models. By comparing with existing baselines, SoliReward's improvements in direct RM evaluation metrics and post-training efficacy were validated.
Results
SoliReward achieved 78.48% accuracy on the physical plausibility task, significantly outperforming VideoAlign's 71.60%. The BT-WT loss effectively mitigated reward hacking, enhancing the model's generalization capability. Experimental results show SoliReward's superior performance across multiple benchmarks.
Applications
SoliReward can be used to enhance the post-training alignment performance of video generation models, applicable in scenarios requiring high-fidelity, temporally coherent visual content generation, such as film production and virtual reality.
Limitations & Outlook
Despite SoliReward's superior performance across multiple benchmarks, single-item binary annotations may not capture all nuances in complex scenarios. Additionally, the cross-prompt pairing strategy may perform poorly on specific datasets. Future research will continue to optimize annotation mechanisms and model architecture.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. SoliReward is like a smart kitchen assistant that helps you choose the best ingredients with simple 'good/bad' labels. Then, it combines these ingredients into delicious dishes. This assistant also ensures that each dish is well-balanced, not overusing any particular ingredient just because it's especially good. In this way, it helps you make tastier and healthier meals.
ELI14 Explained like you're 14
Imagine you're playing a game, and SoliReward is your game assistant. It helps you pick the best gear and ensures you don't lose balance in the game because one piece of gear is too strong. This way, you can perform better in the game and score higher! Isn't that cool?
Glossary
Reward Model
A model used to quantify human preferences, helping generative models align with human expectations.
Used in post-training alignment of video generation models.
Annotation Noise
Data inaccuracies caused by annotator subjectivity and ambiguity.
A key factor affecting reward model accuracy.
Reward Hacking
A phenomenon where generative models exploit reward model vulnerabilities, deviating from true human preferences.
An issue to be addressed during post-training.
Hierarchical Progressive Query Attention
An architecture that enhances feature aggregation by fusing multi-layer features.
Used to improve the robustness of reward signals.
BT-WT Loss
A modified loss function that considers win-tie scenarios, regularizing score distributions for positive samples.
Used to mitigate reward hacking issues.
Open Questions Unanswered questions from this research
- 1 How to further reduce annotation noise in complex scenarios? Current methods may not be precise enough in some cases.
- 2 How does the cross-prompt pairing strategy perform on different datasets? More experiments are needed for validation.
Applications
Immediate Applications
Video Generation Optimization
SoliReward can be used to optimize the post-training alignment of video generation models, enhancing the quality and consistency of generated content.
Long-term Vision
Intelligent Film Production
By enhancing video generation model performance, SoliReward can drive the development of intelligent film production, reducing manual intervention and achieving automated production.
Abstract
Post-training alignment of video generation models with human preferences is a critical goal. Developing effective Reward Models (RMs) for this process faces significant methodological hurdles. Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise. Concurrently, the architectural design of VLM-based RMs, particularly their output mechanisms, remains underexplored. Furthermore, RM is susceptible to reward hacking in post-training. To mitigate these limitations, we propose SoliReward, a systematic framework for video RM training. Our framework first sources high-quality, cost-efficient data via single-item binary annotations, then constructs preference pairs using a cross-prompt pairing strategy. Architecturally, we employ a Hierarchical Progressive Query Attention mechanism to enhance feature aggregation. Finally, we introduce a modified BT loss that explicitly accommodates win-tie scenarios. This approach regularizes the RM's score distribution for positive samples, providing more nuanced preference signals to alleviate over-focus on a small number of top-scoring samples. Our approach is validated on benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, demonstrating improvements in direct RM evaluation metrics and in the efficacy of post-training on video generation models. Code and benchmark are available at https://github.com/lian700/SoliReward.