VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
VA-Judger employs chain-of-thought reasoning and multi-dimensional reinforcement learning to align video-audio generation with human preferences, outperforming metric-based rewards.
Key Findings
Methodology
This work constructs the VAPref-10K large-scale preference dataset, combining structured chain-of-thought reasoning with a multi-dimensional reward model VA-Judger. The training involves three stages: initial learning from clear quality gap pairs, human-verified hard pair rejection sampling, and dimension-wise reinforcement learning (GRPO). The model learns to interpret preferences through detailed explanations, avoiding reward hacking common with metric-based rewards. It leverages human annotations to produce dense, interpretable reward signals that guide joint video-audio content optimization.
Key Results
- VA-Judger achieves 65.91% overall accuracy in preference prediction, surpassing traditional metrics like VideoAlign and AudioBox, especially maintaining performance in out-of-domain tests with 62.35%. It effectively models human preferences across diverse scenarios.
- Using VA-Judger as a reward in post-training LTX-2 models significantly improves subjective content quality, with a 15% increase in user-rated scores. Its multi-dimensional reward enables finer discrimination of visual, auditory, synchronization, and semantic aspects.
- The multi-dimensional reinforcement learning approach results in models that better capture subtle cross-modal failures, leading to more coherent and human-aligned video-audio outputs, validated through extensive subjective and objective evaluations.
Significance
This research addresses the core challenge of aligning multimodal content generation with human preferences, overcoming the limitations of existing metric-based evaluation. By introducing a structured, interpretable reward model grounded in human feedback, it paves the way for more natural, faithful, and user-centric content synthesis. The dataset and benchmark set new standards for evaluating preference alignment in joint video-audio tasks, with broad implications for future research and industry applications in multimedia AI.
Technical Contribution
Key innovations include the development of VA-Judger, a chain-of-thought reasoning model that integrates preference explanation distillation and rejection sampling, and the application of multi-dimensional reinforcement learning (GRPO) to refine preferences at a granular level. The creation of VAPref-10K and VA-Judger-Bench provides valuable resources for systematic evaluation. These advances enable dense, interpretable, and human-aligned reward signals, significantly improving preference prediction and content quality in multimodal generation.
Novelty
This is the first work to explicitly model human preferences for joint video-audio generation using a structured, multi-dimensional reward framework. Unlike prior unimodal preference models, VA-Judger captures cross-modal coherence and semantic fidelity, setting a new paradigm for preference-based optimization in multimodal content synthesis. Its combination of chain-of-thought reasoning, human verification, and reinforcement learning represents a novel methodological integration.
Limitations
- Despite improvements, VA-Judger still struggles with highly complex or ambiguous content, where subtle preferences are difficult to discern even for humans, indicating room for richer data and modeling techniques.
- The training process relies heavily on human annotations and computationally intensive rejection sampling, limiting scalability and real-time deployment.
- Generalization to unseen models and out-of-distribution scenarios requires further validation, and current experiments are confined to specific datasets and architectures.
Future Work
Future directions include developing more scalable preference data collection methods, integrating self-supervised learning to reduce annotation costs, and extending the framework to real-time interactive content generation. Additionally, enhancing the model’s robustness across diverse domains and exploring explainability for end-users will be key to broader adoption.
AI Executive Summary
The rapid advancement of multimodal content generation has brought about the challenge of evaluating and optimizing joint video-audio synthesis in a manner aligned with human perception. Traditional metrics such as VideoAlign, AudioBox, and CLIP-based scores measure isolated aspects like visual fidelity, audio quality, or semantic similarity, but fail to capture the holistic coherence that humans intuitively perceive. This disconnect often leads to reward hacking, where models optimize for metric scores rather than genuine content quality.
To address this, the authors introduce VA-Judger, a novel preference modeling framework that employs structured chain-of-thought reasoning and multi-dimensional reinforcement learning. Central to their approach is the creation of VAPref-10K, a large-scale dataset of human preferences derived from real-world video-audio clips, annotated through pairwise comparisons with detailed explanations. This dataset enables training VA-Judger to interpret complex cross-modal preferences, moving beyond simple binary labels.
The model’s training involves three stages: starting with easy pairs with clear quality gaps, then refining on hard pairs verified through human rejection sampling, and finally employing GRPO to optimize across multiple quality dimensions such as visual fidelity, audio quality, synchronization, and semantic faithfulness. This multi-stage process ensures the model learns to reason jointly over audio and video, producing dense, interpretable reward signals.
Experimental results demonstrate that VA-Judger outperforms traditional metrics in predicting human preferences, achieving over 65% accuracy in preference prediction and significantly improving subjective content quality when used as a reward in post-training generative models. Its ability to capture subtle cross-modal failures and align with human judgments marks a substantial step forward in multimodal content synthesis.
This work has broad implications for both academia and industry, enabling more human-aligned, faithful, and engaging multimedia content. Nonetheless, challenges remain in scaling the approach, reducing computational costs, and enhancing generalization. Future work will focus on more efficient data collection, real-time applications, and expanding the framework’s robustness across diverse scenarios, ultimately pushing multimodal AI closer to human-like understanding and creation.
Deep Dive
Abstract
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.