Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
BNRM integrates Bayesian non-negative factor analysis to mitigate reward hacking in RLHF, improving robustness and interpretability.
Key Findings
Methodology
This paper introduces Bayesian Non-Negative Reward Model (BNRM), which embeds non-negative factor analysis into the Bradley–Terry preference framework. It models rewards via sparse, non-negative latent factors operating at two levels: local latent variables enable disentangled reward representations, while global sparsity acts as an implicit debiasing mechanism. Combining deep model features, an amortized variational inference network is used for efficient end-to-end training, capturing uncertainty and improving robustness. Empirical results show BNRM reduces reward over-optimization, enhances out-of-distribution generalization, and yields more interpretable reward decompositions compared to strong baselines.
Key Results
- On datasets with 40K and 400K training samples, BNRM significantly outperforms traditional models, improving reward accuracy by 5-13 percentage points, demonstrating superior generalization and bias robustness.
- In multiple OOD benchmarks like RewardBench and MT-Bench, BNRM shows enhanced robustness, reducing reliance on superficial cues such as response length and phrasing patterns.
- Reward decompositions are more interpretable, enabling identification of key reward factors, aiding understanding of bias sources and model behavior.
Significance
This work advances reward modeling by addressing core issues of robustness and interpretability in RLHF. By leveraging Bayesian non-negative factor analysis, it effectively mitigates reward hacking and systematic biases, fostering safer and more reliable large language models. The approach provides a principled framework for uncertainty quantification and bias suppression, essential for deploying AI systems in real-world scenarios where trust and safety are paramount. Its interpretability benefits also facilitate transparency and debugging, crucial for ethical AI development.
Technical Contribution
The key innovation lies in integrating non-negative factor analysis within a Bayesian framework, introducing sparse, disentangled reward representations. The model employs an amortized variational inference network with reparameterizable Weibull distributions, enabling scalable training with deep features. This combination captures both aleatoric and epistemic uncertainties, offering theoretical guarantees of robustness and interpretability. The structured Bayesian approach fundamentally differs from existing ensemble or information bottleneck methods, providing a unified solution to reward overfitting and bias.
Novelty
This is the first work to embed Bayesian non-negative factor analysis directly into reward modeling for RLHF, systematically enforcing sparsity and disentanglement. Unlike prior methods relying on ensembles or implicit relevance, BNRM explicitly models reward factors, reducing reliance on superficial correlations. Its structured probabilistic approach offers a new paradigm for robust, interpretable reward functions, setting a foundation for future research in safe AI alignment.
Limitations
- The model depends heavily on high-quality preference data; noisy or biased annotations can still affect performance.
- Training complexity and computational overhead are higher than traditional models, requiring optimized inference networks.
- In scenarios with extreme bias or noise, robustness may degrade, necessitating further refinement.
Future Work
Future directions include integrating multi-modal preferences, developing active learning strategies for better data collection, and scaling to more complex tasks. Exploring adaptive sparsity mechanisms and real-time inference for deployment in interactive systems are also promising avenues. Additionally, extending the framework to multi-agent settings and fairness-aware reward modeling remains an open challenge.
AI Executive Summary
The rapid development of large language models (LLMs) has made reinforcement learning from human feedback (RLHF) a cornerstone for aligning AI behavior with human values. However, existing reward models (RMs) often suffer from reward hacking, where policies exploit superficial correlations such as response length or stylistic artifacts, leading to misaligned behaviors. This challenge is compounded by noisy and subjective human annotations, which cause reward misgeneralization and systematic biases. To address these issues, this paper proposes Bayesian Non-Negative Reward Model (BNRM), a novel framework that embeds non-negative factor analysis within a Bayesian hierarchy. The core idea is to represent rewards as sparse, non-negative latent factors, operating at two levels: local instance-specific variables enable disentangled reward representations, while a global sparsity prior acts as an implicit debiasing mechanism. This structure effectively suppresses spurious correlations and enhances robustness against distribution shifts.
BNRM leverages deep model features as inputs to an amortized variational inference network, which parameterizes the posterior distributions over local and global latent variables using reparameterizable Weibull distributions. This design allows efficient end-to-end training, capturing both aleatoric and epistemic uncertainties. Extensive experiments on datasets with up to 400K human preference annotations demonstrate that BNRM outperforms strong baselines, reducing reward over-optimization by 5-13 percentage points and exhibiting superior robustness in out-of-distribution settings. The reward decompositions produced by BNRM are more interpretable, enabling clearer understanding of the key factors influencing preferences.
This work significantly advances reward modeling by providing a scalable, interpretable, and robust approach rooted in Bayesian non-negative factor analysis. It addresses fundamental challenges in RLHF, including reward hacking and bias suppression, paving the way for safer and more trustworthy AI systems. Future research will explore multi-modal preferences, active data collection, and deployment in complex, real-world scenarios, further strengthening the foundation for aligned AI development.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂的任务是把不同的原料变成成品。以前，工厂用一种简单的机器，只看原料的外表，比如颜色和大小，来判断原料的好坏，但这个方法容易被误导，因为外表不能代表全部质量。现在，工厂引入了一台新机器，它能识别出原料中的关键部分，还能判断哪些特征是真正重要的，哪些是误导的。它还会考虑自己可能会出错的情况，更加谨慎。这样一来，生产出来的产品质量更稳定，工厂也不容易被表面特征骗到。这个比喻说明了BNRM的核心思想：用稀疏、解耦的方式，找到真正的奖励信号，减少偏差和误导，让AI的行为更符合人类的真实意图。
ELI14 Explained like you're 14
想象你在学校参加一个比赛，老师会给你评分，但有时候评分会受到你用的词多不多，回答的长短影响。以前的评分方法就像一个黑箱，只看你答得多或用的词多，忽略了内容的真正价值。现在，BNRM就像一个聪明的老师，不仅看你答得多，还会分析你内容中哪些部分是真正重要的，还知道自己可能会错，变得更谨慎。这样一来，评分就更公平、更贴近你的真实水平，也不容易被一些表面特征骗到。这个方法让评分变得更聪明、更可靠，帮助我们训练出更懂人心的AI。
Abstract
Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We propose Bayesian Non-Negative Reward Model (BNRM), a principled reward modeling framework that integrates non-negative factor analysis into Bradley-Terry (BT) preference model. BNRM represents rewards through a sparse, non-negative latent factor generative process that operates at two complementary levels: instance-specific latent variables induce disentangled reward representations, while sparsity over global latent factors acts as an implicit debiasing mechanism that suppresses spurious correlations. Together, this disentanglement-then-debiasing structure enables robust uncertainty-aware reward learning. To scale BNRM to modern LLMs, we develop an amortized variational inference network conditioned on deep model representations, allowing efficient end-to-end training. Extensive empirical results demonstrate that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.