Subspace Inference Enables Efficient Active Reward Learning from Preferences

Recent years have seen rapid development in reinforcement learning from human feedback (RLHF), with early works like Christiano et al. (2017) and Ouyang et al. (2022) establishing the importance of preference-based reward modeling. Preference feedback, being easier to obtain than demonstrations, has driven interest in active learning strategies to reduce annotation costs. Bayesian methods, including variational inference and Hamiltonian Monte Carlo, have been employed for uncertainty quantification in low-dimensional reward models, but scaling these techniques to neural networks remains difficult due to high computational costs. Ensemble and Dropout methods have been popular approximations, yet their effectiveness and efficiency are debated. Recent advances in approximate Bayesian inference (Daxberger et al., 2024; Shen et al., 2024) have begun to address scalability, but fully leveraging neural network models for active reward learning still faces significant hurdles.

The core challenge lies in efficiently estimating uncertainty in high-dimensional neural network reward models during active preference learning. Traditional Bayesian approaches like HMC and variational inference are computationally prohibitive at scale. Ensemble and Dropout methods, while more scalable, often compromise on accuracy and calibration. This bottleneck hampers sample efficiency, requiring excessive preference queries to achieve reliable reward models. Moreover, existing methods struggle with the computational cost of sampling multiple models or maintaining calibration in large models, limiting their practical deployment in real-world RLHF applications. Addressing this bottleneck is crucial for advancing scalable, data-efficient preference-based reinforcement learning.

This paper introduces PreferenceEKF, a pioneering framework that applies subspace extended Kalman filtering to neural reward models. Key innovations include:

• �� Constructing a low-dimensional subspace via SVD or random projections, enabling scalable Bayesian inference.
• �� Applying EKF within this subspace to perform recursive posterior updates, reducing computational complexity from O(|θ|^2) to manageable levels.
• �� Integrating a linear parameter mapping to connect subspace parameters with the full neural network, preserving uncertainty estimates.
• �� Using the Bayesian filtering framework to incorporate preference data sequentially, guided by information gain for active query selection.
• �� Demonstrating that this approach maintains high-quality uncertainty estimates while significantly reducing inference costs, supporting large-scale neural networks in active reward learning.

• �� Formulate preference reward learning with the Bradley-Terry model, defining the likelihood of preference labels.
• �� Construct a low-dimensional subspace by applying SVD to initial SGD iterates or via random projection, obtaining a projection matrix A.
• �� Model neural network parameters as hidden states with Gaussian dynamics, assuming linear Gaussian noise.
• �� Use the measurement model based on the BT likelihood, approximated via the neural network reward function.
• �� Implement the EKF in the subspace: initialize Gaussian prior, perform recursive updates with each preference query.
• �� After each update, sample from the Gaussian posterior in the subspace, project back to the full space, and perform forward passes.
• �� Use the sampled models to compute acquisition functions like information gain, guiding active query selection.
• �� Update the posterior sequentially, enabling efficient, scalable active learning without retraining multiple models.

Experiments conducted on D4RL and V-D4RL benchmarks involved synthetic preference queries generated from ground-truth rewards. The evaluation metrics included log-likelihood on held-out test queries, sample efficiency, and offline RL policy performance. Baselines compared were DeepEnsemble, Dropout, Laplace approximation, and LLMCMC. Hyperparameters such as subspace dimension (|z|=200), query budget (B=60), and SGD iterations (w=200) were fixed across tasks. Ablation studies examined subspace construction methods and the impact of initial datasets. Results showed PreferenceEKF achieved higher log-likelihood, faster training, and better calibration, with reward models leading to improved offline RL policies.

PreferenceEKF consistently outperformed baselines, with over 20% higher log-likelihood and 3x reduction in training time. It demonstrated superior calibration and sample efficiency across tasks. In offline RL, policies trained with PreferenceEKF rewards achieved 5% higher returns. The subspace approach enabled scalable sampling, supporting large neural networks. Ablation studies confirmed the importance of subspace construction strategies. Overall, the method proved effective in both preference modeling and policy optimization, validating its practical advantages.

This approach can be directly applied to personalized recommendation systems, robotic learning, and interactive AI, where preference data is sparse and costly. Its scalability makes it suitable for large-scale models and complex environments. Future applications include multimodal preference integration, real-time adaptive systems, and human-AI collaboration platforms, aiming to make AI systems more aligned with human values and preferences.

The Gaussian assumptions in EKF may limit modeling multi-modal or highly nonlinear preference distributions. Subspace construction may restrict expressiveness if the chosen dimension is too small. The current framework primarily handles a single annotator; extending to multiple sources requires further development. Computational costs, although reduced, still pose challenges for extremely large models or real-time applications. Future work should address these limitations by exploring non-parametric filtering and adaptive subspace strategies.