Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

Preference-based reinforcement learning (PbRL) learns reward functions from pairwise comparisons of trajectory segments, making it attractive when humans cannot provide dense numeric rewards. Representative methods such as PEBBLE, SURF, and RUNE improve efficiency through unsupervised exploration, semi-supervised learning, data augmentation, and uncertainty-aware querying. Separately, sparse training has become a major tool for reducing parameter count and improving robustness. Dynamic sparse training (DST) methods such as SET and RigL start from sparse networks and evolve topology during learning, while L1 regularization and DropConnect offer alternative sparsification mechanisms. Prior noisy-RL work showed that sparsifying input layers can help filter irrelevant state features, but the intersection of DST and PbRL had not been explored.

The paper studies a precise and difficult problem: in PbRL, can an agent learn a reliable reward model and policy when the state contains a large fraction of irrelevant Gaussian noise features? In the ENE setting, the observation vector is expanded by concatenating random features sampled i.i.d. from N(0,1), with noise fractions as high as 90%. This creates a confounding feature-selection problem: the agent receives only preference labels, yet must infer which inputs matter for reward and control. The challenge is compounded by sparse preference feedback, making standard dense models prone to overfitting distractors and learning brittle policies.

R2N introduces three linked innovations.

• �� Reward-model DST: the reward learner itself is sparsified, not just the policy network. This is necessary because in PbRL the reward estimator is the bottleneck that converts preferences into a learning signal.
• �� RigL-based rewiring: unlike SET, which regrows random connections, RigL grows weights at locations with highest gradient magnitude. The paper argues this is especially appropriate for supervised preference fitting.
• �� Dual-module sparsification: the actor and critic input layers are also sparsified, so both reward inference and policy optimization can suppress noisy features. This differs from prior noisy-RL work that only modified the control side, and from static sparsity methods that keep topology fixed throughout training.

• �� Problem setup: each environment is converted to an ENE by appending task-irrelevant Gaussian noise features, producing high-dimensional observations with 70% or 90% noise.
• �� Preference collection: a simulated teacher compares two trajectory segments (σ0, σ1) and returns a label y∈{0,0.5,1}, stored as tuples (σ0, σ1, y).
• �� Reward learning: the reward model ˆr_θ is trained with the Bradley-Terry preference predictor P_θ and binary cross-entropy L_CE. This learns a scalar reward surrogate from pairwise comparisons.
• �� DST initialization: the reward input layer is randomly pruned to sparsity s_R=80%.
• �� Topology updates: every ΔT_R=100 gradient steps, the smallest-magnitude active weights are pruned at a fraction d_f^R=0.2, then the same number of new weights are grown using RigL.
• �� RL control: the same sparse-training schedule is applied to actor and critic input layers while the policy is optimized with SAC.
• �� Evaluation: all methods train for 1 million timesteps, with offline greedy evaluation every 5,000 timesteps on the ground-truth reward.
• �� Statistics: final return and learning efficiency (AUC) are compared using one-tailed Welch’s t-tests at p<0.05.

The benchmark is the DeepMind Control Suite (DMControl) with five tasks: Cartpole-swingup, Walker-walk, Quadruped-walk, Cheetah-run, and Humanoid-stand. The noise fractions are 0.90 for Cartpole-swingup, Walker-walk, and Cheetah-run, and 0.70 for Quadruped-walk and Humanoid-stand, chosen so the ENE state sizes remain comparable across tasks. The resulting observation dimensions are expanded from 5→51, 17→171, 24→241, 68→260, and 67→224. Baselines include SET, Static Sparse Training, L1 Regularization, and DropConnect for sparse-training comparisons, plus PEBBLE, SURF, and RUNE when testing R2N as a plug-in improvement. The RL backend is SAC throughout. Preference budgets are increased to 400, 1000, 4000, 4000, and 10000 depending on task difficulty.

The strongest result is breadth: R2N improves PbRL across all 15 tested baseline-environment combinations when inserted into PEBBLE, SURF, and RUNE, with significant gains in learning efficiency (p≤0.021) and final return (p≤0.006). On the sparse-training side, R2N-PEBBLE is the only method that stays consistently strong across all five noisy tasks; it significantly outperforms L1 Regularization and DropConnect in 4/5 environments on learning efficiency (p≤0.034). The paper also reports significant gains in Humanoid-stand versus both Static-PEBBLE and SET-PEBBLE on AUC and final return (p≤0.024), and in Cheetah-run versus Static-PEBBLE on AUC (p≤0.018). Ablations show that sparsifying only the reward model or only the policy side is not enough: the full dual-module design is needed.

Household robotics: a cleaning robot can use preference feedback to learn a cleaning style while ignoring irrelevant sensory clutter such as colors, sounds, or temperatures. The benefit is lower sample complexity and better robustness without manual feature engineering.

Interactive RL systems: PbRL frameworks like PEBBLE, SURF, or RUNE can be upgraded with R2N as a drop-in sparsity module. This is valuable for research labs and industrial teams that already rely on SAC-based control stacks and want better robustness under noisy observations.

The method assumes that many observed features are irrelevant noise and that relevance can be discovered by sparse rewiring. If distractors are highly correlated with the task or if the true signal is extremely weak, dynamic sparsity may still struggle. Moreover, the current experiments do not measure human preference elicitation cost, latency, or safety implications in real deployments. More work is needed to quantify scalability and robustness under non-Gaussian, structured, or adversarial noise.