Agent-Centric Visual Reinforcement Learning under Dynamic Perturbations

Visual reinforcement learning has made significant progress in recent years, particularly in simulated benchmarks and robotic manipulation. However, existing methods perform poorly under dynamic visual perturbations, leading to severe performance degradation. Traditional visual reinforcement learning methods often rely on reconstruction-based objectives, which inevitably entangle perturbation artifacts into latent representations, affecting the agent's decision-making ability. To address this issue, researchers have proposed various methods, including data augmentation and domain randomization, but these methods still have limitations when dealing with non-stationary perturbations. To systematically study the performance of visual reinforcement learning under dynamic perturbations, this paper introduces the Visual Degraded Control Suite (VDCS), a benchmark extending the DeepMind Control Suite to simulate real-world non-stationary perturbations.

The robustness of visual reinforcement learning under dynamic perturbations is a pressing issue. Existing methods perform poorly under non-stationary perturbations, leading to severe performance degradation. Specifically, model-free methods learn policies directly from pixel observations; when foregrounds are physically occluded by rain, snow, and haze, or their textures are altered, the encoder confuses task-relevant states with perturbation artifacts, leading to policy confusion. Model-based methods face a more severe failure mode, as world models like DreamerV3 are trained with reconstruction objectives that incentivize the latent representation to encode corruption-specific features. Under dynamically switching perturbations, the world model must simultaneously represent multiple corruption patterns, contaminating the latent state and severely degrading the imagined rollouts used for policy optimization.

The ACO-MoE method proposed in this paper introduces a Mixture-of-Experts approach focusing on restoring perturbed visual inputs and extracting task-relevant foregrounds. ACO-MoE leverages unique agent-centric restoration experts, achieving restoration and foreground extraction without requiring prior perturbation labels. Through information-theoretic analysis, the paper proves the effectiveness of foreground extraction as an information bottleneck surrogate, successfully decoupling task-relevant information from dynamic perturbations. Additionally, ACO-MoE demonstrates plug-and-play compatibility with existing models, supporting seamless integration.

The core steps of the ACO-MoE method include:

• �� Introducing the Visual Degraded Control Suite (VDCS) to simulate real-world non-stationary perturbations and systematically evaluate the robustness of visual reinforcement learning.
• �� Proving through information-theoretic analysis that reconstruction-based objectives inevitably entangle perturbation artifacts into latent representations.
• �� Proposing Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), which achieves restoration from corruptions and task-relevant foreground extraction through agent-centric restoration experts.
• �� Decoupling perception from perturbation before the RL agent processes the observation, enhancing the robustness of visual reinforcement learning under perturbations.

The experimental design includes evaluating the performance of ACO-MoE on the VDCS benchmark, comparing its performance with existing baseline methods. The VDCS benchmark extends the DeepMind Control Suite to simulate real-world non-stationary perturbations, including physical occlusions and texture changes such as rain, snow, and haze. The experiments also evaluate ACO-MoE's performance on the DMControl Generalization benchmark under random-color and video-background perturbations. Key hyperparameters include the number and severity of perturbation modes, and the experiments validate ACO-MoE's robustness through multiple repetitions.

Experimental results show that ACO-MoE recovers 95.3% of clean performance on the VDCS benchmark, significantly outperforming other baseline methods. On the DMControl Generalization benchmark, ACO-MoE achieves state-of-the-art results under random-color and video-background perturbations. Additionally, through information-theoretic analysis, it is proven that reconstruction-based objectives inevitably entangle perturbation artifacts into latent representations, while ACO-MoE effectively eliminates this entanglement through foreground extraction.

ACO-MoE has broad application prospects in fields such as autonomous driving and robotic manipulation. In these scenarios, agents need to make decisions in dynamically changing environments, and ACO-MoE enhances visual perception robustness, improving agent performance in complex environments. Furthermore, ACO-MoE's plug-and-play compatibility allows seamless integration with existing models, further expanding its application scope.

Despite ACO-MoE's excellent performance in most tasks, it underperforms in certain fine-grained tasks, such as the finger_spin task, likely due to limitations of the underlying model rather than preprocessing failures. Additionally, the performance of this method in high computational cost environments has not been fully verified, potentially posing efficiency issues. Future research directions include further optimizing the computational efficiency of ACO-MoE, exploring its application in more real-world scenarios, and integrating it with other reinforcement learning algorithms to enhance its generality and adaptability.