State Representation Learning for Control: An Overview

Robotics and control depend on compact, expressive representations of sensor data. Historically, these features were hand-crafted, but deep learning opened the door to learning them from experience. SRL focuses on learning a low-dimensional state from observations, actions, and sometimes rewards, with the aim of preserving task-relevant information while discarding nuisance factors. In the reinforcement learning view used by the paper, a state is not just a compressed observation; it is the information needed to choose an action and predict what will happen next.

The core problem is to learn a state representation without access to the true latent state. This is hard because observations are high-dimensional, noisy, and often partially observable; actions can have multiple effects; and real-world interaction is expensive. If the learned representation is not Markovian or not aligned with controllable factors, downstream RL may require more data, converge slowly, or fail to generalize. The challenge is therefore both statistical and behavioral.

The paper’s main innovation is a control-centric taxonomy of SRL objectives. First, it distinguishes reconstruction-based learning, where an encoder-decoder pair preserves information through compression. Second, it groups forward-model methods that learn latent dynamics, often under linearity assumptions. Third, it covers inverse-model learning, which encourages states to encode action-relevant information. Fourth, it elevates priors such as simplicity, locality, temporal continuity, and controllability into first-class objectives. This organization makes it easier to match a method to a task.

• �� Formal setup: define environment E, observation ot, action at, optional reward rt, and latent state st=φ(o1:t), with learning performed without true-state supervision.
• �� Reconstruction family: PCA and auto-encoders minimize reconstruction error of ot or ot+1 via encoder-decoder architectures; variants add sparsity, denoising, or dimensionality constraints.
• �� Forward-model family: encode ot→st, then predict ˆst+1 with f(st,at); many works assume linear latent dynamics, e.g., ˆst+1=W·ˆst+U·at+V.
• �� Variational variants: E2C, DVBF, and related models treat next state as a distribution and train with KL divergence, improving uncertainty handling.
• �� Inverse models: g(st,st+1)→at forces the latent code to retain what is needed to infer the action that caused the transition.
• �� Priors: temporal continuity, locality, simplicity, and controllability are encoded as loss terms; one explicit form is Controllability_i = e^{-cov(at,i, st+1,i)}, which rewards alignment between action dimensions and state changes.

Because this is a survey, it does not present a single benchmark suite. Instead, it synthesizes reported evaluations across prior work. The paper cites experiments on Super Mario, Swimmers, Mountain Car, PR2 robot manipulation, a real pole-balance task, navigation tasks, and VizDoom. Compared baselines include raw-observation RL, reconstruction-only AEs, PILCO, and methods without dynamical constraints. Reported metrics in the literature are task performance, convergence speed, latent dimension, and qualitative interpretability.

Across the reviewed literature, the common result is that latent states become much more useful once time and action are built into the learning objective. PCA with 3–4 dimensions already accelerates learning in some game and physics tasks. Simple AEs can compress real images into 2-D states that support control. More importantly, dynamic methods such as E2C and DVBF outperform pure reconstruction approaches when the environment has complex nonlinear dynamics. ICM adds that forward-model error can be repurposed as intrinsic reward, improving exploration under sparse external reward.

The most immediate applications are robotic manipulation, navigation, exploration, and multimodal perception. SRL is particularly attractive when camera, depth, tactile, or audio streams are high-dimensional but the task itself depends on only a few physical variables. In industry, SRL can reduce training time and data requirements for RL-based controllers. In research, it helps disentangle what parts of the world are controllable, predictable, and relevant to policy learning.

The survey also makes clear that SRL is not solved. Many methods assume simple latent dynamics, local smoothness, or a direct correspondence between actions and state changes, which may fail in highly nonlinear or partially observed settings. Cross-paper comparison is difficult because benchmarks and evaluation criteria differ widely. Future progress will require better metrics for disentanglement, stronger transfer tests, and more stable learning on real robots rather than only in simulation.