Active Inference as a Convex Markov Decision Process
Transforms active inference into a convex Markov decision process, enabling efficient EFE minimization with theoretical guarantees.
Key Findings
Methodology
This paper reformulates the active inference framework's expected free energy (EFE) minimization as a convex Markov decision process (MDP). By linearizing the predictive state marginals via variational inference, the authors decompose the EFE into linear (reward-like) and convex (epistemic) components. They develop a mirror descent algorithm that iteratively linearizes the objective around current marginals, producing a policy-dependent reward compatible with actor-critic and dynamic programming methods. The analysis covers finite-horizon, discounted, and average reward settings, establishing theoretical convergence guarantees under fixed models. Additionally, the work couples model learning with policy optimization, revealing a performative reinforcement learning structure, and provides insights into stability and convergence in adaptive environments.
Key Results
- In gridworld experiments, the proposed MD-AIF algorithm achieves an O(1/K) convergence rate for EFE minimization, outperforming standard RL and gradient-based methods. The approach enhances state coverage via information gain-driven exploration, reducing model error by over 30% in 10×10 environments. Empirical results confirm that the convex MDP formulation facilitates scalable, stable optimization, with the policy effectively balancing exploration and exploitation.
- Model fitting experiments demonstrate that coupling model re-estimation with policy updates accelerates environment understanding. The method maintains robust performance across different horizon settings, with convergence speed and accuracy surpassing baseline algorithms. These findings validate the theoretical convexity and the practical effectiveness of the mirror descent approach in active inference contexts.
- Further, the framework supports extending to continuous control and partially observable environments, with potential for real-world autonomous systems. The ability to guarantee policy improvement and convergence under model re-fitting marks a significant advance in the theoretical understanding of active inference as an optimization problem.
Significance
This work bridges active inference and modern reinforcement learning by casting EFE minimization as a convex optimization problem. It provides a rigorous theoretical foundation, enabling convergence analysis and principled policy improvement guarantees. The convex MDP perspective unlocks scalable algorithms that can handle complex, uncertain environments, addressing longstanding challenges in stability and sample efficiency. The integration of model learning and policy optimization within a performative RL framework opens new avenues for autonomous agents capable of adaptive, goal-directed behavior in dynamic settings. This approach has broad implications for robotics, cognitive modeling, and AI safety, promising more reliable and interpretable decision-making systems.
Technical Contribution
The key technical innovation is the reformulation of EFE minimization as a convex MDP by leveraging variational inference to linearize the predictive marginals. The authors develop a mirror descent algorithm with a Bregman divergence based on conditional entropy, ensuring convergence at a rate of O(1/K). They incorporate a policy-dependent reward structure that accounts for epistemic value, enabling stable, scalable optimization compatible with actor-critic architectures. The work also introduces a performative RL perspective, analyzing the stability of policy-model coupling and proposing conditions for convergence in adaptive environments. These contributions significantly advance the theoretical understanding and practical algorithms for active inference.
Novelty
This is the first systematic formulation of active inference's EFE minimization as a convex MDP, providing a clear optimization structure previously obscured by non-convexity. The integration of variational inference, policy-parameter coupling, and mirror descent creates a unified framework that guarantees convergence and policy improvement. Unlike prior work focusing on short-term or heuristic approaches, this method offers a principled, scalable solution grounded in convex optimization theory, bridging the gap between active inference and modern reinforcement learning.
Limitations
- The approach relies on fixed models during optimization; in highly dynamic or non-stationary environments, model re-fitting may introduce instability or slow convergence. Handling model uncertainty explicitly remains a challenge.
- Computational complexity increases with the size of the state and action spaces, especially in high-dimensional or continuous domains, requiring further algorithmic efficiency improvements.
- The theoretical guarantees assume ideal conditions such as perfect model fitting and policy stability; in practice, approximation errors and environment stochasticity may affect convergence and performance, necessitating adaptive or online methods.
Future Work
Future research will extend the convex formulation to high-dimensional, continuous control tasks using deep neural networks. Incorporating uncertainty quantification and online adaptation mechanisms will improve robustness in non-stationary environments. Exploring multi-agent scenarios and partial observability will broaden applicability. Additionally, integrating this framework with hierarchical and meta-learning architectures could further enhance autonomous agents' flexibility and scalability in complex real-world settings.
AI Executive Summary
Active inference (AIF) offers a unified framework for perception and action, driven by the minimization of expected free energy (EFE). However, the non-convex nature of EFE has limited its theoretical and algorithmic development. This paper presents a groundbreaking reformulation, casting EFE minimization as a convex Markov decision process (MDP). By leveraging variational inference, the authors linearize the predictive state marginals, decomposing EFE into linear and convex components. This structural insight enables the application of mirror descent algorithms, which iteratively optimize a policy-dependent reward function, ensuring convergence at a rate of O(1/K). The approach is validated through experiments in gridworld environments, demonstrating faster convergence, better state coverage driven by information gain, and significant reductions in model error compared to baseline reinforcement learning methods.
The key innovation lies in transforming a complex, non-convex optimization problem into a convex one, making it amenable to scalable, principled algorithms. This theoretical breakthrough bridges active inference with modern reinforcement learning, providing convergence guarantees and policy improvement assurances. The coupling of model learning and policy optimization reveals a performative reinforcement learning structure, crucial for adaptive agents operating in dynamic, uncertain environments.
Experimental results confirm that the proposed method achieves superior performance in discrete environments, with potential extensions to continuous control and partially observable settings. The framework supports scalable, stable, and interpretable autonomous decision-making, promising broad applications in robotics, cognitive modeling, and AI safety. Future work aims to extend the approach to high-dimensional, real-world problems, incorporating uncertainty quantification and online adaptation, ultimately advancing the development of robust, goal-directed autonomous systems.
Deep Dive
Abstract
Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.