Expected free energy as an information constraint on the Bethe Lagrangian

Active inference has become a foundational framework for modeling autonomous agents capable of goal-directed behavior under uncertainty. Building on Friston’s free energy principle, the expected free energy (EFE) formalizes the trade-off between exploration and exploitation. However, traditional EFE lacks a structure compatible with message passing algorithms like belief propagation, limiting scalability in complex models. Recent efforts attempted to embed EFE into factor graphs or approximate message passing, but faced challenges due to its non-KL structure. Bethe free energy, a KL-based variational approximation, offers a promising avenue, yet integrating EFE’s expectation over unobserved outcomes remains problematic. This paper advances the field by proposing a novel formulation that supports message passing while explicitly controlling epistemic drives via information constraints.

The core challenge lies in reconciling the non-KL structure of EFE with the message passing framework of Bethe free energy. Existing methods either ignore the expectation over unobserved outcomes or treat epistemic drives as fixed priors, limiting flexibility and scalability. How to embed a dynamic, adjustable epistemic drive into the Bethe framework, ensuring that the agent actively seeks informative observations without sacrificing computational tractability, remains unresolved. Addressing this gap is crucial for developing scalable, adaptive active inference agents capable of operating in complex, real-world environments.

The main innovation is the integration of a mutual information-based inequality constraint into the Bethe free energy framework, enforced via a KKT multiplier. This allows the agent to dynamically regulate its epistemic drive, balancing goal pursuit and information gain. The approach transforms the traditional EFE into a form compatible with message passing, supported by a theoretical guarantee of convergence. By solving the dual problem, the method automatically tunes the epistemic influence without manual hyperparameter tuning, enabling flexible exploration strategies aligned with task demands. This unifies goal-directed and information-seeking behaviors within a single principled variational framework.

• �� Model the agent’s planning problem using a Bethe free energy functional, incorporating the model’s transition and emission factors.
• �� Introduce an inequality constraint on the mutual information between future observations and latent states/parameters, ensuring it exceeds the goal prior entropy.
• �� Formulate the Lagrangian with normalisation, marginalisation, form, and information constraints, introducing KKT multipliers for each.
• �� Derive the stationary policy by optimizing the Lagrangian, resulting in an exponential form modulating the prior with goal cross-entropy and mutual information weighted by the KKT multipliers.
• �� Analyze the KKT multipliers’ regimes (inactive, interior, saturated) to understand exploration-exploitation dynamics.
• �� Validate the approach through experiments on discrete tasks, comparing with standard EFE and Q-MDP, demonstrating improved exploration and learning efficiency.

The experimental setup involves three tasks: a canonical T-maze, an information grid, and a learning environment with unknown transitions. Each task tests the agent’s ability to balance exploration and exploitation under different conditions. The agents compared include the constrained Bethe agent, standard EFE, and Q-MDP. Metrics such as goal success rate, exploration frequency, and average steps are used. Hyperparameters like planning horizon and information floors are tuned to assess the influence of the information constraint. Results show that the constrained Bethe agent consistently outperforms baselines in exploration efficiency and goal achievement, validating the theoretical benefits of the information regulation mechanism.

The constrained Bethe agent achieves perfect goal reachability in the T-maze, with 100% success and minimal steps, surpassing Q-MDP’s 82%. In the grid task, it actively explores cues, reaching 84% success, while the unconstrained EFE only reaches 45%. During learning, the agent adapts its probing behavior based on the information constraint, efficiently reducing uncertainty about unknown transitions and rewards. These results confirm that the information constraint effectively guides exploration, leading to superior performance across diverse scenarios. The method’s ability to modulate epistemic drives dynamically is demonstrated by the regime shifts of the KKT multipliers, aligning exploration intensity with environmental demands.

This framework is applicable in autonomous robotics, where balancing exploration and exploitation is critical for navigation and task completion. It can enhance decision-making in uncertain environments, such as search-and-rescue, exploration, and adaptive control systems. The explicit control over epistemic drives allows for safer, more efficient exploration strategies, reducing unnecessary probing while ensuring sufficient information gathering. Additionally, it provides a theoretical basis for designing agents capable of self-regulated curiosity-driven learning, applicable in AI systems requiring autonomous adaptation and long-term planning.

The computational overhead of solving the dual KKT problem increases with state and action space complexity, limiting scalability. The reliance on accurate prior entropy estimates may lead to suboptimal policies if priors are mis-specified. The current implementation focuses on discrete models; extending to continuous, high-dimensional spaces poses significant challenges. Further research is needed to optimize the dual solver and incorporate function approximation techniques for real-world applications. Additionally, the approach’s performance in highly dynamic or adversarial environments remains to be tested.