Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Exploration is a fundamental prerequisite for autonomous agents to acquire meaningful behaviors, especially in sparse-reward, long-horizon tasks common in robotics and embodied AI. Early psychological studies, such as Edward Tolman's latent learning experiments, demonstrated that animals can learn complex environmental representations without explicit rewards, highlighting intrinsic motivation as a key driver. In reinforcement learning, curiosity-driven methods operationalize this by providing intrinsic rewards based on prediction errors of learned forward models, encouraging agents to seek novel and uncertain states.

Classic approaches like the Intrinsic Curiosity Module (ICM) use forward dynamics models to estimate prediction errors as novelty signals. While effective in simple or simulated environments, these methods struggle in complex, photorealistic 3D scenes due to local loop traps and reward deception. Agents tend to revisit forgotten states repeatedly, receiving spurious novelty rewards, which impedes efficient exploration.

Moreover, many existing methods rely on short-term memory or statistical priors rather than persistent spatial representations, limiting their ability to plan long-term exploratory trajectories. Some approaches incorporate explicit geometric maps to aid navigation, but these often abstract away semantic richness and restrict end-to-end learning and generalization. Advances in 3D reconstruction, particularly online 3D Gaussian Splatting (3DGS), offer spatially consistent and dynamically updatable scene representations, presenting new opportunities for persistent world modeling.

This paper builds upon these insights to develop a curiosity-driven exploration framework that integrates persistent 3D reconstruction with a Transformer policy encoding long-term episodic context, aiming to overcome the limitations of prior work and enable scalable exploration in realistic 3D environments.

The core problem addressed is enabling effective curiosity-driven exploration in complex, photorealistic 3D environments under sparse reward conditions. Key challenges include:

1. Lack of persistent spatial world models: Without a continuously updated and consistent representation of the environment, agents suffer from spatial forgetting, leading to repeated visits to the same locations and false novelty rewards.

2. Insufficient episodic memory in policies: Short-term or no memory policies cannot leverage historical trajectories to plan efficient exploration paths, limiting long-horizon exploration capabilities.

3. Reliance on privileged sensors: Many methods depend on depth sensors or explicit maps, complicating deployment and reducing flexibility.

4. Exploration collapse: Sparse rewards often cause policies to degenerate into repetitive or random behaviors, hindering sustained discovery of novel states.

Addressing these bottlenecks is critical for advancing autonomous agents capable of robust, scalable exploration and learning in real-world-like 3D settings.

This work introduces several core innovations:

1. Persistent 3D World Model: Employing online 3D Gaussian Splatting (3DGS) as a persistent, dynamically updated world model provides spatial consistency and reliable intrinsic rewards based on prediction errors, overcoming spatial forgetting issues inherent in prior statistical models like ICM.

2. Transformer-based Episodic Policy: Designing a policy network that processes sequences of RGB observations and actions using causal temporal self-attention and a global linear attention memory module enables encoding of long-term episodic context, facilitating complex exploratory behaviors such as backtracking and branch discovery.

3. Pure RGB Deployment: While training leverages depth and pose data to build the 3DGS model, the policy requires only RGB inputs at test time, enhancing deployment flexibility and generalization.

4. Exploration Diversity via Random Action Mixing: Integrating a random action mixing strategy within PPO training maintains exploration diversity and prevents policy collapse in sparse-reward environments.

Together, these innovations address the dual challenges of spatial persistence and episodic context, establishing a new paradigm for curiosity-driven exploration in photorealistic 3D environments.

• �� Problem Setup: The agent interacts with a static 3D environment, executing discrete actions to move and receiving RGB observations. Training uses privileged inputs (depth, camera pose) to build the world model; testing relies solely on RGB.

• �� Persistent World Model: An online 3D Gaussian Splatting (3DGS) model Gt is constructed incrementally from RGB-D frames and camera poses. Each pixel contributes a Gaussian primitive, optimized periodically for reconstruction quality and densified.

• �� Intrinsic Reward Computation: At each timestep, the 3DGS model renders a predicted view ˆIt+1 from the agent's new pose. The prediction error et is computed as the mean squared difference between low-pass filtered and downsampled predicted and actual RGB images. A binary curiosity reward rcur_t is assigned based on whether et exceeds a threshold τ, rewarding novel views and penalizing redundant ones.

• �� Policy Network Architecture: Inputs are sequences of paired RGB images Ii and previous actions ai-1, where actions are encoded as Plücker ray images representing intended camera transformations.

• �� Image Encoding: Each RGB-action pair is encoded via a convolutional encoder and enriched with DINOv2 self-supervised visual features. A learnable query token cross-attends to patch tokens and DINOv2 features to produce frame tokens.

• �� Temporal Modeling: Frame tokens pass through sliding-window causal self-attention layers capturing local temporal context, interleaved with a global linear attention memory module that maintains a running memory state, enabling long-term episodic context encoding.

• �� Output Heads: Actor and critic heads produce action probability distributions πθ and value estimates Vθ respectively. Actions are sampled from πθ during training and testing.

• �� Training Procedure: The policy is optimized using PPO with the curiosity reward as the sole signal. A random action mixing mechanism samples actions from a mixture of the learned policy and a uniform random distribution with annealed mixing coefficient β to maintain exploration diversity.

• �� Deployment: The agent operates purely on RGB inputs without access to depth or pose information, relying on learned episodic memory and policy for navigation.

• �� Datasets and Environment: Training is conducted on the Habitat Matterport 3D (HM3D) training set comprising 800 indoor scenes. Evaluation occurs on the HM3D validation set (100 scenes), Gibson dataset (86 scenes), and two AI-generated environments (Hobbit World and Spaceship) for zero-shot generalization.

• �� Baselines: Compared against map-based RL methods including Active Neural SLAM (ANS) and Occupancy Anticipation (OccAnt) variants with RGB, depth, and RGB-D inputs.

• �� Metrics: Exploration is quantified via 3D scene completeness (percentage of ground truth mesh points observed within a threshold) at 256, 512, and 1024 steps, and average distance from ground truth points to nearest observed points.

• �� Ablations: Independently vary world model memory (ICM vs 3DGS, short vs persistent memory) and agent memory capacity (RNN vs Transformer with varying context lengths) to assess their impact on exploration performance.

• �� Downstream Tasks: Fine-tune the pretrained exploration policy on apple-picking and image-goal navigation tasks, comparing performance against from-scratch training baselines.

• �� Training Details: Use Adam optimizer with learning rate 1e-5, train for 110 million steps. Random action mixing starts at 20% and decays to zero over 5 million steps. The agent simulates a spherical drone with four discrete actions (move forward, look left/right, pause) in Habitat simulator.

• �� On HM3D, the proposed method achieves 74.94% completeness at 1024 steps, surpassing OccAnt-RGBD's 74.62%, with average point distance reduced to 0.14cm, indicating superior coverage and precision.

• �� On Gibson, the agent attains 82.42% completeness, outperforming baselines and demonstrating cross-dataset generalization.

• �� Zero-shot tests on AI-generated Hobbit World and Spaceship show coherent exploratory behaviors with only 2-3 collisions over 256 steps, evidencing robustness to novel environments.

• �� Ablations confirm that persistent 3DGS world model significantly improves exploration compared to ICM or short-memory variants. Transformer policies with longer context windows outperform RNNs and no-memory policies.

• �� Fine-tuning on downstream tasks yields higher success rates than from-scratch training, especially under sparse reward conditions, highlighting the benefit of exploratory pretraining.

This approach is directly applicable to autonomous indoor robot navigation, enabling efficient exploration without reliance on depth sensors or explicit maps, facilitating deployment on resource-constrained platforms. The pretrained curiosity-driven policy serves as a strong initialization for downstream vision-based tasks such as object retrieval and goal-directed navigation, improving learning efficiency in sparse-reward scenarios. Additionally, the framework can be leveraged in virtual reality and gaming for intelligent agent exploration, enhancing environment understanding and interaction. Long-term, it can support dynamic environment monitoring, warehouse automation, and other robotics applications requiring robust, scalable exploration capabilities.

The method assumes static environments, limiting applicability in dynamic or highly changing scenes where persistent modeling is more complex. The computational cost of 3DGS reconstruction and rendering is substantial, potentially restricting real-time use in large-scale settings. Training requires privileged depth and camera pose data, increasing sensor and setup complexity. The policy's performance in multi-agent or highly dynamic environments remains unexplored. Addressing these limitations is essential for broader real-world deployment and robustness.