OGBench: Benchmarking Offline Goal-Conditioned RL

The evolution of reinforcement learning has seen rapid advances with algorithms like DQN, PPO, and SAC, primarily focusing on online learning where agents interact directly with environments. However, online methods are often impractical in real-world scenarios due to high data collection costs and safety concerns. As a solution, offline RL emerged, leveraging pre-collected datasets to train policies without environment interaction. Early works such as BCQ, CQL, and MOReL demonstrated success in single-task settings, but extending these to multi-task, goal-conditioned contexts remains challenging. Recent efforts incorporate goal-conditioned RL to enable flexible, multi-goal behaviors, but lack of standardized evaluation hampers progress. Consequently, the community recognizes the need for comprehensive benchmarks to systematically assess and compare algorithms across complex, realistic tasks.

The core challenge in offline GCRL is enabling algorithms to learn versatile, goal-reaching policies from diverse, often suboptimal datasets. Key issues include goal stitching—combining trajectory segments to achieve new goals—long-horizon reasoning—planning over extended sequences—and handling environment stochasticity. Existing benchmarks are limited, often focusing on simple tasks or deterministic environments, which do not reflect real-world complexity. Without a unified evaluation platform, it is difficult to measure progress or identify algorithm limitations. Addressing these issues requires designing tasks that are both challenging and representative of practical applications, with datasets that support multi-faceted evaluation.

This work introduces OGBench, a comprehensive benchmark that addresses the limitations of prior evaluations. Key innovations include: 1) Environment diversity—covering locomotion, manipulation, and drawing tasks with varying complexity; 2) Dataset richness—85 datasets combining expert and exploration trajectories to test stitching, generalization, and stochasticity; 3) Multi-capability evaluation—assessing stitching, long-horizon planning, and stochastic control simultaneously; 4) Reference implementations—of six algorithms like CQL, BCQ, and MOReL, enabling reproducibility. The benchmark emphasizes realistic, high-fidelity environments and provides scalable, adjustable difficulty levels, fostering systematic research and development in offline GCRL.

• �� Environment design: Developed 8 environment types, including maze navigation, manipulation, and drawing, with diverse challenges such as goal stitching, long-horizon tasks, and stochastic dynamics.
• �� Dataset construction: Collected 85 datasets with expert trajectories, exploration data, and stochastic environments, ensuring coverage of suboptimal and high-diversity data.
• �� Algorithm implementation: Provided reference code for 6 algorithms—CQL, BCQ, MOReL, TD3+BC, BEAR, and D4RL—tuned for each task.
• �� Evaluation metrics: Success rate across five predefined goals, stitching ability, long-horizon success, and robustness to stochasticity.
• �� Experimental procedure: Repeated trials on each dataset, analyzing performance variation, ablation studies on dataset quality and environment complexity.

Experiments involved running six algorithms across all environments and datasets, including maze navigation, manipulation, and drawing tasks. Each environment had multiple dataset versions—navigate, stitch, explore—to evaluate different capabilities. Performance was measured via success rates, stitching success, and robustness to stochasticity. Results demonstrated significant performance gaps, with success rates in stitching tasks reaching up to 55%, and long-horizon success around 40%. Variability in stochastic environments highlighted the need for improved stochastic handling. Ablation studies confirmed the importance of dataset diversity and environment complexity in influencing algorithm performance. The experiments validated OGBench's ability to expose strengths and weaknesses of current methods systematically.

The best algorithms achieved success rates up to 55% in stitching tasks, showing the importance of goal stitching for overall performance. Long-horizon tasks saw success rates around 40%, indicating room for improvement. In stochastic environments, success rates varied from 15% to 45%, emphasizing the challenge of environment randomness. These results reveal that stitching and long-term planning are critical bottlenecks. The performance differences across algorithms and environments demonstrate OGBench's effectiveness in diagnosing strengths and weaknesses, guiding future research toward more robust, scalable solutions.

OGBench can accelerate development of autonomous robots, manipulators, and creative agents capable of multi-task goal-reaching from offline data. It provides a testing ground for algorithms intended for real-world deployment, such as autonomous navigation, industrial automation, and assistive robotics. The benchmark's insights can inform design choices, data collection strategies, and algorithm improvements, ultimately reducing development costs and increasing safety in deployment scenarios.

While comprehensive, OGBench relies on simulated environments, which may not fully capture real-world complexities like sensor noise and unmodeled dynamics. Transferability of algorithms trained on these datasets remains uncertain. The focus on MuJoCo and NumPy environments limits direct applicability to physical systems. Additionally, high stochasticity environments still challenge current algorithms, indicating the need for further research into robustness and generalization. Future work should incorporate real-world datasets and explore transfer learning to bridge the simulation-reality gap.