GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI

The evolution of robotic simulation has transitioned from CPU-based, sequential engines like Gazebo and PyBullet to GPU-accelerated platforms such as Isaac Gym and Genesis. These advancements enable large-scale parallel environments, significantly accelerating training and policy optimization. However, existing evaluations primarily focus on task success rates or throughput, neglecting core simulation engine properties like physical fidelity and reproducibility. As simulation scales up, issues such as numerical inaccuracies, synchronization artifacts, and inherent GPU non-determinism become prominent, threatening the reliability of learned policies. The need for standardized benchmarks that comprehensively evaluate these aspects is urgent, especially as embodied AI applications demand both high fidelity and large environment counts.

Current GPU-based simulators face a fundamental challenge: scaling up environment counts often leads to degraded physical accuracy and increased non-determinism. Batch processing on GPUs introduces subtle numerical variations, synchronization artifacts, and floating-point inconsistencies, causing results to vary across runs and environments. This variability hampers reproducibility, critical for research validation and industrial deployment. Existing benchmarks lack systematic metrics to quantify these issues, making it difficult to compare simulators or optimize their configurations. Addressing this problem requires a comprehensive evaluation framework that captures the trade-offs between throughput, physical fidelity, and stochasticity, guiding the development of more reliable simulation tools.

The core innovation is the development of GPUSimBench, a benchmark suite that systematically evaluates GPU simulators along three axes: scalability, physical consistency, and non-determinism. It introduces a controlled inclined-plane task to quantify the distributional alignment with real-world physics via Earth Mover’s Distance. Additionally, it classifies four empirical stochastic regimes based on intra- and inter-run variability, revealing how GPU batch processing induces numerical variations. The framework enables cross-simulator comparisons, providing insights into the fundamental trade-offs and guiding simulator selection for embodied AI tasks. This comprehensive approach surpasses traditional task-centric benchmarks by focusing on core engine properties.

• �� Design a controlled inclined-plane collision experiment, capturing object positions at fixed time intervals. • Use Earth Mover’s Distance (EMD) to quantify the distributional difference between simulated and real-world outcomes. • Measure throughput (FPS) and GPU memory footprint across varying environment counts (Nenv). • Analyze intra-run variability by comparing parallel environments within a single run, computing pairwise EMD. • Assess inter-run variability by comparing aggregated distributions from multiple independent runs. • Classify four stochastic regimes based on the presence or absence of intra- and inter-run variability. • Compare multiple simulators (Isaac Lab, Genesis, Madrona, MuJoCo variants) to evaluate performance and fidelity at scale.

The experiments involve running parallel inclined-plane collision scenarios with 16 environments, measuring physical distribution alignment via EMD, and assessing scalability by increasing environment count until hardware limits. Multiple independent runs quantify variability, while real-world measurements serve as empirical references. The study compares simulators like ManiSkill, MJX, MuJoCo Warp, and Isaac Lab, focusing on throughput, memory usage, and distributional fidelity. Additional experiments include free-fall scenarios and articulated robot control, providing a comprehensive performance profile. Hyperparameters such as time step and environment count are standardized to ensure comparability. Results are analyzed to identify the impact of GPU batching on simulation accuracy and reproducibility.

ManiSkill and MJX achieved the best physical fidelity with EMD scores of 2.520cm and 3.970cm, respectively. As environment count increased, throughput scaled linearly, reaching up to 47 FPS in Isaac Lab, but non-determinism also increased, especially in Madrona and MuJoCo Playground, with intra- and inter-run variability becoming significant. Four stochastic regimes were identified: some simulators maintained deterministic results within runs but varied across runs, while others showed both intra- and inter-run variability. The analysis revealed that GPU batch processing is a primary source of numerical noise, impacting reproducibility, especially at large scales. These insights inform the trade-offs in simulator selection based on task fidelity and stability requirements.

The benchmark framework can guide developers in selecting appropriate simulators for embodied AI training, robotics research, and industrial automation. It enables systematic evaluation of physical fidelity and reproducibility, crucial for policy learning, simulation-to-real transfer, and safety-critical applications. The insights into GPU batch effects can inform hardware and software optimizations, leading to more reliable large-scale simulation environments. Long-term, this work supports the development of standardized evaluation protocols, fostering industry adoption of robust, scalable simulation platforms that meet the demands of autonomous systems and complex multi-physics scenarios.

The current evaluation focuses on rigid-body collision scenarios, limiting applicability to deformable or multi-physics environments. Parameter tuning was constrained, potentially affecting generality. Hardware variability across different GPU architectures was not extensively studied, which could influence results. The stochasticity classification, while comprehensive, may not fully capture all real-world sources of randomness, such as sensor noise or environmental disturbances. Future work should extend to more complex scenarios and multi-modal physics to improve robustness and applicability.