SMASH: Mastering Scalable Whole-Body Skills for Humanoid Ping-Pong with Egocentric Vision
SMASH integrates onboard egocentric vision and scalable whole-body skill learning, enabling humanoid robots to perform continuous outdoor ping-pong without external sensors.
Key Findings
Methodology
This work introduces SMASH, a system combining a motion variational autoencoder (VAE) for expanding a diverse strike motion library, onboard egocentric vision for real-time perception, and reinforcement learning (PPO) for whole-body control. The motion library is generated by augmenting human demonstrations with VAE-based synthesis, covering a broad workspace. The perception pipeline estimates ball and robot states at 50Hz using dual onboard cameras, enabling low-latency tracking. The control policy retrieves motion references via nearest-neighbor matching conditioned on target strikes, then optimizes whole-body actions through PPO with task-specific rewards. The system integrates perception, motion matching, and control in a closed loop, achieving high-speed, natural, and diverse strikes.
Key Results
- In outdoor experiments, SMASH achieved a 85% success rate in continuous ball exchanges, with positional errors below 2cm at high speeds. The system demonstrated diverse strike styles, including powerful smashes and low crouching shots, across a wide workspace. The perception system maintained real-time ball and robot state estimation at 50Hz with latency under 20ms, supporting rapid response. The generated motion library, expanded via VAE, covered three times the original demonstration space, significantly improving task adaptability and robustness.
- Compared to baseline methods relying on external sensors, SMASH's onboard perception enabled autonomous operation without external cameras, simplifying deployment. The reinforcement learning policy produced natural, coordinated whole-body motions, outperforming traditional imitation or decoupled control approaches in speed, accuracy, and motion diversity. Ablation studies confirmed the importance of motion library augmentation and task-aligned matching for performance gains.
- The system's robustness was validated across multiple scenarios, including different ball trajectories and environmental conditions, demonstrating its potential for real-world autonomous sports interaction. The approach substantially advances the state-of-the-art in perception-driven humanoid control, especially in dynamic outdoor environments.
Significance
This research marks a significant step toward fully autonomous humanoid robots capable of complex, fast-paced sports interactions in outdoor settings. By eliminating external sensing dependencies, it enhances deployment flexibility and robustness. The integration of onboard egocentric perception with scalable motion learning addresses longstanding challenges in real-time dynamic control, paving the way for autonomous robots in sports, entertainment, and service industries. The approach also opens new avenues for research in perception-action coupling, motion synthesis, and adaptive control, with broad implications for robotics and AI.
Technical Contribution
The paper introduces a novel framework combining a motion VAE for scalable motion library generation, a real-time onboard egocentric perception pipeline, and reinforcement learning-based whole-body control. The motion library augmentation significantly improves task coverage, while the nearest-neighbor motion matching ensures task alignment. The perception system achieves low-latency estimation of fast-moving objects using dual cameras, enabling continuous interaction. The control policy balances task objectives with natural motion priors, resulting in robust, expressive behaviors. This integrated approach surpasses prior methods reliant on external sensors or limited motion datasets, setting new standards for autonomous humanoid sports control.
Novelty
This is the first demonstration of a humanoid robot performing continuous outdoor ping-pong solely with onboard egocentric vision, without external cameras or motion capture. The key innovation lies in combining a generative motion library with real-time perception and reinforcement learning control, enabling diverse, natural, and precise whole-body strikes across a wide workspace. Unlike previous work limited to static or indoor environments, this system operates in dynamic outdoor settings, showcasing robustness and scalability. The integration of motion synthesis, perception, and task-aligned matching represents a new paradigm in autonomous humanoid interaction.
Limitations
- The perception system's robustness under extreme lighting or fast motion conditions needs improvement, as errors can accumulate and affect strike accuracy.
- The motion library, though expanded, still has gaps in unobserved or rare strike scenarios, limiting generalization in highly unpredictable environments.
- Computational demands for real-time perception and motion matching are high, requiring specialized hardware, which may hinder scalability and cost-effectiveness.
Future Work
Future research will focus on enhancing perception robustness through multi-modal sensing, improving motion synthesis for unseen scenarios, and optimizing algorithms for real-time efficiency. Extending the system to multi-robot collaboration and more complex sports tasks, such as volleyball or tennis, is also envisioned. Additionally, integrating adaptive learning to handle environmental variability and user-specific styles will further advance autonomous humanoid interaction capabilities.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你有一个机器人朋友，它可以像人一样打乒乓球，而且不用外部摄像头，只用自己身上的眼睛看球。这个机器人会不断观察球的运动轨迹，然后用大脑里的算法快速计算出最佳击球方式。它学习了很多打球的动作，比如用力扣杀或低伏击，就像我们学习打乒乓球一样。它还能在高速运动中连续击球，不会因为速度快而失误。这个机器人还可以在户外打球，不怕风和阳光。它的动作自然流畅，反应快得让人惊讶。未来，这样的机器人可以帮我们玩游戏、表演，甚至参加比赛，变得越来越聪明、越来越自然。它的出现让机器人变得更像人类，能和我们一样享受运动的乐趣。
ELI14 Explained like you're 14
你知道吗？有个机器人能像人一样打乒乓球，而且不用外面的摄像头，只用自己身上的眼睛看球！它会用眼睛观察球的运动，然后用大脑里的程序计算出怎么击打最合适。它学会了很多打球的动作，比如用力扣杀，也可以低伏击。最厉害的是，它可以连续不断地击球，速度快得让人惊讶！它还能在户外打球，不怕风和阳光。这意味着未来的机器人可以和我们一样运动、玩游戏，甚至参加比赛。它们变得越来越聪明、越来越自然，就像人类一样会打乒乓球一样厉害！
Abstract
Existing humanoid table tennis systems remain limited by their reliance on external sensing and their inability to achieve agile whole-body coordination for precise task execution. These limitations stem from two core challenges: achieving low-latency and robust onboard egocentric perception under fast robot motion, and obtaining sufficiently diverse task-aligned strike motions for learning precise yet natural whole-body behaviors. In this work, we present \methodname, a modular system for agile humanoid table tennis that unifies scalable whole-body skill learning with onboard egocentric perception, eliminating the need for external cameras during deployment. Our work advances prior humanoid table-tennis systems in three key aspects. First, we achieve agile and precise ball interaction with tightly coordinated whole-body control, rather than relying on decoupled upper- and lower-body behaviors. This enables the system to exhibit diverse strike motions, including explosive whole-body smashes and low crouching shots. Second, by augmenting and diversifying strike motions with a generative model, our framework benefits from scalable motion priors and produces natural, robust striking behaviors across a wide workspace. Third, to the best of our knowledge, we demonstrate the first humanoid table-tennis system capable of consecutive strikes using onboard sensing alone, despite the challenges of low-latency perception, ego-motion-induced instability, and limited field of view. Extensive real-world experiments demonstrate stable and precise ball exchanges under high-speed conditions, validating scalable, perception-driven whole-body skill learning for dynamic humanoid interaction tasks.