Simulation-Driven Imitation Learning for Biosignals-Free Shared-Autonomy Prosthetic Grasping

The quest for intuitive, reliable, and low-effort control of upper-limb prostheses has long challenged researchers and clinicians alike. Traditional methods relying on surface electromyography (sEMG) signals demand continuous user engagement and calibration, often leading to fatigue and inconsistent performance. Semi-autonomous approaches, which leverage computer vision to infer intended actions, have alleviated some burdens but still require explicit triggers for each grasp, limiting natural interaction. Fully autonomous, biosignals-free control systems promise a paradigm shift, enabling prostheses to infer user intent from minimal input—such as simple positioning—while executing complex grasping behaviors independently.

However, developing such systems hinges on the availability of large, diverse, and high-quality demonstration datasets. Collecting real-world human demonstrations is prohibitively costly, time-consuming, and fraught with safety concerns, especially when involving vulnerable populations. This bottleneck hampers the training of robust neural policies capable of generalizing across objects, environments, and user variations.

In response, the authors propose a novel simulation-driven framework that automates the generation of reach-to-grasp demonstrations. By leveraging physics-based simulation environments, they synthesize physically feasible grasp configurations, retarget natural reaching trajectories, and procedurally generate indoor scenes with diverse objects. The system records multimodal observations—wrist-view RGB images, proprioceptive joint states, and action commands—forming a comprehensive dataset for imitation learning. This approach circumvents the limitations of real-world data collection, enabling scalable, diverse, and cost-effective training data generation.

Extensive benchmarking demonstrates that policies trained solely on these simulated demonstrations transfer effectively to real prosthetic hardware, achieving over 90% grasp success across multiple scenarios. The models outperform baseline methods, exhibit strong generalization to unseen objects and environments, and maintain robustness under challenging conditions such as clutter and occlusion. The study compares several state-of-the-art imitation learning algorithms—such as Action Chunking with Transformers (ACT), VTM-VAE, and HannesImitation—highlighting the versatility and effectiveness of the simulation-generated data.

This work marks a significant advance in autonomous prosthetic control, addressing core challenges of data scarcity and transferability. Its implications extend beyond prosthetics, offering a scalable blueprint for simulation-based training in robotic manipulation and assistive technologies. Future directions include integrating domain adaptation techniques, expanding sensory modalities, and developing online learning capabilities to further enhance system robustness and adaptability. Overall, this research paves the way toward practical, user-friendly, and intelligent prosthetic systems that can operate seamlessly in complex real-world environments, fundamentally transforming assistive robotics.