A Gesture-Based Visual Learning Model for Acoustophoretic Interactions using a Swarm of AcoustoBots

As multi-agent systems evolve, swarm robotics has become a research hotspot. These systems coordinate multiple simple autonomous robots to perform complex tasks, offering high fault tolerance, scalability, and adaptability. Traditional robotic systems rely on explicit planning, while swarm robotic systems operate through local rules, producing complex global behaviors from simple agent interactions. These characteristics make swarm robotics applicable in dynamic, real-world environments. However, achieving intuitive real-time interaction between humans and autonomous agent collectives remains a major challenge. Traditional human-swarm interaction methods often rely on abstract command languages or low-level input devices, which are inconvenient for non-expert users and unsuitable for fast-paced or dynamic environments.

Current swarm robotic systems face significant challenges in achieving intuitive real-time interaction between humans and autonomous agent collectives. Traditional methods rely on abstract command languages or low-level input devices, which are inconvenient for non-expert users and unsuitable for fast-paced or dynamic environments. Additionally, existing implementations rely on scripted commands, lacking an intuitive interface for real-time human control. How to achieve natural, contactless interaction without relying on text commands is a pressing issue that needs to be addressed.

The core innovation of this paper lies in applying visual language models to swarm robotic gesture interaction, providing a natural interaction method without relying on text commands. • Real-time gesture capture and recognition are achieved through ESP32-CAM gesture capture and PhaseSpace motion tracking. • An OpenCLIP-based visual learning model classifies three hand gestures and maps them to haptics, audio, and levitation modalities through linear probing. • A centralized processing strategy is adopted, which, although deviating from the ideal of fully distributed autonomy, offers a practical solution under current hardware limitations. • The method's effectiveness in multimodal human-swarm interaction is demonstrated through experimental validation.

This paper presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform. • The system combines ESP32-CAM gesture capture, PhaseSpace motion tracking, centralized processing, and an OpenCLIP-based visual learning model (VLM). • Gesture Capture: Real-time hand gesture images are captured using ESP32-CAM and transmitted to the central server for processing. • Motion Tracking: Precise tracking between users and robots is achieved through the PhaseSpace system, providing positional information. • Visual Learning Model: Based on OpenCLIP, three hand gestures are classified and mapped to haptics, audio, and levitation modalities through linear probing. • Centralized Processing: Gesture recognition and control command generation are performed on the central server, ensuring system real-time performance and coordination.

The experimental design includes two parts: 1) evaluating the performance of the gesture classification model, and 2) evaluating the integration of the model with the AcoustoBot platform. • Datasets: Different scales of datasets are used for training and validation to assess the model's generalization ability. • Baselines: Compared with traditional CNN-based gesture recognition methods. • Evaluation Metrics: Validation accuracy, training and validation loss, response time, and modality switching accuracy. • Hyperparameters: AdamW optimization algorithm is used with a learning rate of 1e-3, batch size of 5, and training for 50 epochs. • Ablation Studies: Performance changes are evaluated through different dataset scales.

The experimental results show significant enhancement in the model's generalization ability as the dataset size increases. Validation accuracy improved from about 67% with a small dataset to nearly 98% with the largest dataset. In integrated experiments with two AcoustoBots, the system achieved an overall gesture-to-modality switching accuracy of 87.8% across 90 trials, with an average end-to-end latency of 3.95 seconds. These results demonstrate the feasibility of using a vision-language-model-based gesture interface for multimodal human-swarm interaction. Ablation studies reveal that dataset size significantly impacts model performance, with larger datasets providing higher accuracy and better generalization.

The system can be applied in various scenarios, including: 1) Human-Robot Collaboration: In manufacturing, workers can collaborate with robots through gestures, improving production efficiency. 2) Entertainment Interaction: In gaming or virtual reality, users can interact with virtual characters or environments through gestures, enhancing immersion and interactivity. 3) Educational Training: In education, teachers can interact with teaching robots through gestures, enhancing teaching effectiveness. These application scenarios demonstrate the system's potential in different fields, especially in situations requiring natural, contactless interaction.

Despite the system's potential in multimodal human-swarm interaction, there are some limitations. First, the system relies on centralized processing, limiting the autonomy of each robot, which may lead to performance bottlenecks under limited computational capacity. Second, the current gesture set is static, unable to adapt to more complex interaction needs, limiting the system's scalability. Additionally, the experiments were conducted only in controlled environments, lacking validation in real-world dynamic environments. Future research directions include exploring decentralized processing architectures, expanding the gesture set to support more complex interaction modes, and validating the system in real-world dynamic environments.