GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations

In recent years, robotic manipulation has increasingly leveraged Vision-Language-Action (VLA) models that integrate visual perception, natural language understanding, and action generation into a unified framework. Notable works such as PaLM-E and SayCan have demonstrated the potential of large-scale vision-language pretrained models to enable robots to follow open-world instructions and perform complex tasks. These models typically adopt hierarchical architectures combining high-level planning with low-level policy execution. Despite these advances, existing VLA systems predominantly rely on textual instructions as the sole modality for conveying human intent. This reliance presents challenges in real-world scenarios where language alone is insufficient to disambiguate spatial references, especially in cluttered scenes with multiple similar objects. Humans naturally use deictic gestures like pointing to resolve such ambiguities, but current VLA models rarely incorporate gestures as a primary input modality. Prior works often treat gestures as auxiliary signals or convert them into text, resulting in information loss and limited spatial grounding precision. Moreover, large-scale datasets aligning gesture, language, and action with precise spatial annotations are scarce due to the high cost and difficulty of manual collection and labeling. These limitations motivate the development of models that tightly integrate gesture with language and vision, supported by scalable data generation pipelines.

The core problem addressed is the spatial ambiguity in robotic instruction following when relying solely on language. Ambiguous commands such as “pick up this one” or “place it there” fail to uniquely specify targets in environments with multiple similar objects. Existing VLA models lack mechanisms to incorporate spatially precise gesture cues, limiting their ability to ground instructions accurately. Challenges include: 1) How to represent and fuse gesture information with language and vision in a unified model without losing spatial detail; 2) How to obtain sufficiently large and diverse training data with accurate gesture annotations to enable robust learning; 3) How to design model architectures that effectively leverage gesture cues for both intent reasoning and action generation. Addressing these challenges is crucial for enabling robots to understand and execute instructions reliably in complex, cluttered real-world settings.

The paper introduces several key innovations: 1) Gesture as a first-class instruction modality: Unlike prior works that treat gestures as auxiliary or convert them to text, GesVLA encodes gesture features directly into continuous latent tokens, preserving spatial precision and enabling deep multimodal fusion. 2) Dual-VLM architecture: The model separates intent reasoning (VLMint) from online perception and action generation (VLMper and action expert), connected via cross-attention that allows latent interaction without intermediate discretization, improving efficiency and robustness. 3) Semi-synthetic gesture data engine: By rendering articulated hand models onto real scene images with precise 3D pointing annotations, the pipeline generates diverse, scalable gesture-language-action data, mitigating the sim-to-real gap and enabling effective pretraining. 4) Two-stage training strategy: Pretraining VLMint on semi-synthetic data for gesture-conditioned spatial reasoning, followed by freezing VLMint and training VLMper and the action expert on real robot demonstrations, ensures robust sim-to-real transfer and improved task performance.

• �� Input modalities: The model receives RGB-D visual observations, natural language instructions, and gesture video sequences. Gesture videos are processed to extract keyframes based on hand motion dynamics.

• �� Gesture embedding: Using MediaPipe, four keypoints per keyframe (wrist and three index finger joints) are extracted as (x,y,depth) coordinates, concatenated into 12-dimensional vectors, and projected into latent space via a multi-layer perceptron (MLP).

• �� Dual-VLM architecture:
• VLMint performs multimodal intent reasoning by jointly modeling gesture embeddings and language instructions, outputting textual and visual reasoning representations.
• VLMper takes scene observations and attends to VLMint’s latent states through cross-attention, producing latent representations for action prediction.

• �� Action expert: A flow-based policy iteratively denoises an initial noise vector conditioned on VLMper’s latent representation and current robot state to generate smooth, continuous action trajectories.

• �� Data generation pipeline:
• GroundingDINO detects candidate objects in real RGB-D scenes.
• Targets are randomly sampled; their 3D coordinates are computed using depth maps and camera intrinsics.
• Hand pointing motions are synthesized by interpolating hand poses from random initial positions toward targets, incorporating parabolic lifting for naturalness.
• Hand meshes are rendered onto real scene images to produce semi-synthetic gesture videos paired with language instructions and precise spatial annotations.

• �� Training:
• Stage 1: VLMint is pretrained on the semi-synthetic dataset using teacher-forced autoregressive cross-entropy loss to jointly learn semantic reasoning and spatial grounding.
• Stage 2: VLMint is frozen; VLMper and the action expert are trained on real robot demonstration data using a flow matching objective to optimize action generation.

The experimental evaluation consists of two main parts: intent reasoning and robotic manipulation. Intent reasoning is tested on 88 real-world scenes with varying object counts and arrangements. The fixed instruction “Pick this up and put it there” is used, and the model must correctly identify both the pointed block and plate. Three cameras capture the scene: two global views and one mounted on the robot gripper. Robotic manipulation tasks include: 1) Pick-and-Place Block — grasping a specified block from clutter and placing it on a plate; 2) Select Jelly — sequentially picking jelly cups from plates in indicated order; 3) Select Fruit/Vegetable — sequentially picking bananas from bins. Each task is repeated 20 times under simple (few objects) and hard (multiple objects) conditions. Baselines include text-only VLA, prompted multimodal large language models combined with VLA, geometric pipeline plus VLA, and a decoupled VLM variant of GesVLA. Ablation studies analyze the impact of gesture encoding, data augmentation, training strategies, and prompt design. Training uses AdamW optimizer with cosine learning rate scheduling on RTX 4090 GPUs.

GesVLA’s VLMint achieves 94.3% accuracy on the intent reasoning test set, outperforming the geometric pipeline baseline (59.1%) and prompted multimodal LLM baseline (38.6%) by large margins. In robotic manipulation, GesVLA attains an average success rate of 83.3%, significantly higher than text-only VLA (31.7%) and geometric pipeline augmented VLA (41.7%). Ablations show that removing the gesture MLP embedding reduces intent reasoning accuracy to 84.1%, omitting data augmentation drops it to 89.8%, and disabling coordinate jitter causes a severe accuracy decline to 42.0%. Two-stage training with frozen VLMint outperforms joint training, indicating the importance of staged optimization. Visual prompts are critical for action generation, while textual prompts add no significant benefit. These results demonstrate the effectiveness of tightly integrating gesture and language in latent space for robust spatial grounding and action execution.

GesVLA is applicable in multimodal human-robot interaction scenarios requiring precise spatial grounding, especially in cluttered or complex environments. Use cases include warehouse picking where workers can use gestures and language to direct robots, service robots assisting in homes or healthcare settings with natural hand-gesture commands, and industrial assembly lines where operators guide robots via combined gesture-language instructions to improve accuracy and flexibility. The system relies on multi-camera setups and depth sensing, suitable for structured or semi-structured environments. Future extensions may enable multi-robot collaboration and dynamic task execution in unstructured settings.

The model depends on MediaPipe for hand keypoint detection, which suffers from reduced robustness under occlusion or challenging lighting, leading to degraded gesture recognition. Although the semi-synthetic data generation pipeline reduces the sim-to-real gap, it cannot fully capture the diversity of real-world hand appearances and motions, limiting generalization. Current experiments focus primarily on pointing gestures, lacking coverage of richer gesture vocabularies needed for complex interactions. Additionally, the flow-based action generation incurs computational overhead, impacting real-time performance. Addressing these limitations requires improved hand detection algorithms, expanded gesture datasets, and optimized model architectures.