Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task

The evolution of human-robot interaction (HRI) has seen significant progress with deep learning and multimodal perception, exemplified by systems like OpenAI's GPT series and vision models like ViLD. Prior works focused on rule-based or single-modal methods, such as gesture recognition or speech commands, which limited interaction naturalness. Recent advances in large language models (LLMs) have demonstrated broad reasoning and contextual understanding, inspiring their application in robotics. However, integrating visual, verbal, and environmental cues remains a challenge, especially for real-time intention prediction. Existing systems often lack the ability to fuse multimodal information seamlessly, leading to reduced interaction fluidity and misinterpretations. This research builds upon prior efforts by combining perception pipelines with LLMs, aiming to address these gaps and enable robots to infer human goals more accurately and naturally.

The core challenge is enabling robots to accurately infer human intentions in dynamic, multimodal scenarios. Existing methods often rely on rule-based heuristics or isolated perception modules, which fail to capture the full context of human signals. The difficulty lies in integrating diverse cues—gestures, facial expressions, speech, and environmental context—into a unified reasoning framework capable of real-time inference. This is compounded by the variability in human behavior, environmental noise, and the need for scalable, adaptable models. Overcoming these limitations is crucial for deploying socially intelligent robots capable of proactive, nuanced interactions in real-world settings.

The main innovations include: 1) a Hierarchical architecture that separates perception and task reasoning, enabling modular, scalable inference; 2) multi-model fusion combining neural perception modules (MediaPipe, ViLD) with LLMs for contextual understanding; 3) leveraging zero-shot reasoning capabilities of LLMs to interpret multimodal cues without extensive training data; 4) real-time integration on a physical robot platform, NICOL, demonstrating practical viability. These innovations address the limitations of prior rule-based and unimodal systems, providing a flexible, robust framework for intention prediction.

• �� Data collection: Visual perception modules detect hand gestures, pose, and facial expressions; speech is recognized via Whisper.
• �� Data transformation: Non-verbal cues are converted into textual tokens, e.g., 'user points at apple', 'user smiles'.
• �� Environment sensing: ViLD detects objects and environment states, providing contextual info.
• �� Hierarchical reasoning: Perceptive layer analyzes cues and environment, forming beliefs; task layer combines explicit commands, prompts, and interaction history.
• �� LLM inference: Multiple models process combined textual input, perform reasoning, and generate predictions.
• �� Action generation: Based on inference, robot executes gestures, speech, or movements, closing the interaction loop.

The system was evaluated on a dataset of 150 object categorization trials involving NICOL robot. The models tested included GPT-4, GPT-3.5, Vicuna, and Mistral, with performance metrics including accuracy in intention prediction and task understanding. The experiments involved various prompts to test task comprehension, object categorization, and intention inference, with success rates compared across models. Ablation studies assessed the contribution of each modality and reasoning layer, confirming the effectiveness of multimodal fusion and hierarchical reasoning. The system's real-time performance was validated through latency measurements, averaging below 1 second, demonstrating practical applicability.

GPT-4 achieved 95% accuracy in task understanding and 92% overall intention prediction, outperforming other models such as Vicuna (33%) and Mistral (18%). The system effectively fused gestures, facial expressions, and speech cues, maintaining robustness across different scenarios. The experimental results showed that the hierarchical reasoning approach significantly improved prediction accuracy, especially in ambiguous or multi-turn interactions. The real-time response capability was confirmed with an average latency under 1 second, validating the system's deployment feasibility on the NICOL robot.

This system can be directly applied in social robots for assistance, education, and healthcare, where natural human-like understanding is essential. It enables robots to interpret subtle cues, anticipate needs, and respond proactively, enhancing user experience. The framework also supports multi-modal interaction in smart environments, facilitating more intuitive human-machine collaboration. Long-term, it can contribute to autonomous social agents capable of complex reasoning and adaptive behaviors in diverse settings.

Current limitations include reduced robustness in multi-user or cluttered environments, high computational costs for models like GPT-4, and sensitivity to environmental noise or occlusion. The system's scalability is constrained by hardware requirements, and handling ambiguous multi-turn dialogues remains challenging. Future improvements should focus on model compression, multi-agent reasoning, and robustness under real-world conditions to broaden applicability.