Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

In the field of mixed reality (MR), users need to interact with objects in virtual environments that are often beyond their physical reach. Traditionally, target selection in MR relies on single cues, such as directional cues (e.g., pointing with a finger or controller) or gestural cues (e.g., grasp gestures). However, these methods have limitations when dealing with complex scenes, particularly when target objects are densely packed or occluded. Recently, researchers have begun exploring multimodal interaction techniques, combining multiple user-generated cues to improve selection accuracy and efficiency. The research background of this paper is based on this trend, aiming to address the shortcomings of existing methods through a probabilistic cue integration framework.

Selecting out-of-reach objects in mixed reality is a fundamental task, but existing methods perform poorly under uncertainty. Specifically, when the dominant cue becomes unreliable, system performance degrades significantly. This issue is particularly evident when target objects are densely packed, share similar shapes, or are occluded. Additionally, existing multi-cue methods are often rule-based and lack flexibility, making them unsuitable for adapting to different interaction scenarios. Therefore, achieving efficient and accurate target selection under uncertainty remains a pressing challenge.

The core innovations of this paper include the introduction of a novel probabilistic cue integration framework for selecting out-of-reach objects in mixed reality. First, the framework combines user-generated cues of pointing direction and grasp gestures, utilizing Bayesian inference for intent inference. Second, the researchers collected the Out-of-Reach Grasping (ORG) dataset to train a robust model for gestural cues, capturing grasping patterns not present in existing datasets. Finally, unlike existing rule-based methods, the framework flexibly combines multiple cues to adapt to different interaction scenarios.

The methodology of this paper includes the following key steps:

• �� Dataset Collection: Researchers collected the Out-of-Reach Grasping (ORG) dataset, which includes grasping patterns not present in existing datasets.

• �� Directional Cue Modeling: A probabilistic model for directional cues is constructed by defining the ray's origin and direction vector.

• �� Gestural Cue Modeling: A neural network parameterized model estimates the probability relationship between gestures and candidate objects.

• �� Bayesian Inference: Bayesian inference is used to integrate directional and gestural cues, calculating the posterior probability of candidate objects.

• �� Target Selection: The object with the highest posterior probability is selected as the inferred target.

The experimental design includes two user studies (Study 1 and Study 2) to validate the effectiveness of the Point&Grasp method. In Study 1, researchers compared Point&Grasp with single-cue methods (direction-only and grasp-only) under systematically varied spatial and semantic ambiguities. Study 2 benchmarks Point&Grasp against state-of-the-art selection techniques (BubbleRay and Expand). These experiments used the ORG dataset, with evaluation metrics including selection accuracy, selection speed, and user satisfaction.

The experimental results show significant improvements in accuracy and speed with the Point&Grasp method. Specifically, compared to single-cue baselines, the method increased selection accuracy by approximately 15% and selection speed by about 20% in complex scenes. Moreover, Point&Grasp remains practically effective across various sources of ambiguity, particularly under high spatial and semantic ambiguity conditions, outperforming BubbleRay and Expand in selection time and completion rate. User feedback indicated that gesture-based interaction was natural and consistent with everyday grasping habits.

The method has wide applications in mixed reality, including 3D design, gaming, and everyday tasks. In these scenarios, users need to efficiently and accurately select out-of-reach objects. Point&Grasp improves selection accuracy and speed by combining directional and gestural cues, particularly in complex scenes. Additionally, the method does not require additional sensors, making it easy to integrate into existing MR systems.

Despite the excellent performance of the Point&Grasp method in handling ambiguity, it may experience performance degradation in extremely complex scenes, particularly when multiple objects are densely packed and share similar shapes. Additionally, the system's performance may be suboptimal under poor lighting conditions or when gestures are partially occluded. Future research directions include optimizing gesture recognition algorithms to improve robustness under varying lighting and occlusion conditions, and validating the method's generalizability and practicality in larger-scale user studies.