MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
MV-GEL employs multi-view ranking and VLM reasoning to localize geometric entities on meshes, achieving up to 1.7× IoU improvement.
Key Findings
Methodology
The framework integrates GELviews view ranking and LISA-based semantic segmentation, combining multi-view rendering, geometric visibility assessment, and multimodal fusion. GELviews encodes camera views with CLIP, computes visibility scores via ray casting, and employs a Transformer to model inter-view dependencies, prioritizing views that maximize target entity observability. Selected views are processed by a language-conditioned segmentation backbone, with 2D masks lifted to 3D mesh entities through geometry-aware ray casting, enabling end-to-end localization. Extensive experiments on 54k queries demonstrate significant improvements in face IoU (up to 1.7×) and edge F1 (over 4.5×) compared to baselines, especially on thin, view-sensitive structures.
Key Results
- The model achieved a maximum face IoU of 0.529 and an F1 score exceeding 0.632, outperforming unranked and random view baselines. Ablation studies confirmed the effectiveness of geometric view ranking, with notable gains in complex structures.
- In large-scale testing, the approach consistently outperformed CLIP-only and random sampling methods, validating the importance of view prioritization for fine-grained geometric localization.
- The multi-view fusion and geometric visibility scoring contributed significantly to the robustness and accuracy, especially for challenging entities with occlusion or slender features.
Significance
This work advances the field of 3D geometric understanding by enabling precise, language-driven localization of CAD entities directly on meshes. It addresses the view-dependence and structural complexity challenges, facilitating applications in CAD editing, robotic manipulation, and scientific simulation. The CAD-agnostic design ensures broad applicability across different mesh formats, promoting generalization. The integration of multimodal reasoning with geometric priors marks a significant step toward intelligent 3D scene understanding, bridging the gap between natural language and complex geometric data.
Technical Contribution
The paper introduces GELviews, a novel view ranking module that combines CLIP-based semantic similarity with geometric visibility scores and Transformer-based cross-view reasoning. It innovatively maps 2D segmentation masks to mesh entities via geometry-aware ray casting, enabling precise localization without reliance on CAD-specific representations. The framework's modular design allows independent training of view ranking and segmentation components, enhancing flexibility and scalability. These contributions collectively push the boundaries of multimodal 3D understanding, offering a robust, generalizable solution for fine-grained geometric entity localization.
Novelty
This is the first framework to integrate prompt-conditioned multi-view ranking with geometry-aware mask lifting for natural language localization of mesh entities. Unlike prior work focused on point clouds or voxel-based methods, it operates directly on polygon meshes, preserving topological accuracy. The combination of CLIP-based semantic encoding, visibility-driven view prioritization, and geometry-aware mask projection constitutes a unique approach that effectively addresses the view sensitivity and structural complexity of CAD models, marking a significant innovation in the field.
Limitations
- The approach struggles with highly occluded or extremely complex topologies where visibility estimation becomes unreliable, leading to potential localization failures.
- Computational cost of multi-view rendering and ranking limits real-time deployment; efficiency improvements are needed.
- Natural language descriptions depend on the clarity and specificity of prompts; ambiguous queries can reduce accuracy.
Future Work
Future directions include optimizing view ranking for real-time applications, integrating reinforcement learning for adaptive view selection, and extending the framework to point cloud and volumetric data. Exploring more robust natural language understanding modules to handle ambiguous descriptions and expanding the dataset with more diverse geometries will further enhance generalization. Additionally, incorporating user feedback for interactive refinement could make the system more versatile for industrial and robotic applications.
AI Executive Summary
Accurately identifying and grounding specific geometric features within complex 3D models remains a fundamental challenge in computer-aided design, robotics, and scientific simulation. Traditional methods often rely on manual annotation or global feature extraction, which struggle with view-dependent ambiguities and intricate structures. Recent advances in vision-language models like CLIP have demonstrated remarkable semantic reasoning in 2D images, but extending this to structured 3D geometry, especially CAD meshes, introduces new difficulties. The appearance of a geometric entity varies significantly with viewpoint, making single-view approaches insufficient for precise localization.
This paper introduces MV-GEL, a novel framework that leverages multi-view reasoning and prompt-conditioned view ranking to localize fine-grained geometric entities directly on polygon meshes. The core innovation, GELviews, employs CLIP encodings to evaluate the relevance of candidate views based on both semantic similarity and geometric visibility. By integrating a Transformer-based inter-view reasoning module, the system prioritizes views that maximize the observability of the target entity, addressing the view sensitivity problem effectively.
Once the optimal views are selected, a language-conditioned segmentation backbone, based on LISA, predicts 2D masks of the entity in each view. These masks are then geometrically lifted onto the mesh using a geometry-aware ray casting process, ensuring accurate correspondence between 2D segmentation and 3D topology. Extensive experiments on a newly constructed dataset of 54,000 queries demonstrate that MV-GEL significantly outperforms baseline methods, achieving up to 1.7× higher face IoU and over 4.5× F1 scores on edges. The results highlight the framework’s robustness, especially for thin, view-sensitive structures.
Overall, MV-GEL represents a major step forward in natural language-driven 3D geometric understanding. Its CAD-agnostic design and multi-view reasoning capabilities make it highly adaptable for industrial CAD editing, robotic manipulation, and scientific modeling. Future work will focus on improving computational efficiency, extending to other 3D data formats, and enhancing natural language robustness, aiming to realize real-time, interactive geometric reasoning systems that bridge the gap between language and complex 3D structures.
Deep Dive
Abstract
Identifying and grounding precise geometric entities, such as edges, planar regions, and curved surfaces within 3D objects, is foundational to computer-aided design (CAD), robotic manipulation, and scientific simulation. Although modern Vision Language Models (VLMs) have advanced referring segmentation (RIS) in the image domain, extending such language-driven localization to structured 3D geometry is substantially harder. The 3D object appearance is highly sensitive to viewpoints; a single perspective may render a target entity clearly observable, while another may suffer from severe occlusion or foreshortening. In this work, we attempt to solve these challenges with MV-GEL (Multi-View Geometric Entity Localization), a framework for localizing fine-grained geometric entities on polygon meshes from natural language queries. Our key insight is that reliable CAD entity (i.e., faces, edges or solids) localization depends on selecting views that make the queried entity maximally interpretable. We introduce GELviews, a prompt-conditioned ranking module that prioritizes viewpoints based on language prompted observability of geometric CAD entities. Selected views are processed by a VLM-based reasoning segmentation backbone, and predicted masks are lifted to the corresponding meshes via geometry-aware ray casting. Our framework is completely CAD agnostic and relies only on 3D meshes. Experiments show up to a 1.7X improvement in face-level IoU and over 4.5X gains in edge-level F1 compared to vanilla baselines, substantially outperforming CLIP-based and random view sampling, particularly for thin and view-sensitive structures. The dataset, code and trained checkpoints are available at https://github.com/kbali1297/MV-GEL.