Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation
HyRo uses hyperbolic rotation to decouple hierarchical and semantic alignment, achieving state-of-the-art open-vocabulary semantic segmentation.
Key Findings
Methodology
HyRo employs the Poincaré ball model to encode hierarchical relations via radius adjustment, while using learnable orthogonal matrices to rotate embeddings in angular space for semantic refinement. Key steps include: • Mapping visual and textual features into hyperbolic space via exponential maps; • Adjusting radii with learnable diagonal matrices to encode hierarchy; • Applying parameterized orthogonal transformations for angular rotation to improve semantic alignment; • Decoding pixel-level predictions from refined embeddings. This approach maintains hierarchical structure and enhances fine-grained semantic discrimination across modalities.
Key Results
- On ADE20K, COCO-Stuff, HyRo achieves 12.0 mIoU on A-847, surpassing HyperCLIP's 11.9 by 0.1; on PC-459, reaches 18.9 mIoU, 0.7 higher; on A-150 and PC-59, respectively 31.2 and 57.3 mIoU, demonstrating strong performance. Ablation studies show that angular rotation significantly boosts accuracy, especially on large-vocabulary datasets, and combining radius scaling with rotation yields optimal results.
Significance
This work advances the understanding of geometric operations in hyperbolic space for multimodal embedding alignment, addressing the challenge of simultaneously modeling hierarchy and semantic similarity. It offers a novel solution that improves generalization to unseen categories, enhances fine-grained recognition, and supports scalable open-vocabulary segmentation, with broad implications for scene understanding and AI applications in industry.
Technical Contribution
The paper introduces a framework that explicitly decouples hierarchical and semantic information in hyperbolic space, leveraging parameterized orthogonal transformations to rotate embeddings without disturbing their radii. Theoretical proofs confirm that such transformations preserve hierarchical structure while refining semantic relationships. This dual geometric manipulation surpasses prior radius-only approaches, enabling more precise cross-modal alignment and fine-grained segmentation performance improvements.
Novelty
This is the first work to systematically incorporate angular rotation in hyperbolic space for semantic alignment, explicitly disentangling hierarchy (radius) and semantics (angle). Unlike previous methods like HyperCLIP, which only scale radii, HyRo introduces learnable orthogonal rotations, providing a new geometric paradigm for multimodal embedding refinement, significantly improving open-vocabulary segmentation accuracy.
Limitations
- The approach may face challenges in highly imbalanced class distributions, where semantic distinctions are subtle or data is scarce. The parameterization of orthogonal matrices can be computationally intensive in very high dimensions. Additionally, current validation is limited to static images, with real-time and dynamic scene adaptation still requiring further research.
Future Work
Future directions include exploring more complex hyperbolic transformations, integrating attention mechanisms for adaptive angular adjustments, extending to video and dynamic scene understanding, and incorporating self-supervised learning to improve robustness and generalization in low-data regimes.
AI Executive Summary
Semantic segmentation at the pixel level with open vocabulary remains a core challenge in computer vision, especially when leveraging large-scale vision-language models like CLIP. Existing approaches often struggle to balance hierarchical structure encoding with fine-grained semantic discrimination, leading to semantic ambiguities and misclassifications. This paper introduces HyRo, a novel framework that operates within the hyperbolic Poincaré space to address these issues.
HyRo's key innovation lies in decoupling hierarchical and semantic alignment through geometric manipulations. It adjusts the hyperbolic radii to encode the hierarchical depth of features, while employing learnable orthogonal rotations to refine the angular relationships that encode semantic similarity. This dual approach allows the model to preserve the hierarchical structure while enhancing semantic discrimination, resulting in more accurate pixel-level predictions.
The core technical mechanism involves mapping features into hyperbolic space via exponential maps, adjusting radii with parameterized diagonal matrices, and applying orthogonal transformations to rotate embeddings without changing their radii. The theoretical foundation confirms that such rotations preserve the hierarchical structure while improving semantic alignment. The refined embeddings are then mapped back to Euclidean space for decoding into segmentation masks.
Experimental results on benchmarks like ADE20K, COCO-Stuff, and PASCAL VOC demonstrate that HyRo outperforms prior methods, achieving 12.0 mIoU on A-847, 18.9 on PC-459, and 31.2 on A-150, among others. Ablation studies reveal that angular rotation contributes significantly to performance gains, especially on large-vocabulary datasets. Qualitative visualizations show sharper, more coherent segmentation masks, confirming the effectiveness of hyperbolic angular refinement.
This work significantly advances the field by providing a geometric framework that explicitly models both hierarchy and semantics, enabling more scalable and precise open-vocabulary segmentation. Future work will explore more complex transformations, real-time adaptation, and broader multimodal applications. Despite current limitations in high-dimensional optimization and dynamic scenes, HyRo opens a promising pathway for multimodal scene understanding and AI generalization.
Deep Dive
Abstract
Open-vocabulary semantic segmentation requires adapting image-level vision-language models such as CLIP to dense pixel-level prediction, which is challenging due to the mismatch between hierarchical structure and semantic alignment in the embedding space. While recent works leverage hyperbolic geometry to model hierarchical relationships, they align embeddings across hierarchical levels but overlook semantic misalignment among embeddings within the same level. In this work, we propose HyRo, a hyperbolic fine-tuning framework that decouples hierarchical and semantic alignment in the Poincaré ball model. HyRo aligns hierarchical levels by adjusting the hyperbolic radius and refines semantic relationships through angular alignment using an orthogonal transformation that theoretically preserves the hyperbolic radius. Experiments on standard open-vocabulary semantic segmentation benchmarks demonstrate that HyRo achieves state-of-the-art performance over prior methods.