OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation
OV-Stitcher employs a global attention mechanism without training, boosting mIoU from 48.7 to 50.7 by stitching sub-image features for open-vocabulary segmentation.
Key Findings
Methodology
OV-Stitcher introduces a 'Stitching Attention' module within the final encoder layer, which aggregates fragmented sub-image features into a unified global representation. This involves extracting query, key, and value vectors from each sub-image, then stitching these features spatially to form a comprehensive global feature map. The global attention is computed via a softmax-based similarity measure, capturing long-range dependencies. Incorporating class-biased textual prompts further enhances semantic alignment. This approach leverages pre-trained vision-language models directly, avoiding additional training, and effectively models full-image context.
Key Results
- Across eight benchmarks, OV-Stitcher improves mean IoU from 48.7 to 50.7, outperforming all prior training-free methods. Notably, on Cityscapes, performance increases by 3.5%, demonstrating superior global context integration.
- Using ViT-B/16 and ViT-L/14 backbones, the method achieves state-of-the-art results, with over 2% gains compared to CorrCLIP. The global feature stitching effectively mitigates feature fragmentation, leading to more coherent segmentation boundaries.
- The approach addresses the limitations of sliding-window strategies by enabling cross-sub-image attention, significantly boosting long-range dependency modeling and semantic consistency.
Significance
This work advances the capability of large pre-trained models for dense pixel-level understanding without additional training. It addresses the core challenge of feature fragmentation caused by local processing, enabling scalable, high-quality open-vocabulary segmentation suitable for real-world large-scale applications such as autonomous driving and urban scene analysis. The method's simplicity and effectiveness open new avenues for deploying powerful vision-language models in practical scenarios.
Technical Contribution
The core innovation is the 'Stitching Attention' mechanism, which transforms local sub-image features into a global context by spatially stitching query, key, and value vectors before attention computation. This design overcomes the inherent limitations of sliding-window inference, enabling the model to reason over entire images. The integration of class-biased prompts further refines semantic predictions, making the approach robust and versatile. This framework leverages existing pre-trained models directly, avoiding costly fine-tuning while achieving state-of-the-art performance.
Novelty
This is the first work to implement a training-free, global context-aware attention mechanism that stitches sub-image features within the encoder layer, effectively solving the feature fragmentation problem in high-resolution dense prediction. Unlike prior local attention or post-processing methods, OV-Stitcher directly incorporates cross-region interactions into the core feature extraction process, setting a new standard for open-vocabulary segmentation.
Limitations
- The reliance on external mask generators like SAM increases computational overhead. In extremely complex or cluttered scenes, errors may propagate through the stitching process.
- Handling very large scenes may pose memory and efficiency challenges due to the full-image feature stitching, requiring further optimization.
- Current validation is limited to static images; extending to video or multi-modal data remains future work.
Future Work
Future directions include multi-scale feature fusion, integrating self-supervised learning to improve robustness, and optimizing the stitching process for real-time deployment. Exploring adaptive stitching strategies for dynamic scenes and extending the framework to video understanding are promising avenues.
AI Executive Summary
The rapid growth of large-scale pre-trained vision-language models like CLIP has revolutionized visual recognition tasks. However, applying these models directly to high-resolution dense prediction, such as semantic segmentation, faces significant challenges due to input resolution limits and feature fragmentation. Traditional sliding-window approaches, while effective in managing resolution, fragment the image into independent patches, preventing the model from capturing global context and long-range dependencies. This fragmentation hampers semantic coherence, especially in complex scenes with large objects or intricate backgrounds.
To address this, the proposed OV-Stitcher framework introduces a novel 'Stitching Attention' mechanism that operates within the final encoder layer. By extracting query, key, and value vectors from each sub-image and spatially stitching them into a unified global feature map, OV-Stitcher enables the model to perform attention across the entire image. This process effectively reconstructs global dependencies, leading to more coherent and spatially consistent segmentation maps. The approach also incorporates class-biased textual prompts, which guide semantic alignment and reduce ambiguity.
Extensive experiments across eight benchmark datasets, including PASCAL VOC, COCO, Cityscapes, and ADE20K, demonstrate that OV-Stitcher outperforms existing training-free methods, achieving an average mIoU of 50.7 compared to 48.7. Notably, in large and complex scenes like Cityscapes, performance gains exceed 3%, validating the effectiveness of global feature stitching. The framework's simplicity, combined with its ability to leverage pre-trained models directly, makes it highly scalable and practical for real-world applications such as autonomous driving, urban scene understanding, and robotic perception.
Looking ahead, future work will focus on multi-scale fusion, efficiency improvements, and extending the approach to dynamic and multi-modal data. Overall, OV-Stitcher marks a significant step toward scalable, training-free open-vocabulary dense prediction, unlocking new potentials for vision-language models in large-scale scene understanding.
Deep Analysis
Background
近年来，预训练模型如CLIP、DINO在视觉任务中展现出强大能力，但其在高分辨率像素级任务中的应用仍受限于输入分辨率。传统方法采用滑动窗口策略，逐块处理图像，虽能缓解输入限制，却导致特征碎片化，缺乏全局上下文，影响语义一致性。已有研究如ProxyCLIP、SAM尝试通过空间相似性或掩码增强局部信息，但未能根本解决全局交互缺失的问题。随着大模型的普及，如何在无微调条件下实现跨子图的全局建模，成为亟待突破的难题。
Core Problem
滑动窗口策略虽然解决了输入分辨率问题，但限制了跨区域的注意力交互，导致特征碎片化，影响长距离依赖建模。缺乏全局上下文使得模型在复杂场景中表现不稳定，边界模糊，语义不连贯。现有方法难以在保持高分辨率的同时实现全局一致性，限制了其在实际大场景中的应用潜力。
Innovation
提出“缝合注意力”机制，将子图特征在编码器最后一层空间上“缝合”成全局特征，突破局部窗口限制，增强全局依赖。结合类别偏置文本提示，提升语义对齐的鲁棒性。该方案无需微调，直接利用预训练模型的知识，显著改善特征一致性和边界精度。创新点在于：•在编码器最后一层引入全局特征融合；•利用“缝合”操作实现跨子图信息交互；•结合文本提示增强语义表达。
Methodology
- ��输入图像通过滑动窗口划分为多个子图；•每个子图经过预训练视觉编码器提取特征；•在最后一层，将子图特征的查询、键、值向量空间“缝合”成全局特征；•利用全局注意力机制（如softmax(QK^T/τ)V）计算全局特征的关系；•结合类别偏置文本提示，增强语义一致性；•最终输出融合全局信息的像素级分割结果。整个流程实现了无微调条件下的全局上下文建模，显著提升了分割的连贯性和准确性。
Experiments
在PASCAL VOC、COCO、Cityscapes、ADE20K等8个公开数据集上，采用不同主干（ViT-B/16、ViT-L/14）进行评估。对比基线包括CorrCLIP、MaskCLIP等，指标为mIoU。采用336×336裁剪，步长112像素，结合SAM掩码进行后处理。通过消除碎片化特征，验证“缝合注意力”在复杂场景中的优势。实验证明，OV-Stitcher在所有数据集上均优于对比方法，尤其在Cityscapes中提升3.5%以上。
Results
在8个基准中，平均mIoU由48.7提升到50.7，优于所有训练无关方法。Cityscapes场景中，性能提升显著，达3.5%，验证了全局特征融合的效果。与CorrCLIP等对比，性能提升超过2%，表明跨子图全局建模有效增强了语义连贯性。消除了特征碎片化，提升了边界清晰度和长距离依赖建模能力。
Applications
该方法适用于自动驾驶、城市规划、机器人感知等需要大场景高精度像素级理解的场景。无需微调，直接利用预训练模型，便于快速部署，提升系统的鲁棒性和效率。未来可结合多模态信息，拓展到视频和动态场景，推动智能感知技术的发展。
Limitations & Outlook
依赖SAM等外部掩码模型，增加计算复杂度。在极端复杂或类别模糊场景中，可能出现误差传播。全局特征缝合在超大场景中存在内存瓶颈，需优化算法效率。当前主要验证静态图像，动态视频和多模态场景的适应性仍需验证。
Plain Language Accessible to non-experts
想象你在做拼图游戏，把一幅大画分成许多小块，每块都很清楚，但拼在一起时，可能会出现缝隙或错位。传统的方法就像只看每一块，不能看到整体，导致拼出来的画不完整。OV-Stitcher就像用一种特殊的胶，把这些小块在拼图的最后一刻粘在一起，确保每一块都对齐，形成完整的画面。这样，不仅每一块都清楚，还能看到整幅画的全貌。它不用重新画，也不用调试，只用已有的拼图块，巧妙地把它们拼成一幅完整、清晰的画面。这就像在大场景中拼图一样，能更好地理解每个部分的关系，整体效果更自然、更连贯。
ELI14 Explained like you're 14
想象你在玩拼图游戏，把一幅大图片切成很多小块。每次你只看一块，拼完后发现有些地方不对，整体看起来不自然。这是因为你只关注每一块，没法看到全局。OV-Stitcher就像用一种神奇的胶，把所有的小块在拼图的最后一刻粘在一起，让它们完美对齐。这样，你就可以看到一幅完整的漂亮画面，不会有缝隙或错位。它不用重新画，也不用调试，只用已有的拼图块，就能拼出一幅完整的画。这就像在大场景中拼图一样，能让你更好地理解每个部分的关系，整个画面看起来更自然、更连贯。这个方法让电脑也能像你一样，把很多小图片拼成一张大图，变得更聪明、更懂得整体的意思。
Glossary
全局注意力 (Global Attention)
一种机制，使模型在处理某一部分时，也考虑整个图像的所有区域，从而理解整体关系。
在OV-Stitcher中，用于融合子图特征，增强全局上下文建模。
缝合注意力 (Stitch Attention)
一种创新的注意力机制，将分割的子图特征在空间上“缝合”成全局特征，突破局部窗口限制。
论文的核心技术，用于实现跨子图的全局信息交互。
类别偏置文本提示 (Class-Biased Prompts)
在文本提示中加入类别特定的描述，增强模型对类别的语义区分能力。
提升分割的语义对齐和分类准确性。
滑动窗口 (Sliding-Window)
将大图划分为多个重叠的小块，逐一处理以适应模型输入限制。
传统方法的基础，但导致特征碎片化。
预训练模型 (Pretrained Model)
在大规模数据上训练好的模型，具备丰富的知识和泛化能力。
OV-Stitcher直接利用其潜能，无需微调。
Open Questions Unanswered questions from this research
- 1 如何进一步提升全局特征融合在极大场景中的效率和内存管理仍是挑战。未来需探索更高效的算法和硬件支持，以实现实时大场景理解。
Applications
Immediate Applications
自动驾驶场景理解
利用OV-Stitcher实现道路、行人、交通标志的高精度像素级识别，提升自动驾驶系统的安全性和鲁棒性。
城市规划与监控
在城市大场景中进行高效的场景分析和目标检测，为智能交通和公共安全提供支持。
Long-term Vision
智能机器人感知
赋能机器人在复杂环境中实现自主导航和交互，推动智能制造和服务机器人技术发展。
Abstract
Training-free open-vocabulary semantic segmentation(TF-OVSS) has recently attracted attention for its ability to perform dense prediction by leveraging the pretrained knowledge of large vision and vision-language models, without requiring additional training. However, due to the limited input resolution of these pretrained encoders, existing TF-OVSS methods commonly adopt a sliding-window strategy that processes cropped sub-images independently. While effective for managing high-resolution inputs, this approach prevents global attention over the full image, leading to fragmented feature representations and limited contextual reasoning. We propose OV-Stitcher, a training-free framework that addresses this limitation by stitching fragmented sub-image features directly within the final encoder block. By reconstructing attention representations from fragmented sub-image features, OV-Stitcher enables global attention within the final encoder block, producing coherent context aggregation and spatially consistent, semantically aligned segmentation maps. Extensive evaluations across eight benchmarks demonstrate that OV-Stitcher establishes a scalable and effective solution for open-vocabulary segmentation, achieving a notable improvement in mean Intersection over Union(mIoU) from 48.7 to 50.7 compared with prior training-free baselines.