RbA: Segmenting Unknown Regions Rejected by All
Proposes RbA, a region-level outlier scoring method based on 'rejected by all,' improving unknown object segmentation with minimal supervision.
Key Findings
Methodology
Built on Mask2Former, the approach analyzes object queries' 'one-vs-all' behavior, defining outliers as regions rejected by all known classes. A novel RbA score is formulated, leveraging the spatial smoothness and confidence calibration of mask classification models. By minimally fine-tuning with synthetic outliers, the method enhances unknown object detection while maintaining known class accuracy. Extensive experiments on Cityscapes, SMIYC, Road Anomaly, and Fishyscapes datasets demonstrate significant improvements over state-of-the-art methods, reducing false positives and boundary errors. The core innovation lies in modeling the 'rejection' event, which better captures the uncertainty in ambiguous regions, especially at semantic boundaries.
Key Results
- On SMIYC anomaly track, RbA achieves 78.45% AP and reduces FPR to 11.83%, outperforming existing methods by over 20%, indicating superior unknown object detection.
- On Road Anomaly and Fishyscapes LaF, FPR drops to 6.92% and 6.30%, respectively, with no degradation in known class recognition, validating robustness.
- Minimal synthetic outlier supervision during fine-tuning boosts generalization across diverse scenarios, confirming practical applicability.
Significance
This work addresses the limitations of pixel-level classification by introducing region-level segmentation and a novel rejection-based scoring mechanism. It effectively reduces false positives at boundaries and ambiguous regions, crucial for safety-critical applications like autonomous driving. The approach leverages the inherent properties of mask classification models, such as spatial smoothness and confidence calibration, to improve out-of-distribution detection without extensive external data. It bridges the gap between closed-set recognition and open-set detection, offering a scalable, efficient solution that maintains high inlier accuracy while significantly enhancing unknown object segmentation. The method’s minimal supervision requirement and compatibility with existing architectures make it highly promising for real-world deployment.
Technical Contribution
The key technical innovation is the formulation of the RbA score based on the 'all classes reject' hypothesis, exploiting the implicit binary classifier behavior of object queries in mask models. This approach departs from traditional max-logit or energy-based scores, providing better calibration and boundary handling. The paper introduces a loss function that directly optimizes RbA with limited outlier supervision, enabling effective regularization without retraining the entire model. The analysis of object query independence and the clustering of logits into distinct modes further deepen the understanding of model behavior, guiding the design of more robust outlier detection strategies. These contributions collectively advance the theoretical understanding and practical performance of open-set semantic segmentation.
Novelty
This research is the first to systematically analyze the 'class-specific query' behavior in mask classification models, revealing their 'one-vs-all' nature. It introduces the concept of defining outliers as regions rejected by all known classes, a novel perspective that significantly improves boundary and ambiguous region detection. Unlike existing methods relying on confidence calibration or external outlier data, this approach leverages the intrinsic properties of mask models, offering a more principled and effective outlier scoring mechanism. The minimal supervision fine-tuning strategy further distinguishes this work, enabling high performance with limited labeled outlier data, a notable advancement over prior fully supervised or unsupervised methods.
Limitations
- Despite its robustness, the method may still struggle with extremely cluttered or occluded scenes where object queries cannot reliably distinguish unknowns. The reliance on synthetic outliers for fine-tuning, although minimal, might limit adaptation to highly diverse real-world anomalies. Additionally, the approach's effectiveness across different scales and complex multi-object scenarios requires further validation, especially in real-time applications where computational efficiency is critical.
Future Work
Future research could explore integrating multi-modal data such as LiDAR or radar to enhance detection robustness in adverse conditions. Developing self-supervised or unsupervised strategies for outlier supervision could reduce reliance on synthetic data, improving adaptability. Extending the framework to multi-scale and multi-class scenarios, along with optimizing inference speed, will be crucial for deployment in real-world autonomous systems. Further theoretical analysis of the query behavior and rejection mechanisms may also lead to more refined scoring functions, pushing the boundaries of open-set semantic understanding.
AI Executive Summary
In recent years, semantic segmentation has become a cornerstone of scene understanding, especially in applications like autonomous driving and surveillance. Traditional models excel in closed-set environments, where all categories are known during training. However, real-world scenarios often present unknown objects—wild animals, debris, or novel obstacles—that these models struggle to detect reliably. Existing out-of-distribution detection methods, based on pixel-level classification and confidence scores, suffer from prediction noise, boundary ambiguity, and high false positive rates, limiting their practical utility.
This paper introduces RbA (Rejected by All), a novel outlier scoring mechanism built upon mask classification architectures like Mask2Former. By analyzing the behavior of object queries, the authors discover that these queries tend to behave like independent binary classifiers, effectively acting as 'one-vs-all' detectors. Leveraging this insight, they define the outlier probability as the complement of the sum of known class probabilities, representing the event that a region is rejected by all known categories. This approach inherently benefits from the spatial smoothness and confidence calibration of mask models, leading to more accurate and less noisy unknown region detection.
Extensive experiments on datasets such as Cityscapes, SMIYC, Road Anomaly, and Fishyscapes demonstrate that RbA outperforms existing methods, achieving higher AP scores and lower false positive rates. Notably, the method requires only minimal supervision with synthetic outliers for fine-tuning, avoiding the performance degradation seen in prior approaches that heavily rely on external outlier data. The results confirm that RbA effectively balances the detection of unknown objects with the preservation of known class accuracy, making it highly suitable for safety-critical applications.
The core innovation lies in exploiting the intrinsic properties of mask-based models, transforming object queries into a robust rejection mechanism. This paradigm shift opens new avenues for open-set semantic understanding, with promising implications for real-world deployment. Future directions include integrating multi-modal data, enhancing scalability, and developing fully unsupervised outlier detection strategies, aiming to create autonomous systems that are both intelligent and safe in unpredictable environments.
Deep Analysis
Background
语义分割技术经历了从早期像素级卷积网络到Transformer架构的演变，代表性工作包括FCN、DeepLab、MaskFormer等。尽管在封闭类别集上取得了巨大成功，但在开放集环境中，模型常因过度自信而误判未知目标，带来安全隐患。异常检测方法逐渐兴起，结合贝叶斯、能量评分等技术，但多依赖像素级分类，存在预测不平滑、边界模糊的问题。近年来，区域级掩码模型如Mask2Former展现出空间平滑和置信校准的优势，为未知区域检测提供了新思路。
Core Problem
现有方法在未知目标检测中存在预测不平滑、误报率高、边界模糊等问题。像素级分类的局限性导致模型在复杂场景中难以准确区分未知目标与背景，尤其在边界区域易出现误判。此外，依赖大量外部异常样本进行训练，成本高且泛化能力有限。如何利用区域级模型的空间平滑性，设计更鲁棒的未知区域检测机制，成为亟待解决的核心难题。
Innovation
提出RbA（Rejected by All）评分机制，基于掩码模型中对象查询的“类一对多”行为，将未知区域定义为被所有已知类别拒绝的事件，突破像素级分类的限制。利用空间平滑性和置信校准，降低边界和背景误判。引入少监督的微调策略，结合合成异常样本，增强模型泛化能力。这一创新在保持已知类别性能的基础上，大幅提升未知区域检测的准确性，为区域级语义理解开辟新路径。
Methodology
- �� 构建在Mask2Former架构基础上，利用区域级掩码预测和对象查询机制。
- �� 分析对象查询的“类一对多”行为，发现其表现出类似多个独立二分类器的特性。
- �� 定义RbA评分，将未知区域视为被所有已知类别拒绝的事件，计算公式为：RbA(x) = -∑σ(Lk(x))，其中σ为激活函数，Lk(x)为类别k的对数it。
- �� 设计少监督的微调目标，通过合成异常样本优化RbA评分，提升未知区域检测能力。
- �� 结合空间平滑性和置信校准，降低边界和背景误判，改善检测效果。
Experiments
- �� 在Cityscapes训练，使用Swin-B作为骨架，单层Transformer解码器。
- �� 在SMIYC、Road Anomaly和Fishyscapes等数据集上进行评估，比较FPR、AP、sIoU等指标。
- �� 采用少量合成异常样本进行微调，验证模型泛化。
- �� 进行消融实验，分析不同评分机制和微调策略的效果。
Results
- �� 在SMIYC异常轨道，AP达78.45%，FPR降至11.83%，优于现有方法20%以上。
- �� 在Road Anomaly和Fishyscapes LaF上，FPR分别为6.92%和6.30%，保持已知类别识别性能。
- �� 微调后，模型在多场景中表现出良好的泛化能力，验证了其实用性。
Applications
- �� 自动驾驶：实时检测未知障碍物，提升行车安全。
- �� 智能监控：识别未标注的异常目标，增强安全监控能力。
Limitations & Outlook
- �� 在极端复杂环境中，遮挡或极小目标仍可能导致误判。
- �� 微调策略依赖合成样本，可能在某些场景下表现不稳定。
- �� 多尺度、多类别场景的适应性有待验证，未来需结合多尺度特征增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房准备一道菜，菜单上列出几种常见的食材，比如土豆、胡萝卜和肉，但偶尔会遇到一些菜单上没有写的新奇食材。普通厨师只知道菜单上的食材，遇到新食材时可能会误判或忽略。现在，假设你有一个神奇的厨具，可以判断每个食材是否属于已知类别，或者是新奇的。这个厨具会根据食材的特征，判断它是不是“被所有已知类别拒绝”的新食材。这样一来，你就能更准确地发现未知的食材，避免误用或遗漏。论文中的RbA方法也是一样，它用区域级的识别机制，判断某个区域是否“被所有已知类别拒绝”，从而更好地识别新奇目标。
ELI14 Explained like you're 14
想象你在学校玩一个宝藏游戏，老师给你几种已知的宝藏（比如金子、宝石、金币），但有时候会出现一些新奇的宝藏。普通的规则只认这些已知宝藏，遇到新宝藏时可能会误判成已知的。现在，假设你有一个神奇的检测器，它可以告诉你这个宝藏是不是“被所有已知宝藏拒绝”的，也就是说，它确认这个宝藏不属于任何已知类别。这样你就能更好地找到那些新奇的宝藏，不会把它们误判成已知的。论文里的方法也是一样，它用一种区域级的检测方式，判断某个区域是不是“被所有已知类别拒绝”，从而更好地发现未知目标。这让自动驾驶汽车或监控系统变得更聪明、更安全，因为它们能更好地识别出新出现的危险或异常。
Glossary
掩码分类 (Mask Classification)
一种将图像区域划分为不同掩码的技术，用于目标检测和语义理解，区别于像素级分类。
本文基于Mask2Former模型，利用掩码分类实现区域级目标识别。
对象查询 (Object Queries)
在Transformer架构中，用于引导模型关注特定目标的可学习向量，帮助实现目标区域的预测。
分析对象查询的“类一对多”行为，提出RbA评分机制。
RbA (Rejected by All)
一种基于区域的异常评分机制，定义为区域被所有已知类别拒绝的概率，提升未知目标检测。
核心创新，用于区分已知类别和未知区域。
区域级分类 (Region-level Classification)
将图像划分为多个区域，分别进行类别判定，增强空间平滑性和置信校准。
替代像素级分类，改善未知目标检测。
少监督微调 (Minimal Supervision Fine-tuning)
用少量合成异常样本微调模型部分参数，提升泛化能力。
验证模型在多场景中的应用潜力。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂环境中保持高精度的未知区域检测仍是挑战，尤其在遮挡和极小目标场景下，模型的鲁棒性有待提升。未来需要结合多模态信息和更强的自监督机制，以实现更全面的场景理解和安全保障。
Applications
Immediate Applications
自动驾驶安全系统
利用RbA实时检测未知障碍物，提升自动驾驶在复杂环境中的安全性，减少误判和漏检。
智能监控预警
在安防系统中识别未标注的异常行为或目标，增强监控的智能化水平。
Long-term Vision
全场景自主感知
结合多模态数据，实现对各种未知目标的全面感知，推动自动驾驶和机器人自主导航的普及。
Abstract
Standard semantic segmentation models owe their success to curated datasets with a fixed set of semantic categories, without contemplating the possibility of identifying unknown objects from novel categories. Existing methods in outlier detection suffer from a lack of smoothness and objectness in their predictions, due to limitations of the per-pixel classification paradigm. Furthermore, additional training for detecting outliers harms the performance of known classes. In this paper, we explore another paradigm with region-level classification to better segment unknown objects. We show that the object queries in mask classification tend to behave like one \vs all classifiers. Based on this finding, we propose a novel outlier scoring function called RbA by defining the event of being an outlier as being rejected by all known classes. Our extensive experiments show that mask classification improves the performance of the existing outlier detection methods, and the best results are achieved with the proposed RbA. We also propose an objective to optimize RbA using minimal outlier supervision. Further fine-tuning with outliers improves the unknown performance, and unlike previous methods, it does not degrade the inlier performance.