Prohibited Items Segmentation via Occlusion-aware Bilayer Modeling
Proposed occlusion-aware bilayer mask decoder with SAM integration achieves 94.4% mAP on prohibited item segmentation in X-ray images.
Key Findings
Methodology
This paper introduces an occlusion-aware instance segmentation framework combining SAM (Segment Anything Model) with a bilayer mask decoder. Leveraging SAM’s zero-shot generalization, the approach integrates Mask R-CNN for region proposals, then employs a dual-decoder structure to explicitly model occlusion relationships. The first decoder estimates occlusion areas, while the second refines the segmentation guided by occlusion cues. Manually annotated occlusion masks on large-scale datasets PIDray and PIXray form the supervision basis. The training employs a multi-task loss balancing classification, bounding box, mask, and occlusion estimation, ensuring robust performance in overlapping scenarios.
Key Results
- On the PIDray-A and PIXray-A datasets, the proposed method achieves state-of-the-art performance with [email protected] of 94.4% and 83.7%, surpassing baseline models like Mask R-CNN by 3-4%. The approach excels particularly in occluded scenarios, with over 5% improvement in occlusion region detection accuracy.
- Ablation studies confirm that occlusion modeling and residual feature guidance contribute significantly, with performance gains of over 1% across metrics. The method maintains robustness across various subsets, including heavily occluded images.
- Qualitative results demonstrate the model’s ability to distinguish overlapping prohibited items, correctly segmenting objects like covered hammers and power banks, outperforming existing methods in complex scenes.
Significance
This work addresses a critical challenge in automated security inspection—accurately segmenting prohibited items amid heavy occlusion and overlap in X-ray images. By integrating SAM’s powerful generalization with explicit occlusion modeling, it offers a practical solution for real-world deployment in airports and security checkpoints. The creation of occlusion-annotated datasets further advances the field, enabling more precise training and evaluation. The approach significantly improves detection accuracy, reducing false negatives and enhancing safety measures, thus contributing to safer public environments.
Technical Contribution
The paper’s key innovation lies in combining SAM’s pre-trained image priors with a novel bilayer mask decoder that explicitly models occlusion relationships. This dual-decoder architecture, guided by manually annotated occlusion masks, allows the model to distinguish overlapping objects accurately. The multi-task training framework, incorporating occlusion loss, enhances the model’s ability to handle complex overlapping scenes. Additionally, the construction of occlusion-annotated datasets (PIDray-A, PIXray-A) provides valuable resources for future research, addressing the scarcity of ground-truth occlusion labels in security image datasets.
Novelty
This is the first work to incorporate SAM into prohibited item segmentation in X-ray images, specifically addressing occlusion through a bilayer mask decoder. Unlike prior methods that treat occlusion implicitly, this approach explicitly models occlusion relationships, leading to superior performance in overlapping scenarios. The manual annotation of occlusion masks on large datasets further distinguishes this work, setting a new benchmark for occlusion-aware segmentation in security imaging.
Limitations
- Despite improvements, the model still struggles with extreme occlusion cases where objects are heavily covered, due to limitations in occlusion mask annotations and model capacity. Annotating occlusion regions is labor-intensive, limiting scalability.
- The reliance on manually annotated occlusion masks introduces potential biases and inconsistencies, which may affect generalization across different datasets or real-world scenarios.
- Computational complexity increases with the dual-decoder structure, potentially impacting real-time deployment in high-throughput environments.
Future Work
Future directions include automating occlusion mask annotation via semi-supervised learning, extending the framework to multi-modal data (e.g., infrared, depth), and optimizing the model for faster inference. Further research could explore unsupervised occlusion modeling and domain adaptation to enhance robustness across diverse security scenarios.
AI Executive Summary
Automated detection of prohibited items in security X-ray images is vital for safeguarding public spaces. However, existing methods face significant challenges when objects overlap or occlude each other, leading to missed detections and inaccurate segmentation. Traditional approaches often treat occlusion implicitly, which limits their effectiveness in complex scenarios. Recognizing this gap, the present study introduces a novel occlusion-aware instance segmentation pipeline that leverages the powerful generalization capabilities of the Segment Anything Model (SAM).
The core innovation is a bilayer mask decoder architecture designed to explicitly model occlusion relationships. The first decoder estimates occlusion areas, while the second refines object masks guided by occlusion cues. To supervise this process, the authors manually annotated occlusion regions in large-scale datasets PIDray and PIXray, creating occlusion-annotated versions PIDray-A and PIXray-A. These datasets enable the model to learn occlusion patterns effectively.
Experimental results demonstrate that the proposed method outperforms existing models like Mask R-CNN and Cascade Mask R-CNN, achieving a mean Average Precision (mAP) of 94.4% at IoU=0.5 on the PIDray-A dataset. The approach excels in scenarios with severe overlap, accurately distinguishing and segmenting multiple prohibited items, such as overlapping power banks and covered hammers. Ablation studies confirm the importance of occlusion modeling and residual feature guidance, with performance gains of over 1% across key metrics.
This research significantly advances automated security inspection, providing a robust solution for complex occlusion scenarios. By explicitly modeling occlusion relationships and enriching datasets with occlusion annotations, it paves the way for more reliable and precise detection systems. Future work aims to automate occlusion annotation and incorporate multi-modal data to further enhance robustness and deployment speed, ultimately contributing to safer public environments worldwide.
Deep Dive
Abstract
Instance segmentation of prohibited items in security X-ray images is a critical yet challenging task. This is mainly caused by the significant appearance gap between prohibited items in X-ray images and natural objects, as well as the severe overlapping among objects in X-ray images. To address these issues, we propose an occlusion-aware instance segmentation pipeline designed to identify prohibited items in X-ray images. Specifically, to bridge the representation gap, we integrate the Segment Anything Model (SAM) into our pipeline, taking advantage of its rich priors and zero-shot generalization capabilities. To address the overlap between prohibited items, we design an occlusion-aware bilayer mask decoder module that explicitly models the occlusion relationships. To supervise occlusion estimation, we manually annotated occlusion areas of prohibited items in two large-scale X-ray image segmentation datasets, PIDray and PIXray. We then reorganized these additional annotations together with the original information as two occlusion-annotated datasets, PIDray-A and PIXray-A. Extensive experimental results on these occlusion-annotated datasets demonstrate the effectiveness of our proposed method. The datasets and codes are available at: https://github.com/Ryh1218/Occ