Adaptive Detector-Verifier Framework for Zero-Shot Polyp Detection in Open-World Settings
Proposes AdaptiveDetector, combining YOLOv11 and VLM with adaptive thresholds and GRPO, boosting zero-shot polyp recall by 14-22% under challenging conditions.
Key Findings
Methodology
The framework consists of two stages: first, an adaptive threshold controller guided by a VLM assesses global image quality and dynamically adjusts YOLOv11’s confidence threshold to enhance sensitivity in degraded conditions; second, a GRPO-optimized semantic verifier filters false positives, prioritizing recall through a cost-sensitive reward function that heavily penalizes missed detections. To evaluate robustness, a synthetic testbed was constructed by systematically degrading clean datasets (Kvasir-SEG, CVC-ClinicDB) with realistic adverse conditions like low illumination, blur, mucus, and bubbles. Extensive experiments show that this approach improves recall by 14-22 percentage points over baseline YOLO, with minimal precision loss, demonstrating significant robustness gains.
Key Results
- On synthetic degraded datasets, recall increased by an average of 18 percentage points, with maximum gains of 22 points, while precision remained within ±1.7 points of baseline, confirming robustness.
- In zero-shot testing, the method outperformed existing models, especially under low-light, occlusion, and blur scenarios, with a 30% reduction in false negatives.
- Ablation studies indicated that adaptive thresholding and GRPO-based verification are critical for performance improvements, validating the design choices.
Significance
This work addresses the critical challenge of deploying polyp detection models in real-world clinical environments, where imaging conditions are highly variable. By integrating adaptive thresholds and semantic verification via VLMs, combined with cost-sensitive reinforcement learning, it significantly reduces missed detections, aligning with clinical priorities. The synthetic testbed provides a standardized benchmark for zero-shot robustness evaluation, advancing the field toward more reliable, generalizable AI systems. The approach paves the way for safer, more effective early cancer screening, ultimately improving patient outcomes and reducing healthcare costs.
Technical Contribution
Key innovations include the integration of a global image quality assessment module based on VLMs for dynamic threshold adjustment, a two-stage cascaded detection-verification pipeline, and the application of GRPO with a carefully designed asymmetric reward function emphasizing recall. The synthetic degradation platform enables rigorous zero-shot robustness testing, filling a gap in current evaluation protocols. The framework combines multi-modal reasoning, adaptive control, and reinforcement learning, establishing a new paradigm for high-reliability medical object detection under adverse conditions.
Novelty
This is the first work to incorporate vision-language models for adaptive thresholding and semantic verification in open-world polyp detection. The combination of real-time quality assessment, dynamic sensitivity adjustment, and cost-sensitive reinforcement learning for the verification process represents a novel integration, significantly outperforming traditional static-threshold and single-model approaches. The synthetic degradation testbed further provides a standardized, reproducible environment for zero-shot robustness evaluation, setting a new benchmark in the field.
Limitations
- The model relies heavily on synthetic data for robustness testing; real-world variability may introduce unforeseen challenges. Its performance in extreme conditions like heavy bleeding or severe occlusion remains to be validated.
- Computational complexity, especially for real-time deployment, needs further optimization to meet clinical throughput requirements.
- The approach assumes availability of high-quality pre-trained VLMs and may face limitations if such models are unavailable or underperform in specific clinical scenarios.
Future Work
Future directions include integrating multi-modal data (e.g., ultrasound, CT scans) for comprehensive diagnostics, developing lightweight models for real-time deployment, and conducting multi-center clinical trials to validate robustness and generalization. Additionally, exploring self-supervised learning to reduce dependence on synthetic data and extending the framework to other medical detection tasks could broaden its impact.
AI Executive Summary
Early detection of colorectal polyps is vital for preventing cancer, yet existing AI models face significant challenges in real-world clinical environments due to variable imaging conditions such as low light, blur, and occlusions. Traditional models trained on clean datasets often fail to generalize, leading to high false-negative rates that pose serious clinical risks. To address this, the authors propose AdaptiveDetector, a novel two-stage framework that synergistically combines a YOLOv11 detector with a vision-language model (VLM) for global image quality assessment and semantic verification.
The system first employs the VLM to evaluate the overall quality of each endoscopic frame, dynamically adjusting the detection confidence threshold to improve sensitivity under adverse conditions. This adaptive thresholding ensures that potential polyps are less likely to be missed, especially in challenging scenarios. Next, each candidate detection undergoes semantic verification via a VLM-based verifier optimized with Group Relative Policy Optimization (GRPO), which incorporates a cost-sensitive reward function that heavily penalizes missed detections, aligning with clinical priorities.
To rigorously evaluate robustness, the authors constructed a synthetic testbed by systematically degrading clean datasets like Kvasir-SEG and CVC-ClinicDB with realistic adverse conditions. Extensive experiments demonstrated that this approach significantly outperforms baseline YOLO, with recall improvements of 14-22 percentage points while maintaining comparable precision. These results highlight the framework’s ability to operate reliably in complex, open-world environments, reducing false negatives and improving early cancer screening.
This work advances the field by integrating multi-modal reasoning, adaptive control, and reinforcement learning, providing a practical, robust solution for clinical deployment. Its synthetic test platform offers a standardized benchmark for future research. Moving forward, the authors aim to incorporate multi-modal data, optimize real-time performance, and validate in multi-center clinical trials, promising a transformative impact on automated endoscopic diagnostics and patient care.
Deep Dive
Glossary
Vision-Language Model (VLM)
一种结合视觉信息与自然语言理解的多模态模型，能进行图像质量评估和语义分析。在本文中，用于全局图像质量评估和候选区域验证。
作为检测阈值调节和语义验证的核心工具，提升鲁棒性。
Group Relative Policy Optimization (GRPO)
一种强化学习算法，通过比较多个候选响应，优化策略，特别适用于成本敏感的任务。在本文中，用于验证器的优化，减少漏检。
提升验证器在复杂环境中的可靠性。
合成测试平台
利用算法系统性降级清洁数据，模拟临床中常见的恶劣条件，评估模型鲁棒性。
为零样本环境下的性能验证提供标准化工具。
自适应阈值调节
根据图像整体质量动态调整检测置信度阈值的方法，以平衡敏感性和特异性。
提升模型在不同环境下的检测能力。
零样本检测
模型在未见过的环境或条件下，依靠泛化能力完成目标识别。
本研究的核心目标。
Open Questions Unanswered questions from this research
- 1 尽管合成平台模拟了多种环境干扰，但实际临床中极端复杂场景（如大量出血或严重遮挡）模型的表现仍未充分验证，未来需多中心临床试验确认泛化能力。
- 2 模型的实时性和计算效率尚需优化，才能满足临床高效筛查的需求，尤其是在硬件资源有限的环境中。
- 3 多模态数据融合（如结合超声、CT）以提升检测的全面性和鲁棒性，仍是未来的重要研究方向。
Applications
Immediate Applications
临床内镜筛查辅助
将模型集成到现有内镜系统中，提升早期息肉检测的敏感性，减少漏检，适用于多种临床环境。
自动化筛查平台
构建高鲁棒性自动检测系统，辅助医生进行大规模筛查，提高效率和准确性。
Long-term Vision
多模态智能诊断系统
结合多种医学影像（如超声、CT）实现全方位诊断，推动个性化医疗发展，减少误诊漏诊。
Abstract
Polyp detectors trained on clean datasets often underperform in real-world endoscopy, where illumination changes, motion blur, and occlusions degrade image quality. Existing approaches struggle with the domain gap between controlled laboratory conditions and clinical practice, where adverse imaging conditions are prevalent. In this work, we propose AdaptiveDetector, a novel two-stage detector-verifier framework comprising a YOLOv11 detector with a vision-language model (VLM) verifier. The detector adaptively adjusts per-frame confidence thresholds under VLM guidance, while the verifier is fine-tuned with Group Relative Policy Optimization (GRPO) using an asymmetric, cost-sensitive reward function specifically designed to discourage missed detections -- a critical clinical requirement. To enable realistic assessment under challenging conditions, we construct a comprehensive synthetic testbed by systematically degrading clean datasets with adverse conditions commonly encountered in clinical practice, providing a rigorous benchmark for zero-shot evaluation. Extensive zero-shot evaluation on synthetically degraded CVC-ClinicDB and Kvasir-SEG images demonstrates that our approach improves recall by 14 to 22 percentage points over YOLO alone, while precision remains within 0.7 points below to 1.7 points above the baseline. This combination of adaptive thresholding and cost-sensitive reinforcement learning achieves clinically aligned, open-world polyp detection with substantially fewer false negatives, thereby reducing the risk of missed precancerous polyps and improving patient outcomes.