IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
Proposes IoU-PD, leveraging true bounding boxes as privileged training guidance, significantly improving multimodal large model's visual grounding accuracy by ~4.25% in mIoU.
Key Findings
Methodology
IoU-PD integrates supervised fine-tuning with privileged distillation, using ground-truth boxes as additional visual guidance during training. The student model receives original images and prompts, while a frozen teacher processes images with marked bounding boxes and augmented prompts indicating the target region. An IoU-aware token weighting scheme adjusts the distillation loss based on geometric importance and teacher reliability, aligning training signals with evaluation metrics. This approach enables the model to learn spatial relationships more effectively without altering inference input formats, resulting in consistent improvements across benchmarks.
Key Results
- On RefCOCO, IoU-PD achieves a 4.04-point increase in mIoU (from 81.74 to 85.78) and a 4.25-point boost in [email protected] (from 88.58 to 91.56). Similar gains are observed on RefCOCO+ and RefCOCOg datasets, outperforming strong baselines. Ablation studies reveal that combining supervised fine-tuning, privileged teacher guidance, and IoU-aware token weighting yields the best performance, with the full model reaching 85.78 mIoU and 86.76% [email protected]. The method remains effective across different model sizes and data scales.
- Compared to existing coordinate generation methods, IoU-PD leverages true bounding boxes as privileged visual cues, not just labels, leading to more precise spatial understanding. Results show significant improvements at higher IoU thresholds, especially above 0.9, indicating better localization accuracy. The approach also enhances performance on small and medium objects, demonstrating robustness in challenging scenarios.
Significance
This work addresses the core challenge of aligning training objectives with evaluation metrics in visual grounding. By incorporating true bounding boxes as privileged guidance, it bridges the gap between token-level supervision and geometric overlap measurement. The method enhances the spatial reasoning ability of multimodal large models, facilitating their deployment in real-world applications like autonomous vehicles, robotics, and intelligent surveillance. Its design preserves inference simplicity while delivering substantial accuracy gains, setting a new standard for training strategies in vision-language tasks.
Technical Contribution
The paper introduces a novel privileged distillation framework that integrates IoU-aware token weighting, enabling models to better encode spatial relationships. It innovatively uses ground-truth boxes during training as visual guidance, without changing inference input formats. The IoU-aware weighting dynamically emphasizes critical coordinate tokens based on geometric significance and teacher confidence, improving the alignment between training signals and evaluation metrics. This approach advances the state-of-the-art in coordinate generation for visual grounding, providing a scalable and effective training paradigm.
Novelty
This research is the first to utilize true bounding boxes as privileged visual guidance in the training of coordinate-generating multimodal large language models. It introduces IoU-sensitive token weighting to enhance the distillation process, directly addressing the mismatch between token-level supervision and region-level evaluation. Unlike prior methods that rely solely on coordinate labels, this approach leverages the visual richness of ground-truth boxes, representing a significant innovation in training strategies for visual grounding.
Limitations
- The method depends on high-quality bounding box annotations; noisy or inconsistent labels could diminish effectiveness. It also requires additional preprocessing to generate marked images and augmented prompts during training, increasing complexity.
- In scenarios with severe occlusion or clutter, privileged guidance may provide limited benefits, and the model's performance could plateau. Computational overhead during training is higher due to the teacher-student setup and IoU-aware weighting.
- The approach assumes availability of accurate bounding boxes, which may not be feasible in all datasets or real-world applications. Future work should explore weakly supervised alternatives to reduce annotation dependency.
Future Work
Future directions include developing self-supervised or weakly supervised bounding box generation to reduce reliance on manual annotations. Extending the framework to video and 3D scene understanding could further enhance spatial reasoning. Additionally, integrating more sophisticated spatial relationship modeling and exploring real-time applications in robotics and autonomous driving are promising avenues.
AI Executive Summary
Visual grounding is a fundamental challenge in multimodal understanding, requiring models to accurately localize image regions described by natural language. Existing approaches often rely on coordinate string generation trained with token-level supervision, which mismatches the evaluation metric of geometric overlap (IoU). This discrepancy limits the models' spatial reasoning capabilities. To address this, the paper introduces IoU-PD, a novel training framework that leverages true bounding boxes as privileged guidance during training. By incorporating ground-truth boxes both as coordinate targets and as visual hints for a frozen teacher model, IoU-PD effectively aligns training signals with evaluation metrics.
The core innovation lies in the IoU-aware token weighting scheme, which dynamically emphasizes coordinate tokens based on their geometric importance and teacher confidence. This mechanism ensures that the model learns to produce more accurate spatial boundaries without altering the inference input format. Extensive experiments on datasets like RefCOCO, RefCOCO+, and RefCOCOg demonstrate that IoU-PD consistently outperforms strong baselines, achieving up to 4.25 points higher in mIoU and notable improvements at higher IoU thresholds.
The approach's significance extends beyond benchmark scores, offering a new paradigm for training vision-language models with better spatial reasoning. It bridges the gap between token-level supervision and region-level evaluation, enabling models to better understand complex spatial relationships in real-world scenarios. Despite its reliance on high-quality bounding box annotations, IoU-PD paves the way for more accurate and robust visual grounding systems, with broad implications for autonomous systems, human-computer interaction, and intelligent perception technologies.
Deep Analysis
Background
多模态视觉理解的发展经历了从检测器回归到生成式模型的演变。早期方法如DETR、Grounding DINO通过专门的检测头实现目标定位，性能优异但缺乏灵活性。近年来，结合大规模预训练模型（如Qwen-VL、Liu等）的方法将定位融入自然语言处理框架，实现了端到端的生成式推理。然而，这些方法在训练目标（字符序列）与评估指标（IoU）之间存在差异，限制了空间理解的深度。现有研究多依赖坐标标签，未充分利用边界框的空间信息。
Core Problem
核心问题在于训练时优化字符级似然与评估时的几何重叠指标（IoU）不一致。模型在字符生成上表现良好，但空间关系理解不足，尤其在复杂场景和小目标中表现有限。如何在保持推理输入格式不变的前提下，利用真实边界框作为训练辅助信息，成为亟待解决的难题。这关系到模型的空间推理能力和实际应用效果。
Innovation
提出Privileged Distillation结合IoU感知令牌加权，创新点包括：• 利用真实边界框作为训练中的视觉引导，增强空间关系理解；• 设计IoU感知的令牌加权机制，根据预测边界框与真实边界的IoU、坐标误差、数字位置等动态调整蒸馏损失；• 保持推理输入格式不变，确保模型部署的便利性。这些创新突破了传统字符级监督的局限，显著提升定位精度。
Methodology
- �� 训练阶段，学生模型接受原始图像与描述，教师模型在图像上标记边界框并加入增强提示，条件于特权信息；• 结合监督微调（SFT）和Privileged Distillation，优化模型输出；• 设计IoU感知的令牌加权机制，依据预测边界框与真实边界框的IoU、坐标误差、数字位置等因素调整蒸馏损失；• 训练过程中，边界框作为视觉引导，提升模型空间关系理解；• 推理阶段，模型仅接受标准输入，无需边界框或额外模块。
Experiments
在RefCOCO、RefCOCO+、RefCOCOg数据集上，采用300k训练样本，比较不同模型架构和训练策略。指标包括mIoU、[email protected]、[email protected]。通过消融实验验证引入监督微调、Privileged Teacher、IoU感知加权的效果。模型在不同规模和数据量下均表现优异，验证了方法的稳定性和泛化能力。
Results
在RefCOCO上，IoU-PD提升mIoU从81.74到85.78，[email protected]从88.58到91.56，表现优于所有对比模型。消融分析显示，结合边界框特权输入和IoU感知加权带来最大提升，整体性能稳步提升，尤其在高IoU阈值下效果显著。模型在小目标和复杂场景中表现优异，验证了空间关系理解的增强。
Applications
该方法适用于自动驾驶中的目标检测、机器人导航、智能监控等场景，能显著提升空间定位精度。只需标准图像和描述，无需额外标注或模块，便于部署。未来可结合视频和三维场景，拓展多模态空间理解能力。
Limitations & Outlook
依赖高质量边界框，若数据集标注不准，可能影响性能。复杂场景下Privileged Teacher的引导效果有限，训练成本增加。未来需探索弱监督和自监督策略，降低对标注的依赖。
Plain Language Accessible to non-experts
想象你在厨房做饭，菜单上写着“做一道美味的炒饭”。传统方法就像只看菜单上的文字，按照字面意思去做，但有时候菜单描述不够详细，做出来的饭可能不够好吃。现在，厨师还会看一张示意图，图上用绿色圈出主要的食材和步骤，帮助厨师更准确地理解。这个示意图就像论文中的边界框，提供了额外的视觉信息，让厨师做饭更准、更快。训练时，厨师会用这张图和提示来学习，但在正式做饭时，只用菜单文字就可以了。这就像模型在推理时不用边界框，只用描述。这样的方法让厨师学得更好，做出来的炒饭也更美味。这个过程就像论文中的Privileged Guidance，利用额外信息帮助学习，但实际操作时不用这些信息。通过这种方式，厨师变得更聪明，能做出更符合菜单要求的饭菜。这就像模型变得更精准，能更好地找到图像中的目标区域。
Abstract
Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels. Project page: https://xyzzzh.github.io/IoU-PD/