Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

The evolution of image captioning has transitioned from template-based methods to deep neural models, notably encoder-decoder architectures like Show and Tell, and Transformer-based models. These approaches significantly improved fluency and relevance but still suffer from hallucinations and detail omissions, especially in complex scenes. Recent LVLMs, trained on large-scale multimodal datasets, have pushed the boundary further, enabling models to understand and generate richer descriptions. However, challenges remain in enabling models to perform deep reasoning, explore diverse hypotheses, and avoid hallucinations. Existing reinforcement learning techniques, such as CLIP-based reward optimization, improve caption relevance but are limited by the models’ intrinsic reasoning capacity and susceptibility to reward hacking. Consequently, there is a pressing need for methods that can enhance reasoning exploration without extensive annotation or complex supervision.

The core issue is how to empower LVLMs to explore and generate more accurate, detailed, and diverse captions, particularly in relation reasoning tasks. Current RL methods tend to converge prematurely, favoring high-reward but potentially hallucinated outputs, and lack mechanisms to detect and correct errors dynamically. This results in limited exploration of the caption space, restricting the model’s ability to produce nuanced descriptions. The challenge is to develop an approach that can guide models toward more truthful and comprehensive captions, leveraging implicit reasoning signals, without relying on costly annotations or extensive retraining.

This work introduces several innovations: 1) Retrieval-guided reasoning using CRS and CQA modules, which analyze descriptions of similar images to identify key semantic elements and detect hallucinations or omissions. 2) Application of k-core analysis to filter out less relevant descriptions, ensuring semantic consistency. 3) Decoupled PPO optimization during RL, which removes retrieval dependencies at inference, enabling efficient deployment. 4) The framework’s ability to generate diverse caption candidates by exploring uncharted regions of the caption space, significantly enhancing reasoning depth and detail. These innovations collectively address the exploration bottleneck in RL-based captioning, providing a scalable, annotation-free solution.

• �� Input: Image v and initial captions {ci} generated by LVLMs.
• �� Image retrieval: Use v as query to retrieve TopK similar images from dataset D, constructing graph Gv with nodes as descriptions.
• �� Semantic filtering: Compute sentence similarities with SBERT, perform k-core decomposition to identify consistent semantic content Skv.
• �� Caption refinement: Use CRS to suggest incorporating Skv into captions, improving content accuracy.
• �� Discrepancy detection: Use c as caption, retrieve descriptions Rc, build graphs Gc and Gvc, identify hallucinations (Skc−Skvc) and omissions (Skv−Skvc).
• �� RL optimization: Incorporate retrieval signals into PPO, generate multiple candidate captions, and optimize the policy to maximize reward, encouraging exploration and diversity.

The evaluation used COCO-LN500 and DOCCI500 benchmarks, with models like Qwen2-VL-7B, LLaVA-1.5-7B, and others. Baselines included GRPO and SFT, with metrics such as F1 scores and relation QA accuracy. Hyperparameters included K=3, τ=0.7, training with Adam optimizer for two epochs, sampling M=5 responses per prompt. Ablation studies assessed the contributions of CRS and CQA modules, and different reward functions validated robustness. The experiments demonstrated consistent improvements across models and benchmarks, confirming the effectiveness of the retrieval-guided approach.

Re3Cap achieved an average of 8.64% improvement in relation reasoning over COCO-LN500, surpassing traditional RL and SFT. It notably enhanced object and attribute F1 scores, with gains up to 7.8% and 13.4% respectively. Across models, the method consistently expanded the reasoning boundary, enabling exploration of previously unexplored caption candidates. The results validated that the retrieval-guided strategy effectively mitigates hallucinations and omissions, leading to richer, more accurate descriptions, especially in complex scenarios.

This approach can be directly applied to automatic image captioning, visual question answering, and multimodal content retrieval. Its annotation-free nature makes it suitable for real-world deployment in industries like e-commerce, healthcare, and autonomous systems, where detailed visual understanding is critical. The ability to generate diverse, truthful descriptions enhances user experience and system reliability. Long-term, integrating this framework with multi-turn reasoning and larger retrieval corpora could revolutionize multimodal AI capabilities.

The method depends on the quality of retrieval results; poor retrieval can mislead reasoning signals. In highly complex or ambiguous scenes, k-core analysis may miss subtle semantics. Computational costs during training are high due to retrieval and graph processing. Future work should focus on improving retrieval robustness, reducing overhead, and extending to broader tasks like video captioning and multi-turn reasoning.