CoCo-IR: Contextual Composed Image Retrieval

The evolution of multimodal learning has seen significant milestones, starting from early dual-encoder models like CLIP that aligned image and text representations in a shared space for zero-shot retrieval. These models excel in static scenarios but lack the capacity for deep reasoning or multi-turn interactions. Recent advances introduced foundation models such as Florence and Gato, which integrate vision and language for more sophisticated understanding. Despite these progressions, most existing systems are limited to single-query retrieval, unable to support iterative refinement or complex dialogues. Static datasets like MS COCO and Flickr30k provided foundational benchmarks, but they lack the dynamic, multi-turn interaction annotations necessary for training conversational retrieval systems. The challenge has been to develop models that can understand and leverage context over multiple exchanges, which is crucial for real-world applications like virtual assistants, e-commerce, and virtual tours. This paper builds upon these developments, aiming to create a scalable, context-aware retrieval framework that can handle complex, multi-turn interactions.

Current image retrieval systems predominantly operate in a single-turn setting, where the user provides an initial query and receives a static set of results. This paradigm fails to accommodate the iterative, exploratory nature of human search behavior, where users refine their intent based on intermediate results. The core problem is enabling models to understand and incorporate the entire interaction history—comprising previous images and instructions—into the retrieval process. Existing approaches often rely on prompt summarization or external models to compress context, which leads to information loss and reduced accuracy. Additionally, the lack of large-scale, high-quality multi-turn datasets hampers the training and evaluation of such systems. Addressing these issues requires innovative model architectures capable of dynamic context encoding, as well as scalable data generation methods that do not depend on manual annotation, to facilitate robust learning and real-world deployment.

The primary innovations include: 1) The TIE architecture, which employs a dedicated <EMB> token to encode the entire dialogue history into a single, transformable embedding, allowing the model to adapt dynamically across multiple turns. 2) A hybrid attention mechanism that combines bidirectional attention within each turn for deep multimodal fusion and causal attention across turns to preserve temporal order. 3) An autonomous data engine that leverages large multimodal models for instruction synthesis, self-reflection, and hard negative mining, enabling the creation of extensive multi-turn datasets with minimal human effort. 4) The use of contrastive learning (InfoNCE) to optimize the embedding space, ensuring discriminability and robustness. These innovations collectively enable the model to understand complex, evolving user intents in a conversational manner, significantly advancing the state-of-the-art in multi-turn image retrieval.

• �� Architecture Design: Introduce a special <EMB> token at the end of each turn’s input, which prompts the model to generate a global, transformable embedding representing the entire interaction history.
• �� Attention Mechanism: Within each turn, employ bidirectional attention to fuse visual and textual modalities deeply; across turns, use causal attention to maintain temporal sequence.
• �� Training Objective: Use contrastive learning with InfoNCE loss, contrasting query embeddings with positive target images and hard negatives, to maximize discriminability.
• �� Data Generation: Develop a scalable pipeline that discovers image pairs with meaningful transformations via feature clustering and webpage co-occurrence, then uses LMMs to generate transformation instructions and perform self-reflection for quality filtering.
• �� Hard Negative Mining: Use LMM-based verification to identify challenging negatives that are visually similar but do not satisfy instructions, enhancing model robustness.
• �� Multi-turn Chain Construction: Concatenate multiple single-turn samples into multi-turn sequences, with LMM rewriting instructions to ensure contextual coherence.
• �� Quality Control: Apply multi-stage filtering and human verification to ensure high data quality for training and evaluation.

The evaluation employs both the CIRCO single-turn benchmark and a newly constructed multi-turn benchmark. Metrics include mAP@5 for single-turn and R@1 for multi-turn scenarios. The training involves ablation studies to assess the impact of <EMB> tokens, attention mechanisms, and data quality filters. Baseline comparisons include traditional CIR models, prompt-based summarization methods, and recent foundation model-based approaches. Hyperparameters such as embedding dimensions, number of transformer layers, and attention heads are tuned for optimal performance. The experiments also analyze the effect of sequence length, negative sample hardness, and data filtering thresholds. Cross-scenario tests demonstrate the model’s robustness and generalization capabilities.

The model achieves 39.4 in mAP@5 on CIRCO, outperforming previous state-of-the-art methods by a significant margin. In the multi-turn setting, it maintains a high performance of 44.1 in R@1 over four turns, whereas prior approaches collapse to around 28.2. Ablation studies confirm that the <EMB> token and hybrid attention are critical for performance gains. The autonomous data engine produces high-quality multi-turn datasets with minimal manual effort, enabling training with only a fraction of the data typically required. The model demonstrates strong generalization across different interaction lengths and instruction complexities, validating the effectiveness of the architecture and data strategies.

This technology can be immediately applied to intelligent virtual assistants, enabling users to refine image searches through natural multi-turn dialogues. It also benefits virtual shopping platforms, where consumers can iteratively specify preferences, and virtual tour guides, providing context-aware scene exploration. The framework supports multimodal interfaces that combine images and text seamlessly, enhancing user experience in e-commerce, education, and entertainment. Long-term, the approach paves the way for fully conversational AI systems capable of understanding complex, evolving instructions across diverse modalities, transforming human-computer interaction into a more natural, intuitive process.

Despite its strengths, the model struggles with highly ambiguous or extremely complex instructions that require long-range context understanding. The autonomous data engine, while scalable, may generate less diverse or overly synthetic data, affecting real-world robustness. Computational costs remain high, especially for large models and multi-turn inference, limiting deployment in resource-constrained environments. Additionally, the current system is primarily validated on image datasets; extending to video or 3D data introduces new challenges. Future work must address these limitations through more efficient architectures, richer datasets, and adaptive learning strategies.