Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval
PrediCIR uses a world model to predict missing target features, boosting zero-shot image retrieval by up to 4.45%.
Key Findings
Methodology
PrediCIR employs two modules: a world view generation module constructs source and target views via random cropping combined with caption-based manipulation actions; a target content predictor, based on JEPA, predicts missing visual elements guided by manipulation text. The model encodes views and actions with CLIP, predicts missing content, and maps it into pseudo-word tokens. Loss functions include contrastive and alignment losses, ensuring the predicted features align with target visual features. During inference, the model combines reference images and manipulation texts, predicts missing content, and forms a composite query for retrieval.
Key Results
- PrediCIR outperforms state-of-the-art methods on six ZS-CIR tasks, with improvements ranging from 1.73% to 4.45%. On Fashion-IQ, it achieves R50 of 52.3%, surpassing previous best by 2.8%. On CIRR, mAP@25 reaches 18.6%, about 2% higher than baselines. Ablation studies confirm the effectiveness of the world view generation and prediction modules, especially in scenarios with missing key information.
- The model demonstrates strong generalization across diverse tasks, including attribute, scene, and object manipulation, with consistent performance gains. It effectively captures fine-grained details, significantly improving retrieval accuracy in challenging cases.
- Experimental results validate that the prediction mechanism enhances the model's ability to recover missing visual content, leading to more precise and robust zero-shot retrieval performance.
Significance
This work addresses the fundamental limitation of coarse-grained embeddings in CLIP-based retrieval, introducing a predictive mechanism that effectively reconstructs missing content. It advances the state-of-the-art in zero-shot visual search, enabling more accurate, detail-rich retrieval in real-world applications like e-commerce, content management, and virtual try-on systems. The approach also deepens understanding of vision-language alignment by integrating content prediction into the retrieval pipeline.
Technical Contribution
The paper introduces PrediCIR, a novel framework combining a world model with a pseudo-word mapping network. It innovatively generates source and target views via random cropping, trains a predictor to estimate missing visual features guided by manipulation text, and maps these features into the language space using contrastive learning. This approach surpasses traditional pseudo-word methods, providing fine-grained content prediction and interpretability, while maintaining efficiency comparable to existing models.
Novelty
This is the first work to incorporate a world model for content prediction in zero-shot image retrieval, bridging the gap between coarse embeddings and detailed content understanding. Unlike prior methods relying solely on contrastive embeddings, PrediCIR predicts missing visual details guided by manipulation text, enabling precise content completion and improved retrieval accuracy, especially when target content is absent in reference images.
Limitations
- The model's performance depends on the quality of cropping and the diversity of training data; complex backgrounds or multiple objects can challenge prediction accuracy.
- Inference costs are relatively high due to the prediction process, limiting real-time deployment in large-scale systems.
- Robustness to ambiguous or highly incomplete manipulation texts needs further enhancement, requiring more sophisticated multimodal fusion techniques.
Future Work
Future directions include integrating adaptive cropping strategies and multi-scale features to improve robustness and efficiency. Exploring self-supervised learning and multi-modal fusion could enhance performance in ambiguous scenarios. Extending the framework to video and 3D content retrieval and optimizing for real-time applications are also promising avenues.
AI Executive Summary
In the rapidly evolving field of visual search, zero-shot image retrieval has gained prominence due to its ability to operate without extensive labeled datasets. Traditional methods, relying on coarse embeddings like CLIP, often struggle with content missing in reference images, especially in complex manipulation tasks. This limitation hampers retrieval accuracy when the target visual details are absent or subtle.
To address this, the authors propose PrediCIR, a novel framework that leverages a world model to predict missing target content in the latent space. The core idea involves generating source and target views through random cropping, then training a predictor based on JEPA to estimate the absent visual elements guided by manipulation text. These predicted features are mapped into pseudo-word tokens, enriching the semantic representation used for retrieval. This approach effectively bridges the gap between coarse embeddings and fine-grained content understanding.
Experimental results across six diverse datasets demonstrate that PrediCIR consistently outperforms existing state-of-the-art methods, with improvements ranging from 1.73% to 4.45%. Notably, on the Fashion-IQ dataset, it achieves a R50 of 52.3%, surpassing previous bests by nearly 3%. The model's ability to recover missing details proves especially beneficial in attribute and scene manipulation tasks, where precise content prediction is critical.
This work marks a significant step forward in zero-shot vision-language retrieval, offering a scalable, interpretable, and highly effective solution. Its innovative use of a world model for content prediction opens new avenues for multi-modal understanding, with broad implications for e-commerce, content management, and virtual reality applications. Despite some limitations in computational cost and robustness, the framework sets a new benchmark and points to promising future research directions, including multi-scale fusion and real-time deployment.
Deep Dive
Abstract
Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent across domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to modify a reference image according to manipulation text to accurately retrieve a target image, especially when the reference image is missing essential target content. In this paper, we propose a novel prediction-based mapping network, named PrediCIR, to adaptively predict the missing target visual content in reference images in the latent space before mapping for accurate ZS-CIR. Specifically, a world view generation module first constructs a source view by omitting certain visual content of a target view, coupled with an action that includes the manipulation intent derived from existing image-caption pairs. Then, a target content prediction module trains a world model as a predictor to adaptively predict the missing visual information guided by user intention in manipulating text at the latent space. The two modules map an image with the predicted relevant information to a pseudo-word token without extra supervision. Our model shows strong generalization ability on six ZS-CIR tasks. It obtains consistent and significant performance boosts ranging from 1.73% to 4.45% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/predicir.