Popcorn: A Configurable Benchmark for Visual Evidence in Multimodal Movie Recommendation

The evolution of movie recommendation systems reflects a gradual shift from simple collaborative filtering to sophisticated multimodal approaches. Early methods relied heavily on user-item interaction data, such as ratings and clicks, with limited content understanding. As deep learning advanced, researchers incorporated visual features extracted from posters, trailers, and short clips, utilizing CNNs like VGG and Inception to capture visual cues related to genre, style, and mood. Notable works include He and McAuley’s VBPR, which integrated visual features into Bayesian personalized ranking, and Deldjoo et al.’s MicroLens dataset, which provided micro-video features for recommendation. Despite progress, these approaches often treat visual evidence as a monolithic input, lacking systematic comparison of different evidence sources. The emergence of vision-language models (VLMs) like CLIP has opened new avenues for encoding sparse but semantically rich visual signals, enabling large-scale catalog applications. However, existing benchmarks and datasets do not explicitly isolate the effect of evidence source, nor do they support comprehensive ablation studies on fusion strategies or metadata augmentation. Consequently, understanding how different visual cues contribute to recommendation performance remains an open challenge, limiting the development of more effective and explainable systems.

The core problem addressed by this work is the lack of a systematic, controlled framework to evaluate the impact of different visual evidence sources—full movies, trailers, and thumbnails—on multimodal recommendation performance. Existing datasets and benchmarks often conflate these sources or focus solely on trailers, making it difficult to disentangle their individual contributions. Moreover, current models do not explicitly compare classical CNN-based multi-frame features with modern VLM-based semantic features at scale, especially in terms of scalability, computational cost, and recommendation quality. This gap hampers the development of optimized fusion strategies and limits understanding of how sparse visual signals can be effectively leveraged in large catalogs. Additionally, the absence of standardized evaluation protocols and detailed logging impedes reproducibility and cross-study comparisons, slowing progress in the field. Addressing these issues requires a comprehensive benchmark that isolates evidence source effects, supports configurable fusion and augmentation strategies, and provides detailed metrics for beyond-accuracy considerations such as coverage, diversity, and fairness.

The main innovations of this work include: 1) The creation of Popcorn, a benchmark that explicitly differentiates between visual evidence sources—full movies, trailers, and thumbnails—and supports systematic comparison under controlled settings. 2) The integration of multiple state-of-the-art visual and vision-language models (CLIP, DINOv2, SigLIP) for encoding sparse thumbnail images, enabling semantic-rich features at catalog scale. 3) The design of a flexible, configuration-driven pipeline that supports various fusion strategies (concatenation, PCA, CCA), recommendation algorithms, and LLM-based metadata augmentation, facilitating comprehensive ablation studies. 4) The release of aligned video embeddings for 274 movies and a large thumbnail layer linked to 65,000 MovieLens titles, providing a rich resource for research. 5) The emphasis on reproducibility and detailed logging, including hyperparameters, modality choices, and metrics, setting a new standard for benchmarking in multimodal recommendation. These innovations collectively enable a nuanced understanding of how different visual evidence sources and fusion strategies influence recommendation outcomes.

• �� Evidence Loading: Extract frame-level, shot-level, and pooled embeddings from full movies and trailers using CNNs like Inception-v3 and VGG-19, sampled at 1 FPS for detailed temporal representation. • Thumbnail Encoding: Construct a large-scale thumbnail layer linking approximately 65,000 MovieLens titles with visual features encoded by six models (CLIP, OpenCLIP, DINOv2, SigLIP). These features are aggregated into over 300,000 embeddings, organized into 13 image packs. • Fusion Strategies: Implement configurable fusion methods—concatenation, PCA (retaining 90% variance), and CCA (with 40 components)—as hyperparameters, allowing systematic comparison. • Recommendation Models: Use classical multi-modal recommenders such as VBPR, AMR, and VMF, trained on fused features, with hyperparameter tuning for each. • Metadata Augmentation: Employ LLMs (e.g., LLaMA, OpenAI GPT) to generate descriptive summaries from sparse metadata, embedding these texts for fusion with visual features. • Evaluation Protocol: Conduct offline experiments on MovieLens-1M and a subset of 274 aligned videos, measuring nDCG@10, Recall@10, coverage, diversity, fairness, and calibration. Hyperparameters are tuned via grid search, with detailed logging of configurations and metrics. • Ablation Studies: Vary evidence sources, encoding backbones, fusion methods, and metadata augmentation to analyze their individual and combined effects on recommendation quality, ensuring reproducibility and comprehensive understanding.

The experiments utilize two primary datasets: one with derived embeddings for 274 movies’ full movies and trailers, and another large-scale thumbnail layer linked to approximately 65,000 MovieLens titles. The evaluation metrics include accuracy measures such as nDCG@10, Recall@10, and precision, alongside beyond-accuracy metrics like coverage, diversity, fairness, and calibration bias. Baselines include traditional CNN features from trailers and older visual models. Hyperparameters such as PCA variance retention (90%) and CCA component count (40) are systematically tuned. Experiments compare visual-only, text-only, and fused modalities across different evidence sources, analyzing the impact of fusion strategies and LLM augmentation. Results show that VLM-encoded thumbnails outperform CNN-based trailer features, with significant improvements in recommendation metrics. Fusion strategies like CCA enhance coverage but may reduce diversity, highlighting trade-offs. The study also examines the effect of model size and storage proxies, revealing that larger models do not always guarantee better performance, emphasizing deployment considerations. Overall, the experiments validate the effectiveness of static thumbnail features encoded by modern VLMs and demonstrate the importance of fusion and metadata augmentation in optimizing recommendation quality.

The key results demonstrate that VLM-encoded thumbnails, such as SigLIP-base, outperform traditional trailer CNN features, with nDCG@10 reaching 0.269 versus 0.222 baseline, a 21.2% increase. Fusion strategies, particularly CCA, significantly improve coverage (from 0.767 to 0.918) but may slightly reduce diversity (from 0.766 to 0.749). Visual-only evidence from trailers surpasses full movies in some models (e.g., VBPR trailer nDCG@10=0.433), but after fusion, full movies can match or exceed trailer performance, indicating their complementary nature. LLM-based metadata augmentation yields moderate improvements in recommendation metrics, especially when combined with fusion strategies, but introduces sensitivity to prompt design. Cost-performance analysis shows that models like CLIP offer high coverage with smaller size, while DINOv2-large achieves higher diversity at increased computational cost. These findings highlight the importance of evidence source selection, fusion strategy, and model efficiency in practical deployment.

The immediate application of Popcorn lies in enhancing large-scale movie recommendation platforms by integrating multi-source visual evidence, thereby improving accuracy, coverage, and user satisfaction. The benchmark facilitates systematic evaluation and optimization of multimodal fusion strategies, guiding industry practitioners in model selection and deployment. Additionally, the framework supports content understanding and explainability, enabling platforms to generate transparent recommendations with visual and textual justifications. Long-term applications include developing adaptive, real-time recommendation systems that leverage multimodal cues for personalized content delivery, content creator tools for content analysis and tagging, and cross-media retrieval systems that utilize visual evidence for efficient content discovery. These advancements can transform multimedia content curation, user engagement, and monetization strategies across entertainment industries.

Despite its strengths, the current Popcorn framework faces limitations such as reliance on derived embeddings rather than raw videos, which may limit the richness of visual features. The aligned full-movie subset is relatively small, restricting long-form video analysis. LLM-based metadata augmentation depends on pretrained models and prompt engineering, which can introduce biases and inconsistencies. Computational costs for encoding and fusion at scale remain high, posing challenges for real-time deployment. Offline evaluation cannot fully capture dynamic user interactions and preferences, necessitating future online validation. Additionally, the framework’s focus on static visual features may overlook temporal dynamics and audio cues, which are also crucial for comprehensive content understanding. Addressing these limitations involves expanding datasets, optimizing encoding pipelines, and integrating multimodal temporal modeling to enhance robustness and applicability in real-world scenarios.