Empowering Large Language Model for Sequential Recommendation via Multimodal Embeddings and Semantic IDs
MME-SID combines multimodal quantization and semantic-ID initialization; on Amazon Beauty, tau rises from 0.0550 to 0.3714.
Key Findings
Methodology
MME-SID builds on Llama3-8B-Instruct and jointly uses collaborative, textual, and visual embeddings. Its MM-RQ-VAE separately encodes the three modalities, replaces MSE reconstruction with characteristic-kernel Maximum Mean Discrepancy (MMD), and applies InfoNCE alignment between collaborative–textual and collaborative–visual quantized embeddings. Learned code vectors initialize semantic-ID embeddings, after which LoRA performs multimodal, frequency-aware LLM fine-tuning.
Key Results
- In the Amazon Beauty preliminary study, RQ-VAE quantization of SASRec collaborative embeddings achieved Kendall’s tau=0.3714 against the original distance ordering. Randomly initializing code embeddings for downstream training reduced tau to 0.0550, implying that approximately 94.5% of the previously learned ordering information was lost.
- The paper reports that directly projecting low-dimensional collaborative embeddings into the LLM space causes severe collapse: more than 98% of embedding dimensions collapse in the reported analysis. MME-SID instead combines original and quantized representations across collaborative, textual, and visual modalities.
- Experiments cover Amazon Beauty, Amazon Sports, and Amazon Clothing. The authors report overall superiority over existing LLM-based sequential recommenders, but the supplied text does not include the main tables’ exact Recall or NDCG values.
Significance
The work unifies two underexplored failure modes in LLM-based sequential recommendation: low-rank collapse caused by mapping collaborative embeddings into a large token space, and catastrophic forgetting caused by discarding pretrained code vectors. Its broader contribution is conceptual as well as practical: multimodality is used not merely to add features, but to enlarge the effective representation subspace and preserve geometry. Code-vector initialization may also reduce the cost of retraining enormous item vocabularies in industrial systems.
Technical Contribution
The principal components are MM-RQ-VAE, MMD reconstruction, cross-modal InfoNCE alignment, pretrained code-vector initialization, and LoRA-based frequency-aware fusion. MMD compares kernel mean embeddings rather than only pointwise Euclidean errors; contrastive objectives couple collaborative signals with textual and visual semantics. The LLM input concatenates projected original embeddings with quantized code embeddings, jointly retaining continuous distance information and hierarchical discrete structure.
Novelty
To the authors’ knowledge, this is the first work to formulate embedding collapse and catastrophic forgetting as a joint problem in LLM recommendation. Unlike methods that use only textual semantic IDs and discard learned code embeddings, MME-SID preserves code vectors and quantizes collaborative, textual, and visual representations in a unified framework.
Limitations
- The supplied paper text does not provide complete Recall@K, NDCG@K, confidence intervals, or per-component gains for the three datasets, making the magnitude of the claimed improvement difficult to independently verify.
- MME-SID depends on LLM2CLIP, Llama3-8B-Instruct, and multiple item modalities. Missing images, weak descriptions, or noisy collaborative signals may reduce the reliability of cross-modal alignment.
- Multiple encoders, codebooks, and LLM inference increase computational and systems complexity; the provided material does not fully quantify end-to-end training or serving costs.
Future Work
Future studies should report complete ranking metrics, significance tests, and scaling costs, while evaluating dynamic codebooks, additional modalities, and cross-domain transfer. Important ablations include kernel choice, codebook depth, β and γ, LoRA rank, and frequency fusion. Robustness should also be tested under new-item cold start, long histories, missing modalities, noisy metadata, and live industrial traffic.
AI Executive Summary
Sequential recommendation predicts a user’s next interaction from behavioral history. Traditional systems rely mainly on item IDs and struggle with cold start; LLM-based systems can read rich item text but introduce two less visible failures. Projecting low-dimensional collaborative embeddings into a high-dimensional token space can produce embedding collapse, while semantic-ID systems often discard pretrained code vectors and relearn them from scratch, causing catastrophic forgetting. The paper reports collapse in more than 98% of embedding dimensions and a severe loss of ordering information under random code initialization.
The authors propose MME-SID, built around Llama3-8B-Instruct and three information sources: collaborative, textual, and visual embeddings. Its MM-RQ-VAE creates hierarchical semantic IDs for each modality. Maximum Mean Discrepancy replaces ordinary MSE reconstruction to preserve distributional and geometric information, while InfoNCE aligns collaborative codes with textual and visual codes. Crucially, the learned code vectors initialize the LLM’s semantic-ID embeddings rather than being discarded. LoRA then performs parameter-efficient, multimodal, frequency-aware fine-tuning.
In the Amazon Beauty preliminary analysis, preserved distance ordering reached Kendall’s tau=0.3714, compared with only 0.0550 after random initialization—corresponding to roughly 94.5% information loss in the latter setting. Experiments on Amazon Sports and Amazon Clothing are reported to show overall gains over existing LLM sequential recommenders, although exact table values are absent from the supplied text. The broader message is that semantic IDs should carry reusable representation content, not merely identifiers. Important open questions concern industrial-scale cost, missing modalities, cross-domain generalization, and the reproducibility of the reported ranking improvements.
Deep Analysis
Background
Sequential recommendation evolved from collaborative-ID models such as SASRec to instruction-tuned LLM systems such as TALLRec. RQ-VAE-based semantic-ID methods discretize item embeddings into generative codes, making LLM retrieval more tractable. Yet ID-only models remain vulnerable to cold start, text-only semantic IDs omit visual and behavioral signals, and projecting low-dimensional collaborative vectors into an LLM token space can waste most representational capacity.
Core Problem
The paper studies two bottlenecks. First, for a low-rank collaborative table E_c∈R^{M×D}, a linear projection satisfies rank(WE_c+b)≤rank(E_c)+1, so the representation occupies a narrow subspace. Second, retaining only semantic IDs and relearning their embeddings destroys original distance orderings. On Amazon Beauty, random downstream code initialization achieved Kendall’s tau=0.0550.
Innovation
MM-RQ-VAE quantizes collaborative, textual, and visual modalities jointly but with modality-specific codebooks. MMD replaces MSE so reconstruction targets distributional statistics rather than only pointwise Euclidean proximity. InfoNCE aligns collaborative codes with textual and visual codes. Learned code vectors initialize semantic-ID tokens, preserving pretrained structure. Finally, original projected embeddings and quantized codes are concatenated, while LoRA and frequency-aware fusion adapt the model to cold and warm items.
Methodology
- �� Encoding: SASRec supplies collaborative embeddings; LLM2CLIP supplies textual and visual embeddings.
- �� Quantization: each modality passes through an L-level RQ-VAE; residuals are recursively quantized and summed into a code embedding.
- �� Objective: optimize MMD reconstruction, InfoNCE alignment, and the RQ-VAE commitment loss: L_MMRQ=L_Recon+βL_Align+γΣL_RQ.
- �� Input construction: original modality projections and semantic-ID code embeddings are concatenated and mapped by an MLP to the 4096-dimensional LLM token space.
- �� Fine-tuning: most LLM parameters remain frozen; LoRA updates the model, and item-frequency statistics guide fusion of LLM outputs with modality-specific item representations.
Experiments
The evaluation uses Amazon Beauty, Amazon Sports, and Amazon Clothing, with Llama3-8B-Instruct as the main backbone. Comparisons involve conventional sequential recommenders, semantic-ID approaches, and LLM-based sequential recommenders. Analyses include singular-value inspection, Kendall’s tau for forgetting, MMD-versus-MSE comparisons, random versus pretrained code initialization, and modality ablations. The supplied text explicitly reports Beauty tau values of 0.3714 and 0.0550 but omits the complete main-result tables.
Results
The analysis attributes more than 98% dimensional collapse to direct projection of low-dimensional collaborative embeddings. Pretrained RQ-VAE codes preserve substantially more distance ordering than randomly initialized codes: tau is 0.3714 versus 0.0550. The three-dataset study is reported to favor MME-SID overall, suggesting complementary benefits from multimodal inputs and code inheritance. However, without exact Recall, NDCG, or ablation numbers, the independent contribution of each component cannot be precisely quantified from the supplied material.
Applications
E-commerce can combine titles, descriptions, images, and click sequences, especially for new or content-rich products. Short-video and media platforms can similarly encode visual semantics with viewing histories. Deployment requires reliable multimodal features, item-frequency statistics, trained codebooks, and substantial LLM inference capacity. LoRA lowers fine-tuning memory and parameter costs, but does not eliminate serving latency or the need for codebook maintenance.
Limitations & Outlook
Evidence is centered on public Amazon data, leaving cross-domain, multilingual, and real-time validation open. The method assumes that text, images, and collaborative signals are available and sufficiently aligned; missing or inconsistent modalities may make InfoNCE training fragile. Sensitivity to codebook depth, kernel choice, β, γ, and frequency fusion requires fuller study. Future work should add complete metrics, scaling and latency measurements, privacy analysis, fairness evaluation, and robust tests under noisy or adversarial metadata.
Plain Language Accessible to non-experts
Imagine a huge bookstore recommending the next book to each customer. A traditional system remembers only barcodes: it knows that a customer bought books 3 and 8, but not what those books mean. An LLM is a talented librarian who can read titles and descriptions and inspect covers. Yet if a small note from the old system is stretched onto a giant poster, the information may occupy only a tiny corner, leaving most of the poster useless.
MME-SID creates three catalogs for every book: a behavior catalog, a text catalog, and a picture catalog. Each catalog is converted into several layers of small labels, but the system also keeps the information behind those labels. MMD checks whether two whole collections have similar patterns, not merely whether individual entries sit nearby. Contrastive learning makes the behavior, description, and cover of the same book point toward one another.
The crucial step is to hand the librarian the old catalog content directly instead of throwing away the notes and inventing every label again. On Amazon Beauty, the system’s preserved ordering score was 0.3714, while random relearning produced only 0.0550. In everyday terms, the method helps the recommender remember what it already knew while using several kinds of evidence to understand what a customer may want next.
ELI14 Explained like you're 14
Suppose you recommend games to a friend. You know which games they played, you can read game descriptions, and you can look at screenshots. A basic recommender is like a notebook containing only game numbers: “They played 3 and 8.” It remembers actions but does not understand what the games are. An LLM is the clever friend who can read and look, but translating the old notebook into its own brain can squeeze useful information into a tiny corner.
MME-SID makes three cards for every game: a player-behavior card, a words card, and a picture card. Each card gets several layers of small codes. The important part is that the system keeps the original card information too, so two similar games remain similar after coding. It also makes the three cards for the same game learn from one another—kind of like checking that the title, screenshot, and player history really describe the same game.
When the big model is trained, it starts with the useful code information it already learned instead of guessing everything from scratch. That is like moving schools with your old notes rather than throwing them away. In the Amazon Beauty test, preserved ordering scored 0.3714, while random relearning scored only 0.0550. Big difference!
The researchers also tested Amazon Sports and Amazon Clothing and reported overall improvements over earlier LLM recommenders. The lesson is not simply “make the model bigger.” A strong recommender should remember old knowledge, understand several clues, and know when each clue is useful. Pretty smart, right?
Glossary
Embedding Collapse
In plain terms, many representation dimensions become nearly useless and the vectors crowd into a narrow subspace. Technically, the embedding matrix becomes close to low-rank, with many very small singular values.
The paper reports collapse in over 98% of dimensions and uses multimodal representations to mitigate it.
Semantic ID
A discrete code sequence that represents an item’s learned semantic or structural properties. In an LLM recommender, these codes can be treated as generated tokens.
MME-SID generates separate semantic IDs for collaborative, textual, and visual modalities.
RQ-VAE
Residual Quantized Variational Autoencoder progressively quantizes a latent vector with multiple codebooks. The selected code vectors from all levels are summed to form the quantized representation.
The proposed MM-RQ-VAE produces hierarchical codes for the three modalities.
Maximum Mean Discrepancy
MMD measures the difference between two distributions through their kernel mean embeddings. With a characteristic kernel, it captures richer distributional statistics than a simple pointwise error.
MM-RQ-VAE uses MMD instead of ordinary MSE for reconstruction.
InfoNCE
A contrastive-learning objective that increases similarity for matching pairs and decreases it for mismatched pairs. A temperature controls the sharpness of the comparison distribution.
The paper uses InfoNCE to align collaborative codes with textual and visual codes.
LoRA
Low-Rank Adaptation fine-tunes a model through small low-rank update matrices rather than changing all parameters. This reduces trainable parameters and memory requirements.
MME-SID applies LoRA for efficient multimodal, frequency-aware fine-tuning.
Open Questions Unanswered questions from this research
- 1 The supplied material omits complete Recall, NDCG, confidence intervals, and significance tests, so the exact improvement over each baseline remains unverifiable without the paper’s result tables.
- 2 It is unclear whether MMD and cross-modal alignment remain stable with missing images, noisy text, extreme cold start, or rapidly changing item catalogs.
- 3 The training and serving cost of Llama3-8B plus multiple encoders, as well as codebook maintenance at billion-item scale, is not fully quantified.
Applications
Immediate Applications
Multimodal E-commerce Recommendation
Retail platforms can train MM-RQ-VAE on product titles, descriptions, images, and interaction sequences, then initialize semantic-ID tokens with learned code vectors. This is suitable for content-rich catalogs with new products, provided that behavior logs, stable metadata, and GPU inference are available.
Short-Video and Media Recommendation
Video platforms can combine thumbnails, captions, and viewing histories, using frequency-aware fusion to treat popular and long-tail items differently. LoRA supports periodic updates, but operators must monitor modality noise, latency, diversity, and popularity bias.
Long-term Vision
Persistent-Memory Generative Recommendation
Semantic IDs could become reusable knowledge interfaces: new recommenders inherit old code vectors and adapt through incremental quantization and cross-domain alignment. Realizing this vision requires version-compatible codebooks, privacy safeguards, continual learning, and efficient online updates.
Abstract
Sequential recommendation (SR) aims to capture users' dynamic interests and sequential patterns based on their historical interactions. Recently, the powerful capabilities of large language models (LLMs) have driven their adoption in SR. However, we identify two critical challenges in existing LLM-based SR methods: 1) embedding collapse when incorporating pre-trained collaborative embeddings and 2) catastrophic forgetting of quantized embeddings when utilizing semantic IDs. These issues dampen the model scalability and lead to suboptimal recommendation performance. Therefore, based on LLMs like Llama3-8B-instruct, we introduce a novel SR framework named MME-SID, which integrates multimodal embeddings and quantized embeddings to mitigate embedding collapse. Additionally, we propose a Multimodal Residual Quantized Variational Autoencoder (MM-RQ-VAE) with maximum mean discrepancy as the reconstruction loss and contrastive learning for alignment, which effectively preserve intra-modal distance information and capture inter-modal correlations, respectively. To further alleviate catastrophic forgetting, we initialize the model with the trained multimodal code embeddings. Finally, we fine-tune the LLM efficiently using LoRA in a multimodal frequency-aware fusion manner. Extensive experiments on three public datasets validate the superior performance of MME-SID thanks to its capability to mitigate embedding collapse and catastrophic forgetting. The implementation code and datasets are publicly available for reproduction: https://github.com/Applied-Machine-Learning-Lab/MME-SID.