Taming the Long Tail: Efficient Item-wise Sharpness-Aware Minimization for LLM-based Recommender Systems

Recommender systems are widely deployed across various domains, including news, videos, and medications. Traditional recommender systems heavily rely on limited interaction data and single-form inputs, lacking broad world knowledge and instruction understanding abilities, which limits further improvements in recommendation quality and personalization. Recently, large language models (LLMs) have been integrated into recommender systems due to their strong knowledge and instruction understanding capabilities. Particularly in sequential recommendation, LLMs are directly used as new backbone models, referred to as LLM-based recommender systems (LRSs). However, this new LRS paradigm has not fully explored long-standing problems in traditional recommender systems, such as the long-tail problem. The long-tail phenomenon refers to a scenario where a small number of popular items accumulate the majority of interactions, while the vast majority of tail items remain underrepresented, leading to issues in diversity, novelty, and fairness of exposure.

LRSs are typically obtained by tuning pre-trained LLMs with recommendation data, consequently facing two types of long-tail problems. The first is the long-tail in training data, referred to as data long-tail. The second comes from the prior LLM pretraining corpus, referred to as prior long-tail. Since pretraining data are inaccessible or extremely large and cannot be directly aligned with recommendation data, prior long-tail can only be reflected implicitly through LLM parameters and ultimately manifested in model performance. The study shows that while both contribute to performance disparities between head and tail items, the overall performance distribution, especially on the tail, is primarily dominated by the data long-tail.

To address the long-tail problem in LRSs, the paper proposes a novel optimization framework called Efficient Item-wise Sharpness-Aware Minimization (EISAM). EISAM improves tail-item performance by adaptively regularizing the loss landscape at the item level. Unlike existing global sharpness-aware minimization methods, EISAM allows more granular control of the loss landscape's flatness. Additionally, EISAM introduces an efficient penalty design that captures fine-grained item-specific sharpness while maintaining computational scalability for large language models.

• �� Item-wise Sharpness Regularization: For each item, adaptively regularize the loss landscape to improve tail-item performance.
• �� Frequency-dependent Weighting Function: Introduce a frequency-dependent weighting function to emphasize tail items and enhance their recommendation performance.
• �� Theoretical Analysis: Derive a generalization bound for EISAM and provide theoretical analysis showing that the bound decreases at a faster rate under item-wise regularization.
• �� Experimental Validation: Validate EISAM's effectiveness across multiple real-world datasets, demonstrating significant performance improvements on tail items.

Experiments are conducted on three real-world datasets: MovieLens-1M, Steam, and Amazon Digital Music (ADM). These datasets cover different domains and data distributions, effectively validating EISAM's performance in long-tail recommendation scenarios. The experimental design includes comparisons with existing long-tail recommendation methods, ablation studies to verify the roles of item-wise sharpness regularization and frequency-dependent weighting functions, and performance evaluation under different data distributions. Key hyperparameters include the form of the weighting function and the strength of item-wise sharpness regularization.

Experimental results show that EISAM significantly boosts tail-item recommendation performance. For instance, on the MovieLens-1M dataset, EISAM improved tail-item recommendation accuracy by over 15%. Additionally, compared to existing long-tail recommendation methods, EISAM shows superior performance improvements on tail items, especially on datasets with significant data long-tail effects. Ablation studies reveal that item-wise sharpness regularization is a key factor in enhancing tail-item performance, while the frequency-dependent weighting function effectively emphasizes tail items.

EISAM can be directly applied to recommender systems dealing with long-tail data distributions, such as product recommendations on e-commerce platforms and video recommendations on streaming platforms. By enhancing tail-item recommendation performance, EISAM can improve recommendation diversity and novelty, enhancing user experience. Additionally, EISAM can be applied to other types of recommender systems, such as social network recommendations, to validate its broader applicability.

Despite EISAM's impressive performance in addressing the long-tail problem, its performance improvement under extreme data long-tail distributions may require more complex weighting function designs. Additionally, due to the need for multiple backpropagations, EISAM incurs higher computational costs compared to standard training methods, which may affect training efficiency on large-scale datasets. Future research could explore more efficient weighting function designs and apply EISAM to other types of recommender systems to validate its broader applicability.