TRWH: A Text-Driven Random Walk Heterogeneous GNN for Semantic-Aware Sparse Recommendation
TRWH combines LLM-generated text features with heterogeneous graphs via random walks, boosting sparse recommendation accuracy.
Key Findings
Methodology
TRWH integrates three components: first, uses Word2Vec and LLMs to generate textual profiles for users and items; second, constructs a multi-relational heterogeneous graph and employs HeteroGNN for message passing; third, applies one-hop random walks to add second-order user-user and item-item links, enriching sparse graphs. Experiments on Amazon 2023 Fashion and Beauty datasets show RMSE reductions of 80% and 25.7%, MAE drops of 52.6% and 10.8%, outperforming state-of-the-art baselines. The framework effectively fuses semantic and structural signals, addressing data sparsity.
Key Results
- On Fashion, TRWH achieves 80.0% RMSE reduction and 52.6% MAE reduction over baselines like LightGCN, demonstrating the synergy of text features and graph augmentation.
- On Beauty, the model improves RMSE by 25.7% and MAE by 10.8%, indicating strong generalization across domains.
- Ablation studies reveal that random walk augmentation enhances traditional embeddings but can dilute LLM semantics, highlighting the need for adaptive fusion strategies.
Significance
This work advances recommendation technology by effectively integrating semantic-rich textual features with structural graph information, especially in sparse data scenarios. It addresses longstanding challenges of cold-start and long-tail distributions, providing a scalable solution for real-world large-scale systems. The approach bridges the gap between language understanding and graph modeling, paving the way for more intelligent, context-aware recommendations in industry and academia.
Technical Contribution
The paper introduces TRWH, a novel framework combining LLM-based textual profiles with multi-relational heterogeneous graphs, enhanced by a strategic one-hop random walk to incorporate second-order relations. It innovates by balancing semantic detail preservation with structural augmentation, employing explicit edge-type modeling and adaptive fusion strategies. The design improves information propagation in sparse graphs, offering both theoretical insights and practical efficiency gains validated on large datasets.
Novelty
TRWH is the first to systematically fuse LLM-generated text features with heterogeneous graph structures via targeted random walks, specifically addressing the challenge of semantic dilution in sparse graphs. Its integration of multi-relation edges and adaptive strategies for semantic-structural balance marks a significant step beyond prior works that treat content and structure separately or rely on simple concatenation.
Limitations
- Random walk augmentation can introduce noise, potentially diluting fine-grained semantic signals from LLMs, especially in highly sparse graphs, requiring more sophisticated filtering.
- The computational cost of large LLMs remains high, limiting real-time deployment in resource-constrained environments.
- Model performance may degrade in extreme cold-start scenarios where textual profiles are insufficient or inaccurate, necessitating further robustness improvements.
Future Work
Future research will explore multi-hop and dynamic random walk strategies, integrating reinforcement learning to adaptively select relations. Additionally, multi-modal data such as images and videos could be incorporated to enrich profiles further. Efforts will also focus on reducing computational overhead, enabling real-time deployment, and extending the framework to cross-domain and multi-task recommendation settings.
AI Executive Summary
Recommender systems face persistent challenges in effectively capturing user preferences amid sparse interaction data. Traditional collaborative filtering methods struggle with cold-start and long-tail issues, while content-based approaches often overlook structural relationships. Recent advances leverage graph neural networks (GNNs) to model user-item interactions as graphs, enabling richer representation learning. However, these methods are limited by the sparsity of interaction graphs and the lack of semantic understanding. Concurrently, large language models (LLMs) have demonstrated remarkable capabilities in understanding and generating natural language, offering a new avenue for extracting detailed textual features from reviews, descriptions, and metadata.
This paper introduces TRWH, a novel framework that synergistically combines LLM-generated textual profiles with heterogeneous graph structures through strategic random walks. The core idea is to leverage LLMs to produce semantic-rich representations of users and items, then embed these into a multi-relational graph that captures diverse interaction types such as ratings, reviews, and purchase history. To address the sparsity problem, the framework employs a one-hop random walk mechanism to add second-order links, connecting similar users and items indirectly, thereby enriching the graph’s structural information.
The architecture comprises three main modules: embedding creation, graph construction with multiple relation types, and a HeteroGNN for message passing. Experimental results on Amazon 2023 datasets show that TRWH outperforms state-of-the-art models like LightGCN and HAN, achieving up to 80% RMSE reduction and significant improvements in MAE. These findings demonstrate the effectiveness of integrating semantic and structural signals, especially in large-scale, sparse scenarios.
Beyond performance gains, TRWH offers a new perspective on combining language understanding with graph modeling, opening pathways for more intelligent, context-aware recommendation systems. Nonetheless, challenges remain in balancing semantic detail with structural augmentation and managing computational costs. Future work will explore multi-hop relations, multi-modal data, and efficiency improvements, aiming to push recommendation technology toward deeper semantic-structural integration and broader applicability.
Deep Dive
Abstract
Graph Neural Networks (GNNs) and Large Language Models (LLMs) have each advanced recommendation systems by modeling structural and semantic signals, respectively. However, integrating their complementary strengths remains challenging, particularly in sparse settings where maintaining semantic precision is critical. We propose TRWH (Text-driven Random Walk Heterogeneous Graph Neural Network), a novel framework that fuses LLM-generated textual profiles with heterogeneous graph structures through strategic random walk augmentation. TRWH consists of three core components: (1) Embedding Creation, which produces user and item representations using both Word2Vec and LLM-based profiling; (2) a Heterogeneous Graph Neural Network (HeteroGNN) that propagates information across multi-relational edges; and (3) Random Walk-based Path Construction, which enriches sparse graphs with second-order user-user and item-item links. Experiments on the Amazon-2023 Fashion (2M users, 825K items) and Beauty (631K users, 112K items) datasets demonstrate that TRWH achieves substantial performance gains over state-of-the-art methods, including 80.0% RMSE and 52.6% MAE reductions on Fashion, and 25.7% and 10.8% improvements on Beauty. Notably, while random walks improve performance with traditional embeddings, they can dilute the nuanced representations learned by LLMs, underscoring the importance of adaptive integration strategies.