ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendation
ROCS improves recommendation efficiency by delaying request-candidate interaction, achieving a 3x QPS boost.
Key Findings
Methodology
ROCS improves inference efficiency by delaying request-candidate interactions, isolating candidate-dependent representations, using Generalized Layer Masking (GLM) and Deep Cross Attention (DCA). IKBO is used for GPU acceleration.
Key Results
- On production-scale workloads, ROCS achieves a 3x QPS improvement on retrieval models without quality degradation and a 0.5% LogLoss improvement with a 50% QPS gain on short-form video ranking models.
- Experiments show ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones.
- ROCS has been deployed across ads and organic systems, spanning retrieval and ranking stages, significantly reducing infrastructure costs.
Significance
ROCS significantly enhances inference efficiency in recommendation systems by leveraging compute sharing, addressing production cost constraints, and improving scalability without sacrificing prediction quality.
Technical Contribution
ROCS introduces a new inference paradigm, achieving candidate isolation and request sharing through GLM and DCA, combined with IKBO for efficient GPU execution.
Novelty
ROCS is the first to systematically exploit request-sharing properties in recommendation inference, delaying request-candidate interaction to maximize reusable computation.
Limitations
- ROCS may struggle in scenarios requiring high real-time responsiveness.
- Models may need adjustments for different recommendation contexts.
Future Work
Future work could explore ROCS applications in other domains, such as real-time ad recommendations, and further optimize IKBO to support more complex model structures.
AI Executive Summary
Modern recommendation models enhance prediction quality by scaling feature-interaction and sequence modules, but production costs limit system scalability. ROCS proposes a request-oriented compute sharing paradigm that significantly improves inference efficiency by delaying request-candidate interactions and isolating candidate-dependent representations, while maintaining or improving prediction quality. ROCS develops Generalized Layer Masking (GLM) and Deep Cross Attention (DCA) to extend request-oriented sharing to sequence architectures. To support efficient GPU deployment, ROCS co-designs In-Kernel Broadcast Optimization (IKBO), significantly accelerating model execution. Experiments show ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones. On production-scale workloads, ROCS achieves up to a 3x QPS improvement on retrieval models without quality degradation and a 0.5% LogLoss improvement with a 50% QPS gain on short-form video ranking models. ROCS has been deployed across ads and organic systems, spanning retrieval and ranking stages, significantly reducing infrastructure costs. ROCS significantly enhances inference efficiency in recommendation systems by leveraging compute sharing, addressing production cost constraints, and improving scalability without sacrificing prediction quality. Future work could explore ROCS applications in other domains, such as real-time ad recommendations, and further optimize IKBO to support more complex model structures.
Deep Analysis
Background
Recommendation systems have rapidly evolved, particularly in feature interaction and sequence modeling. Traditional methods like model unification and knowledge transfer, while effective, are limited by domain interference and freshness requirements. ROCS improves inference efficiency by delaying request-candidate interactions and isolating candidate-dependent representations.
Core Problem
Recommendation system scalability is limited by production costs and latency. Improving inference efficiency without sacrificing prediction quality is a key challenge.
Innovation
ROCS improves inference efficiency by delaying request-candidate interactions, isolating candidate-dependent representations, using GLM and DCA. IKBO is used for GPU acceleration, significantly reducing infrastructure costs.
Methodology
- �� Use GLM for candidate isolation.
- �� Use DCA to extend request sharing to sequence architectures.
- �� Combine with IKBO for efficient GPU execution.
Experiments
Experiments use public benchmarks and production-scale workloads to evaluate ROCS across recommendation backbones. Key metrics include QPS and LogLoss.
Results
ROCS achieves a 3x QPS improvement on retrieval models without quality degradation and a 0.5% LogLoss improvement with a 50% QPS gain on short-form video ranking models.
Applications
ROCS has been deployed across ads and organic systems, spanning retrieval and ranking stages, significantly reducing infrastructure costs.
Limitations & Outlook
ROCS may struggle in scenarios requiring high real-time responsiveness. Models may need adjustments for different recommendation contexts.
Plain Language Accessible to non-experts
Imagine a large supermarket where customers need to choose many products while shopping. Traditional recommendation systems are like recalculating all product prices and discounts every time a customer asks, while ROCS is like pre-calculating the customer's shopping list and quickly matching product information when asked. This method not only saves time but also improves efficiency.
ELI14 Explained like you're 14
Hey, imagine you're playing a game, and every time you choose a character, the game has to reload all the character information. That's slow, right? ROCS is like the game preloading all the character information, so when you choose, it quickly matches, making the game run faster! Isn't that cool?
Glossary
ROCS (Request-Oriented Compute Sharing)
An inference paradigm that improves efficiency by delaying request-candidate interaction.
Used to enhance inference efficiency in recommendation systems.
GLM (Generalized Layer Masking)
A technique for candidate isolation.
Used in feature-interaction architectures.
DCA (Deep Cross Attention)
A technique to extend request sharing to sequence architectures.
Used in sequence modeling.
IKBO (In-Kernel Broadcast Optimization)
A GPU acceleration technique.
Used to accelerate ROCS model execution.
QPS (Queries Per Second)
A metric for measuring system processing capacity.
Used to evaluate recommendation system efficiency.
Open Questions Unanswered questions from this research
- 1 How to apply ROCS in scenarios requiring high real-time responsiveness?
- 2 How to further optimize IKBO to support more complex model structures?
Applications
Immediate Applications
Ad Recommendation
ROCS can be used for real-time ad recommendations to improve system efficiency.
Long-term Vision
Complex Model Support
Future exploration of ROCS applications in more complex models.
Abstract
Modern recommendation models gain prediction quality by scaling feature-interaction and sequence modules, but production cost constraints cap how far systems can scale. In this work, we propose Request-Oriented Compute Sharing (ROCS), a modeling and inference paradigm that exploits a unique property of recommendation inference: each user request is evaluated against many candidates, while request-side features are shared across candidates. ROCS defers request-candidate interactions as late as possible, isolates candidate-dependent representations, and evaluates substantial portions of the model once per request rather than once per candidate, significantly improving inference efficiency while maintaining or improving prediction quality. To realize this paradigm, we develop Generalized Layer Masking (GLM) to enforce candidate isolation in feature-interaction architectures, and Deep Cross Attention (DCA) to extend request-oriented sharing to sequence architectures. To support efficient GPU deployment, we co-design In-Kernel Broadcast Optimization (IKBO) that significantly accelerates ROCS model execution. Experiments on public benchmarks show that ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones. On production-scale workloads, ROCS achieves up to a 3x QPS improvement on retrieval models without quality degradation and a 0.5% relative LogLoss improvement with a 50% QPS gain on a short-form video ranking model. ROCS has been deployed across large-scale recommendation systems spanning ads and organic surfaces, retrieval and ranking stages, and more than two orders of magnitude in inference complexity, delivering significant online gains at reduced infrastructure cost.