LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
LeAP learns permutation-based gates and removes 3,600+ dimensions from a 12,000+ dimensional industrial model with zero business-metric degradation.
Key Findings
Methodology
LeAP replaces repeated Permutation Feature Importance with learnable gating in one forward pass. For each feature, batch-wise shuffling produces noise x′i; a temperature-scaled gate gi=σ(θi/τ) forms x̃i=gixi+(1−gi)sg(x′i). The method measures permutation divergence Δi=1/B∑b||x(b)i−x′(b)i||2, smooths it with EMA, sets λi=αΔ̄i, and optimizes Ltotal=Ltask+∑iλigi.
Key Results
- On Avazu, Criteo, ML-1M, and AliCCP, LeAP achieved the best reported SAUC at both 50% and 25% Feature Retention. The paper emphasizes particularly strong gains over AutoField, LPFS, SFS, and SHARK on the more complex Criteo and AliCCP datasets.
- In production, the search-ranking model has 500+ validated fields, over 12,000 dimensions, 2 TB of parameters, and more than one billion daily requests. LeAP removed 3,600+ redundant dimensions—over 30%—with Zero Diff in core online metrics, reportedly 2–10 times the pruning capability of baselines.
- The analysis shows that if signal strength ΔJ=J(0)−J(1) exceeds λi, optimization drives gi toward 1; for redundant features with ΔJ≈0, regularization drives gi toward 0. Unlike SHARK’s greedy elimination, LeAP jointly handles feature interactions.
Significance
The work addresses three persistent industrial obstacles: heterogeneous feature widths, extreme sparsity, and the cost of permutation evaluation. It combines interpretable perturbation-based importance with trainable gates, allowing selection to run inside ordinary model training and enabling aggressive bandwidth and storage reduction. The results suggest that feature compression can preserve rare personalized signals and business performance rather than treating low activation frequency as evidence of irrelevance.
Technical Contribution
The main contributions are an O(1)-pass Learnable Adaptive Permutation mechanism, avoiding N repeated inference passes; a Permutation-Divergence regularizer whose strength follows observed perturbation rather than a uniform λ∑gi; a convexity-based explanation of gate polarization under ΔJ>λi; and a practical plug-in pipeline with threshold or Top-K pruning followed by limited fine-tuning. The method operates directly on native heterogeneous concatenations.
Novelty
Unlike AutoField’s Gumbel-Softmax, LPFS’s smoothed L0 regularization, and SHARK’s efficient permutation strategy, LeAP does not equate parameter or gate magnitude with importance. It learns how much prediction changes when a feature is replaced by distribution-preserving batch noise. Its fundamental novelty is combining permutation interpretability, differentiable joint optimization, and data-adaptive treatment of dimensionality and sparsity in one model-agnostic module.
Limitations
- The public benchmarks use uniform feature dimensions, so they do not fully reproduce the 1D-to-256D heterogeneous and 99% sparse conditions central to the industrial claim. The strongest evidence for those conditions comes from one deployment scenario.
- The polarization theorem assumes convex expected task loss J(gi), whereas deep recommender models are generally non-convex. The theorem is therefore explanatory and conditional, not a universal guarantee for stochastic deep optimization.
- The supplied paper text does not report per-dataset AUC or SAUC values and omits a complete hyperparameter table, limiting precise statistical and engineering reproducibility.
Future Work
Future work should build public benchmarks with native 1D–256D mixtures and controlled 99% sparsity, reporting complete AUC, SAUC, latency, memory, and significance curves. Further directions include non-convex optimization analysis, hierarchical or group gates, drift-aware online reselection, and joint prediction–cost optimization. Applying LeAP to advertising, search, multimodal systems, and dynamic neural architecture selection would test its generality.
AI Executive Summary
Industrial recommender systems now depend on thousands of heterogeneous inputs. A real-time statistic may occupy one dimension, while a behavioral embedding may occupy 256; together they create terabyte-scale models, expensive training, and severe CPU–GPU bandwidth pressure. Classical Lasso and tree methods are poorly matched to large embeddings. AutoField, LPFS, and related mask methods often assume comparable feature spaces, while SHARK and other permutation approaches require repeated inference. Extreme sparsity creates another trap: a feature that is zero in 99% of examples may still encode a valuable long-tail signal.
LeAP, or Learnable Adaptive Permutation, turns this expensive evaluation into a trainable plug-in. Within each mini-batch, each feature is independently shuffled, preserving its marginal distribution and sparsity pattern while breaking its association with the label. A learnable gate gi then interpolates between the original feature and stop-gradient shuffled noise: x̃i=gixi+(1−gi)sg(x′i). All features can therefore be tested in one forward pass, reducing the stated complexity from O(N×model overhead) to O(1×model overhead). Its central safeguard is Permutation Divergence: the L2 distance between original and shuffled values, smoothed by EMA and converted into λi=αΔ̄i.
LeAP achieved the best reported SAUC at 50% and 25% retention on Avazu, Criteo, ML-1M, and AliCCP. In a search-ranking system serving over one billion requests daily, with 2 TB of parameters and 12,000+ input dimensions, it removed more than 3,600 dimensions—over 30%—without degradation in core online metrics. The broader contribution is a practical bridge between interpretable perturbation tests and differentiable feature selection. Nevertheless, public benchmarks are dimensionally uniform, detailed metric tables are absent from the supplied text, and the theory relies on a convexity assumption that does not generally hold in deep networks.
Deep Analysis
Background
Recommendation, advertising, and search models have evolved from compact handcrafted inputs to thousands of scalars, categorical fields, and behavioral embeddings. WideDeep and related deep architectures improve representation power but create terabyte-scale parameters and bandwidth bottlenecks. Lasso, Random Forest, and XGBoost remain interpretable but are not designed for high-dimensional embeddings. AutoField, LPFS, and SFS introduce differentiable masks, yet often use uniform regularization. SHARK improves permutation efficiency but retains repeated evaluation and sparse-feature bias.
Core Problem
The objective is to identify and remove redundant feature dimensions without materially harming prediction. The challenge has four parts: 1D statistics and 256D embeddings are not naturally comparable; default values dominate extremely sparse fields; conventional permutation requires O(N) forward passes; and neural parameter or gate magnitude is not equivalent to predictive importance because nonlinear interactions can create large effects from small weights.
Innovation
- �� Learnable permutation: independently shuffle fields within a batch and interpolate original and noisy values through gates, avoiding one-feature-at-a-time inference. • Adaptive regularization: compute Δi=1/B∑||xi−x′i||2 and use EMA to obtain λi=αΔ̄i. • Polarization analysis: useful features move toward gi=1 when ΔJ>λi, while redundant ones move toward zero. • Deployment design: insert LeAP after concatenation, then prune by a fixed threshold or Top-K ranking and fine-tune briefly.
Methodology
- �� Input: F fields xi∈Rdi with heterogeneous di and batch size B. • Shuffle: sample an independent RandomPermutation(B) for each field, producing x′i that preserves marginal statistics but breaks sample-level association. • Gate: compute gi=σ(θi/τ) and x̃i=gixi+(1−gi)sg(x′i); stop-gradient prevents noise from updating upstream representations. • Regularize: measure batch L2 divergence, smooth it as Δ̄(t)i=βΔ̄(t−1)i+(1−β)Δ(t)i, and set λi=αΔ̄i. • Optimize Ltotal=Ltask+∑λigi. • Prune: remove gi<0.5 or retain a desired Top-K fraction, delete LeAP, and fine-tune.
Experiments
The public datasets are Avazu (40,428,967 samples, 23 fields), Criteo (45,850,617, 39 fields), ML-1M (1,000,209, 9 fields), and AliCCP (85,316,519, 23 fields). WideDeep is the backbone, and the Search-Retrain protocol compares Lasso, Random Forest, XGBoost, AutoField, LPFS, SFS, and SHARK. Metrics are AUC and normalized cross-dataset SAUC, mainly at 50% and 25% retention. The industrial study uses long-video search, impression, click, and interaction logs, with Group AUC offline and strict online A/B testing.
Results
LeAP obtained the highest reported SAUC on all four public datasets at both retention ratios, with especially clear advantages on Criteo and AliCCP. The industrial model contains 500+ fields, 12,000+ dimensions, and 2 TB of parameters while serving over one billion daily requests. LeAP removed 3,600+ dimensions with Zero Diff in core metrics. Baselines showed substantial degradation when removing fewer than 600 dimensions, highlighting LeAP’s robustness to feature coupling, heterogeneity, and sparsity.
Applications
The immediate use case is input compression for search-ranking, recommendation, and advertising models. Teams can place LeAP after feature concatenation, train it on ordinary logs, accumulate EMA statistics, and prune by gate values. The method can reduce GPU memory traffic, CPU–GPU transfer, training cost, and model storage, provided a stable task model, batch data access, and a short post-pruning fine-tuning stage are available.
Limitations & Outlook
The supplied paper text lacks per-dataset AUC/SAUC tables, and uniform public features limit external validation of heterogeneous inputs. The convexity assumption is unrealistic for general deep networks. Batch shuffling may be noisy for small batches, strongly time-dependent fields, or distribution shift. EMA, temperature, and per-field gates add training overhead, while long-term deployment requires periodic reselection to avoid deleting features whose business value changes.
Plain Language Accessible to non-experts
Imagine a huge restaurant whose menu is controlled by thousands of ingredients. Some ingredients are tiny grains of salt; others are whole boxes of vegetables. Traditional methods either judge every ingredient by the same rule or remove one ingredient at a time and cook the entire meal again. Both are slow. Worse, an ingredient used only once in a hundred meals might be treated as useless even though it creates an important specialty dish.
LeAP acts like a clever head chef. During one cooking round, it mixes up the source labels for every ingredient and watches how much the dishes change. If mixing a particular ingredient hardly changes the taste, that ingredient may be unnecessary. If the taste changes strongly, it should stay. Because many ingredients are tested in the same round, the process is much faster than rebuilding the meal separately for each one.
The chef also adjusts the inspection according to how much the ingredient actually changes when mixed. Large ingredients naturally create larger changes, while rare ingredients are not punished merely because they appear infrequently. They are removed only when the final taste shows that they contribute nothing. In the paper’s industrial case, this process removed more than 3,600 of over 12,000 ingredient positions while keeping core business results unchanged.
ELI14 Explained like you're 14
Imagine a video app trying to guess what you want to watch next. It has more than 12,000 clues: what you watched, when you watched it, which creators you follow, and so on. Some clues are just one number; others are giant lists. Too many clues make the system slow, so we need to find the useless ones.
LeAP plays a clever experiment. It scrambles the clues from different users, like swapping students’ name tags in a group project, and checks whether the prediction gets worse. It does this for many clues in one round instead of testing each clue separately. Every clue gets a tiny switch: near 1 means “keep it,” and near 0 means “this probably adds nothing.”
Here is the smart part: a clue that is blank 99% of the time is not automatically thrown away. Maybe it becomes super important when it finally appears! LeAP asks whether scrambling the clue actually changes the information, rather than judging only by how often it appears or how many numbers it contains.
The researchers tested Avazu, Criteo, ML-1M, and AliCCP, then used the method in a search system handling over one billion requests every day. The system had 12,000+ dimensions, and LeAP removed 3,600+ without hurting core metrics. It is like cleaning a gaming backpack by testing whether dropping an item really lowers your battle power—not simply deleting the items you use least!
Glossary
Learnable Adaptive Permutation
A trainable mechanism that replaces repeated random permutation tests with gates. Each gate learns whether the original feature or shuffled noise should be used.
This is the central LeAP module.
Permutation Feature Importance
Feature importance estimated by shuffling a feature and measuring prediction degradation. It is interpretable but expensive when applied separately to many features.
LeAP preserves its intuition while making evaluation joint and differentiable.
Permutation Divergence
The average L2 distance between an original feature and its shuffled counterpart, Δi=1/B∑||xi−x′i||2. It measures the size of the information perturbation.
It determines each feature’s adaptive regularization strength.
Exponential Moving Average
A weighted combination of historical and current statistics that reduces batch noise. Here, Δ̄t=βΔ̄t−1+(1−β)Δt.
It stabilizes permutation-divergence estimates.
Gate
A learnable value between zero and one controlling feature retention. A gate near one keeps the original feature; a gate near zero favors shuffled noise and pruning.
LeAP obtains it through a temperature-scaled sigmoid.
SAUC
A cross-dataset normalized AUC: each method’s AUC is divided by the best AUC on that dataset and then averaged. It supports aggregate comparison.
It is a primary public-benchmark metric.
Open Questions Unanswered questions from this research
- 1 Because public benchmarks use uniform feature dimensions, they do not fully test native 1D–256D mixtures or 99% sparsity. Reproducible heterogeneous benchmarks with complete per-dataset metrics are needed.
- 2 The polarization proof assumes convex J(gi), unlike most deep recommenders. Probabilistic guarantees under non-convex stochastic training, feature interaction, and distribution shift remain open.
- 3 The supplied text does not quantify extra training cost, EMA sensitivity, or the required frequency of online reselection. These determine the total cost of sustained deployment.
Applications
Immediate Applications
Search-ranking compression
Insert LeAP after feature concatenation, train gates on production logs, and remove dimensions by threshold or Top-K ranking. This suits teams facing GPU-memory and bandwidth bottlenecks; the reported case removed 3,600+ dimensions with Zero Diff in core online metrics.
Recommendation and advertising cleanup
Apply sensitivity-based selection to statistics, categorical embeddings, and behavioral vectors instead of deleting fields solely by frequency. After gates converge, remove LeAP and fine-tune on a small data fraction to reduce training and serving resources.
Long-term Vision
Dynamic feature lifecycle management
Integrate LeAP with continual training, drift monitoring, latency budgets, and storage costs. Periodic reselection could maintain different feature sets by scenario, device, or user segment, while protecting newly valuable signals.
Multimodal input optimization
Extend permutation divergence to text, image, sequence, and sparse-ID representations, creating an interpretable input-pruning framework. Major obstacles include cross-modal dependence, temporal causality, and higher evaluation cost.
Abstract
Modern industrial recommender systems rely on thousands of heterogeneous features -- ranging from low-dimensional scalars (e.g., statistical value) to high-dimensional embeddings (e.g., user-id embeddings, MLP representations) -- to achieve high-precision predictions. Given the immense computational costs associated with training, efficient feature selection is critical. However, existing methods encounter three primary bottlenecks: (1) they typically assume uniform feature dimensions or require costly mapping to a fixed size; (2) they struggle with extreme sparsity, where the majority of features (e.g., 99%+) remain at default values; and (3) traditional permutation-based approaches are computationally prohibitive in large-scale settings. To address these challenges, we propose LeAP (Learnable Adaptive Permutation), a novel, model-agnostic plug-in module for feature selection. LeAP transforms the inefficient random permutation process into a learnable mechanism, significantly accelerating the evaluation of feature importance. In addition, we introduce an adaptive regularization strategy tailored for heterogeneous dimensions and extreme sparsity, enabling superior feature importance ranking results across asymmetric input spaces. Experiments on four public recommendation datasets demonstrate that LeAP achieves state-of-the-art performance. Furthermore, LeAP has been deployed in a large-scale industrial search ranking model with over a billion daily requests and a 2TB model parameter scale. In this real-world scenario involving 12,000+ total feature dimensions, LeAP successfully identified and removed over 3,600 redundant dimensions without performance degradation, which is 2 to 10 times the ability of compared baseline methods.