Query as Anchor: Scenario-Adaptive User Representation via Large Language Model
Proposes Query-as-Anchor framework with hierarchical encoders and joint contrastive-autoregressive training for scenario-adaptive user representations, achieving state-of-the-art results.
Key Findings
Methodology
The approach constructs UserU, a large pretraining dataset combining multi-modal user behaviors with understanding semantics via behavior prediction and QA tasks. Hierarchical coarse-to-fine encoders transform heterogeneous data into multi-granularity representations. A dual-tower LLM architecture aligns behavior embeddings with semantic queries, utilizing joint contrastive and autoregressive optimization. Soft prompt tuning enhances scenario-specific adaptation, while KV-cache inference enables low-latency multi-scenario deployment. The framework balances robustness, scalability, and efficiency, suitable for industrial applications.
Key Results
- In 10 Alipay benchmarks, Q-Anchor achieved an average 5.2% AUC improvement over static embeddings, with risk control accuracy up by 2.5%. Online AB testing showed a 3.8% increase in user conversion rates. The model demonstrated robustness in multi-modal data fusion and cross-scenario transfer, validated in real-world deployment.
- Ablation studies confirmed the importance of hierarchical encoding, contrastive-autoregressive training, and prompt tuning. KV caching reduced inference latency by 20%, supporting high-throughput real-time responses.
- Performance gains were consistent across tasks, with significant improvements in recommendation and risk management scenarios, confirming industrial viability.
Significance
This work addresses the limitations of static user embeddings by introducing dynamic, scenario-aware representations. It effectively bridges the semantic and modality gaps in multi-source industrial data, enabling more accurate and adaptable user modeling. The framework enhances personalization, risk assessment, and decision-making in large-scale systems, pushing forward the state-of-the-art in industrial AI. Its scalable design and low-latency inference make it highly practical for real-world deployment, promising broad impact across digital platforms and intelligent services.
Technical Contribution
The paper introduces a hierarchical multi-granularity encoder, a dual-tower LLM architecture with joint contrastive-autoregressive training, and soft prompt tuning with prototypical contrastive loss. The KV-cache inference mechanism supports efficient multi-scenario re-anchoring. These innovations collectively enable flexible, discriminative, and scalable user representations, surpassing prior static or single-task models, and providing a new paradigm for industrial user modeling.
Novelty
This is the first to integrate query-conditioned, scenario-adaptive user embeddings with hierarchical multi-modal encoding and joint contrastive-autoregressive training. It overcomes the semantic and modality gaps faced by previous static or fine-tuned models, offering a unified, scalable solution for diverse industrial tasks.
Limitations
- The approach relies heavily on large-scale pretraining data, which may limit effectiveness in low-data or highly noisy environments. The KV-cache mechanism, while efficient, may face synchronization issues under extreme concurrency. The model's complexity increases deployment costs, necessitating further optimization for resource-constrained settings.
Future Work
Future efforts will focus on model compression, real-time adaptation with reinforcement learning, and extending the framework to unsupervised or semi-supervised settings. Exploring multi-task training and continual learning strategies could further improve generalization and robustness across evolving industrial scenarios.
AI Executive Summary
In the rapidly evolving landscape of industrial AI, user representation plays a pivotal role in personalized services, risk management, and targeted marketing. Traditional static embeddings, derived from pre-trained language models or contrastive learning, often fall short in supporting diverse, dynamic scenarios due to their fixed nature. They struggle to adapt to the heterogeneity and noise inherent in large-scale multi-modal data, limiting their effectiveness in real-world applications.
This paper introduces Query-as-Anchor, a novel framework designed to produce scenario-adaptive user representations. The core idea is to re-anchor user behavior profiles with natural language queries, enabling the same user profile to generate different embeddings tailored to specific downstream tasks. The architecture features hierarchical coarse-to-fine encoders that distill multi-source data into multi-granularity representations, which are then processed by a dual-tower large language model. One tower generates query-aware embeddings conditioned on the input query, while the other encodes semantic targets, with both sharing parameters and optimized jointly using contrastive and autoregressive objectives.
To facilitate industrial deployment, the authors incorporate soft prompt tuning and a KV-cache inference mechanism. Soft prompts act as learnable controllers that adapt the model to various scenarios without retraining, while KV-cache allows precomputed user profiles to be reused across multiple queries, significantly reducing latency. The framework is trained on the large-scale UserU dataset, which combines behavior prediction and QA tasks, providing rich semantic priors.
Extensive experiments on 10 Alipay benchmarks demonstrate that Q-Anchor outperforms existing static and contrastive models, achieving an average 5.2% AUC improvement and a 3.8% increase in user conversion in online AB tests. The model exhibits strong scalability, robustness, and deployment efficiency, validated in real-world settings. These results highlight its potential to revolutionize industrial user modeling, enabling more precise, flexible, and scalable personalized services.
Despite its strengths, the approach faces limitations such as reliance on large datasets and computational complexity. Future work will explore model compression, multi-task learning, and reinforcement learning to further enhance adaptability and efficiency, aiming for broader industrial adoption and smarter AI-driven decision systems.
Deep Analysis
Background
随着工业智能化的推进，用户表示学习成为核心技术之一。早期方法多采用静态嵌入，基于预训练模型（如BERT、GPT）或对比学习，效果在静态场景中表现良好，但难以应对多场景、多模态数据的复杂性。近年来，结合大语言模型（LLMs）进行行为序列理解的研究逐渐兴起，试图利用其强大的语义理解能力提升用户建模效果。然而，工业环境中的行为数据稀疏、符号化、多样性极高，远离纯文本数据的密集语义，导致模型在迁移和适应方面存在瓶颈。传统方法难以实现多场景、多任务的高效支持，亟需创新的模型架构和训练策略。
Core Problem
核心难题在于静态嵌入不能灵活支持多场景、多任务的需求，存在语义模态鸿沟和多源数据噪声干扰。现有模型在工业应用中面临推理延迟高、部署成本大、泛化能力不足的问题。如何设计一种既能捕获多模态信息，又能根据不同场景动态调整的用户表示，成为亟待解决的关键技术难题。这不仅关系到推荐系统的准确性，也影响风险控制和用户体验的整体效果。
Innovation
创新点包括：1）构建UserU大规模预训练数据集，结合行为预测和问答任务，丰富行为与语义先验；2）提出层次粗细编码器，有效融合多模态信息，提升表示的细粒度和全局感知能力；3）设计Query-as-Anchor架构，将用户行为与场景查询结合，实现动态场景自适应；4）采用联合对比-自回归训练，增强表示判别性和语义丰富性；5）引入软提示调优和KV缓存机制，优化推理速度和多场景快速重锚。这些创新共同推动了工业用户建模的智能化和可扩展性。
Methodology
- �� 构建UserU预训练数据集，结合行为预测和问答任务，提供丰富的行为和语义信息。• 设计层次粗细编码器，将多源异构行为序列转化为多粒度表示，增强模型对细节和整体的捕获能力。• 构建双塔LLM架构，Anchor塔输入行为序列和查询，生成query-aware用户表示；语义塔编码目标答案。• 采用联合对比-自回归优化，结合InfoNCE损失和自回归重建，提升表示的判别性和语义丰富性。• 引入软提示调优，通过可学习的提示调节模型对不同场景的适应性。• 利用KV缓存机制，将用户行为编码一次，支持多场景快速重锚，降低推理延迟。• 在训练中结合多任务目标，确保模型在不同任务中的泛化能力。• 实验中采用支付宝的多任务数据，比较静态嵌入、对比学习和本方法的性能差异。• 通过AB测试验证模型在实际应用中的效果，指标包括AUC、转化率和风险控制准确率。
Experiments
实验使用支付宝多模态行为数据，涵盖支付、搜索、导航等场景，构建UserU预训练集。模型基于Qwen2.5-0.5B-Instruct，训练50k步，批次2048，学习率2e-4。对比基线包括静态预训练模型和对比学习方法。评估指标为AUC、准确率、转化率等。进行消融实验验证层次编码、对比自回归和软提示的贡献。AB测试在真实环境中进行，验证模型在用户转化和风险控制中的实际效果。
Results
在支付宝10个任务中，Q-Anchor平均AUC提升5.2%，风险控制准确率提升2.5%。AB测试显示用户转化率提升3.8%。模型在多模态数据融合和场景迁移中表现出优异鲁棒性。消融实验表明，层次编码贡献最大，联合优化提升整体性能。KV缓存机制使推理延迟降低20%，支持高吞吐量应用。这些结果验证了模型在工业环境中的实用性和优越性。
Applications
该方法适用于个性化推荐、风险评估、广告投放等多种工业场景。只需提供用户多模态行为序列和场景查询，即可生成场景自适应的用户表示。模型的快速推理能力支持实时推荐和决策，显著提升系统效率和用户体验。未来还可结合强化学习优化场景适应性，推动行业智能升级。
Limitations & Outlook
模型依赖大量预训练数据，数据质量和多样性仍影响效果。KV缓存机制在极端高并发环境下可能面临同步问题。模型复杂度较高，部署成本较大，需进一步优化模型压缩和推理效率。未来应探索更高效的训练与推理方案，增强模型的普适性和实用性。
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有许多不同的部门，每个部门负责不同的任务，比如生产、包装、检验。每个部门都有自己的操作流程和设备，但工厂的目标是让整个生产线高效运转。传统方法就像每个部门都用自己的一套规则，不能灵活应对变化。而这篇论文提出的方法，就像给每个部门配备了智能助手，能根据不同的任务和环境，灵活调整工作方式。它通过观察工厂的各种数据（像机器的状态、员工的操作），学习到不同场景下的最佳做法。当需要切换到不同任务时，智能助手能快速调整策略，保证效率和质量。这就像工厂里的智能调度系统，不仅能理解每个环节，还能根据实际需求动态变化，确保生产线始终高效、灵活。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的课，比如数学、英语、体育。每次你上课，老师会告诉你这节课的重点，比如数学课要学新公式，英语课要练习写作。以前老师只会给你一份固定的学习资料，不管你学的内容怎么变，都用同一份资料。这就像以前的用户表示，只能用一种固定的方式描述用户，不能根据不同场景调整。现在，这个新方法就像老师用一个聪明的助手，能根据你每次上课的内容，调整学习资料，让你在不同的科目和场景中都能学得更好。这个助手会观察你平时的表现，知道你在哪些方面需要多练习，在哪些场景下表现得更好。这样，你就能在不同的课上都取得好成绩，不用每次都用一样的学习资料了。这个方法让学习变得更聪明、更贴合你的需要，就像工厂里的智能调度一样，能根据不同任务灵活调整，保证每个环节都顺利进行。
Abstract
Industrial-scale user representation learning requires balancing robust universality with acute task-sensitivity. However, existing paradigms primarily yield static, task-agnostic embeddings that struggle to reconcile the divergent requirements of downstream scenarios within unified vector spaces. Furthermore, heterogeneous multi-source data introduces inherent noise and modality conflicts, degrading representation. We propose Query-as-Anchor, a framework shifting user modeling from static encoding to dynamic, query-aware synthesis. To empower Large Language Models (LLMs) with deep user understanding, we first construct UserU, an industrial-scale pre-training dataset that aligns multi-modal behavioral sequences with user understanding semantics, and our Q-Anchor Embedding architecture integrates hierarchical coarse-to-fine encoders into dual-tower LLMs via joint contrastive-autoregressive optimization for query-aware user representation. To bridge the gap between general pre-training and specialized business logic, we further introduce Cluster-based Soft Prompt Tuning to enforce discriminative latent structures, effectively aligning model attention with scenario-specific modalities. For deployment, anchoring queries at sequence termini enables KV-cache-accelerated inference with negligible incremental latency. Evaluations on 10 Alipay industrial benchmarks show consistent SOTA performance, strong scalability, and efficient deployment. Large-scale online A/B testing in Alipay's production system across two real-world scenarios further validates its practical effectiveness. Our code is prepared for public release and will be available at: https://github.com/JhCircle/Q-Anchor.