From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG
EPIC constructs preference-aligned compact index, reducing memory by 2404× and improving accuracy by 18.79%, enabling efficient on-device personalization.
Key Findings
Methodology
EPIC employs a three-stage approach: first, semantic-based coarse filtering using contrastive embeddings to eliminate irrelevant data; second, language model-based fine verification to ensure preference alignment and generate instruction anchors; third, preference-guided query steering shifts user queries toward preference directions in embedding space. The core algorithms include contrastive encoders like Contriever and Transformer-based verification models, integrated with FAISS for fast retrieval. This pipeline compresses data significantly while maintaining high preference fidelity, suitable for device constraints.
Key Results
- Across four benchmarks, EPIC reduces index memory by 2404×, increases preference-following accuracy by 18.79 percentage points, and cuts retrieval latency by 32.17×. Experiments on Jetson Nano, Mac M4, and Galaxy Z Flip 6 show memory use under 1MB, response times from 5.21 to 29.35 ms, supporting streaming preference updates.
- Compared to BM25, Contriever, and NV-Embed, EPIC outperforms in preference accuracy and efficiency, especially under preference drift scenarios, demonstrating robustness and scalability.
- Ablation studies reveal that semantic filtering contributes most to memory reduction, while verification ensures correctness, and query steering enhances recall, confirming the effectiveness of each component.
Significance
This work addresses the critical challenge of enabling personalized AI on resource-constrained devices by focusing on what to store rather than how to retrieve. It offers a scalable solution that balances privacy, efficiency, and personalization, paving the way for widespread deployment of private, on-device AI assistants. The approach significantly reduces memory footprint while maintaining high fidelity to user preferences, which is vital for real-world applications in mobile, IoT, and privacy-sensitive environments.
Technical Contribution
The paper introduces EPIC, a novel framework combining semantic filtering, language model verification, and preference-guided query modulation. It innovates by transforming personalization into a memory construction problem, emphasizing selective storage of preference-relevant data. The method achieves sub-1MB memory footprints with high preference accuracy, offering theoretical guarantees on preference alignment and practical efficiency, thus advancing the state-of-the-art in resource-limited personalized retrieval.
Novelty
EPIC is the first to integrate multi-stage preference-aware filtering, verification, and query steering into a unified, compact index suitable for on-device deployment. Unlike prior works that store all data or rely solely on query rewriting, EPIC emphasizes preference-guided selection and explicit instruction generation, addressing both storage constraints and alignment accuracy simultaneously.
Limitations
- The method depends on predefined explicit preference sets, limiting adaptability to implicit or evolving preferences. Dynamic preference learning remains an open challenge.
- Performance may degrade with noisy or ambiguous data, as filtering and verification rely on semantic similarity and language models, which can be sensitive to input quality.
- The computational overhead of verification and instruction generation, though minimal, still poses challenges for ultra-low-power devices. Further optimization is needed for broader deployment.
Future Work
Future research will explore online learning of preferences to adapt to changing user tastes, integrate multi-modal data for richer personalization, and optimize the pipeline for even lower-power hardware. Additionally, expanding to more complex, multi-turn interactions and multi-user scenarios will be critical for real-world applications.
AI Executive Summary
The rapid growth of large language models (LLMs) has fueled the development of personal AI assistants capable of handling diverse tasks. However, deploying such models on-device faces significant challenges, primarily due to limited memory and privacy concerns. Traditional retrieval-augmented generation (RAG) systems rely on storing vast amounts of data, which is infeasible on resource-constrained devices. Moreover, indiscriminate data storage can lead to responses that misalign with user preferences, reducing personalization quality.
This paper introduces EPIC (Efficient Preference-aligned Index Construction), a novel framework designed to build highly compact, preference-aligned memory for on-device AI agents. EPIC's core innovation lies in its multi-stage filtering and verification process. It begins with semantic-based coarse filtering, which rapidly prunes irrelevant data by leveraging contrastive embeddings like Contriever. Next, a language model performs fine-grained verification, ensuring that retained data genuinely aligns with user preferences, and generates explicit instructions to guide usage.
The final component, preference-guided query steering, dynamically shifts user queries toward preference directions in embedding space, enhancing retrieval relevance. This combined approach results in an unprecedented 2404× reduction in memory usage, while boosting preference-following accuracy by nearly 19 percentage points. Experiments across four benchmarks and three hardware platforms demonstrate the method's practicality, with memory footprints under 1MB and response times as low as 5 milliseconds.
Overall, EPIC offers a scalable, privacy-preserving solution for personalized AI on edge devices. It addresses the core bottleneck of what data to store, rather than how to retrieve, enabling smarter, faster, and more private AI assistants. Future work will focus on dynamic preference learning and multi-modal integration, further broadening its applicability and robustness.
Deep Analysis
Background
随着大型语言模型（LLMs）在自然语言处理中的突破，个性化AI助手逐渐成为研究焦点。早期工作如RAG（Lewis et al., 2020）通过外部知识库增强生成能力，但多依赖静态数据集，难以满足动态偏好需求。后续如EMG-RAG（Wang et al., 2024）和PEARL（Mysore et al., 2024）尝试构建用户特定记忆图谱或内容选择，但存储成本高，难以在设备端实现。传统索引如BM25和DPR（Karpukhin et al., 2020）表现优异，但在存储受限环境难以扩展。近年来，偏好条件化检索（如Pref-QR、PBR）逐步兴起，强调在检索前融入用户偏好信息，但仍面临存储空间和偏好偏移的挑战。总体而言，现有方法多关注检索机制优化，缺乏高效偏好导向的索引策略，限制了个性化的普及。
Core Problem
在设备端实现高效、偏好对齐的个性化检索，面临存储空间极度有限和偏好动态变化的双重难题。传统索引存储大量无关信息，浪费宝贵空间，且偏好偏移导致检索结果偏离用户期望。如何在保证偏好一致性的同时，极大压缩存储空间，成为核心瓶颈。此外，偏好信息的隐式表达和动态更新增加了系统设计难度。解决这些问题关系到个性化AI的实用性和用户体验。
Innovation
EPIC提出多阶段偏好对齐索引策略：
- �� 语义空间筛选：利用对比学习编码器快速筛除偏好无关内容，提升筛选效率。
- �� 细粒度验证：借助语言模型严格验证偏好一致性，生成指令增强索引表达。
- �� 查询调节：偏好引导的查询向量偏移，提升偏好匹配度。
- �� 指令索引：将偏好指令与内容结合，形成紧凑、偏好导向的索引结构。
该架构在极低存储空间（<1MB）下实现偏好个性化，突破了传统索引的存储与偏好偏移瓶颈，兼具理论保证与工程实用性。
Methodology
- �� 输入：多源设备数据（静态知识、动态足迹、对话历史）和用户偏好集。
- �� 粗筛：将数据和偏好编码到共享语义空间，计算余弦相似度，筛除偏好无关内容。
- �� 细验证：用Transformer模型对筛选内容进行偏好一致性验证，生成指令。
- �� 指令存储：结合偏好指令与内容，形成偏好导向的索引结构。
- �� 查询调节：用户查询编码后，偏好向量偏移，增强偏好匹配。
- �� 检索：利用FAISS索引快速检索偏好相关指令，导向内容生成。
Experiments
在PrefWiki、PrefRQ、PrefELI5和PrefEval四个偏好评估基准上，比较EPIC与传统索引和偏好条件化检索方法。指标包括存储空间、检索延迟和偏好跟随准确率。硬件平台涵盖Jetson Nano、Mac M4和Galaxy Z Flip 6。通过消融实验验证粗筛、验证和偏好调节的贡献，确保偏好对齐和存储压缩效果。
Results
EPIC在偏好跟随准确率方面提升18.79%，存储空间压缩达2404倍，检索延迟降低32.17倍。在三平台测试中，内存控制在1MB以内，响应时间在5.21-29.35毫秒。消融分析显示，语义筛选贡献最大，验证确保偏好匹配，调节提升召回。整体验证了EPIC在极端资源限制下的优越表现。
Applications
EPIC适用于移动端智能助手、隐私敏感场景和物联网设备，能在极少存储空间内实现高质量个性化响应。未来结合偏好学习和多模态信息，将推动个性化AI在更广泛场景的应用，改善用户体验。
Limitations & Outlook
依赖预定义偏好集，难以应对偏好动态变化；在噪声或模糊偏好场景中效果受限；验证和指令生成对硬件资源要求较高，需优化算法。未来将探索偏好在线学习和多模态建模，提升系统适应性。
Abstract
With the rapid emergence of personal AI agents based on Large Language Models (LLMs), implementing them on-device has become essential for privacy and responsiveness. To handle the inherently personal and context-dependent nature of real-world requests, such agents must ground their generation in device-resident personal context. However, under tight memory budgets, the core bottleneck is what to store so that retrieval remains aligned with the user. We propose EPIC (Efficient Preference-aligned Index Construction), which focuses on user preferences as a compact and stable form of personal context and integrates them throughout the RAG pipeline. EPIC selectively retains preference-relevant information from raw data and aligns retrieval toward preference-aligned contexts. Across four benchmarks covering conversations, debates, explanations, and recommendations, EPIC reduces indexing memory by 2,404 times, improves preference-following accuracy by 18.79 %p, and achieves 32.17 times lower retrieval latency over the best-performing baseline. In on-device experiments, EPIC maintains under 1 MB memory and achieves 5.21 to 29.35 ms/query latency across three platforms, while supporting streaming updates under preference drift. Our code and data are available at https://github.com/UbiquitousAILab/EPIC.