CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents
CAPED employs task-driven selective exposure, reducing incidental visual privacy leaks by over 70% while maintaining high task utility.
Key Findings
Methodology
CAPED integrates local task understanding, screen context reasoning, and UI element parsing, employing an element-level decision engine. It extracts structured task requirements, classifies screen context via Android signals, detects UI elements through OCR and detectors, verifies task relevance with VQA, and applies a decision table to selectively expose or mask content. The system operates before screenshot upload, ensuring sensitive information is filtered on-device. Experiments on Android environment with 28 seeded tasks show significant leakage reduction, maintaining high task utility. The architecture emphasizes transparency, auditability, and real-time decision-making.
Key Results
- In the seeded evaluation, CAPED reduced success-conditioned weighted seeded leakage from 0.766 to 0.268, effectively minimizing incidental private information exposure, while maintaining a task success rate of 92.9%.
- On AndroidWorld, the full system completed 55.2% of 116 tasks, compared to 66.4% without protection, demonstrating a trade-off between utility and privacy.
- Ablation studies confirmed that task-driven content filtering outperforms full masking and text anonymization, validating the core design principle.
Significance
This work advances privacy-preserving mobile automation by moving beyond static masking, enabling dynamic, context-aware content filtering. It addresses the critical challenge of incidental privacy leakage during visual perception tasks, facilitating safer deployment of GUI agents in sensitive environments. The approach balances privacy and utility, offering a scalable solution adaptable to diverse applications. Its emphasis on device-side filtering before data transmission aligns with privacy-by-design principles, setting a new standard for visual privacy in AI-enabled mobile systems.
Technical Contribution
The paper introduces a novel task-driven content filtering framework that combines structured task extraction, contextual scene classification, and element-level visual verification. It innovates with an on-device decision engine that dynamically determines exposure based on multi-modal signals, ensuring privacy without sacrificing operational utility. The system’s modular design allows for transparency, auditability, and extensibility, representing a significant departure from traditional blanket masking or anonymization techniques. Empirical results demonstrate its effectiveness in reducing privacy leakage while maintaining high task success rates.
Novelty
This research is the first to integrate task understanding, screen context, and UI element analysis into a unified, real-time content filtering system for mobile GUI agents. Unlike prior work that relies solely on static masking or text anonymization, CAPED dynamically assesses content relevance, enabling fine-grained, context-aware privacy control. Its device-side operation ensures privacy preservation before data leaves the trusted environment, representing a paradigm shift in visual privacy protection for mobile AI systems.
Limitations
- The current system relies on heuristic UI parsing and scene classification, which may fail in highly complex or ambiguous scenarios, leading to potential privacy leaks or unnecessary masking.
- Real-time performance in resource-constrained environments needs further optimization, especially for multi-modal verification processes like VQA.
- The approach assumes trusted device-side environment; it does not address risks from compromised OS or malicious apps, which remain open challenges.
Future Work
Future efforts will focus on deep learning-based UI understanding to improve accuracy and robustness, especially in complex multi-task scenarios. Integrating multi-modal data fusion, such as audio and sensor signals, can enhance context awareness. Additionally, extending the framework to cross-platform and multi-device ecosystems, along with user-customizable privacy policies, will be key directions. Long-term, the goal is to develop adaptive, personalized privacy controls that seamlessly balance utility and privacy in diverse real-world applications.
AI Executive Summary
The proliferation of mobile GUI agents has revolutionized how we automate tasks and interact with smartphones, but it also introduces significant privacy concerns. Traditional protective measures, such as full-screen masking or text anonymization, often compromise functionality or fail to prevent incidental exposure of sensitive visual information. For instance, during routine interactions, private contacts, photos, or health data can inadvertently be revealed to remote AI models, risking privacy breaches.
Addressing this challenge, CAPED (Context-Aware Privacy Exposure Defense) proposes a novel, task-driven content filtering framework. The system operates on-device, before screenshots are uploaded, ensuring sensitive information is filtered out proactively. It combines structured task understanding, screen context classification, and UI element analysis, employing an element-level decision engine that selectively exposes only task-relevant content. This approach maintains interface usability while significantly reducing incidental privacy leaks.
Experimental results on Android demonstrate the effectiveness of CAPED. In a controlled seeded evaluation involving 28 tasks, it reduced privacy leakage metrics from 0.766 to 0.268, with a task success rate of 92.9%. Broader testing on AndroidWorld with 116 tasks showed a practical utility trade-off, with 55.2% task completion. These findings validate the core principle: content exposure should be driven by task relevance and context, not blanket masking.
This work marks a significant step forward in visual privacy protection for mobile AI systems. By filtering sensitive content on-device and only exposing what is necessary, CAPED enhances user privacy without sacrificing operational utility. Its modular, transparent design paves the way for future enhancements, including deep learning-based UI understanding and multi-modal context integration. Ultimately, this approach can foster safer, more trustworthy AI-enabled mobile experiences, aligning privacy with the demands of modern digital life.
Deep Analysis
Background
随着移动端智能化的发展，GUI自动化技术逐渐成熟，早期方案多依赖规则匹配和静态遮蔽，难以应对复杂场景中的隐私风险。近年来，深度学习推动UI解析和任务理解技术突破，但隐私保护仍面临挑战。传统方法如文本匿名和全屏遮蔽，虽然能减少敏感信息泄露，但也带来功能损失和用户体验下降。多模态融合和上下文感知成为研究热点，但尚未形成成熟的解决方案。整体来看，移动GUI代理的隐私保护正从静态遮蔽向动态、任务感知方向演进，亟需更细粒度、更智能的技术。
Core Problem
核心问题在于如何在保证任务完成的同时，有效减少偶发的视觉隐私泄露。现有方案如全屏遮蔽和文本匿名，无法区分任务相关和无关的敏感信息，导致功能受限或隐私风险。尤其是在多模态、多任务环境中，内容筛选的实时性和准确性成为瓶颈。系统还需保证操作的透明性和可审计性，以增强用户信任。解决这一问题，要求深度理解UI结构和任务需求，设计高效、可扩展的内容筛选机制，兼顾隐私保护与用户体验。
Innovation
本研究的创新点在于提出任务驱动的选择性曝光原则，结合屏幕上下文和UI元素分析，实现内容的动态筛选。具体包括：• 任务理解模块，提取结构化需求；• 屏幕上下文分类，设定默认隐私策略；• UI元素检测，识别文本、图像、控制元素；• 任务相关性验证，利用VQA和语义匹配确认内容必要性；• 元素级别决策机制，根据内容和环境动态暴露或遮蔽内容。这一机制突破了静态遮蔽的限制，实现了内容的细粒度控制。
Methodology
- �� 任务理解：在设备端解析用户指令，提取{app, subject, type}结构化需求，确保隐私保护目标明确。
- �� 屏幕上下文分类：利用Android系统信号（如包名、活动名）判断场景隐私级别。
- �� UI元素解析：通过OCR和UI检测器，将截图划分为文本、图像和图标三类，识别内容。
- �� 任务相关性验证：对图像采用VQA判断是否包含任务主体，文本进行语义匹配。
- �� 内容筛选决策：结合任务需求、上下文和元素内容，采用决策表，决定暴露、遮蔽或临时解封。
- �� 遮蔽与恢复：对非必要内容遮蔽，必要时允许临时解封，操作可审计。
- �� 输出：生成筛选后截图和任务表示，确保隐私安全。
Experiments
在Android平台，使用28个 seeded 任务评估隐私泄露，比较无保护、文本匿名、全屏遮蔽和CAPED全保护策略。指标包括泄露指标（加权成功泄露率）和任务完成率。还在AndroidWorld平台测试116个任务，评估实用性和隐私保护效果。调优参数包括UI解析阈值、任务匹配策略和遮蔽强度。通过消融实验验证各模块贡献，分析在复杂场景中的表现。
Results
CAPED将泄露指标由0.766降至0.268，显著减少隐私泄露，任务成功率保持在92.9%。在AndroidWorld测试中，完成55.2%的任务，优于未保护状态的66.4%。消融实验显示，任务驱动筛选优于全屏遮蔽和文本匿名方案，验证其有效性。系统在多场景下表现良好，证明其实际应用潜力。
Applications
该技术适用于移动自动化、个人隐私保护、智能助手等场景。用户授权后，系统自动筛选必要信息，避免敏感内容泄露，提升隐私安全。企业可在客户服务、数据分析中部署，确保敏感信息不被无意中暴露。结合深度学习和多模态融合，未来可实现更智能、更细粒度的隐私控制，推动行业标准。
Limitations & Outlook
系统在复杂多变环境中，仍存在漏检敏感内容或误遮非敏感内容的问题，UI解析和任务理解的局限是主要原因。高复杂度场景下实时性和准确性需优化，特别是多模态验证。系统假设设备环境可信，未考虑操作系统或恶意软件的潜在风险。未来需增强鲁棒性和扩展性，提升系统适应性。
Abstract
Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results show that task-driven selective exposure can reduce incidental visual leakage before screenshots are released to a remote GUI agent.