P2Skill: Privacy Preserving Skill Distillation for Cloud-Local LLM Inference Systems
P2Skill employs prompt-based skill distillation to enhance privacy preservation, achieving 1.69× higher inference quality without fine-tuning.
Key Findings
Methodology
P2Skill utilizes a prompt-driven framework with four core skills: decomposition, PII-aware routing, paraphrasing, and reconstruction. Guided by cloud LLM iterations, it enables a local small language model (SLM) to autonomously split tasks, detect and avoid PII, rewrite sensitive content, and synthesize responses. The approach relies solely on prompt design and rule-based methods, avoiding model fine-tuning or auxiliary detectors. Cloud LLMs provide reference responses, evaluate local outputs, and iteratively refine skill prompts, ensuring generalization across domains. This closed-loop process enhances privacy protection while maintaining inference quality.
Key Results
- On a four-domain benchmark, P2Skill outperformed LDP baselines, reducing PII leakage to zero in sensitive domains like medical and banking, while increasing average inference quality scores to 8.07/10. It achieved 1.69× and 3.66× improvements in privacy-preserved inference quality over previous methods, demonstrating robust privacy guarantees and high task performance across models including Gemma, Qwen, and LLaMA.
Significance
This work addresses the longstanding challenge of balancing privacy and utility in cloud-based LLM inference. By introducing a prompt-based skill distillation framework guided by cloud LLMs, it circumvents the need for costly fine-tuning or complex cryptographic protocols. The approach offers a scalable, domain-agnostic solution that can be deployed on local devices, significantly reducing privacy risks while leveraging powerful cloud models. Its generalizable design paves the way for safer, more private AI applications in sensitive fields like healthcare, finance, and legal services.
Technical Contribution
The primary technical innovation lies in leveraging cloud LLMs to iteratively guide the development of four prompt-based skills—decomposition, routing, paraphrasing, and reconstruction—without modifying the local model weights. This method combines deterministic rule-based PII detection with prompt-driven content rewriting, ensuring privacy without semantic distortion. The closed-loop refinement process, based on failure analysis and prompt rewriting, enhances the robustness and adaptability of the skills. This paradigm shifts from traditional fine-tuning or auxiliary detector training to a flexible, prompt-centric approach that can be easily transferred across models and domains.
Novelty
This research is the first to implement a prompt-only, iterative skill distillation framework guided by cloud LLMs for privacy-preserving inference. Unlike prior methods relying on perturbation or trained detectors, P2Skill employs a rule-based, prompt-driven approach that generalizes across domains without retraining. Its iterative refinement mechanism ensures continuous improvement of skills based on real feedback, setting a new standard for privacy-aware AI systems.
Limitations
- PII detection still depends on rule-based prompts, which may miss complex or novel identifiers, risking residual leakage. Improving detection accuracy remains critical.
- The iterative process relies on cloud API calls, which could incur latency and cost, limiting real-time deployment in some scenarios.
- The approach assumes a stable cloud LLM for guidance; model drift or API changes could affect performance. Further robustness and automation are needed.
Future Work
Future directions include integrating self-supervised detection methods to enhance PII identification, reducing cloud dependency by optimizing local models, and expanding the framework to handle more complex, multi-turn dialogues. Additionally, exploring adaptive prompt design and domain-specific skill transfer could further improve robustness and scalability.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized natural language understanding, reasoning, and generation, leading to widespread deployment in personal assistants, enterprise automation, and decision support systems. However, leveraging cloud-based inference raises significant privacy concerns, especially when prompts contain sensitive user information such as medical, financial, or personal identifiers. Traditional privacy-preserving techniques—cryptographic inference, entity masking, and differential privacy—either impose high computational costs or degrade semantic quality, limiting their practical utility.
In response, this paper introduces P2Skill, an innovative framework that employs prompt-based skill distillation guided by cloud LLMs to protect user privacy without model fine-tuning. The core idea involves designing four reusable skills—decomposition, PII-aware routing, paraphrasing, and response reconstruction—that enable a local small language model (SLM) to autonomously process user prompts. The cloud LLM iteratively refines these skills through a closed-loop process, based on response evaluation and failure analysis, without requiring retraining or auxiliary detectors. This approach leverages deterministic rule-based PII detection and content rewriting, ensuring that only PII-free requests are transmitted to the cloud.
Extensive experiments across four domains—medical, banking, tourism, and general knowledge—demonstrate that P2Skill significantly outperforms traditional perturbation-based methods. It reduces PII leakage to zero in sensitive domains while maintaining high inference quality scores (average 8.07/10), achieving 1.69× and 3.66× improvements over baseline methods. The system's modular, prompt-only design ensures broad applicability and easy transferability across models and domains. These results highlight P2Skill's potential to enable privacy-preserving, high-performance cloud-local inference, addressing critical challenges in deploying AI responsibly in privacy-sensitive contexts.
Despite its strengths, the approach faces limitations such as reliance on rule-based PII detection, potential API costs, and dependency on cloud LLM stability. Future work aims to incorporate self-supervised detection, reduce cloud reliance, and enhance robustness, paving the way for safer, more scalable AI systems that respect user privacy while delivering powerful inference capabilities.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做饭，但你不想让别人知道你用了什么秘密调料。于是，你用一种特别的技巧，把这些秘密调料变成普通的调料，让别人看不出原料，但菜还是很好吃。这就像P2Skill用提示技巧，把用户的敏感信息变得看不出来，但系统还能完成任务。它不用重新学习所有厨艺，只用一些聪明的提示和规则，就能保护你的秘密，又能帮你做出好菜。这让厨房既安全又高效，适合各种菜系。这个方法就像魔法一样，既保护隐私，又能完成任务。
ELI14 Explained like you're 14
想象你在学校写作文，但你不想让老师知道你家里的秘密，比如住址或电话号码。于是，你用一种聪明的办法，把这些秘密变成普通的话，比如把“我住在北京”改成“我住在一个大城市”。这样，老师还是能帮你改作文，但不会知道你的秘密。P2Skill就像这个办法，它用提示和规则，把敏感信息变得不明显，确保你的隐私不被泄露，又能得到帮助。它不用重新学习，只用一些聪明的提示，就能保护你的秘密，又帮你完成任务。这就像用魔法一样，既安全又方便！
Abstract
Cloud-local LLM inference systems have the potential to use the reasoning capability of large cloud models while protecting sensitive user data on personal devices. Cloud-bound requests must exclude personally identifiable information (PII) to prevent external data leakage. Existing privacy-preserving methods rely on prompt perturbation, entity masking, or model fine-tuning, but these approaches may distort contextual semantics or require additional training. This paper proposes P2Skill, a prompt-based skill distillation method in which a local small language model (SLM) autonomously performs decomposition, PII-aware routing, paraphrasing, and reconstruction by following the skill prompts. Skills are iteratively refined from execution failures by a cloud LLM, enabling the local SLM to generalize beyond memorized PII patterns, and therefore P2Skill requires no privacy-specific fine-tuning or learned auxiliary detectors. Evaluation on a four-domain benchmark shows that P2Skill achieves $1.69\times$ and $3.66\times$ higher privacy-preserved inference quality than previous baselines.