Advancing the State-of-the-Art in Empirical Privacy Auditing

The rapid development of large language models (LLMs) has revolutionized NLP, enabling applications across industries. However, their tendency to memorize training data raises privacy concerns, especially when models are fine-tuned on sensitive information. Prior works like differential privacy (DP) and membership inference attacks have highlighted vulnerabilities, but practical detection remains challenging. Existing privacy audits often depend on real data or fixed templates, which lack flexibility and sensitivity. As models grow larger, the risk of unintended memorization increases, necessitating more sophisticated, automated auditing tools. Recent studies (e.g., Meeus et al., 2025; Nasr et al., 2023) have explored synthetic data and canary-based methods, but these often rely on manual design or limited generative capacity. The need for scalable, effective, and naturalistic privacy assessment methods remains urgent, especially for deployment in sensitive domains like healthcare, finance, and legal sectors.

The core challenge is designing impact-driven, naturalistic canary samples that can reliably reveal memorization without compromising privacy. Traditional methods use real or templated data, which are either too obvious or unnatural, limiting detection sensitivity. Moreover, as model sizes increase, their capacity to memorize complex data complicates privacy assessment. Existing approaches struggle to balance naturalness, influence, and detectability of canaries, leading to underestimation of leakage risks. Additionally, there is a lack of systematic, automated tools to evaluate privacy leakage from synthetic data, especially when models generate diffuse, distribution-shifting signals rather than verbatim memorization. Addressing these issues requires leveraging generative models' capabilities to produce impactful, naturalistic samples that can be used for scalable privacy auditing.

The key innovations are: 1) Using high-temperature sampling from pretrained models to generate synthetic canaries with high influence and naturalness, overcoming limitations of fixed templates. 2) Employing an auxiliary attack model trained on synthetic data to detect subtle, diffuse leakage signals, bridging data and model auditing. 3) Analyzing the interaction between model capacity (LoRA rank) and canary entropy, revealing that larger models memorize more complex, high-entropy samples but with lower leakage for reconstruction. These innovations enable automated, sensitive detection of memorization, addressing the limitations of prior template-based or similarity-based methods, and providing a scalable framework for privacy risk assessment.

• �� Generate synthetic canaries by prompting a pretrained, instruction-tuned language model at high temperature (T≥0.8), ensuring samples are impactful yet natural. • Use tailored prompts to produce samples that are somewhat in-distribution, balancing naturalness and influence. • Insert generated canaries into the training data during the same training run to simulate realistic memorization scenarios. • Fine-tune the target model with inserted canaries using LoRA or full weights, with hyperparameters such as batch size 64, 7275 steps, and temperature tuning. • Train an auxiliary attack model on synthetic data, fine-tuning it to detect distributional shifts caused by canaries. • Conduct membership inference and reconstruction attacks, measuring μ-GDP and edit distances, to quantify leakage. • Analyze the effects of model capacity (LoRA rank) and canary entropy on memorization and leakage, performing ablation studies to validate findings.

Experiments utilized the Gemma3 12B model fine-tuned on Enron emails, with LoRA rank 64, trained over 7275 steps, inserting canaries with 10 repetitions. Multiple canary types (random, templated, bigram) were tested across sampling temperatures (0.8-3.0). The evaluation involved measuring μ-GDP scores for membership inference, and edit distances for reconstruction. Synthetic data was generated from the fine-tuned model, and an auxiliary attack model was trained on this data to detect leakage. The experiments systematically varied model capacity and canary entropy, performing ablation studies to assess their impact on memorization. Results demonstrated that high-temperature sampling produces more influential canaries, significantly improving attack sensitivity, with the highest μ-GDP reaching 7.17, indicating strong leakage detection capability.

The high-temperature generated canaries in LoRA fine-tuning models achieved μ-GDP scores up to 7.17, indicating high memorization and leakage potential, far exceeding baseline methods. Synthetic data leakage detection via auxiliary models improved sensitivity by over 30%. Larger models (higher LoRA rank) memorized more complex, high-entropy canaries but showed lower leakage for reconstruction attacks. Repetition of canaries increased memorization depth but also privacy risk. Varying sampling temperature revealed that higher T enhances impact, validating the effectiveness of high-temperature sampling for impactful canary generation. The systematic analysis confirmed the interaction between model capacity and canary entropy, providing insights into designing better privacy safeguards.

This approach is directly applicable to privacy-sensitive NLP tasks such as medical record processing, financial data analysis, and legal document handling. It enables organizations to systematically evaluate and mitigate memorization risks before deploying models. The synthetic canary generation and auxiliary detection tools can be integrated into model development pipelines, providing continuous privacy monitoring. Moreover, the methodology supports regulatory compliance by quantifying leakage risks, facilitating safer AI deployment in sensitive sectors. Long-term, this framework can guide the design of privacy-aware training regimes, balancing model capacity and memorization to ensure data confidentiality while maintaining utility.

The method relies on the generative quality of pretrained models, which may be limited in domain-specific or low-resource settings. High-temperature sampling can produce unnatural or less realistic samples, affecting the representativeness of audits. Auxiliary attack models require substantial computational resources for training, especially on large synthetic datasets. Additionally, the approach may not fully capture all forms of leakage, such as subtle distributional shifts or adversarial attacks. Future work should focus on improving sample diversity, reducing computational costs, and extending to multimodal data to enhance robustness and applicability.