Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?

The evolution of LLMs like GPT-3, LLaMA, and ChatGPT has revolutionized retrieval and generation tasks, leading to the rise of RAG systems that combine knowledge retrieval with language generation. Early methods such as DPR and REALM focused on dense retrieval, while later approaches integrated large generative models for more fluent responses. Nugget-based evaluation emerged as a key metric, emphasizing key facts for system validation. However, with the increasing reliance on LLMs as evaluators, concerns about bias, circularity, and information leaks have grown. Recent studies highlight the vulnerability of LLM judges to prompt sensitivity, adversarial attacks, and data leaks, but systematic analysis of leak-induced score inflation remains limited. This gap motivates the current investigation into evaluation security.

The core issue addressed is that leaked evaluation prompts or gold nuggets can be exploited by system developers to artificially boost scores, leading to misleading assessments of system progress. Such leaks undermine the validity of Nugget evaluation metrics, which are designed to reflect factual correctness and relevance. The problem is compounded by the open nature of prompts and nuggets, often shared publicly, increasing the risk of intentional or unintentional leaks. This jeopardizes the trustworthiness of automated evaluation frameworks, especially as they become central to system development pipelines. The challenge lies in designing evaluation protocols resilient to leaks, ensuring genuine performance measurement.

This work introduces Crucible, a system that systematically simulates leak scenarios by manipulating prompts and nuggets, revealing their impact on evaluation scores. It emphasizes the importance of blind evaluation and methodological diversity, proposing strategies like multiple prompt sets and hidden nugget banks. The key innovation is the detailed leak simulation framework, which can test the robustness of current evaluation methods. Unlike traditional approaches that rely solely on manual or static metrics, this study provides a dynamic, adversarial testing environment, pushing the boundaries of evaluation security research. It also offers practical guidelines for safeguarding evaluation elements in real-world deployments.

• �� Constructed Crucible, integrating retrieval, sentence extraction, summarization, and report assembly modules. • Used TREC NeuCLIR dataset with PLAID-X retriever for document collection. • Designed prompt templates for nugget question-answer pairs, simulating leak scenarios. • Manipulated prompts to test their influence on evaluation scores, including leaked and non-leaked versions. • Simulated perfect nugget prediction to assess maximum score inflation potential. • Employed multiple evaluation metrics (Nugget Recall, Density, Citation Support) to quantify impacts. • Implemented blind evaluation strategies, such as multiple prompt sets and hidden nugget banks, to test robustness.

Experiments utilized the TREC NeuCLIR 2024 dataset, comprising manually curated nuggets and system reports. Baselines included Ginger, GptResearcher, and BulletPoints. Evaluation metrics covered Nugget Recall, Nugget Density, relevant sentence count, and citation support. Leak scenarios involved prompt template leaks, gold nugget prediction, and model parameter exposure. Multiple report lengths (short, medium, long) were tested to analyze the impact of leaks. Statistical significance was assessed via paired t-tests. Results consistently showed that leaks caused substantial score inflation, with Nugget Recall reaching above 0.82 and Nugget Density increasing by over 30%. The experiments confirmed that leak mitigation strategies, such as blind evaluation, effectively reduce inflation.

Leak scenarios, especially prompt template leaks, led to near-perfect Nugget Recall scores (>0.82), significantly surpassing baseline performances. Simulating perfect gold nugget prediction caused score inflation exceeding 50%, exposing severe evaluation vulnerabilities. Diversified, blind evaluation methods mitigated score inflation, maintaining more accurate system assessments. The results demonstrate that leaking evaluation components can artificially inflate scores, misleading system development and benchmarking efforts. These findings underscore the necessity of secure evaluation protocols to preserve assessment integrity.

The insights from this study are critical for AI system developers, evaluators, and policymakers. Implementing secure, multi-faceted evaluation frameworks can prevent score manipulation, ensuring genuine system improvements. In industry, this enhances trustworthiness in AI deployment, especially in sensitive domains like healthcare, finance, and legal systems. The methodology can be adopted to audit existing evaluation pipelines, identify leak vulnerabilities, and develop cryptographic safeguards. Long-term, establishing standardized secure evaluation protocols will foster more reliable AI progress measurement, supporting trustworthy AI deployment across sectors.

The experiments are primarily based on the TREC NeuCLIR dataset, which may not fully capture all real-world leak scenarios. The leak simulations focus on prompts and nuggets, without exploring complex leaks like training data or model weights. The proposed mitigation strategies require further validation across diverse tasks and models. Additionally, the assumption of leak access may not reflect all practical situations, where leaks could be covert or harder to detect. Future work should address these gaps, developing more comprehensive leak detection and prevention mechanisms.