Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

The field of artificial intelligence (AI) governance is rapidly evolving, with governments and regulatory bodies continuously introducing new laws, guidelines, and standards. These policy documents are often lengthy, legally dense, and distributed across multiple jurisdictions, making analysis and comparison challenging. Resources such as the AI Governance and Regulatory Archive (AGORA) provide structured collections of AI policy documents, but extracting insights from these materials still requires substantial manual effort. Automated question-answering systems could help researchers and policymakers navigate this growing body of regulation.

Large language models (LLMs) offer powerful tools for analyzing complex text, but they often struggle with legal and regulatory documents due to domain-specific terminology, conceptual ambiguity, and nested references. Moreover, when applied directly to policy corpora, LLMs may generate fluent but unsupported claims. Retrieval-augmented generation (RAG) addresses this limitation by grounding responses in retrieved documents, yet the effectiveness of RAG depends heavily on both retrieval quality and generation alignment.

Despite recent advances in retriever training and preference-based alignment, it remains unclear whether improvements to individual RAG components consistently translate into better end-to-end question answering performance, particularly in complex and high-stakes domains. Domains like AI governance corpora present particularly serious challenges, given dense legal language, sometimes ambiguous policy and technical jargon, as well as evolving and cross-referenced regulatory coverage across sectors and jurisdictions.

In the field of AI governance, policy documents are often lengthy, legally dense, and distributed across multiple jurisdictions, making analysis and comparison challenging. Although resources like AGORA provide structured collections of AI policy documents, extracting insights from these materials still requires substantial manual effort. Automated question-answering systems could help researchers and policymakers navigate this growing body of regulation. However, existing QA systems often struggle with legal and regulatory documents due to domain-specific terminology, conceptual ambiguity, and nested references. Moreover, when applied directly to policy corpora, LLMs may generate fluent but unsupported claims.

The core innovations of this paper include proposing a RAG pipeline that combines contrastive learning and preference alignment for policy analysis tasks. Specifically:

1) The retriever is fine-tuned using contrastive learning, while the generator is aligned using pairwise preference data. This method better adapts to the specific needs of the policy domain.

2) Analyzing how improvements in retrieval metrics can lead to more confident hallucinations, particularly when relevant documents are missing.

3) Providing practical suggestions for designing QA systems over dynamic regulatory corpora, helping researchers and developers better address the complexity of policy texts.

The paper proposes a RAG pipeline that combines contrastive learning and preference alignment for policy analysis tasks. The specific steps are as follows:

• �� Use a ColBERT-based retriever for retrieval. The retriever is fine-tuned using contrastive learning, while the generator is aligned using pairwise preference data.

• �� Conduct experiments using the AGORA corpus to evaluate retrieval performance, answer relevance, and response faithfulness.

• �� In some cases, when relevant documents are missing from the corpus, stronger retrieval leads to more confident hallucinations.

• �� Provide practical suggestions for designing QA systems over dynamic regulatory corpora, helping researchers and developers better address the complexity of policy texts.

The experimental design includes evaluating retrieval-augmented generation using the AGORA corpus. Specifically, the retriever is fine-tuned using contrastive learning, while the generator is aligned using pairwise preference data. Experiments evaluate retrieval performance, answer relevance, and response faithfulness.

Various metrics are used to evaluate retrieval performance, including Mean Reciprocal Rank (MRR), Recall@k, and MAP@k. The generator alignment is performed using Direct Preference Optimization (DPO) with pairwise preference data.

Results show that while retriever fine-tuning improves retrieval metrics, it does not consistently enhance end-to-end QA performance. In some cases, stronger retrieval leads to more confident hallucinations when relevant documents are missing.

Results show that while retriever fine-tuning improves retrieval metrics, it does not consistently enhance end-to-end QA performance. In some cases, stronger retrieval leads to more confident hallucinations when relevant documents are missing.

Specifically, fine-tuning the retriever improved retrieval metrics, such as Mean Reciprocal Rank (MRR) reaching 0.748, but did not significantly enhance QA performance. The GPT-5.4 baseline model achieved significantly higher answer accuracy without web search compared to the RAG system.

These results suggest that improvements to individual components do not necessarily translate into more reliable answers. This is significant for researchers and developers designing grounded QA systems over dynamic regulatory corpora.

The proposed method can be applied to various policy analysis scenarios, helping researchers and policymakers better navigate complex regulatory systems. Specifically:

1) Automated QA systems can help researchers and policymakers navigate the growing body of regulation, reducing the workload of manual analysis.

2) In the field of AI governance, the system can be used to analyze and compare policy documents across different jurisdictions, providing more comprehensive policy insights.

3) The system can also be applied to policy analysis in other fields, such as healthcare and finance, helping researchers and policymakers better understand and address complex policy texts.

Despite the progress made by the proposed method, there are still some limitations:

1) The system may generate confident but inaccurate answers when relevant documents are missing from the corpus.

2) The generator may incorrectly reference unrelated documents when handling cross-jurisdictional policies.

3) The collection of preference data is limited by the availability of domain experts, potentially not fully capturing the expectations of policy researchers.

Future research could explore stronger hallucination mitigation strategies, cross-document contextual grounding, and improved handling of document status changes.