QUEST: A Query and Extraction System for Topics in Asylum Law Application Decisions
QUEST combines synthetic query generation, topic extraction, and LLM-based relevance assessment to identify credibility indicators in asylum appeal documents, achieving MAP@100 of 0.572 and 0.589 on datasets.
Key Findings
Methodology
QUEST integrates a multi-stage pipeline: first, documents are segmented into chunks; second, synthetic queries are generated using few-shot prompts based on a credibility codebook; third, BERTopic embeddings are used for clustering into topics; fourth, neural retrievers (BM25, E5, Qwen3) retrieve relevant chunks; fifth, monoT5 re-ranks results; finally, an LLM judge assesses relevance and credibility indicators, including a novel domain-specific crels metric. This combination enhances detection of credibility factors in lengthy legal texts.
Key Results
- On PUB and PRIV datasets, MAP@100 reached 0.572 and 0.589, NDCG@10 achieved 0.757 and 0.768, outperforming baseline BM25 and neural models. Detecting credibility indicators proved more challenging, with crels scores lower than traditional qrels, reflecting task complexity. Neural re-ranking improved NDCG slightly, validating multi-model fusion. The low Cohen’s κ (~0.02-0.04) between LLM and human labels indicates room for improvement in automatic credibility assessment.
- Different relevance types (qrels, crels, label_rels) significantly affect retrieval performance, with credibility-specific assessments being more difficult. The results highlight the importance of domain-specific evaluation metrics and multi-stage retrieval strategies.
Significance
This work advances legal NLP by enabling automated extraction of credibility indicators from complex asylum documents, addressing limitations of manual annotation and increasing efficiency. It demonstrates how combining deep learning, topic modeling, and large language models can improve legal decision support systems, potentially transforming judicial workflows. The introduction of credibility-specific relevance metrics offers a new perspective for evaluating retrieval systems in sensitive domains, paving the way for broader applications in legal and governmental settings.
Technical Contribution
The paper introduces QUEST, a comprehensive framework integrating synthetic query generation, neural topic clustering, multi-model retrieval, and large language model-based relevance judgment. The novel domain-specific crels metric distinguishes credibility assessment from general relevance, enriching evaluation methods. The multi-stage retrieval and re-ranking pipeline effectively handles long, semi-structured legal texts, setting a new standard for legal NLP applications involving sensitive, lengthy documents.
Novelty
This is the first study to incorporate a domain-specific relevance metric (crels) explicitly targeting credibility indicators in asylum appeals. It innovatively combines synthetic query generation with neural topic clustering and multi-model retrieval, addressing the challenge of long, complex legal texts. The approach offers a new paradigm for automatic credibility detection, distinct from traditional relevance assessment, with potential for broad impact in legal NLP.
Limitations
- The system's performance on extreme credibility cases remains limited, especially when subtle cues are involved, indicating a need for more nuanced models. The reliance on sensitive data restricts dataset size and diversity, limiting generalization. Handling multi-modal data (images, social media) is not yet integrated, which is crucial for comprehensive case analysis.
Future Work
Future research will focus on integrating multi-modal data, enhancing model robustness for subtle credibility cues, and expanding datasets across jurisdictions. Improving LLM judgment accuracy and reducing biases are priorities. Additionally, deploying the system in real-world judicial environments and conducting user studies will be essential to validate practical utility and refine the framework for broader legal applications.
AI Executive Summary
The surge in asylum applications driven by geopolitical upheavals has placed enormous strain on judicial systems worldwide. Traditional manual review processes are slow, resource-intensive, and prone to human error, especially given the length and complexity of asylum case documents. To address this challenge, the QUEST system was developed, aiming to automate the extraction of credibility indicators from lengthy, heterogeneous legal texts.
QUEST employs a sophisticated pipeline: documents are first segmented into manageable chunks; synthetic queries are generated using few-shot prompts based on a legal credibility codebook; these queries are embedded and clustered into meaningful topics via BERTopic; relevant chunks are retrieved through multiple models including BM25, E5, and Qwen3; then, a neural re-ranker (monoT5) refines the results; finally, an LLM judge assesses the relevance and credibility of each chunk, including a novel credibility relevance score (crels). This multi-layered approach effectively handles the length and unstructured nature of legal documents.
Experimental results on Danish datasets demonstrate that QUEST significantly outperforms baseline retrieval methods, achieving MAP@100 scores of 0.572 and 0.589, with NDCG@10 scores exceeding 0.75. The system's ability to detect credibility indicators, although more challenging, provides valuable insights into the complexity of legal reasoning. The introduction of domain-specific crels metrics highlights the task's difficulty but also guides future improvements.
Overall, this research marks a step forward in legal NLP, offering a scalable, automated solution for judicial decision support. It opens pathways for broader adoption in legal systems, enhancing transparency, efficiency, and fairness. Nonetheless, challenges remain in improving the sensitivity of models to subtle cues and expanding datasets for better generalization. Future work will focus on multi-modal data integration, cross-jurisdictional adaptation, and deploying the system in real-world judicial workflows, ultimately aiming to transform legal decision-making through AI-driven tools.
Deep Analysis
Background
Legal NLP在司法领域的应用不断深化，早期主要依赖关键词匹配和规则系统，随着深度学习模型如BERT、GPT的出现，文本理解能力大幅提升。尤其在庇护案件中，长文本、证据多样、信息敏感，传统方法难以满足自动化需求。近年来，结合信息检索、主题模型和大模型的系统逐渐成为研究热点。Bay-Jørgensen等利用LLMs进行可信度标注，Høgenhaug等探索深度模型在案件分类中的应用，但仍面临长文本处理和多源信息融合的挑战。随着大模型的发展，结合检索与推理的系统成为未来趋势。
Core Problem
庇护案件的核心难题在于如何自动识别影响裁决的可信度因素。长文本的复杂性、数据敏感性和多样性使得传统人工标注成本高、效率低。现有自动化方法多依赖关键词或浅层语义，难以捕捉深层次的可信度信号。如何在保护隐私的同时，提升模型对长文本中微妙信号的敏感度，成为亟待突破的瓶颈。这关系到司法公正、效率和透明度的提升。
Innovation
本研究的创新点包括：1）提出QUEST系统，结合合成查询、主题提取、多模型检索和大模型判定，系统性解决长文本中可信度指标的自动识别问题；2）引入domain-specific的crels指标，区别于传统relevance，强调对可信度的专门检测；3）采用多阶段检索与重排序策略，有效应对长文本的复杂性；4）设计保护敏感数据的机制，确保隐私安全。该系统在法律文本理解和信息检索领域具有突破性意义，为自动化司法提供新思路。
Methodology
- �� 文档分块：将庇护案件长文本拆分成句子或段落块，确保信息的可管理性。
- �� 查询生成：利用少样本提示，从法律代码书中生成针对每块内容的合成查询，确保查询的专业性和相关性。
- �� 主题提取：用BERTopic嵌入生成、降维和聚类，得到代表不同信息主题的簇。
- �� 主题命名：用Gemma3为每个簇赋予自然语言标签，便于理解和检索。
- �� 检索：结合BM25、E5、Qwen3模型，检索与主题相关的文本块，覆盖不同模型的优势。
- �� 重排序：用monoT5模型对检索结果进行优化，提高相关性和可信度的检测效果。
- �� 相关性评估：用LLM判定块与主题的匹配度，区分普通相关性（qrels）和可信度相关性（crels），后者专注于可信度指标。
- �� 评价：计算MAP、NDCG、MRR指标，分析不同模型和指标的表现差异，验证系统有效性。
Experiments
采用PUB（公开摘要）和PRIV（敏感完整材料）两个数据集，分别进行多模型检索、重排序和评估。模型包括BM25、SPLADE、E5、Qwen3，重排序采用monoT5。指标包括MAP@100、NDCG@10和MRR@10。通过不同的相关性标签（qrels、crels、label_rels）评估任务难度。模型参数经过调优，采用交叉验证确保结果稳健。还邀请法律专家对LLM判定的相关性进行验证，确保自动评估的可靠性。
Results
在PUB和PRIV数据集上，QUEST系统的MAP@100分别达0.572和0.589，NDCG@10达0.757和0.768，优于传统BM25和单一神经模型。credibility指标检索难度更高，crels得分低于qrels，反映任务复杂性。神经重排序略微提升了NDCG，验证多模型融合的有效性。LLM判定与专家标注的κ值较低，显示自动化仍需优化。整体结果表明，结合多模型和domain-specific指标能有效提升可信度指标的识别能力。
Applications
该系统可应用于司法部门的案件筛查，自动提取案件中的可信度信息，减轻人工负担，提高效率。未来可结合多模态信息（如图片、视频、社交媒体）实现全流程自动化，推广至合同审查、证据分析等法律场景，提升法律服务智能化水平。
Limitations & Outlook
模型在极端可信度样本表现不足，尤其在微妙信号识别方面仍有限。数据集受限于敏感性，影响模型泛化能力。长文本处理和多模态信息融合仍有待突破，未来需结合多模态学习技术。同时，模型对偏见和误判敏感，需持续优化和验证。
Plain Language Accessible to non-experts
想象你在一家非常复杂的工厂工作，工厂里有很多不同的机器和流程，每天都要处理大量的原料和产品。你需要确保每个产品都符合质量标准，但这些产品来自不同供应商，信息又很长、很复杂。以前，你得花很多时间逐个检查每个产品，找出那些可能有问题的部分。现在，有了QUEST，就像是给你配备了一台超级智能的机器人助手，它可以快速扫描所有的产品信息，自动找出那些可能不合格或不可信的地方。它用一种特别聪明的“搜索引擎”技术，结合了很多不同的工具，帮你筛选出最关键的内容，让你不用逐字逐句检查，也能保证产品的质量。这个助手还能用一种“判断力”，告诉你哪些问题最值得关注，节省了大量时间和精力。这样一来，工厂的效率大大提高，产品质量也更有保障。这个系统就像是工厂里的超级助手，帮你在海量信息中找到最重要的线索，让工作变得更快更准。
ELI14 Explained like you're 14
想象你在学校里，有很多学生的作业要批改，但每份作业都很长，而且内容很复杂。以前，你得花很多时间一字一句地看，才能知道哪个学生说得对，哪个地方不可信。现在，有了QUEST，就像是你有一个超级聪明的机器人助手，它可以帮你快速找到作业里最重要的部分，比如学生是否说了谎或者提供了不真实的内容。这个机器人用一种特别厉害的技术，能理解长长的作业，把里面的关键信息提取出来，然后告诉你哪些内容可能不可信，哪些是可信的。它还会用一种“判断力”，帮你决定哪些问题最需要注意。这样，你就不用一字一句地看作业，也能很快判断出学生的答案是真是假。这个助手让你的工作变得简单多了，也让你更快地完成任务，学习也更轻松。就像有个聪明的朋友帮你筛选信息，让你不用担心漏掉重要的线索。
Abstract
Legal decisions on asylum applications consist of long, complex, and heterogeneous documents, covering narrative applicant interviews, original decisions, and additional supporting materials. If an application is rejected, a critical question in processing an appeal is whether the credibility of the information in the original application was a factor that determined the original decision. In this paper, we present the QUEST system (Query and Extraction System for Topics) to extract and identify factors relating to credibility assessments in two datasets of Danish asylum application appeals. QUEST frames this problem as an information retrieval task, combining synthetic query generation, topic extraction, and relevance assessment to identify information related to credibility indicators in appeals board application materials. In addition to standard retrieval evaluation metrics, we propose a new type of domain-specific assessments distinct from the traditional relevance to evaluate the performance of the tested systems with respect to credibility factors. In this way, we obtain insights about how well automatic methods can return answers for different types of indicators appearing in asylum appeals. Our results indicate that there is an increased challenge when estimating performance using credibility-based relevance assessments, thus pointing to the difficulty of the task.