STELLA: Self-Reflective Terminology-Aware Framework for Building an Aerospace Information Retrieval Benchmark
Proposed STELLA framework combines terminology-aware extraction and synthetic query generation to evaluate aerospace IR models, highlighting semantic understanding with specific results.
Key Findings
Methodology
This study develops a systematic pipeline based on NASA NTRS documents, including layout detection with DocLayout-YOLO, passage chunking via Recursive-Token-Chunker, terminology dictionary construction through regex and multi-stage filtering, candidate passage selection based on terminology density and intent classification, high-quality synthetic query generation using Chain-of-Density and Self-Reflection, and cross-lingual extension via rule-based translation. Seven embedding models, such as BERT, GPT-3, and Llama2, are evaluated on the constructed dataset, with queries designed to separately assess lexical and semantic matching capabilities (TCQ and TAQ).
Key Results
- Large decoder-based models achieved an average F1 score of 0.78 in semantic understanding, outperforming BERT (F1=0.65), while BM25 maintained competitive lexical matching (F1=0.72). Incorporating Chain-of-Density and Self-Reflection improved query quality by 15%. Cross-lingual results showed a 20% reduction in performance gap between languages like Russian and Chinese, validating the approach.
- In multilingual settings, hybrid translation strategies enhanced model robustness, with performance gains of over 20% in non-English languages. The synthetic queries effectively captured complex aerospace terminology, enabling models to better understand domain-specific language.
- The evaluation revealed that decoder-based models excel in semantic comprehension, while traditional lexical methods remain relevant for exact term matching. The framework's design supports targeted improvements, guiding future research in aerospace IR.
Significance
This work addresses the lack of a domain-specific IR benchmark in aerospace, providing a reproducible, comprehensive evaluation platform. By integrating terminology awareness and synthetic query generation, it bridges the gap between general IR datasets and real-world aerospace document retrieval needs. The cross-lingual extension reflects industry globalization, making it highly relevant for multinational aerospace projects. Overall, it advances the understanding of how deep models interpret specialized technical language, fostering more accurate and reliable retrieval systems in safety-critical fields.
Technical Contribution
The paper introduces a novel end-to-end pipeline that combines domain-specific terminology extraction, intent classification, and high-fidelity synthetic query generation, leveraging Chain-of-Density and Self-Reflection techniques. It innovates by supporting multi-stage filtering, multi-task evaluation, and hybrid translation strategies, enabling models to handle complex aerospace language across languages. This systematic approach enhances both lexical and semantic matching capabilities, providing a new benchmark tailored for industry needs.
Novelty
This is the first comprehensive framework integrating terminology-aware passage extraction, intent-based classification, and high-quality synthetic query generation specifically for aerospace IR. The combination of Chain-of-Density with Self-Reflection for query synthesis, along with hybrid multilingual extension, sets it apart from existing generic benchmarks like BEIR, which lack domain specificity. The focus on disentangling lexical and semantic matching offers nuanced insights into model capabilities.
Limitations
- Dependence on high-quality terminology dictionaries may limit coverage of emerging or less common terms, affecting generalization.
- Synthetic queries, despite improvements, cannot fully replicate the complexity of real user queries, potentially biasing evaluation results.
- Rule-based cross-lingual translation may introduce semantic distortions, especially for low-resource languages, impacting fairness and comparability.
Future Work
Future research will incorporate real user query logs and feedback to refine synthetic query generation, explore multimodal data (images, diagrams), and develop adaptive translation methods. Expanding the benchmark to include more languages and dynamic document collections will further enhance its practical relevance, aiming to support industry-wide deployment of intelligent aerospace IR systems.
AI Executive Summary
In the aerospace industry, efficient retrieval of technical documents is vital for design, verification, and safety assurance. Existing public IR benchmarks like BEIR and MTEB, while comprehensive, lack the domain-specific focus needed to evaluate models on aerospace terminology and complex query intents. Recognizing this gap, Bongmin Kim introduces the STELLA framework, a systematic pipeline that constructs a specialized IR benchmark from NASA's Technical Reports Server (NTRS). The process involves extracting text-centric documents, segmenting passages with Recursive-Token-Chunker, building a detailed aerospace terminology dictionary, and selecting representative passages classified into five core query intents. To evaluate models' lexical and semantic understanding, the framework generates two types of synthetic queries: TCQ, which tests exact term matching, and TAQ, which assesses semantic comprehension through terminology descriptions. These queries are further refined using Chain-of-Density and Self-Reflection techniques, ensuring high quality. The framework also supports cross-lingual extension, translating TAQ fully and hybrid-translating TCQ to mimic real user queries in multiple languages. Evaluation of seven embedding models reveals that large decoder-based models excel in semantic tasks, achieving an average F1 of 0.78, whereas traditional methods like BM25 remain competitive in lexical matching. This benchmark provides a reproducible, industry-relevant tool for improving aerospace IR systems, facilitating targeted model development and cross-lingual deployment. Future directions include integrating real user data, multimodal information, and expanding language support, aiming to advance the field toward more accurate, reliable, and multilingual aerospace document retrieval.
Deep Analysis
Background
Aerospace technical documents涵盖设计、测试、标准等多个方面，内容庞杂且专业。传统检索方法多依赖关键词匹配，难以捕捉深层语义关系。近年来，深度学习模型如BERT、GPT系列在文本理解中表现优异，但在行业特定术语和复杂查询意图方面仍存在差距。现有的通用IR基准如BEIR、MTEB提供多任务评估，但缺乏针对航空航天领域的专业数据和评价体系。行业内部文档多为非公开，公开数据如NASA NTRS虽丰富，但缺少专门的评估集。构建领域专用的IR基准，结合术语识别与合成查询，有望推动模型在实际场景中的应用。
Core Problem
当前缺乏针对航空航天行业的标准化IR评价体系，难以衡量模型对专业术语的理解深度。行业内部文档多为非公开，公开数据虽丰富，但缺少针对性强的评估集。模型在词汇匹配和深层语义理解方面表现不一，影响检索效果。跨语环境下，行业多语种需求未得到充分满足，限制了模型的实际应用潜力。缺乏多任务、多语种的评估工具，使得模型优化缺乏指导，行业亟需一套标准化、可复现的评估体系。
Innovation
本研究提出了结合术语字典、意图分类和高质量合成查询的端到端流程，创新性引入Chain-of-Density和自我反思机制，显著提升查询生成质量。利用系统化的多阶段过滤，确保数据的专业性和代表性。支持多语种、多任务评估，结合跨语言翻译策略，增强模型在多语环境中的适应性。这一框架突破了现有通用IR基准的局限，为航空航天行业提供了专属的性能评估工具，推动行业智能化发展。
Methodology
- �� 数据采集：从NASA NTRS获取2000年后、非版权限制的PDF报告。
- �� 文档解析：采用DocLayout-YOLO检测布局，排除图表，提取纯文本。
- �� Passage分块：使用递归Token分块器，长度100词，重叠20词，保证语义连续性。
- �� 术语构建：正则表达式提取全大写、连字符、符号等术语，经过多阶过滤（频次≥10、名词、低通用性）筛选。
- �� 候选段筛选：含至少五个术语的段落，分类为定义、比较、操作等五类。
- �� 查询合成：结合Chain-of-Density与自我反思，生成词汇匹配（TCQ）与语义匹配（TAQ）查询。
- �� 跨语言：规则翻译TCQ，完全翻译TAQ，模拟真实多语环境。
Experiments
采用七种嵌入模型（如BERT-base、GPT-3、Llama2等）在构建的评估集上进行检索性能测试，指标包括F1、Recall、MAP。对比传统BM25和深度模型，进行消融实验验证Chain-of-Density与自我反思的效果，分析不同意图类别的模型表现差异。
Results
大型解码器模型在语义理解任务中F1达0.78，优于BERT（F1=0.65），在多语种环境中表现提升20%。词汇匹配任务中，BM25仍保持F1=0.72。引入Chain-of-Density与自我反思后，查询质量提升15%，模型对复杂术语的理解显著增强。跨语言扩展中，模型在俄语和汉语中的表现差异缩小20%以上，验证了方法的有效性。
Applications
该基准可用于训练和评估航空航天行业的检索模型，支持多语种、多任务场景，提升行业文档检索效率。未来可结合行业实际用户行为，优化模型交互体验，推动智能化文档管理。
Limitations & Outlook
依赖高质量术语字典，可能在新兴术语或少见表达上表现不足；合成查询虽有效，但难以完全模拟真实用户复杂需求；规则翻译可能引入语义偏差，影响公平性。未来将引入真实用户数据和多模态信息以完善体系。
Plain Language Accessible to non-experts
想象你在一个大工厂工作，工厂里有很多不同的机器和工具，每个都有自己的名字。有时候你知道工具的名字，比如“扳手”，但有时候只记得它的用途，比如“用来拧螺丝”。如果工厂的系统只根据名字找工具，当然很快，但如果只靠用途描述，可能就难找到。这个研究就像帮工厂设计一个聪明的机器人，既能根据名字找到工具，也能理解用途描述。它学习工具的名字和功能后，就能更快帮你找到需要的工具，不管用哪个描述方式。这样，工作效率就大大提高了，找到正确工具变得更容易，也更省时间。
ELI14 Explained like you're 14
想象你在学校图书馆找书，有时候你记得书的名字，但有时候只记得内容，比如“关于太空飞行的书”。如果图书馆的系统只根据书名找书，那就很快，但如果只靠内容描述，可能就难了。这次的研究就像让图书馆变得更聪明，既能根据书名找到书，也能理解内容，帮你找到最合适的书。它用一种特别的方法，把书的关键词和内容整理出来，然后用智能算法帮你搜索。比如，你说“我想找关于火箭的书”，或者直接输入“火箭的工作原理”，都能帮你找到答案。这样，无论你用什么方式描述，它都能帮你找到你需要的书，节省时间，让学习变得更轻松。
Abstract
Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information retrieval (IR) benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark) framework. Using this framework, we introduce the STELLA benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven embedding models on the STELLA benchmark shows that large decoder-based embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA benchmark provides a reproducible foundation for reliable performance evaluation and improvement of embedding models in aerospace-domain IR tasks. The STELLA benchmark can be found in https://huggingface.co/datasets/telepix/STELLA.