Reducing Technician Search Burden: A Multimodal RAG for Cessna 172 Maintenance Manual

Aircraft maintenance manuals have evolved from paper documents to digital formats, with early efforts focusing on improving readability and accessibility through redesigns and simplified language. Recent advances leverage deep learning for content extraction, such as OCR and document parsing, enabling automated indexing and retrieval. Prior models like Hou et al.'s retrieval-augmented systems used text-only embeddings, which improved search efficiency but lacked visual context support. However, maintenance tasks often rely heavily on diagrams, tables, and images, which are inadequately handled by text-centric models. This gap underscores the need for multimodal retrieval systems that can process and integrate visual information, addressing the limitations of existing approaches and supporting technicians more effectively.

Current retrieval methods in aircraft maintenance are predominantly text-based, relying on keyword searches that are inefficient and often inaccurate due to inconsistent terminology and complex visual data. Technicians face difficulties locating precise information quickly, especially when diagrams or safety notes are involved. These limitations lead to increased manual effort, longer downtime, and higher risk of errors. The core challenge is developing a system capable of understanding and retrieving multimodal content—text, images, and tables—in a unified framework, ensuring fast, accurate, and interpretable support during maintenance procedures.

The main innovations include: 1) adopting ColQwen2-VL-2b-instruct as the backbone for cross-modal embedding, enabling direct image-text alignment; 2) embedding entire manual pages as images, avoiding multi-stage preprocessing and improving retrieval robustness; 3) integrating a vision-language model with GPT-4.1 for end-to-end response generation, providing contextually relevant answers; 4) introducing heatmap-based interpretability to visualize token alignment, enhancing transparency. These innovations collectively address the limitations of prior text-only models, offering a comprehensive multimodal solution tailored for aviation maintenance, with improved accuracy, efficiency, and explainability.

• �� Content extraction: Use OCR and deep learning models to convert manual pages into structured text and images.
• �� Embedding generation: Convert textual and visual content into high-dimensional vectors using ColQwen2-VL-2b-instruct.
• �� Retrieval process: For each query, embed the question and compute similarity scores with page embeddings via MaxSim, selecting top-k pages.
• �� Response generation: Input top-k pages into GPT-4.1 combined with VLM features, generating detailed answers.
• �� Synthetic dataset creation: Generate 400 queries across manual chapters, covering procedural, diagram, safety, and specification categories.
• �� Performance evaluation: Use nDCG@5, recall@5, and BERTscore to assess retrieval quality and response relevance.
• �� Practical testing: Measure inference times, operational costs, and visualize heatmaps for interpretability.

The experiments utilized a synthetic dataset of 400 queries reflecting real maintenance scenarios, covering all major manual chapters. Baselines included traditional keyword search and single-modal retrieval models. Metrics such as nDCG@5, recall@5, and semantic similarity were used for quantitative assessment. Hyperparameters like top-k retrieval size and embedding dimensions were tuned. Ablation studies examined the impact of visual embeddings versus text-only, demonstrating the advantage of multimodal fusion. Cost analysis confirmed low operational expenses, and heatmaps validated interpretability. Results consistently showed that the multimodal approach outperformed baselines, reducing manual search time and improving response accuracy across different query types.

The system achieved a recall@5 of 93.37% and an nDCG@5 of 83.38%, outperforming text-only models by 15-20%. Semantic similarity of responses reached 87.20%, confirming high relevance. Retrieval and response inference times averaged 11.93 and 4.95 seconds respectively, with costs below $0.01 per query. Heatmaps provided clear explanations of retrieval decisions, increasing user trust. These results demonstrate the effectiveness of multimodal fusion in complex maintenance scenarios, significantly reducing manual effort and error rates, and paving the way for intelligent maintenance automation.

This system can be deployed in aircraft maintenance, industrial equipment troubleshooting, and complex system diagnostics, providing technicians with rapid, multimodal access to relevant information. It supports real-time decision-making, reduces manual search efforts, and enhances safety. Future integration into maintenance platforms can enable automated workflows, predictive diagnostics, and remote assistance, transforming traditional maintenance into a smarter, more efficient process.

The model’s understanding of intricate diagrams remains imperfect, occasionally missing fine details. Inference speed, while optimized, may still lag in low-resource environments, affecting real-time deployment. Cost considerations for large-scale industrial use need further evaluation. Additionally, the system's performance depends on the quality of visual data; poor image quality can impair accuracy. Future improvements should focus on enhancing diagram comprehension, reducing latency, and expanding dataset diversity to improve robustness.