A Survey of Model Architectures in Information Retrieval

The evolution of IR has transitioned from early heuristic models (Boolean, vector space, probabilistic) to statistical language models and finally to neural architectures. The advent of transformers, exemplified by BERT and GPT, has revolutionized semantic understanding, enabling models to capture complex contextual relationships. Traditional models like BM25 and TF-IDF, while efficient, struggled with semantic mismatch and scalability. Deep neural models, especially dense retrieval methods and end-to-end rerankers, now dominate research and industry, supported by large datasets such as MS MARCO and TREC benchmarks. This progression reflects a shift towards more intelligent, context-aware systems capable of handling complex queries across diverse data modalities.

Despite progress, IR systems face key challenges: achieving high accuracy at scale, managing computational costs, and understanding deep semantics. Traditional models lack contextual awareness, leading to lexical mismatch issues. Neural models, while powerful, are resource-intensive and difficult to deploy in real-time. Additionally, integrating multi-modal data (text, images, video) and supporting multiple languages remain unresolved. The core problem is designing architectures that balance efficiency, scalability, and semantic depth, especially for complex reasoning and open-domain questions, which are critical for next-generation autonomous search agents.

Major innovations include: 1) Transformer-based encoders (BERT, RoBERTa) for rich semantic representations; 2) multi-stage retrieval pipelines combining fast dense retrieval with deep interaction-based reranking; 3) decoupling architecture design from training strategies, enabling flexible optimization. Introduction of multi-modal fusion techniques supports diverse data types, and the development of scalable, efficient models (via pruning, quantization) addresses deployment challenges. These innovations collectively push IR towards more intelligent, context-aware, and scalable systems.

• �� Feature encoding: Use pre-trained transformers (e.g., BERT, RoBERTa) to generate contextual embeddings for queries and documents. • Candidate retrieval: Employ bi-encoder models to compute similarity scores rapidly for large collections. • Deep reranking: Apply interaction models (Cross-Encoder, ColBERT) to refine candidate rankings through fine-grained matching. • Training: Use ranking losses like LambdaRank, ListMLE, combined with multi-task and contrastive learning to improve generalization. • Multi-modal fusion: Integrate visual features with textual embeddings for richer representations. • Optimization: Implement model compression techniques (pruning, quantization) for deployment efficiency.

Models are evaluated on datasets like MS MARCO and TREC, measuring nDCG@10, MRR, and Recall@100. Hyperparameters are tuned via grid search, with ablation studies to assess component contributions. Comparisons include traditional BM25, neural dense retrievers, and end-to-end rerankers. Experiments demonstrate that transformer-based dense models outperform classical methods by ~30% in nDCG@10, with recall@100 reaching 85%. Multi-modal models show robustness across languages and data types, validating their practical utility.

Transformer models like BERT achieve nDCG@10 of 0.45 on MS MARCO, outperforming BM25 by 30%. Dense retrieval models (ColBERT, ANCE) reach recall@100 of 85%, enabling scalable retrieval. End-to-end rerankers (LambdaRank) improve MRR to 0.40, validating their effectiveness. Multi-modal fusion enhances cross-lingual performance, demonstrating adaptability. These results confirm that architecture innovations significantly advance IR performance and scalability.

These models are applicable in search engines, enterprise knowledge bases, academic literature retrieval, and question-answering systems. They support multi-lingual and multi-modal data, suitable for real-time applications requiring high accuracy and reasoning. The integration into autonomous agents and intelligent assistants will further transform information access, making retrieval more natural and context-aware.

High computational costs hinder deployment in resource-constrained environments. Multi-modal and multilingual models still face integration challenges, with performance gaps across data types. Deep models lack explainability, complicating trust and interpretability. Future work should focus on efficiency, robustness, and explainability to broaden practical deployment.