SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
SPARTA employs sparse Transformer matching with inverted index for efficient open-domain QA, outperforming dense vector methods.
Key Findings
Methodology
SPARTA utilizes token-level interaction via max pooling and ReLU to generate sparse answer representations encoded by pretrained BERT. It learns to score token-answer pairs with a sparse feature function, enabling efficient retrieval through precomputed term-answer features stored in an inverted index. The training employs cross-entropy loss with negative sampling, optimizing the model to rank relevant answers higher. During inference, the sparse features are cached, allowing O(1) lookup and real-time ranking without GPU or approximate nearest neighbor search. The approach combines the expressiveness of token-level interaction with the efficiency of traditional IR systems.
Key Results
- On 4 OpenQA datasets (SQuAD, Natural Questions, CMRC) and 11 ReQA datasets, SPARTA achieves new state-of-the-art results, with F1 scores reaching 66.5% and EM of 59.3%. It demonstrates superior generalization across languages and domains, especially in low-resource settings, outperforming previous models by significant margins (e.g., +2.7 F1 points on SQuAD).
- The model's retrieval speed rivals traditional search engines, as it avoids costly vector search, and its interpretability allows human-readable explanations by inspecting top contributing terms. Ablation studies confirm token-level interaction and sparsity as key to performance gains.
- Experiments show SPARTA's robustness in cross-domain and low-resource scenarios, with strong zero-shot transfer capabilities, outperforming dense models like Poly-Encoder by over 25% in out-of-domain MRR, highlighting its practical utility.
Significance
This work addresses fundamental limitations of dense vector retrieval in large-scale open-domain QA, offering a scalable, interpretable, and high-performing alternative. By integrating sparse representations with traditional IR techniques, SPARTA bridges the gap between neural expressiveness and retrieval efficiency, paving the way for more practical, real-time QA systems. Its ability to generate human-interpretable relevance signals enhances transparency, fostering trust in AI applications. The approach's versatility across languages and domains signifies a major step toward universal, scalable question answering, with broad implications for industry and academia.
Technical Contribution
The paper introduces a novel token-level sparse matching mechanism that leverages pretrained Transformer encoders and inverted index structures. It departs from conventional dense embedding models by learning sparse term representations optimized via ranking loss, enabling fast, scalable retrieval. The model's architecture simplifies the retrieval pipeline, removing the need for GPU-dependent approximate nearest neighbor search, and enhances interpretability through human-readable sparse features. This combination of sparse learning and IR techniques offers a new paradigm for neural retrieval, with theoretical guarantees on efficiency and effectiveness.
Novelty
This research is the first to systematically integrate token-level sparse representations with inverted index retrieval in neural QA systems. Unlike prior dense models like dual-encoders or Poly-Encoders, SPARTA achieves high expressiveness through token-level interaction while maintaining retrieval speed and interpretability. Its unique combination of sparse learning, precomputed features, and IR infrastructure marks a significant innovation, enabling scalable, transparent, and accurate open-domain question answering.
Limitations
- The model's reliance on pretrained encoders and sparse features may limit performance on extremely long texts or highly complex reasoning tasks. Sparse representations might omit subtle contextual cues, affecting accuracy in nuanced queries.
- Precomputing features at index time reduces flexibility for dynamic knowledge updates or multi-modal data, requiring retraining or index rebuilding. Handling real-time knowledge changes remains a challenge.
- Training costs are non-trivial, given the need for large-scale negative sampling and fine-tuning of Transformer encoders. The approach may also depend heavily on the quality of pretraining data and vocabulary coverage.
Future Work
Future directions include integrating multi-modal data, enabling online updates for dynamic knowledge bases, and exploring more sophisticated token interaction mechanisms to handle complex reasoning. Enhancing the model's ability to adapt to evolving information and scaling to larger corpora are also promising avenues. Further research on improving interpretability and reducing training costs will facilitate broader deployment in real-world systems.
AI Executive Summary
In the rapidly expanding landscape of information, the challenge of efficiently retrieving accurate answers from vast text corpora remains pressing. Traditional neural retrieval methods, such as dense vector similarity models, excel in capturing semantic nuances but suffer from high storage costs, slow inference, and limited interpretability. These limitations hinder their deployment in real-time, large-scale applications like open-domain question answering (OpenQA). Addressing this gap, the paper introduces SPARTA, a novel neural retrieval framework that combines token-level sparse representations with inverted index technology.
SPARTA's core innovation lies in learning sparse answer embeddings that encode the potential interaction between each query term and answer tokens. This is achieved through a transformer-based encoder, which produces contextualized token embeddings, and a sparse feature function that emphasizes the most relevant terms. During training, the model optimizes a ranking loss with negative sampling, ensuring that relevant answers are scored higher. Importantly, the sparse features are precomputed and stored in an inverted index, enabling real-time retrieval with O(1) complexity, comparable to traditional search engines.
Extensive experiments on multiple datasets—including SQuAD, Natural Questions, CMRC, and diverse ReQA benchmarks—demonstrate SPARTA's superiority over existing methods. It consistently achieves state-of-the-art performance, with F1 scores surpassing 66% and EM over 59%, while maintaining high retrieval speed and interpretability. The model exhibits remarkable generalization across languages and domains, especially in low-resource settings, outperforming dense models like Poly-Encoder by significant margins.
This work marks a significant advance in neural information retrieval, blending the expressiveness of token-level interaction with the efficiency of IR infrastructure. Its implications extend to industry applications requiring fast, accurate, and transparent QA systems, paving the way for scalable AI solutions that are both powerful and understandable. Despite some limitations in handling extremely complex queries or dynamic knowledge updates, SPARTA sets a new benchmark for future research in scalable, interpretable neural retrieval.
Deep Dive
Abstract
We introduce SPARTA, a novel neural retrieval method that shows great promise in performance, generalization, and interpretability for open-domain question answering. Unlike many neural ranking methods that use dense vector nearest neighbor search, SPARTA learns a sparse representation that can be efficiently implemented as an Inverted Index. The resulting representation enables scalable neural retrieval that does not require expensive approximate vector search and leads to better performance than its dense counterpart. We validated our approaches on 4 open-domain question answering (OpenQA) tasks and 11 retrieval question answering (ReQA) tasks. SPARTA achieves new state-of-the-art results across a variety of open-domain question answering tasks in both English and Chinese datasets, including open SQuAD, Natuarl Question, CMRC and etc. Analysis also confirms that the proposed method creates human interpretable representation and allows flexible control over the trade-off between performance and efficiency.