MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish
MoganColBERT-TR uses MaxSim for token-level retrieval, achieving strong zero-shot performance on five Turkish BEIR datasets, outperforming larger models.
Key Findings
Methodology
MoganColBERT-TR builds on MoganBERT-embed with a 768→128 projection and MaxSim scoring. Training uses title-passage pairs and Turkish QA datasets, distilled via bge-reranker-v2-m3.
Key Results
- Result 1: Achieved 35.53 nDCG@100 on five Turkish BEIR datasets, outperforming ColmmBERT-base-TR by +2.99.
- Result 2: Excelled on SciFact-TR and ArguAna-TR, with +7.24 and +5.97 nDCG@10 improvements, respectively.
- Result 3: Despite having only 148.9M parameters, it outperformed ColmmBERT-base-TR on four datasets.
Significance
This work provides an efficient multi-vector retrieval model tailored for Turkish, addressing tokenization challenges in agglutinative languages. It sets a benchmark for low-resource language retrieval.
Technical Contribution
Introduced a Turkish-specific multi-vector model leveraging MoganBERT-embed geometry and MaxSim scoring, with a novel three-way hard negative mining strategy (rank skipping, group mask, cosine ceiling).
Novelty
The first Turkish multi-vector retrieval model trained from scratch, diverging from multilingual transfer learning approaches by relying solely on monolingual corpora.
Limitations
- Limitation 1: Underperformed on NFCorpus-TR, likely due to insufficient domain-specific data.
- Limitation 2: Fixed 32-token query length limits performance on long-query tasks like ArguAna-TR.
- Limitation 3: Single-epoch distillation may restrict potential performance gains.
Future Work
Future work could explore support for longer queries, improved distillation methods, and extensions to other low-resource languages.
AI Executive Summary
MoganColBERT-TR is a multi-vector retrieval model tailored for Turkish, employing MaxSim late interaction. Unlike traditional single-vector models, it retains token-level representations via a 768→128 linear projection, avoiding information bottlenecks. Built on MoganBERT-embed, it uses single-epoch distillation with bge-reranker-v2-m3 for optimization.
In zero-shot evaluations on five Turkish BEIR datasets, MoganColBERT-TR achieved an average nDCG@100 of 35.53, surpassing the twice-larger ColmmBERT-base-TR by +2.99. It excelled on SciFact-TR and ArguAna-TR, with +7.24 and +5.97 nDCG@10 improvements, demonstrating strength in complex query-document matching tasks.
However, the model underperformed on NFCorpus-TR and is constrained by its fixed query length. Future research could address these limitations and explore its application to other low-resource languages, further advancing multilingual retrieval systems.
Deep Analysis
Background
Information retrieval has evolved from sparse models like BM25 to dense models like DPR and Sentence-BERT. While dense models are efficient, they compress entire documents into single vectors, losing detail. ColBERT introduced MaxSim late interaction, balancing efficiency and token-level granularity.
Core Problem
Turkish's agglutinative nature complicates tokenization, as single words often split into multiple subwords. Mean pooling in single-vector models blends these subwords, losing critical information. A model retaining token-level representations is needed to improve retrieval accuracy.
Innovation
Key innovations include:
- �� Initialization from MoganBERT-embed, leveraging its isotropic geometry.
- �� MaxSim scoring to preserve token-level information.
- �� A three-way hard negative mining strategy (rank skipping, group mask, cosine ceiling).
- �� Single-epoch distillation using bge-reranker-v2-m3 for soft-label optimization.
Methodology
- �� Initialization: Replaced mean pooling in MoganBERT-embed with a 768→128 projection.
- �� Data: Combined title-passage pairs from pretraining corpus with Turkish QA datasets.
- �� Negative mining: Applied rank skipping, group masking, and cosine ceiling to filter negatives.
- �� Training: Used KL-divergence distillation with bge-reranker-v2-m3 as the teacher model.
Experiments
Evaluations used TurkColBERT on five Turkish BEIR datasets, including SciFact-TR and ArguAna-TR. PLAID indexing ensured exact MaxSim scoring. Metrics included nDCG@10 and nDCG@100, with all experiments conducted in a zero-shot setting.
Results
MoganColBERT-TR achieved 35.53 nDCG@100, outperforming ColmmBERT-base-TR by +2.99. It excelled on SciFact-TR and ArguAna-TR, with +7.24 and +5.97 nDCG@10 improvements.
Applications
The model is ideal for Turkish QA systems, document retrieval, and scientific literature analysis, particularly for complex queries and long-text matching.
Limitations & Outlook
Underperformance on NFCorpus-TR suggests domain-specific data limitations. Fixed query length also constrains performance on long-query tasks.
Plain Language Accessible to non-experts
Imagine searching for a book in a library. Traditional methods use just the book's cover to decide if it's relevant. MoganColBERT-TR, however, reads every page and matches them to your query, ensuring nothing important is missed, even for complex questions.
ELI14 Explained like you're 14
Think of it like finding a hidden treasure in a game. Older methods just check one spot on the map, but MoganColBERT-TR breaks the map into pieces and checks each one carefully. It’s like having a super-smart treasure hunter on your team!
Glossary
MaxSim
A scoring mechanism matching each query token to its most similar document token and summing the scores.
Used for query-document relevance scoring.
ColBERT
A retrieval model combining dense encoding with late interaction.
Forms the core architecture of MoganColBERT-TR.
Distillation
A training process where a student model learns from a teacher model's soft labels.
Used to optimize MoganColBERT-TR.
PLAID Indexing
A high-fidelity indexing method supporting exact MaxSim scoring.
Used for evaluation in TurkColBERT.
Hard Negative Mining
Selecting challenging negative examples to improve model performance.
MoganColBERT-TR uses a three-way filtering strategy for this.
Open Questions Unanswered questions from this research
- 1 How can the model handle longer queries more effectively?
- 2 Can this approach generalize to other low-resource languages?
- 3 What optimizations can improve hard negative mining?
Applications
Immediate Applications
QA Systems
Enhance Turkish QA systems' accuracy, especially for complex queries.
Scientific Retrieval
Help researchers quickly locate relevant Turkish scientific literature.
Long-term Vision
Low-resource Language Retrieval
Develop efficient retrieval systems for other low-resource languages, advancing multilingual NLP.
Abstract
We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.