Wontopos Tablet 2: Measuring Multilingual and Multimodal Memory Retrieval Without Lexical Matching
Wontopos Tablet 2 achieves multilingual and multimodal retrieval without lexical matching, scoring 95.7% on LongMemEval-S.
Key Findings
Methodology
Wontopos Tablet 2 is a production-grade memory engine supporting multilingual and multimodal retrieval. Its retrieval path avoids lexical matching, keyword scoring, or language models, using a staged architecture to retrieve captionless photos from text queries.
Key Results
- Scored 95.7% on LongMemEval-S (500 questions), with a confidence interval of [93.4, 97.1] and an engine latency of 393ms.
- Achieved 67.5% on BEAM-1M (700 questions, 2.21M memories), with an 8.9-point gain from re-ask settings.
- On Crossmodal-3600 (14 languages, captionless photos), Recall@5 averaged 91.4%, outperforming BM25's 0%.
Significance
This work demonstrates the feasibility of multilingual and multimodal retrieval without lexical matching, addressing gaps in low-resource languages and captionless photo retrieval. It has significant implications for cross-language IR and multimodal applications.
Technical Contribution
Introduced a retrieval path independent of language models or lexical matching, enabling robust cross-language and multimodal retrieval. The staged architecture separates retrieval from reasoning for greater flexibility.
Novelty
First to achieve high performance in cross-language retrieval of captionless photos, proving dense representations are not inherently language-independent.
Limitations
- Performance drops significantly for low-resource languages like Swahili (53.0%) and Telugu (64.0%).
- Adding English captions reduces cross-lingual retrieval performance by 11.4 points on average.
- Misconfigured retrieval stages caused a 37-point drop in Korean top-1 accuracy.
Future Work
Future work could focus on improving performance for low-resource languages, enhancing robustness in multimodal retrieval, and optimizing context expansion strategies.
AI Executive Summary
Wontopos Tablet 2 is a novel long-term memory engine designed for multilingual and multimodal retrieval, excelling in scenarios where lexical matching is impossible. Its design prioritizes language independence by avoiding reliance on language models or keyword matching.
In text retrieval tasks, Tablet 2 achieved 95.7% on LongMemEval-S and 67.5% on BEAM-1M, demonstrating its efficiency in large-scale memory storage. For multimodal retrieval, it excelled in cross-language retrieval of captionless photos, achieving a Recall@5 of 91.4%, far surpassing traditional methods.
However, challenges remain, including performance drops in low-resource languages and the negative impact of captions on retrieval accuracy. Future research should address these issues while exploring more efficient multimodal retrieval strategies.
Deep Analysis
Background
Multilingual and multimodal retrieval is a critical challenge in information retrieval. Traditional methods rely on lexical matching or language models, which struggle with language independence and captionless photo retrieval.
Core Problem
Existing methods face significant limitations in multilingual and captionless photo retrieval, particularly in low-resource languages and multimodal tasks.
Innovation
Tablet 2 introduces a lexical-free retrieval path with a staged architecture, achieving language independence and addressing gaps in captionless photo retrieval.
Methodology
- �� Utilizes a staged retrieval architecture to separate retrieval from reasoning.
- �� Supports storage and retrieval of multilingual and multimodal data.
- �� Configurable re-ask budget enhances recall.
- �� Enables cross-language matching via dense representations for captionless photos.
Experiments
Experiments used LongMemEval-S, BEAM-1M, and Crossmodal-3600 datasets to evaluate text and multimodal retrieval. Settings included varying re-ask budgets and context expansion strategies.
Results
Achieved 95.7% on LongMemEval-S and 67.5% on BEAM-1M. On Crossmodal-3600, Recall@5 reached 91.4%, significantly outperforming BM25's 0%.
Applications
The technology can be applied to multilingual search engines, captionless photo retrieval systems, and cross-modal data integration in industry.
Limitations & Outlook
Performance in low-resource languages is limited, captions negatively impact cross-language retrieval, and some retrieval stages are sensitive to configuration.
Plain Language Accessible to non-experts
Imagine a massive library where books have no titles and are written in different languages. Wontopos Tablet 2 acts like a super-librarian, finding the right book based on your description without needing keywords or titles. For example, if you describe 'a cat playing on grass,' it will find the book with that photo. What's special is that it works even if you describe it in Russian or Swahili, which traditional methods can't handle.
ELI14 Explained like you're 14
Imagine you're searching for a picture online, but the picture has no title or description. A regular search engine would be clueless. But Wontopos Tablet 2 is like a super detective! You just say, 'a dog running by the beach,' and it finds the picture! And guess what? It understands you no matter what language you use. Cool, right? But it still struggles with rare languages like Swahili. Something to improve for sure!
Glossary
Lexical-free retrieval
A method that avoids keyword or lexical similarity matching.
Used to achieve language independence in retrieval.
Multimodal retrieval
Retrieval tasks involving multiple data types (e.g., text and images).
Critical for captionless photo retrieval.
LongMemEval-S
A benchmark with 500 questions for text retrieval.
Evaluates Tablet 2's text retrieval performance.
BEAM-1M
A large-scale benchmark with 2.21M stored memories.
Tests Tablet 2's scalability.
Crossmodal-3600
A dataset with 14 languages and captionless photos.
Evaluates cross-language and multimodal retrieval.
Open Questions Unanswered questions from this research
- 1 How to improve retrieval performance for low-resource languages?
- 2 How to mitigate the negative impact of captions on cross-language retrieval?
- 3 Are there more efficient multimodal retrieval strategies?
Applications
Immediate Applications
Multilingual search engines
Enable cross-language information retrieval for global users without translation.
Captionless photo retrieval
Help users find images based on descriptions without captions.
Long-term Vision
Cross-modal knowledge base
Build a global knowledge base supporting multilingual and multimodal data.
Abstract
We measure tablet-2, a production long-term memory engine for language models, on the text benchmarks the field already uses and on cross-lingual retrieval of photographs stored with no text at all. Its retrieval path contains no lexical matching, no keyword scoring, and no language model of its own. On LongMemEval-S (500 questions) it scores 95.7% [93.4, 97.1]; on BEAM-1M (700 questions, 2.21M stored memories) 67.5% [64.8, 70.2]. Those are question-sampling intervals, not the run-to-run spread, which is an order of magnitude narrower. Most of the paper is about how little they mean alone. Holding engine, corpus, settings and judge fixed, changing only the reader moves LongMemEval-S by 2.0 points; changing only the re-ask budget moves BEAM-1M by 8.9. Neither is stated in the reports we compare against, and the second exceeds most gaps there, so we give that table as a placement and not a ranking. For the multimodal axis we run two controls. Against BM25, configured as strongly as we could, we reach 95.2% mean recall@5 over 70 store-and-query language cells where BM25 reaches 19.0% and is exactly zero in 54. On captionless photographs a lexical method has no document to score at all. Open dense baselines on 300 Crossmodal-3600 photographs in 14 languages show that density confers no language independence: one scores 91.0% on English and 4.7% on Russian from identical image vectors, and a multilingual variant collapses on Telugu and Swahili. Our spread across languages is 14.0 against their 27.5 and 27.7. Three results run against us and are reported at equal weight: low-resource languages degrade sharply (Swahili 53.0%, Telugu 64.0%), attaching captions lowers cross-lingual retrieval by 11.4 points, and one setting omitted into one stage of our own retrieval cost 37 points of Korean top-1 accuracy while leaving nine languages untouched.