Diffusion-Pretrained Dense and Contextual Embeddings
pplx-embed combines diffusion pretraining with contrastive curricula, reaching 69.66 MTEB and 81.96 ConTEB nDCG@10.
Key Findings
Methodology
The authors continue-pretrain Qwen3-0.6B/4B as bidirectional encoders with an absorbing-state diffusion objective, then apply Pair, Contextual, and hard-negative Triplet Training. Mean pooling, native INT8 quantization, cosine similarity, false-negative masking, and late chunking form the retrieval stack. pplx-embed-context-v1 optimizes local chunk and global document objectives; pplx-embed-v1 merges contextual and triplet checkpoints with Spherical Linear Interpolation.
Key Results
- On MTEB Multilingual v2, pplx-embed-v1-4B INT8 obtains 69.66 nDCG@10, slightly exceeding Qwen3-Embedding-4B at 69.60 and outperforming text-embedding-3-large at 59.27. It stores 390 document embeddings per MB, versus 97 and 81 for those baselines.
- On ConTEB, pplx-embed-context-v1-4B reaches 81.96 nDCG@10, above voyage-context-3 at 79.45 and Anthropic Contextual at 72.4. On ToolRet, its 44.45 average nDCG@10 exceeds NV-Embed-v1 at 42.71.
- The compact model is highly competitive: pplx-embed-v1-0.6B INT8 scores 68.6 on MIRACL, above its 4B counterpart at 66.2 and Qwen3-0.6B at 61.2. In BERGEN, the 0.6B model beats Qwen3-4B on three of five QA tasks.
Significance
The work addresses three persistent deployment tensions: causal encoders do not fully represent bidirectional passage meaning, independent chunking loses document context, and floating-point vectors are expensive at web scale. Diffusion pretraining enables bidirectional attention, mean pooling, and context-aware late chunking; quantization reduces memory and indexing cost. Evaluation spans multilingual and code retrieval, RAG, tool search, query-to-query retrieval, and an internal suite built from one billion production webpages. Thus the contribution is both algorithmic and operational: it targets retrieval quality under realistic memory and scale constraints rather than benchmark accuracy alone.
Technical Contribution
The main contributions are a continuous-time absorbing-state diffusion objective for converting Qwen3 into a bidirectional encoder; an InfoNCE Pair loss with similarity-threshold false-negative masking; a Contextual objective combining in-sequence and in-batch chunk losses with α=0.2; duplicate-document hash masking and a cosine schedule for β from 0.2 to 0.5; hard-negative Triplet loss; and Spherical Linear Interpolation for checkpoint merging. Quantization is applied during every contrastive stage, with a straight-through estimator for rounding, enabling native INT8 embeddings.
Novelty
The novelty lies less in inventing a single new loss than in integrating diffusion language modeling, contextual chunk training, late chunking, quantization-aware contrastive learning, and checkpoint merging into one production-oriented family. Unlike many causal decoder-based systems using last-token pooling or instruction prefixes, the work demonstrates that a bidirectional diffusion backbone can jointly improve long-document context preservation and storage efficiency.
Limitations
- The report provides aggregate benchmark and engineering results but limited ablations, training-cost accounting, and internal one-billion-page details. Consequently, the independent contribution of diffusion pretraining, contextual training, triplets, and merging is difficult to isolate.
- MIRACL does not show monotonic scaling: the 0.6B model averages 68.6 versus 66.2 for 4B. INT8 and binary representations also lose accuracy, and robustness across domains, languages, and distribution shifts remains insufficiently characterized.
Future Work
Future work should study diffusion schedules, sequence length, late-chunking boundaries, and more transparent ablations. Important directions include lower-bit quantization, better hard-negative mining, domain adaptation, latency and energy reporting, multilingual balancing, and multimodal or table-aware contextual embeddings. Releasing richer production statistics would also improve reproducibility and practical comparison.
AI Executive Summary
Modern search and RAG systems must map queries and documents into a shared vector space. Yet many strong encoders inherit causal decoder architectures and last-token pooling, while long-document chunking discards global context. Floating-point vectors further make billion-page indexing expensive. These limitations create a practical gap between semantic quality and deployability.
Perplexity’s pplx-embed family addresses the gap by continuing Qwen3-0.6B and 4B pretraining with an absorbing-state diffusion objective after disabling causal masking. The resulting bidirectional encoders undergo Pair Training, Contextual Training, and hard-negative Triplet Training. Mean pooling, late chunking, similarity-based false-negative masking, native INT8 quantization, and Spherical Linear Interpolation checkpoint merging complete the framework. The contextual model explicitly combines local chunk and global document InfoNCE objectives.
The reported results are strong across settings. pplx-embed-v1-4B INT8 reaches 69.66 nDCG@10 on MTEB Multilingual v2, with 390 documents per MB, while pplx-embed-context-v1-4B reaches 81.96 on ConTEB, exceeding voyage-context-3’s 79.45. ToolRet performance is 44.45, and the 0.6B model beats Qwen3-4B on three BERGEN tasks. The study supports bidirectional diffusion pretraining as a promising route to high-quality, memory-efficient retrieval, although fuller ablations, cost data, and multilingual failure analysis remain necessary.
Deep Analysis
Background
Dense embeddings map queries and documents to vectors for approximate nearest-neighbor search, powering search and RAG. Recent systems such as Qwen3-Embedding and BGE-M3 largely build on decoder-style language models, but causal attention and last-token pooling limit global passage representation. ConTEB highlights an additional problem: chunk retrieval often requires document-level context that independently encoded passages do not retain.
Core Problem
The target is a multilingual retriever that is accurate on web, code, tools, and long documents while remaining memory efficient. Key bottlenecks include learning bidirectional representations, avoiding false negatives from duplicate or semantically related in-batch samples, separating near-miss hard negatives, and preserving quality after INT8 or binary quantization.
Innovation
- ��Diffusion continued pretraining converts Qwen3 into a bidirectional encoder.
- ��Mean pooling and late chunking preserve global context.
- ��Contextual InfoNCE jointly trains chunk-level and document-level relevance.
- ��Pair loss masks likely false negatives using a +0.1 similarity threshold.
- ��Triplet checkpoints are merged with contextual checkpoints through Spherical Linear Interpolation.
- ��Quantization-aware training directly produces INT8 embeddings.
Methodology
- ��Pretraining: sample t∈[0.001,1], independently mask tokens with probability t, and optimize masked-token cross-entropy scaled by 1/t. Training uses 60,000 steps, global batch 1024, length 4096, about 250B multilingual tokens, and AdamW peak rates of 5×10^-4 and 3.16×10^-4.
- ��Pooling: average L token vectors, apply 127·tanh, and round to [-127,127]; compare vectors with cosine similarity.
- ��Pair stage: InfoNCE contrasts positive documents against documents and queries in batch, masking suspicious negatives.
- ��Context stage: combine in-sequence and in-batch losses with α=0.2; increase global-document weight β from 0.2 to 0.5 using a cosine schedule.
- ��Triplet stage: add hard negatives, then merge checkpoints for the standard model.
Experiments
Evaluation covers 18 retrieval tasks in MTEB Multilingual v2, MTEB Code, MIRACL, eight ConTEB datasets, five BERGEN QA tasks, 35 ToolRet tasks, PPLXQ2Q, and an internal suite based on one billion webpages. Metrics include nDCG@10, Recall@K, precision, comprehensiveness, and answer match. Baselines include Qwen3-Embedding, BGE-M3, Gemini, voyage-context-3, NV-Embed-v1, and GritLM-7B; INT8 and binary outputs are compared.
Results
The 4B INT8 model scores 69.66 on MTEB Multilingual and 78.73 on MTEB Code, with 66.2 on MIRACL. Context-v1-4B reaches 81.96 on ConTEB, including 93.04 on Geography and 78.13 on Football. BERGEN scores for 4B across NQ, HotpotQA, TriviaQA, ASQA, and PopQA are 67.7, 51.9, 91.9, 71.5, and 68.7. ToolRet averages 44.45. Binary storage rises to 3,125 docs/MB, while multilingual score declines to 68.22.
Applications
The models can support web search, enterprise RAG, multilingual knowledge bases, long-document QA, code retrieval, and API-tool prefiltering. INT8 vectors are particularly suitable for memory-constrained vector databases and large ANN indexes. ToolRet suggests that semantic tool selection can reduce context explosion before an LLM generates a plan or answer.
Limitations & Outlook
The paper does not fully disclose dataset sizes, training duration, hard-negative construction, or internal webpage evaluation protocols, limiting reproducibility and cost analysis. The models are optimized for retrieval rather than generation, and language performance is uneven; 4B is not consistently better than 0.6B on MIRACL. Future work should add complete ablations, domain-shift tests, latency and energy measurements, improved multilingual balancing, lower-bit quantization, and multimodal evaluation.
Plain Language Accessible to non-experts
Imagine a gigantic library. A conventional system labels each book by looking mainly at its final pages, so it can miss clues from the beginning. It also cuts a long book into chapters and labels each chapter as if it came from nowhere. pplx-embed first trains a reader who can look both forward and backward, then lets that reader understand the whole book before labeling each chapter. Each chapter therefore carries clues about the book it belongs to.
Training resembles a librarian practicing in stages: first matching questions with the right books, then finding the right chapter inside a book, and finally rejecting books that look similar but answer a different question. The labels are compressed into small eight-bit codes, allowing many more books to fit in storage. Even a two-choice binary code remains surprisingly useful.
This makes the system practical for web search, company archives, and tool catalogs. The trade-off is that aggressive compression can lose detail, and the report does not fully reveal which training stage contributes most. Still, the central idea is simple: understand the whole document first, then retrieve the precise part.
ELI14 Explained like you're 14
Think about searching for a game guide. You type, “How do I beat the final boss?” The system must choose the best answer from millions of pages. An older approach is like reading only the last few sentences of every guide. It might miss the important weapon described near the beginning. Cutting a long guide into pieces can also make each piece forget which game it belongs to.
pplx-embed trains a “super reader” that looks at words on both sides of every word. It reads the whole guide, then labels each small section. So a section saying “dodge the fire during phase two” can still be recognized as relevant even if it never repeats the boss’s name. The training also includes tricky almost-correct guides, teaching the system to notice fine differences.
Then the labels are squeezed into eight-bit numbers, like shrinking a game inventory into compact icons. The 4B model scores 69.66 on multilingual retrieval and 81.96 on contextual retrieval. A much smaller 0.6B model is also surprisingly strong.
There are catches: extreme compression loses some accuracy, and languages do not all behave equally. But imagine this powering search, school study tools, or an assistant finding the right software API. Pretty cool, right?
Glossary
Diffusion language model
A model trained to recover text after tokens have been progressively corrupted or masked. Unlike a causal language model, it can use bidirectional attention.
The paper uses an absorbing-state [MASK] diffusion objective to continue-pretrain Qwen3.
Mean pooling
A sequence representation formed by averaging all token vectors. It can capture distributed information when token states have bidirectional context.
pplx-embed uses mean pooling instead of the last-token pooling common in decoder-based embedders.
Late chunking
A long document is encoded first, and chunk vectors are then derived from the relevant token representations. This lets each chunk retain document-level context.
The contextual model uses late chunking in ConTEB evaluation.
InfoNCE
A contrastive objective that increases similarity for positive pairs relative to negatives, with temperature τ controlling concentration. It is widely used for retrieval representation learning.
Pair, Contextual, and Triplet stages use InfoNCE variants.
INT8 quantization
A representation is converted to signed 8-bit integers to reduce memory and accelerate storage and search. Quantization-aware training can reduce the associated quality loss.
The model trains with quantization and outputs INT8 embeddings by default.
nDCG@10
A ranking metric that rewards highly relevant results near the top ten positions and discounts lower-ranked results. Scores are commonly reported as percentages.
The paper uses nDCG@10 for MTEB, ConTEB, ToolRet, and related comparisons.
Open Questions Unanswered questions from this research
- 1 The contribution of each stage is not isolated. Because a complete ablation is absent, it remains unclear how much performance comes from diffusion pretraining, contextual objectives, hard negatives, or checkpoint merging.
- 2 The internal one-billion-page evaluation is only summarized. Query distributions, labeling procedures, latency, indexing cost, and hardware are needed to verify the claimed production advantage.
- 3 Cross-lingual scaling and very low-bit quantization need deeper study, especially because the 0.6B model outperforms 4B on average MIRACL despite having far fewer parameters.
Applications
Immediate Applications
Multilingual web search
Search providers can index INT8 pplx-embed-v1 vectors with approximate nearest-neighbor systems while conserving memory. The 4B model’s 69.66 MTEB Multilingual score makes it suitable for first-stage recall before reranking.
Enterprise RAG and tool retrieval
Organizations can encode report passages with Context-v1 so retrieved chunks retain document context. API platforms can use v1 to prefilter large tool catalogs; its 44.45 ToolRet score indicates useful context reduction before LLM generation.
Long-term Vision
Low-cost semantic infrastructure
Combining binary embeddings with efficient ANN hardware could support semantic indexes over billions of documents at substantially lower cost. The main obstacles are preserving multilingual, domain-specific, and long-document accuracy.
Abstract
In this report, we introduce pplx-embed, a family of multilingual embedding models that employ multi-stage contrastive learning on a diffusion-pretrained language model backbone for web-scale retrieval. By leveraging bidirectional attention through diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard retrieval, and pplx-embed-context-v1 for contextualized embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB benchmark. Beyond public benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world, large-scale search scenarios constructed from 1B production web pages. These results validate the models' effectiveness in production environments where retrieval quality and efficiency are critical at scale.