RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation

Recent advances in large language models (LLMs) and retrieval-augmented generation (RAG) systems have revolutionized knowledge-based AI applications. Early works like FAISS, Proposition Indexing, and Contextual Chunking improved retrieval efficiency but relied on post hoc interpretive steps, which are costly and lack provenance guarantees. As models scale, the cost of repeated semantic decoding during inference becomes prohibitive, especially with long contexts. Existing solutions struggle with source dynamism, verification, and maintenance, leading to unreliable outputs and high operational costs. The need for a persistent, verifiable, and efficient semantic storage mechanism has become urgent, motivating the development of precompiled semantic structures that can be incrementally maintained and validated.

The core challenge is to eliminate the repetitive semantic decoding at query time, which inflates inference costs and reduces reliability. Traditional indexes lack semantic verification, leading to hallucinations and inaccuracies. Dynamic sources further complicate maintenance, as re-computation is expensive and often infeasible at scale. The fundamental problem is designing a semantic storage system that is both cost-effective and verifiable, capable of handling source updates, model upgrades, and multi-user access without sacrificing accuracy or provenance. Addressing this problem is critical for deploying scalable, trustworthy knowledge systems.

The key innovation is the introduction of ingest-time semantic compilation (ISC), which precomputes and maintains a verified semantic substrate. This system employs two layers: a geometric layer of embeddings, maintained via incremental low-rank updates, and a symbolic layer of atomic claims with provenance validation. It formalizes contracts for compilation (semantic DDL), maintenance, migration, and cost optimization, enabling continuous updates aligned with source changes. Unlike prior methods relying solely on raw text or unverified chunks, ISC guarantees provenance, reduces maintenance costs, and improves retrieval accuracy. It leverages database principles—indexing, validation, and incremental updates—in a novel semantic context.

• �� Define semantic DDL to specify which source content is compiled into claims, with validation gates ensuring source span correctness.
• �� Construct a semantic substrate with two layers: geometric embeddings for fast retrieval, maintained via incremental low-rank updates; symbolic claims with provenance data for verification.
• �� Implement maintenance protocols that track source changes, updating only affected parts, reducing costs.
• �� Develop migration strategies for model upgrades, using orthogonal Procrustes alignment to preserve embedding space.
• �� Establish a cost model to decide when to compile based on expected read frequency, balancing compilation and interpretation.
• �� Use validation gates to prevent invalid claims from entering the index, ensuring high provenance integrity.

Experiments involved 500 broadcast interview transcripts, comparing compiled claims against chunking and hybrid retrieval pipelines. Metrics included accuracy, token budget, maintenance cost, and model migration impact. The system was tested under various source change scenarios, measuring incremental update costs and retrieval correctness. Results showed that incremental maintenance was 33.7× cheaper than full re-computation, with maximum principal-angle drift below 10^-11 degrees. In accuracy tests, compiled claims consistently outperformed chunking, especially at lower token budgets, demonstrating robustness and efficiency. The experiments validated the system's scalability and reliability in real-world scenarios.

Compiled claims achieved 85.2% correctness at a 2.2k token budget, outperforming chunking (72.5%) at 16.3k tokens. Maintenance costs scaled linearly with source changes, remaining far below re-computation costs. The system maintained high geometric alignment accuracy after model upgrades, confirming robustness. Across multiple budgets, compiled claims consistently delivered higher accuracy per token, validating the approach’s efficiency and reliability.

This architecture is ideal for large-scale knowledge bases, enterprise document management, and legal or compliance systems requiring verified, up-to-date information. It supports dynamic sources, model upgrades, and multi-user access, enabling reliable AI-powered decision-making, automated compliance checks, and efficient information retrieval in complex environments.

Current validation relies on exact span matching, which may be challenged by paraphrasing or ambiguous references. Handling highly volatile sources or frequent model upgrades could incur higher costs. Multi-tenant deployment, access control, and version management require further development to meet real-world enterprise needs. Scalability to multi-modal data and complex source structures remains an open challenge.