Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data

Over the past decade, large language models (LLMs) like GPT-3, BERT, and PaLM have revolutionized NLP, enabling impressive capabilities in understanding and generating language. However, their effectiveness diminishes when dealing with vast unstructured data sources, such as enterprise documents, legal texts, and web archives, due to token limitations and high inference costs. Existing solutions like Retrieval-Augmented Generation (RAG) and dense retrieval methods improve efficiency but rely heavily on static indexes or pre-structured data, which are costly to build and inflexible to new queries. Recent research has explored dynamic retrieval and multi-hop reasoning, but these approaches still face scalability issues. The core challenge remains: how to efficiently extract and reuse relevant structural information during reasoning, minimizing redundant document access while maintaining accuracy.

The main bottleneck in unstructured data reasoning with large models is the high token and computational cost associated with repeatedly opening large documents to gather scattered evidence. Traditional methods either pre-structure data, which is infeasible given the vast and unknown potential structures, or rely on static retrieval, which lacks adaptability. As queries grow in complexity and scope, these approaches become prohibitively expensive, limiting practical deployment in enterprise scenarios. The key problem is developing an adaptive, query-driven mechanism that can dynamically build and leverage structure during reasoning, reducing token consumption and improving scalability without sacrificing accuracy.

The paper introduces agentic data cracking, which dynamically extracts and stores structural information during reasoning, guided by observed queries. Unlike static pre-structuring, this method employs a cracking sub-agent that runs in parallel when documents are loaded, inferring entities and relations likely to be reused. It leverages semantic understanding to speculate about useful structures, storing them in a lightweight, query-guided database. This enables future queries to access structured data directly, avoiding document re-openings. The approach combines the principles of database cracking with the flexibility of LLMs, supporting multi-round reasoning and knowledge accumulation, thus significantly reducing inference costs while maintaining high accuracy.

• �� When a document is loaded, initiate a cracking sub-agent that uses semantic reasoning to infer relevant entities and relations.
• �� Based on observed queries, decide which structures to extract, focusing on reusable entities and relations.
• �� Store these structures in a lightweight, query-guided database, supporting fast structured reads.
• �� Use shared prefix caching to reuse previously loaded context, minimizing token costs.
• �� During reasoning, access stored structures via structured read APIs, reducing the need to open documents.
• �� When a structure is missing, fall back to opening the raw document, then update the structure.
• �� Continuously update the knowledge base as new queries arrive, improving coverage over time.
• �� Validate and normalize extracted structures to ensure consistency and quality.
• �� Experimental validation shows over 50% cost reduction with maintained accuracy across multiple scenarios.

The evaluation employed FanOutQA, comparing traditional retrieval methods with the proposed adaptive cracking system. The experiments measured token consumption, API costs, and answer accuracy, using Claude-Haiku-4.5 as the backbone model. The workload included single and multi-turn questions, with related questions generated to simulate real-world reuse. Ablation studies tested different structure extraction strategies, cache sizes, and query patterns. Results confirmed that the cracking approach reduces inference costs by over 50%, with minimal impact on accuracy, and scales well with workload size and complexity. The experiments also demonstrated the system’s ability to preemptively extract relevant structures, boosting efficiency in multi-hop reasoning tasks.

The adaptive cracking system achieved a 53% reduction in reasoning costs on FanOutQA, maintaining over 95% accuracy. Cost savings increased with workload duration and query reuse opportunities, reaching up to 9× cheaper at the 10th percentile. The system effectively covered over 80% of relevant facts through dynamic structure accumulation, significantly reducing document re-openings. Ablation results indicated that semantic-guided extraction and shared prefix caching were critical for maximizing savings. The results validate the approach’s scalability, robustness, and practical utility for enterprise-scale unstructured data reasoning.

This approach can be deployed in enterprise document management, legal research, financial analysis, and scientific data exploration, where complex multi-hop queries over large unstructured datasets are common. It enables organizations to build persistent, evolving knowledge bases that improve over time, reducing operational costs and response times. The system’s adaptability makes it suitable for real-time decision support, automated report generation, and intelligent data curation. Future integration with multimodal data sources could further extend its impact across diverse industries.

The method depends on accurate semantic inference; noisy or ambiguous data can impair structure quality. Computational overhead from continuous structure inference remains a concern, especially in very large datasets. The current focus on textual data limits immediate applicability to multimodal sources like images or videos. Further research is needed to enhance robustness, scalability, and multimodal integration, as well as to develop adaptive strategies for highly dynamic or noisy environments.