LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs

Recent advancements in LLMs like GPT-4 and Llama 3 have pushed the boundaries of long-text understanding, enabling applications in summarization, question answering, and reasoning over extensive documents. However, high-quality long-text training data remains limited due to the high cost of manual annotation and the computational expense of synthetic data generation. Existing methods such as random concatenation, relevance-based retrieval, and clustering approaches face trade-offs between efficiency, diversity, and semantic coherence. The need for scalable, resource-efficient data synthesis techniques has become urgent, especially as models grow larger and more capable but require proportionally richer training datasets. External knowledge systems like BISAC offer a promising avenue for structured content organization, but their integration into data synthesis pipelines is underexplored. Multi-agent debate mechanisms, inspired by human collaborative reasoning, have shown potential in generating diverse and high-quality content, yet their application in long-text data creation remains nascent.

The core challenge lies in balancing the quality, diversity, and resource efficiency of long-text data synthesis. Traditional relevance-based methods demand extensive embedding computations and dense clustering, incurring high GPU costs and limiting scalability. Random concatenation produces incoherent long documents, impairing model understanding. Existing approaches lack systematic content organization, leading to redundancy and coverage gaps. Therefore, developing a method that can generate diverse, coherent, and high-quality long-text samples with minimal computational overhead is critical. This is especially important for democratizing access to advanced long-context models, reducing reliance on expensive infrastructure, and enabling broader research and application in complex domains.

The paper introduces several key innovations: 1) External hierarchical content organization using BISAC, which ensures broad coverage and systematic structure; 2) Multi-agent debate mechanism where two LLMs generate and critique topics, enhancing diversity and quality; 3) Lightweight BM25 retrieval for document sampling, drastically reducing computational costs; 4) Seamless integration with long-dependency enhancement techniques like NExtLong, further improving long-range modeling. These innovations collectively address the limitations of prior relevance-based and clustering methods, providing a scalable, efficient, and effective framework for long-text data synthesis. The approach leverages external knowledge, collaborative AI reasoning, and efficient retrieval to produce high-quality datasets suitable for large-scale pretraining.

• �� Hierarchical Content Organization: Using BISAC categories, content is structured into broad and specific topics, facilitating coverage and navigation.
• �� Multi-Agent Debate: Two debate LLMs generate candidate topics independently, critique each other's outputs based on relevance and diversity, and a judge model filters high-quality topics.
• �� Document Retrieval: For each selected topic, BM25 retrieves up to 256 relevant documents from large corpora, ensuring relevance.
• �� Concatenation: Retrieved documents are concatenated into a 128K-token sample, avoiding expensive embedding-based retrieval.
• �� Integration with NExtLong: Selected documents are further processed with chunking and negative mining to model long-range dependencies.
• �� Evaluation: Performance is validated on HELMET and Ruler benchmarks, comparing scores and resource consumption.
• �� Resource Optimization: The entire pipeline minimizes GPU hours, making long-text data synthesis accessible and scalable.

The experimental setup involves fine-tuning LLaMA-3-8B on datasets like FineWeb-Edu and Cosmopedia V2. Baselines include random concatenation, KNN, ICLM, Quest, and NExtLong. Metrics such as recall, reranking, and long QA scores are used. Hyperparameters include a learning rate of 4e-5, batch size of 128K tokens, and 1000 training steps. Ablation studies assess the impact of BISAC categories and multi-agent debate. Results show LiteLong outperforms baselines with an average score of 61.90, requiring only 6 GPU hours for data generation, demonstrating high efficiency and effectiveness.

LiteLong achieves the highest average score of 61.90, surpassing Quest by over 6 points. Its resource efficiency is remarkable, with only 6 GPU hours needed compared to Quest’s 806. Combining with NExtLong further boosts performance to 63.04, especially in reranking and reasoning tasks. Ablation experiments confirm the importance of hierarchical content organization and multi-agent debate, with the topic scale and filtering strategies significantly affecting outcomes. These results validate the method’s ability to produce high-quality, diverse long-text data efficiently.

This approach is suitable for applications requiring extensive document understanding, such as legal analysis, scientific literature review, and educational content creation. It can generate diverse training data for large models, improving their performance on complex tasks. Additionally, it can be adapted to multimodal data and real-time content synthesis, broadening its industrial relevance and enabling smarter AI assistants, automated summarization, and knowledge extraction in various fields.

Dependence on BISAC limits applicability in niche or rapidly evolving domains, requiring dynamic classification updates. The multi-agent debate mechanism, while effective, may still face issues with redundancy or incomplete coverage in extremely long texts. Computational costs, although reduced, are not negligible, especially for very large corpora or multi-modal data. Future work should focus on adaptive classification, more efficient debate strategies, and multimodal integration to address these limitations.