SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

Large language models (LLMs) have made significant strides in the field of natural language processing, capable of learning broad world knowledge and general capabilities from massive web text. However, in specialized domains, particularly those with scarce data, the knowledge coverage of LLMs remains incomplete. To address this gap, researchers have attempted to enhance models' domain knowledge through knowledge injection. Knowledge injection typically involves further fine-tuning or continual pretraining using domain-specific data. However, these domain-specific datasets are often limited in scale and diversity, and directly fine-tuning LLMs on such sparse data often leads to overfitting to specific surface forms rather than robust knowledge acquisition.

In the field of knowledge injection, existing methods face two major challenges: first, reinforcement learning-based methods, while improving token efficiency in small-scale data generation, often suffer from diversity collapse as data scales, leading to diminishing returns. Second, multi-stage prompting methods, although superior to simple augmentation methods in some cases, may lose their advantages after careful prompt tuning. These issues limit the effectiveness of existing methods in large-scale data generation.

The core innovations of the SPA method lie in its simple yet effective prompt design and large-scale data generation strategy. Specifically:

1. Prompt Design: SPA designs seven prompt templates based on learning strategies from cognitive science and educational psychology, including Concept Learning, Critical Thinking, and Generative Learning. These prompt templates help the generator produce more diverse and comprehensive synthetic data.

2. Large-scale Data Generation: By repeatedly prompting a large language model to rewrite the source content, SPA generates a large-scale synthetic corpus for knowledge injection.

3. Single-stage Prompting: Unlike complex multi-stage prompting methods, SPA generates high-quality synthetic data through a single-stage prompt, simplifying system complexity.

The implementation of the SPA method involves the following key steps:

• �� Prompt Design: Design seven prompt templates based on learning strategies from cognitive science and educational psychology, including Concept Learning, Critical Thinking, and Generative Learning.
• �� Data Generation: Use the designed prompt templates to repeatedly prompt a large language model to rewrite the source content, generating a large-scale synthetic corpus.
• �� Model Training: Train the target model on the generated synthetic corpus to enhance its domain knowledge.
• �� Performance Evaluation: Evaluate SPA's performance on multiple benchmarks to verify its effectiveness in knowledge injection tasks.

The experimental design includes the following aspects:

• �� Datasets: Select SQuAD, QuALITY, and MultiHop-RAG as benchmark datasets.
• �� Baseline Methods: Choose complex methods such as SEAL and Active Reading as comparison baselines.
• �� Evaluation Metrics: Use accuracy as the main evaluation metric to assess SPA's performance on different datasets.
• �� Hyperparameter Settings: Match the number of training tokens for all methods to ensure fair comparisons.
• �� Ablation Studies: Analyze SPA's performance changes by progressively scaling the synthetic corpus.

In experiments, SPA demonstrated superior performance across all benchmarks, surpassing various complex methods such as SEAL and Active Reading. Specifically:

• �� On the SQuAD dataset, SPA achieved 91.27% accuracy, surpassing Active Reading's 90.25% and SEAL's 74.23%.
• �� On the QuALITY dataset, SPA achieved 57.03% accuracy, outperforming EntiGraph's 56.22% and Active Reading's 51.13%.
• �� On the MultiHop-RAG dataset, SPA achieved 86.64% on Qwen2.5-7B and 88.36% on Meta-Llama-3-8B, surpassing all baselines.

The SPA method has broad application potential in various fields:

• �� Domain-specific Knowledge Injection: SPA can be used for knowledge injection in specialized fields such as medicine, finance, and law, enhancing the model's domain knowledge coverage by generating large-scale synthetic data.
• �� Data-scarce Scenarios: In data-scarce scenarios, SPA provides a cost-effective and efficient pathway for knowledge injection, helping models better understand and answer domain-related questions.
• �� Real-time Knowledge Updates: SPA can be applied in real-time knowledge update scenarios, generating new synthetic data to help models quickly adapt to the latest domain knowledge.

While SPA excels in large-scale data generation, its flexibility in generator selection may be limited, especially when using weaker generators. Additionally, SPA relies on the quality of prompt templates, which may require adjustment for specific knowledge injection needs in different domains. Future research can explore further optimization of SPA's prompt design to enhance its applicability across different domains.