Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

Recent years have witnessed rapid growth in large language models (LLMs) such as GPT, BERT, and LLaMA, which have revolutionized NLP tasks. These models are typically trained on static datasets, making their knowledge quickly outdated as new information emerges. To address this, knowledge editing (KE) techniques have been developed, allowing targeted updates to model parameters to incorporate new facts without retraining from scratch. Early approaches focused on structured knowledge triples, updating specific object entities. However, real-world knowledge is often unstructured, expressed in free-form passages, leading to the rise of unstructured KE (UKE). Despite progress, existing methods struggle with knowledge decomposition—answering atomic questions about injected facts—and composition—combining facts for multi-hop reasoning. These limitations hinder the practical deployment of KE in dynamic environments.

The core challenge in unstructured KE is enabling models not just to memorize injected passages but to understand and utilize individual facts flexibly. Current methods often rely on passive memorization, which results in poor decomposition and composition capabilities. This leads to models that can recall passages verbatim but cannot answer targeted atomic questions or perform multi-hop reasoning involving newly injected facts. The problem is compounded by the fact that injected knowledge, being novel, is rarely covered in the model’s own rollout trajectories, limiting the effectiveness of pure on-policy distillation. Overcoming these issues requires a proactive approach that guides models to actively incorporate and utilize new knowledge, rather than passively memorizing it.

This work introduces HPSE, which innovatively combines a hybrid rollout strategy with self-distillation. Key innovations include:

• �� Constructing hybrid trajectories that mix the privileged model’s context-aware predictions with the student’s own generation, triggered adaptively when divergence exceeds thresholds.
• �� Enabling proactive, fact-directed supervision that guides the model to incorporate missing facts precisely where coverage fails.
• �� Providing a theoretical guarantee that the coverage advantage scales linearly with the length of injected facts, especially effective for long, entangled passages.
• �� Maintaining a plug-and-play training objective that does not depend on external supervision or parameter-specific updates, ensuring broad applicability across gradient-based KE methods.

• �� Initialize student model πθ and privileged model π⋆ reading passage c.
• �� During training, generate hybrid rollout y by switching tokens between π⋆ and πθ based on divergence thresholds τ and κ.
• �� Trigger step-in when the privileged model’s confidence significantly exceeds the student’s, correcting off-topic or missing facts.
• �� Minimize a combined loss function: the divergence between hybrid and privileged distributions plus passage-level likelihood regularization.
• �� Repeat over multiple rounds, gradually improving the model’s coverage and ability to utilize injected knowledge proactively.

Experiments involve four models—Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B, Gemma-2-9B—and two benchmarks: UnKEBench and MQuAKE-uns, targeting knowledge decomposition and composition. Baselines include MEMIT, AlphaEdit, and others. Hyperparameters τ and κ are tuned for optimal step-in triggering. The evaluation metrics include joint and fact-level accuracy, multi-hop reasoning, and locality scores. Results show consistent improvements, with average gains of 6.8 points in recall and maximum improvements up to 67.9%. Ablation studies confirm the importance of hybrid rollout and step-in mechanisms.

HPSE outperforms existing KE methods across all models and benchmarks, notably improving atomic fact recall and multi-hop reasoning. It significantly reduces coverage failure, enabling better decomposition and composition of injected facts. The method maintains stability in multi-round editing, demonstrating robustness. Quantitative results show an average increase of 6.8 points in knowledge recall accuracy, with some cases exceeding 67%. The ablation confirms that hybrid rollout and adaptive step-in are critical for performance gains. These results validate HPSE’s effectiveness in real-world knowledge update scenarios.

HPSE can be integrated into question answering systems, knowledge base maintenance, and intelligent assistants requiring real-time knowledge updates. Its plug-and-play design allows seamless adoption in existing frameworks, improving factual accuracy and reasoning capabilities. Long-term, it can facilitate continual learning in AI, enabling models to adapt dynamically to new information, supporting applications like personalized assistants, dynamic knowledge graphs, and real-time information retrieval.

Despite its strengths, HPSE may encounter frequent step-in triggers in highly complex scenarios, increasing computational costs. Its reliance on the privileged model’s accuracy can introduce biases if the privileged state is flawed. Scalability to extremely large models or multimodal data remains to be validated. Additionally, hyperparameter tuning for thresholds τ and κ is necessary, which may limit ease of deployment. Future work should focus on adaptive threshold mechanisms and extending to multimodal knowledge sources.