Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Introduces self-patching to diagnose and recover 58-75% of reasoning failure caused by knowledge-circuit misalignment in fine-tuned LLMs.
Key Findings
Methodology
The study fine-tunes large pre-trained models like GPT-3 and LLaMA with unseen knowledge, then employs self-patching to monitor how injected knowledge permeates the internal layers over time. This involves layer-wise activation relocalization, identifying positions where representations can unlock correct reasoning. Using datasets from the STaRK knowledge base, the authors analyze the temporal lag and accuracy gap between memorization and downstream reasoning. Causal interventions, including relocating representations, validate the hypothesis that knowledge and reasoning circuits are spatially misaligned, and manual relocation significantly improves downstream performance, recovering 58-75% of potential gains.
Key Results
- Across multiple models and knowledge domains, memorization accuracy approaches 100%, but downstream reasoning accuracy remains below 70%, demonstrating a clear Knowing–Using Gap. Self-patching restores 58-75% of the potential performance in failure cases, confirming the spatial misalignment hypothesis. Experiments show that relocating knowledge representations to middle layers boosts multi-hop reasoning success rates substantially, indicating that storage and computation pathways are misaligned spatially.
- Layer-to-layer mapping reveals that early and late layers store knowledge, but effective reasoning requires representations in middle layers. Moving representations into these layers via self-patching yields large performance improvements, validated across models of different sizes and knowledge domains. The heuristic strategies based on patch locations outperform prompting baselines like Chain-of-Thought, achieving over 50% performance recovery.
- Causal experiments demonstrate that manually relocating stored knowledge representations causally enhances reasoning, confirming the mechanistic hypothesis. The findings suggest that the primary bottleneck is not knowledge absence but the spatial misrouting of representations, opening new avenues for internal model editing and interpretability.
Significance
This work advances understanding of the internal mechanisms limiting knowledge transfer in large models. By diagnosing the spatial misalignment between stored knowledge and reasoning pathways, it offers practical intervention strategies to improve reasoning capabilities without retraining. The insights bridge the gap between model interpretability and performance, enabling more controllable and explainable AI systems. The discovery of knowledge-circuit misalignment provides a new conceptual framework for future research on internal knowledge dynamics, model editing, and generalization. It addresses a fundamental challenge in AI: how to make models not only memorize facts but also reason with them reliably, which is crucial for deploying trustworthy AI in real-world applications.
Technical Contribution
The paper introduces self-patching, a causal intervention method that maps the permeation of injected knowledge across model layers. It combines layer-wise activation relocalization with causal effect measurement, revealing the spatial misalignment between knowledge storage and reasoning circuits. The authors validate the hypothesis through extensive experiments, demonstrating that manual relocation of representations significantly improves downstream reasoning performance. They also propose heuristic strategies based on patch locations, recovering up to 75% of potential reasoning capacity, thus providing a practical tool for model editing and interpretability. This approach extends mechanistic interpretability by focusing on the internal spatial dynamics of knowledge and reasoning pathways.
Novelty
This research is the first to systematically diagnose the spatial misalignment between knowledge storage and reasoning circuits in large language models during fine-tuning. It combines causal intervention techniques with spatial mapping, revealing that knowledge representations are often stranded in non-effective layers. Unlike prior work focusing solely on knowledge editing or parameter updates, this study emphasizes the importance of internal spatial routing. The self-patching method and the causal validation of the knowledge-circuit misalignment constitute a significant innovation, providing both theoretical insights and practical tools for improving model reasoning.
Limitations
- The methodology relies on manual intervention and layer-wise relocalization, which may not scale efficiently to very large models or real-time applications. Automating the relocation process remains a challenge.
- The experiments are primarily conducted on models up to 13B parameters and specific knowledge bases; applicability to extremely large models like GPT-4 or multimodal models needs further validation.
- The approach assumes that representations can be effectively relocated without loss, which might not hold in more complex or noisy scenarios. Future work should address robustness and automation.
Future Work
Future research will focus on developing automated algorithms for knowledge relocalization, possibly leveraging reinforcement learning or meta-learning. Extending the analysis to multimodal models and more complex reasoning tasks will be crucial. Additionally, integrating the spatial mapping with model pruning or architecture design could optimize the internal routing of knowledge. The ultimate goal is to enable models that not only memorize facts but also dynamically route and utilize knowledge internally, enhancing reasoning, interpretability, and controllability in AI systems.
AI Executive Summary
Despite remarkable progress, large language models (LLMs) still face a fundamental challenge in effectively utilizing memorized knowledge for reasoning tasks. While models like GPT-3 and LLaMA can memorize facts with near-perfect accuracy after fine-tuning, their ability to apply this knowledge in multi-hop reasoning remains limited, exhibiting a significant accuracy gap and a temporal lag. This phenomenon, termed the Knowing–Using Gap, indicates that stored knowledge does not naturally translate into usable reasoning pathways.
To address this, the authors introduce self-patching, a causal intervention technique that maps how injected knowledge permeates through the model’s internal layers over time. By relocalizing representations from storage layers into computation-effective layers, they demonstrate that the core issue is a spatial misalignment between where knowledge is stored and where it is used for reasoning. Extensive experiments across multiple models, knowledge domains, and reasoning tasks confirm that manually relocating representations can recover 58-75% of the potential reasoning performance.
This mechanistic insight reveals that the failure is not due to a lack of knowledge but to the internal routing of information. The findings suggest that models can be artificially activated to utilize stored facts more effectively, paving the way for improved model editing, interpretability, and reasoning robustness. Although the approach currently relies on manual intervention, it opens promising avenues for automating internal knowledge routing, ultimately enhancing the generalization and trustworthiness of large language models.
Deep Dive
Abstract
Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the \textit{\textbf{Knowing--Using Gap}}, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58--75\% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding.