Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
The study examines memory portability during model upgrades, finding fixed-schema knowledge graphs remain stable.
Key Findings
Methodology
The study designed an experimental framework comparing four memory formats during model upgrades: long-context raw history (LC-RAW), retrieval-augmented generation (RAG), compressed notes (NOTES), and fixed-schema knowledge graph (KG-fixed). Variables were isolated to analyze performance differences during migration.
Key Results
- KG-fixed showed only a +0.0004±0.0020 change in accuracy post-migration, indicating stability.
- NOTES exhibited significant directional accuracy shifts, up to +9.91 or -13.28 percentage points.
- In RAG systems, mixed embedding migration captured only a 4.96-point accuracy improvement, far less than the 11.90-point gain from full re-embedding.
Significance
The study reveals significant differences in migration performance across memory formats during model upgrades, highlighting the stability of fixed-schema structures. This is crucial for applications requiring frequent model upgrades, ensuring memory persistence and reliability.
Technical Contribution
The study provides a comparative analysis of four common memory formats under model migration conditions, revealing the advantages of fixed-schema knowledge graphs and identifying key factors in memory migration such as embedding space isolation and source history retention.
Novelty
This is the first systematic comparison of different memory formats during model upgrades, with a detailed analysis of the stability of fixed-schema knowledge graphs.
Limitations
- NOTES format shows instability during migration, heavily influenced by model capabilities.
- RAG format exhibits significant bottlenecks during retrieval, affecting overall performance.
Future Work
Future research could optimize the stability of the NOTES format, explore more complex retrieval mechanisms to enhance RAG performance, and study the impact of different model combinations on migration outcomes.
AI Executive Summary
In the AI field, model upgrades are common, but memory migration is often overlooked. This study examines the performance of four memory formats during model upgrades, focusing on the stability of fixed-schema knowledge graphs. While NOTES format shows instability in different migration directions, fixed-schema knowledge graphs perform exceptionally well, with accuracy remaining nearly unchanged.
Through experiments on 48 synthetic histories, the study reveals performance differences during migration. The RAG format exhibits significant bottlenecks during retrieval, and mixed embedding migration fails to fully leverage the new model's advantages. The study emphasizes the importance of maintaining embedding space isolation and source history retention during memory migration.
These findings are crucial for applications requiring frequent model upgrades, ensuring memory persistence and reliability. Future research could optimize the stability of the NOTES format, explore more complex retrieval mechanisms to enhance RAG performance, and study the impact of different model combinations on migration outcomes.
Deep Analysis
Background
With the advancement of AI technology, model upgrades have become routine. However, the performance of memory migration during model upgrades is rarely studied. How memory systems perform across different models is crucial for ensuring long-term system stability. Previous research focused mainly on memory performance within a single model, neglecting migration issues during model upgrades.
Core Problem
The core problem is how memory systems perform after model upgrades. Different models may interpret the same memory format differently, leading to migration failures. The study needs to clarify the performance differences of various memory formats post-upgrade to ensure system stability and reliability.
Innovation
The study systematically compares four common memory formats during model upgrades for the first time. By designing an experimental framework, isolating variables, and analyzing performance differences during migration, it particularly highlights the stability of fixed-schema knowledge graphs.
Methodology
- �� Design an experimental framework comparing four memory formats
- �� Use 48 synthetic histories for experiments
- �� Isolate variables to analyze performance differences during migration
- �� Record accuracy changes for each format
Experiments
The experiments use 48 synthetic histories with randomized answer codes to ensure models cannot answer based on pretraining knowledge. Two open-weight models with sub-10 billion parameters are used. Exact scoring analyzes performance differences during migration.
Results
Results show fixed-schema knowledge graphs maintain nearly unchanged accuracy post-migration, while NOTES format shows instability in different migration directions. RAG format exhibits significant bottlenecks during retrieval, affecting overall performance.
Applications
The findings are crucial for applications requiring frequent model upgrades, ensuring memory persistence and reliability. Suitable for intelligent agent systems requiring long-term memory preservation.
Limitations & Outlook
In the study, NOTES format shows instability during migration, heavily influenced by model capabilities. RAG format exhibits significant bottlenecks during retrieval, affecting overall performance. Future research could optimize NOTES format stability and explore more complex retrieval mechanisms.
Plain Language Accessible to non-experts
Imagine a library with various books on the shelves. Each time the library is upgraded, the arrangement of books might change, but the content should not be lost. The study explores how different book arrangements perform after library upgrades. Fixed-schema arrangements, like categorizing by genre, remain stable after upgrades, while other arrangements might make books harder to find.
ELI14 Explained like you're 14
Imagine you're playing a game, and every time it upgrades, the rules might change. The study is like analyzing how different game rules perform after upgrades. Some rules remain effective after upgrades, while others need to be relearned. The study helps us understand which rules remain reliable after upgrades.
Glossary
Model Upgrade
Refers to replacing or updating models in AI systems to improve performance.
The study explores the impact of model upgrades on memory migration.
Memory Migration
The process of transferring memory systems from an old model to a new one during model upgrades.
The study compares the performance of different memory formats during migration.
Fixed-Schema Knowledge Graph
A graph that represents knowledge using a fixed structure, typically storing information in subject-predicate-object form.
The study finds fixed-schema knowledge graphs stable during migration.
Retrieval-Augmented Generation
A method that enhances generation model performance by retrieving relevant information.
In the study, RAG format exhibits bottlenecks during retrieval.
Compressed Notes
A format that compresses historical information into natural language notes.
The study finds NOTES format unstable during migration.
Open Questions Unanswered questions from this research
- 1 How to improve the stability of NOTES format during migration?
- 2 How to overcome retrieval bottlenecks in RAG format?
Applications
Immediate Applications
Smart Assistants
Applicable in smart assistants to ensure user preferences and history are maintained after model upgrades.
Long-term Vision
Long-Term Memory Systems
Develop systems capable of maintaining stability during frequent upgrades, suitable for various intelligent applications.
Abstract
Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ($0.467 \pm 0.014$) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ($0.364 \pm 0.012$) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.