Set the Clock: Temporal Alignment of Pretrained Language Models

The rapid development of large-scale pretraining models such as BERT, GPT-3, and LLaMa has significantly advanced NLP capabilities. These models learn from vast, diverse datasets, capturing extensive factual knowledge. However, their static training data leads to knowledge staleness, especially for time-sensitive facts. Prior efforts have focused on knowledge updating via continual learning, retrieval augmentation, and knowledge editing, but these approaches often treat knowledge as static and do not address the internal organization of temporal information. Recent studies highlight that models tend to answer questions based on outdated knowledge, revealing a chaotic internal temporal structure. Understanding and controlling this internal organization is critical for deploying models in real-world, dynamic environments where facts evolve rapidly.

The core challenge is that pretrained LLMs encode a chaotic and often outdated internal representation of temporal knowledge. Despite recent training, models tend to answer questions with knowledge from earlier years, leading to inaccuracies. Existing solutions like retrieval-augmented generation or post-hoc knowledge editing are limited by external dependencies and do not fundamentally alter internal temporal organization. The problem is compounded by the lack of explicit temporal grounding during training, making models insensitive to the temporal context of questions. This results in poor performance on time-sensitive tasks, limiting their practical utility in domains like news, finance, and healthcare where timely information is crucial.

The paper introduces a comprehensive framework for temporal alignment, combining dataset construction, prompting, and fine-tuning strategies. The key innovations include:

• �� TAQA dataset: a large-scale, automatically generated dataset with questions spanning 2000-2023, enabling systematic probing of temporal knowledge.
• �� Time-aware prompting: appending explicit year information to questions to activate relevant knowledge.
• �� Target-year fine-tuning: selecting year-specific training samples based on answer correctness to reorient model internal knowledge.
• �� Adaptive fine-tuning: dynamically assigning each question to its most relevant year, allowing models to learn flexible temporal representations.

These innovations collectively enable models to reorganize their internal knowledge, improving temporal accuracy without retraining from scratch.

• �� Data collection: Extract Wikipedia tables with temporal columns, generate questions via GPT-4 prompts, and curate QA pairs with multiple answers over time.
• �� Dataset construction: Filter questions with at least five distinct answers across 2000-2023, ensuring high temporal sensitivity.
• �� Prompt design: Append ‘as of year y, the answer is’ to questions, and sample few-shot examples to activate temporal knowledge.
• �� Fine-tuning: Select training samples where the model’s answers match ground truth, then optimize model parameters on these.
• �� Adaptive fine-tuning: For each question, iterate over years from 2022 backward, select the most relevant year based on answer correctness, and fine-tune the model to output the corresponding temporal context.
• �� Evaluation: Measure F1 scores across years, compare with baselines, and analyze data selection strategies and model scale effects.

Experiments involved fine-tuning LLaMa2 models (7B, 13B, 70B) on the TAQA dataset, targeting 2021 and 2022. Baselines included unaligned models and models with simple prompts. Metrics focused on F1 scores for answers in specific years, as well as maximum and decayed F1 scores to evaluate temporal robustness. Data selection strategies (correctness-based, random, popularity) were compared. Results showed that targeted fine-tuning consistently outperformed prompting, with performance gains up to 62%. The adaptive approach further improved historical year alignment, achieving 2.8× better results for 2010. Model scale analysis indicated larger models benefited more from alignment strategies.

Fine-tuning LLaMa2 to 2022 improved F1 scores by up to 62%, significantly surpassing prompt-only methods. Aligning to 2010 yielded a 2.8× performance boost, demonstrating effective knowledge transfer across time. Correctness-guided data selection proved superior to random or popularity-based sampling, confirming that internal knowledge activation, rather than memorization, drives improvements. Larger models (70B) exhibited more pronounced gains, highlighting the importance of model capacity for temporal adaptation. These results validate the hypothesis that internal knowledge organization is modifiable via targeted training.

The methods enable deployment of temporally aware NLP systems in real-time information retrieval, news summarization, and financial analysis, where understanding the correct temporal context is essential. They also facilitate continuous knowledge updates without retraining, reducing costs and improving responsiveness. Long-term, this approach supports building AI that can adapt to evolving facts, improving trustworthiness and relevance in dynamic domains like healthcare, legal, and scientific research.

The approach relies heavily on high-quality, temporally annotated datasets, which are costly to produce and may not cover all domains. Fine-tuning requires substantial computational resources, limiting scalability. The models still struggle with very recent or rapidly changing information, indicating the need for real-time data integration. Moreover, the method’s effectiveness diminishes for less-structured data or sparse temporal signals, and further research is needed to generalize across languages and knowledge sources.