Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning

Knowledge graphs (KGs) have evolved from static representations to dynamic, temporal structures (TKGs), capturing evolving relationships over time. Early works like TransE and Know-Evolve focused on embedding entities and relations, but struggled with capturing complex temporal patterns and requiring large labeled datasets. Recent advances include graph neural networks (RE-GCN, xERTE) that model structural dependencies, yet they depend heavily on explicit features and supervised training. Meanwhile, large language models (LLMs) like GPT-3 have demonstrated emergent reasoning abilities, prompting investigations into their applicability for knowledge inference. This paper situates itself at this intersection, exploring how pre-trained LLMs can perform TKG forecasting without task-specific fine-tuning, leveraging their pattern recognition capabilities to address limitations of prior models.

Existing TKG forecasting models rely on supervised learning, structural assumptions, and extensive labeled data, which hinder scalability and adaptability. They often require dataset-specific architectures and struggle with generalization in sparse or evolving environments. The core challenge is to develop a method that can perform accurate future event prediction in a zero-shot manner, utilizing the implicit knowledge embedded in pre-trained models. This involves designing prompts that effectively encode historical information and extracting predictions solely from token probabilities, bypassing the need for explicit structural modeling or additional training. Addressing this challenge could significantly simplify deployment and improve robustness across diverse scenarios.

The paper introduces a novel zero-shot forecasting framework based on in-context learning, where large language models process structured prompts derived from historical facts. Key innovations include: 1) employing multiple fact filtering strategies (entity, relation, unidirectional, bidirectional) to tailor context; 2) designing prompt templates using entity indices or lexical forms; 3) demonstrating that semantic information is non-essential by replacing names with random indices; 4) systematically analyzing the impact of historical length, model size, and prompt design. These innovations enable models to learn complex temporal and structural patterns solely from context, reducing reliance on explicit graph structures and supervised data, thus broadening the applicability of LLMs for knowledge inference.

• �� Select relevant historical facts from previous graph snapshots based on query type and filtering strategy. • Convert facts into structured prompts using either index-based or lexical templates, encoding entity and relation information. • Append the query to the prompt, forming a context for the model. • Use the pre-trained LLM to generate token probabilities for candidate entities, ranking them accordingly. • For multi-step prediction, incorporate previous predictions as additional context, iterating the process. • Experiment with different prompt formats, historical lengths, and model scales to evaluate performance. • Employ time-aware filtering to exclude known valid answers during evaluation, ensuring fair ranking. • Analyze the influence of semantic content by replacing names with random indices, assessing reliance on semantic priors.

The experimental setup involves multiple datasets—WIKI, YAGO, ICEWS14, ICEWS18, and a newly constructed ACLED dataset—covering diverse temporal and structural characteristics. Baselines include supervised models (RE-GCN, xERTE) and heuristic heuristics (frequency, recency). Metrics such as Hits@1, 3, 10 assess prediction accuracy. Models tested include GPT-J, GPT-NeoX, and GPT-3.5-turbo, with parameters ranging from 124M to 20B. Experiments vary historical length, prompt design, and model size, with multiple runs to ensure robustness. Evaluation employs time-aware filtering to prevent penalizing valid predictions, and ablation studies analyze the effects of prompt strategies and semantic content. Results demonstrate that large, pre-trained models can perform competitively in a zero-shot setting, with performance improving with larger context and models.

Pre-trained models like GPT-NeoX achieve Hits@1 scores within ±3.6% of supervised SOTA models across datasets, confirming strong zero-shot capabilities. Replacing entity/relation names with random indices causes performance drops of only about 0.4%, indicating semantic content is not critical. Increasing historical context length and model scale consistently enhances accuracy, validating the scalability principle. The models outperform simple heuristics (frequency, recency) by significant margins (+10% to +28%), demonstrating their ability to learn complex, irregular patterns beyond straightforward temporal cues. These findings underscore the potential of large models to generalize temporal reasoning without task-specific training.

This approach is applicable in real-time event prediction, dynamic knowledge inference, and intelligent decision-making systems where annotated data is scarce or rapidly changing. It simplifies deployment by removing the need for dataset-specific training, making it suitable for industries like finance, security, and social media analysis. The method can be integrated into existing knowledge management platforms, providing adaptive, scalable forecasting capabilities. Long-term, it paves the way for autonomous reasoning systems that leverage pre-trained models for diverse knowledge inference tasks, reducing reliance on handcrafted features and extensive labeled datasets.

Despite promising results, the approach struggles with highly sparse or noisy data where patterns are less explicit. Large models incur high computational costs, limiting practical deployment in resource-constrained environments. The lack of interpretability hampers understanding of specific predictions, raising trust issues. Furthermore, the method's reliance on historical data quality means that biased or incomplete data can lead to inaccurate forecasts. Future work should focus on improving model efficiency, interpretability, and robustness, especially in complex, real-world scenarios where data quality varies.