A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting
Using MidEast-TE-mini, the study compares graph, text, and mixed LLM forecasters; GPT-4 extraction reaches 72.60%, while tuning and RAG help most.
Key Findings
Methodology
The paper builds MidEast-TE-mini and evaluates graph-only, text-only, and graph-text LLM forecasters. Histories are constructed either by rules—query-subject, complex-event, and temporal windows—or by retrieval: an LLM filters candidate entities, while embedding similarity retrieves news chunks. Summaries then compress evidence before an LLM predicts a missing object or relation.
Key Results
- The benchmark contains 12,542 atomic events, 120 complex events, 3,692 documents, 5,909 entities, and 267 relations. Human evaluation reports 72.60% extraction accuracy, versus 55.56% for MidEast-TE and 29.73% for GDELT-TE.
- Directly adding raw text does not improve zero-shot extrapolation, whereas fine-tuning with raw text substantially improves performance. Retrieval modules recover temporal-relational patterns hidden in history, but RAG remains vulnerable to popularity bias and long-tail entities.
- On the authors' error analysis, time, relation, and entity errors account for 16.67%, 27.78%, and 55.56%, respectively. Entity linking is therefore a major reliability bottleneck even before forecasting begins.
Significance
The study places textual understanding and temporal knowledge-graph forecasting in one evaluation framework. It shows that broad LLM world knowledge is not equivalent to reliable temporal extrapolation: context selection, extraction quality, training, and retrieval must be designed jointly. This matters for conflict early warning, diplomatic monitoring, and explainable intelligence systems, where unsupported predictions and hallucinated evidence are costly.
Technical Contribution
The technical contribution is a GPT-4-based, hierarchical CAMEO extraction pipeline with K-means-assisted entity linking, plus a controlled comparison of rule-based and retrieved histories. Graph retrieval uses LLM-selected relevant entities; text retrieval uses embedding similarity, temporal ordering, filtering, and summarization; mixed retrieval lets graph structure guide news selection. The framework exposes both RAG gains and its bias mechanisms.
Novelty
Relative to work focused mainly on discrete graph reasoning, this paper systematically evaluates how raw news participates in LLM temporal forecasting. Its exploratory benchmark jointly contains documents, extracted events, and complex-event clusters. The novelty is therefore an integrated evaluation of representation, extraction, history construction, retrieval, summarization, and popularity bias—not merely attaching text to an existing graph model.
Limitations
- MidEast-TE-mini contains only 120 Middle Eastern complex events with durations restricted to 40–60 days. GPT-4 extraction is better than prior datasets but still reaches only 72.60% accuracy, so label and retrieval noise propagate into forecasting.
- The supplied text does not include complete Hit@K, MRR, significance tests, or inference-cost tables. Thus it supports qualitative conclusions about zero-shot text, fine-tuning, and RAG, but not a precise numerical ranking of forecasting systems.
- Cross-region, cross-language, and cross-domain generalization is not tested.
Future Work
Future work should improve event extraction and entity linking, expand duration and geographic coverage, and develop popularity-aware retrieval, long-tail calibration, noise-robust RAG, and evidence-grounded explanations. Standard reporting should include Hit@K, MRR, calibration, cost, and human faithfulness evaluations.
AI Executive Summary
Forecasting future events is central to conflict early warning and international-affairs analysis, yet existing systems split the problem. Temporal knowledge graphs encode who did what and when, but discard much of the explanatory detail in news. Text-based systems preserve context but struggle with entity consistency, temporal ordering, and long histories. LLMs appear well suited to bridge the gap, but their ability to extrapolate events rather than recall knowledge has been under-tested.
Chang and colleagues construct MidEast-TE-mini, combining news documents, atomic events, and complex-event clusters. The dataset contains 12,542 atomic events, 120 complex events, and 3,692 documents, split chronologically into train, validation, and test periods. The study compares graph-only, text-only, and graph-text inputs. It also contrasts rule-based history with retrieved history: the latter uses LLM entity filtering or embedding-based news retrieval, followed by temporal sorting and summarization.
The findings are deliberately nuanced. Raw text inserted directly into zero-shot prompts does not improve extrapolation, whereas fine-tuning on raw text substantially helps. RAG can expose temporal-relational patterns hidden in historical events, but it does not remove popularity bias or long-tail failures. GPT-4’s dataset extraction reaches 72.60% human-judged accuracy, compared with 55.56% for MidEast-TE and 29.73% for GDELT-TE. The next advances therefore require cleaner extraction, fairer retrieval, stronger rare-entity reasoning, and traceable evidence—not simply larger models.
Deep Analysis
Background
Temporal event forecasting commonly represents an event as (s,r,o,t) and models evolution with a Temporal Knowledge Graph. GDELT and ICEWS support graph reasoning, while ForecastQA and event-script prediction retain natural-language context. GNNs, Temporal Logical Rules, and early LLM methods capture parts of temporal structure, but noisy extraction, hallucination, long context, entity popularity, and the separation of graph and text remain open problems.
Core Problem
Given historical graphs G<t or documents D<t and a query (s,r/o,t), the system must predict a missing object or relation. The task is difficult because one article may express multiple events, extraction may be wrong, relevant context may be distant or lengthy, and future entities may be rare or unseen. The paper explicitly studies representation choice, RAG behavior, and popularity bias.
Innovation
- ��Creates MidEast-TE-mini by sampling 120 complex events lasting 40–60 days. •Uses GPT-4 with the three-level CAMEO hierarchy for extraction and K-means-assisted entity linking. •Unifies graph-only, text-only, and mixed LLM baselines. •Introduces rule-based and retrieved histories, including LLM entity filtering, embedding retrieval, temporal filtering, and LLM summarization. •Makes long-tail and popularity effects first-class evaluation questions.
Methodology
- ��Representation: each atomic event is (s,r,o,t,c), where c identifies a complex event; all events at timestamp t form Gt. •Extraction: news is split into roughly 150-token chunks; GPT-4 performs hierarchical CAMEO extraction, then entities are clustered with K-means and linked in batches. •Graph history: retain recent query-subject events and events from the prior two days in the same complex event, or retrieve relevant entities with an LLM. •Text history: collect linked documents, or chunk them and retrieve by embedding similarity to the query subject; sort, filter stale items, and summarize. •Forecasting: prompt an LLM with definitions, query, candidate options, and nearest, further, related, or relevant history.
Experiments
The chronological split contains 8,999 training events, 1,777 validation events, and 1,766 test events; corresponding complex-event counts are 88, 19, and 18, and document counts are 2,647, 473, and 572. The full dataset has 12,542 events, 120 complex events, 5,909 entities, 267 relations, and 3,692 documents. The study compares graph-only, text-only, mixed, rule-history, and retrieved-history variants, with human extraction evaluation. The supplied text does not report a complete forecasting-score table.
Results
GPT-4 extraction achieves 72.60% accuracy, above MidEast-TE’s 55.56% and GDELT-TE’s 29.73%. Within the new dataset’s errors, entities contribute 55.56%, relations 27.78%, and time 16.67%. Direct raw-text prompting fails to improve zero-shot extrapolation; raw-text fine-tuning helps substantially. RAG uncovers temporal patterns but retains popularity and long-tail weaknesses.
Applications
Potential uses include regional-conflict early warning, diplomatic interaction monitoring, news-event evolution tracking, and risk-intelligence triage. Deployment should preserve source articles, expose retrieved evidence, require human review, and separately audit rare entities and extraction quality. The system is best treated as a ranked lead generator, not an autonomous decision maker.
Limitations & Outlook
The benchmark is small and region-specific, with only 120 complex events and a 40–60-day duration filter. GPT-4 extraction still has 72.60% accuracy, and entity errors dominate. The provided paper text omits full forecasting metrics, compute costs, and significance tests, limiting quantitative comparison. Future work should broaden regions and languages, use debiased retrieval and uncertainty calibration, and evaluate explanation faithfulness.
Plain Language Accessible to non-experts
Imagine a newsroom trying to predict the next headline in a developing story. It has two filing cabinets. One contains tidy cards saying who did what, to whom, and when. The other contains full articles with motives, background, and nuance. The paper asks whether an AI can use either cabinet, or both, to fill in the next missing piece of the story.
The AI can follow a fixed filing rule, such as looking at the same person or the latest two days. Or it can search for the most relevant old cards and articles, then shorten the articles before reading them. The surprising result is that dumping every article into the AI does not automatically make it wiser. Training helps it use text, and searching can reveal links across time, but the system still favors famous people and countries.
The benchmark contains 12,542 events, and GPT-4 organizes them with 72.60% judged accuracy. That is useful, but not perfect: many mistakes come from identifying the wrong person or organization. The central lesson is simple—good prediction needs clean records, fair searching, and visible evidence, not just a powerful reader.
ELI14 Explained like you're 14
Think of this as a game where you must predict the next plot twist. Each news story gives clues: who acted, what they did, who was affected, and when it happened. The researchers ask a large language model to guess the missing character or event. First, GPT-4 turns messy news into event cards so the model can study the story’s timeline.
The model gets three possible play styles: cards only, articles only, or both together. It can also search older clues, like looking through a game wiki for events involving the same character. Long articles are shortened into summaries so the model does not run out of reading space. Here is the twist: throwing all articles into the prompt does not magically improve zero-shot guessing. Training makes the model better at using the words, and retrieval helps it connect earlier and later events.
But the model has a “main-character problem.” Famous countries and people appear often, so the model keeps choosing them. Rare characters are treated like hidden NPCs and are missed. The dataset itself also contains mistakes: GPT-4’s extraction accuracy is 72.60%, and entity mistakes are the largest category. So the coolest future version would not only guess the next event; it would show the exact clues that justify the guess!
Glossary
Temporal Event Forecasting
Predicting a future missing object, relation, or event from observations available before time t. Technically, it combines temporal ordering with relational prediction.
The paper’s central task.
Temporal Knowledge Graph
A graph whose facts carry timestamps; here an atomic event is (s,r,o,t,c). It represents structured temporal and relational history.
Input to graph-only and mixed methods.
Retrieval-Augmented Generation
A pipeline that retrieves relevant historical evidence before an LLM generates a forecast. Retrieval can improve grounding but can also amplify frequency bias.
The retrieved-history variants.
CAMEO ontology
A hierarchical ontology for international event types. The paper uses its three levels to reduce the cost of prompting GPT-4 over more than 200 event types.
Hierarchical event extraction.
Complex Event
A temporally extended process composed of correlated atomic events involving multiple entities. The Israeli–Palestinian conflict is the motivating example.
Clustering and local-history construction.
Popularity Bias
A tendency to predict frequently observed entities or relations instead of rare ones. It can arise from both historical data frequency and retrieval behavior.
A central failure mode analyzed by the paper.
Open Questions Unanswered questions from this research
- 1 The supplied text does not provide complete Hit@K, MRR, significance, or cost tables, so the exact statistical superiority of individual forecasting variants remains unresolved.
- 2 Extraction and forecasting are treated as linked stages, but robust joint learning, uncertainty propagation, and calibrated confidence for noisy events remain open technical problems.
- 3 Generalization beyond English Middle Eastern news, including other regions, languages, and crisis types, requires larger and more diverse benchmarks.
Applications
Immediate Applications
Conflict-news monitoring
Analysts can convert incoming reports into timestamped event chains, retrieve related history with RAG, and generate candidate warnings. Production systems should preserve source evidence, require human review, and audit rare-entity recall rather than trusting model outputs blindly.
Diplomatic-interaction tracking
Research teams can monitor statements, aid, military actions, and cooperation by country or organization. The system can summarize evolving patterns and surface leads, but experts should verify extraction, source reliability, and causal interpretation before publication or policy use.
Long-term Vision
Explainable global risk radar
With multilingual coverage, debiased retrieval, calibrated probabilities, and linked evidence, such systems could continuously map emerging crises across regions. Obstacles include data rights, ontology consistency, extraction noise, adversarial reporting, and the need for credible human evaluation.
Abstract
Recently, Large Language Models (LLMs) have demonstrated great potential in various data mining tasks, such as knowledge question answering, mathematical reasoning, and commonsense reasoning. However, the reasoning capability of LLMs on temporal event forecasting has been under-explored. To systematically investigate their abilities in temporal event forecasting, we conduct a comprehensive evaluation of LLM-based methods for temporal event forecasting. Due to the lack of a high-quality dataset that involves both graph and textual data, we first construct a benchmark dataset, named MidEast-TE-mini. Based on this dataset, we design a series of baseline methods, characterized by various input formats and retrieval augmented generation (RAG) modules. From extensive experiments, we find that directly integrating raw texts into the input of LLMs does not enhance zero-shot extrapolation performance. In contrast, fine-tuning LLMs with raw texts can significantly improve performance. Additionally, LLMs enhanced with retrieval modules can effectively capture temporal relational patterns hidden in historical events. However, issues such as popularity bias and the long-tail problem persist in LLMs, particularly in the retrieval-augmented generation (RAG) method. These findings not only deepen our understanding of LLM-based event forecasting methods but also highlight several promising research directions. We consider that this comprehensive evaluation, along with the identified research opportunities, will significantly contribute to future research on temporal event forecasting through LLMs.