EpiEvolve: Self-Evolving Agents for Streaming Pandemic Forecasting under Regime Shifts
EpiEvolve adapts a frozen epidemic LLM through memory, reaching 0.629 accuracy and cutting post-shift recovery from 5 to 2 weeks.
Key Findings
Methodology
EpiEvolve wraps a warm-start-trained, frozen Qwen3-14B-Base forecaster and performs no deployment-time weight updates. Each week it predicts all 50 states before labels arrive, then uses hierarchical episodic memory, outcome reflection, strategic lesson distillation, drift detection, and regime-aware retrieval to turn forecast errors into prompt context. Retrieval uses cos(ϕt,ϕe)·w(ρt,ρe), with weight 1 for the same regime and α=0.5 otherwise.
Key Results
- Across 81 weeks and 4,050 state-week cases from 2021-06-07 to 2022-12-19, EpiEvolve achieved 0.629 average accuracy, versus 0.561 for static PandemicLLM, 0.566 for streaming fine-tuning, and 0.325 for the CDC ensemble.
- Average recovery after regime transitions fell to 2 weeks, compared with 5 weeks for PandemicLLM and streaming fine-tuning. Accuracy in BA.5 and BQ.1 was 0.563 and 0.582, versus 0.431 and 0.456 for the static backbone.
- Ablations reduced accuracy to 0.604 without strategic memory, 0.591 without reflection, 0.612 without drift detection, and 0.604 without regime-aware retrieval, demonstrating complementary mechanisms.
Significance
The paper reframes epidemic forecasting as deployment-time learning under delayed feedback rather than static supervised prediction. It shows that a released model can exploit validated experience without changing its parameters, potentially improving auditability, reproducibility, and operational safety. For public-health organizations, faster recovery after variant transitions addresses a practical failure mode: a model may remain systematically wrong precisely when decisions are most consequential. The chronological protocol also supplies a stronger benchmark for adaptive forecasters.
Technical Contribution
The system formalizes non-parametric adaptation as Mt=(Et,Lt,ρt). Et stores state-, HHS-region-, and national-level forecast episodes; reflection produces the error explanation ℓ; strategic distillation converts recurring patterns into λ=(preconditions, consequent, c, n, ρλ), with confidence updated by empirical precision. Drift is triggered when cross-region ordinal error exceeds the warm-start mean by τσ or when a new dominant variant appears. Batchwise updates prevent within-week leakage.
Novelty
The authors characterize EpiEvolve as the first self-evolving LLM agent evaluated for streaming epidemic forecasting under recurring regime shifts. Unlike PandemicLLM, Claude-ICL, retrieval-only systems, or streaming fine-tuning, its central innovation is a closed forecast–label–reflection–rule loop. Adaptation occurs through structured memory and regime-weighted retrieval, not a new backbone or gradients.
Limitations
- The evidence covers only 50 U.S. states, COVID-19 hospitalization trends, and five variant-era slices; it does not establish transfer to other pathogens, countries, or prolonged crises.
- LLM-generated reflections and rules may hallucinate or reinforce spurious correlations. The paper gives limited evidence on prompt cost, latency, memory growth, calibration, and portability across backbone models.
- Drift detection depends on threshold τ and observable variant text; real surveillance may be delayed, incomplete, or inconsistent.
Future Work
Future work should test influenza, RSV, international regions, and prospective deployment; add probabilistic calibration and uncertainty-aware rules; prune or revise obsolete lessons; and compare multiple backbones, retrieval budgets, and τ values. Reporting confidence intervals, compute cost, human audit procedures, and behavior under abrupt or unobserved shifts would strengthen the evaluation.
AI Executive Summary
Operational epidemic forecasting is not a conventional static prediction task. Forecasters must issue weekly predictions before hospitalization labels arrive, while viral lineages, immunity, behavior, and policy continually change. A model trained on an earlier period can therefore remain wrong after a regime shift. Streaming fine-tuning offers adaptation, but it changes parameters, costs computation, and can complicate auditability.
EpiEvolve takes a different approach: it freezes a Qwen3-14B-Base backbone trained on a warm-start period and adapts only through external memory. It stores past forecast episodes at state, HHS-region, and national levels; after delayed labels arrive, a reflection module diagnoses errors. A strategic distiller turns recurring diagnoses into predicate-style rules with support counts and empirical confidence. A drift detector reacts to anomalous ordinal error or a newly observed dominant variant, while regime-aware retrieval prioritizes evidence from the current regime. All weekly forecasts are completed before memory updates, preventing future leakage.
On 81 streaming weeks from 2021-06-07 to 2022-12-19—4,050 state-week forecasts across five variant regimes—EpiEvolve reached 0.629 accuracy, compared with 0.561 for static PandemicLLM, 0.566 for streaming fine-tuning, and 0.325 for the CDC ensemble. It achieved 0.563 and 0.582 in BA.5 and BQ.1, and reduced average recovery lag from 5 to 2 weeks. Ablations confirm that reflection, strategic memory, drift detection, and regime-aware retrieval each matter. The study is promising but remains limited by one country, one disease, and the reliability of LLM-generated lessons.
Deep Analysis
Background
Epidemic forecasting combines hospitalization and case time series with policy text, vaccination signals, geography, and genomic surveillance. The COVID-19 Forecast Hub established the value of ensembles, while PandemicLLM and related epidemic LLM forecasters unified numerical and textual evidence. However, most systems use static train/test assumptions. Deployment introduces delayed labels and concept drift, especially when variants change the predictive value of historical signals.
Core Problem
At week t, predictions must use only current xs,t and deployment memory Mt; the batch Bt={(x,ŷ,y)} becomes available only after every state forecast is fixed. The agent must detect distribution change, decide which historical cases remain relevant, and transfer lessons across regions without allowing future outcomes to leak into current predictions. This is difficult because evidence can shift before enough labeled examples accumulate.
Innovation
- ��Frozen-backbone adaptation replaces gradient updates with prompt-mediated memory.
- ��Hierarchical episodic memory combines state, HHS-region, and national views.
- ��Reflection writes selective error explanations and candidate corrections.
- ��Strategic distillation promotes recurring patterns into confidence-weighted rules.
- ��Regime-aware retrieval uses same-regime weight 1 and cross-regime weight 0.5.
- ��A dual drift detector uses abnormal ordinal error or a new observed variant.
Methodology
- ��Input: recent hospitalization/case trends, vaccination summaries, policy text, dominant variant, and regional context.
- ��Forecast: retrieve episodes Et and matching rules Lt; render them in <MEMORY> and <RULES>; frozen fθ outputs one of five ordinal trend classes.
- ��Feedback: after labels arrive, generate one-sentence reflection ℓ and append an episode.
- ��Distillation: every K weeks or on drift, summarize recent entries into λ rules and update empirical confidence c.
- ��Transition: advance ρ, prioritize current-regime evidence, and demote rather than delete obsolete rules.
Experiments
The warm-start period ends 2021-05-31. Streaming evaluation covers 2021-06-07–2022-12-19: 81 weekly rounds, 50 states, and 4,050 forecasts, divided into Late-Alpha/Delta, BA.1, BA.2, BA.5, and BQ.1. Baselines are static PandemicLLM, streaming fine-tuning, Claude-ICL, the CDC/COVID-19 Forecast Hub ensemble, retrieval-only, and reflection-only variants. Metrics include accuracy, ordinal-label MSE, boundary recovery curves, and recovery lag. The backbone is Qwen3-14B-Base.
Results
EpiEvolve achieved 0.629 overall accuracy, a 0.068 absolute gain over PandemicLLM and a 0.304 gain over the CDC ensemble. Regime accuracies were 0.624, 0.751, 0.687, 0.563, and 0.582. Full-system MSE and lag were 0.65 and 2 weeks; removing reflection produced 0.79 and 4, removing strategic memory 0.71 and 3, and removing regime-aware retrieval 0.73 and 3. The largest benefits appeared in BA.5 and BQ.1.
Applications
Public-health agencies can use the framework for weekly state-level hospitalization alerts, provided they have timestamped surveillance data, policy and variant feeds, and reliable delayed labels. Hospitals could apply the same loop to capacity planning. The design may transfer to influenza, RSV, air-quality forecasting, and other nonstationary streams, although disease-specific labels and validation are prerequisites.
Limitations & Outlook
The evaluation is limited to U.S. COVID hospitalization trends; the five regimes are post-hoc reporting partitions, not supplied labels. LLM reflections may hallucinate rules, while retrieval and prompt size create computational and governance costs. The study does not fully assess calibration, confidence intervals, privacy, long-horizon memory growth, or cross-backbone transfer. Future work should add uncertainty-aware and human-audited rule management and prospective multi-pathogen tests.
Plain Language Accessible to non-experts
Imagine a hospital coordinator who must predict next week’s bed demand for every state. On Monday, the coordinator can read recent numbers, public announcements, and information about the circulating virus, but the true answer arrives only later. Once the weekly report is submitted, no one is allowed to change it.
A normal coordinator may keep using an old notebook. When a new virus version appears, that notebook can become misleading, so mistakes repeat. EpiEvolve gives the coordinator a filing system: it first looks for similar cases from the same state, then nearby states, then the wider country. It also writes a short note after every revealed mistake, such as “this signal usually means the increase is larger than expected.” Repeated notes become trusted reminder cards.
The coordinator’s basic brain never changes; the useful part is the organized notebook placed in front of it. That makes the process easier to inspect and prevents cheating with future answers. In the study, EpiEvolve was correct 62.9% of the time, versus 56.1% for the unchanged model, and recovered from major changes in about two weeks rather than five. It is not magic, however: a bad note can mislead later decisions, and the test covered only U.S. COVID data.
ELI14 Explained like you're 14
Picture a weekly prediction game. You must guess whether hospital use in each of 50 states will drop a lot, drop a little, stay steady, rise a little, or rise a lot. The answer is hidden until later, so you cannot peek while making this week’s guesses.
A basic robot learned its strategy before the game started. When a new virus variant changes the map, its old strategy may fail for many rounds. EpiEvolve keeps the robot’s brain frozen but gives it an experience backpack. After each round, it saves the guess, the real answer, and a quick explanation of what went wrong. It searches for similar experiences before making the next guess.
If the same mistake happens repeatedly, the robot turns it into a rule card: “When these clues appear, do not underestimate an increase.” When a new variant is spotted, it pays more attention to recent cards and similar regions. All players finish their guesses before anyone sees the answers, so the game stays fair!
The result: EpiEvolve scored 0.629 accuracy, while the original PandemicLLM scored 0.561 and the CDC ensemble 0.325. After a major change, it recovered in two weeks instead of five. Still, it might learn a bad rule, and researchers tested only COVID hospitalization data in the United States.
Glossary
Streaming forecasting
A forecasting process in which data and labels arrive sequentially. Predictions are made first, and feedback updates the system later.
The paper evaluates 81 weekly rounds under delayed feedback.
Regime shift
A change in the data-generating process caused here by variants, immunity, or policy. Patterns learned in one era may not transfer to another.
The stream contains five variant regimes from Late-Alpha/Delta through BQ.1.
Episodic memory
An external store of concrete past episodes, including inputs, predictions, truths, and reflections. It adapts behavior without changing model weights.
Episodes are organized into state, regional, and national views.
Strategic memory
A compact collection of reusable rules distilled from recurring reflection patterns. Each rule has conditions, a consequence, support, confidence, and regime tag.
Rules populate the <RULES> prompt slot.
Delayed feedback
Ground truth becomes available only after predictions are fixed. Batchwise timing prevents one state’s revealed outcome from influencing another state’s same-week forecast.
Bt updates Mt+1 only after the weekly batch is complete.
Recovery lag
The number of weeks after a regime boundary until accuracy first reaches the new regime’s steady-state level.
EpiEvolve averages 2 weeks versus 5 for static PandemicLLM.
Open Questions Unanswered questions from this research
- 1 Can LLM-generated rules be automatically checked for hallucination, contradiction, and spurious correlation? The paper demonstrates empirical gains but does not provide formal safety guarantees for lesson accumulation.
- 2 Will regime-aware memory transfer to influenza, RSV, other countries, or settings without timely variant text? Prospective cross-domain studies with uncertainty calibration are needed.
Applications
Immediate Applications
State-level hospitalization alerts
Public-health teams can attach delayed hospitalization labels, policy feeds, and genomic surveillance to a frozen PandemicLLM-style backbone. Weekly batch evaluation then writes auditable reflections and improves subsequent alerts, assuming stable labels and timestamped information.
Hospital capacity planning
Hospital networks can retrieve similar historical waves by region and update capacity forecasts after outcomes are known. Required prerequisites include consistent trend classes, reliable data pipelines, and governance for reviewing automatically generated strategic rules.
Long-term Vision
Cross-pathogen adaptive forecasting
A shared frozen backbone plus disease-specific memory could support influenza, RSV, and emerging pathogens. The main obstacles are sparse labels, mechanism differences, privacy, and unsafe transfer of rules; multi-year prospective validation would be necessary.
Abstract
Epidemic LLM forecasters are usually trained and evaluated as static supervised models, whereas operational pandemic forecasting is a streaming process in which labels arrive after predictions and disease regimes shift over time. We study this mismatch in weekly COVID-19 hospitalization trend forecasting across five variant regimes. We introduce EpiEvolve, a self-evolving agent that wraps an LLM forecaster trained on the warm-start period and keeps its weights fixed during streaming. EpiEvolve adapts by storing forecast outcomes in a hierarchical episodic memory, reflecting on delayed labels, retrieving cases relevant to the current regime, and distilling recurring errors into strategic rules. The resulting context lets the forecaster reuse its own past predictions and outcomes in later weeks while following a chronological protocol that prevents future leakage. On the streaming dataset, EpiEvolve reaches $0.629$ average accuracy, compared with $0.561$ for the static backbone and $0.325$ for the external CDC ensemble, and reduces recovery lag after regime shifts from $5$ to $2$ weeks. Ablations show that reflection, strategic memory, and regime-aware retrieval each contribute to the gains.