FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards
FinEvolveBench tests delayed-feedback self-evolution on 31 Chinese A-share industries and 177,324 news articles; generic memory rarely improves IC.
Key Findings
Methodology
FinEvolveBench chronologically replays a financial stream covering 31 Shenwan Hongyuan industries and 177,324 Chinese financial articles. For each date, agents read news from t−4 through t and output continuous market-sentiment scores s∈[-1,1] for 10-, 20-, and 40-trading-day horizons. Feedback is the industry return minus the CSI1000 return, α=rindustry−rCSI1000, revealed only when mature. Pipe, Mem0-style append-only memory, and MemRL utility-updated memory are compared.
Key Results
- Pipe achieved the highest Overall tsIC in all six completed backbone–horizon settings. Its DeepSeek-V4-Flash scores were 0.0243, 0.0517, and 0.0595 at 10, 20, and 40 days; its Qwen3-35B-A3B scores were 0.0349, 0.0676, and 0.0677. Memory availability therefore did not imply predictive improvement.
- Pipe+MemRL underperformed Pipe by 0.0068 and 0.0335 at 10 and 20 days on DeepSeek, and by 0.0163 and 0.0195 on Qwen. At 40 days, Qwen MemRL reached only 0.0135 versus Pipe's 0.0677, showing strong horizon dependence.
- The matched Frozen MemRL ablation found that utility updates helped Qwen at 10/20 days: Overall tsIC gains were 0.0382/0.0731 and csIC gains 0.0392/0.0834. They hurt at 40 days and reduced DeepSeek Overall tsIC at every horizon, indicating that delayed feedback can calibrate or contaminate retrieval.
Significance
The benchmark moves self-evolution evaluation beyond repetitive, verifiable tasks toward noisy, non-stationary decision environments. It shows that storing trajectories is not equivalent to learning from experience when outcomes arrive late and cannot be assigned to individual reasoning steps. For industry, financial forecasting, monitoring, and information triage should not be judged only by offline accuracy; they require causal access control, delayed-outcome handling, stage-wise analysis, and cross-backbone robustness.
Technical Contribution
The main contribution is a causal replay interface rather than a new language-model training algorithm. Predictions are immutable, future news and prices are hidden, and horizons mature independently. The released corpus preserves full titles and contents without deduplication, importance filtering, or outlier removal. Market-adjusted returns provide continuous outcome feedback. Evaluation separates time-series IC from cross-sectional IC and uses a matched Frozen MemRL condition to isolate feedback-driven utility updates.
Novelty
Unlike SWE-bench, GAIA, and ALFWorld, which generally expose clear success signals, FinEvolveBench targets low repetition with implicit, delayed, outcome-level feedback. Unlike FinQA, FinBen, and other static financial datasets, it reconstructs a continuous information stream and supports researcher-defined horizons. Unlike StockBench, it releases 177,324 complete articles rather than retrieving a small daily subset, emphasizing experience adaptation rather than direct trading profit.
Limitations
- Only DeepSeek-V4-Flash and Qwen3-35B-A3B are evaluated, over a Chinese A-share window from January 2, 2025 to March 31, 2026. Results may depend on model architecture, market regime, and temporal composition.
- Market-adjusted returns combine public information, latent factors, and regime effects. They cannot identify which article or reasoning step caused an outcome, leaving credit assignment fundamentally ambiguous.
- Mem0 and MemRL differ in representation and retrieval design; only Frozen MemRL provides a close control for the utility-update component.
Future Work
Future systems should abstract experience across cases instead of merely retrieving individual trajectories, condition memory on market regimes, represent conflicting evidence and feedback uncertainty, and learn when not to reuse history. Broader models, markets, horizons, statistical testing, and reporting of latency, compute, and risk are also needed.
AI Executive Summary
Whether language-model agents can improve after deployment is a central question in autonomous-system research. Existing benchmarks such as SWE-bench, GAIA, and ALFWorld usually offer recurring procedures and explicit outcomes. Real decisions are less forgiving: similar news can precede different market reactions, feedback may arrive weeks later, and no label identifies which observation caused the result.
FinEvolveBench reconstructs a Chinese A-share information stream containing 31 Shenwan Hongyuan industry indices, 177,324 articles from nine financial-media sources, and daily market records. Across 300 prediction dates, agents read news from the preceding four days and produce 10-, 20-, and 40-trading-day directional scores. Feedback is the industry's return relative to CSI1000, revealed only after the horizon matures. The study compares no-experience Pipe, append-only Mem0, and utility-updated MemRL.
The outcome is diagnostic rather than a simple leaderboard. Pipe leads every overall tsIC combination: on Qwen, its 10/20/40-day scores are 0.0349, 0.0676, and 0.0677. A matched ablation shows that utility updates improve Qwen short horizons by 0.0382 and 0.0731 tsIC, yet decline at 40 days; DeepSeek is worse in most settings. Thus memory access, feedback updating, and genuine self-evolution are not interchangeable. Robust agents must model context, regime change, uncertainty, and when past experience should be ignored.
Deep Analysis
Background
Reflexion, Mem0, Evo-Memory, and MemRL established runtime memory as an alternative to weight updates. SWE-bench, GAIA, LifelongAgentBench, and ALFWorld evaluate experience reuse, but usually with explicit or repeatable success signals. FinQA, TAT-QA, and FinBen are static financial datasets; StockBench adds temporal simulation but emphasizes trading profit. FinEvolveBench addresses continuous, weakly repetitive prediction.
Core Problem
An agent must forecast future industry-relative performance from time-ordered news while receiving only delayed, continuous, noisy outcomes. Recurring entities do not imply recurring market responses, and stored cases may fail under a new regime. The bottlenecks are causal access control, cross-horizon credit assignment, non-stationarity, and deciding when experience is reusable.
Innovation
- �� A fixed stream of 177,324 complete articles rather than isolated instances.
- �� Causal replay interleaving new predictions with matured historical outcomes.
- �� Continuous implicit feedback α=rindustry−rCSI1000.
- �� Separate tsIC and csIC, with Cold-Start versus Exploitation reporting.
- �� Frozen MemRL matching all design factors except feedback-driven utility updating.
Methodology
- �� Input: industry-filtered news from natural days t−4 through t, current market context, and causally available memory.
- �� Output: s_i,t,h∈[-1,1] for every industry and selected horizon, representing future market direction rather than lexical sentiment.
- �� Outcome: r_i,t,h=close_i,t+h/close_i,t−1−1; α_i,t,h=r_i,t,h−rCSI1000,t,h.
- �� Systems: Pipe has no external experience; Mem0 appends complete trajectories; MemRL maintains separate 10/20/40-day utilities.
- �� Evaluation: tsIC measures within-industry temporal association, while csIC measures same-day industry ranking; results average three independent runs.
Experiments
The corpus spans news from 2024-01-01 to 2026-05-03 and market labels through 2026-06-30, with 696 active news days and 254.78 articles per day on average. The replay covers 300 trading days; its first third is Cold-Start and final two thirds Exploitation. Backbones are 32,768-token DeepSeek-V4-Flash and Qwen3-35B-A3B. Pipe, Mem0, and MemRL are compared, with Frozen MemRL as the matched ablation.
Results
Pipe leads all Overall tsIC settings: DeepSeek scores 0.0243/0.0517/0.0595, while Qwen scores 0.0349/0.0676/0.0677 for 10/20/40 days. DeepSeek MemRL scores 0.0175/0.0182/0.0269; Qwen MemRL scores 0.0186/0.0481/0.0135. Qwen utility updating helps short horizons, whereas DeepSeek is mostly harmed, confirming dependence on backbone, horizon, stage, and metric.
Applications
The benchmark can support financial-news monitoring, sector-rotation research, risk alerts, and information triage, but its scores are not trading instructions. Deployment requires temporal splitting, delayed labels, benchmark adjustment, leakage prevention, and evaluation of ranking quality, stage stability, and update cost.
Limitations & Outlook
The evidence is limited to two models and Chinese A-shares, so cross-market generalization is unknown. α is an outcome-level proxy and cannot solve article-level attribution; duplicated and noisy news may also stress context windows. Future work should add regime detection, cross-case abstraction, confidence modeling, more markets, significance tests, and compute/latency accounting.
Plain Language Accessible to non-experts
Imagine an analyst who writes daily forecasts for 31 industries. Each morning, the analyst reads newspapers from the last few days and predicts whether each industry will outperform the broad market after 10, 20, or 40 trading days. The forecast is locked; future newspapers cannot be read. Weeks later, the real result arrives, like a delayed exam grade.
Researchers test three habits. One analyst keeps no notes. Another files every forecast. A third gives old notes higher or lower priority after seeing later results. The difficulty is that similar headlines do not guarantee similar market reactions, and a good or bad result may reflect many hidden causes.
The surprising result is that a larger filing cabinet does not reliably make the analyst better. The no-memory Pipe system has the best overall time-series score in all six model–horizon combinations. Feedback-based ranking helps Qwen on short horizons but hurts on the longest one, while it mostly hurts DeepSeek. The lesson is simple: old notes are useful only when the current situation truly matches the past.
ELI14 Explained like you're 14
Imagine a prediction game: every day you read the news and guess which industry will rise or fall. You do not get the answer immediately. It arrives after 10, 20, or 40 trading days, and the result may depend on politics, world events, and secrets nobody saw.
FinEvolveBench is a giant practice arena with 31 Chinese stock industries, 177,324 financial articles, and 300 prediction days. The robot may read only what was available that day. It can forget everything, save old predictions, or give old notes bonus points when later results arrive.
Here is the twist: more memory did not automatically mean smarter. The simplest no-memory method beat the memory methods on all six overall time-series comparisons. Qwen sometimes benefited from feedback on short games, but lost on the 40-day game. DeepSeek usually lost after feedback updates too.
It is like using game攻略 from an old level: the trick may work there but fail on a new map. A truly clever robot needs to ask whether the current situation matches the old one, how trustworthy the old lesson is, and when to say, “I’m not sure!”
Glossary
Information Coefficient (IC)
IC measures association between predictions and outcomes. Time-series IC evaluates tracking within an industry over time; cross-sectional IC evaluates ranking industries on the same date.
The paper reports both rather than collapsing performance into one score.
Implicit reward
An implicit reward is an outcome-level signal revealed after an action rather than an immediate correctness label. It is often continuous, noisy, and difficult to attribute.
Industry return relative to CSI1000 is the delayed feedback.
Causal replay
Causal replay reconstructs interactions in chronological order and exposes only information available at each cutoff. Future data and immature outcomes remain hidden.
FinEvolveBench uses it to prevent temporal leakage.
MemRL
MemRL stores trajectories externally and updates their retrieval utility using later feedback. The benchmark maintains separate utilities for each horizon.
Pipe+MemRL and Frozen MemRL isolate the value of utility updating.
Low-repetition task
A low-repetition task may repeat entities or topics, but the evidence-to-outcome mapping is unstable. Similar cases therefore do not guarantee reusable solutions.
Financial news provides this property because similar stories can produce different reactions.
Open Questions Unanswered questions from this research
- 1 How can agents abstract reusable rules across cases instead of retrieving semantically similar trajectories? Current results do not separate textual similarity, causal similarity, and genuine contextual transfer.
- 2 How should delayed returns be assigned to articles, reasoning steps, and memory entries? Progress likely requires uncertainty modeling, counterfactual evaluation, and cross-market replication.
- 3 Why does utility updating help Qwen at short horizons but harm DeepSeek in most settings? Larger model suites, alternative prompts, and longer streams may reveal the interaction.
Applications
Immediate Applications
Financial information-stream evaluation
Research teams can use the nine-source corpus and 31 industries to test whether a memory mechanism improves 10–40-day forecasts under strict chronological replay, rather than relying on static classification accuracy.
Risk-monitoring prototype
Financial institutions can use sector scores for analyst alerts and news triage, provided that future information remains hidden and α is treated as a research signal rather than a direct trading command.
Long-term Vision
Context-aware evolving analyst
Future agents could identify market regimes, reconcile conflicting evidence, and reduce unreliable memory weights, enabling more robust research assistance and risk decisions across market cycles.
Abstract
Experience-based self-evolution enables language-model agents to improve their behavior by accumulating and updating experience at test time, yet existing evaluations often assume recurring task patterns and explicit success signals. We introduce \textsc{FinEvolveBench}, a benchmark for self-evolving agents on low-repetition tasks with implicit rewards. The benchmark reconstructs a daily financial information stream over 31 Chinese A-share industry indices and aligns 177,324 public news articles with market observations. Researchers can define prediction horizons over this stream; we evaluate predictive market-sentiment factors against delayed market-adjusted returns after 10, 20, and 40 trading days. Unlike static benchmarks that score each prediction independently, \textsc{FinEvolveBench} interleaves new decisions with delayed outcomes from earlier ones, testing whether agents can convert noisy real-world feedback into reusable experience at test time. Experiments with two backbone models show that the evaluated general-purpose memory systems do not consistently outperform the no-experience pipeline. A matched ablation further shows that feedback-driven utility updates help at shorter horizons on one backbone but hurt in most settings on the other. Together, these results position \textsc{FinEvolveBench} as a diagnostic testbed for experience-based self-evolution under noisy, delayed, and outcome-level feedback.