Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)
Using Falcon-7B-Instruct with FAISS-based RAG, achieving ROUGE-1 scores over 0.33 and outperforming Lead-3 in financial news summarization.
Key Findings
Methodology
This study developed a multi-source pipeline collecting news via News API, background from Wikipedia, and stock prices from Yahoo Finance. Numerical stock data were converted into natural language using a structured template to prevent model errors. Two summarization strategies were employed: LangChain's Summarize Chains and FAISS-based Retrieval-Augmented Generation (RAG). The models evaluated included Falcon-7B-Instruct, DistilBART-CNN, and BART-Large-XSum, with GPT-3 used for stock data summaries. Results showed Falcon-7B with Summarize Chains achieved the best coverage and coherence, accurately capturing all news events without hallucinations.
Key Results
- Falcon-7B combined with Summarize Chains scored ROUGE-1 of 0.3361, covering all three news events, and maintained factual accuracy without hallucination, outperforming other models.
- RAG, while theoretically promising, caused severe repetition in Falcon and hallucinated facts in BART-Large, especially at higher k values, indicating practical limitations.
- All deep models outperformed the Lead-3 baseline, with ROUGE-1 improvements over 20%, demonstrating deep learning's effectiveness in financial text summarization.
Significance
This work advances automated financial news summarization by integrating multi-source data, structured data conversion, and retrieval-augmented techniques. It addresses key issues like information overload and factual reliability, offering scalable solutions for real-time financial analysis. The combination of open-source models and innovative data handling paves the way for accessible, reliable AI-driven financial assistants, potentially transforming how investors and analysts process market information.
Technical Contribution
Key innovations include: 1) a structured template-based conversion of numerical stock data into natural language, reducing errors; 2) integration of FAISS for semantic retrieval, improving relevance; 3) comprehensive evaluation of Summarize Chains versus RAG across multiple models, revealing RAG's issues with repetition and hallucination in small models, enriching the theoretical framework of financial text summarization.
Novelty
This is the first systematic comparison of Summarize Chains and RAG in financial news summarization, especially with FAISS-based retrieval. The structured data-to-text approach effectively mitigates numerical hallucinations, enhancing factual reliability. The study fills a gap in applying retrieval-augmented methods to finance, demonstrating their practical challenges and benefits, thus contributing novel insights to the field.
Limitations
- RAG in small models suffers from severe repetition and hallucination, limiting its immediate deployment in high-stakes scenarios. Optimization of retrieval parameters and model fusion is needed.
- The dataset covers only ten companies over a two-month window; broader validation across markets and longer periods is required.
- ROUGE scores, while useful, do not fully capture factual correctness; human evaluation and factual verification are necessary for comprehensive assessment.
Future Work
Future directions include improving retrieval strategies to reduce hallucinations, expanding datasets to enhance model generalization, integrating multi-modal data (e.g., charts, videos), and developing real-time, scalable systems. Additionally, incorporating human-in-the-loop evaluation will refine factual accuracy and trustworthiness, facilitating deployment in real-world financial environments.
AI Executive Summary
In today's fast-paced financial markets, investors and analysts are overwhelmed by the sheer volume of news and data. Manually sifting through hundreds of articles daily is impractical, often leading to missed critical information. Existing automated summarization methods focus mainly on textual content, neglecting the numerical data crucial for financial decision-making. This gap hampers timely insights and increases the risk of misinformation. To address this, the study introduces an innovative system combining large language models (LLMs), retrieval mechanisms, and multi-source data integration.
The core framework involves collecting news via News API, background information from Wikipedia, and stock prices from Yahoo Finance. Numerical stock data are transformed into natural language using a simple template, ensuring factual accuracy and model interpretability. The system employs two summarization strategies: the sequential Summarize Chains and the retrieval-augmented generation (RAG) with FAISS. Extensive experiments on open-source models like Falcon-7B-Instruct, DistilBART-CNN, and BART-Large-XSum demonstrate that Falcon-7B with Summarize Chains achieves the best coverage, coherence, and factual reliability, outperforming baselines.
While RAG offers promising theoretical benefits, practical challenges such as repetition and hallucination are evident, especially in smaller models. All methods outperform the naive Lead-3 baseline, validating the potential of deep learning for financial text summarization. This research not only advances the technical state-of-the-art but also provides a scalable, accessible solution for real-time financial analysis, with broad implications for automating market intelligence and supporting smarter investment decisions.
Looking ahead, future work will focus on refining retrieval strategies, expanding datasets, and integrating multi-modal data sources. Addressing current limitations, such as hallucination and computational costs, will be critical for deploying robust, trustworthy financial AI systems. Overall, this study marks a significant step toward intelligent, automated financial information processing, promising to reshape how market insights are generated and consumed.
Deep Analysis
Background
Financial text summarization has evolved from rule-based and keyword matching methods to deep neural models like BART and T5, achieving significant progress on benchmarks such as CNN/DailyMail and XSum. However, applying这些模型到金融领域面临特殊挑战，尤其是数值信息难以直接输入模型，导致事实可靠性不足。近年来，结合多源数据（新闻、财报、股价）进行信息融合成为研究热点。FinBERT和BloombergGPT在金融情感分析中表现出色，但缺乏系统性摘要方案。本研究旨在突破现有局限，通过结构化数据转文本和检索机制，提升金融文本自动摘要的准确性和覆盖率。
Core Problem
金融市场信息量巨大，手工整理费时费力，且易遗漏关键信息。现有自动摘要多集中于文本，难以有效融合数值数据，导致事实幻觉和信息遗漏。如何在保证事实准确的基础上，结合多源信息实现高质量自动摘要，成为行业难题。特别是在模型对结构化数据理解有限的情况下，设计有效输入格式和检索机制，提升模型的覆盖率和可靠性，是亟待解决的核心问题。
Innovation
本研究的创新点包括：1）提出结构化金融数据的模板化转文本方案，避免模型误算，提高事实可靠性；2）结合FAISS实现高效语义检索，增强信息相关性，减少信息遗漏；3）系统评估多模型、多策略的性能差异，揭示RAG在金融场景中的局限性，丰富了金融文本自动摘要的理论体系。这些创新为实现高效、可靠的金融信息自动化提供了新思路。
Methodology
- �� 数据采集：利用News API、Wikipedia和Yahoo Finance收集十家主要公司（如AAPL、MSFT）新闻、背景和股价信息。
- �� 结构化转文本：将股票的OHLCV数据通过模板转换为自然语言句子，避免模型误算。
- �� 知识库构建：将新闻、背景和转文本数据合并，形成检索语料库。
- �� 检索机制：使用LangChain的FAISS索引，将文本分块，生成向量表示，支持高效相似度检索。
- �� 摘要策略：采用Summarize Chains逐段总结，结合FAISS检索的RAG方案，比较两者性能。
- �� 模型评估：在Falcon-7B-Instruct、DistilBART-CNN和BART-Large上进行新闻摘要，GPT-3用于股价数据，指标包括ROUGE和定性分析。
Experiments
实验设计涵盖多源数据采集、不同模型和策略的比较。采用真实金融新闻和股价数据，ROUGE指标评估摘要质量。通过定性分析验证内容覆盖和事实准确性。模型参数调优包括k值设定（如k=31）和模板优化。对比不同模型在事件覆盖、信息完整性和幻觉控制方面的表现，验证Summarize Chains在内容连贯性上的优势，以及RAG在信息相关性上的潜力与局限。
Results
Falcon-7B结合Summarize Chains在新闻摘要中ROUGE-1达0.3361，全面覆盖三类事件，避免幻觉。RAG在理论上能增强信息相关性，但在实践中出现严重重复和事实幻觉，尤其在模型规模较小时表现不佳。所有深度模型均优于Lead-3基线，验证了深度学习在金融自动摘要中的有效性。模型在不同公司间表现差异明显，揭示了模型规模和策略选择对效果的影响。
Applications
该系统可应用于金融分析师、投资机构的日常信息筛选，自动生成简明、准确的市场动态报告。对金融信息服务平台、智能投资助手具有推广价值，能显著提升信息处理效率和决策速度。未来还可结合实时数据和多模态信息，打造全场景智能金融分析工具。
Limitations & Outlook
RAG在小模型中的幻觉和重复问题限制其应用，模型规模和检索策略需优化。数据集规模有限，泛化能力待验证。ROUGE指标不足以全面评估事实准确性，未来需引入人类评审和事实验证机制。模型计算成本较高，实际部署需考虑效率和成本平衡。
Plain Language Accessible to non-experts
想象你在厨房做饭，准备多种食材（新闻、股票数据、背景信息），每种食材都需要先洗干净（预处理）再放到锅里（转成自然语言）。有时候，菜谱（模型）只能处理文字，不能直接理解数字，所以你用一个简单的食谱（模板）把数字变成描述，比如“今天苹果的收盘价上涨了3%”。这样，厨师（模型）就能更好地理解和总结菜肴的味道（新闻内容）。如果你还用一个智能助手（检索系统）帮你找出最相关的食材（信息检索），效果会更好。最终，你可以得到一份完整的菜单（摘要），既详细又准确，帮助你快速了解当天的厨房动态。这种方法就像用简单的步骤，把复杂的数字和文字变成一份美味的菜肴，既省时又可靠。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的消息和作业（新闻和股票信息），你想快速知道重点。自己一个个看很慢，也容易漏掉重要内容。于是，你请一个聪明的哥哥帮忙，他会用一种特别的方法，把数字变成简单的句子，比如“今天苹果的价格涨了3%”。这样，你只要看哥哥写的总结，就能很快知道发生了什么。哥哥还会用一个搜索工具，帮你找到最相关的消息，确保你看到的都是重要的。最后，你会得到一份简洁又准确的报告，告诉你每天的市场动态。这个过程就像用一个聪明的机器人帮你整理信息，让你不用花太多时间，也能掌握全部重点。这种技术可以帮助投资者更快做出决策，避免遗漏重要信息。
Glossary
Large Language Model (大规模语言模型)
一种基于深度学习的模型，能理解和生成自然语言，像GPT或Falcon。
用于生成金融新闻摘要和股票分析。
检索增强生成（Retrieval-Augmented Generation, RAG）
结合信息检索和生成模型，提升回答相关性和准确性的方法。
在本文中用FAISS实现金融新闻的相关信息检索。
ROUGE指标
衡量自动摘要与参考文本重叠程度的指标，反映内容覆盖和质量。
用来评估新闻摘要的效果。
FAISS
Facebook AI开发的高效向量相似度搜索库，用于大规模语义检索。
实现金融文本的快速相关信息检索。
Summarize Chains
逐段总结后合成完整摘要的策略，适合长文本。
在多源金融信息摘要中使用。
Open Questions Unanswered questions from this research
- 1 如何进一步减少检索中的幻觉和重复，提升模型在金融场景中的事实可靠性。
- 2 多模态数据（如图表、视频）融合以丰富信息内容的技术路径。
- 3 大规模模型在金融实时应用中的效率优化与成本控制。
Applications
Immediate Applications
金融新闻自动摘要工具
为投资分析师提供每日自动生成的市场动态报告，提升信息处理效率，减少遗漏。
智能投资助手
结合自动摘要和可视化，为个人投资者提供简洁、准确的市场分析，辅助决策。
Long-term Vision
全场景智能金融分析平台
整合多源数据、多模态信息，打造实时、全面、智能的金融分析系统，推动金融自动化。
Abstract
Stock market analysts and investors face a daily challenge: too much financial news, too little time. Manually reading and synthesizing hundreds of company-specific articles is impractical, yet missing key information can directly affect investment decisions. This project, conducted at George Washington University in Fall 2023, explores whether Large Language Models can automate this process reliably. We built a pipeline that pulls news articles from the News API, company background from Wikipedia, and stock price data from Yahoo Finance for ten major companies (AAPL, MSFT, GOOGL, AMZN, META, TSLA, JPM, NVDA, WMT, DIS). Because LLMs cannot directly process numerical tables, we developed a simple but effective template that converts stock data into natural language narratives. We then tested two summarization approaches (Summarize Chains and Retrieval-Augmented Generation with FAISS) across three open-source models (Falcon-7B-Instruct, DistilBART-CNN-12-6, BART-Large-XSum) for news, and GPT (text-davinci-003) for stock summaries. Falcon-7B with Summarize Chains gave the best results, covering all news events accurately and coherently. RAG, while promising in theory, caused severe repetition in Falcon and hallucinated facts in BART-Large when k was large. Both LLM-based approaches outperformed a simple Lead-3 baseline on ROUGE-1. We also built a Streamlit dashboard for interactive stock visualization. The work was done in Fall 2023, before RAG-based financial tools became widespread, and the failure modes we document, particularly hallucination under RAG in smaller models, remain relevant today.