SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
SummaRuNNer uses a bidirectional GRU sequence classifier with multi-feature integration, achieving state-of-the-art extractive summarization performance.
Key Findings
Methodology
SummaRuNNer employs a two-layer bidirectional GRU network to encode words and sentences, combining features like content, salience, and novelty through a multi-dimensional decision mechanism. It formulates extractive summarization as a sequence classification task, where each sentence is visited sequentially, and a binary decision is made based on the combined features. During training, a greedy Rouge-based label generation strategy creates pseudo ground-truth labels, enabling end-to-end training without manual sentence labels. The model's architecture is simple yet effective, with built-in interpretability via feature visualization, supporting transparency in decision-making.
Key Results
- On CNN/DailyMail, SummaRuNNer achieves Rouge-1 scores of 26.2 (extractive) and 23.8 (abstractive), outperforming baselines like Lead-3 (21.9). Its performance remains robust across different summary lengths, especially excelling in short summaries (75 words). On DUC 2002, it matches the performance of Cheng and Lapata (2016), demonstrating good generalization. The model's interpretability is validated through feature contribution visualizations, showing clear reasons behind sentence selection.
- The model's extractive version significantly surpasses previous deep models in Rouge scores, indicating superior sentence ranking ability. The abstractive training approach, coupling with a decoder, enhances the model's understanding of summary semantics, leading to better content coverage.
- Across multiple datasets, the model maintains competitive performance, with visualizations providing insights into feature importance, thus bridging the gap between black-box deep models and explainability requirements.
Significance
This work advances extractive summarization by integrating interpretability into deep neural models, addressing the black-box issue prevalent in NLP. Its simple yet powerful architecture enables high performance with fewer parameters, making it practical for real-world deployment. The novel training strategy, which leverages abstractive summaries to generate pseudo labels, reduces reliance on costly annotations. The approach enhances transparency, fostering trust in automated systems. It paves the way for more explainable AI in NLP, with applications spanning news summarization, legal document analysis, and content curation, ultimately contributing to more accessible and trustworthy AI systems.
Technical Contribution
The primary technical innovation lies in combining bidirectional GRU networks with a multi-feature decision layer that captures content richness, salience, and redundancy, enabling precise sentence extraction. The use of Rouge-based greedy label generation for training circumvents the need for manual sentence labels, a significant step toward unsupervised or weakly supervised learning. The model's architecture is streamlined, with fewer parameters than comparable models, facilitating training efficiency. Its interpretability is enhanced through feature visualization, providing insights into the decision process. The coupling with an abstractive decoder during training introduces a semi-supervised paradigm, improving the semantic quality of extracted summaries.
Novelty
This research is the first to integrate multi-dimensional features within a bidirectional GRU sequence classifier for extractive summarization, coupled with a Rouge-guided greedy label generation strategy. Unlike prior models relying on handcrafted features or complex architectures, SummaRuNNer achieves end-to-end training with minimal supervision and offers transparent decision explanations. Its innovative use of an abstractive decoder during training to improve extractive decisions without requiring sentence-level labels marks a significant departure from existing methods, bridging the gap between extractive and abstractive summarization.
Limitations
- The model depends on pre-trained word embeddings, which may limit performance in low-resource languages or domains lacking such resources. The greedy label generation, while effective, can introduce noisy labels, affecting training quality. Its performance on very long or structurally complex documents remains to be tested, as the current design favors relatively concise texts. Cross-domain adaptation might be limited, requiring further fine-tuning or domain-specific adjustments. Computational costs, though moderate, could be a concern for real-time applications in large-scale systems.
- The reliance on heuristic label generation may lead to suboptimal training signals, especially in cases where Rouge scores do not perfectly reflect sentence importance. Future work should explore more robust pseudo-labeling and domain adaptation techniques.
Future Work
Future directions include developing more robust pseudo-labeling strategies, possibly leveraging reinforcement learning to optimize summary quality directly. Extending the model to multi-document summarization and multilingual settings is also promising. Incorporating external knowledge sources, such as knowledge graphs, could further enhance content relevance. Improving the model's efficiency for real-time applications and exploring unsupervised or semi-supervised training paradigms will broaden its usability. Additionally, integrating more sophisticated features or attention mechanisms may boost performance further, especially on longer and more complex documents.
AI Executive Summary
SummaRuNNer introduces a novel neural architecture for extractive document summarization, leveraging a two-layer bidirectional GRU network that encodes words and sentences, integrating content, salience, and novelty features. This design allows the model to make sequential binary decisions on sentence inclusion, guided by a multi-feature decision layer. A key innovation is the use of Rouge-based greedy label generation, enabling end-to-end training without manual sentence labels, thus reducing annotation costs.
Experimental results on CNN/DailyMail and DUC 2002 datasets demonstrate that SummaRuNNer surpasses traditional baselines and matches or exceeds state-of-the-art models in Rouge scores. Notably, the extractive version achieves a Rouge-1 score of 26.2 on CNN/DailyMail, outperforming models like Cheng and Lapata (2016). The model's interpretability is validated through feature visualization, providing insights into why certain sentences are selected, which enhances trust and transparency.
The approach balances performance and explainability, making it suitable for practical deployment in news summarization, legal document analysis, and content curation. Its simplicity and efficiency facilitate training and real-time application. Future work aims to extend the model to multi-document and multilingual scenarios, improve pseudo-labeling robustness, and incorporate external knowledge sources, promising broader impact across NLP tasks.
Deep Dive
Abstract
We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels.