SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

Dialogue summarization has become increasingly vital with the proliferation of online communication platforms. Early efforts focused on structured data like meeting transcripts (e.g., AMI, ICSI) or synthetic datasets, but lacked diversity and realism. Recent advances leverage neural models, especially encoder-decoder architectures like Transformer, trained on large datasets such as CNN/Daily Mail for news. However, these models struggle with the unstructured, informal nature of real conversations, which include slang, typos, and multi-party interactions. The scarcity of high-quality, annotated dialogue datasets has limited progress. The emergence of chat applications like Messenger and WhatsApp has intensified the need for effective dialogue summarization, yet existing datasets are inadequate, often lacking manual annotations or realistic dialogue styles. This gap motivated the creation of SAMSum, designed to reflect natural messenger conversations, and to evaluate models' ability to generate abstractive summaries that are both accurate and human-like.

The core challenge in dialogue summarization lies in the complex, multi-turn structure where relevant information is dispersed across different speakers and utterances. Unlike news articles with linear narratives, dialogues involve informal language, abbreviations, and frequent topic shifts, complicating both understanding and summarization. Existing models often produce incoherent or inaccurate summaries, especially when trying to capture speaker-specific actions or long-distance dependencies. Moreover, traditional evaluation metrics like ROUGE are insufficient, as they rely on n-gram overlap and do not account for semantic fidelity or contextual coherence. The lack of high-quality, annotated datasets further hampers the development of robust models. Addressing these issues requires novel datasets, structural modeling, and evaluation strategies tailored for multi-party conversations.

This work introduces several key innovations: 1) The SAMSum corpus, a large-scale, manually annotated dataset of messenger-like dialogues, capturing diverse topics and styles; 2) Incorporation of dialogue structure markers (separators) to improve model understanding of turn boundaries; 3) Integration of pretrained language models (GPT-2) to enhance abstractive summarization performance; 4) Systematic comparison of multiple neural architectures, demonstrating the superiority of Transformer-based models trained on joint datasets; 5) Critical analysis of ROUGE's limitations in dialogue contexts, advocating for human evaluation. These innovations collectively advance the state-of-the-art in dialogue summarization, providing both data and methodological insights.

• �� Data collection: Linguists simulated natural messenger conversations, covering informal, semi-formal, and formal styles, with topics like gossip, meetings, and politics. Each dialogue was annotated with a concise, third-person summary.
• �� Data cleaning: Automated scripts detected format deviations, corrected typos, and unified speaker names using Levenshtein distance.
• �� Model training: Employed Pointer Generator, Transformer, LightConv architectures, with GPT-2 embeddings. Dialogue structure markers (e.g., separators) were added to encode turn boundaries.
• �� Input processing: Texts were tokenized, truncated to 400 tokens, with summaries limited to 100 tokens. Special tokens distinguished utterances.
• �� Training: Models trained on combined news and dialogue datasets, with beam search (width=5). Hyperparameters tuned for optimal ROUGE scores.
• �� Evaluation: Used ROUGE-1/2/L and manual human judgment to assess relevance, coherence, and informativeness. Ablation studies examined the impact of structure markers and pretrained embeddings.

Experiments involved training models on SAMSum and CNN/Daily Mail datasets, with variations including joint training and ablation of structure markers. Evaluation metrics combined automatic ROUGE scores and manual assessments of summary quality. Human judges rated summaries on clarity, relevance, and factual correctness, revealing that high ROUGE scores do not always correlate with perceived quality. The models' ability to handle multi-turn, multi-speaker dialogues was tested across different topics and styles. Hyperparameters such as input length, summary length, and beam size were optimized. Results demonstrated that models with dialogue structure markers and GPT-2 embeddings achieved the best performance, but still exhibited issues like speaker name errors and information omission, indicating room for improvement.

The top-performing model, DynamicConv with GPT-2 embeddings trained on combined datasets, achieved ROUGE-1, ROUGE-2, and ROUGE-L scores of 39.94, 17.56, and 36.51 respectively. These scores surpass baseline models and are comparable to or better than models trained on much larger news datasets. However, human evaluation revealed that many summaries contained errors such as incorrect speaker attribution and missing key details. The correlation between ROUGE and human judgment was moderate at best, emphasizing the need for better evaluation metrics. Ablation studies confirmed that dialogue structure markers and pretrained embeddings significantly improve performance, especially in multi-turn conversations. Overall, models excelled at extracting surface-level overlaps but struggled with semantic accuracy and coherence.

The findings support the development of intelligent chatbots, virtual assistants, and automated meeting summarizers. High-quality dialogue summaries can streamline information retrieval, improve user engagement, and reduce manual effort. These models require access to annotated dialogue data and robust architectures capable of understanding multi-party interactions. In industry, they can be integrated into customer service platforms, enterprise communication tools, and social media monitoring systems, enhancing real-time summarization and decision-making. Long-term, such technologies could enable fully autonomous conversational agents capable of maintaining context over extended interactions, transforming human-computer communication.

Current models often misidentify speaker actions or confuse entities, reducing summary accuracy. Handling long-range dependencies and topic shifts remains difficult, especially in lengthy dialogues. ROUGE scores do not align well with human perceptions of quality, highlighting the need for better evaluation metrics. Computational costs are high due to large pretrained models and joint training, limiting scalability. Additionally, the dataset, while diverse, is simulated and may not fully capture real-world messengers' informal language and noise. Future work must address these issues by developing more robust models, better evaluation methods, and expanding datasets to cover more naturalistic conversations.