Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations

The evolution of dialogue summarization has seen Transformer-based models like BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and Pegasus (Zhang et al., 2019) dominate the field, primarily focusing on improving summary quality and efficiency. Early datasets such as SAMSum and Twitter-based conversations provided benchmarks, but lacked representation of natural conversational noise. Recent studies began to explore model robustness (Moradi & Samwald, 2021), yet most concentrated on classification tasks or synthetic perturbations of extreme nature. The gap remains in understanding how these models handle realistic, naturally occurring variations like repetitions, hesitations, and grammatical errors, which are prevalent in real-world dialogues. This research addresses this gap by systematically simulating such variations grounded in the natural conversation framework (Moore & Arar, 2019) and evaluating their impact on state-of-the-art models.

Despite advances, current dialogue summarization models exhibit fragility when exposed to natural conversational variations. These variations—common in real interactions—cause significant performance drops, leading to incomplete or inaccurate summaries. The core challenge lies in designing models that can maintain high-quality summaries despite noisy inputs, which is crucial for practical deployment in customer support, meeting transcription, and social media analysis. The difficulty is compounded by biases such as lead bias, long bias, and repetition bias, which distort the model’s focus. Addressing these issues requires understanding the models’ vulnerabilities and developing training strategies that enhance robustness without sacrificing efficiency.

This work introduces a novel perturbation framework based on natural conversation patterns, encompassing both utterance-level (typos, grammatical errors, vocabulary shifts) and dialogue-level (repetitions, greetings, delays) disruptions. It systematically evaluates multiple Transformer models, including fine-tuned and instruction-tuned variants, across three robustness dimensions. The framework’s key innovation is grounding perturbations in real-world conversational behaviors, enabling realistic robustness assessment. Additionally, the study reveals that simply scaling model size does not improve robustness, highlighting the need for targeted training strategies. The combination of comprehensive simulation, multi-metric evaluation, and human validation constitutes a significant methodological advance in the field.

• �� Select datasets: TweetSum, TODSum, SAMSum, representing task-oriented and chit-chat dialogues.
• �� Design perturbations: (a) utterance-level—typos, grammatical mistakes, vocabulary substitutions, speech disfluencies; (b) dialogue-level—repetitions, greetings, delays, splitting/merging utterances.
• �� Ground perturbations in natural conversation patterns (Moore & Arar, 2019).
• �� Fine-tune models (BART, T5, Pegasus) and evaluate zero-shot instruction-tuned models (DIAL-BART0, FLAN-T5).
• �� Generate summaries with beam search (size 5), compare original and perturbed dialogues.
• �� Metrics: BERTScore (for consistency, saliency), ROUGE-L, SummaC (for faithfulness).
• �� Human annotation to validate semantic preservation.
• �� Analyze impact of model size and training data augmentation.
• �� Use bootstrap sampling for statistical significance.
• �� Examine biases (lead, long, repetition) through targeted perturbations.

The experiments involve applying each perturbation type to test datasets, measuring performance changes across the three robustness metrics. The models are evaluated on unperturbed and perturbed inputs, with multiple runs to ensure statistical reliability. Human evaluation confirms the semantic integrity of perturbed dialogues. The study compares models of different sizes and training regimes, including zero-shot instruction tuning, to assess robustness variations. Ablation studies analyze the effect of training with perturbed data, revealing limited gains, thus emphasizing the need for more sophisticated robustness strategies. The experimental setup ensures comprehensive coverage of real-world conversational noise scenarios.

The results show a consistent decline in performance metrics—up to 30%—under various perturbations. Instruction-tuned models, such as FLAN-T5, exhibit the highest sensitivity, with performance drops exceeding 40% in some cases. Faithfulness suffers most, with hallucinations increasing notably. Dialogue biases manifest as models favoring initial utterances (lead bias) and lengthy segments (long bias). Increasing model size (e.g., T5-large vs. T5-base) does not significantly improve robustness, indicating that model complexity alone is insufficient. Training with limited perturbed data yields inconsistent improvements, highlighting the need for more targeted robustness techniques.

The findings inform the development of more reliable dialogue systems for customer service, virtual assistants, and meeting summarization. The evaluation framework can be adopted by industry to benchmark robustness, guiding model selection and training. Enhancing model resilience to natural variations will improve user trust and system accuracy, especially in noisy environments. Long-term, this research supports the creation of adaptive, context-aware dialogue models capable of handling diverse conversational styles and errors, facilitating broader deployment in real-world applications.

While the study covers a broad range of perturbations grounded in natural conversation, it cannot encompass all real-world complexities, such as multi-turn ambiguities or cultural language variations. The limited scope of training with perturbed data suggests that more advanced robustness techniques, like adversarial training, are needed. The focus on Transformer architectures leaves open questions about other model types or multi-modal inputs. Future work should explore these avenues to build truly resilient dialogue systems.