Proposes mBART, a multilingual Seq2Seq denoising pretraining, boosting low-resource MT BLEU by up to 12 points, with broad transfer capabilities.

Yinhan Liu, Jiatao Gu, Naman Goyal et al.

LayoutLM integrates text, layout, and visual features for document understanding, achieving 79.27% F1 on FUNSD.

Yiheng Xu, Minghao Li, Lei Cui et al.

PEGASUS introduces Gap Sentences Generation (GSG) as a pretraining objective, significantly improving abstractive summarization performance.

Jingqing Zhang, Yao Zhao, Mohammad Saleh et al.

TutorialVQA dataset for instructional video QA, including 6,000 manually collected triples.

Anthony Colas, Seokhwan Kim, Franck Dernoncourt et al.

Automatic prompt generation using relation mining and paraphrasing improves LM knowledge extraction accuracy from 31.1% to 39.6%.

Zhengbao Jiang, Frank F. Xu, Jun Araki et al.

Introduces SAMSum corpus with manually annotated abstractive dialogue summaries; models achieve higher ROUGE scores on dialogues than news, but human judgment contradicts.

Bogdan Gliwa, Iwona Mochol, Maciej Biesek et al.

Constructed ATCSpeech, a real multi-language ATC speech corpus, and trained deep learning ASR models achieving CER of 12-15%, enhancing aviation safety.

Bo Yang, Xianlong Tan, Zhengmao Chen et al.

Proposes Attention-Informed Mixed-Language Training (MLT), leveraging few parallel words for zero-shot cross-lingual dialogue, achieving significant performance gains.

Zihan Liu, Genta Indra Winata, Zhaojiang Lin et al.

Proposed ROCC method improves multi-hop QA accuracy, achieving 56.82% F1 on ARC dataset.

Vikas Yadav, Steven Bethard, Mihai Surdeanu

BP-Transformer uses binary partitioning to efficiently model long-range context, improving text classification and translation performance.

Zihao Ye, Qipeng Guo, Quan Gan et al.

Proposed span-based multi-sentence argument linking model achieving 68.3% F1 on RAMS, leveraging a new large-scale dataset.

Seth Ebner, Patrick Xia, Ryan Culkin et al.

TechQA dataset uses real IBM forum questions and Technote answers to advance domain adaptation research.

Vittorio Castelli, Rishav Chakravarti, Saswati Dana et al.

Constructed PrivacyQA dataset with 1750 questions; BERT-based models achieved 62.6% F1 in privacy policy QA.

Abhilasha Ravichander, Alan W Black, Shomir Wilson et al.

Large-scale GPT-based DialoGPT trained on 1.47B Reddit exchanges achieves near-human performance in open-domain dialogue.

Yizhe Zhang, Siqi Sun, Michel Galley et al.

CCNet pipeline automates extraction of high-quality monolingual datasets from Common Crawl, covering 174 languages with 532 billion tokens, boosting multilingual NLP.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau et al.

BART combines bidirectional encoder and autoregressive decoder with diverse noising pretraining, boosting NLP tasks including generation and comprehension.

Mike Lewis, Yinhan Liu, Naman Goyal et al.

Proposed a BERT-based weakly-supervised method, significantly improving factual consistency detection accuracy in summaries.

Wojciech Kryściński, Bryan McCann, Caiming Xiong et al.

MLQA creates a large-scale, multi-language aligned extractive QA benchmark across 7 languages, enabling comprehensive cross-lingual transfer evaluation.

Patrick Lewis, Barlas Oğuz, Ruty Rinott et al.

Proposes DistilBERT, a 40% smaller model with 97% performance, 60% faster inference, via pre-training distillation with triple loss.

Victor Sanh, Lysandre Debut, Julien Chaumond et al.

Introduces Megatron-LM, a model parallel approach enabling training of 8.3B parameter Transformer models with 76% scaling efficiency on 512 GPUs, using minimal code modifications.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri et al.