EVENT5Ws: A Large Dataset for Open-Domain Event Extraction from Documents

Event extraction is a crucial task in the field of natural language processing, aiming to identify the central aspects of events from text. With the increasing volume of information, automated event extraction becomes increasingly important, especially in emergency management and decision-making. Existing datasets are mostly closed-domain, based on predefined event schemas, limiting the algorithms' generalization capabilities to unseen event types. Moreover, there is a lack of large-scale manually verified datasets in open-domain settings, limiting the development of deep learning methods. To address this, researchers have developed EVENT5Ws, a large-scale, manually annotated, and ICR-verified open-domain event extraction dataset, aiming to support the extraction of unconstrained event types.

Existing event extraction datasets are primarily closed-domain, using predefined event schemas, limiting the algorithms' generalization capabilities to unseen event types. Moreover, there is a lack of large-scale manually verified datasets in open-domain settings, limiting the development of deep learning methods. As real-world events are diverse and constantly evolving, enumerating all possible event types is impractical, necessitating the development of open-domain event extraction datasets that support the extraction of unconstrained event types.

The development of the EVENT5Ws dataset has the following innovations: 1) It employs the 5Ws framework, covering the five key aspects of events: where, when, what, who, and why, supporting the extraction of unconstrained event types; 2) It follows a systematic annotation pipeline and is ICR-verified, ensuring the quality and reliability of the dataset; 3) It provides a new benchmark for evaluating existing pre-trained large language models on open-domain event extraction tasks.

• �� Coder Selection and Recruitment: Coders familiar with the geographical and cultural context were recruited through university courses, mailing lists, and student associations.

• �� Annotation Platform: Dataturks, an open-source web application, was used for text annotation.

• �� Annotation Guidelines: Clear guidelines were created, along with illustrative examples to help coders understand the task.

• �� Resolution Policy: A policy was established to systematically handle disagreements among coders, ensuring the final dataset's accuracy and reliability.

• �� Dataset Construction Process: Consisted of four steps: training, dataset preparation, annotation, and resolution of disagreements.

The experimental design includes evaluating several state-of-the-art pre-trained large language models on the EVENT5Ws dataset, including Gemma 3, Llama 3.1, Qwen 3, Mistral v0.3, and T5 Large. Experiments were conducted under zero-shot and five-shot prompting, using metrics such as exact match and ROUGE-L to evaluate models' performance on the five aspects: where, when, what, who, and why. Experiments also included testing generalization capabilities across different geographical and textual contexts.

Experimental results show that pre-trained large language models perform well on 'where', 'when', and 'who', but struggle with 'what' and 'why', highlighting the challenges of open-domain event extraction. Additionally, models demonstrate good generalization capabilities across datasets from different geographical and textual contexts, proving the potential of the EVENT5Ws dataset for developing generalizable algorithms. Through these experiments, the study offers valuable insights and recommendations for future large-scale dataset development.

Application scenarios for the EVENT5Ws dataset include: 1) Supporting the development of more complex algorithms to improve performance in open-domain event extraction; 2) Serving as a benchmark for evaluating existing pre-trained large language models; 3) Testing models' generalization capabilities across different geographical and textual contexts, advancing natural language understanding and downstream applications.

Despite the generalization capabilities of the EVENT5Ws dataset across different geographical contexts, challenges remain in handling 'what' and 'why', which may require more complex algorithms to improve performance. Additionally, the dataset's creation relies on manual annotation, which, despite ICR verification, may still contain subjective biases. As the dataset is primarily based on news reports, it may perform less effectively when dealing with other types of documents. Future directions include developing more complex algorithms, expanding the dataset to cover more types of documents and events, and exploring automated annotation methods to reduce subjective bias in manual annotations.