ETHOS: an Online Hate Speech Detection Dataset
ETHOS dataset offers hate speech detection based on YouTube and Reddit comments, using binary and multi-label classification.
Key Findings
Methodology
ETHOS dataset employs active sampling and Figure-Eight for annotation, ensuring data balance and diversity. Initial classification uses SVM and TF-IDF, followed by human validation.
Key Results
- On Figure-Eight, Fleiss' kappa values exceeded 0.75, indicating good annotator agreement.
- The dataset excels in multi-label classification tasks, especially in gender and race categories.
- Active sampling improved dataset representativeness and model generalization.
Significance
ETHOS provides high-quality annotated data for hate speech detection, supporting multi-label classification, addressing gaps in diversity and accuracy.
Technical Contribution
Introduces a new annotation protocol combining active learning and crowdsourcing, enhancing dataset quality and diversity.
Novelty
ETHOS is the first dataset focusing on multi-label hate speech detection, emphasizing semantic overlap among labels.
Limitations
- The dataset's small size may limit model generalizability.
- The annotation process relies on human annotators, introducing subjectivity.
Future Work
Future work could expand dataset size, exploring hate speech detection across more languages and cultural contexts.
AI Executive Summary
Hate speech is rampant on social media, and existing detection methods struggle with its diversity and complexity. The ETHOS dataset, through active sampling and Figure-Eight annotation, provides a high-quality multi-label dataset focused on YouTube and Reddit comments. Experimental results show ETHOS excels in multi-label classification tasks, particularly in gender and race categories. Although the dataset is small, its design and annotation protocol offer significant insights for future research. Future work will aim to expand the dataset and explore hate speech detection across more languages and cultural contexts.
Deep Analysis
Background
Hate speech spreads rapidly on social media, with existing detection methods often limited to binary classification, overlooking semantic overlap among labels. ETHOS addresses this gap with multi-label classification.
Core Problem
Existing datasets are often single-label, lacking diversity and accuracy, making it challenging to tackle complex hate speech.
Innovation
ETHOS dataset uses active sampling and Figure-Eight annotation to ensure data balance and diversity, focusing on semantic overlap in multi-label classification.
Methodology
- �� Initial classification with SVM and TF-IDF
- �� Human annotation via Figure-Eight
- �� Active sampling ensures data diversity
Experiments
Experiments use ETHOS for multi-label classification, evaluating annotator agreement and model performance. Fleiss' kappa assesses annotation quality.
Results
Results show ETHOS excels in multi-label classification, particularly in gender and race categories. Fleiss' kappa values exceed 0.75, indicating good annotator agreement.
Applications
ETHOS can be used for hate speech detection on social media platforms, helping manage user content more effectively.
Limitations & Outlook
The dataset's small size may limit model generalizability. The annotation process relies on human annotators, introducing subjectivity.
Plain Language Accessible to non-experts
Imagine a library with many books, each labeled with genres like 'fiction', 'history', 'science'. The ETHOS dataset is like a special library collecting and labeling 'books' of hate speech. This way, we can better identify and manage these expressions.
ELI14 Explained like you're 14
Imagine you're at school, and some classmates always use unfriendly words to attack others. The ETHOS dataset is like a super detective, helping teachers spot these unfriendly words and remind everyone to be careful. This way, the school can become more friendly and inclusive!
Glossary
Active Learning
A machine learning method that improves model performance by selectively labeling data.
Used to select the most informative comments for annotation.
Multi-label Classification
A classification task where each sample can have multiple labels.
ETHOS supports multi-label classification.
Figure-Eight
A crowdsourcing platform used for data annotation and validation.
Used in the annotation process of ETHOS.
Fleiss' kappa
A statistical measure for assessing the reliability of agreement among annotators.
Used to evaluate annotator agreement.
TF-IDF
A text vectorization technique that measures the importance of words.
Used in the initial classification stage.
Open Questions Unanswered questions from this research
- 1 How to maintain annotation quality on larger datasets?
- 2 How to achieve hate speech detection in multilingual environments?
Applications
Immediate Applications
Social Media Monitoring
Helps platforms identify and manage hate speech, improving user experience.
Long-term Vision
Global Application
Expand to multilingual and multicultural contexts, enhancing global social media safety.
Abstract
Online hate speech is a recent problem in our society that is rising at a steady pace by leveraging the vulnerabilities of the corresponding regimes that characterise most social media platforms. This phenomenon is primarily fostered by offensive comments, either during user interaction or in the form of a posted multimedia context. Nowadays, giant corporations own platforms where millions of users log in every day, and protection from exposure to similar phenomena appears to be necessary in order to comply with the corresponding legislation and maintain a high level of service quality. A robust and reliable system for detecting and preventing the uploading of relevant content will have a significant impact on our digitally interconnected society. Several aspects of our daily lives are undeniably linked to our social profiles, making us vulnerable to abusive behaviours. As a result, the lack of accurate hate speech detection mechanisms would severely degrade the overall user experience, although its erroneous operation would pose many ethical concerns. In this paper, we present 'ETHOS', a textual dataset with two variants: binary and multi-label, based on YouTube and Reddit comments validated using the Figure-Eight crowdsourcing platform. Furthermore, we present the annotation protocol used to create this dataset: an active sampling procedure for balancing our data in relation to the various aspects defined. Our key assumption is that, even gaining a small amount of labelled data from such a time-consuming process, we can guarantee hate speech occurrences in the examined material.