Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language
Yor-Sarc is the first gold-standard dataset for sarcasm detection in Yorùbá, with 436 annotated instances and a Fleiss' κ of 0.766.
Key Findings
Methodology
The study employed three native speakers to annotate 436 Yorùbá texts for sarcasm using a culturally informed annotation protocol. The protocol incorporated context-sensitive interpretation and community-informed guidelines. Inter-annotator agreement was evaluated using Fleiss' κ and Cohen's κ.
Key Results
- Result 1: Fleiss' κ reached 0.766, indicating 'substantial' agreement, with the A1–A2 pair achieving Cohen's κ of 0.8743, classified as 'almost perfect'.
- Result 2: 83.3% of instances achieved unanimous agreement, while 16.7% were preserved as soft labels for uncertainty-aware modeling.
- Result 3: The dataset sources include social media, news, and crowdsourcing, balancing formal and informal language contexts.
Significance
Yor-Sarc addresses the lack of resources for sarcasm detection in African languages, advancing semantic interpretation and culturally informed NLP. It sets a benchmark for high-quality annotation in low-resource languages.
Technical Contribution
This study introduces the first sarcasm detection dataset for Yorùbá, employs a multi-annotator protocol, and preserves annotation uncertainty through soft labels. It demonstrates the reliability of native speakers for complex semantic tasks.
Novelty
This is the first publicly available dataset for sarcasm detection in Yorùbá, addressing cultural and linguistic challenges unique to tonal languages.
Limitations
- Limitation 1: The dataset primarily covers social media and news, lacking face-to-face conversational contexts.
- Limitation 2: The dataset size is limited to 436 instances, which may constrain deep learning applications.
- Limitation 3: The dataset does not include multimodal data, limiting its applicability to multimodal sarcasm detection.
Future Work
Future work could expand the dataset's size and contextual diversity, incorporate multimodal data, and explore cross-lingual sarcasm detection models.
AI Executive Summary
Sarcasm detection is a challenging task in NLP, particularly for low-resource languages that lack annotated datasets. Yor-Sarc addresses this gap by introducing the first gold-standard dataset for sarcasm detection in Yorùbá, a tonal language spoken by over 50 million people. The dataset comprises 436 instances annotated by three native speakers using a culturally informed protocol that ensures high-quality annotations.
The study achieved substantial inter-annotator agreement, with a Fleiss' κ of 0.766 and a Cohen's κ of 0.8743 for the best annotator pair, surpassing benchmarks for English sarcasm datasets. The dataset sources include social media, news, and crowdsourced examples, capturing both formal and informal language contexts. Soft labels were used to preserve annotation uncertainty, enabling uncertainty-aware modeling.
Yor-Sarc not only advances sarcasm detection for Yorùbá but also sets a precedent for similar efforts in other low-resource languages. Future research could focus on expanding the dataset, incorporating multimodal data, and exploring cross-lingual applications to further enhance the field of African NLP.
Deep Analysis
Background
Sarcasm detection is a key NLP task due to its impact on sentiment analysis and opinion mining. While significant progress has been made in high-resource languages like English, low-resource languages, particularly African languages, lack annotated datasets for sarcasm detection. Yorùbá, a tonal language spoken by over 50 million people, has seen advancements in tasks like sentiment analysis and machine translation but remains underexplored in figurative language phenomena.
Core Problem
The core challenge in Yorùbá sarcasm detection is the absence of annotated datasets. Sarcasm involves complex semantic phenomena requiring cultural and contextual understanding. The unique tonal and morphological features of Yorùbá add to the difficulty, making it challenging to apply existing methods directly.
Innovation
The key innovations of Yor-Sarc include:
- �� Development of the first sarcasm detection dataset for Yorùbá, addressing a critical gap in African NLP.
- �� Creation of a culturally informed annotation protocol to ensure high-quality annotations.
- �� Introduction of soft labels to preserve annotation uncertainty, enabling more nuanced modeling.
Methodology
The methodology involved:
- �� Data collection: 436 Yorùbá texts from social media, news, and crowdsourcing.
- �� Annotation protocol: Three native speakers independently labeled texts for sarcasm using culturally informed guidelines.
- �� Agreement analysis: Inter-annotator agreement was quantified using Fleiss' κ and Cohen's κ.
- �� Data processing: Soft labels were assigned to 16.7% of instances with majority agreement to capture uncertainty.
Experiments
The experimental design included:
- �� Data sources: Social media (28.5%), news (65.4%), and crowdsourcing (3.9%).
- �� Agreement evaluation: Metrics like Fleiss' κ and Cohen's κ were used to assess annotation quality.
- �� Data analysis: Distribution of annotations and agreement patterns were analyzed to validate the protocol.
Results
Key results include:
- �� Fleiss' κ of 0.766, surpassing benchmarks for English sarcasm datasets.
- �� 83.3% of instances achieved unanimous agreement, with 16.7% preserved as soft labels.
- �� The A1–A2 pair achieved a Cohen's κ of 0.8743, classified as 'almost perfect'.
Applications
Yor-Sarc can be applied to:
- �� Sentiment analysis and opinion mining for Yorùbá social media and news.
- �� Cross-lingual sarcasm detection models, advancing NLP for low-resource languages.
Limitations & Outlook
Limitations include:
- �� Small dataset size (436 instances), limiting deep learning applications.
- �� Lack of conversational data, restricting context diversity.
- �� Absence of multimodal data, limiting multimodal sarcasm detection.
Plain Language Accessible to non-experts
Imagine you're chatting with a friend, and they say, 'Wow, you're such a genius!' but their tone makes it clear they're being sarcastic. Detecting sarcasm is about understanding the hidden meaning behind such statements.
For Yorùbá, this is even harder because it's a tonal language where meaning changes with pitch. Researchers collected 436 Yorùbá texts and asked three native speakers to label them as sarcastic or not. They created special guidelines to ensure the labels were consistent and culturally accurate.
The result is a high-quality dataset with excellent agreement among annotators. This dataset helps computers understand sarcasm in Yorùbá and serves as a model for studying other low-resource languages.
ELI14 Explained like you're 14
Imagine your friend says, 'Nice job!' but you know they're joking because you just spilled juice everywhere. That's sarcasm! Computers can't understand this easily, especially in languages like Yorùbá, which uses tones to change meanings.
Scientists wanted to teach computers to detect sarcasm in Yorùbá. They collected 436 sentences from social media and news, then asked three Yorùbá speakers to label them as sarcastic or not. They even made special rules to make sure everyone agreed.
The results were amazing! Most sentences were labeled the same way by all three people. This new dataset will help computers understand sarcasm in Yorùbá and even other languages in the future.
Glossary
Sarcasm Detection
The task of identifying when the literal meaning of a text differs from its intended meaning.
Used to analyze whether Yorùbá texts contain sarcasm.
Fleiss' κ
A statistical measure of inter-annotator agreement, adjusted for chance.
Used to evaluate the consistency of the three annotators.
Soft Labels
Labels that capture annotation uncertainty, often represented as probabilities.
Used to preserve disagreement among annotators.
Low-Resource Language
A language with limited annotated data and research resources.
Yorùbá is a typical example of a low-resource language.
Multimodal Data
Data combining multiple types such as text, images, and audio.
Future research could expand sarcasm detection to multimodal data.
Open Questions Unanswered questions from this research
- 1 How can the dataset be expanded to include more contexts, such as face-to-face conversations?
- 2 What methods can improve sarcasm detection accuracy in multimodal settings?
- 3 Can this approach be generalized to other low-resource languages?
Applications
Immediate Applications
Social Media Sentiment Analysis
Helps platforms detect and manage sarcastic content, improving user experience.
News Text Analysis
Supports media organizations in identifying and categorizing sarcastic reports.
Long-term Vision
Cross-Language Sarcasm Detection
Develop universal sarcasm detection models for multiple languages, enabling broader NLP advancements.
Abstract
Sarcasm detection poses a fundamental challenge in computational semantics, requiring models to resolve disparities between literal and intended meaning. The challenge is amplified in low-resource languages where annotated datasets are scarce or nonexistent. We present \textbf{Yor-Sarc}, the first gold-standard dataset for sarcasm detection in Yorùbá, a tonal Niger-Congo language spoken by over $50$ million people. The dataset comprises 436 instances annotated by three native speakers from diverse dialectal backgrounds using an annotation protocol specifically designed for Yorùbá sarcasm by taking culture into account. This protocol incorporates context-sensitive interpretation and community-informed guidelines and is accompanied by a comprehensive analysis of inter-annotator agreement to support replication in other African languages. Substantial to almost perfect agreement was achieved (Fleiss' $κ= 0.7660$; pairwise Cohen's $κ= 0.6732$--$0.8743$), with $83.3\%$ unanimous consensus. One annotator pair achieved almost perfect agreement ($κ= 0.8743$; $93.8\%$ raw agreement), exceeding a number of reported benchmarks for English sarcasm research works. The remaining $16.7\%$ majority-agreement cases are preserved as soft labels for uncertainty-aware modelling. Yor-Sarc\footnote{https://github.com/toheebadura/yor-sarc} is expected to facilitate research on semantic interpretation and culturally informed NLP for low-resource African languages.