LinGO: A Linguistic Graph Optimization Framework with LLMs for Interpreting Intents of Online Uncivil Discourse
LinGO uses linguistic graph optimization with LLMs to interpret online uncivil discourse, improving accuracy and F1 scores.
Key Findings
Methodology
LinGO combines linguistic structures and optimization techniques by decomposing language into multi-step components, identifying and optimizing error-prone steps. Evaluated using four optimization techniques: TextGrad, AdalFlow, DSPy, and RAG on a dataset from the 2022 Brazilian presidential election.
Key Results
- LinGO consistently improves accuracy and weighted F1 scores across all models, with RAG and Gemini achieving the best performance at 0.690 accuracy and 0.699 F1 score.
- Outperforms zero-shot, chain-of-thought, direct optimization, and fine-tuning baselines.
- RAG is most effective in selecting high-quality examples, significantly enhancing model performance.
Significance
LinGO significantly enhances LLMs' interpretive capabilities in complex semantic tasks by integrating and optimizing multi-step linguistic components. This approach not only improves incivility detection accuracy but also offers new insights for future complex semantic explanation tasks.
Technical Contribution
LinGO introduces a novel framework combining linguistic graph optimization with existing LLM techniques, providing new theoretical guarantees and engineering possibilities, especially in improving stability in multi-step tasks.
Novelty
LinGO is the first framework to systematically integrate linguistic structures with LLM optimization, significantly enhancing models' understanding of complex semantics, particularly in handling indirect expressions of incivility.
Limitations
- LinGO may still err in extremely complex linguistic structures, especially when contextual information is lacking.
- Requires extensive annotated data to ensure effective model training.
Future Work
Future work includes extending LinGO to other languages and domains, exploring more combinations of optimization techniques, and applying it to larger datasets.
AI Executive Summary
Detecting online uncivil discourse is crucial for maintaining safe and inclusive digital environments. However, existing classifiers often misinterpret posts containing uncivil cues but expressing civil intents, leading to inflated estimates of harmful incivility. LinGO introduces a linguistic graph optimization framework with LLMs, leveraging linguistic structures and optimization techniques to classify multi-class intents of incivility. LinGO decomposes language into multi-step components, identifies targeted steps causing errors, and iteratively optimizes prompts and examples for these steps. Evaluated on a dataset from the 2022 Brazilian presidential election, LinGO improves accuracy and weighted F1 scores across all models. RAG, when paired with the Gemini model, achieves the best performance. These findings demonstrate that incorporating multi-step linguistic components into LLM instructions and optimizing targeted components can help models explain complex semantic meanings, which can be extended to other complex semantic explanation tasks in the future.
Deep Analysis
Background
Detecting online uncivil discourse is a multidimensional challenge involving impoliteness, aggression, and threats to democratic values. While AI technologies have been employed to identify uncivil discourse, challenges remain in handling indirect expressions.
Core Problem
Existing models often misinterpret posts with uncivil cues but civil intents, leading to inflated estimates of incivility. This misinterpretation can affect the safety and inclusivity of online communities.
Innovation
LinGO introduces a novel framework by combining linguistic structures with optimization techniques, systematically integrating linguistic graphs with LLM optimization to significantly enhance models' understanding of complex semantics.
Methodology
- �� Decompose language into multi-step components using linguistic graphs
- �� Identify and optimize error-prone steps
- �� Combine four optimization techniques: TextGrad, AdalFlow, DSPy, and RAG
Experiments
Evaluated LinGO on a dataset from the 2022 Brazilian presidential election using three LLMs and four optimization techniques, comparing performance against different baselines.
Results
LinGO consistently improves accuracy and weighted F1 scores across all models, with RAG and Gemini achieving the best performance at 0.690 accuracy and 0.699 F1 score.
Applications
LinGO can be used for content moderation on social media platforms, helping to identify and classify uncivil discourse and improve online community safety.
Limitations & Outlook
LinGO may still err in extremely complex linguistic structures, especially when contextual information is lacking. Future improvements could include extending to other languages and domains.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking a meal. LinGO is like a smart assistant that helps you break down complex recipes into simple steps. Each step is a small task, like chopping vegetables or boiling rice. LinGO identifies which steps are prone to errors and optimizes them to ensure every dish is perfect. Just like in the kitchen, LinGO processes language step by step, optimizing to ensure accurate results.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game. Each puzzle has multiple steps, and you need to solve them one by one. LinGO is like a super helper that identifies which steps are most likely to go wrong and gives optimization tips. This way, you can solve puzzles faster and score high! LinGO processes language similarly, breaking down steps and optimizing to ensure every problem is solved correctly.
Glossary
Linguistic Graph
A structure that decomposes language into multi-step components to identify and optimize error-prone steps.
Used in LinGO to decompose and optimize language processing steps.
RAG (Retrieval-Augmented Generation)
A technique that selects the most relevant information through similarity matching to optimize example selection.
Used in LinGO to select high-quality examples to enhance model performance.
TextGrad
A technique that treats prompts or examples as trainable parameters and optimizes them through gradient descent.
Used in LinGO to optimize prompt text.
AdalFlow
An optimization technique using LLM-generated feedback as a gradient-like signal.
Used in LinGO to optimize prompts and examples.
DSPy
A framework that optimizes user-defined metrics through bootstrapping examples.
Used in LinGO for multi-step task optimization.
Open Questions Unanswered questions from this research
- 1 How to improve LinGO's accuracy in extremely complex linguistic structures?
- 2 What are the challenges of applying LinGO in multilingual environments?
Applications
Immediate Applications
Social Media Content Moderation
LinGO can be used for content moderation on social media platforms, helping to identify and classify uncivil discourse and improve online community safety.
Long-term Vision
Cross-Language Incivility Detection
Extending LinGO to multilingual environments to help global platforms better manage uncivil content.
Abstract
Detecting uncivil language is crucial for maintaining safe, inclusive, and democratic online spaces. Yet existing classifiers often misinterpret posts containing uncivil cues but expressing civil intents, leading to inflated estimates of harmful incivility online. We introduce LinGO, a linguistic graph optimization framework for large language models (LLMs) that leverages linguistic structures and optimization techniques to classify multi-class intents of incivility that use various direct and indirect expressions. LinGO decomposes language into multi-step linguistic components, identifies targeted steps that cause the most errors, and iteratively optimizes prompt and/or example components for targeted steps. We evaluate it using a dataset collected during the 2022 Brazilian presidential election, encompassing four forms of political incivility: Impoliteness (IMP), Hate Speech and Stereotyping (HSST), Physical Harm and Violent Political Rhetoric (PHAVPR), and Threats to Democratic Institutions and Values (THREAT). Each instance is annotated with six types of civil/uncivil intent. We benchmark LinGO using three cost-efficient LLMs: GPT-5-mini, Gemini 2.5 Flash-Lite, and Claude 3 Haiku, and four optimization techniques: TextGrad, AdalFlow, DSPy, and Retrieval-Augmented Generation (RAG). The results show that, across all models, LinGO consistently improves accuracy and weighted F1 compared with zero-shot, chain-of-thought, direct optimization, and fine-tuning baselines. RAG is the strongest optimization technique and, when paired with Gemini model, achieves the best overall performance. These findings demonstrate that incorporating multi-step linguistic components into LLM instructions and optimize targeted components can help the models explain complex semantic meanings, which can be extended to other complex semantic explanation tasks in the future.