Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs

The evolution of large-scale language models (LLMs) such as GPT-3, BERT, and their successors has significantly advanced NLP capabilities, enabling applications from chatbots to content generation. Despite these technological strides, societal biases embedded in training data—ranging from gender and racial stereotypes to disability misrepresentations—pose ethical challenges. Prior research like Bolukbasi et al. (2016) and Bender et al. (2021) has documented biases in word embeddings and model outputs, prompting efforts in bias evaluation and mitigation. However, the representation of disability remains underexplored, partly due to limited datasets and the complexity of societal perceptions. Existing bias assessments often focus on negative stereotypes, neglecting the subtler, positive biases that can be equally harmful. Recent works, such as Li et al. (2023), have begun to address these gaps, revealing that models tend to produce overly optimistic portrayals that obscure real-life struggles. This study builds on these insights, aiming to systematically analyze how models depict disability, considering both negative and positive biases, and to develop comprehensive evaluation tools that reflect societal realities.

Despite the widespread use of LLMs, their capacity to accurately and fairly represent marginalized groups like people with disabilities remains limited. Current models often generate stereotyped or overly sanitized descriptions, which can distort societal understanding and reinforce biases. The core challenge lies in the models' training data, which is often biased or unbalanced, leading to skewed representations. This issue is critical because AI outputs influence public perceptions, policy-making, and social attitudes. Addressing this requires precise bias detection, understanding the mechanisms of bias reinforcement, and developing effective correction strategies. The difficulty is compounded by the subtlety of positive biases, which are less obvious but equally impactful. Therefore, the key problem is designing evaluation frameworks that can detect and quantify both negative and positive biases, and implementing interventions that promote fair, nuanced representations of disability.

The study introduces a multi-dimensional bias evaluation framework combining sentiment analysis, thematic keyword frequency, and bias indices, applied systematically to both real and generated social media content. It innovates by integrating automated annotation tools with statistical comparison methods, enabling large-scale, nuanced bias characterization. Unlike prior work that primarily focused on negative stereotypes, this research emphasizes the risks of positive overcompensation, providing a balanced view of bias impacts. It also employs a comprehensive dataset of real social media posts from diverse disability-related subreddits, enhancing ecological validity. The framework's ability to detect subtle biases and over-idealizations represents a significant methodological advancement, offering a scalable approach for ongoing bias monitoring and correction in LLMs.

• �� Data collection: Gathered posts from Reddit subreddits related to disability and general topics, ensuring diversity. • Annotation: Used sentiment analysis (e.g., VADER), depression detection models (e.g., BERT-based classifiers), and keyword extraction to label posts automatically. • Prompt design: Crafted open-ended prompts for LLMs (GPT-4, Gemini-1.5F, Mixtral-8B) to generate social media posts, varying prompts to simulate different disability types and generic personas. • Response generation: Set temperature=1.0 for variability, producing 360 posts per model per condition. • Comparative analysis: Quantified differences in sentiment, keywords, and bias scores between real and generated data. • Statistical testing: Applied t-tests and ANOVA to verify significance of observed biases. • Bias assessment: Calculated bias indices, thematic divergence, and sentiment polarity differences to evaluate model fairness.

The experiments involved collecting real posts from six disability-focused subreddits, annotated for emotional tone and depression indicators. Generated datasets from GPT-4, Gemini-1.5F, and Mixtral-8B used identical prompts with controlled randomness. Metrics included sentiment polarity, keyword frequency, bias indices, and emotional diversity. Ablation studies tested the impact of different prompts and annotation thresholds. Results showed models favor positive stereotypes, with over 70% of descriptions emphasizing admirable traits, while real posts exhibited more negative emotions. The bias scores confirmed significant disparities, validating the framework's effectiveness in bias detection and quantification.

Quantitative analysis revealed that models tend to generate overly positive descriptions, with positive sentiment scores exceeding 85%, compared to 60% in real data. Keywords related to career and entertainment appeared 30% more frequently in generated texts for nondisabled personas, indicating bias reinforcement. Bias indices showed a 30% higher bias level in simulated content, reflecting over-idealization. The emotional analysis indicated a stark contrast: simulated posts rarely expressed negative emotions, whereas real posts contained 40% negative sentiment, highlighting the models' inability to capture complex emotional states. These results underscore the need for bias-aware training and evaluation.

The proposed bias evaluation framework can be integrated into model development pipelines to identify and mitigate social biases early. It supports creating fairer AI systems for social media moderation, mental health support, and inclusive content generation. Policymakers and developers can use these tools to ensure AI outputs do not reinforce harmful stereotypes, fostering societal trust. Long-term, this approach can guide the design of socially responsible AI, influencing standards and regulations for ethical model deployment across industries.

The dataset's cultural scope is limited to Reddit, potentially biasing results towards Western contexts. Model tuning was not exhaustively optimized, possibly affecting bias manifestation. The evaluation metrics, while comprehensive, do not fully capture societal mechanisms behind bias formation. Future work should incorporate cross-cultural datasets, sociological theories, and explainability techniques to deepen understanding and improve bias mitigation.