Gender and Race Bias in Consumer Product Recommendations by Large Language Models

The evolution of large language models (LLMs) such as GPT-3, BERT, and GPT-4 has revolutionized NLP applications, especially in personalized recommendations. Early efforts focused on improving accuracy and contextual understanding, with models trained on massive datasets from diverse sources. However, these datasets often contain societal biases, which models inadvertently learn and reproduce. Studies like Bolukbasi et al. (2016) and Caliskan et al. (2017) demonstrated that word embeddings encode gender and racial stereotypes. In recommendation systems, bias manifests as unequal exposure, reinforcing stereotypes and social disparities (Ekstrand et al., 2021). Existing bias detection methods primarily analyze static datasets or explicit stereotypes, leaving implicit biases in generated outputs underexplored. As LLMs are increasingly integrated into real-world applications, understanding and mitigating these biases becomes crucial to ensure fairness, transparency, and social trust.

The core issue addressed is the presence of implicit gender and racial biases in LLM-generated consumer product recommendations. These biases manifest subtly through linguistic patterns and category preferences, which can reinforce stereotypes and lead to unequal treatment of demographic groups. Traditional bias detection techniques are limited to static data or explicit stereotypes, failing to capture nuanced, implicit biases embedded in model outputs. This gap hampers efforts to develop fair AI systems, especially in personalized recommendation contexts where societal biases can influence user experience and decision-making. The challenge lies in designing scalable, sensitive detection methods that can quantify these subtle biases without disrupting recommendation diversity or accuracy.

This work introduces a multi-faceted bias detection framework combining prompt engineering, statistical analysis, and machine learning. First, tailored prompts guide GPT-4 to generate demographic-specific recommendations, ensuring diverse outputs. Second, the Marked Words method identifies words with significant statistical differences across groups, revealing cultural and societal stereotypes. Third, SVM classifiers extract the most distinctive linguistic features, providing interpretability. Fourth, JSD measures the divergence in word frequency distributions, quantifying the extent of bias. This integrated approach enables fine-grained, scalable detection of implicit biases in model outputs, surpassing prior static or explicit stereotype analyses. It provides a comprehensive toolkit for researchers and practitioners to evaluate and mitigate biases in AI recommendation systems.

• �� Design prompts instructing GPT-4 to generate top 10 product recommendations for specific demographic groups, outputting JSON format.
• �� Collect 225 recommendation samples across five racial and three gender groups, with 15 responses per group.
• �� Use Marked Words to compute weighted log-odds ratios, identifying words with significant statistical differences (z-score > 2).
• �� Apply support vector machine (SVM) classifiers to distinguish groups based on concatenated recommendation texts, extracting top features.
• �� Calculate Jensen-Shannon Divergence between word frequency distributions of different groups to quantify linguistic disparities.
• �� Validate results through statistical significance tests and cross-method consistency checks.
• �� Interpret findings to identify culturally and socially biased language patterns.
• �� Propose bias mitigation strategies based on identified keywords and features.

The experiments employed GPT-4 with temperature set to 1.0, generating recommendations for five racial groups (White, Asian, Black, Latino, Middle Eastern) and three gender groups (Male, Female, Nonbinary). Each group produced 15 responses, totaling 225 samples. The analysis involved calculating z-scores for word frequency differences, training linear SVM classifiers to identify top biased features, and computing JSD to measure distribution divergence. Results showed consistent bias patterns aligned with societal stereotypes, such as ‘hair’ and ‘oil’ for Black, ‘rice’ and ‘facial’ for Asian, and gendered terms like ‘skincare’ for women. The accuracy of group classification confirmed the robustness of detected biases. Ablation studies examined the impact of prompt phrasing on bias detection sensitivity, demonstrating the importance of prompt design in bias analysis.

The analysis revealed statistically significant biases: for example, ‘hair’ and ‘oil’ for Black groups (z > 2.5), ‘rice’ and ‘cream’ for Asian groups, and ‘skincare’ and ‘fragrance’ for women. SVM features achieved 98% accuracy in racial group classification and 70% for gender groups, indicating strong linguistic distinctions. JSD values confirmed substantial divergence in word distributions, with top contributing words aligning with cultural stereotypes. These findings demonstrate that LLMs tend to reinforce societal biases in recommendations, which could impact fairness and diversity in real-world applications. The multi-method approach provides a comprehensive bias assessment, guiding future mitigation efforts.

The methodology can be integrated into AI recommendation platforms to automatically detect and reduce biases, improving fairness and user trust. It supports developers in auditing models before deployment, ensuring compliance with ethical standards. Additionally, the framework can inform bias-aware training and fine-tuning strategies, fostering inclusive AI systems. Broader applications include social science research on stereotypes and policy-making for AI fairness. Long-term, this approach can contribute to building transparent, accountable AI ecosystems that promote social equity and reduce bias propagation across industries.

The current approach relies heavily on prompt design sensitivity, which may introduce variability in bias detection. It primarily analyzes textual outputs, potentially missing biases embedded in internal model representations. The experiments focus on specific demographic groups, and broader diversity may reveal additional biases. Computational costs for large-scale analysis are non-trivial, requiring optimization for real-time applications. Future work should explore multi-modal bias detection, causal inference methods, and more diverse datasets to enhance robustness and generalizability.