On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

The rapid development of large language models (LLMs) like GPT-3 has transformed NLP, enabling tasks such as question answering, summarization, and reasoning at unprecedented scales. Chain-of-Thought (CoT) prompting emerged as a technique to enhance reasoning by guiding models to produce intermediate steps, significantly boosting performance on logical and mathematical tasks (Wei et al., 2022b; Kojima et al., 2022). However, as these models are increasingly deployed in socially sensitive contexts, concerns about embedded biases and toxicity have grown. Prior work (Caliskan et al., 2017; Nadeem et al., 2021) documented biases related to race, gender, and religion, which can perpetuate stereotypes and social inequalities. Existing bias detection methods are largely static, focusing on corpus-level analysis without considering the reasoning process. This gap underscores the need to evaluate how reasoning prompts influence bias propagation, especially in zero-shot settings where models generate outputs without fine-tuning. This study aims to fill this gap by systematically analyzing bias and toxicity in zero-shot CoT prompts, providing insights into how model size and prompt design affect social biases.

While CoT prompting improves logical reasoning, its application in socially sensitive tasks raises risks of bias amplification and toxic output generation. The core challenge lies in understanding how step-by-step reasoning influences the propagation of stereotypes and harmful behaviors. Larger models, trained on vast and potentially biased datasets, tend to exhibit stronger biases, which can be unintentionally reinforced during reasoning. This poses a significant problem for deploying AI in real-world scenarios involving marginalized groups or sensitive topics. The difficulty is compounded by the lack of comprehensive evaluation frameworks that capture the dynamic nature of bias during inference. Consequently, there is an urgent need to develop methods that can quantify, understand, and mitigate bias escalation in reasoning processes, ensuring AI systems are both capable and socially responsible.

This work introduces several key innovations: 1) transforming existing bias benchmarks into zero-shot reasoning tasks, enabling direct inference-based bias measurement; 2) empirically demonstrating that bias and toxicity increase with model size during CoT prompting, revealing a previously unquantified risk; 3) proposing explicit bias mitigation instructions that, when integrated into prompts, can reduce harmful outputs. These innovations differ from prior static bias detection by focusing on the reasoning process itself, providing a more nuanced understanding of bias propagation. The approach combines statistical analysis with large-scale empirical testing across GPT-3 variants, offering a comprehensive framework to evaluate and control biases during inference. This advances the field by highlighting the importance of prompt design and instruction tuning in mitigating social harms.

• �� Convert bias benchmarks (CrowS-Pairs, StereoSet, BBQ) and HarmfulQ into zero-shot reasoning prompts.
• �� Use GPT-3 models (text-davinci-002/003) with two prompt formats: standard and Chain-of-Thought (e.g., 'Let's think step by step').
• �� Generate multiple outputs per prompt, label outputs for bias and toxicity, and compute accuracy and bias rates.
• �� Apply explicit bias mitigation instructions in prompts to evaluate their effectiveness.
• �� Analyze the impact of model size (TD1, TD2, TD3) on bias amplification, controlling for prompt format.
• �� Perform statistical validation using confidence intervals, repeated experiments, and ablation studies to ensure robustness.

The experiments involve evaluating GPT-3 models across four benchmarks—CrowS-Pairs, StereoSet, BBQ, and HarmfulQ—under both standard and CoT prompts. Each model version (TD1, TD2, TD3) is tested with multiple prompt formats. Metrics include bias accuracy (percentage of outputs matching stereotypes or anti-stereotypes), toxicity encouragement rate, and bias amplification percentage. The models are prompted with carefully designed templates, and outputs are manually labeled for bias and harmful content. The study conducts ablation experiments with bias mitigation instructions, assesses the effect of model scale, and performs multiple runs to ensure statistical significance. Results are analyzed to identify bias trends and evaluate mitigation effectiveness.

• �� Chain-of-Thought prompting increases bias bias by 8.8 percentage points and toxic behavior encouragement by 19.4 points across models.
• �� Larger models (e.g., TD3) exhibit more pronounced bias amplification, with bias rates reaching 78% under CoT prompts.
• �� Explicit bias mitigation instructions reduce bias and toxicity outputs, but biases persist in larger models.
• �� Bias and toxicity outputs correlate strongly with training data biases, and mitigation strategies only partially address these issues.
• �� Bias amplification trends are consistent across different benchmarks and prompt formats, emphasizing the importance of prompt design and model scaling considerations.

• �� Implement bias detection and mitigation in AI-powered content moderation, ensuring safer online environments.
• �� Use in sensitive decision-making systems (e.g., healthcare, legal advice) to prevent biased or harmful outputs.
• �� Long-term, develop adaptive bias control techniques integrated into training and inference pipelines, fostering fairer AI deployment in diverse societal contexts.

• �� Current evaluation focuses mainly on GPT-3; other architectures may exhibit different bias behaviors.
• �� Metrics primarily measure stereotypical bias and toxicity encouragement, lacking deeper cultural or contextual analysis.
• �� Bias mitigation strategies, while effective, do not fully eliminate biases, especially in larger models, indicating the need for multi-faceted approaches.