Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
PALMS uses small curated datasets to steer large language models toward societal values, reducing bias and toxicity effectively.
Key Findings
Methodology
PALMS employs an iterative process involving defining target values, designing prompts, generating annotated samples, fine-tuning models, and evaluating with toxicity scores, human ratings, and co-occurrence bias analysis. It uses prompt engineering to create small, targeted datasets, which are then used for model fine-tuning. The process is repeated, with dataset expansion based on evaluation feedback. Experiments across GPT-3 sizes (125M to 175B parameters) demonstrate that larger models benefit more from this approach. The core algorithms include prompt-based sample generation, supervised fine-tuning, and multi-metric evaluation, integrating Perspective API toxicity scores and human assessments to guide improvements.
Key Results
- In the 175B parameter GPT-3 model, PALMS significantly reduced toxicity scores (by 0.2 on average, p<0.01), increased human evaluation scores by over 20%, and produced more neutral bias word distributions across gender, religion, and race categories. These improvements indicate a substantial alignment with societal values.
- Compared to baseline models, control models fine-tuned on high-quality but non-targeted data showed limited bias reduction, highlighting the effectiveness of targeted value-based datasets. Model capabilities remained within 1% accuracy, demonstrating minimal performance trade-offs.
- Model size correlates positively with adjustment effectiveness, suggesting that larger models require fewer samples for significant behavioral shifts, opening pathways for scalable value alignment in future large-scale models.
Significance
This work provides a practical, low-cost framework for aligning large language models with societal norms, addressing critical issues of bias, toxicity, and harmful stereotypes. It demonstrates that targeted fine-tuning with small datasets can produce meaningful behavioral changes, especially in large models, thus supporting responsible AI deployment. The approach bridges the gap between model capability and social responsibility, facilitating broader acceptance and safer applications of AI systems across diverse cultural contexts.
Technical Contribution
The key innovation lies in integrating prompt-based sample generation with iterative fine-tuning and multi-metric evaluation, enabling precise control over model behavior with minimal data. The methodology departs from traditional large-data training, emphasizing value-driven adjustments. It leverages the scaling law that larger models are more amenable to behavioral shifts, providing a theoretical basis for future research. The multi-dimensional evaluation framework ensures robustness and transparency in the adjustment process.
Novelty
This is the first systematic framework combining small, curated datasets with iterative multi-metric evaluation to steer large language models toward societal values. Unlike prior approaches relying solely on large-scale data or rule-based filters, PALMS emphasizes targeted, explainable adjustments, making it adaptable across cultural contexts. Its demonstration across GPT-3 sizes underscores its scalability and practical relevance.
Limitations
- The approach is validated only in English, limiting cross-cultural applicability without adaptation. Cultural differences in values require customized datasets.
- Sample creation relies on manual prompt design, which introduces subjectivity and limits scalability. Broader topic coverage remains challenging.
- Behavioral adjustments may be less effective for deeply ingrained biases or complex ethical issues, necessitating further research into multi-modal and multi-source data integration.
Future Work
Future research will explore multi-language and multi-cultural adaptations, automated prompt generation, and multi-modal data integration to enhance generalizability. Investigating model interpretability and explainability will also be prioritized, ensuring transparent value alignment. Additionally, scaling laws for behavioral adjustment will be further formalized to optimize sample efficiency in larger models.
AI Executive Summary
Large language models like GPT-3 have revolutionized natural language processing, achieving unprecedented performance across diverse tasks. However, their deployment raises significant ethical concerns, notably biases, toxicity, and harmful stereotypes embedded in their outputs. Traditional mitigation strategies—such as filtering or large-scale retraining—are often costly, inefficient, and insufficiently precise. Addressing these challenges requires innovative approaches that can steer models toward socially acceptable behaviors without compromising their capabilities.
This paper introduces PALMS (Process for Adapting Language Models to Society), a novel framework that leverages small, carefully curated datasets to align model outputs with societal values. The core idea is to define sensitive topics, articulate desired behaviors, craft prompts, generate annotated samples, and iteratively fine-tune the model. The evaluation employs multiple metrics—toxicity scores from Perspective API, human ratings, and bias word co-occurrence analysis—to ensure comprehensive assessment. Experiments across GPT-3 models of varying sizes demonstrate that larger models (up to 175B parameters) respond more effectively to PALMS adjustments, with toxicity scores decreasing by 0.2 points on average and human evaluation scores improving by over 20%. Notably, the behavioral adjustments do not impair the models’ core capabilities, which remain within 1% of original accuracy.
The significance of this work lies in its demonstration that targeted, low-resource fine-tuning can substantially improve model social responsibility. This approach offers a scalable, cost-effective pathway for deploying AI systems aligned with diverse societal norms, especially as models continue to grow in size and influence. While limitations include language scope and sample subjectivity, future directions involve multi-language adaptation, automation, and interpretability enhancements. Overall, PALMS paves the way for more responsible, inclusive AI, balancing technical innovation with social impact.
Deep Dive
Abstract
Language models can generate harmful and biased outputs and exhibit undesirable behavior according to a given cultural context. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset.