CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks
CommunityBench evaluates community-level model alignment, revealing LLMs' limitations in modeling community preferences.
Key Findings
Methodology
CommunityBench, grounded in Common Identity and Common Bond theory, offers four tasks: preference identification, preference distribution prediction, community-consistent generation, and community identification. Built using Reddit data with 12,149 instances, it evaluates 17 foundation models.
Key Results
- Result 1: Current LLMs show limited ability in preference identification tasks, with a maximum accuracy of 40.34%.
- Result 2: In preference distribution prediction tasks, the lowest Jensen-Shannon Divergence is 0.1136.
- Result 3: In community identification tasks, the highest accuracy achieved is 34.54%.
Significance
This research provides the first systematic evaluation of community-level alignment, addressing gaps in existing individual and one-size-fits-all strategies, offering a new perspective for diversified and scalable model alignment.
Technical Contribution
Introduces community-level alignment as a middle ground, leveraging group dynamics to aggregate individual behavior signals while preserving cultural diversity, offering a new theoretical framework and evaluation benchmark.
Novelty
First to propose community-level alignment, distinct from individual and one-size-fits-all strategies, emphasizing intra-group consistency and cultural diversity.
Limitations
- Limitation 1: Models perform poorly in niche communities due to data scarcity, making alignment challenging.
- Limitation 2: The Reddit data used in experiments may not represent the diversity of all communities.
Future Work
Future work could explore better capturing preferences in niche communities and extend to data from other social platforms.
AI Executive Summary
In the study of aligning large language models, existing strategies often focus on individual or one-size-fits-all values, overlooking intra-group value consistency. CommunityBench proposes a middle ground by evaluating community-level alignment, leveraging group dynamics to aggregate individual behavior signals. Experimental results show that current models have limited ability to capture community-specific preferences. By systematically evaluating 17 foundation models, the study reveals the potential of community-level alignment in facilitating individual modeling, offering a new perspective for diversified and scalable model alignment. Nevertheless, models still need improvement in niche communities, and future work could explore broader data sources and more refined alignment strategies.
Deep Analysis
Background
Research on aligning large language models aims to ensure model behavior reflects human values. Existing strategies mainly focus on individual and one-size-fits-all values, overlooking intra-group value consistency. CommunityBench proposes community-level alignment to fill this gap.
Core Problem
Existing alignment strategies overlook intra-group value consistency, marginalizing minority norms. Individual models are costly and data-sparse, making widespread application difficult.
Innovation
CommunityBench proposes a middle ground by evaluating community-level alignment, leveraging group dynamics to aggregate individual behavior signals while preserving cultural diversity.
Methodology
- �� Use Common Identity and Common Bond theory to construct evaluation tasks
- �� Extract 12,149 instances from Reddit data
- �� Evaluate 17 foundation models on four tasks
Experiments
Experiments use Reddit data to evaluate 17 foundation models on preference identification, preference distribution prediction, community-consistent generation, and community identification tasks. Metrics include accuracy and Jensen-Shannon Divergence.
Results
Experimental results show that current models have limited ability to capture community-specific preferences. In preference identification tasks, the maximum accuracy is 40.34%; in preference distribution prediction tasks, the lowest Jensen-Shannon Divergence is 0.1136.
Applications
Community-level alignment can be used in social media analysis, personalized recommendation systems, and cultural diversity research, helping models better understand intra-group values.
Limitations & Outlook
Models perform poorly in niche communities due to data scarcity, making alignment challenging. The Reddit data used in experiments may not represent the diversity of all communities. Future work could explore broader data sources and more refined alignment strategies.
Plain Language Accessible to non-experts
Imagine a large party where each group has its own topics and ways of communicating. Existing models are like someone who only focuses on individuals, ignoring these groups' shared interests and values. CommunityBench is like a social expert who understands each group's unique preferences and communication styles.
ELI14 Explained like you're 14
Imagine organizing an event at school where each class has its own theme and style. Existing models are like a teacher who only focuses on individuals, ignoring the class's overall style. CommunityBench is like a teacher who understands each class's unique style and interests.
Glossary
Community-Level Alignment
An alignment strategy focusing on intra-group value consistency rather than individual or one-size-fits-all values.
Used in the paper to evaluate model performance in community-specific preferences.
Common Identity Theory
A theory suggesting that values and behaviors within a group are consistent.
Used to construct evaluation tasks in CommunityBench.
Jensen-Shannon Divergence
A metric to measure differences between probability distributions.
Used to evaluate model performance in preference distribution prediction tasks.
Preference Identification
Evaluates whether a model can identify a community's preference in a specific context.
One of the four tasks in CommunityBench.
Community Identification
Evaluates whether a model can identify community identity based on behavioral signatures.
One of the four tasks in CommunityBench.
Open Questions Unanswered questions from this research
- 1 How to improve model performance in niche communities to capture finer cultural differences?
- 2 How to extend CommunityBench to data from other social platforms to enhance diversity?
Applications
Immediate Applications
Social Media Analysis
Helps understand intra-group values and communication styles, enhancing personalized recommendation systems.
Long-term Vision
Cultural Diversity Research
Through community-level alignment, deeply study the values and behavior patterns of different cultural groups.
Abstract
Large language models (LLMs) alignment ensures model behaviors reflect human value. Existing alignment strategies primarily follow two paths: one assumes a universal value set for a unified goal (i.e., one-size-fits-all), while the other treats every individual as unique to customize models (i.e., individual-level). However, assuming a monolithic value space marginalizes minority norms, while tailoring individual models is prohibitively expensive. Recognizing that human society is organized into social clusters with high intra-group value alignment, we propose community-level alignment as a "middle ground". Practically, we introduce CommunityBench, the first large-scale benchmark for community-level alignment evaluation, featuring four tasks grounded in Common Identity and Common Bond theory. With CommunityBench, we conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences. Furthermore, we investigate the potential of community-level alignment in facilitating individual modeling, providing a promising direction for scalable and pluralistic alignment.