Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation
Distinguishing emulation and estimation tasks, base models excel in response generation, fine-tuned models in direct distribution prediction.
Key Findings
Methodology
This paper introduces a framework dividing human opinion simulation into two paradigms: emulation and estimation. Using Pew survey data, six paired models (e.g., Qwen3-14B, Olmo-3-7B) are evaluated via open-response generation and JSON distribution prediction. In emulation, base models generate individual responses aggregated into distributions; in estimation, fine-tuned models directly output distributions. Metrics include TVD and Wasserstein distance, assessing performance across demographic conditions.
Key Results
- Base models outperform fine-tuned counterparts in emulation, with lower Wasserstein distances (e.g., Qwen3-14B base: 0.492) and better demographic structure preservation, indicating stronger population diversity retention.
- Fine-tuned models excel in direct distribution estimation, with Claude Opus 4.6 achieving the lowest TVD (0.142), demonstrating superior calibration.
- The results reveal task-dependent performance: emulation favors response diversity, estimation favors accuracy, highlighting the importance of task-specific model selection.
Significance
This research clarifies the strengths and limitations of language models in simulating human opinions, guiding practitioners in choosing appropriate architectures for policy analysis, market research, and social science. By understanding the divergence in model capabilities, it addresses longstanding challenges of demographic sensitivity and persona collapse in population modeling, paving the way for more reliable and nuanced social simulations.
Technical Contribution
The paper formalizes the emulation-estimation distinction, systematically compares paired models, and introduces open-response emulation as an unbiased evaluation method. Empirical validation on real survey data demonstrates how model performance varies with task, offering a theoretical basis for task-driven model selection. It enriches evaluation metrics and emphasizes the importance of aligning model choice with downstream goals, advancing population-level behavioral modeling techniques.
Novelty
First to explicitly differentiate between emulation and estimation paradigms in population opinion modeling, revealing that base models excel in generating diverse responses, while fine-tuned models are better at direct distribution prediction. The introduction of open-response emulation as an unbiased assessment approach addresses prior methodological biases, providing a new lens for evaluating large language models in social science contexts.
Limitations
- Current evaluation is limited to static survey data; the models' ability to adapt to temporal opinion shifts remains untested.
- Performance in multi-turn dialogues and complex behavioral scenarios is not explored, limiting applicability to dynamic social interactions.
- Fine-tuning may introduce biases, reducing population diversity; balancing diversity and accuracy remains a challenge.
Future Work
Future research will explore models' performance in dynamic, multi-turn settings, incorporating temporal data to improve adaptation. Developing multi-task frameworks that balance emulation and estimation will enhance generalization. Additionally, efforts to maintain response diversity while improving demographic sensitivity are needed to better simulate real-world social behaviors.
AI Executive Summary
Large language models (LLMs) have shown promise in simulating human opinions, yet their capabilities are often misunderstood due to task ambiguity. Some studies highlight their ability to generate responses that align with survey data, while others report persona collapse and demographic insensitivity. This paper clarifies these conflicting results by distinguishing between two core tasks: emulation, where models generate individual responses that aggregate into population distributions, and estimation, where models directly predict these distributions.
Using Pew Research survey data, the authors evaluate six pairs of models—three base and three fine-tuned—across multiple demographic conditions. The findings reveal that base models are superior at emulation, producing response distributions that closely match human data and better preserve demographic structures. Conversely, fine-tuned models excel at direct distribution estimation, providing more accurate predictions when asked explicitly.
This distinction underscores the importance of aligning model choice with the intended application. For generating diverse, human-like responses—such as interactive agents—base models are preferable. For tasks requiring precise distributional estimates—like policy modeling—fine-tuned models are more suitable. The study introduces open-response emulation as an evaluation method free from positional biases, offering a robust tool for future research.
Overall, this work offers a nuanced understanding of large language models' strengths and limitations in population opinion simulation. It advocates for task-specific model selection and highlights avenues for future exploration, including dynamic opinion modeling and multi-task learning. By clarifying these distinctions, the paper advances both theoretical understanding and practical deployment of AI in social science and policy contexts.
Deep Dive
Abstract
Large language models are increasingly used to simulate human opinions, but prior work reports conflicting results: some studies find promising alignment with human survey data, while others find persona collapse and weak demographic sensitivity. We propose that much of this conflict stems from conflating two distinct tasks. We call the first task emulation, in which models generate individual responses that aggregate into a population distribution. We call the second task estimation, in which models directly predict the population distribution. Evaluating six matched base and post-trained models on the Pew American Trends Panel, we find that base models are the stronger emulators: they produce response distributions closer to human ground truth and better preserve demographic structure. Post-trained models are generally the stronger estimators, producing more accurate distributional predictions when asked directly. We argue that model selection for human simulation should be guided by whether the task requires generating text or predicting distributions.