The Algorithmic Caricature: Auditing LLM-Generated Political Discourse Across Crisis Events

In recent years, social media platforms have become central infrastructures for political expression, mobilization, and contestation. However, these platforms have also become key sites of manipulation, including coordinated influence operations, misinformation campaigns, and automated amplification. The rapid diffusion of Large Language Models (LLMs) raises a new concern within this landscape: the possibility of generating large volumes of fluent, politically charged synthetic discourse that can imitate grassroots expression at scale. Existing AI-generated text detection methods have primarily focused on local textual signatures, such as token predictability, burstiness, repetition, or perplexity-based irregularities. While these methods can be useful, they are increasingly vulnerable to model improvements, paraphrasing, and stylistic adaptation. In settings such as political communication, where language is noisy, emotional, and event-dependent, a narrow focus on sentence-level cues may miss broader population-level distortions.

The core problem addressed in this paper is: how does synthetic political discourse differ from observed online populations during crisis events? Traditional AI text detection methods primarily focus on local linguistic features, which may fail to capture the overall distortion of synthetic discourse in terms of emotion, structure, and lexicon. This paper proposes a new Computational Social Science (CSS) framework aimed at auditing the social realism of synthetic discourse from a population level. By treating the discourse population rather than the individual text as the unit of analysis, the study connects generative AI evaluation to central CSS concerns: collective behavior, political communication, crisis response, and the measurement of online publics.

The core innovations of this paper include: 1) Proposing a new CSS framework for evaluating whether synthetic discourse reproduces the aggregate behavioral signatures of observed online publics; 2) Constructing a paired corpus of 1,789,406 posts covering nine political crisis events; 3) Introducing the 'Caricature Gap' as a simple event-level measure to summarize the differences between synthetic and observed discourse. Unlike previous methods focusing on local textual features, this paper emphasizes the overall distortion of synthetic discourse in terms of emotion, structure, and lexicon.

The methodology of this paper includes the following steps:

• �� Dataset Construction: Collecting observed and synthetic discourse for nine political crisis events, forming a paired corpus of 1,789,406 posts.

• �� Dimension Evaluation: Comparing divergence along four dimensions: emotional intensity, structural regularity, lexical-ideological framing, and cross-event dependency.

• �� Divergence Measurement: Using mean gaps and dispersion evidence to assess population distortion, introducing the 'Caricature Gap' as a simple event-level measure.

• �� Statistical Analysis: Conducting statistical analysis of differences in emotion, structure, and lexicon using VADER sentiment analysis and TF-IDF lexical analysis.

The experimental design includes the following aspects:

• �� Dataset: Constructing a paired corpus of 1,789,406 posts covering nine political crisis events.

• �� Baseline: Using observed discourse from social platforms as a baseline for comparison with synthetic discourse.

• �� Evaluation Metrics: Evaluating divergence along four dimensions: emotional intensity, structural regularity, lexical-ideological framing, and cross-event dependency.

• �� Hyperparameters: Using specific prompts and parameter settings in generating synthetic discourse to ensure alignment with event context.

The results analysis shows:

• �� Synthetic discourse is more negative in sentiment, with an average sentiment score of -0.215 compared to +0.018 for observed discourse.

• �� Structurally, synthetic discourse averages 23.08 words per post with a standard deviation of 11.29, while observed discourse averages 32.16 words with a standard deviation of 55.93, indicating higher structural consistency.

• �� In terms of lexical-ideological framing, synthetic discourse is more abstract and formalized, lacking the specific, event-dependent lexical markers found in observed discourse.

The findings of this study can be applied in the following scenarios:

• �� Political Communication Analysis: By evaluating the social realism of synthetic discourse, it helps identify and understand potential manipulation and influence in political communication.

• �� Social Media Monitoring: Provides a new tool for social media platforms to identify and filter potential synthetic discourse.

• �� Generative AI System Optimization: Offers a theoretical foundation for future applications of generative AI systems in social sciences, helping improve models to better reproduce observed emotional diversity and structural complexity.

The limitations of this paper include:

• �� The study is limited to nine specific crisis events, which may not comprehensively represent all types of political discourse.

• �� The generation of synthetic discourse relies on specific prompts and parameter settings, which may affect the generalizability of the results.

• �� Synthetic discourse shows greater distortion in fast-moving and decentralized crisis events, possibly due to the model's limitations in handling informal and heterogeneous discourse.