Benchmarking Identity-Sensitive LLM Outputs for Surveillance and Security Robots

The evolution of large language models (LLMs) like GPT-3 and BERT has revolutionized natural language processing, enabling applications in dialogue systems, content creation, and embodied AI. In robotics, these models support early-stage design, interaction policies, and risk assessments, especially for social robots in public spaces. Prior works, such as the Bias Benchmark and Toxicity datasets, have identified biases in language models related to gender, race, and ethnicity. However, most studies focus on textual toxicity or stereotypes without linking these biases to embodied system design. Recent research emphasizes that social robots can inadvertently adopt gendered or racialized behaviors through appearance, voice, and language, influencing societal perceptions and ethical considerations. Despite advances, there remains a gap in systematically quantifying how demographic prompts influence the complexity and fairness of generated design descriptions, which are foundational to safe and equitable deployment.

The core challenge addressed is whether demographic prompts used in LLMs systematically affect the linguistic complexity of generated surveillance and security robot descriptions. Variations in language complexity can obscure safety instructions, reinforce stereotypes, or lead to discriminatory design choices. Existing evaluations lack a comprehensive, multi-metric approach to detect subtle biases that may propagate into physical embodiments or operational policies. The problem is compounded by the fact that prompts often include demographic or geographic inferences that may encode stereotypes, influencing subsequent design specifications. Addressing this issue is crucial for ensuring that AI-generated content supports fair, transparent, and socially responsible robotics, especially in sensitive public security contexts.

This work introduces a novel multi-metric evaluation framework combining six readability indices with mixed-effects statistical modeling to systematically analyze identity-conditioned variations in generated texts. It innovates by integrating demographic and geographic context augmentation within prompts, assessing their impact on linguistic complexity. The approach moves beyond traditional bias detection, focusing on surface-level text features that influence interpretability and downstream decision-making. It also pioneers the use of automated, scalable metrics to quantify subtle variations across multiple output components and identity categories, providing a comprehensive tool for bias auditing in embodied AI design. These innovations facilitate early detection and mitigation of potential biases, promoting responsible AI development.

• �� Constructed a dataset of 236 demographic identity labels spanning ability, age, gender, ethnicity, and geography.
• �� Designed two prompt types: a simple single-label prompt and an augmented prompt including inferred gender, borough, and ZIP code.
• �� Used Gemini 2.5 to generate seven-part robot descriptions: physical design, behavioral traits, speech, interaction, risk prediction, responsibilities, and recommendations.
• �� Calculated six readability metrics (FKGL, FRE, ARI, Gunning Fog, SMOG, CLI) for each output component.
• �� Applied mixed-effects models with fixed effects for prompt type, identity category, output component, and interactions; random intercepts for identity labels.
• �� Conducted statistical tests (Tukey adjustments) to compare conditions and categories, ensuring robust analysis of systematic differences.

The experiments involved generating 472 descriptions (236 identities × 2 prompts) with Gemini 2.5, each sampled once. The analysis focused on differences in readability scores across prompt conditions and identity categories, with particular attention to components related to safety and operational responsibilities. The models controlled for repeated measures and used post-hoc tests to identify significant effects. Additional analysis examined the impact of identity categories on text complexity, revealing consistent patterns across metrics. The experimental setup aimed to quantify how prompt augmentation influences linguistic features, providing insights into bias propagation at the design specification stage.

Results demonstrated that augmented prompts significantly improved readability (FKGL reduced by approximately 2 levels, p<0.01), especially in safety-critical sections like recommendations. Identity categories such as ability/disability and age produced descriptions with lower complexity scores, indicating easier comprehension. Conversely, cultural and geographic identities yielded more complex texts, potentially reflecting model biases. Output components like recommendations and responsibilities consistently had higher complexity scores, which could hinder stakeholder understanding and safety communication. These findings highlight that prompt design and demographic cues influence the linguistic form of generated descriptions, with implications for bias mitigation and ethical design.

The framework enables developers to evaluate and optimize robot design descriptions for fairness and clarity before physical implementation. It supports bias detection in early design stages, reducing risks of discriminatory features. The metrics can guide prompt engineering to produce equitable descriptions, especially in sensitive applications like surveillance. Long-term, integrating these tools into automated pipelines will facilitate responsible AI deployment, ensuring social acceptance and compliance with ethical standards. The approach also informs policy-making by providing quantifiable measures of bias in embodied AI systems, fostering trust and transparency in human-robot interactions.

The study's reliance on Gemini 2.5 limits generalizability; different models may exhibit different bias patterns. Single-sample generation per prompt restricts assessment of stochastic variability. Metrics focus on surface features, lacking semantic bias analysis. The generated descriptions are conceptual, not embodied robots, so real-world impacts remain unverified. Future work should include multiple models, repeated sampling, and semantic bias assessments, as well as physical prototype testing to validate the influence of textual variations on actual robot behavior.