Mimetic Alignment with ASPECT: Evaluation of AI-inferred Personal Profiles
Proposes ASPECT, a psychometrically-guided pipeline for AI personal profile inference, achieving moderate alignment without per-user training.
Key Findings
Methodology
This study develops a prompt-based framework leveraging validated psychometric scales, specifically the CSI, to extract behavioral evidence from workplace data. The pipeline involves evidence extraction, item-level scoring, self-assessment, and scenario response evaluation. Using large language models (OpenAI’s GPT-3), the system identifies behavioral instances linked to communication constructs, scores each item on a 1-5 scale with rationales, and constructs structured, interpretable profiles. Experiments with 20 participants, 1840 paired ratings, and 600 scenario responses demonstrate the method’s capacity to produce profiles with moderate correlation to self-assessments and preferences favoring ASPECT-generated responses over baselines. The evidence-linked review process enables individuals to identify biases and recalibrate their self-ratings, enhancing control and transparency.
Key Results
- The inferred profiles showed an average correlation coefficient of approximately 0.45 with self-assessments, indicating moderate alignment. Participants preferred ASPECT responses in aggregate (~65%), though individual differences were notable. The system exhibited a positivity bias, overestimating socially desirable traits. Participants’ review process allowed them to recognize and correct biases, leading to improved personal alignment. Response preference analysis confirmed that personalized responses grounded in evidence outperformed generic and self-report baselines, especially in expressing style consistency.
- Behavioral evidence linking and item scoring resulted in responses rated 4.2/5 on average, surpassing baselines. The bias analysis revealed tendencies toward overly positive traits, suggesting future bias correction mechanisms are needed. Variability across individuals highlighted the importance of personalized evidence review, which effectively enhanced profile accuracy and user trust. The experimental results validate the approach’s feasibility for scalable, interpretable communication profiling.
- Individual differences in bias sensitivity and scenario context influenced outcomes. The evidence-linked review process proved crucial for calibration, enabling users to negotiate their profiles actively. Overall, the system demonstrated promising potential for transparent, controllable AI personalization, with scope for further refinement in bias mitigation and multi-modal data integration.
Significance
This work advances AI personalization by integrating validated psychometric tools with behavioral data, creating interpretable, adjustable communication profiles. It addresses critical issues of model opacity and user control, paving the way for trustworthy AI agents capable of representing individuals accurately in social and professional contexts. The evidence-grounded approach enhances transparency, enabling users to verify and modify their profiles, thus fostering greater acceptance and ethical deployment of personalized AI systems. It also offers a scalable framework adaptable across domains, from workplace assistants to virtual companions, aligning AI behavior more closely with human communication nuances.
Technical Contribution
The core innovation lies in operationalizing psychometric scales as prompt templates, coupling behavioral evidence extraction with item-level scoring in a multi-stage pipeline. This approach ensures interpretability and individual control, contrasting with preference-based methods like RLHF or DPO that produce opaque reward functions. The evidence linking process enhances traceability, while the structured profile enables direct inspection and adjustment. The method’s modular design allows transferability across models and domains, establishing a new paradigm for transparent, evidence-based AI personalization.
Novelty
This is the first framework to embed validated communication psychometric scales into large language model-based profiling, grounded in behavioral evidence from real workplace data. Unlike prior work that relies solely on self-report or broad behavioral predictions, ASPECT links specific behavioral instances to construct interpretable, adjustable communication profiles. Its evidence-based, multi-stage process offers a novel solution to the challenge of scalable, transparent AI personalization, bridging psychometrics and large-scale language modeling.
Limitations
- The system exhibits a positivity bias, overestimating socially desirable traits, which may reduce profile authenticity. The sample size is limited, affecting generalizability. Evidence extraction depends on text quality; noisy data can impair accuracy.
- Individual differences in bias sensitivity mean some users may find the profiles less trustworthy. The current process involves manual review steps, limiting scalability. Model responses still show bias tendencies, requiring further bias mitigation strategies.
- The approach relies heavily on textual behavioral data, which may not capture all communication nuances. Future work should incorporate multimodal signals and real-time adjustments to improve robustness and fairness.
Future Work
Future directions include integrating multimodal data such as speech and facial expressions to enrich behavioral evidence, developing real-time profile updating mechanisms, and expanding to diverse cultural contexts. Incorporating active learning and user feedback loops can further refine profile accuracy and user trust. Additionally, exploring automated bias correction and extending the framework to broader social domains will enhance its applicability and fairness.
AI Executive Summary
In the rapidly evolving landscape of AI-driven communication, creating personalized, trustworthy, and interpretable models remains a key challenge. Existing approaches often rely on shallow prompts or costly fine-tuning, which either lack depth or scalability. This study introduces ASPECT, a novel pipeline that leverages validated psychometric scales—specifically the Communication Styles Inventory (CSI)—to construct structured, evidence-grounded personal profiles without individual training. By guiding large language models (LLMs) through evidence extraction, item scoring, and participant review, ASPECT produces profiles that reflect authentic communication traits. The process emphasizes transparency, allowing individuals to verify and adjust their profiles, thus fostering trust and control.
Experimental validation with 20 participants demonstrated that ASPECT-generated profiles achieved moderate correlation with self-assessments (average r≈0.45) and responses grounded in these profiles were preferred over generic and self-report baselines (~65%). The linked evidence enabled participants to identify biases, recalibrate self-ratings, and negotiate contextually appropriate representations. These findings highlight the potential of combining psychometric rigor with behavioral data to develop scalable, interpretable AI personalization systems.
While promising, the approach faces limitations such as positivity bias and reliance on textual data quality. Future work aims to incorporate multimodal signals, real-time updates, and bias mitigation strategies. Overall, ASPECT offers a significant step toward transparent, controllable AI agents capable of faithfully representing individual communication styles, with broad implications for workplace, social, and personal AI applications.
Deep Dive
Glossary
Psychometric Scale (心理测量尺度)
一种标准化工具，用于测量个体的心理特质或行为特征，具有验证的信度和效度。
用作构建沟通特质档案的结构化基础。
Evidence Linking (证据链接)
将行为实例与特质评分关联的过程，确保模型评分依据真实行为证据。
在证据提取和项评分阶段应用。
Item-level Scoring (项级评分)
对每个特质尺度中的单个项目进行独立评分，提供细粒度的特质描述。
实现可解释、可调节的沟通档案。
Large Language Models (大型语言模型)
基于深度学习的自然语言处理模型，能生成连贯、上下文相关的文本。
用于行为证据识别和评分。
Bias Bias (偏差偏向)
模型在沟通特质中倾向于高估社会期望的积极特质。
影响档案真实性和可信度。
Open Questions Unanswered questions from this research
- 1 如何在多模态数据（如语音、面部表情）中融合行为证据，提升沟通特质的准确性和丰富性。未来研究需解决多源信息整合与实时更新的技术难题。
Applications
Immediate Applications
职场沟通代理
为企业提供个性化、透明的AI沟通助手，帮助员工更有效表达自己，提升团队协作效率。
虚拟个人助理
构建能理解用户沟通风格的虚拟助手，增强人机交互的自然性和信任感。
Long-term Vision
可信AI系统普及
推动个性化、可调节的AI在教育、医疗、社交等多领域的应用，实现更人性化的智能服务。
Abstract
AI agents that communicate on behalf of individuals need to capture how each person actually communicates, yet current approaches either require costly per-person fine-tuning, produce generic outputs from shallow persona descriptions, or optimize preferences without modeling communication style. We present ASPECT (Automated Social Psychometric Evaluation of Communication Traits), a pipeline that directs LLMs to assess constructs from a validated communication scale against behavioral evidence from workplace data, without per-person training. In a case study with 20 participants (1,840 paired item ratings, 600 scenario evaluations), ASPECT-generated profiles achieved moderate alignment with self-assessments, and ASPECT-generated responses were preferred over generic and self-report baselines on aggregate, with substantial variation across individuals and scenarios. During the profile review phase, linked evidence helped participants identify mischaracterizations, recalibrate their own self-ratings, and negotiate context-appropriate representations. We discuss implications for building inspectable, individually scoped communication profiles that let individuals control how agents represent them at work.