DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
DUET framework leverages linear decodability of emotion in hidden states, combining hidden space bias and mel-space gradient guidance for controllable emotional TTS without retraining.
Key Findings
Methodology
This study reveals that emotion signals in pretrained diffusion and flow-matching TTS models are represented as linearly decodable directions in frozen hidden states, nearly orthogonal to speaker identity vectors. Based on this, DUET framework identifies the most emotion-discriminative layer via probing, extracts emotion directions from weight matrices, and applies a combined intervention: shifting hidden states along the emotion vector and refining mel spectrograms through gradients backpropagated from an external emotion recognizer via a differentiable vocoder. This process enables fine-grained, training-free emotion control during inference, compatible across diverse architectures.
Key Results
- Across five diverse pretrained TTS architectures (DiT, Matcha-TTS, GradTTS, ProDiff, StableTTS) and three datasets (ESD, CREMA-D, IEMOCAP), DUET outperforms 10 supervised emotional TTS baselines, achieving an average emotion accuracy of 75.5%, with the highest human-rated emotion appropriateness. Notably, DUET maintains balanced performance across emotions and speakers, with emotion signals forming well-separated clusters in hidden space, confirmed by probe accuracy peaking at middle layers. Deployment on an Ameca humanoid robot demonstrated highly expressive, emotionally rich speech, validating practical applicability.
- Results also show that emotion accounts for only 8.5% of hidden state variance but can be effectively manipulated via linear biasing. The dual intervention approach, combining hidden space bias and mel-space gradient guidance, significantly improves naturalness and emotional fidelity, especially in complex emotions like anger. The method exhibits robust generalization and can be integrated into real-time embodied agents.
Significance
This work addresses a fundamental challenge in neural TTS—controlling subtle emotional nuances in a model-agnostic, inference-time manner. By uncovering the geometric structure of emotion in hidden states, DUET enables precise, efficient, and versatile emotional modulation without costly retraining or fine-tuning. Its plug-and-play nature facilitates broad adoption in personalized virtual assistants, expressive avatars, and social robots, pushing the boundary of emotionally intelligent speech synthesis. The approach also offers new insights into the internal representations of large generative models, opening avenues for multi-modal emotion control and interpretability.
Technical Contribution
The core technical innovation lies in the systematic discovery that emotion in pretrained diffusion and flow-matching TTS models is represented as a linearly decodable, nearly orthogonal direction in hidden space. The paper introduces a probing-based method to identify the optimal layer and direction, and constructs a combined emotion steering vector by aggregating centroid and discriminative directions via SVD. The dual intervention—hidden state bias and mel gradient guidance—is integrated into a unified per-step update, leveraging the differentiability of the vocoder. This framework generalizes across architectures, enabling training-free, high-precision emotion control during inference.
Novelty
This work is the first to explicitly demonstrate that emotion signals in frozen pretrained TTS models are linearly decodable and nearly orthogonal to speaker identity, providing a geometric basis for inference-time emotion control. Unlike prior methods relying on retraining or explicit labels, DUET exploits internal model geometry, offering a universal, plug-and-play solution. Its combination of hidden space bias and mel-space gradient guidance represents a novel, effective strategy for fine-grained, natural emotional speech synthesis.
Limitations
- The approach's effectiveness diminishes for emotions with rapid, subtle temporal cues such as anger, due to uniform biasing across sequence positions. Handling such emotions may require temporally adaptive strategies.
- Dependence on external emotion recognizers introduces potential errors, especially in noisy or ambiguous contexts, affecting control fidelity.
- Computational overhead from probing and gradient backpropagation, though manageable, may limit real-time deployment in resource-constrained scenarios.
Future Work
Future directions include integrating multi-modal cues (visual, contextual) for richer emotion control, developing adaptive temporal biasing mechanisms for complex emotions, and exploring self-supervised methods to improve emotion direction robustness. Extending DUET to multi-emotion and multi-speaker scenarios, as well as real-time interactive applications, are promising avenues for advancing emotionally intelligent speech synthesis.
AI Executive Summary
Recent advances in neural speech synthesis, particularly diffusion and flow-matching models, have achieved remarkable naturalness. However, explicit control over emotional expression remains a significant challenge, as emotional signals are subtly embedded within high-dimensional hidden states and entangled with speaker identity. Traditional approaches rely heavily on retraining or fine-tuning, which are costly and lack flexibility. To address this, the present work introduces DUET, a unified, inference-time framework that leverages the geometric structure of hidden states to enable precise, training-free emotion control.
The key insight is that emotion information in pretrained models manifests as a linearly decodable direction in the hidden space, nearly orthogonal to speaker identity vectors. By probing the most discriminative layer and extracting the emotion direction via weight analysis, DUET applies a bias to steer hidden states along this direction during each denoising step. Simultaneously, it refines spectral details by backpropagating gradients from an external emotion recognizer through a differentiable vocoder, ensuring spectral fidelity and emotional appropriateness.
Extensive experiments across five architectures and three datasets demonstrate that DUET significantly outperforms supervised baselines, achieving an average emotion accuracy of 75.5%. Human evaluations confirm its superior naturalness and emotional expressiveness. Deployment on an Ameca humanoid robot further showcases its potential for embodied affective interaction, producing richly expressive speech in real-time.
This work fundamentally advances the understanding of internal representations in generative speech models, providing a versatile, plug-and-play solution for emotional speech synthesis. Its implications extend to personalized virtual assistants, social robots, and multi-modal affective systems, paving the way for more natural, emotionally engaging AI interactions.
Deep Analysis
Background
近年来，深度学习推动语音合成技术不断突破，尤其是扩散模型（如Score-based U-Net）和流匹配模型（如DiT、Transformer流匹配）在自然度方面取得显著提升。然而，情感表达的精细控制仍是难点。传统方法多依赖微调或标签引导，存在成本高、调控不灵活的问题。预训练模型的出现，为语音合成提供了强大基础，但情感信号在隐藏空间中的微妙表现限制了调控的精度。近年来，研究者开始关注模型内部的几何结构，试图利用线性偏移实现情感操控，但缺乏系统性分析和通用框架。
Core Problem
当前预训练TTS模型在情感控制方面面临两大难题：一是情感信号在隐藏状态中的微妙表现，难以实现精准调控；二是情感与说话人身份高度耦合，导致调控效果不稳定。传统微调方法成本高，泛化能力差，难以满足多场景、多情感的需求。如何在保持语音自然度的基础上，实现多样化、细粒度的情感表达，成为亟待突破的关键。
Innovation
本研究的创新点主要包括：1）发现情感在预训练扩散和流模型中的隐藏状态表现为线性可解的方向，且与说话人身份几乎正交；2）提出DUET框架，结合隐藏空间偏移和mel空间梯度引导，实现无需微调的情感操控；3）通过探测最优层和方向，构建情感偏移向量，确保调控的鲁棒性和泛化能力。这些创新突破了传统微调依赖，提供了模型内部几何结构的深层理解。
Methodology
- �� 通过线性探测器在不同隐藏层识别最具情感判别能力的层和方向；• 利用线性偏移在该层沿情感方向偏移隐藏状态，调节情感表达；• 结合可微声码器，反向传播外部情感识别器的梯度，优化频谱细节；• 在每一步生成中，结合隐藏空间偏移和mel空间引导，形成单步多模态调控机制；• 采用多架构、多数据集验证其通用性和效果。
Experiments
在五个不同架构（DiT、Matcha-TTS、GradTTS、ProDiff、StableTTS）上，使用ESD、CREMA-D、IEMOCAP数据集，比较10个有监督基线。指标包括情感识别准确率和人类感知评分。调控方向由外部情感识别器提供，超参数在不同模型上调优。还进行消融实验验证隐藏空间偏移和mel空间引导的互补性。
Results
在所有模型和数据集上，DUET显著优于有监督基线，平均情感准确率提升至75.5%，最高达83.7%。在复杂情感如愤怒表现优越，尤其在多说话人数据集表现出良好鲁棒性。部署于Humanoid机器人中，生成的语音情感丰富，获得用户高度认可。消融分析显示，双空间调控策略协同作用显著提升自然度和情感适切性。
Applications
该技术适用于智能语音助手、虚拟主播、情感机器人等场景，实现个性化、多样化的情感表达。无需模型微调，便于快速部署和调节，适合多模态交互系统的实时情感调控。
Limitations & Outlook
目前对极端情感（如愤怒）调控仍有不足，部分情感表现受限于模型内部特征。高质量调控依赖外部情感识别器，识别误差可能影响效果。未来需结合多模态信息和自监督学习，提升鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在厨房做菜，调味料代表情感，菜肴代表语音。传统方法需要重新调整每次的调味料比例（模型微调），很麻烦。现在，有一种新办法，就像用一根魔法棒，只需轻轻一挥，就能让菜变得更咸或更甜，不用重新调料。这根魔法棒就是DUET，它通过观察菜肴的内部结构（隐藏状态），找到调味的最佳方向，然后轻松调整，最后用一种神奇的调味喷雾（梯度引导）让味道更完美。这样，不管做什么菜，只要挥挥魔法棒，就能让味道变得丰富多彩，既省时又自然。这就像让机器人用“魔法”表达不同的情感一样，既方便又灵活。
ELI14 Explained like you're 14
想象你在玩一款游戏，你可以让你的角色变得更勇敢或更搞笑，但不需要重新开始游戏。科学家们发现，电脑里的“记忆”其实像一堆隐藏的线条，只要沿着特定的方向移动，就能让角色表现出不同的情感。以前，要让电脑学会这些情感，要花很多时间重新训练模型，但现在，他们找到了一种“捷径”。他们用一种叫DUET的方法，直接在模型内部找到那些代表“开心”、“愤怒”等情感的线条，然后轻轻一偏，就能让电脑说话更有感情。更酷的是，他们还用一种神奇的“引导”方法，让电脑在生成声音时，加入这些情感，让声音听起来更真实、更有趣。这个技术可以让机器人、虚拟助手变得更有人情味，就像和朋友聊天一样自然！
Abstract
Diffusion and flow-matching based text-to-speech (TTS) models excel in naturalness but often lack explicit emotion control, as emotional signals remain entangled with speaker identity. We discover that emotion embedding emerges as a linearly decodable direction of frozen hidden states, nearly orthogonal to the direction embedding speaker identity. This inspires a plug-and-play framework DUET for emotion control over pretrained diffusion and flow-matching based TTS models. During generation, DUET unifies dual-space control to achieve fine-grained emotion intervention in a single per-step update: hidden space steering shifts generation along the target emotion direction, while mel-space guidance refines spectral details through gradients backpropagated from a differentiable vocoder. We validate DUET on five architecturally diverse pretrained TTS backbones across three datasets, where it outperforms 10 supervised state-of-the-art emotional TTS baselines across paradigms and achieves the highest human-rated emotion appropriateness. To further showcase its qualitative behavior, we deploy DUET on an Ameca humanoid robot, where it produces richly expressive emotional speech on the humanoid, demonstrating the strong potential for plug-and-play affective interaction for embodied agents.