TiCo: Time-Controllable Training for Spoken Dialogue Models

Spoken Dialogue Models (SDMs) have gained significant attention in real-world applications such as voice assistants, wearable devices, and healthcare systems. Traditional voice assistants rely on cascaded ASR, text generation, and TTS modules, while modern SDMs increasingly adopt end-to-end or tightly integrated modeling paradigms. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions. Controlling response duration is crucial for improving user experience and meeting time constraints in practical applications.

Existing spoken dialogue models perform poorly in time control, failing to meet time constraints in practical applications. This is particularly critical in scenarios requiring precise time management, such as medical and emergency situations, where the model's time control capability directly impacts user experience and system usability. Speech generation duration is influenced not only by word count but also by speaking rate and speech realization, making time control a unique and more challenging problem.

The core innovation of the TiCo method lies in introducing Spoken Time Markers and reinforcement learning to achieve precise control over speech generation duration. • Spoken Time Markers: By inserting time markers during generation, the model can estimate elapsed time and adjust content to meet target durations. • Reinforcement Learning: Optimizes the model's time control capability through a reward mechanism, ensuring response quality while adhering to time constraints. • Self-Generation: Eliminates the need for additional data pairs by using the model's own output distribution for training, improving training stability.

• �� Spoken Time Markers: Inserted during generation to help the model estimate elapsed time. • Self-Generation: Utilizes the model's own output distribution for training without additional data pairs. • Reinforcement Learning: Optimizes the model's time control capability through a reward mechanism. • Dataset Construction: Samples extracted from existing datasets with inserted time control instructions to form an evaluation benchmark.

The experimental design includes evaluations on the InstructS2S and UROBench datasets, using MAE and MAPE as primary metrics. Baseline models include commercial models and cascaded systems, with experiments also testing generalization to different duration settings and text queries. Key hyperparameters include the maximum number of generated tokens and the insertion strategy for time markers.

Experimental results show that TiCo significantly improves time control capabilities across multiple datasets, reducing MAE from the baseline model's 13.01 seconds to 4.54 seconds. TiCo also demonstrates good generalization to longer responses and text queries, proving its applicability in various scenarios. In TiCo-Bench tests, TiCo maintains low error across all duration ranges, with MAPE below 20%.

TiCo can be directly applied in voice assistants and interactive agents, especially in scenarios requiring precise time management, such as medical and emergency situations. By enhancing time control capabilities, TiCo can significantly improve user experience, reduce deployment costs, and increase system usability.

Despite the significant progress in time control, TiCo still has room for improvement in relative error for short-duration responses. Additionally, the current time marker mechanism has limited adaptability to variations in speaking rate. In complex speech scenarios, the accuracy of time marker predictions still needs enhancement. Future research directions include improving time marker prediction accuracy, exploring time control in more complex speech scenarios, and extending TiCo to multimodal dialogue systems.