NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
NemotronLabs VoiceChat is a full-duplex speech-to-speech model with tool-calling capabilities, excelling in Full-Duplex-Bench evaluations.
Key Findings
Methodology
The model integrates a streaming speech encoder and a decoder-only language model, supporting parallel output streams for agent text and structured function calls. An auxiliary RNN-T branch is used for incremental user transcription, and a streaming TTS decoder is used for speech generation.
Key Results
- On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieved the lowest pause-handling takeover rates, 100% takeover following user interruptions, and a response quality score of 4.33/5.
- On Full-Duplex-Bench 1.5, it resumed its response after user backchannels in 93% of cases.
- It obtained a 55.1 normalized average on VoiceBench and achieved 82.5% tool-selection F1 on Full-Duplex-Bench 3.0.
Significance
NemotronLabs VoiceChat demonstrates the integration of full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use in a single open speech-to-speech model without sacrificing real-time conversational behavior.
Technical Contribution
The model achieves full-duplex, real-time, seamless voice interaction in a unified architecture, eliminating the need for multiple models or API handoffs, thereby reducing end-to-end latency.
Novelty
NemotronLabs VoiceChat is the first to seamlessly integrate full-duplex speech interaction and tool calling in an open model, significantly reducing latency compared to traditional cascaded architectures.
Limitations
- Parameter accuracy and end-to-end execution of tool calling need improvement.
- The model may underperform in complex conversational scenarios.
Future Work
Future work could focus on improving parameter accuracy and end-to-end execution efficiency of tool calling, and extending the model's applicability to more complex conversational scenarios.
AI Executive Summary
NemotronLabs VoiceChat is an innovative full-duplex speech-to-speech model combining a streaming speech encoder and decoder-only language model with tool-calling capabilities. Existing speech agents typically rely on cascaded architectures, resulting in high latency. NemotronLabs VoiceChat achieves real-time, seamless voice interaction through a unified architecture.
The model achieved the lowest pause-handling takeover rates on Full-Duplex-Bench 1.0, 100% takeover following user interruptions, and a response quality score of 4.33/5. On Full-Duplex-Bench 1.5, it resumed its response after user backchannels in 93% of cases. It obtained a 55.1 normalized average on VoiceBench and achieved 82.5% tool-selection F1 on Full-Duplex-Bench 3.0.
Despite significant advancements in integrating full-duplex interaction and tool calling, parameter accuracy and end-to-end execution of tool calling need improvement. Future research could focus on enhancing these aspects and extending the model's applicability to more complex conversational scenarios.
Deep Analysis
Background
Speech agents are natural interfaces for interacting with intelligent systems. Traditional methods use cascaded architectures connecting automatic speech recognition, large language models, and text-to-speech systems. However, these methods struggle with dynamic conversations as they require detecting when the user stops speaking before responding. NemotronLabs VoiceChat addresses this issue with a unified neural system, allowing direct speech-conditioned reasoning and generation.
Core Problem
Existing speech agents face challenges in handling real-time conversational dynamics, especially during user interruptions and feedback. NemotronLabs VoiceChat aims to address these issues by achieving seamless speech interaction with full-duplex capabilities and tool calling.
Innovation
The core innovation of NemotronLabs VoiceChat lies in its unified architecture, combining a streaming speech encoder and decoder-only language model with tool-calling capabilities. This design eliminates the need for multiple models or API handoffs, significantly reducing latency.
Methodology
- �� Streaming speech encoder processes user audio.
- �� Decoder-only language model tracks conversation and generates agent responses and function calls.
- �� Auxiliary RNN-T branch for incremental user transcription.
- �� Streaming TTS decoder converts responses into speech.
Experiments
Experiments were conducted on Full-Duplex-Bench 1.0 and 1.5 to evaluate the model's pause handling, user interruption, and feedback recovery capabilities. Single-turn response intelligence was tested on VoiceBench, and tool calling was evaluated on Full-Duplex-Bench 3.0.
Results
NemotronLabs VoiceChat achieved the lowest pause-handling takeover rates on Full-Duplex-Bench 1.0, 100% takeover following user interruptions, and a response quality score of 4.33/5. On Full-Duplex-Bench 1.5, it resumed its response after user backchannels in 93% of cases.
Applications
NemotronLabs VoiceChat can be used in real-time voice assistants, customer service, and other scenarios requiring natural conversation. Its tool-calling capability makes it particularly useful in applications requiring external information or actions.
Limitations & Outlook
Despite advancements in integrating full-duplex interaction and tool calling, parameter accuracy and end-to-end execution of tool calling need improvement. Future research could focus on enhancing these aspects.
Plain Language Accessible to non-experts
Imagine a kitchen where NemotronLabs VoiceChat is like a chef who can listen and talk simultaneously. This chef not only hears your instructions but continues cooking while you speak. He can call on tools like opening the oven or mixer without stopping. This ability makes kitchen work more efficient, just as NemotronLabs VoiceChat makes voice interactions smoother.
ELI14 Explained like you're 14
Imagine you're playing a game where your character can hear teammates' instructions and react without pausing the game. That's how NemotronLabs VoiceChat works! It can handle voice input and output simultaneously, just like you listen and talk in the game. It can also call tools when needed, like using special skills in a game—how cool is that!
Glossary
Full-duplex
A communication method allowing simultaneous sending and receiving of signals.
Implemented in NemotronLabs VoiceChat for speech interaction.
Tool Calling
The ability to invoke external tools or services during a conversation.
The model can seamlessly call tools during dialogue.
Streaming Encoder
A component that processes audio input in real-time.
Used for processing user audio input.
RNN-T
A recurrent neural network architecture for speech recognition.
Auxiliary branch used for incremental user transcription.
TTS (Text-to-Speech)
Technology that converts text into speech.
Used to convert generated responses into speech.
Open Questions Unanswered questions from this research
- 1 Improving parameter accuracy of tool calling remains an open question.
- 2 Performance in complex conversational scenarios needs further research.
Applications
Immediate Applications
Customer Service
NemotronLabs VoiceChat can be used in real-time customer service, providing quick responses and tool calling.
Long-term Vision
Smart Home
In smart homes, NemotronLabs VoiceChat can enable more natural voice control and device management.
Abstract
We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100\% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93\% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5\% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.