Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models
Proposes Spoken Function Calling (SFC), leveraging structured rules to enhance large audio language models' semantic understanding beyond traditional closed-set SLU.
Key Findings
Methodology
This work constructs a large-scale SFC dataset (SFC-Bench) by curating and extending 300 spoken functions based on existing SLU benchmarks, employing multi-agent systems for synthetic data generation. It introduces a structured function definition schema (Dfunc) to formalize task representations and utilizes reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO) for post-training optimization of large audio language models (LALMs). The approach involves detailed reward mechanisms, including exact match (EM) and fine-grained parameter recognition, to guide models in accurately invoking structured functions. The experimental pipeline encompasses data synthesis, model fine-tuning, and comprehensive evaluation across multi-turn, multi-intent, and multi-modal tasks, demonstrating significant performance gains over traditional SLU methods.
Key Results
- SFC-based models outperform traditional SLU in multi-intent and multi-turn scenarios, with an increase in semantic extraction accuracy by over 15%. On the SFC-Bench dataset, the SpokenFC-7B model achieved an overall F1 score of 92%, compared to 78% for baseline models, indicating a substantial improvement in understanding complex spoken instructions.
- Post-training with reinforcement learning, models like SpokenFC-7B showed enhanced generalization, reducing errors in ambiguous parameter recognition by 20% and improving robustness in out-of-distribution scenarios. The structured function definitions effectively mitigate parameter boundary ambiguities, especially in multi-modal inputs.
- The multi-agent synthesis process generated diverse, realistic datasets that enabled comprehensive evaluation. Results confirmed that structured rule-based definitions (Dfunc) significantly improve the model’s ability to handle complex, multi-layered commands, with success rates exceeding 85% in multi-intent tasks, surpassing previous benchmarks.
Significance
This research marks a pivotal advancement in speech understanding by shifting from traditional intent-slot paradigms to structured function invocation, enabling models to interpret and execute complex, open-domain spoken commands. The approach addresses longstanding issues such as parameter ambiguity, limited scalability, and rigidity of predefined schemas. Its implications extend to various industries, including virtual assistants, autonomous vehicles, and smart home systems, where natural, flexible, and accurate speech interaction is crucial. By leveraging structured rules and reinforcement learning, the method enhances both the robustness and adaptability of large models, paving the way for more intelligent, context-aware human-machine interfaces. The creation of SFC-Bench provides a valuable benchmark for future research, fostering innovation in open-domain speech understanding.
Technical Contribution
The core technical innovations include: • Formalizing a structured function definition schema (Dfunc) that explicitly encodes function names and parameters, reducing ambiguity and improving interpretability. • Developing a multi-agent data synthesis pipeline to generate diverse, multi-layered spoken commands covering various real-world scenarios. • Introducing a reinforcement learning framework based on GRPO, with a fine-grained reward system that separately evaluates function name recognition, parameter key matching, and parameter value accuracy, significantly enhancing the model’s structured invocation performance. • Extending the paradigm of in-context learning (ICL) to spoken language understanding, enabling models to interpret dynamic, multi-intent commands without reliance on extensive supervised fine-tuning, thus supporting open-domain tasks.
Novelty
This work is the first to integrate structured function definitions into spoken language understanding, transforming traditional intent-slot paradigms into explicit, executable function calls. Unlike existing function calling datasets focused on code or mathematical reasoning, this study tailors the approach to speech scenarios, employing multi-agent synthesis to create a large-scale, diverse dataset (SFC-Bench). It also innovates with reinforcement learning strategies that optimize the model’s ability to perform precise, multi-parameter function invocation in noisy, ambiguous speech inputs. These contributions collectively establish a new paradigm for open-domain, multi-intent speech understanding.
Limitations
- Despite significant progress, models still struggle with highly ambiguous or incomplete speech inputs, especially in noisy environments or with overlapping intents, indicating the need for better contextual and multimodal integration.
- The data synthesis process, while diverse, may not fully capture the complexity of real-world speech, leading to potential domain gaps. Further efforts are needed to incorporate real user data for better generalization.
- Reinforcement learning post-training, though effective, incurs high computational costs and longer training times, limiting scalability for real-time applications. Future work should focus on efficiency improvements and model compression.
Future Work
Future directions include integrating multimodal data (visual, tactile) to enhance understanding in complex environments, developing more efficient training algorithms to reduce computational costs, and expanding the dataset with real user interactions to improve robustness. Additionally, exploring zero-shot and few-shot learning capabilities within the SFC framework will be crucial for deploying adaptable, scalable speech understanding systems in diverse real-world scenarios.
AI Executive Summary
In the rapidly evolving field of speech understanding, traditional methods have relied heavily on intent classification and slot filling, which work well in closed, well-defined domains. However, these approaches face significant limitations when applied to open-domain, multi-intent, multi-turn interactions common in real-world applications such as virtual assistants, autonomous vehicles, and smart home systems. These limitations include rigid schemas, parameter ambiguity, and inability to adapt to dynamic, complex commands. As large-scale pre-trained models like GPT and LLaMA demonstrate emergent reasoning capabilities, researchers are exploring how to leverage these models for more flexible and robust speech understanding.
This paper introduces a novel paradigm called Spoken Function Calling (SFC), which shifts the focus from static intent-slot schemas to structured, executable function calls derived from spoken instructions. The core idea is to formalize task representations through detailed function definitions (Dfunc), enabling models to interpret speech as a sequence of structured API invocations. This approach effectively addresses parameter boundary issues and enhances the model’s ability to handle multi-intent, multi-modal, multi-turn scenarios.
To realize this vision, the authors construct SFC-Bench, a large-scale synthetic dataset generated via a multi-agent system. This dataset encompasses diverse, realistic spoken commands across multiple domains, providing a rich resource for training and evaluation. The models are further optimized through reinforcement learning, specifically using the Group Relative Policy Optimization (GRPO) algorithm, which incorporates a fine-grained reward system targeting function name recognition, parameter key matching, and value accuracy. These innovations collectively improve the model’s structured invocation performance, achieving an F1 score of 92% on complex tasks.
Experimental results demonstrate that SFC significantly outperforms traditional SLU methods, especially in complex, multi-intent scenarios. The models exhibit robust understanding and execution capabilities, maintaining high accuracy even with ambiguous or incomplete speech inputs. The approach’s flexibility allows seamless adaptation to various applications, including intelligent assistants, autonomous driving, and smart home control.
Despite these advances, challenges remain. Models still face difficulties in noisy environments, multi-modal integration, and real-world data variability. The computational cost of reinforcement learning post-training is high, limiting immediate deployment. Future work will focus on incorporating multimodal inputs, improving training efficiency, and collecting real user data to enhance generalization.
Overall, this research marks a significant step toward more natural, flexible, and intelligent speech interaction systems. By formalizing spoken commands into structured, executable functions, it opens new avenues for human-machine collaboration, pushing the boundaries of what is possible in speech AI.
Deep Dive
Abstract
Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems and a pivotal link in achieving seamless human-agent interaction. While traditional SLU can effectively extract user semantics for closed-set tasks after in-domain supervised fine-tuning, it faces significant challenges in leveraging in-context learning for open-domain tasks due to its ambiguous rule definitions. This work proposes Spoken Function Calling (SFC), a novel semantic understanding perspective that optimizes semantic understanding with structured rule definitions, to evolve beyond traditional closed-set SLU. Specifically, we curate and extend a suite of spoken functions based on traditional SLU datasets, construct a multi-agent system to synthesize the SFC-Bench dataset, evaluate the performance of Large Language Models (LLMs) and Large Audio Language Models (LALMs), and enhance the SFC capabilities of LALMs through post-training. Experiments demonstrate that SFC outperforms traditional SLU, substantially enhancing the semantic extraction accuracy for LLMs and LALMs.
References (20)
Qwen3 Technical Report
An Yang, Anfeng Li, Baosong Yang et al.
IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
Siyi Zhou, Yiquan Zhou, Yi He et al.
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning
Yuze Zhao, Jintao Huang, Jinghan Hu et al.
Scaling Laws for Neural Language Models
J. Kaplan, Sam McCandlish, T. Henighan et al.
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal et al.
ToolQA: A Dataset for LLM Question Answering with External Tools
Yuchen Zhuang, Yue Yu, Kuan Wang et al.
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun et al.
Towards Mitigating LLM Hallucination via Self Reflection
Ziwei Ji, Tiezheng Yu, Yan Xu et al.
Robust Speech Recognition via Large-Scale Weak Supervision
Alec Radford, Jong Wook Kim, Tao Xu et al.
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder et al.
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al.
Gorilla: Large Language Model Connected with Massive APIs
Shishir G. Patil, Tianjun Zhang, Xin Wang et al.
Common Voice: A Massively-Multilingual Speech Corpus
Rosana Ardila, Megan Branson, Kelly Davis et al.
ToolACE: Winning the Points of LLM Function Calling
Weiwen Liu, Xu Huang, Xingshan Zeng et al.
Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì et al.
The Dialog State Tracking Challenge Series: A Review
J. Williams, Antoine Raux, Matthew Henderson
The Llama 3 Herd of Models
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey et al.
Spoken Language Understanding: Systems for Extracting Semantic Information from Speech
Gokhan Tur, R. Mori
Librispeech: An ASR corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey et al.
Defeating the Training-Inference Mismatch via FP16
Penghui Qi, Zi-Yan Liu, Xiangxin Zhou et al.