FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
FAMA is the first large-scale open-science speech foundation model for English and Italian, achieving up to 8x speed improvement.
Key Findings
Methodology
FAMA is trained using open-source datasets, comprising over 150k hours of speech data. The model architecture includes a Conformer encoder and a Transformer decoder, with 12 and 6 layers (small model) or 24 and 12 layers (medium model), respectively. The training process is divided into ASR pre-training and ASR+ST joint training stages.
Key Results
- FAMA medium model achieves an average improvement of 4.4 WER (English) and 6.4 WER (Italian) in ASR tasks compared to Whisper medium.
- Compared to OWSM v3.1 medium, FAMA medium improves by 7.3 WER on Italian.
- FAMA models are 5-8 times faster than Whisper in computational efficiency.
Significance
The introduction of FAMA fills the gap in open-source models in the speech processing field, promoting research transparency and reproducibility. By providing open-source code and datasets, FAMA enables researchers to fairly evaluate and compare different speech models, advancing open science in speech technology.
Technical Contribution
FAMA's technical contributions lie in its training entirely on open-source data and code, avoiding data contamination issues. Compared to existing closed models, FAMA maintains high performance while significantly improving computational efficiency.
Novelty
FAMA is the first speech foundation model trained entirely on open-source data and code, addressing the shortcomings of existing models in reproducibility and data transparency.
Limitations
- FAMA's performance in speech translation tasks still lags behind some non-open models.
- Further research is needed on the model's multilingual scalability.
Future Work
Future research directions include extending FAMA to more languages and developing new open-source datasets to improve speech translation performance.
AI Executive Summary
In recent years, speech foundation models (SFMs) like Whisper and SeamlessM4T have significantly advanced the field of speech processing. However, their closed nature limits reproducibility and fair evaluation. FAMA addresses this gap by being the first large-scale open-science speech foundation model for English and Italian. Trained on over 150k hours of open-source speech data, FAMA employs a Conformer encoder and Transformer decoder architecture, significantly improving computational efficiency.
FAMA demonstrates outstanding performance in automatic speech recognition (ASR) tasks, achieving an average improvement of 4.4 WER (English) and 6.4 WER (Italian) compared to existing models. Additionally, FAMA is 5-8 times faster than Whisper. All code, datasets, and models are released under open-source licenses, promoting openness in speech technology research.
While FAMA excels in ASR tasks, improvements are still needed in speech translation (ST) tasks. Future research will focus on extending FAMA to more languages and developing new open-source datasets to enhance ST performance. This will further advance open science in speech technology, fostering broader participation and inclusive applications.
Deep Analysis
Background
Speech processing technology has made significant strides in recent years, particularly in automatic speech recognition (ASR) and speech translation (ST). Existing speech foundation models like Whisper and SeamlessM4T, while performing well, are closed, limiting reproducibility and fair evaluation. Other fields have made significant progress by developing fully transparent models, but efforts in speech processing remain limited.
Core Problem
Existing speech foundation models are closed, making training data and code inaccessible, hindering reproducibility and fair evaluation. Additionally, data contamination issues make current model evaluations unreliable.
Innovation
FAMA's core innovation lies in its training entirely on open-source data and code, ensuring data transparency and model reproducibility. By using a Conformer encoder and Transformer decoder architecture, FAMA maintains high performance while significantly improving computational efficiency.
Methodology
- �� Trained on 150k hours of open-source speech data
- �� Architecture includes Conformer encoder and Transformer decoder
- �� Training process divided into ASR pre-training and ASR+ST joint training stages
- �� Optimized using a combination of loss functions, including label-smoothed cross-entropy and CTC loss
Experiments
The experimental design includes evaluations using datasets like CommonVoice, MLS, and VoxPopuli. Baseline models include Whisper and OWSM. Key hyperparameters include learning rate and sampling probability. Ablation studies were conducted to assess the impact of different training strategies.
Results
FAMA demonstrates exceptional performance in ASR tasks, achieving an average improvement of 4.4 WER (English) and 6.4 WER (Italian) compared to Whisper medium. In terms of computational efficiency, FAMA is 5-8 times faster than Whisper. While improvements are needed in ST tasks, FAMA's ASR performance approaches that of non-open models.
Applications
FAMA's direct application scenarios include speech recognition and translation systems, particularly suitable for scenarios requiring efficient processing of large-scale speech data. Its open-source nature allows researchers to build upon it for further research and development.
Limitations & Outlook
FAMA's performance in speech translation tasks still lags behind some non-open models. Additionally, further research is needed on the model's multilingual scalability. Future improvements include developing new open-source datasets to enhance ST performance.
Plain Language Accessible to non-experts
Imagine you're in a large library, and FAMA is like a super librarian who can quickly find the books (speech data) you want and explain them to you in multiple languages (speech recognition and translation). Traditional librarians, while excellent, keep their methods secret, so others can't learn and improve. FAMA, however, is completely open, allowing anyone to understand its workings and improve upon it. This is like making the library's management methods public, allowing everyone to participate and improve the library's service efficiency.
ELI14 Explained like you're 14
Imagine you're playing a super cool game where you control characters with your voice! But the problem is, the game's developers don't tell you how they did it, so you can't create similar games yourself. FAMA is like an open-source game engine, where anyone can see its code and use it to create their own games. It not only makes the game run faster but also lets you play in multiple languages! Isn't that cool?
Glossary
Conformer Encoder
An encoder combining convolution and self-attention mechanisms for processing speech signals.
FAMA uses a Conformer encoder to enhance speech recognition accuracy.
Transformer Decoder
A decoder based on self-attention mechanisms for generating target sequences.
FAMA employs a Transformer decoder to produce translation results.
Open-source Dataset
Publicly available datasets that anyone can access and use.
FAMA is trained on open-source datasets to ensure data transparency.
Automatic Speech Recognition (ASR)
Technology that converts speech signals into text.
FAMA excels in ASR tasks, significantly improving recognition accuracy.
Speech Translation (ST)
Converting speech in one language to text in another language.
FAMA still needs improvement in speech translation tasks but has made some progress.
Open Questions Unanswered questions from this research
- 1 How can FAMA's performance in speech translation tasks be improved? Current models still lack accuracy in translation, requiring new datasets and training methods.
- 2 What is FAMA's performance in multilingual scalability? Further research is needed to assess its applicability to other languages.
Applications
Immediate Applications
Speech Recognition Systems
FAMA can be used to develop efficient speech recognition systems suitable for scenarios requiring large-scale speech data processing.
Speech Translation Applications
FAMA can be used to develop multilingual speech translation applications, helping users communicate across different languages.
Long-term Vision
Multilingual Voice Assistants
FAMA's open-source nature makes it an ideal foundation for developing multilingual voice assistants, with potential support for more languages and features in the future.
Abstract
The development of speech foundation models (SFMs) like Whisper and SeamlessM4T has significantly advanced the field of speech processing. However, their closed nature--with inaccessible training data and code--poses major reproducibility and fair evaluation challenges. While other domains have made substantial progress toward open science by developing fully transparent models trained on open-source (OS) code and data, similar efforts in speech remain limited. To fill this gap, we introduce FAMA, the first family of open science SFMs for English and Italian, trained on 150k+ hours of OS speech data. Moreover, we present a new dataset containing 16k hours of cleaned and pseudo-labeled speech for both languages. Results show that FAMA achieves competitive performance compared to existing SFMs while being up to 8 times faster. All artifacts, including code, datasets, and models, are released under OS-compliant licenses, promoting openness in speech technology research.