Collaborative research: RI: medium: using systematic relationships and phonetic speech foundation models for universal speech-to-text across varieties of languages

Gettysburg College received a $377,764 project grant from the National Science Foundation (NSF) Division of Information and Intelligent Systems under the Computer and Information Science and Engineering program (CFDA 47.070), with an award date of September 1, 2025, and a completion date of August 31, 2029. This collaborative research initiative develops a Phonetic and Optical Word Speech Model (POWSM) that enables universal speech-to-text systems across low-resource language varieties by leveraging systematic phonetic relationships between related language variants. The project addresses a significant gap in automatic speech recognition (ASR) technology by creating AI models capable of transcribing speech in language varieties with minimal training data by learning predictable pronunciation patterns from well-resourced language varieties and applying those relationships to underfunded variants.

The deliverable represents a computational solution that enables speech-to-text system development for previously unsupported language variants and regional dialects, expanding accessibility to voice-powered tools and services. The project includes collaborative partnerships with Pennsylvania State University and the University of Colorado, combining linguistic expertise with machine learning methodologies to advance automatic speech recognition across dialectal and language variety spectrums. This research directly supports NSF's broader mission to advance knowledge and technological capabilities in computing and information science while addressing practical applications that enhance accessibility for diverse American populations.

Abstract

This project introduces a new way to develop speech-to-text systems for language varieties in which little digital data is available, especially when a related variety already has plenty of digital data. Languages differ from one another, but they also show a great deal of variation within themselves. People in different regions or countries often pronounce words in the same language differently. For example, many people in the southern US pronounce the ?i? In words like ?ride? As an ?a? Sound (similar to the one found in ?RAD?), while most other americans do not. Most speech-to-text systems?which turn spoken words into written text?focus on just one variety of each language. However, in everyday life, many people speak other varieties. Existing speech-to-text systems often do not work well for these varieties. Improving speech recognition for low-resource varieties represents a business opportunity, one that can help more americans access voice-powered tools and services. This project takes one step towards this goal. IT innovates by leveraging the fact that the differences between various varieties of the same language often follow predictable patterns. For example, since the pronunciations of words in different regions change following rules that apply to the whole vocabulary, one can often predict how a word will be pronounced in one variety if one knows how IT is pronounced in another. The project will develop a powerful AI model (powsm) that can transcribe pronunciation (using a universal system developed by linguists to represent sounds). This approach enables the development of speech-to-text systems for previously unsupported language variants, even when little recorded training data exists for them. IT works by learning both the similarities and the systematic differences between well-resourced varieties and others. The project builds an encoder-decoder foundation speech model called powsm (phonetic open whisper-style speech model), which is trained to recognize speech as sequences of phones (consonants and vowels) in any language. The project will include three applications using this model: 1) prompting powsm with vector representations of the systematic sound correspondences between a low-resource variety (LRV) and a high-resource variety (HRV), enabling the model to recognize the LRV as a transformed variant of the HRV. 2) constructing stochastic weighted finite-state transducers that can generate synthetic LRV data based on linguist-curated knowledge about sound changes in the HRV and LRV. Using this synthetic data to train a language model that can be used to decode LRV output from powsm. 3) learning phonetic correspondences between language varieties automatically from transcribed audio using a novel form of unsupervised bilingual lexicon induction (ubli) that leverages both text and audio. Audio aligned in this way can be used to train basic speech-to-speech translation models that, when used in conjunction with powsm, enable speech technologies for lrvs without requiring linguistic annotation beyond transcription. The proposed methods are intended to cover most HRV-LRV scenarios. They will be evaluated on major languages including varieties in english, italian, chinese, german, arabic, german, and dutch, english, as well as endangered languages like nahuatl and mixtec. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.