Collaborative research: CPS: medium: designing a closed-loop unvoiced interaction paradigm

The National Science Foundation Division of Computer and Network Systems awarded $249,972 to the University of California, Merced on August 15, 2026, under the Computer and Information Science and Engineering program (CFDA 47.070) to design a closed-loop unvoiced interaction paradigm for silent human-computer communication.

The project will develop computational models mapping jaw kinematics and facial vibrations to phonetic and linguistic units, design a personalized context-aware closed-loop feedback control system leveraging structured knowledge schemas for real-time adaptation, and build a low-power earable hardware platform optimized for sensing jaw dynamics with programmable computational offloading. The system uses sensors incorporated into an ear-worn device to detect jaw movement and small facial vibrations associated with speech, enabling users to communicate silently with devices by articulating words without producing sound. This approach addresses limitations of conventional voice interfaces in noisy environments, privacy-compromised public spaces, and quiet settings where audible speech is disruptive. The resulting platform will establish foundations for private, hands-free, unobtrusive interaction in environments where traditional voice interfaces are ineffective or inappropriate.

Performance takes place in Merced, California, with a period of performance from August 15, 2026 through July 31, 2029. This is a collaborative research project grant, classified as assistance funding.

Abstract

Speech is the most natural way for people to communicate with technology. Voice-based interfaces allow users to interact with their phones and other devices without using their hands. However, speaking aloud is not always practical. Voice interfaces struggle to function in noisy environments, can compromise user privacy in public spaces, and may be impractical in quiet settings where IT is desirable to avoid disturbing others (for example, libraries, classrooms, hospitals). This project proposes a novel approach that will allow users to communicate silently with their devices by articulating words without producing sound. The proposed system will use sensors incorporated into a socially acceptable ear-worn device to detect jaw movement and small facial vibrations associated with speech. By transforming these signals into words and commands, the system could enable private, hands-free, and unobtrusive interaction in environments where conventional voice interfaces are ineffective or inappropriate. This project will advance the science of human?computer interaction and support the development of more accessible and privacy-preserving computing systems. The objective of this research is to design a robust, scalable unvoiced interface by leveraging the rising trend of sensor-equipped earables to track jaw motion and subtle facial vibrations for speech inference. This research will (1) develop computational models that map jaw kinematics and facial vibrations to phonetic and linguistic units; (2) design a personalized, context-aware closed-loop feedback control system that leverages structured knowledge schemas to adapt feedback in real time; and (3) build a low-power, earable hardware platform optimized for sensing jaw dynamics with support for programmable computational offloading. The resulting low-power, privacy-preserving platform will establish new foundations for silent human?computer interaction across diverse users and real-world environments. The research outcomes from this work will be integrated into a comprehensive education plan and will influence pedagogy at the intersection of cyber-physical systems, human?computer interaction, embedded sensing, and artificial intelligence. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.