Verbal and non-verbal predictors of successful communication within and across neurotype: leveraging context-dependent, next-generation digital phenotyping

The National Institute on Deafness and Other Communication Disorders awarded The Children's Hospital of Philadelphia $717,468 on May 15, 2026, under the Research Related to Deafness and Communication Disorders program (CFDA 93.173).

The award funds research analyzing how the integration of verbal and non-verbal communication cues predicts communicative success in 560 autistic and non-autistic youth during peer-to-peer video conversations. The project examines facial expressions, vocal behaviors, and language measures together to determine how these elements predict self-reported conversational success and satisfaction across neurotypes. The research leverages an existing video-based conversational dataset from the same youth population and employs manual coding of intra-speaker verbal and non-verbal alignment alongside computer vision techniques for analyzing between-speaker synchrony.

Performance takes place in Philadelphia, Pennsylvania, with a period of performance extending from the award date through April 30, 2031. The assistance type is Project Grant.

Abstract

Conversation is a dynamic interplay between what you say and how you say IT, including what your face and voice express. Even autistic individuals who score within the typical range on standardized language tests frequently report less successful and satisfying social communication outcomes than non-autistic peers, which may be related to differences in the integration and use of verbal and non-verbal expressions. Unfortunately, research mostly fails to consider language in its interactional and multi-modal context in two crucial ways: 1) the intra-speaker alignment of language and non-verbal information from facial and vocal expressions of each speaker, and 2) the between-speaker synchrony of verbal and non-verbal features. This represents a major impediment to our understanding of autistic communication. This project takes an innovative approach to analyzing how the integration of verbal and non-verbal cues predict communicative success in 560 autistic and non-autistic youth during peer-to-peer zoom conversations. A key strength of our plan is to analyze novel facial and vocal behaviors in conjunction with language measures from an ongoing study of the same youth during neurotype-concordant and neurotype-discordant conversations. We propose to leverage multiple techniques, the unique combined expertise of this team, and access to a rich, video-based conversational dataset, to determine how verbal and non-verbal elements together predict self-reported conversational success and satisfaction in autistic and non-autistic adolescents. Three key strengths contribute to our project's feasibility: 1) expertise in manual coding of intra-speaker verbal and non-verbal conversational alignment that captures socially relevant details (MPI grossman); 2) expertise in computer vision-driven, between-speaker multi-modal synchrony using large scale datasets (MPI herrington); 3) access to conversational outcomes measures based on a video dataset of 560 autistic and non-autistic adolescents from an ongoing NIH-funded grant (R01 DC021564-01, ready to connect: conversation and language in autistic teens, DR. Grossman, CO-i). The connect grant captures videos of zoom conversations between neurotype-matched and -discordant adolescents and includes self-reported measures of participants' conversational satisfaction. However, the connect project will analyze only verbal predictors of communicative success and satisfaction. The present proposal will exploit this rich video dataset to analyze non-verbal and verbal conversational behavior jointly. We will use cutting-edge computer vision techniques to determine the relative contributions of verbal and non-verbal behaviors, as well as derive network models of nuanced individual communication profiles that can inform more individualized intervention models. The connect team supports this proposal (see letters of support) and both mpis herrington and grossman have full data access and associated IRB approvals. This project represents a huge economy of scale and has the capacity to identify "subgroups and subpopulations" within autism to support individualized interventions (nidcd's strategic plan theme 3).