Visual play: an interactive play idea and AAC grid generator to assist in behavioral interventions for minimally verbal autistic children

The National Institute on Deafness and Other Communication Disorders awarded Su-Kam Intelligent Education Systems, Inc. $314,363 on August 14, 2026, under the Research Related to Deafness and Communication Disorders program (CFDA 93.173) to develop Visual Play, an interactive application that generates augmentative and alternative communication (AAC) speech-generating device grids to support play-based behavioral interventions for minimally verbal autistic children.

Visual Play automates the creation of AAC grids by leveraging computer vision and large language models. During therapy sessions, therapists photograph the play environment; the application analyzes the images to suggest developmentally appropriate play activities and generates corresponding AAC grids with relevant vocabulary (nouns, verbs, adjectives, prepositions) matched to the child's language development level. The application addresses a documented gap: while speech-generating devices have demonstrated efficacy in increasing socially communicative utterances in minimally verbal and non-verbal children with autism, therapists rarely use them due to the cognitive burden of programming and operating the devices during active sessions.

Su-Kam Intelligent Education Systems is collaborating with Kasari's Autism Intervention Research Group at UCLA. Phase 1 aims to optimize the underlying AI model (GPT-4O).

Period of performance runs through July 31, 2027. Performance takes place in Pasadena, California.

Abstract

And project summary 1 in 36 children are autistic. Many have severe speech and language delays - at age 9, 20% of children with ASD use fewer than 5 words per day, while another 19% can say words but are unable to combine them into sentences. These outcomes can be improved through play-based behavioral therapies. In a 2014 study involving minimally verbal and non-verbal autistic children, kasari ET AL added to such therapy a speech generating device (SGD), a form of augmentative and alternative communication (AAC) consisting of a tapping talkable grid of words relevant to the play activity. SGD use resulted in a significant increase in the number of socially communicative utterances, nearly all of which involved natural speech. Though their benefits are clear, sgds are rarely used by speech and behavioral therapists due to the cognitive burden of programming sgds and using them during their sessions. In this proposal, skies - an educational technology company - and kasari's autism intervention research group at ucla are collaborating to build a new APP called visual play (VP) that makes sgds easier to use during therapy. During the session, the therapist takes pictures of the environment, which the APP uses to suggest play ideas at the child's level of play development. Selecting a play idea causes the APP to generate a relevant SGD grid at the child's level of language development. Auto-generating SGD grids from photographs is an active area of research driven by recent advances in computer vision and large language model technology. The key innovation of visual play over prior work is the explicit focus on supporting evidence-based therapy by including play ideas in an SGD, which is used to build grids with relevant nouns, verbs, adjectives, and prepositions. In this phase 1 study, aim 1 is optimizing an AI (GPT-4O LLM) so that IT can generate play ideas on PAR with those generated by human experts, over a wide range of play levels and toy selections. Optimization will proceed via iterative prompt engineering and fine-tuning, with both human and AI-generated play ideas scored by experts in kasari's group on qualities of accuracy, engagement, and repeatability. Aim 2 is making a prototype of VP. Aim 3 is testing the feasibility and usability of the APP to drive iterative APP development. Testing will be performed with a cohort of 10 therapists receiving training in joint attention, symbolic play, engagement, and regulation (jasper) therapy. Therapists will be selected who are actively working with preschool-aged minimally verbal autistic children. The following data will be collected: APP usage per session, % interactions with visual play, and quantitative and qualitative usability feedback. The data will be collected during jasper therapy sessions with children - both within an intensive week-long jasper/VP training, and during a six week follow-up period where therapists use VP at least weekly with their actual clients. Phase i aims to bring the APP's usability to the point IT is rated highly on an end-of-study questionnaire (≥5/7 average rating on a modified mhealth APP usability questionnaire) by at least 80% of therapists; and used an average of >2 times per session by at least 50% of therapists.