Project Grant 2212303

This $165,014 National Science Foundation project grant will fund the development of linguistically-driven sign recognition technologies from continuous American Sign Language (ASL) signing videos. The Rochester Institute of Technology will receive funding under the Computer and Information Science and Engineering program (CFDA 47.070) to advance sign recognition techniques for isolated citations to segmented signs within sentences.

The research will create an end-to-end machine learning approach with three key components: segmentation of continuous signing regions by sign type; segmentation of individual signs within regions; and recognition of segmented signs considering coarticulation effects and leveraging linguistic constraints. This linguistically informed methodology is necessary for successful large-scale continuous sign recognition and applications like ASL-to-English translation, educational tools for ASL learners, and sign search capabilities. All data and new methods will be publicly shared to facilitate further research across various sciences. Broader impacts include application to other signed languages and incorporation into education programs.