ERI: accessible XR system for human motion monitoring, reasoning, and rendering with 3D vision-language models

The National Science Foundation Division of Electrical, Communications and Cyber Systems awarded Knox College $199,736 on July 15, 2026, under the Engineering Research Initiation (ERI) program (CFDA 47.041) to develop an accessible extended reality telepresence system for rehabilitation monitoring and clinical guidance.

The project creates software and hardware systems that reduce bandwidth and sensor requirements for remote visualization of three-dimensional human motion, enabling expert clinicians to deliver rehabilitation services to homes and community settings. The work addresses three technical challenges: high-fidelity rendering with minimal sensors and low network bandwidth, generalizable three-dimensional vision-language models linking body motion and video with natural language, and causal reasoning that explains rehabilitation anomalies. The system targets older adults in rural and underserved communities, athletes recovering from injury, and stroke and injury survivors who face barriers to specialized clinicians and therapy facilities.

Knox College, a primarily undergraduate institution in Galesburg, Illinois, serves as the recipient. The award funds hands-on research experiences for undergraduates, introduction of new courses in 3D computing and large language model programming, and community-based research partnerships with local senior residents and physical therapists as design collaborators. Performance occurs in Galesburg, Illinois, with a period of performance through June 30, 2028.

Abstract

As populations age and life expectancies rise, the demand for rehabilitation services grows steadily. Yet many older adults, particularly those in rural and underserved communities, struggle to access specialized clinicians and advanced therapy facilities. Athletes recovering from injury and individuals rebuilding strength face the same barriers when timely diagnosis, expert feedback, and structured therapy are unavailable. This engineering research initiation (ERI) project develops an accessible extended reality (XR) telepresence system for rehabilitation. The system reduces the hardware, infrastructure, and network bandwidth required to visualize 3D human motion remotely, so that expert care can reach homes and community settings. The research benefits older adults, athletes, stroke and injury survivors, and general users seeking tools that support safe exercise and wellness. By lowering cost and setup complexity, the project narrows gaps in healthcare access and advances equity in rehabilitation. The activity also provides hands-on research experiences for undergraduates at a primarily undergraduate institution. New courses in 3D computing and large language model programming are introduced. Local senior residents and physical therapists serve as design partners through community-based research. Together, these efforts strengthen a diverse science, technology, engineering, and mathematics workforce prepared for interdisciplinary challenges in next-generation healthcare and XR technology. The research addresses three challenges that limit XR telepresence outside the laboratory: high-fidelity rendering with minimal sensors and low network bandwidth, generalizable three-dimensional vision-language models that link body motion and video with natural language, and causal reasoning that explains rehabilitation anomalies rather than merely detecting them. The first thrust targets free-viewpoint reconstruction of dynamic humans from a single camera with reduced network bandwidth, so that photorealistic telepresence becomes feasible on consumer networks. The second thrust builds a three-dimensional vision-language model (3D VLM). IT translates body motion and video into natural language for clinical description, and generates virtual humanoid exemplars from natural language to instruct patients to follow intended movement. The third thrust develops causal reasoning for rehabilitation based on the 3D VLM. IT accounts for kinematic anomalies in terms of biomechanical factors and produces interpretable explanations. Outcomes advance fundamental knowledge in computer vision, multimodal artificial intelligence, human-centered engineering, and rehabilitation science. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.