AI and haptics-enhanced augmented reality for next generation neonatal simulation

The National Institute of Biomedical Imaging and Bioengineering awarded Vanderbilt University $561,616 on September 1, 2026, to develop ENRICH (Education in Next-Gen Reality with Intelligence and Clinical Haptics), a system integrating augmented reality, generative artificial intelligence, and haptic technologies to simulate realistic newborn patients for nursing education and clinical skill development. The award is made under the Discovery and Applied Research for Technological Innovations to Improve Human Health program (CFDA 93.286).

The project addresses gaps in pediatric neonatal nursing simulation by creating an on-demand virtual newborn patient with customizable, diagnosis-appropriate visual and behavioral presentations. The first phase develops a neonatal avatar creation pipeline using reality capture and animation techniques, coupled with an AI-driven system to control the virtual newborn's clinical presentation. The second phase enables palpation of the virtual avatar through a novel wearable haptic device integrating force, vibrotactile, and thermal feedback. The third phase investigates the overall educational effectiveness of the integrated system within Kolb's framework of experiential learning. Design and co-development involve iterative feedback from focus groups of registered nurses and simulation experts (N = 30), alongside pediatric clinicians, nursing simulation specialists, and experiential learning experts.

Work is performed at Vanderbilt University in Nashville, Tennessee, with a period of performance extending from September 1, 2026, through July 31, 2030.

Abstract

Nursing education relies on high-quality, evidence-based simulation-particularly simulated patients-to prepare clinicians for real-world clinical challenges. Yet for pediatric newborn nursing simulation, standard methods of providing simulated patients, such as manikins and trained actors called standardized patients, are often not viable options. Current state-of-the-art alternative methods, including virtual newborns, often lack realistic visual, behavioral, and physical fidelity. This project addresses these gaps by creating enrich (education in next-GEN reality with intelligence and clinical haptics), a system that integrates augmented reality (AR), generative artificial intelligence (genai), and haptic technologies to simulate realistic newborn patients. The design and validation of enrich will demonstrate the potential of such integrated systems for enhancing skill development and clinical decision-making for nurses using three specific aims. All aims are developed within kolb's framework of experiential learning. Aim 1: to create an on-demand virtual newborn patient with customizable presentations of diagnosis-appropriate symptoms and behaviors by: a) developing a neonatal avatar creation pipeline based on state-of-the-art reality capture and animation techniques, and b) developing an AI- driven system to control a virtual newborn's clinically relevant visual and behavioral presentation. Aim 2: to enable palpation of a virtual newborn avatar using a novel wearable haptic device that integrates force, vibrotactile, and thermal feedback without preventing the learner from engaging with their physical environment. Both aims 1 and 2 will be CO-developed with our team's pediatric clinician, nursing simulation, and experiential learning experts, alongside iterative feedback from focus groups of RNS and simulation experts (n = 30). Aim 3: to investigate the overarching experience of registered nurse stakeholders (n=50) with using enrich through: (a) designing and implementing a standardized newborn simulation scenario, (b) evaluating feasibility, usability, and acceptance of the enrich system, and (c) exploring attitudes, satisfaction, observations, and critical feedback attributes of experiential learning while using enrich. Mixed-method analysis will be employed for rigor and reproducibility. By enhancing newborn nursing simulation, enrich aligns with the broader goal of strengthening healthcare education and improving the quality of care for neonates. The knowledge gained from this work will inform future investigations and innovations in simulation-based healthcare education, including emergency care, acute care, intensive care, psychiatric care, school health care, and long-term care.