AI-enhanced multimodal conversational virtual patient for interactive medical student training

The National Institute of Biomedical Imaging and Bioengineering awarded the University of Idaho $139,208 on September 1, 2026, under the Discovery and Applied Research for Technological Innovations to Improve Human Health program (CFDA 93.286) to develop and validate an AI-enhanced multimodal conversational virtual patient system for medical education.

The project will create an interactive training platform that integrates medical imaging with clinical dialogue using large language models and vision-language models to improve how medical students learn complex diagnoses. Specific deliverables include an AI framework for automated generation of clinically plausible patient profiles that fuse medical images with clinical narratives; an interactive conversational virtual patient system capable of real-time, multimodal dialogue with feedback on diagnostic accuracy via image analysis and communication skills via speech and text analysis; and validation of the system's educational efficacy, usability, and pedagogical impact through a pilot course in dermoscopy diagnosis at a medical school, with data collection from student and faculty surveys and focus groups. The system will support dynamic generation of diverse patient scenarios including rare pathologies and real-time feedback on both diagnostic and communication competencies.

Work is performed in Moscow, Idaho, with performance extending through June 30, 2030. The award funds hypothesis-driven research related to discovery, design, development, validation, and application of biomedical imaging and bioengineering technologies as specified by the NIBIB program authority.

Abstract

Current medical training based on conversational virtual patient (CVP) inadequately prepares clinicians for complex diagnoses due to limited integration of medical imaging with patient communication and laborious case creation. This project will develop and evaluate an innovative CVP system using multimodal large language model (LLM) to overcome these limitations. The long-term objective is to advance multimodal AI for superior healthcare delivery and clinical education, supporting the NIH mission to enhance health by improving clinical decision-making and reducing medical errors. The project entails three specific aims: 1. Developing an AI framework for the automated generation of diverse, clinically plausible patient profiles that fuse medical images with clinical narratives, utilizing advanced vision-language models and bayesian networks for synthetic data creation. 2. Engineering an interactive CVP system capable of real-time, AI-driven multimodal dialogue and feedback on diagnostic accuracy (via image analysis and trainee annotations) and communication skills (via speech and text analysis). 3. Validating the CVP system's educational efficacy, usability, and pedagogical impact through a pilot medical school course in dermoscopy diagnosis, gathering feedback from student and faculty volunteers via surveys and focus groups. Key innovations include the seamless integration of image interpretation with dialogue in simulation, dynamic AI-driven generation of varied patient scenarios including rare pathologies, and real-time feedback on both diagnostic and communication competencies. This research will deliver a scalable, validated training platform designed to improve crucial clinical skills, enhance patient safety, and provide a foundation for future AI- assisted medical education and diagnostic tools.