Developing multimodal foundation models for eye disease screening and prognosis

The National Eye Institute awarded The Schepens Eye Research Institute $858,394 on August 1, 2026, under the Vision Research program (CFDA 93.867) to develop multimodal foundation models for eye disease screening and prognosis.

The research develops a vision-language foundation model trained on pairs of retinal images and clinical notes to enable downstream supervised models for detection and progression prediction of age-related macular degeneration, diabetic retinopathy, and glaucoma. The project creates a reciprocal training paradigm called MutualAssist that simultaneously updates clinical text summarization by a large language model and feature learning by the foundation model. For disease detection, the model uses image data alone; for progression prediction, it integrates both image and text components. Ground truth labels are ascertained by clinician assessment. The research addresses three critical gaps in existing ophthalmic foundation models: incorporation of clinical notes, model domain adaptability, and federated learning strategies for cross-institutional training.

Performance occurs at Schepens' Boston, Massachusetts facility through July 31, 2030. Schepens is a nonprofit research organization affiliated with Harvard Medical School and Massachusetts Eye and Ear, operating primarily as a federal grant recipient for basic and translational vision research.

Abstract

Motivation: the three leading irreversible eye diseases in the US causing unrestorable vision loss with currently available clinical treatment are age-related macular degeneration, diabetic retinopathy, and glaucoma, affecting approximately 20 million, 9.6 million, and 3 million individuals, respectively. Timely diagnosis and accurate prognosis enable prompt and personalized treatment, reducing the risk of further vision loss. Supervised deep learning is promising to provide affordable screening and accurate prognosis for eye diseases, but IT requires large amounts of labeled data that are costly to acquire. Foundation models have emerged as a promising approach to best leverage vast amounts of unlabeled data to benefit downstream supervised tasks using labeled data with a smaller sample size for ophthalmic applications. Previous ophthalmic foundation models are limited by the following critical issues: (1) enriched information from clinical notes has not been incorporated; (2) model domain adaptability has not been addressed; (3) federated learning strategies to enable cross-institutional foundation model training have not yet been developed. Overall approach: the vision-language (VL) foundation model is trained using pairs of retinal images and clinical notes and used to initialize downstream supervised models for eye disease detection and progression prediction. The ground truth labels will be ascertained by clinician assessment. For eye disease detection, only the image component is used; for progression prediction, both image and text components are utilized. Aims: (1) developing and optimizing a reciprocal training paradigm mutualassist. We will develop a reciprocal training paradigm termed mutualassist to simultaneously update clinical text summarization by a large language model (LLM) and feature learning by the VL foundation model. Low-rank adaptation method will be used to update the LLM. (2) developing and optimizing a domain adaptation method domainduet. We will develop a domain adaptation method termed domainduet to learn both domain-invariant and domain-specific features either through adversarial and cooperative classifiers or by minimizing and maximizing feature distances between demographic groups. We will integrate domainduet into mutualassist framework with either a concurrent or alternate updating scheme. (3) developing and optimizing a federated foundation model method fedfound. We will develop and optimize a federated learning approach for foundation model training, fedfound, which balances cross-institutional global learning and institution-specific local learning by combining the global full parameter updating with the local low-rank parameter updating. Main deliverables and impacts: we will develop innovative multimodal VL foundation models to improve the performance in downstream supervised tasks for eye disease screening and prognosis by novel methods to improve clinical note summarization, domain adaptability, and federated foundation model training.