Collaborative research: cise crosscutting small: SCH: towards auto-prompt textual annotations generation with domain knowledge for multimodal medical image segmentation

Texas Woman's University received a $210,000 Project Grant from the National Science Foundation's Division of Information and Intelligent Systems under the Computer and Information Science and Engineering (CISE) program (CFDA 47.070), effective October 1, 2025 through September 30, 2028. This collaborative research project develops an automated framework for generating segmentation-specific textual annotations without manual prompt engineering to enhance multimodal medical image segmentation. The core deliverables include: (1) an auto-prompting mechanism utilizing graph-based reasoning to produce task-specific textual descriptions, (2) a knowledge-graph module that encodes and updates medical domain expertise to improve model generalization across diverse imaging modalities and anatomical regions, and (3) a multi-level feature-alignment strategy that leverages the bidirectional relationship between images and text to refine segmentation quality.

The project integrates Vision-Language Models (VLMs) with domain knowledge to create clinically applicable artificial intelligence tools that overcome current limitations in deep-learning approaches that rely solely on imaging data. Broader impacts include student engagement through hands-on research, interdisciplinary collaboration, and development of open-access tools, with clinical validation achieved through close partnership with medical experts. This research addresses real-world healthcare needs in clinical decision-making and disease monitoring while supporting translational impact in the medical imaging field.

Abstract

Medical image segmentation is essential for clinical decision making and disease monitoring, yet current deep-learning approaches are limited by their reliance on imaging data alone and lack of contextual understanding. Vision-language models (vlms) offer a promising alternative by generating textual annotations, but their dependence on manually crafted prompts and poor adaptation to segmentation tasks constrain their clinical utility. Moreover, these models struggle to generalize across imaging modalities and anatomical regions. This project develops an automated framework that generates segmentation-specific textual descriptions without human-created prompts, improving annotation efficiency and segmentation quality. IT further integrates dynamic knowledge-graph reasoning to embed evolving medical expertise into the annotation process, enhancing adaptability across diverse imaging contexts. The approach aims to create robust and generalizable artificial intelligence (AI) tools for real-world clinical use. Broader-impact aspects of the project include the engagement of students through hands-on research, interdisciplinary collaboration, and open-access tools that advance science and education as well as clinical relevance that is ensured through close collaboration with medical experts, allowing the research to address real-world healthcare needs and support translational impact. The project introduces a multimodal framework that leverages the bidirectional relationship between images and text to refine segmentation. Key components include: (1) an auto-prompting mechanism driven by graph-based reasoning to produce task-specific textual descriptions; (2) a knowledge-graph module that encodes and updates domain expertise to improve generalization; (3) a multi-level feature-alignment strategy with asynchronous fusion and bidirectional encoding to enhance multimodal learning; and (4) a closed-loop learning paradigm wherein segmentation and annotation mutually refine each other. Together, these components establish a comprehensive system for automated and adaptive medical image segmentation with high clinical relevance. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.