Navigation assistance using multimodal LLM and computer vision

The National Eye Institute awarded The Schepens Eye Research Institute Inc. $878,988 on May 1, 2026, under the Vision Research program (CFDA 93.867) to develop navigation assistance technology for blind and visually impaired individuals using multimodal large language models and computer vision.

The project will develop and validate three core modules for smartphone-based navigation. All Walk plans end-to-end directions using map services and guides travelers with sub-foot precision on sidewalks. NavGPT provides micro-navigation directions—solving the "last 30 feet" problem where GPS-based apps fail—using a specialized multimodal language model. CamGuide tracks vision language model-identified targets and guides users toward them with auditory feedback. Custom AI models will be trained on datasets collected from actual navigation scenarios of blind and visually impaired users, including a multimodal vision language model for scene understanding and computer vision models for object recognition. In the second phase, the three modules will be integrated into a comprehensive navigation app called SmartWay. Throughout the project, the institute will collaborate with the Carroll Center for the Blind to ensure accessibility for both blind and partially sighted users, and will conduct field evaluation studies comparing the apps with existing navigation solutions.

The award period runs from May 1, 2026, through April 30, 2031, with performance at Schepens' Boston, Massachusetts facility.

Abstract

Navigation can be a significant challenge for blind and visually impaired (BVI) individuals, often requiring assistance, for instance, remote sighted assistance via aira APP. Existing navigation aids, including mobile apps, have limitations in accuracy and reliability, providing only partial solutions. In general, GPS-based apps are not suitable for micro-navigation (short-range navigation like finding building entrances). This is known as the "last 30 feet" problem. We propose a comprehensive, accessible end-to-end navigation solution for BVI travelers, encompassing both macro-navigation and micro-navigation segments. This solution will be implemented as mobile apps that can run on smartphones. In this project, we will first develop and validate three key modules required for the solution. These modules, which can be packaged as standalone assistance apps for BVI navigation, include (1) all walk: plans end-to-end navigation directions using map services and guides travelers with sub-foot precision on sidewalks; (2) navgpt: provides micro-navigation directions using specialized multi-modal language model; (3) camguide: tracks targets suggested by large vision language model and guides BVI individuals towards them with auditory feedback. AI models will be custom-trained, including a multi-modal vision language model for scene understanding and computer vision models for object recognition. The training dataset will be collected from actual navigation scenarios of BVI individuals. In the second phase, the three modules will be integrated into a comprehensive navigation assistance APP, smartway. Throughout the entire project, we will collaborate with the research team from the carroll center for the blind to ensure the apps meet accessibility guidelines for both blind and partially sighted users. We will also conduct field evaluation studies with BVI participants, comparing our apps with existing navigation solutions such as google maps and aira.