Collaborative research: RUI: hnds-r: learning task-relevant visual features by linking large language models and deep convolutional neural networks

Colgate University received a $234,610 Project Grant from the National Science Foundation's Division of Behavioral and Cognitive Sciences under the Social, Behavioral, and Economic Sciences program (CFDA 47.075), effective September 1, 2025, through August 31, 2028. This collaborative research initiative integrates visual artificial intelligence (Convolutional Neural Networks), language-based artificial intelligence (Large Language Models), neuroscience, and cognitive science to advance understanding of how humans flexibly extract and utilize task-relevant visual information guided by contextual goals. The project delivers computational models trained to predict language embeddings of human scene descriptions under varying task objectives, activation maps identifying image regions most relevant to specific tasks, and a novel multivariate analysis technique called dynamic electrode-to-image mapping for tracking task-relevant feature processing in the brain.

The research outputs include validated behavioral and electroencephalography (EEG) experimental data demonstrating how visual, conceptual, and neural systems interact to support goal-directed perception. These products directly support the broader National Science Foundation objective of promoting fundamental research in the social and behavioral sciences while advancing the development of more adaptive, human-aligned artificial intelligence systems that approach human cognitive flexibility. The project emphasizes undergraduate research engagement consistent with Colgate University's commitment to research-focused educational development.

Abstract

Building artificial intelligence (AI) systems that approach human cognitive flexibility requires a better understanding of how the brain uses visual and linguistic information to achieve specific goals. While previous research in cognitive neuroscience and AI has focused on visual classification tasks, such as identifying objects or labeling scenes, real-world behavior is more nuanced and often depends on selecting task-relevant information, guided by the observer?s goals. Critically, this process draws not only on the visual features of the scene, but on conceptual and linguistic knowledge as well. This project examines how people flexibly extract and use visual information in context and how this information is represented in computational models, supporting the goal of advancing theories of cognition and the development of more adaptive, human-aligned AI systems. The project integrates methods from visual AI (convolutional neural networks), language-based AI (large language models), neuroscience, and cognitive science. First, deep networks are trained to predict language embeddings of human scene descriptions elicited under different task goals, capturing how semantic meaning maps onto visual features. Next, these networks are reverse-engineered to generate activation maps that identify the regions of each image most relevant for a given task. These maps are validated using both behavioral experiments and electroencephalography (EEG). A novel multivariate analysis technique (dynamic electrode-to-image mapping) is used to track when and how these task-relevant features are processed in the brain. Finally, the project assesses whether features identified by the brain contribute to successful behavior. This approach reveals how visual, conceptual, and neural systems interact to support goal-directed perception, offering a new framework for understanding scene processing and for building AI systems that better reflect human needs and capacities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.