Federal Project Grant Award Summary

New York University received a $300,000 project grant from the National Science Foundation's Computer and Information Science and Engineering program (CFDA 47.070) effective October 1, 2025, through July 31, 2028. This collaborative research initiative focuses on developing evaluation concepts and automated assessment technologies for Large Language Models (LLMs). The project will identify a comprehensive taxonomy of evaluation concepts to diagnose LLM failure modes—such as insufficient detail in responses—and create specialized reward models (themselves LLMs) capable of reliably scoring LLM outputs against these criteria.

The deliverables include the development of automated evaluation and improvement methodologies for LLM responses, curation of training datasets demonstrating correct behavior for each evaluation concept, and iterative response improvement techniques utilizing the reward models. The project will advance LLM functionality across critical dimensions including factual accuracy, informativeness, safety, appropriate abstraction levels, format compliance, and creativity. Additionally, the project commits to open-sourcing the concept taxonomy and related technologies, thereby contributing to broader advancement of LLM assessment capabilities within the research community. This work addresses gaps in current evaluation practices by moving beyond basic quality metrics to encompass the more nuanced factors essential for effective LLM-based information systems.