III: a declarative data management system for semantic multimodal workflows

The National Science Foundation Division of Computing and Communication Foundations awarded The Regents of the University of California, doing business as University of California, Berkeley, $999,997 on January 1, 2027, under the Computer and Information Science and Engineering program (CFDA 47.070).

The award funds development of an open-source declarative data management system for semantic multimodal workflows that enables users to express complex analyses over videos and documents through a simple, high-level interface while the system automatically optimizes execution. The project builds a unified data model and domain-specific language with composable operators that operate uniformly over document collections and video frame sequences, accompanied by a low-code interface for iterative workflow development. Novel optimization techniques will make large-scale multimodal data analysis accessible to journalists, city officials, transportation planners, and other non-specialists without requiring significant programming expertise or prohibitive artificial intelligence model execution costs. The resulting software and benchmarks will be released publicly to support further research and education.

The project runs from January 1, 2027, through December 31, 2030, with work performed in Berkeley, California. Beyond software development, the award supports workforce development through new course materials at the intersection of databases and artificial intelligence and through training of graduate students and postdoctoral researchers.

Abstract

The amount of data stored as videos, audio recordings, and documents is growing rapidly and now far exceeds traditional structured data. Extracting useful knowledge from these diverse sources often requires combining information across formats. For example, a traffic engineer may need to analyze camera feeds in combination with road network data to form an accurate understanding of the situation. Today, performing such analyses requires significant programming expertise, is error-prone, and can be prohibitively expensive due to the cost of running artificial intelligence models on large volumes of data. This project develops a new open-source software system that allows users to express complex analyses over videos and documents using a simple, high-level interface, while the system automatically finds efficient ways to execute them. This project will advance the science of data management and by making large-scale multimodal data analysis accessible to a broader range of users, including journalists, city officials, and transportation planners. The resulting software and benchmarks will be released publicly to support further research and education. The project also supports workforce development through new course materials at the intersection of databases and artificial intelligence, and through training of graduate students and postdoctoral researchers. Specifically, this project builds a declarative data management system for authoring, optimizing, and executing semantic workflows that span multiple unstructured data modalities, focusing initially on documents and videos. The project designs a unified data model and domain-specific language (DSL) with composable operators that operate uniformly over document collections and video frame sequences, accompanied by a low-code interface for iterative workflow development. Building on this foundation, the project develops novel optimization techniques that transfer strategies across modalities. The system will be evaluated on real-world multimodal workloads drawn from collaborations in records analysis, freeway traffic monitoring, and city council meeting analysis, using benchmarks that measure execution time, inference cost, scalability, and output quality. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.