Collaborative research: frameworks: iowarp: bending the i/o fabric for advancing AI-infused scientific workflows

The National Science Foundation (NSF) awarded a $4,000,000 Project Grant under the Computer and Information Science and Engineering (CFDA 47.070) program to the Illinois Institute of Technology (IIT) to develop IOWARP, a next-generation data management platform designed to address the challenges of modern, data-intensive scientific workflows. The IOWARP project aims to create a scalable, AI-driven solution that will streamline data access and management, accelerating the pace of scientific discovery across fields like genomics, climate modeling, and AI research.

As part of this effort, IIT has planned sub-awards, including a $250,031 sub-grant to The HDF Group to participate as a co-principal investigator. The HDF Group will lead the design and development of a unified data format and API, maintain the IOWARP CI/CD infrastructure, test the project components, and support outreach activities. The overall IOWARP initiative seeks to foster a collaborative, inclusive research environment that democratizes access to data and equips the next generation of scientists with advanced data skills.

Abstract

Scientific discovery is increasingly driven by massive, complex datasets that hold the keys to unlocking new knowledge and solving global challenges. However, the current landscape of data storage and management is struggling to keep pace with this data deluge. Iowarp is designed to create a next-generation data management platform tailored to the unique needs of modern, data-intensive science. This platform will address the fundamental challenges that researchers face when dealing with diverse data types, the exponential growth of data, and the need for rapid access to critical information. By streamlining data workflows and enhancing data accessibility, iowarp will empower scientists and researchers to focus their valuable time and resources on their core mission: making groundbreaking discoveries. Beyond its technical advancements, iowarp will also foster a collaborative and inclusive research environment, democratizing access to data and equipping the next generation of scientists with essential data skills. By accelerating research in critical fields like genomics, climate modeling, and AI-driven discovery, iowarp has the potential to unlock transformative solutions to global challenges and drive innovation. The iowarp project will engineer a modular, adaptable, and scalable data management platform that addresses the specific challenges encountered in modern scientific workflows, particularly those enhanced by artificial intelligence (AI) technologies. Iowarp will significantly reduce data access times, accelerating the pace of scientific discovery by harnessing state-of-the-art technologies like nvme ssds, CXL devices, and CPU-GPU codesigns. IT will foster a collaborative ecosystem, inviting contributions from diverse scientific and engineering communities. Building upon the team's extensive expertise in multi-tiered storage research and leveraging prior NSF investments in this field, iowarp will integrate AI-driven solutions to redefine data management for high-performance computing environments. The anticipated outcomes of iowarp will establish a thriving community-driven platform that continually evolves to meet the changing needs of scientific research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are planned for this award.