The massive volumes of multi-dimensional array-oriented data generated by NOAA programs and the scientific community at large are predominantly stored in industry standard network common data form…

The National Oceanic and Atmospheric Administration (NOAA) awarded Airmettle Inc. a $300,000 Project Grant under the NOAA Small Business Innovation Research (SBIR) Program (CFDA 11.021) to explore integrating in-situ analysis capabilities for multi-dimensional data stored in Network Common Data Form (NetCDF) into its real-time smart data lake solution. The SBIR Program seeks to stimulate technological innovation and commercialization of federally funded research by small businesses. Under this award from September 1, 2022 to December 31, 2022, Airmettle will partner with the University of Wisconsin-Madison to assess the feasibility of enabling in-place analytics of massive NetCDF datasets directly within its data lake platform. This aims to unlock the potential of petabyte-scale NetCDF climate and weather data held in different locations by dramatically accelerating access and insights without moving entire datasets across networks.

The massive volumes of multi-dimensional array-oriented data generated by NOAA programs and the scientific community at large are predominantly stored in industry standard network common data form (netcdf). Key challenges exist in making use of data stored in netcdf: data sets are often too large to be copied and transferred across networks for every user, and each time data is accessed by an analytics tool IT must be retrieved, subsets extracted, and subsequently formatted, among other requirements, which can account for 80-90% of the total time needed to insight. To unlock the enormous potential of petabyte scale netcdf-formatted data stored at different locations, in this sbir phase i project, airmettle, INC. With its research partners from the university of wisconsin madison proposes to explore the feasibility of integrating in-situ analysis capabilities for multi-dimensional data (netcdf) into our highly innovative real-time smart data lake solution. Dramatically accelerated data