Project Grant 2348465

The National Science Foundation (NSF) awarded a $175,000 Computer and Information Science and Engineering (CISE) Program grant to The Trustees of the Stevens Institute of Technology to develop a compressor-assisted collective communication framework for large-scale deep learning on GPU-based systems. The two-year project aims to address challenges with the communication overhead of training massive deep learning models by investigating efficient lossy compression techniques for gradient data and integrating them into an optimized GPU-aware communications framework. Key deliverables include a novel gradient compressor, GPU-accelerated collective communication library, and strategies for optimal resource sharing between training, compression, and communication tasks on the same GPU. This work seeks to enable significantly faster deep learning training speeds while preserving model accuracy, with potential applications in fields like computer vision, natural language processing, and scientific computing.