Career: an architecture-aware optimization theory for deep learning: non-euclidean descent, structured preconditioning, and scale invariance

The National Science Foundation Division of Computing and Communication Foundations awarded Toyota Technological Institute at Chicago $349,963 on June 1, 2026, under the Computer and Information Science and Engineering program (CFDA 47.070) for a CAREER project developing optimization theory architecture-aware for deep learning.

The project addresses the efficiency and cost of training neural networks by linking optimization methods to neural network structure. Research focuses on three complementary directions: structured preconditioning, optimization methods adapted to different notions of distance, and scale invariance induced by normalization layers. The work will analyze representative model components including multilayer perceptrons, attention modules, embedding parameters, and layers preceding normalization; develop theory explaining when architecture-matched optimization methods improve training efficiency; characterize optimization behavior on losses whose landscape geometry is shaped by network architecture; and study how optimization affects learned solution quality beyond training loss, including downstream task performance and robustness when data differ from training distributions. The project includes graduate education in optimization for deep learning, research-preparation activities for undergraduates, and hands-on artificial intelligence learning modules for local high school students, with participation open to all.

Performance runs through May 31, 2031, at Chicago, Illinois.

Abstract

Training modern artificial intelligence systems requires large amounts of computing time, energy, and money. Many of the optimization methods used to train neural networks are still chosen largely through trial and error because existing theory does not adequately explain why some methods work better than others on different model architectures. This project will develop a scientific foundation for making training faster, more reliable, and more resource efficient by linking optimization methods to the structure of the neural networks they are used to train. The project can reduce the cost and energy use of model training, provide more dependable guidance for practitioners, and support efficient and reliable artificial intelligence development. IT will also support graduate education in optimization for deep learning, research-preparation activities for undergraduates, and hands-on artificial intelligence learning modules for local high school students, with participation in project activities open to all. This project studies how neural network architecture influences optimization through three complementary directions: structured preconditioning, optimization methods adapted to different notions of distance, and scale invariance induced by normalization layers. The research will analyze representative model components such as multilayer perceptrons, attention modules, embedding parameters, and layers preceding normalization. IT will develop theory explaining when optimization methods matched to the model architecture improve training efficiency, characterize their behavior on losses whose landscape geometry is shaped by the network architecture, and study how optimization affects the quality of learned solutions beyond training loss alone, including downstream tasks and performance when data differ from the training distribution. These ideas will be tested through controlled experiments on representative neural network architectures and larger-scale validation on production-relevant training pipelines. The project will release open-source software, reproducible experimental configurations, benchmark results, and public documentation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.