Energetic variational inference: foundations, algorithms, and applications

This federal Project Grant award, funded by the National Science Foundation's (NSF) Mathematical and Physical Sciences program (CFDA 47.049), is focused on advancing the field of variational inference, a powerful technique used in machine learning and artificial intelligence algorithms. The primary objectives are to:

1. Establish the theoretical foundations for a unified energetic variational inference framework to support and justify the use of existing and new variational inference algorithms in machine learning applications.

2. Provide a systematic procedure to create new variational inference algorithms and apply them to emerging machine learning problems.

The award, totaling $267,816, was made on May 15, 2024 to the University of Massachusetts. The project is scheduled for completion by June 30, 2025. The work includes the development of open-source software to package the new algorithms for public use, as well as educational and outreach activities to train students and the workforce on state-of-the-art machine learning tools.

The award includes two sub-grants to the IIT Research Institute ($242,932) and the University of California, Riverside ($243,928) to support collaborative research efforts within the project.

Abstract

Variational inference is a powerful tool used to boost efficiency and flexibility in machine learning and artificial intelligence algorithms, particularly those based on large amounts of data. In this project, the investigators plan to create a unified and systematic framework for variational inference methods, making two key contributions. First, the investigators will establish the theoretical foundations for the proposed framework, which will support and justify using existing and new variational inference algorithms in machine learning applications. Second, the investigators will provide a systemic procedure to create new variational inference algorithms and apply them to emerging machine learning problems. In addition to these new scientific developments, the investigators will create new courses and workshops on machine learning, recruit both undergraduate and graduate students for summer, project-based research programs, and provide mentorship to local high school students through hands-on machine learning training programs. Collaborations are planned with industrial data science partners to apply these new algorithms in practice and to train the workforce with the start-of-the-art machine learning tools. The proposed energetic variational inference framework is based on an energetic variational approach, which has been successfully used to study complicated non-equilibrium systems in physics and biology. The investigators will provide a blueprint for generating new algorithms by introducing various options for the four essential components of the proposed framework: the divergence functional, the dissipation functional, the representation of the probability density, and the temporal discretization. The investigators will study convergence in the continuous formulation as well as estimate the error bounds after temporal discretization of the underlying continuous dynamic system. More importantly, these theoretical results can be applied or extended to other flow-based variational inference approaches. These methods will be applied to problems in supervised learning, density estimation, and generative learning. Additional novel applications in machine learning, statistics, and statistical physics will also be developed. The algorithms will be packaged into open-source software for public use. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.