Theoretical foundations of long-term reinforcement learning under uncertainty

The National Science Foundation Division of Electrical, Communications and Cyber Systems awarded The University of Central Florida Board of Trustees $300,461 on October 1, 2026, through the Engineering program (CFDA 47.041) to develop theoretical foundations for long-term reinforcement learning under uncertainty.

The project advances foundational artificial intelligence research by developing decision-making algorithms that optimize for steady-state, long-term performance in autonomous systems. Current reinforcement learning approaches prioritize immediate gains, creating short-sighted behaviors vulnerable to failure in continuous, extended operations. The research addresses this limitation by establishing theoretical and algorithmic foundations for robust average-reward reinforcement learning that maintains reliability over indefinite horizons under model uncertainty. The project specifically targets three primary constraints in existing frameworks: lack of finite-sample complexity guarantees, underdeveloped multi-agent formulations, and restrictive single-recurrent-class assumptions. The work supports safe integration of autonomous systems into safety-critical applications including healthcare, power grid management, and autonomous transportation.

Beyond algorithmic development, the project integrates research with education through open-source software libraries, new curriculum modules, robotics capstone projects, and specialized training for undergraduate and graduate students. Work is performed in Alafaya, Florida, with a period of performance from October 1, 2026, through September 30, 2029. The award is a project grant with $300,461 obligated.

Abstract

Reinforcement learning has achieved remarkable success in simulated decision-making tasks, but deploying such systems in complex, real-world environments remains a critical challenge due to the mismatch between training simulations and actual deployment conditions. Furthermore, standard learning approaches often prioritize immediate gains, resulting in short-sighted behaviors that perform poorly when systems operate continuously over extended periods. In safety-critical applications such as healthcare, power grid management, and autonomous transportation, this vulnerability and lack of long-term focus can lead to severe system failures. This project advances the foundational science of artificial intelligence by developing decision-making algorithms that specifically optimize for steady-state, long-term performance. By ensuring that autonomous agents remain robust against unexpected environmental changes and maintain reliability over indefinite horizons, the research supports the safe integration of autonomous systems into society. The project also integrates research with education by developing open-source software libraries, creating new curriculum modules and robotics capstone projects, and providing specialized training opportunities for undergraduate and graduate students. The technical goal of this project is to establish the theoretical and algorithmic foundations for robust average-reward reinforcement learning, which optimizes long-term steady-state performance under model uncertainty. The research addresses three primary limitations of current frameworks: a lack of finite-sample complexity guarantees, underdeveloped multi-agent formulations, and restrictive single-recurrent-class assumptions. To overcome these challenges, the project is organized into three central thrusts. First, the principal investigator designs a model-free learning algorithm using a quotient-space fixed-point formulation and establishes minimax lower bounds to characterize intrinsic statistical hardness. Second, the project advances multi-agent systems by proving the existence of robust nash equilibria and developing convergent equilibrium-finding algorithms based on robust temporal difference residual minimization. Third, the research develops multichain robust optimality equations by resolving non-unique worst-case transition kernels through a lexicographic gain-bias separation. The algorithms and theoretical models are evaluated through mathematical analysis, continuous-state simulations, and practical deployments on electric vehicle mobility simulators and quadruped robotics platforms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.