Career: improving software test quality and developer skills through developer-in-the-loop test optimization

The National Science Foundation Division of Information and Intelligent Systems awarded North Dakota State University $500,000 on January 1, 2027, under the Computer and Information Science and Engineering program (CFDA 47.070) to improve software test quality and developer skills through developer-in-the-loop test optimization.

The project investigates how software tests can be systematically improved while helping students and developers strengthen testing skills through hands-on interaction with intelligent tool support. Work centers on analyzing how developers identify, understand, and address software test quality issues by examining open-source repositories, developer discussions, surveys, and interviews. These empirical findings inform the development of automated methods combining rule-based program analysis, large language models, and validation mechanisms to identify test quality issues and generate context-aware recommendations to improve tests as software evolves. The project integrates these methods into an evolution-aware developer-in-the-loop framework that continuously identifies test quality issues and suggests improvements.

The project produces openly available datasets, software tools, and educational resources benefiting researchers, educators, and practitioners. Broader impacts include reducing software maintenance costs, improving software reliability, advancing software testing education, and preparing a workforce with stronger software testing expertise.

Performance occurs in Fargo, North Dakota, with a period of performance from January 1, 2027, through December 31, 2031.

Abstract

Software testing plays a critical role in improving software quality by detecting defects before release. However, software tests often contain quality issues that make them difficult to understand, maintain, and evolve, reducing their effectiveness and increasing maintenance costs. Many developers receive limited training in writing high-quality tests, and existing automated tools provide limited support for identifying and addressing these issues. This project investigates how software tests can be systematically improved while helping students and developers strengthen their testing skills through hands-on interaction with intelligent tool support. The project's novelties are the integration of empirical evidence from real-world software development with developer-centered learning to create an interactive environment that supports both test improvement and skill development. The project's broader significance and importance are reducing software maintenance costs, improving software reliability, advancing software testing education, and preparing a workforce with stronger software testing expertise. The project also produces openly available datasets, software tools, and educational resources that benefit researchers, educators, and practitioners. The project investigates how developers identify, understand, and address software test quality issues by analyzing open-source repositories, developer discussions, surveys, and interviews. These empirical findings inform the development of automated methods that combine rule-based program analysis, large language models, and validation mechanisms to identify test quality issues and generate context-aware recommendations to improve tests as software evolves. The project integrates these methods into an evolution-aware developer-in-the-loop framework that continuously identifies test quality issues, suggests improvements, and provides explanations for each recommendation, while enabling developers to review, modify, and learn from the suggestions during development. User studies with students and practitioners evaluate both improvements in test quality and gains in testing skills. The project contributes to more reliable and maintainable software systems and helps students and developers strengthen their skills in writing and improving software tests. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.