Attachment 0006 - Technology Readiness Assessment Guidance.pdf

January 2020

GAO-20-48G

Assessment Guide

Technology Readiness

Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects

Page i GAO-20-48G TRA Guide

Preface 1

Introduction 4 What is a Technology Readiness Assessment? 9 Why TRAs are Important and Understanding their Features 24 The Characteristics of a High-Quality TRA and a Reliable Process for Conducting Them 32 Step 1 – Prepare the TRA Plan and Select the TRA Team 38 Step 2 – Identify the Critical Technologies 47 Step 3 – Assess the Critical Technologies 63 Step 4 – Prepare the TRA Report 71 Step 5 – Use the TRA Report Findings 80 Technology Maturation Plans 88 TRAs and Software Development 97 Development of System-level Readiness Measures 100 Glossary 103

Appendix I Objectives, Scope, and Methodology 105

Appendix II Key Questions to Assess How Well Programs Followed the Five Step Process for Conducting a TRA 106

Appendix III Case Study Backgrounds 110

Appendix IV Examples of Various TRL Definitions and Descriptions by Organization 115

Appendix V Other Types of Readiness Levels 122

Appendix VI Agency Websites where TRA Report Examples can be Found 128

Contents

Page ii GAO-20-48G TRA Guide

Appendix VII Experts Who Helped Develop This Guide 129

Appendix VIII Contacts and Acknowledgments 135

References 136

Image Sources 142

Tables

Table 1: Cost and Schedule Experiences for Products with Mature and Immature Technologies 27

Table 2: Five Steps for Conducting a Technology Readiness Assessment (TRA) and Associated Best Practices and Characteristics 35

Table 3: Software Implementation Project Work Breakdown Structure 55

Table 4: Technology Readiness Levels (TRLs) and the Information Needed to Demonstrate Them for Hardware and Software 66

Table 5: Examples of Program Management Tools and Analytical Approaches Used with Technology Readiness Assessments (TRAs) 86

Table 6: GAO Reports Used as Case Study in the TRA Guide 110 Table 7: DOD Technology Readiness Levels (2011) 115 Table 8: DOD Software Technology Readiness Levels (2009) 115 Table 9: NASA Hardware Technology Readiness Levels (2013) 116 Table 10: NASA Software Technology Readiness Levels (2013) 117 Table 11: DOE Technology Readiness Levels (2011) 118 Table 12: Department of Transportation Technology Readiness

Levels (2017) 120 Table 13: DOD Manufacturing Readiness Levels 122 Table 14: Integration Readiness Levels 125 Table 15: System Readiness Levels 126 Table 16: Experts Who Made Significant Contributions 129 Table 17: Experts Who Made Noteworthy Contributions 130

Figures

Figure 1: Technology Readiness Levels (TRL) 11

Page iii GAO-20-48G TRA Guide

Figure 2: Phased Acquisition or Project Life-Cycle with Decision Points 14

Figure 3: Illustration of Technology Readiness Assessments (TRA) in a Program’s Integrated Schedule 23

Figure 4: Five Steps for Conducting a High-Quality Technology Readiness Assessment (TRA) 35

Figure 5: Four Steps for Selecting Critical Technologies (CT) 52 Figure 6: A Common Element Work Breakdown Structure (WBS) 53 Figure 7: A Contract Work Breakdown Structure (WBS) 54 Figure 8: A Process Technology Flow Diagram (simplified) 57 Figure 9: Three Steps to Evaluate Critical Technologies (CT) 64 Figure 10: Technology Readiness Assessment (TRA) Report

Template 73 Figure 11: Five Steps to Prepare the Technology Readiness

Assessment (TRA) Report 74 Figure 12: Acquisition or Project Life-cycle with Technology

Readiness Assessments (TRA) at Decision Points for Governance 83

Figure 13: Five Steps to Prepare a Technology Maturation Plan

(TMP) 89

Figure 14: Technology Maturation Plan (TMP) Template 94 Figure 14: Technology Maturation Plan (TMP) Template

(continued) 95

Page iv GAO-20-48G TRA Guide

Abbreviations

CDR Critical Design Review CT critical technologies DOD U.S. Department of Defense DOE U.S. Department of Energy EMD Engineering and Manufacturing Development IRL integration readiness level MDA Milestone Decision Authority MDAP Major Defense Acquisition Program MRL manufacturing readiness level NASA National Aeronautics and Space Administration SRA systems readiness assessment SRL system readiness level TMP technology maturation plan TRA technology readiness assessment TRL technology readiness level WBS work breakdown structure

This Guide is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Page 1 GAO-20-48G TRA Guide

441 G St. N.W.

Washington, DC 20548

Federal agencies spend billions of dollars each year to develop, acquire, and build major systems, facilities, and equipment, including fighter aircraft, nuclear waste treatment facilities, electronic baggage screening equipment, and telescopes for exploring the universe. Managing these complex acquisitions has been a long-standing challenge for federal agencies.

Many of the government’s most costly and complex acquisition programs require the development of cutting-edge technologies and their integration into large and complex systems. Such acquisition efforts may also use existing technologies, but in new applications or environments. For two decades, the U.S. Government Accountability Office (GAO) has shown that using effective management practices and processes to assess how far a technology has matured and how this has been demonstrated are fundamental to evaluating its readiness to be integrated into a system and managed for risk in the federal government’s major acquisitions.

A technology readiness assessment (TRA) is a systematic, evidence-based process that evaluates the maturity of technologies (hardware, software, and processes) critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program, including cost and schedule.1 TRAs, which evaluate the technical maturity of a technology at a specific point in time for inclusion into a larger system, do not eliminate technology risk. But when done well, they can illuminate concerns and serve as the basis for realistic discussions on how to address potential risks as programs move from the early research and technology development to system development and beyond.2 In addition, TRAs help legislators, government officials, and the public hold government programs accountable for achieving technology performance goals.

1In this Guide, we use the term “program,” but some agencies may make distinctions between programs, projects, and products. For the purposes of this Guide, these terms can be used interchangeably to accommodate an agency’s particular application. The methodology developed in this Guide is intended for use in any acquisition, program, project, or product that benefits from the use of TRAs to evaluate and monitor the maturity of critical technologies.

2This Guide refers to “technology development” as the design, development, and engineering of a critical technology or group of technologies, whereas “system development” refers to the design, engineering development, or product development of a system.

Letter

Preface

Page 2 GAO-20-48G TRA Guide

This TRA guide (the Guide) is a companion to GAO’s Cost Estimating and Assessment Guide and the Schedule Assessment Guide.3 With the Guide, GAO is establishing a methodology for evaluating technology maturity based on best practices that can be used across the federal government. This is particularly salient as it relates to determining a program or project’s readiness to move past key decision points that typically coincide with major commitments of resources. Similar assessments can also be made by technology developers and program managers conducting self-assessments (referred to in this Guide as knowledge-building TRAs) during the course of a project to help them evaluate technology maturity, gauge technology development progress, and identify and manage risk. Existing TRA guidance in government agencies and industry may include similar strategies for evaluating technology maturity, but there is no widely held or accepted process for doing so. The science and technology, program management, and systems engineering communities each view technology readiness through their own lenses, which can make for variable results. In addition, some agencies have deemphasized the use of TRAs or questioned their value. We hope that this Guide can reinvigorate the use of TRAs in those organizations.

The Guide is intended to provide TRA practitioners, program managers, technology developers, and governance bodies throughout the federal government a framework for better understanding technology maturity, and best practices for conducting high-quality assessments.

Organizations that have developed their own guidance can use the Guide to support and supplement their practices. Organizations that have not yet developed their own policies can use it to begin establishing their own guidance. As a companion to GAO’s cost and schedule assessment guides, this Guide can also help GAO and other oversight organizations evaluate other agencies’ basis for their conclusions and decisions about technology readiness.

We intend to keep the Guide current. We welcome comments and suggestions from experienced practitioners, as well as recommendations from experts in the science and technology, systems engineering, and program acquisition disciplines.

3GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2, 2009), and GAO, Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: Dec 22, 2015).

https://www.gao.gov/products/GAO-09-3SP https://www.gao.gov/products/GAO-16-89G https://www.gao.gov/products/GAO-16-89G

Page 3 GAO-20-48G TRA Guide

If you have any questions concerning the Guide, you may contact Dr.

Timothy Persons at (202) 512-6888 or personst@gao.gov, or Michele Mackin at (202) 512-4841 or mackinm@gao.gov. Contact points for GAO’s Office of Congressional Relations and Office of Public Affairs may be found on the last page of this Guide.

Timothy M. Persons, Ph.D.

Chief Scientist and Managing Director Science, Technology Assessment, and Analytics Team

Michele Mackin Managing Director Contracting and National Security Acquisitions mailto:personst@gao.gov mailto:mackinm@gao.gov

Page 4 GAO-20-48G TRA Guide

Technology readiness assessments (TRA)—evaluations that determine a technology’s maturity—have been used widely at the U.S. Department of Defense (DOD) and National Aeronautics and Space Administration (NASA) since the 1990s.4 This approach has also been embraced by other government agencies, as well as industry in aerospace, maritime, oil and gas, electronics, and heavy equipment that use TRAs to help manage their acquisitions. Relatively few agencies have guides for assessing a technology’s maturity and its readiness for integration into larger acquisition programs, and the federal government has not adopted a generally accepted approach for evaluating technology beyond using technology readiness level (TRL) measures.5 This TRA Guide is intended to help fill those gaps.

The Guide has two objectives: (1) to describe generally accepted best practices for conducting high-quality TRAs of technology developed for systems or acquisition programs, and (2) to provide technology developers, program managers, and governance bodies with useful information to more effectively mature critical technologies, determine a technology’s readiness, and manage and address risk.6 In addition, oversight bodies—such as those with department or agency acquisition officials or government auditors—may use the Guide to evaluate whether the fundamental processes and best practices of effective TRAs have been followed and whether a TRA demonstrates the characteristics (credibility, objectivity, reliability, and usefulness) of a high-quality assessment. Appendix II lists the key questions to evaluate how well programs have followed the steps and best practices in this Guide.

The Guide recognizes that TRAs have various customers within organizations, such as governance bodies charged with program oversight, as well as more narrow audiences, such as the technology

4DOD introduced TRAs in the 1990s. A NASA researcher pioneered the first technology readiness scale in 1974 with seven levels which were not formally defined until 1989.

5TRLs are a scale of nine levels used to measure a technology’s progress, starting with paper studies of a basic concept and ending with a technology that has proven itself in actual usage in the product’s operational environment.

6TRAs do not assess the risk associated with the technical maturity of a technology or system. Rather they identify specific risks (e.g., performance data gaps) associated with the specific technologies, which provides a basis for quantifying those risks through formal risk assessments. Similarly, the technology maturation plan (TMP) resulting from a TRA, described in the section Technology Maturation Plan, provides the basis for appropriate risk mitigation actions.

Introduction

Page 5 GAO-20-48G TRA Guide developers, program managers, or systems engineers that use TRAs to determine progress in achieving technology maturity goals. The Guide discusses TRAs in context of the full range of best practices for governance, and also provides information where steps may be tailored for self-assessments, or knowledge-building TRAs, conducted for narrower audiences.

The Guide’s chapters first introduce the concept of a TRA, its basis in government and commercial best practices for product and systems development, and the benefits a program, agency, or organization might expect to gain from conducting a TRA. It then maps the characteristics of a high-quality TRA, which include credibility, objectivity, reliability, and usefulness, to each of the best practices within the five TRA steps.

Specific chapters are devoted to each of the five steps for conducting a TRA, followed by chapters that discuss the preparation of a technology maturation plan (TMP) for technologies assessed as immature, the current state of practice related to the assessment of software intensive technologies, and tools for evaluating system-level readiness, which is an extension of the concept of technology readiness.

The Guide draws heavily from DOD, NASA, and the Department of Energy (DOE) for best practices and terminology. In addition, the Guide draws from resources, materials, and tools developed and applied by experts and organizations in order to capture the current thinking on technology readiness and maturity. Existing government agency guidance is largely geared toward conducting TRAs to support major acquisition decisions, in particular the decision to authorize the start of product or system development and allocation of substantial resources.

Demonstrating that a program’s critical technologies (CT) have been proven to work in their intended operational environment before making a commitment to product development has also been the focus of GAO’s work on technology readiness since the late 1990s.7

While this Guide focuses on how to conduct high-quality TRAs from the start of technology development to support technology and product

7Critical technologies are technology elements deemed as critical if they are new or novel, or used in a new or novel way, and are needed for a system to meet its operational performance requirements within defined cost and schedule parameters. These technology elements may be hardware, software, a process, or a combination thereof that are vital to the performance of a larger system or the fulfillment of the key objectives of an acquisition program.

Page 6 GAO-20-48G TRA Guide development decisions, the expert community recognizes that more frequent, regular self-assessments of the maturity of a project’s or program’s CTs are also best practices for technology and program managers. However, some experts have been concerned that applying the same set of practices to these more frequent assessments might be too time consuming and cost prohibitive and ultimately dissuade technology and program managers from conducting them. To that end, the Guide emphasizes that best practices for conducting TRAs can in some cases be tailored and routinely applied to meet specific program goals. These goals range from increasing the knowledge of program managers to better understanding transition risks when demonstrating readiness for product development.

The Guide contains a number of case studies drawn from GAO reviews to augment the text. These case studies highlight problems typically associated with technology development efforts, as well as good practices, and emphasize the main points and lessons that the chapters cover. For example, GAO has found that in many programs, cost growth and schedule delays resulted from overly optimistic assumptions about maturing a technology. Experts have also found that many program managers and technology developers suffer from the assumption that they can deliver state-of-the-art technology upgrades within a constrained budget without available evidence that the technology will perform as expected in its operational environment. Appendix III includes background information for each program used in the case studies.

The primary audiences for this Guide are the organizations and the program managers and technology developers who rely on and develop technology for acquisition programs, the governance bodies that oversee acquisition efforts and make important decisions about the commitment of organizational resources, the contractors that develop technology, and the audit community that evaluates these efforts. Organizations that do not have formal policies for conducting or reviewing TRAs will benefit from the Guide because it can inform them of the criteria GAO may use in evaluating their programs. In addition to GAO, other audit organizations including the Inspector General, may also use the criteria prescribed in the Guide for their work. We intend to periodically update the Guide.

Comments and suggestions from experienced users, and recommendations from experts in the relevant fields are encouraged.

The Guide’s Case Studies

The Guide’s Readers

Page 7 GAO-20-48G TRA Guide

To ascertain the generally accepted best practices for conducting high-quality TRAs, we worked with practitioners and technology experts from across the federal government, commercial industry, nonprofits, and academia. We conducted periodic in-person meetings at our headquarters building in Washington D.C., as well as virtual meetings with a community of over 180 experts where we collected information, facilitated focus groups, and elicited feedback on iterative drafts of the Guide’s chapters. To ensure the Guide reflected a broad range of knowledge and viewpoints in order to identify useful information for maturing critical technologies and determine technology readiness and address risk, we consulted with experts from science and technology, systems engineering, nuclear engineering, software and computer sciences, risk management, and acquisition policy and program management disciplines. We released a public exposure draft of the GAO Technology Readiness Assessment Guide (GAO-16-410G) in August 2016 for a 12-month comment period. A GAO expert panel made up of representatives from several mission teams adjudicated more than 400 comments from August 2017 to June 2019.

We conducted our work from January 2013 to December 2019 in accordance with all sections of GAO’s Quality Assurance Framework that are relevant to our objectives. The framework requires that we plan and perform the engagement to obtain sufficient and appropriate evidence to meet our stated objectives and to discuss any limitations in our work. We believe that the information and data obtained, and the synthesis of information conducted, provide a reasonable basis for the guidance in this product. We describe our objectives, scope and methodology in detail in appendix I.

GAO thanks the many members of the technology readiness assessment community who helped make the Guide a reality. After we discussed our conceptual plan to embark on this effort to develop a government-wide TRA Guide, experts from across the federal government, commercial industry, nonprofits, and academia expressed interest in working with us.

From our first kick-off meeting in January 2013 forward, their contributions have been invaluable. Together with these experts, GAO has developed a Guide that outlines the best practices and key characteristics of high-quality TRAs and promotes their use to benefit many agencies in the federal government as well as other organizations in the United States and abroad. We would like to thank everyone who gave their time, attended meetings, provided valuable documentation, and responded to requests for comments. Those who worked with us on this Guide are

The Guide’s Approach

Acknowledgments https://www.gao.gov/products/GAO-16-410G

Page 8 GAO-20-48G TRA Guide listed in appendix VII. Additional contacts and acknowledgments are in appendix VIII.

Page 9 GAO-20-48G TRA Guide

A TRA is a systematic, evidence-based process that evaluates the maturity of CTs (hardware, software, process, or a combination thereof) that are vital to the performance of a larger system or the fulfillment of the key objectives of an acquisition program.8 It is a normal outgrowth of the system engineering process and relies on data generated during the course of technology or system development. The TRA frequently uses a maturity scale—technology readiness levels (TRLs)—that is ordered according to the characteristics of the demonstration or testing environment under which a given technology was tested at defined points in time. The scale consists of nine levels, each one requiring the technology to be demonstrated in incrementally higher levels of fidelity in terms of its form, the level of integration with other parts of the system, and its operating environment than the previous, until the final level where the actual operation of the technology is in its final form and proven through successful mission operations. The TRA evaluates CTs at specific points in time for integration into a larger system.

A TRA can be conducted and updated with regular frequency throughout the acquisition or project life-cycle, and there is no pre-determined number of TRAs or time intervals for conducting these evaluations.9 Similarly, it is not a requirement that each TRA comprehensively consider all CTs. Rather, the key consideration is that each CT should be evaluated during development. While the TRA does not measure or assign a risk level to a project or assess the ability to achieve system cost, schedule, or performance goals, it is a fundamental means for evaluating an important component of risk—the maturity of a technology and its readiness or ability to perform as part of a larger system. The TRA process is a risk identification tool that will help to highlight CT maturity concerns.

8A technology element is considered a critical technology if it is new or novel, or used in a new or novel way, and it is needed for a system to meet its operational performance requirements within defined cost and schedule parameters.

9Certain certifications and determinations are statutorily required to be made prior to a DOD major acquisition program’s milestone B decision. The Department of Energy (DOE), Office of Environmental Management, requires that TRAs and technology maturation plans (TMP) be conducted for major projects where new critical technologies are being developed prior to critical decision 2. The DOE also highly recommends these assessments for smaller projects, as well as operations research efforts, such as technology demonstrations that involve the development and implementation of new technologies or technologies in new operational environments.

What is a Technology Readiness Assessment?

Page 10 GAO-20-48G TRA Guide

GAO has found that the readiness of CTs at the start of technology development affects the schedule and cost of developing a product.10 Therefore, a TRA performed before development begins provides important information for both the technology developer and program manager responsible for the daily management of developing a product and the governance bodies charged with the oversight of an acquisition program.

TRLs are the most common measure for systematically communicating the readiness of new technologies or new applications of existing technologies (sometimes referred to as heritage technologies) to be incorporated into a system or program.11 TRLs are a compendium of characteristics that describe increasing levels of technical maturity based on demonstrated (tested) capabilities. The performance of a technology is compared to levels of maturity (numbered 1-9) based on demonstrations of increasing fidelity and complexity. Other readiness level measures, for example manufacturing readiness levels (MRL), have been proposed with varying degrees of success and use throughout the life- cycle of a program. 12, 13 Although not exhaustive, appendix V describes other types of readiness levels.

Some organizations have tailored the TRL definitions to suit their product development applications. In general, TRLs are measured on a 1-9 scale, where level 1 generally represents paper studies of the basic concept, moving to laboratory demonstrations around level 4, and ending at level 9, where the technology is tested and proven, integrated into a product, 10See GAO-09-3SP, GAO-12-120G, and GAO, Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999).

11Heritage technologies are technologies that have been used successfully in operation.

These technologies may be used in new ways where the form, fit or function changed; the environment to which they will be exposed to in their new application is different than those for which they were originally qualified; or process changes have been made in their manufacture.

12A scale that consists of ten levels designed to assess the maturity of a given technology, system, subsystem, or component from a manufacturing perspective. MRLs provide decision makers (at all levels) with a common understanding of the relative maturity (and attendant risks) associated with manufacturing technologies, products, and processes being considered to meet DOD requirements.

13GAO, Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks are Managed, GAO-10-439 (Washington, D.C.: Apr. 22, 2010).

Overview of Technology Readiness Levels https://www.gao.gov/products/GAO-09-3SP https://www.gao.gov/products/GAO-12-120G https://www.gao.gov/products/GAO/NSIAD-99-162 https://www.gao.gov/products/GAO/NSIAD-99-162 https://www.gao.gov/products/GAO-10-439

Page 11 GAO-20-48G TRA Guide and successfully operated in its intended environment. Figure 1 features the nine TRLs and descriptions that DOD, NASA, and other organizations use. Appendix IV provides additional examples of government agency TRL definitions and descriptions, including those for both hardware and software.

Figure 1: Technology Readiness Levels (TRL)

Page 12 GAO-20-48G TRA Guide

While a TRA uses TRLs as a key measure for evaluating CTs, an assessment is more than just a single number at single points in time.

TRAs are a compilation of lower-level assessments that may span several years, depending on the program schedule and complexity of the development. Assessments can help gauge the progress of a technology’s development, inform project plans, and identify potential concerns for decision makers throughout acquisitions. Conducting TRAs periodically and during the earlier phases of development can identify potential concerns before risks are carried into the later and more expensive stages of system development. TRAs can also facilitate communication between technology developers, program managers, and acquisition officials throughout development and at key decision points by providing a common language for discussing technology readiness and related technical risks. Finally, TRA results can inform other assessments and planning activities, such as cost and schedule estimates, risk assessments, and technology maturation plans.

Acquisition programs and projects in many organizations are broadly divided into phases of technology development, product development, production, and operation activities. These phases may be further divided by decision points or stage gates with criteria and activities that should be met or completed before committing additional resources to the project.

Passing from one decision point to the next requires evidence and documentation, such as test reports, data analysis, and other assessments to demonstrate that these criteria have been met. During the acquisition life-cycle, TRAs can monitor the progress of maturing technologies and determine how ready a technology is to make a transition from technology development to subsequent phases.

In addition to TRAs, organizations use other types of assessments to examine the technical aspects of acquisition, such as critical design reviews to ensure that a system can proceed into fabrication, demonstration, and tests and the technology can meet the performance requirements within cost and schedule. Other assessments include systems engineering reviews used to examine the integration of components into systems, test reports used to detail the outcomes of developmental tests, and manufacturing readiness assessments used to examine the maturity of the processes that will be applied to manufacture

Uses of TRAs

Overview of the Acquisition Program Life-cycle and TRAs

Page 13 GAO-20-48G TRA Guide the product.14 Each of these reviews provides incremental knowledge during the course of a program and helps managers assess how well a project is progressing. Taken together, the different kinds of reviews and assessments develop a picture of how the project is proceeding and may highlight risk areas.

The Guide focuses on strategies that begin with a new or novel solution to a set of needs and requirements that should integrate technologies of varying technology readiness levels into a program or larger system.

Therefore, acquisitions considered here have their origin in a set of performance requirements requiring a materiel solution. These solutions often require some technology development as well as requirements to integrate with other systems as part of the acquisition process. This technology development may involve new invention, technology maturation, or the adaptation of existing technologies or heritage technologies for new applications or environments.

Figure 2 depicts a four-phased acquisition process: technology development, product development, production, and operations. Each broad phase may contain a number of activities designed to increase knowledge about the technologies and product being developed, built, and eventually operated. Each phase has a transition to the next with a documented evidence-based review that demonstrates the knowledge gained during the phase and the progress in development compared to goals and exit criteria established for the phase.

14For some applications, such as the development of complex chemical processing facilities, validation of the performance of all of the technology elements, including critical technologies, in an integrated system is crucial to the technology maturation process.

Such comprehensive assessments may use a system readiness assessment (SRA) methodology that provides a systemic understanding of the development life-cycle of the entire system and its interfaces with external entities. Such assessments of the integrated processing system should be completed as part of key TRAs.

Page 14 GAO-20-48G TRA Guide

Figure 2: Phased Acquisition or Project Life-Cycle with Decision Points

Since each phase comprises multiple activities, the acquisition or project life-cycle may be further divided into “decision points” or “stage gates” where activities are focused on a narrower developmental task than those encompassed by an entire phase. For example, during the technology development phase, one stage may focus on exploring technologies and a subsequent stage may be concerned with maturing selected technologies. A third stage could consist of activities to help in the transition to mature technologies in the product development phase. A decision point at the end of the transition gate would signal the start of product development. Figure 2 is a notional illustration since each organization creates a model that fits the specific types of acquisition processes they use.

The following descriptions highlight characteristics of each phase of the acquisition or project life-cycle and the potential role of TRAs within them.

Technology development is a continuous discovery and development process reflecting close collaboration between the science and technology community, the user, and the system developer. It is iterative, designed to assess the viability of technologies while refining user requirements. In the technology development phase, the science and technology community explores available technologies (e.g., conceptual systems, components, or enabling technology areas) and matures them to a stage in which they may be able to integrate into a product as part of

Technology Development

Page 15 GAO-20-48G TRA Guide a formal acquisition program.15 This typically occurs when the technology reaches at least a TRL 6 and preferably a TRL 7.

When technology is not adequately mature to transition to product development, the program manager should (1) consider using an alternative technology that is mature and that can meet the user’s needs,

(2) engage the user in a dialogue on appropriately modifying the requirements, or (3) continue to mature the selected technology. In this phase, regular assessments of technology progress provide confidence to the product developers that the technology is advancing towards functioning in a product within available resources of time and funding.

Evidence-based documentation may include multiple TRAs that can inform analyses of alternative solutions, or baseline technology strategies, gauge the progress of development efforts, and establish or update maturation plans to increase the likelihood for successful transition of technology into product development.16 It is important to point out that agencies approach alternative technologies in different ways. For example, some agencies conduct an analysis of alternatives to identify the most mature, cost-effective technologies, using a tailored knowledge-building TRA process to select the technology elements that will constitute a system.17 If there are technologies that can perform similar functions that are at similar TRLs, and require technology maturation and additional performance data, parallel technology development and testing is often used in the early stages to develop the data required to determine whether this alternative technology might achieve better outcomes.

Product development should involve the continued reduction of technology risk, especially as it relates to the integration of technologies into a product or system design. Ideally, product development begins with

15Enabling technologies are products of research and development including processes, equipment, and methodologies that facilitate significant gains in performance or capabilities for a particular industry.

16Experts recognize that knowledge-building TRAs are cost-effective and useful tools for informing the technology maturation process, which helps limit the more resource-intensive formal, independent TRAs conducted for key decision points or stage gates.

17An analytical comparison of the operational effectiveness, suitability, and life-cycle cost of alternatives materiel solution that satisfy an established capability need identified in an Initial Capabilities Document. It focuses on identification and analysis of alternatives, Measures of Effectiveness, schedule, Concepts of Operations, and overall risk. An analysis of alternatives also assesses Critical Technology Elements associated with each proposed materiel solution, including; technology maturity, integration risk, manufacturing feasibility, and technology maturation and demonstration needs.

Product Development

Page 16 GAO-20-48G TRA Guide the transition of mature technologies into the project or system and ends when the product design is complete and developmental testing has shown that the various components can work together as an integrated whole and can be manufactured and sustained within established cost, schedule, and quality goals.

Product development activities include the continued maturation of technologies, development and refinement of the design including the preparation of detailed design drawings, construction of higher fidelity prototypes of components and systems, integration activities to ensure that the components work together, testing to ensure that performance and reliability expectations can be met, and demonstrations of manufacturing capabilities to show that the product can be consistently produced within cost, schedule, quality, and performance goals. Product development may be the last phase for organizations, such as NASA who may build a single product where there is no production of multiple units.

TRAs during this phase can ensure that the technologies are fully mature before proceeding into production. That is, the technologies have been demonstrated as an integrated system in an operational environment and are likely to meet key performance requirements.18 Upon entering product development and therefore having achieved at least TRL 6 (system demonstration in a relevant environment) the CT is now considered beyond the reliance of science and technology investment and is dependent on standard systems engineering development practices to achieve a fully mature status. During the product development process, TRAs are important inputs into systems engineering events, such as a project’s preliminary design review and critical design review, and can expose knowledge gaps. If a project has a lower than recommended TRL (i.e., less than TRL 7) by preliminary design review, then the project does not have a solid technical basis of its design and the program could put itself at risk of approving a design that is less likely to remain stable.

The beginning of the production phase marks the point at which the elements of a product—its technologies and design—are sufficiently mature for initial production. Manufactured items are subjected to acceptance testing designed to ensure that the manufactured products are maintaining quality standards, before they are placed in the inventory.

18For some applications, extended factory acceptance testing, which can include integrated testing of the actual components to be placed into service, is used as part of the technology maturation process, and overall risk mitigation strategy.

Production

Page 17 GAO-20-48G TRA Guide

During this period, production processes are under statistical control and used to ensure the product has attained sufficient reliability and can be produced at an efficient rate within cost and quality goals. Depending on quantities to be produced, production may span 10 years or more.

TRAs are not typically conducted during this phase. However, programs considering incremental upgrades of capabilities or changes to system designs to address issues, such as parts obsolescence, may still conduct TRAs.

The operation phase marks the period of actively using the product.

Organizations may subject the product to follow-on operational testing or to inspections to ensure it is performing as designed. Operational time periods vary, depending on the maturity of the products and their average useful life. The life of a product is determined by its use and by its materials. Buildings, such as nuclear containment facilities may have a 30-year life. Military equipment is routinely projected to have a 15-30 year life-cycle. Systems designed for scientific investigation may have life-cycles that run from 5-15 years. During the operational phase, products are maintained and may undergo refurbishing or receive upgrades.

Obsolescence of technologies (that is, when it becomes too costly or infeasible to maintain old technology) is an important factor as is continued supply of the production components, including spare parts and replenishments.

Similar to the production phase, TRAs may be conducted during the operation phase as part of efforts to upgrade system capabilities, address obsolescence, or plan for follow-on efforts to eventually replace the system.

Not every organization develops or manufactures a unique product to meet its needs. While some organizations may develop new products and establish new programs, they may also undertake other work, such as upgrades to existing products or modifications to products developed by commercial vendors, adjusting the products to meet agency standards or needs. If the project is incrementally developing a product to fill emerging needs, the product may meet minimum requirements but not the desired end state. Successive iterations of development bring the product design to its full capability.

TRAs can play an important role in informing the timing of incremental upgrades by providing information on whether the technologies are

Operation

Technology Development Approaches

Page 18 GAO-20-48G TRA Guide mature and ready to be integrated onto a product. In the case of incremental development or evolutionary acquisition, each product iteration depends on the availability of mature technologies. This may entail successive technology development phases. Program strategies, such as block upgrades, pre-planned product improvements, or similar efforts that provide a significant increase in operational capability, may be managed as separate increments.19 In an evolutionary acquisition, identifying and developing the technologies necessary for follow-on increments continues in parallel with the acquisition of preceding increments, allowing the mature technologies to more rapidly proceed into the product development phase.

TRA data collection efforts may be incorporated as an integral part of systems engineering processes upfront and throughout the development and acquisition of a program. When planned and executed well, TRAs are complementary to existing program management activities, system development efforts, and oversight functions by governance bodies. Many practices needed to produce a TRA are a natural outgrowth of sound systems engineering practices, such as identifying CTs, creating a detailed systems structure, developing a plan for ensuring that CTs are evaluated, collecting evidence of the evaluation, and retaining the results as proof that these processes were undertaken as evidence of progress toward maturity. The program manager for the government and for the contractor, the internal project management and engineering teams, as well as technology developers, will use these documents to inform the management of the program and track its progress.

Programs are also subject to periodic oversight from governance bodies and other decision makers who have responsibility for ensuring that acquisitions are progressing and are ready to move forward past key decision points. For these decision points, TRAs provide evidence that the product’s technical development is progressing as desired and that technologies are mature enough to move to the next phase of development. If program managers have conducted multiple TRAs to help inform their management of the technology development process, then they have already built a knowledge base that can provide persuasive evidence that the technology developers have been diligent and thorough in their examination of the CTs, and that the technologies themselves

19Incremental increases in operational capability should be developed based on mature technology and delivered to the user in a useful grouping.

Relationship of TRAs to Program Management and Oversight

Page 19 GAO-20-48G TRA Guide have matured at a pace commensurate with the acquisition phases of the program. In this case, governance requirements might be met by validating a program’s existing body of TRA knowledge rather than by conducting a new assessment.

The TRA process and the content of an assessment can be tailored, depending on the purpose and audience for which it is conducted. While the focus of this Guide and the best practices it describes is on how to conduct high-quality TRAs throughout the life-cycle, the expert community has recognized that tailored self-assessments are useful to help the narrower audience make decisions about the day-to-day management of technology development efforts.

One such example of a tailored approach is through project self-assessments or knowledge-building TRAs, as part of peer reviews. These knowledge-building TRAs are conducted for the program manager, technology developer, or systems engineer for specific purposes, such as:

• learning about specific aspects of technology development (e.g., identifying gaps in maturity or specific areas that may be challenging);

• calculating progress toward achieving technical performance goals for a specific technology or group of technologies;

• identifying potential concerns and risks;

• gathering evidence to continue development efforts or initiate steps toward using an alternative or backup technology;

• demonstrating and maturing the performance of fieldable technologies or prototypes20;

• understanding the transition risks when maturing technologies;

• determining whether technologies are ready to transition to new or existing acquisition programs or larger systems; and

20A prototype is a physical or virtual model used to evaluate the technical or manufacturing feasibility or utility of a particular technology or process, concept, end item, or system.

Prototyping is used in an acquisition program as a tool for risk reduction, technology maturation, identifying and resolving integration risks, controlling manufacturing and sustainability risks, requirements development, and minimizing risks of cost growth due to unknowns in design, assembly and integration.

Ways to Tailor TRAs for Different Purposes

Page 20 GAO-20-48G TRA Guide

• deciding whether CTs are ready for a TRA to be conducted for governance bodies at an upcoming decision point.

In cases where organizations seek to accelerate technology development through shorter life-cycles as a streamlined acquisition approach, the less resource intensive knowledge-building TRA may be the practical choice.

However, tailoring the TRA process may not be appropriate in other situations. In the case of transition to product development, a decision maker would want to ensure that TRA best practices have been followed before making a major commitment of resources.

Organizations have learned that more frequent, regular evaluations on the maturity of CTs are a best practice. During the 1990s when DOD and NASA were conducting TRAs, governance bodies used the TRA reports at major decision points to determine whether programs that depend on CTs were ready to move to the next acquisition phase. Organizations have since expanded the use of TRAs and understand that conducting knowledge-building TRAs in the periods between decision points can put technology developers, program managers, and systems engineers in a better position to gauge progress, monitor and manage technology maturity, and identify and manage risks earlier.

Multiyear acquisition projects that involve the development of highly complex technologies and systems over long periods likely require multiple TRAs as part of an overall system development approach. In such cases, a best practice is to develop a technology maturity assessment strategy that involves identifying all the TRAs to be conducted throughout the acquisition of the system. The technology maturity assessment strategy lays out the broad goals and purposes of the TRAs, including when they may be conducted in the acquisition or project life-cycle, and how many, at a minimum, will be undertaken. The maturity assessment strategy can also include when knowledge-building TRAs should be conducted to serve as waypoints to gauge progress toward achieving specific program goals.

The technology maturity assessment strategy should provide guidance on how stakeholders should achieve consensus on conducting the TRAs.

For example, in developing an overall technology maturity assessment strategy, stakeholders should agree on whether one or more formal TRAs will be required to support one or more key decision points or stage

More Frequent Evaluations of Technology Maturity

Technology Maturity Assessment Strategies

Page 21 GAO-20-48G TRA Guide gates. This strategy should be included in a program’s overall acquisition strategy or equivalent document.

Case study 1 shows how knowledge-building TRAs can inform acquisition strategies.

Case Study 1: Knowledge-Building TRAs Inform Acquisition Strategies, cited in

GAO-19-336SP

In 2019, the Navy was determining the acquisition strategy for the Next Generation Jammer Low-Band (NGJ-LB) program and analyzing potential solutions to meet its capability needs. The NGJ-LB is an external jamming pod system that will be fitted on EA-18G Growler aircraft in order to disrupt adversaries’ use of the electromagnetic spectrum for radar detection, among other purposes. The Navy is planning to execute the program as a middle tier acquisition.