Ref (6) - TRA-Guide-Feb2025.v2-Cleared.pdf

Technology Readiness Assessment Guidebook

February 2025

Office of Systems Engineering and Architecture

Office of the Under Secretary of Defense for Research and Engineering

Washington, D.C.

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

Technology Readiness Assessment Guidebook

Office of Systems Engineering and Architecture Office of the Under Secretary of Defense for Research and Engineering 3030 Defense Pentagon Washington, DC 20301 osd-sea@mail.mil https://www.cto.mil/sea/

Distribution Statement A. Approved for public release. Distribution is unlimited.

DOPSR Case # 25-T-1398.

TECHNOLOGY READINESS ASSESSMENT GUIDEBOOK, FEB 2025

i i i

Approved by Principal Deputy Executive Director for Systems Engineering and Architecture

Office of the Under Secretary of Defense for Research and Engineering

Technology Readiness Assessment Guidebook Change Record

Date Change Rationale

June 2023 Revision of 2011 and earlier editions

Incorporate GAO 2020 recommendations and recent DoD policy and guidance.

Preface

Section 1 - Introduction

Section 4 – TRA Scope, Initiation, and Execution

Section 2.6.6. - Hardware for Space Systems

Incorporate updates due to Section 4252 changes in FY 2025 NDAA (removed references to waivers and certifications).

Clarify and emphasize the independence and importance of conducting a TRA.

Clarify requirements for demonstration in a relevant environment.

iv

This page is intentionally blank.

Contents v

Contents

Preface .................................................................................................................................................. viii

Introduction

Benefit of Conducting TRAs

Statute and DoD Policy

TRAs in the Commercial Sector

GAO 2020 Guide

Critical Technology Elements and Technology Readiness Levels

Definition and Use of CTEs and TRLs

CTEs and Critical Program Information

TRL Concept for Hardware

TRL Concept for Software

Additional TRL Definitions

Assessing Hardware CTEs

Aircraft

Ground Vehicles

Missiles and Guided Weapons

Ships and Ship Systems

Hardware for IT Applications

Hardware for Space Systems

Assessing Software CTEs

Characteristics of a High-Quality TRA

Four Characteristics of a High-Quality TRA

Understanding TRA’s Limitations

Organizational Experience, Culture, and Bias Can Affect TRAs

TRAs Depend on the Quality and Availability of Credible Data

Human Systems Integration Considerations in TRA

TRA Scope, Initiation, and Execution

Key Players and Roles and Responsibilities

Five-Step Process for Conducting a High-Quality TRA

Step 1: Establish a TRA Plan and Select a TRA Team

Step 2: Identify Critical Technology Elements

Step 3: Assess Critical Technology Elements v i

Step 4: Prepare the TRA Report (including USD(R&E) Review and Evaluation)

Step 5: Use the TRA Report Findings

Options for Addressing Immature Critical Technology Elements

Independent Technical Risk Assessment Considerations

Technology Maturation Plan

Purpose of the TMP

Five Steps for Preparing the TMP

Sections of a TMP

TRAs for the Adaptive Acquisition Framework Pathways

Overview

TRAs for Major Capability Acquisition

Milestone B TRA

Milestone C TRA

Other Types of Readiness Levels

Manufacturing Readiness Levels

Manufacturing Readiness Assessments

DoD MRL Deskbook

DoD MRL Descriptions

Integration Readiness Levels

System Readiness Levels

Sustainment Maturity Levels

Appendix A: Guidance and Best Practices for Identifying Critical Technology Elements

Appendix B: Suggested TRA Outline

Appendix C: Technology Maturation Plan Template

Glossary

Acronyms

References

Figures

Figure 4-1. Five Steps for Conducting a High-Quality TRA

Figure 4-2. Identifying Critical Technology Elements

Figure 4-3. Four Steps for Identifying Critical Technology Elements v i i

Figure 4-4. Assessing Critical Technology Elements

Figure 4-5. Preparing and Submitting a TRA Report

Figure 4-6. Five Steps to Prepare a Technology Maturation Plan

Figure 5-1. Adaptive Acquisition Framework

Figure 6-1. Integration Readiness Levels

Figure 6-2. System Readiness Levels

Figure 6-3. Sustainment Maturity Levels

Figure B-1. TRA Report Template

Figure C-1. TMP Template

Tables

Table 2-1. DoD Hardware TRL Definitions, Descriptions, and Supporting Information

Table 2-2. DoD Software TRL Definitions, Descriptions, and Supporting Information

Table 2-3. Additional Definitions of TRL Descriptive Terms

Table 3-1. Characteristics of a High-Quality TRA

Table 4-1. Questions for Determining Initial CTEs

Table 4-2. Steps in Preparing and Submitting a TRA Report

Table 5-1. TRA Guidance for AAF Pathways

Table 6-1. DoD Manufacturing Readiness Levels

Preface v i i i

Preface

The Department of Defense (DoD) relies on Technology Readiness Assessments (TRAs) to evaluate the technical maturity of critical technologies that enable proposed systems. A TRA provides a comprehensive evaluation of the extent to which these technologies have been successfully demonstrated and applied in relevant mission and operating environments, which helps programs determine the likelihood of successful integration of these technologies into the proposed system. Assessing the readiness of these enabling technologies provides decision makers with additional insight about the feasibility, risk, and potential return on investment of proposed systems, ultimately supporting more effective acquisition outcomes.

A TRA involves a fundamental metric, the Technology Readiness Level (TRL), first developed at the National Aeronautics and Space Administration (NASA) in the 1970s.

In 1999, the Government Accountability Office (GAO) (then General Accounting Office) published an influential report, Best Practices: Better Management of Technology Can Improve Weapon System Outcomes, concluding that using immature technology increased program risk and recommending wider use of TRLs. The report illustrated that maturing new technologies before they were incorporated into a product was perhaps the most important determinant of the success of the eventual product. Incorporating immature technologies into products increased the likelihood of cost overruns and delays in product development.

DoD formally endorsed the use of TRLs in 2001. DoD produced a TRA Deskbook in 2003 and revised the guidance in 2005, 2009, and 2011. This guidebook incorporates and supersedes TRA 2009 and TRA 2011, and it incorporates recommendations from the January 2020 GAO Technology Readiness Assessment Guide–Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects (GAO 2020), which describes characteristics and best practices of high-quality TRAs. This TRA Guidebook also discusses TRAs from the perspective of the DoD Adaptive Acquisition Framework pathways introduced in 2020 and discusses additional measures, such as Manufacturing Readiness Levels and System Readiness Levels, that DoD programs use.

For DoD programs, the TRA output is a key component of a comprehensive assessment of technical risks. The TRA includes an assessment of whether the system’s critical technologies have been demonstrated in a relevant environment, to inform Milestone B approval decisions in accordance with 10 USC 4252, “Major Defense Acquisition Programs: Factors to be Considered before Milestone B Approval.”

The Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)) prepared this guide in cooperation with defense subject matter experts (SMEs). OUSD(R&E) will provide periodic updates to incorporate comments and new information.

1. Introduct ion

Introduction This guidebook is intended to assist Department of Defense (DoD) programs to initiate, organize, and conduct high-quality Technology Readiness Assessments (TRAs) on acquisition programs. It supersedes DoD TRA guidance published in 2009 and 2011 and incorporates recommendations from the 2020 Government Accountability Office (GAO) Technology Readiness Assessment Guide–Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects (GAO 2020).

TRAs are not unique to DoD as other Government agencies and commercial developers conduct TRAs. This guide draws on relevant sources and focuses on guidance applicable to DoD acquisition programs. Whereas DoD policy is mandatory, this guidance is not mandatory but offers recommendations and best practices. The guidebook may refer to mandatory statute and policy for information.

The following introductory paragraphs provide background on the benefit of TRAs, the development of the GAO 2020 guide, and other sources of education, policy, and guidance.

Section 2 discusses the concepts of CTEs and Technology Readiness Levels (TRLs) essential to TRAs. Sections 3 and 4 highlight detailed recommendations from GAO 2020. Section 5 discusses TRAs in relation to the DoD acquisition pathways and phases outlined in DoD Instruction (DoDI) 5000.02, “Operation of the Adaptive Acquisition Framework (AAF).”

Section 6 discusses additional measures applicable to DoD Major Defense Acquisition Programs (MDAPs). The appendices include practices for identifying CTEs, a suggested TRA outline, and other supplementary information.

Benefit of Conducting TRAs

TRAs help programs make decisions to safeguard technical development from undue risk.

They provide PMs and program leadership with objective benchmarks for assessing the current maturity of critical technologies, and they inform decisions regarding allocation of resources and schedule for technology maturation activities. The outputs of the TRA, whether as a separate report or in the context of an Integrated Technical Risk Assessment (ITRA), support informed trades among cost, schedule, and risk.

TRAs provide a standard framework for applying measures and methods that identify potential technical risks. The program can respond to these identified risks by preparing a Technology Maturation Plan (TMP), which outlines the steps and level of effort required to mature the identified risky (immature) technologies. TMPs are discussed in Section 4.5.

Statute and DoD Policy

The following statute and DoD policies state requirements for TRAs:

• 10 USC 42521 requires that Milestone B decisions for MDAPs consider the maturity of critical technologies:

A major defense acquisition program or major subprogram may not receive Milestone B approval until the milestone decision authority confirms the following factors were considered in the decision to grant Milestone B approval: […]

(2) The technology in the program or subprogram has been demonstrated in a relevant environment.

• DoDI 5000.88, “Engineering of Defense Systems,” establishes policy and assigns responsibilities in accordance with DoD Directive (DoDD) 5000.01, “The Defense Acquisition System.” Programs already conducting an ITRA should follow TRA guidance to assess technologies but are not required to submit a separate TRA report.

DoDI 5000.88 states for programs for which an ITRA is conducted, a TRA report is not required. Programs will continue to assess and document the technology maturity of all critical technologies2 consistent with the technology readiness assessment guidance. ITRA teams may leverage technology maturation activities and receive access to results in order to perform independent technical reviews and assessments.

DoDI 5000.88 also requires a Systems Engineering Plan (SEP) for all MDAPs and Acquisition Category (ACAT) II and III programs and, for MDAPS, requires the SEP to include a plan to assess and document the technology maturity of all potential critical technologies. All other programs should also include this plan. Regardless of whether a program conducts an ITRA, all programs will continue to assess and document the technology maturity of all critical technologies throughout the development life cycle.

• DoDI 5000.86, “Acquisition Intelligence,” establishes policy and assigns responsibilities in accordance with DoDD 5000.01. Pursuant to this issuance, the Under Secretary of Defense for Research and Engineering (USD(R&E)) “coordinates and provides acquisition intelligence considerations for use in DoD Component and USD(R&E) independent technology readiness assessments.”

1 10 USC 4252 replaced 10 USC 2366b.

2 This guide uses the term “critical technology element (CTE)” in place of “critical technology (CT).” The term CT may appear in examples and related sources and is equivalent in meaning.

TRAs in the Commercial Sector

Commercial organizations also use TRAs to evaluate internal investments and research and development efforts that could be used on future government contracts. Examples of ways in which commercial organizations have developed processes to follow DoD’s TRA steps include the following:

• Identifying potential systems and programs as likely recipients of the technology.

• Using the research team to perform the TRA, supplemented when necessary by internal technology readiness experts.

• Having SMEs and business leaders review assessments for accuracy and to ensure the technology is progressing adequately.

• Relying on mechanisms to change the research plan to accelerate, slow down, or retire the development based upon the technical readiness assessment.

• Ensuring the assessment is objective, particularly with regard to demonstration environments, as system requirements evolve.

GAO 2020 Guide

GAO 2020 is a current, thoroughly researched and compiled guide for conducting a TRA.

While the GAO 2020 recommendations are not specific to one type of organization or agency, the GAO 2020 development involved a rigorous process that considered a variety of practices by different types of organizations (including DoD) that were well-researched and vetted by specialists and experts. For this reason, DoD has used GAO 2020 as a resource for compiling recommendations in this TRA Guidebook.

To document generally accepted best practices, GAO worked with practitioners and technical experts from across the Federal Government including DoD, commercial industry, nonprofit organizations, and academia. Among other agencies, GAO studied DoD’s TRA practices, case studies, and policies. From 2013 to 2019, GAO conducted meetings, focus groups, and interviews with more than 180 experts to collect information and elicit feedback on drafts of the guide. To reflect a range of expertise and viewpoints, GAO consulted with specialists from science and technology (S&T), systems engineering, nuclear engineering, software and computer sciences, risk management, acquisition policy, and program management.

GAO released a public draft (GAO-16-410G) in August 2016 for a 12-month comment period. From August 2017 to June 2019, representatives from several mission teams adjudicated more than 400 comments. The panel vetted each comment and placed it in one of two categories: (1) actionable, meaning the comment could be further adjudicated; or (2) not actionable because the comment did not align with the broader opinion of the specialist community, was outside the guide’s scope, was factually erroneous, or had no basis for specific action. GAO documented the adjudication in the GAO 2020 guide.

2. CTEs and TRLs

Critical Technology Elements and Technology Readiness Levels

TRAs involve two essential concepts, critical technology elements (CTEs) and Technology Readiness Levels (TRLs).

Definition and Use of CTEs and TRLs

A CTE is a new or novel technology on which a program or platform depends to successfully develop a system or to meet an operational threshold. A CTE may be hardware, software, or a process critical to the performance of a larger system or to the fulfillment of a key objective, such as a cyber-related capability.

OUSD(R&E) defines a technology element as “critical” if the system being acquired depends on this technology element to meet operational requirements and if the technology element or its application is either new or novel or in an area that poses major technological risk during detailed design or demonstration.

Programs should identify and document CTEs as early as possible, document the process for choosing the CTEs in the Acquisition Strategy, and, for MDAPs, assess the maturity before Milestone A and in the Analysis of Alternatives.

During a TRA, evaluators examine program concepts, technology requirements, and demonstrated technical capabilities to determine each CTE’s maturity. They assign each CTE a TRL between 1 and 9, with 9 representing the most mature. TRLs are not a measure of design validity, but they indicate the specific CTE’s level of maturity at the time it is measured and therefore the CTE’s relative readiness to be incorporated into the larger system.

CTEs and Critical Program Information

CTEs may often overlap with critical program information (CPI). The emphasis of CTEs is to identify technologies essential to the system design that need to be matured, whereas the emphasis on CPI is to identify potential vulnerabilities and security needs in operations. TRA teams should review the Technology and Program Protection Guidebook to see if any technologies have been identified as CPI. Any CPI technologies would be strong candidates to also be CTEs whose maturity would need to be assessed.

TRL Concept for Hardware

Many TRAs evaluate hardware CTEs that are being developed for weapon systems, communications systems, soldier systems, and so forth. Table 2-1 shows the TRLs DoD uses to assess hardware. It also lists typical documentation that evaluators should extract or reference to support a TRL assignment.

Table 2-1. DoD Hardware TRL Definitions, Descriptions, and Supporting Information

TRL Definition Description Supporting Information

1 Basic principles observed and reported.

Lowest level of technology readiness. Scientific research begins to be translated into applied research and development (R&D). Examples might include paper studies of a technology’s basic properties.

Published research that identifies the principles that underlie this technology. References to who, where, when.

2 Technology concept and/or application formulated.

Invention begins. Once basic principles are observed, practical applications can be invented. Applications are speculative, and there may be no proof or detailed analysis to support the assumptions.

Examples are limited to analytic studies.

Publications or other references that outline the application being considered and that provide analysis to support the concept.

3 Analytical and experimental critical function and/or characteristic proof of concept.

Active R&D is initiated. This includes analytical studies and laboratory studies to physically validate the analytical predictions of separate elements of the technology.

Examples include components that are not yet integrated or representative.

Results of laboratory tests performed to measure parameters of interest and comparison to analytical predictions for critical subsystems.

References to who, where, and when these tests and comparisons were performed.

4 Component and/or breadboard validation in a laboratory environment.

Basic technological components are integrated to establish that they will work together. This is relatively “low fidelity” compared with the eventual system. Examples include integration of “ad hoc” hardware in the laboratory.

System concepts that have been considered and the results of testing laboratory-scale breadboard(s).

References to who performed this work and when. Provides an estimate of how breadboard hardware and test results differ from the expected system goals.

5 Component and/or breadboard validation in a relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so they can be tested in a simulated environment.

Examples include “high-fidelity”

Results from testing a laboratory breadboard system are integrated with other supporting elements in a simulated operational environment.

How does the “relevant environment” differ from the expected operational environment? How do the test results compare with expectations? What problems, if any, were encountered?

Was the breadboard system refined

TRL Definition Description Supporting Information laboratory integration of components.

to more nearly match the expected system goals?

6 System/subsystem model or prototype demonstration in a relevant environment.

Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment.

Represents a major step up in a technology’s demonstrated readiness.

Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated operational environment.

Results from laboratory testing of a prototype system that is near the desired configuration in terms of performance, weight, and volume.

How did the test environment differ from the operational environment?

Who performed the tests? How did the test compare with expectations?

What problems, if any, were encountered? What are/were the plans, options, or actions to resolve problems before moving to the next level?

7 System prototype demonstration in an operational environment.

Prototype near or at planned operational system. Represents a major step up from TRL 6 by requiring demonstration of an actual system prototype in an operational environment (e.g., in an aircraft, in a vehicle, or in space).

Results from testing a prototype system in an operational environment.

Who performed the tests? How did the test compare with expectations?

What problems, if any, were encountered? What are/were the plans, options, or actions to resolve problems before moving to the next level?

8 Actual system completed and qualified through test and demonstration.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications.

Results of testing the system in its final configuration under the expected range of environmental conditions in which it will be expected to operate.

Assessment of whether it will meet its operational requirements. What problems, if any, were encountered?

What are/were the plans, options, or actions to resolve problems before finalizing the design?

9 Actual system proven through successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation (OT&E). Examples include using the system under operational mission conditions.

OT&E reports.

TRL Concept for Software

If the program develops the software and the software is in itself a CTE, it should appear as a software CTE. The hardware technology classification may include software that executes on the hardware if (1) the software is not being developed or modified as part of the acquisition, or (2) the software is not the reason for placing the element on the CTE list.

Table 2-2 shows the TRLs DoD uses to assess software. Although the TRL definitions are similar to those for hardware, the examples and supporting information to support the assessment differ. During planning, assessment teams may add supporting information items that are appropriate for the technology under assessment.

Table 2-2. DoD Software TRL Definitions, Descriptions, and Supporting Information

TRL Definition Description Supporting Information

1 Basic principles observed and reported.

Lowest level of software technology readiness. A new software domain is being investigated by the basic research community. This level extends to the development of basic use, basic properties of software architecture, mathematical formulations, and general algorithms.

Basic research activities, research articles, peer-reviewed white papers, point papers, early lab model of basic concept may be useful for substantiating the TRL.

2 Technology concept and/or application formulated.

Once basic principles are observed, practical applications can be invented.

Applications are speculative, and there may be no proof or detailed analysis to support the assumptions. Examples are limited to analytic studies using synthetic data.

Applied research activities, analytic studies, small code units, and papers comparing competing technologies.

3 Analytical and experimental critical function and/or characteristic proof of concept.

Active R&D is initiated. The level at which scientific feasibility is demonstrated through analytical and laboratory studies. This level extends to the development of limited functionality environments to validate critical properties including cybersecurity and analytical predictions using non-integrated software components and partially representative data.

Algorithms run on a surrogate processor in a laboratory environment, instrumented components operating in a laboratory environment, laboratory results showing validation of critical properties.

TRL Definition Description Supporting Information

4 Module and/or subsystem validation in a laboratory environment (i.e., software prototype development environment).

Basic software components are integrated to establish that they will work together. They are relatively primitive with regard to efficiency and robustness compared with the eventual system. Architecture development initiated to include interoperability, reliability, maintainability, extensibility, scalability, and security issues. Emulation with current/legacy elements as appropriate. Prototypes developed to demonstrate different aspects of eventual system.

Advanced technology development, stand-alone prototype solving a synthetic full-scale problem, or stand-alone prototype processing fully representative data sets.

5 Module and/or subsystem validation in a relevant environment.

Level at which software technology is ready to start integration with existing systems. The prototype implementations conform to target environment/interfaces.

Experiments with realistic problems. Simulated interfaces to existing systems. System software architecture established.

Algorithms run on a processor(s) with characteristics expected in the operational environment.

System architecture diagram around technology element with critical performance requirements defined.

Processor selection analysis, Simulation/Stimulation (Sim/Stim) Laboratory buildup plan. Software placed under configuration management. Commercial-of-the-shelf/ government-off-the-shelf (COTS/GOTS) components in the system software architecture are identified.

6 Module and/or subsystem validation in a relevant end-to-end environment.

Level at which the engineering feasibility of a software technology is demonstrated.

This level extends to laboratory prototype implementations on full-scale realistic problems in which the software technology is partially integrated with existing hardware/software systems. Cybersecurity verification should be included in the testing.

Results from laboratory testing of a prototype package that is near the desired configuration in terms of performance, including physical, logical, data, and security interfaces.

Comparisons between tested environment and operational environment analytically understood.

Analysis and test measurements quantifying contribution to system-wide requirements such as throughput, scalability, and reliability.

Analysis of human-computer (user environment) begun.

TRL Definition Description Supporting Information

7 System prototype demonstration in an operational, high-fidelity environment.

Level at which the program feasibility of a software technology is demonstrated.

This level extends to operational environment prototype implementations, where critical technical risk functionality is available for demonstration and a test in which the software technology is well integrated with operational hardware/software systems.

Critical technological properties, including cybersecurity, are measured against requirements in an operational environment.

8 Actual system completed and mission qualified through test and demonstration in an operational environment.

Level at which a software technology is fully integrated with operational hardware and software systems.

Software development documentation is complete. All functionality and cybersecurity measures tested in simulated and operational scenarios.

Published documentation and product technology refresh build schedule.

Software resource reserve measured and tracked. All severity 1 and severity 2 defects are resolved/confirmed, and a reasonably low level of severity 3 defects remain open.

9 Actual system proven through successful mission-proven operational capabilities.

Level at which a software technology is readily repeatable and reusable. The software based on the technology is fully integrated with operational hardware/software systems. All software documentation verified. Successful operational experience. Sustaining software engineering support in place. Actual system.

Production configuration management reports. Defect resolution system and process is in place for deployed software to address defects discovered in production.

Additional TRL Definitions

Table 2-3 provides additional TRL definitions.

Table 2-3. Additional Definitions of TRL Descriptive Terms

Term Definition

Breadboard Integrated components that provide a representation of a system/subsystem and that can be used to determine concept feasibility and to develop technical data. Typically configured for laboratory use to demonstrate the technical principles of immediate interest. May resemble final system/subsystem in function only.

High Fidelity Addresses form, fit, and function. A high-fidelity laboratory environment would involve testing with equipment that can simulate and validate all system specifications within a laboratory setting. High fidelity models are accredited to represent the system for their defined purpose.

Low Fidelity A representative of the component or system that has limited ability to provide anything but first-order information about the end product.

Low-fidelity assessments are used to provide trend analysis.

Model A functional form of a system, generally reduced in scale, near or at operational specification. Models will be sufficiently hardened to allow demonstration of the technical and operational capabilities required of the final system.

Operational Environment Environment that addresses user operational requirements and specifications required of the final system to include platform/packaging.

Prototype A physical or virtual model used to evaluate the technical or manufacturing feasibility or military utility of a particular technology or process, concept, end item, or system.

Relevant Environment Testing environment that simulates both the most important and most stressing aspects of the operational environment.

Simulated Operational Environment

Either (1) a real environment that can simulate all the operational requirements and specifications required of the final system or (2) a simulated environment that allows for testing of a virtual prototype.

Used in either case to determine whether a developmental system meets the operational requirements and specifications of the final system.

Assessing Hardware CTEs

Applying the TRL definitions to assess the maturity of hardware technologies appears to be straightforward. For a particular technology, evaluators assign the level of technical readiness that best describes the accomplishments and evidence according to the TRL definitions. In practice, this approach is more difficult than it appears because the TRL definitions often fail to account for all situations.

TRL definitions describe characteristics such as the scale of application or the environment.

The scale of application ranges from device to component, subsystem, and system.

Environment includes the laboratory, mathematical models, physical simulations, field tests, and operational use. A TRA should present increasingly representative tests to demonstrate the technologies in relation to the characteristics in the definition.

Some TRL definitions mention characteristics directly; others do not. When a technology fails to match the literal definition during the assessment, the evaluator should make a judgment about the relevance of what the technology has accomplished and decide whether the accomplishment equates with the TRL definition.

Environment is perhaps the most difficult characteristic to interpret. Both TRL 5 and TRL 6 depend on performing in a relevant environment. Although the details of a relevant environment depend on the intended use of a technology, the TRL criteria are as follows:

• A relevant environment is a set of stressing conditions, representative of the full spectrum of intended operational employments, which are applied to a CTE as part of a component (TRL 5) or system/subsystem (TRL 6) to identify whether any design changes to support the required (threshold) functionality are needed.

To assess whether a technology can perform in the full range of required operational employments, it is not enough to conduct only one or a few demonstrations under only the most favorable conditions, or in a few useful environments. An effective assessment requires a body of data or accepted theory to support, with confidence, that the technology will be effective in the full spectrum of employments.

Demonstration of a CTE as part of a component or system /subsystem in a relevant environment requires successful trial testing that either:

• Validates that the CTE satisfies the required functionality across the full spectrum of intended operational employments or

• Validates that the CTE satisfies the functional need for important, intended operational employment(s) and then uses accepted/approved analytical techniques to extend confidence in supporting the required functionality across the required, intended operational employments.

As an example of a demonstration in a relevant environment, a CTE as part of a system or subsystem model or prototype might be tested in a high-fidelity laboratory environment or in a simulated operational environment.

The progression from TRL 6 to TRL 7 involves a shift in the scale of the demonstration of the technology. TRL 6 requires the CTE to be embedded or installed in a representative model or prototype. TRL 7 requires the CTE to be embedded or installed in a prototype of the planned operational system in the operational environment. At Milestone C, hardware (and software) CTEs should be able to achieve TRL 7.

While the details of an operational environment depend on the intended use of a CTE, following is a general description of an operational environment, and what it demonstrates:

• An operational environment is a set of conditions, representative of the full spectrum of employments, which are applied to a CTE, prototype (TRL 7) or actual system (TRL 8) to identify whether any previously unknown or undiscovered design problems might impact required functionality.

Demonstration of a CTE in an operational environment requires successful testing that either:

1. Validates that the CTE satisfies the required functionality across the full spectrum of operational employments or

2. Validates that the CTE satisfies the functional need for important, operational employment(s) and then uses accepted analytical techniques to extend confidence in supporting the required functionality over all the required operational employments.

As an example of a demonstration in an operational environment, a CTE as part of a system might be installed in an aircraft or vehicle, which is then tested in the operational conditions of a test bed or at a test range facility.

Aircraft

Aircraft are likely to have CTEs in aerodynamic configuration and controls, airframe structure and aeroelasticity, flight control systems, and propulsion. In addition, rotary-wing aircraft have CTEs in power transfer, rotor hub, and blades. CTEs could also be factors in mission equipment, secondary power, environmental control, and other systems, depending on the aircraft’s missions. A variety of methods and facilities are used to demonstrate these different technologies.

TRAs for aerodynamic configuration and controls typically use demonstrations such as analysis, computational fluid dynamics (CFD) investigations, wind tunnel tests, and flight tests.3 When aerodynamic configurations are significantly different from those of existing aircraft, TRAs may use free-flight models (manned or unmanned). Similarly, TRA evaluators may use a variety of methods and facilities for airframe, flight control, and other aeronautical disciplines.

Ground Vehicles

Most new military ground vehicle concepts and systems involve CTEs. Combat and tactical vehicles must meet requirements to address new threats and new or extended performance needs of operational forces. Utility and general-purpose vehicles—many of which are adapted versions of commercial vehicles—also may be required to provide special performance characteristics that exploit new technologies or novel application of existing technologies.

The automotive features of any class of military vehicles are likely to exploit CTEs in propulsive power, drive trains, platform stability, suspension systems, and endurance.

Demonstrating the efficacy of CTEs requires various means of test, analysis, and verification.

In most cases, these tests and analyses are unique to the military environment.

The protection requirements of combat and tactical vehicles are unique for combat environments. CTEs should be anticipated in vehicle-integrated passive protection against diverse weapon and munitions threats. Similarly, as threats increase and become more sophisticated, CTEs may provide reactive (e.g., explosive armor) or active (e.g., detection and attack of threat munitions) features. Evaluators often judge the maturity of these technologies by building on existing analysis and test capabilities.

Missiles and Guided Weapons

The development program for a missile or other guided weapon differs from that of a “platform” vehicle, and the program for a solid propellant rocket differs from that of a liquid propellant rocket. Most military missiles involve structure, propulsion, guidance, flight control, and payload. Each of these systems comprises numerous elements that must function together to meet the objectives of the system, and any of these elements may depend upon CTEs. Although each of the technologies will be discussed independently with associated TRLs/technology, the development of the system and associated TRL must be done from a system perspective because of interdependencies.

To assess the maturity of these CTEs, issues that should be considered in performance demonstrations include how the test environments compare with the real environments and

3 Programs may test a variety of scale models in different wind tunnels to obtain data for different flight conditions and mission phases.

how the performance exposes what is required. Each of the missile components can be tested to high TRLs. The integrated ordnance payload is often evaluated with a sled test (penetration testing) or arena testing (fragmentation testing); both tests are often required to achieve TRL 5-6. The fuze system can only achieve TRL 6 via flight test demonstration due to the required inclusion of the safe and arm function.

Missile structural integrity and flight control are highly interdependent. Structural bending modes, placement of accelerometers, control system time constants, aerodynamic loads and control moments, and reaction controls must work together to achieve stable, controlled flight.

Structural rigidity and inertial properties can first be computed during computer-aided design (CAD) and then confirmed by ground tests. Engineers determine aerodynamic characteristics by analysis and wind tunnel tests. High-fidelity, 6-degree-of-freedom (DOF) simulations can represent the complete missile in its intended flight environment. Components tested in hardware-in-the-loop (HWIL) simulations can reasonably be considered TRL 4.

Assuming flight accelerations and vibrations are important to the functioning of a component, testing that component while on a surrogate missile could support the component to achieve TRL 5. After the components have been integrated into a dynamically correct prototype missile and flown with preprogrammed maneuvers, the components are considered TRL 6 if the environment is relevant for those components.

Missile guidance systems can include a variety of sensor types. Several types of test environments are useful for particular types of sensors. These include anechoic chambers for radars and other radio frequency (RF) systems, terrain tables for visual and infrared (IR) detectors, towers overlooking tactical targets, captive carry-on aircraft and missiles, and free flight. The maturity associated with these sensors depends on the fidelity of the relevant features of the environment and the fidelity of the test article when compared with the final product. If a tower can provide the correct viewpoint and range to a target and if motion is not important, perhaps a tower test of a prototype sensor can be adequate to assess TRL 5.

If motion is important, a captive carry test might be necessary to achieve TRL 5. As motion is almost always important to missile guidance systems, captive carry for TRL 5 and demonstration on a prototype or surrogate missile for TRL 6 are the norms.

For liquid fuel rockets, factors to consider include movement and metering of fuel and oxidizer, throttling or multiple starts, and cooling of the nozzle with fuel. Relevant conditions may include very low ambient pressures and longitudinal and lateral accelerations that can be achieved only in flight.

Air-breathing rockets must establish inlet performance and flammability limits over a wide range of Mach numbers and ambient pressures. Demonstrations are to include connected tests (inlet connected to an air source) to merit TRL 4 and free-flow tests including inlet, captive carry, and free-flight tests to merit TRLs 5 and 6, respectively, if the test articles of the free-flight tests are functionally representative prototypes.

Ships and Ship Systems

Ships are likely to have CTEs in hydrodynamic hull form, materials and structures, propulsion, drag reduction, and motion controls. Ship systems, such as sensors (radar/sonar), weapons (torpedoes/missiles), hotel (waste disposal/desalination/material movement), and aircraft interfaces (elevators), will require some additional CTEs. Ships also have CTEs related to survivability, such as signatures, countermeasures, and intact and damaged stability.

A variety of methods and facilities are used to demonstrate these different technologies.

Ships are usually large and complex; therefore, prototyping of a complete system, such as a new hull form, is expensive and time consuming. The types of demonstrations used normally for ship hull-form technologies include analysis, CFD investigations, towing tank model scale tests, and land-based subsystem tests. For ship configurations that represent large departures from the existing base of knowledge, full-scale prototypes are usually needed.

Similarly, a variety of methods and facilities are used for structures and materials, motion control, and other ship-related disciplines. Torpedo development would follow an approach similar to that of a missile system. The technologies of active drag reduction are treated similarly to those of a propulsion subsystem, such as a new propeller, and would follow the propulsion approach. Passive drag reduction systems, such as hull shaping, are treated similarly to the hull form development approach.

Hardware for IT Applications

As an example of hardware for IT, Microelectromechanical Systems (MEMS) technology delivers computer displays in creative designs, such as a high-tech monocle or a helmet-mounted display (HMD) for operational environments that would not be suitable for laptops or other conventional displays.4 Infantry soldiers are expected to carry the equivalent of a laptop computer with them. They require an ergonomic fit and form for their environment, which may include active combat, harsh weather, and traveling long distances on foot.

The military has tested some early prototypes of MEMS. Stryker vehicle commanders have the option to view the onboard battlefield computer with an HMD. Helicopter pilots have a prototype system with a digital display of the battlespace to increase situational awareness.

4 MEMS projects images directly onto the retina. Essentially all light generated enters the eye, so the MEMS device is energy efficient and reduces demand on the local power supply. A system using MEMS is expected to be more rugged than a conventional system; for example, the display is readable in daylight and provides higher resolution.

These tests provided a technical readiness of TRL 6. Achieving a TRL 7 or higher would require the military to test the display in the infantry soldier’s operational environment.

Hardware for Space Systems

Demonstration in a relevant environment is particularly challenging for space systems because of the unique features of the space environment. Ground-based environmental qualification testing (e.g., vacuum, temperature extremes, solar radiation, micrometeorite impact, etc.) for space system hardware could constitute “demonstration in a relevant environment” as needed to substantiate TRL 6, so long as the required stressing environmental factors are sufficiently simulated. SMEs should be involved in identifying the key stressing factors and thresholds.

Assessing Software CTEs

Software and process technologies have grown increasingly critical to defense systems, and it is important to distinguish them from hardware. In recent years, Services, such as the Navy, have made extra efforts to decouple hardware from software within traditional combat systems. Doing this helps to separate the hardware and software development processes so that software is not tied to slower developing hardware processes (Eckstein 2023). This move is an important nod to the importance of distinguishing hardware and software CTEs as well as the process for assessing them.

The definitions of TRLs applied to software involve several facets. At the application level are values of device, component, subsystem, and system for hardware and algorithms, software components, software programs, and software packages. Another facet includes the environment (or application)—integration issues, laboratory user environment issues, logical relationship issues, data environment issues, security environment issues, and possibly interface issues. Other system-wide facets include obsolescence, scalability, and throughput and are usually expressed in terms of system-wide requirements, but the hardware components often contribute to meeting these requirements.

The combination of these facets determines any TRL. When the accomplishment and the definition do not match, the assessor must use judgment regarding the relevance of what has been accomplished and ask whether the accomplishment is equivalent to the TRL definition.

In assessing software’s technical readiness, the terms relevant environment and operational environment indicate a significant progression in accomplishing the demonstration and include cyber-congested and cyber-contested environments. Claiming technical readiness in a relevant environment (TRL 5 or higher) requires a detailed architecture that exposes all components and elements affecting the operation of the critical software element. Claiming technical readiness in an end-to-end relevant environment (TRL 6 or higher) requires evidence of performance on full-scale, realistic problems. Claiming technical readiness in an operational environment (TRL 7 or higher) requires evidence of the acceptable performance of the software element under operational factors such as system loading, user interaction, security, and realistic communications environment (e.g., bandwidth, latency, jitter).

3. Character is t ics of a High-Qual i ty TRA

Characteristics of a High-Quality TRA This section outlines the characteristics that GAO 2020 offered as indicative of a high-quality TRA and the best practices associated with those characteristics. This section also discusses a TRA’s limitations. References to “critical technology element (CTE)” and “critical technology (CT)” are equivalent.

Four Characteristics of a High-Quality TRA

In its research and discussions with experts from government, industry, non-profits, and academia, the GAO found that high-quality TRAs have four common characteristics:

credibility, objectivity, reliability, and usefulness. Table 3-1 (adapted from GAO 2020) summarizes these characteristics and the best practices that support them.

Table 3-1. Characteristics of a High-Quality TRA

Characteristic Description GAO’s Identified Best Practices to Achieve the Characteristic

Credible Conducted with an understanding of the requirements that guide development of the critical technology elements (CTEs) and system, the relevant or operational environment in which it will function, and its integration or interaction with other technologies

• Is comprehensive and includes all of the key information identified in the Technology Readiness Assessment (TRA) plan

• Identifies and has the expertise needed to conduct the assessment

• Considers the newness or novelty of technologies and how they plan to be used as basis or selecting CTEs

• Considers the operational performance environment and potential cost and schedule rivers as a basis for selecting CTEs

• Considers the relevant environment as a basis for selecting CTEs

• Considers the potential adverse interaction with other systems as basis for selecting CTEs

• Selects CTEs during early development

• Selects CTEs at a testable level

Objective Based on objective, relevant, and trustworthy data, analysis, and information; and the judgements, decisions, and actions for planning and executing the assessment are free from…