A07-W81XWH-18-R-0058_Attch-5-Technology_Readiness_Assessment_Deskbook.pdf

DEPARTMENT OF DEFENSE

Technology Readiness Assessment (TRA) Deskbook

July 2009

Prepared by the Director, Research Directorate (DRD)

Office of the Director, Defense Research and Engineering (DDR&E)

This version of the TRA Deskbook accounts for policy and guidance provided by Directive DoDD 5000.01, of May 12, 2003 and certified current as of November 20, 2007;

Instruction DoDI 5000.02, dated December 2, 2008; and the online Defense Acquisition Guidebook.

iii

Contents

Executive Summary .................................................................................................. ES-1

1. Introduction ...................................................................................................... 1-1

1.1 Technology Readiness Assessment (TRA) Definition ............................. 1-1

1.2 TRA Authority .......................................................................................... 1-2

1.3 TRA Importance ....................................................................................... 1-3

1.3.1 Milestone B TRA .......................................................................... 1-3

1.3.2 Milestone C TRA .......................................................................... 1-5

1.4 Purpose and Organization of This Document ........................................... 1-5

2. Initiating and Conducting TRAs .................................................................... 2-1

2.1 Key Players and the TRA Timeline .......................................................... 2-1

2.2 Roles and Responsibilities ........................................................................ 2-1

3. Evolution of Knowledge on Technology Maturity ........................................ 3-1

3.1 Early Evaluations of Technology Maturity ............................................... 3-1

3.2 Summary ................................................................................................... 3-3

List of Acronyms .................................................................................................... ACR-1

Appendixes

A Submitting a Technology Readiness Assessment (TRA) ...................................... A-1

B. Guidance and Best Practices for Identifying Critical Technology Elements (CTEs) .................................................................................................... B-1

C. Guidance and Best Practices for Assessing Technology Maturity ......................... C-1

D. Amplifying Technology Readiness Assessment (TRA) Guidance for Ships ........ D-1

E. Biomedical Technology Readiness Levels (TRLs) ................................................ E-1

F Technology Maturity Policy ................................................................................... F-1

G. The Technology Readiness Assessment (TRA) Process ....................................... G-1

H. Easy-Reference Displays of the Hardware/Software TRLs and Additional TRL Definitions ................................................................................... H-1 iv

Figure

2-1. Representative Schedule for TRA Activities ...................................................... 2-2

Table

3-1. Basis of Technology Maturity Assessments Throughout Acquisition ................ 3-3

ES-1

Executive Summary

A Technology Readiness Assessment (TRA) is a formal, systematic, metrics-based process and accompanying report that assesses the maturity of critical hardware and software technologies to be used in systems. It is conducted by an Independent

Review Team (IRT) of subject matter experts (SMEs).

This formal TRA complements—but does not in any way preclude—the program manager’s (PM’s) responsibility to pursue all the risk reduction efforts needed to ensure that adequate technological maturity is reached before Milestone B approval is sought.

As an activity separate from the formal TRA, an early evaluation of technology maturity conducted shortly before Milestone A should be used to support the planning of these risk reduction efforts.

All Department of Defense (DoD) acquisition programs must have a formal TRA at Milestone B and at Milestone C of the Defense Acquisition System. For ships, a pre-liminary assessment is required at program initiation. TRAs for Acquisition Category

(ACAT) ID and IAM programs must be submitted to the Director, Research Directorate

(DRD) in the office of the Director of Defense Research and Engineering (DDR&E).

Title 10 United States Code (U.S.C.) Section 2366b requires, in part, that the

Milestone Decision Authority (MDA) certify that the technology being used in Major

Defense Acquisition Programs (MDAPs), including space MDAPs, has been demon-strated in a relevant environment before Milestone B approval. The Under Secretary of

Defense for Acquisition, Technology, and Logistics (USD(AT&L)) relies on the DDR&E to provide technical advice to support this certification. In addition, while 10 U.S.C.

2366b is only applicable to MDAPs, the DoD is also requiring Major Automated Infor-mation System (MAIS) acquisitions to meet the same technology maturity standard at

Milestone B. Consequently, the DDR&E is also providing technical advice to the MDA for MAIS acquisitions. The DDR&E is using the approved TRA process and report as the basis of that technical advice.

This document, the Technology Readiness Assessment (TRA) Deskbook, provides

DRD guidance for conducting TRAs. The body of this document is a concise description

ES-2

of suggested best practices, responsibilities, roles, and procedures for meeting the TRA requirements. The appendixes are designed to amplify the material in the main body.

ACAT ID and IAM programs are expected to follow these best practices as a condition for certification. The processes outlined should also be used for other MDAPs.

This Deskbook is intentionally generic and non-prescriptive. The Services and agencies, given their vast organizational structures, are encouraged to establish their own implementation guidance, approved and endorsed by the Component Science and Tech-nology (S&T) Executive. Procedures should be based upon the principles, guidance, and recommended best practices contained in this Deskbook.

1-1

Section 1.

Introduction

1.1 Technology Readiness Assessment (TRA) Definition

A TRA is a formal, systematic, metrics-based process and accompanying report1 that assesses the maturity of technologies called Critical Technology Elements (CTEs)2 to be used in systems. CTEs can be hardware or software. The definition of a CTE is as follows:

A technology element is “critical” if the system being acquired depends on this technology element to meet operational requirements (within acceptable cost and schedule limits) and if the technology element or its application is either new or novel or in an area that poses major techno-logical risk during detailed design or demonstration.

This definition represents an expansion of previous definitions by adding the phrase “or in an area that poses major technological risk during detailed design or dem-onstration.” In the past, some confusion arose in determining whether a CTE is a “tech-nology” or solely a matter of “engineering.” The purpose of this new phrase is to be more encompassing. If the technology represents a major risk, it should be identified as a CTE so that the TRA will include technical information that can be used to mitigate the risk.

An Independent Review Team (IRT) of subject matter experts (SMEs) uses Tech-nology Readiness Levels (TRLs) as the metric to assess CTE maturity.3 The TRL scale ranges from one through nine. The definitions are as follows:

TRL 1: Basic principles observed and reported

TRL 2: Technology concept and/or application formulated

TRL 3: Analytical and experimental critical function and/or characteristic proof of concept

TRL 4: Component and/or breadboard validation in a laboratory environment

1 Appendix A contains an annotated outline of the TRA report.

2 Appendix B addresses the CTE identification process in more detail.

3 Appendix C discusses TRLs and CTE maturity assessments in more detail. Appendix D provides some amplifying guidance for ships. Appendix E addresses biomedical TRLs. Appendix H (at the end of this document) is an easy-reference display of the hardware and software TRLs and additional definitions of TRL descriptive terms.

1-2

TRL 5: Component and/or breadboard validation in a relevant environment

TRL 6: System/subsystem model or prototype demonstration in a relevant environment

TRL 7: System prototype demonstration in an operational environment

TRL 8: Actual system completed and qualified through test and demonstra-tion

TRL 9: Actual system proven through successful mission operations.

CTE lists of varying provenance exist during the TRA. We reserve the term

“CTE” for the final list with the Director, Research Directorate (DRD) concurrence.

“Possible” CTEs are on the list prepared by the program manager (PM), “potential” CTEs are from pre-Materiel Solution Analysis (MSA) early evaluations of technology maturity, and “candidate” CTEs represent the IRT product for DRD coordination.

1.2 TRA Authority

The requirement to conduct a formal TRA is established by the following doc-uments:4,5

Department of Defense Directive (DoDD) 5000.01, The Defense Acquisition System, of May 12, 2003, and certified current as of November 20, 2007

Department of Defense Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System, dated December 2, 2008

Under Secretary of Defense for Acquisition, Technology, and Logistics USD(AT&L) Memorandum on Transition of the Defense Space Acquisition Board (DSAB) Into the Defense Acquisition Board and its interim guidance attachment, dated March 23, 2009

DoDD 5000.01 authorizes the publication of DoDI 5000.02. Together, these doc-uments provide management principles and mandatory policies and procedures for man-aging all acquisition programs. DoDI 5000.02 establishes a regulatory requirement for

TRAs. All Department of Defense (DoD) acquisition programs must prepare a TRA at

Milestone B and at Milestone C of the Defense Acquisition System. For ships, a

4 The 5000 series documents are available at https://akss.dau.mil/dapc/index.aspx. A working knowl-edge of the Defense Acquisition System is assumed in the main body of this document.

5 There is no such thing as an informal TRA. While many assessments of technology maturity will be conducted in the science and technology (S&T) environment and in the context of an acquisition program, the term “Technology Readiness Assessment” applies only to this regulatory requirement.

1-3 preliminary assessment is required at program initiation. TRAs for Acquisition Category

(ACAT) ID and IAM programs must be submitted to the DRD. The TRA processes pre-sented in this document should be adapted to other ACAT programs to fulfill regulatory and statutory requirements.

The TRA complements—but does not in any way preclude—the PM’s responsi-bility to pursue all risk reduction efforts needed to ensure that adequate technological maturity is reached before Milestone B approval is sought. As an activity separate from the formal TRA, an early evaluation of technology maturity conducted shortly before

Milestone A should be used to support the development of the Technology Development

Strategy (TDS).

1.3 TRA Importance

1.3.1 Milestone B TRA

Programs that enter the Engineering and Manufacturing Development (EMD) phase of the Defense Acquisition System and have immature technologies will incur cost growth and schedule slippage. Therefore, Title 10 United States Code (U.S.C.) Section

2366b requires, in part, that the Milestone Decision Authority (MDA) certify that the technology in Major Defense Acquisition Programs (MDAPs), including space MDAPS,6 has been demonstrated in a relevant environment (TRL 6) before Milestone B approval.

The law allows the MDA to waive the certification requirement (i.e., the technology in the program has been demonstrated in a relevant environment) if it determines that such a requirement would hinder the DoD’s ability to meet critical national security objectives.

As a matter of practice, such waivers will be granted only in extraordinary circum-stances.7 The Under Secretary of Defense for Acquisition, Technology, and Logistics

(USD(AT&L)) has directed that all MDAs—including the Component Acquisition

Executives (CAEs) and the Assistant Secretary of Defense for Networks and Information

Integration (ASD(NII))—for MDAPs will certify without delegation, as required by law.8

6 Statutory language refers to Key Decision Point (KDP) B for space programs. This terminology has been made obsolete by the aforementioned USD(AT&L) memorandum, dated March 23, 2009.

7 Whenever the MDA makes such a determination and authorizes such a waiver, the waiver and the reasons for the determination have to be submitted in writing to the Congressional defense committees within 30 days of waiver authorization.

8 Implementation of Section 2366a of Title 10, United States Code, as amended by the National Defense Authorization Act for FY 2008 (P.L. No. 110-181), USD(AT&L) Memorandum, February 25, 2008, as amended by Policy Update Due To Technical Change in Statute – Reference for Requirement for

1-4

The USD(AT&L) relies on the Director of Defense Research and Engineering

(DDR&E) to provide technical advice to support certification. In addition, while

10 U.S.C. 2366b is only binding for MDAPs, the DoD is also requiring Major Automated

Information System (MAIS) acquisitions to meet the same technology maturity standard at Milestone B. Consequently, the DDR&E is also providing technical advice to the

MDA for MAIS acquisitions. The DDR&E is using the approved TRA process and report as the basis of that technical advice.9 DoDI 5000.02 requires Request for Proposal (RFP) language that prevents the award of an EMD contract if it includes technologies that have not been demonstrated to be mature. As such, a generic TRA not based on the planned technical solution is not acceptable. The TRA must be based on the technologies in the system. This means that TRAs must be performed on all the competitors’ proposals in a source selection. Under the DDR&E, the DRD has primary responsibility for overseeing the TRA process and reviewing TRA reports.

PMs have found that the TRA assessment process is useful in managing technol-ogy maturity. The TRA process highlights critical technologies and other potential tech-nology risk areas that require the PM’s attention. The TRA can help identify immature and important components and track the maturity development of those components.

Some programs use TRAs as an important part of their risk assessment.10

For Information Technology (IT) systems, which rely heavily on off-the-shelf components, TRAs have increased management’s focus on finding CTEs that relate spe-cifically to IT issues (e.g., interfaces, throughput, scalability, external dependencies, inte-gration, and information assurance). Since many IT systems have experienced problems in these areas, the TRA has proven useful in understanding potential problems earlier in the process, when solution options are easier to adopt and less costly to implement.

1.3.2 Milestone C TRA

Milestone C marks approval to enter low rate initiation production (LRIP) for hardware systems and limited deployment in support of operational testing for MAIS programs or for software-intensive systems that have no production components. TRL 7 or higher is the expected state of technology maturity at Milestone C.

Milestone B Certification becomes Section 2366b vice 2366a, Director Acquisition Resources and Analysis Memorandum, November 21, 2008.

9 Appendix F provides more information on how the TRA supports certification.

10 Early evaluations of technology maturity also assist in risk reduction. See Section 3.1.

1-5

The Milestone C TRA is important for several reasons. It reflects the resolution of any technology deficiencies that arose during EMD. This TRA serves as a check that all

CTEs are maturing as planned. By Milestone C, all CTEs will have advanced and will continue to be matured through established Technology Maturation Plans (TMPs). Any new CTEs that have emerged should be identified, and their maturation plans should be reviewed.

For software, TRL 7 means that all source codes have been written and tested— not only as an independent module and/or component, but also as integrated into the whole system. The TRA at Milestone C is important for MAIS programs because it

Documents successful developmental test and evaluation (DT&E)

Examines plans for maintenance and upgrades to ensure that no new CTEs are involved

Determines whether algorithms will transfer successfully when host plat-forms are moved and full-scale applications are initiated in a real operational environment

Identifies where new Milestone B reviews are needed for future releases to initiate efforts to improve performance and determines the architectural changes necessary to support these future releases.

1.4 Purpose and Organization of This Document

This document, the Technology Readiness Assessment (TRA) Deskbook, provides

DRD guidance and best practices for conducting TRAs. ACAT ID and IAM programs are expected to follow the best practices as a condition for certification. Section 2 presents an overview of the process and summarizes the roles and responsibilities of the key players in the process.11 Section 3 describes other TRA activities in the context of an evolution of knowledge of technology maturity throughout acquisition. The appendixes are designed to amplify the material in the main body.

11 Appendix G contains a more chronological description of key player roles and responsibilities and highlights best practices.

2-1

Section 2.

Initiating and Conducting TRAs

2.1 Key Players and the TRA Timeline

Key players in the TRA process are as follows:

The PM, the Component S&T Executive, and the CAE are the principal stakeholders for the Component conducting the TRA.

The DRD has primary responsibility for reviewing and evaluating the TRA for the Office of the Secretary of Defense (OSD) for ACAT ID and IAM programs. The Component S&T Executive evaluates the TRA for ACAT ICs. The Component S&T Executive can delegate to the appro-priate MDAs for ACAT II and below. The DRD monitors the TRA process and reports to the DDR&E.

The IRT of SMEs is responsible for conducting the TRA itself.

Figure 2-1 shows a representative schedule of activities for a TRA. The “months” shown across the top of the figure represent the timeline before a milestone decision. The

TRA schedule will vary with the program’s acquisition strategy and should take into account any source selection or down-select activity. As a result, activity start points and duration may vary greatly. The time varies as a function of Component procedures.

ACAT ID, IC, and IAM programs typically take a full year or more. Smaller, less com-plex programs normally require less time.

2.2 Roles and Responsibilities

Key player roles and responsibilities are as follows:

The PM

– Plans and funds the program’s risk reduction activities to ensure that CTEs reach the appropriate maturity levels. For example, the CTEs must be TRL 6 at Milestone B.

– Informs the Component S&T Executive of the need to conduct a TRA.

– Funds the TRA evaluation for his program.

– Designates a responsible individual to organize all TRA activities.

2-2

Figure 2-1. Representative Schedule for TRA Activities

– Prepares a draft TRA schedule and incorporates the approved version in the program’s Integrated Master Plan (IMP) and Integrated Master Schedule (IMS).

– Suggests to the Component S&T Executive the subject matter expertise needed to perform the TRA.

– Familiarizes the IRT with the program.

– Identifies possible CTEs for consideration by the IRT.

– Provides evidence of CTE maturity to the IRT for assessment, including contractor data.

– Provides technical expertise to the IRT as needed.

– Drafts the section of the TRA report containing a brief description of the program (program/system overview, objectives, and descriptions).

The Component S&T Executive

– Directs the conduct of the TRA.

– Coordinates on the TRA schedule.

– Nominates SMEs for the IRT.

– – Only top experts who have demonstrated, current experience in relevant technical disciplines should be nominated. For a joint program, each Service/agency should have representation on the IRT. Overall, the IRT membership should be balanced among

2-3

Component, other government agency (e.g., National Aeronautics and Space Administration (NASA), National Institute of Stan-dards and Technology (NIST), or Department of Energy (DOE)), and non-government representatives (e.g., academia, Federally Funded Research and Development Centers (FFRDCs), or science boards)).

– – Members should be sufficiently independent of the developers (government or industry) so as to not be unduly influenced by their opinions or have any actual or perceived biases. An IRT member should not be directly working for or matrixed to the program to avoid being unduly influenced by the PM.

– Provides the DRD the credentials of all prospective IRT members and sufficient information to confirm their independence from the program.

– Trains IRT members on the TRA process.

– – Training should include an overview of the TRA process, criteria for identifying CTEs, and examples and instructions for the appli-cation of the TRLs.

– Reviews the TRA report and prepares the TRA report cover memoran-dum, which may include additional technical information deemed appro-priate to support or disagree with IRT findings.

– Sends the completed TRA to the CAE for official transmittal to the DRD and furnishes an advance copy to the DRD.

– Maintains continuity in the IRT membership for all TRAs conducted over the life of a program, to the maximum extent possible.

The CAE

– Approves the TRA report cover memorandum.

– Forwards the TRA to the DRD.

The IRT

– Keeps the Component S&T Executive and the DRD informed on progress throughout the entire TRA process.

– Develops a list of candidate CTEs in conjunction with the program.

– – The IRT should make final recommendations (with associated rationale) on the candidate CTEs that should be assessed in the TRA. These recommendations should be based on (1) full access to specific technical planning performed by existing or previous contractors or government agencies, (2) the CTE definition, 2-4

(3) the PM’s answers to questions, (4) professional experience of IRT members, and (5) a PM-prepared initial list of possible CTEs using the most current system design as a starting point. CTE candidates are not constrained to those technologies on the PM’s initial list. Technologies not included on the program’s initial list may be candidates.

– Assesses the TRLs for all CTEs.

– – The assessment must be based on objective evidence gathered during events such as tests, demonstrations, pilots, or physics-based simulations. Based on the requirements, identified capabili-ties, system architecture, software architecture, concept of opera-tions (CONOPS), and/or the concept of employment, the IRT will define relevant and operational environments and determine which TRL is supported by the objective evidence. The IRT can form subteams based on members’ subject matter expertise. These subteams could deliberate on the appropriate TRL and then defend their position to the entire IRT.

– Prepares (or oversees the preparation of) elements of the TRA report including (1) the IRT credentials and (2) IRT deliberations, findings, conclusions, and supporting evidence.

– – The assessment process should not be constrained to a validation of a “program-developed” position on the TRL.

The DRD

– Concurs with the TRA schedule.

– Concurs with the composition of the IRT.

– Reviews the candidate CTE list and identifies any changes necessary to form the final CTE list.

– – Additions to the list can include any special- interest technologies that warrant the rigor of the formal TRA process.

– Exercises oversight by monitoring and evaluating the TRA process and reviewing the TRA report.

– – On the basis of that review, a TRA revision may be requested or the DRD may conduct its own Independent Technical Assessment

(ITA).

– Sends the results of its TRA review to the appropriate Overarching Inte-grated Product Team (OIPT) and/or the Defense Acquisition Board

(DAB).

2-5

– Provides the DDR&E recommendations concerning certification.

– Recommends technology maturity language for an Acquisition Decision Memorandum (ADM), noting, in particular, conditions under which new technology can be inserted into the program.

3-1

Section 3.

Evolution of Knowledge on Technology Maturity

Assessments of technology readiness or TRA-like activities other than the formal

TRAs at Milestone B and Milestone C take place over the acquisition life cycle. Sec-tion 3.1 discusses early evaluations of technology maturity. Section 3.2 contains a sum-mary table illustrating activities throughout acquisition.

3.1 Early Evaluations of Technology Maturity

In the MSA phase, an Analysis of Alternatives (AoA) is conducted to identify potential materiel solutions, based on a cost-benefit analysis. In parallel, early Systems

Engineering activities, such as the proposed Engineering Analysis of Potential System

Solutions, are conducted. These materiel solutions should then undergo an Early Evalua-tion of Technological Maturity,12 provided sufficient technical information exists to sup-port such an evaluation. This evaluation will identify candidate Critical Technologies or

Critical Technology Areas for each of the potential materiel solutions.

This body of work—the AoA, the early Systems Engineering, and the Early Eval-uation of Technology Maturity—forms the basis of the TDS for evaluating the technol-ogy options in the materiel solution to the capability need identified in the approved

Initial Capabilities Document (ICD). The TDS should show how the technologies (those known by Milestone A to be critical for the successful realization of the chosen materiel solution) will be demonstrated in a relevant environment before they are used in EMD. If the AoA and early Systems Engineering work do not result in sufficient technical infor-mation to support evaluation of technology maturity, such an evaluation will be needed before Milestone A so that critical technologies can be matured during the Technology

Development phase.

The key differences between the early evaluation of technology maturity at Mile-stone A and the required evaluation at Milestone B TRA are as follows:

For the early evaluation of technology maturity, the IRT should include sys-tem-level generalists in addition to SMEs.

12 This early evaluation of technology maturity is not a replacement for the Milestone B TRA.

3-2

The candidate CTE list should be based on information from Broad Agency Announcements (BAAs), Requests for Information (RFIs), market surveys, actual results from government- or industry-funded efforts, and any initial system design concepts being considered by the program office.

Because multiple design/technology options may be available early in the program, the PM should develop a potential CTE list that includes technolo-gies associated with all the options. The IRT should use its collective exper-tise to review and refine this list and determine a preliminary technology maturity assessment, without using TRLs, for each CTE based on require-ments and environments specified in the ICD or draft Capability Develop-ment Document (CDD).

The early evaluation of technology maturity should be signed off by the Component S&T Executive’s office and sent directly to the DRD. The DRD review will be forwarded to the PM, the relevant OIPT, and the Joint Requirements Oversight Council (JROC).

A best practice is to use the results of this early evaluation of technology maturity as follows:

To provide a basis for modifying the requirements if technological risks are too high

To support the development of TMPs that show how all likely CTEs will be demonstrated in a relevant environment before preliminary design begins at the full system level

To refine the TDS (Note that the results of the DRD review of the early eval-uation of technology maturity will form the basis of the DDR&E’s concur-rence or non-concurrence with the TDS).

To inform the test and evaluation (T&E) community about technology matur-ity needs

To ensure that all potential CTEs are included in the program’s risk manage-ment database and plan

To establish Technology Transition Agreements (TTAs) to articulate external dependencies on technology base projects and to define the specific technol-ogies, technology demonstration events, and exit criteria for the technology to transition into the acquisition program.

The early evaluation of technology maturity conducted at or shortly before Mile-stone A helps evaluate technology alternatives and risks and, thereby, helps the PM refine the plans for achieving mature technologies at Milestone B.

3-3

The DRD can also perform a “quick-look” TRA in conjunction with an in-process review, typically in response to a request by the MDA. A “quick-look” TRA is usually conducted by the DRD staff, who are schedule driven and do not use TRLs.

3.2 Summary

Table 3-1 summarizes how the knowledge concerning technology maturity evolves over time. It shows the basis of technology identification, the status of the CTEs, the method for assessing CTEs, and how the evaluation is documented.

Table 3-1. Basis of Technology Maturity Assessments Throughout Acquisition

Milestone A Milestone B Milestone C

Basis of CTE Identification

Early evaluation of technology maturity

Current level of design and CDD requirements

Planned LRIP article (or limited deploy-ment version of an IT system), prior TRAs, and final design

CTE Identification Status

Potential CTEs CTEs – actual tech-nologies in a pre-liminary design

CTEs of planned LRIP articles (or limited deployment version of an IT system)

Assessment Method Evaluated in early evaluations of tech-nology maturity and TMPs

Assessed in Mile-stone B TRA

Assessed in Milestone C TRA

Documentation Informal submission to DRD and corresponding updates to TDS appendix

Milestone B TRA Milestone C TRA

ACR-1

List of Acronyms13

510(k) Premarket Notification for Medical Devices

ACAT Acquisition Category

ACAT IAM The MDA is the DoD CIO (the ASD (NII)). The “M” refers to Major Automated Information Systems Review Council

(MAISRC)

ACAT IC The MDA is the DoD Component Head or, if delegated, the Component Acquisition Executive (CAE). The “C” refers to Component

ACAT ID The MDA is USD(A&T). The “D” refers to the DAB, which advises the USD(A&T) at major decision points.

ADM Acquisition Decision Memorandum

ADR Adverse Drug Reaction

AO Action Officer

AoA Analysis of Alternatives

APS active protection system

ASD(NII) Assistant Secretary of Defense for Networks and Information Integration

ATM Asynchronous Transfer Mode

BAA Broad Agency Announcement

BLA Biologics License Application

CAD computer-aided design

CAE Component Acquisition Executive

CBER Center for Biologics Evaluation and Research

CDD Capabilities Development Document

CDER Center for Drug Evaluation and Research

CDR Critical Design Review

CDRH Center for Devices and Radiologic Health

CFD computational fluid dynamics

CFR Code of Federal Regulations cGMP current Good Manufacturing Practice

CI configuration item

13 This is a comprehensive acronym list for the main body of this report and for the appendixes.

ACR-2

CIO Chief Information Officer

CJCSI Chairman of the Joint Chief of Staff Instruction

CMC chemistry, manufacturing, and controls

CONOPS concept of operations

COTS commercial off-the-shelf

CPD Capability Production Document

CTE Critical Technology Element

DAB Defense Acquisition Board

DAU Defense Acquisition University

DDR&E Director of Defense Research and Engineering

DepSecDef Deputy Secretary of Defense

DIR, ARA Director, Acquisition Resources and Analysis

DMR Device Master Record

DoD Department of Defense

DoDAF Department of Defense Architecture Framework

DoDD Department of Defense Directive

DoDI Department of Defense Instruction

DOE Department of Energy

DOF degree of freedom

DRD Director, Research Directorate

DSAB Defense Space Acquisition Board

DT&E developmental test and evaluation

DTC design-to-cost

DUSD(A&T) Deputy Under Secretary of Defense for Acquisition and Technology

EDM Engineering Development Model

EMD Engineering and Manufacturing Development

FD&C Federal Food, Drug, and Cosmetic

FDA Food and Drug Administration

FFRDC Federally Funded Research and Development Center

FOR field of regard

FOV field of view

FR Federal Register

GATES Global Air Transportation Execution System

GCP Good Clinical Practice

GFM Government Freight Management

GIG Global Information Grid

GLP Good Laboratory Practice

ACR-3

GOTS government off-the-shelf

HIPAA Health Insurance Portability and Accountability Act

HIV Human Immunodeficiency Virus

HM&E hull, mechanical, and electrical

HMD helmet-mounted display

HMMWV High Mobility Multipurpose Wheeled Vehicle

HSDP Homeland Security Presidential Directive

HW hardware

HWIL hardware-in-the-loop

IA information assurance

ICD Initial Capabilities Document

ICH International Conference on Harmonisation

IDA Institute for Defense Analyses

IDE Investigational Device Exemption

IER information exchange requirement

IM Information Management

IMP Integrated Master Plan

IMS Integrated Master Schedule

IND Investigational New Drug

IOT&E initial operational test and evaluation

IPT Integrated Product Team

IR infrared

IRT Independent Review Team

IT Information Technology

ITA Independent Technical Assessment

ITAB Information Technology Acquisition Board

ITV in-transit visibility

JROC Joint Requirements Oversight Council

KDP Key Decision Point

L/D lift-to-drag

LAN local area network

LRIP low rate initial production

M&S modeling and simulation

MAIS Major Automated Information System

MAISRC Major Automated Information Systems Review Council

MDA Milestone Decision Authority

MDAP Major Defense Acquisition Program

ACR-4

MEMS Microelectromechanical Systems

MIL-HDBK Military Handbook

MS Milestone

MSA Materiel Solution Analysis

MTS Movement Tracking System

NASA National Aeronautics and Space Administration

NDA New Drug Application

NIC network interface card

NIST National Institute of Standards and Technology

NSA National Security Agency

OIPT Overarching Integrated Product Team

OSD Office of the Secretary of Defense

OT&E operational test and evaluation

OV Operational View

P.L. Public Law

PAI Preapproval Inspection

PDA personal digital assistant

PEO Program Executive Office

PI principal investigator

PM Program Manager; program manager

PMA Premarket Approval

POC point of contact

QoS Quality of Service

QSIT Quality System Inspection Technique

QSR Quality System Regulation

R&D research and development

RAID redundant array of inexpensive disks

RDT&E research, development, test, and evaluation

RF radio frequency

RFI Request for Information

RFID Radio Frequency Identification

RFP Request for Proposal

S&T Science and Technology, science and technology

SAE Serious Adverse Event

SAN storage area network

SE Substantial Equivalence

SEP Systems Engineering Plan

ACR-5

SFC specific fuel consumption

Sim/Stim Simulation/Stimulation

SIPRNet Secret Internet Protocol Router Network

SME subject matter expert

SPO System Program Office

SQL Structured Query Language

SUBSAFE Submarine Safety Certification Program

SV Systems View

SW software

SWAP size, weight, and power

T&E test and evaluation

TDS Technology Development Strategy

TEMP Test and Evaluation Master Plan

TMP Technology Maturation Plan

TRA Technology Readiness Assessment

TRL Technology Readiness Level

TTA Technology Transition Agreement

TV Technical Standards View

U.S. United States

U.S.C. United States Code

USAMRMC United States Army Medical Research and Materiel Command

USD(AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics

WAN wide area network

WBS work breakdown structure

WSERB Weapon Systems Explosive Safety Review Board

XML eXtensible Markup Language

A-1

Appendix A.

Submitting a Technology Readiness Assessment (TRA)

A.1 Skeletal Template for a Technology Readiness Assessment (TRA) Submission ........................................................................................................ A-3

A.2 Annotated Template for a TRA Submission ..................................................... A-4

A.3 TRA Submission Cover Letter .......................................................................... A-6

A-3

A.1 Skeletal Template for a Technology Readiness Assessment (TRA) Submission

The TRA report should consist of (1) a short description of the program including an explicit statement identifying the program increments covered, if relevant, (2) the

Independent Review Team (IRT) credentials, (3) IRT deliberations, findings, conclu-sions, supporting evidence, differing opinions, and a description of the method for adju-dicating differing opinions, (4) other technical information deemed pertinent by the

Component S&T (Science and Technology) Executive, and (5) a cover letter signed by the Component S&T Executive.

The following outline is a skeletal template for anticipated TRA submissions:

1.0 Purpose of This Document

2.0 Program Overview

2.1 Program Objective

2.2 Program Description

2.3 System Description

3.0 Technology Readiness Assessment (TRA)

3.1 Process Description

3.2 Critical Technology Elements (CTEs)

3.3 Assessment of Maturity

3.3.1 First CTE or Category of Technology

3.3.2 Next CTE or Category of Technology

4.0 Summary

A-4

A.2 Annotated Template for a TRA Submission

The following outline is an annotated version of the TRA template.

1.0 Purpose of This Document

Provides a short introduction that includes the program name, the system name if different from the program name, and the milestone or other decision point for which the TRA was performed. For example, “This document presents an independent TRA for the UH-60M helicopter program in support of the Mile-stone B decision. The TRA was performed at the direction of the Army S&T Executive.”

2.0 Program Overview

2.1 Program Objective

States what the program is trying to achieve (e.g., new capability, improved capability, lower procurement cost, reduced maintenance or manning, and so forth). Refers to the Capability Development Document (CDD) (for Milestone B) or the Capability Production Document (CPD) (for Milestone C) that details the program objectives.

2.2 Program Description

Briefly describes the program or program approach—not the system. Does the program provide a new system or a modification to an existing operational system? Is it an evolutionary acquisition program? If so, what capabilities will be realized by increment? When is the Initial Operational Capability (IOC)? Does it have multiple competing prime contractors? Into what architecture does it fit?

Does its success depend on the success of other acquisition programs?

Also, explicitly identifies the increments covered by the TRA, if relevant.

2.3 System Description

Describes the overall system, the major subsystems, and components to give an understanding of what is being developed and to show what is new, unique, or special about them. This information should include the systems, components, and technologies that will later be declared CTEs. Describes how the system works (if this is not obvious).

3.0 Technology Readiness Assessment (TRA)

3.1 Process Description

Tells who led the TRA and what organizations or individuals were included as part of the Independent Review Team (IRT). Identifies the special expertise of these participating organizations or individuals. This information should establish the subject matter expertise and the independence of the IRT. Members should be experts in relevant fields and should be sufficiently independent of the developers (government or industry) as to not be unduly influenced by their opinions or have

A-5 any actual or perceived biases. To avoid being influenced by the program man-ager (PM), a IRT member should not be directly working for or matrixed to the program. Usually, the PM will provide most of the data and other information that form the basis of a TRA. Nevertheless, the assessment should be independent of the PM.

Tells how CTEs were identified (i.e., the process and criteria used and who identified them). States what analyses and investigations were performed when making the assessment (e.g., examination of test setups, discussions with test per-sonnel, analysis of test data, review of related technology, and so forth).

This is only a broad description of the process. Paragraph 3.3 presents an opportunity to include more detail.

3.2 Critical Technology Elements (CTEs)

Shows the technical work breakdown structure (WBS) or systems architec-ture and software architecture and the CTEs. Lists the technologies included in the TRA. Explains the criterion for technologies that were included on the list of CTEs. Describes the environment that surrounds each CTE. Can include a table that lists the technology name and includes a few words that describe the technol-ogy, its function, and the environment in which it will operate. The names of these CTEs should be used consistently throughout the document.

Includes any additional technology elements that the Component S&T Executive considers critical.

3.3 Assessment of Maturity

3.3.1 First CTE or Category of Technology

Describes the technology (subsystem, component, or technology). Describes the function it performs and, if needed, how it relates to other parts of the system.

Provides a synopsis of development history and status. This synopsis can include facts about related uses of the same or similar technology, numbers or hours breadboards were tested, numbers of prototypes built and tested, relevance of the test conditions, and results achieved.

Describes the environment in which the technology has been demonstrated.

Provides a brief analysis of the similarities between the demonstrated environ-ment and the intended operational environment.

Applies the criteria for Technology Readiness Levels (TRLs) and assigns a readiness level to the technology. States the readiness level (e.g., TRL 6) and the rationale for choosing this readiness level. Describes differing opinions for arriving at a particular TRL and the method of adjudication.

Provides extensive references to papers, presentations, data, and facts that support the assessments. Includes data tables and graphs that illustrate the appro-priateness of key facts. These references/tables/graphs can be included as an appendix.

If the CTEs presented are in categories (e.g., airframe or sensors), the infor-mation specified in the previous paragraph (e.g., describing the technology, A-6 describing the function it performs, and so forth) should be provided for each CTE within a category.

3.3.2 Next CTE or Category of Technology

For the other CTEs, this paragraph and the following paragraphs (e.g., 3.3.3, 3.3.4, and so forth) present the same type of information that was presented in paragraph 3.3.1.

4.0 Summary

Includes a table that lists the CTEs and presents the assigned TRL and a short explanation (one sentence or a list of factors).

A.3 TRA Submission Cover Letter

The Component S&T Executive should indicate agreement or disagreement with the IRT’s findings in the cover letter, along with supporting analyses. In effect, the Com-ponent S&T Executive must certify that he/she stands behind the results or provide ratio-nale for any differences of opinion.

The cover letter should be routed through the Component Acquisition Executive

(CAE) and addressed to the Director, Research Directorate (DRD).

B-1

Appendix B.

Guidance and Best Practices for Identifying

Critical Technology Elements (CTEs)

B.1 Introduction ......................................................................................................... B-3

B.2 Systems Engineering Context for Identifying CTEs ........................................... B-4

B.3 Procedures and Practices for Identifying CTEs .................................................. B-7

B.3.1 Overall Description ................................................................................ B-7 B.3.2 Environments ......................................................................................... B-8

B.4 Representative Questions for Identifying CTEs ................................................ B-13

B.4.1 Aircraft ................................................................................................. B-14 B.4.2 Ground Vehicles .................................................................................. B-15 B.4.3 Missiles ................................................................................................ B-16 B.4.4 Ships, Submarines, and Naval Weapons Systems ............................... B-17 B.4.5 Information Systems ............................................................................ B-18 B.4.6 Networked Communications and Data Management Systems ............ B-19 B.4.7 Business Systems ................................................................................. B-20 B.4.8 Mission Planning Systems ................................................................... B-21 B.4.9 Embedded IT in Tactical Systems ....................................................... B-22

B-3

Best Practice CTE identification should be a continuing element of every program. An initial determination of potential CTEs should be completed during MSA.

B.1 Introduction

The definition of a CTE is as follows:

A technology element is “critical” if the system being acquired depends on this technology element to meet operational requirements (within acceptable cost and schedule limits) and if the technology element or its application is either new or novel or in an area that poses major techno-logical risk during detailed design or demonstration.

The disciplined identification of CTEs is important to a program’s success. If a

CTE is overlooked and not brought to the requisite maturity level for exploitation at the start of Engineering and Manufacturing Development (EMD), the system performance, program schedule, and overall cost could be jeopardized. On the other hand, if an overly conservative approach is taken and a plethora of technologies are categorized as critical, energy and resources are likely to be diverted from the few technologies that deserve an intense maturation effort. If a disciplined process with due diligence does lead to an inor-dinate number of CTEs, this process should indicate that the proposed development is reaching too far for its goals.

The last phrase of the CTE definition—“or in an area that poses major technologi-cal risk during detailed design or demonstration”—is an essential update to the early versions of the TRA Deskbook. It helps to ensure that no technological risk areas are over-looked when identifying CTEs by including situations in which the technology is not

“new or novel,” as follows:

The technology application typically leads to problems based on past experience.

Predicted obsolescence may lead to a technology issue.

The performance being demanded from the technology exceeds previous requirements.

A major part of the CTE identi-fication process should occur during

Materiel Solution Analysis (MSA).

The Technology Development Stra-tegy (TDS)—a product of the MSA phase—should reflect the result of a process sufficiently thorough and disciplined to identify those technologies (including CTEs) that have a realistic potential to be

B-4 improved in the Technology Development phase and exploited in the EMD phase. An early evaluation of technology maturity, conducted shortly before Milestone A, provides further insight into CTE identification. Failure to recognize the potential CTEs at this stage will result in a waste of resources—time, money, facilities, and so forth—and could result in an unfavorable Milestone B decision.

As system development proceeds, the likelihood exists—through necessity or opportunity—for exploitation of technologies not previously considered. These technolo-gies deserve full consideration to decide whether they are critical and whether they are mature enough to be included in the detailed system design.

The original Department of Defense (DoD) Technology Readiness Level (TRL) definitions and supporting information were developed primarily for performance-related hardware technologies (see Appendix C, Table C-1). In identifying CTEs and assessing their maturity, the distinction between hardware and software technologies became important because different, but related, procedures and metrics are used to identify and assess the maturity of hardware and software CTEs. The original set of definitions suited hardware technologies but was inadequate for software technologies.

The following sections of this appendix provide suggestions about how to identify

CTEs for a variety of systems.1 These discussions apply equally to Major Defense Acqui-sition Programs (MDAPs) and Major Automated Information System (MAIS) programs.

Section B.2 discusses system engineering as the program context for identifying CTEs, Section B.3 covers procedures and practices for CTE identification, and Section B.4 contains representative questions/inquiries to use when making a detailed examination of a system to identify CTEs.

B.2 Systems Engineering Context for Identifying CTEs

CTE identification should be integral to the systems engineering approach for defense acquisition programs. The following definition of systems engineering is extracted from Chapter 4 of the Defense Acquisition Guidebook:2

1 Distinct technology maturity metrics for drugs, vaccines, and medical devices have also been established. See Appendix E.

2 Chapter 4 of the Defense Acquisition Guidebook provides a thorough discussion of systems engineering.

B-5

Systems engineering is an interdisciplinary approach encompassing the entire technical effort to evolve and verify an integrated and total life-cycle balanced set of system, people, and process solutions that satisfy customer needs.