Attachment 0007 - Manufacturing Readiness Assessment Guidance.pdf

Manufacturing Readiness Level (MRL)

Deskbook

Version

Advance Draft as of 15 July 22

Effective as of 1 October 22

Prepared by the

The Joint Service/Industry MRL Working Group

Chartered by the Joint Defense Manufacturing Technology Panel i

This document is not a DoD directive and is being offered as a “best practice.”

V2.2.2: This version has changes made to the Criteria Matrix in the appendix to agree with version 11.4 dated 15 December 2014.

V2.3: This version has adds changes from the 2014 Workshop to address the new DoD

5000.02 and user comments.

V2.4: This version has corrections in the Matrix to version 11.5.

V2.5 / 2016: This version has changes in Appendix A – to update the Criteria Matrix to version 11.6 including additional criteria under H.1 Tooling and adds Chapter 8, “Effectively Adapting and Utilizing MRL Criteria.”

2017: This version adds Section 6.7, “MRLs in SAE AS6500.”

2018: Adjusted wording in MRL definitions to match 2018 criteria and metrics, including

ESH additions. Updates for clarity and changes in statutes, policies, and guidance

(Chapter 3 and elsewhere). 2018 Matrix update to Appendix A.

2020: Updated MRL Criteria Matrix to incorporate enhanced MRL 1-4 criteria (2020);

updated Section 6, “Applying MRLs in Contract Language;” and Deskbook updates for clarity and current policy and guidance.

2022: Updated section 1 to current documents, Section 2 to reflect refined relationship of MRLs to TRLs, Section 3 to DoD AAF guidance, Section 6 to AS6500A, and addition of Cybersecurity appendix.

ii

Table of Contents

Table of Contents .......................................................................................................... ii

Executive Summary

1. Introduction

1.1 Manufacturing Risks Recognized in DoD Policy

1.2 Purpose and Organization of this Document

2. Manufacturing Readiness Levels

2.1 Overview of Manufacturing Readiness Levels

2.2 Manufacturing Readiness Levels Defined

2.3 Definition of Environments and Other Terms

2.4 MRL Threads and Sub-Threads

2.5 The MRL and TRL Relationship

3. MRLs in the Adaptive Acquisition Framework

3.1 Introduction

3.2 MRLs for Major Capability Acquisition Prior to the Materiel Development

Decision

3.3 MRLs for Middle Tier of Acquisition

3.4 MRLs for Urgent Operational Needs (UON)

3.5 Summary

4. Conducting MRL Assessments

4.1 Introduction

4.2 Determine Initial Assessment Scope

4.3 Determine Assessment Taxonomy and Schedule

4.4 Form and Orient Assessment Team

4.5 Orient Contractors Being Assessed

4.6 Request Contractors Perform Self-Assessment

4.7 Set Agenda for Site Visits

4.8 Conduct the MRL Assessment

iii

4.9 Prepare the Assessment Report

5. Manufacturing Maturation Plans (MMP) and Risk Management

5.1 Introduction

5.2 Development of a Manufacturing Maturation Plan

5.3 Risk Management Best Practices

6. Applying MRLs in Contract Language

6.1 Introduction

6.2 Acquisition Planning

6.3 RFP Language

6.4 SOW Language for Contracts

6.5 Deliverables

6.6 Additional Quality Considerations

6.7 MRLs Relationship to AS6500A and Quality Standards

7. Users Guide for Performing MRL Assessments

7.1 The MRL Users Guide

8. Effectively Adapting and Utilizing MRL Criteria

8.1 Introduction

8.2 MRL Criteria in the S&T Environment

8.3 MRL Criteria for Sustainment/Maintenance, Repair & Overhaul (MRO) and

Depot Activities

8.4 MRLs for Single or Limited System Acquisition

8.5 MRL Criteria for Industry

APPENDIX A – Detailed MRL Criteria

Table A-1. Manufacturing Readiness Levels for the Technology and Industrial Base

Thread

Table A-2. Manufacturing Readiness Levels for the Design Thread

Table A-3. Manufacturing Readiness Levels for the Cost and Funding Thread

Table A-4. Manufacturing Readiness Levels for the Materials Thread

Table A-5. Manufacturing Readiness Levels for the Process Capability and Control

Thread iv

Table A-6. Manufacturing Readiness Levels for the Quality Management Thread ... 5

Table A-7. Manufacturing Readiness Levels for the Manufacturing Personnel and

Facilities Threads

Table A-8. Manufacturing Readiness Levels for the Manufacturing Management

Thread

APPENDIX B – Acronyms

Appendix C – Operational Technology Cybersecurity

List of Figures

Figure 2-1 – MRL Summaries

Figure 2-2 – TRLs and Descriptions

Figure 2-3 – MRLs and TRLs in the Major Capability Acquisition Pathway

Figure 3-1 – Adaptive Acquisition Framework

Figure 3-2 – MRLs in Major Capability Acquisition

Figure 3-3 – Major Capability Acquisition Details

Figure 3-4 – MRLs for Rapid Prototyping

Figure 3-5 – Rapid Prototyping Paths

Figure 3-6 – MRLs and Rapid Fielding

Figure 3-7 – Rapid Fielding Pathway

Figure 3-8 – MRLs for Urgent Capability Acquisition Pathway

Figure 3-9 – Urgent Capability Acquisition

Figure 4-1 – Sample Process Flow for Conducting

Figure 4-2 – Example of Added Detail Derived from Site Visits

Figure 6-1 – Mapping of MRL Threads to AS6500A & AS9100D Requirements

Figure 6-2 – IAQG Standards

Figure 8-1 – Relationship of MRLs to SMLs

Figure 8-2 – Example of Unplanned Depot Activation Circumstance

Figure 8-3 – Single or Limited System Acquisitions – except Ships

Figure 8-4 – Single or Limited System Acquisitions – Ships

Figure 8-5 – Gated Product Development

Executive Summary

Manufacturing status and risk evaluations have been performed as part of defense acquisition programs for years in a variety of forms. These evaluations, while often highly structured and well managed, did not use a uniform metric to measure and communicate manufacturing risk and readiness.

Department of Defense Instruction (DoDI) 5000.88, Engineering of Defense Systems, states, “The production, quality, and manufacturing (PQM) lead, working for the

[Program Manager (PM)], will ensure manufacturing, producibility, and quality risks are identified and managed throughout the program’s life cycle.”1 This instruction establishes general maturity criteria for each life cycle phase leading to the production decision.

Assessments of manufacturing maturity utilizing the Manufacturing Readiness Level

(MRL) criteria have been designed to identify and manage manufacturing risk in acquisition, decreasing the risk of technology transition for new technology to weapon system applications. MRL criteria and metrics create a measurement scale and vocabulary for assessing and discussing manufacturing maturity and risk. Using the

MRL criteria and metrics, an MRL Assessment is a structured approach for evaluation of a manufacturing processes, procedures, and techniques for technology, components, items, assemblies, subsystems, and systems. A MRL Assessment, aka Manufacturing

Readiness Assessment (MRA), is performed to:

Define current level of manufacturing maturity

Identify maturity shortfalls and associated risks and costs

Provide the basis for management of manufacturing maturation and risk

This document provides “best practices” for conducting assessments of manufacturing maturity and risk using the MRL criteria. It is intended for those tasked with conducting

MRL Assessments, as well as acquisition program managers, system engineers, manufacturing managers, and managers of technology development and pre-systems acquisition technology demonstration projects.

DoDI 5000.88, Engineering of Defense Systems, November 18, 2020, Section 3.6.c.

1. Introduction

1.1 Manufacturing Risks Recognized in DoD Policy

Manufacturing status and technical risk evaluations have been performed as part of defense acquisition programs for years in a variety of forms (e.g., Production Readiness

Reviews, Manufacturing Management/Production Capability Reviews, etc.).2 These reviews, while highly structured and well managed, did not use a uniform metric to measure and communicate manufacturing risk and readiness.

Studies by the Government Accountability Office (GAO) cite a lack of manufacturing knowledge and maturity at key decision points as a leading cause of acquisition program cost growth and schedule slippages in major DoD acquisition programs.3

Consequently, DoD policy has been developed to strengthen the way in which manufacturing issues and risks are considered in the defense acquisition system.

There is long-standing law on manufacturing-related content of acquisition strategies.

Defense Federal Acquisition Regulation Supplement (DFARS) Section 207.105, Contents of Written Acquisition Plans4 mandates specific National Technology and

Industrial Base (NTIB) considerations are included in acquisition strategies for Major

Defense Acquisition Programs (MDAP) as follows:

An analysis of capabilities of the NTIB to develop, produce, maintain, and support programs, including consideration of factors related to foreign dependency

Consideration of requirements for efficient manufacture during the design and production of the systems to be procured under the program

The use of advanced manufacturing technology, processes, and systems during the research and development phase and the production phase of the program

To the maximum extent practicable, the use of contract solicitations that encourage competing offerors to acquire, for use in the performance of the contract, modern technology, production equipment, and production systems (including hardware and software) that increase the productivity and reduce the life cycle costs

Manufacturing risk is one element of overall technical risk to the program.

Defense Acquisitions: Assessment of Selected Weapon Programs (GAO-19-336SP), Government

Accountability Office, May 2019. Similar conclusions were made in prior GAO reports issued annually since 2004. These reports may be accessed at https://www.gao.gov/reports-testimonies.

PGI 207.105, Acquisition Plans, Defense Federal Acquisition Regulation Supplement (DFARS), revised Jan 2021; https://www.acq.osd.mil/dpap/dars/pgi/pgi_htm/PGI207_1.htm#207.105 https://www.gao.gov/reports-testimonies https://www.acq.osd.mil/dpap/dars/pgi/pgi_htm/PGI207_1.htm#207.105

Methods to encourage investment by U.S. domestic sources in advanced manufacturing technology production equipment and processes through:

o Recognition of the contractor’s investment in advanced manufacturing technology production equipment, processes, and organization of work systems that build on workers’ skill and experience, and work force skill development in the development of the contract objective o Increased emphasis in source selection on the efficiency of production.

Additionally, Congress has put additional focus on managing manufacturing risk in

Public Law by requiring MDAPs to “identify critical technologies and manufacturing processes that need to matured by Milestone A and that have not been successfully demonstrated in a relevant environment by Milestone B”5

The GAO found that DoD faces problems in manufacturing weapon systems — systems cost far more and take much longer to build than estimated. Billions of dollars in cost growth occur as programs transition from development to production, and unit cost increases are common after production begins. Contributing factors to these problems include the following: inattention to manufacturing during planning and design; poor supplier management; and a deficit in manufacturing knowledge among the acquisition workforce. Essentially, programs did not identify and resolve manufacturing risks early in development, but carried risks into production where they emerged as significant problems. The GAO has recommended DoD adopt the use of MRLs to help manage the manufacturing risk.6

DoDI 5000.88 reinforces the requirement to address manufacturing over the entire system life cycle. It requires the PQM Lead, working for the PM, to identify and manage manufacturing, producibility, and quality risks.7 NOTE: See Section 2.3 for definitions for manufacturability and producibility.

By the end of the Technology Maturation Risk Reduction (TMRR) Phase, manufacturing and quality processes will be assessed and demonstrated to the extent needed to verify that risk has been reduced to an acceptable level.

During the Engineering and Manufacturing Development (EMD) Phase, the PQM lead will advise the PM on the maturity of critical manufacturing and quality processes to ensure they are affordable and executable. Prior to a production decision, the PQM lead will ensure that:

Manufacturing, producibility, and quality risks are acceptable.

National Defense Authorization Act for Fiscal Year 2017, P.L. 114-328, 23 Dec 2016: § 807

Best Practices: DoD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks

Are Managed (GAO-10-439), Government Accountability Office, Apr 2010 DoDI 5000.88, Engineering of Defense Systems, Section 3.6c, Nov 2020

Supplier qualifications are completed.

Any applicable manufacturing processes are or will be under statistical process control.

1.2 Purpose and Organization of this Document

This document provides “best practices” for conducting assessments of manufacturing maturity and risk during the acquisition process using the MRL criteria. It is intended for those tasked with conducting MRL Assessments, as well as acquisition PMs, system engineers, manufacturing managers, and managers of technology development and pre-systems acquisition technology demonstration projects. This guidance is based on lessons learned, “best practices” in manufacturing, DoD policy, and real-life manufacturing experience of industry and government.

Use of MRL Assessments is recognized as a “best practice” to:

Identify areas of risks, issues, and opportunities

Minimize manufacturing risk in product development to transition effectively to production

Provide earlier manufacturing input to the design and decision processes

Allow maximum flexibility and tailorability in application by a diverse industry and contractor community developing and producing various products

The following sections of this document describe:

Each of the MRLs in detail (Section 2)

How manufacturing maturity and risk evolves throughout the acquisition process and is addressed in the Adaptive Acquisition Framework (Section 3)

The process for conducting assessments using MRLs (Section 4)

Manufacturing maturation plans and risk management (Section 5)

Suggested contract language for implementing MRLs and relationship to AS6500A (Section 6)

Users Guide for performing MRL Assessments (Section 7)

Adaptation of MRL Assessments and criteria to specific situations (Section 8)

MRL criteria and metrics by thread over the acquisition life cycle (Appendix A – MRL Criteria Matrix)

Acronyms (Appendix B)

Additional information can be found at the DoD MRL website DoDMRL.org.

https://dodmrl.org/

2. Manufacturing Readiness Levels

2.1 Overview of Manufacturing Readiness Levels

There are ten levels of MRL criteria that begin with pre-systems acquisition; progress through systems engineering reviews, acquisition decision points, and milestones; and culminate in production. Each of these levels is associated with the evolution of system maturity (i.e., developmental state changes such as bread-board, brass-board, prototype, production configuration, Low Rate Initial Production (LRIP), Full Rate

Production (FRP)).

MRLs 1-4: Criteria address manufacturing maturity and risks beginning with pre-systems acquisition (MRLs 1-3); continue through the selection of a solution

(MRL 4).

MRLs 5-6: Manufacturing maturation of the needed technologies through early prototypes of components or subsystems/systems, culminating in a preliminary design.

MRL 7: The criteria continue by providing metrics for an increased capability to produce systems, subsystems, or components in a production representative environment leading to a critical design review.

MRL 8: The next level of criteria encompasses proving manufacturing process, procedure, and techniques on the designated “pilot line” (see Section 2.3).

MRL 9: Once a decision is made to begin LRIP, the focus is on meeting both quality, throughput, and rate to enable transition to FRP.

MRL 10: The final MRL measures aspects of lean practices and continuous improvement for systems in production.

The basic goal of all acquisition programs is to put required capability in the field in a timely manner with acceptable affordability and supportability. To be successful, the two key risk areas of critical technologies8 being assessed, and immature manufacturing capability, including the supply chain, must be identified, mitigated, and managed effectively.

Technology Readiness Assessment Guide (GAO-20-48G), Government Accountability Office, 11 Feb

2020, pg 47: “A technology element is considered a critical technology if it is new or novel, or used in a new or novel way, and it is needed for a system to meet its operational performance requirements within defined cost and schedule parameters.”

Understanding and mitigating these risks will greatly increase the probability of successful technology insertion by the development community. This will ultimately aid the program success through improvements in cost, schedule, and performance.

Manufacturing readiness metrics will help acquisition PMs mitigate these risks.

Manufacturing readiness metrics are also important to technology development managers because they can be used to measure and demonstrate maturity of technology that acquisition PMs will find credible.

2.2 Manufacturing Readiness Levels Defined

Although the MRLs are numbered, the numbers represent a target used to focus the team on the potential risks associated with reaching program goals. Using numbers is simply a convenient designation. The numbers are a non-linear ordinal scale that identifies what the manufacturing maturity should be as a function of where a program is in the acquisition life cycle (as described in Section 3). The following descriptive paragraphs provide only a short summary of the criteria and metrics for each level. The full criteria and metrics are detailed in the MRL Criteria Matrix shown in

Appendix A and available at DoDMRL.org.

Figure 2-1 – MRL Summaries

MRL 1: Basic Manufacturing Implications Identified

This is the initial level of criteria for assessing manufacturing readiness. The focus is manufacturing capability and begins in the form of studies. Criteria include identification https://www.dodmrl.org/ and investigation of global trends in the industrial and supply base, manufacturing science, material availability, supply chain, and metrology.

MRL 2: Manufacturing Concepts Identified

This level of criteria for assessing manufacturing readiness is characterized by identification of manufacturing concepts. Typically, this includes identification and broad-based studies that address analysis of material and process approaches, material effects and availability, potential supply chains, needed workforce skillsets, and potential future investments, etc. Potential manufacturing and quality future requirements are identified and analyzed. An understanding of manufacturing feasibility and risk is emerging.

MRL 3: Manufacturing Proof of Concept Developed

This level of criteria for assessing manufacturing begins with the analysis and evaluation of the producibility and manufacturability of the proposed system concepts through analytical modeling and simulations and/or laboratory experiments. System concepts comparative cost models, analyses, and budgets are identified. Manufacturing and quality requirements for proposed system concepts are identified and analyzed, including initial quality risks and issues, facility capabilities and capacity, and initial materials planning. This level of readiness is typical of technologies in Applied Research and Advanced Technology Development. Experimental hardware models have been developed in a laboratory environment that may possess limited functionality.

MRL 4: Capability to produce prototype components in a laboratory environment

This level of manufacturing maturity is an exit criterion for the Materiel Solution Analysis

(MSA) Phase approaching a Milestone A decision. Manufacturing and quality risks have been identified and included in the Analysis of Alternatives (AoA). These risks lead to building prototypes and documented mitigation plans. At this point, required investments such as capital, manufacturing technology development, and risk mitigation have been identified. Process variables, manufacturing, materials, and special requirement cost drivers have been identified, and cost driver uncertainty has been quantified. Initial producibility assessments of preferred material solution completed. Initial Key

Performance Parameters (KPP) have been identified as well as any requirements for special tooling, special handling, manufacturing skill sets, and workforce requirements and availability of facilities.

MRL 5: Capability to produce prototype components in a production relevant environment

This level of manufacturing maturity is typical of the mid-point in the TMRR Phase of acquisition. The industrial base assessment should have been initiated to identify potential manufacturing sources. The manufacturing strategy developed for Milestone

A’s Acquisition Strategy (AS) has been refined with the technology maturation contractor and integrated into the Risk Management Plan. Identification of enabling/critical technologies and components is complete. With release of product data required for prototype component manufacturing, evaluation of the design to determine

Key Characteristics (KC) has been initiated. Prototype materials have been demonstrated on components in a production relevant environment, but many manufacturing processes and procedures are still in development. Manufacturing technology development efforts, as well as producibility assessments of key technologies and components have been initiated.

MRL 6: Capability to produce a prototype system or subsystem in a production relevant environment

This level of manufacturing maturity is associated with readiness for a Milestone B decision to initiate an acquisition program by entering into the Engineering and

Manufacturing Development (EMD) Phase of acquisition. It is normally seen as the level of manufacturing readiness that denotes acceptance of a preliminary system design. An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined and characterized, but there are still significant engineering and/or design changes in the system itself. However, preliminary design has been completed and producibility assessments and trade studies of key technologies and components are complete. Manufacturing processes and manufacturing technology solutions, materials, tooling and test equipment, as well as personnel skills have been demonstrated on components, subsystems, and/or systems in a production relevant environment. Cost, yield, and rate analyses have been performed to assess how prototype data compare to target objectives, and the program has developed appropriate risk reduction strategies to achieve cost requirements.

Producibility trade studies and producibility considerations have shaped system development plans. Industrial capabilities assessment for Milestone B has been completed. Long-lead and key supply chain elements have been identified.

MRL 7: Capability to produce systems, subsystems, or components in a production representative environment

This level of manufacturing maturity is typical for the completion of system detailed design activity in the EMD Phase leading to the Critical Design Review (CDR). System detailed design activity is nearing completion. Material specifications have been approved and materials are available to meet the planned pilot line build schedule.

Manufacturing processes and procedures have been demonstrated in a production representative environment. Detailed producibility trade studies are completed and producibility enhancements and risk assessments are underway. The cost model has been updated with detailed designs produced in a production relevant environment, rolled up to system level, and tracked against allocated targets. Unit cost reduction efforts have been prioritized and are underway. Yield and rate analyses have been updated with production representative data. The supply chain and supplier quality assurance have been assessed and long-lead procurement plans are in place.

Manufacturing plans and quality targets have been developed. Production tooling and test equipment design and development efforts are initiated and validation plans for

Special Test Equipment/ Special Inspection Equipment (STE/SIE) are complete.

MRL 8: Pilot line capability demonstrated; ready to begin Low-Rate Initial

Production (LRIP)

This maturity level is associated with manufacturing readiness for a Milestone C decision, and entry into LRIP. Detailed system design is sufficiently stable to enter low-rate production. All materials, manpower, tooling, test equipment, and facilities are proven on the pilot line and are available to meet the planned low-rate production schedule. STE/SIE has been validated as part of pilot line validation in accordance with validation plans. Manufacturing and quality processes and procedures have been proven on a pilot line and are under control and ready for low-rate production. Known producibility risks and issues pose no significant challenges for low-rate production.

Cost model and yield and rate analyses have been updated with pilot line results.

Supplier qualification testing and first article inspections have been completed. The industrial base has been assessed for Milestone C and shows industrial capability is established to support LRIP.

MRL 9: Low-rate production demonstrated; Capability in place to begin Full Rate

Production (FRP)

At this level, the system, component, or item is in production, or has successfully achieved low-rate production. This level of readiness is normally associated with readiness for entry into FRP. All systems engineering/design requirements should have been met such that there are minimal system changes. Major system design features are stable and have been proven in operational test and evaluation. Materials, parts, manpower, tooling, test equipment, and facilities are available to meet planned rate production schedules. STE/SIE validation maintained and revalidated as necessary.

Manufacturing process capability in a low-rate production environment is at an appropriate quality level to meet KC tolerances. Risks and issues managed with monitoring ongoing. LRIP cost targets have been met, and learning curves have been analyzed with actual data. The cost model has been updated for FRP and reflects the impact of continuous improvement.

MRL 10: Full Rate Production demonstrated and lean production practices in place

This is the highest level of manufacturing maturity. This level of manufacturing is normally associated with the “Production & Deployment” or “Operations & Support” phases of the acquisition life cycle. Engineering/design changes are few and generally limited to continuous improvement changes or obsolescence issues. System, components, and items are in full rate production and meet all engineering, performance, quality, and reliability requirements. Manufacturing process capability is at the appropriate quality level. All materials, tooling, inspection and test equipment, facilities and manpower are in place and have met full rate production requirements.

STE/SIE validation maintained and revalidated as necessary. Rate production unit costs meet goals, and funding is sufficient for production at required rates. Continuous process improvements are ongoing.

2.3 Definition of Environments and Other Terms

As manufacturing maturity increases, demonstration of manufacturing capabilities should be accomplished in increasingly realistic manufacturing environments. Prior to

Milestone A, the MRL criteria focus on manufacturing feasibility by identifying manufacturability and producibility of the proposed concepts and reducing the production risk. These proposed concepts are generally demonstrated in a laboratory environment.

Laboratory Environment – an environment where scientists, design engineers, manufacturing engineers, quality engineers, and production personnel develop and test processes, procedure, and equipment for making a product.

The results from the laboratory environment should be used to assess:

Producibility and manufacturability

Testing of new equipment

Testing of processes and procedures

Initial manufacturing process quality data

Initial manufacturing process rate data

New workforce skill requirements

During TMRR prior to Milestone B, the MRL criteria focus on a capability to produce prototypes outside the lab in a production relevant environment. The parameters defining a production relevant environment should be based on the risks and uniqueness associated with demonstrating that manufacturing processes, procedures, and techniques meet program requirements. Production realism for this environment is well beyond what is seen in the laboratory. An emphasis should be placed on addressing higher risk areas (e.g., more advanced manufacturing technologies and newer manufacturing capabilities).

Production Relevant Environment – An environment with some shop floor production realism present (e.g., facilities, personnel, tooling, processes, materials, etc.). There should be minimum reliance on laboratory resources during this phase. Demonstration in a production relevant environment implies that contractor(s) must demonstrate their ability to meet the cost, schedule, and performance requirements of the EMD Phase based on their production of prototypes. The demonstration must provide the program with confidence that these targets will be achieved, but does not require a production line.

Furthermore, there must be an indication of how the program intends to achieve the requirements in production representative and pilot environments.

Demonstration of manufacturing capability in a production relevant environment provides a better understanding of the EMD Phase manufacturing risk of the program meeting cost, schedule, and performance requirements.

As a program enters into the EMD phase and hardware is built for qualification testing, the manufacturing processes should become more robust and mature to address production representative activities on the whole program.

Production Representative Environment – An environment that has as much production realism as possible, considering the maturity of the design. Production personnel, equipment, processes, and materials that will be present on the pilot line should be to the maximum extent possible. The work instructions and tooling should be of high quality, and the only changes anticipated on these items are associated with design changes downstream that address performance or production rate issues. There should be no reliance on a laboratory environment or personnel.

The final stage of EMD is producing products that look and operate like they are production units from LRIP. These units need to be built on a pilot production line to adequately demonstrate the ability to migrate from EMD to LRIP. Without this realism, it would be very difficult to obtain confidence that the production processes will be able to meet cost, schedule, and performance requirements for production.

Pilot Line – An environment that incorporates all of the key production realism elements (e.g., equipment, personnel skill levels, facilities, materials, components, work instructions, processes, tooling, temperature, cleanliness, lighting etc.) required to manufacture production configuration items, subsystems or systems that meet design requirements in low-rate production. To the maximum extent practical, the pilot line should utilize full rate production processes.

Production Line – An environment that incorporates all capabilities required to manufacture production configuration items, subsystems, or systems that meet design requirements utilizing manufacturing processes and procedures that are under control and capable of meeting required rates and quantities.

The definitions of production relevant, production representative, pilot line, and production line environments are intended to demonstrate the natural progression of manufacturing maturity throughout the acquisition life cycle. The program office and contractor must reach agreement on the detailed production realism content for each definition above. This agreement must be based on the specific situation and its associated manufacturing risk in order to mitigate that risk in a timely and thorough manner.

Two other definitions are germane to this discussion:

Manufacturability—The characteristics considered in the design cycle that focus on process capabilities, machine or facility flexibility, and the overall ability to consistently produce at the required level of cost and quality. Associated activities may include some or all of the following:

Design for commonality and standardization—uses fewer parts

Design for environmental and safety compliance

Design for multi-use and dual-use applications

Design for modularity and plug-compatible interface/integration

Design for flexibility/adaptability or use “robust design”

Utilize reliable processes and materials

Utilize monolithic and determinant assembly

Design for manufacturing and assembly

Achieve production yield

Producibility—The relative ease of producing an item that meets engineering, quality, and affordability requirements. Associated activities may include some of the following:

Design for specific process capability and control parameters

Perform material characterization analysis

Perform variable reduction analysis, e.g., Taguchi and design of experiments

Develop critical materials and processes before selecting product design

Utilize modeling and simulation for product and process design tradeoffs

Design and development of closed-loop process control on critical items

2.4 MRL Threads and Sub-Threads

Successful manufacturing has many dimensions. MRL threads have been defined to organize these dimensions into nine manufacturing risk areas. The threads are as follows:

Technology and Industrial Base: Requires an analysis of the capability of the National Technology Industrial Base to support the design, development, production, operation, uninterrupted maintenance support of the system and eventual disposal (i.e., for environmental impacts).

Design: Requires an understanding of the producibility, maturity, and stability of the evolving system design, identification, and control of KCs, and any related impact on manufacturing readiness.

Cost and Funding: Requires an analysis of the adequacy of funding to achieve target manufacturing maturity levels. Examines the risks associated with reaching manufacturing cost targets.

Materials: Requires an analysis of the risks associated with materials (including basic/raw materials, components, semi-finished parts, and subassemblies).

Process Capability and Control: Requires an analysis of the risks that the manufacturing processes are able to reflect the design intent (repeatability and affordability) of KCs.

Quality: Requires an analysis of the risks and management efforts to control quality, and foster continuous improvement.

Manufacturing Workforce (Engineering & Production): Requires an assessment of the required skills, availability, and number of personnel to support the manufacturing effort.

Facilities: Requires an analysis of the capabilities and capacity of key manufacturing facilities (e.g., prime, subcontractor, supplier, vendor, and maintenance/repair).

Manufacturing Management: Requires an analysis of the orchestration of all elements needed to translate the design into an integrated and fielded system (meeting Program goals for affordability and availability).

Many of the MRL threads have been decomposed into sub-threads. This enables a more detailed understanding of manufacturing readiness and risk, thereby ensuring continuity in maturing manufacturing from one level to the next. These include:

Technology and Industrial Base includes Industrial Base issues and Manufacturing Technology Development

Design includes Producibility Program and Design Maturity

Cost and Funding includes Production Cost Knowledge (cost modeling), Cost Analysis, and Manufacturing Investment Budget

Materials includes Maturity, Availability, Supply Chain Management, and Special Handling (i.e., Government Furnished Program (GFP); shelf life;

security; hazardous materials; storage environment; Environmental, Safety, and Health (ESH); etc.)

Process Capability and Control includes Modeling & Simulation (product and process), Manufacturing Process Maturity, and Process Yields and Rates

Quality includes Quality Management, Product Quality, and Supplier Quality/Management

Manufacturing Management includes Manufacturing Planning & Scheduling, Materials Planning, Manufacturing Operational Technology Cybersecurity, and tooling/special test and inspection equipment

The MRL Criteria Matrix shown in Appendix A provides detailed criteria for each of the ten MRLs, by thread and sub-thread, throughout the acquisition life cycle. The matrix allows a user to separately trace and understand the maturation progress of each of the threads and sub-threads as readiness levels increase from MRL 1 though MRL 10.

These thread and sub-thread MRL criteria should be applied when appropriate to the situation and may be tailored to a particular technology or application.

As stated earlier, the MRL number is simply a convenience referring to the MRL criteria used by the MRL Assessment. The degree of maturity for the program element being assessed is what is important and should address the following two questions:

1. Has the program element met the appropriate manufacturing maturity; and

2. If not, what has to be accomplished to meet the metric?

This information is determined in the assessment process using the MRL Criteria

Matrix, not by assigning a number to the element being assessed.

2.5 The MRL and TRL Relationship

PMs and Milestone Decision Authorities (MDA) should consider manufacturing-related concerns during technology development, especially for new and novel technologies that impact system-level performance characteristics. This need to consider Technology

Readiness Levels (TRL) within the MRL construct has forged the basis of Moving

Manufacturing Left in policy and practitioner circles. The TRL process has been used for many years as the maturation measurement approach for all DoD programs in tracking technology development and transition into production and fielding. MRL practitioners should understand the interplay between TRLs/MRLs when advising PMs/MDAs of manufacturing risk.

The nine hardware TRLs are:

Figure 2-2 – TRLs and Descriptions

Figure 2-3 depicts the notional relationship in the context of the acquisition pathway presented below.

Figure 2-3 – MRLs and TRLs in the Major Capability Acquisition Pathway

MRLs define manufacturing maturity and risk based on the current phase of technology development. DoDI 5000.02, Operation of the Adaptive Acquisition Framework defines the role and frequency of evaluation of technical and manufacturing risk throughout the acquisition life cycle. As such, MRLs and TRLs are both key metrics used to assess the program’s level of risk in meeting performance (i.e., TRLs) and production (i.e., MRLs) objectives. As MRL Assessments evaluate manufacturing risk, TRL assessments provide design maturity and aid in identifying new/unique/critical processes requiring further manufacturing development. Evaluating technical risk (e.g., through Independent

Technical Risk Assessments (ITRA)) provides evidence for the TRL determination within MRL Assessments.

Due to this interrelationship, the MRL criteria were designed to include an advised level of technology readiness to encourage manufacturing personnel to work closely with the technologist. NOTE: While there is a general relationship between MRLs and TRLs, there is no direct “one-to-one” requirement. While programs under the Major Capability

Acquisition pathway generally have longer development life cycles, Urgent Capability or

Middle Tier of Acquisitions have tighter timelines where design and manufacturing concerns have a greater impact on programmatic risk.

2.5.1 Best Practices for Assessing New or Novel Technologies

PMs/MDAs assess technological maturity, cost, schedule, and performance risks associated with delivering capability to the warfighter. The core of both TRLs and MRLs is demonstrating performance and manufacturability in increasingly relevant environments, respectively. Therefore, PMs balance risk between design stability and technological advancement, especially during the early acquisition phases.9 Programs

GAO-10-439: “An analysis of DoD’s technical reviews that assesses how programs are progressing show that MRLs address many gaps in core manufacturing-related areas, particularly during the early acquisition phases.”

with higher TRLs that haven’t considered manufacturability or later-stage programs with higher MRLs but unstable designs may have higher programmatic risk overall. Best practice has shown that programs are more likely to be successful when manufacturing design and producibility concerns are considered early within technology development to allow manufacturing maturity to be paced by technology maturity. Otherwise, manufacturing processes will not be able to mature without increased risk.10

As stated earlier, Critical Technologies (CT) are defined as technologies that are “new and novel or used in a new and novel way and impact system performance.” 11 PMs may accelerate technological development to correspond with the appropriate acquisition strategy and must evaluate programmatic risks to better understand the impact on achieving the program’s overall cost, schedule, and performance objectives.

Identifying gaps to achieve higher TRLs for CTs earlier in development allows the PM to effectively plan development strategies to reduce risk and is a “best practice” for achieving better cost and schedule outcomes. NOTE: This does not require being at a higher-level TRL; but suggests an objective “look forward” to ensure that the necessary resources, time, and funding are available for transition to the next appropriate phase.

Although technology may be inserted at any phase of the acquisition framework, many

CTs are developed in S&T (or R&D) environments. Additionally, early system developers are the least likely to consider manufacturing concerns based on historical reviews of programs. So, early technology development up to MRL 4 is a prime concern for MRL practitioners as product technology may have been developed and demonstrated in S&T as an actual system prototype within an operational mission environment. Design for manufacturability and “forward-looking” assessments seek to minimize programmatic risk early-on.

At Milestone B, 10 USC 2366B requires the MDA to certify that the technology in the program has been demonstrated in a relevant environment (TRL 6) on the basis of an independent review and assessment. USD(R&E) has authority and responsibility for

ITRAs, retaining Acquisition Category (ACAT) ID review and determining ITRA approval responsibility for ACAT IB/IC programs which may lie with a Service or Component.

During preparation for Milestone B, the program will be assessed against MRL 6 criteria; however, PMs should review MRL 7 criteria to identify any potential gaps and develop strategies in preparation of CDR. Transition from MRL 6 to MRL 7 is a significant step requiring a high level of technical and manufacturing advancement.

Inadequate considerations for design stability may result in manufacturing inferior products at scale that don’t meet warfighter needs or lack of flexibility in manufacturing

GAO-20-48G, pg 122.

Ibid, pg 47.

processes for critical technologies that may result increased cost or schedule delays that impact programmatic risk.

For Urgent Capability and Middle-Tier of Acquisitions pathways, higher TRLs and MRLs are required to meet rapid fielding requirements. CTs which have matured to TRL 5 (or beyond) by the Materiel Development Decision (MDD) become the “reasonable risk” candidates for consideration in the AoA during the Materiel Solution Analysis (MSA)

Phase. However, if manufacturing readiness has not been considered appropriately at this level, the potential risk for re-design delays can impact cost or schedule negatively.

Since the MDA is required to determine with a high degree of confidence that technology development during TMRR will not delay the fielding target of the program, PMs should evaluate CTs against MRL 4 criteria and analyze the technology development’s ability reach TRL 6 by Milestone B as a “best practice”.

As a manufacturing risk assessor, understanding the interplay between design for manufacturability concerns throughout the life cycle and advocating manufacturing risk is key to providing the best support to PMs/MDAs. Many MRL practitioners not only analyze and identify risk, they are critical to developing mitigation and maturation planning strategies.

3. MRLs in the Adaptive Acquisition Framework

3.1 Introduction

The Adaptive Acquisition Framework (AAF) was developed for Defense Acquisition with the objective of delivering timely, effective, suitable, survivable, sustainable, and affordable solutions to the end user. To achieve those objectives, Milestone Decision

Authorities (MDA), other Decision Authorities (DA), and PMs have broad authority to plan and manage their programs consistent with sound business practice. The AAF acquisition pathways provide opportunities for MDAs/DAs and PMs to develop acquisition strategies and employ acquisition processes that match the characteristics of the capability being acquired.12

DoDI 5000.02, Section 1, Operation of the Adaptive Acquisition Framework

Figure 3-1 – Adaptive Acquisition Framework

DoDI 5000.02, Operation of the Adaptive Acquisition Framework, has established multiple acquisition approaches with distinct, associated instructions:

Major Capability Acquisition (DoDI 5000.85)

Middle Tier of Acquisition (DoDI 5000.80) o Rapid Prototyping and Rapid Fielding procedures

Urgent Capability Acquisition (DoDI 5000.81)

Business Systems (DoDI 5000.75)

Defense Acquisition of Services (DoDI 5000.74)

In general, MRL Assessments are not required for Business Systems or Defense

Acquisition of Services, but are considered “best practice” for Major Capability

Acquisition, Middle Tier of Acquisition and Urgent Capability Acquisitions pathways.

Manufacturing risk management plays an integral part in the acquisition of all weapon systems throughout their entire life cycle. DoDI 5000.88, Engineering of Defense

Systems requires both Systems Engineering reviews and Independent Technical Risk

Assessments (ITRA). As defined in the DoD ITRA Framework, MRL Assessments can be used as a “best practice” to inform the ITRA.13 MRL Assessments may need to be adapted based on the program acquisition pathway (similar to MRL adaptations in

Section 8).

The following sections address MRL Assessments for Major Capability Acquisition, Middle Tier of Acquisition, and Urgent Capability Acquisition.

3.2 MRLs for Major Capability Acquisition Prior to the Materiel

Development Decision

3.2.1 Introduction

The MRL Criteria Matrix, Deskbook and Users Guide provide detailed guidance for application of MRLs in traditional DoD major systems acquisition. These criteria show a progression of manufacturing maturity as the program progresses through the acquisition lifecycle.

The Materiel Development Decision (MDD) is the mandatory entry point into the Major

Capability Acquisition process.14 It ends with the MDD decision to initiate a program of

“Department of Defense Independent Technical Risk Assessment Framework for Risk Categorization,” Under Secretary of Defense for Research and Engineering, June 2018, pg 1

DoDI 5000.85, Section 3.5 record (15) that is based upon the transition of mature technologies with manageable risk. Technology developed in Science and Technology (S&T) programs, procured from industry, or other sources entering the development process at Milestone A should be assessed as mature enough to transition smoothly (i.e., meet cost, schedule and performance requirements) into designs.

Figure 3-2 – MRLs in Major Capability Acquisition

Consideration of manufacturing risk and issues should begin early and intensify as product development proceeds so that manufacturing maturity is sufficient to support rapid and affordable development into a system. Some manufacturing-related “best practices” for development are:

Include manufacturing Subject Matter Experts (SME) in all Systems Engineering technical reviews

Perform an initial assessment in each phase to determine maturity based on the MRL criteria

Identify the target MRLs

Use the results of the initial assessment to set priorities and develop a Manufacturing Maturation Plan (MMP) that will achieve the target MRL or minimize risk to an acceptable level within schedule

Fund and execute to enable achieving the target MRL, or mitigate risks to an acceptable level within budget

Perform a final assessment of manufacturing readiness to identify all manufacturing risks prior to transitioning into the next phase

Program of record: An acquisition program that has been formally initiated by the Milestone Decision Authority and has been fully funded throughout the Future Years Defense Program.

Figure 3.3 below shows the relationship between MRLs, program phases, System

Engineering reviews, activities, decision points, and milestones.

Figure 3-3 – Major Capability Acquisition Details

3.2.2 Materiel Solution Analysis (MSA) Phase

The MDD marks the start of the MSA Phase. This presents the first substantial opportunity to influence systems design by balancing technology opportunities, schedule constraints, funding availability, system performance parameters, and manufacturing feasibility. The technical approach for system development should be driven by knowledge of the manufacturing maturity and risk of the various technologies under consideration as well as their associated performance maturity.

In this phase, the viable alternative is selected by conducting an AoA…