Bidders Library Operational Test and Evaluation - JITC OTE Guidebook v2 0.docx

JITC OT&E Guidebook Appendix I: Test Resources

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

JITC OPERATIONAL TEST

AND EVALUATION

GUIDEBOOK

VERSION 2.0

5 October 2017

(This page intentionally left blank.)

JITC OT&E Guidebook Introduction xiii Joint Interoperability Test Command (JTA)

IN REPLY

REFER TO:

MEMORANDUM FOR THE JOINT INTEROPERABILITY TEST

COMMAND WORKFORCE

SUBJECT: Joint Interoperability Test Command (JITC) Operational Test and Evaluation (OT&E) Guidebook Version (V) 2.0

1. The JITC OT&E Guidebook supports the Command in carrying out its role as a Department of Defense (DoD) Operational Test Agency. The guidebook provides JITC Operational Test Teams guidance on the essential concepts, best practices, and recommended standardized processes for planning, executing, and reporting upon test and evaluation activities performed for the purpose of evaluating operational effectiveness, suitability, and cybersecurity.

2. V2.0 supersedes all previously issued versions of the JITC OT&E Guidebook. Major changes from V1.5 to V2.0 of the guidebook include updated information to address DoD; Secretary of Defense, Director, Operational Test and Evaluation (DOT&E); and JITC policies and instructions applicable to the conduct of OT&E, current as of the issuance of this document. JITC will issue further updates to the guidebook as needed to account for any significant changes to DoD, DOT&E, or JITC policies or instructions pertaining to OT&E.

3. The current version of the guidebook is available at <https://disa.deps.mil/org/JTA/Guidebook/>.

4. Direct any questions or comments concerning this document, or requests for technical assistance in implementing the guidance it provides, to the JITC OT&E and Enterprise Services Division, Operational Evaluation Cell (OEC), by sending an email to <disa.huachuca.jt.list.jta-operational-evaluation-cell@mail.mil>.

5. The point of contact for this action is Mr. Terry Powell, OEC Team Lead, (520) 5385178, DSN 879-5178, or email Terry.M.Powell.civ@mail.mil.

DEFENSE INFORMATION SYSTEMS AGENCY

P. O. BOX 549

FORT MEADE, MARYLAND 20755-0549

Introduction Purpose

This guidebook introduces Joint Interoperability Test Command (JITC) personnel to the policies, processes, and products associated with the conduct of Operational Test and Evaluation (OT&E). The guidebook is intended to aid JITC Action Officers and Operational Test Teams (OTTs) in carrying out OT&E activities in a consistent, efficient, and thorough manner based on the proven processes, procedures, and techniques contained within this document.

References

The following references govern JITC's conduct of OT&E as a Department of Defense (DoD) Operational Test Agency (OTA) and are the basis for this guidebook:

· Title 10 United States Code, Section 139 and Section 2399, 3 January 2007

· DoD Directive 5000.01, The Defense Acquisition System, 12 May 2003

· DoD Directive 5141.02, Director of Operational Test and Evaluation (DOT&E), 2 February 2009

· DoD Directive 5105.19, Defense Information Systems Agency (DISA), 25 July 2006

· DoD Instruction 5000.02, Operation of the Defense Acquisition System, 7 January 2015, incorporating Change 2, 2 February 2017

· DoD Instruction 5000.74, Defense Acquisition of Services, 5 January 2016

· DoD Instruction 5000.75, Business Systems Requirements and Acquisition, 2 February 2017

· DISA Instruction 610-225-2, Acquisition Oversight and Management, 8 June 2017

· DISA Instruction 640-195-1, Test and Evaluation - Operational Test Agency Mission, 20 January 2017

· Supplement to the Memorandum of Agreement on Multi-Service Operational Test an Evaluation (MOT&E) and Operational Suitability Terminology and Definitions, 11 August 2017

Applicability

The policies, processes, and products referenced in this guidebook are applicable to JITC's conduct of OT&E performed in support of any:

· DoD Component, Service, Agency or other federal organization with which JITC has entered into an agreement to serve as their designated OTA

· Specific acquisition programs, systems, services, or capabilities, as requested and reimbursably funded by their Program Management Office or sponsor

The guidebook also extends to the OT&E support of DISA programs that JITC performs as DISA's designated OTA. In the case of DISA programs not on Office of the Secretary of Defense Oversight, the JITC Test and Evaluation (T&E) Process Guidebook for DISA Projects and the JITC T&E Scorecard Guide provide additional guidance for the reporting of project test status using a T&E Scorecard approach. Both documents are available at <https://disa.deps.mil/org/JTA/TE%20Scorecard/>.

FYI

JITC is DoD's Joint Interoperability Certifier and only non-Service Operational Test Agency As an OTA, JITC:

· Operational tests Information Technology and National Security Systems acquired by DISA, other DoD and non-DoD organizations

· Plans, executes, and reports on the evaluation of system Operational Effectiveness, Suitability, and Security (cyber) (OESS).

How To Use This Guidebook

The following symbols are used throughout the guidebook to bring the reader's attention to action items, industry best practices, and other useful information:

· Take Action: Actionable information

· Best Practices: Recommended processes

· FYI: Important information

· For Example: JITC-developed products and where to find them The guidebook is organized into chapters that describe the general processes and steps applicable to most systems for which JITC provides OT&E support. The first chapter provides an overview of OT&E. The second chapter explains how OT&E fits into the DoD Acquisition process. The remainder of the guidebook outlines the OT&E Planning, Execution, and Reporting cycle shown in Figure 1. The majority of OT&E activities occur during the planning phase. Although these planning activities are presented in a specific order within the guidebook, in practice many of these activities are interrelated and performed concurrently. For example, developing a system's analysis structure is likely to take place concurrent to identifying data collection methods, devising a data management strategy, and developing an overall test concept.

FYI

"System" The simplifying term "system" is used throughout this guidebook to refer to the information technology-based system, system-of-systems, services, or capabilities on which JITC is performing OT&E, as well as to the associated acquisition program to which JITC is providing OT&E support.

1 December 2:29 pm Figure 1. OT&E Planning, Execution, and Reporting Cycle

Guiding Principles for JITC OT&E

JITC Action Officers and OTTs should consider the following principles when applying the guidance provided in this document:

1. Be Proactive through Active and Early Engagement – Be proactive through early OTT involvement to fully understand the system being evaluated and the capabilities the system provides to users. 1

2. Encourage Communication and Collaboration – To ensure transparency and a sense of cooperation, coordinate frequently with all of the T&E stakeholders: DISA, DOT&E, Component Acquisition Executives, Program Managers, Developmental Testers, Interoperability Testers, Cybersecurity Testers, Operational Testers, and user community test participants.

3. Practice Integrated Testing – Where possible, encourage integrated test activities that efficiently support multiple T&E objectives.

4. Share and Utilize All Relevant Data – Capitalize on existing valid data sources to reduce the scope or dedicated OT&E phase duration required.

5. Focus on Mission Needs – Emphasize understanding the users of the system and the activities/missions they need to accomplish when using the system.

6. Target Areas of Highest Risk – To achieve "affordable confidence," adjust the depth and breadth of testing to focus resources on areas of highest operational risk.

7. Adopt Structured Analysis Approaches – Employ formal analysis structures, Science-Based Test Designs, and other established best test practices to execute OT&E in a thorough and defensible manner.

8. Be Objective and Analytical – Favor objective quantifiable test measures and criteria over subjective evaluation methods to provide credible and repeatable evidence to support acquisition and deployment decisions.

9. Instrument and Automate – Where cost effective, use instrumented and automated data collection to improve overall test effectiveness and efficiency.

10. Continuously Improve Processes – Strive for continuous process improvement by identifying and acting on opportunities to increase overall test effectiveness, efficiency, responsiveness, and quality of products and services. Hold after-action reviews so others can benefit from lessons learned.

Support and Technical Assistance

For questions, comments, or technical assistance with this guidebook or the JITC OT&E process in general, please contact the JITC OT&E and Enterprise Services Division, Operational Evaluation Cell (OEC), by sending an email to the OEC email distribution list at <disa.huachuca.jt.list.jta-operational-evaluation-cell@mail.mil>.

JITC OT&E Guidebook Introduction

Table of Contents Page

JITC OT&E Guidebook Table of Contents

Chapter 1 Operational Test and Evaluation Overview

1.1 Introduction

The Joint Interoperability Test Command (JITC) conducts Operational Test and Evaluation (OT&E) for the purpose of determining a system's Operational Effectiveness, Suitability, and Security (Cyber) (OESS). JITC conducts OT&E in support of system acquisition and deployment decisions. The scope of OT&E typically includes trained system users and support staff exercising operationally representative system configurations, capabilities, and operating procedures under mission-required operating conditions. During OT&E, JITC evaluates the system for:

· Effectiveness – Does the system enable users to accomplish required missions and functions?

· Suitability – Is the system sufficiently reliable, available, maintainable, and usable with provided training and support structure?

· Security (Cyber) – Does the system effectively operate in a secure manner, in its sustained operational environment, while providing the capability to detect and react to penetrations and exploitations, protect and restore data and information, continuously monitor emerging threats, and implement processes required for handling Personally Identifiable Information, Protected Health Information, and Financial Data?

Figure 1-1 provides a generalized overview of the OT&E process as briefly introduced in this chapter and discussed in greater detail in the chapters that follow. As shown in the figure, the OT&E planning phase is reliant on many concurrently performed interdependent activities. Programmatic requirements influence the scheduling of test events. Environmental requirements address factors within the system’s intended operating environment with the potential to affect OESS.

Take Action Early Operational Test Team (OTT) planning, coordination, and engagement is essential

· Interoperability (IOP) – The OTT should coordinate frequently with the IOP Team to ensure all Joint Interoperability Certification data requirements attributable to OT&E are accounted for:

· The OT&E strategy need address the evaluation of the Net-Ready Key Performance Parameter (NR KPP)

· The OTT should involve the IOP Team in analysis structure and OT&E strategy development, test planning, test readiness reporting, and data collection and analysis activities

· Cybersecurity – The OTT should coordinate often with the JITC Cyber Assessment Team (CSAT) to ensure all Cyber evaluation requirements and processes are addressed:

· The OT&E strategy need address the evaluation Cyber Survivability as a key element of the System Survivability KPP

· The OTT should work closely with the CSAT to ensure required resources and expertise such as Red Teams are available to perform cyber assessment activities when needed

LEGEND:

1 Dec 2:29 pm slide 2 "How much not addressed in guidebook Figure 1-1. Operational Test and Evaluation Process Overview

1.2 Identify Requirements

As part of determining a system's OESS, Operational Test Teams (OTTs) identify requirements and decompose applicable operational requirements. Associated with a system's operational requirements are the additional explicit and implied requirements pertaining to the environment and conditions in which the system must operate. Later, these requirements are used to develop an evaluation framework.

Chapter 4, Identify Operational Requirements, discusses the types of requirements and what documentation should be available as resources for acquiring operational requirements. Some of the more relevant documents are the:

· Initial Capabilities Document

· Capability Development Document

· Capability Production Document

· Information Support Plan

· Concept of Operations

· Systems Engineering Plan

· System Architecture in accordance with the Department of Defense (DoD) Architectural Framework

Once operational requirements are identified, the OTT defines the system's environmental requirements, the conditions under which requirements must be satisfied. This allows OTTs to incorporate a Science-Based Test Design (SBTD) into the planning. An SBTD approach provides a means to more effectively accomplish the OT&E mission of determining system OESS. Appendix D provides more detail regarding SBTD.

1.3 Develop Analysis Structure

The OT&E analysis structure is based on the operational requirements identified within the system's documentation. An analysis structure consists of two primary components:

· Evaluation Framework (EF) – The EF presents the topics for evaluation and "good" test measures that an OTT uses to arrive at an OESS determination

· Data Source Matrix (DSM) – The DSM provides the critical test planning details needed to execute the EF

Chapter 5 provides additional information regarding OT&E analysis structure development.

1.4 Formulate OT&E Strategy

The Test and Evaluation (T&E) Working-level Integrated Product Team (WIPT) coordinates and reflects the scope of the program's OT&E in the OT&E strategy. An effective OT&E strategy takes into consideration the data requirements associated with the OT&E analysis structure, the information needed to support the program's development and acquisition milestones, and the operational and technical risks inherent in delivering the required capabilities. Such a strategy may include the conduct of one or more Operational Assessments (OAs) to assess system maturity and readiness for an Initial Operational Test and Evaluation (IOT&E), and, if needed, Follow-on Operational Test and Evaluation. For system capabilities delivered through an incremental acquisition approach, following the conduct of an IOT&E, the T&E WIPT can adjust the levels of OT&E performed on subsequent system capabilities to account for the risks inherent in each incremental release. See Chapter 3 for more information regarding T&E WIPT activities.

The strategy developed by the T&E WIPT members should be comprehensive in addressing the various testing, cybersecurity activities, and artifacts required over the life-cycle of the system. Figure 1-2 shows notional artifacts, activities, artifacts, and timelines for a system undergoing T&E.

Figure 1-2. Notional Test and Evaluation Artifacts, Activities, and Timelines

1.5 Develop Test and Evaluation Planning Artifacts

The JITC OTT works with the T&E WIPT and other stakeholders in assessing a system. The T&E WIPT and the OTT develop documentation throughout the planning phase. Before test execution, the T&E WIPT and the OTT must complete test planning documents, such as the analysis structure and Test and Evaluation Master Plan (TEMP) (see Chapter 3). The OTT must also complete the Test Concept Brief and Test Plan (see Chapter 12).

1.6 Conduct Test and Collect Data

The test execution phase consists of conducting an initial Operational Test Readiness Review (OTRR) and a final OTRR before test execution. Before the formal test execution, the OTT may conduct a pilot test to verify test readiness and data collection methods. Once test execution begins, data is collected according to the approved test plan. OTRRs are further described in Chapter 13, Execute Test.

Data is generated and collected in a variety of ways. Test participants and users generate data for collection and analysis, using the system to perform real-world tasks to support their mission and carrying out test-induced operational scenarios, or user "free play." Data is also collected using instrumentation. Many times the data collected through the use of instrumentation is collated and analyzed to some degree, thereby reducing the amount of effort required by the OTT. Besides OAs and Operational Tests (OTs), testers may also generate and collect data during Table-Top Exercises and Rehearsal of Concept Drills. OTTs can use surveys and interviews to provide needed data, along with observations of demonstrations or users performing mission tasks.

Another means of acquiring relevant data may be through the review of Developmental Test and Evaluation (DT&E) data and other information sources. Chapter 10, Determine Data Collection Methods, discusses this topic in greater detail.

DT&E is often conducted in a simulated laboratory environment with the testers filling in for users and administrators. However, when DT&E is conducted in the intended fielded environment, the OTT can use the data collected to satisfy system requirements, particularly in support of Reliability, Availability, and Maintainability (RAM) requirements. Table 1-1 contrasts DT&E and OT&E.

Table 1-1. DT&E versus OT&E

· Verifies the system was built right

· Support decision to conduct OT&E Determine the extent the system supports user's operational needs:

· Validates that the right system was built

· Supports fielding decision

FYI

OT&E Oversight and Operational Test Agencies (OTAs)

· Title 10 United States Code Section 139 establishes the Director, Operational Test and Evaluation (DOT&E), a civilian appointed by the President, to perform duties to include:

· Prescribing OT&E policies and procedures within the DoD.

· Monitoring and reviewing all OT&E in DoD.

· Coordinating OT&E conducted jointly by more than one OTA; DOT&E typically appoints the lead OTA when two or more Services participate in the OT&E.

· The OTA is the agency empowered to conduct OT&E for a particular Service or Agency:

· Air Force Operational Test and Evaluation Center (AFOTEC).

· Army Test and Evaluation Command (ATEC).

· Marine Corps Operational Test and Evaluation Activity (MCOTEA).

· Commander, Operational Test and Evaluation Force (Navy) (COMOPTEVFOR).

· JITC serves as the OTA for the Defense Information Systems Agency and other select DoD Agencies through Memorandums of Agreement.

Best Practices OTT Involvement in DT&E The benefits of the OTT observing DT&E include:

· A better understanding of system design, architecture, implementation, maturity, and readiness for OT&E.

· An opportunity for collecting data to supplement OT&E data needs – particularly for RAM data.

1.7 Authenticate, Score, and Analyze Test Data

Part of the Execute phase activities include conducting a Data Authentication Group (DAG) to authenticate and score test data and its analysis to determine the extent test criteria are met. If the criteria are not met, the OTT must discuss the operational impact of the failure with the DAG. Chapter 14 addresses conducting a DAG.

FYI

Difference Between Test and Evaluation

· A Test is a structured activity performed under operationally representative conditions sufficient to allow for the collection of data needed to support specific evaluation objectives.

· An Evaluation is the process of logically assembling, analyzing, and comparing data to expected performance to aid in decision-making. It may involve review and analysis of qualitative and quantitative data obtained from design reviews, hardware inspections, Modeling and Simulation, hardware and software testing, measures review, and operational equipment usage.

1.8 Report Test Findings

In an OA, operational capabilities and limitations and/or the status toward achieving OESS are reported. The determination of OESS is made during an OT (IOT&E or Follow-on Operational Test and Evaluation) event. Both types of events and their results are reported in either a letter-form report or a formal report. In addition to a test report, a results briefing for presentation to DOT&E, JITC senior leadership, and/or other system stakeholders may also be required. These briefings emphasize the test results, analysis, and conclusions. Chapter 15 describes the expectations for both report formats.

Table 1-2 summaries the OT&E activities, the artifacts needed, the involved parties, the approximate time frame that the OT&E activities take place, and where more information can be found regarding those activities. The table presents the OT&E activities from a T&E perspective. DOT&E involvement is only needed for programs on Office of the Secretary of Defense oversight.

Table 1-2. JITC OT&E Activities, Artifacts, and Time Frames

· Coordinate with OEC

T-90 days Chapter 10

Table 1-2. JITC OT&E Activities, Artifacts, and Time Frames (continued)

· Pilot with SMEs/Users

Appendix F

Appendix C

· User Test Participants

Conduct OT Event (OA or IOT&E)

· JITC OTT

· User Test Participants "T Day" to T-End (Typically 1 to 3 weeks in duration) Chapter 13

Chapter 13

T+14 days Chapter 14

T+14 days Chapter 11

(if required)

· Pre-brief JITC Leadership

· Formal brief to PM and DOT&E T-End + 15 days Chapter 15

· Present to PM and DOT&E DAG-End + 30 days Chapter 15

+ 60 days Chapter 15

(Legend on next page.)

LEGEND:

1.9 For Further Reading

· Defense Acquisition University

· Defense Test and Evaluation Management Guide, 6th Edition, December 2012

· United States Air Force

· Air Force Test and Evaluation Management Guide, Sixth Edition, December 2012

· Air Force Instruction 99-103, Capabilities-Based Test and Evaluation, October 16, 2013

· United States Air Force Operational Test and Evaluation Center

· Test Director's Toolkit, 12 April 2010

· United States Army

· Army Regulation 73-1, Test and Evaluation Policy, 16 November 2016

· United States Marine Corps Operational Test and Evaluation Activity

· Operational Test and Evaluation Manual, 3rd Edition, 22 February 2013

· United States Navy Operational Test and Evaluation Force

· Operational Test Director's Manual, 26 July 2016

1.10 Summary

OT&E Overview

· JITC conducts OT&E for the purpose of determining a system's OESS.

· Most OT&E activities occur during the planning phase, and many of these activities are interrelated and performed concurrently.

· The OT&E planning process consists of:

· Identifying requirements

· Developing an analysis structure

· Formulating the OT&E strategy to include defining the type of testing required (OA or OT)

· Developing test and evaluation artifacts

· The OT&E execution process consists of:

· Conducting the test and collecting data

· Authenticating, scoring, and analyzing test data

· The last process consists of reporting the test findings.

JITC OT&E Guidebook Chapter 1 OT&E Overview

Chapter 2 Department of Defense Acquisition Process

2.1 Introduction

As discussed in Chapter 1, the primary reason the Department of Defense (DoD) performs Operational Test and Evaluation (OT&E) is to support DoD acquisition and deployment decisions. As such, evaluators need an understanding of the role of OT&E in the DoD acquisition process. The DoD acquisition process is implemented by DoD Instruction (DoDI) 5000.02 "Operation of the Defense Acquisition System," incorporating Change 2. This instruction provides the policies and principles that govern the Defense Acquisition System and forms the management foundation for DoD programs. It also identifies the specific statutory and regulatory documents and other information requirements for each Milestone (MS) Review and decision point. The DoD Acquisition System is an event-based process where a program goes through a series of processes, milestones, and reviews from beginning to end. Each milestone is the culmination of a phase where it is determined if a program will proceed to the next phase.

For systems certified as or declared as being a business capability, DoDI 5000.75 "Business Systems Requirements and Acquisition" governs the acquisition of these systems. Under DoDI 5000.75, Authority to Proceed (ATP) Decision Points replace the Milestones described in DoDI 5000.02. ATP decisions must still be documented through a formal memorandum and approval still rests with a Milestone Decision Authority (MDA). From a Test and Evaluation perspective, there are little to no changes from current test practices. Business systems are still required to undergo OT&E and the Director, Operational Test and Evaluation must still approve the test plans for programs on Office of the Secretary of Defense oversight.

This chapter provides an overview of the DoD acquisition process as it applies to DoD acquisition programs. The sections in this chapter address:

· The role of testing and Science-Based Test Design (SBTD) in the acquisition process

· The activities that occur during each acquisition phase, the exit criteria for proceeding to the next phase, and the key criteria for each milestone

· For business systems governed by DoDI 5000.75, the Business Capability Acquisition Cycle (BCAC) phases and ATPs

· The Acquisition Category (ACAT)/Business System Category levels and their criteria

· Action Officer roles and responsibilities in the DoD Acquisition process

Take Action Action Officer Responsibilities in the DoD Acquisition Process

· Be familiar with the DoD Acquisition process and the Joint Interoperability Test Command's (JITC's) roles and responsibilities in that process as an Operational Test Agency (OTA).

· Establish where the program is in the DoD Acquisition process to determine appropriate courses of action.

2.2 Overview

Figure 2-1 provides an overview of where testing and the activities leading up to testing, including the inclusion of an SBTD, fit in the DoD acquisition process described in DoDI 5000.02

Figure 2-1. Testing and Science-Based Test Design in the Acquisition Process

The left side of Figure 2-1 shows the Developmental Test and Evaluation (DT&E) process. The Program Manager (PM) is responsible for the ultimate success of DT&E and the Program Management Office (PMO) often conducts DT&E. The PMO may also engage an independent agency to conduct DT&E.

The DT&E test team begins by taking the system's capability requirements and using them to identify the system's technical capabilities. The DT&E test team identifies the decision support questions and developmental test objectives and the methodology needed to determine the extent the system has met the criteria for satisfying the Critical Technical Parameters (CTPs). Incorporated into DT&E (and OT&E) is an SBTD that identifies the various factors and their levels that may influence system performance. Appendix D provides more details regarding the use of an SBTD.

FYI

Critical Technical Parameters

· CTPs are measures that, when achieved, allow the attainment of a desired operational capability.

· CTPs are developed by the PMO.

Most DT&E occurs in the Technology Development Phase before MS B and in the Engineering Development and Manufacturing Phase before MS C. Section 2.3 more fully explains the acquisition phases and milestones.

OT&E follows a similar process as shown on the right side of Figure 2-1. The Operational Test Team (OTT) identifies the operational requirements based on the system's capability requirements. Using the system's Critical Operational Issues (COIs), the OTT develops the Measures of Effectiveness and Measures of Suitability needed to resolve the COIs while identifying the operational test (OT) objectives. Incorporated into the analysis structure are SBTDs that identify the various factors and their levels that may influence system effectiveness or suitability. OT&E typically occurs during the Production and Deployment Phase after MS C.

Best Practices Science-Based Test Design SBTD is best suited for DT&E where the factor and factor levels deemed applicable to the system can be explored during the system's engineering and development process. Only those factors and factor levels determined to significantly affect system performance should be included in the SBTD developed for OT&E.

Integrated testing is intended to result in resource efficiencies (time, money, people, and assets) and an enhanced data set for separate evaluations. The goal of integrated testing is to have DT&E and OT&E teams collaborate to produce credible qualitative and quantitative data useful to all evaluators, and to address developmental, sustainment, and operational issues. Integrated testing allows for the collaborative test event planning, where a single test point or mission can provide data to satisfy multiple objectives, without compromising the participating test organization's test objectives. Integrated testing focuses the entire test program (DT&E and OT&E) on designing, developing, and producing a comprehensive plan that coordinates all test activities to support evaluation results for decision makers at required decision reviews.

Figure 2-2 shows the acquisition model adopted for many incrementally deployed software-intensive systems, typically business systems not governed by DoDI 5000.75, where deployment of the full system will occur in multiple increments as the new system is developed. The major differences are these programs do not have an MS C, and OT&E occurs after MS B. The OT&E Planning, Execution, and Reporting cycle described in this guidebook still applies to such programs.

CDD

FD

FDD

Capability Development Document Full Deployment Full Deployment Decision

IOC

OT&E

RFP

Initial Operational Capability Operational Test and Evaluation Request For Proposal

Figure 2-2. Incrementally Deployed Software-intensive Program Acquisition Process

Each acquisition program falls into an ACAT (DoDI 5000.02) or DoD Business System Category (DoDI 5000.75) depending on its overall funding level and importance. The ACAT or Business System Category level determines the level of oversight a program will require. Overseeing each program is an MDA, who is, or is delegated by (1) the Defense Acquisition Executive (DAE), the individual responsible for supervising the DoD Acquisition System after the Secretary of Defense and the Deputy Secretary of Defense, or (2) a Component Acquisition Executive (CAE), an official within a DoD component that is responsible for all acquisition functions within that component. The MDA is the sole and final decision authority for that program.

The Under Secretary of Defense (Acquisition, Technology, and Logistics) (USD(AT&L)) Assistant Secretary of Defense for Acquisition Policy and Oversight[footnoteRef:1] is the DAE and the MDA for Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAISs). The MDAP and MAIS programs are the most expensive programs and have the most extensive statutory and regulatory reporting requirements. [1: The National Defense Authorization Act (NDAA) for Fiscal Year 2017 disestablishes the USD(AT&L)) and divides its existing duties among a new Undersecretary of Defense for Research and Engineering (USD(R&E)) and the renamed Under Secretary of Management and Support. To enable the USD(R&E) to focus on the innovation mission, the NDAA creates a new Assistant Secretary of Defense for Acquisition Policy and Oversight focused on setting defense-wide acquisition policy and overseeing the development of weapons and national security systems by military services. ]

Take Action

· Obtain a Copy of the Document That Describes the Capability Gap The capability gap is the preliminary source of the user's operational requirements and provides the foundation for the Initial Capabilities Document (ICD), Capability Development Document (CDD), and Capability Production Document (CPD). If possible, obtain a copy of a document such as the Analysis of Alternatives (AoA) to gain a better understanding of why users need the system.

· Establish an Operational Sponsor and/or User Representative Who Can Serve as a Point of Contact for the User Community Identify this person as early as possible in the acquisition process. This point of contact will be an invaluable resource for discerning user requirements, mission needs, and desired system performance characteristics. Always include the PMO in any discussions with the user community.

2.3 DoDI 5000.02 Acquisition Phases and Milestones

2.3.1 Materiel Solution Analysis Phase

The Materiel Solution Analysis (MSA) Phase assesses potential solutions for a needed capability in an ICD[footnoteRef:2]. Another objective of the MSA Phase is to satisfy the phase-specific entrance criteria for the next program milestone designated by the MDA. The MSA Phase is critical to program success and achieving materiel readiness because it is the first opportunity to influence system supportability and affordability by balancing technology opportunities with operational and sustainment requirements. The ICD and the AoA Study Plan guide the AoA and MSA Phase activity. [2: Or an equivalent document. This equivalency applies to all documents described in this chapter. ]

The MSA Phase also includes identifying and evaluating affordable product support alternatives with their associated requirements to meet the operational requirements and associated risks. Consequently, when describing the desired performance to meet mission requirements, sustainment metrics should be defined in addition to the traditional performance design criteria.

While an OTA seldom has any responsibilities during the MSA Phase, it is during this phase the CAE selects a PM and establishes a PMO. The PM is the designated individual with responsibility for and authority to accomplish program objectives for the development, production, and sustainment to meet the user's operational needs.

The main task the PM has during this phase is to complete and submit the Acquisition Strategy. The AoA assesses potential cost-effective materiel solutions that could satisfy validated capability requirements documented in the ICD. The approved Acquisition Strategy describes the overall approach to acquiring the capability, including the program schedule, risks, funding, and business strategy.

Another important role the PM has is to develop a draft CDD. The PM compares user capabilities with technologies to determine feasibility and alternatives to fill user needs. Once the requirements have been identified, the PM should perform a gap analysis to determine the additional capabilities required to implement the concept.

The MSA Phase ends when:

· The AoA has been completed.

· The lead DoD Component conducting the AoA recommends the materiel solution options identified in the approved ICD.

· The phase-specific entrance criteria for the initial milestone review have been satisfied.

2.3.2 Milestone A

The goal of MS A is to determine if a program has met all its exit requirements of the MSA Phase to proceed into the Technology Maturation and Risk Reduction (TMRR) Phase. The MDA determines the exit criteria based on the ACAT level. A few of the common requirements are:

· AoA written in accordance with approved AoA Study Guide and AoA Study Plan

· Acquisition…