Bidders Library Cybersecurity - JITC Cyber TE Guidebook.pdf

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

JITC CYBER TEST AND

EVALUATION GUIDEBOOK

VERSION 1.02

September 2019

(This page intentionally left blank.)

DEFENSE INFORMATION SYSTEMS AGENCY

P. O. BOX 549

FORT MEADE, MARYLAND 20755-0549

IN REPLY

REFER TO: Joint Interoperability Test Command (JTA/JTD) September 2019

ATTN: CYBERSECURITY ANALYST SUPPORTING DEVELOPMENTAL

AND OPERATIONAL TEST AND EVALUATION

SUBJECT: Joint Interoperability Test Command (JITC) Cyber Test and Evaluation Guidebook

Version 1.02 – Integrated Cyber Test and Evaluation (ICT&E) Approach for

Cybersecurity, Resilience, and Survivability Assessments

References: (a) JITC "OT&E Guidebook Version 2.0", 5 October 2017

(b) Director, Operational Test and Evaluation (DOT&E) memorandum

"Procedures for Operational Test and Evaluation of Cybersecurity in

Acquisition Programs", 3 April 2018

(c) Chairman of the Joint Chiefs of Staff (CJCS), "Cyber Survivability

Endorsement Implementation Guide (CSEIG) Version 1.01"

(d) Department of Defense (DoD) "Cybersecurity Test and Evaluation (T&E)

Guidebook Version 2.0", 25 April 2018

1. Introduction. This guidebook replaces the current Appendix C of the JITC OT&E Guidebook, reference (a) addressing Cybersecurity Assessments. This guidebook and the supporting appendices address JITC’s implementation of ICT&E testing of Cybersecurity, Survivability, and Resiliency as directed by references (b), (c), and (d). The appendices contain specific processes and procedures replacing the existing process and procedures detailed in reference (a) and are addressed throughout the acquisition life cycle. Because the references (b), (c), and (d) are being updated on a constant basis, the Operational Evaluation Cell (OEC) will update the appendices rather than the overarching process detailed in this guidebook. The only time the process will be updated is in the event of a drastic change in the overarching references. JITC will follow this process and procedures when supporting any customers using JITC services as requested.

2. Objective. This process and procedures outline a repeatable, cost effective, and consistent process for ICT&E implementation. This process and procedures are in response to new requirements and policies that went into effect in April 2018 (references b, c, and d above). All personnel implementing this process shall have a working knowledge of all listed references and be trained in implementing the approach.

3. Summary. This process and procedures will be followed for each event unless otherwise directed by the OEC ICT&E Subject Matter Expert.

JITC Memo, JTA/JTD, Cyber Test and Evaluation (CTE) Guidebook Version 1.02 – Integrated CTE Approach for Cybersecurity, Resilience, and Survivability Assessments, September 2019

4. The JITC point of contact for this memorandum is Mr. Don Tucker, DSN 879-2612 or commercial (520) 538-2612. E-mail address is don.h.tucker2.civ@mail.mil or don.h.tucker2.civ@mail.smil.mil. Mailing address is PO Box 12798, Fort Huachuca, Arizona

85670-2798.

Enclosure: SHAWN ROBERTS

Integrated Cybersecurity Test CAPTAIN, USN and Evaluation (ICT&E) for Commander

Cybersecurity, Survivability, and Resilience Assessments

Distribution:

JITC

DBC

JITC Cyber Test and Evaluation Guidebook Version 1.02

TABLE OF CONTENTS

Page

INTRODUCTION

POLICY AND GOVERNANCE

ROLES AND RESPONSIBILITIES

INTEGRATED CYBERSECURITY TEST AND EVALUATION APPROACH

LIST OF FIGURES

1. RMF and T&E Crosswalk

2. Integrated Cybersecurity/Survivability Approach

3. Cybersecurity Framework Integration and Intended Results

4. Cybersecurity/Survivability Risk Category Determination

5. Characterize the Attack Surface Activities

6. CMEF Scorecard Example

7. Cooperative Vulnerability Identification Activities

8. CEVA Workstream Overview

9. Cyber Survivability Adversarial Assessment Overlay

10. Cyber Survivability Reality

LIST OF TABLES

1. Cyber Survivability Attributes

2. Cyber Survivability Evaluation and Status Determination Criteria

3. Adversarial Assessment Cyber Survivability Measures

(This page intentionally left blank.)

INTRODUCTION

This Joint Interoperability Test Command (JITC) Cyber Test and Evaluation Guidebook Version 1.02 replaces the current Appendix C of the JITC Operational Test and Evaluation (OT&E) Guidebook Version 2.0. This guidebook and the supporting appendices address JITC’s implementation of Integrated Cyber Test and Evaluation (ICT&E) of Cybersecurity, Cyber Survivability, and Cyber Resiliency for any type of acquisition lifecycle. In order to accomplish this, the addition of the Cybersecurity, Systems Engineering (SE) and Developmental Test and Evaluation (DT&E) processes augment the existing Cyber Survivability OT&E assessment processes. The ultimate goal of this assessment approach is to ensure cyber is addressed throughout the System/Application/Capability's (SACs) life cycle. When properly implemented, this process improves a program’s Cybersecurity, Cyber Survivability, and Cyber Resiliency by reducing risk to the deployment, reducing costs, and enabling agility.

Understanding the terminology for these two distinct, but integrated assessment efforts is very important. Cyber Test Events (CTE) refers to all combined Cybersecurity, SE, Developmental Testing (DT), and Operational Testing events spanning the totality of Cybersecurity, Cyber Survivability, and Cyber Resiliency. The term “Cybersecurity” is primarily used for SE and DT phases while the terms “Cyber Survivability and Cyber Resiliency” refers to the Cyber Survivability test efforts conducted during the Operational Test (OT) phases. This updated version avoids restating policy. Instead, it encourages the reader to go directly to policy source documents for more information and understanding.

This guidebook and the appendices contain specific processes and procedures replacing existing process and procedures detailed in reference (a) and are addressed throughout the acquisition life cycle. Because the references (b), (c), and (d) are being updated on a constant basis, the Operational Evaluation Cell (OEC) will update the appendices rather than the overarching process detailed in this appendix. The only time the process will be updated is in the event of a drastic policy change in the overarching references. JITC will follow this process and supporting procedures when supporting any customers using JITC services.

A visual representation of the ICT&E approach is shown in Figure 2. The ICT&E process uses a master test plan template that covers all CTE. Rather than recreate the information in each test plan, the test plan template will refer to the appendixes listed in this guide for the corresponding event. The overall guidebook provides an overview of each CTE JITC supports. Details of these events are outlined in the appendices below:

Appendix A – Acronyms

Appendix B – Cybersecurity Evaluation Framework (includes Cyber Master Evaluation Framework (CMEF) Requirements Traceability, Scorecard, and Data Source Matrix (DSM))

Appendix C – Cybersecurity DT (related to phases 1-4 shown in Figure 2)

Appendix D – Cyber Survivability OT (related to phases 5-6 shown in Figure 2)

Appendix E – Penetration Testing

Appendix F – Cloud testing

Appendix G – Test and Evaluation Strategy Brief (TESB), Request For Information (RFI), and Test and Evaluation Master Plan (TEMP)

Appendix H – References

Appendix I – Points of Contact (POCs)

POLICY AND GOVERNANCE

The processes, procedures, and methodology outlined in this appendix are primarily based on the following references. The actual version and date of each reference is located in Appendix H due to these references being frequently updated.

JITC “OT&E Guidebook Version”

Director, Operational Test and Evaluation (DOT&E) memorandum, “Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs”

Department of Defense (DoD) “Cybersecurity Test and Evaluation (T&E) Guidebook”

DoD Instruction (DoDI) 8510.01, “Risk Management Framework (RMF) for DoD Information Technology (IT)”

DoD Instruction (DoDI) 5000.02, "Operation of the Defense Acquisition System"

DoDI 5000.75, “Business Systems Requirements and Acquisition"

Chairman of the Joint Chiefs of Staff (CJCS), "Cyber Survivability Endorsement Implementation Guide (CSEIG)"

Scaled Agile Framework (SAFe) Version 4.6

Fiscal Year (FY) 2016 National Defense Authorization Act (NDAA) Section 804 Middle Tier Acquisition (MTA)

National Institute of Standards and Technology (NIST) Special Publication

(sp) sp800-53 Revision (r) 4, "Security and Privacy Controls for Federal Information Systems and Organizations," April 2013 and the Draft sp800-53r5 dated August 2017

Personnel implementing this assessment approach must have a working knowledge of these policies identified in Appendix H in order to understand the processes described in this appendix.

ROLES AND RESPONSIBILITIES

This integrated assessment approach requires support and cooperation from a variety of personnel and resources across different roles in numerous organizations. An overview of the personnel and team roles and responsibilities related to CTEs are defined below. A familiarity with these roles will enable better understanding of the process defined later.

Authorizing Official (AO) – Responsible for authorizing the system’s operation based on achieving and maintaining an acceptable risk posture.

(Reference: DoDI 8510.01 RMF)

Chief Developmental Tester – The Chief Developmental Tester (CDT) is responsible for coordination of the planning, management, and oversight of all DT&E activities for the program; maintaining insight into contractor activities and overseeing the T&E activities including Cybersecurity; and helping Program Managers (PMs) make technically informed, objective judgments about contractor DT&E results. (Reference: Title 10 United States Code [U.S.C] 139b)

Chief Engineer/Lead Systems Engineer – Acts as lead engineer for entire system; responsible for engineering analysis and trades made at the system level; works with system security engineer on integrating security into overall engineering efforts. (Reference: Defense Acquisition Guidebook, Chapter 4)

Cybersecurity DT&E Lead and Subject Matter Expert (SME) – The Cybersecurity DT&E Lead SME is appointed by and reports to the Chief Developmental Tester and directs all Cybersecurity DT&E activities contained in the procedures and methodology. The Cybersecurity DT&E Lead SME directs and leads all team leads described below. The Cybersecurity DT&E Lead SME is responsible for ensuring all deliverables required for DT&E Cybersecurity activities are completed and delivered to the customer. (DoDI 5134.17, Deputy Assistant Secretary of Defense for Developmental Test and Evaluation.

Deputy Assistant Secretary of Defense for Developmental Test and Evaluation –The Office of the Deputy Assistant Secretary of Defense (SecDef) for Developmental Test and Evaluation (DASD(DT&E)) was established within the Office of the Assistant SecDef, Research and Engineering effective June 23, 2009. The DASD DT&E, serves as the principal advisor to the Office of the Secretary of Defense (OSD) and the Under SecDef for Acquisition, Technology and Logistics in matters relating to DT&E in defense acquisition programs. The Weapon Systems Acquisition Reform Act of 2009, which amends Title 10, U.S.C (now formally known as Title 10, U.S.C, Section 139d), assigns the DASD DT&E authority to assess system performance across Major Defense Acquisition Programs including:

o Program Oversight o Policy and Guidance o TEMP approval o Advocacy for Acquisition DT&E workforce o Component T&E Capability o Annual Report to Congress http://www.acq.osd.mil/chieftechnologist/index.html http://www.acq.osd.mil/chieftechnologist/index.html

DT&E is also responsible for evaluating DT&E capability within the Department of

Defense. This increased authority acknowledges the critical role of DT&E in systems acquisition and highlights the need to emphasize DT&E much earlier in the acquisition life cycle. The DASD DT&E is now a collaborative partner with DOT&E to ensure acquisition decisions are supported with the right T&E information. The DASD DT&E also will partner with the OSD acquisition organizations and Component Acquisition

Executives to ensure each acquisition program is supported by an appropriate DT&E strategy. DASD DT&E is also the Director, Test Resource Management Center

(TRMC). The TRMC team is dedicated to ensuring the Components have the right T&E

Infrastructure to accomplish the T&E.

Defense Intelligence Agency Threat Analysis Center – Utilizes intelligence and counterintelligence to assess risks that may be introduced intentionally or unintentionally by a particular supplier and provides standardized all-source intelligence assessments to inform program management and support acquisition risk management efforts. (Reference: Defense Acquisition Guidebook, Chapter 9)

Developer – Role may be performed in-house, by another government entity, or by a contractor/system integrator. The developer should understand relevant threats and be able to assess mission needs and capability gaps against likely adversary threat capabilities. Development will be conducted in accordance with security controls related to assurance, system development, and security best practices to reduce vulnerabilities and to design, build, and test security in the system early and cost effectively. (Reference: DoDI 5000.02)

DoD Component Chief Information Officer (CIO) – Responsible for administration of the RMF within the DoD Component ICT&E program;

participation in the RMF Technical Advisory Group (TAG) visibility and sharing of the RMF status of assigned Information System (IS) and Platform Information Technology (PIT) systems; and enforcement of training requirements for persons participating in the RMF. (Reference: DoDI 8510.01)

Director, Operational Test and Evaluation – DOT&E is the principal staff assistant and senior advisor to the SecDef on OT&E in the DoD. DOT&E is responsible for issuing DoD OT&E policy and procedures; reviewing and analyzing the results of OT&E conducted for each major DoD acquisition program; providing independent assessments to SecDef, DASD DT&E, and Congress making budgetary and financial recommendations to the SecDef regarding OT&E; and oversight to ensure OT&E for major DoD acquisition programs is adequate to confirm Operational Effectiveness, Suitability, and Cyber Survivability of the defense system in use.

Information Owner (IO) – Acts as statutory or operational authority for specified information; responsible for establishing the controls for data generation, classification, collection, processing, dissemination, and disposal.

(Reference: DoDI 8510.01/ Committee on National Security Systems Instruction (CNSSI) 4009)

Information System Security Officer (ISSO) – Responsible for maintaining the appropriate operational security posture for an information system or program.

Information System Security Manager (ISSM) – Responsible for ensuring all products, services, and PIT have completed the appropriate evaluation and configuration processes prior to incorporation into or connection to an IS or PIT system.

JITC Cyber Survivability Assessment Team (CSAT) SME – The CSAT SME is responsible for conducting all Cyber Survivability operational assessment or testing activities after the system is deployed in its intended operational environment. The CSAT SME is a member of the Cyber Working Group (CyWG). (Reference: JITC ICT&E Guidebook)

Joint Forces Headquarters/Department of Defense Information Network (JFHQ DoDIN) J2 – United States (US) Cyber Command (USCYBERCOM) is composed of several service components who will provide Joint services to USCYBERCOM. The JFHQ DoDIN, as the execution component, leads efforts to plan, coordinate, integrate, synchronize and conduct activities to:

direct the operations and defense of specified DoDIN and; prepare to, and when directed, conduct full spectrum military cyberspace operations in order to enable actions in all domains, ensure US/Allied freedom of action in cyberspace and deny the same to our adversaries. The J2 element of JFHQ DoDIN provides accurate and active threat Cybersecurity intelligence on adversaries to assist with attack vector development.

Joint Staff’s Functional Capability Board (FCB) – DoD body that is responsible for the organization, analysis, and prioritization of joint warfighting capabilities within an assigned functional area. (Reference: Joint Capabilities Integration and Development System (JCIDS) Manual)

Joint Requirements Oversight Council (JROC)/DoD Component Requirements Authority – Identifies and assesses the priority of joint military requirements to meet the national military and defense strategies, and considers alternatives to any acquisition program that has been identified to meet military capabilities by evaluating the cost, schedule, and performance criteria of the program and of the identified alternatives. (Reference:

Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 5123.01G)

Milestone Decision Authority (MDA) – Final decision authority. Approves entry of an acquisition program into each phase of the acquisition process and ensures programs are structured and resourced to succeed. (Reference:

DoD Directive (DoDD) 5000.01/DoDI 5000.02)

Operational Test Agency (OTA) – Conducts a comprehensive Operational Test for the system per Title 10. Cyber Survivability assessments are conducted as part of Operational Testing in an operational environment to determine readiness for the Cyber Operational Resiliency Evaluation.

(Reference: DoDD 5141.02)

Operational Test Agency Lead – OTA lead person responsible for overseeing all combined test events supporting operational testing.

Program Executive Office (PEO) – Responsible for executive management of assigned programs. Supervises design of acquisition programs, preparation of programs for decisions, and execution of approved program plans. (Reference: DoDI 5000.02)

Program Manager (PM) / System Manager (SM) – Responsible for ensuring the program meets statutory and regulatory requirements for ICT&E, and for incorporating ICT&E requirements into the program from conceptual development through design and sustainment/disposal. (Reference: DoDI 5000.02)

Requirements Sponsor – Responsible for all capability requirements documentation (Initial Capabilities Document (ICD), Capability Development Document (CDD), Capability Production Document (CPD), and Joint Doctrine, Organization, Training, materiel, Leadership and education, Personnel, and Facilities (DOTmLPF-P) Joint Document Change Recommendations (DCRs)), periodic reporting, and funding actions required to support the capabilities development and acquisition process for a specific capability proposal.

(Reference: CJCSI 3170I)

Software Code and Analysis Team (SCAT) – The SCAT is responsible for running the code inspection tools on the system, capability, or application in support the DT&E. This is not the same as the Security Control Assessor (SCA) for RMF implementation. (Reference: JITC ICT&E Guidebook)

Security Control Assessor (SCA) – Develops the Security Assessment Plan and ensures decomposed component security specifications, including verification criteria, are fully defined and traced to the controls delineated in the Security Plan.

Systems Security Engineering – Provides the expertise needed for effective integration of security, including Cybersecurity, into the design and development of the system throughout the acquisition lifecycle. (Reference:

Defense Acquisition Guidebook)

User Representative – Defines the system’s operational and functional requirements, and is responsible for ensuring that user operational interests are met throughout the system’s authorization process. (Reference: DoDI

8510.01/CNSSI 4009)

Vulnerability Assessment Team (VAT) – The VAT is responsible for conducting all test activities related to the vulnerability assessment of the system, capability, or application in support the DT&E. This is synonymous with the term “Blue Team” in the DoD Cybersecurity Test and Evaluation Guidebook. This is not the same as the SCA RMF implementation. The vulnerability assessment is conducted from inside the system, capability, or application outward to the primary defenses. (Reference: JITC ICT&E Guidebook)

INTEGRATED CYBERSECURITY TEST AND EVALUATION APPROACH

The ICT&E approach includes all Cybersecurity, SE, DT, and Cyber Survivability

OT test activities as outlined in the references above as shown in Figure 2. This approach does not replace the test activities that have previously been conducted during those phases as outlined in Figure 1, rather it integrates all efforts and streamlines the assessment process. This approach maps to the RMF 6 step process, the DoD T&E Guidebook 6 step process, and the DOT&E guidance for operational

Cyber Survivability testing.

LEGEND:

ACD Adversarial Cybersecurity Developmental Test and Evaluation DT&E Developmental Test and Evaluation ATO Authority to Operate DTPT Developmental Test Penetartion Test CEVA Cyber Economics Vulnerability Assessment FedRAMP Federal Risk and Authorization Management Program CTT Cybersecurity Table Top RMF Risk Management Framework CVI Cooperative Vulnerability Identification OT Operational Test CVPA Cooperative Vulnerability and Penetration Assessment OT&E Operational Test and Evaluation DT Developmental Test T&E Test and Evaluation

Figure 1. RMF and T&E Crosswalk

AA Adverserial Assessment DOT&E Director, Operational Test and Evaluation ACD Adverserial Cybersecurity DT&E DT Developmental Test AoA Analysis of Alternatives DT&E Developmental Test and Evaluation ATO Autority to Operate FedRAMP Federal Risk and Authorization Management Program CS Cybersecurity KPP Key Performance Parameter CEVA Cybersecurity Economic Vulnerability Assessment MSA Materiel Solution Analysis CNSS Committee on National Security Systems OT Operational Test CSA Cyber Survivability Attribute OT&E Operational Test and Evaluation CTT Cyber Table Top PEO Prpgram Executive Office CVPA Cooperative Vulnerability and Penetration Assessment PM Program Manager CyWG Cybersecurity Working Group SCA Security Control Assessment DASD Deputy Assistant Secretary of Defense TSN Trusted Systems and Networks

Figure 2. Integrated Cybersecurity/Survivability Approach

Section 2 Section 1

Section 3

Section 4

Section

Section

Section

Section 7 Section 8

The sections below explain the process outlined in Figure 2 above. Summaries are provided only for the areas that JITC provides information support to and no actual

CTE support is required. Appendices are provided for the sections where JITC provides support to the CTE. The blocks covered under each section are indicated by the green blocks shown on Figure 2.

Section 1. Materiel Solution Analysis, Analysis of Alternatives, and Control

Assignment

The Materiel Solution Analysis (MSA), Analysis of Alternatives (AoA), and

Control Assignment is shown in the first gray box of Figure 2. Protection of the program starts during this phase of acquisition which includes any Request for Proposal efforts.

Program protection provides the processes, methodologies, and techniques to enable program offices to identify information, components, and technologies, as well as determine the most appropriate mix of measures to protect the information, components, and technologies from known security threats and attacks. These protection measures affect the development of the system being acquired, the operations of the program office, and the means by which the items are acquired. The

Program Management Office's (PMOs) analysis from this phase drive the development of the Systems Engineering Plan (SEP) and the Program Protection Plan (PPP) which outline requirements and parameters for the system. These requirements are binned or expressed at their highest level within the defined Technical Performance Measures

(TPM) categories. These requirements for the protection are further refined into Critical

Technical Parameters (CTPs). CTPs test metrics used to evaluate the mission-critical functions and components during the SE phase of acquisition shown in the blue boxes on Figure 2. This is a critical part that defines the basis for the Attack Surface and is updated during the system lifecycle. The PMO is responsible for creating the TPMs and

CTPs in accordance with the DoD T&E Guidebook.

The CNSSI 1253, "Security Categorization and Control Selection for National

Security Systems," 15 March 2012, outlines the process for identifying the NIST security controls by the potential impact to the mission, due to loss or degradation of the

Confidentiality, Availability, and Integrity (CAI). This categorization allows the PMO to link the TPMs and CTPs to the initial baseline controls overlays. Overlays can be considered as an initial “bulk tailoring” activity, but system-specific tailoring of controls is required for all systems. The Cybersecurity threats and vulnerabilities constantly change; therefore, tailoring must continue throughout the lifecycle. The PM achieves this tailoring by:

Coordinating the initial security control set with the SCA, preparing the SEP, PPP, and other pertinent Cybersecurity documentation.

Deriving TPMs and CTPs for the MS A based on the draft CDD, Concept of Operations (CONOPS), architectures and data flows, initial baseline controls after overlays are applied, and other stakeholder requirements.

Providing Cybersecurity input for the draft system performance specification, along with the statement of work, Contract Documents Requirements List (CDRL), and source selection criteria, which are key sections of the Technology Maturation and Risk Reduction (TMRR) RFP at MS A.

Note: The MSA and AoA include the validation of approved products for a stable Cybersecurity platform and baseline. The development must start out using approved products and appliances. This requires using products listed on the DoD Approved Products List (APL), National Information Assurance Partnership (NIAP), the Unified Capabilities (UC) APL, and includes Supply Chain Risk Management (SCRM) that meets the levels defined for CAI and data to be processed.

Test and Evaluation Master Plan (TEMP) Development – For systems following the DoDI 5000.02 or DoDI 5000.75 acquisition policy, the PMO must publish a TEMP prior to MS A and update for each subsequent milestone decision. The TEMP guidance remains very fluent at the time of this guidebook publication. DASD DT&E and DOT&E guidance on the criteria, size, and information required for the TEMPs is still in flux and changes between programs. Appendix G provides JITC guidance, templates, and examples. For systems following the MTA approach, a Test and Evaluation Strategy Brief (TESB) is still required. Although this document is less formal than a TEMP, it will still outline how ICT&E will inform deployment decisions.

Note: The TEMP should discuss ICT&E and any additional test organization(s).

In Progress Reviews (IPRs) – IPRs will be required once the SAC is assigned to a JITC Action Officer. The IPR will provide leadership with an overview of the SAC's progression towards testing and any impediments to the overall objectives and goals.

Cyber input will be provided to the slides provided by the JITC Action Officer.

Section 2. Trusted Systems and Networks, Federal Risk and Authorization

Management Program, Cybersecurity, System Survivability Key Performance

Parameter Pillars, and Cyber Survivability Attributes

The Trusted Systems and Networks (TSN) analysis, Federal Risk and

Authorization Management Program (FedRAMP) for cloud computing approval authority implementation, System Survivability-Key Performance Parameter (SS-KPP) Pillars, CTPs linkage, and Cyber Survivability Attributes (CSAs) assignment.

Trusted Systems and Networks – TSN as defined in the DoDI 5000.02:

PMs that have a high impact level for any of the three security objectives, CAI; or other DoD information systems that the Component Acquisition Executive (CAE) or Component CIO determines to be critical to the direct fulfillment of military or intelligence missions must identify and protect mission critical functions and components as required by DoDI 5200.44.

TSN plans and implementation activities are documented in PPPs as well as other relevant Cybersecurity plans and documentation. PMs will manage TSN risk by:

a. Conducting a criticality analysis of the mission critical functions and critical components defined during the MSA/AoA phase and reducing vulnerabilities to system functions and components through secure system design.

b. Requesting updated threat analysis based on the critical components.

c. Applying best practices, processes, techniques, and procurement tools prior to the acquisition of critical components or their integration into the system.

Federal Risk and Authorization Management Program – FedRAMP is an assessment and authorization process U.S. federal agencies have been directed to use by the Office of Management and Budget (OMB) to ensure security is in place when accessing cloud computing products and services. Each FedRAMP assessment must be performed by an accredited Third Party Assessment Organization (3PAO). The Joint Authorization Board (JAB) is responsible for establishing accreditation standards. The 3PAO performs the security assessments of cloud solutions. The JAB reviews authorization packages (that include the results from the 3PAO's assessments), and may grant provisional authorization (to operate). The federal agency consuming the service still has final responsibility for final Authority to Operate (ATO). Data collected and consumed by the 3PAO is in the same NIST format required for input into the CMEF detailed in Appendix B.

System Survivability Key Performance Parameters and Cyber Survivability Attributes – The Cyber Survivability Endorsement (CSE) is outlined in the Joint Chiefs of Staff Cyber Survivability Endorsement Guide. This process integrates multiple DoD cyber efforts and builds upon the TPM and CTP metrics and refines the operational implementation. The CSE is mandated for SACs under the JROC. Per the DoD CIO, any SAC that is not under JROC must have a capability that addresses or exceeds the CSE process. The CSAs are defined Cyber Survivability requirements for SACs as well as their supporting infrastructure. The DoD RMF provides information security measures and controls for identification, assessment, and management of cybersecurity risk. CSE integrates with these frameworks and provides implementable system capability requirements and security considerations in order to produce risk-managed measureable, testable and implementable system attributes to manage cyber risks under realistic threat to survive the mission. Figure 3 illustrates the CSE Cybersecurity Framework Integration and Intended Results.

DoDI Department Of Defense Instruction KPP Key Performance Parameter JCIDS Joint Capabilities Integration and Development System NIST National Institute of Standards and Technology

Figure 3. Cybersecurity Framework Integration and Intended Results

The cyber requirements are measured based on the system’s ability to Prevent, Mitigate, and Recover from the effects of cyber-attacks. The requirements are based on the following mandatory pillars for cyber-attack effects:

Prevent – design principles that protect system’s mission functions from most likely cyber threats

Mitigate – design principles to detect and respond to cyber-attacks; enable the mission system to survive attacks and complete the mission

Recover – design principles to enable recovery from cyber-attacks and prepare mission systems for the next fight

The ten CSAs are traceable to the SS-KPP mandatory pillars (Table 1) and can be applied as performance measure requirements to ensure the SAC is cyber survivable and cyber resilient in its intended operational environment.

Table 1. Cyber Survivability Attributes

SS KPP Pillars Cyber Survivability Attributes Prevent CSA-01 - Control Access

C S

A -1

- A c tiv e ly

M a n a g e

S y s te m C o n fig u ra tio n s to

C o u n te r V u ln e ra b ilitie s a t

T a c tic a lly

R e le v a n t S p e e d

CSA-02 - Reduce System's Cyber Detectability

CSA-03 - Secure Transmissions and Communications

CSA-04 - Protect System's Information from Exploitation

CSA-05 - Partition and Ensure Critical Functions at Mission Completion Performance Levels

CSA-06 - Minimize and Harden Attack Surfaces

Mitigate CSA-07 - Baseline & Monitor Systems and Detect Anomalies

CSA-08 - Manage System Performance if Degraded by Cyber Events

Recover CSA-09 - Recover System Capabilities

CSA Cyber Survivability Attribute SS System Survivability KPP Key Performance Parameter

Cyber Survivability Risk Category Assignment – JITC will characterize and define the initial Attack Surface for testing based on the PM’s implementation of the CSE Process as directed in the Chairman of the Joint Chiefs of Staff, "Cyber Survivability Endorsement Implementation Guide.” As covered before, the CSE is mandated for SACs under the JROC oversight. Per the DoD CIO, any SAC that is not under JROC oversight must have a capability that addresses or exceeds the CSE process. The CSE process uses the Cyber Survivability Risk Category (CSRC) to identify the appropriate implementation levels for cyber survivability. The CSRC is accomplished by:

1. Selecting the mission type of the system

2. Determining the system’s cyber dependence

3. Determining the capability tier of the threat actor

4. Determining the impact of system compromise to the mission

Identifying the CSRC leads to the selection of the appropriate exemplar statements for the ICD and AoA guidance. This allows for defined expectations for testing of the fielded capability. Figure 4 illustrates the process detailed in the CSE.

COTS Commercial Off The Shelf GOTS Government Off The Shelf CSRC Cyber Survivability Risk Category TTP Tailored Tactics and Procedures DoD Department of Defense

Figure 4. Cybersecurity/Survivability Risk Category Determination

Test and Evaluation Strategy Brief (TESB) – The TESB is not always required for each program. Leadership will decide if the TESB is required. The TESB contains similar information as the Test Concept Brief (TCB) listed later in the document. The difference is timing and level of detail. The TESB is normally completed early in the testing lifecycle while the TCB is normally completed for OT events. The TESB is a slide deck that identifies the following:

Program Overview o Purpose - Clearly stated o System/Increment description clearly defined o System Architecture identifies system under test o Stakeholders identified o Program Status/Milestone listed o Program Schedule o Key Documentation

Evaluation Strategy o Evaluation Framework

– Dendritic clearly presents defined SS-KPP, Critical

Operational Issues (COI), Measures of Performance (MOPs) and Measures of Effectiveness (MOEs) in a matrix supporting the determination

– Critical capabilities identified with mission impact provided if not met

– High-level approach for requisite data collection

– OEC involvement and approval for T&E strategy

– Modeling & Simulation and Instrumentation application explained

T&E Strategy o Integrated test events identified, defined and coordinated (i.e., Operational Assessment (OA), DT, DT/OT, OT&E) o Test schedule proposed o Data source matrix identifies collection opportunities and methodology strategy

Risk Areas o Risk levels identified and presented o Effect to test planning, data collection, execution, and system analysis presented o Resolution and/or mitigation presented

Limitations o Effect to test planning, data collection, execution, and system analysis identified o Resolution and/or mitigation clearly presented

Backup Slides (if needed)

Section 3. Attack Surface Definition and Categorization

By this step, the PM, in coordination with the user community, has assigned the

CNSSI Security Categorization and overall mission criticality for the system during the

MSA phase in Section 1. This information feeds into the CSRC defined in Section 2.

Figure 6 depicts how this information is used to characterize the initial Attack Surface.

The Data Enablers shown in Figure 5 are constantly updated throughout the lifecycle allowing further refinement of the Attack Surface after testing.

CMEF Cyber Master Evaluation Framework ICD Interconnection Documentation COOP Continuity of Operation Plan IRP Incident Response Plan CRTM Cyber Risk Tracking Matrix ISP Information Support Plan CSP Cloud Service Provider PPP Program Protection Plan CSSP Cybersecurity Service Provider SAC System, Application, or Capability CTE Cyber Test Event SCA Security Control Assessor eMASS Enterprise Mission Assurance Support Service SSP System Security Plan IAVA Information Assurance Vulnerability Alert VOLT Validated Online Lifecycle Threat

Figure 5. Characterize the Attack Surface Activities

It should be expected that new SACs would not have all the information required for the full characterization of the Attack Surface. As the SAC matures, the Attack

Surface is updated to reflect the latest posture. The Attack Surface should be more

Data Enablers

SAC Security Documentation

Architecture Diagrams eMASS Package

SSP/ISP/PPP/ICD

Hardware Baseline Software Baseline Account Procedures

IRP/COOP

Configuration Management IAVA and Patching Special Data Type Definition Special Program Types

Installation Documentation

CTE Testing Results

Interfacing SAC Reports Other CTE Reports Event Logs System Logs Incident Reports SCA Reports Code Reviews Results

Examine Risk and

Threat Posture

Non-Technical and

Technical Review

Report and Log Data

Correlation

Exteranl Data Feeders Reports

CSP/CSSP Reports VOLT/Threat Docs Physical Security Testing

Threat Environmnet Review

C M

E F

/C R

T M

A tta c k

S u rfa c e a n d A c tio n M a p

Section 4 mature for SACs already deployed or after MS B in the acquisition process. In both cases, JITC will utilize the RFI process to obtain the data required.

Request For Information – The RFI process is a deliverable document that is produced by JITC and provided to the PM. The PM acknowledges receipt of the RFI because all planned CTE are scoped around the timely receipt of this information. The

PM is responsible for providing the completed RFI based on the date in the acknowledgement form or runs the risk of CTE postponement and rescheduling at the

PM's cost. The RFI is tailorable for each SAC and is the responsibility of JITC to ensure they only request the pertinent information for the SAC based on the AoA and that no duplicate information request is made. The goal of the RFI is to collect the necessary artifacts that JITC requires to develop and execute the CTE. The RFI process will continue throughout developmental and operational CTE and may require updating as conditions change. The RFI process is outlined in detail in Appendix G.

Data Enablers / Examine Risk and Threat Posture – The Data Enabler information is collected by JITC using the RFI process. Data Enabler information may be in draft form or not exist at all. The expectation is to obtain as much of the information as possible and update as the program progresses. After the Data Enablers are captured, the review is executed. The Data Enablers are reviewed using the non-technical and technical checks listed in the Data Source Matrix discussed in Appendix

B. These checks reveal the non-technical and technical risk to the SAC. After the risk items are identified, the review of potential mitigating factors are reviewed based on the

Report and Log Correlation. Any risk items not mitigated are reviewed against the potential likelihood of exploitation by a threat actor. This information is fed into the

Cyber Risk Tracking Matrix (CRTM) covered in Section 4.

Section 4. Cyber Master Evaluation Framework

The most important aspect of, and the driving force behind, the ICT&E approach is the Attack Surface and CMEF. All ICT&E results collected against the system’s Attack

Surface during the CTE are mapped to the NIST Special Publication 800-53 security controls families. The mapping is further refined to the specific controls, the CTPs, the

TPMs, and the CSAs. For systems processing financial data, these controls are also mapped to the Defense Finance and Accounting Services (DFAS) and Government

Accounting Office (GAO) requirements. A standardized DSM is used to capture the data for the controls. Data must flow constantly and unimpeded into the CMEF from all

CTE as soon as possible after the event conclusion. This includes data from interfacing systems that will be provided to JITC, consumed by other test events, or any other combination. Examples of the types of data that will inform the CMEF are:

Checks from the DSM

Risk Items and Findings identified during test events, Output from CyWG meetings, CRTM, Action Maps, And data from internal or external sources including cloud testing.

The combined input into the CMEF will ultimately produce an interactive scorecard to inform leadership of the SAC’s cyber posture on demand. The scorecard reflects the results at a high level as related to the NIST control families. The color codes correspond risk category of the risk items discovered based on the underlying controls. This is associated with the status of the testing and indicates whether all aspects of the controls have been assessed or if further testing is required. An example of the CMEF scorecard is shown in Figure 6.

NIST Security Control Area/Family CSA (AC) - Access Control CSA-01 / CSA-03 / CSA-06 / CSA-07 (AP) - Authority and Purpose (Privacy Control) CSA-01

(AR) - Accountability, Audit, and Risk Management (Privacy Control) CSA-04 / CSA-07

(AT) - Awareness and Training CSA-04

(AU) - Audit and Accountability CSA-07 (CA) - Security Assessment and Authorization CSA-06 / CSA-07

(CM) - Configuration Management CSA-06 / CSA-10

(CP) - Contingency Planning CSA-08 / CSA-09 (DI) - Data Quality and Integrity (Privacy Control) CSA-03 / CSA-05

(DM) - Data Minimization and Retention (Privacy Control) CSA-04

(IA) - Identification and Authentication CSA-01 / CSA-03 / CSA-06

(IP) - Individual Participation and Redress (Privacy Control) CSA-01 (IR) - Incident Response CSA-07 / CSA-08 / CSA-09

(MA) - Maintenance CSA-03 / CSA-04

(MP)- Media Protection CSA-03 / CSA-04 (PE) - Physical and Environmental Protection CSA-06 / CSA-07 CSA-08

(PL) - Planning CSA 1 through 10

(PM) - Program Management CSA-08 (PS) - Personnel Security CSA-07

(RA) - Risk Assessment CSA-07 / CSA-10

(SA) - System and Services Acquisition CSA-04 / CSA-06 / CSA-07 / CSA-08

(SC) - System and Communications Protection CSA-02 / CSA-03 / CSA-04 / CSA-05 / CSA-06 (SE) - Security (Privacy Control) CSA-07 / CSA-08

(SI) - System and Information Integrity CSA-04 / CSA-06 / CSA-07 / CSA-08 / CSA-10

(TR) - Transparency (Privacy Control) CSA-01 (UL) - Use Limitation (Privacy Control) CSA-04

Note: The privacy controls are grayed out whenever they do not apply to the SAC.

SCORING HIGHLIGHT LEGEND:

Blue Control completely tested with no futther testung required White Not Tested Red Major failure of a control with no mitigation in place Yellow Failure of a control with mitigation pending/expected Green No failure of the control pending further testing

CSA Cyber Survivability Attribute NIST National Institute of Standards and Technology CMEF Cyber Master Evaluation Framework SAC System, Application, or Capability

Figure 6. CMEF Scorecard Example

Each test event has specific goals and the corresponding analysis after each test event allows the test leads to structure the individual test events based on risk to the

Attack Surface. This allows the test teams to be more agile with test implementation and contained in Appendix B. Below is an overview of the areas covered in Appendix B.

Characterization of Risk Items and Findings – The assessment process uses the security control impact levels as defined in the CNSSI 1253, "Security

Categorization and Control Selection for National Security Systems," 15 March 2012, or the Security Technical Implementation Guides (STIGs)/Security Recommendation

Guides (SRGs) category level to rate risk items or findings and further evaluate the likelihood of exploitation during the CyWG.

Cyber Data Source Matrix – The DSM is based on the Cyber Resilience Review

(CRR) NIST Cybersecurity Framework Crosswalk and the NIST SP 171A, “Assessing

Security Requirements for Controlled Unclassified Information.” The DSM identifies checks and test cases that must be completed against the controls based on level of maturity.

Cyber Risk Tracking Matrix – The Cyber Risk Tracking Matrix (CRTM) is not a

Plan of Actions and Milestones (POA&M) (as used in RMF). The RMF POA&M tracks individual violations of policy or vulnerabilities for compliance. The CyWG will use the

CRTM to track and document the risks recorded in the DSM to the overall Attack

Surface and for the development of the CMEF Scorecard and supporting the Action

Maps. The CMEF Scorecard, discussed in Appendix B, will be the primary means to inform program leadership of the risk to the system prior to proceeding to the

Adversarial Assessment (AA).

Cyber Working Group – The goal of the CyWG is to maximize the likelihood of a system’s cyber survivability and resiliency by reducing its Attack Surface. The CyWG is a sub-working group to the overall T&E working group. The T&E working group is chaired by the DT and OT Lead testers. The Cybersecurity DT lead and the Cyber

Survivability OT are co-chairs of the CyWG and report back to the overall T&E working group with the results from the CyWG. In order to evaluate risk items against a system’s Attack Surface, the CTE leads will convene a CyWG at the end of each test event to develop and evaluate Attack Paths. Attack Paths are the plotting of one or more risk items which a potential adversary will use as a path to execute the adversary’s end goals. These risks are documented in the CRTM and the Attack Paths are documented in the form of Action Maps. Action Maps are covered in detail in

Appendix D.

Data From External Sources – JITC consumes data from numerous external entities when developing the Attack Surface. Every instance of an interface has the potential to add risk to the Attack Surface. Data from testing of these interfaces reduce the testing resource requirements and enables the reuse of data for follow-on testing.

DoDI 8510.01 Enclosure 5 of the RMF directs the implementation of Cybersecurity reciprocity as defined below:

“Cybersecurity reciprocity (referred to herein as “reciprocity”) is an essential element in ensuring IT capabilities are developed and fielded rapidly and efficiently across the DoD Information Enterprise. Applied appropriately, reciprocity reduces redundant testing, assessing and documentation, and the associated costs in time and resources. The DoD RMF presumes acceptance of existing test and assessment results and authorization documentation.”

Note: An interfacing system is defined as any system, application, or service that connects to the SAC covered by an agreement with specific rules and processes for connection. In some cases, web interfaces are not considered as an interface.

Interfacing Systems, Applications, or Services – As part of any interface agreement, both parties must provide evidence of each systems compliance with RMF to each party. Any interfacing system to the SAC is an attack path for the attacker and must be evaluated. Therefore, the current interfacing systems' RMF information will be provided by the SAC PMO for analysis. This information will be evaluated for risk to the

Attack Surface for the SAC.

Cloud or Third Party Assessor (3PAO) Testing – JITC will obtain test results through the Secure Cloud Computing Architecture (SCCA) PM or system PM in the event that the SAC is deployed in a commercial (non-DoD managed) cloud environment. JITC will coordinate to obtain data needed to make the ICT&E determination. Additional testing may be scheduled based on the adequacy of the data resulting from the 3PAO test events. The cloud test methodology is contained in

Appendix F.

Other ICT&E Testing – JITC may use any relevant and timely test data available from other unassociated test events if that data applies the SAC or to the specific environment or site where the SAC is being deployed. This includes, but is not limited to: Cybersecurity Service Provider (CSSP) detection and reporting reports, exercise or

Persistent Cyber Operations (PCO) mission reports, or sensor deployment testing.

Section 5. Program Management Office Developmental Test Cybersecurity

Activities

This is an overview of the systems engineering and developmental test events that contribute to the CMEF. The blue box areas CTEs shown on Figure 2 are conducted by the PMO, SE, and the SCA. Procedures detailing the execution of these specific events are produced by the program office and aligned to the CMEF for integration. A high-level description of these events and the reporting requirements of each follows.

System Engineering and Developer Testing – The assessment process is an information-gathering and evidence-producing activity to determine the effectiveness of the safeguards intended to meet the NIST security controls listed in in the DSM. As stated before, not all aspects of the NIST security controls are assessed during one test event. The objective for this phase of testing is to determine the following:

Identify potential problems or shortfalls in the program or systems security and risk management programs;

Identify security weaknesses and deficiencies in the system applied to the proposed supporting environment in which the system will operate;

Prioritize risk mitigation decisions and activities;

Confirm that identified security weaknesses and deficiencies in the system have been addressed;

Support continuous monitoring activities and provide information security situational awareness, and

Analysis is related to the results of TPMs and CTPs evaluated to determine maturity level of the system.

This assessment process is not…