Bidders Library Cybersecurity - DOD Cybersecurity TE Guidebook.pdf

February 10, 2020

Version 2.0, Change 1

CLEARED

FOR OPEN PUBLICATION

FEB 06 2020

CASE # 20-S-0618

Department of Defense

OFFICE OF PREPUBLICATION AND SECURITY REVIEW

Questions and issues regarding the content and format of this document, please email guidebookfeedback@mitre.org

For technical issues and questions regarding Cybersecurity Developmental Test and Evaluation, please email osd.pentagon.ousd-re.mbx.communications@mail.mil

Office of the Under Secretary of Defense, Research and Engineering, Advanced Capabilities, Developmental Test and Evaluation and Prototyping (https://ac.cto.mil/dtep/)

For technical issues and questions regarding Cybersecurity Operational Test and Evaluation, please email

Mr. David Aland (david.j.aland.civ@mail.mil)

Office of the Director, Operational Test and Evaluation (https://www.dote.osd.mil/) mailto:guidebookfeedback@mitre.org mailto:osd.pentagon.ousd-re.mbx.communications@mail.mil https://ac.cto.mil/dtep/ mailto:david.j.aland.civ@mail.mil https://www.dote.osd.mil/

Cybersecurity Test and Evaluation Guidebook 2.0, Change 1 i

Table of Contents

Introduction

1.1 Organization of This Guidebook

1.2 Audience

1.3 Applicability

1.4 Terminology

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

2.1 Operation of the Defense Acquisition System, DoDI 5000.02

2.2 Fiscal Year 2016 National Defense Authorization Act (NDAA) Section 804

2.3 Business Systems Requirements and Acquisition, DoDI 5000.75

2.4 Cybersecurity, DoDI 8500.01

2.5 Cybersecurity Activities Support to DoD Information Network Operations (DODIN), DoDI

8530.01

2.6 Joint Requirements Guidance

2.7 DOT&E Cybersecurity Procedures Memoranda

Cybersecurity Test and Evaluation Overview

3.1 Cybersecurity T&E Phases Overview

3.2 Cybersecurity Working Group

3.3 Cybersecurity Threat Assessments

3.4 DT&E and SE Collaboration

3.5 Early Tester/Analyst Involvement

3.6 Mission-Based Cyber Risk Assessments

3.7 Role of Cybersecurity Developmental Testing

3.8 Integrated Testing

Phase 1: Understand Cybersecurity Requirements (and Plan for T&E)

4.1 Schedule

4.2 Inputs

4.3 Tasks

4.4 Phase 1 Data Requirements

Phase 2: Characterize the Cyber-Attack Surface

5.1 Schedule

5.2 Inputs

5.3 Tasks

5.4 Phase 2 Data Requirements

Phase 3: Cooperative Vulnerability Identification ii

6.1 Schedule

6.2 Inputs

6.3 Tasks

6.4 Phase 3 Data Requirements

Phase 4: Adversarial Cybersecurity DT&E

7.1 Schedule

7.2 Inputs

7.3 Tasks

7.4 Phase 4 Data Requirements

Phase 5: Cooperative Vulnerability and Penetration Assessment

8.1 Schedule

8.2 Inputs

8.3 Tasks

8.4 Outputs

Phase 6: Adversarial Assessment

9.1 Schedule

9.2 Inputs

9.3 Tasks

9.4 Outputs

Acronyms and Glossary of Terms

10.1 Acronyms

10.2 Cybersecurity T&E Glossary of Terms

References

Cybersecurity T&E Phase 1 through 6 Quick Look .................................................. A-1

Incorporating Cybersecurity T&E into DoD Acquisition Contracts ....................... B-1

Considerations for Tailoring the Cybersecurity T&E Phases .................................. C-1

Key System Artifacts for Cybersecurity T&E Analysis and Planning .................... D-1

Guidance for the Cybersecurity Portion of the Developmental Evaluation

Framework (DEF) ......................................................................................................... E-1

Considerations for Staffing Cybersecurity T&E Activities ...................................... F-1

Considerations for Software Assurance Testing ........................................................ G-1

Considerations for Cybersecurity Requirements and Measures for DT&E (FOUO

Document)...................................................................................................................... X-1

Cyber Threat Assessment for Cybersecurity T&E (FOUO Document) .................. X-1

Mission-Based Cybersecurity Risk Assessments (FOUO Document) ...................... X-1 iii

Cybersecurity Test Infrastructure and Environment Planning (FOUO Document) ...

........................................................................................................................................ .X-1

Cybersecurity Test Considerations for Non-IP Systems (FOUO Document) ......... X-1 iv

List of Figures

Figure 2-1. Business Capability Acquisition Cycle

Figure 2-2. RMF Icon

Figure 3-1. Cybersecurity T&E Phases Mapped to the Acquisition Life Cycle

Figure 3-2. Cybersecurity T&E Phases Are Iterative

Figure 3-3. Interaction of SE and T&E Cybersecurity Activities

Figure 3-4. Cybersecurity Testing Requirement Venn Diagram

Figure 3-5. Interaction of RMF and T&E Cybersecurity Activities

Figure 4-1. Phase 1: Understand Cybersecurity Requirements Activities

Figure 4-2. Phase 1 Iteration

Figure 5-1. Phase 2: Characterize the Cyber-Attack Surface Activities

Figure 5-2. Phase 2 Iteration

Figure 5-3. Example Mission Decomposition and Criticality Analysis

Figure 5-4. Example Cyber-Attack Surface System Diagram

Figure 5-5. Cyber Kill Chain

Figure 5-6. Example Attack Surface Analysis

Figure 5-7. Threat Vignette Illustration

Figure 6-1. Phase 3. Cooperative Vulnerability Identification Activities

Figure 6-2. Phase 3 Testing Process

Figure 6-3. Phase 3 Iteration

Figure 7-1. Phase 4: Adversarial Cybersecurity DT&E Activities

Figure 8-1. Phase 5: Cooperative Vulnerability and Penetration Assessment Activities

Figure 9-1. Phase 6: Adversarial Assessment Activities Schedule

Figure C-1. Adaptive Acquisition Framework (AAF) Pathways .............................................................. C-1

Figure C-2. Tailored Cybersecurity T&E 6 Phase Process ....................................................................... C-2

Figure C-3. T&E During Agile Software Development ........................................................................... C-6

Figure C-4. The Sec in DevSecOps .......................................................................................................... C-7

Figure C-5. Cybersecurity T&E Phases Mapped to the BCAC Process ................................................. C-11

Figure E-1. DEF Schedule ........................................................................................................................ E-1

Figure E-2. Developmental Evaluation Framework Format ..................................................................... E-3

Figure E-3. Example DEF Completed Cybersecurity Section .................................................................. E-6

Figure E-4. Cyber Portion of the DEF for Agile Development Process ................................................... E-8

Figure F-1. Vulnerability Analyst Proficiency and Maturity Levels ........................................................ F-5

Figure F- 2. Example RASCI Table .......................................................................................................... F-8 v

Figure G-1. Software Stack Example ....................................................................................................... G-2

Figure G-2. Compromising the Software Stack ........................................................................................ G-2

Figure G-3. Windows Architecture ........................................................................................................... G-4

Figure G-4. Software Testing Strategy ..................................................................................................... G-5

Figure G-5. Software Testing Schedule .................................................................................................... G-7

Figure G-6. Testing Rigor ....................................................................................................................... G-11

Figure G-7. Comprehensive Evaluation of System Reliability ............................................................... G-13 vi

List of Tables

Table 3-1: SS KPP Pillars and Cyber Survivability Attributes (CSAs)

Table 4-1.Cybersecurity and Resilience Requirements and Testing Factors to Consider

Table 5-1. Mobile Navigation System (Notional) Attack Surface List

Table 6-1. Example Program Test Objectives, Technical Test Objectives and Metrics

Table 6-2. Example Test Activities

Table 6-3. Cybersecurity DT&E Activities and Cybersecurity Test Facilities

Table A-1. Cybersecurity T&E Acquisition and Review Decisions Quick Look .................................... A-1

Table A-2. Quick-Look Summary of DT&E Cybersecurity Phases 1 through 4 ..................................... A-2

Table A-3. Quick-Look Summary of OT&E Cybersecurity Phases 5 and 6 ............................................ A-4

Table C-1. BCAC Acquisition Decisions Informed by Cybersecurity T&E .......................................... C-12

Table G-1. Cybersecurity Software Test Requirements ........................................................................... G-7

Table G-2. Characterization Sources ........................................................................................................ G-8

Table G-3. Sample Test Methods ........................................................................................................... G-10 vii

DoDI Cybersecurity T&E Guidebook v2 Change 1 Significant Changes

Cybersecurity Requirements – Cybersecurity standards, operational resilience and system cyber survivability requirement test considerations were added throughout the document. Chapter 3, section 3.7 explains the cybersecurity requirements associated with the Risk Management Framework (RMF), Department of Defense Instruction (DoDI) 8500.01, and the Cyber Survivability Endorsement

Implementation Guide (CSE IG) to include the Cyber Survivability Risk Posture (CSRP). Chapters 4, 5, 6, and 7 explain how these requirements are addressed in cybersecurity Test and Evaluation (T&E)

Phases 1, 2, 3, 4.

Testing Defensive Cyber Operations (DCO) – Testing guidance was added throughout the Guidebook to address analyzing and testing a system’s DCO defensive measures. This guidance adheres to DoDI

8530.01.

Contractor testing and integrated contractor and government test activities – Language was added to each phase that addresses early (during development) contractor test activities, contractor test activities for rapid prototype and rapid fielding programs, and contractor remediation of discovered vulnerabilities.

Appendix B – Contract language considerations were added to address software development practices, development environments, software assurance testing. Suggested language was also added to address contractor-government integrated cybersecurity testing during system development, contractor testing during prototype and rapid fielding development, and contractor remediation of system vulnerabilities discovered during testing.

Appendix C – The current six Phase process is closely coupled with DoDI 5000.02 acquisition phases and milestones. To provide guidance for tailoring the phases, Appendix C adds a new six Phase diagram that is decoupled from any specific acquisition model. Appendix C also includes the following changes:

Agile and DevSecOps: A new section was added to discuss tailoring phases for Agile software development and DevSecOps for contractor and government testers.

Middle Tier Acquisitions (MTA): A new section was added to discuss tailoring of Rapid

Prototyping and Rapid Fielding for MTA programs.

Appendix E: The cyber portion of the Developmental Evaluation Framework (DEF) now includes an example of a DEF tailored for programs using Agile development methods.

Appendix F: The DoD Cybersecurity Developmental Test Cross-Service working group (XSWG) recently recommended a set of Development Test Cyber Vulnerability Analysis (VA) Standards.

Appendix F was updated to include a summary of the VA standards. The VA standards recommend organizational level and analyst level standards. The organizational standards describe recommended administrative capabilities to support test events, staff cyber VA personnel, and develop and retain Cyber

DT VA workforce. The analyst level standards include a set of Knowledge, Skills and Abilities (KSAs) and a progression of knowledge for Cybersecurity VAs.

Additional changes and corrections are listed in the table below.

Record of Changes

Version Effective Date Summary

1.0 1 July 2015 Initial release, version 1.0

2.0 25 April 2018 Initial release, version 2.0

2.0 Change 1 Review Draft 4 November 2019

New section 2.2: Section 804/MTA policy, new section 2.5 DoDI 8530.01

Chapter 3: Updated figure 3-1, 3-3, 3-5, added new figure 3-4, updated sections 3.1, 3.2, 3.3, 3.4, 3.5, 3.8; added new section 3.7: operational viii resilience and cyber survivability testing; updated figure 3-3

Chapter 4: Updated sections 4.1, 4.2, 4.3, 4.4.

Updated table 4-1 and figure 4-1.

Chapter 5: Updated sections 5.1, 5.2, 5.3. Added new figure 5-7 for Threat Vignettes

Chapter 6: Updated sections 6, 6.3, 6.4. Updated tables 6-1 and 6-2

Chapter 7: Updated section 7, 7.1, 7.3

Chapter 8: Updated section 8, 8.1, 8.2, 8.3;

updated figure 8-1

Chapter 9: Updated for use of Red Teams, section

9, 9.3, text box in 9.3.1; updated figure 9-1

Chapter 10: Added acronyms; corrected CSA, CSRC; updated glossary

Chapter 11: Updated and added references

Appendix A: Updated terminology

Appendix C: updated table C-1

Appendix G: Updated figure G-6

Introduction

Introduction

The purpose of this updated guidebook is to promote data-driven mission-impact-based analysis and assessment methods for cybersecurity test and evaluation (T&E) and to support assessment of cybersecurity, system cyber survivability, and operational resilience within a mission context by encouraging planning for tighter integration with traditional system T&E. Cybersecurity T&E starts at acquisition initiation and continues throughout the entire life cycle.

The guidebook supplements information provided in the Test and Evaluation Master Plan (TEMP)

Guidebook. For more information about TEMPs see References. This updated version avoids restating policy, such as that in the Risk Management Framework (RMF); instead, it encourages the reader to go directly to policy source documents for more information. The guidebook includes footnoted references for some content to assist with understanding the source of the content.

1.1 Organization of This Guidebook

This guidebook has nine chapters, including this introductory Chapter 1. Chapter 2 describes the policies and guidance that are the basis for cybersecurity T&E activities described in this guidebook. Chapter 3 provides an overview of cybersecurity T&E. Chapters 4 through 9 provide detailed implementation guidance for Program Managers (PMs) and test organizations on each of the phases of cybersecurity T&E as follows:

Chapter 4: Phase 1—Understand Cybersecurity Requirements

Chapter 5: Phase 2—Characterize the Cyber-Attack Surface

Chapter 6: Phase 3—Cooperative Vulnerability Identification

Chapter 7: Phase 4—Adversarial Cybersecurity DT&E

Chapter 8: Phase 5—Cooperative Vulnerability and Penetration Assessment

Chapter 9: Phase 6—Adversarial Assessment

The appendices provide additional guidance and information on topics as follows:

Appendix A: Cybersecurity T&E Phase 1 through 6 Quick Look

Appendix B: Incorporating Cybersecurity T&E into DoD Acquisition Contracts

Appendix C: Considerations for Tailoring the Cybersecurity T&E Phases

Appendix D: Key System Artifacts for Cybersecurity T&E Analysis and Planning

Appendix E: Guidance for the Cybersecurity Portion of the Developmental Evaluation

Framework (DEF)

Appendix F: Considerations for Staffing Cybersecurity T&E Activities

Appendix G: Considerations for Software Assurance Testing

Appendix X1: Considerations for Cybersecurity Requirements and Measures for Developmental

T&E (FOUO document)

Appendix X2: Cyber Threat Assessment for Cybersecurity T&E (FOUO document)

Appendix X3: Mission-Based Cybersecurity Risk Assessment (FOUO document)

Appendix X4: Cybersecurity Test Infrastructure and Environment Planning (FOUO document)

Appendix X5: Cybersecurity Test Considerations for Non-IP Systems (FOUO document)

For Official Use Only (FOUO) appendices are accessible to government and authorized contractor personnel at the following link: https://intelshare.intelink.gov/sites/resp/CTT

Introduction

1.2 Audience

This guidebook is intended for PMs, Chief Developmental Testers (CDTs), Lead Developmental Test and

Evaluation (DT&E) Organizations, Operational Test Agencies (OTAs), and the cybersecurity test teams for Department of Defense (DoD) acquisition programs.

1.3 Applicability

The guidance applies to all DoD acquisition programs and systems (e.g., defense business systems [DBS], national security systems, weapon systems, non-developmental items, industrial control systems [ICS], hull, mechanical and electrical [HM&E] systems, and supervisory control and data acquisition [SCADA] systems) regardless of their acquisition category (i.e., [Acquisition Category] ACAT I, IA, II III, IV and

BCAT I, II, III) or their phase of the acquisition life cycle unless noted. Acquisition cybersecurity T&E is not a stand-alone domain but is part of the overall program T&E strategy.

Acquisition programs not required to follow DoD Instruction 5000 series guidance will also benefit from following this guidebook.

1.4 Terminology

Cybersecurity T&E is used to describe the activities that encompass all cybersecurity test and evaluation activities, including vulnerability assessments, security controls testing, penetration testing, adversarial testing, and cybersecurity testing related to a system’s operational resilience and system cyber survivability capabilities within a mission context.

The Services/Components and organizations involved in cybersecurity T&E may use different terms for the people or teams discussed in the guidebook. The activities described in this document are more important than the titles of those performing the activities. For example, the term “cybersecurity tester,” as used often in this guidebook, refers to individual analysts and vulnerability or adversarial assessment teams, including Blue and Red Teams, government and contractor/developers, involved in the verification and validation of system cybersecurity capabilities, system cyber survivability, and operational resilience requirements across the life cycle of a system. Appendix F addresses key personnel involved in the testing planning, analysis, and execution.

The Services/Components may also use different terms for their assessments of system cyber survivability and operational resilience. This guidebook uses the phrase Prevent, Mitigate, Recover

(PMR) for consistency with the key attributes described in the Cyber Survivability Endorsement

Implementation Guide (CSE IG)1. PMR is defined as:

Prevent: The ability to protect critical mission functions from cyber threats.

Mitigate: The ability to detect and respond to cyber-attacks and assess resilience to survive attacks and complete critical missions and tasks.

Recover: The resilience to recover from cyber-attacks and prepare mission systems for the next fight.

Further discussion of the CSE IG is found in this guidebook in Section 2.4 and Appendix X1.

1 Cyber Survivability Endorsement Implementation Guide (CSE IG) version 2.01, Joint Staff

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

Cybersecurity Policies and Guidance for Defense Acquisition

Programs and Systems

This chapter summarizes policy for planning and conducting cybersecurity T&E.

2.1 Operation of the Defense Acquisition System, DoDI 5000.02

DoDI 5000.02, Operation of the Defense Acquisition System, updated in 2017 with Enclosure 14, Cybersecurity in the Defense Acquisition System, outlines responsibilities the PM should implement to safeguard DoD acquisition systems from cybersecurity-related risks throughout the system life cycle.

The key T&E elements of the policy are its emphasis on the continuous need to understand adverse mission impacts from cyber-attacks by using evolving system threats to inform operational impacts.

“Paragraph 3.b. (4) explains the goal is to mitigate risks that could impact performance objectives as well as thresholds.2” This updated guidebook integrates this increased emphasis on understanding threats and mission-based cybersecurity risks.

This guidebook outlines the preferred approach for PMs, CDTs, and OTAs to implement the DoDI

8500.01 and DoDI 5000.02 policies for cybersecurity T&E.

2.2 Fiscal Year 2016 National Defense Authorization Act (NDAA) Section 804

In 2016, Congress passed the Section 804 National Defense Authorization Act, titled Middle Tier of

Acquisition for Rapid Prototyping and Rapid Fielding3 that addresses using innovative technology to rapidly develop fieldable prototypes and field capabilities within 5 years of an approved requirement to demonstrate new capabilities to meet emerging military needs. Middle Tier Acquisition (MTA) programs are exempt from DoDI 5000.02 and Joint Capabilities Integration and Development System (JCIDS) processes but are not exempt from requirements approval.

MTA programs may field a prototype that can be demonstrated in an operational environment and provide residual operational capability within 5 years of an approved requirement. MTA programs may also use a rapid fielding pathway for rapidly fielding production quantities of new or upgraded systems with minimal development required. The objective of a rapid fielding program is to begin production within 6 months and complete fielding within five years of the requirement.4 A rapid acquisition program can proceed from approved requirement directly to production with minimal development or as a follow-on to a rapid prototype.

Section 804 MTA programs require tailored cybersecurity T&E processes that keep pace with rapid acquisition and fielding timelines. This implies early planning and analysis to ensure the analysis of alternatives (AOA) research includes cybersecurity requirements for evaluating alternatives and the prototype Requests for Proposals (RFPs) include contractor cybersecurity T&E requirements.

Cybersecurity T&E planning will require automated cybersecurity testing for software, but also should include test tools and test engineers embedded in the system development process. Appendix C describes test tailoring for MTA programs.

2 DoDI 5000.02, Operation of the Defense Acquisition System, Enclosure 14 (7 January 2017) 3 National Defense Authorization Act for Fiscal Year 2010, 10 U.S.C., Pub. L. 111-84 § 804 (2009) 4 USD(R&E) Middle Tier of Acquisition (Rapid Prototyping/Rapid Fielding) Interim Governance, (October 9, 2018)

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

2.3 Business Systems Requirements and Acquisition, DoDI 5000.75

DoDI 5000.75, Business Systems Requirements and Acquisition, first published in February 2017, defines policy and procedures, including cybersecurity for DBS. It outlines responsibilities the PM must implement to safeguard DoD business systems throughout the system life cycle.

The policy describes the use of the Business Capability Acquisition Cycle (BCAC) for business systems requirements and acquisition. DoDI 5000.75 supersedes DoDI 5000.02 for all business system acquisition programs that are not designated as a Major Defense Acquisition Program (MDAP) (based on MDAP thresholds) according to DoDI 5000.02. The notable difference between the BCAC and the traditional acquisition life cycle is that the BCAC has different phase names and six milestone decisions (depicted in

Figure 2-1) instead of three.

Figure 2-1. Business Capability Acquisition Cycle

The policy states that the Program Office’s implementation plan must include cybersecurity processes to reduce technical risk through T&E management procedures that include:

A Developmental Test and Evaluation Framework (DEF)

Cooperative vulnerability identification and adversarial cybersecurity testing in both developmental and operational tests

A Cyber Economic Vulnerability Analysis (CEVA) as outlined in the January 21, 2015, Director, Operational Test and Evaluation (DOT&E) Memorandum—CEVA is required at the discretion of

DOT&E only for DoD systems whose functions include financial or fiscal/business activities or the management of funds

Direction to Milestone Decision Authorities (MDAs) to avoid tailoring cybersecurity T&E solely to meet Authorization to Operate (ATO) requirements

Appendix C includes considerations for tailoring the cybersecurity T&E phases for the BCAC.

2.4 Cybersecurity, DoDI 8500.01

DoDI 8500.01, Cybersecurity, defines the policy and procedures for cybersecurity. The key elements of the policy are that it:

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

Extends applicability to all DoD information technology (IT), including Platform IT.

Emphasizes operational resilience, risk management, integration, and interoperability.

Incorporates cybersecurity considerations early and continuously within the acquisition life cycle.

References the National Institute of Standards and Technology (NIST) Special Publication (SP)

800-53 Security Control Catalog for use in the DoD.

The policy defines the following activities for the CDT, Lead DT&E Organizations, and the T&E community:

Ensure that cybersecurity T&E is conducted throughout the acquisition life cycle.

Plan, resource, and integrate cybersecurity assessments into DT&E and as part of T&E assessments.

Incorporate cybersecurity planning, implementation, testing, and evaluation in the DoD acquisition process and reflect them in the system TEMP.

Ensure that cybersecurity T&E is integrated with interoperability and other functional testing, and that a cybersecurity representative participates in planning, execution, and reporting of integrated

T&E activities.

Enclosure 3 states that acquisition programs must conduct an operational resilience evaluation during cybersecurity DT&E and operational T&E (OT&E). The evaluation includes exercising under realistic cyber conditions the ability to prevent and mitigate penetrations and exploitations and to recover data and information. To inform acquisition and fielding decisions, PMs should test procedures and tactics for workarounds and fallbacks in hostile environments. PMs should:

Conduct periodic exercises or evaluations of a program’s ability to operate during loss of all information resources and connectivity.

Ensure that systems can allocate information resources dynamically as needed to sustain mission operations while addressing cybersecurity failures, no matter the cause.

Ensure that systems can restore information resources rapidly to a trusted state while maintaining support to ongoing missions.

Enclosure 3 also instructs PMs to include an evaluation of cybersecurity during an acquisition T&E event.

The evaluation should include independent, threat representative penetration and exploitation T&E of the complete system cyberspace defenses, including the controls and protection that Cybersecurity Service

Providers (CSSPs) deliver. PMs should plan and resource the penetration and exploitation testing part of

DT&E and OT&E using the appropriate system test documentation.

DoDI 5000.02, Enclosure 14, DoDI 5000.75, and this guidebook contain the policy and guidance to ensure that PMs successfully perform the above defined DoDI 8500.01 activities.

2.4.1 Risk Management Framework, DoDI 8510.01

The RMF, defined in NIST SP 800-37, is mandated for the DoD by DoDI 8510.01, Risk Management

Framework (RMF) for DoD Information Technology (IT). The policy defines procedures for acquisition processes related to RMF and DT&E but does not replace specific DT&E or OT&E guidance. DoDI

8510.01 requires that the test community:

Integrate RMF activities with developmental and operational test activities.

Define specific concepts and rules for testing to support reciprocity between Program Offices to reduce redundant testing, assessments, documentation, and the associated costs in time and resources.

Integration of RMF with DT&E and OT&E processes requires proper and early planning to ensure that data needed for DT&E and OT&E is available. Look for the RMF icon, Figure 2-2, throughout this guidebook as an indicator to highlight RMF and T&E integration:

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

Figure 2-2. RMF Icon

2.5 Cybersecurity Activities Support to DoD Information Network Operations (DODIN), DoDI 8530.01

DoDI 8530.01 “The Cybersecurity Activities Support to DODIN” supports RMF requirements to monitor security controls continuously, determines the security impact of changes to the DODIN and operational environment, and conducts remediation actions as described in DoDI 8510.01. DoDI 8530.01 applies to

DoD IT (e.g., DoD-owned or DoD-controlled information systems (ISs), platform information technology (PIT) systems, IT products and services) as defined in DoDI 8500.01 and control systems and industrial control systems (ICSs) as defined in NISTSP 800-82 that are owned or operated by or on behalf of DoD Components. DoDI 8530.01 also applies to cleared defense contractors and any mission partner systems connecting to the DODIN.

Cybersecurity T&E should verify and validate that the appropriate security measures were effectively integrated into the system boundary by testing Defensive Cyber Operations (DCO) defensive measures during test events. Testing DCO defensive measures allow mission owners and operators, from the tactical to the DoD level, to have confidence in the confidentiality, integrity, and availability of the

DODIN and DoD information to make decisions.

2.6 Joint Requirements Guidance

In January 2017, the Joint Requirements Oversight Council (JROC) approved a proposed update to the

JCIDS manual (ref. JROCM 009-17) that updates the System Survivability Key Performance Parameter

(SS KPP). The SS KPP update encourages requirements developers to leverage the CSE IG developed by the Joint Staff/J6 in collaboration with the Deputy DoD Chief Information Officer (CIO) for

Cybersecurity, the Defense Intelligence Agency (DIA), and the National Security Agency. The CSE IG consists of guidance that helps acquisition programs ensure that cyber survivability requirements are included in system designs as early as possible.

For PMs, CDTs, Lead DT&E Organizations, and the cybersecurity T&E community, the importance of this update to the JCIDS manual is directly tied to Phase 1 of cybersecurity T&E, Understand the

Cybersecurity Requirements. The SS KPP included in a system’s requirements documents (i.e., Initial

Capability Document [ICD], Capability Development Document [CDD], Capability Production

Document [CPD], Capability Requirements Document [CRD], Information Systems [IS]-ICD and IS-

CDD), is used by the system engineers and system security engineers (SSEs) to define the 10 cyber survivability attributes and risk-managed performance measures in their functional and system requirements documents.

Several Services have similar standards for cybersecurity T&E. The Navy has established the

CYBERSAFE process to ensure overall resilience in addition to the RMF process. The Air Force promulgated an update Air Force Instruction (AFI) 99-103 where cybersecurity testing is prominent in the policy for aircraft testing.

Appendix X1 provides considerations for assessing cyber survivability within the framework of the updated SS KPP.

Cybersecurity Policies and Guidance for Defense Acquisition Programs and Systems

2.7 DOT&E Cybersecurity Procedures Memoranda

In April 2018, the DOT&E published their revised Procedures for Operational Test and Evaluation of

Cybersecurity in Acquisition Programs memorandum to provide revised guidance to the OTAs. The memorandum directs OTAs to perform a cybersecurity Cooperative Vulnerability and Penetration

Assessment (CVPA) and an Adversarial Assessment (AA) of all acquisition programs. Phases 5 and 6 in this guidebook amplify the guidance in the DOT&E memorandum.

In addition, in January 2015, DOT&E published the DBS CEVA memorandum. This memorandum directs OTAs to modify their cybersecurity T&E processes as appropriate for DoD systems whose functions include financial or fiscal/business activities or the management of funds. The memorandum also directs the OTAs to add Cyber Economic Threat Analysis, Cyber Economic Scenario Testing, and

Financial Transaction Analysis to their cybersecurity test planning for DBS.

Cybersecurity Test and Evaluation Overview

Cybersecurity Test and Evaluation Overview

This chapter provides an overview of the six cybersecurity T&E phases and discusses topics that are relevant to all phases.

3.1 Cybersecurity T&E Phases Overview

Figure 3-1 depicts the cybersecurity T&E phases aligned to the DoDI 5000.02 acquisition life cycle. A key feature of effective cybersecurity T&E is early involvement of development contractor, developmental testers, and operational testers in test analysis and planning. Each cybersecurity T&E phase is iterative and includes Phase 1 and 2 ongoing planning and analysis activities for the subsequent phases. For example, before Phase 5, the contractor and government test teams should repeat Phases 1 and 2 to ensure the requirements are clear and concise and understand any updates to the attack surface.

Changes in the attack surface will help identify the CVPA scope to a greater fidelity. Tools that can automate the Phase 1 and 2 activities and that include a digital model of the systems and integration points may facilitate a faster planning and analysis effort. When referred to in this Guidebook, the system developer or system integrator may be a government or contractor or shared role.

Figure 3-1. Cybersecurity T&E Phases Mapped to the Acquisition Life Cycle

Phase 1—Understand the Cybersecurity Requirements. The purpose of the Phase 1 is to examine the system’s cybersecurity, system cyber survivability, and operational resilience requirements for developing initial and subsequent approaches and plans for conducting contractor and government cybersecurity T&E.

Phase 2—Characterize the Attack Surface. During Phase 2, government and contractor test teams identify vulnerabilities and avenues of attack an adversary may use to exploit the system and develop plans to evaluate the impacts to the mission.

Phase 3—Cooperative Vulnerability Identification. The purpose of the third phase is to begin testing early to verify cybersecurity and operational resilience and identify vulnerabilities and inform implementing any needed mitigations. Using multiple tailored test events, vulnerability identification informs contractor and government system designers, developers, and engineers of

Cybersecurity Test and Evaluation Overview needed system cyber survivability and operational resilience improvements to reduce risk. Phase

3 is iterative during contractor development and includes regression testing to verify implemented mitigations. Phase 3 is also iterative during government DT&E.

Phase 4—Adversarial Cybersecurity DT&E. During this phase, integrated contractor and government adversarial test teams test critical functionality. Cybersecurity and operational resilience testing are conducted during system development using a mission context. Phase 4 is iterative during contractor system development.

Phase 5—Cooperative Vulnerability and Penetration Assessment. The purpose of this phase is to use data from cooperative cybersecurity test events to characterize the cybersecurity and resilience of a system in an operational context and provide reconnaissance of the system in support of the AA. This Phase includes assessing all test data from prior testing and is not a single test event.

Phase 6—Adversarial Assessment. Phase 6 characterizes the operational mission effects to critical missions caused by threat-representative cyber activity against a unit trained and equipped with a system, as well as the effectiveness of defensive capabilities.

The goal of cybersecurity T&E is to identify and mitigate exploitable system vulnerabilities impacting operational resilience of military capabilities before system deployment to include safety, survivability, and security. Early discovery of system vulnerabilities can facilitate remediation and reduce impact on cost, schedule, and performance. Cybersecurity T&E Phases 1 and 2 are the essential first steps of the

T&E planning process that support system design and development. Phase 1 and 2 should be performed in a cyclic fashion and repeated throughout each phase to ensure a thorough understanding of the requirements and any changes within the attack surface. Many Program Offices successfully perform

Phases 1 and 2 in parallel.

Phase 1 and 2 analyses and planning rely on engagement and collaboration with, and provide feedback to, system engineering (SE) and specialized component engineers during the early stages of prototyping, system design and development to facilitate design changes that improve cybersecurity, system cyber survivability and operational resilience. SE generates most of the system artifacts, described in Appendix

D, required during these analysis and planning phases, and therefore the partnership between SE and the testers is essential. Integrating developers and engineers for specialized components or functionality with the testers is also important to design effective, relevant testing.

Detailed test planning and execution occurs during system prototyping, system development and prior to system deployment. The various planned test events are aligned to Phases 3, 4, 5, and 6, depending on the acquisition life cycle and purpose of the testing. Phases 3 and 4 comprise cybersecurity DT&E execution activities for all sub-components and component integration during prototype development, system development, up through the full system delivered to the government for independent DT&E. Contractor and government cybersecurity testers develop test objectives, plan test activities and events, and plan the cybersecurity test infrastructure for Phases 3 and 4 based on the outcomes from the Phases 1 and 2 analyses.

Phases 5 and 6 comprise cybersecurity OT&E activities for the system. Operational cybersecurity testing supports the evaluation of system effectiveness, suitability, and survivability. The OTA follows the procedures promulgated by DOT&E to plan for and conduct Phase 5 and 6 activities. OTAs will require results from DT&E Phases 1 and 2 analyses and all DT&E test results to inform the OTA Phases 1 and 2 updates and cybersecurity OT&E.

During Operations and Support (O&S), PMs should periodically reevaluate systems for cybersecurity, system cyber survivability and operational resilience. The CSE IG requires all programs implementing the

CSE to include Cyber Survivability Attribute (CSA) 10: Actively Manage System’s Configurations to

Achieve and Maintain an Operationally Relevant Cyber Survivability Risk Posture (CSRP). CSA 10 will necessitate an iterative effort to repeat Phases 1 and 2, plan additional testing if needed, and potentially

Cybersecurity Test and Evaluation Overview re-engineering to address needed mitigations. Previously discovered and un-mitigated low risk to mission vulnerabilities over time may become high risk and system updates and changes in interfacing systems may introduce new exposures and risk. Phases 1 and 2 help to plan and scope government-conducted sustainment testing. Sustainment testing should take continuous monitoring data into account and may inform changes in people, processes or technology in order to mitigate risk.

Follow on Operational Test and Evaluation (FOT&E) is part of the O&S process and may include re-assessing test activities and test data from previous assessments. FOT&E is a vital part of assessing operational resilience. The continual PMR capabilities should be evaluated during FOT&E. There may be other reasons to return to formal T&E other than program-initiated system modifications:

Help ensure adequate funding is planned for the O&S phase through the future of the program

Changes and modifications to the system as part of a system of systems, especially interfaces that may not be under control of the system, for example if a mission planning system is modified, the processes and systems supported by the mission plan may be impacted

Changes to threat capabilities, newly revealed vulnerabilities, and new threat vectors

At a minimum, the program should conduct or update a mission-based cyber risk assessment (MBCRA)

(see Section 3.6 and Appendix X3) for the system when a significant change to the mission, system, threat, or operating environment occurs. Examples of significant changes include system modernization efforts, discovery of new threat vectors (zero-day vulnerabilities), or deployment of a system to a new operational environment. The results of the MBCRA activity may drive additional Phase 1 through 4

T&E activities, depending on the changes to mission risk. Even without significant changes to mission, system, threat, or operating environment, PMs should conduct or update the MBCRA for the system, with a focus on RMF continuous monitoring efforts in support of renewing the system’s ATO. The Life Cycle

Sustainment Plan (LCSP) should include cybersecurity T&E.

3.1.1 Iterative Nature of the Phases

Cybersecurity T&E phases are iterative (i.e., activities should be repeated multiple times due to changes in the system architecture, new or emerging threats, and changes in the operational environment). Here are some common examples of events that would drive iteration of the phases:

Significant change to the system architecture occurs, such as after a Preliminary Design Review

(PDR) or initiation of an Engineering Change Proposal (ECP). The CDT or system test lead usually repeats Phases 1 and 2 to incorporate any changes that may impact test planning and before conducting the next set of testing in any of the phases.

Updates to the TEMP or other test strategy documents concurrently with SE activities to update requirements, architecture, and design would necessitate Phases 1 and 2 iteration.

Changes to the target operational environment that may drive changes in design and subsequent test strategy which may impact the cyber-attack surface (Phase 2), and test planning for Phases 3, 4, and 5.

Changes to the cyber threat environment, if significant, may trigger repeat of Phase 2 attack surface analysis

When testers verify cybersecurity and operational resilience and discover new high-risk vulnerabilities, the PM may need to update requirements to mitigate the discovered vulnerabilities

Phases 1 and 2 should be repeated to examine the updated cybersecurity requirements and assess any changes to the system’s attack surface.

Cybersecurity Test and Evaluation Overview

Figure 3-2 depicts phase iteration after initial testing during Phase 3 and after Phase 4. Phases 3 and 4 are iterated in a find-fix-verify method (described later in chapters 6 and 7).

Figure 3-2. Cybersecurity T&E Phases Are Iterative

3.1.2 Tailoring Phases

PMs should address the six cybersecurity T&E phases regardless of where the system is in the acquisition life cycle. Some systems, however, enter the acquisition lifecycle at Milestone (MS) B, incrementally update major components of the system, or are already well into the acquisition life cycle when cybersecurity T&E phases are initiated. Accelerated acquisition programs may not have time for the full progression through the phases as depicted in Figure 3-1; however, the Program Office should devote time, and resources to include funding for integration of contractor and government cybersecurity testers, to the early analysis that the phases identify (understanding the cybersecurity requirements and characterizing the attack surface) to establish the foundation for efficient cybersecurity standards, system cyber survivability and operational resilience testing. Appendix C describes tailoring considerations for cybersecurity T&E phases and provides examples for DBS that use the BCAC, smaller acquisition programs, Section 804 MTA programs, and other acquisition programs with compressed timelines.

3.2 Cybersecurity Working Group

The recommended approach for planning and implementing the phases of cybersecurity T&E is for the

CDT or test lead for the system to establish, as early as possible, a Cybersecurity Working Group

(CyWG) that reports to the T&E Working Integrated Product Team (WIPT). The CDT or system test lead should ensure that the CyWG roles and responsibilities are documented in Section 2 of the TEMP. The

CyWG is responsible for integrating and coordinating all cybersecurity T&E and supporting the RMF assessment and authorization (A&A) process. The Information System Security Manager (ISSM) is the focal point for RMF A&A activities, and the remaining members of the CyWG are crucial to ensuring the full range of cybersecurity T&E is planned and executed. The Cybersecurity T&E Lead guides the test planning for the CyWG. The CyWG performs the tasks in the phases as described in this guidebook:

analysis, planning, scheduling, and assessment for all cybersecurity T&E. The CyWG focuses on integrating cybersecurity T&E with functional T&E and assessing mission-based cybersecurity risk to inform the PM before acquisition and engineering decisions. The CyWG membership should reflect the system type and size of program to include program specific cybersecurity staff.

The recommended participants in the CyWG are:

CDT or system test lead if the CDT has not yet been appointed

SSE

ISSM

Lead Systems Engineer Representative

Lead Software Engineer/Architect

Lead DT&E Organization Representative

Cybersecurity Test and Evaluation Overview

Critical or specialized sub-component, component or functionality developers and engineers

Operational Test Agency Representative

Cybersecurity DT&E Technical Experts (testers/analysts/assessors)

Cybersecurity OTA Technical Experts (testers/analysts/assessors)

Security Controls Assessor (SCA)

Cybersecurity Subject Matter Experts (SMEs)

Cyber-Intelligence SME

Software Assurance Testing SME

Scientific Test and Analysis Techniques (STAT) and/or Design of Experiments SME

Cyber Test Range Representative

Modeling and Simulation SME

Anti-tamper (AT) Representative

Interoperability Engineers, Representatives and Testers

CSSP and DCO Representatives

Active Duty System Operators

System Maintainers and Logisticians

Service-Specific T&E Policy Representative, if needed

Oversight organizations and stakeholders, if appropriate

Developers

Prime Contractor, if appropriate

RASCI Matrix. A Responsible, Accountable, Supporting, Consulting, Informed (RASCI) matrix defines the team needed to complete project tasks and their assigned role in each task. Using the recommended roles above, PMs may want to develop a RASCI matrix that supports the cybersecurity tasks described in this guidebook. Appendix F provides an example.

3.3 Cybersecurity Threat Assessments

A cybersecurity threat is an actor or a set of conditions that can cause an adverse mission effect. An assessment of cybersecurity threats should scrutinize each element that may bring about mission performance failures. Designing a system without understanding the relevant cybersecurity threat may result in system weaknesses or exposures that a human or automated process could exploit. It is also important to understand how the system may be used in an unintended manner to cause mission performance failures. It is critical to involve Cyber-Intelligence SMEs in this discussion to identify accurate threat intelligence for the specific system to be tested. The CyWG recommends an appropriate frequency for conducting cybersecurity threat assessments throughout the system development life cycle, but at a minimum, the Program Office obtains a validated threat assessment (e.g., Validated Online

Lifecycle Threat [VOLT] report or Service/Component threat assessment report) from the DIA at each acquisition milestone. The Program Office is responsible for evaluating and updating the mission risk assessment and RMF risk assessment if necessary, using updated threat assessment information. For a detailed explanation of developing, updating, and using the cybersecurity threat assessment throughout cybersecurity T&E, see Appendix X2.

3.4 DT&E and SE Collaboration

Early and regular collaboration between T&E, SSEs, and SE helps acquisition programs avoid costly, difficult system modifications late in the acquisition life cycle. The CDT or system test lead should collaborate with SSE and SE providing architecture and design information and derived critical technical parameters (CTPs) including requirements traceability throughout the system life cycle to the CDT.

Requirements traceability documentation provides insight into the design decisions for allocating

Cybersecurity Test and Evaluation Overview cybersecurity requirements which gives testers context to develop more tailored tests. The CDT will use this information to inform T&E activities and scenarios, to include what testing and data to require of the contractor, and to shape government test designs. DoDI 5000.02, Enclosure 14 provides greater detail into the various required cybersecurity activities across the system development life cycle.

Figure 3-3. Interaction of SE and T&E Cybersecurity Activities

SSE and SE derive CTPs required for test planning. Figure 3-3 illustrates the interaction between cybersecurity T&E activities and SSE and SE program protection activities during the traditional acquisition life cycle. Although Figure 3-3 depicts the full acquisition life cycle, this guidebook recommends that PMs integrate SSE, SE and T&E into the program’s acquisition strategy using the applicable DoD policy and guidance.

T&E, SSE, and SE should collaborate early to conduct MBCRAs/Cyber Table Top (CTT) exercises, described in Appendix X3, to inform the design of a system and increase the operational resilience of that system in its intended cyber-contested environment. Early T&E and SE collaboration is important because the contractor will build and test to the requirements only, ensuring that the system meets specifications.