Bidders Library Test and Evaluation Scorecard Guidebook Version 3 0 3 Dec 2020.pdf

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

JOINT INTEROPERABILITY

TEST COMMAND

TEST AND EVALUATION

SCORECARD GUIDEBOOK

VERSION 3.0

Distribution C Distribution authorized to U.S. Government Agencies and their supporting contractors. This document describes procedures or guidelines for operational test evaluation. Such information may be unclassified but is considered sensitive and its distribution is limited to entities that need it for Government purposes or to conduct official business for DOD. Other requests for this document must be referred to Commander, JITC, ATTN: JTA, P.O. Box 12798, Fort Huachuca, Arizona 85670-2798.

NOVEMBER 2020

(This page intentionally left blank.)

JOINT INTEROPERABILITY

TEST COMMAND

TEST AND EVALUATION

SCORECARD GUIDEBOOK

VERSION 3.0

NOVEMBER 2020

Submitted by: Michael Koester

Chief, Operational Test & Evaluation & Enterprise Services Division

Approved by: _________________________________

SHAWN ROBERTS

Captain, US Navy Commander, Joint Interoperability Test Command

Prepared Under the Direction of:

Akalnesh Mamo Joint Interoperability Test Command

Fort Meade, Maryland

Document Information

Title: Joint Interoperability Test Command Test and Evaluation Scorecard Guidebook

Document Identifier: N/A

Version/Status Version 3.0

Date: 20 Nov 2020

Owner/Contact: JITC/JTA/JTA3

Main File Location: https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/

Document Change History

Date Change Request

Change Author

Change Summary

February 2015 N/A A. Mamo Original Author

October 2015 N/A T. Knight Removed Continuity AoE; continuity is covered in the Cybersecurity AoE

November 2015 N/A T. Knight Split the Service Operations AoE into Service Operations and Transition AoEs

November 2015 N/A T. Knight Changed “Service” to “Project”

December 2015 N/A T. Knight Added AoE Annexes and changed title to AoE Modules

April 2020 N/A R. Montgomery Updated Capability module.

May 2020 N/A R. Montgomery Changed Sustainment to Transition AoE

May 2020 N/A R. Montgomery Update Users Experience AoE

June 2020 N/A R. Montgomery Updated the IOP module based on JT4A review feedback, which is based on the JITC and DOD policies and guides changes.

June 2020 Policy change D. Tucker

Rewrote the Cyber AoE based DOT&E cybersecurity T&E policy change and the JITC cyber guidebook update

August 2020 N/A R. Montgomery

T&E Scorecard Working Group collaboration - Updated Availability AoE - maintain the Geographic Failover, Local Redundancy, Restoration within the module but remove the word “Geographic” & “Local” as those concept may be obsolete with cloud computing

September 2020 N/A R. Montgomery T&E Scorecard Working Group collaboration -Updating Capacity AoE

October 2020 N/A R. Montgomery T&E Scorecard Working Group collaboration -Updating Service Operation AoE

November 2020 N/A R. Montgomery T&E Scorecard work group comment updates to Module 1, 2 and 7 i

FOREWORD

This Defense Information Systems Agency (DISA) Test and Evaluation (T&E)

Scorecard Guidebook provides an overview of the criteria used to evaluate the status of any DISA project for each Area of Evaluation (AoE) reported in the T&E scorecard.

The Guidebook includes modules for each T&E Scorecard AoE. Each module provides details covering requirements and design analysis, technical verification, and operational validation, to include suggested standard metrics and measures.

The DISA T&E Scorecard provides a standard means for reporting project status from a T&E perspective, and is used as a guide for the development of the project’s Evaluation Framework (EF). The T&E Scorecard includes the following AoEs:

1. Capability

2. Interoperability

3. Availability

4. Capacity

5. Transition

6. Service Operations

7. User Experience

8. Cyber

The term “project” is used throughout this document to mean any DISA

Information Technology project as defined in DISA Instruction 610-225-2 “Acquisition Oversight and Management,” dated 19 February 2015.

The term “requirement” is used throughout this document to refer to any capabilities technical documentations and/or artifacts, ‘shall’ statement requirements, use case, feature, epic story, user story and/or product backlog items for an Information Technology project.

The DISA T&E Scorecard Guidebook and supporting products, to include the

T&E Scorecard AoE modules, T&E Scorecard Template, and the DISA T&E Process Guide, are located at https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/ ii iii

TABLE OF CONTENTS

Page

FOREWORD .................................................................................................................... i

INTRODUCTION

PURPOSE ....................................................................... Error! Bookmark not defined.

SCOPE

AREAS OF EVALUATION

PROCESS OVERVIEW

AOE MODULE 1 – CAPABILITY

AOE MODULE 2 – INTEROPERABILITY

AOE MODULE 3 – AVAILABILITY

AOE MODULE 4 – CAPACITY

AOE MODULE 5 – TRANSITION

AOE MODULE 6 – SERVICE OPERATIONS

AOE MODULE 7 – USER EXPERIENCE

AOE MODULE 8 – CYBER

APPENDICES

APPENDIX A ACRONYMS

APPENDIX B GLOSSARY

APPENDIX C SUGGESTED SCORECARD MATERIALS

APPENDIX D REFERENCES

APPENDIX E POINTS OF CONTACT

LIST OF FIGURES

FIGURE 1. TEST AND EVALUATION APPROACH

FIGURE 2. STANDARD TEST AND EVALUATION SCORECARD (EXAMPLE)

FIGURE M3-1. INCIDENT LIFECYCLE

FIGURE B1-1. MANAGEMENT OF RISK ....................................................................B-3

LIST OF TABLES

TABLE M2-1.1. INTEROPERABILITY METHODOLOGY OVERVIEW – SUPPORTS

MILITARY OPERATIONS

TABLE M2-1.2. INTEROPERABILITY METHODOLOGY OVERVIEW – ENTER AND

iv

BE MANAGED ON THE NETWORK

TABLE M2-1.3. INTEROPERABILITY METHODOLOGY OVERVIEW – EFFECTIVE

INFORMATION EXCHANGES

TABLE M2-1.4. INTEROPERABILITY METHODOLOGY OVERVIEW – JOINT

INTEROPERABILITY CERTIFICATION (JIC)

TABLE M2-2. EXAMPLE NET-READY PERFORMANCE ATTRIBUTES

TABLE M2-3. INTEROPERABILITY READINESS CHECKLIST QUESTIONS

TABLE M3-1. AVAILABILITY METHODOLOGY OVERVIEW

TABLE M3-2. AVAILABILITY MANAGEMENT PROCESS REVIEW CRITERIA

TABLE M3-3. AVAILABILITY OPERATIONAL READINESS CHECKLIST QUESTIONS

TABLE M4-1.1. CAPACITY METHODOLOGY OVERVIEW – 1 CHARACTERIZATION

OF LOAD

TABLE M4-1.2. CAPACITY METHODOLOGY OVERVIEW – 2 SYSTEM RESOURCES

TABLE M4-2. CAPACITY PLAN REVIEW CRITERIA AND INDICATORS OF

SUCCESS

TABLE M4-3. EXAMPLE CAPACITY EVALUATION FRAMEWORK

TABLE M4-4. CAPACITY OPERATIONAL READINESS CHECKLIST QUESTIONS .. 46

TABLE M5-1. TRANSITION METHODOLOGY OVERVIEW

TABLE M5-2. SUGGESTED TRANSITION EVALUATION FRAMEWORK

TABLE M5-3. TRANSITION READINESS CHECKLIST QUESTIONS

TABLE M6-1.1. SERVICE OPERATIONS PROCESSES/SERVICE DESK FUNCTIONS

1 EVENT MANAGEMENT

LIST OF TABLES (Continued)

2 INCIDENT MANAGEMENT

TABLE M6-1.3. SERVICE OPERATIONS PROCESSES/SERVICE DESK FUNCTIONS

3 PROBLEM MANAGEMENT

4 ACCESS MANAGEMENT

TABLE M6-1.5. SERVICE OPERATIONS PROCESSES/SERVICE DESK FUNCTIONS

5 REQUEST FULFILLMENT

TABLE M6-1.6. SERVICE OPERATIONS PROCESSES/SERVICE DESK FUNCTIONS

6 SERVICE DESK FUNCTION

TABLE M6-1.7. SERVICE OPERATIONS PROCESSES/SERVICE DESK FUNCTIONS

7 TECHNICAL/APPLICATION MANAGEMENT

TABLE M6-2.1. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 1 EVENT

MANAGEMENT

TABLE M6-2.2. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 2

INCIDENT MANAGEMENT

TABLE M6-2.3. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 3

v

PROBLEM MANAGEMENT

TABLE M6-2.4. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 4 ACCESS

MANAGEMENT

TABLE M6-2.5. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 5

REQUEST FULFILLMENT

TABLE M6-2.6. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 6 SERVICE

DESK

TABLE M6-2.7. SERVICE OPERATIONS METHODOLOGY OVERVIEW – 7

TECHNICAL/APPLICATION MANAGEMENT

TABLE M6-3. OPERATIONS READINESS CHECKLIST QUESTIONS

TABLE M6-4. SERVICE DESK EVALUATION FRAMEWORK

TABLE M7-1. EXAMPLE USER EXPERIENCE (UX) METHODOLOGY OVERVIEW . 84

TABLE M7-2. USER EXPERIENCE PLAN REVIEW CRITERIA

TABLE M7-3. EXAMPLE USER EXPERIENCE MEASURES

TABLE M8-1. CMEF SCORECARD EXAMPLE

TABLE M8-2. SCORECARD CROSSWALK CODES

vi

INTRODUCTION

The Defense Information Systems Agency (DISA) Test and Evaluation (T&E) Scorecard provides a standard means for reporting Information Technology (IT) project status from a T&E perspective and should be used to guide the development of the project’s Evaluation Framework (EF) and execution of T&E. Note that the term “requirement” is used throughout this document to refer to any capabilities technical documentations and/or artifacts, ‘shall’ statement requirements, use case, feature, epic story, user story and /or product backlog items for an Information Technology project.

Also the term “project” is used throughout this document to mean any DISA IT project as defined in DISA Instruction 610-225-2, “Acquisition Oversight and Management,” dated 19 February 2015.

The T&E Scorecard includes the following Areas of Evaluation (AoEs):

1. Capability

2. Interoperability

3. Availability

4. Capacity

5. Transition

6. Service Operations

7. User Experience

8. Cyber

This T&E Scorecard Guidebook includes a module for each of these eight

AoEs. Each module provides an overview of the requirements and design analysis, technical verification, and operational validation processes specific to each AoE. The modules also provide suggested measures and metrics for accomplishing T&E approach and management, technical and operational objectives. Effectively addressing each AoE will position the project to meet Acquisition Review Board and Network Operations (NetOps) Readiness Review Board (NRRB) expectations. As such, the T&E Scorecard is used to report the T&E assessment of project status for each AoE.

While this guidebook and the T&E Scorecard are indeed useful for the evaluation of Acquisition Category (ACAT) programs and are traceable to formal EF typically applied to ACAT programs, nothing in this guidebook should be used to replace or supersede formal guidance for ACAT programs and the formal evaluation of Operational Effectiveness, Suitability, and Security (OESS).

Figure 1 depicts the high-level T&E process and shows the alignment of the T&E Scorecard AoEs with the OESS elements.

The following symbols used throughout the guidebook to highlight recommendations and items of importance to the reader:

Best Practices: Recommended processes

For Your Information: Important information

For Your Information

For a non-ACAT I program, the Program Office and test lead are required to identify requirements by other means. This guidebook acknowledges that some programs may not have official ACAT I documentation, such as Analysis of Alternative (AoA), Initial Capability Document (ICD), Capability Development Document (CDD), Capability Production Document (CPD), etc. However, the Program Office and test lead are encouraged to utilize other design and or equivalent documentation to identify requirements that can be traceable, verifiable and validated through a formal EF to report project status accordingly.

Figure 1. Test and Evaluation Approach

LEGEND:

DESMF

GCMP

JIE

Department of Defense Enterprise Service Management Framework Global Information Grid (GIG) Convergence Master Plan Joint Information Environment

NRRB

OESS

T&E

Network Operations Readiness Review Board Operational Effectiveness, Suitability, and Security Test and Evaluation

PURPOSE

The DISA T&E Scorecard Guidebook provides an overview of the criteria used to evaluate the status of IT projects; the AoE modules provide additional AoE focused guidance. The DISA T&E Scorecard provides a standard means for reporting IT project status from the T&E perspective.

SCOPE

Prepared by the T&E team and submitted to project stakeholders, the DISA T&E Scorecard uses eight AoEs to report the status of an IT project at any point in the acquisition lifecycle. This T&E Scorecard Guidebook provides detail on various methodologies, including Development, Security and Operations (DevSecOps) for evaluating each AoE, frames T&E involvement in the requirements and design analysis process, and provides guidance for the initial risk assessment, the evaluation framework, and the initial project specific T&E management approach. Ultimately, the guide prepares the project for an evaluation of operational risk and readiness by the NRRB.

AREAS OF EVALUATION

Based on pilot efforts and lessons learned, projects that leveraged similar documentation experienced fewer issues during development and testing, and reduced the number of schedule slips and associated schedule impacts. The following AoEs have been identified for the T&E Scorecard:

CAPABILITY: Addresses user-community needs and capability performance. In this area, requirements and associated Critical Technical Parameters (CTPs) Technical Performance Measurement (TPM), Technical Performance Parameters (TPP) and Key Performance Parameters (KPPs) are evaluated.

INTEROPERABILITY: Addresses interoperability based on the Net-Ready Key Performance Attribute (NR KPA) construct, i.e., supported military operations, effective end-to-end information exchanges, and the project’s ability to enter – and be managed on – the network. The project’s Joint Interoperability Certification status is also considered and reported under this AoE.

AVAILABILITY: Addresses the project’s ability to meet operational availability requirements, to include the processes to monitor and track availability. This includes the establishment and validation of operational reliability, availability, and maintainability metrics to include failover, redundancy and recovery activities.

CAPACITY: Addresses the project’s capacity requirements in terms of characterization of load of concurrent users, transactions impact on system resources i.e., Central Processing Unit (CPU) utilization, memory, storage etc. The AoE also covers, demand forecasting and system resources scaling processes.

TRANSITION: Addresses the project’s activities for transition planning and support, change management, configuration management, records management, transition/on-boarding, release, and deployment. This is accomplished through the review of the project’s plan and processes documentation.

SERVICE OPERATIONS: This AoE aligns with Department of Defense (DOD) Enterprise Service Management Framework (DESMF) service operations process areas to include event, incident and problem management, technical and application management, request fulfillment, and access management. It also addresses the Service Desk function and technical/application management requirements and capabilities, training and training documentation based on roles and responsibility.

USER EXPERIENCE: Addresses capability usability, user perception of value, documentation i.e., Users Guide, training and training materials, and Section 508 compliance.

CYBER: This AoE addresses and tracks the Integrated Cyber Test and Evaluation (ICT&E) processes used by Joint Interoperability Test Command (JITC) to determine the system’s overall ability to be found Cyber Survivable and Cyber Resilient as deployed in a cyber-contested environment. This process is predicated on the results from the required Developmental Testing Cybersecurity events and Operational Testing Cyber Survivability events outlined and defined in the DOD CIO Cyber Guidebook phases. The requirements under evaluation are the assigned Risk Management Framework (RMF) security controls and the Cyber Survivability Attributes as defined in the system’s Cyber Survivability Risk Category (CSRC). The system must survive based on the Adversary Threat Tier (ATT) defined in the CSRC and the system’s validated threat reference (i.e. Validated Online Lifecycle Threat (VOLT) document).

PROCESS OVERVIEW

Each project should have a designated T&E lead based on the level of risk determined by the Initial Risk Assessment and established in the T&E Approach document. Based on the Risk Assessment, roles, and responsibilities articulated in the T&E Approach, if the JITC is conducting the testing, JITC would be responsible for the project's T&E Scorecard. If JITC is not conducting the testing, then the Program Manager (PM) should designate a T&E lead to be responsible for the project's T&E

Scorecard. The entity conducting the test or assessment is responsible for determining the status rating for each AoE and providing the T&E Scorecard to the stakeholders. The following steps describe the approach:

1. Determine the acquisition strategy and the current phase/position in the project lifecycle.

2. Determine the documentation and expected capabilities, in relation to the project and supporting infrastructure, for the current phases in the lifecycle.

3. Identify any gaps in the availability of documentation and/or information contained in existing documentation, including documentation of test activities, issues, and risks.

4. Evaluate the schedule, plans and processes to complete required documentation and meet test objectives in time to inform appropriate milestones /decision points and ensure readiness for operations.

5. Analyze test results and make a determination regarding operational impact for each AoE.

Based on the previous five steps determine the status of the project, and update the T&E Scorecard using the most recent template located at: https://disa.deps.mil/org/JTA/JTA3/ETM/TestEval/index.aspx#/templates

The T&E Scorecard comments/rationale column provides an example of the criteria for a status of “Green = No/Minor Issues and Low Risk” for each AoE. The comments are meant only as an example. The specific criteria may differ depending on the project and its current position in the lifecycle. The DISA T&E Scorecard presents project status based on eight AoEs. Figure 2 depicts a sample T&E Scorecard.

Figure 2. Standard Test and Evaluation Scorecard (Example)

The T&E lead determines the rating that best describes the current status of the project for each AoE. The T&E Scorecard status ratings are defined as follows:

No/Minor Issues and Low Risk (Green): The project is on schedule, adequate testing is planned or has been conducted, and the AoE programmatic requirements are met with no significant or critical issues. No moderate or high-risk items have been identified. Requirements and processes are documented and adequate to support testing.

Significant Issues or Moderate Risk (Yellow): The project is behind schedule, or adequate testing is not planned; however, the project has developed a schedule to address the shortfall prior to the next major milestone or decision point. Some AoE programmatic requirements are not met. Significant issues or moderate risk items have been identified.

Requirements and/or processes are not fully documented but are adequate to support testing.

Critical Issues or High Risk (Red): The project is behind schedule or adequate testing is not planned, and the shortfall will likely not be addressed before the next major milestone. AoE programmatic requirements have not been met. Critical issues or high-risk items have been identified and do not have a documented, acceptable workaround or a mitigation strategy in place.

Requirements and/or processes are not adequate to support testing.

Not Applicable (N/A) (Black): The AoE is not currently applicable to the evaluation of the project; however, the AoE may become applicable at a later stage in the lifecycle. The rationale as to why the AoE is currently rated as N/A must be included in the T&E Scorecard comments column along with an explanation of when the AoE will/may become applicable. An example of a case where N/A would be applicable for one or more AoE might be a pilot program where continuity, availability, or transition would not be a concern of the test.

Not Evaluated (NE) (Grey): The AoE is automatically noted as a high risk unless it is articulated as to why it has not been evaluated. Agile processes must provide a justification for the lack of testing. The AoE will be evaluated at a later phase in the lifecycle. The rationale for why the AoE is currently rated as NE must be included in the scorecard comments column along with an explanation of when the AoE will/may become applicable. An example of a case where NE would be applicable for one or more components of the capability will be completed in next sprint/iteration of system development.

Not evaluating the specific component or AoE at point/phase of status report, might not yield dependency; hence, operational risk and/or impact on the current status of the project.

Once each AoE status rating is determined, the first statement in the comments/rationale column of the T&E Scorecard should describe the status of testing.

The comments should include information similar to the following examples (two examples are given for each color [Green, Yellow, Red, Black & Grey], one for when the AoE has not been tested, and one for when it has):

Green: Not tested, but adequate testing is planned and on schedule

Green: Tested and results indicate only minor issues and low risk

Yellow: Not tested, adequate testing is planned, but not on schedule

Yellow: Tested and results indicate significant issues or moderate risk

Red: Not tested and adequate testing is not planned

Red: Tested and results indicate critical issues or high risk

Black: Not tested due to lack of applicability to the current project

Grey: Not evaluated in conjunction with implementing agile processes must provide a justification as to why the AoE is not being valuated

Additional comments should include details as necessary to describe the specific information supporting the status. Always articulate the operational impact and include details if requirements or processes are not adequate to support testing.

AOE MODULE 1 – CAPABILITY

Purpose This module provides guidance for completing the Capability AoE portion of the

T&E Scorecard. It identifies Capability-specific review and evaluation criteria and provides some example of measures and methods used to support the scoring definitions.

Scope The Capability AoE addresses:

Does the project meet the user community need(s) and is it sufficient for the latest iteration level of the set Program Increment (PI) boundaries?

o Requirements are documented and traceable to a user need.

o Project roadmap and PI goals and objectives are defined.

o Project has plan and process in place to manage requirements.

o Project is determined Operationally Effective through Operational

Validation

Does the project meet technical and operational performance parameters?

o Critical Technical Parameters (CTPs) o Technical Performance Measurement (TPM), o Technical Performance Parameters (TPPs) o Key Performance Parameters (KPPs) o Requirement level acceptance criteria is generated in support of meeting the user community needs.

While aspects associated with performance will likely be assessed as part of the evaluation of the Capacity AoEs, it is important to state under the Capability AoE how well or how fast a capability must perform in order to satisfy mission requirements.

For Your Information: Important information

“The TPM approach, using the techniques of risk analysis and probability, offers a promising method to incorporate technical assessments resulting systematically from technical parameter measurements to derive more discrete management data sufficiently early to allow for cost avoidance.”

Ref: HTTPS://WWW.DAU.EDU/COP/RISK/DAU SPONSORED DOCUMENTS/TPM EV AND RISK

MANAGEMENT.PDF, RETRIEVED ON: SEPTEMBER 23, 2020

https://www.dau.edu/cop/risk/DAU%20Sponsored%20Documents/TPM%20EV%20and%20Risk%20Management.pdf https://www.dau.edu/cop/risk/DAU%20Sponsored%20Documents/TPM%20EV%20and%20Risk%20Management.pdf

Methodology This section presents methods for assessing the Capability AoE through the review and evaluation of each validated requirements against test indicators of success to determine if the capability satisfies user needs while meeting established technical and operational performance parameters. Within agile environments the evaluated performance parameters and acceptance criteria will be indicative of the maturity level of the product within PI boundaries that sufficiently meets the requirements identified for that program iteration.

Review: The main purpose of the Capability AoE is to ensure validated project requirements are derived from established requirements that meet user needs. The initial requirements and design review and analysis process is based on business level and capability level requirements. Business level requirements are high-level requirements that describe the user need in support of the mission(s) and organization(s) need. Capability level requirements establish the operational capabilities KPPs and TPM. These requirements are each evaluated to determine the AoE status, as they should be easily traceable to one or more user needs and thus back to capability or business level requirements. Requirements are later derived into solution, functional, and non-functional requirements; these requirements will be the focus for technical verification. Derived requirements should trace functionality to capability and establish the CTPs & TPPs.

Evaluate: The T&E team determines the extent to which the stated capabilities of the project satisfy user needs and contribute to mission success. Evaluation is further divided into two categories – technical verification and operational validation.

Technical Verification: Technical Verification (TV) is performance oriented.

During technical verification, functional characteristics of the project are exercised and/or tested against CTPs to satisfy the operational need. Functional characteristics of each capability in the project set are based on solution and derived level requirements.

They are additionally based on requirements that describe features and functions of the project to include its qualities and characteristics, in operational environment.

The T&E lead for DevSecOps environments will ensure that periodic code reviews (automated and manual) and built into the process to enhance cyber resilience of the project, improve system design/solutions, verify the code/tests address the associated requirements, and ensure coding styles and standards are maintained supporting long term support. This activities are critical in identifying any potential cyber vulnerabilities prior to the code being compiled and implemented into the build process.

The T&E lead should ensure that each requirements has one or more test cases that will exercise appropriate CTPs to evaluate technical performance. Every function should have an associated performance parameter that establishes how well and how fast the function must perform in order to support the users’ needs.

Operational Validation: During Operational Validation (OV), the capability level requirements of the project are exercised in an operationally relevant environment with an operationally representative set of users. Operational validation is oriented to user needs. The operational validation should trace operational activities back to the business and capability level requirements. Evaluator(s) validate the implementation plan of the project and ensure that the performance of capabilities meets the users’ need(s).

Metrics and measures are collected from technical verification and operational validation and are analyzed to determine how well the criteria have been met. At a minimum, all key capabilities should be exercised to ensure all associated KPPs are met. It is highly recommended that each capability have at least one associated KPP.

Table M1-1. Capability Methodology Overview

Issue Elements of Evaluation

Indicators of Success

Review

Evaluate

Technical Verification

Operational Validation

Does the project meet the user community need(s)?

Capability #1 through Capability #n*

One or more user needs are fulfilled by this capability

Identify project/mission requirements traced to user needs

Functional testing:

% of test cases successfully completed

OA, LUT, or FA:

% of use cases successfully completed

Does the project meet technical and operational performance parameters?

Capability #1 through Capability #n*

CTPs and KPPs are established and met

Identify technical and operational performance parameters for each capability

Performance testing:

CTPs met

OA, LUT, or FA

KPPs met

LEGEND:

# Number CTP Critical Technical Parameters FA Field Assessment KPP Key Performance Parameter

LUT Limited User Test OA Operational Assessment

* Assessed for each capability, “#n” represents 2nd – last valid capability

AOE MODULE 2 – INTEROPERABILITY

This module provides guidance for completing the Interoperability AoE portion of the T&E Scorecard. It identifies Interoperability specific review and evaluation criteria and provides some example of measures and methods used to support the scoring

This AoE provides an evaluation framework based on the Net Ready

Performance Attributes and required architecture viewpoints. Interoperability must be evaluated early and with sufficient frequency throughout a system's life cycle to capture and assess changes affecting interoperability. The following summarizes the components for the EF:

Supports Military Operations – Requirements Net Ready Performance Attributes define net-centric operational tasks

Entering and being Managed on the Network – Network entrance and management performance

Effective Information Exchanges, completeness and timeliness (includes both internal and external data/information exchanges)

JIC – Project’s achievement of, or progress toward Joint Interoperability Certification

Joint Interoperability Certification Policy. DOD policy dictates that DOD IT components must interoperate to the maximum extent practicable, with existing and planned projects, equipment of joint, combined, coalition forces, other U.S. Government departments and agencies, and non-governmental organizations, as required based on operational context. All IT, including defense acquisition, procurement programs, and enterprise services, must have the Net Ready Performance Attribute as part of its interoperability requirements documentation. The Net Ready Content consists of performance attributes and approved associated DOD architectures, and is used to assess both the technical exchange of information, data, services, and the end-to-end operational effectiveness of those exchanges. The Key Performance Attributes and a set of DOD Architecture Framework (DODAF) viewpoint to provide the joint interoperability requirements for JIC determination. DOD Instruction (DODI) 8330.01 provides current DOD interoperability policy and procedures for certifying the joint interoperability of IT and National Security Systems (NSS). The documents are located at:

HTTPS://WWW.ESD.WHS.MIL/PORTALS/54/DOCUMENTS/DD/ISSUANCES/DODI/83

3001P.PDF

https://www.esd.whs.mil/portals/54/documents/dd/issuances/dodi/833001p.pdf https://www.esd.whs.mil/portals/54/documents/dd/issuances/dodi/833001p.pdf

JITC reviews the interoperability requirements and architecture products to develop a test methodology. Then it evaluates to determine if the technical solution meets the interoperability requirements for information exchanges, network entry, and management and support of military operations.

Review: Evaluators from the Operational test team and Interoperability test team jointly determine the extent to which the project has identified the:

Mission objectives and capabilities the project is intended to accomplish.

Net-ready operational tasks needed to accomplish mission objectives.

Information exchanges for those information elements.

Conditions under which the missions and tasks will be performed.

Networks the project must connect to support net-centric operations.

The test teams should perform these activities jointly to minimize the duplication of team efforts and leverage early developmental testing to assess standards conformance to ensure the system is pursuing a viable pathway to interoperability objectives. Project documentation should provide the project's intended mission objectives, capabilities, and requirements. These documents should also identify the networks the project must connect to (internal, external-joint, and/or external non-joint) and how well those networks must perform in order to support the project’s mission objectives, capabilities, and requirements.

Architecture artifacts should provide information regarding the data/information exchanges, data/information elements needed to support the intended capabilities, and the mission tasks supported by those data/information exchanges. The JITC Interoperability Process Guide provides the list of required and conditional DODAF viewpoints needed for Joint Interoperability Certification to make a JIC. These viewpoints must define the project's operational activities, relationships between the operational activities, and the project's functions, as well as the data/information exchanges needed to accomplish the operational activities and the characteristics of the data exchanged between projects. In addition, evaluators from the Interoperability Test Team will:

Review the project's interoperability requirements Net Ready Key Performance Attributes and architecture viewpoints and provide input if necessary.

Determine the Interoperability Certification status

Ensure Architecture Viewpoints are approved

Determine the extent to which joint or component interoperability requirements are traceable to architecture viewpoints, as defined in Net Ready Key Performance Attributes.

Assign a level of risk for proceeding to the next phase based on this determination and/or ensure a risk mitigation plan is established

Evaluate: The evaluation of the Interoperability AoE focuses on the verification of technical attributes (e.g., information exchanges and standards conformance) in a lab/test environment, if available, and the validation of operational activities in a production representative environment.

Technical Verification: Technical verification of interoperability focuses on performance measures associated with data/information exchange requirements and standards conformance.

Data/Information exchange requirements in support of capability should be evaluated in a lab/test environment prior to deployment to the intended operational environment. The verification is based on the ability of the project to meet information exchange performance measures which should include, but are not limited to, timeliness, completeness, and accuracy for each information exchange.

Data/Information exchange requirements supporting the ability to enter and be managed on the network should also be verified prior to deployment. All performance measures for data/information exchanges and associated with network and application management and monitoring systems, to include cyber defense systems, should be verified in the lab environment.

For data/information exchange requirements dependent upon high-risk standards, conformance verification may be necessary and can be done in a lab/test environment or through analysis of a standards conformance certification provided by the vendor.

Operational Validation: Operational Validation is focused on the ability of the system to support military operations. The assessment should be conducted in an operationally realistic environment with relevant user participation. The assessment includes an evaluation of the operational requirements associated with information exchanges, as well as application/network management and monitoring capabilities. The evaluator will determine the following for assessing the status of the AoE:

The extent to which the thresholds for the measures associated with the “Supports Military Operations” and the “Exchange Information” requirements have been met

The projects status toward achieving a Joint Interoperability Certification

The project’s status towards completing interoperability reporting

The project’s ability to enter and be managed on the network

Validate the system meets the certified net ready requirements threshold criteria

Tables M2-1.1 through M2-1.4 provide the Interoperability Methodology overview.

Table M2-1.1. Interoperability Methodology Overview – Supports Military Operations

Issue Elements of Evaluation Indicators of

Success Review

Evaluate

Technical Verification

Operational Validation

Supports Military Operations

Does the project exchange information, entered and managed in the network, and supports military operations?

Net-centric operational tasks the project supports?

Operational Performance Parameters are associated with each operational task?

Conditions under which the missions and task will be performed?

Operational tasks are successfully completed

Operational Performance Parameters established and met

Documentation should describe the military operations and activities the project will support and establish operational performance parameters for each activity.

User Acceptance Testing

% of use cases successfully completed

OA, LUT, or FA

% of mission threads successfully exercised

Operational Performance Parameter thresholds met

LEGEND:

FA Field Assessment OA Operational Assessment

Table M2-1.2. Interoperability Methodology Overview – Enter and Be Managed on the Network

Elements of Evaluation

Indicators of Success

Review

Evaluate

Technical Verification

Operational

Enter and Be Managed on the

Network

Has the IT project identified:

Which networks the project must connect to in order to support net-centric military operations?

The systems the project must connect to in order to be managed on the network

Network Entry

Connection to network management and monitoring systems

Project has identified the networks it must connect to in order to support net-centric military operations

Data/Information exchanges with network management and monitoring systems are identified

Documentation should describe the networks and monitoring/management systems that the project must connect to, as well as timeliness and accuracy requirements for each associated information exchange.

Integration Testing

Ability to connect to and from identified networks necessary to support net-centric military operations

Timeliness and accuracy criteria are met for all information exchanges

OA, LUT, or FA

Ability to connect from operational networks validated

Network monitoring and management processes exercised and validated

LEGEND:

FA Field Assessment LUT Limited User Test IT Information Technology OA Operational Assessment

Table M2-1.3. Interoperability Methodology Overview – Effective Information Exchanges

Evaluation

Indicators of Success

Review

Evaluate

Technical Verification

Operational

Effective Data/Information

Exchanges

Has the IT project identified the:

The data/information elements produced and consumed by each mission and net-ready operational tasks?

The internal and external data/information exchanges?

Timeliness and accuracy criteria for each data/information exchange?

Internal data/information exchanges

External data/information exchanges

Joint data/information exchanges

Project has identified the data/information elements produced and consumed by each mission and net-ready operations tasks.

Internal and external interfaces are identified.

Timeliness and Accuracy thresholds are established for each data/information exchange.

Documentation should describe all internal, external, and joint interfaces and establish timeliness and accuracy thresholds for each data/information exchange.

Integration Testing

Verify that timeliness and accuracy criteria is met for all data/information exchanges.

OA, LUT, or FA

Validate the data/information elements produced and consumed by each mission and net-ready operational task.

LEGEND:

FA Field Assessment LUT Limited User Test IT Information Technology OA Operational Assessment

Table M2-1.4. Interoperability Methodology Overview – Joint Interoperability Certification (JIC)

Evaluation

Indicators of Success

Review

Evaluate

Technical Verification

Operational Validation

Joint Interoperability Certification

Has the IT project been granted or is it making progress toward Net-Ready and Interoperability Certification?

The Project's:

JS Net Ready Certification

JIC or “JIC with Conditions”

JS Net-Ready certification or waiver to policy granted and interim Certificate to Operate status

Unless waiver to policy is granted by the JS, a JIC or a “JIC with Conditions"

The Interoperability Test Team:

Determines whether the IT project has received a JS Net-Ready Certification

Reviews developmental and standards conformance test results for remaining issues

Verifies publication of NR Certification

Verifies publication of Joint Interoperability Certification

Standards Conformance

Integration Testing

OA, LUT, or FA

Validate that the system supports net-centric Military Operations in accordance with the Net-Ready Certification

LEGEND:

FA Field Assessment JIC Joint Interoperability Certification JS Joint Staff

IT Information Technology

Table M2-2 provides an example of how NR KPPs might be measured for a fictional project.

Table M2-2. Example Net-Ready Performance Attributes

Net Ready Performance

Attribute Net Ready Performance Measures Threshold Objective

Support Military Operations

MOE: Issuance of Certificates

MOP: Percent of successfully issued certificates

Conditions: Certificate Authority keys operational

≥ 97% ≥ 99%

Enter and Be Managed on the Network

Network: SIPRNET

MOP: Time to connect to an operational network from power up

MOP: Operational Availability

≤ 2 minutes

≥ 99.8%

≤ 1 minute

≥ 99.9%

Effective Information Exchanges

Data/Information Element: GDS

MOP: Time to consume GDS data

MOP: Data latency

≤ 4 seconds

≤ 5 seconds

≤ 2 seconds

≤ 2 seconds

LEGEND:

% Percent GDS Global Directory Service MOE Measure of Effectiveness

MOP Measure of Performance SIPRNET Secret Internet Protocol Router Network

Table M2-3 provides checklist questions and sources. Meeting the review criteria of this module will enable program managers to address NRRB concerns.

Table M2-3. Interoperability Readiness Checklist Questions

Interoperability Checklist Questions

Source

If the new/upgraded project will not solely be used by a single C/S/A (i.e., with no joint system interfaces and/or data/information exchanges), have any joint system interfaces and/or information exchanges been identified?

SPC/ARB

If the new/upgraded project has joint interfaces, have all interoperability requirements been specified in measurable and testable forms? Are the requirement documents appropriate for the acquisition pathway (e.g., AAF, CNS, CIP, ISP), complete?

SPC/ARB

Have any cross program, agency or interoperability requirements critical to the success of the program, and the nature of these dependencies (i.e., functional, data, technical, and schedule), been identified?

CEP

Has it been identified how the capability will provide situational awareness of its performance and availability to the supporting DNCs and the DCC via machine-to-machine interfaces with DISA standard monitoring tools?

CEP

For joint systems, has the JIEP* been completed? Have the ITPs been written for individual test/data collection events? Is an existing Joint Interoperability Certification valid?

NRRB

For joint systems, are the Interoperability Evaluation Report, Standards Conformance Test Report, and other C/S/A Test Results available?

NRRB

NOTE: Optional – Describes the requirements and procedures when the details of implementation needed for an ITP such as time, place, and resources are unknown.

If these details are known, a JIEP is not needed.

LEGEND:

AAF

ARB

C/S/A

CEP

CIP

CNS

COCOM

DCC

DISA

DNC

Adaptive Acquisition Framework Acquisition Review Board Commands/Services/Agencies Chief Engineers Panel Capability Implementation Plan Capability Needs Statement Combatant Command DISA Command Center Defense Information Systems Agency DISA NetOps Centers

JIEP

i.e.

ISP

ITP

NetOps

NRRB

SPC

Joint Interoperability Evaluation Plan In Other Words Information Support Plan Interoperability Test Plan Network Operations NetOps Readiness Review Board Service Portfolio Council

AOE Module 3 – Availability

This module provides guidance for completing the Availability AoE portion of the

T&E Scorecard. This module identifies Availability specific review and evaluation criteria and provides some example of measures and methods used to support the scoring definitions.

Scope This module describes a three-phased approach for assessing the area of

Availability which addresses:

Project meets documented thresholds and/or Service Level Agreements (SLAs) – Operational availability (Ao), reliability and maintainability metrics are verified.

Failover, redundancy and restoration from backup processes are documented and exercised

System monitoring and alerts – Are availability monitoring processes documented, implemented and exercised?

This section presents example methods for assessing the Availability AoE through the review of the project's plan for availability and the evaluation of the project's availability implementation and restoration activities.

Review: Evaluators will review the project's documented availability management strategy, which describes how availability requirements will be achieved and the criteria being met. The process should provide an understanding of the agreed current and future availability demands for the project and its ability to continue operations. Specifically the project should:

Produce and maintain an appropriate up-to-date availability plan that reflects the current and future needs of the Enterprise.

Describe how the project will meet the agreed levels of availability in a cost-effective and timely manner.

Describe how the project is positioned to meet future availability demands with respect to system architecture and evolving infrastructure.

Capture and assess risks in a risk registry and have a mitigation plan for issue resolutions

Observe the full resolution process for service and system recovery activities and reporting.

After reviewing, evaluators should be able to determine the project's readiness for operation and the level of risk associated with proceeding to the evaluation phase. The plan's ability to address each of the programmatic review considerations is directly related to the level of risk associated with proceeding to the next phase. The plan should adequately address the applicable activities that satisfy the criteria for implementing system recovery and restoration in the event of hardware, software or any cyber related Denial of Service (DoS), failures; during assessments the increased likelihood of satisfying the performance requirements, decrease the risk to project down.

If fewer of the review criteria are addressed, the likelihood of successfully satisfying the performance requirements decreases and the risk associated with proceeding to the evaluation phase increases. Not having a plan or having a plan that fails to adequately address the review criteria would be considered high risk for processing the evaluation phase.

For Your Information

Note: Any cyber related assessment i.e., DoS, will be addressed extensively in the Cyber AoE

Evaluate: Evaluators will determine how well the project satisfies the quantifiable levels of availability the project must provide. Before proceeding, determine whether the documented availability processes, procedures, and measures have been approved. Evaluation is done through technical verification and operational validation.

Technical Verification: These measures are assessed during Developmental Tests. The assessor will review historical past performance data relevant to the system/service availability of critical and non-critical services. Collect and analyze monitored data for incidents, trends and known issues. When appropriate past data are available, quantitative modeling and simulation techniques may be used to evaluate the availability forecasting processes. Forecasting using a moving average, weighted moving average, or other forecasting methods may be used with caution for Mean Time to Restore Service (MTRS) that meets or exceeds performance levels. The quality of a particular forecasting method is limited by the assumptions about the function prepared by the method. If the method assumes the data are linear (a tangent line), then a non-linear function (a curve) will be a marginal fit. Since the accuracy of forecasting diminishes over time, forecasting is not appropriate for long-range decisions

The assessor will follow the guidance of the Service Level Agreement (SLA) for new system initiatives for availability requirements. They will review failover and recovery measures of internal and external variables (i.e., server, network, or environmental factors) that are in accordance with the associated thresholds and objectives in order to obtain a quantifiable truth for system availability.

The Program Management Office (PMO) may analyze the Availability design to include: Service fault analysis, Component fault analysis and a Fault Tree Analysis (FTA) for predicting the fidelity of the system’s availability.

The assessor may undertake a rehearsal of concept drill, tabletop exercise, simulations or other similar comprehensive evaluation activity to exercise or simulate the forecasting of future availability. These tests or activities may be conducted in the controlled environment to facilitate failure analysis and to determine the project's readiness for testing in such an environment. The following technical parameters can be used as success criteria:

Inherent availability (Ai) and long access duration exposure and system scalability

System reliability and maintainability characteristics are met via regression testing

Automatic system recovery/failover implementation activities

Project availability forecasting roles and responsibilities are understood.

Operational Validation: These measures are evaluated during an Operational Test and Evaluation, or Operational Assessment. Testing will be conducted in an operationally representative environment using the expected operational loads and capacities. Non-Developmental Items (NDIs) are components and services that are not under program control; the program will need to acquire Operational Level Agreements (OLAs) and Underpinning Contracts (UCs) that specify the availability levels of those services or components.

The assessors will devise a plan and process for monitoring, managing, and reporting on the availability of system components with regards the stipulated requirements…