Bidders Library Interoperability Test and Evaluation - NR KPP Evaluation Guidebook.docx

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

JITC NET READY

KEY PERFORMANCE PARAMETER (NR KPP) EVALUATION GUIDEBOOK

VERSION 1.0

22 MAY 2015

JITC NET READY KEY PERFORMANCE PARAMETER

(NR KPP) EVALUATION GUIDEBOOK

MAY 2015

Submitted by: Alan R. Rieffer Chief, Strategy and Policy Branch Strategy, Plans and Engineering Division

Approved by:

DOUGLAS J. ORSI

Colonel, USA Commander

Prepared Under the Direction of:

Timothy K. Taylor Joint Interoperability Test Command Fort Huachuca, Arizona

(This page intentionally left blank.)

TABLE OF CONTENTS

TABLE OF CONTENTS (CONTINUED)

APPENDICES

LIST OF FIGURES

LIST OF FIGURES (continued)

LIST OF TABLES

iv

FOREWORD

Purpose

The NR KPP Evaluation Guidebook is a JITC publication written for JITC Action Officers (AOs) and testers, providing guidance for evaluating joint interoperability based on the attributes of the NR KPP.

The Joint Staff NR KPP Manual promulgates the Department of Defense (DoD) process for producing and certifying NR KPP requirements.

Scope The NR KPP Evaluation Guidebook:

· Establishes the framework for evaluating joint interoperability using the attributes of the NR KPP.

· Summarizes the key Department of Defense (DoD) interoperability policies, publications, organizations, roles and responsibilities that shape JITC’s interoperability mission.

· Provides a compilation of information and references to aid the JITC Action Officers (AOs) through the joint interoperability test, evaluation, and certification (TE&C) life cycle (from developing the interoperability requirements through distributing results.

The information is tailored to JITC’s authority to issue a Joint Interoperability Certification (JIC) in accordance with DoD Instruction (DoDI) 8330.01.

How to Use This Guidebook The guide is written in a conversational style with the JITC workforce in mind (JITC AO or test officer, hereafter referred to as AO, "you" or "your"). The main body is organized into a Foreword section and six main sections, numbered 1 – 6, augmented by appendices. All sections use paragraph numbering for ease of navigation and referencing. The six main sections are:

1. Interoperability Polices, Roles, and Certification

2. Net Ready Key Performance Parameter

3. Requirements Generation

4. Planning

5. Test and Evaluation

6. Reporting.

Icons and Special Text Figure 1 shows the NR KPP Evaluation Guidebook Navigation icon. This graphic is located at the start of each section, with the current section highlighted, and containing hyperlinks to the other sections to assist with navigation. Although the guidebook is organized sequentially, the sections are independent, allowing users to jump to the area that addresses their particular information need.

Figure 1. Navigation Icon

Points of Contact Assistance is available from a variety of sources at JITC. Table 1 provides points of contact.

Table 1. Points of Contact

disa.huachuca.jitc.mbx.nr-kpp-helpdesk@mail.mil

disa.huachuca.jitc.mbx.waiver-recommendation@mail.mil

disa.huachuca.jitc.mbx.doc-review@mail.mil

disa.huachuca.jitc.mbx.jt4-e-form-9-review@mail.mil

disa.huachuca.jitc.mbx.stp-support@mail.mil

disa.huachuca.jitc.mbx.cert-panel@mail.mil disa.huachuca.jitc.list.webmaster@mail.mil

NOTE

* This JITC resource is also available to PMOs/Sponsors ** Assistance is for questions about the document being reviewed, not the tool or portal being used

SECTION 1

POLICIES, ROLES, AND CERTIFICATION

1.1 Policy and Guidance

1.2 Notable Excerpts

1.3 Roles and Responsibilities

1.4 Interoperability Test and Evaluation Process Overview

1.5 Overview: Requirements for Joint Interoperability Certification

“The NR KPP must be specified and included in either JCIDS requirements documents or an information support plan (ISP) (for those systems not covered by JCIDS), and must be updated throughout the IT life cycle when changes affect interoperability.”

For Your Information

NR KPP and the required architecture information can be reviewed stand-alone in the Interoperability Assessment Module (IAM) as a Partial ISP or using other means of coordination. This was authorized by the DoD Interoperability Steering Group (ISG) after DoDI 8330.01 was published.

· Title 10, United States Code–Armed Forces. Establishes roles and responsibilities for the armed forces. Sections 2222 and 2223 assign specific interoperability responsibilities.

· DoDI 5000.02 Operation of the Defense Acquisition System. Directs that all acquired IT must be certified for interoperability in accordance with DoDI 8330.01.

· DoDI 8330.01 Interoperability of Information Technology (IT), Including National Security Systems (NSS). This instruction sets interoperability policy, responsibilities, and procedures, and also assigns the responsibilities for NR KPP development and certification. It also requires that IT be evaluated early and frequently, and certified for interoperability prior to fielding.

· Interoperability Process Guide (IPG). Outlines the procedures and documentation required for joint interoperability test and certification, waiver processing, and associated processes and procedures. It provides implementation guidance for DoDI 8330.01.

· Joint Staff publications. Collectively these documents provide policy, procedure, and instruction on implementation of the JCIDS and NR KPP integration. The NR KPP Manual provides the “How To” processes and procedures for NR KPP development, staffing, and certification and refers to the specific Department of Defense Architecture Format (DoDAF) viewpoints that support the NR KPP.

· CJCSI 5123.01 Charter of the Joint Requirements Oversight Council (JROC).

· CJCSI 3170.01 Joint Capabilities Integration and Development System (JCIDS).

· JCIDS Manual (Document published electronically on Intellipedia)

· NR KPP Manual (Document published electronically on Intellipedia)

· CJCSI 6212.01F Net Ready Key Performance Parameter (NR KPP). Established the three attribute NR KPP. The document is cancelled by CJCSI 5123 but authorized for use until 12 May 2015. The content moved to the CJCSI 5123.01, 3170.01 series, and the JCIDS Manual (includes the NR KPP Manual).

These documents are summarized in Figure 1-1 for quick reference.

Figure 1-1. Policy and Guidance Publications

· Gives authority over interoperability to the DoD Chief Information Officer (CIO), Section 2223, IT:

“Additional Responsibilities of DoD CIO, Ensure the interoperability of Information Technology and National Security throughout the DoD.”

1.2.2 DoDI 5000.02 Excerpt

· Requires all programs to certify interoperability using DoDI 8330.01, enclosure 11, paragraph 12:

“IT, Including NSS, Interoperability. The Program Manager will ensure that interoperability certification is achieved in accordance with DoD Instruction 8330.01.”

1.2.3 DoDI 8330.01 Excerpts

· Requires early and frequent evaluation of IT (test early, test often), paragraph 3.c:

“IT interoperability must be evaluated early and with sufficient frequency throughout a system’s life cycle to capture and assess changes affecting interoperability in a joint, multinational, and interagency environment.”

· Requires interoperability testing and certification prior to fielding IT, paragraph 3.c:

“Interoperability testing must be comprehensive, cost effective, and completed, and interoperability certification granted, before fielding of a new IT capability or upgrade to existing IT.”

· Identifies JITC as the DoD’s joint interoperability certifier for IT, enclosure 2, paragraph 2.n:

DISA “n. Directs the DISA Joint Interoperability Test Command (JITC) to:

(1) Evaluate and certify joint, multinational, and interagency IT interoperability for the DoD.

(2) Serve as the Interoperability Certification Authority for all DoD IT with joint, multinational, or interagency interoperability requirements...”

· Establishes the requirement and composition for the NR KPP, paragraph 3.b:

”All IT, including defense acquisition and procurement programs and enterprise services, must have a net ready key performance parameter (NR KPP) as part of its interoperability requirements documentation. The NR KPP consists of measurable and testable performance measures and metrics derived from associated DoD architectures, and is used to assess both the technical exchange of information, data, and services, and the end-to-end operational effectiveness of those exchanges.”

· Gives the Chairman of the Joint Chiefs of Staff (CJCS) responsibility and authority over the NR KPP, enclosure 2, paragraph 13.a-c:

“13. CJCS. In addition to the responsibilities in section 12 of this enclosure, the CJCS:

a. Provides specific guidance on preparation, format, content, timelines for submission, and review of the NR KPP.

b. Establishes policy and procedures for developing, coordinating, and certifying the NR KPP, in coordination with the USD(AT&L)*, the DOT&E*, and the other DoD Component heads.

c. Serves as the NR KPP Certification Authority, as described in Enclosure 3 of this instruction, for all IT with joint, multinational, or interagency interoperability requirements.”

* Under Secretary of Defense (USD); Acquisition, Technology, and Logistics (AT&L); Director, Operational Test and Evaluation (DOT&E).

· Provides for the IPG and requires JITC to publish it, enclosure 2, paragraph 2.n.(4):

“(4) Publish and maintain an Interoperability Process Guide (IPG) outlining all procedures required to support joint, multinational, and interagency interoperability test and certification, ICTO requests, and waiver submissions.” (Interim Certificate to Operate (ICTO))

1.3 Roles and Responsibilities

Organizations with key roles and responsibilities related to joint interoperability. Figure 1-2 summarizes the organizational responsibilities for joint interoperability.

· DoD Chief Information Officer (CIO). The DoD CIO is the DoD’s interoperability authority, with responsibility for defining and enforcing all interoperability policy. It assigns roles and responsibilities across DoD for all interoperability matters, and publishes interoperability policy in the DoDI 8330.01.

· CJCS, JS J-6. The DoD CIO delegated authority for the NR KPP to the J-6 who sets NR KPP requirements and staffing procedures, and certifies joint NR KPPs. They have authority for joint NR KPP certification regardless of the requirements determination system used (JCIDS or non-JCIDS).

· Interoperability Steering Group (ISG). The ISG is the executive agent for interoperability and is tri-chaired by representatives of the DoD CIO, Under Secretary of Defense for Acquisition, Technology and Logistics, and the CJCS. The ISG coordinates interoperability policies and reviews critical interoperability issues. They adjudicate waivers to policy, Interim Certificate to Operate, and joint interoperability certifications.

· JITC. JITC is the joint interoperability certifier for DoD, responsible for conducting interoperability T&E and issuing a JIC. JITC is also responsible for publishing and maintaining the IPG. (Topics and information for IPG updates are provided by ISG members.)

· Program Management Office (PMO)/Sponsor. Authors the requirements documents, including the NR KPP and architecture viewpoints, and develops the IT system.

· Sets policy for interoperability

· Assigns roles and responsibilities

· Approves waivers to interoperability policy

· Approves ICTOs

· Determines if a certified NR KPP is required

· Certifies all joint NR KPPs

· Is the authority for JCIDS and NR KPP Manuals

· Adjudicates interoperability issues/ICTOs

· Venue to resolve interoperability policy questions

· Maintains the OARL

· Collaborates on the IPG

· Develops the ISP

· Writes and staffs the NR KPP

· Develops architecture viewpoints

· Coordinates with the components (CAEs, Agencies), users, and JITC

· DoD’s joint interoperability certifier

· Publishes the IPG

· Reviews joint interoperability requirements for testability

Figure 1-2. Interoperability Responsibilities by Organization

1.4 Interoperability Test and Evaluation Process Overview IPG Paragraph 2.b. and figure 2-2

Figure 1-3. Joint Interoperability Certification Test and Evaluation Process

For a waiver to the JIC requirement, see the waiver to policy guidance in Section 7 of the IPG.

1.5.1 Requirements documents. The PMO/Sponsor is responsible for developing the interoperability requirements documents and associated architecture information. The IPG states the PMO/Sponsor must provide the following information to JITC prior to any test and evaluation activity that will support Joint Interoperability Certification. The reality is we want to leverage results from suitable tests and this may include DT events, which occur before requirements are final. Be cautious of this when you are planning and developing support agreements, and be sure to articulate the risk of developing test plans before requirements are approved. Mandated requirement documents to support a JIC are: IPG Paragraphs 3.d and 3.f.

· JS Certified NR KPP. The JS certified NR KPP identifies joint critical net-centric operational tasks and associated attribute measures (Measures of Effectiveness (MOEs) and Measures of Performance (MOPs)).

· Approved Architecture Information. The specified architecture information must be approved by the Component Acquisition Executive (or delegated authority) to issue a JIC. The architecture information augments the requirements delineated in the JS certified NR KPP (the NR KPP requirements document has a size limitation and might not list all joint critical tasks and supporting attribute measures). The IPG establishes the DoDAF architecture viewpoints that are:

1) Required – The minimum architecture information required for a JIC.

2) Conditional – Architecture information required when the system meets the stated parameters (for example a Service View may be required if the IT is a shared service).

For Your Information

Typically the requirements documents listed above are required to issue a JIC but there are exceptions. Regardless of the form, requirements must be JS certified for JITC to issue a JIC. Refer to the IPG for additional detail.

JITC uses data collected from suitable tests for joint interoperability evaluation when certain provisions are met. These include but are not limited to–

· JITC reviewed the test plan to ensure the right data and sample sizes were collected.

· A production-representative system is tested in a realistic operational environment.

· Test results are provided in a JITC prescribed format.

(DoDI 8330.01, enclosure 3, paragraph 5.e.(2), see Appendix G).

SECTION 2

NET READY KEY PERFORMANCE PARAMETER (NR KPP)

2.1 Support Military Operations (Attribute 1)

2.2 Enter and Be Managed on the Network (Attribute 2)

2.3 Effective Information Exchanges (Attribute 3)

2.4 Framework for Evaluating Joint Interoperability

2.5 Certifying Joint Interoperability – the JIC

2.6 Measures Overview

1) As a Key Performance Parameter. In this context, the NR KPP comprises three attributes forming the framework for evaluating joint interoperability under Department of Defense Instruction (DoDI) 8330.01. This section (Section 2) focuses on the NR KPP as one of the five required KPPs.

2) As a Requirements Document. The term NR KPP can refer to interoperability requirements developed by Program Management Office (PMO)/Sponsor for a specific information technology (IT), and certified by the Joint Staff (JS). In this context, the NR KPP provides testable measures, organized by attributes of the key performance parameter, and applicable technical requirements (such as requirements for standards conformance) used to evaluate interoperability. An IT system’s NR KPP is contained in various documents including the Information Support Plan (ISP) and Capability Production Document (CPD). The JS maintains the process and guidance for developing an NR KPP requirements document. See Section 3 for more detail regarding requirements generation.

3) As a Joint Staff Instruction. The term NR KPP is the title of a retired Chairman of the Joint Chiefs of Staff Instruction (CJCSI). CJCSI 6212.01F, Net Ready Key Performance Parameter (NR KPP), is grandfathered for PMO/Sponsor use through May 2015 for generating IT requirements and documentation. The reference may be present in approved requirements documents and past reports. Refer to Section 1 for more detail regarding current policy and guidance.

For Your Information

A JS certified NR KPP is required to issue a JIC (DoDI 8330.01, enclosure 3, paragraph 3, see Appendix G).

The NR KPP Construct The NR KPP was redefined in March 2012. Figure 2-1 shows that the NR KPP is comprised of three attributes with an operational focus. The attributes depict how the IT system:

· Supports military operations (Attribute 1).

· Enters into and is managed on the network (Attribute 2).

· Effectively exchanges information (Attribute 3).

The Net Ready Key Performance Parameter (NR KPP) is focused on three attributes that include program specific, validated, verifiable performance measures and metrics.

Figure 2-1. Net Ready Key Performance Parameter Focus The NR KPP attributes are presented in this section using the following format:

· Attribute description

· Measures consideration Measures terminology used in this section is described in paragraph 2.6, Measures Overview. See Section 3, Requirements Generation, and Appendix D, Measures of Effectiveness and Measures of Performance, to read more about measures, applicable architecture viewpoints, and the interoperability requirements approval process (influencing a program’s NR KPP and architecture viewpoints).

“Operational tasks are net-centric if they produce information, products, or services for or consume information, products, or services from external IT (including storing information on external IT).”

· In an ideal case, joint critical net-centric operational tasks are based on (or derived from) a mission or mission thread. In lieu of a mission or mission thread, operational tasks or activities should align to mission essential/critical tasks or functions.

· The ‘conditions’ for evaluating measures under Attribute 1 describe the operational context or environment for performing the task that the measure and criteria are relevant (see paragraph 2.6).

· Measures established for this attribute should be such that they support evaluation of the IT’s ability to perform the joint critical task or military operation.

· Use Measures of Performance (MOPs) to evaluate operational tasks and/or activities.

· MOPs will be presented in numerical form whenever possible.

· A combination of qualitative and quantitative measures may be appropriate to evaluate Attribute 1 (for example, to answer the questions, “can we use it?” or “does it work?”)

· Measures of Effectiveness (MOEs) are used to measure mission effectiveness (success) and can be qualitative or quantitative.

For Your Information

For Attribute 1:

· MOEs ensure operational effectiveness of the military mission or operation that the user must accomplish to meet user intended needs.

· MOPs ensure operational performance of the operational activities or tasks that the system performs to user stated requirements.

The question T&E needs to answer, relative to Attribute 2, is, “Can the IT use the network, interface, and/or the transport medium to support the required information exchanges?” To support a JIC, an interoperability evaluation needs to address, under this attribute, the specified network(s) (transport, joint interfaces, etc.) that enable the required information exchanges (identified in Attributes 3) to perform the ‘joint critical net-centric operational tasks’ (identified in Attribute 1).

Attribute 2 is relatively new and does not have a standardized framework of terminology and metrics. Evaluation should examine performance with regards to network entry and ability to be managed when on the network (for example, time to connect, reconnect, throughput, and management processes).

· IT using the Secure Internet Protocol Router Network (SIPRNet) for transport. Evaluation of interoperability under Attribute 2 should examine performance of activities required to ‘enter and be managed’ on the SIPRNet. The evaluation does not need to assess activities within the Department of Defense Information Network (DODIN) infrastructure to provision SIPRNet.

· IT exchanging data over a tactical data link. Evaluation of interoperability under Attribute 2 might examine: (1) ’timeliness’ to enter the data link, (2) performance to manage throughput (timeslot reallocation) and (3) conformance to applicable data link standards (technical requirements).

· IT that utilizes satellite or radio frequency (RF) medium for transport. Evaluation of interoperability under Attribute 2 might examine (1) performance of the interfaces between the IT and the long haul communication capabilities and (2) conformance to applicable interface standards (technical requirements).

· Do they have quantitative MOPs?

· Do they examine the time from system start up to when the system is connected to the network and is supporting military operations (needs to define observable start and complete points)?

· Do they include performance measures for information exchanges required for net entry and management (for example authentication, health and status, net monitoring, or quality-of-service.)?

· Do they specify the conditions for evaluating each measure? Do the conditions describe the operational context or environment, and is it relevant to the measure and criteria? Are technical requirements, such as a standard or specification, used as a condition for a measure? See paragraph 2.6 for information about measures.

· Does the IT use shared data or services? Does the IT support unanticipated users? If so, do the requirements include MOPs to evaluate the IT’s ability to support unanticipated users and other applicable data and service sharing requirements derived from Department of Defense (DoD) data and services strategies (DSS)*?

· Measures to evaluate accessibility (for example publish, consume or subscribe) to shared data that a capability produces or consumes (data stored in a location accessible to multiple authorized users).

· Measures to evaluate the visibility of shared data or a shared service to unanticipated but authorized users.

* Note: Suggestions derived from the DoD DSS, to evaluate interoperability of enterprise services, might not apply to some mission systems. Refer to DSS in Section 4 for more detail.

For Your Information

There is no one-size-fits all process; you need to adapt the concepts and guidance presented here, and throughout this guide, to your particular situation(s).

Attribute 3 Information Exchanges (IEs) are found in, and are related to, the DoDAF resource flows. (Resource flows are interactions between activities: Operational resource flows relate to missions and mission threads; System resource flows relate to information elements.)

· Do they have quantitative MOPs to evaluate information exchange effectiveness (whether an information element or a data element)?

· Do they include at least one or more of the following MOPs:

· Timeliness?

· Completeness?

· Accuracy?

· Do they clearly specify the information format and associated networks or transport required for exchanging information to support an operational task, such as voice communications, data exchanges, or flashing light?

· Do they need to include measures to evaluate information exchanges with unanticipated users, using shared data, or using shared services?

For Your Information

There is no one-size-fits all process; you need to adapt the concepts and guidance presented here, and throughout this guide, to your particular situation.

“Interoperability requirements include both the technical information exchanges and the operational effectiveness of those exchanges.”

Timely Complete Accurate Enter the Network Supporting the operational tasks is the pinnacle of the joint interoperability evaluation Technical Requirements (Impact/Risk)

- ‘Conditions’ to Measures

- Formal Conformance Evaluation

- Derived from attributes Attribute 3 (MOPs)

- “Can the IT effectively exchange info?”

- End-to-End, To/From Shared Data

- Timely, complete, accurate, etc.

- Derived technical requirements Attribute 2 (MOPs)

- “Can the IT use the Network/Transport?”

- Timely network entry

- Info exchange for net-management

- Unanticipated Users, Shared data/services

- Derived technical requirements Support MilOps Attribute1 (MOPs/MOEs)

- “Can the IT be used to support the task”

- Quantitative but could have qualitative elements (e.g. usable)

- Operational Tasks/Activities (MOPs) - Required for JIC

- Mission (MOEs) - Optional for JIC Effective Information Exchanges Derived Technical Requirements (Standards, Specs, ICAs, etc.)

Be Managed On the Network

Figure 2-2. Hierarch of Net Ready Key Performance Parameter

2.4.1 Assumptions and Terminology

For brevity and clarity, the following terms and assumptions are used to describe the interoperability evaluation framework and the JIC (paragraphs 2.4 and 2.5): IPG Paragraph10.b

· The term IT refers to Information Technology system, to include NSS.

· The term measure refers to MOPs and MOEs.

· The term requirements in the context of requirements documents, approved requirements, or similar terms, refers to a certified NR KPP and the required (approved) architecture information.

· The term task refers to operational tasks or activities that the IT must perform under Attribute 1 to support military operations.

· The term joint critical tasks refers to the joint critical net-centric operational tasks, for example, the tasks an IT must perform or support for a JIC.

· The term transport refers to networks, data links, and all other forms of moving information between producer and consumer under Attribute 2 (includes access to shared data and services).

· The term condition has two different contextual meanings. The first is used as a condition to a measure (see paragraph 2.6 and Section 3) and second, as a condition or limitation to a JIC (see paragraph 2.5 and Section 6).

· The term technical requirements refer to standards, specifications, or ICAs, which enable the operationally effective exchange of information and use of the specified transport (networks and other forms of moving information).

· The term certification letter refers to the product that documents a JIC.

· Assumption: requirement documents provide testable measures for all attributes and applicable technical requirements.

· Assumption: attribute measures are the core requirements; applicable technical requirements are derived from the attribute requirements.

· Technical requirements refer to standards, specifications, ICAs, etc., which enable the operationally effective exchange of information to include use of networks and other forms of transport, and are documented in approved requirements.

For Your Information

Conditions to a certification are based on operational impact. Conditions limit the operational use of the IT to only those operational tasks, information exchanges, and networks (transport) that do not present a Major or Moderate operational impact. See Section 6.

2.4.2 Evaluation Framework

· The operational component of interoperability is based on the IT’s mission effectiveness and operational task performance (measures aligned to Attribute 1).

· The technical component of interoperability is based on the ability of the IT to meet performance measures and conform to technical requirements as follows:

· Performance measures (MOPs) that quantify the ability to use the specified transport (for example, to support required information exchanges) are aligned to Attribute 2. These include measures for accessing shared data and services.

· Performance measures (MOPs) that quantify the (operationally) effective exchange of information required to perform the requisite tasks are aligned to Attribute 3.

· Conformance to technical requirements (derived from attributes requirements) are addressed in the evaluation framework as follows:

· Technical requirements can be specified as a prelude condition to a measure (see paragraph 2.6) or

· Discrepancies identified in formal conformance testing (for example, a standards conformance test) are evaluated for risk (potential issue) or impact (known issue) as it relates to the IT meeting the performance measures under the attributes.

For Your Information