Bidders Library Interoperability Test and Evaluation - JITC IOP Service Area SOP v1 0.pdf

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

JITC INTEROPERABILITY

SERVICE AREA

STANDARD OPERATING

PROCEDURES

VERSION 1.0

JULY 2020

JITC INTEROPERABILITY

SERVICE AREA

STANDARD OPERATING

PROCEDURES

The Interoperability Service Area Standard Operating Procedures are developed by the Service Area Leads. It is effective for the Service Area immediately upon signature and publication.

VERSION 1.0

JULY 2020

Submitted by: Tod Wister Interoperability Service Area Lead

Richard Delgado Jr.

JITC Technical Advisor

Approved by: ____________________________________

SHAWN ROBERTS

Captain, USN Commander, Joint Interoperability Test Command

Prepared Under the Direction of:

Michael Creegan Joint Interoperability Test Command

Fort Huachuca, Arizona

This page intentionally left blank.

SUMMARY OF CHANGES

Version Sections Affected Description of Change

1.0 All

Initial development of JITC Interoperability Service Area Standard Operating Procedures i

TABLE OF CONTENTS

Page

1. INTRODUCTION

1.1 Purpose

1.2 Scope

1.3 Applicability

1.4 Implementation

2. ORGANIZATIONAL STRUCTURE

3. ROLES AND RESPONSIBILITIES

3.1 Joint Interoperability Test Command Interoperability Service Area

3.1.1 Interoperability Service Area Mission

3.1.2 Interoperability Service Area Personnel Duties

3.1.3 Interoperability Service Area Position Duty Descriptions

3.1.3.1 Interoperability Service Area Leadership

3.1.3.2 Division Chiefs

3.1.3.3 Branch Chiefs

3.1.3.4 Action Officers

4. JOINT INTEROPERABILITY TEST CYCLE

4.1 Basic Requirements for the Requirements Analysis Framework for Test-Based Test and Evaluation Process

4.1.1 Requirements Analysis Framework for Test

4.1.2 Joint Interoperability Evaluation Plan

4.1.3 Test Concept Brief

4.1.4 Interoperability Test Plan and Test Support Package

4.1.5 Test Readiness Review

4.1.6 Quick Look Report

4.1.7 Test Report

4.1.8 Joint Interoperability Certification

4.2 Interoperability Service Area Test Cycle Detailed Description

4.3 Test Documentation and Planning Activities

4.3.1 Test Documentation Production

4.3.2 Test Documentation and Certification Products

4.3.3 Requirements Analysis Framework for Test

4.3.3.1 Initial Requirements Analysis Framework for Test

4.3.3.2 Intermediate Requirements Analysis Framework for Test

4.3.3.3 Final Requirements Analysis Framework for Test

4.3.4 Requirements Review Brief

4.3.5 Joint Interoperability Evaluation Plan

4.3.6 Test Concept Brief

4.3.7 Interoperability Test Plan and Test Support Package

4.3.7.1 Interoperability Test Plan

4.3.7.2 Test Support Package

4.3.8 Test Readiness Review

ii

4.4 Test Results Reporting

4.4.1 Quick Look Report

4.4.2 Test Report

4.4.3 Joint Interoperability Certification

4.4.4 Joint Interoperability Assessment

4.4.5 Extensions and Recertifications…………………………………………………16

4.5 Test Preparation and Timelines

5. GENERAL GUIDANCE FOR ACTION OFFICERS

5.1 General Operations for the Interoperability Service Area

5.2 Interoperability Test Processes

6. TEST DOCUMENTATION KNOWLEDGE MANAGEMENT

6.1 Joint Interoperability Test Command Data Management Tool

APPENDICES

ACRONYMS ................................................................................................................A-1 REFERENCES .............................................................................................................B-1

LIST OF FIGURES

1 Organizational Structure 2 Test Cycle Framework 3 Interoperability Project Workflow

1. INTRODUCTION

1.1 Purpose

These Interoperability (IOP) Service Area (SA) Standard Operating Procedures (SOP) establish standard processes and procedures to conduct the Joint Interoperability Test Command’s (JITC) test and evaluation (T&E) services in support of JITC’s Joint Interoperability Certification (JIC) mission. This SOP is prescriptive in accordance with (IAW) the JITC Commander’s Policy Letter 2019-03, “Establishment of Service Areas.”

Report any errors or circumstances that prevent applying SOP guidance to a program or system to your branch chief (BC) for resolution. The Interoperability Integrated Product Team (IIPT) will address administrative or documentation issues within this SOP.

1.2 Scope

This SOP augments authoritative Department of Defense (DoD), Defense Information Systems Agency (DISA), and JITC directives and instructions. This SOP concerns only systems that are under the purview and governance of Department of Defense Instruction (DoDI) 8330.01, “Interoperability of Information Technology (IT), Including National Security Systems (NSS).” In case of contradictions or conflicts between the content of this SOP and an authoritative source, the authoritative source governs. Test divisions and branches may have their own SOPs that further delineate processes and procedures unique to them but may not contradict, modify, or otherwise revise processes and procedures set out in this IOP SA SOP.

1.3 Applicability

This SOP applies to all JITC IOP SA government civilian employees, military members assigned within the IOP SA, and contractors supporting IOP SA tasks. As a compliance requirement, JITC Action Officers (AOs) should reference this SOP in all future task orders for all contracted support.

1.4 Implementation

All new or restarting IOP programs shall follow the provisions and framework described within this SOP. Existing DoDI 8330.01-based JIC programs shall transition to the processes and procedures described herein as soon as practical, but no later than one year from initial publication of the SOP.

2. ORGANIZATIONAL STRUCTURE

Figure 1 shows the organizational structure of the JITC IOP SA.

SERVICE AREA LEADS (JITC TECHNICAL ADVISOR/JTC DIVISION CHIEF)

SERVICE AREA

JTC1

C2 Intel., Surveillance & Recon

JTC2

Coalition and Combat Service Support

JTD2

National Assets

JTB1

Focused Logistics Mission Area

JTE3

Communic ations & Interopera bility

JTC4

Tactical Data Links

JTC3

Combat Systems

JTC5

Test Support

Note: JTA will follow IOP Service Area procedures in cases where they are conducting joint interoperability certification.

Organizational Structure

3. ROLES AND RESPONSIBILITIES

3.1 Joint Interoperability Test Command Interoperability Service Area

3.1.1 Interoperability Service Area Mission

The IOP SA mission is to enable an effective, efficient, and defensible JIC of systems as required by DoDI 8330.01, “Interoperability of Information Technology (IT), Including National Security Systems (NSS).”

3.1.2 Interoperability Service Area Personnel Duties

IOP SA personnel perform the following duties:

Conduct consistent, scientifically sound, and repeatable testing

Support the DoD acquisition community by determining the extent to which systems meet technical standards and interoperate by conducting:

o Requirements review o Pre-test planning o Standards conformance testing o IOP testing and evaluation o Data collection and analysis o Post-test reporting

Use hardware-in-the-loop and operationally realistic environments (when feasible) to validate system implementation of their joint IOP requirements

Evaluate system IOP in a joint or operationally relevant environment to make a JIC determination

3.1.3 Interoperability Service Area Position Duty Descriptions

3.1.3.1 Interoperability Service Area Leadership

IAW the JITC Commander's Policy Letter 2019-03, SA leadership consists of the JTC Division Chief (IOP SA Lead) and the JITC Technical Advisor. The IOP SA Lead has ownership of SA processes, procedures, and products. Additionally, the IOP SA Lead produces the IOP SA SOP to serve as the authoritative source for execution of SA T&E services. The IOP SA Lead co-chairs the IIPT with the JITC Technical Advisor.

The JITC Technical Advisor advises the IOP SA Lead and coordinates with other SA leads to ensure Command-wide unity of effort and alignment with JITC’s strategic objectives.

3.1.3.2 Division Chiefs

The JITC division chiefs (DCs) of each participating SA division are senior civilians responsible for their division’s missions. They ensure their division’s compliance with the procedures in this SOP and with any additional division procedures that they deem necessary to meet the SA’s mission.

3.1.3.3 Branch Chiefs

The JITC BCs of each participating SA branch are senior civilians responsible for their branch’s missions and their branch AOs. The BCs:

Serve as first-line supervisors for direct support personnel (e.g., AOs).

Ensure that AOs adhere to the requirements of the IOP SA SOP, including properly preparing a Requirements Analysis Framework for Test (RAFT) and the Test Concept, Test Readiness Review (TRR), and associated briefs. A particular focus is ensuring the maturity level and completeness of program RAFTs.

3.1.3.4 Action Officers

The AOs report to their respective BCs and perform the following duties.

Plan and execute IOP T&E of the systems assigned to their respective branch

Manage and execute the day-to-day efforts to achieve IOP certification for the programs in their respective branch

Prepare the Plan of Action and Milestones (POA&M) and any required support agreements (e.g., Fiscal Service Form 7600A) for each assigned program

Maintain an up-to-date cost/schedule/performance quad chart for each program in their respective branch

4. JOINT INTEROPERABILITY TEST CYCLE

Figure 2 shows the test cycle framework for the three primary types of testing:

JITC-led testing, Service testing with JITC participation, and Service testing without JITC participation. Section 4.2 describes these categories. Timeline goals for the various products/activities will need tailoring for agile (e.g., middle tier) development efforts. Agile development will be an evolving topic in future iterations of the SOP.

Requirements

Trace

Evaluation

Strategy

Data

Management and

Analysis Plan

Requirements Analysis Framework for Test (RAFT)

JITC-conducted

Test Event(s)

TCB T-120 TRR T-30

Joint

Interoperability

Evaluation

Plan (JIEP)

Service Testing

Without JITC

Participation

TCB T-90 TRR T-15

Test Support

Package(s)

(Draft T -60 / Final T -30)

Test Events

RAFT

Certification/

Assessment

Report (Final T +60)

Quick Look

Report(s)

(T +30)

Certification/

Assessment

Letter (Final T +60)

System

Requirements

(NR KPP: MOPs, MOEs and derived from architectures

Service Testing

With JITC

Participation

Tech Advisor

Approval

(CWL)

RAFT

All test results are reconciled against a system s RAFT

Interoperability Test

Plan(s) & Procedures (Draft T -90 / Final T -60)

Test Events

Service Test

Results

Service Test

Plans

Test Events

TCB T-90

Coordinate against RAFT

Limited applicability

Without prior coordination

Report

(T +60)

Test Products Audit

Form (TPAF)

TRR T-30 Tech Advisor

Concurrence

(CWL)

Tech Advisor

Approval

(CWL)

Tech Advisor

Approval

(CWL)

LEGEND:

CWL Commander's Watch List NR KPP Net-Ready Key Performance Parameter DoDAF Department of Defense Architecture Framework RAFT Requirements Analysis Framework for Test JIEP Joint Interoperability Evaluation Plan T Test JITC Joint Interoperability Test Command TCB Test Concept Brief

MOE Measure of Effectiveness TRR Test Readiness Review

MOP Measure of Performance

Test Cycle Framework

When implementing the test cycle, the test team must first and foremost establish a set framework that creates a defined, repeatable, and defensible process based on an underlying, rigorous analysis (i.e., RAFT). This framework should allow sufficient flexibility for programmatic differences. Second, for Commander’s Watch List (CWL) programs, the test teams should engage early with JT4A via their review of the system’s RAFT and Joint Interoperability Evaluation Plans (JIEPs). Delaying JT4A’s oversight activities could hamper the ability to influence test program execution.

4.1 Basic Requirements for the Requirements Analysis Framework for

Test-Based Test and Evaluation Process

4.1.1 Requirements Analysis Framework for Test

All DoDI 8330 certifications are based on a RAFT. RAFTs are based on the Joint Staff-approved Net-Ready Key Performance Parameter (NR KPP) and associated architecture viewpoints. Both JT4A and the IOP SA Lead review and concur with RAFTs for CWL programs.

4.1.2 Joint Interoperability Evaluation Plan

All DoDI 8330 certifications have a JIEP. The core component of the JIEP is the system-specific RAFT. The JIEP is a living document that is updated as necessary throughout the life of the test program. Both JT4A and the IOP SA Lead review and concur with JIEPs for CWL programs.

4.1.3 Test Concept Brief

A Test Concept Brief (TCB) is conducted at least once early in the joint IOP test cycle. TCBs for CWL programs are actual briefings, which must include invitations to select individuals/organizations (or their representatives) external to the producing division (specifically, the JITC Technical Advisor, IOP SA Lead, JITC National Capital Region (NCR) Liaison Officer (Command Staff), and JT4A). Non-CWL program TCBs can be in person or a DC/BC review of TCB briefing slides (at the DC’s discretion).

4.1.4 Interoperability Test Plan and Test Support Package

An Interoperability Test Plan (ITP) is required for JITC-led test events. A Test Support Package (TSP) is required for test events where JITC is participating but is not the Responsible Test Organization (RTO). Planning focuses on the method of collecting the data necessary to resolve a system’s joint IOP requirements. Both JT4A and the IOP SA Lead review and concur with ITPs and TSPs for CWL programs.

4.1.5 Test Readiness Review

A TRR is conducted prior to each test event. TRRs for CWL programs are actual briefings, which must include invitations to select individuals/organizations (or their representatives) external to the producing division (specifically, the JITC Technical Advisor, IOP SA Lead, JITC NCR Liaison Officer, and JT4A). Non-CWL program TRRs can be briefings or a DC/BC review of TRR briefing slides (at the DC’s discretion).

4.1.6 Quick Look Report

The Quick Look Report (QLR), when requested by a customer, is limited in scope to characterizing the conduct of the test event itself. It describes the scope of testing that occurred during the event and the successes/failures in satisfying data collection objectives. It may address anecdotally the overall effectiveness of the test event, but may not address system performance. DCs review and approve QLRs for programs in their divisions.

4.1.7 Test Report

The Test Report (TR) is the product of a single test event, from either a JITC-led test event or Service test with JITC participation. It includes JITC’s analysis of the data and the evaluation of the system’s performance relative to the joint IOP requirements in the RAFT based on that analysis. Both JT4A and the IOP SA Lead review and concur with TRs for CWL programs.

4.1.8 Joint Interoperability Certification

The JIC is the final product of the totality of JITC’s analysis and evaluation of joint IOP testing for a given version of a system. Both JT4A and the IOP SA Lead review and concur with JICs for CWL programs.

4.2 Interoperability Service Area Test Cycle Detailed Description

The IOP SA uses three categories of testing to make JIC determinations:

JITC-led Testing: This testing is specifically conducted to produce relevant data to support a JIC. It is a dedicated test event (or multiple events) that does not leverage other planned program test events. Therefore, it is the most costly category and potentially more time-consuming for the program.

Service Testing with JITC Participation (Responsible AO or Representative):

This consists of a Service-conducted test activity (developmental test (DT) or operational test (OT)) where JITC’s requirements are coordinated with the Service Program Management Office’s (PMO) Program Manager (PM). The AO is responsible for including the PMO’s required test cards (TCs) into the RTO’s test plan and for delivering test results to JITC. Products from this category of testing may be a single test report or multiple test reports.

Service Testing Without JITC Participation: JITC leverages, to the extent possible, Service test results from test events in which JITC did not participate and may not have had input. The applicability of such results may be limited because Service testing often does not capture JITC’s requirements when no coordination has occurred. This is a last option, because it may result in a JIC with conditions and may require separate testing to capture required test data. The AO coordinates with the PMO to determine the way forward.

4.3 Test Documentation and Planning Activities

Test documentation exists at varying levels of maturity throughout a program or project’s life cycle. The AOs, depending on when a PMO engages JITC for support, can receive a variety of program documents—such as the Initial Capabilities Document (ICD), Capability Development Document (CDD), Capability Production Document (CPD), Information Support Plan (ISP), or other requirements documentation—to prepare for testing. Based on the maturity of the requirements in the applicable document, AOs may be able to completely address the evaluation strategy, test measure(s), or data analysis corresponding to the information exchange (IE) capability of the program or the system under test (SUT).

Due to the varying stages of the requirements document’s maturity, JITC test documentation corresponds to the requirements document’s maturity. To begin to understand the SUT, AOs start requirements analysis immediately after receiving notification of a PMO’s request and funding for IOP support. AOs begin the RAFT as soon as possible after receiving the SUT NR KPP and/or DoD Architectural Framework (DoDAF) architecture viewpoints.

If a system does not have testable or measurable requirements, or other substantive requirements-related issues are discovered, the AO must coordinate with the PMO/sponsor and Joint Staff J6 to resolve the issues in time for testing.

To assist JITC AOs, the joint IOP help desk can answer questions related to joint

IOP test, evaluation, and certification based on DoDI 8330.01. The help desk email address is disa.huachuca.jt.mbx.joint-interoperability-helpdesk@mail.mil.

4.3.1 Test Documentation Production

While the AO is ultimately responsible for creation, adjudication, and approval of all test documentation, JTC5 has resources to assist an AO on a reimbursable basis.

The AO can coordinate with JTC5 to obtain document support for each IOP test program. JTC5 has the document templates, workforce (either government or contract support), and skills needed to draft the documents for an IOP test program. The AO uses the draft document to fill in any necessary details, complete the analysis needed, and submit the completed document for approval.

mailto:disa.huachuca.jt.mbx.joint-interoperability-helpdesk@mail.mil

The SA has a set of mandatory templates and a series of notional templates and briefings to assist in the IOP T&E process. A templates folder will be maintained on the IIPT SharePoint site, https://disa.deps.mil/org/JITC/IPTS/IIPT/SitePages/Home.aspx, with separate folders for mandatory and notional.

4.3.2 Test Documentation and Certification Products

The pre-test generation and approval process for test documentation is as follows:

1. AO collects DoDAF architecture viewpoints and either loads them into the

JITC Data Management Tool (JDMT) or has JTC5 load them into it.

2. AO uses the JDMT to generate the initial RAFT.

3. AO presents the Requirements Review Brief (RRB) to the IOP SA Lead, DC, and appropriate BC. For CWL programs, JT4A is also included.

4. AO creates the initial draft of the JIEP.

5. AO, PMO, and Joint Staff J6 (when needed) resolve discontinuities, data gaps, and inconsistencies.

6. AO uses the JDMT to generate the intermediate RAFT.

7. AO and PMO create and complete the Data Management and Analysis Plan

(DMAP).

8. AO creates the final JIEP with the intermediate RAFT and DMAP. For CWL programs, both JT4A and the IOP SA Lead review and concur.

9. AO presents the TCB to the IOP SA Lead, DC (if not the same), and appropriate BC. For CWL programs, JT4A is also included.

10. AO uses the JDMT to generate the final RAFT.

11. AO uses the JDMT to generate test cards.

12. AO creates the ITP or TSP including TCs and the final RAFT. For CWL programs, both JT4A and the IOP SA Lead review and concur.

13. Appropriate level of leadership signs the ITP or TSP and the final RAFT.

The post-test generation and approval process for report and certification products is as follows:

1. AO generates the QLR (if required).

2. DC approves the QLR (if a QLR is produced).

3. AO generates the TR. For CWL programs, both JT4A and the IOP SA Lead review and concur.

4. DC approves the TR.

5. AO generates the IOP certification document. For CWL programs, both

JT4A and the IOP SA Lead review and concur.

6. DC signs the IOP certification document.

https://disa.deps.mil/org/JITC/IPTS/IIPT/SitePages/Home.aspx

4.3.3 Requirements Analysis Framework for Test

All JIC test programs are supported by an underlying RAFT, which deconstructs and organizes a system’s requirements and IE-level test strategies as well as DMAPs.

RAFT generation is a standardized approach to tracing requirements and defining detailed test strategies, data sources, and analysis methods into one Microsoft Excel spreadsheet. The RAFT is tied to DoDAF architecture viewpoints and the program’s NR KPP table.

The RAFT provides a holistic, executable framework for the IOP evaluation. It clarifies NR KPP and associated DoDAF requirements, includes embedded TCs to simplify support of individual events, and improves accountability and tracking of certification requirements for all stakeholders. The RAFT also serves to generate discussions between system stakeholders to resolve differences related to requirements and requirements attributes and to determine testing methodologies.

The RAFT process leads to more complete reporting of the IOP status. All test results are entered into the RAFT, providing a repository of concise performance information from which to conduct analysis and reach conclusions. Because the information exchange requirements (IERs) are coupled to the NR KPP, the focus is on operational impacts to the users.

Completion of the RAFT results in tighter coupling between desired warfighting capabilities and acquisition community requirements. It clearly illustrates missing requirements related to NR KPP measures and orphaned IERs (i.e., IERs not supported within other architectural viewpoints).

The RAFT is a living document. The stages of RAFT development are initial, intermediate, and final. The BCs confirm the stage of the RAFT (should be final) prior to authorizing a TRR with the DC.

4.3.3.1 Initial Requirements Analysis Framework for Test

At the program’s start, the AO should receive (or request) the initial requirements documentation from the PMO or as part of the Joint Staff review process. Typically, the initial requirements documents include the ICD, CDD, CPD, ISP, Capability Implementation Plan (CIP), Capability Needs Statement (CNS), or other requirements documentation.

The Joint Staff requirements review process begins with the system documentation hosted on either the Global Information Grid (GIG) Technical Guidance Federation (GTG-F) site (https://gtg.csd.disa.mil) or the Knowledge Management/ Decision Support system (only on the Secret Internet Protocol Router Network (SIPRNet)). When conducting these requirements reviews, it is important that AOs examine the DoDAF products for accuracy; that is, the IERs should be understandable and sufficient (have measurable metrics), as well as traceable throughout the

Operational Viewpoints (OVs) and Systems Viewpoints (SVs). This is the first opportunity to identify any inconsistencies in the architecture viewpoints.

Using the DoDAF architecture viewpoints and NR KPP, the AO should determine whether all the mandated JITC “Interoperability Process Guide” (IPG) views have been included. If not, the AO should coordinate with the PMO for the missing viewpoints.

The next step is to produce the initial RAFT. The goal of the initial RAFT is to determine whether the program requirements have traceability with associated measures throughout the IERs, from the NR KPP through the OVs to the SVs. This review should quickly identify any inconsistencies in the broader RAFT view that are more difficult to discern in the individual DoDAF views. More specifically, the initial RAFT contains the IEs of both the Operational Resource Flow Matrix (OV-3) and the Systems Resource Flow Matrix (SV-6) and makes traceability errors more immediately apparent. The TCs are not generated from the IER test threads at this point because the inconsistencies have yet to be resolved. The initial RAFT should not be included in the program's JIEP or TSP. Once the initial RAFT is complete and has been reviewed by the appropriate BC, the AO can send the RAFT to the PMO for inconsistency resolution.

Inconsistency resolution is both a process and an event. The process aims to eliminate any missing information within the architecture viewpoints, building complete IER test thread(s) within the RAFT to include the appropriate measure for the particular IER. The AO should arrange a meeting with the PMO (typically, the PMO assigns a systems engineer and operational user representative). The PMO representative(s) should be able to speak authoritatively on how the SUT operates and how the IEs and networks operate to execute mission activities. It may be beneficial to have those who wrote the IERs and created the DoDAF architecture viewpoints present for the discussion. Completing this step advances the RAFT, with no information missing in any DoDAF viewpoints, to the intermediate stage.

4.3.3.2 Intermediate Requirements Analysis Framework for Test

Once the inconsistencies in the initial RAFT have been resolved, the intermediate RAFT may be created. The initial RAFT documents all IEs contained within the OV-3 and the SV-6, whereas the intermediate RAFT documents those IEs that JITC tests. The scope of the RAFT and IOP evaluation framework (EF) is one of the outcomes from the discussions with the PMO.

The intermediate RAFT is sufficient to generate the AO view version of the RAFT.

The intermediate RAFT still contains all columns as defined by the Joint Staff J6 “Warfighting Mission Area (WMA) Architecture Development Standard,” whereas the AO’s version contains only the columns with the data of interest to the AO.

The next step after creating the intermediate RAFT is to begin the DMAP portion of the RAFT and then to generate TCs. At this point, the DMAP is immature and the TCs lack specific details, but the overall RAFT construct is complete.

To complete the RAFT DMAP, the AO coordinates with the PMO and their personnel, which should include authoritative IOP representatives and systems engineers (if beneficial). These representatives should be knowledgeable about the SUT, IEs, and networks, as well as the system’s source for specific IER data extraction and the tools necessary to read the extracted data. The output of this process is a completed DMAP. Each mission attribute, network, and IER row has a specific methodology, metric, toolset (if necessary), extraction point, and data collector instructions recorded in the RAFT. Methodologies are written to describe to a data collector who, what, when, where, why (the five w’s), and how the attributes, networks, and IERs are collected and recorded.

For CWL programs, the intermediate RAFT must have a completed DMAP and be staffed to JT4A for concurrence. JT4A reviews the requirements and T&E policy and concurs with the intermediate RAFT. Upon concurrence, the intermediate RAFT is included in the system JIEP and becomes the approved JITC requirements traceability baseline. Any changes to the requirements as the RAFT is finalized require the RAFT to be restaffed to and concurred by JT4A.

4.3.3.3 Final Requirements Analysis Framework for Test

The final RAFT is used to evaluate the IOP of the SUT. At this stage, every component of the RAFT, including the DMAP, is complete, and the RAFT is traceable through the NR KPP attributes down to the IER/TC level. The RAFT’s traceability should identify missions, networks, and activities.

DMAP completion means that the five w’s and how for each IER have been negotiated with the PMO systems engineer. Additionally, for each attribute, network, and IER, a specific methodology to collect the appropriate system data has been identified, the data format and definition determined, the necessary toolset identified and procured, and all DMAP fields populated in the RAFT. With all this information, the TCs can be finalized. After RTO coordination, the final TCs can be included in the TSP and delivered to the RTO collecting IOP data.

The TCs (one TC per IER, addressing NR KPP Attribute 3) include the necessary data elements, methodology, performance attributes (measures), and space to record the results of the test. Similarly, TCs for network performance (NR KPP Attribute 2) and support to military operations (NR KPP Attribute 1) are required.

The TSP, with the final RAFT and TCs, is reviewed by the appropriate BC and

JT4A (if the program is on the CWL) prior to the TRR briefing.

Following the system test event(s), the collected test data is entered in the Test Results column of the final RAFT. AOs analyze data to yield test information to populate the Certification Summary Data Tables of the certification issuance (certification or assessment).

Completion and issuance of the certification signals the completion of the RAFT.

The completed RAFT is archived in the JDMT and becomes the document of record for system evaluation.

4.3.4 Requirements Review Brief

The desired outcome of JITC joint IOP evaluation of a system is a JIC based on clearly defined requirements. Valid requirements must meet three criteria: (1) include a Joint Staff-approved NR KPP and approved architecture viewpoints, (2) be traceable and testable (measurable), and (3) allow testing of at least all joint tasks (mission activities), IEs, networks (transport), and interfaces in an operationally realistic environment.

The purpose of the RRB is to influence requirements early in the planning phase and give leadership the status of requirements that support a joint IOP evaluation for a certification decision. The AOs construct and present the RRB for programs assigned to them. The RRB addresses traceability among attributes, completeness of measures, and other characteristics related to the quality of the requirements. The AOs present an RRB that characterizes the quality of the set of requirements to support the IOP evaluation.

4.3.5 Joint Interoperability Evaluation Plan

The JIEP establishes the overall plan of how a system or system of systems is evaluated and serves as JITC’s version of a Test and Evaluation Master Plan (TEMP).

A JIEP is produced for all joint IOP efforts. The core component of the JIEP is the system-specific RAFT. With respect to IOP, the JIEP outlines the test strategy, timeline sequence, test events, and resources needed to complete the evaluation of the SUT’s NR KPP and joint IOP requirements. The JIEP contains the overarching approach to evaluate a system’s compliance to the NR KPP attributes and compiles those attribute evaluations to support the overall compliance to the NR KPP.

If multiple test events are necessary to complete IOP testing, the JIEP shows what is addressed in each test event. The JIEP is supported by one or more TSPs with details of the individual test event(s). The JIEP contains a section detailing the analysis plan for the data collected from the test events. It also includes a concise system description and operational use as derived from the applicable Joint Capabilities Integration and Development System (JCIDS) and/or requirements documentation.

JT4A reviews and concurs with the JIEP, containing the RAFT, for all CWL programs.

The RAFT identifies all the SUT NR KPP requirements to be addressed in the testing program. The JIEP identifies the requirements to be addressed in which test event. The JIEP provides a snapshot of the overall test program and how the system progresses to final IOP determination. As test results are recorded in the RAFT, it can be used to show the progress toward certification.

4.3.6 Test Concept Brief

The TCB answers the question, “How will the system be evaluated and tested?”

The TCB shows the IOP test planning concept for a system using a standardized presentation. It ensures that the system’s JIEP is appropriately constructed to support specific test objectives defined for each test activity. It also ensures that analysis and planning have been sufficient to yield a valid IOP determination.

Typically, the TCB is presented after the DMAP section of the intermediate RAFT is completed. The TCB is normally presented once for a SUT and covers all testing that will be leveraged for a certification effort. However, updates may be required when changes affecting IOP are made to the baseline system and/or programs experience significant requirements changes.

4.3.7 Interoperability Test Plan and Test Support Package

For a given test event(s), the AO produces either an ITP or a TSP to conduct IOP T&E. Planning focuses on collecting all data necessary to resolve a system’s joint NR KPP requirements and all joint missions/tasks, networks, and IEs. As time and resources permit, data to resolve the entirety of a system’s IOP requirements (missions/tasks, networks, and IEs) should also be sought. Refer to Figure 2 for an illustration of the test cycle framework.

4.3.7.1 Interoperability Test Plan

The ITP is written for a single test or data collection event, providing guidance and structure for sufficient data to be captured to enable the IOP determination. This type of plan details the following areas:

System functional description, background needed to understand the test, scope of the particular event, and any limitations of the event

Cybersecurity status for the test, system baseline description, and how JITC validates that the system is in its approved cyber configuration during test

Testing, data collection, and analysis procedures that apply to the event

Data collection and management plan for the event

The system RAFT for requirements and TCs

Analysis strategy for the test event

4.3.7.2 Test Support Package

The TSP is used when there are multiple test events where a single test event evaluates only a subset of the SUT’s IERs and when JITC is participating in a Service test where JITC is not the RTO. The TSP is an abbreviated ITP, yet it contains the necessary information to successfully execute the test event and gather results. The individual TCs contained within the TSP are used to collect test data on individual IERs, networks, and missions. Several test events are necessary to gather sufficient data on the SUT. The TSP explains how to execute the event, and TCs explain how to evaluate the IERs, networks, and missions.

The TSP consists of short narratives describing the system’s functional description, the background needed to understand the test, the scope of the particular event, and any limitations of the event. The narratives address the data collection and management plan for the event. They provide the verbiage needed to understand the SUT portion and the test event including limitations to the event. The narratives consist of:

SUT Functional Description: Describes the important functions, missions, and uses of the system portion under test. The JIEP contains the broader system functional description; the TSP only addresses the portion of the system being tested. The TSP system functional description defines what the users need from the system portion under test; specifically, it addresses the exchange of information with other systems.

Test Background: Explains why the test is needed and why the portion of the SUT is being evaluated.

Test Purpose: Explains the purpose of evaluating the portion of the SUT with regard to evaluation of the complete system.

Scope of the Test: Explains what this specific testing covers. It describes the relationship of the SUT portion to the full range of system operational capability and identifies the organizations involved, type of testing, and where and when testing occurs. It introduces and lists the IER TCs that are used in the test event and relates them to the SUT portion.

Limitations: Provides a short discussion of issues that constrain test conclusions and explains the impact of each limitation on the conclusion for the event and the possible impact on the complete SUT.

TCs: Provide all the information and procedures to test and record data with respect to individual IERs, networks, and missions. The TC is self-contained and answers the five W’s and how questions to evaluate the attribute in question.

4.3.8 Test Readiness Review

The TRR answers the question, “Is JITC ready to conduct the test event?” The TRR provides command-/division-/branch-level review and concurrence that AOs are ready to test or support a specific test event as determined by the ability to achieve established objectives through either an ITP or TSP. AOs conduct a TRR for every test activity to ensure readiness for test.

The initial criteria to conduct the TRR are as follows:

Joint Staff-approved NR KPP is obtained.

RAFT is approved. The final RAFT includes the IER mapping, EF, data management plan, and TCs necessary to evaluate the IERs and associated networks for the TRR’s target test event. The respective AO’s BC reviews and approves the final RAFT prior to the JIEP, TCB, and TRR phases. When a RAFT cannot be fully completed due to time constraints, lack of documentation, or other limitations, portions of the RAFT may be approved based on a level of maturity that the BC determines. The level-of-maturity approval is based on the completeness of at least the RAFT components being evaluated at the TRR. AOs are responsible for completing the remaining RAFT components.

JIEP is approved.

TCB is successfully completed.

Test documentation is approved at the appropriate level.

Data collectors are identified and available for the entire test period.

Tools and instrumentation needed for the evaluation are available and/or in place to support data collection.

Additional readiness items are as follows:

Necessary funding is available and allocated to the task.

Travel for all participants is booked.

Security clearance requirements are satisfied.

Cybersecurity status for the test has been verified.

If applicable, DISA Field Office and DoD Foreign Clearance Guide (FCG) requirements have been met. The FCG is available at https://www.fcg.pentagon.mil.

4.4 Test Results Reporting

Several documents are the output products of test activities. Test documents are written IAW the “JITC Guide to Test Documentation” and such formats and templates as shall be established by the IOP SA.

4.4.1 Quick Look Report

The QLR, when requested by a customer, is a product limited in scope to characterizing the conduct of the test event itself. It is limited to describing the scope of testing that occurred during the event and the successes/failures in satisfying data collection objectives. It may address anecdotally the overall effectiveness of the test event. A QLR does not contain analysis and evaluation of the data relative to the satisfaction of joint IOP requirements. In this case, a standard TR is produced (i.e., a QLR is not a “quick” version of the TR).

https://www.fcg.pentagon.mil/

4.4.2 Test Report

The TR is the output product of a single test event, from either a JITC-led test event or Service test with JITC participation. It includes JITC’s analysis of the data and the evaluation of the system’s performance relative to the joint IOP requirements in the RAFT based on that analysis.

4.4.3 Joint Interoperability Certification

The JIC is the final output product of the totality of JITC’s analysis and evaluation of joint IOP testing for a given version of a system. The JIC may incorporate the T&E results from one or a series of tests. It is written IAW the respective template instructions and is governed by the “JITC Guide to Test Documentation.”

4.4.4 Joint Interoperability Assessment

This document is used to assess a system’s joint IOP strengths and weaknesses, typically when a JIC would be inappropriate (e.g., there is no JS-certified NR KPP or a PMO/sponsor requests the system’s current IOP status). The process to generate an assessment is the same as that for a certification.

4.4.5 Extensions and Recertifications

JIC extensions and recertifications will be handled on a case-by-case basis using the procedures contained in the JITC IPG.

4.5 Test Preparation and Timeliness

The appropriate leadership must approve all test documentation prior to the start of the respective test event. Briefings and planning documents are due earlier when JITC is the RTO. Refer to Figure 2 for an illustration of the test cycle framework.

Completion dates for test documentation are as follows:

POA&M: At program start and updated as required

RAFT: 120 days after program start

JIEP: 120 days after program start

TSP: 30 days prior to test (60 days if JITC is the RTO)

ITP: 60 days prior to test (90 days if JITC is the RTO)

QLR: 30 days after completion of test event

TR: 60 days after receipt of last test data/adjudication of deficiencies

Certification Issuance: 60 days after receipt of last test data

The production and approval timeline for test documentation includes the staffing process. Completion of a test document is defined as being signed by the appropriate approval authority and ready for electronic report distribution or delivery to the distribution list, as applicable. The date of receipt of the last test data starts the analysis phase clock. Analysis must be conducted and document writing begun immediately. If, after analysis begins, additional data is required, contact the RTO and/or PM to obtain the necessary data or report. Inform your BC that you have additional data or documentation needs. For TRs and certification issuances, timing is critical because the respective PM uses this information for programmatic decisions.

If the IOP certification issuance requires an Initial/Follow-on Operational Test and Evaluation (I/FOT&E) report, the AOs must ensure that the PM and JITC NCR Liaison Officer are aware of that need. Generally, JITC does not release its IOP TR and certification issuance prior to the I/FOT&E report being released.

5. GENERAL GUIDANCE FOR ACTION OFFICERS

5.1 General Operations for the Interoperability Service Area

SA personnel shall adhere to the following guidance:

Develop and submit all POA&Ms and support agreements for approval following established DISA and JITC procedures.

Submit test documentation for routing and approvals using established Command procedures.

Store all final documentation regarding program execution IAW the records management plan of the associated division. Official test planning, execution, and results documentation shall be uploaded into the JITC Electronic Report Distribution (ERD) system (https://jit.fhu.disa.mil/tools/erd/) as required.

Uploading to the ERD system ensures that the automated transition process to the JITC System Tracking Program (STP) is completed.

Adhere to DoD Manual 5200.01 to protect, mark, and disseminate controlled unclassified information and classified information.

Obtain (AOs) the latest program Security Classification Guide (SCG) from the respective PMOs annually. AOs are expected to proactively use a program’s SCG to produce all documentation and correspondence and should include it as a reference in all formal SA products.

https://jit.fhu.disa.mil/tools/erd/

5.2 Interoperability Test Processes

Figure 3 shows the notional IOP project workflow.

LEGEND:

JIEP Joint Interoperability Evaluation Plan RAFT Requirements Analysis Framework for Test NLT No Later Than TCB Test Concept Brief

POA&M Plan of Action and Milestones TRR Test Readiness Review

Interoperability Project Workflow

All IOP SA-led/supported test events should have the full range of documentation necessary and approved/concurred with through the appropriate level—BC, DC, IOP SA Lead, or Command—before executing or attending the data collection test event.

The flow of test documentation generation shown in Figure 3 is cumulative; that is, each successive document relies on and reuses the preceding documentation. The RAFT becomes the core of a JIEP, a JIEP expands into an ITP or TSP, results from testing are populated back into the RAFT for evaluation, and those results are reported in a certification issuance. The intent is to maximize reuse to increase speed while reducing cost. The SA works with the JT4 Strategy, Plans, and Engineering Division to develop and refine tools to automate test documentation and reporting to the maximum extent.

The SA also establishes a shared workspace where AOs can find this SOP, other guidance documents, and SA templates for reports and briefings.

IOP certification support programs should produce, and be based on, a RAFT— essentially the combination of an Integrated Architecture Traceability Matrix (IATM), IOP EF, and DMAP. The RAFT also includes TCs for each IER listed. A RAFT traces all requirements through DoDAF products to test the approach through data management and analysis.

All SA IOP certification support programs should have an approved JIEP that includes the RAFT as its core content. The IOP SA Lead must concur with deviations from this requirement.

All SA IOP certification support programs should conduct a TCB and a TRR with the appropriate level of approval before going to test.

For the SUT, the AO is responsible for timing and resourcing of the creation, construction, and approval of test documentation and conduct of the TCB.

The appropriate level of leadership approves test documentation no later than

30 days prior to test unless circumstances prevent adherence to this timeline.

6. TEST DOCUMENTATION KNOWLEDGE MANAGEMENT

6.1 Joint Interoperability Test Command Data Management Tool

The JDMT is a modular, web-based Structured Query Language (SQL)-based data collection, storage, and management tool developed at JITC. The JDMT exists on both the Non-classified Internet Protocol Router Network (NIPRNet) and SIPRNet. The JDMT NIPRNet location is https://tools.nit.disa.mil/test/jdmt, and the SIPRNet location is https://tools.nit-c.disa.smil.mil/jdmt. The JDMT facilitates:

Self-management of project events, users, and data.

Dynamic form builder (surveys, data collection, incident reports, etc.).

Web-based form input.

Dynamic report generation.

Data Authentication Group (DAG) scoring tool.

File and artifact upload/download management.

The Joint Interoperability Evaluation System (JIES) part of the JDMT provides an automated capability to create JIC documentation. JIES supports the full range of IOP testing activities from program engagement through test planning, reporting, and close-out. JIES also provides a document repository and toolsets and generates reports applicable to all programs undergoing IOP T&E.

https://tools.nit.disa.mil/test/jdmt https://tools.nit-c.disa.smil.mil/jdmt

The JIES repository stores mandated architectural artifacts, Joint Staff J6-approved IOP requirements, and ISPs. It also stores all approved JITC products including the requirements analysis, RAFT analysis, JIEPs, TSPs, ITPs, certification issuances, and Test Incident Reports (TIRs).

The overarching goal for JIES is to ingest JITC IPG-mandated architectural artifacts for the SUT identified for IOP test and certification and to produce all test documentation. The JIES capabilities include or will include:

Generation of TCs.

Generation of RAFTs.

Future capability to generate Head Start products.

Future capability to generate a TSP.

Future capability to generate a report based upon the deficiencies of the DoDAF architecture viewpoint documentation.

Future capability to generate a statistical analysis of test data results.

Future capability to generate enclosure tables for certification issuances.

Future capability to generate certification issuances.

JT4 will develop training as new JDMT capabilities are released. JTC5 can provide individual training upon request. Each AO is responsible for requesting an account and managing the rights and permissions for projects assigned to that account.

A-1

APPENDIX A

ACRONYMS

AO Action Officer

BC Branch Chief

CDD Capability Development Document CIP Capability Implementation Plan CJCS Chairman of the Joint Chiefs of Staff CNS Capability Needs Statement CPD Capability Production Document CWL Commander’s Watch List

DAG Data Authentication Group DC Division Chief DISA Defense Information Systems Agency DMAP Data Management and Analysis Plan DoD Department of Defense DoDAF DoD Architecture Framework DoDI DoD Instruction DT Developmental Test

EF Evaluation Framework ERD Electronic Report Distribution

FCG Foreign Clearance Guide

GIG Global Information Grid GTG-F GIG Technical Guidance Federation

IATM Integrated Architecture Traceability Matrix IAW In Accordance With ICD Initial Capabilities Document IE Information Exchange IER Information Exchange Requirement I/FOT&E Initial/Follow-on Operational T&E IIPT Interoperability Integrated Product Team IOP Interoperability IPG Interoperability Process Guide ISP Information Support Plan IT Information Technology ITP Interoperability Test Plan

A-2

JCIDS Joint Capabilities Integration and Development System JDMT JITC Data Management Tool JIC Joint Interoperability Certification JIEP Joint Interoperability Evaluation Plan JIES Joint Interoperability Evaluation System JIT Joint Interoperability Test JITC Joint Interoperability Test Command JT4 Not an acronym; JITC Strategy, Plans, and Engineering Division JT4A Not an acronym; Strategy & Policy Branch, JT4 JTC Not an acronym; JITC Tactical Systems Division JTC5 Not an acronym; Test Support Branch (formerly Test Planning and

Execution Team), JTC

NCR National Capital Region NIPRNet Non-classified Internet Protocol Router Network NR KPP Net-Ready Key Performance Parameter NSS National Security Systems

OT Operational Test OV Operational Viewpoint

PM Program Manager PMO Program Management Office POA&M Plan of Action and Milestones

QLR Quick Look Report

RAFT Requirements Analysis Framework for Test RRB…