Bidders Library Interoperability Test and Evaluation - JITC IPG.pdf

Joint Interoperability Test Command

Interoperability Process Guide

Version 2.0 23 March 2015

(This page intentionally left blank.)

INTEROPERABILITY PROCESS GUIDE VER 2.0

ii

Summary of Changes

Version Sections Affected Description of Change

Version 1.0 All - Initial approved version.

Version 1.0, Change 1 All

- Administrative corrections.

- Fact-of-life changes

- Updated waiver and ICTO sections.

- Added section on Operating at Risk List processes.

- Added Sections 10 and 11 to define the minimum DoDAF architecture requirements needed for interoperability certification. Changes in text to reflect processes associated with required architecture section.

Version 2.0 All

- Administrative corrections.

- Updated references based on new DoDI

8330.01 and cancellation of DoDD 4630.05, DoDI 4630.8, and DoD CIO memorandum, “Interim Guidance for Interoperability of Information Technology (IT) and National Security Systems (NSS).”

- Added staffing guidance on requirements document review.

- Added criticality definitions for requirements review comments.

- Inserted Network Connection clarification.

- Updated Operating at Risk List section.

- Updated Recertification section.

- Updated Waiver to Policy section.

- Updated Architecture section.

iii iv

TABLE OF CONTENTS

SECTION PAGE

1. Purpose

2. Overview of Certification Policy and Process

3. Pre-Joint Interoperability Test and Certification Procedures

4. Joint Interoperability Test and Certification

5. Post-Joint Interoperability Test and Certification Procedures

6. Interim Certificate To Operate (ICTO) Procedures

7. Waivers to Policy

8. Operating At Risk List (OARL)

9. Other Evaluations and Related Information

10. Requirements for Joint Interoperability Certification (JIC)

11. Minimum Set of Architecture Information Required for Joint Interoperability Certification

Appendix A References

Appendix B Abbreviations and Acronyms

Appendix C Definitions v

TABLE OF FIGURES

FIGURE PAGE

Figure 2-1. Interoperability Directives, Instructions, and Guidance

Figure 2-2. Joint Interoperability Certification T&E Overview

Figure 2-3. Notional Joint Interoperability Certification Process

Figure 3-1. Test Preparation Activities

Figure 3-2. NR KPP Focus

Figure 3-3. Defining NR KPP Policy

Figure 4-1. Representative T&E Test Phase Activities

Figure 5-1. T&E Post-Test Activities

Figure 5-2. Interoperability T&E Products

Figure 5-3. JITC Interoperability Reports

Figure 5-4. Recertification Request Procedures Summary

Figure 6-1. Procedures for Processing ICTO Requests

Figure 7-1. Waiver to Policy Process

Figure 8-1. OARL Description

Figure 10-1. Joint Interoperability Certification Requirements Process Overview

Figure 10-2. Joint Interoperability Certification Requirements Process

Figure 11-1. Minimum Set of Viewpoints for Joint Interoperability Certification vi

1. Purpose

This Interoperability Process Guide (IPG) outlines the procedures and documentation required for Joint Interoperability Test and Certification, waiver processing, and associated processes and procedures. It addresses interoperability test and certification based on the Net-Ready Key Performance Parameter (NR KPP).

a. Section 1 provides the purpose of the IPG.

b. Section 2 outlines the governing directives and documents that underpin interoperability testing, and identifies key organizations that participate in interoperability policy making and its implementation.

c. Sections 3, 4, and 5 identify the processes, procedures, and guiding principles that cover preparation, evaluation, and reporting for Joint Interoperability Certification (JIC).

d. Sections 6, 7, and 8 outline the Department of Defense, Chief Information Office (DoD CIO) processes and procedures for Interim Certificate to Operate (ICTO), Waivers to Policy, and Operating at Risk List (OARL).

e. Section 9 provides information on other evaluations and related information.

f. Sections 10 and 11 describe requirements for JIC, including the review process and a list of minimum required architecture information.

2. Overview of Certification Policy and Process

a. Certification Policy. Several documents govern interoperability for DoD. The paragraphs below summarize key instructions and manuals. Figure 2-1 depicts the high level relationships among these documents.

(1) DoD Instruction (DoDI) 8330.01 (reference (a)) updated policy and procedures for interoperability of Information Technology (IT) and National Security Systems (NSS). This incorporates and cancels DoD Directive (DoDD) 4630.05, DoDI 4630.8, and the DoD CIO memorandum, “Interim Guidance for Interoperability of Information Technology (IT) and National Security Systems (NSS).” DoDI 8330.01 states that all IT systems, including NSS, must be evaluated and certified for interoperability prior to fielding of a new system. This includes upgrades to existing systems as well as periodic interoperability evaluations during the system’s life-cycle.

(2) Under the oversight and direction of the DoD CIO, JITC serves as the Joint Interoperability Certification Authority for all DoD IT with joint, multinational, and interagency interoperability requirements. DoDI 8330.01 further specifies that JITC shall certify all joint IT and NSS for interoperability, based on a Joint Staff certified NR KPP, when applicable.

(3) Although the Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 6212.01 series no longer applies, NR KPP policy can be found in the Joint Capabilities Integration and

Development System (JCIDS) Manual. The NR KPP Manual (reference (b)) contains detailed information regarding the foundation and development process for the NR KPP. The DoD Architecture Framework (DoDAF) viewpoints development processes provide the data used to derive the NR KPP.

(4) This IPG describes the processes for joint interoperability testing, system certification, waiver submission, and updates required for obtaining a Joint Interoperability Certification. The IPG will be updated periodically.

Figure 2-1. Interoperability Directives, Instructions, and Guidance

(5) This IPG does not address certification processes for Unified Capabilities (UC). (UC capabilities are voice, video, data (collaboration), and mobile devices.) Unified Capabilities Requirements (UCR) Document (reference (c)), DoDI 8100.04 (reference (d)), and the Approved Products List (APL) Process Guide (reference (e)) detail the interoperability certification policy and requirements for UC.

(6) Some programs do not develop the NR KPP or DoDAF architecture artifacts. In these cases, the DoD CIO and Joint Staff, with technical assistance from JITC, evaluate the proposed interoperability artifacts and determine if they are acceptable as interoperability requirements. If found acceptable, all other aspects (aside from source and type of requirements) detailed in this IPG will apply.

b. Interoperability within the Acquisition Lifecycle. JITC, as DoD’s sole joint interoperability certifier, has the ability to assist designated test organizations and Program Management Offices (PMOs)/Sponsors with defining interoperability data collection requirements for Joint Interoperability Certification on a cost reimbursable basis. The actual interoperability certification event follows a traditional test and evaluation strategy as shown in Figure 2-2. Programs can use any test organization to conduct the testing, so long as they follow the prescribed processes detailed within this IPG. Programs may engage JITC to support their testing or execute their testing in totality. Regardless of the test method, JITC must evaluate the results and make an interoperability determination.

(1) Interoperability data collection requirements can be fulfilled through the execution of other test and evaluation events. As an example, JITC can obtain data from cybersecurity (previously Information Assurance (IA)) testing, Developmental Test and Evaluation (DT&E), User Acceptance Testing (UAT), Operational Test and Evaluation (OT&E), or any combination thereof. The PMO/Sponsor of the system under test is responsible for ensuring funding for JITC efforts. Because JITC operates through a cost reimbursable model, JITC will always strive to design the most cost effective solution and work to conserve resources while achieving the greatest testing efficiencies.

Figure 2-2. Joint Interoperability Certification T&E Overview

(2) Specific to NSS, JITC will assist National Geospatial-Intelligence Agency (NGA) and National Security Agency (NSA) with interoperability objectives and help define

DoDI

• NR KPP

• Architecture

Develop Test Requirements

Develop Certification Approach

Perform Evaluation

Document Test Results

• PMO/

Sponsor

• Integrated Test Teams

JITC

Interoperability

Process Guide

Denial of Joint Interoperability

Certification

Joint Interoperability

Assessment Joint

Interoperability Certification with Conditions

Joint Interoperability

Certification DoDI

8310.01

IT

Standar ds in the

DoD

DoDI 8330.01 IOP of

IT/NSS

interoperability test and evaluation criteria, measures, and requirements established by intelligence functional managers. JITC will assist with the test planning, data collection, and reporting to ensure systems undergo and successfully complete joint interoperability test and evaluation. As stated above, the PMO/Sponsor of the system under test is responsible for ensuring funding for JITC efforts.

c. Operating at Risk List. If a system is denied certification (due to an interoperability shortfall) or has not made significant progress toward achieving Joint Interoperability Certification, the system may be placed on the OARL. The OARL is monitored by the DoD CIO’s Interoperability Steering Group (ISG). (Refer to Section 8 for OARL policy and procedures.)

d. Network Connection. IT must be certified for interoperability, or possess an ICTO or waiver to policy before connection to any DoD network (other than for test purposes). An interoperability certification with conditions may be issued under certain circumstances. These conditions may constrain use of network interfaces that do not meet all critical requirements (threshold NR KPP), while not limiting use of other interfaces and associated information exchanges. The appropriate Connection Approval Office (CAO) determines final network connection approval, with interoperability certification being merely one of the determining factors. (This clarifies the requirement in DoDI 8330.01, Enclosure 3, paragraph 2.b(3).)

e. Interoperability Process Overview. Sections 3, 4, and 5 detail the Joint Interoperability Certification preparation, test and evaluation, and certification processes. Figure 2-3 shows a notional process flow for a program of record; therefore, the steps would be applicable to most systems requiring a certification.

Figure 2-3. Notional Joint Interoperability Certification Process

Provide Test Data and Results to JITC

4a

Program Manager or Sponsor Activities

Joint Interoperability Test Command Activities

JITC Contact

– Test Doc

3e/f

Provide

NR KPP,

ICA, Cyber, etc. Info

3f/g

Need for Joint Interoperability Certification

(JIC)/NR KPP?

3a/b

Develop and CJCS Cert of NR KPP 3c

Prepare Requirements for JIC 3d

Develop Test Plans & Schedules

3h

3g(1) Evaluate JIC Test Data

4b

Develop

ICEP & ITP

3h(2)/(3)

Test Conduct

(DT,

OT,…)

4a

Identify JIC Test Data Collection Req’s 3h

Define Test Environment & Resources for Testing

3i

Generate Test Report

5a

Generate JIC Cert Product

5b Verify Test Environment

4b

Verify Standards Compliance

4b

Evaluate NR

KPP/JMT….

4b(4)

Define Test Support Needs/Cost

3e

Refers to IPG Paragraph #

3. Pre-Joint Interoperability Test and Certification Procedures

The Test and Evaluation (T&E) checklist shown in Figure 3-1 below typifies the range of test preparation activities that routinely occur during the pre-joint certification phase.

Figure 3-1. Test Preparation Activities

a. Determine Need for Joint Interoperability Certification. All systems with joint interfaces or joint information exchanges with other systems require joint interoperability certification. A joint interface occurs when any system whose mission is joined through a logical connection with a system(s) or data sources from an external partner for the purpose of exchanging common data, sharing situational awareness, or partnering to perform a single mission. An external partner is defined as another DoD Component, U.S. Government Department or Agency (including federal, state, local, and tribal), Coalition partners, non-governmental organizations, or any combination thereof that utilize the same interfaces and/or exchange information produced/consumed/shared or distributed by the system under test. Interfaces and/or information exchanges include all the data products and waveforms used or produced by the system (including sensor platforms). If Joint Interoperability Certification applicability is in doubt, the PMO/Sponsor should contact JITC for assistance with a determination or work through their respective ISG representatives for resolution. A list of Service/Agency ISG representatives can be found on JITC’s ISG Resource website: http://jitc.fhu.disa.mil/projects/isgsite/index.aspx.

b. Determine Need for NR KPP Certification. The Joint Staff is responsible for confirming whether a system has joint interfaces or joint information exchanges and requires a Joint Staff NR KPP certification. The PMO/Sponsor develops the NR KPPs (see reference (b) for policy and procedures on developing the NR KPP and supporting documentation). Generally, certification of interoperability involves evaluating three (3) attributes with the NR KPP (see

Pre-Joint Interoperability Certification Checklist

Determine Need (JIC & NR KPP) Develop NR KPP Requirements Prepare Req’s for Joint

Certification Make Initial JITC Contact Assemble Required

Documentation Include Equipment Configuration

& Cybersecurity Controls Develop Test Plans/Schedule Determine T&E Resources

Joint Staff

Program Office

JITC

All or part of each task may occur during this test phase

System Under Test http://jitc.fhu.disa.mil/projects/isgsite/index.aspx

Figures 3-2 and 3-3), and associated technical requirements defined in, or derived from, the solution architecture data.

Figure 3-2. NR KPP Focus

c. Develop NR KPP Requirements. An early step in preparation for joint interoperability certification is a Joint Staff requirements review, which includes Joint Staff certification of the NR KPP (or equivalent documentation). The NR KPP defines measures of effectiveness (MOEs) and measures of performance (MOPs) with threshold and, if applicable, objective criteria associated with three attributes: support military operations, enter and be managed on the network, and effective information exchange. Early coordination between the PMO/Sponsor and JITC is essential to ensure collection of the required data as part of the testing planned by the PMO/Sponsor.

(1) Support Military Operations. This attribute involves the MOEs and MOPs used in evaluating interoperability aspects of the system being tested within the operational context in achieving the tasks to fulfill its operational mission. Test data for this attribute will normally be obtained from the designated test organization responsible for the OT&E of the system.

(2) Enter and Be Managed on the Network. This attribute provides MOPs addressing the system’s ability to successfully enter into and be managed within the networks it must operate on to perform its operational mission (includes the means for information transport). This also addresses associated technical parameters in or derived from architecture data for the system and the associated networks. Evaluating these details involves addressing both operational and technical requirements associated with the system’s interaction with the specified networks.

This evaluation may make use of data collected from DT&E for some technical requirements and OT&E for the operational requirements. Note: Careful review of assertions that a program does not enter or is not managed in a network is critical here, particularly with sensor programs, payloads, or platforms (manned or unmanned).

NR KPP focuses on three attributes that include program specific, validated, verifiable performance measures and metrics.

(3) Effective Information Exchange. This attribute provides MOPs associated with specific interfaces and information exchanges (resource flows) supporting the operational mission. It also considers interface and exchange technical parameters included in, or derived from, architecture data. As with the previous attribute, evaluation of this attribute must address both operational and technical requirements associated with each interface or exchange. Data to evaluate satisfaction of these requirements may be obtained from DT&E for some technical requirements or OT&E.

Figure 3-3. Defining NR KPP Policy

d. Prepare Requirements for Joint Interoperability Certification. PMO/Sponsor has the responsibility for developing the requisite interoperability requirements documents and associated architecture data. The PMO/Sponsor-JITC coordination on architecture viewpoints needed for interoperability testing and certification should happen early in the architecture development process. The following policy and guidance, as well as appropriate DoD Component requirements, govern preparation of this documentation. (Also see Appendix A for additional references.)

(1) CJCSI 3170.01 (reference (f)) directs JCIDS documentation to be prepared and submitted. Format found in the JCIDS manual (reference (g)).

(2) DoDI 8330.01 (reference (a)) governs Information Support Plans (ISPs).

(3) The NR KPP manual (reference (b)) details preparation of the NR KPP, and related architecture analysis supporting its development.

• NR KPP provides program specific validated, verifiable performance measures and metrics

• NR KPP Architecture development methodology based on DoDAF architecture or optional NR KPP Architecture Data Assessment Template

• Alignment with DoD Information Enterprise Architecture

• Process details in the NR KPP Manual

NR KPP Guiding Principles

NR KPP certification required for all IT and NSS that contain joint interfaces or joint information exchanges

Three NR KPP Attributes

1. Ability to support military operations

2. Enter and be managed on the network

3. Effective information exchange

NR KPP

Manual

CJCSI

3170.01

JCIDS

Manual

Updated Policy

Guidelines

The Joint Staff details processes and procedures for NR KPP development and certification, aligned with DoDAF architectures.

- Defines the NR KPP’s three attributes

- Instructions for developing the NR KPP

- Provides staffing and certification instructions

NR KPP Fundamentals

(4) The above references also delineate the requirement for Interface Control Agreements (ICAs), and reference (b) also points to a descriptive ICA template.

(5) The DoD CIO DoDAF web site (reference (h)) contains a detailed description of the DoDAF, and its application in developing capability solution architectures.

e. Initial Contact with JITC to Schedule Support. To shorten test timelines, the PMO/Sponsor must work with their respective ISG representative to establish contact with JITC as early as possible (during initial development phases) to begin coordination for interoperability evaluation. The JITC public website (http://jitc.fhu.disa.mil/) provides forms and contact information under the “Support” section. The PMO/Sponsor is responsible for arranging funding for planning, testing, analysis, and reporting associated with interoperability certification.

Funding must be in place prior to the start of any JITC activity (typically 120 days before). In the case of Common Data Link (CDL), the CDL Executive Agent funds testing.

f. Required Documentation for Test. The PMO/Sponsor must provide JITC the following information prior to any test and evaluation activity that will support Joint Interoperability Certification (typically 120 days before):

(1) Approved requirements documents (or requirements for Joint Interoperability Certification) with certified NR KPP. Information must include:

(a) A Joint Staff-certified NR KPP. The NR KPP will describe a set of performance measures, to include MOEs and MOPs.

(b) Appropriate supporting solution architecture data, as indicated in the NR KPP Manual (reference b) or the minimum essential architecture information required for Joint Interoperability Certification, as described in Section 11 of this document.

(2) Interface Documentation

(a) ICAs for each external interface of the system to be certified, as defined in the NR KPP Manual.

(b) Interface control documents/specifications (as appropriate) for each external interface of the system to be certified (made available to JITC and other participating test organizations).

(3) For systems employing technology governed by policy mandating specific standards conformance requirements (e.g., specified by DoD Information Technology Standards Registry (DISR)), documentation of appropriate standards conformance shall be provided or cited. For example, Radio Frequency (RF) communications often require over-the-air-interoperability, which involves the ability of two or more radios to process the waveforms generated by the other device. Such policies, for example, include those governing:

(a) DoD Ultra-High Frequency (UHF) Satellite Communications (SATCOM).

http://jitc.fhu.disa.mil/

(b) Geospatial Intelligence (GEOINT) standardization for Still Imagery, Motion Imagery, and Geospatial Intelligence.

(c) Selected High Frequency (HF) and Very High Frequency (VHF) communications capabilities.

(4) Documentation of the cybersecurity configuration sufficient to ensure a realistic cybersecurity testing environment. The PMO/Sponsor must provide documentation, signed by the sponsor Authorizing Official (previously a Designated Approving Authority (DAA)), when claiming exemption from any cybersecurity requirements. When a PMO/Sponsor develops an enterprise application or service that is wholly dependent upon the enterprise infrastructure for security and access control, the requirements for security certification may be waived by the cybersecurity Authorizing Official.

(5) Version identification information for the system or system components (both services and data) to be certified, and for any interfacing capabilities and enterprise components.

(6) Approved PMO/Sponsor and designated test organization test plans and planning documents (see paragraph 3.h below).

(7) A Program Security Classification Guide.

g. Equipment Configuration/Application of Required Cybersecurity Controls.

Interoperability evaluation will be based on testing of production representative systems in as realistic an operational environment as practicable, to include the expected joint operating environment. Testing includes the use of test scenarios with a typical message mix, loading that reflects normal and wartime modes, and benign and hostile environments. System test configurations will represent realistic cybersecurity aspects of the operational environment to include application of the cybersecurity controls. If testing does not use the proper cybersecurity configuration, then the test results may be rejected, requiring additional testing. It is important in the planning stages to recognize the need for a suitable interoperability environment (for the system under test and interfacing systems), including cybersecurity considerations.

h. Develop Test Plans/Schedule. The PMO/Sponsor shall coordinate with JITC to integrate interoperability test requirements and resources into the system's T&E documents (e.g., Test and Evaluation Master Plan (TEMP), test plans). JITC may produce an interoperability assessment strategy, which may be incorporated into an Interoperability Certification Evaluation Plan (ICEP) or an Interoperability Test Plan (ITP). The plan(s) used will depend on several factors:

the complexity of the system (e.g., single item, number of external interfaces); development approach (e.g., commercial-off-the-shelf (COTS), evolutionary with numerous increments); and the anticipated number of JITC and non-JITC conducted test events. Changes in requirements, architecture, concept of operations, or the developmental/operational testing program may require changes in the overall plans. When a program is being developed in increments (phases, blocks, spirals, major releases, etc.), the plans must specify which requirements the system must meet for each increment to be certified.

(1) Test Plan Strategy

(a) Whenever possible, interoperability test data (including standards conformance) shall be obtained from the test program developed by the PMO/Sponsor and designated test organization, with input from JITC regarding data collection required to satisfy interoperability evaluation requirements. JITC shall provide these interoperability data collection requirements to the PMO/Sponsor and designated test organization as early in the lifecycle as possible (after receipt of funding) to be included in TEMPs and test plans. JITC interoperability certification is based on results from events that are as operationally realistic as feasible. This normally entails collection of data obtained from operational testing, operationally realistic exercise events, or from actual operational use.

(b) PMO/Sponsor and designated test organizations shall coordinate test plans with JITC prior to any test event supporting interoperability evaluation.

(c) When test data from the PMO’s/Sponsor’s test efforts are insufficient to perform an interoperability evaluation, JITC (when funded, and in coordination with the responsible PMO/Sponsor and designated test organization) shall develop and execute a plan for interoperability testing for collection and evaluation of the necessary data.

(d) NR KPP MOEs/MOPs (or equivalent requirements) are used to develop the TEMP. Established Joint Mission Threads (JMTs) shall be used to verify the operational effectiveness of information exchanges (reference (i)). If established JMTs are not available, appropriate mission operational tasks (activities) are derived from the Joint Staff certified NR KPP and approved architecture viewpoints (i.e., as in OV-5B and OV-6C).

(e) Standards conformance serves as a foundation for interoperability. If applicable, standards conformance should be assessed prior to joint interoperability testing. The PMO/Sponsor shall coordinate with JITC during the planning of standards conformance testing to ensure interoperability evaluation needs are adequately addressed. This will allow JITC to leverage planned testing for the system’s Joint Interoperability Certification process and minimize additional testing.

(f) Coordination and scheduling considerations should be negotiated by the PMO/Sponsor and designated test organization with proponents of interfacing systems (e.g., the certification process requires interfacing systems be available during interoperability testing).

(2) Interoperability Certification Evaluation Plan (ICEP). An optional JITC test and certification strategy, an ICEP identifies a series of test events at which data collection in support of interoperability evaluation is planned. It is normally developed in coordination with the PMO/Sponsor using the TEMP to identify suitable events for interoperability data collection.

Also, it is used to coordinate development of data collection requirements and procedures with the PMO/Sponsor and associated designated test organizations. An ICEP establishes an overall plan on how a system will be evaluated. An ICEP will usually point to individual test plans for the details on testing component systems.

(3) Interoperability Test Plan (ITP). An ITP describes a system to be tested, test objectives, and detailed test procedures for an interoperability test. JITC develops an ITP when no previous or planned testing will produce the data needed to evaluate interoperability, where programmatic or other constraints preclude inclusion of suitable data collection in planned testing. ITPs are written for individual test or data collection events. These plans detail the testing and data collection and analysis procedures that apply to that event. A variant of an ITP, generalized test plans, may be applicable to some testing programs where the only variable is the specific system under test (i.e., test configuration, procedures, etc., remain the same).

(4) Operational Test Readiness Review (OTRR) Interoperability Statement. JITC evaluates whether a system is ready for OT&E from an interoperability perspective and provides an appropriate recommendation with regard to proceeding to OT&E based on that evaluation.

The statement addresses:

(a) Status of IT interoperability and standards conformance issues.

(b) Confirmation that all required developmental testing relating to IT interoperability has been successfully completed and passed.

(c) Details of any interoperability issues that must be resolved before the start of OT&E.

i. Determining Required Resources for Test & Certification. To be cost effective, the PMO/Sponsor must integrate the evaluation of a system’s interoperability into the overall test, evaluation, and development processes as early in the developmental lifecycle as possible. The PMO/Sponsor and JITC shall jointly establish a strategy for evaluating interoperability requirements in the most efficient and cost effective manner, in an operationally realistic environment, including cybersecurity considerations. This evaluation strategy identifies the data necessary to support an interoperability evaluation, as well as indicates the test events/environments planned to produce that data.

4. Joint Interoperability Test and Certification

During testing, a variety of structured events surround successful interoperability test and certification. Figure 4-1 summarizes the range of activities that typically occur during this key phase.

Figure 4-1. Representative T&E Test Phase Activities

a. Test Conduct

(1) The PMO/Sponsor is responsible for providing the data for interoperability evaluation and follows the plans developed during pre-test activities. While test data is an essential element for analysis, it is critical to pay sufficient attention to the basic tenets of testing. The test environment must be properly configured, including cybersecurity controls, and the correct version of the software and operating system must be loaded and configuration managed.

Version identification is equally important, not merely for the system under test, but for interfacing systems. Documenting this information during testing is critical to a successful test, as often it is unavailable afterwards. Results – good or bad – are meaningless if there are uncertainties about how components were configured (both hardware and software) and what version of a system interoperated with what interfacing system versions.

(2) Integrated T&E (“Test by one, use by all.”) is encouraged to leverage test events to make the most effective and efficient use of scarce resources. However, integrated testing does not result in a single test event that answers all questions. Nor does integrated testing mean that a single organization performs all of the evaluation and reporting. What integrated testing does

Joint Interoperability Certification Test Phase Activities

Establish Realistic Test Environment Employ Authorized Cyber

Configurations DoD Component Approves Required

Architecture for Test & Cert Document Test Configurations Configuration Manage all Test

Components/Material Record Version ID for System and

Interfacing Systems – Load Correct Software!

Conduct Test Events Collect Test Data & Share in

Common Database

Program Office

DOT&E

Service

OT&E

JITC

“Test by One – Use by All”

System Under Test do is allow independent evaluators to share the data from a test, with each performing the appropriate analysis to address their specific test issues and measures. For example, data to support interoperability evaluation should include results from OT testing, with JITC using the shared test data to provide Joint Interoperability Certification. Maximum benefit can be obtained from integrated T&E if developers and test teams collaborate on test planning and execution, and establish common databases to share data.

b. Initial Joint Interoperability Test and Certification Process

(1) Joint interoperability certification is based on test and evaluation of production representative systems (hardware/software) in as realistic an operational joint environment as practicable, including use of authorized cybersecurity configurations.

(2) Joint interoperability certifications provide input to the Milestone Decision Authority (MDA) (post Milestone C), or cognizant fielding authority, for a fielding decision. The Joint Interoperability Certification does not satisfy any other certifications that may be required (e.g., spectrum certifications, network manager approval to connect, and/or other validations/approvals).

(3) A Joint Interoperability Assessment provides preliminary interoperability status.

This can be particularly useful in cases where requirements documents have not been finalized, high risk areas warranting early feedback, etc.

(4) JITC shall evaluate interoperability test results using a variety of resources including:

(a) Joint Staff-certified NR KPP and information prescribed in Section 11 as required for Joint Interoperability Certification. Issues with NR KPP requirements shall be raised with the Joint Staff for resolution.

(b) Mission-related information.

(c) Data from DT&E, OT&E, acceptance testing, exercise venues, or other demonstrations, consistent with any approved TEMP, or other interoperability data collection requirements. The potential operational impacts of all unresolved interoperability deficiencies noted during evaluation must be determined by the appropriate users or user representatives and be reported by JITC in any resulting certification.

(d) Interoperability test and evaluation criteria, measures, and requirements established by intelligence functional managers (e.g., NGA and NSA).

(5) Pre-test activities by JITC include verifying that system and network configurations used in testing are representative of a realistic operational environment, to include cybersecurity (formerly IA) characteristics.

(6) JITC has the capability to evaluate cybersecurity (or portions of cybersecurity requirements) when requested, and shall document any known cybersecurity status as part of reporting the interoperability status. However, some portions of cybersecurity requirements may not be (or able to be) assessed until after JITC interoperability certification, and as such cannot be reported in the certification. Significant cybersecurity issues shall be reported in the Joint Interoperability Certification for systems with an NR KPP, and any deficiency which has a potential critical operational impact, may result in JITC being unable to certify the system.

(7) JITC shall also determine whether any necessary standards conformance certifications have been obtained. This is usually accomplished in DT&E venues because of the nature of standards conformance testing. JITC shall consider test results from previous standards conformance testing conducted during system development.

(8) JITC evaluation of interoperability is not merely an assessment of functional performance, but of the effectiveness of information exchange within the operational environment. Interoperability of a system depends on many factors that the PMO/Sponsor may influence, but not directly control. Interfacing systems, operational network access and loading, atmospheric conditions, satellite transponder or channel availability, ambient electromagnetic conditions, etc., are among the factors that may impact interoperability, and the resulting certification, independent of the performance of the system under test. For this reason, unlike most other forms of testing, deficiencies in interoperability may occur that are not attributable to the system being tested, but may influence the interoperability evaluation and subsequent certification. If a system has no requirement to operate under some set of conditions, failing to do so may be noted but shall not be considered as an interoperability failure. For example, atmospheric dust in excess of specified requirements prevents closing of a link.

5. Post-Joint Interoperability Test and Certification Procedures

This section describes the principal post-test actions required by stakeholders to accomplish interoperability certification. The processes summarized in Figure 5-1 below typify the range of post-test activities that routinely occur during this final phase of the process.

Figure 5-1. T&E Post-Test Activities

a. Reporting. JITC shall provide interoperability test documentation to the PMO/Sponsor, with Joint Interoperability Certifications being sent to ISG members, with a delivery goal of 60 calendar days from the end of testing and receipt of all required test information. The PMO/Sponsor and designated test organization should provide all relevant reports, system/test configuration information (including for interfacing systems), test data, trouble reports, analysis of any discrepancies, etc., in a timely fashion. All parties should keep in mind the JITC processing time required after receipt of test information – the sooner organizations provide the required information, the sooner they can receive their certification.

b. Certification Products. A family of reporting products has been introduced to document test and certification activities. Figure 5-2 and the paragraphs below describe the interoperability T&E products and possible outcomes resulting from successful or failed testing. In addition, see Figure 5-3 for a summary of current reports.

Post-Joint Interoperability Certification Test Process

Incorporate Test Results Leverage DT&E and OT&E Conduct Data Analysis Certify System Distribute to Stakeholders Recertify to Accommodate

Changes

JITC

Program Office

Test Organization

System Under Test

Figure 5-2. Interoperability T&E Products

(1) Joint Interoperability Certification. JITC issues a Joint Interoperability Certification when a system has been evaluated against its joint interoperability requirements and the system's interoperability status is sufficient to support a fielding decision.

(a) Joint Interoperability Certification with Conditions. When appropriate, JITC may issue certifications with conditions (limitations) when only subsets of the requirements are met.

Conditional certifications provide the system/interface interoperability status for cases where useful capabilities are provided, despite not meeting all threshold requirements, and there are no expected critical operational impacts or adverse effects on the joint interoperability environment.

Conditions to certification are based on assessment of operational impact and limit the operational use of the system to only those functions and interfaces that were adequately demonstrated. A PMO/Sponsor must submit the system for additional interoperability testing in order to have these conditions (restrictions) removed.

(b) Certification for Systems Developed in Increments. JITC may issue a Joint Interoperability Certification for each increment of a system. All joint interoperability requirements for a given increment shall be used for evaluation and reporting the status, not just those requirements implemented. If requirements for the system were not delineated by increment (phase, spiral, block, etc.) in the Joint Staff certified NR KPP, all requirements will

Joint Interoperability

Certification

TESTING

• Retest

• Denial of Certification

Interoperability Products

Certification is issued when a system’s interoperability is sufficient to support a fielding decision

Joint Interoperability

Assessment

Assessment of interoperability for a prototype system or early production version

Joint Interoperability

Certification Requirements

OR

apply to the current increment. Changing the increment or criticality of a requirement is a modification to the requirements that may require Joint Staff recertification.

(2) Denial of Joint Interoperability Certification. When interoperability deficiencies are identified that critically impact joint interoperability or joint mission accomplishment, JITC may issue a denial of certification memorandum. This provides CIOs, Joint Staff, MDAs, and PMOs notification of problems that warrant immediate attention.

(3) Joint Interoperability Assessment. A joint interoperability assessment can be issued to assess a system’s interoperability strengths and weaknesses. Assessments are typically provided when a certification is not appropriate (i.e., when there is no certified NR KPP, but the PMO requests an early assessment). Interoperability assessments can be conducted during DT&E or OT&E events, acceptance testing, interoperability exercises, or other test venues. The PMO/Sponsor shall coordinate with and fund JITC to establish the exact assessment needs and identify documentation requirements.

Figure 5-3. JITC Interoperability Reports

(4) Revocation and Reissuance of Joint Interoperability Certifications. Joint interoperability certifications may be rescinded, revoked, or reissued by JITC. This would occur if an issue has been detected due to a change between a fielded configuration and a tested configuration. This would include a change in data exchange partners as well as a change in system configuration, or any interoperability deficiency discovered post test. All organizations that received the original certification notice shall be notified of changes in interoperability certification status.

All Joint Certifications are “Full” or Conditional:

Cert with Conditions

Joint Interoperability

Certification

Denial of

Joint Interoperability

Certification

Conditions will be detailed in

Certification text

Other JITC Reports:

“Full” Certification

Systems With Critical Operational Impacts

May Be Denied Certification:

(5) Certification Extension Process. JITC grants a certification extension to extend coverage of an existing certification to include modifications not affecting interoperability made before the certification has expired. Since certifications are given for specific version numbers, a certification extension would be granted to expand the certification to cover a follow-on version that does not change the original certification – just a new version. The “extended” certification has the same expiration date as the existing certification. The PMO/Sponsor will follow the guidance for "Certification is Scheduled to Expire" in cases where only the period of certification needs to be renewed and no additional testing is required. PMOs/Sponsors will contact their JITC AO to establish required information and cost. At a minimum, certification extension requests must include:

(a) A written statement by the PMO/Sponsor (submitted with the request package) that the modification does not affect interoperability.

(b) Sufficient information for JITC to independently determine the impact of change.

c. Recertification Process. Interoperability can degrade over time. Changes to standards, interfacing systems, and cumulative minor upgrades impact the ability of systems to interoperate and must be carefully monitored throughout the system’s lifecycle. Joint interoperability certifications for a specific increment must be renewed periodically or when system, operating environment, or requirements changes occur that affect joint interoperability. The PMO/Sponsor is responsible for notifying JITC regarding incremental upgrades and other changes affecting interoperability. Coordination with JITC will identify funding requirements for test and certification. The PMO/Sponsor should be aware that the NR KPP may need to be recertified.

(1) Recertification Criteria. Recertification is required when:

(a) The Joint Interoperability Certification is revoked (e.g., critical operational deficiencies are reported after fielding in a given environment).

(b) The system certification expires at the end of 4 years.

(c) Changes to the interoperability environment, including the system, interfacing systems, system requirements have been made or are anticipated to occur (e.g., new increment to be fielded, or other materiel changes) impact interoperability.

(d) A new increment is released, materiel changes (e.g., hardware or software modifications, including firmware) occur to the system that affect interoperability, or materiel changes occur to interfacing systems that affect interoperability. Also, substantive revisions in mandated DISR standards may constitute a change in the interoperability environment that results in a need for recertification.

(e) Non-materiel changes (i.e., Doctrine, Organization, Training, Leadership and education, Personnel, and Facilities – Policy (DOTLPF-P)) occur that affect joint interoperability.

(2) Recertification Procedures. The PMO/Sponsor shall perform the following activities depending upon the specific situation.

(a) If certification is revoked:

1. Make changes to the system, the requirements, or both, to correct discrepancies or operational interoperability issues that were responsible for the revocation.

2. Obtain new certification by following the processes outlined in this guide for attaining an initial certification.

(b) If certification is scheduled to expire and PMO/Sponsor desires recertification without additional testing(procedures summarized in Figure 5-4):

1. The PMO/Sponsor should contact JITC through the applicable DoD Component ISG representative early enough to allow sufficient time for the recertification process to be accomplished. It is recommended this process be started as early as 12 months but no later than 6 months prior to certification expiration.

2. The ISG representative will provide the PMO/Sponsor with the recertification request form, which identifies required technical and funding information. The PMO/Sponsor will work with the respective JITC Action Officer (AO) to complete the recertification request form, which can be found on the ISG Resource website, along with a listing of the ISG representatives, at: http://jitc.fhu.disa.mil/projects/isgsite/index.aspx.

3. The PMO’s/Sponsor’s recertification request will provide written verification that the interoperability environment (including the system and interfacing systems) and joint interoperability requirements (in certified NR KPP and approved architectures) have been reviewed and have not changed such that they affect interoperability. Operation of the system has been verified through exercises, operational use, and deployments (i.e., all moderate or greater operational impacts identified as conditions in the existing Joint Interoperability Certification). If changes have occurred, the written verification will outline the deltas from the prior certified version.

4. The PMO/Sponsor will send the recertification request to the applicable DoD Component ISG representative for review and concurrence.

5. ISG representatives shall ensure requests are complete and valid. If the request is not complete and valid, the ISG representative shall return it to the PMO/Sponsor.

6. Once completed and validated by the ISG representative, the ISG representative shall send the request via e-mail to JITC (disa.huachuca.jt.mbx.jitc-iop-re-certification-requests@mail.mil), Joint Staff J-6, U.S. Strategic Command (USSTRATCOM), and other Components for recommendation.

mailto:disa.huachuca.jt.mbx.jitc-iop-re-certification-requests@mail.mil mailto:disa.huachuca.jt.mbx.jitc-iop-re-certification-requests@mail.mil

Figure 5-4. Recertification Request Procedures Summary

7. Joint Staff J-6, USSTRATCOM, and other Components will provide recommendation to requesting ISG representative, JITC, Joint Staff J-6, USSTRATCOM, and other Components within 30 days of the receipt of all required information. Joint Staff J-6, will verify that requirements (certified NR KPP and approved architectures) are current and changes (from the last certification), if any, do not impact joint interoperability. USSTRATCOM, in coordination with other Components, will review the system for any new operational impacts in the field.

8. JITC shall review all requests and provide an interoperability determination to the PMO/Sponsor and ISG representative within 30 days of receiving recommendations from Joint Staff J-6, USSTRATCOM, and other Components.

9. JITC may issue a new certification (the goal being to do so within 30 days of making the determination) without additional interoperability testing if the joint interoperability requirements, system configuration, and operational environment of the system are current and

Adjudicate / Resolve If Non-Concur

ISG

DoD Components

USSTRATCOM

ISG Reps

Issue New Certification

JITC

Request Review

Review Requirements

Changes Joint Staff J-6

Interoperability Determination

Note:

JIC = Joint Interoperability Certification POA&M = Plan of Action and Milestones MIPR = Military Interdepartmental Purchase Request

Review Operational

Impacts

ISG Rep E-mailed by ISG Rep

POA&M

MIPR

Technical Detail

Recert Request Form

If JITC and PMO disagree as to recertification way ahead, PMO may rebut.

If parties still disagree, ISG decides resolution.

PMO /

Sponsor

Recertification Request

Form

Recommendations

Once completed, ISG Rep e-mails request for Joint Staff (J-6), USSTRATCOM, and Component Reps review and recommendations.

Substantive/Non-concur comments

PMO/Sponsor coordinates with ISG Rep to obtain recertification request form, and both parties work with JITC to identify POC and other info needed to complete form.

ISG Rep reviews request to ensure it is complete and valid. If not, it is returned to PMO/Sponsor.

have not changed in a manner that impacts joint interoperability, and no new operational impacts have been identified.

10. Alternatively, if JITC determines that changes to the system or its environment have impacted interoperability, or if the interoperability requirements have changed, JITC will determine whether a desktop assessment will suffice to issue a new certification or if a new…