Bidders Library Interoperability Test and Evaluation - JTC SOP.pdf

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

TACTICAL SYSTEMS DIVISION

(JTC)

STANDARD OPERATING

PROCEDURES (SOP)

VERSION 2.0

JANUARY 2020

TACTICAL SYSTEMS DIVISION

(JTC)

STANDARD OPERATING

PROCEDURES (SOP)

The JTC SOP is developed and published by JTC in coordination with the Joint Interoperability Test Command Technical Advisor and the Strategy, Plans and Engineering Division (JT4). It is effective for JTC personnel immediately upon publication.

JANUARY 2020

Submitted by: Major Joseph Gertis Operations Officer

Approved by: ____________________________________

TOD WISTER

Chief Tactical Systems Division

Prepared Under the Direction of:

Tactical Systems Division Joint Interoperability Test Command

Fort Huachuca, Arizona

This page intentionally left blank.

SUMMARY OF CHANGES

Version Sections Affected Description of Change

2.0 All Substantial rewrite of the SOP.

i

TABLE OF CONTENTS

Page

1 INTRODUCTION

2 ORGANIZATIONAL STRUCTURE

3 ROLES AND RESPONSIBILITIES

4 GENERAL GUIDANCE FOR TEST OFFICERS

5 TEST DOCUMENTATION KNOWLEDGE MANAGEMENT

6 PROJECT RESOURCING

7 JOINT INTEROPERABILITY TEST CYCLE

APPENDICES

ACRONYMS ................................................................................................................ A-1 DASHBOARD UPDATES ............................................................................................. B-1 SIGNATURE DELEGATION ....................................................................................... C-1 JTC EMPLOYEE TRAINING AND DEVELOPMENT PLAN ........................................ D-1 END-OF-DAY SECURITY CHECKS ............................................................................ E-1 TEST AND EVALUATION MASTER PLAN REVIEW................................................... F-1 REFERENCES AND RESOURCES ............................................................................ G-1

LIST OF FIGURES

1 JTC Organizational Overview 2 Simplified Interoperability Project Workflow 3 Routing Business Document for Branch Chief Approval 4 Routing Business Document for Division Chief Approval 5 Routing Test Document for JTC5 Review and Division Chief Approval 6 Simplified Interoperability Project Workflow—Project Resourcing 7 Initial/Annual TESB Cycle 8 DISA/JITC Support Agreement Process 9 Test Cycle Framework 10 Simplified Interoperability Project Workflow—Test Documentation and Planning

Activities 11 Simplified Interoperability Project Workflow—Test Results Reporting B-1 OpsO Form Excerpt ............................................................................................... B-3

LIST OF TABLES

1 IE Results—Bad Example 2 IE Results—Good Example ii

1. INTRODUCTION

1.1 PURPOSE

The purpose of the Tactical Systems Division (JTC) Standard Operating Procedures (SOP) is to define standard processes and procedures in conducting JTC’s testing and related activities in support of its Joint Interoperability Certification (JIC) and coalition interoperability missions.

This SOP is not all-inclusive. Common sense prevails. As policies and regulations change over time, this SOP will be revised to maintain its relevance. Report any errors or circumstances that prevent applying SOP guidance to a program or system to your Branch Chief (BC) for resolution. Address administrative or documentation issues within this SOP to the Operations Officer (OpsO).

1.2 SCOPE

This SOP augments authoritative Department of Defense (DoD), Defense Information Systems Agency (DISA), and Joint Interoperability Test Command (JITC) directives, instructions, and/or guides. Its primary focus is processes and procedures internal to JTC. In case of contradictions or conflicts between the content of this SOP and an authoritative source, the authoritative source governs.

1.3 APPLICABILITY

This SOP applies to all JTC government civilian and military employees and support contractors. As a compliance requirement, the SOP should be referenced in all future task orders for all contracted support.

1.4 IMPLEMENTATION

All new and existing interoperability programs shall follow the general provisions and framework described within the SOP. Test Officers (TOs) (also known as action officers) and all support efforts shall adhere to all SOP provisions.

2. ORGANIZATIONAL STRUCTURE

The JTC’s internal division structure consists of a Division Chief (DC) (General Schedule (GS)-15), support staff, and five branches (see Figure 1). A BC (GS-14) leads each branch, which consists of six or more government civilian and/or military personnel (GS-12/13/14s; Captain/Major/Lieutenant Colonel).

LEGEND:

C2 Command and Control ISR Intelligence, Surveillance, and Reconnaissance

Figure 1. JTC Organizational Overview

3. ROLES AND RESPONSIBILITIES

3.1 TACTICAL SYSTEMS DIVISION

The JTC mission is twofold: to support JIC and to support coalition Test and Evaluation (T&E).

3.1.1 Joint Interoperability Certification

Division personnel:

• Conduct consistent, sound, and repeatable testing.

• Support the Warfighter by determining the extent to which tactical systems meet technical standards and interoperate via net-centric, operational warfare capabilities by conducting:

o Interoperability testing o Standards conformance testing o Requirements review o Pre-test planning o Data collection and analysis o Post-test reporting

• Use hardware-in-the-loop and operationally realistic environments to validate system implementation of the Net Ready Key Performance Parameter (NR

KPP).

• Certify system interoperability in a joint environment.

• Assess the capability to conduct operational missions in a coalition environment.

3.1.2 Coalition Test and Evaluation

According to the Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 5128.01, the Mission Partner Environment (MPE) Coalition Interoperability, Assurance and Validation (CIAV) Working Group provides an end-to-end Mission-Based Interoperability (MBI) assessment methodology to improve United States (U.S.) MPE operations with mission partners. DISA is a CIAV Working Group member.

MPE provides connections within a mission network construct and Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance (C4ISR) capabilities to enable collaboration and Information Exchange (IE) of the partner nations.

CIAV uses coalition mission threads as a basis to perform MBI assessments.

CIAV has a globally distributed assessment environment that mimics the operational MPE for use during assessments, providing an operationally realistic environment for validating mission requirements.

Each nation’s laboratory consists of its national baseline of capabilities currently fielded or expected to be fielded in support of coalition operations that accurately reflect the MPE and North Atlantic Treaty Organization (NATO) Federated Mission Networking (FMN) paradigm. Where feasible, JTC will use the MBI methodology and the Coalition Verification and Validation Environment to support a customer’s coalition requirements.

3.1.3 JTC Position Duty Descriptions

3.1.3.1 Division Chief

The DC is a senior government civilian (GS-15) responsible for the execution of the division’s missions and personnel. As the leader of JTC, the DC provides overall guidance on the conduct of all JTC missions. The DC serves as:

• A first-line supervisor for direct support personnel (i.e., Management Assistant

(MA) and OpsO) and the five BCs.

• A second-line supervisor for all other branch personnel.

3.1.3.2 Management Assistant

The MA reports to the DC, with secondary direction from the BCs and OpsO.

The MA:

• Provides administrative support to division personnel.

• Tracks and maintains accountability on division taskers and suspense dates, consolidating branch inputs for division-wide responses.

• Ensures proper staffing of documents and products:

o Proofreads documents o Tracks document status

• Serves as the division Defense Agencies Initiative (DAI) timekeeper and monitors DISA Personnel Accountability Management (PAM) daily.

• Maintains the DC’s and JTC’s organizational (Microsoft Outlook) calendars.

• Prepares weekly operations meeting slides and attends meeting to perform secretarial duties.

• Maintains the division contact roster.

• Captures financial information from the Department of Defense (DD) Form

1144 Support Agreement and corresponding funding received to maintain the division’s financial status.

• Provides graphics support (Microsoft Visio) to division personnel in the production of various products.

• Works with OpsO and TOs to ensure activities/products due In Accordance With (IAW) this SOP and the division’s dashboard (e.g., Plans of Action & Milestones (POA&Ms), DD Form 1144 Support Agreements, RAFTs, JIEPs, TSPs/Interoperability Test Plans (ITPs), Quick Look Reports (QLRs), test reports, and certification issuances) are scheduled/conducted/submitted as appropriate. See Appendix B for dashboard details.

• Serves as the:

o JTC primary hand receipt holder and supply point of contact o Purchase order coordinator o Microsoft Joint Enterprise License Agreement coordinator

• Serves as division security monitor:

o Holds the classified document primary account for the Automated

Classified Document Register.

o Conducts annual 100-percent inventory of classified materials.

o Submits and tracks visit requests, combatant command visit requests, and theater and country clearances.

o Maintains end-of-day roster and ensures that security checks are completed.

3.1.3.3 Military Deputy

The Military Deputy (MilDep) is a senior military member (O-5) reporting to the DC. The MilDep:

• Assists the DC in coordinating responses to higher level tasking.

• Represents the organization at designated external meetings, forums, and/or conferences.

• Supports the spend plan development cycle and budget execution reviews.

• Champions equities for all uniformed Service members within the division.

• When the DC is absent:

o Represents the division at senior staff and corporate board meetings (with support from BCs, contingent on topics) on Wednesdays.

o Conducts weekly operations meetings on Tuesdays.

o Conducts scheduled program TCBs and TRR briefings.

3.1.3.4 Operations Officer

The OpsO is a military member reporting to the DC. The OpsO:

• Works in parallel with BCs.

• Maintains the status of all programs/systems undergoing T&E within JTC.

• Maintains the Master Test Schedule for JTC test activity, depicting key dates for Systems Under Test (SUTs). The Master Test Schedule incorporates due dates for each system on a 60/30/7-day cycle and includes the TCB and TRR presentation dates.

• Serves as the lead authority for the division’s dashboard development efforts.

• Interacts with the JTC TOs to populate the dashboard.

• Updates and maintains the JTC SOP.

• Oversees the implementation of knowledge management practices within the division.

• Manages the DoD Enterprise Portal Service (DEPS) within the division.

3.1.3.5 Assistant Operations Officer

The Assistant Operations Officer (AOpsO) is a military member reporting to the OpsO. The AOpsO:

• Assists OpsO to maintain the accuracy and integrity of information on the division’s dashboard.

• Interacts with the JTC TOs to populate the division’s dashboard.

3.1.3.6 Branch Chiefs

The BCs report to the DC and are senior civilians (GS-14) responsible for their branch’s missions and branch TOs. The BCs:

• Serve as first-line supervisors for the direct support personnel (e.g., TOs).

• Ensure that the TO properly prepares TRR briefings and TCBs for presentation to the DC, which includes confirming the maturity level of the

RAFT.

3.1.3.7 Test Officers

The TOs report to the BCs and perform the following duties:

• Plan and execute interoperability T&E of the systems assigned to their respective branch.

• Manage and execute the day-to-day efforts to achieve interoperability certification for the programs in their respective branch.

• Prepare the POA&M and DD Form 1144 Support Agreement for each program.

• Maintain an up-to-date cost/schedule/performance quad chart for each program in their respective branch.

• Serve as the primary or alternate Contracting Officer Representative (COR) if

TOs use contractor personnel for program support in their respective branch.

All aspects of being a COR are detailed at the JITC COR Connection https://disa.deps.mil/ORG/JITC/JITC_COR_Connection/default.aspx?InitialTa bId=Ribbon%2ERead&VisibilityContext=WSSTabPersistence.

3.1.3.8 JTC Technical Editor

The Technical Editor reports to the JTC5 BC and performs the following duties:

• Reviews all JTC products, editing for mechanics (grammar, spelling, and punctuation) and style and for compliance with JITC templates and the JITC Guide to Test Documentation.

• Coordinates as required with the Strategy, Plans and Engineering Division (JT4), Strategy & Policy Branch (JT4A) to ascertain the sufficiency of JTC https://disa.deps.mil/ORG/JITC/JITC_COR_Connection/default.aspx?InitialTabId=Ribbon%2ERead&VisibilityContext=WSSTabPersistence https://disa.deps.mil/ORG/JITC/JITC_COR_Connection/default.aspx?InitialTabId=Ribbon%2ERead&VisibilityContext=WSSTabPersistence documentation with respect to completeness, data sufficiency, rigor, and veracity.

• Conducts periodic brown bag sessions with the TOs to discuss topics related to producing quality test documentation.

3.1.3.9 JTC Chief Architect

The Chief Architect reports to the JTC5 BC and performs the following duties:

• Develops and coordinates Tactics, Techniques, and Procedures (TTPs) for implementing the Requirements Analysis Framework for Test (RAFT) process within the division.

• Performs requirements analysis of architectural artifacts in support of JITC interoperability TOs.

• Develops, maintains, and provides training for new personnel in the use cases for solutions architecture and the inclusion of the architectural requirements into the test planning process.

• Represents JTC to the JIES (pronounced “jees”) Integrated Product Team (IPT) for the development and implementation of automation tools that support the RAFT process.

• Presents the architectural analytical process to JITC customers.

3.1.4 JTC Branches

3.1.4.1 Command & Control/Intelligence, Surveillance, and Reconnaissance Branch (JTC1)

The JTC1 mission is to conduct standards conformance and interoperability testing and certification of C2 and tactical ISR systems.

3.1.4.2 Coalition & Combat Service Support Branch (JTC2)

The JTC2 mission is to conduct standards conformance and interoperability testing and certification of combat support systems and to assess mission-based U.S.

and coalition systems’ ability to interoperate in support of the coalition Warfighter.

3.1.4.3 Combat Systems Branch (JTC3)

The JTC3 mission is to conduct standards conformance and interoperability testing and certification of force application/force protection systems.

3.1.4.4 Data Links Branch (JTC4)

The JTC4 mission is to conduct testing to validate systems’ implementation of approved messaging formats including Tactical Data Link (TDL), still and motion imagery, and Common Data Link (CDL) formats.

3.1.4.5 Test Support Branch (JTC5)

The JTC5 mission is to:

• Provide test documentation for programs/systems being evaluated by JTC.

• Provide data collectors for test efforts.

• Provide approval authority for the RAFT.

• Provide technical editing of all JTC test documentation.

• Function as the Division COR (DCOR) for tasks within the division. As a part of the DCOR role, will also function as the primary or alternate COR for division tasks with contractor support.

3.2 RELATIONSHIPS BETWEEN ELEMENTS

The interoperability testing branches (JTC1, JTC2, and JTC3) conduct T&E and report on the systems contained within each respective branch. The DC reviews, approves, and signs all documentation distributed to external audiences. The DC has delegated signature authority to the BCs for expenditures less than $150,000 IAW a Memorandum for Record (see Appendix C).

The JTC4 personnel have subject matter expertise in TDLs, Integrated Broadcast

Service (IBS), motion imagery, and transmission of imagery. The JTC testing branches (JTC1 through JTC4) employ JTC4 when system attributes and requirements contain these types of messages and IEs. TOs must employ the JTC4 Subject Matter Experts (SMEs) as support reviewers when conducting Joint Capabilities Integration and Development System (JCIDS) and requirements documents reviews that use any TDLs as defined by CJCSI 6610.01E. These TDLs are Link-4A, Link-11/11B, Link-16, Link-16 terminal (Multifunction Information Distribution System (MIDS)), Variable Message Format (VMF), IBS Common Message Format (CMF), Joint Range Extension Application Protocol (JREAP), Link-22, TDL Data Forwarding, Multifunction Advanced Data Link (MADL), and Cursor on Target (CoT).

The JTC testing branches will employ JTC5 for production of test documentation.

JTC5, in conjunction with the TO, will determine whether contractor support will be used. JTC5 support is the default position of JTC T&E for a given system.

JITC has a 5-year indefinite delivery/indefinite quantity support contract with

Jacobs Technology to provide a range of Test, Evaluation, and Certification (TE&C) support services across the command. Defense Information Technology Contracting Organization (DITCO) administers the JITC TE&C contract and appoints CORs, which are government employees.

JTC TOs can obtain TE&C support services in two ways. With the concurrence of the BC, a TO may create a stand-alone task order under the JITC TE&C contract covering either the entire scope or a portion of the joint interoperability T&E of their assigned systems. Alternatively, TOs may obtain support for specific products and services through an existing, overarching JTC interoperability TE&C task order administered by JTC5. Using either method requires the appointment of a COR to oversee the work of the Jacobs contractors at the task order level.

3.3 INTRA-/INTER-DIVISION SUPPORT AGREEMENTS

TOs are responsible for the comprehensive evaluation of interoperability for their assigned programs. JTC should strive for rigorous, consistent technical evaluation and analysis to determine the effectiveness of a system’s Information Exchange Requirements (IERs).

JTC4 provides subject matter technical support for programs that implement

TDLs (as described in CJCSI 6610.01E), IBS, CDLs, National Imagery Transmission Format (NITF) standards, full motion video standards, and Motion Imagery Standard

(MIS).

The JITC Networks/Communications & Department of Defense Information

Network (DoDIN) Capabilities Division (JTE), Communications Interoperability Branch (JTE3), provides subject matter technical support for programs that employ Satellite Communications (SATCOM) and radio frequency communications. The collective term “satellite communications” includes Ultra High Frequency (UHF), super high frequency, extremely high frequency, and military and commercial SATCOM. The collective term “radio frequency” includes high frequency, very high frequency, and UHF voice and data communications.

TOs will meet with the JTC4 Tactical Data Links Team Leader (TDL TL) to coordinate SME support. The process begins with scoping of the Statement of Work (SOW) that details the services that the TO needs to complete the interoperability determination. The SOW within the Intra-/Inter-Division Support Agreement with JTC4 and/or JTE3 determines the appropriate technical support required, due date for documentation products, and associated costs.

To facilitate and document this process, TOs will use the Intra-/Inter-Division Support Agreement template in the SOP folder (T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Ops\SOP\Support Agreement). The support agreement requires the signatures of the supported and supporting BCs. JCIDS document reviews do not require a SOW or Intra-Division Support Agreement, but they should also be coordinated through the TDL TL, as required.

3.4 TRAINING

Training for JTC personnel follows the guidance provided in Appendix D, JTC Employee Training Plan. The plan addresses the necessary academic and DISA mandatory training for establishing and maintaining a continuous learning environment.

For new employees, the respective sponsor assists in obtaining the required reading materials, creating a Defense Acquisition University (DAU) account, and registering in DISA Corporate Management Information System (CMIS) to complete mandatory training. When an action is completed, the new employee reports the status to the responsible BC, and both the new employee and BC initial the plan accordingly.

4. GENERAL GUIDANCE FOR TEST OFFICERS

4.1 GENERAL OPERATIONS FOR JTC

JTC personnel should adhere to the following guidance:

• Log on to your computer to activate the automated PAM updater for accountability. You should receive an Automated PAM Update Success email as confirmation. If not, input your personnel accountability status (typically, this is Present for Duty) into the PAM system by 0900 weekdays.

• When on leave of any kind or Temporary Duty (TDY), use the Projection feature of PAM to account for your daily attendance status. Create and submit absence requests via Defense Agencies Initiative (DAI) for approval.

Annotate periods of nonavailability on the respective branch calendar. Create an Out of Office automatic reply in Microsoft Outlook if you will be gone 1 day or longer (annual leave or TDY).

• Submit your DAI biweekly timecard by 1200 on the Wednesday following a

Friday payday.

• Submit senior staff meeting inputs to the BCs by close of business every

Monday in preparation for the meeting on Wednesday mornings. The DC assembles the division slide from the branch inputs. Division inputs are due to the JITC Operations Division (JT2) by noon every Tuesday.

• Submit all POA&Ms and DD Form 1144 Support Agreements to the DC for signature (BCs may sign for expenditures less than $150,000 annually).

Refer to Section VI for details.

• Submit test documentation for routing to the MA using an eForm 9 (refer to

Section V.4 for details).

• Store all final documentation regarding program execution in the JTC Data

Library (JDL) at T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Data Library. TOs are responsible for placing signed documents in their respective branch’s program folder. The library and storage schema are detailed in Section V.1.

• Adhere to DoD Manual 5200.01 to protect, mark, and disseminate controlled unclassified information and classified information. TOs will obtain the latest program Security Classification Guide (SCG) from their respective Program Management Offices (PMOs) annually. TOs are expected to proactively use a program’s SCG to produce all documentation and correspondence and should include it as a reference in all formal JTC products.

• Perform end-of-day security checks as assigned for a 1-week period, generally twice annually. Refer to Appendix E for details.

4.2 INTEROPERABILITY BRANCHES

This paragraph content is general in nature. Details on test documentation knowledge management, project resourcing, and the joint interoperability test cycle are provided in Sections V through VII, respectively. Figure 2 shows the interoperability project workflow.

DD Department of Defense RAFT Requirements Analysis Framework for Test JIEP Joint Interoperability Evaluation Plan TCB Test Concept Brief NLT No Later Than TESB Test and Evaluation Strategy Brief POA&M Plan of Actions and Milestones TRR Test Readiness Review

Figure 2. Simplified Interoperability Project Workflow

As defined in JITC Instruction 380-50-02, all JTC-led/supported test events will have the full range of documentation necessary and approved through the appropriate level—BC, DC, or Command—before attending the event. Test event support requires an approved Test Plan (TP) or TSP.

The TO will complete a Test Products Audit Form (TPAF) for all JTC interoperability certification support programs at program/project start. The TO will maintain the form as test documentation is developed and approved. The TPAF standardizes the document review process, provides the mechanism for quality control, and collects historical data for record of a given program. The TPAF template location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Ops\SOP\Audit Process-TPAF.

At project start, or during the annual Test and Evaluation Strategy Brief (TESB), TOs will conduct a project resourcing review with their respective BC to ensure that the program’s interoperability certification efforts are adequately supported by fiscal, human, and materiel resources. As an annual requirement for all interoperability projects, the TESB is conducted by 31 May and addresses activities for the following Fiscal Year (FY) to provide funding information to the supported PMOs. The TESB also provides level-of-effort information to JTC leadership. The TESB template location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Ops\SOP\Briefing Templates.

All JTC interoperability certification support programs will produce, and be based on, a RAFT—essentially the combination of an Integrated Architecture Traceability Matrix, Evaluation Framework (EF), and Data Source Matrix. The RAFT also includes Test Cards (TCs) for each IER listed. A RAFT traces all requirements through DoD Architecture Framework (DoDAF) products to test the approach through data management and analysis. The RAFT Development Guide location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Ops\SOP\RAFT\Development Guide.

All JTC interoperability certification support programs will have an approved JIEP that includes the RAFT as its core content. The DC must approve deviations from this requirement. The JIEP template location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Ops\SOP\JIEP.

All JTC interoperability certification support programs will conduct a TCB and a TRR with the appropriate level of approval (DC or command level) before going to test.

The TCB and TRR template location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\ JTC Ops\SOP\Briefing Templates.

For the SUT, the TO is responsible for timing and resourcing of the creation, construction, and approval of test documentation and conduct of the TCB.

The DC will approve test documentation no later than 30 days prior to test unless circumstances prevent adherence to this timeline. Disputes between TOs and their contractor support teams regarding test product timeliness or sufficiency will be communicated to the division COR and applicable BC for resolution or elevation.

Microsoft Visio is the required graphics solution for all JTC-produced test documentation. If a graphic is obtained from another organization, the TO will coordinate with the MA to recreate the graphic in Visio.

4.3 STANDARDS CONFORMANCE BRANCHES

As the military standards conformance branch, JTC4 is responsible for the following:

• TDL interoperability and standards conformance testing is done IAW the JITC

“Joint Interoperability Test Plan 3006 for Testing Tactical Data Links,” November 2018, which is located in T:\\JTC TACTICAL SYSTEMS PORTFOLIO\TPET\Library of Reference Material\JITC Guidance.

• IBS interoperability and standards conformance testing is done IAW the JITC “Joint Interoperability Test Plan 3006 for Testing Tactical Data Links.”

• CDL interoperability and standards conformance testing is done IAW the JITC “Common Data Link Compliance Evaluation Master Plan,” July 2003, and JITC “North Atlantic Treaty Organization Standardization Agreement 7085 Certification Test Plan,” July 2013.

• NITF interoperability and standards conformance testing is done IAW the JITC “National Imagery Transmission Format Standard Compliance Test Plan,” 7 November 2013.

4.4 MISSION-BASED INTEROPERABILITY BRANCH

As the MBI branch, JTC2:

• Conducts MBI assurance and validation assessments aligned to approved operational requirements or identified operational gaps.

• Conducts a detailed Desk Top Analysis (DTA) of the identified operational requirement or gap.

• Uses the DTA process to identify mission threads, supporting services, data requirements, IERs, system and version, and data format aligned to the operational requirement or gap.

• Uses information from the detailed DTA to determine solution sets to include training deficiencies; system, software, or hardware deficiencies; or TTP/SOP shortfalls.

• Provides solutions and recommendations aligned to mission performance improvements to the combatant command, Service, agency, or PMO.

• Ensures that MPE stakeholders are informed of results related to the U.S.

military’s dominant paradigm for operations (a six-phase planning construct)—specifically, Phase 0 (Shape) planning—and ensures that the coalition interoperability considerations are captured.

• Researches MBI considerations for coalition partner nations regarding mission thread analysis and data sharing for the U.S. military’s dominant paradigm for operations Phases 0, 1 (Deter), 2 (Seize Initiative), 3 (Dominate), 4 (Stabilize), and 5 (Enable Civil Authority).

• Coordinates MBI results with coalition partner nations via Joint Staff J6 for distribution to the MPE Executive Steering Committee and NATO FMN Management Group.

• Assists combatant commands, Services, and agencies with coalition interoperability issues and coordinates actionable requirements with the appropriate JTC branches.

5. TEST DOCUMENTATION KNOWLEDGE MANAGEMENT

5.1 JTC DATA LIBRARY

The JDL is a dedicated file structure within the JITC shared network storage system T drive. The top-level folder is the JDL. Each branch has its own data library file within the JDL.

Within each branch folder are individual folders for each program/project (e.g., T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Data Library\JTC3 Combat Systems Branch\JPN 32034 P-8A). A template for the contents of a program/project folder is in each branch’s folder under Template for Data Library Folders.

The individual program/project folders contain the respective test and interoperability evaluation documentation. The folders store documentation that describes the funding, RAFT, audit checklist, TP, TSP, and test report, etc. The certification issuance is also included.

JDL files within each branch’s Template for Data Library Folders are as follows:

• Audit Checklist (TPAF)

• Briefs (TCB/TRR/TESB/Requirements Review and Briefing (RRB))

• Certification Issuance

• JIEP

• POA&M

• RAFT

• Requirements Source

• TP/TSP

• Test Report

Files within each branch’s Training folder are as follows:

• JITC New Employee Training Plans

• JITC Intern Rotation Plans

5.2 JITC DATA MANAGEMENT TOOL

The JITC Data Management Tool (JDMT) is a modular, web-based Structured Query Language (SQL)-based data collection, storage, and management tool developed at JITC. The JDMT exists on both the Non-classified Internet Protocol Router Network (NIPRNet) and Secret Internet Protocol Router Network (SIPRNet).

The JDMT NIPRNet location is https://tools.nit.disa.mil/test/jdmt. The JDMT facilitates:

• Self-management of project events, users, and data.

• Dynamic form builder (surveys, data collection, incident reports, etc.).

• Web-based form input.

https://tools.nit.disa.mil/test/jdmt

• Dynamic report generation.

• Data Authentication Group (DAG) scoring tool.

• File and artifact upload/download management.

JIES, part of the JDMT, provides an automated capability to create JIC documentation IAW JITC Instruction 380-50-02 and the JTC SOP. JIES supports the full range of interoperability testing activities from program engagement through test planning, reporting, and close-out. JIES also provides a document repository and toolsets and generates reports applicable to all programs undergoing interoperability T&E.

The JIES repository stores mandated architectural artifacts, Joint Staff J6-approved interoperability requirements, and Information Support Plans (ISPs). It also stores all approved JITC products including the Head Start analysis, RAFT analysis, JIEPs, TSPs, TPs, certification issuances, and Test Incident Reports (TIRs).

The overarching goal for JIES is to ingest JITC Interoperability Process Guide

(IPG)-mandated architectural artifacts for the SUT identified for interoperability test and certification and to produce all test documentation. The JIES capabilities include or will include:

• JIES generation of the Head Start products.

• JIES generation of TCs.

• JIES generation of RAFTs.

• Future capability to generate a TSP.

• Future capability to generate a report based upon the deficiencies of the

DoDAF architecture viewpoint documentation.

• Future capability to generate a statistical analysis of test data results.

• Future capability to generate enclosure tables for certification issuances.

• Future capability to generate certification issuances.

JT4 is developing training as the tool is developed and matured. JTC5 can provide individual training upon request. Each TO is responsible for requesting an account and managing the rights and permissions for projects assigned to that account.

5.3 JITC DATA REPOSITORY

The JITC Data Repository (JDR) is the central location to archive information for a given task order. The TE&C contract requires contractors to establish task folders for their respective task order awards. Contractors must file task order and task order modifications, invoices, and deliverables/reports (excluding Monthly Status Report (MSR) narratives and MSR financials) in the repository. The JDR also serves as the repository for COR files for the contract. CORs will review files at least annually to ensure compliance. CORs are required to file correspondence as specified in the COR designation letter. The JDR location is https://jitcnet.fhu.disa.mil/scripts/dr/.

5.4 ROUTING OF DOCUMENTS WITHIN JTC

The MA tracks documents as received and serves as the division document gatekeeper. Documents requiring signatures within JTC should use the appropriate DISA eForm 9 (Commander’s Watch List (CWL) or non-CWL) for routing. The eForm 9 location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Data Library\ Templates_Form 9s.

Rather than attach the document for review by email, provide the document’s file location (e.g., T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Data Library\JTC3 Combat Systems Branch\JPN 32034 P-8A). This ensures that reviewers make changes to the original document and prevents the creation of multiple versions. Reviewers will use the Track Changes feature (Review tab) in Microsoft Word until the document is ready for signature. Once the final product has been signed, the TO should place it in the JDL. Figures 3 through 5 show the various routing scenarios.

Figure 3. Routing Business Document for Branch Chief Approval https://jitcnet.fhu.disa.mil/scripts/dr/

Figure 4. Routing Business Document for Division Chief Approval

Figure 5. Routing Test Document for JTC5 Review and Division Chief Approval

5.5 ROUTING OF DOCUMENTS ON THE COMMANDER’S WATCH LIST

CWL documents follow the same internal routing process, with the MA tracking documents as the gatekeeper. However, TOs will use the CWL eForm 9 for routing.

TOs should place documents in the JT4 Review shared file location (Z:\\209.22.104.214\Policy\_JT4 Review). Using the routing matrix on the eForm 9, email the next reviewer that the document is ready for review. Once the DC approves a document, it will be forwarded to the JT4 eForm 9 email box for JT4 review. The TO should place the signed document in the JTC Data Library.

Folders within the JT4 Review file structure are designated according to the

SUT’s System Tracking Program (STP) document tracking number. The STP number is assigned for new systems for T&E, and the number is updated for revised systems that require new certification documentation. These folders contain the document to be reviewed and the corresponding eForm 9.

6. PROJECT RESOURCING

Most interoperability projects within JTC are funded with customer-provided reimbursable dollars budgeted for JITC support and provided via a Military Interdepartmental Purchase Request (MIPR). Customer PMOs rely on TOs to accurately estimate the level of resourcing needed to accomplish interoperability efforts for a given FY or calendar year.

These efforts will be communicated to customer PMOs via the DD Form 1144

Support Agreement and POA&M or Fiscal Service (FS) 7600 United States Government General Terms and Conditions (GT&C). The DD 1144 or FS 7600 is a templated, standardized document that defines recurring reimbursable support between DoD agencies and commands. The POA&M conveys the SOW and the fiscal, human, and materiel resources necessary to achieve the negotiated interoperability goals for a given FY. The POA&M template location is T:\\JTC TACTICAL SYSTEMS PORTFOLIO\JTC Data Library\Templates_POAMs. Figure 6 shows the project workflow for resourcing (circled in red).

JIEP Joint Interoperability Evaluation Plan TCB Test Concept Brief

Figure 6. Simplified Interoperability Project Workflow—Project Resourcing

For a new customer project, determining resources begins with the TO and customer discussing the type of system, existence and maturity of JCIDS or requirements-type documentation, timeline projected to complete interoperability evaluation, and customer’s expectations for obtaining interoperability certification. The TO will use this information to develop an engagement and test strategy that yields the necessary certification product and to build and present the initial TESB to JTC leadership.

For projects that have not yet attained the necessary certification, TOs will conduct a TESB annually to determine the effort required to achieve the next FY’s interoperability goals. TOs will engage with customer PMOs to negotiate the development of the SOW, which includes:

• Requirements documentation reviews.

• Test process and procedures development.

• Test documentation production.

• Travel needs to attend customer Integrated Product Team meetings.

• Travel needs to attend test events.

• Test equipment needs for the evaluation.

The TESB will be completed annually in early April and will cover activities for the following FY to provide customer PMOs accurate planning information and to develop or update the SOW. Resourcing for a program requires development of the DD Form 1144 Support Agreement and POA&M. Preferably, the DISA/JITC Support Agreement should be written for a period of performance lasting at least 3 years, although shorter timeframes are acceptable, and include the POA&M and DD 1144 or Fiscal Service (FS) 7600 United States GT&C. Careful and thorough discussion with the customer regarding interoperability T&E goals is essential. Initial support agreements will nominally take 45 to 60 days for approval. Support agreement packages, including DD 1144 amendments, are provided to customers with recurring support by May 31.

Updated support agreement packages require approval by DISA and the JITC commander.

The output of the TESB for new customer engagement is the POA&M for the next FY. The TESB demonstrates to JTC leadership that sufficient, intellectual rigor and customer engagement have occurred and that the test strategy is sufficient. As the project approaches T&E periods, the TESB will form the basis for the TCB. Figure 7 shows the initial/annual TESB cycle. Figure 8 shows the DISA/JITC support agreement process.

POA&M Plan of Action and Milestones TESB Test and Evaluation Strategy Brief

Figure 7. Initial/Annual TESB Cycle

Agrmnt Agreement JITC Joint Interoperability Test Command AO Action Officer MPS Manpower, Personnel, and Security Directorate ASAM Agency Support Agreements Manager Pg Page COCOM Combatant Command Req Required Coord Coordination RM Resource Management Cust Customer SAM Support Agreements Manager DISA Defense Information Systems Agency V Version Fwd Forward

Figure 8. DISA/JITC Support Agreement Process (1 of 2)

Admin Administrator Fwd Forward Agrmnt Agreement JITC Joint Interoperability Test Command AO Action Officer OGC Office of General Counsel ASAM Agency Support Agreements Manager PDF Portable Document Format Cdr/CDR Commander Pg Page Cmptrlr Comptroller Prev Previous Cust Customer Req Required Dep Deputy SAM Support Agreements Manager DISA Defense Information Systems Agency SAR Support Agreements Repository Div Division Sig Signature FGGM Fort George G. Meade V Version

Figure 8. DISA/JITC Support Agreement Process (2 of 2)

7. JOINT INTEROPERABILITY TEST CYCLE

Figure 9 shows the basic test cycle for the three most prevalent types of testing—JITC-led testing, Service testing with JITC participation, and Service testing without JITC participation, which are described in Section VII.1. Figure 9 also shows the objectives that JITC will address through implementation.

CWL Commander's Watch List RAFT Requirements Analysis Framework for Test DoDAF Department of Defense Architecture Framework T Test JIEP Joint Interoperability Evaluation Plan TCB Test Control Board JITC Joint Interoperability Test Command TPAF Test Products Audit Form NR KPP Net Ready Key Performance Parameter TRR Test Readiness Review

Figure 9. Test Cycle Framework

When implementing the test cycle, first and foremost establish a set framework that creates a defined, repeatable, and defendable process based on an underlying, rigorous analysis (i.e., RAFT). This framework should allow sufficient flexibility for programmatic differences. Second, for CWL programs, engage early with JT4 via their review/approval of system JIEPs and participation in (and concurrence with) TCBs.

Delaying JT4’s oversight could hamper the ability to influence test program execution.

7.1 JTC TEST CYCLE DESCRIPTION

JTC generally uses these three distinct categories of testing to make JIC determinations:

• JITC-led Testing with No Outside Agency Participation. The best example is standards compliance/conformance testing conducted by JTC4, but such testing can also be a specific event(s) conducted to produce relevant data to support a joint certification.

• Service Testing with JITC Participation (Responsible TO or Representative).

The most prevalent type of test event is supporting JIC efforts. It consists of a Service-conducted test activity (developmental test or operational test) where JITC’s requirements are coordinated with the Service PMO Program Manager (PM). The TO is responsible for the inclusion of JTC TCs into the Responsible Test Organization’s (RTO) TP and the delivery of test results to JITC. Products from this type of testing may be a single test report or multiple test reports.

• Service Testing Without JITC Participation. JITC leverages, to the extent possible, Service test results from test events in which JITC had no participation and may or may not have had input. The applicability of such results may be limited because Service testing often does not capture JITC’s requirements when no coordination has occurred.

7.2 TEST DOCUMENTATION AND PLANNING ACTIVITIES

Test documentation exists at varying levels of maturity throughout a program’s or project’s life cycle. TOs, depending on when a PMO engages JITC for support, can receive program documentation at the Initial Capabilities Document (ICD), Capability Development Document (CDD), or Capability Production Document (CPD) level to prepare for testing. Based on the maturity of the requirements reflected in the applicable document, TOs may or may not be able to completely address the evaluation strategy, test measure(s), or data analysis corresponding to the IE capability of the program or SUT.

Due to the varying stages of requirements documentation maturity, JITC test documentation naturally corresponds to the foundational document maturity. To begin to understand the SUT, TOs will begin requirements analysis immediately after receiving notification of a PMO’s request for interoperability support. TOs will begin the RAFT as soon as possible after receiving the SUT NR KPP and/or DoDAF architecture products.

If a system does not have testable or measurable requirements, or other substantive requirements-related issues are discovered, the TO must coordinate with the PMO/sponsor and Joint Staff J-6 with sufficient time for resolution before testing.

To assist JITC TOs, JT4A has set up an NR KPP help desk to answer questions related to joint interoperability test, evaluation, and certification based on DoD Instruction 8330.01. The NR KPP help desk email address is disa.huachuca.jt.mbx.joint-interoperability-helpdesk@mail.mil.

7.2.1 Requirements Head Start Review (JT4A)

JITC Instruction 380-50-02 establishes the Requirements Head Start Review service (referred to as Head Start). The Head Start process and JITC notional guide for architectures are available on the JT4A SharePoint site at https://disa.deps.mil/org/ JT4/JT4A/default.aspx.

Head Start is a quick look review of PMO-provided interoperability requirements.

The review has a limited scope to permit quick response to the TO. Head Start products document gaps and discrepancies based on reviewing a specific set of requirements products. Results include a list of joint missions/tasks, measures by attribute, a standards status review, and a traceability matrix.

Head Start results help TOs discover gaps and discrepancies before test planning. By providing this evaluation early in the test requirements development/ review process, JTC can help the PMO/sponsor clarify/correct system requirements and associate them to operational activities. Clearly defined test requirements and traceability are key to JTC designing focused, cost-effective testing (i.e., building the RAFT), writing plans for risk-based T&E, and providing JICs that explain operational impact.

7.2.2 Test Documentation Production

While the TO is ultimately responsible for creation, adjudication, and approval of all test documentation, JTC5 has resources to assist the TO. JTC5 has the document templates, workforce (either government or contract support), and skills needed to draft all the documents for an interoperability test program. The TO will coordinate with JTC5 to obtain document support for each interoperability test program. The TO uses the draft document, fills in the necessary details, completes analysis needed, and submits the completed document for approval.

JT4A is developing a series of notional guides, including test documentation, to assist in the interoperability T&E process. Current certificate templates and instructions are available on the JT4A SharePoint site at https://disa.deps.mil/org/JT4/JT4A/ mailto:disa.huachuca.jt.mbx.joint-interoperability-helpdesk@mail.mil https://disa.deps.mil/org/JT4/JT4A/default.aspx default.aspx and at T:\\JTC TACTICAL SYSTEMS PORTFOLIO \TPET\Document Templates and Such\IOP Certifications.

7.2.3 JT4A Plans and Policy Review/Approval

JT4A performs the following for CWL programs:

• Provides JITC command-level endorsement of JTC test documentation products.

• Ensures uniform construct/implementation of system RAFT as the core of the

JIEP.

• Ensures adequate/executable data management and analysis approach at the IE level.

• Ensures that CWL test documentation and reporting meet the grammar, policy, and rigor standards required of communications with senior-level customers.

7.2.4 Test Products Audit Form

TOs and JTC5 will use the TPAF to establish new programs and track existing programs through their life cycle. The TPAF contains necessary program information related to internal JTC processes and external JITC processes such as STP entry, determination of CWL or non-CWL, and Electronic Report Distribution (ERD) actions.

The TPAF serves as a process standardization function, audit of test products for sufficiency, and record of test documentation completion. The STP is located at https://stp.fhu.disa.mil/default.aspx. The ERD is located at https://jit.fhu.disa.mil/ tools/erd/.

Figure 10 shows the project workflow for test documentation and planning activities (circled in red).

https://disa.deps.mil/org/JT4/JT4A/default.aspx https://stp.fhu.disa.mil/

JIEP Joint Interoperability Evaluation Plan TCB Test Concept Brief

PO&AM Plan of Actions and Milestones TRR Test Readiness Review

Figure 10. Simplified Interoperability Project Workflow—Test Documentation and Planning Activities

7.2.5 Requirements Analysis Framework for Test

All JIC test programs will be supported by an underlying RAFT, which deconstructs and organizes a system’s requirements and IE-level test strategies, as well as data management and analysis plans.

RAFT generation is a standardized approach to trace requirements and define detailed test strategies, data sources, and analysis methodologies into one Microsoft Excel spreadsheet. The RAFT is tied to DoDAF architecture viewpoints and the program’s NR KPP table.

The RAFT provides a holistic, executable framework for the interoperability evaluation. It clarifies NR KPP and associated DoDAF requirements, includes embedded TCs to simplify support of individual events, and improves accountability and tracking of certification requirements for all stakeholders. The RAFT also serves to generate discussions between system stakeholders to resolve differences related to requirements and requirements attributes and to determine testing methodologies.

The RAFT process leads to a more complete reporting of the interoperability status. All test results are entered into the RAFT, providing a repository of concise performance information from which to conduct analysis and reach conclusions. As the IERs are coupled to the NR KPP, the focus is on operational impacts to the users.

Completion of the RAFT results in tighter coupling between desired warfighting capabilities and acquisition community requirements. It clearly illustrates missing requirements related to NR KPP measures and orphaned IERs (i.e., IERs not supported within other architectural views).

The RAFT is essentially a living document. The stages of RAFT development are initial, intermediate, and final. BCs will confirm maturity level of the RAFT prior to authorizing a TRR with the DC.

7.2.5.1 Initial Requirements Analysis Framework for Test

At program start, the TO should receive (or request) the initial requirements documentation from the PMO or as part of the Joint Staff review process. Typically, the initial requirements documents include the CDD, CPD, and ICD.

The Joint Staff…