Bidders Library Interoperability Test and Evaluation - Instructions for JIC With and Without Conditions.docx

Instructions for the Joint Interoperability Certification (JIC) Template (with and without conditions) v 1.0

Table of Contents

Section 1. Introduction Section 2. General Instructions Section 3. Certification Section 4. Enclosure 1, Additional References Section 5. Enclosure 2, Certification Summary Section 6. Enclosure 3, Certification Summary Data Tables Section 7. Formatting Section 8. Frequently Confused Terms Section 9. Assessing Risk and Reporting Untested Requirements Section 10. Roll-Up Explanation Section 11. Operational Impact Explanation Section 12. Certification Review Process Section 13. JT4A POCs

Please send any comments or suggestions to the NR KPP Helpdesk mailbox.

Section 1. Introduction

· These instructions are applicable to certifications with and without conditions, assessments, and denials. See the General Instructions for more details.

· “Certified document” is used throughout these instructions and refers to the document you are using as the source of interoperability requirements. This document will include the Joint Staff (JS) certified Net-Ready Key Performance Parameter (NR KPP) and associated architectures or an alternative approved requirements document.

· All IT, including NSS, and Defense Business Systems that are the subject of the certification will be referred to as the “system under test” or “SUT” throughout these instructions.

· The phrases ‘joint critical requirements’ and ‘joint requirement’ are used in these instructions as the general term for items specified in certified requirements document(s) that need to be addressed in a JIC. They align with the attributes of the NR KPP, the associated measures, and specific technical requirements and include:

· Joint mission(s) the SUT is required to support

· Joint Critical Tasks/Joint Tasks: Joint Mission Critical Net-Ready Operational Tasks that a SUT must perform to support the joint mission

· Critical IEs: Information exchanges necessary to enable the joint critical task(s)

· Critical Networks/Interfaces: Entry into and use of the networks (interfaces) necessary (aka ‘critical’) to transport or enable the critical information exchanges,

· Standards and/or specifications the SUT must conform to or implement.

· The phrases ‘threshold’ and ‘objective’ in this document refer to measures criteria. See General Instructions for more about measure status, conditions and criteria.

· Highlighted items in the instruction are of particular emphasis either for that section, or for the overall development of the cert.

· Section 2 contains general guidance about these instructions.

· Sections 3 through 6 contain instructions for specific parts of the certification.

· Section 7 provides formatting guidance for these certifications.

· Sections 8 through 11 provide additional clarification for areas that are not generally understood and may be updated based on user feedback.

· Section 12 explains the JITC certification review process.

· Section 13 contains JT4A POCs.

Return to the Table of Contents Section 2. General Instructions

1. General Guidance

a. In the template, replace the bold blue text with the appropriate information or delete it if it is not applicable.

b. Use the formatting guidance explicitly stated in this template for a JIC (template is not the comprehensive formatting guide for other JITC test documentation).

c. The “JITC Guide to Test Documentation” is the authority for other JITC test documentation formatting (except for the formatting guidance provided for JIC in the JIC template and these instructions).

2. Certifications

a. You should use this template when the requirements source is a JS-certified NR KPP developed IAW Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 6212.01F, 21 March 2012 or the NR KPP as described in the latest versions of CJCSI 3170.01 Joint Capabilities Integration and Development System (JCIDS), the JCIDS manual, and the NR KPP manual (i.e., certs based on the three-attribute NR KPP).

b. The certified NR KPP may be contained in a:

(1) Capability Production Document (CPD)

(2) Capability Development Document (CDD) (not the norm)

(3) Information System-Initial Capabilities Document (IS-ICD)

(4) Information Support Plan (ISP) (i.e., such as an annex)

(5) Tailored (partial) Information Support Plan (TISP)

(6) Standalone NR KPP package

(7) Note: The DoD CIO or JS may approve other documents as the requirements source; refer to the Interoperability Process Guide (IPG) Version 2 (or later) and consult with JT4A if needed.

c. You should use this template for Joint Interoperability Certifications (JICs) and JICs with conditions.

d. You should use this template for Joint Interoperability Assessments and Denials of Joint Interoperability Certification. Contact JT4A for assistance modifying/tailoring the verbiage and tables as needed.

NOTE: A Denial of Joint Interoperability Certification is issued for systems/capabilities that fail to accomplish a joint critical task (or mission) or are determined to have a discrepancy resulting in a ‘critical operational impact’. If you have a system that fails a joint critical requirement during testing, determine if there are workarounds to perform joint critical tasks and/or minimize the operational impact. This would preclude you from issuing a Denial of Joint Interoperability Certification. See Section 3, page 5 for levels of ‘operational impact’.

e. Refer to the IPG Version 2 and contact JT4A for assistance with recertification and certification extensions.

f. Statuses of measures. (see also the NR KPP Evaluation Guidebook)

(1) Met. To assess a measure as ‘met’, it must be evaluated under the conditions stated in the NR KPP table and/or architectures (if applicable) and it must satisfy the threshold criteria for the measure.

(2) Not Met. A measure is ‘not met’ if it fails to meet the threshold criteria. In cases where the measure fails to meet threshold criteria when the conditions are not established, the AO needs to determine if the status is ‘not met’ or ‘not tested’.

(3) Not Tested. The AO/Tester must determine if measures were not tested (such as when the measure not evaluated under the stated conditions) and report their status as ‘not tested’. Risk must be explained in the Analysis paragraphs of the Summary Report Section 7 and an operational impact assigned.

g. Conditions for Measures. Conditions are variables of the environment that affect the performance of a task. Conditions for measures are statements that provide the operational and environmental context in order to evaluate success based on the criteria. The template and these instructions assume that that all measures were evaluated under the conditions stated in the NR KPP table and/or architectures (if applicable).

h. Criteria. Criteria are used to evaluate successful performance of a measure under a specific set of conditions. The JIC is based on the ‘threshold’ criteria for a measure.

3. Tailoring

a. The paragraphs in the certification should be adequate for most programs. Coordinate with JT4A (Quality Team Lead) if you need to tailor this template for a special circumstance.

Return to the Table of Contents

Section 3. Certification

1. Header

a. Delete Joint Interoperability Certification Template v 1.0

2. IN REPLY REFER TO:

a. Choose your division designator from the drop-down list; e.g., JTA, JTB, JTC

3. Date

a. Enter the day in two-digit format, e.g., 04, 08, 11, etc.

b. Enter the month in three-letter format, e.g., Apr, Aug, Nov, etc.

c. Enter the year in four-digit format, e.g., 2015

4. JOINT INTEROPERABILITY CERTIFICATION FOR DISTRIBUTION

a. Certifications

(1) Do not change

(2) We are required to send certifications to the ISG members. The ISG member list is the same as the distribution list in the certification and the Interoperability Cert Letter Core List in the ERD. You can send certifications to the PM and other interested parties designated by the PM by creating your own recipients list in the ERD. See JITCI 380-50-02 for more details.

5. Subject line

a. Change Program name to the program name, if applicable.

(1) Many, but not all, systems are part of a program.

(2) The program name in this certification must agree with the program name in STP and should agree with the program name in DITPR, https://ditpr.dod.mil/dodcio.

(3) Note: All AOs and support contractors should have DITPR and STP accounts.

b. Change System name to the system name

(1) The system name in this certification must agree with the system name in STP and should agree with the system name in DITPR.

c. Change JETDS designator to the JETDS designation; e.g., AN-USQ-148B V3, if applicable.

(1) The Joint Electronics Type Designation System or AN (originally Army-Navy, but now just AN) system was developed from an Army nomenclature system and a similar Navy system.

d. Change System version ID to the system version ID

(1) The system version ID in this certification must agree with the system version ID in STP and should agree with the system version ID, if any, in DITPR.

6. References

a. Only reference documents that were used as the basis for or development of this certification

b. Do not alter the policy and guidance documents (already in black text)

(1) Reference (a) – Use the requirement document containing the JS certified NR KPP. If you used an older document as a requirements source because it was referenced by this one (the one containing the NR KPP), include it in Enclosure 1, reference (f) or later.

(2) Other documents are possible but these should cover almost all situations. Contact JT4A if additional guidance is needed.

7. Paragraph 1: (Certification with conditions) Certification and Conditions.

a. You must assess the ‘operational impact’ if a system does not “Fulfill” the NR KPP (i.e., meet all the joint critical requirements).

(1) If you have joint critical requirements that were not met or not tested, AND the operational impact (or potential impact) associated with the discrepancy is Major or Moderate; you should be writing a cert with conditions. Address the operational impacts in paragraph 7 of Enclosure 2 and the applicable data table in Enclosure 3.

(2) If the system has failures that are Minor (or None - no failures) as far as operational impact is concerned, you should write a cert with no conditions; but explain the Minor operational impacts in paragraph 7 of Enclosure 2 and the applicable data table in Enclosure 3.

b. Delete (Certification with conditions.)

c. Change System name and System version ID to agree with the subject line

d. Change all / X of Y to “all” if all of the system’s joint critical tasks were evaluated. You can also use ‘all Y of the system’s joint critical tasks…’ where Y is the total number of tasks. Otherwise, change it to the number of tasks performed (X) of the total number of tasks (Y); e.g., 7 of 9 joint critical tasks.

e. Delete all of the optional Paragraph 1, (Certification with no conditions.) Certification

8. Table 1: (Certification with conditions) Conditions

a. Task – These are the same task (or mission) names (not the task IDs) found in Table 3-1 of Enclosure 3. Complete Table 3-1 prior to creating the conditions table and transfer the information from there. See Section 6 of this instruction for details. Table 2 of the cert should have a corresponding ‘task’ entry.

b. Operational Impact

(1) State the operational impact to the joint operational environment, the warfighter, and/or interfacing systems (inter or intra-Service). Operational Impact categories are:

(a) Critical: Prevents the accomplishment of an operational or mission-essential capability or jeopardizes safety or security. “Critical” would not be used in a JIC with conditions because any expected critical operational impact should result in a Denial of Joint Interoperability Certification.

(b) Major: A discrepancy that adversely affects an operation or mission-essential capability, and an acceptable work-around does not exist. This results in a JIC with Conditions.

(c) Moderate: A discrepancy that causes a ‘Major’ operational impact if the IT/NSS is deployed and used as designed; however, a mitigating circumstance or acceptable work-around DOES EXIST to reduce the impact. This results in a JIC with Conditions. Refer to Section 9 for details on ‘acceptable work-arounds’.

(d) Minor: A discrepancy with no adverse effects on the mission, supports a JIC. (Note: If the information about a Minor operational impact would help the stake holders, it can be documented in the affected Attribute’s Data Table as a remark).

(e) None: No discrepancies, supports a JIC.

(2) By definition, if the operational impact for the requirement is Minor or None, then you should be issuing a cert with no conditions. Contact JT4A for guidance documenting a Condition derived from an aggregate of discrepancies individually assessed as having a Minor operational impact.

(3) For whichever status you choose, make sure what you write under the remarks matches the impact.

(4) See Section 11 of these instructions for additional notes about Operational Impact.

(5) Untested Requirements: The situation when for any reason, JITC does not have enough information to evaluate a measure. See Section 9 for instructions on addressing untested requirements. In summary:

(a) ‘Not Tested’ means there is an untested requirement (applies to measures under any of the NR KPP Attributes). Untested joint critical requirements warrant a Condition to the JIC with a succinct explanation in the Remarks column of Table 1 with the corresponding detailed explanation in paragraph 7 of Enclosure 2 (what was not tested, Risk/potential impact, why was it not tested).

(b) If Risk associated with the untested requirements can be estimated, then the Risk can be used to determine if a Condition to the JIC is warranted. See Section 9 for instructions on assessing the Risk.

c. Remarks – Per the IPG, JITC may issue JICs with Conditions (limitations) when only subsets of the joint critical requirements are met. JICs with Conditions provide the system interoperability status for cases where useful capabilities are provided, despite not successfully demonstrating all joint critical requirements (Not Met or Not Tested), AND there are no expected critical operational impacts (includes severe adverse effects on the joint interoperability environment). A JIC with Conditions limits the operational use of the system to only those joint missions, tasks, and supporting requirements (Information Exchanges (IEs), Networks (transport), and interfaces) that were adequately demonstrated. In otherwords, a Condition can be assigned to joint requirements that failed along with those that were not tested.

9. Paragraph 1: (Certification with no conditions) Certification.

a. You should write a JIC without conditions if the system meets all its joint critical requirements.

(1) The system meets all its joint critical requirements OR

(2) The system has no failures or failures that are Minor as far as operational impact is concerned. Explain the Minor operational impacts in paragraph 7 of Enclosure 2 and the applicable data table in Enclosure 3.

(3) If you have joint critical requirements that were not tested, you should be writing a JIC with conditions. See Section 9 for instructions on assessing Risk and reporting untested requirements.

b. Delete (Certification with no conditions)

c. Change System name and System version ID to agree with the subject line

d. Delete all of Paragraph 1, (Certification with conditions) Conditions of Certification and Table 1

e. Re-number Table 2. NR KPP Compliance Status to Table 1. NR KPP Compliance Status

10. Table 1: (Certification with no Conditions) Conditions

a. Delete this table.

11. Paragraph 1: Assessments

a. If you plan to write an assessment, use the following: 1. Assessment. The System name Version System version ID met (all or some) of the joint critical requirements in the Joint Staff-certified NR KPP contained in Reference (d) (if applicable or) the draft capabilities document of Reference (d). The system was assessed rather than certified (choose one of the following)

(1) as the NR KPP has not yet been JS certified.

(2) as the system experienced critical failures that the program office will correct prior to pursuing certification.

(3) as the program office requested to know the system’s current interoperability status.

b. If the system experienced any failures or any requirements were not tested, do not use the conditions paragraph and table to document these items. An assessment does not confer any interoperability status and stating conditions would imply that the system can be fielded/used except for the conditions. Instead, document what worked and what did not using the instructions in Sections 5 and 6.

12. Paragraph 2: NR KPP Compliance Status

a. When you are submitting a certification with no conditions, change this paragraph to read Table 1 instead of Table 2. Otherwise, there should not be any reason to alter this paragraph.

13. Table 2: NR KPP Compliance Status (change to Table 1 if issuing a JIC without conditions)

a. Mission Statement and Tasks – Replace with the mission statement that the system supports. Tasks in this column are the joint mission critical Net-Ready operational tasks identified in Table 3-1. Complete Table 3-1 prior to populating the NR KPP Compliance Status table. This table has two main parts: NR KPP Attribute Status and Overall NR KPP Compliance.

b. Table shown in the template represents a SUT with a single mission. If multiple missions need to be addressed, repeat the information, adding the additional rows, to address all (joint) missions. There should only be one Overall IT NR KPP Status to cover all missions.

c. NR KPP Attribute Status:

(1) Support Military Operations

(a) Reports on the status of support to military missions or mission threads

(b) Based on the status of the applicable mission and task measures.

(c) Only joint critical tasks for the applicable mission should be included

(d) Possible status options:

1. Satisfied – All measures related to the NR KPP attribute ‘Met’ their respective threshold criteria (status of Met).

2. Partially Satisfied – The status when (1) there are multiple measures, (2) at least one measure related to the NR KPP attribute did not meet its threshold criteria (status of Not Met) or was not tested (status of Not Tested), and (3) at least one measure meets threshold criteria (status of Met).

3. Not Satisfied – This is the status when ALL measures related to the NR KPP attribute did not meet threshold criteria (no measure with a status of Met).

(2) Entered and be Managed on the Network

(a) Reports on the status of the system’s ability to connect to and/or be managed on the required networks

(b) Does not report on the status of interfaces (SV-1/2)

(c) Possible status options:

1. Satisfied

2. Partially Satisfied

3. Not Satisfied

(3) Effectively Exchange Information

(a) Reports on the status of information exchanges (OV-3)

(b) Does not report on the status of interfaces (SV-1/2)

(c) May report on the status of system data exchanges (SV-6)

(d) Does not report on the adequacy of the architecture viewpoints

(e) Possible status options:

1. Satisfied

2. Partially Satisfied

3. Not Satisfied

d. Overall IT NR KPP Compliance Status –

(1) The Overall IT NR KPP Compliance Status represents a roll-up of all attribute measures statuses, for all joint critical tasks.

(2) Roll-up status is discussed in detail in Section 10

(3) Choose the status from the drop down list. Possible status entries are:

(a) Fulfilled – This is the Status given when ALL attributes for ALL joint critical tasks have the status of Satisfied.

(b) Partially Fulfilled – This is the overall status given when (1) a combination of attribute threshold statuses exists (such as a combination of Satisfied, Partially Satisfied, and/or Not Satisfied) OR (2) all attributes have a threshold status of Partially Satisfied.

(c) Not Fulfilled (rare) – This is the overall status given when ALL threshold attribute statuses of ALL joint critical tasks have a status of Not Satisfied.

e. Remarks

(1) Inform the reader of additional information they need to know to understand the status.

(2) Explain the extent the system met the requirements of this task or attribute; i.e., the number of MOPs met, not met, or not tested.

(3) An operational impact statement is required for each mission and task that does not have a status of ‘satisfied’ for all their attributes.

(4) Risk of not testing must be addressed for any part of an attribute that is not tested.

14. Paragraph 3: Test Details

a. Change testers at location(s) to list the test organization(s) that participated in the interoperability evaluation, and to list the location or locations where the test was performed. If no other organizations participated, list JITC.

b. Change test date to test date., and List any post-test activities completed on (date) according to:

(1) The dates of actual testing.

(2) Post-test activities completed on date should include the type of activity and the date it was completed; e.g., data reduction was completed on 23 July 2012.

c. Delete (Create a table if needed. Include any other information that is necessary to understanding the certification.)

(1) If you have multiple testers, locations, and/or test dates, you may create a table to more easily present the information.

(2) You will have to tailor the paragraph to account for the added table.

d. Change Additional test and evaluation is needed to clear the Conditions listed in Table 1 if appropriate according to the following guidance:

(1) Delete the sentence if there are no conditions.

(2) If not all joint critical tasks were tested, use the default verbiage.

(3) If all joint critical tasks were tested and conditions remain that could be cleared by provision of additional documentation, write it as “Additional information is required to clear the Conditions listed in Table 1.” An example would be if data or results from an external source or test event address an untested measure.

(4) If all joint critical tasks were tested and Conditions remain that will require re-testing to clear the Condition; use the default verbiage.

15. Paragraph 4: Certification Authority

a. Do not alter.

16. Paragraph 5: Additional Information

a. There should not be any reason to alter this paragraph.

17. Paragraph 6: POC

a. Make the following alterations:

(1) Change Name to the government AO’s name

(2) Change (xxx) xxx-xxxx to the government AO’s commercial telephone number

(3) Change (xxx) xxx-xxxx to the government AO’s DSN telephone number

(4) Change firstname.mi.lastname.civ@mail.mil to the government AO’s e-mail address

(5) Choose an office code from the drop down list, and then choose the appropriate address from the 2nd drop-down list.

18. Signature Block

a. Change DIVISION CHIEF NAME to the government AO’s division chief's name.

b. Change Division Name to the government AO’s division name.

c. If you have any doubts about this signature block, consult with the division administrative support assistant.

d. 3 Enclosures a/s

(1) 3 is the number of enclosures

(2) a/s means “as stated”

19. Distribution List

a. All certifications are required to be sent to the ISG members. The distribution list in the template is the current ISG membership list (available at the ISG Resource page).

b. If the distribution list does not fit on the signature page, then move the entire list to the next page.

c. If you are not sending the JIC to any of the COCOMs, delete this entry. Otherwise, specify the COCOM(s) to which the JIC should be distributed. You will need to include the entry in the recipients you create in the ERD.

d. Note: You may add any other recipients as agreed to with the PMO by creating a separate recipients list in the ERD. See JITCI 380-50-02 for details.

e. ERD/STP Codes – There are no longer different categories of joint interoperability certifications (e.g. no more limited certifications). As such, JITC needs to classify the three possible certification outcomes, and identify the related certification products, for ERD/STP tracking and reporting purposes. Include one of the following three letter codes at the end of the distribution list to identify the certification result and product.

(1) Joint Interoperability Certification (a ‘recertification will be either (a) or (b))

20. Header (for all certification pages)

a. Choose your office code from the drop-down list; e.g., JTA, JTB, JTC

b. Change Program name to the program name, if applicable.

(1) Many, but not all, systems are part of a program.

(2) The program name in this certification must agree with the program name in STP and should agree with the program name in DITPR, https://ditpr.dod.mil/dodcio.

(3) Note 1: All systems (and the programs when applicable) are required to be listed in STP.

(4) Note 2: All AOs and support contractors should have DITPR and STP accounts.

c. Change System name to the system name

(1) The system name in this certification must agree with the system name in STP and should agree with the system name in DITPR.

d. Change JETDS designator to the JETDS designation; e.g., AN-USQ-148B V3, if applicable.

(1) The Joint Electronics Type Designation System or AN (originally Army-Navy, but now just AN) system was developed from an Army nomenclature system and a similar Navy system.

(2) Delete this portion if your system does not have a JETDS designator.

e. Change System version ID to the system version ID

(1) The system version ID in this certification must agree with the system version ID in STP and should agree with the system version ID, if any, in DITPR.

(2) Your system version ID may be identified as a version, increment, block, or spiral.

(3) Delete this portion if your system does not have a system version ID (applies to some platforms).

Return to the Table of Contents

Section 4. Enclosure 1, Additional References

1. CJCSI 3170.01I (three attribute-based NR KPP)

a. Reference (c) will always be the JITC IPG.

b. Reference (d) will always be the JCIDS Manual with errata. Check periodically for any updates to the manual with errata https://intellipedia.intelink.gov/wiki/JCIDS. Replace the date in blue text with the date of the latest manual with errata.

c. Reference (e) JS NR KPP certification memo

e. CJCSI 6212.01F has been canceled. Do not use this as a reference in your document unless you have referred to it in Table 2-1 of Enclosure 2 or elsewhere in the certification/assessment.

f. Interoperability Certification Evaluation Plan (ICEP), if applicable

f. Interoperability Test Plan (ITP), if applicable

g. Others, as required

2. Reference rules

a. Assign a consecutive lowercase letter enclosed in parentheses starting with (a) for the first reference.

b. Identify references specifically and completely in this order: originating agency, type of document, office code, subject, and date.

c. Only list references you mention in the text AND ones available to the addressee.

d. List references in the order they appear in the text.

e. If a second line is needed, begin typing text under the first letter of the first word in the first line.

Section 5. Enclosure 2, Certification Summary

1. Title

a. Do not alter

2. Paragraph 1. SYSTEM AND REQUIREMENTS IDENTIFICATION

a. State that Table 2-1 contains the system and requirements identification.

3. Table 2-1 System and Requirements Identification

a. System Identification, Sponsor

(1) Enter the name of the sponsoring organization.

(2) For example, GCCS-J Program Management Office (include the acronym and its complete spelling in the Legend)

b. System Identification, Sponsor Point of Contact

(1) Enter the name of your primary POC at the sponsoring organization.

(2) Include name, phone number, and e-mail address

c. System Identification, Program Name

(1) Enter the program name, if applicable.

(2) Many, but not all, systems are part of a program.

(3) The program name in this certification must agree with the program name in STP and should agree with the program name in DITPR, https://ditpr.dod.mil/dodcio.

d. System Identification, System Name

(1) Enter the system name.

(2) The system name in this certification must agree with the system name in STP and should agree with the system name in DITPR.

e. System Identification, Increment and/or Version

(1) Enter the increment or version of the system.

(2) Version identification is mandatory.

(3) We must be able to unambiguously identify what we are certifying.

(4) Use the naming convention the program uses in its documentation, such as:

(a) Version

(b) Increment

(c) Sprint

(d) Block

(e) Other

f. System Background, Milestone/Decision supported

(1) Enter the milestone/decision; e.g., MS C, LRIP, Full Deployment

g. System Background, Previous certifications

(1) Enter the name, subject line from the certification, and date of any previous certifications for the system.

(2) Include previous system versions

h. Tracking, STP #

(1) Enter the STP system number

i. Tracking, DITPR ID

(1) See https://ditpr.dod.mil/dodcio

(2) In some cases you may need to include archived systems in your search; check the “Include Archived” checkbox.

j. Requirements Source, Type of Requirements

(1) Choose the requirements type from the drop-down list. Selections are:

(a) CJCSI 6212.01F NR KPP

(b) JCIDS Manual

k. Requirements Source: Document Control Number, Document Title, and Date (for NR KPP)

(1) Enter the Document Control Number, Document Title, and Date

(2) Example document control numbers:

(a) KM/DSv2

1. 200000571

(b) IAM

1. N2015-ISP-0506

(3) Enter the complete document title “exactly as the title appears on the actual document”

(4) Enter the date of the document, not the date of the NR KPP certification memo

l. Requirements Source, NR KPP Certification Memo Date

(1) Enter the Subject line and date of the JS Certification Memo

(2) For other types of JS certification (PITT, N/A), list the memo here and provide clarification in the Remarks section.

m. Requirements Source: DARS, WMA, other Service source

(1) Enter “DARS Legacy” and this link https://sadie.nmci.navy.mil/jafe/arch_projects/arch_project.aspx?Tab=3 if the program has entered the architectural viewpoints into the DARS legacy catalog.

(2) Enter “WMA-AFIP” and this link https://sadie.nmci.navy.mil/jafe/ if the program has entered the architectural viewpoints under one of the tabs located in the JS J6 Warfighting Mission Area (WMA) Architecture Federation and Integration Portal (WMA-AFIP).

(3) If the architecture viewpoints are located within another Service resource, state the name of the resource and provide the link.

n. Requirements Source, Remarks

(1) Enter any information needed to clarify any items/elements listed elsewhere in the table.

(2) Do not enter something like “The system was certified based on requirements from the system ISP.”

(3) Clarify here if you have an older or unusual requirements source.

(4) If none, enter “None”.

4. Paragraph 2. SYSTEM DESCRIPTION

a. State the all the joint critical mission(s) that the SUT supports. Do not cut and paste from the NR KPP as the statements in the table are not always an adequate description of either the mission(s) or the tasks. It is possible that not all the joint critical missions the SUT supports will be stated in the NR KPP table. Make sure you’ve reviewed the architecture viewpoints to identify any other joint critical missions the SUT must support.

b. Do not include advertising or adjective “fluff” provided by the program office.

c. The description must be understandable by a reader with a technical background but who is not a subject matter expert.

d. The description should be able to answer the question “What mission(s) does this system support?” if available; and provide a brief functional description of the system, and how the system supports the mission(s).

e. Include ALL of the joint operational missions the SUT supports.

5. Paragraph 3. OPERATIONAL ARCHITECTURE

a. Enter a diagram and text of how the SUT fits into its operational architecture. It must show the system under test, its needlines and interfacing operational nodes, and be consistent with the rest of the certification.

b. Typically, an OV-2 or SV-1 is adequate. An OV-2 is acceptable if it contains sufficient detail to explain the operational architecture.

c. Being in a certified document does not make an inadequate OV-2 or SV-1 adequate. In other words, you may need to construct your own diagram.

d. An OV-1 is not sufficient.

6. Paragraph 4. SYSTEM CONFIGURATION

a. There should not be any reason to alter this paragraph.

b. Enter the hardware, software, and firmware versions of the various system components into Table 2-2. This includes the system under test, interfacing systems, and test equipment.

(1) While it is desirable to declare all versions in a system, large systems will predicate a large list. To make the table for a large system manageable and reasonable in size, include only the hardware, software, and firmware needed to perform the core essential functions of the system to support its mission.

(2) For example, peripheral auxiliary devices/hardware and software versions for mice, printers, and scanners should be excluded from the list.

c. This information should be gathered prior to the start of testing. Verify the actual system configuration tested and update this information as needed. Coordinate with the PM early to insure that he/she is aware of the need to disclose the system configuration prior to testing.

d. Certifications are based on specific versions of hardware and software, so a failure to gather this information could make the certification meaningless. JITC certifies IOP on particular builds of the system under test.

e. You may tailor this table to fit your needs, as long as everything is identified.

7. Paragraph 5. TEST CONFIGURATION

a. Provide a diagram and text of the test environment (architecture) and identify significant deviations from the operational architecture (deviations that would impact test results), e.g., using a radar simulator versus actual radar, using a satellite simulator versus the actual satellite link, or evaluating the system only in the European theater rather than in both the European and Pacific theaters.

b. The figure should contain a relatively simple block diagram that shows the system under test, the interfacing systems, and any test equipment used during the test.

c. There is no DoDAF architectural viewpoint that contains ALL this information. An SV-1 is NOT suitable, as it depicts the operational design and not the test configuration.

d. A network diagram that contains the IP addresses, system names, etc., that is used to set up the test network is not appropriate because it contains extraneous information.

e. An example of an incomplete test configuration is one where there is a requirement to evaluate voice exchanges, but the test architecture only shows a configuration for data exchanges.

f. State whether the system was tested in its approved cybersecurity (formerly IA) configuration and provide the configuration information.

g. If the system does not have an approved cybersecurity configuration, discuss the C&A status and the source of the cybersecurity configuration, i.e., what documented source for the cybersecurity configuration you used during the test. The cybersecurity configuration, as tested, must be captured and documented whether it is provided by the Program or documented by the tester.

h. REMINDER: It is important that the test configuration provides the stated conditions for the measure. The AO/Tester must determine if any measures were not evaluated under the stated conditions, and report their status as ‘Not Tested’.

8. Paragraph 6. METHODOLOGY

a. Enter a brief description of the methodology used to evaluate the system. The methodology for each NR KPP attribute must be addressed.

b. Key information to include are sample size considerations, specific conditions for the measure, and significant deviations from the test plan such as a change or a lack of instrumentation.

c. The methodology description should have enough details to be understandable, but it should not be more than a short paragraph for each attribute.

d. When appropriate, include a brief description of how Science Based Test Design (SBTD) was used, which would include Design of Experiments and a description of the statistical analysis. Contact your JITC Operations Research/Systems Analyst (ORSA) for assistance.

9. Paragraph 7. INTEROPERABILITY REQUIREMENTS, RESULTS, AND ANALYSIS

a. These instructions address the required paragraphs and table(s) to describe the requirements, the results, and the analysis you performed to arrive at those results/conclusion(s). You may add clarifying information as needed.

b. If the SUT has more than one mission, you will add “Requirements and Results” paragraphs; joint critical tasks tables; and “Analysis” paragraphs for each mission. They will be grouped according to mission.

c. If the SUT has more than one mission, but each mission has only one task associated with it, then all entries can appear in the same table. You will still add additional paragraph “pairs” for each mission (“Requirements and Results” and “Analysis”).

d. Paragraph 7.1 – Requirements and Results.

(1) Replace the blue text with the mission stated in Paragraph 2 System Description.

(2) The mission(s) will be stated in Table 3-1(Attribute 1).

(3) If no mission is identified, trace through the architecture viewpoints in order to identify the mission(s).

(4) If the system has no mission, then it serves no purpose.

(5) If you are presenting the results for another mission, start the next “Requirements and Results” paragraph after the previous “Analysis” paragraph; and add the corresponding table and “Analysis” paragraph. Modify the table number to correspond to the table whose mission you are currently discussing.

e. Table 2-3 – Joint Critical Task Results

(1) Remove for Mission # if the SUT has only one mission. Otherwise, change the text to black and replace # with the number of the current mission you’re describing. Mission numbers will be 1, 2, etc., and correspond or remain in the same order as those presented in Enclosure 3, Table 3-1. Refer to Tables 3-1, 3-2, and 3-3 in Enclosure 3 for the data needed to populate your results tables.

(2) Mission MOE Results column –

(a) Input the ratio of MOEs Met/Total # of MOEs for the mission.

(b) If there’s only one MOE (and it was “Met”), then you would input a ratio of 1/1.

(c) If there is no MOE shown in the NR KPP, delete the Mission MOE Results column.

(3) Task Description –

(a) List here the joint critical tasks identified in Table 3-1. These will match the tasks listed in the ‘NR KPP Compliance Status in the certification.

(b) Note, if no tasks are present in the NR KPP table, trace through the architecture viewpoints to identify the tasks the SUT must perform in order to accomplish the mission Again, these should be identified in Table 3-1.

(c) Every SUT will perform at least one task or mission (contact JT4A for clarification if the SUT IOP requirements do not specify missions and/or tasks).

(d) Provide the task name, T# from Table 3-1 Enclosure 3, and a BRIEF description of the task.

(4) MOP Results columns –

(a) Input the ratio of MOPs Met/Total # of MOPs for each of the three categories (Task, Network, IE).

(b) Every mission will have at least one MOP for each task, network, and IE.

(c) If these are not listed in the NR KPP table, you will have to trace through the architecture viewpoints to identify them.

(d) All IEs should trace back to joint Task, and to a joint Mission via one or more Networks. If these “orphans” do not, re-trace the information in the architecture viewpoints. If you are unable to trace “orphan” information exchanges back to a joint critical task, then you probably shouldn’t be evaluating that exchange, and you won’t reflect it in Table 2-3. JITC’s focus should be on joint interoperability evaluation and certification, not intra-Service exchanges that do not support a joint critical mission/task.

(5) Condition to the Certification –

(a) Choose “Yes” if any MOPs “Not Met” or “Not Tested” caused one or multiple conditions to the certification (for the associated task).

(b) Choose “No” if any MOPs “Not Met” or “Not Tested” did not cause any conditions to the certification (for the associated task).

(c) Conditions to the certification could range from a particular interface not being recommended for use, specific network configuration changes required for a particular theater, or message formats that are incompatible across other Services or Agencies.

(d) The condition column is tied to the mission and/or task. If there are multiple joint critical tasks for a mission, it is possible to have a condition associated with each task. A mission MOE can be ‘Met’, and there could still be a condition to a task. If, for example, a supporting IE adversely impacts performance of the task (or effectiveness of the mission). Keep this in mind when you are reviewing requirements. Make sure there is a clear traceability between IEs, the enabling transport (networks, interfaces, etc.), and the joint critical missions and/or tasks.

f. Paragraph 7.2 – Analysis.

(1) Provide a description of the analysis associated with any tasks that had a MOP status of “Not Met” or “Not Tested”, for any attribute as indicated in Table 2-3 (e.g., if any of the MOP results (ratios) are not Y/Y, such as 2/3, 5/8, etc.).

(2) For analyses of multiple tasks, create subparagraphs to describe the analysis performed for each task and its elements.

(3) Order the subparagraphs by task # and by operational impact within a given task.

(4) Describe MOP failure (Not Met) and the analysis used to reach the results. Some areas to look at include:

(a) Significance of sample size to show rigor in test

(b) Data adjudication, data processing, etc.

(c) Compare the result (measured value) to the criteria

(5) For MOPs “Not Tested”, provide the reason or rationale for not testing that requirement. Discuss:

(a) The risk assessment (negative consequence and likelihood of occurrence). Refer to the risk instructions in Section 9.

(b) The test environment (test limitations, fidelity of instrumentation, etc.)

(c) Data collection (test deviations, sample size too small, etc.)

(d) Coordination with the Users to determine the operational impact, if any

(6) Describe how the level of operational impact was determined (applies to both “Not Met” and “Not Tested” MOPs). Discuss any work-arounds, to include resulting changes (downgrades) of reported Operational Impact (Table 1), if applicable.

(7) Repeat for each failed or not tested requirement.

g. Paragraph 7.3 – Summary Report Data Tables.

(1) There should be no reason to alter the text for this paragraph.

(2) This will always be the last paragraph of the results section. Change the number according to the number of paragraphs you’ve presented in your results.

(3) The data tables in Enclosure 3 summarize results used in this evaluation.

(4) Tables 3-1 to 3-3 present the status of measures organized by NR KPP attribute.

(5) Table 3-4 presents test results for joint critical/specified interfaces.

(6) Table 3-5 presents deficiencies implementing special technical specifications (standards, specifications, etc.) and whether that deficiency adversely affected the mission or joint critical task.

10. Paragraph 8. TEST LIMITATIONS AND DEVIATIONS

a. Current joint interoperability policy requires tests that employ production-representative systems in as realistic an operational environment as practicable.

b. Identify any testing limitations (planned) and/or deviations (unplanned) that may affect the interpretation of the results, including:

(1) Use and status of authorized cybersecurity configurations

(2) Use of simulation; e.g., the unclassified-but-encrypted network to simulate the SIPRNet

(3) Any non-operational test environments; e.g., developmental test, laboratory test

(4) Not using real operational users, i.e., personnel from the program office acting as users

c. Provide an assessment of the effect of these limitations and/or deviations on the ability to evaluate requirements (measures) and any effect on evaluating joint interoperability (i.e., ‘Conditions’).

d. Additional testing may be warranted in cases where the operational architecture is incomplete or numerous configurations are possible.

e. It is appropriate to recommend verifying the interoperability of untested configurations. This is a cautionary note that relates to a ‘test limitation’, such as when multiple approved configurations are possible but JITC only tests and evaluates a subset of those configurations.

11. Paragraph 9. CONCLUSION(S)

a. State the conclusion(s) in terms of ‘(1) fulfillment’ of the NR KPP and (2) the certification status of the system.

b. Choose which of the two statements in the template apply to the certification (i.e., Certified or Denial of Certification).

c. Tailor the statement to match your conclusion (e.g., fulfills or partially fulfills, etc.).

d. The conclusion should not be more than a sentence or two; it is not necessary to repeat data or discussion that was already presented in the results section.

e. If the format is being applied to something other than a JIC, you may include longer conclusions if needed, but keep the conclusions to the type of information a high-level decision maker would want to know; i.e., did the system meet its joint interoperability requirements.

12. OPTIONAL – Paragraph 10. RECOMMENDATIONS

a. Not required for a JIC.

b. You may add recommendations if you think they are appropriate.

c. If you identify issues that did not result in a ‘Condition’ to the JIC, but you want the users to be aware of them, include them here.

Section 6. Enclosure 3, Data Tables

1. Table 3-1. Support Military Operations (Attribute 1)

a. Mission and Tasks

(1) Enter the mission name (verbatim from the requirements document)

(2) Mission reference number – Assigned by you and based on the number of missions for the SUT. If only one, use M1.

(3) Used to establish cross-references within the certification

(4) Used link to a Task #, Network #, and IE #

(5) Enter the task name (verbatim from the requirement document)

(6) Task reference number – Assigned by you and based on the number of tasks per mission for the SUT. If only one task, use T1.

(7) Reference numbers will be used in Paragraph 7 of the Summary Report and in the tables of Enclosure 3. Reference numbers must be assigned even if only one of each is present for this evaluation.

(8) This includes mission and/or tasks specified in the architecture viewpoints (or equivalent approved requirements document) and explain the source in the Remarks column.

(9) REMEMBER: All IEs should trace back to joint Task, and to a joint Mission via one or more Networks. If these “orphans” do not, re-trace the information in the architecture viewpoints. If you are unable to trace “orphan” information exchanges back to a joint critical task, then you probably shouldn’t be evaluating that exchange, and you won’t reflect it in Table 2-3. JITC’s focus should be on joint interoperability evaluation and certification, not intra-Service exchanges that do not support a joint critical mission/task.

b. Measures and Threshold Criteria

(1) Enter the measure statement verbatim from the requirements document (MOEs for Missions and MOPs for Tasks).

(2) Enter the threshold criteria (value) for the measure.

(3) This includes mission and/or tasks specified in the architecture viewpoints (or equivalent approved requirements document) and explain the source in the Remarks column.

c. Results – Enter the actual test results (measured values) here; e.g., “4 seconds” (measure is timeliness in seconds), “10 out of 10 met the threshold timeliness requirement of 4 seconds” (measure is related to # of successes vs # of attempts), 86% (measure is in percent success), etc.

d. Threshold Status – Choose the status for each mission and task measure(s)

(1) Met – Used if each Mission or Task measure met its Threshold criteria.

(2) Not Met – Used if each Mission or Task measure did not meet its Threshold criteria.

(3) Not Tested – Used if each Mission or Task measure was not tested. This should be the least used status option. A status of “Not Tested’ could result when needed data was not collected for various reasons, such as the “environmental” conditions associated with the measure criteria were not established when collecting data to evaluate the measure.

e. Remarks – USE THESE INSTRUCTIONS FOR ALL SUMMARY REPORT DATA TABLES

(1) Should clarify and provide additional information on the measured values versus the Threshold criteria.

(2) Should identify and explain the operational impact of any failures.

a. Work with the User/User Representative to determine the operational impact for any deficiencies discovered (Not Met).

(3) Should identify and explain the Conditions resulting from untested requirements.

a. Work with the User/User Representative to determine the Risk associated with an event that could adversely affect a joint task or mission and if a Condition to the JIC is warranted (see Section 9).

(4) Should be the same content used for the Conditions table if you are certifying with conditions.

(5) Make sure the remarks you write match the status and operational impact you chose, i.e., don’t write the failure resulted in a Minor operational impact when you reported Major for your operational impact in Table 1.

h. Notes (Same for all tables) – Use as necessary; add a new row between the last table entry and the Legend.

i. Legend (Same for all tables)

(1) Define all acronyms, symbols, and abbreviations (with the exception of common non-technical items such as &, %) used in this table, including the acronyms in the table title, unless they are defined within the table contents.

(2) There are usually two columns of acronyms and two columns of expansions (definitions) in the legend. If the column contents end unevenly, make the left-most of the two columns longer than the right ones.

(3) Double-check all expansions.

(4) Don't rely on the program or your memory.

2. Table 3-2.