Bidders Library Interoperability Test and Evaluation - JITC Notional Guide to TE Documentation.pdf

F

DEFENSE INFORMATION SYSTEMS AGENCY

JOINT INTEROPERABILITY TEST COMMAND

FORT HUACHUCA, ARIZONA

NOTIONAL GUIDE TO TEST &

EVALUATION DOCUMENTATION

VOLUME II – OPERATIONAL TEST &

EVALUATION AND GENERAL TEST

PLANS/TEST REPORTS

VERSION 1.0

JUNE 2020

NOTIONAL GUIDE TO TEST & EVALUATION DOCUMENTATION

VOLUME II – OPERATIONAL TEST & EVALUATION AND GENERAL TEST

PLANS/TEST REPORTS

JUNE 2020

Submitted by: Alan R. Rieffer Chief, Strategy and Policy Branch

Terry Powell Chief, Joint Operational Evaluations Branch

Approved by: ______________________________

AMANOLLAH ADELI

Chief, Strategy, Plans, and Engineering

Prepared Under the Direction of:

Frank G. Diaz Joint Interoperability Test Command

Fort Huachuca, Arizona

INTENTIONALLY BLANK

TEST & EVALUATION DOCUMENTATION

1. Introduction

2. Operational Test & Evaluation and General Test Plans/Test Report Guidance Appendix A. Writer’s Guidance

1. INTRODUCTION

The Joint Interoperability Test Command (JITC) formally documents test and evaluation (T&E) efforts in plans and reports, to include Operational T&E (OT&E) plans and reports. JITC delivers these documents (tangible products) to customers as part of our T&E services. The Department of Defense (DoD) in particular relies on JITC’s OT&E reports to inform the DoD acquisition decision process.

1.1 Purpose

Ensure the quality (consistency and rigor) of JITC’s formal T&E products using standard content and format guidance for documenting JITC’s T&E efforts.

1.2 Applicability

This guide is applicable to JITC’s formal T&E products as specified in “JITC

Instruction 380-50-02 Joint Interoperability and Standards Conformance Test and Evaluation (T&E) and Certification Instruction dated, 9 January 2017” or any test plan/test report without a prescribed format (e.g. joint interoperability).

The guide is not applicable to Unified Capabilities (joint interoperability) or Tactical Data Link T&E products covered by other JITC publications.

2. OPERATIONAL TEST & EVALUATION AND GENERAL TEST PLANS/TEST

REPORT GUIDANCE

Section 2 of this guide provides details to help shape the organization and content of OT&E and general test plans/test reports on the Commander's Watch List, Director OT&E (DOT&E), Office of the Under Secretary of Defense Acquisition and Sustainment, and oversight. As such, tailor each document/product to satisfy stakeholder needs.

The following symbols are used throughout the guidebook to highlight recommendations and items of importance to the reader:

Best Practices: Recommended processes

For Your Information: Important information

For Your Information Seek JT4A assistance in applying the guidance contained in this document and in tailoring the prescribed test plan and test report format to best address stakeholder needs.

2.1 Operational Test & Evaluation and General Test Plans

DESCRIPTION

This test plan format applies to JITC-developed test plans other than those pertaining to joint interoperability. Unless requirements exist to use a test plan format different than that prescribed by this guide, this format applies to all JITC-conducted operational, standards conformance, developmental, system integration, and user/system acceptance test events. Coordinate with JT4A if you have questions concerning application or tailoring of this test plan format.

The “JITC Operational Test and Evaluation Guidebook” provides additional instruction and considerations for developing OT&E plans. A copy of the most current version of this guide is available at: https://disa.deps.mil/org/JTA/Guidebook/.

When JITC is designated as the Responsible Test Organization (RTO), apply the test plan guidance contained in this document for all OT&E and general test plans (i.e., tests other than those specific to joint interoperability). Additionally, when JITC is designated as the lead Operational Test Agency for a Multi-Service OT&E, the “Memorandum of Agreement (MOA) on Multi-Service Operational Test and Evaluation (MOT&E) and Operational Suitability Terminology and Definitions” also applies. The most current signed version of the MOT&E MOA can be found at: https://disa.deps.mil/org/JTA/OTE%20References.

OT&E AND GENERAL TEST PLANS OUTLINE

Front Matter

• Cover page

• Signature page

• Executive Summary

• Table of Contents

Main Body

• Purpose

• System Description

• Required Capabilities

• Background

• Scope

• Limitations

• Test and Evaluation Methodology https://disa.deps.mil/org/JTA/Guidebook/ https://disa.deps.mil/org/JTA/OTE%20References

• Example Results Appendices

• Acronyms & Abbreviations

• Other Appendices

• References

• Points of Contact

Best Practices

Insert “INTENTIONALLY BLANK” pages in accordance with the guidance provided in Appendix A of this guide.

OT&E AND GENERAL TEST PLANS CONTENT

The following sections provide detailed guidance for the OT&E and General Test Plans.

2.1.1 Front Matter

Front Matter (OT&E and General Test Plans):

The Test Plan’s front matter consists of the content below.

Content Information Description Example/Details

Order of Presentation

The front matter content is presented in this order:

• Cover page.

• Followed by a blank page.

• Signature page, which documents the approving official, typically the Division Chief for systems on the Commander’s Watch List.

• “INTENTIONALLY BLANK” page.

• Executive Summary, which provides information for executive-level readers and stakeholders.

• “INTENTIONALLY BLANK” page.

• Table of Contents, which lists (in order) the main body (including the Executive Summary), Appendices, List of Figures, and List of Tables.

Note: The Appendix (Writer's Guidance) in this guide provides document format details.

Front Matter (OT&E and General Test Plans): Executive Summary The Executive Summary serves to collect key test information for executive level readers and stakeholders to use when making decisions. The Executive Summary explains how JITC will test the system. The intended audiences typically read only the Executive Summary for various reasons, including time constraints or lack of technical proficiency. The Executive Summary provides enough information to understand the test and its relevance without reading the entire document.

Content Information Description Example/Details

Order of Presentation

Paragraph 1: Provide the test purpose (Same as the Test Purpose; do not deviate).

Provide a brief explanation of who requested the test and why (Summarized from the Test and Evaluation Methodology section).

o Match functions to procedures

(Summarized from the System Description and the Test and Evaluation Methodology section).

o Mostly method of test and evaluation (Summarized from the Test and Evaluation Methodology section).

o Keep the focus on the test item, not the test or testers.

Paragraph 2: Provide a brief overview of system functions that relate to the system requirements (Summarized from the System Description).

Paragraph 3: Provide a summary of the test and evaluation methodology describing how JITC will test the system and how the test relates to the system requirements. This paragraph is the most important part of the Executive Summary. If there is room, expand the details to an additional paragraph (Summarized from the Test and Evaluation Methodology section).

Paragraph 4: Administrative details covering whom, when, and where (Summarized from the Test and Evaluation Methodology section).

Note: Write the Executive Summary after completing the Main Body and Appendices. The executive summary should not contain tables and figures.

Front Matter (OT&E and General Test Plans): Executive Summary The Executive Summary serves to collect key test information for executive level readers and stakeholders to use when making decisions. The Executive Summary explains how JITC will test the system. The intended audiences typically read only the Executive Summary for various reasons, including time constraints or lack of technical proficiency. The Executive Summary provides enough information to understand the test and its relevance without reading the entire document.

Writing Aids Description Example/Details

Only Critical Information

Devote most of the space to what the system must do, how we will test it, and how we will evaluate data generated from the test.

Delete everything not directly relevant.

o Be concise and relevant.

o No room for hype or program history.

o No room for expressing generally accepted opinion/belief or embellishment.

o No room for tables or figures.

o Keep to one page when feasible.

Target Audience

Write the Executive Summary for high-level decision makers, not engineers or testers. Avoid technical and tester jargon.

o Help the JITC Commander and other stakeholders understand how the testing approach makes sense and that the test will be adequate for its purpose.

o Assume interest, not expertise.

Consistent Keep consistent with the body of the plan, particularly the purpose.

o Be careful not to modify content or meaning when you summarize.

o Use the same terms.

o Do not introduce new information.

Write the Executive Summary after completing the Main Body and Appendices. The Executive Summary should not contain tables and figures.

2.1.2 Main Body

Main Body (OT&E and General Test Plans): Overview The Main Body contains eight sections.

Content Information Description Example/Details

Outline 1. Purpose

2. System Description

3. Required Capabilities

4. Background

5. Scope

6. Limitations

7. T&E Methodology

8. Example Results

Main Body (OT&E and General Test Plans): 1. Purpose JITC uses the Purpose to focus the scope of testing. Establishing the Purpose with clarity is important because what needs to be determined influences every aspect of the test process, including test design. The purpose of all tests are to “find out” or “determine” something. The Purpose clearly sets the tone for intended accomplishments.

Content Information Description Example/Details

Primary Purpose

Identify the single most important purpose of the test.

o Focus on what most of the testing is about.

o No longer than a sentence.

o Write clearly and to the point.

o Use the Purpose statement in the test plan’s main body in the Executive

Summary.

What We Want to Find Out

Must be answered in Conclusion:

Purpose:

Determine if System “A” is effective and suitable.

o Conclusion: System “A” is (or is not) effective and suitable.

o Example: “JITC determined if System A met its effectiveness and suitability requirements...”

Addresses an End State

JITC wants to determine an end state.

o For JITC, success is identifying whether the system meets its effectiveness and suitability requirements.

o Use unbiased terminology. Our role as testers is to be objective. JITC conducts T&E to validate or verify system requirements.

o Examples of “End State”:

• Fielding decision

• Operational Assessment

• Acquisition decision

Target Audience

Write the Purpose for the general audience and understandable for all.

o Any stakeholders reviewing the plan should understand the verbiage.

Main Body (OT&E and General Test Plans): 2. System Description System function pertains to the interactions that occur between the SUT and other systems via interfaces. The System Description presents the system’s important functions to include the mission(s), capabilities, tasks, and processes performed or observed during test.

Content Information Description Example/Details

System Description Overview

Brief high level description to include:

o Mission supported.

o Capability, tasks performed

(operational or mission activities).

o Required functions.

Subparagraphs as needed to address content described below.

The example to the right is for illustration purposes only.

Ensure you have a lead-in sentence for your system diagram.

o A system diagram should be in this section.

o This includes diagrams/figures developed by JITC.

Note: Diagrams/figures of configuration, operational environment, etc. are sometimes relevant. If the diagram/figure provided accurately represents the operational environment then we can reuse them in the report.

System function pertains to the interactions that occur between the SUT and other systems via interfaces. The System Description presents the system’s important functions to include the mission(s), capabilities, tasks, and processes performed or observed during test.

Content Information Description Example/Details

OV-1 or Appropriate System Diagram

Place the appropriate system diagram in this section.

Include Mission Perspective

Identify the missions the system supports (as provided in requirements documents). This will clarify for readers the potential impact of failures.

o Explain what capabilities are new or improved for the increment and/or version.

Include System Capability

Describe what required capabilities the entire system needs to perform the mission(s).

Note: Any areas not included in the test should be identified/grayed-out as needed;

clearly stating what capability is being evaluated.

o State what the system is “supposed” to do.

o Identify the mission activities supported by the capability.

Include System Boundary

Identify what comprises the system. Define the limits of the specific functionality delivered by a capability. Knowing the system’s functionality helps to determine necessary interfaces to other capabilities and organizations.

o Define specific people (i.e., users, TPOCs…), equipment, roles and responsibilities.

Include System Interaction

Provide a list of the system interfaces and explain how each interacts with the SUT.

o Consider using a table to present this information.

System function pertains to the interactions that occur between the SUT and other systems via interfaces. The System Description presents the system’s important functions to include the mission(s), capabilities, tasks, and processes performed or observed during test.

Content Information Description Example/Details

Include System Functions

Explain the role of information exchange in fulfilling the functions of the system.

Note: Provide as much information as you have at this point of your evaluation strategy.

Writing Aids Description Example/Details

Target Audience Write the System Description for the general audience and understandable for all.

o Any stakeholders reviewing the plan should understand the verbiage.

Avoid Cut and Paste The intent is to convey what the system should do, not to promote it.

Be cautious when using descriptions from the program documents. The statements in the documents typically do not adequately describe the mission(s) or the tasks.

o Summarize the program’s system description to focus on what JITC needs.

o Avoid copying the program’s system description and pasting it into the plan.

o Avoid Program Manager or vendor hype (i.e., greatest, fastest, most reliable, etc.).

o Avoid importing inaccurate and superfluous information into the document.

o Avoid trade jargon, irrelevant or unsubstantiated capabilities (i.e., not relevant to T&E objectives).

Alignment with Other Sections

Clearly relate content in this section to the OT&E or General Test Plan’s Required Capabilities and T&E Methodology sections.

o If applicable, update details from the System Description if system functions changed, the requirements changed, or the test scope changed.

o If information does not relate to the requirements and methodology, remove it from the section or explain why it applies.

o Make sure the system version stated in the System Description is the same as the version stated throughout the document.

Physical Detail Include physical details only if they are relevant to how the system functions or how JITC will test it.

o Provide as much information needed to explain the physical details.

Main Body (OT&E and General Test Plans): 3. Required Capabilities Identify the requirements and capabilities the system must meet for the specific test event.

Content Information Description Example/Details

Mandatory Either list required capabilities or summarize them here. Without clear requirements, it is impossible to understand the test.

o Cannot just refer to an appendix, standard, or other document, at a minimum provide a summary.

o If no required capabilities, explain why.

o Be aware of the different types of required capabilities.

Type of test should address the right requirements.

Account for Every Requirement For A Particular Test Event

Identify all requirements applicable to the system or item to be tested.

Note: Some requirements may be too lengthy to be in the main body.

o If requirements are clearly organized and delineated in the source document by phases such as increments, version, block or sprints, account only for requirements covered by the test of that particular phase.

o If you summarize requirements in the test plan’s main body, provide the detailed requirements in an appendix

Valid Sources For T&E

Required capabilities should be user validated.

o JITC cannot invent requirements, but we can use experience, best practices and sound engineering judgment for derived requirements.

o May use standards, requirements, capabilities, concept of operations, or other sources validated by users.

o Special cases may occur without user-validated requirements. Be very specific about conclusions from these events.

Target Audience Write the Required Capabilities section for the Program Management Office (PMO), developers, and the test team.

o The respective stakeholders reviewing the plan should understand the verbiage.

Main Body (OT&E and General Test Plans): 4. Background JITC uses the Background to highlight the status and need for testing a system. Provide a brief history of previous test events (including testing of previous releases, etc.), or planned test events. Only provide relevant information supporting what will happen in the test.

Content Information Description Example/Details

Primary Purpose

Provide the “why” for the “what” given in the Purpose.

o Give a logical reason for the purpose of the test.

o Do not re-state what will be the purpose.

o Do not describe this test. Test description belongs in Scope and Methodology.

Why Test Now

State the reason or reasons why a customer asked JITC to test the system; e.g., new capability, system upgrade, new way of using the system, new configuration, and new environment.

This section can provide a brief history of previous test events (including testing of previous releases, etc.), or planned testing events if they are relevant to this event.

o Identify the common sense reason that would make testing the logical thing to do.

• Not just why JITC does testing

• Not just that somebody asked us to

• Not that the system needs to be certified

Only Relevant Background

If, and only if, previous testing will affect what or how we will test, then identify the relevant findings and show how they relate to the current test.

o Do not need program history.

o Do not need history of need for a function.

o Do not need general history of testing program.

o If previous testing occurred, explain how it is relevant (some requirements already assessed).

Target Audience

Write the Background section for senior stakeholders and the PMO.

Main Body (OT&E and General Test Plans): 5. Scope Outline what the testing will cover, emphasizing the extent of the test versus the total real-world requirements of the system.

Be sure to include number/name/location of sites participating, number/type of users participating, length of exposure, period of observation, and environmental factors collected.

Content Information Description Example/Details

Test Versus Real Environments (aka Production Environment)

Explain how well the test environment and/or network represent the actual environment and required conditions in which the system will operate.

Explain how the system represents the full range of potential system operations during the test.

Note: Operationally realistic environments are necessary, as much as possible, in operational testing.

o Are they the same, similar, or different in important ways?

o If different, explain why the differences are important.

o One realistic environment may not represent all real-world environments. If so, identify the differences.

o Live environments are sometimes impractical to stage due to prohibitive logistical costs and large scale of effort and resources involved. However, the closer the test environment approximates the live environment the closer it is to operational realism.

o Even if the environment is realistic, the performance demonstrated may not be. Performance with one or two users may not reflect performance with hundreds.

Configuration Diagram

Use a diagram and explanation to clarify relationships, connectivity, and information flow.

o Show via diagram and narratives the test configuration and operational configuration or architecture.

o Diagrams need to be clear, understandable and readable for them to have any value. Do not use a low-resolution diagram from another source. Create the diagram, if needed, for a clearer presentation. Avoid sacrificing clarity for convenience.

o Especially important if configuration may impact performance.

Requirements To Be Tested

Test coverage of requirements to be tested.

o Simple statement will suffice to say JITC will test all requirements, if that fact is the case. If testing only a subset of requirements affirm that fact by stating, “All requirements will be tested except…” and enumerate exclusion.

o State requirements excluded from test, the reason for non-inclusion and any known information that will cover addressing the excluded requirements in the future.

o Clearly address the testing of all identified requirements in the T&E Methodology section. Do not leave any loose ends.

Data Requirements and Source

Data elements required to support analysis of evaluation measures from a linked data source.

o Extrapolate, from the requirements and measures to be tested, the data requirements needed for data analysis.

Provide a comprehensive list of data requirements mapped to system requirements and measures. Add mapping to data sources where required data elements will be coming

Main Body (OT&E and General Test Plans): 5. Scope Outline what the testing will cover, emphasizing the extent of the test versus the total real-world requirements of the system.

Be sure to include number/name/location of sites participating, number/type of users participating, length of exposure, period of observation, and environmental factors collected.

Content Information Description Example/Details from. Use initial or final data requirements identified in other sources (i.e., the TEMP) to populate the list, as applicable.

What, Where, When and Why They Matter

What type of testing, where and when the testing will occur, and what these mean to the test.

Note: This should support the purpose of the test.

o Indicate what the sites and organizations represent, such as a typical joint task force element at a deployed location.

o JITC may need to use other test sources (i.e., interoperability, DT, or Cyber) to augment the test data collected.

o Identify the number/name/location of sites participating.

o Identify the duration of test.

o Identify the number/type of users participating.

o Identify the period of observation.

o Identify the environmental factors collected.

Target Audience Write the Scope section for the general audience and ensure it is understandable for all.

o Any stakeholders reviewing the plan should understand the verbiage.

Main Body (OT&E and General Test Plans): 6. Limitations Limitations refer to known anticipated constraints that will limit the extent/coverage of testing. By extension, a Limitation can affect the test conclusion. It is, therefore, important to know the conclusion we seek in order to understand if a constraint to testing will affect it. Provide a short discussion of known issues that will constrain what we can conclude from the test.

Content Information Description Example/Details

Only Limitations on Conclusions

If the known limitation will not affect the conclusion, omit it.

However, no limitations mean our conclusion is unequivocal.

o Limitations can affect

Purpose or Scope (e.g., If our purpose is to determine ability to support voice communications, it is not a limitation that we will not test video).

o Not a place for detailing and anticipating all possible testing problems.

o Not just when or what we cannot test.

o Not a limitation if never in Purpose or Scope.

o Type of test limitation is dependent on test type.

o A limitation is known ahead of actual execution of test event.

Main Body (OT&E and General Test Plans): 6. Limitations Limitations refer to known anticipated constraints that will limit the extent/coverage of testing. By extension, a Limitation can affect the test conclusion. It is, therefore, important to know the conclusion we seek in order to understand if a constraint to testing will affect it. Provide a short discussion of known issues that will constrain what we can conclude from the test.

Content Information Description Example/Details

Always Include the Impact on Expected Conclusion

Explain the impact of each limitation in the conclusion.

o No value without a “so what?”

o Not just, “so we can’t conclude anything about . . . .”

o Example: Since video is a critical aspect of surveillance data, the system may not be able to support these key intelligence missions.

Always Include Risk to Users

Assess the risk to users: the likelihood of negative consequence; the potential impact on the mission should the system fail or not be net-ready.

o Identify the risk to users, not to testers.

o How likely is there a problem in the untested area or alternative approach?

o How serious would a potential failure be to users?

o Is there risk to a particular task or mission?

Mitigation Address the possible “work-around” or alternative solutions to employ to mitigate the limitations and allowing the accomplishment of remaining testing.

o If taking mitigation actions, the alternative solutions should remove the likelihood of negative consequences or negative impact on the mission.

Target Audience Write the Limitations section for senior stakeholders and the

PMO.

Main Body (OT&E and General Test Plans): 7. T&E Methodology The T&E Methodology addresses how JITC tests a system, specifically detailing means of conducting the test that may include pre-test (i.e., pilot test) and actual in-test conduct information. This is the most important part of the test plan and requires the most detail. Provide additional details in appendices if the information exceeds 2-3 pages.

Content Information Description Example/Details

Roles and Responsibilities o Main participants needed for an organized test event.

o Indicate what sites and organizations represent, such as a typical joint task force element at a deployed location.

o Identify the Test Director, Test Lead, Data Manager, Data

Collectors, participating users, site leads (for multiple test locations), and any other roles needed to conduct the test.

o Describe roles, responsibilities, and interactions of the test participants.

o Consider presenting this information in a table to organize and condense the content.

Primarily What the System Does

Describe the system supporting the test execution. Focus on the system, not the testers.

o Test execution is more than what testers do. It is mostly about how the system supported the test.

o Use specifics of what will occur during the event; e.g., “Airmen will use System A to direct aircraft, send images, and modify purchase orders.”

Operational Condition

Live environments are sometimes impractical due to prohibitive logistical costs and resources involved.

o Are they the same, similar, or different in important ways?

o Explain how the differences impacted testing (e.g., performance with one or two users may not reflect performance with hundreds).

o Understand the physical/logical boundaries or conditions that affected the execution of a test or measurement of a requirement.

Test Versus Real Operation

Collecting data from real operations is sometimes impractical due to prohibitive logistical costs and resources involved.

However, the closer the test environment approximates the live environment the closer it is to operational realism.

o Questions that are often asked to determine this include:

• Will the test of the system be in its real-world environment?

• Will it be tested using operational networks?

• Will the system interfaces and hardware be the same as when used in the field?

• Will the users during the test be the same as in the real-world environment?

• Will JITC be able to meet the test purpose?

The T&E Methodology addresses how JITC tests a system, specifically detailing means of conducting the test that may include pre-test (i.e., pilot test) and actual in-test conduct information. This is the most important part of the test plan and requires the most detail. Provide additional details in appendices if the information exceeds 2-3 pages.

Content Information Description Example/Details

Configuration Diagram

Use a diagram and explanation to clarify relationships, connectivity, and information flows.

Note: The test configuration can impact performance (i.e., hardware, software, or instrumentation needed for test).

o Show via diagram and narratives the test configuration and operational configuration or architecture.

o Diagrams need to be clear, understandable, and readable for them to have any value. Do not cut and paste a diagram from a low-resolution document source. Draw the diagram, if need be, for a clearer presentation.

o Avoid sacrificing clarity for convenience.

o Especially important if the test configuration has the potential to impact performance.

Requirements Being Tested/Test Procedures

Show how JITC will test the applicable requirements during a test event.

Must specify how JITC will test the system for each requirement or type of requirement.

o If JITC will only test a subset of requirements, affirm that by stating, “All requirements will be tested except…” and list the exclusions.

o State requirements excluded from test, the reason for non-inclusion, and any known information that will cover addressing the excluded requirements in the future.

o Clearly address all identified requirements to be tested.

o Do not leave any loose ends.

o Show how the test will cover all applicable requirements.

o Cover how any required conditions will affect mission task.

o Describe use of the system, not just using questionnaires; e.g., “Personnel will use the system under normal operational conditions for 3 weeks.”

o Requirements are aggregated in different forms such as test cases, test scenarios, use cases, test cards, etc.

o Provide the tester or trusted agent the necessary test details to execute the test plan.

o If there are too many requirements supported by many procedures, summarize the procedures in this section. The details of the procedures can be contained in an appendix.

Test Design Identify specified test conditions (coverage items) required for a system/test item to be tested.

Data Collection Purpose of data collection is to acquire data needed to address test objectives and support the analysis of evaluation measures.

o Describe methods on how data requirements will be collected, the data capture tools and instrumentation used (including verification and validation information and capability to provide precise measurements dictated by requirements), and the data extraction points and data sources.

o No data requirements mean no data collection.

o If using data collection forms, place the forms in an appendix.

Ensure the elements in a data collection form align to collected data requirements.

The T&E Methodology addresses how JITC tests a system, specifically detailing means of conducting the test that may include pre-test (i.e., pilot test) and actual in-test conduct information. This is the most important part of the test plan and requires the most detail. Provide additional details in appendices if the information exceeds 2-3 pages.

Content Information Description Example/Details o Sample size determination is an aspect of testing to know how many observations/data points are needed to have accurate conclusions with adequate confidence. For all test types, discuss planned sample sizes of each data element from each data source that address testing requirements and the analysis that led to the determination of the ampleness of that sample size, i.e.

hypothesis testing, estimation, etc.

Science-Based Test Design

(SBTD)

and Design of Experiments

JITC uses Science-Based Test Design, such as DOE, to provide rigor to testing. It is critical to provide the details to show customers the foundations of how JITC evaluates systems.

Note: SBTD is not always required. Use this guidance when applicable.

o Identify the goal of the experiment (objectives or questions to be answered); expected response variables of interest supporting the goal, factors, and accompanying levels affecting the response variables; method of varying factors; statistical measures of merit;

DOE test design; and test execution specifics and other DOE-centered test artifacts.

o Discuss sampling strategy for handling and evaluating large populations of data.

o Discuss sample sizes JITC plans to obtain from each data source for corresponding DOE-linked test requirement (i.e., hypothesis testing, estimation, confidence level, etc.).

Target Audience Write the T&E Methodology section for the test team and test participants.

o The respective stakeholders reviewing the plan should understand the verbiage.

Consistent with Other Sections

Tie the system functions, requirements, and test procedures together to help the reader clearly see test strategy and thoroughness.

o Must have clear traceability from requirements to methodology.

o A large Required Capabilities section with a short Test and

Evaluation Methodology section usually indicates inadequate methodology. The very nature of a large system implies a large number of requirements addressed by a proportionally sized methodology.

Most Important Part of the Plan

All other sections exist to support the Test and Evaluation Methodology.

The value a reader can get from a plan depends on this section.

o The test plan’s primary purpose is to describe the test. Do that in the methodology.

o The test plan’s other sections provide the background to understand and evaluate the methodology’s content.

o If this section is sketchy, readers cannot assess if the test is adequate.

Biggest Section More specifics and detail belong here than anywhere else in the plan body. “Users will perform their normal operational tasks” is not sufficient.

o Details should provide a clear enough picture that only technical experts will need to read the applicable appendices.

o If using an appendix to provide details, then summarize the information in this section and direct the reader to that specific appendix. However, if the appendix is only a few pages, put that information in the main body of this section.

Main Body (OT&E and General Test Plans): 8. Example Results Present example results and statuses (using simulated content) in graphs, charts, combined charts, tables and/or narratives to improve the clarity of the information provided. Be sure to tie the graphs, charts, etc. to the requirements, methodology, and evaluation framework to address issues and provide answers.

Content Information Description Example/Details

Example Tables and Narrative

Used to begin the process of preparing the report or any subsequent document requiring the presentation of results and/or statuses.

o Should be able to fill in tables while collecting and analyzing data.

o Change tables in report if appropriate.

Result Versus Status

A result is the outcome of the test.

A status is the attainment or non-attainment of measurement criteria of that outcome.

Note: Statuses are not results. In fact, they rely upon comparisons of results to criteria and are typically a basis for an evaluation leading to compliance.

o Results are direct products of the test conducted.

o Statuses are direct products of evaluation and analysis.

o Do not show the reader statuses without showing what JITC is measuring, the results of measurement, and measure’s criterion. Statuses are meaningless if not showing the traceability to the measure.

o Include and address all requirements.

Expressing Results and/or Statuses

Know what method of results and/or status reporting is best suited to use in the test report or other document.

Some base methods for expressing results on the type of document o Do not create a table when a simple statement will do.

o Unnecessary to include statements like “We will present results in narrative.”

o Tables provide a visual method of viewing results and traceability to required capabilities. Tables summarize a large amount of data. Place additional details in an appendix.

Main Body (OT&E and General Test Plans): 8. Example Results Present example results and statuses (using simulated content) in graphs, charts, combined charts, tables and/or narratives to improve the clarity of the information provided. Be sure to tie the graphs, charts, etc. to the requirements, methodology, and evaluation framework to address issues and provide answers.

Content Information Description Example/Details that will report the results and statuses (e.g., In an interoperability certification the tables for reporting results of measures are already pre-determined and shown in a template).

o Narratives are typically suited in presenting small amounts of results that stand alone in the main body of a report.

o Narratives allow great flexibility in extended discussions of results and/or statuses.

Writing Aids Details Example/Details

Target Audience

Write the Example Results for the general audience and understandable for all.

o Any stakeholders reviewing the plan should understand the verbiage.

Layout and Content

Organize the layout of the example results.

o Layout shows what distinctions the reader will be able to make (large versus small files, images versus text).

o Content shows example data (attempts/successes, maximums, percentages).

Simplicity and Clarity

Colocate side-by-side measures, criteria and results, and other data needing comparison.

o The easiest comparisons are between side-by-side placements of information.

o Use space to organize.

o Group to make trends and exceptions stand out.

2.1.3 Appendices

Appendices (OT&E and General Test Plans):

Three appendices are required in a JITC test plan: Acronyms & Abbreviations, References, and Points of Contact.

Additional appendices may be used, and the appendices may include more technical/essential information than found in the main body of the document. This technical/essential information provides an understanding of the planned test that, if omitted, may inhibit execution or replication of the test. Keep the intended audience in mind and tailor the appendices as appropriate for the SUT. Insert the other appendices between the Acronyms & Abbreviations and References appendices, in descending order of importance.

Content Information Description Example/Details

Required Appendices

The OT&E and General Test Plans must have these appendices:

• Acronyms & Abbreviations

• References

• Points of Contact

Note: List all appendices in the Table of Contents and the body of the OT&E or General Test Plan.

Note: Appendices B and C may change based on the additional appendices used in the Test Plan. The above example is for illustration purposes only.

Target Audience for the Required Appendices

Write the Required Appendices for the general audience and understandable for all.

o Any stakeholders reviewing the plan should understand the verbiage.

Additional Appendices Use additional appendices when needed.

o If the information is too lengthy (exceeding 2-3 pages in Main Body), provide a summary of pointer in the Main Body and move the detail to an appendix.

o Over-elaborate details tend to clutter up the document main body and make it difficult for the reader to understand content.

Examples include but are not limited to:

• Analysis Structure

• Data Management

• Data Forms and Reports

• Data Authentication Group

• Test Entrance Checklist

• Test Resources

• Test Roles and Responsibilities

• Supplemental Investigative Surveys (Identify the reason for the surveys in the title)

Appendices (OT&E and General Test Plans):

Three appendices are required in a JITC test plan: Acronyms & Abbreviations, References, and Points of Contact.

Additional appendices may be used, and the appendices may include more technical/essential information than found in the main body of the document. This technical/essential information provides an understanding of the planned test that, if omitted, may inhibit execution or replication of the test. Keep the intended audience in mind and tailor the appendices as appropriate for the SUT. Insert the other appendices between the Acronyms & Abbreviations and References appendices, in descending order of importance.

Content Information Description Example/Details o The sections most likely to require an additional appendix are Required Capabilities and the Test and Evaluation Methodology.

o Other information required to support the OT&E or General Test Plan.

o Be sure to continue sequential labeling for additional appendices.

Target Audience for Additional Appendices

Write each additional Appendix for the test team and test participants.

o The respective stakeholders reviewing the plan should understand the verbiage.

Detailed Requirements and Procedures

Describe the detailed requirements and exact procedures in an appendix if the requirements and test procedures are so complex that the plan body only summarizes them.

o The most important appendix contains the specifics of the test.

o Include criteria, data requirements, test conduct, test cases, use cases, and data collection not described in the body. Provide enough detail that another test team could replicate the test using only the test plan as guidance.

o Must track with the main body. Put details in appendices.

Do not introduce unrelated test procedures.

o If, and only if, the details are too big for itemized enumeration in the appendix, place an embedded file in the appendix. The reader can expand the embedded file to see the details*.

*Note: JITC's distribution process changes files from Microsoft Word to PDF's, which can make an embedded file impossible to open. Additionally, some security systems will remove embedded files from certain documents.

Only as Technical as Necessary

May include more technical information than in the plan body, but keep as readable as possible, especially in procedures.

o Include the details needed by technical experts to understand the test.

o Provide specifics needed by users to implement.

o Provide enough information for the OT&E or General Test

Plan to stand-alone.

2.2 Operational Test & Evaluation and General Test Reports

DESCRIPTION

This test report format applies to JITC-developed test reports other than those pertaining to joint interoperability. Unless requirements exist to use a test report format different than that prescribed by this guide, this format applies to all JITC-conducted operational, standards conformance, developmental, system integration, and user/system acceptance test events. Coordinate with JT4A if you have questions concerning application or tailoring of this test report format.

The “JITC Operational Test and Evaluation Guidebook” provides additional instruction and considerations for developing OT&E reports. A copy of the most current version of this guide is available at: https://disa.deps.mil/org/JTA/Guidebook/.

The report should focus on the operational significance or “so what” factor for the warfighter.

OT&E AND GENERAL TEST REPORTS OUTLINE

Front Matter

• Cover page

• Signature page

• Executive Summary

• Table of Contents

Main Body

• Purpose

• System Description

• Background

• Scope

• Deviations (Conditional)

• Test and Evaluation Methodology

• Results and Analysis

• Conclusion

Appendices

• Acronyms & Abbreviations

• Other Appendices

• References

• Points of Contact https://disa.deps.mil/org/JTA/Guidebook/

Insert “INTENTIONALLY BLANK” pages in accordance with the guidance provided in Appendix A of this guide.

OT&E AND GENERAL TEST REPORTS CONTENT

The following subsections provide detailed guidance for the OT&E and General Test Reports.

Write the Test Report for the general audience and understandable for all stakeholders involved.

2.2.1 Front Matter

Front Matter (OT&E and General Test Reports):

The Test Report’s front matter consists of the sections discussed below.

Content Information Description Example/Details

Order of Presentation

The front matter content is presented in this order:

• Cover page.

• Followed by a blank page.

• Signature page, which documents the approving official, typically the Division Chief for systems on the Commander’s Watch List.

• “INTENTIONALLY BLANK” page.

• Executive Summary, which provides information for executive-level readers and stakeholders.

• “INTENTIONALLY BLANK” page.

• Table of Contents, which lists (in order) the main body (including the Executive Summary), Appendices, List of Figures, and List of Tables.

Note: The Appendix (Writer's Guidance) in this guide provides document format details.

Front Matter (OT&E and General Test Reports): Executive Summary The Executive Summary serves to collect key test information for executive level readers and stakeholders to use when making decisions. The Executive Summary explains how JITC tested the system. The intended audiences typically read only the Executive Summary for various reasons, including time constraints or lack of technical proficiency. The Executive Summary provides enough information to understand the test and its relevance without reading the entire document.

Content Information Paragraph Description Example/Details

Order of Presentation

Paragraph 1: Provide the test purpose (Same as the Test Purpose; do not deviate).

Provide a brief explanation of who requested the test and why (Summarized from the Test and Evaluation Methodology section).

o Match functions to procedures

(Summarized from the System Description and the Test and Evaluation Methodology section).

o Mostly method of test and evaluation (Summarized from the Test and Evaluation Methodology section).

o Keep the focus on the test item, not the test or testers.

Paragraph 2: Provide a brief overview of system functions that relate to the system requirements (Summarized from the System Description).

Paragraph 3: Provide a summary of the test and evaluation methodology describing how JITC tested the system and how the test related to the system requirements. This paragraph is the most important part of the Executive Summary. If there is room, expand the details to an additional paragraph (Summarized from the Test and Evaluation Methodology section).

Paragraph 4: Administrative details covering whom, when, and where (Summarized from the Test and Evaluation Methodology section).

Note: Write the Executive Summary after completing the Main Body and Appendices. The Executive Summary should not contain tables and figures.

Front Matter (OT&E and General Test Reports): Executive Summary The Executive Summary serves to collect key test information for executive level readers and stakeholders to use when making decisions. The Executive Summary explains how JITC tested the system. The intended audiences typically read only the Executive Summary for various reasons, including time constraints or lack of technical proficiency. The Executive Summary provides enough information to understand the test and its relevance without reading the entire document.

Paragraph 5: Add a paragraph summarizing the test findings/conclusions.

Writing Aids Description Example/Details

Mostly How Devote most of the space to what the system did, how JITC tested it, and how JITC evaluated data generated from the test.

o Match functions to procedures (Summarized from the System Description and the Test and Evaluation Methodology section).

o Mostly method of test and evaluation (Summarized from the Test and Evaluation Methodology section).

o Keep the focus on the test item, not the test or testers.

Only Critical Information

Delete everything not directly relevant.

Keep to one page.

o Be concise and relevant.

o No room for hype or program history.

o No room for expressing generally accepted opinion/belief or embellishment.

o No room for tables or figures.

Consistent Keep consistent with the body of the plan, particularly the purpose.

o Be careful not to modify content or meaning when you summarize.

o Use the same terms.

o Do not introduce new information.

2.2.2 Main Body

Main Body (OT&E and General Test Reports): Overview The Main Body contains eight sections.

Content Information Description Example/Details

Outline 1. Purpose

2. System Description

3. Background

4. Scope

5. Deviations (Conditional)

6. Test and Evaluation Methodology

7. Results and Analysis

8. Conclusion

Main Body (OT&E and General Test Reports): 1. Purpose JITC uses the Purpose to focus the scope of testing. Establishing the Purpose with clarity is important because what needed to be determined influenced every aspect of the test process, including test design. The purpose of all tests were to “find out” or “determine” something. The Purpose clearly set the tone for intended accomplishments.

Content Information Description Example/Details

Identify the single most important purpose of the test.

o Focus on what most of the testing was about.

o No longer than a sentence.

o Write clearly and to the point.

o Use Purpose statement in the test plan’s main body in the Executive

Summary.

What We Want to Find Out

Must be answered in Conclusion:

Purpose: Determine if System “A” is effective and suitable.

o Conclusion: System “A” is (or is not) effective and suitable.

Example: “JITC determined if System A met its effectiveness and suitability requirements...”

Addresses an End State

JITC wanted to determine an end state.

o For JITC, success is identifying whether the system met its effectiveness and suitability requirements.

o Use unbiased terminology. Our role as testers was to be objective.

JITC conducts T&E to validate or verify system requirements.

o Examples of “End State”:

• Fielding decision

• Operational Assessment

• Acquisition decision

Main Body (T&E and General Test Reports): 2. System Description System function pertains to the interactions that occur between…