Exhibit_5F_TFITS_System__Test_and_Evaluation_Strategy.pdf

Unclassified

SYSTEMS TEST & EVALUATION STRATEGY (ST&ES)

TOTAL FORCE INFORMATION TECHNOLOGY

SYSTEMS (TFITS)

Version 3.0

Prepared by:

Prepared by:

MARINE CORPS SYSTEMS COMMAND (MCSC)

TFITS

Information Systems & Infrastructure (IS&I)

Quantico, Virginia 22134

TFITS ST&ES

ii

SIGNATURE PAGE

ST&ES

TECHNICAL REVIEW:

Mr. Robert Lovejoy Assistant Product Manager – Engineer (APdM-E)

PMM 110 IS&I, TFITS

MCSC

APPROVED BY:

Mr. Lou Peradotto Product Manager

PMM 110 IS&I, TFITS

MCSC

iii

Table 1: ST&ES Update Record Revision Number Date

Log of Changes Made and Description of Reason/Changes Approved By

1.0 11/13/2013 Initial Version Mr. William Sandlin

2.0 8/27/2014 Added the following sections:

• 2.0 Reference Documents

• 3.3.1 High Level Schedule

• 3.3.2 Developmental Test Process

• 4.1.1 Test Risk Assessment

• 4.3 Government Acceptance Test

(GAT)

• 4.3.1 GAT Structure

• 4.3.2 Requirements Continuity

• 4.4 GAT Test Environment

• 5 GAT Process

• 5.1 Test Readiness Review (TRR)

• 5.2 Entrance Criteria

• 5.2.1 System Maturity

• 5.2.2 Documentation

• 5.2.3 Other Pre-GAT Consideration

• 5.3 GAT Conduct

• 5.3.1 Roles and Responsibilities

• 5.3.2 GAT Execution

• 5.3.3 Data Collection

• 5.3.4 Failure Definition/Scoring Criteria

(FD/SC) Conferences

• 5.4 GAT Closeout and Reporting

• 5.4.1 Test Incident Reports (TIRs)

• 5.4.2 Engineering Change Proposals

(ECPs)

• 5.4.3 GAT Reporting

• 5.5 Post Implementation Validation

Testing (PVT) Updated the following sections:

• 3.3 Schedule

• 4.1 Test & Evaluation (T&E) Strategy

• 4.2 Level of Testing Required

• 4.2.1 Defect Monitoring

• Appendix A: Failure Definition/Scoring

Criteria Deleted the following section:

• Basis of Test Design

Mr. Lou Peradotto

2.1 10/28/2015 Added the following sections:

• Appendix B

Updated the following sections:

• 1.1 Purpose

• 1.2 Scope iv Unclassified

• 2 Reference Documents

• 3.2 Test Personnel

• 3.3.1 High Level Schedule

• 3.3.2 Developmental Test Process

• 3.3.3 Timeline

• 4.1 T&E Strategy

• 4.2 Level of Testing Required

• 5.2 GAT Entrance Criteria

• 5.4.3 GAT Reporting

• Appendix A v

Table of Contents

1. Introduction – Purpose and Update Plan

1.1 Purpose

1.2 Scope

2 Reference Documents 3 Test Program Management and Schedule

3.1 T&E Organization

3.2 Test Personnel

3.3 Schedule

3.3.1 High Level Schedule

3.3.2 Developmental Test Process

3.3.3 Timeline

4 Test and Evaluation Strategy

4.1 T&E Strategy

4.1.1 Test Risk Assessment

4.2 Level of Testing Required

4.3 Government Acceptance Test

4.3.1 GAT Structure

4.3.2 Requirements Continuity

4.4 GAT Test Environment

4.5 Test Limitations

5 GAT Process

5.1 TRR

5.2 GAT Overview

5.2.1 System Maturity

5.2.2 Documentation

5.2.3 Other Pre-GAT Considerations

5.3 GAT Conduct

5.3.1 Roles and Responsibilities

5.3.2 GAT Execution

5.3.3 Data Collection

5.3.4 FD/SC Conferences

5.4 GAT Closeout and Reporting

5.4.1 TIRs

5.4.2 ECPs

5.4.3 GAT Reporting

5.5 PVT

6 Abbreviations and Acronyms Appendix A: FD/SC ................................................................................................................................. A-1 Appendix B. Cybersecurity Test & Evaluation ....................................................................................... B-1 vi Unclassified

Table of Tables Table 1- Technical Artifact Maturity by Milestone Event Table 2-Timelines of GAT Events by Octet Table 3- Level of Testing by Octet Table 4- System Integrator Test Results Score Card Table 5- Roles and Responsibilities

Table of Figures Figure 1- High Level Schedule Figure 2- GAT Results Tracker Metrics

Unclassified

1 Introduction – Purpose and Update Plan

1.1 Purpose

The purpose of this ST&ES is to define the T&E approach for the TFITS Product Office projects.

This ST&ES will provide overarching guidance common to all projects within the TFITS portfolio, across the System Engineering Technical Review (SETR) process.

1.2 Scope

The TFITS T&E effort will ensure that any project that is completing a new release or post deployment update is sufficiently tested. This will include the following:

• Monitor System Integration Testing (unit and integration) via defect tracking and provide metrics

• Develop validation test(s) for enhancements

• Conduct regression tests

2 Reference Documents

a) Defense Acquisition Guidebook, Chapter 9

b) Institute of Electrical and Electronics Engineers Standard 12207-2008, "Standard for Systems and Software Engineering- Software Life Cycle Processes”

c) Interim Department of Defense (DoD) Instruction 5000.02, “Operation of the Defense Acquisition System,”

d) Marine Corps Operational Test and Evaluation Activity (MCOTEA) Information and Business Systems Risk Assessment Process, 17 December 2012

e) Memorandum for Users of the Director, Operational Test and Evaluation Test and Evaluation Master Plan Guidebook, 27 February 2012

f) SECNAV M-5000.2 Department of the Navy Acquisition and Capabilities Guidebook, May 2012

g) SECNAVINST 5000.2E Department of the Navy Implementation and Operation of the Defense Acquisition System and the Joint Capabilities Integration and Development System, 1 September 2011

h) TFITS- Configuration Management Plan

i) TFITS- Performance Management Plan

j) TFITS- Requirements Management Plan

k) TFITS- Systems Engineering Plan

l) United States Marine Corps Integrated Test and Evaluation Handbook, 6 May 2010

m) MCSC Acquisition Guidebook – October 2014

n) United States Marine Corps Integrated Test & Evaluation Handbook – MCSC Annex

o) DoD Cybersecurity T&E Guidebook V1.0 -01 July 2015

p) DoD Cybersecurity T&E Overview 04 May 2015

q) Defense Information Systems Agency (DISA) Security Technical Implementation Guides (STIG, Security Requirement Guides, tools, guides resources) - http://iase.disa.mil/stigs/Pages/index.aspx

r) DoD Instruction 5000.02- Operation of Defense Acquisition System

s) DoD Instruction 8500.01- Cybersecurity

t) DoD Instruction 8510.01Risk Management Framework (RMF) for DoD Information Technology (IT).

Unclassified

3 Test Program Management and Schedule

3.1 T&E Organization

The T&E planning for a project shall include the Project Manager, Test Engineer, System Engineer, Information Assurance, the System Integrator, and the Advocate. For 1st and 2nd octet changes, other participants such as the Joint Interoperability Testing Command, MCSC Systems Engineering, Interoperability, Architectures, and Technology, MCOTEA, Subject Matter Experts (SME) or interfacing project representatives may be required and a T&E working group may be established.

3.2 Test Personnel

• Test Director will be the Government Test engineer.

• Test Conductor may be the Government Test Engineer or Project test support personnel, and will be responsible for the day to day conduct of the test. The test conductor will liaison with the System Integrator (SI) before and during the test event. The test conductor will track test progress against scope and provide daily hot wash reports.

• Recorder. The test recorder(s) may be SI, government or contract support personnel and will be responsible for recording all incidents that occurred during the test event, using the TFITS TIR. All TIRs generated shall be reviewed by the Test Conductor before they are entered into any defect repository, or reviewed at a Failure Definition Scoring Criteria Conference or hot wash.

• Data Analyst. The data analyst will work with the systems engineer to determine the requirements satisfaction post GAT in support of PRR. These results will be contained in the Software Test Report and recorded in the Government approved repository.

• Testers. Typical users of the system.

• System Integrator. The SI will plan and execute all Systems Integration Developmental Testing (SIDT). SIDT shall not require government users to execute the SIDT activities, however, all SIDT activities may be observed by the Government. The SI shall provide a liaison during GAT to facilitate communication between the Government Test Conductor and SI Developers.

3.3 Schedule

3.3.1 High Level Schedule

Test planning should begin with Engineering Review Board and continue through the requirements and design process. Through the design and development process, T&E focus should be on the testability of the requirements. The TRR is the culminating event prior to the GAT, at which point T&E focus shifts to traceability of the requirements to the test procedures, ensuring that all requirements will have been verified by the time GAT is complete. Post GAT activity is focused on verifying (validating) that system function in the production environment enables the users to accomplish their mission. This is accomplished by means of functional verification checklists and monitoring of SI Helpdesk metrics.

Figure 1- High Level Schedule

Table 1 below depicts the technical artifacts that are deliverable to the Government by the SI at each SETR event, and characterizes the state of completion expected. All artifacts are required for all TFITS portfolio projects, regardless of octet level. Where applicable in Table 1 below, the Data Item Descriptions shall be used for the specified artifacts.

Table 1- Technical Artifact Maturity by Milestone Event

Technical Artifacts SR R

SFR

PD

C D

SID

T

TR

SV

PR

PIR

OCTET

1 2 3 4

For combined SETRs, the more mature condition applies RR DR PR Software Requirements Specification (DI-IPSC-81433A) D F U U U U U U U R R R R Software Design Description (DI-IPSC-81435A) D F U U U U U R R R R Software Test Plan (DI-IPSC-81438A) D F U U U U U R R R R Software Test Description (DI-IPSC-81439A) D F U U R R R R Software Test Report (DI-IPSC-81440A) D F R R R R Requirements Traceability Matrix D U U U U U F R R R R PVT Checklist D U F R R R R

D = DRAFT, F = FINAL, U = UPDATE, R = REQUIRED

3.3.2 Developmental Test Process

The decisions proceeding towards GAT should be event driven, not schedule driven. A System Under Test (SUT) must be demonstrated as ready for GAT through an iterative process of SIDT prior to Government acceptance. Individual software units are tested discretely in an increasingly integrated fashion that culminate in a final integration test, and Cyber Security scans. Prior to the beginning of SIDT, the SI shall demonstrate readiness to proceed at a Systems Integration Developmental Testing Readiness Review (SIDTRR). The Software Test Description (STD) will be finalized, and all requirements shall be traceable to one or more test cases. Requirements

Unclassified documentation will be updated, reflecting the current baseline. The method of evaluating the SIDT results will be established (e.g. test incident evaluation, scorecards, and test reports). SIDT testing can be done with or without Government witness, but all SIDT results must be presented for Government review prior to kickoff of the SIDT and finalized before the TRR.

3.3.3 Timeline

Table 2 below reflects the timelines by which GAT and its associated SETR activities should take place, and when SETR artifacts are deliverable. The timelines noted below align to the overarching timelines defined in Table 4-1 of System Engineering Plan, Reference K. Table 1 delineates which artifacts are required by T&E for each review.

Table 2-Timelines of GAT Events by Octet

Activity 1st Octet 2nd Octet 3rd Octet 4th Octet

SIDTRR Artifacts Received

≥ 15 Days before

SIDTRR

≥ 15 Days before

SIDTRR

≥ 7 Days before

SIDTRR

≥ 2 days before SIDTRR, if required TRR Artifacts

Received ≥ 15 Days before

TRR

≥ 15 Days before

TRR

≥ 7 Days before

TRR

≥ 2 days before TRR, if required

GAT Conducted ≥ 5 days after TRR ≥ 5 days after TRR ≥ 3 days after TRR as required

GAT Preliminary Results Published

≤5 Days after

GAT

≤5 Days after

GAT

≤3 Days after GAT as required

PRR Artifacts Received

≥ 15 Days before

PRR

≥ 15 Days before

PRR

≥ 7 Days before

PRR

≥2 Days before

PRR

PVT Conducted ≤5 Days after Deployment

≤5 Days after Deployment

≤3 Days after Deployment

≤3 Days after Deployment

Post Implementation Review (PIR)

Artifacts Received

≥ 15 Days before

PIR

≥ 15 Days before

PIR

≥ 7 Days before

PIR

≥ 2 days before PIR, if required

4 Test and Evaluation Strategy

4.1 T&E Strategy

The TFITS test strategy is based on a verification and validation process. It calls for using the verification performed by the developer and a validation performed by the government to characterize the performance of the system in its environment. The test strategy also relies on the constant monitoring of the design and development of the system through requirements traceability and assessment.

The System Integrator will develop and maintain a comprehensive software test plan (STP). The Program Office will review and approve the STP and all applicable updates. The Government will also review the SI-produced STD to ensure system performance and configuration integrity is maintained throughout the SUT’s life cycle, and to ensure that all system requirements are assessed

Unclassified through inspection, analysis, demonstration, or test, prior to Government acceptance. The internal tests the System Integrator will conduct shall include Unit Testing, Integration Testing, and Regression Testing.

• Unit testing occurs during system development. Unit testing is characterized by the testing of discrete software units.

• Integration testing occurs after the software build has been applied to the pre-production environment. Integration testing ensures that the system is operating as intended prior to government validation by verifying that the individual software units operate cohesively.

• Regression testing will ensure that the application of changes to the existing software does no harm to the existing functionality. The scope of regression testing required will depend on the octet of the change.

These tests may be monitored by the Project Office test lead or project support personnel. Defects noted during testing shall be assessed for severity in accordance with Appendix A of this ST&ES.

SIDT shall result in the production of an SIDT report, which shall characterize the progress of the system towards readiness for Government acceptance. A system with unresolved severity 1 and 2 defects should not be deemed ready for GAT.

4.1.1 Test Risk Assessment

The Test Risk Assessment questions (see Annex A of Reference E) will be evaluated by the project office. This evaluation should be conducted for each SETR event (System Requirements Review (SRR), System Functional Review (SFR), Preliminary Design Review (PDR), and Critical Design Review (CDR)).

4.2 Level of Testing Required

The level of testing will depend on the scope of the release. The octet level is defined in the TFITS Configuration Management Plan. The following chart depicts the anticipated testing for each release octet. The Level of Testing required may be tailored to support a specific project release as determined by the Test Lead.

Table 3- Level of Testing by Octet

Unclassified

4.3 Government Acceptance Test

The GAT serves as the culmination of all industry and government DT. During the GAT, the SUT will demonstrate mission accomplishment or service fulfillment in an environment representative of the operational environments (for programs in TFITS, “mission” equates to service fulfillment). After GAT, any performance limitations are known, bounded, and predictable and the MCSC-designated technical authority will have sufficient information to proceed to PRR.

The GAT design begins with the premise that all lower level unit and integration testing is complete, and that the system is ready to be exercised at representative tempo and loads. The test operators shall concentrate on performing their missions or using the services rather than fault isolation or debugging.

If the operators cannot demonstrate effective operation, as evidenced by the existence of Severity 1 and/or Severity 2 defects, then the system will fail GAT. Defects characterized as Severity 3, 4, or 5 may be fixed and retested prior to PRR. New ECPs will be created for any unresolved defects or for enhancements that were determined not to be within scope of the release.

4.3.1 GAT Structure

To the extent practical, GAT will use a modular design that allows independent execution of GAT segments. This will allow testing of discrete enhancements and fixes, which, when combined with a comprehensive regression test, provide an efficient means of validating configuration-controlled change. Each segment will be in the form of a realistic scenario or vignette, which demonstrates an end-to-end activity (service provider-to-client) in support of a service. The regression test will exercise the entire functional baseline, but at an abbreviated level of detail. It verifies that controlled changes to the baseline have not negatively impacted previously demonstrated capability.

4.3.2 Requirements Continuity

GAT validates fulfillment of requirements in two ways: analytically, GAT reviews the Requirements Traceability Matrix (RTM), both government and System Integrator. This ensures that during the engineering process (deriving secondary and tertiary requirements, refining and then allocating them to software or hardware components at increasing levels of integration— from unit to component to system) the requirements are tracked continuously, accounted for, and matched to test results. In other words, the GAT team validates the entire system engineering and test “V” progression (Figure 1) up to the point that it is delivered to the government.

Demonstratively, GAT addresses high level requirements, including Key Performance Parameters and Critical Operational Issues, from the perspective of the service client. This proves that services are fulfilled as specified.

4.4 GAT Environment

The test environment for GAT shall be operationally representative of the network environment upon which the SUT is to be deployed. The extent to which the test environment represents a loss of fidelity with the operational environment shall be documented in the STP as a test limitation.

Many projects within the TFITS portfolio are used to manage the personnel and training records of Marines. Any production data that is brought into the testing environment will be assessed for the inclusion of personally identifiable information (PII). Where the PII is not essential for the fidelity of

Unclassified testing, such PII shall be obfuscated or removed from the data set. If it is not possible to remove or obfuscate PII, the test environment shall be required to adhere to the same level of Cyber Security (CS) controls as the production environment.

4.5 Test Limitations

The GAT environment for most SUTs within the TFITS portfolio may not be the same environment for the SUT’s operational employment. Performing GAT on a garrison or tactical Marine Corps Enterprise Network environment may be impractical or impossible. Actual users of the SUT may not be available for GAT. Deviations from an operationally representative environment may include, but are not limited to:

• Network performance (bandwidth, latency, physical infrastructure)

• Operator interfaces, user terminals, internet browser versions

• Operator experience level, number of concurrent users

The extent to which these deviations may affect the ability to generate realistic and predictive GAT results shall be documented as test limitations in a project specific STP. Any risks that test results may be less than predictive of the system operating in its intended environment should be identified at

TRR.

5 GAT Process

5.1 TRR

The TRR is a multi-disciplined technical review designed to ensure the subsystem or system under review is ready to proceed into formal acceptance test. The TRR assesses test objectives, test methods and procedures, scope of tests, and safety and confirms that required test resources have been properly identified, made available, and coordinated to support planned tests. The TRR verifies the traceability of planned tests to program requirements and user needs. It determines the completeness of test procedures and their compliance with test plans and descriptions. The TRR also assesses the system under review for development maturity, cost/schedule effectiveness, and risk to determine readiness to proceed to formal testing. The TRR verifies that cross competency (PM, LOG, SE) coordination of test resources (people, facilities, test articles, test instrumentation, updated user manuals, etc.) has occurred prior to TRR and in support of GAT.

5.2 GAT Overview

The likelihood of a successful system GAT is typically indicated by two factors: demonstrated system maturity towards satisfying user needs, and proper documentation. System maturity is demonstrated through satisfactory passage of rigorous SIDT, while proper documentation demonstrates that the process leading up to GAT has been well thought out, and that the requisite level of collaboration between the Government and the System Integrator has been achieved. Proper documentation ensures that user needs are properly understood, test expectations have been adequately vetted, and that all system risks have been identified. These 2 factors, described in greater detail in the following paragraphs, must be verified at TRR prior to proceeding to GAT.

Unclassified

5.2.1 System Maturity

System maturity is verified through rigorous System Integrator testing, which is conducted through the execution of test cases by the SI, in a SI test environment. These test cases must be traceable to the systems requirements via an approved RTM. SI testing may be conducted under Government observation, as determined by the Lead Test Engineer. In order to demonstrate that system maturity is sufficient to proceed to GAT, the SI must present SI Test Results, and the Government must assess those test results to determine the SUT’s readiness for GAT. Typically, a system with unresolved defects assessed to be Severity 1 or 2 will not be deemed ready for

GAT.

5.2.2 Documentation

5.2.2.1 System Integrator Developmental Test Results

System maturity must be demonstrated through proper documentation. As mentioned before, in order for a system to gain entrance into GAT, it must have shown a reasonable likelihood of successfully passing GAT, based on its performance in SIDT. This performance shall be documented in the System Integrator’s report of SIDT results. Any known defects coming out of the SIDT must be documented, and evaluated with respect to their impact on the military utility of the proposed increment of system capability. If the impact of the known defects negates the original purpose for the release, the Government may elect to defer the GAT until such time that there is a reasonable expectation that the release will provide the desired utility. The below table defines the required datapoints that should be included in any report of SIDT Results:

Table 4- System Integrator Test Results Score Card

Test Case (List 1-n)

Number of Trials

Planned

Number of Passed Trials

Number of Failed Trials

Number of Incomplete

Trials

Test Case #1

Test Case #2

Test Case #n

Number Defects Found Number of Defects Open/Unresolved

Number of Defects Resolved

Defect Number

Defect Description Defect Severity

Defect #1

Defect #n

Unclassified

5.2.2.2 RTM

System functionality and performance should be measurable against some defined standards. These standards are expressed as requirements in an RTM, and they form the basis for the evaluation of the GAT results. The RTM provides the traceability of discrete requirements back to a requirements document, or to a source document (System Design Specification, ECP), and forward to an executable test case in the STD. Each requirement on the RTM should also have a defined verification method (Analysis, Demonstration, Inspection, or Test). All testable requirements should be traceable to the individual steps of an executable test case. The RTM shall also state the test results for each requirement for all SIDT and GAT. With the exception of the updates of test results as a product nears production readiness, the RTM should be in a finalized state before the SUT enters GAT.

5.2.2.3 Test Cases

The test cases provide the step-by-step, click-by-click instructions that the testers must execute during GAT. These test cases are based on operationally representative activities that a system operator would undertake in typical system use. The successful completion of a test case should demonstrate that a specific requirement, or some number of requirements, has been met. To ensure that successful completion of a test case can be taken to mean that certain requirements have been met, the discrete steps of a test case should be mapped to those requirements.

5.2.2.4 SUT Baseline

All TFITS projects shall be under configuration management in accordance with the TFITS Configuration Management Plan. A program’s release baseline is documented over the course of the SETR process, and shall be considered “locked down” from the TRR, through the completion of GAT. During GAT, no changes to the software, data or test environment shall be made without prior authorization from the Project Manager, Project Lead Systems Engineer and Lead Test Engineer.

5.2.3 Other Pre-GAT Considerations

A GAT frequently brings together personnel who do not have an established working relationship. A recall roster of all test personnel to include testers, functional advocate, SI and PjMO personnel is necessary to facilitate information flow, and personnel accountability.

Arrangements for all test resources must be finalized at TRR. Test platform configuration (operating system, browser type and version, JAVA settings, etc.) must be understood and the availability and source of all test assets must be confirmed. Any limitations of the test environment must be known and documented. When the test assets and environment are provided by a non-government source, assurances that the test environment will not be subject to SI tampering must be given. Any certificates required for authentication of test users to roles exercised in the execution of test cases, whether they be software certificates or Government-issued Common Access Cards, shall be tested prior to the beginning of the GAT.

Any differences training required for testers to understand the differences between the SUT and previous versions of the program shall be conducted prior to the beginning of GAT. Draft

Unclassified training material, updated training materials and difference training materials shall be delivered to the PjM at TRR. GAT is correctly understood to be a test event, and not a training event.

Deficiencies in operator expertise may be regarded as deficiencies in the training package delivered to the Government.

5.3 GAT Conduct

GAT is an inherently Government function. It is the culminating event to test and evaluate a system prior to its release to a production environment. All System Integrator participation in GAT activities are subject to Government oversight.

5.3.1 Roles and Responsibilities

An overview of the roles and responsibilities of each person involved in GAT is provided below in Table 4, and further elaborated upon below.

Table 5- Roles and Responsibilities

Role Responsibility

Project Manager (PjM) • Coordinate with the functional advocate for availability of users, SMEs, and testers

• Coordinate GAT schedule with the System Integrator and SE team

• Coordinate the test venue and assets with the System Integrator

• Voting member for scoring conference per Failure Definition/Scoring

Criteria (Appendix A)

• Approve any necessary changes to release baseline during GAT

Test Director: Project Office Test Engineer

• Provide GAT Oversight

• Prepare and submit Test Reports in accordance with Table 3

Test Conductor:

Project Office Test Engineer/Support

• Data Collection

• Capture test case execution

• TIR Quality Assurance

• Facilitate Hot wash

Project Systems Engineer

• Work with the PjM and T&E team to schedule GAT events

• Ensure that all necessary artifacts are available prior to TRR

• Ensure cross-functional support (CM, CS, Logistics, SI, Advocate) is available for GAT

• Coordinate with the PjM to approve any necessary changes to release baseline during GAT

• Develop any formal reports necessary for 3rd and 4th octet releases

System Integrator Liaison

• Data Collection:

• Capture TIR Information in SI defect tracker to include screenshots

• Provide test execution clarification as requested

Tester (may be Functional Advocate, System Users, Subject Matter Expert, et al)

• Test Execution

• Provide test case results to test facilitator

• Provide TIR information to SI defect tracker(Any issue will be documented in a TIR)

• Capture Screenshots for any Incident

Unclassified

5.3.1.1 Project Manager

The PjM will support GAT through coordination with the functional advocate with respect to the availability of users, SMEs and testers; with the SI for test resources and venue; and with the Functional Advocate, SE and SI for scheduling. The PjM, in addition to the Functional Advocate, will be a voting member for scoring of TIRs. Any changes made to the SUT release baseline during GAT shall be approved by the PjM.

5.3.1.2 Test Director

The Test Director will be the Government Test engineer, and will have primary responsibility for the conduct of GAT. The Test Director role may be delegated to the Project System Engineer, or their designated representative. The Test Director will prepare and submit test reporting in accordance with Table 3.

5.3.1.3 Test Conductor

The Test Conductor will be either the Government Test engineer or a project test support contractor. In their absence, the Project System Engineer (or their designated support contractor) may fill this role. The Test Conductor is charged primarily with the collection and preservation of test data in its entirety. Test Conductors shall ensure that the testers are properly instructed in their role, and overseeing the activities of the System Integrator in the test environment. They will ensure that the test data collected is complete and properly documented. At the end of each day, the Test Conductor will facilitate a hot wash to ensure that all test incidents were properly recorded and understood in their proper context, and that each TIR is scored for classification, chargeability, and severity in accordance with Appendix A. When a Failure Definition/Scoring Criteria conference is convened at the end of a test event, the Test Conductor will facilitate the discussion

5.3.1.4 Project Systems Engineer

The project Systems Engineer will consult the T&E team to schedule TRR and GAT, and ensure that all necessary test-related artifacts are made available to the test team for adequate review across the SETR process, in accordance with Table 2. The Systems Engineer shall advise the test team on all matters with respect to the system design and requirements and provide any necessary coordination with the PjM should the need for any changes to the release baseline arise. The Systems Engineer will ensure cross-functional support is available as required. The Systems Engineer will develop any formal reports needed for 3rd and 4th Octet GATs.

5.3.1.5 System Integrator (SI)

The SI shall serve in a supporting role during GAT. The SI will provide printed copies of the test cases to be executed. At the Test Conductor’s discretion, the SI may be asked to prepare TIR forms as test incidents occur. The System Integrator, having a high degree of insight into the function and design of the system, may be in the best position to analyze and provide a narrative description of system anomalies. They may also be better suited to articulate the issue with the system to the software engineers who will be tasked with the troubleshooting and resolution than the test conductor. All TIRs prepared by the SI shall be reviewed by the Test Conductor before they are accepted for review at the hot wash. The

Unclassified

Test Conductor may consult with the SI to gain clarification on the interpretation of test case syntax and system anomalies.

5.3.1.6 Tester

The testers are typically actual users of the system that come from the operational forces and supporting establishment. They are made available by coordination between the functional advocate and the Project Office, and are typically well-versed in the use of the SUT. The testers are charged with executing the test cases as printed by the SI. Testers should clearly indicate in writing, on the actual printed test cases, whether each step of a test case passes or fails, and whether the test case passes or fails overall. When the testers experience a system anomaly, they shall alert the Test Conductor, who may engage with the SI to interpret the anomaly, and record the system behavior on a TIR. Testers are encouraged to make use of screen captures of system anomalies, and ensure that those screen captures are associated to a particular TIR, using a distinct file naming convention.

There will also be a space on the test cases for the testers to make comments as they deem necessary to explain any differences between expected and actual results of executing test cases, or to capture their insights where appropriate.

5.3.2 GAT Execution

The Test Conductor will maintain accountability of each tester in the test facility, and all test articles and testing materials. The Test Conductor will manage the execution of the test cases in a logical order, ensuring that any dependencies between the activities of different testers with different roles are considered. The Test Conductor will manage the daily schedule through the course of GAT events.

The System Integrator may be called upon to assist in the writing of TIRs, and to explain any concerns with the syntax of the test cases, or interpretations of system anomalies. There may arise the occasion of a system defect manifesting that can be readily fixed through simple code manipulation. Any changes to the release baseline should be made in the SI development environment. Under no circumstances shall the SI make any changes to the release baseline in the production or test environment without the concurrence of the Project Manager, Project Lead System Engineer, and Test Director.

5.3.3 Data Collection

There are essentially two sources of data that are collected during GAT: the test cases with the testers’ pass/fail marks, comments and insights, and the TIRs with their associated screen captures. As the testers progress through the test cases, the Test Conductor will collect and review test data to ensure that each step is clearly marked as having passed or failed, and that the tester has signed off on each test case. In addition to executing all the test cases, ad hoc testing may be conducted allowing the testers to use the system in a manner consistent with their day-to-day mission. Ad hoc testing can present test challenges with respect to the scope of the test, but it often results in useful insights, and may reveal defects that otherwise may not have been discovered during scripted testing. Diligence must be taken in the writing of TIRs during ad hoc testing, since there is no test case to which a TIR may be associated. Thorough defect descriptions and use of screen captures are required to ensure adequate details are captured during

Unclassified ad hoc testing. All test results and TIRs will be collected, reviewed, and recorded in the Results Tracker by the Test Conductor. The Test Conductor will retain the TIRs and printed material through the end of GAT, and ensure that all TIRs and test cases are properly archived in an electronic format.

5.3.4 FD/SC Conferences

The FD/SC governs the conduct of post-GAT assessments of the root cause of all test incidents, as recorded by TIRs during GAT. The FD/SC provides a structured process for analyzing all test incidents by stakeholders at a FD/SC Conference. The stakeholders include the Functional Advocate and the Project Management office. The TFITS FD/SC is found in Appendix A of this ST&ES. The purpose of the scoring conference is to assess each TIR for classification, chargeability, and severity, with a view towards determining root cause analysis and issue resolution. At the end of each test day, the Test Conductor will ensure that sufficient time is reserved for scoring of TIRS. The testers will often represent the Functional Advocate. The Test Conductor will facilitate the discussion, ensuring that all parties are represented, but also that the process is not encumbered by excess analysis, speculation, or filibuster. The SI may address the stakeholders as requested for the purposes of providing clarity and technical insight.

5.4 GAT Closeout and Reporting

GAT Closeout consists of ensuring that all system anomalies are recorded in a TIR, and the proper disposition of all TIRs. At the closure of GAT, a report shall be delivered documenting the number of passed and failed test cases, trials, and ECPs, as well as the final status of all TIRs generated over the course of the GAT.

5.4.1 TIRs

In order to exit GAT, all planned test cases must be executed with results recorded. All system anomalies shall be documented with a TIR. TIRs recorded during the GAT must be fully scored at a hot wash or a FD/SC conference. The SI shall report the results of root cause analysis for all TIRs before GAT closeout. All TIRs must be demonstrated as having been resolved (closed) or they may be the subject of a newly generated ECP. In the event that root cause analysis does not definitively lead to the closure of a TIR, GAT will either remain open until resolution can be achieved, or the ECP associated with the failure can be removed from that increment of capability and proposed for a future GAT event in accordance with the TFITS SEP. Such a removal would represent a change in the release baseline, and may necessitate a full regression test of those ECPs that had already passed GAT before GAT can be concluded; this regression is necessary to ensure that the removal of the associated code did not adversely affect code that had already passed.

5.4.2 ECPs

ECPs generated as a result of GAT closeout may be related to a new capability or enhancement that was not within the scope of the GAT (Class I), or they may be related to a bug fix (Class II).

5.4.3 GAT Reporting

As seen in Table 2, all GATs will require the completion of the GAT Results Tracker (Figure 2) and the update of the Government approved repository on the TFITS SharePoint Portal. The Results Tracker metrics are:

Unclassified

• Test Case Execution: Number of test cases planned for execution (Passed, Failed, Not Yet Tested)

• Test Trial Execution: Number of discrete attempts of all test cases, in aggregate (Passed, Failed, Not Yet Tested)

• ECPs: Numbers of ECPs tested (Met, Met with Exception, Not Met, Not Tested)

• ECP Requirements: Number of discrete requirements associated to ECPs, in aggregate (Passed, Failed, Not Tested)

• Test Incident Reports: Rollup of numbers of TIRs by severity, per each day of GAT

• Test Incident Report Status: Rollup of disposition of all TIRs over the course of GAT (ECP Generated, Closed, Open)

Figure 2- GAT Results Tracker Metrics

At the conclusion of GAT, the Test Director or the Test Conductor will upload the results of the GAT in to an online Government approved repository. The GAT results will be reported for all GAT events. A formal narrative report format is used to document the results of all 1st and 2nd octet GATs, and as required by the APdM-E.

5.5 PVT

Production Verification Testing will be conducted for all releases, to ensure that system function and performance in the production environment is as expected in support of the PIR. In most cases, the PVT will be conducted by the System Integrators, but may also be conducted by the functional advocate or project office personnel. System Integrators will develop a checklist of activities that could be completed after deployment to determine that there have been no adverse effects to the production environment and that new functionality is operating as designed; these checklists may be tailored for a specific release. Where data manipulation is required to validate functionality, the functional advocate will perform the validation testing as part of normal business operations. The PjMO T&E may observe the PVT on a case-by-case basis.

Unclassified

6 Abbreviations and Acronyms Abbreviation/Acronym Definition APdM-E Assistant Product Manager – Engineering CDR Critical Design Review CM Configuration Management DISA Defense Information Systems Agency DoD Department of Defense CS Cyber Security ECP Engineering Change Proposal FD/SC Failure Definition and Scoring Criteria GAT Government Acceptance Test(ing) IAVA Information Assurance & Vulnerability Assessment IS&I Information Systems & Infrastructure Log Logistics MCOTEA Marine Corps Operational Test and Evaluation Activity MCSC Marine Corps Systems Command PDR Preliminary Design Review PII Personally Identifiable Information PIR Post-Implementation Review PjM Project Manager PjMO Project Management Office PMM Program Management Manager PRR Production Readiness Review PVT Post-Implementation Verification Test/Testing RTM Requirements Traceability Matrix SC Scoring Conference SE Systems Engineer/Engineering SECNAV Secretary of the Navy SECNAVINST Secretary of the Navy Instruction SETR Systems Engineering Technical Review SFR System Functional Review SI System Integrator SIDT System Integrator Developmental Test(ing) SIDTRR Systems Integration Developmental Testing Readiness Review SME Subject Matter Experts SRR System Requirements Review ST&ES Systems Test & Evaluation Strategy STD Software Test Description STIG System Technical Implementation Guide STP Software Test Plan SUT System Under Test T&E Test and Evaluation TFITS Total Force Information Technology Systems TIR Test Incident Report TRR Test Readiness Review

A-1 Unclassified

Appendix A: Failure Definition and Scoring Criteria (FD/SC)

1. General and Administrative

a. General

(1) Application. The Failure Definition and Scoring Criteria (FD/SC) supports the test process in two key areas:

• Defining the organization of the Scoring Conference (SC) and its procedures for Test Incident Report (TIR) scoring

• Assigning criteria that categorize TIRs.

(2) Purpose. The SC officially reviews and classifies test incidents that occur during test events.

The SC determines what system elements have primary responsibility (causality) for each incidents. SC results shall support the evaluation of system Availability, Reliability, and Maintainability (RAM). The Project Management Office (PjMO) may also use SC results to determine the materiel developer’s contractual compliance.

b. Scoring Conference Organization

(1) Chairman. The agency responsible for the test event will appoint a chairman for all SCs for that event.

(2) Membership. Each member shall have one vote. Additionally, each voting member appointed by command or activity will be a uniformed service member or Government civilian. A majority vote constitutes a final scoring decision. If a voting member is unable to attend, the command or activity may designate a replacement. Table 1 lists SC membership.

Table A-1 - Scoring Conference Membership

Command/Activity Role Project Advocate [e.g.: Capabilities Development Directorate (CDD), Command and Control Integration Division (C2ID), POR]

Advocate

PMM-110/Project Manager Total Force Information Technology Systems (PjM TFITS)

Materiel Developer; Chairman, SC for Government Tests

(3) Contractor Participation. Contractor participation in SCs will be limited. At the discretion of the SC chairman, system developmental contractors may help to resolve particular technical questions concerning select TIRs on an “as needed” and “on call” basis. Only the SC voting members and the facilitator will participate in open discussions; all other personnel will speak only to respond to specific questions.

c. Scoring Conference Procedures

(1) SC Schedule. At least one SC will be held per test event. The SCs will be scheduled by the chair and identified in the individual Test Plan. For GAT events, SCs will occur daily during the testing period.

A-2 Unclassified

(2) Recording of Test Incident Data. Test Incidents are system anomalies that occur during testing and shall be recorded by use of TIRs. TIRs can be generated due to any observed system anomaly during a test event, including non-operational times. All Test Incidents pertaining to either a potential system failure or maintenance action are reported on TIRs

(3) Conduct of Scoring Conference

(a) Conference members will score all incidents according to the instructions and definitions of this FD/SC Charter. Members will examine the unique circumstances of each Test Incident and score each one in terms of Classification and Chargeability. In addition, when a TIR highlights system safety concerns, members will perform a Hazard Risk Assessment as well.

(b) No incident will be left unscored at any SC. The incident can still be scored by the SC members; however, depending on the information available, chargeability may be deferred until such time as the information is available to make the incident chargeable to an area.

Each TIR shall be annotated regarding its chargeability or the reason that the chargeability was not provided as well as the suspense date for the information and its chargeability.

Previously scored incidents may be re-examined at a future conference to consider new information if a majority of the conference’s voting members agree to do so.

(4) Scoring Conference Results. Scoring Conference results will be provided as part of the Software Test Report.

2. Classification, Chargeability, and Hazard Severity Guidelines. In scoring test incidents, conference members will follow the Test Incident Scoring Decision Chart (figure A-1) (ref. a).

A-3

Figure A-1 - Test Incident Classification and Chargeability Decision Chart

a. Classification. Classification places TIRs into categories based on failure definitions contained in this FD/SC Charter, as seen in table A-2 (ref. a).

Table A-2 - Classification Categories Classification Circumstances

Failure An incident that results from the system’s inability to perform in accordance with the design specification or failure to provide expected results.

No-Test

Pre-test Checkout. Any incident observed during the designated burn-in, pretest inspection, or other pretest activity is classified as No Test. The Test Plan must specify the length of the burn-in period (the number of miles, rounds, or hours) to permit a determination of when the pretest period has ended.

Equipment Modification. Maintenance done to install a hardware kit or to incorporate a redesigned component is classified as No Test.

However, if the replaced part was not functioning when it was being replaced, that malfunction will be scored on its own merits. A subsequent malfunction of the installed part will also be scored on its own merits.

A-4 Unclassified

Classification Circumstances

Test-Peculiar Incident. An incident caused by someone not acting as a test player (crew member or maintainer), or by equipment not part of the system being tested, is classified as No Test. An example of this is an engineering evaluation and the maintenance done in furtherance of that evaluation. This classification also includes malfunctions to or caused by test instrumentation. However, an incident caused by test-peculiar equipment (equipment used in the test in lieu of the equipment to be fielded) will be scored under its own merits; if the test planners have introduced equipment for the purposes of the test, they have judged it to be an adequate substitute for the equipment to be fielded. Hence, its failures are to be regarded as representative of the failures of the equipment to be fielded.

Daily Checks and Services. These are checks and services, performed by the operator (or by the crew, if applicable) using only repair parts and On-Equipment Material (OEM) in accordance with the equipment publication before, during, or after the operation of the equipment. Checks and services that meet these conditions are classified as No Test.

Test-Directed Abuse. An incident in which the tester directs the deliberate abuse of the system (e.g., a test to over-stress the performance limits of the system), whether called for by the test plan or not. However, damage to the system willfully caused by the operator or maintainer and not directed by the tester will be scored under its own merits.

Non-RAM Oriented. This is a catch-all term to capture those incidents in which a TIR has been prepared, but which have no bearing on the RAM assessment. Examples are suggested improvements; reports on inadequate test procedures; reports on unacceptable replacement parts, provided they were discovered before or during installation; reports on the equipment’s consistent inability to meet performance specifications even though no actual malfunction has occurred; suggested human factors improvements; and recommended changes to the system support package.

b. Chargeability. Chargeability places TIRs into categories according to what caused the incident, as seen in table A-3. The category represents the element of the system primarily responsible for the incident.

Table A-3 - Chargeability Codes

A-5 Unclassified

Category Code Assigned if incident is attributable to

Hardware CH

Characteristics of the hardware. Malfunction or inaccuracy of a hardware component. This includes cables, power supplies.

Computers, LAN/WAN architecture, batteries, and other ancillary items.

Computer Software CS Characteristics of the computer software, firmware, Built-in-Test (BIT), diagnostics, or training tools included in the design.

Includes hosted applications)

Crew-caused CC Operating personnel error or improper crew actions that deviate from technical manuals or training.

Maintenance CM Improper maintenance actions that deviate from technical manuals or training.

Manuals/Publication CT Misleading, incorrect, or missing information in the technical manuals.

Accident CA Accidents not caused by system design, inadequate training, operator/maintenance personnel error, or erroneous manuals.

Training CR Misleading, incorrect, or missing information provided in the operator or maintainer training programs.

(1) Computer Software Alphanumeric Code.