Exhibit3f_TFITS_STES.pdf

FOR OFFICIAL USE ONLY

SYSTEM TEST & EVALUATION STRATEGY (ST&ES)

TOTAL FORCE INFORMATION TECHNOLOGY SYSTEMS

(TFITS)

Version 1.0

Prepared by:

MARINE CORPS SYSTEMS COMMAND

Total Force Information Technology Systems (TFITS)

Information Systems & Infrastructure (IS&I) Quantico, Virginia 22134

TFITS ST&ES September 23, 2013 ii

SIGNATURE PAGE

TECHNICAL REVIEW:

Mr. Andrew Mittura Assistant Product Manager – Engineer (APdM-E)

TFITS, PMM-110 IS&I

Marine Corps Systems Command

APPROVED BY:

Mr. William Sandlin Product Manager (PdM)

TFITS, PMM-110 IS&I

iii

RECORD OF CHANGE HISTORY

Version # Date Figure, Table, or Paragraph

*A, M, D Title of Brief Description

Change Request #

*A-Added, M-Modified, D-Deleted

Questions concerning this Test and Evaluation Strategy should be referred to:

• Andrew Mittura, TFITS APdM-E, 703-784-4740, Andrew.Mittura@usmc.mil mailto:Andrew.Mittura@usmc.mil iv

TABLE OF CONTENTS

1 INTRODUCTION

1.1 Purpose

1.2 Scope

2 TEST PROGRAM MANAGEMENT AND SCHEDULE

2.1 T&E Organization

2.2 Test Personnel

2.3 Schedule

3 TEST AND EVALUATION STRATEGY

3.1 T&E Strategy

3.2 Level of Testing Required

3.3 Government Acceptance Test

3.4 Test Limitations

3.5 Test Conduct

3.6 Scoring Conference

Table of Figures

Figure 2-1: High Level Schedule Figure 3-1: Defect Monitoring Chart Template Figure 3-2: GAT Process

List of Tables

Table 1-1: Government Acceptance Test and Octet Levels Table 3-1: Testing at Each Version Octet

1 Introduction

1.1 Purpose

The purpose of this Systems Test & Evaluation Strategy (ST&ES) is to define the test & evaluation (T&E) approach for the Total Force Information Technology Systems (TFITS) Product Office projects. This ST&ES will provide overarching guidance common to all projects within the TFITS portfolio. This Test and Evaluation Strategy applies to all TFITS system versions being developed that do not fall under Marine Corps Operational Test and Evaluation Assessment (MCOTEA) oversight.

1.2 Scope

TFITS employs a four-octet versioning scheme as defined in the TFITS Systems Engineering Plan (SEP). That schema defines a first octet change as a capability change, the second octet change as a major functional change, the third octet as a minor functional change and the fourth octet as a maintenance change or patch/bug fix. The scope of testing has been tailored as well with the intention of taking advantage of the verification testing done by the systems integrator during development prior to Test Readiness Review (TRR). This approach allows TFITS Engineering to conduct a validation test and regression test of the major enhancements of the version. The breadth of the Government Acceptance Test (GAT) for each octet level is defined in Table 1-1 below.

Table 1-1: Government Acceptance Test and Octet Levels Change level 1st Octet 2nd Octet 3rd Octet 4th Octet Government test

IDT&E GAT GAT No Government test. User validation through help desk

Developer Test Government monitoring of vendor SIT and

SAT

Government monitoring of vendor SIT and

SAT

Government monitoring of SIT and SAT results

Government monitoring of SIT and SAT results

Documentation TEMP/GAT test plan with full regression testing

GAT test plan with validation of enhancement and regression testing

GAT test plan with validation of enhancement and regression testing

Vendor test scripts

Information Assurance

Pen Test, IV&V, Scans, vendor code reviews

Scans, vendor code reviews

Scans, vendor code reviews

Scans, vendor code reviews

2 Test Program Management and Schedule

2.1 T&E Organization

The T&E planning for a project shall consist of the Project Manager, System Engineer, Test Engineer, Information Assurance, the system integrator, and the advocate. Test execution and oversight are the responsibility of the project systems engineer.

2.2 Test Personnel

GAT Test personnel will include the following:

• Test Director will be the Government engineer.

• Test Conductor may be a Government engineer, Project support personnel, or a SME and will be responsible for the day to day conduct of the test. The test conductor will liaison with the vendor during the test event. The test conductor will track test progress against scope and provide daily hotwash reports.

• Recorder. The test recorder(s) may be government or support personnel and will be responsible for recording all incidents that occurred during the test event.

• Data Analyst. The data analyst will work with the systems engineer to determine the requirements satisfaction in support of the Software Version Review/Production Readiness Review (SVR/PRR). These results will be contained in the Software Test Report.

• Testers. Typical users of the system.

2.3 Schedule

Planning for the GAT should begin in the requirements phase with the GAT test plan being an artifact for the Requirements review. The test cases should be provided as an artifact for the critical design review and the test scripts for the Test Readiness Review (TRR). The GAT planning will be developed by the PdMO project engineer and reviewed by the APdM-E.

Figure 2-1: High Level Schedule

3 Test and Evaluation Strategy

3.1 T&E Strategy

The TFITS test strategy is based on a verification and validation process. It calls for using the verification performed by the developer and a validation performed by the government to characterize the performance of the system in its environment. The test strategy also relies on the constant monitoring of the design and development of the system through requirements traceability and assessment.

The System Integrator will develop and maintain a comprehensive software test plan in accordance with IEEE 12207 the Standard for Information Technology – Software Life Cycle Processes and is expected to verify all requirements have been satisfied. The Project Office will review and approve the test plan and all applicable updates. The internal tests the System Integrator will conduct shall include Unit Testing, Integration Testing, and Regression Testing.

These tests are monitored by the project office test lead. Defects found during testing are to be assessed for severity using the criteria below:

• Severity 1: (Showstopper) Prevents the accomplishment of an operational or mission essential capability. Jeopardizes safety, security, or other requirement designated “critical”.

• Severity 2: (High) Adversely affects the accomplishment of an operational or mission essential capability and no work-around solution is known. Adversely affects the technical, cost, or schedule risks of the project or the life cycle support of the system.

• Severity 3: (Medium) Adversely affects the accomplishment of an operational or mission essential capability but a work-around solution is known. Adversely affects the technical, cost, or schedule risks of the project or the life cycle support of the system.

• Severity 4: (Low) Results in user/operator inconvenience or annoyance but does not affect a required operational or mission essential capability. Results in inconvenience or annoyance for development or support personnel, but does not prevent the accomplishment of those responsibilities.

• Severity 5: (Other) So minor in its affect that corrective action would cost more than it would be worth. An enhancement not linked to a requirement that has been validated.

A system is not ready for GAT if there are open severity 1s and 2s. The project office test lead is also expected to provide an assessment of the completeness of developer testing and the risk of going to GAT with open defects. This assessment is to be presented at the TRR.

3.2 Level of Testing Required

The level of testing will depend on the scope of the release. The octet level is defined in the TFITS Configuration Management Plan and System Engineering Plan as determined by the Engineering Review Board (ERB). The following table depicts the anticipated testing for each release octet.

Table 3-1: Testing at Each Version Octet

Level of Change 1st Octet X.x.x.x

2nd Octet x.X.x.x

3rd Octet x.x.X.x

4th Octet x.x.x.X

Testing Required • Validation

• Verification

• Regression

• Monitor SI

Testing

• Regression

• Validation

• Verification

• Monitor SI

Testing

• Regression

• Verification

• Monitor SI

Testing

• Assessment of SI test report

3.2.1 Defect Monitoring

During the test event, the number of defects or incidents shall be monitored at a regular, pre-determined interval (daily, weekly, etc.) as depicted in the following chart. The same interval shall be used to track the status of defects after the end of GAT.

Figure 3-1: Defect Monitoring Chart Template

3.3 Government Acceptance Test

3.3.1 Purpose

The GAT serves as the culmination of testing. During the GAT, the System Under Test (SUT) shall demonstrate mission accomplishment or service fulfillment in an environment representative of the operational environments (for programs in TFITS, “mission” equates to service fulfillment). After GAT, any performance limitations are known, bounded, and predictable and the MCSC-designated technical authority shall have sufficient information to make a deployment risk assessment.

3.3.2 Basis of Test Design

The GAT designer begins with the premise that all lower level unit and integration testing complete, and that the system is ready to be exercised at representative tempo and loads. The test operators shall concentrate on performing their missions or using the services rather than fault isolation or debugging.

If the operators cannot demonstrate effective operation, then the system will fail GAT. This will result in deficiency resolution (through software revision or possibly operator workaround definition) and subsequent retesting.

The Operational View (OV) -5a, Operational Activity Decomposition Tree, and OV-5b, Operational Activity Model form the basis for service-level demonstrations. These identify relationships and hierarchy between roles and tasks and information exchanges between systems.

The GAT design uses these intended relationships to evaluate realistic activity of the service provider and its clients. In the same way that large acquisition programs publish a Design Reference Mission (DRM), TFITS testers shall design a narrative flow of realistic activity that demonstrates SUT capability in the face of expected threats, challenges, and service demands.

For systems that lack Operational View architecture products or CONOPs, the GAT team will collaborate with developers (and users if already deployed) to design the intended activity hierarchy.

Demonstration of service fulfillment corresponds to meeting higher-level system requirements, thereby completing the progression from derived lower level requirements in unit and integration testing.

3.3.3 GAT Structure

To the extent practical, GAT Teams will use a modular design that allows independent execution of GAT segments. This will allow testing of well-defined corrections or improvements, which, when combined with a comprehensive regression test, provide an efficient means of validating configuration-controlled change. Each segment will be in the form of a real-time scenario or vignette, which demonstrates an end-to-end activity in support of a service. Scenarios and vignettes shall be consistent with the activity de-composition in the OV-5a.

The regression test will exercise the entire functional baseline, but at an abbreviated level of detail consistent with the change. It verifies that controlled changes to the baseline have not negatively impacted previously demonstrated capability.

3.3.4 Requirements Continuity

GAT validates fulfillment of requirements in two ways: analytically, GAT reviews the System Integrator’s Requirements Traceability Matrix (RTM). This ensures that during the engineering process (deriving secondary and tertiary requirements, refining and then allocating them to software or hardware components at increasing levels of integration—from unit to component to system) the requirements are tracked continuously, accounted for, and matched to test results. In other words, the GAT team validates the entire system engineering and test “V” progression up to the point that it is delivered to the government. Demonstratively, GAT addresses high level requirements from the perspective of the user. This proves that services are fulfilled as specified.

Figure 3-2: GAT Process

3.3.5 Measured Performance Baseline

Prior to testing, a published test design describes the expected outcomes for all test cases, either as predicted cause-to-effect sequences for logic processes, or as acceptable thresholds for quantitative or stochastic processes. All prediction and measurement thresholds are traceable to requirements. GAT is executed in a configuration-controlled environment, with the test subject being a re-constructible configuration item. All test operations, from set-up to test execution and data collection, shall be accomplished in accordance with detailed procedures. Deviations from approved procedures shall be recorded, and incorporated in the next document update. GAT design, being standardized and configuration-controlled, allows quantitative comparisons of test results and performance against a common frame of reference. GAT will use these comparisons as a basis for OT readiness recommendations and retain quantitative results for long term trend analysis.

All test documents, including procedures, scenarios, and simulation scripts, are associated with specific GAT events and their measured performance baseline. These are also held under configuration control.

3.4 Test Limitations

The GAT environment for most SUTs within the TFITS portfolio will not be the same environment as the SUT’s operational employment. Performing GAT on an NMCI or tactical MCEN network environment may be impractical or impossible. Actual users of the SUT may not be available for GAT. Deviations from an operationally representative environment may include, but are not limited to:

High Level Rqmts Mission/Service Context Warfighter/Client Focus KPPs

Complete DT OT Risk Reduction

USMC GAT

• Network performance (bandwidth, latency, physical infrastructure)

• Operator interfaces, user terminals, internet browser versions

• Operator experience level, number of concurrent users

The extent to which these deviations may affect the ability to generate realistic and predictive GAT results shall be documented as test limitations. Based on these limitations, additional testing may be required once the system is deployed.

3.5 Test Conduct

During testing, all incidents are documented. These incidents are deviations from the expectation defined in the test script or anomalies determined by the testers. The incidents are then assessed at the end of each test day to ensure completeness of documentation and validity.

At the conclusion of the GAT event, a scoring conference is held with the test team to score each incident.

3.6 Scoring Conference

The scoring conference is held at the end of the GAT. It is chaired by the Test Director and the Project Manager. All competencies should be in attendance as well as the functional manager.

Each incident is first evaluated to determine its validity based on repeatability. Those judged valid are assessed to determine if it is related to the test case. If not, the defect is finding. All defects are graded for severity using the definitions above.