Attachment_09_-_Failure_Definition_Scoring_Criteria_(FD_SC).docx

Attachment 09 Failure Definition/Scoring Criteria (FD/SC) 3 Sep 2014

Amphibious Combat Vehicle (ACV) Phase 1 Increment 1

Failure Definition and Scoring Criteria Charter

September 3, 2014

Director Marine Corps Operational Test and Evaluation Activity 2032 Barnett Avenue Quantico, Virginia 22134-5014

Amphibious Combat Vehicle Phase 1 Increment 1 (ACV 1.1)

Charter

This Failure Definition/Scoring Criteria Charter is hereby approved as the governing document for evaluation of the Amphibious Combat Vehicle Phase 1 Increment 1 (ACV 1.1). This charter is focused on the ACV 1.1 capability and hereafter in this document the term ACV will refer to ACV 1.1. All personnel participating in or associated with the evaluation shall be made aware of the contents of this plan and shall abide by its requirements and procedures.

Approved by

Mr. John Garner Program Manager, Advanced Amphibious Assault Program Executive Office – Land Systems

James McArthur LtCol USMC Director Fires and Maneuver Integration Division Headquarters Marine Corps, Combat Development and Integration

Richard J. Cushing Maj, USMC Expeditionary Division Marine Corps Operational Test and Evaluation Activity

Amphibious COMBAT VEHICLE

Charter

1. General and Administrative

a. General

(1) Application.

The Failure Definition and Scoring Criteria (FD/SC) Charter supports the test process in three key areas: organization of the Scoring Conference (SC) and its procedures for Test Incident Report (TIR) scoring; definition of system Mission Essential Functions (mef); and assignment of criteria that categorize TIRs that occur during Developmental Testing (DT) and Operational Testing (OT) based upon the Concept of Employment (COE) for the system. The FD/SC defines the required functionality and allowable levels of degradation (in other words, what constitutes a reliability failure) and establishes a framework for classifying and charging test incidents. The FD/SC is the basis for the Reliability, Availability and Maintainability (RAM) scoring process.

(2) Purpose.

The ACV SC officially reviews, classifies and assesses preliminary chargeability of TIRs that occur during developmental reliability and operational test events. The SC determines initial, or top-level, system elements having primary responsibility (causality) for these incidents. SC results used in the evaluation of system RAM may also be used for determining contractual compliance of the contractors by the program management team.

b. Scoring Conference Organization

(1) Chairman.

The chairman of the Operational Test and Evaluation (OT&E) scoring conference is the Marine Corps Operational Test and Evaluation Activity (MCOTEA) Operational Test Project Officer (OTPO). The chairman of the DT scoring conference is the government Program Manager Advanced Amphibious Assault (PM AAA) representative. Every effort should be made to keep the same chairmen for both DT and OT scoring conferences in order to maintain consistency. The chairman will schedule, announce, and host conferences and furnish review copies of the TIRs to all voting members. Review copies of TIRs shall be distributed as early as possible, but at least 24-hours before the conference to allow for adequate review and preparation. In most cases TIRs will be distributed one week prior to the SC. Incidents occurring between the time of TIR distribution and the convening of the SC will typically be reviewed in a subsequent SC.

(2) Membership.

The commands and activities listed below will each designate one member, via letter addressed to both the DT and OT chairmen, to fulfill the indicated role at the SCs. Each member will have one vote. Additionally, each voting member appointed by a command or activity will be either a uniformed service member or a government civilian. A majority vote constitutes a final scoring decision. All voting members are to be notified of the SC at least 10 working days before it begins. If a voting member is unable to attend, the command or activity may designate a replacement, in writing. Conference members will review the TIRs before arriving at the SC and shall independently assign their own preliminary scores for each report. Table 1 lists SC membership.

Table 1. Scoring Conference Voting Members

Chairman, SC for Operational Tests

(3) System Developmental Contractor Participation.

Title 10, United States Code, Section 2399, paragraph (d) prohibits system developmental contractors from being involved in any way in the evaluation activities for operational testing. System developmental contractors are the members of industry delivering the vehicles. The attendance of system developmental contractor personnel at OT scoring conferences is prohibited unless authorized by the Director, MCOTEA or his designated representative, the chairman of the Scoring Conference. At the discretion of the SC chairman for operational tests, system developmental contractors may be brought in to help resolve particular technical questions concerning select TIRs on an “as needed” basis. This prohibition does not extend to developmental test events, nor does it preclude contractors who are not delivering vehicles. Under no circumstances will support contractors be voting members for OT scoring conferences, unless designated in writing by their respective command.

c. Scoring Conference Procedures

(1) SC Schedule.

At least one SC will be held at the conclusion of each reliability growth period and following the conclusion of each test event where RAM assessments are being made. The SCs will be scheduled by the chair and identified in the individual test plan for either OT or DT events. Multiple SCs may be held to complete the evaluation of test data if necessary.

(2) Recording of Test Incident Data.

Test incidents are system anomalies that occur during testing and shall be recorded by use of TIRs. TIRs can be generated due to any observed system anomaly during a test event, including non-operational incidents. The activity performing the test will ensure that all test incidents pertaining to either a potential system failure or maintenance action are reported on TIRs. The activity performing the test is responsible for collecting all relevant data detailed by the materiel developers and for ensuring the integrity and completeness of the data. The test activity will enter a preliminary score (Classification and Chargeability) for each TIR generated. The test activity will furnish the TIRs in support of all RAM assessments to the SC chairman for distribution per paragraph 1.b. (1) above.

(3) Conduct of the Scoring Conference Conference members will score all RAM related incidents according to the instructions and definitions of this FD/SC Charter. Members will examine the unique circumstances of each test incident and score each one in terms of Classification and Chargeability. In addition, incidents that occur as a result of a failure, and that pose a credible risk to personal injury related to either a Category I (catastrophic) or II (critical) hazard as defined in Table 5 may be classified as an OMF after thorough consideration of the operational impact.

The conference members will decide the Classification, Chargeability, and Hazard Risk for each incident by a simple majority vote. The chairman will record all scoring results, including minority (dissenting) votes. Members may decide not to vote on any TIR if that is their independent decision. Any SC member with a dissenting vote on a TIR score may submit a written minority or dissenting opinion no later than 5 working days after an SC session. In the event of a tie, the chairman from the responsible test activity will cast the deciding vote.

Test incidents may be left unscored until additional information becomes available to support a scoring decision. Previously scored incidents may be re-examined at a future conference to consider new (additional) information if a majority of the conference’s voting members agree to do so.

In accordance with the Director, Operational Test and Evaluation (DOT&E) Independent Operational Test and Evaluation (OT&E) Suitability Assessments Memorandum for the Record dated 5 Oct 2011; the operational test authorities shall evaluate operational suitability based solely on their own independent scoring of reliability failures treating externally-generated failure definitions and scoring criteria as guidance only.

(4) Scoring Conference Results.

Before the conference ends, the chairman will prepare a Memorandum for the Record documenting the results, which each voting member will review and sign. These results become a matter of record and will be maintained by the conference chairman. The chairman will publish meeting minutes and scoring results of each conference for all voting members to review. The results will be used to calculate RAM metrics needed to produce the test reports.

(5) FD/SC Charter Revisions.

The FD/SC Charter has been developed with specific reference to the approved capabilities documents. Revisions to the charter are possible, but shall be based on changing perceptions of developmental or operational test needs and not system performance during a specific test scenario or event. The RAM IPT reviews all proposed changes to the FD/SC Charter and forwards all proposed changes to the participating commands or activities for review, signature, and final approval. Neither the chairman nor other members can change the approved FD/SC Charter without approval by all members, which they will indicate by signing the revised document.

(6) Limitations.

Although it is written in specific terms, the ACV FD/SC Charter cannot address every possible situation or event that could arise during testing. Test incidents outside the definitions and instructions of this charter shall be scored consistent with the charter’s intent and by majority agreement of the appointed SC members. This charter and its contents regarding TIRs, SCs, etc., applies to RAM issues and not system performance or effectiveness.

2. Description and Definitions

a. General Description.

The ACV capability is designed to fill gaps identified in the Expeditionary Armored Forces Initial Capabilities Document (EAF ICD), dated 11 Dec 07. The ACV fulfills the medium combat capability category requirement within the ICD, previously known as the Marine Personnel Carrier (MPC), to support expeditionary protected mobility requirements for ground combat element forces in the execution of future missions across the range of military operations (ROMO). An armored personnel carrier optimized for land operations with a water mobility capability, the ACV’s primary purpose is to provide a reinforced Marine infantry fire team with inland mobility. The ACV’s weapon systems are intended to support dismounted infantry by fire and contribute to vehicle protection. The ACV’s communications and information systems will provide the requisite situational awareness (SA), command and control (C2) and navigation capabilities for the ACV crew and embarked Marine infantry to operate on land. ACVs must operate under all weather, terrain and climatic conditions per the performance specification to enable mounted forces to complete their missions.

Possessing a water mobility capability, the ACV provides the ground combat element of a Marine Air-Ground Task Force with the means to exploit inland water obstacles as they emerge and maintain operational tempo by swimming shore-to-shore in some littorals and striking where the enemy is vulnerable. Additionally, the vehicle will be suited for riverine operations.

b. Components.

The ACV is composed of several main groups. The main groups are the Hull, Power Pack, Drive Train, Communications Suite, and Weapons Station.

c. Contractor-Furnished Equipment/Government-Furnished Equipment.

During the SC, voting members will determine the chargeability of TIRs as they relate to Contractor-Furnished Equipment (CFE) and Government-Furnished Equipment (GFE). A comprehensive list of CFE/GFE for the ACV will be provided to facilitate the SC’s ability to determine chargeability.

d. Mission.

The ACV is an armored personnel carrier. Its primary function is to provide protected mobility to the infantry. The ACV is employed by an Assault Amphibian (AA) battalion, the Marine Division’s general support amphibious lift organization, to accomplish a mission. The mission of ACV-equipped units is to reinforce the surface assault element of the landing force (LF) during amphibious operations and follow-on operations ashore. The ACV is employed in the conduct of mechanized operations during operations ashore. The typical mission is described in the operational Mode Summary/Mission Profile (OMS/MP).

The ACV will be administratively offloaded from Maritime Prepositioned Ships and provide the embarked Marine infantry with sufficient water mobility to conduct shore-to-shore operations, rapidly transition to land, and enable maneuver within the landing force. The ACV is designed to support the full ROMO optimized for inland operations. The tactical employment of the ACV system permits the embarked Marine infantry to dismount and close to the objective under the supporting fires of the ACV’s organic weapons.

e. Mission Essential Functions.

Mission Essential Functions (mef) are the minimum functional capabilities that the system must possess to be considered mission capable. The operators of the system should be able to readily identify the loss of a mef during normal operations. It is understood that the permissible degradation for each of the mef’s may be well below the stated threshold value(s) in the CDD. Levels of degradation (further described in Appendix A) still allow the ACV to perform its assigned mission and are supported by the Classification/ Chargeability Flow Chart. The mef’s for the ACV are as follows and are expected during typical operations over the OMS/MP, within the full range of environmental conditions including day/night:

· Move Land. The ACV must have the capability to maneuver in formation during landward operations.

Water. The ACV must have the capability to maneuver in formation during waterborne operations. The ACV must be able to transit a surf zone in both landward and seaward directions.

· Communicate The ACV must allow external voice or data communication with higher headquarters. The ACV must allow external voice communication with adjacent units. The ACV must maintain intercom communications between the Vehicle Commander (VC) and Driver.

· Protect The ACV must have the capability to protect the crew, occupants, and certain platform-level capabilities from threat attacks.

· Carry The ACV must have the capability to transport Marines and disembark them in combat ready condition near the objective.

· Shoot The ACV must have the capability to engage targets with direct fire.

f. Failure Occurrence.

For the purpose of ACV RAM testing, the representative mission profile described in the ACV OMS/MP (Figure 1) will be used. The loss of any mef, occurring in any of the scenarios, will be scored an OMF.

Figure 1: ACV Mission Profile The ACV Mission Profile (MP) represents the initial forcible entry phases of a major combat operation, consisting of four scenarios: Littoral Penetration (Scenario 1), Expand Lodgment (Scenario 2), Defend (Scenario 3) and Secure a line of Communication (LOC) (Scenario 4). This MP has four separate missions over four days with an opportunity to repair mission critical failures during resets between scenarios. The four scenarios, followed by reset periods, are consistent with how the Marine Corps operates and provides four distinct profiles. The ACV must have the reliability to complete individual scenarios without mission failure. Because each scenario is followed by a reset period during which the unit prepares for future missions, each scenario will be tested individually rather than as segments of a single mission.

Each scenario culminates in an operational pause (or reset period) after a scenario is completed. This reset period covers activities such as reorganization, resupply, repair activities and mission planning for future operations. The ACV crew and available maintainers have between 6 to 8 hours after each scenario to prepare the ACV for the next scenario. During these reset periods, the crew and unit mechanics can perform preventive maintenance and corrective maintenance. The reset period is dedicated to continuing actions and preparing for the next scenario, including fixing mef failures that must be restored before starting the next scenario. During the reset period, the ACV unit will be supporting patrols around assembly areas or resupply runs, but it is not symmetric across the unit. The ACV unit is not considered to be in an operating status; therefore, any failure of a mef will be scored as an EMA and must be restored prior to the start of the next scenario. If the EMA is not restored during the reset period in-between scenarios it will be marked as an OMF.

An individual ACV starts a scenario once it has been classified as mission capable and available for tasking. MC means all mission essential functions are available and that an ACV has no OMFs, EMAs, safety or dead lining issues remaining from previous scenarios.

3. Classification, Chargeability, and Hazard Severity Guidelines.

In scoring test incidents, conference members will follow the Test Incident Scoring Decision Chart (figure 2) (ref. a).

Figure 2: Test Incident Scoring Decision Chart

a. Classification.

Classification explains system performance incidents according to failure definitions contained in the FD/SC Charter. Table 2 presents the classification categories. It should be noted that all safety and environmental related incidents regardless of the classification are unacceptable and corrective action will be addressed.

Table 2. Classification Categories

No-Test Use if incident occurred due to Pre-test Checkout. Any incident observed during the designated burn-in, pretest inspection, or other pretest activity is classified as No Test. The Test Plan must specify the length of the burn-in period (the number of miles, rounds, or hours) to permit a determination of when the pretest period has ended.

Equipment Modification. Maintenance done to install a hardware kit or to incorporate a redesigned component is classified as No Test. However, if the replaced part was not functioning when it was being replaced, that malfunction will be scored on its own merits. A subsequent malfunction of the installed part will also be scored on its own merits.

Test-Peculiar Incident. An incident caused by someone not acting as a test player (crew member or maintainer), or by equipment not part of the system being tested, is classified as No Test. An example of this is an engineering evaluation and the maintenance done in furtherance of that evaluation. This classification also includes malfunctions to or caused by test instrumentation. However, an incident caused by test-peculiar equipment (equipment used in the test in lieu of the equipment to be fielded) will be scored under its own merits; if the test planners have introduced equipment for the purposes of the test, they have judged it to be an adequate substitute for the equipment to be fielded. Hence, its failures are to be regarded as representative of the failures of the equipment to be fielded.

Checks and Services. These are checks and services in addition to before/during/after operations actions prescribed in the applicable TM, directed by the test site, performed by the operator (or by the crew, if applicable) using only repair parts and On-Equipment Material (OEM) in accordance with the equipment publication before, during, or after the operation of the equipment. Checks and services that meet these conditions are classified as No Test.

Test-Directed Abuse. An incident in which the tester directs the deliberate abuse of the system (e.g., a test to over-stress the performance limits of the system), whether called for by the test plan or not.

Non-RAM. This is a catch-all term to capture those incidents in which a TIR has been prepared, but which have no bearing on the RAM assessment. Examples are suggested improvements; reports on inadequate test procedures; reports on unacceptable replacement parts, provided they were discovered before or during installation; reports on the equipment’s consistent inability to meet performance specifications even though no actual malfunction has occurred; suggested human factors improvements; and recommended changes to the system support package.

Crew Correctable Maintenance Action

A CCMA is a malfunction that resulted in a mef failure that was correctable within time limits described in Table 3. CCMA is field-level maintenance with repair parts, tools and personnel available at the platoon level. The recurrence of CCMAs which exceed a total corrective time of 60 minutes or occur more than 3 times within a scenario will be classified as an OMF.

Note (1) Class II weapon incident allow up to 10 minutes to resolve issue with standard supplied weapon tools per TOP 3-2-045, 17 Sep 2007.

b. Chargeability.

Chargeability explains system performance discrepancies according to cause, focusing on the element of the system primarily responsible for the incident. Table 4 presents the categories of chargeability. The Program Manager will provide a master list of all CFE and GFE items for both the hardware and software for the systems under test prior to entering the OT.

Table 4. Chargeability Categories

(CFE)

Malfunction or inaccuracy of hardware component that the contractor designed, bought, modified, or subcontracted and integrated into the system, including all contractor-modified GFEs.

CH

(GFE)

Malfunction or inaccuracy of an unmodified government-furnished component of the system under test.

(CFE)

Malfunction or inaccuracy of software component that the contractor designed, bought, modified, or subcontracted and integrated into the system, including all contractor-modified GFEs.

CS

(GFE)

Malfunction or inaccuracy of an unmodified government-furnished software component of the system under test.

c. Hazard Risk Assessment.

To aid the achievement of the system safety objective, hazards identified during the testing shall be characterized as to hazard severity categories and hazard probability levels (ref. A).

(1) Hazard Severity.

For each incident, regardless of chargeability, a hazard severity category will be assigned per the priority codes presented in table 5.

Table 5. Hazard Severity Categories

(2) Hazard Probability.

For each incident assigned a hazard severity category from I to IV, a hazard probability level will be assigned as described in table 6.

Table 6. Hazard Probability Categories

4. References.

A. Department of Defense. MIL-STD-882E System Safety. May 2012.

B. ACV 1.1 Operational Mode Summary/Mission Profile. April 10, 2014.

AMPHIBIOUS COMBAT VEHICLE

AUGUST 2014

AUGUST 2014

Appendix A: ACV Relationship/Scoring Matrix The table below provides examples of failure-induced capability degradation that should be scored an OMF or failure for each mef described in paragraph 2. Note: the table below is not intended to define and cover all of the potential incidents and functional degradation but rather serve as a guide for the scoring members.

Table 7: Incident Description (Functional Degradation) and Classification for ACV Mission Essential Functions

Incident Classification

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

A-1

AUGUST 2014

A-5

AUGUST 2014

image1.png image2.emf