Attachment J.8 TSA RMA Guide for OST Version 1.0.pdf

Office of Security Technology

RMA Guide

Reliability, Maintainability, Availability (RMA)

Guide

For the

Office of Security Technology

Version 1.0

June 22, 2011

Version Description

Version Change Date

1.0 Baseline document June 2011

Table of Contents

1. Introduction

2. RMA Terminology

3. RMA Requirements in Acquisition Documentation

a. Operational RMA Metrics

b. Inherent RMA Metrics

4. RMA Metric Development

a. Operational RMA Metrics

b. Inherent RMA Metrics

5. RMA Testing

6. RMA Process Control

7. Supportability and Sustainment

Appendix A Acronyms

1. Introduction

This Guide provides the standardized process the Transportation Security Administration (TSA) Office of Security Technology (OST) will use to develop and apply Reliability, Maintainability, and Availability (RMA) requirements for the acquisition of security equipment. The intent of the guide is to ensure:

1. User requirements are adequately considered through the ORD approval process

2. Department of Homeland Security (DHS) Acquisition Directive (AD) 102-01 requirements are met

3. RMA terminology, measures and formulas are used consistently throughout the security equipment acquisition process

4. Operational requirements are realistic and have a logical basis

5. Acquisition strategies include provisions to ensure that acquired security equipment will obtain the required operational availability threshold

2. RMA Terminology

OST will use standard RMA metrics for each security equipment acquisition and will apply these metrics in the same way each time in TSA acquisition documentation. AD-102 provides the following standard definitions for RMA.

1. Reliability: The ability of a system to perform its mission without failure, degradation, or demand on the support system. Reliability is a measure of the degree to which a system can complete it primary mission over a given duration.

2. Maintainability: The ability of a system to be retained in, or restored to a specified condition when maintenance is performed by personnel having the specified skill levels, using prescribed procedures and resources, at each prescribed level of maintenance and repair. Maintainability is a characteristic of design, installation, and supportability expressed as a measure of the degree to which a system will be retained in or restored to a specified condition within a given period of time, when the maintenance is performed in accordance with prescribed procedures and resources.

3. Availability: A measure of the probability that a system will be in an operable state and can be committed at the start of a mission when the mission is called for at an unknown (random) point in time.

4. Inherent Reliability, Maintainability and Availability (RMA) Value: Any measure of reliability, maintainability or availability that includes only the effects of item design and installation, and assumes an ideal operating and support environment.

5. Operational Reliability, Maintainability or Availability (RMA) Value: Any measure of RMA that includes the combined effects of item design, quality, installation, environment, operation, maintenance, and repair.

Tables 1 and 2 below identifies the standard RMA metrics, definitions, formulas, and application OST will use for all security equipment acquisitions, and identifies how OST will apply the RMA metrics in program acquisition documentation.

Table 1: RMA Definitions and Applications

Measure Statistic Definition Application

R eliab ility

Time to Failure

Mean Time Between

Critical Failure

(MTBCF)

For a particular interval, the system operating time1 divided by the total number of critical failures2

For design of a technology, and evaluation of its ability to perform dependably during Developmental Test & Evaluation (DT&E) and Operational Test & Evaluation (OT&E) and during Operations and Maintenance (O&M)

Time Between Maintenance

Minimum Time Between

Preventive Maintenance

(MTBPM)

A measure of reliability that represents the minimum number of days allowed between Level 2 preventive maintenance3 conducted on a security equipment technology.

For design of a technology, and evaluation of its ability to perform dependably during DT&E and OT&E and during O&M

M ain tain ab ility

System Down Time

Mean Down Time

(MDT)

Total downtime4 divided by the total number of failures5. It includes all time a system is not able to perform its mission including field technician response time, lead time for parts not readily available, or other administrative or logistics downtimes.

For evaluation of operational sustainment performance during OT&E and during O&M

Repair Time

Mean Time to Repair

(MTTR)

The total elapsed time (clock hours) to diagnose and repair a security equipment technology divided by the total number of corrective maintenance6 actions during a given period of time. It does not include other delay times such as field technician response time, lead time for parts not readily available, or other administrative or logistics downtimes.

For design of a technology, and evaluation of its inherent capability to be retained in or restored to a given condition during DT&E and OT&E

Measure Statistic Definition Application

A vailab ility

Operational Availability

Operational (Ao)

The percentage of time, during operational hours, that a security equipment technology is available to perform its required mission.

For evaluation of the operational performance of a technology during OT&E and during O&M

Inherent Availability Inherent (Ai)

Availability of a system with respect only to uptime and corrective maintenance. Ai ignores standby and delay times associated with preventive maintenance as well as mean logistics delay time7.

For design of a technology and evaluation of its inherent capability during DT&E

1. System Operating Time – The period of time a system is available to perform its required mission. In test, equal to the number of hours the system is in operation (net of downtime). During the Operations & Maintenance life cycle phase, airport operational hours (period of time within 24-hour day that an airport is operational) minus downtime will be used as a proxy for system operating time.

2. Critical Failure – A failure that causes the system to be non-operational and requires action by the maintenance service provider to restore the system to operation. Events that a TSO can correct via system resets, simple removal of obstructions, etc. are not categorized as critical failures.

3. Level 2 PM – These Preventive Maintenance (PM) activities are performed by trained Contractor technicians and are performed routinely on a set time schedule (e.g., monthly, quarterly, yearly). These do not include activities performed by the operator on a more routine basis (e.g., hourly, daily, start up, per shift, etc.) without a need to open the equipment (e.g., wipe machine, replace paper, etc.).

4. Downtime – The period of time during which a system is not in a condition to perform its required mission.

This includes all times that a system is not available due to corrective, or depot maintenance. Downtime includes the total response and repair time of Contractor technicians.

5. Failure – Any unscheduled event that prevents a system from performing a mission function.

6. Corrective Maintenance – An action performed by a trained maintenance technician to restore a system to a specified condition.

7. Logistics Delay Time - Includes delay time for spares, support equipment, personnel, facilities, transportation, and administrative activities.

Table 2: RMA Metrics, Formulas and Applicable Acquisition Documentation

Measure Metric Formula Acquisition Documents

Reliability

MTBCF

MTBCF = System Operating Time/Number of Critical Failures

ORD1, FRD2

MTBPM

MTBPM = Minimum Days Between Level 2 Preventive Maintenance (Contractor)

FRD

Maintainability

MDT MDT = Total Downtime / Number of Failures ORD

MTTR

MTTR = Corrective Maintenance Time/ Number of Failures

FRD

Availability

Ao Ao = Uptime3 / (Uptime + Downtime) ORD, APB4

Ai Ai = MTBCF/ (MTBCF + MTTR) FRD

1. ORD – Operational Requirements Document

2. FRD – Functional Requirements Document

3. Uptime – The period of time a system is available to perform its required mission (or System Operating

Time)

4. APB – Acquisition Program Baseline

The specific RMA measures applied will depend on whether an inherent or operational value is more appropriate.

3. RMA Requirements in Acquisition Documentation

Various acquisition documents are required throughout the Acquisition Life Cycle. The specific RMA measures that apply depend on whether it describes an operational requirement or an inherent (design and installation related) requirement.

a. Operational RMA Metrics

OST will use three operational metrics; Ao, MTBCF, and MDT. The TSA user community considers Operational Availability a Key Performance Parameter (KPP) to support their need for equipment availability for screening operations. OST will document the Ao requirement in the ORD, specify it as a KPP in the APB, and will evaluate it during OT&E and throughout the O&M phase.

Reliability is the measure that determines the frequency of system failures or maintenance actions and supports the user need for satisfactory performance for a given period of time. OST will document the MTBCF requirement in the ORD, and will evaluate it during OT&E and throughout the O&M phase.

Maintainability supports the user’s need to rapidly restore security equipment to operational status following a failure. MDT measures the combined operational impact of a technology’s maintainability design and the effectiveness of the logistics support system. OST will document the MDT requirement in the security equipment ORD, and will evaluate it during OT&E and throughout the O&M phase.

b. Inherent RMA Metrics

OST will use four metrics to define the inherent RMA requirements for security equipment; A i, MTBCF, MTBPM and MTTR. All inherent RMA requirements will be evaluated during DT&E.

MTBCF and MTBPM will be the reliability requirements specified in the FRD. The MTBCF requirement will be the same value from the ORD. The MTBPM requirement will define the minimum acceptable time period (days) between scheduled Level 2 preventive maintenance (Contractor-performed).

MTTR will be the maintainability requirement specified in the FRD, and OST will derive the A i requirement directly from the MTBCF and MTTR requirement through the formula in Table 2.

4. RMA Metric Development

OST will use a standard methodology and process for developing the different RMA requirements. Threshold and Objective values will be set for each requirement (except MTBPM) to document the performance levels that meet the following definitions:

Threshold - the minimum operational performance, in the user’s judgment, that TSA is willing to accept

Objective – a level of performance significantly beyond the threshold that represents desired performance.

a. Operational RMA Metrics

The logical basis for developing the Operational RMA requirements is the user need and historical RMA performance data from security equipment currently in operation. Following are the steps that OST will use to develop the RMA Operational metrics documented in the security equipment ORD.

Step 1: In coordination with the user, establish a uniform Availability Ao Threshold and Objective KPP for all security equipment procured under either the Electronic Baggage Screening Program (EBSP) or the Passenger Screening Program (PSP). The Ao Threshold represents the minimum acceptable percentage of time the user considers that security equipment must be available for screening. The Ao Objective represents a desired performance level based on historical limits experienced by other OST security equipment. The availability Threshold will be stated as a percentage; the Objective as a ≥ percentage value.

Step 2: OST will evaluate historical maintenance down time data of similar type security equipment currently in operation to establish an MDT Threshold. The MDT Threshold represents the maximum acceptable down time given the existing Contractor Logistics Support (CLS) concept and contracts.

Step 3: OST will derive the MTBCF Threshold by substituting the Ao and MDT values from Steps 1 and 2 above into the formula Ao = MTBCF / (MTBCF + MDT). The MTBCF Threshold represents the minimum permissible time between critical failures or maintenance actions that will permit the security equipment to meet availability requirements.

Step 4: OST will evaluate historical RMA performance data of all security equipment in operation to establish the one best possible MDT Objective consistent with existing Contractor Logistics Support (CLS) contracts. The MDT Objective represents a significant maintainability performance improvement of interest to TSA users within affordability constraints.

Step 5: OST will determine the MTBCF Objective for each new security equipment technology based on historical RMA performance data from similar type security equipment, and ensure the MTBCF Objective and MDT Objective meets or exceeds the desired Ao Objective from Step

1. The MTBCF Objective represents the desired performance levels.

The following table demonstrates how these steps would be used for the development of the Threshold (T) and Objective (O) RMA Operational metrics. Availability (Ao) thresholds and objectives (Column A and B) are set by the user. Maintainability (MDT) requirements (Columns E and F) are established using historical information. Having these two sets of requirements allows the derivation of the Reliability requirements (MTBCF – columns C and D). The derivation is simply the Availability divided by (1 minus Availability) times MDT. Notional examples for two fictitious technologies (Technology X and Y) are provided.

Technology Availability (Ao) Reliability (MTBCF) Maintainability (MDT)

T O T O T O

Column A B C D E F

Step 1 1 3 5 2 4

Source User User Derived

C = A/ (1-A)*E

Historical &

D/(D + F ) ≥ B

Historical Historical

X 96% ≥ 99.5% 288 2388 12 6

Y 96% ≥ 99.5% 216 1990 9 6

b. Inherent RMA Metrics

A logical basis for developing inherent RMA requirements is to derive appropriate metrics from the ORD requirement or from historical RMA performance data for security equipment currently in operation. Following are the steps that OST will use to develop the inherent RMA metrics that will be documented in the FRD.

Step 1: OST will use the MTBCF Thresholds and Objectives from the ORD as inherent reliability requirements in the FRD. The MTBCF Threshold represents the minimum permissible time between critical failures or maintenance actions that will permit the security equipment to meet availability requirements. The MTBCF Objective represents the desired performance levels. For the MTBPM inherent reliability requirement, OST will coordinate with security equipment users to determine the minimum acceptable time period (days) between scheduled Level 2 preventive maintenance (Contractor-performed). This MTBPM requirement will be specified in the FRD.

Step 2: OST will evaluate historical maintainability data of all security equipment currently in operation and establish a uniform MTTR Threshold for all future security equipment. The MTTR Threshold represents the maximum acceptable corrective maintenance time after a critical failure. It does not include repair delay times such as field technician response time, lead time for parts not readily available, or other administrative or logistics downtimes.

Step 3: OST will evaluate the historical maintainability data of all security equipment in operation and determine a uniform MTTR Objective for all future security equipment. The MTTR Objective represents an inherent maintainability design goal.

Step 4: OST will derive an Inherent Availability (Ai) Threshold and Objective for each new security equipment technology from the MTBCF and MTTR values specified. The Ai Threshold represents the inherent availability with respect only to operating time and corrective maintenance. The Ai Objective represents an inherent availability design goal.

The following table demonstrates how these steps would be used for the development of the Threshold (T) and Objective (O) RMA Inherent metrics. Reliability (MTBCF) thresholds and objectives (Column C and D) are taken from the ORD (having been established as described above). Maintainability (MTTR) requirements (Columns E and F) are established using historical information. Having these two sets of requirements allows the derivation of the Availability requirements (Ai - columns A and B). The derivation is simply the Reliability divided by Reliability plus Maintainability. Notional examples for two fictitious technologies (Technology X and Y) are provided.

Technology Availability (Ai) Reliability (MTBCF) Maintainability (MTTR)

T O T O T O

Column A B C D E F

Step 4 4 1 1 2 3

Source Derived

A = C/(C + E)

Derived

B = D/(D + F)

ORD

ORD Historical Historical

X 99.1% 99.9% 288 2388 2.5 1

Y 98.8% 99.9% 216 1990 2.5 1

5. RMA Testing

The fundamental purpose of Test & Evaluation (T&E) will be to verify attainment of technical performance in DT&E before acquisition, and required operational effectiveness and suitability during OT&E.

DT&E will focus on the verification of inherent system RMA technical performance assuming an ideal operating and support environment (e.g. trained operators, readily-available logistics support resources). DT&E will verify the RMA technical requirements in the FRD (e.g. MTBCF, MTTR, and Ai).

OT&E will be a field test of the security equipment under realistic conditions with users who represent those expected to operate and maintain the equipment when it is deployed. As such, it will consider the effects of the environment, operators, and the logistics support system as part of the test. RMA data collection during OT&E will focus on identifying what fails, the condition under which the failure occurs, and what it takes to get the system operating again.

Historical logistics response times, for the specific airport locations selected for operational testing, will be used to realistically depict logistics support performance during OT&E. OT&E will verify RMA performance requirements in the ORD (e.g., MTBCF, MDT, and Ao).

6. RMA Process Control

OST will adopt acquisition strategies and contract provisions to ensure that delivered security equipment will attain the required Operational Availability Threshold, and to demonstrate the program office’s commitment to meeting user objectives. In those cases where a vendor does not meet the Threshold requirement during testing, but compelling operational needs dictate acquisition and deployment; OST will withhold funds from the procurement contract until the deployed security equipment meets or exceeds the RMA Threshold requirement. This may include a vendor initiated reliability improvement program to improve the performance of the security equipment through Engineering Change Proposals (ECP). Any RMA improvement measures will be tracked, measured and reported. In addition, OST will explore the use of incentives in the security equipment procurement contract to drive performance toward the RMA Objective.

7. Supportability and Sustainment

Effective supportability and sustainment includes integrating supportability requirements into the systems engineering process – “designing for support” and developing and obtaining an integrated systems support package (e.g. spares, support equipment, tech manuals, etc.) – “supporting the design.”

Supportability is influenced through the application of effective RMA requirements on system procurement contracts. Logisticians will collaborate with Engineers to develop RMA requirements that promote system performance and maintainability performance through supportable designs.

Sustainment planning is driven by system availability requirements in the ORD and the inherent supportability capabilities of technologies. Sustainment activities are implemented through Contractor Logistics Support (CLS) contracts, which are performance-based and specify an appropriate RMA measure as their essential contract performance requirement. The CLS contracts include penalties if the contractor does not meet the established RMA performance requirement in the CLS contract. Contracting Officer’s Technical Representatives (COTR) will assess CLS performance monthly to ensure Contractors are meeting contract performance requirements.

OST will use Ao as the performance requirement for CLS contracts with Original Equipment Manufacturers (OEM). MDT will be the CLS contract performance requirement for third party Maintenance Service Providers (MSP) that have no control over the inherent reliability and maintainability performance of a technology.

MSPs will be required to document every maintenance action and report them to OST for evaluation to determine whether contract performance requirements are met and to monitor and assess actual versus expected security equipment performance. OST will continually evaluate the performance of deployed equipment throughout the O&M Phase to identify supportability issues, identify potential reliability improvements, or assist in recapitalization planning.

Appendix A Acronyms

Acronym Definition

AD Acquisition Directive

Ai Inherent Availability

Ao Operational Availability

APB Acquisition Program Baseline

CLS Contractor Logistics Support

DHS Department of Homeland Security

DT&E Developmental Test & Evaluation

EBSP Electronic Baggage Screening Program

ECP Engineering Change Proposal

FRD Functional Requirements Document

KPP Key Performance Parameter

MDT Mean Down Time

MSP Maintenance Service Provider

MTBCF Mean Time Between Critical Failure

MTBPM Minimum Time Between Preventive Maintenance

MTTR Mean Time To Repair

O&M Operations & Maintenance

OEM Original Equipment Manufacturer

ORD Operational Requirements Document

OST Office of Security Technology

OT&E Operational Test & Evaluation

PM Preventive Maintenance

PSP Passenger Screening Program

RMA Reliability, Maintainability, Availability

T&E Test & Evaluation

TSA Transportation Security Administration

TSO Transportation Security Officer