AFOTECPAM_99-104 _AFOTEC_Operational_Suitability_Test_ _Evaluation_Guide.pdf

BY ORDER OF THE COMMANDER AIR FORCE OPERATIONAL TEST AND

EVALUATION CENTER PAMPHLET 99-104

1 MAY 2007

Test and Evaluation

AFOTEC OPERATIONAL SUITABILITY TEST

AND EVALUATION GUIDE

ACCESSIBILITY: Publications and forms are available on the e-Publishing website at www.e-Publishing.af.mil for downloading or ordering.

RELEASABILITY: There are no releasability restrictions on this publication.

OPR: HQ AFOTEC/TSE Certified by: HQ AFOTEC/TS (Lt Col Manas) Supersedes AFOTECPAM 99-104, 8 February 2002 Pages: 112

This pamphlet provides a compilation of “best practices” and corporate wisdom of “how to” efficiently and effectively develop the operational suitability portion of the test concept, test plan and final report. It provides definitions of common terms and measures and identifies processes related to suitability test and evaluation. This guide will equip you with a solid foundational knowledge of the concepts and procedures pertaining to operational suitability test and evaluation. Detachments, test teams, and support personnel should use it as a “how to” guide in conjunction with guidance provided in the AFOTEC OT&E Guide and the mandatory direction contained in AFOTECI 99-101, Conduct of Operational Test and Evalua-tion. Refer recommended changes and questions about this publication to the Office of Primary Respon-sibility (OPR) using the AF IMT 847, Recommendation for Change of Publication; route AF IMT 847s from the field through the appropriate functional’s chain of command. Ensure that all records created as a result of processes prescribed in this publication are maintained in accordance with AFMAN 37-123 (will convert to AFMAN 33-363), Management of Records, and disposed of in accordance with the Air Force Records Disposition Schedule (RDS) located at https://afrims.amc.af.mil/.

SUMMARY OF CHANGES

Readers should thoroughly review this significantly revised publication. Two significant changes are the placement of example measures in attachments and the focus on educating newcomers to operational test and evaluation (OT&E). This version removes entire chapters or portions thereof and combines a few chapters. Thus, the chapter order has changed. Some chapter portions have relocated to other chapters or attachments. The formatting now conforms to the direction in Air Force Instruction (AFI) 33-360, Publi-cations and Forms Management, 18 May 2006.

Chapter 1— INTRODUCTION 6

1.1. Guide’s Intent and Organization

1.2. Operational Suitability Defined

1.3. Significance of OT&E for System Suitability

http://www.e-Publishing.af.mil https://afrims.amc.af.mil/

2 AFOTECPAM99-104 1 MAY 2007

Chapter 2— RELIABILITY, AVAILABILITY, MAINTAINABILITY 9

2.1. Introduction

2.2. Repairable and Non-repairable Systems in the Context of Reliability, Availability, and Maintainability (RAM) Planning and Analysis

2.3. Dissecting Total Equipment Time

Figure 2.1. Linear Breakdown of Total Equipment Time

Figure 2.2. Dendritic Breakdown of Total Equipment Time

2.4. Analytical Products

Figure 2.3. Notional State Graph

Table 2.1. Example Failure and Repair Data

2.5. Probability Distributions

2.6. Reliability

2.7. Aspects of Reliability

2.8. Availability

2.9. Maintainability

Chapter 3— LOGISTICS SUPPORTABILITY 19

3.1. Introduction

Table 3.1. Sustainment Planning, Logistics Support and Operational Suitability Linkage

3.2. Definitions

3.3. Logistics Support Concepts

3.4. Other Topics Related to Logistics Support

Chapter 4— USAGE RATES 25

4.1. Introduction

4.2. Definitions

4.3. Sources of Usage Rate Data

4.4. Usage Rate Evaluation Considerations

Table 4.1. Example Sortie per Aircraft (SPA) Requirements

Table 4.2. Example Test Data

4.5. Usage Rate Measures

Chapter 5— COMPATIBILITY AND INTEROPERABILITY 29

5.1. Introduction

AFOTECPAM99-104 1 MAY 2007 3

5.2. Definitions

5.3. Planning For Test

Table 5.1. Notional Factors and Descriptors Affecting EMC OT&E Planning

5.4. Test Execution and Reporting Considerations

Chapter 6— TRANSPORTABILITY 34

6.1. Introduction

6.2. Definitions

6.3. Transportability Concepts

6.4. Transportability OT&E Planning Considerations

Chapter 7— SAFETY AND ENVIRONMENTAL IMPACTS AND EFFECTS 39

7.1. Introduction

7.2. Environment, Safety and Occupational Health (ESOH)

7.3. ESOH Council (ESOHC)

7.4. ESOH Statutory Compliance

7.5. ESOH-Management Survey (ESOH-MS)

7.6. ESOH Planning and Review Process

7.7. Health and Safety Plan (HSP)

Table 7.1. Risk Level Matrix

7.8. ESOH Certification Board (ESOHCB)

7.9. Staff Meteorology Division (TSW) Support

Table 7.2. ESOH Operational Impact Matrix

Chapter 8— HUMAN FACTORS 44

8.1. Introduction

8.2. Definitions

8.3. Human Factors (HF)/Human System Integration (HSI) Policies

8.4. HF/HSI Areas of Expertise

8.5. HF/HSI Test Measures

8.6. HF/HSI Test Plan Inputs

8.7. HF/HSI Test Execution

8.8. Questionnaire Support

4 AFOTECPAM99-104 1 MAY 2007

Chapter 9— DOCUMENTATION 47

9.1. Introduction

9.2. Definitions

9.3. Documentation Test Planning Considerations

9.4. Documentation Measures

9.5. Documentation Data Requirements. Data requirements usually include:

9.6. Documentation Evaluation Considerations

Chapter 10— TRAINING 50

10.1. Introduction

10.2. Training and Training Support

10.3. Training OT&E Methodology

10.4. Training Considerations

Chapter 11— MODELING AND SIMULATION 52

11.1. Introduction

11.2. Models and Simulations

Chapter 12— HOW TO TAILOR TESTING 54

12.1. Introduction

12.2. Tailoring Suitability Testing for Systems (General)

12.3. Tailoring Suitability Testing for Joint Programs

Chapter 13— MANAGING OPERATIONAL SUITABILITY DATA 56

13.1. Introduction

13.2. Data Management Procedures

13.3. Tasks for Suitability Personnel

13.4. Other Data Sources

13.5. Data Management and Analysis Plan (DMAP)

13.6. Joint Reliability and Maintainability Evaluation Team (JRMET)

13.7. Test Data Scoring Board (TDSB)

Chapter 14— DEFICIENCY REPORTING 62

14.1. Introduction

14.2. Deficiency Reporting (DR)

AFOTECPAM99-104 1 MAY 2007 5

Table 14.1. Attributes which may affect OSS&E

Table 14.2. Deficiency Report (DR) Category and Priority Determination

Attachment 1— GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION 65

Attachment 2— MEASURES IDENTIFICATION PROCESS 90

Attachment 3— RELIABILITY MEASURES 93

Attachment 4— AVAILABILITY MEASURES 97

Attachment 5— MAINTAINABILITY MEASURES 102

Attachment 6— LOGISTICS MEASURES 106

6 AFOTECPAM99-104 1 MAY 2007

Chapter 1

INTRODUCTION

1.1. Guide’s Intent and Organization.

1.1.1. The intent of this guide is to provide a general overview of suitability and useful information to test team members assigned the responsibilities of planning, executing, and analyzing tests involving suitability areas of interest. To fulfill this intent, this guide is meant to educate newcomers to Opera-tional Test and Evaluation (OT&E) about the peculiarities of evaluating a system for operational suit-ability.

1.1.2. The main body of this guide provides a translation of real-world operations to operations in OT&E. For example, some metrics from real-world operations are not as meaningful in OT&E due to smaller sample sizes and/or shorter time periods of consideration. It is important for analysts to know this information to understand the sources of data used to assess suitability.

1.1.3. The attachments include sample test measures and provide detailed guidance for collecting and analyzing data. It is important for the operators/maintainers to know this information to understand how the data are to be collected and the goal of data collection.

1.1.4. Detachments and test teams should tailor suitability testing to their specific systems.

1.2. Operational Suitability Defined. The AFOTEC OT&E Guide contains this definition of opera-tional suitability: “The degree to which a system can be placed satisfactorily in field use with consider-ation given to availability, compatibility, transportability, interoperability, reliability, wartime usage rates, maintainability, safety, human factors, manpower supportability, logistics supportability, natural environmental effects and impacts, documentation, and training requirements.” Some elements of this definition, such as safety, human factors, compatibility, interoperability and environmental effects, also apply to operational effectiveness. The “Memorandum of Agreement on Operational Suitability Termi-nology and Definitions to be Used in Operational Test and Evaluation (OT&E)” provides the policy, terms and definitions for the Service Operational Test Agencies (OTA) to use in suitability evaluations.

This memorandum of agreement (MOA) is located on the Agreements & Charters page under the Plans & Policy menu item of the AFOTEC Management Information Network (MIN) home page.

1.3. Significance of OT&E for System Suitability.

1.3.1. During OT&E, test team members will evaluate operational suitability along with operational effectiveness for a given system in order to provide a complete operational perspective to the warf-ighter and determine the overall mission capability of a system. The test team will evaluate the areas listed in the definition above to search for suitability shortfalls, identify them, discover their cause and document the results. In the end, operational suitability testing is essential because the suitability shortfalls identified in OT&E provide the warfighter with key information that plays a large role in determining the overall mission capability of a system. Thus, a carefully orchestrated suitability anal-ysis is necessary for the warfighter to make an informed fielding decision for a given system.

1.3.2. OT&E examines many facets of suitability for a given system. Some of these suitability areas may include the effects the system will have on operations tempo, what support personnel will be required to maintain the system and what level of logistics support will be needed for the system. In

AFOTECPAM99-104 1 MAY 2007 7

addition, the OT&E results will identify how deployable the system is, how often it breaks, how easy it is to fix, how time-consuming it is to fix and what type of training maintainers will need to support the system. Furthermore, OT&E will examine the type of storage and transportation needed for the system at the main and forward operating bases. OT&E will also provide the warfighter with a sub-stantial amount of human factors data on the system’s ease of use, what type of issues the user may have with the system’s design, the effectiveness of system training and the impact the new system might have on the user’s day-to-day mission. A well-planned, properly executed OT&E is key to iden-tifying suitability shortfalls of a given system.

1.3.3. Proper suitability training for operational testers is of paramount importance, because suitabil-ity analysis can be difficult and costly to perform. Some areas of suitability such as reliability might be particularly difficult to test because of the limited number of hours or test articles available for the test. If the system under test (SUT) simply does not fail during the allotted time, a conclusion about mean time between critical failures will be difficult to report without other avenues of analysis.

Because of difficulties such as this, the test team must have a solid plan for conducting suitability analysis, along with a well-defined methodology and a list of the models they will employ to fill any gaps in the data collected from the actual test. It is important for the test team to have a solid under-standing of testing for operational suitability before planning the operational test in order to take into account the full scope of suitability testing.

1.3.4. To the warfighter, a system’s suitability performance could be a major factor that determines the life-cycle cost of the system. Money is not the only cost issue. A system with suitability issues could dramatically increase the number of maintainers needed to sustain it, so fielding it could have an impact on manpower. Furthermore, a system with major human factors issues might increase operator workload dramatically. For instance, if a new chemical suit is very effective at providing protection against chemical agents, yet is significantly more thermally burdensome than the currently fielded chemical suit, the warfighter will need to weigh the added protection against the drawbacks of the excessive thermal burden. In addition, if a system needs to be stored in a climate-controlled environ-ment, that requirement might add significant cost to its deployment in the theater if a special facility is required to store it. Moreover, if a new chemical agent detector has a high false alarm rate for detect-ing chemical/biological agents, the impact on base operations could be catastrophic. Alternatively, the commander may lose confidence in the detector and thus, the system becomes useless for its intended purpose. These examples represent some potential operational impacts that suitability issues may pose for a given system. It is the responsibility of the operational test community to discover these suitabil-ity shortfalls before the system is fielded in order to reduce the warfighter’s risk in purchasing a sys-tem that might be extremely expensive in the long run.

1.3.5. In addition to increasing life-cycle cost, manpower, facilities and workload, suitability issues could also have a significant impact on mission capability. For example, during OT&E, reliability, availability and maintainability data will be gathered and analyzed. If a system has a large mean repair time, but also has a large mean time between critical failures, this might not impact the system’s mis-sion capability. However, if the system experiences a critical failure often and has a lengthy repair time, this could result in low availability. Hence, even though the system might be extremely effective when available, it would not be mission capable because it is rarely available to support the warf-ighter.

1.3.6. OT&E also provides an opportunity to test the system with other systems to identify any prob-lems with compatibility and interoperability. These problems can be fixed at a much lower cost early

8 AFOTECPAM99-104 1 MAY 2007

in the acquisition cycle of a program as opposed to making accommodations after the system has been fielded. Regardless of the system under test, it is essential to determine whether it will work with the other systems in its intended environment. Suitability analysis in OT&E provides the warfighter with a measure to gauge the impact of incorporating a new system into the battlespace environment. Again, if the system is highly effective but not interoperable with the other support/supported systems, it might not be mission capable and worth fielding.

1.3.7. Another area in which suitability analysis pays dividends is identifying any natural environ-mental hazards or impacts the new system might cause. For example, if the combination of four new aircraft engines is found to produce too much noise during OT&E, the refitted aircraft might not be able to fly or land in an area that has restrictions against high levels of noise. If this impact is not iden-tified before the engine is fielded on the fleet, there may be a considerable amount of political fallout from this acquisition decision due to new restrictions on the aircraft’s available basing and operating locations. A proper OT&E suitability study is invaluable to identify such environmental impacts.

1.3.8. Suitability issues can be very transparent without a properly planned and well-orchestrated OT&E for a system. Developmental Test and Evaluation (DT&E) does not provide the full compila-tion of data for a system under test that the warfighter needs to make an informed fielding decision.

Although DT&E does not satisfy the requirements for a fielding decision, it is important for AFOTEC to be involved with DT&E. By integrating OT&E data points into DT&E events, AFOTEC can show the overall growth of the system prior to OT&E and the developer can find operationally relevant problems earlier. In addition, data from DT&E becomes available and useable for OT&E reporting.

Besides using the OT&E results to make a fielding decision, the warfighter can use these results to help employ a system to accomplish the mission. The operating command can use these results to improve the system’s concept of operations (CONOPS), logistics plan, maintenance plan, system training for both users and maintainers, as well as determine the scope of the system’s footprint and life-cycle cost. A system’s overall mission capability is determined by its effectiveness and suitability.

Thus, it is very important to properly scope and plan for suitability analysis early in the OT&E pro-cess, and then carefully conduct the suitability analysis during OT&E in order to correctly resolve the overall mission capability of a given system.

AFOTECPAM99-104 1 MAY 2007 9

Chapter 2

RELIABILITY, AVAILABILITY, MAINTAINABILITY

2.1. Introduction. Evaluating a system’s reliability, availability and maintainability (RAM) plays a key role in determining whether or not the system can be placed satisfactorily in field use. This chapter pro-vides test teams with details concerning RAM concepts and definitions. Specifics on which RAM test procedures will be followed are in the Joint Reliability and Maintainability Evaluation Team (JRMET) charter of each system under test.

2.2. Repairable and Non-repairable Systems in the Context of Reliability, Availability, and Main-tainability (RAM) Planning and Analysis.

2.2.1. For the purposes of RAM test planning and analysis, a system under test is either repairable or non-repairable. Repairable systems are restorable to a functioning status whenever a failure occurs.

Such repair is corrective maintenance. In addition to this corrective maintenance, these systems also receive preventive maintenance (PM). The classic example of a repairable system is an aircraft.

Non-repairable systems are those that are not repaired once they fail; usually they are simply dis-carded upon failure. While such systems receive no corrective maintenance, they may or may not receive preventive maintenance. An example of a non-repairable system that receives no preventive maintenance is a chemical/biological protective suit that is discarded whenever it tears. An example of a non-repairable system that receives preventive maintenance is a chemical/biological protective mask whose filter is periodically replaced, but discarded if the mask develops cracks or tears.

2.2.2. The distinction between repairable and non-repairable systems is important when selecting which measures of RAM are appropriate for the system under test. These measures in turn drive how total equipment time is broken down and what data the test team needs to collect during testing.

2.2.3. RAM analysis entails characterizing the nature of random variables and making inferences regarding expected performance in operational settings based on observations made during testing.

The random variables with which we deal in RAM applications typically involve chronological time:

the waiting time until an occurrence of an event, the time between the successive occurrences of events, or the time to complete some task are examples. The events of interest in RAM applications are typically failures of some sort (plain failures, critical failures (CF), or operational mission fail-ures(OMF)), other events that “down” a system, or perhaps the delivery of parts needed to complete a repair job. A task duration of interest is the time to restore function to a downed system. This task has several subtasks such as diagnosing the problem, obtaining the parts to fix it, actually fixing it, and accruing administrative and logistics delays. We might be interested in the duration of each of these subtasks as well. Again, the actual times we measure depend on the RAM requirements set forth by the user and whether or not the system is repairable.

2.3. Dissecting Total Equipment Time.

2.3.1. A key objective for test teams to remember when planning tests for either repairable or non-repairable systems is that, in order to do a proper RAM analysis after the test, they need to be able to account for and break down all equipment time during the test period. At the simplest level, it is necessary that test teams collect the proper information during test to conclude, after the fact, whether the SUT was “up” or “down” at any given time during the test period. Knowing uptime and downtime

10 AFOTECPAM99-104 1 MAY 2007

will only enable the computation of one of the more basic measures of readiness, namely operational availability (Ao). Ao is simply a function of the total amount of uptime and the total amount of down-time. To compute RAM measures, total uptime and total downtime must each be further divided into several constituent parts. Figure 2.1. illustrates how total equipment time is broken into the parts needed to compute the various measures of RAM.

Figure 2.1. Linear Breakdown of Total Equipment Time.

2.3.2. Uptime (UT), for instance, consists of standby time (ST) and operating time (OT). Downtime (DT) consists of active maintenance time and administrative/logistical delay time. Active mainte-nance time is then further subdivided into corrective maintenance time (CMT) and preventive mainte-nance time (PMT). If the system is non-repairable, corrective maintenance falls out of the breakdown and DT includes only PMT and preventive administrative and logistics delay time (ALDT).

2.3.3. Figure 2.2. is an alternate presentation of the concept illustrated in Figure 2.1., namely, how total equipment time can be broken down into components. Total time is first split into active time and inactive time. Inactive time is not split any further. Active time is composed of uptime and downtime.

Uptime and downtime are each divided into multiple categories, and so on. This breakdown is obvi-ously more detailed than the corresponding graphic at Figure 2.1. The terminology differs somewhat, and most importantly, accommodates certain possibilities that the breakdown in Figure 2.1. does not allow. For example, it accounts for the possibility of doing preventive and corrective maintenance on the system during uptime and/or during mission time. For some systems, this possibility is realistic, while for others, it is not.

AFOTECPAM99-104 1 MAY 2007 11

Figure 2.2. Dendritic Breakdown of Total Equipment Time

2.3.4. Presenting these two breakdowns highlights that the proper method to use depends on the par-ticular system under test. The breakdowns given in Figure 2.1. and Figure 2.2. likely would require tailoring to fit other systems under test. The tailoring would depend primarily on the specific RAM measures being used (which, in turn, will reflect the complexity of the system and whether the system is repairable or not).

2.3.5. In any case, it is very important to clearly define each segment (if using a linear breakdown as in Figure 2.1.) or each node (if using a dendritic breakdown as in Figure 2.2.) such that they are mutually exclusive. The question, “What exactly constitutes active time, inactive time, mission time, standby time, and off time?” should be asked to aid these definitions.

2.4. Analytical Products.

2.4.1. Figure 2.3. depicts a notional “state graph” of a fictional computer system under test. Assume that this computer is required to operate continuously, though it may be in a “hot standby” mode at times (on, but not performing any of its functions). The horizontal axis represents time and the vertical axis represents a binary state variable X(t) that takes value 1 when the computer is functioning or is assumed functional and takes the value 0 when it is not functional (values other than 0 or 1 are not defined). At time t0 the computer is booted up and the observation period begins. Initially, the com-puter is on but standing by for a short period of time, which is counted as standby time (ST11), fol-lowed by a period of operation, counted as operating time (OT11). The computer is considered functional (up) during ST11 and OT11, and thus the total uptime at this point in the observational period is ST11+OT11 and total downtime is 0. At time t1, the computer “crashes” and begins the first period of downtime (DT1) in this test. Maintenance personnel immediately begin diagnosing the problem, which is counted as corrective maintenance time (CMT11). They realize they must obtain parts from supply. The time spent ordering the parts and waiting for their delivery is counted as

12 AFOTECPAM99-104 1 MAY 2007

administrative and logistics delay time (ALDT11). Once they receive the parts, they resume correc-tive maintenance (CMT12) and at t2, the computer is once again up and in standby mode (ST21).

Total downtime at this point is CMT11+ALDT11+CMT12 (or t2 - t1). This graph would continue charting the state of the computer until the test period ends.

Figure 2.3. Notional State Graph.

2.4.2. A tabular form of similar data is in Table 2.1.

Table 2.1. Example Failure and Repair Data

* C/P/A = Corrective/Preventive/Administrative

2.5. Probability Distributions.

2.5.1. The exponential distribution is typically overused to describe a failure distribution, possibly resulting in a misrepresentation. A primary assumption of the exponential distribution is a constant failure rate; that is, a used item is assumed to be as good as a new item and an item is just as likely to fail at the beginning of a test as at the end of a test. Usually, a failure distribution has either an increas-ing failure rate (system fails more often over time) or a decreasing failure rate (system fails less often over time). Since reliability is the probability that a system will perform satisfactorily for a given time (mission length), it is inappropriate to use the exponential distribution without either having relevant a priori data from previous testing or waiting to model the system failure distribution after test.

Downing Event

(DE)

Type Clock Time

(hours) of

DE

Inter-occur rence Time

(hours)

Time (hours) to Restore Function

(C/P/A*)

Clock Time (hours) Back in Service

1 OMF 170 170 10/0/1 181

2 OMF 242 61 12/0/1 255

3 OMF 293 38 12/0/1 306

4 PM 337 31 0/5/0 342

Etc.

AFOTECPAM99-104 1 MAY 2007 13

2.5.2. The exponential distribution is a type that can be used when testing a system that operates as a function of a continuous measure (time, distance, temperature). In such tests, the times at which fail-ures occur and durations of operating time without failure are important to collect. These data aid the determination of the failure rate (constant, increasing or decreasing). (Weibull analysis determines failure rate characteristics.) When a system with a constant failure rate is also repairable, the Poisson distribution is used to analyze the system’s reliability.

2.5.3. In contrast to a continuous measure, a discrete measure takes on only countable values (0, 1, 2, etc.). An example of a discrete measure is the ratio of systems that fail to the total number of systems tested during a specified period. A sampling plan is required to define the interval of testing. The bino-mial distribution is appropriate to analyze test articles that result in a distinguishable success or fail-ure.

2.5.4. Many measures of suitability (MOS) are discrete. Due to limitations (for example, time and assets) during OT&E, meaningful MOSs are not possible in OT&E. For example, a mission capability rate from a test may be reported, but it is not meaningful since the utilization rate was less than that of the operational fleet. Modeling and simulation can be used to calculate MOSs, provided the data required for the model have been collected. Such data may include the times at which failures occur.

2.6. Reliability.

2.6.1. There are various definitions of reliability. The Defense Acquisition Guidebook defines reli-ability as, “The ability of a system and its parts to perform its mission without failure, degradation or demand on the support system.” AFI 10-602, Determining Mission Capability and Supportability Requirements, contains a different definition: “The probability that a system will perform satisfacto-rily for a given time when used under specified operating conditions.” Note that the first definition refers to an ability of an item, namely its ability to perform a mission, rather than some measure of that ability, as in the second definition. Measures of this ability are in Attachment 2. The distinction between an ability and a measure of an ability is fundamental to the discussion that follows in Section 2.7.

2.6.2. In general, reliability is a term used to describe quantitatively how failure-free a system is likely to be during a given period of operation. The International Organization for Standardization (ISO) defines reliability as the ability of an item to perform a required function, under given environ-mental and operational conditions, and for a stated period of time (ISO 4802). Marvin Rausand1 fur-ther interprets the ISO definition, writing that the item to which the ISO definition refers may be any component, subsystem or system that can be considered as an entity and that the function in the defi-nition may be a single function or a combination of functions that is necessary to provide a required service.

2.6.3. The meanings of the phrases “given environmental and operational conditions” and “stated period of time” are important to the understanding of reliability. The phrase “given environmental and operational conditions” refers to the complete definition of the scenario in which the system will oper-ate. For a ground combat vehicle, these conditions include climatic conditions, road surface, and expected loads during a selected mission profile. These conditions should reflect operational usage.

The phrase “stated period of time” refers to the length of the mission described in a mission profile.

1. Rausand, M., and A. Høyland. 2004. System Reliability Theory: Models, Statistical Methods, and Applications, second edition. John Wiley and Sons, Hoboken, New Jersey.

14 AFOTECPAM99-104 1 MAY 2007

The specification of the length of the interval need not be (and in many cases will not be) a simple specification of clock time. For example, a mobile air defense system mission profile will define an interval containing X rounds fired, Y hours of electronics “on” time and Z miles of travel. Note that only one of the three elements of this specification involves clock time. For a simpler system, such as an air-burst artillery round, the interval may include a single event—round detonation.

2.7. Aspects of Reliability.

2.7.1. The basic concept of reliability is that the system performs satisfactorily, where satisfactory implies a lack of a broad variety of undesirable events and subsequent impacts (think risk analysis).

This general concept has two aspects: mission reliability and logistics reliability. Mission reliability refers to the concept of not having undesirable events (i.e. failures) during mission time and the imme-diate impact to that mission. In that sense, mission reliability is actually best considered in the context of effectiveness rather than of suitability. Logistics reliability refers to undesirable events (including potential or pending failures) with the impact being a burden on the support system. Note that a par-ticular undesirable event or failure could contribute to both mission reliability measures as well as logistics reliability measures.

2.7.2. Mission reliability relates to system effectiveness; logistics reliability relates to the burden of owning and operating the system. Measures of mission reliability address only those incidents that affect mission accomplishment. Measures of logistics-related reliability address all incidents that require a response from the logistics system.

2.7.3. Mission Reliability.

2.7.3.1. Measures of mission reliability quantify the probability that a system will perform mis-sion essential functions for a period of time under the conditions stated in the mission profile. Mis-sion reliability for a single-shot system, i.e., a pyrotechnic device, would not include a time period constraint. A system with high mission reliability has a high probability of successfully complet-ing the defined mission.

2.7.3.2. Measures of mission reliability address only those incidents that affect mission accom-plishment. A mission reliability analysis must, therefore, include the definition of mission essen-tial functions. For example, the mission essential functions for a tank might be to move, shoot and communicate. Requirements that are more specific could specify minimum speed, shooting accu-racy and communication range.

2.7.4. Logistics (Maintenance/Supply) Related Reliability. Logistics related reliability measures, as indicated above, must be selected so that they account for or address all incidents that require a response from the logistics system. Logistics related reliability measures may be further subdivided into maintenance related reliability and supply related reliability. These measures respectively repre-sent the probability that no corrective maintenance or the probability that no unscheduled supply demand will occur following the completion of a specific mission profile.

2.7.5. The mathematical models used to evaluate mission and logistics reliability for the same system may be entirely different.

2.7.6. AFI 10-602 defines both reliability and mission reliability as probabilities (“The probability of…”) while it defines logistics reliability as an ability (“The ability of …”). Observe that in the former two cases, the definitions refer to a measure of ability (probability of…), while in the latter case, the definition refers to the ability itself. Additionally, measures of mission reliability are given in

AFOTECPAM99-104 1 MAY 2007 15

AFI 10-602, one of which is also called mission reliability (Rm)—implying that mission reliability is both an ability as well as a measure of that ability. In contrast, logistics reliability is an ability with specific measures of that ability defined later in the document.

2.7.7. For AFOTEC purposes, the definitions of reliability, mission reliability, and logistics reliability found in AFI 10-602 will be viewed as abilities of the system under test and, therefore, will be treated as operational capabilities (OC) of a system. An OC is a system attribute or grouping of attributes that users and subject matter experts have identified as being crucial to the achievement of critical mission elements and/or operational objectives and are, therefore, of significant value to the warfighter. With this perspective, the definitions of reliability, mission reliability, and logistics reliability found in AFI 10-602 will be rewritten in terms of OCs, as:

2.7.7.1. Reliability— The capability of a system to perform satisfactorily for a given time when used under specified operating conditions. This suitability OC is frequently measured by the time2 between successive occurrences of unsatisfactory performance. Exactly what constitutes unsatis-factory performance for a particular system under test must be defined in advance of testing and usually involves various categories of failures (failures, critical failures, operational mission fail-ures), each of which must be clearly defined.

2.7.7.2. Mission Reliability— The capability of a system to perform its required function for a stated mission duration or for a specified time into the mission. Typical measures of this opera-tional capability are mission reliability (note that the term mission reliability refers to the OC itself and also to one of the measures of that OC), weapon system reliability, and break rate. Note that the OC mission reliability relates to system effectiveness rather than suitability.

2.7.7.3. Logistics Reliability— The capability of a system to perform failure free, under speci-fied operating conditions and time without demand on the support system. Typical measures of this suitability OC include time (or cycles, miles, etc.) between maintenance, demand, or remov-als.

2.8. Availability. Operational availability is the probability that a system will be ready for operational use (i.e., any specified purpose) when required. Availability is the desire of operations and the goal of maintenance. Availability is dependent on reliability, maintainability and logistics supportability. A goal of the availability assessment is to determine if the system can meet the user’s availability requirements stated in a capability document (initial capabilities document (ICD), capability development document (CDD), capability production document (CPD)) or another requirements document.

2.8.1. Availability Concepts.

2.8.1.1. Availability translates the reliability, maintainability and logistics supportability charac-teristics of the system into a measure of interest to the user. It is based on the question, “Is the equipment in working condition when it is needed?” To be realistically evaluated, the evaluator should compare the availability of the system with mission requirements contained in existing requirements documents.

2.8.1.2. Measures of availability are generally percentages; i.e., either the percentage of time a system is capable of performing its mission or the percentage of time a fleet is capable of perform-

2. Throughout this document, time does not necessarily mean clock time. Here time includes indirect measures of time like miles, rotations, cycles, etc.

16 AFOTECPAM99-104 1 MAY 2007

ing its mission. During initial operational test and evaluation (IOT&E), little or no meaningful availability data may be available due to limited amounts of time and assets. Under these circum-stances, the availability assessment may require extensive modeling and simulation. There are some cases where data from the combined developmental test (DT)/operational test (OT) test period may help evaluate availability.

2.9. Maintainability. Maintainability is the ability of an item to be retained in or restored to a specified condition when personnel having specified skills, using prescribed procedures and resources at each pre-scribed level of maintenance and repair perform maintenance. A commonly used working definition states that maintainability is the inherent characteristic of a design that determines the type and amount of maintenance required to retain that design in, or restore it to, a specified condition. In this definition, “retained in” refers to preventative maintenance, while “restored to” addresses corrective maintenance.

2.9.1. Maintainability Concepts.

2.9.1.1. Maintainability, reliability, and other logistics support drivers are major system charac-teristics that impact availability. While maintainability is important as a factor of availability, it also merits substantial consideration as an individual system characteristic. Maintainability is a factor of the design process and an inherent design characteristic that is quantitative and qualita-tive in nature and, therefore, lends itself to specification, demonstration and trade-off analysis.

2.9.1.2. Maintenance refers to all actions required to retain an item in, or restore it to, a specified condition. This includes servicing, diagnosis, repair, modification, modernization, overhaul, rebuild, test, reclamation, inspection and condition determination.

2.9.1.3. In assessing maintainability, data such as maintenance time, direct maintenance work hours and system downtime are collected. The data are then reported as averages, either divided by some operational base such as flying hours, sorties or maintenance events/actions, or catego-rized by subsystems to highlight areas needing most attention. The key to effective maintainability assessment is to first measure maintainability, and then to identify factors that are influencing the results.

2.9.1.4. During an OT&E, both quantitative and qualitative aspects of maintainability are addressed.

2.9.1.4.1. Quantitative statistics for maintainability evaluation can be expressed as mainte-nance downtime per sortie, maintenance ratio (e.g., maintenance work hours per flying hour), total required work force (e.g., maintenance personnel (MP) per operational unit), time to restore a system to operational status (mean downtime), etc.

2.9.1.4.2. Qualitative aspects of maintainability include accessibility, serviceability, ease or difficulty of maintenance, safety, and human factors associated with maintenance actions.

These factors affect the quantity, skill levels and specialty codes of MP and the test equipment required to maintain the system. Qualitative evaluations of maintainability are usually done by experienced maintenance technicians using subjective judgment and are supported by quanti-tative maintainability measures.

2.9.2. Maintenance Demonstrations. Testers may conduct maintenance demonstrations (M-demos) during DT&E and OT&E. During DT&E, the system contractor conducts M-demos to demonstrate compliance with specifications. During OT&E, the test team may conduct M-demos for data gather-ing if done in the operational environment. Considerations concerning the use of M-demos follow.

AFOTECPAM99-104 1 MAY 2007 17

2.9.2.1. M-demos performed during OT&E are “staged” maintenance events done in an opera-tional environment to obtain quantitative maintainability information not otherwise available dur-ing OT&E. These are sometimes described as ease-of-maintenance demonstrations (removal and replacement of components, or performance of tasks). Other potential M-demos can be performed by intentionally inserting faulty components into the system to assess troubleshooting and repair capability. This is especially important when testing highly reliable systems where test exposure is small compared with the expected time between failures. Demonstrations can be used to quantify maintenance times for tasks that will not be required during the course of the test.

2.9.2.2. Many times the contribution of M-demos in an OT&E is limited because they account for the operational frequency and mode of failures. Additionally, M-demos do not give data on induced and no-defect failures. When employing M-demos in OT&E, they should be:

2.9.2.2.1. Clearly defined and scoped in the OT&E test plan.

2.9.2.2.2. Coordinated with the implementing command.

2.9.2.2.3. Performed at or near the end of OT so as not to interfere with or alter the equipment under test.

2.9.2.2.4. Conducted in an environment that simulates as closely as possible the operational and maintenance environment planned for the item (i.e., blue-suit maintenance with no con-tractor involvement).

2.9.2.2.5. Representative of the working conditions, tools, support equipment, spares, facili-ties and technical publications that would be required during operational service.

2.9.2.2.6. Videotaped.

2.9.2.3. Some data collected during M-demos may be included in RAM calculations; other data may not. For example, the repair times from an M-demo in which a failure is induced or inserted may be included in the calculation of mean repair time. However, including the maintenance event in the calculation of mean time between maintenance is inappropriate.

2.9.3. Integrated Diagnostics. Integrated Diagnostics (ID) is “an initiative for delivering weapon systems designed for ease of maintenance (with built-in diagnostics) with less test equipment and fewer maintenance specialists. Suggested by industry, it enhances military capabilities by increasing survivability of the support structure and by reducing the logistics task which could degrade unit mobility. By combining the diagnostics equipment into an integrated system, maintenance quality improves” (Defense Acquisition Guidebook).

2.9.3.1. The purpose of integrated diagnostics is to provide a cost-effective capability to detect and unambiguously isolate all faults known or expected to occur in weapons systems and equip-ment in order to satisfy weapon system mission requirements. In wartime, this becomes extremely significant in that it is imperative that critical failures be found and fixed quickly to support com-bat turn-around times, which can equalize battles against numerically superior forces.

2.9.3.2. The term “diagnostics” is often a general term to cover all means of determining that a system fault has occurred, and the means to determine where the fault is and to isolate it to a repairable or replaceable portion of the system. There are many other terms that relating to this area, including built-in test (BIT), built-in test equipment (BITE), built-in test and fault isolation test (BIT/FIT), and automatic test equipment (ATE).

18 AFOTECPAM99-104 1 MAY 2007

2.9.3.3. The key to integrated diagnostics is the successful consideration and integration of the functions of detection, isolation, verification, recovery, recording and reporting, in a comprehen-sive and cohesive fashion, with the operator and with support functions that may be automatically, semi-automatically and/or manually controlled.

2.9.3.4. See the AFOTEC Handbook on Integrated Diagnostics in the Test Support Directorate (TS) Analyst Training & Technical Information Center (ATTIC) portion of the MIN. Although this document is dated 1991 and requires an update, it still contains useful information and exam-ple measures and metrics.

AFOTECPAM99-104 1 MAY 2007 19

Chapter 3

LOGISTICS SUPPORTABILITY

3.1. Introduction. This chapter explains how to plan a logistics supportability evaluation. Logistics sup-portability is the degree to which the planned logistics support (test, measurement and diagnostic equip-ment; spare and repair parts; technical data; support facilities; transportation requirements; training;

manpower; and software) allows meeting system availability and wartime usage requirements. AFI 63-107, Integrated Product Support Planning and Assessment, provides policy for planning integrated acquisition and sustainment product support. AFI 10-602 is the governing document for establishing logistics supportability requirements. The program office and major command (MAJCOM) use the logis-tics support elements in AFI 10-602 and the sustainment planning elements in AFI 63-107 to design and develop the system. AFOTEC bases its suitability test on these documented supportability items. Table

3.1. shows the relationship between the elements of sustainment planning, logistics support, and opera-tional suitability.

20 AFOTECPAM99-104 1 MAY 2007

Table 3.1. Sustainment Planning, Logistics Support and Operational Suitability Linkage.

3.2. Definitions. The terms that follow are suitability areas that we must evaluate to assess mission capa-bility.

3.2.1. Manpower Supportability— The identification and acquisition of military and civilian per-sonnel with skills and grades required to operate and support a materiel system over its lifetime at peacetime and wartime rates.

Sustainment Planning Elements (AFI 63-107)

Logistics Support Elements

(AFI 10-602)

Operational Suitability

(AFOTECPAM 99-104)

System Engineering Design Interface Availability

Compatibility

Interoperability

Reliability

Maintainability

Safety

Human Factors

Natural Environmental Effects and Impacts

Manpower Manpower and Personnel Manpower Supportability

Logistics Supportability Personnel Manpower and Personnel Logistics Supportability Maintenance Maintenance Planning Maintainability

Support Equipment Logistics Supportability Facilities Logistics Supportability

Supportability Computer Resources Support Logistics Supportability Data Management Technical Data Documentation

Logistics Supportability Supply Supply Support Wartime Usage Rates

Logistics Supportability Transportation Packaging, Handling, Storage, and Transportation Transportability

Logistics Supportability Training Training and Training Support Training

Logistics Supportability Configuration Management

AFOTECPAM99-104 1 MAY 2007 21

3.2.2. Supply Support— All management actions, procedures, and techniques used to determine requirements to acquire, catalog, receive, store, transfer, issue, and dispose of secondary items. This includes provisioning for initial support as well as replenishment supply support.

3.2.3. Support Equipment— All equipment (mobile and fixed) required to support the operation and maintenance of a materiel system. This includes associated multi-use end items, ground handling and maintenance equipment, tools, meteorology and calibration equipment, test equipment, and auto-matic test equipment. It includes the acquisition of logistics support for the support and test equipment itself.

3.2.4. Technical Data— Recorded information regardless of form or character of a scientific or tech-nical nature.

3.2.5. Training and Training Support— The processes, procedures, techniques, training devices and equipment used to train civilian, active duty and reserve military personnel to operate and support a materiel system.

3.2.6. Computer Resources Support— The facilities, hardware, software, documentation, man-power and personnel needed to operate and support embedded computer systems.

3.2.7. Facilities— The permanent, semi-permanent or temporary real property assets required to sup-port the materiel system, including conducting studies to define types of facilities or facility improve-ments, location, space needs, utilities, environmental requirements, real estate requirements, and equipment.

3.2.8. Packaging, Handling, Storage, and Transportation— The resources, processes, proce-dures, design considerations, and methods to ensure that all system, equipment, and support items are preserved, packaged, handled, and transported properly, including environmental considerations, equipment preservation requirements for short- and long-term storage, and transportability.

3.2.9. Design Interface— The relationship of logistics-related design parameters, such as RAM, to readiness and support resource requirements. These parameters are expressed in operational terms rather than inherent values and specifically related to system readiness objectives and support cost of the materiel system.

3.3. Logistics Support Concepts.

3.3.1. Product Support. Governed by AFI 63-107, this concept focuses on…