AFI_99-103 _Capabilities-based_Test_ _Evaluation.pdf

BY ORDER OF THE AIR FORCE INSTRUCTION 99-103

SECRETARY OF THE AIR FORCE 26 February, 2008

Incorporating Change 2, 20 Mar 2009

Test and Evaluation

CAPABILITIES-BASED TEST AND EVALUATION

COMPLIANCE WITH THIS INSTRUCTION IS MANDATORY

ACCESSIBILITY: Publications and forms are available on the e-Publishing website at www.e-publishing.af.mil for downloading or ordering.

RELEASIBILITY: There are no releasibility restrictions on this publication.

OPR: HQ USAF/TEP Certified by: HQ USAF/TEP (Col S. Kyle)

Pages: 71 Supersedes AFI 99-103, 12 May 2008 This is the second update to Air Force Instruction (AFI) 99-103 which implements Air Force Policy Directive (AFPD) 99-1, Test and Evaluation Process. It describes the planning, conduct, and reporting of cost effective test and evaluation (T&E) programs as an efficient continuum of integrated testing known as seamless verification. This AFI implements the policies in Department of Defense Directive (DODD) 5000.1, The Defense Acquisition System, and DOD Instruction (DODI) 5000.2, Operation of the Defense Acquisition System (collectively called the DOD 5000-series); National Security Space (NSS) Acquisition Policy 03-01; Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 3170.01, Joint Capabilities Integration and Development System, and CJCS Manual (CJCSM) 3170.01, Opera-tion of the Joint Capabilities Integration and Development System. This AFI must be used in con-junction with AFI 10-601, Capabilities-Based Requirements Development, and AFI 63-101, Opera-tions of Capabilities Based Acquisition System. For recommended non-mandatory guidance, use the Defense Acquisition Guidebook. Any organization conducting T&E may supplement this instruction as desired. This instruction applies to all Air Force organizations, including the Air National Guard, Air Force Reserve Command, major commands (MAJCOM), direct reporting units (DRU), and field operating agencies (FOA). This instruction applies to all acquisition projects and programs regardless of acquisition category (ACAT). Headquarters (HQ) USAF/TE is the sole waiver authority for poli-cies in this AFI. Refer recommended changes and questions about this publication to the Office of Primary Responsibility (OPR) using AF IMT 847, Recommendation for Change of Publication, routed through the functional chain of command. Ensure all records created as a result of processes pre-scribed in this publication are maintained according to Air Force Manual (AFMAN) 37-123, Manage-ment of Records (will convert to AFMAN 33-363), and disposed of according to the Air Force Infor-mation Management System (AFRIMS) Records Disposition Schedule (RDS) located at https://afrims.amc.af.mil/rds_series.cfm.

SUMMARY OF CHANGES

*This interim change adds a new Chapter 8 for Space Systems Test and Evaluation. It also updates who is responsible for live fire test and evaluation (LFT&E) planning; adds direction for a Sufficiency http://www.e-publishing.af.mil/ of Operational Test Review (SOTR); adds direction about a Capabilities and Limitations (C&L) Re-port; revises the definition of operational utility evaluation (OUE); modifies lead operational test or-ganization determination criteria for ACAT III and other programs; deletes “spiral” terminology from evolutionary acquisition; requires consideration of concepts of operation (CONOPS) in early T&E planning; adds direction about the System Engineering Plan (SEP); makes the requirement optional for recommendations in operational test reports; adds responsibilities for the USAF Warfare Center; mod-ifies signature requirements for selected TEMPs; modifies when TEMP information is cited in the LCMP; and adds definitions for new terms. Numerous paragraphs throughout the AFI were changed to support these updates. A margin bar indicates newly revised material. Numerous paragraphs throughout the AFI were changed to support this updated policy. An asterisk indicates newly revised or added material.

TABLE OF CONTENTS

Chapter 1—VISION AND IMPLEMENTATION CONCEPTS

1.1. Purpose of Test and Evaluation (T&E)

1.2. The Acquisition Environment

Figure 1.1. Integration of the Requirements, Acquisition, and T&E Processes

1.3. Seamless Verification

1.4. Integrated Test Team (ITT)

1.5. General T&E Principles

1.6. Applicability

1.7. Areas Not Covered by This AFI

Chapter 2—TYPES OF TEST AND EVALUATION

2.1. Major Types of Testing

2.2. Developmental Testing

2.3. Specialized Types of Developmental Testing

2.4. Live Fire Test and Evaluation (LFT&E)

2.5. Operational Testing

2.6. Types of Operational Testing

Table 2.1. Summary of Operational Testing Options

2.7. Test Support for Technology Transition

2.8. Foreign Comparative Testing (FCT)

2.9. Joint Test and Evaluation (JT&E)

Chapter 3—RESPONSIBILITIES

3.1. Overview of Responsibilities

3.2. Director, Operational Test and Evaluation (DOT&E)

3.3. Headquarters, U.S. Air Force, Director of Test and Evaluation (HQ USAF/TE)

3.4. Assistant Secretary of the Air Force for Acquisition (SAF/AQ)

3.5. Under Secretary of the Air Force (SAF/US)

3.6. Headquarters, Air Force Materiel Command (AFMC)

3.7. Headquarters, Air Force Space Command (AFSPC)

3.8. Program Managers (PM)

3.9. Air Force Operational Test and Evaluation Center (AFOTEC)

3.10. Operational MAJCOM, DRU, and FOA Headquarters

3.11. Headquarters, U.S. Air Force, Deputy Chief of Staff, Air, Space, & Information Opera-tions, Plans, & Requirements (HQ USAF/A3/5)

3.12. Secretary of the Air Force, Office of Warfighting Integration and Chief Information Officer (SAF/XC)

3.13. Responsible Test Organization (RTO)

3.14. Integrated Test Team (ITT)

3.15. Participating Test Organizations (PTO)

*3.16. United States Air Force Warfare Center (USAFWC)

Chapter 4—T&E ACTIVITIES SUPPORTING MILESTONE A DECISIONS

4.1. Early Tester Involvement

Figure 4.1. Integration of Requirements, Acquisition, and T&E Events Prior to Milestone A

4.2. Early Tester Involvement in Requirements Development

Figure 4.2. Integrated Testing Concept Flowchart

4.3. Early Tester Involvement in the Acquisition Process

4.4. Formation of the Integrated Test Team (ITT)

4.5. Determining the RTO

4.6. Determining the Operational Test Organization

Figure 4.3. Determining the Operational Test Organization

4.7. OSD Oversight Programs With Multiple Subparts

4.8. Multiple Operational Test Organizations

4.9. Lead Service Considerations

4.10. Tester Inputs During Concept Refinement

4.11. T&E Strategy Development

4.12. Early Planning for T&E Resources

Chapter 5—T&E ACTIVITIES SUPPORTING MILESTONE B DECISIONS

5.1. Post Milestone A

Figure 5.1. Integration of Requirements, Acquisition, and T&E Events Prior to Milestone B Figure 5.2. Integrated Testing Concept Flowchart

5.2. Initial Integrated Test Design (IITD) Process

5.3. Critical Technical Parameters (CTP)

5.4. Formal Contractual Documents

5.5. Common T&E Data Management

5.6. Limitations on Contractor Involvement in Operational Testing

5.7. Integrating Specialized Testing

5.8. Modeling and Simulation (M&S) In Support of T&E

5.9. Early DT&E Planning

Figure 5.3. Early Planning for Integrated Testing Activities

5.10 Live Fire Test and Evaluation (LFT&E) Planning

5.11. Operational Assessment Planning and Execution

5.12. Tester Involvement in the Capability Development Document (CDD)

5.13. Life Cycle Management Plans (LCMP) and Single Acquisition Management Plans

(SAMP)

5.14. Test and Evaluation Master Plan (TEMP)

5.15. Testing COTS, NDI, and GFE

5.16. T&E Funding Sources

5.17. Deficiency Reporting (DR) Process

5.18. Program Management Directive (PMD)

5.19. Test Deferrals, Limitations, and Waivers

5.20. Precedence Ratings

Chapter 6—T&E ACTIVITIES IN SUPPORT OF MILESTONE C AND PRODUCTION

DECISIONS

6.1. Post Milestone B

Figure 6.1. Integration of Requirements, Acquisition, and T&E Events Prior to Milestone C

6.2. Developing the Integrated Test Concept (ITC)

Figure 6.2. Integrated Testing Concept Flowchart

6.3. Developing Test Plans That Are Integrated

6.4. Realistic Testing

6.5. Integrated Technical and Safety Reviews

6.6. Certification of System Readiness for Dedicated Operational Testing

6.7. Plans and Briefings for Operational Testing

6.8. DOT&E Test Plan Approval

6.9. Management of T&E Data

6.10. Deficiency Reports (DR)

6.11. Integrated Testing During Sustainment and Follow-On Increments

6.12. Disposing of Test Assets

*6.13. Early Fielding of Prototypes or Pre-Production Systems

Chapter 7—TEST AND EVALUATION OVERSIGHT AND REPORTING

7.1. OSD T&E Oversight List

7.2. General Reporting Policy

7.3. DT&E Reports

7.4. DT&E Report Distribution

7.5. Operational Test Reports

7.6. Operational Test Report Distribution

7.7. Electronic Warfare (EW) Programs

7.8. Integrated Test Reports

7.9. “Briefing Trail”

7.10. Control of Test Reports

7.11. Distributing and Safeguarding Test Information

7.12. Information Collection, Records, and Forms

*Chapter 8—SPACE SYSTEMS TEST AND EVALUATION *8.1. Chapter Overview

*Figure 8.1. NSS 03-01 vs. DODI 5000.02 Acquisition Process *8.2 Types of Space Test and Evaluation *8.3. Tester Involvement in JCIDS Process *8.4. Space T&E Activities Supporting KDP A

*Figure 8.2, Space System Integrated Testing Flow Chart for KDP A (Notional) *8.5. Space T&E Activities Supporting KDP B

*Figure 8.3. System Integrated Testing Flow Chart for KDPs B, C, and Build Approval (Notional) *8.6. Space T&E Activities Supporting KDP C and Build Approval *8.7. Space T&E Activities Supporting Decisions Up Through IOC

*Figure 8.4, Space System Integrated Testing Flow Chart for Consent to Ship and Operational Decisions (Notional).

Attachment 1—GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION Attachment 2—INFORMATION REQUIREMENTS FOR OSD T&E OVERSIGHT

PROGRAMS

Chapter 1

VISION AND IMPLEMENTATION CONCEPTS

1.1. Purpose of Test and Evaluation (T&E). The overarching functions of T&E are to mature sys-tem designs, manage risks, identify and help resolve deficiencies as early as possible, and ensure sys-tems are operationally mission capable (i.e., effective and suitable). The Air Force T&E community plans for and conducts integrated testing as an efficient continuum known as seamless verification in collaboration with the requirements and acquisition communities. The T&E community will:

1.1.1. Collaborate with requirements sponsors and system developers to field better systems faster and more cost effectively.

1.1.2. Provide timely, accurate, and affordable information to decision makers to support produc-tion and fielding decisions.

1.1.3. Help manage risks during engineering, acquisition, fielding, and sustainment by accurately characterizing system technical and operational performance throughout the system life cycle.

1.1.4. Help the acquisition and sustainment communities acquire and maintain operationally effec-tive, suitable, survivable, and secure systems for Air Force operators.

1.1.5. Provide information to operators to allow them to assess mission impacts, develop doctrines, improve requirements, and refine tactics, techniques, and procedures (TTP).

1.2. The Acquisition Environment. Agile acquisition is the overarching concept designed to bring new capabilities to operators more quickly. It begins with capabilities-based requirements develop-ment and continues with capability-based acquisition, T&E, and rapid fielding of new systems.

*1.2.1. Evolutionary Acquisition (EA). EA is the DOD’s preferred acquisition strategy for deli-vering warfighting capabilities to operators, and incremental development is the process used to carry out the EA strategy. See Attachment 1 for definitions of these terms.

1.2.2. Collaborative Concepts and Processes. Agile acquisition is based on new concepts and processes described in AFI 10-601, Capabilities Based Requirements Development, AFI 63-101, Operations of Capabilities Based Acquisition System, and this AFI. Figure 1.1 shows the acquisi-tion process as the “master clock” for the integration of requirements, acquisition, and T&E events and activities. Variations of Figure 1.1 are used throughout this AFI to illustrate T&E events dur-ing each phase of acquisition.

Figure 1.1. Integration of the Requirements, Acquisition, and T&E Processes.

IOC

Technology

Development System Development

& Demonstration Production

& Deployment Operations & Support

FOC

LRIP

A B C

Concept Refinement

MS MS MS

FRP

CDD CPD

RSR

AoA

DAB DAB

Select

MDA

Operations of Capabilities Based Acquisition System (AFI 63 -101)

DAB DRR

LCMPLCMP LCMPLCMP

ISPISP

LCMPLCMP

ISPISP

TDSTDS

Modifications

Capabilities Based Test and Evaluation (AFI 99-103)

EOA

OA OA

IOT&E and/or FDE

OT&E Certification

FOT&E and/or FDE

T&E Strategy TEMPTEMP

Integrated Government T&E Contractor Testing

COA

FRP

Decision

Operations & Support

ICD

ADM

Concept Decision

JCD/AFCD

RSR

AFROCC

JROC

FSA

AFROCC

RSR JROC

AFROCC

RSR JROC

AoA Plan

FSA

Plan

PMDPMD

Tester Involvement

AFROCC

Capabilities Based Requirement s Development (AFI 10 -601)

LFT&E

JROC

ITT Stand Up

IOT&E or FDE

TEMP

Note: All acronyms in this figure are listed in Attachment 1. Single acquisition management plans (SAMP) will be phased out as life cycle management plans (LCMP) are phased in.

1.2.3. National Security Space (NSS) Acquisition Process. The acquisition and decision making process described in National Security Space (NSS) Acquisition Policy 03-01 differs from the ac-quisition process in DODI 5000.2 and AFI 63-101. NSS 03-01 uses key decision points (KDP) for NSS acquisition programs that occur earlier than typical DOD 5000-series milestones and decision reviews. However, the basic T&E support provided to NSS systems is similar to non-space sys-tems. Whenever NSS systems are tested, testers must refer to NSS 03-01 for additional guidance.

1.3. Seamless Verification. The seamless verification concept helps testers structure T&E to better support the requirements and acquisition processes. Seamless verification minimizes the transitions between contractor, developmental, and operational testing by implementing integrated test techniques and procedures. Key stakeholders from multiple disciplines must integrate their efforts, produce effi-cient schedules, eliminate “stovepipes,” share information in open T&E databases, identify problems early, engage contractors to fix deficiencies sooner, and ensure systems are ready to enter dedicated operational testing with a high probability of success. Seamless verification is called integrated test-ing in this AFI and does the following:

1.3.1. Provides a new T&E framework to support programs using EA or a single stage of devel-opment.

1.3.2. Refocuses T&E of materiel solutions on capabilities-based requirements and operational mission needs instead of traditional pass-fail measurements of specification-like requirements. Ca-pability-based testing ensures T&E strategies and plans are derived from the operational environ-ment and functionality specified in validated operational capabilities requirements. It requires an understanding of how systems will be employed in operational environments and mandates that T&E strategies and plans be designed to determine whether a new capability solution merits field-ing.

1.3.3. Provides T&E data to support the requirement in Title 10, United States Code, for initial op-erational test and evaluation (IOT&E).

1.3.4. Integrates the various types of T&E described in Chapter 2 as seamlessly as possible through integrated test teams (ITT).

1.3.5. Requires operational test organizations to take advantage of every opportunity to provide early, written feedback to the program manager (PM) to maximize successful operational testing.

Note: This AFI uses the term “PM” for any designated person in charge of acquisition activities, to include those prior to MS A (i.e., before a technology project is officially designated an acquisition program).

*1.4. Integrated Test Team (ITT). An ITT will be formed during the Concept Refinement phase as shown in Figure 1.1 to create and manage the strategy for T&E for the life of each program. Formal direction for establishing the ITT will be in the new program’s first acquisition decision memorandum (ADM). The ITT construct is central to carrying out seamless verification and replaces the old test planning working group (TPWG). The PM and the operational test organization will co-chair the ITT using the general T&E principles outlined in paragraph 1.5. ITT membership will include representa-tives from the responsible test organization (RTO), operational test organizations, participating test or-ganizations (PTO), system contractors, and the acquisition, requirements, intelligence, operations, and support communities. Reference paragraphs 3.14 and 4.4 and the Air Force Test and Evaluation Gui-debook for details about ITT structure, responsibilities, and functions. The Guidebook is available on the HQ USAF/TE portion of the Air Force Portal.

1.5. General T&E Principles. The following T&E principles are based on DOD 5000-series docu-ments and lessons learned. The unifying theme is that all testers must collaborate to the fullest extent possible to make systems better regardless of organizational affiliation. Integrated testing is the pre-ferred way to organize all T&E activities, resources, and information within statutory and regulatory guidelines and sound engineering principles. Note: Because the acquisition process is fluid, testers must ensure the intent of this AFI is implemented at appropriate times. For example, if a project or program has no Milestone (MS) A, an ITT must be formed at an equivalent point in time and carry out all necessary actions.

1.5.1. Tailoring. All T&E strategies and plans must be flexible to fit the needs of acquisition pro-grams consistent with sound systems engineering practices, common sense, statutory and regulato-ry guidelines, and the time-sensitive nature of operators’ requirements. T&E strategies and plans must be tailored for their specific situations.

1.5.2. Early Tester Involvement. The early provision of T&E expertise and operational insight, preferably before the concept refinement phase, is key to successful initiation of new programs.

1.5.3. Early Deficiency Resolution. Deficiencies must be identified and resolved as early as possible to provide efficiency and economy.

1.5.4. Event-Driven Schedules and Exit Criteria. The ITT must plan for adequate time and re-sources for all T&E activities according to DODD 5000.1. T&E activities must demonstrate the system meets established engineering objectives, capability-based requirements, and exit criteria before moving to the next phase of development. The PM must ensure the system is stable and mature before it is certified ready for dedicated operational testing.

1.6. Applicability. The policies and processes in this AFI are for use by all Air Force acquisition programs and projects regardless of ACAT, to include non-ACAT programs, unless otherwise noted.

Note: See DODI 5000.2, Enclosure E2, for details about ACATs. AFOTEC and each MAJCOM or FOA with designated test responsibilities will establish disciplined processes for planning and execut-ing T&E activities according to this AFI. Note: In this AFI, the term “MAJCOM” will include the FOAs, DRUs (except AFOTEC), and other subordinate test organizations. Minor modification pro-grams (i.e., Air Force Form 1067 modifications, etc.) and MAJCOM-directed acquisition programs will comply with the following principles:

1.6.1. A T&E strategy and test plans tailored for the project or program are required.

1.6.2. Operational testing must be conducted and documented to support acquisition and/or field-ing decisions.

1.6.3. The overarching principles of integrated testing must be complied with.

1.6.4. Validated T&E data must be shared to the maximum extent possible within the test and ac-quisition community.

1.7. Areas Not Covered by This AFI. The systems, programs, and activities listed below are not covered by this AFI. Additional items such as T&E range instrumentation may also be excluded after consultation with HQ USAF/TEP. If there is a question of jurisdiction for space and space launch sys-tems, consult with HQ USAF/TEP, SAF/USA, and HQ AFMC/A3F or HQ AFSPC/A5 for guidance.

1.7.1. Nuclear components are governed by joint Department of Defense (DOD)-Department of Energy agreements. However, nuclear and non-nuclear components, sub-systems, and associated logistics support elements that require testing and nuclear certification throughout the system life cycle remain covered as described in AFI 63-103, Nuclear Weapons Program Management, and AFI 63-125, Nuclear Certification Program.

1.7.2. Industrial maintenance inspections.

1.7.3. Activities associated with the space experimentation program described in AFI 10-1202, Space Test Program (STP) Management.

Chapter 2

TYPES OF TEST AND EVALUATION

2.1. Major Types of Testing. Air Force testing falls into two overarching categories, developmental testing and operational testing. If a specific T&E requirement does not fall precisely into one of the following discrete types of testing, consult with HQ USAF/TEP to select and tailor the type of testing that best fits the need. During test planning, these tests will be integrated to the maximum extent poss-ible as described in Chapters 4 through 6. All testers must collaborate as an integrated team to gener-ate an overarching T&E strategy and test plans that leverage all available test activities and resources while minimizing redundant testing and waste. The result is an integrated test approach with interlock-ing test plans that efficiently work together throughout the acquisition program, and not necessarily a single test plan.

2.2. Developmental Testing. Developmental testing is conducted throughout the acquisition and sus-tainment processes to assist in engineering design and development and verify that critical technical parameters (CTP) have been achieved. Developmental test and evaluation (DT&E) supports the de-velopment and demonstration of new materiel or operational capabilities as early as possible in the ac-quisition life cycle. After full-rate production (FRP) or fielding, DT&E supports the sustainment of systems to keep them current or extend their useful life, performance envelopes, and/or capabilities.

As many test activities as practical are conducted in operationally relevant environments without com-promising engineering integrity, safety, or security. Developmental testing must lead to and support a certification that the system is ready for dedicated operational testing according to DODI 5000.2, En-closure E5, and AFMAN 63-119, Certification of System Readiness for Dedicated Operational Test and Evaluation. In addition, developmental testing:

2.2.1. Assesses the technological capabilities of systems or concepts in support of requirements ac-tivities described in AFI 10-601 (e.g., Analysis of Materiel Approaches, Courses of Action (COA)). Conducts research, development, test, and evaluation (RDT&E) to investigate new con-cepts and technologies and collect basic scientific and engineering data.

2.2.2. Provides empirical data for cost, schedule, and performance trade-offs.

2.2.3. Evaluates and uses modeling and simulation (M&S) tools and digital system models (DSM);

performs verification and validation with actual test data.

2.2.4. Identifies and helps resolve deficiencies as early as possible.

2.2.5. Verifies the extent to which design risks have been minimized.

2.2.6. Verifies compliance with specifications, standards, and contracts.

2.2.7. Characterizes system performance, military utility, and determines system safety.

2.2.8. Quantifies contract technical performance and manufacturing quality.

2.2.9. Ensures fielded systems continue to perform as required in the face of changing operational requirements and threats.

2.2.10. Ensures all new developments, modifications, and upgrades address operational safety, sui-tability, and effectiveness according to AFI 63-1201, Life Cycle Systems Engineering.

2.2.11. Supports aging and surveillance programs, value engineering projects, productivity, relia-bility, availability and maintainability projects, technology insertions, and other modifications ac-cording to AFI 63-1101, Modification Management.

2.3. Specialized Types of Developmental Testing.

2.3.1. Qualification Test and Evaluation (QT&E). QT&E is a tailored type of DT&E conducted for commercial-off-the-shelf (COTS) items, non-developmental items (NDI), and government fur-nished equipment (GFE). Depending on operator requirements, these items may require little or no government funded research and development (R&D), engineering, design, or integration efforts.

PMs cannot disregard T&E of COTS, NDI, and GFE simply because these items came from pre-established sources and some pre-existing data may be available. See paragraph 5.15 for more in-formation on COTS, NDI, and GFE.

2.3.2. Production-Related Testing. The PM will ensure T&E is conducted on production items to demonstrate that specifications and performance-based requirements of the procuring contracts have been fulfilled. Defense Contract Management Agency personnel normally oversee this test-ing at the contractor’s facility. Typical tests (defined in Attachment 1) include: first article tests (FAT); lot acceptance tests (LAT); pre-production qualification tests (PPQT); production qualifica-tion tests (PQT); and production acceptance test and evaluation (PAT&E).

2.4. Live Fire Test and Evaluation (LFT&E). LFT&E is a type of DT&E that provides timely, ri-gorous, and credible vulnerability or lethality test and evaluation of “covered” systems as they progress through the System Development and Demonstration (SDD) phase prior to FRP or a major system modification that affects survivability. Survivability consists of susceptibility, vulnerability, and recoverability information derived from the firing of actual weapons (or surrogates if actual threat weapons are not available) at components, sub-systems, sub-assemblies, and/or full up, system-level targets. Modeling, simulation, and analysis must be an integral part of the LFT&E process. The Air Force must initiate LFT&E programs sufficiently early to allow test results to impact system design prior to FRP or major modification decisions. Note: See paragraph 5.10 for more information; At-tachment 1 for key definitions; and Title 10 §2366. The Air Force accomplishes LFT&E to:

2.4.1. Provide information to decision makers on potential operator casualties, system vulnerabili-ties, lethality, and system recoverability while taking into equal consideration the susceptibility to attack and combat performance of the system.

2.4.2. Ensure system fielding decisions include an evaluation of vulnerability and lethality data under conditions that are as realistic as possible.

2.4.3. Assess battle damage repair capabilities and issues. Note: While assessment of battle dam-age repair is not a statutory requirement of LFT&E, test officials should exploit opportunities to assess such capabilities whenever prudent and affordable.

*2.5. Operational Testing. Operational testing determines if capabilities-based requirements have been satisfied and assesses system impacts to both peacetime and combat operations. It identifies and helps resolve deficiencies as early as possible, identifies enhancements, and evaluates changes in sys-tem configurations that alter system performance. Operational testing includes a determination of the operational impacts of fielding and/or employing a system across the full spectrum of military opera-tions and may be conducted throughout the system life cycle. Operational testing may also look at doctrine, operational concepts (as described in AFPD 10-28, Air Force Operational Concepts), system performance, TTPs, logistics support elements, intelligence support elements, system interoperability and security, materiel issues, safety, training, organization, human systems integration (HSI), and per-sonnel.

2.6. Types of Operational Testing. The types of operational testing listed below afford operational testers a range of options for completing their mission. “Evaluations” collect, analyze, and report data against stated criteria with a high degree of analytical rigor and are used to support FRP or fielding de-cisions. “Assessments” usually collect and analyze data with less analytical rigor, need not report against stated criteria, and cannot be the sole source of T&E data for FRP or fielding decisions. All programs that result in a FRP or fielding decision require an appropriate level of operational testing supported by sufficient independent evaluation to support the decision. The operational test organization, in conjunction with the user and OSD oversight organizations (if applicable), will determine the appropriate level of operational testing required, Operational testing of COTS, NDI, and GFE cannot be omitted simply because these items came from pre-established sources. Operational testing must be based on approved requirements documents specifically for the capabilities being fielded. Follow the methodology in paragraph 4.6 for determining the lead operational test organiza-tion for each acquisition program. See the definition of Operational Test and Evaluation (OT&E) in Attachment 1 for further information.

2.6.1. Initial Operational Test and Evaluation (IOT&E). IOT&E is only conducted by the Air Force Operational Test and Evaluation Center (AFOTEC). AFOTEC determines the operational effectiveness and suitability of the items under test using production or production-representative articles with stabilized performance and operationally representative personnel. Additionally, AFOTEC will resolve the mission capability of the system. Tests are conducted under operational conditions, including combat mission scenarios that are as operationally realistic as possible.

IOT&E determines if operational requirements and critical operational issues (COI) have been sa-tisfied and assesses system impacts to peacetime and combat operations. A dedicated phase of IOT&E is required for new ACAT I and II programs according to Title 10 §2399. Dedicated IOT&E is also required for all OSD OT&E Oversight programs according to DODI 5000.2. The determination of appropriate types of operational testing for subsequent modifications and up-grades as well as applicability to other types of programs will be addressed according to paragraph 4.6.

2.6.2. Qualification Operational Test and Evaluation (QOT&E). QOT&E is a tailored type of IOT&E conducted only by AFOTEC. It is used to evaluate military-unique portions and applica-tions of COTS, NDI, and GFE for military use in an operational environment when little or no government-funded R&D takes place. PMs cannot disregard T&E of COTS, NDI, and GFE simp-ly because these items came from pre-established sources. QOT&E supports the same kinds of de-cisions as IOT&E. See paragraph 5.15 for more information on COTS, NDI, and GFE.

2.6.3. Follow-on Operational Test and Evaluation (FOT&E). By definition, FOT&E is the continuation of operational test and evaluation (OT&E) after IOT&E or QOT&E and is conducted only by AFOTEC. It answers specific questions about unresolved COIs and test issues; verifies the resolution of deficiencies or shortfalls determined to have substantial or severe impact(s) on mis-sion operations; or completes T&E of those areas not finished during OT&E. AFOTEC OT&E re-ports will document known requirements for FOT&E. More than one FOT&E may be required.

2.6.4. Multi-Service Operational Test and Evaluation (MOT&E). MOT&E can be IOT&E, QOT&E, or FOT&E when two or more military Services are involved. It can also be a Multi- Service FDE if a MAJCOM is the lead test organization. See the Memorandum of Agreement [MOA] on Multi-Service Operational Test and Evaluation (MOT&E) and Operational Suitability and Definitions, and paragraphs 4.7, 4.8, and 7.9. If MAJCOMs are involved with multi-Service testing without AFOTEC, they should use this MOA as a guide.

*2.6.5. Force Development Evaluation (FDE). FDE is a type of dedicated operational test and evaluation performed by MAJCOM operational test organizations in support of MAJCOM-managed system acquisition-related decisions prior to initial fielding or for subsequent system sus-tainment or upgrade activities. An FDE may be used for multiple purposes to:

2.6.5.1. Evaluate and verify the resolution of previously identified deficiencies or shortfalls, including those not rated in AFOTEC’s OT&E final report as having a substantial or severe impact on mission operations.

*2.6.5.2. Evaluate routine software modifications (e.g., operational flight programs (OFP)), subsequent increments, upgrades, and other improvements or changes made to sustain or en-hance the system.

2.6.5.3. Evaluate and verify correction of new performance shortfalls discovered after fielding of the system.

2.6.5.4. Evaluate operational systems against foreign equipment.

2.6.5.5. Evaluate operational systems against new or modified threats.

2.6.5.6. Evaluate military-unique portions and applications of COTS, NDI, and GFE for mili-tary use.

*2.6.5.7. Support MAJCOM-managed acquisition program decisions and milestones. Note:

Use of OUE or FDE to support MAJCOM-managed acquisition decisions is at the discretion of the appropriate MAJCOM staff or test organization.

2.6.6. Tactics Development and Evaluation (TD&E). TD&E is a tailored type of FDE con-ducted by MAJCOMs to refine doctrine, system capabilities, and TTPs throughout a system’s life cycle according to AFI 11-260, Tactics Development Program. TD&Es normally identify non-materiel solutions to problems or evaluate better ways to use new or existing systems.

2.6.7. Weapons System Evaluation Program (WSEP). WSEP is a tailored type of FDE con-ducted by MAJCOMs to provide an end-to-end evaluation of fielded weapon systems and their support systems using realistic combat scenarios. WSEP also conducts investigative firings to re-validate capabilities or better understand munitions malfunctions.

*2.6.8. Operational Utility Evaluation (OUE). AFOTEC or MAJCOMs may conduct OUEs whenever a dedicated operational test and evaluation event is required, but the full scope and rigor of a formal IOT&E, QOT&E, FOT&E, or FDE is not appropriate or required. OUEs may be used to support operational decisions (e.g., fielding a system with less than full capability) or acquisi-tion-related decisions (e.g., low-rate production) when appropriate throughout the system lifecycle.

Operational test organizations may establish their supplemental internal guidance on when and how to use OUEs. OUEs will not be used when IOT&E, QOT&E, FOT&E or FDE are more appropri-ate per existing guidance and definitions.

2.6.9. Operational Assessment (OA). OAs are conducted by AFOTEC or MAJCOMs in prepara-tion for dedicated operational testing and typically support MS C or low-rate initial production (LRIP) decisions. They are designed to be progress reports and not intended to determine the overall mission capability of a system. They provide early operational data and feedback from ac-tual testing to developers, operators, and decision makers. An OA can also be used to support the assessment of new technologies. OAs will not be used as substitutes for IOT&E, QOT&E, FOT&E, FDE, or OUE. OAs are integrated with DT&E to:

2.6.9.1. Assess and report on a system’s maturity and potential to meet operational require-ments during dedicated operational testing, and possibly augment or reduce the scope of dedi-cated operational testing.

*2.6.9.2. Support long-lead, LRIP, or increments of evolutionary acquisition programs.

2.6.9.3. Identify deficiencies or design problems that can impact system capability to meet op-erational requirements, the mission, and/or employment concepts.

2.6.9.4. Uncover potential system changes needed to update operational requirements, COIs or the acquisition strategy.

2.6.9.5. Support the demonstration of prototypes, new technologies, or new applications of ex-isting technologies as described in paragraph 2.7, and demonstrate how well these systems meet mission needs or satisfy operational capability requirements.

2.6.9.6. Support proof of concept initiatives.

2.6.10. Early Operational Assessment (EOA). EOAs are similar to OAs, except they are per-formed prior to MS B to provide very early assessments of system capabilities and programmatic risks.

2.6.11. Summary of Operational Testing. The key distinctions between types of operational testing and the decisions they support are shown in Table 2.1. The results of TD&Es support the decision to implement (e.g., field or publish) new or revised TTPs. Note: Table 2.1 is intended as a summary and may not cover all possible T&E situations; refer to the descriptions in paragraph

2.6 or consult with HQ USAF/TEP for final guidance of any issues.

*2.6.12. Sufficiency of Operational Test Review (SOTR). For some programs of limited scope and complexity, system development testing or integrated developmental and operational test events may provide adequate operational test data to support MAJCOM fielding decisions. In these situations, the lowest appropriate level of required MAJCOM operational testing may consist of a review of existing data rather than a separate, dedicated operational test event. Each MAJCOM will determine the ap-propriate MAJCOM staff or test organization where decisions are made for using a SOTR. The SOTR may only be used to inform MAJCOM or user system fielding decisions. It may not be used as the sole source of operational test information for any type of acquisition milestone or production deci-sions. The SOTR may not be used for acquisition milestone decisions associated with OSD OT&E Oversight programs unless approved by DOT&E. See paragraph 7.5.5.

Table 2.1. Summary of Operational Testing Options.

FRP, FieldingEOA: MS B

OA: MS C, LRIP

Decision Supported

ACAT I, IA, II,

and OSD T&E Oversight

All programs Type Program

MAJCOMAllAFOTECAllWho Conducts

FDE and TD&EOUEIOT&E, QOT&E, FOT&E

EOA, OA

Type of Operation al Testing

ASSESSMENTS EVALUATIONS

FRP, Fielding when IOT&E, QOT&E, FOT&E, or FDE not warranted

All programsAll programs

FRP, Fielding when IOT&E, QOT&E, or FOT&E not warranted

2.7. Test Support for Technology Transition. DODI 5000.2 creates multiple paths for “technology projects” and experimentation projects to enter the acquisition system in support of warfighter needs and employment concepts. These activities fall outside the traditional acquisition process and beyond the scope of this AFI. Nonetheless, Air Force testers may be required to support these activities by providing T&E expertise, assessing the military utility of new technologies, and ensuring that T&E data can be used to support subsequent acquisition activities. Examples of these types of projects in-clude:

2.7.1. Advanced Technology Demonstrations (ATD). These are fully funded advanced devel-opment efforts used to meet the needs of employment concepts and capability requirements through “proof of principle” demonstrations in operationally realistic environments. ATDs demon-strate the maturity and potential of advanced technologies for enhancing military operational capa-bilities.

2.7.2. Advanced Concept Technology Demonstrations (ACTD) and Joint Concept Technolo-gy Demonstrations (JCTD). ACTD/JCTDs are Office of the Secretary of Defense (OSD)-sponsored projects designed to exploit maturing technologies that have the potential to fulfill near-term, validated, joint military needs and rapidly transfer those technologies to operators.

ACTD/JCTDs are not a formal part of the acquisition process. They are highly tailored “demon-strations,” under the direction of an ACTD/JCTD operator-sponsor, that typically use OAs or mili-tary utility assessments (MUA). For more information, contact SAF/XCOI or the Deputy Under Secretary of Defense for Advanced Systems and Concepts, DUSD(AS&C), at http://www.acq.osd.mil/actd/guidelns.htm.

http://www.acq.osd.mil/actd/guidelns.htm

2.8. Foreign Comparative Testing (FCT). FCT is an OSD-sponsored T&E program pre-scribed by Title 10 §2350a(g) and centrally managed by the Office of the Deputy Under Secre-tary of Defense (Advanced Systems and Concepts) Comparative Testing Office. FCT is con-ducted on foreign nations’ systems, equipment, and technologies to determine their potential to satisfy validated United States operational requirements. Testers participate in FCT projects as directed by the program management directive (PMD). See the Comparative Testing Office Procedures Handbook at http://www.acq.osd.mil/cto/handbook/2005CTOHandbookFinal.pdf.

2.9. Joint Test and Evaluation (JT&E). The JT&E Program charters joint test projects to evaluate non-materiel capabilities and potential options for increasing joint military effectiveness. The JT&E Program focuses on evaluating current equipment, organizations, threats, doctrine, TTPs, test metho-dologies, and system interoperability in realistic environments. The JT&E Program and organizational responsibilities are described in AFI 99-106, Joint Test and Evaluation Program, and DODI 5010.41, Joint Test and Evaluation (JT&E) Program. JT&E projects are not acquisition programs and are total-ly distinct and separate from MOT&E and multi-Service testing. See definitions in Attachment 1.

http://www.acq.osd.mil/cto/handbook/2005CTOHandbookFinal.pdf

Chapter 3

RESPONSIBILITIES

3.1. Overview of Responsibilities. All Air Force testers and the acquisition community will follow the T&E principles articulated in Chapter 1 of this AFI using the types of tests described in Chapter

2. Testers must collaborate with each other, acquisition officials, and requirements sponsors using the ITT as the T&E focal point for each program. The acquisition community must use this AFI to ensure the agile acquisition and seamless verification concepts function as intended.

3.2. Director, Operational Test and Evaluation (DOT&E). DOT&E:

3.2.1. Prescribes OT&E and LFT&E policies for the DOD according to Title 10 §139, §2366, §2399, and §2400; and DODD 5141.2, Director of Operational Test and Evaluation (DOT&E).

3.2.2. Exercises oversight responsibility for ACAT I or other programs in which the Secretary of Defense (SECDEF) has special interest. Monitors and reviews OT&E and LFT&E activities in the DOD. Participates in ITTs and test integrated product teams (TIPT) to foster program success.

3.2.3. Publishes, in conjunction with USD(AT&L)/DS, a combined list of OSD T&E Oversight programs for DT&E, LFT&E, and OT&E. Contact HQ USAF/TEP for the most current list.

3.2.4. Approves, in writing, the adequacy of operational test plans for those programs on OSD OT&E Oversight prior to the commencement of operational testing. Approves the operational test portions of integrated test plans. Approves the quantity of test articles required for operational test-ing of major defense acquisition programs (MDAP).

3.2.5. Approves test and evaluation master plans (TEMP) and T&E strategies for OSD T&E Over-sight programs in conjunction with USD(AT&L)/DS and Assistant Secretary of Defense for Net-works and Information Integration (ASD/NII).

3.2.6. Approves LFT&E strategies and waivers prior to commencement of LFT&E activities.

3.2.7. Submits a report to SECDEF and Congress before systems on OSD OT&E Oversight may proceed beyond LRIP.

3.3. Headquarters, U.S. Air Force, Director of Test and Evaluation (HQ USAF/TE). HQ USAF/TE will:

3.3.1. Function as the chief T&E advisor to Air Force senior leadership according to Headquarters Air Force Mission Directive (HAF MD) 1-52, Director of Test and Evaluation. Is responsible to the Chief of Staff of the Air Force (CSAF) for establishing Air Force T&E policy, determining the adequacy of T&E resources required to support weapons system development, and resolving T&E issues.

3.3.2. Act as the final T&E review authority and signatory for TEMPs prior to Service Acquisition Executive (SAE) approval and signature. See paragraphs 3.4.5, 3.5.5, and 5.14.2.

3.3.3. Collaborate with requirements sponsors and system developers to improve the development, testing, and fielding of Air Force systems or subsystems. Participate in high performance teams (HPT), ITTs, and TIPTs as necessary to help ensure program success.

3.3.4. Respond to and mediate T&E issues between HQ USAF principals, MAJCOMs, Air Force testers, the Services, OSD, and Congress.

3.3.5. Review and/or prepare T&E information for release to OSD and ensure timely availability of T&E results to decision makers.

3.3.6. Oversee the Air Force T&E infrastructure by ensuring adequate resources to support system acquisition activities. Administer various T&E resource processes and chair or serve on various committees, boards, and groups listed in HAF MD 1-52.

3.3.7. Acts as the single point of entry for the Foreign Materiel Program for the Air Force accord-ing to AFI 99-114, Foreign Materiel Program (S).

3.3.8. Function as the certifying authority for T&E personnel for T&E Level 3 in the Acquisition Professional Development Program (APDP) when not delegated to the MAJCOMs.

3.3.9. Provide advice on ITT charter development and membership requirements. Review ITT charters for programs where HQ USAF/TE participation is necessary.

3.3.10. Manage the Air Force JT&E Program according to DODI 5010.41 and AFI 99-106.

3.3.11. Perform other duties listed in HAF MD 1-52.

3.4. Assistant Secretary of the Air Force for Acquisition (SAF/AQ). SAF/AQ or designated repre-sentatives serve as the Air Force SAE for non-space systems. SAF/AQ will:

3.4.1. Direct PMs to assist ITTs and TIPTs in developing T&E strategies as early as possible (i.e., before MS A).

3.4.2. Ensure systems and programs are certified ready for dedicated operational testing according to AFMAN 63-119.

3.4.3. Ensure T&E responsibilities are documented in ADMs, PMDs, TEMPs, life cycle manage-ment plans (LCMP), expectation management agreements (EMA), and other program documenta-tion. Ensure the initial ADM gives direction for establishing the ITT. Note: Life cycle manage-ment plans (LCMP) are replacing single acquisition management plans (SAMP). In this AFI, all references to the TEMP include the SAMP or LCMP, whichever is applicable according to AFI 63- 107, Integrated Product Support Planning and Assessment. See paragraph 5.13 for more details about SAMPs and LCMPs.

3.4.4. Regarding LFT&E, SAF/AQ or designated representatives will:

3.4.4.1. Recommend candidate systems to DOT&E for compliance with LFT&E legislation after coordinating the proposed nominations with HQ USAF/TE.

3.4.4.2. Approve agreed-upon LFT&E strategies and Air Force resources required to accomplish LFT&E plans and forward to DOT&E. Forward LFT&E waivers (and legislative relief requests, if ap-propriate) to DOT&E, if required.

3.4.5. Approve TEMPs for all non-space ACAT I, IA, II, and other programs on OSD T&E Oversight.

Forward approved TEMPs to DOT&E and USD(AT&L)/DS.

3.5. Under Secretary of the Air Force (SAF/US). SAF/US will:

3.5.1. Function as DOD’s Space Milestone Decision Authority (MDA) and Air Force SAE for as-signed space system acquisition programs according to NSS 03-01 and applicable OSD and SECAF delegations. Note: The acquisition process in NSS 03-01 is different than the acquisition process in DODI 5000.2 and AFI 63-101. See paragraph 1.2.3.

3.5.2. Direct PMs to assist ITTs and TIPTs as early as possible (i.e., before KDP A).

3.5.3. Ensure systems and programs are certified ready for dedicated operational testing according to AFMAN 63-119.

3.5.4. Ensure space system T&E responsibilities are documented in the ADM, PMD, TEMP, inte-grated program summary (IPS), and other program documentation. Ensure each program has es-tablished an ITT prior to KDP-A.

3.5.5. Approve TEMPs for all space MDAPs, all major space system acquisitions (non-major sys-tems as required) and other space programs on OSD T&E Oversight. Forward approved TEMPs to DOT&E and USD(AT&L)/DS. Note: These programs are defined in NSS 03-01, paragraph 3.1.

3.6. Headquarters, Air Force Materiel Command (AFMC). HQ AFMC will:

3.6.1. Develop AFMC DT&E policies, procedures, guidance, and MOAs for non-space programs in assigned mission areas to supplement this AFI. Forward draft copies for HQ USAF/TEP and SAF/AQXA prior to publication.

3.6.2. Ensure nuclear weapon system polices and issues are managed according to AFI 63-103 and AFI 63-125. Assist with development and approval of Nuclear Weapon Subsystem Test Plans.

3.6.3. Establish and provide for DT&E training, organization, and T&E infrastructure resources.

3.6.4. Ensure RDT&E representation at pre-Concept Refinement phase activities to assist in early requirements development, early T&E strategy development, and early acquisition planning ac-cording to AFI 10-601, AFI 63-101, and this AFI. Participate in HPTs. Identify organizations re-sponsible for these activities.

3.6.5. Assist the ITT and PM in identifying government DT&E organizations as RTO candidates as soon as possible after the Concept Decision according to paragraphs 4.4 and 4.5.

3.6.6. Establish policy for and assign T&E focal points to provide support at the air logistics cen-ters (ALC) and product centers.

3.6.7. Conduct long-range planning to ensure T&E infrastructure and processes are in place to support required testing.

3.6.8. Ensure test centers participate in T&E resource investment planning processes.

3.6.9. Ensure ALC and product center PMs oversee the conduct of DT&E and support operational testing of fielded systems throughout the life cycle of the system.

3.6.10. Oversee and inspect AFMC compliance with this instruction.

3.7. Headquarters, Air Force Space Command (AFSPC). HQ AFSPC will:

3.7.1. Develop HQ AFSPC T&E policies, procedures, guidance, and MOAs for space and inter-continental ballistic missile (ICBM) programs to supplement this AFI. Forward draft copies for HQ USAF/TEP and SAF/AQXA prior to publication.

3.7.2. Serve as the focal point for T&E of space launch and ICBM acquisition programs, and tech-nology projects in conjunction with SAF/US.

*3.7.3. Ensure RDT&E representation to pre-Concept Decision activities to assist in early re-quirements and CONOPS development, early T&E strategy development, and early acquisition planning.