AFOTEC_99-101 _Conduct_of_Operational_Test_ _Evaluation.pdf

BY ORDER OF THE COMMANDER AIR FORCE OPERATIONAL TEST

AND EVALUATION CENTER

INSTRUCTION 99-101

1 MAY 2007

Test and Evaluation

CONDUCT OF OPERATIONAL TEST

AND EVALUATION

COMPLIANCE WITH THIS PUBLICATION IS MANDATORY

ACCESSIBILITY: Publications and forms are available for downloading or ordering on the e-Publishing website at www.e-publishing.af.mil/.

RELEASABILITY: There are no releasability restrictions on this publication.

OPR: AFOTEC/XPY Certified by: AFOTEC/CC (Maj Gen Robin E. Scott)

Supersedes AFOTECI 99-103, 1 November 2005 Pages: 36

This instruction issues organizationally specific minimum mandatory policies and procedures for plan-ning, executing, and reporting operational test and evaluation (OT&E) and related activities. This docu-ment is to be used in conjunction with those policies, directives, and instructions contained in Department of Defense (DoD) Directive 5000.1, The Defense Acquisition System; DoD Instruction (DoDI) 5000.2, Operation of the Defense Acquisition System (and National Security Space Acquisition Policy Number 03-01, where applicable); CJCSI 3170.01, Joint Capabilities Integration and Development System;

Chairman of the Joint Chiefs of Staff Instruction 6212.01, Interoperability and Supportability of Informa-tion Technology and National Security Systems; Air Force Policy Directive (AFPD) 99-1, Test and Eval-uation Process; Air Force Instruction (AFI) 99-103, Capabilities Based Test and Evaluation; and Air Force Mission Directive (AFMD) -14, Air Force Operational Test and Evaluation Center (AFOTEC); and the AFOTEC OT&E Guide which outlines the AFOTEC/CC’s processes, procedures, checklists, and tech-niques of the various phases of OT&E defined in this instruction. The AFOTEC OT&E Guide is available electronically from HQ AFOTEC/XPY. It applies to individuals planning, executing, and reporting oper-ational test and evaluation and related activities. Ensure that all records created as a result of processes prescribed in this publication are maintained in accordance with AFMAN 37-123 (will convert to AFMAN 33-363), Management of Records, and disposed of in accordance with the Air Force Records Disposition Schedule (RDS) located at https://afrims.amc.af.mil/. Refer recommended changes and questions about this publication to the OFFICE OF Primary Responsibility (OPR) using the AF IMT 847, Recommendation for Change of Publication; route AF IMT 847s from the field through the appropriate functional’s chain of command. Submit requests for waiver to this instruction to headquarters (HQ) AFO- TEC/XP via the HQ AFOTEC/XP Corporate Account, xpcorp@afotec.af.mil. HQ AFOTEC/XP will administratively coordinate the waiver request with the process owner and the affected AFOTEC staff.

http://www.e-publishing.af.mil/ https://afrims.amc.af.mil/ mailto:xpcorp@afotec.af.mil

2 AFOTECI99-101 1 MAY 2007

SUMMARY OF CHANGES

This instruction has been significantly revised. It is recommended to review it in its entirety. This instruc-tion supersedes AFOTECI 99-103, 1 November 2005. Office symbols have been replaced throughout, reflecting the organizational change of HQ AFOTEC/AS and HQ AFOTEC/ST being subsumed into HQ AFOTEC/XO and HQ AFOTEC/XOR, STZ, and TST being subsumed into HQ AFOTEC/XP. A new OT&E construct has been added that now includes resolution of mission capability. AFOTEC continues to determine effectiveness and suitability. The necessary conditions chart has been renamed to evaluation view (EV) throughout. The evaluation summary (ES) has been added to the Scope/Cost, Test Planning, and Test Reporting processes (this includes use of operational capability (OC) and new rating taxono-mies). New information on client requirements documents (CRD) was added. Guidance on the following areas has been added or updated to this document: information protection; technology projects are now referred to as innovation programs; AFOTEC support to innovation programs taking place at a detach-ment; operational utility assessments (OUA) being the activity for innovation programs; clarifying AFO- TEC program involvement timing; Scope/Cost process changes; critical operational issue (COI) development and use; CRD review by the AFOTEC/CV; multimedia documentation support to OT&E;

OT&E evaluation and rating methodology; IOT&E definition; deletion of operational impact assessment from the vernacular; AFOTEC/CC is the approval authority for pause or stop test events; briefing required to AFOTEC/CC for all reports; interim summary reports used only by direction of AFOTEC/CC; test resource plan (TRP) processing and approval; and updated definition of last test event (LTE).

Chapter 1— OVERVIEW 7

1.1. Introduction

1.2. Philosophy

1.3. AFOTEC Staffing

1.4. Information Protection (IP)

1.5. Terms of Reference

1.6. AFOTEC Test Processes

Figure 1.1. AFOTEC Test Processes

1.7. Program Transfer

1.8. Theory of Constraints (TOC) Program Management (TOC/PM)

1.9. Product Review Process

1.10. Briefings to the AFOTEC/CC

1.11. Briefings to External Organizations/Agencies

1.12. AFOTEC Improvement Process (AIP)

1.13. OSD T&E Oversight List

1.14. Special Interest Items (SII)

1.15. Capabilities-Based Requirements Documents

AFOTECI99-101 1 MAY 2007 3

1.16. Integrated Testing (IT)

1.17. Anatomy of an AFOTEC OT&E Program

1.18. Special Access Programs (SAP)

1.19. Templates and Guides

1.20. Management Information Network (MIN) Structure/Electronic File Structure

1.21. AFOTEC Contractor Considerations

1.22. Operational Risk Management (ORM)

Chapter 2— DISCOVERY 12

2.1. Introduction

2.2. Involvement Decision Package/Briefing

2.3. Rescinding an IO

2.4. Innovation Programs

Chapter 3— SCOPE/COST 15

3.1. Introduction

3.2. Scope/Cost Defined

3.3. Understanding the System and Capability

3.4. Developing Candidate Mission Statement and Critical Operational Issues

3.5. Evaluation View-1

3.6. Evaluation View-2

3.7. Evaluation Summary-1

3.8. Initial Test Design

3.9. Environment, Safety and Occupational Health (ESOH)

3.10. Evaluation Framework

3.11. TRP Updates

3.12. Exercise Support

3.13. Program Cancellation

3.14. Tasking Order Decision Package

3.15. Tasking Order

3.16. Rescinding the TO

3.17. Innovation Programs

4 AFOTECI99-101 1 MAY 2007

Chapter 4— TEST PLANNING 18

4.1. Introduction

4.2. Test Planning Defined

4.3. Capabilities Documentation

4.4. Acquisition Program Documentation

4.5. Test and Evaluation Master Plan/Life Cycle Management Plan

4.6. MIN Certification

4.7. Technical Reviews

4.8. Test Concept Development and Briefing

4.9. Evaluation Summary-2

4.10. General Test Planning Considerations

4.11. Test Planning Process

4.12. Test Team Structure

4.13. Test Team Training

4.14. Integrated DT/OT Planning Considerations

4.15. Operational Test Plan

4.16. Determining Last Test Event (LTE)

4.17. System Contractor Involvement and Data Validation

4.18. Support Memoranda

4.19. TRP Updates

4.20. Unfunded Requirements

4.21. Returning a Program to Discovery or Scope/Cost

4.22. Internal-AFOTEC Coordination of Test Plans

4.23. External-AFOTEC Coordination/Distribution of Test Plans

Table 4.1. OT&E Plan Approval Cycle (External)

4.24. Revision of Test Plans

4.25. AFOTEC-Led MOT&E

4.26. Non-AFOTEC-Led MOT&E

4.27. Certification of System Readiness for OT&E

4.28. Environmental Safety and Occupational Health Certification Board

4.29. Test Readiness Review/Briefing

AFOTECI99-101 1 MAY 2007 5

Chapter 5— TEST EXECUTION 24

5.1. Introduction

5.2. Final Preparation Prior to Test

5.3. Dry-runs

5.4. Resource Management

5.5. Test Execution

5.6. Required Reports While in Execution

5.7. Test Information and Data Sharing with Outside Organizations

5.8. Deficiency Reporting

5.9. Pause/Stop Test

5.10. Restarting after a Pause Test

5.11. Stop Test and Decertification

5.12. Decertification Message

5.13. Recertification After Stop Test

5.14. Joint Reliability and Maintainability Evaluation Team (JRMET) –Test Data Scoring Board (TDSB)

Chapter 6— TEST REPORTING 27

6.1. Introduction

6.2. Test Result Ratings

6.3. Evaluation Summary-3

6.4. Report Timelines

6.5. Interim Summary Report (ISR)

6.6. Letter Reports

6.7. Final Report Formats

6.8. Staffing and Coordinating the Final Report

6.9. Publication of Final Reports

6.10. Distribution of Final Reports

6.11. Final Report Briefings

Chapter 7— TEST CLOSEOUT 30

7.1. OT Closeout

7.2. Test Program Closeout

7.3. Inactivation of Operating Location

6 AFOTECI99-101 1 MAY 2007

7.4. Test Data Disposition

7.5. Funds Closeout Procedures

Attachment 1— GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION 31

Attachment 2— MIN REFERENCE LIST 36

AFOTECI99-101 1 MAY 2007 7

Chapter 1

OVERVIEW

1.1. Introduction. This chapter serves as an overview of AFOTEC’s internal organizational responsibil-ities and outlines AFOTEC’s fundamental processes. Each process owner defines their process and ensures correct procedures are followed. The AFOTEC/CC sets broad policy guidelines and is the approval authority for all changes to policy. This instruction outlines minimum mandatory procedures for those processes that pertain to all operational test (OT) programs. A working knowledge of the AFOTEC organization is assumed. The next six chapters discuss the directives and procedures to be followed and products produced during every OT program. The AFOTEC OT&E Guide provides further detailed infor-mation.

1.2. Philosophy. Consistent with “centralized control decentralized execution,” the HQ will facilitate the detachments’ execution of AFOTEC’s mission (AFMD14). The detachment/CC reports to and is respon-sible to the AFOTEC/CC to ensure quality, on-time products and the test mission is successfully and safely executed. The AFOTEC/CC has designated HQ divisions as staff leads and detachments as opera-tional leads. Specific roles and responsibilities of these leads can be found throughout the AFOTEC OT&E Guide. The core team executes AFOTEC test processes from involvement order (IO) to tasking order (TO). The test team, which is comprised of both detachment and supporting HQ personnel, executes AFOTEC test processes from the issuance of the TO to program closeout.

1.3. AFOTEC Staffing. The operational leads must deliver an accurate, balanced, and complete staff package to the HQ staff lead. The staff lead will complete two-letter and Command Section (CS) coordi-nation. All classified products must comply with DoD 5200.1-R/AFI 31-401, Information Security Pro-gram Management. For classified reports, refer to Executive Order 12958, Classified National Security Information (as amended) and DoD 5200.1-R, Information Security Program. Use DoD 5200.1-PH, DoD Guide to Marking Classified Documents, for definitive guidance on how to mark classified material. If in doubt about any marking requirements, contact HQ AFOTEC/MSJ. Criteria for determining signature authority can be found in the AFOTEC OT&E Guide, chapter 1.

1.4. Information Protection (IP). HQ AFOTEC/TS serves as the HQ point of contact (POC) for IP. The detachment technical advisors and test directors (TD) will execute the guidance. IP involves safeguarding any data/information that potentially reveals United States vulnerabilities, capability gaps, or any data/ information that falls under the Freedom of Information Act (FOIA) law exemptions (privacy act, export control, contractor owned information, etc.). IP includes the correct marking, classification, and handling of AFOTEC- originated products. Refer all outside requests for previously published test plans and reports to HQ AFOTEC/CCYH with information copy to the AFOTEC FOIA officer. Refer all requests from the media to AFOTEC Public Affairs. All requests from private parties that cite, evoke, or imply the Freedom of Information Act are to be referred to the AFOTEC FOIA Office for processing under the FOIA. If a request for information is to be denied (in part or full), the request must be referred to the AFO- TEC FOIA Officer. Additional guidance on information protection/release can be found in the AFOTEC OT&E Guide, chapter 1.

1.5. Terms of Reference. To effectively execute the AFOTEC/CC’s policies and procedures for plan-ning, executing, and reporting OT&E and related activities, a common understanding of terms is essen-

8 AFOTECI99-101 1 MAY 2007

tial. For the purposes of this instruction, the term operational test (OT) activities refer to all OT&E, which includes initial OT&E (IOT&E), qualification OT&E (QOT&E), follow-on OT&E (FOT&E) and multi-service OT&E (MOT&E), as well as operational assessment (OA), early operational assessment (EOA), operational utility evaluation (OUE) and force development evaluation. Operational utility assessments (OUA) are used to provide information for innovation programs. The AFOTEC OT&E Guide, attachment 2, has specific definitions.

1.6. AFOTEC Test Processes. The processes covered by this instruction and their corresponding chap-ters are depicted in Figure 1.1.

Figure 1.1. AFOTEC Test Processes.

1.6.1. Discovery. HQ AFOTEC/XOP executes this process. Discovery consists of: understanding global, theater, and continental United States operations; program discovery/identification; program review; determining AFOTEC’s level of involvement and schedule; initial TRP development, involvement decision briefing (IDB); involvement letter development and coordination; and at the appropriate time, issuing the IO. The core team is established by the issuance of the IO. Discovery is detailed in Chapter 2.

1.6.2. Scope/Cost. HQ AFOTEC/XOS is the execution authority and provides oversight of this pro-cess. The TD is the supported operational lead. Scope/Cost consists of: understanding the system;

developing the EV, ES, and evaluation framework (EF); developing and briefing the initial test design (ITD), ITD approval; TRP refinement, TO development; and TO coordination and approval. Once the TO is approved, the core team transitions into the test team. The test team is comprised of HQ and detachment personnel. Scope/Cost is detailed in Chapter 3.

1.6.3. Test Planning. The test team executes this process and all subsequent processes. Test planning consists of: technical reviews; test concept development; test concept briefing; test plan development;

data management analysis plan (DMAP) development; detailed test procedures (DTP) and TRP refinement; coordination, approval, and updating documents; developing the test readiness review (TRR) briefing; and conducting the TRR. Test planning is detailed in Chapter 4.

1.6.4. Test Execution. Test execution consists of completion of all test events, collection of all test data, and completion of test data analysis and evaluation. Test execution is detailed in Chapter 5.

1.6.5. Test Reporting. Test reporting consists of writing, coordination, briefing, report approval and signing, and publishing. Test reporting is detailed in Chapter 6.

1.6.6. Test Closeout. Test closeout consists of closeout checklist and memorandum completion. Test closeout is detailed in Chapter 7.

1.7. Program Transfer. If a program transfers from one detachment to another, the losing TD will ensure all applicable program transfer checklist items are completed. The detachment will prepare a mem-

AFOTECI99-101 1 MAY 2007 9

orandum for record (MFR) and submit it to HQ AFOTEC/XO. Both the transfer checklist and MFR are available on the templates page on the MIN (see Attachment 2).

1.8. Theory of Constraints (TOC) Program Management (TOC/PM). AFOTEC uses TOC/PM as the specific methodology for executing AFOTEC’s OT&E processes. The TOC aids in applying limited resources in the most effective way to achieve success across all of AFOTEC. AFOTEC uses Concerto© as a software support tool to facilitate execution of OT&E. TOC/PM networks will reflect the tasks required to execute OT&E test processes. The AFOTEC project management office (AFOTEC/CVM) is the office of primary responsibility (OPR) for standardizing, implementation and management/execution.

The AFOTEC OT&E Guide, chapters 1 and 4 have greater detail.

1.9. Product Review Process. HQ AFOTEC/XO executes this process. All test and evaluation strategies (TES), test and evaluation master plans (TEMP)/life cycle management plans (LCMP-formerly known as SAMP-single acquisition management plan), and OT plans and reports will undergo the product review process. The goal of this process is to ensure quality products through efficient review. The staff lead will chair the product review meetings. The detachment/CC will approve the document for submittal for exec-utive coordination prior to AFOTEC/CC approval. For details, refer to the AFOTEC OT&E Guide, attach-ment 1.

1.10. Briefings to the AFOTEC/CC. Action officers will request time on the AFOTEC/CC or CV’s cal-endar upon entering documents into 2-letter coordination, or at least 3 weeks prior to requested briefing date, whichever is earlier. All test program-related documents used in briefings presented to the AFO- TEC/CC (or AFOTEC/CV when delegated), and a listing of briefing participants, will be coordinated through the Commander’s Action Group (CAG) at least 1 week prior to the requested briefing day. The briefing OPR will staff any documents for approval/signature for CS coordination in AFOTEC Electronic Coordination System (AECS) at least 3 days prior to the briefing date.

1.11. Briefings to External Organizations/Agencies. Briefings to external organizations and agencies are staffed similarly to briefings to the AFOTEC/CC. The AFOTEC/CC (or AFOTEC/CV when dele-gated) is the release authority for all external briefings. Video teleconferencing is the preferred media.

Exceptions are approved by the AFOTEC/CC. Per DoDI 5000.2, AFOTEC is required to brief Director, Operational Test and Evaluation (DOT&E) on test concepts for DOT&E oversight programs at least 120 days prior to test start. DOT&E may request a briefing from AFOTEC to accompany the final test plan.

This briefing will be coordinated the same way as an OT concept briefing. Final report briefings will be provided to HQ USAF staff and OSD as requested.

1.12. AFOTEC Improvement Process (AIP). Continuous improvement of AFOTEC’s products and processes is accomplished through the AIP. The AIP website on the Management Information Network (MIN) (Attachment 2) contains lessons learned, product/process improvements, and strategic planning.

As part of the AIP, TDs will review and submit lessons learned as required throughout test processes.

1.13. OSD T&E Oversight List. AFOTEC is the operational test agency (OTA) for all Air Force pro-grams on the OSD T&E oversight list. TDs are responsible for knowing the oversight status of their pro-gram and ensuring the current status is stated on the MIN.

10 AFOTECI99-101 1 MAY 2007

1.14. Special Interest Items (SII). HQ AFOTEC/XOS will provide SII technical support. The TD will ensure DOT&E SIIs are addressed in the AFOTEC ITD, plan, and report. DOT&E policy letters on SIIs (i.e., information assurance, electromagnetic environmental effects, and interoperability) are available on the MIN (Attachment 2). The AFOTEC/CC has an SII to address the impacts of Global Positioning Sys-tem signal loss.

1.15. Capabilities-Based Requirements Documents. AFOTEC supports development of capability requirements documents (initial capabilities document (ICD), capability development document (CDD), capability production document (CPD)) through participation in high-performance teams (HPT) chartered to develop each of these documents. Focus on understanding the required capability to ensure the identi-fied requirements are testable. The program manager (PM)/TD is responsible for HPT participation, depending upon where a program resides within the AFOTEC test processes.

1.15.1. Non-testable Capability Requirements. If capability requirements identified in capabili-ties-based documents are found to be non-testable, the HPT will work to develop testable criteria or standards. Non-testable requirements can take the form of comparisons to legacy systems, require-ments missing threshold or objective values, or threshold or objective values that are not measurable.

1.15.2. Coordinating Capability-Based Documents and HPT Participation Issues. I f c r i t i c a l issues with capability-based requirements documents are identified and cannot be resolved at the action-officer level, HQ AFOTEC/XO will develop, staff, and coordinate a letter to the MAJCOM outlining the critical issue. Critical issues may take the form of non-testable requirements, poorly defined, unfunded, unexecutable, or test related issues driven by acquisition strategies or schedules.

The signature authority for this letter is the AFOTEC/CC.

1.16. Integrated Testing (IT). AFOTEC uses the IT concept throughout the life of a test program. This concept is a key tenet of the United States Air Force agile or capabilities-based acquisition process. HQ AFOTEC/XO and the detachment will represent AFOTEC on integrated test teams (ITT) and participate in combined test force (CTF) activities. These practices allow for a seamless transition from contractor testing, to developmental testing (DT), to integrated DT/OT, to OT&E. Reference paragraph 2.2.5. along with the AFOTEC OT&E Guide, chapters 1 and 4, for information on preparing ITT charters.

1.17. Anatomy of an AFOTEC OT&E Program. The Anatomy of an AFOTEC OT&E Program is a depiction of the typical process employed throughout a traditional OT&E program (AFOTEC OT&E Guide insert). It represents the nominal set of actions that may be required to carry out an OT program.

The anatomy was developed as a tool to provide operational testers insight into the scope and depth of effort an OT may require. Testers should use the anatomy as a baseline to trigger their thought process and design their test activities accordingly. The anatomy contains information on the timing for both internal and external products in relation to the acquisition process and uses this information to ensure adequate lead time to produce the product. It is important to note that the testing process is tailored for each OT and the process as depicted represents all steps that could be performed, but not necessarily need to be per-formed.

1.18. Special Access Programs (SAP). HQ AFOTEC/XOZ is the lead for SAPs. HQ AFOTEC/XOZ maintains control of all SAP-associated activities. It is mandatory that any program contacts or issues that pertain to SAPs are brought to the attention of HQ AFOTEC/XOZ. The AFOTEC/CC provides direction for all HQ AFOTEC/XOZ programs. All normal AFOTEC processes (within security constraints) will be

AFOTECI99-101 1 MAY 2007 11

performed for all SAPs. Coordination and staffing of special access required (SAR) documents and brief-ings within AFOTEC will occur via HQ AFOTEC/XOZ. HQ AFOTEC/XOZ serves as SAP/SAR lead for all HQ functions except for safety (HQ AFOTEC/SE).

1.19. Templates and Guides. AFOTEC templates and guides are provided on the MIN. Template use is mandatory. Templates may be tailored for specific use; however, all content contained in both the tem-plates and briefing guides must be considered for inclusion. The Technical Review Guide is under the pur-view of AFOTEC/TS and information on the technical reviews can be found on the AFOTEC/TS organization web page (Attachment 2). Additional guidance on template usage is provided in the respec-tive sections of the AFOTEC OT&E Guide.

1.20. Management Information Network (MIN) Structure/Electronic File Structure. For each pro-gram, the core/test team is required to publish program case file information to the MIN under the “Doc-umentation” tab. Program case file information is also placed in the electronic file structure (AFOTEC O:\drive). Products on the MIN must be the most current information available. If a product is revised, the old version must be purged from the MIN only. The AFOTEC OT&E Guide, chapter 1, gives further details.

1.21. AFOTEC Contractor Considerations. Detachments and directorates identify technical needs required to perform specific tasks along with test support shortfalls that may exist as the TRP is being developed. Contractor personnel are part of the core/test team, but the PM, TD, or the test team cannot directly task them beyond the scope of the contract task order. There are restrictions on releasing intelli-gence information to government contractors. AFI 14-303, Release of Intelligence to US Contractors, governs such releases, and HQ AFOTEC/TSI manages AFOTEC’s intelligence release program. DD Form 254, DoD Contract Security Classification Specification, on each contract specifies whether the release of intelligence information is authorized for that contract. HQ AFOTEC/TSI will work with the detachment or directorate to ensure all intelligence data necessary for the contractors to complete their obligations is released.

1.22. Operational Risk Management (ORM). Detachments and directorates must employ methods to effectively manage risk in all facets of the test process. ORM has been developed to systematically eval-uate possible courses of action, identify risks and benefits, and determine the best course of action for any given situation. HQ AFOTEC/SE, as a member of the core/test team, will provide guidance, advice and direction on the implementation of ORM as described in AFPD 90-9, Operational Risk Management (AFOTEC OT&E Guide, chapter 1).

12 AFOTECI99-101 1 MAY 2007

Chapter 2

DISCOVERY

2.1. Introduction. HQ AFOTEC/XOP executes the Discovery process. Discovery identifies potential programs early in pre-system acquisition as well as acquisition programs with potential Air Force interest.

HQ AFOTEC/XOP monitors programs and gathers operational information and reviews program status on a regular basis. When involvement taxonomy requirements are satisfied, a program is placed on the MIN (program initiation). The AFOTEC OT&E Guide and the Anatomy of an AFOTEC OT&E Program foldout provide additional information and an example of the process flow.

2.2. Involvement Decision Package/Briefing. If resolution of the involvement taxonomy indicates AFOTEC involvement, HQ AFOTEC/XOP will prepare an involvement decision package. The package includes the involvement letter, involvement decision briefing, and a draft IO. Additionally, the initial TRP and decision criteria checklist will be posted to the MIN. HQ AFOTEC/XOP will schedule and present an IDB (to include an involvement recommendation and proposed IO issuance date) to the AFO- TEC/CC and the gaining detachment/CC. The AFOTEC/CC will make an involvement decision. The involvement decision leads to one of four outcomes: 1) keep the program in Discovery; 2) publish an IL;

3) publish the IL and IO simultaneously, or 4) document noninvolvement. Involvement decisions are doc-umented in the IO and recorded on the MIN. For programs resulting in noninvolvement or that are can-celled, HQ AFOTEC/XOP will archive program information both locally and on the MIN. Additionally, HQ AFOTEC/XOP will update the involvement letter sent to HQ USAF/TE to a noninvolvement deci-sion.

2.2.1. Involvement Letter. The involvement letter formally establishes AFOTEC’s level of involve-ment or noninvolvement in a program. After AFOTEC/CC signature, the letter is sent to HQ USAF/ TEP with copies to the user and developer. The planned level of involvement could be one of the fol-lowing: conduct OT activities to inform specific milestones; participate in MOT&E led by another OTA; monitor program testing activities; or noninvolvement. A noninvolvement decision is coordi-nated with the user (major command (MAJCOM) requirements and MAJCOM test offices) and pro-gram office prior to sending to HQ USAF/TEP. The CAG will provide a copy of the signed letter to

HQ AFOTEC/CCYH.

2.2.2. Involvement Order. HQ AFOTEC/XOP prepares the IO. Template use is mandatory. The IO documents AFOTEC’s timely involvement in a program, establishes the core team, identifies the lead detachment (and respective TD), supporting detachments, and subject matter experts (SME); changes the phase of the program from Discovery to Scope/Cost; starts Scope/Cost process activities, specifies what other type of roles and responsibilities the core team has, and authorizes funds to accomplish Scope/Cost activities. The CAG will provide a copy of all IOs to the History Office (HQ AFOTEC/

CCYH).

2.2.3. Initial Test Resource Plan/Client Requirements Document. The HQ AFOTEC/XO test resource manager (TRM), in conjunction with other AFOTEC offices as required, will initiate the TRP. The TRP is a management and planning document and will evolve throughout a test program to include details on AFOTEC contractor support, range time, flying hours, etc. The HQ AFOTEC/XO TRM will coordinate the initial TRP with the detachment. The initial TRP is a rough order of magni-tude estimate useful for AFOTEC budgeting purposes. The TRP is kept current to reflect maturing resource requirements as the test develops. As a minimum, the TRP will be reviewed and updated

AFOTECI99-101 1 MAY 2007 13

every 6 months. Updates will include reasons for change. TRPs are not coordinated outside AFOTEC until approved by HQ AFOTEC/XPP. For innovation programs, a formal TRP is not prepared; a CRD is used. A CRD is used as a record of agreement between AFOTEC and the client for innovation pro-grams or client-funded test activities. Funds are externally provided IAW the agreements between the executing detachment and the sponsoring agency. It lists the expected level of support and deliver-ables, along with the schedule and funding requirements needed. The CRD is used as a guide for AFOTEC and the client. It is not a contractual, legal, or fiduciary document. For munitions, missile, and target (MMT) forecasting requirements, contact HQ AFOTEC/XPR. The AFOTEC OT&E Guide, chapter 3, provides additional information regarding TRP/CRD and MMTs.

2.2.4. Identification of Core Team Members and Responsibilities. Required core team members include the detachment TD; HQ AFOTEC/XO, TS, XP, SC, SE, and detachment TRMs. External par-ticipation by the system program office (SPO), developer, user, and applicable ITT organizations is encouraged. The TD serves as the operational lead for all core team activities from Scope/Cost through Test Closeout. HQ AFOTEC/XOS executes the Discovery and Scope/Cost processes and is responsible for producing a TO by the date published in the IO. The HQ AFOTEC/TS representative attends all core team meetings, assists HQ AFOTEC/XOS with test design methodologies, assesses a program’s IP requirements, and provides assistance developing realistic threat scenarios. The AFO- TEC/MSCS representative provides information technology support to include computers, communi-cations, multimedia, communities of practice/knowledge management, and information/records management. The HQ AFOTEC/XP representative provides policy guidance and leads identification of major test capability requirements and shortfalls to include potential venues for OT insertion such as exercises and distributed test events. The HQ AFOTEC/SE representative provides environmental, safety, and occupational health (ESOH) guidance. The HQ AFOTEC/XO TRM with support from AFOTEC/RM as required, provides test resource support. While not officially tasked as core team members, HQ AFOTEC/PA supports requests from outside media agencies, and information requests.

HQ AFOTEC/CCYH supports core teams with scientific and technical information research.

2.2.5. Charters/Agreements/Support Memoranda. AFOTEC/CC approves all host-tenant support agreements. AFOTEC/CV approves all memoranda of agreement (MOA) and memoranda of under-standing (MOU). The exception is limited-scope software tools memoranda of agreement, i.e., infor-mation assurance and modeling & simulation (M&S) software tools that do not obligate personnel, facilities, or host-base support. These MOA/MOU require HQ AFOTEC/LC and HQ AFOTEC/MSC, and HQ AFOTEC/XP coordination prior to approval by the appropriate 2-letter. For ITT/CTF char-ters, the core/test team reviews and provides input to the charter/agreement, and completes 2-letter coordination (HQ AFOTEC/XO and HQ AFOTEC/XP). Prior to the TO being issued, HQ AFOTEC/ XO is the signature authority for ITT/CTF charters. Post-TO, the detachment/CC is the signature authority for ITT/CTF charters. Regardless of timing, AFOTEC’s signature authority will match or exceed the document’s highest level of signature). Operating locations (OL) also require an agreement (either an MOA or a host-tenant support agreement, depending on requirements of the host base). HQ AFOTEC/XP will post a copy of all memoranda of agreement on the MIN Policy page, and the TD will maintain a copy of charters in the appropriate MIN program document folder. The AFOTEC OT&E Guide, chapter 4, has specific guidance regarding types of agreements and their coordination/ approval processes.

2.3. Rescinding an IO. Programmatic changes may warrant returning a program to Discovery. Should this occur, a rescind order (RO) is required. The RO must contain sufficient detail and document the

14 AFOTECI99-101 1 MAY 2007

action and justification for the decision. The RO is prepared by HQ AFOTEC/XOO and submitted for staffing using the template found on the MIN (Attachment 2). The RO is approved by the AFOTEC/CC.

The CAG will forward a copy to HQ AFOTEC/CCYH for historical archives. When the IO is rescinded and the program is returned to Discovery, the involvement letter is reaccomplished to reflect the revised level of involvement. The AFOTEC OT&E Guide, chapter 2, outlines additional information on rescind-ing IOs.

2.4. Innovation Programs. Air Force innovation programs addressed by AFPD 10-23, Air Force Inno-vation Programs, include advanced concept technology demonstrations (ACTD), Air Force Battlelab ini-tiatives, Joint and Service experimentation, as well as Joint and Service innovative-centered wargames.

For innovation programs, AFOTEC will use OUAs. These programs are assigned to a detachment based on the SME requirements. AFOTEC involvement in an innovation program is “by request” and follows the innovation program involvement decision taxonomy maintained by HQ AFOTEC/XOP. The AFO- TEC OT&E Guide, chapter 2, outlines specific information regarding innovation programs.

AFOTECI99-101 1 MAY 2007 15

Chapter 3

SCOPE/COST

3.1. Introduction. HQ AFOTEC/XOS is the execution authority and provides oversight to this process.

The TD is the supported operational lead. HQ AFOTEC/XOS and the TD will co-chair all core team meetings. HQ AFOTEC/XOS will ensure standardization of the process across all OT programs. Scope/ Cost consists of: understanding the system; developing the EV, ES, and EF; developing and briefing the ITD; ITD approval; TRP refinement; TO development, coordination and approval. The output of the Scope/Cost process is a TO package that documents the AFOTEC/CC’s broad guidance regarding scope of the program’s evaluation, allocation of resources, and responsibilities. TOC/PM networks are the key tools HQ AFOTEC/XOS will use to ensure project plans are standardized. HQ AFOTEC/XOS is respon-sible for drafting (with support from the core team), coordinating, and attaining final approval of the TO.

The Anatomy of an AFOTEC OT&E Program foldout in the AFOTEC OT&E Guide outlines an example of the process flow.

3.2. Scope/Cost Defined. Scope defines the “extent of” or “how much” testing and the operational value of testing. Cost identifies the anticipated resources associated with the planned testing. The goal of the Scope/Cost process is to provide a standardized approach for the corporate allocation of resources for all programs.

3.3. Understanding the System and Capability. The core team researches and collects capabil-ity-based requirements and system information from various sources to include requirements documents and associated architectures, concept of operations (CONOPS), information support plans (ISP), tactics, techniques and procedures, program office documentation/discussions, and intelligence assessments. This information is necessary to understand where the system fits into the operational environment and to understand the system’s OCs and limitations. The operational significance of the required capabilities must be sufficiently understood to properly scope the evaluation and develop a meaningful EV/ES/EF.

3.4. Developing Candidate Mission Statement and Critical Operational Issues. The delegated HQ representatives will develop a candidate mission statement and COIs prior to the initial core team meet-ing. The core team reviews and refines the mission statement and COIs. For the mission statement, the core team should ask, “Why is this capability being acquired?” or “What operational mission is the sys-tem intended to support?” The mission statement should be consistent with information found in docu-ments such as the ICD, CDD, or CONOPS. For COIs, the core team should ask, “What does the system do to support the mission?” and “What are the critical elements of the mission?” Though phrased as a yes/ no question, AFOTEC reports discuss how well the system accomplishes each COI. For innovation pro-grams, the mission statement is developed from an operational utility perspective and is referred to as an operational utility statement. The AFOTEC OT&E Guide, chapter 3, gives additional information on mis-sion statement and COI development.

3.5. Evaluation View-1. HQ AFOTEC/XOS is responsible for developing the EV-1 with support from the core team and other SMEs. The EV-1 is a pictorial depiction of the battlespace, from a “system-of-sys-tems” view including the Joint and Air Force doctrine mission areas and scenarios a new program is intended to support when fielded. The AFOTEC OT&E Guide, chapter 3, has additional information on developing an EV-1.

16 AFOTECI99-101 1 MAY 2007

3.6. Evaluation View-2. HQ AFOTEC/XOS is responsible for developing the EV-2 with support from the core team and other SMEs. The EV-2 builds upon the operational environment depicted in the EV-1.

The EV-2 maps OCs and SIIs from requirements documents such as the ICD, CDD, CONOPS, and ISP into the battlespace depiction (EV-1). The AFOTEC OT&E Guide, chapter 3, has additional information on developing an EV-2.

3.7. Evaluation Summary-1. The use of the ES-1 is mandatory for all test programs. The core team is responsible for developing the ES-1 with support from SMEs. The ES depicts a summary of the OT&E construct. The ES-1 identifies the mission statement, COIs, OCs, and operational requirements.

3.8. Initial Test Design. The core team develops the ITD. The ITD builds upon the foundation depicted in the EV. It identifies a technically adequate, credible test design, the rationale for the costs of OT, high-cost drivers, and long-lead items. ITD development will culminate with the ITD briefing (developed and given by the detachment TD with core team support and approved by the detachment/CC), which will demonstrate a design that is both a technically adequate and a defendable basis of estimate The AFOTEC OT&E Guide, chapter 3, discusses the ITD in further detail.

3.9. Environment, Safety and Occupational Health (ESOH). The core team will incorporate ESOH factors into the ITD. This includes enhancing mission accomplishment in a safe, environmentally con-scious manner. The TD is ultimately responsible system safety and safety during OT. HQ AFOTEC/SE is the POC for all safety related issues. The AFOTEC OT&E Guide, chapter 4 provides additional informa-tion regarding ESOH.

3.10. Evaluation Framework. HQ AFOTEC/XOS develops the EF. The EF is a working document that captures the core team’s rationale for the test design; it serves as an early test plan. It identifies scope and methodology for testing the requirements, test scenarios, resources, threats, contractor support, etc. The EF also includes the mission statement, COIs, and describes the operational battlespace in which the sys-tem will operate. The core team will ensure SIIs are addressed in the EF. Each section of the EF has an identified OPR who is responsible for completing that section. If required, the Office of Aerospace Stud-ies, Air Force Information Operations Center (AFIOC), and Joint Interoperability Test Command (JITC) should be invited to participate in the development of the EF. The EF template is on the MIN (Attach-ment 2). The EF is a supporting document to the TO, which supports development of the test concept, and can be used to populate the test plan. HQ AFOTEC/XOS and the detachment will verify the currency, refine, and format of the EF prior to inclusion in the TO. The AFOTEC OT&E Guide, chapter 3, outlines additional information on developing this document.

3.11. TRP Updates. Post-IO, HQ AFOTEC/XO TRM retains responsibility for review and update of the TRP created in Discovery. The TRP identifies resources required to support testing and documents the rationale for any changes. It helps formulate the basis for AFOTEC’s budget submissions, program objec-tive memoranda, manpower requirements, and operational test resource procurement lead times. If sup-port from external agencies, such as AFIOC, JITC, and the MAJCOMs is required, it will be included in the TRP. The HQ AFOTEC/XO TRM will staff the TRP to external agencies prior to AFOTEC approval of the TO.

AFOTECI99-101 1 MAY 2007 17

3.12. Exercise Support. HQ AFOTEC/XPR monitors exercises to ascertain if a particular exercise would prove useful for future OT&E activities. An assessment of exercises will be available to support Scope/Cost, the TD, and core team for use in developing test concepts and plans.

3.13. Program Cancellation. If programmatics dictate AFOTEC noninvolvement in a program after an IO has been issued (e.g., a program is cancelled), but before issuance of a TO, HQ AFOTEC/XOS will initiate the closeout process. For program cancellation post TO, reference paragraph 4.21.

3.14. Tasking Order Decision Package. The TD and core team prepares the TO decision package. The TO decision package details the scope of the evaluation, the resource allocations, and establishes the test team and responsibilities during planning, execution, and reporting including the required deliverables.

The TO decision package includes the TO and ITD briefing materials. Additionally the EV, EF, and TRP or CRD will be posted to the MIN. AFOTEC/CV will review CRDs as part of the tasking order decision package coordination and staffing. The AFOTEC OT&E Guide, chapter 3, offers additional information for developing the TO package.

3.15. Tasking Order. HQ AFOTEC/XOS prepares the TO. Template use is mandatory (Attachment 2 for MIN location). HQ AFOTEC/XOS is the HQ staff lead for coordinating the TO. The TO is approved by the AFOTEC/CC. The CAG will forward a copy to HQ AFOTEC/CCYH for historical archives. When more than one detachment is identified as the potential OT lead, HQ AFOTEC/XOS, with core team input, will determine the lead detachment. Post TO, the detachment will lead program scope and cost review efforts with assistance from HQ AFOTEC/XO. Resulting changes to the TO will be staffed by HQ AFOTEC/XO. If a conflict exists between the TO and AFOTEC instructions/policy, the TO will take pre-cedence if the TO states the conflict and the rationale for deviation. If the TO does not address the con-flict, the TD will contact their HQ AFOTEC/XOO representative for assistance for future action.

3.16. Rescinding the TO. Programmatic changes may warrant returning a program to Scope/Cost or Discovery. Should this occur, a rescind order (RO) is required. The RO must contain sufficient detail and document the action and justification for the decision. The RO is prepared by HQ AFOTEC/XOO and submitted for staffing using the template found on the MIN (Attachment 2). The RO is approved by the AFOTEC/CC. The CAG will forward a copy to HQ AFOTEC/CCYH for historical archives. When the TO is rescinded and the program is returned to Scope/Cost or Discovery, the involvement letter to HQ USAF/TE is reaccomplished to reflect the revised level of involvement. The AFOTEC OT&E Guide, chapter 3, outlines additional information on rescinding TOs.

3.17. Innovation Programs. The Scope/Cost process for innovation programs mirrors that of other pro-grams except the process is shorter in duration. The notional timeline from AFOTEC IO to final OUA report is 6 to 9 months. There may be a lengthy period of time between the innovation program establish-ment and the reaching of a mature technology level. The TO package will not be staffed until the technol-ogy readiness level (TRL) reaches a minimum of level five (preferably six). AFOTEC does not accomplish multiple OUAs to help a program mature the technology; AFOTEC only does a single OUA after a program reaches TRL five or six. TRL descriptions are contained in the Defense Acquisition Guidebook.

18 AFOTECI99-101 1 MAY 2007

Chapter 4

TEST PLANNING

4.1. Introduction. The test team executes test planning and all subsequent processes. Test planning begins with the release of the TO and ends with approval to execute test at the TRR. Requests for devia-tions from the TO will be coordinated with HQ AFOTEC/XO and approved by the AFOTEC/CC. HQ AFOTEC/XO will coordinate with HQ AFOTEC/XP if the deviation involves resources or is in conflict with published policy or procedures. The Anatomy of an AFOTEC OT&E Program foldout in the AFO- TEC OT&E Guide outlines an example of the process flow.

4.2. Test Planning Defined. The purpose of the test planning phase is to complete detailed planning of test activities in preparation for the TRR. Test planning consists of: developing a test concept, briefing the test concept, forming the test team, activating units, identifying test capability requirements and short-falls, conducting technical reviews, completing test plan development/coordination/briefing/approval, developing support memoranda, updating the test plan, DTP, DMAP, and TRP updates, preparing the TRR briefing; processing certification of readiness for dedicated OT&E, and conducting the TRR. Pro-grammatics determine which test planning activities are required. The AFOTEC OT&E Guide, chapter 4, discusses specific details.

4.3. Capabilities Documentation. Post TO, the TD will lead all activities and ensure the test team par-ticipates in reviews of the analysis of alternatives (AoA), ICD/CDD/CPD, TRP/CRD, CONOPS, system threat assessment report, and ISP, if applicable. Electronic copies of capabilities documents will be loaded on the program’s documentation page on the MIN. The TD will ensure that the test team is fully aware of the operating/using command’s responsibilities as defined in AFI 10-601, Capabilities Based Require-ments.

4.4. Acquisition Program Documentation. Prior to milestone reviews, the TD will review and provide appropriate input to the program management directive (PMD), TEMP, LCMP, request for proposal (RFP), and other documentation pertinent to the program. Particular attention should be given to sections H (Special Contracts Requirements) and L (Instructions, Conditions, and Notices to Offerors/Bidders) of the RFP to ensure it contains any necessary special clauses for implementing the test plan. Review section M (Evaluation Factors for Award) to ensure it contains evaluation criteria necessary to select a contractor to support the T&E requirement. If the T&E concept specifies an integrated DT/OT approach, provisions must be included for protecting the quality and integrity of system contractor test data for later use during

OT&E.

4.5. Test and Evaluation Master Plan/Life Cycle Management Plan. The TD is responsible for updating the TEMP parts IV and V (or the corresponding sections of the LCMP, as applicable) after the TO is issued. The TEMP focuses on the overall structure, major elements, and objective of the T&E pro-gram to support the acquisition strategy. The TD will ensure an assessment of the linkage between the TEMP and required capabilities. The TD ensures all SIIs are properly addressed in the parts III, IV, and V.

This can be achieved by planning to integrate DT and OT events to the maximum extent possible. The TEMP will undergo the Product Review Process (paragraph 1.9.). If a program does not have a TEMP/ LCMP, the detachment will define the OT requirements through the ITT charter (paragraph 2.2.5.) stipu-

AFOTECI99-101 1 MAY 2007 19

lating completion of the applicable Air Force Manual 63-119 templates as part of the DT exit criteria. The AFOTEC/CC approves all TEMPs/LCMPs.

4.6. MIN Certification. The TD will ensure the MIN reflects accurate program information. The TD will review and update information on the MIN and will certify test program information. The “last certi-fied” date should never be more than 30 days old. The TD is responsible to ensure outdated information or preliminary drafts are properly archived. Changes to a program long/short title require HQ AFOTEC/ XO approval.

4.7. Technical Reviews. The TD will coordinate a technical review date and time with HQ AFOTEC/TS technical advisors and schedule the technical reviews no later than 2 months prior to the test concept briefing and the test plan briefing. Long duration programs may require several reviews. HQ AFOTEC/ TS will conduct technical reviews on all programs to ensure technical adequacy and credibility of the test methodology and the analysis/reporting. HQ AFOTEC/TS is responsible for periodic assessment and doc-umenting that the test team is progressing toward collecting the appropriate level of information to answer relevant questions (COIs, system effectiveness, and suitability). Mandatory attendees from the detach-ment are the TD and the detachment technical advisor. HQ AFOTEC/XOO, XPY, XPR, and SE are invited to the technical reviews and participate as needed. Additional information is available on the HQ AFOTEC/TS Organizational webpage (Attachment 2) and in the AFOTEC OT&E Guide, chapter 4.

4.8. Test Concept Development and Briefing. The test team is responsible for developing the test con-cept and associated briefing.