AFOTEC_OT E_Guide _6th_Edition.pdf

AFOTEC OT&E

Guide

6th Edition

12 February 2009

Refer requests for copies of the OT&E Guide to AFOTEC/A3, 8500 Gibson Blvd SE, Kirtland AFB, NM

87117-5558.

Page Intentionally Left Blank

FOREWORD

The AFOTEC OT&E Guide captures the best practices and corporate knowledge we have gained over the years to effectively and efficiently carry out our core mission. Specifically, the guide:

● Provides organizational-specific information for use across

AFOTEC.

● Outlines the processes, procedures, checklists, and techniques for the various phases of OT&E.

Continual process improvements create new challenges and opportunities. Part of our focus is to minimize the seams inherent in the acquisition process between users, contractors, and developmental and operational testers through early and continuous communication. Early influence with these communities through increased communication, collaboration, and information sharing will minimize the seams and ensure timely delivery of required combat capability to warfighters. We will matrix our OT events with DT activities whenever possible to achieve seamless verification of operational capabilities and make better use of scarce test resources. We must be ready to apply new approaches for situations where the traditional OT model doesn‘t meet the needs of the acquisition community as we have with the Space Test Initiative.

When combatant commanders go to war, they need effective and suitable combat capability for their warriors. We must make sure AFOTEC evaluations and assessments are conducted under operationally realistic conditions. We need to clearly understand the warfighter‘s concept of operations and design an operational test that accurately replicates the envisioned battlespace. Ultimately, our job is to provide key acquisition decision-makers with the decision data they need to make acquisition and fielding decisions. We must accurately and succinctly describe the combat capabilities of weapon systems – both the strengths and the weaknesses – in clear, operational terms. Our reports must be accurate, balanced, complete, and timely.

By using the processes and procedures outlined in the OT&E guide, we will continue to improve our testing effectiveness and remain the world‘s best Operational Test and Evaluation organization. Working together employing our comprehensive guide, we will be better prepared to successfully meet any OT&E challenge. We are the right team, at the right time, poised to achieve a new level of acquisition excellence that will serve our Air Force and our Airmen well into the future.

Stephen T. Sargeant Major General, USAF Commander

AFOTECPAM 99-103

Record of Changes

Change # Date of Change Title of Change

SUMMARY OF CHANGES:

HQ AFOTEC office symbols have been converted to A-Staff symbols and AFOTEC Vice-Commander office symbols; Detachment commanders are no longer the signatory for CTF reports; Detachments are the overall lead for early influence and initial test planning with HQ AFOTEC subject matter experts supporting; the new Air Force space systems test approach guidance has been included in the appropriate chapters; guidance on the use of SIPRNET-MIN for program information filing and final report development; coordination and approval; guidance on two-letter coordination and AFOTEC/CC review of combined test force reports prior to Det/CC release; rapid test planning procedures and guidance has been added; AFMC/AFSPC center test authority MIN pointer added; legacy system information assurance process and procedures deleted; additional guidance and separate attachment added on execution of risk assessments for OT&E of software intensive systems; identification of AFOTEC points of contact for DOT&E and AFOTEC special interest items; guidance on AFOTEC test director relationships with DOT&E and external agencies and clarification on DOT&E determined test adequacy added; MIN pointer added to the A6M Information Assurance Guide; AFOTEC Improvement Process replaced with Product and Process Improvement; AFOTEC Liaison officers role and responsibilities added; a review process for capabilities documents has been added; guidance for integrated test team process and procedures for MOT&Es included; applicable AFI 99-103 OT lead determination guidance added; A5U added as a two-letter reviewer of the A3F involvement decision package; guidance on the development of the TES after the involvement decision; test limitations concept clarified and examples added; A2N roles and responsibilities support to the core team included; new requirement for commander approval on the TRP included;

what AFOTEC pays for during a program‘s T&E updated; signature sequence for MOT&E documents has been added; guidance on tasking order deviations added; A2A9 technical review process updated to include the working relationship between the detachments and the HQ‘s SMEs in preparation; design of experiments use emphasized; specific approval requirements for identified standards added; operating location guidance updated to include need analysis procedures and the requirement to obtain commander approval; test capability accreditation guidance updated;

modeling/simulation guidance updated; Microsoft Enterprise Engineering Center test capability use for OT&E description added; support agreement guidance updated; new requirement for ITT charters to be approved by the commander added and ITT procedures for MOT&Es included; deficiency reporting process and procedures updated to reflect the new AF technical order changes; updated AFMAN 63-119 guidance and templates information updated; the technical editors role in test plan development added; the use of the OT&E construct in operational assessments clarified;

last test event concept clarification added; the latest commander guidance on operational test plan content and format added; commander guidance on communicating OT results with the program office added; the requirement for the commander to approve all status reports has been added; safety operational risk management and SIPRNET communication items for the Det/CC test team deployment checklist updated; test director handling of information requests guidance updated; reporting on system shortfalls only and not operator/unit shortfalls guidance added; guidance on determining and reporting a systems operational effectiveness and operational suitability added; reporting examples updated; program office attendance at report briefings updated; latest commander guidance on OT final report contents not containing recommendations and the process of briefing the commander on test report results until after the report is signed has been added; allocable guidance from the commander on reporting format and contents and the guidance test director and Det/CC‘s on external agency briefing ―structuring for success‖ pointers added; test program close-out guidance updated; product review process information;

coordination figures updated and interim summary report and status report processes added; O-6 representation at external agency plan and report briefings requirements added.

FUTURE CHANGES TO THE OT&E GUIDE:

Any recommendations for future changes to the OT&E Guide can be submitted via an e-mail to AFOTEC/A3E.

i

TABLE OF CONTENTS

Chapter 1 Introduction ..................................................................................... 1-1

1.1 How to Use The OT&E Guide ................................................................... 1-1

1.2 How to Use the Anatomy of an OT&E Program ........................................ 1-2

1.3 Commander‘s Signature Approval Guidance ............................................ 1-2

1.4 DoDI 5000.2 Acquisition Model ................................................................. 1-2

1.5 The Joint Capabilities Integration and Development System .................... 1-3

1.6 Space Systems OT&E ............................................................................... 1-4

1.6.1 Integrated Testing for Space Systems............................................... 1-7

1.6.2 OT&E Early Influence ........................................................................ 1-7

1.6.3 Pre KDP-A Activities .......................................................................... 1-7

1.6.4 Post KDP-A to KDP-B, Concept Development Phase ....................... 1-9

1.6.5 Post KDP-B to KDP-C, Preliminary Design Phase .......................... 1-10

1.6.6 KDP-C to Build Approval, Final Design Phase ................................ 1-11

1.6.7 System Production to OT&E Phase I .............................................. 1-12

1.6.8 Launch Range Compatibility Testing ............................................... 1-13

1.6.9 Launch and Early Orbit Operations, OT&E Phase II ....................... 1-13

1.7 Evolutionary Acquisition .......................................................................... 1-14

1.7.1 Evolutionary Acquisition Characteristics .......................................... 1-14

1.7.2 Test Management in Evolutionary Acquisition ................................. 1-15

1.8 Integrated Testing ................................................................................... 1-16

1.8.1 Integrated Test Teams (ITT) ............................................................ 1-17

1.8.2 Integrated Test Planning and Collaboration .................................... 1-17

1.8.3 Integrated Test Considerations ....................................................... 1-18

1.9 AFOTEC Processes ................................................................................ 1-19

1.10 Rapid Test Considerations .................................................................... 1-20

1.11 Operational Risk Management (ORM) .................................................. 1-21

1.12 Early Influence ....................................................................................... 1-23

1.13 The Core Team ..................................................................................... 1-24

1.14 Ongoing Activities .................................................................................. 1-26

1.14.1 Program-Related Document Reviews ........................................... 1-26

1.14.1.1 Analysis of Alternatives (AoA) ................................................ 1-27

1.14.1.2 Course of Action (COA) ......................................................... 1-27

1.14.1.3 Program Management Directive (PMD) ................................. 1-28

1.14.1.4 Capability-Based Requirements Documents ......................... 1-28

1.14.1.5 Threat Assessment Documents ............................................. 1-31

1.14.1.6 Acquisition Decision Memorandum (ADM) ............................ 1-31

1.14.1.7 Concept of Operations (CONOPS) ........................................ 1-31

1.14.1.8 Security Classification Guide (SCG) ...................................... 1-32

1.14.1.9 Request for Proposal (RFP) ................................................... 1-32

1.14.1.10 Modeling and Simulation (M&S) Support Plan ..................... 1-33

1.14.1.11 Information Support Plans (ISP) .......................................... 1-33

1.14.1.12 Other Useful Documents ...................................................... 1-34

1.14.2 Special Interest Items .................................................................... 1-34

1.14.2.1 Information Assurance ........................................................... 1-35

1.14.2.2 GPS Signal Loss .................................................................... 1-35

1.14.3 Test and Evaluation Strategy ........................................................ 1-36

1.14.4 Test and Evaluation Master Plan (TEMP) ..................................... 1-36

1.14.5 Life Cycle Management Plan ......................................................... 1-36 ii

1.14.6 Establishing a Relationship with DOT&E .......................................1-37

1.14.7 Meeting Attendance .......................................................................1-37

1.14.8 SIPRNet-MIN Requirements ..........................................................1-38

1.14.9 Air Force Manual (AFMAN) 63-119 Responsibilities ......................1-38

1.14.9.1 AFOTEC Implementation of AFMAN 63-119 ......................... 1-40

1.14.9.2 Flexibility ............................................................................... 1-40

1.14.9.3 Communication ..................................................................... 1-41

1.14.10 Lessons Learned .........................................................................1-41

1.14.11 Strategic Planning ........................................................................1-41

1.14.12 Technical and Scientific Support ..................................................1-42

1.14.12.1 Determining the Need for Technical & Scientific Support .... 1-42

1.14.12.2 Determining the Need for Contractor Services .................... 1-42

1.14.12.3 Contracts ............................................................................. 1-42

1.14.12.4 Selecting a Contractor ......................................................... 1-43

1.14.12.5 Contract Task Order ............................................................ 1-43

1.14.12.6 Contract Task Order Management ...................................... 1-43

1.14.12.7 Release of Intelligence Information to Contractors .............. 1-44

1.14.13 Program Support by AFOTEC Dets/Operating Locations (OL) ....1-44

1.14.13.1 Establish a working relationship with program offices ......... 1-44

1.14.13.2 Obtain program information ................................................. 1-45

1.14.13.3 AFOTEC Liaison Officers .................................................... 1-45

1.14.14 Training ........................................................................................1-45

1.14.15 Information Protection ..................................................................1-46

1.14.16 Program Management Theory of Constraints ..............................1-47

Chapter 2 Early Influence .................................................................................2-1

2.1 Introduction ................................................................................................2-1

2.2 Program Identification ................................................................................2-2

2.3 Integrated Concept Teams for Space Systems .........................................2-2

2.4 High Performance Teams ..........................................................................2-3

2.5 Program Initiation .......................................................................................2-3

2.6 Program Updates .......................................................................................2-3

2.7 Involvement Package.................................................................................2-4

2.7.1 Involvement Letter .............................................................................2-4

2.7.2 Initial Tasking Order ...........................................................................2-5

2.8 Initial Test Resource Plan ..........................................................................2-5

2.9 ―Pop-Up‖ Program Funding ........................................................................2-6

2.10 TES and Milestone B TEMP Considerations ...........................................2-6

2.11 Initial Test Design Preparation .................................................................2-8

2.11.1 Understand the Capability Being Acquired ......................................2-9

2.11.2 Developing Recommended Mission Statement and COIs .............2-10

2.11.3 Identifying Operational Capabilities ...............................................2-12

2.11.4 Documenting the OT&E Construct in an Evaluation Summary ......2-12

2.11.5 Developing TES or MS B TEMP Measures ...................................2-13

2.11.6 Developing the Evaluation View ....................................................2-14

2.11.7 Developing the Evaluation View (EV 2) .........................................2-15

2.11.8 Operational Conditions (Factors and Descriptors) .........................2-17

2.11.8.1 Identifying Factors and Descriptors ....................................... 2-18

2.11.8.2 Prioritizing Factors ................................................................. 2-20

2.11.9 Types of Test Design .....................................................................2-21 iii

2.11.9.1 Demonstrations ...................................................................... 2-23

2.11.9.2 Case Designs ........................................................................ 2-23

2.11.9.3 Comparison Designs .............................................................. 2-23

2.11.9.4 Factorial Designs ................................................................... 2-24

2.11.9.5 Combinatorial Designs ........................................................... 2-24

2.11.10 Test Events ................................................................................. 2-24

2.12 Core Team Activities During Early Influence ......................................... 2-26

2.12.1 Review of Preliminary TES/TEMP Products .................................. 2-27

2.12.2 Test Methodology .......................................................................... 2-28

2.12.3 Test Scenarios .............................................................................. 2-28

2.12.4 Test Limitations and Issues ........................................................... 2-29

2.12.4.1 Limitations .............................................................................. 2-30

2.12.4.2 Issues .................................................................................... 2-31

2.12.5 Populate the Initial Test Plan ......................................................... 2-31

2.13 Basis of Estimate ................................................................................... 2-31

2.13.1 Basis of Estimate Content ............................................................. 2-32

2.13.2 Initial Resource Estimates ............................................................. 2-33

2.14 Test Resource Plan (TRP) Update ........................................................ 2-34

2.14.1 Procedures .................................................................................... 2-35

2.14.2 TRP Distribution ............................................................................ 2-36

2.15 Innovation Program Involvement ........................................................... 2-36

Chapter 3 Initial Test Planning ......................................................................... 3-1

3.1 Introduction ................................................................................................ 3-1

3.2 The Core Team During Initial Test Planning.............................................. 3-1

3.3 Validating Previous Initial Test Design Work ............................................. 3-3

3.3.1 Validating Understanding of the Capability Being Acquired............... 3-3

3.3.2 Validating the Evaluation View and the Evaluation Summary ........... 3-4

3.3.3 Validating Operational Conditions and Test Events .......................... 3-5

3.4 Validating the Basis of Estimate ................................................................ 3-6

3.5 Develop Updated Tasking Order Decision Package ................................. 3-6

3.5.1 Populate the Initial Test Plan ............................................................. 3-6

3.5.2 Updated Tasking Order ..................................................................... 3-7

3.5.3 The ITP Briefing ................................................................................ 3-8

3.6 Test Resource Plan (TRP) Update ............................................................ 3-8

3.7 Updated TO Decision Package Approval Procedures ............................... 3-9

3.8 Changes to the Tasking Order ................................................................ 3-10

3.9 Early Multiservice OT&E (MOT&E) Considerations ................................. 3-11

3.10 Innovation Programs ............................................................................. 3-11

Chapter 4 OT Planning .................................................................................... 4-1

4.1 Introduction ................................................................................................ 4-1

4.2 Forming the Test Team ............................................................................. 4-1

4.2.1 Test Team Members and the Test Resource Plan (TRP).................. 4-2

4.2.2 Test Team Selection.......................................................................... 4-2

4.2.2.1 Test Team Positions .................................................................. 4-3

4.2.2.2 Test Team Requisition ............................................................... 4-3

4.2.2.3 MOT&E Test Team Structure .................................................... 4-3

4.2.3 Administrative Support Requirements ............................................... 4-3

4.2.4 MAJCOM-Assigned Test Team Members ......................................... 4-4

4.2.5 Test Team OT&E Training ................................................................. 4-4 iv

4.2.6 Test Team System Training ...............................................................4-6

4.2.7 Real Property, Facilities, Furniture, Communications, and Off-Base

Leases ..............................................................................................4-6

4.3 Initial Test Plan Update ..............................................................................4-7

4.3.1 Evolving the OT&E Construct ..........................................................4-13

4.3.1.1 Reviewing Mission Statements ............................................... 4-15

4.3.1.2 Reviewing Critical Operational Issues ..................................... 4-15

4.3.1.3 Reviewing Operational Capabilities ......................................... 4-16

4.3.1.4 Reviewing Capability Requirements ........................................ 4-16

4.3.1.5 Measures and Supporting Measures ....................................... 4-17

4.3.1.6 Developing Measures of Effectiveness/Suitability ................... 4-23

4.3.1.7 Developing Supporting MOEs/MOSs ...................................... 4-24

4.3.1.8 Identifying Criteria ................................................................... 4-28

4.3.1.9 Developing Identified ―Standards‖ ........................................... 4-28

4.3.1.10 Identifying Data Needs .......................................................... 4-29

4.3.2 OT&E Constructs Within Test Plans ................................................4-29

4.3.3 Test Methodology ............................................................................4-31

4.3.4 Realistic Conditions .........................................................................4-31

4.3.5 Test Events ......................................................................................4-32

4.3.6 Test Scenarios .................................................................................4-32

4.4 Technical Reviews ...................................................................................4-33

4.4.1 Technical Review #1 ........................................................................4-36

4.4.2 Technical Review #2 ........................................................................4-36

4.5 Detachment Test Resource Manager ......................................................4-37

4.6 Presenting the ITP Update for Approval ..................................................4-37

4.7 Test Project Network Update by the Det ..................................................4-38

4.8 Activating an Operating Location .............................................................4-39

4.8.1 Initial Request ..................................................................................4-39

4.8.2 Unit Activation Documents ...............................................................4-40

4.8.3 What Type of Agreement is Required for an OL? ............................4-40

4.8.4 Analyze Requirements and Determine Issues .................................4-41

4.8.5 AECS Coordination ..........................................................................4-42

4.8.6 Activation Process Complete ...........................................................4-42

4.9 Support Agreements ................................................................................4-42

4.9.1 Types of Support Agreements .........................................................4-42

4.9.1.1 Base Support Agreement (DD Form 1144) ............................. 4-43

4.9.1.2 Memorandum of Agreement (MOA) ........................................ 4-43

4.9.1.3 Memorandum of Understanding (MOU) .................................. 4-43

4.9.1.4 Service Level Agreements (SLA) ............................................ 4-44

4.9.1.5 Limited Scope MOAs ............................................................... 4-44

4.9.2 Who Develops, Coordinates, and Approves Support Memoranda ...4-44

4.9.3 What if the Other Unit Does Not Want an agreement ......................4-45

4.9.4 How are Activation, Review, & Termination Dates Determined .......4-45

4.10 Integrated Test Team Charter ................................................................4-45

4.10.1 Integrated Test Team and Combined Test Force Charters ............4-45

4.10.2 ITT/CTF Charter Development and Coordination ..........................4-46

4.11 Test Capabilities ....................................................................................4-46

4.11.1 Test Capability Overview ...............................................................4-46

4.11.2 Test Capability Roadmap ...............................................................4-47

4.11.3 Test Capability Shortfall Matrix ......................................................4-48 v

4.11.4 Determining Range Requirements ................................................ 4-48

4.11.4.1 Range Requirement Identification .......................................... 4-48

4.11.4.2 Range Selection Considerations ............................................ 4-49

4.11.4.3 Test Targets ........................................................................... 4-49

4.11.4.4 Range Safety Considerations ................................................ 4-49

4.11.4.5 Verifying Test Site Access Procedures .................................. 4-50

4.11.4.6 Range Instrumentation Considerations .................................. 4-50

4.11.4.7 Range Data Processing Considerations ................................ 4-50

4.11.4.8 Test Article Instrumentation ................................................... 4-50

4.11.5 OL-NN Responsibilities ................................................................. 4-51

4.11.5.1 NTTR CONOPS ..................................................................... 4-52

4.11.5.2 Additional Information Sources .............................................. 4-53

4.11.6 The Use of Modeling and Simulation (M&S) .................................. 4-54

4.11.7 Use of the Secure Distributed OT&E Control Center (SDOCC) .... 4-55

4.11.8 Microsoft Enterprise Engineering Center ....................................... 4-56

4.11.9 Test Capability Accreditation ......................................................... 4-56

4.11.9.1 Definitions .............................................................................. 4-57

4.11.9.2 Background ............................................................................ 4-57

4.11.9.3 Accreditation Scope ............................................................... 4-58

4.11.9.4 When is Accreditation accomplished ..................................... 4-59

4.11.9.5 Accreditation Procedures ....................................................... 4-59

4.11.9.6 Accreditation References and Resources .............................. 4-62

4.12 Test Plan Development ......................................................................... 4-64

4.12.1 Considerations during T&E Planning ............................................. 4-65

4.12.1.1 Baselining Hardware and Software ........................................ 4-65

4.12.1.2 Survivability ............................................................................ 4-65

4.12.1.3 Contractor Involvement .......................................................... 4-67

4.12.1.4 Operational Test Event Observation. ..................................... 4-67

4.12.1.5 Security .................................................................................. 4-67

4.12.1.6 Environment, Safety and Occupational Health (ESOH) ......... 4-71

4.12.1.7 Suitability ............................................................................... 4-74

4.12.1.8 Software Evaluation ............................................................... 4-79

4.12.2 Operational Utility Assessment (OUA)........................................... 4-80

4.12.2.1 OUAs can be used to: ............................................................ 4-80

4.12.2.2 OUA Planning ........................................................................ 4-81

4.12.3 Operational Assessments (OA) ..................................................... 4-81

4.12.3.1 OA Areas ............................................................................... 4-82

4.12.3.2 OA Plans ................................................................................ 4-85

4.12.4 Operational Assessments for Space Systems ............................... 4-86

4.12.5 Operational Utility Evaluations (OUE)............................................ 4-86

4.12.5.1 OUEs can be used to: ............................................................ 4-86

4.12.5.2 OUEs cannot be used to: ....................................................... 4-87

4.12.5.3 Attributes of OUEs ................................................................. 4-87

4.12.5.4 OUE Planning ........................................................................ 4-87

4.12.6 Operational Utility Evaluations for Space Systems ........................ 4-88

4.12.7 IOT&E/QOT&E/FOT&E Plans ....................................................... 4-88

4.12.7.1 Writing the Plan...................................................................... 4-89

4.12.7.2 Briefing the Plan..................................................................... 4-90

4.12.7.3 Finalizing the Plan.................................................................. 4-90

4.12.7.4 Plan Publishing and Distribution ............................................ 4-91 vi

4.12.7.5 Multiservice Plans ................................................................. 4-91

4.12.7.6 Attachments to the OT&E Plan ............................................. 4-92

4.12.7.7 Reviewing the OT&E Plan ..................................................... 4-92

4.13 Determine Last Test Event (LTE) ..........................................................4-93

4.14 Data Collection and Analysis .................................................................4-94

4.14.1 Data Management and Analysis Plan (DMAP) ..............................4-94

4.14.1.1 DMAP Content ...................................................................... 4-94

4.14.1.2 Data Injectors/Test Tools ...................................................... 4-94

4.14.1.3 Integrated Data Requirements List (IDRL) ............................ 4-95

4.14.2 Questionnaires ...............................................................................4-96

4.14.3 Detailed Test Procedures (DTP) ....................................................4-96

4.15 Deficiency Reporting, Investigation and Resolution Process .................4-98

4.15.1 Deficiency Reporting Procedures ..................................................4-99

4.15.1.1 Joint Deficiency Reporting System (JDRS) ........................... 4-99

4.15.1.2 Originator ............................................................................ 4-100

4.15.1.3 Originating Point .................................................................. 4-101

4.15.1.4 Action Point ......................................................................... 4-101

4.15.1.5 Watch Item Tracking (WIT).................................................. 4-101

4.15.2 Test Director Responsibilities in the DR Process .........................4-102

4.15.3 T&E DR Board (DRB) ..................................................................4-102

4.15.4 Material Improvement Project Review Board (MIPRB) Process ..4-102

4.15.5 DR Prioritization ...........................................................................4-103

4.15.5.1 DR Prioritization Techniques ............................................... 4-103

4.15.5.2 Deficiency Analysis Ranking Technique (DART) Process ... 4-103

4.16 Test Readiness Review (TRR) ............................................................4-105

4.16.1 Certification of System Readiness for OT&E ...............................4-106

4.16.2 Certification Acknowledgement Memo .........................................4-107

4.16.3 Acknowledgement Memo Contents .............................................4-107

4.16.4 Visual Information Incorporation (VIDOC) ....................................4-108

Chapter 5 OT Execution ...................................................................................5-1

5.1 OT Execution Introduction .........................................................................5-1

5.2 Final Preparation .......................................................................................5-1

5.2.1 TD OT Execution Preparation ............................................................5-2

5.2.1.1 Personnel .................................................................................. 5-3

5.2.1.2 Establish Guidelines .................................................................. 5-3

5.2.1.3 Establishing External Points of Contact ..................................... 5-4

5.2.2 Mission Management .........................................................................5-4

5.3 Deployment Checklist ................................................................................5-5

5.4 Resource Management During Test ..........................................................5-6

5.4.1 Funds Management ...........................................................................5-6

5.4.2 Government Purchase Card (GPC) ...................................................5-6

5.5 Technical Services .....................................................................................5-6

5.6 Test Range Utilization ................................................................................5-7

5.7 Dry Runs ....................................................................................................5-7

5.8 OT&E Reporting During Test .....................................................................5-7

5.8.1 Situation Reports ...............................................................................5-8

5.8.2 Status Report .....................................................................................5-8

5.8.3 Status Reports for Space Systems ....................................................5-8

5.8.4 Significant Event Report ....................................................................5-9 vii

5.9 Pausing or Stopping the OT&E ............................................................... 5-10

5.9.1 Pausing the OT&E ........................................................................... 5-10

5.9.1.1 Decision Timing ....................................................................... 5-10

5.9.1.2 Pause Considerations .............................................................. 5-11

5.9.1.3 Restarting after a Pause .......................................................... 5-11

5.9.2 Stop Test and Decertification Actions .............................................. 5-11

5.9.3 Recertification .................................................................................. 5-12

5.10 Data Collection, Management, and Evaluation ...................................... 5-12

5.10.1 Data Management ......................................................................... 5-12

5.10.2 Raw Data Collection ...................................................................... 5-14

5.10.3 Data Collection Forms ................................................................... 5-14

5.10.4 Data Quality Assurance ................................................................. 5-15

5.10.5 Data Quality Control ...................................................................... 5-16

5.10.6 Data Evaluation ............................................................................. 5-16

5.10.7 Organizing Data Collected ............................................................. 5-16

5.10.8 Marking Data for Release .............................................................. 5-17

5.10.9 Data Filing and Storage ................................................................. 5-17

5.10.10 Data Processing, Reduction and Analysis ................................... 5-19

5.10.11 Impact of System Modifications ................................................... 5-19

5.10.12 Test Data Disposition .................................................................. 5-20

5.10.13 Visual Information Documentation (VIDOC) ................................ 5-20

5.11 Joint Reliability and Maintainability Evaluation Team (JRMET) ............. 5-20

5.12 Test Data Scoring Board (TDSB) .......................................................... 5-21

5.13 Release of Test Information .................................................................. 5-21

5.13.1 Test Information Sharing ............................................................... 5-22

5.13.2 Handling Information Requests ..................................................... 5-23

5.13.3 Information Requests on Sensitive Programs ............................... 5-23

5.13.4 Public Affairs Office ....................................................................... 5-25

5.13.5 Information Requests for Published Plans/Reports ....................... 5-25

5.14 Last Test Event ..................................................................................... 5-25

Chapter 6 OT Reporting .................................................................................. 6-1

6.1 OT Reporting Introduction ......................................................................... 6-1

6.2 Reporting OUAs ........................................................................................ 6-1

6.2.1 Operational Assessment of Results for OUAs ................................... 6-1

6.2.1.1 Assessment of Measures for OUAs ........................................... 6-2

6.2.1.2 Assessment of COIs for OUAs .................................................. 6-3

6.2.1.3 Assessing Operational Utility for OUAs...................................... 6-4

6.2.2 Ratings for OUAs .............................................................................. 6-5

6.2.2.1 Rating Supporting Measures for OUAs ...................................... 6-6

6.2.2.2 Rating Measures for OUAs ........................................................ 6-7

6.2.2.3 Rating COIs for OUAs ................................................................ 6-9

6.2.2.4 Rating Operational Utility for OUAs.......................................... 6-11

6.3 Reporting EOAs and OAs........................................................................ 6-12

6.3.1 Operational Assessment of Results for EOAs and OAs .................. 6-12

6.3.1.1 Assessment of Measures for EOAs/OAs ................................. 6-13

6.3.1.2 Assessment of COIs for EOAs/OAs ......................................... 6-15

6.3.2 Ratings for EOAs/OAs ..................................................................... 6-16

6.3.2.1 Rating Supporting Measures for EOAs and OAs ..................... 6-17

6.3.2.2 Rating Measures for EOAs and OAs ....................................... 6-18 viii

6.3.2.3 Rating COIs for EOAs and OAs .............................................. 6-19

6.3.2.4 Rating Assessment Area 2 ...................................................... 6-21

6.4 Reporting OUEs and OT&E .....................................................................6-21

6.4.1 Operational Evaluation of Results for OUEs and OT&E ..................6-22

6.4.1.1 Evaluating Measures for OUEs and OT&E ............................. 6-23

6.4.1.2 Evaluation of COIs for OUEs and OT&E ................................. 6-25

6.4.1.3 Determination of Effectiveness and Suitability for OUEs and

OT&E ..................................................................................... 6-26

6.4.1.4 Evaluation of Mission Capability for OUEs and OT&E ............ 6-27

6.4.2 Ratings for OUEs and OT&E ...........................................................6-28

6.4.2.1 Rating Supporting Measures for OUEs and OT&E.................. 6-29

6.4.2.2 Rating Measures for OUEs and OT&E .................................... 6-31

6.4.2.3 Rating COIs for OUEs and OT&E ........................................... 6-33

6.4.2.4 Rating Effectiveness and Suitability ........................................ 6-35

6.4.2.5 Rating Mission Capability for OUEs and OT&E ....................... 6-36

6.5 Evaluation Summary – Results (ES 3) .....................................................6-37

6.6 Interim Summary Report (ISR) ................................................................6-37

6.7 Final Report Formats ...............................................................................6-39

6.8 Letter Reports ..........................................................................................6-39

6.9 Finalizing the Report ................................................................................6-39

6.10 Report Briefings .....................................................................................6-41

6.11 Staffing the Final Report ........................................................................6-42

6.11.1 Technical Advisor Role ..................................................................6-42

6.11.2 Technical Editor Role .....................................................................6-43

6.11.3 Report Medium ..............................................................................6-43

6.11.4 Report Timeline Exceptions ...........................................................6-43

6.11.5 Final Report Publishing ..................................................................6-43

Chapter 7 OT Closeout ....................................................................................7-1

7.1 OT Closeout Introduction ...........................................................................7-1

7.1.1 Test Program Closeout ......................................................................7-1

7.1.2 Inactivation of Operating Location .....................................................7-2

7.2 Test Data Disposition .................................................................................7-2

7.3 Funds Closeout Procedures ......................................................................7-2

.... Attachment A - Coordination Process & Document Reviews ....................... A-1

.... Attachment B - Glossary .............................................................................. B-1

.... Attachment C – Acronym List ...................................................................... C-1

.... Attachment D – MIN/SIPRNET-MIN Reference List .................................... D-1

.... Attachment E – Test Program Case File ..................................................... E-1

.... Attachment F – Software Intensive System Guidelines ............................... F-1

.... Attachment G – Integrated Test Planning Considerations ........................... G-1 ix

TABLE OF FIGURES

Figure 1.1. DoDI 5000.2 Model ......................................................................... 1-3 Figure 1.2. The JCIDS Process ........................................................................ 1-4 Figure 1.3. NSS 03-01 Comparison .................................................................. 1-5 Figure 1.4. Pre-KDP-A Activites........................................................................ 1-8 Figure 1.5. KDP-A to KDP-B Activities .............................................................. 1-9 Figure 1.6. KDP-B to KDP-C Activities ........................................................... 1-10 Figure 1.7. KDP-C to Build Approval Activities ............................................... 1-11 Figure 1.8. Build Approval to Consent to Ship Activities ................................. 1-12 Figure 1.9. Launch and Early Orbit Operations Activities ............................... 1-13 Figure 1.10. The DoD Requirements and Acquisition Process ....................... 1-14 Figure 1.11. Notional Timeline of AFOTEC Products ..................................... 1-19 Figure 1.12. OT&E Risk Points ....................................................................... 1-20 Figure 1.13. Rapid Test Process .................................................................... 1-21 Figure 1.14. ORM Relationship to the AFOTEC Process ............................... 1-22 Figure 1.15. Capability Requirements Review Process. ................................. 1-29 Figure 2.1. Early Influence Overview ................................................................ 2-1 Figure 2.2. Evaluation Summary (ES-1) ......................................................... 2-13 Figure 2.3. Example OT&E Construct Table ................................................... 2-14 Figure 2.4. Evaluation View (EV 1) ................................................................. 2-15 Figure 2.5. Mapping Capability Requirements and SII (EV 2) ........................ 2-16 Figure 3.1. Initial Test Planning Overview ........................................................ 3-1 Figure 4.1. OT Planning Overview .................................................................... 4-1 Figure 4.2. Evolving The OT&E Construct ...................................................... 4-14 Figure 4.3. OT&E Evaluation Structure ........................................................... 4-15 Figure 4.4. Types of Measures and Data. ....................................................... 4-18 Figure 4.5. Evaluation Summary (ES 2) ......................................................... 4-24 Figure 4.6. OL-NN Timeline (time in months) ................................................. 4-53 Figure 4.7. Accreditation Process ................................................................... 4-60 Figure 4.8. Sample Accreditation Plan/Report Format .................................... 4-63 Figure 4.9. Survivability Assessment .............................................................. 4-66 Figure 4.10. Operational Assessment Areas .................................................. 4-83 Figure 4.11. Data Item Description Fields ....................................................... 4-96 Figure 5.1. OT Execution Overview .................................................................. 5-1 Figure 5.2. Example Deployment Checklist ...................................................... 5-5 Figure 6.1. Test Reporting Overview ................................................................ 6-1 Figure 6.2. Innovation Program Assessment Logic Flow .................................. 6-2 Figure 6.3. Rating OUAs for Innovation Programs............................................ 6-6 Figure 6.4. COI Ratings for OUAs .................................................................. 6-10 Figure 6.5. Operational Utility Ratings ............................................................ 6-11 Figure 6.6. Assessment Taxonomy Logic ....................................................... 6-13 Figure 6.7. Rating OAs for Acquisition Programs ........................................... 6-16 Figure 6.8. COI Ratings for EOAs and OAs .................................................... 6-20 Figure 6.9. OA Area 2, Test Planning and Documentation Ratings ................ 6-22 Figure 6.10. OA Area 2, System Design and Performance Ratings ............... 6-22 Figure 6.11. OA Area 2, Test Assets and Support Ratings ............................. 6-23 Figure 6.12. Evaluation Taxonomy Logic ........................................................ 6-24 Figure 6.13. Effectiveness and Suitability Relationship .................................. 6-28 Figure 6.14. Ratings for OUE and I/Q/F&M OT&E.......................................... 6-29 x

Figure 6.15. Measure Rating Exception ......................................................... 6-33 Figure 6.16. COI Rating Definitions for OUE and I/Q/F&M OT&E .................. 6-34 Figure 6.17. Effectiveness and Suitability Ratings ......................................... 6-35 Figure 6.18. MC Ratings for OUE and I/Q/F&M OT&E .................................. 6-37 Figure 6.19. Evaluation Summary (ES 3) ....................................................... 6-38 Figure 6.20. Sample Letter Report ................................................................. 6-40 Figure 7.1. Test Closeout Overview ................................................................. 7-1 Figure A1. Non-MOT&E Product Review Process for Plans and Reports ........ A-6 Figure A2. MOT&E Product Review Process ................................................. A-12 Figure A3. Product Review Process for ISRs and Status Reports. ................ A-15

TABLE OF TABLES

Table 1.1. Specific Core Team Member Responsibilities ............................... 1-25 Table 1.2. Functional Areas of Expertise ....................................................... 1-26 Table 1.3. AFOTEC POCs for SIIs ................................................................. 1-34 Table 1.4. AFOTEC SIPRNet-MIN Program Documentation .