AFTECMAN99-101.pdf

Operational Test Processes and Procedures

Publications and Forms Management

Test Process Products AFOTEC Intranet Location

DEPARTMENT OF THE AIR FORCE

OFFICE OF THE COMMANDER

AIR FORCE OPERATIONAL TEST AND EVALUATION CENTER

KIRTLAND AIR FORCE BASE NEW MEXICO

Test Process Products AFOTEC Intranet Location

Standard Work for Initial Test Design TC Standard Work Test Process Sustainment.

Folder Name Sample Titles of Document

Folder Name Sample Titles of Document

Management of Records

BY ORDER OF THE COMMANDER

AIR FORCE OPERATIONAL TEST AND

EVALUATION CENTER

AIR FORCE OPERATIONAL TEST AND

EVALUATION CENTER MANUAL 99-101

6 AUGUST 2010

Test and Evaluation

OPERATIONAL TEST PROCESSES AND

PROCEDURES

COMPLIANCE WITH THIS PUBLICATION IS MANDATORY

ACCESSIBILITY: Publications and forms are available for downloading or ordering on the e- Publishing website at www.e-Publishing.af.mil.

RELEASABILITY: There are no releasability restrictions on this publication.

OPR: AFOTEC/A-3

Supersedes: AFOTECI 99-101, 1 May

2007; AFOTEC OT&E

Guide, 12 Feb 2009; AFI 99- 103 AFOTECGM 1, 27 Jul

09; AFOTECGM 99-1, 10

Sep 09; and OT&E Information Files: 0810, 0817, 0902, 0903, 0905, 0906, 0907, 0908, 0909, 0910, 0911, 0912, 0913, 0914, 0915, 0916, 0917, 0919, 0920, 0920(1), 0921, 0922, 0923, 0924, 0925, 0926, 0927, 0928, 0929, 0930, and 1001.

Certified by: AFOTEC/A-3 Col David C.

Gould II)

Pages: 281

The AFOTECMAN 99-101 issues organizationally specific minimum mandatory policies and procedures for planning, executing, and reporting operational test and evaluation (T&E) (OT&E) and related activities. The AFOTECMAN 99-101 is to be used in conjunction with policies, directives, and instructions contained in Department of Defense (DoD) Directive (DoDD) 5000.01, The Defense Acquisition System; DoD Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System; Chairman of the Joint Chiefs of Staff (CJCS) Instruction (CJCSI) 3170.01G, Joint Capabilities Integration and Development System; CJCSI 6212.01E, Interoperability and Supportability of Information Technology and National Security Systems;

Air Force Policy Directive (AFPD) 99-1, Test and Evaluation Process; Air Force Instruction (AFI) 99-103, Capabilities Based Test and Evaluation; and Air Force Mission Directive 14, Air

2 AFOTECMAN 99-101 6 August 2010

Force Operational Test and Evaluation Center. This manual assumes a fundamental understanding of the DoD and Air Force weapon system acquisition processes. This manual does not repeat higher headquarters policy and direction and the higher headquarters guidance should be consulted first to understand the roles and missions of AFOTEC. The AFOTECMAN 99-101 outlines the AFOTEC Commander’s (AFOTEC/CC) processes, procedures, checklists, and techniques of the various phases of OT&E. The manual applies to all AFOTEC directorates, detachments, and operating locations. Ensure that all records created as a result of processes prescribed in this publication are maintained in accordance with Air Force Manual (AFMAN) 33-363, Management of Records, and disposed of in accordance with Air Force Records Information Management System (AFRIMS) Records Disposition Schedule (RDS) located at https://www.my.af.mil/gcss-af61a/afrims/afrims/. Additionally, if the publication generates reports, all applicable Reports Control Numbers must be in accordance with AFI 33-324, The Information Collections and Reports Management Program. Refer recommended changes and questions about this publication to Operations Directorate (A-3) using the AF Form 847, Recommendation for Change of Publication; route AF Forms 847 from the field through the appropriate chain of command. This publication may not be supplemented. Compliance with this publication is mandatory for all AFOTEC personnel.

WAIVER OF DIRECTIVES, INSTRUCTIONS, OR PUBLICATIONS

Submit requests for waiver to the AFOTECMAN 99-101 to the A-3 Corporate Account, AFOTEC.A3.workflow@kirtland.af.mil. A-3 will administratively coordinate the waiver request with the process owner and the affected AFOTEC staff.

SUMMARY OF CHANGES

This Manual is a completely new document and must be reviewed in its entirety.

Chapter 1—INTRODUCTION 10

Section 1A—Overview 10

1.1. Purpose

Figure 1.1. AFOTECMAN 99-101 Overview

Section 1B—DoD and Joint Chiefs of Staff (JCS) Processes 11

1.2. DoDI 5000

Figure 1.2. DoDI 5000

1.3. The Joint Capabilities Integration and Development System (JCIDS)

Figure 1.3. The JCIDS Process

1.4. Evolutionary Acquisition

Figure 1.4. The DoD Requirements and Acquisition Process

Section 1C—AFOTEC Processes 15

AFOTECMAN 99-101 6 August 2010 3

1.5. Early Influence

1.6. IDT/OT

1.7. Space Systems OT&E

Figure 1.5. Space Interim Guidance

Table 1.1. Space Test Approach Benefits

1.8. Establishing a Relationship with DOT&E

1.9. Program Support by AFOTEC Liaison Officers (LNO)

1.10. Staffing and Coordination

1.11. Meeting Attendance

1.12. External Briefings

Table 1.2. External Briefing Checklist

1.13. Information Protection

Table 1.3. POCs for Information Requests

1.14. AFOTEC SIPR-Intranet Requirements

Table 1.4. Program Document Storage

Table 1.5. AFOTEC SIPR-Intranet Program Documentation

1.15. AFOTEC-Intranet

1.16. Training

1.17. AFOTEC HQ Standardization and Evaluation (Stan/Eval)

1.18. Lessons Learned

1.19. Technical and Scientific Support

1.20. Operational Risk Management (ORM)

1.21. Rapid Test Considerations

Figure 1.6. OT&E Risk Points

Figure 1.7. Rapid Test Process

Chapter 2—- AFOTEC ACTIVITIES SUPPORTING MS A 34

Section 2A—Overview 34

2.1. General

Section 2B—External Support Documents 35

2.2. Ongoing Activities

2.3. ICD

2.4. Acquisition Decision Memorandum (ADM)

4 AFOTECMAN 99-101 6 August 2010

2.5. CONOPS

2.6. AoA

2.7. COA

2.8. M&S Support Plan

2.9. ISP

2.10. LCMP

2.11. Integrated Program Summary (IPS) for Space System Acquisitions

Section 2C—Processes, Procedures and Products. 37

2.12. Program Identification

2.13. Program Initiation

2.14. Involvement Package

Table 2.1. Involvement Package Staffing

Table 2.2. Functional Areas of Expertise

2.15. Initial Test Resource Plan

2.16. Pre-MS A Activities for Space Acquisition

Figure 2.1. Pre- MS A Activities

2.17. HPT

2.18. ITT

Table 2.3. ITT/CTF Charter Areas of Interest

2.19. Certification of Readiness for OT&E

2.20. The Core Team During Initial Test Design

2.21. ITD

Figure 2.2. EV 1

Figure 2.3. Mapping Capability Requirements and SII (EV 2)

Table 2.4. Example COIs

Table 2.5. Operational Capability Areas

Figure 2.4. Example ESC-1

Table 2.6. Example COIs

Table 2.7. COI – Recover Isolated Personnel

Table 2.8. COI – Platform Generation

Table 2.9. Additional Factors

Table 2.10. COI – Recover Isolated Personnel

Table 2.11. Factor Prioritization

AFOTECMAN 99-101 6 August 2010 5

Table 2.12. Test Design Purposes

Table 2.13. COI – Recover Isolated Personnel – Test Events

Table 2.14. COI – Recover Isolated Personnel – Test Events

Table 2.15. COI – Platform Generation – Test Events

Table 2.16. Examples of Limitations

Table 2.17. Examples of Issues

Table 2.18. Basis of Estimate Contents

Table 2.19. Operational T&E Support Costs

2.22. Updated TO

2.23. TRP Update

2.24. TES

Figure 2.5. Example OT&E Construct Table

Section 2D—Activities and Events. 71

2.25. Space Acquisition EOA for MS A

Chapter 3—- AFOTEC ACTIVITIES SUPPORTING MS B 73

Section 3A—Overview 73

3.1. General

Section 3B—External Support Documents 73

3.2. ADM

3.3. CDD

3.4. AoA

3.5. Threat Assessment Documents

3.6. Security Classification Guide (SCG)

3.7. M&S Support Plan

3.8. ISP

3.9. LCMP

3.10. RFP

3.11. IPS for Space System Acquisitions

Section 3C—Processes, Procedures and Products. 76

3.12. Concept Development Phase for Space Acquisitions

Figure 3.1. MS A to MS B Activities

3.13. HPT

6 AFOTECMAN 99-101 6 August 2010

3.14. ITT/CTF Charter Review, Update and Coordination

3.15. Validating Previous ITD Work

3.16. TEMP for MS B

Figure 3.2. Example OT&E Construct Table

3.17. OT Considerations Prior to MS B

Section 3D—Activities and Events. 83

3.18. EOA

Chapter 4—- AFOTEC ACTIVITIES SUPPORTING MS C 84

Section 4A—Overview 84

4.1. Introduction

Section 4B—External Support Documents 84

4.2. ADM

4.3. CPD

4.4. CONOPS

4.5. Program Management Directive (PMD)

4.6. AoA

4.7. Threat Assessment Documents

4.8. M&S Support Plan

4.9. LCMP

4.10. ISP

4.11. IPS for Space System Acquisitions

Section 4C—Processes, Procedures and Products. 85

4.12. Preliminary Design Phase for Space Acquisitions

Figure 4.1. MS B to MS C Activities

4.13. HPT

4.14. ITT/CTF Charter Development and Coordination

4.15. Forming the Test Team

4.16. Activating an OL

4.17. Support Agreements

4.18. Test Concept Development

Figure 4.2. Evolving the OT&E Construct

Figure 4.3. OT&E Evaluation Structure

AFOTECMAN 99-101 6 August 2010 7

Figure 4.4. Metric – Threshold Relationship

Figure 4.5. Types of Measures and Data

Figure 4.6. Quantitative – Objective

Figure 4.7. Quantitative – Subjective

Figure 4.8. Qualitative – Subjective

Figure 4.9. Qualitative – Objective

Figure 4.10. Quantitative – Qualitative – Subjective

Figure 4.11. Example ESC-2

4.19. Technical Review

4.20. Presenting the Test Concept for Approval

4.21. Test Capabilities

Figure 4.12. OL-NN Timeline (time in months)

Table 4.1. Test Capability Accreditation Examples

Figure 4.13. Accreditation Process

Table 4.2. Summary of VV&A Responsibilities

4.22. ESOH

4.23. OT Plan Development

4.24. DMAP

4.25. DTP

4.26. DRI&R

4.27. Determine Last Test Event (LTE)

4.28. Visual Information Documentation (VIDOC)

4.29. Technical Review #2

4.30. TEMP

4.31. IOT&E/QOT&E/FOT&E Plans

Section 4D—Activities and Events. 127

4.32. OA

4.33. OUEs

Chapter 5—- AFOTEC ACTIVITIES SUPPORTING FRP/IOC/FIELDING DECISION 129

Section 5A—Overview 129

5.1. Introduction

Section 5B—External Support Documents 129

8 AFOTECMAN 99-101 6 August 2010

5.2. ADM

5.3. IPS for Space System Acquisitions

Section 5C—Processes, Procedures and Products. 129

5.4. Final Design Phase thru OT&E for Space Acquisitions

Figure 5.1. MS C to Build Approval Activities

Figure 5.2. Build Approval to Consent to Ship Activities

Figure 5.3. Launch and Early Orbit Operations Activities

5.5. Finalizing the IOT&E Plan

5.6. TEMP

5.7. ESOH

5.8. TRR/Certification

Section 5D—Activities and Events. 133

5.9. OAs for Space Systems

5.10. OUEs

5.11. IOT&E Execution

Chapter 6—- AFOTEC ACTIVITIES FOLLOWING FRP/IOC/FIELDING DECISION 135

6.1. Introduction

6.2. FOT&E Criteria

6.3. Closeout

Table 6.1. OT Closeout/Change Activities

6.4

6.4.1. Adopted Forms:

Attachment 1—GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION 137

Attachment 2—REVIEW OF CAPABILITY-BASED REQUIREMENTS DOCUMENTS 175

Attachment 3—TEST RESOURCE PLAN (TRP) 178

Attachment 4—ENVIRONMENT, SAFETY AND OCCUPATIONAL HEALTH (ESOH) 181

Attachment 5—TEST PLAN DEVELOPMENT 183

Attachment 6—DEFICIENCY REPORTING 200

Attachment 7—TEST EXECUTION 205

Attachment 8—DATA COLLECTION, MANAGEMENT, AND EVALUATION 213

Attachment 9—RELEASE OF TEST INFORMATION 221

AFOTECMAN 99-101 6 August 2010 9

Attachment 10—TEST REPORT DEVELOPMENT 225

Attachment 11—COORDINATION PROCESS 251

Attachment 12—SOFTWARE GUIDELINES 264

Attachment 13—TEST PROGRAM CASE FILE 268

Attachment 14—AFOTEC-INTRANET REFERENCES 270

Attachment 15—TRR AND SYSTEM CERTIFICATION 272

Attachment 16—SPECIAL INTEREST ITEMS (SII) 276

Attachment 17—ACTIVATION AND INACTIVATION OF OPERATING LOCATIONS 279

10 AFOTECMAN 99-101 6 August 2010

Chapter 1

INTRODUCTION

Section 1A—Overview

1.1. Purpose. The intent of the manual is to provide AFOTEC personnel with mandatory “how to” guidance on accomplishing operational test in support of the Air Force acquisition process (see Figure 1.1.). The processes, procedures, checklists, and techniques outlined in the following chapters are a collection of lessons learned and best practices for conducting an operational test program. AFOTEC/CC, Detachment Commanders (Det/CC), and Directors expect the guidance contained within the manual to serve as the standardized processes and procedures. However, the flexibility to deviate from the manual guidance does exist upon AFOTEC/CC approval when warranted.

Figure 1.1. AFOTECMAN 99-101 Overview.

MDD

MS

A

MS

B

MS

C

FRP

IOC

Field

Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Key External Support Documents

Processes, Procedures & Products

Activities & Events

- ICD

- ADM

- CONOPs

- AoA

- COA

- M&S Support Plan

- ISP

- LCMP

- IPS

- Program ID

- Involvement Determination

- Initial Tasking Order

- Initial TRP

- Space Acquisition

- HPT Participation

- ITT

- Initial Test Design

- ESOH

- T&E Strategy

- TO Update

- TRP Update

- EOA for Space Programs

- ADM

- CDD

- AoA

- Threat Documentation

- SCG

- M&S Support Plan

- ISP

- LCMP

- RFP

- IPS

- Space Acquisition

- HPT Participation

- ITT

- Ref ine ITD

- T&E Master Plan

- OT Considerations

- ESOH

- Def iciency Reporting

- EOA

- ADM

- CPD

- CONOPs

- PMD

- AoA

- Threat Documentation

- M&S Support Plan

- ISP

- LCMP

- RFP

- IPS

- Space Acquisition

- HPT Participation

- ITT

- Form Test Team

- Support Agreements

- Test Concept Development

- Technical Review

- Test Capabilities

- ESOH

- OT Plan Development

- DMAP

- DTP

- DR Process

- TEMP

- OA

- OUE

See the Acronym List for Acronym Definitions

- ADM

- IPS

- Space Acquisition

- Finalize OT&E Plan

- TEMP

- ESOH

- TRR & Certif ication

- OA

- OUE

- IOT&E

- FOT&E Determination

- Closeout

1.1.1. Guidance memorandums, interim changes to this manual, and operational test information files (OTIF) are the means for the command section, the directors, and the Det/CCs to provide immediate notice of changes to the AFOTEC/CC’s OT&E policy, procedure, process directives and guidance outlined in the manual. In addition to the procedures on the title page, changes to the manual can be submitted via request to the A-3

AFOTECMAN 99-101 6 August 2010 11 workflow e-mail account. OTIFs are the means for the command section, the directors, and Det/CCs to provide immediate notice of updated information to AFOTEC that is not captured in formal policy.

1.1.2. Use the manual in conjunction with other AFOTEC produced references, such as briefing guides, templates, and technical papers/notes. These additional references provide the reader with further details of a specific process or procedure, as required. Additionally, the AFOTEC Policy link on the AFOTEC-Intranet homepage or the Analyst Training and Technical Information Center (ATTIC) (see Attachment 14) contain direct links to, or electronic copies of, these references. Throughout the manual, you can find references to documents located on the AFOTEC -Intranet. Attachment 14 contains a list of all referenced documents and a pointer to their actual location on the AFOTEC-Intranet or AFOTEC-Portal (located on the AF-Portal). See paragraph 1.15. for more information on the AFOTEC- Intranet. A working knowledge of the AFOTEC organization is assumed.

1.1.3. Terms of Reference. To effectively execute the AFOTEC/CC’s policies and procedures for planning, executing, and reporting OT&E and related activities, a common understanding of terms is essential. For the purposes of the manual, the term operational test (OT) includes initial OT&E (IOT&E), qualification OT&E (QOT&E), follow-on OT&E (FOT&E) and multiservice OT&E (MOT&E), as well as operational assessment (OA), early operational assessment (EOA), and operational utility evaluation (OUE). Operational utility assessments (OUA) are used to provide information for innovation programs.

Section 1B—DoD and Joint Chiefs of Staff (JCS) Processes

1.2. DoDI 5000. 02, Acquisition Model. Figure 1.2. depicts the DoD acquisition model for Major Defense Acquisition Programs (MDAP) and Major Automated Information Systems (MAIS). A key tenet of the DoD acquisition model is that an acquisition program can enter the process at any one of the acquisition milestones (MSs) depending on the maturity of the technology and program being acquired. For example, an advanced technology demonstration (ATD) or joint concept technology demonstration (JCTD) can successfully demonstrate a technology solution is ready to be transitioned to the formal acquisition process at MS B or later.

AFOTEC activities for the acquisition model support each MS; the low-rate initial production (LRIP) decision or limited deployment decision (MS C) may be informed with an OA and the full-rate production (FRP)/initial operational capability (IOC)/fielding decision is informed with completed I/Q/MOT&E. The Defense Acquisition Guidebook (DAG) provides supporting information to DoDI 5000.02. The guidebook provides best practices, lessons learned, and expectations in executing the defense acquisition process. The guidebook can be accessed on the AFOTEC-Intranet (see Attachment 14).

12 AFOTECMAN 99-101 6 August 2010

Figure 1.2. DoDI 5000.

IOCBA

Materiel Solution Analysis

Engineering and Manufacturing Development

Production & Deployment

Systems Acquisition

Operations & Support

C

User Needs

Sustainment

• The Materiel Development Decision precedes entry into any phase of the acquisition management system

• Entrance criteria met before entering phase

• Evolutionary Acquisition or Single Step to

Full Capability

FRP

Decision Review

FOC

LRIP/IOT&EPost-CDR Assessment

Technology Development

(Program Initiation)

Materiel Development Decision

Pre-Systems Acquisition

Technology Opportunities & Resources

= Decision Point = Milestone Review x

1.3. The Joint Capabilities Integration and Development System (JCIDS). The JCIDS, the Defense Acquisition System and the planning, programming, budgeting and execution processes form the principle DoD decision support processes for transforming the military forces to support the national military strategy and the defense strategy. The procedures established in the JCIDS support the Chairman and the joint requirements oversight council (JROC) in advising the Secretary of Defense in identifying, assessing and prioritizing joint military capability needs.

Validated and approved JCIDS documents provide the JROC advice and assessment. The JCIDS ensures the joint force has the capabilities necessary to perform across the range of military operations and challenges. A joint concepts-centric capabilities identification process is required to define how new joint capabilities are identified and developed as outlined in CJCSI 3170.01G (see Attachment 14). The JCIDS process describes how operational capability requirements are developed and fit into the acquisition process; AFOTEC bases its evaluation of operational capability on the JCIDS foundation. Figure 1.3. illustrates the JCIDS process and acquisition decisions.

AFOTECMAN 99-101 6 August 2010 13

Figure 1.3. The JCIDS Process.

DoD Strategic Guidance

Family of Joint Future Concepts Concepts of Operations Integrated Architectures

JROC DAB/

ITAB

DAB/

ITAB

JROC DAB/

ITAB

JROC DAB/

ITAB

IOC

JC

ID

S An al ys is

Re fin e An al ys is

Re fin e An al ys is

Materiel Development

Decision

MS

A

MS

B

MS

C

ICD

AoA

CDD CPD

DAB – Defense Acquisition Board ITAB – Information Technology Acquisition Board

1.4. Evolutionary Acquisition. Evolutionary acquisition is DoD’s preferred strategy for rapid acquisition of mature technology for the user (see Figure 1.4.). An evolutionary approach delivers capability in increments and recognizes, up front, the need for future capability improvements. The approaches to achieve evolutionary acquisition require collaboration between the user, developer, and tester. A viable evolutionary acquisition approach is to use a “single-step-to-full-capability” strategy. Refer to DoDI 5000.02 for additional information on evolutionary acquisition.

Figure 1.4. The DoD Requirements and Acquisition Process.

EMD

Increment 1 CPD1

AoA

Technology Development

JROC

DoD Strategic Guidance

Joint Operating Concepts Joint Functional Concepts

Materiel Solution AnalysisMDDICD

G ap

A na ly si s DAB

A CDD1

DAB

B

DAB

C

JROC JROC

EMD

Increment 2 CPD2Technology

Development

DAB

A CDD2

DAB

B

DAB

C

JROC JROC

EMD

Increment 3 CPD3Technology

Development

DAB

A CDD3

DAB

B

DAB

C

JROC JROC

x

14 AFOTECMAN 99-101 6 August 2010

1.4.1. Evolutionary Acquisition Characteristics. The success of the strategy depends on the consistent and continuous definition of capabilities-based requirements and the maturation of technologies that lead to disciplined development and production of systems that provide increasing capability. Testing should focus on operationally relevant sets of capabilities tested in an operationally representative environment. Evolutionary acquisition programs should:

1.4.1.1. Provide for early involvement of the Service operational test agency (OTA)/Joint Interoperability Test Command (JITC) in development test (DT) and evaluation (DT&E) and test planning.

1.4.1.2. Conduct adequate DT&E, live-fire T&E (LFT&E), and OT&E of each new incremental capability.

1.4.1.3. Integrate, as appropriate, and without compromising the specific requirements of the different types of testing, successive periods of DT&E, LFT&E, and OT&E.

1.4.1.4. Tailor test content and reporting against earlier test results, evaluating at a minimum the increment of mission accomplishment and survivability required of the new increment, plus whether or not performance demonstrated by the previous increment has been degraded.

1.4.1.5. For programs under Office of the Secretary of Defense (OSD) T&E oversight, support Director OT&E’s (DOT&E) schedule for reporting to the Secretary of Defense and Congressional defense committees, whether through phased submittal of dedicated reports or through DOT&E annual reports to Congress.

1.4.2. Test Management in Evolutionary Acquisition. Programs acquired under evolutionary acquisition may employ an incremental structure or a single-step-to-full-capability approach.

For incremental structure programs, each increment has its own set of acquisition decisions and requisite operational testing. In the case of a single-step-to-full capability, there is only one increment and one set of acquisition decisions. Partial capability deliveries within an increment are not intended to be fielded. There are times when the decision maker may accept the risk associated with fielding partial capabilities. The preferred activity for supporting the fielding of a partial capability is an OUE. The OUE enables a tailored evaluation of operational effectiveness, suitability and mission capability to be accomplished.

Additionally, the OUE allows for a tailored resolution of mission capability. For an OUE to be executed successfully, the acquisition and requirements communities need to provide documentation supporting their intent such as: capability requirements for the partial capability to be fielded, plans for accomplishing the certification process for OT and a test readiness review (TRR). AFOTEC staff and test teams should communicate the requirement for program documentation to the acquisition and requirements communities early and often when asked to support fielding of partial capability. Final determination of the level of AFOTEC support is decided by the AFOTEC/CC.

1.4.2.1. When sufficient information is known about an acquisition program, technology demonstration, or JCTD, an involvement decision can be made for the program.

Normally, only a single involvement decision is made for each program.

1.4.2.2. If enough information is available, test planning for the complete program can take place; if not, the first increment can be initially planned and future increments can be

AFOTECMAN 99-101 6 August 2010 15 planned later. When future planning is required, the core team/test team should develop a plan for conducting test planning on future increments and present the approach as part of the “way ahead” during the initial test design (ITD) briefing.

1.4.2.3. For software intensive systems, DOT&E’s memo titled Guidelines for Conducting OT&E for Software-Intensive System Increments can be consulted for information on the DOT&E perspective on OT&E requirements. The DOT&E memo is available on the AFOTEC-Intranet (see Attachment 14). Attachment 12 provides a matrix of risk levels extracted from the DOT&E memo and associated AFOTEC OT&E activity for implementing DOT&E guidelines. Analyses, Assessments and Lessons Learned Directorate (A-2/9) maintains test tools, with documentation, for assessing the risk associated with a given increment of a software intensive program. These tools are used by detachment test teams only for programs having an AFOTEC approved tasking order. The tools are used in conjunction with the information in Attachment 12 for determining the appropriate AFOTEC test activities for increments of software intensive systems. Detachment test team application of these tools and the DOT&E Guidelines are coordinated with A-3, Plans and Programs Directorate (A-5/8), and A-2/9.

1.4.2.4. The single initial capabilities document (ICD) and incremental capability development document (CDD) and capability production document (CPD) approach to operational capability requirements development brings with it AFOTEC involvement in the requirements process. The ICD is normally developed while the program is still in early influence and forms the foundation for the initial test design. The CDD and CPD for each increment are normally developed by the Major Command (MAJCOM)/user following AFOTEC initial test design. For additional information on updating and annexing capability documents, see AFI 10-601, Capabilities Based Requirements Development. High performance teams (HPT) are formed to develop each of these documents. AFOTEC representation on these HPTs is from the detachment responsible for the program, with A-3 assistance, if requested by the detachment.

Section 1C—AFOTEC Processes

1.5. Early Influence.

1.5.1. Early Influence is an approach adopted by AFOTEC for engaging and teaming with the user and acquisition communities to reduce program risk and support delivering mission capable systems to the warfighter. In addition, early influence enables AFOTEC to identify programs for possible involvement. Early influence includes the following activities:

1.5.1.1. Supporting development of capabilities documents, such as ICD, CDD or CPD.

1.5.1.2. Supporting analysis of alternative (AoA) studies through active participation and attendance in HPT.

1.5.1.3. Refinement of requirements for testability and measurability prior to Air Force Requirements Oversight Council (AFROC)/JROC approval.

1.5.1.4. Participating in establishment of the integrated test team (ITT).

1.5.1.5. Supporting development of the T&E strategy (TES).

1.5.1.6. Assisting in shaping the acquisition program and T&E master plan (TEMP).

16 AFOTECMAN 99-101 6 August 2010

1.5.1.7. Exploiting opportunities to find problems/risks and recommending improvements.

1.5.1.8. Influencing system design/development by providing operational assessment feedback.

1.5.1.9. Identifying test capability requirements and optimizing test resources.

1.5.1.10. Supporting development of operationally realistic threat lists.

1.5.1.11. Identifying OT testing capability requirements and addressing any testing capability and documentation shortfalls.

1.5.1.12. Maximizing participation in integrated DT/operational test (IDT/OT) activities, ensuring system readiness for OT&E.

1.5.1.13. Ensuring OT&E documentation is prepared/approved in a timely manner.

1.5.2. Early Influence for Space Systems. The space system T&E approach is based on early influence of the developmental and operational test community in the space system acquisition process. The Space Interim Guidance acquisition model is “front loaded” with the key decisions on requirements, funding, and development. Requirements definition, acquisition planning and T&E planning must be started earlier for space systems than for non-space systems. Space T&E supports the MS decisions before “metal is bent” by providing early and continuous information and analysis about system progress toward meeting the critical operational issues (COI) and critical technical parameters. Early programmatic reviews and decisions are supported with operational assessments developed by the ITT.

1.6. IDT/OT. IDT/OT is the Air Force process to transform T&E in support of the current acquisition process. The IDT/OT concept centers on leveraging DT efforts for a given program to meet OT requirements into a process that streamlines and integrates test activities and events.

IDT/OT facilitates the sharing of common operationally relevant data along with achieving an overall reduction in time and cost for OT and the reduction in risk during OT activities and events. IDT/OT requires a cradle-to-grave AFOTEC focus, and, IDT/OT success is dependent on the diligence of all AFOTEC personnel. While the primary responsibility for executing IDT/OT activities resides in the AFOTEC detachments, the AFOTEC headquarters’ staff also plays an integral role supporting every test director. IDT/OT success is based on the attributes of early and continuous collaboration between the warfighter (user), acquisition, and T&E communities, as well as the program executive office (PEO) (or product center or acquisition community) acknowledgement of AFOTEC as a stakeholder in DT events (see paragraph 4.23.2.). Early and continuous IDT/OT activities and events increase programmatic confidence (e.g., schedule, requirements stability, funding, testability, and test capability requirements).

AFOTEC will normally involve early (prior to MS A) and will start integrating with the other communities before concepts become formal programs of record and before identification of DT organizations (responsible test organization (RTO) nomination). Product center test authorities (CTA) have been established at each product and logistics center. The CTAs can be consulted in the absence of a program office T&E point of contact (POC). See Attachment 14 for a pointer to CTA information on the AFOTEC-Internet. If a RTO is not yet identified, or fully onboard at ITT stand-up, AFOTEC will request the CTA participate in ITT activities in place of RTOs.

AFOTECMAN 99-101 6 August 2010 17

1.6.1. IDT/OT Planning and Collaboration. IDT/OT starts with the initial formation of the ITT to support IDT/OT planning and continues throughout the program. IDT/OT execution can make use of a combined test force (CTF). The only limiting factor to the integration is the statutorily required dedicated phase of OT&E conducted on a production representative system demonstrating stabilized performance in a realistic operational environment to inform FRP, IOC, or fielding decisions. Early and frequent collaboration among all T&E stakeholders helps to ensure that common testing requirements can be satisfied and that data collected is available to all concerned. Identification of IDT/OT requirements should be done early enough to be included in the developmental contract so that data resulting from contractor developmental testing will be available to the ITT (see paragraph 3.10.). It is important to remember that throughout IDT/OT planning and execution, the individual DT and OT objectives should not be compromised. In the best of cases, a significant amount of operationally realistic data from the IDT/OT can be used to resolve OT&E requirements such that the remaining dedicated OT&E consists only of those activities and events not available in an IDT/OT environment.

1.6.1.1. The development of an IDT/OT concept is one approach to IDT/OT, as outlined in AFI 99-103. To support AFOTEC’s IDT/OT culture, IDT/OT activities and events as well as data requirements and sources are included and highlighted in all OT&E plans.

AFOTEC IOT&E plans will include specific DT events. Instead of looking back at DT and determining if an event was operationally relevant enough to apply to OT&E, AFOTEC should influence the development of the IDT/OT planning in such a way that DT will include events that will meet the needs of both DT and OT. Product center commanders can expect AFOTEC OT&E plans to include IDT/OT activities and data requirements and sources, with OT&E plans being approved earlier in order to request and obtain product center acknowledgement.

1.6.2. IDT/OT Considerations. The fundamental goal behind IDT/OT is to maximize the availability of data and obtain operationally realistic data during DT. Test resources (money and people) and time continue to shrink. AFOTEC ITT members need to share test data and add operational realism when possible via integrated DT/OT activities and events. Test teams need to implement IDT/OT with a goal of achieving a process which integrates DT and OT events. Note: DoDI 5000.02 states “The Department of Defense may not conduct OT&E, including OA, IOT&E, or FOT&E, until the DOT&E approves, in writing, the OT&E portions of the T&E plan for programs on the OSD T&E Oversight List and the adequacy of the plans (including the projected level of funding) for the OT&E to be conducted in connection with that program.” The approval can be obtained through several vehicles such as the TEMP, an OA plan, or a separate memorandum from AFOTEC/CC outlining the OT portion of the IDT/OT.

1.6.2.1. The early collaboration with DT counterparts facilitates a total team effort to develop a testing strategy that integrates both DT and OT. Early T&E community identification of test requirements increases the potential to identify shortfalls, develop peculiar capabilities, and increases the preparedness for both DT and OT. Even if the RTO is not yet identified, AFOTEC states IDT/OT intent in all T&E documentation.

IDT/OT can also include other agencies, such as the JITC, to validate and certify technical compliance with compatibility, interoperability, and integration standards, or

18 AFOTECMAN 99-101 6 August 2010

Air Force Information Operations Center (AFIOC) for information assurance (IA) procedures and activities.

1.6.2.2. Recognizing the earliest pre-MS A test design may be limited with regard to integrated test planning because little DT planning has occurred, integrated testing is still an action item on the test design agenda. Initial test design meetings are held with external core team participation. AFOTEC’s IDT/OT intent is clearly stated from first contact with the external communities and remains a consistent theme throughout AFOTEC’s involvement. Early inclusion of IDT/OT language in MS A TES sets the tone for every acquisition. Many of the AFOTEC-developed measures of effectiveness (MOE) and measures of suitability (MOS) can be addressed during DT&E. AFOTEC can then focus on answering remaining measures, mission statements, COIs, and mission impacts during the remaining dedicated OT&E. For example, if the system under test is in its near-final/final configuration during the last portions of DT&E, a fair amount of useable data can be obtained for the OT&E.

1.6.3. IDT/OT for Space Systems. IDT/OT is a key principle of the space systems T&E approach. Because systems already in space cannot be retrieved from their environment to fix hardware deficiencies discovered after launch, a significant amount of operational testing must be accomplished prior to launch. Close integration of DT and OT activities, must occur while satellites are still on the ground. The developer needs to be an integral member of the ITT. To facilitate developer participation, the request for proposal (RFP) needs to include appropriate language to support the IDT/OT process. It is important when working with DT to ensure contractor support is requested, coordinated, and documented through the respective system program office. The TEMP describes how all testing will be integrated and addresses the overall evaluation approach, key evaluation measures, and the major risks or limitations to completing the evaluations. IDT/OT must not compromise the evaluation of test results carried out by both developmental and operational testers.

1.7. Space Systems OT&E. The need for fully informed decisions regarding increasingly expensive, yet indispensible space system capabilities is crucial in today’s environment of constrained resources. AFOTEC uses the Air Force Space Systems OT&E model to provide fact-based, decision-quality data to decision makers in time to support key acquisition decisions.

1.7.1. Acquisition of space major defense acquisition programs, which were granted a waiver to the DoD 5000 requirements, follow procedures contained in the Space Interim Guidance. The Air Force space T&E policy directs early activities and additional actions that provide earlier OT&E influence and better support space acquisition programs developed under the Space Interim Guidance. The Space Interim Guidance uses a streamlined acquisition framework with MS decisions that occur earlier than typical DOD 5000-series MSs and decision reviews. Figure 1.5. depicts the acquisition process for space programs under the Space Interim Guidance and also shows the JCIDS process adaptations for space programs.

AFOTECMAN 99-101 6 August 2010 19

Figure 1.5. Space Interim Guidance.

Interim NSS Guidance

PHASE B

Preliminary Design

PHASE A

Concept

Development

PHASE C

Complete

Design

BA

PDRSDRSRR

Pre - Systems Acquisition Systems Acquisition Sustainment

C

Upgrade Decision

Follow On Buy

Approval

1st Launch

FOCIOC

PHASE B

Approval

PHASE A

Approval

PHASE C

Approval

JROC

CDD

Pre KDP-A Concept Studies

JROC

ICD

Milestones:

JROC

CDD

Materiel Development Decision Mtg

Initial

CDD

CDR

JROC

CPD

Build Approval

PHASE D

Build & Operations

1.7.2. In a traditional acquisition program governed by DoDD 5000.01, expenditures are relatively small in the Research and Development and investment phases, compared to the cost of production and system operation. For these traditional acquisitions, OT occurs just prior to the major investment or production decision and provides data to inform those decisions adequately. Most of the investment for space systems occurs early in the program, most often without a major production decision. In the past, OT&E took place at the same point in the acquisition cycle as the DoDD 5000.01 programs. However, early in Space Interim Guidance programs, most of the investment has been made, most of the key acquisition decisions have been made, and the critical operational decision to launch the satellite has been made and executed. Additionally, the ground station and associated software often lag in deployment making timely post launch OT&E difficult, if not impossible. Making these key decisions prior to the execution of OT&E severely limits the value of OT&E.

1.7.3. AFOTEC’s implementation of the Air Force space T&E model provides an OT&E that fits the space systems’ acquisition model, delivering better value to both the acquisition and operational decision makers by moving OT&E activity well prior to launch. The three key tenets of the Air Force space T&E model are: early and continuous IDT/OT involvement throughout the system’s life cycle; agile analysis and reporting; and focus on system-of-system evaluations.

1.7.4. AFOTEC’s space test approach provides the basis for knowledge-based acquisition and operational decisions throughout the life cycle of our national security space systems.

The approach provides early operational involvement delivering a number of benefits as listed in Table 1.1.

Table 1.1. Space Test Approach Benefits.

Ensuring the warfighter receives needed mission capabilities Providing early clarity and continued update of operational requirements Influencing early and continual development and refinement of the concept of operations

(CONOPS).

20 AFOTECMAN 99-101 6 August 2010

Ensuring frequent reviews of threat documents to ensure the system design addresses current and future threats.

Highlighting program shortfalls and benefits throughout the development process when they can be addressed most efficiently and inexpensively.

Allowing the user to understand and accept acquisition risks and adjust their mission requirements and plans accordingly.

Addressing and correcting systemic suitability issues early in the program’s development.

1.7.5. The Det/CC responsible for space systems needs to refer to the guidance and procedures contained in the remainder of AFOTECMAN 99-101 and comply with the AFOTEC processes as applicable to specific programs. Successful implementation of the space systems T&E approach is dependent on a clear understanding of the policy and procedural concepts contained in the Space Interim Guidance and AFI 99-103, Chapter 8, Space Systems T&E. Note: Using the space systems OT&E model for cyberspace/information technology systems or user equipment segment programs acquired under the Space Interim Guidance acquisition model may not be appropriate. The Det/CC responsible for space systems also needs to refer to the Memorandum of Agreement (MOA) on Multiservice OT&E for specific procedures regarding lead or supporting OTA activities.

1.8. Establishing a Relationship with DOT&E. Early in a program, the Det/CC and test director should identify and meet the DOT&E action officer assigned to the program if the program is on the OSD T&E Oversight List. Test directors (TD) must remember they are representing the AFOTEC/CC and any request to change an approved test plan or other special request from DOT&E shall be provided to the AFOTEC/CC in writing. For example, if DOT&E requests a change to the number of test articles previously determined adequate for OT&E, the TD should not commit. Instead, the TD will request the DOT&E action officer (AO) formalize the request to the AFOTEC/CC in writing. Upon receipt of the request, the AFOTEC/CC will decide if the test plan will be changed. Frequent communication between the Det/CC, TD, and the DOT&E action officer is useful and important.

1.8.1. OSD T&E Oversight List. AFOTEC is the OTA for all Air Force programs on the OSD T&E Oversight List. Det/CCs, TDs, and technical advisors are responsible for knowing the oversight status of their programs and ensuring the current status is updated on AFOTEC’s intranet site and AFOTEC’s Secure Internet Protocol Router (SIPR)-intranet site.

1.9. Program Support by AFOTEC Liaison Officers (LNO). LNOs can greatly benefit AFOTEC by engaging with program offices, product and logistics centers, MAJCOMs, and other key agencies (e.g., DOT&E, AF T&E (AF/TE), JITC, test ranges, etc.) early in the requirements, alternatives, and acquisition processes. The LNOs should support early program activities by discovering emerging requirements and new acquisition programs, and by obtaining key program documentation. By providing program support, LNOs ensure AFOTEC’s planning considerations are addressed, external offices are educated about the benefits of AFOTEC’s involvement, and communications are enhanced between other organizations and AFOTEC. The AFOTEC Program Manager’s Operational Test Toolkit provides a value-added product for AFOTEC personnel to leave behind with program office and MAJCOM personnel.

1.9.1. AFOTEC LNO. The AFOTEC LNO co-located with Air Force Materiel Command (AFMC) or Air Force Space Command (AFSPC) product centers or the Pentagon, act as an

AFOTECMAN 99-101 6 August 2010 21 on-scene representative of the AFOTEC/CC and fulfill a critical role in the AFOTEC early influence process. These liaison officers accomplish the following actions:

1.9.1.1. Interface directly with their assigned product center counterpart and actively participate in the Product Center’s acquisition plan process and test strategy formulation as directed via AFOTEC tasking orders. The LNO should actively look for the following: information to support tasking order (TO) preparation, new program basics (such as program name description and acquisition strategies), RTO, other POCs, related program documentation, program reviews and forecasts, and other information that may be helpful to establish collaborative IDT/OT planning efforts. Additional program information such as program priorities, requirements documents, CONOPS, operating plans, information support plans (ISP) and tactics, techniques, and procedures (TTP) should be obtained as requested.

1.9.1.2. Notify AFOTEC/CC, detachments, and A-3 of emerging projects/programs.

1.9.1.3. Filter project/program information and recommend involvement initiation when appropriate.

1.9.1.4. Notify AFOTEC/CC, detachments, and A-3 of issues of interest and key meetings requiring AFOTEC representation.

1.9.1.5. Attend product center meetings addressing program development.

1.9.1.6. Clarify AFOTEC policy and positions as required and refer questions to appropriate AFOTEC detachment or headquarters staff.

1.9.1.7. The Washington, DC LNO provides an entry point for contacts and meetings in the Pentagon or Washington, DC area.

1.10. Staffing and Coordination. Many different documents originating both inside and outside of AFOTEC will require staff and executive review. Detailed procedures for staffing and coordinating documents and briefings are contained in Attachment 11. AFOTEC/CC requires prompt information update and exchange for OT&E activities. Det/CCs will provide programmatic updates at weekly stand-up or staff meeting when staff packages do not support time constraints. The weekly update serves to quickly inform the Command Staff and key personnel; it does not negate the requirement for officially staffed documents.

1.11. Meeting Attendance. OT planners attend various acquisition meetings either in person, via teleconference, or via video teleconference. Typical meetings include integrated product team meetings, integrated T&E working group meetings, AoA meetings. Depending on the program, the titles of these meetings may vary. The common thread is that they are usually called by the program office director or staff, or a developing contractor. Core team/test team members should actively monitor program office activities for meetings that could be of benefit to test planning activities. Taking an active role and seeking out information on system program office (SPO) activities is beneficial and can lead to preventing surprises. When attending meetings, remember you are representing the AFOTEC/CC; when appropriate, give the most current AFOTEC Command Briefing (see Attachment 14), and be prepared to ask questions and to give answers as a full participant in the meeting. AFOTEC/CC-approved positions (level of involvement, exit criteria, etc.) are not changed without approval of the AFOTEC/CC.

Commitments on behalf of AFOTEC are not made without prior authorization. If in doubt about

22 AFOTECMAN 99-101 6 August 2010 an answer or topic, take an action item to follow-up. A trip report summarizing the important results of the meeting is provided to the AFOTEC/CC by the appropriate director or Det/CC within two duty days upon completion of the meeting. A template for the trip report format can be found on the AFOTEC-Intranet (see Attachment 14).

1.12. External Briefings. The AFOTEC/CC is the release authority for all external briefings.

Per DoDI 5000.02, DOT&E requires a test concept briefing for DOT&E oversight programs at least 120 days prior to test start. DOT&E may request a briefing from AFOTEC to accompany the final test plan delivered 60 days before test start. If requested, the briefing associated with the final test plan submittal to DOT&E is coordinated prior to submitting to DOT&E. Final report briefings are provided to HQ USAF staff and the OSD as requested. External briefings presented to a program office, MAJCOM, or decision maker, should be done in-person. The test director is normally accompanied by an O-6 and a subject matter expert. Remember, when presenting an AFOTEC/CC-approved briefing externally any changes must be approved by the AFOTEC/CC. Prior to presenting an AFOTEC/CC-approved briefing to an external audience, the following guidance should be reviewed to structure the briefing for success (see Table 1.2.).

AFOTECMAN 99-101 6 August 2010 23

Table 1.2. External Briefing Checklist.

1.13. Information Protection. Security (CVIS) serves as the HQ point of contact for information protection (IP). The Det/CC and technical advisor at each detachment execute the guidance. IP involves safeguarding any data/information that potentially reveals U.S.

vulnerabilities, capabilities, capability gaps, or any data/information that falls under the freedom of information act (FOIA) exemptions (privacy act, export control, contractor owned information, etc.). IP includes the correct marking, classification, and handling of AFOTEC originated products (see Attachment 14). The criteria for releasing information (to include review, coordination, and authority to release) is defined in AFI 61-201, The Local Scientific and Technical Information Process, AFI 61-204, Disseminating Scientific and Technical Information, and DoDD 5400.7 AF Sup 1, DoD Freedom of Information Act Program, as well as other directives such as DoDD 3200.12, DoD Scientific and Technical Information (STI) Program (STIP), DODI O-3600.02, Information Operation (IO) Security Classification Guidance (U), DoDD C-5230.23, Intelligence Disclosure Policy (U), DoDD 5230.25, 24 AFOTECMAN 99-101 6 August 2010

Withholding of Unclassified Technical Data From Public Disclosure, USC, Title 10, Chapter 3, §130, Authority to Withhold from Public Disclosure Certain Technical Data.

1.13.1. AFOTEC Generated Documents.

1.13.1.1. The Det/CC and technical advisor provides IP guidance to test directors over the course of the program.

1.13.1.2. The detachment technical advisor ensures proper markings of working papers and data that support test reports.

1.13.1.3. CVIS, as IP POC, supports the detachments as needed.

1.13.2. Information Releases. Forward all requests for information through the appropriate POC, as outlined in Table 1.3. The POC releases the information only after reviewing the distribution and release markings, coordinating through AFOTEC Electronic Coordination System (AECS), and obtaining approval, as appropriate. For further information on release of AFOTEC information, see Attachment 9.

Table 1.3. POCs for Information Requests.

1.13.3. Special Access Programs (SAP). The SAPs Division (A-3Z) is the HQ lead for SAPs. A-3Z maintains control of all SAP-associated activities and HQ functions except for Safety (SE), and A-2/9. It is mandatory that any program contacts or issues that pertain to SAPs are brought to the attention of A-3Z. The AFOTEC/CC provides direction for all SAP programs. All normal AFOTEC processes (within security constraints) will be performed for all SAPs.