AFOTECMAN 99-101 Operational Test Processes and Procedures.pdf

APPROVED – AWAITING PUBLISHING BY AFDPO

BY ORDER OF THE COMMANDER AFOTEC MANUAL 99-101

24 MAR 2010

Test and Evaluation

OPERATIONAL TEST PROCESSES AND PROCEDURES

COMPLIANCE WITH THIS MANUAL IS MANDATORY

ACCESSIBILITY: Publications and forms are available for downloading or ordering on the e-Publishing website at www.e-Publishing.af.mil.

RELEASIBILITY: There are no releasibility restrictions on this publication.

OPR: AFOTEC/A-3 (Mr. Russell K. Foos) Certified by: AFOTEC/A-3 (Col David C. Gould II)

Pages: 284 Supersedes: AFOTECI 99-101, 1 May 2007 AFOTEC OT&E Guide, 12 Feb 2009 Guidance Memorandums: AFI 99-103 AFOTECGM1, 27 Jul 09; and AFOTECGM

99-1, 10 Sep 09.

OT&E Information Files: 0810, 0817, 0902, 0903, 0905, 0906, 0907, 0908, 0909, 0910, 0911, 0912, 0913, 0914, 0915, 0916, 0917, 0919, 0920, 0920(1), 0921, 0922, 0923, 0924, 0925, 0926, 0927, 0928, 0929, 0930, and 1001.

The AFOTEC Manual 99-101 issues organizationally specific minimum mandatory policies and procedures for planning, executing, and reporting operational test and evaluation (OT&E) and related activities. The AFOTECMAN 99-101 is to be used in conjunction with policies, directives, and instructions contained in Department of Defense Directive (DoDD) 5000.01, The Defense Acquisition System; DoD Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System; Chairman of the Joint Chiefs of Staff (CJCSI) 3170.01G, Joint Capabilities Integration and Development System; CJCSI 6212.01E, Interoperability and Supportability of Information Technology and National Security Systems; Air Force Policy Directive (AFPD) 99- 1, Test and Evaluation Process; Air Force Instruction (AFI) 99 103, Capabilities Based Test and Evaluation; and Air Force Mission Directive 14, Air Force Operational Test and Evaluation Center. This AFOTEC Manual assumes a fundamental understanding of the DoD and Air Force weapon system acquisition processes. This manual does not repeat higher headquarters policy and direction and the higher headquarters guidance should be consulted first to understand the roles and missions of AFOTEC. The AFOTECMAN 99-101 outlines the AFOTEC/CC‘s processes, procedures, checklists, and techniques of the various phases of OT&E. The manual applies to all AFOTEC directorates, detachments, and operating locations. Ensure that all records created as a result of processes prescribed in this publication are maintained in accordance with Air Force Manual (AFMAN) 33-363, Management of Records, and disposed of in accordance with Air Force Records Information Management System (AFRIMS) Records

AFOTECMAN 99-101, 24 Mar 2010

Disposition Schedule (RDS) located at https://www.my.af.mil/gcss-af61a/afrims/afrims/.

Additionally, if the publication generates reports, all applicable Reports Control Numbers must be in accordance with AFI 33-324, The Information Collections and Reports Management Program. Refer recommended changes and questions about this publication to AFOTEC/A-3 using the AF Form 847, Recommendation for Change of Publication; route AF Forms 847 from the field through the appropriate chain of command. This publication may not be supplemented.

Compliance with this publication is mandatory for all AFOTEC personnel.

WAIVER OF DIRECTIVES, INSTRUCTIONS, OR PUBLICATIONS

Submit requests for waiver to the AFOTECMAN 99-101 to the A-3 Corporate Account, AFOTEC.A3.workflow@kirtland.af.mil. A-3 will administratively coordinate the waiver request with the process owner and the affected AFOTEC staff.

Refer recommended changes and questions about this publication to the Office of Primary Responsibility (OPR) using AF IMT 847, Recommendation for Change of Publication, routed through the functional chain of command. Ensure all records created as a result of processes prescribed in this publication are maintained according to Air Force Manual (AFMAN) 37-123, Management of Records (will convert to AFMAN 33-363), and disposed of according to the Air Force Information Management System (AFRIMS) Records Disposition Schedule (RDS) located at https://afrims.amc.af.mil/rds series.cfm.

SUMMARY OF CHANGES

This Manual is a completely new document and must be reviewed in its entirety.

https://afrims.amc.af.mil/rds%20series.cfm

Table of Contents

Chapter 1 – Introduction Chapter 2 – AFOTEC Activities Supporting Milestone A Chapter 3 – AFOTEC Activities Supporting Milestone B Chapter 4 – AFOTEC Activities Supporting Milestone C Chapter 5 – AFOTEC Activities Supporting FRP/IOC/Fielding Decision Chapter 6 – AFOTEC Activities Following FRP/IOC/Fielding Decision Attachment 1 – Glossary of References and Supporting Information Attachment 2 – Capability-Based Requirements Documents Attachment 3 – Test Resource Plan Attachment 4 – Environment, Safety and Occupational Health (ESOH) Attachment 5 – Test Plan Development Attachment 6 – Deficiency Reporting Attachment 7 – Test Execution Attachment 8 – Data Collection, Management, and Evaluation Attachment 9 – Release of Test Information Attachment 10 – Test Report Development Attachment 11 – Coordination Process Attachment 12 – Software Guidelines Attachment 13 – Test Program Case File Attachment 14 – SharePoint/Portal References Attachment 15 – Test Readiness Review and System Certification Attachment 16 – Special Interest Items Attachment 17 - Activation and Inactivation of Operating Locations (OL)

Chapter 1 Introduction

Section 1A-Overview

1.1. The intent of the manual is to provide AFOTEC personnel with mandatory "how to‖ guidance on accomplishing operational test in support of the Air Force acquisition process (see figure 1.1). The processes, procedures, checklists, and techniques outlined in the following chapters are a collection of lessons learned and best practices for conducting an operational test program. The AFOTEC Commander (AFOTEC/CC), Detachment Commanders (Det/CC), and Directors expect the guidance contained within the manual to serve as the standardized processes and procedures. However, the flexibility to deviate from the manual guidance does exist upon AFOTEC/CC approval when warranted.

Figure 1.1. AFOTEC Manual 99-101 Overview

1.1.1 Guidance memorandums (GM), interim changes (IC) to this manual, and operational test information files (OTIF) are the means for the command section, the directors, and the Det/CCs to provide immediate notice of changes to the AFOTEC/CC‘s OT&E policy, procedure, process directives and guidance outlined in the manual. In addition to the procedures on the title page, changes to the manual can be submitted via request to the A-3 workflow e-mail account. OTIFs are the means for the command section, the directors, MDD

MS

A

MS

B

MS

C

FRP

IOC

Field

Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Key External Support Documents

Processes, Procedures & Products

Activities & Events

- ICD

- ADM

- CONOPs

- AoA

- COA

- M&S Support Plan

- ISP

- LCMP

- IPS

- Program ID

- Involvement Determination

- Initial Tasking Order

- Initial TRP

- Space Acquisition

- HPT Participation

- ITT

- Initial Test Design

- ESOH

- T&E Strategy

- Tasking Order Update

- TRP Update

- EOA for Space Programs

- ADM

- CDD

- AoA

- Threat Documentation

- SCG

- M&S Support Plan

- ISP

- LCMP

- RFP

- IPS

- Space Acquisition

- HPT Participation

- ITT

- Ref ine ITD

- T&E Master Plan

- OT Considerations

- ESOH

- Def iciency Reporting

- EOA

- ADM

- CPD

- CONOPs

- PMD

- AoA

- Threat Documentation

- M&S Support Plan

- ISP

- LCMP

- RFP

- IPS

- Space Acquisition

- HPT Participation

- ITT

- Form Test Team

- Support Agreements

- Test Concept Development

- Technical Review

- Test Capabilities

- ESOH

- OT Plan Development

- DMAP

- DTP

- DR Process

- TEMP

- OA

- OUE

See the Acronym List for Acronym Definitions

- ADM

- IPS

- Space Acquisition

- Finalize OT&E Plan

- TEMP

- ESOH

- TRR & Certif ication

- OA

- OUE

- IOT&E

- FOT&E Determination

- Closeout and Det/CCs to provide immediate notice of updated information to AFOTEC that is not captured in formal policy.

1.1.2 Use the manual in conjunction with other AFOTEC produced references, such as briefing guides, templates, and technical papers/notes. These additional references provide the reader with further details of a specific process or procedure, as required. Additionally, the AFOTEC Policy link on the AFOTEC-Intranet homepage or the Analyst Training and Technical Information Center (ATTIC) (see attachment 14) contain direct links to, or electronic copies of, these references. Throughout the manual, you can find references to documents located on the AFOTEC -Intranet. Attachment 14 contains a list of all referenced documents and a pointer to their actual location on the AFOTEC-Intranet or AFOTEC-Portal (located on the Air Force-Portal). See paragraph 1.15 for more information on the AFOTEC- Intranet. A working knowledge of the AFOTEC organization is assumed.

1.1.3. Terms of Reference. To effectively execute the AFOTEC/CC‘s policies and procedures for planning, executing, and reporting OT&E and related activities, a common understanding of terms is essential. For the purposes of the manual, the term OT includes initial OT&E (IOT&E), qualification OT&E (QOT&E), follow-on OT&E (FOT&E) and multiservice OT&E (MOT&E), as well as operational assessment (OA), early operational assessment (EOA), and operational utility evaluation (OUE). Operational utility assessments (OUA) are used to provide information for innovation programs.

Section 1B-DoD and JCS Processes

1.2. DoDI 5000.02 Acquisition Model. Figure 1.2 depicts the DoD acquisition model for Major Defense Acquisition Programs (MDAP) and Major Automated Information Systems (MAIS). A key tenet of the DoD acquisition model is that an acquisition program can enter the process at any one of the acquisition milestones depending on the maturity of the technology and program being acquired. For example, an advanced technology demonstration (ATD) or joint concept technology demonstration (JCTD) can successfully demonstrate a technology solution is ready to be transitioned to the formal acquisition process at Milestone (MS) B or later. AFOTEC activities for the acquisition model support each milestone; the low-rate initial production (LRIP) decision or limited deployment decision (MS C) may be informed with an operational assessment (OA) and the full-rate production (FRP)/initial operational capability (IOC)/fielding decision is informed with completed initial/qualification/multiservice OT&E (I/Q/MOT&E).

The Defense Acquisition Guidebook provides supporting information to DoDI 5000.02. The guidebook provides best practices, lessons learned, and expectations in executing the defense acquisition process. The guidebook can be accessed on the AFOTEC-Intranet (see attachment 14).

Figure 1.2. DoDI 5000.02 Model

1.3. The Joint Capabilities Integration and Development System. The Joint Capabilities Integration and Development System (JCIDS), the Defense Acquisition System and the planning, programming, budgeting and execution processes form the principle DoD decision support processes for transforming the military forces to support the national military strategy and the defense strategy. The procedures established in the JCIDS support the Chairman and the joint requirements oversight council (JROC) in advising the Secretary of Defense in identifying, assessing and prioritizing joint military capability needs. Validated and approved JCIDS documents provide the JROC advice and assessment. The JCIDS ensures the joint force has the capabilities necessary to perform across the range of military operations and challenges. A joint concepts-centric capabilities identification process is required to define how new joint capabilities are identified and developed as outlined in CJCSI 3170.01, Joint Capabilities Integration and Development System (see attachment 14). The JCIDS process describes how operational capability requirements are developed and fit into the acquisition process; AFOTEC bases its evaluation of operational capability on the JCIDS foundation. Figure 1.3 illustrates the JCIDS process and acquisition decisions.

IOCBA

Materiel Solution Analysis

Engineering and Manufacturing Development

Production & Deployment

Systems Acquisition

Operations & Support

C

User Needs

Sustainment

• The Materiel Development Decision precedes entry into any phase of the acquisition management system

• Entrance criteria met before entering phase

• Evolutionary Acquisition or Single Step to

Full Capability

FRP

Decision Review

FOC

LRIP/IOT&EPost-CDR Assessment

Technology Development

(Program Initiation)

Materiel Development Decision

Pre-Systems Acquisition

Technology Opportunities & Resources

= Decision Point = Milestone Review x

Figure 1.3. The JCIDS Process

1.4. Evolutionary Acquisition. Evolutionary acquisition is DoD‘s preferred strategy for rapid acquisition of mature technology for the user (see figure 1.4). An evolutionary approach delivers capability in increments and recognizes, up front, the need for future capability improvements.

The approaches to achieve evolutionary acquisition require collaboration between the user, developer, and tester. A viable evolutionary acquisition approach is to use a "single-step-to-full-capability‖ strategy. Refer to DoDI 5000.02 for additional information on evolutionary acquisition.

DoD Strategic Guidance

Family of Joint Future Concepts Concepts of Operations Integrated Architectures

JROC DAB/

ITAB

DAB/

ITAB

JROC DAB/

ITAB

JROC DAB/

ITAB

IOC

JC

ID

S An al ys is

Re fin e An al ys is

R ef in e

A na ly si s

Materiel Development

Decision

MS

A

MS

B

MS

C

ICD

AoA

CDD CPD

DAB – Defense Acquisition Board ITAB – Information Technology Acquisition Board

Figure 1.4. The DoD Requirements and Acquisition Process

1.4.1. Evolutionary Acquisition Characteristics. The success of the strategy depends on the consistent and continuous definition of capabilities-based requirements and the maturation of technologies that lead to disciplined development and production of systems that provide increasing capability. Testing should focus on operationally relevant sets of capabilities tested in an operationally representative environment. Evolutionary acquisition programs should:

1.4.1.1. Provide for early involvement of the Service OTA/Joint Interoperability Test Command (JITC) in DT&E and test planning.

1.4.1.2. Conduct adequate DT&E, live-fire T&E (LFT&E), and OT&E of each new incremental capability.

1.4.1.3. Integrate, as appropriate, and without compromising the specific requirements of the different types of testing, successive periods of DT&E, LFT&E, and

OT&E.

1.4.1.4. Tailor test content and reporting against earlier test results, evaluating at a minimum the increment of mission accomplishment and survivability required of the new increment, plus whether or not performance demonstrated by the previous increment has been degraded.

1.4.1.5. For programs under OSD T&E oversight, support Director OT&E‘s (DOT&E) schedule for reporting to the Secretary of Defense and Congressional defense committees, whether through phased submittal of dedicated reports or through DOT&E annual reports to Congress.

1.4.2. Test Management in Evolutionary Acquisition. Programs acquired under evolutionary acquisition may employ an incremental structure or a single-step-to-full-capability approach. For incremental structure programs, each increment has its own set of

EMD

Increment 1 CPD1

AoA

Technology Development

JROC

DoD Strategic Guidance

Joint Operating Concepts Joint Functional Concepts

Materiel Solution AnalysisMDDICD

G ap

A na ly si s

DAB

A CDD1

DAB

B

DAB

C

JROC JROC

EMD

Increment 2 CPD2Technology

Development

DAB

A CDD2

DAB

B

DAB

C

JROC JROC

EMD

Increment 3 CPD3Technology

Development

DAB

A CDD3

DAB

B

DAB

C

JROC JROC

x acquisition decisions and requisite operational testing. In the case of a single-step-to-full capability, there is only one increment and one set of acquisition decisions. Partial capability deliveries within an increment are not intended to be fielded. There are times when the decision maker may accept the risk associated with fielding partial capabilities. The preferred activity for supporting the fielding of a partial capability is an OUE. The OUE enables a tailored evaluation of operational effectiveness, suitability and mission capability to be accomplished. Additionally, the OUE allows for a tailored resolution of mission capability. For an OUE to be executed successfully, the acquisition and requirements communities need to provide documentation supporting their intent such as: capability requirements for the partial capability to be fielded, plans for accomplishing the certification process for OT and a test readiness review (TRR). AFOTEC staff and test teams should communicate the requirement for program documentation to the acquisition and requirements communities early and often when asked to support fielding of partial capability. Final determination of the level of AFOTEC support is decided by the AFOTEC/CC.

1.4.2.1. When sufficient information is known about an acquisition program, technology demonstration, or JCTD, an involvement decision can be made for the program. Normally, only a single involvement decision is made for each program.

1.4.2.2. If enough information is available, test planning for the complete program can take place; if not, the first increment can be initially planned and future increments can be planned later. When future planning is required, the core team/test team should develop a plan for conducting test planning on future increments and present the approach as part of the "way ahead‖ during the initial test design (ITD) briefing.

1.4.2.3. For software intensive systems, DOT&E‘s memo titled Guidelines for Conducting OT&E for Software-Intensive System Increments can be consulted for information on the DOT&E perspective on OT&E requirements. The DOT&E memo is available on the AFOTEC-Intranet (see attachment 14). Attachment 12 provides a matrix of risk levels extracted from the DOT&E memo and associated AFOTEC OT&E activity for implementing DOT&E guidelines. AFOTEC/A-2/9 maintains test tools, with documentation, for assessing the risk associated with a given increment of a software intensive program. These tools are used by detachment test teams only for programs having an AFOTEC approved tasking order. The tools are used in conjunction with the information in attachment 12 for determining the appropriate AFOTEC test activities for increments of software intensive systems. Detachment test team application of these tools and the DOT&E Guidelines are coordinated with A-3, A-5/8, and A-2/9.

1.4.2.4. The single initial capabilities document (ICD) and incremental capability development document (CDD) and capability production document (CPD) approach to operational capability requirements development brings with it AFOTEC involvement in the requirements process. The ICD is normally developed while the program is still in early influence and forms the foundation for the initial test design. The CDD and CPD for each increment are normally developed by the MAJCOM/user following AFOTEC initial test design. For additional information on updating and annexing capability documents, see AFI 10-601, Capabilities Based Requirements Development. High performance teams (HPT) are formed to develop each of these documents. AFOTEC representation on these HPTs is from the detachment responsible for the program, with AFOTEC/A-3 assistance, if requested by the detachment.

Section 1C-AFOTEC Processes

1.5. Early Influence.

1.5.1. Early Influence is an approach adopted by AFOTEC for engaging and teaming with the user and acquisition communities to reduce program risk and support delivering mission capable systems to the warfighter. In addition, early influence enables AFOTEC to identify programs for possible involvement. Early influence includes the following activities:

1.5.1.1. Supporting development of capabilities documents, such as ICD, CDD or

CPD.

1.5.1.2. Supporting analysis of alternative (AoA) studies through active participation and attendance in HPT.

1.5.1.3. Refinement of requirements for testability and measurability prior to Air Force Requirements Oversight Council (AFROC)/JROC approval.

1.5.1.4. Participating in establishment of the integrated test team (ITT).

1.5.1.5. Supporting development of the test and evaluation strategy (TES).

1.5.1.6. Assisting in shaping the acquisition program and test and evaluation master plan (TEMP).

1.5.1.7. Exploiting opportunities to find problems/risks and recommending improvements.

1.5.1.8. Influencing system design/development by providing operational assessment feedback.

1.5.1.9. Identifying test capability requirements and optimizing test resources.

1.5.1.10. Supporting development of operationally realistic threat lists.

1.5.1.11. Identifying OT testing capability requirements and addressing any testing capability and documentation shortfalls.

1.5.1.12. Maximizing participation in integrated developmental test/operational test (IDT/OT) activities, ensuring system readiness for OT&E.

1.5.1.13. Ensuring OT&E documentation is prepared/approved in a timely manner.

1.5.2. Early Influence for Space Systems. The space system T&E approach is based on early influence of the developmental and operational test community in the space system acquisition process. The Space Interim Guidance acquisition model is "front loaded‖ with the key decisions on requirements, funding, and development. Requirements definition, acquisition planning and T&E planning must be started earlier for space systems than for non-space systems. Space T&E supports the MS decisions before "metal is bent‖ by providing early and continuous information and analysis about system progress toward meeting the critical operational issues (COI) and critical technical parameters (CTP). Early programmatic reviews and decisions are supported with operational assessments developed by the ITT.

1.6. Integrated DT/OT (IDT/OT). IDT/OT is the Air Force process to transform test and evaluation in support of the current acquisition process. The IDT/OT concept centers on leveraging DT efforts for a given program to meet OT requirements into a process that streamlines and integrates test activities and events. IDT/OT facilitates the sharing of common operationally relevant data along with achieving an overall reduction in time and cost for OT and the reduction in risk during OT activities and events. IDT/OT requires a cradle-to-grave AFOTEC focus, and, IDT/OT success is dependent on the diligence of all AFOTEC personnel.

While the primary responsibility for executing IDT/OT activities resides in the AFOTEC detachments, the AFOTEC headquarters‘ staff also plays an integral role supporting every test director. IDT/OT success is based on the attributes of early and continuous collaboration between the warfighter (user), acquisition, and T&E communities, as well as the PEO (or product center or acquisition community) acknowledgement of AFOTEC as a stakeholder in DT events (see paragraph 4.23.2). Early and continuous IDT/OT activities and events increase programmatic confidence (e.g., schedule, requirements stability, funding, testability, and test capability requirements). AFOTEC will normally involve early (prior to MS A) and will start integrating with the other communities before concepts become formal programs of record and before identification of DT organizations (responsible test organization [RTO] nomination).

Product center test authorities (CTA) have been established at each product and logistics center.

The CTAs can be consulted in the absence of a program office T&E point of contact (POC). See attachment 14 for a pointer to CTA information on the AFOTEC-Internet. If a RTO is not yet identified, or fully onboard at ITT stand-up, AFOTEC will request the CTA participate in ITT activities in place of RTOs.

1.6.1. IDT/OT Planning and Collaboration. IDT/OT starts with the initial formation of the ITT to support IDT/OT planning and continues throughout the program. IDT/OT execution can make use of a combined test force (CTF). The only limiting factor to the integration is the statutorily required dedicated phase of OT&E conducted on a production representative system demonstrating stabilized performance in a realistic operational environment to inform full-rate production (FRP), initial operational capability (IOC), or fielding decisions. Early and frequent collaboration among all T&E stakeholders helps to ensure that common testing requirements can be satisfied and that data collected is available to all concerned.

Identification of IDT/OT requirements should be done early enough to be included in the developmental contract so that data resulting from contractor developmental testing will be available to the ITT (see paragraph 3.10). It is important to remember that throughout IDT/OT planning and execution, the individual DT and OT objectives should not be compromised. In the best of cases, a significant amount of operationally realistic data from the IDT/OT can be used to resolve OT&E requirements such that the remaining dedicated OT&E consists only of those activities and events not available in an IDT/OT environment.

1.6.1.1. The development of an IDT/OT concept is one approach to IDT/OT, as outlined in AFI 99-103. To support AFOTEC‘s IDT/OT culture, IDT/OT activities and events as well as data requirements and sources are included and highlighted in all OT&E plans. AFOTEC IOT&E plans will include specific DT events. Instead of looking back at DT and determining if an event was operationally relevant enough to apply to OT&E, AFOTEC should influence the development of the IDT/OT planning in such a way that DT will include events that will meet the needs of both DT and OT.

Product center commanders can expect AFOTEC OT&E plans to include IDT/OT activities and data requirements and sources, with OT&E plans being approved earlier in order to request and obtain product center acknowledgement.

1.6.2. IDT/OT Considerations. The fundamental goal behind IDT/OT is to maximize the availability of data and obtain operationally realistic data during DT. Test resources (money and people) and time continue to shrink. AFOTEC ITT members need to share test data and add operational realism when possible via integrated DT/OT activities and events. Test teams need to implement IDT/OT with a goal of achieving a process which integrates DT and OT events. NOTE: DoDI 5000.02 states "The Department of Defense may not conduct OT&E, including operational assessment (OA), IOT&E, or FOT&E, until the DOT&E approves, in writing, the OT&E portions of the T&E plan for programs on the OSD T&E Oversight List and the adequacy of the plans (including the projected level of funding) for the OT&E to be conducted in connection with that program.‖ The approval can be obtained through several vehicles such as the TEMP, an OA plan, or a separate memorandum from AFOTEC/CC outlining the OT portion of the IDT/OT.

1.6.2.1. The early collaboration with DT counterparts facilitates a total team effort to develop a testing strategy that integrates both DT and OT. Early T&E community identification of test requirements increases the potential to identify shortfalls, develop peculiar capabilities, and increases the preparedness for both DT and OT. Even if the RTO is not yet identified, AFOTEC states IDT/OT intent in all T&E documentation.

IDT/OT can also include other agencies, such as the JITC, to validate and certify technical compliance with compatibility, interoperability, and integration standards, or Air Force Information Operations Center (AFIOC) for information assurance (IA) procedures and activities.

1.6.2.2. Recognizing the earliest pre-MS A test design may be limited with regard to integrated test planning because little DT planning has occurred, integrated testing is still an action item on the test design agenda. Initial test design meetings are held with external core team participation. AFOTEC‘s IDT/OT intent is clearly stated from first contact with the external communities and remains a consistent theme throughout AFOTEC‘s involvement. Early inclusion of IDT/OT language in MS A test and evaluation strategies (TES) sets the tone for every acquisition. Many of the AFOTEC-developed measures of effectiveness (MOE) and measures of suitability (MOS) can be addressed during DT&E. AFOTEC can then focus on answering remaining measures, mission statements, COIs, and mission impacts during the remaining dedicated OT&E.

For example, if the system under test is in its near-final/final configuration during the last portions of DT&E, a fair amount of useable data can be obtained for the OT&E.

1.6.3. IDT/OT for Space Systems. IDT/OT is a key principle of the space systems T&E approach. Because systems already in space cannot be retrieved from their environment to fix hardware deficiencies discovered after launch, a significant amount of operational testing must be accomplished prior to launch. Close integration of DT and OT activities, must occur while satellites are still on the ground. The developer needs to be an integral member of the ITT. To facilitate developer participation, the request for proposal (RFP) needs to include appropriate language to support the IDT/OT process. It is important when working with DT to ensure contractor support is requested, coordinated, and documented through the respective system program office. The TEMP describes how all testing will be integrated and addresses the overall evaluation approach, key evaluation measures, and the major risks or limitations to completing the evaluations. IDT/OT must not compromise the evaluation of test results carried out by both developmental and operational testers.

1.7. Space Systems OT&E. The need for fully informed decisions regarding increasingly expensive, yet indispensible space system capabilities is crucial in today‘s environment of constrained resources. AFOTEC uses the Air Force Space Systems OT&E model to provide fact-based, decision-quality data to decision makers in time to support key acquisition decisions.

1.7.1. Acquisition of space major defense acquisition programs, which were granted a waiver to the DoD 5000 requirements, follow procedures contained in the Space Interim Guidance. The Air Force space test and evaluation (T&E) policy directs early activities and additional actions that provide earlier OT&E influence and better support space acquisition programs developed under the Space Interim Guidance. The Space Interim Guidance uses a streamlined acquisition framework with milestone decisions that occur earlier than typical DOD 5000-series milestones and decision reviews. Figure 1.5 depicts the acquisition process for space programs under the Space Interim Guidance and also shows the JCIDS process adaptations for space programs.

1.7.2. In a traditional acquisition program governed by Department of Defense Directive (DoDD) 5000.01, expenditures are relatively small in the Research and Development (R&D) and investment phases, compared to the cost of production and system operation. For these traditional acquisitions, OT occurs just prior to the major investment or production decision and provides data to inform those decisions adequately. Most of the investment for space systems occurs early in the program, most often without a major production decision. In the past, OT&E took place at the same point in the acquisition cycle as the DoDD 5000.01 programs. However, early in Space Interim Guidance programs, most of the investment has been made, most of the key acquisition decisions have been made, and the critical operational decision to launch the satellite has been made and executed. Additionally, the ground station and associated software often lag in deployment making timely post launch OT&E difficult, if not impossible. Making these key decisions prior to the execution of OT&E severely limits the value of OT&E.

Figure 1.5. Space Interim Guidance

1.7.3. AFOTEC‘s implementation of the Air Force space T&E model provides an OT&E that fits the space systems‘ acquisition model, delivering better value to both the acquisition and operational decision makers by moving OT&E activity well prior to launch. The three key tenets of the Air Force space T&E model are: early and continuous IDT/OT involvement throughout the system‘s life cycle; agile analysis and reporting; and focus on system-of-system evaluations.

1.7.4. AFOTEC‘s space test approach provides the basis for knowledge-based acquisition and operational decisions throughout the life cycle of our national security space systems.

The approach provides early operational involvement delivering a number of benefits as listed in table 1.1.

Table 1.1. Space Test Approach Benefits

Ensuring the warfighter receives needed mission capabilities.

Providing early clarity and continued update of operational requirements.

Influencing early and continual development and refinement of the CONOP.

Ensuring frequent reviews of threat documents to ensure the system design addresses current and future threats.

Highlighting program shortfalls and benefits throughout the development process when they can be addressed most efficiently and inexpensively.

Allowing the user to understand and accept acquisition risks and adjust their mission requirements and plans accordingly.

Addressing and correcting systemic suitability issues early in the program‘s development.

1.7.5. The Det/CC responsible for space systems needs to refer to the guidance and procedures contained in the remainder of AFOTECMAN 99-101 and comply with the AFOTEC processes as applicable to specific programs. Successful implementation of the

Interim NSSGuidance

PHASE B

Preliminary Design

PHASE A

Concept

Development

PHASE C

Complete

Design

BA

PDRSDRSRR

Pre - Systems Acquisition Systems Acquisition Sustainment

C

Upgrade Decision

Follow On Buy

Approval

1st Launch

FOCIOC

PHASE B

Approval

PHASE A

Approval

PHASE C

Approval

JROC

CDD

Pre KDP-A Concept Studies

JROC

ICD

Milestones:

JROC

CDD

Materiel Development Decision Mtg

Initial

CDD

CDR

JROC

CPD

Build Approval

PHASE D

Build & Operations space systems T&E approach is dependent on a clear understanding of the policy and procedural concepts contained in the Space Interim Guidance and AFI 99-103, Chapter 8, Space Systems T&E. NOTE: Using the space systems OT&E model for cyberspace/information technology systems or user equipment segment programs acquired under the Space Interim Guidance acquisition model may not be appropriate. The Det/CC responsible for space systems also needs to refer to the MOA on Multiservice OT&E for specific procedures regarding lead or supporting OTA activities.

1.8. Establishing a Relationship with DOT&E. Early in a program, the Det/CC and test director should identify and meet the DOT&E action officer assigned to the program if the program is on the OSD T&E Oversight List. Test directors must remember they are representing the AFOTEC/CC and any request to change an approved test plan or other special request from DOT&E shall be provided to the AFOTEC/CC in writing. For example, if DOT&E requests a change to the number of test articles previously determined adequate for OT&E, the TD should not commit. Instead, the TD will request the DOT&E AO formalize the request to the AFOTEC/CC in writing. Upon receipt of the request, the AFOTEC/CC will decide if the test plan will be changed. Frequent communication between the Det/CC, TD, and the DOT&E action officer is useful and important.

1.8.1. OSD T&E Oversight List. AFOTEC is the OTA for all Air Force programs on the OSD T&E Oversight List. Det/CCs, TDs, and technical advisors are responsible for knowing the oversight status of their programs and ensuring the current status is updated on AFOTEC‘s intranet site and AFOTEC‘s SIPR-intranet site.

1.9. Program Support by AFOTEC Liaison Officers (LNO). LNOs can greatly benefit AFOTEC by engaging with program offices, product and logistics centers, MAJCOMs, and other key agencies (e.g., DOT&E, AF/TE, JITC, test ranges, etc.) early in the requirements, alternatives, and acquisition processes. The LNOs should support early program activities by discovering emerging requirements and new acquisition programs, and by obtaining key program documentation. By providing program support, LNOs ensure AFOTEC‘s planning considerations are addressed, external offices are educated about the benefits of AFOTEC‘s involvement, and communications are enhanced between other organizations and AFOTEC. The AFOTEC Program Manager’s Operational Test Toolkit provides a value-added product for AFOTEC personnel to leave behind with program office and MAJCOM personnel.

1.9.1. AFOTEC Liaison Officers. The AFOTEC liaison officers co-located with Air Force Materiel Command or Air Force Space Command product centers or the Pentagon, act as an on-scene representative of the AFOTEC/CC and fulfill a critical role in the AFOTEC early influence process. These liaison officers accomplish the following actions:

1.9.1.1. Interface directly with their assigned product center counterpart and actively participate in the Product Center‘s acquisition plan process and test strategy formulation as directed via AFOTEC tasking orders. The LNO should actively look for the following: information to support TO preparation, new program basics (such as program name description and acquisition strategies), RTO, other POCs, related program documentation, program reviews and forecasts, and other information that may be helpful to establish collaborative IDT/OT planning efforts. Additional program information such as program priorities, requirements documents, concepts of operation (CONOPS), operating plans, information support plans (ISP) and tactics, techniques, and procedures (TTP) should be obtained as requested.

1.9.1.2. Notify AFOTEC/CC, detachments, and AFOTEC/A-3 of emerging projects/programs.

1.9.1.3. Filter project/program information and recommend involvement initiation when appropriate.

1.9.1.4. Notify AFOTEC/CC, detachments, and A-3 of issues of interest and key meetings requiring AFOTEC representation.

1.9.1.5. Attend product center meetings addressing program development.

1.9.1.6. Clarify AFOTEC policy and positions as required and refer questions to appropriate AFOTEC detachment or headquarters staff.

1.9.1.7. The Washington D.C LNO provides an entry point for contacts and meetings in the Pentagon or Washington, DC area.

1.10. Staffing and Coordination. Many different documents originating both inside and outside of AFOTEC will require staff and executive review. Detailed procedures for staffing and coordinating documents and briefings are contained in attachment 11. AFOTEC/CC requires prompt information update and exchange for OT&E activities. Det/CCs will provide programmatic updates at weekly stand-up or staff meeting when staff packages do not support time constraints. The weekly update serves to quickly inform the Command Staff and key personnel; it does not negate the requirement for officially staffed documents.

1.11. Meeting Attendance. OT planners attend various acquisition meetings either in person, via teleconference, or via video teleconference. Typical meetings include integrated product team (IPT) meetings, integrated T&E working group meetings, AoA meetings. Depending on the program, the titles of these meetings may vary. The common thread is that they are usually called by the program office director or staff, or a developing contractor. Core team/test team members should actively monitor program office activities for meetings that could be of benefit to test planning activities. Taking an active role and seeking out information on system program office (SPO) activities is beneficial and can lead to preventing surprises. When attending meetings, remember you are representing the AFOTEC/CC; when appropriate, give the most current AFOTEC Command Briefing (see attachment 14), and be prepared to ask questions and to give answers as a full participant in the meeting. AFOTEC/CC-approved positions (level of involvement, exit criteria, etc.) are not changed without approval of the AFOTEC/CC.

Commitments on behalf of AFOTEC are not made without prior authorization. If in doubt about an answer or topic, take an action item to follow-up. A trip report summarizing the important results of the meeting is provided to the AFOTEC/CC by the appropriate director or Det/CC within two duty days upon completion of the meeting. A template for the trip report format can be found on the AFOTEC-Intranet (see attachment 14).

1.12. External Briefings. The AFOTEC/CC is the release authority for all external briefings.

Per DoDI 5000.02, DOT&E requires a test concept briefing for DOT&E oversight programs at least 120 days prior to test start. DOT&E may request a briefing from AFOTEC to accompany the final test plan delivered 60 days before test start. If requested, the briefing associated with the final test plan submittal to DOT&E is coordinated prior to submitting to DOT&E. Final report briefings are provided to HQ USAF staff and the Office of the Secretary of Defense (OSD) as requested. External briefings presented to a program office, MAJCOM, or decision maker, should be done in-person. The test director is normally accompanied by an O-6 and a subject matter expert. Remember, when presenting an AFOTEC/CC-approved briefing externally any changes must be approved by the AFOTEC/CC. Prior to presenting an AFOTEC/CC-approved briefing to an external audience, the following guidance should be reviewed to structure the briefing for success (see table 1.2).

Table 1.2. External Briefing Checklist.

External Briefing Checklist Item Action Yes No

Pre-flight the briefing process For external briefings, pick a room, know who‘s attending (in person or electronically), have backup briefing materials (hard-copies) available.

Identify issues early for Det/CC or AFOTEC/CC involvement.

Verify the security arrangements, to include making sure that only cleared individuals are allowed access to the briefing.

Ensure MAJCOM and SPO reps are invited to (and attend!) AF/TE and DOT&E briefs … inviting the MAJCOM and SPO isn‘t exactly our job, but we must work with all the above and OL-DC to ensure all those organizations are present who can speak to issues that aren‘t in our lane.

Keep OL-DC in the loop for briefings at the Pentagon or the DC area.

Practice the briefing before the presentation.

You are conveying the AFOTEC/CC‘s message – stick to the approved subject material of the briefing.

Test team observations about the system are acceptable; don‘t make recommendations for system improvements.

Briefings discussing program requirements, capabilities or ratings will be transmitted by secure means. The AFOTEC-specific requirement does not change the classification of the briefing.

Be ready for backup presentation methods.

Avoid discussion outside of your lane.

If asked a question by the briefing recipient, listen to the complete question and then look to the senior AFOTEC officer for an indication to proceed with an answer.

Be prepared to answer the following questions: "who knows what you said besides you and the AFOTEC/CC?‖ and "what does the ORD/CPD say in relation to the results?‖

1.13. Information Protection. AFOTEC/CVI serves as the HQ point of contact for information protection (IP). The Det/CC and technical advisor at each detachment execute the guidance. IP involves safeguarding any data/information that potentially reveals U.S.

vulnerabilities, capabilities, capability gaps, or any data/information that falls under the freedom of information act (FOIA) exemptions (privacy act, export control, contractor owned information, etc.). IP includes the correct marking, classification, and handling of AFOTEC originated products (see attachment 14). The criteria for releasing information (to include review, coordination, and authority to release) is defined in AFI 61-201, AFI 61-204, and DoDD 5400.7/AF Supplement, as well as other directives such as DoDD 3200.12, DODI O-3600.02, DoDD 5230.23, DoDD 5230.25, Title 10, USC Chapter 3, Section 130.

1.13.1. AFOTEC Generated Documents.

1.13.1.1. The Det/CC and technical advisor provides IP guidance to test directors over the course of the program.

1.13.1.2. The detachment technical advisor ensures proper markings of working papers and data that support test reports.

1.13.1.3. AFOTEC/CVI, as IP POC, supports the detachments as needed.

1.13.2. Information Releases. Forward all requests for information through the appropriate POC, as outlined in table 1.3. The POC releases the information only after reviewing the distribution and release markings, coordinating through AFOTEC Electronic Coordination System (AECS), and obtaining approval, as appropriate. For further information on release of AFOTEC information, see attachment 9.

Table 1.3. POCs for Information Requests

Type of Request POC Coordination* DoD Agency HO N/A Agency not Authorized by the Distribution Statement HO FOIA Officer

FOIA FOIA Office

(A-6) HO/LC/PA/A-7K/CVI

Media Release PA FOIA Office/HO/LC/CVI Foreign Government CVI FOIA Office/HO/LC/PA/CVI

* Coordination may vary depending on the nature of the information requested.

1.13.3. Special Access Programs (SAP). The Special Test Division (A-3Z) is the HQ lead for SAPs. A-3Z maintains control of all SAP-associated activities and HQ functions except for Safety (SE), and Analysis, Assessments, and Lessons Learned (A-2/9). It is mandatory that any program contacts or issues that pertain to SAPs are brought to the attention of A-3Z.

The AFOTEC/CC provides direction for all SAP programs. All normal AFOTEC processes (within security constraints) will be performed for all SAPs. Coordination and staffing of special access required (SAR) documents and briefings within AFOTEC will occur via A-3Z.

1.14. AFOTEC SIPR-Intranet Requirements. The AFOTEC SIPR-Intranet provides AFOTEC with value-added information and features on a classified network to assist in the daily test management and reporting functions of OT&E. The AFOTEC SIPR-Intranet is a web-based application that integrates a variety of data sources and applications into a centralized information portal. AFOTEC SIPR-Intranet information is stratified into general support information, specialized directorate information, and program specific data. The AFOTEC SIPR-Intranet allows AFOTEC program managers (PM), test teams, and deployed test personnel to submit reports and documents throughout the life of the program. SAP documents are never loaded to the AFOTEC SIPR-Intranet. AFOTEC SIPR-Intranet information allows AFOTEC‘s senior leaders to have greater insight to upcoming events and prompt notification of significant occurrences, to include safety and mishap issues involving AFOTEC programs, personnel, or resources. A training manual is available on the AFOTEC SIPR-Intranet to assist users.

1.14.1. For each program on the AFOTEC SIPR-Intranet, the Det/CC ensures the test director (TD) publishes program information in the test management area under the "documentation‖ tab (see table 1.4). By keeping the AFOTEC SIPR-Intranet continually updated (typically every 30 days) an accurate documentation trail is available. The TD should replace old documents with new versions. Personnel will consult AFOTEC Legal Council (LC) when posting proprietary information on the AFOTEC SIPR-Intranet to address any potential restrictions. The TD will review and update information on the AFOTEC SIPR-Intranet and will certify test program information. The "last certified‖ date shall never be more than 30 days old. The TD ensures outdated information or preliminary drafts are properly archived. Changes to a program long/short title require AFOTEC/CC approval.

Table 1.4. Program Document Storage.

Storage Document

Developed, coordinated and stored on SIPRNET (including drafts/entire lifecycle)

All AFOTEC test plans and associated briefings All AFOTEC test reports and associated briefings All AFOTEC test readiness reviews, associated briefings and acknowledgement letters All Combined Test Force (CTF) reports and associated briefings

Stored on SIPRNET when the document is final

Agreements/Charters Analysis of Alternatives (AOA) Capability documents (ORD, ICD, CDD, CPD) Certification documents

CONOPS

Involvement letters Meeting minutes documenting decisions made Modeling and simulation plans and reports Schedules Tasking Orders Technical reviews and minutes Test and Evaluation Strategies (TES) Test and Evaluation Master Plans (TEMP)

1.14.2. The AFOTEC SIPR-Intranet documentation area of a test program is broken down into several categories as shown in table 1.5. Additionally, the test director (or PM/TDM) is responsible for maintaining the test program case file (TPCF) in the local detachment provided SIPRNet network drive. The TPCF is the official "cradle to grave‖ file and although it requires some duplication of effort, the SIPRNet network drive at each detachment is the government official records repository and the AFOTEC SIPR-Intranet is a protected web environment for AFOTEC information sharing. See attachment 13 for more information on TPCF requirements.

Table 1.5. AFOTEC SIPR-Intranet Program Documentation

Document Folder Typical Documents Agreements/ Charters MOAs/MOUs ITT and CTF Charters

Briefings Involvement Brief Program briefings

AFOTEC briefings (ITD, test plan, TRR, test report)

Certification Documents

Certification of System Readiness Letter

OT Acknowledgement Letter Pause Test Memo

Stop Test/Decertification Message Recertification Message or Letter

Correspondence/ e-mail Relevant correspondence and e-mail Meeting Minutes All minutes documenting decisions made Deficiency Reports Cat I deficiency reports only Miscellaneous Lessons Learned

Modeling & Simulation M&S Support Plans VV&A Plan/Report Accreditation MFR

Multimedia/ Graphic Items

Specific event video/photo data General video/photos

Program/ Requirements Documents

Requirements (ICD, CDD, CPD, RCT, SMM)

Programmatics (APB, ADM, AoA, PMD, Acquisition Strategy, T&E Strategy, CONOPS, ILSP)

Security (STAR, SCG)

Contract (CDRL, RFP, SOW) Classified document register (title/location only) Other (program office reference documents, contractor documents, etc.)

S…