AFMAN_63-119 _Certification_of_System_Readiness_for_Dedicated_Operational_Test_ _Evaluation.pdf

BY ORDER OF THE

SECRETARY OF THE AIR FORCE

AIF FORCE MANUAL 63-119

20 JUNE 2008

Acquisition

CERTIFICATION OF SYSTEM READINESS

FOR DEDICATED OPERATIONAL TESTING

COMPLIANCE WITH THIS PUBLICATION IS MANDATORY

ACCESSIBILITY: Publications and forms are available on the e-Publishing website at www.e-publishing.af.mil for downloading or ordering.

RELEASABILITY: There are no releasability restrictions on this publication.

OPR: HQ USAF/TEP Certified by: SAF/AQX (Mr. Blaise J. Durante) Supersedes AFMAN 63-119, 22 Feb 95 Pages: 74

This Air Force Manual (AFMAN) implements Air Force Policy Directive 63-1, Capability-Based Acqui-sition System. It implements a process for certification of system readiness for dedicated operational test and evaluation (OT&E) as required by Department of Defense Instruction (DODI) 5000.2, Operation of the Defense Acquisition System. National Security Space (NSS) Acquisition Policy 03-01 must also be consulted for space systems. Air Force Instruction (AFI) 99-103, Capabilities-Based Test and Evaluation, requires certification for additional types of operational testing beyond those required by DODI 5000.2.

AFI 63-101, Operation of Capabilities Based Acquisition System, also requires use of this AFMAN and certification process. Use this AFMAN with Air Force 10-, 13-, 14-, 16-, 21-, 33-, 63-, 91-, 99-, and other series publications to ensure full understanding of all policies, terms, and concepts. This AFMAN applies to all Air Force organizations involved in system acquisition or modification, including Air National Guard and US Air Force Reserve Command units and members. Send proposed supplements or changes to this AFMAN to SAF/AQXA, 1060 Air Force Pentagon, Washington DC 20330-1060; and HQ USAF/ TEP, 1530 Air Force Pentagon, Washington DC, 20330-1530. Ensure all records created as a result of pro-cesses prescribed in this AFMAN are maintained according to AFMAN 37-123, Management of Records (will convert to AFMAN 33-363), and disposed of according to Air Force Information Management Sys-tem (AFRIMS) Records Disposition Schedule (RDS) located at https://afrims.amc.af.mil/ rds_series.cfm.

SUMMARY OF CHANGES

This document was substantially revised and must be completely reviewed. It reflects changes in the DOD 5000-series documents, Chairman of the Joint Chiefs of Staff instructions (CJCSI), CJCS manuals (CJCSM), and numerous AFIs and Air Force Policy Directives (AFPD). New templates were added for Information Technology (IT) and National Security Systems (NSS), Test Plans That Are Integrated, and Integrated Test Teams (ITT). Other templates were renamed, combined, or eliminated to make the certifi-cation process more streamlined. All terminology was updated.

http://www.e-publishing.af.mil https://afrims.amc.af.mil/rds_series.cfm

2 AFMAN63-119 20 JUNE 2008

Chapter 1— OVERVIEW OF THE CERTIFICATION PROCESS 5

1.1. Overview

1.2. Applicability

1.3. Delegation of Certification Official

1.4. Responsibilities

1.5. Links to Reference Documents

Chapter 2— THE CERTIFICATION PROCESS 9

2.1. Overview

2.2. Template Subject Matter

Figure 2.1. Matrix of Certification Templates

2.3. Team Effort

2.4. Tailoring the Process

2.5. Continuous Process

Figure 2.2. Notional Timing of Certification Process Reviews

2.6. The Certification Review Cycle

2.7. Certification Memo Purpose

2.8. Updating the Templates

Chapter 3— TEMPLATE STRUCTURE AND USE 17

3.1. Interlocking Matrix

3.2. Consolidation of Multiple Sources

3.3. Answering Template Line Items

3.4. Focus on Ends, Not Means

3.5. Assigning Responsibilities

3.6. Certification Template Tracking Tool

3.7. Information Collection, Records, and Forms

Attachment 1— GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION 19

Attachment 2— ACQUISITION STRATEGY AND SCHEDULE 36

Attachment 3— ANALYSIS OF ALTERNATIVES (AOA) 37

Attachment 4— CAPABILITY BASED REQUIREMENTS DOCUMENTS (CBRD) 38

AFMAN63-119 20 JUNE 2008 3

Attachment 5— THREAT DOCUMENTS 40

Attachment 6— INTEGRATED TEST TEAM (ITT) STANDUP AND ITT CHARTER 41

Attachment 7— TEST AND EVALUATION STRATEGY 42

Attachment 8— AIR FORCE CONCEPTS 43

Attachment 9— LOGISTICS SUPPORT CONCEPTS (LSC) AND STRATEGIES 44

Attachment 10— INFORMATION TECHNOLOGY (IT) AND NATIONAL SECURITY

SYSTEMS (NSS) 45

Attachment 11— TEMP OR SAMP 47

Attachment 12— TEST PLANS THAT ARE INTEGRATED 48

Attachment 13— PROGRAM PROTECTION AND SECURITY 49

Attachment 14— CONTRACTOR TESTING 50

Attachment 15— DEVELOPMENTAL TEST AND EVALUATION (DT&E) 52

Attachment 16— SOFTWARE DEVELOPMENT AND MATURITY 54

Attachment 17— LIVE FIRE TEST AND EVALUATION (LFT&E) 55

Attachment 18— MODELING AND SIMULATION (M&S) 57

Attachment 19— CONFIGURATION MANAGEMANT PLAN (CMP) 58

Attachment 20— DEFICIENCY IDENTIFICATION AND RESOLUTION PROCESS 59

Attachment 21— PRODUCTION REPRESENTATIVE TEST ARTICLES 60

Attachment 22— SYSTEM PERFORMANCE 61

Attachment 23— OPERATIONAL TEST AND EVALUATION PLAN 62

Attachment 24— TEST AND EVALUATION RESOURCES 64

Attachment 25— PROGRAMMATIC ENVIRONMENT, SAFETY, AND

OCCUPATIONAL HEALTH EVALUATION (PESHE) 65

Attachment 26— OPERATIONAL TEST TEAM TRAINING 67

4 AFMAN63-119 20 JUNE 2008

Attachment 27— SUPPORT EQUIPMENT (SE) 68

Attachment 28— SUFFICIENCY OF SPARES 69

Attachment 29— SUPPORT AGREEMENTS 70

Attachment 30— PACKAGING, HANDLING, AND TRANSPORTATION 71

Attachment 31— PERSONNEL 72

Attachment 32— CONTRACTOR SUPPORT 73

Attachment 33— TECHNICAL DATA 74

AFMAN63-119 20 JUNE 2008 5

Chapter 1

OVERVIEW OF THE CERTIFICATION PROCESS

1.1. Overview. This AFMAN provides a structured mechanism for identifying and reducing risks associ-ated with transitioning from developmental test and evaluation (DT&E) to dedicated operational testing.

It establishes a disciplined review and “certification process” in the early stages of acquisition and modi-fication programs, and culminates in more successful operational test outcomes. The certification process is a tool to help acquisition managers at all levels identify risks, reach negotiated agreements on issues, and render more accurate assessments of system readiness to begin dedicated operational testing. The pro-cess is supported by 32 “templates” based on DOD and Air Force policy, historical information, best prac-tices, practical advice, and lessons learned from numerous acquisition programs. The certification process helps document the pursuit of a credible risk reduction program and an effective development program.

This certification process is mandatory and must be implemented as a continuous effort, not a single event in time.

1.2. Applicability. DODI 5000.2, Enclosure 5, requires the Services to establish a process for evaluating and determining materiel system readiness for initial OT&E (IOT&E). This process will be used for pro-grams on the Office of the Secretary of Defense (OSD) Test and Evaluation (T&E) Oversight List, acqui-sition category (ACAT)-designated programs on the Air Force non-space Acquisition Program Master List (APML), and space programs designated by the Air Force Service Acquisition Executive (SAE) for space. Sustainment programs on the Sustainment Program Master List (SPML) and other acquisition projects or sustainment actions falling below APML or SPML thresholds are highly encouraged to follow this process.

1.2.1. DODI 5000.2 requires the SAE to evaluate and determine materiel system readiness to enter the dedicated phase of IOT&E. It requires a review of DT&E results; an assessment of the system’s progress against the critical technical parameters (CTP) documented in the test and evaluation master plan (TEMP); an analysis of identified technical risks to verify that those risks have been mitigated during DT&E; and a review of the OT&E entrance criteria specified in the TEMP or other operational test plans.

1.2.2. The National Security Space system acquisition process described in NSS 03-01 is significantly different than the acquisition process in DoDI 5000.2 and AFI 63-101. NSS 03-01 uses a streamlined acquisition framework that causes the key decision points for NSS acquisition programs to be phased earlier than typical DoD 5000-series milestones and decision reviews. However, the basic T&E sup-port provided to NSS systems is similar to non-space systems for equivalent decision points. The tem-plates in this AFMAN should be modified accordingly. See NSS 03-01, paragraph AP1.1.6 and others, for additional guidance.

1.2.3. In addition, AFI 99-103, Chapters 2, 3, and 6, require Air Force program managers (PM) to use this certification process to evaluate system readiness for operational testing in support of a full-rate production (FRP) and/or fielding decision for all acquisition programs. The certification process will also be used for systems in sustainment that require an FRP and/or fielding decision. Note: Some sys-tems, programs, and activities may be exempt from this AFMAN according to AFI 99-103, paragraph 1.7.

6 AFMAN63-119 20 JUNE 2008

1.2.4. This AFMAN provides 32 “templates” in Attachment 2 through Attachment 33 that list spe-cific problem or risk areas that could hinder the smooth transition to and execution of dedicated oper-ational testing. Use these templates for reviewing important program details in order to assess program readiness in sufficient depth. Note: While use of the certification process is mandatory, the contents of each template are not mandatory and will not supersede existing DOD or Air Force policy or guidance. The templates should be used in parallel with, not substitutes for, formal DOD or Air Force policy and guidance.

1.2.5. For the purposes of this AFMAN, “dedicated operational testing” refers to that phase of opera-tional testing that must be conducted independently of developers and users in support of an FRP and/ or fielding decision. Program offices using an evolutionary acquisition strategy will need to repeat this certification process for each increment of capability developed, produced and/or fielded. Note: The direction in this AFMAN is based on parameters and descriptions given in AFI 99-103, Chapter 2, paragraph 2.6 et seq which explain the differences between various types of operational testing. These paragraphs establish the basis for determining when readiness certification will be required. This AFMAN is only required for the dedicated operational test portions of an integrated test program. It is not required for combined developmental and operational test activities.

1.2.6. This certification process will be the primary OT&E certification method for all programs, to include space systems covered by NSS 03-01, when the Air Force is the lead Service. For programs where the Air Force is not the lead Service, Air Force ITT members will adapt the Air Force process to flow into the other Service's certification process.

1.2.7. Prior to the early deployment of prototypes and ACTD/JCTDs, use the appropriate certification templates, modified as necessary, to review the system’s capabilities and limitations and its readiness for initial deployment. Going through the formal certification process would not be necessary in this situation.

1.3. Delegation of Certification Official. Although DODI 5000.2 requires the SAE to evaluate and determine system readiness for IOT&E, the applicable Air Force SAE (space and non-space) may dele-gate this authority in writing for assigned programs to a lower milestone decision authority (MDA) such as a program executive officer (PEO) or his/her Deputy for Acquisition. The applicable SAE will deter-mine the OT&E Certification Official for ACAT ID programs. OT&E Certification Officials for smaller programs originating at MAJCOM or Center levels may be delegated to a subordinate level as appropri-ate. Note: Under no circumstance shall a PM be the OT&E Certification Official for his/her own program.

1.4. Responsibilities. The certification process cuts across organizational lines and brings together stake-holders from the acquisition, requirements, developer, T&E, and sustainment communities. Other stake-holder organizations are responsible for providing test data, supporting information, studies, analyses, and candid feedback for assigned areas in support of the certification process. Each line item in the templates suggests a single “most likely” office of primary responsibility (OPR) for that item. Additional offices or organizations may also be involved, but only the OPR is cited. The following organizations or officials (or their representatives) are required to participate in the certification process.

1.4.1. OT&E Certification Official. The OT&E Certification Official, with advice from the inte-grated test team (ITT), and as designated in the TEMP, will determine the broad scope and require-ments for certifying system readiness to begin the dedicated phase of operational testing.

1.4.2. Program Managers (PM). PMs will:

AFMAN63-119 20 JUNE 2008 7

1.4.2.1. Ensure a robust systems engineering process is the underlying foundation for systems development and for reviewing these templates.

1.4.2.2. Ensure their system is mature and demonstrates stabilized performance in an operation-ally relevant environment prior to certification. Additionally, all necessary test support must be available and the system must have a high likelihood of a successful operational test.

1.4.2.3. Designate an OPR for organizing the certification process, gathering information, sched-uling reviews, assigning tasks, negotiating consensus on issues and solutions, assembling certifi-cation briefings, and drafting the final certification memo according to this AFMAN.

1.4.2.4. Document the strategy for the certification process in Part III of the TEMP.

1.4.2.5. Request additional stakeholder organizations to participate in this process as necessary to ensure acquisition program success.

1.4.3. Air Force Operational Test and Evaluation Center (AFOTEC). If AFOTEC is the opera-tional test agency (OTA), they will participate in the certification process by assisting the PM and car-rying out responsibilities as agreed. They will lead the effort to mobilize resources required for dedicated OT&E, and provide advice, test support, and test data to the PM and user throughout the development process. Note: For multi-Service programs, the certification policies of the lead Service will be used. If any Air Force operational test organization is the lead operational tester, this AFMAN will be used. If the Air Force is not the lead Service, this AFMAN may or may not be the governing document. Nonetheless, it should be used for Air Force portions of certification activities.

1.4.4. Major Command (MAJCOM) Operational Test Organizations. I f t he MA JC OM i s responsible for conducting operational testing, they will perform the same certification functions as AFOTEC would have performed. MAJCOMs will assist the PM in implementing this certification process for force development evaluations (FDE) or operational utility evaluations (OUE) when FRP and/or fielding decisions are planned. Note: The acronym “OTA” is used in the templates to denote either the MAJCOM operational tester or AFOTEC, whichever applies.

1.4.5. Lead Operating Command. The lead operating command, or using commands as appropri-ate, will participate in the certification process by assisting the PM and operational testers (i.e., AFO- TEC or MAJCOM) and carrying out responsibilities as agreed. The lead operating command will ensure capability requirements documents are complete and up to date according to AFI 10-601, Capabilities Based Requirements Development.

1.4.6. Responsible Test Organizations (RTO). The RTO or equivalent organization will conduct DT&E and support operational testing of systems according to AFI 99-103, AFI 63-101, and MAJ- COM policies. The RTO will participate in the certification process by providing sufficient analysis results and supporting data, operator comments, and recommendations to the PM to support the PM’s responsibilities in paragraph1.4.2..

1.4.7. HQ USAF Staff. Representatives from SAF/AQ, SAF/US, HQ USAF/TE, and others as needed will monitor the certification process for continued effectiveness and periodically update these templates as policy changes dictate. These staff members should attend certification proceedings when HQ USAF assistance is required.

1.4.8. OSD Staff. The Office of the Undersecretary of Defense, Acquisition, Technology, and Logis-tics (OSD(AT&L)) and OSD/DOT&E staff should closely monitor this process if the program is on OSD T&E Oversight.

8 AFMAN63-119 20 JUNE 2008

1.4.9. Other Stakeholder Organizations. Other support organizations and participating test organi-zations (PTO) should support the PM and the certification process as requested. Joint Interoperability Test Command (JITC) staff should be invited to participate for systems with net-ready key perfor-mance parameter (NR-KPP) (i.e., interoperability or information assurance) requirements.

1.5. Links to Reference Documents. The most current versions of documents referenced in this AFMAN are available electronically. For Air Force publications, check http://www.e-publishing.af.mil/ . For DOD publications, check http://www.dtic.mil/whs/directives/.

http://www.e-publishing.af.mil/ http://www.dtic.mil/whs/directives/

AFMAN63-119 20 JUNE 2008 9

Chapter 2

THE CERTIFICATION PROCESS

2.1. Overview. Proper risk management requires the development of a systematic, disciplined plan to identify problems and risks. A proven risk management technique is to examine the successes, failures, problems, and solutions of similar or past programs for "lessons learned" that can be applied to current programs. Another technique is to systematically comb through the entire program using specific decision criteria based on historical data. The certification process combines these techniques with a system for assigning responsibility and tracking accountability for results.

2.2. Template Subject Matter. The matrix of 32 certification templates in Figure 2.1., Matrix of Certi-fication Templates, covers a broad range of subjects that have historically impacted systems transitioning from DT&E to dedicated operational testing. Not all templates apply equally to every program; however, all templates should be considered for applicability at each review to ensure every relevant area at that point in time is covered. The initial template review should reveal where to begin working on long lead items that usually come to fruition much later in development programs. The templates are arranged in three notional groups in approximate chronological order: Test Planning and Documentation; System Design and Performance; and Test Assets and Support. The templates may be re-ordered as desired. These templates may be used in conjunction with the templates in Department of Defense (DOD) 4245.7-M, Transition from Development to Production. All templates are designed to increase the visibility of poten-tial risk factors and facilitate a streamlined, executive-level review.

A

FM

A

N

63-119 20 JU

N E

Figure 2.1. Matrix of Certification Templates.

AFMAN63-119 20 JUNE 2008 11

2.3. Team Effort. Since any risk reduction process is a team function, PMs must provide the right orga-nizational structure and continuous motivation to make it effective. Risk is mitigated only when condi-tions that contribute to risk are adequately addressed. These risk reduction efforts are typically within the scope, reach, and authority of certification process participants to effect necessary changes. These changes will typically occur at levels not normally visible to senior decision makers on a day-to-day basis.

2.4. Tailoring the Process. As early as practical, PMs and OT&E Certification Authorities should tailor the certification process to their need for information. The Certification Review Cycle, described in para-graph 2.6., should be repeated as often as necessary.

2.4.1. Templates Not Program Specific. Since the templates are not program specific, PMs and OT&E Certification Authorities may tailor them, with operational tester and operational command assistance, to fit specific programs or groups of programs. Some templates may require greater or lesser emphasis depending on the program and its phase of development. The templates give PMs maximum flexibility in focusing and structuring their reviews without losing sight of the original objective--providing an executive-level review of the program.

2.4.2. Tailoring Level of Detail. PMs may attach additional information or levels of detail to the templates at their discretion. Some examples might be exit and pass-fail criteria, action plans, require-ments thresholds, lists of acquisition regulations and standards, watch lists, breakdowns of specific line items, and points of contact. Additional templates can be developed to cover other areas. Addi-tionally, aggregation of templates and template line items can reduce redundancy and help managers concentrate on known risk areas. In short, tailor each certification program to attain the best results.

2.5. Continuous Process. The certification process must be viewed as a continuous effort, not a single event in time. It is not tied to any particular acquisition milestone or decision review; however, the final certification of system readiness briefing must be completed no later than 45 days prior to the planned start of dedicated operational testing, or as mutually agreed between the PM and operational testers. Any dedicated operational testing that supports an FRP and/or fielding decision must be supported by a readi-ness certification. Use the following guidelines:

2.5.1. Starting Early. Templates may be reviewed in any order that makes sense for the program and phase of development. All templates will be initially reviewed and considered for applicability. Those that are clearly not relevant to the program may be set aside. Templates previously set aside could become relevant again later as program dynamics change (e.g., when a capabilities based require-ments document (CBRD) is re-issued after an insertion of new technology). To be most effective, the certification process must begin as early as practical in new development programs.

2.5.2. Series of Reviews. The certification process is a series of reviews culminating in a final readi-ness briefing as shown in the notional diagram in Figure 2.2., Notional Timing of Certification Pro-cess Reviews. For example, certification reviews (and briefings, if required) should be planned as progress checks prior to early operational assessments (EOA) or before each operational assessment (OA). The final review and briefing should be scheduled not later than 45 days prior to the start of dedicated operational testing, or as mutually agreed. This series of reviews is designed to aid in resolving problems or correcting deficiencies as soon as they are discovered, rather than waiting until the final certification review and briefing where late remedial action could cause delays in the start of dedicated operational testing.

12 AFMAN63-119 20 JUNE 2008

Figure 2.2. Notional Timing of Certification Process Reviews.

NOTES:

All acronyms in this figure are defined in Attachment 1.

For applicability of this figure to NSS systems, see paragraph 1.2.1.

2.5.3. Initial Review. Early on, the PM may concentrate on templates grouped under Test Planning and Documentation as shown in Figure 2.1. These templates address pre-Milestone B areas of the acquisition process where early fixes to problems generate large future paybacks. The System Design and Performance group of templates focuses on activities that must be complete or nearly complete prior to MS C. The Test Assets and Support group of templates helps ensure all required assets come together before dedicated operational testing begins. All line items in each template are arranged chro-nologically as much as possible. The order of the templates may be changed as desired.

2.5.4. ITT Involvement. Early in the development program, each program’s ITT should consult with their OT&E Certification Official to determine how to structure and tailor the certification process.

The ITT should recommend the best forum for conducting the reviews and how frequently they should be done. Note: The ITT may not be the appropriate group for conducting the certification review itself due to the high-level nature of ITT membership versus the detailed nature of the material.

A suggestion is to form a special Operational Test Readiness Review Group consisting of the stake-holders outlined in paragraph1.4..

AFMAN63-119 20 JUNE 2008 13

2.5.5. Frequency of Reviews. The ITT should recommend to the OT&E Certifying Official how to tailor the reviews to the needs of the program. In general, the frequency of reviews should increase as the program approaches the final certification date. Early in the development program, a year between reviews may be sufficient, but as dedicated operational testing draws near, reviews should be spaced at much closer intervals. Certification reviews should be planned prior to all operational test activities such as an operational assessment (OA).

2.5.6. Final Certification. As a minimum, a final certification review and briefing should occur not later than 45 calendar days (or as mutually agreed) prior to the start of dedicated operational testing.

This lead time helps ensure sufficient time to fix weak areas before starting dedicated operational test-ing, and for preparation of the certification of readiness memo. A certification of readiness memo must be sent to the lead operational test organization a minimum of 15 days prior to the scheduled start of dedicated operational testing. These times ensure the operational testers have a minimum of two weeks to finalize their T&E resources and schedules. Longer or shorter times may be negotiated by mutual agreement.

2.6. The Certification Review Cycle. A systematic series of candid "review-assessment-negotia-tion-reporting" cycles will promote meaningful dialogue among developers, the operational tester, and the operational command(s). The certification review OPR will periodically issue a call for roundtable meet-ings, create an open forum for discussion, consolidate inputs from all participating organizations, and report results to participants and the OT&E Certification Official.

2.6.1. Pre-Certification Reviews. A series of thorough reviews of all operational capability-based requirements and resource needs is the first step in assessing a program's readiness to begin dedicated operational testing. Suggested review points are shown at the bottom of Figure 2.2. Each participant (i.e., subject matter expert) should review assigned areas of responsibility and intensify ongoing efforts to reach unmet goals. The purpose of multiple early reviews is to keep the PM and the OT&E Certification Official better informed as the program nears final certification. Thus, key issues and risks that impact operational testing can be identified earlier, and quality, timely direction and feed-back attained from the PM and OT&E Certification Official.

2.6.1.1. Subject matter experts should compare demonstrated system performance to required system performance and compare available resources to required resources. A coherent, complete linkage should extend from system/program requirements down through the planned methods and resources for demonstrating technical and operational performance. Any flaws, inconsistencies, contradictions, voids, or disconnects are potential issues and areas of risk. Accurate and complete inputs are needed from all participants.

2.6.1.2. For more complex programs and systems of systems, a greater number of pre-certifica-tion reviews may be needed before the final certification review and briefing.

2.6.2. Assessment. The reviewer should next assess the shortfalls identified in the templates for impacts on the dedicated operational test program. Candid assessments of the system's readiness (i.e., the risk of not passing dedicated operational testing) are crucial to the success of the certification pro-cess.

2.6.2.1. Standard for Judging Readiness. Every template and template line item uses the same “ideal standard” for assessing system readiness and risk level: "Will the system be ready for and successfully complete dedicated operational testing in this area?" This judgment should be based

14 AFMAN63-119 20 JUNE 2008

on the most current operational capability based requirements document, or on sound professional judgment if an operational requirement is not at issue. Any available exit criteria should be reviewed against the relevant military standards, specifications, and requirements. The cumulative total of all judgments about these risks will indicate if the complete system is ready for dedicated operational testing. This candid assessment is the heart of the certification process.

2.6.2.2. Develop Exit Criteria. Certification process participants must know what events must occur to achieve program goals before dedicated operational testing begins. Specific and testable performance-based exit criteria should be developed for each identified deficiency or issue. Satis-faction of the exit criteria in terms of demonstrated, stabilized system performance is the best means to ensure readiness for dedicated operational testing. If possible, use an "end-to-end system integration test" before starting dedicated testing to make DT&E more operationally relevant and to serve as a predictor of future operational performance. Subjective value judgments backed up by sound technical and military judgment may also be necessary. Areas judged "not ready" will require explanation and an action plan to reach the exit criteria.

2.6.2.3. If Standards Are Not Met. Some template line items may not reach the "ideal standard" (i.e., are not expected to be ready for dedicated operational testing) after close scrutiny. For exam-ple, technical orders (TO) are often unavailable, produced late, or incomplete at the start of dedi-cated operational testing. Limitations to test may remain despite best efforts to rectify shortfalls.

Although few unavoidable departures from the ideal standard may occur, these areas still require constant, long-term management attention. Negotiation of exit criteria and action plans will be required.

2.6.2.4. Deferred Requirements. If an evolutionary acquisition strategy is used, some capability requirements (and therefore the operational testing of those requirements) may require deferment to a later increment. These deferments may result from program cost-schedule-performance trade offs. Deferment of requirements must be coordinated and documented between the user and PM and eventually reflected in operational capabilities documents. Deferment of any operational test-ing will be summarized in the final certification briefing and memo.

2.6.3. Negotiation. Risk areas persisting after repeated reviews are likely to impact the conduct of operational testing. Certification process participants must negotiate workaround plans and solutions, or agree to some limitations on dedicated operational testing. The program office is the focal point for attaining negotiated consensus on managing risks. Workarounds and solutions must be in the best interests of the Air Force. Operational test organization officials must be satisfied that the strength, objectivity, and independence of operational testing will not be compromised, while the program office must retain sufficient management flexibility to find optimal solutions. Again, sound military and technical judgment is required to reach a corporate Air Force decision on when to proceed into dedicated operational testing.

2.6.4. Reporting. The program office is responsible for consolidating all participants' inputs and observations and preparing the certification briefing and/or report. Explicit action plans and exit crite-ria should be developed for each deficient area.

2.6.4.1. Final Certification Briefing. The length and format of the certification briefing are dis-cretionary and should be tailored to fit the acquisition or modification program. The order of the templates may be changed as desired. The final product should be an executive-level review of the entire program conveying enough information for senior decision-makers to make informed judg-

AFMAN63-119 20 JUNE 2008 15

ments of system readiness. The review must broaden senior leadership's perspective to the "macro" level where overall program risk is assessed along with supporting details, if required.

2.6.4.2. Reporting to the OT&E Certification Official. After reviewing the briefing or report, the PM will forward it to the OT&E Certification Official who is responsible for final certification of system readiness. The PM should brief the OT&E Certification Official not later than 45 days prior to the planned start of dedicated operational testing. Representatives from appropriate levels of the operational command(s), operational tester(s), RTO, and other participating organizations are required to attend the briefing.

2.6.4.3. Certifications for Evolutionary Acquisition (EA) Programs. Systems may be devel-oped using an EA strategy and fielded in increments of increasing capability over extended time periods. These systems require a final certification of readiness for each increment, followed by dedicated operational testing for that increment. The final certification for follow-on increments will be briefed to the OT&E Certification Official and a certification memo sent in the same man-ner and format as for prior certifications.

2.7. Certification Memo Purpose. The certification of readiness memo documents in writing the level of agreement among certification process participants and specifies the extent of system readiness for dedicated operational testing within stated constraints. It confirms the certification process was properly followed and that OT&E entrance criteria were attained. The certification memo must be sent a minimum of 15 days before the scheduled start of OT&E. It serves as a quantifiable benchmark of projected capa-bilities against which to check operational test results.

2.7.1. Contents. The OT&E Certification Official should not simply enumerate what was ready for dedicated operational testing and what was not ready, but summarize the critical areas and processes accomplished. The PM must organize the certification memo to parallel the program's tailored certifi-cation process and discuss any agreed-upon deferments or limitations to operational testing. As a min-imum, the PM must address the following areas:

2.7.1.1. Briefly describe the dedicated operational testing, OT&E entrance criteria, and which acquisition process phase(s) and increment the memo supports. Include anticipated operational test start and end dates.

2.7.1.2. Briefly describe how the certification program was structured and executed.

2.7.1.3. List the templates (or line items, if necessary) that are fully certified as ready for dedi-cated operational testing without caveats or limitations.

2.7.1.4. List the templates (or line items, if necessary) that are not ready or have qualifications and caveats and explain why. Describe any areas of elevated risk and how they were mitigated.

Describe any proposed action plans, workarounds, and exit criteria, if required.

2.7.1.5. List any test limitations or test deferrals, the rationale, and future plans to clear the limi-tations and/or deferrals. Note that approval of deferred items does not eliminate or alter the requirement for operational testing of those areas. Deferred items must be tested in subsequent operational testing, or the operational requirement document must be changed.

2.7.1.6. List any other system attributes not ready for OT&E or not expected to meet operational requirements (e.g., known deficiencies).

16 AFMAN63-119 20 JUNE 2008

2.7.1.7. List any major areas of disagreement with the operational test organization (OTA or MAJCOM), user(s), or other participants and the rationale.

2.7.2. Addressees. The PM will summarize the final certification briefing in a memo to the opera-tional test organization commander, with information copies to SAF/AQ or SAF/US as appropriate, HQ USAF/TE, HQ USAF/A3/5, HQ AFMC/A3/A2/5 or AFSPC/A3/A5 as appropriate, the capability director, the PEO, the RTO, operational/using MAJCOMs, and other participants. The memo will be released by the OT&E Certification Official.

2.7.3. Certification Acknowledgment. The operational test organization commander will acknowl-edge the certification memo before commencing dedicated operational testing. The acknowledgment memo allows the operational test commander the opportunity to concur or non-concur with the OT&E Certification Official’s assessment, and restate any reservations or positions on unresolved issues.

However, operational test commanders are obligated to start testing. Send the acknowledgment memo to the addressees listed in paragraph2.7.2.

2.7.4. Decertification and Recertification. Despite the developer's best efforts, systems may fail to perform as planned, and continuation of dedicated operational testing is not in the best interests of the Government. The OT&E Certification Official may decertify the system and return it to DT&E. Test organizations may recommend decertification to the PM and/or OT&E Certification Official. A decer-tification memo is required to the addressees listed in paragraph2.7.2. Before the system resumes ded-icated operational testing, the OT&E Certification Official must again certify the system via memo according to paragraph 2.7. after appropriate corrective actions have been taken. If a system is decer-tified, all relevant templates should be revisited and the process tailored, if necessary, to improve future certification reviews of the system.

2.7.5. Alternative to Decertification. For system problems of a less serious or temporary nature, the operational test organization may “pause” or stop testing for a brief time to assess the the problem and determine if additional DT&E is warranted. Note: A series of “pauses” may indicate more serious problems that require system decertification.

2.8. Updating the Templates. The certification process and templates are expected to mature through feedback from certifications and as the acquisition process continues to evolve. Further changes will result from advanced technologies, improved test and evaluation methods, revised acquisition procedures, and restructure of the DOD test and evaluation. All certification process users should forward their obser-vations and suggested improvements to HQ USAF/TEP and SAF/AQXA. Feedback is essential to keep the process and templates up to date.

AFMAN63-119 20 JUNE 2008 17

Chapter 3

TEMPLATE STRUCTURE AND USE

3.1. Interlocking Matrix. The templates form a matrix of interlocking subject areas spanning an entire acquisition or modification program. Each template introduces order and helps reduce risk in a specific phase or aspect of a program. Some duplication and cross-referencing between templates are necessary because acquisition and modification programs rely on many overlapping activities. Decisions about risk in one area often affect other areas. Cross-referencing also facilitates broad area reviews as well as special subject area reviews. Closely associated templates are cited (e.g., “See A15”) to help find parallel infor-mation in other templates. Note: Acronyms and abbreviations are not spelled out the first time used in the templates (Attachment 2 through Attachment 33) in order to reduce template size and volume. The tem-plates are intended as checklists to facilitate the review and help structure an executive-level briefing. All acronyms and abbreviations are described in Attachment 1.

3.2. Consolidation of Multiple Sources. Each template consolidates as much practical information as possible from multiple sources into a succinct "checklist." Only a few of the most important AFIs, CJC- SIs, DODIs and DODDs are cited as footnotes to each template since complete document lists are imprac-tical for this type and level of review, and different groups of documents may apply to different programs.

Programmatic and regulatory details are left to OPRs and collateral agencies more thoroughly conversant with specialized guidance. Citation of minimum detail should help PMs, OT&E Certification Authorities, testers, and users stay squarely focused on quality and readiness issues at the executive-level of review.

3.3. Answering Template Line Items. Each template contains line items phrased as statements of fact rather than questions. Each line item should elicit a brief summary of program status in that subject area rather than a superficial "yes" or "no" response. The word “system” refers to software as well as hardware components of the program under review. The entire group of statements covers the template subject area, but further analysis may be required in certain cases. Line items may be answered individually or in groups depending on how the certification OPR, ITT, and/or PM tailor the certification process. Each template can function as a "tailored checklist" and as a road map for future activities in preparation for dedicated operational testing. As a general rule, aggregation of line items (and even whole templates) should increase as the review and briefing rise through the chain of command. The templates may be re-ordered as desired.

3.4. Focus on Ends, Not Means. The templates emphasize "what must be done" rather than "how to do it." No specific problem solving methods are advocated, leaving PMs maximum flexibility to implement their own "best practices." Certification participants are free to discuss and decide on the best way to rem-edy identified problems. The templates focus on the ends, and the participants focus on the means.

3.5. Assigning Responsibilities. A single lead OPR is suggested for each line item on all templates to assist PMs and other participants focus responsibility and increase accountability for results. Final deter-mination of each OPR should be made as required to improve organizational efficiency based on who is best suited to complete each task or final product. Final approval authority for some line items may lie at higher levels. While other agencies are expected to participate on a collateral basis, multiple OPRs and offices of collateral responsibility (OCR) are not listed since responsibility would be defocused, and all variations between programs cannot be covered. Once identified and agreed upon, the OPR must produce

18 AFMAN63-119 20 JUNE 2008

a high quality review in the assigned areas and gain the required level of participation from OCRs. The PM is the OPR for ensuring the entire certification process is properly executed.

3.6. Certification Template Tracking Tool. An automated certification process tracking tool for all templates is available on the HQ USAF/TE portion of the Air Force Portal. Modify this tool as needed to match any changes made to the templates. Also see SAF/AQ’s web site, www.safaq.hq.af.mil/.

3.7. Information Collection, Records, and Forms.

3.7.1. No information collections are created by this publication.

3.7.2. Program records created as a result of the processes prescribed in this publication are main-tained according to AFMAN 37-123 (will convert to AFMAN 33-363) and disposed of according to the AFRIMS RDS located at https://afrims.amc.af.mil/rds_series.cfm.

3.7.3. Prescribed Forms: No forms are prescribed by this publication.

3.7.4. Adopted Forms:

AF IMT 847, Recommendation for Change of Publication

SF368, Product Quality Deficiency Report

Sue C. Payton Assistant Secretary of the Air Force (Acquisition) http://www.safaq.hq.af.mil/

AFMAN63-119 20 JUNE 2008 19

Attachment 1

GLOSSARY OF REFERENCES AND SUPPORTING INFORMATION

References

Title 10, U.S.C. §139, §2302(5), §2366, and §2399, Operational Test and Evaluation of Defense Acquisi-tion Programs

DODD 3200.11, Major Range and Test Facility Base (MRTFB)

DODD 5000.1, The Defense Acquisition System

DODI 5000.2, Operation of the Defense Acquisition System

DOD 4245.7-M, Transition from Development to Production

DODD 5200.1, DOD Information Security Program

DOD 7000.14-R, Department of Defense Financial Management Regulations (FMRS), Vol 2A

DODI 8500.2, Information Assurance (IA) Implementation

DODI 5810.01, DOD Information Assurance Certification and Accreditation Process (DIACAP)

National Security Space (NSS) Acquisition Policy 03-01

Joint Publication (JP) 1-02, Department of Defense Dictionary of Military and Associated Terms

CJCSI 3170.01, Joint Capabilities Integration and Development System

CJCSM 3170.01, Operation of the Joint Capabilities Integration and Development System

CJCSI 6212.01, Interoperability and Supportability of Information Technology and National Security Systems

AFDD 1-2, Air Force Glossary

AFPD 10-28, Air Force Concept Development

AFI 10-601, Capabilities Based Requirements Development

AFI 10-602, Determining Mission Capability and Supportability Requirements

AFI 10-401, Air Force Operations Planning and Execution

AFI 14-111, Intelligence in Force Modernization

AFI 14-206, Modeling and Simulation

AFI 16-1001, Verification, Validation and Accreditation (VV&A)

AFI 16-1002, Modeling and Simulation (M&S) Support to Acquisition

AFI 21-102, Depot Maintenance Management

AFI 21-115 (Interservice), Product Quality Deficiency Report Program

AFI 21-302 (Interservice), Technical Data Requirements for Logistics Support

AFI 21-303, Technical Orders

20 AFMAN63-119 20 JUNE 2008

AFI 25-201, Support Agreements Procedures

AFI 32-7061, The Environmental Impact Analysis Process

AFI 32-7086, Hazardous Materials Management

AFPD 33-1, Information Resources Management

AFPD 33-2, Information Assurance Program

AFI 36-2201, Developing, Managing, and Conducting Training

AFPD 33-3, Information Management

AFMAN 37-123, Management of Records (will become AFMAN 33-363)

AFI 40-402, Protection of Human Subjects in Biomedical and Behavioral Research

AFPD 63-17, Technology and Acquisition Systems Security Program Protection

AFI 63-101, Operation of the Capabilities Based Acquisition System

AFI 63-104, The SEEK EAGLE Program

AFI 63-107, Integrated Product Support Planning and Assessment

AFI 63-125, Nuclear Certification Program

AFI 63-501, Air Force Acquisition Quality Program

AFI 63-1101, Modification Management

AFI 63-1201, Life Cycle Systems Engineering

AFI 65-601, Vol I, Budget Guidance and Procedures, Chapter 14

AFI 90-901, Operational Risk Management

AFPD 91-4, Directed Energy Weapons (DEW) Safety

AFI 91-202, The US Air Force Mishap Prevention Program

AFI 91-204, Safety Investigations and Reports

AFI 99-103, Capabilities Based Test and Evaluation

AFI 99-109, Test Resource Planning

TO 00-35D-54, USAF Deficiency Reporting, Investigation, and Resolution

TO 00-5-1, Air Force Technical Order System

TO 00-5-3, Air Force Technical Manual Acquisition Procedures

MIL-STD-882D, Standard Practice for System Safety

National Security Telecommunications and Information Systems Security Policy (NTISSP) No.11 and No.12

Memorandum of Agreement [MOA] on Multi-Service Operational Test and Evaluation (MOT&E) and Operational Suitability Terminology and Definitions

Glossary, Defense Acquisition Acronyms and Terms, 11th ed, Sept 2003, Defense Acquisition University

AFMAN63-119 20 JUNE 2008 21

Defense Acquisition Guidebook

DOD Guide for Achieving Reliability, Availability, and Maintainability

IT Lean Guidebook

Test and Evaluation Management Guide, 5th ed., Jan 2005, Defense Acquisition University

Abbreviations and Acronyms

A—Attachment

ACAT—Acquisition Category

ADM—Acquisition Decision Memorandum

AFDD—Air Force Doctrine Document

AFI—Air Force Instruction

AFMAN—Air Force Manual

AFMC—Air Force Materiel Command

AFMSRR—Air Force Modeling and Simulation Resource Repository

AFOTEC—Air Force Operational Test and Evaluation Center

AFPD—Air Force Policy Directive

AFRIMS RDS—Air Force Information Management System Records Disposition Schedule, https://afrims.amc.af.mil/rds_series.cfm

AFROCC—Air Force Requirements for Operational Capabilities Council

AFSPC—Air Force Space Command

Ao—Operational—Availability

AoA—Analysis of Alternatives

APB—Acquisition Program Baseline

APML—Acquisition Program Master List

BDRSK—Battle Damage Repair Spares Kit

C—a symbol for "contractor"

C4—Command, Control, Communications, and Computers

C4ISR—Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance

CBRD—Capabilities Based Requirements Document (this acronym used for this AFMAN only)

CDD—Capability Development Document

CDR—Critical Design Review

CJCSI—Chairman of the Joint Chiefs of Staff Instruction

22 AFMAN63-119 20 JUNE 2008

CJCSM—Chairman of the Joint Chiefs of Staff Manual

CLS—Contractor Logistics Support

CMP—Configuration Management Plan

COA—Courses of Action

COI—Critical Operational Issue

COMSEC—Communications Security

COOP—Continuity of Operations Plan

COTS—Commercial-Off-The-Shelf

CPD—Capability Production Document

CTF—Combined Test Force

CTP—Critical Technical Parameters

DAA—Designated Approving Authority

DAB—Defense Acquisition Board

DAG—Defense Acquisition Guidebook

DAU—Defense Acquisition University

DCR—DOTMLPF Change Recommendation

DEW—Directed Energy Weapons

DIA—Defense Intelligence Agency

DIACAP—DOD Information Assurance Certification and Accreditation Process

DITSCAP—DOD Information Technology Security Certification and Accreditation Process

DOD—Department of Defense

DODD—Department of Defense Directive

DODI—Department of Defense Instruction

DOTMLPF—Doctrine, organization, training, materiel, leadership and education, personnel, and facilities

DR—Deficiency Report, Deficiency Reporting

DRR—Design Readiness Review

DT&E—Developmental Test and Evaluation

EA—Evolutionary Acquisition

EDM—Engineering Development Model

EOA—Early Operational Assessment e.g.—for example

ESOH—Environment, Safety, and Occupational Health

AFMAN63-119 20 JUNE 2008 23

et. seq.—and all that follows

FCA—Functional Configuration Audit

FDE—Force Development Evaluation

FMRS—Financial Management Regulations

FOC—Full Operational Capability

FOT&E—Follow-on Operational Test and Evaluation

FRP—F…