Bidders Library Operational Test and Evaluation - DOTE TEMP Guidebook.pdf

OPERATIONAL TEST

AND EVALUATION

OFFICE OF THE SECRETARY OF DEFENSE

1700 DEFENSE PENTAGON

WASHINGTON, DC 20301-1700

JAN 1 9 2017

MEMORANDUM FOR USERS OF THE DIRECTOR, OPERA TONAL TEST AND

EVALUATION (DOT&E) TEST AND EVALUATION MASTER

PLAN (TEMP) GUIDEBOOK

SUBJECT: DOT&E TEMP Guidebook 3.1

DOT&E TEMP Guidebook 3.1 updates the Design of Experiments (DOE), Scientific Test and Analysis Techniques (STAT), Mission-focused Metrics, Operational Evaluation Framework (OEF), Modeling and Simulation (M&S) for Test and'Evaluation, Defense Business Systems, Cybersecurity, and Software-Intensive Systems sett{ons of the DOT&E TEMP Guidebook 3.0. Version 3.1 also contains a new Survey Design.guidance section consistent with the 6 January 2017 DOT &E Survey Pre-testing and Administration memo.

The DOE and STAT sections now highlight the importance of justifying resources, especially long-lead items, using experimental design techniques at Milestone A and in TEMPs supporting Requests for Proposals. The Mission-focused Metrics section was renamed Quantitative Mission-focused Measures. The terms "metrics," "measures," and "response variables" have been used interchangeably in the Mission-focused Metrics, DOE, and STAT sections which has led to some confusion among readers. Thus where it makes sense, we replaced the terms "metrics" and "response variables" with "measures." We added the term "quantitative" to Mission-focused Measures to highlight the importance of evaluating systems with quantitative measures as opposed to qualitative measures.

The cybersecurity guidance includes a statement reminding readers that a Cooperative Vulnerability and Penetration Assessment and Adversarial Assessment are normally required as part of an operational test or assessment supporting a fielding decision. The command and control cybersecurity example now contains a detailed table describing cyber defenders' roles and responsibilities.

Minor content updates were made in the Software-intensive Systems section to align the discussion with Department of Defense Instruction (DoDI) 5000.02 as compared to the interim

DODI 5000.02.

That STAT and Defense Business Systems sections now include DOT&E survey content expectations in TEMPs, along with dedicated survey guidance. The Defense Business Systems section also now discusses the importance of describing software change requests when explaining the defect tracking process.

The measures of merit discussion in the OEF guidance was incorporated into the STAT section.

The M&S section was updated to reflect the 14 March 2016 and 17 January 2017 DOT &E guidance memoranda on the validation of M&S used in operational test and live fire assessments.

Program Managers will use the TEMP as the primary planning and management tool for all test activities starting at Milestone A. Program Managers will prepare and update the TEMP as needed and to support acquisition milestones or decision points. The TEMP should be specific to the program and tailored to meet program needs. Accordingly, the guidance in this guidebook, in DoDI 5000.02, and in the TEMP format are provided to assist in developing the appropriate TEMP format and content for each program. Strict or immediate adherence to the new TEMP format is not required. Use common sense to apply the guidance to fit your program. Evaluation of TEMP adequacy is based on the TEMP's content, not the format.

Questions or suggestions about this guidebook should be addressed to Dr. Catherine Warner, catherine.w.warner.civ@mail.mil, 703-697-3655.

jMic~i~ dSLector

OPERATIONAL TEST

AND EVALUATION

OFFICE OF THE SECRETARY OF DEFENSE

1700 DEFENSE PENTAGON

WASHINGTON , DC 20301-1700

NOV 16 2

MEMORANDUM FOR USERS OF THE DIRECTOR, OPERATIONAL TEST AND

EVALUATION (DOT&E) TEST AND EVALUATION MASTER

PLAN (TEMP) GUIDEBOOK

SUBJECT: DOT&E TEMP Guidebook 3.0

This new version ofthe DOT&E TEMP Guidebook complements the January 2015 version of DoD I 5000.02 by illustrating with selective guidance and examples how to develop

. and document an adequate test and evaluation (T &E) strategy. The Program Manager will use the TEMP as the primary planning and management tool for all test activities starting at Milestone A.

Best practices outlined in this TEMP Guidebook should be applied to all versions of the TEMP, including the Development Request for Proposal (RFP) TEMP.

The Program Manager will prepare and update the TEMP as needed and to support acquisition milestones or decision points. The TEMP should be specific to the program and tailored to meet program needs. Accordingly, the guidance in this guidebook, in DoDI 5000.02, and in the TEMP format guide are provided to assist in developing the appropriate TEMP format and content for each program. Strict or immediate adherence to the new TEMP format is not required. Use common sense to apply the guidance to fit your program. Evaluation of TEMP adequacy is based on the TEMP' s content, not the format.

Summary of the TEMP and TEMP Guidebook Format

The TEMP format has been changed as illustrated below. The previous TEMP format on the left explained in sentences and paragraphs what DOT &E required for adequacy. TEMP Guidebook 2.1 added colored callout boxes with links to the DOT &E Guidebook guidance and examples.

The new TEMP format on the right enumerates in bullets what should be considered for inclusion in each paragraph/section of the TEMP. Callouts with links to DOT&E guidance in 'the Guidebook 3.0 are in bold blue font.

Previous TEMP and Guidebook 2.1 Format 1 2. MiSSKif'l Description Briefty summanze the miss•on need de.criMd tn the program capability reQUirements documents in terms of the capability it will ptOVtde 10 the JOint Forces Commander Describe the m•ssl()(l to be accomplished by 11 ~o.rut equipped with the system using a l applable CONOPS and Concepts Of EmplOyment. IOCOfl)OI'8te an ov-1 Of the system ShOoMnO the mtended operational e~"~V~ronment. Also 1ncJud• thft orgarnutzon In whiCh the system will be integrated as well as signifiCant points from the life Cycle Sustairment Plan. the Information Support Plan, and Program Protection Plan Provide ~nks to each document referenced in the Introduction. FOf business systems, 1rducle a summary of the r>usiness c.asell"\iilysls tor the' program

1.3. System Oesoiptlon. Oesaibe the system c:ot'lfigl.ntio. lderhfy key features and subsystems. both t'lardwara and software (such as arc::h!tecture.

system and UHt interlaces. HO.I'rty levels, and teserves) for lhe ptamed increments Within the Future Years Defense Program (FYOP)

1.3. 1. System Threat Ass.,sment. Sucetnctly sununanze the threat envtronm.m (to Include cybeir-thre.t.s) in wt'ieh lhe system 'lllf'iH operate.

Reference the appropn.te OIA CW' component-validated threat documents for the sySiem.

1 3 2 Program Background. Refentnoe the AnalySis of .AJtematl:Yn (AoA.) . the APB and the materiel de\lebpment decision to provide background Information on the proposed system Bnefly desO"Ibe the overarc:t.ng AcqUisition Strategy (for space systems, the Integrated Program Summary (IPS)), and the Technology Development Strategy (TOS) Addl'ess whether tM system WIU be

=:u~r~i~an~V::::~·;=:,~ ::;::=!~~~:~~~

Ex•roo19 Is>! t 1 3 1 lnlotnlltion Aas~~tanc• (CyberMCutityJ -El5IITJ!Iesbf1 31

New TEMP and Guidebook 3.0 Format

1.2. MISSION DESCRIPTION

1.2. 1 Mission OveMew

Sunmarize the mission need described in the progam cap;Jbility requirements documents in terms of the capo~bility the system will provide to the Warfighter.

Oesaibe the mission to be ac~~shed by a unit that wil be equ;pped with the system.

lnc:c<pota1e an Operation;:ll VitM (OV-1) of the syst<rn showing the intended opernlional Oflviroomenl ln<ixfe sign1ficont points from the LJte Cycle Susta"'"""'t Pion, the Information Support Pion, ond the Progr;vn Protection Pion.

For business systems, incJOOe a summary of the business C3!0e ~lysis for the program.

1.2.2 Conupe of Operations

• Reference all 3JIIIIicable Coocepts of Operations oro Concepts of <rnploymen in describing the O'issoon. Describe test ompfications o CONOPS~and~

1.2.3 Operatioool users

Describe the intended users of the system, how they will eiT'I)foy the

=.:r~:::.=:c:.:':::=,~~·g .

The callouts have been placed throughout TEMP Guide 3.0 at locations where DOT&E and other applicable policies apply. Keep in mind that the examples are notional and apply to a specific or notional system, not to every system. In preparing your TEMP, you should apply the policy guidance and not simply copy the examples provided. The examples might not be appropriate for your system. The policy guidance contains additional links to the source policy documents if you wish to further investigate the underlying policy.

Summary of Milestone A TEMP Requirements in the January 2015 DoD I 5000.02

The Milestone A TEMP should address all major sections of the TEMP outline, but some of the details in the TEMP format may not be mature until Milestone B. The Milestone A TEMP should be complete enough to estimate and plan for the major resources required for adequate test and evaluation. Other specifics that should be included in the Milestone A TEMP include:

• Operational rationale for requirements. A link or reference to the capabilities development document (CDD) or similar document that provides rationale for requirements would be sufficient.

• For software acquisitions, an analysis of operational risk to mission accomplishment covering all planned capabilities or features in the system. The analysis will include commercial and non-developmental items.

• All planned T &E for phase completion. Major test events should have test entrance and test completion criteria.

• A table of independent variables (or "conditions," "parameters," "factors," etc.)

that may have a significant effect on operational performance.

• Strategy and resources for cybersecurity T &E.

Summary of Milestone B and Subsequent TEMP Requirements in the January 2015 DoD I 5000.02

Regarding operational and live fire testing, the Milestone Band subsequent TEMPs should be updated to address all plans of the T &E strategy until system deployment. The detailed focus of each TEMP should be on plans for the Developmental Test and Evaluation (DT &E), Live Fire Test and Evaluation (LFT &E), and Operational Test and Evaluation (OT &E) supporting the next major acquisition decision. In addition to updating the Milestone A content, the Milestone Band subsequent TEMPs should include:

• Expand on details of each LFT &E and OT &E phase/test to include cybersecurity testing.

• Expanded use of scientific and test analysis techniques to design effective and efficient testing.

• Reliability Growth Curves (RGCs) or Software Tracking metrics, updated RGCs (if applicable) that reflect test results to date, and a working link to the Failure Modes, Effects and Criticality Analysis (FMECA) data. A software defect or failure tracking database may replace the FMECA in software acquisitions.

• Operational evaluation framework that shows how the major test events and test phases link together to form a systematic, rigorous, and structured approach to evaluating mission capability across the applicable values of the independent variables.

• The updated table of variables will include the anticipated effects on operational performance, the range of applicable values (or "levels," "settings," etc.), the overall priority of understanding the effects of the variable, and the intended method of controlling the variable during test (uncontrolled variation, hold constant, or controlled systematic test design).

• Plans for Verification, Validation, and Accreditation if applicable.

• Appropriate cybersecurity measures to evaluate operational capability to protect, detect, react, and restore to sustain continuity of operation. The TEMP will document the threats to be used, which should be selected based on the best current information available from the intelligence community.

• Complete test resource requirements. Resources will reflect the best estimate for conducting all test activities. Resources will be mapped against the developmental and operational evaluation frameworks and schedule to ensure adequacy and availability. Ensure that resource estimates identified in the TEMP are matched against the schedule and justified by analysis.

Summary of the TEMP Outline from the January 2015 DoD I 5000.02

As before, the four major sections of the TEMP remain:

• Part I - Introduction

• Part II- Test Program Management and Schedule

• Part III -Test and Evaluation Strategy and Implementation

• Part IV -Resources Summary.

• Appendices may be added as needed for Scientific Test and Analysis Techniques, Cybersecurity, and Reliability.

Questions or suggestions about this guidebook should be addressed to Dr. Catherine Warner. She may be reached at Catherine.W.Warner.civ@mail.mil or (703) 697-3655.

d·M~;::::--- Director

Director, Operational Test and Evaluation (DOT&E) Test and Evaluation Master Plan

(TEMP) Guidebook

Version 3.1

19 January 2017

TEMP Guidebook Table of Contents

PART 1 – Introduction

1.1 PURPOSE

1.2 MISSION DESCRIPTION

1.2.1 Mission Overview

1.2.2 Concept of Operations

CONOPS Guidance and Examples

1.2.3 Operational Users

1.3 SYSTEM DESCRIPTION

1.3.1 Program Background

1.3.2 Key Interfaces

1.3.3 Key Capabilities

1.3.4 System Threat Assessment

Threat Representation Guidance and Examples

Cybersecurity OT&E Guidance and Examples

1.3.5 Systems Engineering (SE) Requirements

Reliability Growth Guidance

1.3.6 Special Test or Certification Requirements

Threat Representation Guidance and Examples

Cybersecurity OT&E Guidance

1.3.7 Previous Testing

LFT&E Strategy Guidance

PART II – Test Program Management and Schedule

2.1 T&E MANAGEMENT

2.1.1 T&E Organizational Construct

LFT&E Strategy Guidance

2.2 COMMON T&E DATA BASE REQUIREMENTS

2.3 DEFICIENCY REPORTING

Defense Business Systems Guidance and Examples

2.4 TEMP UPDATES

2.5 INTEGRATED TEST PROGRAM SCHEDULE

Figure 2.1 – Integrated Test Program Schedule

PART III – Test and Evaluation Strategy and Implementation

3.1 T&E STRATEGY

Integrated Testing Guidance and Best Practices

3.1.1 Decision Support Key

3.2 DEVELOPMENTAL EVALUATION APPROACH

3.2.1 Developmental Evaluation Framework

3.2.2 Test Methodology

3.2.3 Modeling and Simulation

3.2.4 Test Limitations and Risks

Test Limitations Guidance and DT Examples

3.3 DEVELOPMENTAL TEST APPROACH

3.3.1 Mission-Oriented Approach

Integrated Testing Guidance and Best Practices

3.3.2 Developmental Test Events and Objectives

Integrated Testing Guidance and Best Practices

Software Algorithm Testing Guidance and Examples

Reliability Growth Guidance

Cybersecurity OT&E Guidance

3.4 CERTIFICATION FOR INITIAL OPERATIONAL TEST AND EVALUATION (IOT&E)

IOT&E Entrance Criteria Guidance and Examples

3.5 OPERATIONAL EVALUATION APPROACH

Mission Focused Evaluation Guidance and Examples

Baseline Evaluation Guidance with Best Practices

End-to-End Operational Testing Guidance and Examples

Integrated Testing Guidance and Best Practices

Integrated Survivability Assessment Guidance and Best

Practices Force Protection Evaluation Guidance

Cybersecurity OT&E Guidance

Survey Design and Administration Guidance

3.5.1 Operational Test Events and Objectives

Realistic Operational Conditions Guidance and Examples

3.5.2 Operational Evaluation Framework

Operational Evaluation Framework Guidance with Examples

Test Instrumentation Guidance and Examples

Software Evaluation Guidance with Examples

Quantitative Mission Focused Measures Guidance with Examples

Scientific Test and Analysis Techniques Guidance with Examples

Production Representative Test Articles Guidance and Examples

Test Resources Guidance and Examples

3.5.3 Modeling and Simulation

M&S for OT&E Guidance

3.5.4 Test Limitations

Test Limitations Guidance and OT Examples

3.6 LIVE FIRE EVALUATION APPROACH

LFT&E Strategy Guidance

Integrated Survivability Assessment Guidance and Best Practices

Force Protection Evaluation Guidance

3.6.1 Live Fire Test Objectives

3.6.2 Modeling and Simulation

M&S for LFT&E Guidance

3.6.3 Test Limitations

Test Limitations Guidance and LFT&E Examples

3.7 OTHER CERTIFICATIONS

3.8 FUTURE TEST AND EVALUATION

OT of Software Intensive Systems Guidance and Examples

PART IV – RESOURCE SUMMARY

4.1 INTRODUCTION

Test Resources Guidance and Examples

4.2 TEST RESOURCE SUMMARY

4.2.1 Test Articles

Production Representative Test Articles Guidance and Examples

4.2.2 Test Sites

4.2.3 Test Instrumentation

Test Instrumentation Guidance and Examples

4.2.4 Test Support Equipment

4.2.5 Threat Representation

Threat Representation Guidance and Examples

4.2.6 Test Targets and Expendables

4.2.7 Operational Force Test Support

4.2.8 Models, Simulations, and Test Beds

4.2.9 Joint Operational Test Environment

4.2.10 Special Requirements

4.3 FEDERAL, STATE, AND LOCAL REQUIREMENTS

4.4 MANPOWER / PERSONNEL TRAINING

4.5 TEST FUNDING SUMMARY

Test Funding Guidance and Examples

APPENDIX A – BIBLIOGRAPHY

APPENDIX B – ACRONYMS

APPENDIX C – POINTS OF CONTACT

APPENDIX D – SCIENTIFIC TEST AND ANALYSIS TECHNIQUES

APPENDIX E – CYBERSECURITY

APPENDIX F – RELIABILITY GROWTH PLAN

APPENDIX G – REQUIREMENTS RATIONALE

ADDITIONAL APPENDIXES AS NEEDED

1. PART I - INTRODUCTION

1.1. PURPOSE

State the purpose of the Test and Evaluation Master Plan (TEMP).

Identify if this is an initial or updated TEMP.

State the Milestone (or other) decision the TEMP supports.

State if the program is listed on the DOT&E Oversight List or is an MDAP, MAIS, or USD(AT&L)-designated special interest program.

1.2. MISSION DESCRIPTION

1.2.1 Mission Overview

Summarize the mission need described in the program capability requirements documents in terms of the capability the system will provide to the Warfighter.

Describe the mission to be accomplished by a unit that will be equipped with the system.

Incorporate an Operational View (OV-1) of the system showing the intended operational environment.

Include significant points from the Life Cycle Sustainment Plan, the Information Support Plan, and the Program Protection Plan.

For business systems, include a summary of the business case analysis for the program.

1.2.2 Concept of Operations

Reference all applicable Concepts of Operations and Concepts of Employment in describing the mission. Describe test implications.

o CONOPS Guidance and Examples

1.2.3 Operational Users

Describe the intended users of the system, how they will employ the system, and any important characteristics of the operational users (e.g., experience level, training requirements, area of specialization, etc.).

o Cybersecurity OT&E Guidance and Example

1.3 SYSTEM DESCRIPTION

Describe the system configuration.

Identify key features and subsystems, both hardware and software (such as architecture, system and user interfaces, security levels, and reserves) for the planned increments within the Future Years Defense Program

(FYDP).

1.3.1. Program Background

Reference the Analysis of Alternatives (AoA), the Acquisition Program Baseline (APB), the Materiel Development Decision (MDD), and the last Milestone decision (including Acquisition Decision Memorandum (ADM)) to provide background information on the proposed system.

Briefly describe the overarching Acquisition Strategy. Address whether the system will be procured using an incremental development strategy or a single step to full capability.

If it is an evolutionary acquisition strategy, discuss planned upgrades, additional features and expanded capabilities of follow-on increments. The main focus must be on the current increment with brief descriptions of the previous and follow-on increments to establish continuity between known increments.

Describe the nomenclature used for increments, waves, releases, etc.

1.3.2. Key Interfaces

Identify interfaces with existing or planned systems’ architectures that are required for mission accomplishment.

Address integration and modifications needed for commercial items. Include interoperability with existing and/or planned systems of other Department of Defense (DoD) Components, other Government agencies, or Allies.

Provide a DoD Architectural Framework (DoDAF) that shows the different system interfaces, e.g., SV2, SV6, etc., from the Capability Development Document (CDD) or Capability Production Document (CPD).

1.3.3. Key Capabilities

Identify the Key Performance Parameters (KPPs), Key System Attributes (KSAs), Critical Technical Parameters (CTPs), and additional important information for the system. For each listed parameter, provide the threshold and objective values from the CDD / CPD/ Technical Document and reference the CDD / CPD/ Technical Document paragraph.

Identify Critical Operational Issues (COIs).

o COIs should identify key elements for operationally effectiveness, operationally suitability, and survivability; they represent a significant risk if not satisfactorily resolved.

o COIs should be few in number and reflect operational mission concerns.

Existing documents such as capability requirements documents, Business Case Analysis, AoA, APB, warfighting doctrine, validated threat assessments and CONOPS may provide useful insights in developing COIs.

1.3.4. System Threat Assessment

Describe the threat environment (to include cyber-threats) in which the system will operate. Reference the appropriate Defense Intelligence Agency (DIA) or component-validated threat documents for the system.

o Threat Representation Guidance and Examples o Cybersecurity OT&E Guidance and Example

Ensure that the narrative in Part I is consistent with the schedule in Part II, the T&E strategy in Part III, and allocated resources in Part IV. This will require iterative coordination between sub-workgroups and the T&E WIPT.

1.3.5. Systems Engineering (SE) Requirements

Describe SE-based information and activities that will be used to develop the test and evaluation plan. Examples include hardware reliability growth and software maturity growth strategies. Selected Technical Performance Measures (TPMs) from the Systems Engineering Plan (SEP) should be included to show desired performance growth at various test phases.

o Reliability Growth Guidance

Reference the SEP and ensure alignment to the TEMP.

1.3.6. Special Test or Certification Requirements

Identify unique system characteristics or support concepts that will generate special test, analysis, and evaluation requirements.

Identify and describe all required certifications, e.g., cybersecurity, Risk Management Framework (RMF), post deployment software support, resistance to chemical, biological, nuclear, and radiological effects; resistance to countermeasures; resistance to reverse engineering/exploitation efforts (Anti- Tamper); development of new threat simulation, simulators, or targets.

o Threat Representation Guidance and Examples o Cybersecurity Guidance

1.3.7. Previous Testing

Discuss the results of any previous tests that apply to, or have an effect on, the test strategy.

o LFT&E Strategy Guidance https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/850001_2014.pdf

2. PART II – TEST PROGRAM MANAGEMENT AND SCHEDULE

2.1. T&E MANAGEMENT

Discuss the test and evaluation roles and responsibilities of key personnel and organizations such as:

o Program Office o Chief Developmental Tester.

o Lead DT&E Organization o Prime Contractor o Lead OTA o User representative

2.1.1. T&E Organizational Construct

Identify the organizations or activities (such as the T&E Working-level Integrated Product Team (T&E WIPT) or Service equivalent, LFT&E IPT, etc.) in the T&E management structure, to include the sub-workgroups, such as a Modeling and Simulation; Survivability; Transportability; MANPRINT/Human System Integration; Environmental, Safety, and Occupational Health (ESOH); or Reliability.

o LFT&E Strategy Guidance Provide sufficient information to adequately understand the functional relationships.

Reference the T&E WIPT charter that includes specific responsibilities and deliverable items for detailed explanation of T&E management. These items include TEMPs and Test Resource Plans (TRPs) that are produced collaboratively by member organizations.

2.2. COMMON T&E DATABASE REQUIREMENTS

Describe the provisions for and methods of accessing, collecting, validating, and sharing data as it becomes available from contractor testing, Government Developmental Testing (DT), Operational Testing (OT), and oversight organizations, as well as supporting related activities that contribute or use test data.

Describe how the pedigree of the data will be established and maintained. The pedigree of the data refers to understanding the configuration of the test asset, and the actual test conditions under which the data were obtained for each piece of data.

Describe the data acquisition and management approach.

State which organization will be responsible for maintaining the data. For a common T&E database, a single organization is preferred.

In the case where multiple organizations require separate databases, briefly justify their requirement and describe how data will be synchronized among the databases and which database will be the data of record.

Describe how users of test data will access the data. Describe any special permissions or authorizations needed. Describe if any special tools or software are needed to read and analyze the data.

Reference a data dictionary or similar document that clearly describes the structure and format of the database.

2.3. DEFICIENCY REPORTING

(Post MS A TEMP) Describe the processes for documenting and tracking deficiencies identified during system development and operational testing.

Relate this to the Failure Reporting, Analysis, and Corrective Action System (FRACAS) in the SEP. Describe any deficiency rating system. Describe how the deficiency reporting database is different from the common T&E database, if appropriate.

Describe how the information is accessed and shared across the program, to include all applicable T&E organizations. The processes should address problems or deficiencies identified during both contractor and Government test activities. The processes should also include issues that have not been formally documented as a deficiency (e.g., watch items).

o Defense Business System Guidance and Examples

2.4. TEMP UPDATES

Reference instructions for complying with DoDI 5000.02 required updates or identify exceptions to those procedures if determined necessary for more efficient administration of document.

Provide procedures for keeping TEMP information current between updates. For a Joint or Multi-Service TEMP, identify references that will be followed or exceptions as necessary.

2.5. INTEGRATED TEST PROGRAM SCHEDULE

Display (see Figure 2.1) the overall time sequencing of the major acquisition phases and milestones. Include the test and evaluation major decision points, related activities, and planned cumulative funding expenditures by appropriation by year. Ensure sufficient time is allocated between significant test events to account for test-analyze-fix-test and correction of deficiencies, assessments, and reporting.

Include event dates such as major decision points as defined in DoD Instruction 5000.02, e.g., developmental and operational assessments, preliminary and critical design reviews, test article availability; software version releases;

appropriate phases of DT&E; LFT&E; Cybersecurity testing; Joint Interoperability Test Command (JITC) interoperability testing and certification date to support the MS-C and Full-Rate Production (FRP) Decision Review (DR).

Include significant Cybersecurity event sequencing, such as Interim Authorization to Test (IATT) and Authorization to Operate (ATO).

Include operational test and evaluation; Low-Rate Initial Production (LRIP) deliveries; Initial Operational Capability (IOC); Full Operational Capability (FOC);

and statutorily required reports such as the Live-Fire T&E Report and Beyond Low-Rate Initial Production (B-LRIP) Report.

Provide a single schedule for multi-DoD Component or Joint and Capstone TEMPs showing all related DoD Component system event dates.

Ensure that the schedule in Part II is consistent with the narrative in Part I, the T&E strategy in Part III, and allocated resources in Part IV. This will require iterative coordination between sub-workgroups and the T&E WIPT.

Figure 2.1 SAMPLE Integrated Program Test Schedule

Cyber

CVPA

Cyber

AA

3. PART III – Test and Evaluation Strategy and Implementation

3.1 T&E STRATEGY

Introduce the program T&E strategy by briefly describing how it supports the acquisition strategy as described in Section 1.3.1.

The discussions should focus on the testing for capabilities, and address testing of subsystems or components where they represent a significant risk to achieving a necessary capability.

Describe the scientific approach to designing an efficient test program that will characterize system performance across the operational conditions anticipated to be encountered by users. Summarize with details referenced in the appropriate appendix.

The strategy should address the conditions for integrating DT and OT tests.

o Integrated Testing Guidance and Best Practices

Evaluations shall include a comparison with current mission capabilities using existing data, so that measurable improvements can be determined.

o Describe the strategy for achieving this comparison and for ensuring data are retained and managed for future comparison results of evolutionary increments or future replacement capabilities.

o If such evaluation is considered costly relative to the benefits gained, the PM shall propose an alternative evaluation strategy.

To present the program’s T&E strategy, briefly describe the relative emphasis on methodologies (e.g., Modeling and Simulation (M&S), Measurement Facility (MF), Systems Integration Laboratory (SIL), Hardware-In-the-Loop Test (HILT), Installed System Test Facility (ISTF), Open Air Range (OAR), and Live, Virtual, and Constructive (LVC)).

Describe the evaluation products.

o Describe how the products will be linked.

o Identify the organization that is providing the products and to whom they are being provided.

o Identify the decision being supported by the products.

o Ensure sufficient time is allocated for analysis of the products.

3.1.1. Decision Support Key

Connect key test events to the acquisition decisions they support. Describe the information required to support such decisions.

3.2. DEVELOPMENTAL EVALUATION APPROACH

Describe the developmental evaluation approach that will be used to support technical, programmatic, and acquisition decisions.

Identify how the government intends to evaluate the design and development of technologies, components, subsystems, systems, and systems of systems as applicable in order to assess programmatic and technical risk.

Describe the integrated testing approach and how it will support the overall evaluation strategy.

3.2.1. Developmental Evaluation Framework

Embed a Developmental Evaluation Framework (DEF) in the form of a table or spreadsheet. Describe the contents of the developmental evaluation framework, including descriptions of columns and the origin of information contained. Include instructions to the reader on the use of the table or spreadsheet and its contents.

Arrange the table or spreadsheet to show time-phased, iterative test progression toward the achievement of performance goals and measures.

Include elements (columns, rows, or cells) bearing the following essential information:

o Functional evaluation area. Categorical groupings of functional areas brought forward or derived from baseline documentation.

o Decision supported. The significant program decision points where data and information gathered during testing will be used to make decisions or give program direction.

o Decision support question. Key question related to performance, reliability, cybersecurity, or interoperability that when answered determines the outcome of an evaluation for the decision supported.

o Key system requirements and T&E measures (one or more fields of requirements identification and performance measurement).

Technical requirements document reference.

Description.

Technical measures. CTP, TPM, Metrics.

o Method (technique, process, or verification method).

o Test Event.

o Resources. Brief reference may appear here.

o 3Cross-Reference. Used to refer to related requirements, capabilities, and line items to aid in requirements traceability, precedence, interdependency, and causality.

3.2.2. Test Methodology

For each capability and key functional area, address a test methodology that:

o Verifies achievement of critical technical parameters and the ability to achieve key performance parameters, and assess progress toward achievement of critical operational issues.

o Measures the system’s ability to achieve the thresholds prescribed in the capabilities documents.

o Provides data to the Program Manager to enable root cause determination and to identify corrective actions.

o Measures system functionality.

o Provides information for cost, performance, and schedule tradeoffs.

o Assesses system specification compliance.

o Identifies system capabilities, limitations, and deficiencies.

o Assesses system safety.

o Assesses compatibility with legacy systems.

o Stresses the system within the intended operationally relevant mission environment.

o Supports cybersecurity assessments and authorizations.

o Supports the interoperability certification process.

o Documents achievement of contractual technical performance and verifies incremental improvements and system corrective actions.

o Provides DT&E data to validate parameters in models and simulations.

o Assesses the maturity of the chosen integrated technologies.

3.2.3. Modeling and Simulation (M&S)

Describe the key models and simulations and their intended use. Include the developmental test objectives to be addressed using M&S to include any approved operational test objectives.

Identify who will perform M&S verification, validation, and accreditation.

Identify data needed and the planned accreditation effort.

Identify how the developmental test scenarios will be supplemented with M&S, including how M&S will be used to predict the Sustainment KPP and other sustainment considerations.

Identify and describe LVC requirements.

Identify developmental M&S resource requirements in Part IV.

3.2.4. Test Limitations and Risks

Discuss any developmental test limitations that may significantly affect the evaluator's ability to draw conclusions about the maturity, capabilities, limitations, or readiness for dedicated operational testing.

Address the impact of these limitations as well as resolution approaches.

Discuss any known test risks at the time the TEMP is being written. These are risks that may prevent or delay the satisfactory execution of the test events. Any test risks that are included in the program-level risk management database should be included.

Include a risk mitigation plan for the identified test risks.

o Test Limitations Guidance and DT Examples

3.3. DEVELOPMENTAL TEST APPROACH

Describe the approach to test the system performance in a mission context, i.e., how the system will actually be employed.

Discuss how developmental testing will reflect the expected operational environment to help ensure developmental testing is planned to integrate with operational testing.

Describe the use of actual user subjects to support human factors engineering assessments and NET development.

o Integrated Testing Guidance and Best Practices

3.3.1. Mission-Oriented Approach

3.3.2. Developmental Test Events (Description, Scope, and Scenario) and Objectives

For each developmental test event shown in the schedule and the DEF, prepare a subparagraph that summarizes: Who is the lead test organization; the objectives of the test event, the test event’s schedule; other associated test events, location(s), etc.

Summarize the planned objectives and state the methodology to test the system attributes defined by the applicable capability requirement document (CDD, CPD, CONOPS) and the CTPs that will be addressed during each phase of DT.

Subparagraphs can be used to separate the discussion of each phase.

For each DT phase, discuss the key test objectives to address both the contractor and Government developmental test concerns and their importance to achieving the exit criteria for the next major program decision point. If a contractor is not yet selected, include the developmental test issues addressed in the Request for Proposals (RFPs) or Statement of Work (SOW).

Address measurable exit/entrance criteria for each major T&E phase and milestone decision points.

Discuss how developmental testing will reflect the expected operational environment to help ensure developmental testing is planned to integrate with operational testing.

o Integrated Testing Guidance and Best Practices o Software Algorithm Testing Guidance and Examples Include key test objectives related to logistics testing.

Summarize the developmental test events, test scenarios, and the test design concept.

Quantify the testing sufficiently (e.g., number of test hours, test articles, test events, test firings) to allow a valid cost estimate to be created.

Identify and explain how models and simulations, specific threat systems, surrogates, countermeasures, component, or subsystem testing, test beds, and prototypes will be used to determine whether or not developmental test objectives are achieved.

Identify the DT&E reports required to support decision points/reviews and OT readiness.

Address the system’s reliability growth strategy, goals, and targets and how they support the Developmental Evaluation Framework. Detailed developmental test objectives should be addressed in the System Test Plans and detailed test plans (Provide specific details in Appendix F – Reliability Growth Plan).

o Reliability Growth Guidance Discuss plans for interoperability and cybersecurity testing, including the use of cyber ranges for vulnerability and adversarial testing (Provide specific details in Appendix E

– Cybersecurity).

3.4. CERTIFICATION FOR INITIAL OPERATIONAL TEST AND EVALUATION (IOT&E)

Explain how and when the system will be certified safe and ready for IOT&E.

Explain who is responsible for certification and which decision reviews will be supported using the lead Service’s certification of safety and system materiel readiness process.

List the DT&E information (i.e., reports, briefings, or summaries) that provides predictive analyses of expected system performance against specific COIs and the key system attributes – measures of effectiveness (MOE) and measures of suitability

(MOS).

Discuss the entry criteria for IOT&E and how the DT&E program will address those criteria.

o IOT&E Entrance Criteria Guidance and Examples

3.5. OPERATIONAL EVALUATION APPROACH

• Summarize the mission focused evaluation methodology and supporting test strategy, including the essential mission and system capabilities that contribute to operational effectiveness, suitability, and survivability.

o Mission Focused Evaluation Guidance and Examples o Baseline Evaluation Guidance with Best Practices o End-to-End Operational Testing Guidance and Examples o Cybersecurity OT&E Guidance and Example o Survey Design and Administration Guidance

• Summarize the operational test events, key threat simulators and/or simulation(s) and targets to be employed, and the type of representative personnel who will operate and maintain the system.

• Summarize integrated testing strategy to include:

o Developmental test data that will be used for operational evaluation o Conditions on data pedigree and test conduct to make data suitable for operational evaluation o Integrated Testing Guidance and Best Practices o Integrated Survivability Assessment Guidance and Best Practices o Force Protection Evaluation Guidance

3.5.1 Operational Test Events and Objectives

Identify the key operational test objectives for each test event and test phase Outline the approach for characterizing the COIs and important MOEs/MOSs across relevant operational conditions.

o Realistic Operational Conditions Guidance and Examples o OT of Software Intensive Systems Guidance and Examples

3.5.2 Operational Evaluation Framework The evaluation framework should identify and link:

o The goal of the operational test within a mission context o The mission-oriented response variables, the factors that affect those variables, and he required test resources o (Post MS A TEMP) The test designs for strategically varying the factors across the operational envelope o Operational Evaluation Framework Guidance with Examples o Test Instrumentation Guidance and Examples o Software Evaluation Guidance with Examples The evaluation framework should focus on the subset of mission-oriented measures critical for assessing operational effectiveness, suitability, and survivability.

o Mission Focused Metrics Guidance with Examples

(Post MS A TEMP) Use a systematic, rigorous, and structured approach to link major test events and phases to quantitatively evaluate system capabilities across relevant operational conditions.

(Post MS A TEMP) Describe the statistical test design strategy and corresponding statistical measures of merit (e.g., confidence and power).

o Scientific Test and Analysis Techniques Guidance with Examples Identify planned sources of information (e.g., developmental testing, testing of related systems, modeling, simulation) that may be used to supplement operational test and evaluation.

Describe the scope of the operational test by identifying the test mission scenarios and the resources that will be used to conduct the test.

o Production Representative Test Articles Guidance and Examples o Test Resources Guidance and Examples

3.5.3 Modeling and Simulation (M&S)

• If described in either the DT&E or Live Fire sections, do not repeat. Just reference and hyperlink. Only discuss what is unique to OT&E.

• Describe the key models and simulations and their intended use.

• Include the operational test objectives to be addressed using M&S.

• (Post MS A TEMP) Identify who will perform the M&S verification, validation, and accreditation.

• (Post MS A TEMP) Identify data needed and the planned accreditation effort.

• Identify how the operational test scenarios will be supplemented with M&S.

• Identify operational M&S resource requirements in Part IV.

o M&S for OT&E Guidance

3.5.4 Test Limitations

Discuss test limitations including threat realism, resource availability, limited operational (military; climatic; Chemical, Biological, Nuclear, and Radiological (CBNR), etc.) environments, limited support environment, maturity of tested systems or subsystems, safety, that may impact the resolution of affected COIs.

Describe measures taken to mitigate limitations.

Indicate if any system contractor involvement or support is required, the nature of that support, and steps taken to ensure the impartiality of the contractor providing the support according to Title 10 U.S.C. §2399.

Indicate the impact of test limitations on the ability to resolve COIs and the ability to formulate conclusions regarding operational effectiveness and operational suitability.

Indicate the COIs affected in parentheses after each limitation.

o Test Limitations Guidance and OT Examples

3.6. LIVE FIRE TEST AND EVALUATION APPROACH

If live fire testing is required, describe the approach to evaluate the survivability/lethality of the system, and (for survivability LFT&E) personnel survivability of the system's occupants.

o LFT&E Strategy Guidance o Integrated Survivability Assessment Guidance and Best Practices o Force Protection Evaluation Guidance Include a description of the overall live fire evaluation strategy to influence the system design (as defined in Title 10 U.S.C. § 2366), critical live fire evaluation issues, and major evaluation limitations.

Discuss the management of the LFT&E program, to include the shot selection process, target resource availability, and schedule.

Discuss a waiver, if appropriate, from full-up, system-level survivability testing, and the alternative strategy.

3.6.1. Live Fire Test Objectives

State the key live fire test objectives for realistic survivability or lethality testing of the system.

Include a matrix that identifies all tests within the LFT&E strategy, their schedules, the issues they will address, and which planning documents will be submitted for DOT&E approval and which will be submitted for information and review only.

Identify whether full-up, system-level testing will be conducted, or whether a waiver will be required from such testing. If a waiver will be required from full-up, system-level testing, describe the key features of the alternative LFT&E plan, including the planned levels of test realism to support the evaluation of survivability or lethality.

Quantify the testing sufficiently (e.g., number of test hours, test articles, test events, test firings) to allow a valid cost estimate to be created.

3.6.2. Modeling and Simulation (M&S)

• Only discuss what is unique to live fire.

• Describe the key models and simulations and their intended use.

• If M&S is to be used for test planning, describe how M&S will be used as a basis for decisions regarding test scope or test conditions.

• If M&S is to be used for prediction of test results, identify which tests will have predictions based on M&S, and which models will be used for such predictions.

• If M&S is to be used for evaluation of critical LFT&E issues, summarize the degree of reliance on M&S, and identify any evaluation issues that will be addressed solely by M&S.

• Include the LFT&E test objectives to be addressed using M&S to include operational test objectives.

• (Post MS A TEMP) Identify who will perform M&S verification, validation, and accreditation

• (Post MS A TEMP) Identify data needed and the planned accreditation effort.

• Identify how the test scenarios will be supplemented with M&S.

• Identify and describe LVC requirements.

• Identify M&S resource requirements in Part IV.

o M&S for LFT&E Guidance

Ensure that the T&E strategy in Part III is consistent with the narrative in Part I, the schedule in Part II, and allocated resources in Part IV. This will require iterative coordination between sub-workgroups and the T&E WIPT.

3.6.3. Test Limitations

Discuss any test limitations that may significantly affect the ability to assess the system’s vulnerability and survivability.

Also address the impact of these limitations, and resolution approaches.

o Test Limitations Guidance and LFT&E Examples

3.7. OTHER CERTIFICATIONS

Identify key testing prerequisites and entrance criteria, such as required certifications (e.g. DoD Risk Management Framework (RMF), Authorization to Operate, Weapon Systems Explosive Safety Review Board (WSERB), flight certification, etc.)

3.8. FUTURE TEST AND EVALUATION

Summarize all remaining significant T&E that has not been discussed yet, extending through the system life cycle.

o Significant T&E is that T&E requiring procurement of test assets or other unique test resources that need to be captured in the Resource section.

o Significant T&E can also be any additional questions or issues that need to be resolved for future decisions.

Do not include any T&E in this section that has been previously discussed in this part of the TEMP.

4. PART IV-RESOURCE SUMMARY

4.1. INTRODUCTION

In this section, specify the resource elements, both government and contractor, necessary to plan, execute, and evaluate a test event or test campaign.

o Test Resources Guidance and Examples

Resource elements include test articles, models, simulations, test facilities, manpower for test conduct and support, and other items that are described below.

Resource estimates must be quantifiable and defensible, derived from STAT methodologies (identified in the evaluation framework and included in the STAT section or appendix) and where appropriate, based on test experience.

Testing will be planned and conducted to take full advantage of existing DoD investment in ranges, facilities, and other resources wherever practical. Justify use of non-government facilities.

Along with each resource element, include an estimate of element quantity, when the elements will be used (consistent with figure 2.1 schedule), the organization responsible for providing them, and their cost estimate (if available).

Include long-lead items for the next increment if known.

Callout any shortfalls, their impact on planned T&E, and describe an appropriate mitigation.

4.2. TEST RESOURCE SUMMARY

4.2.1. Test Articles

Identify the actual number of and timing requirements for all test articles, including key support equipment and technical information required for testing in each phase of DT&E, LFT&E, and OT&E.

o Production Representative Test Articles Guidance and Examples

If key subsystems (components, assemblies, subassemblies or software modules) are to be tested individually, before being tested in the final system configuration, identify each subsystem in the TEMP and the quantity required. Specifically identify when prototype, engineering development, or production models will be used.

Use of tables to more accurately convey information for each of the sub-paragraphs below is encouraged. See TEMP Guide for real world TEMP examples.

4.2.2. Test Sites

Identify the specific test ranges/facilities and schedule to be used for each type of testing.

Compare the requirements for test ranges/facilities dictated by the scope and content of planned testing with existing and programmed test range/facility capability.

Summarize the results of a cost benefit analysis (CBA) in those cases where government test facilities are not used.

Test Facilities may include the following and other test venues:

o Digital Modeling and Simulation Facility (DMSF).

o Measurement Facility (MF).

o System Integration Laboratory (SIL).

o Hardware-in-the-Loop (HWIL) Facility.

o Installed System Test Facility (ISTF).

o Open Air Ranges (OAR).

o Cyber Ranges.

o Distributed Live, Virtual, and Constructive (DLVC) Environments.

4.2.3. Test Instrumentation

Identify instrumentation that must be acquired or built specifically to conduct the planned test program o Test Instrumentation Guidance and Examples

Identify the specific data classes that the instrumentation will capture and relate it to the

DEFM.

Identify any special tools or software…