Attachment_3,_ATEC_Pamphlet_73-1_System_Test_andEvaluation_Procedures_.pdf

UNCLASSIFIED

ATEC Pamphlet 73-1

Test and Evaluation

Volume I

System Test and Evaluation Procedures

Headquarters US Army Test and Evaluation Command Aberdeen Proving Ground, MD 9 July 2013

This page intentionally left blank

SUMMARY of CHANGE

Army Test and Evaluation Command (ATEC) Pamphlet 73-1, Volume I System Test and Evaluation Procedures

This revision dated, 9 July 2013—

• Incorporates changes reflecting Phase I of the ATEC re-organization effort and the disestablishment of the US Army Developmental Test Command.

• Incorporates changes resulting from the Base Realignment and Closure 2005.

• Incorporates changes in Army and Department of Defense (DOD) organization.

• Updates policy and figures to reflect the name changes of many T&E planning and reporting documents.

• Removes policy regarding Combined Test Organization.

• Removes policy and procedures regarding Rock Drills.

• Incorporates the policies and responsibilities from Developmental Test Command Pamphlet 73-1.

• Incorporates the policies, responsibilities, and procedures for ATEC, AEC, and OTC in collaboration with the Assistant Secretary of the Army for Acquisition, Logistics, & Technology ASA(ALT) for execution of the Agile Process for acquisition during biannual Network Integration Evaluations (NIE).

• Updates the policy and guidance on the Threat Computer Network Operations Team (TCNOT) selection process and IA testing to support Protection, Detection, Reaction, and Restoration (PDRR) metrics required by the evaluator.

• Incorporates the revised policies, procedures, and responsibilities pertaining to Release of Data and ATEC documentation distribution guidance.

Incorporates new guidance on the Evaluation Determination Matrix for ACAT III Non-Oversight programs.

• Supersedes the following publications:

o ATEC Pamphlet 73-1, 16 June 10.

o DTC Pamphlet 73-1, 30 Oct 06.

o ATEC Interim Policy Guidance (IPG) 10-2, Early Strategy Review and the Test and

Evaluation Concept In Process Review, 15 Mar 10.

o ATEC IPG 10-3, Capabilities and Limitations Report Staffing Process, 17 Mar 10.

o ATEC IPG 10-5, Mission Based T&E, 17 Mar 10.

o ATEC IPG 10-6, Standardization for Scheduling and Completing Early Strategy

Reviews, Test Concept In-Process Reviews and Test and Evaluation Concept Briefings, 27 Aug 10.

o ATEC IPG 10-8, T&E Concept Brief Template, 13 Oct 10.

o ATEC IPG 10-10, Development, Staffing, and Approval of Operational Test Agency

Test Plans (OTA TP) for Operational Tests, 17 Dec 10.

o ATEC IPG 10-11, Including DT Results in OTRRs, 30 Dec 10.

o ATEC IPG 11-4, Transmittal of ATEC T&E Documents to Army Test and Evaluation

Office and Director, Operational Test and Evaluation, 4 Feb 11.

o ATEC Policy Bulletin (PB) 2-12, Safety Verification Document Signature Authority, 17

Jan 2012.

o ATEC PB 3-12, Implementation of Design of Experimentation Techniques for Test and

Evaluations, 22 Feb 12.

o ATEC PB 12-12, Coordination with Program Managers (PM)/Program Executive

Offices (PEO) for Evaluation Plans and Reports, 15 Jun 2012.

o ATEC PB 13-12, Software Safety Verification Policy and Guidelines 15 Jun 12.

o ATEC PB 23-12, Test Location Visits, 23 Oct 12.

o ATEC Policy Bulletin 28-12, ATEC System Team (AST) Roles and Responsibilities, 28

Nov 12.

i ATEC Pamphlet 73-1 • 9 July 2013

Department of the Army ATEC Pamphlet 73-1 United States Army Test and Evaluation Command 2202 Aberdeen Boulevard-Third Floor Aberdeen Proving Ground, MD 21005-5001

Test and Evaluation

VOLUME I - SYSTEM TEST AND EVALUATION PROCEDURES

History. This pamphlet was last revised on 16 June 2010.

Summary. The US Army Test and Evaluation Command (ATEC) supports the Army system acquisition and force development processes through overall management of the Army’s T&E programs. This pamphlet implements ATEC methodology for testing and system evaluation in accordance with ATEC Regulation 73-1; provides background information on integrated T&E strategies; and provides guidance and suggestions for preparing and formatting documentation for tests, evaluations, and assessments. This pamphlet supersedes ATEC Pamphlet 73-1, dated 16 June 2010.

Applicability. This pamphlet applies to the US Army Operational Test Command (OTC), US Army Evaluation Center (AEC), ATEC Test Centers, and Headquarters (HQ) ATEC.

Supplementation. Supplementation of this pamphlet is permitted to implement further guidance within respective areas of responsibility. All supplements will be coordinated with the ATEC HQ G-9 Director, Test Management prior to approval by the Technical Director at each command. A copy will be provided to ATEC HQ G-9 (CSTE-TM).

Suggested improvements. The proponent for this pamphlet is the ATEC G-9, Policy and Methodology Division (CSTE-TM). Users are invited to send comments and suggested improvements on DA Form 2028 (Recommended Changes to Publications and Blank Forms) through the chain of command to Commander, ATEC (CSTE-TM-P), 2202 Aberdeen Boulevard, Aberdeen Proving Ground, MD 21005.

Distribution: OTC, AEC, ATEC Test Centers, and HQ ATEC.

FOR THE COMMANDER:

OFFICIAL: KAREN L. TAYLOR

Chief of Staff

JUDITH M. MATTHEWS

Secretary of the General Staff ii ATEC Pamphlet 73-1 • 9 July 2013

This page is intentionally left blank.

5 ATEC Pamphlet 73-1 • 9 July 2013

TABLE OF CONTENTS

Chapter 1 Introduction

1-1 Purpose 1-2. Scope 1-3. References 1-4. Explanation of Abbreviations and Terms 1-5. Foundation for T&E 1-6. ATEC Mission and Vision 1-7. ATEC Organization 1-8. ATEC Interface with Acquisition Community

Chapter 2 Test and Evaluation Management

2-1. Overview 2-2. Overview of the Joint Capabilities Integration and Development System (JCIDS) (Chairman Joint Chief of Staff Instruction (CJCSI) 3170) 2-3. Overview of the Defense Acquisition Framework (DODI 5000.02) 2-4. Considerations in T&E Planning to Support the Acquisition Process 2-5. Overview of the ATEC Integrated T&E Process 2-6. Rapid Acquisition Initiative (RAI) 2-7. ATEC’s Support to Capabilities Development for Rapid Transition (CDRT) 2-8. Agile Capabilities Life Cycle Process 2-9. Technology Demonstrations and Experiments 2-10. Joint T&E (JT&E) Program 2-11. Chemical and Biological Defense Program (CBDP) 2-12. Clothing and Individual Equipment (CIE) and Organizational CIE (OCIE) Programs. .41 2-13. Soldier Enhancement Program 2-14. Rapid Fielding Initiatives (RFI) 2-15. Business Capability Lifecycle (BCL) for Defense Business Systems (DBS) 2-16. Overview of the NR KPP(CJCSI 6212) 2-17. Overview of the Information Assurance Process (DODD 8500.01E) 2-18. Establishing a System in the ATEC Decision Support System (ADSS) 2-19. ATEC System Team (AST) 2-20 Working Groups and Decision Bodies 2-20. Test Schedule and Review Committee (TSARC)

Chapter 3 System Evaluation Planning and Analysis

3-1. Overview

Section I. Steps in Developing the Evaluation Strategy

3-2. Overview 3-3. Gather and Review Relevant Documents

6 ATEC Pamphlet 73-1 • 9 July 2013

3-4. Evaluation Determination Matrix (EDM) for ACAT III, Non-Oversight Systems 3-5. Develop Measures 3-6. Develop a Functional Dendritic Chart 3-7. Develop a Baseline Correlation Matrix (BCM) 3-8. Develop the Data Source Matrix (DSM) 3-9. Develop the Overarching DOE Campaign and Test Matrices using DOE Principles 3-10. Develop a System Evaluation Data Model 3-11. Assess the Need and Provide for M&S and Common Instrumentation

Section II Evaluation Planning Reviews and Documentation

3-12. Early Strategy Reviews (ESRs) 3-13. Concept In-Process Review (CIPR) 3-14. System Evaluation Plan (SEP) 3-15. Providing Input to the TEMP

Section III. Other Evaluation Considerations

3-16. Evaluation Support Outside ATEC 3-17. Software Analysis 3-18. Integrated Suitability 3-19. Reliability, Availability and Maintainability (RAM) 3-20. Integrated Logistics Support (ILS) (Also Referred to as Product Support) 3-21. Manpower and Personnel Integration (MANPRINT) 3-22. Survivability

Chapter 4 Test Planning and Execution

Section I. The Test Planning Process

4-1. Overview of Test Planning 4-2. Developmental and Operational Testing 4-3. Combined and integrated DT/OT 4-4. Contractor DT 4-5. Use of Government Test Facilities

Section II Test Planning

4-6. Considerations in DT Event Planning 4-7. OTA TP for DT

4-8. DT DTP

4-9. Witnessing Tests 4-10. Considerations in OT Event Planning 4-11. Develop a Pattern of Analysis (PoA) 4-12. OTA TP for OT 4-13. Test Support Packages (TSPs)

7 ATEC Pamphlet 73-1 • 9 July 2013

Section III Test Execution

4-14. Overview of Test Readiness Reviews (TRR) 4-15. Evaluation Decision Review (EDR) 4-16. Developmental Test Readiness Review (DTRR) 4-17. Operational Test Readiness Review (OTRR) 4-18. Joint Test Readiness Review (JTRR) 4-19. Overview of Test Execution 4-20. DT Execution 4-21 OT Execution 4-22. Customer Test

Section IV. Data Management

4-23. Data management Overview 4-24. DT Data Management 4-25. OT Data Management 4-26. AEC Data Management Team 4-27. Data Authentication 4-28. Data Release

Chapter 5 Test and Evaluation Reporting

5-1. Overview Section I. Test Reporting 5-2. Test Reporting Policy 5-3. DT Reports 5-4. Live Fire Test (LFT) Report 5-5. Operational Test Reports (OTR) 5-6. Test Incident Report (TIR)

Section II. Assessments/Evaluations

5-7. Data Analysis 5-8. Evaluation Reporting 5-9. OTA Milestone Assessment Reports (OMARs)

5-10. OER/OFER

5-11. OAR

5-12. Materiel Release Documentation 5-13. System Analysis Report (SAR) Section III. Other ATEC Reports/Documents 5-14. Safety Assessment Documentation 5-15. Technical Notes 5-16. ATEC Document Distribution Guidelines

8 ATEC Pamphlet 73-1 • 9 July 2013

Appendix A References Appendix B Support for RAI/Urgent Materiel Release (UMR) Appendix C Capability Documents Review Checklists Appendix D Project Management Module (PMM) Appendix E Test Cost Transparency Appendix F Test Schedule and Review Committee (TSARC) Appendix G Formats Appendix H Mission Based Test and Evaluation (MBT&E) Appendix I Design of Experiments (DOE) Appendix J Reliability, Availability, and Maintainability (RAM) Appendix K Manpower and Personnel Integration (MANPRINT) Appendix L Survivability Appendix M Guidelines for Conducting Operational T&E for Software-Intensive System Increments Appendix N Test Types Appendix O ATEC Test Facilities Appendix P System Safety and Software Safety Appendix Q Data Authentication Group (DAG) Appendix R Other T&E Considerations Appendix S Commercial Use of Test Facilities

Glossary

Table List

Table 2-1. BCL Timelines Table 2-2. AST Responsibilities Table 3-1. Evaluation Process Inputs/Key Activities and Outputs by Acquisition Phase Table 3-2. System Documentation Table 3-3. EDM Scoring Rules by Factor Table 3-4. Supportability Analysis Scoring Rules Table 3-5. Levels of Evaluation for Non-Major Programs Table 3-6. Example Baseline Correlation Matrix Table 4-1. Agenda for Test Concept Brief Table 4-2. Test Support Package Requirements for OT Table 4-3. Data Level Definitions Table 5-1. ATEC DT Test Report Approval Process Table 5-2. Test/Support Reporting Time Frames (Calendar Days) Table 5-3. AEC Statistical Software Package Analysis Comparison Table 5-4. 3 Tier Effectiveness and Suitability Table 5-5. 3 Tier Survivability Table 5-6. ATEC Evaluation Document Distribution Guidelines

Table B-1. CLR Timeline

9 ATEC Pamphlet 73-1 • 9 July 2013

Table F-1. TSARC Schedule Table I-1: Example Overarching DOE Campaign Table I-2: Example Test Event X Factors, Controls, and Conditions Table I-3. Example Test Design Matrix Table I-4. Example DOE Analytic Model Parameters and Power Table I-5. Example Test Trial Run Matrix Table I-6. DOE Product Leads and Placement in ATEC T&E Briefings and Documents Table O-1. OTC’s Core Competencies - Test Directorate Table R-1. Climatic Design Types Table S-1. Summary of MRTBF and Non-MRTBF Costs

Figure List

Figure 1-1. ATEC Organizations and Locations Figure 1-2. Organizations Involved in T&E Figure 2-1. DOD Concurrent Acquisition Processes Figure 2-2. Selection of OCO Systems for CDRT Figure 2-3. Rapid Materiel Acquisition Scenarios Figure 2-4. Rapid Materiel Acquisition Scenario Detail Figure 2-5. The Army Agile Capability Lifecycle Process Figure 2-6. BCL Model Figure 2-7. Distinguishing between IA and TCNO Figure 2-8. Sample AST Charter Figure 3-1. Example System Evaluation Dendritic chart Figure 3-2. Example DSM Figure 3-3. SEP Staffing and Approval Process Figure 4-1. Test Planning Process Figure 5-1. T&E Reporting Process Figure 5-2. LFT&E Documents Timeline

Figure B-1. RAI and UMR Process Figure D-1. Organizational Breakdown Structure for ATEC Figure D-2. ATEC WBS Figure G-1. EDM Example Figure G-2. Form 2010-1 for Confirmation of Low-Risk Evaluations Figure G-3. Notification of Low Risk Evaluation Figure H-1. MBT&E Framework Figure H-2. MBT&E Procedure Figure I-1. 10 DOE Products, Figure J-1. Sample Reliability Growth Planning Curve with Operation and Support Costs Figure J- 2. Sample time-event plot Figure K-1. MANPRINT Impact Rating Scale Figure L-1. CBRN Survivability Figure L-2. DA-approved CBRCS Criteria for Army Materiel Figure L-3. CBRCS Evaluation Data Requirements

10 ATEC Pamphlet 73-1 • 9 July 2013

Figure N-1. Examples of Test Types During Acquisition Phase Figure O-1- Flowchart for Processing Exceptions ot the ATEC 10 series Test Center Mission and Major Capabilities Figure P-1. Software Safety Verification Process Figure S-1. ATEC Form 2000-R, Private Industry Test Information Sheet Figure S-2. ATEC Form 2001-R, Formal Contract for Commercial Testing Figure S-3. Determination and Findings Figure S-4. ATEC Form 2002-R, Document Checklist for Commercial Testing

11 ATEC Pamphlet 73-1 • 9 July 2013

This page intentionally left blank.

12 ATEC Pamphlet 73-1 • 9 July 2013

Chapter 1 Introduction

1-1. Purpose.

This pamphlet implements the policies and responsibilities set forth in ATEC Regulation 73-1.

It establishes guidelines and procedures for test and evaluation (T&E) of Army, Multi-Service and Joint, Interagency, and Multi-National T&E programs. This ATEC pamphlet describes the integrated T&E support that ATEC provides to the Defense Acquisition Process through developmental testing (DT), operational testing (OT), integrated T&E, field experimentation, rapid acquisition initiatives (RAI), and independent system evaluation.

1-2. Scope.

The procedures described in this pamphlet are based on Chairman of the Joint Chiefs of Staff Instructions (CJCSI) 3170 and 6212; Department of Defense (DOD) Directives/Instructions (DODD/DODI) 3200, 4630, 5000 series, and 8500; and Headquarters, Department of the Army (DA) policy documentation including Army Regulations (ARs) 70-1, AR 73-1, and DA Pamphlet 73-1. This pamphlet establishes the common procedures and processes that are used to implement the policies contained in ATEC Regulation 73-1. Additional procedures and information are contained in the appendixes of this pamphlet.

1-3. References.

Required and related publications are listed in Appendix A.

1-4. Explanation of Abbreviations and Terms.

Abbreviations and terms used in this regulation are spelled out in the glossary. Most abbreviations are spelled out on their first occurrence in each chapter. Many common place abbreviations such as United States (US), ATEC, US Army Operational Test Command (OTC) and US Army Evaluation Center (AEC) are only spelled out on the first occurrence in the document.

1-5. Foundation for T&E.

The requirement for T&E as an integral part of the acquisition of materiel systems is mandated by law and delineated in multiple DOD/AR directives and regulations.

a. Title 10 US Code establishes statutory requirements for T&E. The key sections that address T&E are outlined below.

Section:

139 Establishes Director, Operational Test and Evaluation (DOT&E).

2366 Requires Survivability and Lethality Testing before full rate production (FRP).

13 ATEC Pamphlet 73-1 • 9 July 2013

2399 Directs Operational T&E of Major Defense Acquisition Programs (MDAP) and restricts contractor involvement in Initial Operational T&E (IOT&E).

2400 Describes Determination of Quantities to be procured for Low-Rate Initial Production (LRIP) of New Systems.

2681 Establishes DOD Major Range and Test Facility Base (MRTFB).

b. DODD 5000.1 and DODI 5000.02 establish the policy and procedures for the acquisition process of technology projects and acquisition programs for all Services. DODD 3200.11 establishes policy and responsibilities for the MRTFB. DODD 4630 addresses policy for the Net Ready Key Performance Parameter (NR KPP) of Information Technology (IT) and National Security Systems (NSS). DODD 8500 establishes the DOD Information Assurance Certification and Accreditation Program.

c. AR 73-1 implements policies and assigns responsibilities for T&E activities during the system acquisition process.

1-6. ATEC Mission and Vision.

a. Mission: ATEC is a direct reporting unit under the Office of the Chief of Staff of the Army per AR 10-87 Army Commands, Army Service Component Commands, and Direct Reporting Units, dated 4 September 2007 and by DA General Order 2006-13. ATEC’s mission is to plan, conduct, and integrate DT, independent OT, and independent evaluations, assessments, and experiments to provide essential information to fielding, acquisition and equipping decision makers and commanders.

b. Vision: To determine the true capability provided to our Soldiers through developmental, integrated, and operational testing evaluated in an independent and objective manner. We execute our mission in a collaborative environment. We invest in both our people and infrastructure. We seek the voice of our customer as we continuously improve in all aspects of our command to become more interdependent, affordable, effective, and efficient.

1-7. ATEC Organization.

a. ATEC is comprised of a headquarters (HQ) located at Aberdeen Proving Ground (APG), Maryland, AEC, OTC, and eight developmental Test Centers. ATEC’s organizations and their locations are depicted in figure 1-1.

14 ATEC Pamphlet 73-1 • 9 July 2013

EPG/ Ft Huachuca

Ft Hood

Ft Sill YTC/Yuma

Proving Ground

WDTC/Dugway Proving Ground

Ft Bragg

ATC/Aberdeen Proving Ground

OTC

Legend

Headquarters Major Test Facility Operational Test Dir

AEC

Ft Bliss

ATEC

CRTC/Ft Greely

WSTC/White Sands Missile Range

RTC/Redstone Arsenal

Hawaii

Panama Honduras

Suriname

TRTC

Qatar

FOA Team

Figure 1-1. ATEC Organizations and Locations.

b. ATEC HQ.

(1) Responsibilities of all HQ elements are listed in Chapter 2 of ATEC Regulation 73-1.

(2) Support of Army system acquisition, force development and experimentation processes through overall management of the Army’s T&E programs, activities and institutional resources to include:

(a) Liaison Offices (LNO). As part of ATEC’s early involvement strategy, ATEC HQ G-9 maintains a number of LNOs located throughout the continental US with key US Army Training and Doctrine Command (TRADOC) and Army Program Executive Officers (PEOs).

ATEC LNOs are capable of providing T&E advice and insights early in the acquisition life cycle supporting improved design efficiencies and cost savings. For a current list of ATEC LNOs and their contact information, please contact the ATEC G-9. LNO roles and responsibilities are as follows:

(1) Monitor technical demonstrations (pre-Milestone (MS) A) for possible ATEC involvement or assistance.

(2) Provide ATEC information on emerging systems and requirements.

(3) Expedite entry of new program starts into ATEC Decision Support System

15 ATEC Pamphlet 73-1 • 9 July 2013

(ADSS)/Test Schedule and Review Committee (TSARC) process.

(4) Facilitate the flow of information between ATEC and PEO elements and report on matters that impact T&E execution.

(5) Review, comment, and assist with the preparation of T&E documentation.

(Provide Test Resource Plans (TRPs) for logistics demonstrations.)

(6) Provide a focal point for program conflict/issue resolution.

(7) Provide T&E expertise, guidance and information.

(8) Assist the Program Manager (PM) in completing the ATEC Request for Support form and quad chart to get new acquisition systems or RAI programs into ATEC. These two forms will be submitted to the ATEC G9 test operations mailbox at usarmy.APG.atec.mbx.g9-test-ops-division@mail.mil.

(9) Assist PEO/PMs with T&E Program Objective Memorandum development and justification.

(10) Assist with MS reviews and definition of T&E acquisition regulatory requirements.

(11) Provide support to ATEC elements visiting the PEO.

(12) Serve as the focal point for PEO events to include: distributing surveys, coordination and scheduling with PMs and PEOs, and development of the overall briefing package.

(b) Joint Test and Evaluation (JT&E) Office. The ATEC JT&E Office generates operational solutions to urgent, specific, joint problems through a dynamic rigorous test process, principally accomplished through Quick Reaction Tests. They manage the program for providing Army resources to support joint test requirements. The ATEC JT&E office also assists joint tests through the TSARC process by requesting Army money, personnel, and equipment to support a specific test event. Personnel act as liaisons or are attached to joint tests.

See paragraph 2-10 for more information on JT&E.

c. AEC. AEC is located at APG, Maryland. AEC operates ten evaluation directorates. AEC provides integrated T&E management and executes independent materiel evaluations and assessments to determine effectiveness, suitability, survivability and safety verification for all assigned acquisition programs. In addition to developing system evaluation plans and reports for programs of record, AEC also provides safety assessment documents (releases, confirmations, and recommendations) to support program test events/demonstrations, milestone decisions, and fielding. The evaluation directorates are:

mailto:usarmy.APG.atec.mbx.g9-test-ops-division@mail.mil

16 ATEC Pamphlet 73-1 • 9 July 2013

(1) Ballistic Missile Defense Evaluation Directorate (BMDED): Serves as the Army operational test and evaluation arm of the Ballistic Missile Defense System (BMDS) Combined Test Force and lead service member of the BMDS Operational Test Agency (OTA) Team.

BMDED is located at Huntsville, Alabama.

(2) Command and Control Evaluation Directorate: Evaluates Army and joint command, control, and communications, business information and medical information systems.

(3) Evaluation Sciences Directorate (ESD): Provides evaluation planning, analysis, and reporting in the areas of Reliability, Availability and Maintainability (RAM), statistical analysis, Design of Experiments (DOE), and modeling and simulation. ESD conducts integrated suitability with the Integrated Logistics Support Evaluation Directorate and assists effectiveness evaluations through use of modeling, simulation, DOE, and statistical analysis.ESD also has the mission for the Army’s Center for Reliability Growth (CRG), which is co-led by ATEC and the Army Materiel Systems Analysis Activity (AMSAA). The purpose of the Army CRG is to identify reliability lessons learned from acquisition programs to provide the Army acquisition community with the policies, guidance, standards, contractual language, methods, tools, and training required to rapidly develop systems that meet the Soldier's reliability requirements and fit within the Army's budget.

(4) Fires Evaluation Directorate: Evaluates Army fire support (rockets and missiles, cannon, command and control) and air and missile defense systems.

(5) Integrated Logistics Support Evaluation Directorate (ILSED): Evaluates product support elements, manpower, and personnel integration. In coordination with ESD, ILSED provides integrated suitability position on evaluated systems. ILSED shapes and influences DOD and Army policies to improve logistics supportability and supports Deputy Assistant Secretary of the Army for Acquisition, Policy, and Logistics (DASA-APL) in their role as the Army Acquisition Logistician.

(6) Intelligence Evaluation Directorate: Evaluates intelligence-related acquisition programs covering national, theater, coalition, and commercial space, such as surveillance and reconnaissance, electronic and information warfare, and counter-IED equipment.

(7) Maneuver Air Evaluation Directorate: Evaluates manned and unmanned aviation (aircraft, air traffic control, munitions and support) systems.

(8) Maneuver Ground Evaluation Directorate: Evaluates infantry/Soldier systems, wheeled and tracked combat platforms, sensors and target acquisition systems, battle command systems, combat training simulators, and lethal and nonlethal weapons/munitions programs.

(9) Survivability Evaluation Directorate: Provides survivability, ballistic and non-ballistic battlefield threats, DOD-designated Live Fire Test and Evaluation (LFT&E) program management, vulnerability and lethality evaluations and reports of Army and designated joint systems. Also leads ATEC’s Information Assurance Task Force for the combatant commanders.

17 ATEC Pamphlet 73-1 • 9 July 2013

(10) Sustainment Evaluation Directorate: Evaluates programs that provide sustainment, mobility, maneuver support, quartermaster, ordnance, transportation, military police, engineer and chemical-biological defense systems.

d. OTC. HQ OTC is located at Fort Hood, Texas.

(1) OTC operates eight test directorates that test systems in realistic, varying environments to support Army evaluations of effectiveness, suitability and survivability. The eight test directorates are:

(a) Airborne and Special Operations Test Directorate: Plans, conducts, and reports on the Army’s airborne and special operations systems and techniques related operational tests in support of the acquisition decision-making process for the airborne contingency and Joint Special Operations Forces.

(b) Aviation Test Directorate: Plans, conducts and reports on manned and unmanned aviation related operational tests and field experiments, to include attack, reconnaissance, cargo and lift helicopters, fixed wing aircraft, tactical trainers, ground support equipment, and aviation countermeasure systems.

(c) Fires Test Directorate (FTD): The Army’s artillery, air and missile defense operational tester headquartered at Fort Sill, Oklahoma, with a Missile Test Division at Fort Bliss, Texas, FTD plans, conducts, and reports on operational testing of field artillery and air defense systems.

(d) Integration Test and Evaluation Directorate (ITED): The Army’s support to the fielding of an integrated network capability to the operating force, ITED plans, coordinates and conducts integrated operational test and RAI assessments in support of network integration and other priority Army systems.

(e) Intelligence and Electronic Warfare Test Directorate (IEWTD): The Army’s operational tester of intelligence, surveillance, reconnaissance; electronic warfare; biometrics;

and counter-threat systems. As the developers and implementers of the intelligence modeling and simulation for evaluation capability, IEWTD boasts a collaborative environment with their state-of-the-art intelligence systems integration laboratory.

(f) Maneuver Test Directorate (MTD): Lead operational tester for armor and infantry weapons systems and equipment. Providing a full complement of data collection, reduction, and management, MTD plans, conducts, and reports on operational tests and assessments of armor and infantry (light and mechanized) acquisition programs, Rapid Fielding Initiatives, and Rapid Equipping Force (REF) programs.

(g) Maneuver Support and Sustainment Test Directorate (MS2TD): MS2TD conducts operational tests in the areas of combat engineer, chemical, transportation, military police, quartermaster, ordnance, and medical service.

18 ATEC Pamphlet 73-1 • 9 July 2013

(h) Methodology and Analysis Directorate (MAD): Provides operations research/systems analysis in support of the OTC operational testing mission. Additionally, MAD provides design of experiments support within OTC.

(i) Mission Command Test Directorate (MCTD): Plans and tests systems for a net-centric environment that will process and transmit voice, data, messaging, and video information through networks at the tactical, operational, strategic, and sustaining base levels.

(j) Test Technology Directorate: Supports OTC test directorates in the planning, preparation, and execution of operational tests in a joint test environment by acquiring and sustaining a common integrated simulation, instrumentation, and data management capability.

(2) In addition, OTC has small cells at two locations:

(a) The Infantry Support Cell at Fort Benning, Georgia.

(b) A T&E Coordination Office at Fort Leonard Wood, Missouri.

e. ATEC Test Centers.

(1) ATEC operates eight subordinate test centers throughout the US providing the full spectrum of arctic, tropic, desert and other environments under natural or precisely controlled conditions. The test centers are responsible for the planning, execution and reporting of DT for Army acquisition systems.

(a) US Army Aberdeen Test Center (ATC), APG, Maryland. Major missions include:

Automotive – wheeled and tracked vehicles, Direct Fire Testing, Survivability/Lethality, Soldier Systems and Support Equipment, Training Exercises, Military Environmental Technologies, Maritime Systems, and Command, Control, Communications, Computers, Cyber (C5). In this single location, ATC can subject an item to a wide range of tests.

(b) US Army Electronic Proving Ground (EPG), Fort Huachuca, Arizona. EPG provides expertise for technical testing of Command, Control, Communications, Computers, Cyber and Intelligence, and Electronic Combat/Electronic Warfare equipment.

(c) US Army Redstone Test Center (RTC), Redstone Arsenal, Alabama. RTC provides expertise in testing of Missile, Aviation, and Sensor Systems, subsystems and components.

(d) US Army West Desert Test Center (WDTC), Dugway Proving Ground, Utah.

WDTC is the DOD major range and test facility base activity for Chemical, Biological, Radiological, Nuclear and Explosive (CBRNE) systems, to include smoke and obscuration.

WDTC accommodates training on CBRNE to active and reserve military units, civilian emergency responders, and other organizations.

19 ATEC Pamphlet 73-1 • 9 July 2013

(e) US Army White Sands Test Center (WSTC), White Sands Missile Range, New Mexico. WSTC is the primary organization for DT of air/missile defense systems nuclear weapons effects and directed energy weapons. In addition, WSTC provides expertise on electromagnetic environmental effects, indirect fire systems, and missiles and rockets. In addition, WSTC is the primary location for the Network Integration Evaluation Bi-Annual Events.

(f) Yuma Proving Ground, Arizona with subordinate elements at:

i. US Army Yuma Test Center (YTC), Yuma Proving Ground, Arizona. YTC is the Army’s hot weather test site with missions including combat vehicles and automotive systems, air delivery systems/airdrop, aircraft systems, engineering equipment (demolition, mines, and countermines), direct fire, Counter-IED, indirect fire, ground and airborne sensors, and unmanned aircraft systems. In addition, YTC supports Military training exercises.

ii. US Army Cold Regions Test Center (CRTC), Fort Greely, Alaska. CRTC plans, conducts, and reports developmental and operational tests, assessment, and experiments with an emphasis on Soldier system integration and participation, in the natural environments, snow, extreme cold, and sub-arctic conditions.

iii. US Army Tropic Regions Test Center (TRTC), headquartered at Yuma Proving Ground with sites in Schofield Barracks, Hawaii, Honduras, Suriname, and Panama. This suite of locations provides for testing of US Government materiel in extreme natural tropical (basic, high-humidity) environments, and natural, sub-tropical areas, and in temperate areas.

(2) The MRTFB constitutes a national asset-sized, operated, and maintained under uniform policies for DOD T&E support missions. The Army MRTFB is an aggregation of test activities, facilities, ranges, and equipment providing the Army with a first-rate military T&E capability.

The MRTFB is operated and managed under uniform reimbursement policy. DOD test customers utilizing the MRTFB are required to pay only those costs that are directly identified to the test. The indirect or overhead costs are funded by the MRTFB activity’s parent command.

ATEC manages five Army MRTFB activities: YTC (which includes CRTC and TRTC), WDTC, ATC, WSTC, and EPG. RTC is not an activity under the MRTFB. The only Army MRTFB activity not managed by ATEC is the Ronald Reagan Ballistic Missile Defense Test Site at the US Army Kwajalein Atoll. Detailed information on the MRTFB is reflected in DODD 3200.11.

1-8. ATEC Interface with Acquisition Community.

ATEC must interface with organizations within the DOD and the Army in order to accomplish our mission. These organizations are shown in Figure 1-2 and described below.

a. Office of the Secretary of Defense (OSD). OSD oversight extends to MDAPs and other programs designated via the OSD T&E oversight list. A program may be on T&E oversight for OT, DT, or Live Fire Test (LFT) or a combination of the three.

(1) The DOT&E performs operational and live fire T&E oversight.

20 ATEC Pamphlet 73-1 • 9 July 2013

(2) The Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)). The USD AT&L is the principal staff assistant and advisor to the Secretary of Defense and Deputy Secretary of Defense for all matters concerning acquisition, technology, and logistics.

(a) Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) located under the Deputy Under Secretary of Defense for Defense Research and Engineering (DUSD(DR&E)) performs DT&E oversight.

DOT&E

TRMC

OSD

DA

COMMAND

Secretary Of DefenseASD(NII)

DISA

JITC

AFOTEC

MCOTEA

COMOPTEVFOR

Service OTAs

DASD(DT&E)

ASD(R&E)

USD(AT&L)

Secretary of the Army

JIEDDO

MDA

FORSCOM DUSA-TE ASA(ALT)/AAE

Centers of Excellence

Battle Labs

TRADOC

RTC

ATC WDTC

TRTC

CRTCYPG

CoS Army/ Vice CoS Army

AEC

EPG

WSTC

CIO/G-6

Information Management

OTC

Items in Blue are described in this chapter

AMC

AMCOM

CECOM

TACOM

AMSAA

RDECOM

RDECs ARL

Biometrics (BIMA)

Army Surgeon General

DCS, G-1

Personnel

DCS, G-2

Intelligence

DCS, G-4

Logistics

MEDCOM

USA PHC

YTC

PEOs/JPEOs

PM ITTS

DCS, G-8

Resource Management

DCS, G-3/5/7

Operations

REF

Figure 1-2. Organizations Involved in T&E.

(b) Test Resource Management Center (TRMC). The TRMC plans for and assesses the adequacy of the MRTFB to provide adequate testing in support of development, acquisition, fielding, and sustainment of defense systems; and maintains awareness of other T&E facilities and resources, within and outside the Department, and their impacts on DOD requirements. The DASD DT&E is also the Director, TRMC.

(c) Missile Defense Agency. The Missile Defense Agency has developed research, development and test program focusing on missile defense as a single layered defense system.

ATEC participates as the Army operational T&E arm of the BMDS Combined Test Force, and lead service member of the BMDS OT Activity team.

(3) Defense Information Systems Agency (DISA), Joint Interoperability Test Command (JITC). JITC is DISA’s OTA for DISA-managed programs. They also provide T&E support to other government agencies on a cost-reimbursable basis. JITC has responsibility for certifying joint interoperability of all Command, Control, Communications, Computers, Cyber and

21 ATEC Pamphlet 73-1 • 9 July 2013

Intelligence (C5I) systems, and Automated Information Systems (AIS) that produce, use, or exchange information in any form and exchange information between Services, Agencies, or countries. JITC and ATEC have a memorandum of understanding (MOU) to facilitate the coordination of joint interoperability test requirements for IT/NSS defense acquisition and defense technology programs.

(4) JIEDDO. DODD 2000.19 established a Joint IED Test board under the lead of ATEC to coordinate all Joint IED Defeat developmental testing in DOD.

b. OTAs. Office of Management and Budget Circular A-109 and DODD 5000.1 direct that each military department establish an independent operational test activity to plan and conduct operational test and evaluation. The process for designating a lead Service for a multi-service operational test and evaluation is outlined in the Memorandum of Agreement (MOA) between all the Service OTAs. This MOA is updated annually. The current service OTAs are:

• Air Force Operational Test and Evaluation Center (AFOTEC).

• ATEC.

• Marine Corps Operational Test and Evaluation Activity (MCOTEA).

• Navy: Commander, Operational Test and Evaluation Force (COMOPTEVFOR).

c. Army.

(1) The Army Acquisition Executive (AAE) is responsible for all Army acquisition T&E planning, programming, and budgeting. Currently the AAE is the Office of the Assistant Secretary of the Army (Acquisition, Logistics, and Technology) (ASA (ALT)). The PEO reporting chain is through the AAE to the Defense Acquisition Executive (DAE).

(2) The Deputy Undersecretary of the Army for Test and Evaluation (DUSA-TE)/Army T&E Executive is the key Army proponent for Army T&E policy and for the DOD chemical and biological defense T&E program (CBDP). The DUSA-TE is also the Army T&E Executive.

(3) Army Commands.

(a) The US Army Forces Command (FORSCOM) supports T&E by providing user troops and units through the TSARC process.

(b) TRADOC is the principal capabilities developer, doctrine and training developer, and trainer for materiel and tactical C5I/IT systems. The TRADOC Capabilities Manager (TCM) serves as the user representative on the T&E Working-level Integrated Product Team (WIPT). The TCM coordinates with PMs to facilitate and synchronize new equipment/capabilities fielding. TRADOC supports T&E by providing subject matter experts requested through the TSARC process.

(c) The US Army Materiel Command (AMC) manages the Army’s science and technology base through its laboratories and research, development, and engineering centers.

One of its components, the US Communications-Electronic Command’s (CECOM) Army

22 ATEC Pamphlet 73-1 • 9 July 2013

Participating Test Unit supports joint interoperability testing of C5/IT systems by conducting tests, performing analysis, and generating joint interoperability certification reports for all Army elements and systems.

i. The US Army Research, Development, and Engineering Command (RDECOM) develops technologies for the Army through eight major laboratories, such as the Army Research Laboratory (ARL), and research development and engineering centers, such as the Edgewood Chemical and Biological Center.

ii. AMSAA is the AMC’s and the Army’s Center of Excellence for materiel and logistics systems analysis to support decision making throughout the entire system life cycle.

AMSAA’s primary mission is to conduct analyses across the materiel lifecycle to inform critical decisions for current and future Soldier needs. In addition, AMSAA co-leads the Army CRG with ATEC, and provides expertise in reliability engineering and reliability growth methodology.

iii. ARL provides expertise through the Survivability and Lethality Analysis Directorate (SLAD) and through the Human Research and Engineering Directorate (HRED).

(4) Other HQ DA staff elements with involvement in T&E are:

(a) Deputy Chief of Staff (DCS), G-1 provides guidance regarding MANPRINT.

(b) DCS, G-2 provides guidance regarding threat issues.

(c) DCS, G-3/5/7 reviews and coordinates critical operational issues and criteria (COIC) for all non-tactical C5I/IT programs. Although a DOD effort, the Biometrics Identity Management Agency (BIMA) operates under the Office of Provost Marshall General, through the executive agent authority given to the Secretary of the Army. BIMA provides biometric oversight and policy on all DOD biometric standards and requirements as well as providing coordination with non-DOD agencies. The Biometrics Fusion Center (located in West Virginia) provides standards performance, automated biometric identification system interoperability, and some developmental testing of biometric acquisition systems, and coordinates with ATEC to provide results of biometric testing and ensure biometric test support for operational test events, as necessary.

(d) DCS, G-4 provides ILS guidance. DCS G-4 coordinates regularly with ILSED on sustainment issues and collaborates with AEC during the semi-annual ATEC/DASA-APL reviews.

(e) DCS, G-6 provides a joint, net-centric information enterprise that enables Soldier decision-making superiority. The Central Technical Support Facility (CTSF) at Fort Hood, Texas, serves as the Army’s integration, compliance, and certification activity for Soldier and business management areas of the Army’s enterprise information environment. In this role, the CTSF is part of the ATEC T&E environment by playing a role similar to DISA or JITC. DCS, G-6 also manages T&E life cycle activities for IT systems in support of the AAE.

23 ATEC Pamphlet 73-1 • 9 July 2013

(f) DCS, G-8 plans, programs, and budgets research, development, T&E, Army procurement appropriations, and operation and maintenance T&E funds. G-8 also reviews, coordinates, and approves COIC for all materiel and tactical C5I/IT programs. G-8 manages, solicits, and coordinates Army participation in JT&E.

(g) The REF is an organization that takes its guidance from HQDA G-3 and reports directly to the Vice Chief of Staff of the Army (VCSA). The REF’s mission is to rapidly increase mission capabilities while reducing risk to Soldiers and others. It works directly with operational commanders to find promising materiel solutions to their identified operational requirements. ATEC works with the REF and materiel developers to collect and provide as much information as is known about the materiel solution to the operational commanders. The REF accomplishes its mission in three ways:

i. Equips operational commanders with off the shelf (government or commercial) solutions or rapid developmental items that can be researched, developed, and acquired quickly.

ii. Inserts future force technology solutions that engaged and deploying forces require. Subject to the time available, ATEC tests and evaluates key technologies and systems under operational conditions and provides a Capabilities and Limitations Report (CLR) and a Safety Confirmation (SC) prior to a RAI system being issued for troop use. ATEC also provides a safety release for testing of RAI systems/items involving Soldiers as test participants.

iii. Addresses capabilities and advises Army staff of findings that will enable our forces to rapidly confront an adaptive enemy.

(h) US Army Medical Command (MEDCOM). MEDCOM serves as the Materiel Developer (MATDEV), capabilities developer, doctrine developer, training developer, trainer, tester, and evaluator for assigned systems. They also maintain a human research review via the US Army Medical Research and Materiel Command and provide health hazard assessments via the US Army Public Health Command (PHC) for acquisition systems prior to conduct of OT.

The Surgeon General is the tester and evaluator liaison/point of contact for the health hazard assessment program.

d. Other Government agencies (not shown in Figure 1-2).

(1) National Security Agency (NSA). ATEC provides limited testing and evaluation services to NSA for some intelligence-related acquisition programs, surveillance and reconnaissance, electronic and information warfare, covering national, theater, coalition and commercial space.

(2) Department of Homeland Security (DHS). ATEC provides limited testing and evaluation services to the DHS and various DHS components including the Transportation Security Administration, Customs and Border Protection and others. ATEC and DHS have collaborated on, and will continue to dialogue on aspects of T&E policy, including best practices to support federal acquisition programs and T&E training.

24 ATEC Pamphlet 73-1 • 9 July 2013

Chapter 2 Test and Evaluation Management

2-1. Overview.

a. Today’s Army has a growing need to ensure that fielded systems as well as the new systems in development are designed to work as part of a larger System of Systems (SoS). They must be compatible, interoperable, integrated, and usable by the Soldier. ATEC is designated as the independent Tester and Evaluator of Army materiel systems and for designated multi-service and joint systems. AEC performs evaluation planning and uses the test resources of OTC, ATEC Test Centers, and the materiel developer (MATDEV) to carry out detailed evaluations that encompass a system’s effectiveness, suitability, and survivability (ESS) and safety in technical and operational contexts. ATEC provides a vital source of information to the Soldier and decision makers. ATEC also closes the feedback loop with MATDEVs so that every system’s iteration represents an incremental improvement in Soldier capabilities.

b. DOD Acquisition T&E Overall Process. The concurrent and interdependent acquisition-related process that impacts the ATEC mission is illustrated in figure 2-1. Although the figure depicts the DOD 5000 acquisition system timeline from start to finish, systems will enter into the acquisition process at various milestones based upon technology maturity. It is possible that not all processes will apply to some systems. Evaluation programs will be tailored, based upon elements of the overall DOD acquisition process, to the specific requirements, conditions and contexts associated with each program.

Figure 2-1. DOD Concurrent Acquisition Processes.

Post-PDRA Post-CDRA

FRP

Decision Review LRIP/IOT&E

ISP

Pre-Systems

System Acquisition Sustainment

System Specific

= Decision Point = Milestone Review = Decision Point if PDR is not conducted before MS B

A B C Materiel Solution Analysis

Materiel Development Decision

Technology Development

Engineering & Manufacturing Development

Production & Deployment Operations & Support

J-6 I&S Certification

JCD ICD CDD CPD

J-6 I&S Certification

J-6 System Validation

Information Assurance

(DOD8500.1)

ATEC T&E

Process

(ATEC 73-1)

System Acquisition

(DOD 5000)

JCIDS

(CJCSI 3170)

IT & NSS

I&S

(CJCSI 6212)

[NR-KPP]

IA Cap

MAC/CL

IA C&A

Planning

IA DT

C&A Testing

Certification Review

IATO

IATT

IA OT ATO

ESR /

CIPR

OTA

TP/DTP

SEP

TEMP

DT

MS-B

OMAR

SC

SEP-U

OTA

TP/DTP

OTRR

SR

LUT

MS-C

OMAR/

MRM

SC

OTA TP

OTRR

SR/IOT FOT

OER

SC

OFER TEMP-U

DT

25 ATEC Pamphlet 73-1 • 9 July 2013

2-2. Overview of the Joint Capabilities Integration and Development System (JCIDS) (Chairman Joint Chief of Staff Instruction (CJCSI) 3170).

a. The JCIDS is a system for translating mission requirements, current capabilities, concepts of operation and needs into doctrine or materiel–based solution alternatives that can be analyzed and developed through the DOD 5000 framework. The requirements process supports the acquisition process by providing validated capabilities and associated performance criteria to be used as a basis for acquiring the right systems. T&E will assess whether new or modified systems deliver their intended capability within the applicable functional capabilities area. There will be a need to consider realistic test environments, including joint mission and joint test environments, to assess an individual system’s contribution to joint mission capability.

Presently, T&E is often focused on materiel solutions, but in the future non-materiel solutions may be given more emphasis, in light of budgetary constraints.

b. JCIDS document sponsors may pursue a variety of approaches to determine their organizational capability requirements, depending upon the timeliness of the assessment and the scope of the activities being reviewed. Each approach for identifying capability requirements should not presuppose a specific capability solution or end item, but provide data related to forms and functions of potential solutions to support the development of JCIDS documents.

Identified capability requirements will be traceable to approved Joint concepts developed in accordance with a Concept of Operations (CONOPS). Specifics are discussed in the JCIDS Manual.

c. JCIDS documents serve as a means for sponsors to submit identified capability requirements and capability gaps, along with other relevant information, for review and validation.” For materiel solutions, the JCIDS documents of interest to T&E are the initial capability document (ICD), capability development document (CDD), the capability production document (CPD), the urgent operational need (UON), and the joint UON (JUON) or joint emergent operational needs (JEON). The format for each of these documents and guidelines for ATEC review can be found in Appendix C.

d. An ICD is the most common starting point for new capability requirements to provide the desired joint warfighting capability. The ICD, documents one or more new capability requirements and associated capability gaps. The capabilities developer develops the ICD in close coordination with the PM. The ICD supports the analysis of alternatives (AoA) of possible materiel solutions and the Milestone (MS) A acquisition decision.

e. The CDD is the primary means of proposing refined capability requirements, in the form of Key Performance Parameters (KPPs), Key System Attributes (KSAs), and additional performance attributes. The CDD is associated with a particular solution intended to wholly or partially satisfy validated capability requirements and close or mitigate associated capability gaps. The CDD will capture knowledge gained about individual technologies that will support the integrated system. There may be experiments and demonstrations to determine the most effective technologies to integrate in order to close the capability gaps identified in the ICD. The PM, who has sponsored the experimentation or demonstration process, will determine when the experimentation or demonstration is complete. If additional development is required before

26 ATEC Pamphlet 73-1 • 9 July 2013 fielding, the PM will coordinate with the capabilities developer who creates a CDD or modifies an existing CDD to guide the development process. CDDs are system specific and are drawn upon heavily throughout an acquisition program.