TE_1_WORKFLOW_DIAGRAM.pdf

Test Business Processes

White Sands Missile Range

JERMAIN.M.COMPTON

Typewritten Text TE 1: Workflow Diagram

Test Customer Process This process can be very simple or very complex dependent on the system, point in acquisition process and purpose of the test or test series (such as full rate production)

Example Test Types & Factors:

• Army Evaluated Program of Record

• ACAT I / DOTE Oversight

• New Start, Army Wide Fielding

• System of Systems

• Threat / Targets

• Army Product Improvement

• Existing Fielded Capability - Evolving

• Component / System / Software Upgraded

• Threat / Targets

• Air Force / Navy / Joint Interoperability

• WSMR is Not the Responsible Test Organization

• Test Planning Based on Program Test Plan Data Requirements

• Threat / Target

• Research & Development

• Technology Readiness Level (TRL)

• Type of System (Research Development Engineering Command)

• Other Government Agency

• System Type, Objectives, Data Requirements

• Typically Providing Data or Data Reports

• Commercial / Private Industry

• Cannot Compete with Private Industry for Test Services

• Support Must be Unique Capability Not Available in Private Sector

• Space Industry is an exception (Commercial Crew Transport)

• Typically Providing Data or Data Reports

• Foreign Military Sales

• Sales of U.S. Capabilities (Patriot, GMLRS, Thaad…)

• Sale of Test Services (Chu-SAM KAI)

• Training

The Flow Presented Here is a Snapshot of the Process for an Army Evaluated Program as a New Start

T&E In DoD Evolutionary Acquisition

Research and Development

Concept & Tech Development

MS A

System

Dev. & Dem.

(program Initiation)

MS B

LRIP

MS C

Full Rate Production & Deployment

Meeting the Warfighter’s

Needs

• Tech Feasibility Testing

• Tech Expertise Base on T&E Experience

• Test Assets as Surrogate Tech Demonstrators

ATEC Systems Team (AST)

ATEC Evaluator - Chair

WSMR Developmental Test Engr.

Operational Tester

System Evaluator’s (AEC)

User Representative

• Tech Assessment – Demonstration Test – M&S

• Prep/Plan Instr / Data to support Reqmnts Val/Ver

• Engineering Technical Consultation

• SEIPT/ TEIPT - Testability / Instrument / MOPs / MOEs

• TEMP/ SEP / Detailed Test Plan and Execute

• Support Confidence Testing / Proof of Princ

• Share Findings / Rec Solutions Expedite

• System Level PQT, Interoperability, M&S Val

Improvement & Stockpile

Ver/Val Testing

• Production Qual Test

• M&S Extrapolation

• DT>OT - Readiness

Test – Key to Acquisition Life Cycle

Acquisition Life Cycle

Notional Resource Distribution

C U

M U

L A

T

IV

E C

O S

T

R E

S O

U R

C E

S

TIME

D E

S

IG

N

P ro to ty pe

A o

A

C O

E A

T E

S T

P R

O C

U R

E

F

IE

L

D Operate / Maintain / Sustain – Improve –

Adjust to threat & technology changes

C U

M U

L A

T

IV

E C

O S

T

R E

S O

U R

C E

S

TIME

D E

S

IG

N

P ro to ty pe

A o

A

C O

E A

T E

S T

P R

O C

U R

E

F

IE

L

D Operate / Maintain / Sustain – Improve –

Adjust to threat & technology changes

Successful Testing is Key to Acquisition Objectives

- Yet Perceived as Costly & Time Consuming -

EFFECTIVE TESTING

• Finds / Solves Issues

• Characterizes Performance

• Increases Suitability and Survivability > Reduces Sustainment Cost

The shape of this curve is highly dependant on the success of the T&E Phase

R

IS

K t o M

IS

S

IO

N

/C o n fi d e n c e

Testing / Resources - Time/Cost

Integrated Risk Management

It’s ALL about LEARNING - Reducing RISK - Increasing Confidence

REQUIRED INFORMATION

Standard questions

Performance

Effectiveness

Suitability

Survivability

Etc…

WHERE DO YOU GET IT? > M&S, DT&E, OT&E

HOW MUCH IS ENOUGH? > RISK MANGEMENT PLAN

Balancing Act:

What Tool / How Much = Acceptable Risk?

• Too Little Test = Too Much Risk

• Too Much Testing = Too Much Cost/Schedule for Return

• Test Managed by Value Ratio = Risk Reduction / Cost Decision point – Acceptable Risk

Customer Contact / Opportunity Identified Existing Customer

Evaluator / ATEC Systems Chair TEMP / OPORD Colleague Referral / Workforce Contact

Networking / Request for Proposal 1-866 Phone Call Strategic Planning Research

Web Page Inquiry Conference / Symposium Booth

INPUT

Follow-up / Research Test or Support Objectives

Who are the players / Customer?

What is the goal / expected or derived level of effort?

Strategy and Schedule?

Is a visit needed /required?

Evaluate Effort Type of Customer

ROI, Risk, Impact to Primary Mission, Environmental, Asset Availability and Requirements, Partners, TEAM WSMR Inputs

Recommendation to TC / Directors / Commander on

Course of Action (as appropriate)

Terminate Effort

Check/Enter Contact into Leads Database

Store Data

Preliminary Information to TC and Command

<Feedback>

Data

Type of Customer – DoD, Army

Foreign, Commercial, OGA

CMND

Consensus on Course of Action

A

DOD

Customer

ATEC

10 Series

Reg

Exception Waiver

DoD, Army Customer

AR 73-1

Test Service Agreement Test Service Agreement

LessonsLessons

Closure / Monitor

Closure / Monitor

WSTC PRAT

Determine

Lead Org as Appropriate

WSTC PRAT

Determine

Lead Org as Appropriate

ROM $ Est Reqmnts/Concept

Similar Tasks Experience Envirn

Assumptions

WBS

ROM $ Est Reqmnts/Concept

Similar Tasks Experience Envirn

Assumptions

WBS

Win Customer

Win Customer

INPUT Available Info Statement of Work

System Specification RFP – System Concept Draft TEMP, outline TP

INPUT Available Info Statement of Work

System Specification RFP – System Concept Draft TEMP, outline TPSpecific Support, Cost

RFP Response Unsolicited Proposal

Test/Trn/Tech Solutions Test Concept / Location Specific Support, Cost

Test Officer

Capture Plan Assemble Team

Dev Win Strategy Technical Concept

Capture Plan Assemble Team

Dev Win Strategy Technical Concept

TO / Proposal Manager

Initial Cost Estimate Program Management Module

Work Breakdown Structure Planning Funds

Test Officer / Analyst

Test Program Initiation ATEC Decision Support System

(ADSS)

Request For Test Services

Initial Information Evaluator / Program Manager TEMP, Test Plan, Technical

Specification, Capability Development Document, Scope of Work – Flight Safety

Customer Funding Recv’d GFEB WBS Established

Type When Who Basis

1. Acquisition Base Line Level ROM

2 to 5 yrs out Test Officer Historical similarity and experience

2. Planning ROM 6 months to 2 yrs out / annual

Test Officer, Range Engr

Test concept or Historical similarity and experience

3. Official WSTC Test or Test Series Estimate

2 wks to 2 yrs out / annual

Test Officer, Range Engr, Proposal Mngr

SOW, Test Plan, OR

4. Official WSTC Test Proposal (new customer)

30 days to 2 yrs out

BD Proposal Mgr SOW, Test Plan, RFP, Test Requirements Document

5. WSTC Support (to OTC/ other Test Center)

6 months to 1 yr Test Officer, Range Engr

Test Resource Plan, SEP, SOW, Test Plan, Test Requirements Doc

6. FMS Case 6 months to 2 yrs out / annual

Test Officer, Proposal Mgr

SAMD/USASAC, LOA, SOW, Test Concept/ Test Plan

7. Commercial Industry 2 wks to 2 yrs out

Test Officer, Proposal Mgr

SOW, Test Standard, samples, Test Plan, Test Requirements Document

Planning Cost Estimate Used for advanced planning to meet timelines

Test Officer / Analyst

MOA Establish (if not existing) DD1144 Support Agreement Complete

Support Plans Optics, Radar, Telemetry, Met, Flight Safety, Communications Plan, RFA, Evacs, Road Blocks

Universal Documentation System

(UDS)

Operational Requirements

Support / Planning Activities

System Eng. Team T&E Integrated Product Team

ATEC Sys. Team Test Design

MT / Customer Prepare UDS

Documentation

Program Introduction :

Brief, Scope Objectives

Known Data Requirements

IM:

Coms Engineer

Voice / Data Nets Radio Freq Auth

Information Assurance

RO:

Range Engineer

Optics, Telemetry, Radar, Met, Operations Real Time, Flight Safety

Scheduling

Operational Requirements Coordination Document

For Range Operations Only (Open Air Range Assets)

Vision Digital Library

Information Center

Vision Digital Library Defense Technical Information Center

PMM / ADSSPMM / ADSS

Update

(Cost Estimate)

Detailed Test Plan Entire Qual. Test

Specific Test Event Staffing is dependant on purpose

Of test, ACAT Level

APPROVED TEST CENTER, AST

AEC, ATEC, PM, Contractor

Supporting Documents Initial Capabilities Doc Capabilities Devel Doc System Specification System Design / ICD

Threat Assessment Report Safety Analysis Report Security Class Guide

IPT Documents Evaluation Strategy

Sys. Eval Plan (AEC)

TEMP,

Data Source Matrix Event Design Plan

Parallel Planning Efforts

• Test Site Layout and Logistics

• Master Planning Board Actions

•Threat, Electronic Warfare / Targets

• Procurement Actions / Contracting, Supplies

• SECURITY SOP

• SAFETY SOP – based on SAR, Test Procedures

(may be multiple SOPs – each operation)

• ENVIRONMENTAL DOCUMENTAION

(EIS/EA/REC)

• TEST OPERATIONS PROCEDURES

• DATA COLLECTION PROCEDURES

• RAM – data collection plan

• System Configuration Control

• Data release procedures – databases

• Modeling and simulation

• System Level Instrumentation

• (SV) Climatics, Dynamics, Electromagnetic…

• Logistics, latrines, dirt work

• Ammunition Supply (TAMIS, AMCATS…)

• OTHER Supporting Agencies (TSMO, TMO..)

Elements of the DETAILED TEST PLAN

• Test(s) Objectives / Criteria / Specification

• Sub Tests (component and/or System)

• Safety / Human Factors

• Reliability / Maintainability

• Performance

• Environments (climatic/dynamic/e3)

• Survivability / Vulnerability

• INR, E3, Secure lighting, signatures

• Interoperability / Compatibility

• Data Acquisitions Procedures

• Software Independent Verification/Validation

• Data Reduction and Analysis Methodology

• Performance Measures – Samples size

• Effectiveness Measures

• Modeling and Simulation

• Security / Data Management

May be Multiple Test Plans or a Single one Describing the Methodology for Acquiring the Entire Set of Data for the Program Goal

Live Fire Test (HOT) System Ready for Test Event

Missile Build, Target Conditioning, System Inst.

Checks, Beacons, TM, FTS

T-2 Day All Elements and Program

Report on Readiness Go/No Go, Objectives, Systems in Place, Target System Ordinance Delivered

Test Firing Readiness Check (FRC)

Coded mission is often First conducted as a Firing

Readiness Check Serves as an end-to-end Verification of Timing, Adequacy and readiness

For the system under test as Well as most support and Instrumentation Elements

Verifies Countdown Procedures and

Interference

Sub Test Data Supporting Element

Reports / Briefs AST / AEC /Customer

Fix-Test-Fix

Scoring Conf

Test Incident Reports NGT 3 Days of

Incident

ATIRS, VISION

Scoring Conf

System Level Testing Numerous System Test

Measures Such as:

Environmental Static Sensor Tests E3

INR

Dynamics Inspections – xray NDI Chem / micro-bio Software Countermeasures

Test Resource Management System

(TRMS)

OR defines resources for Schedule Schedule based on System Readiness

Supporting Resource Availability

2a

Other/ External Resources Targets, EW, Aircraft, Contracts, Supplies, latrines, propane, fuel, vending, DPW, LRC, Ranchers, State, Local, Public

DATA ACQUISITION & REVIEW (hot wash) Real Time Data Quick Look Data Reduction Collect Data Products /tapes/disks Ground Truth Data

Optics, TM, Radar, Safety, Performance Survey, Impact, Recovery, Target Meteorology, Video

System Data – pressures, accel, temps

Data Reduction and Analysis Verification / Validation

Scaling, Deviations from Expected Performance, Performance Assessment

Quick Look Data Per Data Management Plan

Preliminary Results to Customer And Analysts

T-0 Hot Mission Tracking

Missile Firing, Lasing All Support Elements

Timing, Commo, Optics, Radar, Telemetry, Flt Safety Grnd Safety, POL, Security Roadblocks, EOD, Recover

Customer, Analyst, Warheads Targets, RFA, Real Time Data

After Action Review Lessons Learned

Issues Customer Feedback

Target Results Lethality, Impact Area

Miss Distance Dispersion Pattern, Recovery…

System Data

SW IV&V, RAM /HFE

Operator Survey Test Conductor Log…

Test Report

Firing / Data Report (days) Event Test Report (day-month) Final Test Report (30 days +)

Analyst / Test Officer Generates Appropriate

Report

Internal Review, Approved by

Director / TC Commander Distribution List

Other Activities

Brief to PM / Milestone Decsion Authority / User OT Readiness Reviews Product Improvements

RAM Scoring Conference

Follow on Test Next Phase

Customer Survey / Feedback

Close Project

PMM ADSS GFEBS

Event Closeout / Update

Retire ADSS Close GFEBS

Test Process Project Team Approach

Project Team Members Test Officer Test Conductor Analyst Subject Matter Experts Range Engineer Flight Safety, Safety, Instrumentation, Data Collection

Test Planning Review Test and Evaluation Master Plan Identify Performance Criteria Design Test Prepare NEPA Documentation Prepare UDS

Documentation Prepare Safety Standard

Operating Procedures

Test Execution and Data Collection System Level Test Open Air Flight Testing Environmental Effects Warheads

RAM / MANPRINT

System Performance Electromagnetic Effects

Data Reduction and Test Reporting Near-Real-Time Quick Look Report (same day) Test Incident Report Firing Report Mission Final Report Final Program Assessment Report

Mission Report

Test Process

QUESTIONS?