Naval_Systems_Engineering_Tech_Handbook.pdf

MARCORSYSCOM Order SPAWARINST 5000.1 NAVFACINST 5000.15 5400.5

NAVSUPINST 5000.21 NAVSEAINST 5000.9 NAVAIRINST 5000.24

Version 1.0 Enclosure (2)

Naval Systems Engineering Technical Review Handbook

Version 1.0

DEPARTMENT OF THE NAVY

NAVAL SEA SYSTEMS COMMAND, WASHINGTON NAVY YARD, DC 20376-4065

NAVAL AIR SYSTEMS COMMAND, PATUXENT RIVER, MD 20670-1547

NAVAL SUPPLY SYSTEMS COMMAND, MECHANICSBURG, PA 17055-0791

NAVAL FACILITIES ENGINEERING COMMAND, WASHINGTON NAVY YARD, DC 20374-5065

SPACE AND NAVAL WARFARE SYSTEMS COMMAND, SAN DIEGO, CA 92110-3127

MARINE CORPS SYSTEMS COMMAND, QUANTICO, VA 22134-6050

DISTRIBUTION STATEMENT A: Approved for public release;

distribution is unlimited.

Naval Syscom Engineering Technical Review Handbook

Version 1.0 Foreword-1

Enclosure (2)

FOREWORD

The Systems Engineering Technical Review (SETR) Handbook provides guidance to implement Naval SYSCOM Systems Engineering Policy (MARCORSYSCOM Order 5000.5, SPAWARINST 5000.1, NAVFACINST 5000.15, NAVSUPINST 5000.21, NAVSEAINST 5000.09, and NAVAIRINST 5000.24).

The Handbook identifies planning, execution, and follow-on activities for the SETR process. These activities apply to all Naval System Commands (SYSCOMs), with the exception of NAVAIR, executing engineering programs for acquisition and modernization of naval systems, System-of-Systems, and Family-of-Systems.

The SETR process is integral to naval systems engineering, and it is consistent with existing and emerging commercial standards.

These SETRs provide program management with assessments of program technical health and maturity at key points in the development life cycle. The SETR process consists of several technical assessments. Each SETR assessment is focused on verifying technical health and maturity by examining objective products representing the program work accomplishments to date. Each technical assessment culminates in a formal meeting that documents recommendations to program management concerning the continuation of work into the next stage of development. SETRs formally review and evaluate whether required systems engineering tasks have been completed successfully before proceeding beyond critical events.

The Milestone Decision Authority (MDA) can allow their Program Managers have flexibility to tailor the SETR to match their program circumstances. The SETR must be described and updated in the Systems Engineering Plan (SEP), which is approved by the Milestone Decision Authority (MDA). Tailoring of SETRs is expected to match the complexity and risks of the program. The tailored SETR must have sufficient technical breadth and depth to assess the maturity of technical work to date and to identify risks associated with continuing the development efforts.

The SETR process is event-driven, not schedule-driven. SETR assessments are conducted when the system is ready for review, in accordance with the program SEP. Entry and closure criteria, as well as areas of interest, are identified in the SEP to govern the SETR schedule.

Version 1.0 Foreword-2

A Technical Review Board, chaired by a senior government employee appointed by the SYSCOM Chief Engineer (CHENG), conducts the SETR assessments in collaboration with program management. However, the SETR Chair is independent of the program to avoid conflicts of interest and has expertise reflective of the program complexity to ensure technical integrity of the review.

Version 1 i

TABLE OF CONTENTS

Page

1.0 SCOPE ................................................ 1-1

2.0 REFERENCES /APPLICABLE DOCUMENTS ........................ 2-1

3.0 DEFINITIONS ............................................. 3-1

4.0 GENERAL INTRODUCTION .................................... 4-1

4.1 PURPOSE ................................................. 4-1

4.2 BACKGROUND .............................................. 4-3

4.3 OBJECTIVES .............................................. 4-5

5.0 OVERVIEW ................................................ 5-1

5.1 SETR PROCESS ............................................ 5-1

5.2 ESSENTIAL SETR ASSESSMENTS .............................. 5-5

5.3 TIMING AND TRIGGERING ................................... 5-6

5.4 SETR INFORMATION ...................................... 5-9

5.4.1 INFORMATION HIERARCHY ................................. 5-9

5.4.2 ENGINEERING FUNCTIONAL AREAS ........................ 5-11

5.5 CLOSURE CRITERIA ...................................... 5-12

6.0 SETR PROCESS PLANNING ................................... 6-1

6.1 SETR TAILORING ......................................... 6-22

6.2 TRB MISSION AND MEMBERSHIP .............................. 6-2

6.2.1 TRB CHAIR ............................................. 6-4

6.2.2 RECORDER .............................................. 6-4

6.3 SETR PARTICIPANTS ....................................... 6-5

6.3.1 SUBJECT MATTER EXPERTS ................................ 6-5

6.4 PROGRAM INTEGRATED PRODUCT TEAM ......................... 6-6

6.4.1 PROGRAM LEAD ENGINEER ................................. 6-6

7.0 SETR ASSESSMENT PLANNING .............................. 7-1

7.1 STAFFING AND PREPARATION ................................ 7-1

7.1.1 TECHNICAL REVIEW ACTION PLAN .......................... 7-2

7.2 AGENDA FOR SETR MEETING ................................. 7-3

7.3 WORK-UP TO SETR MEETING ................................. 7-4

7.4 CONDUCT OF SETR MEETING ................................ 7-55

7.4.1 TECHNICAL REVIEW SUMMARY REPORT ....................... 7-5

7.5 CLOSING SETR ASSESSMENT ................................. 7-6

8.0 REQUEST FOR ACTION/REQUEST FOR INFORMATION PROCESS .. 8-1

9.0 SETR INPUTS TO GATE REVIEWS .......................... 9-1

9.1 ITR INPUTS TO GATE REVIEW ............................... 9-1

9.2 ASR INPUTS TO GATE REVIEW ............................... 9-1

9.3 SRR INPUTS TO GATE REVIEW ............................... 9-2

9.4 SFR INPUTS TO GATE REVIEW ............................... 9-2

ii Enclosure (2)

9.5 PDR INPUTS TO GATE REVIEW ............................... 9-2

9.6 CDR INPUTS TO GATE REVIEW ............................... 9-3

APPENDIX A. FACILITIES AND INFRASTRUCTURE SETR CHECKLISTS ... A-1

APPENDIX B. SAMPLE RFA/RFI FORM ............................. B-1

APPENDIX C. SETR DOCUMENTS .................................. C-1

ENCLOSURE 1. SHIP-SPECIFIC SETR GUIDANCE ................... E1-1

ENCLOSURE 2. AIR-SPECIFIC SETR GUIDANCE .................... E2-1

ENCLOSURE 3. C4I-SPECIFIC SETR GUIDANCE .................... E3-1

ENCLOSURE 4. LAND-SPECIFIC SETR GUIDANCE ................... E4-1

ENCLOSURE 5. INTEGRATED WARFARE SYSTEMS

SPECIFIC SETR GUIDANCE ...................... E5-1

iii Enclosure (2)

LIST OF FIGURES

Page

Figure 5-1. Timeline for Activities in Technical Interchange Meetings. ..................................................... 5-5

Figure 5-2. Example of SETR Assessments for Program Initiation at Milestone A. Position of Technical Reviews Is Notional. ...... 5-7

Figure 5-3. Example of SETR Assessments for Program Initiation at Milestone B. Position of Technical Reviews Is Notional. ...... 5-8 Figure 7-1. Planning Timeline for SETR Assessments. .......... 7-2

LIST OF TABLES

Table 5-1. List of Systems Engineering Technical Review Assessments. .................................................. 5-1 Table 5-2. Recommended Systems Engineering Technical Reviews for Acquisition and Modernization Programs. ....................... 5-6

1-1

1.0 SCOPE

The Systems Engineering Technical Reviews (SETRs) described in this Handbook apply to all Naval System Commands and affiliated program authorities as indicated in the Naval SYSCOM Systems Engineering Policy (MARCORSYSCOM Order 5400.5, SPAWARINST 5000.1, NAVFACINST 5000.15, NAVSUPINST 5000.21, NAVSEAINST 5000.09, NAVAIRINST 5000.24). The review procedures are common to all naval engineering programs for systems, subsystems, and configuration items, as well as System-of-Systems and Family-of- Systems.

Enclosures (1) through (5) present interim SETR guidance applicable to ship, air, space, enterprise information system, command-control-communications-computer-intelligence (C4I), land, and integrated warfare systems (IWS), respectively. System-specific SETR guidance is under development and will be included in enclosures (1) through (5) upon first revision of this Handbook.

2-1

2.0 REFERENCES/APPLICABLE DOCUMENTS

Various government policy and guidance documents, as well as public technical information sources, were used to develop the Handbook. These reference materials are listed below.

a. Defense Acquisition University, Interim Defense Acquisition Guidebook of 15 Jun 09 https://acc.dau.dag).

b. Defense Acquisition University, Systems Engineering Community of Practice, (https://acc.dau.mil/se).

c. Defense Systems Management College, Systems Engineering Fundamentals of Jan 01, (http://dau.mil/pubs/pdf/SEFguide2001-01.pdf)

d. DoDI 3150.09, The Chemical, Biological, Radiological, and Nuclear (CBRN) Survivability Policy of 17-Sep 08.

e. DoD Guide to Integrating Systems Engineering into DOD Acquisition Contracts, Version 1.0, of 22 Dec 06.

f. MIL-HDBK-237D, Electromagnetic Environmental Effects and Spectrum Supportability Guidance for the Acquisition Process of 20 May 05.

g. DoN, Acquisition and Capabilities Guidebook, “Chapter 7, Systems Engineering and Human Systems Integration,” of 2005.

h. DoN Joint Letter, Implementation of System Design Specification (SDS) Guidebook and associated system specific appendices of 18 Jul 08.

i. DoN, System Design Specification Guidebook of 17 Jul 08.

j. DoN, Office of the Assistant Secretary of the Navy for Research, Development, and Acquisition, Guidebook for Acquisition of Naval Software Intensive Systems, of Sep 08.

k. DoN, Deputy Chief of Naval Operations, Warfare Requirements and Programs (N6/N7) Memo Serial Number N6N7/5U916276, Requirement for Naval Open Architecture, 23 Dec 05.

l. Electronic Industries of America, Process for https://acc.dau.dag/

2-2

Engineering a System, EIA 632 Standard, 1998.

m. Institute of Electronic and Electrical Engineers, Application and Management of the Systems Engineering Process, IEEE Standard 1220, 1998.

n. International Council of Systems Engineering, Systems Engineering Handbook, Version 2A, 2004.

o. Marine Corps Systems Command, Develop and Demonstrate Process Handbook and Quick Tips, Version 2, July-2004.

p. National Aeronautical and Space Administration, NASA Systems Engineering Handbook, SP-610S, June-1995.

q. NAVAIRINST 4355.19D, Systems Engineering Technical Review Process, 17 Apr 2009.

r. NAVAIR, Systems Engineering Technical Review Handbook, of 17 Apr 2009 (enclosure (1) to NAVAIRINST 4355.1D)

s. NAVSEA Memo, Ship Design Review Guidelines, Ser 05D/096, of 28 Sep 04.

t. Systems Engineering Sage, A.P., Wiley series in systems engineering management. Wiley-Interscience, 1992 (ISBN 0471536393).

u. SECNAVINST 5000.2D, Implementation and Operation of the Defense Acquisition System and the Joint Capabilities Integration and Development System, 16 Oct 08.

v. Naval SYSCOM Risk Management Policy (MARCORSYSCOM Order

5000.3, SPAWARINST 3058.1, NAVFACINST 5000.15,

NAVSUPINST 5000.20, NAVSEAINST 5000.8, NAVAIRINST

5000.21B), 21 Jul 08.

w. VS-JI-22A, Virtual SYSCOM Engineering and Technical Authority Policy, 31 Jan 07.

x. SPAWARINST 5400.3, Systems Engineering Technical Review Process, 26-February-2007.

3-1

3.0 DEFINITIONS

Definitions of key acronyms and initialisms used in the Handbook are listed below.

Table 3-1

List of Acronyms and Initialisms

TERM DEFINITIONS

ACAT Acquisition Category AoA Analysis of Alternatives ASN(RD&A) Assistant Secretary of the Navy for Research, Development and Acquisition ASR Alternate Systems Review ASSIST Acquisition Streamlining and Standardization Information

System C4I Command, Control, Communications, Computing and

Intelligence CAE Component Acquisition Executive CARD Cost Analysis Requirements Description CBRN Chemical, Biological, Radiological, and Nuclear CDD Capability Development Document CDM Competency Domain Manager CDR Critical Design Review CDRL Contract Data Requirements List CHENG Chief Engineer CI Configuration Item CIO Chief Intelligence Officer CJCS Commander, Joint Chiefs of Staff CM Configuration Management CONOPS Concept of Operations COTS Commercial Off-the-Shelf CPD Capability Production Document CPI Critical Program Information CRISD Computer Resources Integrated Support Document CSCI Computer Software Configuration Item CSOM Computer Systems Operator’s Manual DOD Department of Defense

3-2

TERM DEFINITIONS

DON Department of the Navy DOT&E Director, Operational Test and Evaluation DOTMLPF Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel, and Facilities DSP Defense Standardization Program DT&E Developmental Test and Evaluation DT/OT Developmental Test/Operational Test E3/SS Electromagnetic Environmental Effects and Spectrum

Supportability EMD Engineering and Manufacturing Demonstration FCA Functional Configuration Audit FOC Full Operational Capability FoS Family of Systems FRP Full Rate Production FRR Flight Readiness Review HFE Human Factors Engineering HM&E Hull Mechanical and Engineering HSI Human Systems Integration HWCI Hardware Configuration Item IA Information Assurance IBR Integrated Baseline Review ICD Initial Capabilities Document IDD Interface Design Description II Interoperability and Integration IMS Integrated Master Schedule IOC Initial Operational Capability IOT&E Initial Operational Test and Evaluation IPT Integrated Product Team IRB Institutional Review Board IRR Integration Readiness Review IRS Interface Requirements Specification ISP Information Support Plan ISR In-Service Review IT Information Technology ITR Initial Technical Review IWS Integrated Warfare Systems JCIDS Joint Capabilities Integration Development System KPP Key Performance Parameter LCC Life-Cycle Costs LFT&E Live Fire Test and Evaluation LRFS Logistics Requirements Funding Summary

3-3

LRIP Low Rate Initial Production MC-SAMP Marine Corps Single Acquisition Management Plan MCSC Marine Corps Systems Command MDA Milestone Decision Authority MILCON Military Construction MIL-STD Military Standard (document) MS Milestone MSA Materiel Solution Analysis NATO North Atlantic Treaty Organization NAVFAC Naval Facilities Command NBC Nuclear, Biological, and Chemical NBC Nuclear, Biological, and Chemical NDI Non-developmental Item NOA Naval Open Architecture NR-KPP Net-Ready Key Performance Parameter NSS National Security Systems O&MN Operations and Maintenance, Navy (budget) OT&E Operational Test and Evaluation OTRR Operational Test Readiness Review OV Operational View PCA Physical Configuration Audit PCR Physical Configuration Review PDR Preliminary Design Review PEO Program Executive Office PLE Program Lead Engineer PM Program Manager PMO Program Management Office POM Program Objective Memorandum PR Program Review PRR Production Readiness Review R&M Reliability and Maintainability RAM Reliability, Availability, and Maintainability RFA Request for Action RFI Request for Information RFP Request for Proposal RM Risk Management SDD Software Design Description SATD System Architecture & Technology Demonstration SDM Ship Design Manager SDP Software Development Plan SDS System Design Specification

3-4

SECNAV Secretary of the Navy SEP Systems Engineering Plan SETR Systems Engineering Technical Review SFR System Functional Review SME Subject Matter Expert SRTD System Requirements & Technology Development SoS System of Systems SOW Statement of Work SPII Software Process Improvement Initiative SRR System Requirements Review SRS Software Requirements Specification SS Survivability and Susceptibility SSB Stakeholder Steering Board SSDD System/Subsystem Design Description SSR Software Specification Review STP Software Test Plan SUM Software Users Manual SV System View SVD Software Version Description SVR System Verification Review SYSCOM Systems Command TD Technology Development T&E Test and Evaluation TDP Technical Data Package TEMP Test and Evaluation Master Plan TFA Technical Feasibility Assessment TIM Technical Interchange Meeting TRAP Technical Review Action Plan TRB Technical Review Board TRR Test Readiness Review TRSR Technical Review Summary Report TV Technical View TWH Technical Warrant Holder USD Undersecretary of Defense VDD Version Description Document WBS Work Breakdown Structure

4-1

4.0 GENERAL INTRODUCTION

4.1 PURPOSE

This Naval SETR Handbook describes interim guidance for the expected procedure to implement the Naval Systems Engineering Policy (MARCORSYSCOM Order 5400.5, SPAWARINST 5000.1, NAVFACINST 5000.15, NAVSUPINST 5000.21, NAVSEAINST 5000.09, and NAVAIRINST 5000.24). The policy requires that all naval acquisition and modernization programs successfully complete a SETR process. The SETR process involves a series of structured assessments focused on the technical health and design maturity of a program under review, as planned in the Systems Engineering Plan (SEP), before passing key stages of engineering work.

The SETR process is an integral part of systems engineering and the life-cycle management of acquisition and modernization programs. SETRs are a primary method for assessing the technical health of a program at critical points in its development cycle.

Additionally, SETRs provide Program Managers (PMs) with independent assessments of program readiness to enter the next technical phase. Engineering rigor, interdisciplinary communications, and competency insight are applied to the maturing design in the assessment of requirements traceability, product metrics, and decision rationale. SETRs bring additional nonadvocate subject matter experts (SMEs) to the development process in an effort to ensure program success. The overarching objective of the SETR is a well-managed and disciplined technical effort leading to a successful technical and operational system evaluation that ensures the fielding of a suitable and effective system for the warfighter.

The SETRs are held to assist program office management teams in documenting technical requirements, synthesizing certifiable designs, assessing performance and system safety risk, and producing and deploying systems to achieve required capability.

SETRs address the assessment of the total system, which is composed of hardware, software, and human operators, maintainers, and decision-makers, as well as facilities and infrastructure, operating environment, and information.

4-2

The SETR requirement applies to all naval acquisition and modernizations programs, including programs of record in Acquisition Categories (ACATs) I through IV, as well as non-ACAT programs, System-of-Systems (SoS), and Family-of-Systems (FoS) programs. Some system programs within this scope include naval combatant and noncombatant ships; aircraft carriers; submarines;

aircraft; unmanned surface and subsurface vehicles; integrated warfare systems; weapon systems; hull, mechanical and electrical (HM&E) systems; ship facilities; infrastructure and arrangements;

land systems; space; enterprise information system; and command, control, communications, computing, and intelligence (C4I) systems.

This Handbook describes the planning, documentation and recording requirements, as well as roles and responsibilities for personnel involved in SETRs. Specific objectives, timing and triggering, and entry and closure criteria also are discussed for individual SETR assessments. Future SETR guidance for ship, air, C4I, land, and integrated warfare systems will be available in enclosures (1) through (5), respectively, upon first revision of the Handbook.

Current SETR guidance for these specific types of systems reverts to lead System Command (SYSCOM) policies, system-specific guidance, and program reviews (e.g., for ship system, NAVSEA memo ser 05D/096 of 28 Sep 04), “Ship Design Review Guidelines” and Stakeholder Steering Board program review process in enclosure (1).

SETRs assess the technical maturity of a system, along with its associated life-cycle attributes. For example, Appendix A to this Handbook, “Facilities and Infrastructure SETR Checklists” provides checklists directed at facilities and infrastructure. The checklists identify actions to help ensure that facility, real estate, environmental planning, and capital improvements budgeting (e.g., military construction (MILCON), operations and maintenance, Navy (O&MN)) and execution issues (e.g., construction, restoration and modernization) are addressed at appropriate SETRs. Program Management Offices (PMOs) must allow 5 to 7 years to acquire facilities, especially if the program requires land acquisition.

Some SETR activities, such as those described in Appendix A or those that engage the supply system or those needing cost analyses, may require functional and/or subject matter expert (SME) support. Reimbursement from programs for functional and/or SME support will be required. The cost of support for SETR

4-3 activities must be captured in the program SEP, as well as Customer Service Agreements, Task Assignment Agreements, or equivalent documents, to ensure that adequate resources are available to prepare products and to support personnel for the SETRs.

4.2 BACKGROUND

The Department of Defense (DOD) acquisition life cycle has five major phases: (1) Materiel Solution Analysis (MSA), (2) Technology Development (TD), (3) Engineering and Manufacturing Development,

(4) Production and Deployment, and (5) Operations and Support. A designated Milestone Decision Authority (MDA) conducts reviews at transitions between the first four phases, called Milestone A (MS A), Milestone B (MS B), and Milestone C (MS C), to ensure that the acquisition strategy minimizes the time and cost required to satisfy approved capability needs and maximizes affordability throughout the program life cycle.

The Secretary of the Navy (SECNAV) has augmented the acquisition timeline by establishing a two-pass/six-gate process leading up to MS B. This augmentation improves the governance of the acquisition process and helps to ensure that programs better understand the impact of requirements on total life-cycle costs before locking in development decisions.

The acquisition life cycle begins through the Joint Capabilities Integration and Development System (JCIDS) process, which involves an Analysis of Alternatives (AoA) for varying system concepts and strategies, and which leads to an Initial Capabilities Document (ICD) at MS A stating user/system requirements. Systems engineering plays a central role in identifying technological solutions to capability gaps and by providing robust analytical processes to evaluate operating, maintenance, sustainment, and acquisition approaches.

In parallel with the establishment of the ICD, the Undersecretary of Defense (USD) requires that a SEP be developed for MS A and used by the PM throughout the life of the program. The SEP describes the overall technical approach, systems engineering processes, resources, key technical tasks, activities and events, along with metrics and success criteria. Since the SEP guides all technical aspects of a program, it is established early during program definition (i.e., during the Materiel Solution Analysis

4-4 acquisition phase) and it is updated at each MDA MS. The SEP is a living document, providing a tailored roadmap to support program management by defining comprehensive systems engineering activities that translate systems capability needs into an effective, suitable product that is sustainable at an affordable cost.

The Assistant Secretary of the Navy for Research, Development, and Acquisition (ASN (RD&A)) has elaborated on USD acquisition policy by stressing the need for collaboration among programmatic authorities and systems engineering technical authorities. The Department of the Navy (DON) has responded by requiring the development of a System Design Specification (SDS) early in the program definition phase. The SDS accomplishes the following:

• Derives platform-specific Mission Performance requirements and attributes from higher-level capability documents.

• Identifies naval and industry design criteria and standards used during system development.

• Details expected producibility, operability, and maintainability of the system.

The SDS supports decision makers by defining design requirements and capabilities to establish better informed schedule, costs, and risks early in the acquisition process, thereby reducing programmatic and technical risks associated with technology development and system acquisition.

Consistent with the established government policies, guidance, and practices mentioned above, the Naval SYSCOM Systems Engineering Policy requires that SETRs be conducted for all naval acquisition and modernization programs.

A SETR is a structured, technical assessment process that evaluates the technical health and maturity of an evolving system design at key points in the development life cycle and the potential for successful development in the future.

The SETR process provides visibility into the work efforts associated with the design, development, and production of complex naval systems to assure timely and effective attention to technical details. As a system progresses through design, development, and production, SETR assessments are timed to coincide with major transitions in the program activities. The

4-5

SETR provides the events used by Naval Technical Authorities to verify that technical maturity of the system, subsystem, or configuration item under review is sufficient to commitment resources for the next stage of development work.

As a system matures in development, the focus of SETR assessments adjusts to the depth of the design maturity. Early in the system evolution, the primary focus of SETR assessments is on defining the technical requirements that drive design and development activities, that is, to ensure technical feasibility and to confirm that top-level system concepts reflect user requirements.

Additionally, the early stages review the success in establishing the programmatic and technical management processes that are required for effective systems engineering of the weapon system.

As the system continues to mature, subsequent SETR assessments focus on the design at subsystem levels and below. At even later stages, the SETR assessments focus on verifying that physical solutions align with design requirements and can be manufactured within cost, schedule, and risk targets.

Although the comments on design evolution above stem from the usual progression of engineering for a unified system, the point that SETR assessment evolves over the development life cycle holds true for much more complex and diversified SoS and FoS programs.

The schedule and structure of SETR assessments must coincide with strategic, event-driven transitions in the system development activities. The scheduling and tailoring of SETR assessments is addressed further in this Handbook.

4.3 OBJECTIVES

The SETR process is used to assess and verify system design progress and maturity at key event-driven development stages in the acquisition schedule. Documentation on the attained maturity of the system is compared with pre-planned entry and closure criteria to facilitate the scheduling and management of scope for the SETR assessments. Objectives for the SETR process are as follows:

a. Ensure that trade studies used to define concepts and assess risks have been considered and incorporated into work plans.

4-6

b. Assess system requirements and allocations to ensure that requirements are unambiguous, consistent, complete, feasible, verifiable, and traceable to top-level requirements.

c. Assess design maturity based on technical goals, systems engineering accomplishments, and empirical data supporting progress to date.

d. Confirm effects of performance and safety risks on cost, schedule, and performance; and ensure that risk reduction measures, rationale, and assumptions have been approved.

e. Ensure that interfaces among required hardware, human, and software items have been optimized for efficiency, effectiveness, and safety/training risk reduction.

f. Ensure that performance objectives, functional design, cost, schedule, technical performance measurements, and sustainment and infrastructure support are tracked, on schedule, and achievable within existing constraints.

g. Confirm that continued development is warranted. When it is not warranted, confirm that corrective action is executed before proceeding with the development plan.

h. Ensure that resources (e.g., people, funding, support assets, facilities) required for continued development, testing, production, and operation are identified and resourced.

i. Identify and accept residual safety risks prior to fleet transition or fielding.

j. Ensure programmatic and technical processes are established to support successful design and product development.

Review tailoring of underlying programmatic and technical process to scope complexity of system development.

5-1

5.0 OVERVIEW

5.1 SETR PROCESS

The SETR process is a disciplined, yet flexible, approach to review and verify the technical evolution of a system. While the intended focus of the SETR process is a total system, the approach is inclusive of subsystem and component development work.

Applied to a particular program, the SETR process consists of a series of technical assessments conducted by a Technical Review Board (TRB). The TRB established for each technical review has a defined charter and scope of work. The TRB interfaces regularly with the program team to collect, review, and discuss materials documenting program progress and accomplishments relevant to the ongoing technical assessment. As the assessment nears completion, the TRB and the program meet to conduct a formal assessment documenting the completion status of the review and making recommendations for future work directions.

The outcomes of the formal meeting are delivered by the TRB to program management.

Numerous types of assessments are recognized across the naval enterprise; some are focused on program management, others on program cost and schedule, and still others on technical issues.

The SETR assessments fall into the technical category. Table 5- 1 provides a brief description of the standard SETR assessments.

Not all SETR assessments in Table 5-1 are required for an acquisition program. As discussed later, a PM is expected to tailor the SETR to the complexity and risk level of the program.

Subsequent sections in the Handbook provide interim guidance on tailoring SETRs as well as on a recommended minimum number of SETR assessments.

Table 5-1

List of Systems Engineering Technical Review Assessments

ASSESSMENT PURPOSE TIMING

5-2

Initial Technical Review

Supports technical basis for initial cost estimates and POM budget submissions.

Materiel Solution Analysis (pre-MDA

MS A)

Alternative System Review

Reviews results of Materiel Solution Analysis phase and assesses technology development plan and preferred system concept.

Materiel Solution Analysis (pre-MDA

MS A)

System Requirements Review (Note 1)

Assesses technical readiness to enter Engineering & Manufacturing Development phase.

Technology Development (pre-MDA MS B)

Integrated Baseline Review

Assesses risk areas in contract. Produces Performance Measurement Baseline to ensure technical scope of work is realistically and accurately scheduled, has proper resources, utilizes correct techniques, and employs appropriate management processes.

Technology Development (pre-MDA MS B)

Used periodically throughout development cycle when earned value management is required.

System Functional Review

Assesses System Functional Baseline and readiness to begin functional allocation.

Technology Development (pre -

MDA MS B)

Preliminary Design Review (Note 2)

Assesses System Allocated Baseline and readiness to begin detailed design.

Technology Development (pre/post-MDA MS B)

Software Specification Review

Assesses completeness of software specification.

Technology Development (pre-

MDA MS B)

Critical Design Review

Assesses System Product Baseline and supports Design Readiness Review.

Engineering & Manufacturing Development phase (pre-MDA MS C)

5-3

Test Readiness Review

Assesses system readiness to begin Developmental Test and Evaluation

(DT&E).

Engineering & Manufacturing Development phase (pre-MDA MS C)

Flight Readiness Review

Assesses system readiness to initiate and conduct flight tests and flight operations.

Engineering & Manufacturing Development phase (pre-MDA MS C)

Integrated Readiness Review

Assesses readiness of software systems.

Engineering & Manufacturing Development phase (pre-MDA MS C)

Operational Test Readiness Review

Assesses system readiness to proceed into Operational Test and Evaluation (OT&E).

Engineering & Manufacturing Development phase (pre-MDA MS C)

System Verification Review

Assesses system compliance with functional baseline.

Production and Deployment (pre-MDA MS C)

Production Readiness Review

Assesses system readiness to enter production.

Production and Deployment (post-MDA MS C)

Physical Configuration Audit

Assesses the as-delivered system for compliance with the product baseline and supports full-rate production decision.

Production and Deployment (post-MDA MS C during initial operational capability (IOC))

In-Service Review

Assesses the in-service technical health of a fielded system from a risk, readiness, and resources perspective.

Operations and Support (post-MDA MS C during FOC)

Notes:

1. A best practice is for SRR to be accomplished in two parts.

SRR-I is to ensure the government has established performance requirements and non-tailorable design requirements that are directly traceable to the CDD.

SRR-II is a technical assessment of the developing system

5-4 specification under review to ensure a reasonable expectation of the final system being judged operationally effective and suitable.

2. PDR – A best practice is for PDR to be accomplished in two parts, an initial PDR and a Closure PDR. The PM should plan for a PDR before Milestone B, consistent with associated prototyping requirements. If a PDR has not been conducted prior to Milestone B, the PMs shall plan to accomplish a minimum set of PDR requirements to support SDS development. A minimum MS B preparatory PDR represents a physically architected system based on full engagement of subsystem suppliers and knowledge gained through prototyping and identified in the technology development strategy. Following the Closure PDR, the PM shall send a PDR closure report to the MDA.

The SETR assessments employed to evaluate a particular program should be selected as appropriate for the complexity, risk, and constraints of the program, consistent with sound engineering practice. The process of selecting appropriate SETR assessments is referred to as tailoring, and it is discussed further in Section 5.4 below.

Each SETR assessment involves an appropriate period of time for the TRB to engage continuously with the program under review for the purpose of developing a thorough and technically astute understanding of the program work efforts, accomplishments, and issues. Moreover, each SETR assessment period culminates in a formal meeting where the TRB provides the PM with a report of findings and recommendations concerning the readiness of the program to enter the next phase of development work.

The formal review meeting for each SETR assessment may be preceded by a series of Technical Interchange Meetings (TIMs;

see Figure 5-1) between the TRB and program Integrated Product Team (IPT). During the TIMs, the program and its developers, acting as an IPT, and the TRB identify and discuss the technical status of the program. Technical issues, accomplishments, problems and their mitigation solutions, and documentation on the technical maturity of the system are examples of topics for the TIMs. TIMs provide one means for the TRB members to become familiar with the technical details associated with the program under review, its products and documentation, and the quality of

5-5 work performed prior to the formal review meeting.

The TIMs serve as a forum for problem solving and information sharing, whereas formal technical review meetings are the forum for establishing that the problem solving is aligned with the approved program SEP.

Plan Preview Review Resolve Follow-Up

Before During After

Familiarize

� Identify Participants

� Set Agenda � Define Roles � Review

Criteria � Review Areas of Interest

� Overview Issues

� Assign Roles

� Develop Tasks

� Assemble Artifacts

� Assess Criteria

� Define Scope

� Examine Data

� Analyze � Track

Analyses

� Report Findings

� Examine Analyses

� Discuss � Record

Findings

� Develop Strategy

� Assign Actions

� Track Actions & Issues

� Document Results

� Inform Stakeholders

Figure 5-1. Timeline for Activities in Technical Interchange Meetings.

5.2 ESSENTIAL SETR ASSESSMENTS

Table 5-2 lists the essential SETR assessments for new system acquisitions and modernizations of existing systems. These SETRs are discussed further in Reference (a). The SETR assessments in Table 5-2 have been identified through experience as providing essential coverage of technical issues in naval programs. Table 5-2 does not convey a standard set of SETR assessments for all programs; rather, it highlights the most important SETR assessments. It is expected that PMs will tailor the SETR to include these and other SETR assessments as needed to address the scope, complexity, and risk of their programs.

5-6

Table 5-2

Recommended Systems Engineering Technical Reviews for Acquisition and Modernization Programs

SETR ASSESSMENT

Initial Technical Review Alternative Systems Review System Requirements Review System Functional Review Preliminary Design Review Critical Design Review Test Readiness Review

System Verification Review Production Readiness Review Physical Configuration Audit

Statements of Work (SOWs) should incorporate SETR requirements, as appropriate. A good source of contracting guidance is DOD, Guide to Integrating Systems Engineering into DOD Acquisition Contracts. The SEP shall contain the anticipated reviews and should be submitted with the RFP.

5.3 TIMING AND TRIGGERING

The SETR process is event-driven, rather than schedule-driven.

Entry and closure criteria, as well as timing and triggering guidance, determine the events driving the SETR assessments.

These criteria are established in the SEP and evaluated during the formal review meeting for each SETR assessment. Sections

5.4.1 and 5.4.5 discuss entry and closure criteria in detail.

The schedule of technical assessments is important. If a SETR assessment is conducted too early, items for review cannot be defined adequately. Conversely, SETR assessments conducted too late may result in erroneous program commitments, which are difficult and costly to correct. Figures 5-2 and 5-3 illustrate the timing for SETRs for two different program acquisition strategies.

MARCORSYSCOM Order SPAWARINST 5000.1 NAVFACINST 5000.15 5400.5

5-7

Figure 5-2. Example of SETR Assessments for Program Initiation at Milestone A. Position of Technical Reviews Is Notional.

MARCORSYSCOM Order SPAWARINST 5000.1 NAVFACINST 5000.15 5400.5

5-8

Figure 5-3. Example of SETR Assessments for Program Initiation at Milestone B. Position of Technical Reviews Is Notional.

5-9

Timing of SETR assessments is influenced by the availability of materials supporting the system under review, such as documentation, hardware and software, and completion of acceptance qualification tests. Sufficient time for the Naval Technical Authorities to review these materials also is an important determining factor in the timing of SETR assessments. For example, a PDR SETR would be scheduled after the Hardware Development Specification or Software Design Description and Software Test Plan are available because the focus of the PDR SETR is to assess a developer's approach to meeting the requirements of these documents.

Triggering of SETRs is dependent on several factors, as listed below:

a. Documentation availability.

b. Hardware/software availability.

c. Completion of acceptance qualification tests.

d. Sufficient time for review of materials by the TRB.

For example, a trigger event for the Alternative System Review (ASR) is the availability of a preferred system concept.

5.4 SETR INFORMATION

Each technical review requires a considerable amount of information to be managed and processed by the TRB and the program. This information is the objective evidence demonstrating that the product/program is in compliance with the technical criteria set forth in the review. Under the structured framework of entry criteria for the review, the examination of this evidence is used to determine the timing and readiness of the review.

Additional guidance on the type of information and how it is structured is provided below.

5.4.1 SETR INFORMATION HEIRARCHY

To assist in the organization and communication of that information, a structure for the assessment information is provided as guidance. Tiers 1-3 summarize the SETR information for senior reviewers. These tiers are common to all DON programs

5-10 to the extent reasonable. Tiers 4-7 contain and implement all detailed SETR information, including that summarized in Tiers 1-3 and the specific technical products are that are unique to each program. Tiers 4-7 are developed by the individual SYSCOMs to meet individual program needs. A synopsis of the tiers of information is as follows:

a. Tier 1 – Entry Criteria act as primary benchmarks providing senior leadership a summary of program readiness to enter the SETR process. These criteria are standardized across the SETR process. (See Figure 5-2).

b. Tier 2 – SETR Specific Entry Criteria are a summary level categorization of entry criteria which recognize the shifting balance of systems engineering elements from requirements to product over the maturity of design and progressing reviews.

Areas of interest also include elements of the systems engineering and program management processes and their impact on development with respect to total ship system cost, schedule, performance, and risk. Tier 2 summarizes the SETR information related to the areas of interest for senior reviewers. (See Section 5.4.2)

c. Tier 3 – Products is broadly defined as the collection of work accomplished by the Government program office to accomplish their oversight function aligned to tier 2. Tier 3 products also include the documentation that the senior reviewers will use to evaluate entry and closure of SETR events.

d. Tier 4 - Engineering Functional Areas are the common technical areas that represent the product description of the system in acquisition. This should correspond to the program Work Breakdown Structure. Areas of interest also include elements of the systems engineering and program management processes and their impact on development with respect to total ship system cost, schedule, performance, and risk. Tier 2 summarizes the SETR information related to the areas of interest for senior reviewers.

(See Section 5.4.2)

e. Tier 5 – Procedures or Standard Work Packages align to and implement Tier 4 requirements.

f. Tier 6 - Questions define the issues to be addressed during specific reviews and are aligned to Tier 5.

5-11

g. Tier 7 – Products and Artifacts (or links to them) include both the Tier 3 products and additional detailed information to answer the Tier 6 questions.

Additional information on Tiers 4-7 which are unique to each Naval SYSCOM and Program are contained in the enclosures to this document.

5.4.2 ENGINEERING FUNCTIONAL AREAS

SETRs evaluate the breadth of engineering functional areas.

Although naval systems are complex and diverse, several engineering functions are common across Naval SYSCOMs, and thus represent broad systems engineering disciplines impacting the development of all naval systems. Common engineering functional areas should be built using the Tier 4-7 architecture applying the appropriate balance between DON-wide commonality and program specific tailoring, and then summarized in Tier 2. An example of common SETR information for one engineering functional area, Naval Facilities, is contained in Appendix A. It is the responsibility of the program to identify all applicable engineering functional areas and to achieve compliance with the standards. Common engineering functional areas could include those in paragraphs

5.4.2.1 through 5.4.2.10

5.4.2.1 Systems Engineering and Program Management Tasks.

This includes the engineering and programmatic management processes which create the engineering development environment.

Examples of engineering process are those which manage requirements traceability, evolution of mission CONOPs, tracking technical metrics and transition to manufacturing. Examples of programmatic processes are configuration management, integrated master scheduling, cost estimating, assigning earned value, logistics, contracting and risk management.

5.4.2.2 Electromagnetic Environmental Effects and Spectrum

Supportability

As electronic/electrical systems become more complex, Electromagnetic Environmental Effects and Spectrum Supportability (E3/SS) and certification requirements become critical factors in the ability to employ military systems and platforms effectively.

5-12

Failure to consider E3/SS early could result in program delays, additional cost or less than full operational capability (FOC).

E3/SS on equipment, systems, or platforms are critical elements that must be considered throughout the acquisition process to ensure the successful operational effectiveness of these military assets in support of the warfighter.

In the conduct of SETRs, emphasis is placed on assessing the following E3/SS attributes:

a. E3/SS inputs to requirements documents, including those addressing nuclear survivability.

b. E3/SS requirements addressed in SOW, CDRLs, specifications, SEP, and Test and Evaluation Master Plan (TEMP), as needed.

c. E3/SS analyses and predictions programmed, including those to assess the addition of new antennas or apertures and the integration of equipment wireless sensors.

d. Cost estimates for E3/SS tasks and related activities, such as system/platform antenna design.E3/SS approval for next Acquisition Milestone from DOD or Component Chief Information Officer (CIO), Component Acquisition Executive (CAE), or MDA, based on requirements contained in Section 6.8 of MIL-HDBK-237D.

E3/SS actions and focus areas for each step in the SETR process are defined in Section 6.9 of MIL-HDBK-237D. SETRs involving E3/SS peculiar to both SPAWAR and NAVAIR must be coordinated with the NAVSEA E3/SS Technical Warrant Holder.

5.4.2.3 Facilities and Infrastructure

Facilities affect many supportability elements, such as training (simulators and ranges), maintenance (hangars, dry-docks), supply (warehouses, magazines), environmental (National Environmental Policy Act), and support equipment (bridge cranes, material handling equipment). Effective facilities planning increases system reliability and reduces the logistics footprint by integrating design with acquisition program supportability. PMs must allow 5 to 7 years to acquire facilities, especially if the program requires land acquisition. Consequently, PMs must start planning early and evaluate existing facilities and infrastructure

5-13 along with programming for new facilities to ensure that the right facilities are provided at the right locations at the right time.

Appendix A provides the acquisition community with a set of checklists for conducting SETR events directed at successfully planning for facilities and infrastructure. Each checklist is designed to alert the acquisition community to the complexities of providing timely, compatible, and suitable supporting facilities and infrastructure. The checklists identify a progressive set of recommended actions, based upon the maturity of the system, to help ensure that facility, real estate, and environmental planning, capital improvements budgeting (e.g., MILCON, O&MN)and execution (construction, restoration and modernization) issues and requirements are understood and addressed at the appropriate SETRs.

5.4.2.4 Human Systems Integration

Per DOD Directive 5000.1, PMs shall apply Human Systems Integration (HSI) to optimize total system performance and minimize total ownership cost. To do this, PMs should work with the manpower, personnel, training, safety and occupational health, habitability, survivability, and human factors engineering (HFE) communities to translate and integrate the HSI thresholds and objectives contained in the capabilities documents into quantifiable and measurable system requirements (see DOD Instruction 5000.2). The PM should then include these requirements in specifications, the TEMP, and other program documentation, as appropriate, and use them to address HSI in the SOW. The PM should identify any HSI-related schedule or cost issues that could adversely impact program execution; the system's support strategy should identify responsibilities, describe the technical and management approach for meeting HSI requirements, and summarize major elements of the associated training system.

In the conduct of SETRs, emphasis is placed on assessing the following HSI attributes:

a. HSI requirements based on a top-down requirements analysis, including developing an optimal crew concept and allocating system functions to automation and human performance.

b. Position descriptions based on functions for crew position, including duties, jobs, responsibilities, and levels of

5-14 authority.

c. HSI inputs to acquisition documents and specifications.

d. HSI risk assessment and mitigation plans, identifying factors that impact manpower, human effectiveness, workload, survivability, and safety.

5.4.2.5 Information Protection

PMs must protect all information systems, as well as the information processed by those systems, in their programs. The presence of hostile agents, whose intent is to disrupt proper operation of naval systems, is a critical challenge to system development. The DOD has established and mandated procedures to protect systems and the information they handle.

The term “systems assurance” refers to activities focused on ensuring…