Attachment_J-02_TRM-BFTT_Tailored_Source_Selection_Sensitive.docx

Program Executive Officer Integrated Warfare Systems

(PEO IWS)

Naval Sea Systems Command Research & System Engineering Warfare Systems Engineering and Human Systems Integration Directorate (NAVSEA 05H)

TECHNICAL REVIEW MANUAL (TRM)

Tailored by IWS 1IT

Version 2.0

18 December 2009

Tailored 13 February 2013

Department of the Navy Naval Sea Systems Command / Program Executive Office, Integrated Warfare Systems 1333 Isaac Hull Avenue SE Washington Navy Yard, DC 20376

Source Selection Sensitive

DESCRIPTION OF PROGRAM

In February 1992, the Battle Force Tactical Training (BFTT) Operational Requirements Document (ORD) was signed and approved as a Program Of Record (POR) to reflect the current Tactical Training Strategy (TTS) and Fleet training requirements.

The BFTT Program is an Acquisition Category (ACAT) IV-M Milestone C acquisition program that provides the hardware and software required to integrate existing onboard trainers and stimulators into a baseline multi-warfare/multi-ship combat system team training capability.

BFTT, is a project within the BFTT POR accordingly significant tailoring of the NAVSEA PEO IWS TRM has been undertaken.

Blue – Indicates tailoring of the TRM for BFTT specifically.

Red – Indicates tailoring out areas of the TRM for BFTT specifically

This page was left blank intentionally.

A

EXECUTIVE SUMMARY

The Program Executive Office for Integrated Warfare Systems (PEO IWS) and the, Naval Sea Systems Command, Research & Systems Engineering Warfare Systems Engineering and Human Systems Integration Directorate (NAVSEA 05H) require System Engineering Technical Reviews (SETRs) to be conducted of all systems to demonstrate that required systems engineering (SE) and associated efforts necessary for program advancement to the next development stage have been completed. SETRs are an integral part of the SE process and consistent with existing and emerging engineering standards. SETRs should be event-driven, and conducted when the system development has demonstrated sufficient maturity for review of progress in accordance with entrance criteria established for each review and acceptable level of risk.

While the Systems Engineering Operations Manual (SEOM) provides guidance on the SE process, this Technical Review Manual (TRM) describes each SETR in the overall technical review process and provides PEO IWS amplifying guidance on the preparation and conduct of PEO IWS SETRs. Objectives are identified for each SETR. References (a), (b), and (c) require programs to develop and deliver a Systems Engineering Plan (SEP) that describes all technical aspects of an acquisition program, and describe the overall technical approach, SE processes, resources, key technical task, activities and events along with the metrics and success criteria. Program Managers (PMs) are responsible to insure SETRs are planned and held for their program. The PM may tailor a program's SEP to their particular program circumstances. The SEP must include a description of any program-specific tailoring of SETRs and explain how the tailored SETR activities accomplish the objectives of the overall SETR process. Each SEP must be approved by the PEO IWS Technical Director (TD). In addition, SEPs for Acquisition Category (ACAT) I and II and special interest programs must be approved by the Assistant Secretary of the Navy (Research, Development, and Acquisition) Chief System Engineer (ASN RD&A CHSENG) prior to Milestone Decision Authority's (MDA) approval at each acquisition milestone and whenever a significant program change occurs.

Office of the Secretary of Defense (OSD) requirements for review and approval of SE products and processes are being implemented via the Naval Sea Systems Command's (NAVSEA's) Technical Authority (TA) and Certification process. Information Assurance (IA), always a consideration for the entirety of any development, is now made an explicit topic in development processes and reviews. The emerging importance of Open Architecture (OA), Intellectual Property (IP), Human Systems Integration (HSI), and Commercial Off-the-Shelf (COTS) issues are reflected in additional agenda items in all reviews. These engineering areas require assessment in emerging systems.

The point of contact for this manual is the PEO IWS Deputy Technical Director.

This page was left blank intentionally.

ii

CONTENTS

Section Page

1.0

1.1

1.2

1.3

2.0

3.0

3.1

3.2

3.3

3.4

INTRODUCTION ............................................................................................... 1-1

SCOPE ................................................................................................................. 1-1

DOCUMENT ORGANIZATION ....................................................................... 1-3

DOCUMENT UPDATE ...................................................................................... 1-4

REFERENCES .................................................................................................... 2-1

SYSTEMS ENGINEERING TECHNICAL REVIEW (SETR).......................... 3-1

GENERAL BACKGROUND.............................................................................. 3-1

OBJECTIVES ...................................................................................................... 3-2

RECOMMENDED SETRS AND LEVELS........................................................ 3-4

ROLES AND RESPONSIBILITIES ................................................................... 3-5

3.4.1 PROGRAM / PROJECT MANAGER (PM) ....................................................... 3-5

3.4.2 SYSCOMS ........................................................................................................... 3-7

3.4.3 INDEPENDENT TECHNICAL AUTHORITY (ITA) ....................................... 3-7

3.4.4 OTHER CO-CHAIRS.......................................................................................... 3-8

3.4.5 SETR PANEL ...................................................................................................... 3-9

3.4.6 EXTENDED INTEGRATED PRODUCT TEAM (IPT) .................................... 3-9

3.4.7 TECHNICAL AUTHORITY (TA) PARTICIPANTS ...................................... 3-10

3.4.8 TECHNICAL REVIEW BOARD (TRB) .......................................................... 3-11

3.4.9 TECHNICAL INTERCHANGE MEETING [TIM] ......................................... 3-11

3.5

PLANNING FOR SYSTEMS ENGINEERING TECHNICAL REVIEWS..... 3-11

3.5.1 TECHNICAL REVIEW ACTION PLAN (TRAP)/SEP/MODIFIED SEP ...... 3-12

3.5.2 SYSTEMS ENGINEERING TECHNICAL REVIEW (SETR) AGENDA ...... 3-13

3.6

3.7

SYSTEMS ENGINEERING TECHNICAL REVIEW PREPARATION......... 3-13

CONDUCTING FORMAL SETR MEETINGS ............................................... 3-14

3.7.1 SETR ASSESSMENT MEETING .................................................................... 3-15

3.7.2 SETR CLOSURE............................................................................................... 3-15

3.7.3 TECHNICAL REVIEW SUMMARY REPORT (TRSR)................................. 3-16

APPENDIXES

Appendix Page

A Acronyms ............................................................................................................ A-1

B Document and Systems Engineering Technical Review Definitions ..................B-1

C Specific Systems Engineering Technical Reviews ..............................................C-1

D Technical Review Documentation Matrix .......................................................... D-1

E Technical Review Manual Comment Form…………………………………….E-1

ILLUSTRATIONS

Figure Page

FIGURE 1. NAVSEAINST 5000.9 SYSTEMS ENGINEERING TECHNICAL REVIEWS FOR MILESTONE A PROGRAMS .................................................................................................... 1-3

FIGURE 2. NAVSEAINST 5000.9 SYSTEMS ENGINEERING TECHNICAL REVIEWS FOR MILESTONE B PROGRAMS .................................................................................................... 1-4

FIGURE 3. SYSTEMS ENGINEERING TECHNICAL REVIEWS (SETRs) CO-CHAIRS…3-5

FIGURE 4. PLANNING TIMELINE FOR SETR ASSESSMENTS ...................................... 3-12

FIGURE 5. TYPICAL SETR PROCESS ................................................................................. 3-15

1.0 INTRODUCTION

1.1 Scope

ASN(RD&A) is the DON Component Acquisition Executive (CAE) responsible for DON acquisition including establishing policy and procedures and being responsible for all research, development, and acquisition per SECNAVINST 5430.7 series and SECNAVINST 5400.15 series.

PEOs, SYSCOM Commanders, and DRPMs are accountable for the specific responsibilities listed SECNAVINST 5400.15 series, including administration of assigned acquisition programs, and reporting directly to the CAE for such programs. PEOs, SYSCOM Commanders, DRPMs, and PMs have authority, responsibility, and accountability for life-cycle management of all acquisition programs within their cognizance. Title 10 U.S.C. § 1737 : US Code - Section 1737: Definitions and general provisions defines the term "program manager" with respect to a defense acquisition program as the member of the Acquisition Corps responsible for managing a program.

PEOs and DRPMs shall act for and exercise the programmatic authority of the Naval Acquisition Enterprise to directly supervise the management of assigned programs, maintaining oversight of cost, schedule, and performance, and make timely and forthright reports directly to the ASN(RD&A) for all matters pertaining to acquisition and will work with the SYSCOMs to ensure that technical authority processes are an integral part of their program execution and that acquisition issues pertaining to supportability of their systems are coordinated and addressed throughout the entire life cycle per SECNAVINST 5400.15 series. Programmatic authority for acquisition is exercised by Program Managers and is the management of all aspects of assigned programs from concept to disposal, including oversight of cost, schedule, and performance; and direction of life cycle management per SECNAVINST 5400.15 series.

Technical Authority is the authority, responsibility, and accountability to establish, monitor and approve technical standards, tools, and processes in conformance with applicable Department of Defense (DoD) and DON policy, requirements, architectures, and standards per SECNAVINST 5400.15 series to ensure the safety, reliability and performance of Navy products are practical, complete and meet programmatic needs.

ASN(RDA) directed the development of a common System Engineering Technical Review (SETR) process by the SYSCOMs and the institutionalization of this common SETR process which is to be led by a senior and independent person from the acquisition program ( see policy memorandum Systems Engineering Technical Review Process for Naval Acquisition Programs dated 13 Jun 08). NAVSEAINST 5000.9, Virtual SYSCOM Naval SYSCOM Systems Engineering Policy, 2009 and the SETR Handbook provides for this common SETR process.

Emphasizing review preparations and procedures based on the Naval Systems Engineering (SE) Policy and Naval Systems Engineering Technical Review (SETR) Handbook, 1-4 this TRM identifies technical requirements, planning responsibilities, recording requirements, and roles for SETRs for programs within PEO IWS. This manual describes the overall PEO IWS/SEA 05H SETR process as well as the objectives and activities (including entrance and exit criteria) for each SETR. In so doing, this manual reflects policy that governs the SETR process and provides guidance for the conduct of SETRs for development and verification of new systems or system baseline upgrades.

In accordance with the Policy for SE in Department of Defense (DoD), each program is required to submit a System Engineering Plan (SEP) for Milestone Decision Authority (MDA) approval at each milestone review using the DoD Systems Engineering Plan Preparation Guide including addendums. The SEP describes all technical aspects of an acquisition program and describes the overall technical approach, SE processes, resources, key technical task, activities and events along with the metrics and success criteria. SETRs to be conducted by each program should be identified in the program’s SEP. The SEP must include a description of any program- specific tailoring of the SETR and explain how the tailored SETR activities accomplish the objectives of the overall SETR process. Non-acquisition programs within PEO IWS shall prepare a modified SEP that includes as a minimum the Technical Review Planning portion of the DoD Systems Engineering Plan Preparation Guide. The modified SEP shall have the concurrence of the NAVSEA CHENG and be approved by the IWS Technical Director.

SETRs are independent reviews of programs for the benefit of the program manager. Technical Interchange Meetings (TIMs) and program Integrated Product Team (IPT) meetings are critical elements of the SETR process. The primary focus of each SETR assessment is on technical issues; however, SETRs should never exclude cost and schedule considerations when necessary to exercise sound judgment on technical risks and proposed solutions. In all cases, the term “system” implies the total system: products (hardware and software), processes (facilities, equipment, material, and procedures), and people (users, acquirers, developers, operators, and maintainers).

SETRs are held to assist the Program Manager (PM) and his management team in documenting technical requirements, synthesizing certifiable designs, assessing performance and risk, ensuring programs are on track to achieve deployment with required capabilities. SETRs are event-driven, and implement a technical assessment process that evaluates the maturing program design/progress over the life of the program.

PEO IWS/SEA 05H SETRs will be conducted in accordance with Department of Defense (DoD) and Department of the Navy (DON) policies and instructions. Emphasis is placed on review preparations and procedures based on the Naval SE Policy (ref (i)) and Naval SETR Handbook (ref (h)). Section 2.0 lists key references used in development of this manual. The DoD acquisition life cycle, including SECNAV 2-pass/6-gate reviews, and SETR milestones are shown in Figures (1) and (2) for a major program entering at either Milestone A or Milestone B. These figures show where the various SETRs fit within the acquisition life cycle requirements of higher authority.

FIGURE 1. NAVSEAINST 5000.9 SYSTEMS ENGINEERING TECHNICAL REVIEWS FOR MILESTONE A PROGRAMS

Comments on Figure (1) and Figure (2) on the following page:

1) The Acquisition Gate Review line showing Secretary of the Navy’s two pass/six gate review process in accordance with SECNAVINST 5000.2D dated 16 October 2008, ref (i).

2) The “Technical Review” line shows a program’s SETRs based on program maturity across the life of a program.

1.2 Document Organization

Section 2 lists references upon which the SETR process is based.

Section 3 describes the purpose and objectives of SETRs that may apply to a program and general guidelines to conduct a SETR (scheduling, preparations, documentation, request for action, etc.).

Appendix A lists acronyms.

Appendix B lists document definitions.

FIGURE 2. NAVSEAINST 5000.9 SYSTEMS ENGINEERING TECHNICAL REVIEWS FOR MILESTONE B PROGRAMS

Appendix C lists the SETRs, different types of projects, common topics of SETRs, and contains specific guidelines with a detailed description of each type of SETR, entrance and exit criteria, and recommended guidance for scheduling and content.

Appendix D provides a summary of SETR documentation providing a tabular listing of the documentation and artifacts associated with each review, and recommended architectural views for consideration at each review.

Appendix E provides for a Comment Review Matrix

1.3 Document Update

This document will be updated annually based on changes in relevant acquisition policies and instructions and lessons learned while executing the SETR processes in this manual. Also, the next annual update is planned to specifically address System of Systems, Family of Systems, interoperability, open architecture, Objective Architecture and the Appendices. Personnel are encouraged to provide comments as they are executing the processes herein during the year (vice waiting until the formal review and update cycle) and submitting these comments using the Comment Review Matrix.

2.0 REFERENCES

The SETR process and associated procedures of the PEO IWS and SEA 05H Technical Review Manual (TRM) are set forth to comply with the Naval SE Policy (ref (a)) and the Naval SETR Handbook (ref (b)). This TRM does not replace active acquisition, SE, and operational policies, instructions, notices, and directives, but is a tool meant to provide a framework from which to assist Program Managers (PMs), Systems Integration Program Managers (SIPMs), Program Lead Engineers and TDs, Program Offices, and others in preparing for and conducting SETRs.

Below are a few of the key references this manual is based upon (the list is not all inclusive):

a. DOD Directive 5000.01, The Defense Acquisition System, 12 May 2003

b. DOD Instruction 5000.02, Operation of the Defense Acquisition System, 8 December

c. Defense Acquisition Guidebook (DAG), Interim Version [2009]

d. SECNAVINST 5000.2D, Implementation and Operation of the Defense Acquisition System and the Joint Capabilities Integration and Development System, 16 October

e. SECNAVINST 5400.15 series Department of the Navy (DoN) Research and Development, Associated Lifecycle Management, and Logistics Responsibilities and Accountability

f. VS-JI-22A, Virtual SYSCOM Engineering and Technical Authority Policy, 31 January 2007

g. ASN RDA Memo, Systems Engineering Technical Review Process for Naval Acquisition Programs, 13 June 2008

h. Naval Systems Engineering Technical Review Handbook, Version 1.0

i. NAVSEAINST 5000.9, Virtual SYSCOM Naval SYSCOM Systems Engineering Policy, 2009

j. Department of the Navy Joint letter, Implementation of System Design Specification (SDS) Guidebook and associated system specific appendices, 18 July 2008

k. Department of the Navy System Design Specification (SDS) Guidebook, 17 July, 2008 l. CJCSI 3170.01G, Joint Capabilities Integration and Development System, 1 March

2-2

m. DoD Systems Engineering Plan Preparation Guide, Version 2.01, April 2008 with addendum v2 dated 02 July 2009

n. ASN RDA Memo, Amplification of Policy for DON Systems Engineering Plan Review and Approval, 16 November 2007

o. Naval SYSCOM Risk Management Policy Instruction, 21 July 2008 p. Naval Systems Engineering Guide, October 2004

q. DoD Technical Review Checklist, https://acc.dau.mil/TechRevCklist

3.0 SYSTEMS ENGINEERING TECHNICAL REVIEW (SETR)

3.1 General Background

SETRs are scripted events during which the system under development is independently assessed to determine the system’s technical maturity and to assess if its development is proceeding according to plan. The purpose is to determine whether the risk of continued development justifies the commitment of resources. One outcome of a SETR might be the decision to modify the current plan with the goal of mitigating identified risk.

The establishment and tailoring of SETR schedules are to be described in the program's Systems Engineering Plan (SEP). The Program Management Office (PMO) technical lead or Chief Systems Engineer will have the primary role in developing the SEP. The NAVSEA CHENG shall designate in writing an independent technical authority (ITA)) to assist the PM in the development of the SEP and co-chair SETR events. Systems Command (SYSCOM) Technical Warrants (TWs) and the PEO IWS TD must be engaged to ensure quality, completeness, and acceptable level of risk.

As a system progresses through development, the system increases in technical content and maturity according to its development plan. System level SETRs are to be timed to correspond to specific levels of maturity in order to track progress and risk, and to exploit available information and documentation.

The timing of SETRs depends on two key items: the maturity of the system, and the development plan (e.g., Hardware Development Plan (HDP and Software Development Plan (SDP)) as expressed in the SEP (BFTT will utilize the SEP as the governing document. No SEMP will be developed.) and/or the developer’s Systems Engineering Management Plan (SEMP). Involvement of stakeholders, peers, technical authorities, subject matter experts (SMEs), and senior leaders in SETRs is defined in the SEP. SETRs shall be scheduled at key points in the development schedule when the system has achieved a discrete stage of maturity and when the appropriate data is available to assess progress thereby facilitating an independent review, and to identify any variations from the plan or emerging factors that would represent potential risk. SETRs shall also be scheduled to support specific key decision points regarding the continuing development effort. SETRs shall be conducted on all relevant parts of the system, ranging from the entire system down to the lower-level elements (e.g., configuration items or sub-systems).

The focus of SETR assessments will change as a system and/or a product progresses through its development. Early in the process, the primary focus is on defining the requirements on which subsequent design and development activities will be based. SETRs conducted during the early stages of development usually focus on ensuring that top-level concepts and system definitions reflect user requirements. Once system level definition is complete, the focus is on design at sub-system levels and below.

Individual systems are rarely developed and rarely operate in isolation – there are typically multiple parallel development efforts taking place, ranging from the lowest level

3-8 products, up to the entire ship platform, and to Families of Systems (FoSs) which, as a whole, provide warfighting capabilities to the Navy. As a result, reviews often need to be conducted that assess a system’s role in the context of other systems that may form a System of Systems (SoS). This is to ensure that interoperability and product line concerns are adequately addressed. The scope of the system will influence the content of SETRs as well as their timing, since they need to be coordinated with similar events for the other systems. Table C-1 in Appendix C compares different types of projects at different levels of scale and summarizes some of their important characteristics.

Developers are expected to exercise discipline in following their SEP. Failure to adhere to such plans generally results in significantly increased risks to the success of the program. The program or project managers (PMs), applicable TWs, the developers, and other stakeholders are responsible for closely monitoring adherence to the plans to ensure that all activities and events take place as scheduled; that the products and artifacts comply with their requirements; and that the processes defined are closely followed.

This TRM provides specific guidance including entrance and exit/closure criteria for SETRs traditionally utilized in single system development. However; a system in one context can be a subsystem in another (e.g., a radar is a subsystem of a combat system, a combat system is a subsystem of a warfare system). System of Systems Engineering (SoSE) crosses the boundaries of Major Program Manager (MPM) or PEO responsibilities.

The SoSE and Family of Systems Engineering (FoSE) processes are continuing to mature. It is the intent of PEO IWS to further revise this document, as well as existing SE instructions and guidance, to accommodate SoSE and FoSE processes as they mature. The SEP for a program will define the SETRs within a SoS context. The following paragraphs provide interim guidance for the application of SETRs in SoSE.

3.2 Objectives

SETRs have many objectives, including:

a. Assessing the development maturity based on technology maturity and technical development goals from the SEP, SE events and accomplishments, and empirical test data supporting progress to date.

b. Ensuring operational, functional, performance, Information Assurance (IA) and cost requirements, designs, technical performance measurements, and technical plans are being tracked and are on schedule.

c. Assessing the system requirements to ensure that the requirements are unambiguous, consistent, complete, feasible, verifiable, ranked for priority and stability, and traceable to top-level requirements (TLRs).

d. Demonstrating that the relationships, interactions, interdependencies, and interfaces between required items and externally interfacing items, system functions, subsystems, and system elements (including usability and maintainability), as appropriate, have been addressed.

e. Ensuring open architecture (OA) in the emerging system; assessing the degree of Naval Enterprise reuse (availability and potential for component reuse); and critiquing any tradeoff decisions.

f. Ensuring that the results of trade studies are used to define concepts and that risks associated with alternatives have been analyzed.

g. Ensuring that technical designs will be usable by the target warfighter population, meet the Fleet requirements and have viable training options.

h. Ensuring that trade studies include IA and safety requirements and that the risks of failing to meet these requirements have been properly treated.

i. Confirming that the effects of technical risks on cost, schedule, and performance, as well as risk reduction measures, rationale, and assumptions made in quantifying the risks have been addressed.

j. Providing a forum for communication, coordination, and integration across all acquisition disciplines.

k. Establishing a common configuration baseline from which to proceed to the next level of design.

l. Confirming that continued development is warranted (with or without modifications to the program), and when it is not, identifying the technical measures necessary to preserve for reuse as much of the technology, hardware (HW), and software (SW) developed to date as possible. In the case where program redirection or restructuring is considered appropriate, the review should ensure that executable alternatives have been defined (discontinue development, or take corrective action on the products and/or processes of the current phase before proceeding to the next phase).

m. Verifying the system is ready for the appropriate level and type of testing or that appropriate approvals for production and certification have been granted

n. Identifying resources (e.g., people, funding, support assets, test facilities, as appropriate) required for continued development, testing, or production.

o. Reaching technical concurrence with stakeholders.

p. Provide a check of proposed design configuration versus specification and contractual requirements.

q. Evaluate systems configuration at different stages in terms of requirements.

3.3 Recommended SETRs and Levels

The recommended minimum selection of SETRs for a new system or baseline upgrade is noted below. This selection, and any additional reviews deemed necessary, should be called out in the program’s SEP.

a. Initial Technical Review (ITR) (BFTT will not have an ITR)

b. Alternative Systems Review (ASR) (system level) (The Government conducted a BFTT tailored ASR on 10 December 2012)

c. System Requirements Review (SRR) (system level) (The Government will conduct two (2) BFTT will have a SRR’s. SRR #1 will be an Internal Government Review scheduled for fourth quarter FY13. SRR #2 will be conducted by the Government for the Build 6.0 Contractor within 60 days of award.)

d. System Functional Review (SFR) (system level) (BFTT Contractor Lead)

e. Software Specification Review (SSR) (subsystem level) (Not a designated milestone review for BFTT)

f. Preliminary Design Review (PDR) (system (defined by the program system engineer)/ subsystem (HW/SW) level (addressed herein)) (BFTT Contractor Lead)

g. Critical Design Review (CDR) (system (defined by the program system engineer)/subsystem (HW/SW) level (addressed herein)) (BFTT Contractor Lead)

h. Test Readiness Review (TRR)

i. System Verification Review (SVR)

j. Production Readiness Review (PRR) (Not a designated milestone review for BFTT)

k. Functional Configuration Audit (FCA) (system level)

l. Physical Configuration Audit (PCA) (system level) (Not a designated milestone review for BFTT. Government witnessed unit level and Formal Qualification Testing will replace the PCA.)

ASR, SRR, SFR, PDR, CDR, TRR, FCA, and PCA (BFTT will not have a PCA) will be conducted at the system level for those systems being managed as the system. SSRs, PDRs, CDRs and TRRs for those systems being managed at the subsystem or computer software component item (CSCI), should be conducted at the subsystem level.

Specific SETR agenda items for each review and specific timing for SETRs will be in accordance with the Technical Review Action Plan (TRAP) and procedural guidelines in the Naval SETR Handbook and appropriate references or alternatively in the program SEP or modified SEP for acquisition programs that do not require a SEP.

It should be noted that additional technical and programmatic reviews are depicted in Figures (1) and (2); requirements for these reviews are addressed in the SETR Handbook (ref (b)) and specific implementation recommendations in Appendices B and C.

3.4 Roles and Responsibilities

Participants assigned to support a program SETR should be familiar with the key positions, team, and SETR events. SETRs and the Extended Integrated Product Team (IPT) are co-chaired by the PM and the Independent Technical Authority (ITA) and other co-chairs as required. SETR events shall be co-chaired by the PM and the ITA as depicted in Figure 3. The co-chairs shall work together to plan, execute and document the SETRs and shall share information freely in a timely manner with each other.

SETR EVENT

SETR PANEL

PM

Co-Chair

ITA

Co-chairs

Other Co-chairs*

Program SMEs

Technical Authority

Other Members***

Participants (TRB) **

EXTENDED IPT**** (Example NRT/TRT/ERT)

FIGURE 3. SYSTEMS ENGINEERING TECHNICAL REVIEWS (SETRs) CO-CHAIRS

*See section 3.4.4 for information about other co-chairs ** See section 3.4.7 for information about Technical Authority (TA) participants; TA participants are integral to the Extended IPT activities (not a separate review team) and report as required to the ITA. The identified Technical Authority Participants and the ITA is referred to as the Technical Review Board *** Other Members – The Program Manager may include other members as part of the Extended Integrated Product Team (IPT) as required for a SETR event (i.e. legal, contracts) ****The Extended IPT is typically known as the Navy Review Team, Technical Review Team, Engineering Review Team – this is the team that reviews SETR artifacts and provides testimony to the SETR Panel as to their adequacy in addition this team typically provides for a cradle to grave program system engineering activities and test and evaluation activities

3.4.1 Program / Project Manager (PM)

As stated specifically in the SECNAVINST 5400.15C, “Program managers will be vested with the authority, accountability, and resources necessary to manage all aspects of assigned acquisition programs from concept development to demilitarization and disposal. Program managers will exercise management authority, including selection and application, over core capabilities that support the acquisition, in-service support, and demilitarization and disposal of assigned weapons and IT systems”

The PM is responsible for planning, executing and documenting the overall SETR process for a program under review. The PM may propose to the NAVSEA CHENG’s designated representative, SEA 05H / PEO IWS TD the ITA and or concur with the proposed ITA. SETR process planning begins during preparation of the program SEP. The planning must take into account the general timing and tailoring of SETR assessments; securing budget, schedule, and personnel resources; and coordinating with other program reviews, such as program baseline reviews and sponsor programmatic reviews (e.g., two-pass/six-gate review).

PM responsibilities for SEP development and SETR process planning are in the DoDI 5000.02, SECNAVINST 5000.2D, NAVSEAINST 5000.9 and the DoD Systems Engineering Plan Preparation Guide and Addendum and include the following:

a. Plan and prepare the SETR in a timely and effective manner.

b. Identify, allocate and fund resources necessary to accomplish the total SETR effort, including identification of supporting functions such as facilities and infrastructure.

c. Establish program Extended IPT as required.

d. Tailor the SETR consistent with good engineering judgment and program complexity and risk levels.

e. Schedule SETR assessments consistent with availability of supporting materials and adequate time for Naval Technical Authorities (TAs) to review those materials.

f. Establish areas of interest to focus the SETR assessments.

g. Tailor event-driven entry and closure criteria and include in the approved program

SEP.

h. Plan to conduct incremental SETR assessments, as needed.

i. Review all systems functions, including human operational and maintenance functions.

j. Confirm that HW, human, and SW areas of interest are assessed with appropriate priorities.

k. Propose the SETR agenda, TA participants and other Extended IPT participants including co-chairs as appropriate.

l. Develop the TRAP/SEP/modified SEP as appropriate for the SETRs.

m. Co-chair the SETR event with the ITA and other co-chairs, capture action items and monitor closure of the SETR event and provide an event report for concurrence by the ITA and other co-chairs.

3.4.2 SYSCOMs

As specifically stated in the SECNAVINST 5400.15C, “ The SYSCOM Commanders are responsible for: providing for in-service support; providing support services to PEOs and DRPMs without duplicating their management functions; and serving as the technical authority and operational safety and assurance certification authorities for their designated areas or responsibility” NAVSEA is the primary SYSCOM for PEO IWS.

3.4.3 Independent Technical Authority (ITA)

SETR events are conducted by the SETR Panel and Extended IPT. The Independent Technical Authority (ITA) is the designated representative of Technical Authority and is a co- chair of each SETR event specified in a program SEP, is assigned in writing by the NAVSEA CHENG’s designated representative, SEA 05H / PEO IWS TD and has the responsibility to report in writing unresolved technical issues to the NAVSEA CHENG designated representative, SEA 05H / PEO IWS TD after a good faith effort to resolve the issue at the SETR Panel or Extended IPT. The ITA shall be independent from the Program and will normally be a senior individual from the NAVSEA TA chain with technical expertise relevant to the program, but can in some cases be an independent authority within the Naval Enterprise including the PEO infrastructure if so designated by the NAVSEA CHENG office. When approved by NAVSEA CHENG authority, a principal member of the TRB from the program or the Fleet may be assigned.

The role of the ITA includes the following responsibilities:

a. Oversight of the SETR process, co-chair of the SETRs as the ITA including all meetings of the TRB to review and discuss program materials.

b. Concurrence on the Extended IPT participants.

c. Concurrence on the TRAP/SEP/modified SEP as appropriate. d. Concurrence on areas of interest.

e. Concurrence on entry and closure criteria.

f. Concurrence on the SETR Summary Report.

g. Facilitate participation of SMEs in the SETR process, such as early distribution of program materials to be reviewed and meeting announcements, as well as attention to scheduling convenient meeting locations.

h. Provide specific constructive feedback and viable recommended technical options during program execution as risks and issues arise.

i. Support the PM as required at the leadership level if recommendations will impact the cost, schedule and performance risk of the program.

In the event the PM and the ITA do not agree on findings, the issue(s) will be elevated to the next level authorities, the Program Executive Officer for Integrated Warfare Systems and the NAVSEA Chief Engineer.

3.4.4 Other Co-chairs

Integrated Warfare Systems development requires the participation across the Naval Enterprise to ensure that the combat systems and elements for integration on Naval platforms are accomplished in a timely, cost effective and quality manner. The PM may designate additional co-chairs that represent the interests of a significant partner that has a material interest in the system/product under development. An example is the designation of a ACB 12 CDR Missile Defense Agency (MDA) co-chair due to the requirements drivers for the development of Ballistic Missile Defense and the associated funding provided to ACB 12 from MDA.

The responsibilities of the other co-chairs typically include but are not limited to the following:

a. Co-chair the SETRs as required.

b. Provide subject matter experts (SMEs) and other Extended IPT participants as agreed to with the PM.

c. Concurrence on the TRAP/SEP/modified SEP as appropriate to their authorities. d. Concurrence on areas of interest in alignment with their principle area of concern. e. Concurrence on entry and closure criteria as appropriate to their authorities.

f. Concurrence on the SETR Event Report alignment with their principle area of concern.

g. Facilitate participation of SMEs in the SETR process, such as early distribution of program materials to be reviewed and meeting announcements, as well as attention to scheduling convenient meeting locations.

h. Provide specific constructive feedback and viable recommended technical options during program execution as risks and issues arise.

i. Support the PM as required at the leadership level if recommendations will impact the cost, schedule and performance risk of the program.

In the event the PM, ITA and other co-chairs do not agree on findings, the issue(s) will be elevated to the next level authorities, the Program Executive Officer for Integrated Warfare Systems and the NAVSEA Chief Engineer and the co-chair(s)’ organizational Executive Officer.

3.4.5 SETR Panel

The SETR Panel is composed of the PM, ITA, other co-chairs and other selected panel members designated by the co-chairs. All co-chairs shall participate in the SETR planning, execution, and follow-up activities and shall keep all members apprised of program health including risks and issues. It is desired that the co-chairs will be the same individuals throughout an acquisition phase. The Panel shall be designated in writing prior to the SETR event. This provides for continuity and consistency of reviews. If there is a change in an individual panel member it is the responsibility of the respective organization to provide for an appropriate turn- over in a timely manner and obtain concurrence from the PM and ITA.

The SETR panel shall review, process and approve all Requests for Information (RFIs) and Requests for Actions (RFAs) during Panel assessments and deliberations and designate which are critical to close prior to the entering of a SETR event or closure of the SETR event underway (see section 3.6 paragraph 6 for further guidance).

3.4.6 Extended Integrated Product Team (IPT)

The success of SETRs depends on the body of technical reviewers used to review artifacts at all levels in order to collect and assess information that will be used to support the SETR review process. This review is accomplished while these artifacts are still under development to verify the artifacts conform to requirements and standards; to identify record and remove defects; to ensure product line factors/issues are considered; and to provide a basis for subsequent product approval at formal SETRs (e.g., SSR, PDR, CDR). This body is typically called a Navy Review Team (NRT), Technical Review Team, or Engineering Review Team (ERT). The mission of the Extended IPT is to assess the technical health of a program under review and to provide program management with an assessment of readiness to proceed into the next technical phase. Ideally the Extended IPT is a standing body with consistent membership that follows the evolution of the program throughout its acquisition life-cycle. The distinction between the Program SMEs and the Technical Authority participants is that TRB members shall not provide ‘the’ technical solution and then subsequently review and testify to ‘their technical solution’ adequacy.

These Review Teams are established by the PM at project inception and consist of technical staff independent of the developers, and typically include representatives from the PM, SYSCOMs including Technical Authority participants, Naval Laboratories, academia, and other members that represent the interest of other co-chairs that have a material interest in the development of the combat system and or products/elements. Review teams are used throughout the development effort by the PM as a resource to assist in tracking progress, assessing products, predicting risk, and ensuring quality. Review Team membership shall be selected to ensure the team has expertise to adequately cover all relevant topics and to ensure that all members are able to actively participate. The PM is responsible for establishment of the program Extended IPT and shall use the approved SEP for determining the composition of the program Extended IPT; the PM may modify the Extended IPT membership as specified in the SEP upon recommendation and concurrence of the co-chairs.

Typical membership on the program IPT includes the following individuals:

a. PM or designee (Deputy PM)

b. TA participants (e.g.TWH)

c. Program Competency Leads or representatives, as needed d. Performing Activity Leads or representatives, as needed

e. IPT briefer(s), as needed in accordance with SETR agenda

The Extended IPT shall participate during the planning, execution and documenting of all SETRs and shall keep the co-chairs informed in a timely manner of all risks and issues and provide for viable alternatives. The Extended IPT supports SETR assessments by preparing and assembling the program products to be presented to and reviewed by the SETR Panel. Extended IPT members also may brief the program accomplishments and supporting products to the SETR panel during the SETR events.

3.4.7 Technical Authority (TA) Participants

TA participants provide an independent assessment of engineering processes, products and risks associated with a program. TA participants are TWHs and or TWH pyramid members which are associated with specific aspects of the project and are responsible for supporting the PM and the program’s System Engineer. The TA participants shall be designated in writing by the SETR Panel Co-chairs. In the event the PM, ITA and other co-chairs do not agree on membership, the issue(s) will be elevated to the next level authorities, the Program Executive Officer for Integrated Warfare Systems and the NAVSEA Chief Engineer and the co-chair(s)’ organizational Executive Officer.

The TWHs and their pyramid ensure that processes are documented in the SEP, that they are executed with sufficient rigor, and that the resulting products meet acceptable standards. The TWHs guide the System Engineer in all standards implementations and ensure usability, supportability and interoperability of the final products. While the TWHs support the program, the program’s System Engineer is responsible to the PM for engineering efforts within the program.

Ref (f) establishes the Navy’s TA policy. Programs are to include TA representatives on IPTs to ensure quality, completeness and acceptable levels of risk. Programs must coordinate with NAVSEA CHENG or representative for TWH engagement in the program and assignment of TA representatives for SETR participation.

3.4.8 Technical Review Board (TRB)

The Technical Authority participants and the ITA collectively are referred to as the TRB. The Technical Authority (TA) participants are the TWHs or TWH pyramid individuals that participate in the Extended IPT as independent reviewers of SETR artifacts. The TA participants shall participate in all SETR processes and activities as integral members of the Extended IPT and shall assess the system under review with the objectives, criteria, and areas of interest developed during SETR process planning. Ideally, the TRB is a standing body with reasonably consistent membership that follows the evolution of a program through its development life cycle.

The mission of the ITA and TA participants is to provide for an independent assessment of the technical health of a program under review and to provide the Extended IPT and SETR Panel with an assessment of readiness to proceed into the next technical phase in a timely manner. A TA participant shall not provide for ‘the’ technical solution and then assess ‘their’ technical solution adequacy. If a TA participant is required to supply ‘the’ technical solution for the program under assessment, another TWH or TWH pyramid individual must be designated to the TA participant membership. The TA participants shall participate during the planning, execution, and documenting of all SETRs and shall keep the PM, PM’s IPT and ITA informed in a timely manner of all risks and issues and provide for viable alternatives for resolution.

If a technical issue cannot be resolved or otherwise addressed at the Extended IPT or SETR Panel level, the TA participant and ITA will immediately inform in writing the NAVSEA Chief Engineer (CHENG) and PEO IWS of the risk/issue, impact to the program, alternatives considered and proposed way ahead.

3.4.9 Technical Interchange Meeting [TIM]

SETR TIMs are meetings and/or forums where the program Extended IPT members can interact and hold technical discussions and review supporting documentation, seek answers to technical questions, identify any potential issues, and prioritize follow-on topics.

3.5 Planning for Systems Engineering Technical Reviews

The SETR Panel co-chairs are responsible for planning SETRs and all parties have responsibilities that must be met satisfactorily to realize a successful outcome from the SETR assessment.

Moreover, although no single planning approach is mandated, the critical focus for all parties involved must remain on demonstrating and verifying the successful achievement of the pre-established entrance and exit criteria for the SETR assessment.

For each SETR assessment, the SETR Panel and Extended IPT participants must be designated in writing prior to the event. The co-chairs shall be designated early in the SETR assessment cycle.

The SETR panel concurs with the TRAP/SEP/modified SEP at the beginning of the SETR assessment cycle. The TRAP/SEP/modified SEP should be coordinated across the Extended IPT to ensure that all plans are supportable and executable. Once available, the TRAP/SEP/modified SEP should be distributed to all SETR Panel, and Extended IPT members.

Figure 4 depicts a typical timeline for SETR staffing and preparation purposes. The SETR Panel co-chairs shall assess progress toward meeting all entry criteria at a time sufficiently in advance of the formal review meeting to allow for correction of any shortcoming and deficiencies identified (45 days is a nominal period) and prior to the final agenda and letter being released announcing the SETR event. For all entry criteria that are assessed that will not be met by the SETR event, the SETR Panel co-chairs shall activate mitigation plans immediately to avoid a rescheduling of the SETR assessment meeting. If the entry criteria and critical entry related RFA/RFIs cannot be met in an adequate and timely manner, the SETR Panel Co-chairs must replan the SETR event(s) activities and provide the new plan in writing to the SETR Panel and Extended IPT. The SETR Event replan shall be concurred with by all co-chairs in writing. The determination to reschedule a SETR assessment meeting should be made no later than 15 days prior to the scheduled event.

FIGURE 4. TYPICAL PLANNING TIMELINE FOR SETR ASSESSMENTS

3.5.1 Technical Review Action Plan (TRAP)/SEP/Modified SEP

The TRAP is prepared early in the SETR assessment cycle (see Figure 4 above). The TRAP is a useful tool for managing the expectations of the participants and stakeholders involved in a SETR because it communicates government expectations and success criteria. The PM may include the information required in a TRAP in the SEP body or appendix vice developing a separate document. For programs that are acquiring systems and products (elements) that do not require a SEP, a modified SEP shall be developed that includes the minimum information required of a TRAP.

The TRAP/SEP/modified SEP must be aligned with the technical approach defined in the SEP/modified SEP. The TRAP/SEP/modified SEP contains the minimum following information:

a. Program background, including tailoring of the SETR assessment.

b. Schedule of events leading to the formal SETR assessment meeting (including incremental subsystem reviews).

c. SETR Panel co-chair specific names, Extended IPT minimum membership organizations including the TA participants.

d. Entry criteria (including closeout actions and closure criteria of previous SETRs). e. Proposed agenda, including areas of interest.

f. Closure criteria

g. Request For Action (RFA)/Request For Information (RFI) Process

3.5.2 Systems Engineering Technical Review (SETR) Agenda

The agenda contains elements that correlate to technical accomplishments within the pre- established areas of interest and evidence of successful fulfillment of entry and closure criteria for the SETR assessment. The program IPT and their representatives are expected to present these elements, with supporting documentation and products.

Planning the SETR meeting agenda involves assessing the status and complexity of the areas of interest and technical issues that must be addressed during the meeting.