Attachment J-09 - TRM Version 3.0_27MAR20 12MAY2022.pdf

Program Executive Officer, Integrated Warfare Systems

Naval Systems Engineering Directorate Integrated Warfare Systems Engineering

Technical Review Manual

Version 3.0

Department of the Navy Naval Sea Systems Command / Program Executive Office, Integrated Warfare Systems

1333 Isaac Hull Avenue SE Washington Navy Yard, DC 20376

Distribution Statement A: Approved for public release; distribution is unlimited.

Technical Review Manual, Version 3.0

Foreword As a Navy, we are at a confluence of engineering challenges, demands for increased system of systems capabilities, and demand signals to quickly field advanced capabilities. These factors are driven by the increases in system complexity, the need for increased system employment and operational flexibility, and the need to outpace threat systems and environments. As our systems and capabilities have become more complex, our approach to systems engineering and fielding of capability has remained largely unchanged. Our methods of systems capability acquisition, development, and maintenance require significant time and effort. Systems developed by Program Executive Office Integrated Warfare Systems (PEO IWS) form a major part of the foundation of the defensive and deterrent posture of our Country. We cannot afford to continue to acquire, develop, and field systems in the same manner. In concert with the updated System and Element Certification Instructions, we need to balance risk rather than strive to achieve a state of zero risk, and to shorten the timeframe from system development to capability at Sea.

PEO IWS Major Program Managers (MPM) and the Naval Systems Engineering Directorate, Integrated Warfare Systems Engineering Group (NAVSEA 05H) System Integration Managers (SIM) are expected to fully enact (for new systems/programs) and initiate migration (for existing systems/programs) towards the key tenets of systems engineering, capability transition and fielding.

These tenets are:

• Enabling Model-Based Systems Engineering (MBSE) environments between Prime Contractors and Government Engineers to afford early, continuous, and structured design evaluation and feedback

• Implementing Agile Development cycles within programs to ensure that capability and improvements to existing capability flow quickly from design to field

• Establishing methods to rapidly deploy capability from developer to warfighter while addressing continuous feedback from user community (i.e. the Secure Development and Operations paradigm (DevSecOps))

At all levels of the Navy, these tenets are viewed as necessary to regain the Sea Dominance and to outpace our adversaries, there are and will continue to be discussions on how to best implement these tenets. In PEO IWS and NAVSEA 05H, we need to enact and embrace these tenets immediately to support our own rapid progression, while at the same time maintaining responsibility for the sound execution of systems engineering practice across the PEO IWS portfolio. The Technical Review Manual (TRM) is intended to provide a framework for MPMs and SIMs to customize the application of SE throughout the acquisition process by assessing program development progress and reducing risk to deliver capability within cost and schedule. To accomplish this goal, MPM and SIM shall work together in applying SE principles and processes across their systems of responsibility. The TRM is used to support a multitude of systems, including: Acquisition Category (ACAT), non-ACAT, accelerated acquisition, and urgent need programs. The customization and execution of SE principles and processes enables the MPM and SIM to identify and manage risk while providing confidence in system maturity to meet required programmatic milestones.

As we gain experience and understanding in how to effectively manage our programs while embracing the three tenets, best practices can be captured in future revisions of the TRM.

Contents Foreword

1 Introduction

1.1 Scope

1.2 Background

1.3 Document Organization

1.4 Document Update

2 References

3 Systems Engineering Technical Review Process

3.1 Background

3.2 System of Systems

3.3 Program Reviews and In-Process Reviews

3.4 Model Based Systems Engineering (MBSE)

3.4.1 System of Systems Model

3.4.2 System Model

3.4.3 Detailed Models

3.4.4 Design and Manufacture Release

3.5 Agile and Secure Development and Operations (DevSecOps)

3.6 Program Models and Technical Reviews for Existing PEO IWS Programs

3.7 Roles and Responsibilities

3.7.1 Major Program Manager and System Integration Manager

3.7.2 Technical Review Panel

3.7.3 Technical Review Team

3.8 Planning for Technical Reviews

3.8.1 Technical Review Action Plan (TRAP)

3.8.2 Agenda

3.9 Conducting Technical Reviews

3.9.1 Readiness Assessment Meeting

3.9.2 Technical Review

3.9.3 Technical Review Closure

3.9.4 Technical Review Summary Report

Appendix A Acronyms................................................................................................................... A-1

Appendix B Specific Technical Reviews ....................................................................................... B-1

Appendix C Technical Review Documentation Matrix ................................................................. C-1

Appendix D Technical Review Action Plan Template .................................................................. D-1

Appendix E Action Item Process .................................................................................................... E-1

Appendix F Examples of MBSE, Agile, and DevSecOps Project Management Framework ........ F-1

Figures Figure 1: Hardware and Software Development Program Figure 2: Software Development Program Figure 3: Sample Software Increment Review (SWIR) Topics Figure 4: Accelerated Acquisition Program Figure 5: Technical Review Panel Figure 6: Typical Technical Review Process Figure 7: Example of a Use Case Modeling ........................................................................................... F-1 Figure 8: Example of Performance Analysis .......................................................................................... F-2 Figure 9: Example of Threat Driven Requirement Analysis .................................................................. F-3 Figure 10: Example of Requirements Baseline Modeling ...................................................................... F-4 Figure 11: Example of Functional Baseline and Allocation Modeling .................................................. F-4 Figure 12: Example of Allocated Baseline Modeling............................................................................. F-5 Figure 13: Example of Physical Baseline Modeling............................................................................... F-6 Figure 14: Graphical Depiction of Agile Process within DoD Acquisition Phases ............................... F-7 Figure 15: Agile Process Workflow ....................................................................................................... F-8 Figure 16: Notional Pre-Sprint Planning Activities and Timeline .......................................................... F-9 Figure 17: Notional Sprint Process, Meetings, and Deliverables ......................................................... F-10 Figure 18: DevSecOps Embedded with Agile within DoD Acquisition Phases .................................. F-12

Tables Table 1: SE Products to Support Technical Reviews ............................................................................ C-1

1 Introduction

1.1 Scope

In view of the new systems engineering (SE) tenets (Model Based Systems Engineering (MBSE), Agile Development, and Secure Development and Operations (DevSecOps)), the Systems Engineering Technical Review (SETR) processes for new programs will be a continuous set of SE activities and Technical Reviews to assess technical, cost, and schedule performance. The notional timing of the various SETRs and how they fit within the acquisition life cycle is shown in DoDI 5000.02, reference (e). The scope of the Technical Review Manual (TRM) is to provide a tailorable framework for the application of systems engineering, and the implementation of SETR process, across the Program Executive Office Integrated Warfare Systems (PEO IWS) portfolio for Acquisition Category (ACAT), non-ACAT, accelerated acquisition, and urgent need programs. The primary functions of the TRM are to:

• Serve as a guide for Major Program Manager (MPM) and System Integration Manager (SIM) to jointly customize and streamline the SE processes and in assessing the progress of technical maturity of the system,

• Specify the policies that govern the SE processes,

• Provide the description of overall SE processes, as well as the objectives and activities (including entrance and exit criteria), and

• Provide guidance in the conduct of Technical Reviews.

MPMs are responsible for ensuring that SE is applied, planned, and executed for their programs, while SIMs are responsible for providing independent Technical Authority (TA) for establishing, monitoring, and approving technical standards, tools, and processes. Together, they form a cohesive team supporting all aspects of the program, conforming to higher authority policies, requirements, architectures, and standards, while delivering the best product possible to the Fleet.

Through joint concurrence of the MPM and SIM, the program’s SE processes criteria shall be tailored to its technical, cost, and schedule requirements to reflect each program’s unique nature. The MPM and SIM shall work in conjunction with one another, co-chairing Technical Reviews to ensure the systems are resourced, technically capable, and that the processes are applied to meet the established entrance and exit criteria. They shall establish an agreeable set of Technical Review objectives to assess compliance of SE principles and to ensure technical and programmatic maturity to meet Fleet’s needs, including but not limited to the following:

• Completion of requirements analysis and traceability,

• Functional baseline,

• Allocated baseline,

• Design progress,

• Integration and test activities, and

• Membership and participation of the Program Integrated Product Teams (IPT) and Technical Interchange Meetings (TIMs).

While the primary focus of Technical Reviews is on technical content and maturity, the Technical Reviews shall also include cost and schedule considerations to facilitate sound judgment on technical risks and proposed solutions.

The program’s SE processes shall be documented in the program’s Systems Engineering Plan (SEP) for ACAT programs and a modified SEP for non-ACAT programs, in accordance with the Office of the Secretary of Defense (OSD) guidance, reference (d). The SEP, or modified SEP, shall include a description of how the program plans to implement the assessment of system maturity under the new tenets and explain how the tailored activities accomplish the overall SETR process objectives. In addition, the SEP should describe all technical aspects of an acquisition program: SE processes, resources, key technical tasks, activities and events along with metrics and success criteria. The SEP and modified SEP shall be initially signed by PEO IWS and Naval Systems Engineering Directorate, Integrated Warfare Systems Engineering Group (NAVSEA 05H). Updates are to be signed by the MPM and SIM. Regardless of ACAT designation, in the beginning of a program, a Memorandum for the Record (MFR) shall be created to document its scope, known technical challenges, and application of SE as agreed upon by the MPM and SIM.

This revision of the TRM replaces Enclosure 5 of the Naval Systems Engineering Technical Review Handbook. The Naval Systems Engineering Technical Review Handbook is Enclosure 2 of NAVSEAINST 5000.9, reference (c).

1.2 Background

Per the memorandum in reference (b), the Assistant Secretary of the Navy for Research, Development, and Acquisition (ASN (RDA)) directs the development of a common SETR process by the System Commands (SYSCOMs) and the institutionalization of this process. Technical Reviews in the SETR process are chaired by a senior and independent person from the TA. The NAVSEA 05H SIM is the senior technical government official who is independent from the program being reviewed, as required by reference (f).

PEO IWS has the authority, responsibility, and accountability for lifecycle management of all acquisition programs within their cognizance per Title 10 US Code - Section 1737 & 1706. PEO IWS shall act for and exercise the programmatic authority to directly supervise the management of assigned programs, maintaining oversight of cost, schedule, performance, and make timely and forthright reports directly to the ASN (RDA) and/or the Milestone Decision Authority (MDA) for all matters pertaining to acquisition.

NAVSEA 05 is Naval Sea Systems Command’s Chief Engineer (CHENG) and TA for platform and ship systems including areas common to both surface ships and submarines. The responsibility of TA duties are delegated to NAVSEA 05 in NAVSEAINST 5400.111A, reference (g), and executed through the NAVSEA 05 organizations. NAVSEA 05H is the SE and TA working in conjunction with PEO IWS programmatic authority to directly support the Fleet by providing effective, efficient, and technically capable systems to meet warfighter needs. The NAVSEA 05H SIMs are the warranted TA, responsible and accountable for establishing, monitoring, and approving technical standards, tools and processes, in conformance to higher authority policy requirements, architectures, and standards.

PEO IWS and NAVSEA 05H work together to ensure that TA and SETR processes are an integral part of their program execution and that acquisition issues pertaining to the supportability of their systems are coordinated and addressed throughout the entire lifecycle, per SECNAVINST 5400.15, reference (a).

1.3 Document Organization

• Section 1 provides an overview of the document.

• Section 2 lists references upon which the SETR process is based.

• Section 3 provides overviews and general guidelines for MBSE, Agile, and DevSecOps, as well as the SETR process for existing programs, and general guidelines to conduct a Technical Review (scheduling, preparations, documentation, Action Items (AIs), etc.).

• Appendix A lists acronyms found in this manual.

• Appendix B provides an overview of each type of Technical Review that is conducted by existing PEO IWS programs and notional entrance and exit criteria for each review.

• Appendix C provides a summary of Technical Review documentation for the existing programs providing a tabular listing of the documentation and artifacts associated with each review and recommended architectural views for consideration at each review.

• Appendix D provides a template for developing a Technical Review Action Plan (TRAP). The template includes the minimum information that shall be included in a TRAP.

• Appendix E provides a template for recording Action Items during a review.

• Appendix F provides examples of MBSE, Agile Development and DevSecOps templates

1.4 Document Update

This TRM cancels Version 2.0, dated 18 December 2009. It will be updated based on changes to relevant acquisition policies, instructions, and lessons learned while executing the SETR process, especially as related to new methodologies like MBSE, Agile, DevSecOps. This manual should be reassessed every three (3) years or less if there are significant changes that warrant a new update.

2 References The SETR process in this manual complies with reference (c). This manual does not replace active acquisition, SE, and operational policies, instructions, notices, and directives, but is a tool that provides a framework to assist PEO IWS programs in preparing for and conducting Technical Reviews in support of the SETR process.

Below is the list of DoD and Navy policies and directives referenced within the TRM. The policy revisions referenced are current as of the date of the TRM Version 3.0; however, programs are encouraged to verify the correct version of applicable policies as necessary.

a) SECNAVINST 5400.15C, Department of the Navy (DoN) Research and Development, Acquisition, Associated Life-Cycle Management, and Logistics Responsibilities and Accountability, 2 December 2011

b) ASN(RDA) Memo, Systems Engineering Technical Review Process for Naval Acquisition Programs, 13 June 2008

c) NAVSEAINST 5000.9, Naval SYSCOM Systems Engineering Policy, 7 July 2009, this document replaces Enclosure 5 of the Naval Systems Engineering Technical Review Handbook, which is Enclosure 2 of this reference.

d) Systems Engineering Plan (SEP) Outline, Version 2.0, 2 June 2015

e) DOD Instruction 5000.02, Operation of the Defense Acquisition System Change 4, 31 August

f) SECNAVINST 5000.2F, 26 March 2019, Office of the Secretary, Defense Acquisition System, and Joint Capabilities and Development System Implementation

g) NAVSEAINST 5400.111A, NAVSEA Engineering and Technical Authority Policy, 29 December 2014

h) FIPS 199 (Federal Information Processing Standard Publication 199), Standards for Security Categorization of Federal Information and Information Systems, February 2004.

3 Systems Engineering Technical Review Process

3.1 Background

SE ensures the effective development and delivery of capability through implementing a balanced approach with respect to design, development, integration, test, cost, schedule, performance, sustainment, and risk. This is accomplished using integrated, disciplined, and consistent SE activities and processes regardless of when a program enters the acquisition life cycle. The purpose of Technical Reviews is to determine the system's technical maturity and readiness to proceed to the next phase of development, as well as to identify any technical challenges that require additional (or reallocation of ) resources, and mitigation strategies. When the system has achieved a discrete stage of maturity, and appropriate data is available to assess progress, the MPM and SIM will set up a technical review as a decision event to identify any variations from the plan or emerging factors that would represent potential risk. The SETR process shall be conducted on all relevant parts of the system, ranging from the entire system down to the lower-level elements (e.g., Configuration Items (CI), Software (SW), or sub-systems). Appendix B provides an overview of each type of SETR typically conducted by the existing PEO IWS programs.

The focus of Technical Reviews will change as a system and/or a product progresses through its development. Early in the process, the primary focus is on defining the requirements for which subsequent design and development activities shall be based. Once system level definition is complete, the focus is on design at sub-system levels and below. Developers are expected to exercise discipline in following their established SE processes, since failure to adhere to these processes generally results in significantly increased risks to the success of the program. MPM, SIM, Technical Warrant Holders (TWH), developers, and other stakeholders are responsible for closely monitoring progress to ensure that the products and artifacts comply with their requirements. The SE processes have many objectives, including:

a) Involving all impacted stakeholders in assessing development maturity based on technology maturity and technical development goals from the SEP, SE events, and accomplishments, and empirical test data supporting progress to date.

b) Ensuring operational, functional, performance and cost requirements, designs, technical performance measurements, and technical plans are being tracked and are on schedule.

c) Assessing the system requirements to ensure that the requirements are unambiguous, consistent, complete, feasible, verifiable and traceable to top-level requirements documentation.

d) Demonstrating that the relationships, interactions, interdependencies, and interfaces between required items and externally interfacing items, system functions, subsystems, and system elements, as appropriate, have been addressed.

e) Ensuring Open Architecture (OA) in the emerging system, assessing the degree of Naval Enterprise reuse (availability and potential for component reuse), and critiquing any tradeoff decisions.

f) Ensuring that the results of trade studies are used to balance design concepts and that risks associated with alternatives have been analyzed.

g) Ensuring that technical designs shall be usable and supportable by the target warfighter population, meet the Fleet performance and readiness requirements, and have viable sustainment options.

h) Confirming that the effects of technical risks on cost, schedule, and performance, as well as risk mitigation strategies, are in place and executable.

i) Providing a forum for communication, coordination, and integration across all identified stakeholders.

j) Establishing a common configuration baseline from which to proceed to the next level of design and maturing the programs technical data.

k) Verifying the system and sustainment infrastructure is ready for the appropriate level and type of testing or that appropriate approvals for production and certification have been granted.

l) Identifying resources (e.g., people, funding, support assets, test facilities, as appropriate) required for continued development, testing, production and sustainment.

m) Reaching technical concurrence with stakeholders.

n) Providing a check of proposed design configuration versus specification and contractual requirements.

o) Ensuring that Supportability Analysis (e.g. Failure Modes, Effect, and Criticality Analysis

(FMECA), Reliability, Availability, Maintainability, and Cost (RAM-C) Analysis) Modeling and Simulation (M&S) tools are used to validate system Readiness and lifecycle cost requirement.

p) Ensuring that when Condition Based Maintenance Plus (CBM+), Reliability Centered Maintenance (RCM), Fault Detection Fault Isolation (FD/FI), and Prognostics Health Management (PHM)) technology are incorporated, intended system requirements are not compromised and metrics are tracked and reviewed for continuous improvement efforts.

q) Implementing cybersecurity requirements, verifying compliance, and obtaining an Authority to Operate (ATO).

3.2 System of Systems

Individual systems are rarely developed and operated in isolation. There are typically multiple parallel development efforts taking place, ranging from the lowest level products, up to the entire ship platform and to Family of Systems (FoS), which as a whole, provide warfighting capabilities to the Navy. As a result, a System of Systems (SoS) level SETR process is necessary to assess the integration of each individual system and its function within the SoS. The scope and complexity of the SoS being upgraded or introduced will influence the content of the SETR process as well as its timing, since the SETR process need to be coordinated with each system’s development. The TRM provides tailorable guidance including entrance and exit criteria for technical reviews traditionally utilized in single system development. However, the TRM can also be applied to complex System of Systems Engineering

(SoSE) providing a framework to manage requirements and resources that may cross the boundaries of the PEO IWS’s portfolio and responsibilities. It is recommended that a top-level specification document(s) be generated from a capability document or from a bottoms-up or top-down system element process. When resources are not in place, it is recommended that each individual program clearly document SoS risk(s) to address and provide them to the MPM and SIM for action.

3.3 Program Reviews and In-Process Reviews

MPMs across PEO IWS conduct regularly scheduled Program Reviews (PRs) to assess progress being made across that manager's portfolio of programs. These reviews are typically schedule-driven to support annual financial executions, contracting milestones, and budgeting processes. Additionally, MPMs also conduct In-Process Reviews (IPRs) of individual programs to monitor that program’s technical, schedule, and financial performance. There is the potential for cost savings by combining IPRs and PRs. The MPM and SIM should evaluate how and when to best merge these meetings throughout the program. Entrance and exit criteria should be applied to ensure technical rigor, maturity, and risk management. Attention must be given to the basic differences between technical reviews and program reviews, the most significant being inclusion of all stakeholders and the potential for loss of technical rigor.

3.4 Model Based Systems Engineering (MBSE)

PEO IWS is migrating to a new approach to systems engineering with the following three tenets:

Model Based Systems Engineering (MBSE), Agile Development, and DevSecOps for all new systems develop and major modifications. In addition, a combination of these methodologies is the preferred method for PEO IWS, and all MPMs shall continuously seek opportunity to move to a digital process.

MBSE is defined as the practice of developing a set of related models that help define, design, and document a system under development. These models provide an efficient way to explore, update, and communicate system aspects to stakeholders, while significantly reducing or eliminating dependence on traditional documents. Models are used to explore the structure and behavior of system elements, evaluate design alternatives, and quickly validate assumptions as early in the system life cycle as possible. MBSE will be used throughout the acquisition/systems engineering cycle. MBSE allows PEO IWS to take a holistic, systems approach for a specific program or for a system of systems across multiple programs to better manage the system information and data relationships. Data from multiple types of models are linked within the MBSE structure, ensuring commonality between the models. The traceability inherent in MBSE tools provides the objective quality evidence needed to address many certification, regulatory, and contractual compliance requirements. Use of MBSE will provide for efficiency to be gained by early and constant interface to design agent’s efforts (enabled by a distributed collaborative MBSE environment) that obviates the need for extensive and prolonged review periods of paper products. A Systems Modeling Language (SysML)-based system level model is the key enabler for MBSE, and provides a collaborative environment for reviews. The notional MBSE implementation framework has four elements: System of Systems (SoS) Model, System Model, Detailed Models, and Design and Manufacture Release. Appendix F provides several examples of MBSE for Use Case, Performance Analysis, Threat Driven Requirements, Requirements Baseline, etc.

3.4.1 System of Systems Model

The first element is the development of the SoS model to assess mission effectiveness. The SoS model can be at the Navy Integrated Fire Control level or at a Combat System Level. The Integrated Warfare Analysis that establishes the Concept of Employment (CONEMP) and Effects chain will be the basis for the SoS engineering to define gaps and Key Performance Parameters (KPPs) to close the gap.

3.4.2 System Model

The second element is the System Model (Combat System or Element) which will instantiate system level specifications in a model by defining the structure, behaviors, requirements, and parameters. The approach for the System Model development under MBSE is to divide the traditional systems engineering “V” into five segments and define activities required for each segment. These segments are:

1. Systems Requirements Analysis and Design;

2. Capability Analysis and Design;

3. Component Development;

4. Integration and Test, System Integration; and

5. Test and Certification/Final Delivery.

Each segment will have many activities that will be divided into six areas:

1. Activities to be conducted;

2. Types of models required;

3. Data Needed;

4. Data produced; and

5. People, KSAs, and Process consideration.

The MPM and SIM will continuously review these activities during Technical Reviews.

3.4.3 Detailed Models

The third element is the detailed level mechanical, electrical, software, and test models that will instantiate and validate the design.

3.4.4 Design and Manufacture Release

The final element is the design and manufacture release afforded by the insight and oversight available in the collaborative environment. A series of integrated test events will validate the system for delivery.

3.5 Agile and Secure Development and Operations (DevSecOps)

Agile and DevSecOps are two commercial development paradigms being adopted by DoD. Their growing promotion and use are predicated on an acknowledgement that technology and capability requirements are accelerating faster than previous standardized software and system-engineering processes can adapt, and that there are often competing priorities among developers, cybersecurity, operations, and maintenance entities. Though originally envisioned for software development efforts, inherent practices from each methodology can be applied to software, hardware, or combined hardware and software solutions to create efficiencies and enhance effectiveness; they are also not mutually exclusive, and can be used concurrently in a development program.

Agile is an iterative and incremental development approach, which implements a viable subset of the overall system’s required capabilities in smaller, more frequent releases. Each iteration includes planning, building, and testing tasks that validate and verify the deliverable against requirements. At the end of each iteration, the customer is able to see the result and determine if they are satisfied, which will then determine whether re-work or advancement is a priority in follow-on iterations.

Building on the principles of Agile, DevOps emphasizes direct collaboration between system administrators (operations) and development teams to ensure that operational requirements and constraints are accounted for during the development process. DevSecOps is an evolution of DevOps and represents a cultural shift toward security consciousness across the DevOps community.

DevSecOps emphasizes the shared responsibility of application and infrastructure security, while embracing automated security testing to enable continuous application deployment. The core idea is to create a repeatable, reliable, and incrementally improving process for taking software from concept to customer. The development and operations teams work conjointly in an Agile framework to close the traditional communication and collaboration gap between the two teams.

The DevSecOps pipeline is the conduit that connects the stakeholders, field activities, and design agents throughout the lifecycle, enabling continuous delivery of software and artifacts to analysts, testers, test sites, and Fleet operators as quickly and efficiently as possible. The DevSecOps pipeline is a workflow across development, test, and deployment environments that contains a set of engineering tools enabling collaboration amongst stakeholders, and which provides visibility into product development status. These tools facilitate modeling, artifact reviews, baseline management reviews/boards, configuration management (CM), requirements verification and traceability, software development and delivery, analysis, monitoring of metrics, and data exchange.

Implementing Agile and DevSecOps can allow the acquisition community to more rapidly acquire and validate the delivery of secure, usable capabilities and technology, while allowing for upgrades, enhancements, and/or re-prioritization of requirements throughout the process. However, programs must be aware of and prepared for the close and continual interaction required to fully realize the benefits of leveraging these methods. For example, in MBSE contract deliverable artifacts can be reviewed virtually and SETRs can be and should be coordinated through live collaboration using the models further streamlining the process.

Additional information and notional reference frameworks for Agile project management and DevSecOps implementation is included in Appendix F.

3.6 Program Models and Technical Reviews for Existing PEO IWS Programs This section describes three program models from reference (e) and Technical Review options available to the MPM and SIM for application/customization. Per reference (f), the In-Service Review (ISR) is required after Full Operational Capability in the sustainment period of the lifecycle.

Figure 1 is a model of a Hardware (HW) and Software (SW) Development Program in which development of Navy-unique HW is required, such as in a major weapons system. It represents what is typically done in the past for most programs in PEO IWS. The lower level reviews annotated with a lower case “i” are incremental subsystem reviews that can be conducted at the component, subsystem or assembly levels depending on the complexity and technical risk of the program.

Figure 1: Hardware and Software Development Program

Figure 2 and Figure 3 are sample models of Agile SW development when a program incrementally implements a capability, and reviews multiple builds at a varying level of maturity within the same technical review. Some of the PEO IWS programs, which are early adopters of Agile, are utilizing this approach.

Figure 2: Software Development Program

Figure 3: Sample Software Increment Review (SWIR) Topics

Figure 4 is a model of an accelerated acquisition program used when a potential adversary’s capability necessitates Navy leadership to endorse a higher risk acquisition approach in which cost and schedule considerations dominate over technical risk. This model compresses, or eliminates, phases of the acquisition process and accepts risk in order to expedite a deployable capability through concurrent design phases. Technical Reviews need to be completely customized to suit the cost and schedule constraints for the specific accelerated program.

Figure 4: Accelerated Acquisition Program

3.7 Roles and Responsibilities

Participants assigned to support a Technical Review shall be familiar with the key positions, teams, and types of events. Technical Reviews are co-chaired by the MPM and SIM. Other co-chairs can be added as required. The co-chairs shall work together to plan, manage, execute, and document the events, and shall share information freely in a timely manner with each other.

3.7.1 Major Program Manager and System Integration Manager

MPM and SIM roles and responsibilities for the execution of a Technical Review include the following:

a) Oversee and co-chairs of the process providing tasking, funding, technical oversight and stakeholder participation.

b) Plan and prepare events in a timely and effective manner.

c) Identify, allocate, and fund resources necessary to accomplish the total effort, including identification of supporting functions such as facilities and infrastructure.

d) Establish and concur with program Technical Review Team (TRT) membership and ensure stakeholder organization representation.

e) Decide the documentation that will be matured as a part of the SETR process focusing on the appropriate types of documents for each phase/review. This shall only include those documents critical for SE, for example: the system and subsystem specifications, performance/system trade studies, studies/analysis that defined the requirements, design artifacts and any supporting analyses, testing results, the Acquisition Strategy, Systems Engineering Plan, Test and Evaluation Master Plan (TEMP), and the Life-Cycle Sustainment Plan (LCSP). All of the statutory and regulatory documentation required for ACAT programs shall be reviewed by the TRT scheduled to support the gate review and do not have to be reviewed through the SETR process.

f) Facilitate participation of Subject Matter Experts (SMEs) in the process, such as early distribution of program materials to be reviewed and meeting announcements, as well as attention to scheduling convenient meeting locations.

g) Provide specific constructive feedback and viable recommended technical options during program execution as risks and issues arise.

h) Discuss impacts of recommendations that shall affect cost, schedule, and/or performance of the program.

i) Consistently tailor event-driven entrance and exit criteria to align with good engineering judgment, program complexity, and risk levels.

j) Schedule events consistent with availability of supporting materials and adequate time for the TRT to review those materials.

k) Establish and concur on areas of interest to focus the assessments.

l) Plan to conduct incremental assessments, as needed.

m) Confirm that HW, SW, human, safety, risk and cybersecurity areas of interest are assessed with appropriate priorities.

n) Approve the agenda, panel membership, and other participants, including additional co-chairs, as appropriate.

o) Initiate, submit and/or approve the TRAP, SEP and SE MFR.

p) Capture AIs, monitor closure of the event, and provide an event report.

q) Approve final closure of action items from the Technical Reviews.

r) Agree on findings.

3.7.2 Technical Review Panel

The Technical Review panel is composed of MPM, SIM, other co-chairs, and other selected panel members designated by the co-chairs. Participation across the Naval Enterprise is required to ensure that the combat systems and elements for integration on Navy platforms are accomplished in a timely, cost-effective, and quality manner. Members of the panel will include TWHs, program office personnel, warfare center SMEs, academia SMEs, and industry as appropriate. All co-chairs shall participate in the planning, execution, and follow-up activities, and shall keep all panel members apprised of program health, including risks and issues. It is desired that the composition of the panel will be the same throughout an acquisition phase. This provides for continuity and consistency of reviews. If there is a change in an individual panel member, it is the responsibility of the respective organization to provide for an appropriate replacement in a timely manner, and obtain concurrence from the co-chairs. Notional panel membership is depicted in Figure 5. The MPM and SIM can both delegate chairperson responsibility if schedule conflicts occur.

The Technical Review panel shall review, process, and approve all AIs during panel assessments and deliberations and designate which are critical and non-critical. Generally, critical actions are required to be closed prior to the entering of a subsequent SE event (e.g. Systems Requirements Completion). However, the MPM and SIM may choose to accept the risk of the open action(s) and move to the next event. It is more important to manage the risk associated with the action than it is to strictly follow the process.

Figure 5: Technical Review Panel

3.7.3 Technical Review Team

The TRT is a standing body receiving tasking, funding, and technical oversight from the Technical Review co-chairs; it is responsible for assessing the technical health of the program, ensuring the entrance criteria shall be met, and that the risk is well characterized and understood. Membership consists of program office personnel, TWHs, warfare center personnel, academia, and industry leveraging the program’s IPTs to the best extent possible. The team and the TRT chair is established by the MPM and SIM at the program inception to ensure the program has expertise to adequately cover the technology being developed. TRT membership shall be updated as necessary with additional technical experts/stakeholders as the program matures.

The TRT shall conduct its review while the technical artifacts are still under development to verify the artifacts conform to requirements and standards, to identify, record, and remove defects, and to identify and address any existing design and/or development issues. For programs using MBSE the artifacts would be included in the model and the TRT will conduct its review live using the models. The

TRT shall participate during the planning, execution, and documentation of the Technical Reviews, and shall keep the co-chairs informed in a timely manner of all risks and issues, providing viable alternatives and mitigation plans. For most programs, TIMs are meetings and/or forums where the TRT/IPT members can interact, hold technical discussions, review supporting documentation, seek answers to technical questions, identify any potential issues, and prioritize follow-on topics.

It is recommended that the TRT remain involved throughout the development, assessing progress and reviewing artifacts as they become available. A series of TIMs between TRT SMEs and the system developers may be conducted to discuss technical issues and reach consensus prior to a Technical Review. Incremental artifact review issues may also be discussed at the TIMs. Once the TRT has completed their assessment of the program, they shall provide the panel, along with the prime developer or integrator, with an assessment of readiness to proceed to the review.

3.8 Planning for Technical Reviews

The MPM and SIM are responsible for planning Technical Reviews, and all parties have responsibilities that must be met satisfactorily to realize a successful outcome from the review.

Although no single planning approach is mandated, the critical focus for all parties involved must remain on demonstrating and verifying the successful achievement of the established entrance and exit criteria.

The TRT shall assess progress toward meeting all entrance criteria in advance of the review and to brief the co-chairs on shortcomings, deficiencies, risks and associated mitigation plans. For all entrance criteria that will not be met by the review, the co-chairs shall activate mitigation plans immediately to avoid a rescheduling of the review. If the co-chairs deem the risk to be acceptable, then the review may be conducted as scheduled. The co-chairs must inform the panel and TRT members of their decision on whether or not to proceed if all entrance criteria are not met.

3.8.1 Technical Review Action Plan (TRAP)

The TRAP is prepared early in the planning cycle to define roles and responsibilities, schedule, artifacts, and entrance and exit criteria. The plan may be a standalone document, or the information required in a TRAP may be included in the SEP. The MPM and SIM shall decide on TRAP/SEP documentation when they agree on how SE processes and standards will be customized for the program.

Once the plan has been developed and concurred by appropriate stakeholders, it is then approved by the co-chairs. A template is included in Appendix D.

3.8.2 Agenda

The Technical Review agenda contains elements that correlate to technical accomplishments within the pre-established areas of interest and evidence of successful fulfillment of entrance and exit criteria for the review. The program’s TRT/IPT are expected to present these elements with supporting documentation and products.

Planning the review agenda involves assessing the status and complexity of the areas of interest and technical issues that must be addressed during the meeting. The agenda shall focus on the technical aspects of the system, including issues and risks. The content of the agenda shall provide the panel with sufficient and appropriate amount of development and design detail so that the panel is able to fully assess the technical maturity and risk level of the program.

The agenda should be made available to the Technical Review Panel notionally 30 days prior to the review, or otherwise agreed to by the MPM and SIM. The agenda should be documented in the announcement letter.

3.9 Conducting Technical Reviews

All programs shall conduct Technical Reviews in support of the SETR process, and they shall be tailored dependent upon the scope and complexity of the program. These reviews are jointly chaired by the MPM and SIM. It is important to set the duration of the review appropriately to ensure adequate time is planned for a complete review of the depth and breadth of the program. Figure 6 depicts the typical review process.

Figure 6: Typical Technical Review Process

3.9.1 Readiness Assessment Meeting

The primary focus of the meeting is for the co-chairs to decide if the program is ready to have the Technical Review as scheduled. It should be scheduled at a time and place that allows ample time for the team to fully assess the program. The TRT leads shall prepare entrance criteria compliance assessments, and clarify any and all technical cost or schedule issues impacting the program. The prime developer or integrator shall also present entrance criteria compliance assessments and clarify any and all technical, cost, or schedule issues impacting the program. Any entrance criteria that are not met are ordinarily grounds to delay a Technical Review. The co-chairs can decide to proceed with the review if they agree that the entrance criteria are highly likely to be satisfied prior to commencement of the event.

As a good practice, it is recommended the panel members receive notification of any actions put in place before the review.

3.9.2 Technical Review

The primary focus of the Technical Review is for the panel to assess development progress, achievement of the entrance and exit criteria, and technical challenges given the cost and schedule constraints on the program. The co-chairs shall ensure the resulting AIs are appropriate and assigned properly.

AIs are the primary method for the co-chairs to address the program’s challenges, so it is important to have each submitted action clearly written and executable before being accepted by the panel. A formal AI process is important during the conduct of assessments. Formalizing the AI process keeps focus on the identification and resolution of issues critical to completion of the review. The panel shall review, process, and approve all AIs, and the co-chairs shall determine the criticality of the AI relative to the closure of the Technical Review in progress. All identified AIs that do not support the closure criteria for the review will be dispositioned at the discretion of the co-chairs. A sample action item form can be found in Appendix E.

3.9.3 Technical Review Closure

AIs will be addressed and adjudicated with the originator before being presented to the co-chairs for closure. Critical AIs must be closed before the closure of the Technical Review. A Technical Review is closed when all established exit criteria are met, all critical AIs have been closed, and a letter designating closure is signed by the MPM and SIM.

3.9.4 Technical Review Summary Report

As defined in reference (c), the Technical Review Summary Report (TRSR) for each review shall be prepared and signed by the MPM and SIM, and may be distributed by the MPM to the PEO, NAVSEA 05H Director, and Milestone Decision Authority, as appropriate for the program. The TRSR describes the outcomes of the review, including the following:

a. List of panel members, including name and functional area represented.

b. Meeting minutes, including entrance criteria status, exit criteria status, risks and review results.

c. Recommendations pertinent to the technical health of the program and its readiness to enter the next phase of development.

d. List of all critical and non-critical AIs, originator, assignees, and due dates.

A-1

Appendix A Acronyms AA Accelerated Acquisition ACAT Acquisition Category AI Action Item AoA Analysis of Alternatives ASN(RDA) Assistant Secretary of the Navy for Research, Development, and Acquisition ATO Authority to Operate ATR Above Threshold Reprogram BQT Build Qualification Test BTR Below Threshold Reprogram CAE Component Acquisition Executive CBM+ Condition Based Maintenance Plus CDD Capability Development Document CDR Critical Design Review CHENG Chief Engineer CI Configuration Item CIDS Critical Item Development Specification CM Configuration Management CONEMP Concept of Employment CPD Capability Production Document CSC Computer Software Component CSCI Computer Software Configuration Item CSS Cybersecurity Strategy CSU Computer Software Unit DevSecOps Secure Development and Operations DMAP Data Management and Analysis Plan DMSMS Diminishing Manufacturing Sources and Material Shortages DoD Department of Defense DoN Department of the Navy DRM Design Reference Mission DRMP Design Reference Mission Profile FCA Functional Configuration Audit RFD Fault Detection

A-2

FDR Focused Design Review FI Fault Isolation FMECA Failure, Modes, Effects, and Criticality Analysis FoS Family of Systems FRACAS Failure Reporting, Analysis, and Corrective Action System FRP Full-Rate Production HSI Human Systems Integration HW Hardware ICD Initial Capabilities Document or Interface Control Document IDD Interface Design Description IDR Integration Design Review IMP Integrated Master Plan IMS Integrated Master Schedule IPR In-Process Review IPT Integrated Product Team IRS Interface Requirements Specification ISP Information Support Plan ISR In-Service Review IWS Integrated Warfare Systems KPP Key Performance Parameter KSA Key System Attribute LCSP Lifecycle Sustainment Plan LRIP Low-Rate Initial Production M&S Modeling and Simulation MBSE Model Based Systems Engineering MDA Milestone Decision Authority MFR Memorandum for the Record MOP Measures of Performance MP Metrics Plan MPM Major Program Manager MRA Manufacturing Readiness Assessment MRL Manufacturing Readiness Level MRR Mission Readiness Review

A-3

NAVSEA Naval Sea Systems Command NAVSEA 05 Naval Systems Engineering Directorate NAVSEA 05H Naval Systems Engineering Directorate, Integrated Warfare Systems

Engineering Group OPSEC Operational Security PCA Physical Configuration Audit PDR Preliminary Design Review PEO Program Executive Office/Officer PHM Prognostics Health Management PIDS Prime Item Development Specification PM Program Manager POM Program Objective Memorandum PPP Program Protection Plan PR Program Review PRR Production Readiness Review QAP Quality Assurance Plan RAM-C Rel iab i…