Exhibit_9_The_Marine_Corps_Systems_Command_Technical_Review_Handbook_Amend_0001.pdf

TECHNICAL REVIEW HANDBOOK

Version 1.04

April 2009

Technical Review Handbook

Executive Summary

The Engineering Competency Leadership Board prepared this document to assist Program

Managers and improve the application of Systems Engineering principals throughout Marine

Corps Systems Command. This handbook provides guidance on the preparation and conduct of technical reviews throughout a program’s lifecycle.

Technical reviews are conducted during development and verification of a new system or system baseline upgrade to demonstrate that required tasks have been successfully completed before proceeding beyond critical events. This ensures a disciplined and orderly approach to the development and verification process. Technical reviews are an integral part of the systems engineering process and are consistent with existing and emerging commercial standards. Tailoring of each review ensures that the emerging design is ready to enter the next development stage. Reviews should be event-driven and conducted when the system's design is ready for review of progress in accordance with appropriate technical plans.

Accordingly, for each review, entry and exit criteria tied to the required level of design maturity must be defined and certified and they must be applied across all requirements and technical disciplines. Technical reviews will be chaired by an independent Chairperson appointed by the Deputy Commander Systems Engineering, Interoperability, Architectures and Technologies for all programs. Program Managers may co-chair or assign a review co-chairperson if desired. The technical review will also make maximum use of subject matter experts from outside the Program Team.

This Technical Review Handbook describes the purpose, timing, entrance and exit criteria for each technical review in the overall technical review process. Program Managers have the flexibility to tailor reviews to the particular circumstances of each program. Each Program

Manager will develop and deliver a Systems Engineering Plan that describes how the program intends to tailor the technical reviews and how the tailored review activities accomplish the objectives of the overall technical review process.

Table of Contents Executive Summary 1 Introduction

1.1 Purpose

1.2 References

1.3 Technical Review Background

1.4 Objectives

1.5 Recommended Technical Reviews and Levels

1.6 Timing

2 Technical Review Planning

2.1 Technical Review Action Plan

2.1.1 Tailoring

2.1.2.1 Technical Review Board Chairperson

2.1.2.2 Program Lead Engineer

2.1.3 Entrance Criteria

2.1.4 Technical Review Elements and Agenda

2.1.5 Exit Criteria

2.2 Staffing and Technical Review Preparation

3 Conduct of the Technical Review

3.1.1 Closing the Technical Review

3.1.2 Technical Review Summary Report

4 Request For Action Process Appendix A: Acronyms Appendix B: Document Definitions Appendix C System Requirements Review (SRR) C-1 Overview C-2 Timing C-3 SRR Entrance Criteria C-4 SRR Review Elements C-5 SRR Exit Criteria Appendix D System Functional Review (SFR) D-1 Overview D-2 Timing D-3 SFR Entrance Criteria D-4 SFR Review Elements D-5 SFR Exit Criteria Appendix E Preliminary Design Review (PDR) E-1 Overview E-2 Incremental PDRs E-3 Timing E-4 PDR Entrance Criteria E-5 PDR Review Elements E-6 PDR Exit Criteria Appendix F Critical Design Review (CDR) F-1 Overview F-2 Software CDRs F-3 Incremental CDRs F-4 Timing F-5 CDR Entrance Criteria F-6 CDR Review Elements F-7 CDR Exit Criteria Appendix G Test Readiness Review (TRR) G-1 Overview G-2 Timing G-3 TRR Entrance Criteria G-4 TRR Review Elements

G-5 TRR Exit Criteria Appendix H System Verification Review (SVR) H-1 Overview H-2 Timing H-3 SVR Entrance Criteria H-4 SVR Review Elements H-5 SVR Exit Criteria Appendix I Production Readiness Review (PRR) I-1 Overview I-2 Timing I-3 PRR Entrance Criteria I-4 PRR Review Elements I-5 PRR Exit Criteria Appendix J Physical Configuration Review (PCR) J-1 Overview J-2 Timing J-3 PCR Entrance Criteria J-4 PCR Review Elements J-5 PCR Exit Criteria Appendix K: Sample RFA Form

1 Introduction

1.1 Purpose

The Engineering Competency Leadership Board prepared this document to assist Program

Managers and Project Officers with the application of Systems Engineering principals throughout Marine Corps Systems Command (MCSC) and Program Executive Officer (Land

Systems) (PEO-LS) programs. This handbook is directive for all programs within MCSC and PEO-LS.

This Technical Review Handbook (TRH) identifies technical requirements, planning responsibilities, recording requirements, and roles for technical reviews. This handbook also describes the objectives and activities (agenda, entry and exit criteria, etc.) for each technical review as well as the overall technical review process. The technical review should not be limited to an assessment of the technical products; it should also assess the programmatic readiness of the development and testing efforts that lie ahead.

1.2 References

This handbook does not attempt to describe the full Systems Engineering process. The focus of this handbook is on the technical review process as described in one or more of the following:

Defense Acquisition Guidebook, 24 Jul 2006

Defense Acquisition University Systems Engineering Community of Practice, <acc.dau.mil/se>

NAVAIRINST 4355.19C Systems Engineering Technical Review Process, 10 Apr

NAVAIR Systems Engineering Technical Review Handbook, 10 Apr 2006

Marine Corps Systems Command Develop and Demonstrate Process Handbook and

Quick Tips, version 2, Jul 2004

Defense Systems Management College Systems Engineering Fundamentals, Jan 2001

In addition, this handbook derived many of its review elements and entrance and exit criteria from MIL-STD-1521B (USAF) Military Standard Technical Reviews and Audits for

Systems, Equipments, and Computer Software. Although this MIL-STD has been cancelled, program managers may find it a useful resource.

1.3 Technical Review Background

Technical reviews provide visibility into the developer’s implementation of the work effort required under the terms of a contract’s Statement of Work to assure timely and effective attention to the technical interpretation of contract requirements. As a system progresses through design and development, the system-level technical reviews are generally timed to correspond with transition from one stage of development to another. Technical reviews are the events at which the Program Manager verifies that the technical maturity of the system or item under review is sufficient to justify commitment of resources to go to the next stage of development.

As the system/product progresses through the development process, the focus of the technical assessment changes. Early in the process, the primary focus is on defining the requirements that drive design and development activities. Technical reviews conducted during the early stages of development usually focus on ensuring that top-level concepts and system definitions reflect user requirements. Once system-level definition is complete, the focus is on design at sub-system levels and below.

Technical reviews during these stages are typically design reviews that establish design requirements and then verify that physical solutions are consistent with those requirements.

Schedule technical reviews and audits at strategic, event-driven points representing a transition in design focus or phase. Describe the establishment and tailoring of technical review schedules in the program's Systems Engineering Plan (SEP).

1.4 Objectives

Assess system design progress and maturity at key development stages that are event-driven points in the acquisition development schedule. Compare the design to pre-established exit criteria for the particular event to determine if the appropriate level of maturity has been achieved. These events are the technical reviews and audits. The objectives of these reviews are to:

Ensure that the results of trade studies used to define concepts and risk associated with alternatives have been analyzed.

Assess the system requirements and allocations to ensure that requirements are unambiguous, consistent, complete, feasible, verifiable, and traceable to top-level requirements.

Assess the design maturity based on technical development goals, systems engineering events and accomplishments, and empirical test data supporting progress to date.

Confirm that the effects of technical risk on cost, schedule, and performance, as well as risk reduction measures, rationale, and assumptions made in quantifying the risks have been addressed.

Demonstrate that the relationships, interactions, interdependencies, and interfaces between required items and externally interfacing items, system functions, subsystems, and system elements, as appropriate, have been addressed.

Ensure performance, functional, design, cost, and schedule requirements and objectives, technical performance measurements, and technical plans are being tracked, are on schedule, and are achievable within existing programmatic constraints.

Confirm that continued development is warranted, and when it is not, that executable alternatives have been defined (discontinue development, or take corrective action on the products and/or processes of the current application before proceeding to the next application).

Verify the system is ready for testing or production.

Identify resources (people, funding, support assets, test facilities, etc., as appropriate) required for continued development, testing or production.

Precede formal technical reviews by a series of technical interchange meetings where the developer and the Government identify and discuss issues, problems and concerns, and define solutions to mitigate these problems. The formal technical review is not the place for problem solving, but for verifying that problem solving has been done and the approved system engineering process has been followed. Figure 1. summarizes the technical review event process.

Figure 1: Technical Review Event Process

1.5 Recommended Technical Reviews and Levels

Before During After

Plan

Familiarize

Pre-review

Review

Resolve

Follow-up

• Identify participants

• Assign roles and tasks

• Establish guidelines and procedures

• Establish and use entry criteria

• Establish exit criteria based on the event-driven schedule

• Have overview meeting

• Review entrance criteria

• Individual and team reviews

• Examine data

• Analyze data

• Track and document analysis

• Individual and team reports on review findings

• Facilitate and pace meeting

• Examine review data and analyses, record and classify findings

• Address key issues identified by pre-review activity

• Assess severity of problems

• Identify action items

• Assess exit criteria

• Assign responsibility

• Track action items and issues

• Track action item completion trends

• Document and distribute results of review and action item completions

Technical Interchange Meetings (TIMs)

Before During After

Plan

Familiarize

Pre-review

Review

Resolve

Follow-up

• Identify participants

• Assign roles and tasks

• Establish guidelines and procedures

• Establish and use entry criteria

• Establish exit criteria based on the event-driven schedule

• Have overview meeting

• Review entrance criteria

• Individual and team reviews

• Examine data

• Analyze data

• Track and document analysis

• Individual and team reports on review findings

• Facilitate and pace meeting

• Examine review data and analyses, record and classify findings

• Address key issues identified by pre-review activity

• Assess severity of problems

• Identify action items

• Assess exit criteria

• Assign responsibility

• Track action items and issues

• Track action item completion trends

• Document and distribute results of review and action item completions

Technical Interchange Meetings (TIMs)

The following paragraph describes the recommended technical reviews for a new system or baseline upgrade. Describe the program-specific technical reviews and any needed tailoring in the program's SEP. Incorporate technical review requirements into contracts where appropriate. A good source of contracting guidance is Guide to Integrating Systems

Engineering into DoD Acquisition Contracts.

Appendices to this handbook provide suggested entrance and exit criteria as well as agenda items and timing guidance for each of the reviews listed below. Sections 2.13 and 2.15 of this document more fully addressed Entrance and Exit Criteria. The sequence of system and subsystem reviews may be altered as appropriate for the program, consistent with sound engineering practice. Section 2.1.1 discusses tailoring of technical reviews. Appendix C provides a listing of documentation that corresponds to each technical review.

System Requirements Review (SRR) (system level)

System Functional Review (SFR) (system level)

Preliminary Design Review (PDR) (both system and subsystem level)

Critical Design Review (CDR) (both system and subsystem level)

Test Readiness Review (TRR) (subsystem level)

System Verification Review (SVR) (system level)

Production Readiness Review (PRR) (system level)

Physical Configuration Review (PCR) (system level)

1.6 Timing

The schedule for technical reviews is extremely important. Conduct a review too early, and the item for review will not be adequately defined. Conversely, a late review can result in erroneous program commitments whose correction will be both difficult and costly. For planning purposes, a good method for scheduling technical reviews is to relate them to the documentation requirements. For example, schedule a PDR after the Hardware Development

Specification or Software Design Description and Software Test Plan are available, since the essence of the PDR is to assess the contractor's approach to meeting the requirements of these documents. Scheduling of reviews are dependent not only on documentation availability but also on hardware/software availability, and the completion of the acceptance qualification tests. Figure 2 illustrates where various technical reviews typically occur during the acquisition life cycle.

Figure 2. Systems Engineering Technical Review Timing

2 Technical Review Planning

Reviews should consider the state of the developing product analyses, the ability to test, produce, and support the product, and the ability to train operators to use the system. The developer and the government should have the same expectations regarding the content and the outcome of the review. Important considerations for planning include the following:

Timely and effective preparation for the review

Identifying and allocating resources necessary to accomplish the total review effort

Tailoring of the review consistent with program risk levels

Scheduling consistent with availability of appropriate data

Establishing event-driven entrance and exit criteria

Establishing boundaries to define the review’s content

Using appropriate incremental reviews

Reviewing all systems functions

Confirming all system elements are integrated

2.1 Technical Review Action Plan

Prior to the technical review the Program Team will prepare a Technical Review Action

Plan. It is an important tool to manage the expectations of external stakeholders and it clearly communicates to the contractor team the government’s expectations and success criteria. The Action Plan must be consistent with the technical review approach defined in the program’s Systems Engineering Plan.

The Technical Review Action Plan should contain the following:

Program Background

Tailoring of technical review guidance (Section 2.1.1)

Schedule of events leading to the technical review (including incremental sub-systems technical reviews and readiness assessments)

Technical review Board Membership and technical review Participants (Section

2.1.2)

Entrance Criteria (to include closeout of Request For Actions (RFA) from previous reviews and having met exit criteria of previous reviews) (Section 2.1.3)

Proposed Agenda, elements to be reviewed, and boundary conditions (Section 2.1.4)

Exit Criteria (Section 2.1.5)

The RFA process that will be used to document the technical review’s action items.

(Section 4)

2.1.1 Tailoring

The complexity of the System, Subsystem, or configuration item and the type of program are central in determining both the need for and the number of reviews. When developing a small non-complex system some reviews may not be required, or, if required, may be limited in scope. Conversely, in a very complex development the review process will increase in levels and numbers of reviews. The tailoring of reviews enables the Program Manager (PM) to focus on an appropriately scoped technical review.

Tailor each technical review’s agenda, entrance and exit criteria in accordance with the technical scope and risk of the system; described any tailoring in the program’s SEP.

2.1.2 Technical Review Board Mission, Membership and Participants

The Technical Review Board (TRB) is the government body formed to conduct the technical review to the objectives shown in Section 1.4 and the exit criteria developed during technical review planning (see Section 2.1.5). The TRB advises the TRB Chairperson, or Co-chairpersons, who makes a recommendation(s) to the Program Manager on whether the program is ready to proceed to the next stage of development (see Section 3.1.2).

Note

TRB is the generic name that applies to all of the various reviews listed in Section 1.5. In practice, the review board is known by the name of the review (e.g. the PDR Board, the CDR

Board, etc.). The TRB and the event (PDR, CDR, etc.) are not separate.

TRB Membership (typical composition):

Technical Review Board Chairperson or Co-chairpersons (see Section 2.1.2.1)

Competency Lead Engineers, as appropriate. When there are interfacing or supporting systems the appropriate Competency Lead Engineers will be added to the TRB.

Program Manager or representatives (Industry and Government)

The program’s Lead Engineer, The program’s Lead Logistician, who should ensure all relevant supportability requirements are addressed

Cost Team representative, if required

Counsel, if required

Contracting Officer, if required

Recorder; who is responsible for collating RFAs for submission to the TRB. The recorder should have the Technical Review Summary Report (section 3.1.2) prepared for distribution by the Chairperson

Other participants that are not part of the TRB but who represent stakeholders or who support the TRB:

Resource Sponsor (Requirements Officer from Marine Corps Combat Development

Command (MCCDC)

User representatives

Marine Air-Ground Task Force (MAGTF) and Joint Integration Division within

Systems Engineering, Interoperability, Architectures and Technology (SIAT)

Independent Subject Matter Experts (SMEs) as required to address system concepts and enabling technologies. These SMEs represent their MCSC functional areas in the adjudication of RFAs, to include cost and schedule impacts. These assignments should be coordinated with engineering authority. SMEs should be notified at least

30 days prior to the scheduled review date

Developmental and Operational testers (DT/OT)

Integrated Product Team (IPT) member/briefer in accordance with the technical review agenda

Best Practice

The participation of SMEs from outside the program team is important to a successful

Technical Review. SMEs from interfacing systems are especially important. To convince senior leaders that the proper SMEs will participate, the Technical Review Action Plan should list, by name, the SMEs that will attend the Technical Review. A brief synopsis of each SMEs experience and qualifications should also be included.

2.1.2.1 Technical Review Board Chairperson

Program Executive Officer – Land Systems (PEO-LS), Product Group Director (PGD), and

Independent Program Managers (IPM) will request that the Deputy Commander, Systems

Engineering, Interoperability, Architectures and Technology (DC SIAT), as

MARCORSYSCOM CHENG, assign a TRB Chairperson who is independent of the program team. On programs where Commander, MARCORSYCOM had delegated Milestone

Decision Authority (MDA) to a lower level, the CHENG may assign the Competency Lead

Engineer (CLE) as the TRB Chairman. The Program Manager may co-chair or assign the program’s Lead Engineer or another individual to serve as a Co-chairperson. The role of the chairperson includes:

Review and approval of the Technical Review Action Plan, which includes:

1. TRB membership (see section 2.1.2) based on the program’s content

2. Final review elements (see section 2.1.4)

3. Entrance and exit criteria (see sections 2.1.3 and 2.1.5)

Oversight of the technical review and RFA process (see section 4) as documented in the program’s SEP and the Technical Review Action Plan

Issuance of the Technical Review Summary Report (see section 3.1.2).

2.1.2.2 Program Lead Engineer

The role of the program lead engineer includes:

Ensure the performing activity provides the supporting data and participation in the required review

Develop, coordinate, and execute, in cooperation with the performing activity, individual review arrangements

Ensure that all program documentation relevant to the technical review is coordinated across the IPTs

Conduct the review for the TRB

2.1.3 Entrance Criteria

Establish entrance criteria early, prior to the scheduled review. This ensures that the review is event driven, not schedule driven. Derive the criteria from the program plan and capture the work that must be accomplished during the phase preceding the review. Entrance criteria have a quantitative characteristic, measuring accomplishments. “Reverse Engineering” the criteria at the last minute based on what has been accomplished defeats the purpose of the technical review. Entrance criteria answer the question, “Has the developer completed those things that were planned to be accomplished?” Start the technical review with a review of the developer’s response to the entrance criteria. For each entrance criteria, what objective evidence has the developer submitted to demonstrate that the criteria has been satisfied?

Address how qualified, impartial, reviewers have evaluated the evidence. Example entrance criteria for each technical review are included in the corresponding review’s appendix to this handbook.

2.1.4 Technical Review Elements and Agenda

The program’s lead engineer and the assigned Chairperson shall coordinate the development of a preliminary agenda containing elements that correlate to the established exit criteria.

Make this agenda available to the TRB participants 30 days prior to conduct of the review.

In addition, the program team needs to decide what products (i.e., design documents, test plans and reports, management plans, and risk assessments) will be reviewed prior to the technical review. These products may consist of items such as, but not limited to, any or all of the following: design documents, analysis results, test plans and reports, management plans, and risk assessments as necessary to provide a comprehensive overview of work accomplished to date. A list of the products to be reviewed should be included in the

Technical Review Action Plan. Appendices to this handbook contain sample review elements for each of the technical reviews.

2.1.5 Exit Criteria

While entrance criteria have a quantitative characteristic, measuring accomplishments, exit criteria will answer the question, “Is the accomplished work of sufficient quality and completeness to warrant moving forward to the next phase?” Coordinate the exit criteria with the developer and keep them within scope of the development effort. Setting unrealistic criteria may doom the technical review to failure. Likewise, setting criteria that do not adequately measure readiness to proceed adds risk to the program in the long run. Example exit criteria for each technical review are included in the appendices of this handbook.

Best Practice

Write a proof statement for each exit criteria then map the exit criteria to the agenda to ensure that each is covered during the technical review. For example, for a CDR exit criteria, a proof statement and the associated agenda items might read:

Exit Criteria: “Is the design capable of meeting the performance requirements?”

Proof Statement; “The contractor will brief the flow-down of requirements from the contract specification to the allocated baseline and will include the current program manager’s estimate of performance against the contract specification.”

Agenda Items Supporting this criteria: Mr. Smith’s briefing on requirements flow-down process, Ms. Jones’ briefing on the current performance estimates

2.2 Staffing and Technical Review Preparation

Start planning the technical review well in advance of the event. Project Officers should work with the developer, the Competency Lead Engineer, and other members of the program team to develop the Technical Review Action Plan. At the same time, Program Management should request the designation of a Technical Review Board Chairperson. Once a working group prepares an initial draft, it should be formally coordinated with the Chairperson and other stakeholders. Also send the Technical Review Action Plan to all members of the technical review board (see section 2.1.2). Figure 3 shows a typical timeline for this staffing process.

At least 45 days prior to the event, assess the entrance criteria and the overall preparation.

Do this to ensure that sufficient progress is being made toward meeting the published entrance criteria. For unmet entrance criteria, put mitigation plans in place and review them as part of technical review.

Figure 3. Planning Timeline

3 Conduct of the Technical Review

Just as no two programs are alike, no two technical reviews will be conducted the same way.

For some complex programs, the technical review may be broken down into incremental reviews where SMEs from various disciplines each have a chance to review the appropriate part of the program. For other, less complex, programs a single review over a few days may be more appropriate. The PM may want to have smaller groups, or breakout sessions, examine specific risk areas and then have a general, executive-level, session to receive the smaller group’s findings and to review the program’s readiness to proceed. There is no single approach to conducting a technical review.

No matter what approach is chosen, the critical focus must remain on demonstrating achievement of established exit criteria. There should be a direct correlation between the exit criteria and the agenda topics.

Best Practice

At the start of the technical review the Technical Review Board Chairperson should set the context for the review. Firmly establishing expectations and boundaries will focus the discussion and keep attendees from revisiting decisions made at previous reviews.

Presenting a “we are here” chart showing the technical review in relation to the overall program schedule and the technical review process will also focus the attendees and provide an overall context. Do not assume that all participants have equal background knowledge of the program. Time spent at the outset to baseline everyone will pay dividends later in the review.

Best Practice

At the start of the technical review, review the entrance criteria to justify the program’s readiness to hold the technical review.

Best Practice

At the beginning of the technical review the exit criteria should be briefed and each subsequent briefing should explicitly state which exit criteria is being addressed.

Best Practice

Brief the participants on the RFA process that will be used to document action items (see section 4) and provide hardcopies of the RFA forms.

3.1.1 Closing the Technical Review

In general, a technical review is considered closed when the event is complete, established exit criteria have been met and all critical RFAs have been closed (see section 4 for a discussion of Requests For Action). Obtain concurrence of the TRB Chairperson, or Co-chairpersons, with a formal determination that the technical review is closed. These closeout criteria usually form the first two entrance criteria for the next technical review.

If exit criteria remain unmet at the end of the event, the Technical Review Board Chairperson

(see section 2.1.2.1) must decide whether to address them at a future meeting such as a program review, a follow-on session of the technical review, or, in extreme cases, to reconvene the technical review and start over. Whatever approach the Chairperson chooses it should become part of the recommendation passed to the Program Manager (see section

3.1.2).

3.1.2 Technical Review Summary Report

At the conclusion of the technical review, the Recorder prepares a Technical Review

Summary Report. This report documents the outcome of the review and should include the following items:

List of attendees, to include; name, functional area represented, phone number, and email address

Meeting minutes, including Entrance Criteria status, Technical Review results and

Exit Criteria status

Completed RFA forms

The TRB Chairman’s recommendation to the Program Manager on the technical readiness of the program to enter the next phase of development

Forward the Technical Review Summary Report and the technical review results to DC, SIAT, Assistant Commander, Programs (ACPROG), the Competency Lead Engineer, the

Program Manager, and the Milestone Decision Authority, within 30 days of completion of the review.

4 Request For Action Process

The use of a formal Request For Action/Request for Information (RFA/RFI) process is important for the orderly conduct of any technical review. While action items are routinely assigned during program team meetings and are worked by team members, the technical review requires a more formal process. The large number of people who sometimes attend these reviews can inundate the program team with action items and requests for additional information; not all of which are critical to the completion of the review. By formalizing the action item process, many of these spurious actions are avoided and the critical issues are identified.

RFAs are used by SMEs reviewing documentation prior to the events as well as during the event. In the Technical Review Action Plan discuss how RFAs will be gathered from SMEs reviewing documents prior to the event as well as during the event. If the contractor will comment on these RFAs before the technical review, include this process in the Technical

Review Action Plan.

Brief the use of RFAs/RFIs to the technical review participants at the outset of the technical review. This briefing should include the process for submittal, evaluation, tracking and closeout. Appendix K shows a sample RFA/RFI form. Make hardcopies of this form available to all technical review participants.

The Technical Review Board Recorder is responsible for collecting all submitted RFAs/RFIs.

The Technical Review Board, usually in an executive session, evaluates, accepts or rejects, and classifies all RFAs/RFIs. Assign submitted RFAs/RFIs to one of the following classifications:

Critical RFA. In identifying a RFA as critical, Technical Review Board Members must apply reasonable judgment as to whether failure to complete a specific action

(e.g., resolving a predicted performance shortfall) is serious enough to prevent closure of the review. Examples of RFAs that may be critical include those associated with missing or incomplete design definition, performance shortfalls, design deficiencies, incomplete performance assessments, failure to satisfy exit criteria/objectives, design not reproducible, insufficient risk mitigation, or other required design information deemed mandatory to proceed into the next phase. All Critical RFAs must be closed before formally closing out the technical review.

Non-Critical RFA. Non-critical RFA’s are those not considered mandatory for satisfaction of the technical review exit criteria. Careful scrutiny of these RFAs is required by review chairpersons to ensure the proper classification as non-critical and the assignment of the proper need dates and/or milestone.

RFI. Requests for additional information should be classified as RFIs, if they do not require the development team to perform any action other than to provide the information. If the requested information is crucial to evaluating the design or is a

Contract Data Requirements List (CDRL) item that is required for submission at the technical review, the request should be classified as an RFA.

Out of Scope/Response. Sometimes submitted requests can not be met within the contract scope or are out of bounds for the current technical review. Reject these requests.

Page 19 of 62 Appendix A

Appendix A: Acronyms

ACPROG Assistant Command, Programs

ASR Alternate Systems Review

CDD Capabilities Description Document

CDR Critical Design Review

CDRL Contract Data Requirements List

CI Configuration Item

CM Configuration Management

CPD Capabilities Production Document

CSCI Computer Software Configuration Item

CSDM Computer System Diagnostic Manual

CSOM Computer System Operator’s Manual

CRISD Computer Resources Integrated Support Document

DC SIAT Deputy Commander, Systems Engineering, Interoperability, Architectures and Technology

DoD Department of Defense

DT&E Developmental Test and Evaluation

DT/OT Developmental Test/Operational Test

ECP Engineering Change Proposal

ESOH Environmental Safety and Occupational Health

FCA Functional Configuration Audit

FoS/SoS Family of Systems/System of Systems

FRP Full Rate Production

HWCI Hardware Configuration Item

IBR Integrated Baseline Review

IDD Interface Design Description

IPT Integrated Product Team

IRS Interface Requirements Specifications

ISP Information Support Plan

ISR In Service Review

ITR Initial Technical Review

LRFS Logistics Requirements Funding Summary

LRIP Low Rate Initial Production

LSA Logistics Support Analysis

MAGTF Marine Air-Ground Task Force

MCCDC Marine Corps Combat Development Command

MC-SAMP Marine Corps Single Acquisition Management Plan

MCSC Marine Corps Systems Command

NR-KPP Net-Ready Key Performance Parameter

OPEVAL Operational Evaluation

OTRR Operational Test Readiness Review

PCR Physical Configuration Review

PDR Preliminary Design Review

PEO-LS Program Executive Office – Land Systems

PGD Product Group Director

PM Program Manager

PRR Production Readiness Review

RFA Request For Action

RFI Request For Information

SDD System Development and Demonstration

SDD Software Design Description

SDP Software Development Plan

SE Support Equipment

SEP Systems Engineering Plan

SERD Support Equipment Requirements Document

SFR System Functional Review

SIAT Systems Engineering, Interoperability, Architectures and

Technology

SME Subject Matter Experts

SRR System Requirements Review

SRS Software Requirements Specification

SSDD System/Subsystem Design Document

STP System/Software Test Plan

SUM Software User’s Manual

SV System View

SVD Software Version Description

SVR System Verification Review

T&E Test and Evaluation

TBD To Be Determined

TDP Technical Data Package

TLSU Top-Level Software Unit

TRA Technology Readiness Assessment

TRB Technical Review Board

TRH Technical Review Handbook

TRR Test Readiness Review

TV Technical View

VECP Value Engineering Change Proposal

Page 21 of 62 Appendix B

Appendix B: Document Definitions

This appendix defines the documents mentioned in this handbook.

Allocated Baseline. The approved documentation describing a Configuration Item’s (CI's) functional, performance, interoperability, and interface requirements that are allocated from those of a system or higher level configuration item; interface requirements with interfacing configuration items; and the verifications required to confirm the achievement of those specified requirements.

Capability Development Document (CDD). The Capability Development Document captures the capabilities and performance requirements information necessary to develop a proposed program(s). The CDD outlines a militarily useful and logistically supportable capability.

Capability Production Document (CPD). The Capability Production Document (CPD) is a document used to address the production elements specific to a single increment of an acquisition program.

Configuration Management (CM) Plan. The plan describes the process for establishing and maintaining consistency of a product’s performance, functional and physical attributes with its requirements, design and operational information throughout its lifecycle.

Computer System Diagnostic Manual (CSDM). The documentation needed to allow system administrator of a computer system to troubleshoot problems in a fielded system.

Computer System Operator’s Manual (CSOM). The documentation needed to operate a given computer and its peripherals.

Computer Resources Integrated Support Document (CRISD). The CRISD provides the information needed to plan for life cycle support of deliverable software. The CRISD documents the contractor's plans for transitioning support of deliverable software to the support agency.

Database Design Document. The basis for database implementation and maintenance, including data base design; related data, files, and SW/data base management system for access.

Firmware Support Manual. The information required to program and re-program firmware devices in a system.

Functional Baseline. The approved documentation describing the system's functional, performance, interoperability, and interface requirements and the verifications required to demonstrate the achievement of those specified requirements

Page 22 of 62 Appendix B

Hardware Development Specification. The item specification for a Hardware

Configuration Item (HWCI) defines the performance and interface requirements and design and inter-operability constraints that have been allocated to the CI from a system or higher level CI. The Hardware Development Specification provides the contractual basis for the development and verification of HWCI performance. The Hardware Development

Specification(s) will normally be used to establish the allocated baseline for the CI.

Hardware Product Specification. The item specification for a HWCI defines the performance and interface requirements and design and inter-operability constraints that have been allocated to the CI from a system or higher level CI. Item specifications provide the contractual basis for the development and verification of CI performance. An Hardware

Product Specification or Hardware Performance Specification (essentially the same document) or a Hardware Detailed Specification (containing specific design requirements) is used to provide the contractual basis for acquisition of production quantities of the HWCI.

Interface Design Description (IDD). The IDD describes the interface characteristics of one or more systems, subsystems, Hardware Configuration Items, Computer Software

Configuration Items, manual operations, or other system components. An IDD may also describe any number of interfaces. The IDD can be used to supplement the

System/Subsystem Design Description or Software Design Description. The IDD and its companion Interface Requirements Specifications serve to communicate and control interface design decisions.

Interface Requirements Specification (IRS). The IRS specifies the requirements imposed on one or more systems, subsystems, Hardware Configuration Items, Computer Software

Configuration Items, manual operations, or other system components to achieve one or more interfaces among these entities. An IRS can cover any number of interfaces. The IRS can be used to supplement the System/Subsystem Design Description (SSDD) and SRS as the basis for design and qualification testing of systems and Computer Software Configuration Items.

Information Support Plan (ISP). The ISP (formerly called the Command, Control, Communication, Computers, and Intelligence Support Plan (C4ISP)) is intended to explore the information-related needs of an acquisition program in support of the operational and functional capabilities the program either delivers or contributes to. The ISP provides a mechanism to identify and resolve implementation issues related to an acquisition program's

Information Technology (IT), including National Security Systems (NSS), infrastructure support and IT and NSS interface requirements. It identifies IT needs, dependencies, and interfaces for programs in all acquisition categories, focusing attention on interoperability, supportability, synchronization, sufficiency and net-centricity concerns.

Logistics Requirements Funding Summary (LRFS). The LRFS identifies the product support functions and sub-functions required to establish affordable and effective product support. It identifies support resource requirements and the funds available to meet those requirements. The summary displays requirements versus available funding for all ILS elements and related disciplines, by fiscal year and appropriation, and is traceable to logistic support plan tasks and activities.

Page 23 of 62 Appendix B

Marine Corps Single Acquisition Management Plan (MC-SAMP). The MC-SAMP is a

MARCORSYSCOM initiative to assist Program Management Teams in the development of a single document that satisfies both the DoD/DoN acquisition management requirement to have an acquisition strategy document and the MARCORSYSCOM requirements to have a post-production systems management plan, as well as the requirements of Part 7 of the

Federal Acquisition Regulation requirement that mandates the need for a written acquisition strategy and/or acquisition plan.

Manufacturing Plan. Plan documents methods by which design is to be built. Plan contains sequence and schedule of events at contractor and subcontractor levels that define use of materials, fabrication flow, test equipment, tools, facilities, and personnel. Plan also reflects consideration and incorporation of manufacturing requirements in the design process. It includes identification and assessment of design facilities.

Product Baseline. A CI’s approved detail design documentation including those verifications necessary for accepting product deliveries (first article and acceptance inspections.) Based on program production/procurement strategies, the design information contained in the product baseline can be as simple as identifying a specific part number or as complex as full design disclosure.

Request For Action/Request For Information (RFA/RFI). A formal documentation of an action item initiated at a technical review. RFAs require some action on the part of the developer and can be either critical or non-critical. RFIs require only the providing of existing data.

Software Design Description (SDD). The SDD describes the design of a Computer

Software Configuration Item (CSCI). Descriptions of the CSCI-wide design decisions, the

CSCI architectural design, and the detailed design needed to implement the software are contained in the SDD. The SDD is used as the basis for implementing software. It provides the acquirer visibility into the design and provides information needed for software support.

SDDs may be supplemented by Interface Design Descriptions (IDDs).

Systems Engineering Plan (SEP). The SEP is the blueprint for the conduct, management and control of the technical aspects of an acquisition program from conception to disposal.

Software Programmer’s Manual. Information needed by a programmer to program a given piece of software.

Software Requirements Specification (SRS). The SRS specifies the requirements for a

Computer Software Configuration Item (CSCI) and the methods to be used to ensure that each requirement has been met. Requirements pertaining to the CSCI external interfaces may be presented in the SRS or in one or more Interface IRS referenced from the SRS. The SRS, possibly supplemented by the IRS, is used as the basis for design and qualification testing of a CSCI.

Software Version Description (SVD). The SVD identifies and describes a SW version;

used to release, track and control each version.

Page 24 of 62 Appendix B

System Allocation Document Approved documentation that describes the allocation of requirements from the system specification to individual CIs.

System/Subsystem Design Description (SSDD). The SSDD describes the system/subsystem-wide design and the architectural design of a system/subsystem. The

SSDD may be supplemented by IDDs and Database Design Document. The Database

Design Document, with its associated IDDs, is used as the basis for further system/subsystem development. Design pertaining to interfaces may be presented in the SSDD or Database

Design Documents.

Software Development Plan (SDP). The SDP describes the software development effort, processes, methods, schedules, organization and resources.

Software Support Plan. This plan describes the sum of all activities that take place to ensure that implemented and fielded software continues to fully support the operational mission of the system.

Software Test Plan (STP). The STP describes plans for qualification testing of CSCIs and software systems. It describes the software test environment to be used for the testing, identifies the tests to be performed, and provides schedules for test activities. There is usually a single STP for a project. The STP enables the acquirer to assess the adequacy of planning for CSCI and, if applicable, software system qualification testing.

Software Users Manual (SUM). The SUM is a document that details how the hands-on user installs and uses software, a software item group, a software system or subsystem.

Supportability Plan. The Supportability Plan is the comprehensive logistics support document that summarizes the results of logistics analysis, planning and acquisition. Based on the complexity of the program, a Supportability Plan may not need to be developed if sufficient logistics information can be provided in Chapter 7 of the MC-SAMP.

Technical Review Action Plan. The document produced to capture all planning for a technical review

Technical Review Handbook (TRH). This handbook

Technical Review Summary Report. The document produced as a result of a technical review. It contains the technical findings as well as the critical RFAs.

Version Description Document. See Software Version Description https://www.tigersql.marcorsyscom.usmc.mil/epr/operations_support/EPR_FinalFiles/OS_Glossary_Oct06.asp#MCSAMP#MCSAMP

Page 25 of 62 Appendix C

Appendix C

System Requirements Review (SRR)

C-1 Overview

The System Requirements Review (SRR) is a multi-disciplined product and process assessment to ensure that the system under review can proceed into the System Development and Demonstration (SDD) phase, and that all system and performance requirements derived from the Capability Development Document (CDD) are defined and consistent with cost

(program budget), schedule (program schedule), risk, and other system constraints.

Generally this review assesses the system requirements as captured in the system specification, and ensures that the system requirements are consistent with the preferred system solution as well as available technologies resulting from the Technology

Development phase. Of critical importance to this review is the understanding of program technical risk inherent in the system specification and SDD phase program plan. Acceptable level of risk is key to a successful review.

Evaluate the system’s requirements to determine whether they are fully defined, sufficiently well understood and consistent with the preferred system solution, and whether traceability of system’s requirements to the CDD is maintained. A successful review is predicated on the review team’s determination that the system requirements, preferred system concept, available technology, and program resources (funding, schedule, staffing, and processes) form a satisfactory basis for proceeding into the SDD phase.

Tailor the review in accordance with the technical scope and risk of the system. Describe any tailoring in the SEP.

Completion of this review should provide the following:

(1) An approved system specification

(2) A preliminary allocation of system requirements to hardware, human, and software subsystems

(3) Identification of all software components (tactical, support, deliverable, non-deliverable, etc.)

(4) A comprehensive risk assessment for System Development and Demonstration

(5) An approved System Development and Demonstration Phase Systems

Engineering Plan that addresses cost and critical path drivers

(6) An approved Product Support Plan with updates applicable to this phase

Notwithstanding successful completion of the SRR, the contractor remains responsible for the system design/performance requirements within the terms of the contract.

C-2 Timing

Conduct the SRR near the conclusion of the Technology Development phase, following full

Concept Refinement definition, completion of Technology Development definition, and prior to Milestone B (program initiation). In the competitive environment, with multiple contractors competing, conduct SRRs with all contractors. Conduct an additional SRR, if needed, as an initial technical review of the System Development and Demonstration phase for the purposes of establishing the technical baseline and approach. Do not schedule the

Page 26 of 62 Appendix C

SRR at a particular number of months after contract award; rather, SRR should occur relative to the maturity of the system technical baseline as described above.

C-3 SRR Entrance Criteria

The following are typical Entrance Criteria:

(1) If applicable, an Alternative Systems Review (ASR) has been successfully completed.

(2) A preliminary agenda has been coordinated (nominally) 30 days prior to the SRR.

(3) SRR technical products listed below for both hardware and software system elements have been made available to the cognizant SRR participants prior to the review:

(a) System specification

(b) System software functionality description

(c) Preferred system solution definition

(d) Updated risk assessment

(e) SEP

(f) Updated cost and schedule data

(g) Updated logistics documentation

(h) Updated Human Systems Integration related documentation

(i) Software Development Plan is complete. May be a standard organizational or company document tailored to the program

(j) System architecture (hardware, software, human, material as necessary)

(k) Integrated system architecture and supporting views…