GPR 8700.6D Engineering Peer Reviews.pdf

DIRECTIVE NO. GPR 8700.6D APPROVED BY Signature: Original Signed By

EFFECTIVE DATE: September 3, 2019 NAME: Felicia Jones

EXPIRATION DATE: September 3, 2024 TITLE: Director, Engineering and Technology Directorate

CHECK THE GSFC DIRECTIVES MANAGEMENT SYSTEM AT

http://gdms.gsfc.nasa.govgdmsnew/home.jsp TO VERIFY THAT THIS IS THE CORRECT VERSION PRIOR TO USE.

08/16

Goddard Procedural Requirements (GPR)

COMPLIANCE IS MANDATORY

Responsible Office: 500/ Engineering and Technology Directorate

Title: Engineering Peer Reviews

REVALIDATED 09/03/2019

PREFACE

P.1 PURPOSE

This procedure describes the process for Engineering Peer Reviews (EPRs) of Goddard Space Flight

Center (GSFC) projects.

P.2 APPLICABILITY

These requirements apply to the design and development of all GSFC products (hardware and software/flight and ground), regardless of the development approach being used. In cases where products are being acquired from outside of GSFC, projects shall define and implement an effective peer review process, commensurate with the level of maturity, heritage and reliability of the product being procured, and consistent with the intent of this document. While it is recommended that sub-orbital and technology development projects follow the intent of these requirements it is recognized that these types of projects might have their own processes for conducting engineering peer reviews. These processes should be designed and implemented to be consistent with the defined risk posture of the program or project.

a. In this directive, all document citations are assumed to be the latest version unless otherwise noted.

b. In this directive, all mandatory actions (i.e., requirements) are denoted by statements containing the term “shall.” The terms “may” or “can” denote discretionary privilege or permission; “should” denotes a good practice and is recommended but not required; “will” denotes expected outcome; and “are/is” denotes descriptive material.

P.3 AUTHORITIES

NPD 1280.1, NASA Integrated Management System Policy

P.4 APPLICABLE DOCUMENTS

GPR 8700.4, Goddard Systems Reviews

DIRECTIVE NO. GPR 8700.6D Page 2 of 13

EFFECTIVE DATE: September 3, 2019

EXPIRATION DATE: September 3, 2024

P.5 CANCELLATION

GPR 8700.6B, Engineering Peer Reviews

P.6 SAFETY

Relevant project safety requirements shall apply.

P.7 TRAINING

Not applicable.

P.8 RECORDS

Record Title Record Custodian Retention

EPR Presentation Material, attendance list, EPR

Reports including Requests for Action (RFAs), RFA Responses, RFA Originator Decisions, and

Summary Status of RFAs

Product Design Lead

(PDL) using project’s

CM system

*NRRS 8/101 - Permanent. Cut off records at close of program/project or in

3-year blocks for long term programs/projects. Transfer to records center storage. Transfer to National

Archives 7 years after cutoff.

*NRRS 1441.1 – NASA Records Retention Schedules

P.9 MEASUREMENT/VERIFICATION

Audits related to engineering peer reviews, such as effectiveness of early (in the development cycle) identification and correction of design issues, should be used by the Applied Engineering & Technology

Directorate (AETD) to assess the effectiveness of this procedure.

PROCEDURES

1. Overview

Engineering Peer Reviews are an important element in an overall review approach aimed at providing efficient, timely, independent technical feedback to mitigate risks. They typically build on less formal engineering branch/table-top reviews while supporting more formal Code 300-led system reviews conducted at critical project milestones.

The purpose of an EPR is to add value and reduce risk through the infusion of expert knowledge from subject matter experts who are not directly involved in or responsible for development of the subject product. The review should probe the details in order to confirm the approach, validate decisions, and/or make specific recommendations for improvement. An EPR should provide a penetrating examination of design, analysis, manufacturing, integration, test, and operations and should include

DIRECTIVE NO. GPR 8700.6D Page 3 of 13 assessment of items such as drawings, schematics, parts and materials, processes and data. Finally, the

EPR should result in a documented, traceable record of the technical findings associated with the subject of the review. (This differs from the less formal table-top reviews where formal documentation is not necessarily a requirement.)

2. Engineering Peer Review Planning

In Phase A, an engineering peer review plan (a list of the planned set of EPRs) shall be developed and approved by the project. The peer review plan should be developed by the project lead systems engineer

(or equivalent), in collaboration with the project team of Product Design Leads (PDLs)/discipline leads, as well as any other relevant project personnel.

The peer review plan shall be maintained under project CM control for the duration of the project.

The peer review plan should be updated as necessary during Phase B, prior to the project Preliminary

Design Review (PDR), as well as during Phase C, prior to the project Critical Design Review (CDR).

EPRs may be scheduled following CDR to support late-cycle activities.

The engineering peer review plan, and any necessary updates, shall be made available for review by the

Senior ETD Engineer and/or the GSFC Chief Engineer as well as the Project Manager (PM).

In addition to reviewing subsystem designs, EPRs to review systems engineering and focused evaluation of concepts, designs, plans and processes associated with combinations of functions that cross traditional discipline boundaries should be considered. Examples of this include maneuver planning and execution;

fault detection and correction; the end-to-end data path from detection to data archiving and distribution;

or solutions to address, for example, pointing, thermal or contamination constraints. As highlighted by the above examples, EPRs should be topic based instead of being forced to align to a particular Work

Breakdown Structure (WBS).

Multiple peer reviews should be conducted over the development lifecycle with content consistent with the evolving design and development activities. Although peer reviews normally precede reviews at higher assembly levels, they should be timed primarily to maximize effectiveness for the peer reviewed item and should not be used as dry-runs for milestone reviews.

The project lead systems engineer should keep the project management, the GSRT chairperson(s), the

Senior ETD Engineer, and the GSFC Chief Engineer apprised of the current schedule for peer reviews.

3. Engineering Peer Review Team

The goal for an EPR Team (the review panel) is to conduct a thorough, independent, technically-oriented review with a variety of perspectives, experiences, and processes considered. The engineering peer review team shall consist of technical experts with significant practical experience (individuals who have done it before) relevant to the technology and requirements of the item being reviewed.

DIRECTIVE NO. GPR 8700.6D Page 4 of 13

A Chairperson for each EPR shall be selected by the project led systems engineer/PDL discipline lead and the Chairperson should then select the other members of the review team after consulting with appropriate project and discipline personnel (e.g., project lead systems engineer, PDL/discipline lead, the PDL Branch Head, etc.).

Most review team members should be independent of the project team, i.e., not a participant in the project team as provider of hardware, software, or analytical, fabrication and other services. Rare expertise or intimate project knowledge may demand that some review team members be from the project team.

As appropriate, technical experts from outside the GSFC engineering community should be considered, e.g., other NASA Centers or Federally Funded Research and Development Centers (FFRDC), University

Affiliated Research Center (UARC) or private industry, as well as technical experts with end-item systems perspectives. To the maximum extent possible, review team membership should be consistent for the duration of the project in order to ensure continuity and to avoid repeating basic information.

4. Agenda and Contents of Engineering Peer Reviews

In advance of each EPR, the PDL, should submit the proposed objectives, agenda, and scope of content for approval by the PDL Branch Head and EPR Chairperson.

Enabling assessment of potential issues through independent critique is paramount. This implies a level of detail sufficient for this assessment. In addition to technical details, relevant verification

(test/analysis) details (including parts qualification status) along with the planned schedule of these activities should be reviewed.

Potential presentation formats cover a broad range, from a small group examining drawings, analysis results, test data, or software code to speaker/audience forums. Accomplishing the objectives does not mandate using formal presentations (e.g., PowerPoint presentations) and in fact, since EPRs are intended to be detailed, technical reviews, the wide use of bulleted viewgraphs is strongly discouraged.

Viewgraphs, when used, should guide the discussion, and give context (e.g., a block diagram) to the more technical details that are better presented in formats other than viewgraphs.

Appendix C contains a list of engineering peer review sample topics.

5. Conducting Engineering Peer Reviews

Peer review material should be made available to the review team at least one (1) week prior to the review. In advance of the review, the panel should evaluate the material and identify areas that may require further discussion at the review.

The EPR Chairperson shall preside at the review to moderate and facilitate discussion between the panel members and the PDL team, cut off tangent discussions and otherwise keep the review focused on the target material.

DIRECTIVE NO. GPR 8700.6D Page 5 of 13

The PDL should direct the presentation of the review material in a format appropriate to the nature, scope and complexity of the product. Design relevance/expected compliance to driving performance requirements shall be included in the review. The focused examination of certain design details may be required and subgroup or “splinter” sessions are encouraged, as appropriate, with results reported to the full peer review team.

Deliverables (along with the deliverable schedule) pertinent to the subject review shall be presented, and the review panel should assess the cost and schedule risk of these deliverables within the technical context of the review.

At the conclusion of the review, the Chairperson should summarize the review team’s observations and findings, determine if specific objectives of the peer review were met, review the RFAs for clarity, screen the RFAs for their appropriateness and timing, and establish a due date for each RFA (in conjunction with the project and the PDL).

A list of EPR Chairperson Dos & Don’ts are provided in Appendix D and provide some guidance, based on lessons learned, for conducting the review.

6. Engineering Peer Review Documentation

After the EPR, the EPR Chairperson shall compile the final list of findings and RFAs, after reviewing the RFAs for duplication and clarity. No RFA submitted by a review team member should be removed, either because of duplication or some other reason, without agreement of the originator. It is the responsibility of the EPR Chairperson to ensure that the final list of submitted RFAs is appropriately categorized for the phase of the product under review.

The EPR Chairperson shall issue a report, including a summary assessment, findings and the final set of

RFAs to the PDL/Project. Consideration should be given to also sending the report to the PDL’s branch and division management. The report should be issued within 14 calendar days of the completion of the review, but preferably within 7 calendar days. The report would typically be a memorandum summarizing the, agenda, participants, review team observations/findings/issues, RFAs and the board’s assessment if the review met the technical success criteria. Any items whose resolution the board deems necessary prior to a higher level review, should be highlighted in the report.

The PDL is responsible for the implementation/resolution of RFAs, the preparation of responses to the review team, and the reporting of RFA status (open, closed, contested) to the Project.

The EPR Chairperson will assist the PDL in the closure process by ensuring timely review of responses by the RFA originator. The Chairperson and the RFA originator shall review RFA responses for acceptability and inform the PDL/Project of their decisions in writing, including any perceived residual risk. EPR RFAs should not be closed without the consent of the originator. If there is disagreement about the closure, project and engineering technical management should arbitrate.

DIRECTIVE NO. GPR 8700.6D Page 6 of 13

It is recognized that there will often be cases where there will be findings that either do not lend themselves to RFAs or don’t rise to the level of an RFA, but still represent a high level of concern from a reviewer. The PDL and lead systems engineer should make an effort to understand such findings and to work with the reviewer to ensure that these types of findings (i.e., those without an associated RFA) are addressed.

EPR presentation material (e.g. block diagrams, software code, schematics, etc.), the EPR report, attendance list, RFA responses, and RFA dispositions are controlled documents and shall be maintained in the project documentation system throughout the project lifecycle. The status of EPR RFAs should be maintained by the PDL/project and be readily accessible.

7. Engineering Peer Review Interface with Goddard System Reviews

At each GSR, the Project shall present a summary of the results of each EPR conducted since the last

GSR. This presentation should include an overview of the EPR, listing of review team members, key findings, summary RFA listing and status, lessons learned, residual risks, technical and programmatic, and mitigation strategies.

DIRECTIVE NO. GPR 8700.6D Page 7 of 13

Appendix A – Definitions

A.1 Engineering Peer Review (EPR) – A focused, in-depth, technically-oriented, formal review that is convened using non-project related personnel under project and ETD auspices and that assesses the design and development of a product or system of interest at a particular point in its life cycle.

A.2 Table Top or Equivalent Reviews – Informal reviews which are called for and planned by the

Product Design Lead (PDL), or their respective Branch Head in coordination with the PDL, and are intended to bring together independent subject matter experts to review designs and plans. Table-top reviews remain an important and valuable, ad-hoc review step in the engineering community, but should not be considered a substitute for or taken as an EPR.

A.3 Goddard System Reviews (GSR) – One of a series of system-level reviews conducted at critical project/product milestones in accordance with GPR 8700.4, Goddard System Reviews. GSRs build upon the results of a robust set of EPRs. The adequacy of a project’s engineering peer review plan and EPR activity is assessed at the GSRs.

A.4 Product Design Lead (PDL) – The individual, who has overall responsibility and who is the project lead engineer of a particular discipline, hardware or software element.

A.5 Branch Head (BH) – The Branch Head is the first line of management authority responsible for

Engineering Technical Authority and the implementation of Engineering Technical Excellence for their assigned discipline.

A.6 Project Manager (PM) – The individual designated as having management responsibility for a

GSFC project. A Project Manager may be assigned to any directorate and have a title such as

Project Manager, Project Formulation Manager, Instrument Manager, Principal Investigator, etc.

A.7 Lead Project Systems Engineer (LPSE) – The individual designated as the lead systems engineer;

responsible for applying Agency and Center-level engineering requirements, assuring deviations and waivers are submitted and approved, and certifying that the system has met technical performance and verification requirements. Note, at the system level, the LPSE (or Mission Systems Engineer) is typically the Technical Authority (TA) for the project. For instrument development, the LPSE (or

Instrument Systems Engineer) might not be the TA.

A.8 Request for Action (RFA) – A formal written request from the review team, through the review chairperson, that asks for a specific action of the product team. RFAs shall be actionable items that address deficiencies or gaps in the work reviewed and that are appropriate for the project phase.

RFAs can be categorized into the following categories for tracking purposes:

Action: Take some action to address (ideally within 30 days)

Informational: Provide some information (ideally within 7 days)

Advisory: Recommendation to consider doing some action but not required

Flag: An important action that will be done at some later point in time.

DIRECTIVE NO. GPR 8700.6D Page 8 of 13

Appendix B - Acronyms

ETD Engineering & Technology Directorate

BH Branch Head

CDR Critical Design Review

EEE Electrical, Electronic and Electromechanical

EEPROM Electronically Erasable Programmable Read Only Memory

EMI Electromagnetic Interference

EPR Engineering Peer Review

ESD Electromagnetic Static Discharge

FFRDC Federally Funded Research & Development Center

FMEA Failure Mode and Effects Analysis

FPGA Field Programmable Gate Array

GEVS General Environmental Verification Standard

GOLD Goddard Open Learning Design

GSFC Goddard Space Flight Center

GSR Goddard System Reviews

GSRT Goddard Systems Review Team

LPSE Lead Project Systems Engineer

PDL Product Design Lead

PDR Preliminary Design Review

PM Project Manager

RFA Request for Action

SRB Standing Review Board

TA Technical Authority

UARC University Affiliated Research Center

VHDL Very High-level Design Language

WBS Work Breakdown Structure

DIRECTIVE NO. GPR 8700.6D Page 9 of 13

Appendix C

Engineering Peer Review Sample Topics1

Systems, Subsystems and Component Interfaces

System Engineering

Operations Concepts and Operational Modes

Fault Management

Electronic Packaging: custom modules, printed wiring assemblies, box-level thermal and mechanical design

Design Adequacy: Engineering Reports, Drawings, Schematics, Source Code (for software), VHDL/set lists (for FPGAs), Analyses, Parts and Materials

Analysis Adequacy: Modeling and Simulation

Compliance with GOLD Rules and GEVS and System/Subsystem Requirements

Manufacturing Processes and Manufacturability

Verification Approach: Test, Analyses, Simulation

Verification Results: Data Adequacy, Observed Margins, Trends, Anomalies

Lessons Learned (how applied and where learned)

Margin Trends

System Safety including Software Safety

Contamination Requirements and Implementation

Long Lead Items

Radiation Effects

EMI

Material Compatibility

Limited Life Items

Circuit Analyses, including Worst Case Analyses (Thermal, Mechanical, Timing, Signal

Integrity)

Heritage Claims and Associated Limitations (including consideration of performance, production, and environmental exposure)

Analysis Results (all relevant disciplines)

EEPROM Usage

Qualification Approach

FMEA, Interface and Internal

Probabilistic Risk Assessment

EEE Part Selection, Screening, and Qualification

Deployable Devices

Sensors, Telemetry and Anomaly Diagnostics

Functional Testing Accumulated Time, Failure Free Hours

1 Examples only; not intended to be either a required set of review topics or a complete set

DIRECTIVE NO. GPR 8700.6D Page 10 of 13

Engineering Peer Review Sample Topics (Cont.)

Life Testing, Mechanical

Support Equipment Design and Limitations

Pressure Venting

ESD Sensitivity and Precautions

Alignment

Jitter, Sources and Sensitivities

Technology Readiness

Flight Software and Interfaces

DIRECTIVE NO. GPR 8700.6D Page 11 of 13

Appendix D

Top Dos and Don’ts for the EPR Chairperson

DO: Meet with your team before the review to establish expectations for conduct.

DO: Emphasize the working level focus of the peer review. Details should not be skipped.

DO: Keep the review team on subject matter, keeping tangents at a minimum.

DO: Screen and review RFAs at conclusion of the review and get agreed to closure dates on them.

Allow for time to get this done before everyone leaves the room.

DO: Debrief project at end of review with respect to objectives being met or not met.

DO: Work with PDL to get closure on RFAs by originator. Stay involved in this closure process.

Close the RFA when the level of closure reaches the detail required by the subject Peer Review, not the following Integrated Independent Review or later gate; precluding RFA creep.

DO: Get the written report out in two weeks.

DO: Consider whether a proposed RFA is likely to really help the Project meet a requirement, not just use time and resources to make a marginal improvement.

DON’T: Accept RFAs that are not actionable. Categorize them appropriately for the project to add value to the work and not burden them unnecessarily. This means, amongst other things, do not accept

RFAs that simply suggest other ways of doing things, express concern about the current work, or point out future pit-falls. These can be advisories or flag items.

DON’T: Allow project/GSRT presence to hamper or discourage criticisms. Keep discussions open, candid and constructive.

DON’T: Try to do an 8-hour review in a 4-hour time slot. Make sure sufficient time is planned for review, including lunch and breaks.

DIRECTIVE NO. GPR 8700.6D Page 12 of 13

CHANGE HISTORY LOG

Revision Effective Date Description of Changes

Baseline 10/16/01 Initial Release

A 01/26/05

Changes made to update organization and document references and/or distinguish requirements from recommendations in accordance with the NASA rules update mandate. Minor changes to (1) make mandatory the inclusion of technical experts from outside GSFC on the review teams under certain circumstances, whereas before, this was discretionary, and (2) make mandatory, rather than discretionary, for the EPR chairperson to issue his/her report within 30 days of the completion of the review.

Metrics statement added (P.9).

B

06/05/09

General rewrite to incorporate the following:

- Specific AETD responsibilities aimed at ensuring that process, content and quality of the EPRs meets expected

GSFC standards,

- Changes consistent with implementation of TA,

- Peer review related recommendations contained in various

NASA failure reports, and

- Lessons learned from past GSFC experience with effective peer review activities.

Acronym list added (P.11)

C 09/24/14 General rewrite for clarity, document organization and to distinguish EPRs from milestone review dry-runs.

D 09/03/19 Revalidated with Administrative Changes to include:

- AETD updated to ETD throughout

- Striking Class D Constitution

- Updating applicable documents

- IRR updated to GSR throughout

DIRECTIVE NO. GPR 8700.6D Page 13 of 13

- Wording change in section 7

- WBS added

- Section 3 update to who is responsible for selecting the

EPR chairperson.