J.1-E Surveillance Plan.pdf

RFP 80GRC022R0016 Attachment J.1-E

SURVEILLANCE PLAN

FOR THE

SPACE FLIGHT SYSTEMS

DEVELOPMENT AND OPERATIONS III CONTRACT (SpaceDOC III)

80GRC022R0016

DATE: January 6, 2023

NASA GLENN RESEARCH CENTER

CLEVELAND, OHIO 44135

Change Record

Rev. Effective Date Description

Draft 1/6/2023 Initial draft of SpaceDOC III Surveillance Plan

Table of Contents

1 INTRODUCTION

1.1 Purpose

1.2 Scope

1.3 Background

1.4 Applicable Documents

2 SURVEILLANCE STRATEGY DEFINITIONS

2.1 Insight

2.2 Oversight

2.3 Hybrid

3 RESOURCES

3.1 General

3.2 Space Flight Systems Resident Office

3.3 Supply Chain Insight Central (SCIC)

4 CONTRACT LEVEL SURVEILLANCE ACTIVITIES

4.1 Role of Surveillance Leads

4.2 SpaceDOC III Control Board

4.2.1 Membership

4.3 Contract Level Audits

4.3.1 Defense Contract Management Agency (DCMA)

4.3.2 Defense Contract Audit Agency (DCAA)

4.4 Engineering Review Boards (ERBs) and S&MA Engineering Review Boards (SERBs)

5 PROJECT TEAM SURVEILLANCE STRATEGY AND APPROACH

5.1 General

5.2 Surveillance Planning for Projects

5.2.1 General Forms of Surveillance

5.2.2 Surveillance Support through the Safety and Mission Assurance (S&MA) Directorate

5.2.3 Surveillance Support through the Research and Engineering Directorate (Code L) 13

5.3 Implementation

5.3.1 Communication of Surveillance Approach to Contractor

5.3.2 Communication within NASA GRC Surveillance Team

5.3.3 NASA GRC Surveillance Team Interactions with the Contractor

Appendix A S&MA SURVEILLANCE SUPPORT Checklist example

Appendix B Code L SURVEILLANCE support checklist example

Appendix C Acronym List

FOREWORD

Under performance-based contracts, such as the Space Flight Systems Development and Operations III Contract (SpaceDOC III), the contractor assumes more responsibility and greater risk in exchange for more flexibility and less direct NASA Glenn Research Center (GRC) involvement in contract activities. However, NASA GRC still has the responsibility to monitor the contractor’s performance over the course of the contract to ensure it is acceptable. To meet this responsibility, NASA GRC needs sufficient information on how the contractor is performing to be assured contract requirements are being satisfied.

This Surveillance Plan has been prepared to address NASA GRC’s need for information under SpaceDOC III. It reflects a plan, which may be tailored to fit each project’s unique circumstances. It is a guide to the Base/Delivery Order Project Managers (PM) on how to implement surveillance on their project. It communicates NASA GRC’s approach to conduct surveillance on SpaceDOC III.

1 INTRODUCTION

1.1 PURPOSE

The purpose of this Surveillance Plan is to define the overall approach National Aeronautics and Space Administration (NASA) Glenn Research Center (GRC) intends to use to monitor contractor performance on the SpaceDOC III. The Surveillance Plan defines the process NASA GRC intends to follow to obtain data, evaluate the Contractor and determine if contract performance is acceptable. The goal is to balance the level of NASA GRC surveillance with the perceived impacts and risks to mission success.

1.2 SCOPE

This plan identifies the strategy, resources, and surveillance activities used for assessment of contractor performance. The plan covers overall contractor performance in meeting SpaceDOC III Statement of Work (SOW) and Base/Delivery Order (DO) requirements.

NASA GRC surveillance is conducted through the combined efforts of NASA personnel, other NASA representatives and delegated Government agencies.

1.3 BACKGROUND

NASA GRC implements several space-related programs within the Space Flight Systems Directorate (SFSD). GRC has space flight development responsibility for numerous microgravity research investigations on ISS, human research projects, space flight technology developments and demonstrations such as advanced power, propulsion, communications and other systems, and the potential for space science instrumentation packages. NASA GRC and the SpaceDOC III contractor are responsible for the design, development, analysis, fabrication, assembly, test, verification, delivery, and operation of space flight systems, associated support systems and equipment, and related ground development activities, that include research and technology developments, supporting NASA space flight missions under the Science Mission Directorate, the Exploration Systems Development Mission Directorate, the Space Technology Mission Directorate, and the Space

Operations Mission Directorate. The SpaceDOC III contract is one of a number of entities that will be employed to implement the space flight development content within the SFSD.

NASA implements NPR 7120.5, NASA Program and Project Management Processes and Requirements for space flight projects and NPR 7120.8, NASA Research and Technology Program and Project Management Requirements for research and technology projects for formulation and implementation. The SpaceDOC III SOW and individual Base/Delivery Orders will provide the requirements the contractor needs to define what is necessary to produce the deliverables required.

1.4 APPLICABLE DOCUMENTS

Applicable documents for this surveillance effort include but are not limited to:

• 80GRC022R0016, Space Flight Systems Development and Operations Contract III (SpaceDOC III) and all attachments

• SpaceDOC III Base/Delivery Order Requirements

• NPR 7120.5, NASA Program and Project Management Processes and

Requirements

• NPR 7120.8, NASA Research and Technology Program and Project Management

Requirements

• GLPR 7120.5.30 Space Assurance Requirements (SAR)

• SSP 50431, Program Requirements for Payloads

• NPR 7150.2, NASA Software Engineering Requirements

• NPR 7123.1, NASA Systems Engineering Processes and Requirements

• NPR 8735.2, Hardware Quality Assurance Program Requirements for Programs and Projects

• SAE AS9100, Quality Management Systems - Requirements for Aviation, Space, and Defense Organizations

• SAE AS9101, Quality Management Systems Audit Requirements for Aviation, Space, and Defense Organizations

• FAR Part 46, Quality Assurance

2 SURVEILLANCE STRATEGY DEFINITIONS

2.1 INSIGHT

Insight is a surveillance strategy that relies heavily on non-intrusive methods such as evaluating contract deliverables, existing contractor procedures, contractor metrics, and data acquired from contractor records. Insight is a continuum that can range from low intensity, such as reviewing quarterly reports, to high intensity, such as conducting formal reviews and performing surveys.

Insight as applied to the SpaceDOC III will result in lower levels of Government surveillance and allow the contractor to assume increased responsibility and accountability for the integrity of processes.

2.2 OVERSIGHT

Oversight is a surveillance strategy that relies heavily on intrusive methods such as in-line involvement with the contractor’s processes. Oversight is a continuum that can range from low intensity, such as customer concurrence in reviews (e.g., PDR, CDR), to high intensity oversight, in which the customer has day-to-day involvement in the contractor’s decision-making process (e.g., hardware inspections). Oversight is used for problem areas, areas of high risk, areas of high visibility, or areas where the contractor’s experience is limited.

2.3 HYBRID

A hybrid surveillance approach combines elements of insight and oversight and may be instituted at a contractor’s facility when a high level of confidence does not exist regarding the contractor’s ability to identify, manage, and control programmatic risks. This may occur when new technology is acquired, or unproven processes are employed by a contractor. In this situation, oversight surveillance is used until sufficient data exist that demonstrate the contractor has all critical processes under control. Only after the contractor’s demonstration of risk mitigation capabilities will NASA consider transitioning to insight activities that rely predominantly on internal contractor data. The transition period from oversight to insight activities is variable and accomplished incrementally, depending on contractor performance.

3 RESOURCES

3.1 GENERAL

Primary surveillance activities will be implemented using government and support contractor personnel and electronic access to contractor document repositories at the contractor site or at GRC. Should it be necessary, GRC will obtain the support of the Defense Contract Management Agency (DCMA) personnel to conduct specific surveillance. The multi-disciplinary surveillance team may include:

a) GRC Space Flight Systems Directorate personnel.

b) GRC Safety and Mission Assurance Directorate (S&MA) support personnel.

c) GRC Research and Engineering Directorate (Code L) support personnel.

d) Other GRC or GRC contractor personnel, as required.

e) Resident Office or DCMA personnel at the contractor, as required.

f) Others, as needed.

3.2 SPACE FLIGHT SYSTEMS RESIDENT OFFICE

If determined necessary, NASA GRC will establish a Space Flight Systems Resident Office at the contractor facility. The Space Flight Systems (SFS) Resident Office, located at the contractor’s facility and includes electronic remote access, is an extension of the SFSD. The SFS Resident Office is established to enhance NASA GRC insight into contractor activities and to facilitate NASA GRC access to contractor databases used to manage the work. The databases that are accessible will include the configuration management files for documents, drawings, software, project schedules, risk management, corrective action files and technical plans, analyses, and reports under development. The SFS Resident Office will be comprised of office space, complete with computer workstations and network access, telephones, and office furniture. In addition, the contractor will provide remote network access capabilities for the Surveillance Team to access all the SpaceDOC III data systems.

3.3 SUPPLY CHAIN INSIGHT CENTRAL (SCIC)

Supply Chain Insight Central is a NASA database that houses Supply Chain Risk Management (SCRM) data. This data includes audit/assessment reports and other historical data for contractors that have previously or currently are performing work for NASA. Data collected under the SpaceDOC III surveillance activities will be uploaded into

SCIC.

4 CONTRACT LEVEL SURVEILLANCE ACTIVITIES

While individual project surveillance is directed by the PM, there is a contract level surveillance function also being conducted. Contract level surveillance will review and assess contract level functions such as, however not limited to, the following:

• Contract Management

• Business Management System

• Property Management

• Product Assurance

• Quality Management

• Configuration Management

• Consistency of Product Development Approach

• IT/Cyber Security

4.1 ROLE OF SURVEILLANCE LEADS

The SpaceDOC III Control Board designates surveillance leads in each of the discipline areas. Surveillance leads are to be cognizant of surveillance activities across the contract and to assure consistent approaches are being utilized by the various projects. The leads are also aware of issues within the projects and can bring forward issues, not able to be resolved by the Base/Delivery Order Manager, to the SpaceDOC III Control Board for resolution.

4.2 SPACEDOC III CONTROL BOARD

The SpaceDOC III Control Board is used to review contract level deliverables and to review, assess, and resolve contract level issues. As required the board will identify the need for Engineering Review Boards (ERBs) or S&MA Engineering Review Boards (SERBs) and support the activity to resolve contract issues. Surveillance leads and appropriate discipline personnel will conduct contract level audits, will oversee surveillance activities to assure consistent approaches are being utilized across SpaceDOC III and look for trends in contractor performance.

The SpaceDOC III Control Board will meet at least once a month to evaluate problems, concerns, issues, and review metrics for trends and performance indicators. All available information will be evaluated, and any action by the Government will be determined based upon the scope and magnitude of any particular issue or problem. The SpaceDOC III Control Board Chairperson will formally notify the SpaceDOC III Contracting Officer’s Representative (COR) of situations where it is perceived that the contractor has failed to take prudent corrective or preventive action of situations perceived to increase risk or findings of continued contractual non-compliance.

Information will flow from individual team members through respective Base/Delivery Order Project Managers to SpaceDOC III Control Board representatives, who will present issues and achievements at board meetings. Information gained from these formal and informal exchanges of ideas and collection of data will be compiled then evaluated as a continuous measure of contract performance.

4.2.1 Membership

The membership of the SpaceDOC III Control Board is the SpaceDOC III Program Manager and Chairperson (representing SFSD), COR, Alternate Contracting Officer’s Representative (ACOR), Chief Safety and Mission Assurance Officer (CSO), Chief Engineer (CE), Contracting Officer (CO), Resource Analyst, and Configuration Management (CM) personnel. If required, the Base/Delivery Order Project Managers and other Program and Project Representatives will present issues to the Board. The Chairperson may select a designee in their absence.

4.3 CONTRACT LEVEL AUDITS

Supplier audits or assessments are used to generate evidence of prime and sub-tier supplier risks that are related to the robustness of the supplier's QMS and to their design and control of special processes. For additional information, see AS9101F, Quality Management Systems Audit Requirements for Aviation, Space, and Defense Organizations, for the methodology for performing supplier QMS audits. NASA GRC surveillance team members will conduct independent audits, with a frequency and scope determined by status, importance, or performance, of the contractor's activities, processes, products, documentation, and data in order to provide assurance that the program is being implemented according to all requirements and specifications. At a minimum, audits in the risk management, quality assurance, software assurance, and system safety disciplines will be performed at least once a year during the life of the SpaceDOC III contract. The lead for each audit will define which requirements the contractor will be assessed against. These include requirements found in GLPR 7120.5.30 Space Assurance Requirements (SAR) which covers the appropriate NPRs for each Safety and Mission Assurance discipline, and appropriate SpaceDOC III contract requirements. The audit reports will be provided to the SpaceDOC III Control Board before being sent to the contractor to address. The results will also be uploaded to the SCIC database. These audits will normally be conducted with advance notification and coordinated with the contractor. NASA GRC reserves the right to conduct unscheduled audits when evidence indicates that contractor performance is deficient.

A NASA GRC audit, assessment, survey, or equivalent, will be used to evaluate supplier risk where no prior record can be referenced in NASA supplier databases, or where the prior audit, assessment, survey or Government Contract Quality Assurance (GCQA) records are older than three years.

Contract-level audits authorized by the SpaceDOC III Control Board, will be supported by the two Department of Defense Audit Agencies identified in 4.3.1and 4.3.2, if appropriate.

NASA GRC has the right to perform contract quality assurance on subcontracted suppliers or services only when required in the Government’s interest per 48 CFR § 46.405 – Subcontracts.

NASA GRC will follow the requirements outlined therein when it is in the interest of the Government to perform quality assurance on subcontracted suppliers. These activities will be arranged through the Contractor.

When appropriate, FAR clause 52.246-3(c), assures that the Government has the right to inspect and test any or all of the work included in the contract, at all places and times, including the period of manufacture, and in any event before acceptance.

The Contractor will conduct internal audits, per SAE AS9100, Quality Management Systems - Requirements for Aviation, Space, and Defense Organizations, using their own Quality Assurance (or other independent) organization in accordance with Contractor standard practices and policies.

NASA GRC Surveillance Team members may concurrently participate in SpaceDOC III contractor-led audits involving SFSD hardware or processes.

4.3.1 Defense Contract Management Agency (DCMA)

The DCMA is delegated by NASA GRC primarily to audit the following contract areas:

• Property

• Product Assurance

4.3.2 Defense Contract Audit Agency (DCAA)

The DCAA is delegated by NASA GRC primarily to audit the following contract areas:

• Business Management Systems

4.4 ENGINEERING REVIEW BOARDS (ERBS) AND S&MA ENGINEERING REVIEW

BOARDS (SERBS)

ERBs and SERBs are utilized to bring in broader support from the technical community with expertise in the area of discussion. Activities can include independent analysis, detailed review of material received, or approaches taken by the contractor. Review results and recommendations are documented then presented to the SpaceDOC III Control Board for approval and to initiate contract action, as required.

5 PROJECT TEAM SURVEILLANCE STRATEGY AND APPROACH

5.1 GENERAL

An oversight or hybrid approach will be used until NASA GRC is confident that the contractor’s processes are in compliance with the contract and are properly documented and controlled. After

NASA GRC has obtained sufficient objective evidence to have confidence in the contractor’s performance the surveillance strategy may transition to an insight approach.

The overall surveillance goal will be to obtain objective evidence and data that enable NASA GRC to determine whether the contractor's program and processes are functioning as intended in accordance with the terms of the contract. The focus will be on prevention rather than detection, (i.e., emphasizing controlled processes and methods of operation), as opposed to relying solely upon inspection and test to identify problems.

Throughout the life of the flight, research, or technology development project, the surveillance activities will focus on various products. The Contractor will submit plans and procedures for NASA GRC review and approval in accordance with the SpaceDOC III SOW and the Data Item Description (DID) documents called out in the Base/Delivery Order. NASA GRC reviewers from relevant technical disciplines will review contract products when requested by the PM. The PM and reviewers are responsible for observing whether SpaceDOC III products are in compliance with the contract and will submit comments for any areas of noncompliance. After the NASA GRC obtains evidence that SpaceDOC III products are contractually compliant and have been properly implemented; screening is needed to observe whether the Contractor has controls in place to maintain that status continuously.

5.2 SURVEILLANCE PLANNING FOR PROJECTS

Surveillance for projects is delegated from the COR to individual project and Base/Delivery Order managers through the COR delegation letter. The appropriate type of surveillance will vary based on the contracted work being addressed and the contractor’s present performance on the activity or similar activity. Additionally, it is the responsibility of the project manager to trade surveillance costs (both in terms of NASA GRC and contractor resources) against the known project risks and potential benefits from the surveillance activity. The PM defines general areas of surveillance needs (e.g., Government Mandatory Inspection Points (GMIPs)), typically based on the project’s critical or high-risk areas. If support is required from either S&MA or Code L, the PM shall coordinate with the Chief Engineer and CSO for the specific surveillance needs.

5.2.1 General Forms of Surveillance

Typically, surveillance support can be categorized into 3 major categories:

• Formal Reviews

• Review, assess review products

• ERBs and SERBs

• Assessments, surveys, and audits

• Plan/Process/Quality Management System (QMS) review and/or compliance audits

• Initial QMS evaluation

• AS9100 compliance assessment

• Communications and informal activities

• Teleconferences

• Progress Reviews

• Technical Interchange Meetings (TIMs)

• Review of contract deliverables

• Random inspections

• GMIPs

• Witnessing

Formal reviews and assessments, surveys, and audits will have formal reports as an outcome of the activity.

SFSD’s two major providers of surveillance support are S&MA and Code L. Their specific expertise and potential areas of surveillance are described in Section 5.2.2.and 5.2.3below.

5.2.2 Surveillance Support through the Safety and Mission Assurance (S&MA) Directorate

S&MA is responsible for supporting the SFSD with matters pertaining to hardware and software in the discipline areas listed below. In addition, S&MA supports testing, anomaly reporting and resolution, and buyoff and acceptance of final deliverables. The following are specific disciplines supported by S&MA:

• Quality Assurance and Electronic, Electrical, Electromechanical (EEE) Parts

• Risk Management

• Reliability, Availability and Maintainability

• System Safety

• Software Assurance

Examples of detailed checklists of potential surveillance tasks for each discipline are found in Appendix A.

S&MA is available to provide the following types of audits to individual projects:

• Surveillance Plan Audit (SPA): The Surveillance Plan Audits are conducted based on the SMA Disciplines described in the SpaceDOC III Surveillance Plan. These Disciplines are Quality Assurance, Software Assurance, Risk Management, and Systems Safety. The Auditor is selected based on their experience in the subject discipline. The Auditor selects the SpaceDOC project to be audited and the specific discipline topic to be audited using that project’s Space Assurance Requirements Document (SAR). SPA audits are typically conducted by each Safety and Mission Assurance (SMA) Discipline at least once a year unless another means of periodic discipline review is established such as monthly meetings with active prime contractor SMA disciplines.

• End-Product Audit (EPA): The End Product Audits are conducted by the Quality Assurance Discipline since the audit itself is a review of the Project’s “As Built” hardware documentation. The hardware documentation reviewed ensures that the hardware was built according to approved, released, and controlled Drawings, Assembly, Inspection, and Test Procedures. EPAs are typically conducted prior to a project’s System Acceptance Review (SAR) or Engineering Review Board (ERB) /SAR (ERB/SAR). This audit can take place up to a few hours before the projects (SAR) or ERB/SAR, but preferably it should take place at least a day or two prior to the SAR or ERB/SAR.

• In-Process Audit (IPA): The In-Process Audits are typically conducted by the Quality Assurance Discipline, but In-Process audits IPAs may also be directed to be conducted by other SMA Disciplines as well. IPAs are typically conducted by a SMA Discipline at the request of the SpaceDOC III Contract Control Board (CCB) due to the CCB having concerns regarding general on-going processes or special processes taking place at the SpaceDOC III Prime Contractor’s facility.

• Software Assurance Functional Audit: This audit is held prior to the software delivery to verify that all requirements specified in the Software Requirements Document have been met.

• Software Assurance Physical Audit: This audit is held to verify internal consistency of the software and its documentation, and their readiness for release.

• Software Assurance In-Process Audit: In-process audits of samples of the design are held to verify the consistency of the design, including:

a) Code versus design documentation

b) Interface specifications

c) Design implementation versus functional requirements

d) Functional requirement versus test descriptions

Note: S&MA personnel will be available to assist with surveillance activities as required, but generally on a part time basis.

5.2.3 Surveillance Support through the Research and Engineering Directorate (Code L)

Code L is responsible for supporting the SFSD with matters pertaining to design, development and operation of ground and flight hardware and software systems, with buyoff and acceptance of final deliverables. The Code L point of contact is the Chief Engineer or Lead Systems Engineer (LSE). The following are specific disciplines supported by Engineering:

• Systems Engineering

• Avionics

• Diagnostics and Instrumentation

• Fluids

• Thermal

• Power

• Software

• Mechanical Structures/Mechanical Systems

• Acoustics

• Operations

• Integration and Test

• Materials and Processes

• Propulsion

• Communications

Examples of detailed checklists of potential surveillance tasks for each discipline are found in Appendix B.

Note: These engineers will be available to assist with surveillance activities as required, but generally on a part time basis. A minimum of two weeks advanced notice is requested by Code L for adequate resource scheduling.

5.3 IMPLEMENTATION

5.3.1 Communication of Surveillance Approach to Contractor

Once the surveillance team has been established, this plan will be shared with the contractor and written descriptions of activities will be provided.

5.3.2 Communication within NASA GRC Surveillance Team

The PM and surveillance team work together to assess contractor performance and to assure project success. The following are general roles for the PM and the surveillance teams members:

• Become familiar with the Base/Delivery Order requirements, schedule, and deliverables

• Become familiar with the SpaceDOC III SOW, the content and acceptance criteria per the contract CDRL DIDs

• Create a mutual understanding and agreement on what is needed and how requirements should be interpreted

• Meet regularly to review status, future assignments and discuss issues.

5.3.3 NASA GRC Surveillance Team Interactions with the Contractor

The surveillance team is directed by the PM through the SpaceDOC III Control Board, or directly. Base/Delivery Order specific data, or data required for scheduled audits may be requested directly by the PM, CSO, or CE. Issues identified by the surveillance team members should be communicated to the PM, and the PM will initiate any follow-on action.

The following are general roles for surveillance team members:

• Perform review and independent analysis on contractor products.

• Monitor or witness contractor fabrication, assembly, test, or other activities to assess progress and problems.

• Identify potential issues to the PM. Issues can include misinterpretation of requirements, lack of progress per contractor plan, and risky methodology of performing work.

• Technical direction can only be given by the COR. Any change in scope affecting cost, schedule, or technical requirements is considered a contract change and must be approved by the CO.

APPENDIX A S&MA SURVEILLANCE SUPPORT CHECKLIST

EXAMPLE

Note to Project Manager: Use this Appendix to identify Base/Delivery Order -specific surveillance tasks you would like to have conducted by S&MA. Other items may be added to meet the needs of your Base/Deliver Order.

This Appendix may also be used to create regular surveillance plan audits by the S&MA Surveillance Team. The Team may use a selection of these items, or substitute or augment with others derived from past performance, current challenges, status or Base/Delivery Order needs.

Potential topics to be covered in a project-specific Surveillance Plan / SOW:

• Project Risks and Selected Surveillance Approach

• Surveillance Objectives

• Surveillance Status Meetings

• Surveillance Reporting Requirements

• Period of Performance

• Surveillance Resource Requirements

• Project Surveillance Tasks

POTENTIAL SURVEILLANCE TASKS BY S&MA DISCIPLINE

SUPPLIED BY GRC SAFETY & ASSURANCE TECHNOLOGIES DIRECTORATE (S&MA)

QUALITY ASSURANCE & EEE PARTS

1. Review major milestone document deliverables in the areas of Quality Assurance (QA) and EEE parts to assure that the project has adequately addressed applicable requirements and clearly stated the overall project philosophy and implementation strategy for these areas.

2. Provide support during scheduled surveillance assessments of the contractor, reviewing the areas of QA and EEE parts to assess whether contractor plans and procedures are being effectively implemented.

3. Review test plans to verify incorporation of applicable QA controls and monitor testing to assess whether plans are being effectively implemented.

4. Verify detailed inspection of COTS hardware upon receipt. (COTS hardware intended for flight use should be inspected for suitability for such use.)

5. Verify contractor uses appropriate inspection criteria, checklists and has build papers or travelers which control: Handling and routing sequence as hardware is assembled; as-built traceability details such as lot codes and serial numbers; sign-off by the person performing each operation; and inspections performed at key points of the assembly process

6. Verify that suitable tools are in use and, if required, are in current calibration (torque wrenches, wire strippers, instrumentation, etc.)

7. Verify that contractor has suitable handling, storage, and Electro-static Discharge (ESD) controls.

8. Verify contractor follows proper work area practices and controls (cleanliness, ESD, temperature & humidity, controlled access, procedures, configuration management, records)

9. Verify contractor has and follows rework procedures & re-inspection after rework, procedures for assembly, rework, inspection, etc., involving surface-mount components

10. Verify contractor has and follows procedures for ruggedizing COTS hardware

11. Verify contractor has and follows workmanship standards for each of the various assembly activities

12. Verify contractor has and follows procedures for bakeout prior to conformal coating

13. Verify that contractor’s QA activities are in compliance with the contractor’s Product Assurance Plan

RISK MANAGEMENT

1a. Regularly attend contractor risk management working group meetings to gain government insight into effectiveness of contractor risk management process.

1b. Periodically attend contractor risk management working group meetings to gain government insight into effectiveness of contractor risk management process.

2. Periodic interviews with government project managers (whose projects fall under SpaceDOC III) to determine their satisfaction with, or need for assistance with, risk management activity on the project.

3. Periodic audits of contractor personnel working at project level and below to obtain insight on their input status to the risk management process.

4a. Request periodic updated 'risk list' from contractor for individual projects, to obtain insight into risk identification and mitigation status.

4b. Request regular updated 'risk lists' from contractor for individual projects, to obtain insight into risk identification and mitigation status.

5. Require 8a contractor to deliver a Risk Management Plan.

6. Conduct continuous risk management training courses for projects under SpaceDOC III who have not had the training.

7. Facilitate risk management implementation activities for projects.

8. Periodically attend individual SpaceDOC III project meetings to gain insight into risk management activity at contractor project level.

9. Request regular SpaceDOC III management reports on project risk status (like the Program Management Council does for in-house projects.)

RELIABILITY, AVAILABILITY & MAINTAINABILITY (RAM)

1. Providing guidance, assistance, or review to the project of RAM activities planned in Product Assurance Plan.

2. Setting RAM requirements for the project.

3. Providing guidance to the project for implementing RAM activities: analysis, Failure Modes and Effects Analysis (FMEA), limited life items, maintainability, Probabilistic Reliability Assessment, etc.

4. Provide clarification of requirements and technical opinion to the SpaceDOC III contractor

5. Conduct audits or assessments of the contractor activity

a. Desk Audits

b. Face-to-Face Audits

SYSTEM SAFETY

1. Review Project Design Documentation and Drawings to ensure consistency with launch vehicle and flight and ground safety requirements, and to remain cognizant of project progress.

2. Review the various phased Flight and Ground Safety Data Packages to ensure launch vehicle and other applicable safety requirements are met. Interface with S&MA lead safety reviewer where necessary. Coordinate review comments between S&MA and SpaceDOC III safety.

3. Review the phased Safety Review presentation packages for completeness and soundness of material presented. Inform the NASA GRC Project Manager in the event that additional resources are required.

4. Attend SpaceDOC III safety meetings and maintain correspondence with the Project Safety Engineer Inform the NASA GRC Project Manager of any issues and progress.

5. Provide support to Project safety TIMs or Payload Flight or Ground Safety Reviews

6. Review Safety Verification documentation including analyses and test and inspection reports for accuracy, completeness, and agreement with the associated Hazard Controls and Safety Verification Methods from the Safety Data Package Hazard Reports.

SOFTWARE ASSURANCE

1. Software Management and Control – Ensure software risks have been identified and the necessary level of software control has been established. Advise NASA PM on whether software is being managed in accordance with the assigned level of software control

2. Software Documentation Review – Advise NASA PM whether contractor software documentation is consistent with flight/ground software requirements and whether documentation is adequate for software to meet functional and performance objectives.

3. Software Safety – Verify all safety critical software has been identified in requirements, design and code. Review test plans, monitor testing and review test reports that verify/validate safety critical software.

4. Major Milestone Reviews -- Review major milestone presentation packages with respect to software and software assurance. Provide comments, advice, and guidance to NASA PM, including an assessment of how effectively the contractor is implementing their software assurance program.

5. Software Review and Inspections -- Participate in some software reviews and inspections carried out by the contractor’s Software Assurance group over the software development life cycle. Provide input to system and software formal reviews and periodically monitor walk-through and inspections. Advise NASA PM of progress and concerns.

6. Software Verification and Validation (V&V) – Review software V&V plan or software verification matrix to assure coverage of all applicable software requirements. Monitor nominal and off-nominal mission simulation and functional testing (or review test reports). Verify applicable test procedures have been followed and that compliance with software requirements has been adequately demonstrated.

7. Software Product Acceptance – Ensure contractor Software Assurance group completes functional and physical configuration audit to determine if software products are ready for release or need more work. Review audit findings and reports to evaluate the accuracy of the audit results. Provide recommendations to

NASA PM.

APPENDIX B CODE L SURVEILLANCE SUPPORT CHECKLIST

EXAMPLE

Note to Project Manager: Use this Appendix to identify Base/Deliver Order -specific surveillance tasks you would like to have conducted by Code L. Other items may be added to meet the needs of your Base/Delivery Order.

This Appendix may also be used to create regular surveillance plan audits by the Code L Surveillance Team. The Team may use a selection of these items, or substitute or augment with others derived by past performance, current challenges, status or project needs.

Potential topics to be covered in a project-specific Surveillance Plan / SOW:

• Project Risks and Selected Surveillance Approach

• Surveillance Objectives

• Surveillance Status Meetings

• Surveillance Reporting Requirements

• Period of Performance

• Surveillance Resource Requirements

• Project Surveillance Tasks

POTENTIAL SURVEILLANCE TASKS BY ENGINEERING DISCIPLINE

SUPPLIED BY GRC RESEARCH AND ENGINEERING DIRECTORATE

General Description

The overall purpose of the Engineering Surveillance support is to assess the compliance of the design and to identify technical risk areas and mitigation options. The design compliance is evaluated against the project requirements as provided (science, carrier / interface, environmental, product assurance, performance, safety). Engineering disciplinary support can be provided as a course of the planned staged Project milestone reviews, such as Requirements Definition, Preliminary Design, Critical Design, Verification Readiness and Pre-Ship reviews. Disciplinary engineering nominally supports the reviews as members of the Review Board. Engineering support can also be utilized to review and evaluate the acceptability of Data Deliverables (Project documents, analyses, test plans, and test reports).

Engineering disciplinary support can also be provided to:

• Attend key internal Project meetings and TIMs in order to assess the engineering soundness of plans and approach.

• Evaluate and assess Project design approach, direction and issues at intervals prior to major milestone reviews.

• Augment the Project Management Team in regular and continuous review and assessment of contractor’s progress, risks, deliverables, design and verifications.

The additional support is usually evaluated and applied to the major Project risk areas such as software development, or ‘key’ diagnostic development. The focus of all assessments is on the compliance of the design to the requirements.

Another aspect of Surveillance or Risk Mitigation is the use of engineering to perform separate design or verification activities in pursuit of risk mitigation – design options or perform independent verification and validation (such as structural analysis, thermal analysis, software development and testing). These activities are more intensive and are directed to major risk areas.

The following describes, by discipline, some of the activities and roles that might be performed as a part of surveillance. This section listing is not intended to be comprehensive or obligatory but only a guide and starting point for considering what is needed for a given project and project phase.

Systems Engineering

Evaluates the Systems approach focusing on requirement definition, requirement verification and the system and subsystem development approach.

1. Evaluate the integration planning and implementation.

2. Assess the flow of technical requirements / directions for the system / subsystem developments.

3. Assess Risk Management Plan and Reports and, conduct and issue independent risk assessments.

4. Conduct system engineering assessment of subsystem technologies.

5. Support definition and implementation of design reviews, review boards and review dispositions.

6. Evaluate technical development sections of the project plan and assist in the development of the NASA Project Plan.

7. Participate in Program and Project Reviews, TIMs, workshops, and other Program or Project working groups.

Avionics, Diagnostics, and Instrumentation

Evaluates electrical design of the transmission, conditioning and storage of the system instrumentation signals, and the electrical design of the system control electronics and control methods.

1. Review data system specifications and requirement verifications.

2. Review signal conditioning specifications and circuit design (schematics, block diagrams).

3. Review signal wiring, bundling, grounding and routing scheme (integrity, noise susceptibility assessment).

4. Review signal error analysis (data accuracy compliance) and circuit analysis.

5. Review control circuitry and performance capabilities (schematics, block diagrams, analyses and test reports).

6. Evaluate data transmission rate and storage capacity acceptability.

7. Conduct independent design and / or analysis or breadboarding of specified signal circuitry or control circuitry.

Fluids

Evaluates the fluid (liquid, gas) storage, distribution, manipulation, and/or control subsystem designs.

1. Review subsystem design and verifications (design drawings, block diagrams, verification data).

2. Review subsystem design specifications.

3. Review design / safety devices/ component acceptability to flow and pressure conditions (stress and strength, pressure vessel, etc).

4. Review sealed system integrity and sealing methods (leak rate compliance, verification methods).

5. Review analyses (Pressure conditions –design, maximum, nominal; Stress –sealed container, pressure vessel).

6. Conduct independent analyses

a. Stress

b. Fluid Dynamics

Thermal

Evaluates the design of the thermal control subsystem (active or passive) and assesses thermal verifications.

1. Review system thermal loading, thermal requirements and design approach.

2. Review thermal analyses reports and thermal test reports and assess methodology and acceptability.

3. Evaluate thermal verification approach and system/ subsystem / component test program.

4. Conduct independent thermal analysis of system / subsystem.

5. Conduct independent thermal design and verification.

Power

Evaluates the design of the power distribution, control and protection.

1. Review power system, power components, control components specifications and requirement verifications.

2. Review power analyses and protection analyses (protection types, protection sizing, wire derating).

3. Review grounding and bonding methodology and verifications.

4. Evaluate overall EMI/EMC design approach and acceptability.

5. Conduct independent design and breadboarding of specified power circuitry and control circuitry.

Software

Evaluates the software design, management approach, development approach, and verifications.

1. Review software management and development approaches and assess acceptability and options.

2. Review software requirements for completeness, traceability and ability to be verified.

3. Assess language options and choices, development tools and configuration management approach.

4. Assess verification plans and methods.

5. Review and evaluate design, design documentation, architecture, computer software configuration items, computer software components and firmware.

6. Review and inspect documentation, code, test plans and test results per Software Formal Inspection Methodology.

7. Estimate percent completion.

8. Witness system level software testing.

9. Review software Corrective And Preventative Actions (CAPAs) after system integrated tests.

10. Perform independent validation and performance testing (on specific Computer Software Configuration Items (CSCIs), Computer Software Components (CSCs)).

Mechanical Structures / Mechanical Systems

Evaluates the mechanical structures for acceptability for loads, dynamics, functionality and packaging.

1. Review design drawings, materials and manufacturing specifications.

2. Review Safety Critical Structures data package and Fracture Control.

3. Assess design for human factors, manufacturing and assembly.

4. Assess kinematic analyses and evaluate operational loads, clearances and constraints.

5. Assess load conditions and evaluate structural analyses.

6. Assess Mass & Properties reports and approach.

7. Evaluate elements for environment survivability and compatibility (launch loads, vibration environment, microgravity acceleration environment) and evaluate acceptability of ‘ruggedization’ of commercial / sensitive items.

8. Evaluate structural and mechanical operational life.

9. Evaluate environmental, qualification, and flight acceptability verification test program.

10. Conduct independent analyses.

a. Structural Stress

b. Dynamic response

Acoustics

Evaluate the design and approach for acoustic noise generation and control, and the test program.

1. Review design for potential noise-generating systems / devices.

2. Assess the acoustic control approach.

3. Evaluate the noise-dampening design features.

4. Evaluate system, subsystem, device test plans and test reports.

5. Assess compliance to payload acoustic limits /requirements.

Operations

Evaluate the operational plans and approach, and the design of Mission Operations Command & Data systems and ground operational support units

1. Review operational plans and documents.

2. Evaluate the design and verification of the Mission Operations command & data systems and the ground operational support units.

3. Evaluate mission operational staffing, roles and responsibilities, and training / certification plans. Assess staffing schedules and duty hours to ensure staff have adequate rest.

4. Assess nominal, malfunction and contingency operations plans and procedures, and timeline.

5. Review Payload Flight Rules, Payload Regulations, Training Plans and Training products.

6. Assess ground system performance, and reliability / maintenance plans.

Integration and Test

Evaluate the qualification and acceptance test plans and approach.

1. Review qualification and acceptance plans and documents.

2. Evaluate performance parameters critical to operational effectiveness.

3. Trace linkages between operational requirements and test criteria.

4. Evaluate plans and approach against project schedule, performance, and cost goals.

5. Identify potential project risks and evaluate mitigation strategies.

6. Determine that the scope of testing is adequate and objectives are clear and complete

7. Criteria for entry and exit of a test are clearly identified.

8. Determine whether the test plan includes verification and validation of test equipment, test software, test databases, and facilities.

9. Determine whether sufficient test points have been obtained to adequately characterize the item, function, or performance per the objectives.

10. Determine whether potential damage sources and conditions were identified.

11. Assess whether configuration changes have been clearly identified and adequately described.

12. Assess whether test results were properly evaluated by inspection or analysis to either prove or disprove that the test produced the intended results, and whether it produced unexpected results.

13. Assess the levels of data review required to determine whether or not the success criteria have been achieved.

14. Assure that plans are in place to handle test failures or anomalies.

15. Assure that Qualification and Acceptance test procedures have been validated during prior development tests.

Materials and Processes

1. Ensure that the specifications provided by the NASA PM in procurement documents are accurate and the latest revisions available.

2. Ensure that non-compliant materials are resolved in accordance with the Space Assurance Requirements (SAR).

3. Provide guidance to the PM in the formulation of requirements for a materials and processes program.

4. Provide the technical expertise in the review of M&P for flight applications and recommend approval or disapproval. Where a material has been disapproved, assist with recommendation for an alternate material.

5. Provide and maintain a recommended flight approved materials list for SpaceDOC III project usage (currently being developed). Verify contractor is using.

6. Inform PM and S&MA safety engineer regarding use of a safety-related non-compliant material when unable to resolve usage with the contractor and/or project office. Convene SERB to resolve issue.

7. Review Project/contractor developed material list, Materials Usage Agreements (MUA) and provide comments at the appropriate reviews.

8. Provide guidance to Project/contractor personnel for the purpose of gaining a recommendation for a certification letter.

9. Assist with disposition of materials-related safety verifications.

Propulsion

Evaluates the design of the propulsion subsystem and assesses propulsion verifications.

1. Review and evaluate propulsion requirements, verification, and test plans.

2. Review and evaluate thruster design and drawings.

3. Work with S&MA to determine government inspection points of thruster.

4. Review test data, including throttle level, thrust, and impulse.

5. Witness thruster testing.

6. Assess interface requirements and design. In the case of electric propulsion this should include EMI effects on the vehicle.

Communications

Evaluate and review the design and approach of the communication system and subsystems.

1. Review communication system and subsystems specifications and verification requirements.

2. Review and evaluate communication architecture and communication concept of operations including both space and ground for Telemetry, Tracking, and Command (TT&C) and communication system.

3. Review and evaluate communication design such as transceiver, software defined radio, antenna, gimbal system, and optical.

4. Review and evaluate communication analyses such as link budget analysis and frequency/spectrum analysis.

5. Review and verify implementation of Communication Systems standards (i.e.

Consultative Committee for Space Data Systems (CCSDS), Internet Protocol Security (IPSec), etc.).

6. Witness communication subsystem and system level testing including RF compatibility testing.

7. Review communication test results and reports.

APPENDIX C ACRONYM LIST

ACOR Alternative Contracting Officer’s Representative CDR Critical Design Review CDRL Contract Data Requirements List CM Configuration Management CO Contracting Officer COR Contracting Officer’s Representative COTS Commercial Off-the-shelf CSC Computer Software Components CSCI Computer Software Configuration Item CSO Chief Safety and Mission Assurance Officer DCAA Defense Contract Audit Agency DCMA Defense Contract Management Agency DID Data Item Description DO Delivery Order Code L Research and Engineering Directorate EEE Electronic, Electrical, Electromechanical EMC Electromagnetic Compatibility EMI Electromagnetic Interference ERB Engineering Review…