Attachment A6_HLS FY21 IPEP v1.0 HLS-MOA-001_Baseline.pdf

The electronic version is the official approved document.

Verify this is the correct version before use.

Human Landing System FY21 IV&V Project Execution Plan May 2021

HLS-MOA-001

BASELINE RELEASE

National Aeronautics and Space Administration RELEASE DATE: MAY 05, 2021

HUMAN LANDING SYSTEM (HLS) PROGRAM

FY21 SOFTWARE INDEPENDENT VERIFICATION &

VALIDATION (IV&V) PROJECT EXECUTION PLAN

(IPEP)

VERSION 1.0

Publicly Available: Release to Public Websites Requires Approval of HLS Program, IV&V Program, and approval via the Scientific and Technical Information (STI) process.

Revision: Baseline Release Document No: HLS-MOA-001

Release Date: May 05, 2021 Page: 2 of 31

Title: FY21 HLS Program Software IPEP v1.0

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

REVISION AND HISTORY PAGE

Revision No.

Change No.

Description Release

Date

- HLS-

C0159

Baseline Release (Reference HCB Outside of Board Request, dated 03/30/21)

05/05/21

Release Date: May 05, 2021 Page: 3 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE OF CONTENTS

SECTION PAGE

1.0 INTRODUCTION

1.1 PURPOSE

1.2 INTENDED AUDIENCE

1.3 CHANGE AUTHORITY/RESPONSIBILITY

1.4 DOCUMENT ORGANIZATION

2.0 IV&V OVERVIEW

2.1 IV&V GOALS AND OBJECTIVES

2.2 IV&V APPROACH

2.3 IV&V SCOPE AND FOCUS

3.0 ROLES, RESPONSIBILITIES, AND INTERFACES

3.1 ARTEMIS IV&V PROGRAM

3.2 HLS IV&V TEAM

3.3 PROGRAM PERSONNEL

4.0 IV&V PRODUCTS AND COMMUNICATION AND REPORTING METHODS

4.1 IV&V PRODUCTS

Assurance Conclusions Monthly Reports Lifecycle Review Presentations Technical Issue Memorandums (TIMs) Risks Item Tracking, Monitoring, And Escalation

4.2 IV&V COMMUNICATION AND REPORTING METHODS

Routine Tag-ups Lifecycle Review Presentations Agency/Mission Directorate/Center Management Briefings Development Reviews and Working Groups Support

APPENDIX

APPENDIX A: IV&V PBRA SCOPE AND FOCUS ASSESSMENT

APPENDIX B: IV&V RBA RESULT

APPENDIX C: IV&V HERITAGE REVIEW

APPENDIX D: TECHNICAL SCOPE & RIGOR (TS&R)

APPENDIX E: REFERENCE DOCUMENTATION

APPENDIX F: ACRONYMS

APPENDIX G: FY21 IV&V EFFORTS

APPENDIX H: FY22 AND BEYOND IV&V EFFORTS

APPENDIX J: FORWARD WORK

Release Date: May 05, 2021 Page: 4 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE OF TABLES

TABLE 2-1: TARGETED ARTIFACT TYPES FOR VERIFICATION AND VALIDATION

TABLE 3-1: IV&V TEAM AND PROGRAM INTERFACES

TABLE 3-2: PROGRAM CONTACT INFORMATION

TABLE 4-1: TIM SEVERITY RATING AND DESCRIPTION1

TABLE 4-2: MILESTONE REVIEW IV&V PRESENTATIONS

TABLE 4-3: ADDITIONAL REPORTING EVENTS

TABLE E-1: RELEVANT DOCUMENTATION

TABLE F-1: ACRONYMS

TABLE J-1: FORWARD WORK ITEMS

TABLE OF FIGURES

Figure 3-1: Artemis IV&V Program Structure Figure 3-2: IV&V Team and Program Interfaces

Release Date: May 05, 2021 Page: 5 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

1.0 INTRODUCTION

1.1 PURPOSE

This IPEP has two primary purposes. First, it describes the overall HLS IV&V project and defines the basic agreements for the partnership between the HLS IV&V Team (hereinafter referred to as the IV&V Team), the HLS Program (hereinafter referred to as the Program), and the HLS Commercial Partners (hereinafter referred to as the Partners). These agreements include roles and responsibilities, communications paths, IV&V products, IV&V reporting methods, and artifacts anticipated to be shared between IV&V, the Program, and the Partners. Second, the IPEP serves as the operational plan for the software IV&V efforts for the HLS Program being performed by NASA’s IV&V Program.

The decision was made to remove the signature page from the IPEP due to the Program including the IPEP in the Program’s configuration management system and approving the document through the HLS Control Board (HCB). In approving this document, Program personnel understand their concurrence reflects the agreements identified within the body of the document, excluding the appendices.

1.2 INTENDED AUDIENCE

The intended audience of this document includes:

• NASA IV&V Program staff, particularly the IV&V Program Manager, IV&V Office (IVVO) management, the Artemis IV&V Program and project teams, and IV&V Mission Protection Services (MPS)

• Program personnel, particularly the HLS Program Manager, the HLS Program IV&V POC, the HLS Partner IV&V POCs, the HLS Insight Teams, other appropriate HLS Technical Managers, the Center Safety and Mission Assurance (SMA) Director, the HLS Chief Safety Officer (CSO), and other members of the HLS safety community, HLS Information Security personnel, appropriate members of the Partners’ software engineering leadership and development teams

1.3 CHANGE AUTHORITY/RESPONSIBILITY

Proposed changes to this document shall be submitted via a Change Request (CR) to the appropriate Human Landing System Control Board for consideration and disposition.

All such requests will adhere to the Human Landing System Configuration and Data Management Plan, documented in HLS-PLAN-004.

The appropriate NASA Office of Primary Responsibility (OPR) identified for this document is the HLS Systems Engineering & Integration (SE&I) Office.

1.4 DOCUMENT ORGANIZATION

The IPEP is divided into two major parts: the document body and the appendices. The document body describes the overall IV&V project and defines the basic agreements for the partnership

Release Date: May 05, 2021 Page: 6 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021 between the IV&V Team, the Program, and the Partners. Once coordinated and approved, the basic agreements in the document body are not expected to change.

The second part of the document, the appendices, focuses on the fiscal year (FY) activities for the IV&V efforts. The appendices contain data that are more dynamic in nature and are expected to change over the course of the project. The appendices include the results of, or a reference to, the IV&V Heritage Review, IV&V assessments of potential Program and Partner software risks for IV&V effort prioritization, and detailed information for each planned execution year, including items such as IV&V goals and objectives, schedule, and risks.

This is Version 1.0 of the HLS IPEP. Changes to the agreements in the body of this document (Sections 1-4) will trigger an increase in the base version number (i.e., Version 2.0) and subsequent CR review and approval by the Program. This IPEP will be revisited and updated as necessary. At a minimum, a new version of this IPEP will be released for CR prior to the start of each fiscal year (FY) and the base version number will be increased by one (e.g., Version 2.0).

Editorial revisions to the body of the IPEP and any revisions to the appendices which occur outside of the FY update will result in an update to the decimal part of the version number (e.g., Version 1.1). Draft versions of the IPEP will be marked as “DRAFT.”

2.0 IV&V OVERVIEW

2.1 IV&V GOALS AND OBJECTIVES

The IV&V Team will provide objective evidence and recommendations to increase the assurance that the software will operate reliably, safely, and securely in support of critical capabilities in the expected operating environment under nominal and defined off-nominal conditions. The IV&V Team will document any identified issues and risks to this assurance and will work with the Program and Partners to advance these issues and risks to resolution.

IV&V analyses are intended to add evidence-based assurance that minimizes the overall risk of HLS software preventing the Artemis missions from occurring safely and successfully.

Specific IV&V project goals and objectives for each FY are identified in the appendices.

2.2 IV&V APPROACH

The IV&V approach will consist of validation- and verification-related analyses. Validation and verification are described further below, including the artifact types generally required for specific analysis objectives.

Validation-related analyses strive to assure the system software satisfies the user’s capability needs under operational conditions. These analyses evaluate the attributes, features, and qualities exhibited by the Program’s and Partners’ development artifacts for each selected critical capability, in the context of the following three questions defined in NASA IV&V System Level Procedure (SLP) IVV 09-1, Independent Verification and Validation Technical Framework:

1) Will the software do what it is supposed to do?

2) Will the software not do what it is not supposed to do?

3) Will the software respond appropriately to adverse conditions?

http://www.nasa.gov/centers/ivv/ims/slps/index.html

Release Date: May 05, 2021 Page: 7 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

Verification-related analyses determine whether the products of each development activity are of high quality (e.g., are clear, consistent, verifiable, correct, and complete) and fulfill the requirements or conditions imposed by a previous development activity.

Specific assessments and analyses that the IV&V Team may perform will use the following types of Program and Partners’ artifacts: Concept Documentation, Safety and Reliability Documentation, Requirements Documentation, Design Documentation, Test Documentation, Implementation Documentation, Security Documentation (including Information Technology and system security artifacts), and Operations and Maintenance Documentation. The IV&V Team may perform various types of static code analysis on HLS software. The IV&V Team may also perform independent testing using simulators, test environments or other test systems provided by the IV&V Team, the Program, or the Partners to help provide further evidence of HLS software assurance.

Artifact types the IV&V Team may need to support verification and validation related analyses as well as risk assessments are listed in Table 2-1, below. In the event any of these artifact types cannot be provided to the IV&V Team, and/or the IV&V analyses are required to be performed on-site at any of the Partners’ locations, the IV&V PM, Program IV&V POC, and the appropriate Partner IV&V POC will closely coordinate any impacts and document any risks to the performance of the IV&V efforts. The IV&V Team does not drive or mandate the creation of specific software artifacts. While electronic copy in original format is preferred, the IV&V Team will work with available information and content in most formats, as long as the artifacts provided include the data necessary to verify and validate the Partners’ software and draw credible assurance conclusions on the software’s mission suitability.

Results of the verification and validation will serve as a basis for assessing the software’s ability to perform or support expected system and software behaviors for critical capabilities.

Typical outputs of the verification and validation related analyses will include assurance conclusions, issues, and risks. Refer to Section 4 of this document for additional information on these products. For additional information regarding verification and validation related analyses, see NASA IVV 09-1.

Release Date: May 05, 2021 Page: 8 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE 2-1: TARGETED ARTIFACT TYPES FOR VERIFICATION AND VALIDATION

Artifact Type Need/Applicable Analysis

Operations Concept Document/Data Verify and Validate Concept Documentation

Early concept/design review documentation/data

Verify and Validate Concept Documentation

Level 1 requirements Verify and Validate Requirements

Mission Requirements Document Verify and Validate Requirements

Spacecraft Element Requirements Document

Verify and Validate Requirements

Software Requirements Document Verify and Validate Requirements

Interface Requirements Documents Verify and Validate Requirements

Interface Design Documentation Verify and Validate Design / Verify and

Validate Documentation

Traceability Related Data (Requirement Levels 2 – 5)

Verify and Validate Requirements / Verify and Validate Test Documentation

Hazard Analyses (PHA, FTAs, etc.)

Verify and Validate Requirements / Verify and Validate Design / Verify and Validate Test

Documentation / Verify and Validate Implementation

System Test Plan Verify and Validate Test Documentation

System Test Cases Verify and Validate Test Documentation

Build Level Test Plan Verify and Validate Test Documentation

Build Level Test Cases Verify and Validate Test Documentation

Test Scripts Verify and Validate Test Documentation

Integration Test Plans Verify and Validate Test Documentation

Integration Test Cases Verify and Validate Test Documentation

Traceability related data (showing traceability from requirements to test cases) Verify and Validate Test Documentation

Software Design Documentation Verify and Validate Design

Software Design Models Verify and Validate Design

Source Code Verify and Validate Implementation

Software Build delivery/release packages/Version Description documentation/data

Verify and Validate Implementation /

Verify and Validate Documentation

Test results (at varying levels including build level, integration level and system level)

Verify and Validate Requirements /

Verify and Validate Implementation

Release Date: May 05, 2021 Page: 9 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE 2-1: TARGETED ARTIFACT TYPES FOR VERIFICATION AND VALIDATION

Artifact Type Need/Applicable Analysis

Discrepancy reports from test activities Verify and Validate Implementation

Traceability related data (showing traceability from requirements to design – to code to test) Verify and Validate Test Documentation

Compile and build procedures Verify and Validate Implementation /

Verify and Validate Documentation

Build environments Verify and Validate Implementation

Test environment resources (e.g., identification of and supply of the simulators, emulators, and supporting configuration items/data and tools)

Verify and Validate Implementation

System Security Plan Verify and Validate Security Posture

Security CONOPS Verify and Validate Security Posture

Software Security Requirements (if separate)

Verify and Validate Security Posture

Security Design and Architecture Verify and Validate Security Posture

System’s FIPS-199 Classification Verify and Validate Security Posture

System’s FIPS-200 Classification Verify and Validate Security Posture

System Security Test Plan (if separate) Verify and Validate Security Posture

System Security Test Cases (if separate)

Verify and Validate Security Posture

Threat Assessments Verify and Validate Security Posture

Project Protection Plan Verify and Validate Security Posture

NIST 800-53 Control Selection and Rationale

Verify and Validate Security Posture

Interconnection Security Agreements Verify and Validate Security Posture

Security Risk Assessment Verify and Validate Security Posture

Network Models Verify and Validate Security Posture

System Security Management Plan Verify and Validate Security Posture

Communication Security Plan Verify and Validate Security Posture

Key Management Plan Verify and Validate Security Posture

Information Types Workbook Verify and Validate Security Posture

Release Date: May 05, 2021 Page: 10 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

2.3 IV&V SCOPE AND FOCUS

As part of Software Assurance, IV&V plays an important role in the Agency’s overall software and security risk mitigation strategy throughout the entire system lifecycle to improve the safety, reliability, quality, and security of software systems. Due to resource constraints, it is important that IV&V identify and prioritize those parts of the system that are most critical and must receive appropriate assurance services relative to focus and scope. Scope is the part of the system or systems for which IV&V is applicable. Focus is the set of in-scope system capabilities and software entities that IV&V will cover. In some cases, focus is smaller than scope. There are a variety of reasons why this is the case. Where this occurs, the rationale will be documented accordingly.

The IV&V Program seeks to define focus through an understanding of the risk profile for the Program and its Partners by performing software risk assessments. IV&V teams use the IV&V Project-Based Risk Assessment (PBRA) process to assess the required system capabilities for the mission, that are enabled by or dependent on software, in terms of impact of a defect and likelihood of a defect. Impact and likelihood are evaluated considering IV&V-defined scoring criteria. This prioritization ensures application of IV&V resources on the system capabilities with the highest software risk. IV&V places a high degree of emphasis on the capabilities that are necessary for successful mission operation.

Critical software capabilities have been identified for HLS by IV&V; these capabilities were used to develop a set of assurance objectives which captured the items of risk or questions the IV&V Team identified regarding each specific capability. These assurance objectives have been prioritized and will be used to drive the analysis that IV&V will perform to provide added assurance for HLS software. Additional information regarding what an assurance objective is can be found in process document S3106.

In addition to the PBRA process, IV&V teams use the Risk Based Assessments (RBA) process which is used to select critical software entities to further plan and scope the IV&V project. The entity-to-capability mapping produced by this phase provides a view of the system that serves as a useful tool for discussing and deciding where to apply IV&V effort.

The IV&V Team will share the PBRA and RBA results with Program and Partners, and input and feedback on this data from the Program and Partners is encouraged. The IV&V Team will revisit the risk assessments every six months (or more frequently, if warranted), and any changes to this data will be communicated to the Program and Partners as well as documented in the IPEP.

PBRA results are provided in Appendix A, and RBA results are provided in Appendix B. Additional information on the PBRA and RBA can be found in the PBRA and RBA process document S3106.

Applying IV&V’s assurance activities on these high-risk capabilities discussed above is called Capability Based Assurance (CBA). IV&V’s goal is to expand IV&V assurance to as many areas as possible that pose significant risk to mission success. As part of IV&V’s analysis process, one of the early steps for an analyst working in a new capability area is to gain system understanding. As the analyst gains system understanding, they expand their understanding of risks associated with the capability. This understanding of risk will then be used to plan and execute analyses at appropriately targeted levels of rigor. This process the IV&V Team uses to determine what analysis needs to be provided for specific areas of risk in order to add assurance is described in detail in Appendix D. IV&V’s distribution of effort, focus, assurance strategy and https://www.nasa.gov/sites/default/files/atoms/files/s3106_ver_e_-_04-26-2018.pdf https://www.nasa.gov/sites/default/files/atoms/files/s3106_ver_e_-_04-26-2018.pdf

Release Date: May 05, 2021 Page: 11 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021 approach are adapted as appropriate throughout the IV&V operating period based on evolving risk insight to maximize overall assurance for the mission.

3.0 ROLES, RESPONSIBILITIES, AND INTERFACES

To facilitate successful execution of the IV&V efforts as described in this plan, various roles, responsibilities, and interfaces are maintained. These roles and responsibilities can be described in terms of personnel within the IV&V Program and personnel within the HLS Program and Partners. The subsections below describe these roles and responsibilities.

3.1 ARTEMIS IV&V PROGRAM

The Artemis IV&V Program consists of 6 IV&V Project teams and a modeling / independent testing team. The Artemis IV&V structure tree can be found in Figure 3-1 below. The 6 Artemis related projects that makeup the Artemis IV&V Program are Orion, Exploration Ground Systems (EGS), Space Launch System (SLS), Gateway, Mission Control Center (MCC) and HLS. At the Artemis IV&V level, the teams are in continuous communication prioritizing work across the Artemis IV&V teams as well as working integration related tasks between the teams.

The HLS IV&V Team works with all the projects in the Artemis IV&V Program in order to communicate integrated risks and issues across the projects. Artemis IV&V Program leadership as well as the Artemis IV&V project leadership interacts with various external stakeholders including but not limited to: various Program and Project personnel, the Exploration Systems Development (ESD) Cross-Program Systems Integration Director, ESD Cross-Program Integration Team (CPIT) Lead, ESD Integrated Avionics and Software Integrated Task Team (IAS ITT) Lead, Advanced Exploration Systems (AES) Integrated Avionics and Software Working Group Lead, and various SMA personnel.

Figure 3-1: Artemis IV&V Program Structure

3.2 HLS IV&V TEAM

The HLS IV&V Team primarily consists of an IV&V Project Manager (PM), an IV&V Project Lead (PL), a Lead IV&V Engineer (LE), an IV&V analyst team, and groups within the NASA IV&V Program that support the HLS IV&V Team.

The IV&V PM is a civil servant who serves as the primary interface with the Program in support of the IV&V efforts. The IV&V PM is responsible for the overall leadership and direction of the

Release Date: May 05, 2021 Page: 12 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

IV&V efforts. This IPEP is prepared and maintained by the IV&V PM. The IV&V PM coordinates the creation and maintenance of this document with affected individuals and organizations (within the IV&V Program as well as with the HLS Program). The IV&V PM is responsible for performing project management, tracking and oversight; and conducting risk management in support of the IV&V efforts. The IV&V PM, in concert with the PL and LE, is responsible for establishing the goals and objectives of the IV&V efforts through the use of assurance objectives as well as ensuring that the commitments with the Program as defined in this plan are met.

The IV&V PL is the contractor lead of the HLS IV&V Team. The IV&V PL is responsible for delegation of responsibilities and work assignments for the contractor analysts on the HLS IV&V Team.

The IV&V LE is a civil servant who works with the PM and PL on the technical direction of the HLS IV&V project. The LE is designated to be the primary interface for the HLS Program and HLS Partners on technical engineering issues and solutions, and is responsible for communicating appropriate technical information to the HLS IV&V team.

The IV&V analyst team performs the verification and validation related analyses. At times members of the IV&V analyst team may interface with the Program POC, Partner POCs and other individuals within the Program and Partners directly.

A variety of different IV&V Program groups may support the IV&V Team, and personnel from these groups may interact with Program, Safety, and Security personnel, as well as Partner personnel. These interactions will be coordinated thru the Program POC, Partner POC when applicable, and the IV&V PM. Supporting groups include the IV&V Program Independent Test Capability (ITC) Team, the IV&V Software Assurance and Tools (SWAT) Team, and the IV&V Program MPS Group. The IV&V MPS support will be documented in a separate Memorandum of Agreement (MOA).

Formal and informal interfaces between IV&V personnel, Program personnel, and Partners’ personnel are indicated in Figure 3-2 below. Section 4, IV&V Products and Communication and Reporting Methods, describes the formal and informal communication paths in more detail.

Development of informal interfaces is encouraged to enhance communications by addressing questions about potential findings at the lowest levels.

Release Date: May 05, 2021 Page: 13 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

Figure 3-2: IV&V Team and Program Interfaces

TABLE 3-1: IV&V TEAM AND PROGRAM INTERFACES

NASA IV&V Program

Position Name Contact Information

NASA IV&V Director Gregory Blaney 304-367-8387

Gregory.D.Blaney@nasa.gov

IV&V Office Lead Wes Deadrick 304-367-8329

Wesley.W.Deadrick@nasa.gov

Artemis IV&V Program Manager Mike Facemire 304-367-8265

Michael.L.Facemire@nasa.gov

HLS IV&V PM Justin Smith 681-753-5217

Justin.L.Smith@nasa.gov

HLS IV&V PL Greg Stine 304-685-3716

William.Stine@nasa.gov mailto:Gregory.D.Blaney@nasa.gov mailto:Wesley.W.Deadrick@nasa.gov mailto:Michael.L.Facemire@nasa.gov mailto:Justin.L.Smith@nasa.gov

Release Date: May 05, 2021 Page: 14 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE 3-1: IV&V TEAM AND PROGRAM INTERFACES

NASA IV&V Program

Position Name Contact Information

HLS IV&V LE Bill Stanton 304-367-8392

William.M.Stanton@nasa.gov

MPS Lifecycle Lead Keenan Bowens 681-209-7118

Keenan.L.Bowens@nasa.gov

MPS Integration Lead Steve Husty 681-753-5207

Stephen.Husty@nasa.gov

3.3 PROGRAM PERSONNEL

The Program as well as each Partner will provide an IV&V POC for formal interactions as well as informal interactions between the IV&V Team and the Program, and the IV&V Team and the Partner. The Program IV&V POC will facilitate the IV&V tasks to be performed through coordination between Program personnel, Partner IV&V POC, and the IV&V PM. The Program IV&V POC will be informed of all direct communications with other members of the Program and Partners.

The Program and Partners will provide the IV&V Team the necessary interfaces and access it needs in order to effectively and efficiently provide IV&V services. The Program and Partners will provide the IV&V Team with direct, electronic access, to the Program’s, the Partner’s and any subcontractor’s document repositories and data stores, change management systems, and defect tracking systems. The repositories and data stores include but are not limited to: source code, software models, software design, software requirements, software build data, software configuration data, and software test data that define vehicle software configuration, cybersecurity-relevant mission documentation, access to all software/hardware interfaces, and operational plans. Table 2-1 in this document provides an extensive list of example targeted artifacts.

The Program and/or Partners will provide such data/documentation to IV&V at the same time as the information is made available to the Program’s and Partners’ teams. The Program and/or Partners will provide draft and final versions of IV&V-requested development artifacts. It is expected that many of the development artifacts necessary to perform the IV&V analysis will be formal deliverables. However, in some cases non-deliverable or informal documentation (e.g., Software Development Folders, incremental pre-release builds, etc.) may be needed to support the IV&V analysis. In such cases, the Program IV&V POC and/or Partner POCs will make these items available on a case-by-case basis after taking into consideration various factors, including but not limited to, overall impact on the Program and the Partners. The incremental pre-release builds, in particular, are often necessary for the IV&V Team to achieve in-phase identification of issues.

mailto:William.M.Stanton@nasa.gov mailto:Keenan.L.Bowens@nasa.gov mailto:Stephen.Husty@nasa.gov

Release Date: May 05, 2021 Page: 15 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

The Program and Partners, through the IV&V POCs, are responsible for working with the IV&V Team to resolve issues/risks identified by the IV&V Team. Section 4.1 has more detail about the resolution of issues and risks identified by the IV&V Team.

Program and Partner personnel contact information is identified in Table 3-2 below.

TABLE 3-2: PROGRAM CONTACT INFORMATION

HLS Program

Position/Role Name Contact Information

HLS Program Manager Dr. Lisa Watson-

Morgan

256-544-3523

Lisa.A.Watson-Morgan@nasa.gov

HLS Chief Safety Officer John Crisler 256-544-3085

John.P.Crisler@nasa.gov

HLS Program IV&V POC Lien Moore 256-544-3468

Lien.N.Moore@nasa.gov

HLS Software Insight Team

Lead

Rob Morgenstern 301-286-7582

Robert.M.Morgenstern@nasa.gov

HLS Software Assurance Lead Nicole Woodson 256-961-2790

Nicole.L.Woodson@nasa.gov

HLS Cyber Security Embed Justin Jackson 256-544-8474

Justin.Jackson@nasa.gov

HLS Partner A IV&V POC TBD TBD

4.0 IV&V PRODUCTS AND COMMUNICATION AND REPORTING METHODS

4.1 IV&V PRODUCTS

The IV&V Team generates various products and utilizes various communication and reporting methods throughout the Program software lifecycle. The subsections below describe the IV&V products and associated communication and reporting methods further.

Assurance Conclusions

The IV&V Team will provide assurance conclusions upon the completion of analysis of specific capabilities. Analysis is considered to be complete once the analysis team determines there is no more analysis that can be performed with the available artifacts and resources (e.g. time, funding & personnel) or should be performed based on assessment of residual risk. In some cases, analysis and conclusions may be revisited when additional artifacts become available. Assurance conclusions document IV&V’s mailto:lisa.a.watson-morgan@nasa.gov mailto:john.p.crisler@nasa.gov mailto:lien.n.moore@nasa.gov mailto:robert.m.morgenstern@nasa.gov mailto:nicole.l.woodson@nasa.gov mailto:justin.jackson@nasa.gov

Release Date: May 05, 2021 Page: 16 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021 assessment of the capability, which includes any issues, risks, assumptions, forward work, and other caveats realized during the analysis.

Monthly Reports

Every month the IV&V Team will generate a monthly report for the Program that documents any significant findings from the analyses performed that month. The monthly report will be delivered at the beginning of each month and will typically describe what the IV&V Team completed the previous month (assurance conclusions and analysis activities) and any other notable events and activities for the Program’s awareness. Monthly report deliveries will commence the month following authority to proceed (ATP). Each month the report will provide a high level look ahead for the current month and any open assurance objectives being worked by the team. The monthly report content will be tailored to meet the needs of the Program and the Program IV&V POC.

The IV&V Team will generate a similar monthly report for each Partner with only Partner specific information in the report. These reports will describe what the IV&V Team completed that month (assurance conclusions and analysis activities) and any other notable events and activities related to that Partner. The report will provide a high level look ahead for the current month and any open assurance objectives being worked by the team for that Partner. The Partner monthly report content will be tailored to meet the needs of the Partner and the Partner IV&V POC.

Lifecycle Review Presentations

Throughout the lifecycle, the IV&V Team supports formal Program milestone reviews by providing status of IV&V activities accomplished to date and summary level assessments of reliability, safety, and security of HLS system software based on IV&V results at the time of the review. IV&V can also support any Partner led reviews or milestones at the Program and/or the Partner’s discretion. At a minimum, and as required by the NASA Agency’s Chief SMA Officer, the IV&V Team will present status of the IV&V efforts and associated recommendations at the Safety and Mission Success Review (SMSR).

Technical Issue Memorandums (TIMs)

A Technical Issue Memorandum (TIM) is the formal mechanism the IV&V Team uses to document one or more instances of a defect (i.e., issue) identified within a development artifact and formally communicate defect findings to the Program and Partners. Each TIM has a documented Impact and is assigned a Severity rating between 1 (highest severity) and 5 (lowest severity) as defined in Table 4-1. TIMs of Severity rating 1-3 require a formal disposition by the Program and/or Partners and must be verified to have been addressed prior to closure. TIMs of Severity rating 4 or 5 may be reviewed by the Program and/or Partners, but a formal response is not required (i.e., may transition directly to the “Not To Be Verified” state in IV&V Program issue tracking system).

Resolving Severity rating 4 and 5 TIMs, nonetheless, will certainly improve the quality of the Program’s software and reduce or eliminate risks associated with maintenance of the software product.

TIM Delivery: IV&V will deliver all TIMs directly to the Program IV&V POC in order for the Program to disposition all issues prior to them being delivered to the appropriate Partner.

Release Date: May 05, 2021 Page: 17 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TIM Resolution Path: Upon receiving the issues from the IV&V Team, the Program IV&V POC will take the issues to the Systems Engineering Review Forum/Technical Update Review Forum (SERF/TURF) to review each TIM provided by the IV&V Team and confirm the validity of the issue from the Program’s perspective in a timely manner. In general, it is best if the TIM can be reviewed and dispositioned within two weeks. Timely Program review and response is important to avoid propagation of defects into subsequent products, to prevent incorrect IV&V reporting (e.g., to Office of Safety and Mission Assurance (OSMA), HLS Program Office, HLS Safety Office, and other NASA IV&V Program stakeholders), and to minimize IV&V rework as well as Partner rework.

If the SERF/TURF concurs that a TIM with a severity of 1, 2, or 3 is legitimate through their review, the SERF/TURF will recommend that the TIM be presented to the HCB as an issue per the HLS Risk Management Plan (HLS-PLAN-006) in order for the Program to determine if the issue should be delivered to the Partner or formally accept the risk of not resolving the issue. If the SERF/TURF does not agree with the validity of the TIM, it will be presented to the HCB for disposition. The HCB will have final say if a TIM is to be delivered to the appropriate Partner or not. For TIMs with severity 1 or 2 the SERF/TURF will review and recommend a risk be established per the HLS process or the TIM will be presented to the HCB for disposition. If the Program ultimately decides to accept the risk associated with a Severity 1 or 2 TIM, IV&V will document a formal risk related to the TIM.

This approach to TIM resolution will allow the Program to determine what issues they would like the Partner to address in order to mitigate cost and schedule impacts to the Program.

TIM State Transitions: Once a Severity 1, 2, or 3 issue is provided to the Partner and the Partner identifies a plan to fix the defect that is acceptable to the IV&V Team and the HLS Program, the TIM will be put in the “To Be Verified (TBV)” state. After the defect is resolved, the Partner will notify the IV&V Team that the corrective action has been made and will provide the appropriate evidence (e.g., updated development artifacts, etc.) to the IV&V Team for verification and subsequent closure of the TIM. If verification of the corrective action cannot be completed, the IV&V Team will request additional information from the Program and/or Partner. If the Program accepts the risk of not resolving the TIM, information to support this decision will be provided to IV&V and a residual risk may be entered into the HLS risk database and the TIM will be put in the “Project (i.e., Program/Partner) Accepts Risk (PAR)” state. As documented above, if the TIM is a Severity 1 or 2 TIM then a formal risk will be opened for that TIM. Section 4.1.6 describes the escalation process.

If there is a dispute at any time in the issue resolution process, the TIM may be placed in an “In Dispute” state, at which time the Program and/or Partner and IV&V Team can continue dialog on the TIM. Subsequent to these discussions, the TIM may be withdrawn, placed in the “Project Accepts Risk” state, or reverted to the “To Be Verified” state.

If the Program and/or Partner does not concur a TIM is legitimate, the Program and/or Partner will provide appropriate data and/or explanation to support this conclusion. The IV&V Team will review and consider this data, and if the IV&V Team agrees, the TIM will be withdrawn. If the IV&V Team does not agree, additional dialog and discussion between the Program and/or Partner and IV&V Team may be required, and an appropriate course of action will be determined. Section

4.1.6 describes the escalation process.

https://moon2mars.ndc.nasa.gov/HLS/cmw/ReleasedProducts/NR/HLS-PLAN-006%20HLS%20Risk%20Management%20Plan_Baseline.pdf

Release Date: May 05, 2021 Page: 18 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

TABLE 4-1: TIM SEVERITY RATING AND DESCRIPTION1

Severity Capability

Affected

Success

Criteria Safety Test Cost & Schedule Other

Catas-trophic

Loss of an essential capability

OR

Complete loss of mission critical asset

Inability to achieve minimum mission success criteria

Causes loss of life or injury

N/A N/A N/A

Critical

Degradation of an essential capability

OR

Damage/destruction to mission asset which affects performance

Impact to the accomplish-ment of a mission objective

N/A

Essential capability not tested

Significant cost increases or schedule slip resulting from immediate or downstream development impacts

Significant reduction to requirements margins or design margins

Moderate

Degradation of system dependability

OR

Loss of a non-essential capability

Impact to the accomplish-ment of extended/ optional mission objectives

N/A

Essential capability inadequately tested

Cost or schedule impact resulting from redesign, reimplementation, and/or retest

Degradation of an essential capability or inability to accomplish mission objective, but with a known workaround

Minor

Degradation of a non-essential capability

N/A N/A

Non-essential capability inadequately tested

Defect impacting maintainability on current mission or reuse on future missions

Creates inconvenience for operators, crew or other projects' personnel

Communi-cations

Or

Editorial

Defect impacting documentation and communication clarity and poses no risk to the Mission

Project.

1 Source: S3105, Guidelines for Writing IV&V TIMs http://www.nasa.gov/centers/ivv/ims/supportdocs/index.html

Release Date: May 05, 2021 Page: 19 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

Risks

By conducting IV&V analysis, the IV&V Team may become aware of circumstances or information that indicates potential for an undesirable impact to a Partner or the Program. The IV&V Team will document such indications in risks and will formally communicate these risks to the Program.

The IV&V Team will assess all risks based on the likelihood and consequence of the undesired event using the Program’s likelihood and consequence ranking criteria (as defined in the HLS Program Risk Management Plan, HLS-PLAN-006). The IV&V Team may also provide recommendations to eliminate, reduce, or mitigate the risks. The IV&V Team will coordinate all risks with the Program prior to formal submission.

Typically, Programs maintain documentation of residual risks throughout the Program lifecycle.

The IV&V Team will continually assess the Program’s residual risks. At minimum, and as required by the Chief SMA Officer, the IV&V Team will evaluate residual risk data in preparation for the SMSR. The IV&V Team will communicate their stance with regards to such residual risk data to the Program prior to the SMSR.

Risk Delivery: IV&V will deliver all risks directly to the Program IV&V POC. The Program IV&V POC will then follow the HLS Risk Management Process in order for the Program to review all risks and determine if the risks need to be communicated to the appropriate Partner, or if the risk will be accepted by the Program.

Risk Resolution Path: Upon receiving the risks from the IV&V Team, the Program IV&V POC will follow the HLS Risk Management Process. If the Program agrees with the nature of a risk, the Program may choose to take ownership of the risk and mitigate the risk, communicate the risk to the appropriate Partner for potential mitigation, or choose to do nothing and accept the risk that IV&V has identified. Subsequently, the Program can document the risk and associated mitigation plan(s) in the Program’s risk management system, or in the Partner’s risk management system, however the HLS Program would like to track IV&V submitted risks. The IV&V Team will monitor the progress of any activities related to the risk until the risk is closed. This monitoring may be performed independently or via the Program and/or providing status data to the IV&V Team.

If the Program decides to accept the risk and not mitigate, or manage the risk, the Program will provide appropriate information to support this decision and IV&V will update the risk accordingly.

If the Program and/or Partner disagrees with IV&V regarding the risk, the Program and/or Partner will provide information pertaining to the disagreement. The IV&V Team will review this information and, if the IV&V Team is in agreement with the information, IV&V will withdraw the risk. If the IV&V Team is not in agreement, additional dialog between the Program and/or Partner and the IV&V Team may be required, and an appropriate course of action will be determined.

Section 4.1.6 describes the escalation process.

Item Tracking, Monitoring, And Escalation

All data such as issues and risks are recorded and provided to the Program and to the appropriate Partner after Program concurrence as they are identified and/or as per an agreed-to schedule.

The IV&V Team will evaluate Program and Partner responses to this data and update the status of this data in terms of tracking towards resolution in the appropriate IV&V Program repository. In addition, this data along with assurance conclusions will be documented in other IV&V products https://moon2mars.ndc.nasa.gov/HLS/cmw/ReleasedProducts/NR/HLS-PLAN-006%20HLS%20Risk%20Management%20Plan_Baseline.pdf

Release Date: May 05, 2021 Page: 20 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021 including but not limited to lifecycle review presentations, monthly reports, and recurring or ad hoc status reports as applicable.

The IV&V Team will work with independent Technical Authorities (TAs), e.g., the NASA Chief SMA Officer, and the NASA Chief Engineer (CE), to assure communication of risks or issues when the Program and the IV&V Team are unable to resolve a significant risk or issue, or when a significant risk or issue should be given extra visibility. The TAs will be provided full access to IV&V project records directly to facilitate their independent assessment of the records.

4.2 IV&V COMMUNICATION AND REPORTING METHODS

Communications and reporting methods between the IV&V Team, the Program, and the Partners occur via both formal and informal channels. Examples of communication and reporting methods include delivery of IV&V monthly reports and associated technical data, IV&V briefings at milestone reviews, and dialog between the IV&V Team, Program, and Partners regarding scope, priorities, access to resources, etc. consistent with this plan and the Option A Statement of Work.

Informal communications and reporting methods include recurring teleconferences and tag-ups between the IV&V Team and Program IVV POC, as well as the IV&V Team and the Partner IV&V

POC.

Routine Tag-ups

The IV&V Team will work with the Program IV&V POC to establish routine tag-ups to discuss overall IV&V status, development artifacts requests, results of IV&V analyses (delivery of assurance conclusions, issues, and risks), status of Program’s and Partners’ schedule and artifacts, resolution of IV&V issues and risks, and delivery of formal IV&V reports, etc. Such tag-ups may occur on a weekly, bi-weekly, or monthly basis as agreed to by the IV&V Team and the Program IV&V POC. These routine tag-ups represent the preferred method for communicating any issues and/or risks that the IV&V Team has identified.

The IV&V Team will work with each Partner IV&V POC to conduct monthly technical interchange meetings with the IV&V Team and the Program IV&V POC. The purpose of these meetings will be to discuss the resolution and/or implementation of issues or risks found by IV&V which have been provided to the Partner after review, and by the direction of the Program. These meetings are also an opportunity for IV&V to ask questions pertaining to the Partner, share status, and discuss availability of Partner information and data.

Lifecycle Review Presentations

The IV&V Team will provide IV&V status data and associated results of the IV&V efforts at various Program and Partner milestone reviews as defined in Table 4-2 below. If the IV&V Program will be presenting at the milestone review, the IV&V Team will communicate and coordinate the overall content of the presentation with the Program and Partner prior to the actual review.

TABLE 4-2: MILESTONE REVIEW IV&V PRESENTATIONS

Milestone Review Program Recipient Input Due

Applicable Milestone Reviews Program IV&V POC and appropriate Partner IV&V POC

5 working days prior to review

Release Date: May 05, 2021 Page: 21 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

Agency/Mission Directorate/Center Management Briefings

Throughout the course of the Program lifecycle, the IV&V Team is required and/or requested to present IV&V status to various stakeholders including but not limited to Center Management and the Mission Directorates, etc. Given that the IV&V Program is Goddard Space Flight Center (GSFC) Code 180.0, IV&V does provide input to the GSFC Monthly Status Reviews for all projects receiving services from the IV&V Program. All HLS sensitive and/or proprietary information that is provided to GSFC will be properly protected and only made available to individuals that are approved to view HLS data. The IV&V Team will communicate and coordinate the overall content of these presentations with the Program prior to the actual review as defined in Table 4-3 below.

TABLE 4-3: ADDITIONAL REPORTING EVENTS

Milestone Review Program Recipient Input Due

GSFC Monthly Status Review

(MSR)

Program IV&V POC 5 working days prior to review

OSMA Baseline Performance

Review (BPR)

Program IV&V POC 5 working days prior to review

IV&V Board of Advisor (IBA)

Semi-Annual Briefings

Program IV&V POC 5 working days prior to review

Development Reviews and Working Groups Support

IV&V will participate with software teams and in software and cybersecurity events conducted by the HLS Partners and their subcontractors including developer-level system, software, and cybersecurity reviews. While attending these mutually agreed upon Program and Partner working groups or reviews, IV&V can provide findings in these forums when appropriate. This will provide IV&V analysis results in-phase with the Program’s and Partners’ activities to the extent practical, with IV&V findings submitted in the Program’s and Partners’ own formats when possible.

Significant IV&V findings submitted through such forums will also be communicated via either IV&V TIMs (reference Section 4.1.4) or monthly reports (reference Section 4.1.2). These IV&V products will include a reference to the original submission.

Reviews and Working Group Support Resolution Path: the Program’s and Partners’ own review disposition processes will be applied to submitted materials. Applicable dispositions include: (a) the submission is rejected/withdrawn (tracked as a Withdrawn IV&V issue for IV&V metrics), (b) the submission is considered legitimate, accepted by the Program and/or Partner and turned into an IV&V issue or risk. Matters requiring elevation above the working group or review process will be coordinated with the Program IV&V POC and may apply the escalation options described in section 4.1.6, above.

Release Date: May 05, 2021 Page: 22 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

APPENDIX A: IV&V PBRA SCOPE AND FOCUS ASSESSMENT

The scope and focus for the IV&V work stems from the results of the PBRA results completed prior to Option A and updated as part of annual planning. These results will be shared with the HLS Program and added to Appendix A after Option A begins.

Release Date: May 05, 2021 Page: 23 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

APPENDIX B: IV&V RBA RESULT

The initial RBA results have been completed and are updated as part of annual planning. These results will be shared with the HLS Program and added to Appendix B after Option A begins.

Release Date: May 05, 2021 Page: 24 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

APPENDIX C: IV&V HERITAGE REVIEW

The HLS IV&V Heritage Report documents IV&V’s understanding of the heritage of the HLS software entities.

Please contact the HLS IV&V PM if you would like to review a copy of the IV&V Heritage Report.

Release Date: May 05, 2021 Page: 25 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

APPENDIX D: TECHNICAL SCOPE & RIGOR (TS&R)

The HLS IV&V Technical Scope & Rigor (TS&R) will be established for each Partner prior to Option A Award. After Option A Award, the selected Partner(s) TS&R will be added to this document.

Release Date: May 05, 2021 Page: 26 of 31

Human Landing System FY21 Software IV&V Project Execution Plan May 2021

APPENDIX E: REFERENCE DOCUMENTATION

TABLE E-1: RELEVANT DOCUMENTATION

Document Title Link or Date

IVV 09-1

Independent Verification and Validation

Technical Framework

IVV 09-1

S3105 Guidelines for Writing IV&V TIMs S3105

S3106 PBRA and RBA Process S3106

HLS PLAN 006 HLS Program Risk Management Plan HLS-PLAN-006

HLS PLAN 016 HLS-PLAN-016 Tech Management Plan HLS-PLAN-016

SERF/TURF Charter HLS SERF/TURF Charter HLS-CHTR-003 SERF/TURF Charter

HLS PLAN 004

HLS System Configuration and Data

Management Plan

HLS-PLAN-004

http:…