Attachment_0051_-_Joint_Software_Systems_Safety_Engineering_Handbook.pdf

DEPARTMENT OF DEFENSE

JOINT SOFTWARE SYSTEMS SAFETY ENGINEERING

HANDBOOK

DEVELOPED BY THE JOINT SOFTWARE SYSTEMS SAFETY

ENGINEERING WORKGROUP

Original published December 1999 Version 1.0 Published August 27, 2010

Naval Ordnance Safety and Security Activity

3817 Strauss Avenue, Building D-323 Indian Head, MD 20640-5555

Prepared for:

Director of Defense Research and Engineering

Distribution Statement A Approved for public release; distribution is unlimited.

Software Systems Safety Engineering Handbook Table of Contents i

Table of Contents

1 Overview 2 Introduction to the Handbook

2.1 Introduction

2.2 Purpose

2.3 Scope

2.4 Authority and Standards

2.5 Handbook Overview

2.5.1 Historical Background

2.5.2 Management Responsibilities

2.5.3 Introduction to the Systems Approach

2.5.3.1 The Hardware Development Lifecycle

2.5.3.2 The Software Development Lifecycle

2.5.3.2.1 Grand Design and Waterfall Lifecycle Model

2.5.3.2.2 Modified V Lifecycle Model

2.5.3.2.3 Spiral Lifecycle Model

2.5.3.2.4 Object-Oriented Analysis and Design

2.5.3.2.5 Component-Oriented and Package-Oriented Design

2.5.3.2.6 Extreme Programming

2.5.3.3 The Integration of Hardware and Software Lifecycles

2.5.4 A Team Solution

2.5.5 Systems of Systems Hazards and Causal Factors

2.5.5.1 Safety as a System Property

2.5.5.2 Functional Hazard Causal Factors

2.5.5.3 Interface-Related Hazard Causal Factors

2.5.5.4 Zonal Hazard Causes

2.5.5.5 Data Interfaces

2.5.5.6 COTS

2.5.5.7 Technology Issues

2.6 Handbook Organization

2.6.1 Planning and Management

2.6.2 Task Implementation

2.6.3 Residual Safety Risk Assessment and Acceptance

2.6.4 Supplementary Appendices

3 Introduction to Risk Management and System Safety

3.1 Introduction

3.2 A Discussion of Risk

3.2.1 Risk Perspectives

3.2.2 Safety Management Risk Review

3.3 Types of Risk

3.4 Areas of Program Risk

3.4.1 Schedule Risk

3.4.2 Budget Risk

3.4.3 Sociopolitical Risk

ii

3.4.4 Technical Risk

3.5 System Safety Engineering

3.6 Safety Risk Management

3.6.1 Initial Safety Risk Assessment

3.6.1.1 Mishap, Hazard, and Failure Mode Identification

3.6.1.2 Severity Categories

3.6.1.3 Probability Levels

3.6.1.4 Mishap Risk Index

3.6.2 Safety Order of Precedence

3.6.3 Elimination or Risk Reduction

3.6.4 Quantification of Residual Safety Risk

3.6.5 Managing and Assuming Residual Safety Risk

4 Software System Safety Engineering

4.1 Introduction

4.1.1 Section 4 Format

4.1.2 Process Charts

4.1.3 Software Safety Engineering Products

4.2 Software Safety Planning and Management

4.2.1 Planning

4.2.1.1 Establish the System Safety Program

4.2.1.2 Defining Acceptable Levels of Risk

4.2.1.3 Planning for Two Distinct Processes

4.2.1.3.1 Software Safety Assurance and Integrity Process

4.2.1.3.2 Software Safety Hazard Analysis Process

4.2.1.4 Defining and Using the Software Criticality Matrix

4.2.1.5 Defining the Requirements for Level of Rigor

4.2.1.6 Program Interfaces

4.2.1.6.1 Management Interfaces

4.2.1.6.2 Technical Interfaces

4.2.1.6.3 Contractual Interfaces

4.2.1.7 Contract Deliverables

4.2.1.8 Development of the Mishap Risk Index

4.2.1.8.1 Mishap Severity

4.2.1.8.2 Mishap Probability

4.2.2 Managing the Software Safety Program

4.3 Software Safety Task Implementation

4.3.1 Analyze and Comprehend the Conceptual Design Baseline of the System

4.3.2 Define Software Assurance Levels of Rigor

4.3.3 Functional Hazard Analysis

4.3.4 Preliminary Hazard Analysis

4.3.4.1 PHL Development

4.3.4.2 PHA Development

4.3.5 Safety Requirements Analysis

4.3.5.1 Categories of Software Safety Requirements

4.3.5.1.1 Contributing Software Safety Requirements

4.3.5.1.2 Generic Software Safety Requirements

iii

4.3.5.1.3 Mitigating Software Safety Requirements

4.3.5.2 Derive System Software Safety Requirements in the SRA

4.3.5.2.1 Preliminary SSRs

4.3.5.2.2 Matured SSRs

4.3.5.3 Documenting SSRs

4.3.5.4 Software Analysis Folders

4.3.6 Preliminary Software Design, SSHA

4.3.6.1 Attributes of Fault Management

4.3.6.2 Other Design Considerations for Safety

4.3.6.3 Example Techniques of Preliminary Software Design Analysis

4.3.6.3.1 Module Safety-Criticality Analysis

4.3.6.3.2 Program Structure Analysis

4.3.6.3.3 Traceability Analysis

4.3.7 Detailed Software Design, SSHA

4.3.7.1 Participate in Software Design Maturation

4.3.7.2 Detailed Design Software Safety Analysis

4.3.7.2.1 Safety Interlocks

4.3.7.2.1.1 Checks and Flags

4.3.7.2.1.2 Firewalls

4.3.7.2.1.3 Come-From Programming

4.3.7.2.1.4 Bit Combinations

4.3.7.2.2 “What If” Analysis

4.3.7.2.3 Safety-Critical Path Analysis, Thread Analysis, and UML Sequence Diagrams

4.3.7.2.4 Identifying Potential Hazard Causes Related to Interfacing Systems

4.3.7.3 Detailed Design Analysis Related Sub-Processes

4.3.7.3.1 Process Flow Diagram Development

4.3.7.3.2 Code-Level Analysis

4.3.7.3.2.1 Data Structure Analysis

4.3.7.3.2.2 Data Flow Analysis

4.3.7.3.2.3 Control Flow Analysis

4.3.7.3.2.4 Interface Analysis

4.3.7.3.2.5 Interrupt Analysis

4.3.7.3.2.6 Analysis by Inspection

4.3.7.3.2.7 Code Analysis Software Tools

4.3.7.3.2.8 Formal Proofs of Correctness

4.3.8 System Hazard Analysis

4.4 Software Safety Testing and Risk Assessment

4.4.1 Software Safety Test Planning

4.4.1.1 Guidance for Safety-Critical Software Testing

4.4.1.2 Nominal and Functional Requirements-Based Testing

4.4.1.3 Robustness Testing

4.4.1.4 Requirements Coverage Analysis

4.4.1.5 Structural Coverage Analysis

4.4.1.6 Formal Safety Qualification Testing

4.4.2 Software Systems Safety Tests

iv

4.4.2.1 Requirements-Based Tests

4.4.2.2 Functionality Tests

4.4.2.3 Path Coverage Testing

4.4.2.4 Statement Coverage Testing

4.4.2.5 Stress Testing

4.4.2.6 Endurance Testing

4.4.2.7 User Interface Tests

4.4.2.8 Fault Insertion and Failure Mode Tests

4.4.2.9 Boundary Condition Tests

4.4.2.10 GO/NO-GO Path Tests

4.4.2.11 Mutation Testing

4.4.2.12 Perturbation Testing

4.4.2.13 Test Phases

4.4.2.14 Regression Testing

4.4.3 Software Standards and Criteria Assessment

4.4.4 Safety Residual Risk Assessment

4.5 Safety Assessment Report

4.5.1 SAR Table of Contents

APPENDIX A DEFINITION OF TERMS ........................................................................... A-1 A.1 Acronyms ...................................................................................................... A-1 A.2 Definitions..................................................................................................... A-6 APPENDIX B REFERENCES ............................................................................................. B-1 B.1 Government References ................................................................................ B-1 B.2 Commercial References ................................................................................ B-2 B.3 Individual References ................................................................................... B-3 B.4 Other References ........................................................................................... B-5 APPENDIX C HANDBOOK SUPPLEMENTAL INFORMATION................................... C-1 C.1 DoD Authority and Standards ....................................................................... C-1 C.1.1 Department of Defense Directive 5000.01 ................................................... C-1 C.1.2 Department of Defense Instruction 5000.02 ................................................. C-1 C.1.3 Defense Acquisition Guidebook ................................................................... C-3 C.1.4 Military Standards ......................................................................................... C-4 C.1.4.1 MIL-STD-882B, Notice 1 ............................................................................. C-4 C.1.4.2 MIL-STD-882C ............................................................................................ C-5 C.1.4.3 MIL-STD-882D ............................................................................................ C-6 C.1.4.4 DOD-STD-2167A ......................................................................................... C-7 C.1.4.5 MIL-STD-498 ............................................................................................... C-7 C.1.5 Other Government Agencies ......................................................................... C-8 C.1.5.1 Department of Transportation ....................................................................... C-8 C.1.5.1.1 Federal Aviation Administration .................................................................. C-8 C.1.5.1.2 Aerospace Recommended Practice ............................................................... C-9 C.1.5.2 Department of Homeland Security ............................................................. C-11 C.1.5.2.1 Coast Guard ................................................................................................ C-11 C.1.5.3 National Aeronautics and Space Administration ........................................ C-11 C.1.5.4 Food and Drug Administration ................................................................... C-12 C.1.6 Commercial ................................................................................................. C-12 v

C.1.6.1 Institute of Electrical and Electronic Engineering ...................................... C-12 C.1.6.1.1 IEEE STD 1228-1994 ................................................................................. C-12 C.1.6.1.2 IEEE/EIA STD 12207................................................................................. C-13 C.1.6.2 TechAmerica G-48 System Safety Committee ........................................... C-13 C.1.6.3 International Electrotechnical Commission ................................................ C-14 C.2 International Standards ............................................................................... C-14 C.2.1 Australian Defense Standard 5679.............................................................. C-14 C.2.2 United Kingdom Defence Standard 00-56 .................................................. C-15 C.3 Proposed Contents of the System Safety Data Library ............................... C-16 C.3.1 System Safety Program Plan ....................................................................... C-16 C.3.2 Software Safety Program Plan .................................................................... C-17 C.3.3 Preliminary Hazard List .............................................................................. C-19 C.3.4 Safety-Critical Functions List ..................................................................... C-20 C.3.5 Preliminary Hazard Analysis ...................................................................... C-21 C.3.6 Subsystem Hazard Analysis ........................................................................ C-22 C.3.7 System Hazard Analysis ............................................................................. C-23 C.3.8 Safety Requirements Verification Report ................................................... C-24 C.4 Contractual Documentation ........................................................................ C-25 C.4.1 Statement of Operational Need ................................................................... C-25 C.4.2 Request for Proposal ................................................................................... C-26 C.4.3 Contract ....................................................................................................... C-26 C.4.4 Statement of Work ...................................................................................... C-27 C.4.5 System and Product Specification .............................................................. C-28 C.4.6 System and Subsystem Requirements ........................................................ C-29 C.5 Planning Interfaces...................................................................................... C-30 C.5.1 Engineering Management ........................................................................... C-30 C.5.2 Design Engineering ..................................................................................... C-30 C.5.3 Systems Engineering ................................................................................... C-31 C.5.4 Software Development................................................................................ C-32 C.5.5 Integrated Logistics Support ....................................................................... C-32 C.5.6 Other Engineering Support ......................................................................... C-33 C.6 Meetings and Reviews ................................................................................ C-33 C.6.1 Program Management Reviews .................................................................. C-33 C.6.2 Integrated Product Team Meetings ............................................................. C-34 C.6.3 System Requirements Reviews ................................................................... C-34 C.6.4 System and Subsystem Design Reviews .................................................... C-35 C.6.5 Preliminary Design Review ........................................................................ C-35 C.6.6 Critical Design Review ............................................................................... C-36 C.6.7 Test Readiness Review ............................................................................... C-37 C.6.8 Functional Configuration Audit .................................................................. C-38 C.6.9 Physical Configuration Audit ..................................................................... C-38 C.7 Working Groups.......................................................................................... C-39 C.7.1 System Safety Working Group ................................................................... C-39 C.7.2 Software System Safety Working Group .................................................... C-40 C.7.3 Test Integration Working Group/Test Planning Working Group ............... C-41 C.7.4 Computer Resources Working Group ......................................................... C-41 vi

C.7.5 Interface Control Working Group ............................................................... C-42 C.8 Resource Allocation .................................................................................... C-42 C.8.1 Safety Personnel.......................................................................................... C-42 C.8.2 Funding ....................................................................................................... C-43 C.8.3 Safety Schedules and Milestones ................................................................ C-43 C.8.4 Safety Tools and Training ........................................................................... C-44 C.8.5 Required Hardware and Software ............................................................... C-45 C.9 Program Plans ............................................................................................. C-45 C.9.1 Risk Management Plan ............................................................................... C-46 C.9.2 Quality Assurance Plan or Equivalent ........................................................ C-46 C.9.3 Reliability Engineering Plan ....................................................................... C-47 C.9.4 Software Development Plan ....................................................................... C-48 C.9.5 Systems Engineering Management Plan ..................................................... C-48 C.9.6 Test and Evaluation Master Plan ................................................................ C-50 C.9.7 Software Test Plan ...................................................................................... C-51 C.9.8 Software Installation Plan ........................................................................... C-51 C.9.9 Software Transition Plan............................................................................. C-52 C.10 Hardware and Human Interface Requirements ........................................... C-52 C.10.1 Interface Requirements ............................................................................... C-52 C.10.2 Operations and Support Requirements ....................................................... C-53 C.10.3 Safety and Warning Device Requirements ................................................. C-53 C.10.4 Protective Equipment Requirements ........................................................... C-54 C.10.5 Procedures and Training Requirements ...................................................... C-54 C.11 Managing Change ....................................................................................... C-54 C.11.1 Software Defect Resolution ........................................................................ C-55 C.11.2 Technology Insertion and Refresh .............................................................. C-56 C.11.3 Software Configuration Control Board ....................................................... C-58

APPENDIX D COTS AND NON-DEVELOPMENTAL ITEM SOFTWARE ................... D-1

D.1 Introduction ................................................................................................... D-1 D.2 Characteristics of COTS Software ................................................................ D-1 D.2.1 Advantages of COTS Software ..................................................................... D-2 D.2.2 Disadvantages of COTS Software ................................................................ D-2 D.3 Making a COTS Use Decision ...................................................................... D-3 D.3.1 Confidence .................................................................................................... D-4 D.3.2 Influence ....................................................................................................... D-5 D.3.3 Complexity .................................................................................................... D-7 D.4 COTS Software Safety Selection Process Example ..................................... D-8 D.4.1 Confidence Metric ........................................................................................ D-8 D.4.2 Influence Metric ............................................................................................ D-9 D.4.3 Complexity Metric ...................................................................................... D-10 D.4.4 Measure of Success ..................................................................................... D-13 D.5 COTS/NDI Software Safety Process Implementation ................................ D-13 D.6 Safety Risk Reduction Requirements ......................................................... D-13 D.6.1 Applications Software Design .................................................................... D-14 D.6.2 Middleware or Wrappers ............................................................................ D-15 D.6.3 Message Protocol ........................................................................................ D-16 vii

D.6.4 Designing Around COTS ............................................................................ D-16 D.6.5 Analysis and Test of COTS Software ......................................................... D-17 D.6.6 Eliminating Functionality ........................................................................... D-17 D.6.7 Run-Time Versions ..................................................................................... D-17 D.6.8 Watchdog Timers ........................................................................................ D-18 D.6.9 Configuration Management ........................................................................ D-18 D.6.10 Prototyping .................................................................................................. D-19 D.6.11 Testing......................................................................................................... D-19 D.7 Safety Case Development for COTS/NDI Software .................................. D-19 D.8 Summary ..................................................................................................... D-20

APPENDIX E GENERIC SOFTWARE SAFETY REQUIREMENTS AND GUIDELINES

........................................................................................................................ E-1 E.1 Introduction .................................................................................................... E-1 E.1.1 Determination of Safety-Critical Computing System Functions ................... E-1 E.1.1.1 Specifications ................................................................................................. E-1 E.1.2 Safety Criticality ............................................................................................ E-1 E.1.3 Identification of Safety-Critical Computing System Functions..................... E-2 E.1.3.1 Safety Application Functions ......................................................................... E-2 E.1.3.2 Safety Infrastructure Functions ...................................................................... E-3 E.1.3.3 Common SCCSFs .......................................................................................... E-3 E.1.4 Requirement Types ........................................................................................ E-4 E.1.4.1 Behavioral ...................................................................................................... E-4 E.1.4.2 Non-Behavioral .............................................................................................. E-5 E.1.4.3 Design Constraint........................................................................................... E-5 E.1.5 Implementation of Generic Requirements and Guidelines ............................ E-6 E.1.5.1 Integration with the Software Engineering Process ....................................... E-6 E.1.5.2 Audit Tool Post-Software Design or Development ....................................... E-6 E.1.5.3 Full Compliance vs. Partial or Non-Compliance ........................................... E-6 E.2 Design and Development Process Requirements and Guidelines ................. E-7 E.2.1 Configuration Control .................................................................................... E-7 E.2.2 Software Quality Assurance Program ............................................................ E-7 E.2.3 Two Person Rule ............................................................................................ E-7 E.2.4 Program Patch and Overlay Prohibition ........................................................ E-8 E.2.5 Software Design Verification and Validation ................................................ E-8 E.2.5.1 Correlation of Artifacts Analyzed to Artifacts Deployed .............................. E-8 E.2.5.2 Correlation of Process Reviewed to Process Employed ................................ E-9 E.2.5.3 Reviews and Audits ..................................................................................... E-10 E.3 System Design Requirements and Guidelines ............................................. E-10 E.3.1 Designed Safe States .................................................................................... E-10 E.3.2 Safe State Return.......................................................................................... E-10 E.3.3 Stand-Alone Computer ................................................................................ E-10 E.3.4 Ease of Maintenance .................................................................................... E-11 E.3.5 Restoration of Interlocks .............................................................................. E-11 E.3.6 Input and Output Registers .......................................................................... E-11 E.3.7 External Hardware Failures ......................................................................... E-11 E.3.8 Safety Kernel Failure ................................................................................... E-11 viii

E.3.9 Circumvent Unsafe Conditions .................................................................... E-11 E.3.10 Fallback and Recovery ................................................................................. E-12 E.3.11 Simulators .................................................................................................... E-12 E.3.12 System Errors Log ....................................................................................... E-12 E.3.13 Positive Feedback Mechanisms ................................................................... E-12 E.3.14 Peak Load Conditions .................................................................................. E-12 E.3.15 Endurance Issues .......................................................................................... E-12 E.3.16 Error Handling ............................................................................................. E-13 E.3.17 Redundancy Management ............................................................................ E-15 E.3.18 Safe Modes and Recovery ........................................................................... E-15 E.3.19 Isolation and Modularity .............................................................................. E-16 E.4 Power-Up System Initialization Requirements ............................................ E-16 E.4.1 Power-Up Initialization ................................................................................ E-17 E.4.2 Power Faults................................................................................................. E-17 E.4.3 Primary Computer Failure ........................................................................... E-17 E.4.4 Maintenance Interlocks ................................................................................ E-17 E.4.5 System-Level Check .................................................................................... E-17 E.4.6 Control Flow Defects ................................................................................... E-17 E.5 Computing System Environment Requirements and Guidelines................. E-20 E.5.1 Hardware and Hardware/Software Interface Requirements ........................ E-20 E.5.1.1 Failure in the Computing Environment ....................................................... E-20 E.5.2 CPU Selection .............................................................................................. E-21 E.5.3 Minimum Clock Cycles ............................................................................... E-21 E.5.4 Read Only Memory...................................................................................... E-22 E.6 Self-Check Design Requirements and Guidelines ....................................... E-22 E.6.1 Watchdog Timers ......................................................................................... E-22 E.6.2 Memory Checks ........................................................................................... E-22 E.6.3 Fault Detection ............................................................................................. E-22 E.6.4 Operational Checks ...................................................................................... E-22 E.7 Safety-Critical Computing System Functions Protection Requirements and

Guidelines .................................................................................................... E-23 E.7.1 Safety Degradation....................................................................................... E-23 E.7.2 Unauthorized Interaction ............................................................................. E-23 E.7.3 Unauthorized Access ................................................................................... E-23 E.7.4 Safety Kernel ROM ..................................................................................... E-23 E.7.5 Safety Kernel Independence ........................................................................ E-23 E.7.6 Inadvertent Jumps ........................................................................................ E-23 E.7.7 Load Data Integrity ...................................................................................... E-24 E.7.8 Operational Reconfiguration Integrity ......................................................... E-24 E.8 Interface Design Requirements .................................................................... E-24 E.8.1 Feedback Loops ........................................................................................... E-24 E.8.2 Interface Control .......................................................................................... E-24 E.8.3 Decision Statements ..................................................................................... E-24 E.8.4 Inter-CPU Communications......................................................................... E-24 E.8.5 Data Transfer Messages ............................................................................... E-25 E.8.6 External Functions ....................................................................................... E-25 ix

E.8.7 Input Reasonableness Checks ...................................................................... E-25 E.8.8 Full-Scale Representations........................................................................... E-25 E.9 Human Interface........................................................................................... E-25 E.9.1 Operator/Computing System Interface ........................................................ E-25 E.9.1.1 CHI Issues .................................................................................................... E-26 E.9.2 Processing Cancellation ............................................................................... E-27 E.9.3 Hazardous Function Initiation...................................................................... E-27 E.9.4 Safety-Critical Displays ............................................................................... E-27 E.9.5 Operator Entry Errors .................................................................................. E-27 E.9.6 Safety-Critical Alerts ................................................................................... E-27 E.9.7 Unsafe Situation Alerts ................................................................................ E-28 E.9.8 Unsafe State Alerts ...................................................................................... E-28 E.10 Critical Timing and Interrupt Functions ...................................................... E-28 E.10.1 Safety-Critical Timing ................................................................................. E-28 E.10.2 Valid Interrupts ............................................................................................ E-28 E.10.3 Recursive Loops........................................................................................... E-28 E.10.4 Time Dependency ........................................................................................ E-29 E.11 Software Design and Development Requirements and Guidelines ............. E-29 E.11.1 Coding Requirements and Issues ................................................................. E-29 E.11.1.1 Ada Language Issues ................................................................................... E-30 E.11.2 Modular Code .............................................................................................. E-31 E.11.3 Number of Modules ..................................................................................... E-31 E.11.4 Execution Path ............................................................................................. E-31 E.11.5 Halt Instructions ........................................................................................... E-32 E.11.6 Single Purpose Files ..................................................................................... E-32 E.11.7 Unnecessary Features................................................................................... E-32 E.11.8 Indirect Addressing Methods ....................................................................... E-32 E.11.9 Uninterruptible Code ................................................................................... E-32 E.11.10 Safety-Critical Files ..................................................................................... E-32 E.11.11 Unused Memory........................................................................................... E-32 E.11.12 Overlays of Safety-Critical Software Shall Occupy the Same Amount of

Memory ........................................................................................................ E-33 E.11.13 Operating System Functions ........................................................................ E-33 E.11.14 Compilers ..................................................................................................... E-33 E.11.15 Flags and Variables ...................................................................................... E-33 E.11.16 Loop Entry Point .......................................................................................... E-33 E.11.17 Software Maintenance Design ..................................................................... E-33 E.11.18 Variable Declaration .................................................................................... E-34 E.11.19 Unused Executable Code ............................................................................. E-34 E.11.20 Unreferenced Variables ............................................................................... E-34 E.11.21 Assignment Statements ................................................................................ E-34 E.11.22 Conditional Statements ................................................................................ E-34 E.11.23 Strong Data Typing ...................................................................................... E-34 E.11.24 Timer Values Annotated .............................................................................. E-35 E.11.25 Critical Variable Identification .................................................................... E-35 E.11.26 Global Variables .......................................................................................... E-35 x

E.12 Software Maintenance Requirements and Guidelines ................................. E-35 E.12.1 Critical Function Changes............................................................................ E-35 E.12.2 Critical Firmware Changes .......................................................................... E-35 E.12.3 Software Change Medium ........................................................................... E-36 E.12.4 Modification Configuration Control ............................................................ E-36 E.12.5 Version Identification .................................................................................. E-36 E.13 Software Analysis and Testing .................................................................... E-36 E.13.1 General Testing Guidelines .......................................................................... E-36 E.13.2 Trajectory Testing for Embedded Systems .................................................. E-37 E.13.3 Formal Test Coverage .................................................................................. E-38 E.13.4 Go/No-Go Path Testing ............................................................................... E-38 E.13.5 Input Failure Modes ..................................................................................... E-38 E.13.6 Boundary Test Conditions ........................................................................... E-38 E.13.7 Input Rata Rates ........................................................................................... E-38 E.13.8 Zero Value Testing ...................................................................................... E-39 E.13.9 Regression Testing ....................................................................................... E-39 E.13.10 Operator Interface Testing ........................................................................... E-39 E.13.11 Duration Stress Testing ................................................................................ E-39 APPENDIX F LESSONS LEARNED................................................................................... F-1 F.1 Therac Radiation Therapy Machine Fatalities ............................................... F-1 F.1.1 Summary ........................................................................................................ F-1 F.1.2 Key Facts ....................................................................................................... F-1 F.1.3 Lessons Learned............................................................................................. F-2 F.2 Missile Launch Timing Error Causes Hang-Fire ........................................... F-2 F.2.1 Summary ........................................................................................................ F-2 F.2.2 Key Facts ....................................................................................................... F-3 F.2.3 Lessons Learned............................................................................................. F-3 F.3 Reused Software Causes Flight Controls to Shut Down ............................... F-3 F.3.1 Summary ........................................................................................................ F-3 F.3.2 Key Facts ....................................................................................................... F-4 F.3.3 Lessons Learned............................................................................................. F-4 F.4 Flight Controls Fail at Supersonic Transition ................................................ F-4 F.4.1 Summary ........................................................................................................ F-4 F.4.2 Key Facts ....................................................................................................... F-5 F.4.3 Lessons Learned............................................................................................. F-5 F.5 Incorrect Missile Firing from Invalid Setup Sequence .................................. F-6 F.5.1 Summary ........................................................................................................ F-6 F.5.2 Key Facts ....................................................................................................... F-6 F.5.3 Lessons Learned............................................................................................. F-6 F.6 Operator’s Choice of Weapon Release Overridden by Software Control ..... F-7 F.6.1 Summary ........................................................................................................ F-7 F.6.2 Key Facts ....................................................................................................... F-7 F.6.3 Lessons Learned............................................................................................. F-8

APPENDIX G EXAMPLE REQUEST FOR PROPOSAL AND STATEMENT OF WORK

....................................................................................................................... G-1 G.1 Sample RFP .................................................................................................. G-1 xi

G.2 Sample Statement of Work ........................................................................... G-2 G.2.1 System Safety................................................................................................ G-2 G.2.2 Software Safety ............................................................................................. G-3 xii

Table of Figures

Figure 2-1: Management Commitment to the Integrated Safety Process Figure 2-2: Example of Internal System Interfaces Figure 2-3: Defense Acquisition Management Framework Figure 2-4: Relationship of Software to the Hardware Development Lifecycle Figure 2-5: Grand Design Waterfall Software Acquisition Lifecycle Model Figure 2-6: Modified V Software Acquisition Lifecycle Model Figure 2-7: Spiral Software Acquisition Lifecycle Model Figure 2-8: Integration of Engineering Personnel and Processes Figure 2-9: Handbook Layout Figure 2-10: Section 4 Format Figure 3-1: Types of Risk Figure 3-2: Systems Engineering and Risk Management Documentation Figure 3-3: Mishap Risk Example Figure 3-4: Hazard Reduction Order of Precedence Figure 4-1: Chapter 4 Contents Figure 4-2: Software System Safety Interfaces Figure 4-3: Process Chart Format Example Figure 4-4: Software Safety Planning Figure 4-5: Software Safety Planning by the PA Figure 4-6: Software Safety Planning by the DA Figure 4-7: Planning the Safety Criteria is Important Figure 4-8: Two Distinct Processes of Software Safety Engineering Figure 4-9: Graphical Depiction of Software Assurance and Integrity Figure 4-10: Graphical Depiction of Software Safety Hazard Analysis Figure 4-11: Legacy Software Control Category Definitions Figure 4-12: Recommended Terms and Definitions Figure 4-13: Example LOR Template Figure 4-14: Software Safety Program Interfaces Figure 4-15: Proposed SSS Team Membership Figure 4-16: Example of Risk Acceptance Matrix Figure 4-17: Likelihood of Occurrence Example Figure 4-18: Software Safety Program Management Figure 4-19: Software Safety Task Implementation Figure 4-20: Example POA&M Schedule Figure 4-21: Functional Hazard Analysis Figure 4-22: An Example of Safety-Critical Functions Figure 4-23: PHL Development Figure 4-24: PHA Figure 4-25: Hazard Analysis Segment Figure 4-26: Example of a PHA Format Figure 4-27: Safety Requirements Analysis Figure 4-28: SSR Derivation Figure 4-29: Tailoring the Generic Safety Requirements xiii

Figure 4-30: Example Software Safety Requirements Tracking Worksheet Figure 4-31: In-Depth Hazard Causal Analysis Figure 4-32: Preliminary Software Design Analysis Figure 4-33: SSR Verification Tree Figure 4-34: Hierarchy Tree Example Figure 4-35: Detailed Software Design Analysis Figure 4-36: Verification Methods Figure 4-37: Identification of Safety-Significant CSUs Figure 4-38: Example of a DFD Figure 4-39: Flow Chart Examples Figure 4-40: SHA Figure 4-41: Example of a SHA Interface Analysis Figure 4-42: Documentation of Interface Causes and Safety Requirements Figure 4-43: Software Safety Test Planning Figure 4-44: Software Safety Testing and Analysis Figure 4-45: Software Regression Testing Figure 4-46: Software Requirements Verification Figure 4-47: Software’s Contribution to Residual Safety Risk Assessment Figure C-1: Contents of an SwSPP - IEEE STD 1228-1994 .................................................... C-19 Figure C-2: SSHA and SHA Hazard Record Example ............................................................. C-23 Figure C-3: Hazard Requirements Verification Document Example ....................................... C-25 Figure C-4: Software Safety SOW Paragraphs ......................................................................... C-28 Figure C-5: Software Defect Resolution................................................................................... C-56 Figure C-6: Technology Insertion and Refresh ........................................................................ C-57 Figure C-7: Generic Software Configuration Change Process ................................................. C-59 Figure D-1: Initial Considerations for COTS Selection ............................................................. D-3 Figure D-2: COTS Evaluation Space .......................................................................................... D-4 Figure D-3: Example COTS Confidence Criteria Metric ........................................................... D-9 Figure D-4: Example Safety Influence Metric .......................................................................... D-10 Figure D-5: Example Safety Factors of Complexity ................................................................ D-11 Figure D-6: Example Testability Factors of Complexity ......................................................... D-12 Figure D-7: Example Integration Factors of Complexity ......................................................... D-12 xiv

Table of Tables

Table 3-1: Severity Categories Table 3-2: Probability Levels Table 3-3: Risk Assessment Matrix Table 4-1: Software Control Category Definitions Table 4-2: Software Criticality Matrix Table 4-3: Example LOR Tasks or Requirements Table 4-4: Example of Specific LOR Tasks Table 4-5: Example FHA Template Table 4-6: Acquisition Process Trade-Off Analyses Table 4-7: Example of an SSR Verification Matrix Table 4-8: Example of an RTM Table 4-9: Example Safety-Significant Function Matrix Table 4-10: Data Item Example

Software System Safety Engineering Handbook xv

The Office of Primary Responsibility for this guide is the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. Special thanks to Mr. Chris DiPetto, Dr. Liz Rodriguez-Johnson, and the Director of Defense Research and Engineering for their roles in the development of this Handbook.

DoD Office of Primary Responsibility for completion of Handbook Archibald McKinlay, VI

1999 Original Authors 2007 Returning Original Authors David Alberico Michael Brown John Bozarth Dr. Janet Gill Michael Brown Steven Mattern Janet Gill Archibald McKinlay, VI Steven Mattern

2007-2010 New Contributing Authors and Technical Editors Industry NASA Richard Church Steven D. Smith Murray Donaldson Daniel C. Victor Rene Fitzpatrick Martha Wetherholt Christine Hines John C. Wolf Samuel Redwine U.S. Navy Paige Ripani Mary Ellen Caro Frank Salvatore Karen Cooper Robert Schmedake Robert Heflin Sherry Smith Archibald McKinlay, VI Kristin Thompson David Shampine Kathryn G. Whorf U.S. Marine Corps U.S. Army Scott Rideout Jeffrey Marc Fornoff Missile Defense Agency Patty Lyon Grady Lee David Magidson Steven Pereira Frank Marotta U.S. Air Force Josh McNeil Robert Baker Gohan Oraby Todd Griffin Sandy Picinic Douglas Peterson Dr. George Vinansky Daniel Strub U.S. Coast Guard Jeffrey Wethern Chris Kijora

2010 Core Team Karen Cooper Michelle DePrenger Steven Mattern Archibald McKinlay, VI Azi Pajouhesh David Shampine

Software System Safety Engineering Handbook Section 1 Overview

1 Overview

Since the development of the digital computer and computing devices, software and firmware logic continues to play an important and evolutionary role in the operation and control of hazardous, safety-critical functions (SCFs). The reluctance of the engineering community to relinquish human control of hazardous operations has diminished over the last 25 years. Today, digital computer systems have autonomous control over safety-critical functions in nearly every major technology, both commercially and within Government systems. This revolution is due primarily to the ability of software to perform critical control tasks reliably at speeds unmatched by its human counterpart. Other factors influencing this transition are the ever-growing need for increased versatility, higher performance capability, greater efficiency, increased network interoperability, and decreased lifecycle cost. In most instances, properly designed software can meet all of these attributes for system performance. The logic of the software allows for decisions to be implemented with speed and accuracy without the human operator in the decision-making loop.

Within the domain of systems engineering, systems safety engineering identifies and analyzes behavioral and interface requirements, the design architecture, and the human interface within the context of both systems and systems of systems (SoS). In addition, system safety engineering defines requirements for design and systems engineering, taking into account the potential risks, verification and validation (V&V) of effective mitigation, and residual risk acceptance by certification or approval authorities.

It is essential to perform system safety engineering tasks on safety-critical systems to reduce safety risk in all aspects of a program. These tasks include software system safety activities involving the design, code, test, independent verification and validation (IV&V), operation and maintenance, and change control functions within the software engineering development and deployment processes.

The main objective of system safety engineering, which includes software system safety, is the application of engineering and management principles, criteria, and techniques to optimize all aspects of safety within the constraints of operational effectiveness, time, and cost throughout all phases of the system lifecycle.

Program management is ultimately responsible for the development of a safe system. The commitment to provide qualified personnel and an adequate budget and schedule for a software development program begins with the program director or Program Manager (PM). Senior management must be a strong voice of safety advocacy and must communicate this commitment to each level of program and technical management. The PM must provide the necessary resources to support the integrated safety process between systems engineering, software engineering, and safety engineering in the design, development, test, operation, and maintenance of the system software. The purpose of this Joint Software Systems Safety Engineering Handbook (JSSSEH), hereafter referred to as the Handbook, is to provide management and

Software System Safety Engineering Handbook Section 1 Overview engineering guidelines to achieve a reasonable…