SMC-Standard-012_16JAN2015.pdf

SMC Standard SMC-S-012 16 January 2015

Supersedes:

SMC-S-012 2008

Air Force Space Command

SPACE AND MISSILE SYSTEMS CENTER

STANDARD

SOFTWARE

DEVELOPMENT

APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED

Change Log

Issue Date Sections Changes Initial

TOR-2004

(3909)-3537

20 July 2004 All Based on MIL-STD-498 and tailoring for J-STD-016 in Recommended Software Standards for Space Systems TOR-2004(3909)-3406.

Rev A 11 March 2005 5.1.1 Reference to Appendix H instead of SDP DID.

6.2 Replaced SDP DID with “Use Appendix H.”

Appendix H Added Appendix H SDP Template.

Rev B 11 March 2005 (Editor provided November 2005)

None Distribution became public unlimited. No content change was made. Date remains the same.

TOR-

2013- 00083

March 28, 2013 Title Changed title from “Software Development Standard for Space Systems” to “Software Development Standard for Mission Critical Systems” to better reflect the domains covered. Changed report numbering to TOR-2013-00083.

Foreword and Introduction

Deleted.

Many Incorporated tailoring and lessons learned from recent acquisitions.

All All requirements were assigned unique identifiers within their subsections.

All All multipart requirements were split and assigned unique identifiers within their subsections.

Most Deleted all extra shall statements that merely required other shall statements to be met. (No double counting.)

1 Updates to acquisition- and contract-related terms.

2 Deletions, additions, and updates to referenced documents.

3.1 Additions and updates to software terms for

clarification.

4.2.3 Updates and additions to software bidirectional

traceability. All traceability requirements were gathered in this subsection.

4.2.4 Updates and additions to reusable software products.

4.2.5 Updates and additions to assurance of critical

requirements.

4.2.8 Updates and additions to access for acquirer review.

4.2.9 Added contractual requirements for software.

5.1 and 5.2 Updates to software test planning and integration and test environments.

5.3 Updated requirements for system requirements

analysis.

i

Published by SMC as SMC-S-012 (2008)

5.4 Updated requirements for system architectural

design.

5.5 Updated requirements for software requirements

analysis.

5.6 Major update to software design to add software

architecture and rewrite software design.

5.7 – 5.11 Major updates to improve the robustness of software testing at several levels from unit testing to system qualification testing.

5.12 Updated requirements for transition to operations.

5.13 Updated requirements for transition to maintenance.

5.14 Added requirements for configuration management

baselines and clarified requirements for other configuration management aspects.

5.15 Updated and added requirements for peer reviews.

5.16 Updated and added requirements for quality

assurance.

5.17 Updated requirements for corrective action.

5.18 Updated requirements for joint technical and

management reviews.

5.20 Replaced requirements for software measurement.

5.22 Replaced requirements for subcontractors with

requirements for software team members.

5.24 Updated requirements for coordination with associate

developers.

5.25 Added requirements for process improvement and

Process Improvement Plan.

6 Updated information for current acquisition practices.

Included and updated information from old Appendix G. Added references to new templates in Appendix H.

Appendix A Updated acronym list.

Appendix B Made mandatory and updated evaluation criteria for reusable software.

Appendix C Made mandatory and added discrepancy and change report requirements.

Appendix D Reformatted for easier understanding and added more product review criteria.

Appendix E Made mandatory and added build reviews and objectives.

Appendix F Replaced guidance for software indicators with mandatory test log requirements.

Appendix G Deleted all content.

Appendix H Made mandatory and updated Software Development

Plan template (H.1) to match updates to standard and provide project information. Added templates for Software Architecture Description, Software Master Build Plan, Software Measurement Plan, Software Measurement Report, and Process Improvement Plan.

ii

TR-RS-

2015- 00012

17 March 2014 Most Addressed comments from industry reviewers throughout the document. (Industry and stakeholder review is documented in TOR-2013-00797 Rev A.)

1.2.5.6 #6a Removed “validating” other software.

3.1 Changed definitions of discrepancy, software

development file, and added definition of demonstrate, stakeholder, and turnover. Clarified definitions of a few other terms, e.g., software development file, target computer system ,test procedures.

4.2.3 Clarified bidirectional traceability.

5.9.3 Updated and clarified advance notice for software

qualification testing and related events.

5.15.1 Simplified peer review data references.

Appendix E Added advance notification to acquirer for technical reviews.

iii

Contents

Change Log ........................................................................................................................................... i

1. Scope

1.1 Purpose

1.2 Application

1.2.1 Organizations and Agreements

1.2.2 Contract-Specific Application

1.2.3 Tailoring

1.2.4 Compliance

1.2.5 Interpretation of Selected Terms

1.3 Method and Tool Independence

2. Referenced Documents

3. Definitions

3.1 Terms

4. General Requirements

4.1 Software Development Process

4.2 General Requirements for Software Development

4.2.1 Software Development Methods

4.2.2 Standards for Software Products

4.2.3 Traceability

4.2.4 Reusable Software Products

4.2.5 Assurance of Critical Requirements

4.2.6 Computer Hardware Resource Utilization

4.2.7 Recording Rationale

4.2.8 Access for Acquirer Review

4.2.9 Contractual Requirements for Software

5. Detailed Requirements

5.1 Project Planning and Oversight

5.1.1 Software Development Planning

5.1.2 Software Integration and Qualification Test Planning

5.1.3 System Qualification Test Planning

5.1.4 Planning for Software Transition to Operations

5.1.5 Planning for Software Transition to Maintenance

5.1.6 Following and Updating Plans

5.2 Establishing a Software Development Environment

5.2.1 Software Engineering Environment

5.2.2 Software Integration and Qualification Test Environment

5.2.3 Software Development Library

5.2.4 Software Development Files

5.2.5 Nondeliverable Software

5.3 System Requirements Analysis

5.3.1 Analysis of User Input

5.3.2 Operational Concept

5.3.3 System Requirements Definition

5.4 System Architecture and Design

5.4.1 System-wide Architectural Design Decisions

5.4.2 System Architectural Design

5.5 Software Requirements Analysis

5.6 Software Architecture and Design

iv

5.6.1 Overall Software Architecture

5.6.2 Software Item Architecture

5.6.3 Software Item Detailed Design

5.7 Software Implementation and Unit Testing

5.7.1 Implementing Software

5.7.2 Preparing for Unit Testing

5.7.3 Performing Unit Testing

5.7.4 Analyzing and Recording Unit Testing Results

5.7.5 Unit Regression Testing

5.7.6 Revising and Retesting Units

5.8 Unit Integration and Testing

5.8.1 Testing on the Target Computer System

5.8.2 Preparing for Unit Integration and Testing

5.8.3 Performing Unit Integration and Testing

5.8.4 Analyzing and Recording Unit Integration and Test Results

5.8.5 Unit Integration Regression Testing

5.8.6 Revising and Retesting Unit Integration

5.9 Software Item Qualification Testing

5.9.1 Independence in Software Item Qualification Testing

5.9.2 Testing on the Target Computer System

5.9.3 Preparing for Software Item Qualification Testing

5.9.4 Dry Run of Software Item Qualification Testing

5.9.5 Performing Software Item Qualification Testing

5.9.6 Analyzing and Recording Software Item Qualification Test Results

5.9.7 Software Item Qualification Regression Testing

5.9.8 Revising and Retesting Software Items

5.10 Software-Hardware Item Integration and Testing

5.10.1 Testing on the Target Computer System

5.10.2 Preparing for Software-Hardware Item Integration and Testing

5.10.3 Performing Software-Hardware Item Integration and Testing

5.10.4 Analyzing and Recording Software-Hardware Item Integration and Test Results

5.10.5 Software-Hardware Item Integration Regression Testing

5.10.6 Revising and Retesting Software-Hardware Item Integration

5.11 System Qualification Testing

5.11.1 Independence in System Qualification Testing

5.11.2 Testing on the Target Computer System(s)

5.11.3 Preparing for System Qualification Testing

5.11.4 Dry Run of System Qualification Testing

5.11.5 Performing System Qualification Testing

5.11.6 Analyzing and Recording System Qualification Test Results

5.11.7 System Qualification Regression Testing

5.11.8 Revising and Retesting the System

5.12 Preparing for Software Transition to Operations

5.12.1 Preparing the Executable Software

5.12.2 Preparing Version Descriptions for User Sites

5.12.3 Preparing User Manuals

5.12.4 Installation at User Sites

5.13 Preparing for Software Transition to Maintenance

5.13.1 Preparing the Executable Software

5.13.2 Preparing Source Files

5.13.3 Preparing Version Descriptions for the Maintenance Site(s) v

5.13.4 Preparing the ‘As Built’ Software Architecture, Design, and Related Information

5.13.5 Updating the System/Subsystem Design Description

5.13.6 Updating the Software Requirements

5.13.7 Updating the System Requirements

5.13.8 Preparing Maintenance Manuals

5.13.9 Transition to the Designated Maintenance Site(s)

5.14 Software Configuration Management

5.14.1 Configuration Identification

5.14.2 Configuration Control

5.14.3 Configuration Status Accounting

5.14.4 Configuration Audits

5.14.5 Packaging, Storage, Handling, and Delivery

5.14.6 Baselines

5.15 Software Peer Reviews and Product Evaluations

5.15.1 Software Peer Reviews

5.15.2 Product Evaluations

5.16 Software Quality Assurance

5.16.1 Software Quality Assurance Evaluations

5.16.2 Software Quality Assurance Records

5.16.3 Independence in Software Quality Assurance

5.16.4 Software Quality Assurance Noncompliance Issues

5.17 Corrective Action

5.17.1 Discrepancy and Change Reports

5.17.2 Corrective Action System

5.18 Joint Technical and Management Reviews

5.18.1 Joint Technical Reviews

5.18.2 Joint Management Reviews

5.19 Software Risk Management

5.20 Software Measurement

5.20.1 Software Measurement Planning

5.20.2 Software Measurement Reporting

5.20.3 Software Measurement Working Group (SMWG)

5.21 Security and Privacy

5.22 Software Team Member Management

5.23 Interface with Software IV&V Agents

5.24 Coordination with Associate Developers

5.25 Improvement of Project Processes

6. Notes

6.1 Intended Use

6.2 Data Item Descriptions (DIDs)

6.3 Deliverable Versus Nondeliverable Software Products

6.3.1 Philosophy of the Standard

6.3.2 Contracting for Deliverables

6.3.3 Scheduling Deliverables

6.3.4 Format of Deliverables

6.5 Tailoring Guidance

6.6 Related Standardization Documents

Appendix A. List of Acronyms and Abbreviations .................................................................. A-1 A.1 Scope .................................................................................................................................. A-1

A.1.1 Acronyms and Abbreviations ..................................................................................... A-1 vi

Appendix B. Interpreting This Standard for Incorporation of COTS and Other Reusable Software Products ............................................................................................................................ B-1

B.1 Scope .................................................................................................................................. B-1 B.2 Evaluating Reusable Software Products ............................................................................. B-1

Appendix C. Discrepancy and Change Reporting ................................................................... C-1 C.1 Scope .................................................................................................................................. C-1 C.2 Discrepancy and Change Reports ....................................................................................... C-1

C.2.1 Discrepancy and Change Report Definitions ............................................................. C-1 C.2.2 Discrepancy and Change Report Requirements ......................................................... C-5

Appendix D. Product Evaluations ............................................................................................. D-1 D.1 Scope .................................................................................................................................. D-1 D.2 Criteria Definitions ............................................................................................................. D-1

D.2.1 Accurately Describes (an Item) .................................................................................. D-1 D.2.2 Adequate Tests, Test Cases, Procedures, Data, and Results ...................................... D-1 D.2.3 Component-Based ...................................................................................................... D-1 D.2.4 Consistent with Indicated Product(s) .......................................................................... D-1 D.2.5 Contains All Applicable Information ......................................................................... D-2 D.2.6 Covers (a Given Set of Items) .................................................................................... D-2 D.2.7 Feasible ....................................................................................................................... D-2 D.2.8 Follows SDP ............................................................................................................... D-2 D.2.9 Includes Multiple Perspectives ................................................................................... D-2 D.2.10 Internally Consistent ................................................................................................... D-2 D.2.11 Levels of Detail .......................................................................................................... D-2 D.2.12 Meets SOW and Contract, if Applicable .................................................................... D-2 D.2.13 Presents a Sound Approach ........................................................................................ D-3 D.2.14 Shows Evidence that an Item Under Test Meets its Requirements ............................ D-3 D.2.15 Testable ...................................................................................................................... D-3 D.2.16 Understandable ........................................................................................................... D-3 D.2.17 Uses Software Engineering Tools and Techniques .................................................... D-3

D.3 Required Evaluations .......................................................................................................... D-3 Appendix E. Joint Technical and Management Reviews ........................................................ E-1

E.1 Scope .................................................................................................................................. E-1 E.2 Joint Technical Reviews ..................................................................................................... E-1 E.3 Software-Specific Joint Technical Reviews ....................................................................... E-1

E.3.1 Software Build Planning Review (SBPR) .................................................................. E-3 E.3.2 Software Build Requirements and Architecture Review (SBRAR) ........................... E-4 E.3.3 Software Build Design Review (SBDR) .................................................................... E-6 E.3.4 Software Build Test Readiness Review (SBTRR) ..................................................... E-8 E.3.5 Software Build Exit Review (SBER) ....................................................................... E-10

E.4 Joint Technical Reviews Supporting Major Reviews ....................................................... E-12 E.5 Joint Management Reviews .............................................................................................. E-12

E.5.1 Joint Management Review Frequency ..................................................................... E-12 E.5.2 Joint Management Review Topics ........................................................................... E-13

Appendix F. Test Log Requirements ........................................................................................ F-1 F.1 Scope .................................................................................................................................. F-1 F.2 Test Logs ............................................................................................................................ F-1

F.2.1 Test Log Definitions ................................................................................................... F-1 F.2.2 Test Log Requirements ............................................................................................... F-2

Appendix G. Reserved ................................................................................................................ G-1 Appendix H. Product Templates ................................................................................................ H-1

Scope .............................................................................................................................................. H-1 vii

H.1 Software Development Plan (SDP) Template ................................................................. H.1-1 H.2 Software Architecture Description (SAD) Template ...................................................... H.2-1 H.3 Software Master Build Plan (SMBP) Template .............................................................. H.3-1 H.4 Software Measurement Plan (SMP) Template ................................................................ H.4-1 H.5 Software Measurement Report (SMR) Template ............................................................ H.5-1 H.6 Process Improvement Plan (PIP) Template ..................................................................... H.6-1

Tables

Table C.2-1 Categories of Software Products ................................................................................ C-4 Table C.2-2 Categories of Activities .............................................................................................. C-4 Table C.2-3 Categories of Severity ................................................................................................ C-5 Table C.2-4 Categories of Causes .................................................................................................. C-5 Table C.2-5 Discrepancy and Change Report Information ............................................................ C-6 Table D.3-1 Products and Associated Evaluation Criteria ............................................................. D-4 Table E.5-1 Software Management Products ............................................................................... E-13 viii

1. Scope

1.1 Purpose

The purpose of this standard is to establish uniform requirements for software development activities for mission critical systems.

Note 1: Terms that appear in italics are defined in Section 1.2 or Section 3.1 Terms.

Note 2: This standard is based on (DOD MIL-STD-498).

1.2 Application1

This standard applies to the development of mission critical systems that contain software (such as hardware-software systems), software-only systems, and stand-alone software products. The application of this standard is intended as in the following paragraphs.

1.2.1 Organizations and Agreements

This standard can be applied to contractors or acquirer in-house agencies performing software development. Within this standard, the term “acquirer” is used for the organization requiring the technical effort (see Section 3.1); the term “developer” is used for the organization(s) performing the technical effort (see Section 3.1); the term “contract” is used for the agreement between these parties (see Section 3.1); the term “Statement of Work” (SOW) is used for the list of tasks to be performed by the developer; and the term “Contract Data Requirements List” (CDRL) is used for the list of deliverable products.

1.2.2 Contract-Specific Application

This standard is invoked by citing it on a contract as a compliance document. It applies to each product and to each category of software covered by the contract, regardless of storage medium. The acquirer is expected to specify the categories of software to which the standard applies and to tailor the standard appropriately for each category of software. While this standard is written in terms of software items, it applies to software within the covered categories of software that is not designated as a software item, with the term “software item” interpreted appropriately. This standard applies to software installed in firmware devices.

This standard applies to the prime contractor and all software team members. This standard applies to the categories of software defined in Section 1.2.5.

1.2.3 Tailoring

This standard can be tailored for each category of software to which it is applied. General tailoring guidance can be found in Section 6.5.

1.2.4 Compliance1

1.2.4.1 Compliance Definition

Compliance with this standard as tailored for a project and recorded in the contract is defined as:

1. Performing the activities that resulted from tailoring this standard for a specific project, and

2. Recording applicable information resulting from the performance of the activities.

An activity is complete when all actions constituting the activity, as specified in the contract, have been accomplished and all applicable information has been recorded.

1.2.4.2 Compliance Terminology

For this standard, the words “shall,” “should,” and “may” are reserved words used to designate different levels of compliance. The term “shall” is used in statements of mandatory requirements. The term “should” is used to indicate a goal that the developer should attempt to adhere to, if at all

1This subsection is based on information from Section 1 of (EIA/IEEE J-016).

possible. The term “may” is used to indicate an optional method of satisfying a requirement. Use of the simple present and future tenses is limited to descriptive information only. In particular, the terms “will” and “must” are not used in this standard for any level of compliance.

1.2.4.3 Order of Precedence

In the event of conflict between the requirements of this standard and other mandatory standards, the acquirer is responsible for resolving the conflicts.

1.2.5 Interpretation of Selected Terms

Terms that appear in italics are defined in Section 1.2 Application or Section 3.1 Terms. The following terms have a special interpretation as used in this standard.

1.2.5.1 Interpretation of ‘System’ and ‘Subsystem’

The following interpretations of the words “system” and “subsystem” apply:

1. The term “system,” as used in this standard, means:

a. A software-hardware system (for example, a radar system) for which this standard covers only the software portion, or

b. A software system (for example, a mission planning system, or software running on the acquirer’s computers) for which this standard governs overall development.

2. The term “subsystem” is used to mean any system component between the entire system and the individual software items and hardware items. Depending on context, these components could be called subsystems, segments, elements, prime items, critical items, complex items, or other acquirer-specific terminology. For example, a system could consist of segments; a segment could consist of subsystems or elements; and a subsystem or element could consist of software items and hardware items.

3. If a system consists of subsystems, all requirements in this standard concerning systems apply to the subsystems as well. If a contract is based on alternatives to systems and subsystems, such as segments or complex items, the requirements in this standard concerning the system and its specification apply to these alternatives and their specifications.

1.2.5.2 Interpretation of ‘Participate’ in System Development1

The term “participate” in sections regarding system or subsystem activities is to be interpreted as follows:

1. If the software covered by this standard is part of a hardware-software system for which this standard covers only the software portion, the term “participate” depends upon what organization is in charge of the activity:

a. If the activity is performed by the software organization, then “participate” is to be interpreted as “be responsible for.”

b. If the activity is performed by another organization such as an integration and test organization, then “participate” is to be interpreted as “take part in.”

2. If the software (possibly with its computers) is considered to constitute a system, the term

“participate” is to be interpreted as “be responsible for.”

3. If the software in a hardware-software system is modified, but the hardware is not, the term

“participate” is to be interpreted as “be responsible for.”

1.2.5.3 Interpretation of ‘Develop,’ ‘Define,’ Etc.

Throughout this standard, requirements to “develop,” “define,” “establish,” or “identify” information are interpreted to include new development, modification, reuse, reengineering, maintenance, or any other activity or combination of activities resulting in products. Within this standard, requirements to “develop,” “define,” “establish,” or “identify” always include recording the information even if that is not explicitly stated. These terms also include maintaining the previous and current information, software products, facilities, infrastructure, software development environments, capability, etc.

throughout the system development lifecycle. Maintaining the development environments includes such activities as maintaining configuration management over all components of the development environment; keeping the configurations current; keeping the versions of reusable software, including commercial off-the-shelf (COTS) software, current so that supplier support is maintained; and keeping the plans and procedures current.

1.2.5.4 Interpretation of ‘Record’ or ‘Document’ (the Verbs)

Throughout this standard, requirements to “record” or “document” information are interpreted to mean “set down in a manner that can be retrieved and viewed.” The result can take many forms, including, but not limited to, hard copy or electronic documents and data recorded in computer-aided engineering (CAE) and project management tools. This interpretation also applies to implicit requirements to record or document as described in Section 1.2.5.3.

1.2.5.5 Interpretation of ‘Applicable’1

Within this standard, requirements to provide “applicable” information or to perform “applicable” activities are interpreted as meaning to “provide the information or perform the activities that are required for the project in accordance with the methods, tools, and procedures explained in the Software Development Plan.” The term “applicable” suggests that not all development projects will require the information or the activity. The acquirer and the developer jointly agree upon whether providing information or performing activities are applicable.

1.2.5.6 Interpretation of ‘Categories of Software’

The standard applies to the following categories of software including the software portion of firmware:

1. software onboard space vehicles (e.g., spacecraft bus, communications, payload);

2. software onboard launch and upper-stage vehicles;

3. ground operations software (e.g., mission planning; mission data processing; mission data distribution; mission data storage; mission support; telemetry, tracking, and commanding;

infrastructure and services);

4. user equipment software (e.g., software embedded in ground terminals, handheld receivers, receivers onboard wheeled vehicles, spacecraft, aircraft, ships, and weapons);

5. software that allows configuration management of initial and updated data, including constants, command sequences and procedures, and computer instructions, stored in onboard, ground, and user equipment databases or other types of data stores, and software that supports the verification and validation of this data; and

6. other software (e.g., applications, security, safety, training, modeling, simulation, analysis, database support, automatic test equipment, test facility and environment, and maintenance) used in any of the following:

a. satisfying or verifying requirements, or

b. performing or supporting operations.

1.2.5.7 Interpretation of ‘User’

Within this standard, the term “users” includes operators of the system in addition to the end users of the data produced by the system.

1.2.5.8 Interpretation of ‘User Site’

Within this standard, the term “user site” is interpreted to include individual ground operations sites, individual space vehicles, individual launch vehicles, individual mobile ground systems, and individual types of user equipment. This term is used primarily in Section 5.12.

1.2.5.9 Interpretation of ‘Prime Contractor’

Within this standard, the term “prime contractor” is interpreted to mean the organization with which the acquirer has the prime contract for the project. The prime contractor can also be the developer.

1.2.5.10 Interpretation of ‘Subcontractor’

Within this standard, the term “subcontractor” is interpreted to mean any software team member tasked by the prime contractor or another software team member to perform part of the required software-related effort. This definition is broader than the usual legal definition of subcontractor.

1.3 Method and Tool Independence

This standard is independent of any particular software development methods or tools. No requirement in this standard mandates that a particular method or tool be used.

2. Referenced Documents

Abelson 2011. Abelson, L. A., S. Eslinger, M. C. Gechman, C. H. Ledoux, M. V. Lieu, K. Korzac, Software Measurement Standard for Space Systems, Aerospace Report No. TOR- 2009(8506)-6, 5 May, 2011, The Aerospace Corporation.

Adams 2002. Adams, R. J., and S. Eslinger, A Framework for Software Products Within a System Context (2nd Edition), Aerospace Report No. TR-2002(8550)-3, 31 May 2002, The Aerospace Corporation.

Avizienis 2004. Avizienis A., J.-C. Laprie, B. Randell, and C. Landwehr, Basic Concepts and Taxonomy of Dependable and Secure Computing, IEEE Transactions on Dependable and Secure Computing, Vol. 1, No. 1, 2004.

CNSSI 4009 2010. Committee on National Security Systems, National Information Assurance (IA) Glossary, CNSS Instruction No. 4009, 26 April 2010.

Cohen 2005. Cohen, J., D. Plakosh, K. Keeler, Robustness Testing of Software-Intensive Systems:

Explanation and Guide, Report No. CMU/SEI-2005 -TN-015, April 2005, Carnegie Mellon University.

DAU Glossary. Defense Acquisition University, Glossary: Defense Acquisition Acronyms and Terms, Eleventh Edition, September 2003.

Dixon 2006. Dixon, J. M., C. M. Rink, and C. V. Sather, Digital ASIC/PLD Development Handbook for Space Systems, Aerospace Report No. TOR-2006(3904)-1, 19 May 2006, The Aerospace Corporation.

DOD DAU 2001. Systems Management College, Systems Engineering Fundamentals, January 2001, Defense Acquisition University.

DOD MIL-STD-498. Department of Defense, Software Development and Documentation, MIL-STD- 498, December 1994.

DOD-CIO 2009. Chief Information Officer, Clarifying Guidance Regarding Open Source Software (OSS), 16 October 2009, DOD CIO Memorandum.

EIA/IEEE J-016. Electronic Industries Association/Institute of Electrical and Electronics Engineers, Inc., Standard for Information Technology, Software Lifecycle Processes Software Development Acquirer-Supplier Agreement, EIA/IEEE Interim Standard J-STD-016-1995, September 1995.

FAR. Federal Acquisition Regulations (FAR), 2.101 (latest), https://www.acquisition.gov/far/current/html/Subpart%202_1.html (accessed 28 February 2013).

IEEE 1044. Institute of Electrical and Electronics Engineers, IEEE Standard Classification for Software Anomalies, IEEE Std 1044TM-1993, 2 December 1993.

IEEE 1074. Institute of Electrical and Electronics Engineers, IEEE Standard for Developing a Software Project Lifecycle Process, IEEE Std 1074TM-2006, 28 July 2006.

IEEE 1471. Institute of Electrical and Electronics Engineers, IEEE Recommended Practice for Architecture Description of Software-Intensive Systems, IEEE Std 1471-2000, September 2000.

IEEE 610.12. Institute of Electrical and Electronics Engineers, IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990, September 1990.

ISO/IEC 15939. International Organization for Standardization/International Electrotechnical Commission (ISO/IEC), Systems and Software Engineering – Measurement Process, ISO/IEC 15939:2007, 1 August 2007.

ISO 9241-11. International Organization for Standards/International Electrotechnical Commission (ISO/IEC), Ergonomic requirements for office work with visual display terminals (VDTs)

– Part 11, Guidance on usability, ISO 9241-11:1998, 15 March 1998.

McGarry 2001. McGarry, J., D. Card, C. Jones, B. Layman, E. Clark, J. Dean, and F. Hall, Practical Software Measurement: Objective Information for Decision Makers, October 2001, Addison Wesley.

Merriam-Webster. Merriam-Webster, Collegiate Dictionary, Tenth Edition, 1999.

Peresztegy 2009-1. Peresztegy, L. B., and C. E. O’Conner, Technical Reviews and Audits for Systems, Equipment, and Computer Software, Volume 1, Aerospace Report No. TOR-2007(8583)- 6414, 13 January 2009, The Aerospace Corporation.

Peresztegy 2009-2. Peresztegy, L. B., and C. E. O’Conner, Technical Reviews and Audits for Systems, Equipment, and Computer Software, Volume 2, Space Systems Supplement, Aerospace Report No. TOR-2007(8583)-6414, 30 September 2009, The Aerospace Corporation.

Russia-U.S. 2010. EastWest Institute, Russia-U.S. Bilateral on Cybersecurity – Critical Terminology Foundations, April 2011.

Sather 2010. Sather, C. V., C. M. Rink, and J. M. Dixon, Digital Application-Specific Integrated Circuit and Field-Programmable Gate Array Circuit Development Standard for Space Systems, Aerospace Report No. TOR-2010(8591)-10, Revison A, December 2010, The Aerospace Corporation.

SEI 2010. CMMI Product Team, CMMI for Development, Version 1.3, Report No. CMU/SEI-2010- TR-033, November 2010, Software Engineering Institute, Carnegie Mellon University.

Capability Maturity Model® and CMMI® are registered in the U. S. Patent and Trademark Office by Carnegie Mellon University.

Shaw 2013. Shaw, B. E., Systems Engineering Requirements and Products, Aerospace Report No. TR-2013-00001, 20 February 2013, The Aerospace Corporation. Also published as

SMC-S-001.

SMC-S-012. Space and Missile Systems Center Standard, SMC-S-012, Software Development Standard. This is the same as this document.

TAI SPD. The Aerospace Institute, Software Product Development Course, 2012, The Aerospace Corporation (Aerospace internal).

Wikipedia. Wikipedia, The Free Encyclopedia, http://www.wikipedia.org (Accessed 28 Feb 2013).

Definitions extracted in 2012 under the Creative Commons ShareAlike License found at http://creativecommons.org/licenses.

3. Definitions

3.1 Terms

The terms defined in this section appear in italics in the rest of the standard when they are not used as adjectives.

Acquirer. An organization that procures products for itself or another organization.

Agile development lifecycle model. A group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development and delivery, and a time-boxed iterative approach, and encourages rapid and flexible response to change.

It is a conceptual framework that promotes foreseen interactions throughout the development cycle.

Source: Adapted from (Wikipedia). See also evolutionary, incremental, iterative, spiral, and waterfall development lifecycle models.

Anomaly. See discrepancy.

Applicable. See Section 1.2.5.5.

Approval. Written notification by an authorized representative of the acquirer that a developer’s plans, requirements, designs, or other aspects of the project appear to be sound and can be used as the basis for further work. Such approval does not shift responsibility from the developer to meet contractual requirements.

Architecture. The fundamental organization of the system or software embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution. Source: Adapted from (IEEE 1471)

Associate developer. An organization that is neither prime contractor, software team member, nor subcontractor to the developer, but that has a development role on the same or related system or project.

Automatically generated code. Source code that is automatically generated by tools such as model-based software development tools, graphical user interface (GUI) builders, form builders, and computer-aided software engineering (CASE) tools.

Availability. Prior to the start of a mission, the probability that the system will be mission capable when a mission is needed. Availability accounts for system faults and the time it takes to restore the system to a mission capable state following failure. Source: (TAI SPD)

Bandwidth. A measure of available or consumed data communication resources. Examples of types of data communications resources include network, data bus, ground-ground link, ground-space link.

Source: (Wikipedia)

Baseline. A product or a set of work products that has been formally reviewed and agreed on at a particular point in the item’s lifecycle, which thereafter serves at the basis for further development, and which can be changed only through change control procedures. Source: Adapted from (SEI 2010) Note: A software baseline can be a single product or a set of consistent products for a particular build that can include, e.g., requirements, architecture, design, source code files and the associated executable code, build files, test plans, test cases, test procedures, test results, and user documentation. Source: Adapted from (SEI 2010)

Behavioral design. The design of how an overall system or a software item will behave, from a user’s point of view, in meeting its requirements, ignoring the internal implementation of the system or the software item. This design contrasts with architectural design, which identifies the internal components of the system or the software item, and with the detailed design of those components.

Bidirectional traceability. A two-way relationship between two products; for example, the relationship between a requirement and the design of a given software component, and the relationship between the design of the software component and the requirements it satisfies. Source:

Adapted from (IEEE 610.12)

Build. A version of software that meets a specified subset of the requirements that the completed software will meet.

Note: The relationship of the terms “build” and “version” is up to the developer; for example, it can take several versions to reach a build, a build might be released in several parallel versions (such as to different sites), or the terms can be used as synonyms. Other terms that developers often use as synonyms include: block, cycle, drop, increment, iteration, and spiral. This standard uses the term “build” for all of these.

Categories of software. See Section 1.2.5.6.

Change request. A request for a change to products, documents, or processes. Change requests might result from a discrepancy, a new requirement, a changed requirement, or an improvement suggestion.

Characterization testing. Testing software or a system to explore and determine its behavioral, performance, and other characteristics. For example, the capabilities and capacity of an unfamiliar legacy or COTS product need to be analyzed to determine its suitability for use or whether it still behaves the same way after changes.

Checkout. The process after software has been installed at a user site or maintenance site to exercise the software to demonstrate that the software behaves as expected or required. Checkout procedures might use subsets of the regression test suite.

Child requirement. A requirement in a child specification that can be traced upward to one or more requirements in a specification in the specification tree immediately above that child specification. A child requirement may also be a derived requirement. See parent requirement.

Child specification. A specification in the specification tree that is immediately below another specification in the specification tree. See parent specification. Child specifications contain child requirements.

Commercial off-the-shelf (COTS) software. See reusable software.

Note: COTS software is sometimes called “commercial item” software. “Commercial item” is defined in the FAR 2.101. Source: (FAR)

Component. A constituent part. Source: (Merriam-Webster)

Computer database. See database.

Computer hardware. Devices capable of accepting and storing computer and sensor data, executing a systematic sequence of operations on computer data, or producing control outputs. Such devices can perform substantial interpretation, computation, communication, control, or other logical functions.

Computer program. A combination of computer instructions and data definitions that enable computer hardware to perform computational or control functions. The computer instructions and data definitions in a computer program can exist as source code, object code, or executable code. Source:

(IEEE 610.12)

Computer software. See software.

Computer software configuration item (CSCI). See software item.

Computer software unit (CSU). See software unit.

Contract. The agreement between the acquirer and the developer. See Section 1.2.1.

Contract Data Requirements List (CDRL). See Section 1.2.1.

Contractual requirement. A mandatory statement in this standard or another portion of the contract.

Cyber-security. A property of cyberspace that is an ability to resist intentional and unintentional threats, respond, and recover. Source: (Russia-U.S.)

Cyberspace. An electronic medium through which information is created, transmitted, received, stored, processed, and deleted. Source: (Russia-U.S.)

Database. A collection of related data stored in one or more computerized files in a manner that can be accessed by users or computer programs via a database management system.

Database management system. An integrated set of computer programs that provides the capabilities needed to establish, modify, make available, and maintain the integrity of a database.

Data item description (DID). The format and content preparation instructions for a data product generated by the specific and discrete task requirements as delineated in the contract. Examples include DIDs for the Software Test Plan (STP), Software Test Report (STR), and Software User Manual (SUM) and templates for the Software Development Plan (SDP), Software Architecture Description (SAD), and Software Measurement Plan (SMP). See Section 6.2 for a list of DIDs and templates applicable to this standard.

Define. See Section 1.2.5.3.

Deliverable product. A product that is required by the contract to be delivered to the acquirer and other acquirer-designated recipients.

Demonstrate. To prove or make clear by evidence. Adapted from (Merriam Webster) Note: Demonstrating always includes recording the information even if that is not explicitly stated.

This definition of “demonstrate” is not to be confused with the verification method of “demonstration.”

Dependability. The ability: a) to deliver service that can justifiably be trusted, and b) to avoid service failures that are more frequent and more severe than is acceptable. Dependability is an integrating concept that encompasses the following attributes:

1. readiness for correct service;

2. continuity of correct service;

3. absence of catastrophic consequences on the user(s) and the environment;

4. absence of improper system alterations; and

5. ability to undergo modifications and repairs. Source: (Avizienis 2004)

Derived requirement. A system or software requirement that results from architecture or design decisions or from the user’s operational concepts and that is not directly traceable to one or more higher level requirements. A derived requirement may be either a functional or nonfunctional requirement.

Design. Those characteristics of a system or software item that are selected by the developer in response to the requirements. Some characteristics match the requirements; other characteristics are elaborations of requirements, such as definitions of all error messages in response to a requirement to display error messages; and others are implementation related, such as decisions about what software units and logic to use to satisfy the requirements.

Develop. See Section 1.2.5.3.

Developer. An organization that develops software products (“develops” includes new development, modification, integration, reuse, reengineering, maintenance, or any other activity that results in products). The term “developer” encompasses all software team members. See software team member.

Developer-internal software item integration testing. The last stage of software unit integration and testing where all of the components of the software item are integrated and tested. If the software item is developed in multiple builds, then the developer-internal software item integration testing occurs on a build-by-build basis.

Developer-internal system integration testing. The last stage of hardware-software integration and testing where all of the components of the system are integrated and tested. If the system is developed in multiple builds, then the developer-internal system integration testing occurs on a build-by-build basis.

Discrepancy. Any condition that deviates from expectations based on requirements specifications, architecture documents, design documents, user documents, plans, procedures, reports, standards, policy, etc., or from a user’s or other stakeholder’s sound engineering judgment or experiences.

Discrepancies can be found during, but not limited to, the review, test, analysis, compilation, or use of products or applicable documentation. The term discrepancy is used throughout this standard where others might use the terms: anomaly, defect, error, fault, failure, incident, flaw, problem, gripe, glitch, or bug. Source: Adapted from the term “anomaly” in (IEEE 1044).

Discrepancy and change report (DCR). Documentation of change requests, discrepancies, and test incidents. Discrepancy and change reports are written for any potential change, discrepancy, or test incident, even if the final resolution is that no discrepancy exists, or if the change request is rejected.

Discrepancy and change reports are sometimes called problem reports, change requests, trouble reports, test incident reports, issue reports, and other terms.

Document (as a verb). See Section 1.2.5.4.

Document (as a noun) or documentation. A collection of data, regardless of the medium on which it is recorded, that generally has permanence and can be read by humans or machines.

Duration testing. See endurance testing and stability testing. Contrast with stress testing.

End-to-end functional capability. A series of one or more software or system functions that:

1. satisfy a set of related requirements (or allocated portions of requirements);

2. exercise the software or system from inputs to the software or system through the software and hardware, interacting with databases, sensors, communications, and other software or system components, as applicable; and

3. result in outputs from the software to the user, another software or system component, or another system.

Note: The term “end-to-end functional capability,” as used in this standard, applies only to software unit integration and testing (see Section 5.8.2), software item qualification testing (see Section 5.9.3), and software-hardware item integration and testing (see Section 5.10.2), and it does not apply to system qualification testing.

The scope of an end-to-end functional capability differs depending upon the entity under test. For software unit integration and testing, an end-to-end functional capability is defined by inputs to one or more software units, processing and data flows that cross unit boundaries, and outputs from one or more software units, that together satisfy the portion of software or software interface requirements allocated to the units under test. For software item qualification testing, an end-to-end functional capability is defined by inputs to the software item, processing by the software item, data flows through the software item, and outputs from the software item that together satisfy one or more software item or software interface requirements. A similar interpretation applies to software-hardware integration testing. For software-hardware item integration and testing, an end-to-end functional capability is defined by inputs into a subset of the software or hardware items under test, processing and data flows through the software and hardware items under test, and outputs from a subset of the software or hardware items under test that together satisfy the portion of one or more software, software interface, hardware, or subsystem requirements allocated to the software and hardware items under test.

Endurance testing. Testing over prolonged execution time or elapsed time to determine whether longer-term execution shows functional, performance, or other problems.

Note: This testing often checks for clock overflows, buffer overflows, memory leaks, or other discrepancies that do not necessarily occur in shorter tests. Also known as duration testing or stability testing. Source: Adapted from (Cohen 2005). Contrast with stress testing.

Ensure. To make sure, certain, or secure by performing action(s). Source: Adapted from (Merriam- Webster).