Exhibit_8_The_SUPPLEMENT_TO_GUIDEBOOK_for_Acquisition_of_Naval_Intensive_Systems_Amend_0001.PDF

ASN(RD&A)

Software Criteria and Guidance for Systems Engineering Technical Reviews (SETR)

Supplement to Guidebook for Acquisition of Naval Software Intensive Systems

Office of the Assistant Secretary of the Navy (Research, Development and Acquisition)

Version 1.0 September 2010

Supplement to Guidebook for Acquisition of Naval Software Intensive Systems

Cover Reverse Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

This Page Intentionally Left Blank

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) TOC-1

Table of Contents Executive Summary

1. Background

2. Methodology

3. Document Structure

4. Software Artifacts

4.1 Software Artifact Identification and Governance

4.2 Software Artifact Maturity Schedule

4.3 Software Artifact Based Criteria

5. Core Software Metrics

5.1 Metrics Applicability

5.2 Metrics Based Criteria

6. General Software Health Indicators

6.1 Health Indicator Applicability

6.2 General Health Indicator Based Criteria

7. Tailoring Guidance

7.1 Tailoring of Artifacts and Elements

7.2 Merging or Separating SETRs

7.3 Moving SETRs

7.4 Tailoring an Artifact’s Schedule or Creator

7.5 Tailoring for Software Prototyping Efforts

References List of Acronyms

List of Enclosures

Enclosure (1) SETRs Applicable to Software Intensive Systems Enclosure (2) Software Centric Artifacts and Core Metrics, with their governing policy, guidance, and standards Enclosure (3) List of Elements Enclosure (4) SETR criteria for artifacts created by the Government Enclosure (5) SETR criteria for artifacts created by the Developer Enclosure (6) SETR criteria for artifacts created by the Government and/or Developer Enclosure (7) SETR criteria for Core Software Metrics Enclosure (8) SETR criteria for General Software Health Indicators

List of Tables Table 1. Description of scoring for SETR criteria statements Table 2. Pre-Milestone B Software-Centric Artifact Maturity Schedule Table 3. Post-Milestone B Software-Centric Artifact Maturity Schedule

List of Figures Figure 1 - Process used to develop criteria statements

TOC-2 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 1

Software Criteria and Guidance for Systems Engineering Technical Reviews

(SETR)

Executive Summary

This supplement to the Guidebook for Acquisition of Naval Software Intensive Systems provides guidance on software-centric criteria at use during SETR events. The criteria are for use by program management offices (PMO) and technical authorities, regardless of Acquisition Category (ACAT), at all SETR reviews chosen by the PMO as reflected in the approved System Engineering Plan (SEP) or equivalent.

This supplement adds accountability to the policy and practices conveyed in the Guidebook for Acquisition of Naval Software Intensive Systems; and was developed based on the SETR guidance in the Naval Systems Engineering Technical Review Handbook. The software criteria were developed by software acquisition subject matter experts from across the Naval Systems Commands (SYSCOMs) and Program Executive Offices (PEO), along with external subject matter experts. The information in this supplement represents SYSCOM best practices validated with the with Institute of Electrical & Electronics Engineers/Electronic Industries Alliance (IEEE/EIA) standards that the Navy has adopted for software development. The development team also worked with functional area leaders from interoperability, information assurance, and safety to provide criteria inclusive of their concerns and practices.

This supplement contains listings of the core software metrics and Navy enterprise-wide software artifacts with their maturity (see Table 2, Table 3, and Enclosure 2) with criteria statements for SETR events applicable to software intensive systems (see Enclosure 1 for a listing of the SETR events and Enclosures

(4) through (8) for the criteria statements). Section 7 of this supplement provides guidance on tailoring of the artifacts and criteria statements to match system complexity and risk level.

1. Background

The Assistant Secretary of the Navy for Research Development and Acquisition (ASN (RD&A)) commissioned a Software Process Improvement Initiative (SPII) in 2006 under the cognizance of the ASN (RD&A) Chief Systems Engineer (CHSENG). CHSENG established five SPII teams to focus on functional areas for potential process improvements. The resulting process improvements and associated ASN (RD&A) policies are captured in the Guidebook for Acquisition of Naval Software Intensive Systems issued in September 2008.

https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf

Page 2 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

In order to implement and more completely institutionalize software process improvement, a Cross- SYSCOM Software Working Group (SWWG) was established by CHSENG to address enterprise wide software acquisition issues. SWWG membership included software acquisition subject matter experts (SMEs) from the SYSCOMs and PEOs, augmented by SMEs appointed by the CHSENG. The SWWG provided SME reviews to support the development of software criteria for the Probability of Program Success (PoPS) methodology to assess software health in support of the SECNAVINST 5000.2D (2 Pass/6 Gate reviews). These criteria were associated with and based on the four core software metrics discussed in Chapter 2 of the Guidebook for Acquisition of Naval Software Intensive Systems and mandated by ASN (RD&A) policy (see Appendix D of the Guidebook for the policy memorandum dated July 22, 2008).

The SWWG was then tasked in 2009 to generate a software-focused supplement to the Guidebook for Acquisition of Naval Software Intensive Systems which would provide an enterprise approach for software evaluations within the Systems Engineering Technical Review (SETR) framework, maintain consistency within the overall framework of common SETR and technical authority policies and processes detailed in the Naval Systems Engineering Technical Review Handbook, and support linkage to the SPII focus team products, the SECNAVINST 5000.02D (2 Pass/6 Gate) review process, and the PoPS methodology.

A SETR assessment involves a number of Technical Interchange Meetings (TIMs) as part of the work-up to the SETR event (see section 7.3 of the Naval SETR Handbook). Technical Review Board (TRB) members have a number of responsibilities during the TIMs. Among them are to discuss technical issues relevant to the health of the program and its supporting documentation, identify potential Request for Action (RFA)/Request for Information (RFI) submissions, and identify/prioritize recommendations to program management. See section 8.0 of the Naval SETR Handbook for a full description of the RFA/RFI process.

The SETR software criteria presented in this supplement were prepared by the SWWG for program office staffs and technical authorities to enhance their use of enterprise-wide best practices in software system development. Enclosure (1) shows the list of SETRs from the Naval SETR Handbook that were deemed applicable enterprise-wide to software-intensive systems.

2. Methodology

The Navy software SETR criteria were developed by the SWWG by consolidating software-centric artifacts (i.e., software documents and other software related products) based on policies, standards, and best practices from MARCORSYSCOM, NAVAIR, NAVSEA, and SPAWAR. These 'artifacts' are the software work products developed across the acquisition lifecycle. The SWWG found while crafting the enterprise-wide list of artifacts that the artifacts varied by type and level of detail across the SYSCOMs. The SWWG developed a list of artifacts which were common across the SYSCOMs and reflected their best practices.

In some cases the artifacts are required by policy (such as the Software Development Plan (SDP)). In other cases they represent sound practices that existed at the SYSCOM level. The criteria development process also included a consideration of other (non-artifact) sources, including the four core software metrics. Figure 1 illustrates the SWWG process that was followed. All artifacts and core software metrics were validated and aligned with Institute of Electrical & Electronics Engineers/Electronic Industries Alliance (IEEE/EIA) Standard 12207.0, with IEEE/EIA Standard 1058, and as mentioned above, with current Navy standards for SECNAVINST 5000.2D (2 Pass / 6 Gate reviews) and the PoPS methodology. See section

6.3 of the Guidebook for Acquisition of Naval Software Intensive Systems for an overview of how IEEE/EIA Standard 12207.0 is applied to DoN acquisition of software intensive systems.

https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf https://acquisition.navy.mil/rda/content/download/5657/25845/version/1/file/Guidebook+for+Acquisition+of+Naval+Software+Intensive+SystemsSEP08.pdf

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 3

This supplement to the Guidebook for Acquisition of Naval Software Intensive Systems provides a structured set of software criteria statements to be used by software acquisition personnel and technical authorities. These criteria statements provide enterprise-wide guidance in verifying the health and maturity of software and associated software engineering tasks during SETR events, ensuring that they have been completed successfully. The criteria were designed to evaluate software development efforts leading to the software-intensive system end-product of the Engineering and Manufacturing Development (EMD) phase of the acquisition lifecycle, the integration of software with the overall system, and the readiness for transition from acquisition to life cycle support. These criteria do not, at this time, cover criteria for prototyping efforts for software Critical Technology Elements (CTEs) during the Technology Development phase of acquisition. This issue is under study by ASN(RD&A) and the SWWG with plans to develop and document appropriate criteria as an update to the criteria presented in this supplement. See section 7.5 of this supplement for information on tailoring current software SETR criteria for software prototyping efforts.

The criteria statements are organized according to their sources, which are:

Software Artifacts Core Software Metrics General (non-Artifact/non-Metric) Health Indicators

DA

HIEF

YSTEMS

NGINEER

List refinement

Criteria Statement Development Process Overview

Software artifacts

• Critical element(s) by review Mandated 4 core software metrics System artifacts

SYSCOM(s) Body of Work

DAU

PoPS Metrics

Don’t reinvent the wheelDon’t reinvent the wheel

What Needs To Be Evaluated Existing Software Criteria Statements

Draft set of criteria statements per SETR event

Enterprise SETR Software Criteria Statements

Final Criteria in Supplement to Guidebook for Acquisition of Naval Software Intensive

Systems

Figure 1 - Process used to develop criteria statements

The SWWG recognized that not all criteria were equally important to program health and risk, and that one responsibility of the TRB is to identify issues that could prevent closure of the SETR assessment or items

Page 4 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) that need program office visibility for follow-up. The SWWG scored all criteria to provide an enterprise view of which criteria, if not satisfied, should trigger an RFA, as shown in Table 1.

Table 1. Description of scoring for SETR criteria statements

Score Description 1 Crucial to software health and system risk. A criterion scored as ‘1,’ if not satisfied, should trigger a Critical RFA at a SETR review.

2 Important to software health and system risk. A criterion scored as ‘2,’ if not satisfied, should trigger a Non-Critical RFA at a SETR review.

3 Valuable to software health and system risk. A criterion that is useful at a system SETR, but would not trigger RFAs on its own if not satisfied.

4 Retain for future use. The criterion is a candidate for use at software item reviews, could be tailored to the needs of a particular system, or is not an enterprise-wide item.

3. Document Structure

Section 4 provides information on the software artifact-based criteria. It describes the software artifacts, their elements and maturity, and the criteria statements:

Artifacts are documents and other work products that are developed during the acquisition lifecycle.

Elements describe the content of each artifact and are essentially a ‘table of contents’ list for each artifact.

Maturity refers to the state of preparation and configuration management for each artifact through the development lifecycle. Artifacts can be in Draft, Final, or Updated status at a given SETR event.

Criteria statements provide the questions or statements by which the artifact is evaluated.

Section 5 identifies the four core software metrics and presents the criteria statements associated with assessing metrics-related software health at each SETR event. The core software metrics are mandated across the entire software development lifecycle by ASN (RD&A) policy, and link directly to Probability of Program Success (PoPS) evaluations at gate reviews. Section 6 provides the SETR criteria statements for the general software health indicators. Some information on tailoring software SETR criteria is found in Section 7 of this supplement.

4. Software Artifacts

4.1 Software Artifact Identification and Governance

Enclosure (2) provides a listing of the artifacts, along with their governing policy, guidance, standards, and information on artifacts that are required by policy. The process used to develop this artifact list is described in Section 2 to this supplement. Enclosure (3) provides a listing of the elements that describe the content of each artifact.

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 5

4.2 Software Artifact Maturity Schedule

Program artifacts are produced throughout the acquisition lifecycle and a schedule for their maturity is provided in Tables 2 and 3 of this supplement. In general, most of the artifacts produced before Milestone- B are initially generated by the Government (some with contractor support). The contractor/ developer becomes responsible for the creation of most of the artifacts in the Engineering and Manufacturing Development (EMD) phase of the acquisition (SETR event PDR2 (post-Milestone B) and events which follow).

Tables 2 and 3 document a “best practice” schedule for producing and reviewing the software centric artifacts identified in Enclosure (2), along with the entity (government or developer) typically responsible for creation and maintenance of the artifacts. Table 2 documents the schedule leading up to Milestone B and the release of the EMD Request for Proposal (RFP) to build the system. Table 3 documents the artifact maturity schedule after Milestone B. Not all programs will hold all the SETR events detailed in this supplement. The program System Engineering Plan (SEP) will typically detail how the SETR events will be tailored for the individual program’s needs, and some information on tailoring is found in Section 7 of this supplement.

Tables 2 and 3 also indicate the SETR events for which criteria were developed. These are the ‘essential’ SETR events given in Section 5.2 of the Naval SETR Handbook, with the addition of SSR (Software Specification Review) and IRR (Integration Readiness Review). Both SSR and IRR can provide critical, key event-driven occasions for software assessment in the development of complex systems. SSR is designed to review the completeness of the software specification, while IRR assesses the readiness of software for system-level testing.

The terms Draft, Final, and Update describe the stages in the development of an artifact. In this context, the meaning of the terms is as follows:

Draft: In most cases, Draft (D) denotes the first time an artifact appears at a SETR event. An artifact designated with a ‘D’ at a given SETR must also be available at all subsequent SETR events in the same or improved condition.

Update: Update (U) designation is optional, and may occur more than once. Each time it occurs, it signifies that some sort of improvement-related change to the Artifact is expected. The Update designation can follow either a ‘D’ or an ‘F’ designation.

Final: The Final (F) designation must occur once and only once, and denotes that the artifact is under configuration management control. The ‘F’ designation occurs at a SETR event subsequent to its ‘D’ designation, if the Artifact has a mandated Draft.

Page 6 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

Table 2. Pre-Milestone B Software-Centric Artifact Maturity Schedule

Enclosure/Page refers to the location within this supplement of the artifact description and SETR event criteria statements related to that artifact

Artifact Encl / Page

Created By ITR ASR SRR1 SRR2 SFR SSR

Pre-MS

B PDR

Measurement Plan 4 / 2 Gv D F U U U Software Risks/ Mitigation Plans

6 / 11 Gv then

Dev D F U U U

Software Acquisition Management Plan

(SAMP)

4 / 4 Gv D F U U

Computer Software Configuration Items (CSCIs)

6 / 2 Gv then

Dev

D (by Gv)

U U

Software Requirements Description (SRD)

6 / 4 Gv or

(if

Gv)

U (if Gv)

Software Requirements Traceability Matrix

(RTM)

6 / 5 Gv then

Dev D

(by Gv)

Software Life Cycle Sustainment Plan

(LCSP)

6 / 3 Gv then

Gv = Government Dev = Developer D = Draft F = Final U = Update

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 7

Table 3. Post-Milestone B Software-Centric Artifact Maturity Schedule

Enclosure/Page refers to the location within this supplement of the artifact description and SETR event criteria statements related to that artifact

Artifact Encl / Page

Created By

Post_MS

B PDR CDR IRR TRR SVR PRR

Measurement Plan 4 / 2 Gv U Software Risks / Mitigation Plans

6 / 11 Gv then

Dev U

(by Dev) U (by Dev)

U (by Dev)

U (by Dev)

U (by Dev)

Computer Software Configuration Items (CSCIs)

6 / 2 Gv then

Dev U (by Dev)

F (by

Software Requirements Description (SRD)

6 / 4 Gv or Dev

D/U (if Dev/Gv)

F (by Dev)

U (by Dev)

U (by Dev)

U (by

Traceability Matrix (RTM) 6 / 5

Gv then Dev

U (by Dev)

F (by Dev)

U (by Dev)

U (by Dev)

Software Architecture Description (SAD) 5 / 3 Dev D F U U U

Software Build Plan 5 / 5 Dev D F U U Software Design Description (SDD) 5 / 6 Dev D F U U

Prime/Subs Software Development Plan (SDP)

5 / 8 Dev F 1 U

Software Interface Design Description (SIDD) 5 / 15 Dev D F U U

Software Test Plan (STP) 5 / 18 Dev D F System/Software Integration Plan 5 / 20 Dev D F

Test Procedures/ Scripts/Cases

5 / 22 Dev F U

Operator Guide / Users Guide

5 / 2 Dev D U F

Software Product Baseline 5 / 16 Dev D U F Test Problem / Trouble Report 6 / 14

Gv and Dev D U F

Software Life Cycle Sustainment Plan (LCSP) 6 / 2

Gv then Dev F U

Gv = Government Dev = Developer D = Draft F = Final U = Update

1 Navy policy for acquisition of software intensive systems requires that a Software Development Plan (SDP) be submitted by the developer in draft form in response to the Request for Proposals. See sections 7.4.1, 8.1.1, and Appendix E of the Guidebook for Acquisition of Naval Software Intensive Systems for information on the requirements for an SDP, its role in the software acquisition effort, and the flexibilities available to acquisition officials when tailoring the SDP to the needs of the program.

https://acquisition.navy.mil/rda/home/organizations/dasns/rda_cheng

Page 8 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

4.3 Software Artifact Based Criteria

The SWWG developed a set of criteria based on the artifacts listed in Enclosure (2). The criteria are the enterprise-wide software criteria necessary to evaluate software maturity and contributions to program health and risk and verifying software engineering tasks have been successful.

The criteria are displayed by artifact in Enclosure (4) through (6). For each artifact, the enclosure provides the artifact description, creator, SETR events for which criteria apply, maturity schedule as discussed in Section 4.2, score as described in Section 2, and criteria statements. The table also includes a unique ID number for each criteria statement, created by the SWWG as an aid to tracking and discussing criteria.

The element name for which the criteria statement applies is useful to track how criteria statements mature through the acquisition lifecycle, and also trace how related criteria change in scoring between SETR events. Criteria statements are provided for only those elements necessary to evaluate software maturity and status of software engineering tasks. Enclosure (3) provides a listing of elements for each artifact.

5. Core Software Metrics

5.1 Metrics Applicability

As discussed in Chapter 2 of the Guidebook for Acquisition of Naval Software Intensive Systems, the core metrics are mandated across the entire software development lifecycle by ASN (RD&A) policy (see Appendix D of the Guidebook for the policy memorandum dated July 22, 2008). They are to be defined, gathered, analyzed, reported, and used to assess software health during all phases of the acquisition; and as such, they are “always applicable” and do not follow the same “maturity schedule” concept as the software artifacts described in Section 4.2.

The core metrics are not documents to be developed and controlled. These items will not go through Draft, Final, and possible Update phases. Rather, they are generated on numerous occasions for individual SETR events. The metrics based criteria are therefore tagged as Applicable (A) for any and all SETR events at which they are assessed, and do not have the D/F/U designations. The core metrics are assessed at all SETR events. While there are no artifact-based criteria at PCA, the core metrics evaluate the readiness for transition from acquisition to sustainment.

5.2 Metrics Based Criteria

Enclosure (7) provides the metrics based criteria statements. The core metrics are relational across the acquirer and developer organizations. They are updated at each SETR with criteria that reflect the changing nature of software acquisition across the lifecycle and they tie to PoPS software metrics criteria statements that are part of the SECNAVINST 5000.2D (2 Pass/6 Gate) process. See the ASN (RD&A) policy memorandum dated May 12, 2010 for PoPS materials and their role in assessments of program health.

https://acquisition.navy.mil/rda/content/download/6778/31305/file/Implementation%20of%20DON%20Acquisition%20Governance%20Process%20Revision.pdf

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 9

6. General Software Health Indicators

6.1 Health Indicator Applicability

The general health indicators are best practices and are not documents to be developed and controlled.

These items will not go through Draft, Final, and possible Update phases. Rather, they are indicated on numerous occasions for individual SETR events. The criteria associated with general health indicators are tagged as Applicable (A) for any and all SETR events at which they are assessed, and do not have the D/F/U designations. General software health indicators are listed at key SETR events.

6.2 General Health Indicator Based Criteria

Enclosure (8) provides general health indicator criteria. These criteria are indicators of general software development health and software development risk, and span across several artifacts and/or core software metrics. They will be most useful at the Technical Warrant Holder level.

7. Tailoring Guidance

The Naval SETR Handbook gives the Milestone Decision Authority (MDA) the authority to allow program managers to tailor SETRs to match system complexity or risk level. This may involve actions such as merging SETRs for non-complex systems, decomposing SETRs into incremental events for complex systems, or moving SETRs relative to each other or to the MDA milestones. In addition, program circumstances may dictate software artifacts different from the standard software artifacts of IEEE/IEA 12207 or Navy best practice assumed by this supplement. This section provides tailoring guidance for the SETR criteria identified in Enclosures (4) through (8), covering software artifacts, core metrics, and general health indicators.

7.1 Tailoring of Artifacts and Elements

While Navy acquisition policy for software intensive systems states that “there is no requirement that the specific IEEE/EIA Standard 12207.1 documents need to be created,” it does require that “their information content must be provided in some format, as appropriate, for the proposed work effort.” 2

The SETR criteria for software artifacts documented in Enclosures (4) through (6) are based on the artifacts and artifact elements discussed in IEEE 12207 or by Navy enterprise “best practice” (see Enclosures (2) and (3) for the artifact and element lists). If a system development effort calls for the creation and delivery of artifacts and/or elements different from those identified in the enclosures, the criteria should be tailored to apply to those artifacts to be created that will provide the equivalent information.

2 The Department of the Navy policy memorandum dated September 16, 2008 is found in Appendix E to the Guidebook for Acquisition of Naval Software Intensive Systems. The policy memorandum discusses adoption and mandated use of IEEE/EIA 12207 as the standard life cycle framework, the use of standard contract language, the requirement that offerors submit a Software Development Plan as a CDRL, measurement and metrics, and the functional disciplines required to ensure sound application of modern software technologies.

Page 10 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

7.2 Merging or Separating SETRs

The SETR criteria for software artifacts and core metrics are grouped by SETR within the artifact or metric in Enclosures 2 through 5. This should facilitate tailoring criteria, if SETRs have been merged (e.g., a single SRR is scheduled rather than the recommended two) or separated (e.g., multiple CDRs are scheduled to address subsystems of a complex system or system-of-systems). Merging or separating SETR events may be appropriate based on system complexity and/or incremental builds/system development, and will be documented in the tailored SETR schedule in the Systems Engineering Plan (SEP) as required by the Naval SETR Handbook.

When tailoring the occurrence of SETR events, the level of the reviews should be addressed and characterized. For example, there may be multiple incremental reviews for multiple builds, but these reviews may be at a lower level of detail, not requiring top-level attention. The program office should carefully describe in their SEP the relationship of top-level reviews and lower-level reviews to facilitate an effective strategy. When SETR events are tailored, engineering judgment should be used to eliminate criteria reflecting lesser artifact maturity or to reword criteria to reflect correct artifact maturity for the given

SETR.

7.3 Moving SETRs

If SETRs are reordered, relative to each other, or are moved to the other side of an assumed MDA milestone (such as from pre-MS B to post-MS B), all criteria associated with artifacts, metrics, or general software health associated with the SETR, or SETRs, impacted should be reviewed and tailored, as appropriate, to accommodate the actual system development schedule. Documentation of the tailored SETR schedule in the SEP is required by the Naval SETR Handbook.

7.4 Tailoring an Artifact’s Schedule or Creator

If the schedule of an artifact (Draft, Final, or Update) differs from that shown in Tables 2 or 3 (the draft SRD is scheduled for the pre-MS B PDR rather than SFR, for example), the SETR criteria for the artifact should be tailored, as appropriate, with respect to the actual artifact schedule. If the creator of an artifact will be different than the one indicated in Tables 2 or 3 (for example, the Government instead of the Developer) all criteria associated with the artifact should be tailored appropriately to reflect the new creator.

7.5 Tailoring for Software Prototyping Efforts

DoD and Naval policies (DoDD 5000.01, DoDI 5000.02, and SECNAVINST 5000.2D) require that Program Managers reduce technology risk by demonstrating critical technology prior to program initiation. DoDI

5000.02 requires that Critical Technology Elements (CTEs) be identified and assessed during the Material Solution Analysis phase of Acquisition, then prototyped and demonstrated in a "relevant environment" during the Technology Development phase.

The software SETR criteria contained the Enclosures (4) to (8) of this supplement do not, at this time, cover criteria for the identification, development, demonstration, and assessment of software CTE-related

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 11 prototypes during these early phases of Acquisition. This issue is under study by ASN(RD&A) and the SWWG with plans to develop and document appropriate criteria. In the interim, for those programs facing the issue of reviewing technical maturity of software CTE prototyping efforts at SETRs, the following tailoring guidance is offered:

• Criteria from the late Technology Development SETRs (SSR, SFR, and Pre-MS B PDR) and early Engineering & Manufacturing Development SETRs (Post-MS B PDR, CDR, IRR) may be appropriate, with revision, for assessing the health and maturity of software CTEs/prototypes and prototyping tasks.

• As CTEs are identified during Material Solution Analysis and prototyped during Technology

Development, the revised criteria should be added to existing criteria for the Material Solution Analysis and early Technology Development SETRs (ITR, ASR, SRR1, SRR2, and SFR).

Page 12 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

References

Guidebook for Acquisition of Naval Software Intensive Systems, Version 1.0 (September 2008) Office of the Assistant Secretary of the Navy (Research, Development, and Acquisition). Available at https://acquisition.navy.mil/rda/home/organizations/dasns/rda_cheng

IEEE/EIA 12207.0-1996; Industry Implementation of International Standard ISO/IEC 12207:1995 (ISO/IEC 12207) Standard for Information Technology-Software life cycle processes – Software Life Cycle Processes, IEEE/EIA 12207.0 (Mar 1998), IEEE Press, New York, NY, USA.

IEEE/EIA 12207.1-1997; Industry Implementation of International Standard ISO/IEC 12207:1995 (ISO/IEC 12207) Standard for Information Technology-Software life cycle processes – Software Life Cycle Data, IEEE/EIA 12207.0 (April 1998), IEEE Press, New York, NY, USA.

IEEE/EIA 12207.2-1998; Industry Implementation of International Standard ISO/IEC 12207:1995 (ISO/IEC 12207) Standard for Information Technology-Software life cycle processes – Implementation Considerations, IEEE/EIA 12207.0 (April 1998), IEEE Press, New York, NY, USA.

Implementation and Operation of the Defense Acquisition System and the Joint Capabilities Integration and Development System. SECNAVINST 5000.2D (October 16, 2008)

Implementation of DON Acquisition Governance Process Revision Interim Briefing Templates and Probability of Program Success. ASN(RD&A) policy memorandum dated May 12, 2010.

Available at https://acquisition.navy.mil/rda/content/download/6778/31305/file/Implementation%20of%20DON% 20Acquisition%20Governance%20Process%20Revision.pdf

Naval Systems Engineering Technical Review Handbook, version 1.0 (April 2009). Jointly promulgated as MARCORSYSCOM Order 5400.5, SPAWARINST 5000.1, NAVFACINST 5000.15, NAVSUPINST 5000.21, NAVSEAINST 5000.9, and NAVAIRINST 5000.24.

Operation of the Defense Acquisition System, DoD Instruction 5000.02 (December 8, 2008)

Operation of the Defense Acquisition System, DoD Directive 5000.01 (May 12, 2003, Current as of November 20, 2007)

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 13

List of Acronyms

A ....................................Applicable (in reference to maturity of an artifact) ACAT.............................Acquisition Category AoA................................Analysis of Alternatives ASN (RD&A)..................Assistant Secretary of the Navy for Research, Development, and Acquisition ASR ...............................Alternative System Review CDD...............................Capability Development Document CDR...............................Critical Design Review CHSENG .......................Chief Systems Engineer CI...................................Configuration Item CLS ...............................Contractor Logistics Support CM.................................Configuration Management CMMI® ...........................Capability Maturity Model Integration CONOPS.......................Concept of Operations COTS.............................Commercial Off-The-Shelf CPI ................................Continuous Process Improvement CPU...............................Central Processing Unit CSCI ..............................Computer Software Configuration Item CTE ...............................Critical Technology Element CWBS............................Consolidated Work Breakdown Structure D....................................Draft (in reference to maturity of an artifact) DoD ...............................Department of Defense DoDD.............................Department of Defense Directive DoDI ..............................Department of Defense Instruction EMD...............................Engineering and Manufacturing Development ESOH ............................Environmental Safety and Occupational Health EVMS ............................Earned Value Management System F ....................................Final (in reference to maturity of an artifact) FMEA/FMECA ...............Failure Mode and Effects Analysis / Failure Modes Effects and Criticality Analysis FQT ...............................Formal Qualification Testing GOTS ............................Government Off-The-Shelf HWCI .............................Hardware Configuration Item ICD ................................Initial Capabilities Document ICE ................................Independent Cost Estimate IDE ................................Integrated Development Environment IEEE/EIA........................Institute of Electrical & Electronics Engineers/Electronic Industries Alliance IIV&V .............................Internal Independent Verification and Validation

Page 14 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

IRR ................................Integration Readiness Review IRS ................................Interface Requirements Specification ITR.................................Initial Technical Review KPP ...............................Key Performance Parameter KSA ...............................Knowledge, Skill, and Abilities LCSP .............................Life Cycle Sustainment Plan MARCORSYSCOM .......Marine Corps Systems Command MDA...............................Milestone Decision Authority MS .................................Milestone NAVAIR .........................Naval Air Systems Command NAVSEA........................Naval Sea Systems Command NDI ................................Non Developmental Items PCA ...............................Physical Configuration Audit PEO...............................Program Executive Office PM .................................Program Manager PMO ..............................Program Management Office PoPS .............................Probability of Program Success QA .................................Quality Assurance R&M...............................Reliability and Maintainability RFA ...............................Request for Action RFI.................................Request for Information RFP ...............................Request for Proposal RMP...............................Requirements Management Plan RTM...............................Requirements Traceability Matrix SAD ...............................Software Architecture Description SAMP ............................Software Acquisition Management Plan SCDS.............................Safety Critical Digital Systems SDD...............................Software Design Description SDP ...............................Software Development Plan SDS ...............................Systems Design Specification SECNAVINST................Secretary of the Navy Instruction SEP ...............................System Engineering Plan SETR.............................Systems Engineering Technical Review SFR ...............................System Functional Review SIDD..............................Software Interface Design Description SME...............................Subject Matter Expert SPAWAR.......................Space and Naval Warfare Systems Command SPII................................Software Process Improvement Initiative

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Page 15

SQA...............................Software Quality Assurance SRD...............................Software Requirements Description SRR...............................System Requirements Review SRS ...............................Software Requirements Specification SSA ...............................Source Selection Authority or Software Sustainment Activity/Organization SSR ...............................Software Specification Review STP ...............................Software Test Plan SUM...............................Software Users Manual SVR ...............................System Verification Review SW.................................Software SWWG...........................Software Working Group SYSCOM.......................Systems Command TDS ...............................Technology Development Strategy TEMP.............................Test and Evaluation Master Plan TES ...............................Test and Evaluation Strategy TIM ................................Technical Interchange Meeting TOC...............................Total Ownership Cost TRB ...............................Technical Review Board TRR ...............................Test Readiness Review U....................................Update (in reference to maturity of an artifact) VDD...............................Version Description Document

Page 16 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Enclosure (1) Page 1

Enclosure (1). SETRs Applicable to Software Intensive Systems (information from Table 5-1 of Naval SETR Handbook)

SETR Purpose Timing Initial

Technical Review

Supports technical basis for initial cost estimates and POM budget submissions.

Materiel Solution Analysis (pre-MDA MS A)

Alternative System Review

Reviews results of Materiel Solution Analysis phase and assesses technology development plan and preferred system concept.

Materiel Solution Analysis (pre-MDA MS A)

System Requirements

Review (see Note 1)

Assesses technical readiness to enter Engineering & Manufacturing Development phase.

Technology Development (pre-MDA MS B)

System Functional

Review

Assesses System Functional Baseline and readiness to begin functional allocation.

Technology Development (pre-MDA MS B)

Software Specification

Review Assesses completeness of software specification.

Technology Development (pre-MDA MS B)

Preliminary Design Review

(see Note 2)

Assesses System Allocated Baseline and readiness to begin detailed design.

Pre/post-MDA

MS B

Critical Design Review

Assesses System Product Baseline and supports Design Readiness Review.

Engineering & Manufacturing Development

(pre-MDA MS C) Integration Readiness

Review Assesses readiness of software systems.

Engineering & Manufacturing Development

(pre-MDA MS C)

Test Readiness Review

Assesses system readiness to begin Developmental Test and Evaluation (DT&E).

Engineering & Manufacturing Development

(pre-MDA MS C) System

Verification Review

Assesses system compliance with functional baseline.

Production and Deployment

(pre-MDA MS C)

Production Readiness

Review Assesses system readiness to enter production.

Production and Deployment (post-MDA MS C)

Physical Configuration

Audit

Assesses the as-delivered system for compliance with the product baseline and supports full-rate production decision.

Production and Deployment (post-MDA MS C during initial operational capability)

Notes:

1. A best practice is for SRR to be accomplished in two parts.

SRR1 is to ensure the government has established performance requirements and non-tailorable design requirements that are directly traceable to the CDD.

Enclosure (1) Page 2 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

SRR2 is a technical assessment of the developing system specification under review to ensure a reasonable expectation of the final system being judged operationally effective and suitable.

2. PDR – A best practice is for PDR to be accomplished in two parts, an initial PDR and a Closure PDR.

The PM should plan for a PDR before Milestone B, consistent with associated prototyping requirements. If a PDR has not been conducted prior to Milestone B, the PMs shall plan to accomplish a minimum set of PDR requirements to support SDS development. A minimum MS B preparatory PDR represents a physically architected system based on full engagement of subsystem suppliers and knowledge gained through prototyping and identified in the technology development strategy. Following the Closure PDR, the PM shall send a PDR closure report to the MDA.

Enclosures (4) through (8) use a short-hand notation for the two PDR SETR events. PDR1 criteria were developed for use at a pre-Milestone B PDR, while PDR2 criteria were developed for use post-Milestone B.

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Enclosure (2) Page 1

Enclosure (2). Software Centric Artifacts and Core Metrics, with their governing policy, guidance, and standards

This table provides the list of software centric artifacts along with Department of the Navy (DoN) policy, guidance from both Navy and commercial sources, and applicable IEEE/EIA standards. DoN policy requires that offerors submit a Software Development Plan (SDP) with their proposals and after contract award (see Sections 7.4.1, 8.1.1, and Appendix E of the Guidebook for Acquisition of Naval Software Intensive Systems for additional information). DoN policy also requires the use of core metrics for all software intensive systems (see section 5 of this supplement). All other artifacts may be tailored by the program office to meet the needs of the specific program (see section 7 of this supplement). However, policy requires that the information content of IEEE/EIA Standard 12207.1 documents “must be provided in some format, as appropriate, for the proposed work effort.”

Software Centric Artifacts Policy Guidance Standards

Prime/Subs Software Development Plans (SDPs) Requirements Management Plan Configuration Management Plan

ASN(RD&A) memo: Software Process Improvement Initiative (SPII) Contract Language, 17 November 2006

ASN(RD&A) memo: SPII Guidance for use of SPI Contract Language, 13 July

ASN(RD&A) memo: DoN Policy for Acquisition of Naval Software Intensive Systems, 16 Sep 2008

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008, Appendix L.

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

IEEE/EIA Std 12207.1-1997, Clauses 6.5, 6.8, 6.9, 6.11, and 6.14

Software Acquisition Management Plan (SAMP)

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) memo: DoN Policy for Acquisition of Naval Software Intensive Systems, 16 Sep 2008

CMMI® for Acquisition, Ver 1.2, Addison Wesley SEI Series, Traceability Matrix (RTM)

Naval SYSCOM Systems Engineering Policy, 7 Jul 2009

(NAVSEAINST 5000.9

NAVAIRINST 5000.24

SPAWARINST 5000.1

MARCORSYSCOM Order 5400.5)

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

Enclosure (2) Page 2 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

Measurement Plan

ASN(RD&A) memo: DoN Software Measurement Policy for Software Intensive Systems, 22 July 2008

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) memo: DoN Policy for Acquisition of Naval Software

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

CMMI® for Development 2nd

“Measurement for DoD Projects”, DoD Implementation Guidance, Practical Software and Systems Measurement, 24 February 2003

IEEE 1058, section 4.5.3.6

Software Requirements Description (SRD) - Includes:

SW Requirements Specification (SRS) Interface Requirements Spec

(IRS)

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

CMMI® for Development 2nd

IEEE/EIA Std 12207.1-1997, Clause 6.22

Software Architecture Description (SAD)

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

IEEE/EIA Std 12207.1-1997, Clause 6.12

Software Interface Design Description (SIDD)

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

IEEE/EIA Std 12207.1-1997, Clause 6.19

Software Design Description

(SDD)

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

IEEE/EIA Std 12207.1-1997, Clause 6.16

Computer Software Configuration Items (CSCIs)

Clauses 6.7 and 6.13

System/Software Integration Plan

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

CMMI® for Development 2nd

Clause 6.18

Software Test Plan (STP)

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

CMMI® for Development 2nd

Clause 6.27

Software criteria and guidance for Systems Engineering Technical Reviews (SETRs) Enclosure (2) Page 3

Software Build Plan SYSCOM best practice Operator Guide/Users Guide SYSCOM best practice Software Product Baseline SYSCOM best practice

Test Problem/Trouble Report

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

IEEE/EIA Std 12207.1-1997, Clause 6.10

Test Procedures/Scripts/Cases

ASN(RD&A) memo: Software Process Improvement Initiative Contract Language, 17 November 2006

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep. 2008

IEEE/EIA Std 12207.1-1997, Clause 6.28

Software Life Cycle Sustainment Plan (appendix to System LCSP)

CMMI® for Acquisition, Ver 1.2, Software Risks/Mitigation Plans

Naval SYSCOM Risk Management Policy, 21 Jul 2008

(NAVSEAINST 5000.8

NAVAIRINST 5000.21B

SPAWARINST 3058.1

MARCORSYSCOM Order 5000.3) Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) Guidebook for

Intensive Systems, Sep. 2008

Risk Management Guide for DOD Acquisition, Sixth Edition, Version 1.0, Aug. 2006

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

CMMI® for Acquisition, Ver 1.2, IEEE 1058, section 4.5.4

Enclosure (2) Page 4 Software criteria and guidance for Systems Engineering Technical Reviews (SETRs)

Core Software

Metrics Policy Guidance Standards

Software Core Metric

– Software Size/Stability

ASN(RD&A) memo: DoN Software Measurement Policy for Software Intensive Systems, 22 July 2008

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) memo: DoN Policy for

ASN(RD&A) Guidebook for Acquisition of Naval Software Intensive Systems, Sep.

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

“Measurement for DoD Projects”, DoD Implementation Guidance, Practical Software and Systems Measurement, 24 February 2003

IEEE 1058,

section 4.5.3.6

Software Core Metric

– Software Cost/Schedule

ASN(RD&A) memo: DoN Software Measurement Policy for Software Intensive Systems, 22 July 2008

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) memo: DoN Policy for

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

“Measurement for DoD Projects”, DoD Implementation Guidance, Practical Software and Systems Measurement, 24 February 2003

IEEE 1058,

section 4.5.3.6

Software Core Metric

– Software Quality

ASN(RD&A) memo: DoN Software Measurement Policy for Software Intensive Systems, 22 July 2008

Public Law 107-314, Section 804 (FY-03 Defense Authorization Act, dated 2 Dec 2002)

ASN(RD&A) memo: DoN Policy for

CMMI® for Development 2nd Edition, Ver 1.2, Addison Wesley SEI Series, 2007

“Measurement for DoD Projects”, DoD Implementation Guidance, Practical

Software Core Metric

– Software Organization

ASN(RD&A) memo: DoN Software Measurement Policy for Software…