Attachment_0095_-_Best_Practices_for_using_SE_(01APR2017).pdf

Best Practices for Using

Systems Engineering Standards

(ISO/IEC/IEEE 15288, IEEE 15288.1, and IEEE 15288.2) on

Contracts for Department of Defense Acquisition Programs

April 2017

Prepared by:

Office of the Deputy Assistant Secretary of Defense for Systems Engineering

Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

NOTE: As industry consensus standards are published by a non-Government Standards Development Organizations (SDO), users of the 15288 Standards should remain aware of copyright and licensing limitations associated with these documents. Please review the copyright and licensing statements contained in each of the 15288 Standards to avoid inadvertent copyright violations when tailoring and applying them on contract.

Licensing of the standard may limit distribution of industry published standards. Clarification of copyright law or licensing issues associated with using these standards on contract should be addressed with Government and/or SDO legal staff.

Best Practices for Using Systems Engineering Standards (ISO/IEC/IEEE 15288, IEEE 15288.1, and

IEEE 15288.2) on Contracts for Department of Defense Acquisition Programs

Office of the Deputy Assistant Secretary of Defense Systems Engineering 3030 Defense Pentagon 3C167 Washington, DC 20301-3030 http://www.acq.osd.mil/se/

Distribution Statement A: Approved for public release. Distribution is unlimited.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

Contents

Introduction Background Purpose 15288 Standards Overview

Overview

Systems Engineering Planning Prior to Request for Proposal Planning for Use of Systems Engineering Standards Use of 15288 and 15288.1 on Contract Use of 15288.2 on Contract Tailoring Considerations

Request for Proposal and Source Selection Development of the Request for Proposal Suggested Request for Proposal Language Offeror Response to Request for Proposal Proposal Evaluation and Source Selection

Contract Execution Process Compliance Output Compliance Technical Review Compliance

Appendix A – How Project Characteristics May Drive Tailoring of the 15288 Standards

Appendix B – Work Aid for Definition of Outputs to Be Supplied

Appendix C – Example Tailoring Document Template

Appendix D – Definitions and Acronyms Definitions Acronyms

Appendix E – References

Figures

Figure 1: DoD Employment of 15288 Standards in Acquisition Contracts

Tables

Table 1: Systems Engineering Life Cycle Processes

Table 2: Technical Reviews and Audits

Table 3: Project Characteristics That May Drive Tailoring

Table 4: Section C Recommended Clauses

Table 5: Section L Recommended Provisions

Table 6: Section M Recommended Provisions

Table A-1: Mission Application Subfactors

Table A-2: Mission Application Subfactors Examples

Table A-3: Example - Summary of Space Vehicle Risk Class Attributes

Table B-1: Work Aid for Definition of Outputs

Table C-1: Notional Tailoring Document Template

Introduction

BACKGROUND

The Department of Defense (DoD) and the defense industry have found that applying systems engineering (SE) processes and practices throughout the system life cycle improves project performance, as measured by the project’s ability to satisfy technical requirements within cost and schedule constraints.1 Simply put, projects that use effective SE processes perform better than those that do not. Given this knowledge, it is in the best interest of both acquirers and suppliers to ensure that defense acquisition projects use effective SE processes as the core of the technical management effort. 2

In addition, the use of standards in key technical disciplines, such as SE, can enhance project performance and provide a common framework for communicating best practices for implementing effective SE on DoD acquisition projects. Consistent with the Public Law 114-3283, Public Law 104-113,4 and with the Office of Management and Budget (OMB) Circular A-119, 5 DoD encourages the adoption of voluntary consensus standards, where practical, rather than developing new or updating existing Government-unique specifications and standards.

A “voluntary consensus standard” is a non-Government standard (NGS), developed and accepted by subject matter experts in Government, industry, and academia.6 The revised OMB Circular A-119 maintains a strong preference for voluntary consensus standards over Government-unique standards and provides factors for agencies to consider when evaluating whether to use a standard to meet agency needs.

“Adoption” is the process by which DoD expresses formal acceptance of an NGS for use in direct procurement, as a reference in another document, or as guidance in the design, manufacturing, testing, or

1 Weapon System Requirements: Detailed Systems Engineering Prior to Product Development Positions Programs for Success. GAO-17-77, Government Accountability Office, November 2016.

2 For purposes of this guide, the “acquirer” refers to a DoD Program Management Office or other Government procurement organization, although in practice the acquirer may be a prime contractor in relation to subcontractors.

The “supplier” refers to the selected offeror. The term “project” is used throughout, but all references to a project are applicable to a program. In addition, a defense program may consist of multiple projects for different phases, subsystems, or suppliers.

3 Public Law 114-328, National Defense Authorization Act for Fiscal Year 2017.

4 Public Law 104-113, National Technology Transfer and Advancement Act of 1995.

5 Office of Management and Budget (OMB) Circular A-119, “Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformity Assessment Activities,” revised January 2016.

6 DoD Manual 4120.24, Defense Standardization Program (DSP) Procedures, September 24, 2014.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

support of materiel. 7 An adopted NGS is not a mandatory document; it is simply available for use by DoD activities. Therefore, it is up to each Program Management Office (PMO) to determine if and how the standards should be used to support a particular project.

PURPOSE

DoD has adopted the voluntary consensus standard ISO/IEC/IEEE8 15288, “Systems and Software Engineering–System Life Cycle Processes,” for use by acquisition projects. The standard establishes a common process framework for describing the life cycle of man-made systems and defines a set of SE processes and associated terminology typical for the full system life cycle, including conception, development, production, utilization, support, and retirement.

DoD has also adopted the companion standards IEEE 15288.1, “Standard for the Application of Systems Engineering on Defense Programs,” and IEEE 15288.2, “Standard for Technical Reviews and Audits on Defense Programs,” that define requirements for SE processes, technical reviews, and audits for defense projects. Both companion standards, as well as 15288, were all developed to enable direct citation on DoD acquisition contracts. Throughout the remainder of this document, these three standards will be referred to collectively as the 15288 Standards. DoD projects may use the documents collectively or individually to support effective technical performance on DoD contracts.

The purpose of this document is to assist:

• Acquirers in tailoring the 15288 Standards to meet and communicate project needs

• Acquirers in incorporating appropriate language into a Request for Proposal (RFP) to invoke the standards and express relative importance of the standards in proposal evaluations

• Offerors in developing their proposals to leverage existing organizational processes, or propose alternative value-added tailoring, to support the RFP requirements and comply with the standards as tailored

• Acquirers in evaluating an offeror’s ability and commitment to effectively implement SE processes compliant with acquirer’s requirements based on the proposed Systems Engineering Management Plan (SEMP), project plan, master schedule, and past performance

• Acquirers in monitoring and enforcing a supplier’s compliance with the contract and delivery of the product/service/system

7 SD-9, “DoD Guidance on Participation in the Development and Use of Non-Government Standards,” Defense Standardization Program Office, March 9, 2005.

8 International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC)/Institute of Electrical and Electronics Engineers (IEEE)

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

15288 STANDARDS OVERVIEW

ISO/IEC/IEEE 15288, “Systems and Software Engineering–System Life Cycle Processes,” establishes a common framework for describing the life cycle of man-made systems and defines a set of processes and associated terminology from an engineering viewpoint (Table 1). These processes can be applied at any level within a system’s hierarchy structure and along any stage of the system’s life cycle.

The processes involve all stakeholders, with the ultimate goal of achieving customer satisfaction.

ISO/IEC/IEEE 15288 also provides guidance for definition, control, and improvement of the organization or project’s system life cycle processes. Organizations and projects may use these processes when acquiring and supplying systems. The standard addresses man-made systems that may be configured with one or more of the following elements: hardware, software, data, humans, processes (e.g., processes for providing service to users), procedures (e.g., operator instructions), facilities, materials, and naturally occurring entities.

Table 1: Systems Engineering Life Cycle Processes

Agreement Processes

• Acquisition

• Supply

Technical Management Processes

• Project Planning

• Project Assessment and Control

• Decision Management

• Risk Management

• Configuration Management

• Information Management

• Measurement

• Quality Assurance

Technical Processes

• Business or Mission Analysis

• Stakeholder Needs and Requirements

Definition

• System Requirements Definition

• Architecture Definition

• Design Definition

• System Analysis

• Implementation

• Integration

• Verification

• Transition

• Validation

• Operation

• Maintenance

• Disposal

Organizational Project-Enabling

Processes

• Life Cycle Model Management

• Infrastructure Management

• Portfolio Management

• Human Resource Management

• Quality Management

• Knowledge Management

Source: ISO/IEC/IEEE 15288, “Systems and Software Engineering–System Life Cycle Processes”

IEEE 15288.1, “Standard for the Application of Systems Engineering on Defense Programs,” is a companion standard to ISO/IEC/IEEE 15288, which expands on the SE life cycle processes with additional detail specific to DoD acquisition projects. It adds requirements for SE outputs and the attributes (criteria) for each. The addition of outputs reflects a practical information-based management approach while acknowledging that SE processes are the basis for executing the required work activities. The DoD’s SE acquisition approach is based on a contractual acquirer-supplier agreement for the technical activities to be performed, the SE outputs to be produced, and the ability of the acquirer to assess those activities and outputs. IEEE 15288.1 should be part of the acquirer-supplier agreement, tailored by the acquirer to define the requirements for SE processes applicable to the specific project, including tasks to be performed, outcomes to be achieved, and outputs to be developed.

IEEE 15288.2, “Standard for Technical Reviews and Audits on Defense Programs,” is a companion standard that provides detailed definition, requirements, and evaluation criteria for the technical reviews and audits associated with DoD acquisition projects (Table 2). The acquirer should tailor IEEE 15288.2 to specify the

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

reviews, criteria, and expected outcomes appropriate to the specific acquisition project. Through appropriate tailoring and contract negotiation, the acquirer and supplier establish an agreement with specific criteria to be met in each applicable review.9

Table 2: Technical Reviews and Audits

General Technical Reviews and Audits Domain-Specific Technical Reviews

• Alternative Systems Review

• System Requirements Review

• System Functional Review

• Preliminary Design Review

• Critical Design Review

• Test Readiness Review

• Functional Configuration Audit

• System Verification Review

• Production Readiness Review

• Physical Configuration Audit

• Software Requirements and Architecture Review

• Software Specification Review

• Integration Readiness Review

• Flight Readiness Review

Source: IEEE 15288.2, “Standard for Technical Reviews and Audits on Defense Programs.”

SE experts in Government, industry, and academia worked with IEEE to produce standards 15288.1 and

15288.2 to enable direct citation of SE processes and technical reviews and audits on DoD acquisition contracts. The DoD intends to use the 15288 Standards as compliance documents in the acquirer-supplier agreement. As with any standard, the 15288 Standards should be tailored appropriately for the specific contractual application and applied in a manner that satisfies the business and mission needs and constraints of the project. When suitably tailored, these standards provide explicit requirements for SE processes, technical reviews, and audits to be resourced and executed during the course of the project.

“Suitably tailored” means tailoring the breadth and depth of the activities or tasks to be performed and the criteria associated with the outputs, along with removing those outside the scope of the acquisition. It does not mean eliminating standard SE processes, derived from industry best practices. Programs should use Appendix A as a guide in tailoring the standard. The tailored 15288 Standards, along with project requirements and other acquisition planning documents such as the acquirer’s Systems Engineering Plan (SEP), are part of a clearly stated and properly scoped contract. The inclusion and use of these documents in a contract is meant to ensure that the acquirer’s requirements are bid effectively, resourced appropriately, reflected accurately in the supplier’s proposed SEMP, project plan, and/or schedule and, ultimately, executed in a manner commensurate with effective technical practices.

9 IEEE 15288.1 and IEEE 15288.2 are available at no charge to DoD military and civilian employees via ASSIST, the Defense Logistics Agency (DLA)-sponsored database for DoD specifications and standards. DoD military and civilian employees may view, download, and print these standards for personal use but may not further copy, prepare, or distribute copies of the standard, or significant portions of the standard, subject to the terms of use of the standard. Instructions for how DoD military and civilian employees can access the standards via ASSIST are located on the DASD(SE) website, http://www.acq.osd.mil/se/docs/ASSIST-Access-15288.pdf. Others can purchase all three 15288 Standards via IEEE at http://www.ieee.org.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

Overview

To establish a project with an effective SE approach in the competitive environment of most DoD acquisitions, the system acquirer should:

1. Stress the importance of SE within the scope of the overall acquisition

2. Define the acquirer’s expectations, generally expressed in requirements, for a supplier’s SE processes (outcomes, activities, and/or outputs) and technical reviews and audits

3. Levy requirements on the supplier, via the contract, to perform effective SE

4. Ensure the supplier’s SE efforts are appropriately funded and resourced

5. Ensure a means for the supplier to demonstrate compliance with those requirements

The 15288 Standards provide one method to define the acquirer’s expectations and requirements for the supplier’s performance of SE processes and technical reviews and audits. Thoughtful and proper use of these standards can enhance communication and understanding between the acquirer and supplier throughout the solicitation process and contract execution. Figure 1 illustrates how to successfully implement the 15288 Standards in DoD acquisition contracts.

CDRL: Contract Data Requirements List; DAL: Data Accession List; ID: Identifier; IMP: Integrated Master Plan; IMS: Integrated Master Schedule; RFP: Request for Proposal; SE: Systems Engineering; SEMP: Systems Engineering Management Plan; SEP: Systems Engineering Plan; SOW: Statement of Work; TIM: Technical Interchange Meeting

Figure 1: DoD Employment of 15288 Standards in Acquisition Contracts

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

The process starts with the acquirer outlining the SE needs of the project. Not all projects will use SE processes, technical reviews, and audits within the 15288 Standards to the same extent. For example, a project for follow-on production of a previously designed system may not use the Architecture Definition, Design Definition, and other design-related processes (Table 1, p. 3) to a large degree. A project to develop a proof-of-concept prototype for testing and evaluation may not use all elements of the Maintenance process. Similarly, a project contracting for the Engineering and Manufacturing Development phase most likely will not conduct an Alternative Systems Review.

Upon considering the project’s objectives, needs, and constraints, the acquirer should tailor the 15288 Standards by adding, modifying, or removing process, output, and/or technical review requirements to define the scope of work to be included in the contract effort (see ISO/IEC/IEEE 15288:2015; Annex A for guidance on the tailoring process). The resulting tailored standards define the acquirer’s requirements for SE to be performed by the supplier. Direct-cite standards in a DoD RFP clarify and strengthen the Statement of Work (SOW). These expectations should be discussed with potential offerors during Industry Days and/or draft RFP reviews if conducted, and offeror comments may be considered in the development of the final RFP.

The final RFP and SOW, developed and issued by the acquirer, should cite the tailored 15288 Standards, which define the requirements for performance of SE processes and technical reviews for the project. The RFP should:

• Identify the applicable requirements of the 15288 Standards

• Instruct the offerors to respond to the 15288 Standards requirements and describe how they plan to meet the applicable requirements in their proposal, SEMP, project plan, and/or schedule

• Encourage offerors to use existing organizational processes that satisfy those requirements, and/or propose alternative processes with accompanying rationale

The offeror’s response, contained in the proposal, SEMP, project plan, and schedule, should describe the plan to execute and resource the SE effort to meet the requirements of the RFP. Offerors should provide credible assurance of their capability to implement SE processes and conduct technical reviews and audits consistent with the acquirer’s requirements. An offeror’s approach in a response may include:

• Providing assertions of full or partial conformance to the 15288 Standards

• Providing a mapping between the standards and existing organizational processes and outputs that includes identification of any gaps

• Providing certifications or ratings for other relevant standards or models along with a mapping to the acquirer’s requirements reflected in the tailored standards

• Providing evidence of past performance on projects where processes and/or technical reviews consistent with the 15288 Standards were used or conducted

• Proposing alternative tailoring to the 15288 Standards that demonstrate the benefits to the acquirer as well as the risks associated with the alternative

The acquirer should not seek to mandate specific techniques or methods (“how” requirements) for execution of SE processes. Rather the acquirer should enable suppliers to leverage existing organizational processes already in place to satisfy industry best practices with acceptable rigor. Responding to the acquirer’s requirements does

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

not necessitate strict adherence to the 15288 Standards. It should be acceptable for the supplier to reflect the intent of the tailored 15288 Standards in a manner that provides maximum value, both technical and programmatic, to the acquirer and supplier. The offeror should also be encouraged to consider further tailoring of the 15288 Standards if, by doing so, the offeror provides greater value to the acquirer, while still meeting the acquisition goals.

The acquirer reviews the offeror’s proposal to determine whether the proposed approach for implementing SE processes and technical reviews and audits demonstrates compliance with the acquirer’s requirements and shows an understanding of the project’s technical challenges. The acquirer should ensure that all required processes, outputs, and reviews have been addressed and are reflected in the proposed planning artifacts;

SEMP/IMP/IMS) and that management and technical resources (funding, personnel, and facilities/equipment) will be in place to ensure both acquisition and mission success. The acquirer may request additional process descriptions and information to support a more detailed assessment of conformance to the standard’s requirements.

When a contract is awarded, the acquirer-supplier agreement baselines the applicable requirements of the standards and the supplier’s proposed planning documents, including the SEMP, IMP, and IMS. This formalizes the acquirer-supplier agreement for SE – the commitment by the supplier to perform the agreed-upon activities and reviews, achieve the agreed-upon outcomes, and provide the agreed-upon outputs.

Throughout contract execution, the acquirer should assess the supplier’s compliance with the contract as well as satisfaction of project objectives. Assessment of SE processes, outputs, and outcomes may be accomplished through: participation alongside the supplier, during ongoing process monitoring at specified checkpoints or during technical reviews, and/or evaluation of deliverables provided by the supplier. Ideally, the acquirer will assess compliance using appropriate measures, preferably with leading indicators rather than lagging indicators. Monitoring the supplier’s SE activities and outputs along the way can serve as a leading indicator of project performance, enabling the acquirer to identify project or product shortfalls early and address them in a timely and economical manner. Other examples of leading indicators include:

• Requirements trends (requirements stability, stakeholder needs met)

• Interface trends

• Risk burn-down

• Technical Performance Measure (TPM) trends/summary

• Technical maturity

Sections 3 through 5 provide more detailed information on this process.

Systems Engineering Planning Prior to Request for Proposal

Early technical planning is essential to project success. Before issuing an RFP, the acquirer documents the program’s overall SE approach in the SEP, which then guides all technical activities of the program. The SEP describes the integration of SE activities with other program management and control efforts, including the Integrated Master Plan (IMP), Work Breakdown Structure (WBS), Integrated Master Schedule (IMS), Risk Management Plan (RMP), Technical Performance Measures (TPM), and other documentation fundamental to successful program execution. The SEP also describes the program’s technical requirements, engineering resources and management, and technical activities and products as well as the planning, timing, conduct, and success criteria of event-driven technical reviews and audits throughout the acquisition life cycle. The acquirer should use the SEP to communicate the technical approach to offerors and suppliers.

PLANNING FOR USE OF SYSTEMS ENGINEERING STANDARDS

The 15288 Standards describe a generic perspective of SE processes and technical reviews. The applicability of that perspective will vary for specific projects, acquisition phases, and/or suppliers. Therefore, the acquirer should initially tailor the standards to meet the specific business or mission needs and constraints of the project, balancing the cost of using the standards with the risk reduction achieved from their use. This initial tailoring establishes the acquirer’s SE expectations in the RFP and provides a baseline against which offerors should bid. All tailoring decisions should take into consideration the cost-benefit trade-off of including or excluding specific elements of the standards. In doing so, the potential negative impacts and unintended consequences should be considered.

The value of the 15288 standards is not in the rote performance of the processes and technical reviews they define. Rather, it is in the thought processes and resulting outcomes of the activities that enable better decision making from both technical and programmatic perspectives. As such, the objective should not be to mandate strict task-level conformance to the standards or to require task-level verification of compliance, but to ensure that the intent of the standard is met in a way that is most efficient and effective to achieve the outcomes and outputs expected by performance of the processes. Over-prescriptive specification by the acquirer can lead to significant cost impacts and may potentially constrain supplier innovation that could be of mutual benefit.

When deciding how to use standards to support contracting for SE, the acquirer should consider the project characteristics and the scope of the contracted effort. Tailoring the 15288 standard requirements for SE processes and technical reviews and audits should provide a balanced approach to cost, schedule, and performance at acceptable risk. The project characteristics described in this section and in Appendix A provide insight and examples that may be useful to consider as part of the acquirer’s tailoring decisions.

Other acquirer-specific considerations that may impact selection and tailoring of standards include:

• Acquisition model and applicable life cycle phase(s)

• Acquisition type (open and competitive vs. sole source)

• Funding profile, organization of project office, and size of support staff

• Past performance of known or expected bidders

• Mission criticality and risk acceptance levels

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

USE OF 15288 AND 15288.1 ON CONTRACT

The acquirer should identify applicable requirements of the 15288 and 15288.1 standards by clause as part of the solicitation. The acquirer should define and tailor the requirements for SE outcomes, activities, and outputs so they meet the project’s need for information to provide visibility into the success of product or system development and support decision making to promote programmatic and mission success.

The acquirer should focus on ISO/IEC/IEEE 15288 Clause 6, in particular Clauses 6.X.X.2 and 6.X.X.3, when tailoring to identify desired process outcomes and activity requirements for the supplier. Based on a thorough SE analysis, the acquirer should tailor out process outcomes or activities that do not provide enough value to justify their associated costs.

In IEEE 15288.1, the acquirer should focus on Clause 6.X.X.4 when tailoring to remove outputs and/or associated criteria that are clearly outside the scope of the project. The acquirer can further refine those within scope to reflect the specific project needs by tailoring the breadth/depth of the criteria associated with the outputs.

The acquirer should give specific attention to the tailoring of 15288.1 outcome, activity, and/or output clauses that include the phrase “... in accordance with the acquirer-supplier agreement.” Since the SOW, standard, and tailoring will constitute the contractual agreement, the desired outcomes, activities, and outputs should be clearly expressed to ensure the acquirer and supplier have the same understanding of all requirements.

The acquirer should be cautious when eliminating or significantly minimizing any process element in the standard. The acquirer should consider all potential negative impacts and unintended consequences resulting from tailoring, as they may add risk to the project. Tailoring of SE outcomes, activities, and outputs should be balanced with the business/mission needs and project constraints.

USE OF 15288.2 ON CONTRACT

The acquirer should provide the applicable requirements of IEEE 15288.2 as part of the RFP to define the technical reviews and audits to be conducted in support of the project. The acquirer should give consideration to the different philosophies for technical review execution, for example, bottom-up (subsystem reviews culminating in a system-level review) versus top-down (system review before subsystem reviews). In tailoring, the acquirer should carefully consider the information necessary for informed decision making. The requirement for certain information should be balanced by business/mission needs and project constraints.

• The acquirer should remove any technical reviews that are not planned for the acquisition phase associated with the contract’s period of performance.

• For each applicable technical review or audit, the acquirer should review the evaluation criteria in Clause 6 and tailor accordingly. Evaluation criteria that are not applicable should be removed. Each evaluation criterion levied for technical reviews carries an associated element of cost that may not yield sufficient benefit. The selected criteria should reflect the level and type of risk for the project.

The normative requirements for conducting technical reviews and audits are provided in Clauses 5 and 6 of IEEE 15288.2. Clause 5 indicates which reviews are appropriate to the project. As written, all technical reviews and audits in Clause 5 are normative. Therefore in the tailoring, the acquirer should delete and

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

document any technical reviews or audits that are not appropriate. Any required change regarding the timing for technical reviews or audits should be made clear in Clause 5.X.3. Entry, exit, and success criteria for the reviews are normative and provided in Clauses 5 and 6.

Both the acquirer and supplier should consider and mutually agree upon the tailoring of the reviews in the contract to clarify the expectation for each applicable technical review or audit. The domain-specific technical reviews in Annexes A, B, C, and D are informative and therefore should be specifically invoked if desired as a contractual requirement.

The acquirer should also tailor out any requirements or responsibilities that will be performed solely by the acquirer. Specifically, the required preparation and closure actions in Clauses 6.X.2 and 6.X.4 (including those in the tables) should either be assigned to the supplier or tailored out.

When tailoring either the reviews or the criteria, the parties should remember that this standard does not task the engineering work efforts required to produce the review content. Engineering efforts should be tasked in the SOW and may have standards for practices, design criteria, or testing associated with those efforts. The focus of 15288.2 is limited to the review and approval of intermediate outputs by the acquirer to support effective project management.

TAILORING CONSIDERATIONS

All 15288 processes have been shown to add value to projects. However, the degree to which each process may be used on a project and the benefits that each provides can vary greatly based on characteristics of the project. For example:

• Larger or more complex projects may benefit from more comprehensive and formal application of these processes, whereas smaller or lower-risk projects may not need to support the same level of application to gain the requisite level of benefits.

• Projects developing systems that are safety-critical, mission-critical, or within regulated domains (e.g., medical, nuclear) may demand more comprehensive and formal application of some processes, as well as additional or more detailed outputs reflecting the risks inherent to the domain.

The application of the 15288 processes on a project may be influenced by various project characteristics.

Table 3 provides a sample list of project characteristics and sub-characteristics that may influence the tailoring of the 15288 processes and technical reviews and audits. Appendix A discusses characteristics in more detail.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

Table 3: Project Characteristics That May Drive Tailoring

Project Characteristic Sub-Characteristics

Life-cycle considerations • Acquisition life cycle phases covered

• Acquirer / supplier division of effort

• Duration of development effort

• Total cost of ownership

• Development life cycle model (e.g., rapid)

• Known or assumed funding profile

Mission application • Domain

• Mission criticality (“-ilities” required; domain regulations)

• Number of usage scenarios

• Number of deployment sites / environments

• Design for reusability

Organizational complexity • Number of development organizations

• Diversity of organizational viewpoints, for example based on corporate legacy

• Commonality and integration of standard processes or toolsets

• Reuse of existing components or intellectual property

• Staff experience, capability, size, and skills needed

Technical complexity • Number of requirements

• Number of system external interfaces

• Number of different types of users

• Number of system elements / internal interfaces /architectural levels

• Number of KPPs

• Total development cost

Risk • System precedence / technology availability

• Technology obsolescence

• Integration of the technology

• Programmatic / external risk

• Sustainment / disposal risk

• Manufacturing / supply chain risk

• Prior acquisition / system failures or past performance history

Technical understanding • Requirements understanding

• Architecture understanding

• Emergence likelihood

Request for Proposal and Source Selection

DEVELOPMENT OF THE REQUEST FOR PROPOSAL

When developing the RFP, the acquirer should carefully consider and be transparent about the type of response expected from the offerors. Many factors can influence the desired level of fidelity in an offeror’s response, including the relative importance of SE to the particular project, source selection resources, or prior experience with the offerors. Both preparation and evaluation of proposals is cost-intensive, so clear communication of expectations is important for both the acquirer and supplier organizations. This insight is important for offerors to ensure that they provide sufficiently detailed information to clearly convey their offer, and for the acquirer to ensure that they receive the necessary information to properly evaluate the proposals.

When SE is critical to the project, more detailed proposals may be desired. The most minimal response may be an offeror’s self-certification of conformance to the standards, while the maximum response may be a complete mapping of the relationship between the offeror’s SE processes and the requirements specified in the RFP, or provision of the offeror’s organizational process documentation for source selection review.

Intermediate responses may include citation of the offeror’s internal process documentation and description of how they meet the intent of the standards and requirements specified in the RFP. While the response types at the extreme may be desirable, they tend to involve different but significant source selection risks.

The acquisition type can dictate the offeror response. A “full and open” competition allows for limited and controlled communication between the source selection team and the offerors for clarification, whereas a sole source situation allows for much deeper and more interactive communication between the parties. In the former case, the proposal should provide sufficient information to allow the source selection team to assess and gain assurance that the proposal responds to the requirements, has adequate resourcing for effective task performance, and meets the intent of the standards. In the latter case, it is possible for detailed discussions in a more dynamic environment, which may be more amenable to detailed explanation of offeror practices and how they meet the requirements or intent of the standards. Contract type can also dictate the offeror response. A fixed price contract may call for a greater level of detail up front, whereas a cost reimbursable contract may allow for more flexibility at the initiation of the contract.

Tailoring of standards by the acquirer as part of RFP development communicates the acquirer’s expectations for supplier SE performance to prospective offerors, specifically regarding non-specification requirements.

Tailoring may include removal, modification, or addition of requirements. In many cases, the tailoring reflects the scale or rigor of the application of the processes in 15288.1 or the level of detail for the reviews of 15288.2.

In these cases, the tailoring will focus more on the outputs required from the execution of the process or the scope and criteria of the reviews or audits. Use of a direct-cite standard supports the SOW in documenting the work against which the offerors bid, and for which offerors may opt to propose additional or alternative tailoring. Tailoring of the 15288 Standards should be included in the RFP as an attachment to the SOW.

Due to copyright and licensing limitations, tailoring of industry-published standards precludes “marking up” and distributing the standard itself, unless specific permission is obtained from the SDO. Tailoring may need to be documented as change language indicating the requirements modification, deletion, or addition. For example: “Clause X.X.X, Change to read, ...”; “Clause X.X.X, Delete in its entirety”; or “Clause X.X.X, Add

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

the following additional requirements: ...” Appendix C provides an example tailoring document template that can be included in an RFP to convey the acquirer’s tailored set of requirements.

The acquirer should consider the intended delivery mechanism for SE process outputs. If the acquirer desires to receive formal delivery of an output for approval, then a Contract Data Requirements List (CDRL) item may be required. If the acquirer desires to have access to data and/or outputs, but not formal delivery, then those outputs can be specified in a Data Accession List (DAL). When appropriate, use of existing Data Item Descriptions (DID) should be considered for content specification only.

In addition, the acquirer should consider how the offeror will identify, acquire, and manage the technical data, computer software, intellectual property, and their associated license rights to support acquisition and sustainment of the system throughout its entire life cycle. These data rights considerations may affect the contracting strategy for SE process outputs and tailoring of the standards.

SUGGESTED REQUEST FOR PROPOSAL LANGUAGE

Applying the 15288 Standards on acquisition contracts involves two related but distinct concepts:

• Conformance: the extent to which an offeror’s proposed SE activities, tasks, and outputs align with the requirements of the IEEE 15288 standards, as tailored. Conformance of SE processes and technical reviews and audits to the IEEE 15288 standards is evaluated for contract award (i.e., via proposals and documentation submitted in response to a RFP and SOW).

• Compliance: the extent to which a supplier adheres to the contract by performing the required SE processes and technical reviews and audits, and generates the required outputs. Compliance is monitored and evaluated during project execution.

Once the acquirer determines the desired scope of conformance to the 15288 Standards, offerors are required to propose that conformance. The acquirer incorporates the conformance requirements into Section C (SOW), Section L, and Section M of the RFP.

The following tables provide recommended clauses for the acquirer to incorporate into the RFP. The following conventions are used in these clauses:

• Text contained in {} braces is optional and may be deleted

• Text contained in [] brackets is intended to be modified by the acquirer

Table 4 contains recommended clauses for the SE portion of the SOW. These clauses should be considered individually for addition to Section C.

Table 4: Section C Recommended Clauses

Section L of the RFP includes instructions to offerors. The acquirer should encourage offerors to propose alternative tailoring to the 15288 Standards, with appropriate justification. If a tailoring description is requested from the offeror, the response is recommended to be an appendix to the Technical Volume and not be page limited. The rationale for the alternative tailoring shall include risk and cost-benefit analysis (e.g., risk of tailoring compared with the risk of not tailoring). Table 5 contains recommended provisions for Section L, which the acquirer should consider individually for inclusion.

Table 5: Section L Recommended Provisions

Section M of the RFP includes the evaluation factors for how the acquirer will evaluate the proposals. Table 6 contains recommended provisions for Section M, which the acquirer should consider individually for inclusion.

Table 6: Section M Recommended Provisions

(SOWxx1) The Contractor shall define and implement systems engineering processes in conformance with IEEE 15288.1-2014{as tailored by [Ref tailoring document]}. Conformance shall be measured via the outcomes and outputs specified by 15288.1-2014{as tailored by [Ref tailoring document]}.

(SOWxx2) The Contractor shall define and conduct technical reviews and audits in conformance with IEEE 15288.2-2014 {as tailored by [Ref tailoring document]}. Conformance shall be measured via the outputs and criteria specified by 15288.2-2014{as tailored by [Ref tailoring document]}.

(SOWxx3) The Contractor shall deliver a Systems Engineering Management Plan (SEMP) that documents the system development approach applying systems engineering standards, including processes, outputs, technical reviews, and audits, that is consistent with the Government-provided Systems Engineering Plan (SEP) {RFP Document reference}.

(SOWxx4) The Contractor shall deliver an Integrated Master Plan (IMP) and Integrated Master Schedule (IMS) that includes the applicable technical reviews and audits as documented in the SEMP.

(L1:M1) The offeror shall, as part of its technical proposal, provide a Systems Engineering Management Plan (SEMP), Integrated Master Plan (IMP), and Integrated Master Schedule (IMS) to describe the implementation of IEEE 15288.1-2014 and IEEE 15288.2-2014{as tailored by [Ref tailoring document] and/or any alternative proposed tailoring} in its system engineering processes and schedule to meet the requirements of the Statement of Work (SOW), System Performance Specification, and other contractual requirements.

The offeror’s proposal will be evaluated based upon:

1. (M1:L1) The extent to which the offeror’s systems engineering approach and schedule demonstrate an appropriate balance of cost, schedule, and performance risk {including any alternative value-added tailoring options that meet the intent of IEEE 15288.1 and IEEE 15288.2}.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

The rationale for the tailoring should include risk and cost-benefit analysis for the tailoring (e.g., risk of tailoring compared with the risk of not tailoring). Eliminating or significantly restricting SE areas ostensibly to improve cost, schedule or performance may not actually add value when the associated risks are considered.

OFFEROR RESPONSE TO REQUEST FOR PROPOSAL

In general, the offeror’s SE approach will be documented in the proposal’s technical volume – execution details in the SEMP, scheduling details in the project plan and/or schedule (i.e. IMP/IMS), and resourcing details in the cost volume. The acquirer’s SEP, and the supplier’s SEMP where applicable, should define the planned acquirer/supplier implementation of SE processes and common understanding of expected outputs.

The tailored SE standards should be specified in an attachment to the supplier’s SOW. When baselined by contract, this forms a set of requirements for the scope of work to be performed along with the baseline against which compliance will be evaluated.

There are opportunities to improve the implementation of standards on contract after the pre-RFP tailoring.

During contract award and negotiation, the acquirer assesses the feasibility of the approach, and provides an opportunity for further tailoring of the standards to improve SE effectiveness, achieve a common understanding of conformance verification, and factor in affordability improvements. The acquirer should encourage suppliers to propose additional tailoring opportunities to the 15288 Standards, with appropriate justification. If a tailoring description is requested from the supplier, it is recommended the response be an appendix to the Technical Volume that does not count against the page limit.

During the proposal effort, offerors respond to the RFP requirements and may demonstrate alignment of their established organizational processes with those of the acquirer-tailored standard. Offerors may accomplish this by adapting, where necessary, existing organizational processes to conform to the standards, or by proposing alternative tailoring of the 15288 Standards in a manner that preserves the acquirer’s intent but aligns with the established organizational processes.

However, offerors should be aware that tailoring that does not strictly conform to the SOW, CDRL, or other RFP requirements may result in the proposal being rejected as non-responsive. Offerors should use Industry Days, responses to draft RFPs, and other discussions with the acquirer to determine the level of tailoring permitted, including the acceptability of alternate proposals that do not strictly conform to RFP requirements.

Given the acquirer’s requirements for 15288 conformance, as stated in the RFP, the offeror’s objective is to provide the most efficient and effective means of achieving that conformance. Often this is best achieved through the offeror’s existing organizational processes. In this case, the offeror’s challenge is to ensure that these existing processes satisfy the acquirer’s conformance requirements and to ensure that the proposal is responsive to the RFP.

If the RFP specifies that alternative tailoring proposals are acceptable, the offeror may submit proposals for alternative tailoring to the 15288 Standards. Such proposals should address the value added by the alternative tailoring, as well as the associated risks and opportunities. The end result is a collection of processes the offeror is committing to perform, a collection of outputs that the offeror is committing to provide either as CDRL or DAL items, and a commitment to conformance with the processes and outputs of the 15288 Standards.

Attachment 0095 Best Practices for using SE

W56HZV-18-R-0174

Analysis of the driving factors discussed in section 3.4, Tailoring Considerations, and Appendix A will also inform the offeror’s approach to tailoring.

The offeror’s proposed technical solution will further influence the tailoring. For example, consider a proposed solution that is a modification of a previously developed system. Since the system design already exists, the Architecture Definition and Design Definition processes employed will differ from those used on a completely novel design. These Architecture Definition processes will focus not on the creation of a new architecture, but on the adaptation of an existing architecture to the new application. Likewise the Design Definition process will focus on adapting existing designs to the new application. Prior experience will also influence the risks facing the offeror, concentrating them in areas of less experience.

A useful place to start in assuring conformance is to map the organizational processes and the outputs they produce to the 15288 processes and the 15288.1 outputs. Given such a mapping, the offeror may then identify those process and output gaps that need to be filled to provide the required level of conformance.

As a result of the mapping process, offerors should be able to specify a collection of standard or modified organizational processes that they will execute on the project, and a collection of standard or modified outputs that those processes produce. The offerors’ proposals should:

• Identify the organizational processes that will be performed to achieve conformance to the acquirer’s process requirements derived from IEEE/ISO/IEC 15288, and certify that those processes meet the stated requirements

• Identify the work products produced from these organizational processes that will satisfy the acquirer’s output requirements derived from IEEE 15288.1, and certify that those work products meet the stated requirements

• Identify the review and audit activities to be performed that will satisfy the acquirer’s technical review and audit needs derived from IEEE 15288.2, and certify that those review and audit activities meet the stated requirements

PROPOSAL EVALUATION AND SOURCE SELECTION

During source selection, the acquirer will assess offerors’ proposals to determine whether or not they are responsive to the RFP and, with regard to SE and technical reviews, conform to the 15288 Standards. The source selection will also assess bid realism and select the offer that provides the best overall technical and programmatic benefit.

The offeror’s proposal should:

• Clearly address the activities, outputs, reviews, and audits that are incorporated into the project

• Ensure that these conform to the 15288 Standards

• Demonstrate the ability and commitment to:

o successfully…