SMC-S-021_15SEP2009.PDF

BY ORDER OF THE COMMANDER SMC Standard SMC-S-21

15 September 2009

Supersedes:

New issue

Air Force Space Command

SPACE AND MISSILE SYSTEMS CENTER

STANDARD

TECHNICAL REVIEWS

AND AUDITS

FOR SYSTEMS,

EQUIPMENT AND

COMPUTER SOFTWARE

VOLUME 1

APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED

Downloaded from http://www.everyspec.com on 2012-04-12T19:31:41.

Contents

Paragraph Page

Foreword...................................................................................................................................................... iii Acknowledgment ......................................................................................................................................... iv

1. Scope

1.1 Purpose

1.2 Objectives of Technical Reviews

1.3 Classification

1.4 Application

1.5 Scheduling of Technical Reviews

2. Documents

3. Definitions of Technical Reviews and Audits

3.1 System Requirements Review (SRR)

3.2 System Functional Review (SFR)

3.3 Software Requirement and Architecture Review (SAR)

3.4 Preliminary Design Review (PDR)

3.5 Critical Design Review (CDR)

3.6 Test Readiness Review (TRR)

3.7 Functional Configuration Audit (FCA)

3.8 Physical Configuration Audit (PCA)

3.9 System Verification Review (SVR)

3.10 Manufacturing Readiness Review (MRR)

3.11 Production Readiness Review (PRR)

4. Technical Review Objectives

4.1 Reviews Specified

4.2 Technical Review Criteria

4.3 Technical Review Success

5. Roles, Responsibilities, and Authority

5.1 Contractor Participation and Responsibilities

5.1.1 Subcontractors and Suppliers

5.1.2 Chairperson

5.1.3 The Contractor Requirements

5.2 Procuring Contracting Officer (PCO) Participation and Responsibility

5.2.1 Procuring Contracting Officer (PCO) Role

5.2.2 Review Panel and Participant Selection

5.3 Formal Acknowledgment

5.4 Data Requirements List and Cross Reference

6. Relevant Technical Review and Audit Items

6.1 Source Authority and Material

6.2 ACAT Programs and Contractual Guidance

6.3 “Acceptance Criteria”

6.4 Systems Engineering Process Objectives

6.5 Technology Readiness Assessment (TRA)

Appendix A - System Requirements Review (SRR)

10. System Requirements Review (SRR)

10.1 General

10.2 Purpose

10.3 Objective

10.4 SRR “Acceptance Criteria”

10.4.1 Systems Engineering and Architecture Development

10.4.2 System (System of Systems, Family of Systems), Segment, and Subsystem Design

10.4.3 System, Segment, and Subsystem Verification and Validation

10.4.4 Engineering Disciplines and Specialty Engineering

10.4.5 Integrated Technical Risk Management and Mitigation

Appendix B - System Functional Review (SFR)

20. System Functional Review (SFR)

20.1 General

20.2 Purpose

20.3 Objective

20.4 SFR “Acceptance Criteria”

20.4.1 Systems Engineering and Architecture Development

20.4.2 System, Segment, and Subsystem Design

20.4.3 System, Segment, and Subsystem Verification and Validation

20.4.4 Engineering Disciplines and Specialty Engineering

20.4.5 Integrated Technical Risk Management and Mitigation

Appendix C - Software Requirement and Architecture Review (SAR)

30. Software Requirement and Architecture Review (SAR)

30.1 General

30.2 Objective

30.3 Items To Be Reviewed

30.4 SAR “Acceptance Criteria”

30.5 Post Review Action

Appendix D - Preliminary Design Review (PDR)

40. Preliminary Design Review (PDR)

40.1 General

40.2 Purpose

40.3 Objective

40.4 PDR “Acceptance Criteria”

40.4.1 Systems Engineering and Architecture Development

40.4.2 System, Segment, and Subsystem Design

40.4.3 System, Segment, and Subsystem Verification and Validation

40.4.4 Engineering Disciplines and Specialty Engineering

40.4.5 Integrated Technical Risk Management and Mitigation

Appendix E - Critical Design Review (CDR)

50. Critical Design Review (CDR)

50.1 General

50.2 Purpose

50.3 Objective

50.4 CDR “Acceptance Criteria”

50.4.1 Systems Engineering and Architecture Development

50.4.2 System, Segment, and Subsystem Design

50.4.3 System, Segment, and Subsystem Verification and Validation

50.4.4 Engineering Disciplines/Specialty Engineering

50.4.5 Integrated Technical Risk Management and Mitigation

Appendix F - Test Readiness Review (TRR)

60. Test Readiness Review (TRR)

60.1 General

60.2 Objectives

60.3 Items To Be Reviewed

60.4 TRR “Acceptance Criteria”

Appendix G - Functional Configuration Audit (FCA)

70. Functional Configuration Audit (FCA)

70.1 General

70.2 Contract Requirements

70.3 Contractor Responsibility

70.4 Procedures and Requirements

70.5 Post-Audit Actions

Appendix H - Physical Configuration Audit (PCA)

80. Physical Configuration Audit (PCA)

80.1 General

80.2 Contract Requirements

80.3 Contractor Responsibility

80.4 PCA Procedures and Requirements

80.5 Post Audit Actions

Appendix I - System Verification Review (SVR)

90. System Verification Review (SVR)

90.1 General

90.2 Requirements

90.3 Post Review Action

Appendix J - Manufacturing Readiness Review (MRR) and Production Readiness Review (PRR)

100. Manufacturing Readiness Review (MRR) and Production Readiness Review (PRR)

100.1 General

100.2 Manufacturing Readiness Review (MRR)

100.3 Production Readiness Review (PRR)

Appendix K - Application Guide for Tailoring MIL-STD-1521

110. Application Guide for Tailoring MIL-STD-1521

110.1 Scope

110.2 Purpose

110.3 Objective

110.4 Considerations for Tailoring

110.4.1 Relationship to the Statement of Work

110.4.2 Elimination of Redundancy and Ambiguity

110.4.3 Contractor Participation in Tailoring

110.4.4 Complexity

110.5 Scheduling of Technical Reviews and Audits

110.6 Tailoring Guidance for Engineering Data Reviews

Glossary

Part I – Definitions Part II – Acronyms

Figures

Figure 1. Generic Technical Review Process Map Figure 2. TRA Relationship to Systems Acquisition Gates, Milestones, and Events Figure 3. Waterfall Software Development Life Cycle Model Figure 4. Incremental Software Development Life Cycle Model Figure 5. Evolutionary Software Development Life Cycle Model Figure 6. Engineering Data Discrepancy Sheet Figure 7. TRL Relationship to Systems Development Maturity

Tables

Table 1. Acquisition Policy Examples of Specific Technical Review and Audit Objectives Table 2. Sample Action Item Form Table 3. Scheduling Reviews and Audits Table 4. DoD MRL Definitions Table 5. DoD MRL Descriptions Table 6. DoD TRL Definitions Table 7. DoD Hardware and Software TRL Descriptions

1. Scope This document describes the objective requirements for the conduct of Technical Reviews and Audits on such end items (EIs) as a system, equipment, distributed and embedded Software Configuration Item (SWCI), Hardware Configuration Item (HWCI), and/or a combination thereof.

This document is an update to MIL-STD-1521B (USAF) Technical Reviews and Audits for Systems, Equipments and Computer Software, and will be issued as a USAF Space and Missile Systems Center (SMC) standard (STD) for planning and execution of key system engineering reviews. This update also documents a desired process and associated criteria for technical reviews and audits.

This document is composed of two volumes:

Volume 1 – defines a generic set of technical reviews and audits for systems, equipment, and computer software EIs and the guidance for the cost effective application and tailoring of this standard for the reviews described in Appendixes A through K:

Appendix A System Requirements Reviews (SRR) Appendix B System Functional Reviews (SFR) Appendix C Software Requirements and Architecture Review (SAR) Appendix D Preliminary Design Reviews (PDR) Appendix E Critical Design Reviews (CDR) Appendix F Test Readiness Review (TRR) Appendix G Functional Configuration Audit (FCA) Appendix H Physical Configuration Audit (PCA) Appendix I System Verification Review (SVR) Appendix J Production and Manufacturing Readiness Review (P/MRR) Appendix K Application Guide for Tailoring MIL-STD-1521

Volume 2 – provides specific and unique supplemental criteria content for the core technical reviews (SRR, SFR, SAR, PDR, and CDR) in Volume 1 for Space Systems specific equipment, and computer software end items (EIs) to be implemented per the contract Statement of Work (SOW). These criteria also include parts, materials, structures; systems engineering, tests and processes.

This update provides the Government’s requirements for criteria-based technical reviews, specified under the following five major categories:

1. Systems engineering and architecture development

2. System, segment and subsystem design

3. System verification and validation

4. Engineering disciplines and specialty engineering

5. Integrated technical risk and mitigation of the end item (EI) development maturity and operational utilization objectives in greater detail at the following core reviews:

Appendix A System Requirements Review (SRR) Appendix B System Functional Review (SFR) Appendix C Software Requirements and Architectures Review (SAR) Appendix D Preliminary Design Review (PDR) Appendix E Critical Design Review (CDR)

Downloaded from http://www.everyspec.com on 2012-04-12T19:31:41.

https://akss.dau.mil/dag/DoD5000.asp?view=document&doc=2 https://map.aero.org/FetchDocument?volID=-2052&objID=4556811 https://map.aero.org/mapv2/index.cfm?method=docSearch.details&volID=-2000&objID=2500083&subtypeId=1 https://map.aero.org/mapv2/index.cfm?method=docSearch.details&volID=-2000&objID=2500083&subtypeId=1

The update also identifies and delineates criteria for selected disciplines and specialty areas in varying detail and the associated work that shall be performed and documented in support of the above core reviews in terms of generic requirements criteria examples across all DoD systems developments. In addition, this update also defines the required quality attributes, sufficiency, and progress of the contractor’s documented accomplishments, along with the engineering disciplines and specialty processes, to be presented as specific evidence of the contractor’s accomplishments.

Terms and definitions related to general technical reviews are described in Section 3 of this document.

The general technical criteria required for all reviews are provided in Section 4. Section 5 describes the typical working relationships and roles and responsibilities between contractor and contracting agents, which encompass process steps that can be taken in preparation for and closure of each technical review.

The content of Section 5 is intended as source material for planning and negotiations for conduct of the technical reviews between the contracting agents.

1.1 Purpose

The technical reviews and audits are necessary systems engineering (SE) activities performed to assess technical progress within a program, relative to contractual requirements and developmental maturity.

Technical reviews of program progress shall be event-driven and conducted when the system under development meets the review entrance criteria as documented in the Systems Engineering Plan (SEP).

The technical reviews and audits shall include participation by subject matter experts who are independent of the program (i.e., peer review), unless specifically waived by the SEP approval authority as documented in the SEP.

The technical reviews are conducted by the contractors and subcontractors, also referred to as primes and subs at logical transition points in the development and design efforts, for an idealized EI in various phases of an EI’s development to:

1. Determine the contractor’s technical progress achieved to date

2. Compare the EIs’ performance against the requirements

3. Identify potential impediments and risks to each EI’s design execution

4. Determine mitigation plans to avert program schedule delays and unplanned resource expenditures.

The contracting agency shall perform initial tailoring of this document in accordance with (IAW) Appendix K Application Guide for Tailoring MIL-STD-1521, to require only what is needed for each individual acquisition, and address appropriate program scope, program size, and technical progress within the acquisition life cycle.

Technical Reviews and Audits defined herein shall be conducted in accordance with this standard to the extent specified in the contract clauses, Statement of Work (SOW), Compliance Standards List, and the Contract Data Requirements List that is based on the Government's need for technical data required to support the acquisition and life cycle support strategies. Guidance in applying this standard is provided in Section 4 and 5.

These technical reviews provide a method and forum for the primes and subs and the contracting agency to assess whether the status of the end item under development and the supporting documentation have met contract requirements and expectations, with an appropriate level of maturity, to continue to the next phase of the program with manageable risk.

1.2 Objectives of Technical Reviews

DoDI 5000.02 Operation of the Defense Acquisition System (December 2008) instruction, require that the contracting agent schedule and conduct technical reviews and audits (Table 1) with the objectives of enabling means, methods, and forum that provide for timely and critical insight, evaluation and assessment of the contractors technical progress for an evolving system design. Specifically, ascertain that:

1. The product and EI of the technical effort:

a. Will satisfy the intended system’s performance requirements and mission objectives

b. Is in compliance with security, environmental safety and occupational health (ES&OH) objectives

c. Is producible, testable, deployable, operable, supportable, maintainable, and reliable

2. The design and development process of the contractor(s) provide for incremental, realistic, measurable and achievable milestones for contractual deliverables

3. Methodologies are in place to identify potential problems or failures, associated risks to program resources (i.e., cost, schedule, and technical), and prevention or their mitigation thereof

This standard provides the contractor(s) with content to establish and maintain a technical review process that meets the contracting agent’s expectations and intent.

1.3 Classification

The following technical reviews and audits shall be selected by the program manager at the appropriate phase of program development. Each review and/or audit is defined in Appendixes A through J:

SRR - Appendix A System Requirements Review SFR - Appendix B System Functional Review SAR - Appendix C Software Requirements and Architectures Review PDR - Appendix D Preliminary Design Review CDR - Appendix E Critical Design Review TRR - Appendix F Test Readiness Review FCA - Appendix G Functional Configuration Audit PCA - Appendix H Physical Configuration Audit SVR - Appendix I System Verification Review M/PRR - Appendix J Manufacturing and Production Readiness Review

In order to achieve the generic technical review objectives outlined in Section 4.0, procedures, planning, and all other documentation and data that make up the reviews process shall be defined and made available to the contracting agent for review, assessment, and concurrence at a mutually agreed-upon time, prior to the conduct of each EI review. Roles and responsibilities for the preparation, conduct, and acceptance of each review are described in more detail in Section 5.

1.4 Application

The technical reviews and audits defined herein and designated by the contracting agency as called out in the contractual agreements shall be conducted in accordance with this standard to the extent specified in the contract clauses, Statement of Work (SOW), and the Contract Data Requirements List (CDRL) that defines the associated technical data package (TDP) type, elements and data management products required at the appropriate phases of program development. Scheduling of an EI technical review and the actual timing of the review activities shall be tailored for each program, using acquisition strategy and tailoring guidance provided in Section 4 and Appendix K.

Table 1. Acquisition Policy Examples of Specific Technical Review and Audit Objectives

Technical Review or Audit Objective DoDI 5000.02 Phase Analysis of Alternatives

(AoA) Concept Selection, System CONOPS Material Solution Analysis

System Requirements Review

(SRR)

Review SE Program Foundation and Approval of the Initial Requirements Baseline

Technology Development

System Functional Review

(SFR)

Review and Approval of the System Architecture and Functional Requirements Baseline

Technology Development

Software Requirements and Architecture Review

(SAR)

Review and Approval of the Software Architecture and Functional Requirements Baseline

Technology Development

Preliminary Design Review (PDR) Approval of the Allocated Baseline

Engineering and Manufacturing Development

Critical Design Review (CDR) Approval of the Design Baseline

Engineering and Manufacturing Development

Test Readiness Review

(TRR)

Verification of the Contractor’s Readiness to Begin a Formal Verification Testing

Engineering and Manufacturing Development

Functional Configuration Audit

(FCA)

Qualification of the Design Production and Deployment

Physical Configuration Audit

(PCA)

Approval of the Product Configuration Baseline

Production and Deployment

Manufacturing Readiness Review

(MRR)

Readiness for Production, Training, Deployment, Ops, Support, and Disposal

Production, Deployment, Operations and Support

Production Readiness Review

(PRR)

Authorize Follow-On Procurement of Additional System EIs

Complete Initial Small Quantity / Large Quantity Production-Centric Procurement

Production, Deployment, Operations and Support

1.5 Scheduling of Technical Reviews

The scheduling of a specific technical review is extremely important. If it is conducted too early, the end item for review will not be adequately defined. Conversely, if the review is too late, the program commitments could have been made erroneously, and correction will be both difficult and costly.

Scheduling and planning are program management functions and must be addressed in detail in the negotiated Integrated Management Plan (IMP) and Integrated Master Schedule (IMS) after the appropriate SE requirements have been tailored to the type of program contracted, e.g.:

a. Technology Development and/or Technology Demonstration (TD)

b. System Development and Demonstration (SDD)

c. Engineering Development (ED)

d. Risk Reduction and Design Development (RRDD), etc.

In accordance with the practice of good systems engineering principles, SRR and SFR shall be conducted as “top-down” reviews, where the requirements are established and appropriately allocated to the lower elements of the system under review. Conversely, PDR and CDR shall be conducted as “bottom-up” reviews from the lowest elements, culminating in an overall system review. SAR typically follows SFR on a build-by-build basis for an individual Software Configuration Item (SWCI) or a collection of SWCIs.

A good method for scheduling technical reviews is to relate them to the documentation requirements, e.g., schedule a PDR after the availability of hardware development specification or software architecture and detailed design and software test plans, since the essence of the PDR is to assess the contractor’s approach to meeting the requirements of these documents. Scheduling of technical reviews is dependent not only on documentation availability but also on hardware and software availability and the completion of the acceptance qualification tests.

Although a time frame for reviews is defined in the DoDI 5000.02 Operation of the Defense Acquisition System (08 Dec 2008) instruction, the review date for a given program may vary. The initial timing of each EI review shall be provided to the contracting agency by the qualified bidders as part of their proposal or IMP and IMS or as part of their systems engineering management plan (SEMP).

2. Documents Documents named or referenced within this standard, whether they are specifications, standards, handbooks, guides, drawings, CDRLs or Data Item Descriptions (DIDs), are NOT to be considered as compliance documents unless called out specifically by contract Statement of Work (SOW) or compliance documents list for the preparation and conduct of the design reviews. The SOW shall be referenced for applicable documents. (Copies of specifications, standards, drawings, and publications required by contractors in connection with specific procurement functions should be obtained from the contracting agency or as directed by the contracting officer).

The following list of documents were used as source material in preparation of this document

1. AFR 66-14, Equipment Maintenance Policies, Objectives, and Responsibilities

2. AFSPC Manual 91-710 Range SW Safety STD (1 May 2004)

3. ANSI/EIA 649A National Consensus Standard for Configuration Management (2006)

4. ASC/EN GUIDE – USAF Aeronautical Systems Center Technical Reviews/Audits for Aeronautical

Weapon System Acquisition (19 Jul 2006)

5. DoD 5000.4M Cost Analysis Guidance and Procedures (Dec 1992)

6. DoD Defense Acquisition Guidebook (14 October 2004)

7. DoD-D-1000B - Drawings, Engineering And Associated Lists (28 October 1977)

8. DoD Directive 5000.1 “The Defense Acquisition System,” (13 May 2003)

9. DoDI 5000.02 “Operation of the Defense Acquisition System” (08 Dec 2008)

10. DoD MIL-STD 498 Software Development & Documentation (Dec 1994)

11. DoDAF v 1.0 DoD Architecture Framework (AF) Volumes 1 and 2 (9 Feb 2004)

12. DoDD 4630.5 “Interoperability and Supportability of Information Technology (IT) and National

Security Systems (NSS)” (23 Apr 2007)

13. DoD MIL-STD-1833 Test Requirements For Ground Equipment And Associated Computer Software

Supporting Space Vehicles (13 Nov 1989)

14. DoD Risk Management Guide for Acquisition, Sixth Edition, v 1.0 (Aug 2006)

15. DoD Systems Engineering Plan (SEP) Preparation Guide Version 1.02 (10 February 2006)

16. DoD Technology Readiness Assessment (TRA) Deskbook – DUSD(S&T) (May 2005)

17. IMP & IMS Preparation and Use Guide Version 0.9 (21 October 2005)

18. ISO/IEC STD 15939 Software engineering - Software measurement process (11 Jul 2002)

19. MIL-STD-882D DoD Standard Practice for System Safety (10 Feb 2000)

20. MIL-STD-961E Defense and Program-Unique Specifications Format and Content (Aug 2003)

21. MIL-STD-963B Data Item Descriptions (DIDs) Practice (31 August 1997)

22. MIL-STD-1472F DoD Design Criteria STD for Human Engineering (23 August 1999)

23. MIL-STD-1521B (USAF) Military Standard, Technical Reviews and Audits for Systems, Equipments, and Computer Software (4 Jun 1985) including Notices 1 and 2

24. MIL-DTL-31000C Detail Specification Technical Data Packages (9 Jul 2004)

25. SMC Systems Engineering Primer and Handbook, 2nd Edition (29 Apr 2005)

26. SMCS-S-002 Configuration Management (13 June 2008)

27. System and Software Reviews Since Acquisition Reform, Southern California SPIN (2 Apr 2004)

28. Systems Engineering Plan (SEP) Preparation Guide, v 1.02 (12 Feb 2006)

29. TOR-2004(3909)-3537 Rev. B Software Development Standard for Space Systems

30. TOR-2006(8506)-5749, Mission Assurance Tasks for Software (30 Apr 2007)

3. Definitions of Technical Reviews and Audits The terms defined in this section are included to augment the understanding of the dialog and discourse contained in this document, by providing context elaboration and clarification and consistency.

3.1 System Requirements Review (SRR)

The SRR is a multifunctional technical review or a series of reviews that shall be conducted to ensure that all system and performance requirements are derived from the Initial Capabilities Document (ICD) or draft Capability Development Document (CDD) and are defined and consistent with cost (program budget), schedule (program schedule), risk, and other system constraints.

The SRR assesses the system requirements captured in the system specification and ensures the consistency between the system requirements and the preferred system solution and available technologies.

The SRR is typically held well in advance of Milestone B to allow time for issue resolution and proper executive-level concurrence on process and results. The SRR can convene prior to program initiation or during technology development and is typically convened during system development and demonstration when a significant portion of the system functional requirements has been established.

3.2 System Functional Review (SFR)

The SFR is a multidisciplinary technical review or a series of reviews that shall be conducted, to ensure that the system under review can proceed into preliminary design, and that all system requirements and functional performance requirements derived from the Capability Development Document are defined and are consistent with cost (program budget), schedule (program schedule), risk, and other system constraints. Generally this review assesses the system functional requirements as captured in system specifications (functional baseline), and ensures that all required system performance is fully decomposed and defined in the functional baseline. System performance may be decomposed and traced to lower-level subsystem functionality that may define hardware and software requirements. The SFR determines whether the system’s functional definition is fully decomposed to a low level, and whether the Integrated Product Team (IPT) is prepared to start preliminary design.

Completion of the SFR shall provide:

1. An established system functional baseline

2. An updated risk assessment for the System Development and Demonstration phase

3. Current data to update the Cost Analysis Requirements Description (CARD) document, based on the contractor’s proposed system functional baseline

4. An updated program development schedule including system and software critical path drivers

5. An approved SWCI with updates applicable to this phase

The SFR determines whether the system’s lower-level performance requirements are fully defined and consistent with the mature system concept, and whether lower-level systems requirements trace to top-level system performance and the Capability Development Document. A successful SFR is predicated upon the IPTs determination that the system performance requirements, lower-level performance requirements, and plans for design and development form a satisfactory basis for proceeding into preliminary design.

The program manager shall tailor the review to the technical scope and risk of the system, and address the SFR in the Systems Engineering Plan. The SFR is the last review that ensures that the system is credible and feasible before more technical design work commences.

Typical SFR success criteria include affirmative answers to the following exit questions:

1. Can the system functional requirements, as disclosed, satisfy the Capability Development Document?

2. Are the system functional requirements sufficiently detailed and understood to enable system design to proceed?

3. Are adequate processes and metrics in place for the program to succeed?

4. Are the risks known and manageable for development?

5. Is the program schedule executable (technical and cost risks)?

6. Is the program properly staffed?

7. Is the program with the approved functional baseline executable within the existing budget?

8. Is the updated Cost Analysis Requirements Description consistent with the approved functional baseline?

9. Does the updated cost estimate fit within the existing budget?

10. Has the system Functional Baseline been established to enable preliminary design to proceed with proper configuration management?

11. Is the software functionality in the approved functional baseline consistent with the updated software metrics and resource loaded schedule?

3.3 Software Requirement and Architecture Review (SAR)

The Software Requirements and Architecture Review (SAR) consists of a series of multidisciplinary reviews of the software requirements, architecture, and test planning of technical products, software development processes, and the current state of the software development for all software items. SAR is a review of the finalized Software Item (SI) requirements and operational concept. The SAR shall be conducted when SI requirements have been sufficiently defined to evaluate the contractor’s responsiveness to and interpretation of the system, subsystem, or prime item-level requirements. A successful SAR is predicated upon the contracting agency’s determination that the Software Requirements Specification(s), Interface Requirements Specification(s), and Software Test Plan Operational Concept Document form a satisfactory basis for proceeding to preliminary software design cycle. See Appendix C for additional SAR information. Note: Software Configuration Item (SWCI) and software item (SI) have the same meaning and are used interchangeably throughout the technical community.

3.4 Preliminary Design Review (PDR)

The PDR is the formal culmination of a series of multidisciplinary PDRs of the system and of a series of reviews of individual Configuration Items (CIs) that shall be conducted to ensure that the system under review can proceed into detailed design and can meet the stated performance requirements within cost (program budget), schedule (program schedule), risk, and other system constraints.

Typical PDR outcomes include:

1. The assessment that the system preliminary design as captured in performance specifications for each configuration item in the system, the allocated baseline (ABL), and ensures that each function in the functional baseline (FBL) has been allocated to one or more of the configuration items. Configuration items shall consist of hardware and software elements and include such items as airframes, avionics, weapons, crew systems, engines, trainers and training, etc.

2. The determination of the compatibility of the CIs with performance and engineering specialty requirements of the Hardware Configuration Item (HWCI) development specification

3. The evaluation that the degree of definition and assess the technical risk associated with the selected manufacturing methods and processes

4. The establishment of the existence and compatibility of the physical and functional interfaces among the configuration item and other items of equipment, facilities, computer software, and personnel

5. The measurement of the progress, consistency, and technical adequacy of the selected top-level SWCI designs and test approaches

6. The assessment of the compatibility between software requirements and SWCI(s) preliminary design

7. The PDR’s evaluation of the preliminary version of the SWCI(s) operation and support documents

3.5 Critical Design Review (CDR)

The CDR is a multidisciplinary technical review of the system and of a series of CDRs of individual CIs that shall be conducted for each configuration item when detailed design is essentially complete, with the objective to ensure that the detailed design of each individual configuration item that is an integral part of the system under review can proceed into fabrication, system integration, demonstration, and test, and can meet the stated performance and engineering specialty requirements of the configuration item development specifications within cost (program budget), schedule (program schedule), risk, and other system constraints.

1. The CDR establishes the detailed design compatibility among the configuration item and other items of equipment, facilities, computer software and personnel

2. The CDR assesses configuration item risk areas (on a technical, cost, and schedule basis)

3. The CDR assesses the results of the producibility analyses

4. The CDR focuses on the determination of the acceptability of the detailed design, performance, and test characteristics of the design solution, and on the adequacy of the operation and support documents

5. The CDR assesses the system final design as captured in product specifications for each configuration item in the system (product baseline), and ensures that each product in the product baseline has been captured in the detailed design documentation, e.g., product specifications for:

a. Hardware, to enable the fabrication of configuration items, and include production drawings

b. Software, (e.g., software architecture and detailed design documents) of one or more

Software Configuration Item (SWCI), to the extent specified in the Software Development Plan (SDP) based on the selected life cycle model(s)

6. For complex systems, the contractor may conduct a CDR for each subsystem or configuration item. These individual reviews would lead to an overall system CDR. When individual reviews have been conducted, the emphasis of the overall system CDR shall focus on configuration item functional and physical interface design, as well as overall system detailed design requirements

7. The System CDR determines whether the hardware, human, and software final detailed designs are complete to the extent specified in the SDP based on the selected life cycle model(s), and whether the contractor is prepared to start system fabrication, demonstration, and test

3.6 Test Readiness Review (TRR)

The Test Readiness Review (TRR) is a multifunctional and multidisciplinary review and process assessment that shall be conducted, to verify the contractor’s readiness to begin a formal verification testing event for an end item (EI). It is conducted for each EI to determine whether the test procedures are complete and to assure that the contractor is prepared for formal EI testing. Test procedures are evaluated for compliance with respective test plans and descriptions, and for adequacy in accomplishing test requirements. At TRR, the contracting agency also reviews the results of development testing and any updates to the operation and support documents. A successful TRR is predicated on the contracting agency’s determination that the test procedures, environment, and previous test results form a satisfactory basis for proceeding to formal EI testing.

3.7 Functional Configuration Audit (FCA)

The Functional Configuration Audit (FCA) is a verification process that shall be convened periodically to ascertain that the actual performance of a CI meets the requirements stated in its performance specification and to certify that the CI has met those requirements.

1. The FCA is convened to verify that the actual performance of the system meets the requirements stated in the system performance specification. For very large, complex CIs/systems, the audits can be accomplished in increments. Each increment shall address a specific functional area of the CI/system and document any discrepancies that are found in the performance capabilities of that increment

2. The FCA also reviews the completed operation and support documents

3. A formal FCA is performed to validate that the development of a configuration item has been completed satisfactorily and that the configuration item has achieved the performance and functional characteristics specified in the functional or allocated configuration identification

4. A summary FCA can be convened to address the status of all of the action items that have been identified by the incremental FCAs, in order to document and certify their completion

3.8 Physical Configuration Audit (PCA)

The PCA is a technical examination of a designated configuration item that shall be performed, to verify that the configuration item “As Built” conforms to the technical documentation that defines the actual configuration of an item being produced.

1. The PCA verifies that the design documentation matches the EI as specified in the contract

2. The PCA confirms that the manufacturing processes, quality control system, measurement and test equipment, and training are adequately planned, tracked, and controlled. The PCA also validates many of the supporting processes used by the contractor in the production of the item, and verifies other elements of the item that may have been impacted or redesigned after completion of the System Verification Review (SVR)

3. The PCA is convened prior to the full-rate production decision or when the Government plans to control the detailed design of the item it is acquiring via the Technical Data Package.

When the Government does not plan to exercise such control or purchase the item’s Technical Data Package (e.g., performance-based procurement), the contractor shall conduct an internal PCA to define the starting point for controlling the detailed design of the item and establishing a product baseline

4. The PCA is considered complete when the design and manufacturing documentation matches the item as specified in the contract

3.9 System Verification Review (SVR)

The SVR is a test, inspection, or analytical process by which a group of configuration items that make up the system shall be verified to have met specific contracting agency contractual performance requirements (specifications or equivalent). This review does not apply to hardware or software requirements verified at FCA for the individual configuration item.

3.10 Manufacturing Readiness Review (MRR)

MRRs shall be conducted by the prime contractor, to ensure readiness to build a quality product that inherently embodies defense-unique and/or defense-critical manufacturing capabilities characteristic of a desired defense contractor, as appropriate for the program under review, before commencing manufacture of a unit or other contractually designated configuration items by the prime contractor, subcontractor, or critical component supplier.

3.11 Production Readiness Review (PRR)

PRRs shall be conducted to assist the Government in the evaluation of the production risks and the contractor’s methodology to manage those risks and the determination that the contractor, their sub-contractors, and suppliers have resolved to the satisfaction of the Government the PRRs’ findings and specific actions prior to executing a production go-ahead decision.

1. PRRs are accomplished in an incremental fashion during the Full-Scale Development phase (usually two initial reviews and one final review) to assess the risk in exercising the production go-ahead decision. In its earlier stages the PRR concerns itself with gross-level manufacturing concerns such as the need for identifying high-risk and low-yield manufacturing processes or materials or the requirement for manufacturing development effort to satisfy design requirements. The reviews become more refined as the design matures, dealing with such concerns as adequate production planning, facilities allocation, incorporation of producibility-oriented changes, identification and fabrication of tools and test equipment, long-lead item acquisition, etc.

2. The PRR examines risk; it determines if production or production preparations incur unacceptable risks that might breach thresholds of schedule, performance, cost, or other established criteria

3. The PRR evaluates the full, production-configured system to determine if it correctly and completely implements all system requirements

4. The PRR determines whether the traceability of final system requirements to the final production system is maintained

5. Timing of the incremental PRRs is a function of program posture and is not specifically locked in to other reviews

4. Technical Review Objectives Technical reviews are necessary systems engineering activities designed to assess progress within a project relative to its planned technical and/or contractual objectives. These reviews:

1. Serve as a technical review guide for the contracting agency and Government (also interchangeably referred to as “contracting agent”) from program definition through EI development

2. Assure that technical reviews are conducted at logical transition points in the development effort to assess program health and readiness to proceed to subsequent phases with acceptable risk

3. Assure that technical reviews provide a method for program teams to determine progress achieved to date, assess predicted or actual performance against the requirements, identify potential problems and risks, and determine the need for mitigation plans to avert schedule delays and cost growth to the program

4. Assure that technical reviews facilitate contractor and contracting agent concurrence on technical progress, and help ensure a successful review against a mutually agreed-to set of criteria

5. Assure that the technical reviews criteria defined for each review are based on candidate criteria identified in Section 6

6. Assure that, during each review, all criteria elements will be reviewed against the program technical requirements and their associated impacts to program cost and schedule

7. Assure that the components of this document are tailored by the contracting agent in context of the acquisition agent’s program objectives and resources to execute the end item development effort and take into account to the maximum extent possible the existing capabilities and resources that are already in place at the contractor’s and subcontractor’s facilities to support these reviews

These objectives are applicable at all levels of the enterprise (e.g., prime contractor, subcontractor, and vendor levels). Government customer involvement in reviews and audits shall vary according to the needs of the specific program.

4.1 Reviews Specified

The following reviews and audits shall be conducted:

SRR - Appendix A System Requirements Review SFR - Appendix B System Functional Review SAR - Appendix C Software Requirements and Architectures Review PDR - Appendix D Preliminary Design Review CDR - Appendix E Critical Design Review TRR - Appendix F Test Readiness Review FCA - Appendix G Functional Configuration Audit PCA - Appendix H Physical Configuration Audit SVR - Appendix I System Verification Review M/PRR - Appendix J Manufacturing and Production Readiness Review

4.2 Technical Review Criteria

Technical Review Criteria (TRC) for all reviews shall be composed of all items necessary to illustrate that the program is technically progressing according to plan and ready to move into the next phase or event.

Figure 1 illustrates the generic flow of the review process to be conducted as a joint activity between the Contractor officer’s representative and the Contractor, Subcontractor, and EI developer.

The list of TRCs that the contractor needs to provide at technical reviews shall include but not be limited to:

1. A list of measurable metrics and accomplishments that demonstrate contractor-planned progress and developmental maturity for the review conducted

2. Risk assessments for all items or issues that are on a critical path, e.g.:

a. No high-risk items that would prevent a “move forward” decision

b. Any high-risk items to be presented must be accompanied by an appropriate mitigation plan

3. Data that demonstrates performance that is accumulated and ready for review

4. Key decisions that are complete, executable, and fully supportable

5. Other items as specified by the contracting agent or as tailored by contract, prior to and specifically for each review or audit event

6. Elements that correspond with relevant events or items as contractually defined, incorporated, or expanded, i.e.:

a. The program Integrated Master Plan (IMP) and Integrated Master Schedule (IMS)

b. The Statement of Objectives or Statement of Work (SOW)

c. The Requirements Allocation Document (RAD)

7. Action items from previous technical reviews are completed and closed The TRC list is prepared to ensure that the reviews culminate in a determination of readiness, or a direct plan for achieving readiness to move forward. As such, qualification for entrance into the review implies that all criteria elements have been performed or adequately addressed for a given review milestone and are fully ready for presentation and assessment.

4.3 Technical Review Success

Technical review success is achieved when all criteria elements have been demonstrated by the contractor to the satisfaction of the contracting agency, against a mutually agreed-to set of “Acceptance Criteria” by:

1. Concluding and verifying that the end state of a criteria item has been accomplished

2. Correlating each end state to a specific requirement(s)

3. Defining the basis of measurement or metrics used to substantiate and confirm that the end state of the criteria item has been adequately met “Acceptance Criteria” are objective evidence of accomplishment, and when realized, they provide sufficient evidence to the contracting agency’s satisfaction and acceptance that demonstrates that the design solution selected and the completion criteria elements assigned by the contractor to each design review milestone:

1. Are traceable to all technical requirements

2. Correspond to the system architecture

3. Can be synthesized into an EI design that is realistic and physically producible

4. Can be implemented to satisfy the user’s operational needs for a reasonable cost and acceptable risk

5. Have review criteria for the assessment of technical progress that:

a. Are appropriately tailored to reflect pertinent acquisition strategy

b. Are appropriately tailored to reflect major risk drivers peculiar or unique for each program

c. Allow for the preparation of documented evidence of progress

d. Are sufficient to allow for an informed judgment of readiness and contracting agency’s approval to proceed to program execution for the next level of design maturity

General guidance for the “Acceptance Criteria”, generic to any developmental DoD program, is provided in Appendices A through E specific to SRR, SFR, SAR, PDR, and CDR.

http://assist.daps.dla.mil/quicksearch/

Figure 1. Generic Technical Review Process Map.

PRIME / SUB CONTRACTOR(s) GOV’T & SUPPORT ORGs

Start

No

End

Yes

T

IM

E

So ur ce G ui da nc e:

A

FD

-0

A ss ur ed A cc es s T o

Sp ac e, V ol . 3

N o.

_N ov

D oD D

0.

an d

D oD

I

.0

O pe ra ti on o f t he D ef en se A cq ui sit io n Sy st em

A F

I

-1

_O pe ra ti on B as ed

C ap ab ili tie s

In te gr at io n an d

D ev el op m en t S ys te m

R is k M an ag em en t G ui de fo r D

O D

A cq ui si tio n, S ix th E di ti on , V

.0

(A ug

6)

Pr ov id e

T ec hn ic al R ev ie w S up po rt a s R eq ui re d

Provide Technical Review Instructions, Standards, Guidance

Establish Contractual Rqm’ts to Perform Technical Reviews

Identify SPO, Contractors, and Other Stakeholders and Their Responsibilities to Support Review

Establish Participant Commitments to Perform Review

Perform Review Readiness Assessment

Conduct Technical Review

Document Action Items and Closure Plans

Identify Technical Risks and Deficiencies

Conduct Follow-Up Review

Resolve Issues and Disconnects

Is The Contractor

Ready?

Prepare and Document Plans for Next Review

Accept Review Result s?

Post-Award

Identify Review Criteria

Number, frequency and types of the reviews required are negotiated and tailored by the procuring agency

Joint Effort Legend

Yes

No

5. Roles, Responsibilities, and Authority This section describes the typical roles, responsibilities, and decision authority relationships between the contractor(s) and the contracting agency.

Technical review planning and execution responsibility are shared equally by the contracting agency or the Procuring Contracting Officer (PCO), and the contractor or the contractor’s program manager.

5.1 Contractor Participation and Responsibilities

The contractor shall be responsible for conducting the Technical Reviews and Audits in accordance with the following requirements except as amended by the contract.

5.1.1 Subcontractors and Suppliers

The contractor shall be responsible for ensuring that subcontractors, vendors, and suppliers participate in formal reviews and/or audits, as appropriate.

5.1.2 Chairperson

The chairperson for the technical review shall be the contractor’s program manager or his and her designee. The chairperson is responsible for publishing the meeting agenda, recording and publishing the meeting minutes, maintaining a list of attendees, meeting location selection, and coordination, and all administrative items associated with each technical review.

5.1.3 The Contractor Requirements

The contractor shall be responsible for conducting the Technical Reviews and Audits in accordance with the following requirements.

The contractors shall be responsible for establishing the time, place, and agenda for each Review and Audit in consonance with the master milestone schedule, subject to coordination with the contracting agency.

This shall be accomplished sufficiently, well in advance of each review and/or audit to allow adequate preparation for the meeting by both the contractor and the contracting agency.

5.1.3.1 Review, Audit Schedules, and Available Information

The contractors shall ensure that each Review and Audit schedule is compatible with the availability of the necessary information and contract articles, e.g., system engineering data, trade study results, producibility analysis results, risk analysis results, specifications, manuals, drawings, reports, hardware, software, or mock-ups.

5.1.3.2 Review and Audit Preparation

The contractors shall prepare for each Review and Audit in sufficient detail consistent with the scope and magnitude of the Review and Audit.

5.1.3.3 Review and Audit Conduct

The contractors shall designate a co-chairperson for each Review and Audit. Participating contractor and subcontractor personnel or those chosen to make presentations shall be prepared to discuss in technical detail any of the presented material within the scope of the review.

Specifically, the contractor’s co-chairperson or his and her designee shall be responsible for the following:

1. The planning, organization, coordination, and delivery of each review

2. The preparation and approval of the list of individuals participating, representing, and acting on behalf of the contractor at the reviews

3. The preparation of “Acceptance Criteria” to be used for review and approval by the PCO

a. The contractor program manager must assure that the recommended and mutually agreed-to “Acceptance Criteria” can be demonstrated to initiate the review

b. The contractor shall prepare for each review, supporting data, in sufficient detail as objective evidence of accomplishments, consistent with the scope and magnitude of the review per the mutually agreed-to “Acceptance Criteria” under the following five major categories:

(1) Systems Engineering and Architecture Development

(2) System, Segment, and Subsystem Design

(3) System Verification and Validation

(4) Engineering Disciplines and Specialty Engineering

(5)…