TSA IV Section L Attachment 3 - Simulators Division OSEP.pdf

AIR FORCE LIFE CYCLE MANAGEMENT CENTER

(AFLCMC)

Air Force Program Executive Officer, Agile Combat Support

(AFPEO/ACS)

SIMULATORS PROGRAM OFFICE

ORGANIZATIONAL SYSTEMS ENGINEERING PLAN (OSEP)

VERSION 3.0 –2017

CONCURRENCE

Title Signature/Name Office Symbol

Anthony J DalSasso Chief Engineer Simulators Program Office

AFPEO/ACS

AFLCMC/WNE

Philip E. Carpenter, Col, USAF Chief, Simulators Program Office

AFLCMC/WNS

Jacqueline S. Janning-Lask Director of Engineering

AFLCMC/WNE

APPROVAL

LYNDA T. RUTLEDGE, SES, DAF

Air Force Program Executive Officer Agile Combat Support

AFLCMC/WN

Table of Contents

1.0 Introduction – Purpose and Update Plan

2.0 Program Technical Requirements

2.1 Architectures and Interface Control

2.2 Technical Certifications

Simulator Certification (SIMCERT) Programmatic Environmental, Safety, & Health Evaluation (PESHE) Operational Safety, Suitability, and Effectiveness (OSS&E)

Weapon System OSS&E Support Cybersecurity Certification and Accreditation

3.0 Engineering Resources and Management

3.1 Technical Schedule and Schedule Risk Assessment

3.2 Engineering Resources and Cost/Schedule Reporting

3.3 Technical Risk and Opportunity Management

3.4 Technical Organization

Government Program Office Organization Program Technical Staffing Levels Contractor(s) Program Organization Engineering Team Organization and Staffing

3.5 Relationships with External Technical Organizations

3.6 Technical Performance Measures and Metrics

4.0 Technical Activities and Products

4.1 System Engineering Results for the Previous Acquisition Phase

4.2 System Engineering Activities for the Current Acquisition Phase

4.3 System Engineering Activities for the Next Acquisition Phase

4.4 Requirements Development and Change Process

Analysis and Decomposition Requirements Management and Change Process

4.5 Technical Reviews

Program Technical Reviews Technical Status Report and Reviews Request for Proposal (RFP) Technical Review

4.6 Configuration and Change Management Process

Change Request for Proposal (RFP) Review Board

4.7 Design Considerations

4.8 Human System Integration

4.9 Engineering Tools

5.0 Annex A – Acronyms

6.0 Annex B – OSEP Attachment Numbering

List of Tables and Figures in this OSEP Tables Table 1.0-1 OSEP Update Record Table 2.2.3-1 OSS&E Status Reporting Table B-1.1 Attachment Numbering Formats

Table B.1-2 OSEP Attachments by Number

Figures Figure 3.4.1-1 Simulators Program Office Organization Chart Figure 4.0-1 Training System Acquisition Phases Figure 4.5-1 Change Proposal Flow Diagram

List of Tables and Figures Required in Program SEP Attachments Tables Table 2.1-1 Required Memoranda of Agreement Table 2.2-1 Certification Requirements Table 3.4.4-2 Integrated Product Team Details Table 3.6-2 Technical Performance Measures and Metrics Table 4.4-1-n Technical Review Details Table 4.6-1 Design Considerations Table 4.6-2 Reliability and Maintainability Activity Planning and Timing Table 4.7-1 Engineering Tools

Figures Figure 3.1-1 System Technical Schedule Figure 3.3-1 Technical Risk Cube Figure 3.4.1-1 Program Office Organization Figure 3.4.2-1 Program Technical Staffing Figure 3.4.3-1 Contractor Program Office Organization Figure 3.4.3-2 Contractor Technical Staffing Figure 3.4.4-1 Integrated Product Team /Working Group Team Hierarchy Figure 3.6-1 Reliability Growth Curve Figure 4.3.1-1 Requirements Decomposition/Specification Tree/Baselines Figure 4.5-1 Configuration Management Process

1.0 Introduction – Purpose and Update Plan

a. Use – This Simulators Program Office OSEP establishes the technical planning document and blueprint for the conduct, management, and control of the technical aspects of all programs managed by the Simulators Program Office. Each program managed by the Simulators Program Office will follow this OSEP, with the exception of services acquisition (i.e. Contractor Training Support Services contracts) which are exempt from SEP requirements per Air Force (AF) Policy Directive 63-1 paragraph

3.16. Programs will prepare and maintain an attachment that includes program-unique information required by this document, including deviations from the AFLCMC Standard and Internal Process Guides. This OSEP, including the applicable Program Systems Engineering Plan (PSEP) Attachments, will be a “living, go to” Systems Engineering (SE) planning document for programs from concept exploration to disposal.

Documentation of SE processes will begin at program inception. The purpose is to work through the variety of SE issues that confront an Integrated Product Team (IPT) throughout the program lifecycle. Early attention to the SE process should help the IPT determine Request for Proposal (RFP) content, risk management tools, acquisition strategies, manning, and contractor requirements.

b. Program Description – Each PSEP Attachment will include a top level description of the program, including the program phase, contract type and current contractual scope, products, services, and training capabilities (mission). The AFPEO for the program and the using organizations will also be identified.

c. Alignment with Organizational Processes – This OSEP shall follow Standard and Internal Processes as required for various aspects of SE planning and execution. These processes are maintained by AFLCMC to be consistent with Department of Defense (DoD), AF, Command, and Center policies and guidance.

d. Alignment with Prime Contractor’s SE Management Plan (SEMP) – Some

Simulators Program Office programs require contractors to establish and execute program activities in accordance with a contractor-developed SEMP. Each PSEP Attachment to this OSEP will identify the contractor’s SEMP, if one exists (and is contracted to be delivered to the program), and identify/describe any known process inconsistencies, including deviations from the organizational process documents referenced herein. These inconsistencies/deviations will be documented in the appropriate PSEP Attachment sections where these documents are referenced below, and associated risk types will be addressed within the program’s Risk Management Plan.

e. Update Criteria – This OSEP and its PSEP Attachments will be maintained current through a review and update process in accordance with the following criteria:

1. Event Triggers:

• Regulation/Policy and External Document Updates – The Simulators Program Office Chief Engineer will determine if updates to SE regulations, policy, or external documents referenced herein are significant enough to warrant an out-of-cycle update to this OSEP. Upon such determination, this OSEP will be reviewed, and updated as necessary, including impacted PSEP Attachments.

• Defense Acquisition Management System Milestones – Applicable PSEP Attachments will be reviewed prior to each Simulators Program Office milestone review (defined in DoD Instruction (DoDI) 5000.02), and updated if necessary to support evolving program lifecycle requirements.

• Internal Program SE Events – The PSEP Attachments will be reviewed, and updated as necessary, in conjunction with program SE events (e.g. SEMP updates, requirements reviews, design reviews, and audits).

2. Age Triggers:

• OSEP document and PSEP Attachments – The Simulators Program Office Chief Engineer will perform an annual review of the OSEP document and referenced organizational process documents to determine if any update to the OSEP document is required. This determination will consider any recent updates made due to internal program event-driven reviews/updates. Also, the PSEP Attachments will be reviewed by the Program’s Lead Engineer (LE).

f. Approval authorities – Updates to this document and its PSEP Attachments will be approved by the appropriate authority as established below.

1. OSEP: Per AF Instruction (AFI) 63-101, Attachment 1 and AFLCMC Internal

Process Guide for System Engineering Plans, the Simulators Program Office OSEP and subsequent major updates to it (e.g. 2.0, 3.0, etc.) will be approved by AFPEO/ACS Director of Engineering. Minor updates (e.g. 1.1, 1.2, etc.) will be approved by the Simulators Program Office Chief Engineer. Each update to the OSEP will be logged in Table 1.0-1 below. Each OSEP review and update action will be documented in a signed memorandum entered in the Simulators Program Office official files.

Table 1.0-1 OSEP Update Record

Revision Number Date Change Description Approved By

1.0 10 Jan 13 Baseline Release AFPEO/ACS

2.0 19 Aug 14 Annual update, and added “4.7 Human

Systems Integration” section AFLCMC/WNSE

3.0 Annual update, incorporated engineering

practices (formerly annexes), Operating Instructions, and SEP numbering schema (Annex B)

2. PSEP Attachments (New PSEPs): Each program managed by the Simulators

Program Office will produce a PSEP Attachment to the OSEP, formatted the same as the OSEP, documenting program-specific processes and details. The AFPEO/ACS Director of Engineering will approve this PSEP Attachment for any http://www.dtic.mil/whs/directives/corres/pdf/500002p.pdf http://www.af.mil/shared/media/epubs/AFI63-101.pdf https://cs4.eis.afmc.af.mil/sites/1534/APD/New%20Library%204%20Guides/System%20Engineering%20Plans.docx https://cs4.eis.afmc.af.mil/sites/1534/APD/New%20Library%204%20Guides/System%20Engineering%20Plans.docx program over $50M. The Simulators Program Office Chief Engineer will approve the PSEP Attachment for any program valued between $8M and $50M. For programs less than $8M, the Simulators Program Office Section Chief Systems Engineer (CSE) will approve the PSEP Attachment.

3. PSEP Attachments (Legacy SEPs): When a Simulator program has an existing, stand-alone SEP written using an older format, this SEP will be attached to the OSEP without reformatting. This SEP will remain in its existing format as long as the program is past Milestone C and there are no major updates to the program. If the system incurs modifications valued at under $8M, the PSEP Attachment will be updated and re-approved as required. If the system incurs a modification valued at $8M or greater, changes associated with the modification will be documented in the latest SEP format. The existing SEP for that program will not be updated.

Together, the original old-format SEP and the new-format Modification SEP will serve as that PSEP Attachment to the OSEP.

For modifications above $50M, the AFPEO/ACS Director of Engineering will approve the Modification SEP. Between $8M and $50M, the Simulators Program Office Chief Engineer will approve it. For modifications below $8M, the Simulators Program Office Section CSE will approve it, as required.

4. Additional Coordination: The Simulators Program Office Chief Engineer will determine the need for any additional coordination action for PSEP Attachments on a case-by-case basis. Each PSEP Attachment review and update action will be documented in a signed memorandum entered in the official program files. Updates to external documents incorporated into the PSEP Attachment by reference will not require approval.

2.0 Program Technical Requirements

2.1 Architectures and Interface Control

The purpose of the DoD Architecture Framework (DoDAF) is to support its core processes by facilitating the understanding of a system’s scope and complexities from different “viewpoints” using pictorial and textual architecture documentation products. The majority of Simulators Program Office training systems were implemented as stand-alone devices, with no interfaces to external systems. When this is the case, a top-level system functional architecture diagram, system physical architecture diagram, and system software architecture diagram will be included in the PSEP Attachment to convey internal architectural relationships. For programs with external interfaces to other systems (e.g. Distributed Mission Operations (DMO) programs) additional architecture products will be identified in the PSEP Attachment. Additionally, each PSEP Attachment will identify in tabular format:

a. Requirements documents defining the interfaces with external systems

b. Inter-agency activities and interface control/accreditation authorities

c. Any Memoranda of Agreement (MOA) pertaining to these interfaces that must be established and/or maintained

2.2 Technical Certifications

For training systems, several certification requirements are typically established. The following outline those typically applicable.

See the PSEP Attachments for a listing of all technical certifications.

Simulator Certification (SIMCERT) Reference: AFI 36-2251, Management of Air Force Training Systems, 5 June 2009

AFI 36-2251 (and any Major Command (MAJCOM) supplements thereto) is the source of the SIMCERT requirement. As stated therein, “The… reports generated by the SIMCERT program support the accreditation authority’s decision to accredit the associated training system for its intended use.” Each MAJCOM documents its SIMCERT program in a Master SIMCERT Plan, which will be identified in the certification table in the PSEP Attachment along with planned and completed SIMCERT activities.

Programmatic Environmental, Safety, & Health Evaluation (PESHE)

The Simulators Program Office has an Organizational PESHE that was coordinated through the Environmental Office (AFLCMC/EZ) and signed by the AFPEO/ACS. Many of the programs in the Simulators Program Office have tailored attachments to the PESHE which describes any possible safety, environmental, or hazards associated with their programs.

Operational Safety, Suitability, and Effectiveness (OSS&E) References:

a. AF Materiel Command Instruction 63-1201, Implementing Operational Safety, Suitability & Effectiveness (OSS&E) and Life Cycle SE(LCSE), 14 October 2009.

Incorporating through Change 2, 26 July 2012

b. AFI 63-101/20-101, Integrated Life Cycle Management, 7 March 2013. Incorporating through Change 2, 23 February 2015.

OSS&E is an AF requirement that assures weapon systems are operationally safe, are suitable for the environment in which they will be operating, and effectively meet the mission for which they are designed. While by definition training programs are not “operational” in nature, their effectiveness does feed into that of the overall weapon system, and so training should be considered in weapon system OSS&E activities. Foreign Miltary Sales (FMS) programs are not required to comply with OSS&E certification requirements, but are encouraged to follow this engineering practice.

Each program, with the exception of FMS programs, shall develop a signed OSS&E Baseline Document and maintain it in accordance with robust configuration control practices. A template https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/Engineering%20Resources/Operational%20Suitability,%20Safety%20and%20Effectiveness/PESHE/WNS%20PESHE%20signed.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFMCI63-1201.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFMCI63-1201.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFI63-101.pdf https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/pages/Engineering%20Resources.aspx https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/pages/Engineering%20Resources.aspx and a sample are available in Simulators Program Office Engineering SharePoint site at AFLCMC_EN-EZ OSS&E Guidance.

Each program, with the exception of FMS programs, shall also establish a set of metrics that provide insight into OSS&E status of the Simulators Program Office managed training component of the weapon system. Each program’s LE shall log and report the status of OSS&E related metrics. The status shall include the five items in Table 2.2.3-1, and will be maintained in the Simulators Program Office Technical Status Reports (see paragraph 4.5.2)

Table 2.2.3-1 OSS&E Status Reporting

SAFETY

Weapon System Operational Safety

Aircraft incident during the reporting period for which the training system has been identified by the Safety Investigation Board (SIB) as either a causal or contributing factor?

GREEN - No incidents or SIB findings relevant to training system.

YELLOW - SIB findings relevant to training that are in work but not yet deployed RED - SIB findings that are training relevant with no corrective action identified.

Training System Operational Safety

Occurrence of a safety incident during the course of training system development, installation, or operations during the reporting period?

GREEN – No incidents or all safety issues resolved YELLOW – Safety incident reported, corrective action pending RED – Safety incident with no corrective action

SUITABILITY

System Availability Met contracted availability/student throughput rate for the reporting period?

GREEN – met or exceeded requirements YELLOW – did not meet requirements. Corrective action in work or isolated incident.

RED – did not meet requirements. No action being taken, due to contractual/funding issues or other reason.

EFFECTIVENESS

SIMCERT Passed SIMCERT or equivalent operational evaluation in last 2 years?

GREEN – SIMCERT completed within last 2 years; no significant findings or all corrective actions completed YELLOW – SIMCERT not completed in greater than 2 years or was completed with findings and corrective action in work.

RED – Simulator decertified due to SIMCERT findings; significant SIMCERT findings with no corrective action plans in place

Concurrency Does the training system match the weapon system sufficiently for realistic training?

GREEN – Training system matches the configuration of the weapon system to the extent necessary to support required training https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/Engineering%20Resources/Forms/AllItems.aspx?RootFolder=%2fsites%2f77AESW%2f677AESG%2fSYE%2fEngineering%20Resources%2fOperational%20Suitability%2c%20Safety%20and%20Effectiveness%2fAFLCMC%5fEN%2dEZ%20OSSE%20Guidance&FolderCTID=0x012000B063B7F69E6172488E801381AA099615&View=%7bE23D7DAF%2d2152%2d4CD7%2d9705%2d350311D609F9%7d https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/Lists/Technical%20Status%20Reports/AllItems.aspx

YELLOW – Training system configuration does not match that of the weapon system, but funding is available and changes are pending – OR – using command has consciously decided to defer corrective actions indefinitely.

RED – Training system does not match the weapon system configuration to the extent required to support required training, but no corrective actions are pending, due to lack of funding or other reason.

Weapon System OSS&E Support

Each program shall make an effort to establish an agreement with the weapon system program office on the role of the Simulators Program Office in supporting OSS&E of the weapon system.

This agreement will be documented in the form of a MOA/MOU, Service Level Agreement, or inclusion in the weapon system OSS&E Implementation Plan.

Cybersecurity Certification and Accreditation

References:

a. DoD Directive (DoDD) 5000.01, The Defense Acquisition System, Certified Current as of 20 November 2007

b. DoDI 5000.02, Operation of the Defense Acquisition System, 7 January 2015

c. DoDI 8500.01, Cybersecurity, 14 March 2014

d. DoDI 8510.01, Risk Management Framework (RMF) for Information Technology (IT), Incorporating Change 1, Effective 24 May 2016

e. AFI 17-101

f. National Institute of Standards and Technology (NIST) Special Publication 800-37r1, Guide for Applying the Risk Management Framework to Federal Information Systems

g. NIST Special Publication 800-53r4, Security and Privacy Controls for Federal

Information Systems and Organizations

h. NIST Special Publication 800-53Ar4, Assessing Security and Privacy Controls in

Federal Information Systems and Organizations

i. NIST Special Publication 800-137, Information Security Continuous Monitoring

(ISCM) for Federal Information Systems and Organization

j. Federal Information Processing Standards (FIPS) Publication 199, Standards for

Security Categorization of Federal Information and Information Systems

k. Committee on National Security Systems Instruction No. 1253, Security Categorization and Control Selection for National Security Systems

Training systems provide a cooperative, real time training environment over dedicated point-to-point, fully encrypted commercial lines that do not connect or tunnel through any type of general service traffic on Non-Secure or Secure Internet Protocol Router Network (NIPRNet or SIPRNet). Simulator programs follow the AFLCMC Information Systems and Platform Information Technology systems Risk Management Framework (RMF) process. Simulator programs may be determined to be Special-Access Program/Special Access Required programs, which handle Sensitive Compartmentalized information material and are subject to the Joint Special Access Program Implementation Guide process, which aligns with the RMF.

Each Simulator program shall obtain an Authority to Operate (ATO) that is recommended for authorization by the Headquarters AF, Director of Training and Readiness, Operational Training Infrastructure (OTI) Division (HAF/A3TI) Security Control Assessor, and then authorized by the Authorizing Official (AO). Each program information system security manager is responsible for following the required activities and steps below to complete the RMF process to obtain ATO. Each step requires coordination of various documents, bodies of evidence, and memos with security control assessor representative personnel under the purview of the AO. All steps below are further described in DoDI 8510.01 and NIST Special Publication 800-37. In addition, the following document provides sample paragraphs for incorporating cybersecurity requirements and language into a Statement of Objectives (SOO), Statement of Work (SOW) or Performance Work Statement (PWS); RMF Contract Language (v17) 19 September 2016

The RMF process includes the following steps. Each step has entry and exit criteria used for timing purposes on when to proceed to the next step of the RMF process.

Step 1 – Categorize The Information System

1.1 – Categorize the information system

1.2 – Define the information system (define boundary)

1.3 – Register the information system

Step 2 – Select Security Controls

2.1 – Common control identification

2.2 – Select security controls

2.3 – Develop a continuous monitoring strategy

2.4 – Review and approve the System Security Plan (SSP)

Step 3 – Implement Security Controls

3.1 – Implement security controls specified in the security plan

3.2 – Document implementation

Step 4 – Assess Security Controls

4.1 – Develop Security Assessment Plan

4.2 – Assess controls

4.3 – Develop Security Assessment Report (SAR) and report findings

4.4 – Conduct initial remediation actions

Step 5 – Authorize Information System

5.1 – Develop Plan of Actions and Milestones (POAM)

5.2 – Assemble security authorization package (SSP, SAR, POAM)

5.3 – Determine risk

5.4 – Risk acceptance

Step 6 – Monitor Security Controls (Continuous Monitoring)

6.1 – Determine security impact of proposed/actual changes

6.2 – Assess controls as stated in SSP

6.3 – Remediation actions

6.4 – Update documentation

6.5 – Report security status

6.6 – Review security status

6.7 – Implement system decommission strategy

3.0 Engineering Resources and Management

3.1 Technical Schedule and Schedule Risk Assessment

Training Systems should be kept in a configuration concurrent with the real-world systems being simulated. Therefore, the real-world system development and upgrade schedule must be tracked, either by the program office or by the training system contractor if required as a part of the contract requirements. The government LE will ensure that the technical schedule is planned and executed to ensure training device concurrency with fielded real-world systems being trained. The government Program Manager is responsible to ensure that the technical schedule can be executed through proactive resource planning and budgeting activities in conjunction with the responsible MAJCOM, and if appropriate, aircraft program office. After contract award, concurrency-driven technical program tasks will be identified in change proposals and become part of the resultant contract Integrated Master Plan (IMP) and Integrated Master Schedule (IMS). These tasks will be managed by the contractor and tracked by the government LE with insight provided by the monthly IMS updates, and through the Earned Value Management System (EVMS) when that is a contract requirement.

Additionally, scheduling and planning for training system programs will include consideration of impacts to on-going training activities and phased transitions of fielded real-world systems. The government LE will ensure that training system schedule planning is coordinated with the MAJCOM to ensure consistency with known schedules and training needs.

Training system program schedules will include the following significant activities as applicable:

a. SE technical reviews

b. RFP release dates

c. Contract award (including bridge contracts)

d. Hardware/Software Integration events

e. Weapon System milestones that drive trainer upgrades

f. Testing events/phases

g. SIMCERT planned dates

h. Technology Readiness Assessments

i. Logistics/Sustainment events

j. Long-lead or advanced procurements

k. Ready for Training dates

Schedule risks for training systems will be assessed when specific efforts are planned and proposed, to include critical path analysis, training need date, resource availability, and comparisons with historical data on similar efforts, if available. The technical program execution will be closely monitored through IMS updates, and through software estimating techniques and EVMS analysis as applicable. When risks are identified through software estimating techniques or EVMS analysis, the PSEP Attachment will include hyperlinks to the applicable estimation/analysis reports, whether contractor or government program office generated. Whenever schedule risks are identified, they will be categorized and tracked, and mitigation plans will be established and monitored, in accordance with the program Risk Management Plan along with technical and cost risks.

See the PSEP Attachments for the System Technical Schedules.

3.2 Engineering Resources and Cost/Schedule Reporting

The initial (top-level) Work Breakdown Structure (WBS) is developed by the Simulators Program Office internal program team and included in the RFP, from which offerors propose the contract efforts with a WBS that includes a greater level of detail. Based on the program’s acquisition strategy, the IMP, SOW and/or PWS may be developed by either the Simulators Program Office, or proposed by the offeror, but regardless, will continue to be consistent with the WBS. Each PSEP Attachment will include the program’s WBS and tasking documents (IMP, SOW, PWS); schedule documents (IMS and other schedules); and for new acquisitions, the acquisition strategy and team responsible for developing it.

By ensuring consistency among the WBS and tasking/schedule documents, the program team will clearly communicate their expectations, and provide traceability for the management and execution of the program by the IPT, which includes the contractor and MAJCOM representatives. They also provide traceability for systems engineering, and risk management, which together define the products and key processes associated with program success.

Each Engineering Change Proposal (ECP) includes additional tasks for the contractor. These tasks will be included in updates to the program’s IMP, IMS, and WBS to ensure those technical tasks are tracked as well.

When the contract type supports it, EVMS cost reporting is used in conjunction with the IMS to compare what is scheduled to be accomplished with what is actually accomplished. For these contracts, an Integrated Baseline Review process is established and executed subsequent to contract award to ensure consistent and effective application of EVMS to track program efforts from beginning to end.

See the PSEP Attachments for the IMP, IMS, SOW and/or PWS, EVMS reports, and their links to the WBS.

3.3 Technical Risk and Opportunity Management

Each program within the Simulators Program Office will establish and execute a Technical Risk and Opportunity Management process, documented in a program Risk Management Plan, consistent with the AFLCMC Standard Process to Execute Risk and Issue Management in Acquisition Programs. In addition, each program will explain how their processes will integrate with the contractor’(s) processes.

https://cs4.eis.afmc.af.mil/sites/1534/APD/APD/Risk%20and%20Issue%20Management%20(RIM)%20in%20Acquisition%20Programs.docx https://cs4.eis.afmc.af.mil/sites/1534/APD/APD/Risk%20and%20Issue%20Management%20(RIM)%20in%20Acquisition%20Programs.docx

See the PSEP Attachments for the program risk matrix that identifies the program’s most critical technical risks, any optional Risk Burn-Down Plans, and, if applicable, the program’s Opportunity Management Plans.

3.4 Technical Organization

Government Program Office Organization

The Simulators Program Office organization chart is shown in Figure 3.4.1-1.

Figure 3.4.1-1 Simulators Program Office Organization Chart

Government IPT - Each PSEP Attachment will include a figure that illustrates the Government IPT organizational relationships among the internal team members, the Simulators Program Office organizational hierarchy, and external team members. The following elements will be included as applicable:

a. Legend, as applicable (e.g., color-coding)

b. Organization to which the IPT reports

c. Program Manager (PM)

d. LE

e. MAJCOM representatives

f. Functional Leads (e.g. test & evaluation, logistics, risk, software, configuration management, cybersecurity)

g. Core, matrix, and contractor support personnel

h. Field or additional Service representatives

i. Aircraft Program Point of Contact(s)

See the PSEP Attachments for the IPT Organization Chart.

Program Technical Staffing Levels The Simulators Program Office uses an organizational process to establish government staffing requirements for each of its programs. Simulators Program Office staffing requirements at the program level are typically small. A minimum of one functional lead is usually established for each function (some functions are less than full-time), and then additional resources are added based on program complexity, risks, and scope. Each PSEP Attachment will include a figure that summarizes a time-phased technical staffing plan for Simulators Program Office personnel.

See the PSEP Attachments for the Program Technical Staffing Chart.

Contractor(s) Program Organization Contractor staffing is typically determined when the effort is initially proposed, and is assessed against program requirements for risk. When available, PSEP Attachments will include diagrams of the contractor(s) program organization and staffing plans.

See the SEMPs referenced in individual PSEP Attachments for the Contractor Organization Chart and for the Contractor Technical Staffing Chart.

Engineering Team Organization and Staffing Each PSEP Attachment will include an overall IPT diagram that shows the government IPT’s relationship to the contractor’s IPT (when available), and any associated lower level IPTs/working groups. This diagram will show the hierarchy and relationship among lower level IPTs/working groups. The diagram will identify roles, responsibilities, and authority for all lower level IPTs/working groups.

See the PSEP Attachments for the IPT/Working Group Hierarchy and Relationship Diagram.

3.5 Relationships with External Technical Organizations

There are several external organizations that each program interacts with that provides additional program information or guidance. Some of these organizations are listed below:

a. Weapons System Program Office

b. AFLCMC/EZ Home Office

c. MAJCOMs/Users

d. Air Force Research Laboratory

e. National Air and Space Intelligence Center

f. Database Facilities and Organizations

g. DMO Program Teams, as well as other Simulator Networking Organizations

h. Other Services (Joint Programs)

i. Foreign Governments (FMS Programs)

See the PSEP Attachments for the Responsible Organization for management, funding and scheduling integration/interface efforts with external organizations.

3.6 Technical Performance Measures and Metrics

Each PSEP Attachment includes documentation that describes the Technical Performance Measures (TPMs) and metrics for the program. The program’s strategy for identifying, prioritizing, and selecting the set of metrics for monitoring and tracking program SE activities and performance is included. This documentation includes, if available:

a. An overview of the measurement planning and metrics selection process, including the approach to monitor execution to the established plan, and identification of roles, responsibilities, and authorities for this process.

b. How the program adds or deletes any TPMs and changes any TPM’s goal.

c. A minimum set of TPMs and intermediate goals and the plan to achieve them with as-of dates (to provide quantitative insight into requirements stability and specification compliance).

d. When acquiring a new, unproven technology, PMs will use a growth curve to plan, illustrate, and report progress until the reliability threshold is achieved. However, most simulator programs use Commercial Off-The-Shelf (COTS) products or aircraft products with documented reliability calculations. These designs provide the expected reliability when contractor development testing is completed so the programs do not plan to grow their reliability.

e. When applicable, programs will establish availability and mission reliability metrics within their respective PSEP Attachments, and if there are any contractual provisions related to meeting these metrics.

See the PSEP Attachments for the Table of Technical Performance Measures.

4.0 Technical Activities and Products

Training systems operational requirements are addressed in a high-level fashion near the end of the Joint Capabilities Integration and Development System (JCIDS) process. Most training system operational requirements are derived from the characteristics and capabilities of the Weapon System as established by the JCIDS process. Therefore, many training system programs are initiated at or soon after the Weapon System’s Milestone B decision. Also, training systems are required to be ready for training by Initial Operational Test and Evaluation or at Ready Assets Available, i.e., before Milestone C. Because of this, training system don’t follow the standard DoD acquisition phases of Material Solution Analysis (MSA), Technology Development, Engineering and Manufacturing Development (EMD), Production and Deployment, and Operations and Support (O&S). To be ready for training in this time frame, training systems go through the distinct acquisition phases show in Figure 4.0-1; Requirements Definition, Acquisition Planning, Source Selection, System Development, and Sustainment.

Figure 4.0-1 Training System Acquisition Phases

Requirements Definition Phase has elements of the standard DoD phases of MSA and Technology Development. Ideally, Requirements Definition for any training system program begins with a Training System Requirement Analysis (TSRA). TSRAs are based on the Weapon System’s master task lists that define all activities required of operators and maintainers. The TSRA process will distill these tasks into a set of training requirements which live throughout the program’s lifecycle. The TSRA results are incorporated into the program requirements for the next acquisition phase. It is possible to accomplish Requirements Definition without a TSRA; however, it requires conducting several iterations of user meetings with the MAJCOMs to elicit the detail needed to comprehensively define training system operational and performance requirements.

Acquisition Planning and Source Selection Phases have elements of the standard DoD phases of Technology Development and EMD. Acquisition Planning is the conduct of market research, assessing technology readiness levels, development of an acquisition strategy, estimation of program costs, creation of an RFP, etc. necessary to establish a training system acquisition program. Source Selection is the same as for any acquisition program, offeror proposals are evaluated to select a solution that best satisfies the government’s operational needs as documented in the RFP.

System Development Phase has elements of the standard DoD phases of EMD and Production and Deployment. System Development is the design and development of the training system by a contractor and in some instances will include courseware. System Development includes test and evaluation to ensure the training system meets the government's operational and performance requirements, and the initial deployment of the system to operational sites, including spares, manuals, and a contractor-developed training system support center.

Finally, the Sustainment Phase has elements of the standard DoD phases of Production and Deployment and O&S. Most training systems are acquired in such low numbers that full-rate production is superfluous. Further, training systems are designed with maximum use of commercially available components so that manufacturing risk and readiness is generally not a concern. So in this phase the emphasis is on delivering the remaining training systems and O&S.

In some training system programs, the contractor is responsible for operating the training system

Operational Requirement

Sole Source Recompetition Program Justification and Approval (J&A) Required

Acquisition Planning

Phase

Source Selection

Phase Sustainment

Phase System

Development Phase

Req’ts Definition

Phase

Recompetition Programs and training students. Even if not training students, the contractor is responsible for training system operation and maintenance to meet a government-defined rate for availability.

4.1 SE Results for the Previous Acquisition Phase

The PSEP Attachment will summarize the key results of the SE, integration, and verification processes and activities accomplished in the previous training system acquisition phase. These activities will depend on the specific training system acquisition phase of the program, but should demonstrate that the processes and activities necessary to end the previous phase and begin the current phase of the program are complete.

See the PSEP Attachments for SE activity results of the previous phase.

4.2 SE Activities for the Current Acquisition Phase

The PSEP Attachment will summarize the key SE, integration, and verification processes and activities occurring in the current training system acquisition phase. These activities will depend on the specific training system acquisition phase of the program, but should demonstrate that the processes and activities necessary to end the current phase and begin the next phase of the program are in work.

See the PSEP Attachments for SE activities for the current phase.

4.3 SE Activities for the Next Acquisition Phase

The PSEP Attachment will summarize key planned system engineering, integration, and verification processes and activities established or modified since the current acquisition phase, including updated risk reduction and mitigation strategies and technical and manufacturing maturity. These activities will depend on the specific training system acquisition phase of the program, but should focus on the processes and activities necessary to begin and complete the next phase of the program.

See the PSEP Attachments for planned SE activities for the next phase.

4.4 Requirements Development and Change Process

Analysis and Decomposition

When the training system is developed, the basic system performance of the Weapon System being trained is already established and there is a training system requirements allocation and decomposition process that must be followed. The LE will ensure that training system program tasks and performance requirements established in the Program Management Directive, SOO, SOW, PWS, System Requirements Document (SRD), or other capabilities requirements documents are allocated appropriately in a requirements tracking system that supports traceability down to configuration item build-to specifications and Verification and Validation plans. The tool(s) used for requirements traceability will be identified in Paragraph 4.9 as established below.

See the PSEP Attachments for the Requirements Decomposition/Specification Tree/Baseline

Requirements Management and Change Process Changes in training system program tasks and performance requirements will be managed and executed in accordance with the Configuration and Change Management Process described below in paragraph 4.6.

4.5 Technical Reviews

Program Technical Reviews

The LE will ensure technical reviews are planned and included in each program phase as appropriate for implementation of system capabilities and updates. These reviews are to ensure insight is provided into the system design and report the contractor’s progress toward meeting training system program tasks and performance requirements. These typically include requirements reviews, design reviews, technical working groups, test readiness reviews, system verification reviews, and configuration audits. The AFLCMC Systems Engineering Technical Review Guide will be used to define the technical reviews for each program phase.

Entrance and exit criteria for the successful completion of each event will be established in the program IMP. The LE will ensure that the appropriate AF chairperson, functional personnel, external technical organization, and MAJCOM representation is identified for each review event.

The program technical schedule provided in Section 3.0 will identify each technical review event. The PM is the AF person authorized to determine if technical review entry criteria and exit criteria have been met as well as the status of related action items; however the authority for determining status of entrance and exit criteria is usually delegated to the LE.

See the PSEP Attachments for the Technical Review Details, including Manufacturing Readiness Assessments, when applicable.

Technical Status Report and Reviews Each program LE shall keep the cognizant Section CSE and Simulators Program Office Chief Engineer regularly apprised of program technical status, by means of a Technical Status Report and associated review. Reporting shall be done in the Simulators Program Office Engineering SharePoint site for Technical Status Reports. The Technical Status Report will include the following information:

a. General - Descriptive information including the program identification, programmatic information, and the date of the report.

b. Top 3 Issues – Issues are current problems which are adversely affecting the program’s ability to achieve its objectives. The issues section should identify the top 3 problem areas that are of most concern, that consume the most time, and/or that impede the technical execution of the program. The report should describe the issue, identify the impact, and provide a plan for its resolution (i.e. closure criteria & anticipated closure date).

c. Top 3 Risk Items – Risks are potential problems which may become issues if not mitigated. Risks may be related to your top issues, but are more likely to be independent. The report should describe the risk, quantify its probability of occurrence, and state its potential impact. Actions being taken to mitigate each risk should also be reported.

https://cs.eis.afmc.af.mil/sites/AeroEngDisciplines/Systems/SETR/default.aspx https://cs.eis.afmc.af.mil/sites/AeroEngDisciplines/Systems/SETR/default.aspx https://org.eis.afmc.af.mil/sites/77AESW/677AESG/SYE/Lists/Technical%20Status%20Reports/AllItems.aspx

d. Special Interest Items – These are matters which do not qualify as top issues or risks, but are of particular interest to Simulators Program Office engineering leadership.

Examples of special interest items include Unit Compliance Inspection/Self Inspection Program preparation status, Deficiency Report counts, personnel training issues, etc.

Special interest items may be identified by the engineering leadership for tracking and reporting by the program’s LE. LEs are welcome to include additional special interest items based on the specifics of the program. There is no limit to the number of special interest items which may be reported.

e. Six-Month Look-Ahead – This is a rolling schedule of upcoming events and workload, by which the LE can communicate the need for additional resources, as well as lulls in activities. The LE should be prepared to discuss any engineering staff shortfalls which could impede the successful accomplishment of program events, especially manpower-intensive ones such as source selection or test.

f. OSS&E Status – OSS&E status shall include the five items listed in Table 2.2.3-1.

g. Additional Notes - Notes may be added at the end if something else needs to be reported, but does not fit in any of the other sections (e.g., Contract was awarded to … on [date].)

Technical status information shall be reviewed monthly at a minimum (preferably bi-weekly), and updated accordingly. The LE shall review technical status with their Section CSE on a schedule determined by the CSE. The Simulators Program Office Chief Engineer will set reporting schedules for the Section CSEs. Inclusion of other senior engineers, peers, and other team members at the technical status reviews is encouraged, but left to the discretion of the Section CSE.

RFP Technical Review RFP Technical Reviews will be conducted to ensure that the technical elements of Simulators Program Office RFPs for new programs reflect current, preferred engineering practices, with regard to the specification of requirements, definition of contract tasks, identification of deliverables, intellectual property strategy, implementation of SE processes, and proposal evaluation methods.

A combined senior engineer/peer review shall be conducted on RFP technical documents prior to release. The LE shall distribute and request a team review of RFP products, either as a complete package or incrementally, allowing a minimum of three weeks for review of all documents. The following documents, if planned for inclusion in the RFP, shall be addressed during the senior engineer/peer review:

a. Technical performance requirements (SRD, Systems Specification, etc.)

b. Tasking document (SOO, SOW, PWS)

c. Contract Data Requirements List (CDRL)

d. Section L – Instructions to Offerors

e. Section M – Proposal Evaluation Criteria

f. PSEP Attachment

The LE will attempt to resolve all senior engineer/peer review comments through incorporation in the RFP documents and discussion with the reviewer(s). The LE will brief the Simulators

Program Office Chief Engineer on the number and nature of the comments received from the senior engineer/peer review, proposed resolution, and any contentious issues that remain. Based on this information, the Chief Engineer will determine whether a technical review meeting is needed. If required, the technical review meeting will be held to resolve outstanding issues.

The Simulators Program Office Chief Engineer shall chair the technical review meeting with, as a minimum, the program LE, and the Section CSE. Others may be included at the discretion of the Simulators Program Office Chief Engineer. The technical review meeting shall be accomplished no later than four weeks before RFP release. Depending on the magnitude of RFP updates resulting from the technical review meeting, the Simulators Program Office Chief Engineer may direct the LE to re-distribute the updated RFP documents to the senior engineer/peer review group for a follow-up review.

4.6 Configuration and Change Management Process

The Configuration and Change Management Process consists of a series of steps that result in the control and modification of the approved technical product baseline of an awarded contract. The government will take control of the initial technical product baseline at Critical Design Review, or rationale included in the PSEP Attachment for assuming control at another time. When possible, the approved technical product baseline will be included in the PSEP Attachment when the baseline is both established and verified. This process includes key documents and products associated with each step in the process. Simulators Program Office programs follow the Configuration and Change Management procedures, roles, responsibilities, authorities, and change classifications in accordance with MIL-HDBK-61A, Configuration Management (CM) Guidance. In addition, the following Configuration Control Board (CCB) Charter has been established by the Simulators Program Office with CCB members identified by memorandum.

a. CCB charts and presentation shall be the responsibility of the IPT member (or their designated representative) who has cognizance of the subject matter. CCB members will include operational/functional chiefs for Engineering, Contracts, Finance, Program Management, Logistics, Configuration Management, Safety, and Facilities (as required) or their designated representative. It is the responsibility of each IPT member to brief these leads prior to the CCB. The CCB chairperson may designate additional members.

b. The CCB is the team that reviews proposed changes and its primary function is the control of technical baselines and contractual changes. The CCB is not a voting board.

Only the chairperson has approval and disapproval authority. The CCB chair is delegated by the Chief, Simulators Program Office.

c. Once signed, the original CCB directive is provided to the contracting officer for contract modification(s). The CCB directive will not be changed for any reason without the concurrence of the cognizant chairperson and then only through a revised or amended CCB directive.

d. If any CCB member non-concurs with the decision, they will be required to document their position in a memorandum for record and provide it to the Secretariat/IPT CM representative within 2 working days of the CCB.

e. The CCB file will contain as a minimum: proposal comments, CCB charts, CCB directive, attendance list, and minutes.