ePS_RFP_Attach_07_Test and Evaluation Master Plan (TEMP)_20170214.pdf

For Official Use Only (FOUO)

Test and Evaluation Master Plan for electronic Procurement System

(Development Request for Proposal)

TEIN 1841

ACAT I AM

Version: 1.01

7 June 2016

Distribution Statement D: Distribution authorized to the Department of Defense (DOD) and United States (U.S)

DoD contractor only (Administrative Use) (Oct. 2008). Questions concerning technical content or any other requests for this document shall be referred to the Navy Enterprise Business Solutions Program Manager, Program Executive

Office – Enterprise Information Systems, 1325 10th Street, SE, Suite 401, Washington, DC 20374.

Handling and Destruction Notice: Comply with Distribution Statement and destroyed by any method that will prevent disclosure of contents or reconstruction of the document. For Official Use Only (FOUO) – Contains information exempt from mandatory release to the public under Freedom of Information Act (FOIA) exemption 5 applies.

ePS

Filename:

ePS_TEMP.doc Page:

2/96

Version:

Version 1.01

Date of Creation:

7 Jun 2016

SUBMITTED BY

S. E. SEAY Date

Program Manager

Navy Enterprise Business Solutions

CONCURRED BY

V. S. Gavin Date

Program Executive Officer

Enterprise Information Systems

J. Polowczyk Date

RDML, U.S. Navy

Director, Logistics Programs

P. A. Sohl Date

RADM, U.S. Navy

Commander, Operational Test & Evaluation Force

Page:

3/96

NAVY APPROVAL

D. J. Hahn Date

RADM, U.S. Navy

Director, Innovation, Technology, Requirements, and Test and Evaluation

Honorable Sean J. Stackley Date

Assistant Secretary of the Navy (Research, Development, & Acquisition)

OSD APPROVAL

Mr. Frank Kendall Date

Under Secretary of Defense for Acquisition, Technology, and Logistics

Dr. J. M. Gilmore Date

Director, Operational Test and Evaluation

Page:

4/96

Document History

Date Version Description Authors

24 May 2016 1.0 Original Document D. Rodenberger

8 August 2016 1.01 Updated signature page; Updated Roles and

Responsibilities; Clarified T&E schedule; Updated

Cybersecurity Testing Section; Updated RMF

Section; Updated Appendix H: Acronyms and

Appendix I: References

D. Rodenberger

Page:

5/96

Table of Contents

1. Introduction

1.1 Purpose

1.2 Mission Description

1.3 System Description

1.4 Operational View

1.4.1 System Threat Assessment

1.4.2 Program Background

1.4.3 Key Capabilities

2. Test Program Management and Schedule

2.1 Test & Evaluation Management

2.1.1 Test & Evaluation Working Integrated Product Team (T&E WIPT)

2.2 Common Test and Evaluation Data Management Requirements

2.3 Deficiency Reporting

2.3.1 Verification of Correction of Deficiencies (VCD) Process

2.4 TEMP Updates

2.5 Integrated Test Program Schedule

3. Test and Evaluation Strategy

3.1 Developmental Test Objectives

3.1.1 Software Code Quality Checking

3.1.2 Software Functional Testing

3.1.3 Systems Integration Testing

3.1.4 System Performance Testing

3.1.5 Interoperability Testing

3.1.6 Usability Testing

3.1.7 Deployment Testing

3.1.8 Cybersecurity Testing

3.1.9 Training Package Evaluation

3.2 Evaluation Framework

3.2.1 Reliability Growth

3.2.2 Design of Experiments and Scientific Test Analysis Techniques (STAT)

3.3 Developmental Evaluation Approach

3.3.1 Configuration and Integration Test

3.3.2 Integrated Testing

3.3.3 Mission-Oriented Approach

3.3.4 Modeling and Simulation

3.3.5 Test Limitations

3.4 Live Fire Evaluation Approach

3.5 Certification for Operational Test & Evaluation

3.6 Operational Evaluation Approach

3.6.1 Operational Test and Evaluation Objectives

3.7 Other Certifications

3.7.1 Cybersecurity –Risk Management Framework (RMF)

3.7.2 Cyber Economic Vulnerability Assessment – CEVA

3.8 Future Test and Evaluation

4. Resources Summary

4.1 Introduction

4.1.1 Test Articles

Page:

6/96

4.1.2 Test Sites and Instrumentation

4.1.3 Test Sites

4.1.4 Test Support Equipment

4.1.5 Test Target and Test Expendables

4.1.6 Joint Mission Environment

4.1.7 Special Requirements

4.2 Federal, State, and Local Requirements

4.3 Manpower/Personnel Training

4.3.1 User T&E Participation

4.4 Test and Evaluation Funding Summary

List of Figures

Figure 1-1: ePS Operational View (OV) – 1

Figure 1-2: SV-1 of ePS Figure 2-1: ePS Program Schedule

Figure 3-1: Risk Management Framework Process Figure 3-2: Reliability Maturity (Notional) Figure E-1: ePS Continuum of Data Collection of Responses

List of Tables

Table 1-1: SV-7 SPMs List Table 1-2: Key System Attributes

Table 1-3: Technical Certifications Table 2-1: Roles and Responsibilities

Table 2-2: ePS Workgroups and WIPT Coordination Table 3-1: T&E Focus Areas by Phase Table 3-2: ePS Integrated T&E (IT&E) Strategy

Table 3-3: SCQC Entrance and Exit Criteria Table 3-4: Software Functional Testing Entrance and Exit Criteria Table 3-5: System Integration Primary Test Components

Table 3-6: SIT Entrance and Exit Criteria Table 3-7: Primary Components of System Performance Testing

Table 3-8: System Performance Testing Entrance and Exit Criteria Table 3-9: DoDAF Required Viewpoints Table 3-10: CIT Entrance and Exit Criteria Table 3-11: Operational Test Entrance and Exit Criteria Table 4-1: Types of T&E Resources Needed by Test Activity

Table 4-2: Test Tools Table 4-3: Manpower / Personnel Table 4-4: Test Funding Summary

Table A-1: Defect Priority/Criticality

Page:

7/96

Table A-2: Defect Categories

Table B-1: Operational Evaluation Framework Table B-2: Operational Evaluation Framework Legend Table C-1: Developmental Evaluation Framework (Source Selection) Table C-2: Developmental Evaluation Framework (Performance) Table C-3: Developmental Evaluation Framework (Interoperability)

Table C-5: Developmental Evaluation Framework (Reliability) Table D-1: Data Source Matrix

Table F-1: TRR Checklist

List of Appendixes

Appendix A: Defect Management Appendix B: Operational Evaluation Framework Appendix C: Developmental Evaluation Framework

Appendix D: Design of Experiments and Scientific Test Analysis Techniques Appendix E: Data Source Matrix

Appendix F: Test Readiness Review Checklist Appendix G: Glossary Appendix H: Acronyms

Appendix I: References Appendix J: Points of Contact

Page:

8/96

1. INTRODUCTION

1.1 Purpose

This initial Test and Evaluation Master Plan (TEMP) supports the Development Request for

Proposal (RFP) Release Decision Point. The TEMP documents the overarching guidance to coordinate and execute Test and Evaluation (T&E) of the Electronic Procurement System (ePS) that the Department of the Navy (DON) will acquire. It describes the involvement and relationship of various stakeholders to the ePS T&E. This plan will evolve prior to Authority to

Proceed (ATP), which replaces Milestone B (MS B) and serves as a living document. It provides a framework that generates plans for the ePS Program through a detailed Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E). The ATP version of this

TEMP will include approval by Deputy Assistant Secretary of Defense (DASD) for DT&E and

Director, Operational Test and Evaluation (DOT&E) with concurrence from prior signatories, obtained prior to contract award.

The ePS Program is an Acquisition Category IAM Major Automated Information System

(MAIS), Defense Business System (DBS) with the Under Secretary of Defense (Acquisition, Technology and Logistics) (USD (AT&L)) as the Milestone Decision Authority (MDA). The ePS is on the official Office of the Secretary of Defense (OSD) MAIS /Major Defense

Acquisition Program (MDAP) Program assignment list, dated 6 October 2015.

1.2 Mission Description

The Mission of the ePS is to competitively acquire, test, deliver, and successfully transition to a state-of-the-market DON Contract Writing System (CWS). The DON requires a comprehensive electronic contract writing capability, able to meet DON contract writing requirements, including effective deal initiation, management, termination, close out, and auditability across the full spectrum of Navy and Marine Corps contracting environments.

The ePS Program is a non-developmental acquisition of Commercial off the Shelf (COTS) software, in support of the DON contract writing requirements. The Program shall provide a true end-to-end (E2E) CWS which shall effectively optimize the DON Integrated Contracting

Environment (DICE). The ePS Program will support the contracting operations of approximately 5,500 Buyers and 10,000 Non-buyers (~15,500 users) across 445 DON sites both

Contiguous United States (CONUS) and Outside the Contiguous United States (OCONUS).

The Program will comply with DoDI 8500.01 Cybersecurity and DoDI 8510.01 Risk

Management Framework (RMF) to control and manage inherent Cybersecurity risks. The ePS

Program will utilize the Navy and Marine Corps Intranet (NMCI) network for the web-enabled

Common Access Card (CAC), which provides user authentication and controlled access to the network environment in accordance with Department of Defense Instruction (DoDI) 8520.2, Page:

9/96

Public Key Infrastructure (PKI) and Public Key (PK) Enabling, dated 1 April 2004, for authentication and access controls.

An approved Department of Defense (DoD) or DON hosting facility with a current Authority to

Operate (ATO) will host the ePS Program. The Cybersecurity service provider (CSSP) for Navy commercial cloud hosting is Navy Cyber Defense Operations Command (NCDOC). SSC-

Atlantic will provide engineering, transition, and sustainment services for commercial cloud hosted applications in support of Data Center and Application Optimization (DCAO) Office.

NCDOC will cover data types hosted behind the Navy cloud access point (CAP). The ePS will collaborate with the hosting provider to ensure that the COTS/Non-Developmental Item (NDI) software will not impact that certification.

1.3 System Description

The ePS Program will implement two primary components: a COTS CWS and a DON-managed

COTS interfacing capability, known within the market as an Enterprise Service Bus (ESB). This combination will allow the development of DON-unique interfaces to occur quickly and without modification creating a mature and robust contracting writing capability. In the event that a single COTS tool cannot meet the full ePS requirement set, consideration of Business Process

Management (BPM) tools will occur to close the requirement gap as a preferred alternative to customization of the COTS tool.

The ePS model signifies a modular, highly scalable and open-systems design. It complies with the Business Enterprise Architecture (BEA) and its associated Procure-to-Pay (P2P) process, the

DoD Architecture Framework (DoDAF) v2.02, and the DON Enterprise Architecture (EA) v5.0.

The ePS will be a CAC-enabled, web-based, and Section 508-compliant CWS that facilitates the entire contracting process from requirement identification through award, administration and payment, to final closeout. It will handle purchasing data for trading partners throughout the world who have contracts with the DON, including information capture from the data source – whether internally or externally generated – and will not require the re-entry of data elements for any ePS process. Additionally, the ePS will provide seamless (transparent) interfacing for all data exchanges and be available twenty-four hours a day, seven days a week to both CONUS and

OCONUS contracting environments with the exception of regularly scheduled maintenance periods.

1.4 Operational View

The Operational View (OV-1) depicts how ePS aligns with the BEA business domains. The

DON business domains will manage and coordinate the requirements end users generate. The

DON contracting domain will use ePS to satisfy sourcing requirements and share information with industry, federal and other government entities, financial assistance recipients, international partners, and Non-Government Organizations (NGOs). Through those sources, the DON

Acquisition domain will be able to deliver supplies, services, construction, and grants as well as

Page:

10/96 other assistance agreements that meet the sourcing capabilities the user needs to execute the full range of military operations.

As the single CWS across the DON contracting domain, the ePS develops all contract instrument types from commercial purchase orders, Uniform Contract Format (UCF) contracts and delivery/task orders, Non Federal Acquisition Regulation (FAR)-based agreements and other transactions. As such, the ePS shall be subject to the policies, guidance, instructions, and standardized procedures described in the FAR, Defense Federal Acquisition Regulation

Supplement (DFARS), Navy Marine Corps Acquisition Regulation Supplement (NMCARS), and the Department of Defense Grant and Agreement Regulations (DoDGARs).

As a supporting agency to other DoD and Federal activities, the ePS must also be capable of supporting specific policies, regulations, clauses, etc. as appropriate for the supported agency.

Figure 1-1 illustrates the Operational View (OV-1) of the ePS Program:

Figure 1-1: ePS Operational View (OV) – 1

Page:

11/96

1.4.1 System Threat Assessment

The ePS Program Management Office has initially assessed the Impact Value for

Confidentiality, Integrity and Availability as Moderate. Verification of this assessment will take place during the early stages of the Risk Management Framework categorization process. The baseline threat document is the Defense Intelligence Agency (DIA) Information Operations

Capstone Threat Assessment.

Information Operations (IO) poses the most likely and most serious threat to the DON ePS

Program. The ePS Program Management Office (PMO) will interact with the Space and Naval

Warfare Systems Command (SPAWAR) Scientific & Technical Intelligence Liaison Office

(STILO) to develop a tailored threat letter and maintain awareness of threat updates. It will determine if changes are necessary to the ePS budget, policies, processes, or technology. The specific threats listed below are in addition to threats associated with natural disasters.

The most serious threat to the ePS is the “Trusted Insider,” where a person with legitimate access acts either intentionally or unintentionally on behalf of an adversary. Any compromise of elements of the DoD Information Network (DoDIN) may pose a threat to the ePS due to the

Program’s integration with the DoDIN, formerly called the Global Information Grid (GIG). A compromised system may allow an adversary an attack avenue to other DoDIN components.

The most likely IO threats contain computer network operations threats including Computer

Network Exploitation (CNE) and Computer Network Attack (CNA). CNE is the unauthorized access by foreign computers and networks that may allow the gathering of information or intelligence for an adversary. Given the nature of the information of future DON systems within the ePS, CNE may provide critical intelligence for an adversary to develop future exploits.

Adversaries may use CNE to manipulate, disrupt, deny, degrade, or destroy information to damage the ePS. The ePS is likely to be a target for criminals due to the financial nature of the information. Other possible IO threats do not directly attack ePS, but may impact ePS’s capabilities. These include electronic attack and electronic warfare support systems that could intercept or disrupt DoDIN communications, which the ePS relies upon. The physical threats to the ePS are similar to those of the host unit, and should be in the host unit’s threat assessment.

The ePS Program will adhere to the trust and assurances of standard DoD authentication, separation of duties and role-based access controls.

1.4.2 Program Background

In November of 2011, the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD AT&L) set the initial goal that no new contracts, agreements, or orders be awarded through SPS after September 30, 2015, and that all use of SPS cease by September 30, 2017. The memo, revised in 2014, states that contracts issued prior to September 30, 2018 will receive an SPS sunset date of September 30, 2020. The memo further directed each Component to plan and budget for future contracting capabilities. The Investment Review Board (IRB) approved the ePS Problem Statement on February 24, 2015.

Page:

12/96

The ePS Acquisition Strategy/Acquisition Plan (AS/AP) details the acquisition approach for the ePS Program. The design of the AS/AP includes a streamlined approach with multiple builds and/or deployments. Each build allocates user requirements as approved by the Requirements

Governance Board (RGB). A Build Technical Review (BTR) will allocate requirements to functions and review the preliminary design with a systems integrator. Configuration and integration work aligns the COTS solution to meet requirements allocated to that deployment or build. A Fielding Technical Review (FTR) will occur following BTR, to complete the design and ensure readiness to begin testing. Integrated Testing, coordinated with Operational Test &

Evaluation Force (OPTEVFOR), will combine developmental and operational assessment. Upon completion of Integrated Testing, a Limited Deployment Decision (LDD) will authorize the first deployment of the ePS to the initial user commands. Initial Operational Test & Evaluation

(IOT&E) will follow. Following Limited Deployment, successive deployments will improve functionality to support enterprise wide procurement requirements to close functionality gaps, such as automatic order writing. Finally, the Full Deployment Decision (FDD) will authorize deployment to the remaining commands with the full functionality of ePS. A centralized infrastructure will ensure that all commands will use the most current deployment of ePS as it becomes available.

1.4.2.1 Previous Testing

Not Applicable

1.4.3 Key Capabilities

The ePS Program will support many activities and business rules that are consistent across multiple activities. These include the standardized format for contract line item structures, clause logic activities for solicitation instruments and award instruments, routing documents through pre-established channels, as well as ad-hoc workflows to accomplish reviews and approvals, and rendering documents into human-readable format. These general system requirements, segregated within the ePS architecture, allow standardization to facilitate maintenance and sustainment activities. The use of general system capabilities will occur within the ePS as the operational activities described above necessitate.

From the highest level, the ePS will:Support 100% of the DON's contracting actions in a fully data-centric environment including all procurement processes leading up to and associated with contract award, management, and administration

Fully automate acceptance, funds verification, and post 100% of the associated financial transactions supporting DON clean-audit objectives

One of the tenets behind the concept of operations for the ePS is to rationalize and standardize any custom interfaces. The operational environment in which the ultimate solution deploys may require some non-standard interfacing capabilities. Electronic data exchanges between DON financial systems, the distribution of the contract document, associated attachments, and contract

Page:

13/96 data to the Global Exchange (GEX) in accordance with the Procurement Data Standard, and the ability to create, validate, and finalize a Contract Action Report (CAR) on the Federal

Procurement Data System – Next Generation (FPDS-NG) are required capabilities. The ability to operate with the Integrated Acquisition Environment (IAE) modules as they migrate to the

System for Award Management (SAM) as discussed in Section 7 of the Capability Requirements

Document (CRD) is necessary to sustain operations and integrate those functions to reach an operational maturity appropriate for implementation into the ePS. The ePS System

Requirements Specification (SRS) v1.0 and its associated Requirements Traceability Matrix

(RTM) detail further requirements. The initial requirements baseline occured as part of the

Systems Requirements Review (SRR) Systems Engineering Technical Review (SETR) event in

FY 2016.

The ePS must accommodate several non-contracting specific laws, regulations, and policies in order to sustain operations in an integrated environment. Achievement of Requirements must occur as the ePS implements interfaces or standard services across functional domains to ensure seamless operations across the P2P process.

Specifically, ePS must comply with the:

Federal Financial Management Improvement Act (FFMIA): for interfaces with financial systems and the handling of line of accounting data in accordance with the Standard

Financial Information Structure (SFIS)

Federal Information System Controls Audit Manual (FISCAM): for auditability of actions taken within the system

DON BEA: to ensure alignment within the E2E business framework

Federal Information Security Management Act (FISMA): to ensure the security of the system, data, and network

Per the DoDI 5000.02, Defense Business Systems (DBS) do not employ the Joint Capabilities

Integration and Development System (JCIDS) process. The ePS Program is not a Joint Interest

Program and does not have Key Performance Parameters (KPPs). However, the Program will utilize System Performance Measures (SPMs) for those parameters considered most essential for an effective system capability. These SPMs capture the minimum operational effectiveness and suitability attributes needed to achieve the overall desired capabilities for the ePS during the life of the Program. These Measures contain threshold and objective values verifiable by testing or other analysis.

Page:

14/96

Table 1-1 describes the Systems View-7 (SV-7) SPMs required to support the system capabilities delivered with ePS:

Table 1-1: SV-7 SPMs List

SYSTEM

PERFORMANCE

MEASURE

DEFINITION THRESHOLD OBJECTIVE

Data Error

Rejection

Percent of erroneous data identified and rejected prior to data storage.

90% at IOC

95% at FOC 99.9%

Document Preview

Rendering Time

Percent of procurement instruments (e.g., awards, modifications, and orders) forms rendered as measured as a function of time starting with the application of a digital signature.

90% within 10 seconds of user selecting document preview

99.9% within 10 seconds of user selecting document preview

Contract

Instrument

Generation

Percent of procurement instruments (e.g., awards, modifications, and orders) generated as a function of time.

90% within 30 seconds of KO signature

99.9% within 30 seconds of KO signature

Entitlement Data

Transmission

Percent of transmission of contract entitlement data to the appropriate entitlement system from ePS.

90% within 4 hours of award

95% within 4 hours of award

Obligation Data

Transmission

Percent of contract actions finalized in ePS that transmit obligation data electronically to the appropriate financial management system.

90% within 4 hours of award

95% within 4 hours of award

Contract Data

Compliance

Percent of contract actions which are successfully distributed to EDA upon execution (i.e., KO signature). This presumes that all pre-deployment validation checks have successfully occurred prior to contract execution and that entire document chain (i.e., base and all modifications) was generated within the ePS.

95% of eligibility contract actions successfully meet all Federal, DoD and DON data validation requirements as assessed through centralized data validation services(s).

99.9% of eligibility contract actions successfully meet all Federal, DoD and DON data validation requirements as assessed through centralized data validation services(s).

The ePS will not create further infrastructure issues such as environmental quality, embedded instrumentation, electronic attack, and wartime reserve mode requirements. Because it is not a weapons system, ePS will not involve conventional and initial nuclear weapons effects, nor will

Page:

15/96 it give rise to survivability or safety concerns (e.g., system, nuclear, explosive, and flight safety).

Existing protected facilities will house all equipment with no natural environmental conditions

(e.g., climatic, terrain, weather, oceanographic and astro-geophysical factors), and or unplanned stimuli (e.g., fast cook-off, bullet impact, and sympathetic detonation). The ePS will not require host nation approval.

Table 1-2 lists the System Attributes considered critical or essential for an effective DON capability but not selected as SPMs, along with references to the non-functional requirements managed in the Requirements Traceability Matrix (RTM):

Table 1-2: Key System Attributes

SYSTEM

ATTRIBUTES

DEFINITION AND FORMULAS THRESHOLD OBJECTIVE

System

Responsiveness

The ability of the system to provide timely transactional and query process results to the user

Transactional

(OLTP)

95% within 5.0

Sec

Query (OLAP)

95% within

60.0Sec

Transactional

(OLTP)

95% within 2.0

Sec

Query (OLAP)

95% within

15.0Sec

Data

Restoration

The ability of the system to provide restoral capability from a catastrophic loss to include corruption, destruction, loss and compromise of the ePS components and data disruptions from the system.

Recovery Time

Objective (RTO)

= 24 Hrs.

Recovery Point

Objective (RPO)

= 8 Hrs.

RTO = 8 Hrs.

RPO = 8 Hrs.

Reliability

Mean Time Between Failure (MTBF) is measured by

Total Operating Time/Number of Failures

720 Hrs. 1080 Hrs.

Operational

Availability (Ao)

Availability is measured by uptime divided by uptime + [scheduled downtime + unscheduled downtime]

Threshold = 96%

Including

Scheduled

Downtime

99.5%

Discounting

Scheduled

Downtime

Threshold =

Objective

Page:

16/96

SYSTEM

ATTRIBUTES

DEFINITION AND FORMULAS THRESHOLD OBJECTIVE

Maintainability

Maintainability is measured by the amount of time it takes a system or system element to be repaired and is expressed as Mean Time to

Repair (MTTR).

MTTR = Total corrective Maintenance

Time/Total number of corrective Maintenance

Actions

MTTR =

6.7 Hrs.

MTTR =

2.7 Hrs.

Note: RAM reflects the E2E system including all hardware (as applicable) and software.

1.4.3.1 Key Interfaces

The ePS Program will integrate with DoD/DON Financial Systems to broker financial data exchanges to ensure integrity and control in the areas of budget, program management and delivery, external reporting, and data integrity.

The ePS presents the CWS capability along with the integration of, or translation via Enterprise

Service Bus (ESB) of the interfaces below. The ePS will interface with DON financial management, business, authoritative, and reporting systems along with other various DoD sources for authoritative and reporting functions. External systems that will interface with the ePS include:

DON Procurement & Financial Management (FM) Systems which perform the business functions:

o Accounting & Funds Verification o Requirement Generation via Purchase Requisition Creation o Or, both Accounting & PR Generation in one system

Federal & DoD Systems which perform the business functions:

o Authoritative data source o Federal reporting o Contract compliance o Entitlement o Opportunity publishing

Page:

17/96

Figure 1-2 illustrates external systems that will interface with the ePS:

Figure 1-2: SV-1 of ePS

The vision of the ePS management of system interfaces is to replace the translation function of the SPS Adapter with an Enterprise Service Bus (ESB). One of the ESB’s primary functions is to provide the functionality required to translate external Extensible Markup Language (XML) into the Purchase Request Data Standard (PRDS)/Procurement Data Standard (PDS) XML, and translate the DON external Line of Accounting (LOA) into the Standard Line of Accounting

(SLOA) format as available. Evaluation of key elements will reside in a typical Information

Support Plan (ISP) for net-readiness, interoperability, information supportability, and information.

1.4.3.2 Special Test of Certification Requirements

The ePS requires standard and specialized testing, and certification for deployment onto any

DoD network.

Page:

18/96

Table 1-3 portrays the ePS system-level technical certifications required during the DBS life cycle process:

Table 1-3: Technical Certifications

CERTIFICATION

PMO

TEAM/POC

ACTIVITIES TO

OBTAIN

CERTIFICATION

CERTIFICATION

AUTHORITY

EXPECTED

DATE

Clinger Cohen Act (CCA) Acquisition

Management with SE and

IT

Submit CCA package to

DON CIO

PM/DON CIO

(MDAP/MAIS also by DoD CIO)

Q4 FY16

RMF/Authority to Operate (ATO) SE, CS, and

IT

Submit RMF package

NAO (A&A)

-DON Senior

Information

Assurance Officer

(SIAO)/DON CIO

-Authorizing Official

(AO)

Q4 FY18

Certification for Initial

Operational Test and Evaluation

(IOT&E)

SE and

QA/Test

Operational Test

Readiness Review

(OTRR)

PEO EIS Q1 FY19

Certification of system’s compliance with applicable DoD and Federal CWS requirements and regulations, to include:

Compliance with the prescribed use of data and the interface requirements of Federal, DoD, and Navy enterprise systems

Compliance with data standards

SE and

QA/Test

SRR/BTR & FTR DFAS/DASN (AP)/

FMO/DPAP

Q4 FY18

Page:

19/96

CERTIFICATION

PMO

TEAM/POC

ACTIVITIES TO

OBTAIN

CERTIFICATION

CERTIFICATION

AUTHORITY

EXPECTED

DATE

Certification under § 2222

“Defense business systems:

architecture, accountability, and modernization”

System is in “compliance with the enterprise architecture”

“Business process supported by the DBS program is or will be as streamlined and efficient as practicable”

“Need to tailor COTS systems to meet unique requirements or incorporate unique interfaces has been eliminated or reduced to the maximum extent practicable”

SE/business process

Subject

Matter Expert

(SME)

“Defense Business

Systems Investment

Management Process

Guidance,”

June 2012

Navy DCMO/Navy

CIO/IRB/DBSMC

Q4 Annually

Interoperability Certification T&E WIPT

Lead

Obtain Certification:

Operational Test and

Evaluation Force

OPTEVFOR

interoperability testing/JTIC concurrence

JITC Q4 FY18

Spectrum Supportability

Determination -- Generally, spectrum analysis of COTS software products does not apply to DBS software, since there are neither any transmitters nor receivers or antennae

N/A DD Form 1494 DON CIO/ National

Telecommunications and Information

Administration

(NTIA)

N/A

1.4.3.3 Systems Engineering Requirements

The ePS RTM contains all system-performance requirements including SPMs and System

Attributes, and other non-functional requirements. The Naval Enterprise Business Solutions

Program Office Chief Engineer (Navy EBS CHENG) will tailor and follow the requirements management processes already in place within the PEO EIS portfolio as the Naval Systems

Engineering Guide, dated October 2009 documents.

Page:

20/96

Specifically, the Navy EBS CHENG will:

Tailor sub-processes of the Requirements Definition process (reference Naval Systems

Engineering Guide (October 2009) sub-processes SP14, SP15, SP16 and the

Requirements Validation Process sub-processes SP25 through SP29)

Document and baseline the tailored sub-processes

Train the Engineering team members in the tailored sub-processes

Schedule, staff, and implement the sub-process steps in a level-of-effort manner

Measure, track, and control requirements management performance

The requirements in the CRD establish the performance measurements for ePS.

The ePS Program will follow the DON approved Systems Engineering Technical Review

(SETR) process as defined in the ePS Systems Engineering Plan (SEP). Both formal and informal ePS technical reviews are event driven. The designated USN Technical Authorities

(TAs) will be responsible for providing independent assessments associated with each of the

SETR events via an assigned technical warrant. The SETR events will tailor to ePS specific requirements. A detailed checklist specific to each event will encompass all the evaluated areas and the entrance/exit criteria prior to each event. Identification of participants from the PMO and external personnel will occur prior to each event.

Following the Build Technical Decision (BD), a Field Technical Review (FTR) will occur to ensure development efforts meet requirements, and validated via Integrated Testing (a combined developmental and operational testing event coordinated with Commander Operational Test and

Evaluation Force (COMOPTEVFOR)). The Test Readiness Reviews (TRRs) will occur prior to each formal test event and to assess approval of the planned test objective, methods, procedures, and statistical analyses, in addition to assessing whether required test assets and resources were identified, available, and traceable to program requirements. The TRR will repeat on an as needed basis.

When Integrated Testing (IT) is near completion, PMW 220 will request approval from the

MDA to deploy to the field via an LDD.

2. TEST PROGRAM MANAGEMENT AND SCHEDULE

2.1 Test & Evaluation Management

The ePS T&E Program will encompass the broad aspects of testing such as testing scope, objectives, and schedules. Additionally, this effort brings together a diverse group of entities to manage and conduct the ePS testing requirements. The test entities will include, but are not limited to, the ePS T&E Manager, Vendor, DON major commands/claimants, and other applicable test organizations.

Page:

21/96

2.1.1 Test & Evaluation Working Integrated Product Team (T&E WIPT)

The ePS Program Management Office (PMO) Test & Evaluation Working Integrated Product

Team (T&E WIPT) signed Charter dated 27 July 2015, establishes and defines the T&E WIPT in collaboration with the stakeholders. Collectively, the ePS T&E WIPT collaborates, plans, coordinates, and staffs, facilitates, updates, analyzes, and monitors the execution of the TEMP.

The ePS T&E Assistant Program Manager (APM) chairs the T&E WIPT for the ePS PM. They will schedule group sessions (nominally twice a month until contract award, then monthly), document action items, produce minutes, distribute information to members, and document issues for resolution by senior decision-makers. The T&E WIPT will address specific matters and elevate unresolved issues up the chain of command.

Organizations listed below have stakeholder interest in the ePS TEMP. Each of ePS TEMP stakeholder organizations assigns a single empowered representative to participate in the ePS

T&E WIPT. The empowered representative has the authority to speak for their respective organization on topics related to T&E. The WIPT has both core (routinely engaged) and adjunct

(engaged as needed) members. Core and adjunct members may call upon Subject Matter Experts

(SMEs) from their parent organization as needed. The Vendor may participate in the ePS T&E

WIPT as necessary, with the exception of any matter that is inherently Governmental. Table 2-1 outlines the roles and responsibilities of organizations associated with the ePS TEMP:

Table 2-1: Roles and Responsibilities

ORGANIZATION ROLE / POSITION RESPONSIBILITIES

Office of Under Secretary of Defense for Acquisition, Technology, and Logistics

(USD (AT&L))

Milestone Decision

Authority (MDA) Approves entry of the ePS Program into each phase of the acquisition process

Approves the TEMP

Provides approval to proceed into Operational Test

Provides limited and full deployment decisions

Department of the Navy, Chief Information Officer

(DON CIO)

Core T&E WIPT

Member Serves as a core member of the T&E WIPT

Chief of Naval Operations

(CNO) for Director, Logistics Programs (N41)

Core T&E WIPT

Member Advises the ePS T&E WIPT on ePS requirements and funding as the Resource Sponsor and serves as a core member of the T&E

Ensures that LDD, FDD, and FD processes and criteria, concepts of operation, and the Capabilities Requirements

Document are adequately defined to drive test strategies

CNO (N94) (Director, Innovation, Test &

Evaluation, and Technology

Requirements)

Core T&E WIPT

Member Serves as a core member of the T&E WIPT

Approves TEMP for CNO

Provides Navy T&E policy guidance, T&E WIPT coordination and dispute adjudication to acquisition programs

Serves as primary interface between OSD T&E

Page:

22/96

ORGANIZATION ROLE / POSITION RESPONSIBILITIES

Oversight organizations and Navy activities on testing matters

Deputy Assistant Secretary of the Navy for Acquisition and Procurement

(DASN AP)

Functional Sponsor Advises the ASN on Federal Acquisition

Regulation (FAR), Defense Acquisition Regulations

System (DFARS) and Navy-specific acquisition regulations and policies

DON ePS Systems

Interface Partners

Interface Partners Ensure integration with required interfacing systems

Provide resources and assistance required for T&E of integration/interfaces

Assist in Interface Control Agreement (ICA) development and collecting test data

PEO Enterprise

Information Systems

(PEO-EIS)

Program Executive

Officer (PEO) Provides policy and guidance for the execution of the ePS Program

Manages all ePS staff resources supporting execution of the programs

Oversees the activities of the Test Infrastructure (TI) activities supporting the Government-Approved Labs

(GALs)

Chairs the OTRR event

PEO Enterprise

Information Systems

(PEO-EIS) Engineering

PEO EIS Test Lead Under the supervision of the Assistant PEO –

Engineering (APEO-E)

Reviews and provides feedback on ePS T&E products

Approves/recommends approval when PEO-level approval is required for ePS T&E documents

Monitors and provides feedback regarding ePS T&E activities in terms of policy, standardized procedures, and evolving methodology

Routinely represents the PEO staff in ePS T&E WIPT activities

Electronic Procurement

System (ePS) Program

Management Office (PMO) ePS Program

Manager (PM) Executes the ePS Program, providing assessment of program status and risk reporting to stakeholders

Actively manages, within approved resources, program costs, performance, and schedule

Serves as final authority in the use of best practices, and resource authorization

Ensures Deficiency Reporting (DR) system is in place

Navy EBS PMO

Assistant Program

Manager, Engineering

(APM-E) for T&E

Coordinates/directs T&E activities across the Program

Coordinates with the Operational Test Agency (OTA) and Joint Interoperability Test Command (JITC) for

Operational Test and Evaluation (OT&E) and

Interoperability (IOP) respectively

Provides input for Test Resource Budgeting and

Coordination

Provides guidance for planning and execution of test

Page:

23/96

Navy EBS PMO continued events and monitors reports of findings

Provides input for program acquisition documentation

Chairs the ePS T&E WIPT (See Section 2.1.1.)

Serves as the Chief Developmental Tester, in compliance with FY 2012 National Defense Authorization Act

(NDAA) section 835 and the Interim Department of

Defense Instruction (DoDI) 5000.2

Serves as the Lead DT&E Organization, per DoDI

5000.02 Encl 4

Cybersecurity

Support Ensures Cybersecurity is adequately addressed in test processes, test plans, procedures, templates, measures, and metrics

Exercises involvement throughout the life cycle to provide Cybersecurity subject matter expertise

Satisfies the procedural requirements to achieve accreditation in each Department

System Engineering

(SE) Group Exercises involvement throughout the life cycle to provide subject matter expertise on engineering and architectural matters

Identifies key interfaces

Analyzes the architecture for weakness in scalability, performance, security, load, Reliability, Availability, and

Maintainability (RAM)

Coordinates with T&E team to map requirements to test plans

Supports GALs requirements definition and validation

Data Authentication

Team (Consists of the Government

Developmental

Testing (DT) team)

Determines category and priority of defect reports and change requests

Validates regression test results address both the needed change and system stability

Calls upon organizational experts as needed to help resolve anomalies and analytical questions

Prioritizes and tracks defects/change requests

Configuration

Management (CM) Provides repository for configuration baseline management

Manages ePS technical baseline

Risk Management Participates in T&E WIPT

Identifies, reports and monitors risks and issues

Page:

24/96

Independent Verification

& Validation Agent (IV&V)

IV&V Provide an independent, unbiased assessment of the compliance of the ePS CWS with the RTM

Provide more confidence that the new CWS will be fit for operations and satisfy the business needs and associated requirements for availability, quality, and maintainability

Support ePS CWS life cycle processes to ensure compliance with regulatory, performance, schedule, and budgetary requirements and facilitate early detection of their variances

Enhance management insight into process and system risks

Execute the IV&V activities, capture metrics, and deliver quantitative assessment of findings

Navy Commander

Operational Test and

Evaluation Force

(COMOPTEVFOR)

OT&E Support Acts as the OTA for ePS

Serves as an advocate for Component-specific mission needs

Exercises overall responsibility to plan and evaluate

OT&E to verify the operational Effectiveness and

Suitability of ePS

Coordinates OT&E plans, reports, and briefings to

Director, Operational Test & Evaluation (DOT&E), PMO, PEO, Office of the Secretary of Defense (OSD)

T&E Oversight organizations and the DON

Coordinates OT&E input and concurrence of TEMP approval

Evaluates use cases and requirements for testability

Analyzes workflows and assists in the development of detailed independent Government test cases and suitable test data sets in collaboration with the SMEs and DT&E community

Supports systems integration, functional, and performance testing activities in the Government lab

Establishes Test Management Council (TMC) comprised of OTAs to arbitrate issues that cannot be resolved at the working level

Formulates/coordinates with other Test Agencies and consolidates OT documentation

Ensures a comprehensive data collection and management plan is formulated and coordinated with

OTAs

Ensures common data repository is established

Provides DR procedures, formats, and directions to

OTAs

Accepts DRs, develop DR status reports, submit DRs to

PM

Page:

25/96

Space and Naval Warfare

Systems Command

(SPAWAR) 5.0 Engineering

SPAWAR 5.0

Engineering Navy IT Technical Authority, T&E Technical Process

Owner

Evaluates use cases and requirements for testability

Oversees and reviews testing for systems integration, functional, and performance testing activities in the pre-production environment

Provides Independent Verification and Validation

(IV&V) Agents to conduct continuous collection, analysis, and reporting of pre-determined process and product metrics

Augments the Government Independent T&E team to execute planning and conduct of DT&E after

Configuration and Integration Testing (CIT)

Ensures testing coverage of all requirements

Collaborates in the T&E WIPT for planning Integrated

Test and Evaluation (IT&E)

Reviews all test-related documentation, test plans, test cases, and test reports

Advises T&E APM on T&E matters

Director, Operational Test

&Evaluation (DOT&E)

OT&E Oversight

Organization for

DoD

Exercises oversight of OT&E

Determines adequacy of Operational Test Plans, conduct

Conducts an independent assessment of the operational

Effectiveness and Suitability of the system and reports the results to the MDA and to the Congress

Approves the TEMP and operational test plans

Scientific Test and

Analysis Techniques in Test & Evaluation

Center of Excellence

(STAT T&E COE)

T&E SME Provides experience in DoD T&E planning, execution, and assessment

Provides knowledge of advanced analytical techniques

Provides support to the TEMP development and test planning with scientific designs, and processes

Provides support for test execution, data reduction, and analysis with professional statistical software

Provides support to the development of case studies and new statistical methods

Collects and promotes best practices for the use of statistical design/analysis within T&E

Deputy Assistant Secretary of Defense (DASD)

Developmental Test and

Evaluation (DT&E)

DT&E Oversight

Provides DoD policy compliance oversight for

Developmental Test and Evaluation and approval authority for documents mandated by regulation or statute

Monitors and reviews all DT&E plans and activities

Reviews Developmental Test Results as required

Approves TEMP

Approves DT&E planning within the TEMP

Approves DT&E completion and system readiness for

Page:

26/96

OT&E

Provides independent assessment of DT&E results to the

MDA

Joint Interoperability

Testing Command (JITC)

IOP Certification

Authority Reviews ePS documentation such as DoDAF products and ISP related artifacts to ensure joint interoperability requirements are testable and measurable and provides a desktop audit of interoperability requirements of DON

OTA results

Develops the Joint IOP Certification Evaluation Plan

(ICEP)

Coordinates with the DT/OT teams and leverage existing data and planned events to collect joint interoperability data

Analyzes interoperability data and present results regarding compliance with Interoperability requirements in the Interoperability Certification Memo/Report

DoD Testing Organizations Government Test and Evaluation

Team

Plans, coordinates, conducts, evaluates, and reports test activities in the Government lab and operational environments to ensure the ePS requirements are met

Supports the OT activity as needed

Conducts over-the-shoulder testing during the Vendor’s test events

Develops test plans and reports

Participates in WIPT meetings

Plans, executes, analyzes, and reports DT&E and OT&E activities

Provides independent T&E reports to the WIPT

PMW 220 PMO Cybersecurity

Testing Coordinates Cooperative Vulnerability and Penetration

Assessment

Advises T&E WIPT on Risk Management Framework

(RMF) for DoD Information Technology (IT) implications

Collaborates with Commander Operational Test and

Evaluation Force (COMOPTEVFOR) OT&E

Cybersecurity team

The makeup of the “Government T&E Teams” will vary based on testing activities. For example, ePS T&E welcomes DT&E, OT&E, Cybersecurity, and Independent Verification and

Validation (IV&V) personnel to participate in T&E activities, recognizing those personnel would only do so when appropriate, considering their independent T&E missions.

Page:

27/96

Table 2-2 identifies other workgroups with which the T&E WIPT will coordinate:

Table 2-2: ePS Workgroups and WIPT Coordination

NAME WORKGROUP COORDINATION

Risk Management

Workgroup

The T&E WIPT will collaborate with the Navy EBS PMO Risk Management

Workgroup in identifying, documenting, and handling T&E related risks. Risks outside the scope of the authority of the WIPT will be elevated to senior leadership following the ePS risk management process.

Requirements

Management Workgroup

The T&E WIPT will work closely with the ePS Requirements Management

Workgroup to ensure all applicable test cases and test scripts include approved requirements changes.

Cybersecurity

Workgroup

The T&E WIPT will work closely with the Cybersecurity Workgroup whose role is to serve as a support body to the ePS PM. They provide a forum ensuring identification, documentation, and infusion of every applicable cybersecurity requirement in all acquisition program documents, integrate into the Information System during development, validate through testing and the RMF process, and support throughout the Information System’s life cycle.

Schedule Management

Workgroup

The T&E WIPT will identify T&E related schedules for Developmental Testing (DT) and OT activities. It will coordinate with the Schedule Management Workgroup to ensure the ePS Integrated Master Schedule (IMS) incorporates T&E related schedule data.

2.2 Common Test and Evaluation Data Management Requirements

The T&E WIPT will coordinate the processes and procedures for T&E Data Management (DM).

The ePS T&E test data management approach includes the following best practices:

EPS test data repository for all test-related data and documentation (e.g., plans, reports, references, test deficiency/defect reports)

Support DM plans and procedures

Creation, validation/authentication, and archiving of test data sets needed to support execution of test scripts

Collection, analysis, authentication, quality assurance, and maintenance of test data results

Development, archiving, and management of all test-related documentation

PMW 220 Data Management will maintain all DT&E and OT&E data in the approved data repository. The sharing and release of data collected by COMOPTEVFOR will be in accordance with COMOPTEVFORINST 3980.2 Operational Test Director’s Manual. The PMW 220 approved data repository will include OT data shared with the PMO.

Test data refers to all data (Developmental, Integrated, or Operational) that can be processed by the ePS System Under Test (SUT) to verify ePS’s achievement of functional and non-functional

Page:

28/96 requirements. The test data set must not include actual Personally Identifiable Information (PII), only fabricated PII. It must be up to date and maintain current data standards as they evolve.

Test data results are data that has been collected and includes both results and conditions of the executed and observed test cases. PMW 220 Data Management will collect test data results, review the results for accuracy, completeness, and consistency, categorize, prioritize, and evaluate said results.

The T&E WIPT will collaborate to identify the automated tools and techniques necessary to collect the test data results and address the objectives for the testing:

o Assist in fault isolation o Assist in performance data collection o Diagnose defects o Obtain metrics o Obtain system and network monitoring information

The ePS T&E team will authorize access to test data by T&E WIPT members and PEO/PMO staff. The ePS test data repository will include Vendor test-related artifacts and results.

The Navy EBS PMO staff will collaborate with the ePS Data Management Working Group and the Head of Contracting Activities (HCAs) to develop an enterprise test data set for the CWS test purposes. The data set must be correlated for the test data sources across multiple agencies, systematically varied to exercise branches of functional logic, and provide sufficient volume for performance testing.

Throughout the data collection process, the T&E WIPT conducts Quality Control (QC) procedures to ensure the data is accurate, complete, consistent, and supports the evaluator’s needs.

2.3 Deficiency Reporting

The T&E WIPT will follow the defect management process by tracking all reported defects using the PMO-selected automated defect-tracking tool. The defect management process includes the following:

Setting up the defect-tracking tool

Logging, analyzing, resolving, and tracking identified defects

Tracking desired or suggested enhancements

Analyzing defects requires the problem-tracking tool to hold information needed for problem resolution statistics such as status,…