Attch_0023_ACWS_TEMP_Draft_2015_10_17.pdf

ACWS Draft Test and Evaluation Master Plan Version 1.0 9 December 2015

Distribution A: Approved for public release; distribution unlimited

ARMY CONTRACT

WRITING SYSTEM

(ACWS)

Test and Evaluation Master Plan (TEMP) Draft

Version 1.0 9 December 2015

Prepared By:

Product Manager Army Contract Writing System (ACWS)

ACWS Draft Test and Evaluation Master Plan i

This page intentionally left bank

ACWS Draft Test and Evaluation Master Plan ii

SUBMITTED BY:

LTC Robert J. Wolfe Date ACWS Product Manager

DRAFT APPROVED BY:

COL William M. Russell Date GFEBS Project Manager

APPROVAL PAGE

ACWS Draft Test and Evaluation Master Plan iii

Revision History Version Date Summary of Changes

1.0 9 December 2015 PM-approved draft for Final RFP

ACWS Draft Test and Evaluation Master Plan iv

Table of Contents

1.0 Introduction

1.1 Purpose

1.2 Mission Description

1.3 System Description

1.3.1 System Threat Assessment

1.3.2 Program Background

1.3.3 Key Capabilities

2.0 Test Program Management and Schedule

2.1 T&E Management

2.1.1 Role of the Project Management Office

2.1.2 Role of ACWS Test & Evaluation Section

2.1.3 Role of the System Integrator

2.1.4 Role of Army Test & Evaluation Command (ATEC)

2.1.5 Joint Interoperability Test Command (JITC)

2.1.6 Under Secretary of Defense of Acquisition, Technology, and Logistics

(USD(AT&L))

2.1.7 Director, Operational Test & Evaluation (DOT&E)

2.1.8 Deputy Assistant Secretary of Defense for Developmental Test & Evaluation

(DASD(DT&E))

2.1.9 Deputy, Under Secretary of the Army – Test & Evaluation (DUSA–TE)

2.1.10 Army Chief Information Officer (CIO)/G6 – Network Enterprise Technology Command (NETCOM)

2.1.11 Army Agent of the Security Control Assessor (ACA)

2.1.12 Army Research Laboratory/Human Research and Engineering Directorate

(ARL/HRED)

2.1.13 Army Audit Agency (AAA)

2.1.14 Functional Sponsor

2.1.15 PEO EIS

2.1.16 T&E Organizational Construct

2.2 Common T&E Database Requirements

ACWS Draft Test and Evaluation Master Plan v

2.3 Deficiency Reporting

2.4 TEMP Updates

2.4.1 T&E TEMP Sub Working Group

2.5 Integrated Test Program Schedule

2.5.1 Schedule Concurrency Risk Mitigation

3.0 Test and Evaluation Strategy

3.1 T&E Strategy

3.2 Evaluation Framework

3.2.1 Reliability Growth

3.2.2 Design of Experiments (DOE)

3.3 Developmental Evaluation Approach

3.3.1 Mission-Oriented Approach

3.3.2 Developmental Test Objectives

3.3.3 Modeling & Simulation

3.3.4 Test Limitations

3.4 Live Fire Evaluation Approach

3.5 Certification for Initial Operational Test & Evaluation (IOT&E)

3.5.1 Assessment of Operational Test Readiness

3.6 Operational Evaluation Approach

3.6.1 Operational Test Objectives

3.6.2 Modeling and Simulation

3.6.3 Test Limitations

3.7 Other Certifications

3.8 Future Test and Evaluation

4.0 Resources Summary

4.1 Introduction

4.1.1 Test Articles

4.1.2 Test Sites and Instrumentation

4.1.3 Test Support Equipment

4.1.4 Threat Representation

ACWS Draft Test and Evaluation Master Plan vi

4.1.5 Test Targets and Expendables

4.1.6 Operational Force Test Support

4.1.7 Models, Simulations, and Test Beds

4.1.8 Joint Operational Test Environment

4.1.9 Special Requirements

4.2 Federal State and Local Requirements

4.3 Manpower/Personnel and Training

4.4 Test Funding Summary

Appendix A: Acronym List

Appendix B: Points of Contact

List of Figures Figure 1: ACWS Operational View-1 (OV-1)

Figure 2: ACWS Notional Capabilities Roadmap

Figure 3: ACWS Systems Interface Landscape

Figure 4: ACWS Notional Integrated Program Schedule

Figure 5: ACWS High-Level T&E Strategy

List of Tables Table 1: Evolutionary Acquisition Approach

Table 2: ACWS Key Performance Parameters

Table 3: WIPT Members Roles and Responsibilities

Table 4: ACWS Survivability Strategy

Table 5: ACWS T&E Funding Summary

ACWS Draft Test and Evaluation Master Plan 1

1.0 Introduction

1.1 Purpose

This initial Army Contract Writing System (ACWS) Test and Evaluation Master Plan (TEMP) defines and outlines the strategy, stakeholders, events, and resources involved in the overall Test & Evaluation (T&E) planning from system development through operational T&E. The ACWS Acquisition Strategy (AS) and the Systems Engineering Plan (SEP) initiate the development of the TEMP.

This version of the TEMP provided as a draft to support the Milestone Decision Authority (MDA) Request for Proposal (RFP) Release decision for the ACWS.

1.2 Mission Description

The ACWS is a Defense Business System and has been designated an ACAT IAM program in accordance with the requirements of the Department of Defense Instruction (DoDI) 5000.02. It is managed by Product Manager (PdM) ACWS under the Program Manager (PM) General Fund Enterprise Business Systems. PM GFEBS is under the purview of Program Executive Officer Enterprise Information Systems (PEO EIS). The Milestone Decision Authority (MDA) is the Under Secretary of Defense for Acquisition Technology, and Logistics (USD(AT&L)).

The Office of the Deputy Assistant Secretary for Procurement (ODASA(P)) is the Functional Sponsor for ACWS. ODASA(P) desires a premier contract writing system that uses modern technologies to allow for customized workflow, on-demand collaboration, and rapid adaptability to frequent changes in regulation, policies, and procedures using centralized services, standardized business rules, and standard data schemas.

To achieve this vision, the ACWS Product Management Office (PMO) will evaluate, identify, select, implement, and maintain a single, enterprise-wide contract writing and management system. ACWS will replace the current capabilities of the Standard Procurement System (SPS), the Procurement Automated Data and Document System (PADDS), and the Virtual Contracting Enterprise (VCE) suite of tools.

ACWS Draft Test and Evaluation Master Plan 2

Figure 1: ACWS Operational View-1 (OV-1) illustrates the functional sponsor’s high-level view of the intended operational environment and the major mission functions that ACWS will support.

Figure 1: ACWS Operational View-1 (OV-1)

1.3 System Description

The ACWS Commercial-Off-The-Shelf (COTS) based software solution will meet the full scope of Army Contracting requirements and processes, including those in secure and non-secure locations. ACWS will function in low bandwidth/disconnected status for expeditionary forces, will support both classified and unclassified networks, will support combat or non-combat contingencies, within or outside the borders of the Continental United States, will support grants and assistance agreements, and will support weapons systems, construction, installation, and other specialized contracting activities.

The hardware component of ACWS is limited to hosting services provided by Defense Information Systems Agency (DISA). The system utilizes web-based capability to provide users access to information and to exchange operational data related to contract procurement actions and additional financial management information.

ACWS is expected to utilize centralized services such as clause logic, standardized business rules, including contract line item number structure and standard data schemas such as Procurement Data Standard and Standard Financial Information Structure.

ACWS Draft Test and Evaluation Master Plan 3

Figure 2: ACWS Notional Capabilities Roadmap depicts the ACWS System functional capabilities, followed by the Initial Operational Capabilities (IOC) and Full Deployment (FD) definitions.

Figure 2: ACWS Notional Capabilities Roadmap

IOC is achieved when ACWS:

• Has been deployed and trained to at least one (1) Army Procurement command or sub-command

• Can be used to execute basic procurement functions (as defined above)

• Can successfully exchange data with all identified Federal and DoD systems

• Can successfully exchange data with a minimum of one (1) Army Financial system (GFEBS, LMP, or CEFMS) FD is achieved when ACWS:

• Has been deployed and trained across the Army Contracting Enterprise (ACE)

• Can be used to execute all identified capabilities (as defined above)

• Can successfully exchange data with all identified Federal, DoD, Financial and

Mission Specific systems

1.3.1 System Threat Assessment

The ACWS Product Management Office (PMO), planning as an ACAT IAM program, has coordinated with Training and Doctrine Command (TRADOC) G-2 to plan for a System Threat Assessment Report (STAR). A Threat Steering Group will kick off the process in 1QFY16. TRADOC will then prepare a STAR for the ACWS program with a planned completion date to support the ATP-1 decision. DIA, in coordination with the ACWS Product Manager (PdM), will determine the generation of the relevant operational threat environment based on the STAR, the Multi-Service Force

ACWS Draft Test and Evaluation Master Plan 4

Deployment, the Joint Country Forces Assessment, and scenario support products in accordance with DoDI 5000.61.

The 2015 National Security Strategy and the National Military Strategy recognizes the risks and dangers posed by disruptive and even destructive cyber-attacks and understand our growing investments in the cyber realm to protect vital networks and infrastructure. The vast amounts of procurement/contract data ACWS will process and transfer make it a lucrative target of diverse worldwide threats. While the threat predominantly affects the operational phase of the system lifecycle, it is present throughout system development and sustainment.

Army Contracting Centers operate worldwide on or near military bases and are subject to the same operational environment as the communities they serve. Threats to ACWS vary depending on the level of conflict, location/vulnerability of individual nodes, and the sensitivity of the compiled data contained within the enterprise-wide central database.

The ACWS solution will contain a number of potential entry points associated with service layer boundaries (e.g., between the services themselves, between the services and the solution components which they connect). At a minimum, T&E activities will need to address all appropriate potential points for unauthorized access (e.g., remote machine, local machine while in operation, messaging). The Developmental Test & Evaluation (DT&E) and Operational Test & Evaluation (OT&E) events described in Section 3.0 will include cybersecurity T&E consistent with the cybersecurity threats described in the STAR.

Potential IT-related threats to ACWS include:

• Electronic eavesdropping

• Unauthorized disclosure of information

• Alteration of application programs

• Unauthorized access/use

• Denial of service

• Theft of assets

• Employee sabotage

• Vandalism

• Malicious attack

• Shared passwords

• Compromised passwords

• Spoofing

The ACWS PMO will assess the infrastructure operational threat probability of occurrence during RR phase in conjunction with program protection planning activities.

ACWS Draft Test and Evaluation Master Plan 5

The ACWS will have a robust cybersecurity plan to protect acquisition/contracting information.

Environmental threat results from vulnerabilities in the operational physical environment.

These include acts that are intentional (e.g., terrorism, vandalism) and unintentional (e.g., acts of nature). Potential environmental threats include:

• Climate control System failure

• Earthquake

• Electrical system failure

• Explosion/terrorist acts

• Fire/smoke damage

• Flood

• Hardware failure

• Severe storms

• Snow/ice

• Tornados

• Water damage

If hosted in the DISA facility as anticipated, then DISA will manage the prevention and mitigation of environmental threats to ACWS.

The ACWS Cybersecurity Strategy and Program Protection Plan documents address specific threats, vulnerabilities, and countermeasures. As the ACWS PMO updates these documents, the Cybersecurity Lead and Systems Engineering Lead will align the content with the TEMP to ensure cross-coordination. The Cybersecurity Lead will serve as the Point of Contact (POC) for ensuring alignment.

1.3.2 Program Background

Background Information for Proposed System The Army has relied on SPS, PADDS, and VCE as its primary contract writing capabilities over the last decade. The Army is pursuing an alternative as these systems are reaching the end of service life.

In October 2011, DPAP announced SPS termination plans, directing that no new contracts would be input into SPS after 30 September 2015, and that all use of SPS would cease on 30 September 2017. The services were further directed to plan, budget for, and execute plans to support future contracting capabilities to replace SPS.

Also, in January 2012, through the Fiscal Year (FY) 2013 National Defense Authorization Act Section 862, “Uniform Contract Writing Requirements,” Congress directed the DoD to establish electronic contract writing systems that comply with uniform standards, requirements, rules, and controls.

ACWS Draft Test and Evaluation Master Plan 6

The Army conducted an Analysis of Alternatives (AoA) and Cost Benefit Analysis (CBA) in response to higher headquarters and congressional direction to determine a strategy for the identification of a single enterprise-wide contract writing and management system. This system would replace SPS, PADDS, and the VCE suite of tools in order to meet the Army's current and future critical functional requirements.

On 25 September 2014, the USD(AT&L) signed a memorandum that updated the sunset (termination) date for use of the SPS. The USD(AT&L) directed all components to cease awarding any contracts, agreements, or orders through SPS after 30 September 2018. Use of SPS, for all contracts issued prior to the 30 September 2018, is allowed to continue until 30 September 2020. USD(AT&L) confirmed the original direction from October 2011, for each Service to budget for and execute plans to deploy future contracting capabilities.

The ACWS solution will replace SPS upon its sunset and termination and will subsume PADDS and VCE capabilities in accordance with the approved program schedule. Not only will ACWS subsume and replace these systems, it will reduce the current systems’ operational, maintenance and support costs; reduce the number of interfaces; and improve the support to financial auditing capabilities. The effort will start with detailed assessment of the selected vendor solution including re-engineer business processes to gain optimal standardization resulting in streamlined end-to-end processes and operational functionality.

Overarching Strategies and System Procurement As reflected in Table 1: Evolutionary Acquisition Approach, ACWS will be acquired using an evolutionary (incremental) approach featuring software builds to adopt a COTS-based solution to achieve full capability. Market research indicates that competition exists in the commercial marketplace for multiple vendors with a variety of system solutions that can support the ACWS requirements. The overarching strategy is to acquire COTS system solution functionality to meet a significant number of the operational requirements out-of-the box (OOTB).

The Army will select a Systems Integrator (SI) to configure an OOTB solution to meet program requirements, to develop the interfaces, and to ensure the solution incorporates essential business processes, or accommodates changes resulting from Business Process Reengineering (BPR).

A single Request For Proposal (RFP) under Full and Open competition will be used to select a single SI for award of an Indefinite Delivery/Indefinite Quantity (ID/IQ) contract with Cost Plus and Fixed Price contract line items. The initial award (RFP minimum) will be for Task Order (TO) 001 for a negotiated value (based on funds availability). Follow-on TOs will only be awarded upon the successful progress of work and meeting the appropriate Milestone Decision criteria. The Army Contracting Command – Rock Island, in coordination with the ACWS PMO, will administer the contract.

ACWS Draft Test and Evaluation Master Plan 7

ACWS will be developed utilizing the COTS product solution that the SI recommends in its proposal, and the Government approves. The scope of the SI contract involves work from design through operations and sustainment for a ten-year ordering period.

The initial awarded TO will initiate the program management functions and perform the RR phase tasks to include Global Analysis, BPR, and blueprinting. The software solution capabilities will be delivered in a phased approach tailored to the SI’s solution.

Table 1: Evolutionary Acquisition Approach

Phase Activities Risk Reduction • Basic Functionality delivered OOTB to be accessed.

• Global Analysis (Requirements).

• Fit Gap Analysis (Requirements to System Capabilities).

• Business Process Reengineering.

Development and Deployment

• Basic contract writing & management capabilities, Installation contracting.

• Construction procurement, joint operations partners (Air Force financial system), grants, and cooperative agreements.

• Major weapon systems procurement and Foreign Military Sales.

• Disconnected State and Sensitive Activities procurement (including GFEBS-SA).

• Software will be produced by the SI and deployed to a DISA environment. DISA will provide hosting as a service instead of procuring hardware.

Operations & Support

• Composition of the Help Desk will shift from all contractors to a mix of contractor and government labor.

• Scope of on-site support will be reduced by relying on advance user groups within the commands.

Previous Testing There has been no testing conducted yet for ACWS.

1.3.3 Key Capabilities

The initial ACWS Key Performance Parameters (KPPs) are listed in Table 2: ACWS Key Performance Parameters. KPPs are included in the Army High Level Objectives (HLOs). The HLOs are aligned to Business Outcomes with defined measurements, and associated target threshold and objective values. The HLOs and Business Outcomes have agreed upon by the Functional Sponsor and the ACWS PMO. Critical Operational Issues (COIs), Critical Operational Issue Criteria (COIC), and Critical Technical Parameters (CTPs) will be developed during the RR phase.

ACWS Draft Test and Evaluation Master Plan 8

Table 2: ACWS Key Performance Parameters

Key Performance

Parameter Subfactors Threshold Objective Validation Method

Operational Availability

HLO-6

Restorability (Reliability)

Recovery Time Objective (RTO)

= 24 hours

Recovery Point Objective (RPO)

= 12 hours

RTO = 8 hours

RPO = 6 hours

Average amount of time it takes for the system to be restored from a catastrophic loss of data and system operation.

Availability 93.3% availability

(monthly average)

96.7% availability (monthly average)

Percentage of time that ACWS is operationally available (Uptime / (Uptime + Scheduled Downtime + Unscheduled Downtime)

Maintainability < 8 hours < 4 hours Mean Time to Repair

Net Ready (NR

KPP) HLO-5

Support to Military

Operations

95% Procurement

Data Standard (PDS) Validation

Success Rate

98% PDS

Validation

Success Rate

System generated PDS files will be validated as part of the NIPRNet transmission to the Global Exchange (GEX). Invalid PDS files will generate an error report. The number of error reports divided by the total number of awards finalized equals the success rate.

Condition for measurement is that the award was finalized in a NIPRNet connected environment.

Enter and be managed on the network

TBD (will be updated with the completion of the SV-7 architecture document with initial Information

Support Plan)

TBD (will be updated with the completion of the SV-7 architecture document with initial

Information Support Plan)

Threshold and Objective values and validation method will be updated with the completion of the SV-7 architecture document as submitted with the initial Information Support Plan (ISP).

Exchange Information 95% 98%

% of instruments that are successfully obligated and recorded in the financial system

Key Interfaces As depicted in Figure 3: ACWS Systems Interface Landscape, the ACWS solution will integrate with DoD initiatives, specifically those in the Army Contract Mission Area such as System for Award Management (SAM), and Global Exchange (GEX) Procurement Data and Standard Validation. ACWS will also integrate with other applications that are either currently interfaced with legacy contracting systems or will interface/integrate with ACWS. As the project works through the final Business Process Reengineering requirements, these interfaces will be further defined in coordination with the owners of the interfacing systems and formally documented.

ACWS Draft Test and Evaluation Master Plan 9

Figure 3: ACWS Systems Interface Landscape

System interfaces will be defined in the ACWS SV-6 in accordance with the Information Support Plan (ISP).

Special Test or Certification Requirements ACWS will use the DoD Risk Management Framework (RMF) as the primary Assessment and Authorization (A&A) vehicle with the Deputy Program Executive Officer Enterprise Information Systems (DPEO EIS) as the Authorization Official (AO). Risk Management for the program is based on the guidance found in DoDI 5000.02, Operation of the Defense Acquisition System, 7 January 2015, tailored to meet the needs of the ACWS acquisition process and implemented in accordance with the ACWS Risk Management Plan (RMP). The DoD RMF governance structure implements the three-tiered approach to cybersecurity risk management described in NIST SP 800-39, ACWS will conduct a STAR during the Material Solutions Analysis Phase, the results of which will serve as the threat baseline.

ACWS Draft Test and Evaluation Master Plan 10

The T&E program will verify and validate the cybersecurity requirements. The Security Control Assessor (SCA) will ensure Security Test and Evaluation (ST&E) procedures are completed in accordance with the RMF guidance prior to submitting an AO recommendation. The ACWS Cybersecurity Team will work with the SCA to prepare an A&A schedule to ensure RMF procedures are implemented and an accreditation decision is favorable prior to the deployment of the system.

Additionally, the ACWS Cybersecurity Team, in collaboration with the designated SCA and PEO EIS CIO Cybersecurity and Network Operations Division, will develop and coordinate an effective ST&E plan and schedule. Cybersecurity testing not only provides the data to determine compliance to security requirements but it is also the primary method of ensuring cybersecurity diligence throughout the engineering cycle.

The ACWS Cybersecurity testing methodology employs a “two-tier” test approach for each A&A test event. The team will conduct ST&E procedures to verify the system is configured in accordance with the requirements for a moderate to high impact level system. Once verification testing is completed, the SCA conducts the official certification testing to validate the configuration of the security controls in accordance with the RMF requirements. All security tests and A&A activities have been integrated into the ACWS test and evaluation planning, and incorporated into ACWS testing documentation.

The ACWS PdM, Systems Engineering Lead, ACWS Test Manager, Information System Security Manager (ISSM), and Information System Security Officer (ISSO) will coordinate activities to ensure that the program Integrated Master Schedule (IMS) includes ample time to conduct cybersecurity testing and A&A activities.

The PMO will update this section as additional Special Test and/or Certifications requirements are identified.

Systems Engineering Requirements The ACWS SEP contains the detailed engineering processes for the ACWS program.

The SEP focuses on the significance of well-formed requirements, ensuring testability, traceability, and allocation of requirements to solution components, and planning and performing strong baseline Configuration Management.

ACWS requirements will be traceable to the DoD objectives and HLOs, as defined in the ACWS Requirements Traceability Matrix (RTM). The HLOs are aligned to Business Outcomes with defined measurements, and associated target threshold and objective values. Reference Section 1.3.3 Key Capabilities.

In addition, the ACWS solution software and all ACWS generated data will be hosted by DISA, and as part of that implementation will comply with the OSD CIO Net-Centric Data Strategy.

Business outcomes will be further traced to business process descriptions that will be decomposed into system level requirements. During the requirements analysis phase, ACWS Draft Test and Evaluation Master Plan 11 the system level requirements will be further decomposed into technical requirements and specifications. This traceability ultimately aligns to use cases that will be used to test the system to ensure all requirements are successfully implemented, thus concluding with user acceptance and a deployment decision. The ACWS RTM contains this traceability. The RTM is approved by the Procurement Steering Governance Board (PSGB), which is chaired by DASA(P). The RTM will be managed via the ACWS Configuration Management processes and will be reviewed and updated as necessary at each technical review and program acquisition milestone.

The following systems engineering activities, events, and artifacts will be incorporated into the program IMS to ensure the conduct and completion of efforts aligned with overall program milestones, dependencies, and critical path:

• ACWS RR activities, to include BPR, OOTB Capability and Interface Verification, System Characterization, Interface Characterization, Technical Standards Characterization, Performance and Scalability Assessment, Human Systems Integration (HSI) Assessment, Sandbox/Development Environment Installation 1

• Systems engineering development phases: requirements analysis, design (functional/business and technical), build, and test

• Capability builds

• Technical Reviews (e.g., Preliminary Design Review (PDR), Critical Design

Review (CDR), Test Readiness Reviews (TRR))

• Systems Integration

• Test Events (e.g., Integrated DT/OT, IOT&E)

• System Certifications

• Systems Engineering Training activities

• Systems Engineering Deployment activities, such as site surveys and hosting

Technical risks will be managed as part of the ACWS risk management process. The ACWS PMO will manage program risks and mitigation, ensure significant risks are being captured and addressed in order to meet cost, schedule, and capability delivery objectives in accordance with the ACWS RMP.

ACWS Systems Engineering will focus on reliability growth through the reduction of system software defects over time. As a COTS-based system, the software is subjected to vendor release updates, service packs, and security updates associated with our maintenance contract.

The ACWS PdM, Systems Engineer, and ACWS Test Manager will identify and acquire required testing tools to support testing in developmental and operational testing. The

1 Reference the System Engineering Plan (SEP) for an Overview of the Global Analysis and Blueprinting (including OOTB assessment activities).

ACWS Draft Test and Evaluation Master Plan 12

2.0 Test Program Management and Schedule

2.1 T&E Management

ACWS teams will combine DT&E, OT&E, interoperability, and security testing whenever practicable. The ACWS PMO will consider incremental usability testing and innovative testing approaches, such as Agile-based iterative testing with continuous integration and Risk-Assessed Level of Test (RALOT) techniques. The final approved testing strategy and supporting plans will guide the T&E management activities. The ACWS continuous T&E strategy will evolve and mature the TEMP throughout the development process. Numerous organizations and individuals contribute to the success of T&E activities. These are described in the following subparagraphs and their overall responsibilities are summarized in Table 3: WIPT Members Roles and Responsibilities.

Responsibilities will continue to evolve throughout the program life cycle and will be reflected accordingly in future versions of the TEMP.

2.1.1 Role of the Project Management Office

The ACWS PdM has overall responsibility for establishing an effective T&E program for ACWS. The T&E program planning, preparation, execution, management, and reporting activities are delegated to the Deputy PdM.

The ACWS PMO will provide specific, detailed contractor T&E roles following ATP-1 approval and authorization to pursue the recommended acquisition approach, and update this section in subsequent versions of the TEMP.

2.1.2 Role of ACWS Test & Evaluation Section

The ACWS Test Manager is the Chief Developmental Tester in accordance with DoDI

5000.02 and 10 USC 139b and 1706. The ACWS T&E Section is responsible for managing and coordinating all T&E activities, and providing technical expertise on testing and evaluation issues and in reaching technically informed, objective judgments about contractor DT&E results for ACWS.

2.1.3 Role of the System Integrator

The SI is responsible for managing, planning, preparing for, executing, and reporting on DT&E events and supporting the government throughout other T&E events as required.

The SI develops the verification and validation plan for all system models and simulations.

2.1.4 Role of Army Test & Evaluation Command (ATEC)

ATEC is the Developmental and Operational Test Agency (OTA) for the ACWS program. The ACWS Operational Assessment will be conducted by ATEC in accordance with ATEC Regulation 73-1 to determine the operational effectiveness, suitability, and survivability of the system.

ACWS Draft Test and Evaluation Master Plan 13

ATEC prepares the RALOT for OT&E. ATEC prepares the System Evaluation Plan.

The System Evaluation Plan serves as the directive for test planning that will support ATEC evaluation. If required, ATEC develops the accreditation plan for all system models and simulations, and provides the accreditation recommendation for approval.

ATEC monitors key tests and prepares assessment and evaluation reports to support Milestone Decision Reviews (MDRs) and Materiel Release. ATEC is a principal T&E Integrated Product Team (IPT) member.

2.1.5 Joint Interoperability Test Command (JITC)

JITC supports ACWS in the area of joint interoperability assessment(s), evaluation and certification. JITC will observe all joint information exchanges in accordance with the ACWS PMO Department of Defense Architecture Framework (DODAF) architecture products and the Joint Staff J6/DoD CIO Certified Net Ready Key Performance Parameter (NR KPP) attributes as depicted in the ACWS PMO Information Support Plan (ISP).

Interoperability assessment(s)/evaluation(s) will be performed, as appropriate, in parallel with designated test events to support the ACWS Acquisition and T&E strategy.

2.1.6 Under Secretary of Defense of Acquisition, Technology, and Logistics

(USD(AT&L))

USD(AT&L) is the ACWS MDA. USD(AT&L) provides the final approval and authority on all ACWS acquisition-related decisions.

2.1.7 Director, Operational Test & Evaluation (DOT&E)

DOT&E is an approving authority for the TEMP and Operational Test and Evaluation (OT&E) plans. DOT&E will provide an independent assessment of operational effectiveness, suitability, and survivability of ACWS, and report the results to the MDA, Overarching Integrated Product Team leader (OIPT), and Congress.

2.1.8 Deputy Assistant Secretary of Defense for Developmental Test & Evaluation

(DASD(DT&E))

The DASD(DT&E) provides guidance developing and documenting the program evaluation strategy and management approach throughout the program lifecycle. The DASD(DT&E) is the approving authority for the DT&E information contained in the

TEMP.

2.1.9 Deputy, Under Secretary of the Army – Test & Evaluation (DUSA–TE) DUSA-TE ensures final coordination of the TEMP with Headquarters, Department of the Army (HQDA) staff elements and coordinates the TEMP with DASD(DT&E) and

DOT&E.

ACWS Draft Test and Evaluation Master Plan 14

2.1.10 Army Chief Information Officer (CIO)/G6 – Network Enterprise Technology Command (NETCOM)

The Army CIO/G-6 will provide policy oversight and program direction to the U.S. Army NETCOM/9th Army Signal Command. CIO/G6 will observe information exchanges and collaborate with JITC to provide an intra-operability certification determination.

2.1.11 Army Agent of the Security Control Assessor (ACA)

A specific organization has not yet been designated as the ACA for ACWS. ACA will perform testing to determine system compliance with RMF mandated security controls.

The ACWS PMO will update this section in subsequent versions of the TEMP.

2.1.12 Army Research Laboratory/Human Research and Engineering Directorate

(ARL/HRED)

ARL/HRED will conduct ACWS Human Systems Integration (HSI) usability testing and risk management as part of the HSI Plan and HSI Assessment in support of the ACWS Product Manager and Army G-1 in accordance with DoDI 5000.02 and AR 602.

ARL/Survivability/Lethality Analysis Directorate (SLAD) will perform Cooperative Vulnerability and Penetration Assessment (CVPA) and Adversarial Assessment.

2.1.13 Army Audit Agency (AAA)

ACWS is a financial feeder system and is required to be Federal Financial Management Improvement Act (FFMIA) compliant. ACWS does not have a requirement for AAA to perform the FFMIA Compliance Examination Attestation.

2.1.14 Functional Sponsor

ODASA(P) is the Army’s Executive Sponsor and Functional Proponent for ACWS.

System owners will provide participants to support ACWS PMO T&E activities including identification of interfaces; insight and requirements for data conversions and data interpretation; and developing and verifying operational scenarios, test scripts, and use cases for test events.

2.1.15 PEO EIS

PEO EIS provides program management oversight to ACWS PMO, and exercises responsibilities for the technical, lifecycle, and business management aspects of ACWS development.

PEO EIS provides Independent Verification and Validation (IV&V) support. The IV&V process provides independent assessments of the system documents, plans, processes, Configuration Items (CIs), and T&E activities.

ACWS Draft Test and Evaluation Master Plan 15

PEO EIS will provide system security accreditation as the Designated Approving Authority (DAA).

2.1.16 T&E Organizational Construct

The ACWS WIPT Charter describes a T&E Working Integrated Product Team (WIPT) approach to coordinate the testing of the ACWS solution. The ACWS Test Manager will chair the T&E WIPT and will be responsible for the development and coordination of the T&E strategy and plans that will direct the T&E activities for ACWS. Table 3: WIPT Members Roles and Responsibilities illustrates roles and responsibilities as currently understood (pre-ATP-1).

Table 3: WIPT Members Roles and Responsibilities

WIPT

Member Roles & Responsibility

ACWS Deputy PdM

• Performs overall T&E program management

• Reports directly to the ACWS PdM

ACWS T&E

Section

• Coordinates, manages and oversee all DT&E activities

• Coordinates and publishes test strategies and plans

• Maintains insight into SI/Contractor activities under the program and oversee the T&E activities of other participating government agencies under the program

• Provides technical engineers and Subject Matter Experts (SMEs) to support T&E activities

• Assists the PdM / Deputy PdM in making technically informed, objective judgments about

SI/contractor DT&E efforts System

Integrator

• Manages, plans, prepares for, executes, and reports on DT events

• Supports the government throughout other T&E events as required.

• Develops the verification and validation plan for all system models and simulations.

• Supports problem resolution/disposition during testing activities

• Provides software expertise, guidance, maintenance, troubleshooting, and application system support through DT&E, and as required in OT

• Participates in all WIPT activities as needed

ATEC • Serves as the ACWS independent OTA

• Prepares relevant portions of the TEMP and test plans (e.g., System Evaluation Plan)

• Develops the accreditation plan for all system models and simulations, and provides the accreditation recommendation for approval (if required)

• Monitors key tests and prepares assessment and evaluation reports to support MDRs) and

Materiel Release

• Plans for and conducts ACWS OT

• Participates in all WIPT activities as a principal T&E IPT member

• Attends Test Readiness Reviews

JITC • Performs interoperability “evaluations” to support a Joint Interoperability decision for each ACWS Increment build

• Conducts Interoperability Assessments and provide the ACWS PMO periodic updates via a JITC IOP Assessment Quick Look Report to support the program as to where they currently stand in regards to preparation for an IOP evaluation/certification as requested by the ACWS PMO

• Provides recommendations to the DOD CIO and JCS working with the U.S. Army Interoperability Steering Group representative in support of the ACWS PMO Interim Certificate to Operate (ICTO) request processes if/as required

• Performs Standard Financial Information Structure (SFIS) and 5015.2 compliance testing

• Participates in all WIPT activities as needed

ACWS Draft Test and Evaluation Master Plan 16

WIPT

Member Roles & Responsibility

USD(AT&L) • Serves as the ACWS MDA

• Provides the final approval and authority on all ACWS acquisition-related decisions

DOT&E • Serves as approving authority for the TEMP and OT&E plans

• Provides an independent assessment of operational effectiveness, suitability, and survivability DASD(DT&E) • Provides guidance developing and documenting ACWS evaluation strategy and management approach throughout the lifecycle

• Serves as approving authority for the DT&E information contained in the TEMP

• Monitors and reviews all DT&E activities

• Participates in all WIPT activities

• Provides independent assessment of ACWS DT&E, execution and risk

DUSA-TE • Ensures final coordination of the TEMP with HQDA staff elements

• Coordinates TEMP with DASD(DT&E) and DOT&E

CIO/G6 –

NETCOM

• Coordinates with the PMO to obtain any JITC Interoperability assessment(s), as appropriate, as a result of each build;

• Develops Memorandum of Agreement (MOA) between CIO/G6, ATEC, and PdM ACWS to obtain sufficient interoperability data in order to make a proper Army Interoperability Certification (AIC) conclusion

• Participates in all WIPT activities as needed ACA • Performs validation testing to determine the system’s compliance with the RMF mandated cybersecurity controls

• Performs RMF risk assessment for certification recommendation

• Participates in all WIPT activities as needed

ARL/HRED &

SLAD

• Conducts ACWS HSI RR assessment in support of the ACWS Product Manager

• Performs CVPA and Adversarial Assessment

• Participates in all WIPT activities as needed

AAA • Performs the FFMIA Compliance Examination Attestation if required

• Participates in all WIPT activities as needed

Functional Sponsor

• Coordinates internally and externally with ACWS PMO to integrate functional SMEs with ACWS testing

• Ensures functional representation in ACWS PMO T&E WIPT

• Elevates functional issues to appropriate levels for resolution

• Participates in applicable program control boards, technical reviews, and internal/external meetings related to ACWS testing

• Identifies interfaces.

• Provides insight and requirements for data conversions and data interpretation.

• Assists in developing and verifying operational scenarios, test scripts, and use cases.

• Assists in the evaluation of test results

• Coordinates for SMEs and representation as required

• Participates in test events as required

• Participates in all WIPT activities

PEO EIS • Provides system security accreditation as the DAA

• Concurs with TEMP

• Provides IV&V in support of ACWS PdM

The ACWS PMO, Systems Engineering, T&E leadership team, and participating organizations will further refine specific roles and responsibilities for testing stakeholders in subsequent versions of the TEMP.

ACWS Draft Test and Evaluation Master Plan 17

2.2 Common T&E Database Requirements

Prior to ATP-1, the ACWS PMO and all WIPT members will define testing processes and select common testing toolsets for collecting, validating, and sharing T&E data among testing stakeholders. The PMO will utilize the ACWS SharePoint Site, as much as possible, to share test data, artifacts, and other information available to all T&E WIPT members.

One common T&E database will be maintained for the SI’s database.

Once defined, the ACWS PMO will update this section to include the testing processes and common toolsets.

2.3 Deficiency Reporting

Upon attaining ATP-1, the ACWS PMO will define the processes and tools for documenting and tracking deficiencies, in accordance with applicable statues, regulations, and directives. The SI and ACWS PMO will be responsible for maintaining the designated test tools repositories to track deficiencies as they are generated.

One deficiency reporting system will be maintained for the program.

Once defined, the ACWS PMO will update this section to include the processes and tools for documenting and tracking deficiencies.

2.4 TEMP Updates

This ACWS draft TEMP describes the total T&E planning from system development through IOT&E. This includes the required type and number of T&E events along with resource requirements.

The ACWS PMO will update the TEMP whenever changes to the operating environment(s) for test strategy, test planning, and testing, or the T&E stakeholders change. In accordance with DoDI 5000.02, Table 2: ACWS Key Performance Parameters, the PMO will update the TEMP as appropriate throughout the product development lifecycle.

2.4.1 T&E TEMP Sub Working Group

The T&E TEMP Sub Working Group (SWG) is responsible for the document’s structure and content, and for exercising editorial control over any proposed document changes.

The TEMP SWG exercises a change management regiment to insure that any proposed changes are adequately coordinated amongst the TEMP SWG membership.

The TEMP SWG Lead is responsible for (i) facilitating the appropriate coordination of any proposed change, (ii) the final editing of any TEMP SWG work products before they are presented to the T&E WIPT, (iii) insuring that any TEMP updates are made in a manner appropriate to support key program milestones, and (iv) providing status updates. The T&E TEMP SWG members are responsible for supporting the TEMP

ACWS Draft Test and Evaluation Master Plan 18

SWG Lead in the timely exercise of his/her responsibilities. The T&E WIPT will approve any updates and changes prior to formal staffing of the TEMP. The T&E TEMP SWG operates at the bequest of the ACWS Test Manager.

2.5 Integrated Test Program Schedule

The ACWS Notional Program Schedule is shown in Figure 4: ACWS Notional Integrated Program Schedule and details the Notional Integrated Test Schedule. It includes all ACWS planned T&E events.

The detailed build specific Integrated Test Program schedules will be included as annexes in future updates to the TEMP.

Figure 4: ACWS Notional Integrated Program Schedule

2.5.1 Schedule Concurrency Risk Mitigation

The notional schedule includes periods of concurrent build. Assuming that the final build schedule agreed upon by the ACWS PMO and the SI includes concurrent builds, the ACWS PMO has already identified mitigating steps to reduce risk associated with the concurrent build schedule.

Potential PMO mitigating actions include government blueprinting. Business Process Re-engineering, and demonstrations. The combination of the activities will allow the

ACWS Draft Test and Evaluation Master Plan 19 government to assess adequately the technical maturity of the system prior to decisions that will precede later builds. Additionally, the actions will allow the government to evaluate the technical performance of the SI before awarding the subsequent Option.

At contract award, the government will present the SI with artifacts that will reduce the time necessary to conduct blueprinting. This will allow the SI additional time and resources to assess fully the government’s intent, make recommendations, and develop a complete and comprehensive design package.

The ACWS PMO intends to schedule sufficient time between events to incorporate software changes made in response to test results and subsequent test verify the software updates.

3.0 Test and Evaluation Strategy

3.1 T&E Strategy

As described in section 1.3.2.2 Program Background, the ACWS PMO will pursue a COTS-based solution with minimal development to achieve full capability. The overarching strategy is to acquire a COTS system solution that meets a significant number of the operational requirements OOTB. The Army will select a SI to configure the solution and to develop any remaining capabilities (e.g., interfaces) and ensure the solution incorporates essential business processes and accommodates changes resulting from Business Process Reengineering.

As depicted in Figure 5: ACWS High-Level T&E Strategy, the ACWS T&E Strategy uses an adaptive, integrated testing methodology2 that supports a wide range of possible delivery approaches by the selected SI. This T&E strategy provides for dynamic and flexible methods that easily incorporate innovative acquisition and developmental approaches that the SI may propose.

The SI, in coordination with the PMO, will plan and conduct developmental testing. The PMO will review, and as appropriate, approve SI testing artifacts such as test plans, test descriptions, and test reports. The PMO and ATEC will provide testing oversight, witness test events, and collect appropriate test data to support follow-on test events.

The SI will have the latitude to present innovative approaches for a timely and cost effective developmental test program. In accordance with Contract Data Requirements

2 Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation and reporting by all stakeholders particularly the developmental (both contractor and government) and operational test and evaluation communities. Reference SOO Section 6.4.

ACWS Draft Test and Evaluation Master Plan 20

List C0006, the SI will deliver a Coordinated Test Plan. The Coordinated Test Plan will be used as a data source whenever the TEMP is updated.

Figure 5: ACWS High-Level T&E Strategy

ATEC is the Lead OTA and provides data collection and evaluation services in all phases of testing to support preparation for successful operational testing.

The ACWS T&E team envisions a test strategy that will test and evaluate the ACWS solution in both developmental and operational test environments and will culminate with a formal IOT&E. The SI will provide assistance, as necessary, with testing.

Each Build will undergo a Limited User Test (LUT) and its results will be detailed in an Abbreviated Operational Evaluation Report (AOER) to support the Limited Deployment Decision (LDD) for that Build. An IOT&E for all Builds will be conducted to support the Full Deployment Decision (FDD). The Operational Evaluation Report (OER) will provide the data for the FDD.

Special tests such as certification and accreditation activities, will be conducted throughout the T&E phase, and may include Security Test & Evaluation, Usability Testing, Joint Interoperability Test observations, and FFMIA compliance. The ACWS

ACWS Draft Test and Evaluation Master Plan 21 strategy also dictates that some developmental and all operational testing will be performed in “production-like” environments under configuration management control.

The ACWS PMO and T&E Stakeholders will update the content for this section during the RR phase, and update as required.

3.2 Evaluation Framework

The ACWS RTM defines the ACWS requirements and reflects the critical components of the ACWS operational characteristics. The ACWS PMO will work with the Functional Sponsor and the SI to determine which requirements will be met during which build, and the Evaluation Framework Matrices will be developed and provided in a future Annexes to this document.

The initial ACWS KPPs are listed in Table 2: ACWS Key Performance Parameters.

KPPs are included in the HLOs. ACWS will be using the HLOs and the other software metrics defined in the SEP, to analyze performance and apply corrective actions throughout the development process.

Critical Operational Issues (COIs), Critical Operational Issue Criteria (COIC), and Critical Technical Parameters (CTPs) will be developed. CTPs define measurable critical system characteristics that, if not achieved, preclude fulfilling desired operational performance capabilities. CTPs do not represent user requirements but serve as a means to test user requirements; therefore, the PMO will derive these technical measures from desired user capabilities. COIs, COICs, and CTPs will be developed during the RR phase.

Cybersecurity will be tested throughout the product lifecycle; reference Table 4: ACWS Survivability Strategy. Cybersecurity will be evaluated in accordance with DOT&E Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs, dated 1 August 2014, and the DOT&E memorandum titled Cyber Economic Vulnerability Assessments (CEVA), dated 21 January 2015, and the DoD Cybersecurity Test and Evaluation Guidebook, dated 1 July 2015.

Table 4: ACWS Survivability Strategy

Event Description Risk Reduction Phase, Paper Study

ARL/SLAD to conduct systems engineering support to identify potential cyber security capabilities and limitations of the proposed materiel solution(s). ARL/SLAD will review application specific security controls and configuration options. Cyber test data from existing programs will be reviewed to identify potential problem areas. Open source research will be conducted to identify known risks and existing vulnerabilities. The study will provide application specific cyber security capabilities, limitations, and recommend an outline for future test and evaluation and identify risk areas. Occurs once as part of the pre-award source selection process.

ACWS Draft Test and Evaluation Master Plan 22

Event Description Pre PDR, Cyber Baseline Assessment

ARL/SLAD to conduct a baseline Network Vulnerability Assessment (NVA) to identify potential security vulnerabilities associated with the application configuration, user accounts, security controls, etc.