2018-08-03 E-IDS_DraftSIR_Attachment_J.5_TEMP.pdf

ATM-EIDS-A3-TMP-001 1 December 11, 2017 This Document Becomes Uncontrolled Once Printed.

preliminary Test and Evaluation Master Plan (pTEMP) For

Enterprise Information Display System (E-IDS)

FAA WILLIAM J. HUGHES TECHNICAL CENTER

ATLANTIC CITY INTERNATIONAL AIRPORT, NEW JERSEY 08405

DOCUMENT # ATM-EIDS-A3-TMP-001

VERSION # Version 1.0

VERSION DATE December 11, 2017

ATM-EIDS-A3-TMP-001 2 December 11, 2017

Document Version Control

VERSION DESCRIPTION OF CHANGE DATE

0.1 Initial draft of pTEMP 5/5/2017

0.2 Accepted Changes to initial draft 0.1 added comments. 5/30/2017

0.3 Included J. Potter’s comments 6/2/2017

0.4 Included T. Fisher’s Comments 6/5/2017

0.5 Adjudicated J. Potter and T. Fisher’s comments 6/7/2017

0.6 Rough draft complete with VRTM 7/7/2016

0.7 Included Doug’s comments, changes DT figure 8/1/2017

0.8 Included Ken’s comments Changed figures and tables 8/15/2017

0.9 Included new figure 3-2 and accepted changes 8/21/2017

0.10 Included comments from internal PR 8/29/2017

0.11 Added CPRs from iPRD and TSB comments 9/13/2017

0.12 Added PO comments 10/30/2017

0.13 Added PO comments 11/5/2017

0.14 Added new T&E activities diagram on verbiage 11/15/2017

0.15 Added TSB comments 12/11/2017

1.0 Release Version 1.0 12/11/2017

ATM-EIDS-A3-TMP-001 3 December 11, 2017

ATM-EIDS-A3-TMP-001 4 December 11, 2017

TABLE OF CONTENTS

1 PROCESS OVERVIEW

1.1 BACKGROUND

1.2 PURPOSE

1.3 SCOPE

1.4 MANAGEMENT OF THE TEMP

2 REFERENCE DOCUMENTS

2.1 FAA DOCUMENTS

2.2 OTHER GOVERNMENT DOCUMENTS

2.3 NON-GOVERNMENT DOCUMENTS

3 DESCRIPTION

3.1 OVERVIEW

3.2 INTERFACES OVERVIEW

3.3 CRITICAL PERFORMANCE REQUIREMENTS (CPRS)

3.4 CRITICAL OPERATIONAL ISSUES (COIS)

4 E-IDS TEST PROGRAM DESCRIPTION

4.1 PREVIOUS RESULTS

4.2 DEVELOPMENT TESTING (DT)

4.2.1 Test Activities

4.2.2 DT Entrance Criteria

4.2.3 Dt Exit Criteria

4.2.4 Test Activity: DT Software Testing

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.5 Test Activity: DT Hardware Testing

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria

ATM-EIDS-A3-TMP-001 5 December 11, 2017

Test Limitations

4.2.6 Test Activity: DT Factory Acceptance Test

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.7 Test Activity: DT Installation and Integration Testing

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.8 Test Activity: DT System Testing

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.9 Test Activity: DT Production Acceptance Testing

Test Objectives Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.10 Test Activity: DT Site Acceptance Testing

Test Objectives

ATM-EIDS-A3-TMP-001 6 December 11, 2017

Test Approach Test Location Test Configuration Test and Analysis Tools Test Entrance Criteria Test Exit Criteria Test Limitations

4.2.11 Test Activity: DT Regression Testing

Test Approach

4.2.12 Test Limitations

4.3 OPERATIONAL TESTING (OT)

4.3.1 ATC Operations Test Group

NAS Integration Test Activity Objectives Test Approach

NAS Facility Operations Test Activity Objectives Test Approach Non-FAA Users Test Activity Objectives

Test Approach E-IDS Resiliency Test Activity Objectives

Test Approach E-IDS Performance Test Activity Objectives

Test Approach

4.3.2 Technical Operations Test Group

System Maintenance Operations Test Activity Objectives Test Approach

Degraded Operations Test Activity Objectives Test Approach

Failure & Recovery Test Activity Objectives Test Approach

Transition & Fallback Test Activity Objectives Test Approach

4.3.3 Support Operations

Remote Maintenance Operations Test Activity Objectives

Test Approach Data Administration Test Activity Objectives

Test Approach

4.3.4 Specialty Tests Group

Training Test Activity Objectives Test Approach

ATM-EIDS-A3-TMP-001 7 December 11, 2017

Human Factors & Safety Test Activity Objectives Test Approach

Security Test Activity Objectives Test Approach

Reliability, Maintainability & Availability Test Approach

4.3.5 Test Location

4.3.6 Test Configuration

4.3.7 Test and Analysis Tools

4.3.8 OT Entrance Criteria

4.3.9 OT Exit Criteria

4.3.10 OT Regression Testing

4.3.11 Test Limitations

4.4 FIELD FAMILIARIZATION

4.4.1 Field Familiarization Objective

4.4.2 Field Familiarization Approach

5 TEST MANAGEMENT

5.1 TEST MANAGEMENT ORGANIZATION

5.1.1 Test Program Roles and Responsibilities

5.1.2 Other Participating Organizations

5.1.3 Integrated Test Team (ITT)

5.1.4 DT Test Working Group

5.1.5 OT Test Working Group

5.2 TRAINING

5.2.1 Test Developer Training

5.2.2 Test Participant Training

5.2.3 Special Training Requirements

5.3 E-IDS CONFIGURATION MANAGEMENT (CM)

5.4 TEST DOCUMENTATION

5.4.1 Contractor T&E Documentation

5.4.2 FAA T&E Documentation

5.5 REQUIRED T&E TECHNICAL INTERCHANGE MEETINGS (TIMS) AND REVIEWS

5.6 T&E STATUS REPORTING AND TRACKING PROCESSES

5.7 DISCREPANCY REPORTS (DRS)/PROGRAM TROUBLE REPORTS (PTRS)

5.8 PERSONNEL RESOURCE REQUIREMENTS

5.9 TEST CAPABILITY SUMMARY AND ACCREDITATION

5.10 TEST NCP REQUIREMENTS

ATM-EIDS-A3-TMP-001 8 December 11, 2017

5.11 INTEGRATED TEST SCHEDULE

5.12 T&E FUNDING

5.13 TEST PROGRAM RISKS AND LIMITATIONS

6 TRANSITION OF THE TEST PROGRAM TO IN-SERVICE MANAGEMENT

(ISM)

7 ACRONYMS

8 TEST TERMS AND DEFINITIONS

APPENDIX A. COI DECOMPOSITION

APPENDIX B. TEST PHASE MATRIX

LIST OF FIGURES

FIGURE 3-1 CURRENT OPERATIONAL ENVIRONMENT

FIGURE 3-2 E-IDS OPERATIONAL CONCEPT

FIGURE 3-3 E-IDS ALTERNATIVES

FIGURE 4-1 E-IDS TEST ACTIVITIES

FIGURE 4-2 E-IDS OT PROCESS FLOW

FIGURE 4-3 E-IDS OT ACTIVITIES

FIGURE 5-1 E-IDS T&E TEAM STRUCTURE

LIST OF TABLES

TABLE 3-1 CRITICAL PERFORMANCE REQUIREMENTS 15

TABLE 3-2 CRITICAL OPERATIONAL ISSUES 16

TABLE 5-1 TEST SCHEDULE 41

ATM-EIDS-A3-TMP-001 9 December 11, 2017

1 PROCESS OVERVIEW

1.1 BACKGROUND

This Enterprise Information Display System (E-IDS) preliminary Test and Evaluation Master Plan (pTEMP) defines the test strategy and scope. It is developed based upon the concepts and functions documented in the initial Program Requirements Document (iPRD) prior to Initial Investment Decision (IID). E-IDS is an Acquisition Category 1 new investment system, consisting of Commercial Off-the-Shelf (COTS) hardware/software and newly developed software. E-IDS is not designated for Independent Operational Assessment (IOA). This E-IDS pTEMP was developed using the instructions and structure provided in the Test & Evaluation (T&E) Handbook, and the T&E TEMP Template for New Investments. Reference documents are listed in Section 0.

1.2 PURPOSE

This TEMP describes the E-IDS test program, provides a basis to verify and validate the system under test, ensures E-IDS meets the program requirements specified in both the E-IDS PRD and E-IDS System Specification Document (SSD), and ensures the system is operationally suitable and effective.

1.3 SCOPE

This TEMP addresses the E-IDS Development Testing (DT), Operational Testing (OT), Site Acceptance Testing (SAT), and Field Familiarization (FF) covering the Solution Implementation (SI) phase of the acquisition lifecycle. Test activities are traceable to the program requirements.

The TEMP identifies Critical Performance Requirements (CPRs), Critical Operational Issues (COIs), and operational concepts and procedures. The test strategies described in the TEMP are consistent with the Implementation Strategy and Planning Document (ISPD), but contain more refined details which support the higher-level strategies in the ISPD.

1.4 MANAGEMENT OF THE TEMP

This TEMP is a living document and was developed by and will be maintained by the Air Traffic Management (ATM) Branch (ANG-E59), with input from the E-IDS Integrated Test Team (ITT) (see Section 5.1.3) and the E-IDS Program Office. This document will be updated by ANG-E59 to include revisions and additional details throughout the program and is revised during this timeframe to reflect any major changes in the test scope or overall test strategy.

ATM-EIDS-A3-TMP-001 10 December 11, 2017

2 REFERENCE DOCUMENTS

2.1 FAA DOCUMENTS

a) Test and Evaluation (T&E) Handbook VVSPT-A2-PDD-013 Version 4.0 May 16, 2017

b) Test and Evaluation Master Plan Template for New Investments Version 5.0, July 08,

c) initial Program Requirements Document for Enterprise Information Display System (E-

IDS), TBD

d) ATM Branch Document Management and Control (DMC) Plan, Version 3.0, September 8, 2017

e) Concept of Operations, (ConOps) for Enterprise Information Display System, (E-IDS):

Version 2, September 29, 2017

f) Enterprise Information Display System (E-IDS) Functional Analysis, Final version, TBD

g) Implementation Strategy and Planning Document (ISPD), TBD

h) initial Program Sponsor Configuration Management Plan (iPSCMP), Version 1.0, TBD

2.2 OTHER GOVERNMENT DOCUMENTS

None

2.3 NON-GOVERNMENT DOCUMENTS

None

3 DESCRIPTION

3.1 OVERVIEW

E-IDS will be designed to address the IDS shortfalls outlined in three problem areas described in the E-IDS ConOps document:

1. Inefficiencies of sustainment and replacement activities

2. Limited ability to support continuity of operations and capitalize on NextGen capabilities

3. Inefficiencies of Data Access and Data Management The current operational environment is captured in Figure 3-1 showing multiple legacy versions of IDSs. The Enterprise Information Display System (E-IDS) ConOps focuses on the various system design benefits from an operational and costing perspective by replacing multiple legacy versions of IDSs as shown, in the second layer, with one enterprise infrastructure.

ATM-EIDS-A3-TMP-001 11 December 11, 2017

Figure 3-1 Current Operational Environment

E-IDS is a National Air Space (NAS) enterprise solution that will replace the legacy standalone Information Display System (IDSs) with a common NAS-wide platform capability. E-IDS will be integrated with the NAS network infrastructure and Service Oriented Architecture (SOA)-based System Wide Information Management (SWIM) infrastructure using modern data formats and communication systems. The E-IDS displays are separate from the controllers’ primary displays, and their purpose is to provide supplemental operationally essential information for the Certified Professional Controllers (CPCs) controlling aircraft. Other operators who will rely on the E-IDSs include: Front Line Managers (FLMs), Traffic Management Coordinators (TMCs), and Technical Operations (Tech Ops) personnel. System development will occur in segments with new capabilities including resiliency, correlation of operational data sets, data archiving, and reporting. In its initial segments, E-IDS will interface with a number of FAA Next Generation Air Transportation System (NextGen) Enterprise Services via SWIM including Aeronautical Information Management Modernization (AIMM), Common Support Services Weather (CSS-Wx), and TFMData Flow Service. E-IDS is expected to be deployed to the Air Route Traffic Control Centers (ARTCCs), Terminal Radar Approach Control facilities (TRACONs), Air Traffic Control Towers (ATCTs), Combined Center Radar Approach Control facilities (CERAPs), Air Traffic Control System Command Center (ATCSCC), and the Alaska Flight Service Stations (FSS). Additionally, E-IDS will be installed at the William J. Hughes Technical Center (WJHTC) and the Mike Monroney Aeronautical Center (MMAC) for testing and training purposes. E-IDS will provide NextGen capabilities to En Route, Terminal and Oceanic domains. The operational concept for E-IDS is captured in Figure 3-2 and detailed in the E-IDS ConOps document.

ATM-EIDS-A3-TMP-001 12 December 11, 2017

Figure 3-2 E-IDS Operational Concept

E-IDS will provide integrated information from different sources and facilitate the organization and accessibility of the different types of information. This information will include: (a) real-time weather, Notices to Airmen (NOTAM), Pilot Reports, and Special Activity Airspace (SAA) schedules and status, (b) 56-day static digital information (e.g. charts, approach plates, etc.), and

(c) administrative information.

Figure 3-3 E-IDS shows the three alternatives for E-IDS, an integrated system with a common enterprise-based National Data Center to collect, store, update, and provide information to thousands of client displays in the field rather than require that each IDS collect, store and update its own information. Upon a decision from the Program Office on which alternative is chosen, the TEMP will be updated with a hardware and software diagram.

E-IDS is expected to be deployed in accordance with the Program Office segmentation plan, with emphasis first on replacing legacy IDSs prior to their end-of-life dates, and next on evolving the E-IDS to support and integrate with existing NextGen capabilities as well as new capabilities as they become available.

ATM-EIDS-A3-TMP-001 13 December 11, 2017

Figure 3-3 E-IDS Alternatives

3.2 INTERFACES OVERVIEW

E-IDS consists of a number of new functional services, interfaces to FAA and non-FAA systems, and displays to air traffic controllers, tower personnel, traffic management coordinators, and ATCSCC, ARTCCs, TRACON, Towers, CERAPs and the FSS personnel. The system interface connectivity will be further detailed in the Lab Implementation Plan.

The following list identifies the E-IDS system interfaces:

SWIM-Enabled Information Services:

• AIMM Aeronautical Common Services (ACS) – Provides a common, integrated, and authoritative view of aeronautical information, such as information associated with Special Activity Airspace (SAA) and NOTAMs.

• CSS-Wx – Common Support Services-Weather- Provides both standardized weather information products (e.g., current weather conditions, weather forecasts) and integrated weather information products (e.g., weather over charts) derived from the NextGen Weather Process (NWP), the National Oceanic and Atmospheric Administration 4-Dimensional (4-D)

ATM-EIDS-A3-TMP-001 14 December 11, 2017

Weather Cube, and other weather sources. E-IDS will send Hazardous Weather advisories to CSS-Wx

• TFMData (Flow) – Provides Traffic Flow Management (TFM) Flow data including Traffic Management Initiatives (TMI).

• TBFM-MIS – Provides metering information (e.g., when metering is in effect) from TBFM at each ARTCC to SWIM in support of NextGen objectives.

• TFDM Terminal Publication (TTP) – Provides airport configuration information (e.g., current and scheduled arrival and departure runways and rates) with airport demand and delay information.

• NAS Common Reference (NCR) – Enterprise-level exchange service for NAS status information that integrates dynamic NAS status and constraint information from different sources (e.g., NOTAMs, SAAs, ARTCC boundaries, airports, Traffic Management Initiatives (TMIs), CSS-Wx) to provide a consistent view of the information that is correlated spatially and temporally.

• FAA Web Services– Allow access to up-to-date information that is available through secure FAA web services (e.g., FAA publications or national or local orders via the Mission Support Network).

• Other – To indicate the ability to add additional interfaces.

• SWIM Terminal Data Distribution System (STDDS) – Receives Runway Visual Range (RVR) data from E-IDS.

FTI Service:

• Network Time Protocol (NTP) – IP-network-based distributed networking protocol for clock synchronization between computer systems.

Direct-Access Interfaces:

• Special Use Airspace Management System (SAMS) – E-IDS will provide terminal SAA status information to SAMS.

• Non-FAA Users:

o DOD o Legacy IDSs (NIDS, IDS-5, IDS-4, ACE-IDS)

• Weather and Surface Sensors:

o Automation of Field Operations and Services (AFOS) o Automated Surface Observing Systems (ASOS) o Advanced Technologies and Oceanic Procedures (ATOP) o Advanced Weather Information Processing System (AWIPS) o Automated Weather Observing System (AWOS)-C o Ribbon Display Terminal (RBDT) o Surface Weather System (SWS)

• Runway Visual Range (RVR) – Provides real-time visibility and trend data.

• Flight Data Input/Output (FDIO) – Provides General Information (GI) messages.

ATM-EIDS-A3-TMP-001 15 December 11, 2017

3.3 CRITICAL PERFORMANCE REQUIREMENTS (CPRS)

CPRs are program requirements deemed essential to meeting the E-IDS mission needs. CPRs are TBD by the Program Office.

Table 3-1 Critical Performance Requirements

Critical Performance Requirements

No. Requirement Title Requirement CPR-1 Display of operational data within receipt within the E-IDS boundary (95th percentile) 3 seconds

CPR-2 Maximum connected users 5,500 users

CPR-3 Simultaneous user requests 500

CPR-4 System computer processor CPU utilization must not exceed 30%, averaged daily

MAX - TBD

CPR-5 System processor memory utilization must not exceed 50%, averaged daily

MAX - TBD

CPR-6 Bit error rate for transmission of data 10E-06

CPR-7 System availability for essential services 0.999

CPR-8 Mean Time to Restore (MTTR) essential services following detected failure

10 minutes

3.4 CRITICAL OPERATIONAL ISSUES (COIS)

COIs are key questions about the operational use of the capabilities of the E-IDS. They are evaluated throughout the test and evaluation program to assess the operational suitability and effectiveness of the system. The COIs for the E-IDS program are derived from the program’s requirements documentation (PRD) and serve as the basis for OT to determine the system’s overall capability to support mission accomplishment. The COIs for the E-IDS program are as follows:

ATM-EIDS-A3-TMP-001 16 December 11, 2017

Table 3-2 Critical Operational Issues

Critical Operational Issues COI 1 Does E-IDS design support safe and effective operations for the users while performing their overall duties?

COI 2 Does the E-IDS provide Computer Human Interface (CHI) that supports safe and effective operation for the users?

COI 3 Does E-IDS provide accurate and timely information while under load to meet operational needs?

COI 4 Can the E-IDS interface and operate effectively with other National Airspace

System (NAS) equipment and systems without degrading NAS performance?

COI 5 Can E-IDS be maintained without degradation or unintentional disruption to on-going operations?

COI 6 Is the Reliability, Maintainability and Availability (RMA) of the E-IDS suitable for incorporation into the NAS when used in operational environments?

COI 7 Does the E-IDS design and procedures allow the E-IDS to be optimized, transitioned, adapted to site specific conditions, and certified by available personnel without disruption or degradation to operations?

COI 8 Is the E-IDS training and documentation sufficient for personnel to safely and effectively operate and maintain the E-IDS?

COI 9 Does E-IDS have sufficient resiliency to restore a position or site configuration from another facility?

COI 10 Does E-IDS meet the applicable security requirements as defined in relevant FAA policies and orders?

A breakdown of these E-IDS COIs into Measures of Effectiveness (MOEs), Measures of Suitability (MOSs), and Measures of Performance (MOPs), are contained in Appendix A.

The Verification Requirements Traceability Matrix (VRTM), in Appendix B, traces the program requirements to the respective test phase(s), test activities, and COIs/MOEs/MOSs/MOPs.

4 E-IDS TEST PROGRAM DESCRIPTION

4.1 PREVIOUS RESULTS

Currently there are no E-IDS prototypes used to support verification and validation of system performance.

4.2 DEVELOPMENT TESTING (DT)

The Development Contractor is responsible for conducting DT on E-IDS, while witnessed by the Government, to demonstrate that all contractual requirements for E-IDS have been met. The DT test strategy is documented in the Contractor Master Test Plan (CMTP) which is approved by the Government. An additional objective of DT (verified during SAT) is to prove that the E-IDS is fully integrated and stable, and that it causes no adverse effect on the rest of the NAS. Following the successful completion of DT, the FAA accepts the E-IDS via a formal Government Acceptance (GA) process, and the OT phase of the E-IDS test program can begin.

ATM-EIDS-A3-TMP-001 17 December 11, 2017

E-IDS DT activities shown in Figure 4-1, that lead to Government Acceptance (GA) include:

Hardware Testing, Software Testing, Factory Acceptance Testing (FAT), Installation and Integration (I&I) Testing, System Testing, Production Acceptance Testing (PAT), Site Acceptance Testing (SAT) and Regression Testing.

#n #n #2 #2

Hardware Testing

DT Key Site

(SAT)

System Test

Contractor/FAA Regression Testing

Software Testing

Factory Acceptance

Testing

Installation

Integration OT OT Key

Site

TRRTRR

Contractor Development Testing

Government Acceptance

Government Acceptance

PAT #nPAT #n SAT #nSAT #n

PAT #2PAT #2 SAT #2SAT #2

PAT SAT

FAA Operational Testing

Early Look

TRR

Figure 4-1 E-IDS Test Activities

The Development Contractor will execute DT dry runs and formal tests for all DT activities.

The first site locations chosen for SAT will be considered Key Sites and will consist of at least one Tower, TRACON, ARTCC, and any other selected sites. Following successful completion of DT Key Site (SAT) activities and GA the E-IDS DT phase will be complete.

Prior to the conduct of any formal test event, the Development Contractor provides the Government with a copy of the Test Plan and Test Procedures for review and approval.

Following the successful completion of System Test, the FAA accepts the E-IDS via a formal Government Acceptance (GA) process and the conduct of OT can commence.

Regression testing will be conducted as needed to verify any fixes that have been made during testing.

4.2.1 TEST ACTIVITIES

The Development Contractor’s DT program includes the following DT activities:

• Hardware Testing

• Software Testing

• Factory Acceptance Testing (FAT) including a COTS/Non-Development Item (NDI) Screening Requirements Verification

• Installation and Integration (I&I) Testing

• System Testing

• Production Acceptance Testing (PAT)

• Site Acceptance Testing (SAT)

ATM-EIDS-A3-TMP-001 18 December 11, 2017

• Regression Testing

4.2.2 DT ENTRANCE CRITERIA

Prior to entering any formal DT activity, the DT Test Director ensures that the following minimum DT entrance criteria are met, unless otherwise agreed to by the FAA and the development contractor: (Activity specific criteria is listed under each test activity section)

a) All entrance criteria as defined in the approved CMTP and DT Test Plan(s) are satisfied;

b) DT Test Procedures have been submitted, reviewed, and approved;

c) The test configuration is known and documented;

d) Test Capability Accreditation has been completed and approved;

e) Dry run testing has been completed by the contractor;

f) The configuration under test does not have known deficiencies that affect the functions to be verified by the test; and

g) A Test Readiness Review (TRR) has been successfully conducted.

4.2.3 DT EXIT CRITERIA

Prior to exiting any formal DT test activity, the DT Test Director ensures that the following minimum DT exit criteria are met, unless otherwise agreed to by the FAA and the development contractor: (Activity specific criteria is listed under each test activity section)

a) All exit criteria as defined in the approved CMTP and DT Test Plan(s) are satisfied;

b) Completion of all DT tests in accordance with approved Test Plans and Test Procedures;

c) All Post-Test Reviews are complete;

d) Test results are documented and accepted by the FAA;

e) All DT Program Trouble Reports (PTRs) are fully documented, assessed, and statused; and

f) All contractor performance requirements are addressed in accordance with the contract.

4.2.4 TEST ACTIVITY: DT SOFTWARE TESTING

TEST OBJECTIVES

Verification of the specifications at the B-Level (subsystem). DT Software Testing usually addresses new and modified software broken down to the Computer Software Configuration Item (CSCI), Computer Software Component (CSC), and functional design components.

ATM-EIDS-A3-TMP-001 19 December 11, 2017

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

Software Testing will be conducted at the Development Contractor’s facility.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.5 TEST ACTIVITY: DT HARDWARE TESTING

TEST OBJECTIVES

Verification of the specifications at the B-Level. DT Hardware Testing addresses new and modified hardware broken down to the Hardware Configuration Item (HWCI) and Hardware Design items.

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

Hardware Testing will be conducted at the Development Contractor’s facility, WJHTC, and Key Site(s).

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

ATM-EIDS-A3-TMP-001 20 December 11, 2017

TEST EXIT CRITERIA

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.6 TEST ACTIVITY: DT FACTORY ACCEPTANCE TEST

TEST OBJECTIVES

Verification of primarily hardware, firmware, and Commercial Off-the-Shelf (COTS)/Non- Developmental Item (NDI) subsystem components to address B-Level and A-Level (system) specification items.

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

FAT will be conducted at the Development Contractor’s facility.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.7 TEST ACTIVITY: DT INSTALLATION AND INTEGRATION TESTING

TEST OBJECTIVES

Verification of proper installation and functioning of the complete system in the laboratory, including the verification of system interfaces with other NAS equipment and Government Furnished Equipment (GFE).

TEST APPROACH

ATM-EIDS-A3-TMP-001 21 December 11, 2017

TEST LOCATION

I&I will be conducted at the Development Contractor’s facility and the WJHTC.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.8 TEST ACTIVITY: DT SYSTEM TESTING

TEST OBJECTIVES

Following I&I and the subsequent approved Test Readiness Review (TRR), the Development Contractor performs formal System Testing on E-IDS. System Testing is performed under conditions that emulate the projected operational environment to verify the DT contractual requirements in the E-IDS System/Subsystem Specification (SSS) and system stability.

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

System testing will be conducted at the Development Contractor’s facility or the WJHTC.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

ATM-EIDS-A3-TMP-001 22 December 11, 2017

TEST LIMITATIONS

4.2.9 TEST ACTIVITY: DT PRODUCTION ACCEPTANCE TESTING

TEST OBJECTIVES

The Development Contractor will conduct DT PAT to assure system quality prior to site shipment, and ensuring that units under test are free from manufacturing defects and conform to the product baseline. DT PAT will be conducted on a limited number of COTS products in intervals determined by the program office to achieve an acceptable confidence level that there are no manufacturer defects.

The objective of all PAT is the verification of a select subset of SSS requirements and to assure that the equipment shipped to the site is correct in accordance with the site survey report, in working order and configured in accordance to the site survey report and the Configuration Management (CM) documents.

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

PAT will be conducted at the Development Contractor’s facility or the WJHTC.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.10 TEST ACTIVITY: DT SITE ACCEPTANCE TESTING

TEST OBJECTIVES

The purpose of DT SAT is to verify the site installation, system integration, stability, and operational configuration of the E-IDS, as well as to verify the DT contractual requirements that need operational site conditions, live interfaces, and/or configurations/adaptations not available during the E-IDS DT System Testing.

ATM-EIDS-A3-TMP-001 23 December 11, 2017

The DT SAT verifies a selected sub-set of the SSS VRTM requirements. The Development Contractor will conduct DT SAT at the DT key site to verify the site installation, system integration, stability, and operational configuration of the installation. Note: Interface simulators will be used if connection to the operation interfaces are not available/permitted.

Subsequently, the Development Contractor will conduct SAT on each installed E-IDS production system, witnessed by the Government, to verify the site installation, system integration, stability, and operational configuration of the installation using a subset of SSS VRTM requirements. SAT is performed on all systems installed at a Government location.

TEST APPROACH

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LOCATION

PAT will be conducted at the Development Contractor’s facility or the WJHTC.

TEST CONFIGURATION

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST AND ANALYSIS TOOLS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST ENTRANCE CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST EXIT CRITERIA

(To be provided in iTEMP or fTEMP after an alternative is chosen)

TEST LIMITATIONS

(To be provided in iTEMP or fTEMP after an alternative is chosen)

4.2.11 TEST ACTIVITY: DT REGRESSION TESTING

Verification of integrated software and hardware components after changes have been made to the system. DT Regression Testing includes all retesting for any anomalies, failures, issues, deviations, revisits and DR/PTRs discovered during test activities. DT Regression Testing also ensures that any changes made did not inadvertently result in a problem elsewhere in the system.

TEST APPROACH

Based upon the degree of change, the Development Contractor will recommend the appropriate test procedures and gain concurrence from the Government Test Director.

4.2.12 TEST LIMITATIONS

ATM-EIDS-A3-TMP-001 24 December 11, 2017

4.3 OPERATIONAL TESTING (OT)

The Government conducts OT to validate the E-IDS. OT can commence following the completion of E-IDS DT System Testing and a formal Government Acceptance (GA) process.

An OT TRR is held by ANG-E59 prior to the start of OT. The E-IDS OT is performed by ANG- E59. Other OT support includes participation from FAA operational field representatives, other Government representatives, and vendor hardware and software personnel.

Formal OT is performed at the FAA WJHTC in the E-IDS lab and the first Key Site(s). An OT Key Site as shown in Figure 4-1 will be held at an operational field site if the full fidelity of the MOEs, MOSs, MOPs cannot be validated at the WJHTC in a test environment.

The primary objective of OT is to examine performance and determine operational readiness (effectiveness and suitability) of a new or modified system for use in the NAS and that the NAS infrastructure is ready to accept the system. The OT decision points and milestones are shown in Figure 4-2.

NO

Train Field Test

Participants

Develop, Review & Approve

OT Test Plan

Develop, Review and Approve

OT Test Procedures

Develop, Review & Finalize

OT Quicklook Test Report

Develop, Review & Approve

OT Final Test Report

Regression Needed?

YES

Document Development

Perform Activity

LEGEND

Conduct Pre Test Briefing

Conduct Test Readiness

Review

Conduct Formal Test Execution

Conduct Final Post Test

Review

Conduct OT Caucus

Dry Run Test Procedures Incorporate Redlines

SAT

Figure 4-2 E-IDS OT Process Flow

The E-IDS OT program will consist of three (3) main user groups: ATC Operations, Technical Operations (Tech Ops), and Support Operations. A fourth group, Specialty Tests, will be conducted either by questionnaires, time frame, or other test organizations. The OT test activities are outlined in Figure 4-3 and further defined in the OT Plan.

ATM-EIDS-A3-TMP-001 25 December 11, 2017

Figure 4-3 E-IDS OT Activities

The preceding test activities are designed to address all the COIs identified in Appendix A and the program requirements allocated to OT identified in Appendix B. The COIs will be decomposed into MOEs, MOSs, and MOPs.

OT Test participants will have the opportunity to complete questionnaires after the completion of the OT test activities. The ANG-E59 OT test team will review and analyze the quantitative and qualitative data collected and observed. The questionnaire data collected from the test participants will be used augment the operational effectiveness and suitability of the E-IDS.

These test results will be discussed at the OT Caucus with all participants and stakeholders.

At the conclusion of formal OT, an OT Caucus, chaired by the OT Test Director, is conducted to present and review all findings and test results related to operational effectiveness and suitability of the system. At the caucus, each COI must be answered based on the criticality of the problem(s) impacting it. The operational readiness (effectiveness and suitability) determination is made by the ANG –E59 OT Test Team with concurrence from the ITT. All issues and anomalies discovered during OT will be addressed in OT Regression testing if necessary.

The test groups and the associated tests activities are described in the following sections, each with a corresponding test objective and approach.

4.3.1 ATC OPERATIONS TEST GROUP

ATC personnel from various facilities will be the participants for these test activities. Refer to the E-IDS OT Plan for more details. COIs 1, 3, 4 and 9 will be addressed during this testing.

NAS INTEGRATION TEST ACTIVITY OBJECTIVES

The objective of the NAS Integration test is to validate the interoperability between E-IDS and other systems, outlined in section 3.2. The focus of this test activity is to ensure that the E-IDS can operate with other system interfaces and exchange information. Testing will ensure that the data is processed correctly, accurately, and that there is no loss of data. Testing will also ensure

ATM-EIDS-A3-TMP-001 26 December 11, 2017 that the exchange of data between E-IDS and other NAS system interfaces does not adversely affect or degrade E-IDS or NAS system interfaces.

TEST APPROACH

The approach for this test activity is to populate the E-IDS with various forms of information from all of the available live interfaces. This will verify that the E-IDS can interface and exchange data with other existing NAS systems. This test will run concurrently with the NAS Facility Operations test activity. The interfaces will be checked for accuracy and timeliness of data as ATC operators perform their normal duties.

NAS FACILITY OPERATIONS TEST ACTIVITY OBJECTIVES

The objective of the NAS Facility Operations Test is to ensure that ATC personnel at their facility can use utilize the E-IDS to perform their specific tasks efficiently without any interruption to daily operations. ATC users will demonstrate that, with the information they receive from E-IDS, they can look up reference material, view weather events, and identify conditions that may impact ATC operations. This test activity will also validate the E-IDS training, documentation, and ATC procedures are efficient for use. A questionnaire will be presented to the test participants upon completion of this test.

This test will include ATC operations from the following NAS Facilities:

• Air Traffic Control System Command Center (ATCSCC) Operations

• Air Route Traffic Control Center (ARTCC) Operations

• Terminal Radar Approach Control Facilities (TRACON) Operations

• Air Traffic Control Tower (ATCT) Operations

• Combined Center Radar Approach Controls (CERAPs) Operations

• FAA Flight Service Stations (FSSs) Operations

TEST APPROACH

The E-IDS is positioned in a lab environment that emulates one or more of the NAS facilities listed above. ATC personnel from the different NAS Facilities are given a set of operational procedures to exercise the E-IDS system functionalities under specific NAS facility operational conditions. ATC personnel will perform their tasks under normal and non-normal “Day in the Life” conditions. The users are presented with a variety of information to evaluate the E-IDS while performing their duties.

NON-FAA USERS TEST ACTIVITY OBJECTIVES

The objective of the Non-FAA Facility Operations Test is to ensure that personnel at their facility can use the E-IDS to perform their specific tasks efficiently without any interruption to daily Non-FAA operations.

TEST APPROACH

The E-IDS is positioned in a lab environment that emulates a Non-FAA facility via the use of simulators. Personnel from the Non-FAA facility are given a set of operational procedures to exercise the E-IDS system functionalities under the specific Non-FAA facility operational

ATM-EIDS-A3-TMP-001 27 December 11, 2017 conditions. The users are presented with a variety of information to evaluate the E-IDS while performing their duties.

E-IDS RESILIENCY TEST ACTIVITY OBJECTIVES

The objective of the E-IDS Resiliency Test is to ensure ATC personnel can use the E-IDS effectively to respond to events that may affect the capability to provide ATC services to NAS users.

This test activity will demonstrate that the E-IDS system provides various types of information to assist ATC personnel in managing different operational situations, reducing CPC workloads, and increasing situational awareness. Testing will also include a site’s ability to take over another site in the event of a site outage.

TEST APPROACH

The approach for this test activity is to ensure that ATC personnel can access and display various sources of information on one display, perform searches and filters on the displayed information, and distribute the information to other positions. The Data Administrators will disseminate specific information for the facility’s E-IDS, (i.e. Letters of Agreement (LOAs), Standard Operation Procedures (SOPs), FAA orders, and approach charts), tailor the information for each position, and set user privileges. Data backup, data migration and reconstitution capabilities during interruptive events will also be demonstrated.

E-IDS PERFORMANCE TEST ACTIVITY OBJECTIVES

The objective of the E-IDS Performance Test is to validate that the E-IDS performance levels are operationally acceptable to ATC personnel in a full end-to-end configuration using all available interfaces. The E-IDS will be tested using various NAS facility performance loads, data transaction rates, and actual system response times. E-IDS data exchange will be evaluated for timeliness and accuracy in providing current information.

TEST APPROACH

The approach for this test activity is to evaluate the latency under nominal, peak, and stress loads to ensure the E-IDS data accuracy and processing time is suitable for ATC personnel to perform their duties. For non-nominal situations, i.e. any impact to airspace safety, ATC personnel will utilize the E-IDS at increased levels to display multiple windows with multiple sources of information, to ensure that the E-IDS does not experience degradation of capacity, functionality or performance, and does not degrade other interfacing NAS systems or services.

4.3.2 TECHNICAL OPERATIONS TEST GROUP

Technical Operations (Tech Ops) personnel from various facilities will be the participants for these test activities. Refer to the E-IDS OT Plan for more details. COIs 5 and 7 will be addressed during this testing.

SYSTEM MAINTENANCE OPERATIONS TEST ACTIVITY OBJECTIVES

The objective of the System Maintenance Operations Test is to validate that the E-IDS can be maintained by Tech Ops personnel. Tech Ops personnel will conduct routine preventive maintenance tasks utilizing the E-IDS Technical Instruction Books (TIBs) and other user manuals. This test will ensure that the E-IDS tools, training, electronic documentation, and the infrastructure enables maintenance tasks to be performed without interruption to ATC

ATM-EIDS-A3-TMP-001 28 December 11, 2017 operations. Maintenance tasks that will be exercised are software upgrades, hardware repair and/or replacement, system diagnostics, and system/service certification.

TEST APPROACH

The approach for this test is to have Tech Ops personnel perform maintenance functions on the E-IDS. Tech Ops personnel will use the E-IDS displays to monitor the status of E-IDS, as well as view the technical and maintenance manuals needed to maintain the E-IDS hardware systems.

Tech Ops personnel will perform various maintenance functions for hardware and software sustainment, i.e. manage user data, perform data reductions, record data, archive data, play back data, generate and print reports.

DEGRADED OPERATIONS TEST ACTIVITY OBJECTIVES

The objective of the Degraded Operations test is to validate that the loss of any level of E-IDS services do not impact ATC operations while Tech Ops personnel resolve the issues. Tech Ops personnel will use the E-IDS technical and maintenance manuals to diagnose the problems and return the E-IDS to full operation. This test will also verify that RMLS receives and displays the correct degraded status of E-IDS.

TEST APPROACH

The approach for this test is to inject certain failures into the E-IDS causing some of the resources to become unavailable. Tech Ops personnel will be called to diagnose and correct the fault. Tech Ops personnel will utilize the E-IDS status display and maintenance manuals to identify and isolate the issue. Once Tech Ops has determined the cause of the problem, they will repair the E-IDS and return it to full operating status. Tech Ops Personnel will also conduct a service certification of the E-IDS.

FAILURE & RECOVERY TEST ACTIVITY OBJECTIVES

The objective of the Failure & Recovery test is to validate that the E-IDS can recover from a wide range of failure conditions. Software and hardware faults will be induced to generate error notifications, alarms, and indications. Tech Ops personnel will use maintenance manuals to isolate and diagnose the problem. The impact of the failure conditions will be evaluated by ATC and Tech Ops personnel including the integrity of the recovered data. Failure/recovery testing will include a National Data Center failure which would require a manual switchover between Salt Lake (ZLC) and Atlanta (ZTL). An E-IDS failure at a site which would require a switchover to another sites’ E-IDS. Other failure/recovery testing may include, depending on what architecture is chosen, a failure of an E-IDS regional server, which would require a switchover to another regionals’ E-IDS server or a cloud-based service recovery. This test will also verify that RMLS receives and displays the correct failure status of E-IDS servers, clients, and interfaces.

TEST APPROACH

The approach for this test is to induce software and hardware failures identified from the Failure Mode Effects Analysis (FMEA) into the E-IDS. Tech Ops will evaluate the capabilities of the E- IDS to indicate a failure condition, the ability to isolate the failure to a specific root cause, and to identify the necessary recovery actions to restore the E-IDS to full operating status. In addition, ATM-EIDS-A3-TMP-001 29 December 11, 2017 the users will evaluate the recovery and restoration procedures for their operational effectiveness.

Tech Ops personnel will also conduct a system certification of the E-IDS.

TRANSITION & FALLBACK TEST ACTIVITY OBJECTIVES

The objective of the transition and fallback test is to validate that transitioning from the legacy system to E-IDS and falling-back from E-IDS back to the legacy system can be accomplished effectively without unintended degradation of ATC services.

TEST APPROACH

The approach for this test is to transition the legacy IDS over to the E-IDS utilizing the facility’s’ transition procedures. In the event of an anomaly, the facility will fall back to the original legacy IDS. This will validate that the facility has the proper procedures, tools, training, documentation, logistical support, and equipment to transition or fallback.

4.3.3 SUPPORT OPERATIONS

Second Level Engineering (SLE), Data Managers (DM), and System Administrators (SA) personnel will be the test participants for these test activities. Refer to the E-IDS OT Plan for more details. COI 7 will be addressed during these test activities.

REMOTE MAINTENANCE OPERATIONS TEST ACTIVITY OBJECTIVES

The objective of the Remote Maintenance Operations Test is to validate that the E-IDS system can be supported by the SLE Site Support Team. SLE personnel are tasked with remotely identifying problems, providing support to the site in problem resolution, and supplying national software releases. This test activity will validate that SLE personnel can login remotely from the WJHTC and retrieve system log files from a facility’s’ E-IDS, install national generated software. This test will validate that SLE personnel have the tools, documentation, and training required to develop and maintain E-IDS. This testing will ensure that SLE has the resources it needs to analyze data, troubleshoot issues, and sustain all aspects of E-IDS. This test will also validate that E-IDS provides system status and performance information to the RMLS.

TEST APPROACH

The approach for this test is to position an E-IDS in a lab environment that emulates one or more NAS facilities. SLE personnel will use work stations separated from the lab environment and login to the E-IDS. Testing will ensure that SLE can view the E-IDS status, system alerts and outages for the hardware, software and interfaces. This test will demonstrate that the E-IDS capabilities and/or tools used by SLE personnel are efficient for software management, system health monitoring, log reporting, and playback of E-IDS processing. RMLS operators will also verify that they receive the same status and performance information. This test may be run after or during the Degraded Operations and Failure & Recovery tests.

DATA ADMINISTRATION TEST ACTIVITY OBJECTIVES

The objective of the Data Administration test is to validate that the Data Managers (DMs) and System Administrators (SAs) can configure the E-IDS to support domain-specific uses. DMs and SAs will customize and organize the hierarchy of information to meet entry and display needs of specific user positions within a NAS facility. They will create pages with corresponding

ATM-EIDS-A3-TMP-001 30 December 11, 2017 shortcuts and ensure the information is current. DMs and SAs will also respond to any E-IDS alerts that indicate connectivity issues with information sources.

TEST APPROACH

The approach for this test activity is to configure the E-IDS lab for one or more types of NAS or Non-FAA facilities. The E-IDS will be populated with information from various data sources.

Testing will include DMs and/or SAs customizing the E-IDS for different facility positions in addition to configuring them with facility specific information. The DMs and SAs will create, add, delete, manipulate, and organize the page usage information to support the users’ needs pending on the facility position.

4.3.4 SPECIALTY TESTS GROUP

Personnel from all groups may participate in these test activities. Refer to the E-IDS OT Plan for more details. COIs 2, 6, 8, and 10 will be addressed during these test activities.

TRAINING TEST ACTIVITY OBJECTIVES

This test activity will validate that E-IDS training provided to the test participants prior to OT conduct is operationally suitable for NAS Facility personnel to perform basic E-IDS operations.

E-IDS training materials and operational procedures will be available to aid the test participants during this testing. Training will be provided for ATC, Tech Ops, and SLE personnel.

TEST APPROACH

The approach for this test is to have the test participants receive the available training before conducting any OT activity. Questionnaires will be provided to the test participants to evaluate if the training they received was sufficient to perform basic E-IDS functions. The training materials and operational procedures will be available during testing for reference.

HUMAN FACTORS & SAFETY TEST ACTIVITY OBJECTIVES

The objective of the Human Factors test is to verify the E-IDS operational display satisfies Computer Human Interface (CHI) operational needs and that it can be used to conduct safe airspace operations. Testing will ensure that the E-IDS displays information correctly and accurately.