(U-FOUO) NSG_Enterprise_Test_Evaluation_Master_Plan.pdf

UNCLASSIFIED//FOR OFFICIAL USE ONLY

NSG Enterprise TEMP-D

13 June 2018

Total Pages: 81

National Geospatial-Intelligence Agency (NGA)

(U) NATIONAL SYSTEM FOR GEOSPATIAL INTELLIGENCE

(NSG) ENTERPRISE

TEST AND EVALUATION MASTER PLAN (TEMP)

Revision D

13 June 2018

Abstract: (U) The NSG Enterprise TEMP describes the independent life cycle, enterprise-level testing of the NSG and Allied System for Geospatial-Intelligence (ASG).

Key Words: (U) National System for Geospatial-Intelligence (NSG), National

Geospatial-Intelligence Agency (NGA), NGA Test Organization (NTO), NTO Test Environment, System Test, Test and Evaluation (T&E), Testing, Image Quality, Product Quality, NGA Image Quality and Utility

(NIQU)

13 June 2018 ii

(U) NSG ENTERPRISE TEMP

DISTRIBUTION LIST

(U) Copies of this document have been provided to the distribution noted below.

Requests for additional copies or changes to the Distribution List should be directed to NGA CM.

(U) CM Library

NGA Campus East (NCE) CM Library

13 June 2018 iii

TABLE OF CONTENTS

1. (U) INTRODUCTION

1.1 (U) PURPOSE

1.2 (U) SCOPE

1.3 (U) MISSION DESCRIPTION

1.4 (U) CONFIGURATION MANAGEMENT CONTROL

1.5 (U) CRITICAL SYSTEM CHARACTERISTICS

2 (U) TEST STRATEGY

2.1 (U) INTRODUCTION

2.2 (U) GENERAL METHODOLOGY

2.3 (U) TEST SUPPORT FUNCTIONS ROLES AND RESPONSIBILITIES

2.3.1 (U) Business Area Lead

2.3.2 (U) NTO Test Director

2.3.3 (U) NIQU Topic Lead

2.3.4 (U) Program Manager/Project Manager and/or Product Owner

2.3.5 (U) Joint Interoperability Test Command

2.3.6 (U) Infrastructure Service Provider

2.3.7 (U) Cloud Service Provider

2.3.8 (U) Consolidated Systems Engineering Review Board

2.3.9 (U) Applications Services Governance Board

3 (U) INTEGRATED TEST PROGRAM

3.1 (U) INTRODUCTION

3.2 (U) SUPPORT TO DEVELOPER TESTING

3.2.1 (U) Developmental Test and Evaluation, Phase 1

3.2.2 (U) Developmental Test and Evaluation, Phase 2

3.3 (U) GOVERNMENT TESTING

3.4 (U) QUALITY ASSURANCE SERVICES

3.4.1 (U) Engineering Analysis

3.4.2 (U) Test Environment Hosting

3.4.3 (U) Test Execution

3.4.4 (U) Readiness

3.4.5 (U) Security Assessments

3.4.6 (U) Business Process Management Team

4 (U) NGA TEST ORGANIZATION DIVISION

4.1 (U) THE NTO MISSION AND RESPONSIBILITY

4.1.1 (U) Certification to Field

4.1.2 (U) Waivers

4.2 (U) NTO STAFFING

4.3 (U) NTO QUALITY CRITERIA

4.4 (U) NTO TESTING FACILITIES

4.4.1 (U) NTO Test Infrastructure

4.4.2 (U) Engineering Test Data/Simulated Reference Data

4.5 (U) NTO CUSTOMERS

4.6 (U) REQUESTING NTO SUPPORT

13 June 2018 iv

UNCLASSIFIED//FOR OFFICIAL USE ONLY

4.7 (U) NTO TEST PROCESS

4.7.1 (U) Test Planning and Coordination

4.7.2 (U) Test Execution

4.7.3 (U) Test Reporting

4.8 (U) NTO TEST SUPPORT FUNCTIONS

4.8.1 (U) Test Information Technology Infrastructure Operations

4.8.2 (U) NTO DR Management

4.8.3 (U) NTO Business Area Lead Document Quality Review

4.8.4 (U) NTO Gold Copy

4.8.5 (U) Test Automation

5 (U) NGA IMAGE QUALITY AND UTILITY DIVISION

5.1 (U) NIQU STAFFING

5.2 (U) PRODUCT QUALITY ASSESSMENT

5.3 (U) PRODUCT QUALITY CRITERIA

5.4 (U) NIQU TESTING FACILITIES

5.4.1 (U) NIQU Engineering Cell

5.4.2 (U) NIQU Evaluation Facility

5.4.3 (U) NIQU Use of the NTO Test Environment

5.4.4 (U) NIQU Testing in the Cloud

5.5 (U) REQUESTING NIQU SUPPORT

5.6 (U) NIQU TEST PROCESS

APPENDIX A. (U) NGA TEST ACTIVITIES

APPENDIX B. (U) ABBREVIATIONS AND ACRONYMS

APPENDIX C. (U) GLOSSARY

APPENDIX D. (U) REFERENCES

LIST OF TABLES

Table 1. (U) Discrepancy Report Priority Definitions Used in Testing Table 2. (U) NGA Test Activities

LIST OF FIGURES

Figure 1. (U) NGA Enterprise-Level Integrated Test Program Figure 2. (U) Certification-to-Field Decision Process Figure 3. (U) Test Overview Figure 4. (U) NTO and NIQU Agile Test Engagement Figure 5. (U) TAC DevOps Testing Operational View

13 June 2018 v

UNCLASSIFIED//FOR OFFICIAL USE ONLY

UNCLASSIFIED

(U) TBD/TBR LOG

PARAGRAPH/

FIGURE/TABLE

PAGE

NO.

TBD

TBR

DESCRIPTION

UNCLASSIFIED

13 June 2018 vi

(U) CHANGE LOG

TITLE: (U) NSG Enterprise Test and Evaluation Master Plan (TEMP)

SCN/

REV

DATE

ITEMS AFFECTED REMARKS/RFC

AUTHORITY

000 15 Apr 2004 All – Initial Issue RFC N01-0397

NCCB S-0069

dtd 22 Apr 2004

001 26 Jul 2004 iv-2, C-2, G-2, G-10,G-17, G-21, G-30 RFC N01-

0404J NCCB

S-0078 dtd 29 Jul 2004

002 21 Oct 2004 iv-3, A-6, G-3 to G-5, G-9, G-11 to G-14, G-18 to G- 19, G-31

RFC N99-

0002J NCCB

S-0087 dtd 2 Nov 2004

003 18 Nov 2004 i, ib, ic, iib, iic, iid, iie, iii, iv, iv-1, iv-4, 3, A-1, A-5, A-8, A-10, G-1, H-1 to H-17

RFC N01-

0416J NCCB

S-0090 dtd 22 Nov 2004 REV A This Revision A incorporates the following RFCs:

004 21 Oct 2004 All – Revision A RFC 081-

0122J NCCB

S-0087 dtd 2 Nov 2004

005 20 Oct 2005 RFC N01-0438

NCCB S-0127

dtd 21 Oct 2005

006 18 Aug 2005 RFC 001-4793

NCCB S-0120

dtd 22 Aug 05

007 21 Dec 2005 RFC N09-0110

NCCB 0113-05

dtd 21 Dec 05

008 20 Jan 2006 RFC N01-0435

NCCB 0007-

06 dtd 25 Jan

009 15 Jun 2006 RFC N01-0454

NCCB 0052-06

dtd 20 Jun 06

010 4 Oct 2006 RFC N99-0021

NCCB 0092-06

dtd 05 Oct 06

13 June 2018 vii

011 20 Dec 2006 RFC 001-4856

NCCB 0120-

06 dtd 20 Dec

CLP-001 This Change Letter incorporates the following RFC:

012 20 Sep 2007 i, ib, iib, iii, iv, iv-1, iv-6, iv-7, A-1 – A-10, J-1 RFC N01-0490J

NGA CCB

0090-07 dtd 25 Sep 07 REV B This Revision B incorporates the following RFCs:

013 20 Dec 2007 All - Revision B RFC N54-0003

NCCB 0115-07

dtd 27 Dec 2007

014 20 Dec 2007 001-4933J

NCCB 0115

dtd 27 Dec 2007

015 21 Feb 2008 RFC 001-4940J

NCCB 0022-08

dtd 27 Feb 2008

016 18 Sep 2008 RFC N01-0507

NCCB 0100-08

dtd 24 Sep 2008

C/N 001 This Change Notice incorporates the following RFCs:

017 16 Dec 2009 i, 27, 28, A-4, C-1 RFC N01-0549

NGA CCB 0134-

09 dtd 16 Dec

018 18 Feb 2010 i, 32, A-4 RFC N01-0545J

NCCB 0011-10

dtd 24 Feb 2010

REV C This Revision C incorporates the following RFC:

019 16 Sep 2010 All – Revision C RFC N01-0562J

NCCB 0104-10

dtd 20 Sep 2010

C/N 001 This Change Notice incorporates the following RFC:

020 19 Jan 2012 36 (Table 4-1) RFC NAE-

0006J NCCB

0009-12 dtd 26 Jan 2012

C/N 002 This Change Notice incorporates the following RFC:

13 June 2018 viii

021 29 Jun 2012 (U) Contents of TEMP significantly reorganized: 1. Refs to Newington, ITF, Reston, and Bethesda removed. 2. Orgs updated to reflect NGA reorg in April 2012. 3.

NTO’s part in NGA agile development initiatives and NOW framework added. 4.

Section 2 rewritten to remove redundancies.

RFC N01-0581

NCCB 0058-12

dtd 29 Jun 2012

C/N 003 This Change Notice incorporates the following RFCs:

022 13 Mar 2014 31, Table 4-1 RFC NCU-0001J

MTRB 0010-14

dtd 19 Mar 2014

C/N 004 This Change Notice incorporates the following RFCs:

023 18 Dec 2014 (U) Change log, Effectivity table, added requirements tables, and Table 4-1.

RFC NSG-0046JR1

MTRB 0056-14

dtd 19 Dec 2014

C/N 005 This Change Notice incorporates the following RFCs:

024 4 Mar 2015 (U) Change Log, Table 4-1 RFC N52-0015J

MTRB 0013-15

dtd 20 Mar 2015

025 4 Mar 2015 (U) Change Log, Appendix D - Global change to remove NRAI-C.

RFC N01-0615J

MTRB 0016-15

dtd 23 Mar 2015

C/N 006 This Change Notice incorporates the following RFCs:

026 19 Aug 2015 (U) Change Log, Delete FCRD from Appendix B. RFC N01-0620

MTRB 0044-15

dtd 21 Aug 2015

027 20 Aug 2015 (U) Change Log, Update Effectivity log with effectivity and description, modify table 4.1 by adding “delete at ND001” next to WARP, modify Appendix B by adding the phase “Delete at ND001” on the line for WARP.

RFC N57-0007

MTRB 0045-15

dtd 24 Aug 2015

13 June 2018 ix

028 10 Sep 2015 (U) Change Log, Global change “1” - replace

NCMP-D with NCMP and NGA Configuration Management Plan-D with NGA Configuration Management Plan, Global change “2” – replace NCCB and MTRB with NGA CCB and replace Mission Technology Review Board with NGA Configuration Control Board.

RFC N01-0624

MTRB 0048-15

dtd 11 Sep 2015

C/N 007 This Change Notice incorporates the following RFC:

029 24 Sep 2015 (U) Change Log, Transition to reflect association with NCL architecture, Change third paragraph of section 1.1 to reflect another test system that is considered an “Associated facilities”; Change first paragraph of section 4.2.1.2.1 to replace NGL with NCL; change table 4.1 to add NDL-E and NDL-W and remove AGI-IPL, CIL, IDS-D, IPL, NIL, STIL, and WARP; Change last paragraph of section 4.2.1.2.3 to replace association with NIL, CIL, and STIL to NCL;

Change fourth paragraph of section 4.3.3 to replace NIL, CIL, STIL, and IDS-D to NCL;

Change sixth paragraph of section 4.3.3 to replace NIL, STIL, and IDS-D with NCL; Revise Appendix B, Abbreviations and Acronyms, to delete CIL, IDS-D, IPL, NGL, NIL, STIL and WARP and to add NCL, NDL, NDL-E, and NDL- W.

RFC NCL-0013J

MTRB 0051-15

dtd 25 Sep 2015

C/N 008 This Change Notice incorporates the following RFC:

030 16 Nov 2016 (U) Change Log, Transition Effectivity for NE524, delete associated GEOAxIS line in table

4.1 that indicated it was deleted when effectivity NE524 was implemented.

RFC NCH-0006

MTRB 0051-16

dtd 29 Nov 2016

C/N 009 This Change Notice incorporates the following RFC:

031 12 Jan 2017 (U) Change Log, delete any reference to ITC and reword line to incorporate deletion of ITC.

Removed ITC column from table 4.1.

RFC N99-0066J

MTRB 0002-17

dtd 10 Jan 2017

C/N 010 This Change Notice incorporates the following RFC:

032 15 Jun 2017 (U) Change Log, effectivity NE509 is baselined with this RFC. Changes reflect this. Deleted the effectivity page and the effectivity tags in table 4.1

RFC NBN-0001

MTRB 0023-17

dtd 14 Jun 2017

C/N 011 This Change Notice incorporates the following RFC:

13 June 2018 x

033 6 Nov 2017 (U) Change Log, delete any reference to AIT reworded one line in section 1.2 to incorporate deletion of AIT, Appendix B

RFC NBI-0008

MTRB 0039-17

dtd 19 Oct 2017

C/N 012 This Change Notice incorporates the following RFCs:

034 22 Mar 2018 (U) Change Log, Appendix B, Appendix D.

Note: REV C C/N 012 was not formally published, but approved changes have been incorporated in REV D.

RFC N01-0644

CCB 0013-18

dtd 22 Mar 2018

035 28 Mar 2018 (U) Change Log, List of Tables, Table 4-1, Table 4-2 and lead in paragraphs, Section 4.3.3, Appendix B.

Note: REV C C/N 012 was not formally published, but approved changes have been incorporated in REV D.

RFC N29-0008J

CCB 0015-18

dtd 04 Apr 2018

REV D This Revision incorporates the following RFC:

036 13 Jun 2018 All – Revision D RFC N01-0646J

CCB 0028-18

dtd 25 JUN 2018

(a) Indicates added page, and (d) Indicates deleted page

13 June 2018 xi

UNCLASSIFIED//FOR OFFICIAL USE ONLY

UNCLASSIFIED

(U) EFFECTIVITY

LOG

Unless identified otherwise by NGA Effectivity number (NEn), all requirements of this document are applicable, in accordance with approved Program Master Schedules.

NO.

SCN

EFFECTIVE

PER/ON

RFC TITLE (RFC NUMBER)

NE509 (U) GIMS and NGDS Test VHE at NDC-West IOC

(U) New Test VHE for GIMS/NGDS Stub and NGDS at NDC-West

032 (U//FOUO) NGA Dissemination Element-10 (NDE-

10) Deactivation/Retirement and Deletion and Transition of Multiple Effectivities (RFC NBN- 0001)

EFFECTIVITY NE509 BASELINED PER SCN 032

NE524 (U) GEOAxIS Test VHE at NCE Tech Center IOC

(U) New Test VHE for GEOAxIS at NCE Tech Center

030 (U) Virtual RFC: GEOAxIS Documentation Clean up through Release 3.6 (As-Built) and Baselines NE524 and NE527 (RFC NCH-006)

EFFECTIVITY NE524 BASELINED PER SCN 030

ND001 027 NMS/See ESS-VRT (U) NGA Data Interim Store (NDIS) (formerly WARP) Retirement (RFC N57-0007J)

029 (U) Documentation Retirement for the STIL, NIL, IDS-D, TDOC CIL in support of D&D activities and Baseline Multiple Effectivities, Part 1 of 2 (RFC

NCL-0013J)

EFFECTIVITY ND001 DELETED PER SCN029

NOTE: Added requirements are identified with an NGA Effectivity number (NEn) in the right margin.

13 June 2018 xii

(U) WAIVER LOG

WAIVER

NO.

WAIVER

SCN

* DISPOSITION TITLE/REMARKS

*Disposition is “OPEN,” “CLOSED.”

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

1. (U) INTRODUCTION

1.1 (U) PURPOSE

(U) The National System for Geospatial Intelligence (NSG)/Allied System for Geospatial-Intelligence (ASG) Enterprise Test and Evaluation Master Plan (TEMP) provides a master plan for independent test and evaluation (T&E) of all changes to the NSG/ASG Enterprise prior to operational deployment. The NSG/ASG Enterprise TEMP describes how the National Geospatial-Intelligence Agency’s (NGA’s) independent test program provisions, manages, and operates test resources and the organizational roles and responsibilities. The NSG/ASG Enterprise TEMP provides information about advanced and non-traditional technologies, development approaches and systems, and traditional and hybrid development approaches.

1.2 (U) SCOPE

(U) The NSG/ASG Enterprise TEMP describes the independent T&E activities required to support the integration of sensor data, hardware baselines, and software baselines across NGA and the user communities. New requirements and identification and correction of errors and deficiencies drive changes to the applications, services, sensor data, and systems that compose the NSG/ASG Enterprise. These changes require frequent testing. The T&E process is tailorable to support a variety of development approaches, including waterfall, spiral, agile, Development and Operations (DevOps), mobile applications, Application Program Interfaces, and hybrids.

(U) The NSG/ASG Enterprise TEMP references software, sensor data, hardware, systems, applications, services, widgets, desktops, corporate systems, or networks that require testing. For clarity and ease of reference, “entity” will be used to represent those terms unless noted otherwise. The term “test plan” refers to an NTO or NIQU test plan unless stated otherwise. Additionally, the terms “test,” “test event,” and “assessment” are used interchangeably; they are all meant to describe the activities occurring during the test execution and are not intended to convey separate activities.

(U) The NSG/ASG Enterprise TEMP describes the comprehensive T&E strategy for independent testing of the NSG/ASG Enterprise, both present and future.

(U) NGA does not require programs or projects to produce a separate government TEMP or an appendix for this NSG/ASG Enterprise TEMP for their activities. During the development phases of a program or project, development contractors should follow the test processes of their companies; many contractor organizations have an internal TEMP.

(U) Although the test process is adaptable, basic principles and criteria are essential to producing valid test results. As NGA has pursued a more agile development approach to reduce system complexity and deployment time, test processes and strategies have been improved to support accelerated release cycles.

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

1.3 (U) MISSION DESCRIPTION

(U) Enterprise-level testing is conducted by the Integration Test & Image Quality Office’s (TAC’s) Assessment Division (TACA) and Image Quality Division (TACQ) to provide an independent quality assessment of products provided to the user community to meet mission needs. TACA is also referred to as the NGA Test Organization (NTO); TACQ is also referred to as the NGA Image Quality and Utility

(NIQU).

(U) Independent testing is conducted using the NTO Test Environment (NTO TE), NIQU Evaluation Facility (NEF), NIQU Engineering Cell (NEC), and assets at associated facilities. The NTO TE comprises dedicated test resources in two locations: NGA Campus East (NCE) and NGA Campus West (NCW). The NEF and NEC are specialized assessment environments located at NCE. “Associated facilities” refers to other sites where capabilities are located that are routinely used to support or participate in testing through a network connection. In rare circumstances, it may be necessary to conduct testing at additional sites. Such circumstances require prior approval by TAC management.

(U) The focus of an independent test program is to support preservation of the integration and interoperability of the NSG/ASG Enterprise mission production information technology enterprise. To do this effectively, test resources are allocated to those applications, services, systems, sensor data, or infrastructure items that pose the greatest risks to integration and interoperability. Identification of entities requiring test is a shared responsibility of Program, Project, or Product Managers (PMs) and TAC and is based on the risk that the entities pose to other NSG/ASG Enterprise components or user workflows.

The Director of the Integration Test & Image Quality Office (D/TAC) of the Transformation Services Group (TA) is the approval authority for the NSG/ASG Enterprise test process.

1.4 (U) CONFIGURATION MANAGEMENT CONTROL

(U) This document is controlled by the NGA Configuration Control Board (NGA CCB). The NSG/ASG Enterprise TEMP will be modified when the D/TAC determines that the existing NSG/ASG Enterprise TEMP no longer provides adequate guidance to cover the NGA T&E process. All changes to this document shall be coordinated via the Request for Change (RFC) process and approved by the NGA CCB prior to incorporation.

1.5 (U) CRITICAL SYSTEM CHARACTERISTICS

(U) Universally applied critical system characteristics are called key performance parameters (KPPs) and are described in NGA’s Operational Requirements Document (NORD). Required capabilities are described in a Capability Development Document (CDD). Critical technical parameters (CTPs) are derived from the KPPs or the CDD associated with a particular program or project, internal NGA Enterprise system, or contractual agreement. Due to the general nature of KPPs, under most

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

circumstances, TAC will use specific capabilities and requirements documentation that trace back to the KPPs in order to understand new required capabilities and to develop test objectives to verify and validate them. Examples of requirements documentation include the Product Quality Operational Readiness Criteria (PQ- ORC), applicable segment/specification requirements documents (SRDs), segment/system requirements verification traceability matrices (RVTMs), user concept of operations (CONOPS), features, and feature acceptance criteria.

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

2 (U) TEST STRATEGY

2.1 (U) INTRODUCTION

(U) T&E is a process that spans the full lifecycle of a system. The Defense Acquisition System defines several categories of testing, three of which apply to NGA acquisition programs: Developmental Test and Evaluation (DT&E), Operational Test and Evaluation (OT&E), and DevOps.

(U) DT&E is conducted early in a system’s lifecycle and is focused on ensuring a system’s design meets its requirements. DT&E plans for software development document a specific progression of testing: unit testing, component testing, and system testing. For NGA acquisitions, DT&E is conducted by the developer;

however, tests conducted near the end of DT&E are executed under close government program management office (PMO) supervision. Formal requirements verification is accomplished by the government’s PMO during one of the closely supervised tests. One of TAC’s goals during DT&E is to integrate with the developer by collaboration, to identify issues early in the development pipeline, and provide testability assessments for feature/capability acceptance criteria and requirements.

(U) OT&E is conducted late in a system’s lifecycle and is focused on independently verifying that the delivered system or sensor data performs effectively and is suitable for use in its intended operational environment. OT&E is conducted exclusively by the government with the assistance of independent contractors and the user community; Department of Defense (DoD) strictly limits development contractor involvement to ensure the integrity of test results. One of TAC’s goals during OT&E is to ensure the correct implementation of an entity’s design and its suitability for use in its intended operational environment.

(U) NGA is developing a DevOps strategy. The DevOps process is a foundational element to digital transformation, and it allows for collaboration among developers and assessment teams. It also allows organizations to identify issues earlier in the development pipeline. This, along with continual integration and testing, reduces time spent troubleshooting and improves average time of repair.

Higher-quality products are therefore delivered to the end users more quickly and more frequently. One of TAC’s goals during the development to operations stage is to ensure that the developer is using testing best practices that help improve the quality of the products deployed to the end user. TAC will be a collaborative partner in the effort among the developers, product owners (POs), and the end user, to ensure quality products are delivered in the most efficient and effective way possible.

(U) TAC supports the mission of the Geospatial Intelligence (GEOINT) Functional Manager Standards Assessment (GFMSA) Program by ensuring that GEOINT capabilities, segments, and software developed for deployment into the NSG/ASG adhere to mandated interoperability and quality standards. TAC applies early engineering support and rigorous testing to all GEOINT entities to

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

facilitate their seamless integration and to minimize any negative impacts to the

NSG/ASG.

(U) TAC testing is expected to be both realistic and rigorous, while striking a balance between cost (time and resources) and risk. The test program is also adaptable to new technologies (e.g., Internet of Things (IoT) solution space, Microservices) and innovative development approaches (e.g., agile development, DevOps).

2.2 (U) GENERAL METHODOLOGY

(U) TAC conducts most of its testing in a controlled test environment that closely simulates the operational environment in which the entities will function (e.g., data center, cloud). The fidelity of this controlled test environment is enhanced by the use of operational data (e.g., imagery datasets and other products) during testing. Simulated test data (also known as engineering data) is used only when operational data is unavailable.

(U) Most TAC testing employs a thread-based day-in-the-life (DITL) test approach, in which testers verify a new system for product quality (PQ), security, interoperability, and functional integration. Testers also verify the output, by executing typical user procedures to cause the system to perform the specific business processes or workflows it was designed to support. The procedures are selected to ensure all inter-system interfaces are exercised in order to confirm that the new development has been successfully integrated into the operational architecture and that it does not interfere with the functioning or stability of any other systems with which it operates. Because of the flexibility thread-based DITL testing offers, TAC uses thread-based DITL testing for the majority of its test activities, regardless of the specific type of development to be tested (e. g., traditional segment, web-based service, desktop application, or network).

(U) Testing in a controlled environment that closely replicates the operational environment is the most effective way to identify issues that could adversely affect the user and mission. Implementing a risk management strategy that focuses on the areas that pose the greatest risks increases the efficiency of resources. This approach is referred to as Risk-Based Testing (RBT).

(U) NGA strives to develop and field new and improved capabilities on a timeline that keeps pace with the end user’s mission requirements. NGA is increasingly implementing agile software development approaches to decrease the time between identifying and satisfying a need. Current NGA agile development contracts are using a variety of design approaches, development approaches, and delivery rhythms. TAC continues to adapt its processes and align its resources to support agile program requirements while accomplishing its primary function of preserving the integrity of the current systems in the NSG/ASG operational baseline. TAC has created a repeatable agile testing approach to

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

complement the cadence of an agile program. TAC’s strategy is to manage its resources to accommodate various development approaches and to provide the highest quality test support possible.

(U) External entities may be accepted for integration into the NSG/ASG environment, if TAC determines that the test results and test processes from the external agency were acceptable. TAC will review all test artifacts pertaining to the external entity and will determine if additional functional integration, interoperability, security, or image quality testing is needed. TAC will also review all test artifacts pertaining to an external organization’s security testing and will accept the segment for integration, if the test processes are equivalent to TAC’s security testing.

2.3 (U) TEST SUPPORT FUNCTIONS ROLES AND RESPONSIBILITIES

2.3.1 (U) Business Area Lead

(U) A business area lead (BAL) is designated for each NTO and NIQU business area. The roles and responsibilities for both NIQU and NTO BALs (unless specifically stated otherwise) are addressed throughout this NSG/ASG Enterprise TEMP; key roles and responsibilities include the following:

(U) Review proposed test strategy and test plan for technical accuracy prior to approval

(U) Review test objectives and plans for technical clarity, completeness, and achievability

(U) Provide technical guidance on testing methodologies

(U) Approves test requests and projects in Verification and Validation On- Line Tools (VOLT) for all activities in their areas of responsibility

(U) NTO: Conduct a post-test review to ensure that the test objectives were met and that the recommendation to the Program/Project Manager (PM) is clearly stated

(U) NIQU: Conduct a post-test review to ensure that the test objectives were met and that deliverables clearly document the results of the effort

(U) NTO: Share information at team meetings

(U) NIQU: Represent NIQU at NTO planning meetings

(U) NIQU: Coordinate with the Ops Lead(s) to ensure sufficient resources and schedule are available to complete the topic

2.3.2 (U) NTO Test Director

(U) A test director (TD) is designated for each NTO full-service test activity. The TD’s roles and responsibilities are addressed throughout this NSG/ASG Enterprise TEMP; key roles and responsibilities include the following:

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

(U) Establish or lead meetings as appropriate (e.g., Test Integrated Product Team (IPT) meetings, daily start-up meetings, daily wrap-up meetings)

(U) Coordinate with PM(s) to understand the entity under test and devise a collaborative test strategy

(U) Provide advice and best practices to help ensure delivery of quality products

(U) Attend test readiness reviews (TRRs) to provide test readiness status

(U) Review and comment on RFCs and Engineering Change Proposals (ECPs) for test impacts and adequacy of test planning and test resources

(U) Engage with the PM(s) on all design issues in order to better understand how to independently test capabilities being delivered to operations

(U) Engage with the user community and other stakeholders on all issues relating to system CONOPS and user/mission workflow in order to better understand how to independently test capabilities being delivered to users in operations

(U) Coordinate with NIQU topic leads to ensure integration of NTO and NIQU testing, where appropriate

(U) Develop test objectives for test activities

(U) Validate test objectives (during test execution) and revise objectives as appropriate

(U) Write test plans, daily status reports, and final status reports

(U) Coordinate with necessary stakeholders to ensure the availability of the necessary test environment and other test support services

(U) Execute tests and ensure compliance with approved test processes

2.3.3 (U) NIQU Topic Lead

(U) The NIQU topic lead is responsible for the successful execution of a test. The topic lead’s roles and responsibilities are addressed throughout this NSG/ASG Enterprise TEMP; key roles and responsibilities include the following:

(U) Develop and maintain a detailed task list, execution plan, and schedule

(U) Coordinate with internal and external partners and customers

(U) Execute the topic according to the approved plan and NIQU processes

(U) Provide status updates as required

(U) Coordinate the scheduling of all topic-related briefings and meetings

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

(U) Schedule and conduct all necessary meetings with the topic team

(U) Schedule and conduct a topic kickoff meeting to communicate to the team the schedule, assignments, and deliverable requirements associated with each task

(U) Schedule and conduct readiness reviews in accordance with the NIQU readiness process

(U) Promptly report and document any risks and issues associated with fulfilling scheduled deliverables to the NIQU BAL

(U) Ensure topic tracking data is captured in the NIQU portal and monitor any deviations from approved plans

(U) Coordinate with NTO TDs to ensure integration of NTO and NIQU testing, where appropriate

2.3.4 (U) Program Manager/Project Manager and/or Product Owner

(U) The PM or PO responsibilities include the following:

(U) Budget and contract the T&E need of the program throughout program lifecycle

(U) Align appropriate type of T&E activities according to the program’s lifecycle

(U) Manage developmental testing for the product

(U) Support TAC T&E activities, as identified in consultation with the TAC testing’s point of contact (POC)

(U) Work with the TD and, if applicable, the Test IPT to manage the test strategy

(U) Submit Requests for Services (RFSs) to obtain test support

(U) Include in the entity budget the funding to cover the cost of providing TAC with a suitable suite of hardware (physical or virtual) and fully licensed version of the software to support testing of the delivered entity for the life of the entity, unless a waiver is obtained from the D/TAC

(U) Include in the product budget the funding necessary to provide full-time, on-site product engineers to work in the NTO TE to provide all system administration and testing support for all activities pertaining to the entity’s hardware, software, or interfaces for the life of the entity, unless a waiver is obtained from the D/TAC

(U) Provide required documents (CONOPS and SRDs) and other relevant documentation as required (e.g., requirements traceability; operational scenarios; factory acceptance test (FAT) procedures; FAT results; test data; training, testing, and security documentation) to support TAC test

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

planning and execution and the Certification to Field (CtF) process

(U) Coordinate with the TD to ensure development contractors provide adequate T&E support

(U) When applicable, ensure appropriate members of the entity staff regularly and actively participate in the Test IPT

(U) Develop security test plans and procedures in accordance with the governing Director National Intelligence (DNI) and Department of Defense Intelligence Information System (DoDIIS) requirements

(U) Provide approved security test plans and procedures specific to each NTO test plan (each security test plan will be submitted to Cyber Security (TSR) for formal approval)

(U) Certify that the product has completed formal requirements verification, has successfully completed FAT (or equivalent testing) or a Pre-Ship Review and is ready for integration and operational testing

(U) Arrange for NTO test engineers to be trained on new entity capabilities prior to NTO testing

(U) For those systems requiring certification, provide input for the draft and final certification package to the NTO Intelligence Information Support Team (NIIST)

(U) Obtain National Image Transmission Format (NITF) or other necessary certifications prior to TAC testing

(U) Evaluate the entity for potential PQ risks and, if any, indicating the need for PQ testing in VOLT

(U) If appropriate, coordinate with NIQU for PQ testing during entity development to obtain early feedback on program compliance with applicable PQ standards and/or NORD KPPs 1, 3, and 4

(U) Provide NTO with two copies (Gold Copy) of the software baseline (if delivered on media) and a list of all commercial off-the-shelf (COTS) and government off-the-shelf (GOTS) products being delivered to the NTO TE

(U) Provide updated product baseline to the TAC Configuration Management (CM) Library, through TAC CM, upon receiving successful testing results

2.3.5 (U) Joint Interoperability Test Command

(U) The Joint Interoperability Test Command (JITC) responsibilities include the following:

(U) Provide personnel, co-located within TAC, to assess early insight into entity developments that will require JITC evaluation

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

(U) Monitor and validate the interoperability of entities being tested by TAC during Independent Verification and Validation (IV&V) testing

2.3.6 (U) Infrastructure Service Provider

(U) The Infrastructure Service Provider (ISP) responsibilities include the following:

(U) Providing and maintaining the collective NTO TE, including the current desktop test environment, the thin client test environment, the NCE virtual test environment and the cloud test environment (NTO/NIQU determines the configuration of the NTO and NIQU test infrastructure, including networks and the mechanism for coordinating changes to the test environments and obtaining NTO or NIQU approval is the RFS, submitted via VOLT)

(U) Provisioning and maintaining servers and desktop workstations to support any architecture in use by NGA (the concept for ISP maintenance of the test environments is the same as for operations, to include recapping of hardware as part of the recap schedule for operations; ISP maintenance also includes provision of the same upgrades of operating systems, server and database software, and other items as provided by the ISP to both mission and corporate operations; ISP provision of ISP software and software upgrades includes licenses, except for test tools)

(U) Test instantiations of new ISP hardware and software are provided for the test environment prior to deployment to the operational environment so that they can be tested and retained for future testing.

(U) Obtaining test support for ISP hardware and software, including the NGA desktop infrastructure, the ISP organization sponsoring or making the change submits an RFS via VOLT to request NTO test support

(U) For desktop infrastructure changes, the Core Service Division of NGA’s Service Implementation Group provides a test package that includes fundamental data element information, installation and configuration procedures, the Unit Test Report, and other applicable information, to assist in developing appropriate test objectives for the independent test.

(U) NTO’s ability to continue providing effective test support to NGA and the NSG/ASG depends heavily on the ISP ensuring the NTO test infrastructure remains current with operations.

2.3.7 (U) Cloud Service Provider

(U) The NGA Cloud Integration & Adoption (NCI&A) PMO is empowered to work across directorates to establish unified, transparent, and accountable processes and procedures and to provide the full complement of services needed by NGA and its customers for Cloud operations. The NCI&A PMO’s responsibilities

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

include the following:

(U) Provide oversight and guidance detailing architecture and design principles, as well as standards needed for technical conformance enabling Intelligence Community (IC) Enterprise vision and mission realization

(U) Coordinate, advocate, facilitate, and enable Federal and DoD Cloud initiatives and integration across NGA and in partnership with the IC

(U) Execute NGA’s required activities to transition to a hybrid Cloud computing architecture operational within IC ITE, Government Cloud Services (GovCloud), and Commercial Cloud Services (C2S), or in compliance with the DoD Cloud Directives

(U) Establish a procedure to address NGA-required activities for transitioning to a Cloud computing architecture operational within the Intelligence Community Integrated Test Environment (IC ITE)

2.3.8 (U) Consolidated Systems Engineering Review Board

(U) The Consolidated Systems Engineering Review Board (CS ERB) responsibilities include the following:

(U) Reviewing, prioritizing, assigning, and adjudicating all discrepancy reports (DRs) identified during post-FAT integration, test, and transition activities

(U) Reviewing, prioritizing, assigning, and adjudicating all inter-segment DRs for those elements that have not been transferred to operations o Inter-segment DRs identified during the development phase o All DRs identified during the conduct of system-level integration or site acceptance test and transition activities o Inter-segment DRs identified during operations for those elements that have not been transferred to operations.

2.3.9 (U) Applications Services Governance Board

(U) The Applications Services Governance Board (ASGB) is a forum that acts as the NGA’s central governing body for the NGA GEOINT App Store and other similar Programs of Record (PORs). A Terms of Reference (TOR) document for the ASGB, approved and coordinated with all NGA directorates and key staff offices, articulates the centralization of governance as it relates to mission value, prioritizations, and policy recommendations for application deployment to the NGA GEOINT App Store. However, the TOR does not discriminate or preclude other PORs from using this board for vetting and approving purposes. ASGB responsibilities include the following:

(U) Adjudicate or demote applications, web services, data, and any other

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

NGA development

(U) Promote into operations any application, web services, or data

(U) ASGB membership includes voting representatives from directorates across NGA, including:

Chief Information Office (CIO) and Information Technology Services (CIO- T)

Office of the General Counsel (OGC)

Analysis (A) Directorate

Research (R) Directorate

Source (S) Directorate

TAC

TS (Security and T&E)

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

3 (U) INTEGRATED TEST PROGRAM

3.1 (U) INTRODUCTION

(U) This section describes NGA’s integrated test program in terms of the three major categories of testing and the service offerings in each category. The NSG/ASG Enterprise TEMP presents these categories as composing a well-defined sequence of specific services that an entity is subjected to as it progresses from concept to deployment. However, because the Defense Acquisition System and the test and evaluation strategy emphasize the integration of test activities, the test sequencing is adapted to fit the needs of systems being developed in new development processes (i.e., DevOps), even for systems being developed under traditional acquisition approaches. As NGA adopts new acquisition strategies and development approaches, TAC expects to exploit opportunities for greater integration of the TAC testing services.

Therefore, the reader should bear in mind that although the sequence of specific service events presented in this section is useful for instructional purposes, it is not to be interpreted as a rigid prescription for the design and implementation of any specific entity’s test and evaluation plan.

(U) Appendix A contains a tabular summary of the various types of NGA’s integrated test program types discussed in this section and relates them to their Defense Acquisition System test categories.

3.2 (U) SUPPORT TO DEVELOPER TESTING

(U) As mentioned previously, the Defense Acquisition System defines two broad categories of testing that apply to NGA: DT&E, and OT&E. Figure 3.1 illustrates the major test events that compose these two categories as implemented by

NGA.

3.2.1 (U) Developmental Test and Evaluation, Phase 1

(U) Figure 1 depicts DT&E services as consisting of two phases that are primarily conducted by the development contractor under varying degrees of government PMO or PO supervision. This phase of testing begins with unit testing of individual modules of code and culminates in the conduct of a FAT for the final delivery. It is during FAT or an equivalent activity (such as a product demonstration of new capability) that developers accomplish formal requirements verification under government PMO or PO supervision. This includes verifying compliance with any required standards (e.g., NITF). Although most of the first phase of DT&E testing is internally focused, developers may choose to coordinate and conduct early interface tests (EITs) with other developers to provide an opportunity to assess a portion of their products’ capabilities as they interoperate with one or more external entities.

(U) The planning and execution of the first phase of DT&E testing is the responsibility of NGA PMOs or POs and their development contractors and is not governed by the provisions of this NSG/ASG Enterprise TEMP. However, NTO

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

TDs usually attend EITs, FATs, and product demonstrations as observers to gain familiarity with new systems and to better understand the entity-specific requirements for subsequent independent testing. NIQU also uses EITs, FATs, and product demonstrations as opportunities to provide development contractors with early feedback on the format compliance of their products, insight into systems' compression implementation, and in some cases, assessment of the accuracy of the support data with which the products are being populated. NTO’s and NIQU’s early engagement with PMO/PO reduces risk during later formal entity testing.

(U) In the traditional acquisition process for NSG/ASG mission capabilities, the event normally marking the transition from the first phase of DT&E to the second phase of DT&E is the pre-ship review (PSR). Following a successful PSR, all DRs identified as open at the PSR or written subsequent to the PSR are entered into the NGA DR system (Remedy) for tracking and adjudication by the appropriate engineering review board or configuration management board.

3.2.2 (U) Developmental Test and Evaluation, Phase 2

(U) Because developer test environments do not include direct interfaces with all of the NGA entities they integrate with, developers will generally not be able to fully verify successful integration of their products with the operational NSG/ASG enterprise architecture. Therefore, the second phase of DT&E is typically conducted at the NTO TE or another NTO-approved facility. It begins with the delivery of the new or enhanced entity to the NTO TE (or other approved site) where developers perform installation, checkout, and test (IC&T) of their entities to ensure that they have been properly configured and can communicate with all applicable systems in the test environment. The NTO and NIQU test engineers will observe and provide the PMO/PO with a facilitator to assist in any environment related issues. Developers or PMO testers are then afforded the opportunity to evaluate the proper integration and functioning of their entities by leading and conducting one or more integration tests (ITs). The second phase of DT&E concludes with the completion of the last IT, at which time the PMO/PO turn the entities over to the NTO for independent, enterprise-level OT&E.

3.3 (U) GOVERNMENT TESTING

(U) The T&E that occurs after the second phase of DT&E is referred to in this NSG/ASG Enterprise TEMP as “government test,” which is equivalent to the Defense Acquisition System category of OT&E. The earliest and most methodical government test events are those conducted by TAC (NTO in collaboration with NIQU, OCIO, and JITC). If applicable, after this early testing has confirmed that the product is stable and functional and that it poses no risk to the operational architecture, user organizations conduct the remaining OT&E events, usually with the assistance of the sponsoring PMO at a specific site location. The government testing that is conducted at the NTO TE can be combined with user testing when the users are included in the various test and evaluation service offerings that make up the specific T&E strategy for a PMO.

13 June 2018

Figure 1. (U) NGA Enterprise-Level Integrated Test Program

3.4 (U) QUALITY ASSURANCE SERVICES

(U) This section describes, at a high level, the quality assurance (QA) services provided by the divisions within TAC. Those services fall into six categories:

Engineering Analysis

Test Environment Hosting

Test Execution

Readiness

Security Assessments and Scans

Business Process Management

3.4.1 (U) Engineering Analysis

(U) The Engineering Analysis service is the application of scientific and analytic principles and processes to identify the properties and condition of the entity or GEOINT data under review. This service contains studies, requirements analyses, operational issue investigations, quick look assessments, site acceptance tests (SATs), standards and scales development, consultations, and performance profiling.

3.4.1.1 (U) Studies

(U) The Studies subservice provides an analysis that is performed through research or observation of the entity or data to understand the behavior of the entity. Through this subservice and the analysis performed in this subservice, TAC may identify the need to conduct a small-scale preliminary assessment to evaluate feasibility and performance. This analysis is to validate a design or desired outcome of a technological update with respect to sensor data quality

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

(e.g., the cloud environment) and modifications to functionality, interoperability, sensor data quality, and utility requirements for systems and data.

3.4.1.2 (U) Requirements Analysis

(U) The Requirements Analysis subservice provides an analysis of the requirements to ensure that the GEOINT capabilities being delivered provide sensor data quality, interoperability, performance, utility, and data integrity.

Through this subservice, TAC provides the PMO with an independent review of relevant requirement documents.

3.4.1.3 (U) Operational Issue Investigations

(U) The Operational Issue Investigations subservice provides support for resolution of operational sensor data quality discrepancies related to processing, exploitation, or dissemination.

3.4.1.4 (U) Quick Look Assessments

(U) The Quick Look Assessments subservice provides abbreviated assessments (e.g., fewer assessors, datasets, activities, or time) conducted to provide insight into the behavior of an entity or data type under specific circumstances.

3.4.1.5 (U) Site Acceptance Test

(U) The Site Acceptance Test (SAT) subservice provides users the opportunity to gain hands-on experience with a new capability, allowing them to qualitatively assess how well the capability meets operational needs and expectations, and to assess interoperability, security attributes, and performance before transition to operations. SATs occur on servers outside of the NTO TE (i.e., in operations), which TAC does not control. TAC personnel may observe, participate in, or facilitate (but not lead) the testing. TAC resources (e.g., workstations, subject matter experts (SMEs)) may be used to support testing.

(U) An SAT follows successful completion of all other government testing. It may include aspects of entity testing such as functionality, interoperability, PQ, security, and performance testing.

(U) An SAT can be as formal as desired by the user organization. For a well-understood system with few changes, the SAT may simply consist of ad hoc exercising of the product by one or more users. For new entities or major upgrades, the user organization may elect to have a fully scripted test to ensure all major parameters are exercised and to develop user confidence in the new capability. Users may conduct the SAT with development contractors available to provide SME guidance.

3.4.1.6 (U) Standards and Scales Development

(U) The Standards and Scales Development subservice provides the documentation of processes and procedures that are necessary for the NSG/ASG to be compliant with NORD KPPs 1, 3, and 4. The development of quantifiable measures help to define the levels of interpretability, utility, and

13 June 2018

UNCLASSIFIED//FOR OFFICIAL USE ONLY

interoperability provided by GEOINT sensor data.

3.4.1.7 (U) Consultations

(U) The Consultations subservice provides scientific, engineering, statistical, or technical interactions on hardware, software, processes, methodologies, and innovations.

3.4.1.8 (U) Performance Profiling

(U) The Performance Profiling subservice provides analysis of application performance to identify potential bottlenecks or support application optimization. This subservice may include ongoing performance audits to evaluate application performance against established benchmarks.

3.4.2 (U) Test Environment Hosting

(U) The Test Environment Hosting service is the provision of test environment resources to test entity capability across interfacing test systems; the PMO or developer is responsible for conducting the test. This service contains Inter- Segment Message Exchanges (ISMEs), EITs, Enterprise Tests (ETs) and System Demonstrations (SDs), IC&Ts, ITs, Assisted-Service IV&Vs, and Assisted-Service Regression Tests.

(U//FOUO) Government testing of NGA mission capabilities includes joint operational testing or demonstrations with external organizations that interface with the NGA enterprise architecture. Joint testing has essentially the same objectives as internal NGA testing: to ensure that modifications accurately respond to formal requirements and that they will not adversely affect the joint operational environment. The National Reconnaissance Office (NRO) is the NGA’s primary mission partner; the types of testing required for various types of changes to the joint NRO/NGA enterprise are documented and coordinated via the joint RFC process and several joint planning forums. When changes to the NRO side of the enterprise are planned, the NRO test program drives the types of test activities and the schedule, with TAC participating.