(U) Annex_G_Cloud_Data_Guidance_v5.2.pdf

UNCLASSIFIED

ANNEX G:

CLOUD DATA GUIDANCE

Version 5.2 30 June 2018

Annex G: Cloud Data Guidance ii

TABLE OF CONTENTS

Section Description Page

Revision Summary ............................................................................................................................................................. vi

1.0 Introduction

1.1 OpenDataStore

1.1.1 ODS Content Accounts

1.1.2 ODS Service Providers

1.1.3 ODS Assessment & Authorization (A&A)

1.1.4 ODS Exceptions

1.2 Data Access

1.3 Analytical Data Lake

1.4 Conditioning

1.5 Data Migration Categories

1.6 Content-ID

1.7 Roles

1.8 Data Inventory

2.0 Data Guidance

2.1 General Data Guidance

2.2 Mission Data Guidance

2.3 Activity Data Guidance

2.4 Analytical Data Guidance

2.5 Corporate Data Guidance

2.6 Organizational Data Guidance

3.0 Data Migration

3.1 Background .......................................................................................... Error! Bookmark not defined.

3.2 Data Migration Guidance

3.2.1 Data to the Cloud Process

3.2.1.1 Phase 1: Identify Content

3.2.1.2 Phase 2: Approve Request

3.2.1.3 Phase 3: Create/Implement Cloud Change Request

3.2.2 Conditioning for File/Object based Data

3.2.3 Conditioning for Databases

3.2.4 Production Components

3.2.5 Cloud Migration Lessons Learned

3.3 Metadata Management

3.3.1 Challenge #1: Increasing Discoverability of Mission Data

3.3.2 Challenge #2: Securely Sharing Information with External Partners iii

3.3.3 Tagging Mission data to Overcome Challenges

3.3.4 Metadata Management Guidance

3.3.4.1 NGA Metadata Catalog

3.3.4.2 EDH and TDF Use Criteria

3.3.4.3 EDH ................................................................................................... Error! Bookmark not defined.

3.3.4.4 TDF ................................................................................................... Error! Bookmark not defined.

3.3.4.5 TDO ................................................................................................... Error! Bookmark not defined.

3.3.4.6 Assertions ...................................................................................... Error! Bookmark not defined.

3.3.4.7 NGA Web Service Registry

4.0 Data Access Vision

4.1 Responsibility to Provide

4.2 Authorized Data Access

4.3 Data Entitlements

4.4 Data Privileges

4.5 Legacy Data Access

4.6 Legacy Network Access

4.7 Automated Data Access

4.7.1 Access by Authorization Attributes

4.7.2 Access by Role

4.8 Current State of Automated Data Access

4.8.1 S3 Data Access Roadmap

4.8.2 Database Data Access Roadmap

4.9 Impacts

4.9.1 Application Development

4.9.2 Enterprise Utility

Appendix A: Definitions

Appendix B: References

Appendix C: NMF Mandatory Fields

C.1 NGA NMFv3 Mandatory Fields

C.2 NGA Analysis and Source Mandatory Fields

C.3 NGA X Organization Mandatory Fields

Appendix D: NMF Cross Reference with EDH and TDF

Appendix E: IC Mandatory Fields

E.1 Electronic Records Management (ERM)

E.2 Trusted Data Format (TDF) E.3 IC Enterprise Data Header (IC-EDH) iv

Appendix F: NGA Metadata profile

Appendix G: Data in the Cloud – Example Cases

G.1 Data Storage Architecture Evolution – Object Storage (S3) Example Case

G.2 Data Storage Architecture Evolution – Database (RDB) Example Case

Appendix H: ODS S3 Bucket Naming Standard v

List of Figures

Figure Description Page

Figure 1: Legacy and Target Data Architecture

Figure 2: OpenDataStore Services

Figure 3: A&A Examples (TMS and PeopleSoft)

Figure 4: Cloud Migration Dependences

Figure 5: Phase 1 - Identify Content (assign Content-ID)

Figure 6: Phase 2 – Approve Request

Figure 7: Phase 3 – Create/Implement Cloud Change Request

Figure 8: Production Architecture Components

Figure 9: Concept: Metadata and Data in C2S

Figure 10: Concept: NGA to IC GovCloud Exchange

Figure 11: EDH Graphic

Figure 12: TDF Graphic

Figure 13: NGA Database-Based Discovery via Web Service Tagging

Figure 14: Legacy Data Access

Figure 15: Security Authorization via Attribute Criteria (Granted Example)

Figure 16: Security Authorization via Attribute Criteria (Denied Example)

Figure 17: Entitlement Groups Concept

Figure 18: Statutory Restrictions (Use Case)

Figure 19: PUM Entitlement Group (Use Case)

Figure 20: S3 Data Access Flow (Vision)

Figure 21: S3 Data Access for Trusted Activities

Figure 22: Database Access Flow (Envisioned)

Figure H-1: S3 Now – Current State Architecture

Figure H-2: S3 Next - Transitional State

Figure H-3: Cloud OpenDataStore, S3 Target State

Figure H-4: Database As-Is Example

Figure H-5: Option 1 - Lift & Shift Rehost

Figure H-6: Option 2 RDS Solution

Figure H-7: Eventual Target Cloud Database

List of Tables

Table Description Page

Table 1: NGA Unified Data Strategy Goals

Table 2: Service Provider Roles and Responsibilities

Table 3: Data User Roles and Responsibilities file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339522 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339523 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339532 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339536 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339537 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339538 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339539 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339541 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339542 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339543 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339544 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339545 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339546 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339547 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339548 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339549 file://///coe.ic.gov/Home/nce1/nga_home_214/ji67714/COEProfile/Desktop/Annex%20G_Cloud%20Data%20Guidance_v5.2.docx%23_Toc500339550 vi

REVISION SUMMARY

Version Revision Date Revision Summary

1.0 12/31/2015 Initial Release

2.0 02/29/2016 Updated storage guidance. Added migration and metadata guidance.

3.0 4/19/2016 Moved storage guide materials to white paper, revised most sections.

4.0 6/30/2016 Minor edits and corrections. Added information on Open Data Store. Moved cost info to another annex.

4.1 08/05/2016 Updated classification/portion marking to support push to SBU

5.0 9/30/2016 Added data movement process descriptions, and chart. Edited migration sections for consistency.

Added data conditioning explanation. Revamped S3 Example Case. Added Database Example Case, and S3 Bucket Naming Standards.

5.1 1/6/2017 Added data access information. Changed ‘Data Owner’ to ‘Data Custodian’. Revised migration information. Extensive revision to Appendices.

5.2 5/31/2018 Revised A&A and data to the cloud procedures. Replaced references to Content Accounts with references to Open Data Store.

1.0 INTRODUCTION

This Annex provides cloud data guidance, establishes a common terminology for data service of-ferings, provides data migration guidance, and describes how NGA will manage its metadata in the OpenDataStore(ODS) cloud. Key data requirements and recommended best practices are identi-fied for programs adopting cloud services. Cloud data guidance is guided by the NGA Unified Data Strategy goals outlined in Table 1 below.

Table 1: NGA Unified Data Strategy Goals

Goals Definitions

Find the Data Users across NGA, the IC and DoD are able to intuitively discover, access, and retrieve the geospatial data and content

Free the Data Data are required to be separated from individual applications, systems and information silos

Protect the Data Data accessibility is based on user role and access control data attributes, reflecting user rights, roles and data usage; available when and where needed

Automate the Data Data are conditioned with metadata throughout its lifecycle to enable automated access, discovery, pedigree, provenance, retrieval, and analysis

Unify the Data Integrated Intelligence is unified across all boundaries: architectures, security domains, geographic locations, and mission, corporate and functional operations

As NGA migrates activities to cloud environments, data guidance will help define the data architec-ture, tools, and processes required to achieve success. Overarching concepts are that NGA data will be:

• Managed in accordance with NGA data management, retention and records management policy,

• Encrypted in transit and at rest,

• Available to the National System for Geospatial Intelligence (NSG) and Intelligence Commu-nity (IC),

• Cataloged and indexed for discovery (e.g., using Enterprise Catalog and search engine ser-vices)

• Tagged with handling and access information, and

• Exposed via services.

There are distinct intranets serving the IC and DoD communities, and the classified domains are air-gapped. The cloud in this guidance includes:

• Top Secret Cloud (TC)

• Secret Cloud (SC)

• Unclassified Cloud (UC).

In addition, there are interrelated IC cloud resources being the IC GovCloud and IC ITE. The data storage and data services portion of the latter is in the TC’s OpenDataStore.

The key concepts introduced will enable the reader to better understand the guidance for: ODS, Data Access, Analytic Data Lake, Conditioning, Data Migration Categories, ContentID, and Roles.

Information is provided on key concepts and Chief Data Officer (CDO) direction on how to handle data and metadata in the cloud environments, particularly in the OpenDataStore.

1.1 OpenDataStore

The ODS represents an architectural evolution in data management. In the past, data belonged to and was stored and managed by applications often in semi-private data structures. With ODS, data will be managed by enterprise data services and maintained in common areas accessible to all authorized users, as illustrated in Figure 1 and further described in the Design White Paper on Data Architecture.1

The ODS contains both persistent storage of data accessible to end users for normal work and analy-sis, and data services. It is a highly inclusive con-cept covering all data stored for normal end-user use. It encompasses all means and methods to store data, from flat files, to relational databases, NoSQL databases and complex structures like data warehouses and Analytical Data Lakes. It also in-cludes all formats, from structured relational, to object, to triple-store, to unstructured files.

ODS is an amalgamation of two concepts: Open Data and the Data Store. A Data Store is an inclu-sive repository, and the Open Data movement ad-vocates that data should be freely available to everyone to use.

Data can be stored in UCs Amazon Web Services (AWS) or TCs Commercial Cloud Services (C2S) S3 buckets or in file system implementations on C2S Elastic Block Services (EBS) raw data storage area. Relational and NoSQL databases may be supported by C2S Rela-tional Database Services (RDS) or with C2S Dynamo. Other services can be set up in an C2S Elastic Compute Cloud (EC2) in-stance with EBS, as shown in Figure 2.

Data transformation and conditioning ser-vices are among the many types of data services that NGA can stand up. While the Open Data movement is about making data free and open to the public, that can-not be done with classified and controlled

1 Cloud Data Storage: Data Architecture Pattern Choices, 2018.

Figure 1: Legacy and Target Data Architecture

Figure 2: OpenDataStore Services https://intelshare.intelink.ic.gov/sites/nga-cloud/SiteAssets/SitePages/Guidance/NGA%20Cloud%20Guidance%20v5.2/Design%20White%20Paper%20-%20Cloud%20Data%20ArchStructure%20Choices_June_2018.pdf information. The key elements for implementing open data services are:

• Automate the access via Policy Enforcement Points (PEPs) so if the user credentials meet the metadata attributes, the user can access the data.

• Eliminate technical barriers (i.e., organization by application, and use of private data for-mats) to mission data, beyond that required by the classified networks.

• Design data access services that meet user needs, including services that allow dataset wide discovery and access when appropriate.

The ultimate objective is to provide all data as a Managed Service. Initially, this will take the form of providing a shared environment to store data via the OpenDataStore. The ODS will work much like the NGA enterprise file storage and enterprise database clusters of the NGA Data Centers:

• The data infrastructure will be set up for the Application Service Provider (ASP) programs with:

o Data infrastructure access rights and networking o Actual instances for ASP data storage (database or file storage)

• The ASP will be able to configure that data storage instance, and:

o Define role based access controls internal to their storage solution o Configure their storage solution to support their activity needs

One of the goals of the CDO is to ensure that the mission and activity capabilities are not adversely affected by the cloud data migration. As such, initially the CDO will work to enable activities to ac-cess their data the same as in the past, including direct connections to data storage.

Over time, the CDO, working with the ASPs, will work to transition all data access to web-based services instead of application-direct access to the data stores. Eventually, the ASPs will be con-sumers of the data services instead of working directly with the data stores.

1.1.1 ODS Content Accounts

The CDO is creating an ODS account for mission data and a corporate ODS account of corporate data for each security fabric on C2S. All mission data and corporate data will be stored in the ODS.

While other data categories are not required to go in predefined ODS accounts, there is nothing preventing them from being stored there, should the Data Custodian (DC) so choose. There are many instances where activity data shares space with mission data in a common database, and the DC may prefer to store the entire activity data in one account.

1.1.2 ODS Service Providers

There are three (3) service provider roles with respect to the ODS, as identified in Table 2.

Table 2: Service Provider Roles and Responsibilities

Provider Type Rights & Responsibilities

Infrastructure Service Provider (ISP) Creation of AWS accounts.

Configuration/Management of the AWS Identity Access Management (IAM) services in the AWS console in any given account.

Configuration/Management of the network services in the AWS console in any given account.

Data Service Provider (DSP) Management of the ODS data storage and data services.

Configuration/Management of all the AWS services in the NGA content.

AWS accounts except those managed by the ISP.

Application Service Provider (ASP) Create/Read/Update/Delete (CRUD) operations in the various data stores needed by their activity.

CRUD operations against the NGA Metadata Catalog.

Configuration of database accounts for databases related to their activity.

Configuration/creation of database structures and database code.

The ASP will not have the right to access the AWS console, or to spin-up or spin-down instances. They will just be able to access and perform whatever actions granted to them against the specific instances they need to access.

1.1.3 ODS Assessment & Authorization (A&A)

The ODS extends the concept of the shared security model under which C2S operates, such that:

• C2S (i.e., Amazon Web Services) has the responsibility for securing the hardware and vir-tual machines.

• ISP has the responsibility for NGA ODS accounts and ensuring IAM and network security.

• ASP has the responsibility for the A&A of their activity, the storage service they are operat-ing, and configuring within accounts, databases and other ODS aspects.

Security has stated that activity boundaries do not align with ODS C2S account boundaries, and that the activity A&A includes all parts of the activity architecture, not just the parts within their own C2S account. Therefore, since the ASP is configuring their actual data instance in the ODS, they are responsible for showing that the data configuration meets ICD 503 requirements.

Moving a database from an NGA account or Virtual Private Cloud (VPC) on C2S into the ODS VPC on C2S will not impact or invalidate the current A&A of an NGA activity, so long as the basic envi-ronment controls are the same. As these environment controls are set up and controlled in the same configuration for all NGA accounts, this should not be an issue.

Figure 3 illustrates two A&A examples. In the first example, the Target Management System (TMS) activity has an ODS account where web and application server instances reside. The TMS database is on an EBS backed EC2 instance in the ODS mission account, so the TMS architecture spans two accounts, and the TMS activity is responsible for the A&A for its entire architecture.

Figure 3: A&A Examples (TMS and PeopleSoft)

The Data Service Provider (DSP) would spin up the TMS instance, using the standard Oracle with a spatial Amazon Machine Image (AMI) and provide the secure access credentials. The activity would then set up individual developer staff logins within that instance and any other configura-tions required to be ICD 503 compliant in order to achieve Authority to Operate (ATO), and that Oracle database instance. Since they are using the standard AMI, ATO is bounded by the ASP activ-ity.

In the second example, the PeopleSoft application and web servers are in a PeopleSoft account, but the PeopleSoft database is on the RDS managed service in the ODS. The DSP stood up the RDS in-stance for the ASP, but the ASP is responsible for setting up their schema, setting up developer ac-counts in Oracle along with other infrastructure tasks. The ASP is still responsible for the A&A of the entire PeopleSoft architecture, even though it spans the PeopleSoft account and the ODS.

In both cases, there will have to be test cases that prove that the ASP NIST 800-53 controls are sat-isfied. In summary, the ISP, the DSP and the ASP will each have NIST 800-53 controls that will need to be satisfied through ATO test cases. The controls can be unique to a service provider, or shared as negotiated with CIO-T. However, A&A boundaries are not determined by hardware nor account boundaries. For additional information on the cloud A&A process, see the Cloud Security Annex.

1.1.4 ODS Exceptions

The vision for the ODS is to support the data needs of all NGA missions and activities. However, it is understood that there may be exceptions based on security, policy or technical needs. For activ-ities that need more rights with respect to their data, or data that needs greater protections than are possible in a common ODS account, the CDO will work with the activities and the DC on a case-by-case basis to determine what is needed to resolve their issues, and grant exceptions if neces-sary.

1.2 Data Access

Data access has the following control constraints and features, per DNI Directives:

• There is an automated way to discover and access data. All data is metadata tagged, person identity and roles are associated with accessibility attributes, and automated rules and pro-cesses use these tags to automatically enforce accessibility and dissemination policy.

• “Responsibility to provide” is achieved. The default stance on data is that it is shareable, un-less there is a specific dissemination control, exemption or constraint preventing it from being shared. However, “responsibility to provide” does not negate our responsibility to protect sources and methods.

• People with the “appropriate security clearance and an assigned mission need” are able to access data. Entitlements are how “need to know” is implemented. No other “need to know” check is performed – it is implicit in the request.

1.3 Analytical Data Lake

An Analytic Data Lake is a popular approach for analysis of large amounts of data, in that: “A Data Lake is an attempt to bring together physically available data stores in such a way as they can be easily accessible to users.” 2 Data Lake characteristics of include:

• Data is sub-transactional (append only) or non-transactional (read only)

• Data is used by multiple user communities

• Data is of a scale/volume that won’t economically fit into a relational database

• Data is in a native format (i.e., raw)

A Data Lake is an analytical tool. It allows the user to put massive amounts of disparate data in one spot and run analysis across all of it. It is not, however, a data repository for all data at NGA. For example, the production databases for Foundation GEOINT data are up-dated thousands of times per day, so these would not be able to exist in an “append only” or “read only” structure.

A Data Lake might not be permanent, in that, it would be created to just answer a specific ques-tion. The data sets needed would be loaded, the question asked and answered, and then the Data Lake taken down. So, rather than one, massive, permanent Data Lake at NGA, there could be many Data Lakes, created and taken down in response to analytic need.

The implementation of a Data Lake would be via a Hadoop, Spark or an Accumulo cluster set up in C2S. Also, note that the data risks becoming non-discoverable if not properly con-ditioned and metadata tagged.

1.4 Conditioning

Conditioning is the formatting and metadata processing (i.e., cataloging and tagging) of ingested data. Ingest conditioning is defined as that metadata and formatting work minimally required to allow successful load into persistent storage, including the Enterprise Data Header (EDH) han-dling information. The scope of conditioning is ‘just enough’ to achieve ETL processing without impeding the load or causing unnecessary rejection of the ingest data. Conditioning may continue to be performed at a later time to include:

2 The Confusion and Hype Relating to Data Lakes, Gartner, 2014.

• External metadata enhancements (i.e., populating metadata catalogs, search engine indices, and semantic web links).

• Contextual metadata improvements (i.e., database metadata, object wrappers and headers, and in-line tags, such as XML micro formats).

1.5 Data Migration Categories

Data migration categories include:

Mission Data: Data used in performance of the NGA mission, which typically would be GOLD copy or record data. This would include source data (e.g., host nation publications, imagery, commercially purchased datasets), the actual production data stores (e.g., Struc-tured Observation Management (SOM), Aeronautical Migration System (AMS) and NGA products (e.g., Digital Nautical Chart (DNC), Intel Reports, Geodesy Models, Navigation Planning (NAVPLAN) Charts, and Fleet Guides).

Activity Data: Data (e.g., code, configuration files, logs) other than mission content, needed to make an activity/service run.

Analytical Data: Ephemeral working data/products that analysts are analyzing on their desktops or in an analytical tool/service. It is not considered mission data until it is final-ized/published to the repository and/or the customer. The published product created from the results of analytical data is mission data.

Corporate Data: Data used at the enterprise level that enables the business of NGA, but is not directly part of the mission. This would include: PeopleSoft data, Identity and Access Management (IdAM) data, Personal Identifying Information (PII), financial systems, enter-prise services, workflow and process tools.

Organizational Data: Data needed for the business process of individual organizations.

This would include: International Standards Organization (ISO) processes, organization charts, briefings, and data that organizations put in shared organization drives.

1.6 Content-ID

Content-ID is a way to categorize a related set of activity data, for example: “Digital Nautical Charts” or “Absolute Gravity Survey.” A Content-ID is a unique identifier created when a logical set of activity data is registered in the ATLAS’s Data Inventory. Each data set identified by a Con-tent-ID has a DC and Data Steward (DS), retention/lifecycle policies, and access control policies.

The Content-ID is used to track data throughout the migration process. Virtual folders within S3 buckets are created based on Content-ID, and access control and lifecycle policies are then applied to those Content-ID virtual folders based on the rules for that logical set of data.

1.7 Roles

Data is acquired, managed and provisioned through various user roles, as identified in Table 3.

Table 3: Data User Roles and Responsibilities

User Title User Role

NGA Chief Data Officer (CDO)

Responsible for Data Governance at NGA in general, and for all ODS data and accounts in particular. The CDO ensures that the NGA ODS accounts are properly maintained and configured.

As the owner of the C2S CDO Mission Data and Corporate Data, the CDO must approve any new Data or Data Services being added to the Account(s). The CDO also covers direct data migration costs, to move data to the cloud.

Data Custodian Responsible for bringing the data into the agency. Often known as the Data Owner, this might be the person who:

Authorized the expenditure of funds to buy a set of data

Authorized an Analyst to retrieve a set of free data

Approved a deal with a host nation or other entity to obtain data

Is in charge of the group that produces the data/product

One way or the other, the person, with the authority, that is responsible for that set of data at

NGA.

The Data Custodian is responsible for the ongoing maintenance of his/her data. This includes budgeting for storage costs (including in the cloud), paying storage bills, setting retention policies on the data, and making sure data sets are kept up-to-date or retired as appropriate.

Data Custodians must approve of every data set belonging to them being migrated so that they are aware of the scope of their data and can execute their responsibilities.

Data Stewards Charged with making all information collected and all analysis produced by an IC element available for discovery by automated means by authorized IC personnel 3. In doing so, they:

Define and manage access and handling policies for their data set so that the data is treated in accordance with applicable legal and policy constraints.

Issue Data Management instructions throughout the data life cycle.

Determine if the existing cloud services provide access control mechanisms that meet their security needs.

Data Technical Staff Perform technical actions on behalf of the data steward. For example, they may prepare and route data to its intended destination in the desired format with appropriate Smart Data labels applied. Or, perform metadata tagging research. 4

Data Knight/Champion

Often the person delegated to attend the DMWG and coordinates with the Data Custodians and Data Stewards in their organization to answer data related inquiries.

Activity Data Manager

This role is assigned to one or more Activity automated systems, not a specific user. Gives authorized system users rights to manage (Create, Read, Update and Delete) that Content on behalf of the Data Custodian. Systems that are not Activity Data Managers will only be given Read access to data. For example, the AMS Activity is the assigned Activity Data Manager for the AMS Database per the AMS Data Custodian. If AMS were not, the AMS software could not be used to insert, update and delete data in the AMS database.

Data Scientists Determine the relevancy of data with respect to mission need and investigate and prototype data analytic approaches to yield useful analytic conclusions from one or more IC data sources. They may provide input into the indexes, extractions, or correlations to be performed against their submitted data. Note that this role is intended to also cover roles of other IC users (e.g., data specialists, developers) who perform similar activities aimed at extracting knowledge from data.

Developers Design, develop, and implement services and data flows to support needs.5 They interact with/use other services to develop, integrate, and deploy capabilities. May also be known as software solution engineers and by other terms.

3 IC Directive 501, Discovery and Dissemination or Retrieval of Information within the IC.

4 Defined in ODNI ES 00594 as “An organization that is responsible for executing data-related tasks on behalf of an originating element. These tasks may include collecting, tagging, and processing data.” Note that the Data Custodian organization may be different than the originating organization.

5 IC ITE Data-Centric Management Principles, v02, 2013 https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/X/XK/XKD/CCF/Shared%20Documents/ICD%20501%20-%20Discovery%20and%20Dissemination%20or%20Retrieval%20of%20Information%20within%20the%20IC.pdf https://ideonline/sites/sein/csi/edm1/Documents/Archive/NGA-E-EDM/IC-Related%20Efforts/IC%20ITE%20General/IC%20ITE%20Data%20-%20Centric%20Management%20INWORK%20Version%2002v5%20JMS%2020130916.docx

For a user role(s) example, consider the NGA Aeronautical Safety of Navigation group (SFA) DAFIF. A DAFIF Program Manager (PM) is in charge of the product line. The mission DC for DAFIF would be the head of SFA, and the DAFIF PM would be the Data Steward.

1.8 Data Inventory

The NGA’s Data Inventory is maintained in the ATLAS and contains consolidated information col-lected about NGA cloud migration data6. The ATLAS Data Inventory contains:

• Data status information: Data Status, Migration Status.

• Basic information: Name, Description, Category and Classification of the data.

• Custody: Organization it ‘belongs’ to and Point of Contact (POC).

• Storage information: data storage network(s), type of data, method of storage and data size.

• Relations: Production, Dissemination and Consuming activities (systems) that use, produce or host the data.

To view all the information available on a given data set, one can either go into that specific record or download the Authoritative GOLD copy spreadsheet and see it all in one view. A data dictionary is also available.

The data inventory is continuously updated and new functionality will be added over time. Every migrating activity must provide this information to allow for a successful migration. Data will not be stored in the cloud unless the CDO and/or DC sign off. Continued effort is required by NGA data stewards to ensure the accuracy and completeness of the Data Inventory.

2.0 DATA GUIDANCE

2.1 General Data Guidance

Guidance: Data will be managed in accordance with NGA data management, retention and records management policies.

Data Stewards (DSs) and DCs will set lifecycle rules on their ODS data repositories. Those lifecycle rules will follow National Archives and Records Administration (NARA) guidance and will indicate when data/files/objects can be moved to a near-line archive, and when they can be moved off-line.

Electronic records management metadata tags will be applied during conditioning to allow for ef-fective records management.

NGA data management policies shall be followed, as documented in the NGA Data Management Guidance and Unified Data Strategy. 7

Guidance: Data services built for the cloud shall be delivered suitable for use as enterprise services, whenever applicable.

Conditioning, data load, data access and other data services for applications built for the cloud that may possibly have general (i.e., common) utility to the community are to be delivered so as to be suitable for use as enterprise services. These common data services shall meet standards for web and SOAP services and be documented in the Service Registry.

6 Cloud Migration Data Dictionary 7 NGA Data Management Guidance for EDM, 2016; and, NGA Unified Data Strategy, v2.0, 2014.

https://mydata.coe.ic.gov/ee/request/folder?id=41238 https://ngaonline/sites/CIO-T/TC/TCA/DMWG%20Documents/Cloud%20Data%20Migration%20and%20Coordination/Migration%20to%20the%20Cloud/Data%20Dictionary%20for%20AR%20Data%20Inventory/Data%20Dictionary%20for%20AR%20Data%20Inventory.xlsx https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/coo/Governance/SGC/Information%20Management/References/CDO%20Docs/NGA%20Data%20Management%20Guidance.docx https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/coo/Governance/SGC/Information%20Management/Homework/Content/Unified%20Data%20Strategy%20v2.pdf

Guidance: Data at Rest (DAR) and Data in Transit (DIT) will be encrypted.

Data at rest (DAR) and data in transit (DIT) across and between environments will be encrypted.

NGA follows DoD’s Cloud Computing Security Requirements Guide (SRG) v1r2, NIST 800-53r4, and CNSSI 1253, all of which provide guidance for DIT/DAR. An Analysis of Alternatives (AoA) has been conducted to evaluate encryption and key management solutions, which will be identified through the NGA acquisition process. Annex I: Cloud Security includes encryption details.

• Data encryption for Unclassified Commercial Cloud (U-AWS). The U-AWS will store, pro-cess and transmit both Public Releasable Information and Controlled Unclassified Infor-mation (CUI) data (i.e., LIMDIS). The U-AWS will be accredited to process up to Impact Level 4 data. The DoD SRG v1r2 requires that: 1) for all Information Impact Levels, encrypt all DAR, and 2) for CUI data, use a FIPS 140-2 (minimally Level-1) encryption solution.

• Data encryption SC2S and C2S. The Secret Commercial Cloud Services (SC2S) and C2S en-cryption for DIT/DAR will be accredited in accordance with ICD 503. An objective of the aforementioned AoA is to implement enterprise encryption across U-AWS, SC2S and C2S.

2.2 Mission Data Guidance

Guidance: Mission data will belong to the NSG community.

Data does not belong to a specific group or program, but to the National System for Geospatial In-telligence (NSG) community as a whole. Data will be shared unless restricted by access or policy.

Whenever applicable, data must be accessible to the entire community including unintended but authorized users. Data access will be determined by the access associated with data itself and the roles and permissions of those requesting access to the data, following NGA releaseability policies.

The DS role does include marking and otherwise associating access and interpretation rules to the data, but a data steward should not support actions that would restrict access by an authorized user with a need to know, even if the user or use was unanticipated.

Guidance: Metadata tagging minimum standards will be enforced.

Conditioning and validation processes will be part of the workflow; items that fail (e.g., have insuf-ficient handling information or not-discoverable) are so indicated by status metadata or are routed to separate areas (i.e., “dirty” buckets). The minimum standard NGA will follow is dictated by the IC Enterprise Data Header (EDH) coupled with applicable NSG Metadata Foundation (NMF) core elements. Additional fields to handle records management are also mandated but are not nec-essarily considered for minimum access and discovery capabilities.

Trusted activities, such as those designated for the Activity Data Manager, will have direct access to the data whether it is clean or dirty. Access to “dirty” data is necessary to: 1) get the data condi-tioned, and 2) enable legacy programs to continue operating while conditioning is taking place.

Data must be conditioned before it can be discovered.

Metadata tagging, IAM, and business rules define relationships between data attributes and user credentials. Policy Enforcement Points (PEPs) negotiate access and releaseability. Both enable a strengthened data security posture.

Guidance: Mission data access shall be permitted in accordance with responsibility to provide princi-ples while enforcing all necessary access controls.

https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/esc/CSOC/Useful%20Documents/Cloud%20Security/Cloud_Computing_SRG_v1r2.pdf https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/OCIO/503%20Document%20Library/Governing%20Documents/NIST/NIST.SP.800-53r4.1.pdf https://ngaonline.titanium.rttitanium.nima.ic.gov/sites/OCIO/503%20Document%20Library/Governing%20Documents/CNSS/CNSSI%201253%20v2_15%20Mar%202012.pdf https://ngaonline.titanium.rttitanium.nima.ic.gov/org/leadership2/DO/IMI/Shared%20Documents/4.0%20IC%20ITE%20Tenant%20Services/4.12%20Data%20at%20Rest%20Encryption/fips1402.pdf https://ideonline/projects/1/ASG/Phase%201/Reference%20Information/2008-09-16%20ICD%20503.pdf

Mission data access is to be achieved to the extent possible using automated means matching at-tributes and groups detailing data restrictions and user entitlements. Specifically, entitlement at-tributes such as, clearance and special access permission group or role membership or non-membership must be explicitly asserted and tested to ensure that:

Data is tagged with metadata that have classification, dissemination controls, and access group information with associated entitlements policies.

User identities are tagged with metadata that has clearance, role, and group membership information with access and dissemination consequences.

Methods, rules and services exist to automatically test data access criteria against access attempt credentials, in accordance with policy.

Access groups and roles can be created at any broad or fine level of detail, by category, by inclu-sion, by exclusion, or by specific itemization (e.g., listing who, what, and when).

Guidance: Mission data shall be approved for migration with the CDO, DC and DS.

Before any set of mission data can be migrated to C2S, the CDO, the mission DC and DS must ap-prove (by electronic signature) the dataset’s migration. If no one claims ownership or provides stewardship of a set of mission data, that data will not be migrated. If the mission data are deemed no longer relevant or active, then the DC or DS (or CDO) may choose to not migrate the data. Then, data management would fall under NARA records management rules including potential disposal or archiving. The approval is only needed for the initial migration, not new updates or additions to the existing data set.

Guidance: Mission data will be migrated to an OpenDataStore (ODS).

C2S/ AWS shall be the default storage location of all NGA mission data. If IC partners request NGA data on the IC GovCloud for analytical purposes, a copy of the data requested can be made availa-ble at the requested location. IC GovCloud usage may be allowed for other reasons on a case-by-case basis.

Guidance: Mission data will be stored in an OpenDataStore account.

All mission data shall be stored in the ODS under the NGA mission content account in C2S, or other accounts managed by the CDO. This underscores enterprise data ownership, and helps guarantee uniform accessibility while maintaining necessary stewardship and access controls. Within the ODS, numerous different databases, archives, S3 buckets, and other data stores can be created, and managed, with the appropriate permissions and roles.

A system, program, group or individual may still have a separate C2S account with any permis-sions necessary for special purposes, as long as the mission data is in ODS and available to the community.

Guidance: Mission data will be cataloged for discovery and exposed via services.

Activity owners and designers should work to maximize discovery and the potential for reuse.

Specifically, they shall maintain and catalog metadata that describe mission data, such as the EDH and the information contents of the Trusted Data Format (TDF), including provenance and access information. Whenever practical, discovery data and content data are to be not directly accessible;

but, only through services which control access and track data use. Policy Enforcement Points

(PEP) enforce user access to all data stores via a service or view with no direct access. Application data queries will be encapsulated as services exposed to the community and published in the ser-vice registry.

Guidance: Mission data will be managed with integrated security and access control.

The cloud implements multiple layers of defense, or defense-in-depth, to protect against unau-thorized access and the introduction of vulnerabilities (e.g. malware) into the NGA environment.

The cloud infrastructure is protected by packet analysis, data loss prevention, auditing, privileged access management, and encryption for DAR and DIT. In addition, various other security services are employed to provide extra layers of protection. DSs and DCs are to leverage Public Key Infra-structure (PKI), IdAM, role-based access control (RBAC) and PEPs to maintain secure access to data and metadata.

2.3 Activity Data Guidance

Guidance: Activity owners and designers shall determine the most suitable activity data storage type and location, with the exception of logs.

Security and audit files will be accessible by qualified security administrators only. See Annex I:

Cloud Security and the Design White Paper on C2S Storage Service Options8 for additional infor-mation. Redmine and Gitlab are the approved CM/registry applications for DevOps activities.

GitLab will be used to store development code while ODS is suitable for storing NGA content, source, and mission data.

2.4 Analytical Data Guidance

Guidance: Analytical data will be stored as determined by owners of the enterprise analytical ser-vices and exploitation services.

Best practices suggest that most analytic data should be dynamically compiled in cache, if practi-cal, or in temporary data structures and automatically deleted at the conclusion of the analysis.

Certain analytic situations may require other data storage and purge procedures, at the discretion of activity owner.

2.5 Corporate Data Guidance

Guidance: Corporate data will be access-controlled.

The NGA ODS account will have protections in place to address security controls for PII, financial, Human Resource, and other sensitive non-mission data types. Security measures such as RBAC, PEPs, and business rules will be employed to control access to both discovery metadata and data content.

Guidance: Corporate data will be migrated to an ODS.

The CDO has specified that ODS C2S will be the migration destination and storage location of all NGA corporate data.

Guidance: Corporate data will be stored in the ODS.

8 C2S Storage Service Options, 2018.

https://intelshare.intelink.ic.gov/sites/nga-cloud/SiteAssets/SitePages/Guidance/NGA%20Cloud%20Guidance%20v5.2/Design%20White%20Paper%20-%20C2S%20Storage%20Service%20Options_June_2018.pdf

All corporate data will be stored under an ODS account. This underscores enterprise data owner-ship, providing uniform accessibility while maintaining necessary stewardship and access con-trols. A system, program, group or individual may still have separate accounts with any permissions necessary. Within the ODS, numerous different databases, archives, and other data stores can be set up, and managed, with the appropriate permissions and roles.

Guidance: Corporate data will be cataloged for discovery and exposed via services.

Corporate DCs shall publish discovery metadata in a corporate metadata repository distinct from the mission metadata repository. Whenever practical, application data queries will be encapsu-lated as services exposed to the community and published in the service registry

Guidance: Corporate data will be managed with integrated security and access controls.

The cloud implements multiple layers of defense, or defense-in-depth, to protect against unau-thorized access and introduction of vulnerabilities (e.g. malware) into the NGA environment. The cloud infrastructure is protected by packet analysis, data loss prevention, auditing, privileged ac-cess management, and encryption for DAR and DIT. In addition, various other security services are employed to provide extra layers of protection. DSs and DCs will leverage PKI, IdAM, RBAC and PEPs to maintain access to data and metadata.

2.6 Organizational Data Guidance

Guidance: Organizational data currently on a shared drive and/or SharePoint will be migrated to the equivalent NGA enterprise location in the cloud.

No specific, individual organizational actions are needed at this time; DSs may relocate this data at their discretion.

3.0 DATA MIGRATION

3.1 Background

The NGA mandate9 is to migrate all content and activities to the cloud. The key migration objec-tives are:

Do not break mission capabilities, Get off the current infrastructure, Obtain secure, efficient, intelligence integration.

As shown in Figure 4, migration is dependent on both the data and activity migration. Mission mi-gration is not complete until the applications, services, and data are usable together.

9 Memo: NGA Cloud Guidance, U-2016-00069, 10 February 2016 https://ngaonline/sites/CIO-T/InformationCenter/Documents/Publications/U-2016-00069-ACTION-CLOSED.pdf

Figure 4: Cloud Migration Dependences

Mission data migration ahead of the activities helps to accelerate customer consumption, activity test readiness and interim operations.

3.2 Data Migration Guidance

Guidance: Program Managers and developers shall support the migration effort by implementing the Migration Staging Strategy.

The Migration Staging Strategy follows:

Provision staging S3 buckets by source, data type and classification.

Provision additional buckets as needed (e.g., Special Access Program (SAP) data).

Provision a “Manual Review” dirty bucket per migrating system for all source data without appropriately tagged metadata. Properly tagged legacy data may skip the staging dirty bucket.

No end user access to the “Manual Review” dirty bucket. DSs must have access to remediate the content and workload managers must have oversight permissions.

Do not mix data in the staging architecture that is not mixed today.

Migrate non-file-based data by following database criteria guidance.

Guidance: ‘Interim’ operational access will be granted if content is not in desired organizational structure.

Guidance: Program Managers and developers shall incorporate resiliency requirements for produc-tion storage in the design (e.g., multiple availability zones) for operational data.

Guidance: Program Managers and developers shall implement persistent data storage (e.g., S3 Bucket Strategy).

The S3 Bucket Strategy follows:

All persistent storage for all corporate or mission file-based data will be in S3, with ar-chives using Glacier capabilities.

Use standard enterprise S3 buckets, as predefined.

Content-IDs are assigned to data at the logical dataset level.

Virtual folders are set up for each ContentID within an S3 bucket, managed by activity’s data manager.

Lifecycle/data retention rules are set at the ContentID level.

Permission for direct S3 bucket access by trusted activities are set at the ContentID level with:

o Read only access by default.

o Write/delete access (activity data manager) requiring ContentID approval (i.e., via

DC).

o Trusted activities, if not using the enterprise PEP, responsible for ensuring appro-priate access by users.

o Content following S3 bucket naming standards, presented in Appendix H.

S3 buckets used will be the standardized S3 buckets set up under the ODS.

S3 bucket naming convention will be:

o \\<S3 Bucket Name>\<Content-ID>\<Managing Activity>

Special access permissions can be granted at any level of a virtual folder structure.

o AWS enables wildcards within the string for access policies.

Lifecycle policies will normally be set at the Content-ID level per DS decisions.

o AWS only enable wildcards at the end of the string for lifecycle policies.

ATO of the ODS services, including the Metadata Catalog, are a DC responsibility.

3.2.1 Data to the Cloud Process

The process of getting data to the cloud is outlined by the steps to: identify content, approve re-quests and create/implement ODS change requests that will apply to most datasets. This process applies to moving data to the ODS, establishing storage in the ODS with the appropriate accesses, and creating data feeds. Each data set may have unique issues and considerations. ODS cloud mi-gration preconditions are:

1. A Data Custodian must be identified,

2. Data must be registered in the data inventory such that a ContentID is associated with the data, and

3. Essential metadata (a minimum set) must be populated for the particular data set.

This process is the same for both file/object based data and database data with a few exceptions:

1) database data will not be posted to a dirty bucket, but instead to its designated database struc-ture, and 2) while the ISP and DSP will move database data, as they move file/object data; it will be up to the ASP to ETL data into the cloud’s database.

3.2.1.1 Phase 1: Identify Content

The identity phase begins when a party (CDO, DC, DS, or PM) identifies data to be migrated to or established in the ODS.

Identify Content C

D O

/D C

/D S/

P M

D M

W G

Check Architecture

Repository (AR) for correlating

Content ID

Content ID exist?

Data Team update AR and receive

Content ID

AR

NO

YES

Requester Provides Content To Data

Team

Content ID need

Updating?

NO

YES

Requester Submits Changes to Data

Team

New Data Identified

Content Identified – Go to Approve Request Phase

Figure 5: Phase 1 - Identify Content (assign Content-ID)

As shown in Figure 5, all parties must check the ATLAS’s AR (i.e., Data Inventory) to see if their content aligns with any…