SOW-Appendix A-SCDS - Clean_DRAFT.docx

UNCLASSIFIED

UNCLASSIFIED

APPENDIX A

Source Content Dissemination Services System

November 1, 2022

Source Content Dissemination Services System (SCDS) ST developed SCDS to create an easy to use, unified point of access for NGA customers (domestic and foreign) to discover and retrieve Source Foundation products. Unified access to Source Foundation products through SCDS will replace numerous customer specific access points that created situations in which some users knew where to find some products, while others did not. It will also result in simplification of dissemination processes for product development communities.

SCDS does not impact how producers (USG or Contractor) will produce their products. A product producer (USG or contractor) will simply load a product into the SCDS ingest S3 buckets. From there the product will be scanned for viruses, and if cleared will move on to conditioning, where the artifact will be categorized, labeled and tagged. Tagging will include data points to ensure that a product is labeled with appropriate classification designations, and earmarked for appropriate dissemination for international partners IAW governing agreements and protocols. Those identifying tags will be logged into the Source Foundation Catalog (SFC) for ease of discovery. Concurrently, the product will go into long term cloud storage (NIPR, SIPR or COE/JWICS, as appropriate).

From the Users perspective, a user will access the Saber storefront as a unified Point of Entry to access Source Foundation products, as opposed to the previous system, in which a user had to know which website to go to, to locate which documents. The users’ request will be streamed through the SFC to provide a list of products that answers the users’ query, and aligns with their given access. Selected artifacts are then issued to the user electronically, reducing the discovery and delivery process from (in some cases) weeks to minutes.

Further, users will be able to request scheduled updates to requested artifacts through the Enterprise File Delivery System (EFD), and international users will have their requests processed through the Foundation GEOINT Discovery & Release Tool (FGDRT) to ensure that provided products align with governing agreements and protocols.

SCDS does not replace Print on Demand, or hard copy requests, which are staffed via the Defense logistics Agency, though the SCDS User Interface will facilitate those requests.

SCDS FUNCTIONALITY

Product Content (Introduction of Foundation GEOINT Products into S3 Buckets for Conditioning, Tagging and Storage) Optimized systemization for data ingest, validation, and viral scanning and quarantining of infected files. To be maintained throughout the lifecycle, and performer adaptive maintenance, as needed.

· Data Uploader User interface (UI)

· Digital Delivery Upload

· The Digital Delivery Uploader is a script that automates the process of creating a manifest of files in a local directory and then completes a sync of the local directory with an optional specified s3 bucket. Based on the required standardized directory structure expectations, the uploader dynamically creates proper S3 objects for ContentID and Managing Authority. This script tool provides a default UI for enhanced interaction or can be run via command line. Virus scanning/quarantining of infected files.

· Document Upload

· A document upload microservice that uploads either one or more files into a pre-configured S3 location and requires GEOAxIS for authentication in order to upload documents.

· Dirty Bucket Ingest Listener (EC2 and Lambda)

· Lambda and EC2 application to listen for S3 events (via SNS and SQS) and first determine the file size of newly created objects. The lambda will also scan incoming files if the size is small enough to be handled by the lambda. Files with sizes above the lambda threshold will be sent to a queue to be handled by the EC2 scanner.

· Digital Delivery Manifest

· Digital Delivery Checksum (Lambda)

· Calculates checksums of incoming files for digital delivery and routes manifest to the validation queue.

· Digital Delivery Validator (Lambda)

· Validates manifests when they arrive in queue. Missing checksums cause processing to stop until Visibility/Timeout expires

· Lot Delivery

· Delivery Order Listener (Lambda)

· Lot Delivery Inspector (Lambda)

· Product Inventory Listener (Lambda)

Data Maintenance: Metadata Extraction, Product Conditioning, Data Conditioning

Automated metadata extraction, product conditioning, classification and tagging of ingested data. To be maintained throughout the lifecycle, and performer adaptive maintenance, as needed.

· Extract Router

· Metadata Extractors:

· Archive (Lambda)

· GDAL (Lambda)

· Document (Lambda)

· Picture (Lambda)

· GeoPDF (Lambda)

· Mr. SID (Lambda)

· NITF (Lambda)

· Shapefile (Lambda)

· VPF (Lambda)

· Contributed (Lambda)

· No Longer Produced (Lambda)

· MAG Loader (Lambda)

· POD (Lambda)

· FBG (Lambda)

· Datum Conditioner (Lambda)

· Lambda application to listen for SQS events and update geospatial info in Source Foundation Catalog (SFC).

· Datum conditioner connects to the extractor result queue to find messages to process.

· Geospatial Country Conditioner (Lambda)

· Mapper Conditioner (Lambda)

· The configuration management location for the mappings (instruction set(s)) that are used as part of the data conditioning pipeline, specifically in the Mapper Conditioner Lambda application. These instruction sets are used in mapper to map extracted metadata stored in SFC (extractInfo) to the normalized structures

· Mappings are performed by using the content identifier, file type and mime type of an incoming file to retrieve pre-loaded mappings from the SFC database and applying those mappings to the metadata that has been previously extracted from the incoming file by one of the extractors. The layout of this project is critical to the operations of the Mapper Conditioner.

· IC Information Security Marking Validator Labeling Assurance Handler (ILAH) (Lambda)

· Lambda application to listen for SQS events and populate magic records with either default information related to the content Identifier or information from a policy record. The incoming message must have a content Identifier in order to create a magic record.

· ILAH conditioner connects to the mapper-conditioner result queue to find messages to process.

· Metadata Attribute Gathering Information Collector (MAGIC)

· Policy Updater

· NESTJS Docker API application to handle policy updates

· Foundation GEOINT Discovery & Release Tool (FGDRT) Handler (Lambda)

· Product Conditioner (Lambda)

· ATOC

· CIB

· DTED

· No Manifest

· XML

· Gateway Loader (Lambda)

· EFD Handler

· Country Code API (Lambda)

· API for querying Country Code data

· Translates country codes between standards or name to standards (fips, iso2, iso3, genc2, and genc3)

Foundation GEOINT Discovery and Release Tool – FGDRT

Foundation GEOINT Discovery and Release Tool (FGDRT) is a decision management application that provides the business workflow for adjudication of NGA GEOINT sharing requirements with foreign partners. FGDRT enables the creation of a sharing requirement in the form of an International Agreement, Foreign Military Sales or Exception to Policy “Case” that results in the creation of an “obligation” (rule) after undergoing an administrative review process. Each of these case types provide a common wizard interface for easy and consistent generation of foreign sharing requirements. Features unique to FGDRT include an “online-store-like” user interface (UI) providing the ability to search/select Customers in the form of Department of Defense Activity Address Codes (DODAACs), Coalitions, Multinational Organizations and Country Profiles. FGDRT also provides for quick and simple discovery of GEOINT data/products through a set of pre-determined search parameters (Simple and Advanced). Users submit cases for approval by the users through the application that uses a unique interface that provides the case information (customers, products, administrative data) in a format that best enables the foreign disclosure process. FGDRT also allows users to search for previously generated obligations through a stand-alone interface.

FGDRT enables the storage of documents/files with each case as part of the International Affairs Records Management Office File Plan and represents the “Knowledge Management Center for Foundation GEOINT International Sharing.”

FGDRT Integration with SCDS:

FGDRT has implemented an architecture within the Cloud that automates and synchronizes the Foundation GEOINT D&R process through a robust SCDS interface environment, allowing users to research, access, identify, view and request adjudication of NGA FG material in support of mission requirements. Key impacts include:

· D&R decisions turnaround time reduced by ~70%

· Near real time brokering of digital (softcopy) product requirements

· Reduction of printing and delivery costs

· Requirement ingest workload reduced by 30-40%

· Provides the entire USG access to the largely automated process

· Digitally connects FGDRT decisions with dissemination systems

· Direct foreign partner access for requirements entry

· FG D&R fully integrated with SF/ST modernized Gateway West (SCDS) operations

FGDRT Process:

FGDRT Functional Benefits:

· Capable of supporting the digital distribution environment and improved automation with Defense Logistics Agency (DLA) for accelerated traditional paper/disk distribution.

· Fully integrated adjudication processes within the cloud space enables semi-automated distribution to GEOINT partners.

· FGDRT enables data access, and by design data integrity – inherently enabling assured positioning, navigation, timing and targeting.

· FGDRT is non-proprietary software that is future flexible and integrated into Enterprise data operations.

· FGDRT will be the NGA records repository for all partner foundation agreements regardless of NGA office that is the signatory of the agreement.

FGDRT Technical Construct:

FGDRT API

· Typescript programming language

· NestJS API framework

· Node.js hosting runtime

· Server-side: API is hosted via Docker containers capable of running on Kubernetes, or other runtimes including AWS Lambda

FGDRT UI

· Javascript programming language

· React front-end development framework (DevCorps react-template)

· Node.js hosting runtime

· Front-end: can be hosted via Docker containers on Kubernetes, or other runtimes including AWS Lambda

· Matomo: Web analytics

GPM

· Typescript programming language with limited Python as needed

· NestJS API framework

· Lambda running Node.js hosting runtime

· API and UI: APIs and the conditioning pipelines are run as AWS Lambda Functions, and when cost savings can be realized on EC2, the capability is containerized in docker and is capable of running on Kubernetes. All capabilities are capable of running in the data center as needed

Source Foundation Catalog (SFC)

Catalog of all ingested products, allowing external programs (ie GSR, The Globe, etc) to integrate foundational metadata to facilitate automated flows and integration with NGA. In coordination with FGDRT, SFC will create an automated database for sharing content with allied partners. To be maintained throughout the lifecycle, and performer adaptive maintenance, as needed.

· Listener

· Database

· Loader

Dissemination Services:

Allows for retrieval of ingested materials through organized search parameters established in the SFC. Allows external programs (Map of the World [MoW], Global Search and Retrieval [GSR], The Globe, etc) to use automated dissemination. To be maintained throughout the lifecycle, and performer adaptive maintenance, as needed.

· Dissemination Proxy

· Product Manager

· Object Streamer

· Thawing Service

· Order Manager

· Product Updater

· File Converter

· Bundler

· Subscription Service

· GeoServer Cluster

· Tile Servers

Enterprise File Delivery

Within the Source Content and Delivery Services (SCDS), the Enterprise File Delivery (EFD) element is constructed to support users’ requests for Source Foundation products. Once requested and discovered, EFD provides for electronic delivery of those products. Additionally, EFD provides for regular and recurring updates of products, IAW customer requests through its subscription service function. Delivery of larger files are accomplished through usage of Bundlers.

Highlights of EFD include:

· Java based application, working as a server/client relation.

· Customer/client can request data from NGA and synchronize requested data to match GEOINT holdings.

· Customers can customize data types, scheduling, data location for download as well as other options within the EFD application.

Available on NIPRNet, SIPRNet and JWICS.

EFD Functionality:

· Uses TLS encryption standards for secure communication between source and client sites - HTTPS protocol/FIPS 140-2 compliant encryption.

· Systematically adds/deletes data during synchronization tasks.

· Ability to resume dropped connections.

· High volume data transfer, can set multiple feeds per connection.

· Able to set 2 concurrent transfer per each consumer data set.

· Transfer files of 250 GB and greater in size.

· Software has the ability to both consume and supply data.

· Some customers use it to supply their subordinate organizations with the products received from NGA.

· Logging - The system allows for local EFD Admins to examine detailed logs to extract metrics for data supplied as well as system troubleshooting.

· The system is ‘dashboard’ enabled allowing for real time monitoring of current connections/data feeds.

· Start/stop and edit data feeds from the dashboard if needed.

EFD Requirements:

· Windows 2012/16/19 or Linux OS 6/7.x, RHEL 8 will be tested 3/2022

· Minimum 2 CPU’s Cores, 16 GB Memory

· 4 CPU’s Core and or Virtual, 24 GB Memory if expected heavy data sync’s

· Non-Person Entity (NPE) PKI Certificate

· Java 8 Update 311 or greater

· Storage for data

· Ports Used:

· 443 for delivery/transmit

· 8443 local EFD GUI Admin/User access

· 3306 local Derby Database

GEOINT Delivery Platform (GDP)

GEOINT Delivery Platform (GDP) is a web based application that provides a catalog of NGA products and allows a user to find, view and download NGA's digital holdings. To be maintained throughout the lifecycle, and perform adaptive maintenance, as needed.

· GDP API

· SCDS is planning on transitioning the GDP function to the SFC module

Mission Planning System Updater (MPSU):

Mission Planning System Update utility allows users that subscribe to specific products to inventory differences between subscription holdings on the gateway vs. the customer’s holdings to update their local system. To be maintained throughout the lifecycle, and perform adaptive maintenance, as needed.

· MPSU REST Service

· MPSU Manifest Updater

Trans Shipper:

· Used to send data from NGA to DLA to maintain up to date holdings to be maintained throughout the lifecycle, and perform adaptive maintenance, as needed.

· Trans Shipper API

Saber ST created Saber to update the NGA Print on Demand, Replication on Demand, DLA Map Catalog (PoD / RoD / DMC) Portals with a Slippy map and additional download formats, establish a mechanism to place DLA map / chart hardcopy orders on from the NIPR NGA PoD / RoD / DMC Portal, and deliver the updated PoD / RoD / DMC Portal to NGA for use on NIPR, SIPR, and JWICS. Note: NGA must manually push SIPR / JWICS orders down to NIPR to submit to DLA, but NRL will provide a format acceptable to NGA’s cross-domain solution.

Saber provides a unified User Interface for all NGA customers to access Source Foundation materials, and secure their delivery in minutes (on average) compared to the previous architecture in which different products had different access portals, and securing delivery could take up to weeks.

Saber Simplified Architecture:

Saber Page User

ODS

API

SCDS

API

· Captured user data and session activity log will be stored.

· Data will be accessible for analysis.

Saber Functionality:

· Establish User Profiles and track sessions and order History.

· Go directly to individual Subordinate Pages if desired.

· Submit search queries via a Map interface.

· Queries will be run against multiple connected APIs.

· Results will be consolidated and presented on the Storefront, if applicable.

· Display either in a List or Map interface.

· Recommended and related products will also be included in the search results.

· Order Hard copy Maps.

· Fill out the included Order form and submit for DLA to process (Generate MILSTRIP)

· Order Products and Data through a variety of media options.

· Digital Download

· Download Mission Specific Bundles (EVC, NEO, Flight Bags, etc.)

· Filter Results to fit desired categories (Topographic, Elevation, Vector, etc.)

Saber Wiring Diagram: PKI/CAC Holder:

User Saber GeoAxis 404 Error SABER Landing Page Register/ Create Account

NO

YES

https vs www Authenticate Do you have an account?

YES

NO

404 Error Decline

YES

Data Comes In

CD, DVD or Digital Delivery, product files will be routed through a dirty to clean bucket process that will virus scan, verify and persist content.

At risk files are returned to origin for resolution of issues.

Source Foundation Catalog

Accounts for content location in the cloud, and in datacenters, providing synchronization and metadata access.

Obligations/Rules

FGDRT gives adjudicators the ability to create/apply rules and obligations to images/information as they are inserted into the GEOINT database.

Metadata conditioning accounts for country codes, agreements, security rules, releasbility, etc

FGDRT labels data up front which, can reduce dissemination time from hours and/or days to minutes or seconds.

Data Dissemination

EFD, GDP, SCDS provide for rapid dissemination of authorized products to santioned requestors

Processing and delivery time reduced from days to minutes.

UNCLASSIFIED

image2.png image3.png image4.emf image5.png image6.png image7.jpg image8.png image9.emf image10.png image11.png image12.jpeg image1.png