Task_2_-_Reference_Document__EOS_Architecture.pdf

Version 2.5 iii

Document History

Document Version

Publication Date

Changes

1 3/20/2008 Initial Version

2 3/2010 Full Update of all content

2.1 11/2010 Addition of DEM Explorer

2.2 2/2011 Removal of requirements, updated WBS

2.3 3/2011 Components update, new components diagram, new components descriptions

2.4 4/2012 Full review and update.

Removal of retired components.

2.5 4/2013 Full review and update.

iv

Contents

Document History ........................................................................................................ iii Contents ........................................................................................................................ iv List of Figures .............................................................................................................. vi List of Tables ............................................................................................................... vii Section 1 USGS EOS

1.1 Core USGS Functions

1.1.1 Ingest

1.1.2 Archive

1.1.3 Production

1.1.4 Distribution

1.2 EOS Management Plans

Section 2 LP DAAC Architecture

2.1 LP DAAC High Level Architecture

2.2 LP DAAC Operations Concept

2.2.1 LP DAAC Storage

2.2.2 Network

2.2.3 Security

2.2.4 Database

2.2.5 Virtualization

2.2.6 Backup / Recovery

2.2.7 Browse

2.2.8 Production and Distribution

2.2.9 Data Discovery and Access

2.2.10 Metadata Architecture

2.2.11 Web Services

2.2.12 User Registration

2.3 Components of the LP DAAC

2.4 LP DAAC Use Case Examples

2.4.1 Ingest and Archive

2.4.2 Production, Ordering, and Distribution

2.5 LP DAAC Physical Architecture

2.5.1 Network Architecture

2.5.2 Systems Architecture

2.6 LP DAAC Component Detail

2.6.1 AESICS

2.6.2 ASTER DAR Tool

2.6.3 Cloakware

2.6.4 ASTER Ingest

2.6.5 ASTER Expedited Interface to GDS

2.6.6 ANSUR / ASTER Free List

2.6.7 ASTERVAL / EOSVAL

2.6.8 Catalog Reconciliation Tool (CRT)

2.6.9 COTS Tools

v

2.6.10 Data Pool Search Client

2.6.11 Data Management Tool (DMT)

2.6.12 Data Distribution System (DBDirect)

2.6.13 Drupal

2.6.14 DUEs - Startup / Shutdown and Other Scripts

2.6.15 Earth Explorer (EE)

2.6.16 EOS Clearing House (ECHO)

2.6.17 EOSDIS Core System

2.6.18 ESDIS Metrics System (EMS)

2.6.19 External Web Site / Web Tools

2.6.20 Global Change Master Directory

2.6.21 Global Data Explorer

2.6.22 GloVIS

2.6.23 Landsat Metadata in ECHO

2.6.24 LP DAAC Internal tools

2.6.25 LP DAAC Order Flow Components

2.6.26 MRT

2.6.27 MRTWeb

2.6.28 Reverb

2.6.29 S4PM

2.6.30 SA / DBA Tools

2.6.31 Support Center Plus

2.6.32 Total Cloud Cover (TCC)

2.6.33 User Registration System

2.6.34 Web Services Infrastructure

2.6.35 WELD Distribution

2.7 USGS EOS ICDs and Agreements

Appendix A Glossary and Acronyms Appendix B NASA EOSDIS 2015 Goals Appendix C Common LP DAAC PC Software vi

List of Figures

Figure 2-1 LP DAAC Top Level Architecture Figure 2-2 LP DAAC Operational Concept Figure 2-3 EOS SANs Figure 2-4 EROS Network Diagram Figure 2-5 LP DAAC Production and Distribution Figure 2-6 Data Discovery and Access Figure 2-7 LP DAAC Components Figure 2-8 LP DAAC ASTER Ingest Figure 2-9 LP DAAC MODIS Ingest Figure 2-10 LP DAAC ASTER Expedited Ingest Figure 2-11 Reverb Ordering Figure 2-12 LP DAAC Data Pool Order Figure 2-13 LP DAAC GloVIS Order Figure 2-14 LP DAAC MRTWeb Figure 2-15 LP DAAC Systems Figure 2-16 AESICS System Figure 2-17 ASTER DAR Tool Figure 2-18 ASTER Ingest DUE Figure 2-19 ASTER Expedited Figure 2-20 ASTER Expedited Data Set Archive Figure 2-21 ASTER Free List Figure 2-22 ASTERVAL Use Case Figure 2-23 EOSVAL Use Case Figure 2-24 CRT Basic Operational Concept Figure 2-25 DBDirect Process Flow Figure 2-26 LP DAAC Drupal Environment Figure 2-27 Earth Explorer Process Flow Figure 2-28 ECS Modules Figure 2-29 Data Pool, Online Archive, Tape Archive, AIM Figure 2-30 EMS Metrics Example Figure 2-31 LP DAAC Website Figure 2-32 GCMD Home Page Figure 2-33 GDEx Figure 2-34 LMdiE Process Flow Figure 2-35 LP DAAC Order Flow Figure 2-36 MRT Processing Figure 2-37 MRTWeb Processing Figure 2-38 MRTWeb Architecture Figure 2-39 Reverb Interface Figure 2-40 S4PM Processing Figure 2-41 Detailed S4PM Ingest Figure 2-42 TCC Process vii

Figure 2-43 USGS EOS Project Web Services Infrastructure Figure 2-44 WELD User Interface Figure 2-45 WELD Interface Configuration

List of Tables

Table 2-1 LP DAAC Components Table 2-2 LP DAAC ASTER Ingest Table 2-3 LP DAAC MODIS Ingest Table 2-4 LP DAAC ASTER Expedited Ingest Table 2-5 Reverb Ordering Table 2-6 LP DAAC Data Pool Order Table 2-7 LP DAAC GloVIS Order Table 2-8 LP DAAC MRTWeb Table 2-9 AESICS System Table 2-10 ASTER DAR Tool Table 2-11 ASTER Ingest DUE Table 2-12 ASTER Expedited Table 2-13 ASTER Free List Table 2-14 DBDirect Process Flow Table 2-15 Earth Explorer Process Flow Table 2-16 GDEx Table 2-17 LMdiE Process Flow Table 2-18 LP DAAC Order Flow Table 2-19 MRT Processing Table 2-20 MRTWeb Processing Table 2-21 S4PM Processing Table 2-22 TCC Process Table B-1 NASA EOSDIS 2015 Goals Table C-1 LP DAAC PC Software

Section 1 USGS EOS

The Earth Observing System (EOS) project at the USGS Earth Resources Observation and Science (EROS) data center is also commonly referred to as the Land Processes Distributed Active Archive Center (LP DAAC). The LP DAAC is funded by the NASA Earth Science Data and Information System (ESDIS) project, which manages Science Operations for the larger Earth Observing System Data and Information System (EOSDIS) project.

Activities that take place at the LP DAAC include Project Management, Science Integration, System Development, System Maintenance, and Science Operations.

Further analyzing the structure of the LP DAAC, there are many core supporting functions than enable these top-level activities. These functions include ingest, archive, Production, distribution, management, infrastructure planning / system engineering, and mission science support which also includes general customer support. The EOS architecture has been created to support all of these functions.

1.1 Core USGS Functions

There are four primary Core Functions supported by the USGS EOS LP DAAC. They are outlined below.

1.1.1 Ingest

The ingest activities involve the functions necessary to accomplish the ingest of EOS land products including Moderate Resolution Imaging Spectroradiometer (MODIS) products from the AQUA satellite and MODIS and Advanced Space borne Thermal Emission and Reflection Radiometer (ASTER) products from the TERRA satellite.

1.1.2 Archive

Archive activities involve all functions necessary to archive EOS land products including MODIS products from the AQUA satellite and MODIS and ASTER products from the TERRA satellite. These functions include (but are not limited to) data stewardship, inventory management, and archive data validation (full archive check). While core system development efforts for archive are generally outside the scope of the LP DAAC project (supplied by the ECS), the development of DAAC Unique Extensions (DUEs) and software packages to augment the operations of the LP DAAC archive functions are necessary.

1.1.3 Production

Production activities involve the functions necessary to operationally produce higher-level products and/or custom formats for the products archived within the LP DAAC.

The Simple, Scalable, Script-based, Science Processor for Measurements (S4PM) is utilized to produce higher-level ASTER products for users. Product Generation Executables (PGE)s utilized by S4PM are created by GDS and JPL.

1.1.4 Distribution

Distribution activities involve all functions necessary to distribute EOS land products, including MODIS products from the AQUA satellite and MODIS and ASTER products from the TERRA satellite, to science users and to the general public. Examples of these functions include publishing the catalog to external inventories (i.e. ECS Clearing House (ECHO)), reconciliation of external catalogs, support for users utilizing NASA Reverb to order data, support for development and operations of the USGS Global Visualization system (GloVIS) and Earth Explorer for EOS data, development and support of LP DAAC DUEs (i.e. Global Data Explorer (GDEx), MRTWeb). The systems utilized for these activities are provided by the core system, through DAAC Unique Extensions (DUEs), or by external USGS or NASA projects. While the LP DAAC locally develops most of the software for this activity, the Hierarchical Data Format (HDF) to GeoTIFF Conversion Tool (HEG) that performs some data reformatting is delivered as part of the externally developed Earth Observing System Data and Information System (EOSDIS) Core System (ECS).

1.2 EOS Management Plans

The EOS project has a variety of plans for governing and describing LP DAAC processes and procedures. The LP DAAC Management Plans area is located in DocuShare at USGS Documentation >> Project Management >> 1.1 Project Management >> 1.1 Management Plans

• EOS Portfolio Management Plan – The plan for managing the work and tasks of the LP DAAC.

• EOS Risk Management Plan – The plan for managing risks from Work Packages and Tasks of the LP DAAC.

• EOS Schedule Management Plan – The plan for managing the LP DAAC schedule.

• Project & Process Management Plan – General high-level plan for project and process management.

• System Engineering Management Plan - The plan for managing the System Engineering activities and procedures of the LP DAAC.

• Systems Operations Incident Management Plan – Plan for handling and addressing operations incidents and problems.

• Document Management Plan – The plan for managing LP DAAC documents and the artifacts from Work Packages or other DAAC activities.

• Configuration Management Plan – Plan and procedures for managing configuration changes, updates, and fixes for the LP DAAC.

Section 2 LP DAAC Architecture

2.1 LP DAAC High Level Architecture

At the highest level the LP DAAC architecture supports ingest and archive of MODIS and ASTER data, optional Production, and distribution of that data to users. ASTER and MODIS data is collected by the TERRA and AQUA satellites and downlinked to science processing facilities through the Tracking and Data Relay Satellite System (TDRSS) network.

Figure 2-1 LP DAAC Top Level Architecture

The LP DAAC receives ASTER data from Ground Data System (GDS) and MODIS data from the MODIS Adaptive Processing System (MODAPS). The data is ingested and archived in the LP DAAC Archive. After ASTER and MODIS data is archived, it can be retrieved from the LP DAAC through various distribution interfaces. ASTER data archived on the DAAC can also be processed to higher-level products before delivery to users. All the LP DAAC interfaces are available to users over the Internet. Users are categorized into general users and science users. Science users can request extra access to ASTER data from the LP DAAC to acquire products that are not available to general users. In addition to MODIS and ASTER products, the LP DAAC can support other products as well. These products include data sets like the ASTER Global Digital Elevation Model (GDEM) and the MODIS and Shuttle Radio Topography Mission (SRTM) derived Land Water Mask.

LP DAAC data is made available through multiple clients including: Reverb, EarthExplorer, GloVIS, GDEx, and MRTWeb.

2.2 LP DAAC Operations Concept

Building on the Core USGS EOS functions, an Operational Concept for the LP DAAC is shown in Figure 2-2. The elements of this diagram matrix back to the overall core activities of the LP DAAC. In general, the LP DAAC is providing services to EOS Data Users.

Figure 2-2 LP DAAC Operational Concept

Core functions of the LP DAAC include Ingest, Archive, Production and Distribution.

These elements are responsible for all the LP DAAC EOS data activities. There are a lot of smaller actions that occur within these elements.

Various groups support the operation of the LP DAAC. Systems and Software Engineering support the development, integration, installation, and support for the systems of the DAAC. Systems and Database Administration handle the building, configuration, maintenance, implementation, and integration of the LP DAAC hardware and software components. User Services supports the LP DAAC end users and their interaction with the LP DAAC. The Operations group oversees the day-to-day running of LP DAAC systems to meet the users’ needs. Science and Outreach interact with the science communities that are interested in LP DAAC products and provide information about the science uses and applicability of LP DAAC data.

2.2.1 LP DAAC Storage

LP DAAC storage for the data that is ingested, archived and distributed by the LP DAAC is largely comprised of Storage Area Networks (SANs). The LP DAAC has three distinct SANs that it utilizes.

• ECS SAN – Storage for LP DAAC data and products. Often times referred to directly as ‘The Archive’.

• DAAC SAN – Storage to support LP DAAC virtualization, servers, systems, and databases.

• Science Data SAN – A storage area for provisional science data and a distribution cache for some LP DAAC systems.

Figure 2-3 shows the LP DAAC SANs and their usable size.

Figure 2-3 EOS SANs

2.2.2 Network

Figure 2-4 is an illustration of the EROS network. There are two key LP DAAC networks in place. The DAAC network that hosts LP DAAC equipment is placed behind the USGS firewall. DAAC systems that provide data and interfaces externally to users are routed through the USGS Demilitarized Zone (DMZ).

Figure 2-4 EROS Network Diagram

2.2.3 Security

The LP DAAC complies with the rules, policies, and implementations set forth by the Department of the Interior. The LP DAAC is part of the DOI’s Science and Support Systems Enclave. In addition to following the requirements of that security group the LP DAAC also follows applicable NASA security standards. Differences between the two security standards are negotiated between both groups and USGS EOS. The document EOS Project Acceptance of sub A&A covers the configurations, parameters, Standard Operation Procedures, and waivers involved in the LP DAAC’s Security.

2.2.4 Database

The LP DAAC utilizes multiple databases to run and maintain its systems. The ECS system utilizes a Sybase database system. Other LP DAAC systems utilize an Oracle database for storing data and information necessary to their function. The content for the LP DAAC website’s Content Management System (CMS) is stored in a MySQL database.

2.2.5 Virtualization

The LP DAAC uses virtualization to achieve optimal utilization of hardware resources.

In many cases DAAC systems do not require fully isolated servers to run. Using virtualization allows for sharing and distribution of DAAC hardware to power many different DAAC systems. The LP DAAC utilizes VMWare to provide virtualization services. The DAAC has over 60 virtual servers running in its development, test, and operational modes.

2.2.6 Backup / Recovery

A legato tape backup system is used by the LP DAAC to back up LP DAAC systems.

These backups are primarily for system recovery purposes.

LP DAAC archival data backups are managed through ECS. Some LP DAAC data is archived through an agreement with the USGS EROS LTA project.

2.2.7 Browse

Browse information about LP DAAC data is essential for tools that provide direct visualization to LP DAAC users. The LP DAAC delivers browse data through a handful of interfaces for use with clients that are interfaced by the users. The DBDirect system provides and maintains Browse downloading interfaces and information to the Glovis and EE systems. DBDirect is described further in section

2.6.12. Other LP DAAC systems, like GDEx, utilize OGC web services to provide browse visualization of LP DAAC datasets for users.

2.2.8 Production and Distribution

Many LP DAAC systems support processing of LP DAAC data into higher-level products. The inputs and outputs of these systems are shown in Figure 2-5.

Production on the LP DAAC consists of creating higher-level science products using a variety of lower level inputs and ancillary data. Distribution can consist of delivering just the granule requested to the user but may also consist of mosaicking, subsetting, reprojecting, and reformatting data before its delivery.

Figure 2-5 LP DAAC Production and Distribution

2.2.9 Data Discovery and Access

There are multiple methods for users of the LP DAAC to access LP DAAC data.

Figure 2-6 shows an overview of data discovery and access methods for the LP

DAAC.

Figure 2-6 Data Discovery and Access

2.2.10 Metadata Architecture

Metadata for data distributed and archived by the LP DAAC is stored alongside the data in the archive and is also stored in the EOS Clearing House (ECHO). ECHO provides a repository for EOS metadata to be utilized by the EOS DAACs and by EOS clients for searching. The LP DAAC even publishes metadata from other USGS projects to ECHO as is the case with the Landsat Metadata in ECHO system (LMDiE). Metadata about LP DAAC data is also sent to the USGS Earth Explorer system to enable searching through the EE catalog.

Many of the LP DAAC datasets follow Federal Geographic Data Committee (FGDC) guildelines for Metadata formatting.

2.2.11 Web Services

Certain LP DAAC functions are addressable directly over http protocol and can be utilized by scripts and other systems to execute tasks and deliver LP DAAC products to users and systems. Web services offered by the LP DAAC include:

• MRTWeb

• EOSDIS Service Interface (ESI)

• Data Processing Options Service (DPOS)

More web accessible services will be added and adapted in the future.

2.2.12 User Registration

The LP DAAC utilizes user accounts for access to many of its systems. In order to make accessing DAAC systems simpler, the DAAC utilizes User Registration and authentication services (URS) that are available from ESDIS. This enables users to use the same username and password to access LP DAAC systems and ESDIS systems (for example Reverb).

2.3 Components of the LP DAAC

Due to the size of the LP DAAC and the number of features provided by it, the LP DAAC is broken into separate components to improve manageability. Figure 2-7 shows the components that make up the LP DAAC.

Figure 2-7 LP DAAC Components

The components that make up the LP DAAC come from various different projects and entities. The core of the LP DAAC functionality is provided by the ECS. ECS is created and maintained by the EOSDIS Evolution and Development (EED) contract. Modules that are also necessary for ECS operation include NASA’s ECHO and Reverb components. In addition to the ECS archive functionality, DAAC Unique Extensions (DUEs) provide expanded capabilities developed locally at the LP DAAC. A few DUE modules have been developed that have taken on functions previously provided directly by ECS. These modules are shown as DAAC Developed components of ECS. In addition to DUE, ECS, and NASA components, certain EROS Enterprise components developed by EROS Data Center are utilized by the LP DAAC as well, specifically GloVIS, EE, Tracking Routing and Metrics (TRAM), and the USGS Shopping Cart.

These Enterprise Components help broaden the distribution capabilities of the LP DAAC and tie LP DAAC products into the larger realm of Earth Resources Observation and Science (EROS) Data Center products. Some items shown are not directly part of the LP DAAC system, but are used to provide working support, planning, and integration activities (LP DAAC Internal Tools).

Table 2-2 lists the components in Figure 2-7 including descriptions, examples and key architecture function for the components.

Component Description Component Owner Function Archive Inventory Management (AIM)

DB

The AIM DB provides a database of the entire ECS archive.

ECS Archive

ASTER Data Acquisition Request Tool (DAR Tool)

System to allow ASTER science users to request acquisitions from the ASTER instrument.

LP DAAC Ingest

ASTER Emergency Scheduling Interface and Control System

(AESICS)

Component used to schedule ASTER collections for volcano monitoring.

LP DAAC Ingest

ASTER Expedited Used to receive ASTER Level 0 expedited data from Goddard and process that data to Level 1 products. The Expedited Data Set (EDS) interface allows users to view and download recently ingested expedited scenes.

LP DAAC Ingest / Distribution

ASTER Ingest DUE Used for ingesting ASTER data from GDS (Japan).

LP DAAC Ingest

ASTER Secondary Archive

ASTER L1A data is backed up by services provided by the LTA project. This provides a secondary archive for the L1A data.

EROS LTA Archive

ASTER Validation

(ASTERVAL)

Tool to validate EOS ASTER data.

LP DAAC Data Management

Automated NASA Science Users Registration

(ANSUR) / NASA

Affiliated Users

List of science users allowed to receive ASTER data for free.

LP DAAC User Support

Bulk Metadata Generation Tool

(BMGT)

ECS component for performing distribution used to publish data holdings metadata externally.

ECS Ingest/Archive

Catalog Reconciliation Tool

(CRT)

Tool to reconcile catalogs and metadata repositories of LP DAAC Data.

LP DAAC Data Management

Client Subsystem

(CLS)

Subsystem for interacting with ECS clients.

ECS Distribution/ Production

Cloakware Password repository COTS software that enables easier configuration of database passwords for developed systems.

LP DAAC Infrastructure

Communications Subsystem (CSS)

Subsystem to support communication between ECS subsystems.

ECS Infrastructure

Data Distribution (DBDirect) system

LP DAAC DUE that provides DAAC browse and metadata updates to EROS systems external to the LP DAAC.

LP DAAC Archive

Data Management System (DMS)

ECS component to manage metadata in the Online Archive.

ECS Data Management

Data Management Tool

(DMT)

Tool to manage the metadata inventory.

LP DAAC Data Management

Data Pool Public and Hidden areas ECS Archive

(DPL) for on-line data access from spinning disk.

Data Pool Ingest (DPL Ingest)

ECS ingest components providing capability to ingest ASTER and

MODIS.

ECS Ingest

Data Processing Options Service

(DPOS)

Web service that provides processing options for LP DAAC data.

LP DAAC Distribution

Database (ECS Database)

The LP DAAC infrastructure includes databases that support LP DAAC functions. The ECS system also utilizes a database to support its functions.

LP DAAC / ECS Infrastructure

DocuShare Document Management system used by the LP DAAC to handle and organize management, engineering, and support documents.

LP DAAC DAAC Tools

Drupal (LP DAAC Website)

The content management system (CMS) utilized for the LP DAAC website.

LP DAAC DAAC Tools

Earth Explorer (EE) USGS User Interface that can be used to order ASTER and MODIS data from the LP DAAC

EROS Long Term Archive (LTA)

Distribution

ECHO WSDL

Ordering Component

(EWOC)

ECHO component for entering orders, requesting science data or interfacing with S4PM.

NASA Distribution

EOS Clearing House (ECHO)

Provide inventory and metadata content to enable distribution.

NASA Distribution

EOS Validation

(EOSVAL)

Tool to validate EOS MODIS data.

LP DAAC Data Management

EOSDIS Service Interface (ESI)

Service wrapper that enables data services to be applied to orders.

ECS Distribution

ESDIS System Metrics (EMS)

ESDIS system used for metrics reporting, management, and operational support.

NASA Monitoring

Global Data Explorer (GDEx)

Client to allow selection of an AOI and downloading of mosaicked and subsetted data from various global coverage datasets, including the ASTER

GDEM.

LP DAAC Distribution

GloVIS USGS Client that provide visual discovery access to LP DAAC data sets

EROS LTA Distribution

HDF to GeoTIFF Conversion Tool

(HEG)

ECS component that allows for the conversion of products to alternate formats.

ECS Distribution

HTTP / FTP Access to LP DAAC data is provided via the HTTP and FTP protocols.

ECS / LP DAAC Access

Internetworking Subsystem (ISS)

Network configuration and tools to support ECS.

ECS Infrastructure

Landsat Metadata in Echo (LMdiE)

Module that sends Landsat metadata information to NASA

ECHO.

LP DAAC Archive

Log Logic Log Monitoring

EROS utility used to monitor server log files.

EROS (DAAC Led) Monitoring

LP DAAC Web Site (lpdaac.usgs.gov)

Public web site for the LP

DAAC.

LP DAAC User Support / Science Support

Machine to Machine Gateway

(MTMGW)

Facilitates FTP push of subscribed to data to external users.

MODIS

Reprojection Tool

(MRT)

Data reformatting, reprojection, mosaicking, etc. of MODIS data.

MRT is a user downloadable tool.

LP DAAC Distribution

MRT Swath Data reformatting, reprojection, mosaicking, etc. of MODIS Swath data. MRT Swath is a user downloadable tool.

LP DAAC Distribution

MRTWeb Visual browse/selection interface for the MRT that processes tiled MODIS user data on LP DAAC Servers.

Online Archive ECS components for storing the inventory and metadata that may be published externally.

ECS Archive

Order Manager System (OMS)

ECS component to manage ECS orders.

ECS Production/ Distribution

Processing XML Gateway

(PXG)

Utilized to move process flow between ECS, TRAM, and LP DAAC components.

LP DAAC Production / Distribution

Reverb EOSDIS Client that provides discovery access to LP DAAC data sets.

NASA Distribution

SA / DBA Tools There are a handful of packages and programs that are used by the System Administrators and Database administrators to monitor and maintain LP DAAC systems.

LP DAAC DAAC Tools

Simple, Scalable, Script-based Science Processor for Measurements

(S4PM)

Provides ASTER On-demand and Forward Processing, also implements processing algorithms as a subcomponent for other DAAC systems.

LP DAAC Production

Source Code Repository (SVN, Clearcase)

Code repository used by the LP DAAC. Clearcase is supplied by and contains ECS code.

LP DAAC / ECS DAAC Tools

Spatial Subscription Server (SSS)

System to manage user communication triggers when data is ingested.

ECS Distribution

Support Center Plus

Customer Service system used by the LP DAAC.

LP DAAC DAAC Tools

System Monitoring Automation

(SYSMA)

System monitoring and automation system for ECS and the LP DAAC.

ECS Monitoring

Tape Archive ECS components for deep archive of the data on tapes.

ECS Archive

TestTrack Pro Trouble ticket tracking for ECS and the LP DAAC.

ECS DAAC Tools

Total Cloud Cover

(TCC)

Module that reads and forwards cloud cover data for ASTER to GDS.

LP DAAC Data Management

Tracking, Reporting and Metrics

(TRAM)

USGS interface to manage orders through GloVIS and EE.

EROS LTA Distribution

Traffic Cop / Order Controller (TCOC)

Traffic Cop Order Controller that interfaces with TRAM for placing orders.

LP DAAC Distribution

User Registration Service (URS)

URS allows for registration and authentication of EOS users. It provides consistency of access to users of EOS systems and DAAC systems.

NASA Infrastructure

USGS Shopping Cart

USGS module used to track items ordered by users.

EROS LTA Distribution

Web Enabled Landsat Data

(WELD)

WELD archiving and distribution systems.

LP DAAC Distribution

Web Services Infrastructure

Infrastructure to support web services in use by the LP DAAC.

LP DAAC Infrastructure

Table 2-1 LP DAAC Components

2.4 LP DAAC Use Case Examples

The following figures show the basic processes of the LP DAAC at a high level.

Components from the previous section are shown here to present processing flow.

2.4.1 Ingest and Archive

Archiving is the core functionality of the LP DAAC and ingest is the means by which the archive is populated. The LP DAAC is capable of ingesting and archiving many different types of data. The LP DAAC ingests ASTER Data from GDS and MODIS data from MODAPS. In addition the LP DAAC ingests ASTER expedited data from Goddard Spaceflight Center (GSFC), and then sends it out to GDS.

2.4.1.1 ASTER Ingest

The LP DAAC Ingest process for ASTER Data is shown in Figure 2-8.

Figure 2-8 LP DAAC ASTER Ingest

Step Title Description

1 Receive ASTER Data from GDS

ASTER data is received from GDS. The ASTER ingest DUE receives the data.

2 ASTER Ingest DUE The ASTER Ingest DUE prepares the ASTER data for archiving. The data is sent to the Data Pool Ingest.

3 Data Pool Ingest The Data Pool Ingest stores the ASTER data to the Data Pool disk and writes out metadata information to the Data Pool. Note: Due to ASTER data distribution rules the ASTER data and metadata may be on a hidden (restricted) area of the Data Pool.

4 Tape Archive Data from the Data Pool is archived to the Tape Archive.

Table 2-2 LP DAAC ASTER Ingest

2.4.1.2 MODIS Ingest

The ingest process for MODIS data is shown in Figure 2-9.

Figure 2-9 LP DAAC MODIS Ingest

1 Receive MODIS Data from

MODAPS

MODIS data is received from MODAPS. The Data Pool Ingest receives the data.

2 Data Pool Ingest The Data Pool Ingest stores the MODIS data to the Data Pool disk and writes out metadata information to the Data Pool.

3 Tape Archive Data from the data pool is archived to the Tape Archive.

Table 2-3 LP DAAC MODIS Ingest

2.4.1.3 ASTER Expedited

For expedited ASTER data the LP DAAC reverses its role with GDS and receives ASTER data from Goddard Space Flight Center (GSFC), informs GDS of receipt of the expedited data, and sends this data to GDS once it has been ingested. This ASTER Expedited process is shown in Figure 2-10.

Figure 2-10 LP DAAC ASTER Expedited Ingest

1a Receive ASTER Expedited Data from GSFC

ASTER Expedited data is received and saved to the Data Pool Ingest system from the EDOS system at

GSFC.

1b Data Pool Ingest Data Pool Ingest receives the ASTER Expedited data from S4PM.

2 Processing ASTER Expedited Data

The S4PM system receives the ASTER Expedited input files and processed the data to create an AST_L1AE and an AST_L1BE granule.

3 Data Pool The expedited data is stored in the Data Pool.

Metadata is stored in the Data Pool.

4 Tape Archive The data is backed up to Tape Archive.

5 Notification sent to

GDS

The Spatial Subscription Server monitors ingest and sends an email notification to GDS for any AST_EXP data that has been ingested. The data will be sent to GDS only at GDS’s request.

Table 2-4 LP DAAC ASTER Expedited Ingest

2.4.2 Production, Ordering, and Distribution

The other key function of the LP DAAC is to provide data distribution. In some cases ordering systems are used to manage distribution and in other cases distribution is direct. Also, based on the type of data being ordered the user may have the ability to have the LP DAAC process the data to their specifications. S4PM and MRTWeb are some of the processing tools provided by the LP DAAC.

2.4.2.1 Order through Reverb

There are many different ways to discover and order data from the LP DAAC. One of these methods is through Reverb, the ordering tool created by the NASA ECHO project. Reverb ordering is shown in Figure 2-11.

Figure 2-11 Reverb Ordering

1 Reverb User uses Reverb to search for and order granules.

2 ECHO ECHO provides granule data to Reverb and tracks the order.

3 EWOC In ECHO the EWOC component receives the order from TCOC.

4a Order Manager System

An order is entered in the ECS Order Manager System.

4b Processing XML Gateway

For ASTER orders the Processing XML Gateway (PXG) is called and used to feed the order processing options to S4PM.

4c S4PM Data Processing

For ASTER orders S4PM processes the data per the user specifications.

5 OMS puts order to download directory

Ordered granules are put to an order directory on the Data Pool for download.

6 OMS Completes Order

OMS sends a completed order notification to Traffic Cop / Order Controller.

7 TCOC completes ECHO Order

TCOC sends ECHO the completion notification.

8 OMS Sends e-mail OMS sends e-mail to the customer with information on where to download the order from.

9 Receive order via

HTTP

User receives the order via HTTP from the order directory on the Data Pool.

Table 2-5 Reverb Ordering

2.4.2.2 Order data through Data Pool

Another way of ordering and distributing LP DAAC data is through the Data Pool.

Data Pool ordering is shown in Figure 2-12.

Figure 2-12 LP DAAC Data Pool Order

1 User opens the Data Pool Search Client

The user goes to the Data Pool Search Client and enters the search criteria. The search client drills down through selection details to provide a list of matching granules.

2 Data Pool Granules are found on the Data Pool and can be added to the shopping cart. Users may also choose to view browse images, granule metadata or http download the hdf science file.

3a Data Pool (download page)

For data to be downloaded directly the user selects granules desired. The granules are listed in a single directory as links to download.

3b HdfEOS-to- GeoTIFF Converter (HEG) Processing Orders

Some collections in the Data Pool allow for HEG processing. The user would place the desired granules in the shopping cart and select the desired processing options.

4 Order Manager After the HEG tool has processed the data, it sends the order information to Order Manager. Order Manager sends an email notification to the user with information about getting their data.

5 HTTP Download the granules to the user’s computer. For HEG processed orders, the user will download them from the order directory containing the HEG output.

Table 2-6 LP DAAC Data Pool Order

2.4.2.3 Order Data through GloVIS

The USGS provides the GloVIS visualization tool for visualizing and ordering LP DAAC data. The GloVIS ordering process is shown in Figure 2-13.

Figure 2-13 LP DAAC GloVIS Order

1 User opens the GloVIS client

The user goes to GloVIS and searches for desired granules.

2 Order Scenes Granules selected in GloVIS are ordered. Order information is passed to TRAM.

3 Shopping Cart Ordered item are kept in the USGS Shopping cart.

For ASTER orders the shopping cart checks the ASTER Affiliated Users list.

4 Traffic Cop / Order Controller

TCOC collects order from USGS Shopping cart.

5 EWOC ECS EWOC receives the order from TCOC.

6a Processing XML Gateway

For ASTER the order is sent to the Processing XML Gateway.

6b S4PM PXG sends the ASTER order to S4PM for processing.

6 Order Manager The order is sent to OMS.

7 E-mail Notification OMS sends an e-mail to the user indicating order is available for download.

8 HTTP User HTTPs data from order location on Data Pool.

A Update Browse and Metadata

Browse and metadata are periodically updated using the DBDirect data distribution.

Table 2-7 LP DAAC GloVIS Order

2.4.2.4 Distribute and Manipulate data with MRTWeb

A LP DAAC specific tool for downloading MODIS data is the MRTWeb DUE. The MRTWeb interface utilizes graphical interface code from GloVIS and interacts with the Data Pool to get granules to process.

Figure 2-14 LP DAAC MRTWeb

1 User opens the MRTWeb interface

The user goes to the MRTWeb interface and selects the desired product and granules.

2 Data Pool Selected granules are retrieved from the Data Pool.

Note: Only Data Pool granules are orderable from MRTWeb.

3 MRTWeb MRTWeb processes the information and creates the mosaicked, reprojected, and subsetted product.

4 MRTWeb Download Page

The output jobs are downloadable from the MRTWeb download page.

A Update Browse and Metadata

Browse and metadata are periodically updated using the DBDirect data distribution.

Table 2-8 LP DAAC MRTWeb

2.5 LP DAAC Physical Architecture

2.5.1 Network Architecture

The LP DAAC uses a handful of network subnets to connect their systems. The subsets are assigned based on what the system’s function is and also based on what the access restrictions are. All of these subnets are Class B subnets.

• – ECS equipment

• – Non-ECS equipment

• – DMZ systems (open to public access), proxy servers

• – High volume DMZ and proxy servers (open to public access)

At USGS EROS, pages and services delivered via the Internet to end users are filtered by a Web Application Firewall (WAF) appliance. The WAF provides extra security for network traffic to and from EROS. LP DAAC systems that are filtered through the WAF include:

• ASTER DAR Tool

• GDEx

• MRTWeb

• LP DAAC Website

• AESICS

• ASTER EDS

• ASTERVAL / EOSVAL

2.5.2 Systems Architecture

The LP DAAC is comprised of many servers that carry out and support the functions of the DAAC. In Figure 2-15 the systems in the LP DAAC are shown. Servers in the ECS and DMZ (internet accessible) are shown separated from LP DAAC specific servers.

SAN storage is available in the ECS and also for the other DAAC servers.

2.6 LP DAAC Component Detail

The following section contains a list of LP DAAC components, Software Packages, and DUEs as found in Table 2-2 above. The table also includes the Module Owner and function served by each component.

The following paragraphs describe each of these components and their purpose in the LP DAAC in detail.

2.6.1 AESICS

ASTER Emergency Scheduling Interface and Control System (AESICS) provides a streamlined approach for authorized users to enter ASTER expedited scheduling requests, and reduces the time required for personnel at the LP DAAC and the Jet Propulsion Laboratory (JPL) to respond to and monitor these requests. This effort also supported a NASA–funded initiative to automate the ASTER urgent request protocol for volcano monitoring.

Figure 2-16 AESICS System

The AESICS processing flow is shown in Figure 2-16. The steps to the process are outlined below.

1 User logs on to the AESICS web site

The user enters the data for the acquisition request.

1a View the ASTER Overpass predictor

The ASTER overpass predictor is a link from the AESICS Request page. The user may open this link to see possible acquisition times.

2 Submit Request The request is submitted to ASTER JPL. A confirmation e-mail about the request is sent to the user.

3 Confirm Request ASTER JPL confirms the acquisition request and sends it on to GDS. It also sends a confirmation e-mail to the user.

4 Schedule Request GDS schedules the request for the satellite.

5 Acquire The request is acquired according to the schedule.

6 Downlink data The data for the request is downlinked from the satellite.

7 ASTER Expedited The data is fed to the ASTER expedited system for ingest and processing.

8 Recent ASTER Expedited Browse

The recent ASTER Expedited browse window is populated with the latest expedited scenes. E-mail is sent to user indicating receipt of data.

9 Get Data The user can download data from the DAAC HTTP site, or from a link on the Recent ASTER Expedited browse window.

Table 2-9 AESICS System

This system benefits customers, ASTER JPL staff, LP DAAC Operations staff, and LP DAAC Science staff by enabling a simpler method for quickly scheduling expedited ASTER requests. More information on AESICS can be found in the LP DAAC DocuShare repository in systems -> AESICS.

2.6.2 ASTER DAR Tool

The ASTER Data Acquisition Request (DAR) Tool allows users to request acquisition of ASTER data based on specific user specified settings for spatial, temporal, cloud cover and sensor settings. Requests are sent from the DAR Tool to GDS in Japan and a response is sent to the user indicating if their data will be scheduled for acquisition. If the data is acquired successfully, the user will be notified that their DAR has been acquired. If the data is not acquired successfully, the user will be informed by GDS of the problem and the reason that the data was not acquired. After the granules from the request are ingested at the LP DAAC, the user will be notified by the DAR tool that the images are available and provide a link to the user to order the granule(s) requested.

Users of the DAR tool need to be authorized by the ASTER team at JPL to receive a userid, which allows logging into the DAR Tool and requesting acquisitions. Users may log in to the DAR tool as guest, but guests are not allowed to request acquisitions. The number of acquisition requests allowed for the user will be set by the ASTER team and users who use up their allocation of requests must submit an additional request for more acquisitions.

Figure 2-17 ASTER DAR Tool

The ASTER DAR Tool processing flow is shown in Figure 2-17. The steps to the process flow are outlined below.

0 Request DAR tool access.

The user requests access to the DAR tool and an acquisition request allotment. The user-registered account is tied to an ECHO account.

1 User logs on to the ASTER DAR Tool web site

The user logs in to the DAR Tool UI.

2 The user’s DAR Request, DAR Query, or DAR Modification is sent from the User Interface (UI).

The DAR Request, DAR Query, or DAR Modification is sent from the UI through the ASTER DAR Tool Communications interface to GDS. Status of the request is received from GDS and the user is sent an e-mail identifying whether their request has been approved or rejected. If approved the request is scheduled for acquisition. A subscription for the DAR is entered into the ECS system.

3 Acquire Data The ASTER scene is acquired and ingested into

ECS.

4 Inform user of Ingest

The ECS Spatial Subscription server informs the user of the DAR being acquired via e-mail.

Table 2-10 ASTER DAR Tool

Prior to ECS 7.20, the DAR tool was part of the ECS baseline. As part of the ECS evolution activities, the LP DAAC rebuilt and redeployed the DAR tool, interfacing it directly to GDS and JPL.

2.6.3 Cloakware

Cloakware is a tool that is utilized by the LP DAAC to manage and control access to LP DAAC databases. Cloakware has API’s that are used to manage access to databases by applications and modules. This simplifies the integration of applications with the database and enhances the security for database accessing applications.

2.6.4 ASTER Ingest

The ASTER Electronic Data Transfer of L1A data from Japan is necessary to download L1A data and format it so ECS can ingest it. Originally, the interface from GDS to ECS was designed utilizing a manual tape transfer. With the inception of the ASTER Ingest DUE package, the data is electronically transferred from GDS via FTP and formatted for ECS ingest.

Figure 2-18 ASTER Ingest DUE

1 GDS Production ASTER data is received and processed at GDS.

Both Data files and Browse files are produced.

2 GDS Product Delivery Record (PDR) Generation

A Product Delivery Record (PDR) is created. The PDR and the Data and Browse are put onto the GDS Data Server.

3 ASTER Ingest DUE Poll

The ASTER Ingest DUE polls for new PDRs.

4 Pull Data Data (and Browse) is pulled from GDS based on the

PDR.

5 Ingest ASTER The ASTER Data (and Browse) to ingest is sent to Data Pool Ingest and stored to the Online Archive.

6 Notify GDS If the data was transferred to the LP DAAC successfully, a success notification is send to the GDS PDR Generation system.

In the event of a failure to receive data or PDRs from GDS, a failure notification is sent to the GDS PDR Generation system.

Table 2-11 ASTER Ingest DUE

More information on the ASTER Ingest DUE and its interface to ECS can be found in the ECS-LP DAAC Interface Control Document (ICD) located in the ESDIS ROMULUS system. General information on the DUE can be found in the LP DAAC DocuShare repository in Systems >> ASTER Electronic Ingest.

2.6.5 ASTER Expedited Interface to GDS

Originally the ASTER expedited interface to GDS was operated out of the Goddard Earth Sciences (GES) Data and Information Services Center (DISC). With system evolution activities and the termination of ECS in favor of Simple, Scalable, Script- Based, Science Product Archive (S4PA) at the GES DISC, this DUE was transitioned from the GES DISC to the LP DAAC. Figure 2-19 shows a summary of the interfaces transitioned from the GES DISC to the LP DAAC in the fall of 2007.

Figure 2-19 ASTER Expedited

1 EOS Data and Operations System

(EDOS)

EDOS sends L0 data to GES DISC

2 GES DISC GES DISC sends L0 data to LP DAAC.

3 LP DAAC Upon receipt of L0 data, the LP DAAC sends an Expedited Data Notification (EDN) to GDS

4 GDS For expedited data that GDS wants to order an Expedited Data Request (EDR) is send back to the

LP DAAC.

5 LP DAAC Upon receipt of the EDR, the LP DAAC creates an order for the expedited data and pushes it to GDS.

6 Distribution Notification

After distribution of the L0 data, the LP DAAC sends a XFR file and distribution notification to GDS.

Table 2-12 ASTER Expedited

When ASTER Expedited data is received and ingested at the LP DAAC, it can be accessed and downloaded using the ASTER Expedited Data Set Archive (EDS) web interface. The ASTER EDS is shown in Figure 2-20.

Figure 2-20 ASTER Expedited Data Set Archive

2.6.6 ANSUR / ASTER Free List

The Automated NASA Science Users Registration (ANSUR), aka the ASTER Free List, allows users to request access to ASTER data at no charge. The free list is a database of users managed by the LP DAAC that may obtain ASTER data at no charge to the user. To gain approval for ASTER data at no cost to the customer, a customer must request access through a form available on the LP DAAC public web site. Once the user submits the request, it is sent to NASA officials for approval. Upon approval, the

2.6.9.2 Oracle

An Oracle database is used to provide database services to LP DAAC DUE’s and for the LP DAAC website. The USGS has Enterprise Licensing that allows for the LP DAAC to utilize Oracle for its internally facing systems. Extra licensing for Oracle is needed for deploying systems that are accessible externally (i.e. websites).

2.6.9.3 Java Enterprise Edition Services

Many of the LP DAAC’s DUE’s rely on underlying Java Enterprise Edition services.

JEE or J2EE services provide an environment for executing web accessible code and services through an HTTP server. Java Enterprise Edition services are provided by JBoss 5.0, or by Tomcat 6 frameworks. DAAC DUE’s using Java Enterprise Edition functionality include:

• ASTER DAR Tool - Tomcat

• MRTWeb - JBoss

• GDEx - Tomcat

• DPOS – Jboss

• Processing XML Gateway - JBoss

• TC/OC - JBoss

2.6.9.4 StorNext

Access to the ECS SAN as a shared file system is provided by use of StorNext software. StorNext allows multiple servers to access data from the ECS SAN concurrently.

2.6.9.5 DocuShare

The LP DAAC utilizes DocuShare, a document management system, to manage project documentation and information. Documents are stored in the following hierarchy:

2.6.9.6 Legato

Legato backup software is utilized to backup LP DAAC servers and systems.

2.6.9.7 Log Logic

Log Logic is used by the LP DAAC to manage, archive, and consolidate log information for LP DAAC systems.

2.6.10 Data Pool Search Client

The LP DAAC hosts a Data Pool search client. The ECS Data Pool Search Client allows users to search for ASTER and MODIS granules that are found on the Data Pool. Users select the desired product, timeframe, area of interest, and other parameters and are presented with granules that match the criteria.

In addition to the Data Pool Search Client, MODIS and ASTER granules can be manually downloaded directly from the Data Pool via HTTP.

2.6.11 Data Management Tool (DMT)

LP DAAC Data Management staff perform operational activities for the data catalog and handle exporting the catalog by utilizing the Data Management Tool (DMT). The management of the catalog is based on:

1) Science requirements

2) LP-DAAC policies that refer to and interpret the science requirements

3) Operational practices that

a) Implement the LP DAAC policies

b) Improve and maintain catalog integrity with an end result of customer satisfaction.

The requirements and policies affecting the catalog granules vary by collection and by data types within a collection. Implementing the various policies makes the management and validation of the operational catalog a cumbersome and time-consuming activity. This is even true utilizing the current tools.

The majority of catalog maintenance activity involves updating and removing rows from various fields in a relational database as well as removing data from the file storage systems. Currently each LP DAAC policy and/or operational script used to maintain catalog integrity is implemented as separate and independent scripts. Before any script can be activated, each script must be manually edited making them very susceptible to human errors. The DMT provides a Graphical User Interface (GUI) for running scripts and also tracks data management activities for integrity and reporting purposes.

2.6.12 Data Distribution System (DBDirect)

The DBDirect component of the LP DAAC is used to distribute data to other systems that use LP DAAC data (i.e. GloVIS and EE). The external system queries a database that contains information about recent DAAC changes that will need to be propagated through DBDirect.

Figure 2-25 DBDirect Process Flow

1 DB Activity Ingest of new data, and deletes or updates of existing data occur. Those updates are stored to the Archive Inventory Management Database.

2 Shadow Table A shadow table tracks changes to the database and stores them.

3 Transformer Service

The Transformer Service reads the changes from the shadow table and gathers info from the XML inventory files and browse files and outputs the inventory XML and browse jpeg.

4 DB Direct database The inventory XML and browse jpeg are stored to the DBDirect database. The DBDirect database is also reconciled by the Transformer service.

5 Archive Access The Archive Access service allows the external system (GloVIS in this case) to read the new inventory XML and browse files that will be used to update the external system’s inventory.

Table 2-14 DBDirect Process Flow

Browse and metadata information is propagated to other systems through DBDirect.

More information on DBDirect can be found in the LP DAAC DocuShare repository in Systems >> DBDirect.

2.6.13 Drupal

The LP DAAC website utilizes the Drupal Content Management System (CMS). There are multiple Drupal servers that have been created to handle the load put on the LP DAAC website. The User Services, Science, and Operations group manages the content on the website utilizing the Drupal interface to update and change content for the LP DAAC website.

2.6.15 Earth Explorer (EE)

The USGS Long Term Archive (LTA) project allows searching LP DAAC data through its Earth Explorer Interface. Once LP DAAC data information is returned through an EE search, it can either be downloaded directly or ordered. The product selected determines the options available for distribution from the LP DAAC. Products searchable through EE are:

• ASTER L1B

• ASTER Global Digital Elevation Model (GDEM)

• NAALSED

• MODIS Daily products

• MODIS Composite products

Figure 2-27 Earth Explorer Process Flow

A DBDirect DBDirect transfers browse and metadata information for products searchable in EE.

1 Earth Explorer User inputs search and product criteria in EE.

2 Select Product User selects desired products for download.

3 Data Pool Products are downloaded via HTTP from the data pool to the user.

Table 2-15 Earth Explorer Process Flow

2.6.16 EOS Clearing House (ECHO)

The EOS Clearing House is a repository that stores metadata for Earth remote…