Task_1_Reference_Document__LDCM_OCD_007.pdf

LDCM-OCD-007

Version 7.0

Department of the Interior U.S. Geological Survey

LANDSAT DATA CONTINUITY MISSION (LDCM)

DATA PROCESSING AND ARCHIVE SYSTEM (DPAS)

OPERATIONS CONCEPT DOCUMENT (OCD)

May 2013

- ii - LDCM-OCD-007

Executive Summary

The Landsat Data Continuity Mission (LDCM) is a remote sensing mission which provides data continuity to the Landsat satellite series global multispectral data collection and distribution. LDCM is a satellite- and ground-based collection providing the following capabilities:

Global, moderate-resolution, multispectral data collection

Long term LDCM data archiving

Web-enabled access

Continued Landsat International Cooperators (IC) support

Level 0 (L0) and Level 1 (L1) data products

The LDCM Project consists of three major mission components: the Space Segment (SS), Ground Segment (GS), and Launch Services Segment (LSS). Because LDCM is a cooperative effort between National Aeronautics and Space Administration (NASA) and the U.S. Geological Survey (USGS), each agency has specific responsibilities for delivery of major overall mission capabilities. The Data Processing and Archive System (DPAS) is part of the LDCM GS.

The DPAS Operations Concept Document (OCD) presents the operational concepts, functions, and interactions of the DPAS.

- iii - LDCM-OCD-007

Contents

Executive Summary ...................................................................................................... ii

Document History ............................................................. Error! Bookmark not defined.

Contents ........................................................................................................................ iii

List of Figures .............................................................................................................. vi

Section 1 Introduction

1.1 Background

1.2 Purpose and Scope

1.2.1 Document Organization

Section 2 System Inputs and Outputs

2.1 Major Inputs

2.1.1 Block Address Data (BAD)

2.1.2 Digital Elevation Model (DEM)

2.1.3 Ground Control Point (GCP)

2.1.4 Interval Definition File (IDF)

2.1.5 Mission Data

2.1.6 UTC / UT1 / Pole Wander

2.2 Major Outputs

2.2.1 Bias Parameter File (BPF)

2.2.2 Bulk Metadata

2.2.3 Calibration Product

2.2.4 Calibration Parameter File (CPF)

2.2.5 Characterization Data

2.2.6 Full Resolution Browse (FRB)

2.2.7 L0Ra

2.2.8 L0Rp

2.2.9 L1Gt

2.2.10 L1T

2.2.11 Quality Band (QB)

2.2.12 Reduced Resolution Browse (RRB)

2.2.13 Response Linearization Look Up Table (RLUT)

Section 3 LDCM DPAS Overview

3.1 Subsystem Descriptions

3.1.1 Storage and Archive Subsystem (SA)

3.1.2 Inventory Subsystem (INV)

3.1.3 Ingest Subsystem (IS)

3.1.4 Subsetter Subsystem

3.1.5 Level 1 Product Generation Subsystem (LPGS)

3.1.6 Image Assessment Subsystem (IAS)

3.1.7 User Portal (UP) Subsystem

3.1.8 Mission Management Office (MMO) Database Subsystem (MDS)

3.2 Priority Processing

Section 4 DPAS Operations

- iv - LDCM-OCD-007

4.1 Data Processing, Archive, and Distribution Operations

4.1.1 Operations Facility Layout

4.1.2 Day In The Life (DITL) Overview

4.2 Operational Roles

4.2.1 DPAS Operators

4.2.2 Data Management Personnel

4.2.3 Software Developers

4.2.4 System Engineers

4.2.5 System Administrators

4.2.6 Hardware Engineers

4.2.7 Database Administrators

4.2.8 Calibration/Validation (Cal/Val) Analysts

4.2.9 Data Quality Analyst (DQA)

4.2.10 International Cooperators (IC)

4.2.11 Mission Management Office (MMO)

4.2.12 Configuration Management (CM)

Section 5 Scenarios

5.1 Scenario Conventions

5.2 Standard Processing

5.2.1 Ingest Mission Data

5.2.2 Standard Product Generation

5.2.3 Browse Generation and Quality Band

5.3 On-Demand Processing

5.4 Mission Data Reprocessing

5.5 Data Management

5.5.1 Archive Management

5.5.2 Storage Management

5.5.3 Data Manager Interaction

5.6 Reporting (MDS)

5.6.1 Metrics Reporting

5.6.2 Collection Reconciliation

5.7 User Portal

5.7.1 Registration

5.7.2 Search & Order

5.7.3 Information Dissemination

5.7.4 Collection Request

5.8 Calibration Validation Operations

5.8.1 Calibration/Validation (Cal/Val) Processing

5.8.2 Calibration Parameter Generation

5.9 International Cooperators (IC)

5.9.1 Data Validation

5.9.2 Data Exchange

5.9.3 Mission Data Distribution

5.9.4 Metadata Ingest

5.9.5 Communication Products

5.10 Anomaly Resolution

- v - LDCM-OCD-007

5.10.1 Ingest Error Recovery

5.10.2 Product Generation Error Recovery

5.10.3 Block Address Data (BAD) Handling

5.11 Continuity of Operations

5.11.1 Data Recovery

5.11.2 System Recovery

5.11.3 Routine Backups

Appendix A Scenario to Requirements Trace ............ Error! Bookmark not defined.

References

- vi - LDCM-OCD-007

List of Figures

Figure 1-1. LDCM Operations Concept Hierarchy Figure 3-1. LDCM Operations Concept Architecture Figure 3-2. DPAS Block Diagram

Figure 3-3. DPAS Context Diagram Figure 3-4. SA Block Diagram Figure 3-5. SA Context Diagram Figure 3-6. Inventory Block Diagram Figure 3-7. Inventory Context Diagram

Figure 3-8. Ingest Block Diagram Figure 3-9. Ingest Context Diagram Figure 3-10. Subsetter Block Diagram

Figure 3-11. Subsetter Context Diagram Figure 3-12. LPGS Block Diagram Figure 3-13. LPGS Context Diagram

Figure 3-14. IAS Block Diagram Figure 3-15. IAS Context Diagram Figure 3-16. User Portal Block Diagram

Figure 3-17. User Portal Context Diagram Figure 3-18. MDS Block Diagram

Figure 3-19. MDS Context Diagram Figure 3-20. Priority Processing Figure 4-1. Operations Facility Layout

Figure 4-2. Day-In-The-Life (DITL) Timeline

Figure 5-1. DPAS Scenario Conventions Figure 5-2. Standard Processing Data Flows Figure 5-3. Ingest Mission Data

Figure 5-4. Standard Product Generation Figure 5-5. Browse Generation and Quality Band

Figure 5-6. On-Demand Processing Figure 5-7. Mission Data Reprocessing Figure 5-8. Archive Management Figure 5-9. Storage Management Figure 5-10. Data Manager Interaction

Figure 5-11. Metrics Reporting Figure 5-12. Collection Reconciliation Figure 5-13. Registration

Figure 5-14. Search & Order Figure 5-15. Information Dissemination Figure 5-16. Collection Request Figure 5-17. Cal/Val Processing

Figure 5-18. Calibration Parameter Generation Figure 5-19. Data Validation Figure 5-20. Mission Data Distribution

Figure 5-21. Metadata Ingest

- vii - LDCM-OCD-007

Figure 5-22. Communication Products

Figure 5-23. Ingest Error Recovery Figure 5-24. Product Generation Error Recovery

Figure 5-25. Block Address Data Handling Figure 5-26. Data Recovery Figure 5-27. System Recovery Figure 5-28. Routine Backups

- 1 - LDCM-OCD-007

Section 1 Introduction

The Landsat Data Continuity Mission (LDCM) is a joint mission formulated by the National Aeronautics and Space Administration (NASA) and the U.S. Geological Survey (USGS). LDCM is a remote sensing satellite mission providing coverage of the Earth’s land surfaces. This mission continues the 40+ years of global data collection and distribution provided by the Landsat series of satellites.

1.1 Background

LDCM is a component of the Landsat Program conducted jointly by NASA and the USGS. The goal of LDCM is to continue the collection, archival, and distribution of multispectral imagery affording global, synoptic, and repetitive coverage of the Earth's land surfaces at a scale where natural and human-induced changes can be detected, differentiated, characterized, and monitored over time. The LDCM goal is in keeping with the Landsat programmatic goals stated in the United States Code (USC) Title 15, Chapter 82 “Land Remote Sensing Policy” (derived from the Land Remote Sensing Policy Act of 1992). This policy requires that the Landsat Program provide data into the future that are sufficiently consistent with previous Landsat data to allow the detection and quantitative characterization of changes in or on the land surface of the globe.

LDCM was conceived as a follow-on mission to the highly successful Landsat series of missions that provided satellite coverage of the Earth’s continental surfaces since 1972.

The data from these missions constitute the longest continuous record of the Earth’s surface as seen from space.

The LDCM ensures that Landsat-like data will be provided to the USGS National Satellite Land Remote Sensing Data Archive (NSLRSDA) for a ten-year mission life. A ten-year requirement provides a five-year nominal mission lifetime with an additional five years as risk mitigation against delays in future missions.

The Data Processing and Archive System (DPAS) Operations Concept Document (OCD) is one part of the LDCM Ground System OCD, which is part of the overall LDCM Operations Concept (Mission OCD). This hierarchy reflects the operational concepts that correspond to a relative “level” of requirements, which directly relate to a level of detail. The Mission OCD traces its concepts to GSFC-427-02-01 Landsat Data Continuity Mission (LDCM) Science and Mission Requirements Document (SMRD), LDCM-REQ-001 Landsat Data Continuity Mission (LDCM) Ground System Requirements Document (GSRD), and LDCM-REQ-015 Landsat Data Continuity Mission (LDCM) Data Processing and Archive System Requirements Document

(DPASRD).

- 2 - LDCM-OCD-007

Figure 1-1. LDCM Operations Concept Hierarchy

As shown in Figure 3-1, DPAS is part of the overall Ground System (GS), which is part of the overall mission. Functional decomposition and requirements traceability within this hierarchy ensures the capture of overall mission requirements and concepts at lower levels, specifically the LDCM-OCD-002 Landsat Data Continuity Mission (LDCM) Ground System (GS) Operations Concept Document (OCD), and this document. Error!

Reference source not found. contains a scenario to requirements traceability matrix, which forms the basis of test case development for formal qualification testing.

1.2 Purpose and Scope

The primary purpose of this document is to provide a description of the functions and operations of the systems, people, and processes that comprise the DPAS. It presents a functional view of the DPAS based on high-level LDCM program guidance and requirements. This document also aids in the development and validation of the DPAS requirements by facilitating test plan development.

The scope of this document includes all functions associated with the DPAS as well as the external entities that interact with the DPAS.

1.2.1 Document Organization

This document contains the following sections:

Section 1 describes the purpose, objectives, and document content.

Section 2 provides descriptions of the major inputs and outputs of the DPAS.

Section 3 presents an overview of the DPAS and its Subsystems.

Section 4 presents an overview of DPAS Operations, including operational roles.

Section 5 presents the DPAS operational scenarios.

Error! Reference source not found. provides a DPAS requirement to DPAS scenario traceability matrix.

- 3 - LDCM-OCD-007

Section 2 System Inputs and Outputs

2.1 Major Inputs

This section briefly describes the major inputs to the DPAS. For detailed information, consult LDCM-REF-010 Landsat Data Continuity Mission (LDCM) Ground System (GS) Lexicon.

2.1.1 Block Address Data (BAD)

Block Address Data (BAD) is a special collection of data that is manually commanded for acquisition by the Flight Operations Team (FOT) and captured by the Landsat Ground Network (LGN) stations. BAD is automatically transferred by the Data Collection and Routing Subsystem (DCRS) to the Archive Cache, where it is stored and retrieved as needed for analysis, primarily by the LDCM observatory vendor.

2.1.2 Digital Elevation Model (DEM)

A Digital Elevation Model (DEM) is a digital representation of ground topography stored as a series of raster images that is used for terrain correction during Level 1 (L1) processing. DPAS will use the Global Land Survey (GLS) and the Radarsat Antarctic Mapping Project (RAMP) DEM data sets.

2.1.3 Ground Control Point (GCP)

A Ground Control Point (GCP) is a point on the Earth’s surface with a known geographic location used to geo-reference image data sources. Each GCP is associated with one or more image chips. For the DPAS, the GCP image chips reside on the Archive Cache, along with a text file (GCPLib) for each path/row; the chip locations are recorded in an Image Assessment Subsystem (IAS) database. The chip source is the GLS Mid Decadal Land Survey Collection. Spatial / temporal queries are used to select specific GCPs and chips used for geometric corrections to the L1 products.

2.1.4 Interval Definition File (IDF)

The Interval Definition File (IDF) defines the set of Mission Data files belonging to a collection interval. The IDF contains descriptive parameters for the interval, the names of all of the Mission Data files, and (for appropriate collection types) the list of associated scenes. The Ground Network Element (GNE) DCRS generates an IDF for each interval and delivers it to the DPAS Ingest Subsystem along with the Mission Data.

The IDF is also archived with the Mission Data. A complete interval and an IDF are exchanged either to or from International Cooperators (IC) stations. The GNE (DCRS) provides the IDF file to the DPAS.

2.1.5 Mission Data

A Mission Data set is a collection of files created as a result of receiving instrument data from the satellite and processing it in the LGN. Processing includes removal of the Consultative Committee for Space Data Systems (CCSDS) File Delivery Protocol

- 4 - LDCM-OCD-007

(CFDP) protocols in the X-Band downlink. The spacecraft instrumentation implements the framing of the Mission Data files.

The GNE (DCRS) provides Mission Data to the DPAS.

2.1.6 UTC / UT1 / Pole Wander

Universal Time Code (UTC), UTC Corrected (UT1), and Pole Wander data are extracted from the U.S. Naval Observatory (USNO) and used in the development of Calibration Parameters (CP). UTC/UT1 are used for time corrections, which address the need to synchronize the observatory and Earth times. Similarly, Pole Wander information is used to correct for variances in the Earth’s rotation.

2.2 Major Outputs

This section briefly describes the major outputs of the DPAS. For detailed information consult LDCM-REF-010 Landsat Data Continuity Mission (LDCM) Ground System (GS) Lexicon.

2.2.1 Bias Parameter File (BPF)

The Bias Parameter File (BPF) supplies radiometric correction parameters required during L1 processing of Operational Land Imager (OLI) and/or Thermal Infrared Sensor (TIRS) Earth image and calibration data products. For an interval of time immediately after the OLI’s launch, the initial processing BPF contains values derived from analysis of prelaunch test data. Ingest automatically generates subsequent bias parameters and stores them in the IAS database based on analysis of OLI Shutter and TIRS Deep Space data collects throughout the mission lifetime. The IAS delivers the most recent BPF (applicable to a given Earth image or calibration data product) for L1 radiometric processing. BPFs can be updated as deemed appropriate by the Calibration / Validation Team (CVT) using IAS, such as in response to a confirmed change in bias model parameter values.

2.2.2 Bulk Metadata

Bulk metadata refers to a large volume of Searchable Inventory Database (SID) records, which occurs either as a result of a query or as information provided by an IC to indicate their data holdings. Bulk metadata is also available to researchers who want to perform analytical processing on data holdings within their area of interest.

2.2.3 Calibration Product

IAS creates calibration products as specified by CVT. Unlike standard products, which are distributed to the user community, calibration products are special order products, defined by parameters and/or process flow, for specific research needs by the CVT. An example is an L1 product with a mapping projection other than the Universal Transverse Mercator (UTM) projection.

2.2.4 Calibration Parameter File (CPF)

Calibration parameters are kept in the IAS database. When directed, the parameters are used to create an output file that forms the CPF. The Calibration Parameter File

- 5 - LDCM-OCD-007

(CPF) contains all parameters needed to derive the Level 0 Reformatted Product (L0Rp), Level 1 Systematic Terrain (Corrected) (L1Gt), and Level 1 Precision and Terrain (L1T) products from Level 0 Reformatted Archive (L0Ra) data. The Ingest subsystem also uses a CPF during processing of the Mission Data.

Example parameters contained within the CPF include radiometric conversion coefficients, radiometric artifact correction parameters, earth constants, orbit parameters, detector positions, and alignment parameters.

2.2.5 Characterization Data

Characterization data are a set of measurements that are generated during the production of L1 data from the L0Rp data by the Level 1 Product Generation Subsystem (LPGS) and also generated from L0Ra Ancillary, OLI Shutter, TIRS Deep Space and TIRS Blackbody data by the Ingest Subsystem. The measurement information is analyzed and tracked over time in order to describe the performance of the OLI and TIRS instruments, including stability and error rates.

2.2.6 Full Resolution Browse (FRB)

Full Resolution Browse (FRB) images are also known as LandsatLook. These are derived from L1 products (L1Gt or L1T) scaled from 16- to 8-bits per band by the User Portal (UP) Subsystem. The OLI FRB (or LandsatLook natural color) image comprises three spectral bands; the TIRS FRB (or LandsatLook thermal) image is a single band grayscale image with a linear stretch.

2.2.7 L0Ra

L0Ra data are image data with all data transmission and formatting artifacts removed, time provided, spatial, and band-sequentially ordered multispectral digital image data.

L0Ra data are interval-based and stored in a format to allow for faster generation of L0Rp products. L0Ra data are generated for all collection types other than BAD.

The Ingest Subsystem creates L0Ra data from Mission Data.

2.2.8 L0Rp

L0Rp data are image data with all data transmission and formatting artifacts removed, time provided, spatial, and band-sequentially ordered multispectral digital image data.

L0Rp data are subsetted to the Worldwide Reference System-2 (WRS-2) framing conventions.

L0Rp products include data from all instruments operating within the interval. If the interval was recorded with only a single instrument, then data from only one instrument is present in the L0Rp product. If the interval consisted of data from both instruments, the L0Rp product is include data from both.

The Subsetter Subsystem creates L0Rp data from the L0Ra data. The LPGS creates L0Rp products. The distinction is that checksum generation and data packaging are performed for the product.

- 6 - LDCM-OCD-007

2.2.9 L1Gt

L1Gt data are systematically corrected data resampled for registration to a cartographic projection and referenced to the World Geodetic System 1984 (WGS84), G873, or current version. The geometric corrections incorporate the use of observatory ephemeris data; DEM data are used to correct for terrain relief.

LPGS produces L1Gt products when cloud cover or water prevents the use of precision terrain correction.

2.2.10 L1T

L1T data products are radiometrically corrected, intermediate L1R data products resampled for registration to a cartographic projection; these products are referenced to the WGS84, G873, or current version, and are orthorectified and corrected for terrain relief. The geometric corrections use observatory ephemeris data and GCPs; DEM data are used to correct for terrain relief.

LPGS produces L1T products.

2.2.11 Quality Band (QB)

The Quality Band (QB) is created using OLI data, or OLI and TIRS data, for all sun-lit Earth acquisitions. It is a full resolution image band constructed by setting the bits in each pixel according to specified image quality criteria (such as cloud mask information) for the scene. A 16-bit QB is included with the L1 data products, and an 8-bit version is derived from the 16-bit version to allow mask overlay with the FRB.

LPGS produces 16-bit QBs during L1 processing. UP produces 8-bit versions.

2.2.12 Reduced Resolution Browse (RRB)

The Reduced Resolution Browse (RRB) is a subsampled browse image derived from the FRB. The UP generates and provides the RRBs to the Earth Explorer (EE) system and provides a “quick look” assessment capability during data searching by a user.

2.2.13 Response Linearization Look Up Table (RLUT)

The Response Linearization Look Up Table (RLUT) is an optional file that may accompany a CPF and contains a mapping look up table to linearize the output of the OLI detectors. This correction is used to increase the range of Digital Number (DN) values for preciseness and to limit the number of saturated pixels to true saturations (and on the dark end, limit it to missing data). The CVT initially creates the RLUT using the Cal/Val Tool Kit (CVTK), using the characterization algorithm.

Non-linearity characterization uses a series of variable integration time observations of the solar diffuser and corresponding shutter observations to generate a remapping for every digital number (4096 values per detector) for every detector. This remapping is performed via a Look Up Table (LUT) approach.

- 7 - LDCM-OCD-007

The CVT creates the RLUT and keeps it in the IAS database. When published, the RLUT is output to file and cached (Internal Cache) for use in L1 processing.

- 8 - LDCM-OCD-007

Section 3 LDCM DPAS Overview

Figure 3-1. LDCM Operations Concept Architecture

LDCM is a remote sensing mission which provides data continuity to the Landsat satellite series global multispectral data collection and distribution. LDCM is a satellite-and ground-based collection providing the following capabilities:

Global, moderate-resolution, multispectral data collection

Long term LDCM data archiving

Web-enabled access

Continued Landsat IC support

Level 0 (L0) and L1 data products

The LDCM Project consists of three major mission components: the Space Segment (SS), Ground Segment (GS), and Launch Services Segment (LSS). Because LDCM is

- 9 - LDCM-OCD-007

a cooperative effort between NASA and the USGS, each agency has specific responsibilities for delivery of major overall mission capabilities.

Figure 3-1 illustrates the overall LDCM architecture and the high-level interfaces and interactions between the mission components. As shown in Figure 3-1, DPAS is a component of the GS.

Fundamentally, the responsibility of the DPAS is data processing, storage, long-term archiving, and data and product distribution for the LDCM.

Subscriptions

Search/Requests, Orders*

Mission

Data

Processing

Status

L0Rp & L1 Delete Updates

Storage and Archive

Ingest

Image Assessment

Subsystem (IAS)

L0Rp

Request

L0Rp Data

DEM

RLUT

L0Rp & L1

Products

Histogram

Stats

CPF, BPF, GCP

RLUT Name

UTM Zone

Subsetter

L0Ra Data

L0Ra &

L0Rp

Requests

L0Ra Data

L0Rp Data

DEM

RLUT

RLUT

Mission

Data

L0Ra Data

Browse

Quality Band

CPF

BPF

RLUT

Order Status

(Cal/Val)

Inventory

Level 1 Product

Generation Subsystem

(LPGS)

L0Rp & L1

Metadata

L0Rp & L1 Product Orders

Order (Cal/Val)

L0Ra Location

Scene Extent Info

MMO Database

Subsystem (MDS)

Metadata

& Metrics

L0Ra Metadata & Location

Metrics

Order Info/Stats, Scene Transmit Schedule

Characterization Data*

CPF, RLUT Name

Metrics

L0Rp Data

External GS Element DPAS External Entity

Ground

Network

Element

(GNE)

Mission Data, IDF, BAD, Mission Data Move

Data/Products/Browse*

Metadata Search Results, CPF, BPF, RLUT, GCP, Cal/Val Reports, Collection Request

Status

Metrics

Mission

Data

Available

& Location

Scene, L0Ra &

Mission

Metadata

Mission Data Metadata & Location

Browse & QB

Location, IC Metadata

L0Rp Products

L1 Products

Browse

Quality Band

Mission Data

Metrics

Mission, L0Ra, L0Rp, L1, & Bulk Metadata

L0Rp Status

& Location

L0Rp Status

& Location

Order

Status

Deletion Notifications*

Collection

Activity

Planning

Element

(CAPE)

Mission

Operations

Element

(MOE)

Requests*

Scene Metadata (ACCA, Quality)

Scheduling

Products*

IC Mission Data

User

Portal

User

Community

* Additional detail can be found in the DPAS N2

User Request Types &

Parms

Collection Status*

RLUT

Search Requests

Communication

Products*

Mission Data Record Update

Product

Orders ICs

Mission Data

Scheduling Products*, Software*, Validation Reports, Collection Request

Status

Data Exchange

Collection Requests, Communication Messages*, Metadata, Post-Pass reports

Figure 3-2. DPAS Block Diagram

Figure 3-2 shows data flow interactions among DPAS Subsystems and with entities external to DPAS. The set of externals is not comprehensive in Figure 3-2. A complete interface listing for the DPAS is captured in the DPAS N2 diagram (LDCM-DWG-004 Landsat Data Continuity Mission (LDCM) Data Processing and Archive System (DPAS) N2 Diagram). Figure 3-3 is the context diagram for the DPAS, which provides a complete listing of its external interfaces.

Subsequent sections provide more detailed descriptions of each of the DPAS Subsystems.

- 10 - LDCM-OCD-007

Ground Network

Element (GNE)

IC Mission Data

Block Address Data

Metrics

External Collection Requests

Processing Status

Confirmed Contact Schedule

Contact Forecast

Visibility Schedule

U.S. Naval

Observatory

(USNO)

Ground Network

Element (GNE)

Mission Data, Interval Definition File

Collection

Activity Planning

Element (CAPE)

Status Requests

Cancellation Requests

Calibration Scene Request

Mission

Operations

Element (MOE)

Acquisition Data

U.S. Naval

Observatory

(USNO)

User Community L1 / L0Rp Products

Metadata Search Results

CPF / BPF / RLUT / GCP

Cal / Val Team

CPF Edits

Characterization Data

Cal / Val Products

Calibration Scene Requests

Cal / Val Reports

Browse / QB

LDCM Ground System Element Cal / Val External Entity

International

Cooperators

(ICs)

Collection Requests

IC Mission Data

Administrative Messages

Problem Reports

Instrument

Vendors

(OLI / TIRS) L0Ra, L0Rp Data

Pre-Launch Instrument

Data (OLI / TIRS)

Metadata

Search / Request

Collection Request Status

Data

Processing and

Archive

System

(DPAS)

Pole Wander

UTC/UT1

Scene Transmit Schedule

Mission Data Available & Location

User Request Types & Parms

Status Updates

Cancellation Confirmation

Request IDs

Station Description

Conflict Notices

Cal/Val Reports

Acquisition Data

Contact Forecast Visibility

Algorithms

Production Software

Documentation

Scene Transmit Schedule

Confirmed Contact Schedule

Post Pass Reports

L1 Products

Subscriptions

Validation Reports

Cal / Val Product Status

Validation Reports

Scene Metatdata

Orders

Mission Data, Metadata & Location

Search Requests

Subscriptions

Order info

Communication Products

Product Type

Mission Data Record Update

Mission Data Move

Mission Data

Figure 3-3. DPAS Context Diagram

- 11 - LDCM-OCD-007

3.1 Subsystem Descriptions

The DPAS is composed of eight subsystems, as shown in Figure 3-2. A description, block diagram, and context diagram for each Subsystem follow.

3.1.1 Storage and Archive Subsystem (SA)

Storage and Archive (SA) provides shared storage for product distribution and DPAS use, and provides an archive capability for long-term preservation of LDCM data.

SA is designed to be highly automated and requires limited operational support. The primary operational users of the SA are the DPAS Subsystems. External customers do not access SA directly, but receive data via the UP. Engineers need access to the SA for development, analysis, and test activities. System Administrators and Operational support staff require access to the SA for routine administration and monitoring tasks.

The two components within SA are Storage and Archive.

The Storage component provides shared disk storage for the LDCM. It includes an Online Cache and an Internal Cache. The Online Cache hosts data that are available to customers via Web-enabled access. The types of data using the Online Cache include standard products (L1T, L0Rp), FRB, and 8-bit QB. The Internal Cache provides storage space for the DPAS Subsystems. The Internal Cache contains a large-capacity cache for long-term storage of L0Ra data, and a high-performance Processing Cache used for Ingest and LPGS processing. The Processing Cache holds RLUT and DEM data and temporary working files. The Online Cache and Internal Cache consist of high-capacity shared online storage.

The Archive component includes the long-term archive of data collected and generated by the LDCM, including Mission Data, BAD, FRB, 8-bit QB, and milestone backups, documentation, software and test data. The Archive component includes backup copies of data to facilitate recovery from operational data loss. The Archive includes a nearline storage library with an online data cache. This Archive employs robotic media handling, high-density media, offline storage, and Hierarchical Storage Management (HSM) software to manage online, nearline (onsite), and offline (offsite) data.

The Archive component provides the capability to generate offline media for storage at an offsite archive facility for long-term data preservation. The Offsite Archive includes Mission Data, BAD, FRB, 8-bit QB, documentation, software, and milestone backups.

The primary purpose of the offsite archive is to safeguard the data for Disaster Recovery / Continuity of Operations. This site adheres to National Archive and Records Administration (NARA) standards to ensure the safety of the archived data.

- 12 - LDCM-OCD-007

Figure 3-4 is the block diagram for the SA, which shows the components and major data flows.

Figure 3-4. SA Block Diagram

- 13 - LDCM-OCD-007

Figure 3-5 is the context diagram for the SA, which shows a complete listing of its external interfaces.

Figure 3-5. SA Context Diagram

3.1.2 Inventory Subsystem (INV)

The LDCM Inventory Subsystem performs two key database functions for the DPAS: a master mission database (LDCM Mission Database (LMD)), and a Searchable Inventory Database (SID).

The LMD component maintains the master copy of mission metadata including Mission Data contents, archive locations for full and reduced resolution browse of the reflective, thermal and quality images, and version information for processing systems. The LMD is designed to support long-term archiving, isolated for security, and a protected resource.

The SID component contains a subset of the LMD information and is highly optimized to support fast product search and ordering. To support data distribution, storage location information within the SID supports enterprise systems like the Tracking, Routing, and Metrics (TRAM) to order creation of products. It also contains location data for 8-bit QB and FRB supporting fast downloads. In the DPAS, the primary user of the SID is the EarthExplorer (EE) Enterprise component of the UP. Operations staff and the Data Manager interact with the Inventory Subsystem via the Operator User Interface (OUI) component.

- 14 - LDCM-OCD-007

Figure 3-6 is the block diagram for the Inventory Subsystem.

Figure 3-6. Inventory Block Diagram

- 15 - LDCM-OCD-007

Figure 3-7 is the context diagram for the Inventory Subsystem, which shows a complete listing of its external interfaces.

Figure 3-7. Inventory Context Diagram

3.1.3 Ingest Subsystem (IS)

The primary work of the Ingest Subsystem is to format Mission Data retrieved by the GNE into L0Ra data. Along with this formatting, selected data types are analyzed for impulse noise and saturated pixels, data gaps are filled, duplicate lines are eliminated, ancillary data with suspicious timestamps are flagged, metadata are generated, and processing metrics are collected.

Inputs:

o Mission Data intervals o Calibration Parameters (CPs) for alignment and scene framing o RLUT

Processing:

o Perform initial quality checks (e.g., Cyclic Redundancy Check (CRC), line and frame headers) o Flag suspect timestamps in ancillary data o Create interval and scene metadata o Format the data for efficient processing

- 16 - LDCM-OCD-007

o Extract and/or calculate sensor, spacecraft, and image statistics

Outputs:

o L0Ra intervals and metadata o Standardized characterization data to IAS DB o Bias Parameters (BPs) for product generation

Figure 3-8 is the block diagram for the Ingest Subsystem, showing the components and major data flows.

MDS

IAS

Database

Schema

Level-0Ra

Generator

[LG]

MAC

Controller

[MC*]

Ancillary

Data

Corrector

[ADC]

Metadata

Generator

[MG]

Scene

Framing

Component

[SFC]

MAC

Inventory

Loader

[MIL]

SA

Internal

Cache

SA

Archive Cache

IAS Data

Recorder

[IDR]

Shutter

Histogram

Char

[SHC]

L0Ra

Bias

Parameter

Generator

[BPG]

Trending Data

Characterization

Data

Inventory

L0Ra, L0Ra Metadata, Ancillary

Metrics

L0Ra Metadata

L0Ra Location

GNE

DCRS

Mission Data

Location &

Mission Data

Available Notifications

SA

Internal Cache

L0Ra

Operator

User

Interface

(OUI)

IAS

Services

CPF/RLUT

Name Mission

Data

Operations

Staff

L0Ra ancillary

Ingest DB schema

MAC Job

Processor

[MJP]

SA

Processing Cache

Temp and log files

Characterization

Data

Bias

Parameters

Ancillary temp file

Ancillary temp file, CPF

L0Ra Ancillary

L0Ra Metadata

L0Ra Metadata

L0Ra

L0Ra

Mission

Data start

Shutter

& Deep

Space

Shutter

& Deep

Space

Mission Data Processing Status

CPF/RLUT

Request

CPF

SA

Processing Cache

CPF

CPF

RLUT

CPF

CPF

L0Ra Ancillary

* Not Shown: every component takes the Job Parameter File (JPF) as a command line parameter, which is written to the SA Processing Cache by MC

Figure 3-8. Ingest Block Diagram

- 17 - LDCM-OCD-007

Figure 3-9 is the context diagram for the Ingest Subsystem, which shows a complete listing of its external interfaces.

Figure 3-9. Ingest Context Diagram

3.1.4 Subsetter Subsystem (SBS)

The Subsetter Subsystem extracts L0Rp data from L0Ra data. Requests are received from calling systems via a database entry. The Subsetter Controller monitors the database for new requests and initiates the OLI / TIRS Subsetter for each request.

L0Rp data (L0Ra if the request is for an interval) are staged to the Internal Cache and the requestor is notified of processing completion (L0Rp status) and the location of the data.

Figure 3-10 is the block diagram for the Subsetter Subsystem, which shows the components and major data flows.

- 18 - LDCM-OCD-007

Figure 3-10. Subsetter Block Diagram

- 19 - LDCM-OCD-007

Figure 3-11 is the context diagram for the Subsetter Subsystem, which shows a complete listing of its external interfaces.

Figure 3-11. Subsetter Context Diagram

3.1.5 Level 1 Product Generation Subsystem (LPGS)

The LPGS generates L1 data products. The LPGS uses the same algorithms to generate L1 products that the IAS uses within its L1 processor. This allows the LPGS to provide identically calculated characterization data to the IAS for trending and analysis.

- 20 - LDCM-OCD-007

Figure 3-12 is the block diagram for the LPGS, which shows the components and major

Figure 3-12. LPGS Block Diagram

- 21 - LDCM-OCD-007

Figure 3-13 is the context diagram for the LPGS, which shows a complete listing of its

Figure 3-13. LPGS Context Diagram

3.1.6 Image Assessment Subsystem (IAS)

The IAS is responsible for the offline assessment of image quality to ensure compliance with the radiometric and geometric requirements of the spacecraft and the sensors throughout the life of their respective missions. Operational activities occur at the USGS Earth Resources Observation and Science (EROS) Center; less frequent assessments and calibration certifications occur at the Landsat Project Science Office, which is located at the Goddard Space Flight Center (GSFC) in Greenbelt, Maryland.

The IAS characterizes radiometric and geometric artifacts through a series of algorithms. The outputs of the algorithms and their statistics are captured in a relational database for trending, analysis, modeling, and calibration. The IAS processes up to 40 scenes per day for image quality assessment, radiometric and geometric calibrations and characterizations, and artifact correction.

- 22 - LDCM-OCD-007

Figure 3-14 is the block diagram for the IAS, which shows the components and major

Figure 3-14. IAS Block Diagram

- 23 - LDCM-OCD-007

Figure 3-15 is the context diagram for the IAS, which shows a complete listing of its

Figure 3-15. IAS Context Diagram

3.1.7 User Portal (UP) Subsystem

The UP provides multiple access points to the user community, Cal/Val analysts, and support staff. By providing a variety of access mechanisms, both internal and external data consumers have efficient means to discover LDCM data, obtain information, metadata and browse, request and receive products, and submit feedback. To accomplish such operations, the UP has been functionally decomposed into six components: Project Web Site, Data Access Tool (Earth Explorer), Order (Tracking, Routing, and Metrics system), Registration, UP Services, and International Cooperators exchange and validation cache (Exchange Cache).

The Project Web Site and Data Access Tool are internal to the DPAS architecture and provide a mechanism for users to request information and products. Clients external to DPAS can request LDCM metadata, browse, and standard products using Open Geospatial Consortium (OGC) and Web services provided by the Services component.

Registration and Services components serve both internal and external requests, while the Order component only supports requests initiated from EE. The Exchange Cache

- 24 - LDCM-OCD-007

component is a temporary storage area for pickup and delivery of IC validation and exchange data.

Figure 3-16 shows the components and major data flows of the UP.

Figure 3-16. User Portal Block Diagram

- 25 - LDCM-OCD-007

Figure 3-17 shows external interfaces to the UP.

Figure 3-17. User Portal Context Diagram

3.1.7.1 Earth Explorer (EE)

EE is a Web portal into the vast archive of land remote sensing data available from the USGS EROS archive. The archive includes over four million satellite images (1960 to present) and over eight million aerial photographs (1939 to present). EE supports both registered users and the public with access to over thirty data collections. Online help is available, and customers can search the archive using geographic coordinates, place names, zip codes, or unique parameters exclusive to each collection. EE provides cross-inventory searches of these collections. Most of these collections include online browse references. It also provides and supports various services, such as the generic USGS shopping basket and user registration functionality. The Long Term Archive (LTA) Project manages the EE as an enterprise solution.

3.1.7.2 Tracking, Routing, and Metrics (TRAM)

Tracking, Routing, and Metrics (TRAM) component provides a Web service based interface for accepting orders from multiple clients and routing them to the correct production system. The TRAM provides status on orders, manages demographics

- 26 - LDCM-OCD-007

information derived from orders, and gathers information passing through the system for management reports. The LTA Project manages the TRAM.

- 27 - LDCM-OCD-007

3.1.8 Mission Management Office (MMO) Database Subsystem (MDS)

To support Landsat Project operations, the MDS provides mission status information that the Landsat Project requires for managing and directing Project activities. This information includes statistics to track product generation and data distribution. In addition, the Landsat Project management and contractor staff use this information to manage the Project and to track its status. Fundamental to MDS processing is Extract, Transform, Load (ETL), whereby data from different subsystem databases are obtained and organized for analysis.

Figure 3-18 shows MDS components and major data flows.

Figure 3-18. MDS Block Diagram

- 28 - LDCM-OCD-007

Figure 3-19 shows the MDS external interfaces.

Figure 3-19. MDS Context Diagram

- 29 - LDCM-OCD-007

3.2 Priority Processing

The DPAS supports a priority processing mechanism that enables jobs to be processed in a fashion that supports day-to-day concurrent processing of high priority products and the processing of backlogs from system outage events. Priority product generation supports both emergency response and Cal/Val requests, while priority ingest processing supports priority collection requests and the processing of data input backlogs. Figure 3-20 illustrates the priority processing mechanism of the DPAS.

Figure 3-20. Priority Processing

GNE (DCRS), UP, LPGS, IAS, Ingest, and Subsetter Subsystems support priority processing. The GNE / DCRS and the UP assign priorities to processing jobs based on priority collections or priority product requests. The priority values are from 1 to 9, with 1 being the highest priority. The Ingest, Subsetter, LPGS and IAS Subsystems each contain a processing queue. Jobs of equal priority in each queue are processed in oldest to newest order. At any time prior to actual processing, an Operator may change the priority of a job via the OUI of the Ingest or LPGS Subsystem. In this manner, both the day-to-day concurrent processing needs as well as priority processing needs are accommodated. (See subsection 4.1.2 for additional information on concurrent processing activities of the DPAS.)

- 30 - LDCM-OCD-007

Section 4 DPAS Operations

4.1 Data Processing, Archive, and Distribution Operations

On a daily basis, the LDCM data processing and archive facility at USGS EROS ingests and processes at least 400 collected image scenes to L1 products. Mission Data are archived and L0Rp and L1 products are made available for download by users. Data processing, archive, and distribution operations are conducted in parallel with the current Landsat operations capability whenever possible.

4.1.1 Operations Facility Layout

DPAS Operations exists within an expanded operations area at EROS. This expanded area contains both the LDCM (L8) and Landsat (L5/L7) operations. Figure 4-1 illustrates the conceptual layout of the operations area.

Figure 4-1. Operations Facility Layout

As shown in Figure 4-1, DPAS Operations are conducted in the room 1509 area of EROS. A series of workstations with combinations of single and dual head displays are used to perform the day-to-day operational activities. Landsat operations are conducted in the room 1508 area of EROS. Close proximity of the DPAS and Landsat operations teams ensures good communication and coordination.

4.1.2 Day In The Life (DITL) Overview

It is anticipated that DPAS Operations staff will be employed on a shift schedule of 19 hours per day during the work week and 10 hours per day on the weekend. Because DPAS Operations are to be conducted 24x7, system automation is required in the

- 31 - LDCM-OCD-007

DPAS design. DPAS Operations is primarily an oversight and system monitoring activity. Figure 4-2 presents a Day-In-The-Life (DITL) timeline for DPAS Operations, listing the operational activities of the DPAS Operations staff and the DPAS Subsystems.

Figure 4-2. Day-In-The-Life (DITL) Timeline

DPAS Operations start, stop, and monitor the Subsystems, as well as monitor storage cache usage and operational statistics to ensure that the required DPAS throughput, latency, and availability requirements are met on a daily basis. In addition, DPAS Operations monitors application OUIs and processing log files for errors for both attended and unattended times. Finally, select DPAS Operational staff is on-call during unattended hours in the event of a system failure, which would be noted and reported by System Administrators or Data Processing and Capture Facility staff. In all aspects, DPAS Operations communicates status via a daily morning status meeting each week day with information passed to the MMO.

- 32 - LDCM-OCD-007

4.2 Operational Roles

4.2.1 DPAS Operators

The DPAS Operators monitor all DPAS activities, including the jobs being processed and the status of the Subsystems. The DPAS Operators are provided with the functionality to start, stop, suspend, and monitor processing. If any problems arise with the Subsystems, the DPAS Operators notify the appropriate support staff. If any quality issues are identified with the products, a Cal/Val Analyst is notified.

4.2.2 Data Management Personnel

Data Management personnel are responsible for monitoring the inventory, reconciling metadata, and ensuring the inventory reflects archived holdings. Data Managers follow accepted procedures and practices for LDCM Configuration Management (CM).

4.2.3 Software Developers

Software Developers perform routine and emergency bug fixes, upgrades, and additional release support. Software Developers also build new software to accommodate change requests and software enhancements. Software Developers follow accepted procedures and practices for CM.

4.2.4 System Engineers

System Engineers perform failure detection and correction, and plan new hardware and software to accommodate change requests. System Engineers also develop procedures and practices for CM and formal system integration and test.

4.2.5 System Administrators

System Administrators perform system and capacity monitoring, failure detection and correction, operating system support, operating system upgrades, backup configurations, and support system security initiatives and testing.

4.2.6 Hardware Engineers

Hardware Engineers are responsible for all DPAS hardware, networking, capacity and upgrade planning, failure detection and correction, and equipment setup and deployment. Hardware Engineers also coordinate new hardware and software to accommodate change requests. Hardware Engineers follow accepted procedures and practices for procurement and system integration.

4.2.7 Database Administrators

Database Administrators perform database capacity and performance monitoring, failure detection and correction, database upgrades, backup configurations, and support database security initiatives and testing.

4.2.8 Calibration/Validation (Cal/Val) Analysts

The Cal/Val Analysts’ primary function is to ensure the radiometric, geometric, and spatial quality of remotely sensed imagery. The Cal/Val Analyst uses the IAS tools to

- 33 - LDCM-OCD-007

evaluate these qualities, and to test any calibration parameter updates before they are baselined. Once algorithms and code are tested and ready for production use, they are assigned to a software release. The algorithms and code are copied to a test environment that mimics the production environment and are retested before being formally released.

4.2.9 Data Quality Analyst (DQA)

The Data Quality Analyst (DQA) is a Point of Contact (POC) internal to the DPAS who interacts with ICs for Data Validation & Exchange (DV&E) Operations. The DQA performs the validation process and communicates results with the ICs. In addition, the DQA facilitates the exchange of Mission Data with ICs.

4.2.10 International Cooperators (IC)

ICs receive data downlinks from the observatory and may submit data collection requests through the UP. ICs routinely provide metadata to the DPAS and participate in periodic data validation activities. ICs also exchange Mission Data with the USGS, upon request by either party.

4.2.11 Mission Management Office (MMO)

The MMO is the USGS entity responsible for the day-to-day operations of the DPAS.

The MMO interfaces with the FOT as part of the overall Landsat operations. The MMO is responsible for overall data management, DPAS Operations, and data collection reconciliation. The MMO is the primary user of the MDS, which gathers collection and other pertinent metrics from the DPAS for standard and ad-hoc reporting in support of mission management.

4.2.12 Configuration Management (CM)

Configuration Management (CM) personnel are responsible for the day-to-day CM activities of the Landsat operations. CM controls changes of software and hardware, performs official software and document releases, and tracks artifact changes. CM staff interfaces with several operations personnel including Operators, Data Management, Systems Engineers, Software Engineers, Hardware Engineers, Database Administrators, and the Mission Management Office Information Officer (MMOIO).

- 34 - LDCM-OCD-007

Section 5 Scenarios

5.1 Scenario Conventions

Figure 5-1 illustrates conventions used in the scenario diagrams of this section. Various line colors, styles, and weights distinguish the data and communication flows between the DPAS Subsystems and external entities, as shown in the scenario key. Arrow directions indicate data flow, and grey boxes designate Enterprise components and the use of callouts. Sequence blocks indicate the order of events within a scenario.

Figure 5-1. DPAS Scenario Conventions

- 35 - LDCM-OCD-007

5.2 Standard Processing

The fundamental responsibilities of the DPAS are data processing, storage, long-term archiving, and data and product distribution for the LDCM. Central to these responsibilities is the concept of standard processing. Standard processing consists of three key activities: Ingest Mission Data, Standard Product Generation, and Browse Generation. Figure 5-2 depicts the high-level flows that form these three activities.

Each of these activities is shown in detail in the subsequent sections.

Figure 5-2. Standard Processing Data Flows

- 36 - LDCM-OCD-007

5.2.1 Ingest Mission Data

A fundamental responsibility of the DPAS is to format collected Mission Data into L0Ra data. Along with this formatting, the data are analyzed for impulse noise and saturated pixels, dropped lines are filled, duplicate lines are eliminated, ancillary data with suspicious timestamps are flagged, metadata is generated, and processing metrics are collected (see Figure 5-3).

Figure 5-3. Ingest Mission Data

Step Description of Figure 5-3

1 At any time, Operations may monitor Ingest Subsystem processing using the OUI. When processing anomalies are detected, Operations consults their procedures to resolve them (see subsection 5.10.1).

2 When all files related to an interval (sensor on to sensor off) are received, as defined in the Scene to Interval File Mapping Table (SIFMT), DCRS groups the files into a Mission Data interval directory, creates an IDF, and places the data and the IDF in a temporary work area on the Archive Cache.

3 The DCRS updates the STATUS field in the Subsystem Interface table of the DCRS database, indicating that an interval of Mission Data is available for processing. The Ingest Subsystem monitors the STATUS field.

4 The Ingest Subsystem updates the STATUS field in the Subsystem Interface table of the DCRS database, indicating the Mission Data is being processed.

5 The Ingest Subsystem retrieves the appropriate CPF and RLUT name from the IAS and pulls the CPF and RLUT files from the specified locations.

6 The Ingest Subsystem reads the Mission Data and IDF from the temporary work area of the Archive Cache.

7 The Ingest Subsystem writes the L0Ra data to the Internal Cache.

- 37 - LDCM-OCD-007

Step Description of Figure 5-3

8 The Ingest Subsystem writes the interval metadata, scene metadata, and location of the L0Ra data to the LMD.