Project_Slides.pdf

PDF 9 MB Posted May 2, 2018

Attached to: Patent Application Federal contract opportunity
Solicitation number: 80NSSC18Q0542
Issued by: National Aeronautics and Space Administration Shared Services Center

About this file

From the source

Please see the project slides.

View the file

Other files for this federal contract opportunity

Other files attached to Patent Application, newest first.
File	Type	Posted
RFQ_Open_Market_Template.docx	DOCX document	May 2, 2018
Schnase_CMAC2_Proposal.pdf	PDF	May 2, 2018
NASA_Form_1679_(4).pdf	PDF	Apr 27, 2018
NASA_Form_1679_(6).pdf	PDF	Apr 27, 2018
NASA_Form_1679.pdf	PDF	Apr 27, 2018
NASA_Form_1679_(3).pdf	PDF	Apr 27, 2018
NASA_Form_1679_(2).pdf	PDF	Apr 27, 2018
NASA_Form_1679_(5).pdf	PDF	Apr 27, 2018
Questions.docx	DOCX document	Apr 27, 2018
RFQ_Open_Market_Template.docx	DOCX document	Apr 27, 2018
RFQ_Open_Market_Template.docx	DOCX document	Apr 27, 2018
SATPC0009410_Tab_12_RFQ_Open_Market_Template.docx	DOCX document	Apr 27, 2018
Questions.docx	DOCX document	Apr 27, 2018
RFQ_Open_Market_Template.docx	DOCX document	Apr 27, 2018

Show all 14

On GovTribe

Work with this file on GovTribe

Download the original file
Contacts named in this file
Similar government files
Ask GovTribe AI about this file

Try for free Login

Text version

The file's text, extracted by GovTribe without its formatting.

CMAC-14 Project Status Review Extending Climate Analytics as a Service to the Earth System Grid Federation

14-CMAC14-0016

Investigator Team:

John Schnase (PI), Dan Duffy (Co-I), Glenn Tamkin (Co-I)

Savannah Strong, Jian Li, Mark Carroll, and Roger Gill Office of Computational and Information Sciences and Technology

NASA Goddard Space Flight Center

September 19, 2017

Project Description

• This project addresses CMAC Topic Area 2.1 – Our goal is to advance capabilities for data management and data analytics associated with ESGF, CMIP, and related modeling activities

– Builds on prior CMAC-funded work we have done to develop Climate Analytics as a Service (CAaaS)

– Extends existing CAaaS reference models, hardware and software technologies, analytic algorithms and operations, the web service API, the client distribution package including the CDSlib Python library and workflow applications bundle , data sets, customers, and science applications

• Three major science objectives:

1. Improve data availability – by creating a reanalysis ensemble service built on the technologies and principals behind MERRA Analytic Services

2. Improve analytic capabilities – by building a high-performance, near-real-time extension to the new reanalysis ensemble service

3. Improve service interoperability – by extending our web service interface and client Python libraries to support ESGF’s new WPS web service specification

9/19/172

CMAC-14 Project Status Review 9/19/173

1. Reanalysis Ensemble Service (RES)

– RES will support data from five major reanalysis collections, a core suite of commonly-used ensemble operations, and accessibility through a RESTful web services API and a client CDSlib Python library (formerly referred to as the CDS API)

2. Virtual Real-Time Analytics Testbed (vRAT)

– vRAT will support a select set of climate variables from five major reanalysis data sets in a near-real-time HPC environment supporting SQL querying and SciHadoop-based MapReduce analytics over native NetCDF files

3. ESGF/WPS-Compliant Web Service

– The ESGF/WPS-compliant interface will enable greater interoperabilty between our climate data analytics services and next-generation ESGF capabilities. The CDSlib Python client libraries will be extended to accommodate these new capabilities

Project Description

• RES Development Strategy and Schedule

9/19/174

✔ Checkpoint 1 (TRL 7/1) Finished transition of MERRA/AS to MERRA2/AS (TRL 7), finished basic operators (area and vertical avg), developed RES (TRL 1) reference model and design spec, updated/extended CDSlib, developed QA testing protocols.

✔ Checkpoint 2 (TRL 4) <= Today Added ECMWF, CFSR, ESRL, JR55 reanalyses to RES, added ensemble operators, created UQP spec, integrated NASA Wrangler observational data service, updated/extended CDSlib, QA, completed standalone prototype implementation and testing of all components and subsystems.

✔ Checkpoint 3 (TRL 5/6) Completed component integration and test, science/application verification/validation, fully demonstrated engineering/science RES and CDSlib operational feasibility.

✔ Checkpoint 4 (TRL 7/8) <= Today End of system development, end of CDSlib development, documentation complete, RES fully integrated into a beta test environment and in use by customers …

Summary of R&D Activities & Milestones

• Overview

– Last Checkpoint Summary

– Accomplishments Since Last Checkpoint

– Ensemble Analysis / Uncertainty Quantification Rationale

• System / Interface Architecture

– Web service / Web API

– CDSlib Basic and Extended Utilities

– CDSlib UQP Extended Utilities

• Year 2 Current Status

– Use Case 1 – RES w/ Observational Data Integration

– Use Case 2 – Ensemble reanalysis w/ Uncertainty Quantification Package (UQP)

– Hardened RES, CDSlib Python Library, Jupyter Notebook

– RES Platform-as-a-Service (PaaS) (Docker deployment capability)

– GEOS-5 Forward Processing (FP) data analytics service prototype

– Automated multi-source RES data provisioning

• Science Applications, Partnerships, and Related Projects

– GMAO, ESGF, CREATE, MetOffice

– NASA ASP, RECOVER, ABoVE, CyVerse

– Strategic Partnership Office (SPO)

– Intercontinental Exchange Corporation (ICE), Accuweather, Scoville Risk Partners

– Conservation International (involves 3 NASA programs)

– ESDIS CAaaS-to-Cloud Project

• Technology transfer, licensing, CRADA development

• Next Steps

9/19/175

Reanalysis Ensemble Service Outline

Last Checkpoint Summary

• Participating Reanalyses

– Modern-Era Retrospective Analysis for Research and Applications (MERRA-2)

– ECMWF Interim Reanalysis (ERA-Interim)

– NOAA NCEP Climate Forecast System Reanalysis (CFSR)

– NOAA ESRL 20th Century Reanalysis (20CR)

– Japanese 50-Year Reanalysis (JRA-55)

• Service-Side Analytic Operations (Microservices, includes our “canonical ops” ...)

– Spatiotemporal subsetting, descriptive statistics, and arithmetic ops over single and multiple collections

– Completed operator implementation w/ area and vertical averaging, online standard deviation method

• Client-Side CDSlib Python Library Utilities (Our “convenience functions” ...)

– Basic Utilities (One-to-one microservice calls to RES’s Web API)

• Spatiotemporal subsetting, descriptive statistics, and arithmetic ops over single and multiple collections

– Extended Utilities (Scripts/workflows that call Basic Utilities or other internal or external Web services)

• Climatologies, trends, anomalies, normals, custom workflows, and UQPs over single and multiple collections

• Aggregate ChkPt 2 RES / CDSlib Products * – TRL 4 Standalone prototyping and component validation in laboratory environment

1) Single source and ensembled analytic results – Computationally modeled avg, std, max, min + arithmetic ops

2) Single source and ensembled higher-order data products – Climatologies, trends, anomalies, normals, workflows

3) Uncertainty Quantification Packages (UQPs) – RES products + GPCP observational data via external NASA Wranger service

4) Prototype CDSlib Jupyter notebook – Including documentation, use cases, personal project management

* The primary objective of this work is to enable a new capability for the climate research and applications communities – a service where high-performance computing allows the most commonly used reanalysis data products to be created dynamically from petabyte-scale data collections (1,2) and enables the effective interpretation and use of these products (3,4) …

9/19/176

Ensemble Analysis / Uncertainty Quantification

• First Principles ...

– The Reanalysis Ensemble Service enables four major capabilities:

1) Convenient “one-stop” access to participating reanalyses

2) Ensembled, error- and bias-smoothed multi-reanalysis products

3) A way to compare RES products with other reanalysis products

4) A way to compare RES products with observational products

– Making 3) and 4) easier to do is an important feature of the service ...

• Definition

– Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known.

• Basic idea

– Provide an Uncertainty Quantification Package (UQP) that contains requested RES results along with observational data and other products that can be used to characterize uncertainty in the RES results.

9/19/177

CMAC-14 Project Status Review 9/19/178

System / Interface Architecture

CMAC-14 Project Status Review 9/19/179

System / Interface Architecture

Web service / Web API OAIS Archive Data Flow Interactions

Web API http://<base_URL>/(ingest.php?service=<service_name>&request=<operation>&p arameters=<parameters> http://<base_URL>/(query.php?service=<service_name>&request=<operation>&p arameters=<parameters> http://<base_URL>/(order.php?service=<service_name>&request=<operation>&p arameters=<parameters> http://<base_URL>/(download.php?service=<service_name>&request=<operation >&parameters=<parameters> http://<base_URL>/(execute.php?service=<service_name>&request=<operation> &parameters=<parameters> http://<base_URL>/(status.php?service=<service_name>&request=<operation>&p arameters=<parameters>

• Client applications can make direct calls to RES microservices via the REST web service interface.

• These microservices are atomic, run in parallel, and include our “canonical ops” (max, min, avg, std, cnt, sum, diff).

• Web service syntax follows Open Archive Information System (OAIS) standard.

• Treats RES as a dynamic archive that creates ”realizable” products in near real time.

CMAC-14 Project Status Review 9/19/1710

CDSlib Basic Utilities

• Client applications can also bind to CDSlib’s Python libraries and use Basic Utilities to call RES microservice.

• Basic Utilities abstract in/outbound web service traffic into easy-to-use Python functions.

• They have a one-to-one correspondence to the underlying web service calls, so these are also atomic microservice calls that can run in parallel.

• Microservices can run in parallel, microservices are themselves parallel Hadoop operations, and CDSlib is itself an RES client …

Reanalysis Ensemble Service System / Interface Architecture

Canonical	Operations	(Microservices)
Our	“canonical	operations”	— or	microservices— take	as	input	(1)	a	variable	name,
(2)	a	spatial	extent,	and	(3)	a	temporal	extent

Then	(1)	perform	arithmetic	operations,	(2)	extract	spatiotemporal	subsets,
and	(3)	model	data	by	computing	the	descriptive	statistics	of	limits,	central
tendency,	and	dispersion in	the	spatial	and	temporal	domains,	e.g.:

result f average(var, (t0,t1), ((x0,y0,z0),(x1,y1,z1))),

Canonical	ops	can	be	combined	under	programmatic	control	to	create	higher-
order	products	such	as	climatologies,	trends,	anomalies,	and	tailored	workflows

Basic	Utilities
ingest(service_name,	operation,	parameters)
query(service_name,	operation,	parameters)
order(service_name,	operation,	parameters)
download(service_name,	operation,	parameters)
execute(service_name,	operation,	parameters)
status(service_name,	operation,	parameters)

CMAC-14 Project Status Review 9/19/1711

Extended	Utilities
AVERAGE(MAS,	Variable_Name,	(t0,	t1),	((x0,	y0,	z0),

(x1,	y1,	z1)),	Result)
order(MAS,	GetVarByOpTeSe(Variable_Name,	Average,

(t0, t1), ((x0, y0, z0), (x1, y1, z1)), ID) Repeat

status(MAS,	ID)
until	finished

download(MAS,

ID,

Result)

STD(

MAX(

…

MIN(

...

CLIMATOLOGY

(

...

ANOMALY(

...

TREND(

...

etc.

UQP_UC1

(

...

UQP_UC2(

...

UQP_Bosilovich(

...

etc.

CDSlib Extended Utilities

• Clients can also use CDSlib’s Extended Utilities that call on Basic Utilities, the Web API, or other external web services to create higher-order products.

• Extended Utility functions are basically scripted workflows. They use program controls for microservice orchestration.

• CDSlib Extended Utilities also includes functions corresponding to our “canonical ops.”

• We are designing, prototyping, and testing UQP capabilities as Extended Utilities ...

Reanalysis Ensemble Service System / Interface Architecture

Basic	Utilities
ingest(service_name,	operation,	parameters)
query(service_name,	operation,	parameters)
order(service_name,	operation,	parameters)
download(service_name,	operation,	parameters)
execute(service_name,	operation,	parameters)
status(service_name,	operation,	parameters)

CMAC-14 Project Status Review 9/19/1712

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

• Use Case 1 demonstrates RES integration of observational data to produce a simple Uncertainty Quantification Package (UQP):

Compute the average summer-time (JJA) precipitation rate over the Continental US in 1984 using MERRA2. Compare the results to precipitation data obtained from the Global Precipitation Climatology Project (GPCP) data service.

• Use Case 1 is implemented as a client-side CDSlib Extended Utility that calls on CDSlib’s Basic Utilities and an external observational data service.

#	RES/UQP	Use	Case	1	– Extended	Utility	Function	(RES	w/	obs	integration)
#	Average	summer-time	(JJA)	precipitation	rate	over	the	Continental	US
#	in	1984	[MERRA2	(reanalysis),	GPCP	(observation)]

#	Spatiotemporal	parameters
uqp_parms['time_span']		=	"time_span=19840601-19840831"
uqp_parms['time_interval']	=	"time_interval=fullinterval(fullspan)"
uqp_parms['spatial_extent']	=	"spatial_extent=-125,24,-66,50”
uqp_parms['spatial_resolution']	=	"spatial_resolution=native"
uqp_parms['vertical_extent']	=	"vertical_extent=fullextent”

#	Precipitation	parameters
uqp_parms[‘collection’]		=	"tavg1_2d_lnd_Nx”
uqp_parms[‘variable_list’]	=	["PRECTOTLAND”]
uqp_parms[‘obs_data’]	=	“GPCP”

#	CDSlib	Basic	Utility	invocations
cds_lib.avg("MAS",	uqp_parms,	"./output/”) #	Compute	MERRA2	avg
cds_lib.std("MAS",	uqp_parms,	"./output/”)	#	Compute	MERRA2	std
cds_lib.obs("MAS",	uqp_parms,	"./output/") #	Get	Wrangler	GPCP	obs

UQP Output Structure (zip file)

• 1 NetCDF	file	containing	average	rate
for	all	grid	points	in	MERRA2

• 1 NetCDF	file	containing	standard
deviation	for	all	grid	points	in	MERRA2

• 3	GeoTIFF	files	containing
corresponding	average	rate	in	GPCP
observational	data	obtained	from	NASA

Wrangler service

NASA Wrangler

CDSlib’s UQP Extended Utilities (UC1)

CMAC-14 Project Status Review 9/19/1713

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

• UQPs contain RES results along with observational data and other products that can be used to characterize uncertainty in the RES results.

• We are using NASA Wrangler to obtain observational data in our initial UQP prototype.

• NASA Wrangler is a Amazon Cloud-based service that rapidly aggregates data from a wide range of data services.

• NASA Wrangler is a core technology in the Applied Sciences Program-funded RECOVER wildfire decision support system.

#	RES/UQP	Use	Case	1	– Extended	Utility	Function	(RES	w/	obs	integration)
#	Average	summer-time	(JJA)	precipitation	rate	over	the	Continental	US
#	in	1984	[MERRA2	(reanalysis),	GPCP	(observation)]

#	Spatiotemporal	parameters
uqp_parms['time_span']		=	"time_span=19840601-19840831"
uqp_parms['time_interval']	=	"time_interval=fullinterval(fullspan)"
uqp_parms['spatial_extent']	=	"spatial_extent=-125,24,-66,50”
uqp_parms['spatial_resolution']	=	"spatial_resolution=native"
uqp_parms['vertical_extent']	=	"vertical_extent=fullextent”

#	Precipitation	parameters
uqp_parms[‘collection’]		=	"tavg1_2d_lnd_Nx”
uqp_parms[‘variable_list’]	=	["PRECTOTLAND”]
uqp_parms[‘obs_data’]	=	“GPCP”

#	CDSlib	Basic	Utility	invocations
cds_lib.avg("MAS",	uqp_parms,	"./output/”) #	Compute	MERRA2	avg
cds_lib.std("MAS",	uqp_parms,	"./output/”)	#	Compute	MERRA2	std
cds_lib.obs("MAS",	uqp_parms,	"./output/") #	Get	Wrangler	GPCP	obs

UQP Output Structure (zip file)

• 1 NetCDF	file	containing	average	rate
for	all	grid	points	in	MERRA2

• 1 NetCDF	file	containing	standard
deviation	for	all	grid	points	in	MERRA2

• 3	GeoTIFF	files	containing
corresponding	average	rate	in	GPCP
observational	data	obtained	from	NASA

Wrangler service

NASA Wrangler

CDSlib’s UQP Extended Utilities (UC1)

CMAC-14 Project Status Review 9/19/1714

Use Case 1

Average	summer-time	(JJA)	precipitation	rate over	the	Continental	US	in	1984
[MERRA2	(reanalysis),	GPCP	(observation)]

UQP Output Structure (zip file)

• 1 NetCDF file containing average for all grid points in MERRA2

• 1 NetCDF file containing standard deviation for all grid points in MERRA2

• 3	GeoTIFF	files	containing	corresponding	average	in	GPCP	observational	data	obtained
from	NASA	Wrangler service

Standard Deviation Average

MERRA2 shows that the average weekly 1984 summertime rainfall for the Washington area was 1.05 inches (s = 2.14 inches). GPCP shows an average weekly rainfall of about 0.42 inches …

Ensemble Analysis / Uncertainty Quantification

CMAC-14 Project Status Review 9/19/1715

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

NASA Wrangler

CDSlib’s UQP Extended Utilities (UC2)

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

• Use Case 2 demonstrates ensemble averaging and the creation of a more complex UQP:

Compute

the

ensemble

average

summer-time

(JJA)

precipitation

rate over

the

globe

1984 using

MERRA2

and

ECMWF.

Provide

corresponding

GPCP data

for

comparison.

• Use Case 2 taken from Bosilovich et al. 2008. Evaluation of Global Precipitation in Reanalyses.

• Conventional approach involved acquiring data from three sources, each with different interfaces, file formats, and coarse spatiotemporal subsetting. Regridding, averaging, and further transformations done on the client side. RES does it better, faster, cheaper …

Reanalysis Ensemble Service

9/19/1716

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture ...

• Reanalysis data stored in RES’s Hadoop Filesystem in triplicate, each file indexed by <var, time> composite key. All spatial information associated with a given variable and time contained within the file.

• Subsetting is done in two steps on variables of interest: (1) Fast, indexed temporal subsetting (2) computed spatial subsetting.

CMAC-14 Project Status Review 9/19/1717

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture …

CMAC-14 Project Status Review 9/19/1718

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture …

CMAC-14 Project Status Review 9/19/1719

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture …

CMAC-14 Project Status Review 9/19/1720

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture …

CMAC-14 Project Status Review 9/19/1721

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture …

• Subsetting is done in two steps on variables of interest: (1) Fast, indexed temporal subsetting (2) computed spatial subsetting.

Over time, climate datasets tend to increase in size due to temporal extension. Fast temporal subsetting helps us …

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

CMAC-14 Project Status Review 9/19/1722

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture, which in turn enables parallelism at three levels:

1) Instruction-level – Spatial subsetting of indexed files can occur in parallel, and partial products are computed in parallel. The Hadoop runtime environment assembles partial products into a complete final product.

Instruction-level Parallelism

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3	GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

CMAC-14 Project Status Review 9/19/1723

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture, which in turn enables parallelism at three levels:

2) Operation-level – Calls to atomic microservices can be executed sequentially or in parallel.

Instruction-level Parallelism Operation-level

Parallelism

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3 GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

CMAC-14 Project Status Review 9/19/1724

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture, which in turn enables parallelism at three levels:

2) Operation-level – Calls to atomic microservices can executed sequentially or in parallel.

Instruction-level Parallelism Operation-level

Parallelism

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3 GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

CMAC-14 Project Status Review 9/19/1725

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture, which in turn enables parallelism at three levels:

2) Operation-level – Calls to atomic microservices can executed sequentially or in parallel.

Instruction-level Parallelism Operation-level

Parallelism

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3 GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

CMAC-14 Project Status Review 9/19/1726

Extended Utilities

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

#	RES/UQP	Use	Case	2	– Extended	Utility	Function	(Obs	data	integration)
#	Ensemble	average	summer-time	(JJA)	precipitation	rate	over	the	globe
#	in	1984	[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

# Spatiotemporal parameters # Precipitation parameters

#	CDSlib	Basic	Utility	invocations
cds_lib.avg(~~~) #	Compute	MERRA2	avg
cds_lib.std(~~~)	#	Compute	MERRA2	std
cds_lib.avg(~~~) #	Compute	ECMWF	avg
cds_lib.std(~~~)	#	Compute	ECMWF	std
cds_lib.obs(~~~) #	Get	Wrangler	GPCP	obs

cds_lib.regrid(~~~)	#	Regrid	ECMWF	avg
cds_lib.regrid(~~~)	#	Regrid	MERRA2	avg
cds_lib.ensemble_avg(~~~) #	Compute	ensemble	avg
cds_lib.ensemble_std(~~~) #	Compute	ensemble	std

NASA Wrangler UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average for	all	grid
points	in	each reanalysis	collection	(MERRA2,

ECMWF)

• 2	NetCDF	file	containing	standard	deviation	for
all	grid	points	in	each	reanalysis	collection
(MERRA2,	ECMWF)

• 1 NetCDF	file	containing	ensemble	average
regridded	to	coarsest	resolution

• 1 NetCDF	file	containing	ensemble	standard
deviation	of	ensemble	regridded	to	coarsest

resolution

• 3 GeoTIFF	files	containing	corresponding
average	in	GPCP	observational	data	obtained
from	NASA	Wrangler	service

Reanalysis Ensemble Service

• The RES provides a faster and more convenient approach to data assembly. Improved efficiency is enabled by RES’s storage architecture, which in turn enables parallelism at three levels:

2) Operation-level – Calls to atomic microservices can executed sequentially or in parallel.

3) Service-level – Calls to external services can also be made in parallel …

Instruction-level Parallelism Operation-level

Parallelism

Service-level Parallelism

CMAC-14 Project Status Review 9/19/1727

Use Case 2

Ensemble	average	summer-time	(JJA)	precipitation	rate over	the	globe in	1984
[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid	points	in	each reanalysis	collection
(MERRA2,	ECMWF)

• 2

NetCDF

file

containing

standard

deviation

for

all

grid

points

each

reanalysis

collection

(MERRA2,

ECMWF)

• 1 NetCDF file containing ensemble average regridded to coarsest resolution

• 1 NetCDF	file	containing	ensemble standard	deviation	of	ensemble	regridded	to
coarsest	resolution

• 3 GeoTIFF	files	containing	corresponding	average	in	GPCP	observational	data
obtained	from	NASA	Wrangler	service

Reanalysis Ensemble Service Ensemble Analysis / Uncertainty Quantification

MERRA2 shows that the average weekly 1984 summertime rainfall for the Washington area was 1.05 inches (s = 2.14 inches). GPCP shows an average weekly rainfall of about 0.42 inches. Caution: Current rendering of the data may represent alternative facts …

CMAC-14 Project Status Review 9/19/1728

Ensemble Analysis / Uncertainty Quantification

Use Case 2

Ensemble	average	summer-time	(JJA)	precipitation	rate over	the	globe in	1984
[MERRA2	+	ECMWF	(reanalysis),	GPCP	(observation)]

UQP Output Structure (zip file)

• 2	NetCDF	files	containing	average	for	all	grid	points	in	each reanalysis	collection
(MERRA2,	ECMWF)

• 2

NetCDF

file

containing

standard

deviation

for

all

grid

points

each

reanalysis

collection

(MERRA2,

ECMWF)

• 1 NetCDF file containing ensemble average regridded to coarsest resolution

• 1 NetCDF	file	containing	ensemble standard	deviation	of	ensemble	regridded	to
coarsest	resolution

• 3 GeoTIFF	files	containing	corresponding	average	in	GPCP	observational	data
obtained	from	NASA	Wrangler	service

Estimated Assembly Times (secs) UQP Contents Manual Sequential Parallel

MERRA2 std 35.12 – MERRA2 avg Hours 40.12 – ECMWF std to 40.12 – ECMWF avg days … 40.06 – MERRA2 regrid 11.84 – ECMWF regrid 10.95 – Ensemble avg 0.11 – Ensemble std – –

GPCP – 7.22 –

185.54 52.35

CMAC-14 Project Status Review 9/19/1729

Ensemble Analysis / Uncertainty Quantification

CFSR ECMWF

JRA55 MERRA2

ENSEMBLE

CFSR ECMWF

JRA55 MERRA2

Multiple reanalyses, climatological averages and anomalies of global precipitation (2000– 2009) use case … The climatological (10-year) pattern of global precipitation is calculated for each reanalysis (top left), and difference against ensemble mean is evaluated (bottom left).

Microservices involved:

• Computing averages for all grid points in each reanalysis collection (MERRA2, CFSR, ECMWF, JRA55)

• Regridding all climate average to coarsest resolution

• Computing ensemble average of reanalysis collections

• Computing anomalies in each reanalysis collection against ensemble average

Estimated Assembly Times (secs)

Microservices Manual Parallel

Climatology (4 reanalysis) avg 50.25 Regridding (4 reanalysis) regrid hours … 15.23 Ensemble avg 0.12 Anomaly diff 0.27

Total time = 66.4 secs

CMAC-14 Project Status Review 30

CDSlib Utility Status New Utility Description(Long Name)

Basic	Utilities	(1:1	mapping	with	W/S)
ingest() Submit/register	a	Submission	Information	Package	(SIP).
query() Retrieve	data	from	a	pre-determined	service	request	(synchronous).
order() Request	data	from	a	pre-determined	service	request	(asynchronous).
download() Retrieve	a	Dissemination	Information	Package	(DIP).
status() Track	progress	of	service	activity.
execute() Initiate	a	service-definable	extension.		Allows	for	parameterized	growth	without	API

change..

Extended Utilities (and convenience functions)

avg() Calculate

average

and

download

the

result.

sum() Calculate

summary

and

download

the

result.

max() Calculate

maximum

and

download

the

result.

min() Calculate

minimum

and

download

the

result.

var() Calculate

variance

and

download

the

result.

Variance

measures

how

far

set

numbers

count() Calculate

overall

count

and

download

the

result.

anomaly() Calculate

the

difference

between

the

time

span

average

and

each

data

point

and

download	the	result.
Y aavg() Calculate	an	area	average	and	download	the	result.
Y avgd() Calculate	a	diurnal	average	and	download	the	result.
Y anomaly_simple() Calculate	the	anomaly	between	individual	datasets	and	an	ensemble	average	and

download

the

result.

Y ensemble_avg() Calculate

ensemble

average

and

download

the

result.

Y concatenate_files() Join

multiple

files

into

one

and

download

the

result.

Y compute_max() Compute

maximum

grid

values

across

files

and

download

the

result.

Y regrid()

Regrid

file

according

desired

spatial

resolution

and

download

the

result.

Y diff()

Compute

difference

between

two

files

and

download

the

result.

Y poll() Poll

the

RES

service

determine

the

status

order.

Y placeOrder() Place

order.

Y climatology() Compute

climatology

and

download

the

result.

Y getClimatologies() Compute

multiple

climatologies

and

download

the

result.

downloadCopyResult() Download a file and copy it to the appropriate destination.

downloadResult() Download

file

and

move

the

appropriate

destination.

getLogger() Get

handle

the

log

file.

getElement() Get

the

value

associated

with

tag

XML

string.

move() Move

file

from

source

path

destination

path.

encode() Convert

mapping

object

sequence

two-element

tuples

percent-encoded

string,	suitable	to	pass	to	urlopen().
copy() Copy	a	file	from	source	path	to	destination	path.

CDSlib Python Library Jupyter Notebook

CDSlib Utility Status

New Utility Description(Long	Name)
Basic	Utilities	(1:1	mapping	with	W/S)

Extended Utilities (and convenience functions)

Y std() Calculate the standard deviation and download the result.

Y ensemble_std() Calculate an ensemble standard deviation and download the result.

Y orchestrate() apply workflow to retrieve data from multiple sources

Y aggregate() package results according to session id and request context

Y wrangler.order() Request data from Wrangler (asynchronous).

Y wrangler.download() Download a Dissemination Information Package (DIP) from Wrangler.

Y wrangler.status() Track progress of service activity in Wrangler.

Y TBD Additional extended utilities

9/19/17

Accomplishments Since Last Checkpoint

9/19/1731

• Hardened all existing RES services

– Developed UQP capabilities as CDSlib Extended Utilities, Jupyter Notebooks

• Prototyped the GEOS-5 Forward Processing (FP) data analytics service

– Applied MERRA/AS capabilities to GEOS-5 near-real-time forecast data

• Developed automated multi-source RES data provisioning capability

– Created a library (LoaderLib) to retrieve, prepare, validate, and refresh data collections for analytic processing

– Created an application (Loader) to standup remote RES deployments for customized datasets

• Packaged software to serve as a Platform-as-a-Service (PaaS) for remote hosting

– Built a Docker-ready version of MERRA/AS that can be deployed to multiple different hosting environments

• Broadened partnerships and outreach activities

– Established relationships with AccuWeather, Scoville Risk Partners, Conservation International, Applied Sciences

Program, GMAO, ESDIS, etc.

• Broadened beta test community to support GMAO, AIST, ABoVE, AccuWeather

– Example: Temperature trend detection w/ concentric boundaries to observe changes in averages with respect to distances from data collection stations (Mark Carroll)

• Reported six technology innovations to NASA’s Strategic Partnerships Office

CMAC-14 Project Status Review 9/19/1732

GEOS	Forward	Processing	(FP)	:
Starting	at	February	2013,	GMAO	provides	GEOS	data	products	in	near	real
time	in	support	of	diverse	user	communities.

The	Forward	Processing	stream	generates	forecasts	products	that	are
primarily	used	for	real	time	support	for	NASA	field	campaigns.

system

using

the

most

current

GEOS

model

and

updated

the

GEOS

system

improves.

Forecasting	data	highlights:
• 10-day	forecasts	at	00Z,	and	5-day forecasts	at	12Z
• 3-D	collections	are	on	a	5/16	by	1/4	degree	lon-lat	horizontal	grid
• 42	pressure	levels	or	72	model	grid	layers
• 2-D	data	every	hour,	3-D	data	every	3	hours

Support for GEOS Forward Processing products

Automated Multi-Source Data Provisioning

• Application (Loader) builds and automatic updates customized, stand-alone RES systems inside and outside of NASA

• Includes tools to retrieve, prepare, validate, and refresh data collections for analytic processing (LoaderLib)

• Addresses need of a wide range of government and private-sector science, research, and application customers who want reanalysis data subsets or to combine reanalysis data with other types of data for advanced analytics …

• Addresses need for simplified multi-source updates

• The application does the following:

– Connects through a client interface to a remote server where raw data is hosted

– Downloads the raw data from the remote server to a local system

– Sequences the data to local HDFS if not already sequenced

– Validates that the files have been sequenced properly

9/19/17

Platform-as-a-Service (PaaS)

• Wrapped MERRA/AS stack for remote hosting as a distributable PaaS package in order to easily share

• Established a Docker-ready version of MERRA/AS that can be deployed to multiple targets

– This package includes all MERRA/AS-required software and data

• Application tier (Map/Reduce), web service tier (PHP), client tier

(CDSLib/Python), configuration files

• MERRA2 Monthly Means subset

– We are building additional tools to assist with customized system configuration and extension

• Targeting NGAP and ADAPT as MERRA/AS PaaS hosts

– NGAP (Next Generation Application Platform) is ESDIS’s cloud-based platform for NASA-compliant applications.

• We are developing a proof-of-concept cloud-based climate data analytics system to support the delivery of information relevant to the issue of climate resilience (i.e., RES/Docker).

• This PaaS approach is intriguing because it can assist a variety of partners with a single technical solution

9/19/1734

Platform-as-a-service (PaaS) is a category of cloud computing services that provides a platform allowing partners to develop, run, and manage applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an application

Docker uses containers as a way to package software in a format that can run isolated on a shared operating system. Unlike VMs, containers do not bundle a full operating system

- only libraries and settings required to make the software work are needed.

This is the start of the file's text. The full file is on GovTribe.

File details come from the government source that posted it.