Attachment 5 - RCC Data Management Plan.pdf

Data Management Plan for the NCEI Regional Climate Product Area v0.1 (2018-06-19)

Overview

The NOAA National Centers for Environmental Information (NCEI) mission is to serve as steward of the Nation’s environmental information. NCEI constitutes the official NOAA data archive and is a designated NARA Agency Records Center . NCEI maintains, processes, distributes, and provides long-term stewardship for most of NOAA's environmental and geospatial data, and provides a broad range of user services. NCEI is operated by NESDIS but performs data preservation and stewardship on behalf of the entire agency. NCEI supports U.S. and international environmental, societal, and economic well-being through stewardship of the Nation’s treasure of weather, climate, coastal, oceanic, and geophysical information.

In addition to stewarding and preserving data generated and collected by other parts of NOAA, NCEI also creates value-added datasets and products from the archived data. In 2018, NCEI began organizing these value-added products into Product Areas. With consultation from the NESDIS Data Management Working Group, NCEI determined to document one Data Management Plan per Product Area.

Currently, 27 Product Areas have been defined by NCEI’s Center for Coasts, Oceans, and Geophysics (CCOG) and Center for Weather and Climate (CWC). This document is the Data Management Plan for the NCEI Regional Climate Product Area.

1. General Description of Data to be Managed

1.1. Name of the Data, data collection Project, or data-producing Program:

All value-added data and products created and managed by the NCEI/CWC Regional Climate Product Area. For this NCEI product area, the value-added products are created by and served from the Regional Climate Centers (RCCs).

1.2. Summary description of the data:

The Regional Products Area provides climate products to six regions encompassing the United States through NCEI’s Regional Climate Service (RCS) program. Six contracted regional climate centers

(RCCs) produce and deliver climate data, products, information, and services for decision makers and other users at the local, state, regional, and national levels. The product suite is responsive to specific and validated user needs that support the economic and commercial activity in the respective regions. The RCS program is mandated by Congress.

1 www.archives.gov/files/foia/directives/nara1310.pdf

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 1 of 8

The six RCC’s and their locations are:

● High Plains Regional Climate Center (HPRCC) - Lincoln, NE

● Midwestern Regional Climate Center (MRCC) - Champaign, IL

● Northeast Regional Climate Center (NRCC) - Ithaca, NY

● Southeast Regional Climate Center (SERCC) - Chapel Hill, NC

● Southern Regional Climate Center (SRCC) - Baton Rouge, LA

● Western Regional Climate Center (WRCC) - Reno, NV

This NCEI product area currently includes the following products, created and served by the RCCs:

Applied Climate Web Services (aka ACIS, NRCC)

Automated Weather Datasets (HPRCC)

Climate and Forecast Perspectives (SERCC)

Climate and Health (SERCC)

Climate Data Collection (aka WxCoder, WRCC)

Climate Data Quality Control (aka Datzilla, SRCC)

Climate Tools for Agriculture (HPRCC, MRCC, NRCC)

HPRCC, MRCC, NRCC - Analyses for Industry

Extremes (aka Threadex, NRCC)

Freeze/Thaw (NRCC)

Regional Climate Summaries and Webinars (All RCCs)

Summaries

Webinars: HPRCC, MRCC, NRCC

Regional Mesonet Products (MRCC)

Regional Winter Weather (MRCC)

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 2 of 8

Sporting Events Climatologies (SERCC)

Station-Based Climate Data and Summaries (All RCCs)

HPRCC, MRCC, NRCC, SERCC, SRCC, WRCC

Tropical Weather and Impacts (SRCC)

Vegetation Impact Products (MRCC)

While the RCCs use NOAA data to produce these products, RCC-derived products are created by and hosted at the RCC,s and not at NCEI.

1.3. Is this a one-time data collection, or an ongoing series of measurements?

Ongoing

1.4. Actual or planned temporal coverage of the data:

Temporal coverage of the data and products with this product area vary depending on the dataset used to develop the product and are based on the dataset and station period of record.

1.5. Actual or planned geographic coverage of the data:

In most cases, the geographic coverage is regional per the RCC developing the product. Please see the map in section 1.2. In other cases, while one RCC may have developed the product, geographic coverage may be for the entire U.S.

1.6. Type(s) of data: (e.g., digital numeric data, imagery, photographs, video, audio, database, tabular data, etc.)

Types of data and products in this product area include tabular data, graphics, and video.

1.7. Data collection method(s): (e.g., satellite, airplane, unmanned aerial system, radar, weather station, moored buoy, research vessel, autonomous underwater vehicle, animal tagging, manual surveys, enforcement activities, numerical model, etc.)

Products in this product area primarily use data from NWS and CRN weather stations.

1.8. If data are from a NOAA Observing System of Record , indicate name of system(s):

Data used in this product area are primarily from COOP, CoCoRaHS, ASOS, SNOTEL, RAWS and state mesonets.

2 https://docs.google.com/spreadsheets/d/1uC_FxXYP9LYvY6ofhXRPplxiP9sZI3bhCpkKtwDsk3o/edit#gid=1462893252

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 3 of 8

1.8.1. If data are from another observing system, please specify:

Unknown

2. Point of Contact for this Data Management Plan (author or maintainer)

2.1. Name: Tamara Houston

2.2. Title: Physical Scientist

2.3. Affiliation or facility: NOAA National Centers for Environmental Information (NCEI)

2.4. E-mail address: tamara.houston@noaa.gov

2.5. Phone number: (828) 271-4266

3. Responsible Party for Data Management Program Managers, or their designee, shall be responsible for ensuring the proper management of the data produced by their Program. Please indicate the responsible party below.

3.1. Name: Jeff Privette

3.2. Position Title: Deputy Director, Center for Weather and Climate

4. Resources Programs must identify resources within their own budget for managing the data they produce.

4.1. Have resources for management of these data been identified?

Yes

4.2. Approximate percentage of the budget for these data devoted to data management (specify percentage or "unknown"):

Unknown

5. Data Lineage and Quality NOAA has issued Information Quality Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information which it disseminates.

5.1. Processing workflow of the data from collection or acquisition to making it publicly accessible:

The input data used to create the value-added products in this Product Area come from various providers, including NOAA and non-NOAA observing systems. Scientific quality assessment and control algorithms are applied to remove systematic and non-systematic errors. Science-based algorithms are applied to the original data to create higher level products that serve as the core elements of the Product Area. The derived products are quality-assessed. Currently these

3 http://www.cio.noaa.gov/services_programs/IQ_Guidelines_030414.html

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 4 of 8 products are managed outside of the NCEI Archive by the RCCs.

5.2. Quality control procedures employed (describe or provide URL of description):

NCEI data and products follow standard quality procedures which include one or more of the following methods: Employ various statistical methods to check data precision and accuracy;

Publish results in peer-reviewed articles in domain-relevant journals; Adopt quality standards recommended by the National Academy of Sciences and other expert organizations to ensure consistent, reliable, and scientifically defensible products. The RCCs also work directly with NCEI when data errors are found and perform quality control checks on the products they produce.

6. Data Documentation The EDMC Data Documentation Procedural Directive requires that NOAA data be well documented, specifies the use of ISO 19115 and related standards for documentation of new data, and provides links to resources and tools for metadata creation and validation.

6.1. Does metadata comply with EDMC Data Documentation directive ?

Yes, the NCEI-generated input data sources used to create the value-added products in this product area are represented by an ISO 19115 compliant collection-level metadata record in the NOAA OneStop catalog. RCC-generated products are not.

6.1.1. If metadata are non-existent or non-compliant, please explain:

Applicable metadata for RCC-generated products is located at the RCC along with the product.

6.2. Name of organization or facility providing metadata hosting:

Metadata are hosted by NCEI and available through numerous mechanisms detailed below, including Web Accessible Folders (WAFS), a Geoportal Server and REST API, and the NOAA OneStop user interface and API. Metadata hosted by the RCCs are available by contacting the applicable RCC.

6.2.1. If service is needed for metadata hosting, please indicate: No metadata hosting service is required.

6.3. URL of metadata folder or data catalog, if known:

All NCEI metadata records are discoverable through the NOAA OneStop web interface at https://data.noaa.gov/onestop as well as the NCEI Geoportal Server web interface at https://www.ncei.noaa.gov/metadata/geoportal. In addition, NCEI metadata records can be browsed via the Web Accessible Folder (WAF) at https://data.nodc.noaa.gov/nodc/archive/metadata/approved/ (location to be updated to https://data.noaa.gov/wafs/NOAA/NESDIS/NCEI in near future). NCEI’s metadata can also be searched via the following interoperable machine services:

Service Name Base URL Help

OneStop Search

API

https://data.noaa.gov/onestop/api/search Web Interface:

https://data.noaa.gov/one stop/#/help

API:

4 https://www.nosc.noaa.gov/EDMC/PD.DD.php 5 https://nosc.noaa.gov/EDMC/documents/DataDocumentationPD-v2.0.0.signed.pdf

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 5 of 8 https://github.com/cedard evs/onestop/wiki/OneStop -Search-API

Catalog Service for the Web

(CSW)

https://www.ncei.noaa.gov/metadata/geo portal/csw?service=CSW&version=3.0.0&r equest=GetRecords https://www.ncei.noaa.gov/m etadata/geoportal (Please click “Help” to view the detailed search tips) OpenSearch https://www.ncei.noaa.gov/metadata/geo portal/opensearch

JavaScript Object Notation (JSON) https://www.ncei.noaa.gov/metadata/geo portal/opensearch?&f=json

Keyhole Markup Language (KML) https://www.ncei.noaa.gov/metadata/geo portal/opensearch?&f=kml

Comma Separated Value (CSV) format https://www.ncei.noaa.gov/metadata/geo portal/opensearch?&f=csv

Really Simple Syndication (RSS) https://www.ncei.noaa.gov/metadata/geo portal/opensearch?&f=rss

Metadata hosted by the Regional Climate Centers are available by contacting the applicable

RCC.

6.4. Process for producing and maintaining metadata (describe or provide URL of description):

NCEI has a robust process for developing metadata. Data stewards develop standards, best practices and implementation strategies in alignment with the EDMC Procedural Directives.

Technology stewards develop tools to assist in metadata development and verification. The Science stewards are trained in these best practices and tools. Both Science and Data stewards develop and maintain metadata. At the RCCs, various methods are taken to produce and maintain metadata for the value-added products created at each RCC.

7. Data Access NAO 212-15 states that access to environmental data may only be restricted when distribution is explicitly limited by law, regulation, policy (such as those applicable to personally identifiable information or protected critical infrastructure information or proprietary trade information) or by security requirements. The EDMC Data Access Procedural Directive contains specific guidance, recommends the use of open-standard, interoperable, non-proprietary web services, provides information about resources and tools to enable data access, and includes a Waiver to be submitted to justify any approach other than full, unrestricted public access.

6 https://nosc.noaa.gov/EDMC/documents/NAO_212-15_Signed_4pg.pdf 7 https://nosc.noaa.gov/EDMC/documents/EDMC-DataAccess-PD.v1.0.pdf

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 6 of 8

7.1. Do these data comply with the Data Access directive?

Yes. As part of NCEI’s basic data stewardship services, access is provided to the data it archives, except in cases where such access is precluded by agreement with the provider (as might be the case for some commercial or international data), contains Personally Identifiable Information, or is otherwise procedurally restricted, as might be the case during test phases of new observing systems or satellites. These situations are relatively rare, and the vast majority of the NCEI archives are freely and openly accessible without restriction.

All value-data products in the Regional Climate Product Area, created at the RCCs, are publicly available without restriction directly from the RCCs.

7.1.1. If the data are not to be made available to the public at all, or with limitations, has a Waiver (Appendix A of Data Access directive) been filed?

Not Applicable

7.1.2. If there are limitations to public data access, describe how data are protected from unauthorized access or disclosure:

Not Applicable

7.2. Name of organization of facility providing data access: NCEI (for the input data sources) and the RCCs (for the value-added products and services).

7.2.1. If data hosting service is needed, please indicate: No data hosting service is needed.

7.2.2. URL of data access service, if known:

All NCEI data access services are available from https://www.ncei.noaa.gov.

Value-added products created by the Regional Climate Product Area can be found by contacting the RCCs.

7.3. Data access methods or services offered:

NCEI offers HTTPS, THREDDS, LAS, ArcGIS Server, OPeNDAP Hyrax Server, ERDDAP, FTP download, project-specific interfaces, and Customer Support services for the input datasets used by the RCCs to create their value-added products and services. Not all datasets are available through all services.

The value-added products and services created by the RCCs for this product area are available through a variety of methods which can be determined by contacting the RCCs.

7.4. Approximate delay between data collection/production and dissemination:

7.4.1. If delay is longer than latency of automated processing, indicate under what authority data access is delayed:

All NCEI-generated value-added products and input data sources are made available upon completion of automated and/or manual processing and review workflow steps.

No additional delays are introduced. The same is generally true for the value-added products created by the RCCs.

8. Data Preservation and Protection The NOAA Procedure for Scientific Records Appraisal and Archive Approval describes how to identify, 8 https://nosc.noaa.gov/EDMC/documents/NOAA_Procedure_document_final_12-16-1.pdf

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 7 of 8 appraise and decide what scientific records are to be preserved in a NOAA archive.

8.1. Actual or planned long-term data archive location:

All NCEI-generated value-added products are archived at NCEI. The value-added products in this product area created by the RCCs are currently stored for the longer term at the RCCs but are not formally archived by NCEI.

8.1.1. If World Data Center or Other, specify:

NCEI is an ICSU World Data System for Oceanography, Geophysics, Meteorology, and Paleoclimatology.

8.2. Data storage facility prior to being sent to an archive facility (if any):

All NCEI-generated value-added products and input data sources are developed and stored at NCEI. The value-added products created by the RCCs for this product area are developed and stored at the RCCs.

8.3. Approximate delay between data collection and submission to an archive facility:

The input data sources used by the RCCs are typicallyu submitted to the NCEI archival process by the original data collectors and are made available immediately upon completion of automated and/or manual processing and review workflow steps. No additional delays are introduced.

Currently, the value-addedd products created by the RCCs are not submitted to NCEI for archiving.

8.4. How will the data be protected from accidental or malicious modification or deletion prior to receipt by the archive? Discuss data back-up, disaster recovery/contingency planning, and off-site data storage relevant to the data collection:

All NCEI-generated value-added products are produced and processed on FISMA-compliant systems with current system security measures and monitoring in place. Data processing systems are backed-up in order to support disaster recovery. The same is true for the input data sources archived at NCEI and used by the RCCs in their creation of the value-added products. The value-added products themselves are created, served, and protected by the RCCs directly following a variety of methods and approaches.

9. Additional Line Office or Staff Office Questions Line and Staff Offices may extend this template by inserting additional questions in this section.

NCEI Digital Archive Data Management Plan (draft) Effective 2017 Aug 30 Page 8 of 8