Sole_source_justification_with_documentation__PR_40673625_1.pdf

FWS Form (Rev. 2020) U.S. Fish and Wildlife Service

JOINT ADMINISTRATIVE OPERATIONS

Acquisitions Operations

Program / Project Officer

Contracting Officer

EPA/600/R-20/280

February 2021

Performance Testing Protocols, Metrics, and Target Values for Fine Particulate Matter Air Sensors

Use in Ambient, Outdoor, Fixed Site, Non-Regulatory Supplemental and Informational Monitoring Applications

By

Rachelle M. Duvall, Andrea L. Clements, Gayle Hagler, Ali Kamal, and Vasu Kilaru U.S. Environmental Protection Agency

Research Triangle Park, NC 27711

Laura Goodman and Samuel Frederick National Student Services Contract

U.S. Environmental Protection Agency Research Triangle Park, NC 27711

Karoline K. (Johnson) Barkjohn and Ian VonWald Oak Ridge Institute for Science and Education

U.S. Environmental Protection Agency Research Triangle Park, NC 27711

Danny Greene Eastern Research Group, Inc.

Morrisville, NC 27560

Tim Dye TD Environmental Services, LLC

Petaluma, CA 94952 ii

Disclaimer

This technical report presents work performed by the U.S. Environmental Protection Agency’s (U.S.

EPA) Office of Research and Development (ORD) with technical support provided by Eastern Research Group through a task order (Task Order 68HERC19F0071, EPA Contract #EP-C-16-015). The effort represents a collaboration between ORD and the U.S. EPA Office of Air Quality Planning and Standards (OAQPS). Any mention of trade names, manufacturers, commercial products, or research institutions does not constitute endorsement. This report has been reviewed in accordance with U.S. EPA policy and approved for publication. Testing results do not constitute certification or endorsement by the U.S. EPA.

iii

Table of Contents Disclaimer ................................................................................................................................. ii

List of Figures .......................................................................................................................... vi

List of Tables........................................................................................................................... vii

Acronyms and Abbreviations ................................................................................................... ix

Acknowledgments..................................................................................................................... x

Executive Summary ................................................................................................................. xi

1.0 Introduction

1.1 Background

1.2 Motivation

1.3 Objectives

2.0 Performance Testing Protocols for PM2.5 Air Sensors

2.1 Base Testing

2.1.1 Materials and Equipment

2.1.2 Selecting and Setting Up a Test Site

2.1.3 Setting Up the Air Sensors

2.1.4 Conduct Base Testing

2.2 Enhanced Testing

iv

2.2.1 Materials and Equipment

2.2.2 Equipment Set Up in Exposure Chamber

2.2.3 Initial Testing Conditions

2.2.4 Effect of Relative Humidity (RH)

2.2.5 Effect of Temperature (T)

2.2.6 Drift

2.2.6.1 Drift (Day 1) – Low and Mid Concentration Drift

2.2.6.2 Drift (Day 60) to Evaluate Sensor Aging

2.2.7 Accuracy at High Concentrations

3.0 Performance Metrics and Supporting Calculations for Evaluating PM2.5 Air Sensors

3.1 Base Testing Calculations

3.1.1 Daily Averages

3.1.2 Deployment Averages

3.1.3 Precision

3.1.4 Bias and Linearity

3.1.5 Error

3.1.6 Exploring Effect of Meteorology

3.1.6.2 Normalized Concentration

v

3.1.6.3 Concentration Difference and Absolute Concentration Difference

3.1.6.4 Dew Point (DP)

3.2 Enhanced Testing Calculations

3.2.1 Data Averages

3.2.2 Test Averages

3.2.3 Precision

3.2.4 Bias and Linearity

3.2.5 Error

3.2.6 Effect of Relative Humidity (RH)

3.2.7 Effect of Temperature (T)

3.2.8 Drift

3.2.9 Accuracy at High Concentrations

4.0 Target Values for PM2.5 Air Sensors

4.1 Approach

4.2 List of Target Values

5.0 References

Appendix A: Definitions

Appendix B: Supporting Information for Testing Protocols vi

Appendix C: Supporting Information for Performance Metrics

Appendix D: Supporting Information for Target Values

Appendix E: Checklist for Base Testing

Appendix F: Example Reporting Template for Base Testing

Appendix G: Checklist for Enhanced Testing

Appendix H: Example Reporting Template for Enhanced Testing

List of Figures

Figure 2-1. Example Approach for Conducting Base and Enhanced Testing of a Single Set of

Sensors

Figure 2-2. U.S. Climate Regions (https://www.ncdc.noaa.gov/monitoring-references/maps/us-climate-regions.php, last accessed 07/29/2020)

Figure 2-3. Overview of the Enhanced Testing Procedure

Figure 2-4. Drift Testing to Determine Changes After 60 days or more of Continuous Operation vii

List of Tables

Table ES-1. Recommended Testing Protocols for Understanding PM2.5 Air Sensor Performance ....................................................................................................................... xi

Table ES-2. Base and Enhanced Testing – Recommended Performance Metrics and Target Values for PM2.5 Air Sensors .............................................................................................. xii

Table ES-3. Enhanced Testing – Additional Recommended Performance Metrics and Test Conditions for PM2.5 Air Sensors ....................................................................................... xiii

Table 1-1. NSIM Categories and Specific Examples (adapted from U.S. EPA, 2018)

Table 2-1. Test Site Selection Criteria

Table 2-2. Sampling Probes or Monitoring Path Siting Criteria

Table 2-3. Guidance on Air Sensor Setup at Testing Site

Table 2-4. Initial Testing Conditions

Table 2-5. Elevated RH Test Conditions

Table 2-6. Elevated T Test Conditions

Table 2-7. Low Concentration Drift Test Conditions

Table 2-8. Mid Concentration Drift Test Conditions

Table 2-9. High and Higher PM2.5 Concentration Test Conditions

Table 3-1. Summary of Recommended Performance Metrics for PM2.5 Air Sensors

Table 4-1. PM2.5 Sensor Performance Field Evaluation Results from Available Resources

Table 4-2. Base and Enhanced Testing – Recommended Performance Metrics and Target viii

Values for PM2.5 Air Sensors Used in Ambient, Outdoor, Fixed Site NSIM Applications... 37

Table 4-3. Enhanced Testing – Additional Recommended Performance Metrics and Test Conditions for PM2.5 Air Sensors Used in Ambient, Outdoor, Fixed Site NSIM Applications

Table B-1. General locations likely to meet 25 µg/m3 criteria by month based on analysis of 2017-2019 PM2.5 data. Locations typically referred to in terms of climate zones shown in Figure 2-2

Table D-1. Performance Requirements for PM2.5 FRM/FEM Regulatory Monitors (adapted from

U.S. EPA, 2018)

Table D-2. Summary of U.S. EPA PM2.5 Sensor Evaluation Field Results (24-hour Average) 51

Table D-3. Summary of AQ-SPEC PM2.5 Sensor Field Evaluation Results (24-hour Average) 52

Table D-4. Summary of Literature Reviews of PM2.5 Sensor Evaluations (24-hour average) used to Inform Target Values

Table D-5. Summary of Available Resources Used to Inform Target Values ix

Acronyms and Abbreviations

AMTIC Ambient Monitoring Technology Information Center AQI Air Quality Index AQ-SPEC Air Quality Sensor Performance Evaluation Center AQS Air Quality System b intercept CEMM Center for Environmental Measurement and Modeling CFR Code of Federal Regulations CSN Chemical Speciation Network CV coefficient of variation °C degrees Celsius DL detection limit DP dew point Eq equation FEM Federal Equivalent Method FRM Federal Reference Method m slope MEE Ministry of Ecology and Environment (People’s Republic of China) NAAQS National Ambient Air Quality Standards NCore National Core Network NIST National Institute of Standards and Technology NRMSE normalized root mean square error NSIM non-regulatory supplemental and informational monitoring OAQPS Office of Air Quality Planning and Standards ORD Office of Research and Development PM particulate matter PM2.5 fine particulate matter; particles with aerodynamic diameters less than 2.5 micrometers QA quality assurance QAPP Quality Assurance Project Plan QC quality control r Pearson correlation coefficient R2 coefficient of determination RH relative humidity RMSE root mean square error SD standard deviation SLAMS State and Local Air Monitoring Station T temperature g/m3 microgram per cubic meter U.S. United States U.S. EPA United States Environmental Protection Agency x

Acknowledgments

The authors acknowledge the Eastern Research Group technical staff associated with Task Order 68HERC19F0071, EPA Contract #EP-C-16-015 for their research efforts summarized here. This effort was led by the United States Environmental Protection Agency (U.S. EPA), Office of Research and Development (ORD), Center for Environmental Measurement and Modeling (CEMM) with support from the Sensor Performance Targets Deliberation Workgroup that consisted of staff across the U.S.

EPA. We acknowledge the Subject Matter Experts that participated in the U.S. EPA’s 2018 workshop on “Deliberating Performance Targets for Air Quality Sensors”, whose feedback helped inform this report. Libby Nessley and Christina Alvarez (U.S. EPA/ORD/CEMM Quality Assurance) are recognized for quality assurance support in developing this report. We acknowledge U.S. EPA internal reviewers Kristen Benedict, Trisha Curran, Ron Evans, Tim Hanley, Doug Jager, Sue Kimbrough (retired), and Robert Vanderpool. We also acknowledge the following external reviewers: Pratim Biswas (Washington University); Jason Treutel (Wisconsin Department of Natural Resources); and Vasileios Papapostolou (South Coast Air Quality Management District).

Title page photo by South Coast Air Quality Management District as part of work under the U.S. EPA Contract #EP-16-W-000117. Photo shows the custom-built Citizen Science Air Monitors at the Rubidoux Air Monitoring site in Riverside, California.

xi

Executive Summary

Air sensors have become more accessible nationwide and their development continues to expand and evolve at a rapid pace. There has been a dramatic increase in the use of air sensors for a variety of air monitoring applications and data sets have become more available to the public. While air sensors have encouraged innovation in air monitoring approaches, it is widely known that the data quality from these technologies is highly variable. The variability in data quality makes it challenging to understand the performance of any given sensor device and if a sensor will appropriately fit an application of interest.

Additionally, organizations that manage air quality face challenges in responding to air sensor data provided by the public as there is a lack of knowledge on how air sensor technologies perform which can make it more difficult to trust or interpret the data.

While programs such as the United States Environmental Protection Agency’s (U.S. EPA) Federal Reference Method and Federal Equivalent Method (FRM/FEM) Program [Code of Federal Regulations, Title 40 (40 CFR) Parts 50, 53, and 58] contain standards and performance certification processes for air quality instruments used for regulatory monitoring purposes, it is recognized that air sensors will likely not meet those stringent requirements. However, sensors could be useful in many non-regulatory applications such as understanding local air quality trends, identifying hot spots, supplemental monitoring, and promoting educational/environmental awareness. Currently, there are no standard testing protocols or targets for air sensors.

The objective of this report is to provide a consistent set of testing protocols, metrics, and target values to evaluate the performance of fine particulate matter (PM2.5) air sensors specifically for non-regulatory supplemental and informational monitoring (NSIM) applications in ambient, outdoor, fixed site environments. Two testing protocols, base testing and enhanced testing, are recommended (Table ES-1).

Table ES-1. Recommended Testing Protocols for Understanding PM2.5 Air Sensor Performance

Test Type

Setting Description Purpose

Base Testing

Field Consists of field deployments of at least three replicate PM2.5 air sensors with collocated FRM/FEM monitors for a minimum of 30 days each, at two test sites within different climate regions.

Provides information on sensor performance that is relevant to real-world, ambient, outdoor conditions.

Allows consumers to predict how a sensor might perform in similar conditions.

Enhanced Testing

Laboratory Consists of testing at least three replicate PM2.5 air sensors in controlled laboratory conditions to understand the effect of temperature and relative humidity; drift; and accuracy at higher concentration levels.

Allows for evaluation of sensors over a range of conditions that may be challenging to capture in the field.

Characterizes certain performance parameters that are difficult to test in the field.

xii

All testers are encouraged to conduct base testing at a minimum. Enhanced testing is also encouraged although it calls for a controlled laboratory exposure chamber. For NSIM applications where high PM2.5 concentrations are expected (e.g., wildfire smoke applications), it is recommended that testers conduct base testing in more than two locations and include sites impacted by wildfire smoke and higher PM2.5 concentrations.

Performance metrics and corresponding target values have been identified based on the current state-of- the-science, literature reviews, findings from other organizations that conduct routine sensor evaluations, sensor standards/certification programs in development by other organizations, and the U.S. EPA expertise in sensor evaluation research. A summary of the performance metrics and target values for the base and enhanced testing protocols are shown in Table ES-2. For base testing, an additional data visualization called ‘exploring meteorological effects’ is recommended which includes graphing meteorological data to understand its influences on sensor performance. Further for base testing, it is recommended that at the base test sites, at least one day of the testing period has a 24-hour average PM2.5 concentration of at least 25 microgram per cubic meter ( g/m3) or greater. Additional performance metrics and test conditions for the enhanced testing protocols are shown in Table ES-3.

This report provides details on how to calculate the performance metrics for PM2.5 sensors (see Section 3.0) and templates for base and enhanced testing reports for consistent reporting of testing results (see Appendix F and H).

Table ES-2. Base and Enhanced Testing – Recommended Performance Metrics and Target Values for PM2.5 Air Sensors

Performance Metric Target Value

Base Testing Enhanced Testing*

Precision Standard Deviation (SD)

-OR-

Coefficient of Variation (CV) g/m3

No target values recommended;

report results

Bias Slope 1.0 ± 0.35

Intercept (b) - g/m3

Linearity Coefficient of Determination (R2)

Error Root Mean Square Error (RMSE) or Normalized Root Mean Square Error

(NRMSE)

g/m3 or

*No specific target values are recommended due to limited feasibility, lack of consensus regarding testing protocols, and inconsistency in sensor evaluation results that can result from the limited amount of data that will be collected and variation in the tester’s choice of PM surrogate. See Appendix D for further discussion.

†A sensor will meet this target if either the RMSE or NRMSE meet this criterion. See Appendix D for further discussion.

xiii

Table ES-3. Enhanced Testing – Additional Recommended Performance Metrics and Test Conditions for PM2.5 Air Sensors

Performance Metric Test Conditions

Effect of Relative Humidity (RH) Moderate

Elevated

Effect of Temperature (T) Moderate T: 20°C ± 1°C

Elevated T: 40°C ± 1°C

Drift

Low concentration: 10 g/m3

Mid concentration: 35 g/m3 ± 5

Accuracy at High Concentrations High concentration: 150 g/m3 ± 5

Higher concentration: 250 g/m3 ± 5

The performance metrics and target values for base and enhanced testing are recommended based on the current knowledge of PM2.5 air sensors at the time this report was released. Target values for enhanced testing are not included at this time due to limited feasibility, lack of consensus regarding testing protocols, and inconsistency in sensor evaluation results that can result due to the limited amount of data that will be collected and variation in the tester’s choice of PM surrogate.

It is recognized that PM2.5 sensor technologies will likely continue to develop and improve over time.

The U.S. EPA anticipates updating Tables ES-2 and ES-3 as well as other information in this report, as feasible, to reflect advances in PM2.5 sensor technologies and knowledge gained from sensor evaluation results. Updates will likely be shared as an addendum to this report.

The intended audience for this report includes potential testing organizations, sensor manufacturers, and sensor developers. It is anticipated that a variety of consumers (e.g., state/local/tribal agencies, federal government agencies, community groups, citizen scientists, academia) will benefit from the consistent presentation of testing results to identify sensor technologies that would be best suited for their NSIM application and understand the performance of the air sensor technologies. Consumers may also choose to conduct these testing protocols.

Testing results do not constitute certification or endorsement by the U.S. EPA. It is recommended that testers make the testing reports available on their respective websites to inform consumers on the testing results.

1.0 Introduction

1.1 Background

The term ‘air sensor’ refers to a class of non-regulatory technology that are lower in cost, portable, and generally easier to operate than regulatory monitors. Air sensors often provide relatively quick or instant air pollution concentrations (both gas-based and particulate matter) and allow air quality to be measured in more locations. The term ‘air sensor’ often describes an integrated set of hardware and software that uses one or more sensing elements (also sometimes called sensors) to detect or measure pollutant concentrations. Other commonly used terms for air sensors include “low-cost air sensors”, “lower cost air sensors”, “air sensor devices”, “air sensor pods”, and “air quality sensors”. Advancements in microprocessors and miniaturization have led to a rapid expansion in the availability of air sensors to measure a variety of air pollutants. As air sensors have become more accessible nationwide, there has been a dramatic increase in their use for non-regulatory air quality monitoring purposes and greater access to publicly available sensor data sets (e.g., Zamora et al., 2020; Li et al., 2020; Feenstra et al., 2019; Bulot et al., 2019; Badura et al., 2018; Crilley et al., 2018; Zheng et al., 2018; Nakayama et al, 2018; Kelly et al., 2017; Zikova et al., 2017; Mukerjee et al., 2017).

Since 2012, the United States Environmental Protection Agency (U.S. EPA) has been involved in many activities related to air sensors including, but not limited to, hosting workshops and webinars, evaluating new technologies and applications, developing tools to analyze and visualize data, and disseminating information. More details on these efforts can be found on the U.S. EPA’s Air Sensor Toolbox website (https://www.epa.gov/air-sensor-toolbox, last accessed 07/25/2020).

A variety of options for air sensors are available and development continues to expand and evolve at a rapid pace. However, it is widely known that the data quality from these technologies is highly variable (Williams et al., 2019). Some of the key challenges with PM2.5 air sensor technologies include:

Determining whether the sensor will measure the target pollutant accurately and reliably within the expected concentration range for the application;

Determining how different parameters including relative humidity (RH), temperature (T), and variations in particle composition or size can impact measurements;

Estimating how the sensor’s response changes over time and at what point in time the sensor reading becomes inaccurate or unreliable; and

Understanding how sensors perform out-of-the-box and if correction or adjustments are needed to provide more accurate data.

While programs such as the U.S. EPA’s Federal Reference Method and Federal Equivalent Method (FRM/FEM) Program [Code of Federal Regulations, Title 40 (CFR 40) Parts 50, 53, and 58] contain standards and performance certification processes for air quality instruments used for regulatory monitoring purposes, it is recognized that air sensors will not meet those stringent requirements for several reasons. Monitors designated as FRM/FEMs are specifically designed and manufactured to produce reliable, high quality measurements for use in compliance monitoring that meet all acceptance criteria for laboratory and field tests as outlined 40 CFR Parts 50 and 53. Sensors are typically not designed with these criteria in mind. However, some testing requirements and acceptance criteria in Parts 50 and 53 may be adaptable to evaluate sensor performance.

Currently, there is an absence of testing protocols and performance targets that air sensor manufacturers/developers can use to evaluate their devices. The comparability of sensors with FRM/FEMs is highly variable and the ability of sensors to provide consistent, accurate, and precise measurement data under real-world conditions is not well understood. Nevertheless, there is ongoing interest in using air sensors in non-regulatory air monitoring applications. Testing protocols and targets for air sensors would increase confidence in data quality and help consumers in selecting sensors that appropriately suit an application of interest.

1.2 Motivation

Around 2012, when the availability of air sensors began to expand rapidly, questions related to using sensors and interpreting sensor data began to increase significantly among the user community. The U.S. EPA responded by developing the Air Sensor Guidebook (U.S. EPA, 2014). The guidebook was designed to provide basic foundational knowledge to help those interested in using sensors for air quality measurements with a focus on: 1) background information on common air pollutants and air quality, 2) selecting appropriate sensors for different applications, 3) data quality considerations, and 4) sensor performance for different applications. The target audience for the Air Sensor Guidebook was citizen scientists and sensor manufacturers/developers. Since then, the user community has grown to include individuals, communities, schools, air quality and health agencies, medical professionals, and more.

New air sensor technologies continue to flood the market. Despite ongoing research to evaluate these technologies, variability in data quality persists. While several organizations are in the process of developing performance standards or guidance for evaluating air sensors, currently there are no consistent testing protocols that can be used for uniform evaluation and comparison of different technologies. Furthermore, recommended and testable performance metrics that can guide technology improvement, i.e., performance targets, do not exist for air sensors. The lack of testing protocols and targets can lead to confusion in the marketplace for both sensor manufacturers/developers and consumers. Without proper guidance, sensor manufacturers/developers may not know which procedures are needed to appropriately test the performance of a sensor for a given application.

Consumers may struggle to understand the performance of a sensor and which sensors will appropriately fit their desired application. Additionally, organizations that manage air quality (e.g., air or health agencies) may have difficulty responding to air sensor data that is provided by the public, especially when there is interest in using those data to bring attention to air pollution issues and to influence policy decisions. Without knowledge of how air sensor technologies perform, it is hard to understand the comparability of air sensor data with data from regulatory monitors.

While air sensor technologies are creating significant opportunities to monitor air quality, the variability in data quality creates challenges in understanding sensor performance. Having a consistent approach for evaluating the performance of air sensors benefits all stakeholders as it will provide confidence in data quality and help consumers identify appropriate air sensors for their intended application, encourage innovation and product improvement in the marketplace, and reduce uncertainty about the performance of a given technology. A priority for the U.S. EPA is to support technology development toward data that are of known quality and help establish best practices for the use of air sensors and their data.

1.3 Objectives

The purpose of this report is to provide a standard, streamlined, unbiased approach to testing the performance of fine particulate matter (PM2.5) air sensors for non-regulatory supplemental and informational monitoring (NSIM) applications in ambient, outdoor, fixed site environments. NSIM applications (summarized in Table 1-1) are the focus of this report as these areas have been identified as the primary use of air sensors in the U.S.

Table 1-1. NSIM Categories and Specific Examples (adapted from U.S. EPA, 2018)

Category Definition Examples

Spatiotemporal Variability

Characterizing a pollutant concentration over a geographic area and/or time

Daily trends, gradient studies, air quality forecasting, citizen science, education

Comparison Analysis of differences and/or similarities in air pollution characteristics against a threshold value or between different networks, locations, regions, time periods, etc.

Hot-spot detection, data fusion, emergency response, supplemental monitoring

Long-term Trend

Change in a pollutant concentration over a period of (typically) years

Long-term changes, epidemiological studies, model verification

This report provides specific guidance on testing protocols, performance metrics, and target values for those metrics for PM2.5 air sensors used in NSIM applications. This guidance combines U.S. EPA expertise in sensor evaluation and application research, expertise of other organizations who administer routine sensor evaluation programs, as well as findings from organizations that are developing similar guidance on sensors. Additionally, this guidance utilizes information gathered from two literature reviews conducted by the U.S. EPA that informed the development of sensor performance targets and testing protocols for NSIM applications. The first review identified the most important performance attributes to characterize instruments used to monitor air pollutants and quantitative performance metrics describing those performance attributes (U.S. EPA, 2018). The second review had a similar objective but examined more recent literature as well as results from field and laboratory sensor performance evaluations (U.S. EPA, 2020).

The specific objectives of this report are as follows:

Provide a consistent set of testing protocols, metrics, and target values to systematically evaluate the performance of air sensors;

Provide a consistent framework for communicating performance evaluation results; and

Help consumers make informed decisions on choosing sensors that might best suit a NSIM application of interest.

Collectively, these objectives will help provide a streamlined framework to understand air sensor performance for NSIM applications. It should be noted that other applications (e.g., mobile monitoring, indoor monitoring, personal exposure monitoring) may require different testing protocols which are not covered in this report.

The intended audience for this report includes potential testing organizations, sensor manufacturers, and sensor developers. It is anticipated that a variety of consumers (e.g., state/local/tribal agencies, federal government agencies, community groups, citizen scientists, academia) will benefit from the consistent presentation of testing results to identify sensor technologies that would be best suited for their NSIM application and understand the performance of the air sensor technologies. Consumers may also choose to conduct these testing protocols.

Results from these testing protocols do not constitute certification or endorsement by the U.S. EPA. It is recommended that testers make the testing reports available on their respective websites to inform consumers on the testing results.

2.0 Performance Testing Protocols for PM2.5

Air Sensors

The procedures outlined in this section provide standardized test protocols for evaluating the performance of PM2.5 air sensors (also called ‘air sensor’ and ‘sensor’ in this report). These procedures only apply to sensors used in NSIM applications in ambient, outdoor, fixed site environments. Two testing procedures are summarized: 1) base testing which involves field evaluations, and 2) enhanced testing which involves laboratory evaluations. Base testing at an ambient, outdoor evaluation site provides information on air sensor performance that is relevant to real-world conditions and allows consumers to predict how the sensor might perform in similar conditions. For more comprehensive sensor performance information, enhanced testing in a controlled laboratory environment allows air sensors to be evaluated over a range of conditions that may be challenging to capture in an ambient, outdoor environment. Additionally, enhanced testing characterizes some parameters that are difficult to test under ambient, outdoor conditions. All testers are encouraged to conduct base testing at a minimum. Enhanced testing is also encouraged although it calls for a controlled laboratory exposure chamber.

For both the base and enhanced testing, at least three (3) identical air sensors should be tested to help consumers understand the out-of-the-box performance and variation that may be present among identical sensors. As a caution, sensor performance can change over time and during the testing procedures. It may be informative to test sensors from multiple production batches provided that the sensors are the same make, model, and firmware version. A separate set of at least three (3) air sensors can be used to conduct base and enhanced testing if tests will be conducted simultaneously, but the sensors should all have the same make, model, and firmware version. If conducting both base and enhanced testing with a single set of sensors, an example approach is shown in Figure 2-1. To make the most effective use of time, the second field deployment (i.e., Field Deployment 2) in the base testing and the aging period between drift evaluation [i.e., Drift (Day1) and Drift (Day 60)] in the enhanced testing can be conducted simultaneously.

Figure 2-1. Example Approach for Conducting Base and Enhanced Testing of a Single Set of Sensors

2.1 Base Testing

Base testing consists of two (2) field deployments of PM2.5 air sensors with collocated FRM/FEM monitors for at least 30 days for each deployment. Testers may set up their own FRM/FEM monitors using guidance and information on ambient air monitoring and monitoring methods available on the U.S. EPA’s Ambient Monitoring Technology Information Center (AMTIC) webpage (https://www.epa.gov/amtic, last accessed 07/25/2020). FRM/FEM monitors should be calibrated using transfer standards that are certified and NIST traceable. Alternatively, testers may wish to develop relationships with state, local, or tribal air quality agencies to collocate sensors near regulatory FRM/FEM monitors located at existing air quality monitoring sites around the U.S. These sites can be found on the following website: https://www.epa.gov/outdoor-air-quality-data/interactive-map-air-quality-monitors (last accessed 07/25/2020).

Base testing should occur at two (2) test sites to give the greatest possible variety in PM2.5 variables including PM2.5 concentrations and particle sources, types, and size distributions. The combination of field tests should also demonstrate sensor performance over a range of T, RH, and weather conditions that will inform on sensor use across the U.S. For base testing, the sensor and FRM/FEM data will be compared at 24-hour averages which will allow testers to use a variety of technology options.

FRM/FEM monitors provide near-equivalent measurements at this time averaging interval which will lead to greater comparability in the reported results. For NSIM applications where high PM2.5 concentrations are expected (e.g., wildfire smoke applications), it is recommended that testers conduct base testing in more than two (2) locations and include sites impacted by wildfire smoke and higher PM2.5 concentrations.

The procedure in this section outlines the materials and equipment needed, site selection, set up, testing procedure, and data reporting needs to evaluate air sensor performance. To assist testers in ensuring that the requested data before and during the base testing procedure is documented, a checklist is provided in Appendix E. All information for this testing procedure should be recorded in the base testing report (template available in Appendix F). As mentioned previously, it is recommended that testers make the testing report(s) available on their respective websites to inform consumers on the testing results.

2.1.1 Materials and Equipment

The following materials and equipment are needed for this testing procedure:

Three (3) or more PM2.5 air sensors having the same make, model, and firmware version

Calibrated PM2.5 FRM/FEM monitor*

Calibrated RH monitor†

Calibrated T monitor†

Support structures and/or enclosures for air sensors (as recommended by the manufacturer)

*The FRM/FEM monitor must be calibrated on site prior to conducting base testing. Additional materials and equipment may be needed to accomplish the calibration. Calibration procedures are outlined in the Quality Assurance Guidance Document 2.12 (U.S. EPA, 2016; herein referred to as QA Document 2.12) and 40 CFR Parts 50, 53, and Appendix A of Part 58. Calibration procedures for continuous FEMs are detailed in the manufacturer’s user manual which are approved as part of the FEM designation process. If testing is conducted at an established regulatory air quality monitoring station with established calibration and quality control procedures, attach or cite the site Quality Assurance Project Plan (QAPP) to the base testing report (Appendix F). Additionally, if testing at an established site, it is recommended to confirm with the site operators (i.e., state/local agency) whether or not the FRM/FEM monitor(s) passed the monthly checks before and after testing the sensors and include this information in the base testing report.

†Meteorological monitors should be certified by the manufacturer or calibrated, installed, maintained, and audited according to quality assurance procedures outlined in U.S. EPA’s Quality Assurance Handbook for Air Pollution Measurement Systems Volume IV: Meteorological Measurements (U.S. EPA, 2008).

Some additional measurements may be useful in interpreting base testing results. These measurements include: 1) particle size distribution, 2) particle chemical composition (e.g., carbon, nitrogen), and 3) refractive index. Testers may be responsible for collecting these measurements if they choose to include them. The measurements could be particularly useful in understanding the physical reasons behind variations in sensor performance.

Preferably, measurements should be logged internally on each instrument or through a central data acquisition system. If possible, sensors should not be connected to the internet. The main reasons for this are as follows (based on Schneider et al., 2019 and experience):

Relying solely on internet capabilities may lead to data loss in the event of network outages;

It is difficult to verify the integrity of the sensor data if the sensor is connected to the internet (e.g., firmware could update during testing). Many consumers want the ability to trace and verify how data is transformed from a raw format to a final format. This can be especially problematic for sensors that rely on machine learning approaches;

Some consumers may want to use sensors where internet or cellular connections are not available. Consumers may need to know how sensor devices may work in those situations; and

If a sensor uses a nearby measurement (e.g., FRM/FEM, meteorological, other sensor data) to verify proper operation or correct the data, a consumer may not know how the sensor performs when these data are not available.

It is recognized that not all sensors can log internally or be disconnected from the internet and may stream data to a cloud platform or manufacturer server. If an internet or cellular connection is needed to operate the sensor, this information should be reported, and testers should attest that no data from collocated or nearby FRM/FEMs will be used to manipulate sensor data throughout the data processing procedure for primary testing and reporting. It is recommended that, testers issue a second report with the connectivity, enhanced data processing description, and test results if they believe that many consumers will choose to operate the sensors in such a manner.

In order to properly compare the measurements (FRM/FEM, sensor, RH, T), it is important that the data streams are time aligned. This can be done by adjusting instrument times to a common standard clock [e.g., National Institute of Standards and Technology (NIST) time], carefully checking time stamps when devices are started and stopped, and/or using a common data logger. If data from any instrument is reported as an average, it is also important to understand if the data average is ‘time ending’ or ‘time beginning’. For example, when logging hourly averages, the 07:00 time stamp may reflect data collected between 06:01-7:00 (time ending) or 7:00-7:59 (time beginning). This information should be considered when time aligning data.

For base testing, the sensor and FRM/FEM data will be compared at 24-hour averages which will allow testers to use a variety of technology options. FRM/FEM monitors provide near-equivalent measurements at this time averaging interval which will lead to greater comparability in the reported results. It is recommended that testers issue a supplemental test report of 1-hour data but should note that there may be unquantified variation between FEM instruments at this time averaging interval.

2.1.2 Selecting and Setting Up a Test Site

Potential consumers need information on how well they might expect sensors to perform in the area in which they intend to make measurements. Therefore, testing a sensor’s performance over a range of conditions (e.g., T, RH, pollutant concentrations) would be most informative to the widest variety of consumers. Table 2-1 provides recommended criteria for the test sites.

Table 2-1. Test Site Selection Criteria

Base Testing Plan

Location(s) Season Goal 24-Hour Average PM2.5 Concentration (for at least one day)

Two test sites Site 1 Climate Region 1 25 g/m3

Site 2 Climate Region 2 g/m3

As shown in Table 2-1, it is recommended that base testing be conducted in a minimum of two (2) locations in two (2) different climate regions across the U.S. (Figure 2-2). Although all test results are valuable, testing under a range of PM2.5 concentrations is most informative. Therefore, a goal of at least one day of the 30-day testing period with a 24-hour average PM2.5 concentration of at least 25 g/m3 is suggested. It is acknowledged that PM2.5 concentrations across the U.S. may be unpredictable and even sites that are expected to exceed 25 g/m3 may not achieve this level. A level of 25 g/m3 is recommended as a goal as this will help ensure statistics are comparable across sites and that testing does not result in a low R2 due to low PM2.5 concentration ranges. For NSIM applications where high PM2.5 concentrations are expected (e.g., wildfire smoke applications), it is recommended that testers conduct base testing in more than two (2) locations and include sites impacted by wildfire smoke and higher concentrations.

PM2.5 concentrations and particle sources, types, and size distributions can vary across the U.S. as does climate itself; thus, tests conducted in separate climate regions (Figure 2-2) may offer performance information over a range of PM2.5 variables and meteorological conditions. When choosing test sites, it is recommended to select sites located in climate regions that are not adjacent for the greatest possible variation in PM2.5 variables.

Figure 2-2. U.S. Climate Regions (https://www.ncdc.noaa.gov/monitoring-references/maps/us-climate-regions.php, last accessed 07/29/2020)

Based on historical data, there are a number of sites across the U.S. that should offer the conditions shown in Table 2-1 during certain times of the year. If using an existing ambient air monitoring network site [e.g., National Core Network (NCore), Chemical Speciation Network (CSN), State and Local Air Monitoring Station (SLAMS)], a tester can examine historical air quality data, found on the U.S. EPA AirData website (https://www.epa.gov/outdoor-air-quality-data, last accessed 07/25/2020), to see if/when a site is likely to meet the criteria. If using another site that does not have historical air quality data, consult data from the nearest regulatory monitoring site to determine if the site(s) is likely to meet the criteria in Table 2-1. Additional details outlining how to identify sites that will likely meet this criterion and how this criterion was determined are detailed in Appendix B.

Take the following steps when selecting and setting up a test site:

1. Select a test site(s) that meets the criteria in Table 2-1. If using an existing ambient air monitoring network site, record the Air Quality System (AQS) site ID.

2. Record the calibration or certification date for the T and RH monitors and attach a copy of the calibration certificate(s) to the base testing report (Appendix F).

3. If not already set up at a test site, install the FRM/FEM, T, and RH monitors at the test site such that the sampling probe inlet or monitoring path meets the siting criteria in Table 2-2.

Table 2-2. Sampling Probes or Monitoring Path Siting Criteria

Description Distance (meters)

Height from ground 2 to 15

Horizontal and vertical distance from supporting structures

Distance from trees *

Distance from roadways †

*Should be greater than 20 meters from the tree(s) dripline and must be 10 meters from the dripline when the tree(s) act as an obstruction (see 40 CFR Part 58, Table E-4 of Appendix E).

†The roadway average daily traffic, vehicles per day determines the minimum distance (see 40 CFR Part 58, Table E-1 of Appendix E).

2.1.3 Setting Up the Air Sensors

Take the following steps when setting up the air sensors for base testing:

1. Verify that there are at least three (3) PM2.5 air sensors of the same make, model, and firmware version. The firmware version should not be updated during the testing. Use sensors in the same condition as they were received from the manufacturer and do not modify any manufacturer calibration(s).

2. Disconnect the sensors from internet access. Ideally, data should be stored locally on the sensors (such as on a local data card) or on a common datalogger. If an internet or cellular connection is necessary for sensor operation, data from either collocated or nearby FRM/FEM monitors should not be used by the sensors during this testing procedure.

3. In the base testing report (Appendix F), record information about the equipment and set-up, to the extent possible, including the following:

Parameters measured (e.g., pollutant(s), T, RH, dew point) and units Sampling time interval (e.g., 1-minute, 15-minute, 1-hour, 24-hour) Data storage and transmission method(s), including:

o Where the data are stored (e.g., local data card, transmitted to cloud system) o If applicable, where the data are transmitted (e.g., manufacturer’s cloud server) o Form of data stored (e.g., raw data, corrected or cleaned data)

Data correction approach (if applicable), including:

o Procedure used to correct the data including: [a] how the data are corrected (e.g., manufacturer derived multilinear correction), [b] variables used to correct the data (e.g., RH, T), [c] where the correction variable(s) comes from (e.g., on-board RH sensor), and [d] how the data are validated or calibrated (e.g., RH sensor is calibrated by the manufacturer) o If the way data are corrected does not change and is static, record this information and any mathematical approaches used o If the way data are corrected changes or is a dynamic process, record the following: (a) when the process changes, (b) why the process changes, (c) how/where changes are recorded, and (d) how the correction method is validated

Data analysis/data correction scripts (e.g., Jupyter Notebook, R Markdown) Location of final reported data and its format (e.g., website shows raw data and corrected data on user interface, data provided as .csv, expanded definitions of data headers)

4. Install air sensors at the test site using the ideal setup guidance summarized in Table 2-3.

5. Include photographs that clearly show the entire equipment setup at the test site, and document distances, in the base testing report (Appendix F).

Table 2-3. Guidance on Air Sensor Setup at Testing Site

Recommendations Cautions

Mount sensors within 20 meters horizontal of the FRM/FEM monitor

Mount sensors in a location where they are exposed to unrestricted air flow

Ensure the air sampling inlet for the sensors are within a height of ± 1 meter vertically of the air sampling inlet of the FRM/FEM monitor

Mount identical sensors ~1 meter apart from each other

If necessary, install sensors within a weather-protective shelter/enclosure that maintains ample air flow around the sensor (as recommended by manufacturer)

Do not place sensors near structures/objects that can affect air flow to the sensor OR block the sensor air intake (e.g., against a wall, near a vent, or on the ground blocking the inlet)

Do not place sensors near structures/objects that can alter T or RH near the sensor (e.g., vents, exhausts)

Do not place sensors near sources/sinks that can alter pollutant concentrations (e.g., idling cars, smoking)

Do not place sensors in locations with risk of vibration, electrical shock, or other potential hazards

2.1.4 Conduct Base Testing

The step-by-step procedure for conducting the base testing is as follows:

1. Record the calibration date of the FRM/FEM monitor. Calibration should be conducted after the monitor is in place at the test site, not before. If the FRM/FEM monitor requires calibration, follow the procedures as outlined in QA Document 2.12 and 40 CFR Parts 50, 53, and Appendix A of Part 58. Calibration procedures for continuous FEMs are detailed in the manufacturer’s user manual which are approved as part of the FEM designation process.

2. Verify that the system(s) for data logging and data storage will collect all equipment data and store it in a way that can be accessed later. Make sure that there is enough storage capacity available to prevent older data from being overwritten and allow new data to be stored.

3. Use sensors in the same condition as they were received from the manufacturer and do not modify any manufacturer calibration(s). The firmware version should not be updated during testing.

4. Provide a warm-up and stabilization period for all equipment as specified by the manufacturer.

5. Verify that all equipment is reporting measurements.

6. Conduct a one-point flow rate verification check on the FRM/FEM monitor (see procedures outlined in QA Document 2.12 and the manufacturer’s user manual) and record the date of the check.

7. Allow all equipment to run for at least 30 consecutive days. All equipment should be running during the same time period to allow for comparable results.

8. Follow the manufacturer’s maintenance recommendations, as applicable, for all equipment (e.g., sensors, FRM/FEM) throughout testing. Record and report all maintenance or troubleshooting performed, including dates/times, on the instruments (e.g., power cycling, FRM/FEM flow rate verification check).

9. Record and report the rationale for missing or invalidated data. For a full 30 consecutive day run, , with all instruments reporting is ideal. This corresponds to all equipment reporting at least 23 valid 24-hour pairs of time-matched data points over the course of the 30-day deployment.

a. If the sensor fails irreparably before the 30-day deployment is complete, another sensor should not be substituted. In addition, the sensor should not be sent back to the manufacturer for repairs without restarting the testing procedure. A preliminary report could present results with documentation as to why the sensors failed as these details may be useful to potential consumers. Testing can be restarted with three (3) sensors.

b. Occasionally, low uptime or a deployment period of less than 30 days might occur, for example, due to an unplanned electrical outage or weather event (e.g., hurricane, tornado). In those instances, the dates and reasons for missing data should be recorded. In these scenarios, ideally testing would continue/resume until at least 23 valid 24-hour pairs of time-matched data points are collected.

c. If data from any piece of equipment is not available during each 24-hour sampling period, record and report the reason (e.g., outage, maintenance).

d. Additionally, if any of the data are invalidated due to QC criteria, record the reason and criteria used. FRM/FEM instruments have more established QC criteria (visit the AMTIC webpage at https://www.epa.gov/amtic, last accessed 07/25/2020). QC criteria for the sensor may be available from the manufacturer or may be developed as part of these tests. General information on how the U.S. EPA manages data quality can be found at https://www.epa.gov/quality (last accessed 12/07/2020). Reporting QC criteria for the sensor is strongly recommended as this information is beneficial for consumers.

10. Select a test site for the second field deployment…