140G0321P0016 Amendment I - Q-A and Publication_0001.pdf

140G0321Q0016 P. 2 of 20

140G0321Q0016 – Questions and Answers

1. The SOW states that there are currently approx. 1,113 seismic monitoring stations, with a plan to extend to 1,615 stations. The SOW notes that in some cases data links from the monitoring stations are aggregated onto intermediate relay and larger backhaul links. Approximately how many intermediate relay links are currently in the network? Approximately how many backhaul links are currently in the network?

Approximately 15 of each in each of the Southern California, Northern California, and Pacific Northwest Seismic Networks.

2. What are the wireless communication technologies used to connect the seismic monitoring stations to the network?

IP radios, including Freewave and Xetawave. Most are in the ISM 900 MHz band. Some programmable radios in the VHF band are used in the Pacific Northwest. We have tested federal licensed 165 and 900 MHz bands. Data throughput was strongly limited in 165 MHz because of the limited available channel width allocation. Radio pairs are also operating in licensed 400 and 220 MHz. In general, licensed solutions would be preferred, but obtaining licenses, limitations on channel width, and crowding were decisive negatives for tests of licensed 900 MHz band. Other wireless communication technologies include cellular modems and VSAT.

3. What are the wireless communication technologies used to connect the intermediate relay nodes to the larger backhaul links?

Main method for smaller throughput requirements are dedicated IP radios in ISM bands including 900 MHz and 5.8 GHz. Licensed 4.5 GHz Cambiums are being installed in some links for backhaul applications in the Bay Area. Some larger throughput links utilize licensed 4.9 GHz contributed by University partners. At some sites, a cell modem is being used to collect stations for return over the commercial mobile cellular link.

4. For links between seismic monitoring stations and the core network, or between seismic monitoring stations and intermediate relay nodes, what is the maximum link distance achievable using currently implemented technology?

Each network has “maximum link distance” cases, but distance comes at the expense of bandwidth. These tend to be vulnerable to interference and inferior performance during peak demand. The article by Stubailo et al. (ref. question 9 below) gives details.

Links less than 15 mi with clear lines of sight tend to be fairly reliable, but some longer shots perform well at lower modulations in rural areas.

5. For links between intermediate relay nodes and the core network, what is the maximum link distance achievable using currently implemented technology?

140G0321Q0016 P. 3 of 20

Links of 15-25 miles are generally reliable. Longer distances utilizing lower modulations may be used where no good alternative exists. Generally, the longer distances are only working with single sites where overall throughput demand is lower.

6. Related to the discussion of cloud technologies and processing latency, what is the approximate time it currently takes for the detection algorithms to determine an earthquake location from the received data?

Approximately 0.5s.

7. Related to the discussion of cloud technologies and processing latency, approximately how much data (in bytes) does the current network ingest for processing?

Uncompressed data from a seismic station is roughly 19 kbits/sec. Compression reduces this during quiet background periods. During strong shaking the rate increases because data do not compress as well, or not at all in severe cases. State of health data add 10- 15% to the total. See Stubailo et al. 2020 for details. No terrestrial operations center sees all the seismic data because of its volume. Rather, the networks each receive around 35% of the total seismic data, the great majority from their respective regions.

Between operations centers only smaller-volume summary messages are exchanged.

See the 2017 Telemetry Report, provided with the SOW, for further explanation.

8. Related to the wireless communication technologies currently in use; what methods, if any, have been implemented to resolve data collisions resulting from "hidden node" scenarios where two geographically distant sensor sites are attempting simultaneously to transmit data to an intermediate relay node?

Geographically, most sensor sites fit the description for the “hidden node” condition (sites not “seeing” other sites). The main resolution method for collisions is to employ a 1:1 relationship of transmit to receive radios. Packets are combined in switches or routers onto the backhaul mode. This 1:1 TX:RX pairing increases link capacity and reduces latency. In approximately 5% of cases, sensor sites share an access point (AP) and could be susceptible to the “hidden node” condition. Collision potential is helped by the low packet rate from stations – one to a few packets/second. Radio links operate in separate frequencies, so are not interfering on the RF side. Most modern AP radios (e.g., Cambiums) have the capability to allocate time slices (effectively, TDMA), to improve SNR. For the few cases where an AP is shared, the contribution of the hidden node operation to reducing peak packet throughput is not well known.

9. Would you please provide a full text copy of the paper "Stubailo, Igor, Alvarez, Mark, Biasi, Glenn, Bhadha, Rayomand, and Haukson, Egill (2020) Latency of Waveform Data Delivery from the Southern California Seismic Network during the 2019 Ridgecrest Earthquake Sequence and Its Effect on ShakeAlert, Seismological Research Letters, doi: 10.1785/0220200211" referenced in Appendix B?

Latency of Waveform Data Delivery from the Southern California Seismic Network during the 2019 Ridgecrest Earthquake Sequence and Its Effect on ShakeAlert Igor Stubailo*1, Mark Alvarez2, Glenn Biasi2, Rayomand Bhadha1, and Egill Hauksson1

Abstract

Cite this article as Stubailo, I., M. Alvarez, G. Biasi, R. Bhadha, and E. Hauksson (2020). Latency of Waveform Data Delivery from the Southern California Seismic Network during the 2019 Ridgecrest Earthquake Sequence and Its Effect on ShakeAlert, Seismol. Res. Lett.

XX, 1–17, doi: 10.1785/0220200211.

The occurrence of the 4–6 July 2019Mw 6.4 andMw 7.1 Ridgecrest earthquake sequence provided the first full-scale test of the network and telemetry readiness of the Southern California Seismic Network (SCSN), to support the ShakeAlert earthquake early warning (EEW) system in California. ShakeAlert is a U.S. Geological Survey (USGS)-led collabo-ration to detect earthquakes and, when possible, to alert the public before the arrival of the strongest shaking. The SCSN performed well in its regional monitoring role for both the 4 JulyMw 6.4 and the 6 JulyMw 7.1 earthquakes. In the EEW role, it provided timely delivery of 5 s of P-wave data to ShakeAlert, which issued its first alert 6.9 s after origin time. Data delivery at peak data volumes for many stations exhibited some latency, and, as a consequence, some data arrived too late for analysis by one of the EEW algo-rithms.We find that the average link bandwidth for each stationwas sufficient, because all waveform data were delivered automatically to the archive, but link capacity for many stations was insufficient for peak demand. We describe the performance of the data telemetry for the sequence, including cellular, radio, hybrid, and backhaul systems. Cellular-based telemetry systems maintained low latency throughout strong shaking and after, but some stations, even at great distances, experienced subsequent brief increases in latency. Performance of radio links depended mostly on the signal strength of the link, with short-distance direct shots to high-bandwidth backhaul sys-tems showing no latency impact, whereas stations on some long distance or marginal quality links suffered latencies of tens or hundreds of seconds. Improvements are being implemented to move telemetry links onto USGS and partner high-bandwidth microwave systems, and to reduce dependency on less robust long-distance radio shots.

Introduction The 2019 Ridgecrest earthquake sequence began in earnest with an Mw 6.4 earthquake on the morning of 4 July (10:30 a.m. PST, 20:30 UTC). It was the largest event to occur inside the Southern California Seismic Network (SCSN) in 20 yr. This event was widely felt in the southern California area, including most of Los Angeles County. The ShakeAlert earthquake early warning (EEW) system (Given et al., 2018) detected the event and issued an alert in 6.9 s (Chung et al., 2020). A largerMw 7.1 event followed 31 hr later (5 July, 8:19 p.m. PST; 6 July, 03:19 UTC), some 200 s after an Mw 5.1 foreshock. An active after-shock sequence followed, with multipleMw 5 events in the first few days (Hauksson et al., 2020). Besides shaking southern California, these events were the first full-scale readiness test of the SCSN for EEW before the October 2019 roll-out

ShakeAlert EEW system. We review the results of this test in this article.

The Ridgecrest sequence occurred in the northeast part of the SCSN. The SCSN is jointly managed by Caltech and the U.S.

Geological Survey (USGS), under the netcode “CI” (Fig. 1, regional map of stations). The California Geological Survey (net-code “CE”), the National Strong Motion Project (netcode “NP”), and other organizations also share real-time data streams trans-mitted to SCSN, as discussed by Hellweg et al. (2020).

1. Caltech Seismological Laboratory, Pasadena, California, U.S.A.; 2. U.S. Geological Survey, Pasadena, California, U.S.A.

*Corresponding author: stubailo@caltech.edu

© Seismological Society of America

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 1

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf by 15248 on 21 December 2020 https://doi.org/10.1785/0220200211

Evaluating telemetry performance for stations other than the CI network is outside this study.

To estimate earthquake magnitude and location, ShakeAlert relies on two algorithms. One is called Earthquake Point-source Integrated Code (EPIC, Chung et al., 2019). EPIC mod-els the earthquake as occurring at a point. Amplitudes are mea-sured from up to the first 4 s (almost 5 in practice), after triggering by P-wave arrivals, and used in a regression for station magnitude. With four or more station magnitude estimates, a weighted regression for earthquake magnitude is formed (Chung et al., 2020). If less than 4 s of data are available from a station, as from a data outage, the station con-tribution to magnitude is usually an underestimate. The second ShakeAlert algorithm employed during the Ridgecrest sequence is called FinDer (Böse et al., 2012, 2018).

FinDer is designed to measure large ruptures, by comparing observed ground motion to predictions from template ruptures. The published alert magnitude from ShakeAlert combined the EPIC and FinDer estimates, with the estimate weighted toward EPIC for magnitudes smaller than 7 (ShakeAlert has since changed to prefer FinDer magnitude estimates for Mw 6 and larger events). Earthquake magnitude estimates in FinDer evolve with time, as the strong shaking zone expands. Latent or missing data thus also affect the time-liness and accuracy of FinDer alert updates. Chung et al. (2020)

Figure 1. Station coverage of the Southern California Seismic Network and partners contributing to ShakeAlert. Symbols dis-tinguish CI, California Geological Survey (CGS), National Strong Motion Project (NSMP), and other contributors. The 8 GHz microwave backbone link operated by the U.S. Geological Survey (USGS) is shown in blue, with data collection centers colored in black. Brown lines identify the known fault systems. The location of the magnitude 7.1 earthquake near Ridgecrest is shown by the orange star.

2 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf by 15248 give algorithm-related details, as they applied to the Ridgecrest sequence.

Delays in data flow were specific to stations and their local telemetry links. We store metadata for the SCSN telemetry links in the Station Information System (Yu et al., 2018). At the time of the earthquake sequence, there were no systematic network outages on the microwave, fiber, or dedicated (T1) backhaul systems that support real-time data delivery. There also did not appear to be any issues at the SCSN operations center affecting acquisition and data provisioning to ShakeAlert. Where ShakeAlert had difficulty with earthquake magnitude and alert area, the principal problem appears to have been data latency, although EPIC had an additional issue with algorithm-related magnitude saturation (Chung et al., 2020). The ShakeAlert codes were able to develop reasonable estimates of magnitude, with data played back in expected times of arrival (Chung et al., 2020).

There is some variation in seismic instrumentation in the SCSN, with the instrumentation consisting of class A sensors and digitizers (Working Group on Instrumentation, Siting, Installation, and Site Metadata, 2008). In the CI network, sta-tions with both broadband velocity and acceleration sensors use Kinemetrics Q330 dataloggers. At accelerometer sites, most dataloggers are Kinemetrics Basalt and Obsidian models.

At some accelerometer sites, a vertical velocity component (L4-C sensor) is added to contribute phase picks for small earthquake locations.

There is a greater variety among telemetry systems.

Approximately, 40% of the SCSN stations return data via Verizon cellular modems. Bandwidth varies station by station, depending primarily on the quality of the cellular connection.

Based on actual speed tests, 83% exceed 100 kbps, and 75% exceed 1 Mbps (Table 1). Approximately, 26% of the SCSN stations telemeter by microwave radio in the 900 MHz band.

As with the cellular systems, bandwidth is a function of the connection quality and link length. About 10% of such stations transmit data directly to the USGS microwave backbone sys-tem. The data streams for the remaining stations are gathered a few at a time onto high-capacity backhaul lines, such as dedicated T-1 circuits for transmission to SCSN acquisition computers in Pasadena, California. The rest of the SCSN stations use other systems, including direct Internet, various private cooperative networks, and satellite-based systems (very-small-aperture terminal [VSAT]).

The regional processing center for the SCSN is in Pasadena.

The data are recorded on acquisition computers and converted to Earthworm packets (Hartog et al., 2019). These packets are broadcast on a multicast network to which both EEW and regular network-processing applications “listen.” Software to measure data-delivery time, or “transport” latency, is another client of this multicast network. We discuss how latency is measured in a later section.

During the Ridgecrest sequence, SCSN systems for data acquisition and sharing appear to have handled the increased load, without dropping packets or other signs of overloading.

The Advanced National Seismic System Quake Management System (AQMS) (Hartog et al., 2019) located earthquakes in a timely manner, including the main aftershocks of the large Ridgecrest events. Unlike ShakeAlert, AQMS algorithms are relatively robust to short-term data latencies or isolated station unavailability. The fact that data-receiving systems kept up with data flow means that SCSN computer systems contributed little to data-delivery latency, and that the main delays occurred upstream, at the datalogger or in the telemetry systems. ShakeAlert software also continued to run as pro-grammed and without software issues affecting its performance.

However, the ShakeAlert performance for earthquake detection was sensitive to data timeliness issues (Chung et al., 2020).

Latency Measurements as a Forensic Tool Data latency is the time it takes for data to arrive at the data center after being captured by the station. To analyze data timeliness, about 3 yr ago, SCSN began recording a measure of station data latency under the channel code “L1Z.” The latency recorded on the L1Z channel is a one sample per sec-ond derived time series in which the “amplitude” of the series is the time difference between the time when the last sample of

TABLE 1

Mean, Median, and Range of BandwidthMeasurements in kbps from a Representative Set of Seismic Stations in the Field to a Server at the Southern California Seismic Network (SCSN) Regional Data-Processing Center, Measured on 11 December 2019

Description Maximum (kbps) Minimum (kbps) Mean (kbps) Median (kbps)

Very-small-aperture terminal (VSAT) (eight sites) 1023.4 232.4 571.9 427.9

Cellular (94 sites) 104535 14.8 10005.9 4877.2

Radio (54 sites) 10203.6 8.2 794.2 56.2

Ethernet, fiber (19 sites) 11430.8 227.6 7449.0 8135.1

The number of stations in each set is listed in the first column. For this measurement, seismic data flow was temporarily stopped for each station during the measurements. The results for the cellular connections were calculated for the cellular telemetry in isolation, that is, composite connections (like radio and cellular) were not used.

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 3

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf the data packet was recorded and the time when the first sam-ple of the packet is received by the multicast ring of the data center. The latency measurement includes filtering, packetiza-tion, and output delays on the datalogger, delays in radio and/ or IP packetization, and delays in acknowledgment and deliv-ery. Because data handling by the datalogger and receiving-end acquisition software are very rapid (milliseconds), the L1Z measurement can be interpreted as the delay of data within the telemetry system, that is, the packet transport latency.

Other measures of latency can be defined using the recorded versus arrival times of the middle or the end of a packet. To enable unambiguous latency measurements, we changed the data delivery for the SCSN to occur in packets of a standard 1 s length. The L1Z channel enables monitoring the state of health of the telemetry and allows us to identify anomalous conditions as they affect time to data delivery.

Figure 2 shows a variety of signatures of data latency (L1Z observations) from around the time of the Ridgecrest main-shock. Let us consider this example in detail. The horizontal axis spans 800 s. The lowermost trace is a seismic recording of the Ridgecrest mainshock, pre-ceded by an Mw 5.0 foreshock.

Station CI.CLC is representative of a station with low spare bandwidth. The Mw 5.0 fore-shock started a latency episode, and the mainshock amplified it (Fig. 2a). Station latency even-tually recovered on its own, but data were too late to con-tribute to the initial mainshock alert and tens to hundreds of seconds late for over 4 hr fol-lowing the mainshock. Station CI.DTP had no problem at the Mw 5.0 foreshock but became up to 35 s latent over a period of 50 s following the mainshock. It recovered 150 s after the mainshock. The right slope of latency pulses is a mea-sure of bandwidth in excess of instantaneous demand available to resolve a data backlog.

Station CI.LRL experienced sev-eral delays of 8 s or less, but consistently recovered quickly.

CI.SVD appears more sensitive, with interruptions of 12 s or more at the foreshock, mainshock, and many larger aftershocks. Finally, station

CI.SDD exemplifies a high-bandwidth connection in which there is no evidence of shaking-related latency.

Considering latencies for station CI.LRL around the time of strongest shaking, we find that data became significantly latent, by 5–8 s, starting 6 s after shaking onset (see Fig. 3). For the ShakeAlert EPIC algorithm, this delay affects the timeliness of the magnitude estimates and alert generation. At the time of the Ridgecrest sequence, the FinDer algorithm only analyzed the data that was less than 5 s latent (Chung et al., 2020); this has been changed in the latest FinDer release.

Increases in Data and Packet Rates during Shaking Seismic data volume, in the sense of samples per second or bytes per sample, does not depend on time or seismic activity.

Thus, the uncompressed data rate, about 20 kbps for a six-channel datalogger at 100 samples per second, plus overhead provides a rough estimate of minimum capacity required of a telemetry system. With a compression algorithm, the seismic

7 × 106

7 × 106

0 200 400 600 800

Time (s)

CI.CLC.HHZ

Mw 5.0 Mw 7.1

Aftershocks Aftershocks

0.1

0.2

CI.SDD.L1Z

246.9 km

14 CI.SVD.L1Z

190.9 km

30 CI.DTP.L1Z

61.0 km

8 CI.LRL.L1Z

34.0 km

CI.CLC.L1Z

4.3 km La te nc y (s

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2. Seismic recording of theMw 5.0 Ridgecrest foreshock,Mw 7.1 mainshock, and aftershock sequence with latency measures (L1Z) for five stations that illustrate different latency evolution during the events. (a) Station CI.CLC uses a radio connection and started to exhibit unexpectedly large latencies with theMw 5.0 foreshock, with values as large as 800 s by the end of the displayed time window. (b) Station CI.LRL (uses radio) experienced multiple brief periods (tens of seconds) of increased latency, although never above 8 s. (c) Station CI.DTP (uses radio) became 35 s latent at the time of the immediate aftershocks, but recovered over 150 s. (d) Station CI.SVD (uses cellular modem) experienced similar latent pulses to CI.LRL, but with amplitudes of 10–15 s, and spikes in latency at many aftershocks. (e) Station CI.SDD is on a fiber-optic line and experienced no latency effects. The distances from the stations to the magnitude 7.1 mainshock location are given below the station names. (f) The vertical axis for the seismic recording is given in counts. The slope on the right side of pulses in the top four panels is a measure of recovery rate that functions as a proxy of available telemetry bandwidth. Circles with × mark the P-wave arrival times. The scaling of the vertical axis varies by trace to better show the latency variations.

4 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf data are reformatted in a way that typically reduces the average transmitted bit rate by half to ∼10 kbps. Strong seismic shak-ing decreases the efficiency of the compression (Steim and Reimiller, 2014), and the data volume approaches the volume of the uncompressed data, which requires larger data-transmission rates. Figure 4 illustrates this increase for seismic data at station CI.LRL. Packet count increases, because each packet contains a fixed number of bytes of compressed data.

Thus, more packets are required to maintain 1 s net delivery rates from the datalogger. This increases the effective packet throughput demand on telemetry. The CI.LRL data and packet rates increased about threefold, right after the Mw 7.1 mainshock, and rates remained 40% above pre-event levels well into the aftershock sequence (Fig. 4). Despite this impressive increase, the peak data rate of 2800 bps or 22.4 kbps is consistent with peak bit-rate expectations and consistent with the measured capacity of most SCSN telemetry links (Table 1).

Packet and data rate increases can cause telemetry links with marginal surplus capacity to begin to fall behind.

Many stations in the Ridgecrest area fell into this category.

The dataloggers prioritize completeness over timeliness of delivery and so queue data in a first-in, first-out buffer, and oldest data are delivered first. Until this backlog is resolved, data will appear latent. Dataloggers also “know” whether pack-ets are received at the SCSN data center, because the acquis-ition computers return brief acknowledgment messages. If radio packets containing seismic data time out or are lost in transmission, seismic data are not lost; the dataloggers will resend when the link returns, again oldest data first. The time length for latency to recover depends on the available band-width of the link and severity of the backlog. Strong earth-quake shaking during latency recovery may delay the sta-tion’s return to availability (timely data delivery) for ShakeAlert, if it causes an increase in data generation and demand on telemetry throughput.

Dataloggers in the SCSN have an additional packet retransmission capability called “burst mode” that, if enabled, can affect latency recovery time following a telemetry outage. When the datalogger senses that the con-nection is restored, it can begin to send the entire backlog of data as quickly as possible, in an effort to get back to current (low latency) data. For a com-promised or limited bandwidth telemetry system, this “burst” of packets can be overwhelming and lead to further delays.

We have not pursued the exact contribution of burst mode transmissions to Ridgecrest data latencies, because it cannot readily be distinguished from latencies due to other causes.

For dataloggers that support a burst mode (e.g., in SCSN, the Kinemetrics Q330), the capability can be useful in a program to test telemetry throughput.

Data Latency during the 2019 Ridgecrest Sequence The seismic network configuration at the time of the Ridgecrest sequence is shown in Figure 5. Colors applied to the dots representing the seismic stations indicate the maxi-mum data latency observed at the station during the first 30 s after the mainshock. Symbol shapes are used to indicate telemetry type. Fine lines indicate station-to-station or station-to-hub connections to a shared telemetry backhaul link.

Performance of shared cellular modem links at WRV2 and SRT is discussed in a later section. Stations with telemetry that developed significant latency are concentrated around the mainshock (rectangular northwest-trending patch in Fig. 5).

Stations in this area experienced strongest shaking and signifi-cant increases in packet flow because of decreases in data-com-pression efficiency.

The latency impact of the Mw 7.1 Ridgecrest mainshock is shown in Figure 6. Before the Ridgecrest earthquake, 91% of seismic stations in the SCSN had average data latencies under 2 s. In the first 30 s after the mainshock, this value drops to 56%

280 300 320 340 360 380 400

Time (s)

CI.LRL.HNZ

5 × 106

–5 × 106

–2 × 105

2 × 105

CI.LRL.HHZ

CI.LRL.L1Z

(a)

(b)

(c)

Figure 3. (a) Expanded view of latency for station CI.LRL from Figure 2, shown with the

(b) mainshock broadband and (c) accelerometer data. The vertical axes for the latency and seismic recordings are given in seconds and counts, respectively. The ground motion exceeded the recording limits for the broadband sensor for about 20 s. The top trace shows that only ∼6 s of data from LRL are received without significant latency after the P-wave arrival. The P-wave arrivals are marked with × circles.

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 5

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf for stations, as the P-wave travels past these stations (Fig. 6b).

Interestingly, small increases in latency affecting at least 25 sta-tions are observed outside the range of the P-wave at 30 s (Fig. 6c). The cause for a latency increase before any shaking had arrived at these stations is not known.

A detailed timeline of latency for stations nearest the Ridgecrest mainshock is shown in Figure 7. The stations are sorted by distance from the hypocenter. Colors indicate severity of latency as measured by L1Z. Most stations provided 2–5 s of low-latency data after P-wave arrival (X’s). Only seven of 20 sta-tions within 80 km from the epicenter provided low-latency data for at least 20 s, the approximate duration of the mainshock rup-ture. Data from three stations, CI.CLC, CI.WRC2, and WCS2, were already significantly latent before the mainshock occurred.

Figure 7a shows that latencies for these stations had not yet recovered from anMw 5 foreshock 200 s before the mainshock.

Latency effects from the Mw 5 foreshock appeared on fewer stations, consistent with a smaller radius of significant shaking. The Mw 6.4 event (Fig. 7c) caused latency effects similar in scope to the Ridgecrest mainshock to a dis-tance of ∼45 km, but were somewhat less severe at larger distances.

Latency Performance by Telemetry Type In the following subsections, we consider latency performance of specific telemetry types.

The review highlights strengths and weaknesses brought to light by the Ridgecrest sequence.

Laurel hub latency The Laurel hub is a point where data from nine sites are received over separate radio links. These data are then com-bined with a seismic station at the hub (CI.LRL) and retrans-mitted over a single, high-capacity radio to the Navy Geothermal Program Office (GPO), where a land-based T-1 circuit connects back to the SCSN operations center.

Figure 8 shows telemetry latencies for CI.LRL and five stations that connect to it. Vertical-component broadband velocity sensor data are also shown for station CI.CCC.

Ground motions exceeded the sensor recording capacity dur-ing the strongest shaking. Relations among latency observa-tions at the six stations provide some insight into their causes. Latencies on channel CI.LRL.L1Z may be interpreted as delays in the high-speed trunk connection from Laurel hub to the GPO under the data load of six seismic stations.

Very near the onset of strong shaking (circled “X”), this con-nection developed latencies of 5–8 s but recovered after 30 s for a period near 330 s on the plot with <1 s latencies. The small maximum latencies at CI.LRL and the recovery indicate that the T-1 circuit from the GPO, and the high-speed connector to it stayed up, with a maximum outage of under 8 s. CI.LRL developed two additional, shorter-duration latent periods, then returned to <1 latencies, thereafter. Interruption of the trunk

(a)

(b)

(c)

Figure 4. (a) Increase in packet and data output by the datalogger during periods of (b) elevated ground shaking and (c) the resulting latency. When shaking intensifies, both the packet and data rates increase, because of the decreased efficiency of compression. Peak data rate of 2800 Bps translates to 22.4 kbps of delivered data. Actual telemetry demand is higher, because of packet framing and delivery overhead. Even after 1.5 hr, the packet rate is 50% above the pre-event rate.

The higher data and packet rates cause increased demand on all telemetry components. (c) The data latency, approximately, correlates with strong ground shaking that is correlated with increased packet rate.

6 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf line also interrupted stations CI.CCC and CI.WBS. They recovered over periods of 105 and 80 s, respectively. The cor-relation of latency increases and recovery with CI.LRL.L1Z is approximate, but consistent with sound telemetry links from the respective stations. Stations CI.DTP and CI.CCA are char-acteristic of weaker links, with less capacity to recover. Both exceeded 5 s latency only a few seconds after the P-wave arrival, and both required over 150 s to recover completely to premainshock latency values. Station CI.WRC2 began being latent with theMw 5 foreshock (not shown), and latencies only increased. This pattern reflects a marginal radio link.

Latency of cellular connections Many cellular-connected sites registered short interruptions of data-packet delivery near the time of large events (Figs. 9–11).

Three types of cellular telemetry station latency are shown in Figure 9. Stations CI.ISA, CI.WHF, and, CI.WOR exemplify individual stations on cellular modems, in lightly populated areas. The short periods of latency at CI.WOR and CI.WHF did not start with the P wave, but during the strongest phase of S waves over 20 s later. The shorter latent period at CI.ISA occurred some 80 s after the P wave, thus well after the strong-est shaking had passed. CI.WMF relays by radio to shared cellular telemetry at CI.WRV2 (Fig. 5). CI.WMF is closer to the Ridgecrest epicenter, but the brief latent period it devel-oped was even later, relative to the time at strongest shaking (CI.WMF waveform, Fig. 9b,f). CI.SRT delivered data with low latency for 26 s, after P-wave arrival and throughout the

Radio 900 MHz, 4.9 GHz

Cell

VSAT

Fiber dwp frad ether MW

Figure 5. Peak latency values for the first 30 s after the mainshock for the SCSN stations active during the Mw 7.1 event near Ridgecrest. The telemetry type for each station is given by the symbol shape, whereas the latency values are shown by the symbol color. Dark blue symbols mark the hubs that were col-lecting data for transferring onto the USGS microwave trans-mission line (medium-blue line with black squares), and blue lines display the radio telemetry connections. Details for several sites and hubs (Geothermal Program Office [GPO], Laurel hub) are shown in later figures. To reduce clutter, we label the stations with their station names only, the network code CI is omitted.

Note that the Laurel hub symbol overlaps with CI.LRL and is underneath the yellow triangle. The site CI.SLA colored in red is connected with a Frame Relay line with a 56 kbps max throughput. The P-wave front after 30 s is colored in purple, and the S wave is colored in white. The light-shaded area under Ridgecrest shows the approximate rupture extent for theMw 7.1 event.

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 7

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf strongest shaking (Fig. 10), despite being very close to the epi-center. Together these sites show that strong physical shaking at cellular telemetry sites does not itself cause loss of connec-tion or station data latency.

Cellular telemetry station CI.SRT developed more signifi-cant latencies, but not because the cellular telemetry system itself lacked capacity (Fig. 10). CI.SRT is an example of a hybrid cellular relay site. Two other seismic stations, CI.TOW2 and CI.WNM, link to CI.SRT by 900 MHz radio (Fig. 5), and the three stations communicate through a cellular modem physically located at CI.SRT. Latency patterns for CI.WNM and CI.TOW2 provide clues to cellular system per-formance. Data latencies at both increase to some new average value (3 and 18 s for CI.WNM and CI.TOW2, respectively), but then remain near that new average. This indicates that, except during onset (t � 325 s in Fig. 10), the modem throughput was adequate for both stations at their full delivery rates. In addition, the downward-sloped portions in SRT, the longest example being plotted at t � 335 s to t � 375 s, indi-cate periods in which CI.SRT data flowed faster from CI.SRT than it was being recorded. Thus, the nearest cellular modem to the Ridgecrest earthquake sustained a delivery rate sufficient for two or three seismic stations, throughout the first minutes after the Ridgecrest mainshock.

If cellular modem throughput was adequate to sustain CI.WNM and CI.TOW2, then, what is the source of the aver-age delays experienced by these stations? In detail, we do not know. There are several candidate contributors. On the station side, the datalogger writes packets and sends them to radios.

The station side radios package and queue data for delivery to the receiving radio. The receiving radio delivers packets to a router for the cellular modem, and the cellular modem com-municates with SCSN acquisition computers with Tranmission Control Protocol/Internet Protocol (TCP/IP) packet protocols.

Then, periodically (but not for every packet), an acknowledgment must be returned to the datalogger. If the acknowledgment is interrupted, the datalogger will attempt to resend. Because the datalogger sends data on a first-in, first-out basis, late acknowledgments may lead to cycling around an average delay, until the buffer clears.

Figure 11 shows latencies for a subset of SCSN cellular-tele-metered seismic stations farther from the Ridgecrest main-shock. Latency signals are seen to concentrate after the times of arrival of the P wave (circled X symbols), indicating some relationship to the earthquake. The favorable perfor-mance of closer cellular modem links (Fig. 9) makes unlikely an explanation of physical shaking of the towers or cellular equipment. The onset of latency ∼25 s after P-wave arrival at the nearest two sites, CI.LMR2 and CI.SHO, could be related to shaking in the S-wave coda, as a short-term response to less-efficient data compression. The quality of cellular connection appears to be a factor in latency response. Small increases in packet rate (corresponding to small decreases in data compres-sion efficiency) are not a problem for strong links based on performance at sites with stronger shaking (examples in Fig. 9). Weak connections may be vulnerable to interruption.

Stations including CI.SVD, CI.MUR, CI.JEM, and CI.IPK seem to be in this pattern. Based on distance, ground shaking at CI.SVD and more distant stations would be so weak that it could escape detection by human observers. Lesser latency sig-natures could reflect stronger links to seismic stations, Figure 6. (a) Transport latency for 358 SCSN stations calculated as a mean for the day of 3 July 2019, shortly before the start of the Ridgecrest sequence. The latency for the majority of the sites (91%) was under 2 s. (b) Peak data latency for 105 SCSN stations inside and (c) 363 stations outside of the P-wave 30 s circle (Fig. 5) during the entire 30 s interval after the Mw 7.1 event origin. Stations nearer to the epicentral area show the greatest increases in latency.

8 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf

Figure 7. (a) Station data latency in 1 s increments for stations near the Ridgecrest mainshock. Station distances from the mainshock epicenter are shown on the right side. “x” symbols mark P-wave arrival time. White bar for WVP2 shows a period of no data from L1Z. Only seven of 20 stations within 80 km from the epicenter maintained timely data delivery for the full ∼20 s of active rupture of the mainshock. (b) Expanded view showing station latency response to an Mw 5 foreshock 200 s before the mainshock. Severe latency at three stations continued through the mainshock. (c) Station latency histories for theMw 6.4 event.

Stations are shown by distance from the epicenter, and thus in a different order from the mainshock. Data from many stations became latent soon after the P-wave arrival. (Continued)

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 9

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf although, we cannot exclude options such as increased cellular system demand somehow also having a role. We are evaluating with Verizon whether a Mobile Broadband Priority service could improve modem connectivity during future earthquakes.

VSAT

The SCSN has two Ka-band VSAT providers—HughesNet and Viasat. The Ka-band VSATs operate in the 18.3–30 GHz band-width, using multiple spot beams, usually targeting rural areas.

Most SCSN VSAT stations performed well to capture the first few seconds of the Ridgecrest mainshock P-wave arrivals but struggled to maintain latency values below 5 s (Fig. 12) during the aftershock sequence, especially for the HughesNet equipped sites. A few seconds after the shaking onset, two HughesNet stations fell tens of seconds behind and took 3–5 min to recover. One had a 40 s period in which data latencies were larger than 5 s, and one had only short-term interruptions (Fig. 12). These observed latency increases cor-respond to local shaking of the satellite dish by the arrival of surface waves. For example, CI.NJQ is, approximately, 280 km from the epicenter. Using an estimated surface-wave velocity of 3 km=s, the arrival of surface waves at CI.NJQ is, approximately, 93 s after event origin time, which corresponds to the rise in latency in Figure 12. Three Viasat satellite stations showed no event-related latency and another developed latency larger than 5 s during the mainshock that lasted for about 100 s. For both providers, the pattern of latency is consistent with an effective bandwidth limitation. Given our limited sample of VSAT equipped stations and varied performance during this event, no strong conclusions can be drawn.

High-Bandwidth Links Seismic stations on direct, high-bandwidth connections did not exhibit earthquake-related latencies. They can accommo-date data bursts well over 50 kbps and continuous stable con-nections around 50 kbps per site (Steim and Reimiller, 2014).

The exposure of the EEW system to bandwidth-related latency increases could potentially be mitigated by increasing telem-etry bandwidth at vulnerable sites.

As an example, the station CI.RRX that is delivering data over a fiber line (Fig. 13) exhibits data and packet rate increases but without any associated latency increase. Please see Figure 4 to compare the telemetry performance for bandwidth-limited connections.

Figure 7. Continued

10 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf

The total rate of all data arriving at the SCSN regional processing center from Q330 dataloggers is measured every 100 s (but configurable to be as low as 15 s), with link-monitoring software qmaserv. The data rate increased by more than twice compared with the pre-event rate.

Tests and Investigations into Data Delivery We tested a cellular link with several consecutive large file downloads during 5 min intervals over a secure copy protocol (scp) connection a month after the Ridgecrest event, while the regular seismic data were still being delivered once per second.

No problems were found, and the bandwidth consistently mea-sured 1.7 Mbps. Unexpectedly, we discovered that the latency improved during the test data transfers (Fig. 14). Discussion of these results with cellular provider Verizon suggested that because the cellular modem packets have certain sizes, at the increased data rate, they fill up and leave the devices sooner, and hence decrease the data latency.

The results from the large file transfer test could explain aspects of cellular modem latency transients that follow the

P-wave arrival. Brief transients delaying data by a few seconds or less (e.g., CI.BBR and CI.SHO, Fig. 11) are not explained by data volume increases, because they affect stations at distances where shaking intensity is too large to trigger this phenomenon.

This indicates the cause is not insufficient bandwidth, but rather some condition affecting the device waiting time, related perhaps to connection or traffic congestion at the cellular tower.

Compared to voice traffic, seis-mic data are slow and intermit-tent, with only one or a few packets being delivered per second.

We investigated the poten-tial role of data compression on data-delivery latency. We first considered whether the compression process itself con-tributed to delays. This does not appear to be the case. In testing, compression by the dataloggers is extremely effi-cient (J. M. Steim, written comm., 2019). In addition, latency is not observed on high-bandwidth connections, which it would be if the datalogger was responsible for the delay.

We also investigated disabling data compression as a way to even out demand on telemetry and improve latency perfor-mance. In this mode, data delivery would be less efficient on average, but not vulnerable under a higher packet rate dur-ing peak demand. We found that compression cannot be dis-abled for most dataloggers of the SCSN array. For the Q330 family of dataloggers, which comprise the bulk of SCSN data-loggers, all data are delivered as Steim2-compressed records.

The Kinemetrics brand Rockhound-based dataloggers (Basalt and Obsidian models) are not only less strict about data format but also do not support uncompressed data delivery.

Turning off data compression would also affect day-to-day telemetry performance, by forcing higher sustained throughput rates on all links. Links with limited excess capacity for com-pressed data would be more likely to saturate due to uncom-pressed data and develop gaps after any interruption. Thus, turning off data compression might be attractive for flattening peak demand but would not be a viable alternative for the

SCSN.

1 × 107

1 × 107

200 300 400 500 600 700 800

Time (s)

CI.CCC.HHZ

30 CI.CCA.L1Z

79.2 km

30 CI.DTP.L1Z

61.0 km

10 CI.WBS.L1Z

56.1 km

30 CI.CCC.L1Z

35.0 km

8 CI.LRL.L1Z

34.0 km

CI.WRC2.L1Z

19.6 km

La te nc y

(s

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 8. Data latency for radio-connected stations telemetering through Laurel Hub near Ridgecrest. Before the P-wave arrivals (marked in circles with x), data-delivery latency except for CI.WRC2, was under 2 s. (a–f) Station data for these six sites are telemetered from Laurel Hub to a dedicated high-speed circuit drop, using a single high-capacity 900 MHz radio link. The scaling of the vertical axis varies by trace, to better show the latency variations. (g) The vertical axis for the seismic recording is given in counts. The distances from the stations to the magnitude 7.1 mainshock location are given below the station names.

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 11

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf

7 × 106

7 × 106

100 200 300 400 500 600 700 800

Time (s)

CI.WMF.HHZ

5 CI.ISA.L1Z

80.5 km

14 CI.WHF.L1Z

69.0 km

14 CI.WOR.L1Z

59.2 km

CI.WMF.L1Z

44.4 km

1500 CI.SRT.L1Z

16.9 km

La te nc y

(s

(a)

(b)

(c)

(d)

(e)

(f)

Figure 9. (a–e) Stations with cellular communication have varied latency responses to the main events. (f) The scaling of the vertical axes for the latency varies by trace, to better show the latency variations; the vertical axis for the seismic recording is given in counts. Stations CI.ISA, CI.WHF, CI.WOR, and CI.WMF are located in less populated areas, whereas CI.SRT supports three seismic stations and is near the Ridgecrest population center. The distances from the stations to the magnitude 7.1 mainshock location are given below the station names.

3 × 105

3 × 105

300 350 400 450 500 550 600

Time (s)

CI.SRT.HNZ

8 CI.WNM.L1Z

29.0 km

CI.SRT.L1Z

16.9 km

CI.TOW2.L1Z

15.7 km

La te nc y

(s

(a)

(b)

(c)

(d)

Figure 10. (a–c) Illustration of the delay between shaking onset and eventual very high latency over a long period of time for three stations near Ridgecrest. The latency increase is observed starting at ∼26 s after the Mw 7.1 P-wave arrival at the seismic station, SRT. (d) The vertical axis for the seismic recording is given in counts. The distances from the stations to the magnitude 7.1 mainshock location are given below the station names.

12 Seismological Research Letters www.srl-online.org • Volume XX • Number XX • – 2020

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf

200 240 280 320 360 400 440 480 520 560 600

Time (s)

CI.IKP.L1Z

372.9 km

4 CI.SWP.L1Z

358.8 km

4 CI.PDM.L1Z

355.0 km

4 CI.SDR.L1Z

342.7 km

4 CI.JEM.L1Z

312.9 km

4 CI.GOR.L1Z

293.0 km

4 CI.SMW.L1Z

269.4 km

4 CI.CSH.L1Z

253.8 km

4 CI.MUR.L1Z

244.3 km

4 CI.ELS2.L1Z

236.6 km

4 CI.VLY.L1Z

230.0 km

4 CI.SMT.L1Z

205.9 km

4 CI.SES.L1Z

204.6 km

4 CI.SVD.L1Z

190.9 km

4 CI.BBR.L1Z

179.0 km

4 CI.SHO.L1Z

120.1 km

4 CI.LMR2.L1Z

94.0 km

La te nc y

(s

Figure 11. Latency records for 17 sites using Verizon cellular modems. For all sites shown, the cellular modem supports a single, collocated single seismic station. Data are sorted by epicentral distance, from 94 to 373 km (given below the station names); the y-axis scale is 0–8 s. Picked P-wave arrival times on the seismic HNZ channels are marked by a circled “x.” Incidence of latency is observed to concentrate after P-wave arrivals.

Volume XX • Number XX • – 2020 • www.srl-online.org Seismological Research Letters 13

Downloaded from http://pubs.geoscienceworld.org/ssa/srl/article-pdf/doi/10.1785/0220200211/5183520/srl-2020211.1.pdf

Next Steps and Recommendations Our investigation identified several areas for further investigation and opportunities for network improvement for latency perfor-mance. These areas touch on station telemetry, data acquisition and delivery, network operations, and datalogger configurations.

We include these details for the benefit of other network oper-ations interested in improving real-time performance. Readers primarily interested in latency measurements during Ridgecrest may wish to skip ahead.

1. Complete the investigation of conditions under which latency can develop in radio systems on links nominally expected to have sufficient capacity. Beside link bit-rate capacity, review packetization overhead, configurations and capacities of system elements, and how system elements respond when some portion of the system cannot keep up.

2. Upgrade or replace radio links that are unlikely to sustain reasonable peak packet and bit rates in a future earthquake.

Current candidate radio replacement solutions include higher performance licensed 900 MHz radios, higher frequencies such as 4.9 and 5.8 GHz, and shortening radio shots using relays or alternate paths.

3. Complete the investigation of cellular system performance and link quality. Seek carrier input on available useful tests and on the utility of Mobile Broadband Priority for seismic network communications.