26ILDC-MERLIN_Performance_(2016).pdf

Evaluation of the Performance Characteristics of

MERLIN and NLDN Based on Two Years of

Ground-Truth Data from Kennedy Space Center/Cape

Canaveral Air Force Station, Florida

Jonathan D. Hill and Carlos T. Mata

Scientific Lightning Solutions

Titusville, FL, USA d.hill@sls-us.com

Amitabh Nag

Vaisala, Inc.

Louisville, CO

William P. Roeder

45th Weather Squadron

Patrick Air Force Base, Florida

Abstract— From May-October, 2015, a local lightning locating system at Cape Canaveral Air Force Station (CCAFS)/Kennedy

Space Center (KSC) recorded high-speed video images and wideband rate of change of electric field waveforms for nearby lightning return strokes with accurate strike termination locations and times of the order of 10 m and 100 ns, respectively. A subset of data including 321 return strokes having strike points determined with ground-truth accuracy (10 m or better) was selected to evaluate the performance characteristics of the newly installed Mesoscale Eastern Range Lightning Information

Network (MERLIN), a local network of Vaisala TLS-200 sensors being installed to locate lightning strikes at KSC/CCAFS. The results were compared with a similar analysis performed on data collected by the National Lightning Detection Network (NLDN).

The detection efficiencies, strike location errors, error ellipse parameters, and reported peak currents for both MERLIN and

NLDN are evaluated and presented in this paper.

I. INTRODUCTION

In recent years, the lightning environment at Kennedy Space

Center (KSC) and the adjoining Cape Canaveral Air Force

Station (CCAFS) has been continuously monitored by both the

Cloud-to-Ground component of the Four Dimensional

Lightning Surveillance System (CG-4DLSS) (e.g., Boyd et al., 2005; Ward et al., 2008) and the National Lightning Detection

Network (NLDN) (e.g., Cummins et al., 1998; Cummins and

Murphy, 2009; Nag et al., 2011; Mallick et al., 2014). The performance characteristics of these lightning location systems have been evaluated in prior studies (e.g., Mata et al. 2012, 2014; Murphy et al., 2008) based on ground-truth strike location data collected by the lightning monitoring system located at Launch Complex 39B (LC-39B) (e.g., Mata et al.

2010). Recently, the legacy six station CG-4DLSS network are being been replaced by the Mesoscale Eastern Range Lightning

Information Network (MERLIN), a network of 10 Vaisala

TLS-200 sensors, in order to provide enhanced lightning detection and characterization capability for the high-valued infrastructure and assets at KSC/CCAFS (e.g., Roeder and

Saul, 2012).

During the summer and fall months of 2015, the performance of MERLIN was thoroughly evaluated using ground-truth lightning strike location data obtained by a local lightning locating system (LLS) at KSC/CCAFS. Although the 10

MERLIN sensors are supplemented by 10 in-range NLDN sensors, only the performance of the nine MERLIN sensors available at that time was evaluated, that is, the NLDN sensors were excluded. The final MERLIN network will have 10 sensors. The 10th MERLIN sensor will likely improve the network performance slightly, but will also increase the robustness of that performance to missing sensors. The exclusion of the NLDN sensors was done since inclusion of

NLDN is an option that may not be funded in the future, and mailto:d.hill@sls-us.com

Fig 1. MERLIN (cyan) and local LLS (yellow) sensor locations in addition to the ground truth strike points (magenta) of the 321 cloud-to-ground return strokes.

Prominent landmarks are annotated in the expanded view at right. All 10 of the MERLIN sensors are shown, although only 9 sensors were available during the testing period.

thus, a conservative estimate of the MERLIN performance was desired. The local LLS consists of a compact network of 13 high-speed cameras that record cloud-to-ground lightning return strokes terminating on KSC/CCAFS property, with geographic emphasis to the areas surrounding Launch Complex

39B (LC-39B), Launch Complex 39A (LC-39A), Launch

Complex 41 (LC-41), and the Vehicle Assembly Building

(VAB). Many of the cameras are configured with intersecting fields of view to provide multi-angle views of the same discharge. Eight high-speed cameras are located on tall structures at altitudes greater than 150 m, providing downward vantage points for accurately determining lightning strike point locations. The high-speed cameras sample at either 3,200 or

16,000 frames/sec. In addition, the local LLS is equipped with six wideband rate of change of electric field (dE/dt) sensors.

The digitization time bases of these geographically independent sensors are synchronized with RMS accuracy of 15 ns. The dE/dt network is used to locate the strike points of lightning return strokes via time-of-arrival (TOA) techniques.

Comparative analysis using high-speed video data and supporting evidence from detailed Monte Carlo simulations has shown that the mean strike location error for strokes terminating within the sensor network is of the order of 10 m.

For this study, lightning strike point locations determined via high-speed video, dE/dt TOA, or a combination thereof, are considered to be ground truth. A total of 321 return strokes detected by the local LLS between May and October, 2015 were chosen to evaluate the performance of MERLIN. Note this is a small subset of the total local LLS data set. In determining the data set, strong preference was given to those events with both well-resolved six station dE/dt measurements and multi-angle high-speed video data. A map of the MERLIN

(cyan) and local LLS (yellow) sensor locations overlaid on an aerial photograph in shown in Fig. 1 at left, with an expanded view of the local LLS sensor network area at right. The strike locations of the 321 return strokes detected by the local LLS are shown in magenta. Prominent landmarks are annotated for reference.

The primary goals of the MERLIN performance evaluation were to, 1) determine the return stroke detection efficiencies for the full data set, first strokes, subsequent strokes, strokes attaching directly to ground, water, or short structures, and strokes attaching to tall structures, 2) characterize the latitude and longitude errors and error distributions, 3) determine the validity of the reported error ellipses at the 50%, 95%, and 99% confidence levels, 4) determine if the MERLIN location error ellipses had a bivariate normal distribution, and 5) determine the return stroke peak current distribution reported by

MERLIN. For reference, the same parameters were computed for the corresponding NLDN data set.

II. RETURN STROKE DETECTION EFFICIENCIES

The 321 return strokes were associated with 84 different flashes. MERLIN detected 83 of the 84 flashes (i.e., at least one stroke in the flash was reported), a flash detection efficiency of 98.8%, while NLDN detected 100% of the flashes.

The 321 stroke data set captured by the local LLS was

TABLE 1. SUMMARY DETECTION EFFICIENCY STATISTICS FOR GROUND TRUTH RETURN STROKES REPORTED BY MERLIN AND NLDN

Total Strokes First Strokes Subsequent

Strokes

Strokes Attaching to

Ground, Water, or Low

Structures

Strokes Attaching to Tall Structures

Ground Truth 321 133 188 276 45

MERLIN 296 (92.2%) 122 (91.7%) 174 (92.5%) 258 (93.5%) 38 (84.4%)

NLDN 297 (92.5%) 121 (91.0% 176 (93.6%) 253 (91.7%) 44 (97.8%)

composed of 133 first strokes to ground (41.4%) and 188 subsequent strokes (58.6%). Stroke order was determined by examination of both dE/dt waveforms and high-speed video records. A total of 276 strokes (86%) attached directly to the ground or water while 45 strokes (14%) attached to tall structures. MERLIN reported a total of 296 of the 321 ground truth return strokes, a detection efficiency of 92.2%, while

NLDN similarly reported 297 strokes, a detection efficiency of

92.5%. This MERLIN stroke detection efficiency is somewhat consistent with the analysis of a smaller data set performed by

Wilson [2016] that found a stroke detection of 86% ± 6%.

MERLIN and NLDN reported 122 (91.7%) and 121 (91.0%) first strokes and 174 (92.5%) and 176 (93.6%) subsequent strokes, respectively. For the 276 strokes that attached directly to the ground, water, or low structures, MERLIN and NLDN reported 258 (93.5%) and 253 (91.7%) strokes, respectively.

Finally, MERLIN and NLDN reported 38 (84.4%) and 44

(97.8%) of the 45 strokes that attached directly to tall structures.

The detection efficiencies calculated for the NLDN data set were all similar to those calculated in the previous study reported by Mata et al. [2014] for a data set of 54 ground truth return strokes recorded exclusively in the LC-39B area. The results of the detection efficiency analyses are summarized in

Table 1.

A total of 10 of the 25 return strokes (40%) not reported by

MERLIN were associated with flashes having at least two ground attachment points in sub-millisecond time succession.

In seven cases, MERLIN successfully reported the first ground attachment point but failed to report the second ground attachment. Two cases followed the opposite convention. In the final case, MERLIN failed to report a flash with three ground attachment points, two that contacted ground directly and one that attached to the LC-41 lightning protection system.

Similarly, 12 of the 24 return strokes (50%) not reported by

NLDN were associated with multiple ground attachment flashes. NLDN reported the first ground attachment point but not the second ground attachment point in seven cases. In one case, the second ground attachment point was reported while the first was missed. Finally, in two cases, both the first and second ground attachment points were not reported.

III. EVALUATION OF MERLIN AND NLDN STRIKE LOCATION

ERRORS

The total distances D between the ground truth strike locations and the MERLIN and NLDN strike locations were calculated using the Earth’s radius as the reference sphere. The total distances were then further decomposed into the longitude

(x) and latitude (y) horizontal components according to (1) and

(2), where 𝜃 is the angle between the reported error ellipse center and the ground truth strike location with respect to the rotated semi-major axis.

𝑥 = 𝐷 ∙ sin(𝜃) (1) 𝑦 = 𝐷 ∙ cos(𝜃) (2)

Histograms of the longitude and latitude errors for MERLIN and NLDN relative to the ground truth strike locations are shown in Fig. 2 and Fig. 3, respectively. Positive longitude and latitude errors correspond to a reported strike location that is east and north of the ground truth strike location. Clearly, the error distributions for both MERLIN and NLDN contain data points that are significant outliers. Prior to further analyzing these data, the outliers were examined both quantitatively and qualitatively to determine if they should be removed from the data sets. The inter-quartile range (IQR) for each distribution was first computed. The upper and lower bounds for the minor and major outliers for each distribution were computed by multiplying the IQR by a factor of 1.5 for minor outliers and 3 for major outliers, then adding and subtracting those values to and from the third and first quartile values of the data set, respectively (e.g., NIST/SEMATECH e-Handbook of Statistical

Methods, http://www.itl.nist.gov/div898/handbook/, pp. 2001-

2003, 2013). The thresholds for the minor and major outliers for each distribution are given in Table 2.

The MERLIN errors for a total of 19 strokes in the data set violated the criteria for being major outliers. Four of these strokes were associated with flashes having multiple ground attachment points in sub-millisecond time succession. Seven of the strokes had either abnormally high chi-squared goodness of fit values (greater than 4) and/or were detected by only two

MERLIN stations. Finally, eight strokes had significant location errors for unknown reasons despite being located by a sufficient number of stations and having well-determined solutions. Interestingly, all eight of these strokes were located in the vicinity of the VAB.

NLDN errors for a total of 34 return strokes violated the major outlier criteria given in Table 2. A total of 23 of these strokes were detected by only 2 or 3 stations and/or had reported peak current less than 10 kA. The remaining 11 strokes were detected by a sufficient number of stations and had well-determined solutions, but violated the major outlier criteria by having high value error in latitude (one stroke), high value error in longitude (four strokes), or high value error in both latitude and longitude (six strokes).

Fig 2. Histogram of MERLIN longitude and latitude errors relative to the ground truth strike locations with major outliers included. Summary statistics are provided in the plot insets.

Fig 3. Histograms of NLDN longitude and latitude errors relative to the ground truth strike locations with outliers included. Summary statistics are provided in the plot insets.

TABLE 2. CRITERIA FOR DETERMINING MINOR AND MAJOR OUTLIERS IN MERLIN AND NLDN LONGITUDE AND LATITUDE ERROR

DISTRIBUTIONS.

Peak Negative Error Peak Positive Error Minor Outlier

Thresholds

Major Outlier

Thresholds

MERLIN Longitude

Errors [m] -1507.2 935.8 [-158.6, 114.1] [-260.9, 216.4]

MERLIN Latitude

Errors [m] -622.8 950.0 [-67.0, 84.7] [-123.9, 141.5]

NLDN Longitude

Errors [m] -15073.6 17230.1 [-629.8, 363.8] [-1009.5, 736.4]

NLDN Latitude

Errors [m] -609.3 1148.0 [-321.9, 232.6] [-529.8, 440.5]

The major outliers have been removed from the MERLIN and NLDN data sets for the purpose of evaluating the characteristic error distributions of both lightning location systems. It is important, however, that users of these data understand that the major outliers are included in the bulk data set. Histograms of the MERLIN and NLDN longitude and latitude errors are re-plotted in Fig. 4 and Fig. 5 with the major outliers removed. With major outliers removed, the total mean/median strike location errors for MERLIN and NLDN were 58.1 m/53.6 m and 224.7 m/190.3 m, respectively.

Mean/median longitude error for MERLIN and NLDN were -

23.9 m/-24.2 m and 131.4 m/109.7 m, while mean/median latitude errors were 10.2 m/7.8 m and -31.4 m/-49.4 m, respectively. Strike location error data for MERLIN and

NLDN are summarized in Table 3.

The distributions of MERLIN and NLDN errors in each horizontal direction were analyzed for normalcy by plotting the data on normal probability plots. The results are shown in Fig.

6 and Fig. 7. The MERLIN error distributions follow the normal distribution for data points between the first and third quartiles, however, both the longitude and latitude error distributions also exhibit heavy tails. The longitude errors demonstrate more significant heavy tails at the lower end of the distribution, while the latitude errors demonstrate more significant heavy tails at the upper end of the error distribution.

These observations indicate that, for this 277 return stroke data set, the actual MERLIN error distribution is not bivariate normal, but may be better modeled using a t Location-Scale distribution (e.g., NIST/SEMATECH e-Handbook of Statistical

Methods, http://www.itl.nist.gov/div898/handbook/, pp. 338-

340, 2013) to account for the heavy tails at the upper and lower ends of the distribution.

The NLDN longitude error distribution shown in Fig. 7 follows the normal distribution between the 10% and 90% probability levels, although there is a slight oscillation about the straight line plot, which is indicative of a somewhat bimodal distribution. This characteristic is also visible in the NLDN longitude error histogram shown in Fig. 5. Like the MERLIN error distributions, the NLDN longitude error distribution also exhibits significant heavy tails at the upper and lower ends of the distribution. With the exception of one outlying data point, the NLDN latitude errors follow the normal distribution with minor deviation between the 1% and 80% probability levels, above which the latitude errors exhibit a significant heavy tail.

Considering the geographic area where the data were collected is located on the fringe of the NLDN sensor network, it is possible that the deviations from normal shown for the NLDN error distributions are strongly influenced by the geometry of the strike locations relative to the sensor network.

IV. EVALUATION OF MERLIN AND NLDN ERROR ELLIPSE

PERFORMANCE

The error ellipse semi-major and semi-minor axes lengths along with the ellipse rotation angle reported by MERLIN and

NLDN correspond to the 50% confidence ellipse, that is, 50% of the located return strokes should have ground truth strike locations that occur within the error ellipse. For each event located by MERLIN and NLDN, the minimum distance R between the ellipse center and the ellipse edge in the direction of the ground truth strike location was calculated using

Equation 3, where a is the semi-major axis length, b is the semi-minor axis length, and 𝜃 is the angle between the ellipse center and the ground truth strike location with respect to the rotated semi-major axis.

𝑅(𝜃) = 𝑎 ∙ 𝑏

√(𝑏 ∙ cos(𝜃))2 + (𝑎 ∙ sin(𝜃))2 (3)

Equation (3) provides a convenient method for programmatically determining whether the ground truth strike location occurred inside or outside the reported error ellipse.

For a given stroke, if the computed distance D between the ellipse center and the ground truth strike location is less than or equal to the computed value for R in (3), then the strike occurred inside the reported error ellipse. Otherwise, the event occurred outside the reported error ellipse.

The percentages of reported events for MERLIN and NLDN that occurred within the 50% confidence ellipses were calculated separately for the data sets with major outliers included and omitted. The 50% MERLIN confidence ellipse contained 85.4% of the ground truth strokes for the full data set, and 90.6% of the strokes for the data set with no major outliers.

For the 50% MERLIN ellipses, the mean/median semi-major and semi-minor axes lengths were 118.6 m/100 m and 101.4

Fig 4. Histogram of MERLIN longitude and latitude errors relative to the ground truth strike locations with major outliers omitted. Summary statistics are provided in the plot insets.

Fig 5. Histograms of NLDN longitude and latitude errors relative to the ground truth strike locations with major outliers omitted. Summary statistics are provided in the plot insets.

TABLE 3. STRIKE LOCATION ERROR DATA FOR MERLIN AND NLDN WITH BOTH MAJOR OUTLIERS INCLUDED AND OMITTED. ALL ERROR

MEASUREMENTS ARE GIVEN IN UNITS OF METERS.

MERLIN NLDN

N = 296 N = 277 N = 297 N = 263

Mean Total Error 87.1 58.1 664.9 224.7

Median Total Error 56.5 53.6 208.4 190.3

StD Total Error 146.3 37.0 2003.3 172.8

Min/Max Total Error [3.5, 1538.0] [3.5, 227.0] [6.5, 17230.1] [6.5, 966.0]

Mean Longitude Error -25.5 -23.9 -258.7 -131.4

Median Longitude Error -24.2 -24.2 -126.3 -109.7

StD Longitude Error 133.0 54.4 2086.9 221.9

Min/Max Longitude Error [-1507.1, 935.8] [-225.8, 130.7] [-15073.6, 17230.1] [-965.9, 576.7]

Mean Latitude Error 19.8 10.2 -26.5 -31.4

Median Latitude Error 8.4 7.8 -56.0 -49.4

StD Latitude Error 101.4 33.6 183.1 113.9

Min/Max Latitude Error [-622.8, 949.9] [-92.9, 102.4] [-609.3, 1148.0] [-493.2, 381.3] m/100 m for the full data set and 113.0 m/100 m and 100.7 m/100 m for the data with no major outliers, respectively.

These values suggest that the semi-major and semi-minor axes lengths for the MERLIN 50% confidence ellipses may be overestimated. The confidence level of the MERLIN ellipses needs to be scaled accordingly to determine the actual lengths of the semi-major and semi-minor axes that most closely contain 50% of the ground truth strike locations in the data set.

Given a default confidence level of 50%, the semi-major and semi-minor axes can be scaled by the factor k obtained from

(4).

𝑘 = √−2 ∙ ln (1 − 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦)

√−2 ∙ ln (1 − 0.5) (4)

The confidence level of the MERLIN ellipses was iteratively adjusted until the error ellipses contained 50% of the ground truth strokes. For the full dataset, the 18% confidence ellipse contained 50% of the ground truth strokes, with mean/median semi-major and semi-minor axes lengths of 63.4 m/53.5 m and

54.2 m/53.5 m, respectively. With no major outliers, the 16.5% confidence ellipse contained 50% of the ground truth strokes.

In this case, the mean/median semi-major and semi-minor axes lengths were 57.6 m/51.0 m and 51.4 m/51.0 m, respectively.

The NLDN 50% confidence ellipses contained 45.8% of the ground truth strokes for the full data set (mean/median semi-major and semi-minor axes lengths of 455.6 m/200.0 m and

159.6 m/200.0 m) and 51.3% of the ground truth strokes for the data set with no major outliers included (mean/median semi-major and semi-minor axes lengths of 274.5 m/200.0 m and

149.0 m/200.0 m). With outliers removed, the mean semi-major and semi-minor ellipse axes lengths for the actual 50%

MERLIN ellipse were about factors of 4.8 and 2.9 shorter than those reported for the 50% NLDN confidence ellipse.

The validity of MERLIN and NLDN error ellipses were also evaluated for the 95% and 99% confidence levels by scaling the reported 50% ellipses using (4). The MERLIN 95% confidence ellipse contained 94.6% of the ground truth strokes for the full data set and 99.6% of the ground truth strokes for the data set with no major outliers. The mean/median semi-major and semi-minor axes lengths of the MERLIN 95% confidence ellipse were 246.5 m/207.9 m and 210.7 m/207.9 m for the full dataset and 234.9 m/207.9 m and 209.4 m/207.9 m for the data set with no major outliers, respectively. While the 50%

MERLIN ellipse appears to be underestimating the true lengths of the ellipse axes, for the full dataset, the 95% MERLIN confidence ellipse seems to perform nominally.

The 95% NLDN confidence ellipses contained 85.6% and

90.1% of the ground truth strokes for the full data set and the data set with no major outliers, respectively, both values being less than the ideal 95% level. This suggests that the NLDN ellipse semi-major and semi-minor axes lengths are perhaps underestimated. For the 95% NLDN confidence ellipse, the mean/median semi-major and semi-minor ellipse axes lengths were 947.1 m/415.8 m and 331.8 m/415.8 m for the full dataset and 570.7 m/415.8 m and 309.9 m/207.9 m for the data set with no major outliers. MERLIN 95% confidence ellipse semi-

Fig 6. a) Normal probability plot for MERLIN longitude errors, and B) normal probability plot for MERLIN latitude errors.

Fig 7. a) Normal probability plot for NLDN longitude errors, and B) normal probability plot for NLDN latitude errors.

TABLE 4. MERLIN CONFIDENCE ELLIPSE PERFORMANCE STATISTICS FOR DATA SET WITH MAJOR OUTLIERS INCLUDED (N = 296) AND

OMITTED (N = 277). ALL LENGTH MEASUREMENTS ARE GIVEN IN UNITS OF METERS.

16.5%

Ellipse

18%

Ellipse 50% Ellipse 95% Ellipse 99% Ellipse

N = 277 N = 296 N = 277 N = 296 N = 277 N = 296 N = 277 N = 296

% of Ground Truth

Strokes Inside Ellipse 50 50 90.6 85.4 99.6 94.6 100 95.3

Mean Semi-Major

Axis Length

57.6 63.4 113.0 118.6 234.9 246.5 291.3 305.7

Median Semi-Major

Axis Length

51.0 53.5 100.0 100.0 207.9 207.9 257.8 257.8

Mean Semi-Minor

Axis Length

51.4 54.2 100.7 101.4 209.4 210.7 259.6 261.2

Median Semi-Minor

Axis Length

51.0 53.5 100.0 100.0 207.9 207.9 257.8 257.8 major and semi-minor axes lengths were about factors of 2.4 and 1.5 shorter than those reported by NLDN.

For the full and abbreviated data sets, the 99% confidence ellipses included 95.3% and 100% of the MERLIN events and

90.9% and 95.1% of the NLDN events. MERLIN mean/median semi-major and semi-minor axes lengths were 305.7 m/257.8 m and 261.2 m/257.8 m for the full data set and 291.3 m/257.8 and 259.6 m/257.8 m for the data set with no major outliers.

The computed NLDN semi-major and semi-minor axes lengths were 1174.2 m/515.5 m and 411.4 m/515.5 m for the full data set and 707.6 m/515 m and 384.2 m/257.7 m with major outliers removed. For the 99% confidence ellipses, the calculated

MERLIN semi-major and semi-minor axes lengths were about factors of 2.4 and 1.5 shorter than those reported by NLDN for the data set with no major outliers.

The MERLIN error ellipse parameters for the dataset used in this study are reported with a resolution of 100 m. Only 35 of the 296 strokes (11.8%) detected by MERLIN had reported semi-major axes length that did not equal 100 m. Similarly, only four strokes (1.3%) reported by MERLIN had semi-minor axes length that differed from 100 m. The lack of resolution in the reporting of the MERLIN error ellipse parameters is likely influencing the observed disparity between the expected and actual performance of the 50% error ellipse. With the latest software update, the MERLIN central processor now has the ability to report error ellipse parameters with 10 m resolution.

This new capability will allow future studies to more accurately compare the reported semi-major axes lengths to the measured location errors.

Summary statistics for the error ellipse validation of

MERLIN and NLDN are provided in Table 4 and Table 5, respectively.

V. COMPARISON OF MERLIN AND NLDN PEAK CURRENT

ESTIMATES

The return stroke peak currents for the 283 events reported by both MERLIN and NLDN were compared. The MERLIN return stroke peak currents were sorted in order of increasing magnitude (note all 283 of the events were of negative polarity).

In Fig. 8A, the sorted MERLIN return stroke currents are plotted in blue with the corresponding NLDN currents for each event plotted in red. The percent differences of the NLDN peak currents relative to the MERLIN peak currents are plotted in

Fig. 8B in order of increasing peak current magnitude.

Histograms of the overall distributions of MERLIN and NLDN peak currents are shown in Fig. 8C and Fig. 8D, respectively.

For typical return stroke peak currents magnitudes above about

20 kA, the NLDN peak currents tend to exceed the MERLIN peak current magnitudes by 5-10%. For current magnitudes below 20 kA, there is more variability in the comparison of

MERLIN and NLDN peak currents. There are about 15 data points in the bottom half of the peak current distribution shown in Fig. 8A and Fig. 8B where there is significant (15% or larger) deviation between the MERLIN and NLDN currents. For the full dataset, the mean values of the MERLIN and NLDN peak current distributions were -22.3 kA and -23.5 kA, respectively.

VI. SUMMARY

The performance characteristics of the newly installed

MERLIN lightning location network at KSC/CCAFS were evaluated using a set of 321 return strokes with ground truth strike point location accuracy. The results were compared with similar statistics computed for the existing NLDN lightning location network. The flash detection efficiencies for MERLIN and NLDN were 98.8% and 100%, respectively, for the 84 total flashes. The stroke detection efficiencies for MERLIN and

NLDN were comparable, 92.2% and 92.5%, respectively.

About half of the strokes not reported by MERLIN and NLDN were associated with flashes having multiple ground attachment points in sub-millisecond time succession. Strokes reported by MERLIN with large location errors relative to

Fig 8. A) Comparison of MERLIN and NLDN reported peak return stroke currents for 283 events, plotted in order of increasing peak current magnitude, B) percent difference between NLDN and MERLIN peak currents for 283 events, plotted in order of increasing peak current magnitude, C) distribution of MERLIN peak currents, D) distribution of NLDN peak currents.

ground truth tended to be associated with flashes having multiple ground attachment points and/or strokes located in a relatively small geographic area around the VAB. NLDN strokes reported with large location errors also tended to be associated flashes having multiple ground attachment points, but were also frequently associated with strokes only detected by 2-3 stations and/or having reported peak currents below 10 kA. For the subsets of strokes that did not have outlying location errors (277 strokes for MERLIN and 263 strokes for

NLND), the error distributions for both MERLIN and NLDN were found to be more-or-less normally distributed between the first and third quartiles of the data sets. The MERLIN latitude and longitude errors exhibited heavy tails at both extreme ends of the error distributions, as did the NLDN longitude error. For the data set with no major outliers, the reported MERLIN 50% confidence ellipses contained 90.6% of the ground truth strike locations, suggesting that the MERLIN semi-major and semi-minor axes lengths may be overestimated. It was found that the

16.5% MERLIN confidence ellipse actually contained 50% of the ground truth strike locations. The 50% NLDN confidence ellipse contained 51.3% of the ground truth strokes for the data set with no major outliers, indicating nominal performance.

MERLIN mean/median semi-major and semi-minor axes lengths for the 50% ellipse were 57.6 m/51.0 m and 51.4 m/51.0 m, respectively, about factors of 4.8 and 2.9 shorter than those reported by NLDN. MERLIN 95% and 99% confidence ellipses were found to contain 99.6% and 100% of the ground truth strokes, while NLDN 95% and 99% ellipses contained

90.1% and 95.1% of ground truth strokes, suggesting that the

NLDN semi-major and semi-minor axes lengths for the 95% and 99% may be slightly underestimated, at least for the geographic area where the data were collected. The return stroke peak current distributions reported by MERLIN and

NLDN were similar, with NLDN peak current magnitudes tending to exceed those reported by MERLIN by typically 5-

10% for return stroke peak current magnitudes larger than about

20 kA.

ACKNOWLEDGMENT

The authors would like to thank Jon Saul of the 45th Weather Squadron for providing the MERLIN data set, and William Brooks and John Cramer from Vaisala Inc. for providing the NLDN data set.

TABLE 5. NLND CONFIDENCE ELLIPSE PERFORMANCE STATISTICS FOR DATA SET WITH MAJOR OUTLIERS INCLUDED (N = 297) AND (N =

263). ALL LENGTH MEASUREMENTS ARE GIVEN IN UNITS OF METERS.

50% Ellipse 95% Ellipse 99% Ellipse

N = 263 N = 297 N = 263 N = 297 N = 263 N = 297

% of Ground Truth

Strokes Inside Ellipse

51.3 45.8 90.1 85.6 95.1 90.9

Mean Semi-Major

Axis Length

274.5 455.6 570.7 947.1 707.6 1174.2

Median Semi-Major

Axis Length

200.0 200.0 415.8 415.8 515.5 515.5

Mean Semi-Minor

Axis Length

149.0 159.6 309.9 331.8 384.2 411.4

Median Semi-Minor

Axis Length

100.0 200.0 207.9 415.8 257.8 515.5

REFERENCES

Boyd, B.F, W. P. Roeder, D. L. Hajek, and M. B. Wilson, “Installation, upgrade, and evaluation of a short baseline cloud-to-ground lightning surveillance system in support of space launch operations”, (1st) Conference on Meteorological Applications of Lightning Data, 9-13 Jan 05, San Diego, California, 9-13 January 2005.

Cummins, K.L., and M. J. Murphy, An Overview of Lightning Location Systems: History, Techniques, and Data Uses, With an In-Depth Look at the US. NLDN, IEEE Transactions on Electromagnetic Compatibility, Vol. 51, No. 3, August 2009.

Mallick, S., et al. (2014), Performance characteristics of the NLDN for return strokes and pulses superimposed on steady currents, based on rocket-triggered lightning data acquired in Florida in 2004–2012, J. Geophys.

Res. Atmos., 119, 3825–3856, doi:10.1002/2013JD021401.

Mata, C.T., V. A Rakov, T. Bonilla, A. G. Mata, E. Navedo and G. P. Snyder, “A new comprehensive lightning instrumentation system for PAD 39B at the Kennedy Space Center, Florida”, International Conference on Lightning Protection 2010, Cagliari, Italy, September 2010.

Mata, C.T., V. A Rakov, A. G. Mata, A. Nag and J. Saul, “Evaluation of the performance characteristics of CGLSS II and U.S. NLDN using ground-truth data from Launch Complex 39B, Kennedy Space Center, FL”, International Lightning Detection Conference, Broomfield, Colorado, April 2012.

Mata, C.T., J.D. Hill, and A.G. Mata, “Evaluation of the Performance Characteristics of the CGLSS and NLDN Systems Based on Two Years of Ground-Truth Data from Launch Complex 39B, Kennedy Space Center, Florida”, International Lightning Detection Conference, Tuscon, Arizona, March 2014.

Murphy, M.J., K. L. Cummins, N. W. S. Demetriades, and W. P. Roeder, “Performance of the new Four-Dimensional Lightning Surveillance System (4DLSS) at the Kennedy Space Center/Cape Canaveral Air Force Station Complex”, 13th Conference on Aviation, Range, and Aerospace Meteorology, New Orleans, LA , Ppaer 8.6, 20-24 January 2008

Nag, A., et al. (2011), Evaluation of U.S. National Lightning Detection Network performance characteristics using rocket-triggered lightning data acquired in 2004–2009, J. Geophys. Res., 116, D02123, doi:10.1029/2010JD014929.

Roeder, W.P. and J.M. Saul, “Four Dimensional Lightning Surveillance System: Status and Plans”, International Lightning Detection Conference, Broomfield, Colorado, April 2012.

Ward, J.G., K. L. Cummins and E. P. Krider, “Comparison of the KSC-ER Cloud-To-Ground Lightning Surveillance System (CGLSS) and the U.S.

National Lightning Detection Netwrok (NLDN)” 20th International Lightning Detection Conference,Tucson, USA, April 2008.

Wilson, M., “Technical Analysis Report (TAR) Cape MERLIN cloud-to-ground ground truth evaluation”, CDRL A062, Contract Number FA8806-15-C-0001, 13 January 2016.

http://dx.doi.org/10.1002/2013JD021401