Lightning Probability Upgrade Presentation to 45 WS.pdf
PDF 737 KB Posted
- Attached to
- Upgrade Lightning Probability Tool - Add Model Predictors Federal contract opportunity
- Solicitation number
- FA252123QB112
About this file
This presentation details the results of an analysis to upgrade an existing lightning probability prediction tool. It compares the performance of the current tool against a new tool using four datasets: with and without a low wind speed regime, each split by flow regime and month. Across all analyses, the new tool outperformed the current tool in key metrics including Brier Skill Score, probability distributions, reliability diagrams, and contingency table statistics. The dataset with the low wind speed regime split by month showed the best overall results for the new tool.
The related federal contract opportunity seeks proposals to upgrade the existing lightning probability tool by adding additional model predictors. The solicitation is issued by the Department of the Air Force Space Command under solicitation number FA252123QB112 for the product titled "Upgrade Lightning Probability Tool - Add Model Predictors." Proposals are sought to improve the tool's predictive performance through incorporation of new variables in the statistical model.
View the file
Other files for this federal contract opportunity
| File | Type | Posted |
|---|---|---|
| 23QB0112 Q and A 29Aug23.pdf | ||
| Thunderstorm forecasting at Cape Kennedy_ Florida_Neumann.pdf | ||
| Wind Speed Paramter in Lightning Flow Regimes.pdf | ||
| 23QB0112 SOO.pdf | ||
| 23QB112 Solicitation.pdf | ||
| Class_Deviation_2023-O0002_Attachment_1.pdf | ||
| WD 2015-4555.pdf |
On GovTribe
Work with this file on GovTribe
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Lightning Probability Tool - Final Results Results for the following are shown:
• Slide 2: Dataset with no low wind speed regime, dataset split by flow regime
• Slide 3: Dataset with no low wind speed regime, dataset split (stratified) by month
• Slide 4: Dataset with low wind speed regime, dataset split (stratified) by flow regime
• Slide 5: Dataset with low wind speed regime, dataset split (stratified) by month
• Slides 6-9 show the same data, but with similar statistics grouped together for easy comparison, areas highlighted in blue showed the best results
• The ‘split by’ designation means that the data was stratified by either flow regime or month and then predictor selection and logistic regression was run on the stratified data
Included on each slide:
• Brier Skill Score: show how each equation performs against corresponding standard forecast methods
• Probability Distributions: show how well the equations distinguish between lightning and non-lightning days
• Reliability Diagrams: shows the distribution of forecasts and their associated observations, indicates how the equations perform in terms of under- or over-forecasting lightning occurrence
• Contingency Table Statistics: calculates several accuracy measures and skill scores blue
Brier Skill Score by flow regime BSS SW-1 SW-2 SE-1 SE-2 NW NE Other All vs Current Tool 10.1 15.0 -1.6 -6.9 5.8 -0.7 -3.2 4.8 vs Persistence 32.9 41.4 33.2 32.5 45.1 46.3 50.8 41.0 vs Daily Climo 30.3 27.3 15.5 11.9 14.8 37.7 12.0 22.2 vs Monthly Climo 30.7 28.8 16.5 12.6 18.5 39.9 12.8 23.7 vs Flow Regime 9.4 20.1 9.2 3.8 16.4 23.6 11.4 14.1
Dataset with NO LOW WIND SPEED regime, split by flow regime
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
No low wind speed FR Data split by FR
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Li gh tn in g Fr eq ue nc y
Forecast Probability
Reliability Diagram No Low Wind Speed FR, Split by FR
Current Tool New Tool Perfect
Current Tool New Tool
POD 0.660 0.684
FAR 0.335 0.325
HR 0.748 0.758
CSI 0.495 0.515
HSS 0.461 0.485
KSS 0.464 0.480
No Low Speed Flow Regime, Split by FR
Dataset with NO LOW WIND SPEED regime, split by month
Brier Skill Score by month BSS May Jun Jul Aug Sep Oct All vs Current Tool -5.7 15.1 9.9 -1.2 -0.3 25.7 7.2 vs Persistence 36.3 52.5 46.7 55.3 32.0 43.4 46.7 vs Daily Climo 20.5 30.7 31.8 26.4 35.6 27.7 29.4 vs Monthly Climo 22.6 32.0 32.1 27.7 36.0 30.8 30.6 vs Flow Regime 15.8 21.7 21.5 17.9 28.7 26.8 21.9
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
No low wind speed FR Data split by month
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Li gh tn in g Fr eq ue nc y
Forecast Probability
Reliability Diagram No Low Wind Speed FR, Split by Month
Current Tool New Tool Perfect
Current Tool New Tool
POD 0.677 0.709
FAR 0.311 0.290
HR 0.764 0.783
CSI 0.519 0.550
HSS 0.495 0.536
KSS 0.502 0.537
No Low Speed Flow Regime, Split by Month
Dataset WITH LOW WIND SPEED regime, split by flow regime
Brier Skill Score by flow regime BSS SW-1 SW-2 SE-1 SE-2 NW NE Other Low All vs Current Tool 16.4 7.2 2.0 -6.1 0.6 3.8 -6.2 4.7 4.5 vs Persistence 46.8 42.2 36.7 46.7 39.3 49.8 40.9 44.7 43.1 vs Daily Climo 30.1 14.0 12.4 18.4 19.5 34.8 6.9 21.9 19.4 vs Monthly Climo 30.4 15.3 10.6 19.4 22.1 35.7 10.1 22.6 20.3 vs Flow Regime 7.4 3.4 1.7 12.0 18.8 17.2 9.9 20.1 9.8
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
With low wind speed FR Data split by FR
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
L ig h tn in g
F re q u e n c y
Forecast Probability
Reliability Diagram
With Low Wind Speed FR, Split by FR
Current Tool New Tool Perfect
Current Tool New Tool
POD 0.629 0.665
FAR 0.376 0.333
HR 0.719 0.750
CSI 0.456 0.499
HSS 0.402 0.466
KSS 0.399 0.467
With Low Speed Flow Regime, Split by FR
Dataset WITH LOW WIND SPEED regime, split by month
Brier Skill Score by month BSS May Jun Jul Aug Sep Oct All vs Current Tool -2.2 14.6 7.1 0.6 2.9 13.9 6.4 vs Persistence 42.9 42.9 52.4 51.2 40.1 46.9 46.8 vs Daily Climo 22.6 25.8 29.3 24.6 24.4 11.8 24.3 vs Monthly Climo 25.1 29.3 29.7 25.8 25.0 13.2 25.9 vs Flow Regime 17.7 17.9 18.9 15.0 15.9 8.7 16.2
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) No-LTG (New)
With low wind speed FR Data split by month
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
L ig h tn in g
F re q u e n c y
Forecast Probability
Reliability Diagram
With Low Wind Speed FR, Split by Month
Current Tool New Tool Perfect
Current Tool New Tool
POD 0.651 0.677
FAR 0.330 0.313
HR 0.749 0.764
CSI 0.493 0.517
HSS 0.461 0.494
KSS 0.472 0.500
With Low Speed Flow Regime, Split by Month
Brier Skill Score
BSS SW-1 SW-2 SE-1 SE-2 NW NE Other All vs Current Tool 10.1 15.0 -1.6 -6.9 5.8 -0.7 -3.2 4.8 vs Persistence 32.9 41.4 33.2 32.5 45.1 46.3 50.8 41.0 vs Daily Climo 30.3 27.3 15.5 11.9 14.8 37.7 12.0 22.2 vs Monthly Climo 30.7 28.8 16.5 12.6 18.5 39.9 12.8 23.7 vs Flow Regime 9.4 20.1 9.2 3.8 16.4 23.6 11.4 14.1
Brier Skill Score, No Low Speed Flow Regime, Split by FR
BSS May Jun Jul Aug Sep Oct All vs Current Tool -5.7 15.1 9.9 -1.2 -0.3 25.7 7.2 vs Persistence 36.3 52.5 46.7 55.3 32.0 43.4 46.7 vs Daily Climo 20.5 30.7 31.8 26.4 35.6 27.7 29.4 vs Monthly Climo 22.6 32.0 32.1 27.7 36.0 30.8 30.6 vs Flow Regime 15.8 21.7 21.5 17.9 28.7 26.8 21.9
Brier Skill Score by month
BSS SW-1 SW-2 SE-1 SE-2 NW NE Other Low All vs Current Tool 16.4 7.2 2.0 -6.1 0.6 3.8 -6.2 4.7 4.5 vs Persistence 46.8 42.2 36.7 46.7 39.3 49.8 40.9 44.7 43.1 vs Daily Climo 30.1 14.0 12.4 18.4 19.5 34.8 6.9 21.9 19.4 vs Monthly Climo 30.4 15.3 10.6 19.4 22.1 35.7 10.1 22.6 20.3 vs Flow Regime 7.4 3.4 1.7 12.0 18.8 17.2 9.9 20.1 9.8
Brier Skill Score by flow regime
BSS May Jun Jul Aug Sep Oct All vs Current Tool -2.2 14.6 7.1 0.6 2.9 13.9 6.4 vs Persistence 42.9 42.9 52.4 51.2 40.1 46.9 46.8 vs Daily Climo 22.6 25.8 29.3 24.6 24.4 11.8 24.3 vs Monthly Climo 25.1 29.3 29.7 25.8 25.0 13.2 25.9 vs Flow Regime 17.7 17.9 18.9 15.0 15.9 8.7 16.2
Brier Skill Score by month
Dataset with no low wind speed regime, split by flow regime
Dataset with no low wind speed regime, split by month
Dataset with low wind speed regime, split by flow regime
Dataset with low wind speed regime, split by month (BEST)
Green indicates where new tool beat the current tool Orange indicates where current tool beat the new tool
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
No low wind speed FR Data split by FR
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
No low wind speed FR Data split by month
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) N0-LTG (New)
With low wind speed FR Data split by FR
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
O cc ur re nc e
Forecast Probabilities
Forecast Probability Distributions for Lightning (LTG) and Non Lightning (No-LTG) Days
May - October 1989-2021
LTG (Current) No-LTG (Current) LTG (New) No-LTG (New)
With low wind speed FR Data split by month
Probability Distribution
Datasets with no low wind speed regime
Datasets with low wind speed regime
All similar, no definitive best result
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Li gh tn in g Fr eq ue nc y
Forecast Probability
Reliability Diagram No Low Wind Speed FR, Split by FR
Current Tool New Tool Perfect
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Li gh tn in g Fr eq ue nc y
Forecast Probability
Reliability Diagram No Low Wind Speed FR, Split by Month
Current Tool New Tool Perfect
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
L ig h tn in g
F re q u e n c y
Forecast Probability
Reliability Diagram
With Low Wind Speed FR, Split by FR
Current Tool New Tool Perfect
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
L ig h tn in g
F re q u e n c y
Forecast Probability
Reliability Diagram
With Low Wind Speed FR, Split by Month
Current Tool New Tool Perfect
Reliability Diagrams
Datasets with no low wind speed regime
Datasets with low wind speed regime
BEST
Current Tool New Tool
POD 0.660 0.684
FAR 0.335 0.325
HR 0.748 0.758
CSI 0.495 0.515
HSS 0.461 0.485
KSS 0.464 0.480
No Low Speed Flow Regime, Split by FR Current Tool New Tool
POD 0.677 0.709
FAR 0.311 0.290
HR 0.764 0.783
CSI 0.519 0.550
HSS 0.495 0.536
KSS 0.502 0.537
No Low Speed Flow Regime, Split by Month
Current Tool New Tool
POD 0.629 0.665
FAR 0.376 0.333
HR 0.719 0.750
CSI 0.456 0.499
HSS 0.402 0.466
KSS 0.399 0.467
With Low Speed Flow Regime, Split by FR Current Tool New Tool
POD 0.651 0.677
FAR 0.330 0.313
HR 0.749 0.764
CSI 0.493 0.517
HSS 0.461 0.494
KSS 0.472 0.500
With Low Speed Flow Regime, Split by Month
Contingency Table Statistics
Dataset with no low wind speed regime, split by flow regime
Dataset with no low wind speed regime, split by month (BEST)
Dataset with low wind speed regime, split by flow regime
Dataset with low wind speed regime, split by month
File details come from the government source that posted it. Updated .