2024_IE_ARC_Outline.pdf

State of California – A Natural Resources Agency Gavin Newsom, Governor

OFFICE OF ENERGY INFRASTRUCTURE SAFETY Caroline Thomas Jacobs, Director 715 P Street, 15th Floor | Sacramento, CA 95814

916.902.6000 | www.energysafety.ca.gov

Independent Evaluator Annual Report on Compliance Outline for Wildfire Mitigation Plan Compliance Year 2024

1. Introduction This document sets forth the Office of Energy Infrastructure Safety’s (Energy Safety) Independent Evaluator Annual Report on Compliance (IE ARC) evaluation requirements and associated report outline for reporting evaluation work.1 Where there is disagreement with the below information and the Compliance Guidelines or the Request for Qualifications (IE RFQ), the Compliance Guidelines will prevail over the IE RFQ, which will prevail over this document. This document has the following main sections:

1. Evaluations Process – Contains descriptions of seven steps for independent evaluators (IEs) to take in completing their tasks and is provided for the purposes of guiding the IEs in their work.

2. Annual Report on Compliance Structure – Contains both the required outline for the IE ARCs and specifies content and methodologies for each section or sub-section of the report.

3. Definitions – Is a list of words used in this document and their meanings.

4. Appendices – Provides all the appendices for this document which may contain further specifications to sub-sections.

2. Evaluation Process

2.1. Create a catalog of initiatives (Step 1)

The IE must read through the latest approved version of the Wildfire Mitigation Plan (WMP) to create its own categorized list of all initiatives with accompanying targets.

In Steps 2 through 5 the IE should assess the compliance with initiatives. Initiatives are measures or activities designed to reduce the consequence and/or probability of wildfire or a Public Safety Power Shutoff (PSPS) event. Initiatives are typically assigned a unique initiative identifier (e.g., WMP.459), have associated annual targets (e.g., replace 3,500 poles in 2023), and are listed in the “Targets” sub-sections in the WMP (e.g., Table 8-3. Grid Design, Operations, and Maintenance Targets by Year).2 While the electrical corporation (EC) will

1 The words “evaluation” and “assessment” are used interchangeably throughout the document and mean the same.

2 Pages 78-79 of the 2023-2025 WMP Technical Guidelines available on the Energy Safety website.

provide the IE with a list of initiatives, the IE should cross-reference this list with the latest version of the WMP to ensure there are no initiatives that are missing.

2.2. Divide initiatives by prioritization and field verifiability (Step 2)

Each IE must develop a list of ten to fifteen Focus Initiatives from the comprehensive list of initiatives from the WMP. Focus Initiatives must be EC initiatives that contribute to the most risk-reduction, address the most consequential ignition risks (e.g., risk impact % in WMPs) and/or assess initiatives with which an EC is or has been substantially non-compliant. Focus Initiatives will receive additional scrutiny. The number of Focus Initiatives should increase with the relative size of the EC’s territory; an IE evaluating the largest California EC by territory should choose 15 Focus Initiatives. If an EC has fewer than 10 initiatives, each initiative should be considered a Focus Initiative. The IE should share the list of Focus Initiatives with Energy Safety for input prior to confirming them and beginning their assessment. Refer to Appendix A of this document for a non-exhaustive list of example criteria.

All other initiatives should be considered Non-Focus Initiatives.

To select Field Verifiable Initiatives, the IE should reference the IE RFQ, Appendix A, for a complete list of field verifiable initiatives. Focus Initiatives that are tagged “Recommended” and “Context-dependent” must be verified in the field. Non-Focus Initiatives that are tagged “Recommended” must be verified in the field. The IE should use its best judgment and initiative contexts to decide if field verification is feasible for specific initiatives and can seek exceptions from Energy Safety. Permitting time and resources, the IE is also encouraged to field verify any other initiatives where the field visits would yield meaningful findings, though they should prioritize other parts of the IE ARC process first.

Given the guidance above, each initiative should be designated by the IE as one of the following four types:

• Focus and Field Verifiable

• Non-Focus and Field Verifiable

• Focus and Non-Field Verifiable

• Non-Focus and Non-Field Verifiable

2.3. Calculate sample sizes and identify locations (Step 3)

To select sampling sizes, the IE should use the simplified version of Cochran's sample size formula with a finite population correction.3, 4 Refer to Appendix B for the full formula and the

3 Sampling Techniques, Third Edition by William G. Cochran, Section 4.5 "Sample Size for Proportions," starting on page 75

4 Sampling Techniques, Third Edition by William G. Cochran, Section 4.3 "The Finite Population Correction," starting on page 24 list of inputs. A sample size formula must be used for initiatives with a population larger than 15; for initiatives with a smaller population, all units must be reviewed.

To select sampling locations (i.e., where the samples should be distributed), the IE should examine the distribution of initiative units across different California Public Utilities Commission (CPUC)-defined High Fire-Threat District (HFTD) areas. Next, the IE must divide the initiative population into three strata: HFTD tier 3, HFTD tier 2, and non-HFTD areas. The IE then must select up to 10% of the sample from non-HFTD areas and select the rest of the samples from HFTD tier 3 and tier 2 in approximately the same ratio as their population ratio.5

For initiatives with a population that does not have HFTD stratification, random sampling must be used.

All efforts must be made to ensure the selected sample is representative of the population and area where the initiative took place. Clustered sampling should be avoided. For field-verifiable activities only, the IE may consider practicality considerations such as accessibility, observability of work completed, and density of field verifiable assets.

To calibrate samples, the IE must also select an additional 25% of samples to be used in the case that any of the primary samples are unusable or inaccessible. If unusable or inaccessible samples outnumber the predefined alternate samples of 25%, additional samples must be selected to complete the sample size according to the guidelines above.

2.4. Perform verifications and extrapolate results (Step 4)

The IE must perform the verifications by adhering to the following guidance:

For Field Verifiable Initiatives, the IE must conduct field verifications on selected samples to confirm installation, commissioning, and adherence to applicable EC standards for such work.

For this type of initiative, the IE must request a copy of relevant initiative construction standards from the EC and either physically go into the field to inspect the selected samples against the standards or utilize unmanned aerial vehicles (UAVs) for samples that are difficult to access. Other technology solutions must be submitted in Field Plans and approved by Energy Safety. When verifying an initiative in the field, the IE should note any collateral observations from Appendix D beyond findings for the initiative itself.

For Non-Field Verifiable Initiatives, the IE must seek additional documentation and conduct subject matter expert (SME) interviews, as needed, to verify that the selected samples were completed and commissioned, and that they adhere to applicable EC standards for such work.

To calculate the sample and initiative validation rates, the sample validation rate must be multiplied by the EC-reported population to estimate the total number of completed items.

5 If the population does not have units in one or more of the HFTD areas, distribute the sample proportionally across available zones using a similar judgment.

Then, this estimate shall be divided by the WMP target population to determine the actual initiative validation rate. There are three scenarios where the extrapolation may vary:

• If the EC reports an actually completed population larger than the target specified in their WMP, the actual initiative validation rate should be higher than the sample validation rate.

• If the EC reports a population equal to the target specified in their WMP, the actual initiative validation rate should match the sample validation rate.

• If the EC reports an actually completed population smaller than the target specified in their WMP, the actual initiative validation rate should be lower than the sample validation rate.

If the total initiative validation rate is greater or equal to 95%, the initiative is considered validated by the IE. If not, the initiative is not validated by the IE. The IE should note the initiative validation rate in Column 7 of Table 1.

2.5. Verify funding compliance (Step 5)

The IE must review the WMP to record the budget for each initiative and then compare the values reported in the EC ARC with supporting documentation to verify the actual expenditures. The IE shall report and highlight any funding discrepancies identified during the examination and include the correct values for those entries and the likely causes of the discrepancy. The IE should also calculate the absolute percent difference of the initiative’s expenditure from the budget according to the formula below:

𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑝𝑝𝐴𝐴𝑝𝑝𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝐴𝐴𝑝𝑝𝐴𝐴𝑝𝑝𝑝𝑝𝐴𝐴 = 𝑎𝑎𝐴𝐴𝐴𝐴(𝐵𝐵𝐴𝐴𝑑𝑑𝐵𝐵𝐴𝐴𝐴𝐴 − 𝐸𝐸𝐸𝐸𝑝𝑝𝐴𝐴𝑝𝑝𝑑𝑑𝑑𝑑𝐴𝐴𝐴𝐴𝑝𝑝𝐴𝐴)/𝐵𝐵𝐴𝐴𝑑𝑑𝐵𝐵𝐴𝐴𝐴𝐴.

Initiatives with absolute percent difference greater than 10% require additional rationale to explain why the expenditure differs from the budget. The following questions should be answered as part of the rationale:

• What drivers contributed to a difference between the budget and expenditure?

• Did the drivers contributing to the difference between the budget and expenditure impact the initiative’s progression or the EC’s ability to mitigate risk?

The IE must review EC rationale provided in the EC ARC to answer the above questions. If the EC rationale is insufficient or not included, the IE must request additional documentation and explanation to address the guiding questions above.

2.6. Evaluate Quality Assurance/Quality Control Programs

and Effectiveness (Step 6)

The IE must review Sections 6.6, 8.1.6, and 8.2.5 of the WMP and any other Quality Assurance/ Quality Control (QA/QC)-related sections. The IE must review additional documentation and conduct SME interviews,6 as needed, to understand the policies, procedures, systems, and qualifications of auditors and contractors. The IE must describe the process and documentation reviewed when performing the assessment. The IE must also reference the program assessment framework in Appendix E to evaluate the EC’s QA/QC programs level of maturity for each of the five QA/QC dimensions and provide recommendations for improvement. QA/QC programs include all wildfire-related QA/QC activities that take place in the organization.

Please note that compliance of QA/QC initiatives (i.e., Sections 8.1.6 and 8.2.5) should be verified in Section 3.4 in the same way as for all other initiatives.

2.7. Populate the IE ARC Outline (Step 7)

The IE must use the instructions and section headings specified in the Annual Report on Compliance Structure section of this report to write the IE ARC.

3. Annual Report on Compliance Structure The following reporting sections should be used by IEs as the basis for structuring the IE ARC and each section of the report should provide the specificity described below and include all applicable tables.

3.1. Executive Summary

The Executive Summary must contain high-level information and overall impressions from the IE’s evaluation of the ECs’ WMP initiatives, funding, and QA/QC programs.

In the Executive Summary, the IE must at least:

• Report the total number and percentage of initiatives that were completed within 5% of the WMP target and a list of all initiatives that missed their target or could not be validated by the IE.

• Report the total budget, total expenditure, total variance between budget and expenditure, and the number and percentage of initiatives with absolute percent differences greater than 10%.

• List the 10 largest initiatives by planned expenditure and the number of these initiatives that have been “failed to fund” (i.e., funded below 100%).

• Provide recommendations for the EC’s future WMPs, recordkeeping, and data management, if applicable.

6 If possible, some contractors and on-the-ground personnel should be interviewed to understand actual execution of QA/QC.

3.2. Focus Initiatives and Discussion

In this section, the IE must:

• List the Focus Initiatives that the IE selected for further evaluation,

• Describe the methodology for determining each Focus Initiative,

• Discuss the criteria the IE considered to choose Focus Initiatives.

3.3. Site and Sample Selection and Discussion

In this section for both focus and non-focus initiatives, the IE must:

• List all initiatives that were field verified,

• Describe rationale for scenarios where:

o IE has not field verified an initiative that qualified for field verification, o IE has added another initiative for field verification that is not required as described in the guidance below,

• Describe the method for choosing sample locations for both field and non-field verifications and include a map of the service territory overlaid with sampling locations.

3.4. Review of Initiatives across WMP Categories: Compliance

and Funding

The WMP initiative categories are:

• Risk Methodology and Assessment

• Wildfire Mitigation Strategy Development

• Wildfire Mitigations o Grid Design, Operations, and Maintenance o Vegetation Management and Inspections o Situational Awareness and Forecasting o Emergency Preparedness o Community Outreach and Engagement

• Public Safety Power Shutoff

In this section, the IE must:

• For the above list of WMP initiative categories, list the number of initiatives that belong to each of the following four sub-categories of compliance for:

o Focus and Field Verifiable Initiatives, o Focus and Non-Field Verifiable Initiatives, o Non-Focus and Field Verifiable Initiatives, o Non-Focus and Non-Field Verifiable Initiatives.

• Describe the high-level methodology for evaluating funding compliance

3.4.1. Grid Design, Operations, and Maintenance

3.4.1.1. Initiative Summary Table

In this section, the IE must include in the Initiative Summary Table (Table 1) all initiatives in this WMP category.

The IE must ensure every initiative from the WMP in the catalog of initiatives is included in Table 1 even if it was not reported in the EC ARC. The IE must organize the initiatives in Table 1 in the same order that they appear in the WMP. Please refer to Appendix C for a detailed description of the content for each column that Table 1 must have.

Table 1: Example Initiative Summary Table

In iti at iv e N um be r, W

M P

Se ct io n

N um be r, an d

N am e

W M

P

In iti at iv e Ta rg et

EC

-C la im ed

Pr og re ss la at iv e St at us

Sa m pl e

Si ze pl e

Va lid at io n Ra te

Ve rif ic at io n

M et ho d

IE

F in di ng o n In iti at iv e

(In at iv e Va lid at io n

Ra te

W M

P

Pl an ne d

Sp en d

EC

-C la im ed A ct ua l Sp en d a nd fr om b ud ge t) tis fie d

Ri sk

Re du ct io n

Go al

WMP.972

8.1.2.1

Pole Replacement

Replace 14,850 poles

15,850 Target met

400 97.5% Field inspection

SAP

Notification

(DR064)

Initiative validated (104%)

$2,507,000 $1,435,000

(-43%)

No

Poles were replaced in lower-risk areas, reducing overall risk mitigation

WMP.550

8.4.2.3

Lightning Arrestor Replacement

Replace 2,216 arrestors

1,500 Target not met;

120 83% Field inspection

SAP Work Order

(DR082)

Initiative not validated (56%)

$3,407,000 $4,000,888

(+15%)

No

EC did not meet its target by >30% and did not meet its intended risk reduction at iv e N um be r, W

M P

Se ct io n

N um be r, an d

N am e

W M

P

In iti at iv e Ta rg et la

Pr og re ss la at iv e St at us pl e

Si ze pl e

Va lid at io n Ra te

Ve rif ic at io n

M et ho d

IE

F in di ng o n In iti at iv e

(In at iv e Va lid at io n

Ra te

W M

P

Pl an ne d

Sp en d

EC

-C la im ed A ct ua l Sp en d a nd fr om b ud ge t) tis fie d

Ri sk

Re du ct io n

Go al

WMP.843

8.2.6

QA/QC for Asset Inspections

Inspect 35 line miles for vegetation

35 Target met

35 100% QA/QC

documenta tion

(DR034)

SME

interview

(SME02)

Initiative validated (100%)

$3,150,987 $3,149,987 (0%)

Yes

3.4.1.2. Written Detail for Initiatives

The IE must list each initiative in the same order as it appears in Table 1 and include the EC’s unique tracking ID for each initiative (e.g., SH-14), the WMP section number (e.g., 8.3.2.4), the official name of the initiative from the WMP (e.g., Expulsion Fuse Replacement), and the initiative type (e.g., Focus and Field Verifiable).

3.4.1.2.1. Initiative Review – Findings & Method

For each Focus Initiative in this WMP category, the IE must elaborate on any findings and detail the method and evidence collected for verification.

• Findings: The IE must:

o State whether the target has been met; if the target was not met, discuss detailed drivers for not meeting the target.

o State whether targeted risk reduction was accomplished; if it was not accomplished, provide detailed reasons for not meeting it.

o Include any observations on recordkeeping and data management for this initiative.

o Include any findings, observations, and key risk areas that the IE has noticed while evaluating compliance for the initiative and provide recommendations for improving this initiative.

• Method & Evidence: The IE must:

o For desktop review, elaborate on the data request sent, describe the documents received by the EC, and make a statement on the how well the provided documents addressed the original questions in the data request.

o For SME interviews, describe the number of interviews, the titles of those interviewed, and the implications of the interviews on the initiative.

For each Field-Verifiable Initiative in this WMP category, the IE must elaborate on any findings and detail the method and evidence collected for verification.

• Findings: The IE must:

o Discuss any findings and observations from field inspections or data/document reviews, including collateral observations highlighted in Appendix D.

o Highlight any deviations from relevant industry and construction standards and include pictures of non-conformance in Section 3.7.5.

• Method & Evidence: The IE must:

o For field inspections, discuss how many samples were verified and the method of field verification (e.g., UAV, ground inspections, etc.)

o For any data/document reviews, elaborate on the data request sent, describe the documents received by the EC, and make a statement on the how well the provided documents addressed the original questions in the data request.

For both Focus and Field-Verifiable Initiatives in this WMP category, the IE should combine the Findings and Method & Evidence as outlined above for both Focus and Field-Verifiable Initiatives.

For non-Focus and non-Field-Verifiable Initiatives in this WMP category, the IE must elaborate and add written detail in this section including:

• Why the initiative or targeted risk may not have been achieved.

• Description of nature and number of documents reviewed, or interviews performed.

• If the IE has observed anything of importance during the initiative review (for example, if the risk reduction goal has not been met) that has not been captured in Table 1 already.

3.4.1.2.2. Funding Verification – Findings

For initiatives in which the absolute percent difference between budgeted and actual amount spent is greater than 10% in this WMP category, the IE must elaborate on the following findings.

• Findings: The IE must:

o Document the EC’s detailed rationale for spending more than the targeted or budgeted amount or spending less and provide additional comments as necessary.

o Comment on whether funding played a significant role in the progression or lack of progression of the initiative.

o Comment on implications of the amount spent on targeted risk reduction achievement.

For all other initiatives in this WMP category, Table 1 information should be sufficient, and no additional commentary is required. IE should state “refer to Table 1”. The IE should elaborate and add written detail in this section only if the IE has observed anything of importance during the funding verification (for example, if different documents reported significantly different figures for actual spend) that has not been captured in Table 1 already.

3.4.1.3. Synthesis of Findings

3.4.1.3.1. Synthesis of Findings – Initiative Review

In this section, the IE must summarize, by initiative category as defined in WMP, the initiative review findings across all initiatives within each WMP category. The IE must, by category:

• Summarize the EC’s missed targets and most common reasons for missing targets.

• Discuss overall level of wildfire risk reduction achieved within the category, how that determination was made, and any gaps from actual to projected risk reduction.

• Summarize insights on recordkeeping and data accuracy or management within this category and, if relevant, provide recommendations for EC on areas to improve around data collection, accuracy or management issues.

• Highlight any year-over-year changes and trends in compliance on a category-level, such as missing initiative targets within this WMP category and any improvements in this cycle.

• Highlight any anecdotes, impressions, areas of concern or importance that the IE has come across.

• Describe any gleanings of how the EC intends to improve future performance in any initiative assessed in the ARC.

The IE should not simply repeat findings for individual Focus Initiatives but rather synthesize and analyze findings for the entire category.

3.4.1.3.2. Synthesis of Findings – Funding Verification

In this section, the IE must summarize the funding verification findings across all initiative categories within each WMP category. The IE must, for each category:

• Document the category’s budget from the WMP, actual expenditure, and percent departure from the budget and expenditure.

• Document how many initiatives’ absolute percent differences were at least 10% and the most common reasons for the variances.

• Summarize key trends and findings regarding funding compliance and how funding allocations relate to the overall initiative progression, risk mitigation, and category objectives.

Sections 3.4.2 through 3.4.5 should have the same sub-sections and information as Section 3.4.1.

3.4.2. Vegetation Management and Inspections

3.4.2.1. Initiative Summary Table

3.4.2.2. Written Detail for Initiatives

3.4.2.2.1. Initiative Review – Findings & Method

3.4.2.2.2. Funding Verification – Findings

3.4.2.3. Synthesis of Findings

3.4.2.3.1. Synthesis of Findings – Initiative Review

3.4.2.3.2. Synthesis of Findings – Funding Verification

3.4.3. Situational Awareness and Forecasting

3.4.3.1. Initiative Summary Table

3.4.3.2. Written Detail for Initiatives

3.4.3.2.1. Initiative Review – Findings & Method

3.4.3.2.2. Funding Verification – Findings

3.4.3.3. Synthesis of Findings

3.4.3.3.1. Synthesis of Findings – Initiative Review

3.4.3.3.2. Synthesis of Findings – Funding Verification

3.4.4. Emergency Preparedness

3.4.4.1. Initiative Summary Table

3.4.4.2. Written Detail for Initiatives

3.4.4.2.1. Initiative Review – Findings & Method

3.4.4.2.2. Funding Verification – Findings

3.4.4.3. Synthesis of Findings

3.4.4.3.1. Synthesis of Findings – Initiative Review

3.4.4.3.2. Synthesis of Findings – Funding Verification

3.4.5. Community Outreach and Engagement

3.4.5.1. Initiative Summary Table

3.4.5.2. Written Detail for Initiatives

3.4.5.2.1. Initiative Review – Findings & Method

3.4.5.2.2. Funding Verification – Findings

3.4.5.3. Synthesis of Findings

3.4.5.3.1. Synthesis of Findings – Initiative Review

3.4.5.3.2. Synthesis of Findings – Funding Verification

3.5. Evaluation of QA/QC Programs

In this section, the IE must:

• Provide a rating based on slides provided in Appendix E.

• Report what documents were reviewed, which SMEs were interviewed, and how the IE used this data to determine the quality of the QA/QC programs.

• Describe the QA/QC programs by including the key program components, procedures for documenting QA/QC activities, qualifications, recent changes, and plans to further develop the program.

• Use the program assessment framework, as detailed in Appendix E, to provide an assessment of QA/QC across each of the five dimensions.7

• Provide recommendations to improve the QA/QC programs in the future for each of the five dimensions.

• Include a sunburst chart depicting QA/QC maturity overview. See example in Figure 1 of how to structure the chart showing five levels of maturity for each of the five dimensions of QA/QC evaluation.

Figure 1: Example of Sunburst Chart Structure

7 Roles & responsibilities; quality culture; quality management system; quality inspections/audits; QA/QC technology adoption

3.6. Conclusion

In this section the IE must provide a summary of findings and make concluding statements similar in structure to the Executive Summary. However, this section may include more detail than what is required in the executive summary.

3.7. Attachments

The IE must include a list of all attachments with File Format and File Name for each attachment. The IE should submit the IE ARC and all attachments in one ZIP file. Energy Safety will provide the Excel Attachment with applicable columns for the IEs to fill out.

3.7.1. Catalog of Initiatives

Excel Attachment – Tab 2: Comprehensive list of initiatives across all WMP categories:

• The same columns and information as in Table 1.

• Two additional columns o EC-Claimed actual units from Q4 QDR for each initiative.

o EC-Claimed actual units from EC ARC for each initiative.

3.7.2. Data Requests

Excel Attachment – Tab 3: Comprehensive list of documents reviewed including:

• All public documents reviewed with document publication date.

• All data requests with date sent, date response received, section / initiative, data request number, list of documents received.

3.7.3. SME Interviews

Excel Attachment – Tab 4: Summary of all SME interviews conducted including:

• SME interview date, initiative discussed, interview number, positions interviewed, and a summary of interviews.

3.7.4. List of “Fail-to-Fund” initiatives

Excel Attachment – Tab 5: The list of all failed-to-fund initiatives (i.e., funded less than 100%) with the following information included:

• WMP category – e.g., Grid Design, Operations, and Maintenance

• WMP section number – e.g., 8.1.2.3

• WMP page number – e.g., 87

• Initiative tracking ID – e.g., SH-01 (if applicable)

• Initiative name – e.g., “Pole replacement”

• WMP – planned spend – e.g., $10,000

• EC-Claimed actual spend – e.g., $9,000

• Variance – e.g., 10%

3.7.5. Pictures of Non-Conformance

Attachment [PDF]: Any pictures of non-conformance identified during field inspections by the IE with the location of the picture and a brief explanation of the non-conformance.

4. Definitions “Budget” – The amount of money the electrical corporation estimated, in its WMP or in its change orders, that it would spend on an initiative.

“Electrical Corporation (EC)” – Every corporation or person owning, controlling, operating, or managing any electric plant for compensation within California, except where the producer generates electricity on or distributes it through private property solely for its own use or the use of its tenants and not for sale or transmission to others.

“Energy Safety” – The Office of Energy Infrastructure Safety.

“Expenditure” – The amount of money spent by the electrical corporation on a WMP initiative within the compliance period.

“Field Plan” – The plan with the number of field samples, locations of samples, and methods for field verification that the IE must submit to Energy Safety.

“Field Verifiable Initiative” – Initiative that can be directly inspected, observed, or measured in the physical environment where it is implemented to confirm its completion, quality, and adherence to established standards and protocols.

“Focus Initiative” – An initiative that contributes to the most risk-reduction, addresses the most consequential ignition risks, and/or has missed its target currently or historically.

“Goals” – The electrical corporation’s general intentions and ambitions.

“Independent Evaluator (IE)” – A third-party entity, approved by Energy Safety and contracted by the EC, that is responsible for objectively assessing and reviewing the EC’s annual compliance with their approved WMP

“Initiative” – Measure or activity, either proposed or in process, designed to reduce the consequences and/or probability of wildfire or PSPS.

“Mitigation” – Activities to reduce the loss of life and property from natural and/or human-caused disasters by avoiding or lessening the impact of a disaster and providing value to the public by creating safer communities.

"Objectives” – Specific, measurable, achievable, realistic, and timely outcomes for the overall WMP strategy, or mitigation initiatives and activities that a utility can implement to satisfy the primary goals and subgoals of the WMP program.

“Target” – A forward-looking, quantifiable measurement of work to which an EC commits to in its WMP. ECs will show progress toward completing targets in subsequent reports, including Quarterly Data Reports (QDRs) and WMP Updates.

“Vegetation Management (VM)” – Trimming and removal of trees and other vegetation at risk of contact with electric equipment.

“Wildfire Mitigation Plan (WMP)” – A comprehensive strategy developed and implemented by ECs to reduce the risk and impact of wildfires and public safety power shutoffs associated with their infrastructure.

5. Appendices Appendix A: Criteria for Identifying Focus Initiatives The IE must develop its own criteria for identifying Focus Initiatives. The following non-exhaustive list shows example criteria to consider:

• Initiative is projected to result in significant fire risk reduction compared to other initiatives (e.g., for certain ECs, IE can reference the risk impact % in the WMP).

• Initiative has a relatively high projected spend compared to other initiatives.

• EC has missed funding, or units target in the current year’s EC ARC by more than 20% or has repeatedly missed target in the past.

• EC has been issued high-severity Notices of Violation or Defect (NOVs, NODs) for a specific initiative and/or has shown limited responsiveness to implementing corrective actions.8

• Any other criteria that the IE chooses to reflect the unique context of each EC.

8 Defects can also be referred to as wildfire safety concerns (WSCs).

Appendix B: Sampling Methodology Formula The IE should use the simplified version of Cochran's sample size formula with a finite population correction to calculate the sampling sizes. The formula is stated below:

𝑝𝑝 =

𝑁𝑁 ⋅ �̂�𝑝 ⋅ (1 − �̂�𝑝) ⋅ 𝑧𝑧2

(𝑁𝑁 − 1) ⋅ 𝑀𝑀𝑀𝑀𝐸𝐸2 + �̂�𝑝 ⋅ (1 − �̂�𝑝) ⋅ 𝑧𝑧2

Where:

( 𝑝𝑝 ) 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝐴𝐴𝑎𝑎𝑠𝑠𝑝𝑝𝐴𝐴𝐴𝐴 𝐴𝐴𝑑𝑑𝑧𝑧𝐴𝐴

( 𝑁𝑁 ) 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴𝐴𝐴𝑎𝑎𝐴𝐴𝑑𝑑𝐴𝐴𝑝𝑝,𝑤𝑤ℎ𝑑𝑑𝑝𝑝ℎ 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝐴𝐴𝑎𝑎𝑝𝑝𝐵𝐵𝐴𝐴𝑝𝑝 𝐴𝐴𝑑𝑑 (1) 𝑊𝑊𝑀𝑀𝑊𝑊 𝑑𝑑𝑝𝑝𝑑𝑑𝐴𝐴𝑑𝑑𝑎𝑎𝐴𝐴𝑑𝑑𝑖𝑖𝐴𝐴 𝐴𝐴𝑎𝑎𝑝𝑝𝐵𝐵𝐴𝐴𝐴𝐴 𝑎𝑎𝑠𝑠𝐴𝐴𝐴𝐴𝑝𝑝𝐴𝐴, 𝐴𝐴𝑝𝑝

(2) 𝐸𝐸𝐸𝐸 𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴𝐴𝐴𝑑𝑑 𝑎𝑎𝑠𝑠𝐴𝐴𝐴𝐴𝑝𝑝𝐴𝐴 𝐴𝐴𝑑𝑑 𝑝𝑝𝐴𝐴𝑠𝑠𝑝𝑝𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑 𝐴𝐴𝑝𝑝𝑑𝑑𝐴𝐴𝐴𝐴

( �̂�𝑝 ) 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑𝑠𝑠𝑎𝑎𝐴𝐴𝐴𝐴𝑑𝑑 𝐴𝐴𝑎𝑎𝑠𝑠𝑝𝑝𝐴𝐴𝐴𝐴 𝑝𝑝𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴𝑑𝑑𝐴𝐴𝑝𝑝

( 𝑧𝑧 ) 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝑧𝑧 𝐴𝐴𝑝𝑝𝐴𝐴𝑝𝑝𝐴𝐴 𝑝𝑝𝐴𝐴𝑝𝑝𝑝𝑝𝐴𝐴𝐴𝐴𝑝𝑝𝐴𝐴𝑝𝑝𝑑𝑑𝑑𝑑𝑝𝑝𝐵𝐵 𝐴𝐴𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝑝𝑝𝐴𝐴𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝐴𝐴𝑝𝑝𝑝𝑝𝐴𝐴 𝐴𝐴𝐴𝐴𝑖𝑖𝐴𝐴𝐴𝐴 (𝐸𝐸𝐶𝐶)

( 𝑀𝑀𝑀𝑀𝐸𝐸 ) 𝑑𝑑𝐴𝐴 𝐴𝐴ℎ𝐴𝐴 𝑠𝑠𝑎𝑎𝑝𝑝𝐵𝐵𝑑𝑑𝑝𝑝 𝐴𝐴𝑑𝑑 𝐴𝐴𝑝𝑝𝑝𝑝𝐴𝐴𝑝𝑝

For initiatives where the population size is 15 or less, the IE shall verify the entire population.

For population sizes larger than 15, the IE shall utilize the inputs detailed in Table 2 (Sampling Inputs) depending on the type of initiative.

For the Estimated Sample Proportion (�̂�𝑝 ), use the “sample validation rate” from the prior compliance year’s IE ARC, for particular initiative, if the value is between 85% and 95%. If the prior year sample validation rate is less than 85%, use 85%. If the prior year sample validation rate is greater than 95%, use 95%. For example, if the prior year sample validation rate is 96%, use 95%. If it is 94%, use 94%. If there is no information on the prior year sample validation rate from the IE ARC (if a particular initiative didn’t exist in prior year or other reason for no information), assume 90%.

Table 2: Sampling Inputs

Initiatives Inputs Focus Initiatives Non – Focus Initiatives

Field verifiable ( �̂�𝑝 ) 𝐴𝐴𝑑𝑑 sample validation rate from prior year IE ARC with minimum of 85% and maximum of 95% (90% if none provided in prior compliance year IE ARC)

( 𝑧𝑧 ) 𝐴𝐴𝑑𝑑 1.96 𝑑𝑑𝐴𝐴𝑝𝑝 𝑎𝑎 95% 𝐸𝐸𝐶𝐶

( 𝑀𝑀𝑀𝑀𝐸𝐸 ) of 5%

( �̂�𝑝 ) 𝐴𝐴𝑑𝑑 sample validation rate from prior year IE ARC with minimum of 85% and maximum of 95% (90% if none provided in prior compliance year IE ARC)

( 𝑀𝑀𝑀𝑀𝐸𝐸 ) of 10%

Initiatives Inputs Focus Initiatives Non – Focus Initiatives

Non-Field verifiable ( �̂�𝑝 ) 𝐴𝐴𝑑𝑑 sample validation rate from prior year IE ARC with minimum of 85% and maximum of 95% (90% if none provided in prior compliance year IE ARC)

( 𝑧𝑧 ) 𝐴𝐴𝑑𝑑 1.96 𝑑𝑑𝐴𝐴𝑝𝑝 𝑎𝑎 95% 𝐸𝐸𝐶𝐶

( 𝑀𝑀𝑀𝑀𝐸𝐸 ) of 5%

( �̂�𝑝 ) 𝐴𝐴𝑑𝑑 sample validation rate from prior year IE ARC with minimum of 85% and maximum of 95% (90% if none provided in prior compliance year IE ARC)

( 𝑀𝑀𝑀𝑀𝐸𝐸 ) of 10%

Appendix C: Descriptions of Initiative Summary Table Columns The Initiative Summary Table (Table 1) must have the following columns:

• Initiative Number and Name: EC’s unique tracking ID if used by the EC (e.g., SH-14, WMP.972), the WMP section number (e.g., 8.3.2.4, 8.2.1.1), and the official name of the initiative from the WMP (e.g., expulsion fuse replacement, pole replacement).

• WMP – Initiative Target: Initiative target as it appears in the latest approved WMP (e.g., Install 84 weather stations, Convert 11 circuit miles of overhead to underground, Complete analytics report summarizing assessment of historical consequence data for improved fire spread modeling).

• EC-Claimed Progress: EC’s self-reported progress on initiative target; this is the number of “actuals” from which samples are selected (e.g., 80, 64).

• EC-Claimed Initiative Status: Whether an EC has met its target based on EC self-reporting (e.g., target met, target not met); for initiatives where the EC has not met its target, the IE must provide rationale for missing the target (e.g., due to permitting delays and staff turnover).

• Sample Size: The number of samples selected by the IE for verification (e.g., 1,400, 800) based on the population as defined in Appendix B. For small populations where the sample size is equal to population size, the IE should use the population size.

• Sample Validation Rate (%): The percent of samples that were found compliant by the IE (e.g., 90%, 83%); for example, if the IE sampled 400 sites and found the EC initiative compliant with 380 of the sites, the sample validation rate is 380/400 = 95%.

• Verification Method: Identification of method used by IE to verify initiative.

o Field Inspection, o Desk Review: document type and data request number (e.g., SAP Work order

(DR064), Photos of construction (DR039)), o SME interview: include SME interview number (e.g., SME interview (SME02)).

• IE Finding on Initiative and Initiative Validation Rate (%): Indication of whether the initiative was validated or not validated based on the IE’s analysis (e.g., Initiative Validated, Initiative Not Validated) and the Initiative Validation Rate, which is the Sample Validation Rate (%) times the EC-Claimed Progress and then divided by the WMP-Initiative Target. Reminder: the Initiative is Validated if the Initiative Validation Rate is greater than or equal to 95%; else the Initiative is Not Validated.

• WMP – Planned Spend: Budgeted spend reported in the WMP (e.g., $134,500).

• EC-Claimed Actual Spend: Actual spend self-reported in the EC ARC (e.g., $235,000) and % variance between budgeted and actual spend (e.g., +56%).

• Satisfied Risk Reduction Goal: IE finding on whether expected risk reduction goal claimed in the WMP has been met (e.g., yes / no); if not, rationale for why risk reduction was not satisfied (e.g., poles were replaced in lower-risk areas, reducing overall risk mitigation); note that if an EC intended to reduce risk, installation without commissioning does not satisfy the expected risk reduction.

Appendix D: Collateral Observations When verifying initiatives in the field, the IE should pay attention to other collateral observations it can make beyond completion of the work for the specific initiative. The IE should alert Energy Safety and the relevant EC to any conditions that the IE believes poses an imminent safety risk.

Appendix E: QA/QC Program Assessment Framework This section describes a structured approach for assessing an EC’s QA/QC program. The assessment is based on five key dimensions, which together ensure comprehensive coverage of the QA/QC processes. The objective is to highlight areas in which the EC is demonstrating strong QA/QC practices, identify best practices that can be shared across the industry, and find opportunities for future improvements for the EC.

Five Dimensions of the QA/QC Framework:

Roles and Responsibilities: Cross-functional allocation of quality-related responsibilities across the organization. This includes assessing how well-defined these roles are and whether all relevant stakeholders are aware of their specific responsibilities in relation to QA/QC. The goal is to ensure that everyone involved in the EC’s activities understands who is responsible for managing quality programs and that the organization operates cohesively to achieve its QA/QC targets.

Quality Culture: The organization’s quality culture includes fostering and encouraging quality-conscious behavior among all employees. It is essential to evaluate whether there is organization-wide awareness of the importance of quality and whether this awareness translates into behaviors and practices that prioritize quality in all activities. A strong quality culture is one where employees, regardless of their role, are committed to maintaining high standards.

Quality Management System (QMS): The IE should evaluate the EC’s Quality Management System, which should document QA/QC processes, procedures, and data necessary for achieving quality targets and objectives. The effectiveness of the QMS is measured by its ability to standardize processes, leading to sustainable improvements and consistency in performance. The assessment should determine whether the QMS is robust enough to support the organization’s QA/QC efforts and whether it aligns with recognized industry standards.

Quality Inspections and Audits: The IE should consider the systematic execution and follow-up of quality inspections and audits. Regular inspections and audits are crucial for identifying areas that require improvement and ensuring that quality standards are consistently met. The assessment should review how effectively the EC conducts these activities, how it uses the resulting data to highlight areas for improvement, and how it implements procedural changes that impact operational effectiveness.

QA/QC Technology Adoption: Adoption of advanced technology can enhance the efficiency and efficacy of QA/QC activities. This includes evaluating the types of technology the EC has adopted and how these technologies are integrated into their QA/QC processes. The goal is to determine whether technology is being used effectively to improve QA/QC programs, activities, and outcomes.

Contextual Considerations:

It is important to recognize that different ECs will have varying levels of resources, capabilities, and requirements. This means that the benchmarks for what constitutes satisfactory or exemplary practices will differ depending on the context. For instance, a small EC with limited resources might consider a well-maintained Excel-based QMS to be satisfactory, while a larger EC might require a more sophisticated digital system to achieve the same designation.

Therefore, the assessment should be tailored to the specific circumstances of each EC, considering their size, scope, and the challenges they face. What is satisfactory for one EC could be exemplary for another, depending on these contextual factors.

For each of the five dimensions, the IE should identify a maturity level the IE judges that the EC has attained based on the following five maturity level metrics listed in the following boxes, one for each dimension. These maturity level rating for each dimension will be used to plot the sunburst figure shown in Figure 1.