2024_02_Instruction_107-1-2_ActionandEvaluationImplementation_F.pdf

Instruction 107-1-2: Continuous Improvement Program Action & Evaluation Implementation

February 2024 Version 1.0

Deanne Criswell

Administrator

Date

February 26, 2024 i

Table of Contents

OVERVIEW

PURPOSE

OVERARCHING APPROACH

SECTION 1: FINDING DEVELOPMENT

DISPOSITION OF DRAFT FINDINGS & VALIDATION

RECOMMENDED COURSES OF ACTION

PATTERN IDENTIFICATION & PRIORITIZATION

SECTION 2: REVIEW & REFERRAL

CONTINUOUS IMPROVEMENT PROGRAM SUBMISSION

REGIONAL LEADERSHIP REVIEW

LEVEL OF RESPONSIBILITY DETERMINATION

DIRECT TO RESPONSIBLE PROGRAM OFFICE

CONTINUOUS IMPROVEMENT WORKING GROUP OR HEADQUARTERS COMPONENT REFERRAL

DETERMINATION

DECISION MAKER IDENTIFICATION & REFERRAL

SECTION 3: DECISION

DECISION REQUEST

RESPONSIBILITY REVIEW

INITIAL ACTION DETERMINATION

ACTION MANAGER ASSIGNMENT

SUPPORTING PROGRAM OFFICES & STAKEHOLDER ASSIGNMENTS

SECTION 4: ACTION

DEVELOPMENT

REPORTING

CLOSURE

Instruction 107-1-2 ii

ACTION COMPLETION

SECTION 5: FINDING COMPLETION

FINDING COMPLETION

SECTION 6: EVALUATION

EVALUATION SELECTION & PLANNING

EVALUATION MONITORING

COLLECTION & INITIAL EVALUATION RESULTS

EVALUATION DETERMINATION

SECTION 7: INFORMATION SHARING & ANALYSIS

TRANSPARENCY OF CONTINUOUS IMPROVEMENT PROGRAM INFORMATION

REPORTING

OTHER RESOURCES

REQUIRED APPROVALS

DATA VISIBILITY IN CONTINUOUS IMPROVEMENT PROGRAM SYSTEMS

AUTHORITY HIERARCHY DIAGRAM

ADDITIONAL INFORMATION

REVIEW CYCLE

AUTHORITIES

REFERENCES

DEFINITIONS

MONITORING AND EVALUATION

QUESTIONS

Overview

Instruction 107-1-2 provides the Continuous Improvement Program (CIP)1 and responsible program office (RPO) personnel with requirements and guidance for managing findings from after-action reviews (AAR), determining actions to be taken on findings, and tracking and evaluating accomplishments to continuously improve disaster operations.

This Instruction expands on concepts and governance in Federal Emergency Management Agency (FEMA) Directive 107-1: Continuous Improvement Program, which outlines the high-level process and expectations for management of findings and actions in Major Disaster Declarations (MDDs).2 This Instruction explains the processes and rules covering the Action and Evaluation phases of the CIP four-phased assessment process.3

This Instruction also establishes the information requirements for findings from AARs to support actionability and provides a standardized approach across CIP to review and refer findings and recommended courses of action to RPOs, request action decisions from RPOs, track actions taken to address findings, and ultimately evaluate findings and actions to confirm improvement.

This Instruction establishes baseline requirements and expectations. Individuals and regions may exceed the requirements or implement additional protocols and processes within the scope of their purview and resources. Any additional requirements must be managed by the individual or region while maintaining documentation within the authoritative data source4 for AAR findings and actions.5

The implementation of this Instruction will, among other things, enable trend analysis to encourage action on issues that have greater frequency and effect, and enable Agency

1 CIP, which is in Resilience, includes staff in the Continuous Improvement Division (CID) based at FEMA Headquarters (HQ), Regional Continuous Improvement Coordinators (RCIC) based throughout the FEMA regional offices, and field staff, including Continuous Improvement Coordinators at Recovery Offices.

2 CIP is responsible for conducting AARs for real-world incidents that receive Major Disaster Declarations.

While rare, CIP may develop AAR products for other real-world incidents if the Department of Homeland Security (DHS) Secretary or FEMA Administrator determines it is appropriate for FEMA to generate such reports. Regional Administrators (RA), Associate Administrators, and Assistant Administrators may also request AAR products (FEMA Directive (FD) 107-1: Continuous Improvement Program (2023), p. 3, 4).

3 CIP conducts assessments through a consistent, replicable, and methodologically appropriate approach by following a defined, four-phased process. The four phases are: Discovery, Validation, Action, and Evaluation (FD-107-1, p. 4).

4 FD-103-1: Data Management (June 2018).

5 FEMA Data Lexicon.

https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/198/Directive%20-%20Continuous%20Improvement%20Program.pdf https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/198/Directive%20-%20Continuous%20Improvement%20Program.pdf https://usfema.sharepoint.com/teams/OPPAEAD/EDAMI/DM/Data%20Management%20Directives%20and%20Instructions/Forms/AllItems.aspx?id=%2Fteams%2FOPPAEAD%2FEDAMI%2FDM%2FData%20Management%20Directives%20and%20Instructions%2FData%20Management%20Directive%2Epdf&parent=%2Fteams%2FOPPAEAD%2FEDAMI%2FDM%2FData%20Management%20Directives%20and%20Instructions https://usfema.sharepoint.com/teams/OPPAEAD/EDAMI/DM/SitePages/Data-Lexicon.aspx personnel to independently access AARs results and improvement actions while maintaining the trust of those who contribute and programs seeking to improve.

Purpose

Establish a consistent process by which FEMA determines actions to take to address findings from CIP AARs and tracks and evaluates accomplishments to continuously improve disaster operations.

Overarching Approach

During the Action and Evaluation phases, CIP facilitates the referral and tracking of three key results of the analytical process for AARs: findings, recommended courses of action (COAs), and actions (see Figure 1 ).

CIP collects information about disaster operations for every MDD and identifies findings and observations by conducting structured and repeatable assessments across the disaster lifecycle. Findings and observations6,7 are the objective analytical conclusions resulting from a CIP collection categorized as a strength, potential best practice, or area for improvement. Findings and observations are then documented in an AAR product.

For each finding, CIP works with pertinent subject matter experts (SMEs) from FEMA field, regional, and HQ program offices, as well as state, local, tribal, and territorial (SLTT) stakeholders (as appropriate), to validate findings and develop recommended COAs which suggest activities that they believe will address the root cause(s) of the finding. Regional Continuous Improvement Working Groups (CIWGs) or HQ program components8 review and refer findings and recommended COAs from AARs to the program office with primary responsibility to address the finding, and request that the RPO decide whether to take action.

6 Findings include in-depth analysis of quantitative and qualitative data, should consider root cause(s), and include evidence that elicits higher confidence in the accuracy of the results. Generating findings is the primary objective of CIP collections. Observations at a minimum include qualitative data from multiple objective sources; however, the evidence may be less detailed or comprehensive on a topic and therefore may warrant additional analysis by the RPO to determine an appropriate action. (FD-107-1, p. 18).

7 Unless observation is specifically termed in the description, the term finding will be used throughout this document to describe either a finding or an observation as the process would be applied in the same way.

8 FEMA Instruction 259-3-1: Planning, Programming, Budgeting, and Execution defines “components” as:

FEMA [regions,] programs, and offices; including but not limited to, offices headed by a Deputy Administrator, Associate Administrator or equivalent, Director, Chief Counsel, Chief Financial Officer, as well as other offices within the Office of the Administrator. [For purposes of this Instruction, regions are excluded as the CIWG is the appropriate authority for functions described.] https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/198/Directive%20-%20Continuous%20Improvement%20Program.pdf https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/310/Instruction%20-%20Planning%20Programming%20Budgeting%20and%20Execution.pdf

If an RPO accepts a recommended COA for action, the RPO will then develop an action

Figure 1: Action & Evaluation Lifecycle outlining the specific details of the measures the RPO plans to take to address the finding and remedy deficiencies or standardize strengths or best practices. The RPO provides updates on the status of actions no less than every 90 calendar days as they work toward completing it. If at any point a RPO can no longer commit to working on an action, the RPO may request it be withdrawn before completion. When a RPO completes an action, they notify CIP with a status update and submit documentation showing that the action has been accomplished.

As time and resources allow, CIP conducts a post-completion evaluation to assess recurrence of the finding. A new finding is developed based on the results of the evaluation to capture if the actions were successful or unsuccessful in addressing the root cause(s) of the original finding. The CIWG or HQ component may determine that multiple evaluations are required before designating an evaluation as successful or unsuccessful. If unsuccessful, CIP works with RPOs to consider what additional action is required to improve the outcome and routes the evaluation finding to the RPOs for continued improvement and tracking. If successful, CIP works with RPOs to determine opportunities to disseminate the results of the evaluation across the Agency so others may consider similar actions.

CID maintains two systems to manage this process and ensure Agency staff can learn from AAR findings and actions. CID maintains the Continuous Improvement Information System (CIIS) which is the information management system intended to manage data collection; develop, track, and analyze findings, recommended courses of action, and program actions; and catalog resulting AAR products. CIIS provides restricted access to licensed CIP personnel on a case-by-case basis, designed for individuals actively involved in the AAR development or action tracking process. This serves as the authoritative data source for AAR findings, recommended COAs, actions, and all associated workflows. Additionally, CID maintains the Comprehensive After-Action Review Analysis Tool (CAARAT) to serve as a searchable library of AAR findings and actions which is accessible to all FEMA staff.

Section 1: Finding Development

Outcome: CIP creates and delivers products that drive decision-making, strategy development, and operational improvements for leadership and staff.

CIP performs both qualitative and quantitative analysis to develop findings as part of the AAR process. FD-107-1 and Instruction 107-1-19 establish expectations and requirements for these analysis results. This Instruction extends this guidance by establishing additional criteria to assist RPOs in making informed choices and clarifying the opportunity for impactful change where issues repeatedly occur or have great consequence in the field.

9 Instruction 107-1-1: After-Action Review Implementation (January 2022).

https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/1059/Instruction-%20After-Action%20Review%20Implementation%201.6.2022.pdf

Disposition of Draft Findings & Validation

During the Validation Phase, CIP works with pertinent FEMA field, regional, and HQ program offices, as well as SLTT stakeholders (as appropriate), to review all findings for accuracy, ensure that conclusions recognize comprehensive perspectives, and identify and corroborate recommended COAs.

1. Before submission to the RCIC (regional collections) or CID leadership (HQ-led collections), any member of the Collection team may permanently delete a draft finding.

2. Once a finding is submitted to the RCIC or CID leadership for review, they may permanently delete a draft finding or choose to close it.

a. Closing a finding also closes all associated recommended COAs and removes it from the RCIC list for submission to CIWG Review but makes it available for CIP internal searches to conduct trend analysis and research.

This is relevant for findings that may be difficult to fully validate but may inform future findings.

b. All personnel who serve a CIP assessment function may access closed findings from any region’s collection for trend analysis.

3. In the process of validating findings with subject matter experts, CIP documents the requests for validation and adjudication of feedback provided. A SME who provides validation feedback should be someone who can speak on behalf of their program with a level of technical knowledge or authority beyond the average program staff.

a. Validation with the expected RPO and CID Peer Review (for more information on CID Peer Review, see Decision Maker Identification & Referral section) are preferred well in advance of referral for action to add context to the finding and prevent unnecessary requests for revision at the point of decision. This is especially important in the case of referrals to HQ program offices from region-led AARs where the HQ program office is not engaged in the AAR approval or CIWG processes.

b. Information provided by validators used to confirm, clarify, or nullify findings that are inaccurate is considered attributional. Program offices and leadership rely on SMEs to serve this function and may request confirmation that SMEs in positions of trust were consulted. If a potential validator expresses concern in contributing to a finding as a named individual, they may provide written sources of evidence to contribute, expressly request they contribute confidentially, or CIP will attempt to source an alternative validator.

4. If applicable program office leadership or SMEs responsible for reviewing a finding determine that it is not valid and should not be considered for action, then the program will provide a rationale for why the finding is inaccurate. For regional findings, this information will be available to the regional CIWG for visibility.

5. The Regional Administrator (RA), or their designee, has final approval authority over the validation of findings and approval of after-action review products from operations in their region and can also request the reopening, or closure of findings as needed.10

Recommended Courses of Action

A recommended COA is a suggestion or a set of suggestions to a program office of action(s) that may address the root cause(s) of a finding.11 While recommended COAs for areas for improvement seek to address deficiencies, recommended COAs for strengths and potential best practices seek to standardize or replicate better than expected performance in other operations.

1. One or more recommended COAs are expected for all findings that are referred for action, including areas for improvement, strengths, and potential best practices.

a. If no effective recommended COAs can be identified, a finding may still be referred for situational awareness.

2. CIP works with pertinent SMEs from FEMA field, regional, and HQ program offices, as well as SLTT stakeholders (as appropriate), to develop recommended COAs.

3. Recommended COA(s) should establish the activity(ies) believed necessary to fully address the root cause(s) of the finding and prevent it from recurring.

4. Recommended COAs are effective when they are addressed to the parties with the responsibility to act, and are specific, feasible, cost-effective, and measurable.12

5. Recommended COAs are part of a finding but only serve a temporary purpose to convey the finding to one or more program offices for consideration.

a. Each COA may only be referred to one RPO, who then may decide to develop the action to address a finding (see Section 2: Review & Referral).

i. If a single complex or interdependent action requires more than one program’s authorities to be successful, other program offices should be considered as Supporting Program Offices (SPOs) later in the process (see Section 3: Decision). However, a single RPO must be designated to serve as the overarching manager of the decision and action processes.

b. If multiple, independent actions from different program offices will be required to address the finding, these should be crafted as separate COAs.

c. Once a RPO makes a decision regarding the recommended COA, the COA no longer serves an active purpose and is maintained for reference.

10 FD-107-1: Continuous Improvement Program (2023), p. 6.

11 FD-107-1: Continuous Improvement Program (2023), p. 18.

12 GAO-21-368G: Government Auditing Standards (April 2021), p. 201.

https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/198/Directive%20-%20Continuous%20Improvement%20Program.pdf https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/198/Directive%20-%20Continuous%20Improvement%20Program.pdf https://www.gao.gov/assets/gao-21-368g.pdf

If an action was created, the action(s) serve as the mechanism to track measures to address a finding.

Pattern Identification & Prioritization

1. CIP will provide the structural capability and criteria to estimate a prioritization rating13 for each finding based upon the likelihood the finding will occur again and the effect of a finding. For each finding, this rating is optional and may occur at any time. When provided, this rating seeks to proactively provide RPOs with a basic understanding of the risk posture when accepting or declining action and differentiate actions with critical effects for operations and less critical actions.

Additionally, it is intended to assist in aligning requests for action with the broader enterprise resource decision process14 and in assessing capability targets.15

2. Likelihood is the chance of something happening.16 CIP defines this in terms of patterns of findings where a pattern is a collection of two or more findings occurring in different disaster operations which describe the same area for improvement, strength, or potential best practice (PBP).

a. CIP will conduct trend analyses, such as the annual Summary of Findings, which may serve to assess the extent to which a finding has occurred previously. Patterns must detail the same problem or strength within a topic or process to be actionable, not simply the topic or process itself.

i. For example, an appropriate pattern would be multiple findings describing “insufficient debris removal guidance for hazardous trees due to wildfire,” whereas a pattern describing just the topic of “debris removal” would be insufficient.

b. Whenever a similar finding that identifies a trend or pattern of recurrence is found, CIP will define a pattern of findings in the CIIS and provide this information to Decision Makers and Action Managers.

c. The research process at a minimum will include a review of other CIP findings, as well as Government Accountability Office (GAO) and DHS Office of the Inspector General (OIG) reports with FEMA referred recommendations.

d. Counterexamples are findings that are associated with the same problem or strength defined in a pattern, but which yielded the opposite effect (e.g., a strength associated with a pattern of areas for improvement).

13 “Accountability quantified: What 26 years of GAO reports can teach us about government management.” (February 2015) recommends a score to assess the importance of an individual recommendation.

14 FD-304-103-2: Enterprise Governance (June 2023) requires that FEMA “identify and assess risks to FEMA's mission, programs, and strategic goals when reviewing, recommending, and deciding upon the Agency's approach to resourcing.” FEMA Instruction 259-3-1: Planning, Programming, Budgeting, and Execution (November 2016) defines the process for the programming phase to allow leaders to distribute limited resources to maximize achievement of the goals, priorities, and desired performance outcomes.

15 Ready FEMA Framework (December 2022), p. 15.

16 DHS Risk Management Fundamentals (April 2011), p. 20.

https://www2.deloitte.com/us/en/insights/topics/analytics/text-analytics-and-gao-reports.html https://www2.deloitte.com/us/en/insights/topics/analytics/text-analytics-and-gao-reports.html https://usfema.sharepoint.com/sites/OPPACom/SPD/Lists/PolicyMasterList/DispForm.aspx?ID=2376&e=LbwN0c https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/310/Instruction%20-%20Planning%20Programming%20Budgeting%20and%20Execution.pdf https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/310/Instruction%20-%20Planning%20Programming%20Budgeting%20and%20Execution.pdf https://usfema.sharepoint.com/sites/OAI/SL/Ready%20FEMA%20Framework/Ready%20FEMA%20Framework%20-%20Final%20Draft%20-%20Signed_508%20Reviewed.pdf https://www.dhs.gov/xlibrary/assets/rma-risk-management-fundamentals.pdf

i. Counterexamples do not contribute to the recurrence count of a pattern.

ii. Counterexamples offer insight for action or serve to contextualize the pattern and situations where additional root cause analysis is required before taking action.

e. A pattern associated with four or more findings in the previous four years will be considered “high recurrence.” CIP uses Table 1: Likelihood Scale to define levels of likelihood.

3. Consequence is the effect or impact of an occurrence.17 CIP uses Table 2:

Consequence Scale to define levels of consequence and, when assigning a rating, work with program SMEs to assist.

4. The likelihood and consequence ratings will be used to focus and spur prioritization and resource conversations in conjunction with other planning efforts (see Section 7: Information Sharing & Analysis). The notional Prioritization Matrix in Figure 2 may be used as a decision support tool in guiding those discussions during the Decision, Action, and Evaluation phases.

Table 1: Likelihood Scale

Likelihood What is the likelihood of this risk occurring in the future?

5-Very Likely “High recurrence” patterns (4 or more findings identifying the same issue in the preceding four years).

4-Likely 3 findings identifying the same issue in the preceding four years.

3-Possible At least 2 findings identifying the same issue in any timeframe.

2-Unlikely No similar findings identified.

1-Very Unlikely No similar findings and highly unusual or unique18 circumstances.

17 Derived from DHS Risk Management Fundamentals (April 2011), p. 20.

18 Examples of “highly unusual or unique” circumstances might include unique disaster types that are unlikely to reappear within four years (e.g., volcano, tsunami) or unique geographies or cultural circumstances that are unlikely to occur in disasters occurring in other places (e.g., coordination across government entities including a checkerboard tribal entity, or lessons in re-establishing heritage sites during community relocation).

https://www.dhs.gov/xlibrary/assets/rma-risk-management-fundamentals.pdf

Table 2: Consequence Scale

Consequence What Consequence will this risk have on mission program goal progress?19

Examples

5-Very High Extremely degrades ability to carry out functions that result in no progress on achieving objectives and would have catastrophic adverse effects on organizational operations, organizational assets, individuals, other organizations, or the Nation.

Violations of law which could lead to severe scrutiny, penalties, or litigation.

Numerous survivors or state, local, tribal, and territorial (SLTT) partners unable to receive eligible assistance.

For Strengths/Potential Best Practices (PBP) – prevented or addressed this degraded functionality.

4-High Severely degrades ability to carry out functions that directly impact ability to make progress on achieving objectives and would have a severe adverse effect on organization operations, organizational assets, individuals, other organizations, or the Nation.

Many survivors or SLTT partners were significantly delayed in receiving eligible assistance (well beyond programmatic performance metrics).

For Strengths/PBPs – prevented or addressed delays that could have affected many or significantly increased quality or timeliness of assistance.

3-Moderate Somewhat degrades the ability to carry out certain functions that directly impact ability to make progress on achieving objectives and would have a serious adverse effect on organization operations, organizational assets, individuals, other organizations, or the Nation.

Some survivors or SLTT partners were moderately delayed in receiving eligible assistance (beyond programmatic performance metrics).

For Strengths/PBPs – prevented or addressed delays that could have affected some or moderately increased quality or timeliness of assistance.

19 FY2022 FEMA Operational Risk Register – Risk Rating Scales.

https://usfema.sharepoint.com/:x:/r/teams/RESNPD/NPAD/CII/_layouts/15/Doc.aspx?sourcedoc=%7B11118F1D-445D-4837-8BE6-9C066B1CDF8F%7D&file=Copy%20of%20FY2022%20SOA%20-%20Attachment%20E%20-%20Updated%20FEMA%20Operational%20Risk%20Register.xlsx&action=default&mobileredirect=true

Consequence What Consequence will this risk have on mission program goal progress?19

Examples

2-Low Temporarily disrupts the ability to make progress on achieving objectives and could be expected to have a limited adverse effect on organization operations, organizational assets, individuals, other organizations, or the Nation.

Confusion or inconvenience required additional staff time and effort to resolve an issue.

For Strengths/PBPs – supported internal operational efficiency and communications with no external (i.e., SLTT, partner) effect.

1-Very Low Little or no adverse effect on achieving objectives and could be expected to have a negligible adverse effect on organizational operations, organizational assets, individuals, other organizations, or the Nation.

[CIP does not identify findings with very low consequence]

Figure 2: Prioritization Matrix (notional)

Section 2: Review & Referral

Outcome: FEMA region and HQ leadership are empowered to codify issues and define responsible parties to track corrective actions.

Continuous Improvement Program Submission

Once the validation process is complete, CIP reviews the final finding and recommended COAs to confirm that they meet CIP standards and are ready for review.

For region-led AARs, Regional Continuous Improvement Coordinators conduct this review. For HQ-led AARs, CID staff conduct this review.

1. CIP reviews all findings and recommended courses of action to ensure they are accurate, actionable, and significant.

a. If CIP determines that an analytical conclusion is not comprehensively assessed or does not contain enough evidence to support the finding’s assertion, it will be designated as an observation with the option to close or move forward for further review.

Figure 3: Finding Review Process

2. For each recommended COA, CIP also may provide a suggested “level of responsibility” to address the action; the options for levels of responsibility are field, region, or HQ. CIP may also identify the anticipated RPOs associated with the COA, if known. These are not binding designations but offer planning considerations to leadership and program staff before the findings are approved.

3. While developing an AAR, some findings will be developed and validated more quickly than others. To provide timely information to leadership and program offices, it is recommended that findings be reviewed as soon as validation is complete rather than waiting for AAR product finalization.

a. If this is not feasible for regions, particularly those conducting AAR approval processes which require finalization of all findings for a collection before referral, that is understandable and appropriate. However, once all findings for a collection are included in a final AAR product (submitted within 15 days of finalization),20 findings and recommended COAs must be documented in the authoritative data source within a further 30 calendar days for a total of 45 calendar days after finalization. Adjustments to findings, recommended COAs, and actions must subsequently be documented in the authoritative data source on an ongoing basis along with all associated workflow decisions/determinations, and statuses.

4. Upon submission of a finding for Regional Leadership Review, a finding may no longer be permanently deleted; the finding may only be closed.

a. Whenever a finding is closed, CIP captures the rationale for that decision.

b. CIP will directly communicate finding closures throughout the process with the regional CIWG and HQ program components.

c. At any point after a finding has been closed a RPO Decision Maker may request the finding and/or recommended COA be reopened so they may take action to address it. If the finding was closed before an RPO Decision Maker being identified, the RA or regional CIWG has the discretion to review the rationale and request that CIP reopen the finding.

5. Throughout subsequent stages in this Instruction, parties are given the opportunity to request revisions to the finding or recommended COAs. The originator of the finding (i.e., RCIC or CID) maintains ownership over this content.

Others (e.g., RPOs) cannot make revisions independently. Any request for adjustment to the finding or recommended COAs are referred back to the RCIC for region-led AARs or to CID for HQ-led AARs.

Regional Leadership Review

For region-led AARs, before referral directly to a program office, regional leadership reviews the finding and recommended COAs to determine if they are ready to move forward for CIWG Review, RPO Initial Action Decision, or require additional revisions.

1. Regional Leadership Review will be conducted by the Federal Preparedness Coordinator (FPC), Federal Coordinating Officer (FCO), or CIWG Chair at a

20 Instruction 107-1-1: After-Action Review Implementation (January 2022), p. 18.

https://usfema.sharepoint.com/teams/OPPAPolicy/Lists/Policy%20Master%20List/Attachments/1059/Instruction-%20After-Action%20Review%20Implementation%201.6.2022.pdf minimum. Regions may establish a higher required level of authority (i.e., RA, Deputy RA) or additional required leadership review (i.e., Division Directors) at their discretion. This responsibility should be designated in the region’s AAR product approval process.

a. If a finding is referred through the CIWG and/or the region does not allow a direct to RPO referral, this step is optional as it is duplicative of CIWG determination.

b. For HQ-led AARs, this process occurs during HQ review with program components and therefore is not duplicated.

2. Regional leadership conducts a review of the finding and has three options: (1) approve the finding, (2) request a revision, or (3) close the finding.

a. When reviewing findings, regional leadership should consider the same criteria used during CIP Submission; findings must be accurate, actionable, and significant.

b. If approved, regional leadership either makes the level of responsibility determination (see Level of Responsibility Determination section) and advances the finding directly to the RPO, in regions where this is allowable (see Direct to section), or designates the finding to move forward to the CIWG for review and determination.

c. If revision is requested, the finding is referred back to the RCIC. A comment is required describing the necessary revisions to ensure the RCIC has enough detail to make changes as expected.

d. Closing a finding is requested when regional leadership determines that a finding is not valid or appropriate to move forward. The rationale is captured for this decision.

e. Regional leadership may determine that an analytical conclusion is not comprehensively assessed or does not contain enough evidence to support the finding’s assertion. In this case it will be designated as an observation with the option to close or move forward for further review.

Level of Responsibility Determination

For each recommended COA, the appropriate level of FEMA’s organization to address a recommended COA must be determined (level of responsibility) before referral to a program office for decision. Recommended COAs can have a level of responsibility determined to be either: field, region, or HQ. For region-led AARs, regional leadership or the CIWG determines the level of responsibility. For HQ-led AARs, CID leadership determines the level of responsibility.

1. For region-led AARs, regional leadership defines the level of responsibility and works through regional RPOs and the CIWG to confirm it is accurate.

2. For HQ-led AARs, CID defines the level of responsibility and works through component senior advisors to confirm it is accurate.

3. Recommended COAs with field specified as the level of responsibility are considered disaster-specific, where the scope or responsibility to act on the recommended COA should be completed by staff deployed to a single disaster.

4. Recommended COAs with “region” specified as the level of responsibility may be relevant for a single disaster or multiple operations within a region and can be completed by regional staff. These recommended COAs are unlikely to be appropriate for Agency-wide action or may be pilots that could serve as precursors to enterprise action.

5. Recommended COAs with “HQ” specified as the level of responsibility are those that exceed the region’s scope, responsibility, or resources and have operational consequences requiring Agency action.

Direct to Responsible Program Office

In order to expedite actions, regions may allow regional leadership to route recommended COAs directly to regional RPOs or FCOs without approval through the

CIWG.

1. As CIP and regional leadership work through the validation process, it often becomes apparent that program offices are immediately willing to act. To expedite and encourage this, regional leadership may refer recommended COAs with a regional level of responsibility directly to the program office or with a field level of responsibility directly to the FCO bypassing CIWG review.

a. The CIWG must pre-approve the region’s choice to allow use of the “Direct to RPO” option.

2. Regional leadership designates the recommended COA as “Direct to RPO” then defines the RPO and the RPO Decision Maker.

3. CIP requests an RPO action decision about whether the program intends to act.

For actions accepted, the CIWG will have awareness for tracking purposes. If the RPO declines action or indicates they are not the correct program office, the recommended COA will be routed back to the CIWG for determination.

Continuous Improvement Working Group or Headquarters Component Referral Determination

For each recommended COA, the CIWG or program component reviews the finding and recommended COAs for referral to a program office or acknowledges the regional RPO decision.

1. Each region will establish and maintain a CIWG, or related body, to support referral of findings and recommended COAs to appropriate program offices, assign and track actions, confirm completion of actions, and review the results of evaluations.

a. For region-led AARs, the regional CIWG serves as the reviewing body and RCICs facilitate their review.

b. The CIWG should be convened at least four (4) times per year to fulfill its roles and responsibilities. Regions with more disasters and resulting AARs may need to meet more frequently.

2. For HQ-led AARs, program component leadership serve as the reviewing bodies and the CID Director facilitates their review.

3. If valid and appropriate, the reviewing body makes a CIWG/Component Referral Determination21 to refer the finding for action.

a. Recommended COAs whose findings are approved for referral— either for action or situational awareness—are also approved for referral by default unless individually designated for closure.

b. An individual recommended COA can be closed before referral in order to keep a record of the suggested measures which were not determined to move forward for action.

4. If the reviewing body deems that an analytical conclusion is not comprehensively assessed or does not contain enough evidence to support the finding’s assertion, it will be designated as an observation and may be closed or referred to the RPO for situational awareness or referred for action.

a. A finding may be referred for situational awareness if it is not fully actionable but still likely to be of interest to a RPO (e.g., an observation that relates to a pre-identified RPO priority or an observation that has limited evidence but anecdotal merit). Additionally, if a finding is not appropriate to Refer for Action (e.g., program is in a pilot phase or the effect is clear but negligible) or the finding is a strength or PBP and therefore recommended COAs are less relevant, it may be referred for situational awareness.

b. A finding or observation that is referred for situational awareness will prompt an RPO to acknowledge they have reviewed it, but an action decision is optional.

5. If the reviewing body deems that additional research, validation, or adjustment to the finding is possible and necessary before referring the finding for action, the body makes a CIWG/Component Referral Determination to request revision of the finding or recommended COA(s).

a. When requesting revisions, the reviewing body must include a comment to specify the actions and revisions necessary.

6. The reviewing body may also make a determination to extend the referral decision to reassess at a later date. Finding referrals may be extended for any reason, but often it is because other pending research or activities may influence recommended COAs. Extended finding referrals should be reconsidered at least every 30 calendar days to ensure timeliness of finding referral.

7. If the reviewing body makes a determination that the finding should not be pursued for revision or future action, the body makes a CIWG/Component Referral Determination to close the finding.

a. When closing a finding, the reviewing body provides a rationale for this decision.

21 CIWG/Component Determination options are: Refer for Action, Refer for Situational Awareness, Request Revision, Request Extension, or Close.

Decision Maker Identification & Referral

Each recommended COA is referred to a Decision Maker for a RPO who makes an action decision.

1. Recommended COAs that are approved for referral, either for action or situational awareness, require identification of a Decision Maker within a RPO who is responsible for making an action decision.

a. An RPO Decision Maker is an individual authorized to approve or decline responsibility for, acceptance of, and assignment of resources within a RPO to address recommended COAs. This level of authority is at the branch leadership or above.

b. The referral process operates differently depending on the level of responsibility (i.e., HQ, region, field).

2. For field recommended COAs, the FCO for the relevant disaster is determined to be the Decision Maker unless they specifically delegate this authority to the Deputy FCO or Chief of Staff (COS).

a. The CIWG will identify the current FCO as the RPO Decision Maker and refer the finding for an action decision.

i. If the region allows for a direct to RPO referral, regional leadership (FPC or above) may identify the current FCO as the RPO Decision Maker and refer the finding directly.

b. If the collection disaster is not an MDD, then an equivalent leader managing the disaster is determined to be the Decision Maker if they have the authority to approve the assignment of resources.

3. For region recommended COAs:

a. The CIWG determines the Decision Maker who is generally the relevant

RPO Division Director but may be branch leadership or above.

i. RPO Division Directors, or their designees, participate as members of the CIWG and therefore are already engaged in the decision-making process to accept the finding for action. To prevent duplication of decisions that have already occurred, CIWG approval for referral indicates that the regional RPO has agreed to take action. The Division Director over the RPO assigns appropriate staff or resources to develop and implement an action.

b. If the region allows for a direct to RPO referral, regional leadership (FPC or above) will coordinate with the program office to identify the RPO Decision Maker and refer the finding directly.

4. For HQ recommended COAs:

Figure 4: Finding Referral Process

a. CID conducts a peer review22 which considers whether a finding has an appropriate depth of analytical rigor to warrant Agency programmatic action, and if there are opportunities to make the argument for action more persuasive. In this peer review, CID may refer the finding and recommended COA for action, Refer for Situational Awareness, or Request a Revision. The peer review will be conducted within 14 calendar days.

b. If the CID Peer Review Determination is to Refer for Action, then CID has identified the finding as appropriate to move forward and defines an HQ RPO for each recommended COA in order to request an action decision.

i. If CID has certainty of the RPO and the ability to directly coordinate with the program office, the CID identifies the RPO Decision Maker and refers the finding directly.

ii. In all other cases, CID proposes a RPO and routes the finding and recommended COA to the appropriate Action Office. The Action Office then further identifies the correct RPO and the appropriate Decision Maker.

c. If the CID Peer Review Determination is to Request Revision, the finding is referred back to the RCIC. CID will provide a clear comment with suggested edits to the region for consideration.

i. This peer review does not mandate revisions but serves as a quality control step in the process to ensure findings include root cause analysis and information about operational consequences that enables RPOs to take appropriate action.

d. If the CID Peer Review Determination is to Refer for Situational Awareness, then either the CIWG/Component Determination-requested referral for situational awareness or revisions to the finding did not fully address requirements.

i. Following the same process for referral for action, CID identifies the most likely RPO and routes the finding and recommended COA to the appropriate Action Office to further identify the correct RPO and identify the appropriate Decision Maker.

Section 3: Decision

Outcome: Program offices utilize findings and recommended COAs to identify, understand, and make informed choices in addressing programmatic challenges.

22 HQ Peer Review Determination options are: Refer for Action, Request Revision, or Refer for Situational Awareness.

Decision Request

Once a finding and recommended COA are referred to the RPO Decision Maker, a response is requested from the Decision Maker within 14 calendar days.

1. The RPO is the office with primary responsibility to develop and implement an action to address an identified finding and recommended COA.

a. Each recommended COA and action may only be assigned to one RPO.

b. Complex actions may require engagement of more than one program office (see Supporting Program Offices & Stakeholder Assignments section). However, the program office with the primary responsibility to address a recommended COA is considered the RPO. In cases of equal authority or responsibility across multiple program offices, the program office designated to facilitate progress updates is considered the RPO.

2. When requesting review of the finding, CIP will include the likelihood and consequence rating to enable the program office to make an informed decision (see the Pattern Identification & Prioritization section).

a. If a finding is either high recurrence or high/very high consequence, CIP will seek to communicate directly with the RPO to discuss and encourage action to address the finding.

b. In cases where process improvement or program consulting could eliminate the barrier to action for high recurrence or high consequence findings, pending resource availability, CIP may provide planning and facilitation support to assist the RPO as they work to identify, develop, or plan implementation of actions.

c. While CIP supports and coordinates with program offices to address findings, program offices are responsible for developing and implementing actions.

Figure 5: Action Decision Process

Responsibility Review

The Decision Maker reviews the referred finding and recommended COA and confirms that the RPO identified has the authority or responsibility to address the finding.

1. The RPO Decision Maker identified reviews the finding and the recommended COA and can indicate that they are not the correct RPO, meaning that the program office does not have the authority or responsibility to address the finding or recommended COA.

2. If the Decision Maker indicates the RPO is correct, they are requested to make an Initial Action Determination (see Initial Action Determination section for additional details).

a. If the Decision Maker indicates the RPO is not correct, they are prompted to provide a comment clarifying their role or lack of responsibility. CIP then facilitates a discussion with the reviewing body (i.e., CIWG or HQ component) to adjudicate responsibility.

i. If the reviewing body agrees that the program office identified is not responsible, they identify a different RPO and Decision Maker.

ii. If the reviewing body determines the program office does have responsibility, they communicate with program office leadership and request that the RPO make an Initial Action Determination.

3. For regional level of responsibility-recommended COAs that were referred through the CIWG, a separate responsibility review is not conducted because the RPO agreed to responsibility during the CIWG process.

Initial Action Determination

The RPO Decision Maker provides a determination23 regarding their intent to take action to address a finding and recommended COA.

1. The RPO Decision Maker accepts the finding if they intend to take action to address the finding or designates the finding as previously complete if the RPO has already completed an action which they believe has addressed the finding in the time since it was identified.

a. If accepted, the RPO may choose to build their action based on the recommended COA provided or create an entirely different action if the RPO feels it will more appropriately address the finding. Additionally, the RPO may add more than one action within their responsibility to address the finding.

b. If previously completed, the RPO provides a short narrative on the project and any attachments which document the action’s completion and effect.

The Decision Maker may complete this or may assign an Action Manager (see Action Manager Assignment section) to provide the narrative and

23 RPO Initial Action Determination options are: Accept [Previously Completed], Decline, Revise, [Incorrect RPO].

documentation. This action then follows the procedures for Completion Review (see Action Completion section).

c. If the finding is part of an identified pattern and the RPO Decision Maker feels that one or more of the ongoing actions will address the finding, the RPO Decision Maker may associate new actions with the ongoing finding and action. This reduces the duplication of reporting for the Action Manager.

d. For regional level of responsibility recommended COAs that were referred through the CIWG, the action determination must result in acceptance because the RPO agreed to take action during the CIWG process.

2. The RPO Decision Maker declines the finding and recommended COA if they are unable to take action to address the finding.

a. CIP expects that some RPOs will decline actions given programmatic priorities and finite resources; when this occurs the RPO will provide a rationale and explanatory comment for this decision (e.g., lack of staffing resources, lack of financial resources, not a current priority, finding not accurate, finding no longer relevant, leadership direction, other).

b. The originator of the finding is notified of the declination and rationale for awareness and the recommended COA is closed.

3. The RPO Decision Maker may also request a revision to the finding if they determine it is essentially accurate but requires adjustment to be more accurate, clear, or actionable.

a. If the RPO requests a revision to the finding, the RPO will provide suggested edits to the originator (i.e., RCIC or CID) for consideration.

b. Once revisions are considered, the originator resubmits the finding and decision request back to the Decision Maker who requested the revision with a comment on how the request was adjudicated.

c. For findings with multiple recommended COAs, there may be multiple concurrent decisions and/or requested adjustments. Finding language may vary as the originator of the finding responds to requests for revision.

4. If the RPO Decision Maker needs additional time to make a decision, they may request an extension.

a. The RPO may defer the decision on a recommended COA for up to 90 calendar days from the date of the decision request. In most cases, the Decision Maker will receive reminders through CIIS to complete the decision at 30-, 60-, 80- and 90-day increments. If no decision has been made at 91 calendar days, the recommended COA will be returned to the CIWG or HQ component for discussion and/or re-referral.

5. CIP provides transparency on all Initial Action Determinations to the appropriate reviewing body based on the level of responsibility.

a. For regional level of responsibility-recommended COAs, the RCIC will provide information to the CIWG and/or RA on any action decisions outside of their CIWG review process (e.g., Direct to RPO). The RA has the authority to require the regional RPO to adjust any decision they do not concur with at this level. This authority may be delegated to the CIWG.

i. The RA or CIWG (if delegated) will have the option to request that CIP resubmit the entire finding for CIWG review or to resubmit the Initial Action Determination to the RPO Decision Maker of their choice.

b. For HQ level of responsibility recommended COAs, CID will provide information to the HQ component regarding Initial Action…