Sources Sought/RFI | 1 September 2021

SOURCES SOUGHT/REQUEST FOR INFORMATION (RFI)

I. CONTRACT OPPORTUNITY

+ Name: FAA Safety Culture Assessment + Notice ID: 693KA9-27-FAA Safety Culture Assessment + Related Notice: N/A + Department/Ind. Agency: TRANSPORTATION, DEPARTMENT OF + Sub-tier: 6920 FEDERAL AVIATION ADMINISTRATION + Office: AAQ-460

II. GENERAL INFORMATION

+ Contract Opportunity Type: Sources Sought (Original) + Original Response Date: N/A

III. CLASSIFICATION

+ Product Service Code: R431 SUPPORT- PROFESSIONAL: HUMAN RESOURCES + NAICS Code: 541612 Human Resources Consulting Services + Place of Performance: Vendor and Various FAA Locations

IV. DESCRIPTION

The acquisition strategy for the procurement has not been determined at this time; however, either an unrestricted competition or a set-aside may be chosen depending on the responses to this Sources Sought / RFI Notice.

a) This is not a Screening Information Request or Request for Proposals of any kind;

b) The FAA is not seeking or accepting unsolicited proposals;

c) The FAA will not pay for any information received or costs incurred in preparing the response to the market survey; and

d) Any costs associated with the market survey submittal are solely at the interested vendor’s expense.

State that the FAA is seeking sources or information for specific supplies and or services. Provide a description of the supplies and/or services to be provided as part of the potential requirement. The length of the description will vary depending on the nature of the requirement.]

THIS IS NOT A REQUEST FOR PROPOSALS OR OFFERS. A SIR/Request for Proposal may be made under a separate announcement in the future. The FAA is not seeking or accepting unsolicited proposals. The FAA will not pay for any information received or costs incurred in preparing vendor responses to this market survey.

Therefore, any cost associated with a market survey submission is solely at the interested vendor’s expense.

The purpose of this announcement is to solicit statements of interest and capabilities from large businesses, small businesses, service-disabled veteran-owned small businesses and 8(a) certified firms, capable of providing the supplies outlined in this document, in the timeframe stated in this document. The responses to this RFI will help determine the acquisition strategy and whether adequate competition exists to set-aside the competition among small businesses, service-disabled veteran-owned small businesses or Socially & Economically Disadvantage Business (SEDB/8(a)) certified firms. Below is a list of required items. Please see attachments for specific information.

Note: The FAR references cited in SAM.gov are not applicable to the Federal Aviation Administration (FAA) as the FAA has its own policies and guidance referenced in the Acquisition Management System (AMS). https://fast.faa.gov/ https://fast.faa.gov/PPG_Procurement.cfm https://fast.faa.gov/ https://fast.faa.gov/PPG_Procurement.cfm

Sources Sought/RFI | 2 September 2021

V. ATTACHMENTS

Attachments

1. Draft Requirement & Submission Instructions

2. The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations (2024)

VI. Contact Information: Contracting Office: AAQ-460; E-mail: 9-AFN-AWA-AAQ460-Contracting-Team@faa.gov mailto:9-AFN-AWA-AAQ460-Contracting-Team@faa.gov

Draft Requirement Scope FAA is conducting market research to identify qualified industry sources capable of performing a comprehensive, multi-phase safety culture assessment. Organizational culture factors that contribute to safety, and that impact FAA’s safety management and safety oversight capabilities, should be included in the assessment. The selected partner will conduct a multi-method assessment of the safety culture within the FAA, develop a proposed process by which the FAA can conduct recurring culture assessment, and recommend improvements that would advance FAA’s safety culture maturity. The multi-method assessment will include a climate survey, site visit pilot, and environmental scan of existing data and leading practices.

The selected team is expected to:

• Establish or Liaise with Governance and Develop and Manage Detailed Requirements

• Conduct Climate Survey of FAA Workforce and Manage Safe Handling of Confidential Employee Data

• Conduct a Current State Environmental Scan of Existing Data, Artifacts, and Leading Practices

• Conduct Site-Based Assessments of 3-5 Pilot Sites Including Qualitative and Observational Analysis

• Deliver a Baseline Results Technical Report

• Deliver an Approach to Recurring Culture Assessment Technical Report

• Deliver an Improvement Plan Roadmap

The final deliverable(s) for this initiative will be the baseline results technical report, a technical report providing FAA a framework and guidance for conducting ongoing assessment, and a report giving the FAA a roadmap for enterprise improvement. The future award would be Firm Fixed Price.

Vendor Capability Questions – No more than 10 pages

Interested vendors must directly address the following questions within their capability statement submission:

• Methodological Expertise: Describe your organization's experience in designing, statistically validating, and deploying large-scale, enterprise-wide quantitative safety climate surveys. Or, describe how your organization might deploy an existing instrument such as the FAA Safety Culture Survey.

• Qualitative Facilitation: Detail your approach to conducting qualitative listening sessions or focus groups across geographically dispersed and operationally diverse workforces, ensuring psychological safety and high participation rates.

https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/media/202418.pdf

• Federal Experience, Oversight & Compliance: Provide examples of how your culture assessments have successfully aligned with federal mandates, executive orders, or specific regulatory frameworks. Describe how you have accounted for considerations such as labor union coordination, protection of employee information from FOIA, generating high response rates from operational (non-desk job) employees, and other related considerations.

• Reporting & Strategy Delivery: Provide an overview of your data analysis and reporting capabilities, specifically how you translate complex raw quantitative and qualitative data into actionable, high-level executive findings for ongoing organizational strategy.

• Data Security & Privacy: Confirm your organization’s capability to handle sensitive employee feedback data in compliance with federal privacy standards, ensuring complete anonymity for participants.

• Knowledge & Experiences (Optional): Demonstrate your organization’s familiarity with FAA Safety Management System (SMS) policies and regulations, and with guidance and assessment performed by aviation authorities like the International Civil Aviation Organization and Civil Air Navigation Services Organization.

• Organizational Improvement: Demonstrate your organization’s understanding of safety and organizational culture, and your experience helping industry or federal organizations develop and/or implement comprehensive improvement plans and strategy for advancing safety culture maturity.

Required Corporate Capabilities & Vendor Qualifications – No more than 5 pages

• Past Performance: Minimum of 3–5 years of experience executing organizational culture and/or safety assessments of similar size, scope, and complexity, ideally within the federal government or in aviation/transportation or other high-hazard industry sectors. Corporate experience must include development of improvement plans and strategies supportive of assessment results.

• Personnel Qualifications: Availability of key personnel possessing advanced degrees (Ph.D. or Master's) in Industrial-Organizational (I-O) Psychology, Organizational Behavior, Sociology, or a closely related behavioral science field.

• Personnel Availability: Vendors must be able to staff a team of sufficient size and expertise that the team can rapidly conduct multiple methods of assessment (e.g., survey, site visits / interviews) simultaneously, rather than sequentially, in order to deliver a quick-turn report to the FAA and support rapid progression to the improvement planning phase

• Survey Infrastructure: Possession of or access to a secure, commercially field-tested enterprise application or survey platform capable of hosting thousands of concurrent users while protecting data integrity.

• Third-Party Neutrality: Ability to act as an objective, unbiased third-party research team to ensure the credibility of the culture assessment baseline for internal stakeholders and external oversight bodies.

Industry Feedback – No more than 3 pages Industry feedback is requested on the draft requirement. Please list any challenges or potential performance roadblocks related to fulfilling this type of requirement and any suggestions for improving. Feedback is also requested regarding the proposed PSC and NAICS codes.

Comments on PSC/NAICS and suggestions for alternative must include supporting rationale.

Rough Order of Magnitude (ROM) – No more than 2 pages Interested vendors are requested to provide a Rough Order of Magnitude (ROM) estimate based on the preliminary requirements provided in this notice. The ROM will be used strictly for internal market research, feasibility analysis, and budgetary planning. The ROM is for informational purposes only and is not a legally binding commitment or a formal contract offer.

The Government does not require rigorous cost or pricing data. Responses should provide a high-level, "ballpark" estimate broken down by major cost drivers. Vendors are encouraged to briefly state the critical assumptions or risk factors underlying their cost/price ranges. This notice does not commit the Government to contract for any supply or service whatsoever.

Respondents are advised that the Government will not pay for any information or administrative costs incurred in response to this market survey. Not responding to this RFI does not preclude participation in any future RFP, if issued.

Submission Instructions & Administrative Timeline Submission Format: Responses must be submitted via email to the designated Contracting

Officer in PDF format, not exceeding the page limitations above.

Required Administrative Details: Vendors must include their company name, address, Unique

Entity ID (UEI), CAGE code, socio-economic status (e.g., 8(a), SDVOSB, WOSB, Small Business), and designated Point of Contact (POC) information.

Questions and Clarifications: Industry questions regarding this notice must be submitted in writing no later than 10 business days following the posting date.

Final Response Deadline: All capability statements must be submitted no later than the date identified in the SAM.gov announcement.

Government Review Window: The program office, in coordination with the Enterprise Safety Culture Program, will review submissions to determine the appropriate acquisition strategy

Federal Aviation Administration

Aviation Research

Statement of Work

Maturation of FAA Safety Culture Assessment

A. Introduction and Overview

The Federal Aviation Administration (FAA) is an operating mode of the U.S. Department of Transportation responsible for the safety of civil aviation. The FAA’s major roles include issuing and enforcing regulations, encouraging and developing new aviation technology, and managing air traffic in the U.S. airspace. Its continuing mission is to provide the safest, most efficient aerospace system in the world.

The FAA recognizes that an organization’s safety culture (“the shared values, actions, and behaviors that demonstrate a commitment to safety over competing goals and demands”) is integral to effective safety management and to positive safety outcomes.

The National Airspace System (NAS) continues to rapidly evolve and as new entrants and technologies are integrated, safety culture continues to be of critical importance. As a regulator and as an air navigation services provider, the health of FAA’s own internal safety culture plays a critical role in its effective management of safety and on the overall safety of the NAS.

B. Background

The FAA has a requirement to thoroughly document its current safety culture by 2027 to initiate targeted steps for improvement and to provide guidance for recurring safety culture assessment of the organization. In 2027, the FAA must thoroughly assess its current safety culture, initiate targeted steps for improvement and establish guidance for recurring safety culture assessment of the organization, in alignment with FAA Safety Management Order 1030.8.

By fulfilling the Order’s requirement to assess the FAA safety culture, the agency acknowledges and demonstrates its responsiveness to related requirements and recommendations from Congress, the Civil Air Navigation Services Organization (CANSO), and the Transportation Research Board (TRB). This requirement will also enable ongoing culture assessment and improvement past the current Department of Transportation (DOT) Office of the Inspector General (OIG) audit, and past the Congressionally mandated annual assessment of FAA’s Aviation Safety Organization https://www.faa.gov/documentLibrary/media/Order/Order_1030.8.pdf

(AVS) safety culture under the Aircraft Certification, Safety, and Accountability Act

(ACSAA).

The Aviation Safety Management Organization is sponsoring this research to provide a baseline assessment of current state FAA-wide safety culture, to provide a roadmap to culture improvement, and to provide the FAA with a structured approach for performing ongoing assessment and monitoring.

C. Scope of Work

The contractor must provide support to perform a comprehensive, multi-phase safety culture assessment focusing on organizational culture factors that contribute to safety, and that impact FAA’s safety management and safety oversight capabilities. The work includes conducting a multi-method assessment of the safety culture within the FAA, developing a proposed process by which the FAA can conduct recurring culture assessment, and documenting recommended improvements that would advance FAA’s safety culture maturity. The multi-method assessment must entail, but not necessarily be limited to, a climate survey, site visit pilot, and environmental scan of existing data and leading practices.

D. Requirements

The tasks include:

1. Immediate Enterprise Assessment:

a. Establish or liaise with governance and develop and manage detailed requirements: Stand up a governance model (e.g., executive sponsor, steering committee, field representation) to support assessment planning and implementation. Governance should inform decisions, remove barriers, and oversee local versus enterprise action tracking. Facilitate regular engagements to ensure organizational requirements are captured and stakeholders are kept informed and engaged.

b. Establish Enterprise Baseline: Conduct an environmental scan of FAA-accessible data and artifacts, such as reports, findings, training, corrective actions, and reporting program data, to inform enterprise culture baseline. Where possible, use culture-related data and metrics, informed by current FAA research and assessment, to support baseline.

c. Conduct a climate survey of FAA workforce and manage safe handling of confidential employee data: Collaborate with the sponsor office in ASM to prepare and execute an FAA-wide climate survey in 2027. Teams should look to The Development and Implementation of a Safety Culture Survey for High- Performing Aviation Organizations for guidance and may suggest possible alternatives if recommended.

d. Build the FAA Safety Culture Assessment Toolkit: Design a multi-method assessment framework that establishes an accurate safety culture baseline for the FAA, including defining criteria for selecting pilot sites representative for the diversity across FAA operational environments for evaluations. A standard toolkit, which may include communication templates, observation guides, interview guides, action plan templates, and other tools should be also developed.

e. Conduct Site Visit Pilot: Conduct assessments across 3-5 pilot sites while minimizing employee time burdens to validate the designed tools, with emphasis on regional hubs where multiple FAA operations and oversight activities can be assessed in a single trip. A Standard Operating Procedure (SOP) for comparable site visit assessment, gap analysis, and leading practice identification should be developed. This preparation will include coordination with the ASM sponsor, pilot site leads, and labor partners.

2. Scale Blueprint & Actionable Improvement, Expansion & Sustainment:

a. Deliver Baseline Results Technical Report: Produce detailed FAA-level, organization-level, and pilot site assessment report documenting results of quantitative and qualitative analyses, including analysis of strengths, barriers, and current safety culture practices.

b. Deliver Approach to Recurring Culture Assessment Technical Report: This product will shift the research activity to enable operational sustainment, accountability, and continuous improvement of FAA’s ability to perform culture assessment. This report should provide tools for implementation, such as the SOP, facilitator guides, quality assurance checks, and references on culture assessment leading practices to enable expanded, recurring FAA-wide culture assessment to meet the operational requirement defined in FAA Safety Management Order 1030.8.

c. Deliver an Improvement Plan Roadmap: Develop improvement recommendations informed by the climate survey results, the environmental scan of artifacts, and the insights from the site visits pilot. This may include FAA-level, organization-level, and pilot site-level improvement recommendations supported by the results and by methods proven to support culture change in other safety-focused organizations.

E. Milestones

Item No.

Milestone Activities Delivery Schedule

1 Conduct Kick-off Meeting

Project kickoff Within 1 week after contract award

2 Develop Governance Framework

Establish and document the data management plan and the governance model to

Within 1 month after contract award support assessment planning and implementation

3 Develop Safety Culture Assessment Toolkit

Complete the design of a toolkit to support the multi-method assessment (e.g., data collection templates, interview guides, SOPs, etc.)

Within 3 months after contract award

4 Complete Enterprise Baseline Environmental Scan

Review available data, documentation, and other enterprise information to inform FAA current state culture baseline.

Within 6 months after contract award

5 Complete Climate Survey

Conduct a climate survey of FAA workforce and manage safe handling of confidential employee data

Within 7 months after contract award

6 Complete Site Visit Pilot

Conduct assessments across 3-5 pilot sites and collect data including qualitative methods such as staff listening sessions or interviews

Within 9 months after contract award

7 Brief Initial Assessment Results including Climate Survey, Baseline Study, and Site Visit Pilot

Conduct initial analysis and Integrate findings and analysis from Climate Survey, Baseline Study, and Site Visit Pilot to provide a snapshot of findings

Within 9 months after contract award

8 Finalize Assessment Results Report

Complete analysis and develop a detailed FAA-level assessment results report including results and analysis of the climate survey, baseline study, and site visit pilot findings

Within 11 months after contract award

9 Finalize Recurring Culture Assessment Technical Report

Complete a final report to document approach for implementation of recurring FAA-wide culture assessment

Within 12 months after contract award

10 Finalize Improvement Plan Roadmap

Develop improvement recommendations informed by the climate survey results, the environmental scan of artifacts, and the insights from the site visits pilot.

Within 12 months after contract award

F. Deliverables

The final deliverable for this initiative will be a comprehensive report documenting all findings, which FAA will utilize to design a long-term strategy for periodic culture evaluations. The general criteria for acceptance are that deliverables must:

• Be complete and accurate in presentation and technical content.

• Be legible and free of spelling, grammar and typographical errors.

• Adhere to applicable formats, templates, style guides, and standards.

• Be submitted timely and in accordance with the applicable SOW.

The deliverables may be returned for correction and resubmission should general criteria not be met. Final reports containing data or result documentation shall incorporate a mandatory two-week FAA draft review period followed by a minimum of one week for the contractor adjudication of the FAA’s feedback prior to final acceptance.

Item

No.

Deliverable Acceptance Criteria

Format Delivery Schedule

1 Kick-off Meeting Report Timely, Accurate and Complete

Microsoft Office Suite

Ten (10) business days after Kick-off meeting 2 Monthly Status Reports

Integrated Project Schedule status to include a breakdown of the work to be completed, the status of deliverables, key actions.

Timely, Accurate

Monthly.

Updates must be delivered by the 15th of each month, to FAA CO, COR and Technical PM.

3 Monthly Financial Status Report

Financial Status Report must include the following:

• Cost/hours funded

• Projected date when

80% of funds spent

• Funds/hours remaining

• Other Direct Costs

(ODCs) estimated, used, and remaining by month

Summary of expenditures by resource and labor category for the monthly reporting period, as well as cumulative year-to-date totals for the performance period

Timely, Accurate and Complete

Microsoft Office Suite

Monthly Updates must be delivered to FAA COR and CO, by the 15th of each month.

4 Governance Framework Document documenting the data management plan and the governance model to support assessment planning and implementation

Timely, Accurate

Within 1 month after contract award

5 Safety Culture Assessment Toolkit including Templates for Data Collection and Interviews, Supporting Documentation, SOPs, etc.

Timely, Accurate

Within 3 months after contract award

6 Enterprise Baseline Environmental Scan Project Plan including Calendar, Documentation of Procedures, Data and Observations for Planned Collection and Analysis, and Data Management Approach

Timely, Accurate

Within 4 months after contract award

7 Climate Survey Project Plan including Calendar, Updated Survey Instrument, Communication Package and Labor Coordination Requirements, and Data Management Plan for safeguarding employee data and confidentiality

Timely, Accurate

Within 4 months after contract award

8 Site Visit Pilot Project Plan and Calendar Including Access and Labor Coordination Requirements, Data Management Approach

Timely, Accurate

Within 6 months after contract award

9 Enterprise Baseline Environmental Scan Update Briefing initial snapshot of findings

Timely, Accurate and Complete

Microsoft Office Suite

Within 6 months after contract award

10 Climate Survey Project Status Update Briefing initial snapshot of findings

Timely, Accurate and Complete

Microsoft Office Suite

Within 7 months after contract award

11 Assessment Results Report Update Briefing

Draft report integrating findings and analysis from Baseline Study, Climate Survey, and Site Visit Pilot

Timely, Accurate and Complete

Microsoft Office Suite

Within 9 months after contract award

12a Assessment Results Report

Final report of full results and complete analysis

Timely, Accurate

Within 10 months after contract award

12b Assessment Results Data File(s)

Complete data collection according to the data management plan

Timely, Accurate

Within 10 months after contract award

13 Recurring Culture Assessment Technical Report Including Approach for FAA to Operationalize Ongoing Culture Assessment

Final report

Timely, Accurate

Within 12 months after contract award

14 Improvement Plan Roadmap including Recommendations to Improve on Baseline Findings and Approach for FAA Implementation

Final report

Timely, Accurate

Within 12 months after contract award

Optional Task: Post Assessment Transition and Implementation Support

The objective of this optional task is to provide the Agency with specialized change management, transition guidance, and hands-on implementation support to operationalize the culture improvement recommendations.

This phase is strictly optional and contingent upon the successful completion, delivery, and Agency acceptance of the Improvement Plan Roadmap. The Agency reserves the right, at its sole discretion, to exercise this option within 60 calendar days following the formal acceptance of Improvement Plan Roadmap. There is no implied or explicit guarantee that the Agency will execute this phase, and the Vendor shall not perform any work under this section until a formal, written contract modification or notice to proceed is issued by the Contracting Officer (CO). Period of Performance for the Optional Task is TBD.

Scope:

If exercised, the Vendor shall collaborate closely with Agency leadership, regional heads, and designated change champions to translate approved cultural recommendations into measurable institutional practices across the agency. The Vendor shall provide the following support functions:

• Implementation Roadmap & Action Planning: Convert the approved high-level Improvement Plan Roadmap into a tactical, phased implementation schedule.

This plan must account for the agency's Flight Plan goals and other related priorities, as well as assessment and improvement requirements described in Safety Management Order 1030.8 and its subsequent guidance documentation.

Charles Mayfield Highlight

• Change Management Support: Develop and implement a comprehensive change management plan working with stakeholders to affect change in their areas (e.g., change management templates, sponsor tools, stakeholder map).

• Communication Support: Develop and implement a comprehensive communication plan and toolkit (e.g., key messages, leadership talking points, agency-wide messaging, town hall agendas) supporting the rollout of assessment results and the approved Improvement Plan.

• Monitoring, Sustainment, & KPIs: Establish qualitative and quantitative key performance indicators (KPIs) and continuous feedback loops and conduct monitoring of behavioral adoption, track progress against the cultural baseline, and ensure long-term sustainability.

• Implementation Support for 1030.8 Guidance and Ongoing Culture Assessment: Aid the Agency in socializing and implementing the ongoing culture assessment guidance, once approved, and the related guidance following the publication of 1030.8 (e.g., host webinars with affected stakeholders to raise awareness and education, iterate on guidance products, build web-based tools).

Deliverables:

• Implementation Roadmap: Tactical, phased schedule for implementing the approved Improvement Plan activities across the affected organizations.

• Change Management Plan: Tools to support change practitioners, culture program liaisons and leaders working with stakeholders to affect behavior change in their areas (e.g., change management templates, sponsor tools, stakeholder map).

• Communication Plan: Tools to support the rollout of assessment results and the approved Improvement Plan (e.g., key messages, leadership talking points, agency-wide messaging, town hall agendas, graphics and multimedia products).

• Playbook for Leaders: Tools to support leaders in their unique role leading change and setting the tone for acceptable behaviors and achievement of desired cultural behaviors and indicators.

• Guidance Tools: Support implementation of assessment and improvement guidance to FAA organizations by developing web-based tools and engagement opportunities that raise awareness, understanding and ability to conduct culture assessment and design and monitor improvements.

F. Travel

The Contractor must travel to support the safety culture assessment. The FAA COR must approve all travel in advance of its occurrence. The Contractor will not be reimbursed for any unauthorized travel.

G. Period of Performance

The period of performance is anticipated to be 12-18 months.

Contract/Agreement type will be firm fixed price.

DOT/FAA/AM- 24/18

Aviation Safety

Office of Aerospace Medicine

Washington, D.C. 20591

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations Kylie K. Worthington1

Peter T. Hu2

David J. Schroeder2

Inchul Choi2

1 Civil Aerospace Medical Institute (CAMI) Federal Aviation Administration Oklahoma City, OK 73125

2 Cherokee Federal Oklahoma City, OK 73125

July 2024

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations ii

N OT I C E

This document is disseminated under the sponsorship of the U.S.

Department of Transportation in the interest of information exchange. The United States Government assumes no liability for the contents thereof.

This publication and all Office of Aerospace Medicine technical reports are available in full text from the Civil Aerospace Medical Institute’s publications website (www.faa.gov/go/oamtechreports) and at the National Transportation Library’s Repository & Open Science Access Portal

(https://rosap.ntl.bts.gov).

https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports https://rosap.ntl.bts.gov/ https://rosap.ntl.bts.gov/

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations iii

Technical Report Documentation

1. Report No.

DOT/FAA/AM-24/18

2. Title & Subtitle The Development and Implementation of a Safety Culture Survey for High- Performing Aviation Organizations

3. Report Date July 2024

4. Performing Organization Code

5. Author(s) K. Worthington, P. Hu, D.Schroeder, I. Choi

6. Performing Org Report Number

7. Performing Organization Name & Address Civil Aerospace Medical Institute (CAMI), FAA Oklahoma City, OK 73169

8. Contract or Grant Number

9. Sponsoring Agency Name & Address Office of Aerospace Medicine, Federal Aviation Administration, 800 Independence Ave., S.W., Washington, DC 20591

10. Type of Report & Period Covered Technical Report

11. Supplementary Notes

12. Abstract A positive safety culture requires an intentional approach and periodic assessment to determine where things stand and where improvements can be made. To fulfill requirements of the Aircraft Certification, Safety, and Accountability Act (ACSAA, §132(c)), the FAA developed and deployed a novel survey to help assess safety culture in the regulatory environment. The survey was developed in substantial part by leveraging insights from both the scientific literature and existing regulatory frameworks of safety culture.

The survey measures the ten core traits set forth in the Harmonized Safety Culture Model developed by the International Atomic Energy Agency. As designed, the survey covers important aspects of any safety-critical workplace, including individual and leader commitment to safety, just and reporting culture, competing pressures, resource allocation, problem identification and resolution, workload and work planning, and others. This report provides (a) an overview of the safety culture survey development and design; (b) a study examining safety culture perceptions within a large participating FAA line of business; and (c) a brief discussion of survey administration best practices and follow-up. The value of the survey is that it provides a comprehensive assessment of safety culture perceptions in regulatory organizations; however, its utility is only fully realized when used judiciously and when the results are acted upon as part of a larger strategy for managing safety culture change.

13. Key Words safety culture, survey, assessment

14. Distribution Statement Document is available to the public through the National Transportation Library:

http://www.faa.gov/go/oamtechreports/

15. Security Classification (of this report) Unclassified

16. Security Classification (of this page) Unclassified

17. No. of Pages http://www.faa.gov/go/oamtechreports/

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations iv

Acknowledgments

We would like to thank Dr. Casey Yetter, Rebecca DiDomenica, Blake Nesmith, and Hanna Forbat for providing their expert research support. This research was funded by the FAA Office of Aviation Safety.

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations v

Table of Contents Acknowledgments ...................................................................................................................... iv

List of Tables .............................................................................................................................. vi

List of Figures ............................................................................................................................. vi

List of Abbreviations ................................................................................................................... vii

Introduction

Phase 1: Documenting Stakeholder Priorities and Assessment Requirements

Phase 2: Survey Development

Framework of Positive Safety Culture Traits

Item Selection

Item Validation

Content Validity

Face Validity

Final Safety Culture Survey

Phase 3: Survey Deployment to a Large FAA Line of Business

Survey Administration

Survey Logistics

Confidentiality Assurances

Survey Results

Sample Representativeness

Survey Usability Ratings

Phase 4: Documenting Lessons Learned and Recommended Practices

Survey Administration

Survey Follow-Up

Discussion

Limitations and Future Directions

Conclusions

References

Appendix A. Safety Culture Survey

Survey Introduction Page

Question Items

Communication

Individual Responsibility

Questioning Attitude

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations vi

Leader Responsibility

Decision-Making

Respectful Work Environment

Continuous Learning

Problem Identification and Resolution

Environment for Raising Concerns

Work Planning

Open-ended Feedback

Demographics

Survey Usability

Appendix B. Supplemental Modules

Supplemental Module Question Items

Engagement

Job Satisfaction

Pressure and Workload

Performance

Well-Being

Underperformance

Personal Growth

Autonomy

List of Tables

Table 1 Participation Response Rates by Staff Office Table 2 Survey Usability Items

List of Figures

Figure 1 Respondent Years of Service

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations vii

List of Abbreviations

Acronym Abbreviation Explained

ACSAA Aircraft Certification, Safety, and Accountability Act

AVS Aviation Safety

CAA Civil Aviation Authority

CAMI Civil Aerospace Medical Institute

FAA Federal Aviation Administration

FEVS Federal Employee Viewpoint Survey

IAEA International Atomic Energy Agency

ICAO International Civil Aviation Organization

INSAG International Nuclear Safety Advisory Group

LOB Line of Business

M-SCAIT Maintenance Safety Culture Assessment and Improvement Toolkit

NASA National Aeronautics and Space Administration

National Academies National Academies of Engineering, Science, and Medicine

NIOSH National Institute for Occupational Safety and Health

NOSACQ-50 Nordic Occupational Safety Climate Questionnaire

NRC Nuclear Regulatory Commission

SCMT Safety Culture Measurement Toolkit

SHo-Me Safety Health of Aviation Maintenance Engineering (Tool)

SME Subject Matter Expert

SMICG Safety Management International Collaboration Group

TRB Transportation Research Board

WellBQ Worker Well-Being Questionnaire

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations 1

Introduction

“Safety culture is arguably the single most important influence on the management of safety. If an organization has instituted all the safety management requirements but does not have a positive safety culture, it is likely to underperform” (International Civil Aviation Organization [ICAO], 2018; p. 3-1).

The FAA defines safety culture as “…the shared values, actions, and behaviors that demonstrate a commitment to safety over competing goals and demands” (Department of Transportation Safety Council; see Morrow & Coplen, 2017). A positive safety culture requires shared responsibility across individuals and leaders, demonstrating a clear commitment to safety. Further, a positive safety culture is characterized by a questioning attitude and a respectful work environment where trust, honesty, and open communication are fostered.

Creating this environment for raising safety concerns empowers problem identification and resolution, where safety issues are identified, evaluated, and properly resolved. As safety concerns are addressed, organizations with a positive safety culture will continuously learn and share the lessons learned, promoting decision-making that prioritizes safety. Finally, there should be a support infrastructure in place, including communications and work planning that mutually emphasize safety.

A positive safety culture requires an intentional approach and periodic assessment to determine where things stand and where improvements can be made. It also requires frameworks to enable positive safety culture behaviors and continuous actions, informed by periodic assessments, to strengthen and improve.

The need for safety culture assessment became visible with the passage of the Aircraft Certification, Safety, and Accountability Act (ACSAA, §132(c); as part of the Consolidated Appropriations Act of 2021, 2020), which requires the Federal Aviation Administration (FAA) to conduct safety culture assessments of all Aviation Safety (AVS) employees annually through 2031. This requirement is reflected in the FAA’s strategic planning; initiating a safety culture campaign is one of the initiatives supporting the Safety pillar in FAA’s Flight Plan 21 (2022).

To fulfill ACSAA requirements, the FAA developed and deployed a novel survey to help assess safety culture in the regulatory environment. The survey was designed to balance the goals of scientific robustness with the need for generalizability and ease of use across the FAA. This was achieved by taking a collaborative approach to the project, where researchers and FAA-level safety culture community of practice representatives worked together to define and develop the survey. To test its efficacy, the survey was deployed to one large FAA line of business (LOB).

This report provides (a) an overview of the safety culture survey development and design; (b) a study examining safety culture perceptions within a large participating FAA LOB; and (c) a brief discussion of survey administration best practices and follow-up. This report describes four phases of research:

1. Documenting stakeholder priorities and assessment requirements

2. Survey development

3. Survey deployment to a large FAA LOB

4. Documenting lessons learned and recommended practices for administering and acting on the safety culture survey

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations 2

Phase 1: Documenting Stakeholder Priorities and Assessment Requirements

The researchers hosted brainstorming sessions to identify stakeholder priorities and requirements for a successful safety culture assessment. The brainstorming session participants included leadership and other stakeholders from the FAA-level safety culture community of practice, with diverse representation invited from all FAA LOBs.

The stakeholders prioritized the development of a comprehensive safety culture assessment that could (a) quickly and independently gather a snapshot of the workforce perceptions of the safety culture at a given timeframe,1 (b) monitor safety culture over time, and (c) track the effectiveness of safety culture promotion and improvement actions. Specific requirements of the safety culture survey include:

• Generalizable for use across the FAA

• Scientifically robust (i.e., assess a known framework of safety culture traits, measuring and leveraging multiple indicators)

• Anonymous aggregate results

• Collaborative with labor partners

Stakeholders agreed that the assessment should consist of a survey because it provides a systematic and inclusive approach to assessment, such that all employees have an opportunity to share their perspectives. Results can be compared readily across demographic variables of interest, such as department, location, or job role (National Academies of Science, Engineering, and Medicine [National Academies], 2016); this helps identify culture silos that may require targeted improvements.

Stakeholders prioritized that the survey should be both comprehensive and quick to administer (i.e., use pulse survey methodology). The LOB and stakeholder groups mutually agreed that the survey must assess all ten traits of a positive safety culture across all members of the LOB’s workforce, but the amount of time needed to answer the survey could be shortened during administration by asking each respondent to answer only a randomly selected fraction of the question items needed to fully assess each safety culture trait.

1 Snapshots of workforce perceptions at a given timeframe are often referred to as measures of safety climate. The terms safety culture and safety climate are often used interchangeably, though there are some academic differences. Safety culture refers to the more enduring behaviors, norms, and assumptions in the organization, whereas safety climate is a more temporal, dynamic construct (Wiegmann et al., 2002).

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations 3

Phase 2: Survey Development

The survey was designed as an a priori (planned) measurement of a safety culture based on an existing, well-used framework (i.e., the Harmonized Safety Culture Model, described below). It assesses internationally recognized traits of a positive safety culture (Appendix A), along with supplemental modules that can help address specific work unit needs (Appendix B). The researchers developed this survey to be used in its entirety, but to meet the dual needs of assessing the ten safety culture traits while maintaining brevity, a pulse survey methodology can be adopted where respondents complete a randomly selected subset of question items per safety culture trait.

Framework of Positive Safety Culture Traits The survey was developed in substantial part by leveraging insights from both the scientific literature (see Key et al., 2023) and existing regulatory frameworks of safety culture. The survey measures the ten core traits set forth in the Harmonized Safety Culture Model developed by the International Atomic Energy Agency (IAEA, 2020). The model represents an evolution of assessments dating to 1992 when the International Nuclear Safety Advisory Group (INSAG) identified human factors as having contributed to the 1986 Chernobyl nuclear disaster and noted, among other findings, a general lack of safety culture (1992).

The Harmonized Safety Culture Model consists of ten traits:2

• Individual Responsibility. “All individuals are personally accountable for safety.”

• Questioning Attitude. “Individuals remain vigilant for assumptions, anomalies, conditions, behaviors or activities that can adversely impact safety and then appropriately voice those concerns.”

• Communication. “Communications support a focus on safety.”

• Leader Responsibility. “Leaders demonstrate a commitment to safety in their decisions and behaviors. Leaders are role models for safety.”

• Decision-Making. “Decisions are systematic, rigorous, thorough, and prudent.”

• Respectful Work Environment. “Trust and respect permeate the organization.”

• Continuous Learning. “Learning is highly valued.”

• Problem Identification and Resolution. “Issues potentially impacting safety are systematically identified, fully evaluated, and promptly resolved according to their significance.”

• Environment for Raising Concerns. “Personnel feel free to raise safety concerns without fear of retaliation, intimidation, harassment, or discrimination.”

• Work Planning. “The process of planning and controlling work activities is implemented so that safety is maintained.”

Nine of the traits were recommended by the Transportation Research Board (TRB; IAEA, 2020;

National Academies, 2022); an additional trait captures the priority of safety in decision-making.

2 Quotes are reproduced from IAEA (2020).

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations 4

These ten core traits are further characterized and organized by the IAEA into attributes of a positive safety culture.

Item Selection A panel of four researchers (i.e., the authors) specializing in human factors research, each with a minimum of three years of expertise on the topic of safety culture, convened to develop the survey. The researchers have previously developed safety culture assessments for use in other aviation environments (i.e., regulatory organizations, certificate holders).

To start the development process, the researchers identified available survey instruments relevant to safety climate/culture. The selection criteria for instruments to review were that they must be (a) published, with the constituent question items publicly available, and (b) developed for and/or used by a governance organization. This search yielded the following survey instruments of safety climate/culture:

• Civil Aviation Authority (CAA), Safety Health of Aviation Maintenance Engineering (SHo- Me) Tool (CAA, 2003).

• EUROCONTROL, Safety Culture Measurement Toolkit (SCMT) for European Air Navigation Service Providers (Gordon & Kirwan, 2004; Mearns et al., 2013).

• FAA, 2020 AVS Safety Culture Assessment (MITRE, 2020).

• FAA, Maintenance Safety Culture Assessment and Improvement Toolkit (M-SCAIT) (Key et al., 2023).

• FAA, [2022 AVS] Safety Culture Assessment (Human Synergistics; Cooke et al., Unpublished Manuscript).

• Office of Personnel Management (n.d.), 2022 Office of Personnel Management Federal

Employee Viewpoint Survey results [FEVS].

• IAEA, Safety Culture Perception Questionnaire for License Holders (IAEA, 2017).

• National Aeronautics and Space Administration (NASA), Safety Culture Survey (SCS;

NASA, 2015).

• Nuclear Regulatory Commission (NRC), Safety Culture and Climate Survey (NRC, 2006, 2009, 2016).

• National Institute for Occupational Safety and Health (NIOSH), Worker Well-Being

Questionnaire (WellBQ; Chari et al., 2022).

• Nordic Occupational Safety Climate Questionnaire (NOSACQ-50; Kines et al., 2011).

• Safety Management International Collaboration Group (SMICG), Industry Safety Culture

Evaluation Tool (SMICG, 2019).

• Transport Canada, Score Your Safety Culture (Transport Canada, n.d.).

A set of items evaluating the effectiveness of voluntary reporting programs was included without modifications from a previous safety culture assessment to allow year-on-year comparison (Cooke et al., unpublished manuscript).

Cumulatively, these instruments totaled over 700 items for review. Each researcher independently rated each item for relevance. Specifically, each researcher reviewed each item

The Development and Implementation of a Safety Culture Survey for High-Performing Aviation Organizations 5 and categorized them into one of the ten IAEA traits. The researchers then ranked each item by how closely it aligned with the IAEA attributes. The researchers found that the attributes overlapped, resulting in some items being classified in a secondary attribute as well. In lieu of sampling-based inter-rater reliability methods, researcher consensus for the attribute categorization of each item was reached through discussion. Many items were found not to align well with any attributes, reducing the list to approximately 300 items.

Item Validation Next, the question items were reviewed by stakeholders to ensure their operational readiness.

An iterative two-step evaluation was performed to confirm the content and face validity of the selected items;3 consequently, the number of items in the final survey instrument was reduced.

Content Validity

A separate and independent group of safety culture subject matter experts (SMEs) evaluated the content validity of question items. SMEs held Ph.D.s in psychology or human factors, each of whom has previously led research studies assessing safety culture. SMEs rated the extent to which each of the items measured the IAEA’s attribute; the definitions of each attribute were provided to the SMEs in the rating exercise. More specifically, the SMEs were asked to “Please rate the extent to which each item measures the attribute”, using a rating scale of 0 to 100 in 10-pt increments. Items were not retained if the average agreement that the item measured the attribute was less than 70 of 100. The SMEs were also provided the opportunity to suggest item wording changes or request additional items to measure the attributes. This process reduced the count from 300 items to 146 items.

Face Validity

Representatives were sampled from an agency-level community of practice on safety culture to assess the face validity of each question item. Notably, one of the aims of the community of practice is to develop methods to measure and monitor safety culture across FAA LOBs. The representatives evaluated the face validity of each item to ensure that they were worded appropriately and could be answered by individuals within their respective work units. More specifically, representatives were asked to “Please rate whether employees in your work group (i.e., routing code) can comprehend and answer each item”, using a rating scale of 0 (‘Not at all’) to 10 (‘Definitely’). Items were not retained if the average agreement that the item was understandable was less than 7 of 10. This process reduced the count from 146 items to 75 items.

Final Safety Culture Survey The researchers scoped the survey instrument by…