03282023 E-E_IDIQ SOW_Draft.docx

HRSA Evaluation Studies IDIQ

Statement of Work Project Title: Evidence-Building and Evaluation Services

I. BACKGROUND:

The mission of the Health Resources and Services Administration (HRSA) is to improve health and achieve health equity through access to quality services, a skilled health workforce and innovative programs. HRSA assures the availability of quality health care to low income, uninsured, geographically isolated, special needs, and priority populations. As the primary federal agency for improving health care of the priority populations it serves and the current Administration’s emphasis on health equity; HRSA is focused on conducting robust and relevant evaluations and evidence-building activities.

HRSA is strengthening its programs and the evaluations it conducts to promote health equity and identify opportunities for future evaluation investments.

The Bureaus and Offices within HRSA that administer these programs are responsible for ensuring policy analysis, data analysis and program evaluations are conducted to improve the design, quality, and performance of these programs and to introduce refinements/improvements as needed.

The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) and associated OMB guidance provide the impetus to assess the evidence-building and evaluation capacity within HRSA. Evidence building activities include policy analyses, program evaluations, foundational and descriptive research, and performance and quality measurement activities. In addition, the results of these evaluation efforts inform programmatic, legislative, budgetary and policy decision-making by the senior leadership at the agency.

To accomplish these activities, HRSA staff often need to analyze and assess current and emerging health policy issues, conduct descriptive research, and evaluate HRSA programs under relatively tight time constraints. Activities frequently require personnel to augment HRSA skills and capabilities with skills and experience not available within the Agency.

These activities typically target high priority issues which are likely to influence the budgetary, legislative, regulatory, and planning cycles of the Agency.

II. PURPOSE/GENERAL DESCRIPTION:

The purpose of this Indefinite-Delivery-Indefinite-Quantity (IDIQ) requirement is to provide a contract mechanism to facilitate the production of relevant, high-priority, and robust evaluations, and evidence-building activities to support various Bureaus and Offices within HRSA. These activities may include evaluations, policy analyses, descriptive research studies, as well as activities such as the development, implementation, and analyses of quality improvement projects to identify programmatic best practices, assessment of evaluation capacity, and the development and implementation of performance and quality measurement tools and data collection methods to monitor program performance.

The following describes the typical tasks which may be required of the Contractor in the performance of awarded task orders. Task orders may not necessarily include the entire set of task areas listed below; however, they will specify the actual tasks to be performed along with the project-specific information required for completion of the task order.

The activities related to evaluation studies and services are organized to support the following four (4) key Task Areas:

1. Task Area 1 – Program Evaluations. Conducting program evaluations, including formative, process, outcome, impact, and economic evaluations.

2. Task Area 2 – Evidence Building. Conducting contextual/policy analyses of issues impacting HRSA-supported activities; including conducting descriptive research; developing and implementing performance and quality measures; developing evaluation plans and frameworks; building logic models.

3. Task Area 3 – Communication and Dissemination. Communicating and disseminating findings from evaluations and evidence-building activities.

4. Task Area 4 – Data Analytics to Support Evaluations and Evidence-building. Collecting, cleaning, validating, analyzing, and visualizing data for the purposes of evaluations and evidence-building activities.

III. PERIOD OF PERFORMANCE:

The period of performance is a 12-month Base Year and four 12-month Option Years.

IV. TASKS:

The Parent IDIQ scope includes the following task areas:

TASK AREA 1: Program Evaluations.

Overview: The development, implementation, and support of evaluation studies related to HRSA projects or programs are conducted with the goal of collecting empirical information about program performance, outcomes and/or impact of the programs and to enhance internal HRSA programmatic decision-making and judgments about various aspects of programs. To accomplish these activities, staff often need to analyze and assess current and emerging health policy issues, develop data systems, and evaluate HRSA programs under relatively tight time constraints. Such activities frequently require personnel to augment HRSA skills and capabilities with skills and experience not available within the Agency. These activities typically target high priority issues which are likely to influence the budgetary, legislative, regulatory, and planning cycles of the Agency. HRSA conducts a range of program evaluations, including formative, process, outcome, impact, and sometimes economic evaluations. Typically, these evaluation studies or projects are conducted to:

· Understand factors that may impede or contribute to a program’s success.

· Explain linkages between project or program activities and outcomes.

· Assess effects of a project or program beyond its intended objectives.

· Estimate what would have occurred in the absence of a project or program.

· Assess effect of enhancements to projects or programs.

· Assess net impact of a project or program.

· Compare the effectiveness of alternative interventions, projects or programs aimed at the same objectives.

Provided below are tasks focused on supporting the program functions that the contractor shall undertake:

(1) Develop overall research/evaluation design for the study.

(2) Monitor and analyze implementation activities associated with planned program models or designs.

(3) Conduct a systematic evaluation of operations (i.e., the best way to accomplish program goals).

(4) Conduct assessment of the efficiency of program management activities.

(5) Assess the relationship between HRSA-supported activities and various observed outcomes.

(6) Assess consumer/service recipient perception of program availability and accessibility.

(7) Identify “best practices” implemented among several similar program sites. “Best practices” may include practices, policies or programs that have been demonstrated by HRSA programs/recipients to be effective. These practices may reflect evidence-based best practices as demonstrated by the literature.

To successfully address HRSA program questions, needs, and concerns the Contractor shall determine and, propose evaluation study questions, determine evaluation study design, and perform other tasks such as:

(1) Obtain and reviewing background information on the program to be evaluated using a variety of strategies such as HRSA program subject matter experts (SMEs) and research conducted by Contractor.

(2) Conduct broad-based environmental scans.

(3) Conduct literature review on relevant and pertinent questions to be addressed.

(4) Conduct interviews with HRSA program and recipient staff as well as other stakeholders (e.g., study participants).

(5) Review data on program’s operations and assessing current research on the program to be evaluated.

(6) Recommend the appropriate evaluation design, based on program needs, evaluation type, evaluation aims, and other factors.

(7) Develop new data collection instruments, including survey design. Plain language principles shall be incorporated into development of all data collection instruments.

(8) Revise existing data collection instruments, including survey design. Plain language principles shall be incorporated into development of data collection instruments.

(9) Prepare and submit to the Office of Management and Budget (OMB), clearance packages, Institutional Review Board (IRB) Human Subjects Protection requests, and prepare documents for IRB and OMB review, as appropriate.

(10) Collect quantitative and qualitative data using appropriate strategies, including survey tools, from a variety of HRSA programs, recipients, and special populations (e.g., rural, maternal/child, hard to reach populations).

(11) Coordinate and comply with HRSA Offices of Communication and Information Technology, regarding policies and procedures related to review and clearance of data collection instruments, including surveys.

(12) Implement procedures such as inter-rater reliability to ensure consistency among data collectors for the same observation(s).

(13) Ensure data quality and security, integrity, and availability of data to evaluation team members.

(14) Conduct ongoing analyses of data using appropriate methodologies, including small sample sizes.

(15) Prepare data visualizations or other products to effectively communicate evaluation findings to program stakeholders.

(16) Prepare short reports, topical reports, summaries of preliminary findings, final reports, and other analytical reports, including detailed methodological appendices where appropriate.

(17) Incorporate program stakeholder comments/recommendations into a final report and/or provide briefings.

(18) Ensure all deliverables including final reports, summaries of preliminary findings, and briefings shall be 508 compliant.

(19) Prepare publications of evaluation study methodology and results for the peer- reviewed literature.

(20) Ensure alignment with current HHS and HRSA policies (e.g., through the Office of Communications and/or Office of Information Technology) and other federal regulations regarding execution of Task Orders for evaluations and evidence-building activities and other tasks.

(21) Implement policies and procedures to track and assess progress toward the production, submission and/or quality of evaluation project deliverables to HRSA Program. This assessment shall include a performance standard or other metric to assess level of Contractor performance for project deliverables.

(22) Identify the target audience(s) and develop a communication and dissemination plan in preparing summary reports. The plan shall include:(1) addresses the information need(s) of Program including providing evidence-based findings that promote enhanced programmatic decision-making; and (2) convey complex information in intuitive and meaningful ways for all target audience(s).

(23) Account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

TASK AREA 2: Evidence-Building.

Overview: Evidence-building activities encompass contextual/policy analysis as well as a wide range of activities, supporting evaluations and policy analysis. For example, it is often necessary to develop forecasts of current trends or likely future scenarios when preparing policy/program analyses and planning for evaluations. Evidence-building activities also include the development of logic models, development of performance and quality measures, and evidence-based intervention strategies for enhancing program effectiveness and efficiency. It also includes qualitative and quantitative data/collection, analyses, and reporting for several subject areas all of which lead to stronger evaluations and evidence-building activities.

The analysis of contextual/policy issues which may have an impact on HRSA-supported activities are focused primarily on one or more of the following:

(1) External context of programs and how forces in the health landscape affect HRSA programs and their progress toward achieving goals and objectives. Examples include health and human service system changes (e.g., increases in managed care or changes in the welfare system); social, economic, and political changes (e.g., an increase in populations without health care coverage, etc.); and social determinants of health and health IT operations and limitations.

(2) Assessment of various policy alternatives related to HRSA programs or interests (i.e., how actual, planned, or potential changes in current public and private policies and implementation strategies affect achievement of program goals and objectives).

(3) Identification of policy alternatives for consideration by HRSA programs; these alternatives would include potential benefits (pros) as well as challenges (cons).

(4) Description and better understanding of areas for innovation and/or adapting to technological advances and the impact on health care (e.g., telehealth, mobile health delivery systems, health IT).

(5) Assessment of effectiveness and/or impact of policy-related strategies implemented in the field to address a specific public health issue.

(6) Provision of recommendations and expert opinion(s) on the implementation of various legislative requirements impacting HRSA programs.

Other activities centered around evidence-building included in Task Area 2 are:

· Assessment of performance data for programs with small sample sizes (e.g., clinic settings with a low volume of patients) and experience working with small populations.

· Development of logic or conceptual models to theoretically link program inputs, activities with program outputs and outcomes.

· Program performance and quality improvement, for example,

(1) Implementation of quality improvement approaches and results.

(2) Achievement of operational targets, meeting standards, and following national guidelines.

(3) Development of performance measures and data collection instruments to assess program performance and improve program monitoring efforts (e.g., programmatic, clinical, fiscal).

· Measurement of program effectiveness, for example,

(1) Development, implementation and/or analysis of basic/immediate programmatic outputs and outcomes (e.g., program products/materials such as training/health promotion) to assess short-term program effectiveness and efficiency.

(2) Development of cross cutting metrics that may comprehensively help to assess programs with a single/common focus (e.g., oral health services; health IT).

(3) Determination of program sustainability (i.e., is the program sustainable without fiscal or programmatic resources from HRSA).

(4) Assessment and support of alignment of programmatic performance and quality measures with national benchmarks (e.g., Healthy People objectives).

(5) Assessment of the effectiveness and/or impact of the implementation of specific public health related programmatic strategies.

(6) Preparation of demographic characteristic and population projections, development of financial and service projections, evidence-based projections about service outcomes and system performance indicators, and identification of policy alternatives, including strengths and challenges for program consideration.

· Policy assessment and analysis, for example,

(1) Develop a clear statement of the relevant existing policy issue(s).

(2) Compile qualitative and quantitative data, previous analyses, previous policy/mission statements, previous projection analyses and other relevant information.

(3) Prepare an analysis of policy/program alternatives, using an appropriate methodology(ies).

(4) Identify the strengths and challenges of each identified policy/program alternatives, together with an assessment of their likely future success and possible impediments.

(5) conduct scans of relevant Federal and state laws and policies that impact HRSA programs.

Task area 2 will necessitate working with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data and vital records).

All analyses and findings shall be described in reports suitable for a variety of audiences (e.g., internal reports, public reports, reports to Congress, and publications for peer review) as well as in a comprehensive report that provides clear and complete documentation of the issue(s); methods and data sources; findings; analysis limitations; and conclusions.

Briefings to various audiences, including both well-informed lay audiences and technical audiences shall likely be required. The Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

TASK AREA 3: Communication and Dissemination.

Overview: Communicating and disseminating findings from evaluations or other evidence-building activities is a significant and critical task. This task area shall include the development of approaches to communicate and disseminate results/findings from HRSA evaluation studies, evidence-building activities, policy analyses, and studies. For example, some of these approaches may include established Dissemination and Implementation based approaches as well as a variety of strategies designed for maximizing HRSA’s impact of its programs and policies for the populations it serves. The Contractor shall develop approaches that effectively promote, market, and disseminate, as needed, results/findings from HRSA evaluations (e.g., previously conducted studies and/or evaluation-related work). The Contractor shall coordinate with HRSA’s Office of Communication (e.g., compliance with the policies and practices of this Office that involve publishing data and evaluation results to a HRSA.GOV website or internal HRSA-only website, for example 508 compliance). Examples include use of the HRSA style guide (including HRSA branding standards on all materials), use of HRSA-approved PowerPoint templates, etc.

The Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s). Under this task, the contractor shall:

(1) Develop mechanisms (e.g., reports, graphical representations, dashboards) to disseminate results/findings, models of care, and evidence-based best practices among HRSA programs, recipients, and stakeholders. The development of these tools and resources shall include data visualizations and web-based tools, as appropriate and shall be disseminated on HRSA internal or external websites.

(2) Develop and disseminate resources that promote translation of evaluation findings into improved practice for service providers.

(3) Develop, promote, and market strategies that leverage existing HRSA communications mechanisms (e.g., radio media tour, social media accounts).

(4) Develop plans for optimal implementation and replication of successful models of care among HRSA programs and recipients.

(5) Develop reports, publications, and presentations that are 508 compliant.

(6) Develop communication plans, including identification of stakeholder engagement roles and points of contact.

(7) Prepare manuscript(s) for publication.

(8) Prepare Draft and Final PowerPoint presentation(s) or abstract(s) for meeting/conference presentation(s).

(9) Develop policy and research briefs.

TASK AREA 4: Data Analytics to support Evaluations and Evidence-building.

Overview: This task will include strategies for collecting, cleaning, validating, analyzing, and visualizing data to support evidence-building, evaluation activities, and internal HRSA decision making. This task area shall also include the development of strategies, documentation, and mechanisms to support a continuous quality improvement loop based on the findings from evaluations and evidence building activities conducted in Task Areas 1 and 2. The Contractor’s activities in this task area shall align and be compatible with policies and procedures associated with patient protection (e.g., alignment with OMB and HRSA Human Subject Protection Requirements). Under task 4 the contractor shall:

(1) Convert raw data files into summary/analytic files.

(2) Compile, link, aggregate, and consolidate data from multiple data sources into a summary/analytic file.

(3) Work with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data and vital records).

(4) Conducting statistical analysis and developing statistical models using SAS, STATA, R, SPSS, Excel, Python, Tableau, Nvivo (for qualitative analyses) or other appropriate analytical and/or programming/modeling tools.

(5) Developing data documentation such as metadata and data dictionaries.

(6) Develop instruction manuals and training materials to support data collection for evaluation studies and evaluation-related activities.

(7) De-identify/anonymize/aggregate data to ensure no PII or sensitive data is released to the public.

(8) Conduct activities such as data cleaning, checking accuracy and consistency needed to prepare data for analysis, or develop data documentation.

(9) Develop 508-compliant mock-ups of dashboards, reports, maps, or other data visualizations using HRSA-approved software.

(10) Prepare results of analysis, or other data-related visualizations, presentations, or dissemination.

(11) Provide relevant training to HRSA staff on the analysis of data related to evaluation and/or evaluation-related activities.

(12) Conduct data collection, analysis, and preparation in support of evaluation studies and evaluation-related activities among HRSA programs and recipients.

(13) Coordinate with HRSA’s Offices of Information Technology and Communication to ensure compliance with the policies and practices of these Offices regarding evaluation studies and evaluation-related activities that involve publishing data to a HRSA.GOV website or internal HRSA-only website.

(14) Store datasets/files on a secure server that is in compliance with HHS security policies on data collection, transmission, and storage. The Contractor shall follow all HRSA, HHS, and Federal IT policies including firewalls, data encryption, two- factor authentication, privacy, security, etc.

V. APPENDIXES

Appendix A: Examples of Evaluations that may be conducted in Task Area 1: Evaluations.

· Formative Evaluation is typically conducted to assess whether a program, policy, or organizational approach-or some aspect of these-is feasible, appropriate, and acceptable before it is fully implemented. It may include process and/or outcome measures. A formative evaluation focuses on learning and improvement and does not aim to answer questions of overall effectiveness.

· Impact Evaluation assesses the causal impact of a program, policy, or organization, or aspect thereof, on outcomes relative to those of a counterfactual[footnoteRef:1]. In other words, this type of evaluation estimates and compares outcomes with and without the program, policy, or organization, or aspect thereof. Impact evaluations include both experimental (i.e., randomized controlled trials) and quasi-experimental designs. An impact evaluation can help answer the question, “does it work?” or “did the intervention lead to the observed outcomes?” [1: https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf ]

· Process or Implementation Evaluation assesses how the program or service is delivered relative to its intended theory of change, and often includes information on content, quantity, quality, and structure of services provided. These evaluations can help answer the question, “was the program, policy, or organization implemented as intended?” or “how is the program, policy, or organization operating in practice?”

· Outcome or Summative Evaluation measures the extent to which a program, policy, or organization has achieved its intended outcome(s) and focuses on outputs and outcomes to assess effectiveness. Unlike impact evaluation above, it typically cannot discern causal attribution. Importantly, it is distinct from, but complementary to, performance measurement, as noted below. An outcome evaluation can help answer the question “were the intended outcomes of the program, policy, or organization achieved?”

· Economic Evaluation Public health professionals use economic evaluations to identify, measure, value, and compare the costs and consequences of different public health interventions. Allocating resources and implementing these interventions—whether policies or programs—require an understanding of the relationships between resources used and health outcomes achieved by the program or intervention. Economic evaluations can consider both resources used, and health outcomes achieved simultaneously. They are also useful in supporting decision-making when resources are limited. The types of economic evaluations anticipated by HRSA include, program cost analysis, cost of illness or disease burden, cost-effectiveness analysis, and cost benefit analysis.

Appendix B: Examples of Evidence-Building activities that may be conducted in Task Area 2: Evidence-Building.

For Contextual Policy analysis, the Contractor shall have the capacity to demonstrate experience in assessing policy issues at the Federal, State, and local level that may be relevant to HRSA programs and recipients. The table below provides some examples of work undertaken for the Evidence-Building task. In addition, the Contractor shall also demonstrate the capacity to manipulate data sets of varying size and complexity, and the statistical expertise to use appropriate methods/procedures to conduct and report projection analyses. The Contractor shall also demonstrate an understanding of the nuances associated with data at various levels, particularly at the patient level.

TABLE 2. EXAMPLES OF TYPES OF EVIDENCE-BUILDING ACTIVITIES THAT MAY BE CONDUCTED IN TASK AREA 2

should be designed or modified to achieve those goals.

whether an implementation, outcome, or impact evaluation is appropriate to conduct for a given program.

and development of recommendations regarding evaluation capacity building requirements within the organization.

1. Impact of local, state, and Federal policies that affect the implementation of HRSA programs and recipients.

Impact of changes in financial policy by Federal, state, local and/or private payers on the recipients of HRSA funds and their clients or patients.