Section J Attachment A - HRSA E-E IDIQ SOW - FINAL.pdf

HRSA Evidence-Building and Evaluation Services IDIQ Section J Attachment A

Statement of Work

Project Title: Evidence-Building and Evaluation Services

I. BACKGROUND

The mission of the Health Resources and Services Administration (HRSA) is to improve health and achieve health equity through access to quality services, a skilled health workforce and innovative programs. HRSA assures the availability of quality health care to low income, uninsured, geographically isolated, special needs, and priority populations. The Bureaus and Offices (B/Os) within HRSA that administer these programs are responsible for conducting policy analysis, data analysis, and program evaluations to improve program design, quality, and performance and introduce refinements/improvements as needed.

The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) and associated OMB guidance drive HRSA’s investments in evidence-building and evaluation. Evidence-building activities include policy analyses, program evaluations, foundational and descriptive research, and performance and quality measurement activities. Results of these evaluation efforts inform programmatic, legislative, budgetary and policy decision-making.

HRSA will use evidence-building and evaluation efforts to promote equity. Executive orders1 charged HRSA with advancing equity and opportunity in government practices, including management; policymaking (through regulation and guidance); procurement, contracting, and budgeting; delivery of benefits and services; and data collection, reporting, and use. Core to HRSA’s mission is a dedication to reducing health disparities and achieving health equity; HRSA’s evidence-building and evaluation efforts will inform planning, implementation, and continuous quality improvement of its health care and public health programs.

To achieve these goals, HRSA staff often need to analyze and assess current and emerging health policy issues, conduct descriptive research, and evaluate HRSA programs under relatively tight time constraints. Activities frequently require personnel to augment HRSA skills and capabilities with skills and experience not available within HRSA.

These activities typically target high-priority issues which are likely to influence HRSA’s budgetary, legislative, regulatory, and planning cycles.

II. PURPOSE/GENERAL DESCRIPTION

The purpose of this Indefinite-Delivery-Indefinite-Quantity (IDIQ) requirement is to provide a contract mechanism to facilitate the production of relevant, high-priority, and robust evaluations and evidence-building activities to support B/Os within HRSA. These

1 Executive Order 13985, Advancing Racial Equity and Support for Underserved Communities through the Federal Government (January 20, 2021); Executive Order 14091, Further Advancing Racial Equity and Support for Underserved Communities Through The Federal Government (February 16, 2023).

activities may include evaluations, policy analyses, descriptive research studies, as well as activities such as the development, implementation, and analyses of quality improvement projects to identify programmatic best practices (e.g., through assessing grantee satisfaction, patient satisfaction, or customer experience), assessment of evaluation capacity, and the development and implementation of performance and quality measurement tools and data collection methods to monitor program performance. The evidence-building and evaluation work conducted under this IDIQ contract will be awarded through individual task orders developed by HRSA Bureaus and Offices.

Section IV, Task Order Requirements, provides additional information about required project management workplans, records and information management, Section 508 compliance, and the HHS Policy for Information Technology Security and Privacy.

The following describes the typical tasks which may be required of the Contractor in the performance of awarded task orders. Task orders may not necessarily include the entire set of task areas listed below; however, they must specify the actual tasks to be performed along with the project-specific information required for completion of the task order.

The activities related to evidence-building and evaluation studies and services are organized to support the following four (4) key Task Areas:

1. Task Area 1: Program Evaluations. Conducting program evaluations, including formative, process, outcome, impact, and economic evaluations.

2. Task Area 2: Other Evidence Building Activities. Conducting other evidence-building activities beyond program evaluation, such as the development of logic models/theories of change, needs assessments, evaluability assessments, organizational assessments, performance and quality measurement, systematic reviews, policy analyses, exploratory studies, descriptive studies, and predictive analytics.

3. Task Area 3: Communication and Dissemination. Communicating and disseminating findings from evaluations and evidence-building activities.

4. Task Area 4: Data Optimization. Optimizing data, filling data gaps, and identifying and implementing improvements to data, including collecting, cleaning, validating, linking, and managing data, to support evaluations and evidence-building activities.

III. PERIOD OF PERFORMANCE

The period of performance is a 12-month Base Year and four (4) 12-month Option Years.

TASKS

TASK AREA 1: Program Evaluation

Overview: HRSA conducts a wide range of program evaluations to ensure integrity, efficiency, and effectiveness of the public health programs and the populations served. The development, implementation, and support of evaluation studies are conducted with the goal of collecting information about program performance, outcomes, and impact. This information enhances HRSA’s internal programmatic decision-making about various aspects of the programs. These activities typically target high priority issues which are likely to influence the budgetary, legislative, regulatory, and planning cycles at HRSA.

HRSA conducts a range of program evaluations, including formative, process, outcome, impact, and sometimes economic evaluations; see Appendix A for a definition of these evaluation studies. HRSA is committed to conducting rigorous, relevant, and meaningful evaluations, and supporting the use of evidence from systematic evaluations to inform policy and practice; see Appendix B for HRSA evaluation guidelines. Typically, these evaluation studies or projects are conducted to:

• Understand factors that may impede or contribute to a program’s success.

• Explain linkages between project or program activities and outcomes.

• Assess effects of a project or program beyond its intended objectives.

• Estimate alternative outcomes, what might have occurred in the absence of a project or program, or if an alternative implementation model had been deployed (i.e., later distribution).

• Assess effect of enhancements to projects or programs.

• Assess net impact of a project or program.

• Compare the effectiveness of alternative interventions, projects or programs aimed at the same objectives, benchmark results against other national datasets.

• Estimate the cost effectiveness, return on investment, and cost-benefit/public value of projects or programs.

• Determine long-term sustainability of a program, with or without HRSA resource investments.

Program evaluations can be experimental or non-experimental in design and can include a range of qualitative and/or quantitative data/information for several subject areas.

Below are program evaluation activities included in Task Area 1. These activities are organized by evaluation phases. Activities conducted through Task Area 1 may include one or more of the activities listed below and may be isolated to this Task Area or involve activities in other Task Areas in this IDIQ.

Evaluation Planning and Design

The Contractor shall:

• Develop evaluation questions, an evaluation plan, and an overall research/evaluation design for the study including methodology, sampling technique, data collection, reporting and analysis, and how the overall program will be evaluated.

• Conduct broad-based environmental scans to obtain and review background information on programs, populations served, and contexts.

• Conduct literature and data reviews (e.g., systematic literature reviews, scoping reviews, meta-analyses, and thematic analyses), on relevant and pertinent questions to be addressed as directed by the needs of specific programs within HRSA.

• Develop an evaluation matrix to include evaluation questions, measures, indicators, and instruments that define, describe, and provide a rationale for each within the context of an evaluation plan.

• Account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

• Obtain and review background information on the program to be evaluated using a variety of strategies, such as engaging HRSA program subject matter experts (SMEs), stakeholders, and people with lived experiences.

• Recommend appropriate evaluation design, based on program needs, evaluation type, evaluation aims, and other factors. Include sampling design, quantitative and qualitative data collection processes, statistical plans of analyses, and presenting data results.

Implementation2

• Review existing administrative and operational program data (e.g., application data, progress reports, performance data, and current research, to make recommendations on whether new or revised data collection strategies are necessary for successful implementation of the program evaluation.

• Develop new or revise existing data collection instruments, (including surveys), complying with HHS and OMB standards for data collection.

2 Note: Task 4: Data Optimization presents additional tasks related to ensuring the quality and integrity of data collected.

• Incorporate plain language principles into all data collection instruments.

• Comply with HRSA policies and procedures related to review and clearance of data collection instruments, including surveys, and coordinate with the appropriate HRSA offices responsible for review and clearance.

• Prepare and submit to the Office of Management and Budget (OMB) clearance packages, Institutional Review Board (IRB) Human Subjects Protection requests, and prepare documents for IRB and OMB review, in compliance with HRSA and HHS policies and procedures.

• Collect qualitative data, including through interviews, surveys, focus groups, and observations, using tools and resources approved by HRSA.

• Conduct site visits to support the evaluation (e.g., assessing program implementation, gathering data, and engaging stakeholders).

• Engage HRSA program and recipient staff as well as other stakeholders (e.g., study participants) during the implementation phase.

• Implement procedures such as inter-rater reliability to ensure consistency among data collectors for the same observation(s).

• Collect and abstract utilization and cost data for programs conducting cost-effectiveness studies, cost-benefit analyses, and return on investment analyses.

• Ensure data quality and security, integrity, and availability of data to evaluation team members.

• Conduct ongoing analyses of data using appropriate methodologies, including small sample sizes where and when relevant, and justify why and how certain sampling methodologies overcome issues related to small sample sizes while maintaining the rigor of the evaluation study.

• Collect and abstract utilization and cost data for programs studying new treatments or comparing treatments for HRSA populations of interest.

• Assess awardee and individual perceptions of program availability, accessibility, and program satisfaction.

• Identify best practices and lessons learned for a particular program or among several similar program sites. Best practices and lessons learned may include practices, policies or programs that have been demonstrated by HRSA programs/recipients to be effective. These practices may reflect evidence-based best practices as demonstrated by the literature.

• Through formative evaluation activities, assess whether a program, policy, or organizational approach-or some aspect of these-is feasible, appropriate, and acceptable before it is fully implemented.

• Through process or implementation evaluation activities, assess how a program or service is delivered relative to its intended theory of change.

• Through outcome evaluation, measure the extent to which a program, policy, or organization has achieved its intended outcome(s), without determining the causal impact. Assess the relationship between HRSA-supported activities and various observed outcomes.

• Through impact evaluation activities, assess the causal impact of a program, policy, or organization, or aspect thereof, on outcomes relative to those of a counterfactual3.

• Through economic evaluation, identify, measure, value, and compare the costs and consequences of different public health interventions. The types of economic evaluations anticipated by HRSA include program cost analysis, cost of illness or disease burden, cost-effectiveness analysis, cost-benefit analysis, and return on investment.

Completion4

• Prepare data visualizations or other products to effectively communicate evaluation findings to program stakeholders.

• Prepare reports written to reach a variety of audiences (e.g., internal reports, public reports, reports to Congress, and publications for peer review) as well as in a comprehensive report that provides clear and complete documentation of the issue(s);

methods and data sources; data visualizations; findings; analysis limitations; and conclusions.

• Incorporate program stakeholder comments/recommendations into a final report and/or provide briefings.

• Ensure all deliverables including final reports, summaries of preliminary findings, and briefings shall be 508 compliant, in alignment with HRSA’s current style guides, and reviewed by an editor prior to submission.

• Prepare publications of evaluation study methodology and findings for peer-reviewed publications.

3 https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf 4Note: Task 3: Communication and Dissemination presents additional tasks related to communicating and disseminating findings.

https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf

• Ensure alignment with current HHS and HRSA policies (e.g., through the Office of Communications and/or Office of Information Technology) and other federal regulations regarding execution of Task Orders for evaluations and evidence-building activities and other tasks.

• Implement quality control and quality assurance processes to track and assess progress toward the production, and submission of project deliverables to HRSA.

This assessment shall include performance standards or other metrics to assess level of Contractor performance for project deliverables.

• Engage HRSA program and recipient staff during the completion phase, to inform how findings are disseminated and communicated.

TASK AREA 2: Other Evidence-Building Activities

Overview: Evidence-building activities beyond program evaluation encompass a wide range of activities, such as the development of logic models/theories of change, needs assessments, evaluability assessments, organizational assessments, performance and quality measurement, systematic reviews, policy analyses, exploratory studies, descriptive studies, and predictive analytics, described further in Attachment C. Like program evaluations discussed in Task 1, other evidence-building activities help the ensure integrity, efficiency, and effectiveness of HRSA’s programs and influence budgetary, legislative, regulatory, and planning cycles at HRSA. HRSA is interested in how external context and forces in the health landscape affect HRSA programs and their progress toward achieving goals and objectives. Examples include health and human service system changes (e.g., increases in managed care or changes in the welfare system); social, economic, and political changes (e.g., an increase in populations without health care coverage, etc.); and social determinants of health and health IT operations and limitations.

Below are examples of activities that may be addressed through Task Area 2. These activities align with the topic areas listed in Attachment C. Activities conducted through Task Area 2 may include one or more of the examples listed below and may be isolated to this Task Area or involve activities in other Task Areas in this IDIQ.

Many of the activities within Task Area 2 will necessitate working with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data and vital records).

Logic Models/Theories of Change

• Develop a logic model or theory of change for programs or initiatives, performing the necessary research and review activities to identify and link contexts, program inputs, program activities, program outputs, and short-term, intermediate term, and long-term outcomes, ensuring that assumptions are included, and incorporating key evidence as available and appropriate.

Needs Assessments

• Design and conduct needs assessments to identify potential service populations in their catchment area and the individualized needs of diverse and underserved populations.

• In compliance with HRSA policies and procedures, conduct surveys, questionnaires, interviews, focus groups, and/or observations, and/or analyze data such as performance measures, to inform the development of effective programs that address existing needs and wants.

• Identify discrepancies or gaps between the current state, situation, or condition, and the preferred/desired state, condition, or situation.

• Develop needs assessment findings to inform how appropriate interventions and outreach efforts shall be tailored to address the needs of targeted populations.

Evaluability Assessments

• Design and conduct evaluability assessments to inform whether, when, and how a program evaluation shall be conducted.

• Review a program’s design, theory of change/logic model, and program documentation to study whether a program will achieve desired goals.

• Engage program stakeholders (e.g., program staff, program participants or beneficiaries) individually or in groups through structured interviews, observation, or focus groups to learn how the program operates in practice.

• Assess the program’s capacity for data collection, management, and analysis.

• Develop analyses describing the feasibility of conducting a program evaluation (evaluability assessment report).

Organizational Assessments

• Design and conduct organizational assessments to investigate the processes, work environment, and structure of an organization, Bureau/Office, or subcomponent.

• Collect and analyze stakeholder input, administrative and operational data, and related information to support the organizational assessment.

• Develop reports outlining the methodology, findings, and conclusions of the organizational assessment.

Performance and Quality Measurement

• Assess current performance and quality measures and associated data collection for the purposes of determining and improving their value and utility or proposing alternatives.

• Assess performance data for programs with small sample sizes (e.g., clinic settings with a low volume of patients) and experience working with small populations.

• Assess and support alignment of programmatic performance and quality measures with national benchmarks (e.g., Healthy People objectives).

• Solicit or develop recommendations on performance and quality measures, indicators, and metrics, engaging internal and external stakeholders as well as subject matter experts.

• Identify and analyze data elements that need to be collected to assess program performance.

• Develop performance measures and associated data collection instruments to assess program performance, improve program monitoring efforts (e.g., programmatic, clinical, fiscal), to support continuous quality improvement.

• Develop cross-cutting metrics that may help to assess multiple programs with common areas of focus (e.g., oral health services; health IT; COVID-19 response and recovery).

• Develop a comprehensive performance measurement plan that satisfies performance and accountability requirements, such as the requirements established by the Office of Management and Budget in Circular A-11, Part 6.

• Provide recommendations to incorporate equity into performance and quality measurement activities; where appropriate, ensure that performance and quality measures reflect best practices in health equity measurement, to inform public health systems approaches, health disparities research, public health programming, sustainability, and cost-effectiveness of HRSA health initiatives at the community, local, county, and state levels.

Systematic Reviews

• Design and conduct systematic reviews to identify, appraise, and synthesize relevant studies on a HRSA-relevant topics.

• Follow established best practices for conducting systematic reviews, that would include formulating review questions defining inclusion and exclusion criteria, developing a search strategy, selecting studies, extracting data, assessing study quality, and analyzing and interpreting results.

• Prepare summary reports presenting methodology, findings, and conclusions.

Policy Analyses

• Develop a clear statement of the relevant existing policy issue(s).

• Compile qualitative and quantitative data, previous analyses, previous policy/mission statements, previous projection analyses, and other relevant information.

• Conduct scans of relevant Federal and state laws and policies that impact HRSA programs.

• Develop case studies to conduct in-depth descriptions or analyses of specific components of a program, project, or policy.

• Assess effectiveness and/or impact of policy-related strategies implemented in the field to address a specific public health issue.

• Solicit and document recommendations and expert opinion(s) on the implementation of various legislative and regulatory requirements impacting HRSA programs.

• Identify the strengths and challenges of identified policy/program alternatives, together with an assessment of their likely future success and possible impediments.

• Prepare an analysis of policy/program alternatives, using appropriate methodologies.

• Assess various policy alternatives related to HRSA programs or interests detailing how actual, planned, or potential changes in current public and private policies and implementation strategies affect achievement of program goals and objectives.

• Identify policy alternatives for consideration by HRSA programs; these alternatives would include potential benefits (pros) as well as challenges (cons).

• Describe areas for innovation and/or adapting to technological advances and the impact on health care (e.g., telehealth, mobile health delivery systems, health IT).

Exploratory Studies

• Design and conduct exploratory research to investigate a new topic(s) or issue(s) or generate/propose new ideas or new hypotheses on an existing topic.

• Conduct observations, surveys, interviews, focus groups, literature reviews and/or case studies to answer the key questions of the research study.

Descriptive Studies

• Design and conduct descriptive statistics and/or descriptive analytics to examine key health care and public health questions, summarizing the basic features of study data in a quantitative manner.

• Identify, combine, and analyze current and historical data to identify trends and relationships.

• Use statistical software to identify trends and relationships between variables and prepare data visualizations.

• Prepare summary reports presenting methodology, findings, and conclusions, and data visualizations as appropriate.

• Prepare dynamic data visualizations demonstrating key findings.

Predictive Analytics

• Use historical program/project data or other data sets, such as the Area Health Resource Files, to develop and use predictive analytics to strengthen methodologies for predicting health outcomes and understanding complex health care issues. These predictive analytics shall use a variety of analytic tools, such as statistical modeling, data mining techniques, machine learning algorithms, and/or artificial intelligence.

• Identify patterns and develop projections in data, such as for demographic groups, program financial and service projections, service outcomes and program performance, to identify risks and opportunities for policy and programmatic changes.

• Prepare summary reports presenting methodology, findings, and conclusions, and data visualizations as appropriate.

• Prepare dynamic data visualizations demonstrating key findings.

Completion of Evidence-Building Activities5

• Prepare data visualizations or other products to effectively communicate findings from other evidence-building activities to program stakeholders.

• Prepare reports written to reach a variety of audiences (e.g., internal reports, public reports, reports to Congress, and publications for peer review) as well as in a comprehensive report that provides clear and complete documentation of the issue(s);

methods and data sources; data visualizations; findings; analysis limitations; and conclusions.

• Incorporate program stakeholder comments/recommendations into a final report and/or provide briefings.

• Ensure all deliverables including final reports, summaries of preliminary findings, and briefings shall be 508 compliant, in alignment with HRSA’s current style guides, and reviewed by an editor prior to submission.

• Prepare publications of methodology and results for peer-reviewed publications.

• Ensure alignment with current HHS and HRSA policies (e.g., through the Office of Communications and/or Office of Information Technology) and other federal regulations regarding execution of Task Orders for evaluations and evidence-building activities and other tasks.

• Implement quality control and quality assurance processes to track and assess progress toward the production, and submission of project deliverables to HRSA.

This assessment shall include performance standards or other metrics to assess level of Contractor performance for project deliverables.

Engage HRSA program and recipient staff during the completion phase, to inform how findings are disseminated and communicated.

TASK AREA 3: Communication and Dissemination

Overview: Communicating and disseminating findings from evaluations or other evidence-building activities is a significant and critical task. This task area includes the development of approaches to communicate and disseminate results/findings from HRSA

5Note: Task 3: Communication and Dissemination presents additional tasks related to communicating and disseminating findings.

program evaluations and other evidence-building activities. The Contractor shall develop approaches that effectively promote, market, and disseminate, as needed, results/findings from HRSA evaluations (e.g., previously conducted studies and/or evaluation-related work). The Contractor shall coordinate with HRSA’s Office of Communication (e.g., compliance with the policies and practices of this Office that involve publishing data and evaluation results to the HRSA.gov website or internal HRSA-only website). For all materials, the Contractor shall develop reports, publications, and presentations that comply with Section 508 of the Rehabilitation Act, the Plain Language Act, and the HRSA style guide. The Contractor shall ensure that all materials include HRSA branding standards and use HRSA-approved templates. Additionally, communication and dissemination efforts shall be culturally competent and responsive to the diverse HRSA audience.

Below are examples of activities that may be addressed through Task Area 3. Activities conducted through Task Area 3 may include one or more of the examples listed below and may be isolated to this Task Area or involve activities in other Task Areas in this

IDIQ.

Summarizing Program Evaluation and Other Evidence-Building Activities Findings

• Develop policy and research briefs, fact sheets and summary reports from program evaluations and other evidence-building activities.

• Prepare manuscript(s) for publication.

Promoting Use of Program Evaluation and Other Evidence-Building Activities Findings

• Develop plans for optimal implementation and replication of successful models of care among HRSA programs and recipients.

• Develop mechanisms (e.g., reports, graphical representations, mapping, dashboards) to disseminate results and findings, models of care, and evidence-based best practices among HRSA programs, recipients, and stakeholders, including data visualizations and web-based tools, as appropriate.

• Develop toolkits on evaluation and other evidence-building activities for HRSA audiences.

• Develop and disseminate resources that promote the translation of evaluation findings into improved practice for service providers. For example, using established Dissemination and Implementation frameworks to strengthen HRSA’s dissemination efforts.

Translating Program Evaluation and Other Evidence-Building Activities Findings into Other Formats

• Maintain an image library for communication materials that is inclusive of HRSA audiences.

• Prepare presentation materials for meetings, conferences, and webinars, such as PowerPoint presentations, abstracts, and handouts.

• Develop videos and other audio-visual material, depending on the need of the programs.

• Develop media materials, social media graphics (e.g., infographics), web content (text, audio, and video), blogs, and articles.

• Develop targeted messaging designed for specific audiences using a variety of methods and research mediums.

• Prepare briefings for various audiences, including lay audiences and subject matter experts, considering the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s). Plan and develop OMB/GPRA clearance packages.

• Leverage HRSA communications mechanisms (e.g., radio media tour, social media accounts) and existing HRSA platforms (i.e., development of messages for posting) to disseminate materials.

• Draft listserv or broadcast messages for dissemination in HRSA newsletters or to share with HRSA stakeholders for dissemination.

• Develop mechanisms for gathering stakeholder feedback on meetings, conferences, and webinar presentations.

• Plan and manage for Continuing Medical Education (CME) credits and Continuing

Educations Units (CEU).

Planning and Documentation of Communication and Dissemination Activities

• Identify the target audiences for communication and dissemination efforts, including recommendations from HRSA, and develop a communication and dissemination plan for deliverables from program evaluations and other evidence-building activities.

• Develop and execute communication plans that address the information need(s) of HRSA programs, including providing evidence-based findings that promote enhanced programmatic decision-making and that convey complex information in intuitive and meaningful ways for the identified target audience(s)

• Identify stakeholder engagement roles and points of contact and incorporate into communication and dissemination plans.

• Prepare summaries of communications and dissemination activities undertaken during the period of performance.

• Provide monthly social media metrics report with recommendations for improvements based on analysis where required.

• Optimize the use of social media by providing strategies, development, consulting, design, and message execution for multiple social media platforms.

TASK AREA 4: Data Optimization

Overview: This Task Area includes implementation actions for optimizing data, filling data gaps, and identifying and implementing improvements to data, including collecting, cleaning, validating, linking, and managing data, to support program evaluations and other evidence-building activities. The Contractor’s activities in this task area shall align and be compatible with policies and procedures associated with patient protection (e.g., alignment with OMB and HRSA Human Subject Protection Requirements).

Below are activities that may be addressed through Task Area 4. Activities conducted through Task Area 4 may include one or more of the examples listed below and may be isolated to this Task Area or involve activities in other Task Areas in this IDIQ.

Data Collection, Cleaning, Validation, Linking To Support Data Analysis

• Conduct data collection and preparation in support of evaluation studies and evaluation-related activities among HRSA programs and recipients.

• Identify appropriate data analysis methods and data for analysis, and design analytical approaches.

• Assess gaps and inconsistencies or variations in data to ensure consistency and the highest quality of data and analyses.

• Conduct activities such as data cleaning, checking accuracy and consistency needed to prepare data for analysis.

• Convert raw data files into summary/analytic files.

• Compile, link, aggregate, and consolidate data from multiple data sources into a summary/analytic file.

• Develop statistical code for repeatable processes for consistently integrating and merging disparate data sets, to support data analysis.

• Develop approaches to improve the interoperability of HRSA-collected and HRSA-acquired data, including consistent use of geographic data, unique identifiers or data formats that are easier to manipulate and import into analytic software packages.

• Develop statistical models using SAS, STATA, R, SPSS, Excel, Python, Tableau, NVivo (for qualitative analyses) or other appropriate analytical and/or programming/modeling tools.

• Prepare results of analysis, or other data-related visualizations, presentations, or dissemination.

• Identify trends in data; interpret data tabulations and graphs; identify critical factors such as data limitations, biases, sampling errors, and inconsistencies.

• Revise existing data collection instruments, including survey design.

• Incorporate plain language principles into the development of data collection instruments.

Data Purchase and Acquisition

• Identify and acquire/purchase data to support evaluation and other evidence-building activities.

• Work with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data, and CDC vital statistics records).

Documentation

• Develop data documentation such as metadata and data dictionaries.

• Develop instruction manuals and training materials to support data collection for evaluation studies and evaluation-related activities.

Privacy

• Ensure no Protected Health Information (PHI) or Personally Identifiable Information (PII), or other sensitive data is released to the public, such as through de-identifying, anonymizing, or aggregating data.

• Store datasets/files on a secure server that complies with HHS security policies on data collection, transmission, and storage; follow all HRSA, HHS, and Federal IT policies including firewalls, data encryption, two-factor authentication, privacy, security.

Training

• Provide relevant training to HRSA staff on the data optimization activities to support program evaluation and other evidence-building activities.

Support for Communication and Dissemination

Coordinate with HRSA’s Offices of Information Technology and Communication to ensure compliance with the policies and practices of these Offices regarding program evaluation and other evidence-building activities that involve publishing data to a HRSA.gov website or internal HRSA-only website.

IV. TASK ORDER REQUIREMENTS

Task orders awarded under this parent IDIQ contract shall include the following Contractor requirements (see below).

Project Management Workplan

• Include a project management work plan. The plan shall include:

o tasks, activities, milestones, deliverables, and communications strategies.

o provisions for formal and informal written and oral communication with the Task Order COR on project progress.

o status updates throughout the project period.

o The dates for submission of the monthly/quarterly progress reports.

o risk identification, assessment, and mitigation strategies.

HRSA Records and Information Management

• Comply with HRSA Records and Information Management requirements, including certified completion of Records Management Training within 30 days after contract award and upon new staff onboarding, and thereafter completing the annual refresher course during the life of the contract.

Section 508 Compliance

• Align/Comply with all Federal standards and guidelines related to section 508.

Specifically, the contractor shall ensure all products and services delivered under this contract are compliant with Section 508 in accordance with the Health and Human Services Acquisition Regulation (HHSAR).

These Section 508 Standards were issued by the United States Access Board (https://www.access-board.gov/) and published in the Federal Register, on January 18, 2017, as the final rule (https://www.access-board.gov/ict/). The final rule updates the Section 508 Standards along with accessibility guidelines for telecommunication products and equipment covered by section 255 of the Communications Act. The HHS Section 508 Product Assessment Template (PAT) that is referenced above in 352.239- 73 Electronic Information and Technology Accessibility Notice, Paragraph (c), has been replaced with the Voluntary Product Accessibility Template Version (VPAT) 2.1 or later, and is considered to meet HHS PAT requirement. The new form can be downloaded from https://www.itic.org/policy/accessibility/vpat.

HHS Policy for IT Security and Privacy Language

• Comply with HHS Policy(ies) on security and privacy for information technology procurements. More detailed information about these policies may be found in Appendix D.

https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.access-board.gov%2F&data=05%7C01%7CEDeCoster%40hrsa.gov%7C3968fb6e896d47812d9108db6377881a%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638213137258212631%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XPJ6XTp9qnN%2FIAbNYdBMuqKLxC0Q6xBrVEon%2Ftwk1Go%3D&reserved=0 https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.access-board.gov%2Fict%2F&data=05%7C01%7CEDeCoster%40hrsa.gov%7C3968fb6e896d47812d9108db6377881a%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638213137258212631%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PbPONzS6LGcNmkRtZgSmDonNb35sxS07tYIRfEIofhc%3D&reserved=0 https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.acce%2F&data=05%7C01%7CEDeCoster%40hrsa.gov%7C3968fb6e896d47812d9108db6377881a%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638213137258212631%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GDRk4bdg%2FioRtYQEKjfCXdbS4f0xIeYNar8KrpPFEQU%3D&reserved=0 https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.itic.org%2Fpolicy%2Faccessibility%2Fvpat&data=05%7C01%7CEDeCoster%40hrsa.gov%7C3968fb6e896d47812d9108db6377881a%7C14b77578977342d58507251ca2dc2b06%7C0%7C0%7C638213137258368849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xbSqxBL8XoP9ztDFuLGsfmPzYEwFs8Bbjp9303RP1js%3D&reserved=0

Section J Attachment A – Appendix A

APPENDIXES

Appendix A: Examples of Evaluations that may be conducted in Task Area 1: Evaluations.

The following definitions align with guidance provided by OMB.

FORMATIVE EVALUATION

Formative Evaluation is typically conducted to assess whether a program, policy, or organizational approach-or some aspect of these-is feasible, appropriate, and acceptable before it is fully implemented. It may include process and/or outcome measures. A formative evaluation focuses on learning and improvement and does not aim to answer questions of overall effectiveness.

IMPACT EVALUATION

Impact Evaluation assesses the causal impact of a program, policy, or organization, or aspect thereof, on outcomes relative to those of a counterfactual6. In other words, this type of evaluation estimates and compares outcomes with and without the program, policy, or organization, or aspect thereof. Impact evaluations include both experimental (i.e., randomized controlled trials) and quasi-experimental designs. An impact evaluation can help answer the question, “does it work?” or “did the intervention lead to the observed outcomes?”

PROCESS OR IMPLEMENTATION EVALUATION

Process or Implementation Evaluation assesses how the program or service is delivered relative to its intended theory of change, and often includes information on content, quantity, quality, and structure of services provided. These evaluations can help answer the question, “was the program, policy, or organization implemented as intended?” or “how is the program, policy, or organization operating in practice?”

OUTCOME OR SUMMATIVE EVALUATION

Outcome or Summative Evaluation measures the extent to which a program, policy, or organization has achieved its intended outcome(s) and focuses on outputs and outcomes to assess effectiveness. Unlike impact evaluation above, it typically cannot discern causal attribution.

Importantly, it is distinct from, but complementary to, performance measurement, as noted below. An outcome evaluation can help answer the question, “were the intended outcomes of the program, policy, or organization achieved?”

ECONOMIC EVALUATION

Economic Evaluation identifies, measures, values, and compares the costs and consequences of different public health interventions. Allocating resources and implementing these interventions whether policies or programs require an understanding of the relationships between resources used and health outcomes achieved by the program or intervention. Economic evaluations can

6 https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf

Section J Attachment A – Appendix A consider both resources used, and health outcomes achieved simultaneously. They are also useful in supporting decision-making when resources are limited. The types of economic evaluations anticipated by HRSA include program cost analysis, cost of illness or disease burden, cost-effectiveness analysis, and cost-benefit analysis.

Section J Attachment A – Appendix B

Appendix B: HRSA Evaluation Guidelines

These Evaluation Guidelines reaffirm HRSA’s commitment to conducting rigorous, relevant, and meaningful evaluations, and supporting the use of evidence from systematic evaluations to inform policy and practice. HRSA seeks to promote the standards of rigor, relevance and utility, independence and objectivity, transparency, ethics, and professional competence in alignment with the HHS Evaluation Policy. These standards are further defined below.

RIGOR

HRSA is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within statutory, budget, and other constraints. Rigor is required for all types of evaluations, including impact and outcome evaluations, implementation and process evaluations, descriptive studies, needs assessments, and feasibility studies. Rigor requires ensuring that (a) inferences about cause and effect are well-founded (internal validity); (b) clarity about the populations, settings, or circumstances to which results can be generalized (external validity) is maximized; and (c) measures are used that accurately capture the intended information (measurement reliability and validity). HRSA Bureaus and Offices should select appropriate analytic approaches and methods based upon the evaluation questions, the type of program, the status of program implementation, the timeline needed for findings, the use of the findings, and the intended audience for findings.

RELEVANCE AND UTILITY

Evaluations should reflect legislative requirements and Congressional interests, HHS and HRSA priorities; HRSA partners (e.g., states, territories, tribes, and local grantees) and the diverse populations they serve; and other stakeholders. Evaluations should be designed to inform key programmatic decisions and program effectiveness and efficiency. Evaluation results should be incorporated into planning, developing, managing, and implementing HRSA programs.

INDEPENDENCE AND OBJECTIVITY

Independence and objectivity are core principles of evaluation. Agency and program leadership, program staff, grantees, and other HRSA constituents should participate actively in (a) setting evaluation priorities, (b) identifying evaluation questions, and (c) assessing the implications of findings. Evaluation functions are insulated from undue influence and from both the appearance and the reality of bias and conflict of interest. To promote objectivity, HRSA protects independence in the design, conduct, and analysis of evaluations.

Section J Attachment A – Appendix B

TRANSPARENCY

Evaluation planning should include the identification of appropriate constituencies and decision-makers (except when inconsistent with the Freedom of Information Act or Privacy Act) with whom evaluation findings shall be shared in a timely and meaningful way. HRSA Bureaus and Offices should disseminate information about evaluations to potential applicants, program providers, program administrators, policymakers, and funders. HRSA will share the evaluator, study plans, methods, and findings related to evaluations. HRSA will share results of all evaluations that are not specifically focused on internal management, legal, or enforcement procedures or that are not otherwise prohibited from disclosure. Evaluation reports will present all results, including favorable, unfavorable, and null findings. HRSA will release evaluation results in a timely manner and will archive evaluation data for secondary use by interested researchers (e.g., public use files with appropriate data security protections).

ETHICS

HRSA-sponsored evaluations will be conducted by agency staff and contractors in an ethical manner and safeguard the dignity, rights, safety, and privacy of participants. Evaluations will comply with both the spirit and the letter of relevant requirements such as regulations governing research involving human subjects and reducing public burden.

PROFESSIONAL COMPETENCE

Evaluations, whether conducted by internal agency staff or outside contractors, should be performed by professionals with appropriate training and experience for the evaluation activity.

HRSA should encourage diversity and cultural and linguistic competence among those carrying out evaluations. HRSA maintains an evaluation workforce with training and experience appropriate for planning and overseeing a rigorous evaluation portfolio, recruiting staff with advanced academic degrees and experience in varied disciplines (e.g., sociology, psychology, economics, public policy, and public health). HRSA commits to providing professional development opportunities so staff can keep their evaluation and methodological skills current.

Contractors and grantees conducting evaluations on behalf of HRSA should have appropriate expertise to follow these evaluation guidelines

Section J Attachment A – Appendix C

Appendix C: Examples of Evidence-Building activities that may be conducted in Task Area 2: Other Evidence-Building Activities.

The following descriptions provide some examples of work undertaken for the Evidence- Building task.

DESCRIPTIVE STUDIES

Descriptive studies can be quantitative or qualitative in nature, and seek to describe a program, policy, organization, or population without inferring causality or measuring effectiveness. While not all descriptive studies are evaluations, some may be used for various evaluation purposes, such as to understand relationships between program activities and participant outcomes, measure relationships between policies and particular outcomes, describe program participants or components, and identify trends or patterns in data.

EVALUABILITY ASSESSMENT

Evaluability assessments examine whether it is conceptually and methodologically reasonable and feasible to conduct an evaluation. Evaluability assessments are used to identify the scope, scale, and type of evaluation that can be accomplished. Evaluability assessments can guide decisions about whether a program or initiative is appropriate or ready for evaluation (e.g., the program is operating as planned and at a level which shall be expected to produce impacts), and the type of evaluation designs that are suitable. Evaluability assessment findings can help determine whether an implementation, outcome, or impact evaluation is appropriate to conduct for a given program.

LOGIC MODEL/THEORY OF CHANGE

A logic model illustrates the relationship between a program’s activities and its intended outcomes.

A theory of change is a comprehensive description of why a desired change is expected to happen in a specific context.

NEEDS ASSESSMENT

Needs assessments aim to determine the nature and extent of the issues that a proposed or existing program shall address including: (1) assessing the needs of different stakeholders (e.g., program administrators, participants); (2) developing appropriate program goals for proposed programs; and (3)…