Section_J_Attachment_A_-_Statement_of_Work.pdf

HRSA Evaluation IDIQ; 19-250-SOL-00046 Section J Attachment A

Statement of Work Project Title: Evaluation Studies and Evaluation-Related Services

I. BACKGROUND:

The mission of the Health Resources and Services Administration (HRSA) is to improve health and achieve health equity through access to quality services, a skilled health workforce and innovative programs. HRSA assures the availability of quality health care to low income, uninsured, isolated, vulnerable and special needs populations.

The Bureaus and Offices within HRSA that administer these programs are responsible for ensuring policy analysis, data analysis and program evaluation studies are conducted to improve the design and performance of current programs and introduce refinements/improvements as needed. In addition, the results of these evaluation efforts inform programmatic, legislative, budgetary and policy decision-making by the Administrator, the Deputy Administrator and other senior officials.

The development, implementation, and support of evaluation studies, and evaluation-related services related to HRSA projects or programs are conducted with the goal of collecting empirical information about program performance, outcomes and/or impact to enhance programmatic decision-making and/or make judgments about various aspects of programs.

To accomplish these activities, staff often need to analyze and assess current and emerging health policy issues, develop data systems, and evaluate HRSA programs under relatively tight time constraints. Activities frequently require personnel to augment HRSA skills and capabilities with skills and experience not available within the Agency.

These activities typically target high priority issues which are likely to influence the budgetary, legislative, regulatory and planning cycles of the Agency.

II. PURPOSE/GENERAL DESCRIPTION:

The purpose of this Indefinite-Delivery-Indefinite-Quantity (IDIQ) requirement is to provide a contracting mechanism to facilitate the production of focused, high-priority, evaluation activities in support of various Bureaus and Offices within the Health Resources and Services Administration (HRSA). These activities may include evaluation studies, evaluation-related services, and policy studies. Activities may also include the development, implementation, and analysis of quality improvement projects to identify programmatic best practices. Additionally, contract activities may include assessing evaluation capacity and the development and implementation of performance measurement tools and data systems to monitor program performance.

The following describes the typical tasks which may be required of the contractor in the performance of awarded task orders. Task orders shall not necessarily reflect/include the entire set of task areas listed below, but shall specify the actual tasks to be performed along with the project-specific information required for completion of the task order.

The Contractor shall furnish all personnel, materials, facilities, services, and equipment necessary for the performance of work as stated in any individual Task Order (TO) issued under this Multiple Award IDIQ Contract.

The activities related to evaluation studies and services may be organized into support for five (5) key Task Areas:

1. Task Area 1 – Evaluation Studies and Evaluation-Related Services

2. Task Area 2 – Contextual/Policy Analyses of Issues Which May Impact HRSA- Supported Activities

3. Task Area 3 – Targeted Support for Evaluation Study, Evaluation-Related Service and Policy Study Components

4. Task Area 4 – Communication and Dissemination of Evaluation Study, Evaluation-Related Service and Policy Study Findings

5. Task Area 5 – Evaluation-Related Data Collection, Analysis and Preparation

III. PERIOD OF PERFORMANCE:

The contract shall include a 12-month Base Year and four 12-month Option Years.

IV. TASKS:

The IDIQ contract shall include the following task areas:

TASK AREA 1: Evaluation Studies and Evaluation-Related Services

Overview: Evaluation studies of projects or programs are typically conducted to:

(1) understand factors that may impede or contribute to the program’s success;

(2) explain linkages between project or program activities and outcomes;

(3) assess effects beyond intended objectives;

(4) estimate what would have occurred in the absence of a project or program;

(5) assess effect of enhancement to project or program;

(6) assess net impact of a project or program; and/or

(7) compare the effectiveness of alternative interventions, projects or programs aimed at the same objectives.

Evaluation studies can be experimental or non-experimental in design and can include a range of qualitative and/or quantitative data/information for a number of subject areas.

Evaluation-related services can encompass a wide range of activities, but does not necessarily constitute conducting a full evaluation. Such activities may include the development of logic models, performance measures, and/or evidence-based intervention strategies and can include a range of qualitative and/or quantitative data/information for a number of subject areas.

Examples of Evaluation Studies and Evaluation-Related Services That May Be Addressed In Task Area 1:

Program functioning

(1) Monitoring and analysis of implementation activities associated with f planned program models or designs.

(2) A systematic evaluation of operations (i.e., the best way to accomplish program goals).

(3) Program functioning, including assessment of the efficiency of program management activities.

(4) Assessment of the relationship between HRSA-supported activities and various observed outcomes.

(5) Needs assessment or gap analysis to identify program improvements.

(6) Assessment of related HRSA public health practice activities and intended and unintended outcomes.

(7) Consumer/service recipient perception of program availability and accessibility.

(8) Identification of “best practices” implemented among a number of similar program sites. “Best practices” may include practices, policies or programs that have been demonstrated by HRSA programs/recipients to be effective. These practices may reflect evidence-based best practices as demonstrated by the literature.

(9) Assessment of performance data for programs with small sample sizes (e.g., clinic settings with a low volume of patients) and experience working with small populations.

(10) Development of logic or conceptual models to theoretically link program inputs, activities with program outputs and outcomes.

Program performance and quality improvement

(1) Implementation of quality improvement approaches and results.

(2) Achievement of operational targets, meeting standards, and following national guidelines.

(3) Development of performance measures and data collection instruments to assess program performance and improve program monitoring efforts (e.g., programmatic, clinical, fiscal).

(4) Analysis of a program’s cost-effectiveness and cost benefit(s).

(5) Assessment of validity/reliability of data, indicators and data collection instruments/measures used to evaluate performance.

(6) Determination of unintended effects of HRSA-supported activities.

Measurement of program effectiveness

(1) Development, implementation and/or analysis of basic/immediate programmatic outputs and outcomes (e.g., program products/materials such as training/health promotion) to assess short-term program effectiveness and efficiency.

(2) Development, implementation and/or analysis of intermediate programmatic outputs and outcomes to assess intermediate/mid-range program effectiveness and efficiency.

(3) Development, implementation and/or analysis of longer-term programmatic outputs and outcomes to assess longer term/range program effectiveness and efficiency.

(4) Development of cross cutting metrics that may comprehensively help to assess programs with a single/common focus (e.g., oral health services; health IT).

(5) Determination of program sustainability (i.e., is the program sustainable without fiscal or programmatic resources from HRSA).

(6) Assessment and support of alignment of programmatic performance measures with national benchmarks (e.g., Healthy People objectives).

(7) Assessment of the effectiveness and/or impact of the implementation of specific public health related programmatic strategies.

Table 1: Examples of Types of Evaluation Studies That May Be Conducted in Task Area 1

TABLE 1. EXAMPLES OF TYPES OF EVALUATION STUDIES THAT MAY BE

CONDUCTED IN TASK AREA 1

STUDY TYPE DESCRIPTION

Needs Assessment

Aim to determine the nature and extent of the issues that a proposed or existing program should address including: (1) assessing the needs of different stakeholders (e.g., program administrators, participants); (2) developing appropriate program goals for proposed programs; and (3) determining how a new or existing program should be designed or modified to achieve those goals.

Evaluability Assessment (EA)

Address whether or not it is conceptually and methodologically reasonable and feasible to conduct an evaluation. EAs are used to identify the scope, scale, and type of evaluation that can be accomplished. EAs can guide decisions about whether a program or initiative is appropriate or ready for evaluation (e.g., the program is operating as planned and at a level which would be expected to produce impacts), and the type of evaluation designs that are suitable. EA findings can help determine whether an implementation, outcome, or impact evaluation is appropriate to conduct for a given program.

Organizational Assessment (OA)

Assess an organization’s (e.g., HRSA program and/or recipient) evaluation capability, capacity and/or readiness to conduct evaluation activities. OAs can include needs assessment, identification of strategic priorities, objectives and/programmatic goals, and development of recommendations regarding evaluation capacity building requirements within the organization.

Implementation (e.g., process) Evaluation (IE)

Address questions about how and to what extent programs are carried out as planned.

IEs can address and/or assess (1) how well the actual program operations align with the intended program design; (2) the quality and/or efficiency of program operations;

and/or (3) program activities’ conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations. An IE may take place early in the life of the program to help with near-term refinements.

TABLE 1. EXAMPLES OF TYPES OF EVALUATION STUDIES THAT MAY BE

CONDUCTED IN TASK AREA (cont.) 1

STUDY TYPE DESCRIPTION

Performance Measurement

(PM)

Entails ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals. Performance measures may address: 1) type or level of program activities conducted (process); 2) direct products and services delivered by a program (outputs); or 3) the results of those products and services (outcomes) or 4) cross-cutting activities involving programs with a related topic. PM tracks progress toward intended program outcomes, so program managers and/or administrators can (1) identify who a program is serving; (2) monitor implementation of programs or policies; and/or (3) identify challenges with implementing a program.

PM focuses on whether a program has achieved its objectives, expressed as measurable performance standards. PM also aims to support resource allocation and inform policy decisions to improve service delivery and program effectiveness and enhance public accountability.

Outcome Evaluation (OE)

Assess whether a program is achieving its intended purpose(s) or objective(s) and/or whether outcomes differ across program components, providers, or recipients. OE focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are generated. Examples of program outputs and outcomes include number of youth reached, number of clients served, number of research publications or patents, or the results of program products and services (e.g., change in behavior or environmental conditions after exposure to the program). An OE can indicate whether outcomes are occurring, are moving in the expected direction, and/or whether a program is reaching a defined target or benchmark.

Cost-Benefit and Cost- Effectiveness Analyses (CBA)

Compare a program’s outputs or outcomes with the costs (resources expended) to produce them. Cost-effectiveness analysis assesses the cost of meeting a single goal or objective and can identify the least costly alternative for meeting that goal. Cost-benefit analysis aims to identify all relevant costs and benefits, usually expressed in dollar terms. CBA also reviews new business development – conducting market research to leverage the least costly alternative for meeting that goal.

Systematic Reviews (SR)

Assess the effectiveness of individual interventions and/or summarize the research behind broader strategies or programmatic approaches. SRs emphasize transparent procedures to synthesize and evaluate the results of previous research and summarize the best available research on a given topic area.

Case Studies Conduct case studies to assess program or organizational effectiveness. Activities may include site visits, key informant interviews, and/or review of program documents.

Evaluation studies and evaluation-related services shall generally include some, if not all, of the following tasks:

(1) Facilitating stakeholder engagement in the study using appropriate mechanisms (e.g., in person meetings; conference calls; webinars, etc.);

(2) Developing overall research/evaluation design for the study;

(3) Identifying and recruiting study participants;

(4) Coordinating all aspects of study administration;

(5) Convening and conducting a technical expert panel;

(6) Developing data collection instruments;

(7) Collecting qualitative and/or quantitative information;

(8) Developing measurable performance standards related to the study, (e.g., timeliness or quality assurance surveillance plans);

(9) Analyzing collected data; and

(10) Preparing a report that summarizes the essential questions, design, results, discussion and recommendations, as warranted.

Role of Contractor in Task Area 1: For Task Area 1, the Contractor shall determine and, if needed, elucidate evaluation study questions and design an evaluation study and/or evaluation-related service capable of successfully addressing HRSA program/recipient questions, needs, and/or concerns.

The Contractor shall perform a number of activities while executing the above evaluation and evaluation-related tasks, including:

(1) Obtaining and reviewing background information on the program to be evaluated using a variety of strategies such as HRSA program subject matter experts (SMEs) and research conducted by Contractor.

(2) Conducting broad-based environmental scan and/or literature review on relevant/pertinent questions to be addressed.

(3) Conducting interviews with HRSA program and recipient staff as well as other stakeholders (e.g., study participants).

(4) Reviewing data on program operations, and assessing current research on the program to be evaluated.

(5) Choosing the appropriate/needed evaluation design, based on evaluation type, evaluation aims, and other factors.

(6) Developing or revising data collection instruments, including survey design. Plain language principles shall be incorporated into development of data collection instruments.

(7) Preparing and submitting Office of Management and Budget (OMB) clearance packages, Institutional Review Board (IRB) Human Subjects Protection requests, and preparing documents for IRB and OMB review, as appropriate.

(8) Collecting quantitative and/or qualitative data using appropriate strategies, including survey tools, from a variety of HRSA programs, recipients, and special populations (e.g., rural, maternal/child, hard to reach populations).

(9) Coordinating and complying with HRSA Offices of Communication and

Information Technology, as needed, regarding policies and procedures related to review and clearance of data collection instruments, including surveys.

(10) Implementing procedures such as inter-rater reliability to ensure consistency among data collectors for the same observation(s).

(11) Ensuring data quality and security, integrity and availability of data to evaluation team members.

(12) Conducting ongoing analyses of data using appropriate methodologies, including small sample sizes.

(13) Preparing data visualizations or other products to effectively communicate evaluation findings to program stakeholders.

(14) Preparing short reports, topical reports, summaries of preliminary findings, final reports and/or other analytical reports, including detailed methodological appendices where appropriate.

(15) Incorporating program stakeholder comments/recommendations into a final report and/or provide briefings.

(16) Ensuring all products/deliverables including final reports, summaries of preliminary findings, and briefings shall be 508 compliant.

(17) Preparing publications of evaluation study methodology and results for the peer-reviewed literature.

(18) Ensuring alignment with current HHS and HRSA policies (e.g., through the Office of Communications and/or Office of Information Technology) and other federal regulations regarding execution of Task Orders for evaluation studies and/or evaluation-related services

(19) Implementing policies and procedures to track and assess progress toward the production, submission and/or quality of evaluation project deliverables to the HRSA client. This assessment may include a performance standard or other metric to assess level of Contractor performance for project deliverables.

In preparing summary reports, the Contractor shall work to identify the target audience(s) and develop a communication and dissemination plan that (1) addresses the information need(s) of the requestor including providing evidence-based findings that promote enhanced programmatic decision-making; and (2) conveys complex information in intuitive and meaningful ways for all target audience(s).

In addition, the Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

TASK AREA 2: Contextual/Policy Analyses of Issues Which May Impact HRSA- Supported Activities

Overview: It is often necessary to develop forecasts of current trends or likely future scenarios when preparing policy/program analyses. The analysis of contextual/policy issues which may have an impact on HRSA-supported activities are focused primarily on one or more of the following:

(1) External context of programs and how forces in the health landscape affect HRSA programs and their progress toward achieving goals and objectives. Examples include health and human service system changes (e.g., increases in managed care or changes in the welfare system); social, economic and political changes (e.g., an increase in populations without health care coverage, etc.); and social determinants of health and health IT operations and limitations.

(2) Assessment of various policy alternatives related to HRSA programs or interests (i.e., how actual, planned, or potential changes in current public and private policies and implementation strategies affect achievement of program goals and objectives).

(3) Identification of policy alternatives for consideration by HRSA programs; these alternatives would include potential benefits (pros) as well as challenges (cons).

(4) Description and better understanding of areas for innovation and/or adapting to technological advances and the impact on health care (e.g., telehealth, mobile health delivery systems, health IT).

(5) Assessment of effectiveness and/or impact of policy-related strategies implemented in the field to address a specific public health issue.

(6) Provision of recommendations and expert opinion(s) on the implementation of various legislative requirements impacting HRSA programs.

Examples of Task Area 2 Studies May Include:

(1) impact of local, state, and Federal policies that affect the implementation of HRSA programs and recipients.

(2) impact of changes in financial policy by Federal, state, local and/or private payers on the recipients of HRSA funds and their clients or patients.

Studies in Task Area 2 may necessitate working with data and data use agreements at the national, state and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid and public health data sets (e.g., Medicaid claims data and vital records).

Role of Contractor in Task Area 2:

For Task Area 2, the Contractor shall have the capacity to demonstrate experience in assessing policy issues at the Federal, State, and local level that may be relevant to HRSA programs and recipients, prepare demographic characteristic and population projections, develop financial and service projections, make evidence-based projections about service outcomes and system performance indicators, and identify policy alternatives, including strengths and challenges for program consideration.

To prepare such projections, the Contractor shall have the capacity to manipulate data sets of varying size and complexity, and the statistical expertise to use appropriate methods/procedures to conduct and report projection analyses. The Contractor shall also demonstrate an understanding of the nuances associated with data at various levels, particularly at the patient level.

The Contractor shall perform a number of tasks related to policy assessment and analysis, including:

(1) developing a clear statement of the relevant existing policy issue(s);

(2) compiling qualitative and quantitative data, previous analyses, previous policy/mission statements, previous projection analyses and other relevant information;

(3) preparing an analysis of policy/program alternatives, using an appropriate methodology(ies);

(4) identifying the strengths and challenges of each identified policy/program alternatives, together with an assessment of their likely future success and possible impediments; and

(5) conducting scans of relevant Federal and state laws and policies that impact

HRSA programs.

Analyses and findings shall be described in reports suitable for a variety of audiences (e.g., internal reports, public reports, reports to Congress, and publications for peer review) as well as in a comprehensive report that provides clear and complete documentation of the issue(s); methods and data sources; findings; analysis limitations;

and conclusions. Briefings to various audiences, including both well-informed lay audiences and technical audiences shall likely be required.

The Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

TASK AREA 3: Targeted Support for Evaluation Study, Evaluation –Related Service and Policy /Study Components

Overview: This task area shall address specific areas, activities, or components of an evaluation, evaluation-related activity or a policy study. These targeted activities would involve conducting specific tasks to support HRSA evaluation, evaluation-related or policy study needs (rather than conduct a complete evaluation study/activity).

Examples of Task Area 3 Activities May Include:

(1) planning and implementing stakeholder engagement using appropriate mechanisms (e.g., in person meetings; conference calls; webinars, individual interviews, focus groups, etc.);

(2) establishing and managing/coordinating assembly of technical expert panels to provide needed subject matter expertise;

(3) compiling and synthesizing background information (e.g., program history) to guide the evaluation, evaluation-related activity or policy study;

(4) designing and implementing key components of the evaluation study, evaluation-related services activity or policy study, (e.g., development of quantitative and qualitative data collection instruments; designing and administering surveys;

abstracting data from program documents);

(5) collecting data to support the evaluation study, evaluation-related service or policy study;

(6) performing quantitative or qualitative data analyses

(7) developing reports, summaries and assessments based on data analyses and other relevant components of the study and/or services; and/or

(8) disseminating findings through variety of mechanisms (e.g., infographics;

dashboards, etc.).

Role of Contractor in Task Area 3:

The Contractor shall provide targeted activities for evaluation studies, evaluation-related services, and policy studies including administrative/project coordination (e.g., convening stakeholder meetings); technical support (e.g., developing data collection instruments) and data analysis (e.g., performance of qualitative data analysis). These activities may be “stand alone” or part of a larger evaluation /evaluation-related effort.

The Contractor shall account for the organizational capacity, available resources (e.g., TASK AREA 4: Communication and Dissemination of Evaluation Study, Evaluation-Related Service and Policy Study Findings

Overview: This task area shall include the development of approaches to communicate and disseminate results/findings from HRSA evaluation studies, evaluation-related services and policy studies. These approaches shall include a variety of strategies designed for HRSA programs and recipients as well as other parties interested in incorporating this information into their change management, policy development and/or service delivery practices.

Examples of Task Area 4 Activities May Include:

(1) Development of mechanisms (e.g., reports, graphical representations, dashboards) to disseminate results/findings, models of care, and evidence-based best practices among HRSA programs, recipients, and stakeholders. The development of these tools and resources shall include data visualizations and web-based tools, as appropriate and shall be disseminated on HRSA internal or external websites.

(2) Development of dissemination resources that promote translation of evaluation findings into improved practice for service providers.

(3) Development of promotion and marketing strategies that leverage existing HRSA communications mechanisms (e.g., radio media tour, social media accounts).

(4) Support of optimal implementation and replication of successful models of care among HRSA programs and recipients.

(5) Assistance in the development of reports, publications, or presentations which are 508 compliant.

(6) Development of communication plans, including identification of stakeholder engagement roles and points of contact.

(7) Preparation of manuscript for publication.

(8) Provision of PowerPoint presentation(s) or abstract(s) for meeting/conference presentation(s).

(9) Development of policy briefs.

Role of Contractor in Task Area 4: The Contractor shall develop approaches to effectively promote, market, and disseminate, as needed, results/findings from HRSA evaluations (e.g., previously conducted studies and/or evaluation-related work). The Contractor shall also coordinate with HRSA’s Office of Communication (e.g., compliance with the policies and practices of this Office that involve publishing data and evaluation results to a HRSA.GOV website or internal HRSA-only website, for example 508 compliance). Examples include use of the HRSA style guide (including HRSA branding standards on all materials), use of HRSA-approved PowerPoint templates, etc.

The Contractor shall account for the organizational capacity, available resources (e.g., TASK AREA 5: Evaluation-Related Data Collection, Analysis and Preparation

Overview: This task area shall include the development of strategies, documentation, and potential dissemination mechanisms to support data collection, analysis and dissemination for evaluation studies and evaluation-related activities among HRSA programs and recipients.

Examples of Task Area 5 Activities May Include:

(1) Converting raw data files into summary/analytic files.

(2) Compiling, linking, aggregating and consolidating data from multiple data sources into a summary/analytic file.

(3) Conducting statistical analysis and developing statistical models using SAS, STATA, R, SPSS, Excel, Python, Tableau, NVivo (for qualitative analyses) or other appropriate analytical and/or programming/modeling tools.

(4) Developing data documentation such as metadata and data dictionaries.

(5) Developing instruction manuals and training materials to support data collection for evaluation studies and evaluation-related activities.

(6) De-identifying/anonymizing/aggregating data to ensure no PII or sensitive data is released to the public.

(7) Conducting other related activities (e.g., data cleaning) needed to prepare data for analysis, or develop data documentation.

(8) Developing mock-ups of dashboards, reports, maps, or other data visualizations (OIT will develop and publish the final products). Contractor may use Tableau or other OIT approved tools to develop mock-ups/prototypes.

(9) Conducting other related activities (e.g., develop charts, graphs, tables etc.) to prepare results of analysis, or other data-related visualizations, presentations or dissemination.

(10) Providing relevant training to HRSA staff on the analysis of data related to evaluation and/or evaluation-related activities.

Role of Contractor in Task Area 5

The Contractor shall conduct data collection, analysis and preparation in support of evaluation studies and evaluation-related activities among HRSA programs and recipients.

The Contractor shall coordinate with HRSA’s Offices of Information Technology and Communication to ensure compliance with the policies and practices of these Offices regarding evaluation studies and evaluation-related activities that involve publishing data to a HRSA.GOV website or internal HRSA-only website.

The Contractor shall store datasets/files on a secure server that is in compliance with HHS security policies on data collection, transmission, and storage. The Contractor shall follow all HRSA, HHS, and Federal IT policies including firewalls, data encryption, two-factor authentication, privacy, security, etc.

The Contractor’s activities in this task area shall align and be compatible with policies and procedures associated with patient protection (e.g., alignment with OMB and HRSA Human Subject Protection Requirements).

The Contractor shall account for the organizational capacity, available resources (e.g.,