JJi 03282023 HRSA Evaluation IDIQ Sources SoughtI.docx

DEPARTMENT OF HEALTH AND HUMAN SERVICES

March 28, 2023

The Health Resources and Services Administration (HRSA), Office of Acquisitions Management & Policy is issuing this sources sought notice as part of market research to determine the availability and capability of small businesses to provide the services described under task areas below. This is NOT a solicitation for proposals, proposal abstracts, or quotations. The purpose of this notice is to only obtain information regarding availability and capability of qualified small business. Your responses to the information requested will assist the Government in determining the appropriate acquisition method, including whether set-aside is possible. An organization that is not a small business under NAICS code 541720 should not submit a response to this notice.

Background: This Indefinite-Delivery-Indefinite-Quantity (IDIQ) requirement will provide a contracting mechanism to facilitate the acquisitions of focused, high-priority, short-term and long-term evaluation activities in support of various Bureaus and Offices within HRSA.

The overall purpose of the Evaluation IDIQ is to assist the bureaus and offices with:

(1) development, implementation, and support of evaluation studies related to HRSA projects or programs are conducted with the goal of collecting empirical information about program performance, outcomes and/or impact to enhance programmatic decision-making and/or make judgments about various aspects of programs;

and

(2) provision of technical assistance (TA) and training to support evaluation-related needs and to assess and enhance the capacity of HRSA programs to conduct evaluation studies.

Services and experience:

Evidence-Building and Evaluation Services

The development, implementation, and support of evaluation studies related to HRSA projects or programs are conducted with the goal of collecting empirical information about program performance, outcomes and/or impact to enhance programmatic decision-making and/or make judgments about various aspects of programs. These activities related to evaluation studies and services may be organized into four key areas:

· Program Evaluation

· Evidence Building

· Communication and Dissemination

· Data Analytics to Support Evaluations and Evidence-building

Program Evaluation Overview: The development, implementation, and support of evaluation studies related to HRSA projects or programs are conducted with the goal of collecting empirical information about program performance, outcomes and/or impact of the programs and to enhance internal HRSA programmatic decision-making and judgments about various aspects of programs. To accomplish these activities, staff often need to analyze and assess current and emerging health policy issues, develop data systems, and evaluate HRSA programs under relatively tight time constraints. Such activities frequently require personnel to augment HRSA skills and capabilities with skills and experience not available within the Agency. These activities typically target high priority issues which are likely to influence the budgetary, legislative, regulatory, and planning cycles of the Agency. HRSA conducts a range of program evaluations, including formative, process, outcome, impact, and sometimes economic evaluations. Typically, these evaluation studies or projects are conducted to:

· Understand factors that may impede or contribute to a program’s success.

· Explain linkages between project or program activities and outcomes.

· Assess effects of a project or program beyond its intended objectives.

· Estimate what would have occurred in the absence of a project or program.

· Assess effect of enhancements to projects or programs.

· Assess net impact of a project or program.

· Compare the effectiveness of alternative interventions, projects or programs aimed at the same objectives.

Provided below are tasks focused on supporting the program functions that the contractor shall undertake:

(1) Develop overall research/evaluation design for the study.

(2) Monitor and analyze implementation activities associated with planned program models or designs.

(3) Conduct a systematic evaluation of operations (i.e., the best way to accomplish program goals).

(4) Conduct assessment of the efficiency of program management activities.

(5) Assess the relationship between HRSA-supported activities and various observed outcomes.

(6) Assess consumer/service recipient perception of program availability and accessibility.

(7) Identify “best practices” implemented among several similar program sites. “Best practices” may include practices, policies or programs that have been demonstrated by HRSA programs/recipients to be effective. These practices may reflect evidence-based best practices as demonstrated by the literature.

To successfully address HRSA program questions, needs, and concerns the Contractor shall determine and, propose evaluation study questions, determine evaluation study design, and perform other tasks such as:

· Obtain and reviewing background information on the program to be evaluated using a variety of strategies such as HRSA program subject matter experts (SMEs) and research conducted by Contractor.

· Conduct broad-based environmental scans.

· Conduct literature review on relevant and pertinent questions to be addressed.

· Conduct interviews with HRSA program and recipient staff as well as other stakeholders (e.g., study participants).

· Review data on program’s operations and assessing current research on the program to be evaluated.

· Recommend the appropriate evaluation design, based on program needs, evaluation type, evaluation aims, and other factors.

· Develop new data collection instruments, including survey design. Plain language principles shall be incorporated into development of all data collection instruments.

· Revise existing data collection instruments, including survey design. Plain language principles shall be incorporated into development of data collection instruments.

· Prepare and submit to the Office of Management and Budget (OMB), clearance packages, Institutional Review Board (IRB) Human Subjects Protection requests, and prepare documents for IRB and OMB review, as appropriate.

· Collect quantitative and qualitative data using appropriate strategies, including survey tools, from a variety of HRSA programs, recipients, and special populations (e.g., rural, maternal/child, hard to reach populations).

· Coordinate and comply with HRSA Offices of Communication and Information Technology, regarding policies and procedures related to review and clearance of data collection instruments, including surveys.

· Implement procedures such as inter-rater reliability to ensure consistency among data collectors for the same observation(s).

· Ensure data quality and security, integrity, and availability of data to evaluation team members.

· Conduct ongoing analyses of data using appropriate methodologies, including small sample sizes.

· Prepare data visualizations or other products to effectively communicate evaluation findings to program stakeholders.

· Prepare short reports, topical reports, summaries of preliminary findings, final reports, and other analytical reports, including detailed methodological appendices where appropriate.

· Incorporate program stakeholder comments/recommendations into a final report and/or provide briefings.

· Ensure all deliverables including final reports, summaries of preliminary findings, and briefings shall be 508 compliant.

· Prepare publications of evaluation study methodology and results for the peer- reviewed literature.

· Ensure alignment with current HHS and HRSA policies (e.g., through the Office of Communications and/or Office of Information Technology) and other federal regulations regarding execution of Task Orders for evaluations and evidence-building activities and other tasks.

· Implement policies and procedures to track and assess progress toward the production, submission and/or quality of evaluation project deliverables to HRSA Program. This assessment shall include a performance standard or other metric to assess level of Contractor performance for project deliverables.

· Identify the target audience(s) and develop a communication and dissemination plan in preparing summary reports. The plan shall include:(1) addresses the information need(s) of Program including providing evidence-based findings that promote enhanced programmatic decision-making; and (2) convey complex information in intuitive and meaningful ways for all target audience(s).

· Account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

Evidence-Building Overview: Evidence-building activities encompass contextual/policy analysis as well as a wide range of activities, supporting evaluations and policy analysis. For example, it is often necessary to develop forecasts of current trends or likely future scenarios when preparing policy/program analyses and planning for evaluations. Evidence-building activities also include the development of logic models, development of performance and quality measures, and evidence-based intervention strategies for enhancing program effectiveness and efficiency. It also includes qualitative and quantitative data/collection, analyses, and reporting for several subject areas all of which lead to stronger evaluations and evidence-building activities.

The analysis of contextual/policy issues which may have an impact on HRSA-supported activities are focused primarily on one or more of the following:

1. External context of programs and how forces in the health landscape affect HRSA programs and their progress toward achieving goals and objectives. Examples include health and human service system changes (e.g., increases in managed care or changes in the welfare system); social, economic, and political changes (e.g., an increase in populations without health care coverage, etc.); and social determinants of health and health IT operations and limitations.

2. Assessment of various policy alternatives related to HRSA programs or interests (i.e., how actual, planned, or potential changes in current public and private policies and implementation strategies affect achievement of program goals and objectives).

3. Identification of policy alternatives for consideration by HRSA programs; these alternatives would include potential benefits (pros) as well as challenges (cons).

4. Description and better understanding of areas for innovation and/or adapting to technological advances and the impact on health care (e.g., telehealth, mobile health delivery systems, health IT).

5. Assessment of effectiveness and/or impact of policy-related strategies implemented in the field to address a specific public health issue.

6. Provision of recommendations and expert opinion(s) on the implementation of various legislative requirements impacting HRSA programs.

Other activities centered around evidence-building included in this Task Area are:

· Assessment of performance data for programs with small sample sizes (e.g., clinic settings with a low volume of patients) and experience working with small populations.

· Development of logic or conceptual models to theoretically link program inputs, activities with program outputs and outcomes.

· Program performance and quality improvement, for example,

(1) Implementation of quality improvement approaches and results.

(2) Achievement of operational targets, meeting standards, and following national guidelines.

(3) Development of performance measures and data collection instruments to assess program performance and improve program monitoring efforts (e.g., programmatic, clinical, fiscal).

· Measurement of program effectiveness, for example,

(1) Development, implementation and/or analysis of basic/immediate programmatic outputs and outcomes (e.g., program products/materials such as training/health promotion) to assess short-term program effectiveness and efficiency.

(2) Development of cross cutting metrics that may comprehensively help to assess programs with a single/common focus (e.g., oral health services; health IT).

(3) Determination of program sustainability (i.e., is the program sustainable without fiscal or programmatic resources from HRSA).

(4) Assessment and support of alignment of programmatic performance and quality measures with national benchmarks (e.g., Healthy People objectives).

(5) Assessment of the effectiveness and/or impact of the implementation of specific public health related programmatic strategies.

(6) Preparation of demographic characteristic and population projections, development of financial and service projections, evidence-based projections about service outcomes and system performance indicators, and identification of policy alternatives, including strengths and challenges for program consideration.

· Policy assessment and analysis, for example,

(1) Develop a clear statement of the relevant existing policy issue(s).

(2) Compile qualitative and quantitative data, previous analyses, previous policy/mission statements, previous projection analyses and other relevant information.

(3) Prepare an analysis of policy/program alternatives, using an appropriate methodology(ies).

(4) Identify the strengths and challenges of each identified policy/program alternatives, together with an assessment of their likely future success and possible impediments.

(5) conduct scans of relevant Federal and state laws and policies that impact HRSA programs.

Task area 2 will necessitate working with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data and vital records).

All analyses and findings shall be described in reports suitable for a variety of audiences (e.g., internal reports, public reports, reports to Congress, and publications for peer review) as well as in a comprehensive report that provides clear and complete documentation of the issue(s); methods and data sources; findings; analysis limitations; and conclusions.

Briefings to various audiences, including both well-informed lay audiences and technical audiences shall likely be required. The Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s).

Communication and Dissemination Overview: Communicating and disseminating findings from evaluations or other evidence-building activities is a significant and critical task. This task area shall include the development of approaches to communicate and disseminate results/findings from HRSA evaluation studies, evidence-building activities, policy analyses, and studies. For example, some of these approaches may include established Dissemination and Implementation based approaches as well as a variety of strategies designed for maximizing HRSA’s impact of its programs and policies for the populations it serves. The Contractor shall develop approaches that effectively promote, market, and disseminate, as needed, results/findings from HRSA evaluations (e.g., previously conducted studies and/or evaluation-related work). The Contractor shall coordinate with HRSA’s Office of Communication (e.g., compliance with the policies and practices of this Office that involve publishing data and evaluation results to a HRSA.GOV website or internal HRSA-only website, for example 508 compliance). Examples include use of the HRSA style guide (including HRSA branding standards on all materials), use of HRSA-approved PowerPoint templates, etc.

The Contractor shall account for the organizational capacity, available resources (e.g., staffing capacity), stage of program planning or implementation, Federal program requirements, and objectives of the relevant HRSA program(s). Under this task, the contractor shall:

(1) Develop mechanisms (e.g., reports, graphical representations, dashboards) to disseminate results/findings, models of care, and evidence-based best practices among HRSA programs, recipients, and stakeholders. The development of these tools and resources shall include data visualizations and web-based tools, as appropriate and shall be disseminated on HRSA internal or external websites.

(2) Develop and disseminate resources that promote translation of evaluation findings into improved practice for service providers.

(3) Develop, promote, and market strategies that leverage existing HRSA communications mechanisms (e.g., radio media tour, social media accounts).

(4) Develop plans for optimal implementation and replication of successful models of care among HRSA programs and recipients.

(5) Develop reports, publications, and presentations that are 508 compliant.

(6) Develop communication plans, including identification of stakeholder engagement roles and points of contact.

(7) Prepare manuscript(s) for publication.

(8) Prepare Draft and Final PowerPoint presentation(s) or abstract(s) for meeting/conference presentation(s).

(9) Develop policy and research briefs.

Data Analytics to support Evaluations and Evidence-building Overview: This task will include strategies for collecting, cleaning, validating, analyzing, and visualizing data to support evidence-building, evaluation activities, and internal HRSA decision making. This task area shall also include the development of strategies, documentation, and mechanisms to support a continuous quality improvement loop based on the findings from evaluations and evidence building activities conducted in Task Areas 1 and 2. The Contractor’s activities in this task area shall align and be compatible with policies and procedures associated with patient protection (e.g., alignment with OMB and HRSA Human Subject Protection Requirements). Under this task area, the contractor shall:

· Convert raw data files into summary/analytic files.

· Compile, link, aggregate, and consolidate data from multiple data sources into a summary/analytic file.

· Work with data and data use agreements at the national, state, and local levels, such as: 1) exploration, development, and implementation of methodologies for linking national data sets; and 2) linking national data sets with state data and/or across states such as analysis of linked Medicare, Medicaid, and public health data sets (e.g., Medicaid claims data and vital records).

· Conducting statistical analysis and developing statistical models using SAS, STATA, R, SPSS, Excel, Python, Tableau, Nvivo (for qualitative analyses) or other appropriate analytical and/or programming/modeling tools.

· Developing data documentation such as metadata and data dictionaries.

· Develop instruction manuals and training materials to support data collection for evaluation studies and evaluation-related activities.

· De-identify/anonymize/aggregate data to ensure no PII or sensitive data is released to the public.

· Conduct activities such as data cleaning, checking accuracy and consistency needed to prepare data for analysis, or develop data documentation.

· Develop 508-compliant mock-ups of dashboards, reports, maps, or other data visualizations using HRSA-approved software.

· Prepare results of analysis, or other data-related visualizations, presentations, or dissemination.

· Provide relevant training to HRSA staff on the analysis of data related to evaluation and/or evaluation-related activities.

· Conduct data collection, analysis, and preparation in support of evaluation studies and evaluation-related activities among HRSA programs and recipients.

· Coordinate with HRSA’s Offices of Information Technology and Communication to ensure compliance with the policies and practices of these Offices regarding evaluation studies and evaluation-related activities that involve publishing data to a HRSA.GOV website or internal HRSA-only website.

· Store datasets/files on a secure server that is in compliance with HHS security policies on data collection, transmission, and storage. The Contractor shall follow all HRSA, HHS, and Federal IT policies including firewalls, data encryption, two- factor authentication, privacy, security, etc.

Required Expertise/Experience:

Program Evaluation

1. Expertise and experience in identifying and implementing a variety of methodological approaches to conducting evaluation studies including 1) obtaining background information on the program to be evaluated (e.g., data on program operations) and the issue to be addressed (e.g., through a 5-literature review and/or environmental scan); 2) determining the appropriate evaluation designs (e.g., non-experimental, experimental, quasi-experimental) based on evaluation type, evaluation aims, and other factors; 3) knowledge of approaches for generating, revising, and prioritizing evaluation questions for the development of evaluation plans; 4) designing credible and feasible evaluations that address identified purposes and questions; (e.g., quantitative, qualitative, and mixed methods); and 5) identifying strengths and limitations of evaluation designs and methods.

Expertise and experience in developing logic models to describe complex problems and programs including 1) developing and articulating program goals and objectives in a structure supporting evaluation; 2) identifying assumptions that underlie methodologies and program logic; and 3) expertise in planning, designing, and developing materials and documents related to data and evaluation.

2. Expertise and experience serving as the primary lead in designing and conducting national evaluation studies designed to evaluate health-related and public health programs like those administered by HRSA.

3. Expertise and experience in developing and coordinating with a variety of web-based tools to support evaluation activities including: 1) development of electronic formats (e.g., web interface) and backend database servers for data entry; and 2) coordination with and support of existing web-based data collection systems.

Evidence-Building Activities

1. Expertise and experience serving as the primary lead in designing and conducting large-scale complex surveys (e.g., several hundred to several thousand participants) designed for health-related and public health programs like those administered by HRSA.

2. Expertise and experience in developing and coordinating a variety of web-based tools to support evidence-building activities including 1) development of electronic formats (e.g., web interface) and backend database servers for data entry; and 2) coordination with and support of existing web-based data collection systems.

3. Experience and expertise in 1) conducting policy analysis and research, using a variety of methodologies; 2) assessing policy issues that may affect HRSA programs at the macro level (nationwide) and micro level (community); and 3) identifying near-term and long-term benefits, costs, and risks of each identified policy alternative.

4. Expertise and experience in 1) preparing demographic characteristic/population and Improve data collection, use, and evaluation, to increase evidence-based knowledge that leads to better health outcomes, reduced health disparities, and improved social well-being, equity, and economic resilience; and 2) making evidence-based projections about service outcomes and systems indicators and measures; 3) use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.

Communication and Dissemination of Evidence

1. Expertise and experience in the communication and dissemination of a variety of products designed to effectively communicate findings from evaluations and other evidence-building activities, including briefings, executive summaries, topic reports, final reports, Reports to Congress, analytical reports, detailed methodological appendices, and manuscripts for submission to the peer-reviewed literature.

2. Expertise and experience in developing tools and other resources to disseminate findings from evaluations and other evidence-building activities, including dashboards, interactive or dynamic data visualizations, and other web-based resources.

Data Collection and Management

1. Experience and expertise in use of data and statistical programming languages and computer-based model tools to support evaluation and evidence-building for public health programs and policies.

2. Expertise and experience in developing, assessing, and manipulating data sets (with both large and small sample sizes) including the following: 1) development of sampling methodology; 2) development or revision of data collection instruments, including, but not limited to, survey design; and 3) collection of quantitative and qualitative data, including reduction of non-response.

3. Experience and ensuring data quality, security, and integrity; including the following: 1) conduct ongoing analyses of data using appropriate methodologies (e.g., dealing with item variance estimation for data gathered using complex survey designs; 2) addressing statistical issues associated with small sample sizes and/or patient level data); and 3) collecting, cleaning, validating, analyzing, and visualizing data for the purpose of evaluations and evidence-building activities.

4. Expertise and experience in development of data analysis files (using variety of statistical packages (e.g., SAS) and associated user guides, analysis data file documentation and codebooks).

Knowledge of HRSA’s Programs and Target Populations

1. Subject matter expertise in all the populations that HRSA serves (e.g., safety net communities -uninsured, isolated, or medically underserved) as well as in all major HRSA programmatic areas. Specific focus should be on the populations that are served by the following bureaus and offices: BHW; BPHC; HSB; HAB; MCHB; FORHP, OAT, and PRB.

Organizational Capacity and Management

1. Have established working relationships with a range of organizations (e.g., public and private sector, governmental and nongovernmental organizations, academic institutions) with demonstrable evaluation-related experience conducting program evaluations including formative, process, outcome, impact, and economic evaluations to ensure that studies benefit from the cost savings and other efficiencies incurred through joint projects.

2. Responders should also include a statement about their accounting system and if it has been audited and/or approved by a federal agency. If the responder has an approved accounting system, please provide the certification in which the accounting system was deemed adequate (e.g., name of the audit agency and audit number). Submit as an attachment.

3. Offeror's Capability to meet the requirement and perform large scale evaluations? YES/NO

4. Ability to conduct multiple projects concurrently, complete complex tasks within narrow time spans, assure quality of projects, initiate new orders within two weeks, have sufficient on-board capacity regarding program evaluation, other evidence-building activities, communication and dissemination efforts, and data collection and management tasks.

Capability Statement – How to submit a response

Small business concerns that believe they possess the capabilities necessary to undertake the work as described in all the Task Areas should submit complete documentation of their capabilities to the Contracting Officer.

The capability statements must address the following points:

1. Demonstrate in detail, including examples of related/relevant work, the ability to provide services described under the tasks including addressing in-depth all the required expertise and experience.

2. Business information:

a. UEI

b. Company Name

c. Company Point of Contact (phone number and email address)

d. Company NAICS/PSC code, must be able to validate in System for Award Management (SAM). Additional, information pertaining to NAICS codes can be found at www.sba.gov.

e. Corporate structure (corporation, LLC, sole proprietorship, partnership, limited liability partnership, Professional Corporation, etc.)

f. Responders should also include a statement about their accounting system and if it has been audited and/or approved Federal agency. If the responder has an approved accounting system, please provide the certification in which the accounting system was deemed adequate (e.g. name of the audit agency and audit number). Submit as an attachment.

The capability statement must not exceed 50 single sided pages in length and using a 12-point font size minimum. Interested small business organizations are required to identify their size standards in accordance with the Small Business Administration. The government requests that no proprietary or confidential business data be submitted in a response to this notice. However, responses that indicate the information therein is proprietary will be properly safeguarded for Government use only.

Point of Contact:

Deepa Zaveri, Contract Specialist, HRSA OAMP, Email: dzaveri@hrsa.gov

Health Resources and Services Administration, Office of Acquisitions Management & Policy, 5600 Fishers Lane, Rockville, MD 20857.

image1.wmf