Performance Work Statement DRAFT for RFI - Continuing to Understand TRLECE_2.pdf

Performance Work Statement ACF-GCS-TMP-2023-35 (Sept 2023)

PERFORMANCE WORK STATEMENT

Administration for Children and Families

Office of Planning, Research, and Evaluation

Performance Work Statement for the

Continuing to Understand The Role of Licensing in Early Care and Education

<Insert Date>

Document Revision History

[Example – Post-Award]

Mod.

No.

Description of Changes

Requested

By

Approved

By Date

Table of Contents

1.0 General

1.1 Introduction

1.2 Objective

1.3 Scope of Work

1.4 Background

1.5 Type of Contract Contemplated

1.6 Period of Performance

1.7 Place of Performance

1.8 Applicable Documents

2.0 Specific Requirements/Tasks

3.0 Contractor Quality Assurance

4.0 Delivery or Deliverables

5.0 Required Travel and Other Direct Costs (ODCs)

6.0 Special Instructions

7.0 Government Furnished Property/Equipment/Information

8.0 Glossary of Abbreviations and Acronyms

**Note: as a matter of policy, some agencies require the government to prepare the PWS, the guidance at FAR

37.602(a) stipulates that “a Performance work statement (PWS) may be prepared by the Government or result from a Statement of objectives (SOO) prepared by the Government where the offeror proposes the PWS.”

https://www.acquisition.gov/far/part-37#FAR_37_602 https://www.acquisition.gov/far/part-37#FAR_37_602

Performance Work Statement (PWS) for the

<Insert Acquisition Title>

1.0 General

1.1 Introduction

The mission of the Administration for Children and Families (ACF) within the

Department of Health and Human Services (HHS) is to foster health and well-being by providing federal leadership, partnership and resources for the compassionate and effective delivery of human services. The mission of the Office of Planning, Research, and Evaluation (OPRE) is to study the ACF programs and the children and families they serve through rigorous research and evaluation projects. The purpose of this task order is to build on work previously funded by OPRE to continue exploring how key features of child care and early education (CCEE) licensing systems are related to CCEE quality and outcomes for children, families, and key partners (e.g., CCEE providers, CCEE licensing agencies, and CCEE program administrators and staff).

1.2 Objective

This task order seeks to contribute to our understanding of the role of licensing in supporting the broader CCEE system through the following specific objectives.

Within the context of these objectives, CCEE licensing agencies can include those at the local, State, Territory, or Tribal level:

1. Review and update the current research, policy, and administrative practice literature about licensing in CCEE;

2. Refine a set of conceptual frameworks that articulate the role of licensing in supporting CCEE quality and outcomes for children, families, and key partners in

CCEE licensing systems (e.g., CCEE providers, CCEE licensing agencies, and

CCEE program administrators and staff);

3. Support CCEE licensing agencies to build capacity to research, analyze, and evaluate their licensing data to inform decision making;

4. In collaboration with CCEE licensing agencies, conduct primary research data collection and analyses that examine underexplored topics related to licensing

(e.g., family perspectives of licensing, differential monitoring, license exempt providers);

5. Conduct secondary analyses of data about licensing (e.g. profiles of states, profiles of providers based on experiences and perceptions about licensing);

6. Update and expand visualizations of licensing data through interactive, web-based dashboards or data visualization tools;

7. Archive and preserve data for dissemination and secondary analyses to replicate and build on findings, including preparation of data files, user guides and codebooks, and related documentation; and

8. Disseminate findings of primary and secondary data collection to key audiences

(e.g., State, Territory, and Tribal CCEE licensing agencies, front-line licensing staff, CCEE providers, researchers, ACF staff studying and supporting CCEE programs).

Ultimately, the work under this task order will address key questions on underexplored topics related to licensing and its role in supporting the CCEE system through data collection and analysis and provide resources for licensing agencies and key partners who work with them, researchers, and the Federal Government.

1.3 Scope of Work

The work under this contract will continue to explore how key features of the CCEE licensing system are related to CCEE quality and outcomes for children, families, and key partners (e.g., CCEE providers, CCEE licensing agencies, and CCEE program administrators and staff). This task order seeks to: synthesize the literature and refine conceptual frameworks about CCEE licensing; build research and analytic capacity of licensing agencies to use their licensing data; build new evidence through primary data collection and analysis, as well as secondary analysis of data; and disseminate to the field through data visualization, data archiving, and other products formatted to support and inform efforts by licensing agencies operating across State, Territory, Local, and possibly

Tribal levels, as well as the Federal Government.

The contractor shall furnish the necessary qualified personnel, equipment, materials, supplies, services, facilities, and otherwise do all that is necessary for, or incidental to, the performance of the work set forth below. The contractor is expected to work with the

Contracting Officer’s Representative (COR) to refine the activities and proposed plans once contract is awarded.

1.4 Background

This project seeks to continue strengthening the field’s understanding of the role of the CCEE licensing system in supporting CCEE quality and outcomes for children, families, and key partners involved in CCEE licensing systems (e.g., CCEE providers, licensing agencies, CCEE program administrators and staff). Features of licensing systems include, but are not limited to, establishing licensing standards, practices to monitor provider compliance, pre-service and ongoing health and safety training for CCEE providers, and training and management of licensing staff to perform roles related to monitoring and ensuring compliance.

While CCEE licensing is integral to ensuring the health and safety of children, the scientific literature on licensing is limited. The Administration for Children and Families (ACF) began investing in this area with a contract in 2019 to support The Role of Licensing in Early Care and

Education (TRLECE) project in order to strengthen the field’s understanding of the CCEE licensing system. Much of the work to date has been descriptive in nature and there is still much work to be done to move beyond descriptive studies and understand connections with policy and the mechanisms driving those relationships. This is especially true for Tribal licensing systems, as little research has been conducted in this area with Tribal communities. As a result, it is unclear how information gathered about State and Territory licensing systems applies to Tribal licensing systems. The work conducted under this task order will support OPRE in continuing to understand the key features of licensing systems, how they differ across the country and develop resources and build research, data and evaluation capacity to support licensing agencies.

ACF's Evaluation Policy demonstrates ACF’s commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice. The five principles of ACF’s Evaluation Policy are: rigor, relevance, transparency, independence, and ethics. It is expected that the work described above will be carried out in accordance with the following principles.

Rigor. ACF is committed to using the most rigorous methods that are appropriate to both the evaluation questions and the populations, circumstances, and settings that are the focus of study; and that are feasible within budget and other constraints. Rigor requires ensuring that inferences about cause and effect are well founded (internal validity);

requires clarity about the populations, settings, or circumstances to which results can be generalized (external validity); and requires the use of measures that accurately capture the intended information (measurement reliability and validity). When conclusions about program efficacy or effectiveness are examined, studies should use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences, such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative. ACF will develop and use methods that are appropriate for understanding diverse populations, taking into account historical, contextual, and cultural factors. Where possible, evaluations will design data collections to allow disaggregation of data and analyses of sub-groups to support understanding of equity.

Relevance. It is ACF’s policy to integrate existing evidence and opportunities for further learning into all activities. Where an evidence base is lacking, ACF is committed to building evidence through strong evaluations. To be most relevant, ACF will require contractors to meaningfully engage groups and individuals from the programs and communities invested in the outcomes of studies to improve clarity of presentations, https://www.acf.hhs.gov/opre/project/understanding-role-licensing-early-care-and-education-trlece-2019-2024 https://www.acf.hhs.gov/opre/project/understanding-role-licensing-early-care-and-education-trlece-2019-2024 https://www.acf.hhs.gov/opre/report/acf-evaluation-policy accuracy of interpretations, and effectiveness of dissemination activities; ACF partners such as states, territories, tribes, and local grant recipients; the populations served;

researchers; and other groups and individuals. Further, evaluations designed, and measurement used under this task order should represent the diverse populations served by ACF. Finally, findings must be disseminated in ways that are clear, accessible, and useful to our diverse range of audiences; this includes using plain language, using inclusive language, adhering to principles of clear communication, and developing products accessible to people with disabilities. In order to ensure the findings are accessible and useful to policymakers, service providers, the communities that ACF services, and other groups and individuals.

Transparency, Data Archiving, and External Communication. ACF values transparency in research and evaluation and is committed to making information about planned, ongoing, and completed evaluations easily accessible.

First, any studies conducted under this contract must be registered with an appropriate registry (e.g., clinicaltrials.gov) prior to the initiation of data collection. Study plans, including analysis plans, should be specified in advance.

Second, ACF is interested in making evaluation data and analyses available for secondary use by other interested researchers. A thoughtful data archive plan will establish procedures and parameters, well in advance of any data collections, to support archiving information and data collected under this task order.

Finally, ACF is interested in ensuring that its research and evaluation work is communicated to a range of relevant audiences throughout the course of project activities. Ongoing and continuous activities are considered essential to a strong communication approach. Audiences that may be appropriate include, but are not limited to: policy-makers, intermediaries, researchers, advocates and opinion leaders, program administrators, training and technical assistance providers, practitioners, and the communities that ACF services. ACF will require contractors to meaningfully engage groups and individuals from the programs and communities involved in studies to improve clarity of presentations, accuracy of interpretations, and effectiveness of external communications activities. ACF is particularly interested in the creation of project external communication plans that identify specific audiences and approaches and products to communicate to those audiences. ACF is also interested in communication plans that use new and emerging media, as well as innovative approaches to communication products that are compatible with the ACF website and other communications platforms.

Independence and Active Engagement. ACF’s evaluation policy upholds the idea that independence and objectivity are core principles of evaluation. Agency and program leadership, program staff, service providers, populations and communities studied, and others participate actively in setting evaluation priorities, identifying evaluation questions, and assessing the implications of findings to help ensure the relevance of the work. While active engagement of groups and individuals who are invested in the outcomes of evaluation can support relevance as well as the accuracy of program implementation details, it is important to insulate evaluation functions from undue influence and from both the appearance and the reality of bias. To promote objectivity, ACF protects independence in the design, execution, analysis, and reporting of evaluations, as outlined in the evaluation policy.

ACF aims to be informed by the practice and empirical knowledge available to inform such topics as measurement choices, technical assistance, training and site support, and the contextualizing of findings. Meaningful involvement of groups and individuals in the evaluation may increase the rigor of the activities of the task order itself as well as understanding, acceptance, and utilization of evaluation findings. Groups and individuals who are invested in the outcomes of research or evaluation include agency and federal, state or local program leadership, program staff (both at ACF and at the grantee or other programs being evaluated), service providers, developers of innovations or practices, participants in the programs, and the communities where programs occur. As appropriate, groups and individuals should have the opportunity for input at multiple phases of a project, but this input must be accomplished in a transparent way while safeguarding the objectivity and independence of the study.

Ethics and Data Safety. ACF-sponsored evaluations are conducted in an ethical and equitable manner; safeguard the dignity, rights, safety and privacy of participants; and comply with both the spirit and the letter of relevant requirements such as regulations governing research involving human subjects. ACF will expect contractors to meaningfully engage groups and individuals from the programs and communities involved in studies to ensure programmatic, cultural, linguistic and historical nuances are accurately and respectfully addressed from the initial study design, through execution, analyses and reporting.

ACF is committed to ensuring that data collected as part of this contract are maintained and stored securely. For the purposes of this section:

• Personally Identifiable Information (PII) is defined as information that can be used to distinguish or trace an individual’s identity, such as their name, social security number, biometric records, alone or when combined with other personal or identifying information, which is linked or linkable to a specific individual, such as date and place of birth, mother’s maiden name, and so on.

The Role of Active Engagement for OPRE Projects

OPRE defines active engagement as the intentional involvement of groups and individuals who are invested in the outcomes of the research or evaluation. This should occur in all relevant phases of the project. The purpose of active engagement is to improve the quality, rigor, and relevance of OPRE projects, facilitate the uptake of findings, and increase capacity to promote equity (see “Making Evaluations Rigorous and

Relevant” https://www.acf.hhs.gov/opre/report/making-evaluations-rigorous-and-relevant-role-active-engagement-development-learning). OPRE seeks input from people with practice, empirical, and lived expertise to inform project activities. This includes but is not limited to refinement of conceptual models, developing research questions, measurement choices, evaluation technical assistance, training and site support, the contextualization of findings, and external communication priorities.

To this end, OPRE expects contractors to develop plans and implement strategies to maximize opportunities for active engagement in ACF projects that are appropriate to the goals of each phase of a project’s life cycle. Approaches can range from shared ownership of projects (e.g., co-creation) and community based participatory research to consultation for specific activities. Opportunities to provide input shall be considered for all phases of the project from refining research questions to external communication of findings, as appropriate for the scope and goals of the project.

Groups and individuals to actively engage may include but are not limited to service providers; developers of interventions, innovations, or practices; program participants;

subject matter experts; federal, grantee, or other program staff; federal, state, or local leadership; groups and individuals in programs’ communities; individuals with lived expertise; and those in broader fields of study related to human services research and evaluation.

To promote objectivity, ACF protects independence in the design, conduct and analysis of evaluations, as outlined in the agency evaluation policy. Active engagement input shall be obtained in a transparent way while safeguarding the objectivity and independence of the study, to insulate evaluation functions from undue influence and from both the appearance and the reality of bias.

1.5 Type of Contract Contemplated

This is a non-commercial Cost Plus Fixed Fee (CPFF) for non-severable services.

1.6 Period of Performance

https://www.acf.hhs.gov/opre/report/making-evaluations-rigorous-and-relevant-role-active-engagement-development-learning https://www.acf.hhs.gov/opre/report/making-evaluations-rigorous-and-relevant-role-active-engagement-development-learning

The period of performance of this contract is 60 months from the date of award.

1.7 Place of Performance

The work to be performed under this task order will be performed at the Contractor’s address and may include multiple states and localities.

1.8 Recognized Holidays

New Year’s Day Labor Day

Martin Luther King Jr.’s Birthday Columbus Day

President’s Day Veteran’s Day

Memorial Day Thanksgiving Day

Independence Day Christmas Day

Juneteenth

1.9 Applicable Documents

The following documents are essential for performance of the contract:

• Guidance for OPRE Product Preparation Review and Communication o Federal Plain Language Guidelines o OPRE Clear Communications Checklist

• OPRE Document Review Cover Page Template

• Quality Assurance Surveillance Plan (QASP)

• Privacy Impact Assessment (PIA)

• OCIO Security Requirements

• Wage Determination

• Pricing Sheet

• Problem Notification Report

• Organizational Conflict of Interest

2.0 Specific Requirements/Tasks

Task 1: Project Management and Team Communication

Effective communication between the Contractor and the federal government, coordinated through the Contracting Officer’s Representative (COR), and successful management of the project as a whole are essential to the success of this task order.

Subtask 1.1 Orientation discussion

The need for communication begins at the very start of the contract. Within fifteen (15) working days of the award of the task order, or at a time mutually agreed upon with the COR after the task order has been awarded, the Contractor shall initiate communication with the COR https://www.plainlanguage.gov/guidelines/ and other federal staff to conduct an orientation discussion. The discussion will focus on a range of topics, including, but not necessarily limited to, clarification of ACF’s interests and objectives for the project, coordination with other ACF program offices and research and policy experts who will be an integral part of the activities under this task order; milestones and timelines for all activities and products on an efficient schedule and budget; procedures to ensure quality of products; dissemination activities; and the challenges anticipated and potential solutions. Note that if this orientation discussion will occur later than 15 working days of the award, it would be helpful to schedule an initial informal project call before the orientation discussion. For the orientation discussion, the Contractor shall send an agenda and any relevant materials to the

COR no less than five (5) working days prior to the discussion. Within two (2) weeks of the discussion date, the Contractor shall submit to the COR a memorandum summarizing major decisions reached and follow-up issues identified during this discussion. that includes a summary of the discussion, major issues discussed during the discussion, resolutions agreed upon, and issues remaining to be resolved. The Contractor shall include proposed approaches for resolving outstanding issues. The memorandum will also outline a specific project work plan and schedule that will detail products to be delivered and a timetable for delivery.

Subtask 1.2: Project Work Plan

In order to ensure effective project management between the project director, principal investigator(s), key staff, sub-Contractors, and the COR, the Contractor shall submit, along with the final memorandum in Subtask 1.1, a project work plan and schedule that reflects an understanding of the activities required to meet the goals and objectives of the project and to carry out the tasks (see Section 3). At a minimum, the work plan shall include a detailed discussion by task of the methods to be used and activities to be conducted to ensure the successful completion of each task; a timeline for accomplishing tasks and ensuring timely delivery of all work products; a detailed description of major work products; a discussion of staff responsibilities for specific activities or tasks; and a detailed plan for addressing and resolving concerns or problems that may emerge during the course of the project.

If the Contractor will engage external partners in acquisition or analysis of the data, the work plan should include the methods, activities, and timing for key milestones involving partners.

The work plan should make clear which activities are contingent upon completion of activities or delivery of products/materials by external partners. The Contractor shall send a draft work plan and schedule to the COR within two (2) weeks following the orientation discussion. Within two

(2) weeks of receipt of comments from the COR, the Contractor shall submit a revised work plan and schedule that reflects comments received from the COR (see Section 3). The work plan will be considered final upon written approval from the COR. The Contractor shall not proceed with the implementation or execution of items in the work plan without the affirmative, written consent of the COR. Expenses incurred absent the prior approval of the COR are subject to denial consistent with existing federal rules and regulations.

Subtask 1.3: Continued communication with COR

Ongoing communication is also necessary to ensure successful completion of this task order, and should be both verbal (e.g., regularly scheduled conference calls) and written (e.g., monthly progress reports). The Contractor shall propose a schedule for bi-weekly meetings to update federal staff on progress and issues. Key Contractor staff, sub-Contractor staff, and/or consultants shall participate as needed to address progress and issues for discussion. These updates should also include close communication around making any significant decisions about the focus or direction of key project activities and deliverables (e.g., making a decision about what content to leave out when synthesizing existing evidence and literature). The schedule for meetings may be changed as agreed upon by the COR. The Contractor shall submit a proposed agenda for each call at least three (3) working days in advance of the scheduled date to the

COR for comment. If there are materials that federal staff will need to review prior to the call date, such materials shall be submitted at least five (5) working days in advance of the call date.

The Contractor shall submit a summary of meetings to the COR for review and approval no later than three (3) working days after each regular meeting.

Subtask 1.4: Monthly progress reports

In addition to the ongoing meetings, the Contractor shall submit written monthly progress reports throughout the period of the task order, which includes the following information: activities completed and progress on each task for the report month, including expenditure data by task, monthly and cumulative expenditures, and the balance before the invoice and balance remaining after the invoice. All IT-related expenses shall be highlighted for ease of reporting. The

Contractor reports shall include details related to project performance. Monthly progress reports shall be submitted to the COR at the same time as the invoices related to that period of performance, no later than the 15th day of each month.

If sub-Contractors are employed, the Contractor must detail a process whereby it can obtain information from the subcontractor in order to comply with this requirement. If sub-Contractors are employed, the Contractor should estimate the sub-Contractor’s costs in the monthly detailed spending report. (For any Contractor‐caused performance variances, the Contractor shall implement corrective actions, if and when approved by the COR. For non‐Contractor‐caused performance variances, the Contractor and COR shall jointly determine the corrective actions to be taken.) The Contractor shall report on consultant activities, including work on publications, presentations, or other dissemination projects as part of the monthly progress report.

Task 2: Active Engagement Approach

For the current study, the contractor shall develop an approach, a plan, and implementation strategies that maximize opportunities for active engagement that are appropriate to the goals of each phase of this project’s life cycle.

The Contractor may engage with various experts, key constituents and partners who have valuable knowledge to inform the project and an interest in the project findings. In the current task order, this may include individuals such as researchers, policymakers, policy analysts, subject matter experts, technical experts in research and analysis of data related to CCEE licensing policy and practice, program and policy leadership and staff at the Federal, State, Territory, Tribal or local levels, licensing and monitoring agency leadership and staff, CCEE community leadership, resource and referral agencies, ACF program partners, CCEE providers who are subject to licensing and monitoring regulations; families, parents, and other participants in CCEE programs; developers of innovations or practices; and other professionals and organizations with insight into CCEE quality and licensing policies, regulations, and practices.

Where appropriate, individuals should have the opportunity for input at multiple phases of a project, but this input must be accomplished in a transparent way while safeguarding the objectivity and independence of the study. The topics on which individuals may be engaged may include, but not limited to:

• Providing knowledge, expertise and perspective on topics related to CCEE licensing systems, regulations, and practices, including:

o States’ and Territories’ experiences with licensing policies and regulations, such as monitoring, compliance, changes in supply, and decisions about caseloads o Current or existing research examining CCEE licensing;

o Barriers and supports for providers to participate in quality improvement efforts to meet licensing requirements o Families’ experiences with providers and using consumer education tools o Training and support for licensors and determination of viable size of caseloads;

• Reviewing and providing feedback on study design options, data collection protocols and instruments;

• Responding to information gathered and developed by the Contractor; and

• Participating in discussions and presentations

The engagement plan and the list of individuals to engage (described below) shall be updated throughout the project as necessary, particularly upon initiation of new activities under this contract, in order to document the decisions and agreements made around areas of engagement.

Subtask 2.1 Active Engagement Plan

To accomplish the goals of including meaningful and appropriate active engagement of experts, partners, and key constituents while upholding the principle of independence in ACF sponsored evaluations, there should be explicit conversations about and documentation of active engagement from the start of the project and across tasks. The Contractor shall provide a plan for active engagement in project tasks and deliverables. The Contractor shall develop a plan for active engagement with the COR no later than three (3) months after task order award to delineate project activities where experts, partners, and key constituents will be engaged (see

Section 3).

The plan should begin with a clearly articulated rationale for the Contractor’s approach to active engagement to guide the entire project and achieve its purpose. This rationale shall guide engagement across all tasks. The approach to active engagement shall describe the reason for inclusion of the individuals and groups proposed to be engaged, how and when they will be engaged, and how input will be addressed and incorporated to achieve the purpose of the study.

The plan should follow from this approach and:

• delineate the project tasks and activities where active engagement will occur;

• demonstrate how active engagement in these tasks and activities will meet specific goals of this project;

• identify potential individuals and groups who may be involved in the active engagement and indicate why or to what end they will be engaged, how they will be engaged, on what parts of the project they will be engaged and the timeline for when the project plans to engage them;

• describe the budget considerations associated with the proposed active engagement; and

• delineate how the plans for active engagement will be refined throughout the project.

Additionally, the plan should include explicit discussion of how the independence of research and evaluation functions and findings will be protected from both the appearance and reality of bias throughout the planning for and execution of active engagement. If appropriate, a

Memorandum of Understanding with some individuals and groups may be warranted, particularly in high-stakes evaluations or sensitive relationships. The Contractor’s plan shall be updated throughout the project as necessary to document the decisions and agreements made around areas of active engagement.

Subtask 2.2: Section of Individuals to Engage

The exact number of experts, partners, and key constituents to be involved will be determined after award, likely ranging from 8 to 12. In order to select individuals for this task, the Contractor shall prepare a list of 15 to 20 proposed experts, partners, and key constituents, including in this list of individuals identified by the COR and other key staff in OPRE, and discuss with the COR the combination of expertise necessary to develop and address the needs of this project. The

Contractor shall provide a list of potential experts, partners, and key constituents and topics for discussions no later than one (1) week after the orientation discussion (Subtask 1.1). The list will include a clear articulation of each individual’s area of expertise, skills and knowledge and what he/she will contribute to the work. The Contractor, with input from the COR and other key staff in OPRE, shall select approximately 8-12 individuals to constitute a panel of stakeholders no later than two (2) weeks following receipt of comments from the COR. The Contractor shall seek agreement from these individuals for participation as an active engagement consultant

(a draft of the correspondence shall be reviewed and approved by the COR prior to sending to potential individuals). The Contractor shall submit a final list of recommended individuals to the

COR, which will include contact information and confirmation of the individuals’ availability and interest in providing consultation. The Contractor’s list of individuals to engage shall be updated throughout the project as necessary.

Subtask 2.3 Implementation of Active Engagement Activities

The Contractor shall, in consultation with the COR and other federal staff members, carry out all activities in the COR-approved Active Engagement Plan (Subtask 2.1). The mechanisms for engaging individuals may include in‐person consultations, tele- or web conferences, and electronic and phone communications, as necessary. The Contractor shall consult with and receive approval from the COR prior to scheduling any consultations. Topics for all discussions shall be developed by the Contractor in consultation with the COR.

The Contractor shall submit to the COR all draft materials (e.g., agenda, participant list, slides) for each discussion three (3) weeks in advance of the date of the discussion (see Section 3). The

Contractor shall incorporate feedback from the COR and send the final materials to the COR and the participants no later than one (1) week prior to the date of the discussion (see Section 3).

Following the discussions/consultations, the Contractor shall submit detailed draft summaries of the consultations within two (2) weeks of such discussions. Within two (2) weeks of receipt of comments from the COR, the Contractor shall incorporate feedback and submit final summaries for approval by the COR (see Section 3).

Task 3: Synthesis of Research, Policies, and Administrative Practices, and Refinement of

Conceptual Framework(s)

The Contractor shall review and synthesize research, gray literature, policies, administrative practices, and published conceptual frameworks since 2020 in order to summarize both the current state of knowledge and our gaps in understanding the role of CCEE licensing standards and practices in supporting CCEE quality, and outcomes for children, families, and key partners in CCEE licensing systems (e.g., CCEE providers, licensing agencies, and State, Territory, and/or Tribal CCEE program administrators). Under the Understanding the Role of Licensing in

Early Care and Education project awarded by ACF in 2019, there is a forthcoming literature synthesis that covers publications through 2020. The purpose of the synthesis under this task order is to build on and update, rather than duplicate, existing information. The synthesis will serve multiple purposes, including but not limited to: providing an up to date overview of the existing knowledge and important gaps on these topics; informing and updating relevant conceptual framework(s); and informing the selection of specific research questions to be addressed and analyses to be conducted under this task order.

The Contractor shall not limit the type of information in the review to research material and shall suggest innovative ways to collect and synthesize relevant material to inform the rest of the project. For example, the Contractor may include research studies, Technical Assistance (TA) materials, grey literature, policy reports, descriptions of administrative practices, and other resources. The final topics chosen for review and synthesis will be selected through active engagement activities with experts, key constituents and partners in the CCEE licensing system and/or through direct requests from the COR. The synthesis and review shall define key terms related to licensing in CCEE if necessary. The synthesis will also include a description of how the Contractor conducted the review and synthesis, including but not limited to: what search criteria were used, how and/or where the relevant resources were found, what decisions were made by the Contractor and COR to narrow down to a specific number of resources to include in the final review, and if certain types or specific informational resources were prioritized and why. The results of the synthesis will be summarized in a public-facing report or series of briefs that are accessible to key audiences such as policy makers and licensing agency staff.

Subtask 3.1: Plan for Synthesis and Update of Conceptual Framework(s)

The Contractor shall meet with the COR and any experts, key constituents and partners in the

CCEE licensing system selected for this task to gain a broader understanding of the research and/or policy issue(s) in need of study, the policies and/or research to be reviewed and summarized, as well as the objectives of the task(s), task requirements, and parameters.

The Contractor shall develop a detailed plan for pursuing a targeted review and synthesis of relevant research literature, grey literature, empirical findings, descriptive reports, published conceptual frameworks, licensing policies, and State, Territory, and possibly Tribal, administrative practices, and conceptual frameworks related to CCEE licensing published since

2020. The plan shall also include steps to update conceptual framework(s). Within six (6) weeks of the award of the task order, the Contractor shall submit to the COR a draft plan describing the proposed approach to the synthesis and updating of conceptual frameworks (see Section 3). The

Contractor shall submit revisions for a final plan based on feedback from the COR within two

(2) weeks of receiving comments (see Section 3).

Subtask 3.2: Synthesis and Update of Conceptual Framework(s)

The Contractor shall conduct and complete the targeted review and synthesis of research, policies, and administrative practices, as well as update conceptual frameworks following the approved plan under Subtask 3.1.

Within four (4) months of the approval of the synthesis plan under Subtask 3.1, the Contractor shall submit to the COR draft annotated outline(s) and annotated bibliography matrix synthesizing the research, policies, administrative practices and updated conceptual framework(s) related to CCEE licensing (see Section 3). The Contractor shall submit revisions to the annotated outline(s) and annotated bibliography matrix based on feedback from the COR within two (2) weeks of receiving comments (see Section 3).

Within two (2) months of approval of the annotated outline(s) and annotated bibliography matrix, the Contractor shall submit to the COR a first draft of the synthesis product(s) (see

Section 3). The Contractor shall submit revisions to the synthesis product(s) based on feedback from the COR within three (3) weeks of receiving comments (see Section 3).

Subtask 3.3: Final Memo Summarizing Research Gaps and Areas for Future Research

The Contractor, informed by the synthesis and update of conceptual frameworks completed in

Subtask 3.2 and in consultation with and with prior approval from the COR, shall submit a memoranda to the COR. The memo shall (1) identify and define the issue(s) in need of research and analysis to inform policies related to ECE licensing standards and practices; (2) assess, through limited review, the potential relevance and value of addressing each issue in order to inform policies related to CCEE licensing standards and practices; (3) explore the types and sources of existing information and data available for review, assessment, and analysis that could be used to address the key issues; (4) identify the types of primary data that could be collected to address the key issues; and (5) highlight a select number of issues/research questions, out of the total list of identified issues, that ACF/OPRE may want to prioritize and describe why.

Within one (1) month following submission of the final draft of the synthesis product(s)

(Subtask 3.2), the Contractor shall submit to the COR an annotated outline of the memo (see

Section 3). The Contractor shall submit revisions to the outline based on feedback from the COR within two (2) weeks of receiving comments (see Section 3).

Within one (1) month following approval of the annotated outline of the memo, the Contractor shall submit to the COR a draft memo (see Section F.3). The Contractor shall submit revisions to the memo based on feedback from the COR within two (2) weeks of receiving comments (see

Task 4: Capacity Building of Licensing Agencies

Critical to the TRLECE project is the need to support licensing agencies to build their capacity to evaluate, research, and analyze their administrative licensing data. Previous research indicates that data quality, data use, and data management vary widely among licensing agencies and that additional support is needed to build the research and evaluation capacity of licensing agencies.

Through supporting licensing agencies to build their data evaluation, research, and analysis capacity, the project can help licensing agencies to implement improvements to collection, management, and maintenance of administrative licensing data, effectively use licensing administrative data to inform CCEE and licensing decision making and programmatic changes, and to streamline licensing policies and procedures.

In supporting licensing agencies to build their evaluation, research, and data analysis capacity, the Contractor may collaborate with licensing agencies on activities that focus on topics such as:

• Evaluation of currently available administrative licensing data:

o What data is currently available (and what data might be missing)?

o What is the quality of the data?

o How often is this data collected and entered into the data management system?

o How current is the data?

o Are there potential areas for improvement?

• Assessment of the administrative licensing data management system:

o What data management system is used by licensing agencies?

o How long has the data management system been in use and when was it last upgraded?

o What are the procedures for data collection and data entry (who is responsible for collecting data, entering data, how often is data entered, etc.)?

o Who has access to the data management system?

o What is the user experience of the data management system?

o How is the data management system maintained (e.g., through an outside contractor, by staff within the licensing agency)?

o How does the data management system link to other relevant agencies and/or administrative data systems (e.g., child care subsidy data, Head Start, QRIS, K-12 education data)?

o Are there potential areas for improvement?

• Understanding how administrative licensing data can be used for data analysis:

o How do licensing agencies use their data to inform decision making and programmatic changes and/or streamline licensing policies and procedures?

o What research questions are important to licensing agencies?

o Do adjustments need to be made to current data elements or do any new data elements need to be collected to explore research interests?

o Are there potential areas for improvement?

Subtask 4.1: Capacity Building Work Plan

Activities designed to build the capacity of licensing agencies to explore these topics may be conducted in a number of ways and may include, but are not limited to, written products (e.g., short briefs, resource guides), webinars, and expert convenings.

The Contractor shall submit a Capacity Building Work Plan to the COR that outlines a plan for conducting capacity building activities, including the nature of the activities, topical foci, and timing of activities. The Capacity Building Work Plan shall be submitted on a timeline determined in consultation with the COR. A revised work plan, incorporating comments received from the COR, shall be submitted within ten (10) working days of receipt of COR comments.

Subtask 4.2: Capacity Building Activities

If directed by the COR, the Contractor shall conduct activities outlined in the Capacity Building

Work Plan. The nature and frequency of these activities will be determined in communication with the COR based on the approved Capacity Building Work Plan. The Contractor shall submit draft and final deliverables specific to the activities in the Capacity Building Work Plan to the

COR for review and approval on a schedule proposed by the Contractor and approved by the

COR.

Task 5: Primary Data Collection

The Contractor may develop multiple, separate primary data collection studies informed by the synthesis of prior literature (Task 3), ongoing capacity building work with licensing agencies

(Task 4) and in consultation with and with approval from the COR.

Subtask 5.1: Work Plan for Primary Data Collection

In order to ensure effective project management between the project director, principal investigator(s), key staff and sub-Contractors, and the COR, the Contractor shall submit a primary data collection work plan and schedule that reflects an understanding of the activities required to meet the goals and objectives of the primary data collection and to carry out the tasks

(see Section 3). At a minimum, the primary data collection work plan shall include a detailed discussion by task of the methods to be used and activities to be conducted to ensure the successful completion of each task; a timeline for accomplishing tasks and ensuring timely delivery of all work products; a detailed description of major work products; a discussion of staff responsibilities for specific activities or tasks; and a detailed plan for addressing and resolving concerns or problems that may emerge during the course of the project.

The primary data collection work plan shall include, but shall not be limited to:

a) research questions to be answered;

b) research design that will be used to answer each of the research questions;

c) data elements to be collected;

d) proposed data collection methods (e.g., site visits, observations, interviews, focus groups, surveys, etc.);

e) proposed data collection instruments, and whether these instruments already exist or if they would need to be developed;

f) proposed approach for selecting, recruiting, and training data collectors;

g) proposed timeline required for data collection;

h) a data analysis plan (details provided below);

i) a proposed schedule for reporting progress, completing analyses, producing and delivering data tables, reports, briefs and other materials resulting from data collection;

and

j) a proposed timeline and plan for archiving all data (see Task 7).

The data analysis plan for primary data collection should include a discussion of the rigor of the proposed approaches and, at a minimum, address the following elements:

a) for each research question identified, a narrative discussion of the analysis techniques to be utilized in answering each research question;

b) relative advantages or disadvantages of approaches or methods;

c) any challenges anticipated in the analysis of each research question; and

d) draft table shells.

For the sake of efficiency, primary data collection work plans described above should align as much as possible with the format and content required for the submission of a full OMB package.

The Contractor shall submit a draft primary data collection work plan and schedule to the COR on a timeline determined in consultation with the COR (see Section 3). Within two (2) weeks of receipt of comments, the Contractor shall submit a revised work plan and schedule that reflects comments received from the COR (see Section 3). The work plan and schedule will be considered final upon written approval from the COR. The Contractor shall not proceed with the implementation or execution of items in the primary data collection work plan without the affirmative, written consent of the COR.

DEPARTMENT OF HEALTH & HUMAN SERVICES

Administration for Children Families

GOVERNMENT CONTRACTING SERVICES

Subtask 5.2: Study Design

The Contractor shall submit a fully specified study design and methodologies, which shall update and expand the work plan outlined in Subtask 5.1. The Contractor shall submit a draft study design in the form of multiple memos to the COR. For the sake of efficiency, the memos should align as much as possible with the format and content required for an OMB package. The study design shall include, but is not limited to:

a) research questions to be answered;

b) research design that will be used to answer these questions;

c) data elements to be collected;

d) proposed data collection methods (e.g., site visits, observations, telephone interviews, focus groups, web-based surveys, etc.);

e) proposed sampling approach and recruitment strategy of respondents and key informants;

f) proposed data collection measures and instruments, including existing measures/instruments, as well as an approach to measurement development for constructs for which existing measures are not available;

g) proposed plan to pilot instrument and study protocols, if deemed necessary;

h) proposed schedule and logistics for data collection;

i) proposed management plan for data collection (e.g., includes a process for managing and overseeing data collection that includes quality assurance practices and will identify and address, as quickly as possible, any problems that could arise);

j) proposed approach for selecting, recruiting, and training data collectors;

k) a data management plan (e.g., data storage security; process for cleaning of data);

l) a preliminary data analysis plan;

m) a data archiving, preservation and disposition plan; and

n) proposed schedule for reporting progress, completing analyses, production and delivery of data tables, reports, briefs and other materials resulting from data collection.

The…