Residency_PD_PWS.5.2.19.pdf

Attachment A – Draft Performance Work Statement 5.2.19

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

Performance Work Statement

FEASIBILITY AND CONDUCT OF AN IMPACT EVALUATION OF TEACHER

RESIDENCY PROGRAMS OR PROFESSIONAL DEVELOPMENT FOR CLASSROOM

TEACHERS

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (“NCEE”) of the U.S. Department of Education’s (the “Department”) Institute of Education Sciences (IES) requires a contract to assess the feasibility of conducting an impact evaluation of teacher residency programs and study design work to identify professional development for classroom teachers that complements existing policy and research.

The purpose of the feasibility work is to determine if sufficient numbers of teachers who entered teaching through a residency program exist to provide study sample for an impact study. The purpose of the design work related to professional development is to identify professional development or policies that could be used within an impact study to provide information about the effectiveness of promising professional development strategies. Based on the feasibility and design work conducted, the Department will decide whether to move forward with an impact study estimating the effectiveness of: (1) teachers from residency programs (Study A), or (2) a professional development strategy (Study B). Only one of the two impact studies will be conducted within this contract.

This contract includes a 25-month Base Period, which will include conducting feasibility and design work, study site recruitment, and initial data collection. It also includes four option periods if Study A is conducted, resulting in a 70 month contract, and three option periods if Study B is conducted, resulting in a 60 month contract.

For Study A, if selected:

Option Period I includes a second year of sample recruitment and data collection in the Year 1 recruited sample. Option Period I includes a 26-month period of performance and overlaps with the Base Period by 15 months.

Option Period II includes data collection in the Year 2 recruited sample. Option Period II includes a 21-month period of performance and overlaps with the Option Period I by 13 months.

Option Period III includes data analyses, a technical working group (TWG) meeting, and report writing.

Option Period III includes a 20-month period of performance and overlaps with the Option Period II by 10 months.

Option Period IV includes cost effectiveness analyses, finalizing the report writing, and responding to peer review feedback from IES’s Standards and Review Office (SRO). Option Period IV includes a 24-month period of performance and overlaps with the Option Period III by 8 months.

Base Option Period I-Study A Option Period II-Study A Option Period III- Study A

Option Period IV- Study A

Oct. 2019 – Oct. 2021

Aug. 2020 – Sept. 2022 Sept. 2021 – May 2023 Aug. 2022 – March

Aug. 2023 – July

25 months 26 months 21 months 20 months 24 months Overlaps with base by 15 months Overlaps with Period I by 13 months

Overlaps with Period II by 10 months

Overlaps with Period II by 8 months

Months 1-25 Months 11-36 Months 24-44 Months 35-54 Months 47-70

For Study B, if selected:

Option Period I includes a provider competition, initial implementation of the selected PD strategy, and initial data collection. Option Period I includes a 17-month period of performance and overlaps with the Base Period by 17 months.

Option Period II includes additional implementation of the PD strategy and data collection. Option Period II includes a 24-month period of performance and overlaps with the Option Period I by 3 months.

Option Period III includes data analyses, a technical working group (TWG) meeting, and report writing.

Option Period III includes a 27-month period of performance and overlaps with the Option Period II by 13 months.

Base Option Period I-Study B Option Period II-Study B Option Period III-Study B

Oct. 2019 – Oct.

June 2020 – Oct. 2021 Aug. 2021 – July 2023 July 2022 – Sept. 2024

25 months 17 months 24 months 27 months Overlaps with base by 17 months Overlaps with Period I by 3 months

Overlaps with Period II by 13 months

Months 1-25 Months 9-25 Months 23-46 Months 34-60

A. Legislative Authority

This evaluation is authorized by Title II, Part A of the Elementary and Secondary Education Act (ESEA), section 2121-2123 as amended by the Every Student Succeeds Act (ESSA, P.L. 114-95).

B. Background

Improving the effectiveness of teachers is a central goal for ESSA Title II-A funds. With reauthorization of ESEA, teacher residencies are a new allowable activity within Title II-A to reform or improve teacher preparation. High-poverty schools disproportionately have novice teachers who are, on average, less effective than their more experienced teachers. Given that various studies indicate that teacher performance improves with experience, one potential approach to improve preparation is to provide more classroom experiences to teacher candidates before they become the teacher of record. Teacher residencies, which include a meaningful clinical experience as a central program component, provide such opportunities to their teacher candidates. Despite the promise of this teacher preparation model and its growing use, there is no rigorous evidence of its effectiveness.

Teacher professional development (PD) continues to be the most commonly funded strategy under Title II- A for improving teacher effectiveness. There is an extensive literature in the area of teacher PD, including impact studies conducted by NCEE and NCER/NCSER grantees. Despite the number of studies, there is limited evidence of strategies that improve teachers’ effectiveness. Changing teacher knowledge and practice through professional development is most common. However, changing practices that lead to improvements in student achievement is more elusive. The most compelling evidence on affecting student achievement has to do with improving teachers’ classroom management. Evidence about the effectiveness of content related to instructional strategies has met with more limited success, particularly in improving student achievement within the context of large-scale studies. One hypothesis is that the focus of these PD efforts does not change knowledge/practice extensively enough to affect student achievement. Another hypothesis is that the practices focused on by the PD are not those that most improve student achievement (see https://ies.ed.gov/ncee/pubs/20174010/), particularly related to subject area content.

This study will (1) assess the feasibility of conducting an impact study of teacher residencies or (2) design an impact study of professional development. Based on the information provided in the course of this contract, the Department will make a decision to conduct one of these studies in the context of this contract.

C. Feasibility and design of an Impact Study of Teacher Residencies (for Study A)

The purpose of this work is to assess the feasibility of cost-effectively obtaining teacher sample for an effectiveness study of teachers who enter teaching through residency programs. Specifically,

1. Are there enough residency route programs with graduates for a study? Are the programs sufficiently similar in approach to warrant combining the impact of teachers from multiple programs within one study? Can programs provide information about where past program graduates are teaching?

2. Is there a large enough number of study relevant graduates that are located in a manageable number of districts to make a study of teachers from residency programs feasible to conduct? Do they teach in tested grades/classrooms. More generally, how cost effectively might a study obtain sample?

D. Identify Teacher Professional Development (for Study B)

The purpose of this work is to identify teacher professional development that would inform policy on evidence-based PD. Existing studies are already providing or studying information about the potential importance of improved classroom management teaching practices. Potential areas for future studies include learning about the effectiveness on improving student achievement of particular pedagogical practices, particularly in STEM areas. Such studies should be designed to build off of and complement findings from existing studies (completed and ongoing) to study likely high-leverage pedagogical practices for the PD focus. Another area of interest is learning about more effective ways to deliver PD. Existing PD studies have typically changed classroom practice, to some extent, but perhaps not enough to result in improvements in student achievement. There is also some evidence for the effectiveness of more individualized support. This line of work relies on the hypothesis that PD needs to make more substantive changes in teachers’ classroom practice.

The following are a few PD focus areas of interest:

• Providing support for teachers to collaborate with their peers to collectively improve their classroom practice and performance.

• Training focused on effective use of curricula. For example, a random assignment study of elementary math curricula (Agodini et al. 2010) found that teacher practices were influenced by curricula and that some curricula were more effective than others at improving student achievement, which suggests training focused on particular curricula could be promising).

• Providing individual teachers funding so that they have the flexibility to acquire PD that best meets their needs. Of particular interest is the feasibility of teachers purchasing PD that fits their professional needs (i.e., what is available for purchase? What is the nature of such support?).

The contractor shall address the following in the course of identifying PD for study and designing this work.

1. What are the most promising and policy relevant aspects of PD that are feasible to study? Are there important variations of the feature or differing features that are policy relevant to study and thus should be varied within the study design?

https://ies.ed.gov/ncee/pubs/20174010/

2. Who are the providers? What is their capacity to deliver PD? Is their PD in wide use? Are they interested in potentially participating in an impact evaluation?

3. In assessing the feasibility of teacher PD vouchers, what are the PD opportunities that teachers could purchase with a voucher? What is the nature of these offerings? Do they appear to be promising/consistent with what are evidence-based practices? How much do these offerings cost?

E. Key Study Research Questions

The contractor shall provide options for feasible and credible study designs for impact studies outlined in Sections C and D. NCEE intends to conduct one of these evaluations within this contract. The other may be conducted in a subsequent contract.

The study shall address the following research questions, depending on the study selected to be conducted.

For Study A:

1. For the residency programs participating in the evaluation, what preparation experiences do they provide to their teacher candidates?

2. For teachers in the impact study, what were their preparation experiences?

3. How effective are teachers from residency programs compared with other teachers in the same grade/subject in the same schools who are not trained through a residency model? Does this vary by teachers’ experience level?

For Study B:

1. Was the professional development strategy implemented with fidelity? What were the features of the implemented PD?

2. What was the impact on teachers’ classroom practice and on their students’ academic achievement?

F. Evaluation Approach

The offeror shall consider the following assumptions:

• Study A: students with a grade/subject or classrooms shall be randomly assigned to teachers; Study

B: teachers shall be randomly assigned to treatment or control.

• The Study A sample shall include 100 treatment teachers who received their certification through a residency program and approximately 250 control teachers who received their certification through a non-residency program (assumes 3-4 teachers total per grade with one of them being a residency program trained teacher); it is expected that this study sample will be recruited over two years

• The Study B sample shall include 100 teachers to receive the professional development.

• Data collection shall include: a residency program interview (in year 3 of the contract, Study A only);

observations of both treatment and control teacher classrooms - 2 observations at the beginning and 2 observations at the end of each data collection year for a total of 4 observations per teacher (Study A and B); administrative records to include student achievement tests (Study A and B); intervention implementation participation information (Study B only); and a teacher survey (Study A and B).

• Study B: The contractor shall provide training or resources to implement the strategy/practice. The strategy/practice shall be implemented during the 2021-22 school year.

II. Scope of Work

This section specifies the tasks and subtasks that the contractor shall perform.

Task 1—Opening Meeting with the Department

The contractor’s project director and up to three project staff shall meet with the Contracting Officer’s Representative (COR) and other appropriate Department staff within 15 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables: Opening Meeting summary

Task 2—Identify Residency Programs and Assess the Feasibility of their Participation in an Impact Study

2.1 Develop Feasibility Protocol. The contractor shall develop a protocol to guide discussions with teacher residency programs. The contactor shall include questions to inform both study feasibility and design issues.

2.2 Conduct Program interviews. The contractor shall conduct interviews for up to 9 programs to accomplish the goals outlined in subtask 2.1.

Deliverables: Residency program protocol

Task 3—Identify Professional Development Strategies for Study Focus

3.1 Develop Protocol. The contractor shall develop a protocol to guide discussions with professional development providers. The protocol must be relevant or adaptable for discussions with a variety of providers (e.g., academicians and organizations that provide professional development). The contactor shall include questions to inform what are likely promising professional development strategies and providers that can be implemented feasibly within an evaluation.

3.2 Conduct Provider Calls and Other Information Search. The contractor shall conduct interviews for up to 9 providers to accomplish the goals outlined in subtask 3.1. In addition, the contractor shall gauge the interest of providers in participating in an impact evaluation. The contractor shall also inquire about whether providers can provide PD to individual teachers. In addition, the contractor shall collect information about other ways that a teacher might purchase individualized support (e.g., through participation in internet offerings, postsecondary coursework, etc.).

Deliverables: PD provider Protocol

Task 4—Study Design Options Memo

Based on evidence from information collected in tasks 2 and 3, the contractor shall prepare a study design memo that summarizes interview findings and makes a recommendation about the feasibility of a residency study and the recommendations for the content and approach of a PD study. The memo shall outline a recommended study design for each study (one to estimate the effectiveness of teachers from residency programs and one to estimate the impact of PD). For the PD study, the memo shall provide a detailed description of potential PD treatment(s) and justification of the study’s value in informing policy. The memo also shall include information about the provider capacity and interest in participating in an impact study. The study design(s) shall incorporate information obtained in Sections C and D and the issues raised in Section E. The contractor shall provide a draft of the study design memo to the COR four weeks prior to the TWG meeting. The study design memo shall be used as materials for informing the TWG of the study design prior to the TWG meeting (Task 5).

Deliverables: Study Design Option Memo

Task 5—Technical Working Group for the Evaluation

5.1 Establish Technical Working Group. The role of the TWG will be to advise the contractor and the Department on the conduct of the study including, but not limited to: study design, instrumentation, data collection, and data analysis. The TWG shall be comprised of approximately 12 individuals, and must include experts in: teacher certification and preparation, teacher professional development, teacher quality, teacher policy, teacher preparation providers, and evaluation methodology. The contractor shall submit a list of proposed TWG members no later than six weeks after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members (no later than 10 weeks after contract award), the contractor shall contact each proposed member to secure interest and intention to serve on the TWG.

5.2 Convene Technical Working Group. The contractor shall convene a meeting of the TWG no later than March 2020. The TWG meeting shall focus on proposed recruitment feasibility, specifics of the preparation experience and professional development to include (and potentially vary), study design options, data collection, and analysis issues.

The contractor shall submit a written summary of the meeting to the COR within 1 week after the meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in the Department’s facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

Deliverables: List of proposed TWG members Summary of TWG meeting

Task 6—Prepare OMB package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the impact study.

The package shall include a description of the study design and data collection instruments as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions.

The contractor shall design data collection instruments to address the key impact study questions. The contractor shall submit a draft forms clearance package to the COR no later than December 2020. The COR will submit the accepted forms clearance package for clearance. Prior to the final 60 days in the OMB submission process, the contractor shall pilot test the instruments with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contactor’s procedures and testing systems for administering the instruments are effective.

Based on comments from internal and external clearance (after the first 60 day clearance process) the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB for final clearance(60 days). The contractor also shall ensure that the instruments are revised as needed based on pilot test results.

Data collection for the impact study shall consist of:

• Teacher Survey: The contractor shall administer a teacher survey.

• Administrative records: The contractor shall collect information available in school records relevant to the outcomes of the strategy studied (student achievement, student characteristics).

• NOTE: additional instruments are noted in option I, dependent on which study (A or B) is selected.

The contractor shall coordinate its own IRB approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Original submission OMB clearance package (first 60 days)

Revised OMB clearance package (second 60 days) Documentation of IRB approvals

Task 7—Site recruitment

Using sample selection criteria developed in consultation with the COR, the contractor shall recruit teachers, schools, and districts to participate in the impact evaluation.

The contractor shall complete recruitment activities for study B or complete year 1 recruitment activities for study A no later than spring 2021.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of understanding from districts.

Task 8—2021-22 Data Collection

Using instruments approved by OMB, the contractor shall begin school year 2021-22 data collection to estimate the impact on the outcomes of interest and to answer the impact evaluation questions. The contractor shall also begin collection of teacher classroom practice observations of all teachers participating in the impact study. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data, such as out-of-range variables, missing data, and unusual distributions of scores, so that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The data monitoring activities shall be in place within 18 months after contract award. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. The contractor shall maintain these records and provide information on these procedures to the Government upon request. When possible, individuals collecting and/or coding data shall have no knowledge of each school’s treatment condition.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any PII that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 U.S.C. 552a; the E-Government Act of 2002, 44 U.S.C. §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-12. The Contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

The contractor shall provide monthly information about data collection activities as part of the monthly report in Task 9.

Task 9—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

OPTION PERIOD I-Study A (Effectiveness of Teachers from Residency Programs)

Task 10A—Added OMB forms for Study A

To meet the data collection needs of study A, the contractor shall include the following additional instruments in the OMB deliverables in task 6.

• Structured interviews of Residency program administrators (for information on the nature of recruiting, selection, placement of program participants as well as support strategies during the residency period).

Task 11A— 2021-22 Data Collection for Study A (con’t)

Using instruments approved by OMB, the contractor shall finalize school year 2021-22 data collection to estimate the impact on the outcomes of interest and to answer the evaluation questions. The contractor also shall finalize observations of teachers’ classroom practice. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 8 above and provide monthly information about data collection activities as part of the monthly report in Task 15A.

Task 12A— Random Assignment for Study A (2021-22 school year)

The contractor shall conduct random assignment in order to create equivalent classrooms across teachers (within the same grade/subject and school) participating in the study during the 2021-22 school year.

Task 13A—Additional Site Recruitment Using sample selection criteria developed in consultation with the COR, the contractor shall recruit teachers, schools, and districts to participate in the impact evaluation.

The contractor shall complete year 2 recruitment activities no later than spring 2022.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of understanding from districts.

Task 14A— Random Assignment for Study A (2022-23 school year)

The contractor shall conduct random assignment in order to create equivalent classrooms across teachers (within the same grade/subject and school) participating in the study during the 2022-23 school year.

Task 15A—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, OPTION PERIOD II-Study A (Effectiveness of Teachers from Residency Programs)

Task 16A— 2022-23 Data Collection for Study A

Using instruments approved by OMB, the contractor shall begin school year 2022-23 data collection to estimate the impact on the outcomes of interest and to answer the evaluation questions. The contractor shall also begin classroom observations of study teachers. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 8 above and provide monthly information about data collection activities as part of the monthly report in Task 17A.

Task 17A—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, OPTION PERIOD III-Study A (Effectiveness of Teachers from Residency Programs)

Task 18A—Data Analyses

The contractor shall conduct data analyses that clearly address the evaluation questions in preparation of a TWG meeting to discuss the study’s impact findings.

Task 19A—Technical Working Group

Convene Technical Working Group. The contractor shall convene a meeting of the TWG members to discuss the study’s impact findings. The meeting shall be held no later than March 2024. After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members, and any lodging required by the TWG members.

Finally, the COR will secure a meeting space at Department facilities.

The contractor shall provide copies of all reviews of products by the panel to the COR within five (5) work days of receipt by the contractor. If TWG members recommend any significant deviation from the data analysis plan, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR.

Deliverables: Summary of TWG meeting

Task 20A—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

OPTION PERIOD IV-Study A (Effectiveness of Teachers from Residency Programs)

Task 21A—Cost Effectiveness and Impact Report

The contractor shall conduct cost effectiveness analyses and prepare a report that clearly addresses the impact evaluation questions.

The contractor shall make sure that the impact report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. Two versions of the report shall be created, including:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A 15-page report. Full technical aspects of the report shall be included in appendices.

The contractor also shall prepare a restricted use data file for the study report. The file shall contain all data collected for the evaluation with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, weights, and how missing data were addressed. In addition, the contractor shall submit documentation accompanying the data to include the unit of observation for each data file (e.g., child, classroom, school, etc.).

For each file, the contractor shall also provide IES with a risk disclosure analysis for submission to National Center for Education Statistics (NCES). Once the codebook and risk disclosure analysis are approved, the contractor shall submit to the COR the restricted use data file consistent with NCES standards1. ASCII format is acceptable. The contractor shall submit the restricted use file for the report to the COR by 69 months after contract award.

Deliverables: Draft Impact report

Revised draft Impact report, SRO-ready Final Impact report Print and web-ready Impact report

Restricted use data file with documentation for study report

Task 22A—Reporting Requirements

1 http://nces.ed.gov/statprog/style.asp

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

OPTION PERIOD I-Study B (Impact of PD Strategy)

Task 10B—Selection of Professional Development Provider

10B.1 Expert Provider Selection Panel. In consultation with the COR, the contractor shall propose an expert panel of three persons to advise on the selection of providers within 9 months after award. These experts may also serve on the technical working group described in Task 5. The contractor shall contact each proposed member and obtain commitment to serve on the provider selection panel.

10B.2 Request for Proposals (RFP). In consultation with the COR, the contractor shall prepare by month 10 a provider request for proposals including the timelines for proposals, requirements for the training, and the criteria by which the proposals shall be evaluated. The RFP shall also specify the maximum amount of funding that will be available to support the training. The contractor shall publicly post the RFP. Prior to release of the RFP, the contractor shall send the advance announcement to relevant providers and post it on the website(s) of relevant associations.

10B.3 Selection of Training Provider(s). The contractor shall have each member of the expert panel review the provider proposals and write a brief review of each proposal, including each provider’s strengths and weaknesses, along with recommendations of up to four training providers for inclusion in the panel meetings. The contractor shall provide the reviews and recommendations of these experts to the COR within 2 weeks after the provider proposal submission date. Expert panel members will participate in a one- or two-day meeting, organized by the contractor, with up to four providers in which each provider makes a short presentation of their professional development activities and responds to questions from expert panel members.

In consultation with the COR and the expert panel, the contractor shall secure the recommended provider to participate in the evaluation.

Deliverables: List of expert panel members to review proposals

Request for Proposals (RFP) Document Expert panel review of proposals

Task 11B— Added Data Collection for Study B (con’t)

Using instruments approved by OMB, the contractor shall finalize school year 2021-22 data collection to estimate the impact on the outcomes of interest and to answer the evaluation questions. The contractor shall also include the following data collection:

• Classroom observations: of all participating teachers in the impact study (treatment and control) – three observations at the beginning and three observations at the end of the school year during the year of data collection.

• Monitoring protocols: to track PD intervention implementation and participation

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 8 above and provide monthly information about data collection activities as part of the monthly report in Task 9.

Task 12B—Professional Development Implementation and Implementation Support

As part of the monthly reporting requirement (task 9), the contractor shall provide ongoing information about progress in implementing the professional development activities and as well as participant attendance information.

PD activities. The contractor, under subcontract with the PD provider, shall provide initial professional development/support to 100 teachers assigned to treatment status during the 2021-2022 school year.

Provider support. The contractor shall work with the provider and study districts to arrange the initial PD and support solid implementation.

OPTION PERIOD II-Study B (Impact of PD Strategy) Task 13B—Professional Development Implementation and Implementation Support (con’t)

As part of the monthly reporting requirement (task 15B), the contractor shall provide ongoing information about progress in implementing the professional development activities and as well as participant attendance information.

PD activities (con’t). The contractor, under subcontract with the PD provider, shall complete providing the professional development/support to 100 teachers assigned to treatment status during the 2021-2022 school year.

Provider support (con’t). The contractor shall work with the provider and study districts complete arranging the PD and support solid implementation.

Task 14B— Data Collection for Study B (con’t)

Using instruments approved by OMB and observations of teacher classroom practice, the contractor shall finalize school year 2021-22 data collection to estimate the impact on the outcomes of interest and to answer the evaluation questions. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

Task 15B—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contract Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.

The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, OPTION PERIOD III-Study B (Impact of PD Strategy) Task 16B—Data Analyses, including Cost Effectiveness and Report Preparation

The contractor shall conduct data analyses that clearly address the evaluation questions in preparation of a TWG meeting to discuss the study’s impact findings. The contractor also shall conduct cost effectiveness analyses and prepare a report that clearly addresses the impact evaluation questions.

The contractor shall make sure that the impact report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers.