Attachment_A-_DRAFT_TPrepPD_PWS.pdf

Attachment A – DRAFT TPrep/PD Performance Work Statement MAR 24, 2016

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

Performance Work Statement

IMPACT EVALUATION TO INFORM TEACHER PREPARATION AND

PROFESSIONAL DEVELOPMENT

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (“NCEE”) of the U.S.

Department of Education’s (the “Department”) Institute of Education Sciences (IES) has a requirement to evaluation the impact evaluation of support relevant for both teacher preparation and professional development. The purpose of this evaluation is to provide information about effective practices in both of these areas.

PERIOD OF PERFORMANCE and OVERVIEW

The Period of Performance consists of an 18-month base period and four option periods that would result in a 69-month contract in total.

The base period shall include the selection of intervention providers, recruitment of a novice teacher evaluation sample as well as the development of data collection forms and development of the Office of Management and Budget (OMB) forms clearance package, a Technical Working Group (TWG) meeting, a design report, initial intervention implementation, and baseline data collection. Based on documentation of successful intervention provider(s) selection and evaluation sample recruitment, the Department will decide whether or not to exercise Option Period I by September 2017.

Option Period I is 19 months and it overlaps with the base contract by 7 months. Option Period I includes intervention implementation, data collection during the 2017-18 implementation, recruitment of early career teachers, and initial implementation and baseline data collection for the early career teacher evaluation. Based on documented successful conduct of activities during year 2017-18, the Department will decide whether or not to exercise Option Period II by September 2018.

The term of Option Period II is 19 months and it overlaps with the Option Period I by 7 months. Option Period II includes continued early career teacher intervention implementation of the treatment for a more experienced teacher sample, analyses of data from the novice teachers’ evaluation sample, a TWG meeting to discuss novice teacher findings, and an impact report for the novice teacher sample. Based on documented successful conduct of evaluation activities during 2018-19, the Department will decide whether or not to exercise Option Period III by September 2019.

The term of Option Period III is 19 months and it overlaps with the Option Period II by 7 months. Option Period III includes report 2 preparation and a TWG meeting to discuss findings on the more experienced teachers sample and the year 2 findings on the novice teacher sample. Based on the findings for the more experienced teacher sample, the Department will decide whether or not to exercise Option Period IV by September 2020.

The term of Option Period IV is 22 months and it overlaps with the Option Period III by 7 months. Option Period IV includes finalizing report 2 and a synthesis report using year 2 data, as well as a data file for the final report.

Base Option Period I Option Period II Option Period III Option Period IV Oct. 2016-Mar. 2018 Sept.2017-Mar 2019 Sept.2018-Mar 2020 Sept.2019-Mar 2021 Sept.2020-Jun 2022 18 months 19 months 19 months 19 months 22 months Overlaps with base by 7 months Overlaps with Period I by 7 months

Overlaps with Period II by 7 months

Overlaps with Period III by 7 months

Months 1-18 Months 12-30 Months 24-42 Months 36-54 Months 48-69

A. Legislative Authority

This evaluation is authorized by Title II sections 2001-2002 and Title VIII section 8042 of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 6301 et. seq.) as amended by the Every Student Succeeds Act (ESSA).

B. Background

Current research provides evidence of significant variation in novice teacher effectiveness, some of which has been documented in prior NCEE evaluation publications (for example, see p. 55 -59 of the NCEE report, “An Evaluation of teachers Trained Through Different Routes to Certification,” http://ies.ed.gov/ncee/pubs/20094043/pdf/20094043.pdf). In addition, the literature has explored the effectiveness of various ways of providing intensive professional development with a variety of emphases (for example, see NCEE evaluations Elementary School Reading Professional Development Impact Evaluation, http://ies.ed.gov/ncee/projects/evaluation/tq_reading.asp; Middle School Mathematics Professional Development Impact Study, http://ies.ed.gov/ncee/projects/evaluation/tq_mathematics.asp;

Elementary School Math Professional Development Impact Evaluation, http://ies.ed.gov/ncee/projects/evaluation/tq_mathpd.asp; Impact Evaluation of Teacher Induction Programs, http://ies.ed.gov/ncee/projects/evaluation/tq_induction.asp; Impact Evaluation of Professional Development for Teachers on Data-Driven Instruction, http://ies.ed.gov/ncee/projects/evaluation/tq_datadriven.asp. Even so, the characteristics and skills that translate into an effective teacher have been largely elusive, but are critical to informing certification policy and policy related to teacher preparation and professional development.

Several recent studies of opportunities for teachers to receive feedback is a promising strategy to improve teacher effectiveness. Two studies of feedback through teacher evaluation systems found positive effects on student achievement (see Taylor and Tyler 2012; and Steinberg and Sartain 2015). Three studies of coaching interventions found positive effects (see Allen et al. 2011, 2015 and Campbell and Malkus 2011).1

C. Prior Design Work

In February 2016, NCEE held a TWG Meeting to discuss design options for an impact evaluation of teacher preparation and professional development. The TWG was planned and executed by Mathematica Policy Research, Inc. under contract ED-IES-15-C-0034. The meeting focused on potential options for informing teacher preparation and professional development. During the meeting, the TWG considered whether a random assignment evaluation within schools of education was feasible. The TWG also considered whether an evaluation of support for novice teachers and those early in their teaching career would inform policy and practice in the areas of teacher preparation and professional development. The discussion included core features of support that should be present in an intervention to accomplish these goals. The TWG also considered evaluation designs that included the timing of support to include the summer before a teachers’ first year of teaching, during the first year of teaching, and in their early years of teaching (such as years 2-4). The design report from this work and a summary of the TWG meeting are contained in Appendix A.

1 Full research cites are found in the design report from contract, ED-IES-15-C-0034, “Feasibility and Design of an Impact Evaluation of Teacher Preparation and Professional Development,” included in Appendix A.

http://ies.ed.gov/ncee/pubs/20094043/pdf/20094043.pdf http://ies.ed.gov/ncee/projects/evaluation/tq_reading.asp http://ies.ed.gov/ncee/projects/evaluation/tq_mathematics.asp http://ies.ed.gov/ncee/projects/evaluation/tq_mathpd.asp http://ies.ed.gov/ncee/projects/evaluation/tq_induction.asp http://ies.ed.gov/ncee/projects/evaluation/tq_datadriven.asp

D. Key Research Questions

This evaluation will test intensive opportunities for teachers in elementary schools (grades 3-5) to be videotaped and receive ratings and high-quality feedback for improving practice with follow-up as the core focus of the evaluation. The evaluation includes a novice teacher sample and an early career teacher sample (teachers who are in their 2nd to 4th year of teaching). The resulting evaluation provides information that is useful for both teacher preparation and in-service teacher professional development. It also has the potential to inform teacher certification.

The Department will decide whether or not to execute Option Periods I-IV to address the following key research questions:

1. What is the impact on student achievement of multiple feedback sessions using videos of a teacher’s own classroom practice for improving novice teachers’ effectiveness?

2. What is the impact on student achievement of using multiple videos of a teacher’s own classroom practice as a tool for self-reflection to improve novice teachers’ effectiveness?

3. What is the impact on student achievement of multiple feedback sessions using videos of a teacher’s own classroom practice for improving early career teachers’ effectiveness?

Evaluation samples

• 300 novice teachers – who are randomly assigned to receive: i) video, ratings, and feedback, ii) just video and ratings as a tool for self-reflection, or iii) control

• 200 early career teachers – who are randomly assigned to receive: i) video, ratings, and feedback, or ii) control

The novice teacher evaluation sample estimates the impact on student achievement of intensive support for novice teachers to improve classroom practice and student achievement, and the impact of a less intensive video tool that could be used for self-reflection to improve practice. The evaluation will also estimate the impact on student achievement of using videos with feedback for early career teachers with limited experience (in their 2nd to 4th year of teaching).

E. Teacher Support Intervention

The Department anticipates that the contractor, through a subcontract(s) with support provider(s), shall offer the full intervention (video, ratings, and feedback) to approximately 100 novice teachers, a low-cost version of the intervention (video, ratings, and no feedback) to an additional 100 novice teachers, and the full intervention (video, ratings, and feedback) to 100 teachers who are early in their career (such as those in their 2nd-4th years of teaching). During the summer of 2017, the contractor shall randomly assign novice teachers to one of three conditions: i) the full intervention, ii) a less intensive intervention, or iii) control.

If Option Period I is exercised, the contractor shall offer treatment to novice teachers during 2017-2018 school year according to the assigned treatment status. The contractor shall offer the full intervention to early career teachers, if exercised, during the summer of 2018 (Option Period II) and, if exercised, the 2018-2019 school year (Option Period III). The full intervention shall include a focus on high-leverage topics such as: (1) classroom/behavior management, classroom environment and student engagement; (2) class format (use of small group vs. whole class instruction); and (3) strategies for questioning students to facilitate learning.

The following includes additional desired features of the full intervention:

• Use of videos, ratings and feedback for teachers’ information to improve their practice. Specifically:

o Videos and rubrics of teacher practice that guide an assessment of a teacher’s own practice and identify high leverage practices for improvement. Videos that also allow teachers to view their practice to see exactly what they are and are not doing in the classroom and how it is received by their students;

o Expert videos to demonstrate the high quality practices highlighted in feedback for a teacher to improve upon;

• Frequent feedback (approximately 6 feedback cycles during the school year);

• Targeted feedback – that is actionable and targeted on topics that have evidence of improving student achievement (e.g., classroom/behavior management, classroom environment and student engagement, use of small group vs. whole class instruction, questioning strategies of students to facilitate learning);

• Use of multiple videos (approximately 3 videos per feedback cycle) to provide the coach with information for each feedback cycle;

• One-day summer training for teachers to learn about the use of videos and feedback cycle process.

The following includes desired features of the less intensive intervention:

• Use of videos and rating information for teachers to view and reflect on their teaching, on their own, to improve practice;

• Frequent videos – approximately 6 feedback cycles (approximately 3 videos per feedback cycle) during the school year;

• One-day summer training for teachers to learn about the video cycle process and how to interpret the video ratings.

F. Intervention Selection

The contractor shall be responsible for selecting the professional development model to be evaluated.

After the contract award, the contractor shall establish a group of three experts in professional development and teacher preparation effectiveness (approved by the Department). In consultation with the expert panel and the Department, the contractor shall develop an open solicitation and selection criteria. Then, the contractor shall publish the solicitation and contact providers to ascertain their interest and ability to participate in a randomized trial of their program. In response to the solicitation, interested providers shall submit a description of their program and the requirements for implementation, along with any evidence of effectiveness. The contractor shall oversee the expert panelists’ review of proposals in order to identify up to 4 providers for inclusion in follow-up presentations of their program to the panelists in Washington DC. After consideration of the Contracting Officers Representative (COR) and expert panel’s recommendations, the contractor shall select up to two providers for inclusion in the evaluation.

G. Evaluation Approach

This Performance Work Statement (PWS) assumes the following:

The contractor shall implement the programs of up to two different providers through subcontracts with the intervention providers.

The evaluation shall recruit and complete data collection from a sample of 300 novice teachers and an additional 200 early career teachers.

The contractor shall randomly assign 100 novice teachers to Treatment 1 (T1) status (full intervention), 100 novice teachers to Treatment 2 (T2) status (less intensive intervention), and 100 novice teachers to control (C) status; the contractor shall also randomly assign an additional 100 early career teachers (2-4 years of experience) to T1 status (full intervention), and 100 early career teachers (2-4 years of experience) to control status.

The evaluation shall be powered for an effect of approximately 0.10 on school student achievement.

The evaluation shall include approximately 10-12 districts, with geographical diversity.

Data collection shall consist of the intervention implementation information, a teacher survey for the implementation year (background, postsecondary information and route to certification, teacher preparation experiences, professional development activities and hours), baseline and outcome classroom practice measure, student records data for students of the novice and early career teacher samples. The contractor shall collect student achievement data at the end of the treatment year for the novice and early career teacher samples. The contractor also shall collect student achievement data the year after treatment has ended for the novice teacher sample.

DATA COLLECTION by Evaluation Design Novice teacher sample Early career teacher sample Observation ratings x x Coach logs and teacher participation forms x

Teacher Survey x x Competency Test (pre/post) x Baseline classroom observations x (T1, also used for first observation session; C during first month of implementation year) x (T1, also used for first observation rating)

Classroom observation (outcome measure) x-near end of implementation year x- near end of implementation year

Student records (achievement and demographic info) x-implementation year and year after implementation x-implementation year

300 Novice Teachers Teachers in their first year of teaching

T1 T2

C

200 Early career Teachers Teachers with 2-4 years of experience

T1 C

Specific Design Issues

Discussed below are a number of factors that will affect the evaluation design or successful execution of the evaluation. Other design issues may arise as the evaluation is carried out that will need to be resolved by the evaluation team.

• Random Assignment of Teachers. The contractor shall address plans for carrying out random assignment. The contractor shall give particular attention to plans for carrying out random assignment for the early career teacher sample.

• Intervention. The Department understands that there are various ways to implement the intervention for teachers—frequency of sessions, method of delivering the sessions, amount of information (lessons) per cycle, to name a few. The contractor shall identify the most promising ways to deliver the intervention that is policy relevant, while minimizing costs. In addition, the contractor shall identify content that is most likely to have measurable impacts.

• Counterfactual. Teachers are anticipated to be randomly assigned to be offered the treatment or not. A key issue is the nature of support experienced by teachers in the control condition, the counterfactual for those not assigned to the treatment/intervention.

• Power Analyses. The evaluation must be powered for an effect of .10 on school student achievement.

The contractor shall fully explain power calculation assumptions.

II. Scope of Work

This section specifies the tasks and subtasks that the contractor shall perform.

A. Base Period

Task 1—Opening Meeting with the Department

The contractor’s project director and up to four project staff shall meet with the COR and other appropriate Department staff within 5 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the contractor’s final proposal revision, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the evaluation.

Deliverables: Opening Meeting summary

Task 2 –Selection of Intervention Provider(s)

2.1 Expert Provider Selection Panel. Based on names submitted in the contractor’s final proposal revision and in consultation with the COR, the contractor shall propose an expert panel of three persons to advise on the selection of providers within 2 weeks after contract award. These experts may also serve on the TWG described in Task 5. The contractor shall contact each proposed member and obtain commitment to serve on the provider selection panel.

2.2 Request for Proposals (RFP). Based on the contractor’s proposal and in consultation with the COR, the contractor shall prepare, by week 3 after contract award, a provider request for proposals including the timelines for proposals, requirements for the training, and the criteria by which the proposals shall be evaluated by the contractor. The RFP shall also specify the maximum amount of funding that will be available to support the training. The contractor shall publicly post the RFP. Prior to release of the RFP, the contractor shall send the advance announcement to relevant providers and post it on the website(s) of relevant associations.

2.3 Selection of Provider(s). The contractor shall request that each member of the expert panel review the provider proposals and write a brief review of each proposal, including each provider’s strengths and weaknesses, along with recommendations of up to four providers for inclusion in the in-person panel meetings. The contractor shall provide the reviews and recommendations of these experts to the COR within 2 weeks after the provider proposal submission date. Expert panel members will participate in a one- or two-day meeting at the Department’s offices in Washington, DC, organized by the contractor, with up to four providers in which each provider makes a short presentation of their intervention activities and responds to questions from expert panel members.

In consultation with the COR and the expert panel, the contractor shall secure the recommended provider(s) to participate in the evaluation.

Deliverables: List of expert panel members to review proposals

RFP

Expert panel review of proposals

Task 3– Prepare OMB Package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments. While the preparation of the data collection forms occurs as part of the base contract, data collection using the forms will only occur if the option period(s) are executed.

The package shall include a description of the evaluation design, site recruitment, and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the evaluation mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the evaluation and its potential use for policy decisions. The contractor shall submit a draft forms clearance package to the COR no later than 4 months after contract award. The COR will review within 2 weeks. The contractor shall submit an acceptable revised forms clearance package for original submission no later than 5 months after contract award. The COR will submit the accepted forms clearance package for clearance. Based on comments from internal and external clearance (after the first 60 day clearance process), the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB not later than 7 months after contract award. The COR will then submit the final revised forms clearance package to OMB.

Data collection for the core evaluation Option Periods I-IV shall consist of:

Observation ratings –The contractor shall keep records of observation video ratings and their provision to teachers for the core evaluation.

Coach logs and teacher participation forms – These forms will provide information on full intervention participation and feedback content and less intensive intervention participation. The contractor shall collect these data after each intervention video cycle. The contractor shall keep records of observation video recordings, ratings, and their provision to treatment teachers.

Teacher survey – This survey will provide data on teacher characteristics, their preparation routes and relevant prior preparation experiences, and their support experiences during the year of implementation. This survey shall be administered in the spring of the intervention year.

Teacher competency test – This test will provide pre-post competency information for the novice teacher sample. These tests shall be administered in the fall of 2017 and spring of 2018.

Classroom observation ratings – These will provide pre-post practice information. The contractor shall collect these data in the fall of 2017 and spring of 2018 for the novice teacher sample. The contractor shall collect these data in the fall of 2018 and spring of 2019 for the early career teacher sample.

Student records data – These will include data on student characteristics and test scores, as well as other relevant covariate and subgroup information. For the novice teacher sample, the contractor shall collect data for the year prior to the first implementation year (2016-17), the year of implementation (2017-18), and the year following implementation (2018-19). For the early career teacher sample, the contractor shall collect data for the year prior to the first implementation year (2017-18) and the year of implementation (2018-19).

The contractor shall design data collection instruments to address the key evaluation questions. The contractor shall make use of or refine existing instruments to the extent possible and applicable.

For instruments where validity or reliability are not yet well established, the contractor shall pilot test each of the instruments with no more than nine respondents per instrument in no less than two schools to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Draft original submission OMB clearance package

Original submission OMB clearance package (first 60 day OMB submission) Revised OMB clearance package (second 60 day OMB submission) Final OMB clearance package (cleared package)

Documentation of IRB approvals

Task 4— Selection and Recruitment of Evaluation Districts and Teachers

During the selection and recruitment process, the contractor shall frequently update the COR. The contractor shall provide monthly selection and recruitment updates in the monthly progress reports (see Task 8).

The contractor shall, in consultation with the COR, develop a strategy for selecting evaluation sites and prioritize potential sites based on criteria that the COR and contractor agree are pertinent to addressing the evaluation objectives. These shall include policy-relevant districts with capabilities sufficient to support the evaluation’s needs and a willingness to support and participate in the professional development PD.

Based on the exploratory contacts with potential sites and sample selection criteria developed in consultation with the COR, the contractor shall recruit evaluation districts and teachers to participate in and cooperate with the evaluation. The contractor shall secure final agreements to participate through signed memoranda of understanding from districts so that a total of approximately 10-12 districts and 300 novice teachers are included in the novice teacher evaluation sample. The contractor shall also obtain buy-in to recruit 200 early career teachers in the spring of 2018 to participate in the early career teachers evaluation sample.

The contractor shall complete all district recruitment activities no later than 8 months after contract award. Random assignment of novice teachers shall occur prior to the summer training session in 2017.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverable: Memoranda of Understanding from districts.

Task 5—Technical Working Group for the Evaluation

5.1 Establish Technical Working Group (TWG). The role of the TWG will be to advise the contractor and the Department on the conduct of the evaluation including, but not limited to: evaluation design, instrumentation, data collection and analysis, and draft evaluation reports. The TWG shall be comprised of approximately 10 individuals, and include experts in teacher certification and preparation, teacher quality, teacher policy, teacher preparation providers, as well as experts in evaluation methodology. The contractor shall submit a list of proposed TWG members no later than 8 weeks after contract award.

When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the evaluation.

After the COR approves the proposed TWG members, the contractor shall contact each proposed member to secure interest and intention to serve on the TWG.

5.2 Convene Technical Working Group. The contractor shall convene a meeting of the TWG at the Department’s offices in Washington, DC in the base period. The TWG meeting shall be held no later than 6 months after the contract award and shall focus on remaining design issues, data collection, and analysis issues. The contractor shall prepare an evaluation design document that refines the evaluation design and incorporates a detailed description of the selected treatment to be studied. The evaluation design document will be used as materials for informing the TWG of the evaluation design prior to the TWG meeting. A draft of the evaluation design document shall be provided to the COR 4 weeks prior to the TWG meeting.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions.

Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S.

Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

If TWG members recommend any significant deviation from the proposal, evaluation design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the

COR for approval prior to implementation. The contractor shall not make any significant changes in the evaluation based solely upon the recommendation of TWG members without the written approval of the COR. The contractor shall also use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables: List of proposed TWG members Refined evaluation design document

Summary of TWG meeting

Task 6—Implement and Support the Intervention for Novice Teachers

As part of the monthly reporting requirement (task 8), the contractor shall provide ongoing information about the intervention activities as well as participant attendance information.

6.1 Summer Session and Fall implementation of full intervention. The contractor, under subcontract(s) with the intervention provider(s), shall provide and implement the full intervention to those novice teachers assigned to T1. The contractor shall work with the provider(s) and evaluation districts to:

• Arrange a one-day training for novice evaluation teachers assigned to treatment status T1 during the summer prior to the 2017-2018 school year.

• Provide the full intervention in the fall of 2017 to those teachers assigned to T1.

6.2 Full Intervention Implementation Support. The contractor shall work with the intervention provider(s) to ensure implementation with fidelity consistent of the full intervention during the fall of 2017. The contractor shall also ensure implementation consistent with the requirements of the evaluation design.

6.3 Summer Session and Fall implementation of less intensive intervention. The contractor, under subcontract(s) with the intervention provider(s), shall provide and implement the less intensive intervention to those novice teachers assigned to T2. The contractor shall work with the provider(s) and evaluation districts to:

• Arrange a one-day training for novice evaluation teachers assigned to treatment status T2 during the summer prior to the 2017-2018 school year.

• Provide the less intensive intervention in the fall of 2017 to those teachers assigned to T2.

6.4 Less intensive Intervention Implementation Support. The contractor shall work with the intervention provider(s) to ensure implementation with fidelity consistent of the less intensive intervention during the fall of 2017. The contractor shall also ensure implementation consistent with the requirements of the evaluation design.

Task 7 – Baseline and Fall Data Collection

Using instruments approved by OMB, the contractor shall collect baseline for the novice teacher evaluation sample (video observations and ratings and competency test, as described in Task 3). The contractor shall also collect fall implementation information (coach logs and teacher participation forms, as described in Task 3). The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

The contractor shall establish a contact person and/or telephone number for district staff to contact with questions or for assistance in adhering to the planned random assignment, and to minimize “contamination” and “cross-over” between groups. The contractor shall carefully monitor intake of these inquiries and note any breaches in the planned random assignment in the monthly reports.

http://nces.ed.gov/pubs92/92021.pdf

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The data monitoring activities shall be in place within 11 months after contract award. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide information on these procedures to the Government upon request. When possible, individuals collecting and/or coding data shall have no knowledge of each school’s treatment condition.

As part of the data collection process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Compliance with the Department’s IT Security Policy

The contractor, and all sub-contractors, shall comply with the Department of Education’s IT security policy requirements, specifically those set forth in the ‘Handbook for Information Assurance Security Policy (OCIO-01)’, and other applicable procedures and guidance. The contractor, and all sub-contractors, shall develop and implement management, operational and technical security controls to assure required levels of protection for information systems. The contractor, and all sub-contractors, shall further comply with all applicable Federal IT security requirements including, but not limited to, the Federal Information Security Management Act (FISMA) of 2002, Office of Management and Budget (OMB) Circular A-130 Appendix III, Homeland Security Presidential Directives (HSPD), the National Institute of Standards and Technology (NIST) standards and guidance, and the Federal Risk and Authorization Management Program (FedRAMP) requirements and guidance.

These security requirements include, but are not limited to, the successful Security Authorization (SA) of the system (includes commercially owned and operated systems managed by the commercial vendor and its sub-contractors, supporting Department programs, contracts, and projects); obtaining a full Authority to Operate (ATO) before being granted operational status; performance of annual self-assessments of security controls; annual Contingency Plan testing; performance of periodic vulnerability scans; updating all information system security documentation as changes occur; and other continuous monitoring activities, which may include, mapping, penetration and other intrusive scanning. Full and unfettered access for the Department’s third party Managed Security Services Provider (MSSP) must be granted to access all computers and networks used for this system. Additionally, when there is a significant change to the system’s security posture, the system (Federal and commercial prime- and sub- contractors included) must have a new SA, with all required activities to obtain a new ATO, signed by the Authorizing Official (AO).

System security controls shall be designed and implemented consistent with NIST SP 800-53 Rev 3, ‘Recommended Security Controls for Federal Information Systems and Organizations.’ All NIST SP 800-53 controls must be tested / assessed no less than every 3 years, according to federal and Department policy. The risk impact level of the system will be determined via the completion of the Department's inventory form and shall meet the accurate depiction of security categorization as outlined in Federal Information Publishing Standards (FIPS) 199, ‘Standards for Security Categorization of Federal Information and Information Systems.’

System security documentation shall be developed to record and support the implementation of the security controls for the system. This documentation shall be maintained for the life of the system. The contractor, and all sub-contractors, shall review and update the system security documentation at least annually and after significant changes to the system, to ensure the relevance and accurate depiction of the implemented system controls and to reflect changes to the system and its environment of operation.

Security documentation must be developed in accordance with the NIST 800 series and Department of Education policy and guidance.

The contractor, and all sub-contractors, shall allow Department employees (or Department designated third party contractors) access to the hosting facility to conduct SA activities to include control reviews in accordance with NIST SP 800-53, Rev. 3 and NIST SP 800-53A. The contractor, and all sub-contractors, shall be available for interviews and demonstrations of security control compliance to support the SA process and continuous monitoring of system security. In addition, if the system is rated as ‘Moderate’ or ‘High’ for FIPS 199 risk impact, vulnerability scanning and penetration testing shall be performed on the hosting facility and application as part of the SA process. Appropriate access agreements will be reviewed and signed before any scanning or testing occurs.

Identified deficiencies between required NIST SP 800-53 Rev. 3 controls and the contractor’s, and all sub-contractor’s implementation, as documented in the Risk Assessment Report, System Security Plan (SSP) and Security Assessment Report (SAR), shall be tracked for mitigation through the development of a Plan of Action and Milestones (POA&M) in accordance with the ‘Handbook for Information Assurance Security Policy (OCIO-01).’ Depending on the severity of the deficiencies, the Department may require remediation before an ATO is issued.

All awarded contracts shall ensure that:

1. Their IT product/system is monitored during all hours of operations using entrusted detective/preventive systems;

2. Their IT product/system has current antiviral products installed and operational;

3. Their IT product/system is scanned on a reoccurring basis;

4. Vulnerabilities are remediated in a timely manner on their IT product/system; and

5. Access/view for cyber security situational awareness on their IT product/system is made available to the Department CIRC (cyber incident response capability).

Internet Protocol version 6 (IPv6) For IPv6, the contractor shall provide commercial off-the-shelf COTS solutions that are IPv6 capable. An IPv6 capable system or product shall be capable of receiving, processing, transmitting and forwarding IPv6 packets and/or interfacing with other systems and protocols in a manner similar to that of IPv4.

Specific criteria to be deemed IPv6 capable are:

• An IPv6 capable system that meets the IPv6 base requirements defined by the USGv6 Profile

(http://www.antd.nist.gov/usgv6/profile.html).

• Systems being developed, procured or acquired shall maintain interoperability with IPv4 systems/capabilities.

• Systems shall implement IPv4/IPv6 dual-stack and shall also be built to determine which protocol layer to use depending on the destination host it is attempting to communicate with or establish a socket with. If either protocol is possible, systems shall employ IPv6.

The contractor shall provide IPv6 technical support for system development, implementation and management.

Reporting of Data Security Breaches If there is a suspected or known breach/disclosure of Personally Identifiable Information (PII) due to lost, theft, intercepted transfer, or other, the contractor must ensure that this breach is reported to the agency as soon as the contractor has knowledge of it. Per Office of Management and Budget Memorandum M-06- 19, Federal agencies have a requirement to report breaches of PII security to a Federal incident response center. IES must notify the department within 30 minutes of discovering the incident (and the agency should not distinguish between suspected or confirmed breaches). The data security plan must be written to reflect this requirement, and the contractor must provide sufficient notification and documentation of the suspected loss, as it is understood at the time of notification to the agency for this requirement to be met. Follow-up reports of the final status of loss events will also be prepared by the contractor within a reasonable period of time as advised by the COR.

Task 8- Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contracting Officer (CO) and one copy to the COR, simultaneously with the monthly invoice. The base phase includes performance reports for months 1 through 18.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive work and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

A. Option Period I – Implementation of and Data Collection for Novice Teachers’ Intervention http://www.antd.nist.gov/usgv6/profile.html

Task 9 – Intervention Implementation and Support for Novice Teacher Sample

As part of the monthly reporting requirement (Tasks 8 and 14), the contractor shall provide ongoing information about the intervention activities and as well as participant attendance information.

9.1 Winter Implementation of Full Intervention. The contractor, under subcontract(s) with the intervention provider(s), shall provide and implement the full intervention to those novice teachers assigned to T1. The contractor shall work with the provider(s) and evaluation districts to provide the full intervention in the winter of 2018 to those teachers assigned to T1.

9.2 Full Intervention Implementation Support. The contractor shall work with the intervention provider(s) to ensure implementation with fidelity consistent of the full intervention during the winter of 2018. The contractor shall also ensure implementation consistent with the requirements of the evaluation design.

9.3 Winter Implementation of Less intensive Intervention. The contractor, under subcontract(s) with the intervention provider(s), shall provide and implement the less intensive intervention to those novice teachers assigned to T2 during the winter of 2018.

9.4 Less intensive Intervention Implementation Support. The contractor shall work with the intervention provider(s) to ensure implementation with fidelity consistent of the less intensive intervention during the winter of 2018. The contractor shall also ensure implementation consistent with the requirements of the evaluation design.

Task 10 — Data Collection during 2017-18 Schoolyear for Novice Teacher Sample

Using instruments approved by OMB, the contractor shall collect all evaluation administered data (observation ratings, coach logs and teacher participation forms, spring 2018 teacher survey, and 2016-18 student records data, as described in Task 3) to estimate the impact of the intervention on the outcomes of interest and to answer the evaluation questions during the implementation year. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in task 7 above.

Task 11—Recruitment and Random Assignment of Early career Teacher Sample

The contractor, in consultation with districts recruited in task 4, shall identify 200 early career teachers in elementary schools to participate in an evaluation of the full intervention during the 2018-19 school year.

After obtaining teacher buy-in, the contractor shall randomly assign 200 early career teachers to receive the full intervention or the control condition.

Task 12—Implement and Support the Intervention for Early career Teachers

As part of the monthly reporting requirement (Tasks 8 and 14), the contractor shall provide ongoing information about the intervention activities and as well as participant attendance information.

12.1 Summer Session and Fall Implementation of Full Intervention. The contractor, under subcontract(s) with the intervention provider(s), shall provide and implement the full intervention to those early career teachers assigned to T1. The contractor shall work with the provider(s) and evaluation districts to:

• Arrange a one-day training for early career evaluation teachers assigned to treatment status T1 during the summer prior to the 2018-2019 school year.

• Provide the full intervention in the fall of 2018 to those teachers assigned to T1.

12.2 Full Intervention Implementation Support. The contractor shall work with the intervention provider(s) to ensure implementation with fidelity consistent of the full intervention during the fall of 2018. The contractor shall also ensure implementation consistent with the requirements of the evaluation design.

Task 13: Baseline data collection (observations, report 2)

Using instruments approved by OMB, the contractor shall collect baseline observation videos and ratings for early career teacher evaluation sample, as described in Task 3. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 7 above.

Task 14- Reporting Requirements

In addition…