Attachment_A-DRAFT_PWS_Impact_Evaluation_of_Teacher_Preparation.docx

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT)

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

DRAFT Performance Work Statement

FEASIBILITY AND DESIGN OF AN IMPACT EVALUATION OF TEACHER PREPARATION AND PROFESSIONAL DEVELOPMENT

INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (“NCEE”) of the U.S. Department of Education’s (the “Department”) Institute of Education Sciences (IES) requires a contractor to assess the feasibility of conducting impact evaluations of teacher preparation and professional development. The purpose of this design work is to inform future evaluation efforts that will provide much needed information in these areas. Both preparation and professional development strategies that are productive require more credible information about what works. What are the core teacher practices that lead to effective teaching? What are the effective ways of providing support to improve such practices for both pre-service and in-service teachers? The ultimate study design options must provide information to the field about effective ways to improve novice teacher performance using the most rigorous scientific study methods.

The contract will include work to provide options of which aspects of teacher preparation and professional development are most amendable and policy relevant to study. The contract will also include site visits to preparation programs and information gathering from potential providers and experts in the field to inform feasible study options. The work includes a Technical Working Group (“TWG”) meeting and a study design report.

Legislative Authority

This evaluation is authorized by Title II, Part A of the Elementary and Secondary Education Act (ESEA), section 2121-2123 as amended by No Child Left Behind (NCLB) (20 USC 6621-6623).

Background

Current research provides evidence of significant variation in teacher effectiveness, some of which has been documented in prior NCEE evaluation publications (for example, see p. 55 -59 of the NCEE report, “An Evaluation of teachers Trained Through Different Routes to Certification,” http://ies.ed.gov/ncee/pubs/20094043/pdf/20094043.pdf; pp. 57 and 93 of the NCEE report, “the Effectiveness of Secondary math Teachers from Teacher For America and the Teaching Fellows Programs,” http://ies.ed.gov/ncee/pubs/20134015/pdf/20134015.pdf). In addition, the literature has explored the effectiveness of various ways of providing intensive professional development with a variety of emphases. Aspects of the Measures of Effective Teaching (MET) project have explored the relationship between teacher practices and student achievement. Even so, the characteristics and skills that translate into an effective teacher have been largely elusive, but are critical to informing certification policy and policy related to teacher preparation (see discussion in “Addressing Teacher Shortages in Disadvantaged Schools: Lessons From Two Institute of Education Sciences Studies,” http://ies.ed.gov/ncee/pubs/20134018/pdf/20134018.pdf). Therefore, research using the most credible and feasible methodology to provide causal information about what leads to an effective teacher is needed. Specifically:

· What are the competencies and characteristics that are most critical in improving teacher effectiveness? Identifying and also being able to reliably measure these competencies would be very informative for certification policy. Understanding the relative importance of minimal competencies in the areas of content knowledge, pedagogical knowledge, and classroom management skills could provide guidance for preparation programs, both at entry and exit from the program. Understanding key attributes of effective teachers could be informative for program entry requirements. In addition, it would be useful to understand the extent to which these competencies and attributes might vary or not, based on the characteristics of the students taught, once candidates become the teacher of record.

· What are promising ways to prepare an effective teaching force? Identifying effective preparation activities/experiences that lead to more effective teachers could provide guidance for preparation program design for both alternative and traditional routes.

· What are promising ways to support ongoing teacher improvement? Identifying effective professional development experiences and ways of providing such support would provide much needed guidance for the millions of Title II dollars that support current professional development efforts.

Given that various studies indicate that teacher performance improves with experience, one potential approach to improve preparation is to provide more classroom experiences before becoming the teacher of record. This has resulted in many suggesting that a robust clinical experience, borrowing language from the medical profession, is likely to be a promising feature to include in teacher preparation. Effective professional development that provides teachers with the support they need to improve their performance is also needed.

Key Research Questions

The contractor shall provide options for feasible and credible study designs that address as many of the areas of policy interest as possible outlined in Section B: Background. NCEE intends to conduct a future evaluation of teacher preparation and professional development in a subsequent contract or contracts to provide information for improving preparation programs and professional development. Ideally the study would provide useful information for both alternative and traditional routes to certification and for both pre-service and in-service teachers. In performing the tasks described in this Performance Work Statement (“PWS”), the contractor shall address the feasibility of conducting the study and proposed designs.

Currently the Department thinks an intensive in-situation practice and feedback is the most promising feature to be studied. However, if the work of this contract unveils other feasible features for study, those shall be included in the option discussions. The design options report for an impact evaluation shall address the following questions:

· What are the most promising and policy relevant aspects of preparation and professional development that are feasible for study? Are there important variations of the feature or differing features that are policy relevant to study and thus should be varied within the study design?

· Would preparation program operators be interested in participating in such a study, and how could an evaluation team work with programs to conduct the study?

· What would be needed to interest teachers in participating in the study (pre-service and in-service)? What incentives or strategies would make study participation for teachers appealing?

· What are the outcomes of greatest importance for the field to have causal inferences about and what are the most credible measures to capture the outcomes? That is, what is the primary focus of the evaluation effort (and thus requires the most credible study design) versus what aspects could be more secondary (and thus could involve more exploratory analyses)?

· What is the timing of program and candidate recruitment for the preparation aspect of the study that would be needed relative to looking at outcomes when candidates become the teacher of record?

· What are the key supports/incentives required for preparation program, teacher preparation candidate, and ultimately school district participation in the study? Who would deliver the preparation aspects to be studied and how?

Scope of Work

This section specifies the tasks and subtasks that the contractor shall perform.

Task 1—Opening Meeting with the Department

The contractor’s project director and up to four project staff shall meet with the Contracting Officer’s Representative (COR) and other appropriate Department staff within 15 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables: Opening Meeting summary

Task 2—Identify Preparation Experiences for Study Focus

1.1 Develop Feasibility Protocol. The contractor shall develop a protocol to guide discussions with teacher preparation programs. The protocol must be relevant or adaptable for discussions with both programs associated with alternative routes and programs associated with traditional routes. The contactor shall include questions to inform both study design issues as well as provide guidance about necessary incentives to elicit interest in participating in an evaluation of teacher preparation. The contactor shall also develop a protocol to guide discussions with teacher candidates within the preparation programs(again, relevant for candidates from either traditional and alternative routes) to inform both study design issues as well as provide guidance about necessary incentives to elicit interest in participating in an evaluation of teacher preparation.

1.2 Conduct Site Visits/Calls. The contractor shall conduct interviews for up to 12 programs (approximately six programs associated with alternative routes and six programs associated with traditional routes) to accomplish the goals outlined in subtask 2.1.

Task 3—Identify Professional Development Experiences for Study Focus

3.1 Develop Feasibility Protocol. The contractor shall develop a protocol to guide discussions with professional development providers. The protocol must be relevant or adaptable for discussions with a variety of providers (e.g., academicians and organizations that provide professional development). The contactor shall include questions to inform what are likely promising and feasible professional development activities that can be implemented feasibly within an evaluation.

3.2 Conduct Calls/Literature Overview. The contractor shall conduct interviews for up to 8 programs to accomplish the goals outlined in subtask 3.1.

Deliverables: Protocol

Task 4—Study Design Options Report

Based on evidence from existing research and information collected during the site visits conducted in subtask 2.1 and information gathering conducted in task 3, the contractor shall prepare a study design document that refines the study design and incorporates a detailed description of the selected treatment(s) to be studied. The study design(s) shall address the issues raised in the Section B: Background and the key research questions listed in Section C. The contractor shall provide a draft of the study design document to the COR four weeks prior to the TWG meeting. The study design document shall be used as materials for informing the TWG of the study design prior to the TWG meeting (Task 5).

The contractor shall revise the study design report as appropriate, subsequent to the discussion with the technical working group and in consultation with the COR. If TWG members recommend any significant deviation from the study design, data collection, or data analysis indicated in the study design report, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study design report based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Task 5—Technical Working Group for the Evaluation

5.1 Establish Technical Working Group. The role of the TWG will be to advise the contractor and the Department on the conduct of the study including, but not limited to: study design, instrumentation, data collection and analysis. The TWG shall be comprised of approximately 12 individuals, and must include experts in: teacher certification and preparation, teacher professional development, teacher quality, teacher policy, teacher preparation providers, as well as experts in evaluation methodology. The contractor shall submit a list of proposed TWG members no later than four weeks after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members (no later than 6 weeks after contract award), the contractor shall contact each proposed member to secure interest and intention to serve on the TWG.

5.2 Convene Technical Working Group. The contractor shall convene a meeting of the TWG. The TWG meeting shall focus on proposed recruitment feasibility, specifics of the preparation experience and professional development to include (and potentially vary), study design options, data collection, and analysis issues.

The contractor shall submit a written summary of the meeting to the COR within five business days, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in the Department’s facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

Task 6 -- Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contracting Officer (“CO”), Contract Specialist (“CS”), and one copy to the COR.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task or subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the COR. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

Schedule of Deliverables

All deliverables noted below shall be submitted to the Department’s COR, CS, and CO electronically, unless otherwise noted. Unless indicated otherwise, the COR will notify the contractor within two weeks of receipt of any changes required in the deliverables, and the contractor shall have at most two weeks to make the necessary revisions and submit a final deliverable to the Department.

The COR reserves the right to inspect the contractor’s work while in progress. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, in accordance with FAR 52.246-5: Inspection of Services- Cost-Reimbursement.

BASE PERIOD

Task 1—Opening Meeting with the Department

Task 2— Identify Preparation Experiences for Study Focus

Task 3— Identify Professional Development Experiences for Study Focus

Task 4—Study Design Options Report

Draft Study Design Option Report

18 weeks after award

Task 5 – Technical Working Group for the Evaluation

Task 6 – Reporting Requirements