DRAFT_PWS_Parent_Information_EDIES15R0023.pdf

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 1

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

Performance Work Statement

EXPERIMENTS TO IMPROVE SCHOOL CHOICE INFORMATION:

EXAMINING FORMAT, AMOUNT, AND ORGANIZATION

I. INTRODUCTION

The United States Department of Education’s (ED) Institute of Education Sciences (IES), National Center for Education Evaluation and Regional Assistance (NCEE) division will be conducting an evaluation to identify the best ways to present school choice information so that it can be understood and used by parents, including those with limited time, economic resources, or English language literacy challenges. Using experimental methods, the evaluation will answer questions about optimal ways to format information, provide the right amount of information, and organize or sequence information so that parents can knowledgeably engage in the choice process.

IES requires a contractor to develop parent information materials for use in a series of small scale low cost quick turn-around “lab” based experiments. The contract will include an option to carry out the experiments and to prepare a short user-friendly guide, based on the evaluation’s findings that will help school districts design or refine their own parent information materials.

II. THE EVALUATION

A. Legislative Authority

OII’s Charter Schools Program (Title V, Part B, Subpart 1 of ESEA, 5205) provides for evaluation and technical assistance efforts directed toward improving choice programs.

B. Background

School choice is among the most visible K-12 education policy trends to emerge over the last two decades. Through inter- and intra-district open enrollment, charter schools, magnet schools, and private school voucher programs, options for public school students have increased substantially. The number of students enrolling in choice schools each year has steadily grown too. For example, the percent of students enrolled in charter schools nationwide grew 13 percent from 2012-13 to 2013-14, continuing a pattern

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 2

documented over the last ten or so years . The number of students taking advantage of inter- and intra-district open enrollment, including magnet school options, has also steadily increased. Despite the fact that more families are engaging in school choice, studies examining the impact of choice on improved student achievement have yielded mixed findings with a great deal of variation within and across types of schooling options

One way to potentially improve the effectiveness of school choice is to focus more attention on the choosers themselves. Market based school choice theories are based on the idea that parents know what is best for their children and will select schools accordingly. This should benefit the students attending selected schools and also force lower performing schools to improve or to close because of low enrollment. However, research suggests that the information parents need to make knowledgeable choices is often hard to access and available data frequently exceeds the comprehension abilities of intended audiences . Low-income parents, those for whom alternative options to neighborhood schools is likely to be particularly important, often have limited ability to locate and use the information meant to help them choose . This suggests that parents may be making choices based on limited information that is difficult to understand and may then make choices not best aligned with their needs and values. For example, parents who state that academic achievement is among the school characteristics they value most may not make choices reflecting this viewpoint. Efforts to arm parents--including those with limited resources--with accessible understandable information for use during the decision making process are worth pursuing.

Surprisingly, few studies have rigorously examined school choice information strategies or attempted to identify effective methods of information presentation and delivery.

Given this and the potentially important role of parent information, IES commissioned a design and feasibility study that aimed to (a) identify promising strategies for providing parents with high-quality information and (b) examine the feasibility of an impact evaluation focused on this topic.

The design and feasibility contract was awarded to IMPAQ International and their subcontractors in the fall of 2012 (Contract No. ED-IES-12-C-0100). The key products resulting from this work--which included a synthesis of existing literature, an examination of current information practices in large districts offerings choice, and convening a group of experts to discuss what is known, what is not known, and what we

National Alliance for Public Charter Schools (2014). “Dashboard; A Comprehensive Data Resource from the National Alliance for Public Charter Schools.”

E.g., Teasley, Bettie. (2009). Charter school outcomes. In Berends, Mark, Matthew G. Springer, Dale

Ballou, and Herbert J. Walberg (Eds.). Handbook of research on school choice. Taylor & Francis, Inc. pp. 209‐225.

Goldhaber, Dan and Eide, Eric. “What Do We Know (and Need to Know) About the Impact of School

Choice Reforms on Disadvantaged Students?” (2002). Harvard Educational Review, 72(2): 157-176.

Schneider, M., & Buckley, J. (2002). What do parents want from schools? Evidence from the internet.

Educational Evaluation and Policy Analysis, 24, 133–144.

E.g., Lubienski, C. (2007). Marketing schools: Consumer goods and competitive incentives for consumer information. Education and Urban Society, 40(1), 118–141.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 3

need to know about effective information strategies--are attached. Findings strongly suggest that conducting a field trial testing out the effectiveness of information strategies would be premature. Current practices are so diverse and the body of research on effective strategies is so limited, that it is not clear what interventions would be promising enough to test at scale. Moreover, while getting information into the hands of those who need it most is certainly a central issue deserving more research, the design work helped IES to conclude that before studying dissemination strategies there needs to be more research focused on what it is that would be disseminated. Giving parents access to confusing, overwhelming, information where data are hard to interpret and options hard to decipher would be unlikely to help.

This evaluation represents a narrowly focused first step in examining the role of parent information in the context of school choice. Using methods to minimize cost, burden, and turnaround time, the evaluation will answer questions about how information should be displayed, how much of it should be offered, and how it can be optimally organized

Information related to school performance (i.e., academic information), school location, transportation options, school demographics, school climate and safety, and clear guidance about application and enrollment procedures--the information prior research indicates parents need and care about most--will be the focus. Findings from this evaluation will lay the foundation for future work that might examine effective and/or novel dissemination methods, actual decision making behavior when understandable and useable information is disseminated, and ultimately impacts on achievement.

C. Key Policy and Research Questions

The general research questions that the contractor shall answer in carrying out this evaluation are as follows:

• What is the optimal way to present school choice information? (E.g., Which presentation formats make school performance/achievement data easiest to understand? How should transportation options and location be conveyed?)

• What is the right amount of information to provide to parents whom might be making a school choice? (E.g., Is there a limit to the number of choice options and amount of information that parents can reasonably consider?)

• How is school choice information best organized? (E.g., Does the order in which information is presented shape how it is understood and used? Is there a way to make the information parents most often report caring about salient?)

The matrix below illustrates these general research questions and offers a non-exhaustive sampling of the many specific forms they may take and the number of contrasts that an experiment addressing these questions might test. For example, when

Studies examining such questions have been conducted in fields outside of education (e.g., healthcare, financial services, marketing) and have found that these factors do impact the extent to which information can be understood and used.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 4

examining the right amount of information to offer, an experiment might focus on how many pieces of information a parent can consider for a given school or how many schools a parent can compare.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 5

INFORMATION

(topics and specific type of information to present)

PRESENTATION

What are the best ways to present information so that data, instructions, etc.

are understandable and used?

AMOUNT

How much information can and will parents process?

ORGANIZATION

How can information be sequenced and organized so that key content is salient and used?

Achievement

▪ AYP

▪ % Proficient or advanced on achievement tests

▪ Achievement growth

▪ Graduation rates

▪ Post‐secondary enrollment

▪ Raw score

▪ Raw score converted to numerical rating

(e.g., 1-5, 1-10, 1-100)

▪ Iconic display of rating

(e.g., green to red stop light, quality stars)

▪ Chart to show how school compares to district average

Location

▪ Map display

▪ List organized by distance from home

▪ List organized in other ways

Transportation

▪ District funded

▪ Publically funded (e.g., metro)

▪ Privately funded (e.g., carpools)

▪ Icons (next to school name) to show transportation provided

▪ Written document with description of options and conditions for use

Demographics

▪ Race/ethnicity

▪ Percent free lunch

▪ Special education students

▪ ELL students

▪ Pie charts showing composition

▪ Tables with text

▪ Chart to show how school compares to district average

School Climate/Safety

▪ Suspensions/expulsions

▪ Student attendance rates

▪ Student engagement

▪ Parent satisfaction

▪ Crime statistics

▪ Raw score (e.g., parents survey average)

▪ Raw score converted to numerical rating

(e.g., 1-5, 1-10, 1-100)

▪ Iconic display of rating

▪ Chart to show how school compares to district average

Enrollment Process/Procedures

▪ Enrollment periods

▪ Selection/lottery process

▪ Key dates

▪ Acceptance rates

▪ Waitlist procedures

▪ Glossaries with key terms

▪ Calendars to display dates

▪ Charts to show likelihood of acceptance

▪ How many information topics can parents consider for each school?

▪ ▪ E.g., Are there topics that should be excluded from basic information sources

(to avoid too much)?

▪ How much specific information for each topic can be shown?

▪ ▪ E.g, How many types of performance information should be shown?

▪ ▪ E.g., Is a performance composite score useful and understandable or confusing?

▪ ▪ E.g., Does using the same presentation for all performance data help parents make use of more information?

▪ For how many schools should parents be shown information (all options, a set of options, etc.)?

▪ Should certain topics of information always come first?

▪ ▪ E.g., Should achievement information be shown first?

▪ Should information be presented sequentially or together? I.e., Are certain pairings of information important and optimal?

▪ ▪ E.g., Would achievement information organized by school location be most useful?

▪ Are interactive tools, where parents can self select topic and type of information, helpful?

▪ Are comparison tools, where parents can look across schools all at once helpful?

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 6

D. Evaluation Approach

The evaluation will use experimental methods to answer the research questions posed above. Unlike many IES studies, this work will not entail carrying out a field trial or testing interventions in natural settings. Rather, the nature of the research questions and the evaluation’s goals call for carrying out a series of “lab” based studies. These studies shall take advantage of opportunistic samples of parents and will be carried out in highly controlled environments where the impacts of relatively subtle contrasts may be examined on very proximal outcomes (e.g., self-reported usefulness of information, time spent examining information displays) that theoretically might link to actual choice behavior and longer term outcomes such as student achievement.

E. Design Issues

Several concerns must be taken into account in designing this evaluation. These include:

Specific questions each experiment will answer and contrasts that will be tested.

As the matrix above illustrates, there are numerous specific questions about format, amount of information, and organization that might be asked. For each question, there are also multiple contrasts that an experiment might test (e.g., do parents better understand performance data when given raw achievement test scores or ratings based on a five point scale for each school in the district?).

Given the time and cost constraints of the evaluation, it will not be possible to examine every possible question or compare every potential format, amount, or method of organization.

Creating a sound organizational framework to guide the study will involve making judicious decision based on: (a) Areas where there is consensus on a topic and additional research would not be cost efficient (e.g., research has consistently supported that readability should be in the 5 th

-8 th grade range); (b) An understanding of those contrasts experts in school choice and research in other areas such as healthcare, financial services, marketed and information and behavioral sciences suggest are most promising to have an impact on the outcomes of interest; and (c) Consideration of the extent to which experimental findings would provide actionable guidance to districts.

At the same time, creative evaluation designs may include experiments that address more than one general research question. For example, one issue that is within the scope of the current evaluation is whether and how academic information would be used, by parents stating they value academics, if it was easier to understand and/or more salient. An experiment addressing this issue might explore both presentation and organization of academic information.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 7

Outcome measures. The primary outcomes of interest for this evaluation are related to understandability, perceived usability, and actual use of information during the decision making process. Whenever possible, already validated measures of outcomes should be used, even if these have been developed outside the field of education. Due to the limitations inherent in relying on self-reported data only, the contractor should consider other methods – used in prior research efforts or developed for the current experiments – that will provide more direct assessments of these constructs. For example, the amount of time it takes a participant to identify the three schools, among a set of options, with the best academic performance might provide an assessment of how easy it was for participants to understand data displays showing achievement information. The contractor will have to use creative methods for examining the extent to which information is understood and used.

Experimental design (within versus between subject) and power: As suggested by the design report, there are multiple experimental designs, each with benefits and trade-offs that will need to be considered. The contractor will have to consider whether or not more than one research question can be examined within one experiment, how many contrasts can be compared within an experiment and by each participant, and how to balance cost efficiency, parsimony, and validity.

Taking this information into account, and considering the outcomes that will be assessed and assumed effect sizes, the contractor will carry out power analyses to estimate needed sample sizes for each experiment.

Parent/participant recruitment. This evaluation is meant to provide basic information that can be used on its own, for districts wanting to improve their parent information, and in a future field trial that will examine the effects of dissemination strategies on parents’ choice related behaviors and on their children’s achievement. Time and cost efficiency will need to be balanced with maintaining external validity. We expect that the contractor will take advantage of opportunistic samples (e.g., appropriate standing panels of survey participants).

However, it may be beneficial to recruit additional participants representative of the target population (i.e., low income parents with school aged children) in order to have large enough samples in each experiment or to assure that subgroups of interest are included in large enough numbers. It may also be important to target parents for whom English is not a first language if this group can feasibly be included.

F. Period of Performance

The period of performance for this requirement shall be for one 10-month base period and one 14-month option period.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 8

III. SCOPE OF WORK

Base Period Tasks: Planning and Preparation

Task 1a--Meetings with ED and Progress Reports (Fixed Price)

1.1a Opening Meeting: The contractor’s project director and up to two key project staff members shall meet with the Contracting Officer (CO)/Contract Specialist (CS) and ED’s

Contracting Officer’s Representative (COR) and other appropriate IES staff within 10 work days after the effective date of the contract award. At least three working days prior to the meeting, the contractor must provide an agenda that includes discussion of the various tasks outlined in their final proposal revision, and that allots time for the COR to provide further details concerning the focus of task specific activities. The contractor must also come prepared to discuss the scheduling of activities, to identify any areas of concern, to suggest ways of addressing potential problems, and to review other issues related to the contract. Within two weeks following the meeting, the contractor must prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

1.2a Monthly Progress Report/Exception Reports: The contractor must prepare monthly progress reports due within 10 work days after the end of each month. The reports must summarize the major activities and accomplishments for the reporting period. In addition, they must provide information for each project task regarding significant events/findings, problems encountered, and staff use. The reports must also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the management plan. If there are deviations, the contractor must describe the plan for resolving the problems and should identify and discuss any decisions which may be needed from the COR.

1.3a Monthly Manpower/Expenditure Reports: The contractor must prepare monthly expenditure reports due within 10 work days after the end of each month. These reports, prepared and signed by the project director, must summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report must project similar assignment information for the upcoming month. The reports must also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Task 2--Refine and Finalize Evaluation Design and Materials (Cost

Reimbursement)

The contractor must revise the evaluation design submitted in the final proposal to elaborate on, refine, and ultimately finalize plans for caring out the series of experiments.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 9

2.1 Refined evaluation design memo: The contractor must prepare a memo that lays out the refined evaluation design, building on any organizational framework and/or areas selected for particular focus laid out in the final proposal. The memo must center around a table that provides the following information for each experiment: the specific research questions being addressed, the contrasts that are being considered for testing

(independent variables), the outcomes that will be measured (the dependent variables), the experimental design, and sample size requirements. The table must also make clear which of the general research questions each experiment addresses.

In a “materials” appendix, the contractor must include for each experiment: (a) sample visual representations of the contrasts that will be tested (e.g., for an experiment comparing different formats for offering school performance data the contractor shall include screen shots showing data displays to be compared); (b) the outcome measures

(existing or proposed for piloting); and (c) an outline of the procedures that will be followed and the instructions that participants will receive. It is expected that similar measures, procedures, and instructions will be used for multiple experiments.

The memo must also include a revised plan for obtaining the necessary number of evaluation participants, largely comprised of low income parents and potentially those for whom English is second language and any revisions to the analysis plan laid out in the proposal. Any materials that will be used to recruit parents living in districts offering school choice shall also be included, including a handout that makes it clear that no personally identifying information will be collected.

The draft evaluation design memo must be submitted to the COR no more than 12 weeks after the contract has been awarded. The COR shall provide feedback within two weeks and the contractor must make any revisions necessary in order to prepare the forms clearance package (OMB package) discussed in Task 4.

2.2 Refined materials appendix: In preparation for the first Technical Working Group

(TWG) meeting discussed below, the contractor must submit a refined version of the memo’s materials appendix. Depending on feedback from the COR, this revised appendix may include fewer proposed contrasts but must include more comprehensive and elaborate experimental materials so that TWG members can evaluate and provide guidance on the specific variants the contractor is considering for testing. The refined materials must be submitted to the COR four weeks prior to the first TWG meeting.

2.3 Final evaluation design memo: The final evaluation design memo will take into account feedback from the TWG discussed in Task 3 and from the COR. Any changes needed must be reflected in the final evaluation design memo. For example, changes to the specific questions being examined, the contrasts being tested or the manner in which they will be visually represented, or the outcome variables being measured must be reflected in the final evaluation design memo. In addition, the methods for measuring outcomes and the set of data collection instruments that will be used must be finalized.

For instruments where validity or reliability have not been well established, the contractor must have pilot tested with no more than nine respondents.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 10

The final evaluation design will also elaborate on the procedural details laid out in deliverable 2.1. For each experiment, a study protocol must be added which outlines exactly how the experiment will be implemented. For example, instructions for participants and the manner in which they will be delivered must be detailed. What participants will learn and when, how they will encounter and engage with study materials, the directions they will be given, and any other details pertaining to how the experiments will be carried out must be included. This will vary to some extent across the experiments as some may use straightforward methods for assessing outcomes and others may use scenarios or create actual choice situations to gather behavioral information.

After the TWG meeting the contractor and the COR will agree upon any changes to the evaluation design and materials. The final deliverable shall be submitted four weeks after the TWG meeting.

2.4 Final materials and implementation plan: Once OMB clearance has been obtained the contractor must finalize all materials and take all other steps necessary to implementing the experiments. This task will likely require the contractor to communicate with the operator of a standing panel in order to make sure that materials are in appropriate format for use with this sample. The contractor must also have materials ready to be used in experiments where participants are directly recruited. The final materials must be ready for COR review six weeks after OMB clearance has been obtained.

Task 3--Technical Working Group (TWG)

The contractor must convene a group of experts who will provide advice and guidance concerning specific research questions, key contrasts for testing, material development, testing procedures, and interpretation of findings.

3.1 Establish TWG: The contractor must recruit a team of seven to eight TWG members whose expertise areas include (but are not limited to): school choice research, policy, and practice; parent engagement; lab based experimental design and analysis/behavioral science; and consumer information strategies and science. The contractor must submit a list of potential TWG members, that includes a description of each expert’s credentials and specific area of expertise, to be reviewed and approved by the COR no more than two weeks after the kick off meeting. When the proposed TWG members are approved by the COR, the contractor shall contact each proposed member and submit letters of commitment within two weeks of approval.

3.2 Convene First TWG Meeting: The contractor must convene one full day meeting of the TWG approximately 22-24 weeks after the contract has been awarded. The meeting will focus on discussing the organizational framework for the set of experiments and soliciting feedback regarding the specific questions being explored and the contrasts being tested. Methods and materials for the series of experiments will also be discussed.

The refined evaluation design memo and enhanced methods appendix (see Task 2 above) will serve at the basis for the PowerPoint slide presentation that the contractor must

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 11

prepare for the TWG. Three weeks prior to the meeting, the contractor must submit to the

COR the following: (1) A set of PowerPoint slides; (2) Any handouts necessary to illustrate the contrasts that will be studied for each experiment; (3) A 3 page evaluation summary abstracted from the refined evaluation design memo and/or the draft forms clearance package (see Task 4); (4) A list of proposed discussion questions that TWG members will be asked to consider (this list should include questions regarding appropriateness of the questions being asked, the contrasts being tested, and the outcomes measures being used). The latter two documents, along with a meeting agenda and logistical information will be distributed to panel members at least one week prior to the meeting. The meeting will take place in ED facilities in Washington, DC and the contractor must cover the travel, per diem, and honorarium expenses of the TWG members.

No more than two weeks after the panel meeting, the contractor shall submit a written summary to the COR, detailing the panel’s recommendations, remaining issues, and the contractor’s proposed actions. The proposed actions shall include a list of any questions or contrasts the TWG members recommended adding or subtracting from the proposed plan and that the contractor plans to consider.

Task 4--Forms Clearance Packages

Once the sample size calculations, participant recruitment strategies, experimental materials and data collection forms presented in the refined study design memo (see Task

2.1 above) have been approved, the contractor must prepare all forms and documents necessary for Office of Management and Budget (OMB) clearance and must coordinate its own Institutional Review Board (IRB) approval.

4.1 OMB Package: The contractor shall prepare an OMB forms clearance package that includes all required forms, calculations of participant burden, and data collection materials and instruments. In addition the package must incorporate: (1) a description of the evaluation’s general purpose, (2) the research questions, (3) a cross walk that illustrates how the information collected as part of each experiment addresses one or more of the research questions, (4) participant recruitment materials, (5) a description of and justification for incentives, (6) limitations of the evaluation and its potential use for policy decisions, and (7) table shells indicating how data will be tabulated and analyzed.

While the preparation of the data collection materials and forms occurs as part of the base contract, data collection will only occur if the option period is executed.

The first draft of the OMB package shall be submitted to the COR no more than two weeks after comments have been received on the refined evaluation design memos (14 weeks after contract award).

4.2 Revised OMB Package: Based on comments received after internal and external review (gathered during the first 60 calendar days of the clearance process) the contractor must make all needed revisions and submit a revised OMB package to the COR. The contractor must be prepared to make one additional round of revisions to forms and

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 12

justification documents as required during latter stage of the clearance process. The contractor must keep in mind that clearance requires a minimum of 120 calendar days and that the schedule of deliverables is designed so that the final revision is submitted prior to the conclusion of the base contract.

4.3 IRB approval: The contractor must coordinate its own Institutional Review Board

(IRB) approval for the data collection, ensuring that all aspects of the evaluation have been approved. Documentation of the contractor’s IRB approval shall be submitted to the COR no later than 34 weeks after the contract has been awarded.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 13

Option Period I Tasks: Experiments and District Guide

Task 1b-- Progress Reports (Fixed Price)

This is a continuation of Task 1a from the base period.

1.2b Monthly Progress Report/Exception Reports: The contractor shall prepare monthly progress reports due within 10 work days after the end of each month. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant events/findings, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the management plan. If there are deviations, the contractor shall describe the plan for resolving the problems and should identify and discuss any decisions which may be needed from ED.

1.3b Monthly Manpower/Expenditure Reports: The contractor shall prepare monthly expenditure reports due within 10 work days after the end of each month. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Task 5--Data collection and Analysis

If Option Period I is executed the contractor will carry out the designed experiments, collect data, and complete data analysis.

5.1 Gather sample/recruit participants and run experiments

The contractor must gather participants and carry out the experiments as described in the plans approved by the COR and described in the OMB package. While opportunistic samples will be used, the contractor must record the number of refusals and to the extent possible, collect basic information about those who decline to participate once invited.

No personally identifiable information will be collected at any point. When data collection is complete, to the extent possible, the contractor shall compare how those who refused to participate compare to those who did participate.

Progress reports that track data collection for each experiment must be provided to the

COR on a bi-monthly basis. These reports must include the number of participants that have been identified or recruited, the number of completes for each experiment, and how many additional completes are needed. Any problems with data collection should be

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 14

noted. Data collection is expected to begin no more than three weeks the option period has been exercised.

5.2 Analyze data and produce tables with findings

The contractor must examine the effects of parent information variants on the measured outcomes. Where appropriate the role of a small number of covariates may be explored.

The contractor must prepare tables showing findings from each experiment and highlighting significance. These tables must be followed by a concise set of bullet points which summarize the main findings from each experiment and which spell out the implications for offering districts guidance.

The findings tables and key findings bullets must be submitted to the COR three weeks after data collection and cleaning are completed. It is expected that data collection will be completed within 10 weeks of the option period being exercised. The data tables and findings document will serve as the basis for the second TWG meeting. The COR shall provide feedback within two weeks. Revisions will be made as discussed in Task 6 below.

Task 6--Technical Working Group (TWG) Focused on Findings and Draft Guide

The contractor must convene one half day meeting of the TWG using teleconferencing technology during the option period. The meeting will focus on discussing experimental findings, interpretation of findings, and take home messages for the district guide. In particular the contractor must seek feedback regarding the feasibility of district level implementation of recommended strategies so that the district guide (see Task 7) can acknowledge potential barriers and difficulties districts may encounter. Three weeks prior to the meeting, the contractor must submit to ED: (1) A revised summary of key findings, data tables and take home messages from Task 5 and (2) A list of proposed discussion questions that TWG members will be asked to consider. The two documents, along with a meeting agenda and logistical information will be distributed to panel members at least one week prior to the meeting. The contractor must cover the honorarium expenses of the TWG members.

No more than two weeks after the panel meeting, the contractor must submit a written summary to the COR, detailing the TWG members’ recommendations for any additional analyses and interpretation of findings, remaining issues, and the contractor’s proposed actions.

Task 7--District Guide

The contractor must author a guide that is evidence based but that is tailored for district level policy makers and practitioners and that provides guidance on the format, organization, and amount of school choice information that is offered to parents. The guide must be largely based on the evaluation’s experiments and findings but can also cite evidence gleaned from the literature where available.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 15

7.1 Draft guide

The contractor must draft a short guide, roughly 40 pages, written in straightforward and nontechnical language, that incorporates examples, illustrations, figures, vignettes, and other visuals as appropriate. Base on feedback from the TWG -- where appropriate -- the guide will also acknowledge potential barriers to implementation and suggest approaches for proactively addressing these barriers. The document must include a technical appendix which provides details regarding the experimental methods and analyses. This appendix should not exceed 15 pages and must be written in such a way that a district level official with an interest in how the experiments were carried out and how the analyses were conducted can understand but with enough detail for the guide to be cleared by the IES

Standards and Review Office (SRO). The practice guide can be designed to be viewed online but it should be possible to produce hardcopy versions of the final guide as well.

The contractor must conduct a thorough quality review and editing of the draft guide prior to submitting to the COR for review. The draft guide should be submitted no more than four weeks following the second TWG meeting. The COR shall review the draft practice guide and provide comments within two weeks. The contractor must make revisions and submit a second draft within four weeks. The COR shall review the second draft within two weeks. It is expected that the next draft will be the final draft discussed in Task 7.2.

7.2 Final guide

The COR will submit the draft guide to SRO for an independent peer review. After the first round of comments are received, the contractor must review them and with COR guidance revise the draft. To document all revisions, the contractor shall draft a memo that links each change made to the guide with the reviewers’ comments. The contractor must re-submit the guide to IES for a second round of SRO peer review. This iterative process occurs until the practice guide is approved by IES for production. Once the guide has been approved by SRO the contractor must prepare a 508 compliant version, a version that can be printed and shared with districts, and a web based version that may include interactive examples. The 508 compliant files must be submitted no more than 102 weeks after the contract has been awarded.

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 16

III. SCHEDULE OF DELIVERABLES

All deliverables must be submitted to the Contracting Officer (CO)/Contract Specialist

(CS) and ED’s Contracting Officer’s Representative (COR) electronically as specified in the delivery schedule. Unless otherwise indicated, the COR shall notify the contractor within 10 business days of receipt of any changes required in the deliverable, and the contractor shall have up to 10 business days to make the necessary revisions and submit a final version. Unless specified in the scope of work, deliverables should require no more than two rounds of editing.

Base Contract (Months 1-10)

Task # Task Description Due Date

1 a Meetings with ED and Progress Reports

1.1a Kick off Meeting Agenda 1 week after award

1.2a Kick off Meeting Summary memo 4 weeks after award

1.3a Monthly progress report Due by the 16th of each month

1.4a Monthly man power report Due by the 16th of each month

2 Refine Evaluation Design

2.1 Draft refined evaluation design report

12 weeks after award

2.2

Enhanced materials appendix 4 weeks before

TWG meeting

2.3 Final evaluation design report

4 weeks after TWG meeting

2.4 Final materials

6 weeks after

OMB approval

3 In Person TWG Meeting

3.1 Bios for recommended participants

3 weeks after award

3.2 Letters of commitment

5 weeks after award

3.3 Questions and materials

3 weeks before

TWG meeting

3.4 Summary memo

2 weeks after TWG meeting

4 Forms Clearance Package

4.1 Draft OMB package

14 weeks after

ATTACHMENT A – PERFORMANCE WORK STATEMENT (DRAFT) 17

4.2 Revised OMB package

24 weeks after award

4.3 Documentation of IRB approval

38 weeks after

Option Period I (Months 11-24)

Task # Task Description Due Date

1b Progress Reports (cont)

1.b Monthly progress report Due by the 16th of each month

1.3b Monthly man power report Due by the 16th of each month

5 Experiments and Analyses

5.1 Data collection progress reports

Due by the 16th and 30th of each month

5.2 Findings tables and bullets

Due 3 weeks after data collection is complete

6 Web Based TWG Meeting

6.1 Questions and materials

3 weeks prior to

2nd TWG meeting

6.2 Summary memo

2 weeks after 2nd

TWG meeting

7 District Guide

7.1 Draft guide

4 weeks after 2nd

TWG meeting

7.2 Final guide

10 weeks after 2nd

TWG meeting

7.3 508 compliant and web ready versions 102 weeks after