Attachment_A_Performance_Work_Statement.pdf

U.S. DEPARTMENT OF EDUCATION

OFFICE OF EDUCATIONAL TECHNOLOGY

RAPID CYCLE TECH EVALUATIONS (RCTE)

PERFORMANCE WORK STATEMENT

August 5, 2015

Summary

The goal of this Rapid-Cycle Technology Evaluation is to evaluate educational software applications purchased with Elementary and Secondary Education Act (ESEA) program funds, including Titles I, II, and III and Individuals with Disabilities Act (IDEA), to help schools, districts, and States make evidence-based decisions.

In this Performance Work Statement (PWS), educational software applications (apps) are defined as applications, platforms, and other tools implemented in an educational setting.

Some examples include vocabulary games, adaptive math platforms, apps that adapt a passage into four different reading levels, research apps, professional development simulations, note-taking apps, college counseling tools, embedded language supports for ELLs, apps that claim to increase creativity or persistence, curriculum sharing platforms, and family engagement apps.

This list is intended to be illustrative and not comprehensive.

This evaluation will help establish a high standard for low-cost, quick turnaround evaluations of apps, establish a technical working group, and field test rapid-cycle technology evaluations to understand how to improve outcomes of ESEA programs. In addition to generating evidence on specific apps, this work will help develop protocol tools for conducting rapid cycle evaluations of apps that practitioners, developers, and researchers can use beyond the scope of this evaluation.

The project shall be divided into two phases: 1) a design phase that includes 3-6 pilot evaluations and 2) field testing using evaluation strategies, designs and tools refined through the pilot evaluation stage. The contract also includes a 24-month option to conduct large-scale rapid-cycle technology evaluations.

The rapid cycle tech evaluations shall be designed to meet What Works Clearinghouse (WWC) standards with or without reservations. Final reports of findings shall be submitted to the WWC for review, and user-friendly summaries of findings shall be disseminated to practitioners. In addition, the research design for conducting rapid cycle technology evaluations shall be public, free and openly licensed under the most current version of the Creative Commons Attribution license (CC BY) and disseminated widely to be used by schools, districts, developers and researchers.

Background

As schools continue to invest heavily in education technology, there is pressing need to generate evidence about the effectiveness of these investments and also to develop evaluation tools that developers and practitioners can use to conduct their own evaluations that are quicker and incur lower costs than traditional evaluations. Additionally, ESEA programs allow

Local Education Agencies (LEAs) to use formula funds for education technology, so the

Department is interested in learning about the effectiveness of apps LEAs invest in with formula funding, including ones such as those aimed at improving student’s academic outcomes and family engagement under Title I, providing professional development tools for teachers under

Title II, and/or improving support for English Learners under Title III.

Multiple factors are impacting the pressing need for these kinds of low-cost, quick turnaround evaluations. In the next two years, tens of thousands of schools are expected to gain high speed Internet access, partly through the ConnectED Initiative, which includes a one-time $2 billion investment by the Federal Communications Commission (FCC), a yearly $1.5 billion increase in the FCC’s E-Rate program, as well as an additional $2 billion in private sector contributions.

The field of educational technology changes rapidly and apps are launched daily; developers often claim that their technologies are effective when there is no high-quality evidence to support these claims. The product evaluations supported by this contract are meant to demonstrate whether certain types of studies – for examples, studies that look at effects on outcomes but do not try to explain the mechanism by which any effect occurred, and/or studies that use administrative data – can be conducted rapidly enough to meet the need of educators for information about effectiveness of technology in this fast-changing landscape.

All of these factors are increasing the need to identify what’s working and what’s not more efficiently and more effectively. Without effective evidence or evaluation methods in place, schools rely heavily on marketing materials and the opinions of a small group of peers leading to inefficient use of limited fiscal and human resources.

Scope of Work

The selection process for the apps to be evaluated shall focus on interventions that can be acquired by grantees under major ESEA programs, including Titles I, II, and III and IDEA. Much of the selected educational software applications will be relevant to additional programs and in many cases will be relevant to more than one. Several examples are listed in the Background section of this document; more examples may be found in the “Dear Colleague Letter: Federal

Funding for Technology.”1 The criteria for selection of apps, sites and evaluators shall be developed by the contractor with input from the Contracting Officer’s Representative (COR) and a technical working group (TWG). All apps must be approved by the COR prior to selection for a rapid-cycle tech evaluation. Similarly, the LEA sites and evaluators for the rapid-cycle tech evaluation must be approved by the COR.

The contractor shall establish a technical working group composed of researchers, K-12 practitioners, and private sector educational technology experts. With input from the COR and the technical working group (TWG), the contractor shall design evaluation tools and training materials for rapid cycle technology evaluations. Evaluation tools may include templates for use in establishing clear expectations for all participants, protocols for best practices, applications (for developers or educators) to participate in study, surveys, checklists, or quality assurance materials. Training materials may include resources for pre-, during and post-study such as self-assessments for participating educators (to indicate readiness for study), technical training, resources for developers on working with schools, and how to interpret study results.

These lists are intended to be illustrative and not comprehensive. While the evaluation of a specific tool is the focus of this work, building capacity among participants is an important expected outcome.

The criteria for selection of the apps, sites and evaluators shall be developed by the contractor with input from the TWG and the COR. All apps must be approved by the COR prior to selection for a rapid-cycle tech evaluation. Similarly, the LEA sites and specific evaluators for the rapid-cycle tech evaluation must be approved by the COR. If appropriate, the contractor shall coordinate with other rapid cycle evaluation work conducted by the Department of Education.

After conducting a set of 3 - 6 pilot evaluations, the contractor shall write short reports

(approximately 20 pages) summarizing the results of the studies and disseminate the early findings to appropriate audiences. These reports should meet WWC standards with or without reservations. Working with the TWG and the COR, the contractor shall refine the evaluation design before conducting field testing on an additional set of 8 - 12 apps. The reports from the field tests shall be submitted to WWC, made publicly available under an open license, and disseminated broadly. If the 24-month option is exercised, the contractor shall conduct large-scale rapid-cycle technology evaluations. The contractor shall create and disseminate these additional findings to the broader community.

The contractor shall also create an interactive guide and implementation support tools for schools, districts, developers, and researchers to conduct rapid cycle tech evaluations.

http://tech.ed.gov/federal-funding-dear-colleague-letter/

The Rapid-Cycle Technology Evaluation project includes two (2) distinct phases with an Option for a third phase:

Task 1: Communication Between Contractor and Department of ED: Develop a communication plan that meets identified requirements. (ongoing)

Task 2: Create and Support a Technical Working Group (TWG): Create and support

TWG that shall provide expert advice to the Department in designing the studies and associated evaluation tools, conducting the tech evaluations, collecting and analyzing the data, and recommending report formats. (ongoing)

Phase 1: Design and Pilot (8 - 9 months)

Task 3: Research Design: Design research approach, create a fair and transparent process for selecting sites and apps, and develop evaluation tools and appropriate training materials. (2 - 4 months)

Task 4: Conduct Pilot Evaluations: Conduct 3 - 6 pilot evaluations. (3 - 4 months)

Task 5: Prepare Reports and Disseminate Findings: Prepare public reports that meet

WWC standards and share developed tools and reports. (2 months + dissemination)

Phase 2: Field Test (8-10 months)

Task 6: Refine Research Design: Review and refine design based on pilots and feedback from TWG and Department. (2 - 3 months)

Task 7: Select Sites and/or Apps for Field Testing. Using revised research design, select field test sites and apps using a fair and transparent process. (1 - 2 months)

Task 8: Field Test: Conduct tech evaluations at 8-12 sites. (3 - 4 months)

Task 9: Prepare Report and Disseminate Findings: Prepare public reports that meet

WWC standards, submit reports to WWC, and share developed evaluation tools and reports broadly under a free and open license. (2 - 3 months)

Task 10: Create an Interactive Guide and Implementation Support Tools for

Conducting Rapid Cycle Tech Evaluations: Create an interactive guide and implementation support tools for conducting rapid cycle tech evaluations to be used by schools, districts, developers and researchers. (6 months)

Note that several tasks will overlap in that work will be done on different tasks simultaneously.

Similarly, Phase 1 and 2 will overlap in that the dissemination of initial findings will likely occur when the selection process begins for the field tests.

Optional Phase 3: Large Scale Rapid Cycle Tech Evaluations (24 months)

Task A: Conduct Rapid Cycle Tech Evaluations: Conduct up to 30 rapid-cycle technology evaluations per year using refined research design developed in Phases 1 and 2. (16 - 20 months)

Subtask A.1: Prepare OMB Clearance Package (2 - 6 months)

Subtask A.2: Review and disseminate key findings to the field. (2 - 3 months)

The contractor shall be responsible for carrying out all phases of the work associated with the major activities under this performance-based contract. Key evaluation questions, along with the research design, outcome measures, implementation fidelity measures, methodology for data collection, timeline, and preliminary data analysis plans shall be developed by the contractor in consultation with the COR during Phase 1 of the Rapid-Cycle Technology

Evaluation project. Phase 1 will begin during the 2015-16 school year. Phase 2 will begin in the

2015-2016 year and will extend into 2016-2017. (See Exhibit A for suggested timelines.)

The period of performance for the base contract shall be sixteen (16) months with an option

(Phase 3) to conduct large-scale rapid cycle tech evaluations in Years 2 and 3. If exercised, this option would overlap the last four (4) months of the sixteen (16) month base contract for a total period of performance of 36 months.

Tasks

The contractor shall provide the services described in the following tasks:

TASK 1 - COMMUNICATION BETWEEN CONTRACTOR AND DEPARTMENT OF ED

Throughout the duration of the contract, the contractor shall maintain effective communication to keep the COR informed by phone, electronic mail, meetings, and other means. The contractor shall meet with the COR and other Department staff once every month. At all meetings, the contractor (and subcontractor staff if requested by the COR) shall brief staff and discuss plans, progress, difficulties and solutions. In addition, the contractor shall submit a weekly progress report every Thursday, and convene a weekly conference call (with subcontractor staff if requested by the COR) to discuss the weekly progress report every Friday.

● Kickoff Meeting. The contractor shall meet with the Department, at ED offices, to kickoff work under the contract within one week of the effective date of this contract.

At the meeting, the contractor and ED shall discuss expectations for the tasks and subtasks comprising the contract and discuss the schedule of deliverables. The contractor shall submit to the COR draft minutes of the meeting. Allowing one week for the COR to comment on the draft minutes, the contractor shall submit to the COR revised minutes that incorporate the COR’s comments.

● Monthly Meetings with ED. The contractor shall meet monthly with the Department via a video conference platform.. If requested by the COR, these monthly meetings shall include subcontractor staff. One week before each meeting, the contractor shall prepare and submit to the COR a draft agenda and any supporting materials for the meeting. If requested by the COR, the contractor shall submit revisions within two days after receiving the COR’s comments. Within one week after each meeting, the contractor shall prepare and submit to the COR a draft summary of approximately one page, covering the main points discussed during the meeting, any changes in plans, and any further actions to be taken, and a list of participants. If requested by the COR, the contractor shall submit a revised summary within two days after receiving the COR’s comments.

● Weekly Updates/Conference Calls. The contractor shall provide the COR with written weekly updates on project status by close of business every Thursday. These weekly reports shall include a bulleted list for each of the following for Tasks 2-4 (1)

Accomplished this week; (2) Planned for next week; and (3) Obstacles/Solutions/Action.

the contractor (and subcontractor staff requested by the COR) shall participate in a weekly conference calls with the COR (and any additional ED staff included by the COR) every Friday to review the weekly report and discuss all obstacles/solutions/action in depth.

● Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due within 10 working days of the end of each month. These reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant activities, findings, and events, problem encountered, and staff used. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive work, and identify and discuss decisions that may be needed from the COR.

If there are no exceptions, the reports shall state this fact. If there are exceptions, the contractor shall describe the proposal for resolving the problems.

Task Deliverables Due Dates

1 Kick-off meeting Within one week of the effective date of this contract

Draft minutes from kickoff meeting 1 week from meeting

Revised minutes from kickoff meeting 3 weeks from meeting

Agenda for quarterly progress meeting 2 weeks before each meeting

Draft summary of monthly progress meeting 1 week after meeting

Revised summary of monthly progress meeting

2 days after receiving COR comments

Weekly Progress Report COB every Thursday

Weekly Progress Update Conference Calls Every Friday

TASK 2 - CREATE AND SUPPORT A TECHNICAL WORKING GROUP (TWG)

The TWG shall be made up of 8-15 people who are educational technology researchers, practitioners and private sector educational technology experts. The purpose of this group shall be to provide expert advice to the Department in designing the studies and associated evaluation tools, conducting the tech evaluations, collecting and analyzing the data, and recommending report formats. They shall also review final reports and recommend changes for future evaluations. The Department will provide a list of potential TWG members within a week of contract award.

Within two weeks after contract award, the contractor shall prepare and provide to the COR a preliminary TWG management plan. In the preliminary TWG management plan, the contractor shall include a list of primary and alternate candidates that the contractor is considering as

TWG members, accompanied by background information on the candidates as well as detailed rationales for their inclusion. The COR and other key Department staff identified by the COR will make the final determination regarding the TWG’s membership. The contractor shall contact and invite the members to serve on the TWG. The contractor shall convene and support TWG meetings and other TWG communication in accordance with the TWG management plan.

In the TWG management plan, the contractor shall also include a schedule for meetings and other TWG operations. The first meeting of the TWG shall occur within two months of contract award. The contractor shall notify the COR if an alternate candidate replaces a member who is no longer available to serve on the TWG, or if topic areas warrant the addition of TWG members with additional areas of expertise. The contractor shall convene four (4) face-to-face meetings in Washington, DC, and four (4) virtual TWG meetings over the base period, for a total of eight (8) TWG meetings.

The contractor shall send information and materials to TWG members for their review and shall consider all input from the TWG and, where appropriate, the contractor shall refine and revise its plans, activities, and recommendations to the Department based on the feedback from the

TWG. At the request of the COR, the contractor shall ensure that its project director and other senior staff members (if approved by the COR) participate in all meetings of the TWG. Each

TWG meeting shall be conducted for one-day only, and will run from early morning to late afternoon.

The contractor shall prepare briefing materials to be sent to the TWG at least one week prior to each meeting. The contractor shall include at least the following in the briefing materials: the agenda, status reports, background information on issues to be discussed, and any draft reports to be discussed at the meeting. The contractor shall submit the draft briefing book to the COR three weeks prior to each meeting. After a three-day review by the COR and other key

Department staff identified by the COR, the contractor shall revise the briefing materials incorporating all comments received from the COR and send them to all participants so that they receive it one week before each scheduled meeting.

The contractor shall prepare and submit to the COR summary minutes of the TWG meetings one week after they take place. After a one-week review by the COR, the contractor shall revise the minutes based on COR comments and submit a final copy to the COR no later than two weeks after the TWG meetings.

The contractor shall pay all expenses for the TWG, including travel, honoraria ($500 per TWG member per meeting), communication, meetings and other expenses necessary to fulfill its obligations under this option. The Department anticipates that each of the 8-15 TWG members shall invest a total of approximately 10 days over the 16 months including time spent reviewing drafts and materials, time in TWG meetings, and preparation time. The contractor shall provide administrative support necessary for the TWG to fulfill its obligations under this contract. The contractor shall also ensure that proposed TWG members do not have conflicts of interest in performing the work under this contract.

2 TWG Management Plan Within two weeks of the effective date of this contract

Final draft TWG Management Plan Within 1 week after receiving comments on first draft from ED

Draft agenda for TWG meetings 3 weeks before each meeting

Revised agenda 1 week before each meeting

Draft briefing materials 3 weeks before each meeting

Revised briefing materials 1 week before each meeting

Draft summary of meeting 1 week after each meeting

Revised summary of meeting 2 weeks after each meeting

TASK 3 - RESEARCH DESIGN

The contractor shall develop a research design for conducting rapid cycle tech evaluations.

Subtask 3.1 - Work Plan

The contractor shall develop a detailed work plan for this task that includes (at a minimum):

● A set of design options, including sample sizes (based on statistical power analysis), timelines, appropriate outcome measures, measures of implementation fidelity, and cost for each proposed design.

● Preliminary schedule with tasks, milestone, decision points and deliverables.

● Management and staffing plans, with key personnel, as well as a high-level calendar of key activities, including all subcontracts.

● Quality Assurance Plan, including the contractor’s oversight of subcontractors and key decision points for seeking ED input.

● Draft list of proposed pilot sites, including alternates, and basis for selection. This shall include a brief description of the sites, including geographic and demographic information, any information that may be known about academic performance, and other pertinent information that provides useful background information on the sites.

● Recruitment strategy for sites, participating organizations, and any individual participants.

● Plan for fair and transparent selection process for apps to be evaluated.

● Process for determining research question and for each site.

● Process and timetable for developing evaluation tools and training materials.

● Estimated costs/budget.

● Resources needed from other project partners and subcontractors.

The work plan shall be submitted to the COR in draft form for review. After receiving feedback on the draft, the work plan shall be revised and submitted as final.

Subtask 3.2 - Research Design for Pilot Studies

The contractor shall develop a research design for conducting rapid cycle tech evaluations. The

TWG shall play a central role in identifying the approach and creating the research design. The research design shall be approved by the COR.

Site and App Selection Process: Approaches to consider for identifying the pilot sites and apps could be focusing on the effectiveness of apps on a specific population, such as English

Language Learners (ELL) or special education; choosing a specific genre of apps, such as math or professional development; or selecting sites based on readiness for this kind of work. The selection process for identifying sites and apps for the pilot evaluations may differ from the selection process for the field tests, recognizing that a priority should be placed on conducting pilots in scenarios most likely to yield useful data and information about the research design itself.

Evaluation Tools: The contractor shall create all evaluation tools necessary for an effective pilot evaluation. Evaluation tools may include, for examples, recruitment templates, applications (for developers or educators) to participate in study, templates for use in establishing clear expectations for all participants, protocols for best practices, resources for developing effective research questions, surveys, checklists, and quality assurance materials. All evaluation tools shall be written in non-technical language.

Training Materials: All training materials necessary for a successful rapid cycle tech evaluation shall be created by the contractor. Training materials may include resources for pre-, during and post-study such as self-assessments for participating educators (to indicate readiness for study), technical training, resources for developers on working with schools, and resources for educators on working with researchers. While the contractor is not expected to design training materials for using the selected apps, the contractor shall include in the research design steps to ensure that the developers provide the educators with effective training on how to use the apps. One evaluation tool that could be created is a checklist template for developers on training resources that should be available during rapid cycle app evaluations.

OMB Clearance: The contractor shall design the pilots so that Office of Management and

Budget (OMB) clearance is not necessary and that the pilots do not violate the Paperwork

Reduction Act and 5 CFR 1320.

Priority should be given to creating research designs that are easily replicable. While the evaluation of specific tools is a key focus of this work, building capacity among participants is an important expected outcome.

3 Draft of Research Design

Management Plan

Within two weeks of the effective date of this contract

Final draft Research Design

Management Plan

Within 1 week after receiving comments on first draft from ED

TASK 4 - CONDUCT PILOT EVALUATIONS (3-6 sites)

In consultation with the COR and the TWG, the contractor shall conduct rapid-cycle technology evaluations at schools and districts using the research design developed in Phases 1-3. All sites, apps, and researchers/evaluators must be approved by the COR.

Based on the approved Research Design (Task 3), the contractor shall submit a proposed list of evaluation sites, including alternates, to the COR. The contractor shall include as part of the list a brief description of the sites, including demographic information, any information that may be known about the academic performance of the school districts and schools at the site, and other information that provides useful background information on the sites including information about existing regional partnerships. Priority for the pilot sites should be given to sites within an existing collaborative regional education innovation cluster; i.e., sites connected to an organization or network with developing collaboration among educators, entrepreneurs, policymakers, and/or research institutions. The description of the sites should include details about the existing technology infrastructure, i.e., level of Internet access, existing tech platforms, ratio of mobile devices to students, and other information that would indicate readiness and ability to match with specific apps.

The COR and key Department staff identified by the COR will review the list and provide comments. Based on the COR’s comments and the COR’s approval, the contractor shall notify the sites of their selection and seek their participation. The contractor shall submit to the COR a final list of participating sites.

The plan for conducting tech evaluations at the identified sites shall be developed as part of

Task 3. The contractor is responsible for overseeing and ensuring the quality of the rapid cycle tech evaluations including work conducted by subcontractors. Each pilot evaluation shall run approximately 1 to 3 months, depending on the app studied.

Once sites have been selected, the contractor shall create a specific management plan for each site and app using the protocols developed by the contractor. Specifically these plans should include the following:

● Timetable and milestones for app evaluation(s)

● Criteria and identification process for participating educators at selected site

● Criteria and selection process for app, tool, or platform to be evaluated

● Initial launch team meeting – educators, entrepreneur/developer, researcher/evaluator

● Plan for PD for participants in tech evaluation, including development of PD resources

● Ongoing feedback process - how will information on what’s working and not working in the evaluation process itself be collected

Research Question: The research question is key to each specific tech evaluation and thus shall include input from multiple stakeholders. The process for finalizing the research question to be measured shall include review by the COR and TWG members.

Training Materials: The contractor, with input from the COR and the TWG, shall design appropriate professional development resources for participants (educators, developers, researchers and students/families if appropriate) around conducting successful tech evaluations. These materials shall be revised based on initial feedback from participants, the

TWG, and the COR in preparation for use in future tech evaluations. For example, educators may be provided with resources that explain best practices for working with researchers as well as training materials provided by the app provider.

Data Collection Activities: For all data collection activities, the contractor shall examine the data for completeness and consistency, communicating with respondents as needed to obtain complete data. The contractor shall develop coding materials for entering the data collected, if appropriate, and prepare the data for analysis as data are received. To ensure accuracy, the contractor shall verify all key data entered, conduct editing and consistency checks, and track response rates, if applicable.

The contractor shall notify the COR of the beginning and end of data collection activities on the dates specified in the approved Research Design (Task 3). During the data collection period, the contractor shall provide the COR with weekly updates on response rates when applicable.

The contractor shall design the pilots so that they do not violate the Paperwork Reduction Act and 5 CFR 1320 to avoid the process of OMB clearance.

4 Draft list of selected sites On the date identified in the approved Management Plan

Final list of selected sites 1 day after review and approval by ED of draft list of selected sites

Draft list of selected apps On the date identified in the approved Management Plan

Final list of selected apps 1 day after review and approval by ED of draft list of selected apps

Notify the COR of beginning of data collection

On date specified in the approved Research Design Plan

(Task 3)

Notify the COR of end of data collection

On date specified in the approved Research Design Plan

(Task 3)

Weekly updates on data collection

Weekly during data collection

TASK 5 - Prepare Short Reports and Disseminate Findings

Each short report that includes findings from an effectiveness study shall include information needed by the WWC to conduct its review, shall be submitted to the WWC, and disseminated to the public for practitioner use. In addition, the research design for conducting rapid cycle technology evaluations shall be public, free and openly licensed under the most current version of the Creative Commons Attribution license (CC BY) and disseminated widely to be used by schools, districts, developers and researchers.

After conferring with the COR, TWG, and key Department staff identified by the COR on the most appropriate analyses for the data collections, the contractor shall prepare a detailed outline for the format of each report. These outlines shall provide the COR with information on the structure, content, and analysis. Report structures may be modified for different kinds of apps and types of research questions. The contractor shall be responsible for the final reports.

The COR and key Department staff identified by the COR will review and provide comments on the first set of draft reports. Within two weeks of receiving comments from the COR on the first draft, the contractor shall revise the reports incorporating all comments received from

Department staff. The COR and key Department staff identified by the COR will review and provide comments on the second draft reports. Within one week of receiving comments from the COR on the second drafts, the contractor shall revise the set of reports incorporating all comments received from Department staff. If necessary, the COR and key Department staff identified by the COR will review and provide comments on the set of draft reports a third time.

Within one week of receiving comments from the COR, the contractor shall revise the set of short reports incorporating all comments received from Department staff and submit a final draft to COR. A separate written summary of how all Department comments were addressed shall be submitted with the second, third and final drafts of the report.

In these short reports, the contractor shall address the evaluation questions agreed on by the

COR and key Department staff identified by the COR as well as the TWG in any subsequent meetings or correspondence.

After the tech evaluation reports are completed, the contractor shall draft a non-technical, stand-alone executive summary not to exceed 10 pages, summarizing the key findings across the set of short reports. This summary shall include context, process, lessons learned, and preliminary recommendations for future rapid cycle tech evaluations, if appropriate.

The contractor shall write the executive summary in a manner suitable for distribution to a broad audience, using plain, non-technical language, and that follows the guidelines in the

Department report, Guide to Publishing at the U.S. Department of Education

(http://www.ed.gov/internal/PubGuide.pdf).

The contractor shall not present findings from evaluation reports or data tabulations that have not been reviewed and released by the Department. Prior to final clearance by the

Department, the contractor shall present only methodology at any conferences or other public presentations.

The contractor shall disseminate project resources via tech.ed.gov, outreach to Title I, II and III directors, through existing professional learning networks, and other ED technical assistance efforts such as the Regional Education Laboratories, Race to the Top, Teacher Incentive Fund, Statewide Longitudinal Data System and Small Business Innovation Research programs. In addition, the contractor shall partner with external stakeholders to develop and disseminate supporting materials and resources. Information may be disseminated in non-traditional formats such as infographics, short videos, blog posts and/or a high profile event that highlights the successes of teams doing the rapid cycle tech evaluation work. These are suggestions and do not represent a comprehensive list.

http://www.ed.gov/internal/PubGuide.pdf

The contractor shall develop a detailed work plan for this task that includes the work requirements identified above, and shall suggest additional options for dissemination. The work plan shall be submitted to the COR, in draft form for review. After receiving feedback on the draft, the work plan shall be revised and submitted as final.

5 Draft report outline On date specified in the approved Research Design Plan

(Task 3)

Revised report outline Two weeks after receiving comments from ED on the draft report outline

First draft report Two weeks after receiving comments from ED and the TWG on the revised report outline

Second draft report Two weeks after receiving comments from ED on the first draft of the report

Third draft report One week after receiving comments from ED on the second draft of the report

Executive summary draft Two weeks after final tech evaluation reports are completed

Executive summary second draft

One week after receiving comments from ED on the first draft

Final executive summary One week after receiving comments from ED on the second draft

PHASE 2: FIELD TEST

The contractor shall provide the services described in the following Phase 2 Tasks 6 through 10.

TASK 6 - REFINE RESEARCH DESIGN

Based on the results of the pilots and input from TWG and the COR, the contractor shall refine the research design including revising all associated resources. Additional resources are likely to need to be created by the contractor based on needs identified during the pilots.

The revised Research Design must be approved by the COR before conducting field tests.

Optional Subtask 6.1: Prepare OMB Clearance Package

If the revised research design makes OMB clearance necessary, the contractor shall prepare the necessary forms required for OMB clearance for the evaluation specified in the approved

Research Design Plan (Task 5) under procedures of the Paperwork Reduction Act and 5 CFR

1320. The clearance package must justify the necessity for collecting the data and comprehensively respond to each required item in the instructions for submitting OMB package. The COR will provide guidelines and other information on completing a package to the contractor, as necessary. In general, the package shall include (each separated in different file documents): 1) the IC Data Forms (Parts I and 2); 2) a supporting statement with Parts A and

B in separate file documents; 3) a copy of the statute that authorizes the collection of information; 4) regulations applicable to the collection; 5) the instruments which needs OMB approval; and 6) notification materials. The contractor shall devote sufficient time and resources to this product to ensure a timely clearance since the conduct of the data collection depends on obtaining OMB clearance. As required by the Department, the notification materials shall include notification letters for the state information technology officers, superintendents of the selected school districts, and principals and teachers of the selected schools. In the notification materials, the contractor shall include information on topics such as:

general information on the data collection as well as specific information on schedule and plans; a discussion of the importance of the data collection, its purposes, products, scheduled data collection and sample; provisions for maintaining anonymity of survey participants and data security; the organizations and persons involved in the data collection; and the benefits to be derived from the data collection. Notification materials also shall include any other information to be sent to recipients of the notification letters.

The contractor shall submit the first draft of the OMB package to the COR. Following comment from the COR and key Department staff identified by the COR on the first draft, the contractor shall prepare a final draft of the OMB package within one week incorporating all comments received from the COR.

The contractor shall schedule three weeks to five months (depending on kind of clearance necessary) for review of the clearance package by the Department’s Regulatory Information

Management Services (RIMS) and by OMB prior to OMB approval. RIMS or OMB may require revisions to parts of the clearance package prior to approval. The contractor shall make the required revisions and respond to questions from OMB and the public upon request and submit the revised materials to the COR. The contractor shall, if necessary, meet with the COR and

OMB staff to discuss the clearance package and its revisions and provide other support for the clearance process.

Subtask Deliverables Due Dates

6.1 First draft OMB package Three weeks after revised Research Design is approved

Final draft OMB package Within one week of receiving comments from ED

Memo on pilot testing results During the first public comment period, if applicable

TASK 7 - SELECT SITES AND APPS FOR FIELD TESTING

Using the criteria and process developed in the revised Research Design (Task 6), the contractor shall select the sites and apps for the set of 8 to 12 field tests using a fair and transparent process.

7 Draft list of selected sites On the date identified in the approved Management Plan

Final list of selected sites 1 day after review and approval by ED of draft list of selected sites

Draft list of selected apps On the date identified in the approved Management Plan

Final list of selected apps 1 day after review and approval by ED of draft list of selected apps

TASK 8 - FIELD TEST SELECTED APPS

In consultation with the COR and the TWG, the contractor shall conduct rapid-cycle technology evaluations at schools and districts using the research design developed in Task 3 and refined during Task 6. All sites, apps, and researchers/evaluators must be approved by the COR.

Based on the approved revised Research Design (Task 6), the contractor shall submit a proposed list of evaluation sites, including alternates, to the COR. The contractor shall include as part of the list a brief description of the sites, including demographic information, any information that may be known about the academic performance of the school districts and schools at the site, and other information that provides useful background information on the sites including information about existing regional partnerships. The description of the sites should include details about the existing technology infrastructure, i.e., level of Internet access, existing tech platforms, ratio of mobile devices to students, and other information that would indicate readiness and ability to match to specific apps.

The COR and key Department staff identified by the COR will review the list and provide comments. Based on the COR’s comments and approval, the contractor shall notify the sites of their selection and seek their participation. The contractor shall submit to the COR a final list of participating sites.

The plan for conducting tech evaluations at the identified sites should be developed as part of

Task 3 and refined during Task 6. The contractor is responsible for overseeing and ensuring the quality of the rapid cycle tech evaluations including work conducted by subcontractors. Each pilot evaluation shall run approximately 1 to 4 months, depending on the app studied.

Once sites have been selected, the contractor shall create a specific management plan for each site and app using the protocols developed by the contractor. Specifically these plans should include the following:

● Timetable and milestones for app evaluation(s)

● Criteria and identification process for participating educators at selected site

● Criteria and selection process for app, tool, or platform to be evaluated

● Initial launch team meeting – educators, entrepreneur/developer, researcher/evaluator

● Plan for professional development for participants in tech evaluation, including development of supporting resources

● Ongoing feedback process - how will information on what’s working and not working in the evaluation process itself be collected

Research Question: The research question is key to each specific tech evaluation and thus shall include input from multiple stakeholders. The process for finalizing the research question to be measured shall include review by the COR and TWG members.

Training Materials: The contractor, with input from the COR and the TWG, shall design and prepare appropriate professional development resources for participants (educators, developers, researchers and students/families if appropriate) around conducting successful tech evaluations. These materials shall be revised based on initial feedback from participants, the TWG, and the Department in preparation for use in future tech evaluations. For example, educators may be provided with resources that explain best practices for working with researchers as well as training materials provided by the app provider.

Data Collection Activities: For all data collection activities, the contractor shall examine the data for completeness and consistency, communicating with respondents as needed to obtain complete data. The contractor shall develop coding materials for entering the data collected, if appropriate, and prepare the data for analysis as data are received. To ensure accuracy, the contractor shall verify all key data entered, conduct editing and consistency checks, and track response rates, if applicable.

The contractor shall notify the COR of the beginning and end of data collection activities on the dates specified in the approved Research Design (Task 3). During the data collection period, the contractor shall provide the COR with weekly updates on response rates when applicable.

8 Notify the COR of beginning of data collection

On date specified in the approved Research Design Plan

(Task 3)

Notify the COR of end of data collection

On date specified in the approved Research Design Plan

(Task 3)

Weekly updates on data collection

Weekly during data collection

TASK 9 - REPORT AND DISSEMINATE FINDINGS

In addition to preparing individual short reports for each rapid cycle tech evaluation, the contractor shall create a longer summary report of collective findings and shall create a how-to guide for conducting rapid cycle tech evaluations. All reports shall be public, free and openly licensed under the most current version of the Creative Commons Attribution license (CC BY, see https://creativecommons.org/licenses/) and disseminated widely to be used by schools, districts, developers and researchers.

Short Reports: Each short report (approximately 20 pages) that includes findings from an effectiveness study shall include information needed by the What Works Clearinghouse (WWC) to conduct its review, shall be submitted to the WWC, and disseminated to the public for practitioner use. In addition, the research design for conducting rapid cycle technology evaluations shall be public, free and openly licensed under the most current version of the CC

BY and disseminated widely to be used by schools, districts, developers and researchers.

After conferring with the COR, TWG, and key Department staff identified by the COR on the most appropriate analyses for the data collections, the contractor shall prepare a detailed outline for the format of each report. These outlines shall provide the COR with information on the structure, content, and analysis. Report structures may be modified for different kinds of apps and types of research questions. The contractor shall be responsible for the final reports.

The COR and key Department staff identified by the COR will review and provide comments on the first set of draft reports. Within two weeks of receiving comments from the COR on the first draft, the contractor shall revise the reports incorporating all comments received from the

COR. The COR and key Department staff identified by the COR will review and provide comments on the second draft reports. Within one week of receiving comments from the COR on the second drafts, the contractor shall revise the set of reports incorporating all comments received from Department staff. If necessary, the COR and key Department staff identified by the COR will review and provide comments on the set of draft reports a third time. Within one week of receiving comments from the COR, the contractor shall revise the set of short reports incorporating all comments received from Department staff and submit a final draft to COR. A separate written summary of how all Department comments were addressed shall be submitted with the second, third and final drafts of the report.

In these short reports, the contractor shall address the evaluation questions agreed on by the

COR and key Department staff identified by the COR as well as the TWG in any subsequent meetings or correspondence.

Executive Summary: After the tech evaluation reports are completed, the contractor shall draft a non-technical, stand-alone executive summary not to exceed 20 pages, summarizing the key findings across the set of short reports. This summary shall include context, process, lessons learned, and preliminary recommendations for future rapid cycle tech evaluations, if appropriate.

The contractor shall write the executive summary in a manner suitable for distribution to a broad audience, using plain, non-technical language, and that follows the guidelines in the

Department report, Guide to Publishing at the U.S. Department of Education

The contractor shall not present findings from evaluation reports or data tabulations that have not been reviewed and released by the Department. Prior to final clearance by the

Department, the contractor shall present only methodology at any conferences or other public presentations.

Dissemination: The contractor shall disseminate project resources via tech.ed.gov, outreach to

Title I, II and III directors, through existing professional learning networks, and other ED technical assistance efforts such as the Regional Education Laboratories, Race to the Top, Teacher Incentive Fund, Statewide Longitudinal Data System and Small Business Innovation

Research programs. In addition, the contractor shall partner with external stakeholders to develop and disseminate supporting materials and resources. Information may be disseminated in non-traditional formats such as infographics, short videos, blog posts and/or a high profile event that highlights the successes of teams doing the rapid cycle tech evaluation work. These are suggestions and do not represent a comprehensive list.

Work Plan: The contractor shall develop a detailed work plan for this task that includes the work requirements identified above, and shall suggest additional options for dissemination. The work plan shall be submitted to the COR, in draft form for review. After receiving feedback on the draft, the work plan shall be revised and submitted as final within one week of receiving comment from the COR.

9 Draft report outline On date specified in the approved Research Design Plan (Task 6)

Revised report outline Two weeks after receiving comments from ED on the draft report outline

First draft report Two weeks after receiving comments from ED and the TWG on the revised report outline

Second draft report Two weeks after receiving comments from ED on the first draft of the report

Third draft report One week after receiving comments from ED on the second draft of the report

Executive summary draft Two weeks after final tech evaluation reports are completed

Executive summary second draft

One week after receiving comments from ED on the first draft

Final executive summary One week after receiving comments from ED on the second draft

TASK 10 - CREATE AN INTERACTIVE GUIDE AND IMPLEMENTATION SUPPORT TOOL FOR

CONDUCTING RAPID CYCLE TECH…