Attachment__A-DRAFT_Performance_Work_Statement_(REL_Peer_Review).pdf

Performance Work Statement (April 2017)

DRAFT PERFORMANCE WORK STATEMENT

INDEPENDENT REVIEW AND TECHNICAL AND LOGISTICAL SUPPORT FOR REGIONAL EDUCATIONAL LABORATORIES

I. Introduction

The National Center for Education Evaluation and Regional Assistance (NCEE) is required to provide a system for technical and peer review of applied research activities, research-based reports, and products produced by the ten Regional Educational Laboratories (RELs). This contract will include the refinement of existing review criteria, thorough review of REL products, editing and production of REL products prior to release by IES, and creation and ongoing maintenance of a web-based application in the Institute of Education Science’s (IES) Data Center to facilitate this process. The contract will also develop as-needed analytic and methods oriented projects. In addition, the contract will provide support to the REL program in the following ways: maintenance of the REL Intranet, development of a portal for administration of the REL Stakeholder Feedback Survey, logistical support for biannual REL Directors meetings and occasional webinars for the REL contractors, and production of an updated REL Program brochure.

II. Background

The REL program was established under the Elementary and Secondary Education Act (ESEA) of 1965.

Goals set for the RELs have changed throughout its history. In the 2006-2011 contracts, RELs became “emissaries of science,” providing policymakers and practitioners with training and technical assistance to inform effective decision-making based on the highest-quality evidence. RELs were also expected to engage in high quality, rigorous research to address the effectiveness of programs, policies, and/or practices intended to improve educational outcomes. In the 2012-2017 contracts, the RELs maintained the focus on rigorous and scientifically valid evidence while making the processes and products of these efforts more relevant and accessible to practitioners and policymakers across the country. The 2012- 2017 REL program sought to address the disconnect between researchers and practitioners through researcher-practitioner partnerships known as research alliances. The premise of this approach was that partnerships of researchers, practitioners, and policymakers would result in high quality research studies and products that answered pressing practical questions, communicated findings in plain-spoken but technically precise ways, were engaging and actionable for practitioners, and — most important — built the capacity in states and districts to apply research results.

The current REL program — the 2017-2022 REL contracts — builds on and extend its recent history of high-quality research, genuine partnership, and effective communication. The 2017-2022 RELs are tasked with conducting rigorous research, working in partnership with state and district staff, and effectively communicating about research. In addition, the Department expects the RELs to apply these capabilities to high-leverage education problems in their regions. Focusing on high-leverage problems increases the likelihood that REL support ultimately will contribute to improved student outcomes. High-leverage problems are those that:

1. if addressed could result in substantial improvements in education outcomes for many students or for key subgroups of students;

2. are priorities for regional policymakers, particularly at the state level; and

3. require research or research-related support to address well.

This contract will support the REL program in achieving its goals by providing independent external review of REL products and by providing technical and logistical support.

III. Legislative Authority

The current authorization for the Regional Educational Laboratories program is under the Education Sciences Reform Act (ESRA) of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES) National Center for Education Evaluation and Regional Assistance (NCEE) (go to <Authorizing Legislation> to download a PDF file). Under Section 174(5) Standards, the legislation authorizes the establishment of a process for technical quality and peer review for NCEE products and those produced by the RELs:

The Evaluation and Regional Assistance Commissioner shall establish a system for technical and peer review to ensure that applied research activities, research-based reports, and products of the regional educational laboratories are consistent with the research standards described in section 134 and the evaluation standards adhered to pursuant to Section 173(a)(2)(A)(B).

IV. Contract Type

The contract type is a hybrid contract consisting of Firm-Fixed Price (FFP) and Cost Reimbursement (CR) tasks.

V. Period of Performance

The period of performance for this contract is 60 months and consists of a Base Period and three Option Years whose durations are as follows:

• Base Period: 24 months

• Option Year 1: 12 months

• Option Year 2: 12 months

• Option Year 3: 12 months

VI. Scope of Work

In accordance with the purpose of this contract, the objectives are as follows:

• Review REL research proposals and products to assure alignment with NCEE standards and goals.

o Refine and improve the existing criteria for reviews that can be shared with NCEE and REL staff.

o Review analytical technical support products and tools produced by RELs (REL Task 5).

o Review materials related to research and evaluation projects (REL Task 6), including:

project proposals, OMB Clearance packages, research reports, and restricted use data files and documentation.

o Review materials related to systematic evidence reviews (REL Task 6), including: project proposals and systematic evidence review reports.

o Create an application in the IES Data Center to manage the review and tracking of REL work submitted for review.

http://www2.ed.gov/policy/rschstat/leg/PL107-279.pdf o Provide technical editing, formatting, and production of reports.

o Update the “Writers’ and Style Guide” and REL report templates.

o Provide as-needed guidance for improvement of REL products.

• Develop as-needed analytic and methods oriented projects (optional task).

• Provide general analytic and technical support to the RELs and the IES REL staff.

o Maintain the REL Intranet.

o Develop a portal in the IES Data Center for administration of the REL Stakeholder

Feedback Survey.

o Provide logistical support for REL Directors meetings in Washington, D.C.

o Provide technical and logistical support for webinars for REL staff (optional task).

o Update the REL Program brochure and create REL posters.

VII. Task Requirements

This section specifies the tasks and subtasks that the contractor shall perform.

Task 1: Communication with IES

1.1 Kick-off Meeting (Fixed Price, Non-IT)

Within 1 week of award, the Contractor shall participate in an opening kick-off meeting with the COR, contract specialist (CS), and NCEE staff to discuss plans for the activities to be performed soon after the award date. Within 1 week of this meeting, the contractor shall prepare a short (3-5 page) summary memo on the key issues addressed and follow up recommendations.

After Year 1 of the contract, the kick-off meeting will be replaced with an annual meeting to discuss the work plan for the next year.

Deliverable Summary of kick-off meeting

1.2 Periodic telephone calls with COR (Cost reimbursement, Non-IT)

The contractor shall participate in regular telephone calls with the COR to discuss issues related to the review contract, including any anticipated problems before they occur and proposed solutions to those problems. During the base period of the contract, these telephone calls shall occur every 2 weeks. This will continue during the option periods should they be exercised.

1.3 Monthly Progress/Status Reports (Fixed Price, Non-IT)

The contractor shall submit monthly reports containing the following: information on the progress on each task and activity, including accomplishments and problems that have been resolved or are in need of resolution; information concerning expenditures and other relevant budget related details; and an Excel file that describes IT spending by project in the format specified in the solicitation. These reports shall be submitted by the 7th business day of each month.

Deliverable:

Monthly progress reports

1.4 Annual Program Assessment Information (Fixed Price, IT)

Within 4 weeks of contract award and the start of each option year, the contractor shall submit a work breakdown structure of all IT deliverables, down to the individual personnel level, in Microsoft Project format; a 2-3 page overview of the quality management plan for IT deliverables; and a 2-3 page overview of the risk management plan for IT deliverables. Additional guidance for these documents are provided in the solicitation.

Deliverables:

Work breakdown structure of IT deliverables Quality management plan for IT deliverables Risk management plan for IT deliverables

Task 2: Refine, Improve, and Implement Review Criteria and Processes for REL Work (Cost reimbursement, Non-IT)

2.1 Refine the Existing Criteria for Review

The contractor shall refine the existing review criteria and procedures for each type of REL product. In order to accomplish this task, the contractor shall have access to existing NCEE materials and shall review them to identify which can be improved/refined and areas in which new materials may need to be developed. Review criteria shall be written in a fashion that will make them user-friendly for REL staff and shall address relevance, technical quality, readability, and utility (see below for more information on these review categories). The review criteria shall take into account all existing IES and NCEE standards.

The contractor shall have four weeks from the contract award date to submit a draft set of refined review criteria documents and procedures. The COR will provide feedback on these documents within two weeks. The contractor will have four more weeks to submit a final version of review criteria and procedures.

Deliverables Review criteria and procedures (draft and final)

2.2 Review Research Proposals and Products

The contractor will review project proposals for descriptive data analyses, technical assistance or research guidance documents and tools, systematic evidence reviews, and impact evaluations. The contractor shall also review the products that result from these proposals, as well as OMB Clearance packages necessary to carry out the research, and restricted use data files and documentation resulting from REL work and intended to be made available to approved researchers. Over the five-year period of this contract, NCEE anticipates up to 450 proposals and reports for descriptive data analyses and technical assistance or research guidance documents and up to 35 proposals and reports for impact evaluations and systematic reviews. For each year of the contract, NCEE anticipates up to 150 proposals and reports for descriptive data analyses and technical assistance or research guidance documents and up to 15 proposals and reports for impact evaluations and systematic reviews.

The contractor shall identify a team or teams to be led by senior-level policymakers, practitioners and researchers who can evaluate the relevance, technical quality, readability, and usability for all REL https://ies.ed.gov/ncee/edlabs/relresources/pdf/NCEE_Guidance_for_REL_Products_042013.pdf research proposals and products. This will include adherence to all IES standards. The categories for review are described below.

Review teams must have expertise relevant to the particular REL products being evaluated. The name and qualifications (in the form of a short biographical summary) of each reviewer for any given product must be presented to the COR within two business days of receiving the proposal or product to be assessed. The COR will agree to the reviewers or request changes within two days. Should new reviewers be requested by COR, a revised set of reviewers must be provided by the contractor within three business days.

The contractor shall submit to COR written review comments for each proposal or product within two weeks of receipt of the proposal or product. In most cases, the contractor will be asked to review proposals or products two to three times as they are revised and improved by REL staff. The contractor must have a system for tracking the issues from previous reviews that were successfully addressed and those that require additional attention. The review shall address the criteria described below.

Comments related to a proposal or product shall be presented to COR through careful use of comments in Microsoft Word, as well a detailed memo providing overall comments related to broader issues of strengths or weaknesses in a document. All comments and memos shall be written constructively with the aim of helping authors correct or address any deficiencies in the REL product while also helping to build the knowledge and capacity of the author to prevent similar errors in the future. Reviews shall include recommendations about whether or not a proposal or product should move forward to implementation or publication.

Review Criteria

Relevance The contractor shall provide written reviews on the relevance of project proposals and products. For project proposals, the review memo shall assess the extent to which the proposal has the potential to provide actionable information to a particular audience, the extent to which similar information is or is not already accessible, and the degree to which any potential findings will be worth the data gathering burden imposed on any public entities. The review memo shall also consider the extent to which a rationale has been presented for the work including through the provision of evidence from a needs assessment or input from the partnerships of which the REL is a part.

For reports or products, the section of the review memo dealing with relevance shall assess the extent to which the evidence provided is actionable, the degree to which the presentation of research findings is appropriate for a practitioner/policymaker audience, and whether time consumed reading the research product is balanced by potential benefit or learning.

All review memos provided by the contractor shall also include recommendations for strengthening the research proposal or product, highlight any ambiguities in the proposal or product, and recommend whether or not the proposal or product should move toward execution or publication.

Individuals charged with reviewing the relevance of REL proposals and products must be individuals who would be likely to use REL products; such individuals can include current and former school district leaders, policymakers and program directors from the state and district levels, classroom teachers with more than ten years of experience, leaders in the education industry, and leaders of not-for-profit organizations focused on education advocacy, program provision, and support.

Technical Quality The contractor shall provide written reviews on the technical quality of research proposals and products. The written reviews shall provide feedback on the extent to which REL proposals and products meet the standards and expectations of IES and NCEE related to research design, data collection, data analysis, and reporting of findings. For project proposals, the review memo shall assess the strengths and weaknesses of the proposal and the feasibility of completing the proposed work in the timeline proposed. This will include review of the technical approach used to meet NCEE standards, sampling and statistical inference methods, data sources or outcome measures to be used, and analysis plans.

Depending on the scope and content of the research proposal, review will also include assessment of the proposed methodology, recruitment of study subjects, issues related to sample size and minimum detectable effects, instruments to be used, strategies for dealing with attrition and missing data, and OMB and IRB approval processes. The review will also provide recommendations for strengthening any of these aspects related to the design or analysis proposed in project proposals. All proposal review memos provided by the contractor shall also include recommendations for strengthening the research, highlight any ambiguities in the proposal, and recommend whether or not the proposal move forward.

The contractor shall also review materials that are prepared for submission to the Office of Management and Budget (OMB) in order to secure clearance for information collection. Review of these materials should help REL teams to address all foreseeable questions from OMB and provide guidance about the most effective and efficient way to prepare a submission that meets all relevant guidelines.

The contractor can anticipate a maximum of 20 OMB submissions per year for the first three years of the REL contracts.

In addition, the contractor will be responsible for reviewing the quality and accuracy of the data documentation and data analysis plans related to restricted use data files. These data files are prepared at the conclusion of impact evaluation studies, with the intention of making the data available to researchers for additional analyses. Review of these materials must meet ED standards related to the protection of data. The contractor can anticipate a maximum of 10 sets of materials related to restricted use files during each of the last three years of the REL contracts.

Individuals charged with reviewing the technical quality of REL proposals and products must be senior level researchers or analysts who have demonstrated the ability to design, manage, and lead high quality descriptive and evaluative studies in the field of education. Individuals who will be reviewing evaluation studies or methods papers must have extensive familiarity with evaluation methods and IES standards for scientifically valid research and evaluation. This shall include familiarity with the evidence and reporting requirements of the What Works Clearinghouse. In addition, reviews of technical quality may require psychometric expertise in some cases.

Readability/Writing Quality and Style The contractor shall provide written reviews of the readability and writing quality for all reports expected in each year of this contract. (Reviews of readability are not requested for proposals.) The reviews shall provide feedback on the extent to which REL products are written in a clear and engaging fashion. Reviewers shall comment on the writing style of all products and the extent to which they can be understood by a practitioner or policymaker. (See the Federal Plain Language Guidelines.) Review http://www.plainlanguage.gov/howto/guidelines/FederalPLGuidelines/TOC.cfm comments focused on readability shall help the authors to eliminate redundancies; perfect word and syntax choices; structure paragraphs, chapters, and reports in a logical manner; and create products that are easily consumed by their intended audiences.

Unlike the previous two categories mentioned above (i.e., relevance and technical quality), which focus on research proposals as well as products, the review for readability will be largely focused on products intended for public release.

Individuals charged with reviewing the readability of REL products must have demonstrated expertise in the clear and careful written presentation of research findings for diverse audiences. In some cases, reviews related to readability may be conducted by the same individuals reviewing products for relevance or technical quality while in other cases, reviewers for readability will be other individuals who are expert in issues of writing and presentation.

Utility The contractor shall provide written reviews on the utility of tools, technical assistance, data sets, or other products intended for use by researchers, policymakers, or practitioners. The written reviews shall provide feedback on the extent to which REL products are able to be used by their intended audiences.

Reviews of utility shall be conducted on proposals and products. For proposals, reviews related to utility will provide written comments on the extent to which any tools, technical assistance programs and documents, data sets, or other products are likely to be in a form, and inclusive of content, that could be easily used by the intended audience. Reviews for the utility of products shall include an assessment of the likelihood that the intended audience will be able to use the product as designed.

Individuals charged with reviewing the usability of REL products must represent experiences similar to those of the intended audience for a REL product. For example, products that are targeted toward district administrators shall be reviewed for usability by individuals who have experience working in school district central offices. Products that are targeted at researchers shall be reviewed by researchers with experience in a similar field. The range of experiences that will be needed for usability reviews will include school and district leadership, state education agency leadership, and researchers with significant quantitative and qualitative research experience.

Deliverables (for each proposal or product) Review team list and qualifications Revised review team list (if requested) Review comments and memo for each proposal or product

Task 3: Build and Support a Web-Based Tracking and Review Application for REL Work

3.1 Build a Web-Based Tracking and Review Application (Cost reimbursement, IT)

The contractor shall build a web-based review and tracking application that allows IES staff, REL staff, and contractor staff to follow the progress of documents that are submitted for external review. This application will track all stages of a product, from the time of initial proposal submission to the COR to the final product released on the IES website. The application should create a work flow application that will notify users at each stage that they need to interact with a product, based on the process defined in Task 2. The current workflow is attached in Appendix A.

All systems developed for this contract shall be hosted in the IES Data Center. To be hosted in the IES Data Center, all websites and web applications must be developed using ASP.NET (.NET Framework 3.5, 4.5, 4.6, and MVC 3 are currently supported) and SQL Server 2012 Standard Edition on Windows 2012 Servers and comply with IES's web development standards. A copy of the standards is available upon request. Other platforms and databases are not authorized to run in the IES Data Center.

Third party software (e.g., COTS, open source) must to be approved by the ED Office of the Chief Information Officer and IES and be licensed appropriately or be freely available through open source licensing. The contractor should not assume 3rd party software will be approved. All software developed under this contract should be transferred to IES free of any restrictions, licenses, or fees in accordance with the Rights and Data Clause contained in the contract. All IT security requirements listed in this contract are covered by the current IES Data Center security authorization.

The contractor shall develop a list of requirements for the application based on the information provided in the solicitation and the discussions that occur at the kickoff meeting. This shall be submitted to the COR within 5 days of the kickoff meeting. Upon acceptance, the contractor shall develop a proposed approach to meet the requirements within three weeks. This approach shall include a timeline with specific objectives and milestone dates. All development must take an agile approach and there must be usable functionality every 6 months, if not sooner. The COR will provide written feedback and the contractor shall incorporate the comments and submit a second draft within two weeks of receiving comments.

The contractor shall provide a beta version (version 0.1) of the application within 6 weeks of having an approved plan. This version must be fully functional and allow the COR to see all aspects of the application. The COR will provide feedback within 2 weeks and the contractor is expected to provide a revised beta version (version 0.2) within 2 weeks. The version 0.2 will be tested with one REL, one REL COR, the Associate Commissioner, and the Commissioner. Users will provide feedback to the COR who will provide written feedback to the contractor within 2 weeks. The contractor is expected to provide a revised version (version 0.3) within 2 weeks. This version shall have all work currently in progress loaded into the system and will be piloted with all RELs for 1 month. The COR will collect feedback and will provide it to the contractor. The contractor is then expected to make any revisions and launch the initial version (1.0) within 2 weeks of COR feedback. The contractor is expected to maintain the application and release updates as needed to fix known problems.

Deliverables:

List of Requirements Proposed Approach Memo (draft and final) Version 0.1 Version 0.2 Version 0.3 Version 1.0 Revised versions

3.2 Develop Training Materials and Provide Technical Support (Fixed Price, IT)

The contractor shall provide training for relevant NCEE and REL staff on the use of the tracking and review application. This training for the RELs should be internet based via a webinar that is recorded and archived in the application. The training for IES staff may be internet-based or in-person. If the contractor chooses in-person training, the contractor will be responsible for all related costs and logistical arrangements. However, the training can be conducted at ED-provided facilities in Washington, DC. This training shall occur within 1 week of the launch of version 0.3.

The contractor shall also provide training materials for NCEE and REL staff. These shall include a user’s guide and tip sheets that shall reside electronically in the application. A draft version of these guides shall be submitted with version 0.2 to the COR (within 10 weeks after the Approach Memo is approved).

The COR will provide feedback and final versions shall be included in version 1.0. All user guides and tip sheets shall be updated with each release.

The contractor shall provide technical support for application users. A help desk email address and telephone number shall be provided so that application users can have questions answered during the business day (11 am to 6pm Eastern Time). Wait times for responses shall not exceed 60 minutes during the business day. The contractor shall provide a summary of common issues experienced by users in each monthly report.

Deliverables:

Archived training webinar Users Guide and Tip Sheets (draft and final) Help Desk email address and telephone number Summary of common issues experienced by users

3.3 Manage Proposals and Reports through Temporary Email Process (Cost reimbursement, non-IT)

While the application is under development, the contractor shall manage and track progress of proposals and reports through email. This temporary review and tracking system shall enable IES and the RELs to be notified of the review status of all proposals and draft products and shall allow for the continued review of all proposals and draft products by the COR, REL COR, independent peer reviewers employed by the contractor, the NCEE Associate Commissioner for Knowledge Utilization and the NCEE Commissioner. The contractor shall expect to administer this temporary, email-based system for up to 25 weeks, until version 0.3 is released. The contractor shall submit draft email templates that will be sent through the temporary process to the COR within 5 days after the kickoff meeting and final versions within 5 days of COR feedback.

Deliverable:

Email templates for the temporary process (draft and final)

Task 4: Technical Editing, Formatting, and Production of Reports (Cost reimbursement, Non-IT)

4.1 Update the “Writers’ and Style Guide”

The contractor shall, within 4 weeks of contract award, update the “Writers’ and Style Guide” that provides guidance to the REL authors about how to meet IES and NCEE requirements and prepare a report for review and that provides information about references, footnotes, table format, bibliography format, and other style requirements and formatting. This Guide can be an updated version of the existing Guide. NCEE will review and provide comments within one week of receipt. The contractor shall incorporate NCEE comments and provide the final draft within two weeks of receipt from NCEE.

Deliverable:

https://ies.ed.gov/ncee/edlabs/relresources/pdf/REL_Program_Writers_and_Style_Guide_05262015.pdf

Updated Writers’ and Style Guide (draft and final)

4.2 Report Editing, Formatting, and Production

Because of the high volume of reports expected to be produced each year and the variation in report size from approximately 10 pages to over 100 pages, IES requires a way to ensure report quality and consistency across the 10 RELs. REL and other NCEE products must follow the same organizational platform, and have consistency in layout, handling of footnotes, references, tables, and statistical nomenclature.

The contractor shall provide technical editing and production service for approximately 50-100 reports that will be produced each year for a total of up to 275 reports that will be produced. The contractor shall be provided with a final version of the report in a Word file and shall conduct a technical edit, assuring that all grammar, spelling, and other conventions for written research are met. In addition, the contractor shall be responsible for formatting reports from a Word document or Word template into a design template.

The contractor shall have no more than two weeks to fully edit the Microsoft Word document. NCEE will then review the edited document and provide feedback if needed; NCEE will also request revisions from the REL if needed. The contractor shall then prepare the document in the design template. The REL and NCEE will then proof the formatted document. The contractor shall then make any necessary modifications and deliver a final PDF version of the report within one week of receiving approval and any feedback from NCEE, and will assure that the final PDF complies with Section 508 of the Rehabilitation Act (click this link for further information about the relevant legislation).

As part of the editorial review, the contractor shall:

• Revise and update a Writers’ and Style Guide for preparing reports that meet NCEE standards;

• Organize the entire report;

• Ensure tone is appropriate for the target audience;

• Ensure a logical presentation of findings;

• Ensure the proper use and placement of tables, bullets, and “call outs;”

• Be responsible for overall copyediting and proofreading to ensure there are no errors and that a style consistent with NCEE guidelines is used for citations, tables, footnotes, references, and title and verso pages.

As part of the production process, the contractor shall:

• Revise and update report templates (as requested by NCEE) to be used with various product lines, using design and layout that incorporates the use of pictures, charts, tables, bullets, call outs, and other features designed to make reports attractive and informative for the target audience;

• Ensure design and layout meet ED/IES 508 standards;

• Format all reports from Word files to the design templates;

• Include a verso page that uses IES-approved disclaimer language; and

• Provide PDF file of the final report.

Deliverables:

Reports (draft and final) https://www.section508.gov/content/learn/laws-and-policies

PDF file of each final report

4.3 Update Report Templates

As requested by NCEE, the contractor shall provide NCEE with sample report templates that are designed to be used for various REL report types. NCEE will review and provide comments within one week of receipt. The contractor shall incorporate NCEE comments and provide the final draft within two weeks of receipt from NCEE.

Deliverable:

Updated Report Templates (as requested, draft and final)

Task 5: As-Needed Guidance for Improvement of REL Products (Cost reimbursement, Non-IT)

Following the written review process, NCEE and REL staff will occasionally require additional guidance and consultation about how to improve proposals and products. The required guidance will be related to any of the four review criteria described above (i.e., relevance, technical quality, readability, and utility) and shall be provided in telephone conversations between the contractor’s reviewers and REL and NCEE staff members. The oral as-needed guidance will focus primarily on methodological issues (technical quality) related to impact studies and methods papers prepared for NCEE. Less frequently, as-needed guidance will be expected about other issues related to REL products.

Telephone calls for as-needed guidance will take place approximately 10 times per year and will each last approximately one hour. The contractor will provide a call-in number for participants to use and will be responsible for scheduling the call at a time that is acceptable to all participants (REL staff, NCEE staff, and reviewers). Within a week of the telephone call, the contractor will provide a written memo summarizing the content of the call, guidance provided, and next steps.

Deliverables (for each telephone call) Memo summarizing guidance and next steps

Task 6: As-Needed Analytic and Methods Oriented Projects (Optional, Cost Reimbursement, Non-IT)

The contractor shall design up to three projects throughout the duration of the base period of the contract that provide exemplars of managing complicated analytic tasks or addressing technical issues.

All proposals will be subject to NCEE approval. The contractor shall design activities that are responsive to immediate REL needs that will help advance the state of knowledge, and that contribute meaningfully to improving the conduct and reducing the cost of education evaluations.

Activities will range in scope from relatively basic (e.g., producing a simple primer that suggests methods for treatment of missing data when using SLDS data) to more complex (e.g., providing a project demonstration from start to finish). This work can involve using IES-sponsored restricted use files or taking advantage of currently planned studies in the following ways:

• Identify opportunities in experimentally designed studies conducted by the RELs or by other NCEE contractors to test innovative methodological approaches;

• Review REL draft study designs for particular studies that offer opportunities to test methodological techniques in order to learn whether these techniques may represent improved cost-effective methods;

• Conduct replication studies that compare the results of non-experimental methods with experimental methods; and

• Organize and produce written reports related to methodological and analytical issues related to the empirical investigations.

6.1 Analytic and Methods Oriented Projects

The contractor shall produce proposals that detail the main goals of a project, the questions that will be answered, and the expected benefit of the project to NCEE, in particular to the RELs, and education researchers at large. The contractor shall submit a draft proposal. The COR will provide comments within 2 weeks of receipt. The contractor shall submit a revised proposal within 2 weeks of receiving COR comments.

After NCEE has approved the final study design proposal, the contractor shall carry out the work. The contractor will be responsible for any OMB clearance that may be required related to the approved project. Interim updates shall be shared with the COR in monthly progress update memos throughout the project life cycle.

The contractor shall submit draft reports to NCEE. Reports shall be comprehensive and appropriate for the intended audience of researchers and evaluation experts to further the advancement of cost effective evaluation methods and improve NCEE evaluation design. The COR will provide comments within 2 weeks of receipt. Once approved by the COR, reports will be submitted for approval by IES’s Standards and Review Office (SRO). Review by SRO can take up to four to six months. The contractor shall submit revised drafts to SRO within 4 weeks of receipt of SRO comments. The contractor should expect at least 2 rounds of SRO review.

Deliverables:

Project proposal (draft and final) Monthly progress update memo Report (draft and final)

6.2 Meetings to Discuss Findings

The contractor shall organize and convene up to three meetings of experts in the field of education evaluation to review the final reports submitted under Task 6.1. The contractor shall work closely with the COR to identify and propose a suitable list of between 4 and 6 experts to participate in each of the three meetings. These experts will review final reports, assure they meet scientific standards, and recommend changes where necessary. Upon COR approval of the experts, the contractor shall invite these individuals to participate in the meeting. The contractor shall assess whether potential invitees may have any conflict of interest pertaining to the outcomes of the study and provide the list of proposed invitees to ED for approval. The contractor shall prepare a draft agenda for each meeting held, and submit it to NCEE at least 4 weeks before the date of the meeting. The COR will provide feedback on the draft agenda within 2 weeks. The contractor shall submit a final agenda 1 week prior to the meeting.

The meeting(s) shall take place in Washington, DC at facilities provided by IES. The contractor shall provide remuneration to the experts participating in such meeting(s) at a reasonable rate, which shall take into consideration the amount of time spent participating in the meeting and preparing for it, as well as the consulting rates that are usual and customary for these experts. The contractor will be responsible for all travel costs associated with these meetings (both their own and all participants) and shall coordinate all logistics related to these meetings. Within 1 week of each meeting, the contractor shall prepare a short (3-5 page) memo regarding what was learned at the meeting and any recommendations to NCEE that relate to possible follow-up activities.

Deliverables:

Agenda for meeting to discuss findings (draft and final) Memo regarding lessons learned from meeting

Task 7: Logistical Support

7.1 Maintain REL Intranet (Cost Reimbursement, IT)

The contractor shall provide support regarding the maintenance of the REL intranet. The REL Intranet is a SharePoint 2013-based application in the IES Data Center. It is separate from the application to be developed in Task 3. The REL Intranet allows the REL contractors to share information about projects to support cross-REL collaboration, houses an inventory of Ask-A-REL responses, provides a space to upload and store IES Guidance documents and other important REL documents, and has an event calendar. The RELs upload Ask-a-REL responses to the REL Intranet on a monthly basis. The REL Intranet allows REL contractors and IES to categorize the Ask-a-REL responses by a controlled vocabulary by topic and allows REL contractors and IES to sort and filter the existing responses by topic and other appropriate metadata. For a video walkthrough of the REL Intranet, click here.

The contractor shall help ensure users have the appropriate levels of access and control; address broken links or other glitches; and help improve the usability of existing pages, sites, and libraries. The contractor shall also work with the COR to identify problems that the current users are experiencing and resolve such issues. The contractor shall submit a memo within 10 working days after the end of each quarter that briefly summarizes the problems that the contractor addressed through the maintenance of the REL Intranet.

The contractor shall submit a memo within 10 working days after the end of each quarter that reports the number of Ask-a-REL responses uploaded by each REL and by each topic over each quarter.

Deliverables Quarterly reports on maintenance of REL Intranet Quarterly summary of Ask-a-REL responses

7.2 Stakeholder Feedback Survey (SFS) Portal (Cost Reimbursement, IT)

The contractor shall develop a portal in the IES Data Center for administration of the Stakeholder Feedback Survey, which REL contractors administer after each of their webinars, events, and technical assistance activities. The portal will contain a public-facing web form that recipients of REL technical assistance can use to respond to the survey. A unique URL will be created for each survey administration. Survey responses will be accessible by the RELs through the REL Intranet. A report generator shall be developed so SFS results can be tabulated for each event/activity, each REL, and across the program.

All systems developed for this contract shall be hosted in the IES Data Center. To be hosted in the IES Data Center, all websites and web applications must be developed using ASP.NET (.NET Framework 3.5, https://www.youtube.com/watch?v=LjiVFGrfh6g&feature=youtu.be

4.5, 4.6, and MVC 3 are currently supported) and SQL Server 2012 Standard Edition on Windows 2012 Servers and comply with IES's web development standards. A copy of the standards is available upon request. Other platforms and databases are not authorized to run in the IES Data Center.

Third party software (e.g., COTS, open source) must to be approved by the ED Office of the Chief Information Officer and IES and be licensed appropriately or be freely available through open source licensing. The contractor should not assume 3rd party software will be approved. All software developed under this contract should be transferred to IES free of any restrictions, licenses, or fees in accordance with the Rights and Data Clause contained in the contract. All IT security requirements listed in this contract are covered by the current IES Data Center security authorization.

The contractor shall deliver an approach memo about how they anticipate creating the portal within 6 weeks of contract award. This memo shall include a timeline of milestones and an estimated date of completion.

Deliverables:

Proposed approach memo Stakeholder Feedback Survey portal

7.3 Logistics and Support for REL Directors Meetings (Cost Reimbursement, Non-IT)

Two times per year – typically in March and September – IES convenes a meeting of the REL Directors and key staff. The contractor shall provide support for these meetings in the following ways:

• Take detailed notes during the meeting, share those draft notes with IES, make revisions based on IES comments, and prepare final summary notes.

• As requested by NCEE, assist IES in arranging for guest speakers at each meeting. Arrange and pay for travel for each speaker.

Deliverables:

REL Directors Meeting notes (draft and final)

7.4 Logistics and Support for Webinars for REL Staff (Optional, Fixed Price, IT)

IES may have the need to provide two, 60-to-90 minute webinars per year for the RELs in order to explain guidance or discuss a REL program issue. The contractor shall run the technical aspects of each webinar using the latest webinar technologies. IES will identify the need for each webinar at least two weeks in advance and will work with the contractor to identify a time and a date for each webinar. Each webinar may include up to 200 participants. All webinars must have live closed captioning. Within 2 weeks of the webinar, the contractor shall submit an archived video of the webinar and a closed caption file in a format that can be uploaded to YouTube.

Deliverables:

Webinar video recording Webinar closed caption file

7.5 Update REL Program Brochure and Create 3 REL Posters (Fixed Price, Non-IT)

Within one week of the award of the contractor, the COR will provide the contractor with guidelines for the creation of a new REL Program brochure. Within two weeks of receiving those guidelines, the contractor shall submit a draft REL brochure. The COR will provide feedback on the brochure within two weeks. Within two week of receiving that feedback, the contractor shall submit a revised version of the brochure. The COR will provide any additional feedback within one week and within two weeks of receiving that feedback, the contractor shall submit 500 hardcopies of the final REL brochures and an electronic file with the brochure.

Within 6 weeks of the award of the contract, the COR will provide the contractor with guidelines for the creation of a REL poster that is to be approximately 3 feet by 5 feet in size. Within two weeks of receiving those guidelines, the contractor shall submit a draft REL poster. The COR will provide feedback on the draft poster within two weeks. Within two weeks of receiving that feedback, the contractor shall submit a revised version of the poster. The COR will provide any additional feedback within one week and within two weeks of receiving that feedback, the contractor shall submit 6 hardcopies of the final REL poster and an electronic file with the poster.

Within 52 weeks of the award of the contract, the COR will provide the contractor with guidelines for the creation of a second REL poster that is to be approximately 3 feet by 5 feet in size. Within two weeks of receiving those guidelines, the contractor shall submit a draft REL poster. The COR will provide feedback on the draft poster within two weeks. Within two weeks of receiving that feedback, the contractor shall submit a revised version of the poster. The COR will provide any additional feedback within one week and within two weeks of receiving that feedback, the contractor shall submit 6 hardcopies of the final REL poster and an electronic file with the poster.

Within 104 weeks of the award of the contract, the COR will provide the contractor with guidelines for the creation of a third REL poster or a second REL brochure. (The poster is to be approximately 3 feet by 5 feet in size.) Within two weeks of receiving those guidelines, the contractor shall submit a draft REL poster or brochure. The COR will provide feedback on the draft poster or brochure within two weeks.

Within two weeks of receiving that feedback, the contractor shall submit a revised version of the poster or brochure. The COR will provide any additional feedback within one week and within two weeks of receiving that feedback, the contractor shall submit 6 hardcopies of the final REL poster(or 500 copies of the REL brochure) and an electronic file with the poster or brochure .

Deliverables First draft of first REL brochure 2nd draft of first REL brochure 3rd draft of first REL brochure Final version of first REL brochure

First draft of first REL poster 2nd draft of first REL poster 3rd draft of first REL poster Final version of first REL poster

First draft of second REL poster 2nd draft of second REL poster 3rd draft of second REL poster Final version of second REL poster

First draft of second REL brochure (or third REL poster)

2nd draft of second REL brochure (or third REL poster) 3rd draft of second REL brochure (or third REL poster) Final version of second REL brochure (or third REL poster)

Option Year One - Continuation of work on Tasks 1, 2, and 4 through 7 The contractor shall perform work on Tasks 1, 2, and 4 through 7. Subtask 2.1, all of Task 3, subtask 4.1, and subtask 7.2 do not continue after the base period. For Task 1, the kick-off meeting will be replaced with an annual meeting to discuss the work plan for the option period.

Tasks and subtasks to be performed during Option Year One:

Task 1 Communication with IES

1.1 Annual Meeting

1.2 Periodic telephone calls with COR

1.3 Monthly Progress/Status Reports

1.4 Annual Program Assessment Information

Task 2 Refine, Improve and Implement Review Criteria and Processes for REL Work

2.2 Review Research Proposals and Products

Task 4 Technical Editing, Formatting, and Production of Reports

4.2 Report Editing, Formatting, and Production

4.3 Update Report Templates

Task 5 As-Needed Guidance for Improvement of REL Products Task 6 As-Needed Analytic and Methods Oriented Projects

6.1 Analytic and Methods Oriented Projects

6.2 Meetings to Discuss Findings

Task 7 Logistical Support

7.1 Maintain REL Intranet

7.3 Logistics and Support for REL Directors Meetings

7.4 Logistics and Support for Webinars for REL Staff

7.5 Update REL Brochure and Create 3 REL Posters

Option Year Two - Continuation of work on Tasks 1, 2, and 4 through 7 The contractor shall perform work on Tasks 1, 2, and 4 through 7. Subtask 2.1, all of Task 3, and subtask

4.1 do not continue after the base period. Subtask 7.5 does not continue after Option Year 1. For Task 1, the kick-off meeting will be replaced with an annual meeting to discuss the work plan for the option period.

Tasks and subtasks to be performed during Option Year Two:

Task 1 Communication with IES

1.1 Annual Meeting

1.2 Periodic telephone calls with COR

1.3 Monthly Progress/Status Reports

1.4 Annual Program Assessment Information

Task 2 Refine, Improve and Implement Review Criteria and Processes for REL Work

2.2 Review Research Proposals and Products

Task 4 Technical Editing, Formatting, and Production of Reports

4.2 Report Editing, Formatting, and Production

4.3 Update Report Templates

Task 5 As-Needed Guidance for Improvement of REL Products Task 6 As-Needed Analytic and Methods Oriented Projects

6.1 Analytic and Methods Oriented Projects

6.2 Meetings to Discuss Findings

Task 7 Logistical Support

7.1 Maintain REL Intranet

7.3 Logistics and Support for REL Directors Meetings

Option Year Three - Continuation of work on Tasks 1, 2, and 4 through 7 The contractor shall perform work on Tasks 1, 2, and 4 through 7. Subtask 2.1, all of Task 3, and subtask

4.1 do not continue after the base period. Subtask 7.5 does not continue after Option Year 1. For Task 1, the kick-off meeting will be replaced with an annual meeting to discuss the work…