REL_RFI_-_Resource_Guide_2012-17.pdf

i

Analytic Technical Assistance and Development

Resource Guide for Regional

Educational Laboratory (REL) Staff

2012–17 Contract Cycle

July 2015

Acknowledgments

The Resource Guide for Regional Educational Laboratory (REL) Staff 2012–2017 Contract Cycle was developed collaboratively through the Analytic Technical Assistance & Development (ATA&D) contract. Decision Information Resources, Inc. (DIR), acting as the ATA&D contractor for the 2012–2017 contract cycle, was responsible for organizing the work of all contributions leading to the development of this resource guide for the Institute of Education Sciences (IES) under Contract ED-IES-12-C-0057. NCEE and ATA&D would like to thank Lauren Scher of Concentric Research & Evaluation (ATA&D); Russell Jackson and Ron McCowan of DIR (ATA&D); Mike Puma of Chesapeake Research Associates, LLC (ATA&D);

other ATA&D reviewers; IES staff;; and the REL Directors and staff for their contributions to this publication.

Contents

Preface........................................................................................................................................... iv

Purpose.................................................................................................................................... iviv

Why the REL Program Is Important—a Word from the Commissioner .................................... v

Where the REL Program Fits at the Department of Education ................................................. vi

Organization of the Guide.......................................................................................................... vi

Chapter 1. The Regional Educational Laboratory (REL) Program

The REL Program Has a Long-Standing and Evolving History

Initial Implementation of the REL Program (1965 to 2005) Focused on Technical

Assistance

The REL Program from 2006 to 2011: the REL Shift toward Scientific Rigor

The 2012–17 REL Program

Organization and Focus

Key Tasks

REL Support and Monitoring Provided by IES

REL COR

Other Key Personnel at IES Who Interact with the REL Program

NCEE Commissioner

Associate Commissioner for Knowledge Utilization

REL Peer Review (RPR) COR

Analytic Technical Assistance and Development COR

NCEE Communications and Support Coordinator

NCEE Requirements

Frequently Asked Questions for the NCEE Approval Process

Chapter 2. Research Alliances

What Is a Research Alliance?

How Do the RELs Determine Their Yearly Work?

How Do RELs Work with Research Alliances?

Chapter 3. Standards and Expectations for Writing and Communication

Reports and Other Public Documents

Additional Communications Guidance

From Inception to Completion: Navigating the REL Review Process

Chapter 4. Standards and Expectations for Methodological Rigor

Relevance and Utility

Research Questions ii

Study Designs

Measures, Data Collection, and Analysis

Abbreviations

Glossary of Terms Relevant to the 2012–17 REL Program

Exhibits

Exhibit 1. The Ten Regional Educational Laboratories

Exhibit 2. REL Product Lines

Exhibit 3. REL Peer Review, Phase I: Proposal and Report Review Process Using

SharePoint Workflows

Exhibit 4. REL Peer Review, Phase II: Report Production Process Using SharePoint

Workflows

Tables

Table 1. How Different Funding Mechanisms Vary in Purpose, Goal, and the Level of

Government Involvement

Table 2. Roles of the Contracting Officer and Contracting Officer’s Representative

Table 3. Information and Resources for REL-Specific Research Alliances

Table 4. Additional Resources for Dissemination Guidance

Appendixes

Appendix A. Document Resource List .................................................................................... A-1

Appendix B. IES Organization Chart ......................................................................................B-1

Appendix C. REL Research Alliances .................................................................................... C-1 iii

Preface This section gives the purpose of this resource guide, explains why the Regional Educational Laboratory (REL) program is important, tells how the REL program fits into the U.S.

Department of Education’s mission, and describes the organization of this guide.

Purpose

This guide provides critical information relevant to all REL staff. Because each REL has different internal operating procedures, new REL staff members should also consult resources specific to their individual REL (for example, the Updated Annual Plan). This guide should help new REL staff become familiar with IES requirements and standards developed specifically for the REL program. It provides a general summary of the history and operations of the REL program and describes the process for developing and producing REL products—particularly IES’s expectations for writing, communication, and standards of rigor.

This guide provides links, if available, to key existing REL and other IES guidance documents.

Other nonlinked guidance documents are available to REL contractors through the REL intranet.

In particular, the following five most important documents, entitled “Top REL Resources,” should be considered required reading for all REL staff:

• The Performance Work Statement for Regional Educational Laboratories (2012–17)

• Your REL’s annual plans (current and past)1

• REL Program Writers Guide and Style Guide2

• The REL Peer Review webinar and flowcharts3

• NCEE Guidance for REL Study Proposals, Reports, and Other Products 4

This document contains hyperlinked text that will redirect the reader to external Internet content, key terms in the current document’s glossary, or the list of abbreviations included at the end of this document. Appendix A lists all of the documents that are referenced in this guide. Current REL staff may find links to these documents on the REL intranet.

1 Available from individual RELs.

2 Available from the REL Resources Web page: http://ies.ed.gov/ncee/edlabs/RELResources/index.asp.

3 REL staff will need to obtain login information from their individual REL.

4 Available from the REL Resources Web page: http://ies.ed.gov/ncee/edlabs/RELResources/index.asp.

iv http://ies.ed.gov/ncee/edlabs/RELResources/index.asp https://www.fbo.gov/utils/view?id=50688bfda3cc31d77469838a79649491

Why the REL Program Is Important—a Word from the Commissioner

The Regional Educational Laboratory program is one of several efforts funded by the U.S.

Department of Education that fall under a broad category known as “technical assistance” (TA).

These efforts include the Comprehensive Center Program, the Equity Assistance Centers, and technical assistance supports for grantees who receive funds from a variety of different ED funding sources. The general purpose of technical assistance efforts is to provide expert consultation that will help states, districts, and/or grantees do something better. That may be planning initiatives, developing policy responses, analyzing data, or conducting evaluations of effectiveness.

In this constellation of Department-funded technical assistance, the REL program has a unique role. RELs are charged with building the capacity of state and/or district educators and policymakers to use data and research to improve student outcomes. To maintain the distinctive mission of the REL program, it is important for REL staff to critically examine requests from their regions to see whether the work is appropriate for a REL or whether it should be referred to a different technical assistance program. Similarly, the other ED-funded programs will consider which requests are appropriate for them to undertake and which should be referred to the RELs.

An important feature of the 2012–2017 REL Program cycle is the central focus on researcher-practitioner partnerships, known as research alliances. Working with their RELs, research alliance members identify specific education challenges that can be better understood and addressed by using data and research—by working in partnership with research alliances to:

• conduct applied education research;

• offer opportunities for educators to learn about findings from credible, up-to-date research in education; and

• provide training and other support on education research design, data collection and analysis, and reporting of findings.

The RELs have an exciting and unique opportunity to provide both rigorous and relevant support for use of data and research in places where it is needed most.

This is no small undertaking. The success of the REL Program is possible only when REL contractors (and their extensive network of subcontractors and consultants), research alliances, IES staff, and the Department of Education’s Contracts staff all work together to produce the excellent work that is at the heart of the endeavor. We’re glad to have you on board!

Ruth Curran Neild Commissioner, National Center for Education Evaluation and Regional Assistance v

Where the REL Program Fits at the Department of Education

The REL Program is administered by the Institute of Education Sciences (IES), which is one of 27 offices at the U.S. Department of Education.5 There are four centers at IES:

• The National Center for Education Research (NCER) supports rigorous research that addresses the nation’s most pressing education needs from early childhood to adult education (primarily through grants).

• The National Center for Special Education Research (NCSER) sponsors a comprehensive program of special education research designed to expand the knowledge and understanding of infants, toddlers, and children with disabilities (primarily through grants).

• The National Center for Education Statistics (NCES) is the primary federal entity for collecting and analyzing data related to education (primarily through contracts).

• The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased large-scale evaluations of education programs and practices supported by federal funds; provides research-based technical assistance to educators and policymakers;

and supports the synthesis and the widespread dissemination of the results of research and evaluation throughout the United States (primarily through contracts).

The REL program is located in the Knowledge Utilization Division of NCEE, along with the What Works Clearinghouse (WWC), the Education Resources Information Center (ERIC), and the National Library of Education (NLE). NCEE’s other division, the Evaluation Division, conducts rigorous impact studies of promising federally funded education programs. See Appendix B for the IES organization chart.

Organization of the Guide

This document is organized into four chapters:

Chapter 1 provides an overview of the REL program by describing its evolution over the past half century, explaining how the current REL program operates, and giving a synopsis of the key support and monitoring functions provided by IES and its contractors.

Chapter 2 discusses REL research alliances—a critical piece of the current REL contracts structure. This chapter also discusses alliances and how RELs incorporate the needs of an alliance into the development of annual work plans.

Chapter 3 provides an overview of the standards and expectations for REL program writing and communication. This chapter summarizes and documents numerous resources available to REL staff for meeting these standards and also describes the layered REL review process for all project proposals and reports.

Chapter 4 focuses on standards and expectations regarding methodological rigor for REL activities. This chapter also summarizes current guidance for all IES-published REL publications, including randomized controlled trials (RCTs), literature reviews, and systematic reviews.

5 A full list of U.S. Department of Education offices is available at http://www2.ed.gov/about/offices/list/index.html?src=ft.

vi http://www2.ed.gov/about/offices/list/index.html?src=ft

Chapter 1. The Regional Educational Laboratory (REL) Program This chapter discusses the history of the REL program; the 2012–17 cycle of the program, including its focus and key tasks; support and monitoring by IES; and REL program requirements.

The REL Program Has a Long-Standing and Evolving History

The REL Program is one of the longest running federal education initiatives. It was established under the 1965 Elementary and Secondary Education Act (ESEA) to help bridge the gap between research and development in education and school districts. The REL program has changed over time to reflect shifts in regional needs as well as in the U.S. Department of Education’s (ED’s) priorities.

Initial Implementation of the REL Program (1965 to 2005) Focused on Technical Assistance

For the most part, the early RELs focused on achieving general educational improvement by providing direct technical assistance to support the needs of education practitioners and policymakers. In 1985, ED sought to increase the effectiveness of the RELs by focusing their work on the needs of educators in their designated regions, with a particular emphasis on school and classroom improvement. Over the next 20 years, the REL priorities continued to evolve and, in some cases, shift emphases: (1) the 1990–95 REL program focused on the needs of at-risk children; (2) the 1995–2000 REL program focused on promoting excellence and equity through the scaling-up of educational reform to encompass all schools and students; and (3) the 2000–05 program focused on the dissemination of knowledge about ways to transform low-performing schools into effective learning communities, including providing support for comprehensive school reform activities at the state, school district, and school levels.

The REL Program from 2006 to 2011: the REL Shift toward Scientific Rigor

The 2006–11 period represented a significant shift in the focus and operations of the REL program after the creation of the Institute of Education Sciences (IES). Created as part of the 2002 reauthorization of the Elementary and Secondary Education Act (ESEA) and the subsequent Education Sciences Reform Act of 2002 (ESRA),6 IES has a mission “to provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study, in order to provide parents, educators, students, researchers, policymakers, and the general public with reliable information about

• The condition and progress of education in the United States, including early childhood education;

• Educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and

• The effectiveness of federal and other education programs.”7

6 Public Law (P.L.) 107–279.

7 Ibid, Section 111.

http://ies.ed.gov/director/pdf/ESRAreauth.pdf

ESRA further requires IES to “compile statistics, develop products, and conduct research, evaluations, and wide dissemination activities in areas of demonstrated national need.” All research conducted by IES must use “scientifically based research standards that include, where appropriate, making claims of causal relationships only in random assignment experiments,” and “must be subjected to rigorous peer review before being published or otherwise made available to the public.”8

In line with this role, IES directed the RELs to become “emissaries of science, providing policymakers and practitioners with training and technical assistance based on the highest-quality evidence, as defined by scientifically valid research principles. Further, where such evidence is not available, the RELs were expected to fill the gap with their own applied research and development.”9 At a minimum, IES expected the work of the RELs to include “scientifically valid research on teaching methods, assessment tools, and high quality, challenging curricula (at a minimum covering math, science, and reading); English language acquisition; education technology; and promising practices and new educational methods, including professional development and educational technology.”10

The most significant change to the traditional role of the RELs in the 2006–11 cycle was the shift from (1) serving primarily as a provider of technical assistance to (2) developing and distributing knowledge about the best ways to increase the effectiveness of educational policies and practices through conducting research that meets established standards for “high-quality” evidence.

During this period, the RELs produced two types of research:

• “Fast response” studies based on the use of existing data or studies, or small-scale investigations, which were limited to noncausal conclusions

• Studies that used randomized controlled trials (RCTs) to draw causal conclusions about the effectiveness of well-established educational interventions

The 2012–17 REL Program

This section describes the organization and focus of the 2012–17 REL program and its key tasks.

Organization and Focus

The REL program continues to focus on the creation and dissemination of high-quality evidence to inform policy and practice. The competition for the 2012–17 cycle took place in 2011. Ten RELs (see Exhibit 1) were selected through a competitive process to serve their respective regions through research, training, and technical assistance on the basis of the best available evidence about effective practices.11 In particular, the RELs are required to help

8 Ibid, Section 173.

9 The 2006–11 REL Performance Work Statement.

10 Ibid.

11 http://ies.ed.gov/ncee/edlabs/about/.

Top Five REL Resources

#1 Performance Work Statement for Regional Educational Laboratories

(2012–17) http://ies.ed.gov/ncee/edlabs/about/ https://www.fbo.gov/utils/view?id=50688bfda3cc31d77469838a79649491 https://www.fbo.gov/utils/view?id=50688bfda3cc31d77469838a79649491 https://www.fbo.gov/utils/view?id=921fb8d5f465b7886cfd21681ab6e604 states and districts in the regions build their internal capacity to “use data and research for education improvement.”12

To ensure that the work of the RELs is useful and relevant, the contract tasks require the RELs to work in close partnership with “research alliances” (which are made up of education practitioners and policymakers at the district or state levels) and others, such as representatives of higher education. As part of a collaborative effort, REL staff and alliance members seek to identify pressing education challenges in their region. RELs work with alliances to support increased regional capacity to use data and to frame high-quality research and evaluation studies that will inform decisions related to improving current practices and incorporating data and research into everyday decision-making. Alliances also play pioneer roles to try newly developed or identified policies and practices in REL projects before wide dissemination. (Chapter 2 provides more information about the research alliances.)

The use of data and analysis to improve student outcomes is expected to “address issues of importance within the region that also are of national concern. The following issues are of highest national concern: early childhood education; identifying and retaining effective teachers and principals; adopting and implementing rigorous academic standards and assessments;

increasing college readiness, access, and completion; and improving low-achieving schools.”13

Exhibit 1. The Ten Regional Educational Laboratories

Source: http://ies.ed.gov/ncee/edLabs/regions/.

12 “NCEE Guidance for REL Study Proposals, Reports, and Other Products” available from the REL Resources web page: http://ies.ed.gov/ncee/edlabs/RELResources/index.asp.

13 Performance Work Statement for Regional Educational Laboratories (2012–17).

http://ies.ed.gov/ncee/edLabs/regions/

Key Tasks

The 2012–17 REL work is organized into seven primary tasks and a coordinating (eighth) task:14

• Task 1: Assess regional needs. RELs use various information collection and review methods to assess state and local needs for training, technical assistance, and analyzing research and data. While IES has set five national priorities for the REL program, each REL is expected to conduct needs-sensing activities that are used to set initial regional priorities and inform refinement of REL annual plans and research agendas. The RELs use a variety of approaches to conduct their ongoing assessment of needs, including (1) reviewing regional news and publications focused on education policy or legislation, (2) interviewing regional practitioners, agency officials, other relevant stakeholders, and governing board members, (3) reviewing response data from stakeholder feedback surveys (SFSs), (4) soliciting information from alliance membership, (5) reviewing Ask A REL inquiries (for details, see Task 5 below or the definition in the glossary), (6) using online forums, webinars, and social media, and (7) participating in professional conferences, cross-REL discussion groups, and cross-REL working groups.

• Task 2: Maintain and refine research alliances. As discussed in more detail in Chapter 2, most of the work conducted by the RELs is guided by “research alliances,” which are made up of education practitioners and policymakers at the state and district levels and other educational staff within the region (such as higher education authorizers). Under Task 2, REL staff seek to build relationships and develop a community of practice by engaging in meaningful discussions with researchers and practitioners to identify and respond to regional challenges. In collaboration with the RELs, the alliance members (1) identify policy or practice concerns or both, (2) develop plans to collect data related to those issues, and (3) analyze data to develop and test strategies for educational improvement. This work helps to frame Task 3 and Task 4 efforts (described in the following summaries). Engagement of research alliance members is an important task that RELs must accomplish because “at least 85 percent of the resources spent on Tasks 3 and 4 shall be associated with work conducted for specific research alliances.”15

• Task 3: Provide analytic technical support. Such support can include training and technical assistance related to available data systems and other research and data analysis activities. Task 3 activities build local education agency (LEA) and state education agency (SEA) capacity through the use of data and research results to identify and solve problems in states, districts, and schools. RELs publicly release some Task 3 resources, such as tool kits for education practitioners, workshops materials, and software packages. RELs perform much of this work through research alliances but also offer these services to the broader education community. Services available to the broader community, such as analytic technical support, may be publicized through REL websites.

• Task 4: Conduct applied research and evaluation studies. Research activities are intended to “model for LEAs and SEAs how to define clear and researchable questions, select appropriate data and methods of analysis, and examine and report research and evaluation

14 Performance Work Statement for Regional Educational Laboratories (2012–17), pages 4 – 6.

15 Ibid, page 7.

evidence in clear and accessible ways.”16 All research and evaluation activities must adhere to IES standards and expectations (see Chapters 3 and 4). REL staff can learn more about current projects, priority areas, and study and project designs across the REL program by referring to Task 4 abstracts and research questions for REL works in progress.17 Each REL webpage features a news section that provides abstracts and newsflashes for recently released products. RELs also provide links to external resources on their webpages.

• Task 5: Disseminate technical support products and study findings. RELs disseminate the work developed as part of Tasks 3 and 4 and other work that meets IES standards for rigor and quality. REL products and findings are disseminated to educators, administrators, policymakers, and other key stakeholders in the region “in an accessible way” that conveys “information that is timely and meaningful to the relevant audiences.”18 The RELs reach stakeholders with strategies such as research-to-practice “bridge events;” webinars; in-person or phone meetings with state or district personnel; trainings; blogs and e-newsletters; and the development and dissemination of technical assistance briefs, data templates, or tools. RELs frequently post archived forum or webinar videos and presentation slides on their webpages.

Some RELs have created social media accounts or online discussion forums.

RELs also answer questions submitted through the “Ask A REL” link on their website by providing references, referrals, or brief responses.19 RELs can then publish the “Ask A REL” submission and approved response on their website.

Stakeholder feedback surveys (SFSs) are required for all dissemination products, activities, and events. The document entitled “Overview and Administration of Stakeholder Feedback Surveys” provides guidance and instructions on requirements for SFSs.20

• Task 6: Collaborate and coordinate with other RELs. This task is intended to promote efficiency in meeting regional needs, address national concerns as appropriate, prevent unnecessary duplication of activities, and ensure sharing and best use of available resources across regions. Collaboration includes working with other RELs and IES to ensure strategic deployment of resources across high-priority areas; connecting stakeholders to other technical assistance providers funded by ED, such as the Comprehensive Centers, for needs that are outside the scope of the REL work; and participating in national and local conferences.21 As for all tasks, staff should consult with their individual REL leadership team to determine REL-specific procedures regarding the activities that fall within Task 6.

REL “Working Groups” are one way that RELs collaborate across the REL Program.

Detailed information on all of the working groups is located in a Task 6 Working Group folder on the REL intranet, accessible to limited staff from each of the REL contractors.

16 Performance Work Statement for Regional Educational Laboratories (2012–17), page 5.

17 REL Lookup is a search tool designed to help you find REL publications or REL work in progress appropriate for your interests. Visit the “REL Lookup” webpage and select “ALL REL Work in Progress” in the “Search for:” filter at: http://ies.ed.gov/ncee/edlabs/projects/selector.asp.

18 Performance Work Statement for Regional Educational Laboratories (2012–17), page 5.

19 http://ies.ed.gov/ncee/edlabs/askarel/.

20 Available from the REL Resources web page: http://ies.ed.gov/ncee/edlabs/RELResources/index.asp.

21 Performance Work Statement for Regional Educational Laboratories (2012–17), page 5.

http://ies.ed.gov/ncee/edlabs/RELResources/index.asp http://ies.ed.gov/ncee/edlabs/askarel http://ies.ed.gov/ncee/edlabs/projects/selector.asp http://www.ies.ed.gov/ncee/edlabs/projects/selector.asp https://www.fbo.gov/utils/view?id=50688bfda3cc31d77469838a79649491

• Task 7: Management and reporting. This task calls for general project and financial management for the REL, which includes the creation of a governing board to help direct the REL’s activities and the formation of internal quality controls—for example, internal staff review processes and Technical Working Groups (TWGs). Staff should consult their own REL’s specific guidance for these internal processes because they may vary from REL to

REL.

• Task 8: Serve as a coordinating entity. After the REL contracts were awarded for 2012–17, a coordinating contractor was selected from the group of successful bidders who had also included a Task 8 proposal as part of their overall proposal. For the 2012–17 REL program, REL Mid-Atlantic was selected. The primary responsibility of the coordinating entity is to manage and organize cross-REL working group meetings, identify additional opportunities for collaboration, and develop an intranet to facilitate cross-REL communication.

Key REL tasks 1 through 5 are intended to work together. Ongoing needs sensing is expected to inform the work of the research alliances and to guide decisions about analytic and technical support as well as ongoing research and evaluation activities. The research alliances are expected to identify appropriate technical assistance and research projects and activities, participate in their implementation, and be the primary beneficiaries of the results. Dissemination of technical assistance and research results serves both regional and national needs and feeds back into ongoing needs-sensing activities.

REL Support and Monitoring Provided by IES

The RELs operate under contract to the U.S. Department of Education, through the Institute of Education Sciences (IES). Contracts are different from grants in a number of key ways (see Table 1). Specifically, grants can be thought of as “an award of financial assistance, the principal purpose of which is to transfer a thing of value from a federal agency to a recipient to carry out a public purpose of support or stimulation authorized by a law of the United States.”22

Conversely, contracts are agreements between the government and another entity “used to acquire property or services for the federal government’s direct benefit or use.”23 When the government contracts with a business, the federal government wants to ensure that it receives only high-quality goods. Therefore, the government employs Contracting Officer’s Representatives (CORs) to manage and inspect the quality of the work to ensure that the government purchases something of value (see Table 2). Current program authorizing legislation (P.L. 107–279) requires RELs to enter into contracts rather than grants or cooperative agreements.

IES is required to support and monitor all of its contracts (not just the RELs) according to the specifications contained in the Federal Acquisition Regulations (FAR).24 The FAR is a lengthy set of regulations that describes how employees of the federal government must direct, oversee, and approve all work done under contract to the U.S. government.

22 See http://www.grants.gov/web/grants/support/general-support/glossary.html#G .

23 Ibid.

24 Current FAR specifications can be found at http://www.acquisition.gov/far/ (see also ESRA http://ies.ed.gov/director/pdf/ESRAreauth.pdf , P.L. 107).

http://ies.ed.gov/director/pdf/ESRAreauth.pdf http://www.acquisition.gov/far/ http://www.grants.gov/web/grants/support/general-support/glossary.html#G

Table 1. How Different Funding Mechanisms Vary in Purpose, Goal, and the Level of Government Involvement

Funding Reviewed for Grants Cooperative Agreements Contracts

Purpose: To transfer money, property, services, or anything of value to a recipient. (31 U.S.C. §6304)

To transfer money, property, services, or anything of value to a recipient. (31 U.S.C. §6305)

To obtain (purchase, lease, or barter) goods (property) or services (31 U.S.C. §6303)

Goal: To accomplish a public purpose To accomplish a public purpose For the direct benefit of the U.S.

government.

Government Involvement:

Minor Substantial Major—government directs activity within the terms of the contract.

Source: https://www.cga.msu.edu/PL/Portal/DocumentViewer.aspx?cga=aQBkAD0AMgAxADQA, which also contains more information about the differences between a grant, cooperative agreement, and contract.

The 2012–17 RELs operate under firm fixed-price contracts, with the exception of travel costs, which are cost reimbursable. Each year, the REL submits an annual plan for work, along with the cost of each “deliverable.” Within each REL project, multiple deliverables are specified—for example, events such as governing board meetings, webinars, concept papers, proposals, draft and final research reports, tools or additional disseminated products, monthly reports, event summary reports, annual progress assessments, and annual plans. When these deliverables are “recommended for quality acceptance” by the COR, a REL may include the cost in its monthly invoices. The contractor is not allowed to issue an invoice for the cost of the deliverable if it is not recommended for acceptance.

Table 2. Roles of the Contracting Officer and Contracting Officer’s Representative

Position Office within

ED

Role

Management ensuring performance of all necessary actions for effective contracting and ensuring (CAM) compliance with the terms of the contract.

Contracting IES The CO delegates specific contract administrative functions to a representative of the Officer’s program office who has functional or technical expertise to oversee the substantive work Representative described in the contract. This individual is the Contracting Officer’s Technical (COR) Representative (COTR) or Contracting Officer’s Representative (COR). The COR recommends acceptance of deliverables to the CO.

Source: Internal Department of Education document entitled Acquisition and Procedures Manual.

https://www.cga.msu.edu/PL/Portal/DocumentViewer.aspx?cga=aQBkAD0AMgAxADQA

REL COR

As described in each REL contract (and required by the FAR), each REL COR is responsible for overseeing all of the work of his or her REL or RELs. This oversight includes approving all plans and materials that will be used for research, technical assistance, events, and research alliance activities. REL CORs represent the interests of the U.S. government (the entity that “owns” all of the deliverables produced under contract) and ensures that the REL and IES “brands” meet high standards for rigorous and relevant work. CORs have regular phone calls and email communication with the REL leadership teams to provide ideas, guidance, feedback, and approval for all of the work that RELs undertake. When a deliverable is submitted to a COR and the COR finds it acceptable (which can occur after a COR-only review, a second COR review, a review by the associate commissioner, or completion of the REL Peer Review process—all depending on the work), the COR will recommend this deliverable for acceptance to the CO, who typically follows the recommendation of the COR. CORs tend to be individuals with substantial training and experience in educational research or a related field. Each REL also has an alternate COR (REL ACOR) to provide backup and support to the COR by participating in check-in calls, reviewing deliverables, and acting as a general collaborator.

Task 8 REL COR. One of the REL CORs serves as the “Task 8” COR. The Task 8 contractor (one of the RELs) is responsible for cross-REL coordination (see the preceding section for details).

Other Key Personnel at IES Who Interact with the REL Program

The IES REL team consists of the following additional NCEE staff who work together closely to support the work of the RELs:

• NCEE Commissioner

• Associate Commissioner for Knowledge Utilization (KU)

• REL Peer Review (RPR) COR

• Analytic Technical Assistance and Development (ATA&D) COR

• NCEE Communications and Support Coordinator

The entire IES REL team works together closely, drawing on the strengths of the different members of the team. The team also relies regularly on other IES staff and offices, including the IES website team, the Standards and Review Office (SRO), and the executive office for reporting on spending for information technology (IT), annual budget planning, and similar administrative tracking requirements. Each of the preceding key roles (as related to the REL Program) is described briefly in the following sections.

NCEE Commissioner

The NCEE Commissioner oversees all programs in NCEE and approves all REL reports. The commissioner provides vision, direction, and oversight to the associate commissioner, the REL CORs, and the REL Program at large.

Associate Commissioner for Knowledge Utilization

The associate commissioner oversees programs in the KU division of NCEE, including the REL Program. The associate commissioner provides direction and oversight to the REL CORs, the

RPR COR (described in the following section), and the ATA&D COR (described in a following section); approves all REL research proposals; and recommends approval for final reports to the commissioner. The associate commissioner sends a weekly email to all REL directors (the “Directors Email”) with information and REL Program guidance. Current REL staff can view archived weekly emails on the REL Intranet.

REL Peer Review (RPR) COR

According to the Education Sciences Reform Act (ESRA), all IES research must be peer reviewed.25 REL research is peer reviewed mainly through the REL Peer Review (RPR) process, which is performed under contract to IES. The RPR COR oversees this contract (which is one of two “support” contracts for the REL Program) and the peer review process. The RPR contractor provides reviews of all REL proposals and reports and makes recommendations to the RPR COR about areas to be fixed, clarified, considered, or addressed later. The reviews are conducted by a combination of content and methods experts who have no conflicts of interest related to the product being reviewed. The peer review focuses on relevance and significance of the product, the data and analysis methods used, and the presentation of the goals, data, methods, and findings. The RPR COR reviews reports and provides all IES and peer-review comments to the REL COR who provides them to the REL. The RPR COR tracks all proposals and publications in the pipeline, primarily through the RPR SharePoint system, an online file-sharing system that facilitates efficient and well-organized review and documentation of drafts of REL products.

More information on the standards used by the RPR reviewers is provided in Chapter 4, and more information about the RPR SharePoint system is provided in the Chapter 3 section entitled “Navigating the REL Review Process.”

Analytic Technical Assistance and Development COR

The Analytic Technical Assistance and Development (ATA&D) contract is the second of two support contracts for the REL Program, and work under this contract also provides support across NCEE. The ATA&D contract (1) provides technical assistance to the RELs and to NCEE in an effort to promote and support quality research and effective communication and (2) develops methodological publications and products to advance the field. The ATA&D team solicits input from the REL directors, REL CORs, and NCEE leadership when developing their annual plan. All work is done under the oversight of the ATA&D COR, who is also part of the IES REL Team. ATA&D guidance documents and webinar recordings for the REL program are available on the REL Intranet.

NCEE Communications and Support Coordinator

The NCEE communications and support coordinator is part of the IES REL team and helps to coordinate IES REL newsflashes, IES Web updates, calendar events on the IES website, tracking

25 All IES funded research is required to undergo external peer review. The ESRA requires that “all research, statistics, and evaluation reports conducted by, or supported through, the Institute shall be subjected to rigorous peer review before being published or otherwise made available to the public.” In addition, the Act requires that IES products be “objective, secular, neutral, and non-ideological and are free of partisan political influence and racial, cultural, gender, or regional bias.” For select publications (primarily randomized controlled trial studies) the IES Standards and Review Office (SRO) reviews reports that are intended for official IES publication. See http://ies.ed.gov/director/sro/.

http://ies.ed.gov/director/sro/ of IT spending, acquisition management (including incremental funding, contract modifications, and so forth), and other IES-specific REL dissemination and communication efforts.

NCEE Requirements

Because RELs work under contract, NCEE has approval procedures in place to ensure that REL products are aligned with IES’s standards and expectations. This guide provides sources that clarify IES’s standards, expectations, and review procedures. The text box “Frequently Asked Questions for the NCEE Approval Process” provides a basic list of frequently asked questions regarding NCEE approval processes.

Frequently Asked Questions for the NCEE Approval Process

Q: What deliverables need IES approval, and who gives the approval?

A: REL CORs review and recommend for approval all deliverables including, for example, monthly reports, concept papers, proposals, and reports. Most task 3 and 4 proposals and reports are also reviewed through the REL peer review (RPR) process, which is described in detail in Chapter 3.

Q: What is the timeline for approval?

A: Because REL projects vary in scope and content, the approval process and timeline for approval of REL activities and products varies. The timeline for each project should be discussed with the REL COR.

Q: How do new projects get approved?

A: In general, RELs must first identify a challenge (usually based on alliance interest) and produce a concept paper and project plan that must be approved by the REL COR. For research-related Task 3 and Task 4 projects, after the REL COR approves of the concept paper, RELs must then produce a formal proposal, which is reviewed through the RPR process. REL CORs must approve the proposal before work on the project can commence.

Q: When is OMB approval required?

A: As part of the Paperwork Reduction Act, data collection efforts that include more than nine subjects require OMB approval. RELs cannot commence data collection until OMB provides clearance for each data collection effort.

Q: Can RELs provide anything to the public without first receiving IES approval?

A: No! Any publicly available information, for example, Facebook or Twitter posts or electronic mail Newsflashes, must be approved by the REL COR.

Chapter 2. Research Alliances As discussed in Chapter 1, the defining characteristic of the 2012–17 REL Program is the requirement to work with, and in support of, REL research alliances. This chapter focuses on how this process typically operates and the implications for developing and implementing research and technical assistance projects.

What Is a Research Alliance?

As defined in the Performance Work Statement, a research alliance is a “group of stakeholders who share a specific education concern and agree to work together to learn more about the concern so that they can make sound decisions to improve education outcomes.”26 The number, structure, size, and focus of each alliance are determined by individual RELs to best meet the needs of their region and in consideration of their ability to provide high-quality services to the selected alliance. IES guidelines make the following suggestions:27

• “Research alliances are intended to maximize the number of states and districts within the region that each REL can serve by linking together entities with similar data, research, and evaluation needs.”

• “RELs are encouraged to form regional, cross-state, or cross-district research alliances where appropriate, and/or to partner with existing large-scale alliances.”

• “In order to ensure equitable service within a region, RELs shall ensure that each state is represented in at least one large-scale alliance. As appropriate, RELs should seek to involve relevant state agencies in alliances.”

As part of Task 2, REL staff are expected to work with research alliance members to connect stakeholders around regional educational issues and to develop research and dissemination agendas that address these issues. The REL is expected to support each alliance by (1) identifying and providing appropriate technical assistance (Task 3); (2) conducting and supporting research and evaluation studies (Task 4); and (3) disseminating useful and accessible products based on the research (Task 5). Alliance members are expected “to learn from each other, as well as from REL staff and to develop ongoing collaborative structures or contacts that can persist without ongoing REL support.”28 Research alliances can expand or contract, or they can terminate and be replaced by new alliances as regional needs and capacity change over time.

Appendix C provides a list of the 79 REL research alliances as of December 2014.

How Do the RELs Determine Their Yearly Work?

In the fall of each year, RELs are required to submit their proposed work for the coming year as part of their “updated annual plan” that describes the previous year’s accomplishments and articulates the projects and services that the REL proposes to continue or begin in the coming year.

26 Performance Work Statement for Regional Educational Laboratories (2012–17).

27 Ibid.

28 Ibid.

The plan for research activities and other REL services is typically derived from three sources:

(1) regional “needs sensing” that is generally a result of a variety of communication and data-gathering efforts by a variety of education leaders at the state and district levels; (2) input from each REL’s governing board, made up of regional education leaders that oversee its activities and provide guidance about the most pressing needs for research and technical assistance in the region; and (3) ongoing collaboration with members of their research alliances. IES reviews these proposed plans, provides comments and questions, and (once the revised updated annual plan is approved), determines the REL’s work plan for the year.

How Do RELs Work with Research Alliances?

According to the Performance Work Statement (PWS) for the REL contracts, RELs have the following goals:

• “Development of a cohesive and potentially deep body of knowledge in core priority topics that address regional, and often national, needs;

• Increased use of evaluation, data, and analysis by educators and education policymakers to identify problems, choose programs and strategies, and learn from initiatives;

• Completion of a range of rigorous evaluation and research studies that are methodologically appropriate to the questions that the studies attempt to answer;

• Expansion of state and local capacity to use their own data, conduct high-quality research and evaluation, and appropriately incorporate findings into policy and practice;

• Distribution of REL work across the region with a transparent and equitable process for determining where REL assistance is applied; and

Top Five REL Resources

#2 Your REL’s annual plans (current and past)

• Establishment of strong partnerships among practitioners, policymakers, and researchers who are not dependent on ongoing REL support.”29

How each REL seeks to achieve these goals varies in terms of internal organization structures, staffing arrangements, individual roles and responsibilities, operating procedures, and communication mechanisms. Consequently, no single set of guiding principles is completely applicable to each regional setting. Each REL website provides general descriptions of the alliances that the REL works with, and some RELs have provided specific guidance related to engaging with alliances (see Table 3 for links to these resources). In addition, a working group associated with a cross-REL Research Alliance holds regular conference calls to share information related to working with and maintaining relationships with research alliance members. Individual REL staff should consult with their colleagues for REL-specific protocols and procedures.

29 Ibid.

Table 3. Information and Resources for REL-Specific Research Alliances

Regional Laboratory Key Links and Resources Related to Research Alliances

REL Appalachia http://www.relappalachia.org/alliances/

REL Central http://www.relcentral.org/about-our-research-alliances/

REL Mid-Atlantic https://www.relmidatlantic.org/research-alliances

REL Midwest http://www.relmidwest.org/research-alliances

REL Northeast and Islands http://www.relnei.org/research-alliances.html

REL Northwest http://relnw.educationnorthwest.org/research-alliances

REL Pacific http://relpacific.mcrel.org/research-alliances.html

REL Southeast http://rel-se.fcrr.org/research-alliances/

REL Southwest http://relsouthwest.sedl.org/research-alliances/

REL West http://relwest.wested.org/alliances http://www.relappalachia.org/alliances/ http://www.relcentral.org/about-our-research-alliances/ https://www.relmidatlantic.org/research-alliances http://www.relmidwest.org/research-alliances http://www.relnei.org/research-alliances.html http://relnw.educationnorthwest.org/research-alliances http://relpacific.mcrel.org/research-alliances.html http://rel-se.fcrr.org/research-alliances/ http://relsouthwest.sedl.org/research-alliances/ http://relwest.wested.org/alliances

Chapter 3. Standards and Expectations for Writing and Communication NCEE is committed to ensuring that REL products are clear, engaging, and useful to their stakeholders and audience. To that end, IES has provided resources to the RELs that clarify expectations for writing, communication, and dissemination. This chapter summarizes the key resources provided by IES and provides basic information about other guidance documents and papers available on the REL intranet that give concrete and annotated examples or guidance on specific types of dissemination products (for example, newsflashes and abstracts). Unless otherwise noted, writing-related guidance referenced in this chapter is currently available to REL contractors on the REL intranet.

Reports and Other Public Documents

RELs report and communicate findings from Task 3 and Task 4 efforts in a number of ways. In general, published reports fall under seven product lines (Exhibit 2).

RELs also produce other written or online documents aimed at dissemination of project findings;

many of those documents are companions to published reports. In particular, RELs are required to produce newsflashes and abstracts for each report. RELs also create “tweets” or other products for social media. RELs may also produce Stated Briefly publications that are often companion pieces to other reports.

The REL Program Writers…