Evidence Checklist Tier.pdf

Source: Education Department General Administrative Regulations (EDGAR) 34 CFR 77.1

Education Department General Administrative Regulations (EDGAR) Evidence Definitions

Available at: https://www.ecfr.gov/cgi-bin/text-idx?SID=393301a7cdccca1ea71f18aae51824e7&node=34:1.1.1.1.24&rgn=div5

Tiers of Evidence

Demonstrates a rationale means a key project component included in the project's logic model is informed by research or evaluation findings that suggest the project component is likely to improve relevant outcomes.

Promising evidence means that there is evidence of the effectiveness of a key project component in improving a relevant outcome, based on a relevant finding from one of the following:

(i) A practice guide prepared by WWC reporting a “strong evidence base” or “moderate evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC reporting a “positive effect” or “potentially positive effect” on a relevant outcome with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single study assessed by the Department, as appropriate, that—

(A) Is an experimental study, a quasi-experimental design study, or a well-designed and well-implemented correlational study with statistical controls for selection bias (e.g., a study using regression methods to account for differences between a treatment group and a comparison group); and

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome.

Moderate evidence means that there is evidence of effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations or settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “strong evidence base” or “moderate evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “positive effect” or “potentially positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study or quasi-experimental design study reviewed and reported by the WWC using version 2.1 or 3.0 of the WWC Handbook, or otherwise assessed by the Department using version 3.0 of the WWC Handbook, as appropriate, and that—

(A) Meets WWC standards with or without reservations;

relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version

2.1 or 3.0 of the WWC Handbook; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites.

Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy this requirement.

Strong evidence means that there is evidence of the effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations and settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “strong evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study reviewed and reported by the WWC using version 2.1 or 3.0 of the WWC Handbook, or otherwise assessed by the Department using version 3.0 of the WWC Handbook, as appropriate, and that—

(A) Meets WWC standards without reservations;

relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version

2.1 or 3.0 of the WWC Handbook; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites.

Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy this requirement.

Other Relevant Definitions

Evidence-based means the proposed project component is supported by one or more of strong evidence, moderate evidence, promising evidence, or evidence that demonstrates a rationale.

Experimental study means a study that is designed to compare outcomes between two groups of individuals (such as students) that are otherwise equivalent except for their assignment to either a treatment group receiving a project component or a control group that does not.

Randomized controlled trials, regression discontinuity design studies, and single-case design studies are the specific types of experimental studies that, depending on their design and implementation (e.g., sample attrition in randomized controlled trials and regression discontinuity design studies), can meet What Works Clearinghouse (WWC) standards without reservations as described in the WWC Handbook:

(i) A randomized controlled trial employs random assignment of, for example, students, teachers, classrooms, or schools to receive the project component being evaluated (the treatment group) or not to receive the project component (the control group).

(ii) A regression discontinuity design study assigns the project component being evaluated using a measured variable (e.g., assigning students reading below a cutoff score to tutoring or developmental education classes) and controls for that variable in the analysis of outcomes.

(iii) A single-case design study uses observations of a single case (e.g., a student eligible for a behavioral intervention) over time in the absence and presence of a controlled treatment manipulation to determine whether the outcome is systematically related to the treatment.

Logic model (also referred to as a theory of action) means a framework that identifies key project components of the proposed project (i.e., the active “ingredients” that are hypothesized to be critical to achieving the relevant outcomes) and describes the theoretical and operational relationships among the key project components and relevant outcomes.

Project component means an activity, strategy, intervention, process, product, practice, or policy included in a project. Evidence may pertain to an individual project component or to a combination of project components (e.g., training teachers on instructional practices for English learners and follow-on coaching for these teachers).

Quasi-experimental design study means a study using a design that attempts to approximate an experimental study by identifying a comparison group that is similar to the treatment group in important respects. This type of study, depending on design and implementation (e.g., establishment of baseline equivalence of the groups being compared), can meet WWC standards with reservations, but cannot meet WWC standards without reservations, as described in the WWC Handbook.

Relevant outcome means the student outcome(s) or other outcome(s) the key project component is designed to improve, consistent with the specific goals of the program.

What Works Clearinghouse Handbook (WWC Handbook) means the standards and procedures set forth in the WWC Procedures and Standards Handbook, Version 3.0 or Version 2.1 (incorporated by reference, see 34 CFR 77.2). Study findings eligible for review under WWC standards can meet WWC standards without reservations, meet WWC standards with reservations, or not meet WWC standards. WWC practice guides and intervention reports include findings from systematic reviews of evidence as described in the Handbook documentation.

**Note: Relevant evidence definitions will be updated to reflect WWC Version 4.0.