PWS Attachment 2 REL Research Proposal Templates.docx

Applied Research (Subtask 5.1 and Subtask 5.6) Concept Paper and Proposal Template

This template is used for both the “concept paper” and “proposal” phases of reviewing REL Applied Research projects. Fill out only those sections in gray for the concept paper phase of review. All sections of the template must be completed in the proposal phase of review. Note that all relevant information including both timeline and cost should also be updated in the proposal phase of the review.

While there is no page limit for applied research proposals, IES’ expectation is that these proposals will include the requested information as completely and concisely as possible to expedite review and approval. When possible, proposals should be limited to 25 pages of main text (excluding timeline, budget, references, and appendices).

Project Title:

High-leverage need or problem. State the high-leverage need for the study or the high-leverage problem the project will address, as determined in partnership with stakeholders. Describe the short- and medium-term partnership goals the study seeks to achieve, and how the study will address the stated need or problem. If the work is proposed with an existing partnership, note whether these short- and medium-term goals of this work are not already included in the partnership’s logic model. If they are not, briefly explain how they fit with the existing goals of the partnership. (1-2 paragraphs)

Audience and actionability. Describe the primary audience for the project (e.g., specific individuals, groups, SEAs and/or LEAs), and any possible actions by the audience that would be informed by the results of the study.

Evidence Base: Describe the existing evidence base for this study, and how this study will contribute to the evidence base to support the needs of partners. Include appropriate citations. (1-2 paragraphs, excluding references)

Describe the evidence base of the Applied Research project in greater detail, expanding on the content included in the concept paper. Include a general description of the quality of this evidence base (i.e., what types of research studies – causal, correlational, case study? – underpin this evidence base?) for each proposed research question. Describe how this study will contribute to the evidence base to support the needs of partners. Include appropriate citations.

Research questions: List the research questions to be addressed by the study in bullet points.

Data sources and sample: Provide an overview of the data sources that will be used in the study and provide an overview of the sample. (1 paragraph)

Describe the data source(s) that will be used to address each research question in greater detail, expanding on the information in the concept paper. Provide an assessment of the likelihood of securing necessary data. Include the outcome variables and the sample that each data source covers (grades, school years, etc.). If the data includes a sample, rather than universe of participants, please discuss the representativeness of the sample and plans to examine nonresponse bias. If the proposed project includes data collection, please provide an overview of the proposed data collection instrument(s) and include the draft instrument(s) in an appendix.

Outline how the study team will manage data, ensure data security, and follow applicable records management and privacy laws. If applicable, outline the plan to secure Office of Management and Budget approval for information collection and submit the final study to the Registry of Efficacy and Effectiveness Studies database and What Works Clearinghouse.

Research design and analysis methods: Describe the methods you plan to use to analyze the data, including justification for these methods. (2 paragraphs)

Describe the research design and analysis methods that will be used to address each research question in greater detail, expanding on the information in the concept paper. Provide an explicit description of analyses that will be used in the study, organized by research question. Describe all quantitative and qualitative analyses, any literature reviews or meta-analyses, and any alignment (of standards, curricula, assessments) analyses.

Challenges and limitations. Provide a brief overview of any challenges or limitations you anticipate in the study. (2-3 sentences)

Describe the challenges and limitations in greater detail, expanding on the content included in the concept paper. Please provide a more detailed description of challenges that could occur in this study and how those challenges will be mitigated.

Participation by families/community stakeholders (Subtask 5.6 projects only):

Describe the stakeholder group(s)—such as students, parents, or community members—that will be recruited for this project and explain why this group(s) was selected. In addition, describe the recruitment and selection process.

Briefly describe how the stakeholder group(s) will be involved in shaping the project from design to dissemination. If the REL plans to involve the stakeholder group(s) in the development of the proposal, please fully describe how they will be involved. Describe the potential challenges to engaging the stakeholder group(s) and how the REL will mitigate such challenges.

Participation by families/community stakeholders, expanding on the content included in the concept paper (Subtask 5.6 projects only):

Describe the stakeholder group(s) that will be recruited for this project and the recruitment and selection process. Describe how the stakeholder group(s) will be involved in shaping the project from design to dissemination, including how the REL will mitigate potential challenges, expanding on the content included in the concept paper. If applicable, describe how the stakeholder group(s) contributed to the development of this proposal.

Project monitoring metrics – Submit a list of metrics that the REL theorizes will be important indicators of progress towards the success of the project. In most cases, success will be defined as the project contributing to the partnership meeting its short-, medium-, and long-term outcomes. These metrics will be reported on in the Monthly Progress Report’s “Partnership and Project Progress Monitoring” spreadsheet. While the measurement plan measures whether an outcome was achieved, these metrics are formative assessments of progress towards the partnership’s outcomes or the success of the project. See the Monthly Progress Report guidance for project example items.

Plan to ensure the objectivity of the research (only for an efficacy evaluation of an intervention developed under Subtask 5.3). Such a plan might include how assignment of units to treatment and comparison conditions, supervision of outcome data collection and coding, and data analysis are assigned to persons who were not involved in the development of the intervention nor in the provision of any implementation support.

Reporting/dissemination: Describe the format for reporting findings from this study and justify how the chosen format will best serve the needs of both the stakeholders who requested the study and the broader research field. Options include an IES-published study report, or an alternate format such as a PowerPoint presentation. Identify the expected additional dissemination products that will be developed (for example, a blog, infographic, event, or companion product). Note: Dissemination products shall be developed under Subtask 6.2 and simply noted here.

Principal Investigator(s): Identify the individual(s) who will be primarily responsible for conducting this work and note their affiliation. The first Principal Investigator listed will be understood to be the first author on the report. If proposing a research synthesis,[footnoteRef:2] the Principal Investigator must be certified in What Works Clearinghouse Group Design Standards, Version 4.1. (1-2 sentences) [2: Refer to the “Consolidated Guidance on Research Syntheses” document to determine whether the proposed work constitutes a research synthesis.]

Subject Matter Expert (SME) Review. Identify the SME(s) name, title, affiliation, and 1-2 sentences describing the relevant experience or expertise. The SME(s) must be external to the project and cannot work on the project in any other capacity.

Data Management Plan (if applicable). All causal studies require a Data Management Plan (DMP) that will be published on the REL website. If your project requires a DMP, please add the draft DMP as an appendix. The DMP should contain the information below:

1. How the REL will obtain all data for the project. Is the REL going to collect the data, use administrative data, use data from some other source, or some combination?

1. The types of data that will be included in the data file. This should be a brief description of the variables that the REL expects to include in the final analytic sample. This should also include a brief, general description of the final sample whose data will be included (i.e., sample size, grade range, state(s) in which the study will take place).

1. Procedures for managing and for maintaining the confidentiality of Personally Identifiable Information (PII), if applicable.

1. Whether the final data file will be a restricted use file, a public use file, or use the alternate template (see forthcoming Open Science Guidance document for more information on these options.)

1. Any circumstances that prevent all or some of the primary data from being made publicly accessible (either as a public-use file or a restricted-use file). This includes data that may fall under multiple statutes and, hence, must meet the confidentiality requirements for each applicable statute (e.g., data covered by Common Rule for Protection of Human Subjects, FERPA, and HIPAA). This could also be a specific state and/or local statute or policy that prohibits the data from being shared. RELs are expected to fully explore viable options for data sharing. It is important to consider data sharing requirements in recruiting schools and organizations that will contribute to the study so that adequate IRB and consent processes can be established prior to any data collection or sharing.

Timeline. Complete the activity/milestone table below. Designate important milestones and expected submission dates for each deliverable (including but not limited to, project proposal draft, IRB approval, MOU signed and executed, data analysis completed, draft report submitted, report published). Designate which milestones will be submitted for IES review. Add additional years as needed. Update the timeline as needed when resubmitting for proposal review.

Research-Based Toolkits Based on WWC Practice Guides Proposal Template (Subtask 5.2): Toolkit Design, Development, and Usability

The toolkit proposal should be an updated and expanded version of the toolkit proposal that was submitted in response to the Request for Proposals. Please also reference the “Guidance for RELs to Create Spinoff Products Based on WWC Practice Guides” document for more detail about stating Practice Guide recommendations and interpreting implementation steps. Additional guidance on the toolkit projects is forthcoming.

Section 1. Toolkit Design, Development, and Usability Toolkit title.

Related WWC Practice Guide.

High leverage need. Describe the high leverage need for the proposed project and the student and educator outcomes designed to be affected by the toolkit. If the toolkit is focused on a certain student subgroup (e.g., low-income students), justify the choice.

Primary audience. Describe the primary audience for the toolkit and any secondary audiences for whom materials will be developed (e.g., families):

Recommendations included and justification. Describe each recommendation from the Practice Guide that will be addressed in the toolkit and their relevance to the high-leverage need identified above. If the REL does not propose to use all recommendations in a Practice Guide to develop a toolkit, please note the rationale behind this decision.

Toolkit logic model. Depict core components of the toolkit in the logic model, lettering or numbering each for purposes of identification. Depict relationships between core components of the toolkit and the outcomes they are meant to affect.

Toolkit structure and content. Provide an overview of the structure and content of the toolkit that:

· References the logic model developed above as appropriate.

· Includes the participant learning outcomes associated with each component described in the PWS (initial diagnostics, professional development resources, steps for institutionalizing supports, and on-going monitoring instruments) and specific learning artifacts that will be developed (e.g., videos, slide decks, infographics, practice audits).

· Indicates the end user for each component and any relevant secondary users (e.g., families), and how the end user will use each component. For example, if the toolkit requires expert or peer facilitation, describe the role of the facilitator.

· Includes pacing of each component within the toolkit, and the length of time needed to implement the entire toolkit and its components.

· Describes how the roadblocks to implementing the practice guide recommendations will be incorporated into the toolkit.

Usability testing. Describe the process for usability testing on the toolkit, with an emphasis on the methods and tools that will be used. Describe the sample that will be recruited for usability testing, and why the sample reflects likely users of the toolkit. For an overview of usability testing tools and methods, visit www.usability.gov.

Toolkit development and improvement. Describe how the toolkit will be developed and how those results will be used to improve the design of the toolkit, along with a timeline of major milestones in the design process. Describe the partner(s) that will be involved in the development of the toolkit and their role throughout the process.

IT/interactive components. Describe any IT or interactive components of the toolkit, including any components that will be posted on the IES website or any components that will collect user data. IT components may include interactive modules, quizzes, or webpages. These components will be created by the REL Website contractor.

REL support of toolkit dissemination. Describe the steps the REL will take to support broad dissemination of the toolkit. Identify the expected additional dissemination products that will be developed (for example, a blog, infographic, event, or companion product). Note: Dissemination products shall be developed under Subtask 6.2 and simply noted here.

Project lead. Confirm that the person listed as the project lead from your contract proposal is still proposed as the project lead. If the proposed project lead has changed, please contact your COR immediately.

SME for Toolkit Development. Provide the name, title, affiliation for the individual that will serve as the SME for the toolkit development project. Include a short (250 word) description of their relevant experience and expertise to serve as the SME for this project.

Section 2. Timeline, Deliverables, References, and Costs Timeline for this project. Include dates for submitting the proposal, key milestones in toolkit development, submitting the usability memo, and key evaluation milestones. RELs must submit a combined timeline for the toolkit development and evaluation projects.

Activity/Milestone

Research-Based Toolkits Based on WWC Practice Guides Proposal Template (Subtask 5.2): Toolkit Efficacy Evaluation

The toolkit proposal should be an updated and expanded version of the toolkit proposal your REL submitted in response to the Request for Proposals.

Note: IES anticipates that, at a minimum, the efficacy and implementation evaluation undertaken under Subtask 5.1 will require Information Collection Clearance (ICR) from the Office of Management and Budget. The questions below should inform, but will not fully address, requirements for preparing a complete ICR package. Usability testing under subtask 5.2, if done at significant scale (n > 9), may also require OMB clearance; however, this may not be feasible given the time necessary to develop and evaluate a toolkit.

Section 1. Toolkit Evaluation Design Research questions. Describe the proposed research questions to be tested in the required evaluation of efficacy and implementation.

Evaluation approach. Describe your proposed approach to the required evaluation. For each evaluation question, include information regarding your sample(s) and the population to which your sample is meant to generalize; how that sample will be selected and recruited, including any compensation offered to participants; data collection(s), including data sources, collection mechanisms, and instrumentation; hours of burden to students, teachers, or others providing information to the evaluation; and steps undertaken to refine information collection procedures from sample members that reduce burden and improve quality. If your toolkit requires facilitation, identify what facilitators will be used during the efficacy trial and how they will be recruited and trained. To expedite the OMB approval process for data collection, RELs are strongly encouraged to coordinate and identify similar data collection methods and items that could be employed across multiple evaluations. RELs are also strongly encouraged to identify existing usability or user experience measures for their proposed studies through resources such as usability.gov.

Analytic approach. Describe your analytic approach and how the proposed approach will yield a credible estimate of efficacy and assessment of implementation. Identify your inferential framework (e.g., Fisher’s, Neyman-Pearson, null hypothesis significance testing, Bayesian), the key statistical assumptions on which your efficacy analysis will be based and justify why they should be considered valid. Identify the strengths and weaknesses of your approach, justifying how the former outweighs the latter.

Data Management Plan (if applicable). All causal studies require a Data Management Plan (DMP) that will be published on the REL website. Please add the draft DMP as an appendix. The DMP should contain the information below:

1. How the REL will obtain all data for the project. Is the REL going to collect the data, use administrative data, use data from some other source, or some combination?

2. The types of data that will be included in the data file. This should be a brief description of the variables that the REL expects to include in the final analytic sample. This should also include a brief, general description of the final sample whose data will be included (i.e., sample size, grade range, state(s) in which the study will take place).

3. Procedures for managing and for maintaining the confidentiality of Personally Identifiable Information (PII), if applicable.

4. Whether the final data file will be a restricted use file, a public use file, or use the alternate template (see forthcoming Open Science Guidance document for more information on these options.)

5. Any circumstances that prevent all or some of the primary data from being made publicly accessible (either as a public-use file or a restricted-use file). This includes data that may fall under multiple statutes and, hence, must meet the confidentiality requirements for each applicable statute (e.g., data covered by Common Rule for Protection of Human Subjects, FERPA, and HIPAA). This could also be a specific state and/or local statute or policy that prohibits the data from being shared. RELs are expected to fully explore viable options for data sharing. It is important to consider data sharing requirements in recruiting schools and organizations that will contribute to the study so that adequate IRB and consent processes can be established prior to any data collection or sharing.

Premortem. Imagine that, at the end of the contract cycle, this project has not provided evidence of efficacy along one or more dimensions. In which potential areas is this most likely, and what are their likely causes? How have you adjusted your design and evaluation proposal, above, to mitigate them?

Plan to ensure the objectivity of the research. Such a plan might include how assignment of units to treatment and comparison conditions, supervision of outcome data collection and coding, and data analysis are assigned to persons who were not involved in the development of the toolkit nor in the provision of any implementation support. Additional guidance on ensuring objective research is forthcoming.

Principal Investigator: Please confirm that the person listed as the project lead from your contract proposal is still proposed as the principal investigator. If the proposed principal investigator has changed, please contact your COR immediately.

SME for Toolkit Evaluation. Provide the name, title, affiliation for the individual that will serve as the SME for the toolkit evaluation. Include a short (250 word) description of their relevant experience and expertise to serve as the SME for this project.

Section 2. Timeline, Deliverables, References, and Costs Timeline for this project. Include dates for submitting the proposal, key milestones in toolkit development, submitting the usability memo, and key evaluation milestones. RELs must submit an updated version of the combined timeline for the toolkit development and evaluation projects submitted with the toolkit development proposal.

Activity/Milestone

Design and Test New Interventions to Improve Student Outcomes (Subtask 5.3) Concept Paper and Proposal Template

This template is used for both the “concept paper” and “proposal” phases of reviewing REL Projects under Subtask 5.3. Fill out only those sections in gray for the concept paper phase of review. All sections of the template must be completed in the proposal phase of review. Note that all relevant information including both timeline and cost should also be updated in the proposal phase of the review.

While there is no page limit for intervention proposals, IES’ expectation is that these proposals will include the requested information as completely and concisely as possible to expedite review and approval. When possible, proposals should be limited to 25 pages of main text (excluding timeline, budget, references, and appendices).

The required evaluation component of the intervention should be proposed under Subtask 5.1 and follow the Applied Research concept paper and proposal template. RELs must submit a separate concept paper for the intervention evaluation under Subtask 5.1 and submit it to the COR along with the intervention design and testing proposal.

Project Title:

High-leverage need or problem. Describe the high-leverage need for the work or the problem of practice to be solved, and how the proposed intervention will address partner or stakeholder needs. Describe the short- and medium-term goals the intervention seeks to achieve. Please include a draft logic model for the intervention. Note that due to the limited time in a REL cycle, the intervention should emphasize improvements in short-term outcomes. (1-2 paragraphs + logic model)

Evidence Base: Describe the existing evidence base for this intervention, and how this intervention will contribute to the evidence base to support the needs of partners and students or educators across your region. Include appropriate citations. (1-2 paragraphs, excluding references)

Describe the evidence base for the proposed intervention in greater detail, building from the information in the concept paper. Describe the existing evidence base for this intervention, expanding on the content included in the concept paper, including a general description of the quality of this evidence base (i.e., what types of research studies – causal, correlational, case study? – underpin this evidence base?). Describe how this intervention will contribute to the evidence base and support the needs of partners and students or educators across the region. Include appropriate citations.

Content. Describe the components of the intervention, the specific audience for the intervention and the size of the audience, and the outcomes the intervention seeks to improve. Include appropriate citations. (2-3 paragraphs, excluding references)

Format. Describe the format of the intervention. How will the intervention be delivered? What is the pacing of the intervention? Describe any interactive or IT components of the intervention (see definition of interactive from the Subtask 5.2 proposal template). If the intervention includes an IT component, the project will require coordination with the REL website contractor. (1-2 paragraphs)

Describe the content and format of the intervention in greater detail, expanding on the content included in the concept paper. Provide an overview of the structure and content of the intervention that:

· References the logic model developed above as appropriate.

· Includes the participant learning outcomes associated with each component and specific learning materials that will be developed (e.g., handouts, videos, slide decks, infographics, practice audits).

· Indicates the end user for each component and any relevant secondary users (e.g., families), and how the end user will use each component. For example, if the intervention requires expert or peer facilitation, describe the role of the facilitator.

· Includes pacing of each component within the intervention, and the length of time needed to implement the entire intervention and its components.

· Provides additional information as requested on any IT or interactive components of the intervention, including a plan to coordinate with the REL website contractor.

· Adds a “premortem.” Imagine that, at the end of the contract cycle, this intervention has not provided evidence of efficacy along one or more dimensions. In which potential areas is this most likely, and what are their likely causes? How have you adjusted your design to mitigate them?

Uniqueness of the proposed intervention. Describe any similar programs/practices that are currently available and justify the need for the new program or practice.

Intervention development, usability testing, and improvement. Describe how the intervention will be developed, how usability will be tested, and how those results will be used to improve the design of the intervention. When describing the process for usability testing, emphasize on the methods and tools that will be used. Describe the sample that will be recruited for usability testing, and why the sample reflects likely users of the intervention. For an overview of usability and user experience testing tools and methods, visit www.usability.gov.

REL support for intervention dissemination. Describe the steps you will take to support broad dissemination of the intervention. Identify the expected additional dissemination products that will be developed (for example, a blog, infographic, event, or companion product). Note: Dissemination products shall be developed under Subtask 6.2 and simply noted here.

Principal Investigator(s): Identify the individual(s) who will be primarily responsible for developing this intervention. The first Principal Investigator listed will be understood to be the first author on the intervention. (1-2 sentences)

Co-developers and other partners. Identify the agency, organization, or partner that will co-develop the intervention with the REL. Identify any other partners that will be involved in the intervention design, implementation, and usability testing.

Staffing. Identify the individual(s) who will be primarily responsible for conducting this work alongside the PI. Include the staff members’ name, title, and 1-2 sentences describing the relevant experience or expertise as it relates to the activity.

Subject Matter Expert (SME) Review. Identify the SME’s name, title, and 1-2 sentences describing the relevant experience or expertise. The SME must be external to the project and cannot work on the project in any other capacity.

Timeline. Complete the activity/milestone table below and designate which milestones will be submitted for IES review. All work under this Task must be completed within the five-year contract, including the review and publication of the intervention and study. Add additional years as necessary. The draft evaluation report under Subtask 5.1 should be submitted no later than September 2025 and should be included in this timeline. Update the timeline as needed when resubmitting for proposal review.

Develop Tools to Support Research Processes and Research-Based Practices (Subtask 5.4) Concept Paper and Proposal Template

This template is used for both the “concept paper” and “proposal” phases of reviewing REL Projects under Subtask 5.4. Fill out only those sections in gray for the concept paper phase of review. All sections of the template must be completed in the proposal phase of review. Note that all relevant information including both timeline and cost should also be updated in the proposal phase of the review.

While there is no page limit for tool proposals, IES’ expectation is that these proposals will include the requested information as completely and concisely as possible to expedite review and approval. When possible, proposals should be limited to 25 pages of main text (excluding timeline, budget, references, and appendices).

Project Title:

High-leverage need or problem. Describe the high-leverage need for the work or the problem of practice to be solved, and how the proposed tool will address partner or stakeholder needs, as well as the needs of students or educators across the region. Describe the short- and medium-term goals the tool seeks to achieve. (1-2 paragraphs)

Evidence Base: Describe the existing evidence base for this tool, and how this tool will contribute to the evidence base to support the needs of partners and students or educators across your region. Include appropriate citations. (1-2 paragraphs, excluding references)

Describe the evidence base for the proposed tool in greater detail, building from the information in the concept paper. Describe the existing evidence base for this tool, expanding on the content included in the concept paper, including a general description of the quality of this evidence base (i.e., what types of research studies – causal, correlational, case study? – underpin this evidence base?). Describe how this tool will contribute to the evidence base and support the needs of partners and students or educators across the region. Include appropriate citations.

Content. Describe the components of the tool, the specific audience for the tool and the size of the audience, and the outcomes the tool seeks to improve. Include appropriate citations. (2-3 paragraphs, excluding references)

Format. Describe the format of the tool. How will the tool be delivered? Describe any interactive or IT components of the tool (see definition of interactive from the Subtask 5.2 proposal template). If the tool includes an IT component, the project will require coordination with the REL website contractor. (1-2 paragraphs)

Describe the content and format of the tool in greater detail, expanding on the content included in the concept paper. Provide an overview of the structure and content of the tool that:

· Includes the participant learning outcomes associated with each component of the tool.

· Indicates the end user for each component and any relevant secondary users (e.g., families), and how the end user will use each component.

· Provides additional information as requested on any IT or interactive components of the tool, including a plan to coordinate with the REL website contractor.

Uniqueness of the proposed tool. Describe any similar tools that are currently available and justify the need for the proposed tool.

Tool development, usability testing, and improvement. Describe how the tool will be developed, how usability will be tested, and how those results will be used to improve the design of the tool. When describing the process for usability testing, emphasize on the methods and tools that will be used. Describe the sample that will be recruited for usability testing, and why the sample reflects likely users of the tool. For an overview of usability and user experience testing tools and methods, visit www.usability.gov.

REL support for tool dissemination. Describe the steps the REL will take to support broad dissemination of the tool. Identify the expected additional dissemination products that will be developed (for example, a blog, infographic, event, or companion product). Note: Dissemination products shall be developed under Subtask 6.2 and simply noted here.

Principal Investigator(s): Identify the individual(s) who will be primarily responsible for developing this tool. The first Principal Investigator listed will be understood to be the first author on the tool. (1-2 sentences)

Co-developers and other partners. Identify the agency, organization, or partner that will co-develop the tool with the REL. Identify any other partners that will be involved in the tool design, implementation, and usability testing.

Staffing. Identify the individual(s) who will be primarily responsible for conducting this work alongside the PI. Include the staff members’ name, title, and 1-2 sentences describing the relevant experience or expertise as it relates to the activity.

Subject Matter Expert (SME) Review. Identify the SME’s name, title, and 1-2 sentences describing the relevant experience or expertise. The SME must be external to the project and cannot work on the project in any other capacity.

Timeline. Complete the activity/milestone table below and designate which milestones will be submitted for IES review. All work under this Task must be completed within the five-year contract. Add additional years as necessary. Update the timeline as needed when resubmitting for proposal review.