2011_OSEP_Evaluation_Report_Draft.docx

Annual OSEP/RTP Performance Measures—2011

ANNUAL EVALUATION REPORT

(PRELIMINARY DRAFT)

Submitted to:

Office of Special Education Programs Research to Practice Division U.S. Department of Education 550 12th Street, SW Washington, DC 20202

Submitted by:

11785 Beltsville Drive, Suite 300 Calverton, MD 20705

Revised Draft Submitted September 9, 2011

U.S. Department of Education Draft

U.S. Department of Education

U.S. Department of Education

TABLE OF CONTENTS

Introduction…………………………………………………………………………… Preliminary Results and Findings………….…………………………………………………………………………..3 Parent Training and Information Centers……………………………………… Personnel Development…………………………………..……………………………..13 Technical Assistance and Dissemination…………………………………………….23 Technology and Media…………………………………………………………….….….33 Suggestions for Improvement…………………………………………………………..44 Appendices

INTRODUCTION

In recent years, the Federal Government has increasingly recognized the importance of holding federal agencies accountable for program results and quality of service. As a result, the Government Performance and Results Act of 1993 (GPRA), and more recently the enactment of the GPRA Modernization Act of 2010 (GPRAMA), requires the Office of Management and Budget (OMB) to provide government-wide priority goals, increase the frequency and enhance quality of agency reporting, and improve the transparency of performance reporting. The new law also improves federal performance management by establishing a Chief Operating Officer (COO) within each agency tasked with improving the performance of an agency. It also codifies and strengthens existing resources for performance management, including the Performance Improvement Officers (PIOs) within the federal agencies and the interagency Performance Improvement Council (PIC). The Individuals with Disabilities Education Act (IDEA) Part D program performance measures authorized by the 1997 IDEA, and reauthorized in 2004, are administered by the Office of Special Education and Rehabilitation Services (OSERS) through the Office of Special Education Programs (OSEP). These measures are the focus of this evaluation.

The Division of Research to Practice (RTP) has sole responsibility for collecting, analyzing, and reporting data for the IDEA Part D program performance measures specified in the OSEP GPRA Plan. Part D programs are as follows: Personnel Development, Technology and Media Services, Technical Assistance and Dissemination, and Parent Training and Information Centers (the State Personnel Development Program is also an OSEP component, but it was not evaluated this year). OSEP’s GPRA Plan specifies discrete objectives for each of these programs. The Plan contains indicators and data sources to measure its progress in achieving each objective.

This report aims to fully describe the methods and results implemented for each indicator (annual as well as long-term), provide an assessment of each indicator, describe the changes in the indicator data from 2006 to 2011, discuss the strengths and limitations of each indicator and measure, and provide recommendations for future development.

PRELIMINARY RESULTS AND FINDINGS

The following is a summary of the measures, methods, scoring calculations, and results for all annual and long-term measures from the 2010 Part D Performance Measures Data Collection. The results are organized by program area. Appendices are provided that contain all correspondence and forms used for the Expert Panels, as well as other supplemental materials and information for each program.

IDEA: Special Education Parent Information Centers (OSERS)

FY 2011 Program Performance Report

Strategic Goal 1

Discretionary

IDEA, Part D-3, Sections 671–673

Document Year 2011 Appropriation: $

Measure 1.1 of 4: The percentage of materials used by Special Education Parent Information Centers projects deemed to be of high quality by an independent review panel of experts qualified to review the substantive content of the products or services. (Desired direction: Increase) 89a0e4

(or date expected) Status

Source: Description of Expert Panel Review—Two panels were involved in the review of products and services used by Parent Information Centers this year.

The Office of Special Education Programs (OSEP) provides an outside contractor with a list of 15–20 potential Parent Expert Panel members and 15–20 potential Science Expert Panel members. Potential Parent Expert Panel members are parents of children and young adults with a wide range of disabilities who also have professional or advocacy experience in special education, including serving as peer reviewers for the Parent Training and Information Program. Potential Science Expert Panel members are nationally known experts in special education research, policy, and/or practice. The contractor randomly selects, contacts, and secures the review services of six to eight (6–8) members for each panel, ensuring that no panelist is currently employed by an OSEP-funded program, grant, or project.

The Parent Expert Panel and the Science Expert Panel review a randomly selected sample of products and services made available through OSEP-funded Parent Information Centers (84.328 grantees). The program office defines a “product” as “a piece of work, in text or electronic form developed and disseminated by an OSEP-funded project to inform a specific audience on a topic relevant to the improvement of outcomes for children with disabilities.” A “service” is defined as “work performed by an OSEP-funded project to provide information to a specific audience relevant to the improvement of outcomes for children with disabilities.” In 2011, the Parent Expert Panel reviewed products and services identified as “policy-based” and the Science Expert Panel reviewed products and services identified as “evidenced-based.”

The grantees selected to provide data for this measure supply a copy of one product and one service for each panel member, along with Product and Service Description forms that explain each product and service. The purpose of the forms is to standardize the information used in the panel review. Information categories contained in the forms include product or service name, target audience, and classification of the product or service as evidence-based or policy-based.

Panelists independently assess the quality of each product and service according to criteria described in the Explanation section below.

Frequency of Data Collection: Annual Data Quality: This measure applies to all 84.328 grants (101 grants: 71 Parent Training and Information Center grants [84.328M] and 30 Community Parent Resource Center grants[84.328C]). Fiscal Year (FY) 2010 funding for these grants totaled $28,028,000.

The grant sample is selected by an outside contractor following a cohort design with parameters established by the program office. The cohort sample design ensures a random selection of a new group of grants annually while avoiding sample replication in any 2 consecutive years.

All 84.328 grants funded in the prior fiscal year (FY 2010) that were not included in the previous year’s sample are included in the sampling frame. Fifty percent (50%) of the eligible 84.328 grants are randomly selected. The sample is tested to ensure that it is representative of the number of 84.328M grants funded at each of three intervals (i.e., under $225,000, between $226,000 and $300,000, and $301,000 or more). The specific product or service from each grantee is chosen in the following way: The contractor requests a list of five products and services from each grantee and randomly selects one product and one service from each grantee’s list. The contractor then requests that the grantee send copies of the selected product and service for review, along with the Product and Service Description forms.

In 2011, forty-three (43) 84.328M grants were included in the sampling frame, as were twenty-two (22) 84.328C grants. The total number of grants in the 2011 sample for this measure was 33.

The contractor provided copies of each item to the Science Expert Panel and Parent Expert Panel, along with instructions for accessing, reviewing, and assessing the quality of each product and service. Panelists rated the new products and services using a Quality Assessment Scoring Guide described below.

The AM-ICC inter-rater reliability score for the Parent Training and Information Program product and service quality review was .443, representing moderate consistency across the reviewers.

Note: Values of AM-ICC from 0.40 to 0.50 are considered moderate, 0.51 to 0.70 are considered substantial, and 0.71 or higher are considered outstanding (Landis, J.R., Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics 33:159–174).

Target Context: Targets for this measure were established on the basis of 2 years of trend data from 2007 and 2008.

Explanation Description of Method: Panelists individually assess the quality of each product and service on two criteria: (1) Substance—Does the product reflect evidence of conceptual soundness and quality, grounded in recent scientific evidence, legislation, policy, or accepted professional practice (2) Communication—Is the product presented in such a way that it is clearly understood, as evidenced by being well-organized, free of editorial errors and appropriately formatted? The extent to which the product or service meets the substance criterion is measured using a 7-point scale ranging from 0 = Unacceptable to 1 or 2 = Low, 3 or 4 = Acceptable, and 5 or 6 = Superior. The extent to which the product or service meets the communication criterion is measured using a 4-point scale ranging from 0 = Unacceptable to 1 = Low, 2 = Acceptable, and 3 = Superior. Panelists complete the Quality Assessment Scoring Guide for each product and service. In 2011, products and services identified by grantees as “evidence-based” were reviewed by the Science Expert Panel. Products and services identified by grantees as “policy-based” were reviewed by the Parent Expert Panel.

In 2011, average panel scores of the quality of products and services ranged between 5.14 and 8.86. Products and services receiving an average total panel score of 6 or higher across the two criteria were deemed to be of high quality.

Description of Calculation: The calculation provides the proportion of products and services that the Science Expert Panel and the Parent Expert Panel deem to be of high quality. The calculation is as follows:

Total number of 84.328 grantee products and services with average panel scores of 6 or higher divided by the total number of 84.328 grantee products and services reviewed, times 100.

2011 Result = [59 products and services with average panel scores totaling 6 or higher / 66 products and services reviewed] x 100 = 89.4%

Measure 1.2 of 4: The percentage of Special Education Parent Information Centers products and services deemed to be of high relevance to educational and early intervention policy or practice by an independent review panel of qualified experts with appropriate expertise to review the substantive content of the products or services. (Desired direction: Increase) 89a0e5

(or date expected) Status

Source: Description of Expert Panel Review—OSEP provides an outside contractor with a list of 15–20 potential Parent Expert Panel members who are parents of children and young adults with a wide range of disabilities, and who have professional or advocacy experience in special education, including serving as peer reviewers for the Parent Training and Information Program. The contractor randomly selects, contacts, and secures the review services of 6–8 panel members, ensuring that no panelist is currently employed by an OSEP-funded program, grant or project.

The Parent Expert Panel reviews a randomly selected sample of products and services made available through OSEP-funded Parent Information Centers (84.328 grantees). The program office defines a “product” as “a piece of work, in text or electronic form developed and disseminated by an OSEP-funded project to inform a specific audience on a topic relevant to the improvement of outcomes for children with disabilities.” A “service” is defined as “work performed by an OSEP-funded project to provide information to a specific audience relevant to the improvement of outcomes for children with disabilities.”

The grantees selected to provide data for this measure supply a copy of one product and one service for each panel member, along with Product and Service Description forms that explain each product and service. The purpose of the forms is to standardize the information used in the panel review. Information categories contained in the forms include product or service name, target audience, and classification of the product or service as evidence-based or policy-based.

Panelists independently assess the relevance of each product and service according to criteria described below in the Explanation section.

Frequency of Data Collection: Annual.

Data Quality: This measure applies to all 84.328 grants (101 grants: 71 Parent Training and Information Center grants [84.328M] and 30 Community Parent Resource Center grants [84.328C]). FY 2010 funding for these grants totaled $28,028,000.

The grant sample is selected by an outside contractor following a cohort design with parameters established by the program office. The cohort sample design ensures a random selection of a new group of grants annually while avoiding sample replication in any 2 consecutive years.

All 84.328 grants funded in the prior fiscal year (FY 2010) that were not included in the previous year’s sample are included in the sampling frame. Fifty percent (50%) of the eligible 84.328 grants are randomly selected. The sample is tested to ensure that it is representative of the number of 84.328M grants funded at each of three intervals (i.e., under $225,000, between $226,000 and $300,000, and $301,000 or more). The specific product or service from each grantee is chosen in the following way: The contractor requests a list of five products and services from each grantee and randomly selects one product and one service from each grantee’s list. The contractor then requests that the grantee send copies of the selected product and service for review, along with the Product and Service Description forms.

In 2011, forty-three (43) 84.328M grants were included in the sampling frame, as were twenty-two (22) 84.328C grants. The total number of grants in the 2011 sample for this measure was 33.

The contractor provided copies of each item to the Parent Expert Panel, along with instructions for accessing, reviewing, and assessing the relevance of each product and service. Panelists rated the products and services using a Relevance Assessment Scoring Guide described below.

The AM-ICC inter-rater reliability score for the Parent Training and Information Program product and service relevance review was .550, indicating substantial consistency across the reviewers.

Note: Values of AM-ICC from 0.40 to 0.50 are considered moderate, 0.51 to 0.70 are considered substantial, and 0.71 or higher are considered outstanding (Landis, J.R., Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics 33:159–174).

Target Context: Targets for this measure were established on the basis of results for 2007 and 2008.

Explanation Description of Method: Panelists individually assess the relevance of each product and service on three criteria: (1) Need—Does the content of the product or service attempt to solve an important problem or critical issue (2) Pertinence—Does the content of the product or service tie directly to a problem or issue recognized as important by the target audience and (3) Reach—To what extent is the content of the product or service applicable to diverse segments of the target audience? The extent to which the product or service meets each criterion is measured using a 4-point scale ranging from 0 = Unacceptable to 1 = Low, 2 = Acceptable, and 3 = Superior. Panelists complete the Relevance Assessment Scoring Guide for each product and service.

In 2011, average panel scores of the relevance of products and services ranged between 5.57 and 9.00. Products and services receiving an average total panel score of 6 or higher across the three criteria were deemed to be of high relevance.

Description of Calculation: The calculation provides the proportion of products and services that are deemed by the Parent Expert Panel to be of high relevance. The calculation is as follows:

Total number of 84.328 grantee products and services with average panel scores of 6 or higher divided by the total number of 84.328 grantee products and services reviewed, times 100.

2011 Result: [65 products and services with average panel scores totaling 6 or higher / 66 products and services reviewed] x 100 = 98.5% Measure 1.3 of 4: The percentage of all Special Education Parent Training and Information Centers products and services deemed by an independent review panel of qualified experts to be useful to improve educational or early intervention policy or practice. (Desired direction: increase) 1953

(or date expected) Status

Source: Description of Expert Panel Review—OSEP provides an outside contractor with a list of 15–20 potential Parent Expert Panel members who are parents of children and young adults with a wide range of disabilities, and who have professional or advocacy experience in special education, including serving as peer reviewers for the Parent Training and Information Program. The contractor randomly selects, contacts, and secures the review services of 6–8 panel members, ensuring that no panelist is currently employed by an OSEP-funded program, grant, or project.

The Parent Expert Panel reviews a randomly selected sample of products and services made available through OSEP-funded Parent Information Centers (84.328 grantees). The program office defines a “product” as “a piece of work, in text or electronic form developed and disseminated by an OSEP-funded project to inform a specific audience on a topic relevant to the improvement of outcomes for children with disabilities.” A “service” is defined as “work performed by an OSEP-funded project to provide information to a specific audience relevant to the improvement of outcomes for children with disabilities.”

The grantees selected to provide data for this measure supply a copy of one product and one service for each panel member, along with Product and Service Description forms that explain each product and service. The purpose of the forms is to standardize the information used in the panel review. Information categories contained in the forms include product or service name, target audience, and classification of the product or service as evidence-based or policy-based.

Panelists independently assess the usefulness of each product and service according to criteria described below in the Explanation section.

Frequency of Data Collection: Annual Data Quality: This measure applies to all 84.328 grants (101 grants: 71 Parent Training and Information Center grants [84.328M]and 30 Community Parent Resource Center grants[84.328C]). FY 2010 funding for these grants totaled $28,028,000.

The grant sample is selected by an outside contractor following a cohort design with parameters established by the program office. The cohort sample design ensures a random selection of a new group of grants annually while avoiding sample replication in any 2 consecutive years.

All 84.328 grants funded in the prior fiscal year (FY 2010) that were not included in the previous year’s sample are included in the sampling frame. Fifty percent (50%) of the eligible 84.328 grants are randomly selected. The sample is tested to ensure that it is representative of the number of 84.328M grants funded at each of three intervals (i.e., under $225,000, between $226,000 and $300,000, and $301,000 or more). The specific product or service from each grantee is chosen in the following way: The contractor requests a list of five products and services from each grantee and randomly selects one product and one service from each grantee’s list. The contractor then requests that the grantee send copies of the selected product and service for review, along with the Product and Service Description forms.

In 2011, forty-three (43) 84.328M grants were included in the sampling frame, as were twenty-two (22) 84.328C grants. The total number of grants in the 2011 sample for this measure was 33.

The contractor provided copies of each item to the Parent Expert Panel, along with instructions for accessing, reviewing, and assessing the usefulness of each product and service. Panelists rated the products and services using a Usefulness Assessment Scoring Guide described below.

The AM-ICC inter-rater reliability score for the Parent Training and Information Program product and service usefulness review was .613, indicating substantial consistency across the reviewers.

Note: Values of AM-ICC from 0.40 to 0.50 are considered moderate, 0.51 to 0.70 are considered substantial, and 0.71 or higher are considered outstanding (Landis, J.R., Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics 33:159–174).

Target Context: Targets for this measure were established on the basis of data collected in 2007 and 2008.

Explanation Description of Method: Panelists individually assess the usefulness of each product and services on three criteria: (1) Ease—Does the content of the product or service address a problem or issue in an easily understood way, with directions or guidance regarding how the content can be used to address the problem or issue (2) Pertinence—Does the content of the product or service tie directly to a problem or issue recognized as important by the target audience and (3) Replicability—Is it a likely that the information derived from the product or service will be used repeatedly in multiple setting to achieve the intended benefit? The extent to which the product or service meets each criterion is measured using a 4point scale ranging from 0 = Unacceptable to 1 = Low, 2 = Acceptable, and 3 = Superior. Panelists complete the Usefulness Assessment Scoring Guide for each product and service.

In 2011, average panel scores of the usefulness of products and services ranged between 4.71 and 8.71. Products and services receiving an average total panel score of 6 or higher across the three criteria were deemed to be of highusefulness.

Description of Calculation: The calculation provides the proportion of products and services that are deemed by the Parent Expert Panel to be of high usefulness. The calculation is as follows:

Total number of 84.328 grantee products and services with average panel scores of 6 or higher divided by the total number of 84.328 grantee products and services reviewed times 100.

2011 Result = [61 products and services with average panel scores totaling 6 or higher / 66 products and services reviewed] x 100 = 92.4%.

Measure 1.4 of 4: The federal cost per unit of output provided by the Special Education Parent Training and Information Centers, by category. (Desired direction: decrease) 1954

(or date expected) Status

Source: The number of parents and professionals served by 84.328 grantees in each of the technical assistance categories is derived from the National Parent Training and Information Center‘s annual report, “Parents Helping Families: Outcome Data” which reports results of an annual survey of State Parent Training and Information Centers (PTIs) conducted by the Center.

Frequency of Data Collection: Annual.

Data Quality: The federal cost of technical assistance services rendered by the PTIs is currently expressed as a weighted index of cost per unit.

Technical assistance services provided by 84.328 grants are classified into four technical assistance categories, each of which is assigned an ‘ intensity‘ weight reflecting the amount of interaction and support required to render each service. The four Technical Assistance (TA) categories and their intensity weights (in parentheses) are: parents and professionals accessing website materials and receiving newsletters (1); parents and professionals served through letters, telephone calls, and e-mail exchanges (2); parents and professionals provided training, in-person participation in meetings and receiving home visits (3); and parents and professionals receiving extended services including grantee attendance or participation in IEP meetings, and participation in mediation sessions and resolution sessions (4).

Target Context: Targets for this measure were established on the basis of data collected in 2007. OSEP plans to revise this measure in 2012 to ensure that it yields more appropriate management information

Explanation Description of Calculation: The calculation of the federal index of cost per unit of technical assistance is the total 84.328 funding for a given fiscal year divided by the sum of units of technical assistance weighted by category. The result is expressed as an index of cost rather than a cost in dollars.

2011 Result = $28,028,000 / [19,600,299(1) +2,005,771(2)+402,417(3)+12,658(4)]

= $28,028,000 / 24,869,724

= 1.13

Measure 2.1 of 1: The percentage of parents receiving Special Education Parent Information Centers services who report enhanced knowledge of IDEA rights and responsibilities. (Desired direction: Increase) 1956

(or date expected) Status

Source: National Parent Technical Assistance Center Annual Parent Survey.

Frequency of Data Collection: Biennial.

Data Quality: Data for this long-term measure are collected through a telephone survey developed by the National Parent Technical Assistance Center and conducted by all 84.328M, 84.328R, and 84.328C grants, with a randomly selected sample of 25 parents who had received information from the grantee in person by telephone in the prior fiscal year.

The seven-item selected-response survey evaluates the quality of information and services parents receive. The values of “Yes” responses to three items on the survey are used to calculate the value of the present measure. The value of “Yes” responses to these items is weighted to reflect the relevance of these responses to “enhanced knowledge of rights and responsibilities” as follows: survey item #1 = weight of 1, survey item #4 = weight of 2, and survey item #6 = weight of 4.

Target Context: Targets for this measure were established on the basis of a 2009 validation study.

Explanation Description of the Calculation: The calculation for this measure is as follows:

The total number of parents responding “Yes” to survey item #1 plus the total number of parents responding “Yes” to item #4 (times 2), plus the total number of parents responding “Yes” to item #6 (times 4), divided by the total number of parents responding to item #1, plus the total number of parents responding to item #4 (times 2), plus the total number of parents responding to item #6 (times 4), times 100.

2011 Result = [3118 + 2630(2) + 2087(4) / 3294+ 3286(2) + 2480(4)] x 100 = [16,726/19,786] x 100 = 84.5%

Measure 3.1 of 1: The percentage of parents receiving Special Education Parent InformationCenters services who promote scientifically- or evidence-based practices for their infants, toddlers, children and youth. (Desired direction: increase) 000014

(or date expected) Status

Source: National Parent Technical Assistance Center Annual Parent Survey.

Frequency of Data Collection: Biennial.

Data Quality: Data for this long-term measure are collected through a telephone survey developed by the National Parent Technical Assistance Center and conducted by all 84.328M, 84.328R, and 84.328C grants with a randomly selected sample of 25 parents who had received information from the grantee in person on by telephone in the prior fiscal year.

The seven-item selected-response survey evaluates the quality of information and services parents receive. The values of “Yes” responses to four items on the survey are used to calculate the value of the present measure. The values of “Yes” responses to these items are weighted to reflect their relevance to “enhanced knowledge of rights and responsibilities” as follows: survey item #2 = weight of 1, survey item #3 = weight of 2, survey item #5 = weight of 1, and survey item #7 = weight of 4.

Target Context: Targets for this measure were established on the basis of a 2009 validation study.

Explanation

Description of Calculation: The calculation for this measure is as follows:

The total number of parents responding “Yes” to survey item #2 plus the total number of parents responding “to a great extent “ to survey item #3 (times a weight of 2), plus the total number of parents responding “to a great extent” to survey item #5, plus the total number of parents responding “Yes” to item #7(times a weight of 4), divided by the total number of parents responding to survey item #2, plus the total number of parents responding to survey item #3 (times a weight of 2), plus the total number of parents responding to survey item #5, plus the total number of parents responding to survey item #7 (times a weight of 4)], times 100.

2011 Result = [3146 + 2179(2) + 1598 + 2483 (4) / 3394+ 3342(2) + 3324 + 2802(4)] x 100 = [19,034/24,610] x 100 = 77%

IDEA: Special Education Personnel Preparation (OSERS)

FY 2011 Program Performance Report

Strategic Goal 1

Discretionary

IDEA, Part D-2, Sections 662

Document Year 2011 Appropriation: $

Measure 1.1 of 2: The percentage of Special Education Personnel Preparation projects that incorporate evidence-based practices into their curricula. (Desired direction: increase) 1944

(or date expected) Status

Source. A panel of 5–7 external experts in the field of Special Education evidence-based instructional practices review syllabi comprising the curricula of a 50% random sample (in FY 2011, N = 36) of 84.325 Personnel Development grants funded during the previous FY 2010, with the exception of Program Improvement grants whose sample is drawn from grants funded 2 years prior to the current fiscal year (in FY2011, FY 2009 grants). The panel of 5 to 7 members (syllabi panel) is selected annually by an outside contractor from a list of eligible experts. Experts are considered eligible to review syllabi based upon the relevance of their expertise, and the depth and breadth of their experience in the field of special education. No expert serving on the faculty of any grantee currently receiving funds from the Office of Special Education Programs is eligible for appointment to the syllabi panel.

Each panelist reviews syllabi of the randomly sampled (36) grantees, including Early Childhood (6), Leadership (11), Low Incidence (5), Minority Institutions (5), Program Improvement (6), and Secondary Transition (3). A curriculum is rated as “evidence-based” if at least half of the experts judge the syllabi to incorporate evidence-based practices (EBPs).

Data for this measure (syllabi and panel ratings) are collected annually during the summer.

Frequency of Data Collection: Annual.

Data Quality. A 50% random sample of grants within each of the grant categories was drawn to select grantees whose curricula would be evaluated for the presence of EBPs.

A 19-page EBPs protocol developed by American Institutes for Research (AIR), under contract with OSEP is provided to panelists to assist them in systematic evaluation of the curricula for grants in the categories of Early Childhood, Low Incidence, Minority Institutions, Program Improvement, and Secondary Transition. The 19-page EBP protocol included five EBP domains and two additional categories to be applied in evaluating selected grantees, as appropriate. The evidence-based categories are (1) assessment, (2) behavior, (3) inclusive practices, (4) instructional strategies, (5) literacy, (6) early childhood, and (7) secondary transition.

Because Leadership grants focus on the development of administrative and research personnel, whose work is intended to be supportive of special education services (as opposed to providing direct services), curricula for Leadership grants are evaluated with a protocol that reviews syllabi for evidence of coursework supporting the development of expertise in EBPs, including research or evaluation methods or designs, data collection and analysis, research techniques, and data interpretation/applications.

Each expert panelist also used an electronic scoring workbook containing worksheets for each type of grant. Worksheets are pre-populated with the “Institution,” “Grant Number,” and “Syllabus Title” for each syllabus. Expert panelists record all scores in these Excel worksheets.

Two orientation webinars are held with panelists to review instructions, protocol, and scoring rubrics.

Inter-rater reliability analysis for FY 2011 yielded an AM-ICC score of 0.90. This represents an outstanding level of agreement across the 6 Expert Panel reviewers and the 36 curricula that were reviewed.

Note: Values of AM-ICC from 0.40 to 0.50 are considered moderate, 0.51 to 0.70 are considered substantial, and 0.71 or higher are considered outstanding (Landis, J.R., Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics 33:159–174). Most statisticians prefer for AM-ICC values to be at least 0.55 or higher before claiming a good level of agreement.

Target Context. Targets for this measure were established based upon results for this measure in FY 2007. Annual targets were designed to set challenging benchmarks for improvement on this measure, since Personnel Development Program grants are awarded, in part, based upon the extent to which grantee curricula reflect EBPs. Because the program has greatly exceeded targets in FY 2010 and FY 2011, for FY 2012 OSEP is considering using a more rigorous measure calculation methodology that would look at not just incorporation of EBPs in the curriculum, but also evidence that curricula include opportunities for scholars to demonstrate the acquisition of EBP skills.

Explanation

Explanation of Method:

Expert panelists rate the presence of EBPs in five domains (Assessment, Behavior, Inclusive Practices, Instructional Strategies, and Literacy) across the syllabi in the curriculum and across two age spans (Early Childhood and Secondary Transition). For Leadership grants only, expert panelists rate syllabi associated with each Leadership grant for evidence of coursework supporting the development of expertise in any or all of the following EBPs: research or evaluation methods or designs, data collection and analysis, research techniques, and data interpretation/applications.

A curriculum is judged to be evidence-based if 50% or more of the expert panelists judge it to be evidence-based in all five domains of EBP for all grant categories, except for Leadership where expert panelists look for evidence of EBPs in any or all of four domains. (Since a total of 6 expert panelists scored each curriculum, at least three expert panelists had to indicate that a curriculum was evidence-based).

There was a unanimous decision on 31 of the 36 curricula (86.1%), and for 35 of the 36 curricula (97.2%) there was agreement among five or more of the six panelists.

Explanation of Scoring Calculation:

In 2011, all 36 (or 100%) grants in the sample were found to incorporate EBPs in their curricula.

2011 Result: (36 curricula were judged to have included EBPs / 36 curricula reviewed) = 100%

Discussion

In 2011, 36 of the 36 (or 100%) grants in the sample were found to incorporate EBPs in their curricula. There was a unanimous decision on 31 of the 36 curricula (86.1%), and for 35 of the 36 curricula (97.2%), there was agreement among five or more of the six panelists.

The score for 2011 for the Personnel Development Program increased by 8.7 percentage points over the 2010 score; it is the second year in a row that the program scored in the excellent range for presence of EBPs incorporated into the curricula of its grantees. There are a number of explanations for this continued level of high achievement. First, the Personnel Development Program has emphasized the importance to prospective grantees of inclusion of EBPs in proposed curricula. Second, the Personnel Development Program has targeted technical assistance support to grantees in evidence-based instructional and evaluation practices. Finally, expert panelist scoring continues to become more efficient and consistent because of continued improvements in the clarity of instructions, scoring protocols for leadership grantees, pre-population of scoring sheets with grantee and syllabi information, and orientation and follow-up sessions to train the expert panelists.

Measure 1.2 of 2: The percentage of scholars completing Special Education Personnel Preparation funded training programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. (Desired direction: increase) 1945 Comment by : Change to “Personnel Development” instead of “Personnel Preparation?”

(or date expected) Status

Source. Grantees submit data for this measure through the OSEP Personnel Development Program Performance Reporting System (PRS), a web-based data collection (https://www.oseppddprs.ed.gov). The data for a given reporting year pertains to graduates exiting 2 years prior. The 2011 percentage refers to 2009 graduates.

Data Quality. Data reported are calculated from responses to the Scholar Data Report (SDR) Items 4-9 submitted by Institutions of Higher Education (IHEs) during the prior fiscal year and pertain to graduates in the previous year. The accuracy of data (number of graduates) submitted by the IHEs is independently verified by an outside contractor.

Target Context. Targets (a baseline) for this measure were established on the basis of data collected in 2008. Now that a more complete data set is being collected, it may be time to create new targets for 2012 and beyond. Alternatively, OSEP may want to collect one more year of data to establish a more dependable baseline.

Explanation.

Explanation of Method:

The measure reflects the percentage of scholars completing programs funded by Special Education Personnel Development Program who passed a standardized or non-standardized measure. (Scholars trained under Leadership programs are excluded from this calculation).

Explanation of Scoring Calculation:

The numerator for this measure is the number of scholars who pass an exam demonstrating knowledge and skills in EBPs for children with disabilities (2,255).

The denominator is the total number of scholars who completed training programs (2,655). This number includes 2,255 scholars who passed a test, 30 who did not pass a test, 171 who did not take a test, 165 for whom test status was missing, and 34 for whom grantees did not know if they took any test or exam). The denominator includes scholars who graduated but did not take certification tests, scholars whose testing status or results are missing or unknown, and scholars who graduated and passed certification tests.

Numerator = number of scholars who passed an assessment or exam on either the first attempt or a subsequent attempt (N = 2,255).

Denominator = number of scholars who graduated from grant supported training programs in FY 2009 minus students who were enrolled in Leadership grants, minus scholars for whom testing data was not applicable (N = 2,655).

Result is multiplied by 100.

2011 Results:

(2,255 / 2,655) * 100 = 84.9%

Explanation of Results:

The results of this measure reflect an improvement in data collection attributed to training and technical assistance that was provided to grantees to ensure that scores for those scholars who do take a standardized exam are reported, and that results from non-standardized measures of knowledge and skills are also reported by grantees.

Measure 2.1 of 4: The percentage of Special Education Personnel Preparation funded scholars who exit training programs prior to completion due to poor academic performance. (Desired direction: decrease) 1783

(or date expected) Status

Source. Grantees submit data for this measure through the OSEP Personnel Development Program Performance Reporting System (PRS), a web-based data collection (https://www.oseppddprs.ed.gov). The data for this measure refers to scholars funded under 84.325 personnel development program grants who exited due to poor performance in FY 2009.

Frequency of Data Collection: Annual Data Quality. Data for this measure include all scholars funded under 84.325 personnel development grants in FY 2009. Calculations are derived from data provided by grantees and entered into the SDR during the prior fiscal year.

Data are entered into the PRS by grantees based upon their administrative records. Calculations for the percentage of exiters due to poor performance are derived from data submitted on the SDR in FY 2010.

Target Context. Targets were established based upon results in FY 2006. OSEP has established a baseline of 2.0% for this measure, with the objective of reducing the target in the future. OSEP intends to provide technical support to grantees on the importance of provision of ongoing academic and financial support to scholars.

Explanation

Explanation of Calculation:

For this measure, the calculation is the percentage of exiting scholars who leave training prior to completion due to poor academic or field-based performance.

Numerator = number of scholars funded by 84.325 grants who exited training programs in FY 2009 due to poor academic or field-based performance. (n = 66)

Denominator: number of scholars funded by 84.325 grants who exited training programs in FY 2009. (n = 2,678 [completers] + 701 [exiters prior to completion] = 3,379)

Result is multiplied by 100.

2011 Result: (66 / 3,379) x 100 = 1.95%

The result of this measure, 1.95% of all scholars receiving program funds exited preparation programs prior to completion due to poor academic performance in FY 2010, reflects a slight increase from the FY 2009 number that was reported in FY 2010, yet falls below the 2.0% threshold established for this measure.

OSEP has an ongoing interest in providing TA to grantees to ensure grantees maintain high standards when recruiting scholars to receive federal training funds, and that scholars are supported academically and financially to achieve timely completion of the preparation program.

Measure 2.2 of 4: The percentage of Special Education Personnel Preparation funded degree/certification program recipients who are working in the area(s) in which they were trained upon program completion. (Desired direction: increase) 000100

(or date expected) Status

Source. Grantees submit data for this measure through the OSEP Personnel Development Program Performance Reporting System (PRS), a web-based data collection (https://www.oseppddprs.ed.gov). The data for this measure refer to scholars funded under 84.325 grants who exited the training program in FY 2008. Other data used to calculate this measure are derived from the Scholar Data Report (SDR).

Data Quality. Data for this measure include all scholars funded under 84.325 Personnel Development Program grants in FY 2009. Data are entered into the PRS by grantees based upon their administrative records pertaining to the degree earned, and known employment status of scholars at the completion of the program. Calculations for the percentage of degree recipients working in the area in which they were trained are derived from data submitted to the SDR in FY 2011.

Target Context. Targets for this measure were established based upon two years of data reported in 2006 and 2007.

Explanation.

Explanation of Calculation:

For this measure, the calculation is the number of program completers currently employed in the area in which they were trained (or under contract for such employment) divided by the number of scholars currently employed, those not employed, and those for whom employment status is unknown.

Numerator = number of scholars who exited grant-supported training in FY 2009, graduated with a state certificate, and are employed in their areas of training (n = 1,875 in FY 2009).

Denominator = number of scholars who exited grant-supported training in FY 2009 (2,678), minus 1 who was missing data on degree received, minus the 230 who received grantee endorsements/course completion only (n = 2,447 in FY 2009)

Result is multiplied by 100.

2011 Result: (1,875 / 2,447) x 100 = 76.6%.

Explanation of Results:

The denominator includes degree and certification recipients who were employed (1,792), those who were employed but not employed in their area of training (230), those for whom grantees did not know employment status (185), those who were not employed (142), and those for whom employment data were missing (15).

Although the target for this measure was not met, the program saw an increase of 7.2 percentage points, which represents a 10.4% increase over the FY 2010 result. This substantial improvement is in a large measure due to technical assistance provided to the grantees concerning follow-up data collection on the employment status of graduates.