27MATALMB1 - Evidence Form - Final.docx

Evidence Form

27MATALMB1

Please provide evidence for each of the following items listed below. Evidence can include screenshots and/or links to samples. Please ensure that all links are easily accessible to the Review Team.

Required Criteria For responses to be considered, Bidders must provide evidence of meeting the following required criteria. Submissions that do not provide sufficient evidence in the Required Criteria for A-E will NOT be scored.

A. Constructs Measured: The constructs measured at identified assessment periods (beginning of year, middle of year, end of year) include, at a minimum, the following skills in the corresponding grades.

Kindergarten: Vendor must provide evidence of making available the following constructs.

Note: BOY: Beginning of the school year; MOY: Middle of the school year; EOY: End of the school year.

Phonemic Awareness Must measure at least one of the following:

· Phoneme Segmentation (BOY/MOY, EOY)

· Elision (BOY/MOY, EOY)

· Blending (BOY/MOY, EOY)

Alphabetic Knowledge Must measure at least one of the following:

· Letter Identification (BOY/MOY)

· Letter Sound Knowledge (BOY/MOY) Must measure both of the following:

· Letter Sound Knowledge (EOY)

· Decoding Nonsense Words (EOY)

Rapid Automatized Naming Must measure:

· Objects or Letters (BOY/MOY)

· Letters (EOY)

Oral Language Listening Comprehension:

Must measure one of the following:

· Retell (BOY/MOY/EOY)

· Passage level listening with follow-up questions (BOY/MOY/EOY)

· Sentence Repetition (BOY/MOY, EOY)

Grade One (1): Vendor must provide evidence of making available the following constructs.

Note: BOY: Beginning of the school year; MOY: Middle of the school year; EOY: End of the school year.

· Phoneme Segmentation (BOY, MOY, EOY)

· Elision (BOY, MOY, EOY)

· Blending (BOY, MOY, EOY)

· Letter Sound Knowledge (BOY)

· Decoding Nonsense Words (BOY, MOY, EOY)

Word Reading

· Word Identification (BOY)

· Passage Reading Fluency (MOY, EOY)

Rapid Automatized Naming

· Letters (BOY)

Oral Language Listening Comprehension Must measure at least one of the following:

· Retell (BOY/MOY/EOY)

Grades Two (2) and Three (3): Vendor must provide evidence of making available the following constructs.

Note: BOY: Beginning of the school year; MOY: Middle of the school year; EOY: End of the school year.

Alphabetic Knowledge

· Decoding Nonsense Words (BOY, MOY, EOY)

Word Reading

· Passage Reading Fluency (BOY, MOY, EOY)

Reading Comprehension

· Reading Comprehension (BOY, MOY, EOY)

Oral Language: GRADE 2 ONLY Listening Comprehension Must measure at least one of the following:

· Retell (BOY/MOY/EOY)

B. Technical Adequacy Classification Accuracy: Must provide a statement of affirmation that the screening assessment meets the following threshold for the “significantly below” risk level for grades K-3. If available, provide a reference to the page number(s) in your technical manual where this information can be verified.

· Sensitivity: greater than or equal to 0.70

· Specificity: greater than or equal to 0.70

· Area under the Curve (AUC): greater than or equal to 0.70

C. Scoring and Assessment of Risk: Must provide evidence of the descriptors listed below:

· Assessment results in a composite percentile rank as part of the reported score.

· A composite risk level (e.g., at risk/requires intervention) as part of the reported score.

D. Technical Manual: Must provide evidence of the descriptors listed below:

· Provides a link to the Technical Manual which includes, at a minimum, detailed information about test development, scoring, norming, reliability and validity of the assessment.

E. Administration, Usability, and Support: Must provide evidence of the descriptor listed below:

· Provides a user-friendly parent report on student results.

Administration, Usability, and Support:

· Assessment is designed to be administered at least twice per year in kindergarten and three times per year in grades one, two and three at a minimum (fall, winter, spring)

· Statement of willingness to cooperate with DESE on the collection of data three times per year from certain schools.

Administration, Usability, and Support:

· Offers professional development, technical support and customer service to public school educators using the assessment.

Preferred Criteria After bidders provide evidence to meet the required criteria with accompanying descriptors listed above (A-E), submissions will be further evaluated on the evidence provided for additional criteria, including accompanying descriptors listed below.

A. Constructs Measured (Section Value 10 Points)

1. Provide evidence that the assessment includes a measure of Rapid Automatized Naming (RAN) for grades 2 and 3. If RAN is measured as a part of a subtest such as letter naming fluency, evidence demonstrates the validity of this as a measure of RAN OR RAN, if included as a standalone measure, features:

a. Timed Administration

b. A set of at least 36 familiar objects or letters (stimuli) presented serially and pseudorandomized

c. Five or six token items (stimuli)

d. Requires student to read continuously from left to right, and engage in a "return to sweep" to read on the next line

e. Norming sample identified and criterion referenced scoring based on total naming time.

2. Provide evidence that the assessment includes encoding/spelling at any or all grades K-3.

Additional Criteria Vendors must submit evidence for the criteria listed below.

B1. Classification Accuracy (Section Value 25 points) DESE is seeking screening assessments that can efficiently and accurately identify students at risk of reading difficulty. In this section we ask for information about your definition of “at risk,” how your screening assessment identifies students who are at risk, including how risk levels were set (methods, sample, rationale for this approach) and why the criteria used to establish risk benchmarks are meaningful and relevant.

3. Describe the criterion measures used in classification accuracy analyses (e.g., measure name, score, author/vendor, what the assessment measures). Identify how the classification as “significantly below” or the equivalent risk level is based on a cut point between the 10th-20th percentile on the external (outcome) measure.

Note: External criterion should be a measure of "end of year" or spring grade-level expectations." Assessments may use internal or external criterion measures to demonstrate classification accuracy for fall and winter administrations of their screener. If vendors use internal criterion measures to support classification accuracy in fall and winter, evidence needs to be provided on how fall and winter cut scores align to the external spring criterion measure.

4. Describe how the cut scores identified above are currently included in your assessment system. If they are not one of your current risk classifications, describe how these will be built into your system so districts in Massachusetts would automatically be able to use a risk classification between the 10th-20th percentile on the outcome measure.

5. Provide evidence that these analyses were drawn from a sample of at least 50 students per classification analysis (Fall Kindergarten and Winter Kindergarten should have 50 students each) that is representative of students across all performance levels.

6. Complete the tables below with your classification analysis findings for “significantly below” benchmark by grade-level and season. If classification accuracy analyses are not provided for a grade-level by season combination, write “Not Provided” in the criterion measure row and leave all other data for that column blank. Vendors may reformat the tables below for ease of sharing data. Content requirements must not be altered.

Note: If you would like DESE to consider a cross-validation study of classification accuracy, please duplicate these tables and add “Cross-Validation Evidence for” ahead of the grade-level for which cross-validation evidence is submitted.

Kindergarten

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Note: Sample size should include the number of total students used for the classification analyses. All other categories should be based on percentages of that number. Economically disadvantaged includes students participating in the Supplemental Nutrition Assistance Program (SNAP), eligible for the National School Lunch Program, or those whose family reported income was at or below 185% of the federal poverty line.

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Grade 1

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Grade 2

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Grade 3

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

B2. Reliability (Section Value: 15 points) Because composite scores are used for risk classification, identify the types of reliability evidence focused on that type of decision-making, such as SEM and conditional SEM by grade and season (especially near the cut), short-interval test–retest stability assuming 2-3 windows per year. Include information about inter-rater reliability and scorer training/monitoring for human or other scoring approaches. If alternate forms or modes are used, provide evidence of form/mode equivalence. Include subgroup results where available.

7. Using the table below, identify the types of reliability that have been established for your screening assessment. Offer a justification for each type of reliability reported, given the type and purpose of the screening assessment.

Note: Appropriately justified reliability analyses for each grade level submitted must include either (1) at least one model-based approach to reliability (e.g., McDonald’s Omega) or (2) at least two non-model-based reliability analyses (e.g., alternate form, test-retest, inter-rater).

In the table below, report the results of the reliability analyses described above. Add additional rows as needed. Vendors may reformat the table below for ease of sharing data. Content requirements must not be altered.

8. Describe the analysis procedures for each reported type of reliability.

9. Describe the sample(s), including size, characteristics, and representativeness of students across all performance levels for each reliability analysis conducted.

10. Report other types of reliability not compatible with the table, if any.

11. Using the table below, identify the types of reliability that have been established for any subtests that measure oral language. Vendors may reformat the table below for ease of sharing data. Content requirements must not be altered.

12. Describe the analysis procedures for each type of reliability reported in the oral language table.

B3. Validity (Section Value: 15 points) DESE is looking for strong evidence of validity supporting the intended use of composite risk levels in identifying students at risk of reading difficulty in grades K-3.

13. Provide at least two types of validity evidence for each grade level submitted for review. Describe any criterion measures used and explain why each measure is appropriate given the purpose of the screening assessment.

Note: Two analyses that both provide the same type of validity evidence (e.g., two types of predictive validity evidence) count as one type of validity evidence.

14. Describe the analysis procedures for each reported type of validity.

15. Describe the sample(s), including size, characteristics, representativeness of performance levels and race and ethnicity.

In the table below, report the results of the validity analyses described above. Add additional rows as needed. Vendors may reformat the table below for ease of sharing data. Content requirements must not be altered.

16. Report other types of validity not compatible with the table.

17. Using the table below, describe the validity evidence for any subtests that measure oral language. Vendors may reformat the table below for ease of sharing data. Content requirements must not be altered.

18. Describe the analysis procedure for each type of validity reported in the oral language table.

B4. Bias Analysis (Section Value: 10 points)

19. Describe the method (e.g., differential item functioning, multiple-group confirmatory factory models) used to determine the presence or absence of bias for student groups (e.g., gender, economic disadvantage, race or ethnicity, English learner status, country of origin).

Note: A statistical procedure must be used as part of the bias analysis process. Your complete response must be provided below.

20. Describe the student groups (e.g., race and ethnicity, gender, economic (dis)advantage status, students with disabilities, English learners) included in the bias analyses.

21. Describe the results of the bias analyses, including data and interpretative statements. Include the magnitude of the effect (when available) if bias was identified, and what actions were taken as a result of any identified bias.

C. Additional Information: Administration, Progress Monitoring, and Supports

22. Provide evidence that the assessment is brief with clear directions for administration. List the administration time for each submitted season and grade level.

23. Provide evidence of including progress monitoring tools to quantify student's rate of improvement or responsiveness to instruction, and to evaluate instruction. For each progress monitoring tool, provide information about how frequently the tool can be administered and the minimum data points required for a stable slope that can be used to make instructional decisions.

24. Provide evidence of supportive resources to guide school- based administrators and educators in effective data assessment administration, data access, and data based instructional decision-making.