26MACISMB2 Evidence Form - Spanish - Final.docx

Evidence Form: SPANISH

RFR 26MACISMB5

Please provide evidence for each of the following items listed below. Evidence can include screenshots and/or links to samples. Please ensure that all links are easily accessible to the Review Team.

Required Criteria For responses to be considered, Bidders must provide evidence of meeting the following required criteria. Submissions that do not provide sufficient evidence for numbers 1-8 in this section will NOT be scored.

Constructs Measured: The constructs measured at identified assessment periods (beginning of year, middle of year, end of year) include, at a minimum, the following skills in the corresponding grades.

Alphabetic Knowledge and Phonological Awareness: Must provide evidence of measuring at least one of the following skills for kindergarten and grade one:

a. Letter Sound Knowledge (grades K; grade 1 beginning of the year)

b. Letter Combination or syllable sounds (grades K, 1)

c. Blending (grades K,1)

d. Segmenting (grades K, 1)

Word Reading: Must provide evidence of measuring both skills listed for the corresponding grades:

a. Word Reading (grade 1, beginning of the year)

b. Passage Reading (grade 1, middle of the year; grades 2 and 3)

Reading Comprehension: Must provide evidence of measuring reading comprehension in both grades 2 and 3.

Oral Language Proficiency: Must provide evidence of measuring at least one of the following skills for each grade listed:

a. Receptive Vocabulary (grades K-3)

b. Expressive Vocabulary (grades K-3)

c. Listening Comprehension (grades K, 1)

2. Technical Adequacy: Must provide a statement of affirmation that, as outlined on the Table below (see number 14, Additional Criteria section), the screening assessment meets the following threshold for the “significantly below” risk level for grades K-3.

· Sensitivity: greater than or equal to 0.70

· Specificity: greater than or equal to 0.70

· Area under the Curve (AUC): greater than or equal to 0.70

3. Scoring and Assessment of Risk: Must provide evidence of the descriptors listed below:

· Assessment results in a composite percentile rank as part of the reported score.

· A composite risk level (e.g., at risk/requires intervention) as part of the reported score.

4. Administration, Usability, and Support: Must provide evidence of the descriptor listed below:

· Provides a user-friendly parent report on student results.

5. Technical Manual: Must provide evidence of the descriptors listed below:

· Provides a Technical Manual which includes, at a minimum, detailed information about test development, scoring, norming, reliability and validity of the assessment.

· For Technical Manuals that are not available in English, the following translated sections must be available in English

· Classification accuracy

· Reliability

· Validity

· Norming

· Test Development

· Scoring and assessment of risk (how risk is determined)

6. Administration, Usability, and Support: Must provide evidence of the descriptor listed below:

· Assessment is designed to be administered at least twice per year in kindergarten and three times per year in grades one, two and three at a minimum (fall, winter, spring)

7. Administration, Usability, and Support: Must provide evidence of the descriptors listed below:

· Offers professional development, technical support and customer service to public school educators using the assessment

Preferred Criteria After bidders provide evidence to meet the required criteria with accompanying descriptors listed above (#1-7), submissions will be further evaluated on the evidence provided for additional criteria, including accompanying descriptors listed below.

A. Constructs Measured (Section Value 10 Points)

8. Provide evidence that the assessment includes a measure of Rapid Automatized Naming (RAN). If RAN is measured as a part of a subtest such as letter naming fluency, evidence demonstrates the validity of this as a measure of RAN OR RAN, if included as a standalone measure, features:

a. Timed Administration

b. A set of at least 36 familiar objects or letters (stimuli) presented serially and pseudorandomized

c. Five or six token items (stimuli)

d. Requires student to read continuously from left to right, and engage in a "return to sweep" to read on the next line

e. Norming sample identified and criterion referenced scoring based on total naming time.

9. Provide evidence that the assessment includes at least one of the following subtests:

a. Encoding/spelling at any grade level (K-3)

b. Blending (grades K,1)

c. Segmenting ( grades K, 1)

Additional Criteria Vendors must submit evidence for the criteria listed below.

B1. Classification Accuracy (Section Value 25 points) DESE is seeking screening assessments that can efficiently and accurately identify students at risk of reading difficulty. In this section we ask for information about your definition of “at risk,” how your screening assessment identifies students who are at risk, including how risk levels were set (methods, sample, rationale for this approach) and why the criteria used to establish risk benchmarks are meaningful and relevant.

10. Describe the composite cut score points used in your measures (measure name, score, author/vendor, what the assessment measures, assessment format) used in classification accuracy analyses. Identify how the classification as “significantly below” or the equivalent risk level is based on a cut point between the 10th-20th percentile on the outcome measure, if an external criterion is used.

Note: External criterion should be a measure of "end of year" or spring grade-level expectations." Vendors may use internal or external criterion measures to demonstrate classification accuracy for fall and winter administrations of their screener. If vendors use internal criterion measures to support classification accuracy in fall and winter, evidence needs to be provided on how fall and winter cut scores align to the external spring criterion measure.

11. Describe how the cut scores identified above are currently included in your assessment system. If they are not one of your current risk classifications, describe how these will be built into your system so districts in Massachusetts would automatically be able to use a risk classification between the 10th-20th percentile on the outcome measure.

12. Provide evidence that these analyses were drawn from a sample of at least 50 students per classification analysis (Fall Kindergarten and Winter Kindergarten should have 50 students each) that is representative of students across all performance levels.

13. Describe the language of instruction and country of origin of the sample used for each classification analysis and how this sample aligns with the intended use of the tool.

14. Complete the tables below with your classification analysis findings for “significantly below” benchmark by grade-level and season. If classification accuracy analyses are not provided for a grade-level by season combination, write “Not Provided” in the criterion measure row and leave all other data for that column blank.

Kindergarten

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Native speaker of focal language

Grade 1

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Native speaker of focal language

Note. Sample size should include the number of total students used for the classification analyses. All other categories should be based on percentages of that number. Economically disadvantaged includes students participating in the Supplemental Nutrition Assistance Program (SNAP), eligible for the National School Lunch Program, or those whose family reported income was at or below 185% of the federal poverty line.

Grade 2

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Native speaker of focal language

Grade 3

Criterion measure

Cut Points - Percentile rank on criterion measure

Cut Points - Performance score on criterion measure

Cut Points - Corresponding performance score (numeric) on screener measure

Classification Data - True Positive (a)

Classification Data - False Positive (b)

Classification Data - False Negative (c)

Classification Data - True Negative (d)

Area Under the Curve (AUC)

AUC Estimate’s 95% Confidence Interval: Lower Bound

AUC Estimate’s 95% Confidence Interval: Upper Bound

Base Rate

Overall Classification Rate

Sensitivity

Specificity

Date

Sample size (N)

Gender

Male

Female

Nonbinary

Gender unknown

Ethnicity

Hispanic

Non-Hispanic

Ethnicity unknown

Race

American Indian or Alaska Native

Asian

Black or African American

Native Hawaiian or Other Pacific Islander

White

Multi-Race

Race unknown

Economically disadvantaged

IEP or diagnosed disability

English learner

Native speaker of focal language

B2. Reliability (Section Value: 15 points) Because composite scores are used for risk classification, identify the types of reliability evidence focused on that type of decision-making, such as SEM and conditional SEM by grade and season (especially near the cut), short-interval test–retest stability assuming 2-3 windows per year. Include information about inter-rater reliability and scorer training/monitoring for human or other scoring approaches. If alternate forms or modes are used, provide evidence of form/mode equivalence. Include subgroup results where available.

15. Using the table below, Identify the types of reliability that have been established for your screening assessment. Offer a justification for each type of reliability reported, given the type and purpose of the screening assessment.

Note: Appropriately justified reliability analyses for each grade level submitted must include either (1) at least one model-based approach to reliability (e.g., McDonald’s Omega) or (2) at least two non-model-based reliability analyses (e.g., alternate form, test-retest, inter-rater). Your complete response must be provided below.

In the table below, report the results of the reliability analyses described above. Add additional rows as needed.

16. Describe the analysis procedures for each reported type of reliability.

17. Describe the sample(s), including size, characteristics, and representativeness of students across all performance levels for each reliability analysis conducted, and how the sampled students’ language aligns with the language assessed by the tool (e.g., language of instruction, country of origin, native speakers of the focal language, English speaking students in an immersion program). Your complete response must be provided below.

18. Report other types of reliability not compatible with the table.

B3. Validity (Section Value: 15 points) DESE is looking for strong evidence of validity supporting the intended use of composite risk levels in identifying students at risk of reading difficulty in grades K-3.

19. Provide at least two types of validity evidence for each grade level submitted for review. Describe any criterion measures used and explain why each measure is appropriate given the purpose of the screening assessment.

Note: Two analyses that both provide the same type of validity evidence (e.g., two types of predictive validity evidence) count as one type of validity evidence.

20. Describe the analysis procedures for each reported type of validity.

21. Describe the sample(s), including size, characteristics, representativeness of performance levels and race and ethnicity, and how the language of the sampled students aligns with the language assessed by the tool (e.g., language of instruction, country of origin, native speakers of focal language, English speaking students in an immersion program).

In the table below, report the results of the validity analyses described above. Add additional rows as needed.

22. Report other types of validity not compatible with the table.

B4. Bias Analysis (Section Value: 10 points)

23. Describe the method (e.g., differential item functioning, multiple-group confirmatory factory models) used to determine the presence or absence of bias for student groups (e.g., gender, economic disadvantage, race or ethnicity, English learner status, country of origin, native speaker of focal language). A statistical procedure must be used as part of the bias analysis process. Your complete response must be provided below.

24. Describe the student groups (e.g., race and ethnicity, gender, economic (dis)advantage status, students with disabilities, English learners, country of origin) included in the analyses. Your complete response must be provided below.

25. Describe the results of the bias analyses, including data and interpretative statements. Include the magnitude of the effect (when available) if bias was identified, and what actions were taken as a result of any identified bias. Your complete response must be provided below.

C. Additional Information: Administration, Progress Monitoring, Supports, and Costs

26. Provide evidence that the assessment is brief with clear directions for administration.

27. Provide evidence of including progress monitoring tools to quantify student's rate of improvement or responsiveness to instruction, and to evaluate instruction. For each progress monitoring tool, provide information about how frequently the tool can be administered and the minimum data points required for a stable slope that can be used to make instructional decisions.

28. Provide evidence of supportive resources to guide school- based administrators and educators in effective data assessment administration, data access, and data based instructional decision-making.