Project Grant 2201438

EXPLORING THE VALIDITY OF THREE-DIMENSIONAL ASSESSMENTS FOR UNDERSTANDING STUDENT LEARNING OF SCIENCE -THE NEXT GENERATION SCIENCE STANDARDS (NGSS LEAD STATES, 2013) ORGANIZES GOALS FOR STUDENT LEARNING ALONG THREE DIMENSIONS: (1) SCIENTIFIC AND ENGINEERING PRACTICES SUCH AS DEVELOPING AND USING MODELS; (2) CROSSCUTTING CONCEPTS SUCH AS CAUSE AND EFFECT; AND (3) DISCIPLINARY CORE IDEAS WHICH INCLUDES FOUNDATIONAL IDEAS FROM THE LIFE, PHYSICAL, AND EARTH SCIENCES, AND ENGINEERING AND TECHNOLOGY. STUDENTS IN 44 STATES ARE BEING OR WILL SOON BE HELD ACCOUNTABLE TO SCIENCE STANDARDS BASED ON THE NGSS. HIGH-QUALITY ASSESSMENTS ALONG THESE THREE DIMENSIONS (TERMED 3-D ASSESSMENTS) ARE NEEDED TO HELP STATES, TEACHERS, AND RESEARCHERS MEASURE STUDENTS? PROGRESS AND THE IMPACT OF EDUCATIONAL INTERVENTIONS. WHILE THERE IS EMERGING AGREEMENT ABOUT THE FEATURES OF HIGH-QUALITY 3-D ASSESSMENTS, DEVELOPERS HAVE STRUGGLED TO CREATE THEM FOR THE NGSS THAT MEET THE CRITERIA THAT HAVE BEEN SET IN THE STANDARDS. IN THE PROCESS, THEY HAVE DEVELOPED ASSESSMENTS THAT STUDENTS FIND MORE DIFFICULT TO COMPLETE AND SCORE LOWER ON THAN TRADITIONAL ONES. IN ADDITION, THERE IS LITTLE EVIDENCE TO DETERMINE IF CURRENT ASSESSMENTS ARE MEASURING WHAT THEY PURPORT TO MEASURE. THIS PROJECT ADDRESSES THIS ISSUE BY CONDUCTING STUDIES TO EXPLORE WHAT THESE ASSESSMENTS ARE ACTUALLY MEASURING. THIS PROJECT WILL EXTEND PRIOR WORK AROUND ASSESSMENT RESEARCH AND DEVELOPMENT AND PROMISES TO HAVE A SIGNIFICANT IMPACT ON HOW THE FIELD OF SCIENCE EDUCATION THINKS ABOUT 3-D ASSESSMENTS. THE FINDINGS FROM THIS PROJECT WILL PROVIDE INFORMATION ABOUT HOW THE ASSESSMENTS FUNCTION FOR ENGLISH-LANGUAGE LEARNERS LEARNERS WHICH COULD BE USED TO INCREASE THE EQUITY OF 3-D ASSESSMENTS. THE FINDINGS COULD ALSO HELP ASSESSMENT DEVELOPERS BETTER DEVELOP INSTRUMENTS THAT MEASURE STUDENTS? ABILITY TO DEMONSTRATE KNOWLEDGE ALONG THE THREE DIMENSIONS ARTICULATED IN THE NGSS. ADDITIONALLY, IF THE ASSESSMENTS USED IN THIS PROJECT ARE FOUND TO BE VALID MEASURES OF STUDENTS? 3-D SCIENCE UNDERSTANDING, THE PROJECT?S FINDINGS COULD BE USED TO INVESTIGATE THE IMPACT OF 3-D SCIENCE LEARNING AT THE MIDDLE SCHOOL LEVEL. IN ORDER TO ADVANCE THE NGSS, IT IS CRITICAL THAT ADVANCES IN THE ASSESSMENT OF LEARNING ALONG THESE THREE DIMENSIONS OF KNOWING BE MADE. THUS FAR, 3-D ASSESSMENT DEVELOPMENT HAS LARGELY FOCUSED ON DEVELOPING CLUSTERS OF ITEMS THAT ARE BUILT AROUND A FOCAL PHENOMENON, KNOWN AS PHENOMENA-BASED ITEM CLUSTERS (PBICS). THE OVERARCHING RESEARCH QUESTION THAT THIS PROJECT SEEKS TO ANSWER IS TO WHAT EXTENT ARE PBICS VALID MEASURES OF STUDENTS? SCIENCE UNDERSTANDING ALONG THESE THREE NGSS DIMENSIONS? THIS QUESTION WILL BE ANSWERED BY DEVELOPING AN EVIDENCE-BASED VALIDITY ARGUMENT BASED ON THE VARIETY OF EVIDENCE OUTLINED IN THE UPDATED STANDARDS FOR EDUCATIONAL AND PSYCHOLOGICAL TESTING (AERA, APA, & NCME, 1999/2014). FIRST, THE PROJECT TEAM WILL DEVELOP A POOL OF EXTANT PBICS AND INVESTIGATE THE EXTENT TO WHICH THEY MEET THE QUALITY CRITERIA ESTABLISHED FOR 3-D ASSESSMENTS (EVIDENCE BASED ON TEST CONTENT). THE RESULTS OF THIS INVESTIGATION WILL THEN INFORM REVISIONS TO THE ASSESSMENTS THAT WILL BE MADE BEFORE THEY ARE USED IN A SERIES OF STUDIES DESIGNED TO GATHER VALIDITY EVIDENCE. NEXT, THINK-ALOUD INTERVIEWS WILL BE CONDUCTED WITH MIDDLE SCHOOL STUDENTS TO STUDY HOW WELL PBICS TAP INTO THE INTENDED COGNITIVE PROCESSES RELATED TO THE NGSS DIMENSIONS (EVIDENCE BASED ON RESPONSE PROCESSES). CONCURRENTLY, A LARGER GROUP OF MIDDLE SCHOOL STUDENTS FROM ACROSS THE U.S. WHO WILL BE USING THE OPENSCIED MATERIALS WILL BE RECRUITED TO PARTICIPATE IN A PRE-TEST/POST-TEST STUDY. RASCH MODELING OF THE PRE- AND POST-TEST DATA WILL BE USED TO DETERMINE THE DIMENSIONALITY OF THE ITEMS THAT MAKE UP THE PBICS (EVIDENCE BASED ON INTERNAL STRUCTURE), INVESTIGATE THE CORRELATION BETWEEN PERFORMANCE ON PBICS AND DISCIPLINARY CORE IDEA-FOCUSED, MULTIPLE-CHOICE ASSESSMENTS (EVIDENCE BASED ON RELATIONS TO OTHER VARIABLES), AND EXPLORE WHETHER THE PBICS FUNCTION DIFFERENTLY FOR ENGLISH LEARNERS AND NATIVE ENGLISH SPEAKERS (EVIDENCE BASED ON INTERNAL STRUCTURE). HIERARCHICAL LINEAR MODELS WILL BE USED TO UNCOVER WHAT PROPORTION OF VARIANCE EXPLAINED BY PBICS IS EXPLAINED BY MEASURES OF READING OR WRITING ABILITY (EVIDENCE BASED ON RELATIONS TO OTHER VARIABLES). FINALLY, MEASURES OF INSTRUCTIONAL SENSITIVITY FOR BOTH THE PBICS AND THE DISCIPLINARY CORE IDEA-FOCUSED ASSESSMENTS WILL BE CALCULATED TO INVESTIGATE THE EXTENT TO WHICH THESE ASSESSMENTS ARE SENSITIVE TO SCIENCE INSTRUCTION ALONG THE THREE NGSS DIMENSIONS (EVIDENCE BASED ON CONSEQUENCES OF TESTING). THIS PROJECT IS SUPPORTED BY NSF'S EHR CORE RESEARCH (ECR) PROGRAM. THE ECR PROGRAM EMPHASIZES FUNDAMENTAL STEM EDUCATION RESEARCH THAT GENERATES FOUNDATIONAL KNOWLEDGE IN THE FIELD. INVESTMENTS ARE MADE IN CRITICAL AREAS THAT ARE ESSENTIAL, BROAD, AND ENDURING: STEM LEARNING AND STEM LEARNING ENVIRONMENTS, BROADENING PARTICIPATION IN STEM, AND STEM WORKFORCE DEVELOPMENT. THE PROGRAM SUPPORTS THE ACCUMULATION OF ROBUST EVIDENCE TO INFORM EFFORTS TO UNDERSTAND, BUILD THEORY TO EXPLAIN, AND SUGGEST INTERVENTION AND INNOVATIONS TO ADDRESS PERSISTENT CHALLENGES IN EDUCATION. THIS AWARD REFLECTS NSF'S STATUTORY MISSION AND HAS BEEN DEEMED WORTHY OF SUPPORT THROUGH EVALUATION USING THE FOUNDATION'S INTELLECTUAL MERIT AND BROADER IMPACTS REVIEW CRITERIA.