Attachment P-NAEP Reading Survey Questionaire.pdf

ETS Confidential Confidential, unpublished property of ETS. Do not duplicate or distribute without permission.

Use and distribution limited solely to authorized staff, on a need-to-know basis.

NAEP Item Development (ID)

2026 Reading Issues Paper

2026 Reading Survey Questionnaires

Deliverable in response to ID Task 4.2.1

Submitted: June 7, 2021 Last Revised: February 29, 2024

February 2024 ETS Confidential Page 2

Revision History

Version Date Description

1.0 06/07/21 Initial Version

2.0 07/12/21 ETS response to NCES review comments of 6/15/21

3.0 07/23/21 ETS response to NCES review comments of 6/15/21 and Standing

Committee comments from June 23 and 30 meetings

4.0 09/03/21 ETS response to NCES review comments of 8/13/21 and the approved Reading Framework dated 7/23/21

5.0 09/14/21 ETS response to NCES review comments of 9/09/21

6.0 02/29/24 Minor editorial revisions in preparation for publication

February 2024 ETS Confidential Page 3

TABLE OF CONTENTS

Introduction

Reading in the Digital era

Topic 1. Reader Characteristics: Cognition and Metacognition

Reading Strategies

Topical Knowledge

Questionnaire Constructs for Topic 1: Reader Characteristics: Cognition and Metacognition

Topic 2. Reader Characteristics: Engagement and Motivation

Volume of Reading

Reading for Enjoyment

Motivations for Reading

Intrinsic motivation

Extrinsic motivation

Reading self-efficacy

Reading value

Reading interest

Attention

Questionnaire Constructs for Topic 2: Reader Characteristics: Engagement and Motivation

Topic 3. Environmental Characteristics: School and Community Resources, School Social Support and Collaboration

Participation in out-of-school reading/literacy activities

Questionnaire Constructs for Topic 3: Environmental Characteristics: School and Community Resources

Topic 4. Environmental Characteristics: Teacher, Instructional, and Classroom Supports

Teacher support for reading engagement

Teacher support for motivation

Teacher support for students’ experiences

Program and curricular support for reading development

Questionnaire Constructs for Topic 4: Environmental Characteristics: Teacher, Instructional, and Classroom Supports

Conclusions and Next Steps

References

APPENDIX

February 2024 ETS Confidential Page 4

INTRODUCTION

The National Assessment of Educational Progress (NAEP) survey questionnaires help the NAEP program fulfill its federally legislated requirement to collect data and report on demographic reporting categories such as race, ethnicity, gender, and disability, as well as socioeconomic status (NAEP Law, 2002). The National Assessment Governing Board (Governing Board) requires that NAEP survey questionnaires collect general and subject-specific contextual data in addition to the reporting categories outlined in the NAEP Law (NAGB, 2013). These data help the NAEP program provide a context for student achievement data and thus insights into U.S. student learning and changes over time. Since it relies on large, nationally representative samples,1 NAEP program data allow researchers to detect relationships and analyze subgroup differences that would not be discernable otherwise.

The purpose of this paper is to guide, along with other documents briefly described below, the development of the NAEP Reading Survey Questionnaires for the Reading assessment in 2026 and beyond, based on the priority of topics for the collection of contextual data for the reading assessment as specified in the 2026 Reading Framework2 (referred to in this paper as the Framework) (NAGB, 2021). These topics structure the development of constructs and questions that capture a wide range of data that provides a context for student performance in reading.

Figure 1 shows the development process for the 2026 operational reading questionnaires. The current Issues Paper is part of the first major step: the identification of constructs3 that the questionnaires will target. Other documents that are part of this process include the Framework Implementation Plan, which describes the development process for the 2026 reading cognitive assessment and survey questionnaires in light of the recommendations included in the Framework, and the contents of the current item pools and other logistical constraints (e.g., number of blocks needed for the cognitive assessment or the overall response time available for survey questionnaires4), as well as the Existing Item Pool Report.

1 Please note that at grades 4 and 8, NAEP Reading samples are also representative at the state level and at the level of several large urban school districts.

2 At the time this Issues Paper was prepared, the 2026 Reading Framework was not approved. The 2026 Reading Framework referenced in this Issues Paper is the draft dated April 21, 2021. The 2026 survey questionnaires development timeline necessitated this Issues Paper be prepared prior to August—when the final version of the draft 2026 Reading Framework would be voted on by the Governing Board. The Reading Framework was approved on August 5, 2021. This Issues Paper is currently being revised to reflect the differences between the April 21 draft of the Framework and the July 23, 2021, draft approved by the Governing Board. The revisions are minimal in nature and will not affect the Framework topics and subtopics presented.

3 Please note that this step also includes the Core Questionnaire White Paper because some of the constructs, such as socioeconomic status or resilience, are captured in the Core questionnaires.

4 The student questionnaire has a 15-minute time limit for completion. The student survey questionnaire is designed to contain 7 minutes of Core questionnaire content and 8 minutes of subject-specific questionnaire content. The teacher and school administrator questionnaires have no time limit for completion. Teacher and school administrator questionnaires are designed to take approximately 20-30 minutes to complete. Completion time may be longer for school administrators and for teachers who provide instruction for multiple assessment subjects (e.g., fourth-grade teachers).

February 2024 ETS Confidential Page 5

Figure 1. Development Process for Reading Survey Questionnaires

As a first step toward the development of the reading survey questionnaires for the 2026 assessment, this paper presents an overview of the reading-specific contextual variables identified in the Framework for inclusion in the reading survey questionnaires (NAGB, 2021). This iteration of the Framework is the first to encompass both the cognitive reading assessment and the survey questionnaires. It presents reading instruction and learning as dependent on several reader and environmental characteristics. Reading characteristics are intrapersonal aspects of the student and their reading experience including students’ perception of the interest, difficulty, and familiarity of texts, tasks, and contexts on their performances (Pintrich & Schrauben 1992; Eccles, O'Neil et al., 2005; Valencia, Wixson et al., 2014). Conversely, environmental characteristics are more external variables within which the student is situated, including cultural background, community, and school. Environmental characteristics affect reading performance by assigning value to reading, influencing topic familiarity, or by providing resources for reading (Chen, 2005; Elias, & Haynes, 2008; Klem, & Connell, 2004; Košir & Tement, 2014; Malecki & Demaray, 2006; Vaux, Phillips, Holly, Thompson, Williams, & Steward, 1986). This paper is organized around these higher-order contextual variables and several more specific constructs nested within them. Table 1 displays the reader and environmental variable topics and subtopics outlined in the Framework.

February 2024 ETS Confidential Page 6

Table 1. Contextual variables and their definitions as outlined in the Framework

Reader Characteristics

Cognition and Metacognition

Cognitive strategies Cognitive strategies in reading comprehension refer to skills used to understand a text, such as drawing inferences to connect sentences together and checking to be certain that text information is fully understood (OECD, 2018).

Metacognitive strategies Metacognitive strategies in reading comprehension refer to, for example, a student’s use of a mental guidance system to perform such operations as deciding which sections of text are most relevant to an assigned reading goal, how to link two sections, and/or when to reread to seek more information or clarify understanding (Cho & Afflerbach, 2017).

Topical knowledge Topical knowledge refers to students’ use of their pre-existing knowledge of the reading topic to enable them to understand text information and construct new knowledge (O’Reilly, Wang, & Sabatini, 2019).

Engagement and Motivation

Volume of reading Volume of reading refers to the amount of reading a student does for personal interest, pleasure, or learning (Schaffner, Schiefele, & Ulferts, 2013).

Reading for enjoyment Reading for enjoyment refers to the goals, uses, purposes, reasons, and benefits students have for reading in school and out of school (Pitzer & Skinner, 2017).

February 2024 ETS Confidential Page 7

Motivation for reading Motivations for reading refer to students’ attention, effort, sense of self-efficacy, interest, and value for reading a particular text with a unique set of tasks and questions related to it (NAEP Reading Special Study, 2019).

Environmental Characteristics

Reports of School and Community Resources5

School social support

School social support refers to the extent to which students perceive that their teachers and peers believe they contribute positively to classroom reading (through listening, speaking, and interacting well with others) (Vaux, Phillips, Holly, Thompson, Williams, & Steward, 1986).

Participation in outside of school reading/literacy activities

Participation in out-of-school reading/literacy activities refers to the degree to which students have access to resources (i.e., books, computers, media centers, camps, and community organizations) that utilize literacy for enjoyment, communication, learning, and pursuing a variety of activities (Bowen, Bowen, & Ware, 2002).

Reports of Teacher, Instructional, and Classroom Supports

Teacher support for reading engagement

Teacher support for reading engagement refers to the extent to which students perceive their teacher(s) as providing materials and tasks that encourage the development of their reading engagement (Afflerbach, Hurt, & Cho, 2020).

5 In the Framework, school and community resources include a “belonging in school” subtopic. Belonging in school is defined in the Framework as the extent to which students perceive themselves to be accepted members of the school community. Belonging in school is a domain-general construct that is best included in the Core (or domain general) questionnaire. As such, belonging in school is discussed in the appendix of this Issues Paper. Development of survey questionnaire content assessing belonging in school will be part of Core development.

February 2024 ETS Confidential Page 8

Teacher support for motivation

Teacher support for motivation refers to the degree to which students perceive their teacher(s) to support their interests, self-efficacy, and reading goals (Wentzel & Wigfield, 2007).

Teacher support for students’ experiences

Teacher support for students’ experiences refers to the students’ perceptions that their teacher recognizes and uses students’ cultural, language, and social knowledge during reading instruction (Shin, Daly, & Vera, 2007).

Program and curricular support for reading development

Program and curricular support for reading development refers to the extent to which teachers and administrators perceive that the school’s reading program and curriculum enables them to support students’ development of effective reading practices.

February 2024 ETS Confidential Page 9

Opportunity-to-learn (OTL) factors are discussed throughout the Framework and are prominent in educational large-scale assessments, including NAEP. The Framework defines opportunities to learn as “inputs and processes that enable student achievement of intended outcomes.” Examples of OTL factors given in the Framework include the content being taught, time spent on content, instructional strategies, instructional resources, availability of libraries, and access to challenging curricula. In line with OTL definitions and examples given in the Framework, all environmental characteristic topics, subtopics, and the survey questionnaire constructs chosen to reflect them are considered OTL factors.

Before discussing the topics and subtopics outlined in the Framework, digital literacy, the digital divide, and the prevalence of reading on digital screens or “e-reading” are briefly discussed. The purpose of discussing these aspects of reading in the digital era before discussing Framework topics and subtopics is to acknowledge changing reading modalities and their potential to affect how reading is defined and how reading happens. These changes have implications for questionnaire construct development. For example, when assessing how students access reading materials it is important to include virtual access (buying e-books on Amazon or borrowing e-books from the library through apps like Libby) as well as physical access (going to a library or bookstore).

READING IN THE DIGITAL ERA

Traditionally, literacy has been defined as “the ability to read and write to meet society’s needed standards and expectations” (McArthur, Lam-McArthur, & Fontaine, 2018). However, this and similar definitions of literacy are outdated. More contemporary definitions recognize literacy as “the knowledge and the skills needed for contemporary socio-cultural interactions, which comprehend digital (e.g., touch screen tablets) and nondigital tools (e.g., paper books)” (Peled, 2021). Digital literacy encompasses traditional literacy skills, such as comprehension of text, and contemporary skills associated with information and communication technologies (ICT). Digital literacy definitions recognize that 21st-century ICT skills are sometimes required for literacy and often mediate literacy processes. As such, digital literacy refers to “knowledge needed to access digital information effectively, efficiently, and ethically” and highlights digital tools available to communicate with others, to create meaning, and to evaluate digital content (Neumann, Finger, & Neumann, 2017). While there is often disagreement about which digital literacy skills are most important for digital proficiency, educational researchers have classified digital skills into the following categories: information access, online participation, computer ability, search engine skills, and skills required to evaluate found information (McArthur et al., 2018).

Still others classify digital skills into operational, mobile, navigation, social, and creative domains (Peromingo & Pieterson, 2018).

Despite the ubiquity of digital devices and the steadily increasing need for digital literacy in education and society, more generally, there is a digital divide separating students with and without access to the internet and digital devices (Lai & Widmar, 2021; Ramsetty & Adams, 2020). This digital divide creates different levels of digital literacy in today’s students. Though progress has been made closing the digital divide, students in higher-income households are more likely than students in lower-income households to have a desktop or laptop computer at home or to have their own digital devices like personal laptops or smartphones (Rideout, 2019). Additionally, students from higher-income households are more likely than those from lower-income households to report using computers for homework. Sixty-four percent of teens in higher-income households report using a computer for homework daily, compared to only 51 percent of teens from lower-income households. On the other hand, only 8 percent of teens in higher-income households report using a computer for homework only monthly, compared to 22 percent of

February 2024 ETS Confidential Page 10 teens from lower-income households. During an average day, teens from higher-income households report spending 55 minutes using computers for homework, compared to 34 minutes reported by their peers in lower-income households. Conversely, on average, teens from lower-income households report spending 21 minutes a day using their phones for homework, compared to only 12 minutes a day reported by teens from higher-income households. It is not clear whether differences in daily use of computers or smartphones for homework is due to availability of these devices or differences in the amount or nature of homework assignments. Research also shows that the digital divide disproportionately affects students of color, especially African-American students (Ellison & Solomon, 2019). On the whole, the digital divide still exists and research regarding e-reading indicates it is increasing for some students but not others and increasing in some forms but not others.

The Common Sense Census on media usage of tweens (8- to 12-year-olds) and teens (13- to 18-year-olds) reports media and digital device usage of nationally representative samples in the United States (Rideout & Robb, 2019). The 2019 report indicates that 8- to 18-year-olds spend a relatively small proportion of their screen time “e-reading.” Tweens reported about 4.75 hours of daily screen time (excluding time spent doing schoolwork or homework) and only 2 percent of that screen time was dedicated to e-reading. Roughly the same percentage was reported in 2015. Teens reported almost 7.5 hours of daily screen time (excluding time spent doing schoolwork or homework) and only 2 percent of this screen time was dedicated to e-reading. Roughly the same percentage was reported in 2015. While the percentage of time spent e-reading is small relative to other screen activities (i.e., watching TV and videos, gaming, browsing websites and engaging with social media), e-reading among this population is increasing for some students, albeit slowly. However, from 2015 to 2019, the percentage of 8- to 12-year-olds that reported reading electronic books did not increase significantly. From 2015 to 2019 the percentage of 13- to 18-year-olds that reported reading electronic books increased significantly from 4 percent to 9 percent. For both groups print book reading remained relatively stable.

Additionally, from 2015 to 2019, the average time spent e-reading increased significantly from 3 to 5 minutes for tweens and from 3 to 8 minutes for teens.

It is unclear whether the 8- to 18-year-olds surveyed included reading done while using social media or texting in their reports of e-reading. The percentage of “everyday” social media usage did not increase significantly for tweens (10% to 13%); however, it increased significantly for teens (45% to 63%), from 2015 to 2019. Questions about texting were only asked of teens. Teens reported a significant increase in the percentage of daily texting (53% to 78%) and a significant decrease in the average number of texts sent daily (55 to 39 daily texts). Rideout (2019) suggests this decrease may be due to the availability of other means of communication like Snapchat.

During the COVID-19 pandemic, student screen time generally increased (Carroll, Sadowski, Laila, Hruska, Nixon, Ma, & Haines, 2020; Guan, Okely, Aguilar-Farias, del Pozo Cruz, Draper, El Hamdouchi, A., ... & Veldman, 2020; Kovacs, Starc, Brandes, Kaj, Blagus, Leskošek, B., ... & Okely, 2021; Schmidt, Anedda, Burchartz, Eichsteller, Kolb, Nigg, ... & Woll, 2020), and research indicates reading on screens or e-reading increased for some texts but not others. In general, reading for enjoyment increased during the pandemic (Abimbola, Shabi, & Aramide, 2021; Clark & Picton, 2020). Research published by The National Literacy Trust reports that students’ enjoyment of reading has increased 47.8 percent (pre- COVID) to 55.9 percent (post-COVID); about 25 percent of students reported enjoying reading more during the pandemic, and about a third of students reported reading more during the pandemic (Clark & Picton, 2020). Studies indicate print reading was the preferred reading modality and e-reading was

February 2024 ETS Confidential Page 11 underutilized (Sun, Loh, & Nie, 2021). More specifically, students reported reading “Fiction,” “Magazines,” “Comics/graphic novels,” “Non-fiction” and “Poems” in print more often than on screens.

However, students reported reading “Song lyrics” and “News” more often on screen than in print.

Similarly, audiobook use increased among boys during the pandemic (Clark & Picton, 2020).

On the whole, research regarding e-reading, social media usage, and texting are mixed, showing increases in e-reading for some age groups and some types of text but not all; increases in social media usage for some age groups but not all; and increases in the percentage of everyday texting, but a decrease in the number of texts sent daily. Despite these mixed results, it is still important to acknowledge and assess different aspects of reading in the digital era in the NAEP reading survey questionnaire to (1) monitor and report emerging trends and relationships with reading achievement, and (2) pose and potentially answer the question: “What do students perceive as reading?” Additionally, the large nationally-representative samples used in NAEP have the potential to reconcile mixed or conflicting results regarding relationships between e-reading and reading achievement.

In the following sections, we discuss relevant literature pertaining to the topics and subtopics outlined in the Framework and describe the association between constructs related to each subtopic and student achievement in reading. We also propose constructs that correspond to each subtopic, along with examples of potential components of those constructs (i.e., sub-constructs). The topics are presented in the order in which they are listed in the Framework.

TOPIC 1. READER CHARACTERISTICS: COGNITION AND METACOGNITION

Cognition and metacognition are components of self-regulatory behavior that impact reading comprehension and performance (Bol & Hacker, 2001; Dent & Koenka, 2016; Hacker, Bol, & Bahbahani, 2008; Soto, de Blume, Jacovina, McNamara, Benson, Riffo, & Kruk, 2019; Zabrucky, Moore, Agler, & Cummings, 2015). Table 2 lists cognitive and metacognitive strategies outlined by the 2026 Reading Assessment and Item Specifications document. Indeed, cognitive, and metacognitive strategies have direct and indirect effects on student reading performance (Ergen & Kanadli, 2017). The section that follows discusses results of large and meta-analytic studies that examined the impact of cognition and metacognition strategies on reading comprehension and performance. Given that cognitive and metacognitive strategies are related and potentially overlapping topics, they are combined here as Reading Strategies.

February 2024 ETS Confidential Page 12

Table 2. Reading strategies

• Locating information in text(s) to accomplish reading goals

• Integrating and interpreting text-based information to accomplish reading goals

• Analyzing and evaluating text-based information to accomplish reading goals

• Using and applying text-based information in new situations

• Adjusting reading speed and concentration depending on the perceived importance of text to reading goals

• Accessing relevant features of texts to identify key information

• Using affordances in digital texts as appropriate to reading goals

• Revisiting texts as needed for understanding

• Evaluating the importance of text content and determining the usefulness in accordance with reading goals

• Judging style of digital text, including word choices, sentence fluency, information organization, logical structure of main points and supporting details, visual attractiveness, easy-to-read features, and overall layout and design

• Evaluating author’s purposes, intentions, and goals based upon the analysis of the assumptions, worldviews, and beliefs that are often hidden in text

• Assessing the qualities of text, with these evaluations in part affecting whether text has impact on reader’s knowledge, attitudes, behavior, and so on

Reading Strategies According to the Framework, cognitive strategies in reading comprehension are “skills used to understand a text, such as drawing inferences to connect sentences together and checking to be certain that text information is fully understood” (NAGB, 2021). According to Cho and Afflerbach (2017), cognitive strategies are used to make meaning out of what is read, to monitor one’s reading and understanding to ensure comprehension, and to evaluate the text being read. For example, when a student is engaged in the process of making meaning from a text, they are reconsidering and/or revising prior knowledge based on text content.

Dent and Koenka’s (2016) meta-analysis of the relationships between cognitive strategies and academic performance indicate that both cognitive strategies and metacognitive processes have small positive relationships with academic performance, r = .11, 95% CI [0.08, 0.14], and r = .20, 95% CI [0.16, 0.24], respectively. These relationships are moderated by academic subject, grade level, and performance measure. Specifically, the relationship between cognitive strategies and English/Language Arts (ELA) performance is positive and very small (r = .06, 95% CI [0.00, 0.13]). There is no relationship between cognitive strategies and elementary school performance (r = .06, 95% CI [-0.08, 0.20]); and very small positive relationships between cognitive strategies and middle school performance (r = .09, 95% CI [0.05, 0.13]) and high school performance (r = .15, 95% CI [0.11, 0.20]). Moreover, there is no relationship between cognitive processes and standardized test performance (r = .07, 95% CI [-0.02, 0.13]). Overall, meta-analytic results show cognitive strategies have little to no impact on ELA performance; elementary, middle, and high school performance; and standardized test performance.

However, the documented impact of metacognitive strategies is stronger.

February 2024 ETS Confidential Page 13

According to the Framework, metacognitive strategies in reading comprehension include “a student’s use of a mental guidance system to perform such operations as deciding which sections of text are most relevant to an assigned reading goal, how to link two sections, and/or when to reread to seek more information or clarify understanding” (NAGB, 2021, p. 59). More broadly, metacognitive strategies are self-regulatory strategies that enable academic behaviors, like using cognitive strategies and resource management (Dent & Koenka, 2016). Though self-regulatory learning researchers often disagree about what processes or strategies can be categorized as metacognitive, there is some consensus that the following are metacognitive strategies: goal setting, planning, self-monitoring, self-control, and self-evaluation.

Dent and Koenka’s (2016) meta-analysis of the relationship between metacognitive processes and academic performance revealed a small positive relationship, r = .20, 95% CI [0.16, 0.24]. This relationship is moderated by academic subject, grade level, and performance measure. Specifically, the relationship between metacognitive processes and ELA performance is positive and small (r = .23, 95% CI [0.18, 0.29]). Similarly, there are small positive relationships between metacognitive processes and elementary school performance (r = .24, 95% CI [0.15, 0.32]), middle school performance (r = .21, 95% CI [0.16, 0.25]), and high school performance (r = .18, 95% CI [0.10, 0.25]). Moreover, there is a small positive relationship between metacognitive processes and standardized test performance (r = .22, 95% CI [0.16, 0.28]). Overall, meta-analytic results show metacognitive strategies have a small positive impact on ELA performance; elementary, middle, and high school performance; and standardized test performance.

Topical Knowledge According to the Framework, topical knowledge refers to “students’ use of their pre-existing knowledge of the reading topic to enable them to understand text information and construct new knowledge” (NAGB, 2021). The importance of topical knowledge (also called background knowledge or prior knowledge) to reading comprehension is recognized in several models of reading (O’Reilly, Wang, & Sabatini, 2019). For example, the construction-integration model of reading states the reader understands texts by incorporating background knowledge with current text contents (Kintsch, 2004).

Activation models of reading state that reading activates words and concepts that appear in the text as well as those that are related to the concepts in the text. In this case, activation of topical knowledge impacts reading comprehension, particularly when the text is closely related to a topic about which the reader is knowledgeable (Cook & O’Brien, 2014).

Correlational research shows topical vocabulary has a significant effect on high school students’ reading comprehension (O’Reilly, Wang, & Sabatini, 2019). More specifically, the large study showed that above the threshold of 59 percent correct on a topical vocabulary test, topical knowledge had a strong positive relationship with reading comprehension (b = 0.56). Whereas below the threshold, there was not a strong relationship between topical knowledge and reading comprehension (b = 0.12). Topical knowledge impacts reading comprehension, but the strength of the relationship may vary by the amount of topical knowledge.

Questionnaire Constructs for Topic 1: Reader Characteristics: Cognition and Metacognition The constructs that the Framework calls on for capturing cognition and metacognition are discussed below:

February 2024 ETS Confidential Page 14

1) Reading strategies. This construct captures the cognitive and metacognitive strategies students use to achieve reading comprehension including planning, meaning making, monitoring, control, and evaluation strategies. Sub-constructs include those identified in table 2. Potential survey questionnaire items used to operationalize these sub-constructs could ask students to indicate how frequently they use specific reading strategies. Given the fact that different strategies may be more frequently or better utilized at different grades, it may be important to consider measuring this construct differentially across grades 4, 8, and 12. Additionally, given that some reading strategies are complex cognitive and metacognitive processes, it may also be useful to measure some of these sub-constructs with process data6 as well as survey questionnaire items.

Though reading strategies are often complex processes, contemporary research on measuring metacognitive reading strategies provides evidence that self-report survey instruments can validly measure students’ strategy use in generalized reading contexts (Mokhtari, Dimitrov, & Reichard, 2018).

2) Topical Knowledge. This construct captures students’ prior knowledge of topics covered in assigned texts. For example, knowledge of violins in a text about playing the violin. Different aspects of topical knowledge make up its sub-constructs. These sub-constructs include the amount of relevant topics, accuracy of topical knowledge (or the extent to which students’ topical knowledge is accurate), specificity of topical knowledge (or the extent to which topical knowledge is related to the text), and coherence of topical knowledge (or the extent to which topical knowledge is integrated; McCarthy & McNamara, 2021). Survey questionnaire items used to operationalize these sub-constructs may ask students about the amount, accuracy, specificity, and coherence of their topical knowledge for assigned texts. It is worth noting that topical knowledge varies with the text in question. Meaning students’ topical knowledge may vary for each assigned text or passage within and across classroom assignments and standardized assessments, NAEP included. Survey questionnaire items assessing topical knowledge of NAEP reading passages can be administered at the beginning of the questionnaire with other questions that aim to debrief students about the cognitive assessment. Current debrief items include: “How easy or difficult was this test?” and “How much effort did you apply to succeed on this test?” Potential follow-up debrief items assessing topical knowledge may include, “The test was easy (or difficult) because I did (not) know a lot about the reading topics,” or “I applied a lot (little) effort to succeed on this test because I am (not) interested in the reading topics.” Survey questionnaire items assessing topical knowledge of text assigned in ELA class may include, “I know a lot about the topics we read about in class.”; “The topics we read about in class help me learn more about topics I am familiar with.”; and “I do not know much about the topics we read about in class.”

TOPIC 2. READER CHARACTERISTICS: ENGAGEMENT AND MOTIVATION

Engagement and motivation have long been recognized in education and education psychology literature as key determinants of reading performance. Guthrie & Klauda’s heuristic model of reading

6 For all questionnaire constructs that may also be assessed using process data, details about the use of process data will be provided in the Reading Development Plan.

February 2024 ETS Confidential Page 15 engagement posits that reading motivation influences reading engagement which in turn influences reading performance. Overall, research shows that motivation and engagement positively impact reading performance (Toste, Didion, Peng, Filderman, & McClelland, 2020). The section that follows discusses specific aspects of reading engagement (i.e., reading volume and reading for pleasure) and motivation (i.e., intrinsic motivation, self-efficacy, reading value, and attention) and how they relate to reading performance. Meta-analytic and large study results of the relationships between engagement and motivation constructs and reading performance are also discussed.

Volume of Reading The Framework defines volume of reading as “the amount of reading a student does for personal interest, pleasure, or learning.” (NAGB, 2021, p. 59). More generally, reading volume is the amount of time spent reading inside or outside of school (Allington & McGill-Franzen, 2021). Meta-analytic and large studies show moderate to strong positive relationships between reading volume (also called “print exposure”) and reading comprehension and performance. Specifically, Mol and Bus’s (2011) meta-analysis examined the relationship between reading volume and reading achievement from preschool to post-secondary education and found moderate to strong relationships, where correlations with print exposure and reaching achievement become stronger for technical reading skills and intelligence from elementary school to middle school. A large longitudinal study conducted with students in 1st–9th grades found that in 1st–3rd grades reading achievement predicts leisure reading volume; but in 3rd grade and beyond, leisure reading volume, especially leisure book reading, predicts reading achievement (Torppa, Niemi, Vasalampi, Lerkkanen, Tolvanen, & Poikkeus, 2020).

Furthermore, research suggests that reading volume plays an important role in the reading motivation-engagement-performance relationship posited by the heuristic model. Specifically, studies found positive associations among reading volume, intrinsic motivation, and reading comprehension but negative associations between reading volume and extrinsic motivation, and extrinsic motivation and reading comprehension (Miyamoto, Pfost, & Artelt, 2019; Schaffner, Schiefele, & Ulferts, 2013; Stutz, Schaffner, & Schiefele, 2016). Moreover, reading volume has been found to mediate the positive relationship between intrinsic motivation and reading performance. Hence, the positive relationship between intrinsic motivation and reading performance is explained (or partially explained) by reading volume. However, the mediational role of reading volume on intrinsic motivation and reading performance is not always replicated. For example, a large study conducted with 4,000 elementary school students in 59 U.S. schools replicated positive associations between intrinsic motivation and reading volume and between reading volume and reading comprehension; but neither partial nor full mediation was replicated. Additionally, relationships among reading volume, intrinsic motivation, and reading comprehension are moderated by parental value of the reading language for bilingual migrant students (El-Khechen, Ferdinand, Steinmayr, & McElvany, 2016) and by immigrant status (Miyamoto et al., 2019). Overall, the relationship between reading volume and reading performance is positive and moderate to strong.

Reading for Enjoyment The Framework defines reading for enjoyment as “the goals, uses, purposes, reasons and benefits students have for reading in school and out of school” (NAGB, 2021, p. 59). More generally, reading for enjoyment (or “reading for pleasure”) is defined as reading that is done of the readers own volition with the expectation of gratification or fulfilment that will result from reading. Reading for enjoyment, however, also includes reading initiated at another’s request but that continues due to the readers own interest (Clark & Rumbold, 2006). Positive relationships between reading for enjoyment and scores on

February 2024 ETS Confidential Page 16 large-scale assessments are well-documented (PIRLS, 2006; OECD 2002, 2010, 2011). Meta-analytic studies show that leisure reading has positive relationships with basic reading skills and reading comprehension (Mol & Bus, 2011). Additionally, for elementary and middle school students, relationships between leisure reading and basic reading skills were stronger for low-ability readers than for high-ability readers.

When asked why they read, students 11–18 years of age indicated that reading helps them understand different people or cultures, increases knowledge of new subjects, and encourages them to try new hobbies. When asked how they would describe reading, students indicated reading was informative or educational, relaxing, and fun (Nestle Family Monitor, 2003). In a large study of 8,000 elementary and middle school students, the most frequently reported reason for reading was because reading is a “life skill,” followed by reading helps students find wanted and needed information, and reading is fun (Clarke & Foster, 2005). Fewer students reported that they read to understand the world, other cultures, and themselves. Lastly, students also indicated they read to relax (Dungworth, Grimshaw, McKnight, & Morris, 2004).

More recent small-scale qualitative research among elementary school students indicates that students’ reasons for reading are varied and guide what they choose to read (McGeown, Bonsall, Andries, Howarth, & Wilkinson, 2020). When asked why they read books, 94 percent of elementary students voiced affective and emotional reasons. About 42 percent of students reported reading to feel calm and relaxed. Specifically, students said, “I can drain all my emotions out on the book, so if I’ve had a hard day, I can just read” and “After a long day at school … it just makes me want to read a book.” Thirty-six percent of students said they read because it makes them happy or is entertaining. For example, students said, “Happy? I just enjoy it”; “And it makes me laugh sometimes”; and “If I read a funny book, it just lightens my mood.” Lastly, 30 percent of students reported reading books for excitement. For example, students said, “You don’t want to put it down, because every page you read, there’s another cliff-hanger” and “Oh, what will happen here, what will happen here? Oh, can I just read for two more minutes?” (p. 601). Middle school students also reported reading for affective and emotional reasons (Louick, Leider, Daley, Proctor, & Gardner, 2016). When interviewed middle school students said, “I'm happy I can read because the reading is just like fun. It's relaxing, and it's nice and quiet” and “I just like to read, and then I feel good when I read and stuff, because it's kind of relaxing. Yeah. I feel like, about myself when I read, because reading's nice, and calm, and stuff” (Louick et al., 2016, p. 264).

Sixty-one percent of elementary students indicated reading books for immersion or escapism. For example students said, “It’s quite nice to sort of feel like you’re sort of letting go from whatever is happening around you, and you can go deep – by yourself – deep in the world of the book”; “I feel like … if it’s written very well, I feel like I’m there”; “I feel like I am actually in that place, and I don’t want to leave it”; and “I kind of feel what the book wants me to feel” (McGeown et. al, 2020, p. 601). Middle school students also reported immersion and escapism as their reasons for reading. For example, one student said, “I like to read … because it, like, um, like, you can imagine things in your mind while you're reading. Like, you can like, pretend like you're actually there” (Louick et al., 2016, p. 264).

Additionally, elementary students indicated reading for “familiarity” or rereading favorite books, reading favorite authors, reading to complete a series, or reading to spend time with favorite characters. Thirty-six percent of students voiced reading for familiarity. For example, students said, “I prefer series, because if it’s one book, there’s always a part of the end that you want to know what happens next, and

February 2024 ETS Confidential Page 17 in series you can almost go on forever” and “Because I’ve read the whole series three times, and I’m thinking about reading it a fourth time, I enjoyed it that much” (McGeown et al., 2020, p. 601).

Overall, relationships between reading for enjoyment and reading performance are moderate to strong and may differ by student-ability level. In the context of qualitative and mixed-method studies, students reported reading for many reasons including gaining factual knowledge, increasing understanding of the world, others, and self. Students reported affective and emotional reasons for reading including happiness, relaxation, excitement and to become immersed in stories. Students reported reading to enjoy the social aspects of reading and reading as a habitual practice. Students also reported external reasons for reading including to gain rewards and comply with requests. Older students reported reading because it is instrumental for future employment. Indeed, small-scale qualitative and mixed-method research indicates students report reading for many purposes including intrinsic reasons (i.e., reading for enjoyment of the act itself) and extrinsic reasons (i.e., reading to attain external rewards).

The section that follows discusses the results of meta-analytic research on the relationship between several aspects of reading motivation, including intrinsic and extrinsic motivation, and reading performance.

Motivations for Reading According to the Framework, motivations for reading are “students’ attention, effort, sense of self-efficacy, interest, and value for reading a particular text with a unique set of tasks and questions related to it” (NAGB, 2021, p. 59). Indeed, Guthrie & Klauda’s heuristic model of reading posits that reading motivation (e.g., intrinsic motivation, self-efficacy, value and pro-social reading behaviors and beliefs) influences behavioral reading, which in turn influences reading performance.

Intrinsic motivation Intrinsic motivation for reading refers to an individual’s desire to read for the sake of reading itself and is often contrasted with extrinsic reading motivation which is reward contingent (Toste, Didion, Peng, Filderman, & McClelland, 2020). The relationship between intrinsic motivation and reading has received much empirical attention. Meta-analytic research found that intrinsic motivation is moderately positively related to reading achievement (r =.32, 95% CI [.22,.41], Toste, Didion, Peng, Filderman, & McClelland, 2020). Moreover, intrinsic motivation was more strongly related to reading achievement than other aspects of motivation, namely performance and mastery goals, beliefs about self (e.g., self-efficacy), beliefs about reading (e.g., expectations, values), reading attitudes, and reading interests.

Extrinsic motivation Extrinsic motivation for reading refers to an individual’s desire to read to attain external rewards (Toste, Didion, Peng, Filderman, & McClelland, 2020). The relationship between extrinsic motivation and reading has received much empirical attention, culminating in the belief that extrinsic motivations for reading can threaten intrinsic motivation. Meta-analytic research found that extrinsic motivation (operationalized as performance achievement goals) has a very small positive relationship with reading achievement (r =.05, 95% CI [.004,.09], Toste, Didion, Peng, Filderman, & McClelland, 2020). Moreover, extrinsic motivation had the weakest relationship to reading achievement among all other aspects of motivation, namely mastery goals, beliefs about self (e.g., self-efficacy), beliefs about reading (e.g., expectations, values), reading attitudes, and reading interests. Research conducted at Educational Testing Service (ETS) is consistent with these meta-analytic results. Specifically, 2017 NAEP reading and

February 2024 ETS Confidential Page 18 mathematics data revealed that performance achievement goals had weak relationships with NAEP reading achievement scores, in general, and compared to mastery achievement goals (Gill, 2018).7

Reading self-efficacy Reading motivation literature identifies self-efficacy as an important motivating factor (Gill & Fox, 2017;

Cho, Kim, Ju, & Lee, 2021; Conradi, Jang, & McKenna, 2014). Those with high self-efficacy tend to be more diligent, persistent, and willing to seek help to accomplish a task (Linnenbrink & Pintrich 2003);

Pajares, 1996; Schunk, 1996). Self-efficacy is viewed as an affirming motivation that is positively correlated with achievement (Guthrie, Klauda, & Ho, 2013). Reading self-efficacy is an individual’s belief about their ability to successfully complete reading-related tasks (Unrau, Rueda, Son, Polanin, Lundeen, & Muraszewski, 2018). Meta-analytic research found that beliefs about self (e.g., self-efficacy, self-concept, self-determination) are moderately positively related to reading achievement (r =.28, 95% CI [.24,.32], Toste, Didion, Peng, Filderman, & McClelland, 2020). Moreover, beliefs about self were more strongly related to reading achievement than all other aspects of motivation except intrinsic motivation.

Reading value Reading motivation literature identifies reading value as an important motivating factor (Cantrell, Rintamaa, Anderman, & Anderman, 2018; Cartwright, Marshall, & Wray, 2016; Guthrie, Klauda, & Ho, 2013; Guthrie, Wigfield, & You, 2012; Wigfield, Gladstone, & Turci, 2016). Theories that identify reading value as an aspect of motivation draw from expectancy-value theory, which states that motivation is a combination of one’s expected reading performance and the value they placed on reading (Neugebauer & Fujimoto, 2020). Reading value is viewed as an affirming motivation that is positively correlated with achievement (Guthrie, Klauda, & Ho, 2013). Reading value is the belief that reading is important either because it is useful or interesting. Toste et al.’s (2020) meta-analysis revealed that beliefs about reading, including reading value, have a small positive relationship with reading achievement (r =.16, 95% CI [.09,.23]). The relationship of reading value with reading achievement is positive but smaller than that of intrinsic motivation and reading self-efficacy.

Reading interest Reading motivation literature identifies reading value as an important motivating factor (Ahmadi, Ismail, & Abdullah, 2013; Chiu & Chow, 2010; Guthrie, Klauda, & Ho, 2013; McTigue, Solheim, Walgermo, Frijters, & Foldnes, 2019). Reading interest is viewed as an affirming motivation that is positively correlated with achievement (Guthrie, Klauda, & Ho, 2013). Interest refers to ‘‘relatively stable evaluative orientation toward a certain domain’’ (Schiefele, 1999, p. 258). Interested readers are those who feel involved and situated in the things they read (Renninger, 2001). In the reading literature, interest is often conceptualized as a general interest in reading (e.g., “When my parents give me a book as present, I am interested in it,” “I read to learn about my topics of interest” [Becker, McElvany, & Kortenbruck, 2010; Retelsdorf, Köller, & Möller, 2011]). For younger students, reading interest is also conceptualized as strong positive affect for a book or topic, having a favorite author or book, and rereading part or all of a book (Guthrie, Hoa, Wigfield, Tonks, Humenick, & Littles, 2007). Reading interest has also been conceptualized as the types of text (e.g., books, newspapers, magazines, internet articles) or topics (e.g., travel, nature, romance, humor) students prefer to read (Dollinger, 2016).

7 Due to their weak relationship with NAEP reading achievement scores, performance achievement goal items were removed from the reading survey questionnaire, creating space for constructs with stronger relationships to achievement.

February 2024 ETS Confidential Page 19

Generally, relationships between interest in reading and reading achievement are small to moderate and positive (Al Jefri & Areepattamannil, 2019; Karasinski & Anderson, 2017; Marsh, Hau, Artelt, Baumert, & Peschar, 2006). Additionally, research shows that the types of text and topics that students are interested in differentially impact reading achievement (Dollinger, 2016).