Appendix_B_Final_Alignment_Report_10_25_13.pdf

Promoting College and Career Ready Standards in Adult Basic Education

Task 3 Final Report: Alignment Review of the National Reporting System to the College and Career Readiness Standards for Adult Education

October 25, 2013

Larry Condelli Michelle Perry American Institutes for Research

Submitted by RTI International under U.S. Department of Education Contract No.

ED-CFO-10-A-0117-0001

APPENDIX B

Alignment Review of the NRS to the College and Career Readiness Standards for Adult Education iii

Contents

Executive Summary .........................................................................................................................v Approach ....................................................................................................................................v Findings......................................................................................................................................v Issues from the Review for the NRS and EFLs ....................................................................... vi

1.0 Introduction

The National Reporting System

2.0 Approach of the Review

Materials and Procedure

3.0 Analysis and Findings

Summary for ELA Standards Summary by Level for ELA Summary for Mathematics Standards Summary by Level for Mathematics Summary of Findings for the Standards for Mathematical Practice

4.0 Issues from the Review for the NRS EFLs and Assessments

EFL Descriptors Assessment Issues Technical Assistance and Implications for Instruction

5.0 Conclusion

References

Appendix Panel Reviewers Test Summaries for NRS Panels Instructions to ELA Reviewers National Reporting System (NRS) Educational Functioning Level (EFL) Descriptors and Assessment Matrix – Reading and Writing (ELA Standards)

Standards for English Language Arts/Literacy Rating Form Summary of Reviewers Ratings and Comments

Instructions to Math Reviewers National Reporting System (NRS) Educational Functioning Level (EFL) Descriptors and Assessment Matrix – Mathematics

Standards for Mathematics Rating Form Summary of Reviewers Ratings and Comments ..164 Standards for Mathematical Practice Rating Form v

Executive Summary The purpose of the Promoting College and Career Ready Standards in Adult Basic Education project was to identify a set of college and career ready (CCR) standards to prepare adult learners for success in higher education and training programs. To identify the standards, two expert panels, one for English language arts (ELA) and one for mathematics, reviewed the Common Core State Standards (CCSS) and selected the standards that describe appropriate content for adult students (Pimentel 2013). To gain insight into the possible implications of the standards for state adult education accountability requirements, the Office of Vocational Adult Education (OVAE), U.S. Department of Education, also included a second major activity of the project, a review of the alignment of the CCR standards with the educational functioning level (EFL) descriptors and assessments of the National Reporting System for Adult Education (NRS). This report describes the NRS review process and findings from the review and discusses the issues around possible modifications to the NRS that may be needed to accommodate state adoption of the CCR standards.

Approach Separate panels of adult educators, experts in the content areas of ELA and mathematics, conducted the review. Panel members were also knowledgeable of the NRS and CCSS. Project staff provided panel members with a rating scale and a matrix containing compiled test item content provided by test publishers. Panel members independently rated the degree of perceived alignment of each standard and then discussed their ratings and reached consensus at an in-person meeting. Finally, project staff compiled the consensus ratings into one document that was sent to panelists for final review and discussion.

Findings The panelists’ general conclusion for both ELA and mathematics was that the CCR standards are not adequately aligned with the NRS EFL descriptors or the approved assessments. They gave a rating of “aligned” to only one standard in ELA and only three standards in mathematics among all of the standards reviewed. Although panelists gave many ratings of acceptably aligned— meaning that the standard can fit into the EFL and existing tests but some content related to the standard is not included—the ratings often were based on finding the concept represented by the standard in only one assessment. Panel members considered content to be included if it dealt with the standard’s main idea, even if the nuance in wording did not match the CCR standards.

During the meeting discussions and debriefings, panelists expressed uncertainty in some of their ratings caused by the lack of complete information and noted that sometimes their ratings were based on assumptions about the content of assessments.

vi

Issues from the Review for the NRS and EFLs The review revealed the lack of alignment between the NRS EFL descriptors and approved NRS assessments, and the CCR standards. The current EFL descriptors are outdated and do not reflect what adult learners need to know to be college and career ready in the 21st century. A new framework is needed to guide adult education instruction.

The lack of alignment between the CCR standards and the current NRS raises several significant issues for OVAE to address. The issues focus on three major areas—revision of the EFL descriptors, assessment, and technical assistance and instruction.

Revision of the EFL Descriptors

The EFL descriptors need to be rewritten if closer alignment with the CCR standards is desired.

However, when revising the descriptors, OVAE should be mindful of the fine balance between providing a guiding framework and a more definitive statement of what is to be learned. The EFL descriptors outline skills that students should have upon entering an NRS level, but there are no exit-level descriptors that designate what skills the student should have upon completing a level. Several review panel members suggested that OVAE could consider changing this framework to include both entry and exit descriptors as part of the revised NRS.

Since there has been no review of the CCR standards in relation to the NRS EFL descriptors or assessments for English as a Second Language (ESL) instruction, the EFL descriptors for ESL cannot be revised. Although it appears likely that the ESL descriptors and corresponding assessments do not align, further review is needed to determine how to use the CCR standards for adult ESL students.

Assessment Issues

If the EFL descriptors are revised, all of the NRS assessments will be out of compliance and publishers will have to revise their assessments or develop new ones. Before new or revised assessments are available, states that adopt the standards will have to continue to use existing assessments. Additional guidance will be needed on specific areas lacking alignment and how (or whether) to assess these areas.

This review makes apparent that all of the currently approved tests for NRS approval must be revised in some way. While the extent of revision needed may vary by test, the independent review and approval process for NRS assessments will remain necessary to ensure the quality of tests and their ability to reflect the CCR standards validly and reliably. To be approved, test publishers would have to demonstrate that their newly developed or revised tests are aligned with the standards and can measure educational gain according to the new framework.

Adult education lacks a writing test, yet writing is recognized as a critical skill for college and career readiness, as identified through the standards. Although not necessarily required for NRS vii reporting, writing assessments for adult education are needed if writing is to be a focus of instruction.

A future alignment review is likely to identify the need for new or revised adult ESL assessments. The CCR standards also include speaking and listening for adult basic education (ABE) students and assessments may be needed to assess these skills for native English speakers as well.

Technical Assistance and Implications for Instruction

An organized rollout of any changes to the NRS and a technical assistance approach will smooth the path to successful integration of the standards into the adult education system. States will need support in the following areas to help them implement the changes resulting from the adoption of the CCR standards: developing, interpreting, and using the revised EFL descriptors;

defining areas of emphasis for NRS reporting; using assessments and interpreting assessment results; and adapting state data systems and reporting methods. Panel members raised three additional issues related to instruction from the NRS lens.

• The EFL descriptors and NRS assessments for mathematics focus mostly on computation and simple application but not on problem-solving strategies and the deeper analysis of concepts implied by the CCR standards. Panel members noted that there needs to be special emphasis on improving mathematics instruction and assessment of mathematical concepts.

• Some states have invested in their own standards framework and may not want to adopt the CCR standards. OVAE can assist these states by providing guidance or technical assistance in evaluating the level of alignment of their standards against the revised EFL descriptors and CCR standards.

• If states adopt the CCR standards and EFL descriptors are rewritten, there could be a higher proportion of students in the lower levels, than is currently the case, who may need more instructional time to advance. This possible shift in the distribution of students to lower levels could create a misperception that the program’s performance has declined. The field would need to explain why performance on educational gain is lower and manage expectations about the time it takes to complete NRS levels.

1.0 Introduction

The move toward standards-based education has been one of the most significant developments in education in recent years. The Common Core State Standards (CCSS) have identified what is essential for students to know to be prepared for postsecondary education and work, and almost all states have indicated that they will adopt them. In adult education, the Office of Vocational and Adult Education (OVAE), U.S. Department of Education, has supported state efforts to develop content standards for more than a decade through several professional development projects.

Promoting College and Career Ready Standards in Adult Basic Education continues this effort.

This project included two major activities. For the first activity, OVAE asked panels of experts to identify a set of standards appropriate to preparing adult learners for success in higher education and training programs. To identify the standards, two expert panels, one for English language arts (ELA) and one for mathematics, reviewed the CCSS and selected the standards that describe appropriate content for adult students. The final report, College and Career Readiness Standards for Adult Education, released in April 2013, presents the standards and describes the year-long selection process (Pimentel 2013).

States’ adoption of the College and Career Readiness (CCR) standards has the potential to significantly change instruction in adult education, with implications for student assessment and program accountability as measured through the National Reporting System (NRS). To gain insight into the possible implications of the standards for the NRS, OVAE included a second major activity of the project: a review of the alignment of the CCR standards with the NRS educational functioning level (EFL) descriptors and assessments. This report describes the review process and findings from the review and discusses the issues around possible modifications to the NRS that may be needed to accommodate state adoption of the CCR standards.

The National Reporting System The NRS has served as the accountability system for the federal adult education program since 2000. It identifies and defines measures for national reporting, establishes data collection methods, and establishes requirements for annual reporting to OVAE. States must report student demographic information, contact hours, and outcome measures for all students who attend the program for 12 or more hours. Outcome measures include educational gain, receipt of a secondary credential, entry into postsecondary education, entered employment, and retained employment.

Educational Gain and Educational Functioning Levels

The central measure of the NRS—and the one relevant to this review—is educational gain, which is measured through the pre- and posttesting of students. Upon entry into the program, a student is pretested and placed into an EFL based on test results. After a specified number of instructional hours based on the tests used, the student is posttested; the test results indicate whether a student is placed in the same or a higher EFL. Placement into a higher EFL is defined as a level advancement or educational gain.

The EFLs are defined by descriptors, which elaborate basic reading, writing, numeracy, functional, and workplace skills that can be expected from a person functioning at a given level. The skill descriptors illustrate the types of skills students at a particular level are likely to have. They do not provide a complete or comprehensive delineation of all skills at a level, but they provide examples to guide assessment and instruction (U.S. Department of Education 2013). Each of the six EFLs for adult basic education (ABE) students has separate descriptors (beginning literacy, beginning basic education, low- and high-intermediate basic education, and low- and high-adult secondary education); there are six descriptors for English-as-a-second language (ESL) EFLs.

The original EFL descriptors were written more than 20 years ago and were substantially revised in 1998 and 2001 for the NRS to reflect the assessments and instructional goals of states. They represent the skills students have upon entering a level and were intended as a framework to guide states on student placement and assessments for measuring educational gain. The appendix to this report includes the EFL descriptors for ABE.

NRS Assessments and the Test Review Process

OVAE requires that assessments used for state NRS reporting be aligned to the EFLs and be able to measure student advancement across levels through pre- and posttesting. In 2008, OVAE established a regulated process whereby independent expert panels review the tests’ alignment with the EFLs and whether the tests can measure educational gain in a valid and reliable manner, according to accepted psychometric criteria (34 CFR 462). OVAE used the panels’ recommendations to approve the following five assessments for NRS reporting of educational gain for ABE students.

1. Comprehensive Adult Student Assessment System (CASAS), life skills, reading and mathematics tests and functional writing assessment

2. Tests of Adult Basic Education (TABE), mathematics and reading tests

3. Massachusetts Adult Proficiency Test (MAPT), mathematics and reading tests

4. WorkKeys, applied mathematics and reading for information tests

5. General Assessment of Instructional Needs (GAIN), English skills and mathematics skills tests

Additional tests for ESL also were reviewed and approved.

One or more of these tests is benchmarked to each EFL; the tests define the score ranges for student placement into the various levels. For example, a scale score of 465 on the TABE reading test places a student into the low-intermediate level of ABE. States are required to use the tests scores to place students into levels for instruction and NRS reporting.

2.0 Approach of the Review

The intent of the review was to compare the CCR standards against the NRS EFL descriptors for ABE and NRS-approved assessments to identify a level of alignment and gaps in coverage.

Separate panels of adult educators, experts in the content areas of mathematics and ELA, conducted the review. Panel members were also knowledgeable of the NRS and CCSS. In addition, one member of each panel served on a panel that selected the CCR standards from the CCSS. The appendix provides information about panel members.

This review was not intended to be a comprehensive alignment evaluation, such as those conducted by states and school districts for the CCSS (e.g., Webb 1997), where test items are analyzed by instructional and assessment experts to determine a test’s level of alignment with standards. Rather, this review was intended as a first look at the implications of state adoption of the CCR standards for the adult education program’s accountability requirements, as described through the NRS framework. Panelists did not have specific test items and did not have access to the assessment task or content represented by individual test items; instead they relied on content summaries provided by test publishers, including item specification created by publishers as part of test development. This approach precluded detailed item- and task-level analysis. Instead, the panels’ analysis provided a general look at test content in comparison to the content implied by the CCR standards.

A notable example of the implication of the lack of test items for the review is evident in the area of “text complexity.” Research by ACT, Inc. (2006) revealed that the clearest differentiator between students who are prepared for college and those who are not is their ability to read and comprehend complex texts. Additional research shows that the complexity of text in college textbooks and workplace materials has remained constant or increased somewhat over the last 50 years while the complexity of high school texts has decreased, creating a “complexity gap.” The result is that many college students are unprepared for reading at the postsecondary level (see Appendix A of the CCSS and Stenner, Koons, and Swartz [2010] for detailed discussions of this issue). To help overcome this gap, CCSS (and the CCR Standards for ELA) adopted a staircase of text complexity (Reading Standard 10) that starts with beginning reading and raises the bar for text complexity at each level so students are ready for the demands of college and career reading.

Rather than focusing solely on how students read, the standards also focus on the complexity of texts read by students.

For ELA reading standards, then, knowledge of text complexity as defined by the CCSS model is needed to evaluate fully the alignment of the NRS EFL descriptors and test content. For example, Standard 2 for reading entails determining the central idea of a text. This standard is similar at each level, differing mainly as it is to be applied to higher levels of text complexity at each successive level. Likewise, Standard 4 deals with determining the meaning of words and phrases in texts. Again what differentiates the performance is the complexity of the texts that http://www.corestandards.org/assets/Appendix_A.pdf students are asked to read and the vocabulary they contain. The remaining reading comprehension standards grow both in their difficulty and in the complexity of texts to which they are to be applied. Panelists did not have access to the texts used in assessments and thus could not fully evaluate the reading standards on this critical dimension. Therefore, their ratings on Reading Standards 1–9 are based on incomplete information and may not reflect the actual degree of alignment.

Materials and Procedure To compare the CCR standards against the NRS EFL descriptors and NRS-approved assessments, each panel member received the NRS EFL Descriptors and Assessment Matrix,1 Test Summaries for NRS Panels,2 and Standards Rating Form. The mathematics panelists also received the Standards for Mathematical Practice Rating Form. In addition to these materials, all panelists received a set of instructions detailing the materials and review process. Project staff compiled the item content provided by test publishers into the NRS EFL Descriptors and Assessment Matrix. The information from each approved test and EFL descriptor was organized according to ABE level and type of skill (i.e., reading, writing, mathematics, or functional).

Assessments varied in complexity and length and the content summaries in the charts reflected this variation. In addition, test publishers provided different levels of detail about their tests, which affected the amount of information the panelists could distill regarding the assessment.

The Test Summaries for NRS Panels document contained tables for each test that included the intended population, purpose, skill levels covered, and number and type of items in each test.

Staff also developed a rating rubric for panel members to rate whether the EFL descriptor and assessments were aligned with the CCR standards. The Standards Rating Form contained the CCR standards and was organized according to EFL, using grade-level approximations defined in the EFLs (e.g., grade levels 2–3 were considered beginning ABE).

The Standards for Mathematical Practice are part of the CCSS and included in the CCR standards.

They provide guidance for teachers on the skills that mathematically proficient students should demonstrate. Because they are not broken down by level and are not included in the EFL descriptors, the Standards for Mathematical Practice Rating Form that the mathematics panelists received contained the standard and a set of three questions designed to determine whether the standard was reflected in the EFL descriptor. All of these materials are provided in the appendix of this report.

Project staff sent the materials to panel members for advance review and held a conference call to explain the process. Panel members then had three weeks to independently rate the alignment

1 For the NRS EFL Descriptors and Assessment Matrix, publishers provided item specifications for their tests at each NRS level, which was used to summarize test content within the matrix.

2 For the Test Summaries for NRS Panels, publishers provided general information about their tests which was used to create the test summaries. The publishers then reviewed the summaries for accuracy.

of each standard by level, on a scale of 1 to 4, using the Standards Rating Form and NRS EFL Descriptors and Assessment Matrix for reference:

1- Full Alignment. The standard fits into the EFL and existing tests can assess expected performance. The standard is reflected in the NRS descriptors and in the content of all or most of the tests.

2- Acceptable Alignment. The standard can fit into the EFL and existing tests can assess performance adequately or can assess the general underlying concept. The main concepts within the standard are reflected in the EFL and at least some of the assessments. Some content may be left out of some or all of the assessments or EFL descriptors but reflect the main concepts.

3- Nonalignment. The standard does not match the EFL descriptor and approved tests cannot assess expected performance. The standard is not reflected in the EFL descriptors or in the NRS assessments at all or does not include important concepts within the standard.

4- Misaligned. The standard fits into a different EFL than assigned. The standard is reflected in the EFL descriptor or test content of another NRS level.

Each panel then met for a one and a half day, in-person meeting to discuss ratings and reach consensus. During the meeting, panelists were instructed to pay particular attention to CASAS and TABE because all states use one or both of these. Because the Standards for Mathematical Practice are not included in the EFL descriptors, the mathematics panel did not rate these standards for alignment. Instead they were asked to assess whether the practice standard was reflected by the NRS EFL descriptors, had language or topics that could be matched to the standard, or contained a gap in coverage (i.e., the practice was not reflected at all).

During the meetings, the panelists explained that they often based their ratings on the evidence of standard content found in the EFL descriptor, or at least one of the two most commonly used assessments. Consequently, a rating of acceptable alignment may mean that the concept represented by the standard was found in one assessment and not necessarily fully represented by the level.

Project staff recorded the consensus ratings and justification. Panelists’ individual rating forms were then collected and reviewed for accuracy and supporting reasoning. The results were compiled into one document that was sent out to panelists containing only the standards that they had determined were aligned or somewhat aligned to the EFL descriptors and assessments.

(Refer to the appendix for the consensus document.) A final debriefing conference call was held for each panel about one month after the meeting in which the results were reviewed and panelists provided additional comments.

3.0 Analysis and Findings

The panelists’ general conclusion for both ELA and mathematics was that the CCR standards are not aligned with the NRS EFL descriptors or the approved assessments. It is telling that they gave a rating of 1 on our scale (i.e., aligned) to only one standard in ELA and only three standards in mathematics among all of the standards reviewed. Although panelists gave many ratings of 2 (acceptable alignment),3 the ratings often were based on the concept representing the standard found in only a single assessment. In their comments about ratings and during the debriefings, panelists expressed uncertainty in some of their ratings caused by the lack of complete information; ratings were sometimes based on assumptions about the content of assessments. For example, as already discussed, panelists did not have access to the texts used in the ELA assessments and thus could not fully evaluate their alignment to the ELA reading standards. Therefore, their ratings on level of alignment for those standards were based on incomplete information and may not reflect the actual degree of alignment. Unanimous agreement on initial ratings among panelists was infrequent but they came to consensus after discussion. Panelists rated many standards 3 or 4, meaning the standards were unaligned or possibly aligned at a different EFL.

Summary for ELA Standards Panelists’ ratings indicate that the NRS EFL descriptors and current assessments provide incomplete coverage and descriptions of ELA standards and are not aligned. There is little coverage in the current assessments for any content at the two lowest NRS levels. The EFL descriptors lack coverage of writing tasks and the single NRS-approved writing assessment was rated as unaligned with the CCR standards. (The approval for use of this test in the NRS will soon expire, leaving no approved writing tests for NRS reporting.) There were more ratings of acceptable alignment (i.e., some content related to the standard was included) for reading standards, especially at higher levels, but coverage appeared to be inconsistent and incomplete and no measures of text complexity were available, which is particularly consequential at the higher levels.

Most language and reading foundation standards appear to be covered inadequately and received ratings of nonalignment. This lack of alignment is most apparent in the areas of the instructional shifts identified by Lafond (2012) and Pimentel (2013). The ability to cite evidence from texts and comprehend and use content-rich nonfiction to build knowledge are two of the shifts prompted by the CCSS that are believed to be critical for college and career readiness. However, panelists saw little or no evidence that current tests ask students to present careful analysis or well-defended claims nor that the EFLs describe students’ ability to cite evidence from texts to present careful analyses, well-defended claims, and clear information. Another area of weak

3 A rating of acceptable alignment indicates that the standard can fit into the EFL and existing tests can assess performance adequately, or can assess the general underlying concept. However, some content related to the standard is not included in some or all of the assessments or EFL descriptors. See page 6 and the rating form in the Appendix for more detail on the rating rubric.

alignment appears to be with standards that require comprehension of informational texts from content areas and technical subjects or from literature (the ELA standards are divided by literature and informational text).

As discussed earlier, practice with texts of appropriate complexity—the third instructional shift and key emphasis in the CCR standards—could not be evaluated based on the material provided by test publishers, thus panelists gave no rating at all to Reading Standard 10. Summaries of ratings by level are presented below and the appendix includes the summary of ratings for each standard.

Summary by Level for ELA Level A (K–1)

Panelists agreed on acceptable alignment of only one standard at this level, Standard 2 in the language standards. This standard describes command of conventions of English capitalization, punctuation, and spelling. No level of alignment was rated for other language standards or for reading, or reading foundations. There was also no alignment rated for the writing standards.

Level B (2–3)

Within the reading standards, panelists rated acceptable alignment for Standards 1, 2, and 4, which reflect understanding key details in texts, determining main ideas, and determining the meaning of specific words and phrases in texts. Standards 5, 6, and 8—which address an understanding of text features, point of view, and how reasons support specific points in a text— were rated as having acceptable alignment as well. Standards 3, 7, and 9, which reflect the ability to describe the relationship between ideas and events, use information from illustrations, and compare two texts on the same topic, were rated nonaligned. No writing standards were aligned because none of the standards are addressed by the single NRS-approved writing assessment.

Within the language standards, panelists rated standards that address grammar, spelling, punctuation and other conventions (Standards 1 and 2) as having acceptable alignment. They rated Standard 4—concerned with determining the meaning of unknown and multiple-meaning words and phrases—similarly. Standards 3, 5, and 6, reflecting making effective choices for meaning or style, knowledge of figurative language and nuances in word meaning, and using words and phrases acquired through conversations, appeared to be nonaligned. Among the reading foundation standards, panelists saw evidence of acceptable alignment for grade-level phonics and decoding standards but not fluency and accuracy supporting comprehension.

Level C (4–5)

Panelists found acceptable alignment for reading standards 1–5 and 8, which address making logical inferences from texts; determining, summarizing, and analyzing main ideas and themes;

explaining ideas, events or concepts within a text; determining technical, connotative, and figurative meanings of words and phrases; determining the overall text structure; and explaining how reasons and evidence support particular points in a text.4

No evidence of alignment was apparent to panelists for standards relating to assessing how points of view shape style and content, interpreting information presented in diverse formats, or integrating information from multiple texts on the same topic (Reading Standards 6, 7 and 9).

Language and reading foundation standards did not appear to align with the standards and were rated nonaligned. One exception was Language Standard 4, clarifying the meaning of unknown and multiple-meaning words and phrases from context, which panelists rated as acceptably aligned. Once again, writing was rated as unaligned due to the lack of writing coverage by the NRS assessments.

Level D (6–8)

As with Level C, panelists found acceptable alignment for reading standards that concerned making logical inferences from text; determining, summarizing, and analyzing central ideas and themes; analyzing how ideas and events interact and connect within a text; determining technical, connotative, and figurative meanings of words and phrases; analyzing how parts of a text relate to create structure; and delineating and evaluating arguments and specific claims in a text (Standards 1–5 and 8). In addition—unlike for Level C—assessing how point of view and purpose shapes style and content (Standard 6) was rated as acceptably aligned. Panelists rated as unaligned Standards 7 and 9, concerning integrating content from diverse formats and analyzing how multiple texts address the same topic to build knowledge.

For language standards, panelists did not see alignment for standards that address grammar, spelling, punctuation and other conventions (Standards1 and 2), nor alignment for Standard 3, on making effective choices for meaning or style. Panelists found acceptable alignment for Standard 4, on clarifying the meaning of unknown and multiple-meaning words and phrases from context.

They also rated Standard 6 as having acceptable alignment. The latter standard addresses acquiring and using academic vocabulary. Once again, with only one NRS-approved writing assessment, writing standards were rated as nonaligned.

Level E (9–12)

Panelists rated the content of many more reading standards as acceptably aligned. In fact, they considered all of them acceptably aligned with the main assessments or descriptors with two exceptions: Standards 7 and 9. For Standard 7, panelists reached consensus only on the integration of quantitative and technical analysis with qualitative analysis; panelists found no evidence of alignment on the broader concept of Standard 7 that relates to integrating and

4 Ratings about reading standards at all levels should be considered in light of panelists’ lack of information about text complexity within assessments, as discussed in Section 2 above.

evaluating content from diverse formats. They also did not find evidence of Standard 9 on analyzing foundational U.S. documents.

Language Standard 4 in Level E is the only standard in the ELA review to which panelists gave a rating of fully aligned (i.e., 1 on our scale). This standard describes clarifying the meaning of unknown and multiple-meaning words and phrases by using context. They noted that they saw this standard reflected in multiple assessments and implicitly in the EFL descriptor. All remaining language standards in this level, as well as the writing standards, were rated unaligned.

Summary for Mathematics Standards The panelists concluded that current assessments and EFL descriptors focus on computation and basic applications and appear to have little alignment with CCR standards that focus on real problem-solving approaches (especially real-world applications), the use of tools, and graphic representations to understand concepts and higher-order problem-solving skills. There does not appear to be alignment with the deeper skills and foundations represented by the standards; this was especially noticeable at Level E. In the NRS descriptors, geometry, statistics and probability, the use of data, and algebra are focused on basic equations and problem solving but do not represent a deeper conceptual understanding of relationships among mathematical concepts.

Among assessments, there appears to be a lack of alignment with standards that reflect the conceptual understanding of key concepts, procedural skill and fluency, statistics and probability, and the rigorous application of mathematics in real-world contexts. Summaries of ratings by level are provided below and the appendix includes a summary of ratings for each standard.

Summary by Level for Mathematics Level A (K–1)

Panelists identified only three standards as acceptably aligned at this level. Within Number and Operations-Base 10, they noted evidence of acceptable alignment for standards that describe comparing two-digit numbers based on the meaning of tens and ones digits and adding within 100 using models, drawings, and/or strategies based on place value operations or relationships between adding and subtracting. Panelists also rated a standard for geometry concerning the analysis and comparison of two- and three-dimensional shapes as acceptably aligned. Panelists rated the other geometry, or Base-10 standards, or standards for operations and algebraic thinking and measurement and data as nonaligned.

Level B (2–3)

Within Level B, panelists saw evidence of sporadic and inconsistent alignment in all content areas. For Number and Operations-Base 10, panelists found six standards that reflected an understanding of place value rounding to the nearest 10 or 100; reading and writing numbers to 1,000 (including skip counting); adding and subtracting strategies to 1,000; and multiplying numbers by 10 to be acceptably aligned. Standards that reflected a deeper conceptual understanding of place value (e.g., explaining why place value strategies work in addition and subtraction) appeared to panelists to be nonaligned.

Among standards on fractions, panelists rated only one standard area as acceptably aligned: basic understanding of what a fraction is (Standard 3.NF.1). Under operations and algebraic thinking, panelists agreed on acceptable ratings for interpreting products and quotients of whole numbers and the fluent ability to divide and multiply by 100, using strategies such as the relationship between multiplication and division. The apparent absence of content in the EFL descriptors and NRS assessments related to solving word problems in this domain were the reasons panelists rated standards reflecting problem solving as nonaligned for this level. Geometry standards also were rated as nonaligned, and under the measurement and data domain panelists saw acceptable alignment only for standards about measuring length, comparing lengths, and drawing a picture or graph. They rated nonalignment for other standards in this domain that address topics such as problem solving, understanding volume, computing, and understanding area or plotting measurement data.

Level C (4–5 and 6+)

Panelists’ ratings of alignment at this level for the Number and Operations-Base 10 standards were varied. Two standards were rated fully aligned, as were those describing the fluent ability to add, subtract, and divide whole-digit numbers using the standard algorithm. It appeared to panelists that the EFL descriptors and NRS assessments capture these computational skills.

Panelists also found acceptable alignment for standards explaining patterns of zeros when multiplying by powers of 10, and adding, subtracting, multiplying, and dividing decimals with the standard algorithm. However, other standards in this category, for example those reflecting the application of place value to division; computing four-digit decimals; and finding quotients and remainders of four digits and explaining calculations using equations, arrays, or models, were rated nonaligned.

Under fractions, panelists agreed on another rating of fully aligned for standards related to computation (Standard 5.NF.1) and adding and subtracting fractions with unlike denominators.

They rated as acceptably aligned a standard regarding the use of decimal notation for fractions.

However, panelists could see no evidence of alignment for other standards in this domain, many of which focus on solving word problems using equations, visual models or comparison strategies, so they gave ratings of nonaligned.

For operations and algebraic thinking, standards about writing and evaluating numerical expressions to solve word problems were rated acceptably aligned, as were two standards concerning understanding the meaning of variables and the ability to write a simple linear expression to represent a word problem. However, panelists did not see evidence of alignment for the other 15 standards in this domain, which cover a broad range of skills, including evaluating inequalities; using variables to represent quantities in real-world problems; and the ability to read, write, and evaluate expressions where letters stand for numbers. These were rated nonaligned.

For geometry, panelists saw only one standard, about drawing two-dimensional figures (e.g., points, rays, lines), as acceptably aligned. However, panelists found acceptable alignment on standards that reflects skills on the use of the four operations to solve word problems for distance, time, and mass; applying area and perimeter formulas to solve problems; and understanding angles and volume. Panelists rated as unaligned standards reflecting more complicated concepts of measurement and data. For example, standards about recognizing statistical questions on variability, using concepts of volume to solve real-world problems, and using plot lines to display measurement data in fractions were rated as nonaligned.

Level D (6+ and 7–8)

Panelists rated as acceptably aligned standards related to the number system that reflect an understanding of the meaning of positive and negative numbers, rational numbers, and the ability to perform the four operations with these numbers and represent them on lines. However, panelists did not see evidence of alignment with standards applying rational numbers to real-world problem solving or standards about irrational numbers. A standard concerning comprehension of ratios and rate and using these concepts to solve real-world problems also received a rating of acceptable alignment.

Panelists did not see evidence of alignment for standards involving expressions and equations except for a single standard about solving linear equations with one variable, which was rated acceptably aligned. All standards in the category of functions were rated as nonaligned. Within geometry panelists rated only the standards about knowing the formulas for area and circumference as acceptably aligned.

Of the 16 standards at this level about statistics and probability, panelists rated only three as acceptably aligned. These standards concerned understanding that statistics can provide information about a population by using a sample, using measures of center and variability to draw inferences about a population from a sample, and developing a probability model to determine the probability of events.

Level E (9–12)

Level E for mathematics had the unique distinction in this review of having panelists rate all standards as nonaligned with the EFL descriptors and assessments. These standards, which include the real-number system, algebra, functions, geometry, and interpreting categorical and quantitative data, reflect higher-level concepts and analytic skills that panelists did not see reflected in the EFL descriptors or assessments at this level. This apparent lack of coverage represents the largest gap identified in this review.

Summary of Findings for the Standards for Mathematical Practice With few exceptions, panelists concluded that the standards for mathematical practice (MP) do not appear to be reflected in the EFL descriptors or NRS assessments. Their comments are summarized by standard, with more detailed summaries provided in the appendix.

MP Standard 1. Make sense of problems and persevere in solving them. This standard was not evident to panelists in the lowest three levels of the EFL descriptors. There seemed to be some evidence in the upper three levels (i.e., in low-adult secondary education [ASE], there is a reference to texts, charts, and graphs), but it was unclear and did not get to the true meaning of the standard, including “making sense of problems” and the ability to “persevere.” There is some evidence that this practice standard is addressed in some form in the assessments, but the panel did not believe that any EFL descriptor or test adequately reflected the standard.

MP Standard 2. Reason abstractly and quantitatively. Evidence of this standard was not seen in the EFL descriptors until the low-ASE level. However, there, it appears more in work-oriented tasks and less in mathematics-specific tasks. There is some matching language found at low ASE in a few assessments but the richness and depth is not apparent there. In the panelists’ view, the concepts reflected in this standard are processes that are difficult to assess but that the teacher can evaluate in the classroom.

MP Standard 3. Construct viable arguments and critique the reasoning of others. Panelists found little mention of this practice standard in the assessments. There is some mention of constructing arguments in the low- and high-ASE levels. According to panelists, deductive reasoning—the heart of this standard—is missing from all EFL descriptors. The main gap is “critique the reasoning of others,” which was not apparent anywhere to panelists.

MP Standard 4. Model with mathematics. This standard seemed to be implicit at the EFL low-intermediate level in the functional skills. It was also seen at the high-intermediate level and ASE levels in some aspects, as well as in these levels of the assessments. There seemed to be more concrete language and models from the beginning ABE and other levels as they moved up (e.g., form concrete models, assess the reasonableness of answers, illustrate/explain and justify conclusions, and construct functions to model relationships).

MP Standard 5. Use appropriate tools strategically. Although there was some mention of calculators, measuring tools, and computer programs, panelists saw little mention of using tools as described in this practice standard. The EFL descriptors have some limited description of using computer programs, dials, and scales, but there is nothing represented in the low- and high- ASE levels of the descriptors and no apparent description of the use of tools in the assessments.

MP Standard 6. Attend to precision. The panel mentioned four big ideas that relate to this practice standard: 1) communication (explain clearly and precisely using appropriate language);

2) precision; 3) clear labels, definitions; and 4) accuracy, efficiency. In the EFL descriptors, none of these ideas was seen until the low-intermediate (accuracy) and low-ASE (perform basic mathematics functions) levels. Panelists described “weak” coverage of this standard in the assessments and did not find evidence of the notion of “precision,” as described by the standard.

MP Standard 7. Look for and make use of structure. Although some tests included references to “recognizing numerical patterns,” panelist saw only “bits and pieces” in the tests with little pattern or overlap. Panelists concluded that this standard cannot be considered as residing within the EFL descriptors.

MP Standard 8. Look for and express regularity in repeated reasoning. The high-ASE level mentioned “adapt” in the functional areas, but there was little evidence of any other element of this standard in the EFL descriptors. Panelists commented that the EFL descriptors were missing the key element of “repeated reasoning.” In addition, there appeared to be little evidence of the elements of this standard in current assessments.

4.0 Issues from the Review for the NRS EFLs and

Assessments This review revealed the lack of alignment between the NRS EFL descriptors and approved NRS assessments, and the CCR standards. The current EFL descriptors are outdated and do not reflect what adult learners need to know to be college and career ready in the 21st century. A new framework is needed to guide adult education instruction.

The lack of alignment and need for a new framework is not surprising because the descriptors were developed more than 15 years ago, when the educational landscape was substantially different. Nor is the need to revise this type of framework unique to adult education. It parallels a similar need for states to align their standards and curriculum frameworks with the CCSS, although most states so far have chosen not to revise their standards but have instead adopted the CCSS, with some adding a modicum of content.

This lack of alignment raises several significant issues for the field to address to move it toward informing instruction that is more firmly rooted in a college and career readiness framework for adult education. In this section we discuss key issues in three major areas surrounding the modification of the NRS. These areas—revision of the EFL descriptors, assessment, and technical assistance and instruction—are interdependent. Changes in any one area will require changes in each of the others.

EFL Descriptors The EFL descriptors have long guided state decisions regarding the content of instruction and assessment. Somewhat like content standards, they act as a frame for what students are to know and learn. They offer only very general statements of student abilities. By design, the descriptors were written broadly to accommodate the wide range of instructional approaches, state delivery systems, and the goals of ABE and ESL student assessment. They were not intended to provide a complete and comprehensive delineation of what learners are expected to do and know in order to be college and career ready in the 21st century. In addition, state control and flexibility were significant considerations in the development of the descriptors. The broad nature of the descriptors allowed states flexibility regarding what to teach and assess within the NRS accountability framework. Although the NRS required reporting on student progress along the skill levels defined by the EFL framework, specific decisions on instructional content and approach were left to the states.

This review revealed that the EFL descriptors need to be rewritten to provide alignment with a CCR framework.