Tech2_DLPT5 Framework.pdf

Technical Exhibit 4

Defense Language Proficiency Testing System 5

Framework

(DRAFT)

Test Review and Production Division

Language Proficiency Assessment Directorate

Defense Language Institute Foreign Language Center

Table of Contents

1. Purpose

2. Overview of the DLPT5 Testing System

2.1 Test design

2.2 Test Content

2.3 Test Format

2.4 Test Administration

2.5 Examinees and test users

3. Defining the test constructs

3.1 The Interagency Language Roundtable Skill Level Descriptions

3.2 Reading

3.3 Listening Comprehension

3.4 Measuring sustained performance

4. The DLPT5 development process, quality assurance, and DLPT5 calibration

4.1 Personnel selection and training

4.2 Item Development and Review Procedures for DLPT5

4.3 Calibration of DLPT5 in multiple-choice format

4.4 Piloting of DLPT5 in constructed-response format

4.5 CRT scorer training and scorer maintenance

5. Test maintenance

6. Future directions

6.1 Lower-ability examinees

6.2 Sustained performance

6.3 English use in the DLPT5

6.4 Note-taking

6.5 Memory and passage length

6.6 Interaction with audio

6.7 Difficulty and ILR level

6.8 Performance differences among subgroups of examinees

References

Appendix A: Interagency Language Roundtable Language Skill Level Descriptions

Appendix B: Validity and Reliability of DLPT5 Multiple-Choice Tests

Appendix C: Lower-Range DLPT5 Constructed Response Test Scoring Procedures

1. Purpose

The purpose of the DLPT5 Framework is to provide stakeholders and prospective test users with information regarding the purpose of the Defense Language Proficiency Test 5 (DLPT5) testing system, its test design, and how the Interagency Language Roundtable (ILR) Language Skill

Level Descriptions are used as the basis of test constructs, test development, test use, and test score interpretation and decision-making based on scores. Section 2 provides a description of what the DLPT5 testing system is. Section 3 describes the theoretical underpinnings of the test constructs: reading and listening comprehension ability and the sub-abilities that DLPT5 tests measure in relation to the ILR. Section 4 describes the process of generating the DLPT5 operational test forms. Section 5 outlines plans in support of the DLPT5 testing system, and, finally, Section 6 describes future plans to make the testing system more efficient and user-friendly. It is our hope that this framework document will help stakeholders and test users better understand the nature of the DLPT5, the issues relating to the use of the DLPT5 testing system, and the system’s capabilities and limitations, so that they will be able to make informed decisions about using DLPT5 test scores. The Test Review and Production Division (TRP) at the

Defense Language Institute Foreign Language Center (DLIFLC) was tasked to develop this document by the Defense Language Office (DLO). The initial drafts have been developed within DLIFLC with input from the Defense Language Testing Advisory Board (DELTAB).

2. Overview of the DLPT5 Testing System

The DLPT5 testing system is designed to assess the global language proficiency in reading and listening of native speakers of English who have learned a foreign language as a second language and of speakers of other languages who have very strong English skills. The DLPT5s measure proficiency as defined by the ILR Skill Level Descriptions, levels 0+ – 4 (see Appendix A) and are used to document and make decisions regarding operational readiness, incentive pay, assignment and training for civilian and military personnel in the US government who have foreign language skills. All DLPT5s are delivered on computer.

2.1 Test design

DLPT5s in many languages include both a lower-range test and an upper-range test. The lower-range test measures ILR proficiency levels from 0+ through 3, while the upper-range test measures ILR proficiency levels 3+ and 4. Examinees will normally take the lower-range

DLPT5; those who receive a score of 3 on this test may be eligible to take the upper-range test, depending on the policy of their organization.

There are two test formats for the DLPT5: multiple-choice (MC in the tabular displays) and constructed-response (CRT in the tabular displays). The multiple-choice format is used for languages with large examinee populations, and tests are scored by computer. The constructed-response format is used for languages with small examinee populations, and tests are scored by human raters certified to score such tests. Multiple- choice or constructed-response formats may be used for both lower-range and upper-range tests. A description of DLPT5 in both the multiple-choice and constructed-response formats follows.

2.1.1. DLPT5 in Multiple-Choice Format:

Upper-Range tests:

o The Upper-Range Reading Test contains approximately 36 questions with approximately 14 authentic written passages. Each passage may have up to five questions with four answer choices per question.

o The Upper-Range Listening Test contains approximately 36 questions with approximately 14 authentic audio passages. Each passage may have up to three questions with four answer choices per question. All passages are played twice.

o For research purposes, some questions are not scored. These questions do not count toward the final score the examinee receives. Examinees are told that such questions are in the test but are not told which questions are the unscored ones.

Lower-Range tests:

o The Lower-Range Reading Test contains approximately 60 questions with approximately 36 authentic written passages. Each passage may have up to four questions with four answer choices per question.

o The Lower-Range Listening Test contains approximately 60 questions with approximately 37 authentic audio passages. Each passage may have up to two questions with four answer choices per question. Passages at the beginning of the test are played once. Starting from Level 2, examinees hear the passages twice.

o For research purposes, some questions are not scored. These questions do not count toward the final score the examinee receives. Examinees are told that such questions are in the test but are not told which questions are the unscored ones.

2.1.2. DLPT5 in Constructed-Response Format:

Upper-Range Tests:

o The Upper-Range Reading Test contains approximately 35 questions with 12 authentic written passages. Each passage has two or three questions.

o The Upper-Range Listening Test contains approximately 35 questions with 12 authentic audio passages. Each passage has two or three questions and is played twice.

Lower-Range Tests:

o The Lower-Range Reading Test contains 60 questions with 30 authentic written passages. Each passage may have up to three questions.

o The Lower-Range Listening Test contains 60 questions with 30 authentic audio passages. Each passage has two questions and is played twice.

2.2 Test Content

The DLPT5 is designed to measure proficiency in the target language (TL) regardless of how it has been acquired. For this reason, and because of the broad proficiency orientation of the test, its content is not tied to any particular language-training program.

The DLPT5 is designed to measure examinee ability to understand materials produced by native speakers of the target language for communicative purposes. Thus, the test passages need to reflect the characteristics and the types of written and spoken texts in the target-language use domain, rather than being texts produced for instructional purposes. Whenever possible, the

DLPT5 passages are sampled from real-life sources such as signs, newspapers, radio and television broadcasts, the Internet, etc. When such sampling is not possible, passages must nevertheless have the characteristics of written or spoken texts used in real life for communicative purposes, as judged by native speakers of the target language. The passages cover a broad range of content areas, including social, cultural, political, economic, geographic, scientific, and military topics. Table 2-1 provides a select description of the content and topics

DLPT5 samples. It should be noted that Table 2-1 is by no means an exhaustive list of topics the

DLPT5 tests cover, and the ratio of the Final Learning Objective (FLO) content areas in each of the ILR skills levels measured varies slightly from language to language. In order to maximize content authenticity within each language, the DLPT5 test specifications allow for language-to-language variation in the content covered, even though this may lead to somewhat less standardization across languages.

Table 2-1

FLO Content Area Example Subcategories

Level 1 Military-Security Military ranks

Basic police subjects (arrests, etc.)

Customs officials

Traffic regulations

Economic-Political Government ministries

National events

Hiring and promotion

Marketplace activities

Basic bank transactions

Basic travel and tourism

Scientific and Technological Health services (appointments, prescriptions, hospitals, etc.)

Technological devices of daily life

Simple texts on scientific discoveries/ research

Cultural and Social Family event announcements

Holidays

School events

Cultural fairs

Sports

Entertainment

Obituaries

Geography: Physical, Political, Economic

Landmarks and spatial orientation

Weather and climate

Basic geographic relations

Level 2

Military-Security Warfare activities

Military career and training

Casualty reports

Arms sales and disarmament

Weaponry and equipment

Law enforcement issues

Crime and violence

Terrorism

Smuggling

Military intelligence issues

Economic-Political Transportation and travel

Population trends

Agriculture issues

Trade issues (WTO, tariffs, export/import)

Financial issues (budgets, inflation, taxes)

Economic growth

Industrialization

Employment

Political systems

Elections

Scientific and Technological Medical research and trends

Medical/scientific training

Inventions and new treatments

Discoveries

Technological progress

Cultural and Social Family issues (marriage, divorce, etc.)

Women’s rights/status

Educational issues

Customs and traditions

Religious subjects

Leisure, art, and entertainment activities

Media issues (freedom of the press, etc.)

Geography: Physical, Political, Economic

Climate issues

Topography

Urban vs. rural areas

Historical/famous sites

Natural disasters

Pollution and other environmental issues

Water issues

Level 3

Military-Security Analysis and discussion of military events and intelligence activities

Assessment of military cooperation or alliance among countries

Rationales for arms sales or military aid

Description of advanced weapons systems

Analysis of historical battles Analysis of security issues: justifications for security measures, etc.

Economic-Political Economic analysis

Political analysis

Economic policies

Assessment of a country’s economic situation

Development and growth issues

Political strategies

Political reforms and democracy

Analysis of international political and trade relations

Scientific and Technological Explanation of scientific, medical, and technological issues

Analysis of scientific achievements, discoveries, or findings (e.g., the ethics of cloning)

Research policies of governments or institutions

Impact of scientific progress on families, behavior, society, etc.

Scientific theories

Cultural and Social Evaluation or analysis of social events and relationships

Discussion of societal issues

(discrimination, violence, drugs, privacy, etc.)

Private vs. public education, the role of government in education, standards, etc.

Religious issues: the relationship of church and state, fundamentalism, persecution, value systems

Freedom of the press

Commentary on books, movies, artworks

Geography: Physical, Political, Economic

Analysis of environmental policies

Opinions on water issues (dam-building, etc.)

Analysis of border disputes

Use of natural resources

Plans for maintaining habitat (e.g., rainforest preservation, anti-desertification plans, etc.)

Level 4 “Think pieces” pertaining to any topics related to FLO

Written or spoken discourse in the vernacular or exhibiting individual characteristics in writing or speaking styles

2.3 Test Format

The DLPT5 is a bilingual test: The test passages are in the target language, but the rest of the test, including test instructions and test questions, is in English. All tests include instructions on how to take the test, examples of how to answer the questions, and question sets containing the following parts that examinees see on the computer screen:

o Orientation: This is a short statement in English that appears before each passage. Its purpose is to identify the context from which the passage was taken.

o Passage: This is the only element of the test that is in the target language being tested. The maximum length of a listening comprehension passage in the test is approximately 2.5 minutes. The maximum length of a reading comprehension passage is approximately 400 words. Most of the passages are much shorter than the maximum length. Examinees see written passages on the computer screen for the reading test but listen to audio passages though headsets for the listening test.

o Question statement: Each individual question is based on the passage, is written in English, and is posed in the form of a complete question or an incomplete statement. The questions may ask about what is explicitly stated in the passage or, in some cases, what is implied in it. Occasionally questions may ask about expressions that are used in the passage.

o Answer choices (in multiple-choice tests): Each question statement is followed by four answer choices, also written in English, only one of which is the correct answer to the question. Each answer choice is displayed on the screen with a button next to it that examinees will click to select that choice. Note-taking is not permitted, but examinees can change their selection by clicking on a different button.

o Answer box (in constructed-response tests): For each question, there is a box on the screen in which examinees type their answer in English. Examinees may also type notes in these boxes if they wish.

2.4 Test Administration

DLPT5 has two separate tests: a Reading Test and a Listening Test, which are administered at separately scheduled sessions and delivered via computer. Examinees have three hours to complete the Reading Test and three hours to complete the Listening Test. Approximately halfway through each test, examinees will be given a 15-minute break. The break does not count toward the overall test time.

2.5 Examinees and test users

The DLPT5 examinees are US military and government civilian personnel with varying degrees of proficiency in a foreign language. They may be native speakers of English or native speakers of other languages with very strong English skills. Their jobs require them to deal with written and/or spoken texts in a foreign language in many different contexts ranging from teaching to linguistic deciphering to content analysis. Some jobs require rapid text processing to obtain the gist or locate specific pieces of information, while others require careful analysis at the textual level for instructional purposes. These people take the DLPT5 tests annually as required by their specific job.

DLPT5 test users include the DLPT5 examinees, their supervisors, and people in government agencies who are using DLPT5 test scores for decision-making in areas such as job placement, graduation requirements, eligibility for further training, or receiving foreign language proficiency pay (FLPP).

3. Defining the test constructs

The DLPT5 test construct is the ability to understand written and spoken input intended for the general public in terms of the target-language use domain as specified by the ILR. The DLPT5 measures use of receptive language skills in context, i.e., listening comprehension ability and reading comprehension ability in a given target language for a communicative purpose. The

DLPT5 test design is informed by current theoretical understanding of second-language attainment and proficiency levels, research findings on the influence of texts and question types on test performance, and most importantly, the ILR Skill Level Descriptions, which serve as the standard against which examinee linguistic proficiency is measured. The two constructs being measured—reading comprehension ability and listening comprehension ability—can be seen as the ability of examinees to interact in certain ways with certain kinds of authentic written or spoken tests. Measuring these abilities necessarily involves the interplay among three variables:

examinees, authentic written or spoken texts, and tasks. This section explains how we define and operationalize these variables in the context of the DLPT5 so that readers will have a better understanding of how examinee performance is related to the target language use domain, and what is expected with regard to language ability at different ILR levels. This understanding, in turn, will allow test users to make informed inferences about examinee language ability.

An examinee’s ability with regard to communicative language use can be understood as consisting of two components: language knowledge and strategic competence (Bachman &

Palmer, 1996). Language knowledge is further broken down into four components: grammatical knowledge, textual knowledge, functional or illocutionary knowledge, and sociolinguistic knowledge.

Grammatical knowledge and textual knowledge support the functional capacity to use vocabulary, syntax, phonology (in spoken language) or graphology (in written language), cohesion, and rhetorical organization. Grammatical knowledge and textual knowledge involve control of the formal structures of the language in order to understand grammatically acceptable utterances or written sentences, and they involve the understanding of how these utterances or sentences are organized to form texts. Functional and sociolinguistic knowledge, often referred to as the pragmatic component of language use, enable examinees to interpret language appropriate for the particular language use setting and to relate the spoken or written texts to the communicative goals of the speaker or writer.

Strategic competence refers to the strategies or cognitive and metacognitive processes examinees use to interpret language effectively and to answer questions. This ability is necessary for examinees to determine the most effective means to achieve a communicative goal and execute those plans. These strategies include deciding what goal to pursue, what resources are needed to achieve the goal, and how to employ those resources to achieve the goal. Examinees use different strategies at different proficiency levels, and their choices of courses of action influence test performance. For example, those at higher ability levels tend to make better use of both cognitive and metacognitive processes, because their control of the linguistic elements is automatic, freeing more cognitive capacity for higher-level processes, such as making inferences, monitoring their comprehension, paying attention selectively, etc. Lower-ability language examinees, on the other hand, tend to rely on grammatical cues for meanings, and their control of target-language elements is not yet automatic, forcing them to devote much of their cognitive capacity to linguistic processing, thus leaving little room for higher-level processing.

Examinees’ performance on the DLPT5 depends on the integration of their language ability in the target language, their individual attributes apart from this ability, and the strategies they use.

The purpose of the test is to measure ability in the target language. It is therefore important to minimize the influence of extraneous factors on test performance.

Certain other individual examinee attributes may also affect their performance on the test, though these attributes are not part of language ability per se. They include such individual characteristics as age, gender, specific first-language, real-world knowledge (including cultural or topical knowledge related to the target language or knowledge in a specialized area), and affective schemata—meaning examinees’ affective responses to test passages and questions.

Affective responses may influence the way examinees process the passages and respond to the test questions. Various measures are taken in order to minimize the potential impact these factors might have on test performance. For example, in selecting test passages, passages that require specialized outside knowledge to understand are excluded, as are passages that could likely be understood on the basis of outside knowledge without need for the appropriate linguistic knowledge. Topics are avoided that would unduly advantage or disadvantage any particular age group, gender, or ethnic or religious group. Topics are also avoided that could be expected to evoke a strong negative emotional reaction in a large proportion of DLPT5 examinees (who are primarily military personnel). In addition, test anxiety is minimized by providing examinees with extensive familiarization materials.

Although some strategic competencies, as illustrated above, might be considered to be relevant to the construct, Test-taking strategies also exist that are construct-irrelevant. Examples of these include choosing an answer based on its length or not choosing an answer that includes certain key words such as “all” or “never.” The effect of such construct-irrelevant strategies on test performance is minimized to the extent possible. In the construction of questions, care is taken to minimize examinee ability to guess the correct answer when language ability itself is insufficient—for example, making answer choices similar to each other in terms of length and syntax. In addition, the familiarization materials mentioned above also help to reduce differences in test performance that result from differences in familiarity with the types of test passages and questions.

Research findings, to be discussed in sections 3.2 and 3.3, suggest that text types and question types may affect test difficulty and test performance. Care has been given to ensure that all

DLPT5 test passages target the general public and are coherent. The types of questions appearing in the DLPT5 tests are listed in Tables 3.2 and 3.4. Detailed discussions of text types and question types in the DLPT5 context are provided in sections 3.2 and 3.3.

Another significant potential construct-irrelevant variable affecting test performance is the wording and language of the questions themselves. Shohamy (1984) found that multiple-choice and open-ended items in examinees’ native language were easier than in the target language. She speculated that the wording of questions in the examinees’ first language may provide some clues to the general meaning of the passage and therefore help students to guess the correct answer in multiple-choice items. All DLPT5 questions are in English, and the test is designed for native speakers of English. Native speakers of other languages may be disadvantaged by having the questions in English; writers do, however, attempt to keep the English of the questions (and answer choices, for multiple-choice tests) as simple as possible, to help mitigate the effects of differences in English reading proficiency. Less frequently used vocabulary may be unfamiliar to examinees and could pose difficulty for examinees in processing the questions and/or associated answer choices. The DLPT5 test developers attempt to reduce lexical processing load by choosing high-frequency words and simple syntactic structures where possible.

The purpose of the DLPT5 is to make inferences about examinee target-language ability, as reflected in the ILR Skill Level Descriptions, based on samples of language use represented in a range of test passages and questions. The first step in developing the test is thus to define language ability and the characteristics of receptive language use. The next step is to operationalize the assessment of language ability by producing test passages that represent relevant characteristics of the target language and developing questions that target the specific language abilities of interest. The linkage between the test and the ILR is maintained through a series of ILR-based content reviews and analyses of the outcomes of test administrations.

It should be noted that there are some purposes for which the DLPT5 is not an appropriate test.

The DLPT5 is not intended to measure target language proficiency in speaking or writing, nor is it intended to reflect examinees’ job-related performance or ability to perform specific language-related tasks under special circumstances—e.g., reading or listening to a target language passage with an indefinite number of repetitions and the aid of supplemental reference materials and information sources. The DLPT5 is also not intended to be used as a tool to diagnose strengths, weaknesses, or progress in learning a target language.

The ILR Skill Level Descriptions form a functional scale that provides a standard that defines what examinees at different levels of language proficiency can or cannot do with the target language for communication in the target-language environment. They consist of general characterizations of language users at each of five levels of proficiency, the highest level being that of the highly educated native speaker . While these descriptions serve as the standard against which examinee performance is to be measured, they are not adequate in and of themselves to determine test content fully. In developing test material, DLPT5 developers must therefore rely on several supplemental strategies.

This section has four subsections. Section 3.1 explains the use of the Interagency Language

Roundtable Skill Level Descriptions in the definitions of required examinee ability and some strategies that are employed to supplement these descriptions in the development of test material.

Sections 3.2 and 3.3 outline our definitions of the test constructs of reading comprehension and listening comprehension, respectively, and discuss the effects of examinee ability, passage, and questions on test performance. Section 3.4 provides the rationale for measuring examinee performance on the basis of the interplay of examinee, test input and the tasks in the DLPT5.

1 The concept of “nativeness” vs. “non-nativeness” and/or “heritage,” and the terms “native,” “non-native,” and

“heritage” as applied to speakers/listeners/writers/readers of a given language, are controversial in the fields of second-language acquisition and language assessment. For our purposes, we follow the definition of “native speaker” as conceived by the originators of the ILR: a person who has grown up using the language and who has been educated in the language.

3.1 The Interagency Language Roundtable Skill Level Descriptions

The ILR Skill Level Descriptions provide the basis for DLPT5 test development and test score interpretation. This section explains how features of the ILR Skill Level Descriptions are used in

DLPT5 test development and how the descriptions are interpreted to define the scope of personal ability measured by the DLPT5. This section also explains how the descriptions of texts found in the ILR, supplemented by guidelines based on text typology, are used for test passage selection. In addition, explanation is provided for how supplemental strategies that make use of inferences from the functional nature of the ILR scale are used in the selection of test passages and for the development of tasks based on those passages.

The ILR Skill Level Descriptions consist of a set of descriptions that characterize performance typical of each level of proficiency from 0 (no proficiency) through 5 (that of a highly educated native speaker), with “plus” levels from 0+ through 4+. These descriptions provide target-language user profiles in the form of “can-do” and “can’t-do” statements. The ILR also provides some examples of the types of texts second-language users are able to understand at the respective ILR levels in the skill modalities of interest to DLPT5. In the context of assessing reading and listening comprehension abilities, these “can-do” and “can’t-do” statements form a set of explicit and implicit standards used to evaluate the communicative skills second-language users have acquired when they deal with texts produced by native speakers of the target language in real-life communicative situations.

The ILR Skill Level Descriptions deal with general rather than job-related language skills, and they stress proficiency, not achievement (Lowe, 1986). The use of the ILR Skill Level

Descriptions as the standard for DLPT5 development thus implies a focus on whether examinees are able to understand a given target language for given communicative purposes in communicative situations. Secondly, the ILR emphasizes consistent and sustained language ability (Lowe, 1986). The ILR Skill Level Descriptions provide generalized user guidelines for what a typical second-or foreign-language user can or cannot do at any given ILR level. For example, at Reading Level 2, the reader is characterized as being “able to read with some misunderstandings straightforward, familiar, factual material, but in general insufficiently experienced with the language to draw inferences directly from the linguistic aspects of the text” and “typically able to answer factual questions about authentic texts….” The reader “can locate and understand the main ideas and details in material written for the general reader.” In general, “the prose that can be read by the individual is predominantly in straightforward/high-frequency sentence patterns.” The “can-do” statements suggest what a typical Level 2 reader is able to accomplish and provide explicit points of reference for defining the areas of ability most essential at this level. On the other hand, the “can’t-do” statements, such as “inability to discern nuances and/or intentionally disguised meaning” (Reading Level 2+) and “can get the gist of more sophisticated texts, but may be unable to detect or understand subtlety and nuance”

(Reading Level 3), function as implicit references to the limit of what a typical reader is able to

do. In addition, the ILR also acknowledges that language users have the ability to process language minimally at a higher level, e.g., at Reading Level 2+, stating that “the individual is able to get the gist of main and subsidiary ideas in texts which could only be read thoroughly by persons with much higher proficiencies.” In the context of testing, the DLPT5 measures only what examinees can do, and the assignment of ILR levels is based on the success rate of correctly answering questions relating to the “can-do” statements. The “can’t-do” statements provide explicit limitations on passage selection and task development at given levels. For example, Level 2 listeners are “able to understand facts; i.e., the lines but not between or beyond the lines.” Therefore, tasks requiring listeners to make speaker-intended inferences are not included for Level 2 test passages.

When measuring the consistent and sustained aspect of language use ability, DLPT5 test developers focus on the quantity and quality of the examinees’ language performance through tasks designed to elicit the subskills considered important at a given ILR level. For example, at

Reading Level 2, a typical reader can “locate and understand the main ideas and details in material written for the general reader,” and is "typically able to answer factual questions about authentic texts.” Questions at Level 2 thus focus on main ideas and supporting information in authentic texts that are straightforward in conveying factual information. The quantitative aspect of sustained performance is represented by the formulas used to assign scores on the DLPT5 (see section 3.4). The ILR also contains statements related to quality, or the accuracy with which target-language users are able to employ their ability. Level 2 readers read, for example, “with some misunderstandings.” This qualitative aspect of performance is represented by the specific ideas required in selecting or producing the correct answer to a test question. Note that individuals with the same level of proficiency vary considerably in terms of both the qualitative and quantitative factors. It should be stressed that the DLPT5 measures general proficiency; two individuals who receive the same score on the test will often have different strengths and weaknesses.

The ILR Skill Level Descriptions anecdotally provide some descriptions of texts second-language users are able to understand. For example, at Reading Level 3, individuals are “able to read … with almost complete comprehension a variety of authentic prose material on unfamiliar subjects,” and the text types include “news stories similar to wire service reports or international news items in major periodicals, routine correspondence, general reports, and technical material in his/her professional field….” The latter statement describes the types and nature of texts a typical reader at Level 3 is able to handle. However, a systematic treatment of both the nature and complexity of texts and their content is missing, which poses a problem for DLPT5 test developers when selecting reading and listening test passages. Child’s text typology, discussed below, provides a more detailed treatment of texts in spoken and written forms at all ILR levels, and is therefore used by DLPT5 test developers for supplementary guidelines in text selection.

In short, the skill level statements in the ILR provide descriptions of what language users can or cannot do over a wide spectrum of communicative situations, whereas the textual modes delineate the texts produced by native speakers for various communicative purposes.

Child’s text typology (Child, 1987, 1998 & 1999) is a scale in the textual dimension for both written and spoken texts. DLPT5 test developers use Child’s analyses and the text descriptions found in the ILR Skill Level Descriptions as standards for selecting test passages. The typology of texts posits four text modes or purposes to express the ways in which language is used by native speakers for communicative purposes. The four modes in the text typology, presented as a “scheme of increasing complexity” are the Orientation Mode, Instructive Mode, Evaluative

Mode, and Projective Mode, corresponding roughly to the ILR Skill Level Descriptions Level 1, Level 2, Level 3 and Level 4. Thoughts on textual modes from other ILR scholars such as Lowe

(1988, 1998), Clifford (1980), and Edwards on written texts (1996) are included in the current collection of textual analyses used by the DLPT5 test developers.

The most prominent characteristics of these textual modes, taken from Child’s papers (1987, 1998, 1999), are provided below. Refer to Tables 3-2 and 3-4 for more details of textual analyses, descriptions of the texts considered appropriate at targeted ILR levels, and their sources. It should be noted that native speakers do not have textual mode in mind when they speak or write, and, therefore, texts do not usually fall under one single mode. A textual mode is determined by how dominant discourse and linguistic elements in a given mode are in a text.

• Orientation Mode

The purpose of texts covered by this mode is to make information in the form of simple spoken or written texts immediately available to those needing it. Texts in the Orientation Mode are usually concerned with orienting readers or listeners regarding who or what is where, or what is happening or supposed to happen, all within a generally predetermined pattern. Such texts are easily accessible to users who have acquired the basic second-language functions and elements, because such texts are usually short and grammatically simple, yet they are “part of the real world” and convey communicative purposes as considered by native speakers of the target language. Some examples of texts in the Orientation Mode are indications of identity, arrivals and departures of mass-transit carriers, times and places of meetings, and, in listening contexts, greetings or other formulaic exchanges in daily conversation.

• Instructive Mode

The purpose of texts in the Instructive Mode, like the Orientation Mode, is to transmit factual information. However, the variety and scope of texts are much more expanded. Generally speaking, texts in this mode convey written or spoken information about something that “exists or is developing or should take place in the real world” but do not offer “analytical or intuitive judgments” concerning the information conveyed. Writers and speakers who produce such texts assume that the reader or nonparticipatory listener shares a sufficient amount of background information. In other words, topics in the Instructive Mode are either intrinsically familiar to users or they can be dealt with from the context of the content. Although texts in the Instructive

Mode deal with factuality and follow a predictable discourse pattern, speakers or writers may use a variety of expressions to achieve the same communicative purpose. Second-language users must acquire the basic rules of the target language to comprehend texts in the Instructive Mode.

Some examples of texts in the Instructive Mode include accounts of domestic and international events, detailed instructions on the assembly of a complicated piece of equipment, straightforward historical narrative, and a topographical description of a geographical area.

• Evaluative Mode

In texts in the Evaluative Mode, the emphasis on the transmission of factual information is shifted to “a perspective in which facts are selected and pressed into service in order to develop points of view; explain or apologize for personal conduct; state and defend past or projected policies.” The speakers or writers of Evaluative texts intend the products to fulfill a social purpose. Texts in the Evaluative Mode contain analysis and evaluation of things and events about which both the speaker or writer and the intended audience share background information.

Texts in the Evaluative Mode correspond roughly to ILR Level 3, a level at which second-language users are expected to respond intellectually, intuitively, or instinctively to texts. In other words, users are expected to think in the target language when they process texts in the Evaluative Mode. Users must have good command of the target language and have developed a good deal of cultural knowledge of the target language. Some examples of Evaluative texts are newspaper editorials, radio commentaries, biography with some critical interpretation, personal correspondences providing justifications for past actions, and assessment of given policies.

• Projective Mode

Texts in the Projective Mode are the most difficult type to access due to their relative lack of shared information and assumptions on the part of the speaker or writer. The function of these texts is to take a novel or creative approach in order to rethink and verbalize solutions to problems either previously treated in a different way or not addressed at all. The speakers or writers are breaking new ground, and thus the products are unique in their conceptualization and notable for their individuation. Such texts are relatively inaccessible because they reflect unfamiliar cultural values, highly idiosyncratic language use, or a combination of the two.

Second-language users must be well acculturated into the target language society. They also must be sensitive to the ways in which the language used in Projective texts follows or diverges from the assumed linguistic or cultural norms. Some examples of texts in the Projective Mode are “think pieces on the op-ed page of a newspaper or in a journal on the need to reformulate social, economic, or political policy on X, Y, or Z,” and “exchanges between thoughtful people as they address, or reconsider personal problems or goals; discuss merits of a work of art or a performance from personal perspectives; or merely give voice to private feelings.”

• Mixed mode

Mixed-mode texts refer to texts that display textual elements found in two modes. They are roughly equivalent to “plus”-level texts. For example, if a text exhibits a blend of language elements from the Orientation Mode and the Instructive Mode, this text will be treated as a

Mixed-Mode (Orientation/Instructive) text. Such texts are roughly equivalent to 1+ texts.

Standards for determining 1+ texts, according to Child (1999), should be the standards set for the Instructive Mode. By the same token, a Mixed-Mode Instructive/Evaluative text, i.e., a 2+ text, exhibits almost all characteristics found in the Evaluative Mode. It may fall short of the amount or extent of arguments, hypotheses, or evaluation generally found in a typical Evaluative text. In short, mixed-mode texts fall short of the characteristics required for the next mode on the typology hierarchy. Refer to Tables 3-2 and 3-4 for detailed descriptions of the characteristics associated with written and spoken texts at each proficiency level.

Standards in text typology apply to all languages in terms of text content (e.g., advertisement, public announcement, news story, editorial, etc.), function (e.g., to sell something, to attract people to an event, to inform about a recent happening, to offer an opinion), and linguistic complexity (e.g., range of lexicon, syntactic structures used, organizational characteristics of the text). Because the DLPT5 measures proficiency in a wide variety of target languages, selecting representative samples from the target language use domain is the first step toward ensuring test validity in the languages tested. Therefore, DLPT5 test developers map test passages against the text typology standards in terms of the content, function, and linguistic complexity factors.

(expanded upon in Tables 3.2 and 3.4). However, each target language tested has its own unique language features that must be accommodated. Examinees need to understand the language and cultural elements in the target language they have acquired in order to process test passages associated with the different textual modes.

The ILR Skill Level Descriptions and Child’s text typology provide two different perspectives on the ILR level of texts, and test developers make use of these perspectives when selecting texts to be used in the DLPT5. Child’s text typology is a text-based perspective that directly addresses the level of a text based on its features. The ILR Skill Level Descriptions, on the other hand, entail a language-user-based perspective that mentions selected text types at various levels as examples of the types of texts language users at that level can work with. The ILR skill descriptors were developed from observations of second-language user behavior. The textual descriptions, on the other hand, were derived from “the perspectives of the native speakers of the target languages” (Child, 1999). Our approach to testing examinee proficiency suggests that, first, DLPT5 test developers need to select authentic passages used for real communicative purposes according to the textual standards prescribed, and, secondly, they need to develop tasks that tap the abilities relevant to the level of the text in order to measure the examinee’s language use ability described in the ILR Skill Level Descriptions. The use of these two scales has the further implication that examinees considered at a given level of proficiency are supposed to be able to demonstrate the subskills required at all lower levels of proficiency. For example, a reader who is at Level 2 in a target language is supposed to be able to meet the criteria for levels

1 and 1+ as well.

As noted above, however, even the ILR Skill Level Descriptions and Child’s text typology together are inadequate to determine fully the test passages selected, the subskills to be tested, and the tasks developed, and we therefore employ two supplementary strategies whose primary purpose is to describe the kinds of tasks language users can perform and the kinds of written or spoken texts they can comprehend or produce. First, we can determine what constitutes major or important information in a text at a given level based on what the practical effect would be of failing to understand that information. For example, in a Level 1 spoken text about a change in tracks for a train departure, any information that is necessary in order to ensure a traveler would board the right train would be considered important. Second, adhering to a practice followed by

ILR experts, we infer that language-use tasks at a given level have functional equivalents across skill modalities; hence, descriptions of ability, accuracy, and text purpose applicable to one skill can be applied to another skill (see Lowe, 2006). For example, the Skill Level Descriptions for

Level 2 in listening do not provide information about nonparticipatory listening text types. The

Level 2 reading Skill Level Descriptions, however, refer to “descriptions and narrations in contexts such as news items describing frequently occurring events, simple biographical information, and simple technical material written for the general reader.” We apply this description to functionally equivalent tasks in listening and infer that Level 2 listening ability involves ability to comprehend descriptions and narrations in news broadcasts describing frequently occurring events, simple biographical information, and simple technical material for the general public.

The ILR Skill Level Descriptions also sometimes refer to characteristics of second-language users that we cannot practically test on the DLPT5. For example, the Skill Level Descriptions at times describe second-language users’ ability to process texts related to their areas of specialty, i.e., background knowledge—for example, in Reading Level 2, the statement that “persons who have professional knowledge of a subject may be able to summarize or perform sorting and locating tasks with written texts that are well beyond their general proficiency level.” In selecting test passages, it is difficult to avoid all subject matter that could be an area of specialized knowledge for an individual examinee. We do not think background knowledge should play a greater role in examinees’ test performance than their language knowledge, but we acknowledge that background knowledge influences comprehension. It is our practice to control for examinees’ background knowledge as much as possible by including a wide variety of text types and content representing target-language use situations for the general public, in order not to favor particular groups of examinees.

The ILR is a description of a linear progression of proficiency that helps test users interpret test scores. However, it should be noted that language development is multidimensional, and there is no clear indication of how subskills suggested in the ILR levels are ordered. Unless empirically verified, the descriptors are observations of second-language users’ language-use ability in a wide spectrum of communicative situations. The skill descriptors are indicative of what a given examinee most likely is able to accomplish and where this examinee stands in the proficiency continuum.

3.2 Reading

In this section, the operational definition of the construct of reading comprehension ability will be provided. This section includes three subsections: The first offers the DLPT5 definition of reading comprehension. The second discusses factors affecting examinee performance, to include examinee characteristics and text and question characteristics. Finally, the interplay of these three types of factors with ILR level will be described and presented.

3.2.1 What is reading?

Reading comprehension can be viewed from two perspectives: process and product (Alderson

2000). The process perspective looks at the interaction between the reader and the text, whereas the product perspective focuses on comprehension, the product of reading. During the process of reading, readers look at the text, decipher the symbols contained in the text, and decide what those symbols mean as well as how they are related to one another. Readers also think about all or some of the following: what they are reading, what the text means to them, how they can relate to other things they read or already know, and what they expect will come next. Readers may also be thinking about whether the text is useful, interesting or boring. Furthermore, they may be reflecting unconsciously about the difficulty or ease of the text. In short, components of language use are activated during reading, and the entire process is dynamic, and varies among readers of the same text at different times of reading and with different purposes. The purpose of reading also affects text processing and comprehension. The reader’s purpose may be as varied as reading to identify specific information, reading for general comprehension, reading for instructional purposes, and/or reading to integrate information.

A common view of the reading process suggests that reading consists of two components:

decoding and comprehension (Gough et al., 1992). Decoding consists of word recognition fluency and accuracy of word representation; the comprehension process refers to parsing sentences, understanding sentences in discourse, building a discourse structure, and integrating understanding of the text with what one already knows. The comprehension process describes both linguistic skills and cognitive and metacognitive skills. This view of reading argues that fluency and accuracy of word recognition, reading rate, and processing efficiency are fundamental to reading ability because the comprehension process of parsing sentences and building a discourse structure requires a certain level of rapid and automatic processing.