ID06190003_SF33_Attach_5_Tech_Exh_11-21_Test_Items_IDIQ.pdf

DLPT5

General Test Specifications Revision: February 9, 2006

Defense Language Proficiency Test

DLPT5

Upper Range

Multiple-Choice Test Specifications

The full specification document for the DLPT5 consists of a number of parts:

1. A General Description of the test, which is for public dissemination (pp. 2-6)

2. Test Development Specifications, which are kept private and internal to EST. This includes:

a. Tables of Specifications for the Operational Linear Test (p. 8)

b. A table of content areas to be covered in the test (p. 9)

c. The Composition of the Sub-Pools for Each ILR Level (pp. 10-19)

3. General Guidelines on Text Selection (pp. 20-21)

4. Specifications for Test Administration and Scoring

a. Administration of the Operational Forms of the DLPT5 (pp. 22-23)

b. Administration of the Validation Forms of the DLPT5 (pp. 24-25)

c. Standard instructions for test booklets/computer sessions (pp. 26-29)

d. Scoring and Calibration of the DLPT5 (p. 30)

Technical Exhibit 11

ID06190003

Attachment 05

Revision: February 9, 2006

DLPT5-MC

General Description

(For Public Dissemination)

Purpose of the DLPT The purpose of the DLPT5 is to assess the language proficiency of military and civilian government employees, by testing comprehension of a wide variety of realistic written and spoken materials over a wide range of topics. Each DLPT5 is designed to measure proficiency in one target language, regardless of how it has been acquired. Scores are used to determine eligibility for certain jobs and for pay bonuses. The test is designed for native speakers of American English, although it may be used for native speakers of other languages who have near-native English proficiency. Typically, examinees taking the upper-range DLPT5 are those who have taken the lower-range test and achieved a rating of level 3.

Administration of the DLPT5 The DLPT5 has two separate sections, Reading Comprehension and Listening Comprehension, which are administered as two separate tests. Each section is allotted 3 hours for administration.

Each section is constructed from a separate pool of available questions that have been calibrated on an Item Response Theory (IRT) scale, and different forms of the test are constructed from the pools.

Linear tests: At first, tests will be administered in linear form by computer, and will assess proficiency from Interagency Language Roundtable (ILR) levels 3 to 4. Some passages and questions may also appear on the lower-range test.

Computer-adaptive multi-stage tests: Eventually, some languages will have tests administered by computer as a series of smaller tests. The test as a whole will be capable of assessing ILR levels 0 through 4, but examinees will take smaller tests spanning only part of the range, depending on their estimated proficiency level.

Test Format

Reading Comprehension Section The test consists of a series of approximately 10 written passages with questions based on each passage. Passages vary in length and have up to 5 associated questions. The total number of scored questions in this section is approximately 30. In addition, the section contains approximately 2 passages with approximately 5 associated questions that are not scored. These passages and questions are being calibrated for possible use on later operational tests.

The tasks given to the examinee are all multiple-choice questions, each of which consists of a question or incomplete statement followed by 4 answer choices. All questions and answer choices are given in English. A brief orientation in English precedes each passage and sets the stage, informing the test-taker from what kind of source a reading text was selected. Examinees are able to navigate freely between passages, both forward and backward.

Listening Comprehension Section The test consists of a series of approximately 18 spoken passages with questions based on each passage. Passages vary in length and have up to 3 associated questions. The total number of

Technical Exhibit 11

ID06190003

scored questions in this section is approximately 30. In addition, the section contains approximately 3 passages with approximately 5 associated questions that are not scored. These passages and questions are being calibrated for possible use on later operational tests.

The tasks given to the examinee are all multiple-choice questions, each of which consists of a question or incomplete statement followed by 4 answer choices. All questions and answer choices are given in English. A brief orientation in English precedes each passage and sets the stage, informing the test-taker under what circumstances a listening text would have been heard.

In the Listening Comprehension section, examinees are able to control when the audio for the next passage begins, but have to listen to the entire audio for each passage and to all of the passages in order.

Additional Screens In addition to the test content, the examinee sees a number of introductory screens including personal-data-gathering screens, disclaimer screens, instructional screens (both navigational and content-related), and a sample passage and questions. Also, following the final question there is a review screen that allows the examinee to answer any unanswered questions or change any previous answers.

Scores Reported A separate score is provided for Reading Comprehension and Listening Comprehension. These are reported on the ILR scale, from levels 3 through 4. Scores are based on the number of questions on the test answered correctly.

Communicative Skills Assessed Although DLPT5 is a test of comprehension, not production, it is designed to test a wide range of communicative language ability, including:

Grammatical knowledge: Test-takers are tested on their knowledge of vocabulary, syntax, morphology, phonology, the writing system, and so forth, by means of tasks requiring understanding of explicitly stated information and of expressions in context.

Discourse knowledge: Test-takers are tested on their ability to understand discourse features, such as cohesive devices, pronominal reference, and rhetorical schemata, as well as the ability to deal with the structure of unplanned spoken discourse.

Sociolinguistic knowledge: Test-takers are tested on their knowledge of the appropriateness of language use; at the higher proficiency levels, they are also tested on their ability to understand cultural references and non-standard features of the language.

The balance of these abilities will vary depending on the targeted ILR level of the text.

Text Selection Texts are selected based on a variety of criteria, principally, ILR proficiency descriptors, topic, and degree of authenticity. Texts are selected from sources intended for the general reader or listener.

ILR Descriptors The ILR descriptors provide general guidelines for the texts appropriate at each level, and these are used as the main basis for text selection. However, some languages have characteristics that are not covered by the ILR descriptors (e.g., honorifics, use of classical language, tones, etc.), and text selection will take these language-specific issues into account where appropriate.

Topical Content

Technical Exhibit 11

ID06190003

Topics for both listening and reading are based on the following areas:

• military and security issues

• economic and political matters

• scientific and technical issues

• cultural and social issues

• physical, political and economic geography.

The balance of topics commonly found in any target-language situation, or culture, varies considerably from one culture to another—especially at the higher levels. (For example, very little public discussion of military issues is allowed in some countries.) Hence, emphasis is placed on those topics that are relevant to the particular target-language culture.

Degree of Authenticity The DLPT5 will emphasize authentic language. This means that to the extent that it is practical, texts have the characteristics of texts found in the target-language situation, and texts will require the language-processing skills typical of target-language use. Some texts may be modified as necessary.

Construct Definition:

Reading Comprehension is defined as the ability to:

process samples of realistic written discourse automatically and reasonably quickly;

understand the linguistic information that is unequivocally included in the text;

make any author-intended inferences that are unambiguously implied by the content of the text.

DLPT5 emphasizes the ability to understand transactional language (i.e., language intended to convey ideas), but may also include sufficient interactional language (i.e., language intended to maintain social relationships) in order to preserve the authenticity of the text.

Reading Texts These will vary based on characteristics stated in the ILR Skill Level Descriptions. Some examples and characteristics are given below. Note that the examples given do not exhaust the possible text types that might be used.

Level 3 Editorials, commentaries, biographies with critical interpretations, criticisms, reports intended for the general reader on complex issues or specialized topics, philosophical argumentation, opinion pieces, political analysis. These texts demonstrate a wide variety of discourse structures and a wide range of vocabulary. They contain complex argumentation, including hypothesis, supported opinion, analysis, and subtle implications. Main ideas are often not stated explicitly and require “reading between the lines.” From one to four questions may accompany a Level 3 text. A Level 3 text can be up to 400 words longs, measured by its English rendering.

Level 4 Think-pieces, essays, philosophical expositions, satiric writings, other sophisticated humor. These texts are linguistically and conceptually extremely dense. They reflect the author’s unique thinking about an abstract topic or complex issue and usually display unexpected means of

Technical Exhibit 11

ID06190003

argumentation or unusual turns of thought. They use sophisticated and complex language with virtuosity, often in figurative and subtle ways. From one to five questions may accompany a Level 4 text.

Proficiency Skills Tested These vary according to the targeted ILR level of individual items, but overall the DLPT5 targets the following reading skills. The ability to:

understand main ideas, or the gist of a text.

understand supporting detail.

understand the cohesive relationships between parts of the text.

understand grammatical relationships within the text.

understand the meaning of vocabulary, or expressions, in context.

make inferences that the author intended the reader to make.

• understand the author’s tone or attitude.

compare and contrast points of view.

synthesize information or ideas.

Listening Comprehension is defined as the ability to:

process samples of realistic spoken language automatically and in real time;

understand the linguistic information that is unequivocally included in the text;

make any speaker-intended inferences that are unambiguously implied by the content of the text or by intonation, accent, or other oral characteristics.

Listening Texts Spoken texts are either taken from prerecorded sources or created in-house. Texts are genuine or realistic samples of the language as it is spoken in the target-language environment.

The spoken texts used vary with regard to two factors that affect difficulty:

Content, which ranges from less challenging to more challenging, based on the characteristics of the ILR Scale and the characteristics of the target language.

Oral characteristics, (e.g., phonological modification, hesitation phenomena, speech rate, level of formality, colloquialisms, number of speakers, etc.), with informal texts having more oral characteristics, and more formal texts having fewer of them.

Texts with more oral characteristics tend to be more difficult to process than those with fewer.

The degree to which a text has oral characteristics will be based on the characteristics of the target-language-use situation.

Texts are good-quality audio, although some texts may have background sounds added to provide authenticity, and other texts that have been recorded from authentic media sources will have the sound quality of the original source. Texts can be up to about 2 minutes long, although most are considerably shorter.

Examinees will hear each text twice.

ID06190003

Some examples of possible text types and their general characteristics are given below. Note that the examples given do not exhaust the possible text types that might be used.

Level 3 Debates, talk shows, movies, discussion forums, lectures, interviews, speeches, etc. Texts deal with complex concrete or abstract topics and generally contain analysis, opinions, and/or subtle implications. They may be spoken at a normal rate of speech or sometimes at a somewhat faster rate than normal. Texts may have a wide range of phonological modifications and demonstrate a variety of discourse structures. They may contain colloquial or common slang expressions. From one to two questions may accompany a Level 3 text. A Level 3 text may be up to 2 minutes long.

Level 4 Debates, movies, discussion forums, lectures, speeches, intellectual discussion, formal oratory.

These texts may be formal or informal in nature and typically deal with abstract topics that are conceptually dense. They are spoken at normal or fast rates of speech and have an extensive range of “oral characteristics.” They may also contain non-intrusive background noise. The general language use and word choice in Level 4 texts is creative, figurative, and often idiosyncratic in nature, demonstrating the speaker’s virtuosity in expressing him-/herself. From one to three questions may accompany a Level 4 text.

These will vary according to the targeted ILR level of individual items, but overall the DLPT5 targets the following listening skills. The ability to:

understand explicit main ideas.

understand major details.

understand the sequence of events.

understand common indirect speech acts.

understand speaker-intended implications.

understand implications conveyed by suprasegmentals (e.g., intonation, stress, etc.).

separate fact from opinions.

understand register.

understand colloquial and idiomatic expressions.

• understand the speaker’s tone or attitude.

Technical Exhibit 11

ID06190003

Defense Language Proficiency Test

Multiple-Choice Test Development Specifications

Private and Confidential For the Internal Use of the Test Development and Standards Division, DLIFLC

Not for External Distribution

Technical Exhibit 11

ID06190003

Table of Specifications for Operational Linear Forms

For many languages, there will also be a DLPT5 available to assess examinees whose proficiency is above ILR level 3. That test, depending on the target language, may also be a multiple-choice test or it may be a constructed-response test. For some languages, the range of the test as a whole will be limited to the lower range (0 through 3).

Notes: The numbers shown in the tables below are for one operational form of the test.

Typically, two forms are developed, and more items per form than are shown below may be created, in the expectation that items may be lost based on the outcomes of piloting and/or validation. See separate validation specifications on page 34.

The number of passages at any given ILR level will vary depending on the number of test items per passage that successfully meet piloting and statistical standards.

The length of passages for Reading is calculated in terms of the number of words in an English translation of the passage.

Level Approx. % of Total Test Range of Items (scored/unscored)

Approx.

Length of Passages (in words)

Range of Items per Passage

Range of Passages (scored/unscored)

3 10% 3-4 0-3 Up to 400 Up to 4 1-2 0-1 3+ 45% 14-16 0-3 Up to 400 Up to 5 3-5 0-1 4 45% 14-16 0-3 Up to 500 Up to 5 3-5 0-1

Total 100% 30-32 4-6 8-12 1-2 Grand Total 100% 34-36 10-14

Level Approx % of Total Test

Range of Items (scored/unscored)

Max. length of a Passage* (in seconds):

(monologs/dialogs)

Range of Items per Passage

Range of Passages (scored/unscored)

# of Voicings per Passage

3 10% 3-4 0-3 100 120 Up to 2 1-2 0-1 2 3+ 45% 14-16 0-3 120 140 Up to 3 6-9 0-2 2 4 45% 14-16 0-3 140 160 Up to 3 6-9 0-2 2

Total 100% 30-32 4-6 15-21 2-4

Grand Total 100% 33-36 18-24

* In rare cases, a passage exceeding the maximum length may be divided into 2 segments, each administered independently. Such dividing of a listening passage into segments is to be considered an exceptional procedure, not standard practice.

Technical Exhibit 11

ID06190003

Content Areas The DLPT generally covers all the Final Learning Objectives (FLO) content areas. The types of topics within the content areas will, of course, be different for different levels, as will the proportion of each content area represented at each level. Passages should represent both a wide range of topics within each content area and particular situations that are not likely to be known to the examinee. The distribution of content areas at each level should be as balanced as possible, although language- or culture-specific constraints may make an even distribution difficult or impossible to achieve.

The FLO Content Areas

• Military-Security

• Economic-Political

• Scientific and Technological

• Cultural and Social

• Geography: Physical, Political, Economic

Technical Exhibit 11

ID06190003

The Composition of the Sub-pools for Each ILR Level

General Comments A separate sub-pool of passages with their associated items is created for each of the base and plus ILR levels represented in the test. Each sub-pool is balanced according to a number of criteria—text characteristics, sources, topics, skills, and questions to be asked. The content of each sub-pool is intended to provide a representative sample of the universe of all possible items at that level. Explanations and discussion of the specifications for the sub-pools are provided for each level on the following pages. The balance of items testing global tasks and those testing non-global tasks is shown in the table below. Global tasks are those for which the question focuses on the passage as a whole, for example, asking about the author’s general attitude, the main idea of the passage, or the principle underlying the author’s position. Non-global tasks are those for which the question focuses on part of the passage (even if understanding the whole passage is necessary for answering correctly). Examples might be an item asking about a detail of a plane crash, the function of a particular metaphor in the author’s argumentation, or the author’s tone in a particular paragraph or when addressing a particular sub-issue.

Reading

Item type Percent of items in pool for each level 0+ 1 1+ 2 2+ 3 3+ 4

Global 0% 50-75% 50-75% 40-60% 30-50% 30-50% 20-30% 20-30% Non-Global 100% 25-50% 25-50% 40-60% 50-70% 50-70% 70-80% 70-80%

Listening Item type Percent of items in pool for each level

0+ 1 1+ 2 2+ 3 3+ 4 Global 0% 40-70% 40-70% 30-60% 30-60% 30-60% 30-50% 30-50% Non-Global 100% 30-60% 30-60% 40-70% 40-70% 40-70% 50-70% 50-70%

Technical Exhibit 11

ID06190003

Reading: Level 3

Part I: The texts Length Texts should not typically be longer than the equivalent of 400-word English passages. It is desirable to have a substantial proportion of the texts considerably shorter than this.

Linguistic Characteristics

• Texts should show a wide range of grammatical structures and vocabulary.

• Texts should demonstrate a variety of discourse structures.

• Texts should deal with abstract topics. They should generally contain hypothesis, argumentation, analysis, supported opinions and/or subtle implications.

• Texts should usually reflect the target-language culture, and cultural references/cues play an increasingly large role as the difficulty of these texts rises.

• The organization of the texts should:

a. be complex and personalized (‘shaped’ by the author)

b. show clear cohesive discourse with adequate cohesive devices characteristic of the target language.

Text Types Texts may be of the types shown below, but the ultimate selection of a text depends on its meeting the ILR Skill Level criteria.

1. Evaluative texts (whose function is to analyze or assess events).

Examples: Editorials, commentaries, literature or art reviews.

2. Expository texts (whose function is to explain or discuss issues).

Examples: Letters to the editor explaining a problem, articles from specialized magazines, essays, textbooks. There are two subcategories of texts:

a. Specialized texts that discuss concrete but technical issues (scientific, technological or legal texts)

b. Abstract texts that discuss ideas, concepts, or notions.

3. Argumentative texts (whose function is to defend or refute a policy, a strategy, or a program).

Examples: Platforms of political parties, proposals put forth by politicians or officials, letters to the editor to argue in favor of or against an issue, editorials, etc.

Sources Leading newspapers, serious magazines, position papers, reports, the Internet, books, essays, textbooks.

Examples Editorials, commentaries, biographies with critical interpretations, criticisms, reports, philosophical argumentation, opinion pieces, political analysis.

Technical Exhibit 11

ID06190003

Examples of possible topics within the FLO Content Areas

FLO Content Area Example Sub-categories Military-Security Analysis of military events and intelligence activities Assessment of military cooperation or alliance among countries Rationales for arms sales or military aid Description of advanced weapons systems Analysis of historical battles Analysis of security issues: justifications for security measures, etc.

Economic-Political Economic analysis

Political analysis Economic policies Assessment of a country’s economic situation Development and growth issues Political strategies Political reforms and democracy Analysis of international political and trade relations Scientific and Technological Explanation of scientific, medical, and technological issues Analysis of scientific achievements, discoveries, or findings (e.g., the ethics of cloning) Research policies of governments or institutions Impact of scientific progress on families, behavior, society, etc.

Scientific theories

Cultural and Social Evaluation or analysis of social events and relationships

Discussion of societal issues (discrimination, violence, drugs, privacy, etc.)

Private vs. public education, role of government in education, standards, etc.

Religious issues: church/state relationship, fundamentalism, persecution, value systems

Freedom of the press Commentary on books, movies, artworks

Geography: Physical, Political, Economic

Analysis of environmental policies Opinions on water issues (dam-building, etc.)

Analysis of border disputes Use of natural resources Plans for maintaining habitat (e.g., rainforest preservation , anti-desertification plans)

Technical Exhibit 11

ID06190003

Part II: Skills assessed On the above texts, level 3 readers can:

Global:

o understand major ideas which may be explicitly stated or be implicit in the text.

o draw appropriate conclusions from an author’s remarks.

Non-global:

o understand major points supporting a line of argumentation.

o understand author-intended implications.

o understand vocabulary in context, including idiomatic expressions and figurative speech.

o understand grammatical and other references that connect parts of the text and that control meaning.

Can be either global or non-global:

o separate fact from opinion.

o identify the attitude of the author.

The predominance of items should test comprehension of major ideas, conclusions, and author-intended implications; it is desirable to have at least a few items in the test that address supporting a line of argumentation. A text must have either an item testing comprehension of a major idea, an item testing a conclusion, or an item testing an author-intended implication in order to be included in an operational test.

Technical Exhibit 11

ID06190003

Reading: Level 3+ desirable to have a substantial proportion of the texts considerably shorter than this.

Level 3+ text and item characteristics are similar to those for level 3, but items assessing ability to recognize basic rhetorical structure are permissible at this level.

Technical Exhibit 11

ID06190003

Reading: Level 4

Texts should not typically be longer than the equivalent of 500-word English passages. It is desirable to have a substantial proportion of the texts considerably shorter than this.

• Texts are linguistically and conceptually extremely dense.

• Texts may display unusual/unexpected means of argumentation or turns of thought.

• Texts are highly individualized, i.e., reflect the author’s unique thinking about or synthesis of a topic.

• Texts are extremely sophisticated and use complex, subtle language with virtuosity, often in figurative and subtle ways.

Intellectual periodicals, books, the Internet, collections of essays, etc.

Examples Think-pieces, essays, philosophical expositions, satiric writings, other sophisticated humor.

Possible topics within the FLO Content Areas Essentially any topic is appropriate, but many topics are not usually discussed at level 4. Many of the example topics at level 3 would also be appropriate for level 4; see the level 3 table for details.

On the above texts, level 4 readers can:

Global or non-global:

o understand lines of argumentation by synthesizing information and ideas.

o understand major points supporting the author’s argumentation.

o understand the rhetorical structure (i.e., the relationship between the parts of the text).

o understand subtle author-intended implications.

o follow unpredictable turns of thought/understand subtle argumentation.

o read beyond the lines (i.e., understand the full ramifications of a text as it is situated in a wider context).

o understand the author's attitude or tone.

o understand vocabulary and expressions in context.

o understand grammatical and other references.

The items should assess comprehension of the features indicated above. A text must have either an item testing comprehension of a line of argumentation, an item testing comprehension of a conclusion, an item testing the synthesis of information central to the author’s argument, or an item testing an author-intended implication in order to be included in an operational test.

Technical Exhibit 11

ID06190003

Listening: Level 3

Texts should not typically be longer than 100 seconds for monologues, 120 seconds for dialogues.

It is desirable to have a substantial proportion of the texts considerably shorter than this.

• Texts are either dialogues or monologues; most will be delivered at a normal speech rate, and some may be at a faster rate than normal. Texts may have authentic background noise.

• Texts deal with abstract topics or concrete topics that are complex, and they generally contain hypotheses, analyses, opinions, and/or subtle implications.

• Texts may have a wide range of phonological modifications and demonstrate a variety of grammatical and discourse structures. Texts may have colloquial and common slang expressions.

• Authentic audio media sources (e.g. TV/radio programs, movies, public announcements, the Internet).

• Recorded in-house: debates, lectures, interviews, speeches.

activities Assessment of military cooperation or alliance among countries Rationales for arms sales or military aid Description of advanced weapons systems Analysis of historical battles Analysis of security issues: justifications for security measures, etc.

Economic-Political Economic analysis

Political analysis Economic policies Assessment of a country’s economic situation Development and growth issues Political strategies Political reforms and democracy Analysis of international political and trade relations

Technical Exhibit 11

ID06190003

Scientific and Technological Explanation of scientific, medical, and technological issues

Analysis of scientific achievements, discoveries, or findings (e.g., the ethics of cloning)

Research policies of governments or institutions Impact of scientific progress on families, behavior, society, etc.

Scientific theories

Cultural and Social Evaluation or analysis of social events and relationships

Discussion of societal issues (discrimination, violence, drugs, privacy, etc.)

Private vs. public education, role of government in education, standards, etc.

Religious issues: church/state relationship, fundamentalism, persecution, value systems

Freedom of the press Commentary on books, movies, artworks

Geography: Physical, Political, Economic

Analysis of environmental policies Opinions on water issues (dam-building, etc.)

Analysis of border disputes Use of natural resources Plans for maintaining habitat (e.g., rainforest preservation , anti-desertification plans)

The listening skills necessary to comprehend Level 3 texts are as follows:

Global:

o Understand major ideas that may be explicitly stated or be implicit in the text.

o Draw appropriate conclusions from a speaker’s remarks.

Non-global:

o Understand major points supporting a line of argumentation.

o Understand implications conveyed by supra-segmental features (e.g., intonation, stress, etc.).

Can be global or non-global:

o Understand speaker-intended implications.

o Separate fact from opinion.

o Identify the intent of the speaker.

o Understand register.

Most items should test the understanding of major ideas, conclusions, and speaker-intended implications.

Technical Exhibit 11

ID06190003

Listening: Level 3+ Texts should not typically be longer than 120 seconds for monologues, 140 seconds for dialogues.

It is desirable to have a substantial proportion of the texts considerably shorter than this.

Level 3+ text and item characteristics are similar to those for level 3.

Technical Exhibit 11

ID06190003

Listening: Level 4

Texts should generally not be longer than 140 seconds for monologues, 160 seconds for dialogues.

Linguistic characteristics

• Texts are either dialogues or monologues delivered at normal or fast speech rate. They may have non-intrusive background noise.

• Texts have an extensive range of phonological modification.

• Texts may deal with abstract topics. They typically are linguistically and conceptually dense.

• Texts may consist of formal or informal speech.

• Texts have much low-frequency vocabulary, with creative, figurative, or idiosyncratic use of language. They may include features of non-standard dialects.

• Authentic audio media sources (e.g. TV and radio programs, movies, public addresses, the Internet).

• Recorded in-house: informal discussions, formal speeches, debates, intellectual discussions.

On the above texts, level 4 listeners can:

Global or non-global:

o Understand lines of argumentation by synthesizing information and ideas.

o Understand major points supporting the speaker’s argumentation.

o Draw appropriate conclusions.

o Understand subtle speaker-intended implications.

o Follow unpredictable turns of thought and understand subtle argumentation.

o Go “beyond the lines” of the text, i.e., place the speaker’s message in a wider contextual framework.

o Understand register.

o Understand the speaker’s attitude or tone.

o Understand colloquial and idiomatic use of the language.

o Understand implications represented by supra-segmental features.

Most items should test comprehension of lines of argumentation, conclusions, and speaker-

Technical Exhibit 11

ID06190003

General Guidelines on Text Selection Authenticity of Reading Texts Texts are genuine reading texts, taken from the target-language use situation, and intended for the general reader. If necessary, they may be edited, but ideally editing should be kept to a minimum, or avoided altogether. If the range of authentic texts available is too restricted to provide an appropriate range of test items, texts may be purpose-written, but in such cases they are expected to have all the characteristics of genuine reading texts, and require reading skills typical of authentic texts.

Authenticity of Listening Passages Passages are genuine samples of spoken language, intended for the general listener. They may be recorded directly from the target-language use situation, with or without some editing, or they may be created by the test-development team. As far as possible, created passages should be spontaneous, although semi-scripting can be used if necessary. The reading of scripted passages should be avoided if at all possible. The important thing is that the passages contain the characteristics of realistic spoken language, and require listening skills and language knowledge typically required to listen to such passages in the target-language use situation.

The Characteristics of Spoken Language The selection of listening passages should cover the range of spoken-language characteristics found in the target-language use situation, from very oral texts to very literate texts. All the passages should attempt to replicate the characteristics of real-world speech, and at least half of the passages used should be on the oral side of the continuum.

The most common characteristics of oral language to be included in texts are:

• Considerable phonological modification (assimilation, elision, vowel reduction, etc.).

• Relatively rapid speed, even at lower levels, requiring listeners to process speech in real time, automatically and effectively.

• Use of idea units (roughly the amount a listener can store in working memory at one time) rather than sentences. (Idea units typically are short, about 7 words or 2 seconds long, and are connected more often by conjunction than by embedding.)

• Hesitations: pauses, fillers, repairs, repetitions.

• Slang and linguistic forms not found in writing.

• More colloquial, and less conservative, vocabulary and grammar.

• More personal involvement.

• Approximation and exaggeration.

Level of Tasks In order to target a particular proficiency level, tasks should focus on aspects of a text at that level that are part of the core of the text (rather than its periphery). The English used in the stems and options of items should be as simple and straightforward as possible (rather than necessarily matching the level of the target-language text).

Technical Exhibit 11

ID06190003

The DLPT should provide a representative sample of types of sources from the domain of interest, the target-language use situation. Web pages, for example, are just one text genre out of the many that should be included on the test. Generally, no more than 30% of the test content should come from sources that were written specifically for the Web. Other text genres, such as reports, book reviews, news reports, etc., that were not written specifically for the Web, are accessible on-line and are not included in the 30% figure cited above (provided they are in their original form and have not been modified for the Web). In the case of low-volume languages, where sources are not otherwise available, this figure of 30% may be increased. The same considerations apply to other text genres, such as radio broadcasts, television broadcasts, newspapers, textbooks, etc.

Technical Exhibit 11

ID06190003

Administration of the Operational Forms of the DLPT5

Administration of Linear Operational Forms Because the computer-adaptive multi-stage test will require many more items than will have been originally validated, each DLPT5 will be administered operationally as upper-range or lower-range linear tests until the CAMST pool is ready. The operational linear forms of the DLPT5 will be delivered to examinees by computer. The upper-range reading section and listening section will each consist of 33-36 test items, of which about 5 are unscored. (See tables on p. 8.) The placement of the unscored items is randomized within each level, so that examinees cannot tell which items are scored and which are unscored.

Examinees have 3 hours to complete the Reading test and 3 hours to complete the Listening test.

Approximately 90 minutes into each test, the computer will initiate a 15-minute break. For each test, examinees control when to move to the next passage by clicking a [Next] button on the computer screen. Countdown timers are visible on the screen to help the examinee manage the time; the screen also displays how many passages are on the test and where the examinee is in the test. Following the final question in both the listening and the reading sections, a review screen will appear that allows the examinee to use whatever time remains to answer any questions left unanswered or change answers.

Presentation of the Listening Texts

For all listening passages, examinees see on their computer screen the orientation to the passage and all questions associated with that passage. Each listening passage in the test will be preceded by an orientation statement in English. The purpose of the orientation is to let the examinee know what source the listening passage came from or under what circumstances the examinee would hear such a text. Examinees read the orientation and questions and , when they are ready to hear the passage, they activate the playing of the audio by clicking a [Play Audio] button on the screen. The audio includes a recording of the orientation statement, the passage, and instructions to answer the questions. Passages will be played twice, with a short tone in between the two playings. Examinees cannot replay the audio or stop it before it has finished playing. After the final playing of the passage, examinees will have as much time as they want to answer the questions for that passage before moving on to the next passage. The test as a whole is timed, so examinees have to manage their time spent per passage, as they do on the reading section.

The Computer-Adaptive Multi-Stage Test (CAMST) Ultimately, the DLPT5 will be delivered to examinees as a computer-adaptive multi-stage test that will be able to assess the full range of ILR levels. The test will consist of three non-overlapping panels (independent sub-tests), and examinees will typically take only one panel of the test. Each panel will consist of a number of sub-panels at varying levels, and examinees will be routed to a total of three of those sub-panels (a total of 50 items per examinee) in order to determine their score, which will be reported in terms of ILR level. The routing of an examinee to a particular Stage 2 sub-panel will depend on the performance of that examinee on the Stage 1 sub-panel; similarly, the Stage 3 sub-panel selected will depend on the performance at Stage 2.The structures of the three panels are schematized on the next page.

Panel 1

ID06190003

Panel 2

Panel 3

Stage 3-Low 0+, 1

15 items

Stage 3- High 1, 1+ 15 items

Stage 1 2, 2+, 3 20 items

Stage 2-Low

15 items

Stage 2-Low

15 items

Stage 3-Mid 2+

15 items

Stage 3- High 3

15 items

Stage 3-Low 3+

15 items

Stage 3- High 4

15 items

Stage 2- High 4

15 items

Stage 2-Low 3+

15 items

Stage 1 3+, 4

20 items

Stage 2- High 3

15 items

Stage 2-Mid 2+

15 items

Stage 2- High 1, 1+ 15 items

Stage 2-Low 0+, 1

15 items

Stage 1 0+, 1, 1+ 20 items

Technical Exhibit 11

ID06190003

Administration of the Validation Forms of the DLPT5

The validation forms of a DLPT5 will be administered prior to the first operational administration of that test in order to determine the psychometric qualities of the passages and questions that are being considered for use in the operational test. It is expected that some proportion of the passages and questions will prove unusable; therefore, the selection and number of passages and questions are calculated to yield two operational forms.

The validation forms of the DLPT5 will be delivered to examinees in either paper-and-pencil versions or, in some cases, computer versions. The reading test and the listening test will each cover the same levels as the operational test (3 to 4) and will each consist of approximately 150 test items. All keys will be randomized prior to validation administration.

The DLPT5 reading test in its validation form will be administered in four test sessions. In each of these sessions, the examinees will take one of four reading sub-forms, labeled A, B, C, and D, each consisting of approximately 38 items. Each of the 38-item forms will span the entire range of ILR levels for the test as a whole (3 to 4). Examinees will have approximately two hours to complete each reading sub-form. Each form will use a single FL-101 answer sheet (for paper-and-pencil administrations).

As with the reading test, examinees will take the validation forms of the DLPT5 listening test in four sessions. The test form given in each listening session, like its reading counterpart, will consist of approximately 38 items, spanning the entire ILR range of the test as a whole. For validation, the information concerning 1) the number of voicings of a passage, 2) the presentation of the listening passages, and 3) the orientation to the passage is the same as for the operational linear test administration. The examinee, however, will be given a fixed amount of time (10 seconds) per question for prereading prior to hearing the passage, and will also be given a fixed amount of time per question (15 seconds) for answering.

Technical Exhibit 11

ID06190003

Table of Specifications for the Validation Forms of DLPT5-MC

Reading Comprehension

Listening Comprehension

Level Approx.

Percentage of Total Test

Approx.

# of Items

Max. length of a Passage (seconds):

(monologs/dialogs)

Average # of Items per Passage

Range of Passages

# of Voicings of Passages

3 14% 20 100 120 1 to 2 8-10 2 3+ 43% 65 120 140 1 to 3 26-30 2 4 43% 65 140 160 1 to 3 26-30 2

Total 100% 150 60-70

* NOTE: These numbers represent the expected final ranges of passages. The number of passages at each level should be as close to the midpoint of the range as feasible; that way, the total number of passages at any given level, as a percentage of the total number of passages in the test, will approximate the percent of the test intended to be at that level. At each level, the passages must cover an appropriate range of topics.

Level Approx.

Percentage of Total Test

Approx. # of items / level

Approx.

Length of Passages (in words)

Approx. # of items / passage

Range of passages

3 14% 20 Up to 400 2 to 4 5-7 3+ 43% 65 Up to 400 2 to 5 17-22 4 43% 65 Up to 500 2 to 5 17-22

Total 100% 150 40-50

ID06190003

Standard Instructions for Examinees (operational administration)

• Read the short statement before each passage that tells you where the passage came from.

• Read the passage and the question or questions carefully. Some questions may be in the form of an incomplete statement.

• Based on what you have read in the passage, choose the best answer to each question from among the four answer choices. The answer to each question may be stated overtly in the passage or only implied.

• Click on the button next to the answer you have chosen. You may change your answer by clicking on a different button.

• Click on the [Next] button at the bottom right of the screen to go to the next passage.

• There is no penalty for guessing; therefore, it is to your advantage to answer all of the questions.

A sample passage follows on the next screen.

If you need help during the test, click on the [Help] button.

ID06190003

ATTENTION: You should hear a voice reading the instructions below.

If the audio is not clear or loud enough, raise your hand now to alert the Test Administrator.

• When a passage screen opens, you will see a short statement in English that explains the context from which the passage was taken, and below that, the question or questions associated with the passage.

• When you are ready to hear the passage, click on the [Play Audio] button located at the bottom right corner of the screen.

• You will then hear the short statement in English followed by the passage. The passage will be played twice. There will be a short tone signal between the two playings.

• You do not have to wait until the passage has finished playing to begin answering the questions. After hearing the passage you will be reminded to answer the question or questions. Read each question and its answer choices carefully. Some questions may be in the form of an incomplete statement.

• Based on what you have heard in the passage, choose the best answer to each question from among the four answer choices. The answer to each question may be stated overtly in the passage or only implied.

• Click on the button next to the answer you have chosen. You may change your answer by clicking on a different button.

• Click on the [Next] button at the bottom right of the screen to go to the next

• There is no penalty for guessing, so it is to your advantage to answer all of the questions.

ID06190003

Standard Instructions for Examinees (paper-and-pencil validation administration)

This test measures your reading comprehension in XXXXXX. It contains a number of reading passages in XXXXXX, each one followed by one to five questions based on that passage. In taking the test, you should do the following:

• Read the short statement before each passage that tells you where the passage came from.

• Read the XXXXXXX passage and the question or questions carefully. Some questions may be in the form of an incomplete statement.

• Based on what you have read in the passage, choose the best answer to each question from among the four answer choices. The answer to each question may be stated overtly in the passage or only implied.

• Find the question number on your answer sheet and fill in the space that has the same letter as the answer you have chosen.

There is no penalty for guessing; therefore, it is to your advantage to answer all of the

Before starting the test, please read the sample passage and question(s) provided on the following page.

Technical Exhibit 11

ID06190003

This test measures your listening comprehension in XXXXX. It contains a number of listening passages in XXXXX, each one followed by one to three questions based on that

• Before each passage you will hear a statement in English that explains the context from which the passage was taken. This statement is also printed in your test booklet.

• Then you will be given a short amount of time to read the questions pertaining to the passage.

• You will then hear the XXXXX passage itself. The passage will be played twice.

There will be a short tone signal between the two playings.

• After hearing the passage, you will be reminded which questions are to be answered at this point. Read each question and its answer choices carefully. Some questions may be in the form of an incomplete statement. The amount of time you have to answer depends on the number of questions for the passage.

• Based on what you have heard in the passage, choose the best answer to each question from among the four answer choices. The answer to each question may be stated overtly in the passage or only implied.

• Find the question number on your answer sheet and fill in the space that has the same letter as the answer you have chosen.

There is no penalty for guessing; therefore, it is to your advantage to answer all of the

You will now hear a sample passage and see a sample question or questions based on that

Technical Exhibit 11

ID06190003

Scoring and Calibration of DLPT5

Formerly, all examinees taking the validation form of a test were also required to take the 2-Skill Interview test. (See 2-Skill Interview specifications for details.) Validation test items were calibrated based on the results of the 2-Skill Interview test. The current practice is to administer only the multiple-choice tests. The calibration process is as follows:

1) A sample of examinees takes the calibration forms of the tests containing all of the items that the test development team constructed.

2) The parameters of these items (which describe the likelihood of an examinee answering correctly; for example the item difficulty) are computed from the sample.

3) The items are collected together by their nominal ILR level and the relationship between proficiency and the probability of answering correctly is graphed and analyzed numerically.

4) The underlying proficiency (Theta) required to attain the standard at a particular ILR level is determined. Typically the standard is 70% correct for the items at that level.

5) For each level there will be a proficiency cut score that separates those below the level from those at or above the level.

6) The test developers examine the item analysis and discard items that do not behave correctly (negatively related to proficiency; not truly representative of the nominal level, etc.).

7) The items that remain are the “item pool” from which the test development team selects two sets: operational forms A and B.

8) The proficiency levels identified as the cut-scores are applied to the items in each operational form to determine the raw cut-scores for each form.

9) The raw cut scores are, in turn, used to determine the range of raw scores that equates to each ILR level, called a scoring table.

10) The sample of examinees is scored using these scoring tables to determine whether the two forms are parallel and have a sufficiently high degree of agreement.

11) Examinees taking the operational forms are assigned ILR level scores according to their raw scores and the scoring table.

Calibration of items added to the pool after validation Once the base pool of items has been calibrated during the validation process, newly-written items may be added to the pool. Such items will be mixed in with calibrated items in operational forms, and calibrated to the operational items. Uncalibrated items will not be counted for examinee scores; after new items have been administered to an appropriate number of examinees, they will be considered fully calibrated and put into the pool of operational items.

Technical Exhibit 11

ID06190003

Upper Range Constructed-Response Test

Specifications

The full specification document for the DLPT5 consists of a number of parts:

1. A General Description of the test, which is for public dissemination (pp. 2-6)

2. Test Development Specifications, which are kept private and internal to EST. This includes:

a. Tables of Specifications for the Operational Linear Test (p. 8)

b. A table of content areas to be covered in the test (p. 9)

c. The Composition of the…