Tech13-Rules for Writing MC DLPT items.pdf

DLIFLC

Rules for Writing DLPT Items For Multiple Choice Tests

Test Production Division Revision: October 2016

Rules for Writing DLPT Items – MC

Table of Contents

INTRODUCTION AND TERMS

ITEM DEVELOPMENT

1. THE PASSAGE

2. THE ORIENTATION STATEMENT

3. THE ITEM

4. THE STEM

5. THE OPTIONS

Key

Distractors

6. THE ITEM SET

INTRODUCTION AND TERMS

INTRODUCTION

This document was assembled to help test item developers develop multiple choice test material for the Defense Language Proficiency Test (DLPT). It provides specific rules for writing acceptable multiple choice reading and listening comprehension test items, targeting proficiency levels 0+ to 4 on the Interagency Language Roundtable (ILR) scale.

TERMS

TL Target Language (the language being tested)

RC Reading Comprehension

LC Listening Comprehension

Item The unit consisting of a Stem and options, accompanied by an orientation statement and passage

Item Set A set of all item(s) based on one passage

Passage The TL text, whether spoken or written words

OS An orientation statement that informs the examinees of the context in which the passage would commonly be found

Stem The question posed to examinees about information in the passage

Option Any one of the answer choices to the Stem

Key The option that is the correct answer to the Stem

Distractor Any option that is an incorrect answer to the Stem

ITEM DEVELOPMENT

1. THE PASSAGE

The purpose of the passage is to present examinees with a realistic sample of the TL, so that their comprehension of the TL, as it is used for communicative purposes, can be assessed.

• A passage must represent authentic language as used in real-world contexts.

• A passage must contain the characteristics of texts that are typical in communicative TL-use situations.

• A passage must be self-standing.

• A passage must represent the variant of the language that has been specified for the language being tested.

• A passage must be culturally appropriate, plausible, believable, and logical.

• A passage that might evoke a strong emotional reaction must not be used.1

•A passage must meet the ILR and text mode specifications for its level.

•A passage must be no longer than the maximum length specified for its level.

•The topic of a passage must fall under one of the five DLIFLC Final Learning Objective

(FLO) content areas : Military/Security, Economic/Political, Scientific/Technological, Cultural and Social, and Geography-Physical/Political/Economic.

• Some passages may be modified as necessary to conform to specifications regarding passage length, coherence, datedness, vocabulary (in RC), or sound quality (in LC).

• Some passages may be created for the purposes of the test (purpose-written). These passages must represent authentic language as used in the TL; they must reflect and adhere to TL culture, social norms and customs; and they must appear (for RC) or sound (for LC) authentic to a native speaker of the TL.

2. THE ORIENTATION STATEMENT

The purpose of the orientation statement (OS) is to provide examinees with the context for the TL passage.

• The OS must provide information about the source of the TL passage and/or identify a genre for the passage.

• The OS must not include information that can help examinees arrive at a correct answer without reading or listening to the passage.

• The RC OS must be written as either a full noun phrase that includes the articles “A” or “An” before the noun or as a prepositional phrase that begins with “From.” If the passage represents an incomplete text (an excerpt from a text), the OS must clearly signal this fact by beginning with the preposition “From.” The RC OS must begin with a capital letter and not end with a period.

• The LC OS must be a complete sentence that begins with “This passage is...” and ends with a period. If the passage represents an incomplete text (an excerpt from a text), the OS must clearly signal this fact by inserting the word “from”: “This passage is from...”

1 Refer to DLPT Fairness Review Guidelines.

• The LC OS must reflect the order in which the speakers are heard. If a conversation between a man and woman begins with the woman’s voice, the OS must read A conversation between a woman and a man, for example.

3. THE ITEM

The item presents a task that is designed to provide evidence of examinees’ comprehension of the TL passage.

• The task posed in the item must require comprehension of the passage.

•The item must target information that is level-appropriate according to the test specifications.

• The item must have one and only one correct answer.

• The item must not inadvertently assess an examinee’s general knowledge or allow the examinee to answer without having to understand the TL passage.

• The item must not inadvertently require specialized content knowledge or skills other than language proficiency—such as mathematics or critical reasoning—in order to identify the correct answer.

• The item must not unfairly advantage or disadvantage any group of examinees.

4. THE STEM

The purpose of the Stem is to make it clear to examinees what information needs to be found.

• The Stem must be written to specify the task precisely and unambiguously, using the simplest possible vocabulary and syntax to do so.

• The Stem must be either closed (a question) or open-ended (an incomplete statement that is completed by the options).

5. THE OPTIONS

• Options must be clear and concise.

• Each option must express a coherent idea.

• Options must not be longer than 20 words.

• Options must be free of ambiguous or vague expressions (e.g., ambiguous pronoun antecedent or vague reference).

• Options must be structurally balanced.

• Options must be grammatically correct answers to the Stem.

• Options must not be direct opposites of each other.

• One option must not logically entail another option.

KEY

• The Key must be a complete and correct answer to the question posed in the Stem.

• The Key must be recognizable to examinees who have understood the passage at the level targeted by the question.

DISTRACTORS

• The distractors must be plausible according to the content and context of the passage, but incorrect answers to the question posed in the Stem, according to the information in the passage.

• Distractors must not be correct by inference.

6. THE ITEM SET

• Each item in a set must be independent of the other items in the set.

• The Stem of an item must not point to the Key of any other items in the same item set.

• The Key of one item must not influence the Key of another item.

• Items must be presented in the same order that the information appears in the passage.