Tech16-B-IBS_v1_Manual.pdf

TTeecchhnniiccaall EExxhhiibbiitt 2200--BB

DLIFLC

Manual for Item Bank Specifications (IBS) Metadata Coding in Domino

Multiple Choice Test (MCT) – Reading Comprehension (RC)

Version 1

Language Proficiency Assessment Directorate 9/8/2015

FOR OFFICIAL USE ONLY

Table of Contents

Purpose of the IBS manuals

Acronyms and variable notations

MCT-RC variables at the document level

MCT-RC variables at the question level

Description of the MCT-RC document-level variables

ILR*

Setting*

Formality*

Genre*

Genre OTHER

FLO*

Topic*

Passage description*

Text type*

Source type*

Authentic-edited reason

Source country*

Source region

Writing system

Target language*

Translation*

Mode (TRE)

Description of the MCT-RC question-level variables

Item-ILR*

Negation in stem*

Open stem*

Required information location*

Required information redundancy*

Comprehension type*

Task type*

8 September 2015 IBS manual v1, MCT-RC Page 1 of 14

Purpose of the IBS manuals

• To introduce Item Bank Specifications (IBS) metadata variables for coding in Domino.

There are four IBS manuals. Table 1 below presents a summary. The red box indicates information specific to this manual.

Table 1. Summary of the IBS variables, version 1.

Test type Multiple Choice Test (MCT) Constructed Response Test (CRT)

Skill type Reading Comprehension

(RC)

Listening Comprehension

(LC)

Reading Comprehension

(RC)

Listening Comprehension

(LC)

IBS manual v1 MCT-RC MCT-LC CRT-RC CRT-LC

Passage

12 required 4 as needed 1 TRE only

17 required 4 as needed 1 TRE only

12 required 4 as needed 1 TRE only

17 required 4 as needed 1 TRE only

Item

7 required 8 required 9 required (13 required for items with a two-part stem)

10 required (15 required for items with a two-part stem)

Acronyms and variable notations o TL: Target Language o RC: Reading Comprehension o LC: Listening Comprehension o Variable name (TRE): Variables available for coding only by TRE staff.

o * : Indication of required fields.

o / : Indication of synonymous concepts.

o | : Indication of different elements in the same category.

o For drop-down menus, the option ‘Undefined’ is provided as default.

8 September 2015 IBS manual v1, MCT-RC Page 2 of 14

ILR*

Setting*

Formality*

Genre*

Genre OTHER

FLO*

Topic*

Passage description*

Text type*

Source type*

MCT-RC variables at the document level

Authentic-edited reason

Source country*

Source region

Writing system

Target language*

Translation*

Mode (TRE)

Item-ILR*

Negation in stem*

Open stem*

MCT-RC variables at the question level

Required information location*

Required information redundancy*

Comprehension type*

Task type*

8 September 2015 IBS manual v1, MCT-RC Page 3 of 14

Description of the MCT-RC document-level variables

ILR*

The difficulty of comprehending the core communicative intent(s) of the passage, expressed as the minimum required examinee proficiency on the ILR Scale.

• Coding format: a drop-down list, single selection.

• Coding options:

0+ 1+ 2+ 3+ 4+

NR

Setting* A broad sphere of the physical setting where the language act occurs.

Public Personal

• Coding definitions:

Public: Physical setting that is open to general public.

Personal: Physical setting that pertains to individuals.

Formality* Linguistic formality that is represented in the passage through the discourse register and organization, which are concerned with the way lexical items and grammatical structures are put together to form a written or spoken passage that is appropriate to a given communicative purpose and situation.

Casual/Informal Formal Mixed

8 September 2015 IBS manual v1, MCT-RC Page 4 of 14

Casual/Informal: About 80% or more linguistic elements used in the passage represent a casual or informal register and discourse organization (e.g., conversation between two friends, postcard to a brother).

Formal: About 80% or more linguistic elements used in the passage represent a formal register and discourse organization (e.g., lecture, news report, job interview).

Mixed: The linguistic elements used in the passage represent a mixture of formal and informal discourse register and organization (e.g., talk show). For example, more than 20% of the linguistic elements used in the passage represent either a formal or informal discourse register and organization.

Genre* A classification of reading texts characterized by similarities in form, function, and style.

• Note to coders:

o Select OTHER only when none of the options provided captures the particular genre of the passage under consideration.

o When selecting OTHER, code the next variable Genre OTHER.

• Coding options:

Academic/scientific article Advertisement Anecdote | Essay | Story Announcement | Notice Autobiography | Biography | Memoir Bill | Label | List | Receipt Column | Commentary | Editorial | Opinion | Op-ed Correspondence Debate | Discussion FAQ | Inquiries Flyer | Brochure | Pamphlet Form | Chart | Schedule | Table Interview Legal/official document Manual | Prescription | Recipe Miscellaneous (e.g., horoscope, weather map) News headline Note | Memo Poster | Sign Report: news, weather, traffic Report: other Review

OTHER

8 September 2015 IBS manual v1, MCT-RC Page 5 of 14

Genre OTHER The name of a new genre that cannot be captured by using the options provided for the variable

GENRE.

• Coding format: a text box.

• Note to coders: Provide a new genre name that cannot be captured by using the options provided for the variable GENRE.

FLO*

The broad subject domain of information discussed in the passage according to the DLIFLC Final Learning Objectives (FLO) content areas.

Social | Cultural Economic | Political Scientific | Technological Military | Security Geography (Physical | Political | Economic)

Topic* The specific topical domain of information discussed in the passage.

o Select OTHER only when none of the options provided captures the particular topic of the passage under consideration.

• Coding options:

Accidents | Incidents Agricultural issues Arts | Entertainment | Movie | Music | Performances | Shows Business | Investment Corruption | Bribery Crime | Violence Cultural behaviors/attitudes | Customs | Traditions Daily life affairs - Beauty | Fashion Daily life affairs - Errands | Personal/Family needs Daily life affairs - Food | Restaurant Daily life affairs - Personal/Family events Daily life affairs - Repair | Service Daily life affairs - Miscellaneous Democracy | Political reforms Economic/Financial analysis/issues Education | Career Election

8 September 2015 IBS manual v1, MCT-RC Page 6 of 14

Environmental issues | Ecology Events | Fairs | Festivals Health | Medical issues History | Historical sites or events International relations Invention | Discovery Labor issues | Hiring | Unemployment Law | Law enforcement Leisure | Tourism | Travel Media/press issues | Freedom of speech Military operations/activities/exercises National Security | Terrorism Natural disasters People | Persons Places Policy | Initiative | Plans | Program Religious issues Social trends | Social issues Sports | Competitions | Games Trade Traffic | Transportation Weather | Climate

OTHER

Passage description* A short description indicating what the passage is about.

• Coding format: a text box.

• Note to coders: Provide a short description about the passage topic (50 characters maximum).

Text type* The classification of texts based on the primary purpose of writing or speaking.

Interactional | Interpersonal Descriptive Narrative Expository | Informative Procedural | Demonstrative Persuasive Argumentative Entertaining

8 September 2015 IBS manual v1, MCT-RC Page 7 of 14

• Coding definitions for the primary text purpose:

Interactional | Interpersonal: Express social relations and/or personal feelings or attitudes to establish or maintain rapport.

Descriptive: Create an impression or image of a person, object, place, action, and/or event so that the reader/listener can picture what is being described.

Narrative: Gain readers’/listeners’ interest by telling stories about people and events, usually along a dimension of time and/or place.

Expository | Informative: Inform readers/listeners by providing descriptions, explanations, comparisons, contrasts, and elaborations that reflect analysis of information, e.g., explain why something happens, explain causes and effects, describe problems, offer solutions, compare one thing to another, etc.

Procedural | Demonstrative: Provide instructions or directions for completing a task.

Persuasive: Win someone to writer’s/speaker’s viewpoint/position; Promote/sell goods/services; Urge reader/listener to take some form of action by providing some factual information or details, reasoning, justification, discussion, or argumentation.

Argumentative: Justify/prove the validity of an idea, hypothesis, viewpoint, or position of the writer/speaker about an issue; plead a case by presenting facts, reasoning, discussion, and/or argumentation.

Entertaining: Amuse / entertain / provide pleasure to readers/listeners by expressing facts and ideas creatively and esthetically.

Source type* The way the selected passage was prepared as the test stimulus.

o If the option Authentic-edited is chosen, code the next variable Authentic-edited reason.

Authentic Authentic-edited Purpose-written

• Coding definitions (based on DLPT5 framework pp. 23-24 and pp. 48-51):

Authentic: The passage taken from a source is not modified in any way. The selected passage is produced by users of the target language and is intended to be read or heard by other users of the target language in the target-language culture (i.e., “fully authentic” as described in the DLPT5 framework, p. 23 and p. 51).

Authentic texts don’t have to be produced within the target country only; as long

8 September 2015 IBS manual v1, MCT-RC Page 8 of 14 as the texts contain features of language produced by native speakers of the target language, they are considered authentic (Widdowson, 2004).

Authentic-edited: The passage selected from an original source is edited to meet the

DLPT5 passage requirement per the respective ILR level. If this option is chosen, code the next variable called the Authentic-edited reason.

Purpose-written: A reading passage is written by a Target Language Expert (TLE) to fulfill the DLPT5 passage requirement per the respective ILR level. For example, “depending on the target language, material in certain content areas might be difficult or impossible to collect. The same might also be true for certain text types at certain levels. In order to achieve a desirable sampling of text types and subject areas at each level, it is sometimes necessary to purpose-write test passages” (Framework, p. 24).

Authentic-edited reason A specific reason for editing authentic passages.

• Note to coders: Code this variable only when you selected the option Authentic-edited from the variable above, Source type.

• Coding options:

Abridging Limited editing

• Coding definitions:

Abridging: “The fully authentic text might exceed length constraints for test passages at a given level, or it might contain embedded off-topic material. In such a case, the fully authentic text might be abridged for use as a test passage, as long as the test passage remains natural and coherent” (DLPT5 Framework, pp. 23-24).

Limited editing: “Occasionally, an isolated element of vocabulary or syntax will be in conflict with the overall level of the passage and [yet] crucial to comprehending the passage, or there will be a spelling or grammatical error in the original authentic [reading] text. In such cases, limited editing is permitted, although systematic simplification of a text is not” (DLPT5 Framework, p. 24). In case of listening, a passage editing may involve some deletion of a section of the recording, but it does not involve any new studio-recording.

Source country* The name of the country where the source material originated, e.g., Syria, Lebanon, China.

Source region The specific region(s) within the country of the source material. Indicate it by using the cardinal direction words such as north, east, south, and west (e.g., South Yemen) or regional names such as cities, towns, or provinces (e.g., Algiers, Hong Kong).

8 September 2015 IBS manual v1, MCT-RC Page 9 of 14

Writing system The writing system used in the target language written text (e.g., traditional vs. simplified characters for Mandarin or Cantonese; Gurmukhi vs. Shahmukhi for Punjabi).

Target language* The number of target language words in the passage.

o No additional action is required. Domino will capture this information automatically when target language is inserted into the box called Target Language.

Translation* The number of words in the English rendering of the target language passage.

o No additional action is required. Domino will capture this information automatically when a target language translation is inserted into the box called Translation.

o Please note that Domino will count every word entered in the “Translation” box, including additional information in squared brackets (e.g., clarification about a target language expression, explanation of background information), a marking of ‘recognizable’ information notated as [REC], and footnotes that provide further explanations.

Mode (TRE) The typology of texts, or “textual modes,” that can group texts based upon information regarding the author’s intent (Child, 1998). “The way or manner in which language text can be judged according to their evident purposes” (Child, 1987, p. 102).

• Note to coders: This variable is available only for TRE staff.

Orientational mode Instructive (informative) mode Evaluative mode Projective mode Orientational-Instructive Mixed mode Instructive (informative)-Evaluative Mixed mode Evaluative-Projective Mixed mode

8 September 2015 IBS manual v1, MCT-RC Page 10 of 14

Orientational mode: Communication of basic, concrete, factual information to orient recipient on topics of immediate need and/or social or practical nature.

Instructive (informative) mode: Communication of detailed factual and concrete information through narration, description, or instructions to inform the recipient(s).

Evaluative mode: Evaluation of facts, events, or situations of an abstract/societal nature, with analysis, claims, and hypotheses.

Projective mode: Communication through highly abstract/philosophical language/content, deeply embedded culturally; may express author’s unique and unpredictable turns of thought.

8 September 2015 IBS manual v1, MCT-RC Page 11 of 14

Description of the MCT-RC question-level variables

Item-ILR* The intended difficulty of correctly responding to the item, expressed as the minimum required examinee proficiency on the ILR Scale.

0+ 1+ 2+ 3+ 4+

NR

Negation in stem* Indication of the presence of any morpheme in the stem (question) that functions as a negating marker (e.g., not, no, in-, un-).

• Coding format: a check box.

• Note to coders: Check the box only when the stem (question) includes a negative morpheme (e.g., not, no, in-, un-).

Open stem* Indication about whether the stem (question) is an incomplete statement.

− Example of an open stem: According to the mother, her son likes to _.

• Coding format: a check box.

• Note to coders: Check the box only when the stem (question) is an incomplete statement.

Required information location* Indication of the location of the required information, which refers to information in the passage that is required for the examinee to understand in order to find the key to the question (stem). Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

8 September 2015 IBS manual v1, MCT-RC Page 12 of 14

Beginning Middle End Beginning and Middle Beginning and End Middle and End Dispersed throughout

Beginning: The required information appears within the first 33% of the passage.

Middle: The required information appears within the second 33% of the passage.

End: The required information appears within the last 33% of the passage.

Beginning and Middle: The required information appears from the beginning to the middle portion of the passage (taking up approximately 66% of the passage).

Beginning and End: The required information appears within the first 33% and the last 33% of the passage.

Middle and End: The required information appears from the middle to the last portion of the passage (taking up approximately 66% of the passage).

Dispersed throughout: The required information is scattered throughout the passage.

Required information redundancy* The extent to which required information for the item is repeated within the passage.

− Required information refers to information in the passage that is required for the examinee to understand in order to find the key to the question (stem). Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

• Coding format: a drop-down radio button, single selection.

0–25% (little redundancy) 26–50% (some redundancy) 51– 75% (quite a bit of redundancy) 76– 100% (built-in redundancy)

Comprehension type* A broad classification of comprehension that is probed by the item.

Literal

8 September 2015 IBS manual v1, MCT-RC Page 13 of 14

Inferential

Literal: Finding the correct answer to the item requires comprehension of information that is explicitly stated in the passage.

Inferential: Finding the correct answer to the item requires comprehension in which examinees need to integrate information presented in the passage together with their schemata to infer meanings that are intended by the author/speaker in the given communicative context.

Task type* The primary language comprehension task that is assigned to the item by the item developer.

• Note to coders: Select the option that most closely represents the comprehension task type assigned by the item developer.

• Coding options: The item assesses examinees’ ability to:

understand a single word or phrase understand the general subject matter under discussion understand a factual detail that is mentioned in the passage understand a viewpoint, opinion, suggestion, or advice understand sequence of events / sequencing understand causal relationship / cause and effect understand differences and/or similarities of events, issues, entities, viewpoints, etc.

differentiate facts from opinions understand the contextual meaning of words, phrases, or idioms understand the reason/purpose of a given sociolinguistic and/or cultural reference or allusion understand the reasons/purposes of particular examples or illustrations as used by the author or speaker understand the author’s or speaker’s intended purpose understand the author’s or speaker’s attitude about or tone on an issue understand lines of argumentation supporting the author’s or speaker’s view/position draw/reach appropriate conclusions by evaluating details and making a judgment make a prediction based on information presented

8 September 2015 IBS manual v1, MCT-RC Page 14 of 14