Tech15-A-IBS_v1_Manual.pdf

TTeecchhnniiccaall EExxhhiibbiitt 2200--AA

DLIFLC

Manual for Item Bank Specifications (IBS) Metadata Coding in Domino

Multiple Choice Test (MCT) – Listening Comprehension (LC)

Version 1

Language Proficiency Assessment Directorate 9/8/2015

FOR OFFICIAL USE ONLY

Table of Contents

Purpose of the IBS manuals

Acronyms and variable notations

MCT-LC variables at the document level

MCT-LC variables at the question level

Description of the MCT-LC document-level variables

ILR*

Setting*

Formality*

Genre*

Genre OTHER

FLO*

Topic*

Passage description*

Planning*

Text type*

Source type*

Authentic-edited reason

Studio-recorded reason

Source country*

Source region

LC orientation voice*

TL speaker-Male*

TL speaker-Female*

Audio*

Transcription*

Translation*

Mode (TRE)

Description of the MCT-LC question-level variables

Item-ILR*

Negation in stem*

Open stem*

Required information location*

8 September 2015 IBS manual v1, MCT-LC Page 1 of 17

Required information redundancy*

Audio distracter in required information*

Comprehension type*

Task type*

8 September 2015 IBS manual v1, MCT-LC Page 2 of 17

Purpose of the IBS manuals

• To introduce Item Bank Specifications (IBS) metadata variables for coding in Domino.

There are four IBS manuals. Table 1 below presents a summary. The red box indicates information specific to this manual.

Table 1. Summary of the IBS variables, version 1.

Test type Multiple Choice Test (MCT) Constructed Response Test (CRT)

Skill type Reading Comprehension

(RC)

Listening Comprehension

(LC)

Reading Comprehension

(RC)

Listening Comprehension

(LC)

IBS manual v1 MCT-RC MCT-LC CRT-RC CRT-LC

Passage

12 required 4 as needed 1 TRE only

17 required 4 as needed 1 TRE only

12 required 4 as needed 1 TRE only

17 required 4 as needed 1 TRE only

Item

7 required 8 required 9 required (13 required for items with a two-part stem)

10 required (15 required for items with a two-part stem)

Acronyms and variable notations o TL: Target Language o RC: Reading Comprehension o LC: Listening Comprehension o Variable name (TRE): Variables available for coding only by TRE staff.

o * : Indication of required fields.

o / : Indication of synonymous concepts.

o | : Indication of different elements in the same category.

o For drop-down menus, the option ‘Undefined’ is provided as default.

8 September 2015 IBS manual v1, MCT-LC Page 3 of 17

ILR*

Setting*

Formality*

Genre*

Genre OTHER

FLO*

Topic*

Passage description*

Planning*

Text type*

Source type*

MCT-LC variables at the document level

Authentic-edited reason

Studio-recorded reason

Source country*

Source region

LC orientation voice*

TL speaker-Male*

TL speaker-Female*

Audio*

Transcription*

Translation*

Mode (TRE)

Item-ILR*

Negation in stem*

Open stem*

MCT-LC variables at the question level

Required information location*

Required information redundancy*

Audio distracter in required information*

Comprehension type*

Task type*

8 September 2015 IBS manual v1, MCT-LC Page 4 of 17

Description of the MCT-LC document-level variables

ILR*

The difficulty of comprehending the core communicative intent(s) of the passage, expressed as the minimum required examinee proficiency on the ILR Scale.

• Coding format: a drop-down list, single selection.

• Coding options:

0+ 1+ 2+ 3+ 4+

NR

Setting* A broad sphere of the physical setting where the language act occurs.

Public Personal

• Coding definitions:

Public: Physical setting that is open to general public.

Personal: Physical setting that pertains to individuals.

Formality* Linguistic formality that is represented in the passage through the discourse register and organization, which are concerned with the way lexical items and grammatical structures are put together to form a written or spoken passage that is appropriate to a given communicative purpose and situation.

Casual/Informal Formal Mixed

8 September 2015 IBS manual v1, MCT-LC Page 5 of 17

Casual/Informal: About 80% or more linguistic elements used in the passage represent a casual or informal register and discourse organization (e.g., conversation between two friends, postcard to a brother).

Formal: About 80% or more linguistic elements used in the passage represent a formal register and discourse organization (e.g., lecture, news report, job interview).

Mixed: The linguistic elements used in the passage represent a mixture of formal and informal discourse register and organization (e.g., talk show). For example, more than 20% of the linguistic elements used in the passage represent either a formal or informal discourse register and organization.

Genre* A classification of listening texts characterized by similarities in form, function, and style.

• Note to coders:

o Select OTHER only when none of the options provided captures the particular genre of the passage under consideration.

o When selecting OTHER, code the next variable Genre OTHER.

Address | Lecture | Presentation | Speech Announcement | Notice Anecdote | Story Autobiography | Biography | Memoir Command | Instructions | Order Commercial Column | Commentary | Editorial | Opinion piece | Op-ed Conversation Debate | Discussion Interview Report: news, traffic, weather Report: other Review Voice message

OTHER

Genre OTHER The name of a new genre that cannot be captured by using the options provided for the variable

GENRE.

• Coding format: a text box.

• Note to coders: Provide a new genre name that cannot be captured by using the options provided for the variable GENRE.

8 September 2015 IBS manual v1, MCT-LC Page 6 of 17

FLO*

The broad subject domain of information discussed in the passage according to the DLIFLC Final Learning Objectives (FLO) content areas.

Social | Cultural Economic | Political Scientific | Technological Military | Security Geography (Physical | Political | Economic)

Topic* The specific topical domain of information discussed in the passage.

o Select OTHER only when none of the options provided captures the particular topic of the passage under consideration.

• Coding options:

Accidents | Incidents Agricultural issues Arts | Entertainment | Movie | Music | Performances | Shows Business | Investment Corruption | Bribery Crime | Violence Cultural behaviors/attitudes | Customs | Traditions Daily life affairs - Beauty | Fashion Daily life affairs - Errands | Personal/Family needs Daily life affairs - Food | Restaurant Daily life affairs - Personal/Family events Daily life affairs - Repair | Service Daily life affairs - Miscellaneous Democracy | Political reforms Economic/Financial analysis/issues Education | Career Election Environmental issues | Ecology Events | Fairs | Festivals Health | Medical issues History | Historical sites or events International relations Invention | Discovery Labor issues | Hiring | Unemployment

8 September 2015 IBS manual v1, MCT-LC Page 7 of 17

Law | Law enforcement Leisure | Tourism | Travel Media/press issues | Freedom of speech Military operations/activities/exercises National Security | Terrorism Natural disasters People | Persons Places Policy | Initiative | Plans | Program Religious issues Social trends | Social issues Sports | Competitions | Games Trade Traffic | Transportation Weather | Climate

OTHER

Passage description* A short description indicating what the passage is about.

• Coding format: a text box.

• Note to coders: Provide a short description about the passage topic (50 characters maximum).

Planning* Degree of speech planning involved when the listening passage was created.

• Coding option:

Planned Semi-planned Unplanned (Spontaneous) Undefinable

Planned: The listening passage represents a prepared, organized, and polished communication (e.g., news report).

Semi-planned: The speaker(s) had some time to prepare a mental draft or to practice beforehand (e.g., giving a presentation, job interview).

Unplanned (Spontaneous): The listening passage represents spontaneous speech produced without taking much time for preparation or organization (e.g., conversation with family).

8 September 2015 IBS manual v1, MCT-LC Page 8 of 17

Text type* The classification of texts based on the primary purpose of writing or speaking.

Interactional | Interpersonal Descriptive Narrative Expository | Informative Procedural | Demonstrative Persuasive Argumentative Entertaining

• Coding definitions for the primary text purpose:

Interactional | Interpersonal: Express social relations and/or personal feelings or attitudes to establish or maintain rapport.

Descriptive: Create an impression or image of a person, object, place, action, and/or event so that the reader/listener can picture what is being described.

Narrative: Gain readers’/listeners’ interest by telling stories about people and events, usually along a dimension of time and/or place.

Expository | Informative: Inform readers/listeners by providing descriptions, explanations, comparisons, contrasts, and elaborations that reflect analysis of information, e.g., explain why something happens, explain causes and effects, describe problems, offer solutions, compare one thing to another, etc.

Procedural | Demonstrative: Provide instructions or directions for completing a task.

Persuasive: Win someone to writer’s/speaker’s viewpoint/position; Promote/sell goods/services; Urge reader/listener to take some form of action by providing some factual information or details, reasoning, justification, discussion, or argumentation.

Argumentative: Justify/prove the validity of an idea, hypothesis, viewpoint, or position of the writer/speaker about an issue; plead a case by presenting facts, reasoning, discussion, and/or argumentation.

Entertaining: Amuse / entertain / provide pleasure to readers/listeners by expressing facts and ideas creatively and esthetically.

Source type* The way the selected passage was prepared as the test stimulus.

8 September 2015 IBS manual v1, MCT-LC Page 9 of 17 o If the option Authentic-edited is chosen, code the next variable Authentic-edited reason.

o If the option Studio-recorded is chosen, code the next variable Studio-recorded reason.

Authentic Authentic-edited Studio-recorded

• Coding definitions (based on DLPT5 framework pp. 23-24 and pp. 48-51):

Authentic: The passage taken from a source is not modified in any way. The selected passage is produced by users of the target language and is intended to be read or heard by other users of the target language in the target-language culture (i.e., “fully authentic” as described in the DLPT5 framework, p. 23 and p. 51).

Authentic texts don’t have to be produced within the target country only; as long as the texts contain features of language produced by native speakers of the target language, they are considered authentic (Widdowson, 2004).

Authentic-edited: The passage selected from an original source is edited to meet the

DLPT5 passage requirement per the respective ILR level. If this option is chosen, code the next variable called the Authentic-edited reason.

Studio-recorded: A listening passage is recorded by one or more TLEs to fulfill the passage requirement per the respective ILR level. If this option is chosen, code the next variable called the Studio-recorded reason.

Authentic-edited reason A specific reason for editing authentic passages.

• Note to coders: Code this variable only when you selected the option Authentic-edited from the variable above, Source type.

Abridging Limited editing

Abridging: “The fully authentic text might exceed length constraints for test passages at a given level, or it might contain embedded off-topic material. In such a case, the fully authentic text might be abridged for use as a test passage, as long as the test passage remains natural and coherent” (DLPT5 Framework, pp. 23-24).

8 September 2015 IBS manual v1, MCT-LC Page 10 of 17

Limited editing: “Occasionally, an isolated element of vocabulary or syntax will be in conflict with the overall level of the passage and [yet] crucial to comprehending the passage, or there will be a spelling or grammatical error in the original authentic [reading] text. In such cases, limited editing is permitted, although systematic simplification of a text is not” (DLPT5 Framework, p. 24). In case of listening, a passage editing may involve some deletion of a section of the recording, but it does not involve any new studio-recording.

Studio-recorded reason A specific reason for constructing studio-recorded passages.

• Note to coders: Code this variable only when the option Studio-recorded is selected from the variable Source type.

• Coding options:

To improve audio quality To improve the content To improve both audio quality and the content To construct non-scripted content To construct semi-scripted content To construct fully-scripted content

To improve audio quality: The original listening passage includes some distortion in the audio quality (e.g., existence of inaudible portion due to noise). Thus, one or more TLEs re-record the original passage without any content modification.

To improve the content: The original listening passage may include some language features or content that is not appropriate for the intended ILR level. Thus, one or more TLEs edit the content of the original passage and re-record it in the studio.

To improve both the audio quality and the content: Studio-recording is done due to both of the two reasons mentioned above.

To construct non-scripted content: One or more TLEs discuss a scenario of a story (including the topic, main points, participant roles) and then carry out a natural conversation in the assigned roles (without any notes or scripts).

To construct semi-scripted content: One or more TLEs develop a semi-scripted content structure (e.g., an outline without a full transcript). Then, the TLE(s) carry out a natural speaking of the content based on those notes in the studio.

To construct fully-scripted content: One or more TLEs come up with a fully-scripted passage content and read the script as naturally as possible in the studio recording session.

8 September 2015 IBS manual v1, MCT-LC Page 11 of 17

Source country* The name of the country where the source material originated, e.g., Syria, Lebanon, China.

Source region The specific region(s) within the country of the source material. Indicate it by using the cardinal direction words such as north, east, south, and west (e.g., South Yemen) or regional names such as cities, towns, or provinces (e.g., Algiers, Hong Kong).

LC orientation voice* The gender of the speaker who provided the voice for the passage’s orientation statement.

M: Male F: Female

TL speaker-Male* The number of male speaker(s) present in the target language passage.

TL speaker-Female* The number of female speaker(s) presented in the target language passage.

Audio* The number of seconds in target language audio (excluding the duration of the orientation statement).

o No additional action is required. Domino will capture this information automatically when a target language passage is inserted into the platform.

Transcription* The number of words in the target language transcript for a listening passage.

automatically when a target language transcription is inserted into the platform.

Translation* The number of words in the English rendering of the target language passage.

8 September 2015 IBS manual v1, MCT-LC Page 12 of 17 automatically when a target language translation is inserted into the platform.

o Please note that Domino will count every word entered in the “Translation” box, including additional information in squared brackets (e.g., clarification about a target language expression, explanation of background information), a marking of ‘recognizable’ information notated as [REC], and footnotes that provide further explanations.

Mode (TRE) The typology of texts, or “textual modes,” that can group texts based upon information regarding the author’s intent (Child, 1998). “The way or manner in which language text can be judged according to their evident purposes” (Child, 1987, p. 102).

• Note to coders: This variable is available only for TRE staff.

Orientational mode Instructive (informative) mode Evaluative mode Projective mode Orientational-Instructive Mixed mode Instructive (informative)-Evaluative Mixed mode Evaluative-Projective Mixed mode

Orientational mode: Communication of basic, concrete, factual information to orient recipient on topics of immediate need and/or social or practical nature.

Instructive (informative) mode: Communication of detailed factual and concrete information through narration, description, or instructions to inform the recipient(s).

Evaluative mode: Evaluation of facts, events, or situations of an abstract/societal nature, with analysis, claims, and hypotheses.

Projective mode: Communication through highly abstract/philosophical language/content, deeply embedded culturally; may express author’s unique and unpredictable turns of thought.

8 September 2015 IBS manual v1, MCT-LC Page 13 of 17

Description of the MCT-LC question-level variables

Item-ILR* The intended difficulty of correctly responding to the item, expressed as the minimum required examinee proficiency on the ILR Scale.

0+ 1+ 2+ 3+ 4+

NR

Negation in stem* Indication of the presence of any morpheme in the stem (question) that functions as a negating marker (e.g., not, no, in-, un-).

• Coding format: a check box.

• Note to coders: Check the box only when the stem (question) includes a negative morpheme (e.g., not, no, in-, un-).

Open stem* Indication about whether the stem (question) is an incomplete statement.

− Example of an open stem: According to the mother, her son likes to .

• Coding format: a check box.

• Note to coders: Check the box only when the stem (question) is an incomplete statement.

Required information location* Indication of the location of the required information, which refers to information in the passage that is required for the examinee to understand in order to find the key to the question (stem). Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

Beginning Middle

8 September 2015 IBS manual v1, MCT-LC Page 14 of 17

End Beginning and Middle Beginning and End Middle and End Dispersed throughout

Beginning: The required information appears within the first 33% of the passage.

Middle: The required information appears within the second 33% of the passage.

End: The required information appears within the last 33% of the passage.

Beginning and Middle: The required information appears from the beginning to the middle portion of the passage (taking up approximately 66% of the passage).

Beginning and End: The required information appears within the first 33% and the last 33% of the passage.

Middle and End: The required information appears from the middle to the last portion of the passage (taking up approximately 66% of the passage).

Dispersed throughout: The required information is scattered throughout the passage.

Required information redundancy* The extent to which required information for the item is repeated within the passage.

− Required information refers to information in the passage that is required for the examinee to understand in order to find the key to the question (stem). Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

• Coding format: a drop-down radio button, single selection.

0–25% (little redundancy) 26–50% (some redundancy) 51– 75% (quite a bit of redundancy) 76– 100% (built-in redundancy)

Audio distracter in required information* Indication of the presence of any auditory distracter within the portion of the passage that discusses required information.

− Required information refers to information in the passage that is required for the examinee to understand in order to find the key to the question (stem). Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

8 September 2015 IBS manual v1, MCT-LC Page 15 of 17

• Coding format: a drop-down list, multiple selection.

• Note to coders: Select all relevant options.

None White noise or echo Non-human noise (e.g., music, car) Non-speech human noise (e.g., coughing) Inaudible word or phrase Speech overlap at turn-taking (e.g., at sentence beginning or end) Simultaneous talk

Comprehension type* A broad classification of comprehension that is probed by the item.

Literal Inferential

Literal: Finding the correct answer to the item requires comprehension of information that is explicitly stated in the passage.

Inferential: Finding the correct answer to the item requires comprehension in which examinees need to integrate information presented in the passage together with their schemata to infer meanings that are intended by the author/speaker in the given communicative context.

Task type* The primary language comprehension task that is assigned to the item by the item developer.

• Note to coders: Select the option that most closely represents the comprehension task type assigned by the item developer.

• Coding options: The item assesses examinees’ ability to:

understand a single word or phrase understand the general subject matter under discussion understand a factual detail that is mentioned in the passage understand a viewpoint, opinion, suggestion, or advice understand sequence of events / sequencing understand causal relationship / cause and effect

8 September 2015 IBS manual v1, MCT-LC Page 16 of 17 understand differences and/or similarities of events, issues, entities, viewpoints, etc.

differentiate facts from opinions understand the contextual meaning of words, phrases, or idioms understand the reason/purpose of a given sociolinguistic and/or cultural reference or allusion understand the reasons/purposes of particular examples or illustrations as used by the author or speaker understand the author’s or speaker’s intended purpose understand the author’s or speaker’s attitude about or tone on an issue understand lines of argumentation supporting the author’s or speaker’s view/position draw/reach appropriate conclusions by evaluating details and making a judgment make a prediction based on information presented

8 September 2015 IBS manual v1, MCT-LC Page 17 of 17