Tech8-10-F-IBS Contractor Full Introduction and Passage Rating Variables.pptx

IBS Workshop Session #1

LPAD - TRP

Presentation Overview Introduction The IBS What is it?

What does it do?

Passage Variables (RC & LC) coding Practice Item Variables (RC & LC) Coding Practice

Too many pairs of shoes……

Hard to find the ones I want…

Sarah

What should I do?

Labeling

Brooks Purple Yellow stripes Green shoe lace Running Minimalist Exercise

What is IBS?

and What does the IBS do?

Item Bank Specification

What is IBS?

1. The LPAD directorate’s level multi-year effort to construct item-level classification system for DLPT5 item banks

2. To identify and organize DLPT5 test item characteristics per passage and question

3. To help map DLPT5 test construct as guided by ILR skill-level descriptors and in reference to language testing literature

4. To capture the knowledge, skills and abilities that are measured through the items.

3 - “The DLPT5 test construct is the ability to understand written and spoken input intended for the general public in terms of the target-language use domain as specified by the ILR. The DLPT5 measures use of receptive language skills in context, i.e., listening comprehension ability and reading comprehension ability in a given target language for a communicative purpose.” (DLPT5 Framework, p.10) Note: The DLPT5 construct can be understood is the language ability DLPT5 is purported to measure. Inspection of IBS item and passage variables can be used to see if the test items are assessing what they are supposed to measure.

4 - “The DLPT5 Reading Test assesses reading from the following purpose perspectives: reading to find information, reading to understand main ideas or main points, and reading requiring examinees to integrate and connect detailed information conveyed by the author.” (DLPT5 Framework, p.19)

Structure of the IBS variables

1. Per test type Multiple choice test & constructed response test

2. Per skill Reading vs. Listening

3. Levels of variables

1) passage variables

2) item variables

IBS v1 structure (2015)

(LC)

4 as needed 1 TRE only 11 required* 4 as needed 1 TRE only 12 required* 4 as needed 1 TRE only` 17 required* 4 as needed 1 TRE only

(15 required for items with a two-part stem)

RC (12->9, setting, translation, target language removed) LC (17 ->11, setting, planning, translation, TL, LC orientation voice, audio removed)

15 AUG 2016

Passage Variables: MCT

RC & LC

ILR*

Setting* Formality*

FLO*

Topic* Passage description* Text type* Source type* Authentic-edited reason Source Country* Source Region Mode (TRE)

RC

Genre* Genre OTHER Writing System Target Language

LC

Genre* Genre OTHER Planning Studio-recorded reason TL Speaker – Male* TL Speaker – Female*

Meltem

Item Variables: MCT

RC

Item-ILR* Negation in Stem* Open Stem* Required Information Location* Required Information Redundancy* Comprehension Type* Task Type*

LC

Item-ILR* Negation in Stem* Open Stem* Required Information Location* Required Information Redundancy* Comprehension Type* Task Type* Audio distractor in required information*

Where is IBS data stored?

Domino is used as platform for coding and storage of IBS v1 metadata There are 4 different coding formats:

- a drop-down list, single selection

- A check box, single selection

- A check box, multiple selection

- A text box

IBS Passage & Item Variables

Agenda

I will be going over the passage variables for Reading and for Listening comprehension. You already have this information in the IBS manuals, so In the following slides, we will be quickly going over the definitions of each coding variable, the coding options for each variable first for Reading and then for listening. We will then be looking at a sample in Domino.

Passage Variables (RC vs LC)

Coding Practice: Passage Variables

Item Variables (MC vs CR)

Coding Practice: Item Variables

Definitions

Coding options

Overview of Variables

Sample in Domino

Passage Variables RC & LC

(LC)

4 as needed 1 TRE only 11 required 4as needed 1 TRE only 12 required 4 as needed 1 TRE only 17 required 4 as needed 1 TRE only

I will be going over the definitions of the passage variables for Reading and for Listening comprehension that you will be coding. The number of coding options is 11 for RC and 15 for LC as noted in the OBS manual, but we excluded the ones that do not need to be filled out by you.

Passage Variables

RC

ILR*

Setting* Formality*

FLO*

Topic* Passage description* Text type* Source type* Authentic-edited reason Source Country* Source Region Mode (TRE)

RC

Genre* Genre OTHER Writing System Target Language

So here again is an overview of the passage variables for RC. As you can see, the variables that are mandatory are marked with an asterix. The variables in the blue box are the same for LC and RC and while Genre is the same for LC and RC, the subcategories are different for each skill. So let’s go over each variable one at a time.

Passage ILR* Coding Format:

Coding Options:

0+ 1+ 2+ 3+ 4+

NR

The difficulty of comprehending the core communicative intent(s) of the passage, expressed as the minimum required examinee proficiency on the ILR scale.

This variable is self-explanatory, but please taka a moment to read the definition. The coding format is a drop-down list with a single selection and coding options include all ILR levels. You will notice that there is “NR” = not ratable at the end of the list. This option is used by TRE only, so you can disregard it.

Formality* Linguistic formality that is represented in the passage through the discourse register and organization, which are concerned with the way lexical items and grammatical structures are put together to form a written or spoken passage that is appropriate to a given communicative purpose and situation.

Coding Format:

Coding Options:

Casual/Informal Formal Mixed

The next variable is formality. Please again take a moment to read the description. For this mandatory variable we also have a drop-down list where we select one of the three options: Casual/Informal, Formal, Mixed. Before I go to the next slide, I would like to note that this variable for most languages is determined based on Linguistic features and not based on context. Let’s talk about this in the next slide.

Formality* (continued) Conversation between two friends Lecture Postcard to a brother Talk show News report Job interview Conversation between a store clerk and a customer Casual/Informal

Formal

Mixed Coding Options:

EXAMPLES

Sometimes it may be difficult to determine the formality level of a passage. In IBS version 1, we are focusing on linguistic forms to determine the formality level. For example, In English, we do not have formal and informal endings that mark our verbs or we do not have different pronouns we use when we want to change the formality of our language. But for other language like Turkish or Japanese we have linguistic markers. For example, in Japanese a mother while talking to her child could use formal language with honorific endings. While the formality should be “Casual/Informal” as mothers usually use a casual register while talking to their children, in the Japanese example we should code the passage as formal due to the linguistic forms used. Again, it will the linguistic forms that determine the formality.

Another example might be a post card to a brother that we would expect to be casual/informal. However in some languages a post card written by a younger brother or sister to an older brother might contain formal linguistic forms and should be coded “Formal” or maybe mixed depending on the language used.

A good example when ”Mixed” can be chosen for formality level would be Talk show where the host introduces a guest to the audience using a formal register, but when the host talks directly to the guest she switches to a an informal register using informal linguistic forms.

Genre*

A classification of reading texts characterized by similarities in form, function, and style.

Note to Coders:

Select OTHER only when none of the options provided captures the particular genre of the passage.

When selecting OTHER, code the next variable GENRE OTHER.

Our next variable is Genre. Please take a moment to read the description. For this mandatory variable we also have a drop-down list where we select one option. We have quite a few options to choose from and before I show these options on the next slide, please note that we have the option “other” that can be selected if none of the provided coding options captures the genre of your passage. When you choose “other” you will see that another field appears where you will need to type in the genre of the passage.

Genre* (continued) Academic/Scientific Article Advertisement Anecdote | Essay | Story Announcement | Notice Autobiography | Biography | Memoir Bill | Label | List | Receipt Column | Commentary | Editorial | Opinion | Op-ed Correspondence Debate | Discussion FAQ | Inquiries Flyer | Brochure | Pamphlet Form | Chart | Schedule | Table Interview Legal/official document Manual | Prescription | Recipe Miscellaneous (e.g., horoscope, weather map) News headline Note | Memo Poster | Sign Report: news, weather, traffic Report: other Review

OTHER

RC Coding Options:

So here is the list of options for genre. As you can see, We have a long list of subcategories, but we do understand that with authentic passages there may be a case where you cannot choose one of them. In which case You can choose "Other" but please use it sparingly and only when absolutely needed.

Genre OTHER Coding Format:

Note to Coders: Provide a new genre name that cannot be captured by using the options provided for the variable GENRE.

The name of a new genre that cannot be captured by using the options provided for the variable GENRE.

Text Box

If you do select “other” for genre, you will be prompted to fill in the text box with the appropriate genre for your passage.

Do we have any questions up to this point?

FLO*

Coding Format:

Coding Options:

The broad subject domain of information discussed in the passage according to the DLIFLC Final Learning Objectives (FLO) content areas. (DLPT5 Framework pp.5-8)

We have five FLO content areas under which we code our passages for their subject domain. While the FLO topics are self-explanatory, let’s look at the the sub-categories provided in the FLO Handout. I find this document very helpful since it gives a good idea what subject domain to choose for passages at various levels. But before I open the document, I would like to point out that "Accidents" are not listed in the sub-categories. When you have a passage related to an accident, please code it as "military/security". While it could be argued that accidents could be a social topic, our internal criterion for accident passage is military security. It would be nice if dual coding was possible, but currently we can only code one. For passages where you detect two subject domains, choose the FLO topic that is covered more heavily. If the domains are represented 50/50, you can toss a coin.

Topic* The specific topical domain of information discussed in the passage.

Coding format:

Note to coders:

Select OTHER only when none of the options provided captures the particular topic of the passage under consideration.

When selecting the topic for you passage, you will see that that the drop-down list is quite extensive. But, you also have the option to select ”other” if the topic of your passage is not provided in the list.

Topic* (continued) Accidents | Incidents Agricultural Issues Arts | Entertainment | Movie | Music | Performances | Shows Business | Investment Corruption | Bribery Crime | Violence Events | Fairs | Festivals Cultural Behaviors/Attitudes | Customs | Traditions Daily Life Affairs - Beauty | Fashion Daily Life Affairs - Food | Restaurant Daily Life Affairs - Personal/Family Events Daily Life Affairs - Errands | Personal/Family Needs Daily Life Affairs - Repair | Service Daily Life Affairs - Miscellaneous Democracy | Political Reforms Economic/Financial Analysis/Issues Education | Career Election Environmental Issues | Ecology Health | Medical issues History | Historical sites or events International relations Invention | Discovery Labor issues | Hiring | Unemployment Law | Law enforcement Leisure | Tourism | Travel Media/press issues | Freedom of speech Military operations/activities/exercises National Security | Terrorism Natural disasters People | Persons Places Initiative | Plans | Policy | Program Religious issues Social trends | Social issues Sports | Competitions | Games Trade Traffic | Transportation Weather | Climate

OTHER

Please note the options provided with a strikethrough. These are still visible in Domino, but please do not select them and only use “Daily life affairs – Miscellaneous” if you passage is related to daily life event.

Passage description* A short description indicating what the passage is about.

Coding Format:

Note to Coders: Provide a short description about the passage topic (50 characters maximum).

Text type* Coding Format:

Coding Options:

The classification of texts based on the primary purpose of writing or speaking.

Text type is based on the writer’s or speaker’s primary purpose. So we have eight coding options for his variable. Let’s take a look at the description of these coding options on the next slide.

H. Express social relations and/or personal feelings or attitudes to establish or maintain rapport.

A. Create an impression or image of a person, object, place, action, and/or event so that the reader/listener can picture what is being described.

E. Gain readers’/listeners’ interest by telling stories about people and events, usually along a dimension of time and/or place.

G. Inform readers/listeners by providing descriptions, explanations, comparisons, contrasts, and elaborations that reflect analysis of information.

F. Amuse / entertain / provide pleasure to readers/listeners by expressing facts and ideas creatively and esthetically.

D. Justify/prove the validity of an idea, hypothesis, viewpoint, or position of the writer/speaker about an issue; plead a case by presenting facts, reasoning, discussion, and/or argumentation.

C. Win someone to writer’s/speaker’s viewpoint/position; Promote/sell goods/services; Urge reader/listener to take some form of action by providing some factual information or details, reasoning, justification, discussion, or argumentation.

B. Provide instructions or directions for completing a task.

Persuasive/Argumentative are interchangable

Interactional | Interpersonal

Descriptive

Narrative

Expository | Informative

Procedural | Demonstrative

Persuasive

Argumentative

Entertaining

Source type*

Coding Options:

Note to Coders: If the option Authentic-edited is chosen, code the next variable Authentic-edited reason.

Authentic-Edited reason The way the selected passage was prepared as the test stimulus.

When you select passages as test stimulus, you have three options, you can select authentic passages, you can select Authentic – edited and you can select purpose written. If you had to edit the authentic passage either for lengths or other reasons, you have to code it as “authentic - edited”. The next variable Authentic-edited reason will pop up in Domino for you to populate.

Authentic-edited reason Coding Format:

Coding Options:

Note to Coders: Code this variable only when you selected the option Authentic-edited from the variable above, source type.

Note to Coders: When Abridging and Limited Editing occur simultaneously, choose Limited Editing.

A specific reason for editing authentic passages.

If you have edited the RC passage, you need to provide the reson for the editing and have two options. Please note that Note When abridging and limited editing occur simultaneously, choose Limited editing.

Source Country* The name of the country where the source material originated, e.g., Syria, Lebanon, China.

Qb is coded by source country, e.g all the different Spnaish-spekaing latin American countries in addition to Spain.

Source Region The specific region(s) within the country of the source material. Indicate it by using the cardinal direction words such as north, east, south, and west (e.g., South Yemen) or regional names such as cities, towns, or provinces (e.g., Algiers, Hong Kong).

We encourage to code for the specific Source Region, but it is optional. If you code it, use the text box to type in your answer.

Writing System The writing system used in the target language written text (e. g., Traditional vs. Simplified characters for Chinese; Gurmukhi vs. Shahmukhi for Punjabi).

This is another optional coding variable. Not many languages will need to code for this variable. If you have a different writing system used in the target language, please type it in the text box.

Let’s take a look at a sample in Domino.

Meltem - RC

Passage Variables: LC

Passage Variables

LC

ILR*

Setting* Formality*

FLO*

Topic* Passage description* Text type* Source type* Authentic-edited reason Source Country* Source Region Mode (TRE)

LC

Genre* Genre OTHER Planning Studio-recorded reason TL Speaker – Male TL Speaker – Female

Genre* Address | Lecture | Presentation | Speech Announcement | Notice Anecdote | Story Autobiography | Biography | Memoir Command | Instructions | Order Commercial Column | Commentary | Editorial | Opinion Piece | Op-ed Conversation Debate | Discussion Interview Report: News, Traffic, Weather Report: Other Review Voice Message Other Coding Options

Remember, like like in RC, if you choose “Other” for Genre, you will also need to type the Genre in the text box provided.

Source Type* Coding Format:

Coding Options:

Note to Coders:

If the option Authentic-Edited is chosen, code the next variable Authentic-edited Reason.

If the option Studio-recorded is chosen, code the next variable Studio-recorded Reason.

Authentic-edited Reason Studio-recorded Reason The way the selected passage was prepared as the test stimulus.

The source type is very similar to RC. But besides Authentic and Authentic-Edited, you have the “Studio-recorded” option. For both Authentic-Edited and “Studio-recorded, note that you also need to code the reason in the next variable. The coding variables for Authentic edited are the same as we had seen for RC (Abridging and limited editing). So, Let’s look at the Studio-recorded reasons on the next slide.

Studio-Recorded Reason Coding Format:

Coding Options:

To improve audio quality To improve the content To improve both audio quality and the content To construct non-scripted content To construct semi-scripted content To construct fully-scripted content Note to Coders: Code this variable only when the option Studio-recorded is selected from the variable source type.

A specific reason for constructing studio-recorded passages.

TL Speaker-Male*

Note to Coder: If no male voice, enter “0”; do not leave blank.

The number of male speaker(s) present in the target language passage.

For our listening passage, we need to code the number of male and female speakers. Please note that you must not leave the space blank, if for example, you do not have a male speaker. You have to write “0” in the text box provided.

TL Speaker-Female* Coding Format:

Note to Coder: If no female voice, enter “0”; do not leave blank.

The number of female speaker(s) presented in the target language passage.

Same rule applies for the number of female speakers.

Questions?

END of Day1

Before we continue to the item variables, lt’s see if there are any questions.

Let’s take a look at a sample in Domino.

SAMPLE : MCT LC Passage Coding

Item Variables – MCT

(LC)

4 as needed 1 TRE only 11 required 4 as needed 1 TRE only 12 required 4 as needed 1 TRE only 17 required 4 as needed 1 TRE only

LC-Audio distractor in the required information RC (12->9, setting, translation, target language removed) LC (17 ->11, setting, planning, translation, TL, LC orientation voice, audio removed)

Item Variables – MCT

MCT-RC & LC

Item-ILR* Negation in Stem* Open Stem* Required Information Location* Required Information Redundancy* Comprehension Type* Task Type*

MCT-LC

Audio distractor in required information*

CRT-RC

Item-ILR* Negation in Stem* Open Stem* Required Information Location* Required Information Redundancy* Comprehension Type* Task Type* Two part stem* KI required for credit*

CRT-LC

Audio distracter in required information*

Item-ILR*

The intended difficulty of correctly responding to the item, expressed as the minimum required examinee proficiency on the ILR scale.

CODING FORMAT: Drop-down List, Single Selection.

Coding options:

0+ 1+ 2+ 3+ 4+

NR

Negation in stem* Indication of the presence of any morpheme in the stem (question) that functions as a negating marker (e.g., Not, no, never, in-, un-).

CODING FORMAT: a check box.

NOTE TO CODERS: check the box only when the stem (question) includes a negative morpheme (e.g., Not, no, never, in-, un-).

Negation marker

Open stem* Indication about whether the stem (question) is an incomplete statement.

Example of an open stem:

According to the mother, her son likes to CODING FORMAT: Check Box.

NOTE TO CODERS: Check the box only when the stem (question) is an incomplete statement.

Required information location* Indication of the location of the required information*, which refers to information in the passage that is required for the examinee to understand in order to find the key to the question. Required information can be indicated by a direct mention, paraphrases, or presentations of clues adding up to infer the key information.

CODING FORMAT: Drop-down list, single selection.

CODING OPTIONS:

Required Information: the minimum amount of information required to get to the key correcty Beginning and Middle: both a repetition and/or in-between

Required information redundancy* The extent to which required information for the item is repeated within the passage.

CODING FORMAT: Drop-down list, single selection.

CODING OPTIONS:

Important aspect: how many times RI for the item is repeated for the length of a psg (e.g., for a very short psg, if RL was repeated twice, it could be coded as some redundancy but for a long psg, this could be coded as little redundancy.

Comprehension type* A broad classification of comprehension that is probed by the item.

Coding format: Drop-down List, Single Selection.

Coding options:

Coding definitions:

Literal: finding the correct answer to the item requires comprehension of information that is explicitly stated in the passage.

Inferential: finding the correct answer to the item requires comprehension in which examinees need to integrate information presented in the passage together with their schemata to infer meanings that are intended by the author/speaker in the given communicative context.

Note (No need to bring up level and comprehension): Inferential comprehension type is “theoretically” possible even for lower level items. For example, when a stem asks about the purpose of a passage and the purpose is not explicitly stated and needs to be” inferred” by combining several words to come to the correct answer.

Task type* The primary language comprehension task that is assigned to the item by the item developer.

Coding format: Drop-down List, Single Selection.

Note to coders: Select the option that most closely represents the comprehension task type assigned by the item developer.

Put yourself in the item developer’s shoes

Task type* (continued) Coding options: The item assesses examinees’ ability to Understand a single word or phrase (including ‘vocabulary in context’ item in RC at level 3 and above) Understand the general subject matter under discussion Understand a factual detail that is mentioned in the passage Understand a viewpoint, opinion, suggestion, or advice Understand sequence of events / sequencing a detail Understand causal relationship / cause and effect a detail Understand differences and/or similarities of events, issues, entities, viewpoints, etc. A detail or a viewpoint Differentiate facts from opinions N/A Understand the contextual meaning of words, phrases, or idioms a single word or phrase

1.Factual: removed (per the addendum) since “factual” was confusing as there could be inferential details in the passage or some coders were understanding “factual” as being synonymous with “scientifically true”.

2.Domino will still display all the old 16 options, since it is relational database it is not possible to delete/hide any options, as it will affect previously stored data.

3.Remind the audience they should use the 7 reduced task types. Once Domino templates are updated with the updated manuals in version 2, this will be resolved.

Task type*(continued) Coding options: The item assesses examinees’ ability to

Understand the reason/purpose of a given sociolinguistic and/or cultural reference or allusion the author’s purpose Understand the reasons/purposes of particular examples or illustrations as used by the author or speaker the author’s purpose Understand the author’s or speaker’s intended purpose Understand the author’s or speaker’s attitude about or tone on an issue Understand lines of argumentation supporting the author’s or speaker’s view/position a viewpoint Draw/reach appropriate conclusions by evaluating details and making a judgment Make a prediction based on information presented draw conclusions

Audio distracter in required information* (LC items only) Indication of the presence of any auditory distracter within the portion of the passage that discusses required information.

Coding format: Check Box, Multiple Selection.

Note to coders: Select all relevant options.

Coding options:

None White Noise Or Echo Non-human Noise (E.G., Music, Car) Non-speech Human Noise (E.G., Coughing) Inaudible Word Or Phrase Speech Overlap At Turn-taking (E.G., At Sentence Beginning Or End) Simultaneous Talk

Preferably there should be no audio distractor in Required Information for passages below L3.

Sample in domino

Questions?

END of Day1

Before we continue to the item variables, lt’s see if there are any questions.

SAMPLE : MCT Item Coding Let’s take a look at the sample in Domino.

image2.png image10.jpeg image11.png image12.jpeg image13.jpeg image14.jpeg image15.jpeg image4.jpeg image5.jpeg image6.png image7.jpeg image8.png image9.jpeg image23.jpeg image16.jpeg image17.png image18.png image19.png image20.png image21.png image22.png image24.png image25.png image3.png image26.png image27.jpeg image1.png