DLPT5 Fairness Review Guidelines_5July2022.docx

CUI

DLPT5

FAIRNESS REVIEW GUIDELINES

Test Review and Production Language Proficiency Assessment Directorate Defense Language Institute Foreign Language Center

Fairness and the DLPT5 The Language Proficiency Assessment Directorate (LPAD) of the Defense Language Institute Foreign Language Center (DLIFLC) considers test fairness to be one of the fundamental elements that support test validity. “Fairness, an essential quality of a test, has been broadly defined as equitable treatment of all test takers during the testing process, absence of measurement bias, equitable access to the constructs being measured, and justifiable validity of test score interpretation for the intended purpose(s) (AREA, APA, & NCME, 2014).”[footnoteRef:1] This document provides developers and reviewers with guidelines and recommendations for content review to ensure that the content in the Defense Language Proficiency Test 5 (DLPT5) is fair. That is, the DLPT5 does not present unintended biased views; does not discriminate between groups of examinees; and does not unnecessarily distract, anger, irritate, or frustrate examinees. [1: Banerjee, H. L. (n.d.). Tests Fairness in Second Language Assessment. Studies in Applied Linguistics & TESOL. Retrieved from https://academiccommons.columbia.edu/doi/10.7916/D8NG62KG]

The DLPT5 is a high-stakes test. LPAD’s objective is that each DLPT5 test measures key aspects of intended construct. It is the goal of LPAD to maximize the authenticity of content through the use of passages that reflect the characteristics of real-world written and spoken texts, while minimizing the negative impact of construct irrelevant factors. To accomplish this goal, LPAD strives to ensure that unfair content is detected and eliminated at all stages of test development. Concerns of fairness are addressed through all stages of development, including item development, item review, field testing, and test operation.

The guidelines in this document address the avoidance of cognitive and affective sources of construct-irrelevant variance in DLPT5 content. The guidelines are provided so that the DLPT5 neither advantages nor disadvantages any group of examinees. LPAD makes the final determination of content appropriateness based on the evaluation of qualified Target Language Experts and the professional judgment of test assessment specialists.

DLPT Fairness Review Guidelines Content and item fairness review is conducted by test assessment specialists in the Test Review and Production (TRP) division of LPAD. The guidelines in this document are to be adhered to by content developers, test assessment specialists in TRP. The primary guidelines for TRP test assessment specialists are as follows: avoid cognitive sources of construct-irrelevant variance and avoid affective sources of construct-irrelevant variance.

1. Avoid cognitive sources of construct-irrelevant variance

· Fairness in English language use Fairness in language use refers specifically to fairness of the components of an item that include the orientation statement, stem, and options. The level and complexity of English used in DLPT5 items should not negatively impact an examinee’s ability to demonstrate their comprehension of the foreign language passages.

The intention of this review is to ensure that:

· English language used on the DLPT5 is Standard American English.[footnoteRef:2] [2: Standard American English in this document refers to the form of the English language used in the United States, as taught in schools and heard on newscasts, adhering to fixed norms of spelling, grammar, and usage in written and spoken contexts, and neutralizing nonstandard dialectal variation.]

· English vocabulary is accessible to DLPT5 examinees and does not require specialized content knowledge to identify the correct answer.

· Structure in the Stem, Key, and Distractors uses simple syntax to convey the task.

· Fairness in diversity of topics, sources and viewpoints

Fairness in diversity of topics, sources and viewpoints means that no topic, source or viewpoint is more prominent than another.

The intention of this review is to ensure that there is:

· Diversity of topics

The content of DLPT5 passages falls under the following five Final Learning Objectives (FLOs) content areas:

· Social and cultural

· Political and economic

· Scientific and technical

· Military and security

· Physical, political and economic geography

· Diversity of sources and text types

The content of DLPT5 passages is a representative sample of the target language as used by native speakers of that language. The majority of the content of the DLPT5 is derived from sources such as newspapers, magazines, books, TV/radio programs, personal correspondence, text messages, public announcements, the Internet, social media, etc., to represent the various text types and discourse patterns typically found in communication among native speakers of the target language.

· Diversity of viewpoints

The DLPT5 passages are selected from sources that are accessible to the general public in the target language speaking community, represent a wide variety of viewpoints, and do not intentionally favor one viewpoint over another. The associated items do not unfairly advantage or disadvantage any group of examinees by targeting information that would be more familiar or interesting to a certain gender, group, or affiliation.

2. Avoid affective sources of construct-irrelevant variance Test content on the DLPT5 should not be unnecessarily offensive to any examinee. The content of the tests should not unnecessarily evoke strong negative emotions or distract an examinee from the task of understanding a stimulus or responding to an item. All test content is evaluated and reviewed by assessment specialists in TRP.

The following list is illustrative of topics that might contain content that could evoke strong negative emotions on the part of the examinee. Graphic accounts of the topics below should be avoided.

· Abortion

· Abuse of people or animals

· Any graphic accounts of accidents, illnesses, natural disasters, or death and dying

· Atrocities or genocide

· Child marriage

· Contraception

· Controversial gender issues

· Euthanasia

· Experimentation on human beings or animals that is painful or harmful

· Honor killings

· Rape

· Religion (material should be as objective as possible)

· Substance abuse (avoid a focus on details of substance abuse)

· Suicide

· Torture

5 July 2022

CUI 1

image1.png