Attach 7_Standard Setting Study Plan-Example.pdf

Developing a Plan for Setting Proficiency Standards for the DLPT5

Draft Plan

In Response to:

Contract M67004-05-D-0009

Delivery Order 51

Submitted to:

Dr. Donald C. Fischer, Provost Defense Language Institute

Daniel Segall, PhD

Defense Manpower Data Center

DoD Center Monterey Bay 400 Gigling Road

Seaside, CA 93955-6771

Prepared by:

Gregory J. Cizek, PhD

Barbara S. Plake, PhD

October 13, 2009

I. Background

The Defense Language Proficiency Tests (DLPTs) are batteries of tests designed to assess proficiencies in reading and listening in a number of foreign languages. The DLPTs are used by the Department of Defense and other agencies for a variety of purposes, including determining individual qualification levels and pay, organizational readiness, individual graduation from foreign language training at the Defense Language Institute Foreign Language

Center (DLIFLC), qualification for follow-on linguist occupations, and other purposes.

Recently, new editions of the DLPTs (hereafter, DLPT 5) have been introduced for many languages. At its March 2009 meeting, an advisory committee for the DLPTs, the Defense

Language Testing Advisory Board (DELTAB), recommended that it would be appropriate to conduct a review of the procedures used to set cut-scores on the DLPT 5 examinations. In July

2009, the Defense Manpower Data Center (DMDC) executed a Deliver Order with nationally-recognized specialists in standard setting, Dr. Gregory J. Cizek and Dr. Barbara S. Plake, to perform a review and to develop a plan for conducting standard-setting procedures for the DLPT examinations. Biographical information for Drs. Cizek and Plake is included as Appendix A to this document.

On August 17-18, 2009, Drs. Cizek and Plake met with DLIFLC and DMDC representatives in Seaside, California to perform a preliminary step specified in the Delivery

Order—that of gathering information about the DLPT examinations, the existing standard setting methodology, and concerns of the DLIFLC and other stakeholders with regard to the setting of cut scores on the examinations. A pre-meeting planning teleconference was held on July 30, 2009. Additionally, in preparation for the meeting, DLIFLC provided substantial assistance in terms of background material and documentation. A list of the documents reviewed prior to the meeting is provided in Appendix B. Appendix B also includes documents provided at the August

17-18, 2009 meeting. Following execution of a test security agreement by Drs. Cizek and Plake, DLIFLC personnel supervised a review of actual DLPT 5 reading and listening test forms and a demonstration of the computer interface used for test administrations. Finally, review of an initial draft of this report by DLIFLC personnel occurred and suggestions for changes and clarifications occurred in a teleconference held on October 5, 2009. The authors of this report gratefully acknowledge all input, suggestions, and clarifications.

The following sections of this draft report address nine topics as specified in the Delivery

Order Statement of Work: 1) Discussion of Strengths and Weaknesses of the Current DLPT

Standard-Setting Approach; 2) Choice of Standard-Setting Panel Members; 3) Performance

Level Descriptions; 4) Choice of Standard-Setting Process Facilitators; 5) Recommended

Standard-Setting Method, Panelist Training, and Panelist Discussion/Deliberation Protocol; 6)

Information Sources for Standard Setting; 7) Evaluating the Standard-Setting Process; 8)

Validity Evidence and Technical Documentation; and 9) Applying Standards to Future Test

Forms and to CAT Versions of the DLPT5. A final section provides a summary list of recommendations. References, figures, and appendices appear at the end of the report.

In evaluating the current standard setting process for the DLPT 5 examinations and in developing recommendations for the future, the authors of this report were guided by psychometric best practices and established professional guidelines including, primarily, the

Standards for Educational and Psychological Testing (1999), a joint product of the American

Educational Research Association, the American Psychological Association, and the National

Council on Measurement in Education. Finally, for purposes of this report, we note the following definitions:

Content Standards: A collection of statements that express the knowledge, skills, and/or abilities that are intended to be included in a curriculum, the focus of instruction, and assessed by an examination.

Performance Level: A category of performance on an examination, typically denoted by brief verbal or quantitative labels. Examples include Pass/Fail, Basic/Proficient/Advanced and, on DLPT examinations, 0, 0+, 1, 1+, 2 and so on.

Performance Level Description (PLD): A (typically) brief statement describing the general knowledge, skills, and abilities of examinees falling within a given performance level.

Performance Standards: The minimum level of performance on an examination required for an examinee to be categorized into a given performance level. According to the Standards for Educational and Psychological Testing, “A critical step in the development and use of some tests is to establish one or more cut points dividing the score range to partition the distribution of scores into categories.... [C]ut scores embody the rules according to which tests are used or interpreted. Thus, in some situations, the validity of test interpretations may hinge on the cut scores”

(AERA/APA/NCME, 1999, p. 53). Performance standards may be expressed in a raw score metric (e.g., number correct), an item response theory metric (e.g., theta level), or another metric (e.g., transformed or scaled scores). Although subtle and sometimes important distinctions can be made (see Cizek, 2006; Kane 1994), the term cut score is used interchangeably with performance standard in this report.

Standard Setting: The systematic process of deriving performance standards for an examination.

Finally, we note that, although the term standard setting is use throughout this report, it is most accurate to say that the participants in a standard setting study serve as “standard-recommenders.” The results of a standard setting procedure are one or more cut scores, recommended by the panelists to the entity with the authority and responsibility for the examination at hand. The responsible entity may, defensibly and appropriately, accept, reject, or modify the panelists’ recommendations.

II. Strengths and Weaknesses of the Current Standard-Setting Approach

Currently, cut-scores for Levels 1, 1+, 2, 2+, and 3 on the DLPT5 examinations are set using an empirical approach that is based on performance on validation items (i.e., items that are administered to volunteer or non-motivated candidates on a tryout basis). Items are developed to elicit the characteristics of each of these levels and go through an extensive review process to verify they are targeted at the intended performance level. Once these items have been administered and calibrated using a three-parameter logistic (3PL) item response theory (IRT) model, test characteristic curves are generated using, sequentially, only items that are targeted for a particular level, starting with Level 1. The items used to develop each item-level test characteristic curve are de-selected as needed so that the final set of test characteristic curves for each level shows a progression of locations on the ability continuum that are non-overlapping and non-crossing, creating a picture of a true Guttman scale. Once these test characteristic curves have been finalized, the cut-score for each level is set at the theta value on the ability continuum where the probability of obtaining that theta value is .70. These theta values are then translated onto the number correct scale by applying the item parameters for the selected items with theta value for each respective cut-score, and then truncating any decimal value and adding one (always ensuring that the cut-score meets or exceeds the converted number right value). A different approach is used for setting the 0+ cut-score. For this cut-score, examinees whose performance indicates they are randomly guessing (using the sum of the c parameters from the

3PL IRT model) are identified. The distribution of their obtained scores on the assessment is determined and the 80th percentile of that distribution is used as the cut-score for the 0+ level.

There are several features of this method that are noteworthy. For one, it relies on examinee data so it is reasonably easy to compute. In addition, once set, these values can be equated using IRT true score equating to set the cut-scores on other versions of the examination without having to redo these computations.

There are however, some concerns about this method for setting cut-scores on the

DLPT5. First, the analyses depend on the estimates of the item parameters from fitting the data to a 3PL IRT model. This means that the quality and integrity of these item parameters are critical to the validity of the cut-scores. There are two concerns with these parameter estimates.

First, they are based on potentially unmotivated examinees who also may not have had the same curriculum as implemented at the Defense Language Institute (DLI). Therefore it is not clear what the motivation, preparation, qualifications, or representativeness of these examinees are when they were administered the test. The second concern has to do with the stability of the item parameter estimates, particularly for the a (discrimination) and c (pseudo-guessing) parameters since the sample sizes upon which these item parameter estimates are based typically do not exceed 250 – 300 cases. It is generally recommended that examinee sample sizes that exceed 1000 are needed for use of the 3PL IRT model, with some recommendations as high as

3000 for good estimates of the c parameter.

Assuming that there were accurate estimates of the item parameters on which to base the computation of the cut-score, a second concern has to do with the ability of the individual items in the assessment to validly represent the characteristics of the targeted ILR level. The framework document provides test developers with these defining characteristics and each item once developed goes through a review process where two independent raters assess whether the item does indeed elicit the intended characteristics of that ILR level. Disagreements are adjudicated and feedback is provided to the item writers on what changes are needed to ensure the items is situated property on the ILR scale. Whereas correct classification of the items to the

ILR scale categories is identified as a potential concern, the procedures put into place in the item development and review process make this concern less of an issue. However, it is clear that since the current standard setting method is 100% tied to the items in the item bank and their ILR level assignments as well as their IRT parameters, concerns about the coding of the items to ILR levels and the items’ IRT parameters need to be in the forefront when reviewing the appropriateness of this method for setting cut-scores for the DLPT5 examinations.

Another concern with this method is the .70 criterion for setting the cut-scores on the theta scale. Although the decision to set this value for the criterion appears to be a compromise among a group of people who debated whether it should be set at .80 or .70, there is no record that the decision was explicitly grounded in consideration of the particular knowledge, skills and abilities (KSAs) of examinees taking the test. Rather, it appears to be based on the notion the

70% is typically used as the passing score on academic assessments, but without regard to the difficulty of the questions comprising the test or of the KSAs of the target examinees. Further, it appears that the choice of 70% was made based on the performance level requirement for examinees who are within a performance category (e.g., Level 2+) as opposed to examinees who were just at the borderline of a performance category (just at the 2/2+ borderline, for example).

Current best practices in standard setting indicate that cut-scores should be set to distinguish examinees who are just entering a category in transition from the previous category (such as just entering Level 2 from previous performance as a Level 1+).

Overall, it should be noted that, theoretically, there is really nothing inherently wrong with the underlying conceptualization of the current method; it is well-known that all standard setting is, to some extent, a judgmental process. On the other hand, the current method is not one that is found in the standard setting literature, and it is not among the methods that have been documented as a defensible standard setting method. In addition, as currently implemented, some concerns about the current method are apparent. Under ideal circumstances, with appropriately coded items tagged to their ILR levels and accurate estimates of their item parameters, level-specific test characteristic curves can be generated and the items used to generate these curves trimmed to create curves that support the notion of a Guttman scale as the underlying scale for the assessment. Next, with appropriate deliberations by representative stakeholders and test users, decisions could be made about the criteria for setting the cut-score thresholds. This would need to be determined with a consideration of the tasks that comprise the test characteristic curves for each level and may not yield the same criterion for each of the respective ILR levels. Further, these decisions should be made by considering performance on these items by examinees who are transitioning into that performance category, sometimes called the “borderline” examinee for each of the performance categories.

Because of these shortcomings of the current method, it is recommended that a well-established, documented, and researched standard setting method be used to set the performance level cut-scores on the DLPT5. Such a method would be sensitive not only to the difficulty and demands of the items that comprise the test but also set the performance cut-scores with the knowledge, skills, and abilities of the borderline candidates for each of the ILR performance categories and that are set with the input of important stakeholders and test users. The use of such a method would allow DLPT5 stakeholders and users to have more confidence that these cut-scores are appropriate for their respective uses and yield valid and defensible cut-scores.

III. Choice of Standard-Setting Panel Members

Participants in the standard setting process are critical to the success of any standard setting process. In a general sense, the most appropriate group of participants to empanel would be a sufficiently large and representative sample of the population of possible participants that meet whatever criteria the responsible entity has established as defining “qualified” for the standard setting task. The Standards for Educational and Psychological Testing

(AERA/APA/NCME, 1999) provide some guidance on representation, selection, and training of participants (called “judges” in the Standards). For example, regarding the number and characteristics of participants, the Standards indicate that “a sufficiently large and representative group of judges should be involved to provide reasonable assurance that results would not vary greatly if the process were replicated” (p. 54). The Standards recommend that “when a validation rests in part on the opinions or decisions of expert judges, observers or raters, procedures for selecting such experts and for eliciting judgments or ratings should be fully described” (p. 19) and that “the qualifications of any judges involved in standard setting and the process by which they are selected” (p. 54) should be fully described and included as part of the documentation of the standard setting process.

One key finding resulting from fact-finding discussions with personnel associated with

DLPT examinations was that there are several stakeholder groups that must be represented on any standard-setting panels because of the diverse uses that are made of DLPT test scores and because of the important perspectives the various stakeholders bring to bear. Representation of and participation by important stakeholder groups is essential to the success of the standard setting procedure and credibility of its results. Although the identification of stakeholder groups and precise relative proportions of representation are a policy matter, relevant stakeholders that should be sampled for participation in DLPT standard-setting panels would be fluent in the language covered by an examination and would likely include, but not be limited to, representatives of:

* the Defense Language Institute;

* the Defense Language Office;

* the Defense Language Steering Committee;

* the Defense Intelligence Agency;

* the National Security Administration;

* field commanders who utilize linguists subject to DLPT examinations; and

* military language instructors.

Whereas the issue of representativeness just described is the most fundamental concern when choosing standard-setting panel members, another consideration is the number of participants that should be empanelled for each DLPT examination. Research and practical experience in the area of standard setting indicate that the choice of how many panelists is a balance between two factors. On the one hand, the number of panelists should be large enough to provide stable (i.e., reliable, reproducible) estimates of the cut scores. On the other hand, the panel should be of a size that permits full discussion, free interaction among panel members, and adequate facilitation. For standard setting on DLPT examinations, it is recommended that panels for each language examination consist of 16 to 24 members. A fairly large number (i.e., 16 to 24) of participants is highly desirable to foster stability of the eventual recommended cut scores. It is a statistical truism that a larger sample (in this case, of participants) provides a more stable estimate of a desired index (in this case, the cut scores). Reducing the possibility of obtaining cut scores that were “off” simply due to small sample sizes seems to be an appropriate goal of configuring panels. In addition, it is desirable to obtain estimates of reproducibility (reliability) of cut scores in standard setting results. For that reason, a large group of participants is typically broken down into sub-panels. After whole-group training and practice, each of the sub-panels then works independently, arriving at their own recommended cut scores. In this way, variability across independent panels can be assessed. However, to obtain the desired statistical stability, the actual overall recommendation is based on the combined judgments of both panels. For standard-setting on the DLPT examinations, it is recommended to empanel 16 to 24 participants for the large group, with the rationale that 16 to 24 would be sufficiently large to provide reasonable stability.

The large group of 16 to 24 participants would receive common orientation, training, and so on, but would then divide into independent sub-panels of 8 to 12 participants. Such a configuration would lead to stable estimates of the cut scores (based on all 16 to 24 participants), but would also be amenable to effective facilitation and would permit analysis by sub-panel to evaluate replicability of results. Such a configuration would also be well-aligned with the guidance provided by the Standards for Educational and Psychological Testing which recommend that analyses and documentation for standard setting activities “should indicate whether participants reached their decisions independently, and should report the level of agreement reached” (AERA, APA, NCME, 1999, p. 19). Finally, as a first step in the process of establishing new cut scores for the DLPT examinations, it is recommended that policy decisions be made regarding a final list of stakeholder groups that should be represented on the standard-setting panels and the relative proportions of each group that are desirable. It is also recommended that such a policy be considered for application to the empanelling of participants to be selected for developing the Performance Level Descriptions, described in detail in the following section of this report.

IV. Performance Level Descriptions

Nearly all of the most common standard setting methods rely on experts to make judgments about how examinees will perform on test items or on a test as a whole. There are two referents for these judgments. One referent is a hypothetical group or individual test taker that possesses specific characteristics; most commonly, the characteristics are those of examinees who are just at the borderline between two performance levels, say, on the borderline between 2 and 2+ on DLPT examinations. Examinees at this point are sometimes referred to “borderline” or

“minimally proficient” examinees. The standard setting activity recommended here will include activities in which panelists help to delineate the knowledge, skills, and abilities that should be possessed by examinees at each borderline (that is, at each cut score level).

The process of describing the hypothetical borderline examinee at each performance level necessarily begins with the performance level descriptions (PLDs) of each of the performance levels. These are already available and are provided for reading and listening by the Interagency

Language Roundtable (ILR; see Interagency Language Roundtable, a; b). However, it is important to note that the ILR PLDs describe the performance of examinees within a performance category. What are needed are descriptions of examinees at the borderline between adjacent categories. Such examinees will, by definition, possess just enough knowledge, skill, and ability to be classified within a category, but only minimally more than examinees at the highest level of the lower category. These borderline PLDs (hereafter, BPLDs) will be used by panelists in the course of the actual standard setting procedure; however, but the required BPLDs do not currently exist. Thus, another essential recommendation is that they be created in advance of the operational standard-setting study.

It is recommended that a group of qualified experts and stakeholders be empanelled to create the BPLDs as a separate, prior activity. It is recommended that this group not be the same persons as those who will participate in the actual standard setting (see Section III), although the same interests, expertise, and stakeholder groups should be represented. Separate BLPDs would be created for reading and listening for each DLPT language, although the borderline descriptions would be generic as regards specific languages and thus useful across DLPT standard setting activities. The development of “generic” BPLDs would lend an overall consistency to the subsequent standard setting meetings for each language covered in DLPT examinations. The coherence of cut-scores across language examinations would be seriously threatened if, for example, an Arabic BPLD group drafted much more rigorous BPLDs than a

Chinese BPLD group. On the one hand, generic BPLDs would have the advantage of providing a common thread for the subsequent operational standard setting studies, and generic BPLDs would help maintain the policy intention of the stakeholders who helped define the descriptions of acceptable performance at each ILR level. On the other hand, in some cases it may be unavoidable that language-specific knowledge, skills, abilities, or other characteristics will need to be embodied in BPLDs for them to provide clear and adequate guidance to those who serve on operational standard setting panels. Thus, in addition to developing the generic set of BPLDs using the process described in this section, it will be important to subsequently “flesh out” the

BPLDs in a language-specific manner as one of the first activities in the operational standard setting studies for each language (see Section VI).

It is recommended that the BPLD development activity would take place as a separate meeting; the activity would not require more than 1.5 days to complete and would result in clear operationalizations of borderline performance for use during all subsequent standard setting meetings for DLPT languages. BLPDs would be created for each of the six borderline groups

(e.g., the 0/0+ borderline, the 0+/1 borderline, the 1/1+ borderline, etc.) and each would not exceed approximately one-paragraph in length. Optimally, the BLPD development meeting would be conducted by the same facilitators who would then conduct subsequent standard setting meetings.

V. Choice of Standard-Setting Process Facilitators

Facilitation is a critical aspect of any standard setting procedure. Facilitation for a standard setting meeting involves the overall oversight, direction, and management of the process by one or more individuals who have specialized training and experience in the required tasks. Skilled facilitators enable the standard setting meeting to progress smoothly and on schedule; they manage difficult situations or unexpected events that arise during the meeting;

they ensure that all panelists understand the standard-setting tasks and engage in them appropriately, they foster full and complete participation by all participants in discussions and deliberations; and they coordinate evaluations of the procedures, analyses of the data generated by participants, and the provision of feedback information to panelists. Standard setting facilitators must strike a delicate balance to ensure that they direct the activities of the standard setting meeting, but do not direct the results. As Hambleton (2001) has noted, effective facilitation “seems obvious to the validity of the overall process” (2001, p. 111).

For the standard setting on the DLPT examinations, it is recommended that the actual standard-setting meetings be conducted as face-to-face meetings, as opposed to meetings conducted remotely. This recommendation is based on the fact that nearly all of the studies reported in the professional literature have used a face-to-face configuration. Although there is little in the professional literature regarding remote conduct of standard-setting studies, it is recognized that the potential for conducting studies remotely is likely to increase. A remote study is not proposed for the DLPTs in part because, while an adequate research base exists for traditional methods (i.e., in person), it is uncertain how a remote methodology could affect the results of the study. Because of the high-stakes nature of the DLPT examinations, it would seem prudent to adhere as strictly as possible to well-evaluated approaches. Finally, in the proposed approach, panelist interaction would seem to be critical, especially in the crafting of BPLDs and in the consideration of other participants' ratings, feedback, impact information, and in the general discussions. As well, if a remote methodology were used, the ability of facilitators to ensure the integrity of the process would be reduced, and it is likely that the interactions, discussions, etc., would not be as rich as would be desirable.

It is not necessary—indeed, it is probably not desirable—for facilitators of standard-setting activities to have expertise in the content area covered by the examination for which performance standards will be set (in this case, foreign languages). However, facilitators of standard-setting procedures must possess other characteristics that are vital to the success of the standard setting procedure. For the DLPT standard-setting workshops, it is recommended that highly experienced facilitators should be engaged. Facilitators for the DLPT standard-setting procedures should have background in psychometrics; expertise in standard setting; expertise in developing and making presentations; a high degree of competence and experience in managing group work and discussions in standard-setting contexts; and the ability to respond to changing circumstances or challenges as they may arise in group settings. It is also recommended that a similarly-qualified facilitator be engaged to lead the meeting at which the BPLDs will be developed.

Assuming the operational standard-setting configuration described in this report, it is recommended each standard-setting procedure for each language examination be led by two facilitators with the qualifications just described, and that a resource person who is fluent in the language that is the focus of the particular standard-setting study be assigned to each standard setting meeting. For the success of each standard-setting meeting, it seems essential that at least one person involved in conducting the meeting possess fluency in the target language. If it is possible to locate two facilitators who have fluency in the target language of the examination, it would be possible to conduct the studies with only the two facilitators. The facilitator must be able to lead (and hence, understand) the issues that arise in group discussions about the difficulty of an item, the content of the items, the language-based reasons for divergent judgments, and so

on. Because it appears that it would be difficult (impossible?) to identify two language-fluent facilitators with expertise in standard setting, it is instead recommended that one resource person with language fluency be assigned to each standard setting study to serve as just described. It is likely that one such person could “float” between both groups to address language-related issues.

VI. Standard-Setting Method, Panelist Training, and

Panelist Discussion/Deliberation Protocol

When choosing a standard setting method, it is important to consider many factors, including the format of the examination (e.g., multiple-choice versus constructed-response or performance item formats), the stakes associated with setting a cut-score, and other contextual and logistic factors. Based on a review of these aspects of the DLPTs, it is recommended that a variation of the Angoff (1971) standard setting method be considered. The variation is called the

Yes/No method, and it has been described in the psychometric literature (see Impara & Plake, 1997) and used successfully in a variety of contexts.

Unlike the Angoff method, where panelists in the standard setting study make estimates of the proportion of borderline examinees who will likely answer each item correctly, in the

Yes/No (Y/N) method, panelists simply estimate whether the borderline examinees are likely to answer each item correctly or not (Yes or No). The Y/N method is particularly appropriate in situations where there are hierarchical levels of performance, such as is the case with the ILR performance scale. For example, once it is decided that an examinee who is at level 1 on the ILR scale will likely answer an item correctly, it is axiomatic that any examinee who is at a higher performance level on the ILR scale will also likely respond correctly. This feature of the Y/N method reduces the burden on the panelists for the number of independent decisions that need to be made for each of the items that comprise the test. If an Angoff method was used, each panelist would need to make 6 different (but monotonically increasing) estimates of the proportion of borderline examinees who would answer the item correctly (that is, one each for

0/0+, 0+/1, 1/1+, 1+/2, 2/2+, and 2+/3). With the Y/N method, instead the panelists would need only to signal when the examinee’s likelihood of answering the item correctly transitions from

No to Yes; from that point forward the answers would all be Yes and from that point back down the scale it would always be No.

We considered recommending the use of the Y/N method only to set cut-scores for the base ILR levels (i.e., 1, 2, 3) and the use of interpolation to derive the intermediate + levels (i.e., 0+, 1+, 2+). An approach that used interpolation to generate + levels might be defensible if there weren’t important stakes tied to the classification of examinees into a performance level, but was deemed as merely “moving toward the next ILR level” or as “high performance within an ILR level.” Additionally, for an interpolation approach to be defensible (i.e., so that “60%” or some other percentage would not be seen as arbitrary), some systematic process would need to be followed to arrive at the appropriate percentage; in effect, a standard setting would need to be performed to justify the choice of 60% as having meaning beyond an arbitrarily chosen value.

Thus, it is our strong recommendation that BLPDs must be created, and a rigorous, defensible standard setting procedure should be used to establish cut-scores for plus levels because of the importance placed on some + levels; for example, if plus levels are used for FLPP decisions, if plus levels are used for promotion decisions, if plus levels are used to identify some test takers as “more proficient” than other test takers at the “whole” level and, of great relevance, if a graduation criterion is raised from a 2 to a 2+.

The Y/N method is appropriate for these contexts and purposes because it seeks to identify how target groups of examinees (i.e., those at each of the transition points identified above) are likely perform on the test. Therefore it is sensitive to both the questions on the test and to the knowledge, skills, and abilities of the examinees at each of the transition points.

Because of the Guttman-like properties of the ILR scale, with the language ability systematically increasing as performance moves up the ILR scale, this method is uniquely appropriate for setting performance standards on the DLPT5. Panelists at the standard setting study would be asked to make Yes/No decisions for the performance level transition points for each item in the

Reading and Listening tests. Cut-scores would then be computed, as noted below, based on the panelists’ final ratings, for the Reading and Listening tests. These values would form that basis of the group-recommended cut-scores that would then be communicated to a policy body to make the final cut-score decisions for that language’s DLPT5 Reading and Listening components.

A standard setting study would be conducted for each language that has a DLPT5 assessment for Reading and Listening. Each standard setting study would be comprised of five major elements: Orientation, Tailoring of the BPLDs to the language of interest, Training, Operational Ratings, and Evaluation. The following paragraphs describe each of these elements in additional detail.

The first element in each standard setting study is the Orientation where the panelists are given an overview of the purpose of the standard setting study and provided with the generic

Borderline Performance Level Descriptors (BPLDs) that were developed independently by another panel prior to convening the standard setting study. In addition, the orientation will give the panelists a brief overview of the agenda for the process and a conceptual understanding of the standard setting method. The panelists will also be given an opportunity to experience the test in quasi-operational conditions, preferably in the administration mode used for operational delivery, e.g., computer delivery.

Following Orientation, the panelists will engage in the second element of the standard setting study, tailoring the BPLDs to the language of the test. Prior to the standard setting activity, a group of stakeholders will have met to generate generic BPLDs that extend the characteristics identified by the ILR level descriptions to those individuals whose knowledge, skills, and abilities are at the borderline for each of the performance thresholds (0/0+, 0+/1, etc).

The purpose of this activity is to flesh out the generic BPLDs as they apply to the language under consideration for the standard setting study. Panelists will be divided into six groups, one for each of the six cut-points, and will be directed to amplify the BPLDs for the cut-point they have been asked to consider for both Reading and Listening. Once each of these subgroups has completed their work, the panelists as a whole group will review each others’ work to develop a consensus document about the knowledge, skills and abilities, as defined by the BPLD, for their language area. This document will be compiled and distributed to the panelists for their use in making their item level judgments as they complete their ratings. This document will also be included in the technical report from the standard setting study.

The third element of each standard setting study is Training in the selected method. This will include a practice task where the panelists will apply the method to items similar to the ones on the operational tests. They will get experience with making the item-level judgments of the transition point on the ILR scale where the performance of examinees changes from an expected

“no” for getting the item right to an expected “yes”. In addition, they will have practice in interpreting the kinds of feedback that will occur during their operational ratings, including normative, reality, and (if deemed to be appropriate) impact feedback types. In the practice setting, panelists will have the opportunity to ask questions, discuss the nature of their rating decisions, and learn about the format and intended interpretations of the feedback. Experienced facilitators will move thoughtfully and purposely through the training, ensuring that all panelists are comfortable with the tasks that are to be completed when they move from practice to their operational ratings.

The role of guessing will be addressed in the training (and, optionally, via a proposed statistical process described later in this section). In a high stakes test, research has shown that candidates rarely randomly guess, especially in tests that have strong item development procedures where the incorrect options are designed to tap common, anticipated weaknesses in candidate knowledge. During training, panelists will learn about what characteristics of a multiple-choice item affect the likelihood that candidates will be able to answer correctly, including the influences of a) the difficulty of the questions, b) similarity of the incorrect options, and c) the likelihood of getting the item correct by guessing.

The next element in the standard setting study is the Operational Ratings. For this component, panelists will be divided into two groups of 8–12 each. These groups will be led by trained facilitators. The total group will be split into two groups in order to allow for better group dynamics and to enable better discussions among group members. In addition, by having two groups, it will be possible to estimate how stable the cut-scores estimates are for randomly equivalent panels, an important criterion for the technical quality of the recommended cut-scores.

Operational ratings will be performed once the training has been completed, and they will be done separately for Reading and Listening. First the panelists will work with the Reading assessment. The ratings will progress through three iterations (or “Rounds”), with panelists receiving feedback between rounds of ratings. Panelists will work independently, progressing through the test item-by-item, making their Yes/No ratings on a specially designed rating form

(see Figure 1 for an example rating form).

Once the panelists have completed their ratings, these values will be analyzed to compute, for each panelist, recommended cut-scores for each of the six cut-points. This computation is the sum of the “yes” ratings across the items for each cut-point. Traditionally, the cut-scores from the Y/N method for each round are derived by obtaining the average (or median) anticipated number correct across all the items in the test form for each of the cut-points. To illustrate this process, it is easiest to envision a matrix that has six columns (one for each of the cut-scores) and k (number of items in the form) rows. There will be one such matrix for each panelist. Once a panelist decides where the cut-point transition is for getting an item correct, that transition cut-point column would be recorded as a 1 and all the cut-point columns to the right would also be recorded as a 1. All cut-point columns to the left of the transition cut-point would be recorded as a 0. The cut-scores for the panelists are determined by summing over the rows for each cut-point. These cut-points would be calculated for each panelist and then averaged over the panelists to determine the group recommended cut-scores for that round. (Depending of the distribution of group cut-scores, the median value may be used instead of the average).

As indicated previously, it may also be desirable to address concerns about guessing via a statistical adjustment. As an alternative for calculating the cut-points, the following procedure could be considered. (We note that this procedure has not, to our knowledge, been documented in the psychometric literature. However, we believe that the procedure is based on sound principles and is responsive to an important concern about false positive categorical assignments due to guessing.)

The procedure would begin by obtaining the cut-scores for each level in the manner just described. Then, a unique adjustment value would be added to each of the cut-points. We recommend that this additional value be conditional for each of the six cut-points. The value would be calculated based on the probability of answering correctly the remaining items in the test beyond the respective estimated cut-point. Because a 3PL IRT parameter model is used for the DLPT testing programs, every test item has an estimated c parameter that indicates the probability that an examinee would be able to answer the item correctly by guessing. A

“guessing-adjustment value” would be determined by multiplying the remaining number of items in the test by the average pseudo-guessing parameter for the items that comprise the test.

As a hypothetical example to illustrate the use of the adjustment, suppose that the result of applying the Y/N method resulted in recommended cut-scores of 16 for the 0+/1 level and 20 for the 1/1+ level. (Two cut-scores are used here for illustration; the same procedure would also be used for the remaining levels.) Further suppose that the mean c parameter estimate for items in the test was .18. Now, using the borderline 0+/1 level cut-score of 16 items (out of 50) results in 34 remaining items for which examinees would have some probability of responding correctly by guessing. Multiplying the 34 remaining items by the guessing-adjustment constant of .18 equals 6.12. The guessing adjustment of 6.12 would then be added to the base 0+/1 recommended cut-score of 16 to yield a total, guessing-adjusted cut-score recommendation of

22.12. Likewise, for the 1/1+ level, suppose that participants judged that the borderline 1/1+ examinees would be able to answer 20 (out of 50) items correctly. That leaves 30 items for which examinees would have some probability of responding correctly by guessing. Multiplying the 30 remaining items by the guessing-adjustment constant of .18 equals 5.40. The guessing adjustment of 5.40 would then added to the base 1/1+ cut-score of 20 to yield a total, guessing-adjusted cut-score recommendation of 25.40. Once these cut-scores have been calculated, a policy decision will need to be made about converting the cut-score recommendations from the standard-setting meeting to raw score values for the first operational use of the new performance standards. Possible policy decisions could involve truncating, traditional rounding, or adopting the next higher integer value.

The proposal to use a guessing-adjusted value recognizes at least three caveats. First, it is likely that the average c parameter varies for examinees at the various ILR proficiency levels, so a conditional (on ability level) average c-values could be calculated. However, computing conditional average c values does not seem feasible given the sparse data at some levels and instability of the c parameter estimates due to both small sample sizes and few numbers of items on which the conditional averages would be based . Thus, it is proposed that the average c parameter value across all items in the test form used for standard setting could serve as an overall estimate of the propensity for candidates to get the items in the test correct by randomly guessing. Second, although it is possible that guessing-adjustment values may vary if computed separately for each language examination, it is recommended that a single constant be obtained and applied across all language examinations. Further, it is recommended that the single value be derived from the language area for which the greatest number of representative examinees was available for computing item parameters. This recommendation may have the effect of biasing the guessing adjustment slightly for some languages, but we believe that any small bias introduced is preferable to the comparatively larger errors that would likely be introduced if the adjustment was derived from small or non-representative samples from low-volume examinations. (If this option is pursued, it might also be advisable to compute guessing adjustment constants for the two highest volume examinations as a way of assessing the degree of bias.) Finally, it is recommended that the results from both approaches to computing the cut-scores (i.e., the traditional Y/N method and the Y/N method with guessing-adjustment approach described here) be reported to the policy body for their use in making the final cut-point determinations.

The test form that is used by the standard-setting panelists should be a form that either currently exists or is one that is created to be representative of the table of specifications in terms of content and statistical properties. In addition, it is important that panelists view the test items on which participants will be making judgments in the same way that examinees experience the items. Thus, it is assumed that panelists will be viewing the test online. However, unlike an operational test administration, the test items should not be presented in sequence by ILR level.

Instead, the items should be presented in a scrambled order to avoid any cluing that might exist if the items were presented in ILR order. Further, it is recommended that the DLPT computer interface be modified to allow panelists to enter their item ratings via the computer as they progress though the items in any given test form. This procedure would eliminate the need for the panelists to coordinate their viewing of the items on the computer with filling out a separate paper-based form to enter their ratings. The form shown in Figure 1 could be readily adapted for the interface or, if modification of the interface is not practical, a form such as that shown in

Figure 1 could be used. Further, with electronically captured panelists’ ratings, the computations of the panelists’ cut-scores would be far less labor intensive, reducing the amount of time required to conduct the standard setting studies, and ensuring greater accuracy of the data capture process. Without the computer interface and data capture, panelists’ data would need to be entered into preprogrammed spreadsheets that would then compute the panelists’ cut scores. The time to enter panelists’ data can be a hurdle for standard setting studies and is less efficient from a time and accuracy perspective than having electronic entry by panelists and automatic data capture for the analyses.

Following their first round of ratings, panelists will be provided with normative information about how their ratings compare with those of other panelists. Individual panelists will be given a unique identification code, so their personal cut-scores will be known only to them and not the other panelists. This is desirable because some panelists may feel intimidated or pressured by other panelists should their personal cut-scores be revealed. In addition, summary level data will be shared with the panelists based on their Round 1 results: Mean cut-points, median cut-points, measures of variability such as the standard errors of the mean and median, lowest and highest values for each cut-point.

Led by the group facilitators, discussion among the panelists will ensue, allowing panelists to present their perspectives on the results. Panelists will also be given information about items where there is a large discrepancy regarding where the transition location on the ILR scale falls. For these items, a group discussion will be led by the facilitators, eliciting from the panel differing perspectives for that location. The intent of the discussion is for the panelists to hear their fellow panelists’ perspectives, but not to require group consensus. Panelists will be told that their ratings should be made independently, after learning from the perspectives of their fellow panelists. This aspect of the standard-setting process is well-aligned with guidance embodied in the Standards for Educational and Psychological Testing which recommend that “If participants [in a standard-setting procedure] interacted with one another or exchanged information, the procedures through which they may have influenced one another should be set forth” (AERA, APA, NCME, 1999, p. 19).

Next panelists will undertake their second round of ratings, again making their ratings independently. Following completion of the second round of ratings, panelists will again be given the same type of normative data that they received following their first round of ratings, but in addition they will be shown…