Exhibit_2_Technical Requirements_Study_Report_Specs.docx

Standard Setting Study 03 August 2021 Technical Requirements – Study Specifications:

Panelists will be provided with a brief orientation to standard-setting, the DLPT5, the Interagency Language Roundtable (ILR) scale, and the test development process. Following that training, the panelists will use generic Borderline Proficiency Level Descriptors (BPLDs) to assess the minimum target language proficiency needed to know the correct answer to each item in a set of practice items. The panelists will next tailor the generic BPLDs through the provision of language specific examples (e.g., pronunciation, grammar, cultural hierarchies, etc.) to address unique aspects of the target language and skill. This process, from using the generic BPLDs through tailoring the BPLDs for the target language, will be repeated in the event of a two-skill standard-setting study.

After the tailoring process, the panel will be split into two smaller panels in separate rooms, where they will have minimal interaction with the other panelists for the rest of the standard-setting study. The panelists will then experience the test in an approximation of the examinee experience. Following that experience, the panelists will be trained to use the Modified Angoff Yes/No Method. After working through a set of practice items, the panelists will use the tailored BPLDs to assess the minimum target language proficiency needed to know the correct answer to each item on the test form provided. Facilitators will tell panelists how to interpret their individual cut score recommendations; present group cut score recommendations; and provide empirical item-level data and how to interpret those data (if available). Facilitators will coordinate a panelist discussion of some of their rating decisions. Following that discussion, facilitators will conduct a second round of ratings. The second round of ratings will be used to set recommended cut scores. The process, from the test experience to the second round of ratings, will then be repeated for additional test forms or skills as appropriate. After the ratings are completed, the panelists will participate in a debriefing led by the study facilitators.

Specifically, each study will include the following sections:

1. Orientation and Training:

a. Test security briefing and non-disclosure agreements

b. Description of the standard-setting process

c. Description of test development process and DLPT5

d. Description of DLPT5 test population

e. DLPT5 goal

f. Orientation and practice with constructed-response credit schemes, if applicable

g. ILR proficiency scale

h. DLPT5 format and anatomy of a test item

i. DLPT5 test content areas

j. Reading and listening comprehension Knowledge, Skills, & Abilities

k. Description of DLPT5 pre-test sample, when available.

l. Example of test item characteristics

m. Factors influencing examinee test performance

n. Factors influencing test item difficulty

2. Introduction and practice using generic BPLDs with sample passages and items

3. Tailoring generic reading comprehension BPLDs to target language

4. Tailoring generic listening comprehension BPLDs to target language

5. Orientation and training process evaluation

6. Reading comprehension practice with UI

7. Reading comprehension test experience

8. Standard setting: reading comprehension using Modified-Angoff Yes/No method

a. Training on the Modified-Angoff Yes/No method

b. Item-level rating practice

c. Operational ratings – round 1

d. Operational ratings – round 1 feedback

i. Individual cut score recommendations.

ii. Group cut score recommendation statistics (e.g., median and mode)

iii. Item statistics, when available

iv. Discussion of items selected by facilitators

e. Operational ratings – round 2

f. Operational ratings – round 2 feedback

g. Reading comprehension process evaluation

9. Listening comprehension practice with UI

10. Listening comprehension test experience

11. Standard setting: listening comprehension using Modified-Angoff Yes/No method

a. Item-level rating practice

b. Operational ratings – round 1

c. Operational ratings – round 1 feedback

i. Individual cut score recommendations

ii. Group cut score recommendation statistics (e.g., median and mode)

iii. Item statistics, when available.

iv. Discussion of items selected by facilitators

d. Operational ratings – round 2

e. Operational ratings – round 2 feedback

f. Listening comprehension process and overall standard-setting study evaluation

12. Focus group /debriefing

Technical Requirements – Report Specifications:

Facilitators must submit a technical report within 30 days of the completion of each standard-setting study. The report must include necessary data and sufficiently detailed facilitator observations such that DLIFLC may reasonably assess the reliability and validity of panelist recommendations. The report must not include links to external sources. Each report will include the following sections:

1. Executive summary;

2. Purpose of the report;

3. Facilitator’s acknowledgement of confidentiality and the proprietary nature of the report contents;

4. Brief description of study implementation, noting all departures from study specifications and remedial actions taken;

5. Facilitator evaluation of validity evidence supporting standard setting study recommendations and evidence that may threaten validity of the standard setting study recommendations

a. Procedural sources (e.g., panel composition, facilitator adherence to study specifications; panelist engagement, panelist adherence to study design (e.g., repeated reference to scales other than the BPLDs or acting on an expressed desire to artificially inflate/deflate cut scores), the physical environment of the standard setting study)

b. Internal sources (e.g., correlations between panelist ratings and item difficulty values)

c. External sources (e.g., results of the split-panel design);

6. References cited in report;

7. List of attachments;

8. Data in Tabular Format:

a. Standard setting panelist demographic characteristics (to include: US Government affiliation, heritage /native /non-native language ability, panel sub-group affiliation);

b. Panelist evaluation of orientation, training, and BPLD tailoring activities;

c. Reading comprehension cut score recommendations and descriptive statistics;

d. Panelist evaluation of reading comprehension standard setting activities and results

e. Listening comprehension cut score recommendations and descriptive statistics;

f. Panelist evaluation of listening comprehension standard setting activities and results;

g. Panelist evaluation of the workshop overall;

9. Data in Charts /Figures:

a. Mean reading comprehension cut score recommendations;

b. Median reading comprehension cut score recommendations;

c. Dot plot of reading comprehension cut score recommendations by ILR level;

d. Mean listening comprehension cut score recommendations;

e. Median reading comprehension cut score recommendations;

f. Dot plot of listening comprehension cut score recommendations by ILR level;

10. Submitted as appendices

a. Standard setting study agenda and copies of all materials distributed to panelists;

b. Panelist enhancements (tailoring) to generic reading and listening BPLDs;

c. Written comments submitted by panelists on evaluation /feedback forms;

d. Synthesis of comments made by panelists during focus-group /debrief;

e. Reading comprehension: panelist BPLD ratings for each item for both rounds of rating;

f. Reading comprehension: item p-values derived from individual panelist BPLD recommendations for each item for both rounds of rating;

g. Reading comprehension: cut score recommendations for each ILR level for both rounds of rating by panelist and panelists grouped by government affiliation;

h. Listening comprehension: panelist BPLD ratings for each item for both rounds of rating;

i. Listening comprehension: item p-values derived from individual panelist BPLD recommendations for each item for both rounds of rating;

j. Listening comprehension: cut score recommendations for each ILR level for both rounds of rating by panelist and panelists grouped by government affiliation;

k. Panelist demographic information (e.g., education, employer, affiliation, years in the field, experience with the DLPT5, native/non-native/heritage language user status) without PII

i. Study facilitators will provide unique panelist identification numbers such that panelist self-assessments, responses to demographic questionnaire items, and operational ratings, etc., may be linked within a study and across studies (for those panelists that participate in multiple studies).

l. Panelist self-assessments of language proficiency

11. Submitted separately as an encrypted document

a. Crosswalk between PII and unique panelist identifiers used in report.