ID06180012_SF33_Attach_1_PWS_Standard_Setting_04272018.pdf

Solicitation ID06180012 Attachment 01

Performance Work Statement

DLPT5 Standard-Setting Study

April 27, 2018

Directorate of Language Proficiency Assessment (LPAD) Defense Language Institute Foreign Language Center

C.1. INTRODUCTION

The Defense Language Institute Foreign Language Center (DLIFLC) has a requirement to assess the foreign language skills of the US Government military and civilian personnel.

C.2. BACKGROUND

C.2.1. DLIFLC is developing listening and reading tests (Defense Language Proficiency Test5 (DLPT5)) to assess the foreign language proficiency of the US Government military and civilian personnel. The DLPT5 reports scores according to the Interagency Language Roundtable (ILR) Skill Level Descriptions. DLPT5s are available as very low range, covering ILR levels 0+-1+, lower-range tests, covering ILR levels 0+-3, or as upper-range tests, covering ILR levels 3-4.

C.2.2. Each test, with exceptions noted below, has both a reading comprehension and a listening comprehension component; these components are considered separate tests for scoring purposes, but each standard-setting panel will consider both reading and listening tests in the same study, where both are available. In order to determine the number of items on each test that examinees need to answer correctly to achieve a given ILR score, DLIFLC has been tasked to use standard-setting studies. Standard-setting studies involve panels of experts judging the linkage between number of items correct and score awarded. Standard-setting studies are considered to be of great importance in the testing industry, as they provide evidence for the validity of decisions made based on a test. Standard-setting studies are one aspect of the commitment that DLIFLC has made to build quality assessments.

C.3. SCOPE

The Contractor shall conduct up to ten standard-setting studies in the base period and up to 10 studies in each of the two option periods. See the Technical Exhibit #1 - Proposed Listing of Standard Setting Studies in the Base Year.

Schedules corresponding to any exercised option period (if exercised) will be provided to the Contractor no later than 30 days prior to the exercise of the option. Each standard-setting study shall focus on one language and range, setting cut scores for the two skills (reading and listening) independently, unless specified otherwise. The Contractor shall deliver reports on the results of the standard-setting studies.

C.4. APPLICABLE DOCUMENTS

LPAD will provide “DLPT5 Familiarization Guide” and “Technical Requirements for Study and Report Specifications” (Technical Exhibit #2) as part of the solicitation. “DLPT5 Framework documents,” “Borderline Proficiency Level Descriptions” and “Standard-Setting Study Plan” will be provided within 5 days of award of the contract. These three documents include confidential specifications tailored to DLIFLC Standard Setting practices and are not believed to affect solicitors’ cost estimates.

Technical Exhibit # Document Name Description Technical Exhibit #1 Proposed Listing of Standard Setting Studies in the Base Year Languages for the Base Year

Technical Exhibit #2 Technical Requirements – Study and Report Specifications

Description of the study and reports specifications to be used in the standard- setting studies.

Technical Exhibit #3 DLIFLC ATFL-ESD-TD Test Security Agreement Form

Test security agreement form for all participants in standard setting studies process.

Technical Exhibit #4 Borderline Proficiency Level Descriptions (To be provided within 5 days after award of contract)

Descriptions of the abilities and functions of the borderline examinee, to be used by panelists.

Technical Exhibit #5 DLPT5 Framework document (To be provided within 5 days after award of contract)

Lays out basic assumptions underlying DLPT5 testing program.

Technical Exhibit #6 Standard-Setting Study Plan (To be provided within 5 days after award of contract)

Description of the methods to be used in the standard-setting studies.

Technical Exhibit #7 DLPT5 Familiarization Guide (go to https://www.dliflc.edu/resources/dlpt-guides/)

Guide intended for test users, including Interagency Language Roundtable Skill Level Descriptions.

C.5. TECHNICAL REQUIREMENTS

C.5.1. Experts for Standard Setting Studies: The Contractor shall provide experts for standard-setting studies.

C.5.1.1. It is estimated that a maximum of 16 Contractor supplied experts per language shall be required to hold a standard-setting study. The Government shall supplement the Contractor supplied panelists with approximately 5 experts of their own. The exact number of government and contractor provided experts will be coordinated at the task order level.

C.5.1.2. Experts shall have the following qualifications (see exception at C.5.2):

(a) Proficiency of ILR level 3 or greater in the target language (reading and listening).

(b) Proficiency of ILR level 3 or greater in English (reading, listening, and speaking).

(c) Experience in working with the Interagency Language Roundtable Skill Level Descriptions for Reading and Listening,

(d) Experience using the target language in the context of Department of Defense.

C.5.1.3. 50% of the experts shall be native speakers of English with proficiency of ILR level 3 or greater in English and the target language of the standard setting study. 50% of the experts shall be native speakers of the target language of the standard setting study with proficiency of ILR levels 3 or greater in the target language and in English.

C.5.1.4. DLIFLC will determine acceptability of experts on a case-by-case basis.

C.5.2. Language Resource Expert: From amongst the Contractor supplied experts, the Contractor shall designate one as a language resource expert for each standard-setting study.

C.5.2.1. The language resource expert shall have the following qualifications:

(a) Proficiency of ILR level 4 in the target language (reading and listening).

(b) Proficiency of ILR level 3 in English (reading, listening, and speaking).

(c) Experience in applied linguistics or teaching the target language.

(d) Recent experience in the target culture.

C.5.2.2. If it is not possible to find a language resource person meeting all of these qualifications, DLIFLC will determine acceptability of a language resource person on a case-by-case basis.

C.5.3. The Contractor shall conduct standard-setting studies in accordance with “Technical Specifications” provided by DLIFLC. (See Technical Exhibit #2)

C.5.4. Facilitators: Each standard setting study will require at least two (2) facilitators since target language panelists will be divided into two parallel subgroups for the duration of the study. Each facilitator will demonstrate relevant knowledge, skills and abilities through:

(a) Successful facilitation of standard setting studies (of any design) on at least six occasions within the last four years;

(b) Successful demonstration of group facilitation and moderation techniques on at least five occasions within the last five years; and

(c) Successful completion ― at an accredited US educational institution ― of at least 12 units of formal graduate level training addressing any combination of the following topics:

1. Educational and psychological measurement, including classical and modern (item) response theories and applications;

2. The AERA /APA /NCME Standards for Educational and Psychological Testing;

3. Selected- and constructed-response test development and analysis methods.

(d) Any professional experience conducting research and analyses related to the Interagency Language Roundtable (ILR) skill level descriptions is desirable.

C.5.5 Key Personnel:

C.5.5.1 The following personnel are considered key personnel by the Government: Project Manager and Standard Setting Study Facilitators.

C.5.5.2 Project Manager: The Contractor shall provide a Project Manager who shall be responsible for the performance of the work. The name of this person and an alternate who shall act for the contractor when the manager is absent shall be designated in writing to the contracting officer. The project manager or alternate shall have full authority to act for the contractor on all contract matters relating to daily operation of this contract. The project manager or alternate shall be available during normal business hours, Monday thru Friday except Federal holidays or when the Government facility is closed for administrative reasons. Project Managers must have at least three years of experience managing requirements of similar scope and magnitude. Familiarity with the ILR scale is preferred. The Contractor shall submit documentation of the qualifications of the Project Manager(s) in its proposal for Government approval, which may include resumes and/or curriculum vitae.

C.5.5.3 Study Facilitators: The Contractor shall provide a minimum of two Study Facilitators designated as key personnel at the base contract level and assigned to facilitate studies at the task order level. C.5.4 lists facilitator qualifications. The Contractor shall submit documentation of the qualifications of the Study Facilitators in its proposal for Government approval, which may include resumes and/or curriculum vitae.

C.5.6. Site of Studies:

C.5.6.1. Each study will take place with all participants at the same site over a period of 4 or 5 days - 4 days for tests in either the reading or listening skill; 5 days for tests in both the reading and listening skills.

C.5.6.2. The Contractor is responsible for obtaining a venue with space for the whole group and two parallel sessions in the Washington, D.C. area (within a 50 mile radius of the United States capital) unless specified otherwise. To this end, the contractor shall reserve a room for the whole group to sit in a ‘U’ shape for the first day of the standard-setting study, and two rooms big enough for the whole group for the remaining days of the standard-setting study. The Washington D.C. area is for the convenience of the Government in order to facilitate attendance by Government personnel.

C.5.7. Technology Equipment: The Contractor shall provide the following for the standard-setting studies.

C.5.7.1. One laptop computer, mouse, and noise-cancelling audio headset for each language expert's and the language resource person’s exclusive use for the duration of the study. The computers should be password protected and be able to connect wirelessly to the Internet. The computers should contain at least the following software and specifications:

(a) Wired or Wireless Internet connection (minimum - 3 Mbps broadband or faster bandwidth with low latency --preferred - 5 Mbps broadband or faster bandwidth with low latency)

(b) Google Chrome (V.49 or greater) or Mozilla Firefox (V.52 or greater)

(c) Windows OS 7 or greater (Windows native fonts)

(d) Microsoft Access

(d) Adobe Acrobat Professional

(e) Minimum 14" screen

C. 5.7.2. The contractor will arrange secured wireless internet connections for each room at the standard-setting study.

C.5.8. Lessons Learned from Studies: The Contractor shall document the lessons learned from each study and recommendations for refining the conduct of future studies.

C.5.9. Schedule of Standard Setting Studies:

C.5.9.1. The Contractor shall provide one or up to ten standard-setting studies in the base period and up to 10 studies in each of the two option periods thereafter. Standard-setting studies will be performed for any of the following languages:

(1) Italian (MC), (2) Polish (MC), (3) Ukrainian (MC), (4) Indonesian (MC), (5) Portuguese-Brazilian (MC), (6) Thai (MC) (7) Cambodian (CRT), (8) Malay (LR-CRT), (9) Kazakh (CRT), (10) Somali (CRT), (11) Tamil (CRT), (12) Uighur (CRT), (13) Haitian-Creole (CRT), (14) Persian Farsi (MC), (15) Romanian (MC), (16) Bulgarian (MC), (17) Dutch (MC), (18) Yoruba (CRT), (19) Cebuano (CRT-LC only), (20) Albanian (CRT), (21) MSA (MC), (22) Egyptian (MC), (23) Moroccan (MC), and (24) Tagalog (MC).

C.5.9.2. The Government reserves the right to make substitutions for languages; the Government will notify the Contractor of any substitutions at least 60 days before conducting the study.

C.6. DELIVERABLES

C.6.1. A post-award meeting, in person or by telephone, between the Government and the Contractor shall take place within five (5) working days of the award of the contract. At the meeting, the two parties shall review contract deliverables, proposed milestones, and implementation issues.

C.6.2. Additional meetings, in person or by telephone, between the Government and the Contractor shall take place as needed to review contract deliverables, proposed milestones, and implementation issues.

C.6.3. The Contractor shall submit a list of potential experts with their resumes and evidence of their language proficiency of the target language to DLIFLC 30 days before conducting each study. DLIFLC will review and approve qualified experts to participate in each study.

C.6.4. The Contractor shall submit a list of potential language resource experts with their resumes and evidence of their language proficiency of the target language to DLIFLC 30 days before conducting each study. DLIFLC will review and approve a qualified language resource expert to participate in each study.

C.6.5. Any changes from proposed Contractor supplied facilitators identified in the base contract as Key Personnel must be approved 30 days prior to the beginning of each study.

C.6.6. Contractor shall submit a detailed schedule for each standard-setting study panel at least 21 days before the scheduled convening of that panel.

C.6.7. The Contractor shall submit a description of the venue in which the standard setting study will be held. The description shall include a description of the meeting spaces, floor plans, and a map showing the location of the venue and the United States Capitol as well as the scale of the map. This description shall be submitted no later than 7 calendar days prior to the start of any study.

C.6.8. The Contractor shall deliver a report within 30 days following each study including the findings, recommended cut scores, impact data, and recommendations for future studies. See Technical Requirements for Study and Report Specifications for details. (Technical Exhibit #2)

C.6.9. The Contractor shall submit signed confidentiality agreements (Technical Exhibit #3) for all study participants including language experts, facilitators, and project coordinators or managers the day of the study prior to participant receipt of test materials.

C.6.10. The Contractor shall submit inventories of test material following Government specifications.

C.6.11. The Contractor shall return all test materials to the Government following the conclusion of each study and submit certification that all test material has been destroyed or transferred to DLIFLC within 30 days of the conclusion of each study or the end of the contract period of performance, whichever comes first. The certification shall include: 1) Statement that all paper and electronic copies of contract deliverables have been destroyed or transferred to the Government 2) signature of CEO, and 3) Corporate seal.

C.6.12. Base Contract Monthly Status Reports: The Contractor shall submit by the 10th of each following month, a report for the prior month listing all pertinent contract activities/deliverables accomplished, planned activities for the following month, issues encountered, and any other information pertinent to the contract.

C.6.13. Quality Control Plan: The contractor shall implement a quality control plan (QCP) at the beginning of the period of performance (POP) and maintain the plan throughout the POP. The initial plan shall be submitted to the Contracting Officer for review and approval no later than thirty (30) calendar days after contract award. All revisions shall be subject to written approval by the Contracting Officer, upon receipt of which the contractor shall document and record the plan number, distribute revised plans and communicate changes to affected contractor and Government personnel. The contractor shall furnish all changes to the Contracting Officer and the appropriate Contracting Officer’s Representative (COR) at least three (3) working days prior to the effective date of the change.

The contractor’s QCP shall establish performance indicators to ensure deliverables are within established tolerance limits.

Initial quality control requirements will be administered by the Program Manager. QCP shall include:

a. Performance Indicators (e.g., management goals relative to cost, schedule, and quality of technical and administrative service provided)

b. Inspection and test methods (e.g., accepted techniques and procedures)

c. Deficiency identification (e.g., areas exceeding acceptable tolerance limits)

d. Deficiency correction (e.g., establishes and implements corrective actions)

e. Documentation and enforcement (e.g., document/enforce corrective actions)

f. Trend analysis (e.g., evaluates and record trends in performance results)

g. Surveillance methods (e.g., collect real time, on-line performance metrics)

h. Customer complaint program (e.g., respond to valid customer complaints)

The QCP shall be the basis of the contractor’s Quality Control (QC) Program, a basic tenet of which shall be continuous quality improvement. The QCP shall also:

a. Recommend QC staff should be independent of the contractor’s production and supervisory staff and to report directly to management.

b. Contain methods of direct and indirect communication with the Government regarding performance, to include regular and formal meetings with the Government, review and analyses of key process indicators, analyses of process deficiencies, and problem resolution.

c. Contain process control and process performance measurement procedures to include how the contractor plans to effect preventive actions and corrective actions, as opposed to reliance on Government or customer identification of deficiencies prior to resolving the problem.

d. Contain a customer issue tracking ticketing system for identification and correction of validated complaints and to provide feedback.

e. Establish procedures to document and record all inspections or tests conducted by the contractor and any corrective actions taken, and their effectiveness, during the performance period.

The Contractor shall submit a QCP to the Contracting Officer within 30 calendar days of award. The Contracting Officer shall review and approve/disapprove the QCP within 15 calendar days of submission.

C.7. ENTERPRISE CONTRACTOR MANPOWER REPORTING APPLICATION (https://www.ecmra.mil/)

C.7.1. The U.S. Department of Defense Enterprise Contractor Manpower Reporting Application (eCMRA) is a business process to collect information for each contract, on the funding source and related financial data, the organization receiving the services, mission/function performed, and labor hours and costs, by the Government Fiscal Year (1 October through 30 September). Contractors report actual labor hours, dollars, and other information directly from their accounting systems directly into this system, or through their Army Requiring Activity, using their existing timekeeping, personnel, accounting systems as well as information supplied with the contract. Existing financial management and procurement systems do not completely distinguish between goods and services, nor do they identify the specific organization for which the work is being performed. eCMRA will enable the Army to (1) fully understand the composition of the Army workforce - military, civilian, and contractor, and allow for more informed workforce staffing and funding decisions; (2) provide better oversight of our workforce, avoid duplication of effort, or shifting of in-house reductions to contract; and (3) better account for and explain our total Army workforce. Contractor labor hours and associated labor costs are to be treated as proprietary information when linked to contractor name and contract number.

C.7.2. The U.S. Department of Defense operates and maintains a secure data collection site where the contractor will report ALL contractor manpower (including subcontractor manpower) required for performance of this contract. The Contractor is required to completely fill in all the information in the format using the following web address:

https://www.ecmra.mil/.

C.7.3. Reporting inputs will be for the labor executed during the period of performance during each Government Fiscal Year, which runs from October 1 through September 30. While inputs may be reported any time during the FY, all data shall be reported no later than October 31st of each calendar year for which there is performance. Contractors may direct questions to the help desk by clicking on “Send an email” which is located under the Help Resources ribbon on the right side of the login page of the eCMRA website.

C.8. SECURITY

C.8.1. The Contractor shall employ personnel and implement procedures to safeguard the security and confidentiality of all deliverables. Contractor shall comply with Army Regulation 611-5 (2008 revision), paragraphs 3-1 and 3-2, as well as DLIFLC regulation 611-1 (dated July 1, 1997), paragraphs 7 and 8. Government will retain sole ownership of, and exclusive rights to, all deliverables and GFM. See DFARS 252.227-7020 “Rights in Special Works” and 252.227-7021 “Rights in Data – Existing Works” for license rights requirement related to all deliverables under this contract.

C.8.2. Confidentiality Agreement: Contractor and all subcontractors’ personnel working under this contract shall sign a Government supplied confidentiality agreement to protect the security of all foreign language testing initiatives by the United States Government. This agreement shall acknowledge that irreparable harm would be caused to Government if the confidential information obtained by Contractor in the performance of tasks included in this Performance Work Statement were to be revealed to third parties. All contractors and subcontractors must submit a copy of their signed confidentiality agreement to Test Development Division before they that reviewer can begin work. The confidentiality agreement entered into by Contractor or subcontractor personnel in connection with this contract shall survive the termination of this contract.

C.8.3. Security Procedures: Contractor or subcontractor personnel shall not show any materials pertaining to this contract to any other person, or discuss its contents, beyond normal work requirements. Contractor or subcontractor personnel shall keep all contract related materials under lock and key at all times when not actually performing the work.

Contractor or subcontractor personnel shall not retain paper or electronic copies of any contract deliverables or contract related materials following the delivery of contract deliverables to Government, or re-use any of these materials for future private, public, or commercial use.

C.8.4. Operational Security (OPSEC) and Training: Per AR 530-1 Operations Security, the contractor employees must complete Level I OPSEC Awareness training. New employees must be trained within 30 calendar days of their reporting for duty and annually thereafter. OPSEC Awareness for Military Members, DoD Employees and Contractors is available at the following website: http://cdsetrain.dtic.mil/opsec/index.htm The Contractor shall practice OPSEC to protect critical information for specific government contracts and subcontracts. The Contractor will ensure that annual OPSEC training is conducted to protect DLIFLC critical information.

C.9. CONFLICT OF INTEREST

The Contractor shall certify that no panel members have a conflict of interest or the appearance of a conflict of interest.

No panel member shall participate in a study without DLIFLC approval.

C.10. TRAVEL

C.10.1. The Contractor may be required to travel for the purpose of attending Government initiated Standard Setting Studies or other related Government meetings. The Government will reimburse for approved travel within the Continental United States in accordance with listed travel cost CLINs for each task order.

C.10.2. The Contractor shall use only the minimum number of travelers and rental vehicles required to accomplish the requirement. Contractor personnel are responsible to make all arrangements for travel, lodging and local transportation without assistance of Government personnel. The Contractor shall coordinate security-related paperwork (including obtaining access to military installations) and any other requirements pertinent to this Performance Work Statement.

C.10.3. Whenever travel is determined to be necessary by the Government, Contractor personnel shall be available to travel within two weeks’ notice.

C.10.4. The Contractor shall obtain advance COR approval for travel requirements.

C.10.5. The Contractor shall submit an estimated cost of Contractor attendees for in-person meetings/conferences/orientation. Included in their estimate l shall be a breakdown of costs related to the travel costs and shall include as a minimum, (1) transportation costs (airfare/car rental, etc.); (2) lodging and; (3) per diem for each employee. The contractor shall use JTR rates as guidelines for expenses (these can be found at http://www.defensetravel.dod.mil/site/travelreg.cfm).

C.10.6. The Contractor shall submit an expense report to the COR when invoicing for travel costs. This report shall contain the following, at a minimum:

A brief description of the travel to include traveler(s) name(s), dates of travel, destination and purpose of travel.

Copies of receipts for all airfare, lodging and any expense in excess of $75.00.

Claims for payment of per diem shall be accompanied by a corporate worksheet or copy of the employee’s per diem voucher listing the location, dates and per diem rate applied to the claim.

Acceptable submissions include paid receipts from airlines and hotels or copies of corporate credit / purchase card statements providing that the statement lists the date, vendor name and amount of purchase.

All backup documentation shall clearly indicate the actual date that the contractor incurred the claimed expense.

Failure to provide the required documentation may result in rejection of the contractor’s invoice as not being a proper invoice as described in FAR 32.905(b)(2).

Travel invoices shall be submitted separately from labor invoices.

C.11. TASK ORDER ISSUANCE

C.11.1. Administration: Administration for the awarded contract will be by the General Services Administration, Region 6 Assisted Acquisition Services located in Kansas City, MO.

C.11.2. Types of Orders: The type of orders that are authorized for use under this Indefinite Delivery – Indefinite Quantity contract are Firm Fixed Price (FFP) with reimbursable line item for travel (when applicable). Task orders will be awarded on a unilateral basis.

C.11.3. Unauthorized Work: The contractor is not authorized to commence task order performance without a signed task order or direction by the Contracting Officer. Notwithstanding any of the other provisions of this contract, the Contracting Officer shall be the only individual authorized on behalf of the Government to:

C.11.3.1. Accept nonconforming work;

C.11.3.2. Waive any requirement of the task order; or, C.11.3.3. Modify any term or condition of the task order.

C.11.4. Task Order Funding: No unfunded task orders are authorized. Funding must be authorized at the task order level and must be the type deemed appropriate for the services to be acquired.

C.11.5. Security Requirements: Contractor personnel shall conform to all security requirements as specified in each original IDIQ type contract.

C.11.6. Task Order Issuance Procedures: Individual task orders will be awarded with a “Total Price” for FFP task orders. Task orders shall be issued in accordance with the following procedures:

C.11.6.1. The Contracting Officer will issue a unilateral task order that will include:

C.11.6.1.1. Contract and task order numbers;

C.11.6.1.2. Any government-furnished property, material, or base support to be made available for performance of the order;

C.11.6.1.3. The schedule of services and/or supplies and period of performance;

C.11.6.1.4. Any other pertinent information deemed necessary to the performance of the order.

C.11.6.1.5. The preferred method for task order issuance is through electronic transmission.

C.11.6.2. Upon receipt of a written authorization/order issued by the Contracting Officer, the contractor shall furnish to the Government the services and/or supplies as set forth in the schedule.

C.11.6.3. Oral orders will be placed on an “as needed” basis in accordance with the FAR clause 52.216-19 Order Limitations and 52.216-22 Indefinite Quantity. Performance/deliveries shall be made in response to oral orders issued against the contract by the ordering officers. Each individual task order will reflect the actual period of performance.

C.11.6.4. The contractor shall submit invoices of items completed within 30 days by means of GSA’s ASSIST portal.

C.11.6.5. All quantities of supplies required for each individual task order assigned are fixed. The contractor shall only be paid for actual quantities ordered/work performed.

(End of Performance Work Statement)

TECHNICAL EXHIBIT #1

Proposed Listing of Standard-Setting studies in Base Year

Language

1. Somali Form B. (CRT) 6. Indonesian (MC)

2. Italian (MC) 7. Ukrainian (MC)

3. Polish (MC) 8. Thai (MC)

4. Yoruba Form B (CRT) 9. Tagalog (MC)

5. Portuguese-Brazilian (MC) 10. To Be Determined

(End of Technical Exhibit #1 - Proposed Listing of Standard-Setting studies in Base Year)

TECHNICAL EXHIBIT #2

Technical Requirements – Study Specifications:

Panelists will be provided with a brief orientation to standard-setting, the DLPT5, the Interagency Language Roundtable (ILR) scale, and the test development process. Following that training, the panelists will use generic Borderline Proficiency Level Descriptors (BPLDs) to assess the minimum target language proficiency needed to know the correct answer to each item in a set of practice items. The panelists will next tailor the generic BPLDs through the provision of language specific examples (e.g., pronunciation, grammar, cultural hierarchies, etc.) to address unique aspects of the target language and skill. This process, from using the generic BPLDs through tailoring the BPLDs for the target language, will be repeated in the event of a two-skill standard-setting study.

After the tailoring process, the panel will be split into two smaller panels in separate rooms, where they will have minimal interaction with the other panelists for the rest of the standard-setting study. The panelists will then experience the test in an approximation of the examinee experience. Following that experience, the panelists will be trained to use the Modified Angoff Yes/No Method. After working through a set of practice items, the panelists will use the tailored BPLDs to assess the minimum target language proficiency needed to know the correct answer to each item on the test form provided.

Panelist recommendations will be discussed, and empirical item-level data, if any, will be provided to panelists prior to conducting a second round of ratings. Panelists will also discuss some of their ratings decisions. The second round of ratings will be used to set recommended cut scores. The process, from the test experience to the second round of ratings, will then be repeated for additional test forms or skills as appropriate. After the ratings are completed, the panelists will participate in a debriefing led by the study facilitators.

Specifically, each study will include the following sections:

1. Orientation and Training:

a. Test security briefing and non-disclosure agreements

b. Description of the standard-setting process

c. Description of test development process and DLPT5

d. Description of DLPT5 test population

e. DLPT5 goal

f. Orientation and practice with constructed-response credit schemes, if applicable

g. ILR proficiency scale

h. DLPT5 format and anatomy of a test item

i. DLPT5 test content areas

j. Reading and listening comprehension Knowledge, Skills, & Abilities

k. Description of DLPT5 pre-test sample, when available.

l. Example of test item characteristics

m. Factors influencing examinee test performance

n. Factors influencing test item difficulty

2. Introduction and practice using generic BPLDs with sample passages and items

3. Tailoring generic reading comprehension BPLDs to target language

4. Tailoring generic listening comprehension BPLDs to target language

5. Orientation and training process evaluation

6. Reading comprehension test experience

7. Standard setting: reading comprehension using Modified-Angoff Yes/No method

a. Training on the Modified-Angoff Yes/No method

b. Item-level rating practice

c. Operational ratings – round 1

d. Operational ratings – round 1 feedback

i. Individual cut score recommendations.

ii. Group statistics (e.g., median and mode)

iii. Data from field testing, when available

iv. Discussion of items selected by facilitators

e. Operational ratings – round 2

f. Operational ratings – round 2 feedback

g. Reading comprehension process evaluation

8. Listening comprehension test experience

9. Standard setting: listening comprehension using Modified-Angoff Yes/No method

a. Item-level rating practice

b. Operational ratings – round 1

c. Operational ratings – round 1 feedback

i. Individual cut score recommendations

ii. Group statistics (e.g., median and mode)

iii. Data from field testing, when available.

iv. Discussion of items selected by facilitators

d. Operational ratings – round 2

e. Operational ratings – round 2 feedback

f. Listening comprehension process and overall standard-setting study evaluation

10. Focus group /debriefing

Technical Requirements – Report Specifications:

Facilitators must submit a technical report within 30 days of the completion of each standard-setting study. The report must include necessary data and sufficiently detailed facilitator observations such that DLIFLC may reasonably assess the reliability and validity of panelist recommendations. Each report will include the following sections:

1. Executive summary;

2. Purpose of the report;

3. Facilitator’s acknowledgement of confidentiality and the proprietary nature of the report contents;

4. Brief description of study implementation, noting all departures from study specifications and remedial actions taken;

5. Facilitator evaluation of validity evidence supporting standard setting study recommendations and evidence that may threaten validity of the standard setting study recommendations

a. Procedural sources (e.g., panel composition, facilitator adherence to study specifications; panelist engagement, the physical environment of the standard setting study)

b. Internal sources (e.g., correlations between panelist ratings and item difficulty values)

c. External sources (e.g., results of the split-panel design);

6. References cited in report;

7. List of attachments;

8. Data in Tabular Format:

a. Standard setting panelist demographic characteristics (to include: US Government affiliation, heritage /native /non-native language ability, panel sub-group affiliation);

b. Panelist evaluation of orientation, training, and BPLD tailoring activities;

c. Reading comprehension cut score recommendations and descriptive statistics;

d. Panelist evaluation of reading comprehension standard setting activities and results;

e. Listening comprehension cut score recommendations and descriptive statistics;

f. Panelist evaluation of listening comprehension standard setting activities and results;

g. Panelist evaluation of the workshop overall;

9. Data in Charts /Figures:

a. Mean reading comprehension cut score recommendations;

b. Dot plot of reading comprehension cut score recommendations by ILR level;

c. Mean listening comprehension cut score recommendations;

d. Dot plot of listening comprehension cut score recommendations by ILR level;

10. Submitted as appendices

a. Study agenda and copies of all materials distributed to panelists;

b. Panelist enhancements (tailoring) to generic reading and listening BPLDs;

c. Written comments submitted by panelists on evaluation /feedback forms;

d. Synthesis of comments made by panelists during focus-group /debrief;

e. Reading comprehension: panelist BPLD ratings for each item for both rounds of rating;

f. Reading comprehension: item p-values derived from individual panelist BPLD recommendations for each item for both rounds of rating;

g. Reading comprehension: cut score recommendations for each ILR level for both rounds of rating by panelist and panelists grouped by government affiliation;

h. Listening comprehension: panelist BPLD ratings for each item for both rounds of rating;

i. Listening comprehension: item p-values derived from individual panelist BPLD recommendations for each item for both rounds of rating;

j. Listening comprehension: cut score recommendations for each ILR level for both rounds of rating by panelist and panelists grouped by government affiliation;

k. Panelist demographic information (e.g., education, employer, affiliation, years in the field, experience with the DLPT5, native/non-native/heritage language user status)

i. Study facilitators will provide unique panelist identification numbers such that panelist self-assessments, responses to demographic questionnaire items, and operational ratings, etc., may be linked within a study and across studies (for those panelists that participate in multiple studies).

l. Panelist self-assessments of language proficiency

m. Panelist answers to test experiences

(End of Technical Exhibit #2 - Technical Requirement Study Specifications)

TECHNICAL EXHIBIT #3

Technical Exhibit #3 - DLIFLC ATFL-ESD-TD Test Security Agreement Form

I, (PRINT NAME) , having been asked to provide service in the Standard Setting Study Methodological Evaluation for the Testing Directorate at The Defense Language Institute Foreign Language Center, Presidio of Monterey, California have read and understand the following guidelines regarding Test Security:

1. I will have direct or indirect contact with Defense Language Proficiency Test (DLPT) materials, Social Security, Privacy Act, research, and/or statistical data that while it may not be identified as classified information, is considered For Official Use Only, and therefore is not available for public use or distribution at any time.

2. I am directly responsible for the physical and/or electronic security of any and all material that is assigned directly to me or in conjunction with another office or person in this directorate.

3. I will take all necessary measures to ensure that any and all material is either secured and/or contained by myself, a secure container and/or facility, and/or another person within this directorate.

4. I will NOT download or save any electronic or audio files onto any storage devices.

5. I cannot discuss or share test material or specific test items with any person not directly assigned to or approved by the

Directorate of Evaluation and Standardization to participate in the standard setting study.

6. In the event that I am reassigned or my contract expires, I will turn in to the project director any and all test material that I used or had access to during my assignment. This includes all originals and copies of paper printed and/or electronically formatted material, and any personal notes about said materials.

7. I must report any compromise (potential or actual), knowledge of any compromise (potential or actual), whether due to my actions or the actions of others, of any and all material, to the project director and/or chain of supervision immediately upon having knowledge of the situation(s) previously identified.

8. This agreement is binding. There is no expiration date assigned to this agreement.

I have read the above guidelines regarding test security. I understand that failure to comply with this signed agreement will be used in determining possible administrative action against me.

SIGNED ON (DATE)

Signature of Employee Signature of Chief of Testing Department

(End of Technical Exhibit #3 - DLIFLC ATFL-ESD-TD Test Security Agreement Form)

(End of Technical Exhibits)