47QFHA23K0008_T10_08d_Attach 1 - PWS_PRS_ODA_May 2023.pdf

PERFORMANCEWORK STATEMENT (PWS)

ONLINE DIAGNOSTIC ASSESSMENT (ODA) TESTLET DEVELOPMENT

SUPPORT SERVICES

The Defense Language Institute Foreign Language Center (DLIFLC)

1. GENERAL: This is a non-personal services contract to provide Online Diagnostic Assessment (ODA) Testlet Development support services.

1.1. Description of Services/Introduction: The contractor shall provide all personnel, equipment, supplies, facilities, transportation, tools, materials, supervision, and other items and non-personal services necessary to perform Online Diagnostic Assessment (ODA) Testlet Development support services as defined in this PWS except for those items specified as Government furnished property and services. The contractor shall perform to the standards in this contract.

The Defense Language Institute Foreign Language Center (DLIFLC), located at the Presidio of Monterey (POM), California, is tasked to help military and Government linguists maintain and enhance proficiency in selected languages. As a vital component of mission/task accomplishment, DLIFLC is expanding the language capabilities of its Online Diagnostic Assessment (ODA) program.

The base period of this contract requires the Contractor to submit forty (40) Government-accepted Reading or Listening Comprehension testlets (units of assessment) for each of the languages indicated in Technical Exhibit 2 (Arabic-MSA, Russian, North Korean, Chinese-Mandarin, and Persian-Farsi). All testlets are to be delivered within a specified delivery period during the base period (one year).

The option period of this contract requires the Contractor to submit forty (40) Government- accepted Reading or Listening Comprehension testlets (units of assessment) from the languages indicated in Technical Exhibit 3 and the specific languages will be provided no later than 60 days prior to the possible exercise of all option periods.

1.2. Background: Online Diagnostic Assessment (ODA) is a web-based diagnostic assessment tool that identifies learner’s strengths, weaknesses, and provides feedback (may be accessed at https://oda.dliflc.edu.). The primary function of the ODA system is to meet the critical formative assessment needs of resident and non-resident language training and maintenance programs.

One of the challenges teachers and managers of military linguists in the field face is determining the actual language capability and needs of their students. Although they may have comparable proficiency test scores, linguists enrolled in various remedial and enhancement programs typically vary in terms of individual strengths and weaknesses.

The ODA system provides the program managers, teachers, and learners with an instrument to evaluate and manage their foreign-language learning needs.

ODA (1) offers language assessment that adapts to the learner’s performance; (2) determines and verifies floor and ceiling levels of proficiency; (3) collects diagnostic data; (4) generates diagnostic profiles; and (5) provides the learner with individualized feedback. Sampling of learner abilities is systematic across a variety of levels, topics, tasks, and specific linguistic features.

1.3. Objectives: Provide testlets for selected languages for Reading and Listening Comprehension for online diagnostic assessment for foreign language students.

● Provide testlets consisting of passages with content comprehension and linguistic ability assessment items associated with the passages.

● Provide well ILR-rated passages that are appropriate for ODA purposes.

● Provide 2 to 3 content comprehension assessment short-answer items on a selected passage with answer keys

● Provide linguistic ability assessment items on vocabulary, structure, and discourse components of the selected passage.

● Provide user feedback statements (diagnostic profile statements) for the structure and discourse assessment items.

1.4. Scope: ODA Testlets Development.

1.4.1. Reading/Listening Comprehension Testlets. The Contractor shall provide forty (40) each;

Government-accepted, system-ready, reading/listening comprehension testlets (see paragraph 2.1. for the definition of testlet) for each of the languages shown in Technical Exhibit 2 for the base period and Technical Exhibit 3 for option periods.

1.4.2. Delivery Requirements. The Contractor shall deliver all required reading or listening testlets within the specified period (one year) in accordance with the contract’s delivery schedule (see paragraph 5.2.2, Delivery/Acceptance Process).

1.4.3. Government-Approval of Testlets. All four stages shall require government approval. Each testlet shall be first approved by the Government review team before payment is made to the Contractor upon approval of stage four. Payment shall only be made after government approval of stage four (see paragraph 5.2.2).

1.4.4. Proficiency Levels. In the ODA program, Testlets will have different proficiency levels (1; 1+; 2; 2+ and

3) as defined in the Interagency Language Roundtable (ILR) skill level descriptions [see technical exhibits 4 (reading) and 5 (listening)], in accordance with the distribution shown in Technical Exhibit 2 for the base period and Technical Exhibit 3 for option periods.

1.4.5. Languages, skills, and the number of testlets in each skill level for the option period shall be provided 60 calendar days prior to the start date of the option period

1.4.6. List of Languages: Additional languages may be added at any given time in addition to the existing ODA language list below.

Arabic-Egyptian Arabic-Iraqi Arabic-Levantine Arabic-MSA Balochi Chinese Dari English Farsi French Hebrew Korean Korean / North Korean Pashto Portuguese Russian Serbian / Croatian Somali Spanish Tagalog Turkish Urdu

1.5. Period of Performance: The period of performance shall be for one Base Year of 12 months and four 12-month option years.

1.6. General Information

1.6.1. Quality Control: Quality Control is the responsibility of the contractor. The contractor is responsible for the delivery of quality services/supplies to the Government (see FAR 52.246-4, Inspection of Services – Fixed-Price).

The Contractor shall develop, implement and maintain an effective Quality Control System which includes a written Quality Control Plan (QCP). The QCP shall implement a standardized procedure/methodology for monitoring and documenting contract performance to ensure all contract requirements are met. The Contractors’ QCP must contain a systematic approach to monitor operations to ensure acceptable services/products are provided to the Government. The QCP, as a minimum, shall address continuous process improvement;

procedures for scheduling, conducting, and documentation of inspection; discrepancy identification and correction; corrective action procedures to include procedures for addressing Government discovered non-conformances; procedures for root cause analysis to identify the root cause and root cause corrective action to prevent re-occurrence of discrepancies; procedures for trend analysis; procedures for collecting and addressing customer feedback/complaints. The contractor shall upon request provide the Government with their quality control documentation. The QCP shall be provided to the administrative Contracting Officer (KO) and contract specialist via email within 30 days of the PSD. The Government will accept or return the QCP for revision within 30 days. Any change to the QCP after initial acceptance requires the review and acceptance of the KO.

1.6.2. Quality Assurance: The Government will evaluate the contractor’s performance under this contract in accordance with (IAW) the Quality Assurance Surveillance Plan (QASP). This plan is a Government only document primarily focused on what the Government must do to assure that the contractor has performed IAW the requirements of the contract.

1.6.3. Federal Government Holidays: The contractor is not required to make deliveries or attend meetings on Federal holidays as recognized by the Office of Personnel Management, https://www.opm.gov/policy-data-oversight/pay-leave/federal-holidays/. The Government will not work on Federal holidays.

1.6.4. Place of Performance: The work to be performed under this contract will be performed at the contractor facility.

1.6.5. Type of Contract: The Government will award a Firm Fixed Price contract.

1.6.6. Security Procedures:

1.6.6.1. The Contractor or its subcontractor personnel shall not reveal any materials pertaining to this contract to any other person or discuss its contents beyond normal work requirements. The Contractor or its subcontractor personnel shall keep all contract-related materials under lock and key at all times when not actually performing the work.

1.6.6.2. The Contractor or its subcontractor personnel shall not retain paper or electronic copies of any contract deliverables or contract-related materials following the delivery of contract deliverables to Government, or re-use any of these materials for future private, public, or commercial use.

1.6.6.3. Any materials kept outside of DLIFLC, for example, materials kept by test developers at the contractor’s site, or field test materials, must be returned to the COR or destroyed no more than 15 working days after the termination of the activity for which they were used. The Contractor shall inventory all material not at DLIFLC and send to the COR documentation that any materials on paper or removable electronic media have been destroyed or transferred to the COR, as well as a certification that any electronic versions of test material for that set of materials have been removed from reviewers’ computers or electronic storage media.

1.6.7. Security of Deliverables (including but not limited to materials and software security)

1.6.7.1. The Contractor shall implement procedures to safeguard the security and confidentiality of all deliverables. The Contractor shall comply with Army Regulation 611-5 (2008 revision), paragraphs 3-1 and 3-2.

1.6.7.2. The Government will retain sole ownership of and exclusive rights to all deliverables and Government-Furnished Materials provided to the Contractor (for example, intellectual property).

1.6.7.3. Non-Disclosure Agreement (NDA)

1.6.7.3.1. The Contractor and its subcontractor’s personnel shall sign a confidentiality agreement to protect the security of all foreign language testing materials provided to the Contractor by the Government. This agreement shall include an agreement signed by Contractor and subcontractor personnel that the individual will not, directly or indirectly, use, make available, sell, disclose, or otherwise communicate to any third-party materials or services provided under this contract to the Government.

1.6.7.3.2. This agreement shall acknowledge that irreparable harm would be caused to the Government if the confidential information obtained by Contractor in the performance of tasks included in this Performance Work Statement were to be revealed to third parties.

1.6.7.3.3. A copy of each individual’s signed confidentiality agreement must be received by the COR before that individual can begin work.

1.6.7.3.4. The confidentiality agreement entered into by Contractor or subcontractor personnel in connection with this contract shall be in effect indefinitely

1.6.8. Orientation Requirements

1.6.8.1. The Government (DLIFLC) shall provide orientation to the Contractor at the Contractor’s designated site on the date(s) agreed upon by both parties during the pre-performance meeting.

1.6.8.2. Orientation of Contractor-personnel is expected to last approximately two (2) weeks. It will be scheduled to begin within the first month after the contract award. All key personnel are expected to be present during the orientation (see 1.6.12. for the key personnel definitions).

1.6.8.3. Although available for further guidance, the Government expects the Contractor to provide further training to new employees by means of Contractor personnel who have already received the initial orientation by the Government.

1.6.9. Post Award Conference/Periodic Progress Meetings: The Contractor agrees to attend any post-award conference convened by the contracting activity IAW Federal Acquisition Regulation Subpart 42.5, Post Award Orientation. The Contracting Officer, COR, and other Government personnel, as appropriate, may meet periodically with the contractor to review the contractor's performance. At these meetings, the Government will apprise the contractor of how the Government views the contractor's performance and the contractor will apprise the Government of problems, if any, being experienced. Appropriate action shall be taken to resolve outstanding issues. These meetings shall be at no additional cost to the Government.

1.6.10. Contracting Officer Representative (COR): The COR will be identified by a separate letter. The COR monitors all technical aspects of the contract and assists in contract administration. The COR is authorized to perform the following functions: assure that the Contractor performs the technical requirements of the contract;

perform inspections necessary in connection with contract performance; maintain written and oral communications with the Contractor concerning technical aspects of the contract; issue written interpretations of technical requirements, including Government drawings, designs, and specifications; monitor Contractor's performance and notify both the KO and Contractor of any deficiencies; coordinate availability of Government-furnished property; and provide site entry of Contractor personnel. A letter of designation issued to the COR, a copy of which is sent to the Contractor, states the responsibilities and limitations of the COR, especially with regard to changes in cost or price, estimates, or changes in delivery dates. The COR is not authorized to change any of the terms and conditions of the resulting order.

1.6.11. Contractor Employees:

1.6.11.1. By the contract start date, the Contractor shall provide to the Contracting Officer and COR a list of employees who will perform the functions outlined in the PWS, paragraph 1.6.12.1.

1.6.11.2 Within 20 working days of the award of the contract, the Contractor shall provide to the Contracting Officer and COR a list of employees who will perform the functions outlined in the PWS, paragraph 1.6.12.2 & paragraph 1.6.12.3.

1.6.11.3. The Government shall have the right to review the qualifications of each personnel and subsequently approve/disapprove proposed key personnel (paragraph 1.6.12).

1.6.11.3. The Contractor shall maintain a list of key Contractor personnel including the full name of each employee, job title or position held by each employee, work assignment, and Job Position Description for each position under this contract. The Contractor shall maintain this list throughout the life of the contract and shall notify the Contracting Officer and COR, in writing, of any addition, deletion, or change within five (5) working days of such change.

1.6.12. Key Personnel: The following personnel are considered key personnel by the Government: Program Manager, ODA Developers, and English Editors. The contractor shall provide a contract manager who shall be responsible for the performance of the work. The name of this person and an alternate who shall act for the contractor when the manager is absent shall be designated in writing to the contracting officer. The contract manager or alternate shall have full authority to act for the contractor on all contract matters relating to the daily operation of this contract. The contract manager or alternate shall be available between 7:45 to 16:45 PST, Monday thru Friday except for Federal holidays or when the Government facility is closed for administrative reasons. Qualifications for all key personnel are listed below:

1.6.12.1. Program Manager (PM)

1.6.12.1.1. The Contractor shall designate to the Contracting Officer and COR a program manager and alternate program manager (who shall perform the duties of the program manager in the absence of the program manager).

1.6.12.1.2. The program manager or alternate program manager shall provide overall management and coordination of the work required under this contract. The program manager shall be the central point of contact for the Government. The program manager or alternate shall be available for meetings with Government personnel as necessary within 24-hour notice.

1.6.12.1.3. The PM or alternate PM shall establish milestones to meet monthly delivery schedules. The PM shall monitor, track, and document stages of testlet development for each language. The PM shall ensure quality standards are met as well as provide technical expertise in testlet development.

1.6.12.1.4. The PM shall have experience in conducting training, reviewing, and evaluating developers’ work, monitoring all phases of development and review, meeting project deadlines, and communicating with stakeholders. Experience shall be no less than 2 years of recent relevant experience in areas described herein.

1.6.12.2. ODA Developers

1.6.12.2.1. The contractor shall provide ODA developers for each specified language. Each developer shall possess college-educated native or near-native proficiency in reading, writing, and speaking the target language.

Proficiency shall be demonstrated to meet the equivalent of ILR skill level in reading, writing, speaking, and listening.

1.6.12.2.2. Developer Proficiency. The Contractor shall provide written documentation to the Government within 20 working days of the award with the proficiency level of developers of at least level 3 in the target language, and at least level 3 in English, as measured by ILR (or ACTFL equivalent).

1.6.12.2.3. The developer shall have no less than two years’ experience in one or more of the following areas:

curriculum design, language teaching, and or item writing/test development.

1.6.12.2.4. It is recognized that in some circumstances it may not be possible to find developers meeting all these criteria. In such cases, the Government will determine developer eligibility on a case-by-case- basis. Deviations from the above standards must be approved by the Government prior to work being performed on this contract.

1.6.12.2.5. The Contractor shall provide resumes of proposed developers to the Government before a developer begins performance on any testlets. Resumes shall be sent to the Contracting Officer and COR no later than 20 working days after the contract award.

1.6.12.3. English Editor(s)

1.6.12.3.1. English Editor(s) must possess the equivalent of ILR skill level 4 in English and have at least a BA in a language-related field.

1.6.12.3.2. The English editor(s) shall also have no less than two years of experience in one or a combination of the following areas: (1) language testing in general, (2) linguistics, (3) language teaching, (4) literature or (5) editing experience.

1.6.13. Resumes shall be provided by the Contractor for each of the Key Personnel described at C.10.3 within 20 working days of award.

1.6.14. Supervision of Contractor Employees: The Government will not exercise any supervision or control over Contractor or subcontractor employees while performing work under the contract. Such employees shall be accountable solely to the Contractor, not the Government. The Contractor, in turn, shall be accountable to the Government for Contractor or subcontractor employees.

1.6.15. Conflict of Interest (OCI): The Contractor and subcontractor personnel shall certify that no employees working under this contract have a conflict of interest or the appearance of a conflict of interest.

2. DEFINITIONS AND ACRONYMS:

2.1. DEFINITIONS:

Answer Key An array or arrays of possible answers for each constructed response item.

Authentic Materials Materials written, spoken or produced by native speakers for native speakers in a native (target) language context.

Constructed Response Item

A type of question that requires the student to type in the answer, instead of selecting from a set of choices.

Content Item Question about the meaning of a text, events, details, ideas or arguments.

Contractor A supplier or vendor awarded a contract to provide specific supplies or service to the government. The term used in this contract refers to the prime.

Contracting Officer A person with authority to enter into, administer, and or terminate contracts, and make related determinations and findings on behalf of the government.

Note: The only individual who can legally bind the government.

Contracting Officer Representative (COR)

An employee of the U.S. Government appointed by the contracting officer to administer the contract. Such appointment shall be in writing and shall state the scope of authority and limitations. This individual has authority to provide technical direction to the Contractor as long as that direction is within the scope of the contract, does not constitute a change, and has no funding implications. This individual does NOT have authority to change the terms and conditions of the contract.

Defective Service A service output that does not meet the standard of performance associated with the Performance Work Statement.

Deliverable Anything that can be physically delivered, but may include non-manufactured things such as meeting minutes or reports.

Diagnostic Profile Feedback form the student receives at the end of an ODA session.

Diagnostic Profile Statement

A statement provided to the student in the diagnostic profile, generated by the relevant testlet and item tags.

Discourse Question addressing language elements beyond the clause or sentence level.

Examples include cohesive devices, such as conjunctions, or pronoun references.

FLO Topics Final Learning Objective topics: Geography and Environment; Military and Security; Economy and Politics; Science and Technology; and Culture and Society.

ILR Levels Levels described in the Interagency Language Roundtable Language Proficiency Skill Level Descriptions and Scale, as the standard tool of measurement and rating of proficiency and passages.

Item A question or a statement that elicits a response from a test-taker and answer keys associated with it.

Key Personnel

Contractor personnel that are evaluated in a source selection process and that may be required to be used in the performance of a contract by the Key Personnel listed in the PWS. When key personnel are used as an evaluation factor in best value procurement, an offer can be rejected if it does not have a firm commitment from the persons that are listed in the proposal.

Language-specific tags Metadata that is used only for structure (grammar) and discourse items. Each ODA language develops its own set of structure and discourse tags.

Linguistic Item Question about key vocabulary, sentence structures, and discourse relationships between ideas.

Main Proposition Question addressing the main idea, topic or issue of the passage.

Metadata Each testlet and item is assigned specific tags to identify their function in the assessment process.

Modified Audio Versions

For levels 2 and higher; exact transcripts are re-recorded in a controlled environment to minimize the effect of authentic rate of speech, background noise, or accent, etc. The modified version is used in combination with the authentic version for all of the testlets at levels 2 and higher.

Original Text The original text in its entirety as it appears in its source.

Proprietary Information

All information and any idea in whatever form, tangible and intangible, pertaining in any manner to the business of the Government or the Contractor or Subcontractor, which was produced by any Contractor or Subcontractor employee in the course of the employees’ employment or otherwise produced or acquired by or on behalf of the Government.

Quality Assurance The government procedures to verify that services being performed by the Contractor are performed according to acceptable standards.

Quality Assurance Surveillance Plan

An organized written document specifying the surveillance methodology to be used for surveillance of contractor performance.

Simulated Audio Passages

For levels 1 and 1+; audio passages are re-created around authentic situations, but edited for level appropriateness and recorded in a controlled environment.

Structure Question addressing syntactical elements or relations within a sentence or a clause. Examples include parts of speech, verb tense, or conjugations.

Subcontractors One that enters into a contract with a prime contractor. The Government does not have privity of contract with the subcontractor.

Sub-topics Breakdown of the ten (FLO) topics; see paragraph C.6.7.1.

Supporting proposition (1 & 2) Questions addressing supporting facts, details or arguments in a passage.

Testlet An assessment unit, consisting of one passage, content items and linguistic items.

Testlet Passage The abridged or modified version of the original text, as necessary and appropriate. This passage is used for the development of a testlet.

Main Cut An authentic audio recording no longer than 2 minutes used as the basis for a testlet and its items.

(Audio) Segment A smaller section of the main cut used for certain item types within the testlet.

Work Day The number of hours per day the Contractor provides services in accordance with the contract.

Work Week Monday through Friday, unless specified otherwise

2.2. ACRONYMS:

AR Army Regulation CFR Code of Federal Regulations CONUS Continental United States (excludes Alaska and Hawaii) COR Contracting Officer Representative DA Department of the Army DFARS Defense Federal Acquisition Regulation Supplement DMDC Defense Manpower Data Center DLIFLC Defense Language Institute Foreign Language Center DoD Department of Defense FAR Federal Acquisition Regulation FLO Final Learning Objective

ILR Interagency Language Roundtable KO Contracting Officer OCI Organizational Conflict of Interest OCONUS Outside Continental United States (includes Alaska and Hawaii) ODA Online Diagnostic Assessment POC Point of Contact PRS Performance Requirements Summary PSD Performance Start Date PWS Performance Work Statement QA Quality Assurance QAP Quality Assurance Program QASP Quality Assurance Surveillance Plan QC Quality Control QCP Quality Control Plan

3. GOVERNMENT FURNISHED ITEMS AND SERVICES:

3.1. Materials: There will not be any GFP/E in this acquisition. The Government (DLIFLC) will provide access to the necessary software and upgrades to operate the ODA authoring tool, The Generator, including orientation/training, software, guidance, and technical assistance to use the authoring tool and associated templates. Everything else shall be contractor-furnished. Contractor access to the DLIFLC authoring tools shall be limited to this contract.

4. CONTRACTOR FURNISHED ITEMS AND RESPONSIBILITIES:

4.1. General: The Contractor shall furnish all supplies, equipment, facilities and services required to perform work under this contract that are not listed under Section 3 of this PWS.

4.2. Materials. See paragraph 4.1, above.

4.3. Equipment. See paragraph 4.1, above.

5. SPECIFIC TASKS:

5.1. ODA Testlets:

5.1.1. Each reading and listening testlet shall contain target language texts from authentic sources. These print or audio texts shall be accurately rated according to the ILR scales and related text typology standards.

5.1.2. Testlet Contents. Each testlet shall include the following:

5.1.2.1. Passage. One (1) strictly ILR-rated passage that is conducive to diagnostic assessment. See Section II of Technical Exhibit 8 for the passage selection specifications.

5.1.2.2. Content Items. Each testlet should include two to three content items, depending on the ILR level. The items should be clearly worded with well-defined correct answers. The items should include clear indicators of what is being elicited but should contain few or no clues to the passage’s content. See Technical Exhibit 1 for the number of content items required in accordance with ILR level, and Paragraph 3.4 at Section III of Technical Exhibit 8 for the content item writing specifications.

5.1.2.3. Linguistic Items. Each testlet contains 2- 3 linguistic probing items in accordance with the ODA level specifications (lexical, structural and discourse, respectively). The probing items should address relevant linguistic areas needed for comprehending the content items. Lexical items should target keywords or phrases that communicate the meaning of the main and supporting propositions in a passage. Structural items should address syntactical elements, or relations within a sentence or clause from the passage. Discourse items should address language elements beyond the clause or sentence level. See Technical Exhibit 1 for the number of linguistic items required in accordance with ILR level, and Paragraphs 3.5, 3.6, and 3.7 at Section III of Technical Exhibit 8 for the linguistic item writing specifications.

5.1.2.4. Answer Keys for short-answer items. Answer keys should be based upon a model answer, and alternate phrasing of the model answer. The answer keys should contain key words of the answers, and the key words should have synonyms and appropriate scoring for the answers. See Section IV of Technical Exhibit 8 for the answer key development specifications.

5.1.2.5. Options for Fixed-Response Items. The options contain one key and three distractors. The distractors should be plausible and based on likely misinterpretation.

5.1.2.6. Proper Metadata tagging. Testlet metadata includes information on skill/modality, level, FLO topic, and subtopic. Item metadata includes information on item function and language-specific features. See Section V of Exhibit 8 for the testlet metadata tagging specifications.

5.1.3. Each language iteration of testlets shall be accompanied by a list of language-specific tags and diagnostic-profile statements suitable to tag structure and discourse items and generate diagnostic profile statements for those two types of items. The diagnostic statements are to be written for general users in plain language to be delivered as feedback to non-linguists. See Section V of Exhibit 8 for the diagnostic statement development specifications.

5.2. Delivery/Acceptance Process

5.2.1. The Contractor shall deliver all required reading and listening testlets within the current year in accordance with the contract’s delivery schedule (see Technical Exhibits 6a & 6b).

5.2.2. The delivery/acceptance process consists of four (4) stages as follows:

5.2.2.1. Stage 1 – Passage Selection and Rating Approval:

5.2.2.1.1 The Contractor shall submit intended passages to be utilized in the development of testlets at the specified levels. The passages shall not be the same or alike to the ones that are already in the ODA system. The list of the passages that are in the system will be provided to the Contractor in initial post-award orientation.

5.2.2.1.2. Submission shall consist of (1) the source/original text selected from copyrighted material, (2) the testlet passage (abridged version of longer text, when necessary and appropriate (which will be discussed during orientation)); (3) assigned ILR level; and (4) an English translation of the passage that closely mirrors the testlet passage. For audio passages, target language transcriptions of the testlet passages are required.

5.2.2.1.3. The Government will offer guidance and corrective feedback regarding the ILR level of the text as well as suitability for use in the ODA project within four (4) working days. Note: If the Government requires additional time to complete the review process, the Contractor will be notified within four (4) working days.

5.2.2.1.3. In the event that a passage is rejected by the Government, the Contractor shall submit another passage within four (4) working days. Note: The primary criterion for a suitable passage is the designation of the appropriate ILR level.

5.2.2.2. Stage 2 – Review of Testlet Draft 1:

5.2.2.2.1. The Contractor shall submit the completed testlet to the Government in order to determine the level of acceptability of the testlet.

5.2.2.2.2. The Government may (1) Accept the testlet as is; (2) Reject the testlet completely or (3) Conditionally accept the testlet (with the understanding that the testlet requires revision to make it acceptable. The draft testlet will be returned to the Contractor for revision, i.e. the Contractor incorporates Government-recommended revisions and corrections and resubmits its revised draft testlet to the Government).

5.2.2.2.3. In the event of a testlet being returned for revision, the Government will provide in writing guidance and/or corrective feedback within 10 working days upon receipt of the draft 1 submittal.

5.2.2.2.4. If the testlet is acceptable, the Contractor will be notified accordingly.

5.2.2.3. Stage 3 – Review of Testlet Draft 2:

5.2.2.3.1. The Contractor shall resubmit its revised testlet to the Government within four (4) working days of receipt of Government feedback of Draft 1.

5.2.2.3.2. If the testlet is acceptable, the Contractor will be notified accordingly.

5.2.2.3.3. If the testlet is unacceptable as submitted, the Government will provide in writing, within four (4) working days, a second round of guidance and/or corrective feedback regarding technical requirements. If the Government requires additional time to complete its review process, the Contractor will be notified within four

(4) working days.

5.2.2.3.4. This process shall be repeated with the same timeframes until the Government receives an acceptable testlet.

5.2.2.4. Stage 4- English Editing:

5.2.2.4.1. The Contractor shall perform an internal quality control review that involves reviewing the use of English throughout the testlet development, including diagnostic profile statements (see paragraphs 5.1.3 & 5.5).

5.2.2.4.2. The Contractor shall submit the final draft testlet to the Government within four (4) working days from the date of receipt of feedback of Draft 2.

5.2.2.4.3. Upon Government notification of acceptance of the final draft testlet during this stage, the Contractor shall invoice for payment in full of the contract line-item cost of the testlet.

5.3. Audio Passages. All ILR guidelines for listening shall be observed. Audio passages at level 1 & level 1+ may be simulated, based on authentic passages. For levels 2 and higher, authentic audio passages shall be used.

In addition, all passages for level 2 and higher shall be re-recorded in a studio to be used as modified audio versions (see C.3.).

5.4. Listening Audio Files

5.4.1. All listening audio files shall be edited and formatted in accordance with ODA audio recording and processing guidelines (see Appendix J of Technical Exhibit 8).

5.4.2. The listening testlets shall have all audio files properly named and selected as indicated in Appendix J of Technical Exhibit 8.

5.5. List of Language-specific Tags and Diagnostic Profile Statements. Each language iteration of testlets (40 testlets per language) requires a list of language specific tags and Diagnostic Profile Statements. The list shall be used to tag structure and discourse items and generate diagnostic profile statements for those two types of items.

See Section V of Exhibit 8 for the diagnostic statement development specifications (Further details shall be addressed during orientation).

5.6. Final Learning Objectives (FLO) Sub-Topics

5.6.1. Each language shall have a well-balanced distribution of Final Learning Objectives (FLO) topics and sub-topics, as level permits. These topics are: Geography/Environment, Economy/Politics, Science/Technology, Military/Security, and Society/Culture. ODA sub-topics are listed at Paragraph 2.3, Section II of Technical Exhibit 8 (Further details will be provided in initial post-award orientation).

5.6.2. For the number of required FLO topics for each Level see Technical Exhibit 7 - Topic Distribution and Number of Testlets by Level.

5.7. Copyright Clearances

5.7.1. The Contractor shall use materials (audio, video, images, and texts) for which a copyright clearance has been obtained for DLIFLC. In the event that a new copyright clearance needs to be obtained, the Contractor shall secure and provide the requisite copyright clearance(s) in English to DLIFLC (see Technical Exhibits 10a & 10b).

5.7.2. The Contractor shall document and make available copyright clearances to DLIFLC for all materials used.

5.7.3. Proper citation of the copyrighted source shall be included in each testlet.

5.8. Authoring Tools

5.8.1. The Contractor shall use an authoring tool that has been developed by DLIFLC (see Technical Exhibit 9, Generator Manual, for the authoring tool usage).

5.8.2. Method of Access. DLIFLC will decide on the method of access to the authoring tool, either by providing the Contractor an electronic standalone copy or by making it only available online.

5.8.3. Orientation in the Use of Authoring Tools. DLIFLC will provide orientation as necessary in the use of the authoring tool, related teaching methodology, and instructional design requirements at the onset of the project at the Contractor’s site (time and date to be finalized at the pre-performance meeting).

5.8.4. Technical Guidance and Support by DLIFLC. DLIFLC will provide technical guidance and support for the authoring tool throughout the contract period(s). Although DLIFLC may decide to travel to the Contractor’s site for additional training, it is expected of the Contractor to train additional personnel by contractor-furnished personnel who have already received orientation training by DLIFLC.

5.9. Monthly Report. Provide a monthly status report to the COR which includes:

● Contract/Task Order Number and Title

● Period covered by the report

● Date and Name of preparer

● Milestones/Date Due/Government Due/Comments

● Task Title(s) and Number(s)

● Status of deliverables

● Highlight any Deviation from the PWS

● Efforts this Reporting Period

● Efforts Projected for Next Reporting Period

● Problem Areas

● General Comments

Deliverable: Monthly Status Report completed by the 10th working day of the following month. This report shall be submitted electronically to the COR, in an MS Office format.

MAS: No more than 2 reports submitted late. 100% accuracy of the content presented, (minor clerical mistakes that are merely a matter of form and not of substance will be noted for correction).

6. DELIVERABLES

6.1. Project Milestone Schedule

6.1.1. The Contractor shall deliver Project Milestone Charts, for approval by the Contracting Officer and COR, in the form of a bar chart within 20 working days of contract award and award of option periods. The Charts shall reflect the proposed work schedule to meet the required contract delivery schedule.

a. The project execution chart shall include the following minimum information:

(1) A breakdown by major headings for primary work activity including Stage 1, Stage 2, Stage 3 and Stage 4 as described at C.6.2.2.

(2) A breakdown of the various phases under each major stage sufficient to track the progress of the work.

6.1.2. A Project Milestone Chart shall be provided for each of the required 40 testlets per language.

6.1.3. The Contractor shall provide an excel spreadsheet reflecting projected milestones electronically to the Contracting Officer and COR the last workday of each month. The spreadsheet shall reflect actual versus projected dates. A written report shall be submitted with the spreadsheet providing an explanation of any deviation as well as the plan to get the work back on track.

6.2. Testlet Delivery

6.2.1. The Contractor shall deliver 40 testlets per language to the COR within the prescribed delivery period (see Technical Exhibits 6a and 6b).

6.2.1.1. Testlets shall be in digital format, and in the most current version of the DLIFLC ODA authoring tool, The Generator (see Technical Exhibit 8 for the manual).

6.2.1.2. Legal copyright clearances shall be provided at the same time as testlet delivery.

6.2.2. DLIFLC will accept or reject with possible corrections to make acceptable the subject testlets.

6.3. Correspondence. The Contractor shall provide all correspondence through the COR.

6.4. This table lists any reports or documentation that is required as a deliverable to include the frequency, medium/format and who it is to be submitted. A deliverable is anything that can be physically delivered but may include non-physical things such as verbal communications.

DELIVERABLE REQUIRING

DIRECTIVE

FORMAT METHOD AND

FREQUENCY

SUBMIT TO

Quality Control Plan PWS 1.6.1 Word document

Within 30 days after contract award and within 5 working days when changes are made thereafter

KO AND COR

Non-Disclosure Agreement PWS 1.6.7.3 Gov’t provided format

Within 15 days of award and upon changes

KO AND COR

Monthly Status Report

(MSR)

PWS 5.9 Excel spreadsheet and/or Word document

Due by 10th of each month

KO AND COR

Milestone Schedule/Chart PWS 6.1 Excel spreadsheet and/or Word document

Within 20 working days after contract award

KO AND COR

Testlets PWS 6.2 As required Per Milestone Schedule

COR

Legal Copyright Clearances

PWS 6.2.1.2 As required With Testlets per Milestone Schedule

COR

7. APPLICABLE PUBLICATIONS (CURRENT EDITIONS) AND DOCUMENTS

7.1 Publications

Army Regulation (AR) 611-5, Personnel and Classification Testing, paragraphs 3-1 and 3-2

7.2. Documents:

Technical Exhibit 1 – ODA Testlet Content and Structure

Technical Exhibit 2 – ODA Languages, Skills and Distribution of ILR levels, Base Year

Technical Exhibit 3 – ODA Languages, Skills and Distribution of ILR levels, Option Year

Technical Exhibit 4 - Interagency Language Roundtable Reading Skill Level Descriptions. For more details of the ILR, go to the following websites:

Reading-- http://www.govtilr.org/Skills/ILRscale4.htm Home Page-- http://www.govtilr.org/

Technical Exhibit 5 – Interagency Language Roundtable Listening Skill Level Descriptions. For more details of the ILR, go to the following websites:

Listening-- http://www.govtilr.org/Skills/ILRscale3.htm Home Page-- http://www.govtilr.org/

Technical Exhibit 6a – ODA Delivery Schedule for Each Language (Template)

Technical Exhibit 6b – ODA Delivery Schedule by Level

Technical Exhibit 7 – Topic Distribution and Number of Testlets

Technical Exhibit 8 – ODA Developers’ Guidelines

Technical Exhibit 9 – Generator Manual for ODA

Technical Exhibit 10a – Sample Copyright Release Letter

Technical Exhibit 10b – Copyrighted Cleared Sources

Technical Exhibit 11 – Performance Requirements Summary (PRS)

*TECHNICAL EXHIBITS 6a, 6b, 8, 9, and 10b will be given upon the contract award.

TECHNICAL EXHIBIT 1

TESTLET CONTENT AND STRUCTURE

A testlet is the operating unit of Online Diagnostic Assessment. A testlet consists of a passage and four to six items, depending on the ILR level. The items are of two categories: the first category targets core content and functions; and the second category targets the linguistic elements that underlie the core content and functions.

Requirements for each level as follows:

Level 1 Level 1+ Levels 2, 2+ and 3 Passage* Passage Passage

Content Items

1. Main Proposition 1. Main Proposition 1. Main Proposition

2. Supporting Proposition 2. Supporting Proposition 2. Supporting Proposition 1

3. Supporting Proposition 2 Linguistic Items

3. Lexicon – 5 vocabulary items

3. Lexicon – 6 vocabulary items

4. Lexicon – 7 vocabulary items

4. Structure 4. Structure 5. Structure

5. Discourse 6. Discourse

For definitions see PWS 2.1.

Passage* - Passage Length is as follows:

Passage Length is approximate and will vary from one language to another. These are averaged parameters, with occasional outliers.

READING COMPREHENSION

Level 1 Level 1+ Level 2 Level 2+ Level 3 50-100 words 75-150 words 125-250 words 200-300 words 250-350 words

Listening COMPREHENSION Level 1 Level 1+ Level 2 Level 2+ Level 3 15-30 seconds 25-50 seconds 25-50 seconds 50-80 seconds 60-90 seconds

Grammar Level 1 Level 1+ Level 2 Level 2+ Level 3 50-100 words 75-150 words 125-250 word 200-300 words 250-350 words 15-30 seconds 25-50 seconds 25-50 seconds 50-80 seconds 60-90 seconds

TECHNICAL EXHIBIT 2

ODA Languages, Skills and Distribution of ILR levels Base Year

(Arabic-MSA, Russian, North Korean, Chinese-Mandarin, and Persian-Farsi)

Language Skill

ILR Skill Level

Total1 1+ 2 2+ 3 # of testlets # of testlets

# of testlets

# of testlets

# of testlets

Arabic-MSA Reading 6 12 10 6 6 40

Russian Reading 6 12 10 6 6 40

North Korean Reading 6 12 10 6 6 40

Mandarin Reading 6 12 10 6 6 40

Persian-Farsi Reading 6 12 10 6 6 40

ODA Languages, Skills and Distribution of ILR levels Option Year 1

(Arabic-MSA, Russian, North Korean, Chinese-Mandarin, and Persian-Farsi)

Language Skill

ILR Skill Level

Total1 1+ 2 2+ 3 # of testlets # of testlets

# of testlets

# of testlets

# of testlets

Arabic-MSA Listening 6 12 10 6 6 40

Russian Listening 6 12 10 6 6 40

North Korean Listening 6 12 10 6 6 40

Mandarin Listening 6 12 10 6 6 40

Persian-Farsi Listening 6 12 10 6 6 40

TECHNICAL EXHIBIT 3

ODA Languages, Skills and Distribution of ILR levels

Option Year 2, 3, & 4

Language Skill

ILR Skill Level

Total1 1+ 2 2+ 3 # of testlets # of testlets

# of testlets

# of testlets

# of testlets

TBD TBD 6 12 10 6 6 40

TBD TBD 6 12 10 6 6 40

TBD TBD 6 12 10 6 6 40

TBD TBD 6 12 10 6 6 40

TBD TBD 6 12 10 6 6 40

TECHNICAL EXHIBIT 4

Interagency Language Roundtable Language Skill Level Descriptions

Source: http://www.govtilr.org/

Reading R-1: Reading 1 (Elementary Proficiency) Sufficient comprehension to read very simple connected written material in a form equivalent to usual printing or typescript.

Can read either representations of familiar formulaic verbal exchanges or simple language containing only the highest frequency structural patterns and vocabulary, including shared international vocabulary items and cognates (when appropriate). Able to read and understand known language elements that have been recombined in new ways to achieve different meanings at a similar level of simplicity. Texts may include descriptions of persons, places or things: and explanations of geography and government such as those simplified for tourists. Some misunderstandings are possible on simple texts.

Can get some main ideas and locate prominent items of professional significance in more complex texts. Can identify general subject matter in some authentic texts.

R-1+: Reading 1+ (Elementary Proficiency, Plus) Sufficient comprehension to understand simple discourse in printed form for informative social purposes. Can read material such as announcements of public events, simple prose containing biographical information or narration of events, and straightforward newspaper headlines. Can guess at unfamiliar vocabulary if highly contextualized, but with difficulty in unfamiliar contexts. Can get some main ideas and locate routine information of professional significance in more complex texts. Can follow essential points of written discussion at an elementary level on topics in his/her special professional field. In commonly taught languages, the individual may not control the structure well. For example, basic grammatical relations are often misinterpreted, and temporal reference may rely primarily on lexical items as time indicators. Has some difficulty with the cohesive factors in discourse, such as matching pronouns with referents. May have to read materials several times for understanding.

R-2: Reading 2 (Limited Working Proficiency) Sufficient comprehension to read simple, authentic written material in a form equivalent to usual printing or typescript on subjects within a familiar context. Able to read with some misunderstandings straightforward, familiar, factual material, but in general insufficiently experienced with the language to draw inferences directly from the linguistic aspects of the text. Can locate and understand the main ideas and details in material written for the general reader.

However, persons who have professional knowledge of a subject may be able to summarize or perform sorting and locating tasks with written texts that are well beyond their general proficiency level. The individual can read uncomplicated, but authentic prose on familiar subjects that are normally presented in a predictable sequence which aids the reader in understanding. Texts may include descriptions and narrations in contexts such as news items describing frequently occurring events, simple biographical information, social notices, formulaic business letters, and simple technical material written for the general reader. Generally, the prose that can be read by the individual is predominantly in straightforward/high-frequency sentence patterns. The individual does not have a broad active vocabulary (that is, which he/she recognizes immediately on sight), but is able to use contextual and real-world cues to understand the text.

Characteristically, however, the individual is quite slow in performing such a process. Is typically able to answer factual questions about authentic texts of the types described above.

R-2+: Reading 2+ (Limited Working Proficiency, Plus) Sufficient comprehension to understand most factual material in non-technical prose as well as some discussions on concrete topics related to special professional interests. Is markedly more proficient at reading materials on a familiar topic. Is able to separate the main ideas and details from lesser ones and uses that distinction to advance understanding. The individual is able to use linguistic context and real-world knowledge to make sensible guesses about unfamiliar material. Has a broad active reading vocabulary. The individual is able to get the gist of main and subsidiary ideas in texts which could only be read thoroughly by persons with much higher proficiencies. Weaknesses include slowness, uncertainty, inability to discern nuance and/or intentionally disguised meaning.

R-3: Reading 3 (General Professional Proficiency) Able to read within a normal range of speed and with almost complete comprehension a variety of authentic prose material on unfamiliar subjects. Reading ability is not dependent on subject matter knowledge, although it is not expected that the individual can comprehend thoroughly subject matter which is highly dependent on cultural knowledge or which is outside…