OpenMT_Human_Assessments_SOW.doc

1. OpenMT – Human Assessments 2015 National Institute of Standards and Technology (NIST) – Statement of Work (SOW)

2. Background The US Department of Commerce, National Institute of Standards and Technology (NIST), Information Technology Laboratory (ITL), Information Access Division (IAD) advances and promotes the use of technologies for the intuitive, efficient and effective access, manipulation, analysis, and exchange of complex information through collaboration with industry, academia, and other government agencies to research and develop metrics, evaluation methodologies, test data, prototypes, test beds, and standards.

The IAD Multimodal Information Group supports Human Language Technology (HLT) research and development objectives through collaborations with other agencies to advance HLT systems in support of speech analysis, video analysis, machine translation and speaker and language recognition technologies.

Machine Translation is a technology that automates the translation of a data (text, image or audio), either written or spoken, in one language to that of another. This translation technology is emerging as an effective communication tool for industry, academia, and our military, and this technology is enabling greater information access, a goal of the NIST Information Access Division. NIST supports the government-wide effort to increase machine translation capabilities through research and evaluation designed to address factors negatively effecting translation performance. NIST designs experimental test protocols, provides guidance to help ensure that test data is collected as appropriate for addressing identified challenges, develops metrics and implements community evaluation that focus research on specific problems preventing the technology from reaching its full potential.

This procurement requires the performance of tasks and deliverables the Multimodal Information Group requires to fulfill their mission of advancing the measurement science for machine translation evaluation of text input. More specifically this procurement will generate the human assessments of machine translation output for informal genres that can be scientifically compared to automated metrics. The types of assessments are:

· Bilingual Human Assessment of Semantic Adequacy: Bilingual human judges (Arabic/English & separately Chinese/English) view a segment of text representing the original source data and the resulting machine translation output. Each judge provides their assessment of quality based on a Likert scale, answering the question “Does the translation have the same meaning as the original source data?”.

· Monolingual Human Assessment of Semantic Adequacy: English speaking judges view 2-4 versions of a human generated reference translation and an automated machine translation of the same segment. Each judges provides their assessment of quality based on a Likert scale, answering the question “Does the translation have the same meaning as the reference(s)?”.

3. Scope

The general scope and research areas for this procurement are:

· Human Language Technology

· Linguistics

· Machine Translation evaluation

4. Purpose and Objectives The objective for this procurement is to extend existing NIST Test & Evaluation capabilities to include a structured human in the loop assessment metric that will also serve as a reference point for the correlation of automatic machine translation metrics. The purpose of this procurement is to perform tasks in human assessment of machine translation output supporting the NIST 2015 OpenMT evaluation. The tasks required to support these objectives include:

· Provide (5) five independent human judgments of semantic adequacy for each test segment using Bilingual judges (Arabic/English).

Note: the number of words per test segment will vary. SMS/CHAT segments are typically very short. Conversation Telephone Speech transcripts (CTS) are typically much longer. The total number of words to be assessed is measured from the original source data (Arabic). The following table identifies the number of words to be assessed:

500,000 words

· Provide (5) five independent human judgments of semantic adequacy for each test segment using Bilingual judges (Chinese/English).

Note: the number of words per test segment will vary. SMS/CHAT segments are typically very short. Conversation Telephone Speech transcripts (CTS) are typically much longer. The total number of words to be assessed is measured from the original source data (Chinese). The following table identifies the number of words to be assessed:

500,000 words

· Provide (5) independent human judgments of semantic adequacy for each test segment using Monolingual judges (English).

Note: the number of words per test segment will vary. SMS/CHAT segments are typically very short. Conversation Telephone Speech transcripts (CTS) are typically much longer. The total number of words to be assessed is measured from the original source data (Arabic & Chinese). The following table identifies the number of words to be assessed:

1,000,000 words

5. Contract Type

The contract type is Firm-Fixed-Price (FFP)

6. Place of Performance

The work shall be performed the contractor’s site.

7. Period of Performance

The period of performance for this procurement is 6-months (December 1, 2014 through May 30, 2015).

8. Specific Tasks

The following are the specific tasks to be performed by the contractor.

Task 1 – Provide a roster of bilingual and monolingual judges identifying the judges by a coded Personal Identification Number or PIN. This roster will be used by NIST to define the amount of data to assign to each judge for each data set.

· The Bilingual roster shall include a PIN (representing a unique judge), their native language, the language data set they will assess (Arabic or Chinese) and a description of how they were deemed fluent in both the language data they will assess and in English.

· Bilingual judges must be able to read textual content in the language they will assess and in English

· Bilingual judges must be a native of the source language and fluent in English.

· A minimum of 5 and a maximum of 25 bilingual judges may be used to assess the Arabic data set

· A minimum of 5 and a maximum of 25 bilingual judges may be used to assess the Chinese data set

· The Monolingual roster shall include a PIN (representing a unique judge), their native language, the language data set they will assess (English) and a description of how they were deemed fluent in English.

· Monolingual judges must be able to read textual content in English

· Monolingual judges must be fluent in English

· Monolingual judges must have higher than a high school education

· A minimum of 10 and maximum of 50 monolingual judges may be used to assess the English data set Task 2 – Verify that the NIST provided software used to perform the assessments works on the contractor’s computational resources. The software will be written in Java to maximize its portability. Documentation for the assessment tool use and functionalities, the judges' instructions, and the validation data set will be provided. Specific requirements for this task include:

· Contractor is to install the software locally on contractor systems.

· Exercise the tool by performing assessments on a NIST provided validation set.

· Return resulting assessments to NIST for validation. NIST will validate proper format and contractor understanding of the Likert scale.

Task 3 – Instruct bilingual and monolingual judges. Using the software installed and validated in Task 2, contractor will instruct the pool of bilingual and monolingual judges on the assessment tool and the task to be performed using the NIST provided training instructions and a training data set.

· Contractor feedback regarding possible improvements to the tool and instructions is requested but optional.

· Contractor is to instruct the judges on the task to be performed and assess the judges’ competency by having the judges performing assessment on a NIST provided training data set.

Task 4 – Perform Bilingual Assessments. NIST will provide the contractor with an excel spreadsheet that assigns data sets to particular judges (using their PIN). (In the case that a judge drops out midway through the process it is preferred that a completely new judge finishes the assignments. Concerns and issues such as this should be discussed with the NIST technical project lead before continuing.) NIST will provide individualized data sets to be given to each judge. These data sets will follow a specific scientific design to appropriately mix the output of various systems across the panel of judges such that it will maximize NIST’s intended analysis. Because of this, it will be important that a single judge completes their assigned data sets. Specific requirements for this task include:

· Judges shall perform the bilingual human assessments of adequacy using the NIST-provided software for all data assigned. For this test the judge will view a segment of text in the foreign source language and the corresponding machine translation. With this information they will answer the question: “Does the translation have the same meaning as the source?”

· Contractor is to return the files created by the NIST software tool that contains the Bilingual assessments.

Task 5 – Perform Monolingual Assessments. NIST will provide the contractor with an excel spreadsheet that assigns data sets to particular judges (using their PIN). (In the case that a judge drops out midway through the process it is preferred that a completely new judge finishes the assignments. Concerns and issues such as this should be discussed with the NIST technical project lead before continuing.) NIST will provide individualized data sets to be given to each judge. These data sets will follow a specific scientific design to appropriately mix the output of various systems across the panel of judges such that it will maximize NIST’s intended analysis. Because of this, it will be important that a single judge completes their assigned data sets. Specific requirements for this task include:

· Judges shall perform the monolingual human assessments of adequacy using the NIST-provided software for all data assigned. For this test the judge will view 2-4 human generated reference translation of a segment of text and the corresponding machine translation. With this information they will answer the question: “Does the translation have the same meaning as the reference(s)?”

· Contractor is to return the files created by the NIST software tool that contains the Monolingual assessments.

Task 6– Monthly and Final Reporting

· Provide monthly reports to update NIST COR and NIST technical lead of project status and issues

· Provide end of period reports to update NIST COR and NIST technical lead of project status and issues

9. Technical Requirements

The contractor shall identify the qualifying judges that will be used to perform the human assessments. The contractor shall validate installation of NIST provided software (to be used to perform the assessments) on the contractor’s computational resources. The contractor shall perform bilingual and monolingual assessments of the NIST provided data sets, using the NIST provided software tool. The contractor will return to NIST the resulting assessments as stored by the NIST provided software tool in electronic form.

· The dialect for the Arabic data will be Egyptian Arabic

· The dialect for the Chinese data will be Mandarin Chinese

· Bilingual judges shall be native of the language data they will assess (Arabic or Chinese)

· Monolingual judges shall be native of the United States

· Monolingual judges shall have been educated at a level higher than high school

· Bilingual and Monolingual judges shall be fluent in English

· The number of judges used should ensure completion of all assessments in a 3-month window (February-April 2015)

· NIST provided assessment software shall be used

· NIST provided assignments of data sets to judges must be followed to the extent possible

10. Other Pertinent Information or Special Considerations

· The Government estimates 300 person days (6 hr days) are necessary to accurately assess 2,000,000 words of data in support of OpenMT 2015 as required by this procurement.

· Table 1 is a representative sample of the type of source data and resulting machine translation quality that can be expected in the data sets to be assessed:

Table 1: a representative sample of the data included for assessment.

11. Deliverables/Delivery Schedule

Monthly Status and End of Period Progress Summary Reports – The Contractor shall prepare and submit Monthly Status and Progress Summary Reports. Reports can be in either email body or MS Word format and shall be forwarded to the NIST COR and NIST project manager by the 10th calendar day of the month following the reporting period.

The Monthly Report shall be a compilation of management issues, meetings and work performed during the previous month in sufficient detail to ensure NIST understanding of any issues and progress.

The format and content of the delivered Reports shall include the following:

· Work and deliverables completed during the reporting period

· Status of on-going activities

· Summary of tasks performed under section 8 (Specific Tasks) of the Statement of Work

· Problems or issues identified

· Alternatives and/or recommended solution(s) for identified or projected problems or issues

Final Reports – The contractor shall provide a final report in MS Word and/or Excel format(s) to the COR 15-30 days prior to the period’s conclusion of the period of performance. These reports shall summarize objectives achieved and, if any, significant issues, problems, and recommendations to improve the process in the future. The final report shall also include the deliverables listed in the Deliverables table below.

Deliverable to include a summary document identifying the number of judges to be used for the bilingual and monolingual assessments.

Deliverable to include a single electronic file containing an independent assessment from a NIST provided validation set.

Deliverable to include documented training results.

Due April 30th 2015.

Deliverable to include the full collection of electronic files (output from the NIST provide assessment tool) for 100% of the Bilingual judgments (20000 words X 10 systems x 5 independent judgments).

Due April 30th 2015.

Deliverable to include the full collection of electronic files (output from the NIST provide assessment tool) for 100% of the Monolingual judgments (20000 words X 10 systems x 5 independent judgments).

12. Quality Assurance Surveillance Plan (QASP) - Post-Award Administration

The following will be used for evaluation of Contractor work quality and progress.

I. Objective:

The purpose of the Quality Assurance Plan is to provide quality-monitoring procedures to assure that the Contractor is meeting the requirements in a reasonable period of time and of acceptable quality. The Quality Assurance Plan shall help ensure the COR has a method for detecting and remedying any performance deficiencies early on.

II. Performance Standards:

A. Quality Level:

The Contractor shall be monitored by their monthly reports to determine if the requirements are being met. The Contractor is to document plan changes as determined in ad hoc meetings with NIST staff in a monthly progress report to the COR.

B. Frequency:

The COR or their designees shall adjust the frequency of reports or ad hoc meetings with the contractor based on determination of work progress and the need for planning review.

C. Management Responsiveness:

The COR or their designees shall determine whether the Contractor has managed the requirements of the procurement effectively and efficiently with successful and timely compilation of requirements as set forth in the Performance Work Statement. The COR shall confirm whether the Contractor has satisfactorily met all reporting requirements.

III. Evaluation Methods:

The COR shall conduct performance evaluations based upon Section II above and the required performance levels set forth in the order. The following techniques shall be used to perform evaluation:

A. Materials Review:

The COR and/or designees shall review the materials generated by the Contractor including progress reports and any of the following: roster documentation, training documentation, software installation validation, validation response test sets, assessment files, software documentation, technical reports software comments, technical presentations, technical papers, Web pages, data compilations, memoranda and other documents intended for publication.

B. Performance Tests:

The COR and/or designees shall review the data compilations (generated assessments) generated by the Contractor and within NIST will determine if it is meeting the documented categories of semantic adequacy and is of acceptable quality.

Returned assessments shall be reviewed by the NIST technical staff for completeness and reasonable consistency across judges. It is expected that variation amongst judges will exist, but patterns suggesting ineffective assessments or guessing attributed to a particular judge PIN will be investigated and require reassessment by the contractor at their own costs for all data assigned to the identified judge(s), if demonstrable signs of randomness or guessing exist.

C. Function Tests:

NIST users associated with the project work tasks shall implement functional tests of the data compilations generated by the contractor and report the results to the COR or designees.

13. Risk Assessment The information technology security risk for service contract personnel is low.

14. Security

There will be no on-site contractor personnel.

15. Government Furnished Equipment (GFE)/ Government Furnished Information (GFI)

The government will provide:

· Java-based semantic adequacy assessment software as outlined in this statement of work

· Training data and documentation for use as training assessments of potential judges

· An installation validation assessment set

· Data assignments for individual judge assessments

· Assembled data sets for individual judge assessments

· All software licenses necessary for the completion of the work outlined in this statement of work

16. Transition Plan

A transition plan is not necessary for this contract.