Attachment0009DemonstrationPlan.pdf

UNCLASSIFIED

Machine Foreign Language Translation System

(MFLTS)

Text-to-Text

Demonstration Plan

26 April 2018

1. DISTRIBUTION STATEMENT: Distribution authorized to the Department of Defense (DoD) and U.S. DoD Offerors (Administrative Use, 28 March 2013). For other requests for this document, refer to the Product Manager, Machine Foreign Language Translation System, SFAE-IEW-DCG- MFLT, Building 363, 10115 Gratiot Road, Fort Belvoir, VA 22060-5812, 703-704- 4572Introduction.

The Technical Demonstration will be conducted as a portion of the source selection for the MFLTS Text-To-Text (T2T) Language Pack (LP) Solicitation W909MY-18-R-0011. This document describes the Offeror demonstration element of the Government proposal review process. One demonstration will occur for each LP proposed by the Offeror. The Government will evaluate the data acquired after the demonstration(s) as defined in this demonstration plan and Section M of the solicitation. The Offeror will conduct the LP demonstration using Government Furnished Software collection tools, and data to demonstrate that the demonstrated LP(s) meets the Government’s LP stated requirements.

2. Scope.

The intent of the demonstration is for the Government to collect the Offeror’s LP performance data of the Machine Translation (MT) engine, and the Optical Character recognition (OCR) engine in accordance with (IAW) the technical demonstration evaluation non-priced factors, as defined in the solicitation.

3. MFLTS System Description.

3.1 The MFLTS is a family of software applications that are installed on a variety of receiving programs’ host hardware platforms to provide a language translation capability to soldiers in the field. An application is comprised of software and data in several system configurations. To enable language translation for a specific language pair (English and one target language). The MFLTS Text to Text (T2T) SW Application (App), with an installed T2T Language Pack, provides all the software functionality required to perform a document translation task. T2T LPs consist of a MT Language Component (LC), which performs the translation function, and may include an OCR LC that processes image document inputs. MFLTS T2T configurations are as follows:

3.1.1 Mobile Configuration. This version of the MFLTS SW App is configured for host platforms that are typically laptop computers that utilize a Windows Operating System and have limited or no network connectivity. An example of a mobile platform is the Dell M7710 / Dell M7720 or equivalent system with a Windows 10 operating system. This is the configuration that will be utilized to perform the Qualification Test.

3.1.2. This configuration of the MFLTS App is used on a server platform that utilizes a Windows- Web Operating System and implements web services to provide a language translation capability to remote users via closed, restricted network. The users interact with the software via a web-based graphical user interface. This configuration will not be utilized during this Qualification Test.

4. Demonstration Approach.

Demonstration Concept. The intent of the technical demonstration is to demonstrate bilingual T2T products, with MT and OCR products for designated languages IAW the solicitation.

Offerors will conduct a technical demonstration of proposed LP(s), witnessed by the Government at a location designated by the Government. The Offeror must provide the hardware platform with all software necessary to demonstrate the proposed LP(s), such as Operating System, Offeror Language Translation application, Government Furnished performance evaluation software tools (Table 5-1). The Offeror may have up to three (3) personnel participate in the demonstration and is responsible for operating the proposed system and recording all performance data. The Government will provide demonstration Corpora (Table 5-1) for use by the Offeror at the time of the demonstration event.

4.2. Demonstration Procedures. Offeror’s will conduct a Technical Demonstration of the proposed Commercial Off the Shelf (COTS) Language Pack in accordance with the procedures provided in Section 5 of this plan. Offerors are responsible for recording Processing Speed (Words per Minute (WPM)); and processing the required image and electronic File Type Check (FTC) files, which must be processed through proposed MT and/or OCR software. Additionally, the Offeror will provide the translated output associated with the Government provided Corpora for use in a Reading Comprehension Test (RCT). The RCT is an evaluation of the translation effectiveness of the T2T LP that will be conducted by the Government. The Government Technical Demonstration team will evaluate the resulting RCT scores as well as the WPM, and FTC test results IAW section M of the solicitation.

The Offeror will be provided one opportunity to demonstrate each of the below efforts on their primary provided laptop. If there is an issue with performance of the primary laptop, the Offeror will be provided one opportunity to demonstrate each of the below efforts on their secondary laptop. If the Offeror is unable to perform the demonstration with the secondary laptop, the demonstration will be halted and the Offeror will be rated as Unacceptable in the uncompleted demonstrations.

If the Offeror is unable to successfully perform the required language pack demonstration, the Government may determine that the proposed language pack is unacceptable.

If the Offeror is not present for demonstration at the provided date and time, their proposed language pack(s) may also be evaluated as unacceptable.

THE GOVERNMENT ASSUMES NO LIABILITY OR RESPONSIBILITY FOR ANY

EQUIPMENT AND PERSONNEL PROVIDED.

5. Language Pack Components and Associated Performance Metrics.

Language Component (LC). Offeror’s will present all T2T LP components (OCR and MT) being proposed in response to the Request for Proposal (RFP) for demonstration. The Offeror will record the performance of the demonstrated LP components on the Government furnished Technical Demonstration Data Collection Sheet, which the Offeror will complete and a copy will be provided to the Offeror.

5.1.1. Machine Translation (MT). MT language components translate electronic text from one language to electronic text in another language. In order to meet the requirement for two-way text translation, all MFLTS T2T LPs will have the following two (2) MT LCs:

a) One LC that translates from a foreign language to English

b) One LC that translates from English to the foreign language

5.1.2. Optical Character Recognition (OCR). The OCR language component converts an image file (such as PDF Image, JPEG) containing text or characters into an electronic text file, which is a transcription of the text within the image file. The output of the OCR language component becomes the input to the MT LC, and is to be captured by the Offeror as a separate file.

5.2. LP Performance Metrics. There are five Technical Performance parameters that are measured and used to characterize the technical performance of a LP, and are listed below:

a) Translation Effectiveness. The Reading Comprehension Test (RCT) characterizes the translation effectiveness of a LP by testing the ability of human test subjects to read and comprehend the content of text that has been translated by a LP

b) Processing Speed. This is the rate (in Words per Minute) at which a LP processes and translates foreign language documents into English

c) File Type Compatibility. The File-Type Check (FTC) test confirms that a LP can ingest and process documents that are of different file formats or “file-types”.

d) Translation Accuracy. The Bilingual Evaluation Understudy (BLEU) test measures the translation accuracy associated with a Machine Translation LC

e) OCR Accuracy. The Word Error Rate (WER) test measures the performance of an Optical Character Recognition LC

5.2.1. Technical Demonstration performance testing. The tests associated with a) Translation Effectiveness, b) Processing Speed, and c) File Type Compatibility will be performed during the Offeror’s Technical Demonstration of the proposed COTS LP. All of the metrics collected during the Offeror’s demonstration will be evaluated as described in Section M of the RFP, and will become the performance benchmark that will be used by the Contractor when performing the Qualification Test of the integrated LP. Table 5-1 identifies which Government-furnished document Corpus is to be used to perform the demonstration tests.

Table 5-1 Performance Benchmarks Metric Tech Perf measure Test Corpus to be used:

*Processing Speed in Words per Minute (WPM)

This is a calculation of the speed at which the LP processed the Government furnished documents.

Corpus 3 Corpus 4

*Reading Comprehension Test

(RCT)

For machine translations, an effectiveness tool that measures test subjects' ability to understand essential elements of information in a document

Corpus 3 Corpus 4

*File Type Check (FTC) This process verifies the ability to process additional file types using the above tools.

Corpus 5a Corpus 5b

* Performance benchmarks collected during technical demonstration.

5.2.2. Post-Award contractor performance testing. The tests and performance benchmarks associated with d) Translation Accuracy (BLEU), and e) OCR Accuracy (WER) are not part of the Technical Demonstration of the COTS LP. These tests are required in the contract Performance Work Statement (PWS) and will be performed by the contractor subsequent to award of the contract. The metrics collected from these pre-integration tests also become the performance benchmarks for those parameters in the contractors’ subsequent Qualification Testing of the integrated LP.

5.2.3. Processing Speed (Words per Minute (WPM)). This test will record the amount of time required by the COTS LP to complete the processing of the Government provided foreign language documents used in the RCT and will result in a processing speed calculation of WPM. The data for this performance metric will be collected simultaneously with the RCT procedure. The offeror will record the amount of time to complete the processing of each individual document in the applicable test corpora. There are two (2) cases for this test and the proposed COTS LP will be demonstrated in accordance with the applicable procedures for the language pack as follows:

1. WPM Case 1 applies to Dari, Pashto, Urdu, and Farsi COTS LPs: If the Corpus 3 processing speed is less than 500 WPM, the COTS LP will have failed the processing speed test, and the proposed COTS LP will be rated UNACCEPTABLE. The Corpus3 Processing Speed will become the performance specification requirement for Processing Speed and shall be used in the contractor’s Qualification Test procedure.

2. WPM Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin

Chinese, and Ukrainian COTS LPs: If the Corpus 3 Processing Speed is less than 500 Words per Minute (WPM), the COTS LP will have failed the processing speed test, and the proposed COTS LP will be rated UNACCEPTABLE. In addition, if the Corpus 4 processing speed is less than 300 Words per Minute (WPM), the COTS LP will have failed the processing speed test and the proposed COTS LP will be rated UNACCEPTABLE. The Corpus 3 and Corpus 4 Processing Speeds will become the Processing Speed requirements for the contractor’s Qualification Test procedure and shall be used in the contractor’s Qualification Test procedure if the contractor should receive an award..

5.2.3. Reading Comprehension Test (RCT). The RCT measures an English-speaking test subject's ability to understand essential elements of information from an English machine translation of a foreign language document. The Government will provide a set of foreign language test documents to the offeror at the demonstration. The Offeror will then process the test documents through the proposed COTS LP, and provide the corresponding English translation output documents to the Government on a Government provided thumb drive. After the Offeror’s demonstration concludes, the Government will prepare and administer a RCT at separate location for each proposed COTS LP. Scoring for the RCT will be on an ACCEPTABLE/UNACCEPTABLE basis as defined in Section M of the solicitation. The proposed COTS LP will be demonstrated in accordance with the following procedures:

1. RCT Case 1 applies to Dari, Pashto, Urdu, Farsi COTS LPs that contain only a MT language component (LC). The Offeror shall process the set of Government provided documents identified as Corpus 3 through the proposed COTS LP. The Offeror provided translation output shall be evaluated by the Government through use of the RCT.

2. RCT Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin

Chinese, and Ukrainian COTS LPs that contain both an MT LC and an OCR LC. The Offeror will process two (2) sets of Government provided documents identified as Corpus 3 and Corpus 4 through the proposed COTS LP. The Offeror provided translation output shall be evaluated by the Government through use of the RCT,

5.2.4. File Type Check. The FTC procedure confirms that the proposed COTS LP can ingest and process documents that are of different file formats or “file-types”. The FTC procedure requires the Offeror to process a set of Government provided documents through the proposed COTS LP and furnish the corresponding output to the Government for evaluation (Appendix A). Scoring for the FTC will be on an ACCEPTABLE/UNACCEPTABLE basis. The FTC will be demonstrated and evaluated using the appropriate test case as follows:

1. FTC Case 1 applies to Dari, Pashto, Urdu, Farsi COTS LPs that contain only a MT LC.

The Offeror will process a set of Government provided electronic documents identified as Corpus 5a that are in the TXT (UTF-8), RTF, and HTML file formats and shall provide the translation output to the Government.

2. FTC Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin

Chinese, and Ukrainian COTS LPs that contain both an MT LC and an OCR LC. The Offeror will process a set of Government provided electronic documents identified as Corpus 5a that are in the TXT (UTF-8), RTF and JTML file formats. In addition, the Offeror will process a set of image-based documents identified as Corpus 5b, which are in the TIFF, PNG, JPEG and PDF image file formats. The Offeror shall provide all translation output from all demonstrations to the Government.

5.3. Test Data Sources. The Government will provide demonstration Corpora to the Offeror at the demonstration event (see paragraph 5.7). However, the Offeror shall remove all Government provided test material from Offeror systems at the end of each demonstration event. The Offeror must return all Government furnished test materials to the Government at the end of the demonstration event and shall not retain demonstration materials.

5.4. Hardware. Offerors will provide all hardware required during the demonstration. Offerors will provide systems that are similar to the system configuration provided in Table 5-2. The Offeror must provide one (1) primary and (1) backup hardware/demonstration system for the Technical Demonstration. The Offeror shall record the system specifications, such as model, operating system, Random Access Memory (RAM), Hard Drive type & Size, and Processor (processor type & speed) on the data collection scorecard provided by the Government, see Appendix A.

Table 5-2 System Configuration

Mobile Configuration Item Description/Version

Platform Dell Precision M7710 /Dell M7720 or equivalent CPU 2.8 GHz I7 processor, sixth generation or later Physical Memory 32GB RAM System Hard Disk Drive 256GB HD USB Computer must have USB port to be able to read and store data to 16GB USB Drive

5.5. Demonstration Data Sets. The Government will provide language files to the Offeror at the time of the demonstration via 16GB USB thumb drives (one per language pack). The Offeror shall save all demonstration output to a second Government furnished thumb drive. The Offeror shall record demonstration results on a data collection scorecard (Appendix A). Data set outputs and hardcopy results will identify Offerors by an alpha designation (e.g. Offeror A, Offeror B, etc.)

assigned by the Government. The Offeror must provide all output data (hardcopy or digitally recorded data) to the Government Lead or designated representative before being released from the test site.

5.5.1. File Structure. The source and output file structures are intentionally similar for ease when saving output data to the designated thumb drive. The source thumb drive file structure will contain a directory named for the source language; contained within that directory exist a number of subdirectories as represented in Figure 5-1.

Figure 5-1 Language Source Subdirectories

The output or destination thumb drive is labeled with an alpha character; the file structure includes a directory for each language pack demonstrated by the offer, and directory contains a number of subdirectories as represented in Figure 5-2.

Figure 5-2 Offeror Output Subdirectories

5.5.2. Insert demonstration language source thumb drive. The Offeror will copy the source file structure to include (directory, subdirectories and files) to the demonstration laptop and remove the source thumb drive.

5.5.3. Insert Output Source thumb drive. The Offeror will create a directory labeled to represent the demonstrated language (e.g. Russian, Korean, etc.). The Offeror will copy the output file structure from the output thumb drive to include (directory and subdirectories) into that directory noted in fig 5.2 and copy the output file structure (directory and subdirectories) into that directory. The Offeror will save translated output files from RCT (Corpus 3 and 4) and FTC (Corpus 5a and 5b) to the respective subdirectories.

5.5.4. Data. The Offeror must remove all Government furnished materials including language Corpora from the Offeror provided laptops at the conclusion of the Technical Demonstration. .

The Offeror will record FTC and Processing Speed on the Data Collection Scorecard (Appendix A).

A copy of the completed scorecard will be provided to the Offeror at the completion of the demonstration.

5.5.5. Instrumentation/Support Test Equipment.

The Government will provide digital stopwatches for the Offeror to capture the WPM performance metric. The Offeror will record and report the translation times for each language test corpora during the RCT data processing portion of the demonstration. The Offeror will initiate the stopwatch at the time of translation initiation, typically a button push, or similar action resulting in translation initiation. The Offeror will stop the stopwatch timer at the point at which translation completes. Measurement of translation is at the individual file level only (Appendix A).

5.6 Government Observation. A Government Observer must witness all Offeror data collection at the Technical Demonstration event. Results not witnessed by the Government Observer may not be considered. The Offeror will record data collect by using the Data Collection Scorecard (Appendix A).

A-1

Appendix A Data Collection Scorecard

A Government Observer will witness offeror data collection at the Technical Demonstration event. The Offeror will record data collect by using the Data Collection Form (see below).

Table A-1 SAMPLE Data Collection Scorecard

OFFEROR ID: ________________ DATE: __________

LANGUAGE: _________________

Reading Comprehension Test (RCT) Results: After completion of the demonstration a RCT consisting of 25 questions per Corpus aligned to the Offeror’s English language output will be prepared and administered to four (4) Test Subjects, resulting in an average RCT score for the LP to be recorded below.

1. Corpus 3 (MT, processing TXT (UTF-8) files)

a. English Translation Output: ____________(yes/no)

b. RCT MT AVG Score:____________%

2. Corpus 4 (OCR+MT, processing TIFF files)

a. English Translation Output:_____________ (yes/no)

b. RCT OCR+MT AVG Score:____________%

Processing Speed (Words per Minute (WPM)) test and Data Collection Results: Processing Speed

(WPM) metric is calculated by dividing the number of words in a Corpus by the total processing time that was recorded during this test. Word counts for all files are provided in the WPM Spreadsheet.

1. Corpus 3 (MT, processing TXT (UTF-8)files):__________WPM

2. Corpus 4 (OCR+MT, processing TIFF files):_____________WPM

Note: The Corpus 3 and Corpus 4 Processing Speeds shall become the performance requirements for Processing Speed and shall be used in the contractor’s Qualification Test procedure if the contractor should receive an award.

A-2

OFFEROR ID: ________________ DATE: __________

LANGUAGE: _________________

File-Type Check (FTC) Test Results: The FTC requires the Offeror to process a set of Government provided documents through the proposed COTS LP.

1. Corpus 5a FTC (Accept/Unacceptable)

a. TXT (UTF-8) ____________ (demonstrated during RCT processing of Corpus 3)

b. RTF _____________

c. HTML _____________

2. Corpus 5b FTC (Accept/Unacceptable)

a. TIFF _____________(demonstrated during RCT processing of Corpus 4)

b. PNG _____________

c. JPEG _____________

d. PDF _____________

End of LP Demo Check to verify:

- All Offeror generated metrics and output files are recorded and provided to Government

- Backup Data and files are an exact copy of Offeror provided output

- Offeror has purged Government provided Corpora from laptop computers

- Data Collection Scoreboard has been filled in and signed by offeror and Government representative

SIGNATURES:

OFFEROR: Signature indicates offeror submits results and data for review by the Government during the pending source selection evaluation process.

__________________________ DATE: ________ _______________________ DATE: _______

GOVERNMENT REPRESENTATIVE OFFEROR REPRESENTATIVE