Attachment0004MFLTSLanguagePackQualificationTestPlan.pdf

UNCLASSIFIED

Machine Foreign Language Translation System

(MFLTS)

Text-to-Text Language Pack

Qualification Test Plan (QTP)

20 April 2018

DISTRIBUTION STATEMENT: Distribution authorized to the Department of Defense (DoD) and U.S. DoD contractors (Administrative Use, 28 March 2013). Other requests for this document, contact to the Product Director, Machine Foreign Language Translation System, SFAE-IEW-DM, Building 363, 10115 Gratiot Road, Fort Belvoir, VA 22060-5812, 703-704-4572.

Table of Contents

1. Introduction

2. Purpose of the Qualification Test

3. Qualification Test Approach

4. Qualification Test Requirements & Procedures

4.1. LP Performance Requirements

4.2. LP Integration Requirements

5. Qualification Test Execution

6. MFLTS T2T System Description

7. Qualification Testing Requirements

7.1. LP Performance Testing

7.1.1. Processing Speed WPM Test

7.1.2. File-Type Check (FTC)

7.1.3. Bilingual Evaluation Understudy (BLEU) Test

7.1.4. Word Error Rate (WER) Test

7.1.5. Reading Comprehension Test Data Collection Procedure

7.2. Functional Testing

7.2.1 The contractor’s Functional Test

7.2.2 MFLTS SW Configuration used for Qualification Testing

7.3. Contractor’s QT Procedures

7.3.1 Deviations

7.3.2 Test Participants

7.4. Approach to Testing

7.5. Pass/Fail Scoring Criteria

8. Test Resources, Environment, and Constraints

8.1 Resources

8.2 Environmental Requirements

8.3 Hardware

8.4 Platform Hardware Specification

8.5 Data Collection Tools (Software)

8.6 Instrumentation/Support Test Equipment

8.7 Test Data Sets

8.8 File Structure

Appendix A Qualification Test Execution .......................................................................... A-1

Tables

Table 7-1 Text-to-Text (T2T) Functions Checkout Summary Table 8-1 System Configuration Table 8-2 Tools that support QT

Figures

Figure 8-1 Language Source Subdirectories Figure 8-2 Contractor Output Subdirectories Figure A-1 QT Execution Overview ........................................................................................... A-1

1. Introduction.

This Qualification Test Plan (QTP) represents the structure of a contractor’s Qualification Test (QT) that will be followed in order for a contractor’s to assert that a Language Pack (LP) has met the requirements of the contract. Only Qualified LPs will be eligible for licensing by the Army.

The intent of the QTP and the corresponding Contractor-developed QT Procedures is twofold: a) it must validate that the Commercial-Off-The-Shelf (COTS) LP continues to meet the performance requirements that were established during the COTS LP Demonstration, and b) the COTS LP has been successfully integrated with the MFLTS Software (SW) Application (App) and is compliant with the MFLTS Tech Architecture requirements. The contractor shall provide all hardware components and personnel needed to execute the Government approved QT Procedures, operate the System Under Test (SUT) and also perform as data collectors. This QTP describes the purpose, objectives, scope, approach, evaluation environment, resources, and schedule for testing activities. This plan identifies the features and functions to be tested, the resources needed, and the schedule required to complete testing.

2. Purpose of the Qualification Test.

The purpose of the contractor conducted QT event is to determine if the integrated COTS LP meets the requirements of the contract. The QT shall include an assessment of the technical performance of the LPs Machine Translation (MT) and Optical Character Recognition (OCR) Language Components (LCs), verification of conformance of LP interfaces with the MFLTS architecture, and verification of successful integration of the LP with the MFLTS SW App. This QTP provides an outline of the requirements and structure of the LP QT. It also supports the contractor’s development of an associated QT procedures document, which when executed will resulting the successful completion of a LP QT. The contractor’s QT Procedures shall utilize the MFLTS SW App and the requisite software tools in order to document the conformance of the Contractor provided LP with the requirements of the contract.

3. Qualification Test Approach.

The QT of LP shall focus specifically on the LP interfaces and LP performance. The Government will provide an executable version of the MFLTS SW App to the contractor, and the MFLTS App will be used to execute the QT. The contractor shall prepare and execute QT procedures during the QT that are sufficient to characterize language pack performance and validate the LP interface with the MFLTS SW App and the existing MFLTS features and functions. Contractors are expected to provide the host laptop test system(s) with the Government provided MFLTS SW App and requisite SCLITE & Machine Translation Evaluation (MTEval) performance evaluation tools already installed and operational. The Contractors QT will verify LP performance by repeating the tests conducted during the COTS LP Demonstration, to include the Bilingual Evaluation Understudy (BLEU), Word Error Rate (WER), Words per Minute (WPM), and File Type Check (FTC) tests. They shall also perform an installation and uninstallation of LP test, as well as a full range of MFLTS SW functionality verification and with the LP installed.

4. Qualification Test Requirements & Procedures.

The contractor shall prepare and submit for Government review and approval a proposed set of QT Procedures that are sufficient to verify conformance with QT requirements. The QT requirements include LP Performance requirements as well as LP Integration requirements.

4.1. LP Performance Requirements are comprised of the performance values that were recorded during the COTS LP demonstration and submitted to the Government for evaluation. As was stated in the Request for Proposal Section M, all of the tests conducted during the Technical Demonstration and Contractor’s Benchmark Test IAW in the Performance Work Statement (PWS), shall be executed again during the contractors’ QT, to verify that the LP meets or exceeds the values recorded during the demonstration. The Reading Comprehension Test (RCT) Data collection procedure will be conducted as part of the QT, however RCT will not be conducted.

4.2. LP Integration Requirements shall be developed and proposed by the contractor in the QT Procedures, and shall be sufficient to verify that the contractor has successfully integrated the LP with the MFLTS App and this integration conforms to the MFLTS Technical Architecture.

Contractor shall derive specific and sufficient SW requirements and test cases to fully characterize the conformance of the integration effort with the MFLTS SW Architecture. These requirements shall be tested and verified by employing a SW functionality test of the integrated LP and MFLTS SW App.

5. Qualification Test Execution.

The contractor shall provide facilities, equipment, personnel and training necessary to execute the QT using the Government approved QT Procedures. The contractor test team shall adhere to the approved QT Procedures throughout testing activities. The contractor test team shall also use appropriate data recording mechanisms, tools and equipment, as specified in the QT Procedures.

The contractor test team shall record assumptions, constraints along with expected and actual test results. The Contractor test team shall ensure the QT Reports accurately convey test results, test incidents, and/or anomalies and declare whether a QT requirement has been met or not met.

6. MFLTS T2T System Description.

The MFLTS is a family of SW App(s) that are installed on a variety of receiving host program hardware platforms to provide a language translation capability to soldiers in the field. An application is comprised of software and data in several system configurations to enable language translation for a specific language pair (English and one target language). The MFLTS Text-to-Text (T2T) SW App with an installed T2T Language Pack provides all the software functionality required to perform a document translation task. T2T LPs consist of a MT LC which performs the translation function, and may include an OCR LC that processes image document inputs. MFLTS T2T configurations are as follows:

a. Mobile Configuration. This version of the MFLTS SW App is configured for host platforms that are typically laptop computers that utilize a Windows Operating System and have limited or no network connectivity. An example of a mobile platform is the Dell M7710 / Dell M7720 or equivalent system with a Windows 10 operating system. This is the configuration that will utilized to perform the QT.

b. Web-Enabled Configuration. This configuration of the MFLTS App is used on a server platform that utilizes a Windows-web Operating System and implements web services to provide a language translation capability to remote users via closed, restricted network. The users interact with the software via a web-based graphical user interface. This configuration will not be utilized during this QT.

7. Qualification Testing Requirements.

7.1. LP Performance Testing. LP Performance testing consists of a series of tests and data collection procedures designed to characterize the performance of the LP LCs. For the QT, contractors shall execute all of the test and data collect procedures executed during the LP Technical Demonstration in order to verify that the LP meets (or exceeds) the performance values recorded during that demonstration. The measures of performance to be re-tested during the QT are: Processing Speed WPM Test; FTC Procedure; BLEU Test; WER Test; RCT Data Collection Procedure.

7.1.1. Processing Speed WPM Test. This test will record the amount of time required by the COTS LP to complete the processing of the applicable Government provided foreign language documents and will result in a processing speed calculation of WPM. The contractor shall record and report document processing times for each file in the QT corpus, then calculate the QT processing speed using the Government provided word count for the files, There are two (2) cases for this test and the COTS LP will be tested in accordance with the applicable procedures for each case as follows:

7.1.1.1 WPM Case 1 applies to Dari, Pashto, Urdu, Farsi COTS LPs that contain only a MT LC.

The Contractor shall record the processing speed in WPM for Corpus 3 and compare that speed to the processing speed reported in the LP Demo.

WPM Case 1 Test Criterion; The QT speed should be equal to or greater than the speed reported at LP Demonstration.

7.1.1.2 WPM Case 2 WPM Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin Chinese, and Ukrainian COTS LPs that contain both an MT LC and an OCR LC. The Contractor shall record the processing speeds in WPM for Corpus 3 and Corpus 4 and compare those speeds to the speeds reported in the LP Tech Demo.

WPM Case 2 Test Criterion; The QT speed should be equal to or greater than the speed reported at LP Demonstration.

7.1.2. File-Type Check (FTC). Using the same procedures as were used during the LP Demo, the contractor shall test the capability of the LP to successfully process documents of different file formats. The FTC procedure requires the Contractor to process a set of Government provided documents through the proposed COTS LP and evaluate the corresponding output against the output recorded during the LP Demo. All LPs will be tested using electronic text file types (.txt, html, and .rtf), and LPs that include an OCR LC will also be tested using image files types (.jpg, .png, .tiff, .pdf image). The FTC shall be tested and evaluated in only of one of the two test cases as follows:

7.1.2.1 FTC Case 1 applies to Dari, Pashto, Urdu, Farsi COTS LPs that contain only a MT LC.

The Contractor shall process a set of three Government provided electronic documents identified as Corpus 5a that are in the TXT (UTF-8), RTF, and HTML file formats.

FTC Case 1 Test Criterion: If the LP accepts the required electronic file types and outputs a file of the same electronic file-type with a nonzero English word count, the FTC for that specific file type is scored as a PASS. If the COTS LP fails to score a PASS for all three (3) of the required file types then the LP will be rated as not meeting the QT requirement for the FTC.

7.1.2.2 FTC Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin Chinese, and Ukrainian COTS LPs that contain both an MT LC and an OCR LC. The Contractor shall process the set of three (3) electronic file type documents identified as Corpus 5a as well as the set of four Government provided image-based documents identified as Corpus 5b, which are in the TIFF, PNG, JPEG and PDF image file formats.

FTC Case 2 Test Criterion: If the LP accepts the required electronic file types and outputs a file of the same electronic file-type with a nonzero English word count, the FTC for that specific file type is scored as a PASS. If the COTS LP fails to score a PASS for all seven of the required file types then the LP will be rated as not meeting the QT requirement for the FTC.

7.1.3. Bilingual Evaluation Understudy (BLEU) Test. Contractor shall execute the BLEU test procedure during qualification testing of all of the following COTS LPs: Russian, Dari, Pashto, Urdu, Farsi, Korean, and Mandarin Chinese. The BLEU benchmark score that was recorded at the Contractor Benchmark Test is the BLEU performance requirement to be used in the contractors QT procedures. This procedure utilizes the National Institute of Standards and Technology (NIST) MTEval BLEU tool to measure the translation accuracy associated with proposed COTS LP, and will result in one measurement that will be associated with the proposed COTS LP. The BLEU data collect procedure requires the Contractor to process the set of Government provided documents through the proposed COTS LP, and then process the corresponding translated output through the NIST MTEval BLEU tool to measure the translation performance associated with the proposed COTS LP. There may be more than one test case per language, as follows:

7.1.3.1 BLEU Case 1 applies to Russian, Dari, Pashto, Urdu, Farsi, Korean, Mandarin Chinese, COTS LPs. In this case the contractor shall use Corpus 1 to perform a foreign language to English translation. Corpus 1 is comprised of foreign language documents that the contractor shall translate into English using the proposed COTS LP. The Contractor shall record the BLEU score generated by the NIST MTEval tool, and provide all of the output from the tool and the output from the COTS LP to the Government in the QT Report.

BLEU Case 1 Test Criterion: The BLEU score recorded at LP Demo is the BLEU performance requirement to be used in the QT procedure for the associated COTS LP.

The BLEU QT score should be equal to or greater than the BLEU score reported at the Contractor Benchmark Test.

7.1.3.2 BLEU Case 2 applies to Dari, Pashto, and Korean. The Contractor shall use Corpus 1a to perform English to foreign language data collect procedure. Corpus 1a is comprised of English language documents that the contractor shall translate to the specified foreign language using the proposed COTS LP. The Contractor shall record the BLEU score generated by the NIST MTEval tool, and provide all of the output from the tool and the output from the COTS LP to the Government in the QT Report.

BLEU Case 2 Test Criterion: The BLEU score recorded at Contractor Benchmark Test is the BLEU performance requirement to be used in the QT procedure for the associated COTS LP. The BLEU QT score should be equal to or greater than the BLEU score reported at the Contractor Benchmark Test.

7.1.4. Word Error Rate (WER) Test. This test is a performance measurement of the accuracy of an Optical Character Recognition LC. Contractors will perform this procedure for LPs that contain both MT and OCR LCs. The WER test procedure requires the Contractor to process the set of Government provided documents identified as Corpus 2 through the OCR of the LP, and then process the transcribed foreign language OCR output through the NIST SCLite tool to measure the WER. The Contractor will record the WER score generated by the SCLite tool, and include all of the OCR output and the SCLite tool output in the QT Report.

7.1.4.1. Mandarin Chinese. The error rate performance of a Mandarin Chinese OCR LC is measured using the same tools and procedures, but is called Character Error Rate (CER). In addition, Corpus 2 for Mandarin Chinese LPs contains two (2) sets of documents: one using in the Simplified Chinese character set, and one using the Traditional Chinese character set.

Contractors are required to process and report CER scores on both sets of documents in Corpus 2.

WER Test Criterion: The WER score recorded at Contractor Benchmark Test is the WER performance requirement to be used in the QT procedure for the associated COTS LP. The WER QT score should be equal to or less than the WER score reported at the LP Demo.

7.1.5. Reading Comprehension Test Data Collection Procedure. This procedure is included as a data collection event only, there is no Criterion to be met for this procedure. The Government may utilize the translation output generated from this procedure in future RCTs. The Government will provide the foreign language test documents used in the COTS LP Demo to the contractor, and contractor shall process these using the MFLTS SW App and the integrated COTS LP. Contractor shall provide the corresponding English translation output documents to the Government. The government will collect the documents for use in a RCT that will be prepared and administered in the future. There are two possible test cases for the RCT, and the proposed data collection procedure will be performed In Accordance With (IAW) the applicable procedures as follows:

7.1.5.1 RCT Case 1 applies to Dari, Pashto, Urdu, Farsi COTS LPs that contain only a MT LC.

The Contractor shall process the set of Government provided documents identified as Corpus 3 through the proposed COTS LP and provide the translation output to the Government to support the preparation and administration of an RCT.

RCT Case 1 Test Criterion: NA

7.1.5.2 RCT Case 2. RCT Case 2 applies to Russian, Modern Standard Arabic, Polish, Korean, Mandarin Chinese, and Ukrainian COTS LPs that contain both an MT LC and an OCR LC. The Contractor shall process each of the two (2) sets of Government provided documents identified as Corpus 3 and Corpus 4 through the proposed COTS LP and provide the corresponding translation output to the Government to support the preparation and administration of an RCT.

RCT Case 1 Test Criterion: NA

7.2. Functional Testing. Functional Test Procedures shall be developed and proposed by the contractor in the QT Procedures, and shall be sufficient to verify that the contractor has successfully integrated the LP with the MFLTS App and this integration conforms to the MFLTS Technical Architecture. Contractor shall derive specific and sufficient SW requirements and test cases to fully characterize the conformance of the integration effort with the MFLTS SW Architecture. These requirements shall be tested and verified by employing a SW functionality test of the integrated LP and MFLTS SW App.

7.2.1 The contractor’s Functional Test procedures shall include but not be limited to function/feature checks described in Table 7-1. The Intent of the functional testing is to ensure full interoperability between the MFLTS App and contractor LP, and conformance of the LP with the MFLTS Technical Architecture. The contractor shall utilize Government furnished corpora to execute the T2T Checkout.

Table 7-1 Text-to-Text (T2T) Functions Checkout Summary.

Task

ID

Sub Task

ID

Task

A. Launch MFLTS T2T B. Install first Language Pack C. Tutorial & Help

1. Launch Tutorial

2. Open Help

D. Upload Files

1. Upload a Single File

2. Upload Multiple Documents

E. Queue Actions

1. Pause on Queue

2. Restart Processing

3. Sort the Queue

F. Job Actions

1. Cancel a Job

2. Delete Job(s)

3. Job Information

4. Comments

G. View Output

1. .txt files

2. .html files

3. .rtf files

4. Image files (.tiff, .jpg, .png, .pdf Image), (This section applies only if OCR is enabled.)

H. Save Translated Files

Task

ID

Sub Task

ID

Task

1. Text files (.txt, .rtf)

2. .html files

3. Image files (.tiff, .jpg, .png, .pdf Image), (this section only applies if OCR is enabled.)

I. Access Software Version Information J Uninstall first Language Pack K Install second Language Pack L. Repeat steps D. through G.

M. End MFLTS T2T function/feature checkout

7.2.2 MFLTS SW Configuration used for Qualification Testing. The T2T QT shall utilize the Mobile (T2T) configuration, this configuration is representative of both the Mobile and Web Enabled configurations.

7.3. Contractor’s QT Procedures. IAW the Performance Work Statement (PWS) and the associated Contract Data Requirement List (CDRL) the contractor shall develop and submit for Government review and approval the QT Procedures that align to the requirements of this QTP.

Government approval of the QT Procedures is required prior to the start of QT activities. The Contractor shall provide all QT participants with an in-brief regarding the test lab and facilities, test order, test synchronization points, test documentation, data logs retrieval and the suspension and exit criteria associated with the test. All test participants shall be familiar with MFLTS and the associated host platform hardware basic functionality. The contractor must complete familiarization or system specific system training required to conduct the test prior to the QT start date.

7.3.1 Deviations. In the event that test procedure corrections are necessary after the QT start date, the contractor shall notify the Government and seek concurrence. With Government concurrence, the Contractor shall record the change to the test procedures, and proceed to record test results. If the Government disapproves of changes to the test procedures, the Contractor shall proceed with the original test procedure(s) and record output data as planned.

7.3.2 Test Participants. The contractor shall execute the QT using contractor provided operators executing the approved procedures. All test operators are required to complete the tutorial embedded in the MFLTS App prior to the start of the QT.

7.4. Approach to Testing. The contractor shall ensure the SUT has the MFLTS App SW installed and ready for test. All test operators shall receive the Government approved QT Procedures from the contractor along with the respective Government provided test corpora, as applicable. For each test case completed, the system operator in collaboration with the test data collector shall document the test case number, the result of the test, and the current date on the provided test case recording forms from the approved QT Procedures. For every test failure, the contractor shall record and provide the details of the failure in the QT Report provided to the Government at the conclusion of the QT event.

7.5. Pass/Fail Scoring Criteria. The test criteria shall be included in the QT Procedures.

Determination of issue/incident severity includes consideration of the issues found during the execution of the QT, the number of issues found, the likelihood of the issues occurring in the system, and the impact the issues have on the system effectiveness. All assumptions, constraints and test results shall be recorded to substantiate test reporting. The Government will determine Qualification based upon pass/fail conditions and shall adjudicate fail conditions as to the impact they have on usability, functionality, and mission effectiveness.

8. Test Resources, Environment, and Constraints.

8.1 Resources. The Contractor shall provide all necessary facilities, equipment and personnel required to successfully execute the QT. The Government will witness all QTs.

8.2 Environmental Requirements. The Contractor shall provide a test environment suitable to conduct the QT, with adequate power. The SUT shall not utilize external internet functionality.

The Government estimates a QT duration of approximately two days (for four (4) LPs) for the test event at the contractor site; facilities must be available for the duration of the event.

8.3 Hardware. Contractors shall provide all hardware required to conduct the QT.

Contractors shall provide equivalent systems, avoiding use of hardware that provides an unrealistic representation of software performance, as it is unlikely that Government hardware acquisitions will be able to field advanced hardware to the force.

8.4 Platform Hardware Specification. Table 8-1 identifies hardware that previously hosted the MFLTS App. Contractor is requested to select a host platform for the QT that is similar to the system described below. If possible, contractor should configure the host laptop using the Secure Host Baseline-Army.

Table 8-1 System Configuration

Mobile Configuration Item Description/Version

Platform Dell Precision M7710 or M7720 or equivalent CPU 2.8 GHz I7-4900MQ processor, sixth generation or later Physical Memory 32 GB RAM System Hard Disk Drive 256 GB HD USB Computer must have USB port to be able to read and store data to

USB Drive

8.5 Data Collection Tools (Software). The contractor shall use the tools described in table 8-2 to collect performance data during the LP QT: NIST BLEU for T2T MT performance, and NIST SCLite tool for WER testing. These tools can be downloaded from the Reading Room at the following URL: https://www.mycmsportal.com/login.aspx.

https://www.mycmsportal.com/login.aspx

Table 8-2 Tools that support QT

Tool Purpose Test Corpus National Institute of Standards and Technology (NIST) Bilingual Language Evaluation Understudy (BLEU) tool, MTEval version 14c

For machine translations, a software tool that scores individual translated segments by comparing them with a set of reference translations

Corpus #1 Corpus #1a

Word Error Rate (WER) tool, SCLITE from NIST version 2.4.10

For OCR, an automated tool that calculates the

WER

Corpus #2

8.6 Instrumentation/Support Test Equipment. The contractor shall provide test instrumentation such as stop watches as required for use during the QT.

8.7 Test Data Sets. The Government will provide language files at the time of the QT via 16GB USB thumb drives (by language) to the contractor. The contractor shall save all QT output to a second 16GB USB thumb drive (one per contractor) and return the output thumb drive to the Government. The contractor shall submit all output data as appendices to the o the Qualification Test Report. The contractor is responsible for any incidental drivers or data files required for testing.

8.8 File Structure. The source and output file structures are intentionally similar for ease of use when saving output data to the designated thumb drive. The source thumb drive file structure will contain a directory named for the source language; contained within that directory exist a number of subdirectories as represented in Figure 8-1.

Figure 8-1 Language Source Subdirectories

The file structure for the output thumb drive contains a directory representing each language addressed by the contractor LP(s), that directory also contains a number of subdirectories as represented in Figure 8-2.

Figure 8-2 Contractor Output Subdirectories

The contractor shall copy the source file structure (directory and subdirectories) to the SUT and remove the source thumb drive. The contractor shall then create a directory (labeled to represent the language (e.g. Russian, Korean, etc.) and copy the output file structure (see figure 8-2) and deposit output data into the folder locations as indicated by the directories. The contractor shall save BLEU, FTC, and WER data output to the respective location in the output subdirectories.

Note: The contractor collects WPM output data manually and records the data on hard copy forms. The contractor shall surrender the recorded WPM output data to the Government.

Appendix A Qualification Test Execution

Qualification Test Execution Overview. The Government will witness the Contractor load/installation of the LP onto the laptop. The Contractor shall execute the T2T Functional Checkout followed by the BLEU, WER, WPM, and FTC test procedures. At the conclusion of the FTC, the Contractor shall de-install the LP, install a new LP and process sample text and image based files.

Data Management and recovery. The Government will control all source and output data, using an external Hard Drive, which serves as a language file source master. The Government employs individual language thumb drives or DVD to deliver language files to the contractor.

The contractor saves the performance data to the SUT and records the output data to an output thumb drive, DVD, or data collection form as appropriate.

Data Collection. Prior to close out of the QT the Contractor and Government witness should verify that the output results are human readable and accurately reflect the Qualification Test performance results, and are archived appropriately.

Foreign Language

Thumb Drive(s)

Offeror Laptop With Proposed LP(s)

Foreign Language

Thumb Drive(s)

Src Input Raw Output

Qual Tst Rpt

Perf Results Language Corpora (Master)

Language Data Gov.

Note: The Government controls all data products to avoid data compromise

Figure A-1 QT Execution Overview