Attachment0007QualityAssuranceSurveillancePlan.pdf

UNCLASSIFIED

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

FOR

MACHINE FOREIGN LANGUAGE TRANSLATION SYSTEMS (MFLTS)

TEXT TO TEXT (T2T) LANGUAGE PACKS (LPS)

19 April 2018

PREPARED BY:

Product Director, MFLTS

10115 Gratiot Road, Building 363, Room 207

Ft. Belvoir, VA 22060

This document contains information EXEMPT FROM MANDATORY DISCLOSURE under the Freedom of Information Act (FOIA). Exemption 2 is applicable ii

TABLE OF CONTENTS

1. PURPOSE

2. AUTHORITY

3. SCOPE

3.1. Contract Scope

3.2. Contract Quality Assurance

3.3. Government Quality Assurance Surveillance

4. ROLES AND RESPONSIBILITIES

4.1. Product Director (PD) MFLTS

4.2. Procuring Contracting Officer (PCO)

4.3. Contracting Officer Representative (COR)

4.4. PD MFLTS Program Analyst

5. METHODS OF QUALITY ASSURANCE SURVEILLANCE

5.1. Surveillance Folder

5.2. Evaluation Areas of Contractor Performance Assessment Report

5.3. Performance Rating Definitions

5.4. Timely Feedback to Contractor

5.5. Quality Surveillance

5.6. Contract Performance Assessment Reporting System (CPARS)

5.7. Disputes

5.8. Performance Requirements Summary (PRS)

LIST OF TABLES

Table 1: Performance Requirements Summary

Table 2: Performance Ratings (FAR Part 42.1503, Table 42-1)

1. PURPOSE

This Quality Assurance Surveillance Plan (QASP) is a Government-developed document that describes the systematic quality assurance methods and guidelines used by the Procuring

Contracting Officer (PCO) and the Contracting Officer Representative (COR) to measure and monitor Contractor performance. Specifically, this QASP measures the Contractor efforts toward meeting the requirements of the Performance Work Statement (PWS) for the Machine

Foreign Language Translation System (MFLTS) acquisition of commercial Language Packs

(LP).

This QASP:

a. Defines roles and responsibilities;

b. Identifies the performance standards and Acceptable Quality Limits (AQLs);

c. Defines the methodologies used to monitor and evaluate the Contractor’s performance;

d. Describes quality assurance reporting; and

e. Describes the analysis of quality assurance monitoring results.

2. AUTHORITY

This QASP was developed in accordance with Federal Acquisition Regulation (FAR) Part 46, Quality Assurance, and the relevant Clauses for Inspection of products and services to establish the procedures for monitoring, reporting, and evaluating the performance of the Contractor. The

Government acceptance of products and services will be executed by the PCO or designated

COR. The frequency and method of the government’s monitoring (inspection), review responsibilities, and reporting requirements follow. Appendix A identifies acronyms used within this document

3. SCOPE

3.1. Contract Scope. The contract establishes the requirements for acquisition of MFLTS LP.

Under a multiple award, Firm Fixed Price (FFP) contract basis, the Government will issue

Multiple Type C contract awards for Language Packs - MFLTS translation language packs for a designated language pair(s) (English – Foreign Language); to include evidence of testing.

The contract period of performance (POP) is five years, inclusive of a twelve (12) month base period and 3 months. The place of performance is at the Contractor’s facility.

3.2. Contract Quality Assurance. Per DFAR 246.4, Government Contract Quality

Assurance, and the PWS, the Government will monitor and record the Contractor performance, as stated in this QASP. PWS Appendix A, paragraph 2, Quality Control and Management, requires the following:

a. The Contractor shall maintain an effective quality control program to ensure provision of products and services in accordance with the contract and the PWS, and applicable performance specifications.

b. The Contractor shall execute quality assurance and quality control of all MFLTS deliverables, products, and artifacts.

As corrective actions are identified, the PCO may notify the Contractor of required modifications to the Contractor quality assurance plan during the period of performance, see Appendix B for the Corrective Action Report.

3.3. Government Quality Assurance Surveillance. The Government will monitor Contractor performance and adherence to the PWS requirements, as outlined in Table 2, Performance

Requirements Summary (PRS). PWS requirements include Contract Data Requirements List

(CDRLs), work products, meetings, and periodic program reviews that provide progress of the contracted effort. The Government COR will monitor performance and review performance reports furnished by the Contractor to determine how the Contractor is performing against communicated and documented performance objectives. The Government will make decisions based on performance measurement metric data and will notify the Contractor of those decisions.

The Contractor will be responsible for making required changes in processes and practices to ensure effective performance management.

Table 1: Performance Requirements Summary

Required

Products/Services

Performance

Measure Performance Standard

Acceptable

Quality Level (AQL) Methods of Surveillance

Technical Management (PWS 3.1)

Program Reviews

(PWS 3.2.1)

Compliance

Reviews and minutes provide the information needed for Government to assess program status.

No critical comments.

All key meeting topics and actions recorded.

100 % COR inspection of contractor minutes

Version Description

Document (PWS 3.2.9)

Compliance

Contractor shall provide Version Description

Documents (VDD) with each LP Delivery

In accordance with

CDRL A008

100 % COR review of VDDs

Software Quality Control &

Management (PWS Appendix A, Para. 2)

Compliance

Contractor shall provide evidence of Software Quality Assurance

In accordance with PWS

Appendix A, paragraph 2

100 % COR inspection of products delivered under this contract.

Cybersecurity (PWS

3.2.8) Compliance

In accordance with the

PWS

Compliance with 100% of PWS paragraph 3.2.8.

100 % COR review of Contractor’s compliance.

Provide Language Packs (PWS 3.2.2)

Develop an

Automated LP Installer (PWS 3.2.3)

Cost Completed within cost ceiling

In accordance with PWS 100 %COR inspection of invoices

Schedule Completed within delivery schedule

Delivered IAW DD Form

COR accepts IAW DD Form 1423, and Letter of Acceptance.

Performance Compliance with Performance

Specification

In accordance with PWS.

100 % Qualification Test witnessed by the Government

Qualification Test Reports (PWS 3.2.5)

Compliance Government to review for approval

As stated in CDRL A003 100 % COR inspection of Contractor Test Plan(s)

Required

Products/Services

Performance

Measure Performance Standard

Acceptable

Quality Level (AQL) Methods of Surveillance

Test Articles (PWS

3.2.4) Compliance

Completed within the LP Qualification Test period

In accordance with

CDRLs A003 and A004

100% review of A003 and A004 by

COR

Qualification Testing

(PWS 3.2.5)

Compliance

Government approved

Contractor LP Qualification Test

Procedures

In accordance with the PWS and CDRL A003, and applicable MFLTS

LP Performance Specifications

100% COR review of contractor qualification test procedures

Language Pack

Installer (PWS 3.2.3) Demonstration

Demonstrated during the Qualification Test.

LPs install and interoperate with the

MFLTS; de-installation without leaving LP remnants that adversely affect the MFLTS application

100% COR inspection of installation and de-installation of contractor LP

Test and Evaluation

Software License

(PWS 3.2.7.1)

Compliance

The Contractor shall provide a T&E Software License to the

Government to conduct

LP T&E, Integration, and Certification.

In accordance with PWS paragraph 3.2.7.1 100% COR review of agree, Appendix B - Deliverables

Deliverables (PWS – various)

Schedule

Deliverables submitted to the Government within the timeframes stated in the DD Form 1423s.

90% of all deliverables submitted to the Government within the timeframes stated in the

DD Form 1423s. The remaining (i.e. late deliverables) submitted no later than one week after the timeframes stated in the DD Form

1423s. 100% of any deliverables that are granted an extension by the Government must be submitted by the extension date.

100% review of all deliverables and receipt of deliverables

4. ROLES AND RESPONSIBILITIES

The following Government personnel positions will oversee and implementation surveillance activities.

4.1. Product Director (PD) MFLTS. PD MFLTS will provide primary program oversight.

While the PD may serve as a direct conduit to provide Government guidance and feedback to the

Contractor on project matters, the PD is not empowered to make any contractual commitments or any contract changes on behalf of the Government. The PD will recommend a COR to the PCO, ensure the COR is trained before performing any delegated duties, and support the COR performance assessment activities.

4.2. Procuring Contracting Officer (PCO)

The Army Contracting Command-Aberdeen Proving Ground Belvoir Division (ACC-APG-

Belvoir) will assign a duly appoint PCO with the authority to enter into, administer, or terminate contract actions, such as basic awards or modifications, and make related determinations and findings on behalf of the Government. See FAR Part 1.602 for authority and responsibilities.

The PCO will ensure performance of all necessary actions for effective contracting and compliance with the Contract terms and conditions. The PCO will serve as the official

Government agent with authority to modify and interpret the contract with the Contractor. The

PCO will be the only Government official who can authorize contract actions that may result from review of Contractor performance. The PCO will also ensure that the Contractor receives impartial, fair, and equitable treatment.

The PCO will rely on the COR to provide information and recommendations regarding

Contractor performance. While the PCO may delegate many of the day-to-day contractual administration duties to the COR, certain contractual actions will be retained as the sole purview of the PCO, including negotiation and issuance of contract modifications, issuance of Contract

Deficiency Reports, issuance of “show cause” letters, contract termination, contract closeout, and

Contractor claims and disputes resolution. The PCO will designate in writing a COR. The PCO will ensure the COR is trained before performing any duties and supports COR performance assessment activities.

The PCO is ultimately responsible for the final determination of adequacy of Contractor performance.

4.3. Contracting Officer Representative (COR). The COR will be an individual designated in writing by the PCO to act as the authorized representative to assist in the administration of the contract. The COR will be responsible for providing continuous technical oversight of

Contractor performance. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on behalf of the Government. For any changes that may affect contract price, terms, or conditions, the COR will alert the PCO for potential resolution or action. The source of authority for the COR is the PCO Delegation Memorandum.

See FAR Part 1.604 and DFAR 201.602-2 for definition of responsibilities.

4.4. PD MFLTS Program Analyst. The PD MFLTS Program Analyst will be an individual designated by PD MFLTS to act as the authorized representative to assist in the administration of the financial management of the contract.

5. METHODS OF QUALITY ASSURANCE SURVEILLANCE

5.1. Surveillance Folder. A surveillance folder will be used by the primary COR to perform surveillance of Contractor performance and adherence to contract requirements. The folder has the following divisions:

a. COR appointment letter, PCO letter of designation, and Training Record;

b. PWS and other pertinent parts of the contract;

c. Quality Assurance Surveillance Plan;

d. Surveillance Reports;

e. Correspondence: Customer and user comments; and

f. Data and technical reports.

5.2. Evaluation Areas of Contractor Performance Assessment Report. The COR will perform an assessment and will file the assessment of the following evaluation factors in accordance with FAR 42.1502 (a).

a. Quality of Product or Service (Technical Performance) – The COR will assess the

Contractor’s efforts for planning, monitoring, and controlling program risk, costs, schedule, and adherence to the PWS and technical performance requirements. Areas of focus may include the planning and management of program tasks, the quality of support provided throughout all phases of contract execution, and the management of a totally integrated effort of all program management concerns to meet cost, schedule, and technical performance objectives.

b. Schedule and Timeliness of Deliveries – The COR will assess the Contractor’s adherence to the required delivery schedule by assessing the Contractor’s efforts during the assessment period that contribute to or affect the schedule variance. This will take into consideration the significance of scheduled events, and will also assess the effectiveness of corrective actions taken by the Contractor.

In addition, the Contractor will be rated on their conduct of scheduled reviews and their timely submission of review materials, per the agreed upon process and associated timelines as required by the contract.

c. Business Relations or Management – The COR will assess the turnaround, completeness, and quality of problem identification, corrective action plans, the cooperativeness of the Contractor, and customer satisfaction. The COR will assess the

Contractor’s success with management of subcontracts. The Contractor’s ability to identify and apply resources required to meet schedule requirements, assign responsibility for tasks/actions required by the PWS, and communicate appropriate information to the program office in a timely manner will also be assessed. The COR will assess the Contractor’s program management and technical risks and the associated risk mitigation plans to determine if there are any impacts to PD MFLTS’s project objectives.

5.3. Performance Rating Definitions. The Government ratings and narrative must leverage off of FAR Part 42.1503, Table 42-1 “Evaluation Ratings Definitions” as follows:

Table 2: Performance Ratings (FAR Part 42.1503, Table 42-1)

Rating Definition Note

Exceptional – Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective.

To justify an Exceptional rating, identify multiple significant events and state how they were of benefit to the Government. A singular benefit, however, could be of such magnitude that it alone constitutes an Exceptional rating. Also, there should not have been any significant weaknesses identified.

Good – Performance meets contractual requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the Contractor were effective.

To justify a Good rating, identify a significant event and state how it was a benefit to the Government. There should not be any significant weaknesses identified.

Satisfactory – Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the

Contractor appear or were satisfactory.

To justify a Satisfactory rating, there should have been only minor problems, or major problems the Contractor recovered from without impact to the contract/order. There should not have been significant weaknesses identified. A fundamental principle of assigning ratings is that Contractors will not be evaluated with a rating lower than

Satisfactory solely for not performing beyond the requirements of the contract/order.

Marginal – Performance does not meet some contractual requirements. The contractual performance of the element or sub-element being evaluated reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor proposed actions appear only marginally effective or were not fully implemented.

To justify Marginal performance, identify a significant event in each category that the Contractor had trouble overcoming and state how it impacted the Government. A Marginal rating should be supported by referencing the management tool that notified the Contractor of the contractual deficiency (e.g., management, quality, safety, or environmental deficiency report or letter).

Unsatisfactory – Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which Contractor corrective actions appear or were ineffective.

To justify an Unsatisfactory rating, identify multiple significant events in each category that the Contractor had trouble overcoming and state how it impacted the Government. A singular problem, however, could be of such serious magnitude that it alone constitutes an unsatisfactory rating. An Unsatisfactory rating should be supported by referencing the management tools used to notify the

Contractor of the contractual deficiencies (e.g., management, quality, safety, or environmental deficiency reports, or letters).

5.4. Timely Feedback to Contractor. Timely feedback to the Contractor on unacceptable performance will be provided by the COR and documented in writing. Prompt feedback is essential so that the Contractor can develop and implement a corrective action plan. The

Contractor's corrective action must be reported to the PCO for tracking purposes.

5.5. Quality Surveillance. Quality surveillance of Contractor performance will be performed as stated in this QASP. The Government, through implementation of its QASP, will monitor the

Contractor’s performance on a continuing basis. However, the volume of tasks performed by the

Contractor makes technical inspections of every task and step impractical. Accordingly, tasks will be monitored using a variety of surveillance techniques as described below. In addition, precise documentation is required to record, evaluate, and report the Contractor’s performance.

As a result, all documentation resulting from surveillance will be made part of the contract file.

During the program reviews the Contractor, Contracting Officer, and PD MFLTS staff will discuss contract performance, resolve issues, discuss customer complaints, and provide positive interaction and feedback to all parties. This process will provide a list of any unresolved deficiencies to the Contractor’s quality assurance representative.

Performance evaluation will be accomplished by monitoring compliance with the performance requirements as described in the contract and in the QASP. The method of evaluation will depend on the service type, and each service type may be assessed by more than one method.

Generally accepted surveillance methods will be used by the Government and are described in its

QASP. These methods include, but are not limited to:

a. 100% Inspection – The Government will evaluate 100% of designated products/tasks.

This surveillance method is appropriate for products/tasks where 100% inspection is required and is practical.

b. Partial Inspection – The Government will evaluate less than 100% of designated tasks and on a selective basis. This surveillance method is appropriate for tasks that occur infrequently and where 100% inspection is neither required nor practical.

c. Random Sampling – The Government will evaluate less than 100% of designated tasks on a random basis. This surveillance method is appropriate for tasks that occur infrequently and where 100% inspection is neither required nor practical.

d. Performance Reviews - The Contractor will attend Contract Performance Reviews, as required. These meetings will focus on overall contract performance. Specific dates, locations, agenda, and attendance requirements will be specified by the Contracting

Officer at least 30 calendar days prior to each meeting.

5.5.1. MFLTS Performance Assessment Report (PAR). The PAR will be used to report all minor discrepancies and will be generated by the COR and sent to the Contractor for corrective action. Contractors will be given ten (10) days to correct the items stated in the PARs, unless they have requested an extension. If three or more minor discrepancies are found, the COR will combine them and process a Corrective Action Report (CAR).

5.5.2. MFLTS Customer Complaint Form. The COR will furnish instructions for this form to the Customer if the Customer pursues on filing a complaint of a service or product not meeting the Customer’s expectations. In addition, the Customer will receive the contract requirements that are associated with the service or product. The Customer must provide supporting evidence of complaint or feedback. After the COR validates feedback or complaint demonstrating Contractor noncompliance, the COR will process a PAR or CAR to the PCO. All

Customer Complaint forms will be validated by the COR prior to notifying the Contractor, see

Appendix C for the Customer Complaint Form.

5.6. Contract Performance Assessment Reporting System (CPARS). CPARS

(www.cpars.gov) will be prepared at the time the work under a contract is completed, documenting Contractor performance for this contract. See FAR Part 42.1502. More frequent http://www.cpars.gov/ updates may be made as required to document a significant improvement or degradation of the level of contract performance, see Appendix D for the Performance Assessment Report.

5.7. Disputes. Every attempt will be made to resolve all disputes arising under this contact using the Alternate Dispute Resolution (ADR) as outlined in FAR 33.214. If resolution cannot be made under ADR, the PCO will render a final decision.

5.8. Performance Requirements Summary (PRS). The PRS (Table 2) provides the performance standards, AQLs, and monitoring methods that are associated with the required products and services within the PWS. Performance standards are used to define the desired outcomes. AQLs are used define the level of service required under the contract to successfully meet the performance standard. Methods of Surveillance describe how the Government will assess the required support services and system enhancements to determine if the Contractor meets the AQLs.

A-1

ACRONYMS

The following acronyms are in use within this document.

Acronym Definition

ADR Alternate Dispute Resolution

AQLs Acceptable Quality Limits

CAR Corrective Action Request

CDR Critical Design Review

CDRL Contract Data Requirements List

CLINs Contract Line Item Numbers

COR Contracting Officer Representative

CPARS Contract Performance Assessment Reporting System

FAR Federal Acquisition Regulation

FFP Firm Fixed Price

LP Language Pack

MFLTS Machine Foreign Language Translation System

PAR Performance Assessment Report

PCO Procuring Contracting Officer

PD Product Director

POP Period of Performance

PRS Performance Requirements Summary

PWS Performance Work Statement

QASP Quality Assurance Surveillance Plan

VDD Version Description Document

B-1

CORRECTIVE ACTION REPORT

CORRECTIVE ACTION REPORT (CAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACTOR

2. CONTRACT DELIVERY/TASK

ORDER NUMBER

3. TYPE OF SERVICES

4. FUNCTIONAL AREA

5. SUSPENSE DATE

6. CONTROL NUMBER

7. DEFICIENCY MAJOR MINOR

FINDING:

FINDING IMPACT:

Please respond with a written corrective action plan that details the corrective action of the cited deficiency, the cause of the deficiency, and actions taken to prevent recurrence by Suspense Date in Block 5. If date was not entered in Block 5, the Contractor is not required to provide a response.

8. QUALITY ASSURANCE PERSONNEL (QAP)

B-2

TYPED NAME AND GRADE

SIGNATURE AND DATE

9. ISSUING AUTHORITY

TYPED NAME AND GRADE

SIGNATURE AND DATE

10. QAP RESPONSE TO CONTRACTOR CORRECTIVE ACTION AND ACTION TAKEN TO PREVENT RECURRENCE

11. QAP DETERMINATION

ACCEPTED REJECTED

12. CLOSE DATE

C-1

CUSTOMER COMPLAINT RECORD TEMPLATE

CUSTOMER COMPLAINT RECORD

DATE/TIME OF COMPLAINT

SOURCE OF COMPLAINT

ORGANIZATION

BUILDING NUMBER

INDIVIDUAL

PHONE NUMBER

NATURE OF COMPLAINT

CONTRACT REFERENCE

VALIDATION

DATE/TIME CONTRACTOR INFORMED OF COMPLAINT

C-2

ACTION TAKEN BY CONTRACTOR

RECEIVED/VALIDATED BY

D-1

PERFORMANCE ASSESSMENT REPORT

PERFORMANCE ASSESSMENT REPORT (PAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACT DELIVERY/TASK ORDER

NUMBER

2. CONTRACTOR 3. TYPE OF SERVICES

4. QUALITY ASSURANCE PERSONNEL (QAP) SIGNATURE AND DATE

5. QAP PHONE 6. SUSPENSE DATE

I. PERFORMANCE

7. DEFICIENCY (CHECK ALL BOXES THAT APPLY)

NEW

REPEAT

NO DEFICIENCY NOTED

8. SERVICES SUMMARY or SOW PARAGRAPH ITEM REVIEWED

9. BRIEF DESCRIPTION OF DEFICIENCY (IF DEFICIENCY BOX WAS

CHECKED)

10. DETAILED PERFORMANCE ASSESSMENT

II. CONTRACTOR VALIDATION

11. CONTRACTOR REPRESENTATIVE CONCUR NON-

CONCUR

12. CORRECTIVE ACTION ESTIMATED COMPLETION DATE

D-2

13. CONTRACTOR REPRESENTATIVE CORRECTIVE ACTION AND PREVENTION OF RECURRENCE OR REASON FOR NON-

CONCURRENCE OF QAP CITED DEFICIENCY

III. ACTION CORRECTED

14. CONCUR NON-CONCUR QAP SIGNATURE AND DATE

15. QAP REMARKS (REQUIRED)

16. CONTRACTOR REPRESENTATIVE REMARKS