Attachment 1 - QASP.pdf

U.S. Department of Justice Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) Office of Management (OM) Acquisitions Management Division (AMD) Mission Support Branch (MSB) www.atf.gov

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

Laboratory Information Management System (LIMS) Modernization

Solicitation: 15A00025R00000058

Attachment 1

TABLE OF CONTENTS

1 INTRODUCTION

1.1 Purpose

1.2 Performance Management Approach

1.3 Performance Management Strategy

2 ROLES AND RESPONSIBILITIES

2.1 The Contracting Officer (CO)

2.2 The COR

3 IDENTIFICATION OF REQUIRED PERFORMANCE STANDARDS/QUALITY

LEVELS

4 METHODOLOGIES TO MONITOR PERFORMANCE

2.3 Customer Feedback

2.4 Acceptable Quality Levels

3 QUALITY ASSURANCE DOCUMENTATION

3.1 The Performance Management Feedback Loop

1.1 Monitoring Forms

2 ANALYSIS OF QUALITY ASSURANCE ASSESSMENT

2.1 Determining Performance

2.2 Reporting

2.3 Reviews and Resolution

ATTACHMENT 1: PERFORMANCE REQUIREMENTS SUMMARY

ATTACHMENT 2: SAMPLE QUALITY ASSURANCE MONITORING FORM

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

1 INTRODUCTION

This quality assurance surveillance plan (QASP) is pursuant to the requirements listed in the performance work statement (PWS) entitled Laboratory Information Management System (LIMS) Modernization. This plan sets forth the procedures and guidelines Laboratory Services will use in ensuring the required performance standards or services levels are achieved by the contractor.

1.1 Purpose

1.1.1 The purpose of the QASP is to describe the systematic methods used to monitor performance and identify required documentation and resources to be employed. The QASP provides a means for evaluating whether the contractor is meeting the performance standards/quality levels identified in the PWS and the contractor’s quality control plan (QCP), and ensure the government pays only for the level of services received.

1.1.2 This QASP defines the roles and responsibilities of all members of the integrated project team (IPT), identifies the performance objectives, defines the methodologies used to monitor and evaluate the contractor’s performance, describes quality assurance documentation requirements, and describes the analysis of quality assurance monitoring results.

1.2 Performance Management Approach

1.2.1 The PWS structures the acquisition around what service or quality level is required, as opposed to how the contractor should perform the work (i.e., results, not compliance).

This QASP will define the performance management approach taken by Laboratory Services to monitor and manage the contractor’s performance to ensure the expected outcomes or performance objectives communicated in the PWS are achieved.

Performance management rests on developing a capability to review and analyze information generated through performance assessment. The ability to make decisions based on the analysis of performance data is the cornerstone of performance management; this analysis yields information that indicates whether expected outcomes for the project are being achieved by the contractor.

1.2.2 Performance management represents a significant shift from the more traditional quality assurance (QA) concepts in several ways. Performance management focuses on assessing whether outcomes are being achieved and to what extent. This approach migrates away from scrutiny of compliance with the processes and practices used to achieve the outcome. A performance-based approach enables the contractor to play a large role in how the work is performed, if the proposed processes are within the stated constraints.

The only exceptions to process reviews are those required by law (federal, state, and local) and compelling business situations, such as safety and health. A “results” focus provides the contractor flexibility to continuously improve and innovate over the course of the contract if the critical outcomes expected are being achieved and/or the desired performance levels are being met.

1.3 Performance Management Strategy

1.3.1 The contractor is responsible for the quality of all work performed. The contractor measures that quality through the contractor’s quality control (QC) program. QC is work output, not workers, and therefore includes all work performed under this contract regardless of whether the work is performed by contractor employees or subcontractors.

The contractor’s QCP will set forth the staffing and procedures for self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the PWS. The contractor will develop and implement a performance management system with processes to assess and report its performance to the designated government representative. The contractor’s QCP will also set forth the staffing and procedures for self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the PWS. This QASP enables the government to take advantage of the contractor’s QC program.

1.3.2 The government representative(s) will monitor performance and review performance reports furnished by the contractor to determine how the contractor is performing against communicated performance objectives. The government will make determinations regarding incentives based on performance measurement metric data and notify the contractor of those decisions. The contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively.

2 ROLES AND RESPONSIBILITIES

2.1 The Contracting Officer (CO)

The CO is responsible for monitoring contract compliance, contract administration, and cost control, and resolving any differences between observations documented by the Contracting Officer's Representative (COR) and Government Program Manager (PM). The CO will designate one full-time COR as the government authority for performance management. The number of additional representatives serving as technical inspectors depends on the complexity of the services measured, as well as the contractor’s performance, and must be identified and designated by the CO.

2.2 The COR

The COR is designated in writing by the CO to act as their authorized representative to assist in administering a contract. COR limitations are contained in the written appointment letter. The COR is responsible for technical administration of the project and ensures proper government surveillance of the contractor’s performance. The COR is not empowered to make any contractual commitments or authorize any contractual changes on the government’s behalf. Any changes the contractor deems may affect contract price, terms, or conditions must be referred to the CO for action. The COR will have the responsibility for completing QA monitoring forms used to document the inspection and evaluation of the contractor’s work performance.

Government surveillance may occur under the inspection of services clause for any service relating to the contract.

3 IDENTIFICATION OF REQUIRED PERFORMANCE STANDARDS/QUALITY

LEVELS

The required performance standards and/or quality levels are included in the PWS and Attachment 1, Performance Requirements Summary. If the contractor meets the required service or performance level, it will be paid the monthly amount agreed on in the contract.

The required performance standards and/or quality levels are included in the PWS and Attachment 1, Performance Requirements Summary. If the contractor meets or exceeds the required service or performance level, it will be documented favorably within CPARS. If the contractor fails to meet the required service or performance level, it will be documented unfavorably within CPARS.

4 METHODOLOGIES TO MONITOR PERFORMANCE

4.1 Surveillance Techniques

To minimize the performance management burden, simplified surveillance methods shall be used by the government to evaluate contractor performance when appropriate. The primary methods of surveillance are:

• Random monitoring, which shall be performed by the COR designated inspector

• 100 percent inspection – Upon deliverable receipt, the COR shall review the generated documentation and enter summary results into the Surveillance Activity Checklist

• Periodic inspection – the COR typically performs the periodic inspection

2.3 Customer Feedback

The contractor is expected to establish and maintain professional communication between its employees and customers. The primary objective of this communication is customer satisfaction.

Customer satisfaction is the most significant external indicator of the success and effectiveness of all services provided and can be measured through customer complaints.

Performance management drives the contractor to be customer focused through initially and internally addressing customer complaints and investigating the issues and/or problems, but the customer always has the option to communicate complaints to the CO or COR, as opposed to the contractor.

Customer complaints, to be considered valid, must set forth clearly and in writing the detailed nature of the complaint, must be signed, and must be forwarded to the COR. The COR will accept those customer complaints and investigate.

Customer feedback may also be obtained either from the results of formal customer satisfaction surveys or from random customer complaints.

2.4 Acceptable Quality Levels

The acceptable quality levels (AQLs) included in Attachment 1 for contractor performance are structured to allow the contractor to manage how the work is performed while providing negative incentives for performance shortfalls. For certain critical activities, such as those involving the solution deployment to development/test and production environments, existing LIMS data migration, and end-user training, the desired performance level is established at 100 percent.

Other levels of performance are keyed to the relative importance of the task to the overall mission performance at Laboratory Services.

3 QUALITY ASSURANCE DOCUMENTATION

3.1 The Performance Management Feedback Loop

The performance management feedback loop begins with the communication of expected outcomes. Performance standards are expressed in the PWS and assessed using the performance monitoring techniques shown in Attachment 1.

1.1 Monitoring Forms

The government’s QA surveillance, accomplished by the COR, will be reported using the monitoring form in Attachment 2. The forms, when completed, will document the government’s assessment of the contractor’s performance under the contract to ensure the required results are being achieved.

1.1.1 The COR will retain a copy of all completed QA surveillance forms.

2 ANALYSIS OF QUALITY ASSURANCE ASSESSMENT

2.1 Determining Performance

The government shall use the monitoring methods cited to determine whether the performance standards/service levels/AQLs have been met. If the contractor has not met the minimum requirements, it may be asked to develop a corrective action plan to show how and by what date it intends to bring performance up to the required levels. If the contractor fails to meet the required service or performance level, it will be documented unfavorably within CPARS. If the contractor meets or exceeds the required service or performance level, it will be documented favorably within

CPARS.

2.2 Reporting

2.2.1 At the end of each month, the COR will prepare a written report for the CO summarizing the overall results of the quality assurance surveillance of the contractor’s performance.

This report, which includes the contractor’s submitted monthly report and the completed quality assurance monitoring forms (Attachment 2), will become part of the QA documentation. It will enable the government to demonstrate whether the contractor is meeting the stated objectives and/or performance standards, including cost/technical/scheduling objectives.

2.3 Reviews and Resolution

2.3.1 The COR may require the contractor’s project manager or designated alternate to meet with the CO and other government IPT personnel as deemed necessary to discuss the performance evaluation. The CO will define a frequency of in-depth reviews with the contractor, including appropriate self-assessments by the contractor; however, if the need arises, the contractor will meet with the COR as often as required or per the contractor’s request. The agenda of the reviews may include:

• Monthly performance assessment data and trend analysis

• Issues and concerns of both parties

• Projected outlook for upcoming months and progress against expected trends, including a corrective action plan analysis

• Recommendations for improved efficiency and/or effectiveness

1.1.1 The QAR must coordinate and communicate with the contractor to resolve issues and concerns regarding marginal or unacceptable performance.

1.1.2 The COR and contractor should jointly formulate tactical and long-term courses of action. Decisions regarding changes to metrics, thresholds, or service levels should be clearly documented. Changes to service levels, procedures, and metrics will be incorporated as a contract modification at the convenience of the procuring CO (PCO) or

ACO.

Firearms and Explosives (ATF) Office of Management (OM) Acquisitions Management Division (AMD)

ATTACHMENT 1: PERFORMANCE REQUIREMENTS SUMMARY

Performance

Objective Performance

Standard Acceptable Quality

Level Source Method of

Assessment Who is

Responsible How measure is calculated

Incentive

Objective 1

Kick off meeting

Agenda

Meeting minutes

Agenda and meeting minutes delivered

Contractor Program Manager/Project Manager, Training Coordinator, Technical Manager attendance

Kick off meeting Meeting attendance, observation, 100% inspection

Contractor

PM

Meeting attendance of all Contractor key personnel

CPARS:

Manageme nt

Project Management Plan

Project Management Plan delivered

The Contractor shall deliver a Project Management Plan that includes at a minimum, the Objectives listed in this PWS.

Project Management Plan

100% inspection

Contractor

PM

Acceptance of Project Management Plan by Government

PM

CPARS:

Manageme nt/Schedule

Project Schedule Project Schedule delivered

The Contractor shall deliver a Project Schedule that lists at a minimum, the Objectives listed in this

PWS.

Project Schedule 100% inspection

Contractor

PM

Acceptance of Project Schedule by Government

PM

CPARS:

Manageme nt/Schedule

Objective

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive

Support Contact Information

Support Contact Information delivered

Contractor key and supporting personnel are identified by name, title, contact information (email and telephone number), and typical work hours defined.

Support Contact Information

100% inspection

Contractor

PM

Acceptance of Support Contact Information by Government

PM

CPARS:

Manageme nt/Schedule

Hardware/Software Requirements Listing

Hardware/Softw are Requirements Listing delivered.

Detailed listing of LIMS hardware and software requirements provided by the Government

Hardware/Softwar e Requirements

100% inspection

Contractor

PM

Acceptance of training plan by Government

PM

CPARS:

Manageme nt/Schedule

Objective 2

Pilot Requirements Validation Checklist

Pilot Requirements Validation Checklist

Document for use by the Contractor to validate each LIMS environment deployed against Government requirements.

Pilot Requirements Validatiton Checklist

100% document review to ensure understanding of requirements defined.

Contractor

PM

Government

PM

CPARS:

Manageme nt

Objective 3

LIMS Development Environment

Deploy LIMS solution in the

LIMS development environment shall be fully operational and configured to support

LIMS

Development Environment

100% inspection

Contractor Pilot testing criteria based off the LIMS business

CPARS:

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive development environment.

the pilot user acceptance testing.

requirements defined in this

PWS.

Manageme nt/Quality

Objective 4

Data Migration Plan

Data Migration Plan delivered

The Contractor shall deliver a Data Migration Plan that identifies the tasks associated with the data migration efforts from the existing LIMS to the new LIMS solution.

Data Migration Plan

100% inspection

Contractor Acceptance of Data Migration Plan by Government

PM

CPARS:

Manageme nt/Schedule

Data Migration Schedule

Data Migration Schedule

The Contractor shall deliver a Data Migration Schedule that identifies the start/end timelines for each of the Data Migration Plan tasks.

The schedule shall identify the complete data migration timeline.

Data Migration Schedule

100% inspection

Contractor

PM

Acceptance of Data Migration Schedule by Government

PM

CPARS:

Schedule

Data Migration Mapping

Data Migration Mapping

The Contractor shall deliver a Data Migration Map that identifies each data record element that will be migrated from the

Data Migration Mapping

100% inspection

Contractor Acceptance of Data Migration Mapping by

Schedule

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive existing LIMS solution to the new LIMS solution. The data map shall show old table/field identifiers and the new table/field where the data record set will be migrated from/to.

Government

PM

Entity Relationship Diagram (ERD)

ERD delivered The ERD shall include a full listing of database tables and fields to include table joins and primary keys.

Entity Relationship Diagram (ERD)

100% inspection

Contractor Acceptance by the Government

PM

CPARS:

Schedule

Objective 5

LIMS Test Environment

Deploy LIMS solution in the test environment.

LIMS test environment shall be fully operational, configured to support the pilot user acceptance testing and training purposes.

LIMS Test Environment

Pilot testing by trained pilot testing users selected by the Government.

Contractor Pilot testing criteria based off the LIMS business requirements defined in this

PWS.

CPARS:

Manageme nt/Quality

Section 508 Compliance

Section 508 Compliance

Solution shall be 100% compliant with Section

LIMS Test Environment

Pilot testing and use of Section 508 Compliance Checklist

Contractor 100% compliance on Section

CPARS:

nt/Quality

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive

Compliance Checklist

Objective 6

Pilot user training Training results in users’ effective ability to use all aspects of the system.

100% of pilot users are able to effectively use the system after training.

Knowledge check, course survey, participant follow-ups, course observation, all training material provided as referenced.

100% inspection

Contractor training coordinator

Number of students who pass the knowledge check with 100% divided by the total number of students trained.

CPARS:

Quality

Objective 7

Pilot Plan of Action, if applicable

Deploy system fixes/solutions that resolve issues identified during pilot testing.

100% issue resolved LIMS Test Environment

Pilot re-testing by trained pilot testing users selected by the Government.

Contractor Issue identified during pilot testing.

CPARS:

Manageme nt/Cost Control

Objective 8

Training Plan Training Plan delivered

Training plan includes curriculum, and any supporting training material referenced (i.e.

Training Plan

100% inspection

Contractor training coordinator

Acceptance of training plan by

Quality

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive

Quick reference guides, user manual, etc.).

Government

PM

Training schedule Training Schedule delivered

Schedule includes training location, training dates and times, alternate make-up sessions defined.

Training Schedule 100% inspection

Contractor training coordinator

Acceptance of training schedule by Government

PM

CPARS:

Quality

Train-The-Trainer, Training

Training results in users’ effective ability to use all aspects of the system at a capacity level to train a new user.

100% of system users are able to effectively use the system after training.

Knowledge check, course survey, participant follow-ups, course observation, all training material provided as referenced in the Training Plan.

100% inspection

Contractor training coordinator

Number of students who pass the knowledge check with 100% divided by the total number of students trained.

CPARS:

Quality

All-Hands Training Training results in users’ effective ability to use all aspects of the system.

90% of system users are able to effectively use the system after training.

Knowledge check, course survey, participant follow-ups, course observation, all training material provided as referenced in the Training Plan.

100% inspection

Contractor training coordinator

Number of students who pass the knowledge check with 100% divided by the total number of students trained.

Quality

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive

Training Material The Contractor shall deliver a user manual that covers the functionality of all aspects of the system.

User Manual covers all system elements, is accurate, professionally drafted and user-friendly.

Final User Manual is received 1 week prior to system launch.

User Manual 100% inspection

Contractor training coordinator

Acceptance of the User Manual by Government

PM

CPARS:

Quality

Objective 9

Deploy LIMS production environment

Deploy LIMS solution in the production environment.

LIMS production environment shall be fully operational with full license package applied. The system will be fully configured with the Government requirements to include any changes resulting from the Pilot Plan of Action.

LIMS Production Environment

100% inspection and user acceptance testing to include all discipline workflows, system reports and analytics

Contractor User acceptance testing criteria based off the

LIMS

business requirements defined in this PWS and Pilot Action Plan, if applicable.

CPARS:

Manageme nt/Quality

Attain ATO Authorization to Operate

Achieve successful Authorization to Operate that includes successful completion of required security

LIMS Production Environment

100% inspection

Contractor Government executed security vulnerability and

CPARS:

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive vulnerability and compliance scans.

compliance scanning.

Full Software/Licensing Package

Deliver complete LIMS software package

The Contractor shall deliver the full LIMS software installation package to include all executables for any modifications, documentation, training material, license keys for a complete LIMS deployment.

LIMS Software, licensing and Documentation

100% inspection

Contractor Acceptance by Government

PM

CPARS:

Schedule

Objective 10

Data Migration Successful completion of data migration from existing LIMS to new LIMS solution.

A test data migration will be conducted on the test environment prior to final

A successful data migration will include 100% of defined data records in accordance with the Data Mapping with zero (0%) data loss.

Data records shall be transferred without failure or corruption and be retained within the new LIMS specified database

LIMS Production Environment

100% inspection

100% record count matching

Data will be randomly sampled.

Contractor Acceptance by Government

PM

CPARS:

Performance Standard

Acceptable Quality Level

Source Method of Assessment

Who is Responsible

How measure is calculated

Incentive production data migration.

table(s) and field(s), as defined in the Data Mapping.

Migrated data shall be accessible for user access within the new LIMS solution for user record updating and reporting purposes.

Objective 11

Enhanced analytics and reporting

LIMS provided reporting and analytics

Any supplied LIMS reports and analytics included in the software system.

Reports and analytics to include any active dashboard reporting.

This shall include LIMS supplied reporting and dashboard analytics as well as, any Contractor developed.

100% inspection and user testing

Laboratory Services management team

CPARS:

Manageme nt/Quality/ Cost Control

Firearms and Explosives (ATF) Office of Management (OM) Acquisitions Management Division (AMD)

ATTACHMENT 2: SAMPLE QUALITY ASSURANCE MONITORING FORM

SERVICE or STANDARD:

SURVEY PERIOD:

SURVEILLANCE METHOD (check):

Random Sampling 100% Inspection Periodic Inspection Customer Complaint

LEVEL OF SURVEILLANCE (check):

Monthly Quarterly As needed

PERCENTAGE OF ITEMS SAMPLED DURING SURVEY PERIOD: ______ %

ANALYSIS OF RESULTS:

Observed Service Provider Performance Measurement Rate: ______%

Service Provider’s Performance (check): Meets Standards

Does Not Meet Standards

Narrative of Performance During Survey Period:

PREPARED BY: ___________________________________ DATE: _________________