B08 - ATTACHMENT K - APLIS QASP Draft.docx

SOLUTION DELIVERY DIVISION DELIVERING SOLUTIONS, ADVANCING HEALTH CARE

Quality Assurance Surveillance Plan (QASP) for Anatomic Pathology Laboratory Information System (APLIS) Software Solution (Cerner CoPath Plus) Sustainment

Prepared by:

The Quality Management Team for the EHR CORE Program Management Office

Version 2.0 June, 2019

SDD

SOLUTION DELIVERY DIVISION

Document Approval

SUBMITTED BY:

Mala Viswanath Quality Management Lead Solution Delivery Division

*************************************************************************************************** CONCURRED BY:

[Vendor Task Manager Name] [Role] [Company]

APPROVED BY:

Ms. Lorri A. Bovain Contracting Officer’s Representative, APLIS Software Solution Sustainment EHR CORE Program Management Office

Document History

Table of Contents

List of Tables

1. Introduction Purpose A Quality Assurance Surveillance Plan (QASP) is a Government-developed and applied document used to ensure systematic quality assurance methods are implemented in the administration of the performance standards stated in the contract. The intent is to verify that the contractor performs in accordance with performance metrics set forth in the contract documents, that the Government receives the quality of services called for in the contract, and that the Government only provides compensation for the acceptable level of services received. The contractor is the individual, partnership, corporation, or other activity that sells property, goods, or services. This QASP explains the following:

· What will be monitored (including but not limited to metrics and documentation)?

· How will monitoring take place?

· Who will conduct monitoring?

· How will monitoring efforts and results be documented?

This QASP does not detail how the contractor accomplishes the work. Rather, the QASP is created with the premise that the contractor is responsible for management and quality control actions to meet the terms of the contract. It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance. In addition, the QASP recognizes that unforeseen and uncontrollable situations may occur.

This QASP is a “living document” the Government may review and revise on a regular basis. However, the Government shall coordinate changes with the contractor prior to the changes taking effect. Updates shall guarantee the QASP remains a valid, useful, and enforceable document. Copies of the original QASP and revisions shall be provided to Government officials implementing surveillance activities and the contractor.

Scope This is a QASP between the Electronic Health Record (EHR) CORE Program Management Office (PMO) and the contractor identified below:

Contract Title: Anatomic Pathology Laboratory Information System (APLIS) Software Solution (Cerner CoPath Plus) Sustainment

Contract Description:

This support shall include installation and sustainment of all test, training, and production and training environments, site Military Treatment Facility (MTF) installs, customer support, system and infrastructure maintenance at host MAAG sites, and quality assurance. This support also includes change and configuration management of current and future versions of APLIS to ensure continuous support.

This support also includes the configuration, implementation, and maintenance activities required to deploy new versions of APLIS (future releases and/or upgrades of application software throughout the life of the sustainment contract) to include interfaces with other clinical systems. All components of the application software including reports are required to perform accurately and contain no defects on all versions. All supporting and third party software shall be tracked to ensure that all versions are current. Tracking of expiration dates, licenses, end of life notifications, cybersecurity compliances are some of the attributes to be tracked in addition to others that may be identified during the contract period.

In addition, this support will also include the management of the Host Based Security System (HBSS) on the APLIS production, training, and testing servers. HBSS is a DOD Commercial off the Shelf (COTS) suite of software applications used within the DOD to monitor, detect, and counter attacks against DOD computer networks and systems.

Contract Number: [Contract Number]

Contractor: [Contracting Company]

Task Manager: Ms. Lorri A. Bovain Authority Authority for issuance of this QASP is in accordance with Federal Acquisition Regulation (FAR) Part 37.604, the FAR Part 46.4, and Section 3.0 of the Performance Work Statement (PWS). Additionally, the following FAR Clauses may apply depending on contract type:

· 52.246-4 Inspection of Services – Fixed-Price

Section 3.0 of the PWS provides for inspections and acceptance of the services and documentation outlined in contract task orders to be executed by the Contracting Officer (KO) or an authorized representative.

Authority to make changes to this document is designated to the Contracting Officer’s Representative (COR) assigned to the contract identified in Section 1.2 Scope.

Review This document will be reviewed annually and updated as needed.

Roles & Responsibilities Quality Assurance Surveillance Plan vs. Quality Control Plan The QASP is put in place to provide Government surveillance oversight of the contractor’s quality control efforts to assure timely, effective, and accurate delivery of the products and action items specified in the contract or task order. The Government representative(s) will monitor and review performance reports furnished by the contractor to determine how the contractor is performing against specified performance objectives. The contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively. The contractor uses their accepted Quality Control Plan (QCP) to guide and document the implementation of the required management and quality control actions to achieve the specified results.

Roles and Responsibilities The roles and responsibilities listed in Table 1 QASP – Roles and Responsibilities are specific to Quality Assurance and the QASP. The Project Team and the QM team shall oversee and coordinate quality surveillance activities. The roles and responsibilities listed in Table 2 QASP – Contractor Roles and Responsibilities are specific to the contract contractor. The individuals identified may have other project or organizational responsibilities not listed. Contractor personnel aside from the Program Manager and Task Manager may be identified in Table 2.

SDD Roles and Responsibilities The QM team is available to assist the Project Team with tracking and monitoring performance via the QASP Monitoring Tool (QMT), it is also an option for Project Team to control this if they so choose.

Table 1: QASP Roles and Responsibilities

· Ensures compliance with the contract terms

· Safeguards the interests of the United States in the contractual relationship

· Assures that the contractor receives impartial, fair, and equitable treatment under this contract

· Has ultimate responsible for the final determination of the adequacy of the contractor’s performance

· Acts as well as liaison between the Contracting Office and the supporting contracting office

· Observes, documents, and communicates contractor performance to both the KO and contractor

· Completes quality assurance monitoring forms (identified in Section 3.5) used to document the inspection and evaluation of the contractor’s work performance

· Maintains a quality assurance file

· Provides guidance, direction, and oversight of PMO quality assurance activities

· Assigns QA staff to specific projects

· Tracks QASP performance for identified quality metrics then reports to COR on a monthly basis and at each technical review (Note: Includes tracking of deliverable timeliness and quality)

· Customizes QASP Monitoring Tool for project tailored QASP and PWS deliverable due dates

· Works with the CORs, Project Manager, and Project Controllers in proposing metrics and for inputting those metrics accepted by the COR into the final draft QASP

· Provides Project Management support in the administration of the monitoring and controlling process to enable the success of the project.

Contractor Contact Information Table 2: QASP Contractor Contact Information

Task Manager

Contract Manager

Deliverables Lead

Solution Leader

Implementation Manager

QASP Framework Performance Management Approach The PWS structures the acquisition around “what” service or quality level is required, as opposed to “how” the contractor should perform the work (i.e., results, not process). This QASP will define the performance management approach taken by the Government to monitor and manage the contractor’s performance. Performance management rests on developing a capability to review and analyze information generated through performance assessment. The ability to make decisions based on the analysis of performance data serves as the cornerstone of performance management: this analysis yields information that indicates whether expected outcomes for the project are being achieved by the contractor.

Performance management focuses on assessing whether outcomes are being achieved and to what extent. This approach migrates away from scrutiny of compliance with the processes and practices used to achieve the outcome. A performance-based approach enables the contractor to play a large role in how the work is performed, provided the proposed processes are within the stated constraints. The only exceptions to process reviews are those required by law (i.e., federal, state, and local) and compelling business situations, such as safety and health. A “results” focus provides the contractor flexibility to continuously improve and innovate over the course of the contract as long as the critical outcomes expected are being achieved and/or the desired performance levels are being met.

Incentives Incentives shall be based on exceeding, meeting, or failing to meet performance standards. The QASP is an official documentation of the contractor’s performance. The following incentives will be used in this QASP:

· Negative/ Positive Past Performance Evaluation: QASP performance documentation may be used as objective evidence in CPARS evaluations. The Government shall use past performance to evaluate the contractor for future awards.

Methods of QA Surveillance To minimize the performance management burden, simplified surveillance methods shall be used by the Government to evaluate contractor performance when appropriate. The primary methods of surveillance are listed in Table 3: Potential Surveillance Methods. The COR will need to review the Performance Standards Summary Matrix (PSSM) in the contract to determine if the selected monitoring methods are appropriate to monitor each performance standard. Within a QASP, multiple surveillance methods may be used. The method for any given task will depend on the performance standard and Acceptable Quality Level (AQL).

Surveillance results may be used as the basis for actions (as referred to in section 3.2) against the contractor. In such cases, the Inspection of Services clause in the Contract becomes the basis for the KO’s actions.

Table 3: Potential Surveillance Methods

To be considered valid, customer complaints must be signed and submitted in writing to the COR. The COR will validate each complaint to ensure the service was required and that the requirement was not fulfilled.

Ratings Metrics and methods are designed to determine if performance exceeds, meets, or does not meet a given standard and Acceptable Quality Level (AQL). Table 4: Ratings shall be used to determine a positive, neutral, or negative outcome.

Table 4: Ratings

Documenting Performance The COR will maintain a complete quality assurance file for the life of the contract. The file will contain copies of all reports, evaluations, recommendations, and any actions related to the Government’s performance of the quality assurance function. The COR shall forward these records to the KO at termination or completion of the contract.

QASP Monitoring Tool The QMT (Appendix C) will document the Government’s assessment of the contractor’s performance under the contract to ensure that the required results are being achieved. The PQA will provide the updated QMT to the COR monthly. This tool will enable the Government to demonstrate whether the contractor is meeting the stated objectives and/or performance standards The COR will retain a copy of all QMT updates.

Unacceptable Performance When unacceptable performance occurs, the COR shall inform the contractor. The COR shall prepare a QASP Discrepancy Report (QDR) (Appendix B), and present it to the contractor's task manager or on-site representative. The contractor shall acknowledge receipt of the QDR in writing. The QDR will specify if the contractor is required to prepare a corrective action plan to document how the contractor shall correct the unacceptable performance and avoid a recurrence. The COR will retain a copy of all QDRs.

Reviews and Resolution The PQA may require correspondence between the contractor's task manager and the COR and access to CDRLs to effectively complete the QMT. Topics of discussion that might be useful for monitoring purposes that may be useful for quality monitoring:

· Monthly performance assessment data and trend analysis

· Issues and concerns of both parties

· Projected outlook for upcoming months and progress against expected trends, including a corrective action plan analysis

· Recommendations for improved efficiency and/or effectiveness

· Issues arising from the performance monitoring processes

Performance Standards Performance Standards Summary Performance standards are the benchmark against which performance is measured. As described in Section 3.3 Methods of QA Surveillance, the Government will perform surveillance as described in this plan to determine if the contractor exceeds, meets, or does not meet the defined performance standards.

Table 5: Performance Standard Summary Matrix defines performance standards that will be used to measure this contract. The Government shall use these standards to determine contractor performance and shall compare contractor performance to the AQLs.

The performance ratings in Section 3.4 Ratings are attributed to the AQLs used in Table 5: Performance Standard Summary Matrix. For certain critical activities such as those involving submission of invoices, the desired performance level may be 100 percent. Other levels of performance are bound to the importance of the task as related to the overall mission performance of the EHR CORE PMO.

Table 5: Performance Standard Summary Matrix

include all necessary data elements

PWS 9.57 - Deliverables Submit deliverables in accordance with standards and the data elements outlined by the Govt.

100% of the time submit deliverables in accordance with standards and the data elements outlined 100% CDRL Reviews Submit weekly, Monthly, Quarterly CDRL’s Negative/ Positive Past Performance Evaluation

PWS 9.28 - Risk Management Framework (RMF) for DOD Information Technology (IT) Support

(a) Submit accurately formulated risks, issues and opportunities

(b) Submit Risks, Issues and Opportunities on schedule

(c) Ensure Risks, Issues and Opportunities have actionable mitigations

Submit Weekly& Monthly CDRL’s

Negative/ Positive Past Performance Evaluation

Environment

PWS 9.40 - Reliability, Maintainability, and Availability (RM&A) – Deliverable 45 Infrastructure hosting reports indicating operational availability, mission duration, observed down time 99.9% System Availability for Production Environment

Environments

PWS 9.17 - Non-Production Environments Infrastructure hosting reports indicating operational availability, mission duration, observed down time 98% System Availability for Non-Production Environments

PWS 9.48.1 - Training Planning – Deliverable 5

PWS 9.9.2 - APLIS Tier III Support

(a) 12 hour response time to trouble tickets

(b) Trouble Ticket processing follows standards prescribed by the Govt.

Provide a response time within12 hour for 100% of trouble tickets

Appendix A: Document Standards & Checklists The following documentation standards and templates are located in the SDD Process Asset Library (PAL), examples of templates include:

· SDD Requirements Worksheet (SRW)

· Business Continuity Plan (BCP)

· Business Impact Analysis (BIA)

· Disaster Recovery Plan (DRP)

· Cyber Incident Response Plan (CIRP)

· Interface Control Document (ICD) Template

· Other Cybersecurity Templates as required by COR

Appendix B: QASP Discrepancy Report (QDR)

Contract Number:

Date and Time Discrepancy was Observed:

From:

Ms. Lorri A. Bovain

To:

Mr. Jason M. Fitch

If a written corrective action plan is required, include the following:

The written Corrective Action Plan will be provided no later than 10 days after receipt of this QDR

Prepared By:

<<COR signature>> Date:

Received By:

<<TM signature>> Date:

Appendix C1: QASP Monitoring Tool The attached tool will be used to monitor and report vendor performance against the standards of the QASP.

Appendix C2: CDRL Evaluation Template (CET)

00 July 2007 DHIMS CM 6 of 18 image2.emf

APLIS QMT.xlsm Monthly QASP Quality Assessment

QASP Monitoring Summary

Monthly QASP Quality Assessment

Not Met Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 0 0 0 0 0 0 0 0 0 0 0 0 Met Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 0 0 0 0 0 0 0 0 0 0 0 0 Not Recieved Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 0 0 0 0 0 0 0 0 0 0 0 0

Tracking Sheet

APLIS APLIS Software Solution (Cerner CoPathPlus) Sustainment

Monthly Status Status Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Not Met 0 0 0 0 0 0 0 0 0 0 0 0

Met 0 0 0 0 0 0 0 0 0 0 0 0

Not Recieved 0 0 0 0 0 0 0 0 0 0 0 0

N/A 0 0 0 0 0 0 0 0 0 0 0 0

Required Service Activity Frequency Standard AQL Method Comments

Deliverables will include all necessary data elements Sampling Submit deliverables in accordance with standards and the data elements outlined in the

PWS.

95% Direct Observation/ Progress Meetings / Progress Reports / Periodic Inspection / Periodic Sampling

Risks, Issues and Opportunities Management Weekly (a) Submit accurately formulated risks, issues and opportunities

(b) Submit Risks, Issues and Opportunities on schedule

(c) Ensure Risks, Issues and Opportunities have actionable mitigations 98% Direct Observation/ Progress Meetings / Progress Reports / Periodic Inspection / Periodic Sampling

System Availability Weekly Infrastructure hosting reports indicating operational availability, mission duration, observed down time 100% Direct Observation/ Progress Meetings / Progress Reports / Periodic Inspection / Periodic Sampling

Training Approach One-Time Activity Trainee Satisfaction Survey Results "The Four Levels of Learning Evaluation." [Kirkpatrick] or other industry standard for Government Review and Approval Accomplished in accordance with standards and the data elements outlined in the PWS 80% Direct Observation/ Progress Meetings / Progress Reports / Periodic Inspection / Periodic Sampling

Tier III Support Sampling 12-hour response time trouble tickets Tickets resolved in accordance with standards and the data elements outlined in the PWS 100% Direct Observation/ Progress Meetings / Progress Reports / Periodic Inspection / Periodic Sampling

Lists

Action Item

O'Neal, Amy, CTR, OASD(HA)/TMA: O'Neal, Amy, CTR, OASD(HA)/TMA:

NDAs, HIPAA Training certificates, TOMP, WBS, WBS Dictionary, Transition Plans, Closeout Brief, Contract Management Plans CDR Not Met Weekly

Architecture Documentation

O'Neal, Amy, CTR, OASD(HA)/TMA: O'Neal, Amy, CTR, OASD(HA)/TMA:

QCP, RMP, QASP, Lessons Learned, Issue Management Plan, Project Management Plans On-Going Met Monthly

Change Management

Hearn, Jessica, CTR, OASD(HA)/TMA: Hearn, Jessica, CTR, OASD(HA)/TMA:

Software or Hardware Change Requests PDR N/A Quarterly

Coding

Hearn, Jessica, CTR, OASD(HA)/TMA: Hearn, Jessica, CTR, OASD(HA)/TMA:

Software Code Deliveries Post-Contract Award Not Recieved Sampling

Contract Management SRR Ad-Hoc

Information Assurance TRR One-Time Activity

Issue and Risk Management TRR: SAT

Schedule Management TRR: SIT 1

Status Reporting

Hearn, Jessica, CTR, OASD(HA)/TMA: Hearn, Jessica, CTR, OASD(HA)/TMA:

Monthly, Weekly, etc. Status Reports

TRR: SIT 2

Technical Documentation TRR: SIT 3

Testing TRR: SQT image3.emf

CDRL Evaluation Template.xlsx CDRL Evaluation Overview

CDRL Evaluation Template

PROJECT TITLE

PROJECT MANAGER

PMO

Vendor

DURATION % of TASK

CDRL CDRL Description CDRL Frequency Late (Days) Task Order # Times Rejected Notes

CDRL 1 0

CDRL 2 0

Project Data

SBM # CDRLS CDRL # CDRL Name CDR Frequency Days Initial Due Date Task CUE Only Initial Submission Date Submission Days Late Submission Months Late Contract Start Date Reject Count Accepted Date Comment Status

CDRL 1 0.0 0.0

CDRL 2 0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0

0.0 0.0 image1.png