VA118-17-R-2324-010.docx

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

DEPARTMENT OF VETERANS AFFAIRS

Electronic Health Record Modernization (EHRM) Program Executive Office

VA Electronic Health Record Modernization (EHRM) System

Date: October 4, 2017

QASP Version Number: 1.0

THIS DOCUMENT DOES NOT GET INCORPORATED INTO THE CONTRACT; IT IS USED BY GOVERNMENT REPRESENTATIVES TO MONITOR PERFORMANCE AND THE GOVERNMENT HAS THE RIGHT TO CHANGE ITS METHOD OF SURVEILLANCE AT ANY TIME. THE CO/COR SHOULD PROVIDE A COPY TO THE CONTRACTOR THROUGH CORRESPONDENCE.

QUALITY ASSURANCE SURVEILLANCE PLAN

For: VA Electronic Health Record Modernization (EHRM) System

Contract Number: Blank until contract award

Contract Description: The Department of Veterans Affairs (VA) is replacing its current Electronic Health Record (EHR), Veterans Information Systems and Technology Architecture, with a commercial EHR to modernize VA EHR processing and to promote interoperability of medical data between the Department of Defense, VA and community providers. The Contractor shall provide, host, and deploy the EHRM system across the VA enterprise including the following areas: project management, change management, training, testing, deployment services, sustainment and other solutions encompassing the entire range of EHR requirements, to include hosting, software, and hardware incidental to the solution. The Contractor shall develop and maintain interfaces as required to meet EHRM solution requirements. The Contractor shall deploy EHRM to all VA Medical Centers, Outpatient Clinics and other approved users of VA EHR functionality. The Consolidated Mail Outpatient Pharmacies will have access to the EHRM. The Contractor shall support the Veterans Health Administration revenue cycle reporting, business intelligence, data analysis and new employee/continuing education training activities and system optimization. EHRM capabilities will also be deployed to Veteran Benefits Administration facilities and/or to support other VA use cases as required.

Contractor’s Name: Cerner Corporation

Purpose This Quality Assurance Surveillance Plan (QASP) provides a systematic method to monitor Contractor performance. This QASP describes:

What will be monitored How monitoring will take place Who will conduct the monitoring How monitoring efforts and results will be documented

Contractor performance and reporting requirements as they pertain to this QASP are defined in the EHRM Indefinite Delivery/Indefinite Quantity (ID/IQ) Performance Work Statement (PWS) and accompanying attachments such as the Requirements Traceability Matrix (RTM), Software Assurance Standard Operating Procedure, and Veteran-Focused Integration Process (VIP) Major Program Deployment Guide; referred to herein as “PWS”. Additionally, Functional and Non-Functional Key Performance Indicators (KPIs) are defined in Appendix A of this document. The Government may also utilize Cerner’s commercially available KPIs and Service Level Agreements (SLAs) to monitor and measure Contractor performance. Contractor performance and reporting requirements will be further defined for individual Task Order (TO) requirements. Additional KPIs may also be defined at later dates for individual TO requirements, or at the IDIQ level as approved through EHRM governance boards, as needed and as determined by the government in accordance with the organization’s mission. Copies of the basic and individual TO QASPs and revisions shall be provided to the Contractor and Government officials responsible for surveillance activities. The Government can change the method of surveillance at any time without the approval of the contractor.

Performance Management Approach The PWS sets forth “what” service is required as well as the performance standards associated with that task, as opposed to “how” the Contractor should perform the work (i.e., results, not compliance). This QASP will define the performance management approach taken by the EHRM Program Executive Office (PEO) to monitor the Contractor’s performance to ensure the expected outcomes or performance standards communicated in the PWS, Appendix A of this document, and Cerner’s commercially available KPIs and SLAs are achieved. Performance management rests on developing a capability to review and analyze information generated through performance assessment. The ability to make decisions based on the analysis of performance data is the cornerstone of performance management; this analysis yields information that indicates to what extent the expected outcomes for the project are being achieved by the Contractor.

Performance management represents a significant shift from the more traditional quality assurance (QA) concepts in several ways. Performance management focuses on assessing whether outcomes are being achieved and to what extent. This approach migrates away from scrutiny of compliance with the processes and practices used to achieve the outcome. A performance-based approach enables the Contractor to play a large role in how the work is performed, as long as the proposed processes are within the stated constraints. Required processes are those required by law (federal, state, and local) and compelling business situations, such as safety and health. A “results” focus by the Government provides the Contractor flexibility to continuously improve and innovate over the course of the contract as long as the critical outcomes expected are being achieved and/or the desired performance levels are being met.

Performance Management Strategy The Contractor is responsible for the quality of all work performed. The Contractor measures that quality through the Contractor’s own quality control (QC) program. QC is work output, not workers, and therefore includes all work performed under this contract regardless of whether the work is performed by Contractor employees or by Subcontractors. The Contractor’s QC program will set forth the procedures for self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the PWS. The Contractor will implement a performance management system with processes to assess and report its performance to the designated government representative.

The government representative(s) will monitor performance by the Contractor to determine how the Contractor is performing against performance standards. The Contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively. The Contractor will be monitored and assessed throughout the period of performance of the contract, and each resulting TO, as to either meeting or not meeting the performance thresholds stated in the PWS. The Contracting Officer’s Representative (COR) will perform quarterly assessments at the program level. An alternative frequency, in which the COR will perform assessments, may be identified for each individual TO. The Performance Based Service Assessment, or other method, may be used to document this assessment. A Performance Based Service Assessment is provided as Appendix B of this document. When Contractor performance is unacceptable, the Government Task Leader (GTL)/VA Program Manager (PM) will notify the COR, who will then formally notify Cerner’s Engagement Leader and the Contracting Officer (CO). Unacceptable performance is defined as; “the contractor is not meeting the Acceptable Levels of Performance (ALPs) as defined in the PWS, Appendix A of this document, and/or Cerner’s commercially available KPIs and SLAs or is in violation of any contract clause or terms and conditions.” Notification of unacceptable performance issues shall be immediately provided to the Engagement Leader and shall be addressed prior to the end of an assessment period. In order to remediate performance issues in a timely manner, the COR should work collaboratively with the Engagement Leader and GTL/VA PM. The COR/CO will engage the Engagement Leader to resolve the discrepancy.

Performance Feedback The COR will assign a severity level (e.g. low [minimal impact], medium [moderate impact], high [severe impact]) to each performance issue based upon VA PM and/or GTL feedback. For instances where immediate notification of performance issues is not required, the COR should review the assessment in accordance with the quarterly reviews with the Engagement Leader and provide the assessment to CO. The COR/CO will notify the Contractor of the results for a rating of 3 or less, no later than 15 working days after the end of the assessment period.

Government Roles and Responsibilities The following personnel shall oversee and coordinate surveillance activities:

a. Contracting Officer (CO) - The CO shall ensure performance of all necessary actions for effective contracting, ensure compliance with the contract/order terms, and shall safeguard the interests of the United States in the contractual relationship. The CO shall also ensure that the Contractor receives impartial, fair, and equitable treatment under this contract and resulting TOs. The CO is ultimately responsible for the final determination of the level of acceptability of the Contractor’s performance.

Assigned CO: Matthew Truex, Contracting Officer Organization: VA, Office of Procurement, Acquisition and Logistics (OPAL), Technology Acquisition Center (TAC)

b. Contracting Officer’s Representative (COR) - The COR is responsible for technical administration of the contract/order and shall assure proper Government surveillance of the Contractor’s performance. The COR shall keep a quality assurance file. This file shall contain all quality assessment reports. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. A COR shall be designated for each individual TO.

Assigned COR: Name and organization

c. Government Task Leader (GTL) – The GTL is the Requiring Activity’s designated onsite representative. A GTL may be designated for each TO and is the COR’s primary Point of Contact for managing the TO award and administration processes. The GTL will be appointed by the Contracting Officer and duties delegated in an appointment letter. The GTL shall coordinate performance assessment information with the COR, as well as keep adequate performance records to be included in the COR’s quality assurance file. Only Government employees are permitted to be GTLs. The GTL is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. Note: In coordination with the GTL or when a GTL is not designated for a particular TO, the VA PM will assume these responsibilities.

Assigned GTL: Name and organization

d. Other Key Government Personnel:

a. Capt. John Windom, EHRM Program Executive

b. Fred Mingo, EHRM Chief Operating Officer

c. Ash Zenooz, EHRM Chief Medical Officer

d. Jonathan Nebeker, EHRM Chief Medical Informatics Officer

e. John Short, EHRM Chief Technical Officer

f. Roger Drye, EHRM Chief Business Officer

Contractor Representatives

a. Engagement Leader – To be completed at award

b. Other Contractor Personnel – To be completed at award; if any (name and title)

Performance Standards Performance Standards define required performance for each of the performance objectives. The Government performs surveillance to determine if the Contractor exceeds, meets or does not meet these standards.

The Performance Metrics for Performance Standards are defined in the PWS, Appendix A of this document, and Cerner’s commercially available KPIs and SLAs. Additional performance metrics and standards to be met during contract performance will be identified in each individual TO. The schedule of deliverables is outlined in Section B of the contract/order. The Government may utilize the Performance Based Service Assessment, provided as Appendix B of this document, or other methods to compare Contractor performance to the ALPs.

Methods of QA Surveillance Various methods exist to monitor performance. Appendix A of this document defines the methods by which the Government plans to collect and measure performance data; the COR, in coordination with the VA PM/GTL, will then use this data to assess performance. The Government shall use any or a combination of the surveillance methods listed below in assessing performance/data collected using this QASP. Surveillance methods may be further specified for each TO.

1. 100% INSPECTION. (Evaluates all outcomes to include tasks and deliverables.)

a. Each quarter, the COR shall review all of the Contractor’s performance/generated documentation and document the results accordingly. This assessment shall be placed in the COR’s QA file.

2. 100% INSPECTION (of a Specific type of Deliverable) in a Specific Area or Task

a. Each quarter, the COR shall review the Contractor’s performance/generated documentation and document the results accordingly. This assessment shall be placed in the COR’s QA file.

3. Random Sample

a. Each quarter, the COR shall review a random sampling of the Contractor’s performance/generated documentation and document the results accordingly. This assessment shall be placed in the COR’s QA file.

4. Validated Customer Complaint or Validated Below Average Acceptable Level of Performance (ALP) in a Specific Area

a. Each quarter, the COR shall review the Contractor’s performance/generated documentation corresponding to a validated customer complaint or validated inability to perform in accordance with the ALP in a specific area and document the results accordingly. This assessment shall be placed in the COR’s QA file.

5. On-Site Surveillance

a. Each quarter, the COR shall review findings from on-site surveillance and document the results accordingly. This assessment shall be placed in the COR’s QA file.

6. End-User / Patient Feedback (e.g. Focus group, survey, interviews)

a. Each quarter, the COR shall review end-user/patient feedback and document the results accordingly. This assessment shall be placed in the COR’s QA file.

7. Data Collection / Studies / Analyses (e.g. Utilization, Baseline, Comparative)

a. Each quarter, the COR shall review collected data, studies, and analyses and document the results accordingly. This assessment shall be placed in the COR’s QA file.

Acceptable Levels of Performance (ALP) Metrics and methods are designed to determine if performance exceeds, meets, or does not meet a given standard and ALP.

The ALPs are included at both the ID/IQ and TO levels for Contractor performance and are structured to allow the Contractor to manage how the work is performed, while providing negative incentives for performance shortfalls.

Performance and payment considerations The Government shall consider the Contractor’s performance when making a determination to execute future TOs, exercise any options, and authorize TO payments.

Documenting Performance

a. ACCEPTABLE PERFORMANCE

The Government shall document acceptable performance accordingly. Report templates are shown in QASP Appendices B and C. Any report may become a part of the supporting documentation for any contractual action.

b. UNACCEPTABLE PERFORMANCE

When unacceptable performance occurs, the GTL/VA PM will immediately inform the COR, who in turn shall inform the CO. This will always be in writing although when circumstances necessitate immediate verbal communication, that communication will be followed up in writing. The COR shall document the discussion and place it in the COR file.

When the CO determines formal written communication is required, the COR/GTL shall prepare a Contract Discrepancy Report (CDR), provided as Appendix C of this document, and present it to the Engagement Leader.

The Contractor shall acknowledge receipt of the CDR in writing to the CO. The CDR will state how long after receipt the Contractor has to take corrective action. The CDR will also specify if the Contractor is required to prepare a corrective action plan to document how the Contractor shall correct the unacceptable performance and avoid a recurrence. The CO shall review the Contractor's corrective action plan to determine acceptability.

Any CDRs may become a part of the supporting documentation for any contractual action deemed necessary by the CO.

Frequency of Measurement

a. Frequency of Measurement.

During contract/order performance, the COR will periodically analyze whether the negotiated frequency of surveillance is appropriate for the work being performed, and at a minimum shall be quarterly.

b. Frequency of Performance Assessment Meetings.

The COR shall meet with the Contractor quarterly to assess performance and shall provide a written assessment to the CO. An alternative frequency for performance assessment meetings may be identified for each TO.

Quality of submission should also be considered. See examples below. Error rates or resubmits for content flaws would be the measures associated with these standards.

· Accuracy Work Products shall be accurate in presentation, technical content, and adhere to accepted elements of style.

· Clarity Work Products shall be clear and concise. Any/All diagrams shall be easy to understand and be relevant to the supporting narrative.

· Consistency to Requirements All work products must satisfy the requirements of this PWS.

· File Editing All text and diagrammatic files shall be editable by the VA in Windows-based or Adobe environments/platforms.

· Format - Follow specified VA Directives or Manuals and/or best business practices.

· Presentations - Presentations shall be clear, concise, executive-focused, and written in plain, clear English with minimal jargon, understandable by lay persons. The quality of deliverables directly contributes to organizational communications.

· Project Plan - Project Plan shall be comprehensive; recognize and address authority, perceptions, and concerns of stakeholders; incorporate scope of requisite requirements across the organization and/or agency.

· Reports - There shall be no omissions in the reports, documents or functional requirements.

· Publications and other documents - Deliverables shall be in formats appropriate to target audiences; user friendly, clear, thorough and comprehensive.

· Meeting support - Pre-meeting preparations and logistics demonstrate smooth meeting operations; complete comprehensive post-meeting summaries to include but not limited to: Minutes, Action Items, Attendees, Program Objectives and Milestones and major decision points.

· Analyses and Assessments - Analyses and assessments are performed with accuracy, completeness and adherence to industry best practices.

Obtain stakeholder input. Deliverables shall consist of the timely implementation of input mechanisms, and shall consist of an accurate and comprehensive synthesis of results and recommendations. Integration of relevant stakeholder input documented for each deliverable.

EVALUATION RATINGS DEFINITIONS

The Government will evaluate the contractor’s performance for each TO in accordance with the following Contractor Performance Assessment Reporting System (CPARS) ratings:

Exceptional

Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

To justify an Exceptional rating, identify multiple significant events and state how they were of benefit to the Government. A singular benefit, however, could be of such magnitude that it alone constitutes an Exceptional rating. Also, there should have been NO significant weaknesses identified.

Very Good

Performance meets contractual requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

To justify a Very Good rating, identify a significant event and state how it was a benefit to the Government. There should have been no significant weaknesses identified.

Satisfactory

Marginal

Unsatisfactory

Appendix A

FUNCTIONAL AND NON-FUNCTIONAL KEY PERFORMANCE INDICATORS

[Refer to attached files: Appendix A - 1 EHRM Functional Key Performance Indicators and Appendix A - 2 EHRM Non-Functional Key Performance Indicators]

Appendix B

PERFORMANCE BASED SERVICE ASSESSMENT

CONTRACTOR: ________________________

GOVERNMENT REQUIRING ACTIVITY: _____________________

CONTRACT/ORDER NUMBER/ TITLE: _____________________

PERFORMANCE PERIOD COVERED: _____________________________________

NAME AND TITLE OF COR: _______________________________

NAME AND TITLE OF GTL: ________________________________

DATE: __________________________

EVALUATION RATINGS FOR ASSESSMENT

EXCEPTIONAL

VERY GOOD

SATISFACTORY

MARGINAL

UNSATISFACTORY

All value ratings must be supported, objective and explained in the Narrative Section for each Performance Objective.

PERFORMANCE OBECTIVES:

A. TECHNICAL/QUALITY OF PRODUCT OR SERVICE: Rating: <Value>

How well does the contractor meet your Technical Requirement IAW the performance metrics in the PWS?

NARRATIVE: (enter narrative in box)

B. PROJECT MILESTONES AND SCHEDULE: Rating: <Value>

How well does the contractor meet the established schedule IAW the performance metrics in the PWS?

NARRATIVE: (enter narrative in box)

C. INVOICING & STAFFING: Rating: <Value>

Are the staffing levels and expertise appropriate for accomplishing the mission IAW the performance metrics in the PWS?

Were the invoices current, accurate and complete?

D. MANAGEMENT: Rating: <Value>

How well did the contractor integrate/coordinate all activities needed to execute the contract IAW the performance metrics in the PWS?

Appendix C

CONTRACT DISCREPANCY REPORT

1. Contract Number: <insert number>

2. To: <insert name of Cerner’s Engagement Leader>

3. From: <insert name of GTL>

4. Date and Time Discrepancy was Observed:

5. Discrepancy or Problem: <Describe in detail. Identify any attachments.>

6. Corrective Action Plan:

A written corrective action plan <is/is not> required.

<If a written corrective action plan is required include the following.>

The written Corrective Action Plan will be provided to the undersigned no later than <#> days after receipt of this CDR.

Prepared By: <Enter GTL’s name>

Government Task Leader

Concur:

Contracting Officer’s Representative

Received By:

Engagement Leader image1.jpeg