Attachment_4_-_QASP_Template_Revised_06132017.doc

ATTACHMENT 4

Internal Revenue Service

Managed Print Services (MPS)

Quality Assurance Surveillance Plan (QASP) Template

Version 1.0

June 12, 2017

Review and Approval This Quality Assurance Surveillance Plan (QASP) documents how the Treasury will monitor Service Level Objective (SLO) performance under selected Managed Services contracts. The Treasury MPS Program Management Office (PMO) has established a multi-tiered framework to support MPS contract performance surveillance. As part of this framework, Treasury PMO Quality Management (QM) will utilize Contractor-supplied data to monitor performance periodically and will provide the results of this surveillance to the Contractor. The Treasury MPS PMO will augment this primary audit with in-depth reviews (audits), as necessary.

Authority for issuance of this QASP is the Federal Acquisition Regulation (FAR) Subparts 37.6, 46.1, and 46.4, as well as contract Section E – Inspection and Acceptance, which provides for inspections and acceptance of the work and services called for in service contracts, to be executed by the Contracting Officer (CO) or a duly authorized representative.

Review of this QASP signifies formal notification of the periodic surveillance audit to monitor MPS Contractor technical performance. If an in-depth review (audit) is necessary, the Treasury MPS PMO will issue separate formal notification to the Contractor of its intent to perform the review, no later than 30 calendar days before it is scheduled to begin.

Review and Approval:

The following stakeholders have approved this MPS Quality Assurance Surveillance Plan, v1.0:

XXX, MPS Director Date

XXX, MPS Quality Management Date

MPS Contracting Officer Date

MPS Contracting Officer Technical Representative Date

The following stakeholders have reviewed and concur with this MPS Quality Assurance Surveillance Plan:

Contractor Program Manager Date

Contractor Quality Control Date Table of Contents iReview and Approval iiTable of Contents iiiTable of Tables iiiTable of Figures ivRecord of Changes vNotes for Using the QASP Template

11.

Introduction

11.1 Scope

11.2 Authority

11.3 Goals and Objectives

21.4 QASP Update

32.

Roles and Responsibilities

32.1 Treasury

42.2 Contractor

52.3 The MITRE Corporation

63.

Approach

63.1 Overall Approach

73.2 Surveillance Methods

83.3 Evaluation Methodology

114.

Surveillance Activities Table

135.

Reporting Flow

156.

Failure to Perform

16Appendix A: SLO Specifications

18Appendix B: Forms

19Appendix C: References

20Appendix D: Acronyms

Table of Tables 3Table 2-1 Government Roles and Responsibilities

4Table 2-2 Contractor Roles and Responsibilities

5Table 2-3 MITRE Roles and Responsibilities

11Table 4-1 Surveillance Activities Table

Table of Figures 6Figure 3-1 QASP Implementation Approach

9Figure 3-2 Example Quality Evaluation

12Figure 5-1 Surveillance Reporting Flow

Record of Changes

(Table, Figure, or Paragraph) A=Add

M=Modify

D=Delete Brief Description of Change

Notes for Using the QASP Template

NOTE 1:

This Quality Assurance Surveillance Plan (QASP) template may be used for any Managed Services contract. <Bracketed, green highlighted> text indicates areas where additional information is needed.

1. Introduction This Quality Assurance Surveillance Plan (QASP) documents the approach the Treasury MPS Program Management Office (PMO) will take to monitor Contractor technical performance required by the Performance Work Statement (PWS) in effect at the time of the surveillance activity.

1.1 Scope

This QASP documents how the Treasury MPS PMO will monitor Service Level Objective (SLO) performance under selected Managed Services contracts. The QASP will be tailored to the characteristics and requirements of the particular contract by the addition of the germane Surveillance Activity Table in Section 4 of this document. SLOs not contained in this document will not be considered to be within scope of the QASP implementation.

The QASP does not define or dictate how the Contractor should accomplish the work required under contract provisions; instead, the QASP recognizes that the Contractor is responsible for the management and quality control (QC) actions necessary to meet the performance standards documented herein, and assumes the Contractor has in place QC programs to ensure the performance standards are met. Thus, the continuous performance monitoring performed by the Contractor is not part of the QASP implementation activity, but does yield data that are leveraged during QASP activities.

Some of the data developed during QASP activities may be used in other Treasury MPS PMO measurement activities. The descriptions of those other measurement activities are contained in documents external to this QASP.

1.2 Authority

Authority for issuance of this QASP is the Federal Acquisition Regulation (FAR) Subparts 37.6, 46.1, and 46.4, as well as contract Section E – Inspection and Acceptance, which provides for inspections and acceptance of the work and services called for in service contracts, to be executed by the Contracting Officer (CO) or a duly authorized representative.

1.3 Goals and Objectives

The QASP activity is a critical component of the Managed Print Services (MPS) that is documented in the Performance Monitoring Management Plan (PMMP). This QASP will be implemented by the MPS Quality Management (QM) function.

1.3.1 Goal

The overall goal of the QASP activity is to confirm that the MPS Contractor’s delivery and execution of products or services meets the agreed technical performance measures, or SLOs, included in the contract and documented in this QASP.

1.3.2 Objectives

Specific QASP objectives include:

· Determine Contractor performance in meeting agreed technical SLOs in the contract to which applied. This may include audits, examinations, and measurements gathered from reports.

· Identify MPS risks arising from Contractor performance in meeting agreed technical SLOs.

· Identify Contractor corrective action(s) needed to meet agreed SLOs or to address identified risks or issues.

1.4 QASP Update

This Plan will be updated periodically to reflect the results of a joint review, by Treasury MPS PMO and Contractor personnel, of SLOs, SLO definitions, and SLO monitoring approaches.

2. Roles and Responsibilities Treasury bears primary responsibility for implementing this QASP, with support from the Contractor and from The MITRE Corporation.

2.1 Treasury

Table 2-1 describes Government roles and responsibilities regarding QASP implementation.

Table 2-1 Government Roles and Responsibilities

· Provides resources to implement QASP

· Supports fee determinations or achievement of other contract provisions

· Analyzes Quality Assurance Surveillance Review (QASR) and similar feedback and findings, as appropriate

· Interviews business customer(s)

· Reviews Contractor-provided reports

· Conducts liaison with Contracting Officer Representative(s) (COR(s))

· Documents findings

· Provides formal written feedback, including findings

· Participates in project status reviews, as necessary

· Performs in-depth quality review activities at Contractor location, as necessary

· Debriefs Contractor (with CO and/or COR) on the surveillance review results, as necessary

· Develops technical input for award term determinations or achievement of other contract provisions, using QASR results, as appropriate

· Reviews QASR and similar feedback and findings, as appropriate

· As appropriate, executes award term determinations and ensures achievement of other contract provisions

· Provides contractual direction

· Debriefs Contractor (with MPS QM and/or COR) on the surveillance review results, as necessary

· Reviews QASR and similar feedback and findings, as appropriate

· Forwards QASR and similar reports to appropriate business customer

· Debriefs Contractor (with CO and/or Treasury MPS QM) on surveillance review results, as necessary

2.2 Contractor

Table 2-2 describes Contractor roles and responsibilities regarding QASP implementation.

Table 2-2 Contractor Roles and Responsibilities

· Collects, analyzes, documents, produces performance data

· Submits data reports to CO/COR

· Supports Treasury MPS QASRs and similar activities, as necessary

· Identifies and implements corrective actions

· Submits requests to COR for new reports or measures

2.3 The MITRE Corporation

Table 2-3 describes The MITRE Corporation’s roles and responsibilities regarding QASP implementation.

Table 2-3 MITRE Roles and Responsibilities

· Assists Treasury in defining and documenting processes and procedures for conducting audits, monitoring performance measures, documenting findings, and assessing fee determinations or achievement of other contract provisions

3. Approach

The Treasury MPS PMO has established a multi-tiered framework to facilitate quality assurance (QA) monitoring of MPS contractor performance. Treasury MPS QM will leverage Contractor-supplied data to implement this framework. The following sections describe this strategy in more detail.

3.1 Overall Approach

The Treasury MPS QA monitoring framework described in this QASP integrates and leverages ongoing monitoring activities, as depicted in Figure 3-1. Treasury MPS QM performs two types of audits, QA surveillance reviews on a regular basis and in-depth reviews (“deep dives”) on an exception basis, when implementing the QA monitoring framework.

Figure 3-1 QASP Implementation Approach Treasury MPS QM’s basic approach is to: establish an agreed and understood quality baseline; monitor Contractor performance against that baseline; identify any performance gaps on a timely basis; and, ensure corrective actions are applied timely.

· Treasury MPS QM staff and other interested MPS stakeholders meet periodically with Contractor program personnel to review and agree to the SLOs to be included in the QASP, reach a common definition of the SLOs, and review the monitoring and reporting systems to ensure continuous and accurate intake, analysis and reporting of performance data. This QA Program Baseline Review (QA PBR) establishes the quality baseline against which Contractor performance is assessed. This QASP is updated based on the QA PBR results, and changes to service levels, procedures, and metrics are incorporated in a contract modification by the CO.

· The Contractor implements continuous performance monitoring as agreed during the QA PBR, uses internal QC resources to ensure SLO performance thresholds defined by the Acceptable Quality Levels (AQLs) are met, and submits deliverables periodically to report status against the quality baseline. Although these monitoring activities themselves are not included in the scope of the Treasury MPS QA surveillance, they yield measurement data and deliverables that are assessed during QASP implementation.

· The COR holds Project Status Reviews (PSRs) with other relevant MPS stakeholders, including MPS QM and internal Treasury MPS SLO “owners”, to assess SLO status against the established baseline, identify emerging risks and issues, and determine potential “deep dive” opportunities.

· The MPS QM function audits Contractor performance by, at a minimum, checking and synthesizing reported SLO data to develop quality surveillance feedback, including findings and potential corrective action requests, for submission to the CO/COR. These audit results are submitted using the form in Appendix B.

· In addition, based on the results of the surveillance monitoring and per FAR Subpart 52.246-4, the Treasury MPS PMO reserves the right to perform in-depth reviews of Contractor performance (“deep dives”) as necessary to assess perceived or actual performance issues. These in-depth reviews may occur at any time during the evaluation period and may address performance issues arising in the current or any prior evaluation period. If an in-depth review (audit) is necessary, Treasury will issue separate formal notification to the Contractor of its intent to perform the review, no later than 30 calendar days before it is scheduled to begin. To support such deep dives, the Treasury may request additional data from the Contractor, including but not limited to raw data, audit results, evidence of certification, QC program data, etc. Treasury reserves the right to perform these deep dives at Contractor or subcontractor facilities.

3.2 Surveillance Methods

MPS QM has at its disposal several methods with which to perform the QA monitoring function described above:

· Contractor Self-Reporting. Uses contractor-supplied performance data collected by tracking and monitoring tools and/or contractor status reports that identify problems, issues and/or risks. (See Appendix C, Reference number 3.)

· Trend Analysis. Per the US Census Bureau: “This method should be used regularly and continually to monitor the contractor’s ongoing performance over time. Data for tracking trends can be gathered from all other evaluation sources and methods, [and used to populate] a database upon which the trend analysis is based. This database should be created and maintained by government personnel. The contractor’s own metrics may provide most of the information needed for the analysis.” (See Appendix C, Reference number 1.)

· Quantitative Analysis. This refers to the use of statistical and other quantitative techniques to analyze raw performance data to characterize performance and/or identify trends, causal relationships, or root causes of performance issues. Examples would include analyzing results of customer survey and problem tickets against relevant SLOs. This method is appropriate for in-depth analysis of areas of interest.

· Customer Inputs. Per the Office of Management and Budget (OMB): “Although not usually a primary method, this is a valuable supplement to more systematic methods. For example, in a case where random sampling indicates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations where customers can be relied upon to complain consistently when the quality of performance is poor, e.g., dining facilities, building services, customer surveys and customer complaints may be a primary surveillance method and customer satisfaction an appropriate performance standard. In all cases, complaints should be documented, preferably on a standard form.” (See Appendix C, Reference numbers 2, 3.)

· Third Party Audits or Assessment. Per the US Census Bureau: “Third party audits or assessments refers to contractor evaluation by a third party organization that is independent of the contractor. All documentation supplied to, and produced by, the third party should be made available to the government by the contractor.” (See Appendix C, Reference number 1.)

· Periodic Inspection. Per the Office of Management and Budget: “This method, sometimes called ‘planned sampling,’ consists of the evaluation of tasks selected on other than a 100 percent or random basis. It may be appropriate for tasks that occur infrequently, and where 100 percent inspection is neither required nor practicable. A predetermined plan for inspecting part of the work is established using subjective judgment and analysis of agency resources to decide what work to inspect and how frequently to inspect it.” (See Appendix C, Reference number 2.)

· Random Sampling. Per the Office of Management and Budget: “This is usually the most appropriate method for recurring tasks. With random sampling, services are sampled to determine if the level of performance is acceptable. Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained.” (See Appendix C, Reference numbers 2, 3.)

· 100 Percent Inspection. Per the Office of Management and Budget: “This is usually only the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g., where safety or health is a concern. With this method, performance is inspected/evaluated at each occurrence. One hundred percent inspection is too expensive to be used in most cases.” (See Appendix C, Reference numbers 2, 3.)

3.3 Evaluation Methodology

MPS QM will use the cited observation methods to determine whether AQLs have been met over the evaluation time frame. The evaluation period will be quarterly. The specific evaluation methodology is as follows:

· MPS QM has established a Surveillance Activity Workbook (SAW), maintained within the MPS PMO, to capture quality surveillance data. The workbook lists the SLOs described in Appendix A and provides a weighting for each SLO, for each month during the evaluation period, based on the importance of that SLO to achieving Treasury business needs. The weightings range from 1 to 5, with 5 being the highest (most important). Note that Treasury reserves the right to change the weightings for an evaluation period, to reflect changing priorities or emerging issues. If the weightings are changed, the CO will communicate the changes to the Contractor prior to the start of the evaluation period in accordance with the terms of the contract. Note also that some SLOs might not be applicable for a given month during the quarter, and would be excluded from evaluation.

· For each applicable SLO, Treasury MPS PMO QM will determine whether the associated AQL has been met. Treasury MPS PMO QM will base this determination on SLO data contained in Contractor deliverables, using the specification in Appendix A as the baseline definition of the SLO, as well as any additional data gathered by Government personnel. The Met — Not Met determination will be captured in the SAW. In addition, any quality issues or questions noted for the reporting period will be captured as comments in the SAW.

· At the end of the evaluation period, Treasury MPS PMO QM will synthesize the monthly data to determine the weighted percentage of AQLs achieved. Each “Met” SLO will be scored at its associated weight, and each “Not Met” SLO will be scored at 0. The total score of SLOs achieved will be divided by the total possible score (i.e., the total of the weights applied for the reporting period), and the resulting percentage value will be rounded to the nearest whole number. The result will be reported to the CO using the form in Appendix B, along with observations and findings identified from a review of the comments collected over the evaluation period.

Figure 3.2 provides an example of the application of the evaluation methodology.

· The total score of SLOs achieved is equal to: (3+2+0+4+4+2+2+5+5+2) = 29.

· The total possible score (total of the weights applied) is equal to: (3+2+1+4+4+2+2+5+5+2) = 30.

· The weighted percentage of AQLs achieved is equal to: 29/30 = 96.7%, rounded to 97%

Figure 3-2 Example Quality Evaluation

· Based on the evaluation results, the Treasury MPS PMO may determine that additional review of Contractor performance is warranted. In this case, Treasury MPS PMO QM will use Standard Operating Procedures <TBD> to plan and conduct a “deep dive” analysis. Results of this analysis will be provided to Treasury MPS PMO management, the CO, and the COR before discussion with the Contractor. <NOTE: This may be hard to do if on-site at Contractor facility, unless CO/COTR accompany the team or are available for telecom reviews.>

4. Surveillance Activities Table

Table 4-1 lists Contractor performance standards and identifies the surveillance methods that may be used to monitor performance against those standards. Although only a Primary Monitoring Approach(es) is (are) listed, the Treasury MPS PMO reserves the right to engage, as necessary, in any of the monitoring approaches described in Section 3.2 to obtain full insight into Contractor performance.

NOTE: Table 4-1 will be tailored to the specific contract to which this QASP applies.

Table 4-1 Surveillance Activities Table

Contractor Reporting Frequency

· <document>

· <frequency>

· <document>

· <document>

· <document>

5. Reporting Flow The QASP surveillance activity will be performed periodically as required by contract terms, and the results will be reported using the flow outlined in Figure 5-1 below.

Figure 5-1 Surveillance Reporting Flow

Treasury MPS PMO QM will use the cited observation methods to determine whether AQLs have been met over the evaluation time frame. Should the standards notbe met, the CO will determine the degree to which the results may be translated into specific negative incentives that cause adjustments to Contractor payments.

At the end of the quarterly evaluation period, Treasury MPS PMO QM will use the data contained in the SAW to prepare a written report for the CO/COR, summarizing the overall results of the QA monitoring of the Contractor’s performance over the evaluation period. The report, documented in the format presented in Appendix B, will become part of the QA documentation for the contract.

Treasury MPS PMO QM will submit the report to the MPS PMO, the CO, and the COR no later than two weeks after the end of the evaluation period; the data will be used as the technical input for fee determinations or achievement of other contract provisions, as appropriate. Feedback will be provided to the Contractor in accordance with the terms of the contract. Depending on the assessment contained in the report, the CO/COR may require the Contractor’s project manager, or a designated alternate, to meet with the CO/COR, Treasury MPS PMO QM, and other Government personnel as deemed necessary to discuss the performance evaluation.

Treasury MPS PMO QM, in addition to providing documentation to the CO/COR, will maintain a complete QA file. The file will contains copies of all data analyses, reports, evaluations, recommendations, and any actions related to the Government’s performance of the QA function, including the originals of all the SAWs. All such records will be retained for the life of the contract. Treasury MPS PMO QM shall forward these records to the CO upon completion of the contract.

6. Failure to Perform See the Contract Administration Plan.

Appendix A: SLO Specifications A-0 - Template <SLO Name>

SLO Description:

Acceptable Quality Level:

<AQL>

SLO Reporting Frequency:

<e.g., Monthly>

SLO Calculation:

<Put an equation, if possible. Note whether value will be truncated or rounded, and to how many decimal points>

Base Measures:

Data Verification:

Data Summarization:

Reporting Shift:

Reporting & Retention:

Exceptions:

Data Source:

Data Verification:

Recovery:

· <if the data capture mechanism fails, how will the data be captured? This could be N/A, if there is no redundancy.>

Other Notes:

· <as needed>

History:

· <mm/dd/yyyy>: Baseline description approved by TBD

· <mm/dd/yyyy: other changes, including technology or software changes that affected response time>

Appendix B: Forms

TBD

Appendix C: References

1) US Census Bureau, Attachment 2: Performance-based Statements of Work Guidance for Using a Quality Assurance Plan (accessed on June 24, 2010 at www.census.gov/procur/www/fssp/att02.html)

2) A Guide to Best Practices for Performance-based Services Contracting, Office of Federal Procurement Policy (OFPP) in the Office of Management and Budget (OMB), Final Edition, October 1998

3) United States Department of Agriculture Foreign Agricultural Service, Infrastructure Management Quality Assurance Surveillance Plan, Version V0.4, November 2008 (accessed on June 24, 2010 from the “Seven Steps to Performance-based Acquisition” website at https://www.acquisition.gov/comp/seven_steps/library3.html. See Step 5.)

Appendix D: Acronyms

Quality Assurance Surveillance Plan (QASP) Template ■ Version 1.0

STYLEREF Draft \* MERGEFORMAT

STYLEREF Draft \* MERGEFORMAT

STYLEREF Draft \* MERGEFORMAT June 12, 2017