Enclosure 1 - Quality Assurance Surveillance Plan.pdf

NASA CONSOLIDATED APPLICATIONS AND PLATFORM SERVICES (NCAPS)

Quality Assurance Surveillance Plan (QASP)

CONTRACT: 80TECH23R0002

CONTRACTOR: TBD

EFFECTIVE DATE: TBD

APPROVAL:

EBMO, IT Acquisition Management Branch, IT Contracts Lead

Enclosure 1

80TECH23R0002

I. INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) is a Government-developed document used to determine if the Contractor’s performance meets the performance standards contained in the contract, including any Task Order(s). The QASP establishes procedures on how this assessment/inspection process will be conducted and provides a detailed, continuous oversight process, including:

• What will be monitored,

• How monitoring will take place,

• Who will conduct the monitoring, and

• How monitoring efforts and results will be documented.

The Contractor is responsible for implementing and delivering performance that meets contract standards using Attachment B – Data Requirement Descriptions, and Attachment – M, Service Delivery Standards and Metrics. The QASP provides the structure for the Government’s surveillance of the Contractor’s performance to ensure that contract standards are met. It is the Government’s responsibility to be objective, fair, and consistent in evaluating Contractor performance.

The QASP is incorporated by reference into this contract; however, it is not intended to duplicate the Contractor’s plan to meet performance metrics as outlined in Attachment M, Service Delivery Standards and Metrics and DRD MA-001, Contract Management Plan. This QASP is a living document and is the Government’s plan for ensuring conformance to the contract’s minimum requirements. Flexibility in the QASP is required to allow for an increase or decrease in the level of surveillance necessary based on Contractor performance.

The Government may provide a copy of the QASP to the Contractor to facilitate open communication. In addition, the QASP recognizes that unforeseen circumstances outside the Contractor’s control may occur. The QASP will ensure early identification and resolution of performance issues to minimize the impact on service performance. Timely communication is critical to delivering quality services, and the monthly Performance Management Review (PMR) Monthly Performance Management Review Report (DRD MA-015) is the cornerstone for open and constructive communication between the Government and Contractor to ensure excellent performance.

The Contractor, and not the Government, is responsible for management and quality control actions to meet the terms of the contract. The role of the Government is quality assurance to ensure contract standards are achieved.

The NASA Consolidated Applications and Platforms Services (NCAPS) DRD MA-001, Contract Management Plan is the driver for defining a quality assurance approach. The Contractor is required to develop a comprehensive program of inspections and monitoring actions. The first major step to ensuring a “self-correcting” contract is ensuring that the Contractor’s quality assurance program provides the measures needed for success.

Once the Contractor’s quality assurance approach is approved as part of DRD MA-001, Contract Management Plan careful application of the process and standards presented in this document will ensure a robust quality assurance program.

Authority for issuance of this QASP is provided under Part 46 of the Federal Acquisition Regulation (FAR), Inspection of Services clauses, which provides for the inspection, acceptance, and documentation of the service called for in the contract or order. This acceptance is to be executed by the Contracting Officer (CO) or a duly authorized representative.

II. GOVERNMENT REPRESENTATIVES

The following personnel shall oversee and coordinate surveillance activities for the NCAPS contract, including all Task Orders:

1. The Contracting Officer (CO) is responsible for:

a. Ensuring performance of all necessary actions for effective contracting.

b. Ensuring compliance with the contract terms.

c. Safeguarding the interests of the Government in the contractual relationship.

d. Ensuring that the Contractor receives impartial, fair, and equitable treatment under this contract.

e. Determining the final assessment of the Contractor’s performance in the

Contractor Performance Assessment Reporting System (CPARS) and Evaluation Plan.

(1) As part of CPARS, in accordance with FAR 42.15, the Government will rate the Contractor’s performance annually on the following areas:

(a) Technical/Quality of Products or Services

(b) Cost Control (if applicable)

(c) Schedule and Timeliness

(d) Management or Business Relations

(e) Small Business Subcontracting

(f) Regulatory Compliance

(g) Other

2. The Contracting Officer’s Representative (COR) is delegated limited authority from the CO and is responsible for:

a. Providing continuous oversight of the Contractor’s performance to verify that performance standards contained in the contract are met.

(1) The COR uses the QASP to conduct the oversight/surveillance process, with assistance from the Contract Resource Analyst (CRA), Technical Area Monitors (TAMs), Technical POCs (TPOCs), and Task Monitors (TMs), as well, as other Government stakeholders.

b. Keeping a Quality Assurance file that accurately documents the Contractor’s contract performance.

c. Providing overview training for all TAMs, TPOCs, and TMs.

d. Early identification of performance problems and reporting them to the CO.

e. In accordance with Attachment M, Service Delivery Standards and Metrics, the

COR is authorized and may grant exceptions/exclusions to customer requests/incident tickets for specific Service Delivery Standard and Metric (SDSM) calculations.

Assistance may also be required from CRA, TAMs, TPOCs, and TMs in reviewing and approving/disapproving in advance any SLA exclusions that may be proposed by the Contractor.

f. The COR, with input from the CRA, TAMs, TPOCs, and TMs, is required to provide an annual performance assessment to the CO. This assessment will be used in documenting past performance in the CPARS.

g. The COR is not authorized to make any contractual changes to the contract;

however, he or she may make contract change recommendations to Service Line management for review and final approval by the CO.

3. TAMs, TPOCs, and TMs share a common list of contract responsibilities, with a few key differences.

TAMs are nominated by the Centers’ Chief Information Officers (CIOs), Service Line Directors and/or by the COR. The TAMs are notified by the COR as to their duties and responsibilities for the contract they are responsible for. Also, TAM appointments are not re-delegable.

TAMs, TPOCs, and TMs responsibilities include:

a. Monitoring, evaluating, and assessing the Contractor’s technical performance at their Centers in accordance with Attachment A, Performance Work Statement or at the Work Package Level.

b. Attending a required TAM, TPOC, TM Orientation Workshop conducted by the COR prior to appointment. Refresher workshops may also be required.

c. Soliciting input from the Subject Matter Experts (SMEs), CIOs, and other stakeholders.

d. Participating in monthly or quarterly Performance Management Review

(PMR).

e. Verifying and investigating all stakeholder/end-user complaints forwarded to the TAM, TPOC, TM by the COR.

(1) Upon receipt of the stakeholder/end-user’s written complaint, the TAM, TPOC, TM will investigate the complaint and report any recommended resolutions to the COR/CO for timely disposition.

f. Preparing and submitting a quarterly Quality Assessment form to the COR.

(1) This assessment will be considered and incorporated as applicable in the preparation of a single comprehensive performance assessment report by the COR and used in support of the CPARS.

g. Evaluating and recommending changes to the methods used to evaluate Contractor performance.

h. Identifying and securing funding, monitoring technical performance, and concurring on the monthly invoices for approval and payment by the CO.

i. Quality assessment data provided to the COR from the TAM, TPOC, and TM shall also include all Center Work Packages, sometimes referenced as Task Orders.

j. TAM, TPOC, and TMs shall be impartial and treat the Contractor with professionalism and fairness.

k. All disagreements between the TAM, TPOC, TM and the Contractor should be immediately brought to the attention of the CO and COR.

4. The Performance Evaluation Board (PEB) is made up of the OCIO IT Management Board (ITMB) and is responsible to meet semi-annually to review the evaluation and recommendations of the Performance Evaluation Committee (PEC) as to the Contractor’s performance in support of each annual option period.

III. CONTRACTOR REPRESENTATIVES

1. The Contractor Program Manager is responsible for:

a. Serving as the primary point of contact to the Government and shall serve as the responsible party for the management and overall quality control for contract performance in accordance with the SLAs and other terms and conditions in this contract.

b. Presenting a regular status to Service Line and Center CIO management on contract performance and any other programmatic or operational issues.

c. Identifying employees at each service location as defined in the PWS to serve as the Contractor’s points of contact to interface with TAMs, TPOCs, and TMs for all services identified in the Performance Work Statement (PWS).

IV. CONTRACT SURVEILLANCE

The goal of the QASP is to ensure that Contractor performance is effectively monitored and documented. The Government team will be professional and will work to maintain a non-adversarial relationship with the Contractor, which will enable positive, open, and timely communications. The foundation of this relationship is built upon objective, fair, and consistent evaluations of Contractor performance against contract requirements. The COR or designee uses the methods contained in this QASP to ensure that the Contractor is compliant with contract requirements, including the base contract and all task orders. For formal past performance reporting purposes (e.g., CPARS), performance under individual task orders will be included in the overall surveillance of the base contract and not assessed and reported separately, unless specifically requested by the Service Line Director, Center CIOs and approved by the CO. The COR/CO is responsible for a wide range of surveillance requirements that effectively measure objective and subjective data used to evaluate the Contractor’s overall performance. As stated in Section I above, this QASP is based on the premise that the Contractor, not the Government, is responsible for management and quality assurance to successfully meet the terms of the contract.

V. METHODS OF GOVERNMENT SURVEILLANCE

The Government will use various surveillance methods to surveil and report contract performance.

Regardless of the surveillance method, the CO will always keep the Contractor informed of all performance issues so that the Contractor can resolve all issues in a timely manner. The COR, in conjunction with the Service Line Leads and CRMs, shall be responsible for monitoring the Contractor’s performance in meeting a specific performance standard or SDSM. All objective and subjective surveillance results may be used as input into the annual CPARS.

1. Surveillance Data:

a. Surveillance shall be conducted against the objective performance standards identified in Attachment M, Service Delivery Standards and Metrics, and through subjective evaluation methods.

b. NASA’s Information Technology Service Management (ITSM) system shall serve as the official data source for the purpose of surveillance and reporting performance data.

c. These performance standards define the desired outcomes for each service.

Performance standards define the level of service required under the contract to successfully meet the performance objectives. In accordance with Attachment M, Service Delivery Standards and Metrics, proposed changes to the SLAs may be made by the Contractor or Government at any time during contract performance. Specifically, the SLAs will be reviewed annually as part of the PMRs.

d. Any approved changes to the SDSMs shall be implemented via a bilateral contract modification by the CO (Changes to Attachment M, Service Delivery Standards and Metrics.) Based on a recommendation and concurrence from the Service Line Manager the CO has the authority to waive SLA tickets.

2. Quality Assessment Forms:

a. The TAMs are nominated by the Centers’ Chief Information Officers (CIOs), Service Line Directors and/or by the Contracting Officer’s Representative (COR). The TAMs will complete the Government’s Quality Assessment forms.

b. The forms will document the Government’s assessment of the Contractor’s performance under the contract.

c. The Quality Assessment form will align with the performance areas identified in the CPARS for consistent performance assessment by the Government.

d. The CO and COR will retain a copy of all completed Quality Assessment forms.

3. Objective Performance Methods:

a. Random Monitoring/Data Sampling—Periodically performed by the COR or designee.

b. Review of Data Requirements Documents (DRDs)—As required, the COR (or designee) will review the contract deliverables required by Attachment B, Data Requirement Descriptions, and Attachment M, Service Delivery Standards and Metrics, for quality and timeliness in accordance with the terms and conditions.

c. Periodic Assessments—The COR and/or TAMs, TPOCs, and TMs may perform periodic assessments. Any significant findings will be shared with the Contractor as part of the Performance Management Review PMR.

d. Monthly Reports—Monthly performance reports and/or DRDs are reviewed by the assigned owners and the COR for contract compliance for quality and schedule.

e. Management Information Systems—The Service Line will utilize relevant data from Government systems. Some data may be pulled from contractor-provided systems.

4. Subjective Performance Methods:

a. End-user Survey—The Enterprise Service Desk (ESD) will send customer satisfaction surveys for the following items:

(1) Completed service requests (with minor exceptions). Exceptions occur when the Service Line determines that a specific service should not be surveyed on completion.

(2) All incident tickets marked “Resolved,” with the following COR-approved exceptions:

(a) Status inquiries

(b) Tickets referred to Center or another service desk

(c) Tickets marked as duplicates

(d) Cancelled by the customer or ESD

(e) Service Line/Contractor internal incident tickets

(f) Service Line/Contractor internal service requests (RITMs)

(g) Approved vendor constraints

b. The COR will send customer satisfaction surveys on a monthly or quarterly frequency as needed for the Service Line.

c. The Contractor shall comply with the customer satisfaction performance metrics defined in Attachment M, Service Delivery Standards and Metrics.

(1) The Contractor shall use self-service reporting in the ESD’s ITSM platform to view responses and address missed SLAs as required.

d. Direct Observation—TAMS, TPOCs, and TMs will perform and record their direct observations and report them to the COR.

e. End-user Complaints—To be considered a valid complaint, the end-user will, in writing, clearly detail the nature of the complaint and forward this information to the applicable TAMs, TPOCs, and TMs within 10 business days of the incident. Once an assessment has been made and the TAMs, TPOCs, and TMs determines the complaint to be valid, the complaint will be reported to the Contractor’s Center Operations Manager with a cc to the COR/DCOR for investigation and proposed resolution. If an agreed-upon resolution between the TAMs, TPOCs, TMs and the Center Operations Manager cannot be reached, the complaint will be escalated to the COR. The COR will then work with the Contractor’s Program Management Office and applicable Service Line Leads for final resolution.

f. Progress Reports, Meetings, Other—The COR may additionally record observations from reports, meetings, and/or other sources to establish a comprehensive and fair assessment of the Contractor’s performance.

5. Customer Feedback:

The Contractor is expected to establish and maintain professional communication between its employees and customers.

The primary objective of this communication is creating a positive impact on the end-user experience. The Government qualifies end-user experience as the sum of a combination of factors that the Government analyzes, including End-user Perceptions, Mission and Center Outcomes, and End-user Preferences. End-user experience is the most significant external indicator of the success and effectiveness of all services provided and can be measured through ESD customer satisfaction surveys, end-user complaints, personas, journey maps, and Voice of the Customer (VoC) analyses.

6. The Performance Management Feedback Loop:

a. The performance management feedback loop begins with the communication of the NCAPS contract requirements. The PMRs provide a formal and periodic opportunity to facilitate open and timely communications regarding performance. The performance standards are given in Attachment M, Service Delivery Standards and Metrics, and are assessed using the methods of surveillance provided herein.

b. The objective quality standards will be measured in accordance with the performance standards identified Attachment M, Service Delivery Standards and Metrics.

VI. ANALYSIS OF QUALITY ASSURANCE ASSESSMENT

1. Determining Performance:

a. The SLA table Attachment M, Service Delivery Standards and Metrics is structured to allow the Contractor to manage how the work is performed provided that the desired performance standards are still met.

b. The Government shall use the monitoring methods cited in Section V – Methods of Government Surveillance to determine whether the performance standards/service levels have been successfully met. If the Contractor has not met the minimum requirements, the Contractor may be asked to develop a remediation plan at no additional cost to show how and by what date it intends to bring performance up to the required levels. All available information related to the performance of this contract may be used by the Government for performance assessment and determination.

2. Reporting:

a. At the end of the midterm period (6-month period), the COR will prepare a written report summarizing the overall results of the quality assurance surveillance of the Contractor’s performance for the previous 6 months. This written report will become part of the consolidated performance documentation and will include input from TAMs, TPOCs, and TMs. The consolidated input will enable the Government to assess whether the Contractor is meeting the stated objectives and/or performance standards under this contract.

3. Reviews and Resolution:

a. Senior Management Review—The COR may require the Contractor’s Program

Manager (designated or alternate) to meet with the COR and other necessary Government personnel to discuss performance issues. The COR will schedule these reviews with the Contractor as needed; however, if the need arises, the Contractor may meet with the COR and Service Line management as often as required or per the Contractor’s request. This review may cover the topics below, or other topics as requested by the Service Line:

(1) Projected outlook for upcoming months and progress against expected trends, including a corrective action plan analysis

(2) Appropriateness of self-assessments

(3) Recommendations for improved efficiency and/or effectiveness

(4) Remediation/corrective action plans

(5) Performance issues and/or skill mix/staffing concerns on LOE Task

Orders

(6) Transformation and innovation

(7) Invoice accuracy and funding

(8) Contract/business management issues

(9) Issues and concerns from both parties

b. Daily Meetings—As required, daily meetings between the Service Line Leads and TAMs will be scheduled to address performance issues and resolutions.

c. Monthly Performance Management Reviews—A PMR will be conducted monthly by the Contractor’s Program Manager and presented to the Service Line and Center CIO management teams. This review will highlight performance data and discuss trend analyses and any other issues requested by Service Line and Center CIO management in accordance with Attachment M, Service Delivery Standards and Metrics. The COR will schedule the monthly PMR and publish the schedule. This review will include, but is not limited to, all sections of the PWS and other topics identified by the Government or Contractor (PWS)

d. Semiannual (Midterm) Review—The 6-month PMR shall be conducted by the Contractor’s Program Manager and shall include a summary of the Contractor’s overall performance for the previous 6 months. The Service Line Director, Center CIO management or COR will highlight any areas for improvement to ensure any deficient performance objectives can be improved over the next 6 months, with the overall goal for the Contractor to earn an “Exceptional” performance rating (see Section VII for full definition).

(1) Areas of Special Emphasis (ASE) may also be addressed, if applicable, for upcoming next 6 months.

e. Annual Review—The 12th monthly PMR will be conducted by the Contractor’s Program Manager and shall include a summary of the Contractor’s overall performance for the previous 12 months and summarize the items listed under the monthly PMRs. Proposed changes to the SLAs can also be discussed/reviewed at this time.

f. In coordination with Service Line and Center CIO management, the CO/COR must coordinate and communicate with the Contractor to resolve issues and concerns regarding marginal or unacceptable performance.

VII. PERFORMANCE RATING DEFINITIONS

The performance ratings given in the table below may be used to evaluate the quality of the Contractor’s performance.

The following table can be used to align NCAPS adjectival ratings with to the adjectival ratings employed by CPARS for past performance ratings.

CROSS-WALK BETWEEN FAR/NFS PRESCRIBED RATINGS AND CPARS

CPARS

91%-100% EXCELLENT Exceptional Performance meets contractual requirements and exceeds many to the Government's benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective.

76%-90% VERY GOOD Very Good Performance meets contractual requirements and exceeds some to the Government's benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the Contractor were effective.

Note To justify a Very Good rating, identify a significant event and state how it was a benefit to the Government. There should have been no significant weaknesses identified.

51%-75% GOOD Satisfactory Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the Contractor appear, or were, satisfactory.

Note To justify a Satisfactory rating, there should have been only minor problems, or major problems that the Contractor recovered from without impact to the contract or order. There should have been NO significant weaknesses identified. A fundamental principle of assigning ratings is that Contractors will not be evaluated with a rating lower than Satisfactory solely for not performing beyond the requirements of the contract or order.

50%* SATISFACTORY Marginal Performance does not meet some contractual requirements. The contractual performance of the element or sub-element being evaluated reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor's proposed actions appear only marginally effective, or were not fully implemented.

Note To justify Marginal performance, identify a significant event in each category that the Contractor had trouble overcoming, and state how it impacted the Government. A Marginal rating should be supported by referencing the management tool that notified the Contractor of the contractual deficiency (e.g., management, quality, safety, or environmental deficiency report or letter).

49% - 0% UNSATISFACTORY Unsatisfactory Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which the Contractor's corrective actions appear or were ineffective.

Note To justify an Unsatisfactory rating, identify multiple significant events in each category that the Contractor had trouble overcoming, and state how these events impacted the Government. A singular problem, however, could be of such serious magnitude that it alone constitutes an Unsatisfactory rating. An Unsatisfactory rating should be supported by referencing the management tools used to notify the Contractor of the contractual deficiencies (e.g., management, quality, FAR/NFS

91%-100% EXCELLENT

Contractor has exceeded almost all of the significant award-fee criteria and has met overall cost, schedule, and technical performance requirements of the contract in the aggregate as defined and measured against the criteria in the award-fee plan for the award-fee evaluation period.

76%-90% VERY GOOD Contractor has exceeded many of the significant award-fee criteria and has met overall cost, schedule, and technical performance requirements of the contract in the aggregate as defined and measured against the criteria in the award-fee plan for the award-fee evaluation period.

51%-75% GOOD Contractor has exceeded some of the significant award-fee criteria and has met overall cost, schedule, and technical performance requirements of the contract in the aggregate as defined and measured against the criteria in the award-fee plan for the award-fee evaluation period.

50%* SATISFACTORY Contractor has met overall cost, schedule, and technical performance requirements of the contract in the aggregate as defined and measured against the criteria in the award-fee plan for the award-fee evaluation period.

<50% UNSATISFACTORY Contractor has failed to meet overall cost, schedule, and technical performance requirements of the contract in the aggregate as defined and measured against the criteria in the award-fee plan for the award-fee evaluation period.

FAR

*NFS allows only 50% for a Satisfactory Rating