Attachment 3 - QASP Example.pdf

Quality Assurance Surveillance Plan (QASP)

The objective of the QASP is t o use objective performance measures to drive continuous improvement in the support o f business and user-related goals. The primary goal of the QASP is to provide the team with insight as to the difference between expected and actual outcomes of the completed work. Ideally, both the measures and their associated performance level evolve over time as the team’s understanding of the underlying needs improve and challenges are overcome.

Measurement Definition Acceptable

Quality Level Actual

Measurement Reporting Frequency

Method for Surveillance

General Resource Availability Between 6am and 9pm, Monday - Friday. Occasional 24hr support as requested.

Non-Production Support (Dev & Test Environments)

Between 6am and 9pm, Monday - Friday. Occasional 24hr support as requested.

On Call Production Support (Imp & Prod Environments)

24 hours per day, 7 days per week support. During federal holidays and non-production hours, on-call support shall be provided with an acknowledgement, escalation, and resolution plan. Live support should be available to respond to CMS by priority for Critical (P1), Moderate (P2), and Low (P3) service requests.

P1 response times shall be 15 minutes.

P2 response times shall be 15 minutes.

P3 response times shall be 1 day.

Critical/Urgent Communication Requests

Response within 15 minutes and resolution within 1 hour for a P1 and 4 hours for a P2.

Critical Level (P1) Software Defects or Issues

Defects or issues are fixed within 1 hour of identification.

Moderate Level (P2) Software Defects or Issues

Defects or issues are fixed within 4 hours of identification.

Low (P3) Software Defects or Issues

Defects are evaluated and prioritized within 4 days of identification.

General Communication Requests

Response within 8 business hours and resolution within 5 business days.

Contract Deliverables Deliver all contractual deliverables as specified in the delivery schedule.

Critical Security Software and OS Patches

Deployed within 24 hours of release, unless otherwise directed by CMS.

General Server OS Patches Deployed within 48 hours of release, unless otherwise

Server OS Upgrades Completed within 24 months of release, unless otherwise

User Management Requests Respond to requests within 4 hours and complete within 8 hours.

General Security Corrections Correct any security deficiencies, conditions, weaknesses, findings, or gaps identified by all audits, reviews, evaluations, tests and assessments, including but not limited to, the Statement on Auditing Standards (SAS) – 70 Reviews, Inspector General Audits, self-assessments such as the annual security controls testing as reported under FISMA, and Vulnerability Assessments within 30

Corrective Action Plans Provide Corrective Action Plans for all identified weaknesses, findings, gaps, or other deficiencies in accordance with the Business Partner System Security Manual (BPSSM) within 15 business days.

Audit/Evaluation Corrections Correct weaknesses, findings, gaps, or other deficiencies as follows:

General- 90 calendar days of receipt of the final audit or evaluation report.

Critical- 15 calendar days of receipt of the final audit or evaluation report.

Corrective Action Validations Validate and document that corrective actions are implemented, tested, and effective within 15 business days. The Contractor shall also provide attestation and documentation of corrective actions to CMS within 15

Cost Estimates Provide cost estimates within 5 business days, unless otherwise directed by CMS.

Number of User Stories Accepted

User acceptance of capabilities for each user story means that the team is fulfilling requirements and responding to change. This means that Agile projects create high-quality products, well aligned with stakeholder expectations.

Test Case Status Status of test cases by due dates as per project timelines.

Defects Discovered in Production

Number of defects discovered after a release to a production environment. The metric indicates whether the testing team is successfully capturing defects before they go into Production.

Error Discovery Rate Number of total defects found/Number of test procedures executed.

The error discovery rate is used to analyze and support a rational product release decision.

Error Discovery Rate Trend Measures change in Error Discovery Rate over time.

Goal is to identify defects sooner rather than later.

Defect Fix Retest The date a defect was fixed and released in a new build versus the date the defect was retested. The defect fix retest metric indicates whether the testing team is retesting the fixes fast enough, in order to get an accurate progress metric.

Automation Index (%) Number of tests that have been automated as a percentage of all test cases. Percent automated can be defined as the percentage of a set of given test cases that is automated.

ATC No. of test cases automated

AI (%) = ------------- = (------------------------------------------)

TC Total no. of test cases

AI = Automation Index (%) ATC = Number of test cases automated TC = Total number of test cases

Defect Density Used to determine areas to automate. A measure of the total known defects divided by the size of the software entity being measured.

D No. of known defects

DD = ----------------- = (-----------------------------------)

SS Size of software entity

DD = Defect Density D = Number of known defects SS = Size of software entity

Attrition Rate Rate as a % of employees who leave the contract to those who started on the contract per month.

Raw Incident Count Rate of incidents on a weekly basis.

Mean Time to Resolution Average time required for the Contractor to resolve incidents.

Time to Acknowledgement/Time to Response

Average time to acknowledge and respond to an incident.

Application Performance Index

(APDEX)

(Satisfied Count + Tolerating Count/2) / Total Samples

508 Leaderboard Score Meets > 96% score on HHS Leaderboard

Lead Time Measurement of the time taken from when code starts development to successful production deployment.

Deployment Frequency The rate in which code deployed (Based on successful DevOps program this should trend up or remain on a consistence schedule).

Velocity Velocity is measured by the sum of story points for each story completed during an Iteration (sprint(s)).

Percentage of Failed Deployment

The percentage of deployments that have resulted in a negative user impact or caused an outages.

Mean Time to Recovery (MTTR) Measurement of the time taken to restore a service after a deployment including detection, diagnosis and recover phases. CMS will use MTTR to measure the Contractors effectiveness in handling changes and how well the DevOps program is established.

Customer Satisfaction Measured by the number of tickets generated by customers and users.

Percentage of Change in User Volume

Number of new users, interacting with the service and generating traffic.

Availability The percentage of overall uptime of the product and service (this percentage shall remain in alignment of

SLA).

Performance (Response Time) Measurement of product or service response time (the response time of the product or service shall operating within SLA thresholds). This metric should remain unwavering regardless of % change in user volume or deployment activity.

Websites Uptimes Maintain websites up 99.99% of the time. During unexpected periods of downtime, provide a Root Cause Analysis within 1 day of resolution for a P1 and 4 days for a P2.

Page Load Times All webpages shall load in under 3 seconds measured at the 95th and 99th percentile via APM and synthetic test monitoring based on a 30 minute average regardless of geo-region. Measurements shall be recorded at 8:00AM everyday using the last 24 hours of data.

Application Performance Monitoring SLA

Application Performance Monitoring: This set of metrics will be base lined in accordance with 99.99% uptime, availability, and page load times not exceeding 3 seconds.

Error Rate: should not exceed 1% or the established baseline.

Throughput: expectations will be based on anticipated site traffic, load testing, and CMS Program Management expectations.

Browser Metric SLA Page Load: times should not exceed 3 seconds and error rate should not exceed 1% or the established baseline.

Synthetic Testing SLA Uptime: 99.99%

Server Metric SLA Server Metrics: shall be remediated if they sustain or exceed established baseline.