Att 5 Key Performance Indicators.docx

ATTACHMENT 5: EOHHS KEY PERFORMANCE INDICATORS

The performance, delivery, deployment, testing, proposal efforts, and execution of the Key Performance Indicators (KPIs) below and all associated metrics, Deliverables, Plans and other documentation shall be subject to processes outlined in RFR Section 3.

1 Implementation KPIs The KPIs in this section apply to the initial Go Live of the EHS Comprehensive Assessment System (e-CAS), specifically Phases 1 and 2 as described in RFR Section 3.2.

1.1 Goal: Timely, High Quality Documents

1.1.1 KPI: Produce high quality Deliverables to document e-CAS.

Metrics:

A. Initial Deliverable Completion.

a. Minimum Service Level: Draft Deliverable completed by the Vendor submitted to EOHHS for review within 15 Business Days of the agreed upon date for Milestone completion as set forth in the Implementation Plan as described in RFR Section 3.2.1.

b. Desired Service Level: Draft Deliverable completed by the Vendor submitted to EOHHS for review before or on the agreed upon date for Milestone completion as set forth in the Implementation Plan as described in RFR Section 3.2.1.

B. Deliverable Turnaround Time.

a. Minimum Service Level: Every time EOHHS provides comments/edits to a Deliverable, Vendor completes edits in response within 10 Business Days of feedback being provided.

b. Desired Service Level: Every time EOHHS provides comments/edits to a Deliverable, Vendor completes edits in response within 5 Business Days of feedback being provided.

C. Number of Revisions.

a. Minimum Service Level: EOHHS provides comments/edits to a Deliverable no more than three (3) times before the Deliverable is finalized.

b. Desired Service Level: EOHHS provides comments/edits to a Deliverable no more than one (1) time before the Deliverable is finalized.

D. Finalization of Deliverables.

a. Minimum Service Level: Deliverables are finalized within 20 Business Days after draft is first sent to EOHHS by Vendor.

b. Desired Service Level: Deliverables are finalized within 10 Business Days after draft is first sent to EOHHS by Vendor

Dependencies:

Dependent on the establishment and use of a mutually agreed upon Deliverable Acceptance Metrics set forth in Attachment 3, e-CAS Reports and Plans.

1.2 Goal: e-CAS Quality

1.2.1 KPI: Deliver all business Requirements as agreed upon.

Metrics:

A. The percentage of requirements delivered, measured per release.

a. Minimum Service Level: The Vendor delivers within each release 99% of the Requirements in production that were signed off on by both EOHHS and the Vendor for the scope of all planned software releases. Requirements must meet the agreed upon Definition of Done as specified in the Requirements documentation (e.g., RTM). To qualify for this Service Level, the Vendor must meet the threshold for all planned software releases.

b. Desired Service Level: The Vendor delivers within each release 100% of the Requirements in production that were signed off on by both EOHHS and the Vendor for the scope of all planned software releases. Requirements must meet the agreed upon Definition of Done as specified in the Requirements documentation (e.g., RTM). To qualify for this Service Level, the Vendor must meet the threshold for all planned software releases.

Dependencies:

Dependent on the establishment and use of a mutually agreed upon Requirements documentation (including Requirements Traceability Matrix) and related process as input to releases. This would include the process for the mutual vetting and signing off on the requirements.

1.2.2. KPI: Deploy high code quality within functionality delivered.

Metrics:

A. Number of Defects in delivered Requirements.

a. Minimum Service Level: Defects are identified post-production in no more than 2% of Requirements that have been scoped, documented, and completed all necessary testing (unit, integration and user acceptance).

b. Desired Service Level: Defects are identified post-production in no more than 0.5% of Requirements that have been scoped, documented, and completed all necessary testing (unit, integration and user acceptance).

B. Time to fix Defects.

a. Minimum Service Level: The Vendor fixes, tests, and redeploys all identified Defects based upon the mutually agreed upon timeframe for resolution by Severity Level as specified in RFR Section 3.2.8 Deployment and Acceptance Plan.

b. Desired Service Level: The Vendor fixes, tests, and redeploys all identified Defects 50% faster than the mutually agreed upon timeframe for resolution of Defects by Severity Level as specified in RFR Section 3.2.8 Deployment and Acceptance Plan.

C. Percentage of Defects to be fixed for code to be accepted.

a. Minimum Service Level: The Vendor fixes at least 99% of all Defects reported in any test environment.

b. Desired Service Level: The Vendor fixes 100% of all Defects reported in any test environment.

Dependencies:

Dependent on the establishment and use of a mutually agreed upon Requirements documentation (including Requirements Traceability Matrix) and related process as input to releases. This would include the process for the mutual vetting and signing off on the requirements.

Dependent on mutually agreed upon timeframes for Defect resolution and definition of Defects.

Notes:

Please see Defect Severity Definitions in Section 5 of this Attachment.

2 Maintenance and Operations KPIs

2.1 Strategic Focus: Quality

2.1.1 Goal: e-CAS Quality

2.1.1.1 KPI: Deliver all Business Requirements as agreed upon.

Metrics:

A. The percentage of requirements delivered, measured per release.

a. Minimum Service Level: The Vendor delivers within each release 99% of the Requirements in production that were signed off on by both EOHHS and the Vendor for the scope of all planned software releases. Requirements must meet the agreed upon Definition of Done as specified in the Requirements documentation (e.g., RTM).

b. Desired Service Level: The Vendor delivers within each release 100% of the Requirements in production that were signed off on by both EOHHS and the Vendor for the scope of all planned software releases. Requirements must meet the agreed upon Definition of Done as specified in the Requirements documentation (e.g., RTM).

Dependencies:

Dependent on the establishment and use of a mutually agreed upon Requirements documentation (including Requirements Traceability Matrix) and related process as input to releases. This would include the process for the mutual vetting and signing off on the requirements.

2.1.1.2. KPI: Deploy high code quality within functionality delivered.

Metrics:

A. Number of Defects in delivered Requirements.

a. Minimum Service Level: Defects are identified post-production in no more than 5% of the Requirements that have been scoped, documented, and completed all necessary testing (unit, integration and user acceptance).

b. Desired Service Level: Defects are identified post-production in no more than 2% of Requirements that have been scoped, documented, and completed all necessary testing (unit, integration and user acceptance).

B. Time to fix Defects.

a. Minimum Service Level: The Vendor fixes, tests, and redeploys all identified Defects based upon the mutually agreed upon timeframe for resolution of Defects by Severity Level.

b. Desired Service Level: The Vendor fixes, tests, and redeploys all identified Defects 50% faster than the mutually agreed upon timeframe for resolution of Defects based on Severity Level.

C. Percentage of Defects fixed for code to be accepted.

a. Minimum Service Level: The Vendor fixes at least 99% of all Defects reported in any test environment.

b. Desired Service Level: The Vendor fixes 100% of all Defects reported in any test environment.

D. Number of times a release causes Incidents in production each year.

a. Minimum Service Level: No more than one (1) Incident each year resulting from code promoted in a release.

b. Desired Service Level: Zero (0) Incidents each year resulting from code promoted in a release.

E. Quality of scheduled jobs.

a. Minimum Service Level: 99.9% of scheduled jobs are performed on time, to completion, and their outputs are correct.

b. Desired Service Level: 100% of scheduled jobs are performed on time, to completion, and their outputs are correct.

Dependencies:

Dependent on the establishment and use of a mutually agreed upon Requirements documentation (including Requirements Traceability Matrix) and related process as input to releases. This would include the process for the mutual vetting and signing off on the requirements.

Dependent on mutually agreed upon timeframes for Defect resolution and definition of Defects.

Notes:

Please see Defect Severity Definitions in Section 5 of this Attachment.

2.1.2 Goal: e-CAS Data Quality

2.1.2.1 KPI: Ensure data held in e-CAS is high quality.

Metrics:

A. Percent of Bad Data received from other systems that is corrected.

a. Minimum Service Level: The Vendor corrects 50% of Bad Data received from other systems according to a data correction process that is mutually agreed upon by EOHHS and the Vendor.

b. Desired Service Level: The Vendor corrects 80% of Bad Data received from other systems according to a data correction process that is mutually agreed upon by EOHHS and the Vendor.

B. Percent of data in database that contains errors.

a. Minimum Service Level: No more than 5% of records in the e-CAS database contain Bad Data.

b. Desired Service Level: No more than 2% of records in the e-CAS database contain Bad Data.

C. Resolution of Bad Data issues within mutually agreed upon time period.

a. Minimum Service Level: Vendor resolves all Bad Data issues within the agreed upon timeframe established by EOHHS and the Vendor.

b. Desired Service Level: Vendor resolves Bad Data issues 50% faster on average than the agreed upon timeframe established by EOHHS and the Vendor.

D. Number of rejected records when files sent to EOHHS systems.

a. Minimum Service Level: No more than 2% of records sent to any EOHHS system will be rejected due to Bad Data or other data transfer errors.

b. Desired Service Level: No more than 0.5% of records sent to any EOHHS system will be rejected due to Bad Data or other data transfer errors.

E. Percentage of data domains covered by Rulebooks.

a. Minimum Service Level: 99.9% of data is covered by rulebooks as measured by any point in time audit of Rulebooks. EOHHS will conduct random audits of Rulebooks at least once annually.

b. Desired Service Level: 100% of data is covered by rulebooks as measured by any point in time audit of rulebooks. EOHHS will conduct random audits of Rulebooks at least once annually.

Dependencies:

Dependent on a mutually agreed upon definition of Bad Data that has been defined in sufficient detail to operationalize the KPI, specifying what data errors are acceptable and unacceptable, including problems with data completeness, accuracy, consistency, validity, and integrity.

2.2 Strategic Focus: User Experience

2.2.1 Goal: e-CAS Provides Good User Experience

2.2.1.1 KPI: Users indicate satisfaction with the e-CAS user interface.

Metrics A. System Usability Scale (SUS) performance.

a. Minimum Service Level: The average SUS score for the e-CAS as measured via user surveys each quarter is greater than or equal to 68.

b. Desired Service Level: The average SUS score for the e-CAS as measured via user surveys each quarter is greater than or equal to 80.

B. Customer Satisfaction score (CSAT) performance.

a. Minimum Service Level: The average CSAT score for the e-CAS as measured via user surveys each quarter is greater than or equal to 80%.

b. Desired Service Level: The average CSAT score for the e-CAS as measured via user surveys each quarter is greater than or equal to 95%.

Dependencies:

The Vendor must implement regular pop-up surveying of users as described in RFR Section 1.4.1.2.7.

2.2.1.2 KPI: e-CAS is quick and reliable.

Metrics A. Error Rate for e-CAS webpage.

a. Minimum Service Level: The e-CAS webpage has an Error Rate of less than or equal to 0.05%. Errors included in the Error Rate calculation are all 400 and 500 level errors.

b. Desired Service Level: The e-CAS webpage has an Error Rate of less than or equal to 0.001%. Errors included in the Error Rate calculation are all 400 and 500 level errors.

B. e-CAS webpage Time to Interact.

a. Minimum Service Level: The e-CAS webpage Time to Interact is less than or equal to 3.8 seconds.

b. Desired Service Level: The e-CAS webpage Time to Interact is less than or equal to 2.2 seconds.

C. Time for System Logic to display Outputs once assessment is finalized.

a. Minimum Service Level: Once assessment is finalized (i.e., final reviewer signs off and submits the assessment), the System Logic provides Outputs within 1 minute.

b. Desired Service Level: Once assessment is finalized (i.e., final reviewer signs off and submits the assessment), the System Logic provides Outputs within 10 seconds.

D. Number of Help Desk tickets handled by the Help Desk Vendor that require escalation to the e-CAS Vendor.

a. Minimum Service Level: Fewer than 5% of Help Desk tickets handled by the Help Desk Vendor require escalation to the e-CAS Vendor staff according to agreed-upon escalation criteria.

b. Desired Service Level: Fewer than 2% of Help Desk tickets handled by the Help Desk Vendor require escalation to the e-CAS Vendor staff according to agreed-upon escalation criteria.

For the purposes of calculating Metrics A-C, e-CAS shall be measured under conditions of 500 or fewer concurrent users using appropriate devices during Business Hours on Business Days. The Vendor should utilize off the shelf software tools to monitor these Metrics. The Vendor is responsible for ensuring the performance of all subcontractors, platforms, and software tools that it uses to maintain the e-CAS.

Dependencies:

Dependent upon establishing agreed upon escalation criteria for when Help Desk Vendor should escalate tickets to e-CAS Vendor.

2.3 Strategic Focus: System Availability

2.3.1 Goal: Consistent and Full Availability of e-CAS

2.3.1.1. KPI: Maximize e-CAS availability.

Metrics A. Percent of time the e-CAS is Available.

a. Minimum Service Level: e-CAS is Available to users for 99.99% of the time, outside of prior-approved valid time windows for system maintenance and updates each month. This measurement is by application, not by server instance.

b. Desired Service Level: e-CAS is Available to users for 100% of the time, outside of prior-approved valid time windows for system maintenance and updates each month. This measurement is by application, not by server instance.

B. Percent of Time the e-CAS is fully functioning (i.e., no Degradation of Service such as system being unusable, down, or a key function being unusable or down).

a. Minimum Service Level: e-CAS provides non-Degraded Service to users for 99% of the time each month. This measurement is by application, not by server instance.

a. Desired Service Level: e-CAS provides non-Degraded Service to users for 100% of the time each month. This measurement is by application, not by server instance.

Notes:

· Exceptions. No period of Service Degradation or inoperability will be included in calculating Availability to the extent that such downtime or degradation is due to any of the following ("Exceptions"):

· EOHHS’s or any of its Authorized Users' misuse of the Services;

· failures of EOHHS’s or its Authorized Users' internet connectivity;

· internet or other network traffic problems other than problems arising in or from networks actually or required to be provided or controlled by the Vendor or its Subcontractor(s);

· EOHHS’s or any of its Authorized Users' failure to meet any minimum hardware or software requirements set forth in the Specifications; or

· scheduled downtime.

· For purposes of calculating uptime percentage, scheduled maintenance may be excluded up to ten (10) hours per month, but unscheduled maintenance and any scheduled maintenance in excess of ten (10) hours must be included as downtime.

· Scheduled maintenance must occur: with at least two (2) Business Days’ advance notice; at agreed-upon times when a minimum number of users will be using the system; and in no event during Business Hours unless prior approved to address a situation deemed emergent.

· System shall be considered up if it has a response time of 3 seconds or less for 500 or fewer concurrent users during Business Hours on Business Days. Data for sessions greater than 500 concurrent users can be excluded from the calculation of the KPI.

2.3.1.2 KPI: Ensure timely notification and resolution of e-CAS issues.

Metrics A. System unavailability notification time.

a. Minimum Service Level: In 100% of cases, the Vendor notifies EOHHS within two (2) hours of discovering or receiving notice of the system not being Available.

b. Desired Service Level: In 100% of cases, the Vendor notifies EOHHS within 30 minutes of discovering or receiving notice of the system not being Available.

B. Problem circumvention or resolution time.

a. Minimum Service Level: The Vendor circumvents or resolves the problem within the agreed upon timeframe depending on the Priority Level and Service Level as specified in Section 5 of this Attachment.

b. Desired Service Level: Same as Minimum Service Level.

Notes:

Please see Section 5 of this Attachment for Priority Level and Severity Level definitions.

2.4 Strategic Focus: Throughput

2.4.1 Goal: Efficient Release and Change Management

2.4.1.2 KPI: Ensure timely and cost-effective delivery of e-CAS releases.

Metrics A. Approved project timelines specified as part of the Change Request process.

a. Minimum Service Level: The Vendor delivers all e-CAS releases within the timeline agreed upon by EOHHS and the Vendor.

b. Desired Service Level: The Vendor delivers all e-CAS releases 10% faster than the timeline agreed upon by EOHHS and the Vendor.

B. Approved project costs specified as part of Change Request process.

a. Minimum Service Level: The Vendor delivers all e-CAS releases within the costs agreed upon by EOHHS and the Vendor.

b. Desired Service Level: Same as Minimum Service Level.

5.1.3 KPI: Provide high level of e-CAS configurability.

Metrics A. Percent of EOHHS-requested changes to Assessment Tools that are configurable by EOHHS users.

a. Minimum Service Level: Over 70% of requests for changes to all Assessment Tools are configurable by EOHHS users.

b. Desired Service Level: Over 90% of requests for changes to all Assessment Tools are configurable by EOHHS users.

B. Percent of EOHHS-requested changes to e-CAS System Logic that are configurable by EOHHS users.

a. Minimum Service Level: Over 70% of requests for changes to the existing System Logic are configurable by EOHHS users.

b. Desired Service Level: Over 90% of requests for changes to the existing System Logic are configurable by EOHHS users.

2.4.2 Goal: Efficient Data Management

2.4.2.1 KPI: Manage the data lifecycle efficiently.

Metrics

a. Availability of data via API.

a. Minimum Service Level: 100% of data produced within the e-CAS is immediately available to send to Partner Systems and Trading Partners via Application Programming Interface (API).

b. Desired Service Level: Same as Minimum Service Level.

b. Time between data entry into e-CAS and sending data extracts to Partner Systems and Trading Partners.

a. Minimum Service Level: All data entered into the e-CAS is sent to all agreed upon Partner Systems and Trading Partners within 24 hours.

b. Desired Service Level: All data entered into the e-CAS is sent to all agreed upon Partner Systems and Trading Partners within 12 hours.

c. Time between receiving data extracts from Partner Systems and making data available to e-CAS users.

a. Minimum Service Level: All data received from Partner Systems and Trading Partners is available to e-CAS users within 18 hours.

b. Desired Service Level: All data received from Partner Systems and Trading Partners is available to e-CAS users within 6 hours.

3 Liquidated Damages The parties acknowledge and agree that each of the liquidated damages assessed pursuant to this Section 3: (a) is a reasonable estimate of the diminished value of the Services and/or compensation to EOHHS for the anticipated or actual harm that may arise from the corresponding service error or failure, which would be impossible or very difficult to accurately estimate as of the Effective Date of the Contract; (b) is not intended as, and should not be deemed to be, a penalty or forfeiture; and (c) may, at EOHHS's option, be credited or set off against any fees or other charges payable to Contractor under this Agreement or be payable to EOHHS upon demand.

At any point during this Contract, if EOHHS identifies, in its sole judgment, any deficiency in the Vendor’s performance under this Contract, EOHHS may assess liquidated damages in addition to any Credits provided by the Vendor to EOHHS for failure to meet the Minimum Service Level. The intention of liquidated damages is to compensate EOHHS in the event of a serious incident or issue that results in material damage to EOHHS operations. No punitive intention is inherent in such liquidated damages provisions. The following conditions apply:

a. EOHHS may assess liquidated damages in the following cases:

1. System Unavailable or Degraded. In the event of a major system failure or issue, which causes the e-CAS Application, database, or interfaces to not be Available or to have Degraded Service so that it is unusable for an extended period of time beyond the Minimum Service Level and that has not been addressed and resolved according to the procedures for resolution of issues and Defects outlined in Section 5 of this attachment. Applies to issues or Defects of Severity Level S1 and S2 and Priority Level P1 and P2 where no work-around or problem circumvention is implemented prior to issue or Defect resolution or the Defect is not resolved within the specified timeframe. Material damage is caused to EOHHS as Assessor productivity is lost, thereby resulting in delays in MassHealth Members’ assessments being conducted, which may further result in delays to Members receiving Medically Necessary services.

2. Security Breach. In the event of a substantial breach of e-CAS security or e-CAS data security, resulting in significant damage to or loss of Commonwealth Data, Third Party Data, or other personal information or health information held within the e-CAS Application or database. Material damage is caused to EOHHS as the loss of or improper use of Commonwealth Data by unauthorized parties will harm Commonwealth employees and/or MassHealth Members.

3. Vendor Non-Responsiveness or Inaction. In the case of significant non-responsiveness or inaction on the part of the Vendor. These liquidated damages shall be assessed after the Parties have completed the corrective action plan process as described in Section 6.5 of Attachment 1, Statement of Work and the corrective action plan has not been implemented as agreed upon or has not resulted in the agreed upon improvements in the agreed upon timeframe as specified in the corrective action plan. Material damage is caused to EOHHS as the Vendor is responsible for a large proportion of MassHealth’s assessment operations per the Contract, which will be performed to a poor standard, thereby harming EOHHS operations.

b. The Vendor shall not be subject to liquidated damages for any performance failures caused by EOHHS or Commonwealth network performance issues or operator/administrator errors.

c. The EOHHS Project Manager may waive these liquidated damages at his/her reasonable discretion.

d. These liquidated damages shall in no way limit the rights of EOHHS to pursue other non-monetary remedies available to EOHHS under the contract or at law.

e. For any performance failure, EOHHS may choose not to impose liquidated damages and to seek other remedies, including, without limitation, monetary remedies, available to EOHHS under the Contract or at law.

f. Payment of any liquidated damages shall not relieve the Vendor from its obligation to meet the responsibilities established by the Contract.

g. In the event that EOHHS assesses liquidated damages, EOHHS will provide the Vendor a written notice for payment of liquidated damages, which explains the basis for the assessment. The Vendor shall have the opportunity to respond in writing to the EOHHS notice of assessment of liquidated damages within 30 Calendar Days of receipt of such notice. After reviewing any timely-submitted response by the Vendor, including any contention that EOHHS has acted unreasonably in its assessment, EOHHS shall issue a final notice to the Vendor regarding the assessment of liquidated damages.

h. The Vendor shall, within 30 Calendar Days of receipt of the final notice, or if the Vendor chooses not to respond to the original notice, within 30 Calendar Days of receipt of the original notice, either pay the liquidated damages in full, or elect to have EOHHS withhold the amount of the liquidated damages from future payment(s).

i. The maximum liquidated damages that EOHHS may impose on the Vendor in any month shall not exceed 80% of the total amount owed to the Vendor for that month.

j. Guidelines for amount of liquidated damages to be assessed in each case are as follows:

1. System unavailable or degraded as defined in this Section 3.a.1:

2. Security breach as defined in this Section 3.a.2:

3. Vendor non-responsiveness or inaction as defined in this Section 3.a.3:

4 System Usability Scale (SUS) & Customer Satisfaction Survey (CSAT)

4.1 System Usability Scale (SUS)

When a SUS is used, users are asked to score the following 10 items with one of five responses that range from Strongly Agree to Strongly Disagree:

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Calculation of the SUS score will be as follows:

For each response:

1. Add up the total score for all odd-numbered questions (questions 1, 3, 5, 7, and 9), then subtract 5 from the total to get (X).

2. Add up the total score for all even-numbered questions (questions 2, 4, 6, 8, and 10), then subtract that total from 25 to get (Y).

3. Add up the total score of the new values (X+Y) and multiply by 2.5.

Example scoring:

1. Odd = (4+5+3+4+3) = 19 – 5 = 14

2. Even = (2+1+3+1+1) = 25 – 8 = 17

3. SUS Score: (14+17) x 2.5 = 77.5

By following the scoring tabulation methodology above, a SUS score out of 100 will be produced (in the example above: 77.5/100).

4.2 Customer Satisfaction Survey (CSAT)

For the purposes of the e-CAS, EOHHS expects users will be asked to score the following item with one of five responses on a scale of 1-5 that ranges from Highly Unsatisfied (1) to Highly Satisfied (5):

1. Overall, how satisfied are you with your most recent interaction with the e-CAS?

EOHHS expects users will be asked to score the following 2 items with one of five responses on a scale of 1-5 that ranges from Highly Unlikely (1) to Highly Likely (5):

2. Based on your most recent interaction with the e-CAS, how likely are you to use it again?

3. Based on your most recent interaction with the e-CAS, how likely are you to recommend it to a colleague?

Calculation of the CSAT will be as follows:

CSAT score ranges from 0-100. To calculate the average score for each measurement period, take the number of satisfied customers (those with responses of 4 or 5), and divide by the total number of responses. For example, if 62 of 100 responses have a rating of 4 or 5, the score would be 62.

5 Priority Level and Defect Severity Level Definitions

5.1 Priority Levels Definitions

EOHHS defines the following Priority Levels for system availability problems:

1) Priority 1 (P1) – Total system outage. Hosted system or major functionality of system is not available or operational which affects all users, performing critical functionality where there is no workaround.

2) Priority 2 (P2) – Impairment of critical system functions. Hosted system or major functionality of system is not available or operational for multiple users. No workaround exists, or workaround is cumbersome and causes an impact on productivity.

3) Priority 3 (P3) – Impairment of less critical system functions for multiple users. Other functionality is available and operational for users. The function(s) will not perform as expected and the business impact is moderate.

4) Priority 4 (P4) – Inconvenience, annoyance or cosmetic issue. Minor function of hosted system is not operational for single user, but all other functionality is available and operational for users.

While the Vendor may initially classify Priority Levels, EOHHS determines the final classification of system availability issue Priority Levels.

5.2 Defect Severity Level Definitions

EOHHS defines the following Severity Levels for Defects:

1) Severity Level 1 (S1) – A Defect that completely obstructs the execution of a core functionality/feature of the software is classified a critical Defect. This Defect affects the critical functionalities and data and makes the testing of the software difficult.

2) Severity Level 2 (S2) – A Defect that makes a major functionality/feature behave grossly away from what is specified in the requirements specification of the software is classified a major Defect. This Defect affects the major functionalities and data.

3) Severity Level 3 (S3) – A Defect which occurs when a functionality/feature does not behave as intended or exhibits some unnatural behavior, however the functionality/feature as a whole is not much impacted is classified a minor Defect. This minimally affects the functionalities and data.

4) Severity Level 4 (S4) – Any cosmetic Defects such as misplaced images, spelling mistakes or alignment issues or font casing is classified a trivial Defect. This Defect does not affect functionalities and data.

While the Vendor may initially classify Defects, EOHHS determines final classification of Defects.

5.3 Resolution Timeframe

EOHHS and the Vendor will agree upon resolution timeframe guidelines depending up on the Priority Level and Defect Severity Level of issues discovered. EOHHS anticipates the guidelines will be similar to those below:

[REMAINDER OF PAGE INTENTIONALLY BLANK]

Attachment 5, EOHHS Key Performance Indicators 2 6 KPI Calculation Tables As part of the Annual KPI Plan described in RFR Section 3.4.1, information for each Metric within each KPI will be recorded, including but not limited to: Metric description, calculation details, relative weight as compared to other Metrics, measurement timeframe, value of Metric for the Minimum Service Level and Desired Service Level. EOHHS anticipates this information will be recorded and maintained in tables similar to those below.

6.1 Implementation KPIs

a. Minimum Service Level: Upon review of Deliverable, EOHHS staff provide corrections to less than 10% of the Deliverable content.

b. Desired Service Level: Upon review of the Deliverable, EOHHS staff provide corrections to less than 5% of the Deliverable content.

1/10 = 10%

Ex.

Project Initiation completion - $60,000 Requirement Elaboration and Specification Definition - $60,000 Production Deployment - $60,000 Certification Termination of Warranty Period - $120,000 Depends on Base Value of Contract and which Milestone the Metric is calculated for.

Ex.

Project Initiation completion - $30,000 Requirement Elaboration and Specification Definition - $30,000 Production Deployment - $30,000 Certification Termination of Warranty Period - $60,000

6.2 Maintenance and Operations KPIs

a. Minimum Service Level: The average SUS score for the e-CAS as measured via user surveys each quarter is greater than or equal to 68.

b. Desired Service Level: The average SUS score for the e-CAS as measured via user surveys each quarter is greater than or equal to 80.

1/20 = 5%

Ex. $10,000 Depends on Base Value of Contract, length of measurement timeframe, and relative weight.

Ex. $5,000