Attachment A Appendix A Key Performance Measures__20220209.pdf

Massachusetts Health Insurance Exchange Program Project Management and Testing Services

Request for Responses

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 1 Appendix A, Key Performance Measures

Appendix A

Key Performance Measures

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 2 Appendix A, Key Performance Measures

This Key Performance Measure Appendix sets forth the key performance measures (KPMs) and is subject to the terms and conditions contained in the Contract and is made a part thereof.

EOHHS requires that the Contractor perform its responsibilities and services as specified in the Contract, and will monitor Contractor to ensure that all Services and Deliverables under the Contract shall be provided at or above a level acceptable to EOHHS as set forth in the KPMs.

Further, EOHHS has identified a number of KPMs that EOHHS has determined to be of particular importance to the overall success of the HIX Program. Therefore, for the duration of the Contract, the Contractor will be expected to achieve defined targets associated with KPMs, and may be required to provide credits to EOHHS against amounts otherwise owed by EOHHS to the Contractor for Contractor’s failure to meet applicable KPMs (KPM Credits). For the avoidance of doubt, the application of any KPM Credits shall not be considered as liquidated damages, nor shall the application of any KPM Credits constitute a sole or exclusive remedy of Client. Client shall retain all other legal and equitable rights and remedies in addition to KPM Credits.

For the first Contract Year, the Contractor is expected to achieve the KPM targets defined below. These KPMs apply to the period beginning on the Contract Effective Date and ending on December 31, 2023 (the “First KPM Term”).

KPMs for subsequent Contract Years shall be established in writing by negotiation between the Parties, and defined no later than ninety (90) days prior to the start of each subsequent Contract year; provided, however, that in the event Parties are unable to reach agreement on any revised KPMs before thirty (30) days prior to the start of the next Contract Year, the KPMs from the preceding Contract Year shall remain in effect, unless the Client determines that they are no longer applicable.

1. Single Point of Contact

Achievement of these KPMs will require coordination and collaboration with the Client and HIX Program Vendors. The Contractor shall provide a Single Point of Contact (SPOC) for the prompt resolution of issues or conflicts pertaining to the achievement of the KPMs.

2. Periodic Review

The KPMs set forth in Tables 1-3 will be reviewed on a periodic basis by the Client so that it can determine if the defined measures and targets are appropriate. Either the Client or the Contractor may request further KPM review if, in their reasonable determination, any KPM or associated target is no longer appropriate, if new measures and targets need to be added, or if there are changes to the application or infrastructure that will impact the existing targets. The Parties will work together in good faith to consider any changes to the KPMs and will mutually collaborate in an effort to resolve any differences regarding the applicability of any existing or proposed measures or targets. If after a

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 3 Appendix A, Key Performance Measures reasonable period of review, the Parties cannot reach agreement, then the issues will be documented by the Project Managers and escalated in accordance with the terms of the Contract. If questions remain regarding the appropriateness or applicability of any KPM that is the subject of the review, Client will make a final determination, taking into account the respective position(s) of both Parties.

3. Assessment of KPM Credits

Contractor’s performance shall be assessed against each KPM as described in Tables 1, 2, and 3. KPM Credits shall be assessed as credits on the monthly invoice for the relevant Service Category (i.e., Program Management or Testing and Validation Services) for the month following the month during which the Contractor failed to meet the KPM(s), up to the maximum per-month as defined below.

EOHHS’s failure to assess KPM Credits in one or more of the KPMs described in this agreement shall not constitute a waiver of any of its rights to assess liquidated or actual damages, seek any other remedy available or exercise any of its other rights under the Contract, nor constitute acceptance of Contractor’s performance.

4. KPM Reporting

Contractor shall provide the Commonwealth with a written report of its performance against all KPMs, whether or not any KPM Credit trigger was met, in accordance with the timeframes specified in Tables 1-3, below. The report will specifically address any and all occurrences where a target is not met and it will describe: (1) the KPM for which the target was missed; (2) the specific cause of failure to meet the target; (3) remedial action to be taken by the Contractor; and (4) if applicable, any actions the Contractor will take to ensure future targets for that KPM will be met. Upon delivery of the report, the Commonwealth may request a meeting with Contractor to further discuss issues related to the report.

Contractor shall immediately notify Client when Contractor determines that a KPM Credit is triggered, and Client shall immediately notify Contractor when Client determines a KPM Credit is triggered. Such notifications shall be provided by the Project Managers in writing, where "writing" may include the transmission of an email, or documenting the trigger in JIRA or in another system agreed upon by the Parties. Each such notice shall state the specific Target and measure that Contractor has triggered; and, if known at the time of notification, the date such Target and measure was triggered, based on knowledge available at the time of the notice. If any of the required elements referenced in this section is not known at the time of notification, they shall be provided as part of the incident report or other update as soon as practicable.

5. KPMs for First KPM Term

Tables 1-3 detail the KPMs for the First KPM Term. Each table provides the reporting cadence and maximum KPM Credit for that category of KPM.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 4 Appendix A, Key Performance Measures

KPMs: Contract Readiness and Start-Up Activities

• Reporting: Weekly (Contract Effective Date – Contract Operational Start Date)

• Maximum Damages: 35% of total Transition Costs

Table 1: Contract Readiness and Start-Up Activities

KPI IID Assessment Period(s) Summary Description Target Measure KPM Credit

CR01 N/A Submission of Contract Readiness Workplan

The Contractor must submit an acceptable Contract Readiness Workplan no later than 15 Business Days following the Contract Effective Date

Not later than 15 Business Days following the Contract Effective Date

Number of Business Days beyond the due date until the Workplan is submitted

$5,000/Business Day

Exception(s) None. If Contractor is unable to obtain access to Client or any third-party resources (including personnel and documentation) necessary for the completion of the Workplan, Contractor shall document this within the Workplan (for example, as a Risk, Constraint or Assumption as applicable) and shall escalate immediately to the Client’s Project Manager.

CR02 N/A Successful Contract Readiness Reviews – (Section 2.5.A of the Contract)

For each Contract readiness review, Contractor successfully demonstrates that it is able to meet the Contract requirements identified as part of the readiness review; and provides EOHHS and EOHHS-designees access to all facilities, information, materials, documentation, and other information or resources necessary to validate Contractor’s representations

All Readiness Review requirements are completed successfully

For each Contract Readiness Review, the number of readiness review Contract requirements that the Contractor fails to meet, i.e., an Unmet Requirement (see Section 2.5.A of the Contract)

$5,000 per Unmet Requirement per readiness review, up to $25,000 per readiness review

Contractor may recoup $3,500 per Unmet Requirement if the requirement is met by the next Readiness Review

If on the Operational Start Date there are any remaining Unmet Requirements, the Contractor shall be assessed an additional KPM Credit of $25,000

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 5 Appendix A, Key Performance Measures

Table 1: Contract Readiness and Start-Up Activities

KPI IID Assessment Period(s) Summary Description Target Measure KPM Credit

Exception(s) If an Unmet Requirement is solely the result of the Client’s or a third party’s actions or failure to act, and the Contractor is able to demonstrate that it took all reasonable steps to address such action or inaction in a timely manner (including but not limited to escalating to Client’s Project Manager), such Unmet Requirement will not be counted against the Contractor for the purposes of calculating the KPM Credit.

CR03 N/A Transition Workplan Submission of a detailed Transition project plan, consistent with Contract requirements, identifying resource requirements, dependencies, and critical paths of all implementation tasks and activities necessary to ensure a smooth transition to the new contractual landscape

No later than 15 Business Days following Contract Effective Date

Number of Business Days beyond the due date until the Workplan is submitted

$5,000 per Business Day

Exceptions None. If Contractor is unable to obtain access to Client or any third-party resources (including personnel and documentation) necessary for the completion of the Workplan, Contractor shall document this within the Workplan (for example, as a Risk, Constraint or Assumption as applicable) and shall escalate immediately to the Client’s Project Manager.

CR04 N/A OTS Transition Workplan Submission of a detailed OTS Transition Workplan, consistent with Contract requirements, identifying resource requirements, dependencies, and critical paths of all implementation tasks and activities necessary to ensure a smooth transition to the new contractual landscape (i.e., transition to the new OTS Vendor or contractual arrangement)

No later than 30 Business Days following Contract Effective Date

Number of Business Days beyond the due date until the Workplan is submitted

$5,000 per Business Day

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 6 Appendix A, Key Performance Measures

Table 1: Contract Readiness and Start-Up Activities

KPI IID Assessment Period(s) Summary Description Target Measure KPM Credit

Exception(s) None. If Contractor is unable to obtain access to Client or any third-party resources (including personnel and documentation) necessary for the completion of the Workplan, Contractor shall document this within the Workplan (for example, as a Risk, Constraint or Assumption as applicable) and shall escalate immediately to the Client’s Project Manager.

CR05 Contract Readiness

Key Subcontracts Submission to EOHHS for review and approval copies of any Key Subcontracts the Contractor intends to enter into for the purpose of performing any of the responsibilities of the Contract

No later than 10 Business Days following the Contract Effective Date

Number of Business Days beyond the due date until the subcontracts are submitted

$1,000 per Business Day per Subcontract

Exception(s) None

CR06 Contract Readiness

Key Personnel Onboarding All Key Personnel, as approved by EOHHS, are onboarded

No later than 30 Calendar Days following Contract Effective Date

Number of Calendar Days beyond the due date until all Key Personnel are onboarded

$1,000 per Business Day per Key Personnel

Exception(s) If failure to meet this requirement is solely the result of Client delay, such delay will not be counted against the Contractor for the purpose of calculating the KPM Credit

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 7 Appendix A, Key Performance Measures

KPMs: Project Management KPMs: Reporting:

• Bi-Weekly (Contract Effective Date – Contract Operational Start Date)

• Monthly (Contract Operational Start Date – End of Contract Year 1)

Maximum Damages: 10% of the total annual Fixed Price for Project Management Services

Table 2: Project Management

KPI IID Assessment Periods Summary Description Target Measure KPM Credit

PM01 Quarterly, beginning on Operational Start Date

Timeliness of Deliverable Submission

% of Project Deliverables submitted on time

99% For the preceding quarter: (# of Deliverables Contractor was required to submit that were submitted within one Business Day of the due date/: total # of Deliverables Contractor was required to submit) * 100 This measure shall be calculated separately for those Deliverables that the Contractor is required to produce itself, and those Deliverable the Contractor is required to compile based on information provided by Client or HIX Program Vendors

For Deliverables Contractor required to produce itself:

$5,000, per percentage point below the target For Deliverables compiled by Contractor based on information obtained from HIX Program Vendors: $1,000 per percentage point below the target Total maximum combined quarterly credit is $50,000

Exceptions If a late or missing Deliverable is solely the result of the Client’s or a third party’s actions or failure to provide required material or information in a timely manner, and the Contractor is able to demonstrate that it took all reasonable steps to address such (including but not limited to escalating to Client’s Project Manager), such Deliverable shall be excluded from the denominator for the purposes of this measure.

PM02 Quarterly, beginning 3 months following the Operational Start Date

Quality of PM Deliverables

% of Project Deliverables Unconditionally Accepted after one round of revisions

95% For the preceding quarter: (# of Deliverables Unconditionally Approved by the Client upon initial submission plus # of Deliverables Unconditionally Approved by the Client after one round of revisions/ total # of Deliverables submitted by the Contractor) *

$1,000 per percentage point below the target 50% of the penalty will be waived if 100% of Deliverables are Unconditionally Approved by the Client after no more than two rounds of revisions

Exceptions Deliverables and Project Works as defined in the SDLC are excluded from this measure.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 8 Appendix A, Key Performance Measures

Table 2: Project Management

KPI IID Assessment Periods Summary Description Target Measure KPM Credit

PM03 Assessment Period 1: 6 months after Contract Operational Start Date

Assessment Period 2: 6 months after Assessment Period

Assessment of overall

PMO

Effectiveness

Effectiveness of PMO on the basis of four criteria, as reflected in survey responses:

• Streamlining program management processes

• Improving efficiency of Release Deliverables Review processes

• Promoting communication across entities and teams

• Improved accuracy of data in HIX Program’s data repositories and tools

Survey of HIX Program Entities reflects improvement in each of the four criteria:

Assessment Period 1:

• 75% of Program Team and Business Owners agree that there has been improvement in each of the 4 criteria compared to the first day of operations.

Assessment Period 2:

• >90% of HIX Program Team and Business Owners agree that there has been improvement in each of the 4 criteria at the end of the second six months compared to the first day of operations;

and

• - > 50% of HIX Program Vendors agree that there has been improvement in each of the 4 criteria at the end of the second six months compared to the first day of operations

Assessment based on % of survey respondents who say there has been at least some improvement on each of the four criteria:

Assessment Period 1

• % of HIX Program Team and Business Owners = (# of HIX Program Team and Business Owners responding positively (vs neutrally or negatively)/total # of HIX Program Team and Business Owners who responded) * 100

Assessment Period 2

• % of HIX Program Team and Business Owners responding positively = (# of HIX Program Team and Business Owners responding positively (vs neutrally or negatively)/total # of HIX Program Team and Business Owners who responded) * 100; and

• - % of HIX Program Vendors responding positively = (# of HIX Program Vendors responding positively (vs neutrally or negatively)/total # of HIX Program Vendors who responded) * 100

KPM Credits e assessed on a per-criteria basis.

Assessment Period 1:

• 65-75%: $25,000

• 51-64%: $35,000:

• 50% or below: $50,000

Assessment Period 2:

HIX Program Team/Business Owners:

• 81-90%: $25,000

• 76-80%: $35,000:

• 75% or below: $50,000

HIX Program Vendors:

• Below 50%: $25,000

Earn-backs:

If any KPM Credit was assessed in Assessment Period 1, Contractor may earn back a percentage of such KPM Credit in Assessment Period 2 as follows:

For each measure for which a KPM Credit was assessed, if performance on that measure is:

• 75%-80%, earn-back is 50% of

KPM Credit assessed

• >81% earn-back is 75% of KPM

Credit assessed

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 9 Appendix A, Key Performance Measures

Table 2: Project Management

KPI IID Assessment Periods Summary Description Target Measure KPM Credit

Exception(s) None. Client shall seek input on survey design from the Contractor and will make every good faith effort to establish consensus; however, the final survey design is within the Client’s discretion.

PM-

Key Personnel

Identify replacements for any Key Personnel role vacancies within the timeframes specified in the contract

Number of Business Days beyond the due date until the suitable replacement is identified.

$500/business day per position

Exceptions For sudden or unanticipated vacancies, Contractor and Client shall establish a mutually-agreed upon timeframe for the identification of a replacement. Penalty shall apply if replacement is not identified by that date.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 10 Appendix A, Key Performance Measures

KPMs: Testing and Validation Services

Reporting: Monthly

Maximum KPM Credit:

• Table 3.a. KPMs: No maximum

• Table 3.b. KPMs: 10% of the total annual Fixed Price for Testing and Validation Services.

Unless otherwise specified, the types of testing to which each of the KPMs in Table 3.b apply (e.g., functional, system, regression) shall be based on the Contractor’s Software Testing Lifecyle as approved by the Client and as reflected in the approved master test plan, and shall be by mutual agreement of the Parties.

Unless otherwise indicated, there will be no targets for performance during the 1st major release cycle following the Operational Start Date. Performance will be measured to establish a baseline. Targets will be defined for performance in the second and third major functional release cycles:

• 2nd Major Functional Release: Target will be defined by the Client and will be based in part on percentage improvement over baseline

• 3rd Major Functional Release: Target will be fixed based on Release 2 Performance and other factors.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 11 Appendix A, Key Performance Measures

Testing and Validation KPMs Table 3.a

KPI IID Assessment Period(s) Summary Description Target Measure KPM Credit

TV01 Major Functional Release

Regression Suite Maintenance

Regression test suite remains current Regression test suite is updated as of the current release and maintained release over release

Regression test suite is updated to reflect all new or modified functional and non-functional requirements introduced in the current release

1 Major Release behind:

$25,000 2 Major Releases behind:

$$75,000

Exceptions None

TV02 Major Functional Release (3x/year)

Release Schedule Management

Complete all testing activities within the timeframes set forth in the approved Integrated Program Schedule (IPS)

Each test phase is completed no later than the date specified in the approved IPS

For each test phase, number of Calendar Days beyond the identified completion date For phases following the first instance of a missed target, the measure will be based on duration of the phase, and not the completion date unless the schedule was re-baselined by written agreement of the Parties with revised completion dates defined.

For the first instance in a release cycle: $1,000 per Calendar Day beyond the defined completion date For subsequent instances within a release cycle: $1,000 per Calendar Day beyond the defined duration of the phase, or, if schedule was re-baselined, beyond the redefined completion date.

An additional $25,000 KPM Credit will be assessed if delay in test execution necessitates a delay in the final release deployment date.

Exception(s) If failure to meet this requirement is the result of (1) unknown and unforeseeable unavailability of properly configured environments, and/or (2) delays in code delivery or code fixes by the HIX Application vendor, such delay will not be counted against the Contractor for the purpose of calculating the KPM Credit.

If action or inaction on the part of the Client or a third party contracted by the Commonwealth is determined by the Client to be the primary cause for a delay in the release, the Contractor will not be assessed the additional $25,000 KPM Credit.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 12 Appendix A, Key Performance Measures

Testing and Validation KPMs Table 3.b. Test Case Development and Execution

KPI IID Assessment Period Summary Description Measure 1st Major functional release 2nd Major functional release 3rd Major functional release

Baseline Target % improvement over baseline

KPM Credit Target KPM Credit

TV03 Major Functional Release

Requirements Coverage % of requirements that have corresponding test cases

(Number of requirements mapped to test cases / Total number of requirements) * 100

TBD TBD NA 100% $5,000 per percentage point below the target

Exceptions None

TV04 Major Functional Release

Test Case Efficiency # of requirements covered per test case

Number of requirements mapped to test cases/# of test cases

TBD TBD None TBD None

Exceptions: None.

No credits apply to this KPM

TV05 Major Functional Release

Test execution coverage

% of planned test cases that are executed

(Number of test cases executed per cycle/number of test cases planned per cycle)*100

100% % improvement not applicable.

Target=100%

$1000 per percentage point below the target

100% $5,000/ percentage point below the target

Exceptions: Test cases that cannot be executed due solely to the actions or inactions of the Client or third party shall not be included in the denominator for this measure.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 13 Appendix A, Key Performance Measures

Testing and Validation KPMs Table 3.b. Test Case Development and Execution

KPI IID

Assessment

Period Summary Description Measure 1st Major functional release 2nd Major functional release 3rd Major functional release

Baseline Target % improvement over baseline

KPM Credit Target KPM Credit

TV06 Major Functional Release – 3 months post-production

Testing Efficiency % of valid Nonconformities detected during testing

(# of valid Nonconformity detected in the test cycle/ (total # of valid Nonconformities detected during testing + 3 months post implementation))*100

TBD TBD NA 99% $5,000 per percentage point below the target, plus:

$1,000 per Severity 1 Nonconformity, and $500 per Severity 2 Nonconformity

Exceptions Valid Nonconformities detected and reported to the Client and HIX Application Vendor but not fixed will not be included in the denominator for this measure.

If HIX Application Vendor delivers a fix but such fix cannot, as determined by the Client, reasonably be implemented prior to deployment to production, such Nonconformity will not be included in the denominator for this measure.

TV07 Major Functional Release

Testing Effectiveness % of Nonconformities found during testing that are determined to be valid

(Nonconformities accepted by the HIX Application Development Team as valid/total Nonconformities reported in the cycle)*100

TBD TBD NA 85% $1,000/

percentage point below the target

Exceptions In the event that the Client and Development Team disagree on the validity of a Nonconformity, the Client’s determination shall prevail for the purposes of calculating this metric.

TV08 Conclusion of User Acceptance testing

Testing Effectiveness % of Nonconformities detected prior to

UAT

(Total Number of Nonconformities Found prior to UAT/ Total Number of Nonconformities Found Before and during UAT) *

TBD TBD NA 85% $5,000/ per percentage point below the target

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses Attachment A, Model Contract 14 Appendix A, Key Performance Measures

Testing and Validation KPMs Table 3.b. Test Case Development and Execution

KPI IID

Assessment

Period Summary Description Measure 1st Major functional release 2nd Major functional release 3rd Major functional release

Baseline Target % improvement over baseline

KPM Credit Target KPM Credit

Exceptions: Valid Nonconformities detected and reported to the Client and HIX Application Vendor but a code fix is not delivered by the Application Vendor, or in the estimation of the Commonwealth could not reasonably be implemented prior to UAT, the Nonconformity not fixed will not be included in the denominator for this measure.

TV10 Test Automation % of test cases that are automated

(# of automated test cases/total # of test cases) *100

At end of 1st major release, not less than 60% of regression tests are automated

At the end of the 2nd major release, no less than 20% improvement over Baseline performance

NA At the end of 3rd major release, no less than 20% improvement over performance in 2nd Major functional release

$10,000/percent age point below the target % increase

Exception: Test cases that inherently require manual or systemic intervention (e.g., by a 3rd party) will be excluded from the denominator for this measure