Attachment A Appendix C Testing Types__20220209.pdf

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 1 Attachment A, Model Contract Appendix C. HIX Test Types

Massachusetts Health Insurance Exchange Program Project Management and Testing Services

Request for Responses

Appendix C

HIX Testing Types

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 2

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Unit Testing Testing focused on specific component (units) of the software to determine whether it is fully functional. This should be run every time a piece of code is changed. The application owner is performing this on their internal infrastructure

MA HIX

Application

Vendor

No

Smoke Test Basic tests that are run to ensure business critical functionalities of the application are working as expected. This would ensure that the deployment/release is ready for more involved SIT testing. Smoke Testing should be run after every build and before more intensive tests are executed.

Contractor Yes

System Integration Testing (“SIT Testing”)/End-to-End Testing

System Integration Testing is carried out in an integrated software environment to verify the features/behavior and their interactions thereof as a complete system to evaluate compliance with its specified requirements.

Verify the system's coexistence with other entities (internal/external) and test the interface between features of the software application.

Contractor Yes Functional, Integration, System, and User Interface testing are components of SIT Testing.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 3

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Functional Testing Functional Testing will focus on the business requirements and product capability specifications of the application/release functions. The intent is to verify that the output of an action returns expected results as per approved business requirements and/or product specifications provided by the Core Application vendor.

Contractor Yes

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 4

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Integration Testing Integration Testing focuses on ensuring internal or external interfaces (to & from HIX) function as expected. Integration testing will validate and verify that information is transmitted/received as per mutually agreed to specifications with the internal or external integration partner(s).This testing further validates that the information exchanged thereof is processed appropriately.

Contractor Yes External Interface testing may involve coordination with the Infrastructure Vendor, HIX Security Team, and other internal or external stakeholders to allow routing or configuration changes

External Interface changes made as part of a release shall be included in the Release Test Plan

External Interface changes that are proposed outside of a release cycle, whether application or infrastructure related, shall be tested by the Contractor and performed in coordination with the PMO to ensure proper adherence to change control procedures and coordination among HIX Program teams

To fully test External Interfaces, full System Testing or System Integration Testing must occur.

System Testing

Complete application is tested as whole to determine if the software meets the requirements and meets quality standard.

This is testing the complete application prior to involving external stakeholder

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 5

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

User Interface (UI) Testing

Based on the corresponding Interface Guide, UI testing ensures that the User Interface provides the user with the appropriate access and navigation through the functions of the application. In addition, UI testing ensures that the objects within the UI function as expected and conform to corresponding or industry standards.

Contractor Yes The Contractor shall verify, at a minimum, the following:

• Navigation through the MA HIX Application properly reflects business functions and requirements. For the web user interface this includes window-to-window, field-to-field, and use of access methods (tab keys, mouse movements);

• Window objects and characteristics, such as menus, size, position, state, and focus conform to mockups or other descriptions for the look and feel, including Style Guide; and

• Proof for spelling and grammar.

Pre-production (Prod-like) Testing

Pre-production testing will focus on ensuring the new release code works well with prod-like or real time data. Pre-prod testing may also include validating major operational processes.

Contractor Yes The Contractor may be responsible for supporting the operations team in facilitating and validating pre-prod executions of major operational activities.

Regression The process of re-executing functional and/or non-functional tests to ensure that changes to the code base/system have not negatively impacted the application functionality.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 6

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

User Acceptance Testing

(“UAT”)

UAT is used to validate and verify if the software/release/build meets the expectations of the business and performs as expected prior to release go-live. This is a critical and final test to determine whether the software/release/build will be accepted.

EOHHS

Yes Support for EOHHS UAT activities, to include:

• Coordinating environment availability

• Coordinating scheduling and execution of processes and scripts for UAT teams

Database-related testing Database related changes as part of a release shall be included in the release planning testing plan.

Database changes that are proposed outside of a release cycle, whether application or infrastructure related, shall be tested by the Contractor and performed in coordination with the Contractor’s PMO to ensure proper adherence to change control procedures and coordination among HIX Program teams

Ensure that batch jobs and database scripts do not cause data integrity issues and or performance issues.

Contractor Yes Database Administration (“DBA”) functions are provided by the Infrastructure Vendor (ex: foreign keys, database structure, tablespace, capacity planning). Testing is the responsibility of the Contractor

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 7

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Accessibility Testing MA HIX Application Vendor

Software Testing performed to ensure that the application being tested is usable by people with disabilities like hearing, visual impairment, and cognitive disabilities e. Testing shall be to ensure conformance to the Commonwealth of Massachusetts’ Accessibility Standards, using the process required by the state. See https://www.mass.gov/guides/enterprise-it-accessibility-standards

MA HIX

Application

Vendor

Yes Contractor shall review and ensure that the MA HIX Application Vendor testing is complete and accurate, and shall report the results to EOHHS

Accessibility Testing (Contractor)

Software Testing performed to ensure that the application being tested is usable by people with disabilities like hearing, visual impairment, and cognitive disabilities e. Testing shall be to ensure conformance to the Commonwealth of Massachusetts’ Accessibility Standards, using the process required by the state. See https://www.mass.gov/guides/enterprise-it-accessibility-standards

Contractor

Yes In addition to Contractor review and validation of the results of Current Vendor’s Accessibility Testing (as per above row), Contractor shall perform its own Accessibility Testing and shall report the results to EOHHS.

Load/Performance/Stress

Check the performance of the system under varying loads to determine system response under normal as well as peak loads and ensure that response times is within agreed upon SLA's.

Contractor Yes https://www.mass.gov/guides/enterprise-it-accessibility-standards https://www.mass.gov/guides/enterprise-it-accessibility-standards https://www.mass.gov/guides/enterprise-it-accessibility-standards https://www.mass.gov/guides/enterprise-it-accessibility-standards

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 8

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Code Review (Peer Code Review)

Performed internally by the MA HIX Application Vendor and includes a plan for developers to consciously and systematically check each other’s code for mistakes, potential issues, or vulnerabilities.

MA HIX

Application

Vendor

No Database Administration (“DBA”) functions are provided by the Infrastructure Vendor (ex: foreign keys, database structure, tablespace, capacity planning).

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 9

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Disaster Recovery and Business Continuity Testing

The HIX cloud-based infrastructure has business continuity and disaster recovery inherently built into the infrastructure design. However, testing to validate that the HIX Program can remain operational in the event of a system failure (Business Continuity), or can recover and restore services following a critical IT failure or disruption (Disaster Recovery) is an essential testing component.

Infrastructure Vendor

Yes Responsible for coordinating with EOHHS and the Infrastructure Vendor

Develop testing plans to simulate system failures and execute the plan to assess impact to the program.

Coordinate execution of the test with the MA HIX Application Vendor and Infrastructure Vendor, and HIX Program stakeholders

Report on performance against the HIX Recovery Time Objective (“RTO”) and Recovery Point Objective (“RPO”).

RTO – how long the business can be down without causing significant damage to the organization. Measured by the amount of time between system failure and full availability. Focus is on amount of time system is down.

RPO – Refers to loss tolerance; the amount of data that can be lost before causing significant harm. Measured by the loss event to the previous backup. The focus is on the data loss following a failure.

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 10

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Infrastructure Requirements Validation Testing

Testing to ensure the Application can be installed on the specified and agreed upon system infrastructure requirements (e.g.:

software versions, database types)

Privacy Testing

Specific tests that are performed to ensure that privacy requirements are implemented correctly in systems.

The intent is to ensure the system appropriately protects Personally Identifiable Information (“PII”), Protect Health Information (“PHI”) and Federal Tax Information (“FTI”)

Contractor Yes

Security and Compliance

Tests designed to validate HIX Security and Compliance Controls and to support ongoing Security and Compliance objectives of the HIX Program.

Security Testing shall encompass the following activities: Discovery, Vulnerability Scan, Baseline Scans, Vulnerability Assessment, Security Assessment, Penetration Test, Risk Assessment, and Security Review.

MA HIX

Application

Vendor

Yes Results provided to PMO; Contractor to analyze and report to EOHHS.

Some security tests are on a predicable schedule (e.g.: IRS audit every three years, CMS Annual Attestation, while others may be ad hoc such as Commonwealth Inspector General request)

Static Code Analysis / Source Code Analysis

Method of debugging by examining source code before program is run. It is completed by analyzing the code again a set or multiple sets of coding rules

MA HIX

Application

Vendor

Yes PMO to monitor MA HIX Application Vendor to ensure that this is completed.

Testing Unit of Contractor to evaluate results

Massachusetts Health Insurance Exchange Program, Project Management and Testing Services Request for Responses 11

For the avoidance of doubt, all Testing Types described in this Appendix C are subject to all related and applicable terms and conditions set forth in

Attachment A, Model Contract

Name Description Primary Responsible

Party

Contractor Responsibility

(Yes/No)

Notes/Commentary

Vulnerability Testing Testing to evaluate risks in the software to reduce the probability of security vulnerabilities