ATTM 021.pdf

Document number: PGM.PLAN.0010 Version: 1.0

Delivery of Automated Systems for Healthcare

(DASH) Program

Master Test Plan

ATTM 021

PGM.PLAN.0010 – Master Test Plan Ver.1.0

Table of Contents

1. INTRODUCTION

1.1. PURPOSE

1.2. SCOPE

1.2.1. In Test Scope

1.2.2. Out of Test Scope

1.3. ASSUMPTIONS AND CONSTRAINTS

1.3.1. Assumptions

1.3.2. Constraints

1.4. STAKEHOLDERS

1.5. GLOSSARY

1.6. REFERENCES

2. TESTING OVERVIEW

3. TEST ACTIVITIES

3.1. PLAN

3.1.1. Plan Test Phase

3.1.2. Plan Test Cases

3.2. SCRIPT

3.2.1. Test Case Workflow

3.2.2. DevOps Test Case Attributes

3.3. PREPARE DATA

3.3.1. Manage Interface Data

3.4. EXECUTE

4. TEST PHASES

4.1. UNIT TEST

4.2. SYSTEM INTEGRATION TEST (SIT)

4.3. STATE TEST (STT)

4.4. END-TO-END INTEGRATION TEST (E2E)

4.5. SECURITY TEST (SEC)

4.6. PERFORMANCE TEST (PRF)

4.7. USER ACCEPTANCE TEST (UAT)

4.8. TEST PHASE SUMMARY

5. TEST ENVIRONMENTS

5.1. TEST ENVIRONMENT MANAGEMENT

6. TEST TOOLS

7. DEFECT MANAGEMENT

7.1. DEFECT LIFECYCLE

7.2. DEFECT ATTRIBUTES

7.3. CREATE DEFECTS

7.4. REVIEW DEFECTS

7.5. TRIAGE DEFECTS

8. ROLES AND RESPONSIBILITIES

9. TEST METRICS AND REPORTS

10. GOVERNANCE AND PLAN MAINTENANCE

APPENDIX A TEST ESTIMATION MODEL

APPENDIX B TEST STRATEGY MEETING TEMPLATE

APPENDIX C TEST CASE MATRIX TEMPLATE

APPENDIX D TEST CASE TEMPLATE

APPENDIX E DEFECT TEMPLATE

APPENDIX F TEST DASHBOARD

APPENDIX G TEST SUMMARY REPORT TEMPLATE

1. Introduction

1.1. Purpose

This plan describes the methodology and processes used to plan, manage, and execute testing for the DASH Program.

1.2. Scope

The scope of this plan encompasses the test strategy, processes, and resources required to support SCDHHS and Vendor application software testing.

1.2.1. In Test Scope

Test items within the validation scope of this plan include, but are not limited to, the following:

• Approved requirements o State o Federal

• Design documentation o Functional Design o Technical Design o Interface Control Document o Use Cases

• Application components o Web-based functionality

• Screens Screen functionality, validation, error messages, etc.

Help functionality Browser compatibility

• Forms

• Reports

• Automated Processes/Batch

• Interfaces o Mobile functionality

• Mobile platform Android/iOS o Interactive Voice Response (IVR) software o Role-based security and user access

• System performance/Service Level Agreements (SLAs)

• Accessibility/Americans with Disabilities Act (ADA) Section 508 compliance

• Application/Conversion data

1.2.2. Out of Test Scope

Items excluded from the validation scope of this plan include:

• Operational procedures

• Help Desk procedures

• Disaster recovery procedures

• Training deliverables

1.3. Assumptions and Constraints

1.3.1. Assumptions

• The test guidance in this plan applies to SCDHHS and Vendor test phases, activities, and output with the intention of standardizing testing at the enterprise level; Vendors must provide written requests to SCDHHS to alter the processes or templates defined by this plan

• Test entrance and exit criteria is enforced across test phases

• Each test is conducted in a restorable test environment suited to the purpose of the test as agreed upon by SCDHHS and the Vendor

• Data is available per the requested volume and quality including converted data for each test phase

• Test data management adheres to SCDHHS and Vendor security protocols including masking or de-identification based on the test environment

• The Vendor provides timely and accurate release notes for new/modified application components proposed for deployment to a SCDHHS test environment; The SCDHHS Test Manager or designee must approve the release of code, configuration, or data changes prior to deployment in any SCDHHS test environment

• SCDHHS testers may request/modify system date parameters as needed to simulate data processing over time

• Cross-team testing dependencies and milestones are documented in the project schedule in accordance with PGM.PLAN.0007 - Schedule Management Plan

• Adequate resources are allocated to support testing within the allotted time frame as agreed upon by SCDHHS and the Vendor

• The project schedule includes sufficient time for defect retest and regression test

• Regression testing is automated where feasible

• Test-related risks are logged and managed in accordance with PGM.PLAN.0003 -

Risk Management Plan

• Changes to the scope of project requirements are managed in accordance with

PGM.PLAN.0006 - Change Management Plan

• SCDHHS assists the Vendor in coordinating testing activities with Interface

Partners

• Each Interface Partner provides a knowledgeable resource and backup resource to support coordinated testing activities including review of test cases, test data/files, and test results; Each Interface Partner participates in defect triage, resolution, and retesting

• Each Interface Partner defines and performs Unit/SIT/Performance Test for each of their interface components and performs necessary regression testing

1.3.2. Constraints

• Automated testing is limited to stable application components without Severity 1 defects unless approved by SCDHHS

• Coordinated interface testing is conducted based upon the readiness and availability of the Interface Partners

• Licensing limitations may exist for specialized test tools related to performance, accessibility, automation, etc.; SCDHHS will work with the Vendor to ensure appropriate test oversight is supported

1.4. Stakeholders

Test stakeholders include members of the following teams:

• SCDHHS/DASH Program

• Vendor

• Interface Partners

1.5. Glossary

Term Definition Accessibility Testing

Testing performed to validate compliance with ADA/Section 508 compliance for Americans with Disabilities Act (ADA) requirements; Utilizes specialized tools such as screen readers, website evaluators, color/contrast analyzers, etc.

Defect Defects are logged when actual results do not match expected results during testing.

Defects are reviewed during triage meetings and a root cause is subsequently determined by the Vendor and/or SCDHHS; Documentation defects may be logged for design-related issues; Defects may also be referred to as Bugs

Document Defect

Defect logged based on incorrect, insufficient, or conflicting design documentation

End-To-End Integration Test (E2E)

This test phase is performed by the Vendor to validate transactions that span the full breadth and depth of the application perform correctly

Enhancement A change that is deemed by SCDHHS and the Vendor to expand agreed upon application capabilities beyond the approved design; Enhancements are managed via the project change control process

Interactive Voice Response

(IVR)

Automated telephone-based technology that supports caller self-service features such as inquiries, gathering information, and call routing

Microsoft Azure DevOps (DevOps)

DASH project repository used to support SDLC management of requirements, testing, defects, and traceability

Office of Cybersecurity

(OCS)

This group performs oversight of security governance and enforcement initiatives for

SCDHHS

Parallel Testing

Analysis of functionality/output in legacy application compared to the equivalent functionality/output in the proposed application; This test activity may be included within a test phase such as E2E or UAT

Performance Test (PRF)

This test phase is performed by the Vendor to validate and benchmark application behavior and conformance with the approved technical requirements and SLAs

Regression Testing

Test activity to confirm previously tested functionality has not been negatively impacted by new/modified code associated with application releases and defect corrections

SCDHHS Test Team

Group comprised of DASH Test Manager, DASH Test Lead, DASH testers, and SCDHHS staff such as Program Area reviewers/UAT testers, technical staff, etc.

Security Test

(SEC)

This test phase is performed by the Vendor to validate that the application conforms with the approved security protocols and requirements; SCDHHS OCS provides oversight

Service Level Agreements (SLAs)

Documented commitment or expectations for Vendor/application responsiveness, throughput, quality, availability, etc.

Smoke Test A test activity comprised of high-level application testing for each release to confirm successful deployment to the test environment

State Test

(STT)

This test phase is performed by SCDHHS to validate that the application conforms with the approved requirements and designs; State Test usually lags Vendor SIT activities by one release or iteration

Subject Matter Experts (SMEs)

SCDHHS and Vendor personnel identified as primary contacts for business and/or technical aspects of the project

System Integration Test (SIT)

This test phase is performed by the Vendor to validate that the application conforms with the approved requirements and designs

Test Case A series of step-by-step manual instructions to confirm the application functions per the requirements and designs; Test cases are subject to configuration management; Term may be used synonymously with test script

Test Case Matrix

Excel-based listing of test inputs/variables that define proposed test cases; Test cases defined in the matrix are also commonly referred to as test scenarios

Test Phase Phases include Unit Test, System Integration Test (SIT), State Test (STT), End-to-End Integration Test (E2E), Security (SEC), Performance (PRF), and User Acceptance Test (UAT)

Test Script Term commonly used for a series of step-by-step automated instructions to validate application functionality; Automation is usually reserved for core/stable functionality based on the required effort to develop automation; Term may be used synonymously with test case

Test Suite A collection of test cases or test scripts User Acceptance Test (UAT)

This test phase is performed by SCDHHS to validate application compliance with the approved requirements and designs and confirm deployment readiness

Unit Test This test phase is performed by Vendor development staff for new/updated application components

1.6. References

This section references external documents that provide background or supplemental information considered valuable to the intended audience.

[1] PGM.PLAN.0001 – Project Management Plan

[2] PGM.PLAN.0002 – Requirements Management Plan

[3] PGM.PLAN.0006 – Change Management Plan

[4] PGM.PLAN.0007 – Schedule Management Plan

[5] PGM.PROC.0002 – Documentation Management Process

2. Testing Overview The DASH Program test approach is based on the following key concepts:

• Multiple test phases that build upon one another to comprehensively test SCDHHS applications

• Use of manual and automated testing supported by detailed test cases/scripts

• Bi-directional traceability that spans the SDLC

• Peer review of test deliverables and work products

• Timely and collaborative triage and resolution of defects

• Targeted regression testing for application changes/defect corrections

• Availability of real-time test dashboards and metrics

This plan defines the following software test phases and the corresponding SCDHHS and Vendor responsibilities to conduct or support each phase:

• Unit Test

• System Integration Test (SIT)

• State Test (STT)

• End-To-End Integration Test (E2E)

• Security Test (SEC)

• Performance Test (PRF)

• User Acceptance Test (UAT)

Each test phase has a defined scope and level of detail intended to contribute to a comprehensive validation of the application as illustrated in Figure 1 - Testing V-Model.

Figure 1 - Testing V-Model

The Testing V-Model depicts the evolution of major SDLC outputs and the corresponding test phases. Requirements receive final validation during UAT by SCDHHS. Use cases, designs, components, and relevant requirements are tested by SCDHHS and/or the Vendor during multiple integration test phases including SIT, State Test, E2E, Security, and Performance. Initial testing of application components is conducted by the Vendor during Unit Test. Test phases may lag or overlap one another based on completion of the defined entrance and exit criteria.

SCDHHS defines each test phase as being comprised of the following major test activities:

• Plan

• Script

• Prepare Data

• Execute

Each activity includes tasks common to each test phase and as illustrated in Figure 2 - Test Process Overview.

Figure 2 - Test Process Overview

Additional phase-specific tasks are addressed in the subsequent sections of this document.

Test planning requires two distinct activities. Test phase-level planning is typically performed by the DASH or Vendor Test Manager for each test phase with the output reflected in the project schedule. Test case-level planning is performed by testers with support from test stakeholders to define test scope and identify test cases.

Subsequent test activities to script, prepare data, and execute testing are performed by SCDHHS for State Test and UAT or the Vendor for the remaining test phases.

3. Test Activities SCDHHS will collaborate with each Vendor to define required test activities and mutually agree on test planning and estimation methods to support development of a test schedule.

Joint planning sessions will occur to finalize and incorporate to the project schedule.

The following sections describe each test activity in further detail.

3.1. Plan

The test planning effort requires high-level planning to manage coordination of the overall test phase and low-level planning to scope and define test cases/scripts. These activities may be performed concurrently.

3.1.1. Plan Test Phase

The SCDHHS and Vendor Test Team managers plan each test phase with assistance from their respective teams by completing the following steps:

Step Task Comments

Schedule Develop the overall test timeline based on the test scope and anticipated resources

Refer to Appendix A Test Estimation Model or provide an alternate Vendor model as approved by

SCDHHS

Project schedule maintained in SharePoint

Step Task Comments

Participants Identify SCDHHS/Vendor Test Team and other stakeholders

May include other SCDHHS/Vendor stakeholders or other Interface Partners to participate in test planning, document review, test execution, and defect triage

Training Assess need for test participant training on application and test processes/tools

Manage training activities and timeframes in project schedule

Work Items Create and assign test work items in DevOps

Test work items are used to manage test case assignments and provide test case development status

Test Strategy Meeting

Conduct joint SCDHHS/Vendor Test Strategy meeting

Address the following:

• Previous Test Phase Results

• Test Overview and Schedule/Cycles

• Test Scope: In/Out

• Test Contacts

• Test Phase Entrance Criteria

• Testing Tools/Process/Training

• Planned Test Cases

• Test Environments

• Test Data

• Defect Management

• Test Reporting and Deliverables

Refer to Appendix B Test Strategy Meeting Template for additional information

Communicate Publish test schedule and communicate with participants

This includes conducting test meetings, walkthroughs, defect triage, etc.

3.1.2. Plan Test Cases

SCDHHS/Vendor Test Team members participate in planning efforts for their respective test cases. Test case planning includes the following steps:

Review Review test input including requirements, designs, use cases, supporting documents, etc.

Evaluate:

• Any existing test cases

• Open defects

• Listing of Business Processes as prioritized by

SCDHHS Subject Matter Experts (SMEs) by factors including business criticality, volume, functional complexity, and technical complexity

Step Task Comments

Scope/Plan Test Cases

Identify planned test scope and test cases needed using Test Case Matrix

Evaluate:

• Most common success/failure paths

• Exception-based scenarios that may result in success or failure

• Scenarios to evaluate maximum/minimum allowable data/field lengths

• Boundary testing scenarios to assess maximum/minimum values

• Interface files/data

Refer to Appendix C Test Case Matrix Template for additional details

Review Planned Test Cases

Confirm full coverage for test scope and priorities with Test Team and test stakeholders

Review Test Case Matrix

Assess test case priority on a four-point scale with one being the highest

Assign Test Case Work Items

Assign work items to each tester

DevOps workflow is used to track test case development status as Not Started, In Progress, and Completed

3.2. Script

A tester translates each scenario defined in the Test Case Matrix to develop a detailed test case that specifies each test action and expected result. Test cases are developed using a standard template/naming and subject to configuration management. Each test case adheres to a scripting convention to preview the test data, perform the test action, and confirm the resulting test data. Positive and negative testing actions are incorporated where applicable.

Testers trace each test case to the relevant requirements, designs, use cases, and components during the scripting process as illustrated in Figure 3 - Test Case Traceability.

Figure 3 - Test Case Traceability

Additional requirements traceability details are defined in PGM.PLAN.0002 – Requirements Management Plan.

Peer review is conducted by Test Team members to confirm adherence to the checklist defined in the test case template including the following items:

• Template usage/naming standards

• Accurate traceability

• Scripting standards for consistency, level of detail, and conciseness

• Test scope/completeness

• Test case quality (free of formatting/typographical errors)

Scripting includes the following steps:

Develop Test Case/Script

Tester develops test case using template per the assignments in the Test Case Matrix

Refer to Appendix D Test Case Template scripting tab

Author completes test case checklist and closes test case work item

Trace Test Case Identify each requirement, design, use case, and component;

Upload test case to SharePoint repository

Refer to Appendix D Test Case Template traceability tab

Peer Review Reviewer completes peer review checklist

Refer to Appendix D Test Case Template review checklist tab

Import Test Case Test Admin imports test case to test tool

DevOps test dashboard indicates test case is available in Design state for SCDHHS test cases

3.2.1. Test Case Workflow

Test cases/scripts are developed using the test tool appropriate to the test phase or activity. Figure 4 - Test Case Workflow illustrates the states supported by DevOps for test case development and execution.

Figure 4 - Test Case Workflow

Vendor may propose alternate tools for specific test phases as approved by SCDHHS.

Alternate tools and supporting test processes must be documented as mutually agreed upon with SCDHHS and the Vendor.

3.2.2. DevOps Test Case Attributes

The test case attributes available in DevOps are described in the following table.

These attributes are used to support test case development, execution, and status reporting.

Attribute Definition Values Required

ID DevOps identifier for test case Number Auto Work Item Type

DevOps designation for test case Test Case Yes

Title Test case name formatted per project-specific naming convention Example: EVV TC0001 IVR Personal Care Check in V1

Text Yes

Test Step DevOps identifier for test condition Number Auto Step Action Test action to perform Text Yes Step Expected Expected results Text Yes Area Path DevOps mapping to test project Text Yes Assigned To DevOps username Active Directory Yes State (Test Case Development)

Default value for new test case Design Yes

Active/available to execute Ready Inactive/retired Closed

State (Test Case Execution)

Not yet executed Not Run Yes Execution temporarily halted by tester Paused Not applicable for test phase/cycle N/A Successfully executed with zero defects or only Severity 4 defects

Passed

Executed with Severity 1, 2 and/or 3 defects identified

Failed

Unable to execute pending active defect resolution

Blocked

Converted Data Test case requires or utilizes converted data

Yes No No

Phase Name/number of project phase Text Yes Reviewed By Peer reviewer Active Directory Yes – at time of review

Test Phase Vendor System Integration Test SIT Yes SCDHHS Integration Test STATE Vendor End-to-End Integration Test E2E SCDHHS User Acceptance Test UAT SCDHHS Pilot (as applicable) PILOT Regression testing for Production defects

PROD

Attribute Definition Values Required

Cycle Test cycle number 1 No

Regression Indicator that test case exercises core functionality required during retest

Yes Yes No

Functional Area High-level business or functional area that provides high-level context as defined by SCDHHS

Claims Yes Constituent Services

EDI

Finance Interfaces Member Prior Authorization Provider Reference Reporting None

Application Type

Identifies software type Web No Mobile

IVR

Priority Rank of relative test case importance with 1 being highest; Default value is 2

1 Yes

Description Brief summary of test case purpose (one to two sentences)

Text Yes

Prerequisites Numbered list of pre-conditions for test execution

Text Yes

Overview Numbered list of major test case actions Text Yes Role Numbered list of user roles required for test execution Text Yes

Execution Comments

Captures tester comments related to test runs

Text No

Outcome Value manually set for test case work item based on final test case run

Passed No Failed Blocked

Tags Admin-defined grouping label Text No

3.3. Prepare Data

Test data is generated using a variety of methods. Each test phase has a prescribed level of detail and a required level of data realism to assess application functionality. Each Test Team will identify test data based upon the test phase needs and test case specifics.

Testing may be performed using converted data when conversion is included in project scope. Converted data should be available to support SIT and State Test and all subsequent test phases.

Test data is generated via the following means:

• Application data entry

• Converted data

• Vendor or third-party provided data/files

• Master/reference data

Unmasked, converted Production data must be used in accordance with defined SCDHHS security procedures/protocols.

3.3.1. Manage Interface Data

Test data for interfaces will be developed and managed according to the following:

• A request will be communicated to the Interface Partner for test data and test support

• Partner-provided data files will be used for new or modified interfaces

• When partner-provided data files are not available, the Vendor will assist

SCDHHS as necessary

• Specific interface data requirements will be discussed during the Test

Strategy meeting

• Requests for interface test files will be coordinated with the Functional and

Test Team managers for the respective applications/test environments Data preparation for each test phase includes the following steps:

Review Data Needs

Review requirements, design, and use cases to identify data needs and sources

Evaluate user roles and privileges, file layouts, and business rules

Stakeholder Data Meeting

Meet with business owners, Vendor, and Interface Partners for a data overview and discussion

Evaluate test environment, Vendor capabilities, and security requirements for test data

Develop Test Data Listing

Develop detailed list of required test data based on defined test cases

Testers compile and maintain listing of volume and nature of data needed; List managed in SharePoint

Develop/Request Data

Create data using application as feasible; Request data from Vendor/Interface Partner(s)

Request via agreed upon communication method

Validate Data and Application Readiness

Confirm data received satisfies request and supports testing needs

Confirm Vendor/Interface Partner(s) are ready to support test execution and application components were successfully tested in lower test phases

Communicate Provide updates to Vendor/Interface Partner(s) Stakeholders

Includes conducting test meetings, defect triage, status updates, etc.

3.4. Execute

Test execution for each test phase is managed using a published checklist to ensure required tasks are completed and occur in the proper sequence. The scope of the checklist includes activities related to the following:

• Test planning activities related to schedule, resources, test cycles, etc.

• Test case development

• Test data management including required data/files and use of converted data

• Test environment readiness including code deployment/demonstration, user profiles, connectivity for Interface Partners, and Smoke Test

• Batch schedule/parameters required for test execution

• Test execution and defect management

At the conclusion of testing for each test phase, a Test Summary Report is produced that summarizes test purpose, content, testing method/tools, results achieved, and defects identified.

Test execution for each test phase includes the following steps:

Step Task Comments

Develop Execution Checklist

Use execution checklist template Refer to SharePoint template

Execute Test Cases

Execute using DevOps or other tool as agreed upon

DevOps dashboard provides real-time test status

Manage Defects Log defects via DevOps

Conduct triage with Stakeholders

DevOps dashboard provides real-time defect status

Resolve Defects and Perform Regression

Re-execute test cases associated with defect corrections; Regression testing must be performed in lower test phases/environments

Refer to Release Management Plan for additional detail

Generate Test Summary Report

Prepare report summarizing test phase Refer to Appendix G Test Summary Report Template for additional details

4. Test Phases Software testing is conducted in multiple phases aligned with design and development activities. Each test phase has a defined scope and level of detail and is conducted in a pre-determined environment. Multiple test phases may be conducted within the same environment based on the needed data volume/quality. Each test phase is planned, managed, and executed by the Vendor or SCDHHS as defined in the project schedule.

4.1. Unit Test

Unit Test is the initial phase of testing performed by the Vendor. This test is highly localized with the objective of testing a new or modified software component in isolation from other components. Unit Test provides validation that each application component is operating as designed. Developers perform Unit Test in conjunction with coding activities per the Vendor development methodology and tools. Vendors are required to submit a Development Methodology Document that addresses Development and Unit Test capabilities.

4.2. System Integration Test (SIT)

System Integration Test (SIT) is conducted by the Vendor Test Team after successful completion of Unit Test. This test verifies integrated system functionality, as defined by the requirements, designs, and use cases, is addressed by the sprint/release. SIT scope includes testing of web-based screens, forms, reports, batch programs, interfaces, mobile/IVR functionality, etc. Accessibility testing for Section 508 compliance should occur during SIT. Results from accessibility testing are submitted via the Accessibility Test Summary Report separately from the SIT Summary Report.

The primary function of SIT is to provide testing and defect discovery to confirm readiness for State Test execution by SCDHHS. As each sprint/release is developed, the scope of SIT will increase based on the newly available functionality and regression testing of previous functionality.

SIT is intended to validate the application’s ability to operate as expected with other modules and may require coordination with third parties across multiple applications.

Considerations for SIT execution include test case sequencing, potential date manipulation/advancement, and batch scheduling. Obvious performance-related issues such as slow response time for pages, searches, report generation, batch execution, etc.

should be identified as defects during SIT. Formal performance testing is completed in a subsequent test phase.

SIT will occur when SIT entrance criteria have been met.

System Integration Test Summary

Goal Test application components within a full or partial business function from a functional point of view using realistic data to simulate end-user activity

Input • Application release including new/updated components and interfaces

• State and Federal requirements

• Design documents

• Use cases

• Prior test phase defects

Output • Completed SIT test cases/scripts

• SIT defects

• SIT Summary Report

Entrance Criteria

1. Conduct SIT Test Strategy meeting with test stakeholders

2. Communicate test schedule as defined in project schedule; Must address defect correction/retest

3. Communicate approved defect management process; Confirm no open

Severity 1 defects exist; Remaining defects have mutually agreed upon target resolution dates

4. Confirm configuration management process for project documentation changes is approved

5. Confirm release management process is approved

6. Confirm Unit testing is complete and results documented as agreed upon

7. Confirm SIT test cases/scripts exist in test tool as agreed upon

8. Confirm SIT test cases have been traced to the approved requirements/designs/components and verify full test coverage

9. Confirm test environment is ready, including all necessary applications, configuration, interface connectivity, and batch

10. Conduct successful Smoke Test

11. Confirm test data is ready including test files/interface data in accordance with defined procedures; Reference data/tables are populated and required converted data is available

12. Confirm third parties are prepared to exchange needed files/extracts required for validation

Exit Criteria 1. Execute planned SIT test cases; Results are available in testing tool

2. Document SIT pass rate

3. Assign any deferred test cases/scripts to a subsequent test phase

4. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS;

Remaining defects have mutually agreed upon target resolution dates

5. Complete documentation updates or capture as SIT defects

6. Publish SIT Summary Report and obtain SCDHHS approval

Concurrent Test Phases

None

Environment Vendor SIT Tools • Vendor test tools as agreed upon

• Accessibility tools as agreed upon Test Data As necessary to address full test scope including converted and Interface

Partner data Vendor Role Plan, manage, and execute SIT SCDHHS Role Perform SIT oversight activities Considerations Test Coverage: Each requirement within the sprint/release is tested in one or more test cases related to a design/use case and component Targeted Cycles: As defined in the schedule for each sprint/release Test Suspension Test Resumption

Smoke Test failure SIT resumes after successful Smoke Test

Risks Mitigation

Unit Test/code release completed late

Modify SIT schedule or include in next release

Test environment is unavailable

Modify or extend schedule

4.3. State Test (STT)

State Test is conducted by the SCDHHS Test Team after successful completion of SIT and/or each SIT sprint/release. This test verifies application functionality performs as defined by the requirements, designs, and use cases. This includes the testing of web-based screens, forms, reports, batch programs, interfaces, mobile/IVR functionality, etc.

The Vendor will coordinate with SCDHHS to complete any necessary system training before State Test begins. The Vendor will support test data and credential preparation as needed for SCDHHS testers.

SCDHHS may perform limited 508 testing during this Test Phase.

State Test Summary

Goal Test single or multiple, related components within a full or partial business function from a functional point of view using realistic data to simulate end-user activity

Input • Application release including new/updated components

• State and Federal requirements

• Design documents

• Use cases

• Prior test phase defects

Output • Completed State Test cases/scripts

• State Test defects

• SCDHHS State Test Summary Report

Entrance Criteria

1. Conduct State Test Strategy meeting with test stakeholders

2. Communicate test schedule as defined in project schedule; Must address defect correction/retest

3. Communicate approved defect management process; Confirm no open Severity

1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

4. Confirm configuration management process for project documentation changes is approved

5. Confirm release management process is approved; Release Notes format and frequency is mutually agreed upon

6. Validate testing user credentials/profiles are ready

7. Confirm approved State test cases/scripts exist in DevOps

8. Confirm State test cases have been traced to the approved requirements/designs/components and verify full test coverage

9. Confirm test environment is ready, including all necessary applications, configuration, interface connectivity, and batch

10. Conduct successful Smoke Test

11. Confirm test data is ready including test files/interface data in accordance with defined procedures; Reference data/tables are populated and required converted data is available

12. Confirm third parties are prepared to exchange needed files/extracts required for validation

Exit Criteria 1. Execute planned State test cases; Results are available in testing tool

2. Document State Test pass rate

3. Assign any deferred test cases/scripts to a subsequent test phase

4. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS;

Remaining defects have mutually agreed upon target resolution dates

5. Publish State Test Summary Report

Concurrent Test Phases

None

Environment State Test Tools • Excel – Used to create test cases

• DevOps – Used to manage and document test cases execution and defects

• Applicable Accessibility tools

Test Data As necessary to address full test scope including converted and Interface Partner data Vendor Role Support SCDHHS testing activities and deliver defect corrections

State Test Summary

SCDHHS Role Plan, manage, and execute State Test Considerations Test Coverage: Each requirement within the sprint/release is tested in one or more test cases related to a design/use case and component Targeted Cycles: As defined in the schedule for each sprint/release Test Suspension Test Resumption

Smoke Test failure State Test resumes after successful Smoke Test

Risks Mitigation

SIT/code release completed late Modify State Test schedule or include in next release

Test environment is unavailable Modify or extend schedule

4.4. End-to-End Integration Test (E2E)

End-to-End Integration Test (E2E) will concentrate on high-level and critical end-to-end business scenarios that will demonstrate the capabilities of the system to support vital business activities. The scope of E2E testing includes:

• Full system functionality including interfaces

• Full scope of user profiles and privileges

• Comprehensive coverage of converted and production data as feasible

• Parallel testing of legacy transactions to confirm accurate application processing/output as agreed upon by SCDHHS and Vendor SCDHHS and the Vendor will collaboratively identify test scenarios for E2E testing. E2E test cases may be created or derived from existing test cases. Test cases for E2E Testing will be managed in the testing tool as agreed upon.

E2E testing will occur when exit criteria have been met for SIT and entrance criteria for E2E have been met. For E2E testing, the application including all interfaces must be functional to allow testing of the complete system lifecycle.

End-to-End Integration Test Summary

Goal Test business transactions that span the application including interfaces from a functional point of view; The scope of E2E testing includes validating secondary capabilities and downstream functions such as payment processing, reporting, etc.

Input • Fully coded and available application, interfaces, and master/reference data

• State and Federal requirements

• Design documents

• Use cases

• Prior test phase defects

• Converted and Production data as feasible

• Interface files and connectivity

• User profiles

Output • Completed E2E Test cases/scripts

• E2E defects

• E2E Test Summary Report

End-to-End Integration Test Summary

Entrance Criteria

1. Conduct E2E Test Strategy meeting with test stakeholders

2. Communicate test schedule as defined in project schedule; Must address defect correction/retest

3. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

4. Confirm release management process is approved

5. Confirm configuration management process for project documentation changes is approved

6. Confirm successful completion of SIT phase and SIT Summary Report has been approved

7. Confirm E2E Test cases/scripts exist in test tool as agreed upon

8. Confirm E2E Test cases have been traced to the approved requirements/components and SCDHHS approves full E2E test coverage

9. Confirm test environment is ready, including all necessary applications, configuration, interface connectivity, and batch

10. Conduct a Successful Smoke Test of full application

11. Confirm test data is ready including test files/interface data in accordance with defined procedures; Reference data/tables are populated and required converted data is available

12. Confirm third parties are prepared to exchange needed files/extracts required for validation

Exit Criteria 1. Execute planned E2E Test cases/scripts; Results are available in testing tool

2. Document E2E Test Pass rate

3. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

4. Complete documentation updates or capture as E2E defects

5. Publish E2E Test Summary Report and obtain SCDHHS approval

Concurrent Test Phases

Security and Performance

Environment As agreed upon Tools Vendor test tools as agreed upon Test Data A large volume of production-quality data is created/obtained utilizing converted and

Interface Partner provided data as available; Any automated batch stream is executed in full

Vendor Role Plan, manage, and execute E2E SCDHHS Role Collaborate in E2E planning

Collaborate with Vendor to validate E2E results Considerations Test Coverage: Proposed application end-to-end

Targeted Cycles: As defined in the schedule Test Suspension Resumption

Smoke Test failure E2E resumes after successful Smoke Test

As necessary based on volume or nature of defects identified

At the discretion of the Vendor Test Manager

Risks Mitigation Application code is late Modify schedule and/or resources Test environment is unavailable Modify schedule and/or resources New/changed requirements

• Delay testing of component until new code received

• Create/modify test cases as necessary

• Modify schedule if necessary

End-to-End Integration Test Summary

Interface Partners not ready/available for test

Delay test of that interface or proceed with simulated data

Turnaround time for code corrections Adjust testing schedule for impacted component

4.5. Security Test (SEC)

The purpose of Security Test is to identify and mitigate security vulnerabilities in the delivered application. SCDHHS will collaborate with each Vendor as necessary to define the level and scope of technical/vulnerability security testing. The SCDHHS OCS Group will participate in planning and oversight of Security Testing.

Application security access testing confirms the application has been properly configured and correctly enforces role-based access to screens, reports, etc. Application security testing must be addressed during SIT, State Test, E2E, and UAT test phases at a minimum.

4.6. Performance Test (PRF)

Performance Test is executed to evaluate technical application behavior and capabilities. The following performance test objectives must be addressed:

Test Objective

Load Determine application behavior under both normal and anticipated peak load conditions in terms of users and transactions to gather system baseline metrics

Stress Analyze system performance and behavior under conditions that overload its resources and cause the system to break/fail

The Vendor must collaborate with SCDHHS to identify the number and nature of business transactions and processing volumes defined for Performance Test.

Performance testing must include relevant requirements/SLAs, front-end processing response for screens, searches, transactions, reports, etc. as mutually agreed upon and back-end processing transactions such as batches, interfaces, etc.

The scope of Performance Test includes the following:

• Identify and define the Performance Test transactions

• Identify the metrics that will be collected in order to verify against Performance objectives

• Identify Production or Production-equivalent environment

• Define Performance Test cases and required data to simulate the load/stress conditions

• Analyze the metrics/data captured during the tests

• Document performance issue bottlenecks and identify corrective action At the completion of performance testing the Vendor will provide a scorecard of Performance metrics as measured against benchmarks such as:

• Screen Performance

• Response vs. Time Summary

• Transaction Throughput

• Server Health Summary

• System Health Summary

Performance Test Summary

Goal Test application at typical and peak levels of concurrent users and transactions to define performance benchmarks and identify underperforming areas and bottlenecks; Stress the application to the point of failure to confirm application recovery

Input • Fully coded and available system application and master/reference data

• Converted data

• Complete batch schedule

• Network and server configuration and parameters

• Performance-related requirements

Output • Completed Performance Test cases/scripts

• System performance benchmarks for online and batch transactions

• Performance Test defects

• Performance Test Summary Report

Entrance Criteria

1. Define Performance Test transactions in collaboration with SCDHHS and Vendor

2. Confirm SCDHHS has approved Vendor transaction/data/user volume estimates

3. Conduct Performance Test Strategy meeting with test stakeholders

4. Communicate test schedule as defined in project schedule; Must address defect correction/retest

5. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

6. Confirm successful completion of SIT; E2E/Security Test in progress or complete

7. Confirm performance-related requirements, design documents/components are approved and traced to Performance Test cases/plans

8. Confirm Performance Test cases/scripts have been developed within the

Performance test tool

9. Confirm test environment is ready, including all necessary applications, configuration, interface connectivity, and batch

10. Confirm Performance Test data is ready including test files/interface data in accordance with defined security procedures; Reference data/tables are populated and required converted data is available

Exit Criteria 1. Execute planned Performance Test cases/scripts; Results are available in testing tool

2. Document Performance Test Pass rate

3. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

4. Confirm system performance benchmarks for online and batch transactions comply with performance requirements/SLAs

5. Publish Performance Test Summary Report

Concurrent Test Phases

E2E and Security

Performance Test Summary

Environment Production-equivalent Tools Performance tools as agreed upon Test Data A large volume of Production-quality test data is available including converted and interface data as available Vendor Role Plan, manage, and execute Performance Test State Role Collaborate in Performance Test planning

Collaborate with Vendor to validate Performance Test results Considerations Test Coverage: Full application within Production-equivalent environment

Targeted Cycles: As defined in the schedule Test Suspension Resumption

Smoke Test failure PRF resumes after successful Smoke Test As necessary based on volume or nature of defects identified

At the discretion of the Vendor Test Manager/Technical Manager

Risks Mitigation Application code is late Modify schedule and/or resources Test environment is unavailable Modify schedule and/or resources

• Delay testing of component until new code received

• Create/modify scripts as necessary

• Modify schedule if necessary

Interface Partners not ready/available for test

Delay test of that interface or proceed with simulated data

Turnaround time for code corrections Adjust testing schedule for impacted component

4.7. User Acceptance Test (UAT)

User Acceptance Test is planned and executed by SCDHHS to confirm the application functions per the approved requirements, designs, and use cases. UAT participants undergo Vendor-led application training prior to test execution. UAT results are a key factor to support a favorable Deployment Go/No Go decision.

UAT Test Summary

Goal Exercise the full scope of the application with emphasis on key processes/transactions; Validate State/Federal requirements and CMS KPIs

Input • Fully coded application and master/reference data

• SCDHHS UAT Plan

• State and Federal requirements

• Design documents

• Use cases

• Converted data

• Complete batch schedule/programs

• Network and server configuration and parameters

• System performance benchmarks for online and batch transactions

• Parallel testing transactions as applicable

Output • Completed UAT Test cases/scripts

• UAT defects including Vendor system documentation design/training updates

• SCDHHS UAT Test Summary Report

UAT Test Summary

Entrance Criteria

1. Approve E2E Summary Report; Validate coverage and traceability

2. Conduct UAT Test Strategy meeting with test stakeholders

3. Communicate test schedule as defined in project schedule; Must include periodic planned releases for defect corrections

4. Confirm release management process is approved; Release Notes format and frequency is mutually agreed upon

5. Communicate approved configuration management process for project documentation changes

6. Communicate approved defect management process; Confirm no open Severity

1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

7. Configure DevOps and any supporting testing tools including access permissions for UAT stakeholders

8. Confirm DevOps UAT test cases are ready; Validate full coverage for approved requirements and designs/components

9. Confirm UAT environment is ready, including any supporting applications, configuration, interface connectivity, and batch

10. Conduct a successful UAT Smoke Test

11. Confirm test data is ready including test files/interface data in accordance with defined security procedures; Reference data/tables are populated and required converted data is available

12. Confirm UAT participant training is complete

13. Validate testing user credentials/profiles are ready

14. Confirm third parties are prepared to exchange UAT files/extracts

15. Confirm UAT support resources are available; Help desk is staffed and operational Exit Criteria 1. Execute planned UAT test cases; Results are available in testing tool

2. Document UAT pass rate

3. Confirm no open Severity 1 or 2 defects unless approved by SCDHHS; Remaining defects have mutually agreed upon target resolution dates

4. Confirm workarounds are documented for Severity 3 and 4 defects and are accepted by Business Owners

5. Complete documentation updates or capture as UAT defects

6. Confirm training materials, e-Learning modules, and online help are complete and approved

7. Publish UAT Test Summary Report

Concurrent Test Phases

None

Environment UAT Tools • Excel – Used to create test cases

• DevOps – Used to manage and document test cases execution and defects

• Performance tools as needed

Test Data • A large volume of data is required including converted and interface data in accordance with defined security procedures

• Full set of defined security roles/privileges SCDHHS Role • Plan, execute, and manage UAT

• Develop UAT test cases

• Train UAT participants on SCDHHS UAT process and tools

• Facilitate UAT defect triage

• Assess overall UAT results as input to Go/No Go decision

Vendor Role • Support UAT planning and execution

• Provide application training for SCDHHS UAT participants

UAT Test Summary

• Provide UAT test data including converted data and interface data as specified by

SCDHHS

• Support batch and interface processing to mirror Production

• Troubleshoot and resolve UAT defects

• Perform regression testing for UAT defects in lower test environments including

Unit, SIT, E2E, and PRF; Update applicable system documentation Considerations Test Coverage:

• Full application functionality

• Performance test cases as applicable Targeted Cycles: 2-3 cycles as defined by SCDHHS Test Suspension Resumption

Smoke Test Failure UAT resumes successful Smoke Test Initiated by SCDHHS as necessary based on volume or nature of testing defects identified

At the discretion of the SCDHHS Test Manager

Risks Mitigation E2E Test completed late Modify UAT schedule and/or resources Test environment is unavailable Modify schedule and/or resources Business owner/testers unavailable

Identify backup resource personnel

• Delay testing of component until new code received

• Create/modify test cases as necessary

• Modify schedule if necessary

4.8. Test Phase Summary

Phase Goal Input Output Entrance

Criteria Exit

Criteria Unit Test

Vendor developer executes low level testing on system component

Requirements, Use Case/Design, Unit Test Plan

Successfully committed code to Unit/Dev

Completed development object

Successful Unit, Peer review

SIT Vendor functionally tests single/related modules within a full/partial system function; Validate Accessibility/508 compliance

Application release, Requirements, Use Cases/Designs, Components

Completed test cases/scripts, SIT defects, SIT Test Summary Report

Successful Unit & Smoke Test

Successful

SIT,

No open Level 1 Defects, Test Summary Report

STT SCDHHS tests system components within a full or partial business function from a business point of view using realistic data, Validate Accessibility/508 compliance

Application release, Requirements, Use Cases/Designs, Components

Completed test cases/scripts, State Test defects, State Test Summary Report

Successful SIT & Smoke Test

State Test, No open Level 1 or 2 Defects, Test Summary Report

E2E Vendor tests business transactions that span system including testing of interfaces and security roles and privileges

Fully coded application, Master/Reference data, Converted data, Requirements, Use Cases/Designs, Components,…