PL 008.pdf

SCDHHS Data Classification: Confidential Distribution

MMRP System Design Document v8.2

South Carolina Department of Health and Human Services (SCDHHS)

Member Management Replacement Program

System Design Document

Version 9.0

Last Revised: May 26, 2021

Prepared by:

South Carolina Department of Health and Human Services (SCDHHS)

Clemson Health Care Information Services (HCIS)

Clemson Medicaid IT Services (MITS)

IBM

PL 008

SCDHHS Data Classification: Confidential Distribution Page 2 of 67

MMRP System Design Document v9.0

Table of Contents

1. Revision History Review History

2. Review and Approval

3. Introduction

3.1 Purpose of the SDD

4. General Overview and Design Considerations

4.1 General Overview

4.2 Assumptions / Constraints / Risks

4.2.1 Assumptions

4.2.2 Constraints

4.2.3 Risks

4.3 Alignment with the Federal Enterprise Architecture

5. Design Considerations

5.1 Goals and Guidelines

5.2 Development Methods & Contingencies

5.3 Architectural Strategies

5.4 Performance Engineering

5.5 Test Methodology

5.5.1 Load Testing

5.5.2 Stress Testing

5.5.3 Endurance Testing

5.5.4 Performance Testing

6. Operational Scenarios

6.1 Pre-Screening

6.2 Application Intake

6.3 Eligibility Determination

6.4 Caseload Management

6.5 Documentation

6.6 Change of Circumstance (CoC)

6.7 Annual Review

6.8 Reporting and Business Intelligence

7. System Architecture and Architecture Design

7.1 Logical View

7.2 Hardware Architecture

7.2.1 Security Hardware Architecture

7.2.2 Performance Hardware Architecture

7.3 Software Architecture

7.3.1 Security Software Architecture

SCDHHS Data Classification: Confidential Distribution Page 3 of 67

7.3.2 Performance Software Architecture

7.4 Information Architecture

7.4.1 Records Management

7.5 Internal Communications Architecture

7.6 Security Architecture

7.7 System Architecture

8. System Design

8.1 Business Requirements

8.1.1 Connectivity

8.2 Database Design

8.2.1 Security and Privacy

8.2.2 Data Objects and Resultant Data Structures

8.2.3 File and Database Structures

8.2.4 Database Management System Files

8.2.5 Non-Database Management System Files

8.3 Data Conversion

8.4 User Machine-Readable Interface

8.4.1 Inputs

8.4.2 Outputs

8.5 User Interface Design

8.5.1 Section 508 Compliance

9. Detailed Design

9.1 Hardware Detailed Design

9.2 Software Detailed Design

9.3 Security Detailed Design

9.3.1 Logging and Auditing

9.4 Performance Detailed Design

10. System Integrity Controls

11. External Interfaces

11.1 Interface Architecture

12. Acronyms

13. Glossary

14. Appendices

SCDHHS Data Classification: Confidential Distribution Page 4 of 67

1. Revision History

Date Author Rev Notes

6/18/13 John Supra 1.0 Initial version for DDR review

10/23/14 Stasey Tackett 2.0 Revised for FDDR review

3/4/16 Rudy Rodriguez 3.1 Revised for FDDR review

3/29/16 Stasey Tackett 4.0 Finalized version 4.0

6/16/17 Stasey Tackett 4.1 IBM, HCIS, and Clemson Security updates

6/29/17 Stasey Tackett 4.2 O&M updates

7/6/17 Stasey Tackett 4.2 IBM Internal review edits

8/08/17 Stasey Tackett 4.3 Address business feedback, O&M input

8/23/17 Stasey Tackett 5.0 Finalize version name to 5.0 based on business approval

11/15/2017 Lisa Carlyle 5.1 Updated to add link to access info in

Appendices

6/29/2018 Stasey Tackett 5.2

IBM MMRP Quarterly review and update; updates for Performance

Testing and Architecture and

Database Design

7/18/2018 Stasey Tackett 5.3

Updates to address comments from

SCDHHS MMRP Technical and

PMO staff

SCDHHS Data Classification: Confidential Distribution Page 5 of 67

7/20/2018 Stasey Tackett 5.4 Updates to address IBM MMRS

O&M final review and approval

4/30/2019 Eric Mittnight

Mahmoud Al Taher 5.5

Updates to address CGIS upgrade to

MMRS, to include updated diagrams

5/08/2019 Eric Mittnight

Mahmoud Al Taher 5.6

Updated to address CGIS interfaces and additional non-database management system files

5/20/2019 Erin Coleman 6.0

Updated version number and date to reflect approval, accepted all track changes, and removed comments.

8/30/2019 Eric Mittnight

Mahmoud Al Taher 6.1

Updated to address Database

Design, Data Dictionary, External

Interfaces, included updated diagrams, and incorporated Clemson updates

9/20/2019 Mahmoud Al Taher 6.2

Updates to address comments from

SCDHHS Technical staff and MMRP staff

10/02/2019 Mahmoud Al Taher 6.3 Updates to address comments from

SCDHHS Technical staff

11/14/2019 Erin Coleman 7.0

Updated version number and date to reflect approval, accepted all track changes, and removed comments.

11/04/2020 Mahmoud Al Taher 7.1

Updated to address CGIS release changes, Data Dictionary, Database

Configuration and incorporated

Clemson updates

1/6/2021 Mahmoud Al Taher 7.2 Updates to address comments from

SCDHHS Technical staff

SCDHHS Data Classification: Confidential Distribution Page 6 of 67

2/3/2021 Erin Coleman 8.0

Updated version number and date to reflect approval, accepted all track changes, and removed comments.

4/19/2021 Mahmoud Al Taher

Vijay Mujumdar 8.1

Updated external interface table, MMS System architecture diagram, and MMS Non-Database

Management System Files table

5/6/2021 Mahmoud Al Taher 8.2 Updates to address comments from

SCDHHS Technical staff

5/26/2021 Erin Coleman 9.0

Updated version number and date to reflect approval, accepted all track changes, and removed comments.

Review History

Version Date Author(s) Comments

5.5 4/1/2019 Erin Coleman Reviewed and added comments

5.5 4/11/2019 Renjan Bhasi Reviewed and added comments.

5.6 5/9/2019 Erin Coleman Reviewed and closed addressed comments.

5.6 5/11/2019 Tim Cozine Reviewed

6.1 9/3/2019 Erin Coleman Reviewed and added comments

6.1 9/16/2019 Renjan Bhasi Reviewed and added comments.

6.2 9/20/2019 Erin Coleman Reviewed updates and closed comments.

6.2 9/26/2019 Renjan Bhasi Reviewed, closed some comments, added some follow up queries.

7.1 11/5/2020 Erin Coleman Reviewed updates

SCDHHS Data Classification: Confidential Distribution Page 7 of 67

Version Date Author(s) Comments

7.1 11/13/202

Tim Cozine Reviewed.

7.2 1/11/2020 Erin Coleman Reviewed and responded to comment.

8.1 4/20/2021 Erin Coleman Reviewed

8.1 4/27/2021 Tim Cozine Reviewed

2. Review and Approval

The undersigned acknowledge that they have reviewed this System Design Document and agree with the information presented. Changes to this document are coordinated with and approved by the undersigned or their designated representatives:

Name Title Approval Date

Rhonda Morrison Member Management Replacement Program (MMRP) Director, SCDHHS

5/26/2021

TBD (delegate Jason Dale)

Deputy Chief Information Officer (CIO), Information Management, SCDHHS

5/19/2021

Thomas Taylor MITS Executive Director, IT Services, Clemson University

5/25/2021

Jeff Leyh MMRP Operations and Maintenance Manager, IBM

5/25/2021

3. Introduction

This System Design Document (SDD) describes the technical design supporting the South

Carolina Department of Health and Human Services (SCDHHS) implementation of the

Member Management Replacement System (MMRS). In order to reflect the future state of

MMRS as the sole eligibility determination system for SCDHHS, the name of the system has been updated to Member Management System (MMS). SCDHHS established the

SCDHHS Data Classification: Confidential Distribution Page 8 of 67

Member Management Replacement Program (MMRP) organization to provide the program and project management support required for the MMS implementation efforts.

MMS is a web-based system providing a portal for consumers, community-based counselors, state workers and agents. It helps users determine an individual’s eligibility for

Medicaid, facilitates enrollment in Managed Care Organizations (MCOs), and interfaces with the Federally-Facilitated Marketplace (FFM) and the Centers for Medicare and

Medicaid Services (CMS).

The goal of MMS is to simplify eligibility and enrollment, making Medicaid more accessible to South Carolinians, while replacing the Legacy system known as Medicaid Eligibility

Determination System (MEDS), and meeting the latest requirements of the Patient

Protection and Affordable Care Act (PPACA or ACA).

Currently, MMS provides data to the MEDS for any South Carolina citizens determined eligible for Medicaid. The data exchange format mirrors the current data format required for

MEDS. MEDS then feeds the Medicaid Management Information System (MMIS) current eligibility data to facilitate downstream claims operations and payment.

MMS currently leverages the IBM Cúram Social Program Management (SPM) product as its core solution to implement the Medicaid eligibility and enrollment system for the Modified

Adjusted Gross Income (MAGI)-based coverage groups as part of an ongoing SCDHHS modernization initiative. IBM’s Cúram SPM modules supports the lifecycles of an eligibility system, with the Health Care Reform (HCR) being the module specifically developed to support ACA functions.

In 2018, SCDHHS initiated a new project to expand MMS to support non-MAGI based

Medicaid programs, based on the IBM Cúram SPM Global Income Support (CGIS) module.

The CGIS project will enable SCDHHS to conduct eligibility determination, program enrollment, and comprehensive verification and reporting for non-MAGI Medicaid applicants and beneficiaries.

The CGIS enhancements are to be accompanied by a newly enhanced Universal Access module to provide an integrated application experience for the citizens of South Carolina across all Medicaid benefit delivery programs included in the scope for MMS. With CGIS, SCDHHS gains an improved user experience for the current user population in both the

MMS Citizen Portal and Worker Portal.

With the CGIS project, SCDHHS acquires an integrated system, built upon the existing

MMS framework, to begin to migrate SCDHHS off the legacy MEDS mainframe system.

Overall, the CGIS project will seamlessly integrate with the existing MAGI functionality currently enabled in MMS, integrate with the SCDHHS Medicaid Enterprise Architecture including MMIS services through the existing MMS Enterprise Service Bus (ESB), integrate

SCDHHS Data Classification: Confidential Distribution Page 9 of 67 with the SCDHHS content management platform OnBase® and meet federal and state regulations pertaining to Medicaid eligibility.

3.1 Purpose of the SDD

The SDD documents and tracks the necessary information required to effectively define architecture and system design of the MMS. The SDD is updated in compliance with the

SCDHHS Project Management Office (PMO) Consolidated Deliverable Management List

(CDML) requirements, which have been adapted to meet the needs of the MMRP CGIS project.

4. General Overview and Design

Considerations

4.1 General Overview

MMS supports SCDHHS in its initiatives to modernize and transform Medicaid eligibility operations while also meeting the State’s obligations regarding the ACA, primarily with interaction with the FFM and CMS. MMS is compliant with the CMS Medicaid Information

Technology Architecture (MITA).

The driving force behind MMS has been a modernization of business processes that drive

Medicaid eligibility, moving away from a caseworker-centric model to a contemporary, web interface model. This modernization enables the citizens of the State significantly greater access to the application and enrollment process.

MMS utilizes several connections to state and federal interfaces for the purpose of evidence

(income, citizenship, lawful presence, etc.) verification. MMS also connects to the Federal

Data Services Hub (FDSH) to support near real-time verification of member evidence, potentially allowing for near real-time eligibility determinations for members.

MMS serves as the System of Record (SoR) for SCDHHS in managing Member applications for Medicaid services, rendering eligibility determinations, enrollment of eligible members in particular program categories, subsequent redeterminations of Member eligibility, and all associated ancillary support activities. For all cases converted from MEDS to MMS as part of the CGIS effort, the SoR for all converted members is to be changed from MEDS to MMS.

MMS has improved the eligibility and enrollment process by:

• Featuring a transparent, understandable, easy-to-use online process that enables citizens to apply and enroll in the South Carolina Medicaid Program

• Adding additional non-MAGI-based Medicaid programs as part of the CGIS project

SCDHHS Data Classification: Confidential Distribution Page 10 of 67

• Accommodating a full range of user capabilities, languages, and access considerations

• Offering seamless integration between multiple systems and alleviating the need for citizens to know in advance for which programs they may qualify.

• Providing strong privacy and security protections

For Medicaid MAGI Programs, Cúram is an eligibility and case management solution that helps SCDHHS to address ACA requirements and meet CMS critical success factors. MMS meets the State of South Carolina’s obligations for interactions with the FFM.

The deployment of the Medicaid non-MAGI eligibility components will integrate with the

MMS Cúram HCR module and include the following functional areas. Please refer to

Section 6: Operational Scenarios for functional area details, to include:

• CGIS Program Rules for Application Intake, Change of Circumstances and

Renewals;

• CGIS Evidence for Application Intake, Change of Circumstances and Renewals;

• CGIS Verifications and Data Validations for Application Intake, Change of

Circumstances and Renewals;

• CGIS Program Alignment for Application Intake, Change of Circumstances and

Renewals;

• CGIS Case Maintenance & System Processing for Application Intake, Change of

Circumstances and Renewals;

• CGIS Tasks and Work Assignment for Application Intake, Change of Circumstances and Renewals;

• CGIS Intelligent Evidence Gathering (IEG)/Script/Wizards for Application Intake, Change of Circumstances and Renewals;

• CGIS Notices, Reports, and Appeals

Key components in the State’s enterprise include:

• The Cúram Application – utilizing IBM’s WebSphere Application Server (WAS) and the Oracle database

• MuleESB – to facilitate communications with the FDSH

• OnBase – document management system

• Oracle Managed File Transfer – provides end-to-end secure transfer of files at the

SCDHHS Enterprise level

The diagram below is an illustration of the MMS boundary, updated to include SCDHHS

Enterprise Platform Services applications to be deployed as part of the CGIS project. These applications include Oracle Managed File Transfer (MFT) and PathOS.

SCDHHS Data Classification: Confidential Distribution Page 11 of 67

Figure 1 MMS Boundary

SCDHHS Data Classification: Confidential Distribution Page 12 of 67

SCDHHS is utilizing Clemson University’s data center in Anderson, South Carolina as its primary hosting location.

The State’s operations team is comprised of SCDHHS and IBM Design, Development &

Implementation (DDI) teams, both located in Columbia, SC. MMRP employs the Agile

Scrum Sprint methodology for MMS development and implementation.

MMS utilizes existing connections to state and federal interfaces for evidence verification

(i.e., income, citizenship, lawful presence, etc.). MMS relies on the FDSH to support real-time verification of member evidence, potentially allowing for near real-time eligibility determinations for members. The CGIS project is adding a number of new interfaces with

CMS and Social Security Administration (SSA) services which are detailed in Section 11 below.

MMS integrates with the FFM, allowing it to receive member applications that have been assessed by the FFM as Medicaid eligible. Transmission of the applicant’s record releases the applicant from having to create an additional application. MMS also transmits member applications to the FFM.

Cúram supports two different groups of users: citizens and caseworkers. These are consumers of the Cúram Citizen Portal and the Cúram Caseworker Portal respectively.

The Cúram Citizen Portal (https://apply.scdhhs.gov/), used for screening and application process, is located at SCDHHS’ official web site, Healthy Connections

(https://www.scdhhs.gov/), and is accessible by any South Carolina citizen with internet access.

The Cúram Caseworker Portal is used by state and county staff to manage and support applications for benefits. The Caseworker Portal can only be accessed through the State’s internal network with a valid SCDHHS User ID.

4.2 Assumptions / Constraints / Risks

4.2.1 Assumptions

For the State of South Carolina to meet its objectives for releases, the following assumptions were made. The MMS design depends upon the following:

The ability of the IBM Cúram HCR module to provide the core functional capabilities including Medicaid single-streamlined application and connections/interactions with the

FDSH (lookup and account transfer).

IBM and SCDHHS collaborating on functionality for IBM Cúram HCR module to meet

CMS’ basic requirements.

Ability to work with Clemson Health Care Infrastructure Services (HCIS) to deploy new infrastructure.

https://www.scdhhs.gov/

SCDHHS Data Classification: Confidential Distribution Page 13 of 67

The IBM Cúram SPM framework operates as expected within the new infrastructure.

The technical infrastructure provided to the MMRP team meets all security requirements applicable to the storage, retrieval and transmission of PHI data.

HCIS manages the maintenance and licensing of the hardware and operating system of all servers.

The licensing required for the MMRP team, including Oracle, WAS, and others, are made available by SCDHHS when requested, and approved as appropriate.

4.2.2 Constraints

MMS is currently hosted and managed in a data center that offers primarily virtualized services. VMWare products are used for individual servers and NetApp products are used for storage arrays.

Data management for MMS is constrained by the fact that PHI data must be handled and protected in a specifically regulated manner. All data management plans must comply with applicable security policies when PHI is involved.

MMS data management is further constrained by format requirements of the commercial product IBM Cúram, which is the central application MMS.

MMS must comply with all legally mandated data retention policies, and as a result, experiences data storage constraints to retain all necessary data for a predefined number of years.

4.2.3 Risks

The Out-of-the-Box (OOTB) Cúram SPM data model stores a significant amount of

Extensible Markup Language (XML) data in the Binary Large Object (BLOB) and

Character Large Object (CLOB) fields directly in the database. This design decision has greatly hindered efforts to automatically mask data, as well as generate test data at scale.

Refer to the SCDHHS MMRP Risk Register located in the MMRP Project Web

Application (PWA) SharePoint site.

4.3 Alignment with the Federal Enterprise Architecture

The Cúram (MMS) Service-Oriented Architecture (SOA) is compatible with the Federal

Enterprise Architecture (FEA) and enables the Federal government to expand the volume of claims processed, easily adapt to changing legislation and economic conditions, and eliminate the need for costly technology overhauls.

MMS is compliant with the Medicaid Information Technology Architecture (MITA) Version

4.0 framework and standards. MITA has adopted the standards promulgated by the FEA, and the Federal Health Architecture (FHA), among other standards, to promote interoperability, flexibility and the use of common standards (i.e., SOA).

https://schhs.sharepoint.com/sites/pwa/MMRS%20CGIS%20(Curam%20Global%20Income%20Support)/Lists/Risks/AllItems.aspx

SCDHHS Data Classification: Confidential Distribution Page 14 of 67

Overall, MMS incorporates MITA-aligned performance focused processes and technology that is aligned with the seven conditions of enhanced funding: Modularity, MITA, Industry

Standards, Leverage, Business Results, Reporting and Interoperability.

5. Design Considerations

5.1 Goals and Guidelines

The design decisions and architectural strategies for MMS were driven by recognition that certain architecture selections were needed to drive the South Carolina Enterprise

Architecture – including its MEDS replacement strategy.

The design decisions and resulting architecture represent a significant change and associated improvement to the technologies used by the State of South Carolina, requiring significantly more attention to design, potential for re-use, and analysis on the overall impact to the enterprise.

5.2 Development Methods & Contingencies

The MMS design continues to be centered on the IBM Cúram SPM framework – with a goal to maintain and continuously improve upon the end user experience for South Carolina citizens. The MMS design process is a collaborative process between the SCDHHS and

IBM DDI teams, SCDHHS Program staff, and SCDHHS Eligibility staff. The decision points regarding customization vs. business process changes often arise during the Joint

Application Design (JAD) sessions and would be approved as part of SCDHHS design approvals.

The project adopted two key approaches that continue to drive the development methodology. These key strategies are consistent with modern agile approaches to software development and system implementation:

• Configuration over customization: When practical, the IBM MMRP DDI team proposes adjusting the business process to accommodate how Cúram OOTB was designed to work, versus requiring customization to meet an SCDHHS business process.

• Focusing on a Minimally Viable Product (MVP): Develop, implement, and build the leanest product possible, while meeting the State’s requirements.

To drive the appropriate amount of documentation and ensure rapid development cycles, the project adopted an approach that focuses on the following:

• SCDHHS Business Analysts (BAs) developing Business Process Models using business process modeling notation

SCDHHS Data Classification: Confidential Distribution Page 15 of 67

• SCDHHS BAs developing Business Requirements Documents (BRD) containing business rules and requirements required to implement the Business Process

Models

• IBM DDI BAs developing Functional Design Specification (FDS) documents, containing functional requirements to meet the business rules/requirements

• IBM BAs providing FDS documents for Developers and Quality Assurance (QA) team members

• IBM DDI BAs creating User Stories that connect to specific business requirements, which are leveraged by IBM Development and QA Teams for development/testing efforts

• MMRP Team implementing by Sprint (a series of one or more User Stories)

• MMRP Team organizing into Sprint teams

• IBM DDI providing Technical Design Specification (TDS) documents for final development structure and historical changes to system components.

For the CGIS project, all FDS and TDS documentation reside in integrated Architectural

Description Documents (ADD) which are maintained on the SCDHHS MMRP

SharePoint site. For the HCR project, the FDS and TDS documents are stand-alone documents.

5.3 Architectural Strategies

The MMS architecture is designed to meet the following goals:

1. Architecture that is multi-tier and service-based

a. Utilizes multi-tier architecture that includes presentation layer (web), application/service layer, and data layer (databases)

b. Provides horizontal scalability at each tier

2. Software and hardware architecture that is horizontally scalable based on the following:

a. Clustering technology

b. Load balancing to distribute load across multiple services/servers

c. Service-based to separate load among multiple services

d. Operations/infrastructure based on virtual machines (VMs)

e. Implementation that includes high-availability and/or redundancy at each level

3. Use of COTS technologies when practical

a. COTS application framework, ESB, and databases

b. Standardization of service design for re-use

The following applications are in use for MMS.

SCDHHS Data Classification: Confidential Distribution Page 16 of 67

Application Purpose

Cúram Social Program

Management

Cúram SPM provides core of Medicaid eligibility functionality, connections to FDSH and Federal Exchange

WebSphere Application

Server (WAS)

Java Virtual Machine (JVM) container, utilized with WebSphere

ND to support session clustering; JVMs are used for the Cúram framework operation

Oracle Database Server Primary database; Cúram framework operates within Oracle database

Mule Enterprise Service Bus

(MuleESB)

MuleESB for interoperability between Cúram framework and the

FDSH, Medicaid Management Information System (MMIS) and other State systems

Atlassian Confluence Knowledge base for the MMRP

Atlassian Jira Toolset for development processes, including issue tracking, requirements (known as “user stories”), business cases, test cases, development and traceability, test management and automated testing

Eclipse Development

Environment

Developer end-user integrated development environment.

Selected for integration with Cúram framework and Java development

Hyland OnBase Enterprise Document Management System; integrates with

Cúram framework

Oracle MFT Managed File Transfer (MFT) solution used to provide end-to-end secure transfer of files at the SCDHHS Enterprise level

Oracle Linux (OL) Currently in use in production for SCDHHS MMS. For use with all

Java-based applications – Cúram, ESB, Oracle

Microsoft Windows Server Currently in use in production for SCDHHS MMS. For use with

Windows applications and OnBase

Microsoft SQL Server Currently in use in production for SCDHHS MMIS solutions and was initial selection for OnBase database

Microsoft Active Directory Currently in use in production SCDHHS solutions

SCDHHS Data Classification: Confidential Distribution Page 17 of 67

Application Purpose

Rivest-Shamir-Adleman

(RSA) Secure ID

Software and hard tokens for multi-factor authentication

Cúram XML Server Part of the Cúram framework for processing of XML files in and out of the Cúram framework

WildFly Application server for FDSH web services; communicates with

MuleESB console

IBM HTTP Load-balancing proxy server; a pass-through to WebSphere

Cúram application server

Table 1 MMS Applications

5.4 Performance Engineering

MMS employs a balanced approach, leveraging COTS to jump-start the functional aspects of the solution while establishing a comprehensive and sustainable SOA foundation to facilitate interoperability. This foundation is designed to:

• Maximize reuse

• Minimize data replication

• Provide necessary security and privacy

This architecture also enhances the capability of the system to adapt to changes in technology while delivering the agility to support the business needs. MMS application is scalable and built on the concepts of SOA. The SOA environment allows loosely coupled applications to communicate seamlessly.

5.5 Test Methodology

5.5.1 Load Testing

The purpose of load testing is to investigate how a system or component performs under various load factors. Load tests are run at peak-hour volume – highest volume of transactions that a system can execute in one hour without failures or performance degradation. This defined volume is also known as a “1x or 100% load” or “peak load.” The system is tested at various load levels. Performance issues found during load testing are considered for remediation.

5.5.2 Stress Testing

The purpose of stress testing is to investigate how far beyond the 100% load a system or component can operate under before it shows signs of deteriorated performance. The

SCDHHS Data Classification: Confidential Distribution Page 18 of 67 system is tested at increasingly higher loads until the system shows degradation of performance or the load achieved without degradation is high enough for foreseeable needs of the business. Performance issues found during load testing may be considered for remediation immediately or may be deferred to a later release.

5.5.3 Endurance Testing

The purpose of endurance testing is to investigate how well a system or component operates under a constant high load for a longer period (such as annual enrollment when the system needs to be able to handle a higher load for multiple hours.) Endurance testing is used to find issues, such as slow memory leaks, that may not show during other types of testing.

5.5.4 Performance Testing

The purpose of performance testing is to ensure the system can successfully create, process and modify applications in both Citizen and Worker portal environments.

Detailed monitoring of all aspects of the system is required so that the performance of the system can be analyzed after the test.

The test team recommends the following tools be installed on the target environment:

• Nigel's performance Monitor (nmon) monitoring suite on all Linux servers

• Oracle Enterprise Manager (OEM) for database monitoring

• WAS Performance Tuning Toolkit (PTT)

• Splunk

Logs and charts for each test run are collected and analyzed. MMS uses Jira to track defects.

The MMRP Load and Performance Test Plan used by the test team is stored in Confluence.

6. Operational Scenarios

The key functional areas of the operational production system include the following:

6.1 Pre-Screening

The pre-screening calculator is used to identify potential SCDHHS programs for which an individual may qualify. The pre-screening calculator is accessible through the Citizen Portal and uses a series of questions to assess potential eligibility through IBM Cúram’s Intelligent

Evidence Gathering (IEG) process.

SCDHHS Data Classification: Confidential Distribution Page 19 of 67

6.2 Application Intake

Applications may be submitted through a variety of channels including FFM, online via the

Citizen Portal, in person, by mail, drop off, fax, or telephone. Applications entered by a worker through the Caseworker Portal uses IEG to gather all information necessary to determine eligibility. Certain evidence gathered, such as citizenship, requires verification.

Evidence verifications are gathered via external data interfaces as well as paper documentation. Once all required evidence is verified, the system applies verified data against program-specific rules to determine eligibility for the individuals on an application requesting benefits.

6.3 Eligibility Determination

The system applies rules to a household’s evidence to determine eligibility, providing specific eligibility determination for each specific program for which the individual has requested benefits. MMS evaluates the evidence and applies program specific rules to determine eligibility. There are different sets of eligibility rules for the MAGI Medicaid and non-MAGI Medicaid programs, per Federal and State of South Carolina policies and requirements.

6.4 Caseload Management

The system’s Workflow Management component was configured to meet SCDHHS business needs. The main functions that are utilized to manage workload are listed below:

• Work Effort Requests – One of the functions of the Workflow Management system is to generate work effort requests, or “tasks.” Tasks are used to assign and track the work of system users, for example, Eligibility Workers. Tasks have deadlines and priorities, which allow users to improve time management and balance a busy workload. By focusing on those tasks with highest priority, Workers can complete their work by prescribed deadlines and improve performance metrics, such as timeliness ratings.

• Work Queues – Tasks or work effort requests are sent to the inbox of specific individuals, roles or jobs, or assigned to work queues. A work queue is a subscription-based list that can maintain a list of related work effort requests from which a user can pull tasks that need to be performed. For example, submission of online applications by clients may trigger a task into an Online Application Work

Queue if they require manual intervention. A “Get-Next” feature is provided to avoid workers selectively choosing the tasks on which to work, and to help SCDHHS manage tasks based on prioritization criteria when necessary. SCDHHS utilizes

Work Queues to effectively tackle tasks as a team so that no one staff member is overburdened, and critical tasks can be worked immediately.

• Supervisor Functionality – Supervisors manage work queues through a Work

Queue Workspace. The Workspace provides summaries of work effort requests

SCDHHS Data Classification: Confidential Distribution Page 20 of 67 associated with their staff or organizational unit(s) as well as capabilities to drill down into details so that Supervisors can be actively engaged in employee productivity and proactively prevent bottlenecks or address concerns.

6.5 Documentation

The system leverages the State of South Carolina’s existing state-based interfaces as well as the FDSH for electronic documentation of evidence requiring verification to determine eligibility. Paper documentation is requested for any required verifications that cannot be satisfied through external databases and can be submitted through a variety of channels including in person, mail, drop off, or fax. Eligibility determinations require a large amount of documentation and managing that documentation is an integral part of the eligibility determination process.

6.6 Change of Circumstance (CoC)

Client changes can enter the system from multiple channels:

• Self-reported by the client via the Citizen Portal

• Self-reported via a paper change form

• Verbally communicated directly to the Intake or Eligibility Worker

• Received from verified data sources

Once the change is received, SCDHHS policies dictate whether the MMS automatically applies the change based on verification from external data systems, whether the change requires verification and if so, what type – or if the change requires an Eligibility Worker’s review. Once the change is applied, the system automatically reassess eligibility for those programs and eligibility periods impacted by the change and generates notifications to the

Client. Workers can then view the history of changes within case records.

6.7 Annual Review

The MMS utilizes the IBM Cúram renewal and recertification processes to complete the annual review determination process. External data sources are compared against client eligibility data to determine if the information is reasonably compatible. If the information is reasonably compatible as determined by comparison of data provided to external data systems, the annual review is processed, and a notice is sent to the client regarding the resulting eligibility determination.

If the information is not reasonably compatible or there is not sufficient information through external data sources to re-determine eligibility, a pre-populated annual review form is sent to the client requesting any changes be identified and reported within a specific time frame.

The MMS is updated based on reported changes once all required verifications are provided. The changes are applied, and eligibility is reassessed by the system and a notice is sent to the participant regarding any changes to eligibility.

SCDHHS Data Classification: Confidential Distribution Page 21 of 67

6.8 Reporting and Business Intelligence

Powerful reporting and analytics are provided by Crystal Reports, leveraging built-in

Cúram–based functionality that supports ongoing State business processes and workflows, and provides an integrated and flexible reporting structure to meet the needs of the State.

CMS-required reports as well as specific reports based on State business needs are generated based on identified time requirements. Ad hoc reports can also be generated based on business need such as legislative inquiries.

7. System Architecture and Architecture

Design

Cúram’s framework is an n-tier service-oriented architecture product that uses a robust rules engine to determine eligibility. It supports Web Service standards required by the Web

Service Interoperability (WS-I) Basic Protocol and Basic Security Protocol, Hypertext

Transfer Protocol (HTTP), Simple Object Access Protocol (SOAP), Web Service Definition

Language (WSDL), XML Schema, and Web Services (WS)-Security. Additionally, it provides support to business process modeling standards for interoperability via WS-

Business Process Execution Language (BPEL). The Cúram COTS product supports the full generation of inbound and outbound web services.

To ensure data management and interoperability within the solution, the MMS solution includes internal architectural components including the MMS database. The purpose of this database is to serve as a storage location for data that is not stored elsewhere in the

COTS products.

7.1 Logical View

The South Carolina MMS architecture design provides a three-tier architecture that is based on the Cúram framework, the use of the OnBase document management system, and the

ESB. These major components interact to deliver the required functionality for MMS. The diagram below provides a logical overview of the MMS:

SCDHHS Data Classification: Confidential Distribution Page 22 of 67

Figure 2 MMS Logical Design

7.2 Hardware Architecture

The MMS infrastructure is housed at the primary data center in Anderson, South Carolina.

The secondary data center is in Columbia, South Carolina. Each site hosts one virtual environment – known as “POD.”

When not operating as production, the Secondary Site hosts a production cloud in passive mode that receives updates from the Primary Site.

Each site is fully capable of operating the anticipated production load.

The Columbia data center is the Disaster Recovery site.

Please see Appendix 1 for MMS Primary and Secondary Site diagram.

7.2.1 Security Hardware Architecture

SCDHHS equipment is housed in secured enclosures separate from other data center and client infrastructure. All physical equipment is locked and secured with access limited to trained and approved individuals. Physical access is logged, tracked, and auditable at any time.

SCDHHS Data Classification: Confidential Distribution Page 23 of 67

SCDHHS servers take connectivity only from data center edge routers/switches (edge devices) in a redundant, High Availability (HA) manner. End-point encryption terminates within the SCDHHS environment.

Clemson’s Office of Information and Privacy (OISP) employs the following rules as they relate to perimeter devices, including firewalls and routers:

1. User log-on (successful and unsuccessful) and log-off functions are logged and sent to Splunk log correlation servers in the SCDHHS POD

2. All system administration activities performed on network devices are logged and sent to Splunk log correlation servers in the SCDHHS POD

3. Account provisioning is logged and sent to Splunk log correlation servers in the

SCDHHS POD

4. System shutdown and reboot tasks are logged and sent to Splunk log correlation servers in the SCDHHS POD

5. System errors on network devices are captured and sent to Splunk log correlation servers in the SCDHHS POD

6. Modification of proxy services such as front-end web services are captured on the F5

Big-IP and sent to Splunk log correlation servers in the SCDHHS POD

The SCDHHS-managed Active Directory (AD) is the authoritative source for all MMS user accounts. Users who require privileged accounts are granted access via “SUDO” (meaning

“Superuser DO”) group membership. All SUDO commands are logged and sent to Splunk where the data is retained.

Users requesting privileged rights must receive authorization from the data owner or the designee for privileged use, after which they are assigned a role with a defined business case that warrants the privileged use. After having demonstrated a bona fide business case for the access, a user submits the request through the FootPrints ticketing system, where it is routed for the required approvals.

Database administrator roles are the only privileged users authorized to administer the database management systems.

Users attempting to execute privileged commands must invoke the SUDO command, which requires authentication to validate the privileged role.

Only authorized users are permitted access to MMS-related files, directories, drives, servers, network shares, ports, protocols, and services required to complete job duties.

Access to MMS information is always authorized by a designated data owner of such information and is limited to a restricted subset of people per role-based access. Users are granted minimal file system access until escalated access is approved.

SCDHHS Data Classification: Confidential Distribution Page 24 of 67

Administrative access requests must have additional justification for the elevated access and management and/or project director approval. All approvals are documented within the

FootPrints ticketing system.

The integrity of all transmitted information is protected using encryption. Web tiers employ secure Hypertext Transfer Protocol Secure (HTTPS), using Secure Sockets Layer (SSL) certificates. Internet Protocol Security (IPSec), VPN, SSL, and Web Services are used to externally transmit data.

Virtual Local Area Networks (VLANs) are employed on the internal Local Area Network

(LAN) for traffic isolation. The F5 Big-IP appliances use FIPS 140-2 compliant cryptographic mechanisms. All web services are secured via Transport Layer Security (TLS). No data is sent in the clear over an unsecured network.

The following events are logged and sent to the Splunk log correlation tool:

1. Server alerts and error messages are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

2. Failed and successful logons are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

3. Successful logoffs are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

4. Password changes occurring on MMS servers are captured and sent to the

Splunk log correlation server in the SCDHHS POD.

5. Audit records for the use of administrative privileges are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

6. Audit records for the use of SUDO (Linux) and RUNAS (Windows) are captured for the MMS servers and sent to Splunk log correlation servers in the SCDHHS

POD.

7. Audit records related to the creation/modification of super-user groups such as the SUDOers file is captured and sent to Splunk log correlation servers in the

SCDHHS POD.

8. Security-related functions performed by the security admin account are captured and sent to Splunk log correlation servers in the SCDHHS POD.

9. System administrator commands are performed by invoking the SUDO command while logged on as a standard user. These administrative commands are captured and sent to Splunk log correlation servers in the SCDHHS POD.

10. All commands issued that modify or clear an audit log file are captured and sent to Splunk log correlation servers in the SCDHHS POD.

11. System startup and shutdown audit logs are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

SCDHHS Data Classification: Confidential Distribution Page 25 of 67

12. Use of identification and authentication mechanisms for Cúram administrative and organizational access are captured and sent to Splunk log correlation servers in the SCDHHS POD.

13. Change of file or user permissions or privileges are captured and sent to Splunk log correlation servers in the SCDHHS POD.

14. Audit logs related to remote access connections such as VPN and VDI are captured and sent to Splunk log correlation servers in the SCDHHS POD.

15. Cúram batch jobs create changes to both the application and the database.

Batch servers produce their own logs via Automic. Process monitoring displays detailed information about the tasks being monitored, including status, success and failure of the batches, along with batch history.

16. Within Cúram, personal details (social security number, addresses, medical history, income, etc.) can be changed or corrected.

17. If AD makes user account changes (account creation, security role, activation, deactivation, etc.) to its system, AD pushes those changes to Cúram via a web service called SyncCúramFromLDAP, and these changes are automatically updated in Cúram without manual intervention.

18. Failed and successful logons are captured for the MMS applications and servers.

The logs are sent to Splunk log correlation servers in the SCDHHS POD.

19. System shutdown and reboot audit logs are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

20. System error audit logs are captured for the MMS servers. The logs are sent to

Splunk log correlation servers in the SCDHHS POD.

21. All Cúram application errors route to the Splunk Security Information and Event

Management (SIEM) tool. They are also captured in Cúram’s application server, WebSphere, where the administrator can learn more about root cause. History of errors is available for auditing purposes.

22. Cúram restarts route to the Splunk SIEM. They are also captured in Cúram’s application server, WebSphere, where a history is maintained.

23. Cúram application errors are captured in Cúram’s application server, WebSphere, where a history is maintained.

24. Security policy modifications are captured for the MMS servers. The logs are sent to Splunk log correlation servers in the SCDHHS POD.

25. Application administrators do not have a business need to print sensitive information, but any printing would be captured via the user’s system logs.

26. Social Security Administration data audit logs generated by Cúram.

7.2.2 Performance Hardware Architecture

The MMS Infrastructure consists of Oracle Enterprise Linux servers and Windows servers divided into environments to support the Software Development Lifecycle. These servers are VMWare virtual machines hosted on Lenovo ESX servers built in PODs located in

SCDHHS Data Classification: Confidential Distribution Page 26 of 67

Anderson, SC and Columbia, SC. The PODs are built with identical NetApp storage that includes SSD, 10k, and 7k arrays. The two data centers have primary and secondary network redundancy to ensure accessibility for both locations. Each data center receives network traffic from a point of presence (POP) where the primary and secondary channels converge.

The IBM team maintains configuration information in Confluence of the MMS servers and the environment to which they belong.

All MMS VMs are placed in the associated networks based on their role within the three-tier system described below:

Figure 3 MMS Tiered Architecture https://devtools.scdhhs.gov/confluence/display/ProdSupp/Curam+-+ESB+Server+Environments

SCDHHS Data Classification: Confidential Distribution Page 27 of 67

At the top level, the F5 load balances and distributes incoming the HTTPS requests across web and Hub servers.

All firewalls follow a deny-by-default, allow-by-exception methodology. All firewall exception requests are logged in the Medicaid Services ticketing system, evaluated by Clemson’s

OISP, and implemented only if no serious existing vulnerabilities exist between the servers requested for connectivity.

Underneath the F5 are the three layers that denote levels of accessibility.

Tier one, the topmost tier of the system – the presentation tier – is accessed directly by users. The presentation tier displays information in the user interface and communicates with other tiers by displaying the results to the consumers, and to all other tiers in the

Cúram network.

Tiers two and three represent the application and database layers, respectively. The components at the application layer manage the business logic of the MMS/Cúram application and permit access to the third-tier services. The application layer is where most of the Cúram processing work occurs. Without an application logic layer, client components access the product database directly. Separating the second and third tiers reduces the load on the third-tier services, supports more effective connection management, and can improve overall network performance. Read more about the three-tier architecture at the

IBM Cúram Knowledge Center.

The MMS infrastructure is composed of primary and secondary data centers (or “sites”).

The primary production site is in western Anderson County, South Carolina. It is located approximately ten (10) miles from the heart of the Clemson University campus in what is known as the Clemson Research Park.

The Secondary Site is in the SC Department of Administration, Division of Technology Data

Center (SCDTO), located in Columbia, South Carolina. It is located approximately seven (7) miles from the SCDHHS offices in downtown Columbia. This Secondary Site serves as the

Disaster Recovery site for the MMS. See Appendix 1 for infrastructure diagrams.

7.3 Software Architecture

7.3.1 Security Software Architecture

Security is built into the infrastructure that underpins the development of Cúram application.

It supports the authentication of a user at login and provides support for the authorization process. Authentication is needed when a user must be verified to access a secure resource on a system.

Form-based authentication means a user is presented with a form allowing him or her to enter user name and password credentials. These credentials are compared against the https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/covr_3-tier.html

SCDHHS Data Classification: Confidential Distribution Page 28 of 67 credentials stored on the system or in Active Directory for this user name; if they match, the user is considered an “authenticated user” for the system. HTTPS/SSL is turned on by default to ensure the form-based login authentication mode is secure. The authentication result returned by this method is automatically logged in the AuthenticationLog database table.

Different authentication modes can be configured (depending on authentication requirements) via the Cúram Java Authentication and Authorization Service (JAAS) login module.

The following are the authentication modes supported:

• Default Authentication (Authentication is performed by Cúram)

• Identity Only Authentication (Authentication is performed by third-party software, LDAP)

• External Access Security Authentication (Custom authentication for external users)

SCDHHS has implemented Identity Only Authentication for internal users and Default

Authentication for external users.

Identity Only Authentication for Internal Users:

Identity only verification means that…