PL 007.pdf

SCDHHS Data Classification: Confidential Distribution

Member Management Replacement Program

Configuration Management Plan

Version 14.2

June 4, 2021

Prepared by:

South Carolina Department of Health and Human Services (SCDHHS), Clemson Medicaid IT Services (MITS)

IBM

PL 007

MMRP Configuration Management Plan V15.0

SCDHHS Data Classification: Confidential Distribution Page 2 of 50

Table of Contents

1. Revision History

1.1 Review History

2. Review and Approval

3. Introduction

3.1 Identification

3.2 Purpose

3.3 MMRP Configuration Management Roles

3.4 MMRP Configuration Items (CIs)

3.4.1 Infrastructure Configuration Items

3.4.2 Application CIs

4. MMS Server Procurement Process

4.1 Server VM Naming Conventions

4.2 SaltStack States Applied by HCIS to all MMS Server Builds

5. Server-Level Configuration Management

5.1 MMS Server Configuration Control

5.1.1 Baseline Configurations

5.1.2 Server Baseline Configuration

5.1.3 Configuration Change Process

5.2 Configuration Change Types

5.2.1 Configuration Changes to Existing Systems

5.2.2 Configuration Changes to New Server Builds

5.3 Baseline Configuration Review

5.4 Mandatory Configurations

5.4.1 Server Hardening

5.4.2 Hardening Process

5.4.3 Physical Hardening

5.5 Unauthorized Changes

5.6 Configuration Exemptions

6. SCDHHS Server Inventory

6.1 MMS Access Control

6.2 MMS Server Updates

6.3 Server Security Patching

6.4 Server Maintenance

6.5 Log Auditing and Monitoring

6.5.1 Logged Events on Servers

6.5.2 Splunk Logging

6.5.3 Auditing Review

6.5.4 Additional Monitoring

7. Infrastructure Change Control

SCDHHS Data Classification: Confidential Distribution Page 3 of 50

7.1 Change Management

7.2 Changes to Configuration Items

8. MMS Application-Level Configuration Management

8.1 Cúram Configuration Control

8.2 Bamboo Builds

8.2.1 How the IBM MMRP Team Uses Bamboo

8.3 SaltStack Configuration Process

8.3.1 How the IBM MMRP Team Uses SaltStack

8.4 MMS Codebase Configuration Management

8.4.1 Codebase Baseline Management

8.4.2 Baseline Integrity

9. MMS Software CI Support

9.1 Vulnerability Management

9.1.1 CMS Standards

9.1.2 Vulnerability Remediation

9.2 Patch Management

9.2.1 Maintenance Schedule

9.3 Log Auditing and Monitoring

9.4 Changes to Configuration Items

10. MMS Change Control

11. IBM Cúram Build and Release Process

11.1 Developer Estimates and Evaluations

11.2 Developer Changes

11.3 Issue / Bug Resolution

11.4 Daily Commit / Trunk Merge Evaluations

11.5 DevOps Actions - Release Branch, Builds and Code Merges

11.6 DBA Actions

11.7 QA Actions

11.8 Project Manager Actions

11.9 Business Analyst Actions

11.9.1 Testing Coordination

12. Release to Production

12.1 Release

12.2 WAS Configuration Management

13. Terms

13.1 Acronyms

13.2 Glossary

SCDHHS Data Classification: Confidential Distribution Page 4 of 50

1. Revision History

Revisions

Date Author Rev Notes

10/04/14 Stasey Tackett 1.0 Initial version

10/12/14 Stasey Tackett 1.0 Edits

7/27/15 Stasey Tackett 2.0 Refresh for July 2015 SSP

2/18/16 Rudy Rodriguez 3.0 Updates for FDDR

3/10/16 Stasey Tackett 3.0 Edits

3/15/16 Stasey Tackett 4.0 SaltStack Changes (Matt Garrett)

3/15/17 Abbie Tremblay 4.1 Clemson Security updates

4/19/17 Stasey Tackett 4.2 IBM updates; full document edit

5/31/17 Stasey Tackett 4.3 Business feedback, additional content and terms; New template

6/23/17 Stasey Tackett 5.0 Final edits

9/06/17 Stasey Tackett 5.5 Quarterly Review completed by

Clemson OISP and IBM MMRP O&M

CoreOps teams

10/04/17 Stasey Tackett 5.6 Additional comments from SCDHHS reviewers; IBM MMRP

Patch/Vulnerability Management content added; IBM MMRP O&M

CoreOps Lead review comments

10/23/17 Stasey Tackett 5.7 Comments per 10/19/17 meeting with state

SCDHHS Data Classification: Confidential Distribution Page 5 of 50

11/06/17 Stasey Tackett 6.0 Updated Section 10 (Change

Management) with a reference to the

SCDHHS MMRP PMP

1/18/18 Abbie Tremblay 7.0 Quarterly Review: Clemson Security updates to section 5.3.1 (Server

Hardening) and 6.3 (Server Security

Patching)

1/19/18 Stasey Tackett 7.0 Quarterly Review: Version 7.0 edits

2/12/2018 Stasey Tackett 7.1 Address SCDHHS comments and feedback to create final document for approval

6/22/2018 Abby Tremblay 7.2 Completed full document review and updated references to CCIT HCIS and OISP functions

6/28/2018 Eric Mittnight 7.3 IBM MMRP Quarterly review and update; updates to hyperlinks and technical changes

7/18/2018 Stasey Tackett 7.4 Updates to address comments from

SCDHHS MMRP Technical and

PMO staff

7/20/2018 Stasey Tackett 7.5 Updates to address IBM MMRS

O&M final review and approval

10/19/2018 Eric Mittnight 7.6 Incorporate updates from IBM O&M, Clemson CCIT and OISP and

SCDHHS OIA teams

12/14/2018 Eric Mittnight

Mahmoud Al Taher

7.7 Incorporated updates on Application

CIs, Change/Defect Management processes and technical edits

3/29/2019 Eric Mittnight

Mahmoud Al Taher

7.8 Quarterly review and update

SCDHHS Data Classification: Confidential Distribution Page 6 of 50

4/25/2019 Eric Mittnight

Mahmoud Al Taher

7.9 Updated to address comments from

SCDHHS MMRP Technical and

PMO staff

5/20/2019 Erin Coleman 8.0 Updated version number and date to reflect approval, removed all comments, and accepted track changes.

8/6/2019 Mahmoud Al Taher 8.1 Incorporated updates from Clemson

CCIT and OISP to the Hardening

Process, Unauthorized changes, and

Glossary sections

8/9/2019 Nicholas McElveen

Eric Mittnight

Mahmoud Al Taher

8.1

Quarterly review and update

9/9/2019 Erin Coleman 9.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

12/5/2019 Nicholas McElveen

Mahmoud Al Taher

9.1 Quarterly review and update

12/19/2019 Mahmoud Al Taher 9.2 Addressed SCDHHS comments

1/17/2020 Erin Coleman 10.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

3/3/2020 Mahmoud Al Taher

Nicholas McElveen

10.1 Updated version number and

hyperlinks, Incorporated Clemson updates

4/1/2020 Mahmoud Al Taher 10.2 Addressed SCDHHS comments

4/27/2020 Erin Coleman 11.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

6/5/2020 Mahmoud Al Taher

Nicholas McElveen

11.1 Quarterly review and update.

SCDHHS Data Classification: Confidential Distribution Page 7 of 50

Abbie Abbott

6/25/2020 Erin Coleman 12.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

8/14/2020 Mahmoud Al Taher

Nicholas McElveen

Abbie Abbott

12.1

Quarterly review and update.

9/29/2020 Mahmoud Al Taher

Matthew Garrett

12.2 Addressed SCDHHS comments

11/13/2020 Erin Coleman 13.0 Updated version number and date to reflect approval, updated watermark, and accepted track changes.

1/15/2021 Mahmoud Al Taher

Nicholas McElveen

13.1

Quarterly review and update.

2/9/2021 Erin Coleman 14.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

3/25/2021 Mahmoud Al Taher 14.1 Quarterly review and update

5/6/2021 Mahmoud Al Taher 14.2 Addressed SCDHHS comments

6/4/2021 Erin Coleman 15.0 Updated version number and date to reflect approval, removed all comments, updated watermark, and accepted track changes.

Version Date Author(s) Comments

7.6 11/06/2018 Erin Coleman Completed technical edit, addressed comments for SCDHHS OIA team, SCDHHS Data Classification: Confidential Distribution Page 8 of 50 requested update on Change & Defect Management process details

7.6 11/19/2018 Renjan Bhasi Requested updates for Application CIs

7.6 11/10/2018 Tim Cozine Comments added to the document

7.6 11/13/2018 Noddie Gibson Comments added to the document

7.6 11/14/2018 IV&V Team Minor comments

7.7 1/14/2019 Renjan Bhasi Reviewed and closed my comments, all are addressed.

7.8 4/1/2019 Erin Coleman Reviewed updates and added comments.

7.8 4/6/2019 Tim Cozine Reviewed

7.8 4/11/2019 Renjan Bhasi Reviewed and added 1 comment.

7.9 5/1/2019 Renjan Bhasi Reviewed and closed the comment.

8.1 8/16/2019 Erin Coleman Reviewed updates

8.1 8/28/2019 Renjan Bhasi Reviewed, no new comments.

9.1 12/6/2019 Erin Coleman Reviewed; comments added

9.1 12/6/2019 Tim Cozine Reviewed, no comments.

9.1 12/18/2019 Renjan Bhasi Reviewed, added 2 minor comments.

9.1 1/7/2019 Renjan Bhasi Reviewed, okay with IBM response.

10.1 3//2020 Erin Coleman Reviewed updates

10.2 4/2/2020 Erin Coleman Reviewed updates

10.2 4/6/2020 Tim Cozine Reviewed.

11.1 6/10/2020 Erin Coleman Reviewed updates

SCDHHS Data Classification: Confidential Distribution Page 9 of 50

11.1 6/24/2020 Tim Cozine Reviewed.

12.1 8/18/2020 Erin Coleman Reviewed updates, no comments.

12.1 8/26/2020 Tim Cozine Reviewed updates, no comments.

12.1 9/1/2020 Renjan Bhasi Reviewed updates, no comments.

12.2 10/1/2020 Erin Coleman Reviewed updates, no comments.

12.2 10/2/2020 Tim Cozine Reviewed updates.

13.1 1/19/2021 Erin Coleman Reviewed updates

13.1 1.19.2021 Tim Cozine Reviewed.

13.1 2/1/2021 Renjan Bhasi Reviewed, no new comments.

4/2/2021 Tim Cozine Reviewed.

14.1 4/13/2021 Renjan Bhasi Reviewed, added 2 comments.

2. Review and Approval

The undersigned acknowledge that they have reviewed this Configuration Management

Plan and agree with the information presented. Changes to this document will be coordinated with and approved by the undersigned or their designated representatives:

Name Title Approval Date

Rhonda Morrison

Member Management Replacement Program (MMRP) Director, SCDHHS

6/3/2021

(delegated to Jason Dale)

Deputy Chief Information Officer (CIO), Information Management, SCDHHS

4/2/2021

SCDHHS Data Classification: Confidential Distribution Page 10 of 50

Name Title Approval Date

Thomas Taylor MITS Executive Director, IT Services, Clemson University

6/1/2021

Jeff Leyh MMRP Operations and Maintenance Manager, IBM

6/3/2021

3. Introduction

The South Carolina Department of Health and Human Services (SCDHHS) Member

Management Replacement Program (MMRP) Configuration Management Plan establishes the overall approach for Configuration Management planning requirements for the Member

Management System (MMS). In order to reflect the future state of MMS as the sole eligibility determination system for SCDHHS, the name of the system has been updated to

Member Management System (MMS).

This plan addresses the process for updating and releasing the Configuration Management

Plan as the principal configuration management process artifact, described below:

In compliance with the Centers for Medicare and Medicaid Services (CMS)

Minimum Acceptable Risk Standards for Exchanges (MARS-E) Configuration

Management CM-9 control, the SCDHHS System Security Plan (SSP) attests that the MMRP Configuration Management Plan is updated at least yearly.

The plan is reviewed quarterly to capture critical changes to configuration management activities and updated as needed.

This Configuration Management Plan aligns with the SCDHHS governance model for

Configuration Management, which ensures the following best practices are administered in order to:

1. Define and control the components of services and infrastructure.

2. Maintain accurate configuration information on the historical, planned and current state of services and assets.

The Configuration Management Plan addresses the configuration management processes for the MMS infrastructure and associated change and configuration management artifacts.

The MMS infrastructure is managed and maintained by Clemson University’s Infrastructure

Service and Operations (ISO) Healthcare Information Services (HCIS) organization.

SCDHHS Data Classification: Confidential Distribution Page 11 of 50

This plan also addresses the configuration processes executed by the IBM MMRP team in support of the MMS Cúram code base and all associated application components and change and configuration management artifacts. The IBM MMRP Program Management

Office (PMO) is responsible for the management, maintenance and update of the

Configuration Management Plan.

The Configuration Management Plan describes the processes used to control changes to the MMS through a formal process of change requests, reviews, testing, change approvals and post-change validations. This plan provides MMS with consistency within operations and life cycle management. Through the processes and tools described in this plan, the

MMS support organizations can compare new configurations to baseline configurations for executing operational, functional and security assessments.

The Information Technology Infrastructure Library (ITIL) defines a Configuration Item (CI) as follows:

Any Component that needs to be managed in order to deliver an IT Service.

Information about each CI is recorded in a Configuration Record within the

Configuration Management System and is maintained throughout its Lifecycle by Configuration Management. CIs are under the control of Change

Management. CIs typically include hardware, software, and formal documentation such as process documentation.

The plan identifies how requests and approvals for new CIs are executed and provides information on the Change Management processes that have been implemented for MMS, including submitting, planning, approving, and implementing changes to the system. To this end, the plan addresses how changes to the system are executed through the MMRP

Release Management process.

CIs are identified through a process of defining functional and physical characteristics of a systems component (e.g., hardware, software, specification documents, change artifacts) in sufficient detail so that it may be developed, tested, evaluated, produced, procured, accepted, operated, maintained, and supported. CIs are defined and established by baselines plus approved changes. The identification of CIs includes the selection, creation, and specification of software services to be delivered, requirements and design documents and change and configuration management artifacts.

It is the goal of Configuration Management to maintain a detailed record of the system’s configurations in order to:

• Allow HCIS to revert to a previous server patch level,

• Allow the IBM MMRP Design, Development and Implementation (DDI) team to revert to an earlier Cúram release, and

SCDHHS Data Classification: Confidential Distribution Page 12 of 50

• Allow tracking of MMS software or database patches as they are deployed and tested across the MMS development and test environments.

There are several entities involved in the configuration control activities for MMS and they reside within different organizations. For an overview of the roles and responsibilities of all

SCDHHS, IBM and Clemson MMRP team members and key stakeholders, please refer to the MMRP Roles and Responsibilities matrix maintained in the SCDHHS MMRP

SharePoint site.

Organization Role

SCDHHS MMRP - Serves as the governing authority and owner of the MMS and is currently led by Rhonda Morrison, MMRP Program Director.

- The MMRP is a part of the Office of Systems Development and the Program Director reports to the Deputy Director of Systems Development.

MMRP Change Advisory Board

(CAB)

- Responsible for reviewing DDI Change Requests (CRs) and determining if the CR will be submitted for further consideration by the MMRP Executive Oversight Committee.

SCDHHS CAB - The SCDHHS CAB is an integral part of the Change Management process designed to balance the need for change with the need to minimize inherent risks.

- The SCDHHS CAB provides oversight over SCDHHS and Clemson infrastructure changes.

MMRP Eligibility Executive Oversight Committee

(EEOC)

- Responsible for defining the vision and strategic direction of the MMRP, providing executive and organizational support, reviewing and approving major project decisions, and serving as the overall project oversight

- Responsible for the final decision on CR approvals

- Consists of representatives from SCDHHS leadership, to include the MMRP Program Director, MMRP DDI Manager, Eligibility Operations Program Manager, Deputy Director of EEMS and the Deputy Director of Health Programs.

SCDHHS OIM

Steering Committee

- Responsible for providing strategic oversight at the enterprise level and directing operating principles to SCDHHS OIM Contractors

- Responsible for approving Enterprise level CRs https://schhs.sharepoint.com/:x:/r/sites/1211207/_layouts/15/Doc.aspx?sourcedoc=%7BDA5120A5-9F65-4E11-A623-5AC6A72F6ACB%7D&file=MMRP%20Roles%20and%20Responsibilities.xlsx&action=default&mobileredirect=true&cid=21546293-70cb-4d8a-8d5c-cab48490c94e

SCDHHS Data Classification: Confidential Distribution Page 13 of 50

Organization Role

- Consists of the CIO and OIM Program Managers, to include the Medicaid Enterprise Services (MES) Program Director

CCIT ISO

(Clemson)

- Clemson University’s Clemson Computing Information Technology (CCIT) Infrastructure Services and Operations (ISO) is a Clemson University entity and the umbrella organization that provides all MMRP infrastructure and infrastructure support.

CCIT Healthcare Infrastructure Services (HCIS) (Clemson)

- HCIS is the core team within the CCIT ISO that supports MMS operations and maintenance.

- HCIS plays a critical role in all change and configuration management processes. HCIS issues a Request for Change (RFC) for the proposed configuration changes to the production system.

CCIT Office of Information Security and Privacy (OISP) (Clemson)

- OISP is Clemson University’s security assurance organization and supports MMS security operations and maintenance.

- OISP plays a critical role in change management assessment functions.

MMRP Design, Development & Implementation

(DDI) (IBM)

- The IBM MMRP DDI team is the IBM application development and operations support component where the MMS Cúram development and production releases are developed and tested.

MMRP Operations and Maintenance (O&M) Core Operations (CoreOps) (IBM)

- The IBM MMRP O&M CoreOps team configures and maintains the development, test and staging environments, code repositories and tools needed to support the project.

- The O&M CoreOps team is responsible for managing MMS application and database changes to the MMS production environment, to include software bug fixes and configuration changes, database configurations, package upgrades, patches, fix packs, etc.

MMRP PMO (IBM) - The IBM MMRP PMO Information Assurance (IA) team is responsible for the development, maintenance and updates to all documents in the Consolidated Deliverables Management List (CDML), including the Configuration Management Plan, Systems Design Document, Data Management Plan, among others.

SCDHHS Data Classification: Confidential Distribution Page 14 of 50

All MMS infrastructure CIs are managed by Clemson’s ISO HCIS group. After proposed changes to CIs are approved through the Change Management process, updates to the

Configuration Management Database (CMDB) are implemented to reflect the current state of the infrastructure. HCIS generates a report from the MMRP configuration tool, Salt, on the 15th of each month, so that the IBM MMRP O&M team may be aware of current inventory of server virtual machines (VMs) currently commissioned.

HCIS builds servers based off a baseline template (i.e., server image) that provides a definitive basis for change control. (See Section 7, Infrastructure Change Control.) After the baseline image is built using the SaltStack configuration management tool, the server is added to the weekly Nessus vulnerability scanning list, and the VMs are then delivered to

MMS O&M CoreOps team from HCIS with the following:

• Requested OS

• Firewall rules, which allow specific servers in the same tier to communicate with each other (upon request)

Once VMs are built, IBM MMRP O&M CoreOps team adds the VM names and properties to an internal secure server specifications spreadsheet maintained in Confluence. The document is the inventory of all servers, categorized by environment, use and properties.

This information is only available and updated by specific authorized individuals and serves to assist in proactive decision-making by key stakeholders in order to maintain the project’s delivery goals.

All MMS application CIs are managed by IBM’s MMRP DDI and O&M CoreOps teams.

Cúram CI baselines are tracked through code tagging and branching using Subversion

(SVN) version control. The MMRP teams utilize SVN to maintain a history of all code changes, build code bases for testing, and maintain a history of Cúram releases and upgrades. Before a specific code base is released into production, that code is tagged from the SVN Release branch to signify a production release. SVN is also used to manage and maintain application properties and configuration baselines.

Cúram and other internal agents rely on the Enterprise Service Bus (ESB) (MuleESB) version 3.8.7 hosted on Clemson Linux servers to send and receive Simple Object Access

Protocol (SOAP) based web service calls to CMS external network. ESB handles the orchestration of the web service calls received from the South Carolina user agents and routes such internal communication traffic via a secure network layer called the Edge-Hub proxy. This Edge-Hub proxy is hosted on a WildFly 16 app server which acts as the authenticated proxy server to communicate with CMS server endpoints.

SCDHHS Data Classification: Confidential Distribution Page 15 of 50

The SaltStack tool is used to support ESB configuration management requirements.

SaltMaster is used to maintain server configurations identical on all servers: Saltscripts for

ESB application server installation, setup, WildFly Application server, Java Version, Apache

Tomcat, XML server and release management. Cúram operational configuration control is driven by MMS’s development tracking system, Jira. Jira is the MMRP team’s ticketing and change management tool that works in concert with the IBM MMRP knowledge base, Confluence. The IBM MMRP DDI team manages all Cúram releases via SVN in order to preserve baseline integrity and enable rollback.

The IBM MMRP O&M team is responsible for managing critical application and database upgrades, patches and other configuration changes to these components. The O&M team is responsible for maintaining the following software components:

• Cúram Social Program Management (Cúram),

• WebSphere Application Server (WAS),

• IBM Java,

• Oracle Database Server,

• Oracle Data Integrator (ODI),

• Mule ESB,

• Apache Tomcat,

• Cúram XML Server,

• WildFly,

• Bamboo (Atlassian),

• Fisheye (Atlassian), and

• Jira (Atlassian).

Crystal Reports and Oracle Managed File Transfer (MFT) are currently supported by the

SCDHHS Information Technology Services (ITS) organization.

The configuration changes for these components are derived from vendor upgrades, patches for application bug fixes and security vulnerabilities identified through security scans and vendor security bulletins and other sources. MMS operational configuration change control is supported by MMRP’s change tracking system, Jira, working in concert with Confluence. In addition, SaltStack is leveraged to provide up to date configuration data for MMS hardware, software and databases for all active MMS environments, which is made available to MMS system administrators.

Back to top https://devtools.scdhhs.gov/confluence/display/ProdSupp/Curam+-+ESB+Server+Environments#expand-Details

SCDHHS Data Classification: Confidential Distribution Page 16 of 50

4. MMS Server Procurement Process The process by which the IBM MMRP O&M team requests and commissions its VMs is listed below:

1. Specifications for the requested server(s) are maintained by the HCIS in Salt. These specifications include CPU, RAM, Swap, filesystems, and Network File System

(NFS) and shared mounts.

2. The IBM MMRP O&M CoreOps team creates a ticket in the Clemson ticket management system, where the details from the server specifications spreadsheet are included. This ticket is assigned to HCIS as well as the MMRP Program Director.

3. The MMRP Program Director must approve all server builds. The ticket does not proceed to server build or delivery until this approval is inputted.

4. HCIS works with the Network Operations Center (NOC) to configure Internet

Protocol (IP) addresses and Domain Name System (DNS) name(s) for the server request.

5. HCIS builds the server according to the server specifications from the ticket.

6. HCIS applies the tier-specific platform set to grant users access to the server(s).

7. The ticket is updated to confirm that the servers are delivered and ready for use.

The Server Specifications spreadsheet is maintained by the IBM MMRP O&M CoreOps team, who requests the server name from Clemson HCIS as part of the server procurement process. HCIS maintains the conventions for naming servers.

Example: hhs-mem-d-was01, hhs-cgs-d-was01 hhs Organization / SCDHHS server mem or cgs

Project / This is a reference to the project the server is supporting.

• ‘mem’ for MMS servers

• ‘cgs’ for MMS Cúram Global Income Support (CGIS) servers d Tier / This letter represents the tier that this server is supporting.

(Example: “d” for development, “q” for qa, “s” for staging, “p” for production, “z” for sandbox, “t” for training etc.)

was Type / These letters represent the function of the server. (Example: Hub, ESB, XML, Web, Ora, Was)

SCDHHS Data Classification: Confidential Distribution Page 17 of 50

01 Number / general server number

The SaltStack states are contained in Git, a version control system that manages files as they change over time. If an update is made, Git pushes the update to all the servers. The HCIS-managed Git server is hosted on-premise at the Clemson University data center in Anderson, SC and follows the same security guidelines as other HHS infrastructure servers.

When changes to the default Salt states are required, HCIS follows these procedures:

• Salt administrator clones the repository and checks out the latest state into a test environment.

• Salt administrator makes the necessary changes and tests.

• Salt administrator submits a merge request.

• Merge requests are handled by a team of 5 Salt admins. These administrators review the code change by hand, apply the Salt state to a new machine, and build a machine using the new changes. If all tests pass and the new code does perform the new actions correctly, the Salt administrator will merge the code into the code base.

NOTE: A Salt admin cannot merge their own code into the base.

HCIS applies all of the following SaltStack states by default when the MMS server is built:

• resolv - fix DNS to search clemson.edu by default

• hostfile - add the SCDHHS domain controllers for Lightweight Directory Access

Protocol (LDAP) queries

• spacewalk - add to spacewalk for patching

• rootPassword - update root password to our default

• postConfig - apply standard packages for administration

• ntp - add the time servers

• legato - add the backup software

• asam - adds the usernames to the machines from SCDHHS

SCDHHS Data Classification: Confidential Distribution Page 18 of 50

• sshd - updates default ssh daemon

• splunk - installs Splunk client and configures logs to point to HHS Splunk

• mail - configures default mail client

• nist - NIST server hardening

• hhsFirewall - installs local firewall

• vmware - installs VMWare Tools

• ksplice / kspliceUserspace - adds software to allow for updating OS without reboots

• servicesHcis - updates the ports list so that what service is using which port is visible

• grub - fixes GRUB for better defaults

• bash - updates bash so that has history timestamp

• ldconfFix - fixes the /etc/ld.so.conf to read /etc/ld.so.conf.d directory

• linkTickets - creates all the TT tickets and link them together

• extend Disks - converts default disks to bigger disks (if defined in build)

• system Upgrade - fully patches (and reboots) the server

5. Server-Level Configuration Management

HCIS builds and maintains configuration settings for MMS servers using the SaltStack methodology and tool.

SaltStack, also known as “Salt,” is a configuration management and remote execution tool supporting the infrastructure-as-code approach to deployment and cloud management. It maintains remote nodes in defined states, ensuring that specific services are running. Salt also allows servers to be targeted not only by hostname, but also by server properties.

Configuration control is both established and maintained by SaltStack. Server settings can be queried from the SaltStack head node, and all baseline configurations can be generated from Salt, ensuring that documentation is always current and accessible.

SCDHHS Data Classification: Confidential Distribution Page 19 of 50

All baseline configuration Salt states are stored in a Git repository. Rollback of configuration is easily accomplished by an HCIS administrator logging into the SaltStack head node, checking out an earlier baseline from Git, and applying the previous configuration to the specific environment.

Documentation specific to any configuration controls in line with any install or change is accessible from the SaltStack head node by the HCIS System Administrator, who has access to view configuration control and server properties. (See Section 7, Infrastructure

Change Control.)

HCIS performs multiple operations to configure an MMS server for operation.

This configuration represents baseline MMS server configuration, and the steps for installation of the operating system configuration are stored in Clemson’s secure

Confluence ISO knowledge base.

Configuration changes to existing production servers or changes that will affect new builds of SCDHHS MMS Linux Servers are initiated in Clemson’s ticket management system. In the initial ticket, the change required is thoroughly documented, as described below:

1. IBM MMRP O&M CoreOps/HCIS submits the proposed change to SCDHHS stakeholders to vet the change.

2. IBM MMRP O&M CoreOps/HCIS issues an RFC for the proposed configuration change to the production server.

3. IBM/HCIS engages Clemson CCIT OISP and/or SCDHHS OIA to evaluate the requested change.

4. OISP/OIA engages the appropriate applications teams to identify potential problems.

5. Once the proposed change is vetted by the MMRP CAB, the proposed change is production-ready.

6. The change is applied to production environment by HCIS.

7. The Standard Operating Procedure (SOP) for SCDHHS MMRP Baseline

Configuration of Linux Servers documentation is updated to reflect the changes.

8. Server administrators confirm that the proposed changes have been implemented successfully.

SCDHHS Data Classification: Confidential Distribution Page 20 of 50

The modification of baseline configurations for existing servers are driven by a combination of processes involving SaltStack and specialized scripts.

In most cases, HCIS changes the Salt State so that if a new server is built, it will have the correct configuration; however, occasionally specialized scripts are required for these modifications.

These modifications are delivered by HCIS and applied en masse to the supported servers and incorporated into the procedures and scripts of the baseline configuration via the

Configuration Change Process, Section 5.1.3.

For changes requested by IBM MMRP O&M CoreOps, such as a new server build, see

Configuration Change Process, Section 5.1.3.

The modification of baseline configurations for new servers is also approved via the

Configuration Change Process, Section 5.1.3.

In compliance with MARS-E Control CM-02(01), OISP reviews baseline configurations at least every 365 days. Linux administrators coordinate with OISP to review the security controls. Any updates to baseline configurations are documented in the Baseline

Configuration Plan and then presented to HCIS for implementation.

HCIS then implements the applicable changes based on the security review according to the Server Baseline Configuration Change procedures. (See Section 3.3, MMRP

Configuration Management Roles, for complete definitions of all entities involved in

Configuration Management.)

Server hardening is the process of enhancing the server’s security using Center for Internet

Security (CIS) level 1 and level 2 audit scans. The hardening procedures described below are documented within the MMS baseline configuration management document. The server hardening process is managed by the Clemson University ISO HCIS organization.

SCDHHS Data Classification: Confidential Distribution Page 21 of 50

Clemson Security performs hardening scans on all new system builds that do not already have a hardening configuration. The recommended hardening settings from these scans are reviewed by HCIS Systems Administrators to determine the required settings that can be enforced while maintaining expected business functionality. The process allows for the minimal system access necessary to maintain a “least privilege” standard across the MMS environments. All configuration settings are captured by the Salt configuration management tool and retained within the HCIS GIT repository.

• Deviations from approved configuration settings must be supported by a business justification and be approved by Clemson Security; all approved deviations will be documented and regularly reviewed to ensure the business justification remains.

• Mandatory configuration settings are enforced by the Salt configuration management tool and all changes are captured and monitored within the Splunk log correlation system.

All SCDHHS MMS servers are secured in a locked rack within the SCDHHS POD within

ISO and managed by HCIS. The POD refers to the module of MMS network components that work together to deliver services. The SCDHHS POD contains high-density infrastructure racks to support MMS.

Temporary manual exceptions to the configuration can be made under emergency circumstances. The exceptions must follow the same documentation and approval procedures before being implemented. OISP utilizes various tools in order to detect abnormal changes and responds/escalates accordingly.

Various SaltStack commands are issued to test if the machine in question is in compliance.

For example, the following commands are executed to test if the configuration on the target machine is in compliance:

salt-master # salt ‘target-machine-id’ state.highstate test=true

To check if machines have updates available:

salt-master # salt ‘target-machine-id‘ pkg.upgrade_available

To check version of http installed on a machine (example):

salt-master # salt ‘target-machine-id‘ pkg.version httpd

SCDHHS Data Classification: Confidential Distribution Page 22 of 50

Servers requiring an exemption from any mandatory configuration settings must have the exemptions documented with approval from OISP and/or the SCDHHS Office of Information

Assurance (OIA).

Back to top

6. SCDHHS Server Inventory All new MMS server builds are documented in Clemson’s ticket management system and approved by OISP for network placement.

ISO maintains an inventory of servers and components for each environment in the CMDB that is readily available by both system administrators and OISP.

As new servers are provisioned by HCIS via SaltStack to SCDHHS environments, part of the build process is for the HCIS Hardware Architecture Team to update the CMDB and any other appropriate inventory repositories.

After proposed changes to CIs are approved through the Change Management process, updates to the CMDB are implemented to reflect the current state. HCIS generates a report from the MMS configuration tool, Salt, on the 15th of each month so the MMRP team may be aware of current inventory of VMs.

SCDHHS governs the account creation and management policy; which states that individuals may not authenticate as a service account. Access to SCDHHS MMS is governed by means of the Identity Manager Fan-Out Driver’s Platform Sets. In this context, HHS-MMS servers are called “platforms” and are contained in a grouping called a “Platform

Set.”

a. HHS-MMS PLATFORM SETS:

i. EEMMS_PrdWeb

ii. EEMMS_PrdApp

iii. EEMMS_PrdDB

iv. EEMMS_PrdUTSutil

v. EEMMS_DevWeb

vi. EEMMS_DevApp

vii. EEMMS_DevDB

SCDHHS Data Classification: Confidential Distribution Page 23 of 50

viii. EEMMS_Devutil

ix. EEMMS_UTSWeb

x. EEMMS_UTSApp

xi. EEMMS_UTSDB

xii. EEMMS_QaWeb

xiii. EEMMS_QaApp

xiv. EEMMS_QaDB

xv. CGIS_DevApp

xvi. CGIS_DevDb

xvii. CGIS_DevWeb

xviii. CGIS_QaWeb

xix. CGIS_UtsApp

xx. CGIS_UtsDb

xxi. CGIS_UtsWeb

b. HHS_MMS PLATFORM SET USER/GROUP PRIVILEGES

To ensure the least privilege necessary is given to users at login, user groups defined for the EEMS / HHS-MMRP platform set are maintained in SCDHHS Active Directory (AD) and passed down to the SCDHHS Vault. The following groups are defined to regulate this:

Group Privilege Level

Linux_Admins Sudo

MMRS-DBA Subset of files based on business needs

MMRS-Dev_web Subset of files based on business needs

MMRS-Dev_App Subset of files based on business needs

MMRS-Dev_Util Subset of files based on business needs

Misc_Admin Sudo

OISP_Admin Sudo

c. APPROVAL PROCESS FOR SERVER ACCESS REQUESTS

SCDHHS Data Classification: Confidential Distribution Page 24 of 50

A user requiring access to one or more servers in the HHS POD seeks approval by sending the request via Clemson’s ticket management system. A business need that necessitates the requested access is part of the ticket template and is a mandatory field. If the data owner and security groups deem that the business need is bona fide, then the access is approved.

The request requires approval from Clemson OISP and the MMS Data Owner with

SCDHHS OIA having veto rights. After those approvals are documented in the ticket, the user is added to the appropriate SCDHHS group.

Administrative (Sudo for Linux) access requests must have additional justification for the elevated access, as well as the appropriate level of management approval. All approvals are documented within Clemson’s ticket management system.

OISP and SCDHHS OIA reserve the right to request justification annually to maintain access to the associated systems.

d. USER ROLE CHANGE OR TERMINATION

If a user's role changes or a termination occurs, the associated technical support group (or manager) is responsible for requesting that the user have access removed from all appropriate servers by submitting a ticket to OISP. OISP reserves the right to immediately remove all server access for a disabled or terminated user.

In the event of a hostile termination, a request can be made via the OISP on-call with a follow-up ticket for tracking purposes.

e. GROUP MEMBERSHIP MONITORING

OISP, SCDHHS, and IBM monitor membership in the various access groups. OISP, SCDHHS, and IBM review the group membership in SCDHHS AD at least every 365 days.

Required updates and/or patching to the SCDHHS MMS Linux environment servers are done by HCIS through the Oracle Linux Spacewalk server. Since the patch set on the

Spacewalk server is updated daily, the Spacewalk server can query each server nightly to determine which software components, if any, require updating.

The last step in building a server is to patch everything to the latest version. Going forward, patching will be driven by security requirements and will be pushed to the server on a weekly (as needed) and/or monthly cadence. Change control procedures must be followed for proposed changes to Production. (See Section 7, Infrastructure Change Control.)

SCDHHS Data Classification: Confidential Distribution Page 25 of 50

MMS servers are scanned weekly by Nessus to identify any new vulnerabilities, which are aggregated into Splunk. OISP reviews the scan reports and submits any patching information to HCIS and IBM.

Details relating to the vulnerabilities can be viewed in Splunk or as reports produced by

SecurityCenter, the application which front-ends Nessus. OISP makes the reports available to MMRP through Clemson’s SharePoint site and sends to OIA weekly. OISP and OIA review all vulnerabilities and reserve the right to elevate a vulnerability rating based on widespread exploitation or imminent threat to the environment.

In the event of a vulnerability that affects the MMS environment, security patches will be applied to all affected servers within 7 days of escalation by OISP and/or OIA. All patches are tested by the IBM MMRP O&M CoreOps team in non-Production environments prior to

Production deployment. If a security vulnerability includes a work around, the work around may be applied in an effort to give the IBM MMRP O&M CoreOps team more time to evaluate the patch.

All other updates, bug fixes, or vulnerabilities that are determined to be low are applied on a

30-day cycle. The security patching processes for MMS are currently being revised to align with MARS-E security vulnerability remediation guidelines.

An RFC is required for all changes to Production, including patching. RFCs are submitted to

MMRP CAB for approval.

Any HCIS server maintenance involving a major system change, including Oracle Linux version upgrades or OS changes, undergo the following process before the server is reinstated into rotation:

1. HCIS releases the server to MMS for testing and validation.

2. MMS performs standard testing and confirms normal functionality.

3. The server is scheduled to return to normal operation.

4. The server is scanned for vulnerabilities using Nessus as part of the set weekly scanning process.

Detection of vulnerabilities and presence of malware are managed by Nessus and Trend.

Nessus uploads the reports to Splunk, allowing for security posture measurement at any time.

SCDHHS Data Classification: Confidential Distribution Page 26 of 50

Per MARS-E Audit and Accountability Control AU-2, the following events are logged to the

SystemChangeLog, which feeds into Splunk:

1. Server alerts and error messages

2. Log onto server

3. Log off server

4. Change of password

5. All system administrator commands

6. Switching accounts or running privileged accounts from another account

7. Creation or modification of super-user groups

8. Subset of security administrator commands while logged on in the security administrator role

9. Subset of security administrator commands while logged on in the user role

10. Clearing of the audit log file

11. Startup and shutdown of audit functions

12. Use of identification and authentication mechanisms

13. Change of file or user permissions or privileges

14. Remote access

15. IBM: At the application level, Cúram batch jobs create changes to both the application and the database. Batch servers produce their own logs via Automic’s

UC4. UC4’s Process Monitoring displays detailed information about the tasks being monitored, including status, success and failure of the batches, along with batch history.

16. IBM: At the application level, Person details (social security number, addresses, medical history, income, etc.) can be changed or corrected. These changes are not on the Cúram Audit Trail table page, but all changes are viewable in the database and on the application screen itself, along with the user(s) who made a critical change.

17. IBM: At the application level, if AD makes user account changes (account creation, security role, activation, deactivation, etc.) to its system, AD pushes those changes to Cúram via a web service called SyncCúramFromLDAP, and these changes are automatically updated in Cúram without manual intervention. However, other user account feature changes (change in user position, location, etc.) that are not directly

SCDHHS Data Classification: Confidential Distribution Page 27 of 50 present/supported in AD must be made by Cúram Administrators manually via

Administrator privileges. Auditing processes are accomplished through AD.

18. User log-on and log-off

19. Server shutdown and reboot

20. Server errors

21. IBM: At the application level, Cúram application errors, including shutdown, route to

Splunk. They are captured in Cúram’s application server, WebSphere.

22. IBM: At the application level, Cúram restarts route to Splunk. They are also captured in Cúram’s application server, where a history is maintained.

23. IBM: At the application level, Cúram application errors are captured in WebSphere.

24. Security policy modifications

25. Application administrators do not have a business need to print sensitive information, but any printing would be captured via the user’s system logs.

26. Social Security Administration data audit logs generated by Cúram.

Splunk monitors the following files on all Linux servers:

- /var/log/messages

- /var/log/secure

- /var/log/*.log

- /security_logs/*

- /app/IBM/WAS_8.5.5/AppLogs/PersonAuditInfo.log

- /app/IBM/WAS_8.5.5/AppLogs/CaseAuditInfo.log

- /app/IBM/WAS_8.5.5/AppLogs/SDXAuditInfo.log

- /app/IBM/WAS_8.5.5/AppLogs/BendexAuditInfo.log

- /app/IBM/WAS_8.5.5/AppLogs/SVESAuditInfo.log

- /app/IBM/WAS_8.5.5/AppLogs/MSPAuditInfo.log

Clemson OISP, IBM MMRP, and SCDHHS OIA review the list of auditing events at least every 365 days (to include infrastructure and application-level monitoring). The review process includes the following steps:

1. OISP/IBM/OIA review current auditable events list.

SCDHHS Data Classification: Confidential Distribution Page 28 of 50

2. OISP/IBM/OIA check the list against current auditing requirements from MARS-E

AU-2.

3. OISP/IBM also check newly documented changes to the servers/applications to determine if new audit events are needed.

4. OISP/IBM present any new audit changes to ISO HCIS.

5. HCIS/Splunk Admins implement changes via normal RFC processes.

6. Splunk Admins/OISP/IBM verify that the new audited events are being logged to

Splunk.

Splunk monitors the following events:

1. All successful and unsuccessful authentication attempts.

2. All changes to logical access control authorities (e.g., rights, permissions).

3. The enabling or disabling of audit report generation services.

4. Command line changes, batch file changes and queries made to the server (e.g., operating system, application, and database).

Back to top

7. Infrastructure Change Control Managing infrastructure changes is a critical part of MMRP’s configuration control processes.

HCIS infrastructure change control ensures that standardized procedures are used for efficient handling of all changes, minimizing the impact of change-related incidents to service, and continuously improving day-to-day operations.

All changes are tracked in Clemson’s ticket management system. Proposed changes are presented to the SCDHHS CAB.

All documentation, voting history, and follow-up is maintained by the SCDHHS CAB team.

Every change to the MMS core infrastructure, such as acquiring new VMs and databases, is requested by IBM MMRP O&M CoreOps via the Clemson’s ticket management system.

(See Section 5.1.3, Configuration Change process.)

The MMRP CAB is responsible for assessing and approving urgent changes because of a system or service incident. Emergency changes are those that do not follow the review period and voting procedures (often because the timeline for implementation is shorter than

SCDHHS Data Classification: Confidential Distribution Page 29 of 50 the typical review period). Emergency changes affecting services deemed “critical” are communicated to the CAB as required.

The SCDHHS CISO, Deputy CIO, and MMRP Program Manager may deny any change for reasons including, but not limited to, inadequate planning or back-out plans, because the timing of the change may negatively impact a key business process, or security risk.

The practice of ensuring that all changes to configuration items are carried out in a planned and authorized manner includes ensuring the following steps. There is a business reason behind each change, and it is recorded in a ticket within Clemson’s ticket management system.

- Identifying the specific configuration items and IT services affected by the change

- Planning the change

- Testing the change

- Having a rollback baseline, should the change result in an unexpected state of the configuration item

Back to top

8. MMS Application-Level Configuration

Management

The primary application used for the MMS is Cúram, an IBM commercial-off-the-shelf

(COTS) product customized to meet SCDHHS requirements.

Cúram configuration control is driven by MMRP’s development tracking system, Jira. Jira is the MMRP team’s ticketing and change management tool that works in concert with the

IBM MMRP knowledge base, Confluence. Jira and Confluence are both part of the

Atlassian product suite and are deployed in a separate VLAN from the MMS production servers.

Jira's SVN integration allows for access to SVN “commit” information relevant to each issue, known within Jira as a “story.”

Jira/SVN integration captures the SVN commit information applicable to each development issue with each release and provides for on-demand documentation configurable by desired criteria.

SCDHHS Data Classification: Confidential Distribution Page 30 of 50

Cúram codebase configuration baselines are under Change Management via the MMRP

Jira submittal process, governed by the SCDHHS and IBM MMRP DDI teams, after which approved changes to the baseline are formally documented and approved.

The IBM MMRP development team manages all Cúram releases via SVN in order to preserve baseline integrity and enable rollback.

In addition, an Atlassian tool, Bamboo is a continuous integration server that can be used to automate the release management for a software application, creating a continuous delivery pipeline. The Bamboo configurations are administered by the IBM MMRP O&M

CoreOps team.

Bamboo is the central management server that schedules and coordinates all work.

Bamboo provides a way to manage builds that have different requirements or targets, or automatic deployment to a server. Hence, the build and test process are not dependent on a specific local environment. Bamboo optimizes build performance through parallelism and leverages elastic resources implementing release management.

To ensure code that has not been approved for release is not committed, any code being released to Production resides in SVN’s Release branch. The build lead manages this by validating all SVN commits from the IBM O&M Release Manager.

The Configuration changes fall under the following two categories:

- Build configuration (Bamboo)

- WAS Application server configuration (Manual)

Bamboo audit logs are available for each build, which record all configuration changes made.

Any changes to the default WAS configuration are handled, tracked and audited via

CoreOps Jira stories following any existing code management change requests within Jira.

For…