MCS SOW OGE.docx

Exhibit 2.1.2 (MCS Statement of Work – Other Government Entities) Solicitation No. 5400028075

Managed Cloud Services

Exhibit 2.1.2 Statement of Work – Other Government Entities

Solicitation No. 5400028075

Table of Contents

Figures Figure 1 – MCS Other Government Entity Operating Model 4

Tables

Introduction Overview This Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities) sets forth the Services that Service Provider shall provide as of the Commencement Date unless otherwise specified to support Other Government Entities that elect Managed Cloud Services (MCS). Further, this Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities) sets forth the processes and systems that the Service Provider will provide, including the Service Provider's obligations to work with the Customer, to deliver integrated end-to-end Services. The Service Provider shall coordinate with the Customer, Customer’s cloud security teams and providers to effectively provide the Services.

The Service Provider shall provide a solution that supports all the requirements described in this Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities) and its Exhibits in accordance with Exhibit 4.2 (Financial Responsibility Matrix). All Services are included within the Charges described in Exhibit 4.0 (Business Model). Accordingly, the Service Provider also confirms that the Customers will not incur any other Charges for the requirements described in this Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities).

The Service Provider shall deliver the Services in the best interests of the Customer and the State. The Service Provider shall be responsive to the State’s current and future requirements by proactively anticipating needs and adjusting Services accordingly within the Charges. Requirements for New Services will be handled in accordance with Exhibit 1.4 (General Provisions), and the Service Provider shall assess the impact of these requirements on Customers’ operating environments and supported applications in accordance with the terms of the Agreement.

The Service Provider shall be responsible for all activities necessary to ensure Services are performed in accordance with all standards and processes contained in the Agreement, the Service Levels in Exhibit 3.1 (Service Level & Deliverables Matrix), and the Service Management Manual (SMM).

Service Provider shall integrate and fully cooperate with Customers as required for smooth and efficient operation during the development, implementation, and execution of its Service Management Processes to ensure consistency and integration across all Service Providers. Service Provider will work with Customers to deliver integrated Services as defined in Exhibit 3.1 (Service Level & Deliverables Matrix).

Operating Model Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities) contains specific cross-functional requirements that all SCPs must meet to perform the requested Services and responsibilities defined in Exhibit 2.0 (Service Model).

Figure 1 - MCS Other Government Entity Operating Model below depicts the relationships between Customers, SCPs, and SFAA.

Figure 1 – MCS Other Government Entity Operating Model As the overall service owner for Other Government Entity-consumed services including Managed Cloud Services (MCS), the Service Provider is to provide services at the Customer Engagement, Service Management, and SCP operating model levels. The Service Provider is to provide high-quality delivery of MCS services and objective performance transparency to enable each Other Government Entity the ability to self-govern the functional and financial aspects of their consumed MCS services. For these services, the Service Provider provides operations functions that provide customer relationship management, service catalog and service desk services, IT service management services, and integration with the Customer’s information security functions. The Non-Shared Service Customers are responsible for their own cloud security and incident response with support from the Service Provider.

Cloud Services The Other Government Entity Cloud Service offerings include Base Managed Cloud Services, available to all Other Government Entities that elect Cloud Services, and Optional Managed Cloud Services, as elected by the Customer. Table 1 Other Government Entity Cloud Services summarizes the high-level services the Service Provider is to design, deploy, and deliver either as base services or optionally.

Table 1 – Other Government Entity Cloud Services

Customer to CSP Network Connections

Optional

Networking within CSPs

Optional

RFS - Cloud Solution Design & Project Mgt.

Optional

Managed Technical Support (bundle)

Optional Bundle

Operations Monitoring & Reporting

Optional Bundle

Performance and Capacity Management

Optional Bundle

Backup and Recovery Services

Optional Bundle

Disaster Recovery Services

Optional Bundle

Real-Time Forensics Support Services

Optional Bundle

Virtual Machine Middleware Services

Optional

Virtual Machine Administration

Optional

Cloud Native Middleware Services

Optional

Distributed Denial of Service Protection Services

Optional

Database Management Services

Optional

Technology Roadmap

Optional

In the above Table 1 - Other Government Entity Cloud Services, the following meaning is intended:

1. Scope flag:

a. Yes - In-scope Services provided by Service Provider.

b. Yes, qualification - In-scope Services the Service Provider delivers only applies to specific criteria.

c. Optional – Services provided by the Service Provider that are optional as elected by a Customer.

Base Managed Cloud Services Cloud Service Provisioning and Management One of the goals of public cloud is to allow Customers to provision and consume services in the public cloud as natively as possible with technical, delivery, and security assurances. In accordance with the Services provided for Base Managed Cloud Services, the Service Provider shall, at a minimum:

1. Manage and administer initial access to the cloud services, systems, networks, operating software, system files, and Customer’s data, including:

a. Enable role-based access to public cloud services and functions for Customer and enterprise user roles.

b. Assign initial and, as requested, reset existing privileged account passwords per established procedures.

c. Provide Customer access to create user IDs, suspend and delete inactive logon IDs, research system security problems, and maintain network access authority.

2. Provide cloud service provisioning with State-approved Cloud Service Provider (CSPs) in accordance with the Services provided for Base Managed Cloud Services to support service provisioning and billing for each approved CSP, including, but not limited to:

a. Accounts, subscriptions/organizations, organizational units, resources/services, etc.

i. Other Government Entity services will be deployed in the Customer’s tenant.

b. Communicate the approved information electronically with the requesting Customer, enabling the Customer to access and use the provisioned service.

c. Support the Customer with questions regarding how to gain access to the provisioned service.

d. Provide Customers with the ability to provision and support services aligned with public cloud capabilities by leveraging native interfaces.

3. Develop and maintain, within the SMM, Customer-facing and internal operational documentation on the steps to request, provision, and support the provisioning of cloud services for Customers.

4. Implement access controls to services and operational functions based on defined Customer and Service Provider support responsibilities.

5. Integrate Service Provider Services with CSP native console and/or service catalogs, including:

a. Provisioning access to allow fulfillment of Customer provision requests based on templates aligned with State standards and policies, and/or application tools available from CSPs.

b. Enable deployment of the templatized cloud infrastructure with role-based security and automated fulfillment, leveraging public cloud best practices delivered with the Service Provider’s service catalog integration.

i. Role-based security shall be delivered based upon Customer requirements and Cloud Services, within defined State policies, and governed by the principle of “least privilege,” required to balance Customer requirements with program goals effectively.

6. Provide the ability to integrate provisioning and management through automated orchestration and integration of CSP service catalogs (e.g., custom catalogs).

7. Provisioning infrastructure components based on State-defined and Customer-approved reference architecture standards, including compute, network, storage, and security resources.

8. Extend cloud service capabilities by integrating with cloud native, cloud-agnostic, and third-party tools.

9. Ensure that security and certificates meet State program requirements.

10. Implementing IT Financial Management (ITFM) billing and chargeback of cloud services consumed directly to the Customer, including integrated tagging of services, products, and components for tracking chargeback.

11. Develop cloud service tagging automation and orchestration to inform service management, billing processes, and capabilities.

12. Perform necessary service integration with CSPs to ensure necessary brokering and orchestration can occur in an automated fashion to provision service requests successfully.

Provisioning Automation The Service Provider shall enable continuous integration and automated testing of cloud service provisioning deployments using Infrastructure as Code (IaC), including but not limited to:

1. Provide, deploy, and manage IaC using State-approved tools to support cloud services' deployment, provisioning, scaling, networking, and security.

2. Automate service deployment, operational monitoring, and security monitoring using IaC concepts.

3. Comply with established reference architecture standards and standard products for IaC deployments.

a. Where these standards may not exist or contain a complete view of best practices, the Service Provider shall recommend updates to the documentation owner.

4. Document all IaC processes, procedures, and Source Code documentation, aligning with the SMM.

5. Implement a data protection solution to ensure the backup and recoverability of IaC configurations in a State-approved Source Code library.

6. Develop rollback procedures as applicable for restoration to previous configurations.

7. Perform quality assurance testing for IaC deployments.

8. Utilize best practices relating to IaC, including but not limited to:

a. Develop and deploy IAC using templates, images, and stacks that define the resources and interconnections.

b. Maintain detailed documentation to ensure resources and properties, metadata, parameters, mappings, conditions, and outputs are defined.

c. Provide transferable solutions and approaches to IaC that minimize vendor lock-in and protect the State against an over-reliance on a single vendor solution.

Base Managed Cloud Security Services State Customers are recommended to adhere to the Admin SCDIS-210 publication security framework and guidelines. In support of Customers, the Service Provider shall, at a minimum:

1. Provide recommended configurations and support to Customers in adherence to the SCDIS-210 Information Security Technology Coverage Measurement Standards published by Admin.

a. Provide, as requested by Customers, the capture and routing of security logs to the Customer-designated SIEM.

b. Provide support, as requested by Customers, for research and resolution of security incidents.

2. As requested by Customers, evaluate the security controls and data protection standards implemented in public cloud services and report on performance and compliance requirements adherence to SCDIS-200 Information Security and Privacy Standards as published by Admin, and CIS Benchmarks as approved by Admin.

Optional Managed Cloud Services As requested by Customers, the Service Provider will offer optional services.

Network within the CSPs As an optional service, deploy and manage networking within CSPs to enable Customers to connect to CSP services securely.

Service Provider shall, at a minimum:

1. Architect virtual network reference architecture standards to define best practices, design principles, and security guidelines for setting up networking within CSPs that support Customer segregation, billing, audit, reporting, security, and operations management.

2. Create and manage Customer cloud environments using virtual network reference architecture standards as recommended by the State, aligning with and supporting customer segregation, billing, audit, reporting, security, and operations management.

3. Provide installation, testing, operating, troubleshooting, and maintenance of cloud virtual network solutions (e.g., Virtual Private Clouds (VPCs), virtual cloud network), compute resources, operating system(s), configuration items, software, services, and other elements that comprise the Service.

4. Implement zone-based network security (e.g., VPCs, Virtual Networks, Subnets) with stateful firewall filtering between isolated networks, utilizing stateful packet inspection to manage packet flows. This ensures that only packets from known active connections can pass.

5. Provide cloud virtual network reporting (e.g., subnet in use, port mapping, internet usage).

6. Provide a solution to control network access to cloud services and associated functions based on State-approved policies and customer requirements. (i.e., ability to restrict/grant appropriate services access based on State-approved services, support models, and reference architectures using features like cloud-native firewall and network security groups).

a. The solution shall be auditable and reportable, enabling identification of levels of Service Provider support services.

7. Provide and support standardized virtual network (or comparable) build automation, as well as the ability to support the utilization of existing VPCs when required for new compute build requests.

8. Provide and support the ability to deploy custom virtual network solutions (e.g., support VNet/Subnet counts and sizes).

9. Request available IP address ranges from the Customer and allocate and manage IP addresses in accordance with State IPAM strategies, processes, and standards to support public cloud services.

10. As requested by the Customer, where needed to resolve Customer overlapping IP address issues, develop and deploy a Network Address Translation (NAT) solution as needed within CSPs.

Request for Solution (RFS) Requests for Solution (RFS) are Customer requests where requirements are captured, and the Service Provider develops solutions and cost estimates for Customer review and approval. These solutions typically assume the Service Provider builds and implements the solution. For Customer Requests that require the Service Provider to propose a solution, the Service Provider shall, at a minimum:

1. Support the State in developing and maintaining RFS processes in the SMM, including supporting systems and appropriate mechanisms for fulfilling complex requests that require design, pricing, solutions, and proposals. This includes clear and effective communication to set expectations and promote excellent customer service.

2. Support the State in developing and maintaining RFS ROM processes in the SMM, including appropriate mechanisms to support rapid proposal development that provides accuracy for budgetary information without requiring a complete solution (e.g., rough order magnitude (ROM) pricing and high-level architecture).

3. For all RFSs requested by the Customer:

a. Review RFS to validate for completeness.

b. Coordinate and lead meetings to review requests, gather requirements, solution, and develop the proposal.

c. Coordinate the attendance of all necessary subject matter experts in solution and requirement gathering sessions.

d. Once requirements are complete, provide a timeframe for delivering the solution proposal, including cost estimates.

e. Develop the solution, including the technical solution, effort, acceptance criteria, solution design document, and pricing.

f. Develop the solution to conform to the State-approved architecture, standards, and pricing.

g. Develop the solution in conformance with the State Security Policies, procedures, and guidelines of Customer and the State.

h. Develop the solution in conformance with State-approved security baselines.

i. Develop the solution in conformance with the boundaries of the State technical guidelines.

j. Coordinate and facilitate solution reviews across the Service Provider as required to review and gain approval for the solution and pricing.

k. Track all Project Change requests in accordance with established procedures in the SMM.

l. Provide a single proposal to the requesting Customer as required.

m. Iterate and adjust the solution and cost estimating template to adhere to the requesting Customer’s feedback and requirements.

n. Document Customer approvals in accordance with established processes as per the SMM.

o. Gather and validate that the proposal acceptance comes from an appropriately authorized user.

p. Provide status to Customers for all outstanding requests.

q. Initiate Project Management as appropriate upon acceptance of the proposal by the Customer to implement approved proposals.

Customer Virtual CSP Connection As an optional service, as requested by Customers who do not have connectivity to CSPs, solution and provide a service that offers Customer virtual connectivity from the Customers network to the CSP services.

Service Provider shall, at a minimum:

1. Design, implement, monitor, and manage cloud access using VPN to enable connections to CSPs.

Customer Direct CSP Connection As an optional service, as requested by Customers that do not have connectivity to CSPs, solution and provide a service that offers Customer direct network connectivity from the Customers network to the CSP services.

Service Provider shall, at a minimum:

1. Design, implement, monitor, and manage cloud access using a Virtual Cross Connect (VXC) direct connect solution to enable Customers to connect securely to CSPs.

2. Provide an optional service to establish and maintain a VPN connection within the VXC direct connect solution.

Managed Technical Support Managed steady-state services include responsibilities and activities to support public cloud products and corresponding workloads, services, processes, systems management roles, virtual networks, microservices, container technologies, serverless architectures, and other public cloud services. The Service Provider shall, at a minimum:

1. Through ongoing support and maintenance control processes, support the current implementation and solutions developed by the Service Provider during the Agreement term.

2. Manage the implementation of public cloud products, services, and related technologies to support required business applications.

3. Install tools and processes to enable delivery of all in-scope cloud services, operations, monitoring, systems management, event response, and service restoration for the in-scope environment.

4. Perform 24x7 operations monitoring, systems management, event response, and service restoration for the in-scope cloud services.

5. Execute agreed-upon State Incident and Problem Management processes as defined in the SMM, and interface with Customer teams to support Resolution and service restoration.

6. Resolve all events, warnings, and alarm messages and notify Customers as appropriate.

7. Support cloud-native, cloud-agnostic, and third-party tools and products to support defined Services.

8. Define and provide technical support for cloud services, in accordance with the SMM for operations, including:

a. OS administration.

b. Patch management.

c. Virtualization of resources.

d. Storage management.

e. Backup and recovery.

f. Disaster Recovery.

g. Virtual server support.

h. Install/Move/Add/Change (IMAC).

i. Capacity planning and reporting.

j. Performance tuning.

k. Problem resolution and Root Cause Analysis.

9. Verify that deployed cloud services are compliant with State baseline standards.

10. Provide cloud service performance optimization, capacity management, and auditing of configurations to ensure alignment with State-defined standards. The Service Provider shall notify the Customer where deviations exist.

11. Perform all measurements and reporting for public cloud-hosted platforms as required to support Services as defined in Exhibit 3.0 (Performance Model).

12. Comply with established reference architecture standards and standard products, including deployed software and Services within the managed service.

13. Ensure an auto-scaling design for the following:

a. Tools integration.

b. Host uniqueness.

c. Backup reporting.

d. Billing.

e. Tagging.

f. Compliance implications.

g. Naming standards.

h. Auditability of these Instances.

14. Provide technical advice and support to Projects, applications, application development, and service operations teams as required.

15. Enable and manage native public cloud services for dynamic scaling to meet overall capacity needs, including auto-scaling groups where necessary.

16. Design and implement user and system security measures in alignment with defined security policies.

17. Comply with the requirements of Customers for Configuration Management items in the Service.

18. Follow State-defined Incident Management processes, and interface with the Customer process.

19. Design, implement, and manage capabilities to enable cross-cloud migration techniques to ease system migration and re-platforming.

20. Implement and manage an automated software deployment and patching set of procedures and tooling, leveraging cloud best practices.

21. Identify and implement required service or configuration changes to address solution defects.

22. Maintain service documentation (technical specifications and testing documentation) as well as common problems, root causes, and remedies to aid in identifying and remedying underlying system incidents.

23. Participate in applicable acceptance testing or review of any changes arising from break/fix or patch/release performed by the Service Provider.

24. Verify and ensure compliance with any security-mandated patches, configuration settings, or system levels required, given the nature of the security mandate, and report to the Customer, in writing, any risks or issues that the Service Provider becomes aware of while providing the Service to the Customer.

25. Assist Customers by referring incidents to the appropriate third-party entity for resolution and coordinating with the third-party service to help minimize the Customer's role in problem management.

26. Notify the Customer of Service Provider’s planned and reactive maintenance activities, including service-impacting and non-service-impacting operations, as appropriate.

27. Assess the utilization of all MCS assets (e.g., public cloud products, platforms, storage, network interface points) within the defined lifecycle services within the Service Provider’s Services, emerging products, and capabilities to deliver Services more effectively. Drive the overall consistency and reliability of the Service through:

a. Simplification of service offerings and support tiers to move the Service to a support model that is highly repeatable and reliable.

b. Implement repeatable templates, automation, and programmatic orchestration to drive initial quality in service implementations (i.e., “right first time”) and reduce Incident, Problem, and Change Management service requests wherever possible.

c. Automate data collection to drive better and more timely data collection to facilitate Service decision-making, cross-functional coordination, and long-term planning.

28. Actively support the demand management process by developing technical solutions, including intra-tower solution development, as initiated through Request for Solution procedures.

29. Develop and maintain support for cross-platform cloud support solutions (e.g., Pivotal, OpenShift, etc.)

30. Upon request, perform multi-homing of installed tooling alerts to include Customers' systems where feasible.

31. Assess CSP services in the public cloud environment to ensure alignment with financial and service optimization activities.

32. Leverage the approved Customer mechanism to enable the tracking, management, and implementation of security certificates used to secure confidential sessions (e.g., SSL) for Internet and Intranet transactions and communications, including:

a. Establish processes and procedures for renewals, as required by Customers.

b. Schedule, apply, and support security certificates as the Customer requires.

c. Coordinate and advise Customers regarding certificates that are embedded in Applications.

d. Notify and advise Customers of certificate renewals.

33. Service Provider shall provide system administration assistance with Customer Software installations, upgrades, and patches. Service Provider shall, upon request, assist in the installation, patching, and upgrades of Customer Software. In these instances where Customer requests support, Customer shall provide work instructions for requested activities per the SMM.

34. Maintain MCS-approved service listing as instructed in the SMM, including developing automations for improved service and data quality management.

35. Monitor services for deprecation notices and proactively provide upgrade planning and implementation support to ensure services remain at supported levels and API versions.

36. The Service Provider shall provide storage management for all in-scope service elements, including but not limited to:

a. Configure, monitor, and manage a robust and highly available cloud-native storage solution to meet the overall needs of Customers.

b. Perform storage, backup, and restore service testing, and maintain the backup system and service documentation.

c. Implement cloud-native platform-level encryption on all Customer storage in accordance with State standards and security baselines. Due to data classification and compliance requirements, implement and support customer-managed keys for encryption where necessary.

Operations Monitoring The Service Provider shall provide operations monitoring services and support for Customer-elected environments and Services such as virtual networks, microservices, serverless architectures, and native middleware services, with responsibilities including, but not limited to:

1. Install, configure, and provide ongoing CSP services monitoring and reporting tools and services.

2. Perform operational monitoring, including real-time mechanisms for monitoring systems, including but not limited to availability, auto-scaling, performance, capacity, and overall environment health as defined in the SMM and required to support Service Levels as defined in Exhibit 3.0 (Performance Model).

3. Integrate operational monitoring with the Service Provider ITSM platforms to allow electronic event and Incident Management integration and provide real-time systematic notification of performance issues and events.

4. Conduct 7x24x365 operational monitoring using CSP native health monitoring tools (e.g., Azure Monitor, Azure Resource Health, Azure Application Insights, Amazon Cloud Watch, AWS Service Health Dashboard) and third-party monitoring tools (e.g., Datadog, Splunk).

a. Perform service restoration activities of public cloud platforms, including real-time monitoring and reporting of capacity, stability, and performance of Services and platforms.

b. Establish and monitor proactive alarms in accordance with thresholds defined in the SMM.

c. Provide end-to-end visibility to Customers to view performance statistics (real-time and historical) on public cloud platforms.

5. Provide full monitoring integration with Customer operational support systems (e.g., Customer SIEM).

6. Provide status and trending reports to Customers as required, including:

a. Reports listed in Exhibit 3.3 (Reports).

b. Other Reports as required in this Exhibit 2.1.2 (Managed Cloud Services SOW - Other Government Entities), Exhibit 4 (Business Model), and related attachments.

Performance and Capacity Management To ensure that Services achieve the desired business and service-based outcomes, the Service Provider shall, at a minimum:

1. Develop performance and capacity management processes.

2. Actively participate in the exchange of data and information amongst Customers to ensure successful delivery of Services, including the ability to validate capacity planning.

3. Integrate performance and capacity management tooling and process outputs with other Service Management processes and systems.

4. Conduct performance and capacity planning and management activities for all supported products, platforms, and applications.

5. Participate in scheduled capacity planning meetings.

6. Create, update, and report a Capacity Plan that includes:

a. Developing an agreed-upon formula for measuring capacity.

b. Services, components, and resources measured.

c. Current, trending, and forecasted capacity based on technical planning and demand management.

d. Detailed performance and consumption characteristics.

e. Identify workload landscape (Prod, Test, etc.).

f. Risk areas (including over- and under-capacity or approaching quota limits).

g. Actual consumption compared to plan.

7. As required, provide performance and capacity technical advice and support to projects, applications, application development, and database teams.

Backup and Recovery Services Service Provider shall be responsible for backup and recovery of enrolled Customer Services. In establishing, monitoring, and managing backup/restore operations, the Service Provider shall, at a minimum:

1. Design, install, monitor, and manage backup policies and services to satisfy the overall needs of Customers within the public cloud, including support for virtual machines, storage, snapshots, virtual grids, virtual tape libraries (VTL), volumes, databases, tables, files, and filesystems.

2. Design and manage storage, file, and volume gateway solutions.

3. Develop backup and recovery-related management processes and automation.

4. Assess, develop, and formally recommend opportunities to reduce (or avoid) costs associated with a backup environment.

5. Perform environment/supported backup tuning, job, and environment restructuring, and provide tools and other efforts to help improve the efficiency and reliability of storage and backup operations and to help reduce ongoing maintenance requirements.

6. Maintain backup environments in accordance with Customer strategies and standards relating to technical, data, and applications architectures as agreed upon in this Exhibit, and as required by Customer projects, environments, the SMM, or other supporting documents.

7. Establish, publish, and maintain a production backup calendar, including daily and periodic backup and service maintenance activities.

8. Generate and provide access to monthly summary reports that track the progress of the CSP and Service Provider’s backup and service maintenance work performance.

9. Perform ad hoc backup/restore operations reporting as agreed by the parties.

10. Meet with Customers at least once a year to review existing backup schedules for suitability and identify any required updates to existing backup schedules and retention policies.

11. Monitor and remediate backup failures on the Service Provider's standard backup service.

12. Perform backup restoration testing.

13. Install, configure, and manage file-level backup agents and services where appropriate.

14. Backup all databases deployed within the Customer’s environment using database-aware technologies.

15. Encrypt all backups, regardless of media and location, to ensure compliance with the SC DIS 200 security framework or subsequent versions.

16. Implement and monitor backup and restore services operations for all regions and availability zones.

17. Update the backup platforms and services as new tools and technology are available to improve the Customer’s business processes and performance.

18. Verify backup outcomes correctly achieve service delivery requirements for Service Provider responsible data (i.e., operating system recoverability).

19. Provide an auditable solution for Customer to view the schedule, retention, and target information as configured in the backup systems, with the ability for Customer to correlate to Customer’s requested requirements.

20. Provide reporting on backups and backup infrastructure (e.g., success/failure, schedules, retention, targets, archive, capacity, performance, storage media types, and storage location).

21. Perform backup administration and monitoring, including:

a. Verify backup jobs start as scheduled.

b. Monitor scheduled production backup jobs.

c. Perform job restart, as necessary, in accordance with resolution and restart procedures.

d. Resolve backup scheduling conflicts.

e. Ensure that failed backup jobs are restarted once the failure condition is identified and resolved.

22. Monitor scheduler-related incidents and develop and recommend changes to the scheduler database.

23. Schedule backup jobs, as requested by the Customer, that require expedited or ad hoc execution.

Disaster Recovery Services Customers will be solely responsible for overall business continuity and application recovery plans. The Service Provider retains responsibility for business continuity plans for Service Provider Services and, if requested by the Customer, supports the Customers with the recovery of cloud services. Service Provider Disaster Recovery (DR) does not apply to non-native cloud middleware and application software configurations, application presentation, or customizations, and is limited to in-scope cloud services, unless otherwise agreed upon with the Customer.

The Service Provider shall, at a minimum:

1. Design and deploy solutions, via the RFS process, that align with the defined DR class requirements as required by Customers, and complement Customer activities in support of Customer and business continuity plan(s), including:

a. Leverage region failover and availability zones.

b. Document procedures to restore primary operations.

2. Monitor and manage recoverable cloud services within the public cloud to satisfy the Customers' needs.

3. Upon Customer request, participate in Customer DR planning sessions, DR testing sessions, or actual DR recovery efforts for in-scope service elements.

Real-Time Forensics Support Services Upon Customer’s request, Service Provider shall secure, acquire, and preserve evidence data within the public cloud, and allow third-party forensic teams to perform analysis without potentially losing critical data. If requested, the Service Provider shall, at a minimum:

1. Implement and manage a forensics capability and support within any defined public cloud hosting environment.

2. Support a third-party at the Customer’s direction, in securing, acquiring, and preserving evidence data within the public cloud hosting environment.

3. Provide a mechanism for a third-party forensics team to access collected data for analysis.

Virtual Machine Middleware Services As an optional service, requested and approved by the Customer, the Service Provider is responsible for monitoring and supporting Middleware (e.g., Microsoft .NET Framework, Apache Kafka, IBM WebSphere, etc.). The Service Provider shall, at a minimum:

1. Perform monitoring of defined Middleware environments.

2. Provide administrative support for enrolled Middleware services.

3. Provide effective technical support (e.g., patching, event resolution, advice, Third-Party Vendor coordination).

4. As appropriate for state entities, integrate with the Customer’s security monitoring and access control management standards (e.g., SIEM, Identity and Access Management (IAM)) as requested by the Customer.

5. Support Customer application developers in supporting Middleware runtime environments.

6. Assist the Customer in managing the lifecycle of provisioned Middleware environments, including specification, installation, implementation, monitoring, management, backup/restore, and disaster recovery of Customer Middleware environments.

7. Monitor and manage the Middleware systems components on a 24/7 basis to ensure environments meet performance standards.

8. Alert defined resources if process, application, system events, or thresholds have been exceeded. This notification will occur based on defined SMM procedures and the agreed escalation matrix.

9. Monitor capacity and proactively provide performance and capacity planning.

10. Monitor services for deprecation notices and proactively provide upgrade planning and implementation support to ensure services remain at supported software levels.

Virtual Machine System Administration The Service Provider shall, at a minimum:

1. Provide systems management and administration services for all in-scope Services.

2. Perform installation, patching, and upgrades of Customer software upon request.

3. Install productivity tools/utilities and perform all required operational modifications for the efficient and proper delivery of the Services.

4. Execute the Operating System’s maintenance, patching, and support, including shared libraries, print, .NET, FTP, mail services, etc.

5. Identify, test, coordinate patching, and provide other updates associated with the supported operating system(s), configuration items, software, and other elements that comprise the Service.

6. Implement additional security-related fixes associated with the operating system(s), configuration items, software, and other elements that comprise the Service.

7. Release upgrades for packaged infrastructure software initiated through scheduled releases, including, but not limited to, operating systems, patches, virus scanners, etc.

8. Work with Customers to identify required access for system administration activities within Customer environments.

9. Enable console access for Customers to have visibility into available and consumed Services for effective resource management.

a. This includes API and command line interface account access.

b. These accounts should be governed by the “least privilege” principle required to support requirements to perform work responsibility effectively.

10. Assist in analyzing and correcting endpoint and/or network Incidents and Problems associated with cloud Service processing.

11. Provide technical support for virtual machines, storage, and networking environments in the delivery of Services.

12. Ensure that all solution delivery elements are maintained at currency levels within vendor standard support levels.

13. Support Instance Schedulers in configuring custom start and stop times within public cloud services to assist Customers in reducing operational costs.

14. Provide public cloud tagging enablement for Service Provider-managed tags based on predetermined rules and Customer-managed tags.

15. Actively work with Customer to ensure that tagging includes relevant CI details as defined by the SMM.

16. Ensure all public cloud-hosted instances are tagged with necessary details to support billing, audit, reporting, security, and operations management to meet Customer requirements.

17. Ensure secure public cloud-hosted instance tagging governance policies are in place to prevent unauthorized modification of tags.

Cloud Native Middleware Services Service Provider monitors and supports CSP-offered cloud-native Middleware (e.g., Azure Integration Services, Azure Event Grid, AWS SQS). The Service Provider shall, at a minimum:

1. Provide administrative support for enrolled Middleware services.

2. Provide adequate technical support (e.g., patching, event resolution, advice, Third-Party Vendor coordination).

3. Integrate with Customer security monitoring and access control management standards (e.g., SIEM, Identity and Access Management (IAM)) as required by the Customer.

4. Support Customer application developers in supporting Middleware runtime environments.

5. Assist the Customer in managing the lifecycle of provisioned Middleware environments, including specification, installation, implementation, monitoring, management, backup/restore, and disaster recovery of Customer Middleware environments.

6. Monitor and manage the Middleware systems components on a 24/7 basis to ensure environments meet performance standards.

7. Alert defined resources if process, application, system events, or thresholds have been exceeded. This notification will occur based on defined SMM procedures and the agreed escalation matrix.

8. Monitor capacity and proactively provide performance and capacity planning.

Distributed Denial of Service Protection Services If requested by the Customer, as an optional service, the Service Provider shall, at a minimum:

1. Provide intrusion prevention system (IPS) and intrusion detection system (IDS) services.

2. Manage intrusion detection, prevention, and distributed denial of service (DDOS), including prompt Customer notification of such events, reporting, monitoring, and assessing security events.

3. Work with Customer to support the denial of communications to and from known malicious IP addresses.

4. Ensure the public cloud network architecture separates internal systems from DMZ and extranet systems.

5. Require remote login access to use two-factor authentication.

6. Support Customer’s monitoring and managing devices remotely, logging into the internal network.

7. Support Customer in configuring firewall session tracking mechanisms for addresses that access the public cloud.

Database Management Services Database management is an optional service that Customers may request. The Service Provider shall, at a minimum:

1. Design and deploy database components and services with appropriate high availability and resilience configurations to meet the requirements.

2. Complete management of the lifecycle of provisioned database environments, including implementation, monitoring and management, encryption, backup/restore, data protection, restricted data identification, disaster recovery, data replication, and decommissioning (e.g., archive, deletion, destruction) of Customer databases.

DBMS Administration and Operations If elected by the Customer, the Service Provider shall, at a minimum, administer the following within the Customer database platforms and in compliance with the required SLAs per Exhibit 3.2 (Service Level Definitions):

1. Develop, document, and maintain database standards and procedures.

2. Perform monitoring, management, and operational support of databases.

3. Perform environment creation.

4. Perform database refreshes.

5. Perform database exports.

6. Maintain and follow OEM-developed software standards.

7. Provide Customers with system-level performance statistics commonly available through Microsoft and AWS-provided system consoles and functions, including operational performance statistics summarized per Customer.

8. Provide production environment performance tuning based on Customer direction, including tuning scripts or instructions.

9. Perform database defragmentation upon the Customer's request, should the Service Provider determine that proactive defragmentation is advisable.

10. Monitor and provide alerting for the Customer database components on a 7x24 basis to ensure databases meet performance and availability standards.

11. Monitor all database objects (e.g., database, file system servers/services, static data, log and table spaces, indexes) and work with Customers to identify and report all databases containing restricted data types.

12. Configure, schedule, and execute backups to storage per Customer requirements.

13. Upon receipt of an authorized Customer service request, restore and recover the database and data/files in accordance with the applicable service schedules.

14. Provide the capability to archive and purge historical or obsolete data from production environments, including databases, files, logs, swap space, and other data stores prone to accumulating historical data.

15. Patch all database components and system/platform software.

16. Implement OEM technology stack updates (patches, updates, fixes, etc.

17. Provide OEM software server startup and shutdown functions using the provided scripts.

18. Perform database performance analysis.

Technology Roadmap As an optional service, if the Customer requests, the Service Provider shall support the Customer in reviewing and creating a multi-year cloud service roadmap, including all projects, optimization, and transformation initiatives to achieve the Customer’s cloud objectives as requested.

As directed by the Customer, the plans may include:

1. Schedules, dependencies, and requirements for introducing Customer cloud technology changes using State cloud services, including acquiring, supporting, and retiring software and hardware.

2. Specification of the solutions, outcomes, benefits (i.e., Financial, Operational), and schedules for achieving Customer technology evolution goals for State cloud services.

3. Recommendations and adjustments to the plan from prior year plans.

4. Provide and review with the Customer a solution accelerator library with frameworks, templates, tools, and methodologies to streamline and accelerate solution documentation and implementation.

5. Support and make available technology standards that support the Customers cloud technology roadmap, including as requested by the Customer:

a. Make available the description of Services and offerings by provided by the Service Provider.

b. Make available standards for supporting open-source software.

Service Management Service Management Systems Generally, for Other Government Entity-provided services, the Service Provider shall be responsible for using Service Provider-provided service management systems and complying with the ITIL framework in cooperation with and as established by the Service Management Manual (SMM). The service management functional capabilities and high-level responsibilities for providing the platform and processes to support the capability (tools), providing the staffing to support the capability operations (operate), ensuring the capability is performing in accordance with targets (manage), and the responsibilities to approve policies, and manage cloud-program level service management decisions (govern), are outlined below.

Table 2 - Service Management Capabilities

(tools may change) Provide Tool (provide Systems and Processes) Operate (provide staffing and perform a role) Manage (oversee and ensure performance) Govern (approve policies, manage program-level decisions)