Attch J1 CRIMS SLA Draft.docx

Revision History

Page | 2

Table of Contents

Page | 2

LIST OF TABLES

1.0 Introduction

This Service Level Agreement (SLA) describes the levels of services CONTRACTOR [TBD] will provide to National Library Service for the Blind and Physically Handicapped (NLS) for the Currency Reader Information Management System (CRIMS).

1.0.1 Terminology

The APPLICATION shall refer to the Azure environment, the CRIMS and/or the KTA application for the purpose of this SLA.

1.1 Parties Involved

National Library Service for the Blind and Physically Handicapped (NLS)

Washington, DC 20542

[TBD] (Contractor)

1.2 Period of Performance

This agreement begins on the date signed below and will be in effect until the end of the contract per the statement of work (SOW).

1.3 Security Directive Compliance

CONTRACTOR shall be responsible for ensuring the APPLICATION adheres to the Library of Congress Security Directive1 that is received at the time of signing this document.[footnoteRef:1] NLS shall monitor the security directives and, in the event of a rule change, modify the APPLICATION to support the new security directive. [1: Security Directive 5-410-1_v3.2]

1.4 Time Zones

Any times listed in this document will be in a 12-hour format and are considered Eastern Time zone.

1.5 Time Windows

Any operational times, performance, and/or monitoring metrics will be Monday through Friday from 6:30am to 9:00pm (NORMAL WORKING HOURS). This excludes Federal holidays as defined by the Office of Personal Management.[footnoteRef:2] [2: https://www.opm.gov/policy-data-oversight/snow-dismissal-procedures/federal-holidays/

2.0 Support Services

CONTRACTOR shall provide the services described herein to maintain the APPLICATION in good working order, keeping it free from material defects so that the APPLICATION system shall function properly and in accordance with the accepted level of performance as set forth in this service agreement.

2.1 Help Desk

CONTRACTOR shall maintain a web-based helpdesk system (the “helpdesk”) that will operate during NORMAL WORKING HOURS. The URL for the helpdesk is: [TBD]

2.2 Submission of Requests

A helpdesk request (TICKET) is required for all requests including defects, changes, or maintenance requests. CONTRACTOR will provide any reports, metrics, and/or alerting (unless otherwise listed in a section) via the helpdesk system. Requests will not be accepted via any other method; CONTRACTOR will not be held accountable for any request outside that process.

2.3 Process for Support

CONTRACTOR and NLS shall adhere to the following process for submission and resolution of remedial support:

1. Request is submitted to CONTRACTOR helpdesk.

2. CONTRACTOR categorizes the request based on information provided.

3. CONTRACTOR will determine a resolution for the issue.

4. CONTRACTOR and NLS will test the resolution.

5. CONTRACTOR will initiate a request to the Change Control Board (CCB) to approval the fix.

6. CONTRACTOR will follow the CCB process, following approval CONTRACTOR will implement fix.

2.4 Categorization of Requests

2.4.1 Urgent

Produces an emergency in which the APPLICATION is inoperable, produces incorrect results, or fails catastrophically. The resolution will be delivered to NLS as a work-around or as an emergency fix for the APPLICATION. If CONTRACTOR delivers an acceptable work-around, the severity classification will drop to a High priority.

2.4.2 High

Produces a detrimental situation in which performance (throughput or response) of the APPLICATION degrades substantially under reasonable loads, such that there is a severe impact on use; the APPLICATION is usable, but materially incomplete; one or more mainline functions or commands is inoperable, or the use is otherwise significantly impacted. There is a work around, but the workaround is difficult. The resolution will be delivered to NLS as a work-around or as an emergency fix for the APPLICATION. If CONTRACTOR delivers an acceptable work-around, the severity classification will drop to a Normal priority.

2.4.3 Normal

Produces an inconvenient situation in which the APPLICATION is usable but does not provide a function in the most convenient or expeditious manner, and the user suffers little or no significant impact. The resolution will be delivered to NLS as a fix in the next maintenance release.

2.4.4 Low

Produces a noticeable situation in which the use is affected in some way which is reasonably correctable by a cosmetic change. The resolution will be delivered to NLS as a fix in the next maintenance release.

2.5 Support Response Times

The following are the response times for TICKETS during NORMAL WORK HOURS.

Table 2-1: Ticket Levels and Response Times

2.6 Training

Any personnel from CONTRACTOR working on the APPLICATION is required to complete training that NLS requires; including but not limited to Incident Response (IR) training requirements. For the purposes of this engagement any such training required will not exceed 2 hours per quarter per person.

3.0 Change Management

Any change to the APPLICATION shall be approved by the NLS CCB and follow the approval process below, unless otherwise noted.

3.1 Process for Change Approval

CONTRACTOR and NLS shall follow the following process for requesting CCB approval:

1. Submit a request to the NLS CCB Facilitator

2. Request reviewed by CCB evaluation team

3. If approved by the CCB evaluation team, the request is sent to the CCB for review

4. CRIMS CCB votes on request either in a formal meeting or electronically.

5. CONTRACTOR acts based on the vote of the CCB. NLS accepts any risk for changes requested by CONTRACTOR but rejected by the CCB.

4.0 Backups and Disaster Recovery

The APPLICATION has a 71-hour recovery time objective (RTO) with a 24-hour recovery point objective (RPO). A disaster is termed as the total loss of data, code, or files in the APPLICATION in the production environment.

4.1 Daily Backups

Daily incremental data backups will be executed outside of NORMAL WORKING HOURS to copy the data changes from the APPLICATION’s Production database to the APPLICATION Disaster Recovery (DR) environment database.

4.2 Weekly Backups

A full data backup will be executed every week on Sunday, starting at 12am CST. This backup will include a full backup of the APPLICATION’s Production database to the APPLICATION’s DR database. backup.

4.3 Delivery of Backups

At the completion of the Daily or Weekly backup, CONTRACTOR shall deliver a copy of the backup performed to the NLS Sun9 ingest server upon completion of the backup. The backup procedure will follow the “NLS Sun9 Backup Procedure” those files will be pushed to the NLS BACKUP SERVER using protocol below.[footnoteRef:3] It is the sole responsibility of NLS to maintain the NLS BACKUP SERVER, and to ensure that CONTRACTOR has the correct technical details to connect. If the NLS BACKUP SERVER is not available, the backups for that interval will be skipped. [3: General/CRIMS/CRIMS%20Sun9%20Backup%20Procedure.docx]

More information on how the backups is run on the APPLICATION can be found in Addendum A to this SLA.

4.4 DR Site

A standby copy of the APPLICATION will be maintained as a disaster recovery site. All changes applied to the production site will be applied to the DR site. In the event of a disaster, the website will be ready to use as soon it is pointed to a database, and the production URL is redirected to the DR site. The information in the APPLICATION’s DR DB will be 24 hours old. The APPLICATION’s DR database will be used as both a backup DB in the case of a disaster and a data backup in the case of a data loss in the Production DB. Should the NLS move to a Hot / Hot (or High Availability site) this section will no longer be enforced.

4.5 DR Tests

DR tests will be completed on a yearly basis. The test scenarios for the APPLICATION are:

The DR test process can be found in Addendum A this document.

More Information on a previous DR test with the APPLICATION can be found in Addendum B to this document.

5.0 Security

The APPLICATION is hosted in a FedRAMP moderate certified Microsoft Azure cloud instance as requested by NLS. In the event of a potential security breach or violation, Microsoft will notify CONTRACTOR, who will immediately notify NLS and work with NLS teams to evaluate the impact and next steps.

5.1 Application Security Requirements

Security measures have been built into the APPLICATION in accordance with NLS security requirements. CONTRACTOR will adhere to all applicable security compliance standards required by the Library of Congress detailed in the Information Technology Security Directive received at the time of signing this document.

5.2 Emergency Situations

In the event of an emergency arises, and it is necessary to take the system offline to prevent damage to its operation, CONTRACTOR shall attempt to notify the NLS personnel listed in this section. If CONTRACTOR is unable to reach the designated NLS personnel, CONTRACTOR shall have the authority to take the system offline to prevent further damage. Once the situation is resolved and CRIMS is online, CONTRACTOR shall provide a summary of the issue and resolution to NLS.

NLS is responsible for notifying CONTRACTOR of any changes to this list, CONTRACTOR is not responsible for sending alerts to outdated contact info.

Table 5-1: Emergency Point of Contact List

5.3 Audit Logs

CONTRACTOR will setup audit logs for the APPLICATION that records add/update/delete of all records within the APPLICATION, as well as all errors that occur. Records in the audit log will be kept for 180 days.

5.4 Firewall and IDS

CONTRACTOR will maintain a firewalled environment that only allows public access to what is needed for the APPLICATION users. Any other access will be protected from public access.

5.5 Continuous Security Scanning

CONTRACTOR shall conduct automated security scanning of the APPLICATION at a minimum of 1 time per month using tools such as Qualys. The results of those scans should be logged into a TICKET for all staff to review.

6.0 Monitoring

CONTRACTOR will provide a continuous monitoring system for all servers. CONTRACTOR will monitor performance of vital system processes and connections which ensure that performance metrics listed in section 7.0 are measured. In addition, CONTRACTOR will provide system level scanners (e.g. Anti-virus systems). Notices of performance issues should be auto logged within CONTRACTOR helpdesk system and closed with a resolution.

6.1 Daily Log Monitoring

CONTRACTOR will review the log file daily to check for suspicious or repetitive errors. If an issue is suspected the CONTRACTOR will create a helpdesk notice and notify the NLS Coordinators in Section 5.2 of the issue. The issue shall be classified, and the Section 2.3 support process shall be followed.

6.2 Weekly Performance Monitoring

CONTRACTOR will review the Azure performance reports including CPU Usage, Disk Read, Disk Write, Network In, Network Out, and Data Storage every Friday before 7pm. CONTRACTOR will create a helpdesk ticket and provide a weekly summary in or attached to that ticket. If average performance drops below the acceptable levels for the WEEK, the CONTRACTOR will create a helpdesk notice and notify the NLS Coordinators in Section 5.2 of the issue. The issue shall be classified, and the Section 2.3 support process shall be followed.

6.3 Monthly Performance Monitoring

On the first business day of every month, a report shall be compiled and provided to NLS of reported tickets, SLA performance, and system performance for the preceding month. This report shall be reviewed with the NLS Project Manager/Coordinator to ensure understanding and address any concerns.

6.3.1 Continuous Monitoring Support

NLS will submit any security directive changes to CONTRACTOR by 5pm EST on the 15th of every month (or the next business day in the event of holidays or weekends). NLS is responsible for providing all necessary information for the changes. Once all necessary changes are received by CONTRACTOR, they must notify NLS of the level of effort for those changes within 7 business days. If changes are not received as specified above, changes can be resubmitted on the next 15th.

The exact continuous monitoring metrics that are reported to The Library of Congress can be found in Addendum C to this SLA.

6.4 OS Level Security Scanning

On the first business day of every month, CONTRACTOR shall conduct an OS level scan and must provide the results of the scan to the Coordinators in Section 5.2. The results of the scan will be submitted to the CONTRACTOR helpdesk for customer review.

7.0 Performance and Capacity Metrics

NLS recognizes that there are many factors to the metrics and capacity defined in this section, some of which are outside of CONTRACTOR control including, but not limited to, 3rd party software bugs or cloud provider service issues). Items not defined in this document are considered outside of CONTRACTOR control. In those events, CONTRACTOR will provide diagnostic information to NLS. CONTRACTOR will work with NLS to determine the cause, propose a fix with time and cost estimate, and will follow all processes defined in this document.

7.1 Web Page Response

The following pages will be monitoring, via monitoring points, within the territory of the United States. Average rendering times during NORMAL WORKING HOURS for the endpoints defined in this section should be less than 5 seconds.

· https://crims.nlsbph.org/Account/Login

7.2 Storage Capacity

The mount point “C:\” will be monitored on all Windows based systems of the APPLICATION. Monitoring will be performed via an agent provided by the CONTRACTOR. Average capacity for the WEEK should be less than 95% Capacity (237.50 GB).

7.3 Cloud Service Availability

Cloud services should have a 99.99% uptime for NORMAL WORKING HOURS.

7.4 Network Ping

Ping times to any production server should be less than 50ms round trip.

7.5 RAM and CPU Usage

Daily average RAM and CPU usage should be below 70% during NORMAL WORKING HOURS.

8.0 System Patching

Any operating system, software, and/or application updates or patching must follow the process as defined below. Any patching will be conducted by CONTRACTOR during a planned maintenance window, with the exception instances defined in Section 8.1. System patching does not include major system updates (i.e. Windows 2016 to Windows 2019).

8.1 Patching Process

1. CONTRACTOR identifies the necessary change

2. CONTRACTOR creates a TICKET and document what is needed, what systems are affected, and why it is needed. The ticket description will denote whether the change is MINOR or a ZERO-DAY SECURITY PATCHING event. See Section 8.2 for Zero-Day Security Patching process.

3. NLS will “conditionally” approve the change, this change is done in a DEVELOPMENT environment by CONTRACTOR.

4. Once the update has been performed, CONTRACTOR will update helpdesk ticket asking for NLS review within 15 days.

5. NLS will update the TICKET with an approval or rejection. If NLS indicates a rejection, NLS assumes all risk associated with non-update.

6. On approval CONTRACTOR will schedule a maintenance window, in the PRODUCTION environment, as defined in Section 9.0.

7. CONTRACTOR will conduct the change during that time, and once complete will update the TICKET notifying the NLS and the ticket will be closed.

8.2 Zero-day Security Patching

CONTRACTOR shall have the ability to implement zero-day security related patches without CCB approval, when the patch represents a minor change with minimal impact to the system or when the need for the security patch is critical and a CCB meeting cannot reasonably be held before the implementation is needed. In such cases, CONTRACTOR may omit steps 3, 4, 5, and 6 from Section 8.1. As stated in 8.1.2, CONTRACTOR will denote the ticket as a ZERO-DAY SECURITY PATCHING event in the description of the ticket.

8.3 Major System Updates

Due to the nature of Major system updates, such as (but, not limited to) moving from Windows 2016 to Windows 2019 those type updates are considered out of scope for this SLA. Major updates time and costs will be provided to NLS. No SLA will apply to software or servers that have reached End-of-Life status by their vendor. On NLS approval CONTRACTOR will use the Section 8.1 process, skipping step 7 from Section 8.1, and adding the further steps defined to update the major version of the software.

1. CONTRACTOR will create a new DEVELOPMENT environment and migrate any data or settings necessary for the operation of the APPLICATION.

2. CONTRACTOR will notify NLS and provide information for NLS to test

3. On approval from NLS CONTRACTOR will set a “cut over time” at which time the system will be switched to the new system.

4. After 10 business days, CONTRACTOR will permanently remove the old environment. NLS can request that environment be kept up and running past that time; NLS will be appropriate costs daily for maintaining that environment.

9.0 Outages and Maintenance Windows

9.1 Maintenance Window

A planned outage is the result of a planned activity to perform maintenance task, diagnose detected faults, or software patching/software updates. Maintenance windows can originate from one or more of the following entities: Microsoft Azure, CONTRACTOR, and / or NLS.

9.2 Notification

CONTRACTOR will notify the Coordinators listed in Section 5.2 of all planned maintenance windows at least 72 business hours before the start time.

9.3 Outage Timeframe

Planned outages must be scheduled outside of NORMAL WORKING HOURS.

9.4 Unplanned Outages

An unplanned outage is termed as any unplanned downtime or interruption to daily operations. Unplanned downtime may be caused by failures such a: cloud service failure, software bug, human error, equipment error, or overload due to exceeding capacity. Unplanned outages may occur for the Microsoft Azure cloud instance the APPLICATION is hosted on and the APPLICATION itself.

9.4.1 Remediation

CONTRACTOR will implement monitoring procedures to prevent unplanned outages, but if an unplanned outage occurs CONTRACTOR will (1) notify NLS immediately via email and phone, (2) open a TICKET, and (3) provide updates via the ticket, and provide resolution according to the remedial support procedures.

9.4.2 Root Cause Analysis

CONTRACTOR will provide the NLS coordinators, as defined in Section 5.2, a Root Cause Analysis of the unplanned outage within 48 business hours after the unplanned outage is fully resolved.

10.0 Escalation Policy and Procedure

In the event Support Response Times, as defined in Section 2.5, and/or performance metrics under CONTRACTOR control are not met, the NLS shall follow the escalation process detailed below until a resolution is reached. The NLS shall escalate the issue until a resolution is reached or until it reaches the level of Delivery Manager.

10.1 Escalation Process

In the event Support Response Times and/or performance metrics are not met, NLS shall follow the following escalation process:

1. Email notification to CONTRACTOR project manager, noting what time and / or metric is not being met. The project manager must respond within 24 business hours setting up a meeting with the necessary parties within 48 business hours. During that meeting a remediation plan must be put in place; or if more discovery is required, a follow up meeting must happen within 24 business hours of the first meeting.

2. CONTRACTOR will have 15 business days to implement the remediation plan. NLS must approve any changes, if any, that require additional costs before work on the implementation plan is started.

3. If after 15 business days, the issue has not been resolved to NLS satisfaction, NLS will escalate to the CONTRACT Delivery Manager via email, the Delivery Manager will work to schedule the appropriate meetings and establish a remediation plan

4. CONTRACTOR must schedule a status meeting every day (or with NLS approval, provide an end of day email update), until the issue has been resolved to NLS satisfaction.

11.0 Stakeholder Review and Approval

I have reviewed the content of this Service Level Agreement, and I agree to abide within its provisions. This agreement is subject to revision and review prior to the renewal of each option period.

[Insert Signature Block] Page | 20