14-Appendix M ESInet-NGCS Service Level Agreement.pdf

Performance Standards and Service Level Agreements

Service Level Agreement An agreement between the State and the Respondent that specifies, in measurable terms, the services that the Respondent will furnish.

System Performance Standards and Reporting Respondents must identify the SLAs and metrics for the system components that will be utilized to formulate the system performance measurements for each performance standard.

System Availability The service must be available 99.999 percent of the time and is measured on a per-PSAP basis.

Service Level and Service Management Performance Standard Services referenced here are limited to those provided under the agreement. All times are averages over a rolling 12-month measurement period. However, there are provisions for declaring an SLA violation in cases where repeated instances occur over a short period of time.

All time intervals are calculated to the nearest minute. Performance requirements are applicable to managed and non-managed services.

Regular Business Hours Hours between 8:00 a.m. and 5:00 p.m. US Eastern time.

NOC/SOC Availability Resources will be available (via toll-free telephone number) 24 x 7 x 365 to process requests for support.

Technician Availability Onsite technicians will be available remotely and/or onsite as required, on a 24 x 7 x 365 basis.

Incident Severity Levels

Incident severity level definitions are consistent with those in Acceptance Testing. For the post-cutover environment, some examples are provided below of issues that would be classified according to the specified severity.

Severity Level 1 – Critical An incident shall be categorized as “Severity Level 1” if it renders a critical system, service, software, equipment, or network component unavailable or substantially unavailable, or seriously impacts normal business operations; in each case, prohibiting the ability to process a request for assistance for one or more PSAPs, or a single individual performing a critical function. Examples of conditions may include:

o Isolation of any single site or sites from the rest of the network, resulting in the inability for affected sites to communicate with the rest of the network.

o A complete loss of functional elements within the NGCS (e.g. ECRF, ESRP, etc.)

o A software defect without a workaround that impacts any site’s or sites’ ability to maintain business operations.

o A reduction in call-processing capacity at a single site of 50 percent or more. Call-processing capacity shall be defined as the number of staffed CHaaS workstations at a PSAP.

o Cybersecurity attack detected to be in progress or highly likely/imminent.

o Faults affecting critical 9-1-1 service resulting in:

The inability for a caller to connect through voice dialing or texting with a PSAP.

The inability for a PSAP to receive audio with an emergency call, (when audio is expected).

The inability for a PSAP to receive location information with a 9-1-1 call.

The inability for a PSAP to receive callback information with a 9-1-1 call.

Severity Level 2 – Major An incident shall be categorized as “Severity Level 2” if the incident does not render a critical system, service, software, equipment, or network component unavailable or substantially unavailable, but a function or functions are not available, substantially available, or functioning as they should; in each case, impacting the ability to process a request for assistance for one or more PSAPs, or a single individual performing a critical business function. Examples of conditions may include:

o Loss of network or redundancy, including both ESInet/NGCS connectivity and PSAP connectivity.

o System or component problem that could result in loss of a site without timely repair.

o A reduction in call-processing capacity at a single ESInet or NGCS site of 20 percent or more.

o Inability to accurately display critical information like callback number or location information.

o Call misroutes (delivery of calls to the wrong PSAP, voice or text) o Location failures resulting in default routing of calls

Severity Level 3 – Minor An incident shall be categorized as “Severity Level 3” if the incident causes a group or individual to experience trouble accessing or using a system, service, software, equipment, or network component, or a key feature thereof, and a reasonable workaround is not available, but does not prohibit the execution of productive work. Examples of these conditions may include:

o Inability to access supplemental data.

o Failure of a common function like monitoring or a CHE display problem, requiring manual intervention or other additional steps to achieve the desired result.

Severity Level 4 – Non-Service-Impacting An event shall be categorized as “Severity Level 4” if the incident impacts a PSAP, a group, or individual with minor usability problems but does not prevent or hinder functionality. Examples of these conditions may include:

o Errors in data display/presentation with an application.

o Errors with peripheral systems such as MIS, workstation equipment, etc.

o Errors with monitoring systems, dashboards, etc.

o Items committed to by the Respondent that are planned for future release (software development)

Response Time A qualified resource acknowledges requests for assistance by the State or PSAP via phone or email and commences investigation of trouble within the timeframes identified in the table below.

Table 1: Response Time

Trouble Category Response Time

Severity Level 1 15 minutes

Severity Level 2 30 minutes

Severity Level 3 8 hours

Severity Level 4 12 hours

Notification Time The time from when an issue is identified and the State or PSAP is informed of the issue. Contractor shall have the ability to provide customized notifications based on desire of the State or PSAP.

Table 2: Notification Time

Trouble Category Notification Time

Severity Level 1 15 minutes

Severity Level 2 30 minutes

Severity Level 3 8 hours

Severity Level 4 Next Business Day

Repair Time Issues documented via the ticketing process will be resolved in the timeframes identified in the table below. Resolution may be either the final repair that returns the system to its normal functioning condition, or a PSAP-accepted workaround accompanied by a plan to achieve the final repair.

Table 3: Repair Time

Trouble Category Interval

Severity Level 1 2 hours

Severity Level 2 4 hours

Severity Level 3 48 hours

Severity Level 4 96 hours

Escalation In cases where an interval identified in the table below has elapsed from the time the ticket was created, a request for assistance to the next higher level of technical support must be executed.

Table 4: Escalation Intervals

Trouble Category Interval

Severity Level 1 2 hours

Severity Level 2 4 hours

Severity Level 3 48 hours

Severity Level 4 96 hours

Once an escalation has occurred, Contractor will provide the State and affected parties with a status update within an agreed-upon interval until resolved.

IP Network Measurement and Reporting Requirements

Network Performance

The selected Respondent must measure and report on the network performance against the service levels monthly. For any circuit downtime, outages, or interruptions, the selected Respondent must provide a written report describing the degradation of service or outage, including the root cause and the plan to prevent similar occurrences in the future. Trend data must be supplied with this report that shows current and previous monthly performances.

Service Interruption and Outage Reporting

As defined in 47 CFR § 4.5, the Respondent is responsible for complying with all federal reporting requirements for any outages or interruptions to 9-1-1 voice or SMS text delivery, and/or processing systems and services provided under a contract. The Respondent is required to electronically report significant network outages and information about outages that exceed specific thresholds—in terms of duration and magnitude—to the FCC’s Network Outage Reporting System (NORS).

The electronic report shall also include information regarding communications disruptions affecting any 9-1-1 facilities.

Root Cause Analysis Reporting

In the event of an unplanned service interruption or outage that results in a Severity 1 or 2 incident or in an SLA being missed, the Contractor must provide the State with a Root Cause Analysis (RCA) report. An initial written explanation of the problem and the planned resolution will be provided by the vendor within 5 business days, with a final report provided within 30 calendar days, measured from the beginning of the service affecting issue. This report will include the timeline and duration of the interruption/outage, the cause of the interruption/outage, the corrective actions taken to resolve the interruption/outage, and any actions/processes undertaken by the Respondent and its subcontractors to prevent similar issues from occurring in the future, or to mitigate their impact, if they do.

Bandwidth Management

The Respondent shall provide monthly bandwidth performance and network utilization reports.

State must be able to monitor overall bandwidth usage and specific usage between sites. State must be able to view real-time or near-real-time bandwidth performance and utilization reports. Users of the monitoring tools should be able to filter on various traffic characteristics (e.g., protocol, QoS classification, media type, etc.). A web-based portal or browser-enabled viewer is required.

The Respondent’s call-processing capacity shall be capable of 125% busy hour call traffic for the PSAP.

The Respondent’s service shall be able to increase call-processing capability by up to 50% over contract duration with no critical hardware upgrades required.

Voice Quality and Quality of Service

Voice quality must be maintained at traditional Public Switched Telephone Network (PSTN) levels. Voice data (RTP1) must have priority over any other IP traffic and this priority must be respected and enforced by all network infrastructure elements, end-to-end. The service shall use an uncompressed, high quality voice codec. The ESInet/NGCS must not degrade the Mean Opinion Score (MOS) of calls traversing the ESInet by more than two-tenths (0.2) from the point of network ingress to the point of delivery to the CHaaS system. The Respondent must maintain a MOS score of 4.0 or better across the ESInet and through the NGCS to the CHaaS demarcation point.

Network Management and Monitoring

The Respondent must provide a NOC/SOC to respond to network issues and meet the service levels stated within this document, including requirements for a secondary or backup NOC, and monitor and analyze activity on networks, servers, endpoints, databases, applications, websites, and other systems to identify anomalous activity that could be indicative of a security incident or compromise.

Proactive Monitoring

The Respondent must provide active monitoring of its Service as a whole, including; network, replication, hardware, software, call processing, etc. The Respondent must proactively generate incident tickets and alert the State and affected PSAPs in accordance with the incident severity level tables above regarding Response or Notification Times.

SLA Reporting

A secure online SLA reporting dashboard is required. The SLA reporting tool shall include both real-time and/or near real-time performance data captures in no greater than five-minute averages. The SLA reporting tool shall summarize network

1 Real-time Transport Protocol performance metrics by hour, day, week, month, quarter, and year. The mechanism shall deliver automated SLA results to State on a monthly and quarterly schedule. (The quarterly results are simply an aggregation of the monthly reports).

Quality of service (QoS) reporting shall present traffic by type. Reports shall include, at a minimum, statistics for latency, jitter, packet loss, and bandwidth utilization, and shall be available on demand with near real-time data. Other relevant data also may be reported.

Contractor shall conduct end-of-month and end-of-quarter service reviews, accounting for any degradation of service to include service failures, as well as incidents and problems, and their resolution.

SLA remedies shall be credited to their full amount, but the maximum credit per month shall not exceed 25 percent of the total monthly service fee.

Incidents shall be tracked via support tickets and the ticket contents shall be made available to the State and the affected PSAPs. Contractor shall have automated systems to track all SLA deliverables and provide the State with monthly reports detailing Contractor performance.

The monthly SLA compliance report shall include the following detail and a one-page summary for every Severity 1 and 2 problem that occurred during the month:

1) Report period;

2) Contractor’s trouble ticket number;

3) Name(s) of affected PSAP(s);

4) FCC ID(s) of affected PSAP(s);

5) Service type;

6) Brief trouble symptom;

7) Brief restoration description;

8) Trouble symptom category;

9) Ticket open date and time;

10) Severity level;

11) Problem resolution date and time;

12) Total duration;

13) Yes/no if qualified for SLA;

14) Yes/no if FCC reporting required;

i. If yes, link to FCC report and FCC response, if any

15) The applicable SLA; and

16) Rights and remedies applied to each ticket when applicable.

Table 5: SLA Violations, Damages, and Requirements

SLA Definition Measurement Method Objective Rights and Remedies

Final master project plan (MPP) shall be delivered to the State within sixty (60) calendar days of contract execution.

Calendar Days Delivery of MPP within 60 days.

Failure to meet the objective shall result in a

$5,000.00 credit/or adjustment for each calendar day that the report is not delivered after the objective date.

Contractor shall achieve all milestone dates identified in the

MPP.

Calendar Days

Completion of MPP milestones on or before the date agreed by the State and Contractor.

Any failure to meet the objective shall result in a

$5,000.00 credit/or adjustment for each calendar day that the milestone is not delivered after the objective date (for milestones with a majority of the underlying task ownership being that of the

Contractor and/or its subcontractors).

SLA Remedy Delivery:

Timely credit/or adjustment of remedies due to the State for missed SLA objectives.

Calendar Days

Contractor’s credit/or adjustment shall be issued no more than sixty (60) calendar days after written notice of the violation from the State.

Each occurrence of an SLA remedy (credit/or adjustment) that is not issued within sixty (60) calendar days shall result in a $5,000.00 credit/or adjustment for each calendar day that the credit/or adjustment is not issued.

Contractor shall provide a ticketing interface and monitoring dashboard 24 hours a day, 7 days a week, 365 days a year (24 x 7 x

365)

The monthly availability percentage equals the Scheduled Uptime less

Unavailable Time divided by Scheduled Uptime per month multiplied by 100.

Scheduled Uptime is based on 24 hours x number of days in the month x 60 minutes.

The monthly Availability percentage shall be based on the cumulative total of all outage durations for each calendar month.

99.9% (8.77 hours per year

43.83 minutes per month)

Failure to meet the SLA objective for one month shall result in a $2,500 credit/or adjustment of the total Monthly Recurring

Charge (MRC) of all services under contract for that month.

When a manual (i.e., via phone call to NOC) emergent request is made to reroute calls to an alternate PSAP, reroute will be effective in five minutes or less, from the time of the PSAP call being answered by the NOC for pre-provisioned routes.

If the request is for standard maintenance, the reroute will be effective in seven minutes or less for pre-provisioned routes; if a new route is required, that will be completed within twenty minutes of request.

Time from NOC call received by human to time the routing change becomes effective.

5 minutes (emergency) 7 minutes (maintenance) 20 minutes (new route)

Failure to meet the SLA objective shall result in a

$5,000.00 credit/or adjustment of the total

MRC.

Contractor shall report all outages and service interruptions that potentially impact the delivery of 911 traffic (voice, ANI, ALI) within

15 minutes of the occurrence.

Any outage or service interruption that potentially impacts the delivery of 911 traffic, regardless of traffic type

Notification within 15 minutes or less.

Any failure to meet the

$5,000.00 credit/or adjustment

Every next consecutive minute that the Contractor fails to meet the SLA objective shall result in an additional $1,000.00 credit/or adjustment per minute, up to the total MRC for all services.

Redundant ESInet connectivity shall be available to each end-point PSAP connection (except as described in the following requirement).

The monthly availability percentage equals the Scheduled Uptime less

Unavailable Time divided by Scheduled Uptime per month multiplied by 100.

Scheduled Uptime is based on 24 hours x number of

The ESInet connectivity availability requires two diverse IP network connections to each PSAP.

For those PSAPs where diverse IP network connections are not available and when approved by the State, the individual ESInet connectivity SLA applies.

on the cumulative total of all outage durations for each calendar month.

99.999% (5.26 minutes per year

26.3 seconds per month of unplanned downtime) shall result in a 25% credit/or adjustment of the

MRC for the population served by that end-point

PSAP that month.

objective for the next consecutive month shall result in a 50% credit/or adjustment of the MRC for the population served by that end-point PSAP that month.

objective for each additional consecutive month shall result in a

100% credit/or adjustment of the MRC for the population served by that end-point PSAP that month, plus an additional $25,000.

Individual (non-redundant) ESInet connectivity shall be available to each end-point PSAP and will only apply when diverse NG911 trunks are not available and when approved by the State.

The monthly availability percentage equals the Scheduled Uptime less

Unavailable Time divided by Scheduled Uptime per month multiplied by 100.

Scheduled Uptime is based on 24 hours x number of on the cumulative total of all outage durations for each calendar month.

99.9% (8.77 hours per year

43.83 minutes per month) shall result in a 25% credit/or adjustment of the

MRC for the population served by that end-point

PSAP that month.

The second consecutive month of failure to meet the SLA objective shall result in a 50% credit/or adjustment of the MRC for the population served by that end-point PSAP that month.

The third consecutive month of failure to meet the SLA objective shall result in a 100% credit/or adjustment of the MRC for the population served by that end-point PSAP that month.

Contractor shall transition or migrate all ingress traffic from legacy selective routers (LSRs) to its NGCS in accordance with the timeline established by the FCC

Report and Order in the Matter of Facilitating Implementation of

Next Generation 9-1-1 Services (NG911) 2 . All traffic shall be migrated as soon as the respective OSP supports SIP-based interconnection.

Decommissioning of legacy selective routers and elimination of associated costs.

Successful decommissioning of legacy selective routers.

Failure to meet the SLA objective shall result in a

$5,000 credit or adjustment per OSP for each month that the OSP is not connected to the NGCS via

SIP).

Contractor shall require SIP-capable OSPs to connect directly to the ESInet/NGCS via SIP in accordance with the timelines established by the FCC Report and Order in the Matter of Facilitating Implementation of

Next Generation 9-1-1 Services (NG911). All traffic shall be transitioned as soon as the respective OSP supports SIP-based interconnection.

Elimination of TDM aggregation sites.

100% ingress SIP call delivery

$5,000.00 credit/or adjustment per OSP for each month that the OSP is not delivering calls via

SIP.

Failure to provide updates to solution to conform to NENA i3 standards within 18 months of ratification.

Identification by the State or its designee of i3 functionality not supported.

100% conformance to NENA i3 standards within 18 months of ratification.

20% credit/or adjustment of total monthly invoice until conformance issue is resolved to State’s satisfaction.

2 FCC 24-178 Facilitating Implementation of Next Generation 911 Services (NG911) - https://www.fcc.gov/document/fcc-takes-action-expedite-transition-next-generation-911-0 https://www.fcc.gov/document/fcc-takes-action-expedite-transition-next-generation-911-0 https://www.fcc.gov/document/fcc-takes-action-expedite-transition-next-generation-911-0

NGCS must forward voice calls with little or no degradation of voice quality of the call from the ingress demarcation point to the egress demarcation point, as measured and monitored by an automated Mean Opinion Score

(MOS) measurement tool between various ingress and egress points at times when the ESInet is meeting its performance parameters.

Ingress demarcation point shall be defined as the Protocol Interworking

Function (PIF) of the LNG (TDM call delivery) or the ingress SIP interface on the SBC (SIP call delivery).

Egress demarcation point shall be the CHaaS host facing port of the ESInet/NGCS edge router at the CHaaS host sites.

Maintain a MOS score of 4.0 or better across the ESInet and through the NGCS to the CHaaS demarcation point.

Failure to meet the SLR objective for one month shall result in a 25% credit or adjustment of the amount at risk for the ESInet for all affected

PSAPs that month.

A second consecutive month failure to meet the SLR objective shall result in a 50% credit or adjustment of the amount at risk for the ESInet for all affected PSAPs that month.

Each additional consecutive month failure to meet the SLR objective shall result in a 100% credit or adjustment of the amount at risk for the ESInet for all affected

PSAPs that month.

The NGCS will deliver 911 traffic including location, callback information, and Additional Data to the appropriate NG911 CHE.

Single outage with a duration of six (6) minutes or more.

Preventing outages of six (6) minutes or more.

100% credit/or adjustment of the MRC of NGCS for the affected PSAP.

The NGCS will deliver 911 traffic including location, callback information, and Additional Data to the appropriate NG911 CHE.

Single outage of greater than two (2) minutes and less than six (6) minutes.

Preventing outages greater than two (2) minutes, but less than six (6) minutes.

50% credit/or adjustment of the MRC of NGCS for the affected PSAP.

The Contractor shall provide the SLA reporting required by this contract for each month of activity during the term of the contract by the 10th business day of the following month.

Business Days

The Contractor shall deliver accurate and complete reporting by the 10th calendar date of the month following the end of the applicable reporting month.

Failure to meet the objective shall result in a

$5,000.00 credit or adjustment for each business day that the reporting is not delivered after the objective date.

Contractor shall provide a cybersecurity vulnerability notification upon recognition of a cybersecurity threat or breach within three (3) business days of issue identification.

Business Days

Within three (3) business days of the recognition of a vulnerability or identified cybersecurity threat or breach, the Contractor shall notify the State and the impacted PSAP(s) of the event, and measures taken to mitigate impact and avoid future risk.

Failure to meet the objective shall result in a

$5,000.00 credit or adjustment for each business day that the report is not delivered after the objective date.

Contractor shall provide an initial Root Cause Analysis (RCA) report for Severity Level 1 and 2 issues within five (5) business days.

Business Days

The Contractor shall deliver initial root cause analysis to the State and the affected

PSAPs within five (5) business days of service affecting issue.

Each occurrence of a failure to meet the objective shall result in a $15,000.00 creditor adjustment for each business day that the

Contractor shall provide a final RCA report within thirty (30) calendar days from the beginning of the service affecting issue.

Calendar Days

The Contractor shall deliver final RCA to the State and the affected PSAPs within thirty (30) calendar days from the beginning of the service affecting issue.

Each occurrence of a failure to meet the objective shall result in a $15,000.00 credit/or adjustment for each business day that the

Contractor shall provide accurate and timely billing that complies with the agreed upon rates and schedule.

Identification by the State or its designee of billing inaccuracies or deviations from the agreed schedule.

100% conformance with agreed upon billing rates and schedules.

Failure to meet the SLA objective shall result in a credit of 25% of the amount of the bill in question for each week within the month that the bill is late or found to be in error, until the billing issue is resolved to the State’s satisfaction.