Appendix E Service Level Agreements.docx

CONTROLLED//FDIC BUSINESS

Appendix E – Service Level Agreements Key Performance Indicators

Service Level Title K1: Deliverables Submitted on Time (Status Reports, Invoices, Accruals, SLA Reports, Staffing, …)

· For purposes of this performance metric, Deliverables is defined to include all reports, documentations, product output with established commitments for the AS-NG project. Deliverables for Contractor support whether related to Production or other environments.

· Reports and documents include but are not limited to:

· Status Report – weekly

· Invoices – monthly (10 days after the end of each month)

· Accruals - monthly (5 days before the end of each month)

· Staffing Report - monthly (10 days after the end of each month)

· SLA report – monthly (15 days after the end of each month)

Reports, documentations, as designated by

FDIC

Measured and reported monthly for previous month

(Number of Reports and Documents with established commitments completed within the commitment duration) * 100 / Total Required Number of Reports and Documents submitted for the month

Performance Failure results in Service Level Default, triggering a Service Level Credit as described in the “Performance Management” section of the Agreement

Service Level Title K2: Service Requests Completed on Time

Production and non- Production Service Requests completed on schedule

· For purposes of this performance metric, Service Request is defined to include all Requests for Contractor support whether related to Production or other environments

· Examples include Requests for Services such as setting up a test bed, running reports ad hoc, refreshing databases, resetting passwords, DBA services in support of a major migration, tools support, etc.

o Incidents and Application change requests are NOT included

· When a Service Request is recorded in the Incident Management System with an initial commitment duration, the commitment duration is expected to come from one of three sources:

· Durations listed in a service catalog

· Durations specified by the requestor in the Service Request

· Durations negotiated between the requestor and the subject matter expert

· The Contractor may update - based on negotiation with the customer - the initial commitment duration only once for each Request Incident Management System, as designated by

FDIC

Measured and reported monthly for previous month

(Number of Requests with established commitments completed within the commitment duration + Number of Requests without established commitments completed by the initial commitment entered into the tracking system of record) * 100 / Number of Requests completed

Performance Failure results in Service Level Default, triggering a Service Level Credit as described in the “Performance Management” section of the Agreement

Service Level Title K3: Deliverables Submitted on Time (Status Reports, Invoices, Accruals, SLA Reports, Projects, …)

Defects

· Measures the rate of security Defects injected by the Contractor for a developed Application, upgrade, revision, Enhancement or product released into the Production environment

· For purposes of this performance metric, the security Defect count includes:

· Any security Defect existing when Software is delivered for User Acceptance Testing

· Any known security Defect left over from a previous release

· Any security Defect recorded in GitHub or the applicable repository (as designated by FDIC) after the start of formal UAT

· Any security Defect identified or confirmed and recorded in OpenFISMA during the Warranty period

· Any instance of SQL injection identified in any code component developed or modified by the Contractor at any time

· Any instances of cross-site scripting identified in any code component developed or modified by the Contractor at any time

· Any Defect that enables unauthorized access to data due to incorrect configuration of platform or tool (such as database roles, privileges and grants)

· The only security Defects not included in the count are as follows:

· Those that result from circumstances that could not have been known prior to releasing the Software into Production (such as newly discovered and exploited weaknesses in an underlying technology)

· Duplicate Defects recorded in more than one repository (in which case only the Defect in the OpenFISMA repository will be counted)

· Defects for which the FDIC has executed a formal Acceptance of Risk (AOR)

· Defects which have been confirmed by the FDIC PM as fixed but which have not yet been reevaluated by Security and formally closed

· Findings directly attributable to the hardware or physical host environment for the Application or configurations thereof which are beyond the control of the Contractor

· For purposes of this performance metric, Project includes Projects with over 320 hours

· Calculations are to be performed once for each Project immediately after the end of the Warranty period

· Calculations are to be performed for each Project but reported in aggregate for determining performance to the AQL OpenFISMA, GitHub, or as designated by FDIC Measured and reported monthly for previous month

Sum of security Defects on all Projects / Number of Projects completed

YELLOW

Performance Failure results in Service Level Default, triggering a Service Level Credit as described in the “Performance Management” section of the Agreement

RED

Service Level Title K5: User Acceptance Test Defects – Severity 1 & 2

(Severity 1 and 2)

· Measures the rate of UAT Defects in a developed Application, upgrade, revision, Enhancement over 320 hours (or at a threshold identified in the TBOA) delivered for UAT

· For purposes of this performance metric, the Project hours worked count includes the most recent approved PWP baseline hours but does not include any hours related to resolving UAT Defects

· The number of UAT Defects includes all Severity 1 and 2 Defects uncovered during UATs for all Projects conducted by Contractor under a TBOA

· Calculations are to be performed once for each Project immediately upon completion of UAT

· Calculations are to be performed for each Project but reported in aggregate for determining performance to the AQL

GitHub or as designated by FDIC Measured and reported monthly

(number of Severity 1 UAT Defects + number of Severity 2 UAT Defects) * 100 project hours / total new development and Enhancement project hours worked

YELLOW

>.80

RED

Performance Failure results in Service Level Default, triggering a Service Level Credit as described in the “Performance Management” section of the Agreement

Operational Performance Indicators

OPIs are Service Level metrics without automatic impacts (financial and/or non-financial) to the contractor for failure to meet the AQL. The table below lists the OPIs that may be applicable.

Operational Performance Indicators

Service Level Title O3: Average Task Price Variance

Contractor price estimates

Measures the accuracy of Contractor’s price estimates for Task Orders (TOs), Task Assignments (TAs) or underlying Projects over 320 hours (or as specified in the TO/TA) completed in the Production environment during the measurement period

· For purposes of this performance metric the following definitions apply:

· Actual price is the actual amount invoiced for the effort by the Contractor o Approved price is the Not-to-Exceed (NTE) price from the original TO / TA plus, where applicable, additional NTE price changes that result from FDIC directed changes or delays, or from new scope added to the Project

· Approved price does not include NTE price changes attributed to the Contractor (attribution at the discretion of the OM)

· Calculations are to be performed for each Project, Application, product or tool completed in a given month but reported in aggregate for determining performance to the AQL

· Calculations are to be performed once for each Project immediately after the Warranty period

Approved Invoices, Contract documents Measured and reported monthly

((Actual price – Approved price) /Approved price) * 100

Service Level Title O4: Average Task Schedule Variance

Contractor’s schedule estimates

· Measures the accuracy of Contractor’s schedule estimates for Task Orders (TOs), Work Orders (WOs) or underlying Projects over 320 hours (or as specified in the Task Order) completed in the Production environment during the measurement period

· Approved Business Days are days included in the approved baseline Project plan plus those changes that result from FDIC directed changes or delays, or from new scope added to the Project

· Elapsed Business Days begins on period of performance start date or authorization to start, whichever is earlier, and ends at:

· the end of the Transition phase (Project Completion Milestone Review) for Development projects, less any approved adjustments

· the date on which the last activity in the Project plan is completed for Shared Services projects

· Calculations are to be performed for each Project, Application, product or tool completed in a given month but reported in aggregate for determining performance to the AQL

· Calculations are to be performed once for each Project immediately after the end of Warranty period Project Server, GitHub, or as designated by FDIC Measured and reported monthly for previous month

((Elapsed Business Days – Approved Business Days) / Approved Business Days) * 100

Service Level Title O5: Assessing Work Performed

Contractor’s ability to complete iteration assessments

· Measures, for applicable projects, the Contractor’s ability to complete assessment documentation for each iteration

· Applicable Projects include those that meet all the following criteria:

· Projects with duration equal to or greater than six calendar weeks, and

· Projects that adhere to iterative software development approaches, such as RUP or Agile methodologies, regardless of whether a system is finally deployed to the Production environment, and

· Projects that end in the measurement period, with end occurring either through Software being deployed in Production (use deployment date as end date), the Project completing per the date in an FDIC-approved Project plan (use date completed as end date), or the Project being cancelled or put on hold (use cancellation or hold date as end date)

· All applicable Projects are scored individually using the metric calculation below with results expressed as a percent

· Acceptable evidence that iteration assessments have occurred includes assessment documents or email chains (stored in FDIC approved repository with “Created Time” or “Sent Time” attributes that are prior to the end date of the iteration) that indicate FDIC approval or acceptance of the assessment and / or the assessment documentation

· In case of disagreement, the OM will make the final determination regarding acceptability of assessment content

· Calculations are to be performed once for each Project immediately after the end of the Warranty period after release to Production GitHub, or as designated by FDIC Measured and reported monthly

Number of iteration assessments conducted with evidence / number of iterations

Service Level Title O6: Quality Plan Compliance

Contractor compliance with approved Quality Plans

· Measures, for Projects underway and Application portfolios supported by Contractor, the rate at which Contractor complies with reporting and performance requirements outlined in FDIC approved Quality Plans

· Applicable Projects / portfolios include those that meet all the following criteria:

· Projects / portfolios that are underway during the measurement period

· Projects / portfolios that have approved Quality Plans

· For purposes of this performance metric the following definitions apply:

· “In compliance” means meeting or exceeding the reporting and performance requirements outlined in the approved Quality Plans

· “Reporting requirements” means the format, content and timing of reports as outlined in the approved Quality Plans

· “Performance requirements” means the quality-related performance targets set in the approved Quality Plans

· Acceptable evidence that Quality Plans have been approved and reports on performance to Quality Plan details have been submitted includes documents and / or email chains (stored in in FDIC approved repository with “Created Time” or “Sent Time” attributes that are prior to reporting deadlines outlined in the Quality Plans) that indicate FDIC approval of the Quality Plan and / or receipt of the Quality Plan performance reports

· In case of disagreement, the OM will make the final determination regarding compliance

· All applicable Projects / portfolios are scored individually using the metric calculation below but reported in aggregate for determining performance to the AQL with results expressed as a percent GitHub or as designated by FDIC

Measured and reported monthly

Number of Projects underway or Application portfolios supported where Contractor performance is in compliance with reporting and performance requirements in approved Quality Plans / number of Projects underway or Application portfolios supported that are required to have Quality Plans

Service Level Title O7: Test Plan Compliance

Contractor compliance with approved Test Plans

· Measures for Projects underway and Application portfolios supported by Contractor, the rate at which Contractor complies with reporting and performance requirements outlined in FDICapproved Test Plans

· Applicable projects and Application portfolios include those that meet all the following criteria:

· Projects / portfolios that are underway during the measurement period

· Projects / portfolios that that are required to have approved Test Plans

· For purposes of this performance metric the following definitions apply:

· “In compliance” means meeting or exceeding the reporting and performance requirements outlined in the approved Test Plans

· “Reporting requirements” means the format, content and timing of reports as outlined in the approved Test Plans

· “Performance requirements” means the quality-related performance targets set in the approved Test Plans

· Acceptable evidence that Test Plans have been approved and reports on performance to Test Plan details have been submitted includes documents and / or email chains (stored in FDIC approved repository with “Created Time” or “Sent Time” attributes that are prior to reporting deadlines outlined in the Test Plans) that indicate FDIC approval of the Test Plan and / or receipt of the Test Plan performance reports

· In case of disagreement, the OM will make the final determination regarding compliance

· All applicable Projects / portfolios are scored individually using the metric calculation below but reported in aggregate for determining performance to the AQL with results expressed as a percent GitHub or as designated by FDIC

Measured and reported monthly

Number of Projects underway or Application portfolios supported where Contractor performance is in compliance with reporting and performance requirements in approved Test Plans / number of Projects underway or Application portfolios supported that are required to have Test Plans

Service Level Title O8: Measuring Project Velocity

Contractor ability to report their project team’s velocity

· Measures how many of the Contractor projects are able to measure and report their team velocity. The velocity is calculated by adding up the estimates (or size) of units of work completed in an iteration. Estimates, or sizes, are determined at the beginning of each iteration and the velocity is calculated at the end of an iteration

· Units of work, or items, may include features, stories, requirements, tasks, or backlog items

· To calculate velocity, a team first must determine how many units of work (sizing) each item is worth. During development, the team must keep track of completed items and, at the end of the iteration, sum the size of work completed during the iteration. The team then captures the calculated velocity in a chart or on a graph

· All applicable Projects are scored individually using the metric calculation below with results expressed as a percent

· Acceptable evidence that velocity has been measured and reported is the velocity chart or graph stored in GitHub or approved repository by the OM

· In case of disagreement, the OM will make the final determination regarding compliance

· Applicable projects include those that meet all the following criteria:

· Projects with duration equal to or greater than six calendar weeks, and

· Projects that adhere to iterative Software development approaches, such as RUP or Agile methodologies, regardless of whether a system is finally deployed to the Production environment, and

· Projects that end in the measurement period, with end occurring either through Software being deployed in Production (use deployment date as end date), the Project completing per the date in an FDIC-approved Project plan (use date completed as end date), or the Project being cancelled or put on hold (use cancellation or hold date as end date). All applicable Project are scored individually (by phase) using the metric calculation below

· Example calculation for a Project ending during the measurement month with 10 total iterations over the life of the Project, of which 9 of those iterations had a velocity graph or chart stored in approved repository:

· 9/10 = .9 or 90% (therefore, a performance failure has occurred)

· Calculations are to be performed once for each Project GitHub, or as designated by FDIC Measured and reported monthly

Number of iterations conducted with evidence that a velocity graph or chart was produced / number of iterations

Service Level Title O9: Root Cause Analysis (RCA)

RCAs completed within an acceptable timeframe

· Calculations are to be performed for each Incident in a given month requiring an RCA. Calculations are reported in aggregate for determining performance to the AQL, unless otherwise directed by FDIC

· For purposes of this performance metric, RCA is determined complete when the following has occurred:

· RCA has been submitted to the Technical Monitor for the related task

· Technical Monitor approves the RCA

· Timeframes assume the clock starts at the time the earliest ticket associated with the Incident was assigned to Contractor in the Incident Management System and stops at “Time Posted “in SharePoint or GitHub for the completed RCA SharePoint, GitHub, ServiceNow, or as designated by FDIC Measured and reported monthly for previous month

Number of RCAs completed within AQL timeframe / number of Incidents requiring RCAs

image1.png image2.png