ATT M - Service Delivery Standards and Metrics.pdf

ATTACHMENT M

NASA CONSOLIDATED APPLICATIONS AND

PLATFORM SERVICES (NCAPS)

SERVICE DELIVERY STANDARDS AND METRICS

RFP 80TECH23R0002

CONTRACT #TBD

DATE: JANUARY 2023

NCAPS Contract – 80TECH23R0002 Attachment M Service Delivery Standards and Metrics

Table of Contents

1.0 SERVICE DELIVERY STANDARDS AND METRICS

2.0 GUIDELINES

3.0 EXAMPLES OVERVIEW

4.0 METRIC EXAMPLES

4.1 SERVICE DELIVERY STANDARDS

4.1.1 SHAREPOINT AS A SERVICE (SPaaS)

4.1.2 COMMERCIAL OFF-THE-SHELF (COTS) UPGRADES

4.1.3 IDU DELIVERY

4.1.4 IR DELIVERY

4.2 SYSTEM PERFORMANCE METRICS

4.2.1 APPLICATION & SYSTEM RETURN TO SERVICE

4.2.2 APPLICATION & SYSTEM AVAILABILITY

4.3 REPORTING ONLY METRICS

4.3.1 INCREMENTAL DEVELOPMENT UNIT (IDU) PLANNING

ACCURACY

4.3.2 OUTAGE NOTIFICATION

4.3.3 SERVICE TICKET RESPONSE

4.3.4 COMMITMENT RELIABILITY

4.3.5 PLANNED VERSUS UNPLANNED WORK

4.3.6 STORY POINT COMPLETION

4.3.7 PRODUCT HEALTH

4.3.8 QUALITY IMPROVEMENT

4.3.9 AUTOMATION ADOPTION

4.4 IT SECURITY COMPLIANCE METRICS

4.4.1 ANNUAL IT SECURITY TRAINING

4.4.2 IT SECURITY ACTIONS

4.4.3 PLAN OF ACTIONS & MILESTONES (POA&M’S)

4.4.4 VULNERABILITY REMEDIATION

Attachment M

1.0 SERVICE DELIVERY STANDARDS AND METRICS

Table M-1, Service Delivery Standards

# Service Area PWS Ref. Service Standard

1 SharePoint as a Service (SPaaS) 8.4 B. Provide SharePoint Server Major Software

Upgrades No later than 24 months after major version release

Commercial Off-the-

Shelf (COTS) Upgrades

8.1 E.

Provide COTS upgrades

No later than 12 months after major version release

3 IDU Delivery 8.2 E. xvi. Deliver IDU

Within 30 calendar days of Task Monitor (TM) approved start date or within 60 calendar days of being placed on contract if no TM approved date

4 Investigation Report (IR) Delivery 8 B Deliver IR Deliver IR within 10 Business Days

Table M-2, Service Delivery Metrics

# Service Area Description Target Metric

Reporting Period

Associated Catalog ID(s) for Deduction

(i)

Deduction Schedule (ii)

Assessment Frequency

SharePoint as a Service (SPaaS)

Delivery of each SharePoint upgrade in accordance with the requirement due date.

Pass/Fail Event

SP-001

SP-002

SP-004

10% reduction in the price of each Site Collection per month after the missed upgrade

Per Event

COTS

Upgrades Delivery of each COTS upgrade in accordance with the requirement due date.

Pass/Fail Event All Catalog Items

$1000 per month after missed upgrade Per Event

3 IDU

Delivery Delivery of IDU Pass/Fail Event DEV-002

Initial $200;

Additional $50 for every week not delivered

Per Event

4 IR Delivery Deliver IR Pass/Fail Event DEV-001 10% for every week not delivered Per Event

Table M-3, System Performance Metrics

Metric Title Service Standard Target Metric

Reporting Period

Associated Catalog

ID(s) for

Deduction (i)

Deduction Schedule

(ii)

Assessment Frequency

Return to Service (RTS)

Application & System Criticality

1, Incident Severity 1

Resolution or restoration of issues resulting in a loss of production accessibility, functionality, or availability for the following:

• Applications, systems, and subsystems categorized as Application Criticality 1

> 4 Hours

> 16 Hours

> 7 Calendar Days

Pass/Fail Monthly All Catalog Items

$5,000

$7,500

$25,000

Per Event

Application & System Criticality

1, Incident Severity 2

Resolution or restoration of issues resulting in a loss of production

Application Criticality 1

> 8 Hours

> 32 Hours

> 7 Calendar

Pass/Fail Monthly All Catalog

$7,500

$25,000

Metric Title Service Standard Target Metric

Reporting Period

Associated Catalog

ID(s) for

Deduction (i)

Deduction Schedule

(ii)

Assessment Frequency

Application & System Criticality

Incident Severity 3

Resolution or restoration of issues resulting in a loss of production

Application Criticality 1

> 14 Calendar

> 21 Calendar

> 28 Calendar

Pass/Fail Monthly All Catalog

$2,500

$10,000

Application & System Criticality

2 and 3 Incident Severity 1

Resolution or restoration of issues resulting in a loss of production

Application Criticality 2 and 3

> 8 Business Hours

> 32 Hours

Pass/Fail Monthly All Catalog

$1,000

$2,000

Metric Title Service Standard Target Metric

Reporting Period

Associated Catalog

ID(s) for

Deduction (i)

Deduction Schedule

(ii)

Assessment Frequency

Application & System Criticality

2 and 3 Incident Severity 2

Resolution or restoration of issues resulting in a loss of production

Application Criticality 2 and 3

> 16 Business

> 64 Hours

Pass/Fail Monthly All Catalog

Application & System Criticality

2 and 3 Incident Severity 3

Resolution or restoration of issues resulting in a loss of production

Application Criticality 2 and 3

> 60 Calendar

> 120 Calendar

Pass/Fail Monthly All Catalog

Metric Title Service Standard Target Metric

Reporting Period

Associated Catalog

ID(s) for

Deduction (i)

Deduction Schedule

(ii)

Assessment Frequency

Service Availability

Application Criticality 1

Availability of an individual application or system categorized as Application Criticality 1 in Attachment S, Application Inventory.

99.75% Monthly All Catalog

Items

99.25% - 99.75%:

98.25% -

99.2499%:

$10,000

< 98.2499%:

$15,000

Monthly

Application Criticality

2 & 3

Availability of an individual application or systems categorized as Application Criticality 2 in Attachment S, Application Inventory.

95% Monthly All Catalog

Items

90% - 95%:

80%-90%

$2000

< 80%:

$3,000

Monthly

Table M-4, Reporting Only Metrics

Metric Title Service Associated

Catalog ID(s)

Standard Target Metric

Incremental Development Unit

(IDU) Planning Accuracy

Number of User Story Points delivered in an IDU as a percent of the Number of Story Points planned in an IDU.

This metric is calculated against the aggregates.

DEV-002 >= 90% Monthly

Outage Notification

Notify Service Customer (SC)s for each non-emergency outage of Criticality 1 Applications no less than 1 week before outage.

This metric is calculated against the aggregates.

All Catalog Items >= 95% Annually

Outage Notification

Notify SCs for each non-emergency outage of Criticality 2 and 3 Applications no less than 72 hours before outage.

This metric is calculated against the aggregates.

All Catalog Items >= 95% Annually

Service Ticket Response

Initial Service Ticket Response with the customer within 4 business hours.

This metric is calculated against the aggregates.

All Catalog Items = 100% Annually

Commitment Reliability

A measure of the story points accepted at the end of the committed time against the story points planned.

This metric is calculated against the aggregates.

All FPT-, All ITU- >=95% Quarterly

Table M-4, Reporting Only Metrics

Metric Title Service Associated

Catalog ID(s)

Standard Target Metric

Planned vs. Unplanned Work <= 25% of stories/tasks completed should be unplanned All FPT-, All ITU- <= 25% Bi-Annually

Story Point Completion Story points must be completed and accepted on time (per team) All FPT-, All ITU- >= 95% Quarterly

Product Health Percent of healthy product deployment without defects and compliance issues

All FPT-, All ITU- >=98% Quarterly

Quality Improvement Percent of critical defects must be found before turning over to customer for testing

All FPT-, All ITU- >=95% Bi-Annually

Automation Adoption Increase in the adoption of test automation All FPT-, All ITU- >=15% Annual

Table M-5, IT Security Compliance Metrics

Metric Title Description Target Metric

Reporting Period

Deduction Schedule (ii)

Assessment Frequency

Annual IT Security Training

Successful completion of employee training in SATERN by the designated due date.

This metric is calculated against the aggregates.

100% Annually

If 100% is not reached, then $500 per delinquent employee per month

Annually

IT Security Actions

On-time completion of action items and Security Operations Center - Mitigation Action Requirements (SOC-MAR)s based upon the due date when issued (Note:

approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (AR) count as closure).

Any day a system is in a Configuration Freeze does not count toward the timeline.

This metric is calculated against the aggregates.

>= 95% Monthly $25,000 Annually

POA&Ms

On-time closure of POA&Ms, without extensions.

This metric is calculated against the aggregates.

>= 98% Monthly $25,000 Annually

Vulnerability Remediation

Vulnerability Remediation within defined timeline of vulnerability identification.

This metric is calculated against the aggregates.

>=98% Monthly $25,000 Annually

2.0 GUIDELINES

Table M-6, Service Delivery Standards Metrics Guidelines

Delivery Timeframe

Government Approval Required

Metric Calculation Methodology Examples

Days

Yes Day count starts the next business day following Government approval.

A request is submitted on Wednesday, November 1, 2022. If the request is approved on Friday, November 3, 2022, and the metric is three business days, the target delivery date is close of business Wednesday, November 8, 2022.

No Day count starts the next business day following submission by the SC.

A request is submitted on Wednesday, November 1, 2022. If the metric is three business days, the target delivery date is close of business Monday, November 6, 2022.

Hours No Hour count starts upon submission by the SC.

A request is submitted on Wednesday, November 1, 2022, at 3:30 p.m. If the metric is four business hours, the target delivery time is 9:30 a.m. Thursday, November 2, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Same Day Yes

Requests approved by the TM at 2:00 p.m. or prior are to be completed by close of business the same day.

A request is approved on Wednesday, November 1, 2022 at 2:00 p.m. The request must be completed by close of business the same day.

Requests approved by the TM after 2:00 p.m. are considered next business day.

A request is approved on Wednesday, November 1, 2022 at 2:01 p.m. The request must be completed by close of business the next day.

Table M-7, System Performance Metrics Guidelines

Metric Type Standard Measured In

Metric Calculation Methodology Examples

Return to Service (RTS)

Days Day count starts the day following loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 9:00 a.m. If the RTS metric is one business day, the target RTS is 5:30 p.m. Thursday, November 2, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Hours Starts immediately upon loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 9:00 a.m. If the RTS metric is four hours, the target RTS is 1:00 p.m. Wednesday, November 1, 2022.

Business Hours Starts immediately upon loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 1:00 p.m. If the RTS metric is 8 business hours, the target RTS is 11:00 a.m. Wednesday, November 1, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Availability N/A Starts immediately upon loss of service. N/A

Table M-8, Incident Severity Guidelines

Incident Severity Definition

1 The Incident is an immediate and total loss of availability and/or functionality. Examples:

Table M-8, Incident Severity Guidelines

Incident Severity Definition

• Entire app/system not functioning. User can login but cannot perform anything. User cannot perform primary function of application. Host service for a web application stopped.

• Total loss of Availability/functionality – Lack of access to any service provided under this Contract.

The Incident is a loss of significant/critical/Inoperable functionality or severe degradation of performance.

Examples:

• User can login but some critical function(s) do not work. The user can access the site collection home page of a

SharePoint site collection, but not any of the other pages. A user can access a page and view the existing data, but not enter new information

• Significant/Critical/Inoperable functionality – The loss of functionality would impede the application or system from completing primary functions of the applications. These incidents have no workaround available. Portions of an application or system that cease to perform as designed or at all.

The Incident is a loss of secondary or non-critical functionality or minor degradation of performance.

All other issues that affect an application, system, or service under this contract from operating as designed.

Examples

• Need information on the front page is not displayed but the information is still available on a separate page.

Main search does not work but advance search does.

• Secondary or non-Critical functionality – The customer has an acceptable workaround; the customer can still perform the function in an alternative way.

• Minor degradation of performance – The performance of the application has been diminished but is still usable to the user but inconvenient.

• Layout, color, or alt text do not function correctly. Bugs that do not stop the user from being able to perform a function.

• All other issues include spelling errors, graphical mis-alignments that don’t impair functionality

Table M-9, Application Criticality Guidelines

Application Criticality Support Hours

Incident Severity 1 Return to Service

Incident Severity 2 Return to Service

Incident Severity 3 Return to Service

1 24/7/365 4 hours 8 hours 14 calendar days

2 7:30 a.m. - 8:30 p.m. ET 8 business hours 16 business hours 60 calendar days

3 7:30 a.m. - 5:30 p.m. Local 8 business hours 16 business hours 60 calendar days

Table M-10, Vulnerability Response Timeline Guidelines

Vulnerability Rating Definition

Critical Critical Vulnerability Remediation within 15 calendar days of vulnerability identification.

High High Vulnerability Remediation within 30 calendar days of vulnerability identification.

Medium Medium Vulnerability Remediation within 30 calendar days of vulnerability identification.

Low Low Vulnerability Remediation within 60 calendar days of vulnerability identification.

3.0 EXAMPLES OVERVIEW

The information below provides examples of how the Government will calculate the metrics defined in Tables B-2 through B-5.

As used in these metrics calculations, "aggregate" is defined as "taking all units as a whole." For example, Outage Notification, in which the "aggregate" is defined as the combined availability of all Outage Notifications during the assessment frequency (monthly).

All percentages will be rounded to the nearest 2 decimal places (e.g., 94.583% rounds to 94.58%).

Disputes regarding whether a specific site is affected by an incident downtime will be resolved by the Contracting Officer (CO).

Each Example has the following format:

• Service – Description of the metric.

• Assumption – Considerations taken into account when performing the calculations.

• Calculations – The formula/approach the Government will use when calculating the metric.

• Target Metric – Value of the target metric as defined above.

• Examples – Illustrations of the metric being calculated or, in the case of pass/fail metrics, a demonstration of when the metric is or is not met.

All tables in the examples should not be construed as representing true estimates.

Some calculations use an average hours/month (i.e. ‘H’) which is calculated as follows:

𝐻𝐻 =

day ∗ 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑦𝑦𝑑𝑑𝑦𝑦 months year

24 ∗ 365

12 =

12 = 730

Equation 1

4.0 METRIC EXAMPLES

4.1 SERVICE DELIVERY STANDARDS

4.1.1 SHAREPOINT AS A SERVICE (SPaaS)

Service:

Delivery of each SharePoint upgrade in accordance with the requirements due date.

Assumptions:

The SharePoint as a Service Upgrade Delivery metric is based on the due dates specified in the Performance Work Statement (PWS), Section 8.4.B; therefore, the date is either met or it is not.

SharePoint site collections SP-001, SP-002 and SP-004 will receive the discount.

Target Metric: Pass / Fail

Example:

The SharePoint 2022 upgrade is delivered November 1, 2024. SharePoint 2022 was released October 1, 2022. PWS 8.4 B requires the update to be completed 24 months after major version released by the vendor, therefore the metric is failed. SharePoint site collections SP-001, SP-002 and SP-004, will each be assessed a 10% discount on the invoice for the month of October.

4.1.2 COMMERCIAL OFF-THE-SHELF (COTS) UPGRADES

Service:

Delivery of each COTS upgrade in accordance with the requirements due date.

Assumptions:

The Commercial Off-the-Shelf (COTS) Upgrades Delivery metric is based on the due dates specified in the PWS, Sections 8.1 E; therefore, the date is either met or it is not. O&M Catalog items with the defined PWS section will receive the discount.

For COTS products running in production which are several versions behind commercially available version, the contractor shall work with NASA to determine a course of action and timeframe for implementation of commercially available version.

Target Metric: Pass / Fail

Example:

The Atlassian Jira upgrade 8.20 is delivered November 1, 2024. Atlassian Jira 8.20 was released October 1, 2023. PWS 8.1 E requires the update to be completed 12 months after major version released by the vendor, therefore the metric is failed. Catalog item KSC-002-S is for Atlassian Jira support and references PWS 8.1 E. Catalog item KSC-002-S will be assessed a $1000 a month invoiced deduction for the month of October.

4.1.3 IDU DELIVERY

Service:

Delivery of each IDU in accordance with the requirements due date.

Assumptions:

None Target Metric: Pass / Fail

Example:

There is an Iteration approved to be completed Oct 24th, 2022 that consists of 4 IDUs. The Iteration is delivered October 26th, 2022. PWS 8.2 E xvi requires the IDU to be Deliver IDU within 30 calendar days of TM approved delivery date or within 60 calendar days of being placed on contract if no TM approved date, therefore the metric is failed. The 4 catalog items DEV-002 for the Iteration will be assessed a $200 invoiced discount each for a total of $800 on the invoice for the month of October.

4.1.4 IR DELIVERY

Service:

Delivery of each IR in accordance with the requirements due date.

Assumptions:

The start date is when the IR is officially on the contract if purchased or when the Government request the contractor to deliver an IR, whichever is later.

Target Metric: Pass / Fail

Example:

There is an IR is requested by the Government on completed Oct 5th, 2022. The IR is scheduled to be delivered October 19th, 2022. The IR is delivered October 21st, 2022. PWS 8 B requires the IR IDU to be delivered within 10 business days, therefore the metric is failed. The IR catalog item DEV-001 will be assessed a 20% invoiced discount on the invoice for the month of October.

4.2 SYSTEM PERFORMANCE METRICS

4.2.1 APPLICATION & SYSTEM RETURN TO SERVICE

Service:

Resolution or restoration of issues resulting in a loss of production accessibility, functionality, or

A. Applications, systems, and subsystems categorized with an Application & System Criticality 1 that received an unplanned outage categorized with an Incident Severity of 1

B. Applications, systems, and subsystems categorized with an Application & System Criticality 1 that received an unplanned outage categorized with an Incident Severity of 2

C. Applications, systems, and subsystems categorized with an Application & System Criticality 1 that received an unplanned outage categorized with an Incident Severity of 3

D. Applications, systems, and subsystems categorized with an Application & System Criticality 2 & 3 that received an unplanned outage categorized with an Incident Severity of 1

E. Applications, systems, and subsystems categorized with an Application & System Criticality 2 & 3 that received an unplanned outage categorized with an Incident Severity of 2

F. Applications, systems, and subsystems categorized with an Application & System Criticality 2 & 3 that received an unplanned outage categorized with an Incident Severity of 3

a. The RTS time will start for accessibility upon actual loss of service until service is restored regardless of reported incident timestamp.

b. The RTS time for functionality will start upon first reported incident timestamp.

c. Any application or subsystem in a multi-application platform or system will be considered individually.

d. Expiration of cryptographic certificates are included in RTS.

e. RTS System Performance Metrics are additive with other metrics.

f. Non-production environments provided to external parties under Hosted Only are considered production environments for metric purposes.

Target Metric: Pass / Fail

Example 1:

On July 6, 2024, TechDoc (an KSC-001-S Application & System Criticality 1) has experienced an issue resulting in loss of service starting at 7:00 a.m. The system is returned to service at 12:15 p.m. the same day, resulting in an RTS incident of 5.25 hours.

This incident is greater than 4 hours and will result in a deduction of $5,000 for catalog offering KSC-001-S for the month of July.

In this example, no deduction will be assessed against other catalog items.

Example 2:

On March 6, 2024, BOSS Maximo (an KSC-003-S Application & System Criticality 1) has experienced an issue resulting in loss of service starting at 9:00 a.m. The system is returned to service at 2:30 p.m. the same day, resulting in an RTS incident of 5.50 hours. This incident is greater than 4 hours and will result in a deduction of $5,000.

On March 7, 2024, SAP (an Application & System Criticality 1 system maintained under APS-003-S) has experienced an issue resulting in loss of service starting at 9:00 a.m. The system is returned to service at 5:30 p.m. the same day, resulting in an RTS incident of 8.50 hours. This incident is greater than 4 hours and will result in a deduction of $5,000

In this example, the month of March will be assessed a RTS deduction for both KSC-003-S and APS-003-S for a total of $10,000 for the month of March.

4.2.2 APPLICATION & SYSTEM AVAILABILITY

The availability of an individual application or system categorized as an Application & System Criticality 1, 2, & 3.

Assumptions:

a. The availability time will start for accessibility upon actual loss of service until service is restored regardless of reported incident timestamp.

b. The availability time for functionality will start upon first reported incident timestamp.

c. Any application or subsystem in a multi-application platform or system will be considered individually.

d. Expiration of cryptographic certificates are included in availability.

e. Availability System Performance Metrics are additive with other metrics.

f. Non-production environments provided to external parties under Hosted Only are considered production environments for metric purposes.

Calculation Directions:

ASCA = �1−𝐷𝐷

𝐻𝐻� ∗ 100, where

Equation 2

ASCA = Application & System Criticality Availability D = the sum of hours that an individual application or system has experienced RTS incidents in a month, including both accessibility and functionality.

H = average hours in a month (See Document Overview for calculation).

Target Metric: >=99.75% Monthly

Example 1:

On April 4, 2024, CAPI (an KSC-001-S 3.1.4 Application & System Criticality 1 system) experienced a 3.75-hour RTS incident. On April 6, 2024, CAPI experienced another RTS incident taking 2.00-hours, and a third 2 hr. incident on April 16, 2024. All three total 7.75 hours of RTS incidents in the month of April.

Using Equation 1 and Equation 2 where

D = 3.75+2+2 = 7.75, and H = 730.

ASCA = �1− 𝐷𝐷

𝐻𝐻� ∗ 100 =

�1 − 7.75

� ∗ 100 = 98.94%

In this example, the Application & System Criticality 1 Availability is below the target 99.75% and falls in the 98.25% - 99.2499% range and will result in a deduction of $10,000 for catalog offering KSC-001-S for the month of April.

Table M-11, Applications Criticality 1 Availability Example Application Criticality 1

Availability Deduction Schedule System A Availability

99.25% - 99.75%: $5,000 98.25% - 99.2499%: $10,000 X

< 98.2499%: $15,000

Example 2:

In this example two applications experienced outages as follows.

On April 4, 2024, CAPI (an KSC-005 Application & System Criticality 1) experienced a 5.00-hour RTS incident. On April 6, 2018, CAPI experienced another RTS incident taking 3.00-hours, and on April 16, 2024, CAPI experienced a 1.00-hour incident. All combined CAPI experienced 9.00-hours of RTS incidents.

On April 10, 2024, TechDoc (also an KSC-005 Application & System Criticality 1) experienced a 14.00 hour RTS incident. On April 12, 2024, TechDoc experienced another RTS incident of

10.00 hours. On April 19, 2024 a third RTS incident occurred taking 6.00 hours. All combined, TechDoc experienced 30.00 hours of RTS incidents.

Using Equation 1 and Equation 2 where

For CAPI

D = 5+3+1 = 9, and

ASCA𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 = �1− 𝐷𝐷

𝐻𝐻� ∗ 100 =

�1 − 9 � ∗ 100 = 98.77%

For TechDoc D = 14.00+10.00+6.00 = 30.00, and

ASCA𝑇𝑇𝑇𝑇𝑇𝑇ℎ𝐷𝐷𝐷𝐷𝑇𝑇 = �1− 𝐷𝐷

𝐻𝐻� ∗ 100 =

�1 − 30.00

� ∗ 100 = 95.89%

In this example, the Application & System Criticality 1 Availability for CAPI is 98.77% and for TechDoc it is 95.89%. Both of these fall below the target metric of 99.75%. This will result in two deductions: one for CAPI and one for TechDoc, as shown in Table M-12.

Table M-12, Application Criticality 1 Availability Example Application Criticality 1

Availability Deduction Schedule System A Availability

99.25% - 99.75%: $5,000 98.25% - 99.2499%: $10,000 X

< 98.2499%: $15,000 X

In this example, the deductions are $10,000 for CAPI and $15,000 for TechDoc for a total of $25,000 for the month of April against catalog offering KSC-005.

4.3 REPORTING ONLY METRICS

4.3.1 INCREMENTAL DEVELOPMENT UNIT (IDU) PLANNING ACCURACY

Service:

Number of User Story Points delivered in an IDU as a percent of the Number of Story Points planned in an IDU. This metric is calculated against the aggregates.

Assumptions:

None

Calculation Directions:

IDUPA = �

𝐷𝐷𝐷𝐷𝐷𝐷

𝐷𝐷𝐷𝐷𝐷𝐷� ∗ 100, where

Equation 3

IDUPA = the IDU Planning Accuracy DUD = the sum of all story points delivered from all IDUs DUP = the number of story points planned for all IDUs

Target Metric: >= 90% Monthly

Example:

The following table shows the story points planned and delivered for several IDUs in a month.

Table M-13, Story Point Example

IDU A IDU B IDU C IDU D IDU E Total Delivered

(DUD) 22 18 60 35 27 162

Planned

(DUP) 20 20 60 30 30 160

Using Equation 3 and the values of Table M-13, where

DUD = 162 story points delivered DUP = 160 story points planned

IDUPA = �

𝐷𝐷𝐷𝐷𝐷𝐷

𝐷𝐷𝐷𝐷𝐷𝐷� ∗ 100 =

160� ∗ 100 = 101.25%

The IDU Planning Accuracy is greater than the 90% target. This metric is a reporting only metric.

4.3.2 OUTAGE NOTIFICATION

Notify SCs for each non-emergency outage of all Applications. This metric is calculated against the aggregates.

Assumptions:

None

The Monthly Outage Notification Metric is calculated in accordance with the following formula.

MONM = �1 −

𝑀𝑀𝑀𝑀

𝑂𝑂

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 4

MONM = the Monthly Outage Notification Metric, MN = the number of Missed non-emergency outage Notifications for the month, O = the total number of non-emergency Outages for the month.

The annual outage notification metric is calculated using the following formula.

AONM = � (𝑀𝑀𝑂𝑂𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑇𝑇𝑆𝑆

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

, where

Equation 5

AONM = the Annual Outage Notification Metric, MONM = the Monthly Outage Notification Metric, n = the Government Fiscal Year months (i.e., October thru September) X = the number of months in a performance period (12 for a full Fiscal Year).

Target Metric: >= 95% Annually

The following table outlines the number of non-emergency outages (O) taken in a given month and the number of times the contractor failed to give proper notification (MN).

Table M-14, Non-Emergency Outages Example

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept

Missed Notification

(MN)

1 3 2 0 3 2 1 2 1 0 0 0

Outages (O) 10 10 10 14 13 10 10 10 10 20 25 30

Monthly Outage

Notification Metric

(MONM)

90% 70% 80% 100% 76.92% 80% 90% 80% 90% 100% 100% 100%

Using Equation 5 and the data in Table M-14, where n = Oct thru Sept, MONM = the metrics in Table B-14 and X = 12

AONM = � (𝑀𝑀𝑂𝑂𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑇𝑇𝑆𝑆

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

(90 +70 + 80 + 100 + 76.92 + 80 + 90 +80 + 90 +100 + 100 +100)/12 = = 88.08%

In this example, the Annual Outage Notification Metric of 88.08% fell below the 95% target.

4.3.3 SERVICE TICKET RESPONSE

Service:

Respond to service tickets for all Applications. This metric is calculated against the aggregates.

Assumptions:

None

Calculation Directions:

The Monthly Service Ticket Response Metric is calculated in accordance with the following formula.

MSTRM = �1 −

𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀

𝑂𝑂

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 6

MSTRM = the Monthly Service Ticket Response Metric, MSTR = the number of Missed Service Ticket Responses for the month, ST = the total number of Service Tickets for the month.

The annual outage notification metric is calculated using the following formula.

ASTRM = �

(𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)

𝑋𝑋

𝑆𝑆𝑇𝑇𝑆𝑆

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

, where

Equation 7

ASTRM = the Annual Service Ticket Response Metric, MSTRM = the Monthly Service Ticket Response Metric, Target Metric: = 100% Annually

The following table outlines the number of service tickets (ST) taken in a given month and the number of times the contractor failed to respond in time notification (MSTR).

Table M-15, Service Ticket Response Example

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept

Missed Service Ticket

Response

(MSTR)

1 3 2 0 3 2 1 2 1 0 0 0

Service Tickets (ST) 10 10 10 14 13 10 10 10 10 20 25 30

Monthly Service Ticket

Response Metric

(MSTRM)

Using Equation 7 where

MSTRM=the metrics in Table B-15 and X = 12

ASTRM = �

(𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)

𝑋𝑋

𝑆𝑆𝑇𝑇𝑆𝑆

(90+70+80+100+76.92+80+90+80+90+100+100+100)/12 = 88.08%

In this example, the Annual Service Ticket Response Metric of 88.08% fell below the 100% target.

4.3.4 COMMITMENT RELIABILITY

A measure of the story points accepted at the end of the committed time against the story points planned in an iteration. This metric is calculated against the aggregates.

Assumptions:

a. The story points accepted will be calculated at the end of the agreed to time frame and not when the work is invoiced.

b. For fixed price teams the number of story points accepted will be calculated at the end of the 2-week period.

c. For Incremental Team Units (ITU)s the number of story points accepted will be calculated at the end of the agreed to period for the Iteration.

The Monthly Commitment Reliability Metric is calculated in accordance with the following

MCRM = �1 −

𝑀𝑀𝐷𝐷𝑃𝑃

𝑀𝑀𝐷𝐷𝐷𝐷

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 8

MCRM = the Monthly Commitment Reliability Metric, SPA = the total number of Story Points Accepted for the month.

SPP = the total number of Story Points Planned for the month.

The Quarterly Commitment Reliability Metric is calculated using the following formula.

QCRM = �

(𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)

X

𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

, where

Equation 9

QCRM = the Quarterly Commitment Reliability Metric, MCRM = the Monthly Commitment Reliability Metric, X = Number of months in the quarter

Target Metric: >= 95% Quarterly

The following table outlines the number of Story Points Accepted (SPA) taken in a given month and the number of Story Points Planned (SPP) in a given month.

Table M-16, Commitment Reliability Example Oct Nov Dec

SPP 64 64 66

SPA 64 62 64

MCRM 100% 96.88% 96.97%

Using Equation 9 where n = Oct thru Dec, MCRM = the metrics in Table B − 16 and X = 3

QCRM = �

(𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)

X

𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

(100 +96.88 + 96.97)/3 = 97.95%

In this example, the Quarterly Commitment Reliability Metric of 97.95% and is above the 95%

4.3.5 PLANNED VERSUS UNPLANNED WORK

A measure of the number of stories completed and accepted that indicates how many were planned and how many were unplanned. This metric is calculated against the aggregates.

Assumptions:

Planned Work is work that has been approved by the Government at the beginning of an iteration.

Unplanned Work is work that is not approved by the Government or planned at the beginning of a specific iteration or sprint.

The Monthly Planned versus Unplanned Work Metric is calculated in accordance with the following formula.

MPvUPWM = �

UPW

𝐷𝐷𝑃𝑃 + 𝐷𝐷𝐷𝐷𝑃𝑃

� ∗ 100

Equation 10

MPvUPWM = the Planned versus Unplanned Work Metric UPW = the cumulative Unplanned stories/tasks for the duration

PW = the cumulative Unplanned stories/tasks for the duration

The Bi-Annually Planned versus Unplanned Work Metric is calculated in accordance with the following formula.

BPvUPWM =

∑ (𝐷𝐷𝐷𝐷𝑃𝑃𝑛𝑛)𝑀𝑀𝑀𝑀𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝐷𝐷𝑃𝑃𝑛𝑛)𝑀𝑀𝑀𝑀𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 + ∑ (𝐷𝐷𝐷𝐷𝑃𝑃𝑛𝑛)𝑀𝑀𝑀𝑀𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∗ 100

Equation 11

BPvUPWM = the Planned versus Unplanned Work Metric UPW = the cumulative Unplanned stories/tasks for the duration

PW = the cumulative Unplanned stories/tasks for the duration n = the Government Fiscal Year months (i.e., October thru March)

Target Metric: <= 25% Bi-Annually

The following table indicates the Planned and Unplanned completed and accepted stories for a six-month duration.

Table M-17, Planned vs Unplanned Work Example Oct Nov Dec Jan Feb Mar

PW 50 60 58 56 62 58

UPW 10 14 12 10 16 12

Using Equation 11 and the values of Table M-17 where

BPvUPWM =

∑ (𝐷𝐷𝐷𝐷𝑃𝑃𝑛𝑛)𝑀𝑀𝑀𝑀𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝐷𝐷𝑃𝑃𝑛𝑛)𝑀𝑀𝑀𝑀𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 + ∑ (𝐷𝐷𝐷𝐷𝑃𝑃𝑛𝑛) 𝑀𝑀𝑀𝑀𝑀𝑀

(10+14+12+10+16+12) / ((50+60+58+56+62+58) + (10+14+12+10+16+12)) *100 = 17.7% 74 / (344 + 74) *100 = 17.7%

In this example, the % of Unplanned Work is 17.7% and within the target range (<=25%).

4.3.6 STORY POINT COMPLETION

A measure of Story Points completed and accepted on time (per team). This metric is calculated against the aggregates.

Assumptions:

a. The Story Points completed and accepted will be calculated at the end of the agreed to time frame and not when the work is invoiced.

b. For fixed price teams the Story Points completed and accepted will be calculated at the end of the 2-week period.

c. For Incremental Team Units (ITU)s the Story Points completed and accepted will be calculated at the end of the agreed to period for the Iteration.

The Monthly Story Point Completion Metric is calculated in accordance with the following

MSPCM = �

𝑀𝑀𝐷𝐷𝑃𝑃

SPP

� ∗ 100

Equation 12

MSPCM = the Story Point Completion Metric SPA = the total Story Points Accepted for the month SPP = the total Story Points Planned for the month

The Quarterly Story Point Completion Metric is calculated in accordance with the following

QSPCM =

∑ (𝑀𝑀𝐷𝐷𝑃𝑃𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝑀𝑀𝐷𝐷𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∗ 100

Equation 13

QSPCM = the Quarterly Story Point Completion Metric SPA = the total Story Points Accepted for the month SPP = the total Story Points Planned for the month n = the Government Fiscal Year months (i.e., October thru December)

Target Metric: >= 95% Quarterly

The following table indicates the Story Points Planned and Accepted by a team in a respective quarter.

Table M-18, Story Point Completion Example

Team B (24-point capacity) Oct Nov Dec

SPP 28 30 28

SPA 22 28 28

Using Equation 13 and the values of Table M-18 where n = Oct thru Dec, QFPM =

∑ (𝑀𝑀𝐷𝐷𝑃𝑃𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝑀𝑀𝐷𝐷𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

∗ 100 =

(22+28+28) / (28+30+28) * 100 = 90.7%

In this example, the SPCM of 90.7% missed the target (<=95%).

4.3.7 PRODUCT HEALTH

A measure of healthy product deployment without defects and compliance issues. This metric is calculated against the aggregates.

Assumptions:

a. Compliance issues are defined per ATT D, Applicable Documents List.

b. Deployment is defined as code migration to a production platform

c. The applicable time frame is one year

The Monthly Product Health Metric is calculated in accordance with the following formula.

MPHM = �

𝐷𝐷𝐷𝐷𝐷𝐷+ 𝑀𝑀𝐶𝐶

𝐷𝐷 � ∗ 100, where

Equation 14

MPHM = the Monthly Product Health Metric D = Deployments DEF = Defects CI = Compliance Issues

The Quarterly Product Health Metric is calculated in accordance with the following formula.

QPHM = �1 −

(∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 + ∑ (𝑀𝑀𝐶𝐶𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 )

∑ (𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 15

QPHM = the Quarterly Product Health Metric D = Deployments DEF = Defects CI = Compliance Issues n = the Government Fiscal Year months (i.e., October thru December)

Target Metric: >= 98% Quarterly

The following table indicates the number of deployments, defects, and compliance issues for a given quarter.

Table M-19, Product Health Example Oct Nov Dec

D 60 75 65

DEF 1 2 1

CI 2 3 1

Using Equation 15 and the values of Table M-19 where n = Oct thru Dec, QPHM = �1 −

(∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 + ∑ (𝑀𝑀𝐶𝐶𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 )

∑ (𝐷𝐷𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

� ∗ 100 =

(1 - ((1+2+1) + (2+3+1)) / (60+75+65)) * 100 = 95%

In this example, the Product Health Metric of 95% and is below the >=98% target.

4.3.8 QUALITY IMPROVEMENT

A measure of critical defects identified before customer testing. This metric is calculated against the aggregates.

Assumptions:

a. Defects are not identified during programmer unit testing.

b. Defects are counted once delivered for Quality Assurance testing.

The Monthly Quality Improvement Metric is calculated in accordance with the following

MQIM =

𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀

𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀 + 𝐷𝐷𝑀𝑀 ∗ 100

Equation 16

MQIM = the Monthly Quality Improvement Metric DEFPC = Defects identified prior to customer testing (functional, regression, integration)

DC = Defects identified by the customer

The Bi-Annually Quality Improvement Metric is calculated in accordance with the following formula.

BQIM =

∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

(∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀𝑛𝑛) + 𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 ∑ (𝐷𝐷𝑀𝑀𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 )

∗ 100

Equation 17

BQIM = the Monthly Quality Improvement Metric DEFPC = Defects identified prior to customer testing (functional, regression, integration)

DC = Defects identified by the customer

Target Metric: >= 95% Bi-Annually

The following table outlines the amount of Business Value Planned (BVP) in a given month and the amount of Business Value Delivered (BVD) in a given month.

Table M-20, Quality Improvement Example Oct Nov Dec Jan Feb Mar

DEFPC 25 40 30 45 20 30

DC 2 1 0 1 2 3

Using Equation 17 and the values of Table M-20 where n = Oct thru Mar, BQIM =

∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

(∑ (𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝑀𝑀𝑛𝑛) + 𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 ∑ (𝐷𝐷𝑀𝑀𝑛𝑛)𝐷𝐷𝑇𝑇𝑇𝑇

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂 )

((25+40+30+45+20+30) / ((25+40+30+45+20+30) + (2+1+0+1+2+3))) * 100 = 95.48%

In this example, the Bi-Annually Quality Improvement Metric of 95.48% does not meet the <=95% target.

4.3.9 AUTOMATION ADOPTION

A measure of the increase in test automation. This metric is calculated against the aggregates.

Assumptions:

None

The Test Automation Metric is calculated in accordance with the following formula.

TA = (

𝐷𝐷𝑀𝑀𝐷𝐷

𝐵𝐵𝑀𝑀𝐷𝐷

− 1) ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 18

TA = the Test Automation Metric BRP = total automated tests at the Beginning of the Reporting Period ERP = total automated tests at the End of the Reporting Period

Target Metric: >= 15% Annually

The following table identifies the number of automated tests from one contract year to the next contract year.

Table M-21, Automation Adoption Example

BRP 500

ERP 600

Using Equation 18 and the values of Table M-21 where

TA = (

𝐵𝐵𝑀𝑀𝐷𝐷

𝐵𝐵𝑀𝑀𝐷𝐷

− 1) ∗ 100 =

(600 / 500 – 1) * 100 = 20%

In this example, the Test Automation Metric of 20% meets the >=15% target.

4.4 IT SECURITY COMPLIANCE METRICS

4.4.1 ANNUAL IT SECURITY TRAINING

Service:

Successful completion of employee training in SATERN by the designated due date. This metric is calculated against the aggregates.

N/A

Calculation Directions:

AD = ( � DE𝑛𝑛) ∗ $500

𝑀𝑀

,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 19

AD = Annual Deduction, DE = Delinquent Employees in a M, M = the final month of the summation, n = the Government Fiscal Year months (e.g. October thru September)

Target Metric: 100% Annually

Table M-22, Training Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

FY19

Training Incomplete Employees

N/A N/A 149 130 120 100 10 5 2 1 0 0

The FY24 training course was released by NASA on December 1, 2023 with a due date of April 30, 2024.

Metrics are calculated at the end of the month.

Table M-22 indicates the number of employees who have not completed the training December through March, and the number of delinquent employees April through September.

Using Equation 19 where M = September, 𝐷𝐷𝐷𝐷𝐶𝐶𝑆𝑆𝑀𝑀 = 10, 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝑀𝑀 = 5, 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝑛𝑛 = 2, 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝐽𝐽 = 1, 𝐷𝐷𝐷𝐷𝑂𝑂𝑇𝑇𝑂𝑂,𝑁𝑁𝐷𝐷𝑁𝑁,𝐷𝐷𝑇𝑇𝑇𝑇,𝐽𝐽𝑀𝑀𝑛𝑛,𝐹𝐹𝑇𝑇𝐹𝐹,𝑀𝑀𝑀𝑀𝑀𝑀,𝐶𝐶𝐽𝐽𝐴𝐴,𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂 = 0

AD = ( � DE𝑛𝑛) ∗ $500

𝑀𝑀

�𝐷𝐷𝐷𝐷𝑂𝑂𝑇𝑇𝑂𝑂 + 𝐷𝐷𝐷𝐷𝑁𝑁𝐷𝐷𝑁𝑁 + 𝐷𝐷𝐷𝐷𝐷𝐷𝑇𝑇𝑇𝑇 + 𝐷𝐷𝐷𝐷𝐽𝐽𝑀𝑀𝑛𝑛 + 𝐷𝐷𝐷𝐷𝐹𝐹𝑇𝑇𝐹𝐹 + 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝑀𝑀 + 𝐷𝐷𝐷𝐷𝐶𝐶𝑆𝑆𝑀𝑀 + 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝑀𝑀 + 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝑛𝑛 + 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝐽𝐽 + 𝐷𝐷𝐷𝐷𝐶𝐶𝐽𝐽𝐴𝐴 + 𝐷𝐷𝐷𝐷𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂� ∗ $500 =

(0 + 0 + 0 + 0 + 0 + 0 + 10 + 5 + 2 + 1 + 0 + 0) ∗ $500 =

18 ∗ $500 = $9,000

Since 18 of the employees did not complete the required training by April 30, 2024, the contractor will be penalized $9,000.

4.4.2 IT SECURITY ACTIONS

On-time completion of action items based upon the due date when issued (Note: approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (AR) count as closure). This metric is calculated against the aggregates.

Assumptions:

a. Contractor initiated extensions even with NASA approval is not considered on time closure.

b. Approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (ARs) count as closure.

The metric will be computed using the following equation:

𝐶𝐶𝑀𝑀𝑀𝑀 𝑃𝑃𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝑀𝑀𝐶𝐶𝑀𝑀𝑛𝑛𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ 𝑃𝑃𝑛𝑛𝑀𝑀

∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 20

CLT = Total number of actions closed on time during the FY up to the current month.

A = Total number of actions issued from beginning of FY to include those with due dates in the current month.

M = the final month of the summation.

Target Metric: >= 95% Annually

Table M-23, Action Metric Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Total Completed On Time 5 4 4 2 6 4 5 5 4 3 1 3 46

Due 5 6 4 2 6 4 7 5 4 3 1 3 50

Using the data in Table M-23 and Equation 20 we get the following values for this example.

CLT = {5,4,4,2,6,4,5,5,4,3,1,3}

A = {5,6,4,2,6,4,7,5,4,3,1,3} n = September

𝐶𝐶𝑀𝑀𝑀𝑀 𝑃𝑃𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝑀𝑀𝐶𝐶𝑀𝑀𝑛𝑛𝑀𝑀

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ 𝑃𝑃𝑛𝑛𝑀𝑀

∑ 𝑀𝑀𝐶𝐶𝑀𝑀𝑛𝑛

𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ 𝑃𝑃𝑛𝑛

𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

(5 + 4 + 4 + 2 + 6 + 4 + 5 + 5 + 4 + 3 + 1 + 3) (5 + 6 + 4 + 2 + 6 + 4 + 7 + 5 + 4 + 3 + 1 + 3) =

50 = 92.00%

In this example, the metric was not met and the contractor will be accessed a $25,000 penalty.

4.4.3 PLAN OF ACTIONS & MILESTONES (POA&M’S)

Service:

On-time closure of Plan of Action & Milestone (POA&Ms), without extensions. This metric is calculated against the aggregates.

Assumptions:

a. Contractor initiated extensions, even with NASA approval, is not considered on time closure.

b. Extension due to third party vendor constraints would require CO approval.

Calculation Directions:

𝐷𝐷𝑂𝑂𝑃𝑃&𝑀𝑀 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐷𝐷𝑀𝑀𝑂𝑂𝑀𝑀𝑛𝑛

𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝐷𝐷𝐷𝐷𝑛𝑛 − 𝐷𝐷𝑃𝑃𝐷𝐷𝑛𝑛)𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 21

PCOT = Total number of all ITSSP POA&M’s closed on time during the FY up to the current month.

PD = Total number of all ITSSP POA&M’s issued from beginning of FY to include those with due dates in the current month

PAE = POA&M’s with approved Contracting Officer (CO) extensions.

Target Metric: >= 98% Annually

Table M-24 has representative values for a fiscal year.

Table M-24, Fiscal Year POA&M Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Total POA&M Closed on Time 10 10 7 8 3 8 7 10 4 7 5 10 89

POA&M Due 10 12 7 8 4 12 7 10 8 7 5 10 100 CO Approved Extensions 0 2 0 0 0 0 0 0 0 0 0 0 2

POA&M Due - CO Approved Extensions

10 10 7 8 4 12 7 10 8 7 5 10 98

For this example.

Using the data in Table M-24 and Equation 21 we get the following values for this example.

PCOT = {10,10,7,8,3,8,7,10,4,7,5,10}.

PD = {10,12,7,8,4,12,7,10,8,7,5,10}

PAE = {0,2,0,0,0,0,0,0,0,0,0,0}.

𝐷𝐷𝑂𝑂𝑃𝑃&𝑀𝑀 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐷𝐷𝑀𝑀𝑂𝑂𝑀𝑀𝑛𝑛

𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

∑ (𝐷𝐷𝐷𝐷𝑛𝑛 − 𝐷𝐷𝑃𝑃𝐷𝐷𝑛𝑛)𝑆𝑆𝑇𝑇𝑆𝑆𝑂𝑂

(10 + 10 + 7 + 8 + 3 + 8 + 7 + 10 + 4 + 7 + 5 + 10)

(10 − 0) + (12 − 2) + (7 − 0) + (8 − 0) + (4− 0) + (12− 0) + (7− 0) + (10 − 0) + (8 − 0) + (7 − 0) + (5 − 0) + (10 − 0) ∗ 100 =

98 ∗ 100 = 90.82%

For this example, the target metric was missed, and the contractor will be accessed a $25,000 penalty at the end if the FY.

4.4.4 VULNERABILITY REMEDIATION

On-time remediation of vulnerabilities within the defined timeline per vulnerability rating level.

This metric is calculated against the aggregates.

Assumptions:

a. Remediation of a vulnerability occurs under the following conditions:

a. Vulnerability is removed.

b. A POA&M is created and approved by NASA.

c. A RBD is created and approved by NASA.

b. The clock stops when submitted to NASA for review and starts again if the POA&M or RBD is rejected.

c. The clock starts over if a RBD expires.

Calculation Directions:

The Monthly Vulnerability Remediation Metric is calculated in accordance with the following

MVRM = �1 −

𝑀𝑀𝑀𝑀

𝑀𝑀

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 22

MVRM = the Monthly Vulnerability Remediation Metric, MV = the number of Missed Vulnerabilities for the month, V = the total number of Vulnerabilities for the month.

The annual vulnerability remediation metric is calculated using the following formula.

AVRM = � (𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑇𝑇𝑆𝑆

𝑛𝑛=𝑂𝑂𝑇𝑇𝑂𝑂

, where

Equation 23

AVRM = the Annual Vulnerability Remediation Metric, MVRM = the Monthly Vulnerability Remediation Metric, Target Metric: >= 98% Annually

The following table outlines the number of vulnerabilities (V) identified in a given month and the number of times the contractor failed to address the vulnerability in time (MV).

Table M-25, Non-Emergency Outages Example

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept

Missed Vulnerability

(MV)

1 3 2 0 3 2 1 2 1 0 0 0

Vulnerabilities

(V) 10 10 10 14 13 10 10 10 10 20 25 30

Monthly Vulnerability Remediation

Metric (MVRM)

Using Equation 23 where

MVRM=the metrics in Table B-25 and X = 12

AVRM = � (𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑇𝑇𝑆𝑆

(90 +70 + 80 + 100 + 76.92 + 80 + 90 +80 + 90 +100 + 100 +100)/12 = = 88.08%

In this example, the Annual Vulnerability Remediation Metric of 88.08% fell below the 98%