ATT M - Fixed Price Service Delivery Standards and Metrics - Draft.pdf

ATTACHMENT M

NASA CONSOLIDATED APPLICATIONS AND

PLATFORM SERVICES (NCAPS)

SERVICE DELIVERY STANDARDS AND METRICS

RFP 80TECH22R0002

CONTRACT #TBD

DATE: TBD

NCAPS Contract – 80TECH22R0002 Attachment M Service Delivery Standards and Metrics

Table of Contents

1.0 SERVICE DELIVERY STANDARDS AND METRICS

2.0 EXAMPLES OVERVIEW

3.0 INFRASTRUCTURE EXAMPLES

3.1 Application Operation & Software Services

3.1.1 Outage Notification

3.1.2 SharePoint as a Service (SPaaS)

3.1.3 Application & System Criticality 1, Incident Severity 1

3.1.4 Application & System Criticality 1

3.1.5 Incremental Development Unit (IDU) Planning Accuracy

4.0 IT SECURITY COMPLIANCE METRICS

4.1 Annual IT Security Training

4.2 IT Security Actions

4.3 Plan of Actions & Milestones (POA&M’s)

Appendix B Service Delivery Standards and Metricsi

1.0 SERVICE DELIVERY STANDARDS AND METRICS

Table B-1, Service Delivery Standards

# Service Area PWS/Catalog Ref. Service Standard

1 SharePoint as a Service (SPaaS) 6.1.4 b. Provide SharePoint Server Major Software

Upgrades No later than 24 months after major version release

2 COTS Upgrades All Provide Major COTS upgrades No later than 12 months after major version release

3 Startup Delivery 6.1.2 A. vii. Deliver Startup

Within 30 calendar days of TM approved start date or within 60 calendar days of being placed on contract if no TM approved date.

4 IDU Start 6.1.2 E. xvi. Start of IDU implementation.

Start delivery of Iterations within 1 month after Startup completion or within 1 month after completion of previous Iteration.

5 IDU Delivery 6.1.2 E. xvii. Deliver IDU Complete each Iteration within 1 month of Iteration start unless otherwise approved by the TM.

6 Master Data All Resolve Emergency Master Data tickets. Within 1 Business Day

7 Master Data All Resolve Exceptions Master Data tickets. Within 8 Business Days

8 Master Data All Resolved other Master Data tickets. Excludes Emergency and Exception Master Data tickets. Within 2 Business Days

Table B-2, Service Delivery Metrics

Service Area Description Target Metric

Reporting Period

Associated Catalog ID(s) for Deduction

Deduction Schedule

Assessment Frequency

SharePoint as a Service (SPaaS)

Delivery of each SharePoint upgrade in accordance with the requirement due date.

Pass/Fail Event

SP-001

SP-002

SP-004

10% reduction in the price of each

Site Collection per month after the missed upgrade

Per Event

COTS Upgrades Delivery of each COTS upgrade in accordance with the requirement due date.

Pass/Fail Event All $1000 per month after missed upgrade

Per Event

Startups Delivery of the Startup Pass/Fail Event

DEV-003

DEV-004

DEV-005

$500 per week after missed delivery Per Event

Iteration Start Start the implementation of an Iteration Pass/Fail Event DEV-002

$500 per week after missed delivery Per Event

Iteration Completion

Delivery of Iteration Pass/Fail Event DEV-002

Initial $1000;

Additional $500 for every week not delivered per

Iteration

Per Event

Master Data Resolve Emergency Master Data tickets. >= 99% Monthly

APS-002-S

thru

APS-056-S

$5K Semi-

Annually

Master Data Resolve Exceptions Master Data tickets. >= 96% Monthly

APS-002-S

thru

APS-056-S

$5K Semi- Annually

Table B-3, System Performance Metrics

Metric Title Service Standard Target Metric

Reporting Period

Deduction Schedule

Assessment Frequency

Return to Service (RTS)

Application & System Criticality 1, Incident Severity 1

Resolution or restoration of issues resulting in a loss of production accessibility, functionality, or availability for the following:

• Applications, systems, and subsystems categorized as Application Criticality 1

4 Hours > 16 Hours

> 7 Calendar Days

Pass/Fail Monthly $5,000 $7,500 $25,000

Per Event

Application & System Criticality 1, Incident Severity 2

Resolution or restoration of issues resulting in a loss

8 Hours > 32 Hours

> 7 Calendar Days

Pass/Fail Monthly $5,000 $7,500 $25,000

Per Event

Application & System Criticality 1 Incident Severity 3

Resolution or restoration of issues resulting in a loss

14 Calendar Days

> 14 Calendar Days

> 28 Calendar Days

Pass/Fail Monthly

$2,500

$5,000

$10,000

Per Event

Master Data Resolved other Master Data tickets.

Excludes Emergency and Exception Master Data tickets.

>= 98% Monthly

APS-002-S

thru

APS-056-S

$5K Semi- Annually

Metric Title Service Standard Target Metric

Reporting Period

Deduction Schedule

Assessment Frequency

Application & System Criticality 2 and 3 Incident Severity 1

Resolution or restoration of issues resulting in a loss categorized as Application Criticality 2 and 3

8 Business Hours

> 32 Hours

$1,000

$2,000

Application & System Criticality 2 and 3 Incident Severity 2

Resolution or restoration of issues resulting in a loss

16 Business Hours

> 64 Hours

Application & System Criticality 2 and 3 Incident Severity 3

Resolution or restoration of issues resulting in a loss

60 Calendar Days > 120

Calendar Days

Application Criticality 1

Availability of an individual application or system categorized as Application Criticality 1 in Appendix XXX, Application Inventory.

99.75% Monthly

99.25% - 99.75%:

$5,000

Monthly 98.25% - 99.2499%:

$10,000 < 98.2499%:

$15,000

Metric Title Service Standard Target Metric

Reporting Period

Deduction Schedule

Assessment Frequency

Application Criticality

2 & 3

Availability of an individual application or systems categorized as Application Criticality 2 in Appendix XXX, Application Inventory.

90% Monthly

80% - 90%:

$1,000 Monthly < 80%:

Table B-4, Reporting Only Metrics

Metric Title Service Standard Target Metric

Incremental Development Unit (IDU) Planning Accuracy

Number of User Story Points delivered in an IDU as a percent of the Number of Story Points planned in an IDU. This metric is calculated against the aggregates.

>= 90% Monthly

Outage Notification Notify SCs for each non-emergency outage of Urgent Applications no less than 1 week before outage.

This metric is calculated against the aggregates.

>= 95% Annually

Outage Notification Notify SCs for each non-emergency outage of Standard Applications no less than 72 hours before outage. This metric is calculated against the aggregates.

>= 95% Annually

Service Ticket Response Initial Service Ticket Response with the customer withing 4 business hours.

This metric is calculated against the aggregates.

= 100% Annually

Flow Velocity The measure of the flow velocity of an Agile Team >= 80% PI Event

Table B-4, Reporting Only Metrics

Metric Title Service Standard Target Metric

Flow Predictability The measure of the flow predictability of an Agile Team >= 80% and <= 100% PI Event

Flow Efficiency The measure of the flow efficiency of an Agile Team (>70% will be considered excellent) >= 50% PI Event

Team Health Radar

Team Health Radar (>= 70% with no more than 20% of items measured below 70%, will be considered excellent if >= 80% with no more than 10% of items measured below 80%)

>= 70% PI Event

Scrum Master Health Radar

Scrum Master Health Radar (>= 70% with no more than 20% of items measured below 70%, will be considered excellent if >= 80% with no more than 10% of items measured below 80%)

>= 70% PI Even

Product Owner Health Radar

Product Owner Health Radar (>= 70% with no more than 20% of items measured below 70%, will be considered excellent if >= 80% with no more than 10% of items measured below 80%)

>= 70% PI Event

Release Train Engineer Health Radar

Release Train Engineer Health Radar (>= 70% with no more than 20% of items measured below 70%, will be considered excellent if >= 80% with no more than 10% of items measured below 80%)

>= 70% PI Event

Table B-5, IT Security Compliance Metrics

Metric Title Description Target Metric

Reporting Period Deduction Schedule Assessment

Frequency

Annual IT Security Training

Successful completion of employee training in SATERN by the designated due date.

This metric is calculated against the aggregates.

100% Annually

If 100% is not reached, then $500 per delinquent employee per month

Annual

IT Security Actions

On-time completion of action items and SOC-MARs based upon the due date when issued (Note: approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (AR) count as closure).

Any day a system is in Mission Freeze does not count toward the timeline.

This metric is calculated against the aggregates.

>= 95% Monthly $25,000 Annual

Plan of Action and Milestones (POA&Ms)

On-time closure of Plan of Action & Milestone (POA&Ms), without extensions.

This metric is calculated against the aggregates.

>= 98% Monthly $25,000 Annual

Vulnerability Remediation

Critical Vulnerability Remediation within 15 calendar days of vulnerability identification.

This metric is calculated against the aggregates.

>=98% Monthly $10,000 Monthly

Vulnerability Remediation

High Vulnerability Remediation within 30 calendar days of vulnerability identification.

This metric is calculated against the aggregates.

>=98% Monthly $8,000 Monthly

Table B-5, IT Security Compliance Metrics

Metric Title Description Target Metric

Reporting Period Deduction Schedule Assessment

Frequency Vulnerability Remediation

Medium Vulnerability Remediation within 30 calendar days of vulnerability identification.

This metric is calculated against the aggregates.

>=95% Monthly $5,000 Monthly

Vulnerability Remediation

Low Vulnerability Remediation within 60 calendar days of vulnerability identification.

This metric is calculated against the aggregates.

>=95% Monthly $3,000 Monthly

Table B-6, Service Delivery Standards Metrics Guidelines

Delivery Timeframe

Government Approval Required

Metric Calculation Methodology Examples

Days

Yes Day count starts the next business day following Government approval.

A request is submitted on Wednesday, November 1, 2022. If the request is approved on Friday, November 3, 2022, and the metric is three business days, the target delivery date is close of business Wednesday, November 8, 2022.

No Day count starts the next business day following submission by the SC.

A request is submitted on Wednesday, November 1, 2022. If the metric is three business days, the target delivery date is close of business Monday, November 6, 2022.

Hours No Hour count starts upon submission by the SC.

A request is submitted on Wednesday, November 1, 2022, at 3:30 p.m. If the metric is four business hours, the target delivery time is 9:30 a.m. Thursday, November 2, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Table B-6, Service Delivery Standards Metrics Guidelines

Delivery Timeframe

Government Approval Required

Metric Calculation Methodology Examples

Same Day Yes

Requests approved by the TM at 2:00 p.m. or prior are to be completed by close of business the same day.

A request is approved on Wednesday, November 1, 2022 at 2:00 p.m. The request must be completed by close of business the same day.

Requests approved by the TM after 2:00 p.m. are considered next business day.

A request is approved on Wednesday, November 1, 2022 at 2:01 p.m. The request must be completed by close of business the next day.

Table B-7, System Performance Metrics Guidelines

Metric Type Standard Measured In

Metric Calculation Methodology Examples

RTS

Days Day count starts the day following loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 9:00 a.m. If the RTS metric is one business day, the target RTS is 5:30 p.m. Thursday, November 2, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Hours Starts immediately upon loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 9:00 a.m. If the RTS metric is four hours, the target RTS is 1:00 p.m. Wednesday, November 1, 2022.

Business Hours

Starts immediately upon loss of service.

A loss in service occurs on Wednesday, November 1, 2022, at 1:00 p.m. If the RTS metric is 8 business hours, the target RTS is 11:00 a.m. Wednesday, November 1, 2022 (assuming Standard Hours of 7:30 a.m. to 5:30 p.m.).

Availability N/A Starts immediately upon loss of service. N/A

Table B-8, Incident Severity Guidelines Incident Severity Definition

The Incident is an immediate and total loss of availability and/or functionality.

• Eg. Entire app/system not functioning. User can login but cannot perform anything. User cannot perform primary function of application. Host service for a web application stopped.

• - Total loss of Availability/functionality – Lack of access to any service provided under this Contract.

The Incident is a loss of significant/critical/Inoperable functionality or severe degradation of performance.

• - Eg. User can login but some critical function(s) do not work. The user can access the site collection home page of a SharePoint site collection, but not any of the other pages. A user can access a page and view the existing data, but not enter new information

• - Significant/Critical/Inoperable functionality – The loss of functionality would impede the application or system from completing primary functions of the applications. These incidents have no workaround available.

Portions of an application or system that cease to perform as designed or at all.

The Incident is a loss of secondary or non-critical functionality or minor degradation of performance.

All other issues that affect an application, system, or service under this contract from operating as designed.

• - Eg. Need information on the front page is not displayed but the information is still available on a separate page. Main search does not work but advance search does.

• - Secondary or non-Critical functionality – The customer has an acceptable workaround; the customer can still perform the function in an alternative way.

• - Minor degradation of performance – The performance of the application has been diminished but is still usable to the user but inconvenient.

• - Eg. Layout, color, or alt text do not function correctly. Bugs that do not stop the user from being able to perform a function.

• - All other issues include spelling errors, graphical mis-alignments that don’t impair functionality

Table B-9, Application Criticality Guidelines Application Criticality Support Hours

Incident Severity 1 Return to Service

Incident Severity 2 Return to Service

Incident Severity 3 Return to Service

1 24/7/365 4 hours 8 hours 14 calendar days 2 7:30 a.m. - 8:30 p.m. ET 8 business hours 16 business hours 60 calendar days 3 7:30 a.m. - 5:30 p.m. Local 8 business hours 16 business hours 60 calendar days i Reference Clause B.6, Deduction Schedule.

2.0 EXAMPLES OVERVIEW

The information below provides examples of how the Government will calculate the metrics defined in Tables B-2 through B-9.

As used in these metrics calculations, "aggregate" is defined as "taking all units as a whole." For example, Outage Notification, in which the "aggregate" is defined as the combined availability of all Outage Notifications during the assessment frequency (monthly).

All percentages will be rounded to the nearest 2 decimal places (e.g., 94.583% rounds to 94.58%).

Disputes regarding whether a specific site is affected by an incident downtime will be resolved by the Contracting Officer (CO).

Each Example has the following format:

• Service – Description of the metric.

• Assumption – Considerations taken into account when performing the calculations.

• Calculations – The formula/approach the Government will use when calculating the metric.

• Target Metric – Value of the target metric as defined above.

• Examples – Illustrations of the metric being calculated or, in the case of pass/fail metrics, a demonstration of when the metric is or is not met.

All tables in the examples should not be construed as representing true estimates.

Some calculations use an average hours/month (i.e. ‘H’) which is calculated as follows:

𝐻𝐻 =

Hours day ∗ 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑦𝑦𝑑𝑑𝑦𝑦 months year

24 ∗ 365

= 730

Equation 1

3.0 INFRASTRUCTURE EXAMPLES

3.1 Application Operation & Software Services

3.1.1 Outage Notification

Service:

Notify SCs for each non-emergency outage of all Applications. This metric is calculated against the aggregates.

Assumptions:

None

Calculation Directions:

The Monthly Outage Notification Metric is calculated in accordance with the following formula.

MONM = �1 −

𝑀𝑀𝑀𝑀

𝑂𝑂

� ∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 2

MONM = the Monthly Outage Notification Metric, MN = the number of Missed non-emergency outage Notifications for the month, O = the total number of non-emergency Outages for the month.

The annual outage notification metric is calculated using the following formula.

AONM = � (𝑀𝑀𝑂𝑂𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑆𝑆𝑆𝑆

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

, where

Equation 3

AONM = the Annual Outage Notification Metric, MONM = the Monthly Outage Notification Metric, n = the Government Fiscal Year months (i.e., October thru September) X = the number of months in a performance period (12 for a full Fiscal Year).

Target Metric: >= 95%

Example:

The following table outlines the number of non-emergency outages (O) taken in a given month and the number of times the contractor failed to give proper notification (MN).

Table B-9, Non-Emergency Outages Example

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept

Missed Notification

(MN)

1 3 2 0 3 2 1 2 1 0 0 0

Outages (O) 10 10 10 14 13 10 10 10 10 20 25 30

Monthly Outage

Notification Metric

(MONM)

90% 70% 80% 100% 76.92% 80% 90% 80% 90% 100% 100% 100%

Using Equation 3, where n = Oct thru Sept, MONM = the metrics in Table B− 9 and X = 12

AONM = � (𝑀𝑀𝑂𝑂𝑀𝑀𝑀𝑀𝑛𝑛)/X

𝑆𝑆𝑆𝑆𝑆𝑆

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

(90 +70 + 80 + 100 + 76.92 + 80 + 90 +80 + 90 +100 + 100 +100)/12 = = 88.08%

In this example, the Annual Outage Notification Metric of 88.08% fell below the 95% target.

3.1.2 SharePoint as a Service (SPaaS)

Service:

Delivery of each SharePoint upgrade in accordance with the requirements due date.

Assumptions:

The SharePoint as a Service Upgrade Delivery metric is based on the due dates specified in the Attachment A, Performance Work Statement (PWS), Section 6.1.4.B; therefore, the date is either met or it is not. SharePoint site collections SP-001, SP-002 and SP-004 will receive the discount.

Target Metric: Pass / Fail Example:

The SharePoint 2022 upgrade is delivered November 1, 2024. SharePoint 2022 was released October 1, 2022. PWS 6.1.4.B requires the update to be completed 2 years after major version released by the vendor, therefore the metric is failed. SharePoint site collections SP-001, SP-002 and SP-004, will each be assessed a 10% discount on the invoice for the month of October.

3.1.3 Application & System Criticality 1, Incident Severity 1 Return to Service

Resolution or restoration of issues resulting in a loss of production accessibility, functionality, or

A. Applications, systems, and subsystems categorized with an Application & System Criticality 1 that received an unplanned outage categorized with an Incident Severity of 1

Assumptions:

a. The RTS time will start for accessibility upon actual loss of service until service is restored regardless of reported incident timestamp.

b. The RTS time for functionality will start upon first reported incident timestamp.

c. Any application or subsystem in a multi-application platform or system will be considered individually.

d. Expiration of cryptographic certificates are included in RTS.

e. RTS System Performance Metrics are additive with other metrics.

f. Non-production environments provided to external parties under Hosted Only are considered production environments for metric purposes.

Target Metric: Pass / Fail

Example 1:

On July 6, 2024, TechDoc (an KSC-001-S Application & System Criticality 1) has experienced an issue resulting in loss of service starting at 7:00 a.m. The system is returned to service at 12:15 p.m. the same day, resulting in an RTS incident of 5.25 hours.

This incident is greater than 4 hours and will result in a deduction of $5,000 for catalog offering KSC-001-S for the month of July.

In this example, no deduction will be assessed against other catalog items.

Example 2:

On March 6, 2024, BOSS Maximo (an KSC-003-S Application & System Criticality 1) has experienced an issue resulting in loss of service starting at 9:00 a.m. The system is returned to service at 2:30 p.m. the same day, resulting in an RTS incident of 5.50 hours. This incident is greater than 4 hours and will result in a deduction of $5,000.

On March 7, 2024, SAP (an Application & System Criticality 1 system maintained under APS-003-S) has experienced an issue resulting in loss of service starting at 9:00 a.m. The system is returned to service at 5:30 p.m. the same day, resulting in an RTS incident of 8.50 hours. This incident is greater than 4 hours and will result in a deduction of $5,000

In this example, the month of March will be assessed a RTS deduction for both KSC-003-S and APS-003-S for a total of $10,000 for the month of March.

3.1.4 Application & System Criticality 1, Incident Severity 1 Availability

The availability of an individual IAS application or system categorized as an Application & System Criticality 1.

Assumptions:

a. The availability time will start for accessibility upon actual loss of service until service is restored regardless of reported incident timestamp.

b. The availability time for functionality will start upon first reported incident timestamp.

c. Any application or subsystem in a multi-application platform or system will be considered individually.

d. Expiration of cryptographic certificates are included in availability.

e. Availability System Performance Metrics are additive with other metrics.

f. Non-production environments provided to external parties under Hosted Only are considered production environments for metric purposes.

Calculation Directions:

ASCA = �1−𝐷𝐷

𝐻𝐻� ∗ 100, where

Equation 4

ASCA = Application & System Criticality Availability D = the sum of hours that an individual application or system has experienced RTS incidents in a month, including both accessibility and functionality.

H = average hours in a month (See Document Overview for calculation).

Target Metric: >=99.75%

Example 1:

On April 4, 2024, CAPI (an KSC-001-S 3.1.4 Application & System Criticality 1 system) experienced a 3.75-hour RTS incident. On April 6, 2024, CAPI experienced another RTS incident taking 2.00-hours, and a third 2 hr. incident on April 16, 2024. All three total 7.75 hours of RTS incidents in the month of April.

Using Equation 1 and Equation 4 where, D = 3.75+2+2 = 7.75, and H = 730.

ASCA = �1− 𝐷𝐷

𝐻𝐻� ∗ 100 =

�1 − 7.75

� ∗ 100 = 98.94%

In this example, the Application & System Criticality 1 Availability is below the target 99.75% and falls in the 98.25% - 99.2499% range and will result in a deduction of $10,000 for catalog offering KSC-001-S for the month of April.

Table B-10, Urgent Applications and Systems Availability Example

Urgent Applications and Systems Availability

Deduction Schedule System A Availability

99.25% - 99.75%: $5,000 98.25% - 99.2499%: $10,000 X

< 98.2499%: $15,000

Example 2:

In this example two applications experienced outages as follows.

On April 4, 2024, CAPI (an KSC-005 Application & System Criticality 1) experienced a 5.00-hour RTS incident. On April 6, 2018, CAPI experienced another RTS incident taking 3.00-hours, and on April 16, 2024, CAPI experienced a 1.00-hour incident. All combined CAPI experienced 9.00-hours of RTS incidents.

On April 10, 2024, TechDoc (also an KSC-005 Application & System Criticality 1) experienced a 14.00 hour RTS incident. On April 12, 2024, TechDoc experienced another RTS incident of

10.00 hours. On April 19, 2024 a third RTS incident occurred taking 6.00 hours. All combined, TechDoc experienced 30.00 hours of RTS incidents.

Using Equation 1 and Equation 4 where, For CAPI

D = 5+3+1 = 9, and

ASCA𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 = �1− 𝐷𝐷

�1 − 9 � ∗ 100 = 98.77%

For TechDoc D = 14.00+10.00+6.00 = 30.00, and

ASCA𝑇𝑇𝑆𝑆𝑂𝑂ℎ𝐷𝐷𝐷𝐷𝑂𝑂 = �1− 𝐷𝐷

�1 − 30.00

� ∗ 100 = 95.89%

In this example, the Application & System Criticality 1 Availability for CAPI is 98.77% and for TechDoc it is 95.89%. Both of these fall below the target metric of 99.75%. This will result in two deductions: one for CAPI and one for TechDoc, as shown in Table B-11.

Table B-11, Urgent Applications and Systems Availability Example

Urgent Applications and Systems Availability

Deduction Schedule System A Availability

99.25% - 99.75%: $5,000 98.25% - 99.2499%: $10,000 X

< 98.2499%: $15,000 X

In this example, the deductions are $10,000 for CAPI and $15,000 for TechDoc for a total of $25,000 for the month of April against catalog offering KSC-005.

3.1.5 Incremental Development Unit (IDU) Planning Accuracy

Service:

Number of User Story Points delivered in an IDU as a percent of the Number of Story Points planned in an IDU. This metric is calculated against the aggregates.

Assumptions:

None

IDUPA = �

𝐷𝐷𝐷𝐷𝐷𝐷

𝐷𝐷𝐷𝐷𝐷𝐷� ∗ 100, where

Equation 5

IDUPA = the IDU Planning Accuracy DUD = the sum of all story points delivered from all IDUs DUP = the number of story points planned for all IDUs

Target Metric: >= 90%

Example:

The following table shows the story points planned and delivered for several IDUs in a month.

Table B-12, Story Point Example IDU A IDU B IDU C IDU D IDU E Total

Delivered

(DUD) 22 18 60 35 27 162

Planned

(DUP) 20 20 60 30 30 160

Using Equation 5 and the values of Table B-12, where

DUD = 162 story points delivered DUP = 160 story points planned

IDUPA = �

𝐷𝐷𝐷𝐷𝐷𝐷

𝐷𝐷𝐷𝐷𝐷𝐷� ∗ 100 =

160� ∗ 100 = 101.25%

The IDU Planning Accuracy is greater than the 90% target. This metric is a reporting only metric.

4.0 IT SECURITY COMPLIANCE METRICS

4.1 Annual IT Security Training

Service:

Successful completion of employee training in SATERN by the designated due date. This metric is calculated against the aggregates.

Assumptions:

N/A

Calculation Directions:

AD = ( � DE𝑛𝑛) ∗ $500

𝑀𝑀

,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 6

AD = Annual Deduction, DE = Delinquent Employees in a M, M = the final month of the summation, n = the Government Fiscal Year months (e.g. October thru September)

Target Metric: 100%

Table B-13, Training Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep FY19 Training Incomplete Employees

N/A N/A 149 130 120 100 10 5 2 1 0 0

The FY24 training course was released by NASA on December 1, 2023 with a due date of April 30, 2024.

Metrics are calculated at the end of the month.

Table B-13 indicates the number of employees who have not completed the training December through March, and the number of delinquent employees April through September.

Using Equation 6 where, M = September, 𝐷𝐷𝐷𝐷𝐶𝐶𝑆𝑆𝐴𝐴 = 10, 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝑀𝑀 = 5, 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝑛𝑛 = 2, 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝐽𝐽 = 1, 𝐷𝐷𝐷𝐷𝑂𝑂𝑂𝑂𝑂𝑂,𝑁𝑁𝐷𝐷𝑁𝑁,𝐷𝐷𝑆𝑆𝑂𝑂,𝐽𝐽𝑀𝑀𝑛𝑛,𝐹𝐹𝑆𝑆𝐹𝐹,𝑀𝑀𝑀𝑀𝐴𝐴,𝐶𝐶𝐽𝐽𝐴𝐴,𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂 = 0

AD = ( � DE𝑛𝑛) ∗ $500

𝑀𝑀

�𝐷𝐷𝐷𝐷𝑂𝑂𝑂𝑂𝑂𝑂 + 𝐷𝐷𝐷𝐷𝑁𝑁𝐷𝐷𝑁𝑁 + 𝐷𝐷𝐷𝐷𝐷𝐷𝑆𝑆𝑂𝑂 + 𝐷𝐷𝐷𝐷𝐽𝐽𝑀𝑀𝑛𝑛 + 𝐷𝐷𝐷𝐷𝐹𝐹𝑆𝑆𝐹𝐹 + 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝐴𝐴 + 𝐷𝐷𝐷𝐷𝐶𝐶𝑆𝑆𝐴𝐴 + 𝐷𝐷𝐷𝐷𝑀𝑀𝑀𝑀𝑀𝑀 + 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝑛𝑛 + 𝐷𝐷𝐷𝐷𝐽𝐽𝐽𝐽𝐽𝐽 + 𝐷𝐷𝐷𝐷𝐶𝐶𝐽𝐽𝐴𝐴 + 𝐷𝐷𝐷𝐷𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂� ∗ $500 =

(0 + 0 + 0 + 0 + 0 + 0 + 10 + 5 + 2 + 1 + 0 + 0) ∗ $500 =

18 ∗ $500 = $9,000

Since 18 of the employees did not complete the required training by April 30, 2024, the contractor will be penalized $9,000.

4.2 IT Security Actions

Service:

On-time completion of action items based upon the due date when issued (Note: approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (AR) count as closure). This metric is calculated against the aggregates.

Assumptions:

a. Contractor initiated extensions even with NASA approval is not considered on time closure.

b. Approved exceptions (Plan of Action & Milestones (POA&Ms) and Accepted Risks (ARs) count as closure.

The metric will be computed using the following equation:

𝐼𝐼𝐼𝐼𝐼𝐼 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐶𝐶𝐶𝐶𝐼𝐼𝑛𝑛𝑀𝑀

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

∑ 𝐴𝐴𝑛𝑛𝑀𝑀

∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 7

CLT = Total number of actions closed on time during the FY up to the current month.

A = Total number of actions issued from beginning of FY to include those with due dates in the current month.

M = the final month of the summation.

Target Metric: >= 95%

Table B-14, Action Metric Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Total Completed On Time 5 4 4 2 6 4 5 5 4 3 1 3 46

Due 5 6 4 2 6 4 7 5 4 3 1 3 50

Using the data in Table B-14 and Equation 10 we get the following values for this example.

CLT = {5,4,4,2,6,4,5,5,4,3,1,3}

A = {5,6,4,2,6,4,7,5,4,3,1,3} n = September

𝐼𝐼𝐼𝐼𝐼𝐼 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑑𝑑 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐶𝐶𝐶𝐶𝐼𝐼𝑛𝑛𝑀𝑀

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

∑ 𝐴𝐴𝑛𝑛𝑀𝑀

∗ 100 =

∑ 𝐶𝐶𝐶𝐶𝐼𝐼𝑛𝑛

𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

∑ 𝐴𝐴𝑛𝑛

𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

(5 + 4 + 4 + 2 + 6 + 4 + 5 + 5 + 4 + 3 + 1 + 3) (5 + 6 + 4 + 2 + 6 + 4 + 7 + 5 + 4 + 3 + 1 + 3)

= 92.00%

In this example the metric was not met and the contractor will be accessed a $25,000 penalty.

4.3 Plan of Actions & Milestones (POA&M’s)

Service:

On-time closure of Plan of Action & Milestone (POA&Ms), without extensions. This metric is calculated against the aggregates.

Assumptions:

a. Contractor initiated extensions, even with NASA approval, is not considered on time closure.

b. Extension due to third party vendor constraints would require CO approval.

Calculation Directions:

𝐷𝐷𝑂𝑂𝐴𝐴&𝑀𝑀 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐷𝐷𝐶𝐶𝑂𝑂𝐼𝐼𝑛𝑛

𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

∑ (𝐷𝐷𝐷𝐷𝑛𝑛 − 𝐷𝐷𝐴𝐴𝐷𝐷𝑛𝑛)𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

∗ 100,𝑤𝑤ℎ𝑦𝑦𝑦𝑦𝑦𝑦

Equation 8

PCOT = Total number of all ITSSP POA&M’s closed on time during the FY up to the current month.

PD = Total number of all ITSSP POA&M’s issued from beginning of FY to include those with due dates in the current month PAE = POA&M’s with approved Contracting Officer (CO) extensions.

Target Metric: >= 98%

Table B-27 has representative values for a fiscal year.

Table B-15, Fiscal Year POA&M Example Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Total POA&M Closed on Time 10 10 7 8 3 8 7 10 4 7 5 10 89

POA&M Due 10 12 7 8 4 12 7 10 8 7 5 10 100 CO Approved Extensions 0 2 0 0 0 0 0 0 0 0 0 0 2

POA&M Due - CO Approved Extensions

10 10 7 8 4 12 7 10 8 7 5 10 98

For this example.

PCOT = {10,10,7,8,3,8,7,10,4,7,5,10}.

PD = {10,12,7,8,4,12,7,10,8,7,5,10}

PAE = {0,2,0,0,0,0,0,0,0,0,0,0}.

𝐷𝐷𝑂𝑂𝐴𝐴&𝑀𝑀 𝑀𝑀𝑦𝑦𝐴𝐴𝑦𝑦𝐴𝐴𝐴𝐴 =

∑ 𝐷𝐷𝐶𝐶𝑂𝑂𝐼𝐼𝑛𝑛

𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

𝑛𝑛=𝑂𝑂𝑂𝑂𝑂𝑂

∑ (𝐷𝐷𝐷𝐷𝑛𝑛 − 𝐷𝐷𝐴𝐴𝐷𝐷𝑛𝑛)𝑆𝑆𝑆𝑆𝑆𝑆𝑂𝑂

(10 + 10 + 7 + 8 + 3 + 8 + 7 + 10 + 4 + 7 + 5 + 10)

(10− 0) + (12− 2) + (7− 0) + (8− 0) + (4− 0) + (12− 0) + (7− 0) + (10− 0) + (8− 0) + (7− 0) + (5− 0) + (10− 0) ∗ 100 =

∗ 100 = 90.82%

For this example, the target metric was missed and the contractor will be accessed a $25,000 penalty at the end if the FY.