Exhibit_5c_TFITS_Performance_Measurement_Plan.pdf

Unclassified

PERFORMANCE MEASUREMENT PLAN (PMP)

TOTAL FORCE INFORMATION TECHNOLOGY

SYSTEMS (TFITS)

Version 2.0

November 2015

Prepared by:

MARINE CORPS SYSTEMS COMMAND (MCSC)

TFITS

Information Systems & Infrastructure (IS&I) Quantico, Virginia 22134

TFITS Performance Measurement Plan i

SIGNATURE PAGE

CONCURRENCE

Mr. Greg Kellenberger Lead Logistician, TFITS

PMM110 IS&I

MCSC

Mr. Robert Lovejoy Assistant Product Manager – Engineering, TFITS

APPROVED BY

Mr. Lou Peradotto Product Manager (PdM), TFITS ii

Record of Changes

Version # Date Figure, Table, or

Paragraph # *A, M, D Title of Brief Description Change Request #

1.0 12 Dec 2013

1.1 30 Apr 2014

2.0 15 May 2015 Signature block A Added signature block for TFITS lead logistician

2.0 15 May 2015 multiple M Format updates to align with other Engr docs

2.0 15 May 2015 Section 1.4 A Added section 1.4:

Glossary/Terms of

Reference

2.0 15 May 2015 Section 2 A Added Table Captions

2.0 15 May 2015 Table 1 M Added Priority column;

Clarified Language in

Reporting Details Column

2.0 15 May 2015 2.4.7, 2.4.8, 2.4.12 M Changed from priority 1 metric to priority 2 metric

2.0 15 May 2015 2.3.6, 2.6.9, 2.6.10,

2.6.11 M Changed to priority 1

metric

2.0 15 May 2015 Section 2.1 A Added metric definition paragraphs

2.0 15 May 2015 Section 2.1 M Moved System Interface Errors metric to Section

2.2

2.0 15 May 2015 Section 2 M Various updates to clarify and augment metric definitions

2.0 1 November 2015 Section 2.1 & 2.6 A

Updated to annotate that TPMs from this section shall be documented in project System Engineering Plan (SEP).

2.0 15 May 2015 Appendix A M Updated Appendix A to match changes to metric definitions/reporting requirements

2.0 1 October 2015 Section 2.5.3 A Updated IA Compliancy metrics to include Marine Corps Directives (MCDs)

A-Added, M-Modified, D-Deleted iii

Table of Contents Signature Page ........................................................................................................................................... i Record of Changes .................................................................................................................................. ii 1 Identification

1.1 General Scope

1.2 Purpose

1.3 Reference Documents

1.4 Glossary/Terms of Reference

2 Performance Metric Definitions

2.1 Application Specific Metrics

2.2 Standard System Profile

2.3 Service Operations Metrics

2.4 Engineering and Integration Metrics

2.5 Compliancy Metrics

2.6 System Performance Metrics

Appendix A Sample TFITS Performance Measurement Form ......................................................... A-1 iv

List of Tables Table 1 Glossary/Terms of Reference Table 2 Application Specific Metric Definitions Table 3 Standard System Profile Metrics Definitions Table 4 Service Operations Metrics Definitions Table 5 Engineering and Integration Metrics Definitions Table 6 Compliancy Metrics Definitions Table 7 Code Review Findings Report Format Table 8 System Performance Metrics Definitions

TFITS

PMP

1 Identification

1.1 General Scope

The guidelines contained within this document define Information Technology (IT) and Systems Engineering metrics standards set by the TFITS Product Management Office (PdMO) for its portfolio of projects and their supported IT systems. This guide provides all managed projects with a framework for reporting a set of core and specialized metrics to baseline system element performance and report a standardized set of performance measures to stakeholders.

1.2 Purpose

Performance measurement is a critical function supporting the system/software acquisition lifecycle and product management functions. To ensure a proper user experience, the PdMO using the Metrics Report Form in Appendix A will collect metrics outlined in this guide or future Government supplied Commercial off the Shelf/Government off the Shelf tools. These metrics help the project manager, system engineer and supporting contract team to analyze and quantify normal behavior of a system and evaluate the maturation process for each system. As new changes are introduced to each environment, having a properly defined and established baseline will help all stakeholders understand the impact of each change and help the evaluation process.

TFITS Performance Measurement is tracked and monitored by Tier II and Tier III projects and reported to Tier I Integrated Product Team leads for analysis. All processes and best practices for systems engineering functions are identified in section 1.3, Reference Documents. All TFITS performance measurement activities will remain compliant with these plans, to include proper configuration management and test and evaluation strategies for data collection and testing standards.

The measures in this PMP are intended to provide guidance regarding the minimum set of required measures providing the greatest value to TFITS Project Managers and Engineers. Tailoring is permitted and should consider three main objectives: 1) identification of the measures that apply to the project, 2) definitions of the measures and 3) collection and analysis mechanisms.

1.3 Reference Documents

• TFITS SEP

• TFITS Requirements Management Plan

• TFITS System Test and Evaluation Strategy

• TFITS Configuration Management Plan

1.4 Glossary/Terms of Reference

Table 1 Glossary/Terms of Reference Term Definition

Active User A user who has logged into the system.

Class I

Engineering Change Proposals

(ECPs)

A classification (Class) I ECP represents an engineering change that impacts system form, fit, or function. For example, a significant change to form, fit, or function driven by enhancement or multiple hardware and/or software upgrades

Class II ECPs

Class II ECPs are changes that do not affect form, fit, or function, interoperability, or interchangeability and provide minimal to no impact on any controlled documentation under configuration control. For example, corrections to code which do not affect form, fit, or function or a defect, or bug fix that does not constitute not a measurable change or enhancement ECP Engineering Change Proposal

Emergency Change A Change that must be introduced as soon as possible. For example, an Emergency Change may resolve a Major Incident or implement a Security patch.

Incident Request

An Incident is an unplanned interruption to an IT service or a reduction in the quality of an IT service. Failure of a Configuration Item that has not yet impacted a service is also an Incident. An Incident Request provides the details of an Incident and documents the lifecycle of a single Incident.

Inherent Availability (AI)

Availability of a system with respect only to operating time and corrective maintenance. AI ignores stand by and delay times associated with preventive maintenance. The goal for measuring AI is to document the system up time and exclude any downtime associated with the environment. For example, if the system was available for 700 hours over a 30-day period (30 days x 24 hours/day = 720 total hours) then AI would be 97.2% (700 hours/720 hours) regardless of whether the hosting environment was functioning or not.

Operational Availability (AO)

Operational availability is a measure of the availability that the customer actually experiences and includes all experienced sources of downtime. AO =

Uptime/Operating Cycle.

Security Technical Implementation Guide (STIG)

STIGs are the configuration standards for Department of Defense Information Assurance (IA) and IA-enable devices/systems. STIGs contain technical guidance to “lock down” information systems/software that might otherwise be vulnerable to a malicious computer attack.

Vulnerability Weakness in an information system, system security procedures, internal controls, or implementation that could be exploited or triggered by a threat source.

2 Performance Metric Definitions All performance metrics are provided a metric priority. This category of metric defines the Product Office’s priority for establishing this as a baseline metric within each project.

• Priority 1: Metric is essential. It provides greater value to analysts when determining overall health and maturity of an IT system.

Priority 2: Metric is vital but, may incur unplanned costs or additional resources that require proper planning and/or funding allocation.

Priority 3: Metric provides value however, additional costs are likely and must be planned and prioritized behind priority 1 and 2 metrics.

2.1 Application Specific Metrics

Application specific metrics are those Technical Performance Metrics (TPMs) pertinent to one, or a small subset of TFITS systems. These metrics will focus specifically on the performance elements of that environment which directly impact user experience and critical mission functions.

For this reason, all application specific metrics are considered priority 1 and essential for measuring overall system quality. Example TPMs may include: 1) Optical Digital Imaging- Records Management System web page load times for Temploan/Charge out Reports, 2) Total Force Structure Management Systems and Tables of Organization and Equipment Change Request processing time, or 3) time to run an Officer Longevity Report for Total Force Data Warehouse.

Due to the exhaustive list of TFITS applications and range of services provided and system objectives they fulfill, PdMO TFITS engineers will work with each project to define TPMs and document objective and threshold measures within the system’s project SEP.

Sample metrics common for system/application performance monitoring may include:

Table 2 Application Specific Metric Definitions

ID # Performance Metric

Reporting Frequency Reporting Details Metric

Priority

2.1.1 Search Query

Execution Time

Monthly/Pe r system Change

List

- Identify each critical search criteria query performed on the system

- Identify baseline version number, current version number

- Identify baseline query execution time

- Identify current version execution time

- Identify query environment

2.1.2 Key Web Page

Load Time

Monthly/Pe r System Change

List

- Identify each critical web page load time performed on the system

- Identify baseline version number, current version number

- Identify baseline web page load time

- Identify current version web page load time

- Identify page load environment

2.1.3 Workflow Execution

Times

Monthly/ Per System

Change

List

- Identify each critical workflow

- Identify baseline query execution time

- Identify baseline version number, new baseline version number

- Identify workflow environment.

2.1.4 User

Generated Reports

Monthly/ Per System

Change

List

- Critical system reports accessed by stakeholders

- Baseline time to run report

- Identify baseline version number, new baseline version number

- Identify report environment.

Search Query Execution TimeThis metric quantifies the execution time in 2.1.1 seconds to complete query searches in an identified environment. A comparison of results from the current and previous reporting periods, expressed as numeric and percentage deltas, is required. Performance deltas will be determined by the Project Manager (PjM) in coordination with the Systems Engineer.

Key Web Page Load Time 2.1.2 This metric quantifies the execution time in seconds to complete web page loads in an identified environment. A comparison of results from the current and previous reporting periods, expressed as numeric and percentage deltas, is required. Performance deltas will be determined by the PjM in coordination with the Systems Engineer.

Workflow Execution Times 2.1.3 This metric quantifies the execution time in seconds to complete standard workflows performed by users in an identified environment. A comparison of results from the current and previous reporting periods, expressed as numeric and percentage deltas, is required.

Performance deltas will be determined by the PjM in coordination with the Systems Engineer.

User Generated Reports 2.1.4 This metric quantifies the time in seconds to complete execution of standard reports in an identified environment. A comparison of results from the current and previous reporting periods, expressed as numeric and percentage deltas, is required. Performance deltas will be determined by the PjM in coordination with the Systems Engineer.

2.2 Standard System Profile

The standard system profile metrics characterize an IT system to help Product Management Office and associated System Integrators (SI) to understand and define system behavior.

Table 3 Standard System Profile Metrics Definitions

ID # Performance Metric

Reporting Frequency Reporting Details Metric

Priority

2.2.1 Total Number

of Users Monthly

List

- (Polling Rate) 1st day of month

- List Number of Users

2.2.2 Total Number

of Active Users

Monthly List

- (Polling Rate) 1st day of month List Number of Users

2.2.3 Max Number of Concurrent Connections

Monthly List

- Total number of connections

- Date & time of activity

2.2.4

Average Number of Concurrent Connections

Monthly

Trend Chart

- (Polling Rate) Continuous Monitoring

- (Duration) 1 Month

- (X axis) Day (Y axis) Number of Connections

2.2.5 System

Throughput (mb/s)

Monthly

Trend Chart

- (Polling Rate) Continuous Monitoring

- (Duration) 4 x 1 Week Charts

- (X axis) Day/Hour (Y axis) mb/s

2.2.6

Standard System

Maintenance Windows

Monthly

List

- Date(s)

- Time of Day

- Duration

2.2.7

System Interface

Errors (impacting web/report/ query load times)

Monthly/ Per System Change

List

- Identify all system interfaces connecting to 3rd. party systems (e.g. Joint Operation Planning and Execution System, Wide Area Work Flow)

- Identify number of connection errors

- Identify connection availability

Total Number of Users 2.2.1 This metric defines the total number of users who have account access to the system. For public web based systems that are currently hosted or will be hosted in the future by PM TFITS, total number of unique users accessing the system is acceptable.

Total Number of Active Users 2.2.2 This metric quantifies the total number of active users in the reporting period, where active is defined as a user who has logged into the system.

Max Number of Concurrent Connections 2.2.3 This metric defines the number of simultaneous, or concurrent, sessions or connections made to a system during a reporting period. Maximum number of connections should be labeled with a date and time stamp.

Average Number of Concurrent Connections 2.2.4 This metric defines the average number of sessions or connections expected to be open with the system at a given time.

System Throughput (mb/s) 2.2.5 This metric defines the network/system throughput based upon user activity and requests.

System Throughput is measured in mb/s and shall be plotted on a line chart that shows time of day and day of the week.

Standard System Maintenance Window(s) 2.2.6 All PdM TFITS projects shall report a standard maintenance window with time of day and duration of window. By reporting this information, stakeholder notifications and planning can occur that helps reduce impact of planned system downtime. The Standard Maintenance Window planned for the project should be identified. Actual Maintenance Windows should be compared to Planned in order to identify trends and to inform the PjM if the standard window duration is sufficient.

System Interface Errors 2.2.7 This metric identifies other systems with which the application interfaces, the number of connection errors, and the availability of the interface connection during the current reporting period.

2.3 Service Operations Metrics

Service Operations metrics show the volume of customer requests and interaction, as well as the efficiency in which support teams can accommodate those requests. Service operations metrics will provide key performance indicators that help Project Managers to determine the proper level of effort to support a system, which in turn will drive program cost estimates and future forecasting for capacity needs. Detailed service operations metrics can also provide key indicators for engineers and architects to identify systemic problems impacting stakeholder experience.

Table 4 Service Operations Metrics Definitions

Total Number of Incident Requests 2.3.1

This metric identifies the number of new stakeholder requests to the system support team for help/assistance in remediating incidents, problems, or system failures during the reporting period, including aggregated monthly data from prior reporting months. Requests shall be categorized by “Type” and “Severity”, where “Type” and “Severity” of request is defined by the Systems Engineer and Project Manager. (Example: Severity as Urgent, High, Medium, Low, etc.)

Total Number of Active/Open Incident Requests at End of Reporting Period 2.3.2

This metric identifies the number of open requests at the end of each period by: Type and Severity (defined in 2.3.1), age, and a brief description of the open service request.

Total Number of Incident Requests Closed 2.3.3 This metric identifies the total count of incident requests closed during the reporting period and aggregated monthly data from prior reporting months by: Type and severity (defined in

ID # Performance Metric

Reporting Frequency Reporting Details Metric

Priority

2.3.1 Total Number

of Incident Requests

Monthly Table and Bar Graph with trend line

- (Polling Rate) Last day of performance period

- Categorize by Tier

2.3.2

Total Number of Active/Open

Requests at End of

Reporting Period

Monthly

Table and Bar Graph with trend line

- (Polling Rate) Last day of performance period

- List request age

- List request description

- Categorize by Tier

2.3.3

Total Number of Incident Requests Closed

Monthly List

- (Polling Rate) Last day of performance period

- Categorize by Tier

2.3.4 Mean Time to

Resolve Incidents

Monthly List

- (Polling Rate) Last day of performance period

- Categorize by tier

2.3.5

Total Time Spent Per Incident Request

Monthly List

- (Polling Rate) Last day of performance period

- Categorize by tier

2.3.6

Total Number of ECPs

Generated from Incident

Request

Monthly

List

- (Polling Rate) Last day of performance period

- List ECP number

- List incident request number

Para 2.3.1). For incident requests closed during the period, age and a brief description of the closed service request will be included.

Mean Time to Resolve Incidents 2.3.4 This metric identifies the average amount of time it takes from opening incident requests to the time of closure.

Total Time Spent Per Incident Type 2.3.5 This metric identifies the total amount of time system support staff spent resolving all incidents within each project defined category. Note that it is acceptable to report total time spent by ‘Type’ category, it is not necessary to be specific for every request unless designated by the system PjM.

Total Number of ECPs Generated from Incident Request 2.3.6 This metric identifies the total number of incident requests that were properly correlated to system problems necessitating change, with corrective actions documented into an ECP.

2.4 Engineering and Integration Metrics

Performance metrics within the Engineering and Integration category are geared towards engineering process, planning and integration efficiency. Engineering performance metrics are tailored to program System Engineering Technical Reviews (SETR) events and seek to identify sustainability of the current baseline design.

Table 5 Engineering and Integration Metrics Definitions

ID # Performance Metric

Reporting Frequency Reporting Details Metric

Priority

2.4.1 Total Number

of New ECPs Monthly

Table

- (Polling Rate) Last day of reporting period

- List each unique ECP

2.4.2 Total Number of Open ECPs

in Development Monthly

Table

- (Polling Rate) Last day of reporting period

- List each open ECP

- List status of ECP

2.4.3 Total Number

of Closed ECPs Monthly

Table

- (Polling Rate) Last day of reporting period

- List each closed ECP

- List description of ECP

2.4.4

Total Number of ECPs rejected at Engineering

Review Board (ERB)/Configur ation Control Board (CCB)

Monthly

Table

- (Polling Rate) Last day of reporting period

- List rejected ECPs

- List reason for rejection

ID # Performance Metric

Reporting Frequency Reporting Details Metric

Priority

2.4.5 Total Number of Backed-Out

ECPs Monthly

Table

- List numbers of ECPs backed-out

- Provide description as to why back-out occurred

2.4.6

Total Numbers of Emergency/

Unplanned Changes

Monthly

Table

- List numbers of emergency changes

- Provide description of emergency change

- Date/time change occurred

2.4.7

Total Number of Failure

Reporting and Corrective

Action System

(FRACAS)

Reports

Per Failure

Table

- List of numbers of FRACAS reports

- Provide description of system failure Date/time failure occurred

2.4.8

Total Number of Government

Acceptance Test (GAT) Test Cases Executed

Per Release

Table

- Number of GAT test cases executed 2

2.4.9 Total Number of GAT Test Cases Failed

Per Release Table

- Number of failed test cases executed

- Description and test ID number

2.4.10 Total Number

of Defects Monthly

List

- Number of identified defects

- Defect description

2.4.11 Total Number

of Defects Closed

Monthly List

- Number of defects closed

- Defect ID & description

2.4.12 Total Number

of New Defects Introduced

Monthly Table

- Total number of new defects

- Description of new defect

- By Severity level

2.4.13

Total Number (Request for

Action ) RFAs / SETR Event

Per SETR Event

Table

- Number of RFAs categorized by SETR

- RFA description

- RFA open date

Total Number of New ECPs 2.4.1 This metric indicates the total number of new ECPs submitted during the reporting period.

These ECPs are categorized by severity/priority level and classified at the FRB/ERB/CCB in accordance with the TFITS Configuration Management Plan and in coordination with systems engineer guidance. If the ECP has not been classified through the appropriate review board, report ‘not-prioritized’.

Total Number of Open ECPs 2.4.2 This metric identifies the total number of open ECPs in development at the end of the reporting period. These ECPs will be categorized by severity level and classified in accordance with the TFITS Configuration Management Plan and in coordination with systems engineer guidance.

Total Number of Closed ECPs 2.4.3 This metric identifies the total number of closed ECPs during the reporting period. These ECPs will be categorized by severity/priority level and classified in accordance with the TFITS Configuration Management Plan and in coordination with systems engineer guidance.

Total Number of ECPs Rejected at Engineering Review Board 2.4.4 (ERB)/Configuration Control Board (CCB) This metric indicates the number of ECP changes that have been reviewed at ERB or CCB meeting conducted during the reporting period and were formally rejected.

Total Number of Backed-Out ECPs 2.4.5 This metric indicates the number of ECP changes executed or in process of execution and eventually backed-out or rolled-back due to technical issues or newly introduced defects. A description must be provided with each ECP.

Total Number of Emergency/Unplanned Changes 2.4.6 This metric indicates the number of changes to the system executed outside the scope of standard maintenance windows and results from customer problems and incidents.

Unplanned or emergency changes may still follow the ECP and SETR process; however these are driven by defects or failures that require unplanned cycles to remediate.

Total Number of FRACAS Reports 2.4.7 Failure Reporting and Corrective Action System (FRACAS) are generated upon system failures: system failures are identifies as 1) total outages and 2) failures of a service within the system that impact its ability to meet customer requirements. FRACAS reports are required to be provided upon each system outage and will detail the cause and duration of outage. This metric identifies the total number of FRACAS reports and provides a summary of each.

Total Number of Government Acceptance Test (GAT) Test Cases Executed 2.4.8 This metric identifies the total number of GAT tests executed during the reporting period.

Total Number of GAT Failures 2.4.9 This metric identifies the total number of failures experienced during GAT test execution.

Total Number of GAT Defects 2.4.10 This metric indicates the total number of known defects within application code or in network/hardware resources. Each defect should provide a brief synopsis of the defect and correlate to an existing ECP or release in which it will be corrected.

Total Number of GAT Defects Closed 2.4.11 This metric identifies the total number of defects closed in the reporting period. A brief synopsis of each defect should be provided along with the ECP number closing that defect.

Total Number of New Defects Introduced 2.4.12 This metric indicates the number of defects introduced after a release to production or identified by users/operators during the reporting period. Each new defect should provide a brief synopsis of the defect, change or ECP that introduced the defect, and whether it is scheduled in a future release for remediation.

Total Number of RFAs/SETR Event 2.4.13 The Request for Action (RFA) is a method to document SETR issues and identify steps for resolution. This metric indicates the number of critical RFAs and RFA descriptions categorized by SETR event.

2.5 Compliancy Metrics

System compliancy metrics demonstrate compliancy with imposed federal mandates and program requirements, to comply with Federal Information Security Modernization Act (FISMA) 2014 and Clinger Cohen Act of 1996. At any time, analyzing compliancy metrics can provide insight to the overall system security posture, as well as overall sustainability metrics useful in measuring personnel performance and aptitude.

Table 6 Compliancy Metrics Definitions

ID # Performance Metric Reporting Frequency Reporting Details Metric Priority

2.5.1 Number of open STIG findings

(annotated by individual STIGs/per server).

Monthly Table

- (Polling Rate) Monthly

2.5.2

Information Security Vulnerability Management (IAVM) Compliancy

-Relevant IAVMs new in the month

-Relevant IAVMs closed in the month

-Relevant IAVMs POAM’d (include IAVM number)

Monthly Table

- (Polling Rate) Monthly

2.5.3

MCDs

-Number of MCDs received new in the month

-Number of MCDs closed in the month

- Number of MCDs POAM’d

Monthly Table 2

2.5.4 Total Number of All

Open IA Vulnerabilities being carried on POAM

Monthly Table

- Categorized by Severity Level (CAT I, II, III, IV)

2.5.5 Total Number of IA

Vulnerabilities Remediated

Monthly Table

- Categorized by Severity Level (CAT I, II, III, IV)

2.5.6 Code Review(s) Quarterly or

As Performed

List

- Number of code reviews performed

- Total number of code review findings by severity level

- Number of code security findings remediated during reporting period

Number of Open STIG Findings 2.5.1 This metric identifies the security vulnerabilities from a STIG perspective. The Defense Information Systems Agency (DISA) Information Assurance Support Environment (IASE) provides methodology to support teams to assist in the collection of this information (http://iase.disa.mil/ ).

For each applicable STIG, include the total number of checks (line items), number of STIGS that were tested and compliant (closed), number of non-compliant (open);

categorized by severity level (1, 2, 3) for each device (server, F5, etc.) and STIG (i.e., Application and development STIG). If a new or updated STIG is released in the month, provide the deltas and annotate the number of additional or updated checks (line items) that were assessed and either closed or remain open and were added to the system POAM. (The STIG findings that have changed can be obtained by checking the release notes in the updated STIGs.)

The STIG statistics shall include total system to include the application, database and servers. The findings will be enumerated not unique findings (for example: if there is one STIGs finding on 12 servers then we have 12 open STIGs).

IAVM Compliancy 2.5.2 This metric identifies the open security vulnerabilities from an IAVM perspective. IAVMs consist of Information Security Vulnerability Alerts (IAVAs) and Information Assurance Vulnerability Bulletins. This information can be obtained at the US Cybercom website (https://iavm.csd.disa.mil) and will be confirmed by scans.

The IAVM statistics will include total system to include the application, database and servers. The findings will be enumerated individually and not reported by unique findings (for example: if there is one IAVM finding on 12 servers then we have 12 open IAVAs).

The open IAVMs will include:

- Applicable IAVMs new in the month

- Applicable IAVMs closed in the month

- Applicable IAVMs that were unable to be closed and were added to the POAM (include IAVM number) Note: if your system (OMS) is patched via FEN instead of IAVM, please report FEN status here.

MCDs 2.5.3 MCDs are usually vulnerabilities identified by the Marine Corps Network Operation and Security Command that are relevant to a specific system. If a system is notified of an open MCD on the system the vulnerability will be tracked by:

- Number of new MCDs received in the month,

- Number of MCDs closed in the month and

- Number of MCDs that were unable to be closed and were added to the POAM (include MCD number).

Total Number of Open IA Vulnerabilities being carried on POAM 2.5.4 This metric identifies the total number of security vulnerabilities present within the application system at the end of the reporting period. Total vulnerabilities should be reported by total number in each category severity level (CAT).

IA controls are tested at the IV&V and are part of the overall health of a system along with the STIGs, IAVA, and Fortify. There can be open items that need to be addressed per the POAM and Authority to Operate letter. These can be New, Open, or Closed. This is the total from all sources (STIGS, IAVA, CODE, IA controls.)

Total Number of IA Vulnerabilities Remediated 2.5.5 This metric identifies the number of IA vulnerabilities remediated during the reporting period so that program management and IA personnel can evaluate the total progress and efficiency with which engineering and administrators can identify, assess, test, implement, and close vulnerabilities.

Code Reviews 2.5.6 This metric identifies the summarized results of Code Reviews. Data from applicable code review tools (Hewlett Packard Fortify and others) should be used to present findings by severity level and category. Please indicate if any findings were remediated or any new findings were introduced during the reporting period.

Table 7 Code Review Findings Report Format Number of Prior

Findings Findings Removed

Findings Introduced

CRITICAL: Exploitable 0 CRITICAL: Bad Practice 0 CRITICAL: Stability Issue 0 CRITICAL: Not an Issue 0 CRITICAL: Suspicious 0 CRITICAL: None* 0 High: Exploitable 0 High: Bad Practice 0 High: Stability Issue 0 High: Not an Issue 0 High: Suspicious 0 High: None* 0 Medium: Exploitable 0 Medium: Bad Practice 0 Medium: Stability Issue 0 Medium: Not an Issue 0 Medium: Suspicious 0 Medium: None* 0 Low: Exploitable 0 Low: Bad Practice 0 Low: Stability Issue 0 Low: Not an Issue 0 Low: Suspicious 0 Low: None* 0

TOTAL 0

TOTAL Exploitable: 0

2.6 System Performance Metrics

System performance metrics form the basis for TPMs. These metrics directly correlate to how back-end server and software code are interacting with all elements to produce an expected user experience. To conduct true system performance measurement, all system services must undergo decomposition down to system elements, where traceability can be established for root cause analysis on partial outages or slow performance to a specific web query or request. In addressing the Priority 1 metrics in this category, consideration and accommodation should be given to systems that are hosted in environments where availability of this data is not readily available to the SI or Government resources. The intent is to provide data that provides insight into current and historical system performance in order to support trending and availability analysis.

As mentioned in Section 2.1, TPMs shall be identified in project SEPs to address objective and threshold measures as identified and agreed upon with the system user community.

Table 8 System Performance Metrics Definitions

ID # Performance Metric

Reporting Frequency

Reporting Details Metric Priority

2.6.1 Planned System

Outage Monthly

List

- (Polling Rate) Continuous Monitoring

- Number of planned outages

- Duration per outage (minutes)

2.6.2 Unplanned System

Outage Monthly

List

- (Polling Rate) Continuous monitoring

- Number of unplanned outages

- Duration per outage (minutes)

2.6.3 Downtime During

Peak Hours Monthly

List

- (Polling Rate) Continuous monitoring during peak performance times

2.6.4 System Element

Availability Monthly

List

- (Polling Rate) Continuous Monitoring

- Database Tier Availability

- Web Tier Availability

- Application Tier Availability

- Network Availability

- Storage/SAN Availability

2.6.5 Mean Time to

Restore Per Incident

List

- List each failure

- List time to repair each failure

- Average time to restore all outages

ID # Performance Metric

Reporting Frequency

Reporting Details Metric Priority

2.6.6 Backup &

Restoration Success Rate

Monthly

Table

- (Polling Rate) Verify after each planned backup or restore operation

- Total number of backup attempts

- Total number of backup failures

- Total number of restore attempts

- Total number of restore failures

2.6.7 Disk Storage

Utilization Monthly

Table and Trend Line (Polling Rate) Last day of reporting period

- Categorize storage space Available by drive/volume

- Categorize storage space Consumed by drive/volume

- Identify total storage available on Hardware storage device(s) (unallocated storage) Aggregate monthly data from prior months

2.6.8 Memory

Utilization Monthly

Table

- (Polling Rate) Continuous monitoring

- List average utilization by system server

2.6.9 Central Processing

Unit (CPU) Utilization

Monthly

Table

- (Polling Rate) Continuous monitoring

- List average utilization by system server

Planned System Outages 2.6.1 This metric indicates the number of planned system outages during a reporting period. A planned outage is a coordinated event with the Program Office and stakeholders where the user community of the system is provided ample notification that the system will not be available for use. Planned outages may consist of (but not limited to) preventative maintenance, logistical delays, or system upgrades. Each planned outage should be recorded and the total downtime during that planned outage provided to assist in calculating system inherent and operational availability.

Unplanned System Outages 2.6.2 This metrics indicates the number of unplanned system outages during a reporting period.

Unplanned outages are often caused by a system failure that require corrective maintenance.

Each unplanned outage should be recorded and the total downtime during that unplanned outage provided to assist in calculating system inherent and operational availability.

Downtime During Peak Hours 2.6.3 This metric indicates the total downtime experienced during peak system usage and performance hours. Peak hours will vary by system and should be determined through analysis of historical experiential data for each system.

System Element Availability 2.6.4 This metric indicates subsystem element operational availability of the application system.

Categories include 1) Database Availability, 2) Web Tier Availability, 3) Application Tier Availability, 4) Network Availability, and 5) Storage/SAN availability.

Mean Time to Restore (MTTR) 2.6.5 This metric indicates the average time it takes to restore the application system from failure. MTTR only factors system failures and does not factor incidents or problems that do not cause a failure.

Backup and Restoration Success Rate 2.6.6 This metric indicates the total number of backups taken during the reporting period, along with the success rates of those backups. This metric should also indicate the total number of restore procedures conducted during the period and the success rate of those restore procedures.

Disk Storage Utilization 2.6.7 Disk storage utilization is an important metric for capacity planning. This metric will indicate (in tabular form): 1) Raw storage available, 2) Storage allocated (by volume), and 3) Storage consumed (by volume).

Memory Utilization 2.6.8 Memory utilization is an important metric for capacity planning. This information will be provided in a table for each system server and should show 1) observed spikes in memory utilization with date/time a n d 2) average memory usage on the server.

CPU Utilization 2.6.9 CPU utilization is an important metric for capacity planning. This information will be provided in a table for each system server and should show 1) observed spikes in CPU utilization and 2) average CPU utilization per server

A-1

Appendix A Sample TFITS Performance Measurement Form Appendix A contains a populated TFITS Performance Measurement form. This form has been populated with representative data to demonstrate the proper method of data collection and reporting. Note that this is a sample and may differ depending upon the specific TFITS project based upon critical metrics of the system.

PdM TFITS Monthly Performance Measurement Reporting Form (SAMPLE)

Project: TRAP Reporting Date: March 2015

1.0 Application/System Specific Metrics

Baseline System Version: 4.3.5.1

Please note that these are sample metrics not properly mapped to a specific TFITS system. These are reflective of application functions throughout the portfolio that will be recorded and monitored.

Current System Version: 4.3.5.3

Search Query Execution Time

Search Query Baseline Search Query Time (sec)

Current Search Query Time (sec) Delta

General Officer Search 14.9 15.5 0.6

104%

Senior Enlisted Search 21.4 20.4 -1

95%

Civilian Personnel Search 29.1 31.5

2.4 108%

Web Page Load Times

Web Page Baseline Load Time (sec)

Current Load Time (sec) Delta

Commanders' Dashboard 4.3 9.3

216%

Finance Reporting Dashboard 9.2 19.5

10.3 212%

Joint Force Planning Page 5.5 5.1

-0.4 93%

General's Blog Website 3.4 2.4 -1

71%

A-2

Workflow Execution

Workflow Name Baseline Workflow Execution Time (sec)

Current Workflow Execution Time (sec) Delta

Submit New User Request 8.4 10.8

2.4 129%

Create New Report 31.5 35.5

113%

Notify All Users Alert 74.8 246.7 171.9 330%

Create New Calendar Entry 5.8 6.5

0.7 112%

User Generated Reports

Report Name Baseline Report Generation Time (sec)

Baseline Report Generation Time (sec) Delta

General Officer Report 8.2 10.8 2.6

132%

Enlisted Report 23.6 20.1 -3.5 85%

Civilian Personnel Report 15.2 45.5

30.3 299%

A-3

2.0 System Profile Information

Last Month Current Month Total Users 1844 1906 Active Users 1732 1092

User Role (if Applicable) Total/Active Last Month Current Month

Role 1 Total 29 32 Active 20 20

Role 2 Total 47 47 Active 46 15

Role 3 Total 66 68 Active 61 42

Role 4 Total 80 85 Active 74 42

Role 5 Total 85 85 Active 64 47

Role 6 Total 292 307 Active 284 228

Role 7 Total 1245 1282 Active 1183 698

A-4

Last Month Current Month

Max Concurrent Connections 875 823

2/22/2015 3/16/2015 Average Concurrent Connections 613 689

A-5

Standard System Maintenance Window Third Saturday-Sunday of Month, 2100-0500

3.0 Service Operations Metrics

Request Type

New This Month

Closed This Month

Mean Time To Resolve

Total Open Total Closed

Type X 18 14 20 hours 18 46 Type Y 20 17 13 hours 43 55 Type Z 21 22 17 hours 32 82

A-6

Active Incidents (beyond X hours/days) (repeated for each incident type)

Type Incident # Description Status Age

Type X

INC1555 INC1555

Description INC1555 Status INC1555 Age

INC1478 INC1478

Description INC1478 Status INC1478 Age

INC1592 INC1592

Description INC1592 Status INC1592 Age

INC1769 INC1769

Description INC1769 Status INC1769 Age

INC1631 INC1631

Description INC1631 Status INC1631 Age

INC1388 INC1388

Description INC1388 Status INC1388 Age

INC1925 INC1925

Description INC1925 Status INC1925 Age

INC1265 INC1265

Description INC1265 Status INC1265 Age

INC1185 INC1185

Description INC1185 Status INC1185 Age

A-7

INC1865 INC1865

Description INC1865 Status INC1865 Age

INC1688 INC1688

Description INC1688 Status INC1688 Age

INC2000 INC2000

Description INC2000 Status INC2000 Age

INC1666 INC1666

Description INC1666 Status INC1666 Age

INC1085 INC1085

Description INC1085 Status INC1085 Age

INC1801 INC1801

Description INC1801 Status INC1801 Age

INC1313 INC1313

Description INC1313 Status INC1313 Age

INC1671 INC1671

Description INC1671 Status INC1671 Age

INC1627 INC1627

Description INC1627 Status INC1627 Age

A-8

4.0 Engineering and Integration Metrics

ECP

Classification

New ECPs this Month

ECPs Closed

This Month

Total Open ECPs

Total Closed ECPs

ECPs Rejected at ERB/CCB

Backed out ECPs

Class 1 0 3 4 11 2 1 Class 2 3 1 8 9 0 0

Open ECPs Class ECP# ECP Description ECP Status

Class 1

SYSNAME

SYSNAME13

Description SYSNAME13 Status

SYSNAME

SYSNAME9

Description SYSNAME9 Status

A-9

SYSNAME

SYSNAME31

Description SYSNAME31 Status

SYSNAME

SYSNAME13

Description SYSNAME13 Status

Class 2

SYSNAME

SYSNAME3

Description SYSNAME3 Status

SYSNAME

SYSNAME20

Description SYSNAME20 Status

SYSNAME

SYSNAME19

Description SYSNAME19 Status

SYSNAME

SYSNAME22

Description SYSNAME22 Status

SYSNAME

SYSNAME34

Description SYSNAME34 Status

SYSNAME

SYSNAME12

Description SYSNAME12 Status

SYSNAME

SYSNAME17

Description SYSNAME17 Status

SYSNAME

SYSNAME26

Description SYSNAME26 Status

ECPs Rejected at ERB/CCB Class ECP# ECP Description Reason for Rejection

Class 1

SYSNAME

SYSNAME35

Description SYSNAME35 Rejection Reason

SYSNAME

SYSNAME15

Description SYSNAME15 Rejection Reason

Class 2 Backed Out ECPs

Class ECP# ECP Description Reason for Backing Out ECP

Class 1 SYSNAME

SYSNAME32

Description SYSNAME32 Back out Reason

Class 2 None None None Emergency/Unplanned Changes FRACAS Reports

Emergency/Unplanned Change Description 1 FRACAS ID FRACAS Summary

Emergency/Unplanned Change Description 2 FRACAS 2015_03_14

System search service failed;

search index corrupted, rebuilt index by running new scan

A-10

GAT Results Total # GAT

Tests Executed Total # GAT Test Failures

Total # GAT Defects

Total # GAT Defects Closed

Total # Defects Introduced

1 (v4.3.5.3) None Identified None Identified N/A None

A-11

5.0 Compliancy Metrics

Percentage of STIG Compliance 86% Percentage of IAVM Compliants 94%

Server Category Number of Vulnerabilities Open

Number of Vulnerabilities

Remediated

WEBSERV001

CAT I 1 1

CAT II 8 3

CAT III 13 4

CAT IV 0 0

APPSERV001

CAT I 3 2

CAT II 9 1

CAT III 5 1

CAT IV 0 0

DBSERV001

CAT I 2 0

CAT II 5 1

CAT III 7 0

CAT IV 0 0

DBSERV002

CAT I 1 0

CAT II 5 1

CAT III 11 0

CAT IV 0 0

TOTAL

CAT I 7 3

CAT II 27 6

CAT III 36 5

CAT IV 0 0

A-12

6.0 System Performance Metrics

Inherent Availability 100% Operational Availability 99.93% Peak Hours Downtime 0 minutes

Database Tier Availability 99.93% Web Tier Availability 100%

Application Tier Availability 100% Mean Time to Restore FRACAS 2015_03_14, 30 minutes

Backup and Restore Success Rate Number of Backups Taken 30

Backup Failures 1 Backup Success Rate 96.67%

Disk Storage Utilization

Storage Available (Unallocated) 4.70 TB Raw (Estimated) 2.70

TB @ RAID 5

Server Storage

Allocated (GB) Storage Used

(GB) Utilization Rate

WEBSERV001 C:\ 200 88 44.00%

D:\ 350 275 78.57%

APPSERV001 C:\ 200 89 44.50%

D:\ 350 260 74.29%

DBSERV001 C:\ 200 87 43.50%

D:\ 350 348 99.43%

DBSERV002

C:\ 200 92 46.00% D:\ 350 292 83.43%

Total Storage 2200 1531 69.59%

A-13

CPU & Memory Utilization

Server Avg. CPU Utilization

Observed

CPU

Spikes

Average Memory

Utilization

Observed Memory Spikes

WEBSERV001 42.00% 73.00% 12.00% 39.00%

APPSERV001 9.00% 31.00% 14.00% 34.00%

DBSERV001 55.00% 89.00% 43.00% 74.00%

DBSERV002 49.00% 80.00% 59.00% 91.00%