Attachment_7_-_QASP_-_Task_Order_1_Final_Draft.docx

QUALITY ASSURANCE SURVEILLANCE PLAN

FOR

DTRA Agency-Wide Information Technology Support Services Task Order 1; IT Service Integration, Provision and Operations

RFP HDTRA1-18-R-0002

1. Purpose of Contract: The contractor shall provide IT service integration, provision, transition, and operations to ensure DTRA and its partners have reliable, secure, and efficient IT capabilities and services necessary to perform the critical DTRA mission. In the performance of this Task Order (TO), the contractor(s) shall provide support services necessary to meet the DTRA Information Integration and Technology Services Directorate (I2TS)/Chief Information Officer’s (CIO) strategic goals consistent with DoD policies and industry best practices. The effective date of the contract is from (insert date) for a base period of five years and one (1) five-year renewable option, thus possibly extending the period of performance to (insert date).

2. Review Responsibilities: TBD will be designated as the Contracting Officer's Representative (COR) for this contract. As COR, TBD is responsible for interfacing with the contractor in the performance of duties such as those set forth below.

3. Methods and Frequency:

3.1 Surveillance will be accomplished in several methods: random basis; weekly, monthly, quarterly and bi-annual reviews. The method used for evaluation will be based upon document reviews, reports, observation, and received customer feedback.

3.2 Timely feedback to the contractor on unacceptable performance will be provided in writing by the COR.

3.3 A summary of any performances that did not meet the terms and conditions of the contract shall be prepared by the COR and forwarded to the Contracting Officer (KO).

3.4 The surveillance report and a summary of the surveillance logs will be prepared by the COR and submitted to the KO annually. The COR may submit out-of-cycle reports to the KO to support claims of unacceptable performance or when there is a significant change in performance that alters the assessment in one or more evaluation areas.

4. Performance Areas:

The following performance areas will be evaluated as described. However, they are not all inclusive, and any service on this contract may be evaluated at any time; all services provided on this contract are subject to inspection and evaluation at any time. Specific performance areas, standards, metrics, acceptable quality levels, and frequency of assessments are shown in the attached QASP Table.

4.1 The contractor will be evaluated on the effectiveness of their task order management. Specifically, the government will evaluate the timeliness of staff on-boarding, timeliness of deliverables, and consistently meeting performance measures in other performance measures within this task order.

4.2 The contractor will be evaluated on their support of IT operations and systems administration as defined in the PWS. Specifically, the government will monitor and evaluate network availability, efficiency, and security.

4.3 The contractor will be evaluated on their ability to effectively support a service desk operation. Specifically, the government will monitor and evaluate the contractor’s performance based on effectiveness, responsiveness, and overall customer satisfaction.

4.4 The contractor will be evaluated on the consistency with which IT applications and assets (software or hardware components of the network) are upgraded, patched, or replaced before they exceed their usefulness and serviceability and impact mission availability or effectiveness. Specifically, the government will monitor and evaluate the contractor’s performance based on consistency and accuracy of applications and asset management reporting.

4.5 The contractor will be evaluated on the consistency with which they support the planning for and integration of cloud solutions and capabilities that become part of the DTRA IT Enterprise. Specifically, the government will monitor and evaluate the seamless integration of and transition to full usage of cloud services and capabilities.

4.6 The contractor will be evaluated on their support to new service integration by performing transition analysis and supporting change management activities associated with the transition and integration of new capabilities into the DTRA IT Enterprise. Specifically, the government will monitor and evaluate the thoroughness of the transition analysis to help minimize realized transition impacts and the effectiveness of change management efforts and the speed of adoption of new systems.

4.7 The contractor will be evaluated on the consistency of support to systems functional and security validation prior to deployment of new capabilities and systems onto the DTRA IT Enterprise. Specifically, the government will monitor and evaluate the on consistent application of validation requirements for newly introduced services or capabilities.

4.8 The contractor will be evaluated on their ability to effectively transition into full performance following task order award. Specifically, the government will monitor and evaluate the on-boarding of support staff per the transition-in plan included in the contractor proposal.

5. Evaluation Criteria:

· Exceptional: Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

· Very Good: Performance meets contractual requirements and exceeds some to the Government’s benefits. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective action taken by contractor was effective.

· Satisfactory: Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taking by the contractor appear or were satisfactory.

· Marginal: Performance does not meet some contractual requirements. The contractor’s performance of the element or sub-element being evaluated reflects a serious problem for which the contractor has not yet identified corrective actions. The contractor’s proposed actions appear only marginally effective or were not fully implemented.

· Unsatisfactory: Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which the contractor’s corrective actions appear or were ineffective.

6. Surveillance Documents: Documentation used and referenced to perform surveillances will consist of Reports, the Performance Work Statement, the Quality Control Plan, Customer Feedback, and the Surveillance Delivery Summary/Performance Surveillance Checklist. Documentation shall include the date, time, and location of the surveillance; applicable remarks to the surveillance; and signature of the COR. Unsatisfactory ratings must be signed by the contractor.

7. Rating: All contractor tasks will be rated as Exceptional, Satisfactory, Marginal, or Unsatisfactory in the Surveillance Delivery Summary/Performance Surveillance Checklist. At the end of the contract or contract order, the KO will collect the Surveillance Delivery Summary/Performance Surveillance Checklist and any supporting documentation and provide an overall rating for each of the performance areas using the evaluation criteria in Paragraph 5 of this plan.

8. Recording Observations: The Surveillance Delivery Summary/Performance Surveillance Checklist will be used to document observations and problems and rate the individual contractor tasks. The Performance Assessment Report (PAR) Form will be used to record each observed performance issue noted, for performance areas identified in Paragraph 4 or any other tasks identified in the PWS. The Surveillance Checklists and PARs become the formal record for reference. The contractor is told each time a performance issue is found during the observations.

9. Potential Unacceptable Performance: The COR must identify the specific service that is unacceptable and the possible causes and ask a number of questions that will likely identify the source of the problem. The COR shall attempt to resolve the problem with the contractor.

9.1 The COR shall forward the original Surveillance Delivery Summary/Performance Surveillance Checklist and the PAR documenting the observation and attempted solution (along with an evaluation of the results) to the KO, when a specific service becomes unsatisfactory during a surveillance period. The surveillance reports will be used to provide an overall rating for the performance areas identified in Paragraph 4 of this plan.

9.2 The contractor shall complete a response to the unacceptable evaluation if requested by the KO or by its own choice.

10. Taking Action:

10.1 The COR shall evaluate the contractor's performance and document any non- compliance, but only the KO may take action against the contractor for an unacceptable rating.

10.2 When the contractor's performance is unacceptable and a formal action is indicated, the COR and KO will determine what action is appropriate for the specific circumstances. The KO will determine the final action.

11. Records: During the performance of the contract, the COR will retain a copy of all Surveillance Delivery Summary/Performance Surveillance Checklists and Performance Assessment Report (PAR) Forms, with performance issues noted. The Surveillance Delivery Summary/Performance Surveillance Checklist and PARs become the formal record for reference. At the end of the surveillance period, the COR will forward all inspection records to the KO for inclusion in the contract file.

QASP Table

After the initial transition period, the contract shall ensure staffing is available as required to meet performance requirements, and unplanned staff vacancies are held to a minimum.

Staff Transition Metric = # of planned staff positions fully on-boarded at DTRA / # of staff positions scheduled to be onboarded at end of performance period.

Average Staff Availability Metric (Key Personnel) = (Total hours accumulated within period for Key Personnel) / (# of key personnel positions X available workhours in period) Average Staff Availability Metric (Non-Key Personnel) = (Total hours accumulated within period for Non-Key Personnel) / (# of key personnel positions X available workhours in period) 1st Month AQL > 85% 2nd Month AQL > 95% 3rd Month AQL > 98%

Metric begins after transition period AQL for Key Personnel > 95%

AQL for Non-Key Personnel > 90% Data will be collected from Monthly Status Report and compared to Staffing Plan in the Proposal. Metrics will be as of the cut-off date of the MSR. Metric can be validated using DTRA security records.

Data will be collected from MRS and will be as of the cut-off date of the MSR for each period.

NOTE: on-time deliverable assumes the item is accepted as compliant by the government reviewer. A rejected deliverable will be withdrawn from the delivered count and the metric will be recalculated.

For each network the metric is calculated as follows: Possible Availability = (# of devices)*(seconds in month) Actual Availability = (Possible Availability) - (sum of all outages in seconds for servers as measured in Remedy by taking Asset Outage records and subtracting the Actual Start Time from the Actual End Time) Availability = Actual Availability/Possible Availability

Average time to incorporate = (date installation and checkout is complete – date of receipt of patch/update) / # of patches/updates completed in the period.

Metrics will be separate for patches/updates received as “urgent” and all others.

There will be two AQLs for this performance requirement;

AQL for Urgent patches/updates < 2 days

AQL for other than Urgent patches/updates < 7 days The contract shall maintain a log of the receipt date/time for all patches and updates. The contractor shall calculate this metric each month and include in the MSR.

First Contract Resolution Rate = # of user service restoration incidents (received during performance period) resolved during the initial contact / Total # of user service restoration incidents within the performance period.

Monthly AQL for First Contact Resolution Rate shall be equal to or greater than 75%.

Beginning with the sixth month of performance, a second metric will be added to represent the most recent cumulative 6-month rate. The rolling 6-month cumulative rate AQL shall be greater than 80% The contractor shall collect data to support this metric from their QA program and include the calculated metric in the MSR.

User Service Restoration Mean Time to Restore/Resolve (MTTR) Executive Support; 4 hours or less.

User Service Restoration Mean Time to Restore/Resolve (MTTR) (non-Executive) Work Stoppage; 8 hours or less.

The AQL for these metrics is 90%.

Thus, the contractor shall meet the stated MTTR performance standards 90% of the time as measured within a performance period.

The contractor shall collect data to support this metric from their QA program and include the calculated metric in the MSR.

Accuracy rate = (# of records inspected - # of errors identified) / (# of records inspected).

Note: an error could be an unverifiable entry, or the failure to have recorded an entry at time of inspection.

The AQL for this measure will be

Accurate rate > 96%

Sample size and frequency of inspection will be at COR discretion This metric will be based on government random inspection of asset management records.

The Seamless Cloud Integration Rate = (Number of users who access the new cloud service or application - # of users reporting issues with root cause determined to be integration issues) / Number of users who access the new cloud service or application within the first 60 day.

The performance standard for the Seamless Cloud Integration Rate is 95%, and will be calculated for each new cloud service added to the Enterprise.

Note: the intent is to exclude service tickets associated with user error or unfamiliarity with new or replacement service.

The Seamless New Service Integration Rate = (Number of users who access the new service or application - # of users reporting issues with root cause determined to be integration issues) / Number of users who access the new service or application within the first 60 day.

The performance standard for the Seamless New Service Integration Rate is 95%, and will be calculated for each new service added to the Enterprise.

Note: the intent is to exclude service tickets associated with user error or unfamiliarity with new or replacement service. This metric will include new cloud services as well as non-cloud services.

Service Validation Consistency rate = (# of new systems presented - # of new systems rejected) / # of new systems presented for deployment within the period of performance. (note: since the rate of introduction will vary, this metric will be calculated on actions within the previous 3-month period)

Contractor shall develop and deliver for government approval, the customer survey before initiating the survey.

End User Adoption Rate = # of unique identifier hits per month / # of new system users defined in the transition analysis.

End User Adoption Rate should grow steadily in the early months after deployment and stabilize when anticipated user volume is reached.

AQL for End User Adoption rate will grow in monthly increments where;

AQL > 35% at end of month 1 (after deployment);

AQL > 50% at end of month 2 AQL > 75% at end of month 3 AQL > 90% after month 4 The contractor shall collect data to support this metric from their QA program and include the calculated metric in the MSR for each new system deployed.

After the 4th month, or when the rate equals 90% or more, the metric will only be monitored and reported quarterly.

The Seamless New Service Integration Rate

Lessons Learned Capture Rate = # of documented post-transition “retrospectives” / total number of new service transitions from start of task order.

Metric will be a cumulative to date measure.

AQL=95%

This is a cumulative measure and recognizes that not all transitions may not complete the post-transition retrospective but most will.

The contractor shall collect data to support this metric from their QA program and include the calculated metric in the MSR.

Staff Transition Metric = # of planned staff positions fully on-boarded at DTRA / # of staff positions scheduled to be onboarded at end of performance period.

1st Month AQL > 85% 2nd Month AQL > 95% 3rd Month AQL > 98%

The contractor shall collect data to support this metric from their QA program and include the calculated metric in the MSR