Updated_Attachment_J.8_O&M_QASP_V.1_6-12-17.doc

QASP

Quality Assurance Surveillance Plan (QASP)

U.S. Department of Homeland Security Federal Emergency Management Agency

Operations and Maintenance Indefinite Delivery

And Indefinite Quantity

HSFE30-16-R-0009

June 12, 2017

V.1

Table of Contents

1Section 1:

Introduction

11.1.

Purpose

11.2.

Relationship to the Solicitation

11.3.

Task Order Communication

2Section 2:

General Information

22.1.

Requirements

22.2.

Methodology

22.3.

Contractor Responsibility

22.4.

Government Responsibility

22.4.1.

Contracting Officer

32.4.2.

Contracting Officer Technical Representative (COR)

32.4.3.

Customers

32.4.4.

Quality Assurance Evaluators (QAE)

32.5.

Non-Conformance

42.5.1.

Allowable Deviation

42.6.

Remedies

5Section 3:

Quality Assurance Methods

53.1.

General

53.2.

Inspections

53.2.1.

100 Percent Inspection

53.2.2.

Planned Sampling

63.2.3.

Random Sampling

63.2.4.

Special Inspections

63.3.

Customer Feedback Program

63.3.1.

Value of Customer Complaints

73.3.2.

Use of Customer Complaints

73.3.3.

Validating Customer Complaints

73.3.4.

Taking Action Based on Customer Complaints

73.4.

Quality Assurance Approach

73.5.

Interpretation of Results of Analysis

73.5.1.

Excellent Performance

73.5.2.

Satisfactory Performance

83.5.3.

Unacceptable Performance

83.6.

Considerations in Choosing Inspection Methods

83.6.1.

Population Size

83.6.2.

Relative Importance

83.6.3.

Availability of Surveillance Resources

9Section 4:

Surveillance Documentation

94.1.

Record Keeping and Documentation

94.2.

Surveillance Schedule

104.3.

Monthly Performance Summary

104.4.

Performance Evaluation Meeting

Section 1: Introduction

Purpose

This Quality Assurance Surveillance Plan (QASP) has been developed to assure that the selected contractor provides quality support services. The QASP is designed to assist the Government in monitoring the quality and quantity of specific services provided by the contractor. The contractor must develop and implement a quality control system that will ensure the quantity and quality of services complies with the specified requirements of the Task Order.

This QASP has been prepared with the intent of encouraging the highest levels of performance, and efficiency by the contractor. Surveillance will be performed to ensure the performance requirements described in the Task Order (TO) are met. It also defines comprehensive procedures to be followed by the Government to ensure that the contractor accomplishments are properly assessed in comparison with established performance requirements. The Government monitors work performance through observations of Quality Assurance Evaluators. The QASP provides comprehensive Quality Assurance Evaluators activities and has been developed in a format to ensure ease of understanding and to simplify implementation.

1.1. Relationship to the Solicitation

This QASP is not part of the solicitation nor will it be made part of any resulting Task Order. The Federal Emergency Management Agency (FEMA) will retain the right to change the surveillance methods and Quality Assurance (QA) procedures or increase or decrease the degree of surveillance efforts at any time necessary to assure Task Order compliance.

1.2. Task Order Communication

All communication regarding questions or issues related to quality performance assurance and inspection will be directed to the Contracting Officer (CO). As such, the CO must approve any revision to the QASP processes or standards.

Section 2: General Information

2.1. Requirements

Task Order performance requirements are shown in the Performance Requirements Summary (PRS). The PRS defines the key services and sets forth the maximum allowable deviation from standard performance for these services that may occur before further Government action will be invoked (e.g., more detailed Inspection, directed changes to the quality control plan, or re-performance). The PRS also sets forth the surveillance methods the Government will use to evaluate contractor performance for the listed tasks.

2.2. Methodology

Quality Assurance (QA) is a program undertaken by FEMA to evaluate contractor performance of the Quality Assurance Indefinite Delivery and Indefinite Quality (IDIQ) task orders. The purpose of QA is to ensure FEMA is receiving the services paid for and to confirm that the customers are satisfied with the products and services received from the contractor.

For contracted services there must be some means to attest to the value received for the funds spent. While the contractor is ultimately responsible for the performance and quality of all products and services specified in the Task Order, FEMA must be able to confirm that the quantity and quality of goods and services received conform to Task Order requirements.

2.3. Contractor Responsibility

The FEMA Quality Assurance Evaluators evaluates the output products and services provided by the contractor. The output products and services can result from either 1) contractor-developed procedures or from 2) Government/FEMA-specified procedures. When the output is based on contractor-developed procedures, the procedure is examined on an exception basis; that is, satisfactory service output as specified in the Task Order normally indicates that the contractor is using satisfactory procedures. The Quality Assurance Evaluators should be concerned with the contractor procedure only when services are not adequate. When the FEMA specifies the procedure, compliance with the procedure is the desired output.

2.4. Government Responsibility

This section of the QASP briefly defines the duties and responsibilities of the key personnel for Task Order Administration and QA. The key personnel who will be responsible for QA are the CO, Contracting Officer Representative (COR), and Quality Assurance Evaluators.

2.4.1. Contracting Officer

The CO has the authority to administer the Task Order. The CO may delegate many of the day-to-day Contract Administration duties to the COR and Quality Assurance Evaluators. However, certain contractual actions such as certification of invoices, negotiation and issuance of Task Order modifications, resolution of contractor claims and disputes, issuance of Contract Discrepancy Report (CDR), issuance of cure notices, issuance of show-cause letters, termination of the Task Order, and Task Order close-out are retained by the CO.

2.4.2. Contracting Officer Technical Representative (COR)

The COR may serve as the daily manager of one, several, or all requirements. The COR represents the CO and functions as the contractor’s point of contact. The COR supervises the Quality Assurance Evaluators and ensures that they conduct QA properly. If modifications to the Task Order are necessary, the COR will assist the CO in preparing and negotiating the modifications. If there are problems with contractor performance, the COR must inform the contractor of the problems and recommend to the CO what adverse Contractual actions should be taken (such as CDR or issuance of a cure notice). Finally, the COR must coordinate matters of Task Order interpretation with the contractor and the CO.

2.4.3. Customers

Customers are the FEMA staff supported by the contractor. Customers are responsible for assisting the COR in conducting QA by providing information on contractor performance through a Customer Feedback Program.

2.4.4. Quality Assurance Evaluators (QAE)

The Quality Assurance Evaluators play a key role in Contract Administration. They serve as the eyes and ears of the CO and COR. The Quality Assurance Evaluators perform the actual Task Order surveillance and report to the COR. Some of the key Contract Administration duties of Quality Assurance Evaluators include the following:

· Complete surveillance as required by this QASP and make recommendations to the COR for issuance of CDRs or letters of commendation;

· Make recommendations to the COR for the validation of satisfactorily completed work and for administrative actions based on unsatisfactory work or non-performed work;

· Assist the COR in identifying necessary changes to the Task Order, preparing FEMA estimates, conducting QA meetings, approving submittals, and maintaining work files; and

· Promptly furnish the COR with any requests for changes, deviations, or waivers to the Task Order.

The Quality Assurance Evaluators only have authority delegated by the CO. They have no authority to allow the contractor to deviate from Task Order requirements. The Quality Assurance Evaluators also have no authority to direct or interfere with the methods of performance by the contractor or to issue modifications directly to any of the contractor’s personnel unless methods being used are unsafe.

2.5. Non-Conformance

QA facilitates determination of the effects of quality deficiencies through Contract Administration. FEMA’s concern is with the products and services provided and not with the procedures used to produce them. If performance of any required product or service is unsatisfactory and that performance is the result of actions (or lack of action) by the contractor, the CO may take action in accordance with the FAR clause 52.246-4 “Inspection of Services – Fixed-Price” or 52.246-6, “Inspection of Services – Time and Material or Labor-Hour”.

A defect is the lack of something necessary for completeness. In cases where an element necessary to the Task Order is clearly present or absent, or works or does not work, a defect is easy to identify. A problem arises in trying to define a defect where a Task Order element is open to subjective interpretation. Since many service Task Orders deal with service outputs that are subjective, criteria for acceptable performance and for defects must be defined. Defects may be made up of one or more deficiencies. The concept of “substantially complete” should be the basis for inspections.

Work is considered "substantially complete" where there has been no willful departure from the terms of the Task Order and no omission of essential work. The contractor has honestly and faithfully performed the work required and the only variance consists of minor omissions or deficiencies. In general, work is substantially completed when 90 - 95 percent or more is satisfactorily completed. The percentage selected depends upon the type of work performed. This concept relies somewhat on subjective judgment, however, and there are no clear guidelines established.

2.5.1. Allowable Deviation

The allowable deviation from standard performance for a service is reflected in the Acceptable Level of Performance (ALP). The ALP represents the maximum percent defective or the maximum number of defects per hundred units considered satisfactory for purposes of a sampling inspection. It is the allowable variance from a standard expressed by narrative description before the Government will reject the specific service. Exceeding this allowable variance from the standard will cause a service level to be unsatisfactory. It must be emphasized an ALP should not be construed as providing the contractor an outlet to avoid performing a certain percentage of the required service. The ALP does not imply the contractor may knowingly offer defective service to the Government; best efforts are required.

2.6. Remedies

Penalties are defined as those actions taken under the direction of the CO against the contractor within the general provisions of the Task Order for nonconformance to the requirements of the Task Order.

In accordance with FAR 52.246-6, the Government may require the contractor to re-perform any services that do not conform to Task Order requirements. If the defects cannot be corrected by re-performance, the CO may either require the contractor to take the necessary action to ensure that future performance conforms to the requirements, or he/she may reduce the Task Order price to reflect the reduced value of the services performed. At an extreme decision point, penalties may include a decision not to exercise Task Order options. The CO will determine the penalty for nonconformance based upon his or her judgment and the severity of the nonconformance.

Section 3: Quality Assurance Methods

3.1. General

This QASP has been developed to provide the Quality Assurance Evaluators with an effective and systematic surveillance approach for assessing the performance of functions set forth in the Task Order.

It provides a method to evaluate the services the contractor is required to furnish (i.e., a particular level of performance as defined by a performance requirement) and not the details of how the contractor accomplishes the work. It also identifies the various surveillance methods that, when properly used, provide the Government with the ability to determine acceptable performance.

This QASP is based on the premise that the contractor, and not the Government, is responsible for management and quality control actions to meet the terms of the Task Order. The APL listed in the PRS in the RD recognizes that contractor performance will not be perfect and unforeseen and uncontrollable problems do occasionally occur. However, good management and the use of an adequate QC Plan will allow the contractor to operate within specified performance requirements.

QA resources are extremely scarce and costly and they must be used efficiently. To achieve the desired efficiency, FEMA will employ two approaches for its quality assurance: inspections and customer feedback.

3.2. Inspections

The best means of determining whether the contractor has met all Task Order requirements is to inspect the results. Such inspections confirm that value is received for the money spent or that problems exist. Documented inspection results are the most effective tool used to enforce Contract administration actions.

Under this QASP, there are four primary surveillance methods used to evaluate the work performed for the work performed by the IDIQ contractors: One Hundred Percent Inspection, Planned Sampling, Random Sampling, and Special Inspections. These methods are briefly described below.

3.2.1. One Hundred Percent Inspection

One Hundred Percent Inspection is an evaluation method that requires total inspection of a Task Order requirement. These inspections should be used for requirements that are especially critical or where there is some reason for suspecting that the performance standard is not being met (and therefore, should be more closely monitored). One Hundred Percent Inspection should also be used for monitoring scheduled Task Order requirements (such as one-time deliverables, scheduled submissions, and infrequently identified requirements). Although One Hundred Percent Inspections are the most costly, they provide positive proof of delivery of critical requirements.

3.2.2. Planned Sampling

Planned sampling provides a systematic way of looking at service output and forming conclusions about the contractor’s level of performance. Evaluation by planned sampling is designed to inspect some part but not all of the services being monitored. Specific occurrences of Task Order requirements that are to be monitored are selected for evaluation prior to their scheduled accomplishment.

Planned sampling differs from random sampling in the way in which samples are selected. Sample selection is based on some subjective rationale and sample sizes are usually arbitrarily determined. With this type of evaluation, the contractor knows work performed in specific functions or selected locations is more likely to be monitored than work in other areas or locations. The COR directs efforts to those areas where inspection is most needed.

Planned sampling, unlike random sampling, does not provide a sound statistical means of making comparisons between observed and overall performance, and the contractor’s overall level of performance cannot be determined. The cost of planned sampling varies with the level of inspections. Such latitude is important to manage limited resources and focus inspections on known or suspected problems areas.

3.2.3. Random Sampling

Random Sampling is a method whereby some part, but not all, of contractor performance is evaluated. Random Sampling, which is based on statistical principles, estimates the contractor’s overall level of performance for a given Task Order requirement. What distinguishes it as Random Sampling is that each service output in the lot has an equal chance of being selected for inspection. In this manner, the Quality Assurance Evaluators need only make a few observations from which to project the quality of the entire lot. The organization performing the work does not know which service output will be observed; consequently all must be done correctly. Also, the Quality Assurance Evaluator is prevented from biasing the sample by his/her own judgment.

Random Sampling is preferred when surveillance resources are limited, the surveillance lot is large and relatively homogeneous, the service requirements occur continuously or frequently (otherwise an inspector may have to be devoted full time to one activity), the activities are located in the same general geographical area, and a well documented performance audit trail is available to use for evaluation, such as completed customer complaints or work logs. Random Sampling is the preferred method for this QASP because it provides a non-biased, comprehensive evaluation of contractor performance with an efficient use of limited inspector personnel.

3.2.4. Special Inspections

Special inspections consist of impromptu evaluations of Task Order requirements whenever the contractor is not delivering a monitoring tool such as the Customer Complaint Record. This method is very similar to planned sampling except for the omission of a planned schedule. The costs of special inspections are usually higher than for regular planned sampling because they are used in reaction to identified quality problems.

3.3. Customer Feedback Program

The Government shall have a Customer Feedback Program. At the beginning of the Task Order, the COR will use service level agreements with customers to assessment any problems with services they receive or identify when superior services are received. Copies of all complaints shall be provided to the FEMA COR.

3.3.1. Value of Customer Complaints

Validated customer complaints are a QA method based on customer experience and require a high level of customer awareness. Customers are made aware of Task Order requirements and monitor the performance of the contractor. Where there is a case of poor performance or nonperformance, customers notify the COR. The COR then investigate the report and, if found to be valid, document their findings. The numbers of complaints and resulting inspections depend upon customer awareness and response. Since special inspections based on validated customer complaints cannot be scheduled prior to work accomplishment, this method may supplement other surveillance techniques.

3.3.2. Use of Customer Complaints

Customer complaints will be used to monitor quality and provide an indication that quality problems may exist. The actual documentation of the quality problem, should it exist, will be by performing an inspection of the specific services provided.

3.3.3. Validating Customer Complaints

Customer Complaints Records submitted to the contractor and COR must be validated. It is primarily the responsibility of the contractor to investigate each complaint to determine the problem (even if the problem is the customer). While CORs can also investigate customer complaints, the responsibility for initial review shall remain with the contractor. At the Government’s discretion, the COR will investigate problems from customer groups and complaints involving major problems with the services being provided. After the complaint has been investigated, the contractor shall take appropriate action, complete the complaint form, and forward the form to the COR.

3.3.4. Taking Action Based on Customer Complaints

The contractor must take action every time a Customer Feedback Record is received. If a valid complaint exists, the contractor shall re-perform the product or service until the customer is satisfied. The contractor shall also use the complaint as an indicator that the quality control program and process methodology needs improvement to prevent the recurrence of similar problems in the future or that similar problems will be detected and fixed before a product or service is delivered to the customer. If the customer complaint is found not to be valid, the contractor shall educate the customer about what can and cannot be expected from the Contracted services. When the complaint has been resolved, the contractor shall provide a copy to the Quality Assurance Evaluators accompanied by a written explanation of the actions taken to resolve the complaint.

3.4. Quality Assurance Approach

The Quality Assurance Evaluators shall use the Performance Requirements Summary as guidance for conducting their inspections of the contract tasks. The Quality Assurance Evaluators will evaluate the contractor’s performance in functional areas, Task Order requirements and specific work requirements. To receive an acceptable performance rating, the contractor must meet or exceed the allowable deviation for the performance requirement being evaluated.

3.5. Interpretation of Results of Analysis

Analysis of all types of Task Order monitoring will result in one of the following outcomes: excellent performance, satisfactory performance, or unacceptable performance.

3.5.1. Excellent Performance

When the requirements of the PRS are met and no validated customer complaints are lodged and the contractor has performed in the best possible manner, FEMA may reduce its level of surveillance. The COR might also suggest that the contractor be notified by the COR that the performance has been better than satisfactory and to continue the good work.

3.5.2. Satisfactory Performance

The contractor’s performance is satisfactory when the number of validated complaints does not exceed the complaint threshold or Allowable Deviation for Planned Sampling, Random Sampling, or 100 Percent Inspection. The CO may suggest to the COR that an increased level of surveillance be used for important services which show defect rates approaching the Allowable Deviation, and that the contractor be notified that the performance is marginal.

3.5.3. Unacceptable Performance

When the allowable deviation for any service has been exceeded, the contractor’s performance is unsatisfactory and is, therefore, unacceptable. The COR should suggest one or more of the following actions be taken.

· The level of surveillance is increased until the contractor demonstrates acceptable performance over a period of time.

· The CO or COR meets with the contractor to discuss discrepancies, trends, and intended corrective measures.

3.6. Considerations in Choosing Inspection Methods

A number of inspection methods can be used for monitoring performance under Task Order requirements. Various factors should be considered in deciding which surveillance method to use with each performance requirement. For example, in deciding whether to conduct 100 Percent Inspections or Random Sampling, two primary factors should be considered – lot size and relative importance. If the inspection methods used in this QASP need to be modified in the future, the factors briefly described below should be considered in making the modifications.

3.6.1. Population Size

Population size, or lot size, refers to the number of expected occurrences of a service during the surveillance period. Population size is determined from work schedules or estimated from historical and projected data. For small sample populations the contractor will normally be evaluated using 100 Percent Inspection. Large sample populations will normally use Random Sampling, as it is ideally suited to large, homogeneous populations. One Hundred Percent Inspection would be extremely time consuming and expensive for the Government to implement, although relative importance may necessitate use of this surveillance method.

3.6.2. Relative Importance

Some work requirements and related performance requirements are so critical (e.g., safeguarding sensitive material) that poor performance under any of these requirements would pose a direct risk to the safety of personnel and result in a failure to meet its mission requirements. One Hundred Percent Inspection has usually been considered for critical work requirements.

3.6.3. Availability of Surveillance Resources

The number of Quality Assurance Evaluators assigned to a Task Order is a driving factor in selection of an inspection method. With limited resources, inspection methods are frequently selected for both their effectiveness and efficiency. Random Sampling is the preferred method when surveillance resources are limited. Similarly, using customer comments through Customer Satisfaction ratings and the Customer Complaint process augments surveillance resources.

Section 4: Surveillance Documentation

4.1. Record Keeping and Documentation

An accurate and complete record of all inspections performed is required. The Quality Assurance Evaluator is responsible for updating the monthly surveillance schedule, recording customer complaints and preparing other documentation that reflects the quality and quantity of the contractor’s performance. In accordance with the “Consequences of contractor’s Failure to Perform Required Services” clause, the Government is required to give the contractor prompt notice of defects observed by the Quality Assurance Evaluators in the course of his or her inspections. The requirement does not imply that the Government is obligated to carry out quality control inspections for the contractor, but that on discovering defects, the Government will inform the contractor in detail of the defects location. Sending copies of inspection reports to the contractor is considered notice. It is recommended that the contractor’s Representative be requested to acknowledge the receipt of the notice of the defect by initialing a copy of the inspection report.

There should be a complete audit trail from the schedule, to the performance of surveillance inspections, to the completion of the surveillance observation records, and finally to the submission of CDRs, when applicable. An auditor must be able to track the above items to the CO's final disposition.

The records maintained by the Quality Assurance Evaluators are the basis for determining unacceptable performance and taking any corrective actions under the Task Order. The Quality Assurance Evaluators will maintain the follow material on file:

· Quality Assurance Evaluators Appointment Letter

· Copy of Task Order, with Attachments

· Task Order Modifications and Amendments

· QA Plans

· Contractor’s Quality Control Plan

· List of contractor’s key personnel, and the designated contractor Representative for this Task Order

· Memoranda of Record (phone conversations, meeting minutes)

· Copies of all surveillance schedules, surveillance records, customer complaints and other supporting data

The Quality Assurance Evaluators will forward the documents to the CO's Task Order file at the close of the Task Order.

4.2. Surveillance Schedule

One of the duties of the COR is to develop a monthly schedule for inspections based on the QASP requirements. This schedule indicates when the various RD requirements will be monitored. The COR must complete the monthly schedule by the last workday of the preceding month before the inspection. Include in the schedule all required services and those areas considered essential to adequately monitor performance. Remember that surveillance must cover all hours of operation. The COR needs to schedule random observations at night, on weekends, and on holidays if the service is performed during these periods. The COR/Quality Assurance Evaluators Surveillance Schedule provides documentation showing which activities the Quality Assurance Evaluators will monitor and the inspection times during the month.

This monthly schedule shows where to add what the Quality Assurance Evaluators are monitoring at all times. The COR needs to give copies of the Surveillance Schedule to the Quality Assurance Evaluators as soon as it is finished so the Quality Assurance Evaluators can use the information to plan their activities and perform their inspections.

Mark the Surveillance Schedule “FOR OFFICIAL USE ONLY,” and do not show it to the contractor to preserve the anonymity of the inspection procedures. If the schedule is shown to the contractor, they will have advance warning of the areas you plan to inspect and may use this information to skew their performance. Submit a copy of the Surveillance Schedule to the CO for his/her information and review.

When preparing the Surveillance Schedule, first program those functions involving Planned Sampling or One Hundred Percent Inspection. The remaining days are designated to examine the logs, documentation, reports, and working days of the randomly sampled functions.

The COR must document and explain the reasons for any changes to the Surveillance Schedule, post the changes, and send a copy of the revised schedule to the CO. Actual surveillance activity recorded on the checklists must be comparable to the monthly schedule. Also, the CO and the COR must be able to monitor the Quality Assurance Evaluator’s performance by using the monthly schedule as updated.

4.3. Monthly Performance Summary

The Quality Assurance Evaluators will develop a report indicating the level of performance under each work requirement and performance requirement evaluated during the period. The report should include any additional information relevant to the contractor’s performance for each task inspected during the month. The Quality Assurance Evaluators will advise the COR of the results and recommend appropriate course(s) of action. Unacceptable performance under any performance requirement translates into COR action. In this situation, the Quality Assurance Evaluators may decide to include a recommended CDR.

Evaluating the contractor’s performance provides the basis for deciding if the contractor needs to implement corrective actions, whether to increase, decrease or maintain the level of future surveillance, and whether any Task Order penalties need to be imposed.

4.4. Performance Evaluation Meeting

An initial step in avoiding disputes and claims is to settle minor problems and misunderstandings at the lowest possible level. In addition to the on-site evaluations, Performance Evaluation Meetings could be held on a regular basis if considered useful by the CO or COR. The Quality Assurance Evaluators will prepare a formal agenda covering a review of performance, procedural problems, communication problems, status of outstanding work, and any other matters of concern to either the Government or the contractor. Recognition of good performance should also be discussed. The Quality Assurance Evaluators will record the minutes and sign and date them along with the contractor’s Representative.

ATTACHMENT 1: PERFORMANCE REQUIREMENTS SUMMARY

Required Services

(Tasks) Performance

Standards Acceptable Quality

Levels Methods of

Surveillance

Program and project Monthly Status Report and executive monthly status briefing

System Maintenance, Code Deployments, Availability of Systems and Capacity reporting.

Meet the performance standard (acceptable quality level) as defined in the Service Level Agreement (SLA) with customers/programs.

Perform the full suite of code maintenance tasks using Agile methodologies, including, but not limited to: participating in creating user stories for both business functionality, technical requirements and defining acceptance criteria; estimating the size of stories; solution design; development; and testing.

Support FEMA in its core Mobility Services business to include: identifying business needs, recommending wireless plans and applications, wire line, and satellite technologies to meet those needs, and managing these technologies from financial, contractual, and operational perspectives Provide a Tier 1 (i.e., phone-based) technical end-user support for FEMA as well as a Tier 2 (i.e., in-person) technical end-user support for end-users systems (desktops, laptops, tablet PC’s, and other peripheral devices).

Support for centralized operations, monitoring, failure and intrusion detection, and restoration of FEMA’s nationwide telecommunications and network systems, including video teleconferencing operations for FEMA senior-level management, federal, state, and local partners and other DHS components.

Quality of Work Force - qualified contractor staff Provide surge staffing at the request of the Contracting Officer (CO). The Contractor shall provide this staff in time of emergencies when current personnel have been deployed in an emergency response and backfill is needed to provide security guidance for the systems.

100% accurately depict current status

Contractor is compliant 99% of the time Contractor must provide 99% uptime Must complete deployment schedules 99% of the time.

Monthly status reports on MSC status are 100% accurate Customer Survey maintain 98% successfully rating 99% Network uptime Staff is onboard within 30 days and meets position requirements Surge staff is onboard within two weeks 98%

98% 98% 98% 98%

96%

98%

98%

98% File reviews, periodic inspections, and random, observations File reviews, NETIQ reports, and random, observations

File reviews, periodic inspections, and random, NETIQ reports, Customer complaints

File reviews, periodic inspections, and random, observations, Customer complaints

File reviews, periodic inspections, and random, observations, Customer complaints

Surveys, observations, Customer complaints

Observation via NOC tools and reports, Customer complaints, inspections.

File reviews, observations

File reviews, observations

ATTACHMENT 2: SAMPLE QUALITY ASSURANCE

MONITORING FORM

SERVICE or STANDARD:

SURVEY PERIOD:

SURVEILLANCE METHOD (Check):

Random Sampling 100% Inspection Periodic Inspection Customer Complaint

LEVEL OF SURVEILLANCE (Check):

Monthly Quarterly As needed

PERCENTAGE OF ITEMS SAMPLED DURING SURVEY PERIOD:

ANALYSIS OF RESULTS:

Observed Service Provider Performance Measurement Rate: ______%

Service Provider’s Performance (Check):

Meets Standards

Does Not Meet Standards

Narrative of Performance During Survey Period:

PREPARED BY: ___________________________________

DATE: _________________

i

PAGE