QASP_-_Best_Execution.doc

Performance Requirement Summary

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

Property Valuation Services

TABLE OF CONTENTS

31.0

INTRODUCTION

31.1

PURPOSE

31.2

PERFORMANCE MANAGEMENT APPROACH

41.3

PERFORMANCE MANAGEMENT STRATEGY

52.0

ROLES AND RESPONSIBILITIES

73.0

IDENTIFICATION OF SERVICES TO BE PERFORMED

74.0

METHODOLOGIES FOR MONITORING PERFORMANCE

105.0

QUALITY ASSURANCE REPORTING

106.0

ANALYSIS OF QUALITY ASSURANCE MONITORING RESULTS

117.0.

FAILURE TO PERFORM

LIST OF ATTACHMENTS:

APPENDIX A:

PERFORMANCE REQUIREMENTS SUMMARY

HUD will periodically evaluate the Contractor’s performance by monitoring performance to ensure services are received in accordance with the Performance Work Statement (PWS). The Government Technical Representative/Government Technical Manager (GTR/GTM) will evaluate the Contractor’s performance through one or more of the surveillance methods described in paragraph 4.0—Methodologies to Monitor Performance. In addition, HUD may increase the number of quality control reviews if deemed appropriate because of repeated failures discovered during quality control reviews. Likewise, HUD may decrease the number of quality control reviews if service provider has established a good performance record. The GTR/GTM or the Contracting Officer shall make final determination of the validity of all complaint(s).

If any of the services do not conform to contract requirements, HUD may require the Contractor to perform the services again in conformity with contract requirements, at no increase in contract amount. When the defects in services cannot be corrected by re-performance, HUD may:

(a) Require the Contractor to take necessary action to ensure that future performance conforms to contract requirements; and

(b) Reduce the contract price to reflect the reduced value of the services performed. Performance scoring will be in accordance with the acceptable quality level (AQL) identified in the performance metrics table.

QUALITY ASSURANCE SURVEILLANCE PLAN

(QASP)

1.0

INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) is pursuant to the requirements listed in the performance-based Performance Work Statement (PWS) for Operations Assessment Services. This performance-based plan sets forth the procedures and guidelines the Federal Housing Administration (FHA) will use in evaluating the technical performance of the Operations Assessment Contractor.

1.1

PURPOSE

1.1.1. The purpose of the QASP is to describe the systematic methods used to measure performance. The QASP provides a means for evaluating whether the contractor is meeting the performance standards identified in the PWS. This QASP describes the procedures that the GTR/GTM will use to monitor the service provider’s performance. HUD’s primary concern is with the products and services provided and not with the procedures used to produce them. Therefore, the QASP focuses on examining the products and services provided and less on the processes used to produce them. It is intended that the QASP be a tool to guide the GTR/GTM in assessing performance. In some cases, specific metrics are used to measure performance, in other cases subjective judgment and evaluation by HUD personnel will be the determining criteria. This plan describes the methodology utilized to make both quantitative and qualitative evaluation of performance under the contract.

1.1.2 This QASP is designed to define roles and responsibilities, identify the performance objectives, define the methodologies used to monitor and evaluate the contractor’s performance, describe quality assurance reporting, and describe the analysis of quality assurance monitoring results.

1.2

PERFORMANCE MANAGEMENT APPROACH

1.2.1 The performance-based PWS structures the acquisition around “what” service is required as opposed to “how” the contractor should perform the work. This QASP will define the performance management approach taken by the HUD to monitor, manage, and take appropriate action on the contractor’s performance against expected outcomes or performance objectives communicated in the PWS. Performance management rests upon developing a capability to review and analyze information generated through performance metrics. The ability to make decisions based on the analysis of performance data is the cornerstone of performance management. The data generated in a performance management approach provides information that indicates whether or not expected outcomes for required services are being achieved adequately by the contractor.

1.2.2 Performance management also represents a significant shift from the more traditional Quality Assurance (QA) concepts in several ways. Performance management focuses on assessing whether or not outcomes are being achieved and migrates away from scrutiny on compliance with the processes and practices used to achieve the outcome. The only exceptions to process reviews are those required by law (Federal, State, and local) and compelling business situations such as safety and health. An outcome focus provides the contractor flexibility to continuously improve and innovate over the course of the contract as long as the critical outcomes expected are being achieved at the desired levels of performance.

1.3

PERFORMANCE MANAGEMENT STRATEGY

Performance will be monitored through various surveillance methods described in Section 4: METHODOLOGIES FOR MONITOR PERFORMANCE. Performance data gathered will be evaluated to assess contractor performance against contract requirements.

1.3.1 The contractor’s internal quality control system will set forth the staffing and procedures for self inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the PWS.

1.3.1.1 The Government representative will monitor performance and review performance reports and performance objectives:

1.3.1.2 Ensure that all functions and processes related to the research and data gathering for the requirements of the PWS are performed in a manner that is in compliance with local, state and national laws and guidelines.

1.3.1.3 Ensure that the data researched and delivered to HUD are accurate and comprehensively identifies the source of the data collected, including how it was obtained, collected, and evaluated.

1.3.1.4 Ensure electronic delivery of the operating data in an acceptable format within the timeframes specified in the PWS.

1.3.1.5 Employ efficient automation and quality assurance measures to ensure smooth, reliable, and seamless flow of data between the Contractor and parties designated by HUD.

1.3.1.6 Ensure the integrity of the data and prevent loss of sensitive business data that is stored and transferred between the Contractor’s technology environment and parties designated by HUD.

1.3.1.7 The contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively.

2.0

ROLES AND RESPONSIBILITIES

The purpose of quality assurance is to ensure that the goods and/or services provided are in accordance with the PWS and other requirements of the contract. The roles and responsibilities of the stakeholders involved in QA are described below.

2.1 Service Provider

The service provider is responsible for delivering products or services in accordance with the contract. The service provider is responsible for implementing its Quality Control Plan (QCP), which is incorporated in the contract. The QCP describes the service provider’s methods for ensuring all products and services provided under the contract meet established performance standards and AQLs. The service provider is responsible for producing, maintaining, and providing for audit, quality control records, reports, and all records associated with the investigation and resolution of customer complaints. The service provider should appoint a single quality control point-of-contact to act as a central recipient of communication from the Government.

2.2 Contracting Officer (CO) The CO is responsible for monitoring contract compliance, contract administration and cost control; and resolving any differences between the observations documented by the Contracting Officer’s GTR/GTM and the contractor’s performance. The CO delegates many of the day-to-day contract administration duties to the GTR/GTM. However, certain contractual actions such as negotiation and issuance of contract modifications, resolution of service provider claims and disputes, issuance of cure notices (notification that unless unacceptable performance is corrected, the Government may terminate the contract for default, IAW FAR 49.607), issuance of show-cause letters (following a cure notice, requesting facts bearing on the case), termination of the contract, and contract close-out functions are retained by the CO. Administrative actions such as invoice approval and issuance of Contract Discrepancy Reports are delegated by the CO to the GTR. For tasks and/or subtasks which include incentive arrangements (award fee, shared savings, award term, penalties, etc.), the GTR shall provide recommendations to the CO for action. All communication regarding questions or issues related to quality assurance and inspection will be directed to the CO or the GTR. The CO shall approve any revision to the QASP processes or standards.

The CO will designate one full-time GTR/GTM as the Government authority for performance management. The number of additional representatives serving as technical inspectors depends upon the complexity of the services measured as well as the contractor’s performance.

2.3 Government Technical Representative/Government Technical Manager (GTR/GTM)

The GTR/GTM is responsible for monitoring, assessing, and communicating the technical performance of the contractor. The GTR/GTM will have the responsibility for completing quality assurance monitoring forms (refer to Attachments II and III) used to document the evaluation of the contractor’s work performance.

The GTR/GTM is a federal employee who is designated by name and/or position to act as a liaison between the Government and the service provider on all issues pertinent to the daily performance of the contract. The GTR/GTM represents the CO. In turn, the GTR/GTM may delegate some of his/her responsibilities, such as supervision of the quality assurance evaluators, to another individual in the organization in order to ensure that the quality assurance function is efficiently executed. If modifications to the contract are necessary, the GTR will assist the CO in preparing and negotiating the modifications. If there are problems with service provider performance, the GTR/GTM will inform the service provider, in writing, of the problems and recommend to the CO any adverse contractual actions as appropriate (e.g., cure notice) if the service provider fails to correct the problem. The GTR refers differences of contract interpretation to the CO.

2.4 Quality Assurance Evaluators

The GTR/GTMs play a key role. They serve as the on-site representatives of the CO and the GTR. The GTM performs the actual contract surveillance and report to the GTR. Some of the key administrative duties of GTM include, but are not limited to, the following:

· Perform surveillance as required by this QASP, and make recommendations to the GTR;

· Make recommendations to the GTR for the acceptance or rejection of completed work and for administrative actions based on unsatisfactory work or non-performed work;

· Assist the GTR in identifying necessary contract modifications;

· Make recommendations to the GTR for changes to the QASP;

· Assist the GTR in preparing reports on performance and cost.

The GTM only has the authority delegated to them in writing by the GTR and/or CO. They have no authority to direct or to allow the service provider to deviate from contract requirements. The GTM also has no authority to direct or interfere with the methods of performance by the service provider or to issue directions to any of the service provider’s personnel. These actions are reserved to the CO or to the GTR.

The GTM may use the form provided in Attachments II and III, or such other forms as approved by the GTR.

2.5 Customers

Customers are the various organizations whose organizational performance is or can be affected or supported by the service provider. Customers may be requested to assist the GTM and GTR in conducting quality assurance by providing information on service provider performance through customer feedback. The information gained from customer feedback may be used in conjunction with other methods of observation to rate the performance of the service provider.

3.0

IDENTIFICATION OF SERVICES TO BE PERFORMED

The contractor shall provide property valuation services to establish a value for properties insured, to be insured, or formerly insured by the Federal Housing Administration. In the performance of these services, the Contractor shall use its own proprietary valuation model, analysis, and staffing resources to complete the requirements and tasks as outlined in the PWS. The performance standards are established in the paragraph of the PWS that covers the specific category of work.

4.0

METHODOLOGIES TO MONITOR PERFORMANCE

The surveillance methods used in the QA process are the Government’s tools to monitor the service provider’s products and services to determine whether the performance is in compliance with the contract requirements. Documented inspection results are an effective tool in contract administration. Inspections either confirm successful achievement of the performance requirements or highlight areas where defects exist and improvements are necessary.

The primary methods of surveillance are reports and quality control reviews. In addition, the government may also perform internal audits and/or obtain customer input/feedback to measure the Contractor’s performance. The Government will use appointed representatives, as well as reports and quality control reviews as sources to monitor the contractor’s performance.

The surveillance methods described below that will be used in assessing the quality of contractor performance include as periodic inspection of the credibility of the proprietary AVM/AVE as HUD determines, a 100% inspection of all AVM and AVE deliverables as assigned for completeness, timeliness, and accuracy. In addition, contractor performance will also be assessed based on customer feedback.

If the Contractor is identified to have poor performance, HUD may increase the level of surveillance and focus on known problem areas. In either case, the reasons for the change in surveillance will be documented.

4.1 100 Percent Inspection

The 100 percent inspection method requires complete inspection of a contract requirement and will be used for requirements that are especially critical or where there is some reason for suspecting that the performance standard or AQL is not being met (and therefore, should be more closely monitored). Evaluation schedules for 100 percent inspections will be prepared each month.

Performance Standards and AQLs

The performance standards and AQLs may be stated as either percentages or absolute numbers.

Evaluation Procedures

Observed defects for a service monitored by 100 percent inspection is compared to the performance standard and AQL.

4.2 Periodic Inspection

Periodic inspection provides a systematic way of looking at service outputs and forming conclusions about the service provider's level of performance in accordance with a planned schedule of surveillance. Evaluation by periodic inspection is designed to inspect some part but not all of the products and services being monitored.

Application

Specific contract requirements that are to be monitored are selected for evaluation prior to their scheduled accomplishment. Periodic inspection differs from random sampling in the way in which samples are selected – periodic inspection sample selection is based on some subjective rationale and sample sizes are usually arbitrarily determined. With this type of evaluation, the GTM’s are able to direct efforts to those areas where inspections are most needed, and the service provider knows that those areas are more likely to be monitored than others. Periodic inspection, as compared with random sampling, provides a less sound statistical means of making comparisons between observed and overall performance, and the service provider's overall level of performance. Periodic inspection is generally used in two ways. First, it can provide a one-time subjective evaluation of service provider performance. Second, it can be used to detect a change in the service provider’s level of performance (i.e., trend analysis). This method requires that the sample selection criteria be well documented and consistently applied from period to period, and that there are no other intervening factors. The cost of periodic inspections varies with the level of inspections. Such latitude is important to manage limited resources and focus inspections on known or suspected problems areas.

Performance Standards and AQLs

Performance standards and AQLs are usually stated in terms of the number of defects detected per time period (e.g., three times per month). There is no specific relationship between sample size and performance standard/AQL. However, when the AQL is expressed as a percentage, it is recommended that the maximum sample size be chosen such that one defect does not exceed the AQL.

Evaluation Procedures

The levels of evaluation appropriate for periodic inspection are judgmental. In order to perform trend analysis from periodic inspection, criteria for sample selection should be applied consistently from period to period. To ensure valid results, the GTR/GTMs will use periodic inspection evaluation sheets and follow a detailed inspection schedule. Schedules may be developed monthly to coincide with the service provider's monthly schedule of work, and regularly updated after receiving the service provider's definitive weekly schedule. Observed defects for services monitored by periodic inspection will be totaled at the end of each month. For each service, the total number of defects will be compared to the performance standard and AQL.

4.3 Customer Feedback

Validated customer feedback is a quality assurance method based on customer and service provider interaction. Customers continually receive the outputs of Contractor performance and are in a position to evaluate the service provider on a recurring basis. Because customers have a clear stake in the quality of service provider services, they are valuable resource for the GTM’s.

Application

Customers are made aware of contract requirements and monitor the services provided by the service provider, both positive and negative. Where there is a case of poor performance or non-performance, customers notify the GTR/GTM. The GTR/GTM then investigate the report and, if found to be valid, document their findings. The numbers of complaints and resulting inspections depend upon customer awareness and response. If the complaint is valid and caused by poor performance or non-performance by the service provider, the service provider must take appropriate corrective action. A valid complaint is one in which the GTR/GTM confirms that poor performance or non-performance violates contract requirements.

Customer Feedback Process

Upon contract award, the GTR/GTM should send letters to all or selected customer points-of-contact. These letters will inform them of the need for their active participation in the overall Quality Assurance Program. The GTR/GTM will also provide a customer feedback record (sample at Attachment III) for the customer to use to either document performance problems or identify when superior services are received.

The GTR/GTM will validate the Customer Feedback Records submitted. It is primarily the responsibility of the service provider to investigate each complaint to determine the problem. While GTR/GTMs can also investigate customer complaints, the responsibility for initial review shall remain with the service provider. At the Government’s discretion, the GTR/GTM will investigate problems from customer groups and complaints involving major problems with services being provided.

The service provider shall take action when a customer feedback record is received. If a valid complaint exists, the service provider shall re-perform the product or service. The service provider may use the complaint as an indicator that the QCP needs improvement. Corrective actions shall be implemented to prevent the recurrence of similar problems in the future or detect and fix such problems before a product or service is delivered to a customer. If the customer complaint is found to be invalid, the GTR shall educate the customer regarding contract requirements as they pertain to the customer’s expectations.

Evaluation Procedure

The service provider shall report validated complaints each month, so the GTR/GTM may review the valid complaints and formulate action items if necessary. Trend analysis may be used to test for variations in the number of complaints received each month and identify changes in service provider performance.

5.0

QUALITY ASSURANCE REPORTING

5.1 The performance management feedback loop begins with the communication of expected outcomes. Performance standards are expressed in the PWS and measured by the required performance metrics listed in Appendix A.

5.2 The Government’s QA monitoring, accomplished by the GTR/GTM (and others as designated) will be reported using the monitoring forms in Attachments II and III. The forms, when completed, will document the GTM/GTR’s understanding of the contractor’s performance under the contract to ensure that the PWS requirements are being met.

5.2.1 The GTR/GTM will retain a copy of all completed QA monitoring forms.

6.0

ANALYSIS OF QUALITY ASSURANCE MONITORING

RESULTS

6.1 The Government shall use the methods cited to determine whether the acceptable quality levels have been met. The Government’s evaluation is then translated into the specific positive or negative incentives that cause adjustments to the contractor’s payments.

6.2 At the end of each month, the GTR/GTM will prepare a written report for the Contracting Officer (CO) summarizing the overall results of the quality assurance monitoring of the contractor’s performance. This written report consists of the contractor’s submitted monthly progress report and the completed Quality Assurance Monitoring Forms (Attachment II) that will become part of the QA monitoring documentation.

6.3 The CO may require the contractor’s project manager, or a designated alternate, to meet with the CO and other Government personnel as deemed necessary to conduct/discuss a performance evaluation. The GTR/GTM will define a frequency of in-depth reviews with the contractor. However if the need arises, the contractor will meet with HUD as often as required or per the contractor’s request. The agenda of the reviews may discuss:

· Monthly performance measured by the objectives outlined in the PWS.

· Issues and concerns of both parties

· Projected outlook for upcoming months and progress against expected trend

· Recommendations made to/by the GTR/GTM based on contractor information

· Issues arising from QC reviews and review of the Contractor’s reports 6.4 In addition to QA monitoring, the GTR/GTM will use the information contained in the contractor’s monthly report to assess the contractor’s level of performance for each objective outlined in section 1.3. The GTR/GTM must coordinate and communicate with the contractor to resolve issues and concerns of marginal or unacceptable performance. The contractor will discuss with the GTR/GTM satisfaction ratings receiving a “less than acceptable” rating. For such cases, the contractor should highlight its perspective on factors driving customer satisfaction and present a corrective action plan to address any deficiencies noted within a time frame that is agreed upon by the GTR/GTM..

6.5 The GTR/GTM and contractor should jointly formulate tactical and long–term courses of action. Decisions regarding changes to metrics, thresholds, or service levels should be clearly documented. Changes to service levels, procedures, and metrics will be incorporated as a contract modification at the convenience of the CO.

7.0.

FAILURE TO PERFORM

7.1 The contractor may receive deductions or even termination based on failure to perform. The following criteria apply for determining appropriate action:

1. Notifications. Consistent with FAR Part 49, the CO shall notify the Operations Assessment Contractor of a failure to meet standards through QA monitoring forms, cure notices, and shall inform the service provider project manager or designated alternate of such notices.

2. Deductions. The Government has the right to withhold the entire agreed to payment for failing to perform particular services of a task order based on a failure to meet performance standards.

3. Termination. If the CO determines that the contractor has failed to perform to the extent that a termination for default is justified, the CO shall issue a notice of termination, consistent with FAR Part 49.

Appendix A

The Operations Assessment Contractor is expected to establish and maintain professional communication between its employees and HUD. The primary objective of professional communication between employees and customers is customer satisfaction. Customer satisfaction is the most significant external indicator of the success and effectiveness of all services provided and can be measured through customer complaints. Performance management drives the contractor to be customer focused through addressing HUD’s complaints and investigating the issues and/or problems.

The GTR/GTM will investigate customer complaints using the Quality Assurance Monitoring Form – Customer Complaint Investigation identified in Attachment III.

The performance measurement requirements and acceptable quality levels of performance are outlined below:

-and-

Customer Feedback 10% of the total order invoice Disincentive for each dissemination error

Quality Assurance Surveillance Plan

SPECIFIC REQUIREMENTS FOR QUALITY ASSURANCE REVIEWS

OF PROPERTY VALUATION SERVICES CONTRACT

1. Element to be monitored: Contractor provided complete and correct AVMs for 100% of the properties in the order.

Severity of Risk – High. This is the main service the contractor is being asked to perform and wrongful evaluations lead to a direct financial and procedural risk for the FHA.

Monitoring Procedure – Performed Per Order

· GTM obtains weekly order report deliverable for Property Valuation Services requested.

· GTM verifies that the valuations are related to the correct properties by ensuring valuations are not duplicates of other valuations within the same file or valuations are not blank.

· GTM verifies with customers that all valuations requested and received were complete and accurate

Corrective Action Steps – A disincentive of the cost per each AVM/AVE within the order request that does not meet the standard of completion or accuracy

Performance Analysis

· Desired Standard – 100% of AVM/AVEs ordered are complete and accurate

2. Element to be monitored: Contractor completed the order for AVMs on time.

Severity of Risk – High. Turnaround time on completion of these order is very important to the follow up actions for which these services are required.

· GTM logs when order was placed with Contractor

· GTM logs when deliverable was received from Contractor

· GTM verifies if deliverable was completed within the timeframes outlined in the contract

Corrective Action Steps – A disincentive of 5% in the reduction of the total order invoice for each business day the Contractor does not meet the performance requirement.

Performance Analysis

· Desired Standard – 100% of AVM/AVEs ordered are completed on time.

4. Element to be monitored: The Contractor corresponds to all order requesters in a timely fashion.

Severity of Risk – Medium. Timely correspondence mitigates procedural risks for FHA programs and program participants.

Monitoring Procedure – Performed Quarterly

· GTM logs customer complaints received about Contractor correspondence

· GTM investigates the credibility of customer complaints

· GTM calculates how many complaints were determined to be credible

Corrective Action Steps – Disincentive of 1% for the total amount invoiced for the quarter for every five (5) complaints verified by the GTM/GTR

Performance Analysis

· Desired Standard – Contractor returns all requests for information within one (1) business day for simple communication correspondence and two (2) business days for written or report deliverable correspondence.

5. Element to be monitored: The Contractor disseminates order deliverables to the right parties.

Severity of Risk – High. Ensuring the right information goes to the right parties protects against a major security risk for all stakeholder participants in our insurance programs.

· GTM logs customer complaints received about Contractor correspondence

· GTM investigates the credibility of customer complaints

· GTM calculates how many complaints were determined to be credible

Corrective Action Steps – A disincentive of 50% of the total order invoice for each order that is not disseminated to the correct party.

Performance Analysis

· Desired Standard – The deliverable for 100% of orders processed for valuation services is disseminated to the party that requested it.

6. Element to be monitored: The Contractor performs Quality Control of AVM models and orders for property valuation services as outlined in their Quality Control Plan

Severity of Risk – High. Quality controls on the AVM/AVE product and the contractor’s performance in using them to complete property valuation services prevents financial, security, and procedural risks.

Monitoring Procedure – Performed Quarterly

· GTM identifies random sample of 20% of AVM/AVE order requests, uploaded information, and case level property valuation data.

· GTM assesses how these services were performed in accordance to the Quality Control Plan submitted by the contractor.

· GTM identifies any discrepancies that the contractor’s QCP should have eliminated or discovered and logs the occurrence

· GTM calculates how many of these occurrences exists

Corrective Action Steps – A disincentive resulting in a refund of 10% of past total amount invoiced for the quarter in review for each instance where evidence of the Quality Control Plan not being followed is discovered.

Performance Analysis

· Desired Standard – The Contractor performs there outlined Quality Control measures on 100% of all work performed under this contract

PAGE