Attachment 2 QASP - HMTAP TSS_05142019.pdf

Federal Insurance and Mitigation Administration

Hazard Mitigation Technical Assistance Program (HMTAP)

TRAINING SERVICES

Quality Assurance Surveillance Plan

U.S Department of Homeland Security

Federal Emergency Management Agency

Federal Insurance and Mitigation Administration

Risk Reduction Division

500 C Street

Washington, DC

Procurement Sensitive Information - See FAR 3.104

Procurement Sensitive Information - See FAR 3.104, Page 2 of 24

TABLE OF CONTENTS

Federal Insurance and Mitigation Administration

1. INTRODUCTION

2. ROLES AND RESPONSIBILITIES

3. QUALITY CONTROL PLAN

4. METHODS OF SURVEILLANCE

5. ACCEPTABLE QUALITY LEVELS

6. QUALITY ASSURANCE DOCUMENTATION

7. ANALYSIS OF QUALITY ASSURANCE ASSESSMENT

8. INSPECTION OF DELIVERABLES

9. QUALITY ASSURANCE SURVEILLANCE PLAN

Appendix A: Performance Requirements Summary Table

Appendix B: Site Visit Quality Assurance Monitoring Form

Appendix C: Customer Service Feedback Investigation Form

Appendix D: Task Order Performance Evaluation Form

Appendix E: EMI Course Student Evaluation Form

Appendix F: Non-EMI Course Student Evaluation Form

Procurement Sensitive Information - See FAR 3.104, Page 3 of 24

1. INTRODUCTION

The Government’s Quality Assurance Surveillance Plan (QASP) is based on the premise that the service provider, and not the Government, is responsible for the day-to-day performance of the work and all the management and quality control actions required to meet the terms of the contract. The role of the Government in quality assurance is to ensure performance standards are achieved and maintained. The service provider is required to develop a comprehensive program of inspections and monitoring actions and to document its approach in a Quality Control Plan (QCP). The service provider’s QCP, upon approval by the Government, will be made a part of the resulting contractual agreement.

This QASP is designed to provide an effective surveillance method to monitor the service provider’s performance relative to the requirements listed in the agreement. The QASP illustrates the systematic method the Government (or its designated representative) will use to evaluate the services the service provider is required to furnish.

This QASP is based on the premise that the Government must validate that the service provider is complying with Government-mandated quality standards in all aspects of the services provided. Performance standards address all facets of operations, including quality; accuracy; completeness; timeliness; safety and health; conformance with statutory, regulatory, and policy guidance; appropriate use and care of government furnished property, information, or facilities; data and records management; etc. Good management by the service provider and use of an approved QCP will ensure that the services are provided within acceptable quality levels.

2. ROLES AND RESPONSIBILITIES

Contracting Officer The Contracting Officer (CO) is responsible for monitoring contract compliance, contract administration, and cost control and for resolving any differences between the observations documented by the Contracting Officer’s Representative (COR), the HMTAP Program Manager and the contractor. The CO will designate one full-time COR as the government authority for performance management. The number of additional representatives serving as project monitors and technical monitors depends on the complexity of the services measured, as well as the contractor’s performance, and will be identified and designated by the CO for each task/delivery order.

Assigned CO: Gary Topper Organization or Agency: FEMA Telephone: 301.447-7280 Email: gary.topper@fema.dhs.gov

Contracting Officer’s Representative The COR is designated in writing by the CO to act as his or her authorized mailto:gary.topper@fema.dhs.gov

Procurement Sensitive Information - See FAR 3.104, Page 4 of 24 representative to assist in administering a contract. COR responsibilities are contained in the written appointment letter. The COR is responsible for day to day administration of the program and ensures proper government surveillance of the contractor’s performance. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the government’s behalf.

Any changes discussed by the COR that the contractor or COR determines may affect contract price, terms, or conditions shall be referred to the CO for action. The COR will have the responsibility for completing QA monitoring forms used to document the inspection and evaluation of the contractor’s work performance.

Assigned COR: Tiffany Ryes Telephone: 202.212.5855 Email: Tiffany.Ryes@fema.dhs.gov

Other Key Government Personnel Title: Tanya Canady, Alternate COR Telephone: 202.646.3896 Email: Tanya.Canady@fema.dhs.gov

Title: Peter Fiess, Alternate COR Telephone: 202.227.9193 Email: Peter.Fiess@fema.dhs.gov

Title: Shabbar Saifee, Program Manager Telephone: 202.646.3142

Email: Shabbar.Saifee@fema.dhs.gov

3. QUALITY CONTROL PLAN

As a part of its agreement with the Government, the Contractor is required to develop, implement, and maintain a Quality Control Plan (QCP) that illustrates the methods it will use to review its performance to ensure it conforms to the performance requirements (Appendix A Performance Requirements Summary Table). Such reviews are performed by the service provider in order to validate its operations and assure the Government that the services meet the performance standards.

The contractor’s QCP should include monitoring methods that ensure and demonstrate its compliance with the performance standards. This includes inspection methods and schedules that are consistent with the regular reviews conducted by the government.

The reports and other results generated by the service provider’s QCP activities should be provided to the COR as requested.

The frequency and type of the service provider’s reviews should be consistent with what is necessary in order to ensure compliance with the performance standards, but no less frequent than what is described in the Government’s monitoring instrument/worksheets (See Appendix A).

mailto:Tiffany.Ryes@fema.dhs.gov mailto:Tanya.Canady@fema.dhs.gov mailto:Peter.Fiess@fema.dhs.gov mailto:Shabbar.Saifee@fema.dhs.gov

Procurement Sensitive Information - See FAR 3.104, Page 5 of 24

The service provider is encouraged not to limit its inspection to only the processes outlined in the Government’s standard; however, certain key documents must be provided by the contractor to assure the Government that the services meet the performance standards. Some of the documentation that must be generated and made available to the COR for inspection is listed below:

• QCP Reports

• Written policies and procedures to implement and assess operational requirements of the standard

• Documentation and recording keeping that ensure ongoing operational compliance with the standards (e.g.; monthly reports, quarterly reports, invoices, etc.)

• Staff training records

The list is intended as illustrative and is not all-inclusive. The service provider must develop and implement a program that addresses the specific requirement of each standard and the means it will use to document compliance.

4. METHODS OF SURVEILLANCE

In an effort to minimize the performance management burden, simplified surveillance methods shall be used by the government to evaluate contractor performance whenever possible. The primary methods of surveillance are:

4.1 Site Visits: Site visits can be scheduled or ad-hoc. Site visits are used to observe actual performance to determine the extent of compliance with performance standards, and to ensure any noted deficiencies are effectively addressed and corrected as quickly as possible.

Scheduled site visits will be performed no less than one (1) per contract period as funding allows. Scheduled site visits may involve direct observation of contractor personnel performing tasks, interacting with government staff or the public. Site visits may be performed by the COR or by other parties designated as representatives of FEMA. The COR or other designated representative of FEMA will record all findings on the Site Visit Quality Assurance Monitoring Form (Appendix B).

Ad-hoc site visits are unscheduled and will be conducted as a result of special interests arising from routine monitoring of the service provider’s QCP, an unusual occurrence pertaining to the agreement, or other FEMA concerns.

When visiting a site, either the COR or a designated representative of FEMA may conduct their own inspections of contractor performance activities, or accompany the service provider’s designated Quality Control Inspector (QCI) on scheduled inspections.

The COR and QCI may inspect the same event simultaneously but independently and compare results to determine if any surveillance areas were overlooked. The COR or other designated representative of FEMA will record all findings on the Site Visit Quality Assurance Monitoring Form (Appendix B). Site visit findings will be provided to the contractor personnel as appropriate.

Procurement Sensitive Information - See FAR 3.104, Page 6 of 24

4.2 Customer Feedback:

The contractor is expected to establish and maintain professional communication between its employees and customers. The primary objective of this communication is to determine the level of customer satisfaction. Customer satisfaction is the most significant external indicator of the success and effectiveness of all services provided and can be measured through customer compliments and/or complaints. The COR will accept those customer compliments and complaints and investigate using the Customer Service Feedback Investigation Form, identified in Appendix C.

Performance management drives the contractor to be customer focused through initially and internally addressing customer complaints and investigating the issues and/or problems. The customer always has the option to communicate complaints to the CO and COR, as opposed to the contractor.

Customer complaints, to be considered valid, must be set forth clearly and in writing with the detailed nature of the complaint, must be signed, and must be forwarded to the

COR.

4.3 Other Feedback:

For all activity types

The COR or other designated representative of FEMA will complete a performance evaluation as part of the closeout of each task order to ascertain the extent of compliance with the performance standards. The COR or other designated representative of FEMA will record all findings on the Task Order Performance Evaluation Form (Appendix D).

For Course Instruction

Following the instruction of each course students will complete course evaluations. If the course is designated as an Emergency Management Institute (EMI) course with an EMI course number the students will complete the most recent version of the EMI student evaluation form. An example of the current EMI student evaluation form is provided in Appendix E. If the course is not designated as an EMI course with an EMI course number the students will complete the student evaluation form in Appendix F.

The information contained in the student evaluation forms will be taken into consideration when the COR or other designated representative of FEMA completes the task order performance evaluation prior to close out of the task order.

5. ACCEPTABLE QUALITY LEVELS

The acceptable quality levels (AQLs) included in Appendix A, Performance Requirements Summary Table, for contractor performance are structured to allow the contractor to manage how the work is performed while providing negative incentives for

Procurement Sensitive Information - See FAR 3.104, Page 7 of 24 performance shortfalls. For certain critical activities that may be identified in a task/delivery order, the desired performance level maybe established at 100 percent.

Other levels of performance are keyed to the relative importance of the task to the overall mission performance as indicated in the task/delivery order.

6. QUALITY ASSURANCE DOCUMENTATION

6.1 Performance Management Feedback Loop

The performance management feedback loop begins with the communication of expected outcomes. Performance standards are expressed in the task/delivery order and are assessed using the methods of surveillance shown in Appendix A.

6.2 Monitoring Forms

The Government’s quality assurance surveillance will be reported using the specific monitoring tool identified in Section Four (4) of this QASP. The forms, when completed, will document the Government’s assessment of the contractor’s performance under the contract to ensure that the required results identified in the task order are being achieved.

7. ANALYSIS OF QUALITY ASSURANCE ASSESSMENT

7.1 Determining Performance

The Government shall use the monitoring methods cited herein to determine whether the performance standards/service levels/AQLs have been met. If the contractor has not met the minimum requirements, it may be asked to develop a process improvement plan or a corrective action plan to show how and by what date it intends to bring performance up to the required levels. Any mandatory deduction for performance requirements will be addressed in each individual task order (fixed-price only). In addition, failure to meet the AQL may result in a less than favorable rating in the Contract Performance Assessment Reporting System (CPARS) as indicated in Appendix A.

7.2 Reporting

At the end of each quarter, the Contractor shall prepare a written report for the COR and Project Manager summarizing the overall results of the quality assurance surveillance of the contractor’s performance. This written report, which includes any of the aforementioned completed quality assurance monitoring forms, will become part of the QA documentation. It will enable the government to demonstrate whether the contractor is meeting the stated objectives and/or performance standards, including cost/technical/scheduling objectives.

7.3 Reviews and Resolution

The CO, COR or Project Manager may require the contractor’s Program Manager, Task Order Manager, or a designated alternate, to meet with government personnel as deemed necessary to discuss the performance evaluation. The CO will define a

Procurement Sensitive Information - See FAR 3.104, Page 8 of 24 frequency of reviews with the contractor, including appropriate self-assessments by the contractor; however, if the need arises, the contractor will meet with the CO as often as required or per the contractor’s request. The agenda of the reviews may include:

• Monthly performance assessment data and trend analysis

• Issues and concerns of both parties

• Projected outlook for upcoming months and progress against expected trends, including a corrective action plan analysis

• Recommendations for improved efficiency and/or effectiveness

• Issues arising from the performance monitoring processes

The COR must coordinate and communicate with the contractor to resolve issues and concerns regarding marginal or unacceptable performance.

The CO and contractor should jointly formulate tactical and long-term courses of action.

Decisions regarding changes to metrics, thresholds, or service levels should be clearly documented. Changes to service levels, procedures, and metrics will be incorporated as a contract modification at the convenience of the CO.

8. INSPECTION OF DELIVERABLES

All deliverables will be inspected for content, completeness, accuracy, and conformance to requirements specified in the SOO. Unless specifically noted and mutually agreed upon, the Government may take up to 15 business days after receipt of the deliverable to inspect and review the deliverable, provide comments and feedback, and accept or reject the deliverable. If necessary, the contractor may take up to 15 business days to address the comments or feedback provided by the Government, unless specifically noted and mutually agreed upon.

9. QUALITY ASSURANCE SURVEILLANCE PLAN

The absence of any task order requirement from the QASP shall not detract from its enforceability to limit the rights or remedies of the government under any other provisions of the contract, including the clauses entitled “Inspection of Services” and “Default.”

The QASP is organized in accordance with:

• Desired Output: Lists the performance requirement that the government will survey.

• Performance Standard: Sets forth the required performance levels.

• Acceptable Quality Levels (AQLs): AQLs for Contractor performance is structured to allow the Contractor to manage how the work is performed. Levels

Procurement Sensitive Information - See FAR 3.104, Page 9 of 24 of performance are keyed to the relative importance of the task to the overall mission performance.

• Method of Surveillance: Sets forth the surveillance methods the Government will use to evaluate the contractor’s performance for the listed tasks and the level/frequency of surveillance.

• Performance Incentive/Disincentive: The contractor’s performance will be monitored in accordance with the QASP. Deficiencies in contractor performance will be fully and accurately documented to (1) identify the discrepancy, (2) record the steps taken to correct the discrepancy, and (3) detail the outcomes and disposition of the deficiency. Likewise, successful or exceptional performance will be recorded in the COR file and appropriate actions will be taken to recognize the contractor (i.e., Contractor Performance Assessment Reporting System

(CPARS).

Appendix A: Performance Requirements Summary Table

Procurement Sensitive Information - See FAR 3.104, Page 10 of 24

Required Service Performance

Standard Acceptance

Level Methods of

Surveillance Incentive

Administrative Performance Requirements

Administrative paperwork (such as monthly invoices, monthly reports, quarterly reports, task order closeout reports, etc…) complete, accurate, and timely

Administrative report accurately depict current status 100% of the time

98%

File reviews, periodic inspections, random observation, and Customer Satisfaction

Investigations

Less than favorable

CPARS

Rating

Administer quality control program including subcontractor management in accordance with Quality Control Plan

Contractor is in compliance with Quality Control Plan 100% of the time

98%

File reviews, periodic inspections, random observation, and Customer Satisfaction

Investigations

Less than favorable

CPARS

Rating

Contract Objectives Performance Requirements

Ensure a high quality of instruction for all course types and subjects

Quality assurance ratings have a satisfactory rating or higher 100% of the time

98%

Site Visits, Customer Feedback, Student Evaluations, and

Customer Satisfaction Investigations

Less than favorable

CPARS

Rating

Ensure quality course development that meets all required FEMA design requirements, as well as EMI and/or DHS 508 compliance requirements as established in each individual task order

SOO

Quality of product or service ratings have a satisfactory or better rating

100% of the time

98%

Performance Evaluation and

Customer Satisfaction Investigations

Less than favorable

CPARS

Rating

Ensure quality research information analysis and tool development that meets the breadth and depth of the need established in each individual task order SOO

Quality of product or service ratings have a satisfactory or better rating

100% of the time

98%

Performance Evaluation and

Customer Satisfaction Investigations

Less than favorable

CPARS

Rating

Contract Management Performance Requirements

Procurement Sensitive Information - See FAR 3.104, Page 11 of 24

Ensure satisfactory or better cost efficiency for all task orders

Quality assurance ratings have a satisfactory rating or higher 100% of the time

98% Performance

Evaluation Form

Less than favorable

CPARS

Rating

Ensure satisfactory or better business relations for all task orders and general contract activities

Quality assurance ratings have a satisfactory rating or higher 100% of the time

98% Performance

Evaluation Form

Less than favorable

CPARS

Rating

Ensure satisfactory or better key personnel management for all task orders and general contract activities

Quality assurance ratings have a satisfactory rating or higher 100% of the time

98% Performance

Evaluation Form

Less than favorable

CPARS

Rating

➢ NOTE: A satisfactory CPARS rating is considered favorable and higher ratings require better than favorable performance.

Appendix B: Site Visit Quality Assurance Monitoring Form

Procurement Sensitive Information - See FAR 3.104, Page 12 of 24

Observer:

Date:

Course Number:

Course Title:

Instructor Name:

Module Number:

Module Title:

Location:

Check the box that most closely indicates your level of agreement with each statement.

Use the scale of 1 (strongly disagree) through 5 (strongly agree). If you strongly disagree or strongly agree, please provide comments for the instructors so they can improve and enhance course effectiveness.

Evaluation Criteria

Strongly Disagree

2 3 4

Strongly Agree

Comments

Objectives

The session's objectives were clearly stated.

The students understood what the session activities and schedule would be.

Classroom Management

The instructor exercises appropriate time management.

The instructor could be clearly heard throughout the room.

The instructor promoted a respectful, positive, and encouraging learning environment.

Quality of Instruction

The instructor was knowledgeable on the subject.

The instructor demonstrated knowledge and understanding of cultural sensitivity.

The instructor communicated course information in an organized way.

Procurement Sensitive Information - See FAR 3.104, Page 13 of 24

Evaluation Criteria

Strongly Disagree

2 3 4

Strongly Agree

Comments

The instructor kept the students engaged.

The instruction was correctly paced to the level of the students' knowledge and abilities.

The instructor effectively used the Participant Manual as part of the learning process.

The instructor used visual aids effectively.

The instructor used examples effectively to highlight key points.

The instructor used guided questioning techniques to elicit student participation.

The instructor promoted participation from all students.

The instructor used a variety of teaching methods to reach students.

The students had enough time to think about and ask questions before the next topic.

The instructor clearly understood the student questions.

The instructor answered questions in a way that students could understand.

The instructor asked open-ended questions that check for conceptual understanding, analysis, and synthesis of subject matter.

The instructor responded appropriately when they didn’t know an answer to a question.

Appendix C: Customer Service Feedback Investigation Form

Procurement Sensitive Information - See FAR 3.104, Page 14 of 24

Memorandum for: _______, Contracting Officer

Memorandum From: __________, Contractor Officer’s Representative

Subject: Customer Service Feedback Investigation

General Information

1. Contractor:

2. Contract Number:

3. Task Order Number:

4. Project Title:

5. Evaluation Date:

6. Government Evaluator Name:

7. Government Evaluator Title:

Customer Service Feedback Information

8. Name of Customer:

9. Date of Customer Service Feedback:

10. Customer Service Feedback specifics (Describe in Detail and reference task order proposal…Include any backup documentation provided by complainant.):

Contractor Response to Customer Service Feedback

11. Narrative of Contractor Response:

Government Analysis of Customer Service Feedback

12. Is Government surveillance investigation necessary? _______ (If so, complete questions 13 – 17.)

13. Surveillance Type:

14. Date of Surveillance:

15. Narrative of Performance during Survey Period:

16. Government Evaluation of Contractor Response/Resolution:

17. Recommended Government Action:

By signing below, the COR confirms the information contained herein for the Contracting Officer.

Procurement Sensitive Information - See FAR 3.104, Page 15 of 24

COR Signature Date

Printed Name – FEMA COR

Appendix D: Task Order Performance Evaluation Form

Procurement Sensitive Information - See FAR 3.104, Page 16 of 24

HMTAP

PERFORMANCE EVALUATION

Contractor

Contract Number Project Title

Task Order Number Not Applicable Evaluation Date

FEMA COR

FEMA PM

Contractor TOM Period of Performance through start date end date

PURPOSE OF PROJECT (This should be taken from SOO):

* All ratings must include supporting comments.

Upon opening this document you will need to disable the security warning by clicking on the OPTIONS button at the top of the document. When the Security Alert-Macro Warning comes up, select the Help protect me from unknown content (recommended) option. This will allow you to make changes to the document.

To complete this form, fill out each of the five following evaluation comments and the rating recommendations section.

For each evaluation factor, select the appropriate rating from the drop-down menu in the left column. In the right column provide supporting comments that substantiate that rating. The evaluation comments must directly apply to the performance requirements in the agreed upon proposal. These comments are required in order for the evaluation to be accepted for review by the Contracting Officers Representative.

QUALITY OF PRODUCT OR SERVICE

(use below requirements guidelines for quality rating)

Exceptional No quality issues; Performance is “Outstanding”; Deliverables are precedent-setting; Reports are concise, thorough, and easy to understand; Deliverables document innovative or creative approaches; The explanation must include an example of a significant event that lead to the rating; No weaknesses were noted.

Very Good Minimal quality issues; More than one performance work statement revision required; Technical analysis incomplete; A few recommendations not accepted;

Reports adequately written, need minimal rework; Data accurate; The explanation must include an example of what was done to meet the requirements and what was done to exceed them in some “minor” way; No significant weaknesses.

Procurement Sensitive Information - See FAR 3.104, Page 17 of 24

Satisfactory Meets the requirements of the Performance Work Statement; If a correction was required, the explanation should include examples of what was done that needed correction and that the correction was done.

Marginal Numerous quality issues; Technical Analysis marginally effective but not fully implemented; Recommendations marginally effective; Reports marginally effective; Some data flawed; The explanation must include what was done poorly or incorrectly.

Unsatisfactory Serious quality issues; Technical analysis incomplete; Recommendations ineffective; Reports not usable; Some data flawed; Reports poorly written, not usable, and ineffective; The explanation must include an example of what was not done correctly and that it was not corrected or repeated; The explanation must state what magnitude of the contractor’s failure to the government was addressed.

Quality of Product or Service Evaluation Comments

(select a rating)

COST CONTROL/EFFICIENCY

(use below requirements guidelines for quality rating)

Exceptional No cost issues; Work assignment completed within budget in spite of extraordinary circumstances; Significant cost avoidance measures implemented with excellent documentation; Cost avoidance measures exemplary; The explanation must include an example of a significant event that lead to the rating;

No weaknesses were noted.

Very Good Rare cost issues; Rare failure to provide satisfactory documentation for cost increase; Cost avoidance efforts are demonstrated; The explanation must include an example of what was done to meet the requirements and what was done to exceed them in some “minor” way; No significant weaknesses.

Satisfactory Met the requirements of the agreed too cost schedule; If a correction was required, the explanation should include examples of what was done that needed correction and that the correction was done.

Marginal Numerous cost issues; Costs not adequately managed; Management of costs ineffective; Failure to provide satisfactory documentation for cost increases; The explanation must include what was done poorly or incorrectly.

Unsatisfactory Cost issues; Failure to manage costs; Budgets exceeded without adequate warning or justification; Failure to provide documentation for cost increases;

Requirements for additional funding to complete the project; The explanation must include an example of what was not done correctly and that it was not corrected or repeated; The explanation must state what magnitude of the contractor’s failure to the government was addressed.

Neutral Rating of neutral is used for fixed price projects.

Cost Control/Efficiency Evaluation Comments

SCHEDULE

Procurement Sensitive Information - See FAR 3.104, Page 18 of 24

Exceptional No timeliness issues; All deliverables submitted ahead of schedule; Precedent-setting creative and innovation means to shorten timeframes; Proactive in identifying issues and in responding to project demands; The explanation must include an example of a significant event that lead to the rating; No weaknesses were noted.

Very Good Rare timeliness issues; All deliverables are submitted on time; Adequate documentation for requests to extend due dates; Efforts taken to reduce time well documented; Consistent response to issues and requests; The explanation must include an example of what was done to meet the requirements and what was done to exceed them in some “minor” way; No significant weaknesses.

Satisfactory Meets the Schedule agreed upon in the Performance Work Statement; If a correction was required, the explanation should include examples of what was done that needed correction and that the correction was done.

Marginal Numerous schedule issues; Slippage of schedule without justification or warning;

Frequent failure to provide satisfactory documentation for requests to extend due dates; The explanation must include what was done poorly or incorrectly.

Unsatisfactory Serious timeliness issues; Frequent slippage of schedule without adequate warning or justification; Frequent requests to extend due dates as a result of contractor caused issues; Failure to respond to issues and requests; The explanation must include an example of what was not done correctly and that it was not corrected or repeated; The explanation must state what magnitude of the

Schedule Evaluation Comments

BUSINESS RELATIONS (Contract Requirements)

Exceptional No business relations issues; Excellent balance of staff; Proactive communications;

Identify solutions as problems are identified; Outstanding selection and management of staff and subcontractors; Exceptional partnering with HMTAP partners; The explanation must include an example of a significant event that lead to the rating; No weaknesses were noted.

Very Good Rare business relations issues; Appropriate assignment of personnel; Successful identification and resolution of issues; Maintenance of regular communication;

Responsive and thorough dealings with all aspects of the program; Effective monitoring of subcontractors; Effective partnering with HMTAP partners; The explanation must include an example of what was done to meet the requirements and what was done to exceed them in some “minor” way; No significant weaknesses.

Satisfactory Met contract requirements; If a correction was required, the explanation should include examples of what was done that needed correction and that the correction was done.

Marginal Numerous negative business relation issues; Assignment of Marginally effective;

Utilization of resources marginally effective; Communication protocol not fully implemented; Implementation of contract requirements marginally effective;

Management of subcontractors marginally effective; The explanation must include what was done poorly or incorrectly.

Procurement Sensitive Information - See FAR 3.104, Page 19 of 24

Unsatisfactory Significant negative business relations issues; Failure to assign appropriate personnel; Poor utilization of resources; Failure to maintain regular communication; Implementation of contract requirements ineffective; Failure to manage subcontractors; Failure to partner with HMTAP partners; The explanation must include an example of what was not done correctly and that it was not corrected or repeated; The explanation must state what magnitude of the

Business Relations Evaluation Comments

MANAGEMENT OF KEY PERSONNEL

Excellent No management of key personnel issues; Contractor performed excellent in selecting, retaining, supporting, and when necessary, replacing, key personnel;

key staff exceeded the required contract qualifications for the position; if a key person did not perform well, the contractor quickly corrected this; if a replacement of key person was necessary, the replacement exceeded the required contract qualifications; The explanation must include an example of a significant event that lead to the rating; No weaknesses were noted.

Very Good Rare management of key personnel issues; Contractor performed satisfactory in selecting, retaining, supporting, and when necessary, replacing key personnel;

key staff met the required contract qualifications for the position; if the key person did not perform well, the contractor corrected this; if a replacement of key person was necessary, the replacement met the required contract qualifications;

The explanation must include an example of what was done to meet the requirements and what was done to exceed them in some “minor” way; No significant weaknesses.

Satisfactory Key personnel defined in project work statement executed the work or if a replacement was required they fully met the contract requirements; If a correction was required, the explanation should include examples of what was done that needed correction and that the correction was done.

Marginal Numerous management issues of key personnel; Contractor had numerous issues with selecting, retaining, supporting, and when necessary replacing key personnel;

If a replacement of key personnel was required they marginally met the contract requirements; The explanation must include what was done poorly or incorrectly.

Unsatisfactory Serious management of key personnel issues; Agreed upon key personnel did not perform tasks; Contractor failed in selecting, retaining, supporting, and when necessary, replacing, key personnel; key staff did not met the required contract qualifications for the position; if the key person did not perform well, the contractor did not correct this; if a replacement of key person was necessary, the replacement did not meet the required contract qualifications; The explanation must include an example of what was not done correctly and that it was not corrected or repeated; The explanation must state what magnitude of the

Management of Key Personnel Evaluation Comments

Procurement Sensitive Information - See FAR 3.104, Page 20 of 24

CONSIDERATIONS AND RECOMMENDATIONS FOR FUTURE PROJECTS OF THIS TYPE

Were “Official” corrective actions required?* (If yes, corrective action documents must be attached)

Yes if yes, continue to answer the questions below.

No

Appendix E: EMI Course Student Evaluation Form

Procurement Sensitive Information - See FAR 3.104, Page 21 of 24

Procurement Sensitive Information - See FAR 3.104, Page 22 of 24

Appendix F: Non-EMI Course Student Evaluation Form

Procurement Sensitive Information - See FAR 3.104, Page 23 of 24

Observer:

Course Number:

Course Title:

Instructor Name:

Location:

Check the box that most closely indicates your level of agreement with each statement.

Use the scale of 1 (completely disagree) through 5 (agree). If you strongly agree or strongly disagree, please provide comments for the instructors so they can improve and enhance course effectiveness.

Evaluation Criteria

Strongly Disagree

2 3 4

Strongly Agree

Comments

Objectives

The session's objectives were clearly stated.

The objectives of the course were met.

The course improved my understanding of the topic.

Organization of the Course

The material was presented in a clear manner that facilitated understanding.

Course exercises were well placed and added to student understanding of course concepts.

The pace of the course was appropriate.

The course had a logical flow.

Participation/Discussion

The course provided ample opportunities for participation.

There was an appropriate amount of discussion and student involvement.

Procurement Sensitive Information - See FAR 3.104, Page 24 of 24

Evaluation Criteria

Strongly Disagree

2 3 4

Strongly Agree

Comments

The instructor promoted participation from all students.

Course Materials

Course material was up-to-date, well-organized, and presented in sufficient depth.

Audio-visual materials used were relevant and of high quality.

Handout materials enhanced course content.

Quality of Instruction

The instructor was knowledgeable on the subject.

The instructor communicated course information in an organized way.

The instructor kept the students engaged.

The instructor responded to student questions in a way the student could easily understand.

Please answer the following questions:

Would you recommend this course to others? Yes _____ No_____

Why or why not?

What skills or knowledge did you acquire as a result of attending this course?

What are the strengths of the course?

What, within the course materials, could be improved?