Attachment 6 - Quality Assurance Surveillance Plan (QASP).pdf

FORM # USPTO-OP-014.v1.0 (06/2015) 1

Training and Instructional Support Services

Quality Assurance Surveillance Plan

(QASP)

Attachment 6

1/16/2025

FORM # USPTO-OP-014.v1.0 (06/2015) 2

Table of Contents

1. INTRODUCTION

2. PURPOSE

3. SCOPE

4. CONTENT OF THE QASP

5. SUMMARY OF WORK TO BE PERFORMED

6. QASP ROLES AND RESPONSIBILITIES

6.1 USPTO Personnel

6.2 Contractor Personnel

7. QUALITY SURVEILLANCE PROCESS

7.1 Methods

8. STEPS IN THE SURVEILLANCE PROCESS

8.1 Scheduling Surveillance

8.2 Performing Surveillance

8.3 Resolving Performance Issues

8.4 Documenting Surveillance

9. SURVEILLANCE STANDARDS AND TOOLS

9.1 Quality Assurance Standards

9.2 Tools

10. OTHER QA COMMUNICATION AND REPORTING . ERROR! BOOKMARK NOT

DEFINED.

11. RESOLVING QUALITY ASSURANCE ISSUES ............ ERROR! BOOKMARK NOT

DEFINED.

11.1 Notification

11.2 Response

12. PERFORMANCE REQUIREMENTS SUMMARY

FORM # USPTO-OP-014.v1.0 (06/2015) 3

1. INTRODUCTION

The USPTO has the requirement for training and instructional support services designed to advance and enhance the provision of information and automation instructional support to Patent employees both within a virtual environment setting and through direct, in-person engagement.

There is also a requirement for contractor support for digital media services to assist with the development of computer and web-based training modules and all aspects of digital video production. The USPTO utilizes contractor services to furnish these specialized instructional and training support services to enhance operational efficiency and knowledge dissemination.

All work performed under this contract shall be subject to surveillance to ensure all services rendered and deliverables are of the highest quality and acceptable to the government.

2. PURPOSE

This QASP sets forth standard surveillance for monitoring the performance based contract and guidelines that the USPTO will use in evaluating the technical performance of the contractor.

A copy of this plan will be furnished to the contractor so that the contractor will be aware of the methods that the Government will employ in evaluating performance on this contract and address any concerns that the contractor may have prior to initiating work. The Surveillance/Evaluation Methods identified in this QASP, in concert with the contractor’s Quality Control Plan (QCP), will assure the Government of satisfactory contractor performance.

The QASP is intended to accomplish the following:

Summarize the types of services to be performed and work products to be delivered that are covered under the contract and the desired outcomes associated with them.

Define the specific contractor and government responsibilities for evaluating the performance of the contractor in delivering the products and services covered by this contract.

Describe the surveillance or evaluation methods to be used by the government to confirm the contractor’s performance.

Provide initial surveillance draft formats of the reports to be used to evaluate and report on the contractor’s performance.

Describe the sequence and timing of the performance reporting process.

3. SCOPE

The QASP shall be utilized as the tool for use in Government administration of the contract and remains subject to revisions at any time by the USPTO throughout the contract performance period. USPTO will retain the right to change the surveillance methods, metrics, and Quality Assurance (QA) procedures described in the QASP, or to increase or decrease the degree of surveillance efforts at any time necessary to assure contract compliance. The contractor, and not is responsible for management and Quality Control (QC) actions to meet the terms of the contract.

FORM # USPTO-OP-014.v1.0 (06/2015) 4

4. CONTENT OF THE QASP

The purpose and content of the QASP is identified below:

Providing a summary of the work to be performed on this contract (Section 5) Ensuring that QASP roles and responsibilities are clear for this contract (Section 6) Defining QA surveillance methods to be applied in this contract (Section 7) Defining the standards and tools and that will be applied to the contractor’s QA activities on this contract (Section 9) Communications (Section 10) Resolving QA issues and concerns specific to this contract (Section 11) Provide the Performance Requirements Summary that ties requirements to the methods and outcomes (Section 12)

5. SUMMARY OF WORK TO BE PERFORMED

In support of the USPTO Automation Training Services Division (ATS), the contractor shall administer training and helpdesk support to all USPTO patent examiners on the utilization of mission-critical patent automation, desktop and search tools. The majority of this training shall be administered virtually but there is a requirement for limited onsite staff at the USPTO Alexandria headquarters and other USPTO offices as identified in the PWS. To successfully support the mission of ATS, the contractor shall administer training so that new patent examiners become proficient with all search and collaboration tools. The contractor shall provide helpdesk support in accordance with the PWS. The USPTO shall utilize the criteria listed in the Performance Requirements Summary (PRS) and scoring sheets to access the contractor’s performance. All terms of the PWS apply to support ATS.

The contractor shall also provide all support as identified in the PWS to support the USPTO Digital Media Serivces (DMS). The contractor shall provide support to DMS for the development of all aspects and professional contract and eLearing modules, including computer-based training (CBT) and web-based training (WBT). The contractor shall be fluent with a number of rapid eLearning authoring tools as identified in the PWS. The contractor shall be fluent utilizing design software and web design software as identified in the PWS. All deliverables shall be 508 compliant. The USPTO shall utilize the criteria listed in the Performance Requirements Summary (PRS) and scoring sheets to access the contractor’s performance. All terms of the PWS apply to support DMS.

6. QASP ROLES AND RESPONSIBILITIES

This section details the Roles and Responsibilities for successful execution of the QASP. This includes both USPTO’s and the contractor’s personnel. Roles and responsibilities are listed below.

FORM # USPTO-OP-014.v1.0 (06/2015) 5

6.1 USPTO Personnel

Contracting Officer (CO)

The Contracting Officer (CO) has overall responsibility for the administration of all USPTO contracts and task orders (TOs). The USPTO CO is part of the USPTO management team and is the only individual authorized to take actions on behalf of the Government to approve, amend, modify, or deviate from the contract terms, conditions, requirements, specifications, details, and/or delivery schedules. Within that authority, the CO has the ability to delegate to the Contracting Officer’s Representative (COR) for the everyday administration of the contract. The CO, who is responsible for the overall administration and final closeout of the contracts, and as it relates to the implementation of the QASP, shall:

Ensure compliance with contracting requirements Issue all modifications against the contracts Negotiate and issue contract modifications Resolve contractor claims and disputes Issue cure notices (notification that unless unacceptable performance is corrected, the

Government may terminate the contract for default, IAW FAR 49.607) Issue show-cause letters (following a cure notice, requesting facts bearing on the case) Terminate contracts

Contracting Officer’s Representative (COR)

The USPTO COR is responsible for the day-to-day coordination of the USPTO contract. The USPTO COR is part of the USPTO management team. The COR does not have any authority to make any contractual changes the will increase cost, schedule or scope.

In implementing the QASP, the COR responsibilities include managing Quality Assurance (QA) activities to ensure compliance and completeness by:

Overseeing the quality aspects of the project Serving as the technical liaison with the Contractor Quality Control (QC) staff Responsible for the final inspection and acceptance of all reports Other such other responsibilities as may be specified in the contract.

Program Office Subject Matter Expert (SME)/Government POC

The government shall designate a subject matter expert and/or government monitor to perform the inspections and surveillance of all training materials to ensure they are in compliance with all performance standards.

Conducting in person performance reviews of training Serves as the SME/Government POC to report to the COR that services rendered, and deliverables are acceptable Serves as the point of contact for all day-to-day operations and communications Reviews all reporting to ensure the information is accurate and acceptable. Requests all other reporting as required.

FORM # USPTO-OP-014.v1.0 (06/2015) 6

6.2 Contractor Personnel

Program Manager

The program manager shall oversee all day-to-day operations to provide oversite, supervision and technical direction to all contractor subordinates. This is the primary POC for communication between the contractors and SME/COR.

Adhere to the Quality Control Plan and to ensure all services rendered are being performed successfully on a day-to-day basis

Performs Quality Control and makes immediate corrections if services are unacceptable Provide metrics to the COR/SME Respond to discrepancy reports Meet with the USPTO SME/Government POC/COR/ACOR on a monthly basis to review contract quality Provide all meeting minutes to the COR within three (3) business days

7. QUALITY SURVEILLANCE PROCESS

This section describes the various surveillance processes that may be used on this contract. The generic surveillance process should be adapted to the specific requirements of the contract.

Additional procedures may be required and will be added if necessary, during the life of the contract.

7.1 Methods

The SME/COR will perform evaluations using as many of the various methods applicable and outlined in the PRS table included in the QASP. The following methods and techniques may be appropriate for the contracts and are provided as examples and may be incorporated into the PRS to monitor contract performance. Each QASP must select and specify the appropriate methods that will be utilized on the contract.

100% Inspection

This method is an inspection where specific characteristics of every performance unit are examined and tested to determine conformance with requirements. This surveillance method is preferred for tasks that occur infrequently, as well as for tasks that cannot be randomly inspected.

Random Surveillance

Random Surveillance is a sampling method in which each unit of the population has an equal chance of being selected. This method is done to determine whether the contractor’s performance of a particular task for a given period of time meets the performance requirement based on randomly selected samples of the task. This method is the most appropriate for frequently recurring tasks.

Periodic Surveillance

Evaluates tasks selected using a methodology other than a 100% or random inspection. It may be appropriate for tasks that occur infrequently and where 100% inspection is neither required nor practical. An example of periodic surveillance is weekly inspections when the

FORM # USPTO-OP-014.v1.0 (06/2015) 7

COR/Government POC chooses the location and time in other than a statistically random manner. This is not a preferred method of surveillance but may provide a basis for an incentive payment to the contractor.

Customer Feedback

This method is an independent evaluation of an activity or process to assess compliance with the contract’s terms of the contract. Customer Feedback may be necessary for certain types of tasks that do not lend themselves to random sampling or 100% inspection. Customer feedback may be obtained from the results of formal customer satisfaction surveys or from random customer complaints. It may provide the basis for an incentive payment to the contractor, but it cannot be used on the basis of deduction from payment since it is not a statistically valid method of surveillance. The CO may use validated customer complaints as the basis for actions (other than payment deductions) against the contractor. For a Customer Feedback form see section 10.

8. STEPS IN THE SURVEILLANCE PROCESS

Actual surveillance must be done and shall implement the methods identified in section 7.1 to include scheduling, performing, resolving performance issues discovered through surveillance, and documenting surveillance. Steps in the surveillance process are identified below. Due to the nature of the this contract the SME or COR could attend any training session or observe any service at any time without prior notification to the contractor.

8.1 Scheduling Surveillance (if applicable)

The SME/Government POC assigned to this contract is responsible for developing a monthly (or perhaps more frequent) schedule of surveillance activities based on the QASP’s requirements.

The schedule should be completed by a specific date, e.g., no later than 7 calendar days before the beginning of the period it covers. Computer-generated or locally devised forms may be used.

8.2 Performing Surveillance

Surveillance shall be performed according the methods described in this QASP and meeting the schedule established between the contractor and the organization’s personnel. The Contractor shall provide whatever assistance is required in the surveillance procedures.

8.3 Resolving Performance Issues

It is not unprecedented that USPTO and the contractor may discover performance issues or deficiencies through the surveillance process. The SME/COR will provide a list of any unresolved deficiencies to both the USPTO CO and contractor's Quality Assurance personnel to afford him/her the opportunity to show where the problem is already in the process of being worked. If evidence is shown where the contractor’s quality program has already documented the deficiency and timely corrective action is resolving the problem, the COR/ACOR will follow up with the contractor’s quality assurance personnel to ensure the deficiencies are remedied in a timely manner, in accordance with the PWS.

All deficiencies associated with performance requirements listed on the Performance Requirements Summary, whether remedied by the contractor or not, will be included in the SME/CORs Status Report regardless of who identified the deficiency.

FORM # USPTO-OP-014.v1.0 (06/2015) 8

The SME/Government POC/COR/ACOR and the Contractor QA Personnel will meet monthly or as needed to discuss contractor performance. The QA Personnel will provide a copy of the meeting minutes to all attendees within five 5 business of the meeting. Should the need arise for an emergency meeting because of unexpected performance issues, these meetings can be called by either COR/ACOR or the contractor’s quality personnel.

8.4 Documenting Surveillance

Documentation used and referenced to perform surveillance shall consist of monthly reports, contractor plans and procedures, schedules, customer feedback and contract data requirements.

This documentation provides the CO with contractor status as it applies to the performance criteria.

All documentation resulting from surveillance must be made part of the contract file. The COR must keep the documentation files during the surveillance period, but either monthly or at the conclusion of the contract, as directed by the CO, the COR/ACOR must give the files to the CO for inclusion in the official contract file.

Surveillance Folder

A surveillance folder must be developed and maintained by SME/COR who is assigned to accomplish QA for a performance requirement. The folder is contained electronically within the contract folder.

Appointments. COR/ACOR nomination letters and copy of the CO’s Letter to the Contractor appointing the new COR/ACOR to the contract.

Contract Documents. The Contract with all modifications, directive documents, references, pertinent terms defined by the contract, equipment listings, PWS and QASP.

Surveillance Paperwork. Documented surveillance inspections performed by the SME/COR will be kept in this section. All documentation that supports the surveillance should be kept in this section as well. The types of information that may be included are details of inspections or data gathering, conversations or meetings with the contractor, notes and comments that support the inspection paperwork. Quality Assurance Surveillance Reports will be kept in this section.

Unacceptable Performance Documentation. In accordance with FAR 49.607, a section used for filing all documentation associated with contract quality assurance, e.g., Customer Complaints (both active and resolved), notices of contract deficiencies, Customer Discrepancy Report, the CO’s inputs/determinations, Cure Notices, Show Cause, and all supporting documentation.

9. SURVEILLANCE STANDARDS AND TOOLS

The COR/Government POC will assess the Contractor’s performance to ensure the Contractor is performing up to the specified standards. The performance standards are outlined in the Performance Requirements Summary (PRS) table included in this QASP.

FORM # USPTO-OP-014.v1.0 (06/2015) 9

9.1 Quality Assurance Standards

Unless otherwise stated in the PRS table below, all services rendered and deliverables shall conform with a 95% rate of acceptability.

9.2 Tools

Visual surveillance will be conducted by the SME/COR and contractor program management.

For the development and deployment of any computer or web-based training/instructional services digital programming commercially available tools may be used to ensure acceptability if quality concerns are present. Customer feedback to include surveys, emails or phone calls from USPTO personnel may be utilized to determine the level of performance as well.

10. COMMUNICATION AND REPORTING

All quality issues that may arise during the execution of the contract will be managed through the resolution methods defined in the QASP.

11. RESOLVING QUALITY ASSURRANCE ISSUES

This section identifies the resolution process if quality assurance issues arise.

11.1 Notification

The COR/ACOR will notify the Contracting Officer, in writing, of unacceptable quality levels.

The Contracting Officer will promptly provide written notification of discrepancies on a monthly basis, or immediately if the situation significantly affects the well-being of the project. The Contractor will be given the opportunity to respond, in writing, to each discrepancy.

11.2 Response

The Contractor shall provide a written response to the cause and the corrective action to prevent recurrence within ten (10) business days after receipt of Contracting Officer’s notification. The Contractor shall cite specific quality assurance program procedures or new procedures instituted to prevent recurrence.

12. PERFORMANCE REQUIREMENTS SUMMARY

The Performance Requirements Summary (PRS) is the culmination of quality planning for each specific task based on the PWS and sureveillance activities as summarzied in the table below.

Task

Performance Objective

Performance Standard

Acceptable Quality Level (AQL) (All need a percentage)

Surveillance Method

Training Services Support Instructional Content ATS

Quality of Instructional Content

Instructional Content adheres to the Analyze, Design, Develop, Implement and Evaluate (ADDIE) process to ensure high quality training materials (training needs analysis, training plans, lesson plans, and training materials, knowledge checks, hands-on/practical exercises).

98% of the materials reviewed meet the stated requirements.

Random surveillance

Scheduled inspections

Customer Feedback

Deliver Training

ATS

Quality and Efficiency

Formal and informal instruction and materials are delivered in a professional manner that adhere to industry standard training methods and the required training schedule

98% of the time, courses start and end on time; delivery issues are reported within five (5) business days, unless otherwise stated.

Random surveillance

Scheduled inspections

Customer Feedback

Deliver Training

ATS

Quality and Efficiency

Learners can utilize principles learned in training sessions and apply after the training is received.

98% of the time Random Surveillance

Continuity in Training ATS

Quality and Efficiency

Contractor staff are trained and proficient with USPTO’s custom software/hardware systems so that it is completed within four (4) months following the person’s first day of hire.

99% of the time 100% Inspection

Customer Feedback

Student/Customer Support ATS

Quality of training support

Training support - Ensures accurate LMS support (entering sessions, creating WebEx links and pulling WebEx attendance reports).

Ensures professional OPT admin support (in-person classroom and virtual instructional)

98% of the time, remedial action is taken within five (5) business days for any training material that fails the quality review.

Random surveillance

Scheduled inspections

Customer Feedback

ITRP Assists

ATS

Quality and Efficiency of Training Assists

Ensures timely and efficient support: 8:00am-8:00pm Chatroom and ITRP Hotline.

95% of the time, positive customer feedback is received.

Random surveillance

Training Evaluation

Quality and Efficiency of contractor’s multi-faceted training evaluation program

Evaluation program ensures all instructors are and remain knowledgeable of USPTO automation tools and training content per USPTO standards.

98% of the time, remedial action is taken within five (5) business days for any training material that fails the quality review.

Random surveillance

Scheduled inspections

Customer Feedback

Training Support Services Instructional Content DMS

1. Quality of Instructional Content

2. Quality of Grammar and Typography

3. Quality of Online Content Visuals

4. Quality of Audio

1. Instructional Content – Instructional theory is utilized to ensure the quality of instruction/collaboration with DMS and customers as required. Lesson sequencing, planning/storyboarding is of high quality prior to deployment.

2. Fonts, grammar rules, spellings, and consistency in brand styles are accurate before deployment.

3. Visuals of the online content

– Graphics and Images – Assessment related to effective use of graphics and visuals to enhance the experience of eLearning content are of high quality.

4. Audio, voice-over, no external noises and distracting sounds, proper cadence and pronunciation of words in the script, music, sound effects, and syncing are all tested and operable prior to deployment.

The Quality of all instructional content is adhered to 98% of the time.

Random Surveillance

Training Support Services Course Development

DMS

1. Operability and Efficiency of Course Formatting

2. Operability of Course Navigation

3. Operability of Interactivity

4. Quality of Quiz and Assessment

1.Documents, line, breaks, transcripts, sub-titles, white space, mobile view responsive are accurately displayed.

2.Navigation of course content, best practices for navigation – tabs, UI elements are tested and operable before deployment.

3. The interaction of various features and functions within the course are tested and operable before deployment.

Immersive qualities and functionality are tested and operable.

4. The quality and operability of knowledge checks, quizzes and scoring users, and all functions are all tested and operable before deployment.

The operability, efficiency and quality is acceptable 98% of the time.

Random Surveillance

5. Utilization of Technology

5. Cross-browser compatibility, LMS (Learning Center) compatibility, SCORM 2004 compliance and Web compatibility courses are all tested and operable before deployment.

Training Support Services Accessibility

ATS/DMS

Accessibility All training and materials are compliant with section 508 for individuals with a disability.

Section 508 standards are adhered to 100% of the time.

100% Inspection

Reporting and Deliverables

ATS/DMS

Quality and Efficiency

Complete and deliver all reporting requirements per the

PWS.

98% of the time 100% Inspection

Assists Observations Checklist Form

OITP

Office Of Information Technology for Patents

Contract No.:

XXXXXXXXXXX

Contract Requirement Contract Observer Method of Surveillance Date Accomplished Compliance

Customer Greeted ATS Federal Random surveillance 2/19/14 – 9am Yes

Customer Helped/Assisted ATS Federal Random surveillance 2/19/14 – 11am Yes

Customer Referred ATS Federal Random surveillance 2/19/14 – 4pm No. Referral not offered for follow-up.

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

[Contract requirement]

Contract Discrepancy Report Sample

CONTRACT DISCREPANCY REPORT (CDR)

1. Contract No.: XXXXXX 2. Report No. for this Discrepency: XXX

3. To: (Contractor and Manager’s Name)

4. From: (Name of COR/SME/Government

POC)

5. Date Prepared: XX/XX/XXXX Date Returned by Contractor:

XX/XX/XXXX

Date Action Completed: XX/XX/XXXX

6. Discrepency or Problem: (Describe in detail; include refernec to PWS Directive; attach continuation sheet if necessary)

There have been 20 unscheduled Helpdesk requirements during this quarterly surveillance period (Jan – Mar). On four of the requirements, the contractor did not responds withing four minutes of the agree time as required by 5.2.2 and the PRS item

3. Of all four occasions, the service was delivered over 5 hours late. The performance requirement is 85%, therefore the requirement was performed unsatisfactorily since only 3 noncompliance events are allowed.

7. Signature of Contrating Officer:

8. To: (Contracting Officer) From: (Contractor)

9. Contractor response as to Cause, Corrective Action, and Actions to Prevent recurrence;

Attach continuation sheet if necessary. (Cite applicable quality control program procedures or new procedures)

10. Signature of Contractor Representative:

Date:

11. Government Evaluation (Acceptance, partial acceptance, rejection; attach continuation sheet if necessary)

12. Government Action (Reduced payment, cure notice, show cause, other)

CLOSEOUT

Name – Title Signature Date

Contractor Notified

7. Instructor responds appropriately to students' questions, if asked. 3 N/A

8. Instructor actively monitors student learning throughout the class time. 3 N/A

9. Instructor demonstrates proficiency with instructional technology software (ex. PowerPoint, Windows) and hardware (ex. dual monitors, projectors).

4 N/A

Instruction Content Weight Pass Fail N/A Weighted Score

10. Instructor demonstrates proficiency in all key aspects of the subject matter.

4 N/A

11. Content is presented in a logical sequence or referenced at the appropriate time.

4 N/A

12. Instructor provides clear and concise directions for students to follow. 4 N/A

Class Conclusion Weight Pass Fail N/A Weighted Score

13. Instructor reviews accomplishment of stated course objectives with students.

1 N/A

14. Instructor reminds students how to contact staff for further assistance.

1 N/A

Comments:

Comments:

Comments:

Comments:

Comments:

Comments:

Comments:

15. Instructor ends the class on time. 2 N/A

Observation Scoring Pass Fail N/A

Weighted Score per category 0 0 36

Weighted Passing Percentage*

*For ratings of Not Applicable (N/A), the score on the form for that rating is not factored into the calculation of the overall percentage.

Passing 90% or higher

Additional Comments:

Total Points (36 possible, based on 15 Factors. N/A points deducted)