Quality-Assurance-Surveillance-Plan_4-4-2018.pdf

Museum Specialist Services Quality Assurance Surveillance Plan (QASP) April 4 2018

The Exhibition Program at the National Library of Medicine, part of the National Institutes of Health, produces stories about the social and cultural history of science, medicine, and technology for audiences that include subject matter experts, professionals, and the general public around the world. Exhibition Program staff transform these stories into content for presentation on multiple formats.

The Quality Assurance Surveillance Plan outlines how the government will evaluate the requirements, services, and expected outcomes of the Museum Specialist Vendors.

TABLE OF CONTENTS

1 Introduction

1.1 Purpose

1.2 Performance Management Approach

1.3 Performance Management Strategy ....................................................... ………… …. …2

2 Roles and Responsibilities

2.1 The Contracting Officer

2.2 The Contracting Officer’s Representative

3 Identification of Required Performance Standards/Quality Levels

4 Methodologies to Monitor Performance

4.1 Surveillance Techniques

4.2 Customer Feedback

4.3 Acceptable Quality Levels

5 Quality Assurance Documentation

5.1 The Performance Management Feedback Loop

5.2 Monitoring Forms

6 Analysis of Quality Assurance Assessment

6.1 Determining Performance

6.2 Reporting

6.3 Reviews and Resolution

Attachment 1: Performance Requirements Summary

Attachment 2: Sample Quality Assurance Monitoring Form

Museum Specialist Services Quality Assurance Surveillance Plan

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

1 Introduction

This Quality Assurance Surveillance Plan (QASP) is pursuant to the requirements listed in the Performance Work Statement entitled Museum Specialist Services. This plan sets forth the procedures and guidelines the Exhibition Program, National Library of Medicine will use in ensuring the required performance standards or services levels are achieved by the Vendor(s).

1.1 Purpose

1.1.1 The purpose of the QASP is to describe the systematic methods used to monitor performance and to identify the required documentation and the resources to be employed.

The QASP provides a means for evaluating whether the Vendor is meeting the performance standards/quality levels identified in the Performance Work Statement and the Vendor’s Quality Control Plan (QCP), and to ensure that the government pays only for the level of services received.

1.1.2 This QASP defines the roles and responsibilities of all members of the Integrated Project Team, identifies the performance objectives, defines the methodologies used to monitor and evaluate the Vendor’s performance, describes quality assurance documentation requirements, and describes the analysis of quality assurance monitoring results.

1.2 Performance Management Approach

1.2.1 The Performance Work Statement structures the acquisition around “what” service or quality level is required, as opposed to “how” the Vendor should perform the work (ie, results, not compliance). This QASP will define the performance management approach taken by the Exhibition Program to monitor and manage the Vendor’s performance to ensure the expected outcomes or performance objectives communicated in the Performance Work Statement are achieved. Performance management rests on developing a capability to review and analyze information generated through performance assessment. The ability to make decisions based on the analysis of performance data is the cornerstone of performance management; this analysis yields information that indicates whether expected outcomes for the project are being achieved by the Vendor.

1.2.2 Performance management represents a significant shift from the more traditional Quality Assurance concepts in several ways. Performance management focuses on assessing whether outcomes are being achieved and to what extent. This approach migrates away from scrutiny of compliance with the processes and practices used to achieve the outcome. A performance-based approach enables the Vendor to play a large role in how the work is performed, as long as the proposed processes are within the stated constraints. The only exceptions to process reviews are those required by law (federal, state, and local) and compelling business situations, such as safety and health. A “results” focus provides the Vendor flexibility to continuously improve and innovate over the course of the contract as long as the critical outcomes expected are being achieved and/or the desired performance levels are being met.

1.3 Performance Management Strategy

1.3.1 The Vendor is responsible for the quality of all work performed. The Vendor measures that quality through the Vendor’s own Quality Control Plan. Quality Control is work output, not workers, and therefore includes all work performed under this contract regardless of whether the work is performed by Vendor employees or by subcontractors. The Vendor’s Quality Control Plan will set forth the staffing and procedures for self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the Performance Work Statement. The Vendor will develop and implement a performance management system with processes to assess and report its performance, and order burn rate, to the designated government representative.

1.3.2 The government representative(s) will monitor performance and review performance reports furnished by the Vendor to determine how the Vendor is performing against communicated performance objectives. The government will make determinations regarding performance based on performance measurement metric data and notify the Vendor of those decisions. The Vendor will be responsible for making required changes in processes and practices to ensure performance is managed effectively.

2 Roles and Responsibilities

2.1 The Contracting Officer

The Contracting Officer is responsible for monitoring contract compliance, contract administration, and cost control and for resolving any differences between the observations documented by the Contracting Officer Representative and the Vendor. The Contracting Officer will designate one full-time Contracting Officer Representative as the government authority for performance management. The number of additional representatives serving as technical inspectors depends on the complexity of the services measured, as well as the contractor’s performance, and must be identified and designated by the Contracting Officer.

2.2 The Contracting Officer Representative

The Contracting Officer Representative is designated in writing by the Contracting Officer to act as his or her authorized representative to assist in administering a contract. Contracting Officer Representative limitations are contained in the written appointment letter. The Contracting Officer Representative is responsible for technical administration of the project and ensures proper government surveillance of the Vendor’s performance. The Contracting Officer Representative is not empowered to make any contractual commitments or to authorize any contractual changes on the government’s behalf. Any changes that the Vendor deems may affect contract price, terms, or conditions shall be referred to the Contracting Officer for action. The Contracting Officer Representative will have the responsibility for completing Quality Assurance monitoring forms used to document the inspection and evaluation of the Vendor’s work performance. Government surveillance may occur under the inspection of services clause for any service relating to the purchase order.

3 Identification of Required Performance Standards/Quality Levels

The required performance standards and/or quality levels are included in the Performan ce Work Statement and in Attachment 1: Performance Requirements Summary. If the contractor meets the required service or performance level, it will be paid for the work.

4 Methodologies to Monitor Performance

4.1 Surveillance Techniques

In an effort to minimize the performance management burden, simplified surveillance methods shall be used by the government to evaluate contractor performance when appropriate. The primary methods of surveillance are:

• Random observations, which shall be performed by the Contracting Officer Representative or designated inspector.

• File reviews—Contracting Officer Representative or designated inspector will review files periodically, consistently, or 100% review.

• Periodic Inspection – Contracting Officer Representative or designated inspector will perform periodic inspection at a minimum on a monthly basis.

Contracting Officer Representative or designated inspector completes surveillance documentation on a monthly basis.

4.2 Customer Feedback

The Vendor is expected to establish and maintain professional communication between its employees or subcontractors and customers—including the federal government and customers of the federal government. The primary objective of this communication is customer satisfaction. Customer satisfaction is the most significant external indicator of the success and effectiveness of all services provided and can be measured through customer complaints.

Performance management drives the Vendor to be customer focused through initially and internally addressing customer complaints and investigating the issues and/or problems but the customer always has the option to communicate complaints to the Contracting Officer or the Contracting Office Representative, as opposed to the Vendor.

Customer feedback may also be obtained either from the results of formal customer satisfaction surveys or from random customer complaints .

4.3 Acceptable Quality Levels

The Acceptable Quality Levels included in Attachment 1: Performance Requirements Summary for Vendor performance are structured to allow the contractor to manage how the work is performed. There are no financial penalties for failing to achieve Acceptable Quality Levels other than additional tasks will not be awarded until performance is brought into compliance.

5 Quality Assurance Documentation

5.1 The Performance Management Feedback Loop

The performance management feedback loop begins with the communication of expected outcomes. Performance standards are expressed in the Performance Work Statement and is assessed using the performance monitoring techniques shown in Attachment 1: Performance Requirements Summary.

5.2 Monitoring Forms

The government’s Quality Assurance surveillance, accomplished by the Contracting Officer Representative or designated inspector will be reported using the monitoring forms in Attachment 2: Sample Quality Assurance Monitoring Form. The forms, when completed, will document the government’s assessment of the Vendor’s performance under the contract to ensure that the contractor is achieving the required Acceptable Quality Levels of performance.

The Contracting Officer Representative will retain a copy of all completed Quality Assurance surveillance forms.

6 Analysis of Quality Assurance Assessment

6.1 Determining Performance

Government shall use the monitoring methods cited to determine whether the performan ce standards/service levels/Acceptable Quality Levels have been met. If the Vendor has not met the minimum requirements, it may be asked to develop a corrective action plan to show how and by what date it intends to bring performance up to the required lev els.

6.2 Reporting

At the end of each month, Contracting Officer Representative or designated inspector will prepare a written report for the Vendor’s file summarizing the overall results of the quality assurance surveillance of the Vendor’s performance. This written report, which includes the Vendor’s submitted monthly report and the completed quality assurance monitoring forms (Attachment 2), will become part of the Quality Assurance documentation. It will enable the government to demonstrate whether the Vendor is meeting the stated objectives and/or performance standards, including cost/technical/scheduling objectives.

6.3 Reviews and Resolution

The Contracting Officer Representative may require the Vendor’s project manager, or a designated alternate, to meet with the Contracting Officer Representative or designated inspector as deemed necessary to discuss performance evaluation. The Contracting Officer Representative will define a frequency of in-depth reviews with the Vendor, including appropriate self-assessments by the Vendor; however, if the need arises, the Vendor will meet with the Contracting Officer Representative as often as required or per the Vendor’s request. The agenda of the reviews may include:

• Monthly performance assessment data and trend analysis

• Issues and concerns of both parties

• Projected outlook for upcoming months and progress against expected trends, including a corrective action plan analysis

• Recommendations for improved efficiency and/or effectiveness

• Other issues affecting performance: The Contracting Officer Representative or designated inspector must coordinate and communicate with the Vendor to resolve issues and concerns regarding marginal or unacceptable performance.

The Contracting Officer Representative or designed inspector and Vendor should jointly formulate tactical and long-term courses of action. Decisions regarding changes to metrics, thresholds, or service levels should be clearly documented. Changes to service levels, procedures, and metrics will be incorporated as a contract modification at the convenience of the Contracting Officer.

Attachment 1 Quality Assurance Surveillance Plan Performance Requirements Summary

The Museum Specialist services Vendor requirements are summarized into performance objectives that relate directly to mission essential items. The performance threshold briefly describes the minimum acceptable levels of service for each requirement. These thresholds are critical to mission success

Regarding accessibility: Documentation and reports prepared by the Vendor in performance of the terms of this agreement will conform to Section 508 accessibility requirements for Documentation and Support Services (36 CFR 1194, Appendix C, 602.1 and 602.3). Common application checklists are available at the Section508.gov web site. Look for “Harmonized guidance from the AED Community of Practice (COP)”

Performance Objective Standard Performance Threshold or Acceptable

Quality Levels

Method of Surveillance

12.1 Museum Specialist: Exhibition coordination services

12.1.1 All project related

information is relevant, accurate, appropriately named, captured, and distributed in a timely manner, archived and documented.

Confidential information is protected.

1. Asset related information is maintained in keeping with Exhibition Program protocols and archival systems.

2.Distributed information does not need to be recalled, corrected, and redistributed

2. All information documents are labeled, dated, pagination assigned, and file naming conventions followed

3. Contact information, business information

1. All project related information is accurate and up-to-date within 48 hours of changes or corrections or sooner if required by production schedules.

2. Re-issuing corrected information to a project team occurs not more than twice per project.

3. All project related information is distributed on time as per established project schedules.

Late release of no longer than 48

File reviews Periodic inspections Random observations https://section508.gov/best-practices including values, and lender information are not shared without staff approval

4. Requests for project information are appropriately prioritized and answered with project team experiencing no delay of work

5. Meeting notes, content and technical information, asset information is delivered within 24 hours of request unless otherwise negotiated hours occurs less than once a month.

4. Work-in-process files are deleted, all final files updated and finalized, vendor production files acquired and archived within 4 weeks of project launch

5. Contact information, business information including values, and lender information are not shared without staff approval

12.1.2 The project team has

access to project assets including appropriately sized digital assets for multiple production purposes.

1. Project has required digital assets of appropriate sizes for multiple production purposes

2. Project team can access digital assets through Exhibition Program standard naming conventions and archival protocols

1. Work-in-progress digital assets and final production files are available throughout the project life cycle and especially in anticipation of milestone junctures

2. Within 1 week of receipt, all digital assets are named and filed in keeping with Exhibition Program best practices

File reviews Periodic inspections Random

12.1.3 Project scripts adhere to

Exhibition Program and project style guides.

Scripts are current, accurate, and error free.

Online scripts have relevant alt tags.

1. Project scripts are accurate, up to date, and error free throughout the project life cycle

2. Project scripts adhere to Exhibition Program and project style guides

3. Project team can access script documents through Exhibition Program-wide standard naming conventions and archival protocols

1. Changes and updates to active scripts are cascaded across all project documents within 48 hours

2. Application of Exhibition Program and project style guides to all scripts occurs 2 weeks prior to release for review and design production

3. Script files are always saved in accordance with Exhibition Program naming conventions and archival protocols.

File reviews Periodic inspections Random observations

12.1.4 Project team members

have and adhere to project dates and schedules.

1. Project schedules are kept current

2. Project team can access current schedules

1. Project schedules are current within 24 hours of changes or updates inspections Random observations

12.1.5 Implement Quality

Control over design and production deliverables from exhibition designer.

1.Design and production deliverables accurately reflect original source content and all technical specifications.

2. Changes and corrections to design and production deliverables are accurately

1. During review period, team members are alerted, design deliverables are inspected, comments collated with 100% on-time response to designers 2.Communications regarding quality control is clear with no

File reviews Periodic inspections communicated and documented.

misunderstanding 95% of the time.

12.1.6 Content deliverables for

exhibition designer and to web producer are accurate, and complete.

Exhibition and web deliverables are reviewed for quality assurance of functionality, accessibility, and implementation of content.

1. Content for designers and producers are accurate, complete, well organized, and delivered and received on time.

2. Content adheres to Exhibition Program style guidelines

2. Web producer programming delivered to Exhibition Program is reviewed and tested for accuracy, functionality, completeness, adherence to Exhibition Program style guidelines, and 508 compliance.

1. Content is distributed on time as per established project schedules.

Late release of no longer than 48 hours occurs less than once a month.

2. Quality and Assurance reviews are scheduled 2 weeks in advance, reviewers notified and committed to the task; work completed, compiled and available for release on time with no longer than 48 hours delay.

3. Projects are 100% compliant with Exhibition Program style guidelines

File reviews Periodic inspections

12.2 Museum Specialist: Exhibition guest curator services

12.2.1 Project related research

is responsive to cotemporary best practices. National Library of Medicine collections are showcased. Results of research will appeal and engage target audiences.

Original sources are identified.

1. Research acknowledges existing scholarship and adheres to best practices as outlined in the American Historical Association Statement of Professional Conduct.

1. Research process is transparent, and deliverables are professionally documented, sources attributed, and work is verifiable 100% of the time.

File reviews Periodic inspections

2. Chicago Manual of Style guides development of work products

12.2.2 Project assets are

relevant, and accessible and available for inclusion in the project. Assets are responsive to Program-wide standards. Assets respect the privacy and dignity of individuals depicted.

1. Recommended assets are primarily drawn from the collections of the National Library of Medicine

2. Other recommended assets are reliably sourced and available for loan

3. A majority (more than 90%) of recommended assets convey meaning visually and not through text

1.All recommended assets are responsive to Exhibition Program-wide standards and best practices

2. Occasionally, from time to time, project specific assets that fall outside standard practices are acceptable

File reviews Periodic inspections Random observations

12.2.3 Textual interpretation of

narratives and assets engages young readers.

1. Project scripts adhere to Exhibition Program standards and museum best practices, as outlined in literature such as Beverly Serrell’s On Labels.

2. All writing targets project audiences.

3. Project scripts are repurposed for different media.

1. Scripts are factually accurate 100% of the time.

2. Scripts clearly communicate project messages to diverse audiences 100% of the time.

3. Final scripts are error free 100% of the time.

File reviews Periodic inspections

12.3 Museum Specialist: Exhibition educator services

12.3.1 Promote use of exhibition

content with emphasis on

1. Online exhibition

1. Education resource content

NLM resources in classroom instructions.

education resources, such as lesson plan and university module, help educators engage students in learning with exhibition content. (See Reference A.

Lesson Plan style guide and Reference B.

University module style guide.)

2. Online educational resources make selected exhibition topics and materials, relevant and appropriate for existing academic curricula and/or programs.

3. Online educational resources use effective instructional strategies that reflect student-centered and educator-guided learning.

4. Other formats of exhibition-based educational resources may include online is factually accurate 100% of time.

2. Each education resource features at least one (1) exhibition topic/theme and three (3) specific exhibition materials, as well as minimum of three (3) National Library of Medicine items/resources.

3. Each online education resource involves at least one (1) current or previous classroom educator from the targeted education level(s) who has expertise in educational theories and practices.

4. All formats of exhibition educational resources adhere to style guide for each format.

Periodic inspections interactives, downloadable stand-alone activity package, or others to be defined and developed as needed and when resources permit.

12.3.2 Raise awareness of

National Library of Medicine and Exhibition Program resources among education communities.

1.Awareness campaign informs targeted education communities about relevant exhibition educational resources.

2. Impact and outcome of an awareness campaign is measured to inform future development of educational resources.

1. Web analytics or other tools track the implementation of all awareness campaign elements.

2. Reports document the assessment(s) outcomes of awareness campaign.

3. Queries about the exhibition education resources are responded to within 48 hours.

File reviews Periodic inspections

12.4 Museum Specialist: Traveling exhibition services

12.4.1 Make available to

borrowing institutions a standard package of online resources related to individual exhibition titles. Provided enhanced package of resources when relevant.

Online resources reflect accurate information for each exhibition title; customers have access to information to properly install, display, ship traveling exhibitions without incident

100% of online resources are provided to web team at least one month prior to launch of online traveling exhibition booking page;

100% of exhibitions are shipped on time

File reviews Periodic inspections Random observations

12.4.2 Itinerary and booking

data is accurate across

Adhere to Traveling

100% of booking data is accurate traveling exhibition services tracking systems.

Exhibitions How- To-Guide and updated weekly to reflect new bookings and new customers

Periodic inspections Random observations

12.4.3 Meaningful

correspondence with customers is recorded and archived.

Customer inquiries and correspondence are maintained and organized by exhibition and venue

100% of customer inquiries and requests are responded to within 24 hours of receipt.

inspections Random observations

12.4.4 Customers are aware of

and responsive to the importance of completing and submitting customer surveys after each booking period.

Adhere to Traveling Exhibition Services How-To- Guide

Customer survey response percentage exceeds 75% inspections Random observations

12.4.5 Records reflect accurate

and current customer and audience data for analysis, planning, and reporting purposes.

Adhere to Traveling Exhibition Services Standard Operating Procedures

Data is consistent and accurate 100% of the time.

100% of data captured is relevant to the mission of the Exhibition Program

File reviews Periodic inspections Random observations

12.4.6 Data reports visualize the

significance of traveling exhibition services statistics.

Adhere to Traveling Exhibition Services Standard Operating Procedures

Data is consistent and accurate 100% of the time.

100% of data captured reflects the significance of traveling exhibition statistics

File reviews Periodic inspections Random observations

12.4.7 Customers engage in

peer-to-peer networks to share public programming information and engage with Regional Medical Libraries.

Adhere to Traveling Exhibition Services Standard Operating Procedures

The number of customers participating in peer-to-peer networks and connecting with Regional Medical Libraries increases over time

File reviews Periodic inspections Random observations

12.4.8 Records for traveling

exhibition services inventory outline

Adhere to Traveling Exhibition

Records for inventory accurately reflect

File reviews inspections requirements for 12 months, 3 years, and 5 years.

Services Standard Operating Procedures projected long-range programmatic requirements

Random observations

12.4.9 Reports, recommendations, and day-to-day oversight of traveling exhibition services responds to Exhibition Program objectives.

Adhere to Traveling Exhibition Services Standard Operating Procedures and How-To-Guide

100% of traveling exhibition tasks respond to Exhibition Program objectives

File reviews Periodic inspections

12.5 Museum Specialist: Registrarial support

12.5.1 Assets borrowed from

other organizations, institutions, or individuals are safely transported between owner and National Library of Medicine and are physically and legally secure while in the custodial care of the National Library of Medicine.

Adhere to American Alliance of Museums best practices as outlined in Museum Registration Methods, 5th edition.

100% of assets are shipped, received, protected and secured, and returned safely

File reviews Periodic inspections Random observations

12.5.2 Install/deinstall assets in

special displays and other exhibitions.

Adhere to American Alliance of Museums best practices as outlined in Museum Registration Methods, 5th edition.

100% of assets are safely installed and managed with no incidents

File reviews Periodic inspections Random observations

12.5.3 The wider project team

including conservation libraries and History of Medicine Division staff have timely access to assets and relevant information regarding asset conditions.

Adhere to loan agreement requirements;

adhere to accepted museum professional and conservation standards

HMD conservation has 100% access to assets in a timely manner

File reviews Periodic inspections

12.5.4 Exhibition Program

environments are safe, secure, and pest free.

Adhere to professional museum standards for environmental conditions including temperature, humidity, security and pest management

Exhibition Program spaces are 100% pest-free and environmental levels maintained within required parameters

File reviews Periodic inspections Random observations

12.5.5 National Library of

Medicine has rights and permissions to include reproductions of borrowed assets or images with rights owned by others in Exhibition Program projects.

Adhere to LO Policy for Fair Use of Copyrighted Materials and Exhibition Program protocols for securing copyright

Permission is received in writing for 100% of assets

File reviews Periodic inspections Random observations

12.5.6 The public has access, when permissible, to image assets from Exhibition Program legacy projects.

Adhere to LO Policy for Fair Use of Copyrighted Materials

100% of all permissible image assets are accessible to the public

File reviews Periodic inspections

Attachment 2 Sample Quality Assurance Monitoring Form

SERVICE or STANDARD:

SURVEY PERIOD:

SURVEILLANCE METHOD (Check):

Random Sampling 100% Inspection Periodic Inspection

LEVEL OF SURVEILLANCE (Check):

Monthly Quarterly As needed

PERCENTAGE OF ITEMS SAMPLED DURING SURVEY PERIOD: ______ %

ANALYSIS OF RESULTS:

Observed Service Provider Performance Measurement Rate: ______%

Service Provider’s Performance (Check): Meets Standards Does Not Meet Standards

Narrative of Performance During Survey Period:

PREPARED BY: ___________________________________ DATE: _________________