15-223-SOL-00185_Atch_7_Award_Term_Perf._Evaluation_and_Measurement_plan.pdf

15-223-SOL-00185 Atch 7 Award Term Performance Evaluation and Measurement Plan

1. INTRODUCTION

This Award Term Performance Evaluation and Measurement Plan (hereinafter referred to as the “Plan”) serves as the charter which will be used to evaluate the contractor's performance of work required by this contract and to determine whether the award term ordering period (hereinafter referred to as the “award term” or “term”) will be granted. FDA requires a top-level software solution and contractor performance to meet program requirements. Hence, this performance plan is designed to provide an additional incentive to the contractor for outstanding performance that will benefit FDA.

The intent of this plan is to establish procedures for the evaluation of contractor performance by furnishing guidelines and procedures for: (1) evaluating the contractor's performance during the evaluation period; and (2) furnishing sufficient data to enable the Award Term Determination Official (ATDO) to determine whether the award term will be granted.

2. GENERAL

General provisions of the award term are provided in Section 2.2 of the IDIQ contract.

3. PURPOSE

This plan and the specific contract provisions shall serve as a guide to FDA personnel directly involved in the evaluations of contractor performance.

4. OBJECTIVES

The objective of this performance award term feature is to incentivize contractor performance in the areas delineated in paragraph 6.1 and to assist the Government in selecting the most highly rated solution(s) for further implementation. Therefore, the contractor should emphasize these areas in its performance of this contract.

5. PERFORMANCE AWARD TERM REVIEW TEAM

5.1. Organization

The organization of the Team is described in the following paragraphs.

5.1.1 Award Term Determination Official. The Award Term Determination Official is the cognizant Contracting Officer for the FDA. The Contracting Officer may appoint another individual to perform this function for their organization.

5.1.2 Performance Award Term Review Team. The Award Term Determination Official will appoint a Team to assist in evaluating the contractor's performance. If warranted by the size or complexity of the contract, the Award Term Determination Official may appoint a Chairman to the Team or the Award Term Determination Official may serve as the Chairman. The team members will ensure a fair and accurate assessment of the contractor's performance for the period being evaluated. The Team Chairman may also use non-voting advisors as necessary.

5.2. Duties of the Team.

The duties of the Team are as follows:

• Implement the plan and propose timely modifications to the Plan if required, throughout the period of contract performance.

• Evaluate contractor performance for each performance evaluation period.

• Prepare and submit to the Award Term Determination Official a written evaluation of the contractor's performance.

5.3. Responsibilities.

5.3.1 Award Term Determination Official. Approves the award term plan and the evaluation factors and scoring methodology. Approves the composition of the Team. Determines the contractor performance rating and whether the award term period will be granted based on the factors of the plan. Advises the contractor in writing of annual evaluation results and award term decision and documents the basis for the decision.

5.3.2 Team Chairman. Structures the Team membership to provide representation that reflects all appropriate aspects of contract performance and provides membership to adequately assess contractor performance for the period being evaluated. Conducts the evaluation under this plan.

Schedules Team meetings and serves as a recorder at these meetings. Leads the team in developing a consensus evaluation and in resolving significant differences in ratings. Provides brief summary documentation for the Award Term Determination Official.

5.3.3 Performance Award Term Review Team. Monitors and evaluates contractor performance for the period under consideration, utilizing the factors set forth in the plan. Makes written evaluations and formulates award term recommendations. Briefs the Award Term Determination Official on evaluations, when requested, and provides supporting data/documentation to support the assessment of performance. Prepares the evaluation report and accompanying narrative justification. For this plan, the Performance Award Term Review Team will consider input of Principal Investigators (PI) and user community representatives and may delegate selected review team activities to Principal Investigators and user community representatives.

5.3.4 Contract Specialist. Prepares and distributes contract modifications awarding the term authorized by the Award Term Determination Official. Maintains term documentation as part of the official order file. Retains historical files and other documentation relating to term matters for the contract.

6. PERFORMANCE EVALUATION AND FACTORS

FDA will evaluate the contractor's performance in achieving contract requirements for the term period using the evaluation factors below.

6.1. Evaluation Factors and Scoring

The contractor's performance will be evaluated on the basis of factors with subjective rating criteria and assigned evaluation ratings of excellent, highly satisfactory, satisfactory, and unsatisfactory. An evaluation rating of “excellent” and “highly satisfactory” will only be given when the contractor’s performance substantially exceeds satisfactory:

6.1.1. Factor 1 – Ability Perform the Use Cases Provided

Overview: This document contains a set of short Use Cases that are meant to demonstrate the capabilities of ELNs. In this set of Use Cases, a Principal Investigator designs an experiment that has two workflows, assigns the experiment to lab technicians, and reviews the results. The Lab Technicians conduct experiments and collaborate with each other. An Auditor audits the experiments.

Notes:

• It is NOT necessary to follow the steps in sequence.

• If the ELN does not support some of the functionality, it may be omitted.

• The experiment examples depicted here may not technically be experiments.

Evaluation of Use Cases: FDA will provide an experiment to which the ELN can be used to evaluate the capability of each vendor product to meet the criteria identified in the Use Cases. In order to achieve a rating of satisfactory, the product must be capable of performing all criteria listed in the Use Cases. In order to receive a rating of highly satisfactory or excellent, the product must have features or functionalities that exceed the criteria identified in each Use Case but are still applicable to the Use Case category. An unsatisfactory rating means that the software does not have the functionalities to meet the steps identified in the Use Case.

• Use Case 1 Name: Design the Experiment

Actor: Principal investigator (PI)

Brief Description: The Principal investigator sets up an experiment that will be conducted by Lab

Technicians. The PI as well as the Lab Technicians will eventually need to collaborate on the experiment.

# Experiment Steps

1 Pull up a list of templates to design experiments and pick one. [Note: Try to include unstructured data and experimentation].

2 Enter Experiment Name “Prepare cell culture of a sample from site A”.

3 Create two workflows: Collect Sample (CS) and Grow Culture (GC). [See Use Case 3 for workflow steps].

4 Assign Collect Sample to Lab Technician 1 and Grow Culture to Lab Technician 2

5 Assign the experiment to a series of experiments (that are predefined before the demo). Then remove the experiment from the series.

6 If the ELN supports importing and exporting of workflows, then import the Collect Sample workflow. Change the workflow and export it.

• Use Case 2

Name: Administer Experiments

Actor: Principal investigator (PI)

Brief Description: The Principal investigator performs various administration roles. At the end of the experiment, the PI evaluates the results and requests that the experiment be redone.

1 Authorize (or start) the experiments after searching for other experiments and making sure that duplication does not exist.

2 During Use Case 3, reassign work from LT2 to LT3.

3 Before, during and after the experiments, look at dashboard that shows the various experiments, their status, and who they are assigned to. For the purpose of the demo, additional experiments and other data may be created.

4 Review the experiment from a PI perspective.

5 At the conclusion of the experiment, indicate that the experiment is not acceptable and should be repeated.

• Use Case 3

Name: Conduct Experiments (This use Case follows Use Case 1 and some parts of Use Case 2) Actors:

Lab Technician 1 (LT1), Lab Technician 2 (LT2) and Lab Technician 3 (LT3)

Brief Description: Lab Technicians 1, 2, and 3 execute the workflows. The two workflows are dependent on each other and go side by side. CS and GC identify which steps the workflows belong to, and are described in Use Case 1.

# Actor Experiment Steps

CS1 LT1 Indicate that the sample is about to be collected.

GC1 LT2 Receive notification that 500 ml of “Broth A” should be prepared. Broth A contains ingredients Ing1, Ing2, and Ing3.

GC1 LT2 Look up inventory of Ing1. Weigh 50g, and add it to a beaker. Return the rest of the Ing1 back to storage. Note the amount that was weighed (example: 49.984 g)

Note: In this step, Inventory Management features are to be displayed or discussed.

Capability to map sample location in fridge/freezer; by lab room#, freezer #, drawer#, shelf#, rack# would be valuable. A grid or spreadsheet that displays the locations would be useful.

GC2 LT2 Weigh 1g of Ing2. In this step, the scale should be connected to the ELN and the reading obtained directly from the scale. For the demo, the step may be modified to use a connection to any small instrument/appliance such as a scale, pH meter, pipette…etc. If the connection is not possible, then a simulator may be used. At the very least, a short statement on how this step would be handled by the ELN should be made.

GC3 LT2 And Ing2 to the beaker (that contains Ing1).

GC4 LT2 Measure 50 ml of Ing3. Add it to the beaker. Record that 60 ml was added. Then correct the entry to 50. Add a comment or note that 60 was a typo.

GC5 LT2 Leave the beaker in the incubator at 37C.

CS2 LT1 Collect the sample from Site A and generate a unique sample ID (print bar code if possible) for it. This is Sample A.

CS2 LT1 Attach a picture of the sample collection site (any picture file may be attached in the demo).

CS3 LT1 Trigger notification to PI and LT2 that the sample has been collected.

GC6 LT2 Receive Sample from LT1.

GC7 LT2 Add 5 ml of sample to beaker and incubate it for 24h (or for a few seconds at the demo).

***See Use Case 3***. Work assignment has been changed from LT2 to LT3.

GC9 LT3 Check the pH of the solution in beaker. If the pH is less than 5, start workflow to “Add alkali”. If the pH is greater than 8, start workflow “Add Acid”. At the demo, show that either one of the two distinct workflows are executed, depending on the decision point.

GC10 LT3 Show or discuss support for math functions, stoichiometric calculations, mass, empirical and molecular formulas, chemical equations, percent yield, etc.

GC11 LT3 Scribble a note with finger or a pen (if the feature is available on the ELN)

GC12 LT3 Sign off that the work is complete. Demonstrate or discuss whether this can be done using PIV cards.

• Use Case 4 Name: Collaborate on Experiments

Actor: Principal investigator (PI), Lab Technician 1 (LT1), Lab Technician 2 (LT2) Brief Description: Actors collaborate on the experiment

# Actor Experiment Steps

1 LT1 See whether anybody else is working on or has recently worked with a sample from Site A.

2 LT1 Check the status of Broth A preparation (which is prepared in Use Case 3) at various stages of Use Case 3.

• Use Case 5 Name: Audit the experiment

Actor: Auditor

Brief Description: Review the experiment

1 Trace the various steps taken during the experiments. All steps with the start and end timestamps, comments, and corrections must be visible.

2 The chain of custody for Sample A (from Use case 3) must be visible.

6.1.2. Factor 2 – Meets Job Requirements

The Principal Investigators (PI) participating in the pilot will be directed to use the ELN solution for a period of one month to conduct their daily scientific research. PI’s will be requested to document the ability of the selected ELN solution to perform daily conducted scientific activities. This will be reviewed by the chief PI to determine the efficacy of the product in a real life scenario.

Evaluation of the ability to meet factor 2 is a subjective judgment made by the chief PI. Features that will be considered are:

• General ease of use

• Minimal effort need to configure ELN solution for use cases

• Ability to easily navigate between functions

• Ability to organize, archive, or retrieve documentation

• Ability to integrate with other systems

• Ability to share data with other stakeholders

• Responsive to future needs

• Document management

• Simultaneous use by multiple users

6.1.2. Factor 2 – Meets Job Requirements