GADS3_Attachment_J_3_TaskOrder_2_Final.docx

ATTACHMENT J-3 “PMR” WORK STATEMENT TASK ORDER 2

ATTACHMENT J-3 WORK STATEMENT – Task Order 2 User Interface Development for PCC Machine Reading (“PMR”)

NATIONAL INSTITUTES OF ENVIRONMENTAL HEALTH SCIENCES

RESEARCH TRIANGLE PARK, NC

Table of Contents

1.0 INTRODUCTION:

This task order work statement is to obtain Application Development, Modernization, and Enhancement (DME) services for the DERT Program Class Code (PCC) classification process for incoming grant applications. The objective of this task is to support the development of a user interface (UI) that integrates a Machine Reading Process to facilitate PCC assignment of incoming grant applications (“PCC Machine Reading” or PMR Phase I task order). The Government intends to award this Labor Hour (LH) task order with the IDIQ base contract award. The PMR task order is the first part of a two-phased effort, where the second phase (issued under a later, separate PMR Phase II task order under this IDIQ) will require changes to IMPAC II.

2.0 BACKGROUND AND SCOPE:

DERT uses the PCC schema to classify all 1,500 incoming applications per year into scientific topics and assign them to DERT program and grants management staff. The driving variable within the PCC schema is a 2-digit scientific code. Once assigned, the primary science code drives the assignment of the DERT program officer and the grants management specialist.

Currently the process is done manually; however, NIEHS has collaborated with Oak Ridge National Laboratory (ORNL) to develop a “Machine Reading Process” that analyzes application titles, abstracts and specific aims and matches each application to one or more of the roughly 100 scientific codes. The top match is assigned an uncertainty score of 0-6 (6 being most certain; 0 being most uncertain). The certainty score reflects the confidence that the reading machine has assigned the right score. The script uses the machine reading process to pick the best science code, based on two years of previous data classification (“learning”). Going forward, the prior year data classification file (“training data set”) needs to be updated periodically (not more than once per quarter.)

The code for this project is currently being tested and prepared for transfer to NIEHS. The Government intends to expedite the current PCC assignment approach through development and implementation of a UI that integrates with the Machine Reading Process. With this UI, DERT will be able to further automate and streamline the PCC assignment process. The users (“PCC coders”) consist of approximately three staff. The Machine Reading Process is command line based, uses a csv file as input/output, and requires Python. The input and output .csv file columns are identified in Appendix A.

3.0. TASKS:

The Contractor shall provide the following services under this task order.

3.1 Task Order Management

The GTM is responsible for technical monitoring of Contractor task performance on behalf of the task order Contracting Officer Representative. The GTM will ensure the Government effectively measures service levels and continuously conducts quality assurance to assess the Contractor’s performance at all times. The Contractor’s performance shall be monitored using the attached GADS3 Quality Assurance Surveillance Plan (QASP)(Base Contract Attachment J-9).

3.1.1 Task Order Project Plan

Within 10 calendar days after task order award, the Contractor shall provide a Project Plan (see DEL DME.C.7.2-1 in base contract) that addresses the following lifecycle phases of the PMR Phase I project:

· Need/Concept

· Definition

· Design

· Execution of Solution

· Deployment

· Operate and Maintain The Contractor shall use the plan to identify the project approach as either “Conventional” with project milestones or “Agile” with sprint outcomes and structure the plan accordingly. The Contractor shall meet or exceed DME Project Performance Objectives identified in Table 5 (“Lifecycle Maintenance Response and Resolution Levels) and Table 7 (Lifecycle Maintenance Application Availability Service Levels”) in the GADS3 base contract. The project plan shall identify whether and how prototypes and/or pilots will be used as part of the PMR Phase I effort.

3.1.2 Task Order Quarterly Review

The task order quarterly review shall take place as part of the agenda of the GADS3 Contract Quarterly Review.

3.1.3 Weekly Status Meetings

The Contractor shall meet weekly with the COR and GTM, as part of the weekly status meeting required for the base contract, to provide a status on task activities, task management issues, and any other task-related information that the Government needs to know in order for the Contractor to successfully perform the requirements of the PMR task order. For the PMR Phase I task, the Contractor shall use a PowerPoint briefing deck to deliver the PMR weekly agenda, meeting topics, and any supporting data (“PMR Weekly Status Briefing”) (PMR DEL 3.1-001). Briefing decks should be concise and generally consist of one (1) to three (3) slides. Meetings can take place on-site at the Government facility or via teleconference or WebEx. The Government may determine that weekly status meetings include additional DME task order projects, depending on the size and complexity of the projects.

3.1.4 Contractor Quality Control

The Contractor shall adhere to the GADS3 Quality Control Plan (QCP) that is submitted as part of the GADS3 contract proposal.

3.1.5 Task Order Start-Up Meeting

The task order start-up meeting shall take place as part of the agenda of the GADS3 Contract Start-Up Meeting.

3.1.6 Project Management

The Contractor shall meet all project management requirements as stated in the base contract Section 6.6. The Contractor shall use NIEHS-provided project management software to define and track application development phases in a transparent manner with the Government. NIEHS maintains environments that are available to the Contractor to develop and manage project tracking, ticket tracking, knowledge repositories, and workflows. These currently include:

The Contractor shall not use Internet-based or corporate web sites/portals to full this requirement. For security reasons, this requirement must be fulfilled using NIEHS network-confined tools or software.

3.1.7 Knowledge Management Portal

Within 30 calendar days of the task order award, the Contractor shall begin utilizing the Knowledge Management Portal that it has established under the GADS3 contract for posting all PMR Phase I project documentation and deliverables.

3.2 UI Design and Development

The Contractor shall develop a web-based UI that:

· Provides a user-friendly interface that allows the PCC coder to easily interact with the Machine Reading Process

· Is accessed by the PCC Coder and their backup(s)

· Easily filters data by output confidence level

· Keeps a record of the PCC codes for easy viewing

· Shows the user applications that have not been assigned PCC Codes

· Easily exports records to .xls or .csv

· Uses and periodically updates the training data set

· Communicates results to stakeholders

· Provides hyperlinks to abstracts and original applications

· Resides in the NIEHS data center, is behind the NIEHS firewall, is accessible only to DERT staff, and complies with all security guidelines of the parent contract

· Functions as a utility within the GFDT

In addition, the Contractor shall ensure the UI delivers the following functionality:

· Any secondary application that receives an NIEHS PCC code should automatically be displayed in GFDT

· The system shall clearly notify the PCC Coder when the PCC code for an application that has been in the system over 20 days is empty and needs attention.

· The following roles shall be supported:

a. Primary PCC coder

b. Backup PCC coder

c. ‘Read-only’ role that would allow anyone with access to GFDT to view a log of PCC assignments in a specified date range

i. Would require selecting a radio button (“View log”) and entering a date range.

ii. Corollary feature to GFDT.

· The Primary PCC coder shall have the ability to change or add to the automated email recipients.

3.3 Process and Design Documentation

The current workflows and business process are provided in Appendix B. The Contractor shall document the To-Be PMR Phase I Business Process (PMR DEL 3.3-001) that supports the concept workflow provided in Appendix C. The Contractor shall provide a Requirements Definition Document (PMR DEL 3.3-002) outlining the detailed requirements. The Contractor shall develop system design documentation to deliver the business functionality, including changes needed to the Solution Architecture Document (PMR DEL 3.3-003). The documentation shall include a solution overview, technical considerations, configuration management, physical architecture diagram, and other items as needed.

3.4 Testing, User Acceptance Testing, and Release

The Contractor shall conduct testing, user acceptance testing, and release per the requirements stated in the base GADS3 contract Section 7.2. The Contractor shall deliver a User Acceptance Test Plan (DEL DME.C.7.2-2), a Test Report (DEL DME.C.7.2-3), Release Notes (DEL DME.C.7.2-4), and a Release Package (DEL DME.C.7.2-5).

4.0 CONTRACT TYPE:

The Government anticipates awarding a FFP task order.

5.0 PLACE OF PERFORMANCE:

There are no additional clauses applicable to this section of the task order beyond those in the base contract.

6.0 HOURS OF OPERATION:

There are no additional clauses applicable to this section of the task order beyond those in the base contract.

7.0 PERIOD OF PERFORMANCE:

The total period of performance for this task order shall be for a base period of four months and two two-month option periods. This task order also includes the clause at FAR 52.217-8, Option to Extend Services.

8.0 DELIVERABLES/DELIVERY SCHEDULE:

9.0 GOVERNMENT FURNISHED PROPERTY, EQUIPMENT, AND SERVICES:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

10.0 CONTRACTOR FURNISHED PROPERTY, EQUIPMENT, AND SERVICES:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

11.0 SECURITY:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

12.0 TRAVEL:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

13.0 PACKAGING, PACKING, AND SHIPPING INSTRUCTIONS:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

14.0 DATA AND INTELLECTUAL RIGHTS:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

15.0 INSPECTION AND ACCEPTANCE CRITERIA:

There are no additional clauses applicable to this section of the Task Order beyond those in the base contract.

16.0 ACCOUNTING AND APPROPRIATION DATA:

Funds are available for this task order.

17.0 KEY PERSONNEL:

Certain skilled experienced project management and/or technical personnel are essential for accomplishing the work to be performed:

· Written and verbal communications skills to clearly articulate conditions of success for work tickets and create scenarios for user acceptance testing

· Maintenance of program schedules and deliverables completed according to Government requirements

· Management of service level agreements and/or performance metrics as key metrics of success for contract performance

· B.S. or B.A. degree. Two years of related experience may be substituted for the education degree requirement established for this labor category.

18.0 INVOICING:

Emails that contain invoices sent to the CO and the COR for review and approval must contain, in the email header, the name of this task order “PMR”. There are no additional clauses applicable to this section of the task order beyond those in the base contract.

Appendix A: Input and Output .csv File Columns The input .csv file columns for both the training data set and the applications to be coded include:

1. Activity Code

2. PI Name

3. Project Number

4. Appl Id

5. Council SRG

6. IRG Group

7. IRG Cluster

8. RFA/PA Number

9. RFA Title

10. Institution

11. Human

12. Project Title

13. Abstract Text (with Public Health Relevance)

14. Specific Aims

15. RCDC Cats

The output .csv file columns currently include:

1. Activity Code

2. PI Name

3. Project Number

4. Appl Id

5. Council SRG

6. IRG Group

7. IRG Cluster

8. RFA/PA Number

9. RFA Title

10. Institution

11. Human

12. Project Title

13. Abstract Text (with Public Health Relevance)

14. Specific Aims

15. RCDC Categories

16. Predicted Science Codes (No particular order)

17. Top Science Code

18. Confidence (for Top Science Code) (0 - Low; 6 - High)

Appendix B: Current Workflow

1. The PCC Coder extracts the application data from QVR and prepares the .csv file by hand.

2. The PCC Coder opens a command line viewer from their work laptop.

3. The PCC Coder enters a command line prompt

a. For example: ./ornl-ml-tool.py --training_data=my_old_labelled_data.xls --new_data=new_proposal_data.xls --output=new_proposal_data_with_ml_predictions.xls

4. The Machine Reading Process (which resides on a server in the NIEHS data center) analyzes the data provided and produces a .csv file with results.

5. The PCC Coder then uses the .csv file to review data, combines it with other aspects of the PCC process and enters the PCC codes that will be assigned to each application into the spreadsheet for record keeping.

6. The PCC Coder then opens the ICO module in IMPAC II and updates the PCC code for each application.

Appendix C: UI To-Be Concept (Workflow and other features)

1. Each night, the UI automatically identifies any competing (Type 1, 2, 9) applications with null PCC codes and runs the Machine Reading Process, adding all needed information to the display.

2. Users are able to update the training data set periodically (no more than once per quarter).

3. As needed (2-3 times/week), the PCC Coder opens the tool and sees a list of the applications not yet assigned PCC codes.

4. Display features include (See also Figures 1, 2 and 3 below.):

a. Three separate navigation sections (e.g., top tabs): Type 1 applications, all other Types (2 and 9), and manual import.

i. The Type 1 applications have an empty field for adding a PCC

ii. The Type 2 and 9 applications display the prior PCC code and provide a column to track whether the PCC Coder has reviewed or not. The PCC column will already be filled in since PCC codes for these applications automatically port over. If the user selects “reviewed” the PCC code will remain unchanged and the application would be removed from view with no further action needed. If the code should be changed, the user can enter change the new PCC code in the PCC column.

iii. The manual import tab will include a page similar to the “search” page in GFDT and will allow the user to input information of a specific application. This application would then be brought into the tool (in the type 1 or type 2/9 tabs).

b. The user can access relevant information (via hyperlink) and record the PCC to be associated with the application.

c. Color is used to highlight data produced by the Machine Reading Process

d. The user can filter applications based on three confidence level groups (5-6, 4 and <3)

e. The user can save or submit

i. Save records the manually entered data but does not remove the application from the view

ii. Submit” records the manually entered data and removes the application from the view.

f. Applications will be ordered by the time at which they were entered in the system (i.e. the oldest application will be on top).

g. Columns will be sortable

h. The user records PCC code in an empty field for each application

5. At 10PM each evening, any codes that were “submitted” by the user will be compiled into a .csv file and attached to an email.

a. Email recipients may differ based on the initial confidence level assigned by the Machine Reading Process. (e.g., Person A would receive a list of applications with scores of 0-3; while person B would receive a list of applications with scores of 4-6).

b. For Type 1 applications, an email will be sent out containing newly assigned applications

c. For Type 2/9 applications, an email will only be sent out if the PCC code for an application is changed.

d. Email .csv file, which should contain the columns shown in the output file listed in Appendix A

6. Off line, the PCC Coders will enter the data manually into the ICO Module of IMPAC II

7. Applications will continue to appear on the .csv file until a PCC code is entered in IMPAC II, at which time it would disappear from the display

Figure 1:

Figure 2:

Figure 3:

image3.png image1.png image2.png