RFP_Attachment_4_-_Task_Order_0001_PWS_Amendment_0001.pdf

Performance Work Statement

FY18 DoD Sentiment of OPA Survey Comments: A Pilot

Task Order 1

1.0 INTRODUCTION

The Office of People Analytics (OPA) Data Science Directorate requires an analysis of the current state of sentiment analysis to conduct complex natural language processing (NLP) tasks in support of OPA’s text analytics platform and projects.

2.0 BACKGROUND

The Data Science division is an official DoD Program for full spectrum data science research. Its mission is to augment internal OPA analytic outcomes and enhance external OPA analytic impacts from data to decision. A Data Science primary research objective is to conduct state-of-the-art natural language processing on DoD projects. OPA has collected over 500,000 text elements from service members from

2002 to present. This information is a resource and respondents have given their valuable time to provide insight for the Department of Defense. This resource is under used by the clients because of the time required to organize and gain a complete understanding of the comments. The Data Science division is augmenting the comment analysis process through automation by embracing state of the art tools and methods. One specific need of the OPA Data Science division is for a text analytic industry standard research and hand comment labeling. Industry research provides insight on how text analytics is being functionalized in the industry and comment labels will provide a supervised learning method for further algorithm development.

3.0 SCOPE

The Contractor shall provide all personnel, equipment, materials and other items necessary to perform sentiment analysis literature review and a pilot study including hand labeling of a subsample of survey responses to inform model development and validation at specified metric given the topic.

4.0 REQUIREMENTS

The Contractor shall:

4.1 Attend a kickoff meeting with Data Science personnel in person or via teleconference. The purpose of the kickoff meeting is to clarify Data Sciences’ objectives and to discuss how the Contractor will address those objectives. The Contractor shall provide Data Science a written summary of the kick-off meeting discussion along with a plan for research.

4.2 Provide a literature review detailing state of the art sentiment analysis. The literature review shall include bag of words vs. other state-of-the-art methods and also functionalizing NLP tasks in business settings. Provide a recommended method to Data Science for pilot testing and how to create products that will use sentiment information to add value to client needs and objectives.

4.3 Using a sample of 6000 text comments identified by the Data Science Division, hand label all

6000 text comments and then further subdivide the sample into n = 4000 for a training set, n = 1000 for a hold-out test set, and n = 1000 for a validation set.

4.4. Additionally hand-label 30,000 comments by topic areas (e.g., transgender, sexual assault, training, and/or don’t ask don’t tell). The Data Science Division will hold a workshop on how to code comments for consistency.

4.5 Perform an analysis on the training set (from PWS 4.3) using a “bag of words” approach and then pilot other state-of-the-art methods using labeled data as identified in 4.2 in R/Python. The results of the modeling will be set at an agreed upon benchmark, determined from 4.2. Provide draft and final technical reports.

4.6 Provide organized, final script/code.

4.7 Provide a technical report detailing the recommended methodology performance vs. the bag of words approach including the sentiment analysis results and a demonstration how the metric validation was achieved.

4.8 Provide related briefing materials demonstrating the value of the other state-of-the-art methods of words methodology, in particular how this information / tools can be used to meet DoD client objectives.

4.9 Provide Monthly Status Reports, which includes the status and relevant issues associated with each tasking, a description of the remaining work to be done on the task, and total funds expenditure to date.

4.10 Contractor Manpower Reporting

Report all contractor labor hours (including subcontractor labor hours) required for performance of services provided under this contract for the Government via a secure data collection site. The Contractor shall completely fill in all required data fields using the following web address: http://www.ecmra.mil/.

Reporting inputs shall be for the labor executed during the period of performance during each

Government fiscal year (FY), which runs from 1 October through 30 September. While inputs may be reported any time during the FY, all data shall be reported no later than 31 October of each calendar year.

Contractors may direct questions to the help desk at: http://www.ecmra.mil/.

5.0 DELIVERABLES

Deliverable Ref. Delivery Date (on or about)

Kick-off meeting 4.1 Within 1 week of award (15th June)

Written summary of Kick-off meeting 4.1 2 weeks following kick-off meeting

Literature Review and

Recommendation 4.2

Deliverable schedule set during kick-off meeting

Hand-label 30,000 comments 4.4 01 July 2018

Draft Technical Report 4.5 15 September 2018

Final Technical Report 4.5 15 December 2018

Finalized script and code 4.6 15 December 2018

Draft Briefing Materials 4.8 15 September 2018

Final Briefing Materials 4.8 15 December 2018

Monthly Status Report 4.9 Monthly during PoP

6.0 PLACE OF PERFORMANCE.

The place of performance will be at the government’s worksite in Alexandria, VA.

7.0 PERIOD OF PERFORMANCE.

The period of performance shall be twelve (12) months.

8.0 GOVERNMENT FURNISHED PROPERTY (GFP)

See contract PWS paragraph 13.

9.0 SECURITY CLEARANCE.

No classified work shall be done under this task order.