StatementOfRequirements.pdf

Office of Contracting and Logistics Management Revised 05 Mar 2015

Statement of Requirements

Global Social Media Archive Department of Defense Analysis

Naval Postgraduate School

1.0 Background/Introduction:

This project is part of ongoing research efforts conducted through the Department of Defense and Analysis at the Naval Postgraduate School. Our research aims to provide enhanced understanding of fundamental social dynamics, to model the evolution of linguistic communities, and emerging modes of collective expression, over time and across countries. As a central requirement for this research, we seek to acquire a large-scale global historical archive of social media data, providing the full text of all public social media posts, across all countries and languages covered by the social media platform. We aim to use this research to advance knowledge through scientific publications.

This data will also be used for pedagogical purposes in the classroom, giving students new opportunities for thesis research and the development of “big data” analytic skills.

2.0 Minimal Technical Requirements/Salient Characteristics:

Given the aims of our research, a core requirement is that the records must provide the full text and all associated meta-data for all publicly available posts on a single social media platform, over a specified historical period, unfiltered by country, language, search terms, or any other factors that would systematically bias the representativeness of the records provided in the historical archive.

In addition, the archive must meet the following minimum specifications:

• Minimum Time Period Covered: 7/1/2014 - 12/31/2016

• Minimum Number of Total Records: 350,000,000,000

• Platform Requirements:

o Data must consist of records drawn from a single social media platform.

o Data must consist of all publically available messages, comments, or posts transmitted on the platform over the specified time period.

o Data must be drawn from a global platform, including messages from at least 200 million unique users in at least 100 countries, with no single country accounting for more than 30% of users.

o Data must include messages written in at least 60 languages, with at least 50% of the messages written in non-English languages.

• Record Requirements:

o All records must consist exclusively of publically available information. No private communications or private user data will be included.

o Each record in the archive must provide the full text of a social media post, unaltered from its original content and formatting, with all publicly available meta-data, including country, language, hashtags, location, handle, timestamp, and URLs, that were associated with the original posting.

o At a minimum, all records must include the time and date at which each message was sent and the public user handle associated with the message.

o Approximate location information, providing self-reported user hometowns, or other publically available geo-location information, must be included for at least 20% of the records.

Office of Contracting and Logistics Management Revised 05 Mar 2015

• Delivery Requirements:

o Data will be delivered as compressed text files, using UTF-8 encoding, with all records in JSON format.

o Data will be delivered in bulk, as a one-time purchase of a single historical archive.

Data will be stored on an ongoing basis by the customer, with no ongoing subscriptions or services, following completion of the delivery.

o Data may be transmitted either via a secure, dedicated online hosting (e.g. SFTP), or via shipping of physical storage media (e.g. portable hard drives). Transmission via physical media will be preferred.

o For transmission mechanisms requiring fixed time periods (e.g. online hosting), vendor must allow at least 30 days, following initial delivery, for data to be transferred and stored by the customer, and vendors must ensure that sufficient transmission bandwidth will be available to allow transferring the archive within this 30 day period.

o Any transmission or hosting fees associated with data delivery must be covered by the vendor as part of their proposed pricing.