SOR For Posting.pdf

Revised June 2023

Statement of Requirements (SOR) Open Source Mapping of the Information Environment

Department of Defense Analysis Naval Postgraduate School

1.0 Background/Introduction:

This project is part of ongoing research efforts conducted through the Department of Defense and Analysis at the Naval Postgraduate School. Our research aims to provide enhanced understanding of fundamental social dynamics, to model the dynamics of influence in the information environment, over time and across countries.

2.0 Scope:

To support this research, we seek to acquire a global historical archive of social media data, providing the full text of all available public online media posts, or an unbiased sample of all available public online media posts, across all countries and languages, across multiple social media platforms.

3.0 Tasks:

The contractor shall perform the following task(s) as follows:

3.1 Delivery of electronic files containing full records from a global database of social media posts meeting the minimum technical requirements (5).

4.0 Deliverables:

The contractor shall be responsible for providing the following deliverables:

4.1 Global Social Media Archive

Database records must provide the full text and all associated meta-data for an unbiased sample of posts across at least 3 global social media platforms, over at least a 6-month period, unfiltered by country, language, search terms, or any other factors that would systematically bias the representativeness of the records provided in the historical archive.

5.0 Minimum Technical Requirements:

The database must meet the following minimum specifications:

5.1 Record Requirements:

5.1.1 Platforms Covered: The archive must provide records from at least 3 global social media platforms, each with at least 200 million unique registered users per platform.

5.1.2 Minimum Time Period Covered: 6 months. Time period covered by the archive must begin no later than 10/1/2021 and must end no sooner than 3/31/2022.

5.1.3 Minimum Number of Total Records: 6 billion

5.1.4 Sampling rate: The archive must provide at least 5% of the total volume of publicly available messages from each of the sampled platforms.

5.1.5 Sampling mechanism: If less than 100% of publicly available messages from a given platform will be included, proposals must include a description of how sampling will be performed, including whether sampling was conducted fully randomly, or through some other means, and must confirm that the sampling does not rely on any systematic filtering or exclusions based on message content, search terms, language, or geography.

5.1.6 Each record in the archive must provide the full text of an online social media post, unaltered from its original content and formatting, with all publicly available meta-data, including country, language, hashtags, location, handle, timestamp, and URLs, that were associated with the original posting. Proposals must include a full listing of the fields that will be included from each platform.

5.1.7 Approximate location information, such as self-reported user hometowns, or other publicly available geo-location information, must be included for at least 20% of the records.

Revised June 2023

5.1.8 All records must consist exclusively of publicly available information. No private communications or private user data will be included.

5.1.9 All data must be collected and delivered in compliance with applicable regulations, including the GDPR and CCPA, and in compliance with the Terms of Service of the recorded platforms.

5.2 Delivery Requirements

5.2.1 Data will be delivered as compressed text files, using UTF-8 encoding, with all records in JSON format.

5.2.2 Data will be delivered in bulk, as a one-time purchase of a single historical archive. Data will be stored on an ongoing basis by the customer, with no ongoing subscriptions or services, following completion of the delivery.

5.2.3 Data may be transmitted either via secure online hosting (e.g. AWS or SFTP), or via shipping of physical storage media (e.g. portable hard drives).

5.2.4 For transmission mechanisms requiring fixed time periods (e.g. online hosting), vendor must allow at least 30 days, following initial delivery, for data to be transferred and stored by the customer, and vendor must ensure that sufficient transmission bandwidth will be available to allow transferring the archive within this 30-day period.

5.2.5 Data will be licensed exclusively for non-commercial, academic research conducted by NPS faculty and students and NPS research sponsors.

6.0 Delivery: An award will be made NLT 30 September 2024. ARO is 30 days from date of award.

7.0 Delivery Address:

Naval Post Graduate School (NPS) 1 University Circle Monterey, CA 93943-0005