SOW CF version Generic (20 Jul 2022)v2.pdf

FOR PUBLIC RELEASE

Defense Threat Reduction Agency 8725 John J. Kingman Drive, MS 6201

Fort Belvoir, VA. 22060-6201

STATEMENT OF WORK (SOW)

Forecasting Methodologies, SaaS Subscription

Security and Counterintelligence Division Counterintelligence Operations

20 July 2022

1 Introduction

1.1 Mission

The Defense Threat Reduction Agency (DTRA) enables Department of Defense (DoD), the U.S.

Government, and International Partners to counter and deter Weapons of Mass Destruction. The Operations and Integration Directorate is responsible for counterintelligence support to DTRA. DTRA counterintelligence programs include Countering Weapons of Mass Destruction.

1.2 Background

The Defense Threat Reduction Agency (DTRA) is the DoD lead Combat Support Agency for combating weapons of mass destruction (CWMD). DTRA’s Operational Analysis Department (OI-OA) is responsible for providing CWMD planning support and analysis for US Combatant Commands (CCMDs). DTRA’s OI-OA requires an application to gather internal forecasts using prediction markets and other methods to uncover and quantify ground truth.

1.3 Objective

This effort will be used to research and test the operational feasibility of using crowd-sourcing techniques to improve forecasting methodologies. This will support finding previously unknown WMD threats, improving forecasting threat development timelines and event horizons, and improving DTRA’s support to CCMD planning cells. The goal of this effort is to leverage the results of research funded by IARPA (part of ODNI) in their Aggregative Contigent Estimation (ACE) program which developed processes for crowdsourced geopolitical forecasts from non-experts.

2.0 Software as a Service (SaaS) Forecasting Methodologies Tool Capabilities The desired tool provides the capability to create forecasting questions that will allow team members to make probabilistic forecasts that can be objectively scored. The platform is utilized to post questions which allows analysts to make and revise forecasts that can be aggregated into a consensus forecast. When a question resolves, a score can be calculated for each individual and the team as a whole. The platform is utilized to generate these crowdsourced forecasts, to collect rationales to understand why people have made the forecasts they have, and to provide feedback to analysts to help them improve their forecasting.

2.1 The Software as a Service (SaaS) Tool will allow DTRA personnel to:

• Employ crowdsourcing methodologies for evaluating the likelihood of WMD threats.

• Improve forecasting threat development timelines and event horizons.

• Improve DTRA's support to CCMD.

• Access Administrative Training (Training of Software).

• Access Training and Consulting (Training and Consulting of Software).

• Have SaaS licensing for up to 200 (two hundred) users for each period of performance (PoP).

FOR PUBLIC RELEASE

3.0 Requirements

• A private web-based probabilistic crowdsourced forecasting platform only accessible by our organization's users who are logged in.

• The platform shall enable the creation of scorable forecasting questions (including Yes/No, multiple bin, and ordinal bin) by administrators.

• The platform shall allow 50 to 200 human forecasters to log in, to make new or revised probabilistic forecasts on forecasting questions with rationales. Forecasters shall be able to review other forecasters' forecasts and rationales, comment on them, and to see current and historic trends in forecasted probabilities for each forecasting question.

• The platform shall score individual forecasts using standard and ordinal Brier scores and the Relative variants of each. The platform shall also score consensus forecasts using standard and ordinal Brier scores.

• The platform shall provide overall and per-question leaderboards, and support distinct forecasting seasons and challenges consisting of subsets of forecasting questions specified by the administrators.

• The platform shall provide search and browse features for forecasters to find questions, forecasts, and comments/rationales efficiently.

• The platform shall provide an Application Programming Interface (API) and reporting capabilities for administrators to summarize and analyze results, to include individual forecasts, consensus forecasts, per question forecasts, accuracy reports, and daily consensus forecasts.

• The platform shall provide mechanisms to automatically email forecasters with site announcements, and notifications about new questions, resolved questions, forecasting reminders, and comments in response to or mentioning the forecaster.

• The platform shall provide mechanisms for the publication of blog posts for forecasters.

• For continuity of ongoing forecasts, it is desirable the platform be able to support approximately

120 open questions from existing commercially available platforms -- displaying historical user forecasts and rationales and enabling continued forecasting for the duration of the questions.

• For continuity of existing resolved forecasts, it is desirable the platform be able to incorporate the results of approximately 350 previously resolved forecasts from commercially available forecasting platforms for search and browsing, and into ongoing leaderboards.

4.0 Security

The provided data will come from UNCLASSIFIED, commercially available sources. The vendor will not have access to classified U.S. Government information.