HR001118S0047.pdf

HR001118S0047 SCORE 1

Broad Agency Announcement Systematizing Confidence in Open Research and Evidence

(SCORE)

Defense Sciences Office

HR001118S0047

June 12, 2018

HR001118S0047 SCORE 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. Deliverables H. Government-furnished Property/Equipment/Information I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Other Eligibility Requirements

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Frequently Asked Questions (FAQs) B. Collaborative Efforts/Teaming C. Proposers Day

BAA Attachments:

Attachment A: Abstract Summary Slide Template Attachment B: Abstract Template Attachment C: Proposal Summary Slide Template Attachment D: Proposal Template Volume 1 Technical & Management Volume Attachment E: Proposal Template Volume 2 Cost Volume Attachment F: Proposal Template Cost Summary Spreadsheet Attachment F-2: Proposal Template Volume 2 Cost Breakdown Attachment G: Proposal Template Volume 3 Administrative & National Policy Volume Attachment H: Milestones and Deliverables Summary Table

HR001118S0047 SCORE 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Systematizing Confidence in Open Research and Evidence

(SCORE)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001118S0047

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: June 12, 2018 o Proposers Day: June 8, 2018. See Section VIII.C.

o Teaming Profiles: June 15, 2018, 4:00 p.m. EST See Section VIII.B.

o Abstract Due Date:

For TA1 Abstracts: June 20, 2018, 4:00 p.m. EST For TA2 Abstracts: June 20, 2018, 4:00 p.m. EST For TA3 Abstracts: November 1, 2018, 4:00 p.m. EST o FAQ Submission Deadline TA1 and TA2: July 20, 2018, 4:00 p.m. EST See Section

VIII.A.

o FAQ Submission Deadline TA3: October 17, 2018, 4:00 p.m. EST See Section VIII.A.

o Full Proposal Due Date:

For TA1 Proposals: August 1, 2018, 4:00 p.m. EST For TA2 Proposals: August 1, 2018, 4:00 p.m. EST For TA3 Proposals: December 12, 2018, 4:00 p.m. EST

Anticipated Individual Awards: DARPA anticipates multiple awards under each

Technical Area (TA)

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements or other transactions

Agency contacts o Technical POC: Adam Russell, Program Manager, DARPA/DSO o BAA Email: SCORE@darpa.mil mailto:SCORE@darpa.mil

HR001118S0047 SCORE 4

o BAA Mailing Address:

DARPA/DSO

ATTN: HR001118S0047

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Teaming Information: See Section VIII.B for information on teaming opportunities.

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.A for further information.

http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name

HR001118S0047 SCORE 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 CFR § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR 15.4 for procurement contracts.

A. Introduction The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals for the development and deployment of automated tools to assign Confidence Scores (CSs) to different kinds of Social and Behavioral Science (SBS) research results and claims. CSs are quantitative measures that should enable someone to understand the degree to which a particular claim or result is likely to be reproducible and/or replicable. These tools will assign explainable CSs with a reliability that is equal to, or better than, the best current human expert methods and will enable a consumer of SBS research to quickly calibrate the level of confidence in the Reproducibility and Replicability (R&R) of a given SBS result or claim. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

B. Background

Given the accelerating sociotechnical complexity of today’s world—a world that is increasingly connected but often poorly understood—there are growing calls to more effectively leverage Social and Behavioral Sciences (SBS) to help address critical complex national security challenges in the Human Domain.i Current examples include the National Academies’ ongoing “Decadal Survey of Social and Behavioral Sciences for Applications to National Security,”ii the Minerva Research Initiative,iii and a host of other efforts to use SBS to develop solutions for a wide range of national security challenges, including enhancing deterrence,iv supporting stability,v increasing trust and influence,vi reducing extremism,vii and enhancing “social-behavioral modeling.”viii

The pressing importance of addressing national security challenges means that the Department of Defense (DoD) would like to leverage SBS research to design plans, guide investments, assess outcomes, and build models in order to make decisions in the Human Domain.

For the purposes of the SCORE Program, the Social and Behavioral Sciences (SBS) include those major SBS disciplines chiefly focused on understanding human social systems and behaviors, such as sociology, political science, economics, psychology, etc., as well as their respective sub-disciplines, such as computational social science, behavioral economics, social psychology, etc.

HR001118S0047 SCORE 6

Unfortunately, a number of recent empirical studies and meta-analyses have revealed that many SBS results and claims vary dramatically in terms of their ability to be independently reproduced and/or replicated.ix This is worrying since the degree of R&Rx of the SBS research upon which these decisions and models may depend can have real world implications. Research that appears confirmatory when it is in fact exploratory,xi suffers from other Questionable Research Practices (QRPs),xii or makes assertions or predictions that are unlikely given known practical or theoretical limitations,xiii can lead an SBS consumer to have inappropriate confidence in that research.

Additionally, current approaches to estimating the R&R for SBS research are often slow, inefficient, opaque, and unreliable.xiv,xv Taken in the context of growing numbers of journals, articles, and preprints, this current state of affairs could result in an SBS consumer mistakenly over-relying on weak SBS research or dismissing strong SBS research entirely.

Efforts to address these concerns include new calls for, and early investments in, methods for identifying, promoting, and rewarding best research practices and new incentive structures among different communities in the hope that these changes may increase the R&R of future SBS research.xvi However, there is generally widespread recognition that substantive improvement, if possible, will take significant time and resources to achieve.xvii

DARPA hypothesizes that an effective automated tool for helping an SBS consumer rapidly, accurately, and repeatedly calibrate the confidence they should have in research R&R would have significant positive impacts for increasing the effective use of the SBS literature to address Human Domain challenges. By identifying, aggregating, and analyzing a wide variety of signals, such a tool could provide expert-like “Confidence Scores” (CSs), or quantitative measures, that enable a user to understand the degree to which a particular claim or result is likely to be Reproducible and/or Replicable. DARPA anticipates that CSs may be derived from signals that may include (but not be limited to): evidence of one or many QRPs, retractions, evidence of publication bias or file drawer problems in a topic or field, (un)successful efforts at reproducing or replicating similar claims, (un)declared conflicts of interest, public criticism or xviii popular press coverage, sentiment analysis and social media/blog posts, post-publication review and crowd-sourcing,xix social networks of reviewers or co-authors, incentive structures, journal transparencyxx, ratio of exploratory to confirmatory studies in a topic or area, and/or the use of pre-registration or Registered Reports.

Further, such a tool should enable an SBS consumer to rapidly update CSs based on changes in those signals. Finally, such a tool should also explain why it has assigned the CSs that it has, including which signals it may have used and how it used them, allowing an SBS consumer to understand the results and appropriately leverage the SBS claim for use in the Human Domain.xxi

Given the lack of widely agreed upon definitions of R&R, the SCORE Program will define Reproducibility as “the degree to which a particular research claim or result can be independently computationally reproduced by reanalyzing the study using the original data.” Likewise, the SCORE Program will define Replicability as “the degree to which a particular research claim or result can be independently replicated by rerunning the study as similarly as possible with a new sample.”

HR001118S0047 SCORE 7

C. Program Description/Scope The vision of the SCORE program therefore is to test, validate, and demonstrate the feasibility and utility of one or more automated tools for assigning CSs to a wide range of SBS claims. SCORE seeks to realize this vision through a two-phase program. The first Phase will focus on developing the initial Common Task Framework (CTF) for SCORE, with a curated dataset of SBS research claims using methods for rapidly but accurately labeling those data with human expert CSs. Early algorithm development will also occur in Phase 1 as proof of principle. In Phase 2, performers will use those labeled data to train and test algorithms that will assign quantitative CSs. These algorithm-based CSs will be compared to CSs assigned by the best performing human expert methods to see how they overlap. If successful, this program will enable SBS consumers within the DoD and the U.S. Government to use SCORE algorithms to quickly, accurately, and iteratively calibrate the confidence they should have in a particular SBS claim’s R&R. SCORE deliverables should have significant positive impact on DoD and USG’s abilities to leverage SBS for modeling, planning for, and operating in, the Human Domain.

D. Program Structure SCORE is a 36-month program, comprising two phases with durations of 18 months each.

Figure 1: SCORE Program Structure

To achieve its vision, the SCORE program will fund research in three Technical Areas (TAs), with an independent Test and Evaluation (T&E) team providing oversight. The three TAs are:

• TA1: Data

• TA2: Experts

• TA3: Algorithms

HR001118S0047 SCORE 8

As outlined in Figure 1 above, SCORE will kick off by having its TA1 and TA2 community of performers and relevant subject matter experts agree upon common CSs criteria, scales, and metrics for the program. These will be determined in Month 1 of the program at the SCORE kickoff meeting. The kickoff meeting for TA3 performers will occur in Month 7 of the program.

Phase 1 of the SCORE program will then focus on TA1 teams developing the CTF with a curated SBS research dataset, while TA2 teams will focus on the implementation of methods for rapidly assigning expert CSs to articles. TA1 teams will also empirically evaluate the accuracy of each TA2 team’s expert-assigned CSs by independently reproducing/replicating a representative subsample of the CTF dataset and comparing outcomes with TA2 predictions. TA3 teams will use some TA1-supplied data (plus any proposed supplementary data) to conduct early algorithm development to provide proof of principle by Month 18.

If Phase 1 is successful in developing the initial CTF with validated CSs, Phase 2 will see TA3 teams develop out algorithms that are capable of efficiently and accurately assigning CSs with convincing overlap with the best performing TA2 method(s). These algorithms should allow SBS consumers in DoD to assign and update a claim’s CSs and understand how the algorithm(s) came to the specific quantitative assessment(s).

Figure 2: SCORE Technical Area Interactions

DARPA is soliciting proposals for TA1, TA2, or TA3 but is not soliciting proposals for participation on the T&E team. Proposals to any of the TAs must address the full program timeline. As stated in Part II, please note that TA1 and TA2 will have an earlier proposal due

HR001118S0047 SCORE 9

date and kickoff date than TA3. As discussed further in Section E, TA3 proposals should be submitted in accordance with the TA3 proposal deadline listed in Part I, and TA3 proposals should reflect a six-month later start: TA1 and TA2 teams will start work in Month 1, while TA3 teams will start work in Month 7. Proposers1 should structure their proposals with Phase 1 as the base period (18-months for TA1 and TA2; 12-months for TA3), and Phase 2 as an option for funding (18-months for all TAs).

Each proposal should only address a single TA. Please note that to avoid conflicts of interest among TAs, no person or organization may be a performer on more than one TA, either as a prime or sub-contractor. The same person or organization may be on multiple proposals if those proposals are submitted to the same TA. However, proposers should provide sufficient approaches for managing potential conflicts of interest and/or firewalls among different teams/proposals in that TA and provide evidence that they have sufficient resources to mitigate any technical, cost, and/or schedule risk should they or their team members be on multiple proposals selected for negotiation.

E. Technical Area Descriptions

1. Technical Area 1 (TA1): Data

The goal of TA1 is to provision data for the program by curating SBS research claims, including (but need not be limited to) journal articles, previous R&R studies, abstracts, and preprints, into a single CTF dataset for TA2 and TA3 teams. In doing so, TA1 teams will have prime responsibility for developing the CTF for the SCORE program by providing training/test data to TA2 and TA3 performers as well as empirically validating the R&R of a representative subsample of claims in order to evaluate the accuracy of TA2 CSs expert methods.

Proposals: TA1 proposals must include clear, credible, and (where appropriate) quantitative descriptions that include (but need not be limited to) the following tasks deemed critical for SCORE program success:

Identification of, and justification for, SBS topics/literatures to be curated as well as non- SBS topics/literatures that might be used to assess the generalizability of TA2/TA3 methods (e.g., assigning CSs to biomedical or neuroscience research). Proposals should also identify research areas or claims that are likely to be impossible or impractical for their approach to CTF creation;

Recommendations for what should constitute an SBS research “claim” (e.g., article synopsis, abstract, specific conclusion, discrete statistical relationship, model) along with proposed approach for efficiently and reliably capturing these “claims” from various SBS research articles for TA2/TA3 use;

1 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

HR001118S0047 SCORE 10

Proposed criteria for operationalizing CSs for R&R, such as:

o Factors that should be used for scoring R&R;

o Relative weights assigned to R&R in order to come up with an aggregated CSs;

o Type and format of a CSs scale (e.g., range of 1-10 versus 1-100);

o Proposed visualizations (e.g., possible colors, chart types).

Complete and feasible description of the curation process, including anticipated level of manual versus automated processes in each Phase;

Proposed approach for representative sampling of CTF claims to be empirically validated for evaluating TA2 accuracy in Phases 1 and 2;

Identification of literature or data repositories to be leveraged and a detailed data management plan (see Section I.I.3), with credible mechanisms to curate, store, and release data for TA2 CSs assignment and train/test data for TA3;

Proposed schedule (or approach to developing a schedule) for data transfer and/or making data accessible to TA2 and TA3 teams during Phase 1 and Phase 2 efforts (e.g., daily, weekly, bi-weekly)

Principled plan for the empirical evaluation of SBS research studies in each phase, including the anticipated relative number of replications, number of reproductions (where, for example, replication may be infeasible), and number of joint replication/reproductions;

Approaches for Phase 2 testing of TA3 algorithms’ updating capability and ability to detect efforts to game the algorithms (including what kind and how much real or synthetic data would be injected/ablated and proposed metrics for evaluating TA3 updating/gaming detection success);

Identification of any anticipated Human Subjects Research (HSR) and Human Research Protection Office (HRPO) requirements and plans to expeditiously receive necessary approvals that enable reaching the milestones on, or before, schedule (see Section VI.B.6 for more information on HSR requirements);

Anticipated/recommended data output formats;

Identification of specific risks to the proposed approach(es) and credible mitigation plans;

Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in TA1 relevant areas in order to increase confidence in the specific proposed approach(es).

Performance Metrics: To assess TA1 performers’ progress towards realizing SCORE’s vision, TA1 teams will be assessed according to the following metrics:

Curation Rate – Ability to curate SBS research articles efficiently and accurately.

Proposers should offer credible approaches to achieve a curation rate of at least 3,000 SBS articles and claims per 12 months by Month 12, or if this is not possible, proposers should provide a reasonable justification and mitigation strategy that can still realize SCORE’s vision. In addition, proposers should offer credible plans for an increased curation rate in Phase 2, in order to exceed 3,000 articles and claims per 12 months. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

HR001118S0047 SCORE 11

Empirical Evaluation – Capabilities supporting the empirical evaluation of the R&R of at least 200 representative research claims in each phase. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

Platform/Methodology Efficacy – Ability to maintain and continually grow a CTF of curated SBS research datasets that are useful for SCORE TA2/TA3 performers.

Capabilities should also include rapid (preferably semi-automated) processes for cleaning and aligning articles as well as developing representative samples. Credible approaches that make a compelling argument that this CTF platform/method may be useful beyond the SCORE program may be more favorably reviewed.

Nominate TA1 Metrics: Proposers should nominate additional metrics for evaluating their specific capabilities and performance in terms of the TA1 goals. Examples might include metrics for evaluating TA1 R&R empirical validation methods and outcomes, metrics for evaluating TA2 R&R accuracy against TA1 empirical validations, and/or metrics for evaluating TA2 and TA3 CSs overlap using TA1 data. Final TA1 metrics will be decided in coordination with DARPA and the T&E team after selection for award.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA1 teams may be expected to maintain some control during the program period over database contents, platforms, curation processes, etc.

Out of Scope: Use of proprietary data or datasets may significantly impede SCORE program goals, including (but not limited to): limiting TA1 deliverables or teaming capabilities, reducing the Government’s ability to independently evaluate those data, limiting the capabilities of TA2/TA3 teams to conduct their research, and/or restricting the capabilities and evaluation of TA2/TA3 deliverables. Given these concerns, use of proprietary data or datasets may be out of scope for the SCORE program. However, if use of some proprietary datasets is deemed appropriate, licenses should be negotiated that do not prevent SCORE performers, the T&E team, or DARPA from being able to use those datasets to accomplish their respective and collective technical and programmatic goals. SCORE will not fund efforts that propose the use of data or datasets that violate copyright protections or other legal restrictions.

2. Technical Area 2 (TA2): Experts

The goal of TA2 teams is to assist in the creation of the SCORE CTF by using or developing scalable methods and/or platforms for eliciting expert Confidence Scores for the R&R of SBS claims in the TA1 datasets. The focus of TA2 is on assigning human expert CSs to TA1’s data (claims, studies, abstracts) at a rate that is sufficient to enable TA1 to build up the SCORE CTF datasets required for Phase 2. TA2 teams may use innovative mechanisms that enable rapid but accurate TA2 CSs assignments, where accuracy is measured as the ability of TA2 teams to predict the outcomes of TA1’s empirical evaluations of the R&R of a subsample of studies and claims.

Proposals: TA2 proposals must include clear, credible, and (where appropriate) quantitative descriptions that include (but need not be limited to) the following tasks deemed critical for

HR001118S0047 SCORE 12

SCORE program success:

Identification of an approach for acquiring expert assessments via prediction markets, expert surveys, online games, and/or other innovative and technically compelling approach(es);

Proposed approaches to capturing and conveying CSs “forecasts” for TA1 testing (e.g., summed averages, market value, rank order; judgmental reasoning to derive probabilities or CSs), including any approach(es) for capturing expert scoring processes/signals and potentially sharing identified signals and/or expert processes with TA3 teams;

Recommendations for what should constitute an SBS research “claim” for CSs assignment (e.g., article synopsis, abstract, specific conclusion, discrete statistical relationship, model), along with

Proposed criteria for operationalizing expert CSs for R&R, such as:

o Factors that might be used for scoring R&R;

o Relative weights assigned to R&R in order to come up with an aggregated CSs;

o Type and format of CSs scale (e.g., range of 1-10 versus 1-100);

o Proposed visualizations (e.g., possible colors, chart types).

Proposed approach(es) for capturing/comparing the impact of different variables (numbers or kinds of experts, skillsets, infrastructure, etc.) on the performance/accuracy of TA2 methods;

Proposed suggestions for SBS topics or literatures best suited for approach, including identifying potential datasets and/or literature repositories for TA1 to leverage and identifying extensions to other topics/literatures, including non-SBS research;

Identification of any anticipated HSR/HRPO requirements and plans to expeditiously receive necessary approvals that enable reaching the milestones on, or before, schedule;

Proposed flexible mechanisms and/or platforms for receiving/storing TA1 data and returning CSs to TA1 such that TA2 CSs can be easily and persistently associated with specific claims;

Identification of anticipated requirements of/for data sets, structure, formats, and data management plan per Section I.I.3;

Identification of specific risks to the proposed approach(es) and credible mitigation plans;

Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in relevant areas in order to increase confidence in the specific proposed approach(es).

Performance Metrics: To assess TA2 performers’ progress towards realizing SCORE’s vision, TA2 teams will be assessed according to the following metrics:

Evaluation Rate – Ability to assign CSs to SBS research articles efficiently. Proposers should offer credible approaches to achieve an evaluation rate of 3,000 articles per 12 months by Month 12, or if this is not possible, proposers should provide a reasonable justification and mitigation strategy that can still realize SCORE’s vision. In addition, proposers should offer credible plans to allow for an increased evaluation rate in Phase 2, exceeding 3,000 articles per 12 months. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

HR001118S0047 SCORE 13

Accuracy – Ability to assign CSs to SBS claims that are at least 80% accurate in Phase 1 when tested against TA1 R&R empirical evaluations in Month 15. Proposers should also offer credible plans for an improved accuracy rate in Phase 2, ideally exceeding 80% by Month 30. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

Nominate TA2 Metrics: Proposers should nominate additional metrics for evaluating their capabilities and performance in terms of the TA2 goals. Examples might include metrics for estimating cost per CSs assignment per claim and/or metrics for “expert agreement” and interrater reliability. Final TA2 metrics will be decided in coordination with DARPA and the T&E team after selection for award.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA2 teams may be expected to maintain some control during the program period over expert assessments, processes, platforms, etc.

Out of Scope: SCORE will not fund performers that use platforms in a way that may violate or otherwise disregard acknowledged and in good standing IP claims, restrictions, or legal statutes in applicable states or countries.

3. Technical Area 3 (TA3): Algorithms

Successful TA3 teams will develop SCORE algorithms using training data provided by TA1 (as well as other training data they may propose) to automatically assign CSs to held-out TA1 test sets. TA3 algorithms will be evaluated in terms of their CSs overlap with the best performing TA2 CSs, ultimately seeking to achieve 95% confidence in an algorithm’s overlap with those TA2 CSs.

Towards the end of Phase 2, successful TA3 teams will also demonstrate the utility of their algorithms/systems for DoD users and applications.

Proposals: TA3 proposals should include clear, credible, and (where appropriate) quantitative descriptions that include (but need not be limited to) the following tasks deemed critical for SCORE program success:

Credible plan for achieving Phase 1 “proof of principle” by Month 18;

If appropriate, identification of potential training/test data, whether in SBS or elsewhere, that may be used to complement data provided by TA1;

Credible plan for achieving increasing overlap with best performing TA2 expert CSs over the course of each testing and training sprint in Phase 2, reaching 75% confidence of overlap by Month 25, 85% confidence of overlap by Month 30, and 95% confidence of overlap by Month 36;

Proposed approaches to testing TA3 updating capability (including what kind and how much real or synthetic data TA1 might provide, proposed metrics for evaluating TA3 updating success, and innovative but credible approaches to testing TA3 algorithms susceptibility to “gaming” attempts);

HR001118S0047 SCORE 14

Identification of any anticipated HSR/HRPO requirements and plans to expeditiously receive necessary approvals that enable reaching the milestones on, or before, schedule;

Identification of anticipated requirements of/for data sets, structure, formats, and data management plan per Section I.I.3;

Identification of specific platforms and/or software for algorithm development and deployment;

Identification of specific risks to the proposed approach(es) and credible mitigation plans;

Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in relevant areas in order to increase confidence in the specific proposed approach(es).

Performance Metrics: TA3 performers will be assessed according to the following approach capabilities:

Scoring Rate – Ability to evaluate SBS research articles efficiently. Proposers should offer credible approaches to achieving an ability to assign an R&R CSs to a claim within 60 minutes by Month 18, and under 30 minutes by Month 36. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

Overlap with TA2 – Ability to assign Confidence Scores to SBS claims that overlap with best performing TA2 methods. Proposed approach should provide a credible plan for achieving early proof of principle in Phase 1 by Month 18 and offer credible plans for achieving 75%, 85%, and 95% CS overlap over the course of each six-month sprint in Phase 2, respectively. If a proposer believes this metric is not achievable, they should provide a reasonable justification and mitigation strategy. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

Algorithm/Platform Usability – Capability for algorithms to assign and pass algorithm CSs to TA1 teams efficiently and the degree to which algorithms are transparent and their CSs assessments are interpretable/tailorable as evaluated by an above average score on a System Usability Scalexxii by the end of Phase 2.

Nominate TA3 Metrics: Proposers should nominate additional metrics for evaluating their capabilities and performance in terms of the TA3 goals. Examples may include metrics to evaluate usability of algorithms and supporting platforms and/or estimating cost per CSs assignment per claim. Final TA3 metrics will be decided in coordination with DARPA and the T&E team after selection for award.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA3 teams may be expected to maintain some control during the program period over algorithm code, assessments, processes, platforms, etc.

Out of Scope: “Blackbox” algorithms, whose final functions or outcomes in assigning Confidence

HR001118S0047 SCORE 15

Scores cannot be reasonably explained to a DoD user, are explicitly out of scope for SCORE.

Additionally, SCORE will not fund algorithms developed using hardware and/or software that limit TA3 deliverables to less than Government Purpose Rights, unless strongly justified.

4. Test and Evaluation (T&E)

The SCORE T&E team will assist with coordinating among the TAs, including helping to manage all communications among TAs to ensure accordance with SCORE programmatic goals. The T&E team will also be involved in adjudicating the representativeness of proposed TA1 samples for empirical evaluation, as well as TA2 accuracy as compared to TA1 empirical evaluations, and TA2/TA3 CSs overlap. Lastly, the T&E team will also assist in coordinating TA3 system usability tests and metrics for successful TA3 teams in Phase 2.

F. Schedule/Milestones Proposals to any of the three TAs must address the full program timeline (TA3 proposals, due at the later TA3 proposal closing date stated in Part I, should address the full program timeline assuming a Month 7 kickoff). Proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents an aggressive plan to fully address all program goals, metrics, milestones and deliverables. In addition, the task structure must be consistent across the proposed schedule, Statement of Work (SOW), and cost volume in order to enable DARPA to evaluate the technical, programmatic, and financial credibility of individual tasks as well as the overall approach. Schedules will be synchronized across performers, as required, and monitored/revised as necessary throughout the program. For TA1 and TA2 teams, a target start date of January 2019 may be assumed for planning purposes. For TA3 teams, a target start date of July 2019 may be assumed for planning purposes.

All SCORE TA1 and TA2 performers should expect to attend a kickoff meeting in the Washington, D.C. area. TA3 performers should expect the Month 7Principal Investigator (PI) Meeting to serve as their kickoff. After Month 7, TA3 performers should plan for site visits and PI Meetings as would TA1 and TA2 teams per the program schedule outlined below in Figure 3 and Tables 1 and 2.

DARPA expects all performers to attend PI Meetings every 6 months, as shown in Figure 3. The purposes of the PI Meetings are to (i) provide the Program Manager and other SCORE performers with updates on progress towards milestones and goals; (ii) summarize outstanding technical challenges; (iii) support test and evaluation; and (iv) provide Government and potential transition partners with opportunities to provide input, comments, and suggestions for the SCORE program and its performers. For budgeting purposes, TA1 and TA2 proposers should assume a two-day kickoff meeting, while PI meetings will require three days and will alternate between Washington, D.C. and a west coast location. The Government and T&E team will schedule regular teleconference meetings for progress reporting as well as problem identification and mitigation.

Proposers should anticipate at least one site visit every 6 months by the DARPA Program Manager during which they will have the opportunity to demonstrate progress towards agreed-upon milestones. Additional anticipated programmatic events are included in Tables 1 and 2, below.

HR001118S0047 SCORE 16

For budgeting purposes, all teams should plan for six three-day PI meetings over the course of 36 months: three meetings in the Washington, D.C. area and three meetings in the San Francisco, CA area. For TA1 and TA2 teams, these meetings are in addition to a two-day kickoff meeting in the Washington, D.C. area (TA3 teams will attend the Month 7 PI Meeting, which will serve as their kickoff).

Figure 3: SCORE Program Gantt Chart

Table 1: Technical Goals by Phase Technical Area Phase 1 Phase 2

TA1:

Data

• Curate a database of SBS research to develop a common task framework (CTF)

• HSR secondary approvals

• Curation at a rate of 3,000 articles or more per 12 months

• Empirical evaluation of representative sample of claims at a rate of 200 or more per 15 months

• Continue curation of CTF database

• Curation at a rate of 3,000 articles or more per 12 months

• Empirical evaluation of at least 200 representative samples of SBS claims

TA2:

Expert Assessments

• Launch methods for rapidly getting expert assessments of SBS research studies

• HSR secondary approvals

• Assessment at a rate of 3,000 articles or more per 12 months

• Assessment at a rate of 3,000 articles or more per 12 months

• Accuracy greater than 80% as compared to TA1 empirical evaluations

1a

1b

1c

2a

2b

HR001118S0047 SCORE 17

• Accuracy of at least 80% as compared to TA1 empirical evaluations

TA3:

Algorithm Assessments

• Develop algorithms for rapid assessment of SBS research studies based on endogenous and exogenous signals

• Assessment at a rate of 1 article per 60 minutes

• Demonstration of algorithm’s efficacy and explainability

• Assessment at a rate of 1 article per 30 minutes

• Increase percentage of distribution overlap between TA2 and TA3 assessments over the course of three consecutive sprints:

o Sprint 1 – 75% overlap o Sprint 2 – 85% overlap o Sprint 3 – 95% overlap

• Demonstration of system usability for both experts and non-experts

Table 2: Program Events by Month Months After Award

Event Description

PHASE 1

1 Program

Kickoff

• All teams agree on SCORE R&R criteria, scales, definitions

• TA1 and TA2 teams present technical approach and work plan

• T&E team provides test and evaluation plan, candidate metrics

1 HSR Process • TA1 and TA2 teams submit protocols for IRB approval

1 TA1/TA2

Start Work

• TA1 teams begin data curation and work with T&E to develop a representative sample for empirical evaluation

• TA2 launches platforms for scaling up CSs assignments

3 Site Visits • TA1 and TA2: demonstrate progress towards milestones 6 Milestones 1a and 2a

• TA1 and TA2 receive all secondary HSR approvals for Phase 1

7 PI meeting and TA3 Kickoff

• TA1 and TA2: review technical progress

• TA3 launches platforms for rapidly assessing SBS research articles

• TA1 provides preliminary training data to TA3

9 Site Visits • All teams: demonstrate progress towards milestones 12 PI meeting • All teams: review technical progress 12 Milestones 1b and 2b

• TA1 teams demonstrate curation rate of at least 3K claims per 12 months

• TA2 teams demonstrate CSs assignments at rate of 3K claims per 12 months

15 Milestone 1c • TA1 submits results of at least 200 R&R empirical evaluations for assessing TA2 accuracy to T&E

15 Milestone 2c • T&E team evaluates TA2 accuracy against TA1 empirical evaluations

2c

3a

3b

3c

3d

HR001118S0047 SCORE 18

15 Site Visits • All teams: demonstrate progress towards milestones 18 Milestone 3a • TA3 demonstrates algorithm efficacy and explainability 18 PI Meeting -

End of Phase

• Phase 1 results for all TAs presented

HR001118S0047 SCORE 19

PHASE 2

18 Empirical

Evaluations

• TA1 begins second round of R&R empirical evaluations for assessing TA2 accuracy

21 Site Visits • All teams: demonstrate progress towards milestones 24 PI meeting • All teams: review technical progress

• T&E presents results of TA3 overlap with TA2 24 Milestone 3b • TA3 completes Sprint 1 and achieves 75% confidence overlap with best performing TA2 CSs methods 27 Site Visits • All teams: demonstrate progress towards milestones 30 Empirical

Evaluations Complete

• TA1 submits results of at least 200 R&R empirical evaluations for assessing TA2 accuracy to T&E

30 PI meeting • All teams: review technical progress

• T&E presents results of TA3 overlap with TA2, TA2 accuracy compared with TA1 R&R empirical evaluations

30 Milestone 3c • TA3 completes Sprint 2 and achieves 85% confidence overlap with best performing TA2 CSs methods

30 TA3 Usability • TA1 and TA3 begin usability testing re: algorithm updating and detecting of gaming, coordinated by T&E

33 Site Visits • All teams: demonstrate progress towards milestones 36 Milestone 3d • TA3 completes Sprint 3 and achieves 95% confidence overlap with best performing TA2 CSs methods 36 Final PI

Meeting

• All teams: review final product deliverables

• T&E presents results of TA3 overlap with TA2, TA3 usability testing

G. Deliverables DARPA expects performers to provide at a minimum the following deliverables:

Comprehensive monthly technical reports due within ten days of the end of each given month, describing progress made on the specific milestones as laid out in the SOW.

A phase completion report submitted within 30 days of the end of each phase, summarizing the research done.

Other negotiated deliverables specific to the objectives of the individual efforts. These may include:

o For TA1: CTF datasets, access to database, results of empirical evaluations, and other deliverables stemming from the research funded under SCORE o For TA2: access to expert assessment tools/platforms and other deliverables stemming from the research funded under SCORE, including data and Confidence Scores o For TA3: algorithms and systems, and other deliverables stemming from the research funded under SCORE, including data and Confidence Scores o For all: publications, intermediate and final versions of software libraries, code, and APIs, including documentation and user manuals and/or a comprehensive assemblage of design documents, data, and results.

Reporting as outlined in Section VI.C.

HR001118S0047 SCORE 20

H. Government-furnished Property/Equipment/Information Performers are not expected to need Government-furnished Property (GFP). Any requests for GFP must be accompanied by a credible justification and explanation of why other options would not be suitable for the proposed work.

I.Other Program Objectives and Considerations

1. Collaboration

DARPA expects all performers to work collaboratively with one another to realize the program objectives outlined herein, so proposers should carefully review the goals for the entire program in order to fully understand the context of each program objective within the overall program structure. Furthermore, throughout development of program technologies, it will be necessary for all performers to share relevant information regarding their technology development to support the larger program goals. For example, TA2 performers will need access to the CTF datasets developed by the TA1 performers. All proposals must clearly describe plans for interfacing and integrating their proposed technologies/approaches with other performers. Proposals that fail to include interface and integration plans may be deemed non-conforming2 and removed from consideration for award.

DARPA expects SCORE performers to collaborate closely with the T&E team, so proposals should describe approaches for providing the T&E team with access to all necessary and relevant software, data, and domain knowledge.

2. Intellectual Property As discussed above, there is an emphasis on creating and leveraging open source technologies and architectures, making data sharing and collaboration key aspects of this program. Therefore, intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.4 for more information related to intellectual property.

3. Data Management Plan

This BAA requires a Data Management Plan (DMP) be included as part of the proposal submission. DARPA/DSO’s view of what constitutes the scope of applicable data products to be covered in a DMP is quite broad, potentially encompassing all digital activity related to a project.

DARPA’s approach to an effective and practical DMP is predicated with two goals:

First, data are increasingly the key products of research and engineering endeavors. To ensure the reproducibility of results and the accessibility of program accomplishments to future users, we require proposers document the necessary and sufficient scope of data that may be applicable to these goals. Performers will be expected to document both the proprietary and nonproprietary products of the program, including but not limited to raw unprocessed data, software source code and executables, build scripts, process sequence, programmatic communication and other

2 “Conforming” is defined as having been submitted in accordance with the requirements outlined herein.

HR001118S0047 SCORE 21

collaboration activities; rarified, experimental, test and measurement data; design of experiments and simulations; models or simulations (computational or mathematical); recordings of various physical phenomena (including images, videos, sensor data, etc.); access to and use of institutional, organizational or scientific community repositories and archives. An important goal of the DMP is to ensure the retention and potential reusability of this information.

Second, when possible, DARPA may also share some or all of the program-generated data with the broader research community as open data (with permission to access, reuse, and redistribute under appropriate licensing terms) to the extent permitted by applicable law and regulations (e.g., privacy, security, rights in data, and export control). The complete scope of program-generated data described above may go considerably beyond the scope of data to be made public. Hence, DARPA expects that as part of a DMP proposers will delineate their specific data products that are suitable for public release and how they intend to capture and represent this information. In this way, it is DARPA’s intention to enable reproducibility of results and establish (or contribute

to) digital collections that can advance this and other scientific fields. Note that this provision is not meant to require disclosure of otherwise proprietary internal component or process intellectual property, but to ensure all performers can meet the overall program objectives.

Proposals submitted without a DMP may be deemed non-conforming and may not be reviewed.

Note that the DMP does not count against the page-limit for Volume 1. A DMP should include enough detail to ensure that the data products delivered to DARPA (or made public) are adequate for use by an independent third party, both for further exploratory research as well as for reproducibility and verification of the scientific results. As this is an effort that is required to execute the program, DARPA does not expect the existence of a DMP to produce additional cost burden on performers for data management requirements during or after the period of performance.

II. Award Information A. General Award Information

DARPA anticipates multiple awards.

The level of funding for individual awards made under this BAA will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous to the Government, all evaluation factors considered. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund awards in increments with options for continued work at the end of one or more

HR001118S0047 SCORE 22

phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or other transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program.

For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared…