HR001118S0047_TA3_Amendment_clean_version_no_track_changes.docx

Broad Agency Announcement Systematizing Confidence in Open Research and Evidence (SCORE) Defense Sciences Office HR001118S0047 June 12, 2018

HR001118S0047 SCORE

Table of Contents

BAA Attachments:

· Attachment A: Abstract Summary Slide Template

· Attachment B: Abstract Template

· Attachment C: Proposal Summary Slide Template

· Attachment D: Proposal Template Volume 1 Technical & Management Volume

· Attachment E: Proposal Template Volume 2 Cost Volume

· Attachment F: Proposal Template Cost Summary Spreadsheet

· Attachment F-2: Proposal Template Volume 2 Cost Breakdown

· Attachment G: Proposal Template Volume 3 Administrative & National Policy Volume

· Attachment H: Milestones and Deliverables Summary Table

PART I: OVERVIEW INFORMATION

· Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

· Funding Opportunity Title: Systematizing Confidence in Open Research and Evidence (SCORE)

· Announcement Type: Initial Announcement

· Funding Opportunity Number: HR001118S0047

· Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

· Dates (All times listed herein are Eastern Time.)

· Posting Date: June 12, 2018

· Proposers Day: June 8, 2018. See Section VIII.C.

· Teaming Profiles: June 15, 2018, 4:00 p.m. EST See Section VIII.B.

· Abstract Due Date:

· For TA1 Abstracts: June 20, 2018, 4:00 p.m. EST

· For TA2 Abstracts: June 20, 2018, 4:00 p.m. EST

· For TA3 Abstracts: January 31, 2019, 4:00 p.m. EST

· FAQ Submission Deadline TA1 and TA2: July 20, 2018, 4:00 p.m. EST See Section VIII.A.

· FAQ Submission Deadline TA3: February 18, 2019, 4:00 p.m. EST See Section VIII.A.

· Full Proposal Due Date:

· For TA1 Proposals: August 1, 2018, 4:00 p.m. EST

· For TA2 Proposals: August 1, 2018, 4:00 p.m. EST

· For TA3 Proposals: March 12, 2019, 4:00 p.m. EST

· Anticipated Individual Awards: DARPA anticipates multiple awards under each Technical Area (TA)

· Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements or other transactions

· Agency contacts

· Technical POC: Adam Russell, Program Manager, DARPA/DSO

· BAA Email: SCORE@darpa.mil

· BAA Mailing Address:

DARPA/DSO

ATTN: HR001118S0047

675 North Randolph Street Arlington, VA 22203-2114

· DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with- us/opportunities

· Teaming Information: See Section VIII.B for information on teaming opportunities.

· Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.A for further information.

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 CFR § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR 15.4 for procurement contracts.

A. Introduction The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals for the development and deployment of automated tools to assign Confidence Scores (CSs) to different kinds of Social and Behavioral Science (SBS) research results and claims. CSs are quantitative measures that should enable someone to understand the degree to which a particular claim or result is likely to be reproducible and/or replicable. These tools will assign explainable CSs with a reliability that is equal to, or better than, the best current human expert methods and will enable a consumer of SBS research to quickly calibrate the level of confidence in the Reproducibility and Replicability (R&R) of a given SBS result or claim. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

B. Background

Given the accelerating sociotechnical complexity of today’s world—a world that is increasingly connected but often poorly understood—there are growing calls to more effectively leverage Social and Behavioral Sciences (SBS) to help address critical complex national security challenges in the Human Domain.i Current examples include the National Academies’ ongoing “Decadal Survey of Social and Behavioral Sciences for Applications to National Security,”ii the Minerva Research Initiative,iii and a host of other efforts to use SBS to develop solutions for a wide range of national security challenges, including enhancing deterrence,iv supporting stability,v increasing trust and influence,vi reducing extremism,vii and enhancing “social-behavioral modeling.”viii

For the purposes of the SCORE Program, the Social and Behavioral Sciences (SBS) include those major SBS disciplines chiefly focused on understanding human social systems and behaviors, such as sociology, political science, economics, psychology, etc., as well as their respective sub-disciplines, such as computational social science, behavioral economics, social psychology, etc.

The pressing importance of addressing national security challenges means that the Department of Defense (DoD) would like to leverage SBS research to design plans, guide investments, assess outcomes, and build models in order to make decisions in the Human Domain.

Unfortunately, a number of recent empirical studies and meta-analyses have revealed that many SBS results and claims vary dramatically in terms of their ability to be independently reproduced and/or replicated.ix This is worrying since the degree of R&Rx of the SBS research upon which these decisions and models may depend can have real world implications. Research that appears confirmatory when it is in fact exploratory,xi suffers from other Questionable Research Practices (QRPs),xii or makes assertions or predictions that are unlikely given known practical or theoretical limitations,xiii can lead an SBS consumer to have inappropriate confidence in that research. Additionally, current approaches to estimating the R&R for SBS research are often slow, inefficient, opaque, and unreliable.xiv,xv Taken in the context of growing numbers of journals, articles, and preprints, this current state of affairs could result in an SBS consumer mistakenly over-relying on weak SBS research or dismissing strong SBS research entirely.

Efforts to address these concerns include new calls for, and early investments in, methods for identifying, promoting, and rewarding best research practices and new incentive structures among different communities in the hope that these changes may increase the R&R of future SBS research.xvi However, there is generally widespread recognition that substantive improvement, if possible, will take significant time and resources to achieve.xvii

Given the lack of widely agreed upon definitions of R&R, the SCORE Program will define Reproducibility as “the degree to which a particular research claim or result can be independently computationally reproduced by reanalyzing the study using the original data.” Likewise, the SCORE Program will define Replicability as “the degree to which a particular research claim or result can be independently replicated by rerunning the study as similarly as possible with a new sample.”

DARPA hypothesizes that an effective automated tool for helping an SBS consumer rapidly, accurately, and repeatedly calibrate the confidence they should have in research R&R would have significant positive impacts for increasing the effective use of the SBS literature to address Human Domain challenges. By identifying, aggregating, and analyzing a wide variety of signals, such a tool could provide expert-like “Confidence Scores” (CSs), or quantitative measures, that enable a user to understand the degree to which a particular claim or result is likely to be Reproducible and/or Replicable. DARPA anticipates that CSs may be derived from signals that may include (but not be limited to): evidence of one or many QRPs, retractions, evidence of publication bias or file drawer problems in a topic or field, (un)successful efforts at reproducing or replicating similar claims, (un)declared conflicts of interest, public criticism or xviii popular press coverage, sentiment analysis and social media/blog posts, post-publication review and crowd-sourcing,xix social networks of reviewers or co-authors, incentive structures, journal transparencyxx, ratio of exploratory to confirmatory studies in a topic or area, and/or the use of pre-registration or Registered Reports.

Further, such a tool should enable an SBS consumer to rapidly update CSs based on changes in those signals. Finally, such a tool should also explain why it has assigned the CSs that it has, including which signals it may have used and how it used them, allowing an SBS consumer to understand the results and appropriately leverage the SBS claim for use in the Human Domain.xxi

C. Program Description/Scope The vision of the SCORE program therefore is to test, validate, and demonstrate the feasibility and utility of one or more automated tools for assigning CSs to a wide range of SBS claims. SCORE seeks to realize this vision through a two-phase program. The first Phase will focus on developing the initial Common Task Framework (CTF) for SCORE, with a curated dataset of SBS research claims using methods for rapidly but accurately labeling those data with human expert CSs. Early algorithm development will also occur in Phase 1 as proof of principle. In Phase 2, performers will use those labeled data to train and test algorithms that will assign quantitative CSs. These algorithm-based CSs will be compared to CSs assigned by the best performing human expert methods to see how they overlap. If successful, this program will enable SBS consumers within the DoD and the U.S. Government to use SCORE algorithms to quickly, accurately, and iteratively calibrate the confidence they should have in a particular SBS claim’s R&R. SCORE deliverables should have significant positive impact on DoD and USG’s abilities to leverage SBS for modeling, planning for, and operating in, the Human Domain.

D. Program Structure SCORE is a 36-month program, comprising two phases with durations of 18 months each. Figure 1: SCORE Program Structure

To achieve its vision, the SCORE program will fund research in three Technical Areas (TAs), with an independent Test and Evaluation (T&E) team providing oversight. The three TAs are:

· TA1: Data

· TA2: Experts

· TA3: Algorithms

As outlined in Figure 1 above, SCORE will kick off by having its TA1 and TA2 community of performers and relevant subject matter experts agree upon common CSs criteria, scales, and metrics for the program. These will be determined in Month 1 of the program at the SCORE kickoff meeting. The kickoff meeting for TA3 performers will occur in Month 10 of the program.

Phase 1 of the SCORE program will then focus on TA1 teams developing the CTF with a curated SBS research dataset, while TA2 teams will focus on the implementation of methods for rapidly assigning expert CSs to articles. TA1 teams will also empirically evaluate the accuracy of each TA2 team’s expert-assigned CSs by independently reproducing/replicating a representative subsample of the CTF dataset and comparing outcomes with TA2 predictions. TA3 teams will use some TA1-supplied data (plus any proposed supplementary data) to conduct early algorithm development to provide proof of principle by Month 18.

If Phase 1 is successful in developing the initial CTF with validated CSs, Phase 2 will see TA3 teams develop out algorithms that are capable of efficiently and accurately assigning CSs with convincing overlap with the best performing TA2 method(s). These algorithms should allow SBS consumers in DoD to assign and update a claim’s CSs and understand how the algorithm(s) came to the specific quantitative assessment(s).

Figure 2: SCORE Technical Area Interactions

DARPA is soliciting proposals for TA1, TA2, or TA3 but is not soliciting proposals for participation on the T&E team. Proposals to any of the TAs must address the full program timeline. As stated in Part II, please note that TA1 and TA2 will have an earlier proposal due date and kickoff date than TA3. As discussed further in Section E, TA3 proposals should be submitted in accordance with the TA3 proposal deadline listed in Part I, and TA3 proposals should reflect a nine-month later start: TA1 and TA2 teams will start work in Month 1, while TA3 teams will start work in Month 10. Proposers1 should structure their proposals with Phase 1 as the base period (18-months for TA1 and TA2; nine-months for TA3), and Phase 2 as an option for funding (18-months for all TAs).

Each proposal should only address a single TA. Please note that to avoid conflicts of interest among TAs, no person or organization may be a performer on more than one TA, either as a prime or sub-contractor. The same person or organization may be on multiple proposals if those proposals are submitted to the same TA. However, proposers should provide sufficient approaches for managing potential conflicts of interest and/or firewalls among different teams/proposals in that TA and provide evidence that they have sufficient resources to mitigate any technical, cost, and/or schedule risk should they or their team members be on multiple proposals selected for negotiation. If an offeror submitted a proposal to TA1 or TA2 and the proposal was not selected for negotiation for award, that offeror may submit to TA3 if the TA3 requirements below are met.

E. Technical Area Descriptions

1. Technical Area 1 (TA1): Data

The goal of TA1 is to provision data for the program by curating SBS research claims, including (but need not be limited to) journal articles, previous R&R studies, abstracts, and preprints, into a single CTF dataset for TA2 and TA3 teams. In doing so, TA1 teams will have prime responsibility for developing the CTF for the SCORE program by providing training/test data to TA2 and TA3 performers as well as empirically validating the R&R of a representative subsample of claims in order to evaluate the accuracy of TA2 CSs expert methods.

Proposals: TA1 proposals must include clear, credible, and (where appropriate) quantitative descriptions that include (but need not be limited to) the following tasks deemed critical for SCORE program success:

· Identification of, and justification for, SBS topics/literatures to be curated as well as non- SBS topics/literatures that might be used to assess the generalizability of TA2/TA3 methods (e.g., assigning CSs to biomedical or neuroscience research). Proposals should also identify research areas or claims that are likely to be impossible or impractical for their approach to CTF creation;

· Recommendations for what should constitute an SBS research “claim” (e.g., article synopsis, abstract, specific conclusion, discrete statistical relationship, model) along with proposed approach for efficiently and reliably capturing these “claims” from various SBS research articles for TA2/TA3 use;

1 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

· Proposed criteria for operationalizing CSs for R&R, such as:

· Factors that should be used for scoring R&R;

· Relative weights assigned to R&R in order to come up with an aggregated CSs;

· Type and format of a CSs scale (e.g., range of 1-10 versus 1-100);

· Proposed visualizations (e.g., possible colors, chart types).

· Complete and feasible description of the curation process, including anticipated level of manual versus automated processes in each Phase;

· Proposed approach for representative sampling of CTF claims to be empirically validated for evaluating TA2 accuracy in Phases 1 and 2;

· Identification of literature or data repositories to be leveraged and a detailed data management plan (see Section I.I.3), with credible mechanisms to curate, store, and release data for TA2 CSs assignment and train/test data for TA3;

· Proposed schedule (or approach to developing a schedule) for data transfer and/or making data accessible to TA2 and TA3 teams during Phase 1 and Phase 2 efforts (e.g., daily, weekly, bi-weekly)

· Principled plan for the empirical evaluation of SBS research studies in each phase, including the anticipated relative number of replications, number of reproductions (where, for example, replication may be infeasible), and number of joint replication/reproductions;

· Approaches for Phase 2 testing of TA3 algorithms’ updating capability and ability to detect efforts to game the algorithms (including what kind and how much real or synthetic data would be injected/ablated and proposed metrics for evaluating TA3 updating/gaming detection success);

· Identification of any anticipated Human Subjects Research (HSR) and Human Research Protection Office (HRPO) requirements and plans to expeditiously receive necessary approvals that enable reaching the milestones on, or before, schedule (see Section VI.B.6 for more information on HSR requirements);

· Anticipated/recommended data output formats;

· Identification of specific risks to the proposed approach(es) and credible mitigation plans;

· Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in TA1 relevant areas in order to increase confidence in the specific proposed approach(es).

Performance Metrics: To assess TA1 performers’ progress towards realizing SCORE’s vision, TA1 teams will be assessed according to the following metrics:

· Curation Rate – Ability to curate SBS research articles efficiently and accurately. Proposers should offer credible approaches to achieve a curation rate of at least 3,000 SBS articles and claims per 12 months by Month 12, or if this is not possible, proposers should provide a reasonable justification and mitigation strategy that can still realize SCORE’s vision. In addition, proposers should offer credible plans for an increased curation rate in Phase 2, in order to exceed 3,000 articles and claims per 12 months. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

· Empirical Evaluation – Capabilities supporting the empirical evaluation of the R&R of at least 200 representative research claims in each phase. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

· Platform/Methodology Efficacy – Ability to maintain and continually grow a CTF of curated SBS research datasets that are useful for SCORE TA2/TA3 performers. Capabilities should also include rapid (preferably semi-automated) processes for cleaning and aligning articles as well as developing representative samples. Credible approaches that make a compelling argument that this CTF platform/method may be useful beyond the SCORE program may be more favorably reviewed.

Nominate TA1 Metrics: Proposers should nominate additional metrics for evaluating their specific capabilities and performance in terms of the TA1 goals. Examples might include metrics for evaluating TA1 R&R empirical validation methods and outcomes, metrics for evaluating TA2 R&R accuracy against TA1 empirical validations, and/or metrics for evaluating TA2 and TA3 CSs overlap using TA1 data. Final TA1 metrics will be decided in coordination with DARPA and the T&E team after selection for award.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA1 teams may be expected to maintain some control during the program period over database contents, platforms, curation processes, etc.

Out of Scope: Use of proprietary data or datasets may significantly impede SCORE program goals, including (but not limited to): limiting TA1 deliverables or teaming capabilities, reducing the Government’s ability to independently evaluate those data, limiting the capabilities of TA2/TA3 teams to conduct their research, and/or restricting the capabilities and evaluation of TA2/TA3 deliverables. Given these concerns, use of proprietary data or datasets may be out of scope for the SCORE program. However, if use of some proprietary datasets is deemed appropriate, licenses should be negotiated that do not prevent SCORE performers, the T&E team, or DARPA from being able to use those datasets to accomplish their respective and collective technical and programmatic goals. SCORE will not fund efforts that propose the use of data or datasets that violate copyright protections or other legal restrictions.

2. Technical Area 2 (TA2): Experts

The goal of TA2 teams is to assist in the creation of the SCORE CTF by using or developing scalable methods and/or platforms for eliciting expert Confidence Scores for the R&R of SBS claims in the TA1 datasets. The focus of TA2 is on assigning human expert CSs to TA1’s data (claims, studies, abstracts) at a rate that is sufficient to enable TA1 to build up the SCORE CTF datasets required for Phase 2. TA2 teams may use innovative mechanisms that enable rapid but accurate TA2 CSs assignments, where accuracy is measured as the ability of TA2 teams to predict the outcomes of TA1’s empirical evaluations of the R&R of a subsample of studies and claims.

Proposals: TA2 proposals must include clear, credible, and (where appropriate) quantitative descriptions that include (but need not be limited to) the following tasks deemed critical for

SCORE program success:

· Identification of an approach for acquiring expert assessments via prediction markets, expert surveys, online games, and/or other innovative and technically compelling approach(es);

· Proposed approaches to capturing and conveying CSs “forecasts” for TA1 testing (e.g., summed averages, market value, rank order; judgmental reasoning to derive probabilities or CSs), including any approach(es) for capturing expert scoring processes/signals and potentially sharing identified signals and/or expert processes with TA3 teams;

· Recommendations for what should constitute an SBS research “claim” for CSs assignment (e.g., article synopsis, abstract, specific conclusion, discrete statistical relationship, model), along with

· Proposed criteria for operationalizing expert CSs for R&R, such as:

· Factors that might be used for scoring R&R;

· Relative weights assigned to R&R in order to come up with an aggregated CSs;

· Type and format of CSs scale (e.g., range of 1-10 versus 1-100);

· Proposed visualizations (e.g., possible colors, chart types).

· Proposed approach(es) for capturing/comparing the impact of different variables (numbers or kinds of experts, skillsets, infrastructure, etc.) on the performance/accuracy of TA2 methods;

· Proposed suggestions for SBS topics or literatures best suited for approach, including identifying potential datasets and/or literature repositories for TA1 to leverage and identifying extensions to other topics/literatures, including non-SBS research;

· Identification of any anticipated HSR/HRPO requirements and plans to expeditiously receive necessary approvals that enable reaching the milestones on, or before, schedule;

· Proposed flexible mechanisms and/or platforms for receiving/storing TA1 data and returning CSs to TA1 such that TA2 CSs can be easily and persistently associated with specific claims;

· Identification of anticipated requirements of/for data sets, structure, formats, and data management plan per Section I.I.3;

· Identification of specific risks to the proposed approach(es) and credible mitigation plans;

· Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in relevant areas in order to increase confidence in the specific proposed approach(es).

Performance Metrics: To assess TA2 performers’ progress towards realizing SCORE’s vision, TA2 teams will be assessed according to the following metrics:

· Evaluation Rate – Ability to assign CSs to SBS research articles efficiently. Proposers should offer credible approaches to achieve an evaluation rate of 3,000 articles per 12 months by Month 12, or if this is not possible, proposers should provide a reasonable justification and mitigation strategy that can still realize SCORE’s vision. In addition, proposers should offer credible plans to allow for an increased evaluation rate in Phase 2, exceeding 3,000 articles per 12 months. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

· Accuracy – Ability to assign CSs to SBS claims that are at least 80% accurate in Phase 1 when tested against TA1 R&R empirical evaluations in Month 15. Proposers should also offer credible plans for an improved accuracy rate in Phase 2, ideally exceeding 80% by Month 30. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

Nominate TA2 Metrics: Proposers should nominate additional metrics for evaluating their capabilities and performance in terms of the TA2 goals. Examples might include metrics for estimating cost per CSs assignment per claim and/or metrics for “expert agreement” and interrater reliability. Final TA2 metrics will be decided in coordination with DARPA and the T&E team after selection for award.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA2 teams may be expected to maintain some control during the program period over expert assessments, processes, platforms, etc.

Out of Scope: SCORE will not fund performers that use platforms in a way that may violate or otherwise disregard acknowledged and in good standing IP claims, restrictions, or legal statutes in applicable states or countries.

3. Technical Area 3 (TA3): Algorithms

Successful TA3 teams will develop SCORE algorithms using training data provided by TA1 (as well as other training data they may propose) to automatically assign CSs to held-out TA1 test sets. TA3 team(s) will be responsible for all aspects of algorithm development. This responsibility extends to all essential feature identification, extraction, and engineering steps, whether carried out over TA1-curated data or over TA3-provided data. TA3 algorithms will be evaluated in terms of their CSs overlap with the best performing TA2 CSs, ultimately seeking to achieve 95% confidence in an algorithm’s overlap with those TA2 CSs. Towards the end of Phase 2, successful TA3 teams will also demonstrate the utility of their algorithms/systems for DoD users and applications.

Proposals: TA3 proposals should include clear, credible, and (where appropriate) quantitative descriptions that address (but need not be limited to) the following elements:

· Credible plan for achieving Phase 1 “proof of principle” capabilities by Month 18 and for regularly documenting and demonstrating incremental progress toward that goal. Quantitative milestones should be proposed for Months 12, 15, and 18;

· As relevant, identification of additional training/test data, whether in the SBS or otherwise, that the proposer intends to use to complement data provided by TA1;

· As relevant, identification of minimum essential data features and elements that the proposer requires in the TA1-provided CTF data set (to include requirements associated with data content, structure, format, or data interchange standards), in order for the proposed technical approach(es) to be feasible;

· Identification of data features and elements that the proposer plans to create and/or assemble, including (as relevant) details of which specific data sources will be leveraged to generate those features, which feature extraction processes will be employed, and what licensing and reuse restrictions, if any, govern their distribution to and use by other research teams, the government, and government transition partners;

· Identification of non-essential, but potentially useful, ancillary data features and elements that the proposer may wish to leverage, including (as relevant) details of candidate data sources and associated terms of use;

· Identification of the specific algorithms (or, as appropriate, algorithm classes or algorithmic approaches) that the proposer intends to develop and/or implement, including a description of the desirable properties of those algorithms or approaches in the SCORE context;

· Specification of a credible and complete data management plan, addressing all data products entailed by the proposal, as detailed in Section I.I.3;

· Credible plan for achieving increasing overlap with best performing TA2 expert CSs over the course of each testing and training sprint in Phase 2, reaching 75% confidence of overlap by Month 24, 85% confidence of overlap by Month 30, and 95% confidence of overlap by Month 36;

· Proposed approaches to testing TA3 updating capability (including what kind and how much real or synthetic data TA1 might provide, proposed metrics for evaluating TA3 updating success, and innovative but credible approaches to testing TA3 algorithms susceptibility to “gaming” attempts);

· Identification of any anticipated HSR/HRPO requirements and plans to expeditiously receive necessary approvals that enable reaching program milestones on, or before, schedule;

· Identification of specific computing platforms and/or software required for algorithm development and deployment;

· Additional information necessary to understand and evaluate the feasibility and innovation of the approach(es) being proposed, which may include evidence of prior work or research in relevant areas.

Performers may face technical and logistical challenges in developing, implementing, and testing TA3 algorithms. Proposals should explicitly identify and address key challenges and risks. At a minimum, proposals should address:

· Whether proposed algorithms or systems will require interactive inputs from humans. Proposers should clearly explain when and how any human inputs will be leveraged and detail what, if any, human qualifications are assumed;

· How the proposer’s technical approach will remain robust in the face of topically, formally, and qualitatively diverse scientific research claims and data;

· How algorithmic models will be updated or retuned (as applicable), and what costs this might impose upon the government post-SCORE technological transition;

· How algorithmic outputs will be associated with human-interpretable evidence and descriptions (see “Out of Scope” section, below);

· Whether the proposed technical approach will require access to any proprietary or subscription-based data and, if so, the costs and terms associated with such access; a compelling argument for why such data are essential to success of the proposer’s technical approach; and a concrete plan for how the proposer intends to mitigate any negative impacts on the program’s objectives. Proposers are strongly discouraged from leveraging data to which only they have access or data which are unlikely to be licensable or deliverable to the government for purposes of technology transition due to restrictive terms of use or privacy-related restrictions, or because they are not easily replicated on government systems. Additionally, if proposers plan to incorporate existing intellectual property (IP) into their algorithms or systems, they must specify what the cost to the government will be to use this IP both during the SCORE program and after the program is completed and transitioned.

Proposals should describe offeror past performance in research areas relevant to SCORE. Proposals should also clearly describe performer team-internal research efforts and pilot testing that will be undertaken in pursuit of SCORE goals. Internal research efforts and pilot testing may include other modeling or data collection activities, including but not limited to: running of novel experiments and data collection exercises; analysis or mining of extant data sets; development and testing of novel algorithms; running simulation studies; development of novel formal or analytical methods; and other possible avenues of knowledge generation.

Performance Metrics: TA3 performers will be assessed according to the following performance dimensions:

· Overlap with TA2 – Ability to assign Confidence Scores to SBS claims that overlap with best performing TA2 methods. Proposed approach should provide a credible plan for achieving early proof of principle in Phase 1 by Month 18 and offer credible plans for achieving 75%, 85%, and 95% CS overlap over the course of each six-month sprint in Phase 2, respectively. If a proposer believes this metric is not achievable, they should provide a reasonable justification and mitigation strategy. Credible approaches that make a compelling argument that these metrics will be exceeded may be more favorably reviewed.

· Algorithmic Efficiency – Ability to evaluate SBS research articles efficiently and at scale. TA3 algorithms will be formally evaluated to assess their resource utilization and computational efficiency. Specific algorithmic efficiency metrics and targets will be finalized after awards are granted, but TA3 proposers should provide credible plans for achieving a claim processing throughput of not fewer than 20 new claims per 24-hour period by Month 18, with throughput doubling by Month 36, and with no attendant drop in system accuracy.

· Algorithm/Platform Usability – Capability for algorithms to assign and pass algorithm CSs to TA1 teams efficiently and the degree to which algorithms are transparent and their CSs assessments are interpretable/tailorable as evaluated by an above average score on a System Usability Scalexxii by the end of Phase 2.

Nomination and Selection of TA3 Metrics: Proposers should nominate suitable metrics for evaluating the capabilities and performance of their systems Examples may include (but need not be limited to) specific metrics and metric targets for the assessment of algorithmic efficiency, algorithm/platform usability, cost per processed claim, ease of updating, and system accuracy. Final TA3 performance metrics will be decided in coordination with DARPA and the T&E team after award determinations have been made.

Publication of Research: Note DARPA anticipates that all research conducted for SCORE will be fundamental, unclassified research and, therefore, encourages performers to publish and/or distribute deliverables and results. However, given SCORE goals, TA3 teams may be expected to maintain some control during the program period over algorithm code, assessments, processes, platforms, etc.

Out of Scope: The following technical approaches are out of scope for TA3:

· “Blackbox” algorithms, whose final functions or outcomes in assigning Confidence Scores cannot be reasonably explained to a DoD user

· Methods in which Confidence Scores are based exclusively on the weighting or transformation of human analytic judgments without meaningful incorporation of other machine/statistical data or models;

· Approaches that require substantial, ongoing engagement of subject matter experts (SMEs) to maintain and operate. Engagement of SMEs should be to a degree that is scalable and not prohibitive in the context of post-program transition.

Additionally, SCORE will not fund algorithms developed using hardware and/or software that limit TA3 deliverables to less than Government Purpose Rights, unless strongly justified.

4. Test and Evaluation (T&E)

The SCORE T&E team will assist with coordinating among the TAs, including helping to manage all communications among TAs to ensure accordance with SCORE programmatic goals. The T&E team will also be involved in adjudicating the representativeness of proposed TA1 samples for empirical evaluation, as well as TA2 accuracy as compared to TA1 empirical evaluations, and TA2/TA3 CSs overlap. Lastly, the T&E team will also assist in coordinating TA3 system usability tests and metrics for successful TA3 teams in Phase 2.

F. Schedule/Milestones Proposals to any of the three TAs must address the full program timeline (TA3 proposals, due at the later TA3 proposal closing date stated in Part I, should address the full program timeline assuming a Month 7 kickoff). Proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents an aggressive plan to fully address all program goals, metrics, milestones and deliverables. In addition, the task structure must be consistent across the proposed schedule, Statement of Work (SOW), and cost volume in order to enable DARPA to evaluate the technical, programmatic, and financial credibility of individual tasks as well as the overall approach. Schedules will be synchronized across performers, as required, and monitored/revised as necessary throughout the program. For TA1 and TA2 teams, a target start date of January 2019 may be assumed for planning purposes. For TA3 teams, a target start date of October 2019 may be assumed for planning purposes.

All SCORE TA1 and TA2 performers should expect to attend a kickoff meeting in the Washington, D.C. area. TA3 performers should expect the Month 10 Principal Investigator (PI) Meeting to serve as their kickoff. After Month 10, TA3 performers should plan for site visits and PI Meetings as would TA1 and TA2 teams per the program schedule outlined below in Figure 3 and Tables 1 and 2.

DARPA expects all performers to attend PI Meetings every 6 months, as shown in Figure 3. The purposes of the PI Meetings are to (i) provide the Program Manager and other SCORE performers with updates on progress towards milestones and goals; (ii) summarize outstanding technical challenges; (iii) support test and evaluation; and (iv) provide Government and potential transition partners with opportunities to provide input, comments, and suggestions for the SCORE program and its performers. For budgeting purposes, TA1 and TA2 proposers should assume a two-day kickoff meeting, while PI meetings will require three days and will alternate between Washington, D.C. and a west coast location. The Government and T&E team will schedule regular teleconference meetings for progress reporting as well as problem identification and mitigation. Proposers should anticipate at least one site visit every 6 months by the DARPA Program Manager during which they will have the opportunity to demonstrate progress towards agreed-upon milestones. Additional anticipated programmatic events are included in Tables 1 and 2, below.

For budgeting purposes, all teams should plan for six three-day PI meetings over the course of 36 months: three meetings in the Washington, D.C. area and three meetings in the San Francisco, CA area. For TA1 and TA2 teams, these meetings are in addition to a two-day kickoff meeting in the Washington, D.C. area (TA3 teams will attend a Month 10 kickoff meeting).

Figure 3: SCORE Program Gantt Chart

Table 1: Technical Goals by Phase

TA1:

Data

• Curate a database of SBS research to develop a common task framework (CTF)

• HSR secondary approvals1a

• Curation at a rate of 3,000 articles or more per 12 months1b

• Empirical evaluation of representative sample of claims at a rate of 200 or more per 15 months1c

• Continue curation of CTF database

• Curation at a rate of 3,000 articles or more per 12 months

• Empirical evaluation of at least 200 representative samples of SBS claims

TA2:

Expert Assessments

• Launch methods for rapidly getting expert assessments of SBS research studies

• HSR secondary approvals2a

• Assessment at a rate of 3,000 articles or more per 12 months2b

• Accuracy of at least 80% as compared to TA1 empirical evaluations2c

• Assessment at a rate of 3,000 articles or more per 12 months

• Accuracy greater than 80% as compared to TA1 empirical evaluations

TA3:

Algorithm Assessments

· Develop algorithms for rapid assessment of SBS research studies based on endogenous and exogenous signals

• Assessment at a rate of 1 article per 60 minutes

• Demonstration of algorithm’s efficacy and explainability 3a

• Assessment at a rate of 1 article per 30 minutes

• Increase percentage of distribution overlap between TA2 and TA3 assessments over the course of three consecutive sprints:

· Sprint 1 – 75% overlap3b

· Sprint 2 – 85% overlap 3c

· Sprint 3 – 95% overlap3d

• Demonstration of system usability for both experts and non-experts

Table 2: Program Events by Month Months After Award

Event

Description

PHASE 1

· TA1 and TA2 teams present technical approach and work plan

· T&E team provides test and evaluation plan, candidate metrics

Start Work

· TA1 teams begin data curation and work with T&E to develop a representative sample for empirical evaluation

· TA2 launches platforms for scaling up CSs assignments

Site Visits

· All teams: demonstrate progress towards milestones

· TA1 provides preliminary training data to TA3

· TA2 teams demonstrate CSs assignments at rate of 3K claims per 12 months

PHASE 2

· T&E presents results of TA3 overlap with TA2

· T&E presents results of TA3 overlap with TA2, TA2 accuracy compared with TA1 R&R empirical evaluations

· T&E presents results of TA3 overlap with TA2, TA3 usability testing

G. Deliverables DARPA expects performers to provide at a minimum the following deliverables:

· Comprehensive monthly technical reports due within ten days of the end of each given month, describing progress made on the specific milestones as laid out in the SOW.

· A phase completion report submitted within 30 days of the end of each phase, summarizing the research done.

· Other negotiated deliverables specific to the objectives of the individual efforts. These may include:

· For TA1: CTF datasets, access to database, results of empirical evaluations, and other deliverables stemming from the research funded under SCORE

· For TA2: access to expert assessment tools/platforms and other deliverables stemming from the research funded under SCORE, including data and Confidence Scores

· For TA3: algorithms and systems, and other deliverables stemming from the research funded under SCORE, including data and Confidence Scores

· For all: publications, intermediate and final versions of software libraries, code, and APIs, including documentation and user manuals and/or a comprehensive assemblage of design documents, data, and results.

· Reporting as outlined in Section VI.C.

H. Government-furnished Property/Equipment/Information Performers are not expected to need Government-furnished Property (GFP). Any requests for GFP must be accompanied by a credible justification and explanation of why other options would not be suitable for the proposed work.

I. Other Program Objectives and Considerations

1. Collaboration DARPA expects all performers to work collaboratively with one another to realize the program objectives outlined herein, so proposers should carefully review the goals for the entire program in order to fully understand the context of each program objective within the overall program structure. Furthermore, throughout development of program technologies, it will be necessary for all performers to share relevant information regarding their technology development to support the larger program goals. For example, TA2 performers will need access to the CTF datasets developed by the TA1 performers. All proposals must clearly describe plans for interfacing and integrating their proposed technologies/approaches with other performers. Proposals that fail to include interface and integration plans may be deemed non-conforming2 and removed from consideration for award.

DARPA expects SCORE performers to collaborate closely with the T&E team, so proposals should describe approaches for providing the T&E team with access to all necessary and relevant software, data, and domain knowledge.

2. Intellectual Property As discussed above, there is an emphasis on creating and leveraging open source technologies and architectures, making data sharing and collaboration key aspects of this program. Therefore, intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.4 for more information related to intellectual property.

3. Data Management Plan

This BAA requires a Data Management Plan (DMP) be included as part of the proposal submission. DARPA/DSO’s view of what constitutes the scope of applicable data products to be covered in a DMP is quite broad, potentially encompassing all digital activity related to a project. DARPA’s approach to an effective and practical DMP is predicated with two goals:

First, data are increasingly the key products of research and engineering endeavors. To ensure the reproducibility of results and the accessibility of program accomplishments to future users, we require proposers document the necessary and sufficient scope of data that may be applicable to these goals. Performers will be expected to document both the proprietary and nonproprietary products of the program, including but not limited to raw unprocessed data, software source code and executables, build scripts, process sequence, programmatic communication and other

2 “Conforming” is defined as having been submitted in accordance with the requirements outlined herein.

collaboration activities; rarified, experimental, test and measurement data; design of experiments and simulations; models or simulations (computational or mathematical); recordings of various physical phenomena (including images, videos, sensor data, etc.); access to and use of institutional, organizational or scientific community repositories and archives. An important goal of the DMP is to ensure the retention and potential reusability of this information.

Second, when possible, DARPA may also share some or all of the program-generated data with the broader research community as open data (with permission to access,…