IARPA-BAA-22-01_HIATUS_Amend 001_C.pdf

IARPA

BROAD AGENCY ANNOUNCEMENT

IARPA-BAA-22-01

Human Interpretable Attribution of Text using Underlying Structure (HIATUS) Program

IARPA-BAA-22-01

Release Date:

February 25, 2022

Amendment 001: April 04, 2022

ALL CHANGES FOR AMENDMENT 001 ARE HIGHLIGHTED IN YELLOW.

i

BROAD AGENCY ANNOUNCEMENT: IARPA-BAA-22-01

HIATUS Program

TABLE OF CONTENTS

GENERAL INFORMATION

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

1.A. Program Overview

1.A.1. Technical Challenges and Objectives

1.A.2. Program Phases

1.A.2.1. Phase 1

1.A.2.2. Phase 2

1.A.2.3. Phase 3

1.B. Team Expertise

1.C. Program Scope and Limitations

1.D. Program Data

1.D.1. Development Data

1.D.1.1. Government Research and Test Sets

1.D.1.2. Researcher Collections

1.D.1.3. External Data Sources

1.D.2. Evaluation Data

1.D.3. Privacy Plan

1.E. Test and Evaluation (T&E)

1.F. Program Metrics

1.G. Program Waypoints, Milestones, and Deliverables

1.G.1. Program Waypoints

1.G.2. Program Milestone, Waypoint, and Deliverables Timeline

1.G.3. Software Deliverable Formatting

1.G.3.1. Program API

1.H. Meeting and Travel Requirements

1.H.1. Workshops

1.H.2. Site Visits

1.I. Period of Performance

1.J. Place of Performance

SECTION 2: AWARD INFORMATION

SECTION 3: ELIGIBILITY INFORMATION

ii

3.A. Eligible Applicants

3.A.1 Organizational Conflicts of Interest (OCI)

3. A.2 Multiple Submissions to the BAA

3. B. U.S. Academic Institutions

3. C. Other Eligibility Criteria

4. A. Proposal Information

4. B. Proposal Format and Content

4. B.1 Volume 1: Technical and Management Proposal

4. B.1.a Section 1: Cover Sheet & Transmittal Letter

4. B.1.b Section 2: Summary of Proposal (see below for page limit)

4. B.1.c. Section 3: Detailed Proposal Information

4.B.1.d. Section 4: Attachments

4.B.2. Volume 2: Cost Proposal (No Page Limit)

4. B.2.a Section 1: Cover Sheet

4. B.2.b. Section 2: Estimated Cost Breakdown

4. B.2.c. Section 3: Supporting Information

4. C. Submission Details

4. C.1. Due Dates

4. C. 2 Proposal Delivery

4. D. Funding Restrictions

SECTION 5: PROPOSAL REVIEW INFORMATION

5.A. Technical and Funding Availability Evaluation Factors

5.A.1. Technical Evaluation Factor (technical criteria listed below)

5. A.1.a. Overall Scientific and Technical Merit

5. A.1.b. Effectiveness of Proposed Work Plan

5. A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal

5. A.1.d Relevant Experience and Expertise

5. A.1.e Resource Realism

5.A.2. Funding Availability Factor

5. A.2.a. Budget Constraints

5. A.2.b. Program Balance

5.B. Method of Evaluation and Selection Process

5.C. Negotiation and Contract Award

5.D. Proposal Retention iii

SECTION 6: AWARD ADMINISTRATION INFORMATION

6.A. Award Notices

6.B. Administrative and National Policy Requirements

6.B.1. Proprietary Data

6.B.2. Intellectual Property

6.B.3. Human Use

6.B.4. Animal Use

6.B.5. Publication Approval

6.B.6. Export Control

6.B.7. Subcontracting

6.B.8. Reporting

6.B.9. System for Award Management (SAM)

6.B.10. Representations and Certifications

6.B.11. Lawful Use and Privacy Protection Measures

6.B.12. Public Access to Results

6.B.13. Other Contract Requirements

6.B.13.a. Provisions

6.B.13.b. Clauses

Appendix A: Templates for Volume 1: Technical Proposal

A.1 Cover Sheet for Volume 1: Technical Proposal

Appendix A.2 Academic Institution Acknowledgment Letter

Appendix A.3 Intellectual Property Rights

Appendix A.4 Organizational Conflicts of Interest Certification Letter

Appendix A.5 Three Chart Summary of the Proposal

Appendix A.6 Research Data Management Plan (RDMP) BAA 22-01

Appendix B: Templates for Volume 2: Cost Proposal

Appendix B.1 Cover Sheet for Volume 2: Cost Proposal

Appendix B.2 Prime Contractor/Subcontractor Cost Element Sheet for Volume 2: Cost Proposal

Appendix B.3 - Software and IP Costs

Appendix B.4 – Travel Costs Trip breakdown

Appendix B.5 – Contract Deliverables Table

GENERAL INFORMATION

This notice constitutes a Broad Agency Announcement (BAA) and sets forth research of interest in the area of authorship attribution and privacy. The solicitation process will follow Federal Acquisition Regulation (FAR) Part 35, Research and Development Contracting, as supplemented with additional information included in this notice. Awards based on responses to this BAA will be considered the result of full and open competition.

1. Federal Agency Name – Office of the Director of National Intelligence (ODNI)/Intelligence Advanced

Research Projects Activity (IARPA)

2. Funding Opportunity Title – Human Interpretable Attribution of Text using Underlying Structure (HIATUS) Program

3. Announcement Type – Initial

4. Funding Opportunity Number – IARPA-BAA-22-01

5. Catalog of Federal Domestic Assistance Numbers (CFDA) – Not applicable

6. Questions

Submit questions on administrative, technical, or contractual issues by email to DNI-IARPA-BAA-22- 01@iarpa.gov. All requests must include the full name and affiliation of a point of contact. Do not send questions with proprietary content. A consolidated Question and Answer response will be posted on SAM.gov for Contract Opportunities website (https://SAM.gov/) and linked from the IARPA website (https://www.iarpa.gov/index.php/research-programs/hiatus). No answer will go directly to the submitter. IARPA will accept questions until March 18, 2022 @ 5:00 PM EST.

7. Dates

7.1 Posting Date: February 25, 2022

7.2 Proposal Due Date for Initial Round of Selections: April 18, 2022 @ 4:00 PM EST

7.3 BAA Closing Date: April 29, 2022 4:00 PM EST

8. Anticipated individual awards – Multiple awards anticipated

9. Types of instruments that may be awarded – Procurement Contracts and Other Transactions 1

10. Agency Points of Contact

ATTN: IARPA-BAA-22-01

Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington, DC 20511 Electronic mail: DNI-IARPA-BAA-22-01@iarpa.gov

11. Program Manager (PM) – Dr. Timothy McKinnon

12. Program Website – http://www.iarpa.gov/index.php/research-programs/hiatus

1 Procurement Contract: This is a standard government contract that follows the processes, format and terms and conditions as outlined in the Federal Acquisition Regulations (FAR) and supplementing Agency specific regulations.

Other Transaction Authority: Agreements generally are not subject to the federal laws and regulations governing procurement contracts and thus are not required to comply with the Federal Acquisition Regulation (FAR), its supplements, or laws that are limited in applicability to procurement contracts. They may be used with non-traditional contractors under certain circumstances.

13. BAA Summary –The HIATUS program seeks to develop novel human-useable systems for attributing authorship and protecting author privacy through identification and leveraging of explainable linguistic fingerprints. HIATUS Performers will focus on three task areas (TAs). The program will develop novel methods to generate a stylistic feature space in which each author is represented in stable fashion across diverse text types (TA1). Using this feature space, the program will also create human-interpretable algorithms for authorship attribution (TA2) and author privacy (TA3). Technical approaches must scale to other languages.

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

IARPA often selects its research efforts through the BAA process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA will appear under Contract Opportunities on https://sam.gov as well as the IARPA website at http://www.iarpa.gov. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-22-01) is for the Human Interpretable Attribution of Text using Underlying Structure (HIATUS) Program. IARPA seeks innovative solutions for the HIATUS Program which is envisioned to be a 42-month effort, beginning approximately September 30, 2022 through March 29, 2026.

1.A. Program Overview

Authorship attribution is an important capability for the Intelligence Community (IC), facilitating efforts to understand and combat increasingly sophisticated malicious information campaigns online, fight human trafficking, identify counterintelligence risks, etc. Authorship privacy (also known as “authorship obfuscation”) refers to technology and methods for modifying text to remove features predictive of a specific author. Authorship privacy technologies have a wide variety of uses including protection of individuals and groups whose writing, if attributed, could place them in personal danger.

The HIATUS program seeks to develop novel human-useable systems for attributing authorship and protecting author privacy through identification and leveraging of explainable linguistic fingerprints.

HIATUS will create technologies that automatically identify and generate verifiable ‘linguistic fingerprints’ of individual authors (both human and machine), enabling authorship attribution and privacy. Successful systems will perform robustly for diverse authors, topic domains, genres and languages.2 The overall vision of the program is shown in Figure 1.

Figure 1: The HIATUS vision: A combined authorship attribution and privacy system that can be trusted and audited by human operators.

2 The term ‘genre’ refers here to differences in text based on audience, purpose, activity, and related factors.

HIATUS addresses the limitations in currently available authorship attribution and authorship privacy techniques. Manual text attribution is time-consuming and requires highly specialized expertise. State-of-the-art authorship attribution systems built on powerful neural language models are far too complex for humans to interpret or audit, undermining user trust. Older computational approaches (Logistic Regression, Support Vector Machines, etc.) expose model rationale to a greater extent than neural models, but they rely on text preprocessing schemes that place a priori limits on type and quantity of stylistic-linguistic features extracted from text and require voluminous writing samples from each author. See Figure 2 for a graphical representation of this trilemma. HIATUS addresses these limitations by developing automated, human interpretable systems that maximize use of the linguistic feature space and provide robust performance across diverse genres, topic domains and languages.

Figure 2: The HIATUS Trilemma: Categories of author attribution and their strengths and weaknesses. HIATUS aims to minimize human effort, maximally leverage the grammatical feature space, and furnish human-interpretable feedback for trust and verification of system behavior.

High performing authorship attribution/privacy systems are of limited value if their behavior cannot be trusted and validated by human users. Explainable Natural Language Processing (NLP) techniques to date have relied heavily on delineating spans of input text that a model ‘attends’ to when predicting a label. Such explanations do not identify actual linguistic features or structures (syntactic, morphological, discourse-level, etc.) A central goal of the HIATUS program is to provide human-interpretable representations of the stylistic features in text. Offerors shall propose innovative techniques in explainable NLP that achieve this goal and shall work with a Test & Evaluation (T&E) Team to develop and implement methods and protocols to evaluate explanations of system behavior.

The HIATUS program casts attribution and privacy as different aspects of the same underlying challenge:

understanding author-level stylistic variation by elucidating stable identifiers of individual authors across diverse types of text. The program fully embraces authorship attribution/privacy as an adversarial machine learning problem. Throughout the development and testing lifecycle, attribution and privacy algorithms are placed in constant competition (Figure 3 shows the HIATUS evaluation workflow). At the beginning of the program, performers will be provided with a software evaluation harness developed by the T&E team, and they shall implement team-internal competition between attribution and privacy systems to ensure optimal system performance3. The same software testing harness will be used at official evaluation milestones, where systems are submitted to the T&E Team for blind evaluation against opponent team systems on a sequestered dataset comprising documents representing diverse text and author characteristics.

The intended outcome of this scheme is to improve the robustness of authorship privacy algorithms against attribution algorithms (and vice versa) and simultaneously incentivize teams to build higher fidelity encodings of author-level grammatical variation (see Figure 4). To remain competitive throughout the program, teams must build increasingly sophisticated representations of the stylistic feature space, the representation upon which both privacy and attribution algorithms operate.

3 Systems developed by one performer will not be shared with other performers, only submitted to T&E for official evaluations.

The HIATUS Program will address the technical needs and challenges outlined above by pursuing research into stylometric algorithms, domain adaptation, text generation, and explainable NLP. Performers will conduct research to develop robust, high-throughput software systems that generate representations of text in stylistic feature space, perform attribution and text modifications to ensure document privacy. Systems must be capable of performing such functions over an array of document types, including texts representing diverse genres, domains, languages and author backgrounds. The delivered interim and final systems will be evaluated by the T&E Team according to a set of program metric targets detailed in section 1.F. below.

A robust data collection initiative will be part of the HIATUS Program to assemble diverse text types for use in Research and Development (R&D) for algorithm training, and in T&E for comparing the algorithm performance against ground truth. The T&E Team will conduct dataset development activities throughout the life of the program using Institutional Review Board (IRB)-approved Human Subject Research (HSR) protocols. A portion of this data will be made available to Performers for R&D. In addition, Performers will be required to conduct their own supplemental IRB-approved dataset collections. At the conclusion of each program phase, Performers shall make all datasets used for system development available to the T&E Team for the purpose of distribution to other Performers. Additional details on program data can be found in section 1.D. Program Data below.

The HIATUS Program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and algorithm performance is measured across the full range of data conditions. T&E activities will not only inform Government stakeholders on HIATUS research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development.

The HIATUS Program will work closely with Government leaders in Human Language Technologies (HLT) to continually refine and improve T&E methodologies.

1.A.1. Technical Challenges and Objectives

Offerors shall address three technical TAs to meet the goal of HIATUS to develop novel, human-interpretable, author attribution and privacy systems. Offerors must propose novel approaches to each of these TAs, and if selected as a Performer, Performers will be required to create an end-to-end technology that incorporates software components from each TA.

Feature Space Generation (TA1): The goal of TA1 is to research and develop an encoder that projects text into an optimized stylistic feature space wherein observations (texts) from the same class (author) are projected near one another (and far from those of other authors), regardless of non-authorial properties (genres, domains, languages and author backgrounds). Successful approaches must utilize features that are both predictive of authorship across texts of diverse types, as well as features that are predictive of authorship within specific topic domains or genres only. Successful representations shall also enable segmentation of machine- vs. human-generated documents. Successful technical approaches will take both feature types into account. The TA1 deliverable will be evaluated by the T&E Team using a rank and retrieval task as described in section 1.F. below.

Authorship attribution (TA2): The goal of TA2 is to research and develop a system that will successfully identify unique authors within a corpus using the stylistic characteristics of author text contained within.

The TA2 software deliverable must leverage representations from the TA1 stylistic encoder to determine whether two or more documents in a collection were produced by the same author. Systems will be evaluated using an identification task as described in section 1.F. The TA2 deliverable will be evaluated by the T&E Team using an open set search task.

Authorship privacy (TA3): The goal of TA3 is to research and develop a system that will successfully modify aspects of a document that provide evidence of authorship without altering non-authorial features such as the fluency and meaning.4 For the purposes of analysis, the T&E Team shall develop a clear set of criteria to evaluate TA3 system output. These criteria shall be announced at program kickoff, but Offerors may assume criteria like those found in the machine-generated text and paraphrase generation literature.

One important additional aspect of fluency is stylistic self-consistency: different parts of the same modified document, or multiple modified documents from the same author, should not look like they were produced by different authors; they should be stylistically consistent with one another (though stylistically distinct from their original author). To achieve these objectives, technical approaches to TA3 should leverage ‘upstream’ information from TA1 and TA2 components e.g., to isolate stylistic features in the document and gauge how specific modifications of the text affect the document’s location in stylistic feature space.

The success of TA3 systems will be measured by the ability to thwart attribution of a modified document while ensuring all modifications preserve that document’s original fluency and meaning. Ability to thwart attribution will be measured by running TA3 system outputs through opponents’ TA2 systems. Both human and automated methods will be used to assess preservation of fluency and meaning. This task is described in section 1.F. below.

System explainability (TA2 & TA3): Systems developed in both TA2 and TA3 must provide human-interpretable feedback explaining why specific documents likely are (or are not) attributed to the same author. Offerors must propose novel explainable NLP techniques to address this goal, and methods to evaluate this approach that are potentially generalizable to other Performer approaches, including justifiable metrics for Phase 2 and Phase 3. T&E Teams will finalize protocol for evaluating explanations in Phase 1 and implement this protocol in evaluations during subsequent phases.

Figure 5 provides an example of how the TA1, TA2 and TA3 software components may ultimately fit together into an end-to-end authorship attribution and privacy system. T&E Teams will design component interfaces to ensure that systems are modularized, allowing for addition or removal of components to meet government transition needs.

4 Fluency and meaning are general terms encompassing a variety of considerations. For example, preservation of meaning may include faithfulness to lexical semantics, propositional semantics, pragmatic felicity, etc. Fluency refers to grammatical well-formedness with respect to a putative author’s mental grammar as well as genre-appropriateness.

1.A.2. Program Phases

The HIATUS program is anticipated to be a 3.5-year (42 month) effort, comprised of three (3) Phases. All three (3) Phases are being solicited under this BAA. Phase 1 will be 18 months in duration, Phase 2 will be 12 months, and Phase 3 will be 12 months. Each phase will encompass the three Task Areas (TAs) described above.

Offerors are required to propose to all three Phases and all Task Areas of the Program under this BAA. Proposals that submit to less than the three Task Areas or less than three Phases per Task Area will be considered non-compliant.

HIATUS aims to create systems that can accomplish authorship attribution and privacy on the vast array of text-types (differing in topic, genre, language, length, author background, etc.) encountered by IC stakeholders in actual operational scenarios. Program phases are designed to test Performer systems against increasingly challenging data scenarios. The following paragraphs describe the phases and Table 1 summarizes and distinguishes the differences among the program phases.

Table 1: HIATUS Program Phases

Area Phase 1 (18 months) Phase 2 (12 months) Phase 3 (12 months)

Number of Government research datasets

20+ from diverse genres and domains

20+ additional corpora per language from diverse genres and domains

20+ additional corpora per language from diverse genres and domains

Query document long texts are >700 words Medium texts are estimated at 100-200 words on average

Short texts are estimated at 20-30 words on average

Number of ground truth documents presented to Performer system during evaluation as queries5

1-3 per author 3-5 per author 7-9 per author

5 Query documents may reflect different genres and domains.

Evaluation Language(s) English English + Phase 2 surprise language (announced at phase kickoff)

English + Phase 2 + two new Phase 3 surprise languages (announced at phase kickoff)

Multi-authorship Single author per document

Single author per document

Multi-author for some larger documents

Human feedback for privacy

Two conditions: human-in-the-loop (HITL) and fully automatic

Same as Phase 1 with reduction in allowed human effort for HITL condition

Same as Phase 2 with reduction in allowed human effort for HITL condition

Within each program phase, there are three evaluation milestones at which Performers submit software deliverables to the T&E team for evaluation against hidden datasets. The first of these three milestones (1.1, 2.1, and 3.1) evaluates the quality of Performer TA1 software deliverables on a rank and retrieval task. The remaining two milestones (1.2, 1.3, 2.2, 2.3, 3.2, 3.3) are mixed open search/rank and retrieval tasks that simultaneously evaluate the quality of Performer TA2 and TA3 software deliverables (see Figure 6).6

Figure 6: Mapping of TAs to Milestones and Phases7

Each evaluation will be performed on sequestered datasets using Performer-submitted containerized software components.8 Aspects of evaluation that involve human adjudication or feedback will be performed by the T&E Team.

In Phase 1, IARPA will provide development datasets and an evaluation harness developed by the T&E team. The same harness will be used for testing and evaluation of Performer-submitted software components at each subsequent evaluation milestone. The purpose of the evaluation harness is to enable Performer-site training, testing and competition between attribution and privacy components. IARPA will also furnish to Performers documentation of the program Application Programming Interface (API)

6 In Phase 1, a ‘dry run’ evaluation milestone allows performers an opportunity to test out their TA2 systems. This risk reduction exercise is indicated by an asterisk in the timeline figure.

7 See 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline for a detailed timeline of evaluation milestones 8 Systems shall be permitted to ‘fine-tune’ on evaluation corpora, but ground-truth author labels will not be exposed to systems for the purpose of fine-tuning.

developed by the HIATUS T&E Team. This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner.

Datasets developed by the T&E Team for each program phase will be partitioned into development and evaluation datasets. Although the T&E Team will deliver some development data, Performers are expected to create their own development datasets aggregated from open source or self-generated and using creative techniques and heuristics to infer authorship labelling. These datasets must be submitted to the T&E Team at the conclusion of each phase, combined into a single dataset, such that it is releasable to all Performers as discussed in section 1.D.1.2.

Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are sequestered. A portion of these datasets will be shared with Performers after each evaluation to enable Performers to conduct error analyses. The baseline API provided in Phase 1 will standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in section 1.D. and more details on the API are available in section 1.G.3.1.

1.A.2.1. Phase 1

The goal of Phase 1 is to develop novel human-useable systems for author fingerprinting, attribution, and privacy. Attribution systems developed during this phase should succeed at performing rank and retrieval (TA1) and open search (TA2) tasks using 1-3 single-author (human or machine), large-length query documents (>700 words) in English. Systems will use query documents to find other documents from the same author in a corpus comprising documents from 20+ domains and genres. Performers shall also deliver author privacy (TA3) systems that successfully remove author-identifying features from large-length English documents, thereby effectively thwarting authorship attribution systems. TA3 systems must function both with and without human feedback.

Throughout the first phase of the program Performer research will focus on foundational explorations and investigations into different approaches, theories, and concepts to establish the building blocks of their HIATUS system(s). During this time, Performers and the T&E Team will collaborate on a protocol to evaluate human explainability for TA2 and assessment of fluency and sense preservation in TA3. This protocol will be implemented in Phases 2 and 3. As a precursor to this collaboration, Proposals in response to this BAA must articulate a novel approach to the explainability challenge, addressing how their HIATUS systems would furnish explanations to users allowing them to understand and verify system behavior. Two categories of an end user are envisioned: a non-expert analyst using the system explanations to understand system rationale, flag spurious attributions, gauge attribution confidence, etc.; and an expert with sophisticated linguistic knowledge conducting highly nuanced authorship analysis. Offerors must be explicit in their proposals about how their explanatory techniques could be evaluated.

With the aims of mitigating risk and investigating competing approaches, Performers are encouraged to pursue several novel approaches under the same subtopic. Performers will compare and contrast various approaches using Government furnished development datasets as well as datasets collected by teams internally.

1.A.2.2. Phase 2

The goal of Phase 2 is for Performers to extend the capabilities of author fingerprinting, attribution and privacy systems developed in Phase 1. Attribution systems should succeed at performing rank and retrieval (TA1) and open search (TA2) tasks using 3-5 single-author (human or machine), medium-length query documents (~100-200 words). In this phase, performer systems will be evaluated using separate corpora in English and a second target language which will be announced at the phase kick-off. Systems will use query documents to find other documents from the same author in a corpus comprising documents from 40+ domains and genres. It is expected that performer systems will be required to perform cross-/multi-lingual attribution by finding documents created by the same author in multiple languages, given a query document by this author in one language. While each evaluation haystack is expected to contain documents in a single language, some level of naturally occurring code-switching or language mixing should also be expected.

Performers shall deliver author privacy (TA3) systems that successfully remove author-identifying features from long and medium-length documents in English and the Phase 2 target language. In addition to performing attribution, TA2 systems will furnish to the human user feedback explaining system rationale for authorship attributions. TA3 systems must function both with and without human feedback, but human effort must be limited.

1.A.2.3. Phase 3

The goal of Phase 3 is for Performers to further extend author fingerprinting, attribution and privacy capabilities developed in Phases 1 and 2. Attribution systems should succeed at performing rank and retrieval (TA1) and open search (TA2) tasks using 7-9 human or machine-authored, short-length query documents (~20-30 words) in English, the Phase 2 target language, and the two new target languages to be announced at the phase kick-off, with some documents produced by multiple authors. Systems will use query documents to find other documents from the same author in a corpus comprising documents from 60+ domains and genres. As in Phase 2, cross-/multi-lingual attribution may be required. Performers shall deliver author privacy (TA3) systems that successfully remove author-identifying features from large, medium, and short-length documents in English and the identified target languages. TA2 systems will furnish improved feedback explaining system rationale for authorship attributions. TA3 systems must function both with and without human feedback, but human effort must be minimal. Texts from the same author modified by TA3 should exhibit a high level of stylistic self-consistency.

1.B. Team Expertise

Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and should include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research. Proposals should include a description and the mix of skills and staffing that the Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

• Adversarial machine learning

• Authorship attribution

• Data visualization

• Deep metric learning

• Distributed processing

• Domain adaptation

• Explainable NLP

• Forensic linguistics

• Human computation

• Information retrieval

• Language embeddings

• Software engineering

• Style transfer

• Systems engineering

• Systems integration

• Text dataset development

• Theoretical linguistics

Although the HIATUS program is focused on computational approaches, teaming schemes that include meaningful collaborations with forensic linguists and practitioners of non-computational text attribution are highly encouraged.

1.C. Program Scope and Limitations

Proposals shall explicitly address all the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each identified risk.

• Software development: Proposals shall describe the approach to software architecture, modularization, and integration.

The following areas of research are out of scope for the HIATUS program:

• Research that does not have strong theoretical and experimental foundations.

• Approaches to authorship attribution and privacy that rely on non-text features (e.g., timestamps, network structure, multimedia). The program is limited in scope to text-only approaches to attribution and privacy.

• Approaches to authorship attribution or privacy that rely exclusively on predetermined sets of linguistic features selected by humans

• Development of hardware.

• Research that utilizes proprietary data that are not made available to other Performers.

• Approaches that consist merely of integrating currently existing software.

When creating development datasets, Performers are permitted to use non-text features to establish ground-truth and near-ground-truth authorship labels for text documents collected from open source. However, testing corpora will each comprise a collection of text documents in which metadata is concealed from Performers at test time.

Delivered software will be evaluated by the T&E Team on sequestered evaluation datasets. During each performance period, Performers will build software systems that will be run and evaluated by the T&E Team. Testing protocols do not allow for expert operators, but limited human-in-the-loop operation is permitted for privacy systems and will be carried out by the T&E Team. Notwithstanding testing and evaluation tasks that explicitly require human feedback, systems must be fully automated. Systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided they run autonomously.

1.D. Program Data

The HIATUS program seeks to simulate the challenge of attribution and privacy in real-world circumstances. Common datasets used for modeling authorship have one or more of the following shortcomings that render them inadequate for the development of robust attribution and privacy systems:

1. Closed world: Finite known set of authors

2. Limited genre

3. Limited topic domains

4. Lack of cross-topic/genre authorship: samples of the same author across diverse text types are limited or absent

5. Lack of author diversity: authors derive from similar backgrounds or demographics

6. Lack of linguistic diversity: focus on single language or dialect

The HIATUS dataset is an open world collection (i.e., unknown number of authors) that is both massively multi-genre and multi-topic. Ground-truth authors represent diverse demographic and linguistic backgrounds. To ensure sufficient data are provided for system development and statistically reliable evaluations, the program T&E Team will implement a robust and explicit data collection workflow.

The program will include two types of data – Development Data and Evaluation Data. Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the HIATUS program metrics. A summary of HIATUS datasets across the entire life of the program is included in Table 2.

Offerors shall include in their proposals the cost and resources to acquire, implement, and maintain the computing infrastructure necessary to carry out all work described in their proposals, including dataset development.

Table 2: HIATUS Program Datasets

Phase Type Origin Dataset9 Month

# Ground-truth-labelled

Human Authors10

# Ground-truth-labelled

Machine Authors

# Corpora11

1 Development (English) T&E HRS1 1 50 Est. 50 Est. 5+ corpora

Est.

1 Evaluation (English) T&E HTS1 5 200 Est. 200 Est. 20+ corpora

Est.

1 Development (English) Performer HRC1 Ongoing Unlimited Unlimited Unlimited

Development

(Phase 2 target Language)

T&E HRS2

(phase kickoff)

50 Est. 50 Est.

5+ corpora per language Est.

2 Evaluation (Phase 2 target Language) T&E HTS2 21 200 Est. 200 Est.

20+ corpora per language

Est.

Development

(Phase 2 Target Language)

Performer HRC2 Ongoing Unlimited Unlimited Unlimited

2 Development (English) T&E HRS2

(phase kickoff) 50 Est. 50 Est.

5+ corpora per language

Est.

9 HRS = HIATUS Research Set, HTS = HIATUS Test Set, HRC = HIATUS Researcher Collection 10 Minimum number of unique individuals in the probe set with mated entries in the dataset gallery.

11 Corpora contain an unknown (and often large) number of anonymous authors.

Phase Type Origin Dataset9 Month

# Ground-truth-labelled

Human Authors10

# Ground-truth-labelled

Machine Authors

# Corpora11

2 Evaluation (English) T&E HTS2 23 200 Est. 200 Est.

20+ corpora per language

Est.

2 Development (English) Performer HRC2 Ongoing Unlimited Unlimited Unlimited

Development

(Phase 3 Target Languages)

T&E HRS3

(phase kickoff)

100 Est. (50 each)

100 Est. (50 each)

10+ corpora per language

Est.

3 Evaluation (Phase 3 Target Languages) T&E HTS3 33 400 Est. (200 each) 400 Est. (200 each)

40+ corpora per language

Est.

Development

(Phase 3 Target Languages)

Performer HRC3 Ongoing Unlimited Unlimited Unlimited

3 Development (English) T&E HRS3

(phase kickoff) 50 Est. 50 Est.

5+ corpora per language

Est.

3 Evaluation (English) T&E HTS3 33 200 Est. 200 Est.

20+ corpora per language

Est.

Development

(English and all target languages)

Performer HRC3 Ongoing Unlimited Unlimited Unlimited

1.D.1. Development Data

Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training. Each Performer is anticipated to have a unique technical solution to the HIATUS challenge, and accordingly may require different development data depending on their approach.

Proposals must specify the development data needed to carry out the proposed research and what data characteristics are necessary for the Offeror’s approach(es) to be successful at meeting program objectives.

Additionally, as part of Volume 1 (Technical & Management Proposal) Section 3 (Detailed Proposal), Offerors should present a dataset development plan detailing how the Offeror intends to obtain these data.

Offerors’ unique data needs in a particular technical area should not be interpreted as requirements input into the program’s T&E data collections, but part of a comprehensive research approach that communicates a possible path towards meeting HIATUS program objectives.

There will be three types of development data used in the HIATUS program:

• Government Research and Test Sets – Datasets collected, annotated, and curated by the T&E Team. Most of these data will be sequestered, but a small portion will be provided to Performers for R&D (prior to evaluation) and error analysis (following evaluation). Performers should anticipate receiving less of this data in Phases 2 and 3.

• Researcher Collections – Datasets collected, annotated, and curated by R&D Performers. The delivered datasets shall not limit the Government’s transition goals in accordance with BAA Section 4.B.1.d. Section 4: Attachments, Attachment 2: IP Rights.

• External Data Sources – Data obtained by Performers that are available from third parties or that have been collected by a Performer outside of HIATUS shall be delivered with sufficient rights to allow the United States Government (USG) and the T&E Team to share this data with all Performers for their use in connection with the HIATUS Program.

1.D.1.1. Government Research and Test Sets

The T&E Team will develop HIATUS Research Sets (HRS) and HIATUS Test Sets (HTS) throughout the program. The Government will release HRS to Performers to facilitate system development. HTS will be sequestered for the purpose of blind testing and evaluation, though small portions of this dataset may be released for the purpose of error analysis following official evaluation milestones. The volume of documents in the HTS is anticipated to be much larger than the HRS.

See Table 2 for a preliminary summary of all HRS datasets. These datasets will include numerous ‘haystacks’ i.e., collections of documents drawn from various sources of text content from the internet and elsewhere. A haystack may, for example, consist of a collection of posts from an online hobbyist forum.

Each haystack will consist only of text documents. The T&E Team will implement filtering to remove Personally Identifiable Information (PII) and other metadata (e.g., timestamps, network data). The haystack portion of datasets may incidentally contain metadata, such as information within the body of a document indicating that two or more documents are authored by the same individual, but Performers are prohibited from using such metadata in their systems.

The T&E Team will augment the haystack collection with human-authored and machine-authored ground-truth-labelled documents. Human-authored documents will be elicited from individuals contracted by the T&E Team to author documents for specific haystacks. The T&E Team will vet human authors and the text content they produce to verify contributions are appropriate to and not otherwise aberrant to the target haystack. Machine-authored documents will be produced by a variety of text-generation models selected by the T&E Team and fine-tuned to the target haystack. Each ground-truth-labelled author may be asked to contribute to one or more target haystacks. There will be no overlap between the ground-truth-labelled authors included in the HRS and HTS datasets.

The T&E Team will continuously develop HRS and HTS data sets throughout the program, altering the characteristics of the datasets they develop to meet evaluation goals. The specifics of these collections will be determined by the T&E Team during the program, but for purposes of this BAA, Offerors can assume the following characteristics within each of the HRS datasets:

• HTS and HRS will be a collection of dozens of haystack corpora collected from open source

• Each haystack originates from a unique source (e.g., comment posts from the same web domain)

• Documents from the same haystack will tend to be from similar genres and topic domains, but this is not necessarily the case

• HTS documents will not be provided with labels for authorship or handle at test time, nor will any additional metadata be included other than non-informative labels (e.g., numeric codes) indicating the haystack to which a document belongs

• HTS and HRS contain a subset of documents labelled for ground truth authorship

• Documents will be normalized using methods that ensure consistency across sources, but minimize loss of document characteristics that are potentially predictive of author

1.D.1.2. Researcher Collections

As indicated in the previous section, the Government will provide limited datasets (HRS) to Performers for system development. Each Performer is required to plan and carry out internal authorship dataset development efforts. To minimize the risk of system biases, Performers must plan and implement steps to ensure that datasets include demographically diverse author populations. Datasets assembled by Performers are referred to as HIATUS Researcher Collections (HRC). Performers are required to submit their most current HRC to the Government in a standard format at each evaluation milestone. Performers are permitted to withhold their internal HRC from other Performers until the end of each program phase. At the conclusion of Phases 1 and 2, the T&E Team will combine each Performer’s HRC into a unified collection and will release this collection to all Performers. Sharing of the combined HRC helps ensure that phase-over-phase improvement in performance on evaluation tasks is attributable to a team’s superior technical approach rather than a greater accrual of development data resources.

As part of their proposal, each Offeror should provide a data collection plan and describe how their HRC will be used in R&D. Performers cannot control the quality and depth of data from other teams’ collections, but they can ensure their own collections meet or exceed their own research needs. Details regarding text normalization techniques will be released to Performers by the T&E Team during kickoff. Normalization techniques will provide a consistent document format across all datasets while minimizing removal or modification of author-predictive text attributes.

Performers will be responsible for annotating all HRC data for their R&D needs. All annotation data are to be delivered to the Government regardless of whether they are developed after the collection or after initial delivery of the HRC dataset. Annotation methodologies should be consistent with current research best practices to facilitate supervised machine learning algorithm development.

All data collected, curated, annotated, synthesized, or generated by Performers created under the HIATUS program for R&D or any other purpose will be provided to the Government as Deliverables. Software tools or algorithms created for the purpose of data annotation, synthesis, or generation under the HIATUS program for R&D purposes or otherwise shall also be provided to the Government as Deliverables with unlimited rights (but will consider GPR in limited circumstances) in accordance with FAR 52.227-14, to allow use by the USG. IARPA reserves the right to require delivery of incomplete or partially annotated versions of HRC datasets with 30 days’ advance notice.

1.D.1.3. External Data Sources

Performers may utilize external datasets in addition to HRS and HRC datasets. External data are data obtained by Performers that are available from third parties or that have been collected by the Performer outside of the HIATUS program. An example of a third-party dataset would be an authorship attribution dataset, collected by a university under an IRB-approved HSR protocol, that has been approved by the dataset owner for release to the research community. Data collected by a Performer under a different program are considered external data, even if the other program’s data collection was Government-sponsored.

All external datasets must be approved for use in the HIATUS program by IARPA during contract performance, in accordance with HSR and applicable privacy policies, statutes, and regulations.

Performers may not use proprietary datasets unless these datasets are made available to all R&D Performers and the T&E Team in the HIATUS program. Public release of proprietary datasets is not a requirement;

however, release for use within the HIATUS program is required. Moreover, for any dataset not collected under the scope of the HIATUS program, Performers must provide the Government with an accounting of all resources used and sources from which data are drawn and describe how the data will be used for development, testing, and training of algorithms.

1.D.2. Evaluation Data

HIATUS will utilize distinct test data to evaluate Performer subcomponents, modules, and systems against program goals, objectives, and metrics. Each HIATUS Test Set (HTS) will consist of diverse documents in English and target languages (in Phases 2 and 3) representing a broad sampling of topic domains and genres.

These data are constructed to evaluate program metrics in a balanced and intentional manner. The set of ground-truth-labelled authors appearing in this collection will not overlap with that of the HRS dataset.

Moreover, to the extent that HTS data is released to Performers following evaluation milestones for the purpose of error analysis, all documents produced by ground-truth-labelled authors appearing in these releases will be removed from the portion of the HTS that remains sequestered. This ensures that no authors whose content was used for system development will be included in the testing set, and that systems performance generalizes.

Intended uses of the HTS evaluation datasets include both use by the T&E Team for independent evaluation of program Deliverables against target metrics, and use by Performers to refine and improve their algorithms. Small portions of the evaluation datasets will be provided to Performers to enable internal T&E and exploratory error analysis by Performers and to improve the consistency and communication between Performers and the T&E Team. Additional sequestered or external datasets may be used to supplement performance evaluations at the discretion of the HIATUS PM.

1.D.3. Privacy Plan

As part of their proposal, Offerors shall prepare a HIATUS Privacy Plan v1.0 that comprehensively describes the efforts the Offeror will take to protect personally identifiable information and safeguard the security of any personal data collected or services involved in collection, transmission, processing, and storage of this data. Any claims that data are anonymous must be based on evidence and supported with sufficient information regarding how the data have been anonymized. Each Performer’s Privacy Plan shall cover all data categories described in this BAA; relevant information on GFI datasets will be provided by the T&E Team in advance of data release.

The initial version of the HIATUS Privacy Plan shall be included in the Offeror’s proposal (as Attachment

12) that covers all external datasets to be leveraged as part of the proposed research approaches. The HIATUS Privacy Plan shall be updated at the beginning of each Phase and whenever new sources of data or datasets are proposed for use within a Performer’s HIATUS research activities, including data used for either development or evaluation purposes. Details on government-provided research datasets (see 1.D.1.1.

Government Research and Test Sets) will be provided by IARPA at the Phase 1 Kick-off Meeting for inclusion in the HIATUS Privacy Plan v1.1 Deliverable (see Table 4).

1.E. Test and Evaluation (T&E)

T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of Performer research Deliverables using program test datasets and protocols. The HTS data and test protocols (see 1.D.2. Evaluation Data) will be the primary mechanism by which the T&E Team carries out their evaluations.

The HIATUS Program…