QA Round 2 HIATUS IARPA-BAA-22-01_C.pdf

Human Interpretable Attribution of Text using Underlying Structure (HIATUS) Program

IARPA-BAA-22-01

Industry Questions and Government Responses Round 1 (Q.1 – Q.4) Published on 03/11/2022

Round 2 (Q.5 – Q.46) Published on 04/04/2022

Question Round Industry Question Paragraph & Page #

Government Response Change to BAA (Yes/No)

Paragraph & Page # for Change

1 1

Is it possible to streamline involvement of foreign team members from Five Eyes countries with respect to export controls and security regulations?

Would coordination between partners' foreign governmental sponsors and IARPA be possible, to help ensure that all such requirements are properly met?

3.A. p.29

Per Section 3: Eligibility Information, 3.A. Eligible Applicants, of the BAA, “Foreign entities and/or individuals may participate but only as a part of a U.S. based team. The prime contractor must be a U.S.

organization. Foreign entities and individuals may participate as subcontractors or employees of a U.S. organization; however, all foreign participation must comply with any necessary Non-Disclosure Agreements, Security Regulations, Export Control Laws, and other governing statutes applicable under the circumstances. Offerors are expected to ensure that the efforts of foreign participants do not either directly or indirectly compromise the laws of the United States, nor its security interests. As such, both foreign and domestic Offerors should carefully consider the roles and responsibilities of foreign participants as they pursue teaming arrangements.” At this time, IARPA cannot commit to streamlining Export Controls and Security Regulations.

No N/A

Resultant contract awards will include the following IARPA Export Control Clause.

IA52.204-703 -- Export Control (Jul 2013)

(a) The contractor shall comply with all U.S. export control laws and regulations, including the International traffic in Arms Regulations (ITAR), 22 CFR Parts 120 through 130, and the Export Administration Regulations (EAR), 15 CFR Parts 730 through 799, in the performance of this contract. In the absence of available license exemptions or exceptions, the contractor shall be responsible for obtaining the appropriate licenses or other approvals, if required, for exports of (including deemed exports) hardware, technical data, and software, or for the provision of technical assistance.

(b) The contractor shall be responsible for obtaining export licenses, if required, before utilizing foreign persons in the performance of this contract, including instances where the work is to be performed on-site at any Government installation (whether in or outside the United States), where the foreign person will have access to export-controlled technologies, including technical data or software.

(c) The contractor shall be responsible for all regulatory record keeping requirements associated with the use of licenses and license exemptions or exceptions.

(d) The contractor shall appropriately mark all contract deliverables controlled by ITAR and/or EAR.

(e) The contractor shall ensure that the provisions of this clause apply to its subcontractors.

2 1 Can IARPA resolve OCI's prior to proposal submission? 3.A.1., p.29-

IARPA does not have the resources to review and mitigate all OCI's for potential offeror prior to proposal submission. Within Section 3.A.1, Organizational Conflicts of Interest, of the BAA offerors are advised that, "The Government will make OCI determinations, as applicable, for proposals that are otherwise selectable under the BAA Evaluation Factors". Please refer to section 3.A, Eligible Applicants, to determine if you are eligible to submit a proposal, and section 3.A.1, Organizational Conflicts of Interest, for instructions on how to submit an OCI Mitigation Plan.

No N/A

3 1

The BAA says that the HTS and HRS will each be a collection of dozens of haystack corpora. It sounds like there will be a numeric code at both development and test time indicating which haystack a given document comes from.

Should we expect to see the same author across multiple haystacks?

(That is, blogger_jane_smith

1.D.1.1, p.17

Performers should expect that documents produced by the same author may appear in multiple haystacks.

writing posts in both haystack21 and haystack29.) Or are the authors contained in each haystack unique to that haystack?

4 1

The HIATUS BAA says that the output of the privacy system should be stylistically self-consistent. That is, if five documents from a single author have been anonymized, they should still look, stylistically, like they are produced by the same author. The BAA does not comment on whether it is required that documents authored by two distinct authors still look like they were authored by two distinct people (just different people). Is a TA3 system that modifies multiple documents, each from a different author, such that those modified documents exhibit stylistic properties consistent with a single, anonymous author in scope?

1.A.1, p.9;

1.F, p.21

1.F, p. 21 of the BAA states “A final key technical consideration for TA3 is that the output of the privacy system should be stylistically self-consistent. By way of example, five documents from a single author that have each been modified by a Performer’s TA3 system should not exhibit stylistic features that make them appear to have been produced by five distinct authors; rather, all five documents should ideally exhibit stylistic properties consistent with a single author.”

In cases where a TA3 system modifies multiple documents, each from a unique author, the program requires that the modified versions of these documents be stylistically distinct, i.e., they must not be consistent with a single author.

5 2

The address http://www.iarpa.gov/faq.html does not seem to be in working order. Could you please get it fixed or refer me to the contents?

Thank you.

N/A

The BAA does not reference this web address. If this web address is referenced in IDEAS, the reference is in error. The web address is no longer active. An offeror does not need to access http://www.iarpa.gov/faq.html to submit a proposal or to register in

IDEAS.

In Table 4 of IARPA-BAA-22- 01, there appears to be a typo related to the deliverable in

1.G.2, p.24 Yes, this is a typing error.

Amendment 001 corrects this error. Yes Amendment 001: 1.G.2, p.24 - Table 4

2 Month 9. The deliverable description is “System submission for Milestone 1.2a”, and the last column lists the following Deliverables: “TA1 System; …”. Should the latter be “TA2 System; …”?

Does IARPA consider the topic(s) and/or technology developed under this project to be export controlled (either by the EAR or by ITAR)? Does the government anticipate providing any export controlled technology to performers?

6.B.6, p.49

It is the responsibility of the contractor to determine if export control is applicable, and if so, comply with applicable regulations.

See BAA Section 6.B.6, Export Control and the IARPA Export Clause (provided in Q.1) which will be included in resultant contract awards. At this time the Government cannot comment on whether or not export controlled technology will be provided to performers.

No N/A

Will there be foreign national restrictions put in place on the awarded contract?

3.A. p.29

See BAA Section 3: Eligibility Information, 3.A. Eligible Applicants No N/A

Under section I “Detailed Management Plan” table 4 outlines an example of the Team Organization information. Is a “brief biography for all Key Personnel… and significant contributors” required in addition to the information provided in table 4?

4.B.1.c, p.36

As part of the Detailed Management Plan, the offeror shall provide a brief biography of all Key Personnel (including alternates, if desired) and significant contributors who shall be involved in the research along with the amount of effort to be expended by each person during the year.

Additionally, a Team Organization table shall be provided. The recommended format for the table is provided in Section 4.B.1.c. I.

Detailed Management Plan.

Amendment 001 renumbers this table from Table 4 to Table 7.

Yes Amendment 001: 4.B.1.c.

I., p.36-37

RDMP - section L of the technical volume requires a RDMP. An RDMP (using the template provided in Appendix A) is also required as attachment

11. Should the same RDMP be included in both sections?

4.B.1.c., p.

37-38 and 4.B.1.d., p.

In the BAA, Section 4, Proposal and Submission Information, subsection,

4. B.1.c. Section 3: Detailed Proposal Information, paragraph L.

Research Data Management Plan (RDMP) of the technical volume requires the submission of an RDMP.

The information required by the RDMP shall be provided as Attachment 11 only.

Attachment 7 of the technical volume requires “human use documentation”. Section 6.B.3.

Indicates that institutions “awarded funding…must provide documentation of a current Assurance of Compliance with Federal regulations for human subject protection” (e.g. DHHS OHRP FWA). For the proposal, will an indication of the institution’s FWA number suffice for this requirement or is additional documentation required?

6.B.3., p.48-

Per Section 6.B.3., Human Use, of the BAA, "For all proposed research that will involve human subjects, the institution must provide evidence of or a plan for review by an Institutional Review Board (IRB) with the final proposal submission to IARPA as outlined in the management plan. (Reference Section 4 of the BAA). The IRB conducting the review must be the IRB identified on the institution’s Assurance. The informed consent document must comply with federal regulations (45 CFR Part 46)." The offeror can also provide the institution's Federal Wide Assurance (FWA) number.

Can you advise on when we can anticipate hearing back on the “initial round of selections”?

5.C., p.47

Per Section 5.C., Negotiation and Contract Award., "After selection and before award, the CO will contact Offerors whose proposals were selected or selected with modifications to engage in negotiations."

For the “estimated cost breakdown” (section 4.B.2.b), item C requires the cost broken down by “major program task by

4.B.2.b., p.41

The Government's fiscal year.

Amendment 001 clarifies BAA section 4.B.2.b.

Yes Amendment 001: 4.B.2.b., p.41 fiscal year” - is this the government’s fiscal year or the contractor’s fiscal year?

Three chart summary - can you clarify if the three chart summary required in the attachment section is one powerpoint slide with 3 charts (those outlined in Appendix A) or 3 slides with one chart on each slide?

4.B.1.d.,p.40

Three (3) slides with one (1) chart on each slide. Amendment 001 clarifies BAA Section 4.B.1.d. Additionally, the three (3) slides in Appendix A.5, Three Chart Summary, of the Proposal are now outlined to show they are three (3) separate slides.

Yes Amendment 001: 4.B.1.d., p.40 and Appendix A.5, p.59

We are performing independent research on the Combined Synopsis/Solicitation (IARPABAA2201). We are hoping to find out if there is (was) an incumbent contractor performing these services, or if this is a brand new contract. If there is an incumbent, could you please provide the current / previous contract number?

N/A This is a new requirement. There is not an incumbent contractor or contract number.

Are performers necessarily required to create their own datasets for development? Or is it permissible for them to use existing datasets that are open source, assuming they can find sufficient ones?

1.D.1.2.

pg.17-18 and 1.D.1 pg.16

The use of existing open-source datasets is permitted provided such use does not limit the government’s transition goals in accordance with BAA Section 4.B.1.d. Section 4:

Attachments, Attachment 2: IP Rights.

Yes Amendment 001: 1.D.1 pg.16

In Phase III, the system will have to deal with multi-author documents. In that case, what are the AA and AO systems expected to do? Should AA give high attribution scores for all authors in the draft? And should AO obfuscate all authors in the draft?

Sec 1.A.2.3.

pg.13

When provided with a multi-author document query, a successful attribution system will rank documents from one or more of the query authors higher than other documents in a collection. Likewise, a privacy system should thwart attribution of all authors in a multi-author query.

No

N/A

Given that the goal is to target style-driven attribution across multiple languages, is the expectation that the Performers will need to build a single attribution model that can address multiple languages?

Performers may choose to build a single model or a battery of models to address multiple languages. Also please note edits in Amendment 001 Section 1.A.2.2 page 12.

Yes

Amendment 001: 1.A.2.2 pg.12-13 and 1.A.2.3 pg.13

The HIATUS Program has phase one as 18 months phase two as 12 months and phase three as 12 months. I’m not sure how to put a budget together for this. I never had a proposal with 3 ½ years.

Any suggestions.

It is the offeror's responsibility to develop their budget. The Government cannot comment, assist, or advise an offeror on how they should develop their budget.

No

N/A

Authorship attribution limited to one language or across several languages? i.e., do we assume text data from one author is available in multiple languages?

You should assume that some of the authors are bilingual and produce documents in multiple languages. No

N/A

The BAA contains this statement on page 8: “In addition, Performers will be required to conduct their own supplemental IRB-approved dataset collections.” For our methods, we do not require any human-generated data during the program nor will we use human annotators to annotate data. In addition to the government-provided data, we will leverage only historical literature with expired copyrights, existing datasets widely available in the academic community, and previously-provided government data. All other data we will produce for this project will be machine-generated. Is it a

1.A., p.8

It is not a requirement that Performers collect IRB-relevant data.

requirement that performers collect IRB-relevant data even if not required by the performer’s methods?

If a nontraditional offeror requests a FFP or OTA contract type as a non-negotiable condition, is the offeror permitted to provide fixed pricing in proposal section 3.F for the government’s evaluation of funding availability (as opposed to providing cost/price information by cost element)?

We note that the Resource Realism evaluation is based on the solicitation section 4.B.1.b(F) Technical Resource Summary, so the Cost, schedule, milestones response should not be needed for that purpose.)

4.B.1.c.(F), p.36

4.B.1.b(F),

p. 34

If a nontraditional offeror requests a FFP or OTA contract type, they are still required to describe the cost, schedule, and milestones for the proposed research, including cost estimates by cost element for base period, the option period(s) and the total program summary, and company cost share, if any, as well as, costs by technical area(s) and tasks in accordance with BAA Section 4.B.1.c. Section 3: Detailed Proposal Information, paragraph F.

Cost, Schedule, Milestones. This is meant to be a high level fully burdened cost estimate where overhead, indirect rates and fees are not apparent.

The Technical Resource Summary, Section 4.B.1.b(F), only describes the resources: labor ( i.e. hours, labor categories), items such as material and equipment and travel (i.e.

number of trips/personnel per trip) without describing the costs.

No

N/A

Will the government please confirm the following: The research conducted under the HIATUS program is designated as “applied research” as this term is defined in FAR subpart 35 and is not “fundamental research.”

The Prime Contractor can designate subcontracted components as fundamental

The research conducted under the HIATUS program is applied research in accordance with FAR Part 35, Research and Development Contracting. IARPA does not provide guidance or feedback on what subcontracted components are fundamental research. The prime contractor is responsible for ensuring research without violating the terms of the awarded contract.

compliance with the ITAR and EAR as it relates to export restrictions and should not assume the fundamental research exemption. The prime contract will include clause IA52.204-703 -- Export Control (Jul 2013) that is required to flow down to subcontractors. See Answer to question #1, round 1.

confirm that the Contractor and subcontractors, including those universities listed as subcontractors, shall comply with all U.S. export control laws and regulations, including the International Traffic in Arms Regulations (ITAR), 22 C.F.R.

Parts 120 through 130, and the Export Administration Regulations (EAR), 15 C.F.R.

Parts 730 through 799, and any amendments thereto, including clause IA52.204 703, in the performance of this contract.

6.B.6., p. 49

In accordance with Section 6.B.6, Export Control, of the BAA:

(1) The Offeror shall comply with all U.S. export control laws and regulations, including the International Traffic in Arms Regulations (ITAR), 22 C.F.R. Parts 120 through 130, and the Export Administration Regulations (EAR), 15 C.F.R. Parts 730 through 799, and any amendments thereto, in the performance of this contract.

(5) The Offeror shall be responsible for ensuring that the provisions of this section apply to its sub-contractors.

Additionally, see response to Question 1, Round 1, regarding clause IA 52.204-703.

No

N/A confirm that the Contractor and subcontractors, including universities listed as subcontractors, shall comply with clauses 252.204-7012, IA

A full list of applicable FAR clauses, to include full text IARPA Clauses, will be made available for review after selection and prior to contract award. The contractor will be required to comply with clauses in any resultant award. The

52.227-701, IA 52.204-713b, IA

52.204-703, IA 252.227-7015.

subcontractor, including universities listed as subcontractors, will be required to comply with the required flow down clauses.

Clause 252.204-7012, Safeguarding Covered Defense Information and Cyber Incident Reporting, and 252.227-7015, Technical Data – Commercial Items, are both Defense Federal Acquisition Regulation (DFAR) clause. IARPA does not fall under the Department of Defense;

therefore, DFAR clauses are not applicable.

Regarding IA 52.204-703, see response to Question 1, Round 1, and Question 24, Round 2.

Regarding IA 52.227-701 and IA 52.204-713b, these are IARPA specific clauses. A full list of applicable FAR clauses, to include full text IARPA Clauses, will be made available for review after selection and prior to contract award.

Will the FAR University clauses such as Audit and Negotiations FAR 52.215-2 (Alt II), Allowable Cost and Payment 52.216-7 (Alt II), Pre-determined Indirect Cost and Rates -FAR 52.216-15, Cost Accounting Standard, Educational Institutions FAR 52.230-5, and Rights in Data General, FAR 52.227-14(IV), and Termination for Convenience of the Government (Educational and Other Non-profit

A full list of applicable FAR clauses, to include full text IARPA Clauses, will be made available for review after selection and prior to contract award. The contractor will be required to comply with clauses in any resultant award. The subcontractor, including universities listed as subcontractors, will be required to comply with the required flow down clauses.

Institutions) 52.249-5 be included in the contract to be flowed down to the university subcontractors?

The clauses that you listed are typical for what has been included in prior IARPA contracts.

Has the Government determined that human subject research is being conducted under this contract?

6.B.3., pg.

There is no requirement that HIATUS Performers conduct human subjects research. Section 6.B.3 Human of the BAA states “All research involving human subjects, to include use of human biological specimens and human data, selected for funding must comply with the federal regulations for human subject protection, namely 45 CFR Part 46, Protection of Human Subjects.”

No

N/A confirm that the Service Contracts Act does not apply to the research in this program.

The Service Contract Act, also referred to as the McNamara-O’Hara Service Contract Act (SCA), does not apply to this program.

No

N/A

Are texts in a haystack collection of document datasets listed in order of when they were created and if so, are most recent on top?

No. It should not be assumed that documents within the haystack are sorted in any fashion. No

N/A

Is each haystack collection of document datasets from only one

(1) source (e.g., the same online hobbyist forum) or are sources merged with others?

In some cases a single haystack collection may comprise documents from distinct sources. In such cases T&E will implement procedures to ensure that sources used to create the haystack collection comprise similar types of documents (e.g., culinary blog posts from multiple internet domains).

No

N/A

May university subcontractors retain Government property in accordance with FAR subpart 35?

Resultant contract awards may include FAR 52.245-1, Government Property. If required, the applicable alternate will be applied.

No

N/A

Is there a possibility performers will be working with text plus metadata for training or learning patterns? If so, can you advise

1.C., pg.14

Offerors are allowed to use any metadata for training. However, as stated in Sec 1.D.1.1 pg.17 HIATUS Test Sets used in evaluations "will what specific types and/or data formats?

not be provided with labels for authorship or handle at test time, nor will any additional metadata be included other than non-informative labels (e.g., numeric codes) indicating the haystack to which a document belongs"

Will performer-shared datasets be required to meet some balance of machine-generated and human-generated data?

No.

N/A

Will the same domains and/or modality (type of document) be included for each language?

No.

N/A

Will any evaluation require the identification of an author in a language that differs from the language of provided examples for that author? Meaning, will we need to address multiple languages in the same evaluation?

Yes, in later phases of the program systems will be required in some cases to perform cross-/multi-lingual attribution by finding documents created by the same author in multiple languages, given a query document by this author in one language.

Yes

Amendment 001: 1.A.2.2, pg.12-13 and 1.A.2.3, pg.13

Will there be additional hidden training data at evaluation? If so, is the hidden data in English or in the target language?

There will be no additional training data. However, performer systems will be permitted to be fine-tuned to unlabeled haystack collections while running on the T&E server.

No

N/A

Are we given the evaluation data ahead of time, or is it all hidden for the three evaluation milestones within each phase?

1.A., pg. 5

Evaluation data is all hidden. As stated on page 11 in Section 1.A.2, performers will be submitting their containerized systems to T&E ahead of the evaluation, and T&E will run those systems on sequestered evaluation data located within their T&E infrastructure.

No

N/A

Does IARPA allow nested subcontractor arrangements?

Specifically, can a University subcontract to another University

It is the responsibility of the Offeror to determine their teaming arrangement. The BAA does not that is a subcontractor to the prime, or do all subcontractors need to contract directly with the prime?

prohibit a University from subcontracting to a University.

What constitutes verifiable 'linguistic fingerprints'? Currently existing automated AI programs do not process Authorship Attribution via identifiable linguistic features other than very general ones such as lexical richness and general sentence structure such as passive v.

active, etc. Do these rise to the level of verifiable 'linguistic fingerprints'? Further, what level of “obfuscation” will be needed to satisfy the BAA? A “reversal” of these general patterns would likely be the expected pathway to automatically obfuscate.

Sec 1.A, pg.5

"Verifiable linguistic fingerprints" are representations of author-predictive features that can be demonstrated to be true, accurate and/or justified. The program seeks novel techniques in explainable NLP to achieve greater levels of verifiability, significantly exceeding current state-of-the-art. For the satisfactory level of obfuscation, Section 1.F of the BAA (page 21) states: “The goal of TA3 is authorship privacy. The HIATUS program defines the success of a Performer’s privacy algorithm in terms of (i) how well it thwarts the attribution systems developed by opponent Performer teams, while at the same time (ii) ensuring that the modified document maintains fluency (soundness) and remains faithful to the semantics of the original query document (sense).”

The same section then describes how TA3 systems will be evaluated.

No

N/A

Would IARPA consider using the type of automated authorship identification technology being designed by FLINT Inc. that uses linguistic features identified by Dr. Robert Leonard, and admitted via his testimony under both Daubert and Frye in cases of murder and other serious crimes?

(See, e.g., 2017. “Forensic Linguistics” in Handbook of

5.A.1., pg.

45 and 1.B.

pg., 13-14

IARPA seeks solutions to the HIATUS challenge that significantly exceed current capabilities. As indicated in Section 5.A.1.a. (p.45), proposed approaches must advance the state-of-the-art. Criteria for evaluation of technical merit of each Offeror’s proposed approach can be found in Section 5.A.1. of the BAA.

In general, IARPA does not fund incremental improvement of current

Behavioral Criminology:

Contemporary Strategies and Issues. Van Hasselt, Vincent, and Michael Bourke, eds. Springer.

and 2017. “Forensic Linguistics:

Applying the science of linguistics to issues of the law.”

Co-authors Juliane Ford and Tanya Karoli Christensen. 45 Hofstra Law Review 881.) The Flint authorship identification algorithm under construction was primarily aimed at combating academic deception, so-called “contract cheating,” in which students purchase newly created bespoke writings and submit them as their own academic work. Flint’s algorithm aims to automate the techniques and provide academic integrity officers precise ‘reason codes,’ that is, identification of non-random patterns of matching linguistic features, when confronting suspicious manuscripts.

capabilities. For HIATUS, IARPA welcomes novel, high-risk/high-reward approaches that draw on the theoretical and empirical insights of forensic linguistics. Moreover, section 1.B. (p.13-14) states “teaming schemes that include meaningful collaborations with forensic linguists and practitioners of non-computational text attribution are highly encouraged.”

The BAA specifies that systems “furnish human-interpretable feedback for trust and verification of system behavior.” Please confirm that the HIATUS Program intends to meet or exceed the results of programs such as the Flint Authorship Attribution algorithms that are aimed at combating academic deception and “contract cheating” and that provide precise “reason codes,” pg.6

IARPA seeks solutions to the HIATUS challenge that significantly exceed current capabilities. As indicated in Section 5.A.1.a. (p.45), proposed approaches must advance the state-of-the-art.

i.e., features explainable by a human. (For background see e.g., 2017. “Forensic Linguistics” in Handbook of Behavioral Criminology: Contemporary Strategies and Issues. Van Hasselt, Vincent, and Michael Bourke, eds. Springer. and 2017.

“Forensic Linguistics: Applying the science of linguistics to issues of the law.” Co-authors Juliane Ford and Tanya Karoli Christensen. 45 Hofstra Law Review 881.)

Not a question, but a possible typo to correct: Figure 2.

"minimize minimal" -> "minimize" pg.6

It is a typing error; corrected in Amendment 001, Section 1.A, pg.6 Yes

Amendment 001: Sec 1.A, pg.6

BAA indicates that the query document may be in a genre or domain that differs from the test documents at the evaluation time.

For multiple languages, our understanding is that the approach will need to scale to multiple languages, though there won't be a cross-lingual setting (that is, the query document is in a language that differs from the evaluation documents). Please correct us if this understanding is wrong.

Query documents may differ from the evaluation documents in terms of genre, domain and language.

However, cross-lingual attribution scenarios (e.g., attribution of a non- English text using an English query) will be introduced only at a later phase of the program Yes

Amendment 001: 1.A.2.2, pg.12-13 and 1.A.2.3, pg.13

Figure 6: could you clarify what are "non-linked authors" in milestone 3?

1.A.2, Fig 6,

pg. 11

A ‘non-linked author’ is an author for which there are queries but no matching documents in the haystack collections

No

N/A

For human-in-the-loop (HITL) for TA3, are there any constraints that our proposed HITL work

1.F., pg. 21

Constraints on HITL interactions will be denominated in terms of time and edits. As stated in Section 1.F, pg.21, should follow (e.g, time allowed, types of HITL interactions)?

‘[t]he T&E Team will provide baseline measures of human effort for TA3 calculated in terms of edits made or time spent curating a document.’ Total allowed human interaction with TA3 systems will be calculated as a percentage of these baseline measures.

Can you clarify the difference between the material requested in: (i) Volume 1, Section 3.L (Research Data Management Plan), and (ii) Volume 1, Section 4, Attachment 11: RDMP?

See response to Question 10, Round 2.