IARPA-RFI-22-08_Final_fixC.pdf

Request for Information (RFI) on Annotated Malicious Binaries and Surrogate Source Code for Authorship Exploration

IARPA-RFI-22-08

IARPA is seeking information about annotated binary and source code datasets to explore authorship (for benign tools, malicious tools, or penetration testing tools) that have been annotated with relevant information that could be obtained and made available to research partners in potential future programs. This RFI is issued for planning purposes only and does not constitute a formal solicitation for proposals or suggest the procurement of any material, data sets, etc., and it is intended to serve as due diligence to assist in identifying potential datasets which may not be as easily identified through a scientific literature review.

Background

A challenge in the ability to explore the authorship of malicious binaries and source code is the availability of sizeable annotated datasets. This challenge is caused by the need for annotations in the data indicating authorship. Datasets containing additional metadata, beyond authorship, may also be useful in bridging additional scientific gaps in source code or binary authorship research. One of the largest binary datasets in academic literature is the APT Class dataset [1], and at least one research paper references datasets obtained from commercial companies [2]; few details are provided about these datasets.

In order to explore malicious source code authorship, surrogate data may be necessary, as malicious source code may be limited or non-existent. Google Code Jam (GCJ) datasets appear to be one of the more common datasets used for exploring authorship of source code [3-7], however they are generally smaller snippets of code. The concept of determining authorship of both source code and binaries is an interesting challenge, but few datasets merge source code with compiled binaries. One known example is in Caliskan-Islam et al. [3], where they compiled C/C++ code within the Google Code Jam datasets.

Scope

We are aware that a focus of malicious software means that data may not exist – especially for source code. Surrogate datasets that can be used to explore authorship features of source code and/or binaries are an acceptable response to this RFI. The below questions will help IARPA evaluate any datasets that would be sufficient for exploring authorship features in these domains.

1. What commercial or academic datasets exist that contain authorship annotations of malicious binary executables?

a. What metadata is provided in such datasets?

b. What is the general size and significance of the dataset?

c. How has this data been ground-truthed? (e.g., usernames of publishers of binaries, threat intelligence groups, etc.)

d. What is the mechanism for obtaining identified datasets? If a commercial company maintains a potentially relevant dataset for internal use, can this information be purchased for research purposes?

2. What commercial or academic datasets exist that contain authorship annotations of Source Code? Surrogate benign software, surrogate malicious code (i.e. has there been a “malware” coding competition to design tools exploiting vulnerabilities that may have collected authorship tied to source code), or pen-testing tools.

a. What metadata is provided in such cases?

b. What is the general size and significance of the dataset?

c. How has this data been ground-truthed? (e.g., usernames, threat intelligence groups, etc.)

d. What is the mechanism for obtaining identified datasets? If a commercial company maintains a potentially relevant dataset for internal use, can this information be purchased for research purposes?

3. What commercial or academic datasets exist that contain source code tied to specific binaries with authorship annotations?

a. What metadata is provided in such cases? Are authors, groups, or companies included in the annotations?

b. What is the general size and significance of the dataset?

c. How has this data been ground-truthed? (e.g., usernames, threat intelligence groups, etc.)

d. How was source code obtained? (e.g., reverse engineering, dark web, etc.)

e. What is the mechanism for obtaining identified datasets? If a commercial company maintains a potentially relevant dataset for internal use, can this information be purchased for research purposes?

4. Are there any ongoing efforts to build datasets that are reflected in questions 1-3?

Preparation Instructions to Respondents:

IARPA requests that respondents submit responses to the above questions for use by the Government in formulating a potential program. IARPA requests that submittals address one or more of the above questions (to include all sub-bullets of the question). If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) estimate regarding what purchase of commercial datasets might require in terms of funding and other resources to execute a program that might require sharing such datasets with additional research partners. This announcement contains all information required to submit a response. No additional forms, kits, or other materials are needed.

IARPA welcomes responses from all capable and qualified sources including academia, industry, government, national laboratories, and other federally funded research entities from within and outside of the U.S. This includes perspectives and associated capabilities for effectively evaluating potential research approaches.

Reponses must meet the following formatting requirements:

1. A one-page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with IARPA-

RFI-22-08;

2. A response to one, some, or all questions posed above with a maximum of three pages.

3. A list of references as necessary – citations do not count against the page limit;

4. Additional Appendices further describing the dataset in full detail can be provided – especially if the respondents are owners, maintainers, or familiar with the datasets from prior research. The sub-bullets list types of appendices would be beneficial to our understanding of the utility of potential datasets:

1. Data structure diagrams

2. Entity-Relation diagrams of database structure

3. JSON data structure

4. A list of associated metadata

5. Statistics about the dataset

5. Additional Appendices estimating the break-down of costs of the Dataset can be provided

– especially if the respondents are the owners and maintainers of relevant commercial datasets that can be purchased.

6. All documents are limited to 12-point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11-inch paper with 1-inch margins.

7. DO NOT submit samples of malicious binaries or source code with your response.

Submission Instructions to Respondents:

Responses to this RFI are due no later than 5:00 p.m., Eastern Time, October 10, 2022. All submissions must be electronically submitted to dni-iarpa-rfi-22-08@iarpa.gov as a PDF document. Inquiries to this RFI must be submitted to dni-iarpa-rfi-22-08@iarpa.gov. Do not send questions with proprietary content. No telephone inquiries will be accepted.

Disclaimers and Important Notes:

This is an RFI issued solely for information and planning purposes and does not constitute a solicitation or authority to enter negotiations for a contract. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information or to provide feedback to respondents with respect to any information submitted under this RFI. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract.

Respondents are solely responsible for all expenses associated with responding to this RFI.

IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner. The Government does not intend to award a contract based on this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. No proprietary and no classified concepts or information shall be included in the submittal. Should a respondent wish to submit classified / proprietary concepts or information, prior coordination must be made with the IARPA Chief of Security. Email the Primary Point of Contact with a request for coordination with the IARPA

Chief of Security. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate nondisclosure requirements.

Contracting Office Address:

Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity Washington, District of Columbia 20511 United States

Primary Point of Contact:

Dr. Kristopher Reese Program Manager Intelligence Advanced Research Projects Activity

REFERENCES:

[1] J. Gray, D. Sgandurra, and L. Cavallaro, “Identifying Authorship Style in Malicious Binaries: Techniques, Challenges, & Datasets,” Computing Research Repository, 2021 [Online] Available:

https://arxiv.org/abs/2101.06124v2

[2] S. Alrabaee, P. Shirani, M. Debbabi, and L. Wang, “On the Feasibility of Malware Authorship Attribution,” in Foundations and Practice of Security. FPS 2016., vol. 10128, F. Cuppens, L. Wang, N. Cuppens-Boulahia, N.

Tawbi, and J. Garcia-Alfaro, Eds. Springer, 2017, pp. 256–272.

[3] A. Caliskan, F. Yamaguchi, E. Dauber, R. Harang, K. Rieck, R. Greenstadt, and A. Narayanan, “When Coding Style Survives Compilation: De-anonymizing Pogrammers from Executable Binaries,” 2018 Network and Distributed System Security Symposium, San Diego, CA, 2018. doi: 10.14722/ndss.2018.23304

[4] N. Rosenblum, X. Zhu, and B. P. Miller. “Who wrote this code? identifying the authors of program binaries.”

Computer Security – ESORICS 2011, pages 172–189, 2011. Springer: Berlin Heidelberg. ISBN 978-3-642-23822-2.

[5] S. Alrabaee, P. Shirani, L. Wang, M. Debbabi, and A. Hanna. “On leveraging coding habits for effective binary authorship attribution.” Computer Security, pages 26–47, Cham, 2018. Springer International Publishing. ISBN 978- 3-319-99073-6.

[6] L. Simko, L. Zettlemoyer, and T. Kohno, “Recognizing and Imitating Programmer Style: Adversaries in Program Authorship Attribution.” in Proceedings on Privacy Enhancing Technologies, Jan. 2018, vol. 2018, no. 1,

pp. 127-144. doi: 10.1515/popets-2018-0007

[7] B. Alsulami, E. Dauber, R. Harang, S. Mancoridis, and R. Greenstadt, “Source Code Authorship Attribution Using Long Short-Term Memory Based Networks,” in Proceedings of the European Symposium on Research in Computer Security, 2017, pp. 65-82. doi: 10.1007/978-3-319-66402-6-6