EndGen_RFI_C.pdf

UNCLASSIFIED

Request for Information (RFI) on Generative AI/ML Models for Math, Algorithms, and Signal Processing (MASP) Input/Output Modalities

Agency: Office of the Director of National Intelligence Office: Intelligence Advanced Research Projects Activity

RFI Number: IARPA-RFI-24-03

Synopsis:

The Intelligence Advanced Research Projects Activity (IARPA) seeks information regarding innovative approaches to generative artificial intelligence (AI) or machine learning (ML) models to achieve a revolutionary leap in applications of science and engineering by generating smaller evolutionary products of math, algorithms, or signal processing (MASP). While significant progress has been made for generators of text, image, and audio (TIA) modalities, AI/ML generators for more complex sciences, including the MASP modalities, have not received the same attention. It is important to note this RFI is not for AI/ML solutions that perform such calculations of math, algorithms, or signal processing; rather, this RFI is looking for AI/ML solutions that create math, algorithms, or signal processing products themselves at the output of the generator.

The envisioned models and systems could enable exponential advances in scientific and engineering fields as the AI/ML generates many small evolutionary products, unfettered from delays in human creativity, to quickly accumulate into generational discoveries.

Background:

Modern uses of AI/ML are commonly determined by two segments of their framework:

input/output modalities and the machine goal. Single, or multiple, input modalities define what domain the AI/ML is working on, with the most recognizable domains being: text, image, or audio (TIA); single, or multiple, AI/ML outputs can be in that same, or perhaps different, modality. The AI/ML machine goal itself is often further distinguished by two types: a discriminative AI attempts to find boundaries within some input modality while a generative AI attempts to understand the entire distribution of that modality. And while discriminative AIs may perform better at class labeling (1), generative AIs have a more robust knowledge of the entire input distribution and can create original samples from the input mode. Generative AIs, with uncommon modalities, that can create original samples in those modalities, are the focus of this RFI.

For example, text-to-text generative AI, such as OpenAI’s ChatGPT (2), has taken on a nearly ubiquitous spot in the population’s lexicon with its ability to help human users generate useful snippets of text for their own purposes. Text-to-image generative AI, such as DALL-E (3) was created recently to generate unique images, using a similar transformer model as ChatGPT, but with output data images rather than natural language text. Image-to-text generation (4), often called image captioning, is used to produce natural language descriptions of input images. With generative audio, the most common use is realistic text-to-speech (5) synthesis, or even audio-to-text transcription, but audio-to-audio frameworks are also garnering attention. Voicebox (6) is a generative AI framework that creates audio infill based on large audio models and multimodal text cues.

There is already much work in moving beyond the traditional modalities seen in existing TIA generators. For example, using GPT-4 it is possible to produce a text-to-code generator (7) that can respond to strict coding questions or even execute passable code (8) itself. Generators have also been used in the biological sciences, where one might think of biological outputs as a different modality, such as in (9) where generative AI was used to model novel protein structures. Decision-making is another form of output modality where generative AI is used for such purposes as business model innovation (10). Closer to this current RFI, while the so-called FunSearch AI (11) solved a niche combinatorics problem, the results certainly suggest that large models can be used to make mathematical discoveries. Or, in a different scientific field, (12) uses a physics inspired neural network to solve optimization problems better than other traditional algorithms because mathematical knowledge is trained directly into the machine. Finally, frameworks for in situ uses of AI/ML in communications or signal processing, such as Deepmod (13), are an interesting use of AI/ML in MASP systems; however, Deepmod directly implements MASP, rather than creating novel MASP output products, and therefore is not applicable to this current RFI.

Scope:

Traditional generative AI is certainly a robust field of study. Researchers will be busy for decades with the numerous combinations of single- or multi-modal input-to-output TIA modes available.

This robustness translates to broad usage of generative AI. These machines can be anything from an AI teaching assistant for students (14) to an AI that wins art competitions (15). And while using generative AI to help humans write readable reports, understand complex topics, or create unique art concepts are interesting uses of modern technology, this RFI is asking a different question:

How can generative AI be used to directly advance science and engineering, even in marginal ways? What input and output modalities, coupled with novel AI/ML frameworks, can be used to create evolutionary advances in science that, so far, have generally come from humans? Consider the framework needed, the datasets to train with, and the prompts (16) required to generate some of the following anecdotes:

• A machine is trained on text inputs exclusively from knowledge in mathematics and philosophy available prior to 600 BC. The machine is prompted and then generates an output in the math modality that describes a relationship between the two sides and hypotenuse of a right triangle (17) (but is not itself a triangle solver).

• A machine is trained on symbolic or graph inputs and tasked with placing a list of integer numbers in ascending order in an efficient manner. The machine is prompted and generates a sorting algorithm that is human useable (but does not perform the sorting).

• A machine is trained on the functional relationship between signals and linear time-invariant systems. The machine is prompted and generates a digital filter that maximizes output signal-to-noise ratio (but does not perform the filtering operation).

TIA modalities with generative AI have seen great application so far and can certainly help scientists and engineers perform better in their fields; however, TIA modalities may not have the robustness necessary to perform science itself. A human engineer that designs a novel algorithm may use generative TIA AI to help create the abstract for a journal paper or use the AI to better understand a complex mathematical topic; however, this RFI is interested in a generative MASP AI that helps discover the novelty itself. Unfortunately, the use of TIA modalities for this purpose may be hindered by their ability to capture the meaning of scientific or engineering topics to make these discoveries. Furthermore, while TIA AI/ML suffers from opinion and bias, a MASP AI/ML must require fact for all inputs and outputs. Different modalities other than TIA, such as: math, graphs, signals, or symbolic inputs/outputs may be necessary to create generative MASP output products.

The objective of this RFI is to receive input from respondents regarding methods of AI/ML frameworks that will demonstrate that evolutionary advances in MASP are possible, in response to human prompts, and how to measure the performance of such frameworks.

Requests:

This RFI seeks understanding of innovative systems consisting of MASP input and output modalities for generative AI/ML frameworks. These systems, when fully realized, should have the opportunity to create marginal advances to MASP problems by generating novel MASP products.

These marginal improvements, fed back into future machine iterations, in a positive feedback fashion, may provide generational improvements in some science and engineering fields.

Output Modalities of Interest

1. Math: Generative math is created by an AI/ML framework that is trained on, and consumes, TIA, MASP, or other symbolic inputs to generate outputs in the math modality: theorems, lemmas, equations, etc. For this RFI, these products may be any preexisting human-discovered math knowledge that shows the generative AI framework would have the ability to create such products if trained on appropriate inputs. Generative math is not a “math solver” itself; rather, generative math will take appropriate inputs to create something novel in an output math modality that is human readable.

2. Algorithms: Generative algorithms are created from an AI/ML framework that is trained on, and consumes, TIA, MASP, or other symbolic inputs to generate an algorithm (18), or sequence of: steps, variables, feedback loops, updates, etc. to perform some action, based on a given objective function. A generative algorithm AI/ML does not perform steps of an algorithm itself; rather, it learns from the symbolic input modality how to arrange the steps of the algorithm, the truth behind each step, and what the objective optimization is. The output mode can be defined indeterminately but should be translatable to human understanding.

3. Signal Processing: Generative signal processing is created from an AI/ML framework that is trained on, and consumes, TIA, MASP, or other symbolic inputs to generate signal processing products useful to an engineering discipline: filtering, modulations, transforms, waveforms, etc. For example, generative signal processing may create a known error correction coding scheme, after being trained on topics such as bit manipulation and additive noise. The generative signal processor should output a signal processing function the human can use and not perform those operations itself.

In general, any response to this RFI is of interest to IARPA that can show usable MASP products, generated by AI/ML, even if those products have already been discovered by humans. The purpose of this RFI is to understand the feasibility of MASP discoveries and not necessarily to create new discoveries immediately. In short, where current generative AI platforms can certainly help scientists and engineers perform their occupational duties, this RFI is looking for generative AI frameworks that help perform science and engineering itself. Keep in mind, responses that perform MASP operations, but do not generate MASP products, are not appropriate for this RFI.

Specific Questions for Respondents

1. How will the respondent generate, or acquire, training datasets geared toward the MASP input and output modalities in order to generate the desired products? Nontraditional, non-text datasets are of particular interest to this RFI.

2. What types of AI/ML frameworks can be used to move beyond well understood TIA outputs and generate MASP outputs instead? How are MASP outputs human readable?

3. How will prompting work for MASP modalities? How would the respondent’s framework avoid bias/opinion to maintain factual outputs given human generated prompts?

4. What metrics can be used to gauge the effectiveness of an AI/ML framework when used with MASP modalities? How can these metrics be measured effectively between two significantly different approaches?

And/Or

5. What anecdotal experiments can the respondent perform to show that generative AI for MASP can produce marginal improvements given the state of the machine?

Preparation Instructions to Respondents:

IARPA requests that respondents submit responses to the above questions for use by the Government in formulating a potential R&D program. IARPA requests that submittals briefly and clearly describe the potential approach or concept, the training datasets to use in the approach, describe how the approach may be quantified against other approaches, and comment on generative responses to the approach prompts. If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) estimate regarding what such approaches might require in terms of funding and other resources for one to four years to execute the respondent’s vision for a generative AI/ML framework the operates in the math, algorithm, and/or signal processing modalities. This announcement contains all the information required to submit a response. No additional forms, kits, or other materials are needed.

IARPA welcomes responses from all capable and qualified sources from within and outside of the U.S.

Reponses must meet the following formatting requirements:

1. A one-page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone numbers, and email addresses of all co-authors, and clearly indicating its association with IARPA-RFI-24-03;

2. A substantive, focused, one-half page executive summary;

3. A description of the technical challenges and suggested approaches to create an AI/ML framework that generates novel products of math, algorithms, or signal processing.

Responses should answer at least questions 1-4 and/or question 5 above and be limited to 3 pages (excluding cover pages, executive summary, and citations) in minimum 12-point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11-inch paper, with 1-inch margins. Please provide diagrams, figures, charts to help illustrate concepts when possible; and

4. A list of citations (any significant claims or reports of success must be accompanied by citations). There is no page limit for citations.

Submission Instructions to Respondents:

Responses to this RFI are due no later than 5:00 p.m., Eastern Time, March 29,2024. All submissions must be electronically submitted to dni-iarpa-rfi-24-03@iarpa.gov as a PDF document. Inquiries to this RFI must be submitted to dni-iarpa-rfi-24-03@iarpa.gov. Do not send questions with proprietary content. No telephone inquiries will be accepted. In response to submissions to this RFI, IARPA may organize an invitation-only workshop for the purpose of reviewing and discussing relevant current and future research. Should a workshop occur, participants may be asked to make formal presentations based on their RFI submissions or associated research. Written submissions and information discussed at a workshop may assist in the formulation of future U.S. Government research.

Disclaimers and Important Notes:

This is an RFI issued solely for information and planning purposes and does not constitute a solicitation or authority to enter into negotiations for a contract. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information or to provide feedback to respondents with respect to any information submitted under this RFI. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner.

The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. No proprietary and no classified concepts or information shall be included in the submittal. However, should a respondent wish to submit classified concepts or information, prior coordination must be made with the IARPA Program Security Officer. Email the Primary Point of Contact with a request for coordination with the IARPA Program Security Officer. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.

Contracting Office Address:

Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity Washington, District of Columbia 20511 United States

Primary Point of Contact:

Dr. Adam L. Anderson, Program Manager dni-iarpa-rfi-24-03@iarpa.gov Intelligence Advanced Research Projects Activity

Works Cited

1. Generative or discriminative? getting the best of both worlds. Bernardo, J M, et al. 3, 2007, Bayesian statistics, Vol. 8, pp. 3-24.

2. ChatGPT. OpenAI. 2023, https://chat.openai.com/chat.

3. A very preliminary analysis of DALL-E 2. Marcus, Gary, Davis, Ernest and Aaronson, Scott. 2022, arXiv preprint arXiv:2204.13807.

4. Deep learning for image-to-text generation: A technical overview. He, Xiaodong and Deng, Li. 6, s.l. :

IEEE, 2017, IEEE Signal Processing Magazine, Vol. 34, pp. 109-116. 1053-5888.

5. Text-to-speech synthesis. Shiga, Yoshinori, et al. s.l. : Springer, 2020, Speech-to-Speech Translation,

pp. 39-52. 9811505942.

6. Voicebox: Text-guided multilingual universal speech generation at scale. Le, Matthew, et al. 2023, arXiv preprint arXiv:2306.15687.

7. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. Phung, Tung, et al. s.l. : ACM, 9 2023, Conference on International Computing Education Research, pp. 41-42.

8. Generative AI for software practitioners. Ebert, Christof and Louridas, Panos. 4, s.l. : IEEE, 2023, IEEE Software, Vol. 40, pp. 30-38. 0740-7459.

9. SE (3) diffusion model with application to protein backbone generation. Yim, Jason, et al. 2023, arXiv preprint arXiv:2302.02277.

10. The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective. Kanbach, Dominik K, et al. s.l. : Springer, 2023, Review of Managerial Science, pp. 1-32.

1863-6683.

11. Mathematical discoveries from program search with large language models. Romera-Paredes, Bernardino, et al. s.l. : Nature Publishing Group UK London, 2023, Nature, pp. 1-3. 0028-0836.

12. Solving real-world optimization tasks using physics-informed neural computing. Seo, J. 202, s.l. : Sci Rep, 2024, Vol. 14.

13. Deep modulation (Deepmod): A self-taught PHY layer for resilient digital communications. Anderson, Adam, et al. 2019, arXiv preprint arXiv:1908.11218.

14. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Baidoo-Anu, David and Ansah, Leticia Owusu. 1, s.l. : İzmir Academy Association, 2023, Journal of AI, Vol. 7, pp. 52-62.

15. "Theatre D'opera Spatial" — Wikipedia, The Free Encyclopedia. contributors, Wikipedia. 2023.

16. A Brief History of Prompt: Leveraging Language Models.(Through Advanced Prompting). Muktadir, G. M. s.l. : arXiv e-prints, 2023, Vols. arXiv-2310.

17. Maor, Eli. The Pythagorean theorem: a 4,000-year history. s.l. : Princeton University Press, 2019.

0691196885.

18. What is Algorithm | Introduction to Algorithms. 2023, https://www.geeksforgeeks.org/introduction-to-algorithms/.