CJF CRA QA 2024-09-12 1800 v2.pdf

Published 2024-09-13

ARMY RESEARCH LABORATORY

BROAD AGENCY ANNOUNCEMENT

Questions and Answers

FOR

Collective Judgment Formation Collaborative Research Alliance

Fiscal Year 2024

W911NF-24-S-0010

ISSUED BY:

U.S. Army Contracting Command Aberdeen Proving Ground

Research Triangle Park Division P. O. Box 12211

Research Triangle Park, NC 27709-2211

Q&A for Collective Judgment Formation CRA 2

Slides

Q01. Is there a recording available for the CJF CRA Proposers Day?

A01. There is no recording of the Proposers Day event; however, the slides and Q&A are posted to grants.gov and sam.gov

Q02. Will proposer day slides be available after this meeting?

A02. The slides are available at grants.gov and sam.gov

Q&A for Collective Judgment Formation CRA 3

Teaming

Q03. How many PIs/institutions do you envision to participate in a team?

A03. Each proposing team can have a maximum of 6 members. Smaller teams are acceptable and will need to have the right mix of disciplinary expertise to carry out the proposed work. (See BAA p. 24.) Teams are not required to be from multiple institutions and can all come from the same institution.

Q04. Regarding Section II.D, are UARCs allowed to be subcontractors on proposals?

Are there special criteria that UARCs must fulfill to qualify for participation?

A04. UARCs are allowed to compete except when it is prohibited by it contract.

Q05. The BAA states that TA3 proposers must have facilities capable of TS/SCI work.

Is it possible for this facility to be owned by a subcontractor? Or must the facility be with the Prime organization? Additionally, is it sufficient for the individuals within the proposing organization to have TS/SCI clearances along with facilities that are cleared at the TS level?

A05. No. The Prime will need to be capable of receiving a TS/SCI contract.

Q06. The BAA states that a computer scientist needs to lead TA2. Would a computational scientist (e.g., engineer, computational statistician) qualify?

A06. Yes. For TA2, computer scientist can be interpreted more broadly to include relevant/equivalent computational scientists. Like for TA1, Social Scientist comes from a set of social sciences such as Communication, Psychology, Sociology, or Political Science, to name a few.

Q07. Should an individual proposal team consist of research, academic, and industry organizations? Or is a subset sufficient – e.g., one university and one research lab? Is there preference for who is the prime and who are the subs?

A07. There is no requirement in the BAA to have a mixed (academic and industry) proposing team. It is up to the proposers to have the right mix of expertise to complete the proposed work.

Q08. Do proposers need to include an ARL scientist as part of their team (i.e., included in the 6-member limit)? If not, do proposers need to indicate in the proposal an ARL scientist or ARL scientists with whom we would directly collaborate?

A08. ARL scientists should NOT be part of any proposing team. Proposing teams are welcome to name any ARL scientists with whom they have a working relationship.

Q&A for Collective Judgment Formation CRA 4

Q09. Does the "6-member limit" mean 6 individual PIs or 6 institutions?

A09. Six (6) member limit is the number of PIs and Co-PIs on a proposing team. They may all come from the same institution.

Q10. Can there be “Senior Personnel” (e.g., research faculty not designated as PIs) not counted towards the 6 person PI limit?

A10. No.

Q11. Do you have a strong programmatic preference for performers or team members with an established physical presence close to those facilities?

A11. No. There is no physical distance assessment as part of the evaluation process.

See BAA p. 41 for proposal evaluation criteria.

Q12. Do you find it plausible that performers who are not physically proximal to those facilities can be effective performers in this program, even though their access to the program’s CUI will be limited?

A12. There is no physical distance assessment as part of the evaluation process. See

BAA p. 41 for proposal evaluation criteria.

Q13. We noted the eligibility on FFRDCs in the BAA, as well as the preferred award instrument of a cooperative agreement or contract. Should this application result in an award, would it be possible to fund a FFRDC through an Interagency Agreement?

A13. FFRDCs are allowed to apply except when it is prohibited by FFRDC sponsor agreement’s terms and conditions or sponsoring organization. Funding to FFRDC will be provided through Military Interdepartmental Purchase Request (MIPR) or the appropriate Interagency Agreement.

Q14. Will it be possible to get a list of meeting participants for teaming purposes (if they voluntarily share their information)?

A14. No, The government does not have a publicly publishable list of meeting attendees.

Q15. For TA3, is there also a multi-disciplinary team requirement? This is not explicitly stated in the BAA, but is mentioned as a requirement for TA1 and TA2

A15. The proposing team is required to have the right expertise to perform the proposed work. Given the tasks outlined under TA3, it will likely necessitate a multi-disciplinary team.

Q&A for Collective Judgment Formation CRA 5

Q16. Does the TA1 computer scientist need to be a PhD level researcher, or would a software engineer be an acceptable team member to support integration?

A16. The proposing team is required to have the right expertise to perform the proposed work. A software engineer counts as a computer scientist under the terms of the BAA. Experience and qualifications of all team members will be evaluated for contribution to the research proposed. See Q6

Q17. As a small business which already has relationships with ARL, how do you envision collaboration with academia? Is partnership with academia encouraged and/or a part of your evaluation criteria?

A17. Prior relationships with ARL are not required. See BAA p. 41 for proposal evaluation criteria.

Q18. Can a researcher be part of two separate teams for white paper submissions?

A18. Yes, the same person can be part of two separate teams for white paper submission under the same TA; however whole teams may NOT submit multiple white papers under the same TA. The Government reserves the right to make the determination if a team (in whole or parts thereof) submitted multiple white paper under the same TA.

Q19. The requirement for TA1, led by a social scientist, is not well defined. Does psychologist or public health social modeling count? Or is this requirement more aimed at higher-level social process disciplines such as sociology, anthropology and economics?

A19. All social scientists are eligible to be TA1 lead. Their expertise should be directly relevant to the work being proposed. See Q6

Q20. There have been references to a maximum Team Size of 6 per TA proposal. Is

“team” defined as PIs, Co-PIs, and Post-Docs, or would it also include graduate students?

A20. The BAA limits the number of senior personnel on the proposal to six (6) and should all be listed as PI and Co-PIs.

Q21. Could other senior personnel and/or consultants be included in the research effort for a topic area? If so, where in the white paper would you recommend describing these additional personnel and how they would contribute to the efforts?

A21. The BAA limits the number of senior personnel on the proposal to six (6). All senior personnel should be listed as PI and Co-PIs.

Q&A for Collective Judgment Formation CRA 6

Q22. Are there any restrictions on UARC scientists being lead, PI, and/or co-PI for

TA3?

A22. There are no restrictions in the BAA for limiting UARC scientists from being the PI or Co-PI an any of the TAs. See Q4

Q23. Do 'consultants' count as the 6 senior personnel?

A23. Consultants should not be included as part of the proposal. If their expertise is required for the proposed work, they should be included as a Co-PI.

Q24. Is it required that each team for a TA contain a university and a research lab? Can there be other configurations for teams, such as two research labs, with one prime and one subordinate?

A24. There are no requirements for the types or combinations of institutions that apply.

See Q7 and Q9

Q25. Should we discuss collaboration plans between the various organizations in the team that we are proposing, or collaboration plans between the three TA teams, or both?

A25. Proposers can discuss collaboration plans that are helpful for evaluators to understand the work that they are proposing. Only the criteria listed in the BAA will be used for evaluation purposes.

Q&A for Collective Judgment Formation CRA 7

Controlled Unclassified Information (CUI)

Q26. Relevant to section II.D.2., are there any circumstances that would permit a US person who is not a US citizen to work on the effort, for instance, having no access to the Government sites mentioned?

A26. PIs, Co-PIs, and postdocs must be US Citizens to apply for funding in this program. (See BAA p. 22)

Q27. Relevant to section II.D.2. Some universities have a policy that no graduate student can work on a CUI project, since that would limit the student’s opportunities to publish. In what way(s) are the BAA’s emphasis on fundamental advances and graduate student eligibility consistent with this being a CUI effort?

A27. The government does not control the policies of universities. The program is clearly indicated as CUI upfront. It will be up to individual proposing teams to decide if this funding opportunity is right for them.

Q28. Regarding Section II.D.2.iv, can you provide any guidance about what will be

CUI? For instance, perhaps only the data accessible through the Government is CUI? (The BAA states Access to CUI on information systems is limited to government owned and accredited information systems (i.e., computers) only at Adelphi Laboratory Center (ALC) and Aberdeen Proving Ground (APG), MD.)

Perhaps the testing & evaluation sessions supported by the TA3 performer will be CUI? Or perhaps all concepts, discussions, and lessons learned wherever held will be CUI? Can you provide any guidelines, ideally sooner than later?

A28. It is expected that there will be multiple aspects of the program that will be CUI such as some data and activities at ALC and APG. The program will follow all government rules managing Controlled Unclassified Information (CUI).

For additional information about CUI, please see: https://www.dodcui.mil/.

Q29. Given the cooperative agreements for TA1 and TA2 assuming a US Citizen PI, will the prime be able to carve out enclaves of exemptions for work that supports the effort: (1) For staff/faculty/postdocs that are US persons, but not US citizens (e.g. Green Card holders)? Does nation of citizenship impact this? (2) For broader non-CUI, without restrictions that could be performed by non-US people?

A29. PIs, Co-PIs, and postdocs for TA1, TA2, and TA3 must be US Citizens (see BAA

p. 22); the BAA does not have a restriction on graduate student citizenship.

Q30. I will be eligible to apply for US citizenship later this year. Can I be a PI on a proposal while my application for US citizenship is still pending?

A30. No. Only US Citizens can compete for funding under this BAA.

Q&A for Collective Judgment Formation CRA 8

Q31. The BAA indicates in Section 2.a.i that access to CUI in this program will be restricted to on-site access at Adelphi Laboratory Center and Aberdeen Proving Ground. What assumptions are you making regarding the required frequency with which CUI will need to be accessed?

A31. There is no predetermined frequency for CUI access/engagement.

Q32. The BAA states that eligible applicants are based in the United States or its

Territories. We are based in the UK but were sent the BAA so are wondering if there are exceptions to this, i.e. if we have previously received ARL funding or if we collaborate with a US partner who act as lead?

A32. All performers must be US citizens and be based in the US. (See BAA p. 22.)

Q33. When the Award information states the TA1 and TA2 awards will be made in the form of CUI cooperative agreements, can you clarify what that means? Does that mean it will include publication approval language, making the research subject to the export control regulations and therefore Export Controlled CUI?

Does it mean that CUI will be shared with the performers? If the latter, do you know what category of CUI the performers will likely receive?

A33. There will be CUI aspects to the program whereas specific access and dissemination controls protecting information resulting from the research are applicable, and the appropriate restrictions will be applied to those aspects of the program based on the types of data and/or engagements which may result in publication restrictions placed on resulting awards.

Q&A for Collective Judgment Formation CRA 9

Submissions

Q34. We're considering bidding on TA1 and TA2, could this be one proposal, or should they be separate?

A34. Each proposal may address only a single TA.

Q35. On page 8, section II.A.5, the BAA states, “Proposers may submit multiple proposals if they seek to address multiple TA, but not multiple proposals to the same TA.” In a large university, there may be distinct teams with differing approaches that wish to bid on a specific TA. Is that disqualified?

A35. Not disqualified. Universities may submit white papers from multiple teams to the same TA. See Q18

Q36. Relevant to section II.A.5, is there any prohibition against an award to TA1 and one to TA2 involving the same individual?

A36. The same individual may be on multiple teams; the same individual may not serve as the Lead PI on TA1 and TA2. See Q18 and Q20

Q37. Do you have any advice for teams that are looking to submit multiple proposals for different topic areas (in our case two teams interested in TA1 and TA2 respectively). Can we refer to the other team's proposal in our own proposal? Will you expect (or recommend) the two teams to have some overlap or is it OK to have completely disjoint teams?

A37. All proposals must be able to stand alone and will be independently evaluated..

Q38. Are bibliographic references included in the 7-page limit?

A38. Section E.2 of the BAA address white paper format and content. There is no limit on the number of bibliographical reference page(s).

Q39. Can tables include font that is smaller than 12 point Arial?

A39. Section E.1 of the BAA address white paper format, the font description applies to tables and figures.

Q&A for Collective Judgment Formation CRA 10

Programmatic Clarifications

Q40. Will there be a cooperative agreement such that TA3 PIs/co-PIs/senior personnel can also be on publications and provide hypotheses?

A40. TA3 will be a FAR/DFARS based contract. Like TA1 and TA2 teams, TA3 performers will be able to collaborate and publish with any/all members of CJF (See BAA pp. 67-68 for the Associate Performer Agreement). There will be CUI aspects to the program where specific access and dissemination controls protecting information resulting from the research are applicable, and the appropriate restrictions will be applied to those aspects of the program based on the types of data and/or engagements which may result in publication restrictions placed on resulting awards.

Q41. Should all the participant costs be covered in the budget for TA3, or split with TA2 and TA1?

A41. TA1 and TA2 performers should not expect that all experimentation costs are borne by TA3; rather the intention is for TA3 to test the major research hypotheses. All TAs may choose to budget for participant costs to the extent relevant to the work proposed.

Q42. Since the TA3 is only for one year with possible outyears, what happens if the

TA3 group is not renewed?

A42. While there is a possibility that any outyear is not renewed, the intention for the program is a 5-year effort. If the CJF Program Managers determine that the TA3 outyear will not be renewed, that will be communicated in accordance with the terms of the contact.

Q43. Is TA3 responsible for all of the participant recruiting and selection as well?

A43. TA3 will be the lead driver for CJF-wide experiments. All performers must take responsibility for the success of the program and contribute as appropriate (See BAA pp. 67-68 for the Associate Performer Agreement).

Q&A for Collective Judgment Formation CRA 11

Q44. Regarding Section Ii.B.3, in the first year, before TA3 has provided a testbed and before the first official experiment, are TA1 and TA2 performers to propose and create their own scenarios, data, and internal testing & evaluation sessions for setting and tracking progress? If so, will that require HSR review, or does the HSR approval cover all internal performer research?

A44. During the first year, all three TAs and ARL Scientists will work towards defining scenarios, data, and types of human subjects research needed for the program.

If a TA1 or TA2 team wants to run their own small experiments during that time, they may do so, so long as they get IRB and HRPO approvals. For CJF, the scope of the IRB and HRPO approvals will be limited to the defined protocols that are collaboratively designed by the performers and ARL scientists.

Q45. Is it accurate that all work in the first three years is basic research (6.1) and the proposed effort need not involve CUI access until the two option years?

A45. No.

Q46. Could you clarify who is providing the storage and compute cycles (CPU and

GPU)? Is it TA3? Is it ARL’s facilities? Can the TA1 and TA2 performer bid costs for development and component testing of algorithms prior to each of the four test and evaluation periods?

A46. The government will provide storage for CUI data. Performers are responsible for other storage and computing resources for their proposed work. Costs of acquiring computing hardware (e.g., large equipment purchases such as CPUs and GPUs for ML) will not be approved.

Q47. Regarding II.B.2, the diverse types of models and the generative AI capabilities listed on page 13 can require significant compute resources, particularly GPUs for generative AI. Are there restrictions on the size of models that we propose to use?

A47. Proposals should be realistic in their team's capabilities and corresponding statements of work. Understanding that it will be a distributed program, proposed models should be within scope of execution.

Q48. Does DoD have specific applications and data sets in mind?

A48. The government does not have specific applications or data sets in mind. The research developed under this program will drive those.

Q&A for Collective Judgment Formation CRA 12

Q49. Do the scenarios we study need to directly include actual intelligence data or are proxy collaborative tasks that model the flow of information through a group acceptable?

A49. This program will not provide actual intelligence data. Proxy collaborative tasks would be appropriate.

Q50. Who determines the scope of the intelligence domain and domain examples to motivate the research?

A50. Proposers should identify their own proxies to the intelligence domains most relevant to their interests and expertise. The ARL scientists will help shape the program direction after all performers are selected.

Q51. How does ARL anticipate coordinating focus between TAs, for instance, if a TA2 performer is focused on LLMs but TA1 is interested in multimodal data that requires a computer vision component?

A51. The collaborative process between all performers and Army scientists will be necessary to create fruitful synergy around the various focal points over the course of the 5-year program.

Q52. Is there any preference for or against any particular intel work/workflows (e.g., All-source, GEOINT, SIGINT, OSINT)? Or use cases (e.g., cyber, targeting, non-proliferation)?

A52. No. The program will evolve based on the performers and Army scientists. That said, it would not surprise us if OSINT was the dominant focus for much/all of the program given the focus on generative AI capabilities.

Q53. What’s the relationship between the Team Lead and the embedded ARL Bench

Scientist?

A53. All PIs and Co-PIs will be collaborators with ARL Scientists. See BAA pp. 67-68 for the Associate Performer Agreement.

Q54. Is this funding opportunity considered to be for Fundamental Research as defined in 15 CFR 734.8 (EAR) and 22 CFR 120.34 (ITAR)?

A54. There will be CUI aspects to the program where specific access and dissemination controls protecting information resulting from the research are applicable, and the appropriate restrictions will be applied to those aspects of the program based on the types of data and/or engagements which may result in publication restrictions placed on resulting awards.

Q&A for Collective Judgment Formation CRA 13

Q55. Will the Cooperative Agreements for TA1 and TA2 be issued for the performance of Fundamental Research (as defined by National Security Decision Directive (NSDD) 189, National Policy on the Transfer of Scientific, Technical, and Engineering Information?

A55. See Q54

Q&A for Collective Judgment Formation CRA 14

Technical Clarifications

Q56. What kinds of participants should be in the research studies? E.g., crowdsourced workers, intelligence analysts, students, some combination?

A56. These choices are at the discretion of the proposing team.

Q57. Are proposals that respond to Topic Area 2 (computational modeling) expected to carry out any human subjects research, or will human subjects research activities exclusively handled by the other two teams?

A57. These choices are at the discretion of the proposing team.

Q58. Can you define "Wizard of Oz" solution?

A58. The BAA (p. 11) states that no Wizard of Oz solutions will be accepted. A Wizard of Oz solution is a human experiment where a participant (unknown to the research participant) thinks that they are working with an AI/computer when in reality there is an unseen human (experimental confederate) responding to the participant.

Q59. For TA1, must proposals address all 3 focus areas? Would a proposal focusing on information highlights but not shift work be in scope or out of scope?

A59. If possible, including all three focal areas outlined in TA1 could create a stronger proposal. However, if the team can only focus on one or two of them, they will still be considered responsive to the BAA.

Q60. The formal and empirical sides of TA2 *could* entail very different kinds of work.

Presumably there is a premium on coherent TA2 that marries the formal and empirical aspects nicely? And can we also assume that the formal aspect is well-informed by the TA1 social science theories? My impression is that the formal aspects of TA2 seems to be a bridge between TA1 and TA2?

A60. The formal and empirical sides of TA2 are intended to be complementary, contributing to insights into the same basic science question(s). The formal aspect is further intended to be informed by TA1 social science theories, with the result being integrative science, with the formal work indeed bridging between the social sciences and computational sciences.

Q61. Would AI System Vulnerabilities be of interest for TA2?

A61. System vulnerabilities may be considered under the Game Theory section of TA2;

however, this is not a "hardening" nor "penetration testing" program, rather a basic science program.

Q&A for Collective Judgment Formation CRA 15

Q62. Will TA1 (or TA2 for that matter) be able to conduct any internal HSR studies to test hypotheses / develop models, or will TA1 and TA2 be entirely reliant on TA3 for HSR?

A62. TA1 and TA2 should conduct their own HSR studies if that is what is needed to contribute to the larger program goals.

Q63. In TA3 Section, Page 14, the BAA states, “Example measures to quantify during experiments include but are not limited to agent usefulness, trust, adaptation and resilience time, coordination effort required after perturbation, team effectiveness, development of transactive memory system, and development of human and agent theory of mind. Strong proposals will discuss additional relevant metrics and measures to be quantified during evaluation exercises.” Is TA3 expected to integrate new or existing algorithms to measure/quantify these constructs for TA1/TA2 or are these suggestions meant for TA1/TA2 to consider as a part of their work?

A63. Under the Associate Performer Agreement (BAA pp. 67-68), all parties will work together to do this work. That said, TA1 and TA2 must offer definitions and measurement methodologies for the constructs they put forward.

Q&A for Collective Judgment Formation CRA 16

Other

Q64. Specific ties may already exist with a PEO or an official Program of Record. What do you envision the relationship with PEOs or other Army/DoD Programs will be to CJF? Is there thought in using/providing Government-Owned or Government Purpose Rights technical documentation to provide greater opportunity to deliver capabilities with high-transition probability?

A64. Proposing teams are welcomed to share their connections to larger Army / DoD programs that they may already have, and the government welcomes transition suggestions as/when appropriate.

Q65. What are the restrictions and/or processes on publishing for this effort?

A65. There will be CUI aspects to the program where specific access and dissemination controls protecting information resulting from the research are applicable, and the appropriate restrictions will be applied to those aspects of the program based on the types of data and/or engagements which may result in publication restrictions placed on resulting awards.

Q66. The US Presidential election cycles offer historical and current opportunities to study the TA1 processes outlined or for TA2 modeling. Are there any issues with focusing on US elections?

A66. No projects focused on US elections at any level will be considered.

Q67. How many subawards are you expecting each TA to include?

A67. There are no specific expectations. Subawards are not necessary if you have all the expertise at your institution.