CJF Proposers Day 2024-09-04 1100.pdf

UNCLASSIFIED

U.S. ARMY COMBAT CAPABILITIES

DEVELOPMENT COMMAND

ARMY RESEARCH LABORATORY

04 September 2024 1

Controlled by:

Controlled by:

CUI Category:

Distribution Statement:

POCs:

N/A

N/A

N/A

Distro A: Approved for public release; distribution is unlimited

Army Research Office Army Contracting Command

Collective Judgment Formation CRA Proposers Day

04 SEPT 2024

Contact:

cjf-cra-baa@army.mil

AGENDA

ARMY CONTRACTING COMMAND

–General Comments

–Key Dates

–White Paper/Proposal Process

–Award Information

COLLECTIVE JUDGMENT FORMATION

–Program Intent

–TA1: Social Science Theories

–TA2: Computational Models

–TA3: Testing, Evaluation, and Transitions

–Budget

–Associate Performer Agreement

QUESTIONS AND ANSWERS

–Q&A for pre-submitted questions

–Q&A for today’s questions

04 September 2024 2

ARMY CONTRACTING COMMAND

04 September 2024 3

U.S. ARMY CONTRACTING COMMAND

ABERDEEN PROVING GROUND

RESEARCH TRIANGLE PARK DIVISION (ACC-APG-RTP)

BAA: W911NF-24-S-0010

▪ Posted 14 August 2024

▪ Application Inquiries and Questions on BAA (Deadline): 10 September 2024

▪ Questions and Responses Posting: 13 September 2024

▪ Questions and Responses will be posted to Grants.gov and SAM.gov

▪ TA1 and TA2 proposers are required to submit white papers

▪ TA3 is encourage but not required to submit white paper

▪ White Papers Due: 25 September 2024 by1500 Eastern Standard Time (EST)

▪ White Papers Feedback: 18 October 2024

▪ Invitation to submit proposals for TA1 and TA2 White Papers: 18 October 2024

04 September 2024 4

U.S. ARMY CONTRACTING COMMAND

ABERDEEN PROVING GROUND

RESEARCH TRIANGLE PARK DIVISION (ACC-APG-RTP)

BAA: W911NF-24-S-0010

▪ Deadline for Questions on Proposals: 30 October 2024

▪ Proposals Due: 21 November 2024 1500 Eastern Standard Time (EST)

▪ All proposals for Assistance Instruments (TA1/TA2) and Contracts (TA3) must be submitted electronically through Grants.gov using Workspace.

▪ Notification to Recipients: estimated 19 December 2024

▪ Estimated Award Date: February 2025

04 September 2024 5

U.S. ARMY CONTRACTING COMMAND

ABERDEEN PROVING GROUND

RESEARCH TRIANGLE PARK DIVISION (ACC-APG-RTP)

BAA: W911NF-24-S-0010

▪ All white papers must be emailed directly to the following email address:

cjf-cra-baa@army.mil

Email subject line: “White paper Submission: CJF CRA,” the BAA number W911NF- 24-S-0010, and the Technical Area

Example: White paper Submission: CJF CRA, BAA W911NF-24-S-0010, TA1

04 September 2024 6

U.S. ARMY CONTRACTING COMMAND

ABERDEEN PROVING GROUND

RESEARCH TRIANGLE PARK DIVISION (ACC-APG-RTP)

BAA: W911NF-24-S-0010

▪ Number of awards:

▪ TA1- One (1) Cooperative agreement (36 months Base, 24 months Option)

▪ TA2- One (1) Cooperative agreement (36 months Base, 24 months Option)

▪ TA3- One (1) Contract (12 months Base, four (4) 12 months Options)

▪ Optional renewal periods are subject to availability of funds based on the annual budget appropriations.

▪ The content of the Broad Agency Announcement posted to grants.gov/SAM.gov supersedes any information provided during this Proposer Day, to include any answers provided to questions during today’s session.

▪ The Government will post a Question and Answer document to the grants.gov/SAM.gov BAA Announcement following the deadline for question submissions. The Question and Answer document posted to grants.gov/SAM.gov WILL constitute an Amendment to the FOA.

04 September 2024 7

ARMY RESEARCH LABORATORY

04 September 2024 8

PROPOSERS DAY

Today’s meeting will be an UNCLASSIFED Discussion

▪ BAA is available at:

https://grants.gov/search-results-detail/356024

▪ CJF BAA is the authority …any contradictions between what is said today and the BAA, the BAA wins

04 September 2024 9

COLLECTIVE JUDGMENT FORMATION

What:

Basic research to create and expand scientific understanding human-agent interactions in AI/ML-supported team decision making for intelligence activities

Why:

The DoD and IC rely heavily on deriving knowledge for effective decision making; the challenges for processing high volumes of information are becoming exponentially more difficult with the expanding capabilities of generative artificial intelligence technologies, yet the need for high confidence assessments remains steadfast

How:

Multiple disciplinary perspectives working in concert to discover novel knowledge on

a) humans working with each other and with novel forms of software agents

b) teams of intelligence analysts responding to novel methods that create verbose structured and unstructured content

04 September 2024 10

COLLECTIVE JUDGMENT FORMATION

04 September 2024 11

Who:

ARL scientists to partner with multi-disciplinary performers to advance the scientific state-of-the-art in human-agent systems for intelligence activities https://arl.devcom.army.mil/collaborate-with-us/opportunity/collaborative-research-alliance/

COLLECTIVE JUDGMENT FORMATION

Program Structure

▪ To address the human-agent systems for intelligence activities (HAS-IA) problem space, the CJF program will consist of three technical areas (TAs):

–TA1: Social Science Theories for Human-Agent Systems for Intelligence Activities

–Cooperative Agreement; CUI

–TA2: Computational Models for Human-Agent Systems for Intelligence Activities

–Cooperative Agreement; CUI

–TA3: Testing, Evaluation, and Transitions for Human-Agent Systems for Intelligence Activities

–Contract; CUI; Cleared Facility; TS/SCI eligible

04 September 2024 12

COLLECTIVE JUDGMENT FORMATION

04 September 2024 13 https://www.dodcui.mil/

TA1: SOCIAL SCIENCE THEORIES FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

▪ Grounded in social science theories Must include at least one of:

1. Transactive Memory Theory

2. Theory of Mind

▪ Focus on:

1. Shift handover coordination

2. Information consolidation and synthesis

3. Critical information highlights to accommodate dynamic team environments

▪ These theories provide a theoretical lens in which to understand collectives through shared mental models

▪ Extend these theories to incorporate expanded teaming capabilities arising with

1. Generative artificial intelligence

2. Drive toward

Artificial Social Intelligence (https://artificialsocialintelligence.org)

04 September 2024 14

TA1 Primary Objective:

Formulate and test novel hypotheses about collective judgment formation within human-agent systems engaged in information processing for intelligence activities

TA1: SOCIAL SCIENCE THEORIES FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

Select at least two:

1) Cognitive biases and their impacts on collective judgment formation

• Individual cognitive biases

• Human-teaming biases

• Emergent human-agent teaming biases

• Neurodiversity with respect to information processing strengths

2) Exogenous pressures and obstacles on collective judgment formation

• Time constraints

• Turnover

• Dynamic global events

3) Impact of information source types on collective judgment formation

• For example: free-form text, instructions, tables, maps, diagrams, pictures, audio, and video

4) Narratives: building, following, detecting changes

• Story arcs

• Patterns of life

• Building and tracking knowledge threads

• Counter-narrative information integration

04 September 2024 15

TA1: SOCIAL SCIENCE THEORIES FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

Select at least two:

5) Impact of discovering violations to data integrity & provenance on belief systems

6) Misinformation/deception within corpus of data

• Identification

• Negation

• Management

7) Behavioral indicators of collective judgment

• Individual and collective predictors of values, beliefs, and opinions

• Indicators for the dynamics of belief formation

• Predictors of successful coordination and information consolidation

8) Uncertainty management

9) Game theory

• Establishing the rules and predicting value/utility for information

10) Measurable team performance

• Enhancements from humans teaming with agents

• Detriments from humans teaming with agents

• Agent support for distributed analyst teams

04 September 2024 16

TA1: SOCIAL SCIENCE THEORIES FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

Guidance for TA1 proposals:

▪ Describe the state-of-the-art and how your team will advance it

▪ Describe how you will minimize the ambiguity of their claims

▪ Describe how your constructs will be measured

▪ Describe methods for verifying the robustness of analyses

▪ Wizard of Oz solutions will NOT be accepted for consideration

▪ TA1 performer will work closely with TA2 performer to inform design of agent architectures

▪ TA1 performer will work with the TA3 performer to inform the design of the testbed for agent interaction and for hypothesis testing

▪ Goal: humans can be tested as a part of actual human-agent teams within this program

04 September 2024 17

TA2: COMPUTATIONAL MODELS FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

04 September 2024 18

Primary Objective 1:

Create formal models of collective judgment formation that ground the social science models of TA1 in appropriate logical, mathematical, and computational terms

Primary Objective 2:

Design intelligent systems that can be used to demonstrably improve collective judgment formation

The systems developed will support human analysts doing knowledge work, unobtrusively learning who knows whom, who knows what, who needs what information, and when. Improvements are expected to apply to the judgments produced (e.g., with greater accuracy and speed) as well as to the functioning of the team (e.g., alternative organizational structures or communication pathways, robustness in team substitutions).

TA2: COMPUTATIONAL MODELS FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

04 September 2024 19

Models of rationality

Models of inference (e.g., probabilistic networks, theorem proving, neural models)

Big data and data-driven decision making

Centralized and distributed decision making

Generative artificial intelligence, including large language models

Theory of Mind

“Genres” of knowledge representation and reasoning (e.g., causality, narrative, norms)

Computational game theory

Models of human cognition and social cognition

Models of explanation, instruction, advice taking, i.e. the communication of judgments

Potential research areas (though other proposed areas will be seriously considered), of which two or more should focus on:

TA2: COMPUTATIONAL MODELS FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

Example:

Analysts modeled as a dynamic hierarchical network that represents team structure, with information propagating from one node to another until some conclusion has been reached. Points of interest:

• Quality of data (e.g., accuracy, precision, uncertainty, completeness, timeliness) and effects on decision making

• Patterns in network structure, as a social system, that make agreement on a correct interpretation easier or harder to develop

• Mechanisms (in a game theoretic sense) that create incentives for more or less collaboration and information sharing, as appropriate to context

• Models of how information is processed by individual analysts (e.g., how disparate uncertain evidence is combined) and implications for how evolving context contributes to understanding at a more abstract organizational level

04 September 2024 20

TA2: COMPUTATIONAL MODELS FOR HUMAN-AGENT

SYSTEMS FOR INTELLIGENCE ACTIVITIES

(COOPERATIVE AGREEMENT; CUI)

Guidance for TA2 Proposals:

• TA2 proposals should describe the state-of-the-art and how it will be advanced:

models, algorithms, evaluation metrics, and evaluation procedures

• TA2 research should lead to models of collective judgment formation that have explanatory value

• TA2 is a bridge between TA1 and TA3.

• Ideally, TA2 models aligned and validated alongside TA1 models

• Rigorous analysis, theoretical and empirical evaluation

• Collaboration with TA3 to inform the testbed design for agent interaction

04 September 2024 21

TA3: TESTING, EVALUATION, AND TRANSITIONS FOR HUMAN-

AGENT SYSTEMS FOR INTELLIGENCE ACTIVITIES

(CONTRACT; CUI; CLEARED FACILITY; TS/SCI ELIGIBLE)

▪ Testbed: The TA3 performer will leverage an existing testbed through adaptation, extension, and implementation for a scalable testbed environment

▪ Experiments: In collaboration with the TA1 and TA2 performers, the TA3 performer will organize, design, and execute at least four experiments (approximately one per year)

▪ HSR: The TA3 performer will be responsible for managing all Human Subjects Research (HSR) for the program, will likely serve as the Single IRB (Institutional Review Board) for research oversight, and leading coordination with the Army’s Human Research Protection Office (HRPO)

04 September 2024 22

TA3: TESTING, EVALUATION, AND TRANSITIONS FOR HUMAN-

AGENT SYSTEMS FOR INTELLIGENCE ACTIVITIES

(CONTRACT; CUI; CLEARED FACILITY; TS/SCI ELIGIBLE)

▪ Data management: The TA3 performer will work with government-provided and other publicly available dataset(s), and handle identifying, gathering, cleaning, and making available to the program dataset(s) that are capable of testing relevant hypotheses for the program

▪ Technical Exchanges: The TA3 team will host technical exchange meetings with all performers approximately three months before each experiment

▪ Transitions: Strong TA3 proposals will include plans for transition support for the program as theoretical and technical advancements occur, as well as at the end of the program

04 September 2024 23

COLLECTIVE JUDGMENT FORMATION

▪ Each TA team is expected to draw performers from multiple disciplines (max 6 members)

▪ TA1 will require a team led by a social scientist, with a focus on social science approaches to the problems space; the team must contain at least one computer scientist to support integration with TA2 and TA3

▪ TA2 will require a team led by a computer scientist, with a focus on computational modeling approaches to the problem space; the team must contain at least one social scientist to support integration with TA1 and TA3

▪ Each proposal may address only a single TA

▪ Proposers may submit multiple proposals if they seek to address multiple TA, but not multiple proposals to the same TA

▪ A proposing team may submit proposals for all three technical areas

▪ The decision as to which proposal to select for award is at the discretion of the Government

04 September 2024 24

COLLECTIVE JUDGMENT FORMATION

Anticipated Funding:

Pending availability of funds, we anticipate that one award will be made for each TA and that they will have the following funding profiles ($k)

04 September 2024 25

ASSOCIATE PERFORMER

AGREEMENT

04 September 2024 26

COLLECTIVE JUDGMENT FORMATION

Timeline:

04 September 2024 27

QUESTIONS AND ANSWERS

▪ Q&A for pre-submitted questions

▪ Q&A for today’s questions

▪ CJF CRA BAA is the authority Any contradictions between what is said today and the BAA, the BAA wins https://grants.gov/search-results-detail/356024

04 September 2024 28

THANK YOU.

Contact:

cjf-cra-baa@army.mil

Subject:

CJF Proposer Question

04 September 2024 29