SOW_NeuroX_Fusion.pdf

March 9, 2026

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and does not constitute “development” or “production” technology under applicable export control regulations.

Statement of Work

NeuroX-Fusion: LLM-Foundation Model Alignment

Period of Performance:

Twenty-four (24) months from subcontract start date.

1. Background and Purpose The DOE ASCR–funded NeuroX-Fusion project aims to construct a unified, multimodal foundation model of the human brain—integrating MRI, electrophysiology, video, and language—using a Large Language Model (LLM) as a shared semantic space. The roadmap culminates in NeuroX-Fusion, a 130B-parameter “digital twin” that unifies NeuroX-MRI and NeuroX-Ephys via LLM-based alignment.

The Subcontractor shall be a leader in:

• Naturalistic fMRI and intracranial recordings

• Brain–language mapping and LLM–brain encoding

• Neural coupling during storytelling and real-world communication

The Subcontractor will help refine LLM–brain alignment, validate cross-modal fusion between fMRI and electrophysiology, and co-develop protocols to use the digital twin for in-silico experiments.

This Statement of Work (SOW) formalizes a focused, two-year subcontract to:

• Design and validate LLM–Foundation Model (FM) alignment strategies and benchmarks for NeuroX-MRI, NeuroX-Ephys, and NeuroX-Fusion.

• Co-design neuroscience-aligned pretraining and auxiliary objectives that embed neurocognitive constraints directly into NeuroX foundation model training.

• Deliver datasets, benchmarks, and prototype pipelines that integrate early NeuroX checkpoints and drive rapid iteration.

• Academic dissemination (conference and journal publications) is encouraged, consistent with DOE and institutional requirements.

2. Scope of Work and Major Task Areas Work under this SOW is organized into three major task areas:

• Task Area 1 – LLM–Foundation Model Alignment

• Task Area 2 – Neuroscience-Aligned Pretraining Objectives

• Task Area 3 – Prototyping, Datasets, Benchmarks, and Hackathon

Each task area includes specific subtasks and deliverables.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and

3. Detailed Tasks and Deliverables

Task Area 1 – LLM–Foundation Model Alignment

Objective:

Ensure that NeuroX foundation models (NeuroX-MRI, NeuroX-Ephys, and ultimately NeuroX-Fusion) learn latent representations that are neurally and cognitively plausible when aligned to an LLM, especially under naturalistic narrative and communication stimuli.

Task 1.1 – Alignment Design for Naturalistic Language and Narratives

The Subcontractor shall:

• Specify alignment strategies between:

o Token- and layer-level representations from an LLM applied to continuous language (stories, conversations, movie dialogue); and o Brain representations from MRI and electrophysiology for the same stimuli.

• Provide guidance on:

o Which LLM layers and NeuroX feature levels should correspond to different cortical hierarchies and timescales of processing.

o How to handle speaker–listener coupling, inter-subject shared versus idiosyncratic variance, and narrative/event structure in the alignment design.

Deliverable T1.1:

“LLM–Foundation Model Alignment Design Note” (approximately 2–4 pages, PDF), explicitly tailored to NeuroX-MRI, NeuroX-Ephys, and NeuroX-Fusion.

Task 1.2 – Alignment Benchmarks and Evaluation Metrics

The Subcontractor shall:

• Define quantitative benchmarks to test whether NeuroX models, when projected into the

LLM latent space, capture:

o Semantic structure of continuous narratives (event boundaries, topic shifts, suspense points).

o Known hierarchical processing patterns in cortex during naturalistic stimuli.

• Propose metrics including, but not limited to:

o Encoding/decoding performance (correlation/R²) for fMRI and Ephys.

o Inter-subject correlation (ISC) and shared-variance measures.

o Region- and layer-resolved alignment curves.

Deliverable T1.2:

Benchmark specification document (PDF) plus example evaluation scripts (e.g., Jupyter notebooks) runnable on modest-scale models and subsets of NeuroX data.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and

Task 1.3 – Cross-Modal Validation for NeuroX-Fusion

The Subcontractor shall:

• Define tests to evaluate whether MRI and Ephys branches, when aligned through the shared

LLM space, produce consistent semantic trajectories for the same stimuli.

• Identify canonical paradigms (e.g., story listening, movie-viewing tasks) and expected cross-modal patterns (spatial vs temporal signatures, hierarchical processing) that serve as validation targets for NeuroX-Fusion.

Deliverable T1.3:

“NeuroX-Fusion Cross-Modal Validation Playbook” (PDF), with specific contrasts, ROIs, and example analyses.

Task Area 2 – Neuroscience-Aligned Pretraining Objectives

Objective:

Design, prototype, and specify neuroscience- and cognition-aligned pretraining/auxiliary objectives that encode realistic constraints from human communication and brain dynamics directly into NeuroX-MRI, NeuroX-Ephys, and NeuroX-Fusion training. These objectives will complement other neuro-informed loss functions and fusion layers developed by NeuroX collaborators.

Task 2.1 – Objective Design: Naturalistic Narrative and Communication Priors

• Co-design with BNL and collaborators a set of pretraining or auxiliary objectives that encourage NeuroX models to:

o Predict/encode brain responses to continuous narratives given text, video, and/or internal NeuroX representations.

o Capture speaker–listener coupling and shared semantic spaces across individuals.

o Respect multi-timescale structure (fast lexical/phonetic vs slow narrative/event-level dynamics).

• Express each objective as a well-defined loss term compatible with NeuroX training (e.g., contrastive objectives, prediction heads, regularizers).

Deliverable T2.1:

“Neuroscience-Aligned Objective Specification” (PDF) describing each objective, required inputs, pseudo-code for loss computation, and its role in MRI, Ephys, and Fusion training.

Task 2.2 – Prototype Implementations on Small-Scale Models

• Implement reference versions of at least two to three core neuroscience-aligned objectives using:

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and o Public LLMs and/or moderate-scale variants, and o Lab and/or publicly available brain datasets.

• Demonstrate that:

o The objectives are numerically stable; and o They improve alignment metrics defined in Task 1 (e.g., encoding correlation, ISC) compared with simple baselines.

Deliverable T2.2:

Prototype code modules (e.g., PyTorch / HuggingFace style) implementing selected objectives, with example training scripts and a short results note (figures or summary tables).

Task 2.3 – Integration Requirements for NeuroX Training

The Subcontractor shall:

• Translate objective designs into clear API and data requirements for the NeuroX training stack, specifying:

o Necessary internal activations (e.g., layer-wise embeddings, temporal pooling outputs).

o Annotation formats and batching schemes for aligned brain–text–video windows.

• Coordinate with BNL and external collaborators (e.g., Cha/Moon labs) so these objectives are compatible with other knowledge-constrained loss functions and multi-modal fusion layers planned for NeuroX.

Deliverable T2.3:

“NeuroX Training Integration Requirements” (PDF), optionally co-signed with BNL, for engineering implementation.

Task Area 3 – Prototyping, Datasets, Benchmarks, and Hackathon

Objective:

Provide concrete datasets, benchmarks, pipelines, and a focused hackathon that operationalize Task

Areas 1 and 2 and accelerate integration with early NeuroX checkpoints.

Task 3.1 – Dataset Curation for Pretraining and Evaluation

• Curate one or more naturalistic neuroimaging datasets (subject to IRB and data-use agreements) suitable for:

o Neuroscience-aligned pretraining objectives (Task 2).

o Alignment and evaluation benchmarks (Task 1).

• For each dataset, provide:

o Preprocessed fMRI or electrophysiology signals, or standardized preprocessing instructions.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and o Time-aligned transcripts and/or annotations (e.g., words, sentences, events, scenes).

• Recommended splits for model training, validation, and evaluation-only usage.

Deliverable T3.1:

Dataset package(s) or secure access instructions, plus accompanying documentation targeted to

NeuroX engineers and scientists.

Task 3.2 – Benchmark Definitions and Leaderboard Templates

The Subcontractor shall:

• Define benchmark tasks that NeuroX models should be evaluated on, such as:

o Predicting brain activity from text/video embeddings.

o Inferring semantic/event labels from NeuroX brain latents.

o Capturing shared vs idiosyncratic components across subjects.

• Provide:

o Formal task definitions and metric formulas.

o Reference baselines using simpler models (e.g., linear models, smaller transformers).

Deliverable T3.2:

Benchmark specification (PDF) plus runnable evaluation scripts and an optional “leaderboard” template

(CSV/JSON formats).

Task 3.3 – End-to-End Prototype Pipelines

• Assemble minimal end-to-end pipelines mirroring planned NeuroX-Fusion workflows, for example:

o Input: stimulus (text/video) ± brain data → o LLM + prototype brain model → o Neuroscience-aligned losses and alignment metrics → o Evaluation and visualization outputs.

• Ensure these pipelines are structured to allow NeuroX-MRI/Ephys/Fusion checkpoints to be inserted later with minimal code changes.

Deliverable T3.3:

Example notebooks/scripts plus a brief summary of findings on current (non-NeuroX) models.

Task 3.4 – NeuroX-Fusion Hackathon

Objective:

Conduct an intensive joint hackathon to integrate early NeuroX checkpoints into pipelines and stress-test neuroscience-aligned objectives and benchmarks.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and

• In collaboration with BNL, plan and execute a 2–3 day hackathon (virtual or hybrid) during

Year 2 focused on:

o Plugging early NeuroX-MRI / NeuroX-Ephys / NeuroX-Fusion checkpoints into pipelines and benchmarks.

o Rapidly prototyping and comparing neuroscience-aligned pretraining objectives from Task 2.

o Identifying failure modes, gaps in data or interfaces, and high-value follow-up experiments.

• Before the hackathon:

o Prepare a small set of ready-to-run tasks (e.g., narrative/fMRI benchmarks, speaker– listener alignment tests).

o Provide template notebooks and short documentation for participants.

• After the hackathon:

o Summarize experiments, results, and recommended adjustments to objectives, benchmarks, and integration plans.

Deliverables T3.4:

• T3.4-D1: “Hackathon Plan” (1–2 pages: objectives, agenda, roles, and technical requirements).

• T3.4-D2: “Hackathon Outcome Report” (slides or short written report summarizing key results and follow-ups).

4. Academic Publications and Dissemination

Subject to DOE and BNL policies on publications and intellectual property, the Subcontractor is expected to pursue scholarly dissemination of key scientific findings arising from this work, including but not limited to:

• Co-authored conference papers (e.g., NeurIPS, ICML, ICLR, CogNeuro meetings) on:

o LLM–brain/LLM–FM alignment results using NeuroX components.

o Neuroscience-aligned pretraining objectives and benchmarks.

• Journal articles in appropriate neuroscience or computational journals, highlighting:

o New insights into brain–language mapping from NeuroX models.

o Methodological advances in multimodal brain foundation modeling.

Deliverable P1 (Academic Outputs):

A brief annual list of peer-reviewed conference or journal manuscripts submitted and/or published that substantially involve work funded through this SOW, including DOIs where available.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and

5. Schedule and Milestones (24 Months)

Actual dates may be adjusted by mutual agreement, but the following structure is anticipated.

Year 1 (Months 0–12)

• Months 0–2 o Project kickoff meeting (remote) between BNL and Subcontractor.

o Finalize data-sharing and IRB/data-use constraints.

• Months 3–4 o T1.1: LLM–FM Alignment Design Note.

o T2.1 (initial): Neuroscience-Aligned Objective Specification (first version).

• Months 5–6 o T1.2: Alignment Benchmarks and Metrics (docs + initial scripts).

o T3.1 (v1): Initial dataset package(s) and documentation.

• Months 7–9 o T1.3: NeuroX-Fusion Cross-Modal Validation Playbook.

o T3.2 (v1): Initial benchmark definitions and baselines.

• Months 10–12 o T2.2 (v1): Prototype implementations of core objectives on small models/datasets.

o T3.3 (v1): First end-to-end prototype pipelines (without NeuroX checkpoints).

o P1 (Year 1): Short summary of any manuscripts submitted/accepted.

Year 2 (Months 13–24)

• Months 13–15 o Refresh objectives/benchmarks in light of NeuroX architecture decisions.

o T3.4-D1: Hackathon Plan.

• Months 16–18 o Conduct the NeuroX-Fusion Hackathon.

o T3.4-D2: Hackathon Outcome Report.

• Months 18–20 o T2.2 (final): Updated objective implementations incorporating hackathon findings.

o T2.3: Final Integration Requirements for NeuroX Training.

o T3.1 (v2): Extended/updated dataset packages as needed.

• Months 21–24 o T3.2 (final): Stable benchmark suite and leaderboard formats.

o T3.3 (final): End-to-end pipelines refactored for direct use with NeuroX-

MRI/Ephys/Fusion checkpoints.

o P1 (Year 2): Updated list of manuscripts submitted/accepted.

BNL procurement specifications for public solicitation. Contains no export-controlled technical data and o Final closeout meeting and technical summary to BNL.

6. Meetings, Reporting, and Communication

• Kickoff Meeting:

Held within the first month after subcontract start, to refine scope, deliverable timing, and interfaces.

• Regular Progress Meetings:

Approximately monthly remote meetings between Subcontractor and BNL, covering:

o Status of tasks and deliverables.

o Integration issues with NeuroX models and infrastructure.

o Planning for the Year 2 hackathon and publication strategy.

• Progress Updates:

Brief written or slide-based updates may be requested quarterly or aligned with BNL’s reporting schedule.

• Closeout Review:

At or near Month 24, the Subcontractor will present a summary of work completed, main scientific/technical outcomes, and recommendations for future collaboration.

Requester Signature: ________________________ Date: ______________

Tech Rep Signature: ________________________ Date: ______________