DARPA-BAA-15-58_(MediFor)_Amendment_0002.pdf

Broad Agency Announcement

Media Forensics (MediFor)

DARPA‐BAA‐15‐58

September 29, 2015

Amendment 2 (Amended on October 20, 2015)

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203‐2114

DARPA-BAA-15-58 MEDIFOR 2

Table of Contents

PART I: OVERVIEW

PART II: FULL TEXT OF ANNOUNCEMENT

I. FUNDING OPPORTUNITY DESCRIPTION

II. AWARD INFORMATION

A. Awards

B. Fundamental Research

III. ELIGIBILITY INFORMATION

A. Eligible Applicants

B. Procurement Integrity, Standards of Conduct, Ethical Considerations and Organizational

Conflicts of Interest (OCIs)

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. APPLICATION AND SUBMISSION INFORMATION

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. APPLICATION REVIEW INFORMATION

A. Evaluation Criteria

B. Review and Selection Process

VI. AWARD ADMINISTRATION INFORMATION

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. AGENCY CONTACTS

VIII. OTHER INFORMATION

A. Frequently Asked Questions (FAQs)

B. Associate Contractor Agreement Clause (ACA)

C. Proposers Day

D. Submission Checklist

DARPA-BAA-15-58 MEDIFOR 3

PART I: OVERVIEW

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Media Forensics (MediFor)

Announcement Type: Initial Announcement

Funding Opportunity Number: DARPA‐BAA‐15‐58

Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910

Dates o Posting Date: September 29, 2015 o Proposers Day: October 2, 2015 o Abstract Due Date: October 9, 2015, 12:00 noon (ET) o Proposal Due Date: November 24, 2015, 12:00 noon (ET) o BAA Closing Date: November 24, 2015, 12:00 noon (ET)

Anticipated Individual Awards: DARPA anticipates the MediFor program will consist of three phases: one 24‐month phase (base) and two 12‐month phases (options). DARPA anticipates multiple small research awards for part or all of Technical Areas (TAs) 1.1, 1.2, and 1.3. TA2 will require significantly more effort than individual TAs 1.1, 1.2, and

1.3 efforts, and DARPA anticipates multiple awards addressing this TA. It is anticipated that there will be multiple awards for TA3 data collection, manipulations and annotations

Types of Instruments that may be awarded: Procurement contracts, cooperative agreements or Other Transactions (OTs).

Technical POC: Dr. David Doermann, Program Manager, DARPA/I2O

BAA EMail: MediFor@darpa.mil

BAA Mailing Address:

DARPA/I2O

ATTN: DARPA‐BAA‐15‐58

675 North Randolph Street Arlington, VA 22203‐2114

I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities

DARPA-BAA-15-58 MEDIFOR 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. FUNDING OPPORTUNITY DESCRIPTION

DARPA is soliciting innovative research proposals in the area of visual media forensics, the science and practice of determining the authenticity and establishing the integrity of visual media. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that only results in evolutionary improvements to the existing state of art.

This broad agency announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 35.016. Any negotiations and/or awards will use procedures under FAR 15.4. Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and, as applicable, the Grants.gov website (http://www.grants.gov/).

The following information is for those wishing to respond to this BAA.

TERMINOLOGY

The following terms are used throughout this document:

Visual Media Images and videos in a recognized digital format and audio that is associated with a video.

Visual Media Asset An individual image or video.

Manipulate To change, in any way, the representation of visual media as it was obtained from an imaging device.

Authenticate The process of determining that a visual media asset has not been manipulated.

Integrity The quality that a visual media asset can be trusted.

Integrity Score A score from 0 to 1 developed to quantify the integrity of an image or video, where a score of 0 indicates that an image is heavily manipulated and a score of 1 indicates that the image or video is unmanipulated.

Integrity Basis The documented justification for an image or video asset with an integrity score less than 1.

DARPA-BAA-15-58 MEDIFOR 5

Integrity Report An integrity score and associated integrity basis suitable for use by automated systems or intelligence analysts.

MediFor Platform The automated analytics and reasoning components that process images and videos to produce integrity reports.

MediFor Console The MediFor component that provides access to the

MediFor platform for the analyst to configure the platform and analyze results.

Visual Media Forensics The process of establishing the integrity of a Visual Media

Asset.

INTRODUCTION/BACKGROUND

A. Program Overview

Historically, the US Government deployed and operated a variety of collection systems that provided imagery with assured integrity. In recent years, however, consumer imaging technology (digital cameras, mobile phones, etc.) has become ubiquitous and it is estimated that an average of 1.8 billion images and videos were loaded to social media per day in 2014.

This imagery represents a huge opportunity for the Department of Defense (DoD) except for one critical drawback: a growing proportion of this visual media has been manipulated. Many manipulations are benign, performed for fun or for artistic value, but some are for adversarial purposes, such as propaganda or misinformation campaigns. While deciding the reason for a given manipulation is not within the scope of MediFor, having a complete understanding of what manipulation was done is essential for analysts and systems to ultimately decide whether to use the image or video.

The manipulation of visual media is enabled by the ready availability of sophisticated image and video editing applications (many available as freeware downloads) that enable even novice photographers and videographers to manipulate visual media in ways that are very difficult to detect either visually or with current image analysis and visual media forensics tools. The forensic tools used today lack robustness and scalability and address only some aspects of media authentication; an end‐to‐end platform to perform a complete and automated forensic analysis does not exist. Although there are a few applications for image manipulation detection in the commercial sector, they are typically limited to a yes/no decision about the source being an “original” asset, obtained directly from an imaging device. As a result, media authentication is typically performed manually using a variety of ad hoc methods that are often more art than science, and forensics analysts rely heavily on their own background and experience.

The MediFor program aims to level this playing field, which currently favors the image manipulator, by developing technologies for the automated assessment of the integrity of an image or video. The program will integrate these technologies in a visual media forensics platform that, for a given image/video, will automatically detect manipulations, provide

DARPA-BAA-15-58 MEDIFOR 6

analysts and decision makers with detailed information about the types of manipulations performed, how they were performed, and their significance/importance in order to facilitate decisions regarding the intelligence value of the image/video. The MediFor platform will also automatically discover associations across visual media collections as another means for confirming the veracity of an image/video.

B. Program Scope and System Concept

The MediFor program has adopted a model for image and video integrity comprised of three principal elements:

Digital Integrity Indicators: Are the pixels or the representation of the image or video inconsistent? Are there examples of pixel‐level features that cast doubt on the digital integrity such as edge discontinuities, blurred pixels, or repeated image regions? Do metadata and/or representation artifacts suggest manipulation?

Physical Integrity Indicators: Are there image or video features that appear to violate the laws of physics? Do features from the 3D scene include shadows, reflections, and/or kinematics (video) that are inconsistent?

Semantic Integrity Indicators: Do other information sources corroborate or contradict results of the digital or physical analyses or any assumptions made about the asset? Is there evidence that the date, time, or location is not correct using external knowledge, or that there are inconsistencies in digital or physical features within a group of assets?

In attempting to discover if an asset may have been repurposed, is there other evidence that shows an asset is not what it is claimed to be?

The program envisions that as a first step in quantifying the integrity of an image/video one must first consider its integrity in terms of these three principal elements. Accomplishing this will require the development of appropriate algorithms for the automated computation of digital, physical, and semantic integrity indicators. Each of these analytics will accept an image/video as input and output the following:

A score expressing the confidence that the indicator is present.

A set of characteristics, such as regions of the image/video, where indicators have been detected.

Explanation of nature and potential significance of detected indicators, for example, smoothed image region suggestive of deletion/obscuration of image features (digital), inconsistent shadows (physical), or similar images that show the same scene but contain additional details/features (semantic).

Establishing the overall integrity of an image/video will require advanced reasoning techniques for automatically assimilating detected elements of digital, physical, and semantic integrity into a comprehensive quantitative assessment of integrity. For each image/video, the following outputs are required:

A single summary integrity score; and

DARPA-BAA-15-58 MEDIFOR 7

A summary integrity report that explains the key factors that influenced the integrity scoring.

As can be seen, a key technical challenge for the MediFor program is the creation of a quantitative score for integrity. Proposers must provide an implementable algorithmic description of a process for computing an integrity score from the set of digital, physical and semantic indicators for a given image or video. The basis for this computation is left to the proposer, but should reflect both application‐specific and application‐independent concepts of operations. That is, the integrity score may be computed on the basis of features internal to the image or embedded image metadata, or it may be computed on the basis of external features such as provider reputation, associated images or video, or other information that is available from the established source. Proposers must also provide a procedure for evaluating their proposed integrity scores. For example, if one conceptualizes the integrity score as a test statistic, it can be evaluated in terms of its probability of detection (PD) for manipulated images and its probability of false alarm (PFA) for original images, through the use of receiver operating characteristics (ROCs). (Note: a similar approach was taken for the evaluation of an algorithm for computing the “truth value” of a textual statement.1)

The development of integrity scores for images and videos from these integrity indicators is considered one of the high risk aspects of the program, in part because of the subjective nature of integrity and in part because the envisioned use of the integrity score may in fact preclude the use of a single measure for all applications. To elaborate, consider the following question:

Does compression decrease the integrity of an image? If the result of the compression is to remove critical detail, most analysts would answer “yes.” But when is detail critical? That will generally depend on the application. Thus alternative integrity metrics that together provide a comprehensive assessment of various aspects of integrity might be needed for use in diverse applications. Proposers should explore these considerations in their proposals to the extent they deem necessary.

The program will integrate these manipulation detection and integrity analytic technologies in a visual media forensics platform that, for a given image/video, will automatically detect manipulations, provide analysts and decision makers with detailed information about the types of manipulations performed, how they were performed, and their significance/importance in order to facilitate decisions regarding the intelligence value of the image/video. The MediFor platform will also automatically discover associations across visual media collections as another means for confirming the veracity of an image/video. It is envisioned that the MediFor platform will automatically triage incoming image and video assets to assess their integrity and to determine linkages to other images/videos (through visual, temporal, spatial, source, and other means of association) prior to use by analysts.

The MediFor platform will be augmented by a MediFor console that will be developed to facilitate operator use and interaction. For example, using the MediFor console an analyst can provide general contextual information, including hypotheses to be tested, application

1 Ciampaglia GL, Shiralkar P, Rocha LM, Bollen J, Menczer F, Flammini A (2015) Computational Fact Checking from Knowledge Networks. PLoS ONE 10(6)

DARPA-BAA-15-58 MEDIFOR 8

constraints, and appropriate world knowledge. The console may also be used to examine/drill‐ down into an integrity report and associated analytics, but should not be viewed as a tool for interacting directly with individual images or videos as part of the analysis process.

The MediFor platform and the MediFor console will be developed so as to interoperate seamlessly as the MediFor system. It is intended that the MediFor system will be capable of automatically processing images and videos at very large scales (millions of images/videos per day) in support of diverse applications spanning intelligence analysis, criminal prosecution, and combating misinformation campaigns and propaganda. Figure 1 depicts the MediFor system concept.

Figure 1: The MediFor System Concept where xi is the ith indicator for x = d(digital), p(physical) or s(semantic)

C. Program Structure and Plan

The overriding objective of the MediFor program is to produce the best possible technologies to advance the field of media forensics. To this end, DARPA intends MediFor to be a collaborative program in which all performers (within and across Technical Areas) constructively interact with one another. To facilitate the open exchange of information, performers may have an associate contractor agreement (ACA) clause included in their award (see Section VIII.B). This clause is intended to ensure appropriate coordination and potential integration of work done by the MediFor performers. Once selections have been made, selectees should have their ACAs in place prior to the first program kick‐off meeting.

DARPA-BAA-15-58 MEDIFOR 9

A key goal of the program is to establish an open, standards‐based, multisource, plug‐and‐play architecture that allows for interoperability and integration. This goal includes the ability to easily add, remove, substitute, and modify software and hardware components. Rapid innovation will be facilitated by providing a base for future users or developers of program technologies and deliverables. Therefore, DARPA desires that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government with a minimum of Government purpose rights (GPR), as lesser rights may adversely affect the lifecycle costs of affected items, components, or processes

The MediFor program will emphasize creating and leveraging open source technology and architecture. Intellectual property rights asserted by proposers are encouraged to be aligned with open source regimes and are strongly preferred for TA1 and TA2 research. Any proposer claiming the use of proprietary technology or choosing to explicitly exclude their technology from the open source regime will need to provide adequate justification.

It is planned that the development of the MediFor platform will be driven in part by the creation of a MediFor data corpus that includes both high integrity (e.g., original) and manipulated images/videos for development and testing purposes. Due to the highly dynamic/reactive/adversarial nature of image/video manipulation, it is envisioned that the quantity or diversity of data made available may be useful for evaluation and system development but may not be sufficient to support training in the sense that has become standard in the computer vision and machine learning communities.

MediFor will promote community‐wide involvement by conducting open evaluations and periodic challenges as described in this document. All performers will be expected to participate in these evaluations as a program requirement.

It is anticipated that the MediFor program will consist of three phases: Phase 1 will be 24 months long, and Phases 2 and 3 will each be 12 months long. MediFor Technical Areas (TAs) will be as follows and are described in detail further below.

TA1 – Integrity Analytics Research and Development TA1.1 – Digital Integrity TA1.2 – Physical Integrity TA1.3 – Semantic Integrity

TA2 – Integrity Reasoning Engine and MediFor Console Development

TA3 – Media Resource Development: Corpora Creation, Manipulation, and Annotations

DARPA-BAA-15-58 MEDIFOR 10

TA1 – Integrity Analytics Research and Development (R&D)

TA1 will focus on the development of analytics that can be used to detect manipulations to assist in measuring integrity. Proposals submitted for TA1.1, TA1.2, and/or TA1.3 are not required to address all parts of any subarea, nor all forms of media. For example, a proposal that deals with the digital (TA1.1) and semantic (TA1.3) integrity of video will still be considered responsive. DARPA anticipates multiple research awards for TA1.1, TA1.2, and TA1.3.

TA1.1 – Digital Integrity

Research in TA1.1 will entail:

Analysis of the representation and structure of the image or video including, but not limited to file structure, metadata, compression, and color tables.

Analysis of the image or video content at the pixel level.

Identification of trace evidence and media DNA suggestive of a manipulation history.

Proposals to TA1.1 should focus on R&D of novel techniques for analyzing the format, structure and content of images and videos at the pixel level in order to discover irregularities and the causes for these irregularities. Irregularities may arise from a variety of operations such as resizing, compression, cropping, rotation, shearing, blurring, and alteration by cut‐and‐paste operations. TA1.1 algorithms are expected to provide more information than just a yes/no decision on manipulation. They are expected to detect the type(s) of manipulation used (when the algorithm is a detector for that particular manipulation), a parameterization of the manipulation, the location in the asset, and a measure of the uncertainty of detection.

Proposals for TA1.1 may focus on tracing the origin of the image or video and, in case of alterations, the sources of the material used to create the altered asset.

TA1.2 – Physical Integrity

Research in TA1.2 will involve analyzing characteristics of the scene captured by the image or video that either support or dispute an interpretation of the scene as a valid 2D projection of the 3D world, or provide additional knowledge that can be used to increase or decrease an integrity measure.

Proposals to TA1.2 should focus on R&D of novel techniques for determining if physical properties of the imaged instance of the scene such as kinematics (video), shadows, reflections, and the effects of various natural phenomena, are consistent and well behaved. Because images and videos are 2D projections of the 3D world, it is difficult to maintain this type of consistency when images are altered. Proposals for TA1.2 should detail the methods for detecting the physical features and the degree of consistency for each feature. Proposals can address a single or multiple features, but must not assume the existence of specific physical properties a priori and, thus, must avoid hypothesizing features when they do not exist. TA1.2 algorithms are expected to provide more information than just a yes/no decision on manipulation; they must summarize a basis for the manipulation detection decisions.

DARPA-BAA-15-58 MEDIFOR 11

TA1.3 – Semantic Integrity

Research in TA1.3 will require:

Confirmation or rejection of a hypothesis (provided through the console [see below]) about the asset or set of assets.

Determining whether the image or video was repurposed from its original or intended use.

Identification of spatial, temporal or visual associations between images and/or videos in potentially large collections.

Establishing the provenance of an image or video through its manipulations or modifications.

Proposals to TA1.3 should be focused on novel techniques for establishing the semantic integrity of media. Semantic integrity can be established when the media is captioned or when a hypothesis is provided to the algorithm to be confirmed such as a date, time, location or description of content. Certain features can be performed in the absence of such a hypothesis (e.g., association with other assets). TA1.3 algorithms must accept a hypothesis or other criteria as input, whose format will be a templated frame or structured query (to be described early in Phase I). TA1.3 systems will need to rely heavily on external resources including publically available data sources that will be gathered by performers and served through the console and the association of the assets with other assets in a world dataset (see below).

The TA1.3 results must provide significant improvement on current state‐of‐the‐art classification and association techniques, and not just repurpose existing techniques designed for image and video retrieval.

During Phase 1, TA1.1, TA1.2, and TA1.3 performers are expected to work with the TA2 performers to integrate algorithms, knowledge, resources, or other results into a TA2 performer's workflow. TA1 proposers must provide their algorithms, knowledge, resources, and other results to the TA2 performers for this purpose. TA2 performers are expected to consider and, when appropriate, integrate new ideas, algorithms, and resources from multiple TA1 performers over the course of the program.

The TA1 results must not be solely theoretical in nature. TA1 performers are expected to provide substantive support to the overall program goal of verifying the integrity of all images and videos. Results from TA1 are expected to be incorporated into algorithms, knowledge, and models that contribute to the MediFor platform. For that purpose, the TA1 algorithms will have to be enclosed in an application programming interface (API) layer and shared with all TA2 performers providing the required information. The API will be developed by the performers in the early stages of the program in conjunction with DARPA and government representatives.

TA1 performers should not feel constrained by standard information exchange representations.

They should feel free to develop novel ideas and progressive information representation that form a basis for manipulation detection decisions, and the program will decide how to share

DARPA-BAA-15-58 MEDIFOR 12

this information. TA1 and TA2 performers will also work closely during Phase I to develop an appropriate representation for any external resources to be shared with all performers through the console.

TA1 performers will be required to participate in annual National Institute of Standards and Technology (NIST) evaluations to determine the accuracy and effectiveness of their algorithms.

The NIST evaluations will consist of a set of manipulated images and videos distributed among a larger collection of assets. The algorithms developed by the TA1 performers will attempt, at a minimum, to identify which have been manipulated, and determine the type, scope, location, and confidence of the manipulation detected. TA1 systems specify which types of manipulations they are addressing, and will not be penalized for not detecting out‐of‐scope manipulations. As part of semantic integrity, corroborating evidence such as the corresponding unmanipulated versions of the assets should also be identified through association. The performance will be determined by accuracy of the algorithms (more details are provided in Section I.E).

Performers in TA1 are expected to show improvements in the accuracy of their algorithms over the state of the art, either through the NIST evaluation or through performer self‐run evaluations when the NIST evaluation is not sufficient to fully evaluate their technologies2. At the midterm of Phase 1, approximately 10 months into the program, all TA1 algorithms and/or tools will be evaluated, the results of which need to meet or exceed the existing state of the art and be used as a baseline for further improvement of the algorithms. At the end of each phase, performers will have to measurably decrease the error rate of the algorithms from the previous phase or baseline while increasing the efficiency of their algorithms. Performers in TA1 not meeting the goals of the program, not making important contributions to the program, or being out performed by competing approaches, may have not have their options exercised.

Cooperation among teams is strongly encouraged and will be considered as a way of objectively contributing to the program.

TA2 – Integrity Reasoning and MediFor Console Development

Proposals to TA2 should be focused on architecting the MediFor platform that integrates the TA1 algorithms, on the R&D of novel logic and inference techniques for orchestrating the output of the TA1 algorithms, and on the creation of the MediFor console. The system developed under this TA must fuse the models, rules, assertions, or extracted information produced by TA1 algorithms to create an integrity report for each image or video the system receives. The TA2 performers are expected to develop the means to establish a quantitative integrity score, and an associated scale and relate the score to a generic belief that an asset is not manipulated as described above.

The console will serve as a way for the user/analyst to configure the MediFor platform and

2 If TA1 algorithms are not covered by existing community evaluation protocols, a description of how techniques could be evaluated, the data required for evaluation, and how this data could be obtained must be included in the statement of work section of the abstract and in the full proposal in Appendix B.

DARPA-BAA-15-58 MEDIFOR 13

interact with the resulting MediFor integrity reports. The console should be designed to handle at least the following types of information:

Hypotheses about a set of images or videos (e.g., location or time of day or world knowledge such as geospatial information).

An external set of potentially corroborating or disputing images or videos presented to the system.

External databases of world facts that can be used to support or dispute the above hypotheses.

The console will need to manage and provide this information when requested by the TA1 algorithms. A proposed format and process for this communication should be detailed in TA2 proposals, and will be finalized in the early stages of Phase I after collaboration with TA1 performers. The console must also communicate with external sources such as external corpora to provide additional assets to the TA1 algorithms to enable association of image and videos to establish corroborating evidence or to find the sources in the case of cut‐and‐paste manipulations. The program will focus only on the visual aspects of the images and videos, and not, for example, have access to external text from associated html pages, beyond what is supplied through the console. The console will expose the user/analyst to information from the integrity report for the analyzed media including an integrity score, an integrity basis, information about the genealogy of the assets, or sources used for copy/paste manipulations.

The MediFor architecture must include and run multiple algorithms performing the same or similar analyses, combining the results if appropriate. The NIST evaluation team will assess the effectiveness of the algorithms when used in isolation or as a part of system combination. TA2 performers will collaborate with TA1 performers, and other TA2 performers regarding the design of a common set of representations and APIs to enable them to communicate with the TA1 algorithms.

TA2 performers will participate in NIST evaluations during Phases 2 to determine the accuracy and effectiveness of their platforms.

TA2 performers will conduct several transition exercises with combatant command (COCOM) partners or other transition partners. For each exercise, the TA2 performers will integrate the platform and console into the transition partner's test environment, integrate with the transition partner’s data streams, and conduct tests with transition partner supplied personnel attempting to produce an analysis of the media corpus. Integration and testing exercises may be conducted at U.S. transition partner sites. Performers should budget for three such exercises in the Washington D.C. area, to include software installation, training and support for a three month test and evaluation period.

DARPA anticipates that TA2 work will start at a low level at the start of Phase 1 and involve planning and coordination activities, including the definition of APIs and coordination with TA1 performers regarding the specific media processing tools. In Phase 2, the level of effort for TA2 will increase and include the development of the console, as well as integration and

DARPA-BAA-15-58 MEDIFOR 14

coordination with transition partners. In Phase 3, it is anticipated the work of TA2 will increase more, as prototypes are tested in transition partner spaces3.

After the end of Phase 2, the TA2 platforms should be sufficiently architected and stable so others (in particular, the evaluation team) can upload collections of images or videos (including manipulated and unmanipulated media files) for automatic judgments.

DARPA anticipates providing a suite of basic search and authentication algorithms that are currently available in the literature or open source, as well as select knowledge resources at the program kickoff. These tools may or may not provide the output information that the program will ultimately require, but should be sufficient as place holders for TA2 teams to begin development of their architecture, without waiting for TA1 teams to complete prototypes of their algorithms. The knowledge resources may include, for example, a database of camera parameters, jpeg compression tables or sun angles, but are not intended to be a comprehensive list of what TA1.3 performers will need. TA2 teams are welcome to use additional existing or open source algorithms, provided these algorithms are shared with all performers and require minimal new development efforts.

TA3 – Corpora Creation, Manipulation, and Annotations

TA3 proposals will focus on collection, manipulation, and annotation of media resources to create corpora in support of TA1 and TA2 development and testing by the NIST evaluation team. In both cases, the goal is to accurately reflect real world problems and test the state of the art in manipulation detection algorithms. (See the Performance Evaluation section for details regarding the NIST evaluations).

The resources developed under TA3 will consist of large world image and video dataset and a smaller manipulated image and video dataset. The world dataset will be used by the TA1 and TA2 teams to establish provenance and association among image and video assets. It is intended to represent the types of media assets of unknown origin that one may find in open source material on the Internet. It will be limited to visual media and possibly associated captions, but not full text. The manipulated images and videos will be embedded in a larger set of images and videos, resulting in a probe dataset for evaluation.

For the world dataset, the TA3 performer will collect 45 million images and 450,000 videos. The collection will occur during the first phase of the program (24 months) and be distributed in nearly equal amounts at 3‐month intervals to the evaluation team. It is envisioned that the TA3 performer will gather a significant percentage of the necessary media from open source material with creative commons licenses. Although the program is interested primarily in the visual media, performers should collect as much associated metadata and context as possible.

This data set will not be shared outside of the MediFor program without DARPA’s approval.

The data collected should have image content that includes a variety of scenes ‐ indoor (office, home, airport, shopping mall); outdoor (city street, farm, sea) and a variety of light source(s), 3 At the time of proposal submission, all proposers to TA2 should have personnel with a Top Secret clearance who are eligible for SCI, so that SCI discussions can be held with transition partners.

DARPA-BAA-15-58 MEDIFOR 15

different lighting conditions/directions, color/grey images, shadow cases, and reflection cases.

Image content should reflect a variety of imaging devices and imaging parameters. For example: camera model, focal length, aperture setting, and distance to scene, etc.; and for scanners, the scanner resolution, color/gray scale, and scan modes. It is critical for the TA3 performer to be familiar with the state of the art in manipulation detection, knowing for example what types of images representations and content make detection difficult. TA3 performers will also be asked to assimilate data gathered and provided by TA1 performers that have data needs beyond the envisioned scope of the technical area.

At the start of the program, DARPA intends to seed this collection with a baseline world dataset of 5 million images and 50,000 videos.

For the manipulated dataset, the TA3 performers will manipulate approximately 5,000 images and videos in the first two phases of the program. The performer will be required to develop various scenarios where the evidence of manipulation may either be limited to a given image or video or may be distributed across the larger world dataset. It is envisioned that TA3 performers will provide an “adversarial team” to research, develop and carry out specific manipulations that encompass realistic challenges faced in the field for the specific purpose of testing MediFor technologies. Ideally, the team will be formed from various fields of expertise to produce a wide variety of manipulations with various complexities. The TA3 performer team should be familiar with the state of the art in manipulation detection, knowing for example that good visual manipulations do not necessarily indicate difficult detection.

There are many types of manipulations from basic image processing through complex scene changes and fully computer‐generated scenes. It is expected that most of the images and videos will be manipulated by multiple techniques. For example in a paste manipulation, some form of resizing, rotating, recompression, and smoothing or blurring may be used. TA2 performers will be expected to identify all sources of manipulation in their integrity basis. The TA3 performer responsible for manipulations will track and record the type and location of each manipulation in a separate database. The test corpora will be developed and exposed to performers as part of the evaluation by the NIST Evaluation Team.

At the start of the program, DARPA will provide approximately 100 manipulated images. When augmented with unmanipulated images and videos, this set will act as an initial probe set for evaluation and as examples for TA1 and TA2 performers. The TA3 performer will also work with the Government and the NIST Evaluation Team to develop a baseline set of detection algorithms representing the state of the art, to ensure the manipulations they are producing are sufficiently challenging for the program.

The schedule for manipulation production will follow the schedule for the collection, where each quarter after the beginning of the program the performer responsible for manipulation will provide images and videos. The manipulation performer will lag the data creation performer by 3 months so the first contribution will arrive six months after the start of the program.

DARPA-BAA-15-58 MEDIFOR 16

D. Submissions to Multiple Technical Areas

A proposer may submit proposals for any number of TAs using a separate proposal for each TA (i.e. TA1, TA2 and TA3), but only one proposal is required for addressing any number of sub areas within TA1 (i.e., TA1.1‐1.3).

While proposers may submit proposals for multiple TAs, proposers selected for TA3 cannot be selected for any portion of TA1 or TA2, whether as a prime, subcontractor, or in any other capacity from an organizational to individual level.

E. Performance Evaluation

NIST will be responsible for executing three types of program evaluations for MediFor. The first will be an open evaluation that will measure the specific capabilities of individual TA1 algorithms. All MediFor TA1 performers are expected to participate in these open challenges.

A second will be directed to the MediFor platform to determine the accuracy of the integrated system. The third will focus on field tests and will assess the adequacy of the MediFor platform for the kinds of analysis tasks that are of interest to the transition partners.

In addition to the formal performance evaluations, NIST will host the “Fool My MediFor” challenge where participants (not limited to the MediFor program) will be encouraged to submit manipulated images or videos to the MediFor platform to see if their image manipulations can be detected.

TA1 Evaluations:

The output of integrity indicators may vary according to the different methods of editing and/or manipulating images or videos. Generally, systems will report evidence of the manipulation, which includes but is not limited to the type of manipulation, how it was applied, the region(s) that was manipulated, and the forensic technologies used in the detection.

For example, the manipulated images/videos can be generated using combinations of the following processes, among others.

Media header (metadata) modification

Enhancement and marking (typically benign) o Histogram equalization o Color enhancement o Contrast adjustment o Filtering

Geometric modifications ‐ enlargement, rotation, zoom, cropping, shearing, …

Color change

Cloning ‐ copy‐pasting including adding, replacing, moving, and removing object(s) in the original

DARPA-BAA-15-58 MEDIFOR 17

Splicing ‐ copy‐pasting including adding, replacing object(s) in the original from another sources

Recapturing ‐ recapturing partial/segmented or full media object (maybe manipulated)

Computer generated content (CGC) ‐ realistic images or videos generated by computer graphics software.

Signs of these manipulations can be subtle or counter forensics can be used to hide them. Each algorithm will report which type or types of manipulations it is detecting and the parameters of the manipulation.

The evaluations will be broadly grouped into the following tasks:

Analytics Evaluation (Manipulation) ‐ Determine if assets are genuine/edited/manipulated, and if so, how.

Association Evaluation (Provenance) ‐ Map the who, what, and when (genealogy) of an asset.

Verification Evaluation (Hypotheses) ‐ Given a claim about an asset, prove, disprove, and/or identify inconsistencies based on external resources.

The integrity basis will be viewed as all of the evidence gathered to support the detected manipulation, to justify the provenance of the asset, and/or to support or counter the claim made about the asset.

The technology evaluations run by NIST will occur four times during the program. The first evaluation is expected to occur during month 10 or 11 of Phase 1 and will serve as a baseline.

The following three evaluations will occur toward the end of each phase.

TA2 Evaluations:

TA2 platform evaluations will measure the capabilities of each TA2 technology using the TA1 algorithms. This evaluation will occur during month 10 of Phases 2 and 3 and will be organized by NIST. It will measure the capability of the entire system while also measuring the effectiveness of each algorithm integrated into the platform by a careful analysis of a report generated by the platform. The platform’s output will be measured in terms of accuracy and the completeness of the integrity report, which include enumeration of the types of manipulation found, their location, severity and confidence.

TA2 transition evaluations will provide information on the adequacy of the MediFor system to support the kinds of analysis tasks that are of interest to the transition partners. These partner evaluations will be conducted annually beginning in Phase 2 and run by NIST in conjunction with DoD partners. For each evaluation, several scenarios will be developed by the NIST led evaluation team in consultation with the transition partner and an appropriate corpus will be made available. The evaluation will measure the accuracy of the MediFor system and the effectiveness in producing meaningful results to the user.

DARPA-BAA-15-58 MEDIFOR 18

Early in Phase 2, for calibration of the field evaluations, multiple analysts/mission planners will perform the same tasks that later will be required of analysts/mission planners using MediFor.

This evaluation will test the console’s ability to find anomalies and problem images and perform association of trusted images. The corpora used in this evaluation will include challenges for the MediFor platform, either naturally occurring or seeded by the TA3 performer adversarial team.

The purpose of these performance evaluations is to inform the adjustment of research directions in the program. Evaluation results will be used as a feedback mechanism to determine which technologies require further improvement and which are mature enough for implementation.

F. Schedule/Milestones

Table 1 below shows the schedule for the program. All evaluations are shown for both the technologies and the integrated system. A kick‐off meeting will be scheduled at the start of the program and Principal Investigator (PI) meetings will occur twice a year.

Table 1: Notional MediFor Schedule

G. Meetings and Travel

PI meetings will be held approximately every 6 months, in various locations in the continental United States as specified by the Government. For planning purposes, assume that PI meetings will alternate between US locations on the east coast and west coast by phase. The goals of the PI meetings will be to:

review system architecture and integration progress;

review the accomplishments of each performer;

DARPA-BAA-15-58 MEDIFOR 19

demonstrate prototypes and other accomplishments; and coordinate APIs and other points of interaction between performers.

Additionally, TA2 performers should anticipate one to two trips to the Washington DC metro area per phase for meetings with the Government or transition partners and hosting one site visit per year from the program manager.

Regular teleconference meetings are encouraged to enhance communications with the program manager, the evaluation team, and the Government’s technical representatives.

Should important issues arise between program reviews, the program manager will be available to support informal interim technical interchange meetings and reviews.

Academic performers are encouraged to involve graduate and post‐graduate students (in addition to faculty and research staff) in PI meetings, site visits, and teleconferences.

Especially during Phases 2 and 3, personnel cleared to SCI from the TA2 performer will be expected to have multiple trips to COCOM transition partner locations.

H. Deliverables

During Phase 1, TA1.1, TA1.2, and TA1.3 performers will demonstrate value to the overall program goals. The preferred method for demonstrating value is to work with a TA2 performer to integrate TA1.1, TA1.2, or TA1.3 algorithms, knowledge, resources, and other results into a TA2 performer's workflow. Proposers for TA1.1 – 1.3 must provide their algorithms, knowledge, resources, and other results to the TA2 performers for integration and evaluations.

TA2 performers are expected to be receptive to new ideas, algorithms, and/or resources from other TA1 performers over the course of the program.

All performers (TA1, TA2 and TA3) shall be required to provide the following deliverables via DARPA’s Technical‐Financial Information Management System (TFIMS) database:

Technical papers and reports ‐ Initial reports shall be submitted within 1 month of the program kickoff meeting and after each annual review.

Quarterly progress reports ‐ A quarterly progress report describing progress made, resources expended, and issues requiring the attention of the Government team shall be provided within 15 days of the end of each fiscal quarter.

Monthly financial reporting.

Final report ‐ The final report shall concisely summarize the effort.

In addition to reports, TA1 and TA2 performers under procurement contracts/OTAs shall be required to provide the following deliverables to DARPA:

Intermediate and final versions of software libraries, code, data, and prototypes (Intermediate versions 1 month after each evaluation).

DARPA-BAA-15-58 MEDIFOR 20

Implementation documentation ‐ Documentation shall be provided one month after each code drop documenting any algorithms, source code, hardware descriptions, language specifications, system diagrams, part numbers, and other data necessary to replicate and test the designs.

Application user manual and training material.

In addition to reports, TA3 performers under procurement contracts/OTAs shall be required to provide data, manipulated images and videos, and annotations as detailed in the TA3 description.

I. Government‐furnished Property/Equipment/Information

The Government will provide the world dataset released periodically throughout the program, and approximately 50% of the probe data will be released after each evaluation.

J. Intellectual Property (IP)

A key goal of the program is to establish an open, standards‐based, multi‐source, plug‐and‐play architecture that allows for interoperability and integration. This open architecture includes the ability to easily add, remove, substitute, and modify software components. The architecture will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables.

Intellectual property rights asserted by proposers (especially to TA1) are strongly encouraged to align with open source regimes. Proposers who wish to assert IP rights that are not aligned with open source regimes must explain in detail why the asserted IP rights will aid in effective transition and use of the technologies, and why an open source solution is not feasible. DARPA desires all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government purpose rights (GPR), as lesser rights may adversely affect the lifecycle costs of affected items, components, and processes. See Section VI.B.1 for more details on intellectual property.

DARPA-BAA-15-58 MEDIFOR 21

II. AWARD INFORMATION

A. Awards

Multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous and provide the best value to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre‐priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, cooperative agreement, or Other Transaction Agreement (OTA) depending upon the nature of the work proposed, the required degree of interaction between parties, and…