Draft SOW_ASL_Dataset.pdf

ASL Dataset and Lip-Reading Dataset Statement of Work (SOW)

Background

The National Telecommunications and Information Administration’s Institute for

Telecommunication Sciences (NTIA/ITS), on behalf of the National Institute of Science and Technology (NIST) Public Safety Communications Research (PSCR) Division is creating datasets that investigates the relationship between video quality and the ability of people to communicate with American Sign Language (ASL).

This dataset will be made freely available on the Consumer Digital Video Library

(CDVL) for research and development purposes only. The dataset will support the development of no reference (NR) metrics for video quality assessment (VQA) that allow future camera systems to detect and mitigate problems that would hinder first responder applications. The dataset will also support advances in modern camera systems that better accommodate the needs of the Deaf and hard of hearing.

Objectives

The goal is to create an ASL dataset. The ASL dataset will contain short, silent videos of people communicating with ASL at a natural pace, as if talking with someone in the room. Preferably, each video recording will be unique (e.g., the dataset does not contain the same video segment recorded at two different bitrates).

The final experiment design will be agreed upon jointly by NTIA/ITS and the Contractor.

The main variables for the ASL datasets are video resolution, compression, and ASL language grammatical element (e.g., body language, finger spelling). The dataset will include the “original” recording (after editing and format conversion to the test monitor resolution) to assess the upper quality limit imposed by the camera, video format, and test monitor.

The dataset will include subject ratings collected from subjects on the 5-level Absolute

Category Rating (ACR) scale, in accordance with ITU-T Rec. P.913. The test requires at least 24 subjects who are proficient in ASL.

The dataset (videos and ratings) will be distributed on for research and development purposes on the Consumer Digital Video Library (CDVL, www.cdvl.org).

The Contractor shall collaborate with NTIA/ITS to co-author paper(s) that describe and analyze the datasets.

Scope

The Contractor must:

• Provide a lead researcher who has published at least fifteen (15) papers on related research (e.g., image or video quality subjective tests, objective metrics that assess image or video quality, and video for ASL).

http://www.cdvl.org/

ASL Dataset and Lip-Reading Dataset SOW

• Provide subject matter expertise on ASL.

• Ensuring the sound planning and execution of the experiment.

• Provide NTIA/ITS with permission to redistribute the dataset on CDVL.

• Assist NTIA/ITS with finalizing the experiment design.

• Determine a suitable length for the edited videos in each dataset, the goal being a shortest duration that allows subjects to comfortably watch, understand the message, and rate the videos.

• Edit the ASL footage into short videos that contain appropriate content for these tasks. The edited “original” videos must be saved in a high bit-rate format that is visually lossless (e.g., H.264/AVC at 20 Mbps).

• Create the dataset, conduct pre-testing, and provide ratings from 24+ subjects.

• Conduct the test on monitors with resolution at least (1280 × 720).

• Follow standard procedures for video quality testing, as per ITU-T Rec. P.913, including the ethical treatment of subjects.

• Provide a readme file listing acknowledgements to be distributed with the dataset, including any language the Contractor requires around privacy concerns.

• Co-author a paper that describes and analyzes the dataset.

NTIA/ITS will provide the following:

• Professionally produced footage that depicts people communicating conversationally in ASL in front of a mottled grey background (video only).

• Draft experiment design for the ASL datasets.

• Consultation and advice on all aspects of the dataset creation and subjective testing.

• Co-author a paper that describes and analyzes the dataset.

Tasks

The Contractor’s first task will be to establish a schedule. The Contractor shall (1) create a dataset that investigates the relationship between video quality and the ability of people to communicate with ASL, and (2) document the dataset.

Delivery

The dataset must be delivered in electronic form (video files) to Margaret H. Pinson, mpinson@ntia.gov, at 325 Broadway, Boulder CO, 80305. Electronic delivery is preferred. Receipt will be confirmed via email.

Delivery is expected no later than 1 year after the award of this SOW.

Government-Furnished Property, Material, Equipment, or Information (GFP, GFM, GFE, or GFI)

No GFP, GFM, GFE, or GFI will be used.

Security

There are no security considerations.

mailto:mpinson@ntia.gov

ASL Dataset and Lip-Reading Dataset SOW

Travel

NTIA/ITS has no travel requirements for this task, but the Contractor may include a travel budget for cover their needs (e.g., to conduct the subjective test at a remote location with Deaf subjects).

Special Material Requirements

No special materials are required.

Other Unique Requirements

The Contractor must provide video editing software and computers to conduct the subjective test.

Place of Performance

The tasks may be conducted at any location.

Period of Performance

The datasets shall be delivered 1 year after contract award.

This period of performance can be extended up to 1 year if the Contractor encounters difficulties (e.g., related to the pandemic) or if a scheduled event must occur more than 1 year after contract award (e.g., conference presentation).