SON - New Video Content ITU Testing.pdf

Scope of Need (SON) U.S. Department of Commerce

National Telecommunications and Information Administration (NTIA) Institute for Telecommunication Sciences (ITS)

Validation Test Footage Specifications

Background

The International Telecommunications Union (ITU), which gets its charter from the United Nations (UN), is performing validation tests of no reference (NR) metrics that assess the quality of a video stream using only pixel values, resolution, and frame rate. Despite 30 years of research into NR video metrics, no such metric has been proven accurate enough for commercial deployment. U.S. industry is very interested in this type of metric because it could enable automatic feedback loops in cameras and various video systems. The NR metric would allow the camera or video system to detect quality problems, understand why the quality is bad, and automatically fix the problem.

The U.S. Department of Commerce’s NTIA/ITS is participating in this validation test of NR metrics, nicknamed J.noref. We need a set of footage to be used in an experiment that will contain 400+ short videos. Those videos will show various responses of modern video cameras, to scenes and filming scenarios chosen to solicit different responses from the cameras. After the purchase is produced, NTIA will conduct an experiment where a panel of people will assess the quality of the videos. The NR metrics then try to estimate the quality of each video. The videos will be shared with other researchers around the world. The videos will also be shared for research and development (R&D) purposes only (no commercial applications) on the Consumer Digital Video Library (CDVL, www.cdvl.org).

This purchase is inspired by the 2015 study, Impact of Camera Pixel Count and Monitor Resolution Perceptual Image Quality (Conference Paper) - ITS, performed by NTIA/ITS, Intel, and the University of Ghent. That study compares the performance of 23 digital cameras, each used to photograph the same scene. Figures 1, 2, and 3 show the types of scenes selected and the range of camera impairments that were produced. The full set of images can be viewed by creating a free account on CDVL, logging in, and then searching for “CCRIQ”.

The goal is to compare the quality response of different cameras in responses to major variables (e.g., scene complexity, camera motion, in-view motion, lighting, and focal distance).

Basic Requirements

• Minimum of 400 unique videos, each 5 second duration (edited or as filmed)

• Photograph the same scenarios with a set of modern digital cameras

• Use a diverse set of 16 cameras

• Record 25+ scenarios with each camera

• Cameras are typically handheld and used one after the other (i.e., some variations will result)

• Tripods may be used for professional cameras and DSLRs

• Handheld photography is preferred if few people use tripods (e.g., cell phones)

• Some scenarios will intentionally include camera impairments (e.g., low light, too quick pan)

• We would appreciate good aesthetics where possible

• 1920 x 1080 resolution

• Landscape orientation

• Frame rates from 24 to 60 fps (as appropriate for the camera)

• Standard dynamic range (not HDR)

• Sound is optional (the validation test will be visual only)

• Each 5 second scene must be filmed continuously (no scene cuts) http://www.cdvl.org/ https://its.ntia.gov/publications/details?pub=2820 https://its.ntia.gov/publications/details?pub=2820 https://its.ntia.gov/publications/details?pub=2820

Choice of Cameras

We would like to take advantage of the camera DXOMARK rankings to make sure the cameras have different quality responses, from good to bad. See https://www.dxomark.com/ tab “rankings.”

• The set of 16 cameras must include:

o Two DSLRs o One compact / point-and-shoot o Two action cameras or bodycams, including one GoPro o Two spy cameras (e.g., nanny cam) o One dash cam (installed by the driver, not for law enforcement) o Six smartphones

Mostly phones released recently (2023 to 2025) Ratings spread from high to low rankings on https://www.dxomark.com/smartphones/ o One tablet, preferably iPad o One professional camcorder (the highest quality you own)

• The deliverables are the footage and rights to use and distribute the footage.

• Some changes to the above list of cameras can be made, if agreed by NTIA/ITS and the vendor

(e.g., using the vendor’s expertise)

Scenario Characteristics

The vendor will choose a set of 25+ scenarios to film by each camera with input from NTIA/ITS. The goal is that the scenarios show different situations where cameras are used, and the different responses from the cameras (some better, some worse).

The scenario characteristics are to be selected for specific characteristics, with the intention of stressing the cameras in different ways. Since each camera is used one after another, we expect larger differences among the recordings from the cameras than in the 2015 experiment (e.g., when filming a dog running in a park the dog’s position will change, scenes will be framed slightly differently and pan speeds will not be identical). It is allowed for the scenario to change slightly among cameras (e.g., film a different portion of a longer dance, stage magician demonstrates a variety of tricks).

The videographer should use their best judgement when operating the high-end cameras (e.g., settings for DSLRs and professional camcorders). The other cameras should be used in their default configuration (automatic mode) and handheld.

The vendor may want to consult https://www.dxomark.com/ for insights into types of scenarios that will make the cameras respond in different ways. The following scene elements and scenarios are of interest. Some scenarios may not be possible on all cameras.

• Skin tones, including a diversity of genders, ages, and skin colors / genetic background

• Indoors and outdoors, including landscape with distant objects

• Various lighting levels and lighting types (e.g., sun shining into the lens, full sun, partial shade, backlit, low light, night, artificial light, streetlights, and professional lighting)

• Various focal distances and focal depth, including small objects very near the lens (e.g., ants moving) to very long (e.g., landscape)

• Different camera motions including pans with various speeds and moving forward or backward

• Various camera zooms, including zoom past the optical zoom limit, static zoom, and dynamic zoom of different speeds

• Jiggling or bouncing picture (e.g., handheld camera)

• Follow a fast-moving object (e.g., car driving on the road) https://www.dxomark.com/ https://www.dxomark.com/smartphones/ https://www.dxomark.com/

• A flat object (e.g., painting or diagram)

• Different amounts of motion in the scene o Low motion = most of the pixels stay the same from frame to frame o High motion = most of the pixels change dramatically from frame to frame, and the scene contains a huge variety of colors

• Very fast motion, object moves across the screen in ≈1 second (e.g., boxing)

• Different amounts of details in the scene (little to lots)

• Different amounts of color in the scene (nearly black/white to lots of vibrant colors)

• Different stabilization conditions from no stabilization used up to external mechanical stabilization.

Note that high motion scenarios are surprisingly hard to create. The most famous high motion scene used for our research shows a race with maybe 100 people running from far away / left to close / right. Sports, races, and dancing tend to be high motion, especially if combined with camera motion.

The following unusual elements are of interest:

• Repetitious or indistinguishable fine detail (e.g., gravel, grass, hair, rug, pinstripes)

• Sharp black/white edges

• Blurred background, with an in-focus foreground

• Entire picture is intended to be in-focus

• Motion blur

• Night or dimly lit scene (camera noise, noticeable white level problems)

• Too bright lighting (potentially washing out colors, noticeable black level problems)

• Ramped color (e.g., sunset)

• Water, fire, or smoke (for unusual shapes and shifting patterns)

• Flashing lights or other extremely fast events

• Action in a small portion of the total picture

• Colorful scene

• Small amounts of analog noise (e.g., camera gain from dim lighting)

• Multiple objects moving in a random, unpredictable manner

• Visually simple imagery (e.g., black birds flying across a blue sky)

• Very saturated colors

• Many straight edges

• A person with pierced tongue, tattoos, or painted fingernails o Note: American Sign Language interpreters told us that these may cause odd or distracting camera effects, as the mouth or tattoo or fingernails move

• Something behind glass (e.g., painting or display behind a window), for the reflections on the glass

• Something that has only shades of black, white, and grey (e.g., ink drawing)

Where possible, each scenario should be recorded on all cameras. Deviations are allowed. We recognize that some interesting scenarios might not be possible on a few cameras (e.g., the zoom option is not available). Or a camera might break, as occurred during the 2015 study.

Subject Matter

The following subject matter are of particular interest. Many of these have been particularly useful for video quality research. This list is “nice to have” not “need to have” due to potential issues with fees, licensing, and availability.

• Amateur sports (e.g., tennis, martial arts, boxing, soccer, baseball, volleyball)

• Action sports (i.e., the person holding the camera is moving quickly)

• Crowd of people (e.g., busy sidewalks, stadium stands, people exiting an event)

• Many people running

• Racing (e.g., horses, cars, bicycles, dogs, swimming, motorcycles)

• Dancing (e.g., ballet, street dancing)

• The look and feel of a segment from a music video or concert

• DJ with light show in front a large pixel wall showing new age digital art or animations

• Tiny objects moving erratically (e.g., confetti falling, sparklers, fireworks)

• Physical comedy with brightly colored costumes (e.g., stage magic, mime, slapstick, juggling)

• Someone conversing fluently in American Sign Language (ASL)

• Look and feel of a nature documentary scene (e.g., trees, leaves, moving water, animals with high detail)

• Close-up macro shot of insect(s)

• Scenarios inspired by the most popular short videos (e.g., Instagram, TickTok)

We would like enough visual interest and variety among subject matter to keep people entertained. Where possible, the subject should be interesting and visually pleasing. The NR metrics cannot assess aesthetics. Good aesthetic elements (including framing) will help keep people entertained as they watch the videos and rate their quality.

Licensing Terms

NTIA/ITS must have rights to distribute all delivered footage freely on the Consumer Digital Video Library (CDVL, www.cdvl.org) for R&D purposes by NTIA/ITS (no commercial applications) without time limit.

Register on CDVL to see the licensing terms. The website provides a license clarification in plain English.

Video Editing and File Names

If video recordings must be edited after the original filming (which is likely), then the transcoded bitrate will be high enough to not add new impairments, and the file saved in mp4 format.

The videos do not need to be exactly 5 seconds. Small variations in duration are okay (a few frames), especially if this makes it possible to retain the original recording from the camera instead of editing afterward.

The 5 second video files must be named with a clear and simple naming convention that indicates the scenario, camera, and camera type. Cameras will be assigned letters (A, B, C) for the purpose of obscuring camera details when the dataset is distributed on CDVL. The scenario names should be short (e.g., 8 to 15 letters). Use underlines only to separate the scenario, camera, and camera type. Examples:

• WinterPeaks_T_compact

• Aspen_A_tablet

NTIA/ITS will be given a file that includes:

• The association of letters to actual cameras (e.g., make, model, and year)

• A short description of each camera, that includes the typical recording bitrate and whether additional devices were used (e.g., tripod)

• A short description of each scenario that includes interesting design elements

Number of Scenarios and Cameras

The vendor will communicate with NTIA/ITS on the selection of cameras and scenarios, to ensure a diverse range of expected quality.

Extra video files are desirable but not required.

http://www.cdvl.org/

The minimum number of video files includes a small buffer that considers the likely number of video files that the ITU is likely to discard, based on tests over the past 25 years. These discards will only occur if someone involved in the validation test believes a visual impairment in the video file is inappropriate for the validation test. Neither ITS nor the vendor can anticipate this, so the vendor should not be too concerned about this issue.