Amendment 03_Attachment 02_Attachment 04_OPS Scan Annotation Protocol_v4.1.pdf

On-Person Screening (OPS) Scan Annotation Protocol Version 4.1

Introduction As TSA pursues development of automatic target recognition (ATR) algorithms for future On- Person Screening (OPS) technology, a critical prerequisite is collecting and curating a high-quality dataset to train those algorithms.

The AIT ATR process requires that data-labeling address at least 3 issues dealing with ATR detection and output accuracy:

1. Identifying the location of a target on the body in a raw image.

2. Collecting meta-data about the target (e.g. gun, knife, explosive).

3. Translating the location of a target from the raw image to a representative human figure

(RHF).

Additionally, any labeling protocol must also provide a systematic, repeatable method such that any properly trained annotator can create high-quality annotations.

Keypoint Detection The intent of “keypoint” detection is to annotate points on the body that indicate joints. The objective of keypoint algorithms is to identify joints or connection points in an image, so a ‘stick figure’ can be made from their connection. These stick-figures allow a coordinate transform function (CTF) to translate the location of a bounding box on a raw image to an RHF by calculating the relative distance between joints on the raw image and RHF using these keypoints.

To maintain consistency, entries in the in the annotations metadata sheet (see Metadata Entry) after each keypoint annotation must be made.

Labeling Keypoints

In the dataset, the following keypoints must be labeled:

Joints: (shoulders, elbows, ankles, wrists, knees, hips) should be approximated as the center of the joint, regardless of the angle of the scan.

Head: Parts of the head and face (ears, eyes, nose) should be labeled only if visible, and treated as the center of the body part.

The location of a keypoint should only be proposed if it is identifiable in the image.

Class List (17)

Left Shoulder Left Ankle Left Hip Left Ear

Right Shoulder Right Ankle Right Hip Right Ear

Left Elbow Left Wrist Left Knee Left Eye

Right Elbow Right Wrist Right Knee Right Eye

Nose

Object Detection Object detection models identify a region within an image that contains a specific type or class of object. To maintain consistency, entries in the in the annotations metadata sheet (see Metadata Entry) after each keypoint annotation must be made.

Labeling Body Regions

In the dataset, the following regions must be labeled:

• Arms: The border of the arms can be thought of as the extent that would be exposed by wearing a tank top. Hands and fingers should be approximated as if the individual were wearing mittens (e.g. do not delineate individual fingers).

• Legs: The upper border of the legs should start where a tight fitting undergarment would end. This can be approximated in the front and rear view scans by the location of the hip flexors.

• Head: The head should be thought of as the extent that would be exposed by wearing a crew neck t-shirt. This includes the neck.

• Torso: The torso can be bounded at the neck and arms, and include the midsection down to a horizontal bound that crosses the belly button.

• Upper sensitive: The upper sensitive label overlaps the torso on the front and back of the body only. The label should encompass the breasts and can be approximated by the shape of a sports bra or athletic support for the upper chest.

• Lower sensitive: The lower sensitive label does not overlap the torso or legs. It can be described as the area left unlabeled by the legs and torso, capturing the midriff below the belly button, and ending at the hip flexors.

Other images from the same scan should be used to help with annotations, but the location of a body region should only be proposed if it is identifiable in the image.

Class List (8)

Labeling Targets: Using Polyline Annotations

To create the most accurate polyline annotation possible, an external image editing software to manipulate the brightness and contrast of the image may be needed.

Each image with a target has scan metadata associated with it, which identifies the target item and location. The target item will be labeled with the ‘THREAT_OBJ’ class label.

Other images from the same scan should be used to help with annotations, but only the area of a target that is identifiable in the image should be proposed.

Labeling & Annotation Procedure

1) Open the image labeling software and associated datasets.

Left Arm Left Leg Head Upper Sensitive

Right Arm Right Leg Torso Lower Sensitive

a. Image metadata sheet – spreadsheet containing fields to populate with additional data associated with each labeled object in each scan.

b. Ground truth (GT) datasheet – spreadsheet with each scan’s GT metadata, including threat type and threat location.

c. Reference scan supplement – a slide deck with example annotations for ideal and edge case scenarios for each class of object to be labeled.

d. Image editing software – software used to manipulate image contrast / brightness for difficult to see images, (e.g. ImageJ, Gimp, Photoshop)

2) Annotate all of the images for a single class of label (e.g. body part, threat, keypoint) rather than annotating a single image for all of the labels. Focusing on a single class at a time will help maintain consistency. For body part and threat annotations, use polylines to delineate the objects rather than bounding boxes.

3) Start with the keypoint class, which is a good way to get familiarized with the data. Step through each image and assign labels to the parts of the body according to the reference scan supplement.

4) Labeling all body parts for all of the images, which is a good way to get familiarized with poly line annotations and become comfortable using external applications to adjust the contrast / brightness of the scans for difficult regions.

5) Annotate the targets. These classes can be challenging, so it’s important to have experience using the polyline annotation tool, including adjusting the brightness and contrast in adjacent software.

Annotation Guidance

Other images from the same scan should be used to help with annotations, but only visible portions of the body part or target object should be included in the polyline mask. If an object appears to be split (no data visible through a portion of the object), multiple instances of that object should be created to bound what is visible. Do not label what is not present in the image.

To the maximum extent possible, every part of the scan that is visible should be covered with a label. For front and rear view scans, try to create tight margins between body parts (e.g., arms and torso), so no gaps exist. Overlaps between parts of the body may occur for off-angle scans, if the overlapping areas can be visually resolved.

Ensure that left vs right body parts are relative to the individual in the scan, meaning that understanding the orientation of the scanned individual is requisite for labeling body parts correctly.

Off-Axis Considerations

Off-axis images pose additional challenges – the primary rule of thumb is to only label what can be seen. If objects overlap and elements of both can be seen, label both. Where objects are incomplete, or split into multiple pieces, create multiple instances of the label to reflect that.

Missing or low intensity parts of the image may also make labeling contiguous regions impossible. Maintain consistency when determining when there is too little information to label. Consider using tools that enhance image brightness or contrast to ensure that all available image data is leveraged when labeling, especially in off-axis views.

Special case - the upper-sensitive region

Because the upper sensitive region may be challenging to label consistently in all views, it is described as a lateral cross section of the chest that contains the breasts, consequently including regions of the arm pits and upper back.

Metadata Entry

Each annotation project should be associated with an annotations metadata spreadsheet. The spreadsheet includes all of the information about each image and annotation that may be relevant for downstream algorithm development. The metadata sheet must have the following fields:

• Filename: The name of the file being annotated.

• Label Date: The date that the specific annotation was generated

• Labeler ID: Unique identifier of the annotator (can be hashed for PII)

• Model: Form controlled, either:

o object detection (OD) o Keypoint (KP).

• Label: The label name being annotated. These should be form controlled.

• GT Threat: The specific threat ID being annotated, should match the associated GT spreadsheet and be form controlled.

• Label Location: The location of the body where the annotation is located. Only applies to target or unknown labels.

• QA Flag: Contains a form controlled identifier that describes the quality of the annotation.

o Clear – no issues with annotation.

o Difficult – Annotation was possible but there were irregularities, either due to lack of image data or noise, etc.

o Impossible – unable to complete the labeling process due to either image or metadata challenges.

• Notes: Used to explain of any issues associated with the annotation. If the QA flag is not

‘Clear’, the Notes field must be used to explain why.

Filename Label Date

Labeler

ID

Model Label GT Threat

Label Location

QA Flag Notes

17.png Apr-04-

05 OD Left Arm NA NA Clear None

17.png Apr-04-

05 OD Threat_1 Solids Rear Torso Clear None

17.Png Apr-04-

05 OD Unknown NA Left Wrist Difficult Looks like a watch, difficult to capture margin against the wrist.

17.Png Apr-05-

06 KP Left Wrist NA Left Wrist Clear None