Public_ST_Test_Plan_Version_01132016.pdf

RFP Attachment F

TSA Pre✓® Application Expansion Test and Evaluation Plan

January 13, 2016 ii

This page intentionally left blank.

iii

Signature Page

Mike Turner Deputy Director Program Management Division Office of Intelligence and Analysis Transportation Security Administration Department of Homeland Security

(571) 227-2388 Michael.Turner2@tsa.dhs.gov

Steve Dennis Technical Director Science & Technology Directorate Homeland Security Advanced Research Programs Agency Department of Homeland Security

(202) 254-5788 stephen.dennis@hq.dhs.gov mailto:Michael.Turner2@tsa.dhs.gov mailto:stephen.dennis@hq.dhs.gov iv

Table of Contents 1 Introduction

2 Purpose

3 Definitions

4 Test and Evaluation Methodology

5 Test and Evaluation Milestones and Data

6 Test Design

6.1 Scope

6.2 Test Design Assumptions

6.3 Test Design Constraints

6.4 Test Design Description

6.4.1 Overview

6.4.2 Identity Assurance Testing (Pass/Fail)

6.4.3 Criminality Screening Testing (Pass/Fail)

Appendix A DHS S&T DA-E Support .................................................................................. A-1

Appendix B DHS S&T Review (of Volume IV Written Proposals) (applies to post-award) B-2

Appendix C Protecting Critical Data and Information .......................................................... C-3

Appendix D Communications................................................................................................ D-1

Appendix E Generating Variations ........................................................................................ E-1

Appendix F References .......................................................................................................... F-1

Appendix G Abbreviations and Acronyms ............................................................................ G-1

Appendix H Country Codes ................................................................................................... H-1 v

List of Figures Figure 1: Test and Evaluation Steps within the RFP Process vi

List of Tables Table 4-1: TSA Criteria Metrics and Description

Table 5-1: TSA Milestones and Descriptions

Table 6-1: Test Dataset Fields

Table 6-2: Offeror Identity Assurance Response Fields

Table 6-3: TSA Identity Assurance Metrics and Descriptions

Table 6-4: Offeror Criminality Screening Response Fields

Table 6-5: TSA Criminality Screening Metrics and Descriptions vii

1 Introduction The Transportation Security Administration (TSA) developed a Request for Proposal (RFP) (HSTS02-16-R-OIA011), which it released in October 2015, seeking vendors for the TSA Pre✓® Application Expansion initiative to deploy ready-to-market application capabilities to expand the public’s enrollment opportunities for TSA Pre✓® and provide the ability for TSA to identify known travelers. Successful Offerors will receive an Other Transaction Agreement (OTA) that will allow these approved contractors to market, enroll and pre-screen individuals for TSA Pre✓® eligibility. TSA will then make the final eligibility determination for actual TSA Pre✓® enrollment upon completion of a Security Threat Assessment (STA) process.

The Department of Homeland Security (DHS) Science & Technology (S&T) Data Analytics – Engine (DA-E) will conduct a Test and Evaluation of Offerors’ Identity Assurance and Criminal History Records Check (CHRC) solutions. This Test and Evaluation will inform the source selection team on whether an Offeror’s solution passes/fails the Identity Assurance and CHRC requirements.

2 Purpose This Test and Evaluation Plan provides guidance to Test and Evaluation participants on the pass/fail test design and schedule to assist TSA in making a decision to award OTAs to selected Offerors.

3 Definitions The key terms below will assist the reader’s understanding of the Test and Evaluation Plan.

1.) Offeror. An offeror is a private industry company who responded to the RFP and has been chosen to participate in the Test and Evaluation.

2.) Private Data Broker. A private data broker is a commercial company whose business model contains the collection, maintenance, and selling of individual’s data (these data brokers could be referred to as subcontractors during the acquisition if they are supporting an Offeror).

3.) Verified Identity. A verified identity matches the identity of an actual living person containing the present legal name, current address, and accurate date of birth for a US citizen, US national, or Legal Permanent Resident (LPR).

4.) Data field, data attribute, data feature: For the purposes of this assessment plan, these are synonymous terms for the additional data record information associated with a record index or identification.

5.) Biographic. A biographic is a data field on the TSA Pre✓® application; the data fields include such items as, last name, first name, address, telephone number, etc.

6.) Biometric. A biometric is the unique personal identifier submitted with the TSA Pre✓® application; currently all 10 fingerprints. Applicant biometrics are essential to confirming identity and cross-referencing criminal data bases for disqualifying offenses.

7.) Ground Truth Test Dataset. The ground truth test dataset consists of both biographic and limited biometric data from recent applicants to the TSA Pre✓® program. This ground truth dataset will remain with DA-E; Offerors will be presented only with a subset (number of features and, number of applicants, Federal Bureau of Investigations (FBI) rap sheets) of this data, to perform their identity and criminal screening solutions and submit their results for evaluation.

8.) FBI-Approved Channelers. FBI-Approved Channelers are companies who can submit requests for the FBI to conduct a criminal background check on an individual. The criminal background checks are submitted with both biographic and biometric data for the individual and checked against the Department of Justice (DOJ) Criminal Justice Information Services (CJIS) database.

4 Test and Evaluation Methodology Only Offerors who pass the Independent Verification and Validation (IV&V) process will proceed to DHS S&T Test and Evaluation phase. If the Offeror’s information technology and operating systems (including an Offeror’s subcontractor’s information technology and operating systems that will process the Government test data) do not receive a “Pass” rating under the Independent IV&V process, then the Offeror will not proceed to the DHS S&T Testing phase and will not receive any Government test data. See RFP Attachment D for further details regarding the IV&V process.

The DHS S&T DA-E office will evaluate the Offerors’ solutions used for digital identity assurance and criminality screening. The pass/fail test will use a digital test dataset sent securely to the Offerors to quantitative assess how the Offerors will meet TSA established identity and criminal screening criteria. Offerors will evaluate the individual records within this test dataset for verifiable identities and then for criminal disqualifying offenses as specified in the TSA Pre✓®

Disqualification List, parts A, B, and C (available at http://www.tsa.gov/Disqualifying- Offenses-Factors).

A test dataset will not be provided to the Offeror until they have substantiated their ability, in addition to any of their subcontractors or data sources, to provide information security of the TSA sensitive test data and purge all government dataset records after the completion of the S&T evaluation, or as directed by TSA. Further, a test dataset will not be provided to the Offeror until they have documented the privacy protections in place with any subcontractors or data sources to ensure that individuals with names used in test dataset do not receive any commercial notifications (e.g. credit check inquiries) as a result of the test.

The test dataset will contain verifiable/non-verifiable identities and criminals/non-criminals individual records. The test dataset comes from an “authoritative” government source and individual records will have associated fields/attributes which could include but are not limited to, record ID, full name, gender, date of birth, address, etc.. Offerors will be expected to return results for the ID assurance digital testing within a 24 hour time-frame. The Offeror can use any public, commercial, and private data sources as appropriate for identity assurance. The criminality screening results will be expected to be returned to TSA within two weeks (14 consecutive days). In addition to using public, commercial, and private data sources for criminality screening the Offeror can use FBI-Approved Channelers or publicly available services. Extensions will not be allowed for criminality screening results from a service provider that takes longer than 2 weeks (14 consecutive days).

Identity assurance and criminality screening results will be scored in accordance with the RFP (HSTS02-16-R-OIA011), Version 2.1 December 29, 2015 pass/fail criteria and metrics, depicted below.

http://www.tsa.gov/Disqualifying-Offenses-Factors http://www.tsa.gov/Disqualifying-Offenses-Factors

Table 4-1: TSA Criteria Metrics and Description

Metrics Description

1.) Greater than or equal to (>=) 99% of the identities considered “not verifiable” are correctly documented in the Offeror’s test results as unverifiable identities

2.) Greater than or equal to (>=) 98% of the identities considered valid/verifiable by the Government are correctly documented in the Offeror’s test results as verifiable identities

3.) Greater than or equal to (>=) 99% of the individuals criminally disqualified by the Government are correctly documented in the Offeror’s test results as disqualified

4.) Greater than or equal to (>=) 98% of the individuals approved by the Government are correctly documented in the Offeror’s test results as approved/recommended*

*Explanatory note regarding metric #4:

• Example 1: Offeror disqualifies an individual who has a criminal record, but none of the crimes are disqualifying based on TSA’s requirements (e.g. Driving Under the Influence (DUI) conviction is not a disqualifier per TSA requirements). This erroneous disqualification would count against the Offeror in this metric.

• Example 2: The Offeror identifies an individual as disqualified based on the belief that a disqualifying crime conviction exists (e.g. murder), but the Offeror’s criminal history check logic/process was incorrect. This erroneous disqualification would count against the Offeror in this metric.

• Example 3: The Offeror identifies an individual as disqualified based on the belief that a disqualifying crime conviction exists (e.g. murder), AND the Offeror’s criminal history check logic/process was accurate based on DHS S&T DA-E’s analysis of the results, then these records will not be counted against the Offeror for this metric. In this case, the Offeror identified a disqualified individual that DHS did not know existed in the test dataset, but did so accurately therefore does not count against the Offeror. For those individuals who are criminally disqualified by the Offeror, the Offeror shall be responsible for providing the disqualification reason(s) in accordance with the TSA Pre✓® Disqualification List (Parts A, B, and C) as part of the test results.

DHS S&T DA-E will answer questions regarding the evaluation, and work collaboratively with the Offerors during the evaluation process. The following sections of this evaluation plan will explain in more detail the evaluation milestones, test design, assumptions, constrains, and data exchange procedures.

5 Test and Evaluation Milestones and Data The below table illustrates the anticipated milestones for the Test and Evaluation events.

Table 5-1: TSA Milestones and Descriptions

DHS S&T DA-E Milestones Date

1.) Begin identity assurance and criminality screening testing

TBD (completion of IV&V Phase) + 10 days

2.) Complete identity assurance and criminality screening testing

TBD (completion of IV&V Phase) + 24 days

3.) Provide pass/fail results for identity assurance and criminality screening test to

TSA

TBD (completion of IV&V Phase) + 25 days

6 Test Design

6.1 Scope

The focus of the Test and Evaluation is to test the two distinct areas Offerors must perform to complete the initial, front-end pre-screening (identity assurance and a CHRC) of applicants for TSA Pre✓® eligibility.

Concretely, this means that DHS S&T DA-E will provide technical advice and test results:

• Technical advice to TSA of the Offerors’ end-to-end solution as presented to TSA in written and oral presentations and during the quantitative (pass/fail) test component

• Quantitative testing (pass/fail) of the front-end pre-screening process includes:

o Identity Assurance: Given a test dataset and a finite number of applicant individual record attributes, evaluate an Offeror’s ability to verify a specific digital identity o Criminal Disqualifiers: Given an applicant’s identity, evaluate an Offeror’s ability to determine if the applicant has one or more disqualifying criminal offenses

6.2 Test Design Assumptions

1.) The Offerors can securely receive, evaluate, and securely return test dataset results to DHS

S&T DA-E

2.) The Offerors can verify an identity using only commercial, public, and private data sources not using physical documentation, or have the person present for an interview 3.) For the criminality testing, the Offerors have the capability to read and fully understand FBI rap sheets (which will be provided for selected digital records) 4.) Testing dataset closely replicates the population of TSA Pre✓® applicants, before the addition of false identities, criminal populations, deceased individuals, etc. into the data for quantitative test purposes

5.) The Offerors will enter into a CRADA with DHS S&T DA-E

6.3 Test Design Constraints

1.) Offerors will return the identification assurance results within 24 hours 2.) Offerors will return the criminality screening results within 2 weeks (14 consecutive days) 3.) DHS S&T DA-E will be able to begin the quantitative (pass/fail) testing in accordance with the RFP pass/fail criteria within 10 days from TSA notification 4.) The evaluation is limited to use of DHS TSA-approved additional commercial, public, or private data attributes (based on guidance in the RFP)

6.4 Test Design Description

6.4.1 Overview

Upon selection by TSA to advance to the identity assurance and criminality screening testing component, Offerors will coordinate with DHS S&T DA-E to receive a test in-brief. This in-brief will provide the overall test methodology; criteria and metrics; timelines; how the test dataset will be exchanged; and data security guidelines. One (1) test dataset will be exchanged with the Offerors for the identity assurance and criminality testing. Specific data exchange formats and results exchange requirements are explained below.

Figure 1 depicts the Test and Evaluation steps within the RFP process.

Figure 1: Test and Evaluation Steps within the RFP Process

6.4.2 Identity Assurance Testing (Pass/Fail)

Given a test dataset, the Offeror will verify (or disqualify) the applicants digital record. For each applicant’s digital record the Offeror will use their solution to verify the digital ID. The Offeror will determine if the ‘identity score’ exceeds their threshold score for a verified ID result. The Offeror is expected to determine a confidence score based on the digital footprint from their available data sources.

The evaluation team envisions several scenarios, including:

1.) An identity is quite unique in the digital domain leading to a single footprint with high confidence.

2.) An identity is not completely unique and hence a determination is required to either verify with lower confidence or reject this particular applicant.

3.) An identity is really quite common or so close to several competing identities that it cannot be verified and hence must be rejected.

4.) An identity cannot be isolated within privately available data with any confidence at all or the applicant is deceased leading to a clear rejection outcome.

At the start of the evaluation period, Offerors will be provided a test dataset containing several biographic fields about the applicants. Offerors will have 24 hours to return the results of the identity assurance portion of the pre-screening methodology.

The test dataset will be exchanged with the Offerors via Secure File Transfer Protocol (SFTP) of a CSV file with the following data fields defined in the below Table 6-1.

Table 6-1: Test Dataset Fields

Data Type of Field

- Length Field Name Description

Biographic Dataset

Number App ID Value that is unique to the applicant

Person Name Applicant's Primary Name

VARCHAR2 – 40 Person First Name First Name

VARCHAR2 - 40 Person Middle Name Middle Name

VARCHAR2 - 40 Person Last Name Last Name

NUMBER Gender (Person Sex Code) Gender

DATE Person Birth Date Date of birth

Preferred Communication Method Applicant's preferred method of communication

VARCHAR2 - 50 Email Applicable contact information (email)

Phone Applicable contact information (phone)

VARCHAR2 - 20 WORK_PHONE Work phone number

VARCHAR2 - 5 WORK_EXT Work phone number extension

NUMBER WORK_DAYNIGHT Preferred work phone number

VARCHAR2 - 20 HOME_PHONE Home phone number

VARCHAR2 - 5 HOME_EXT Home phone number extension

NUMBER HOME_DAYNIGHT Preferred home phone number

Place of Birth Applicant's birth city

Char - 2 POB_COUNTRY (Country of Birth) Country

Char - 2 POB_STATE State

VARCHAR2 - 20 POB_COUNTY Zip

VARCHAR2 - 30 POB_CITY City

Char - 2 OTHER_POB_COUNTRY (Country of Birth) Country

VARCHAR2 - 2 Citzenship_Country_CD Country of citizenship

Address Address

Address Type Type of Address

VARCHAR2 - 40 Street 1 Street address1

VARCHAR2 - 40 Street 2 Street address2

VARCHAR2 - 30 City City

VARCHAR2 - 30 County County

CHAR - 2 State State

VARCHAR2 - 16 Zip Zip

CHAR - 2 Country Country

VARCHAR2 - 30 Other_Country Other Country

Date FROM_DATE From Date

Date TO_DATE To Date

Rap Sheet Dataset (Only if Offerors propose an FBI-based CHRC)

NUMBER APP_ID Value that is unique to the applicant

CLOB FULLSHEET Rap sheet with redacted SSN

Offerors will transmit their identity assurance results using the same SFTP process. Note the dataset key is the unique data record field and only the resultant 1 (ID verified) or 0 (ID not verified) needs to be returned.

Table 6-2: Offeror Identity Assurance Response Fields

Data Record

Identity Status

Numeric Binary 1 – Valid

0 - InValid

Scoring of Offeror results will be pass/fail and measured against the two identity assurance metrics.

Table 6-3: TSA Identity Assurance Metrics and Descriptions

Metrics Description

1.) Greater than or equal to (>=) 99% of the identities considered “not verifiable” are correctly documented in the Offeror’s test results as unverifiable identities

2.) Greater than or equal to (>=) 98% of the identities considered valid/verifiable by the Government are correctly documented in the Offeror’s test results as verifiable identities

The successful Offeror will have correctly verified 98% or more of the true applicant identities and correctly found 99% or more of the false applicant identities.

6.4.3 Criminality Screening Testing (Pass/Fail)

Given the same test dataset, the Offeror will determine all the criminal disqualifying offenses (or lack thereof) for each applicant in the data set. For each applicant, the Offeror will use their solution to determine if this applicant should be disqualified for past criminal offense(s) in their record. The Offeror will determine which (if any) offenses an applicant has committed and create a list of those offenses. With the understanding that upon finding any of the TSA disqualifying offenses, that the applicant would then be disqualified. However for evaluation purposes, the evaluation team seeks to assess each Offeror for completeness; thus, the generated list of offenses for each applicant will assist in understanding the complete picture of the applicant’s digital record.

Certain digital records in the test dataset will have a FBI rap sheet appended and the Offerors can use this information to assist in making an “approval” or “denial” recommendation.

The evaluation team envisions each applicant falling into one of three cases:

- Applicant has not committed any disqualifying offense

- Applicant has committed one disqualifying offense (list the offense)

- Applicant has committed more than one disqualifying offense (list all offenses)

The Offeror will provide the evaluation team with the results of the disqualifying offense search and by association, the resultant accept or rejection Offeror decision for each applicant in the data set.

Offerors have two (2) weeks (14 consecutive days) to complete the criminality screening for the applicants in the test dataset and return the results, recommending “approval” or “denial”.

Using the same SFT protocol, Offerors will provide the results of their criminality screening in the following format.

Table 6-4: Offeror Criminality Screening Response Fields

Data Record

Criminal Disqualifications

All TSA Pre✓® Disqualifications

Jurisdiction Statute Disqualification Offense Reference

Numeric 1 – Disqualifications 0 – No Disqualifications

Text

None or List Each Disqualification

Offense (In accordance with TSA Pre✓® Part

Text

Federal, State, Local

Alpha Numeric

Violated Statute (e.g. specific line and verse in the

United States Code (USC), Alpha Numeric

Documentation Sufficient for Offeror Determination (e.g.

source of evidence for disqualifying crime

Data Record

Criminal Disqualifications

All TSA Pre✓® Disqualifications

Jurisdiction Statute Disqualification Offense Reference

A, B, and C disqualifying offenses)

State Law, or Local Law violation) data, USC, law, date, points of contact, etc.)

Scoring of Offeror results will be pass/fail and measured against the two criminality screening metrics.

Table 6-5: TSA Criminality Screening Metrics and Descriptions

Metrics Description

3.) Greater than or equal to (>=) 99% of the individuals criminally disqualified by the Government are correctly documented in the Offeror’s test results as disqualified

4.) Greater than or equal to (>=) 98% of the individuals approved by the Government are correctly documented in the Offeror’s test results as approved/recommended*

The successful Offeror will have correctly identified and recommended approval for 98% or more of those applicants without criminal disqualifications and identified and recommended denial for 99% or more those applicants found to have one or more criminal disqualifications.

A-1

Appendix A DHS S&T DA-E Support Representatives from TSA and DHS S&T DA-E will be present in their advisory role at all Offeror oral presentations; participate in questioning proposed methodologies; and provide input to the TSA Program Manager. Specifically, DHS S&T DA-E representative(s) will attend the oral presentations and will support this process by providing technical evaluation expertise in regards to the Offeror’s proposed solution.

B-2

Appendix B DHS S&T Review (of Volume IV Written Proposals) (applies to post-award)

The quantitative assessment using the test dataset will not evaluate the following:

• Use of physical documentation to verify an identity

• Use of Knowledge Based Authentication (KBA) for on-line identity assurance

• Fraudulent document detection

• Verification of an identity by comparing a collected biometric to a presented biometric

Instead, and for post-award purposes only, DHS S&T DA-E will be able to review Offeror’s technological solutions (to include technologies used by data brokers to the extent possible based on business proprietary solutions) for overall design as described here in Appendix B. To facilitate this Appendix B process, and ensure the Offerors’ solutions are as transparent as possible to the TSA, the Offerors will be required to enter into a CRADA with DHS S&T. Offerors are expected to provide as much detail as possible regarding their identity assurance and criminality screening solutions.

Offerors will be required to answer technical or procedural questions posed by DHS S&T regarding design, information architecture, privacy, security, and data integrity and storage during the evaluation. DHS S&T will use the Volume IV information provided by Offerors to understand the algorithms used for identify assurance and criminal history checks, infrastructure and architecture framework, and system specifications as well as participate in meetings to discuss algorithm(s) and system(s) as required. Offerors should be prepared to provide additional information during the DHS S&T testing and evaluation phase. The Volume IV documentation shall serve solely for post-award informational purposes of the Offeror’s Technology and Technical Capabilities and not serve as part of the Pass/Fail RFP criteria for Element 6 or as any part of Contract Award selection. Specifically, Volume IV shall contain the following information:

• Detailed description of the systems, logical methodologies, and processes for identity assurance and criminal history records check.

• Explanation of proposed technical architectures and components for the proposed identity and criminal history records check system including capabilities for information security, privacy protection, and data storage specifically as to how they relate to DHS S&T testing.

• Use case examples that illustrate identity assurance and criminal history records check systems, logical methodologies, and processes (to include proposed technological, system, and data features and decision-making information flow).

C-3

Appendix C Protecting Critical Data and Information This appendix presents summarized information on data collection, storage, transmission, and safeguard processes. For detailed guidance refer to the CRADA (Disposition Plan).

In general, the DHS S&T will secure data by following the findings of the DHS Data Privacy & Integrity Advisory Committee’s report, “The Use of Commercial Data” (Report Number 2006- 03 Adopted December 6, 2006) and an agreement with DHS TSA, “016 Transportation Security Technology Testing System of Records.”

C.1 Data Requirements Data will include Personally Identifiable Information (PII) collected by means of purchase and direct collection of open and Government sources. DHS TSA will coordinate with DHS S&T to request, collect, and assemble evaluation data.

C.2 Data Restrictions Data will be handled in accordance with DHS S&T policies and procedures for safeguarding Sensitive Security Information (SSI) and PII. Refer to the CRADA for specific details. DHS S&T will provide for the storage of evaluation data sets for the duration of the effort until otherwise directed. Specific measures include:

• Identification of the individuals with access to evaluation data and search requests and results.

• Segregated storage of prescreening data on a protected network that is designed to house sensitive sponsor data.

• Limited access to prescreening data using two-factor authentication.

• Storage of backup copies of the data on approved, secure, and encrypted external hard drives. When not in use, external hard drives are disconnected from the unclassified network and stored in locked cabinets in access-controlled, secure areas.

All data stored on government equipment will be encrypted and password protected following conventions for the creation of strong passwords. DHS S&T provides training on security procedures and privacy controls to ensure proper handling of any PII. Equipment will be checked to confirm up-to-date security settings and antivirus software.

Evaluation data will be safeguarded in accordance with applicable rules and policies, including all applicable DHS automated systems security and access policies. Strict controls have been imposed to minimize the risk of compromising information that is being stored. Access to the computer system containing the records in this system is limited to those individuals who have a need to know the information for the performance of their official duties and who have appropriate clearances or permissions. Information in this system is protected from unauthorized access through appropriate administrative, physical, and technical safeguards. Unauthorized personnel are denied physical access to the location where records are stored. For computerized records, safeguards are in accordance with generally acceptable information security guidelines via use of security codes, passwords, personal identification numbers (PIN), and other similar safeguards.

C-4

The evaluation team members have reviewed the policy and applicable, published System of Record Notices (SORN) on the requirements and best practices for handling and managing PII and understand the responsible stewardship for handling data appropriately.

C.3 Data Transfer Protocol Data will be transmitted in accordance with DHS S&T policies and procedures for safeguarding SSI and PII. SFTP will be used to transfer files to Offerors for the evaluation, with written receipt of delivery and destruction required.

C.4 Data and Search Purging Evaluation data will be maintained in accordance with the National Archives and Records Administration-approved General Records Schedule 20 -1a (N1-GRS-95-2 item 1a), “Electronic files or records created solely to evaluate system performance, as well as hard copy printouts and related documentation for the electronic files/records.” These records will be deleted or destroyed when TSA and DHS S&T DA&E determines that they are no longer needed for administrative, legal, audit, or other operational purposes. Offerors will destroy all training and evaluation data sets upon completion of the Test and Evaluation.

Similarly, all data, search records, logs, etc. exchanged between an Offeror and any sub-contractor of private commercial data must be purged from the subcontractors’ systems as directed in the CRADA. Any person or company acting in concert with an Offeror for the TSA Pre✓® Application Expansion is expressly prohibited from maintaining any record whatsoever of the applicants or any portion of their data.

D-1

Appendix D Communications D.1 General Data sets containing PII will be electronically transferred securely via Secure File Transfer to participating Offerors and results by record number will be transferred to test personnel via non-secure email.

Formal written documentation of the receipt, handling, and destruction of evaluation data is required. Reference Appendix A

D.2 Test and Evaluation Data Exchange Times The test team will securely transfer the test data to each Offeror via electronic file transfer:

tentatively scheduled for the first day at 9:00AM of the DHS S&T Test and Evaluation period.

Offerors will email evaluation identification results, not later than 24 hours at 9:00AM, consisting of the fields record number and a binary (0, 1) indicating: identity validated or invalid to the test team in accordance with the format in Table 6-2.

Offerors will email evaluation criminality results, no later than two weeks later by Friday 1:00PM, consisting of the fields record number and a binary (0, 1) indicating: no criminal disqualifiers or criminal disqualifications present along with a listing of those criminal disqualifiers to the test team via format in Table 6-4.

All times indicated reflect Eastern Time (UTC-05:00).

D.3 Reporting Results A test report will be provided to the Contracting Officer.

E-1

Appendix E Generating Variations DHS S&T DA-E recognizes that errors exist in our nation’s databases and will seek to replicate some of the more common forms of these errors to evaluate an Offeror’s ability to resolve these common identity field variations.

E.1 Element and Structural Variations DHS S&T DA-E will introduce permutations to applicant data to simulate the data errors, arising from element variations: particles, short forms, and spelling variations as well as structural variations: additions and deletions, fielding variations, permutations, placeholders, and element segmentations that are commonly found in identity field data. Below are several examples of these element and structural variations:

• Francis Huger = Francis linger

• Mark = Marc, Katherine = Catherine, Smith = Smithe, Rahman = Rahamn

• Franklin = Frankln, Peter = Perer, Sam ≠ Pam, Tom ≠ Tim, Keller ≠ Teller

• Jennifer = Jen, Robert = Bob

• John Andrew Smith = J Andrew Smith, John Andrew Smith = John A Smith

• Carstanjen = Carstanj

• Abd al Rahman = Abdal Rahman

• Smith-Jones = Smith Jones

• De Los Angeles = Delosangeles

• McHenry = Mc Henry

• Maria Rodriguez De Gonzalez = Maria Rodriguez Gonzalez

• Maria del Carmen = Ma del Ca

• C = Charles

• Catherine = Catharine = Katherine = Kathryn

• Husayn = Hussein = Hossein

• Robert = Bob = Bobby

• Joseph = Giuseppe = Yusif

• John C Smith = John Smith

• Maria Gonzales Lopez = Maria Gonzales

• Kim JONG JR = Jong Jr KIM

• Clara Lucia Bolivar = Lucia Clara Bolivar

• Nur Mohammed Amin = Nur Mohammedamin

E.2 Identity Scores, Threshold, and Confidence Offerors will necessarily have to determine how their solution will resolve the above types of errors and their confidence in their resolution method.

F-1

Appendix F References For additional information, the following references are provided for clarification, background, and further research:

1. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011

2. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011, Amendment #0002

3. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011, Attachment 2, High Level Requirements

4. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011, Attachment A, Final RFP Version 2.1

5. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011, Attachment D, Independent Verification and Validation Process

6. Request for Proposals, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011, Attachment E, Cooperative Research and Development Agreement

7. Statement of Work, TSA Pre✓® Application Expansion, HSTS02-16-R-OIA011

8. TSA Pre✓® Disqualifying Offenses/Factors (source: http://www.tsa.gov/tsa-precheck/eligibility-requirements)

9. National Institute of Standards and Technology (NIST) Publication 800-63, Standards

10. Apaydin, E., 2010 Introduction to Machine Learning, chapter Design and Analysis of

Machine Learning Experiments. MIT Press, 475–515.

11. Batista, G. E. A. P. A., R. C. Prati, and M. C. Monard, 2004 A study of the behavior of several methods for balancing machine learning training data. Sigkdd Explorations 6: 20–29.

12. Giraud-Carrier, C., and M. H. Dunham, 2010 On the importance of sharing negative results.

Sigkdd Explorations 12: 3–4.

13. Guo, H., and H. L. Viktor, 2004 Learning from imbalanced data sets with boosting and data generation: the DataBoost-IM approach. Sigkdd Explorations 6: 30–39.

14. LaFree, G., L. Dugan, and E. Miller, 2012 Integrated United States Security Database

(IUSSD): Terrorism data on the united states homeland, 1970 to 2011. Final report to the resilient systems division, DHS Science and Technology Directorate, U.S. Department of Homeland Security, START, College Park, MD.

15. Phua, C., D. Alahakoon, and V. Lee, 2004 Minortiy report in fraud detection: classification of skewed data. Sigkdd Explorations 6: 50–59.

16. Zheng, Z., X. Wu, and R. Srihari, 2004 Feature selection for text categorization for imbalanced data. Sigkdd Explorations 6: 80–89.

http://www.tsa.gov/tsa-precheck/eligibility-requirements http://www.tsa.gov/tsa-precheck/eligibility-requirements

G-1

Appendix G Abbreviations and Acronyms The following sections contain technical terms, acronyms, and abbreviations that are used in this document.

Acronym Definition

CHRC Criminal History Records Check CJIS Criminal Justice Information Services CRADA Cooperative Research and Development Agreement CSV Comma Separated Values

DA-E Data Analytics – Engine DB Database DHS Department of Homeland Security

DOB Date of Birth

DOJ Department of Justice DUI Driving Under the Influence HSARPA Homeland Security Advanced Research Project Agency

ID Identification IPT Integrated Project Team IV&V Independent Verification and Validation KBA Knowledge Based Authentication LPR Legal Permanent Resident OTA Other Transaction Agreement PII Personally Identifiable Information

PIN Personal Identification Number

PVI Previously Verified Identity S&T Science and Technology Directorate

SFTP Secure File Transfer Protocol SORN Systems of Record Notice

SSI Sensitive Security Information STA Security Threat Assessment TSA Transportation Security Administration

H-1

Appendix H Country Codes The following table contains the acceptable values for 2-letter country codes for the countries named. These country codes are used for the TSA Pre✓® applicant’s country of citizenship (CTZ) and place of birth (POB), Residential and Mailing Addresses, and non U.S.-based employer addresses. Table is sorted by Country.

Note: This list is not representative of an official list of country codes. Countries denoted by * are non-official countries that were used in prior releases.

2-Letter Country Code Country

AF AFGHANISTAN

AX ÅLAND ISLANDS

AL ALBANIA

DZ ALGERIA

AS AMERICAN SAMOA

AD ANDORRA

AO ANGOLA

AI ANGUILLA

AQ ANTARCTICA

AG ANTIGUA AND BARBUDA

AR ARGENTINA

AM ARMENIA

AW ARUBA

AU AUSTRALIA

AT AUSTRIA

AZ AZERBAIJAN

BS BAHAMAS

BH BAHRAIN

BD BANGLADESH

BB BARBADOS

BY BELARUS

BE BELGIUM

H-2

BZ BELIZE

BJ BENIN

BM BERMUDA

BT BHUTAN

BO BOLIVIA, PLURINATIONAL STATE OF

BQ *BONAIRE, EUSTATIUS AND SABA

BA BOSNIA AND HERZEGOVINA

BW BOTSWANA

BV BOUVET ISLAND

BR BRAZIL

IO BRITISH INDIAN OCEAN TERRITORY

BN BRUNEI DARUSSALAM

BG BULGARIA

BF BURKINA FASO

BI BURUNDI

KH CAMBODIA

CM CAMEROON

CA CANADA

CV CABO VERDE

KY CAYMAN ISLANDS

CF CENTRAL AFRICAN REPUBLIC

TD CHAD

CL CHILE

CN CHINA

CX CHRISTMAS ISLAND

CC COCOS (KEELING) ISLANDS

CO COLOMBIA

KM COMOROS

CG CONGO

CD CONGO, THE DEMOCRATIC REPUBLIC OF THE

CK COOK ISLANDS

H-3

CR COSTA RICA

CI CÔTE D'IVOIRE

HR CROATIA

CU CUBA

CY CYPRUS

CZ CZECH REPUBLIC

DK DENMARK

DJ DJIBOUTI

DM DOMINICA

DO DOMINICAN REPUBLIC

EC ECUADOR

EG EGYPT

SV EL SALVADOR

GQ EQUATORIAL GUINEA

ER ERITREA

EE ESTONIA

ET ETHIOPIA

FK FALKLAND ISLANDS (MALVINAS)

FO FAROE ISLANDS

FJ FIJI

FI FINLAND

FR FRANCE

GF FRENCH GUIANA

PF FRENCH POLYNESIA

TF FRENCH SOUTHERN TERRITORIES

GA GABON

GM GAMBIA

GE GEORGIA

DE GERMANY

GH GHANA

GI GIBRALTAR

H-4

GR GREECE

GL GREENLAND

GD GRENADA

GP GUADELOUPE

GU GUAM

GT GUATEMALA

GG GUERNSEY

GN GUINEA

GW GUINEA-BISSAU

GY GUYANA

HT HAITI

HM HEARD ISLAND AND MCDONALD ISLANDS

VA HOLY SEE (VATICAN CITY STATE)

HN HONDURAS

HK HONG KONG

HU HUNGARY

IS ICELAND

IN INDIA

ID INDONESIA

IR IRAN, ISLAMIC REPUBLIC OF

IQ IRAQ

IE IRELAND

IM ISLE OF MAN

IL ISRAEL

IT ITALY

JM JAMAICA

JP JAPAN

JE JERSEY

JO JORDAN

KZ KAZAKHSTAN

KE KENYA

H-5

KI KIRIBATI

KP KOREA, DEMOCRATIC PEOPLE'S REPUBLIC OF

KR KOREA, REPUBLIC OF

KW KUWAIT

KG KYRGYZSTAN

LA LAO PEOPLE'S DEMOCRATIC REPUBLIC

LV LATVIA

LB LEBANON

LS LESOTHO

LR LIBERIA

LY LIBYAN ARAB JAMAHIRIYA

LI LIECHTENSTEIN

LT LITHUANIA

LU LUXEMBOURG

MO MACAO

MK MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF

MG MADAGASCAR

MW MALAWI

MY MALAYSIA

MV MALDIVES

ML MALI

MT MALTA

MH MARSHALL ISLANDS

MQ MARTINIQUE

MR MAURITANIA

MU MAURITIUS

YT MAYOTTE

MX MEXICO

FM MICRONESIA, FEDERATED STATES OF

MD MOLDOVA, REPUBLIC OF

MC MONACO

H-6

MN MONGOLIA

ME MONTENEGRO

MS MONTSERRAT

MA MOROCCO

MZ MOZAMBIQUE

MM MYANMAR

NA NAMIBIA

NR NAURU

NP NEPAL

NL NETHERLANDS

AN *NETHERLANDS ANTILLES

NC NEW CALEDONIA

NZ NEW ZEALAND

NI NICARAGUA

NE NIGER

NG NIGERIA

NU NIUE

NF NORFOLK ISLAND

MP NORTHERN MARIANA ISLANDS

NO NORWAY

OM OMAN

PK PAKISTAN

PW PALAU

PS PALESTINIAN TERRITORY, OCCUPIED

PA PANAMA

PG PAPUA NEW GUINEA

PY PARAGUAY

PE PERU

PH PHILIPPINES

PN PITCAIRN

PL POLAND

H-7

PT PORTUGAL

PR PUERTO RICO

QA QATAR

RE RÉUNION

RO ROMANIA

RU RUSSIAN FEDERATION

RW RWANDA

BL SAINT BARTHÉLEMY

SH SAINT HELENA

KN SAINT KITTS AND NEVIS

LC SAINT LUCIA

MF SAINT MARTIN

PM SAINT PIERRE AND MIQUELON

VC SAINT VINCENT AND THE GRENADINES

WS SAMOA

SM SAN MARINO

ST SAO TOME AND PRINCIPE

SA SAUDI ARABIA

SN SENEGAL

RS SERBIA

SC SEYCHELLES

SL SIERRA LEONE

SG SINGAPORE

SX SINT MAARTEN

SK SLOVAKIA

SI SLOVENIA

SB SOLOMON ISLANDS

SO SOMALIA

ZA SOUTH AFRICA

GS SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS

ES SPAIN

H-8

LK SRI LANKA

SD SUDAN

SR SURINAME

SJ SVALBARD AND JAN MAYEN

SZ SWAZILAND

SE SWEDEN

CH SWITZERLAND

SY SYRIAN ARAB REPUBLIC

TW TAIWAN, PROVINCE OF CHINA

TJ TAJIKISTAN

TZ TANZANIA, UNITED REPUBLIC OF

TH THAILAND

TL TIMOR-LESTE

TG TOGO

TK TOKELAU

TO TONGA

TT TRINIDAD AND TOBAGO

TN TUNISIA

TR TURKEY

TM TURKMENISTAN

TC TURKS AND CAICOS ISLANDS

TV TUVALU

UG UGANDA

UA UKRAINE

AE UNITED ARAB EMIRATES

GB UNITED KINGDOM

US UNITED STATES

UM UNITED STATES MINOR OUTLYING ISLANDS

UY URUGUAY

UZ UZBEKISTAN

VU VANUATU

H-9

VE VENEZUELA, BOLIVARIAN REPUBLIC OF

VN VIET NAM

VG VIRGIN ISLANDS, BRITISH

VI VIRGIN ISLANDS, U.S.

WF WALLIS AND FUTUNA

EH WESTERN SAHARA

YE YEMEN

ZM ZAMBIA

ZW ZIMBABWE