24 Attachment G - TSA T and E Guidebook Rev 6 - Final.pdf

Test and Evaluation Guidebook

Revision 6

August 27, 2024

T&E Guidebook Revision 6 i

APPROVAL:

Thomas Tomaiko Independent Test Agent

Executive Director, Test and Evaluation Division Acquisition Program Management

Transportation Security Administration

THOMAS A

TOMAIKO

Digitally signed by

THOMAS A TOMAIKO

Date: 2024.09.03 18:30:40 -04'00'

T&E Guidebook Revision 6 ii

TABLE OF CONTENTS

SECTION NO. PAGE NO.

1.0 PURPOSE

1.1 T&E OVERVIEW

1.2 T&E PRINCIPLES

1.3 T&E ORGANIZATION

1.4 ACQUISITION PROGRAM CATEGORIES

1.5 T&E METRICS AND INFORMATION SHARING

2.0 T&E ROLES AND RESPONSIBILITIES

2.1 SYSTEM EVALUATOR

2.2 OPERATIONAL TEST DIRECTOR

2.3 T&E MANAGER

2.4 QUALIFICATION TEST DIRECTOR

2.5 DESIGNATED GOVERNMENT CERTIFICATION AGENT

2.6 T&E STAKEHOLDER ORGANIZATIONS

2.6.1 APM/PEO

2.6.2 APM/Capability and Acceptance Management Branch

2.6.3 APM/Deployment and Sustainment Division

2.6.4 CAE Staff

2.6.5 Contracting and Procurement

2.6.6 IT/IAD

2.6.7 IT/Governance, Risk, and Compliance

2.6.8 Intelligence and Analysis

2.6.9 Occupational Safety, Health, and Environment

2.6.10 Requirements and Capabilities Analysis/Analysis and Engineering

2.6.11 RCA/Capabilities Management and Innovation

2.6.12 Training and Development

3.0 T&E STRATEGY AND PLANNING

3.1 T&E IPT

3.2 SYSTEM EVALUATION TEAM

3.3 REVIEWING AND ASSESSING REQUIREMENTS

T&E Guidebook Revision 6 iii

3.4 USE OF THIRD PARTY TESTING IN A SYSTEM EVALUATION

3.5 DEVELOPING EVALUATION CRITERIA

3.6 TES DEVELOPMENT

3.7 TEMP DEVELOPMENT

3.7.1 Evaluation Framework

3.7.2 Modeling and Simulation

3.8 SYSTEM EVALUATION PLAN

3.9 TEST AND EVALUATION CONCEPT BRIEFING

3.10 DETERMINING THE SCOPE OF TESTING AFTER A SYSTEM CHANGE

4.0 DEVELOPMENTAL, CERTIFICATION, AND QUALIFICATION TESTING ... 25

4.1 TEST EVENT OVERVIEW

4.2 QT PLANNING

4.2.1 Planning Documentation

4.2.2 Coordination of System Delivery and Installation

4.2.3 Establishing the Configuration Baseline Prior to Qualification Testing

4.2.4 Test Readiness Notification

4.2.5 Threat Items for CERT

4.2.6 Test Article Selection and Development

4.2.7 Test Tool Development and VV&A

4.2.8 Coordination with Supporting Organizations

4.2.9 QT Readiness Review

4.3 QT EXECUTION AND REPORTING

4.3.1 Test Execution

4.3.2 QT DAG

4.3.3 Data Analysis and Reporting

4.3.4 Results Adjudication

4.3.5 Regression QT

4.3.6 Test Article Management

4.3.7 Configuration Control During Test Execution

4.3.8 Maintenance Activities

4.3.9 Termination of QT Activities

T&E Guidebook Revision 6 iv

5.0 OPERATIONAL TESTING

5.1 OT EVENTS OVERVIEW

5.2 OT PLANNING

5.2.1 Operational Test Plan

5.2.2 OA Plan and OUE Plan

5.2.3 Site Selection

5.2.4 Test Site Coordination and In-Briefs

5.2.5 Establishing Baseline Capability

5.2.6 Test Team Preparation and Training

5.2.7 OT TRN

5.2.8 Verifying the System Configuration Prior to Test Execution

5.2.9 Configuration Control During Test Execution

5.2.10 Operational Test Readiness Review

5.3 BURN-IN AND TEST EXECUTION

5.3.1 Burn-In

5.3.2 Test Execution and Data Collection

5.3.3 Early Termination of Operational Testing

5.3.4 Test Closeout and Site Restoration

5.4 TI TESTING

5.4.1 TI Test Planning

5.4.2 TI Execution

5.4.3 Working with TI Data

5.4.4 Test Article Security

5.5 OT DATA AUTHENTICATION

5.5.1 OT DAG Personnel

5.5.2 DAG Scoring Criteria

5.5.3 DAG Procedures

5.6 OT EVENT REPORTING

5.6.1 OT QLR

5.6.2 OT ERB

5.6.3 OA Report

T&E Guidebook Revision 6 v

5.6.4 OUE Report

6.0 SYSTEM EVALUATION REPORT

6.1.1 Test Event Reporting in Support of the SER

6.1.2 SER Development

6.1.3 SER Review and Approval

6.1.4 SER Redress Memorandum

7.0 ACCEPTANCE TESTING

7.1 ACCEPTANCE TESTING

7.1.1 First Article Test and Evaluation

7.1.2 Factory Acceptance Test

7.1.3 SAT and ORT

7.1.4 AT Reporting

7.2 INTEGRATION ACCEPTANCE TESTING

7.2.1 Network Throughput Test

7.2.2 Integrated System Acceptance Test

7.2.3 Operational Run-in & Closeout

7.2.4 ISAT and Operational Run-in Data Analysis

7.2.5 Integration Testing Reporting

7.3 BAGGAGE ASSESSMENT AND SYSTEM EVALUATION

7.3.1 CBIS PGDS

7.3.2 BHS CBIS Support

7.3.3 CBIS Design Reviews

7.3.4 BASE Efforts

7.3.5 CBIS R&D

7.4 LABORATORY/FACILITY BHS SUPPORT

7.4.1 TSIF BHS

7.4.2 TSALC BHS Loop

7.5 ACCEPTANCE TESTING AND SYSTEM ASSESSMENT TEST ARTICLES

7.5.1 Image Quality Test Kits

7.5.2 SOC Representative Test Articles

7.5.3 ISAT Test Articles

T&E Guidebook Revision 6 vi

7.5.4 Donations: ATSA Logistical Support

APPENDIX A: ACRONYM LIST .................................................................................... A-1

APPENDIX B: TEST AND EVALUATION PLANS, REPORTS, AND OTHER

SUPPORTING DOCUMENTATION ............................................................................... B-1

APPENDIX C: REFERENCES......................................................................................... C-1

APPENDIX D: CYBERSECURITY EVALUATION GUIDANCE.................................. D-1

APPENDIX E: DOCUMENTATION GUIDELINES ....................................................... E-1

APPENDIX F: TEST AND EVALUATION ROLES AND RESPONSIBILITIES ...........F-1

T&E Guidebook Revision 6 vii

LIST OF TABLES

TABLE NO. PAGE NO.

Table 3-1: Change Package Tier Designations

Table 4-1: Developmental / QT Events

Table 4-2: QT DAG Personnel

Table 4-3: Determination of Compliance with Requirements

Table 4-4: QT Severity Ratings (Adjudicated)

Table 5-1: Operational Testing Activities

Table 5-2: OT DAG Personnel

Table 7-1: Types of ATs

Table 7-2: Types of Integration ATs

LIST OF FIGURES

FIGURE NO. PAGE NO.

Figure 1-1: Acquisition Lifecycle Framework

Figure 3-1: TES and Evaluation Planning Process Flow

Figure 3-2: ITA Review of External Data

Figure 3-3: Example Criteria and Derived Measures

Figure 3-4: Relationship of COIs, Criteria, and Measures

Figure 3-5: TEMP to Reporting Hierarchy

Figure 3-6: Changes to Deployed Systems Process

Figure 4-1: Qualification Test Planning Process Flow

Figure 4-2: QT Execution Process Flow

Figure 5-1: OT Planning Process Flow

Figure 5-2: Site Selection Process

Figure 5-3: OT Execution Process Flow

Figure 6-1: Evaluation Reporting Process Flow

Figure 7-1: AT Process

Figure 7-2: ISAT Process

T&E Guidebook Revision 6 1

1.0 PURPOSE

The Transportation Security Administration (TSA) Test and Evaluation (T&E) Guidebook describes how TSA T&E organizations, in collaboration with stakeholders, plan and execute an effective and efficient T&E Strategy (TES). The guidebook aligns with the TSA T&E Management Directive (MD) 2600.4, dated August 21, 2024, and meets the spirit and intent of overarching Department of Homeland Security (DHS) policy as described in the Department of Homeland Security (DHS) Acquisition Management Directive 102-01, dated February 25, 2019, and DHS Directive 026-06, Test and Evaluation, dated October 01, 2020. Additionally, this guidebook aligns with the TSA Acquisition Manual (TSAAM), dated June 2023.

The T&E activities described in this guidebook support the DHS Acquisition Life Cycle Framework (ALF). The ALF, depicted in Figure 1-1, identifies the four phases required for system acquisition at DHS and the associated Acquisition Decision Events (ADEs).

Figure 1-1: Acquisition Lifecycle Framework

A successful ADE reflects approval to continue to the next ALF phase as well as the commitment of effort and funds. T&E provides evaluation results and conclusions to inform decision makers during the Acquisition Review Board (ARB) meeting that assists the Acquisition Decision Authority (ADA) in determining whether to proceed to the next phase of the acquisition life cycle.

T&E Guidebook Revision 6 2

1.1 T&E Overview

TSA/Acquisition Program Management (APM) performs T&E to determine whether a system, inclusive of technology, operator, and procedures, meets operational and functional requirements. TSA T&E organizations support the T&E of a wide variety of systems from checked baggage and passenger screening Transportation Security Equipment (TSE) to Information Technology (IT) systems. This revision of the guidebook focuses on the T&E procedures for the assessment of TSE.

Acquisition Management Directive 102-01 applies to Acquisition Level 1 and Level 2 programs and programs designated as special interest by the Undersecretary of Management. TSA T&E organizations may also assess capabilities not directly associated with an acquisition program (e.g., technology demonstration of emerging capability). This guidebook is primarily focused on the T&E processes associated with acquisition programs. The guidebook clearly identifies any explanation of processes not associated with acquisition program T&E when presented.

Throughout this guidebook, there are recurring references to Operational Effectiveness, Suitability, and Resilience. These terms are critical to the overall goals and outcomes of T&E and defined as follows:

• Operational Effectiveness is the overall ability of a system to accomplish mission objectives when used by representative personnel in the environment expected for operational employment considering organization, procedures, supportability, vulnerability, and threat.

• Operational Suitability is the overall degree to which a system can be satisfactorily used with consideration given to transportability, interoperability, reliability, maintainability, availability, safety, human-systems integration, manpower, logistics supportability, and training requirements.

• Operational Resilience is the overall ability of the system to protect, detect, counter react to threats, and restore capabilities, to include exploitation potential and mission impact. A critical component/criterion of operational resilience is cyber resilience. Cyber resilience is an assessment of the system’s ability to remain effective and suitable to satisfy the mission while under duress from offensive cybersecurity attack.

1.2 T&E Principles

TSA Test Teams conduct T&E following a set of four T&E principles. These principles enable T&E Teams to support the material objectives of speed to field and increased performance by discovering problems early using efficient and proven T&E methods. The four T&E principles are:

• Early Test Involvement

• Tailor to the Situation

• Integrated and Parallel Testing

• Streamline Processes and Products

T&E Guidebook Revision 6 3

In support of early test involvement, TSA T&E Teams conduct Operational Utility Evaluations (OUEs) in the field and/or Data Collection Activities in laboratory environments in support of Research and Development (R&D) activities, such as in support of the Innovation Task Force (ITF). These events allow for the early identification of system capabilities and limitations to assist vendors in their system design and development. The events also assist TSA in deciding whether a system is of a sufficient level of maturity to proceed through the acquisition/qualification process.

T&E tailors its early T&E support to the event sponsor’s needs, ensuring the T&E Team collects and provides the precise sought after data in an expeditious manner. To accomplish these goals, T&E typically develops a concise, incremental, test plan (e.g., OUE Plan) and report (e.g., OUE Report). T&E Teams will help sponsors develop data collection strategies and approaches when documented requirements are not available.

Further tailoring may occur in support of acquisition programs when the Independent Test Agent (ITA) conducts Operational Assessments (OAs) to assess overall readiness to proceed to an Operational Test (OT). If documented in the approved Test and Evaluation Master Plan (TEMP), the ITA may streamline the OT event by utilizing data from the OA as a primary system evaluation data source.

When planning for a test event, the ITA is responsible for assigning a primary test entity to verify individual test measures (and requirements). During this activity, the ITA should pursue methods to increase test execution efficiency such as pursuing an integrated test event if feasible and/or parallel execution of test activities. Integrated testing is the collaborative planning and execution of test phases and events to provide data in support of independent analysis, evaluation, and reporting by all stakeholders, particularly the Qualification Test (QT), and OT entities. The ITA attempts to minimize the sequential/serial execution of test events as it delays acquisition and fielding timelines.

1.3 T&E Organization

The TSA Executive Director of the APM/T&E Division (hereafter referred to as the Executive Director, ITA) leads the TSA ITA. The ITA performs the independent evaluation of Operational Effectiveness, Suitability, and Resilience and is comprised of the APM/T&E Division as approved by the Director, Office of Test and Evaluation (DOT&E). The APM/T&E Division consists of the branches listed below.

• The System Evaluation Branch (SEB) is responsible for the overall evaluation of the Operational Effectiveness, Suitability, and Resilience of a system. The SEB produces OUE Reports in support of R&D efforts and System Evaluation Reports (SERs) in support of acquisition decisions.

• The Operational Test Branch (OTB) plans and conducts OAs and OTs in support of the SEB overall system evaluation. The OTB also plans and conducts OUEs in support of R&D efforts.

T&E Guidebook Revision 6 4

• The TSA Systems Integration Facility (TSIF) Test Branch (TTB) plans and conducts QT events at the TSIF to verify system technical capability in support of the overall system evaluation. The TTB generally verifies non-detection-related functional requirements, however, they may verify detection-related requirements when needed. The TTB also serves as the Designated Government Certification Agent (DGCA), or in a supporting role to the DGCA, for non-explosives detection and false alarm verification (e.g., Prohibited Item [PI] detection). The TTB also supports Data Collection Activities in support of R&D efforts.

• The Test Infrastructure and Operations Branch (TIOB) supports TTB and OTB test activities by providing facility, test article, Configuration Management (CM), and other related logistical and infrastructure support.

• The Acceptance Test and System Assessment (ATSA) Branch (hereafter referred to as ATSA) is responsible for providing Acceptance Test (AT) support activities for T&E events and in support of deployment activities post-ADE-3.

The following entities provide data to the ITA evaluation of candidate systems as a DGCA, or in support of the DGCA, as documented in the program TEMP. Additional information concerning the role of the DGCA can be found in the TSA Transportation Security Equipment Certification Policy, dated April 21, 2021. As such, the APM/T&E Division includes functional leadership who coordinate directly with these entities:

• The DHS/Science and Technology/Transportation Security Laboratory (TSL)/Independent Test and Evaluation (IT&E) Division conducts Certification Testing (CERT) in the areas of explosives detection and false alarm verification against approved detection standards. TSL/IT&E also supports ad-hoc special studies and exploratory testing on an as needed basis. The System Evaluator typically uses the CERT as a primary input to the Operational Effectiveness determination. The TSL/Developmental Test and Evaluation Division supports the development of candidate technologies by providing Certification Readiness Assistance and Readiness Testing prior to formal CERT and supporting ad-hoc special studies as needed. The TSL Deputy Director is functionally aligned within the APM/T&E Division.

• The Federal Bureau of Investigation’s (FBI’s) Terrorist Explosive Device Analytical Center (TEDAC) Improvised Explosives Detection and Synthesis (TIEDS) Center supports FBI, Science and Technology, and TSA’s Research, Development, Test, and Evaluation priorities to assess the safety and performance characteristics of improvised explosives and assess performance of detection technologies against existing and emerging threats. In this regard, the TSL and TIEDS operate in tandem to assess TSA system performance against explosives detection requirements. The TIEDS Center is comprised of an explosive synthesis and testing laboratory, a trace explosives laboratory, the detection technology center, and explosives testing range. The TIEDS TSA onsite infrastructure lead is functionally aligned within the APM/T&E Division.

The ITA may leverage data from other data sources both internal to TSA (e.g., IT/Information Assurance and Cybersecurity Division [IAD]) and external to TSA (e.g., third party test organizations) in its evaluation under certain conditions. Stakeholder organizations are

T&E Guidebook Revision 6 5 documented in section 2.6. One can find additional details on the ITA’s use of external data sources in the Third Party Testing Framework, dated October 04, 2021, and in section 3.4 of this guidebook.

1.4 Acquisition Program Categories

Throughout this guidebook, there are references to acquisition programs in four categories as defined below.

• Major Programs (Non-Delegated Systems): Any system within a major program as designated in the DHS Master Acquisition Oversight List (MAOL) in a non-delegated status. These generally consist of systems within Level 1 and Level 2 programs that have not received an initial successful ADE-3 decision. The ADA for major programs (non-delegated systems) is the DHS Chief Acquisition Officer.

• Major Programs (Delegated Systems): Any system within a major program, as designated in the DHS MAOL in a delegated status. These generally consist of systems within Level 1 and Level 2 programs that have received a successful ADE-3 fielding decision with delegation status formally granted by the Chief Acquisition Officer at the ADE-3 and documented in the associated Acquisition Decision Memorandum (ADM). The ADA for major programs (delegated systems) is the Component Acquisition Executive (CAE).

• Major Program (Post-Full Operational Capability [FOC] Delivery): A major program designated as post-FOC delivery in the DHS MAOL. These programs generally differ from those previously described in that all systems within these programs are in the acquisition sustainment / operations and maintenance phase. The ADA for major programs (post-FOC delivery) is the CAE.

• Non-Major Programs: Programs designated as “non-major” in the DHS MAOL. These generally consist of Level 3 programs. The ADA for non-major programs is the CAE.

This guidebook denotes any differences in the planning, execution, and reporting based on program categorization in the appropriate sections.

1.5 T&E Metrics and Information Sharing

The T&E Division test branches are responsible for the ongoing tracking and reporting of T&E activities to include test events planned and completed, plans and reports delivered, and event tracking against program-defined milestones. This tracking feeds a division dashboard that is utilized to compile accomplishments for end-of-year reporting and to understand potential deficient areas and improvement opportunities.

The T&E Division also requires its test branches to maintain an easily accessed repository for maintaining and sharing T&E policy and procedure, templates, signed plans and reports, and tracking mechanisms for T&E Division-provided T&E recommendations and for any ADM action items assigned to the T&E Division. As such, the test branches have processes in place for ensuring these repositories are current and documents are easily accessed (e.g., intuitive folder structure, standard naming convention) to allow for easy reference and effective sharing across the division.

T&E Guidebook Revision 6 6

2.0 T&E ROLES AND RESPONSIBILITIES

The primary functional roles responsible for T&E at TSA are listed in overarching documentation, specifically, the DHS Acquisition Management Directive 102-01, DHS Management Directive 026-06, the TSA T&E Management Directive 2600.4, and the TSA Transportation Security Equipment Certification Policy.

The roles critical to support the execution of T&E at TSA are the T&E Manager, System Evaluator, Operational Test Director (OTD), QT Director (QTD), and the DGCA. The following sections describe the responsibilities for these critical roles.

2.1 System Evaluator

The System Evaluator is a member of the ITA and responsible for leading the evaluation of Operational Effectiveness, Suitability, and Resilience. Specifically, the System Evaluator:

(1) Leads the System Evaluation Team (SET) and ensures the SET activities are in accordance with the program TEMP, T&E Policy, and T&E guidelines. Ensures coordination across all T&E stakeholders on subject matter that supports the system evaluation.

(2) Reviews all requirements to ensure that they are testable, measurable, and achievable.

(3) Works with the T&E Manager and the OTD to determine the TES and the test activities required to evaluate a system to support the acquisition strategy identified by the Program Management Office (PMO). Briefs the TES to the Executive Director, ITA in the form of an Early Strategy Review (ESR). The Executive Director, Program Execution Office (PEO) should also attend the ESR for major programs (non-delegated systems).

(4) Collaborates with the T&E Manager to develop the Evaluation Framework (EF).

(5) Provides the TES inputs in the TEMP.

(6) Provides input to the TES briefing. As requested by the PMO, leads the presentation of the T&E approach in the TES briefing.

(7) For deployed systems, develops a Determination of Test Scope (DTS) for Tier 3-proposed system changes as further defined in section 3.8.

(8) Develops the System Evaluation Plan (SEP).

(9) Develops the Test and Evaluation Concept Briefing (TECB). Presents the TECB to

DOT&E.

(10) Leads the threat working group to define the threat/threat set required to support Certification test efforts for the program’s detection requirements. In collaboration with the T&E Manager and DGCA, facilitates the endorsement or approval of the threat sets.

(11) Authors the threat validation memorandum for Threat Inject (TI) test articles.

(12) Reviews and concurs with all test plans to ensure the test and data collection methodologies are adequate to support the evaluation plan/strategy.

T&E Guidebook Revision 6 7

(13) Responsible for the final selection of OT sites using input from the OTD and the SET.

(14) Chairs Data Authentication Group (DAG) meetings, Reliability, Maintainability, Availability (RMA) scoring conferences (if held separately from the DAG), and Adversarial Assessment (AA) vulnerability scoring conferences with support from the OTD. Considers all inputs from DAG members and provides the final authentication decision for test data.

(15) Issues a CERT Letter Concurrence memorandum to the PMO prior to PMO issuance of a vendor Certification Letter.

(16) Authors the OT Quick Look Report (QLR), SER (and corresponding TI Report if required), OA Report, and OUE Report. Drafts a SER Communication Memorandum (SCM) and works with the APM Front Office to communicate the SER to stakeholder organizations. Archives all final approved reports in the OTB/SEB document library.

(17) Reviews and provides feedback to the DOT&E-developed Letter of Assessment (LOA).

(18) Upon PMO request, supports the development of program/acquisition materials that contain T&E results. Upon request, supports the program’s presentation of these results to leadership at Acquisition Review Team meetings and ARB meetings.

(19) Upon PMO request, supports Technical Interchange Meetings (TIMs) with vendors where the results of T&E are discussed.

2.2 Operational Test Director

The OTD is a member of the ITA and is responsible for leading the planning and execution of OA and OT activities. Specifically, the OTD:

(1) During TES Development and Evaluation Planning:

a. Reviews all operational requirements to ensure that they are operationally meaningful, testable, measurable, and achievable and ensures the OT design is adequate to address requirements.

b. Communicates required OT level of effort (e.g., sites, duration, test staff) with ITA leadership and stakeholders to ensure appropriate resources are available for a timely OT start.

c. Provides input as requested by the PMO or the T&E Manager for the PMO-managed Integrated Master Schedule (IMS) concerning test activity timelines.

d. Provides OT inputs to the TEMP, as requested by the T&E Manager, and SEP.

e. Supports the T&E Manager and System Evaluator by providing input to EF development with a focus on the OT data source assignments for requirements.

f. Provides input to the ESR briefing and TECB, and presents OT portions in coordination with the System Evaluator. Coordinates with the T&E Manager and PMO during the development of the TES and presents the OT portions of the TES briefing as needed.

T&E Guidebook Revision 6 8

g. Coordinates the development of threat articles with the SET and coordinates test article approvals for testing as needed.

(2) During OT/OA/OUE Planning:

a. Leads the site selection process to include coordination with the user organization and deployment stakeholders, and conducting site surveys as required, to understand site usage of relevant systems and provides a list of recommended sites to the System Evaluator for final selection.

b. Develops the Operational Test Plan (OTP) and OA Plan to include AA planning and all other required test and data methodologies in alignment with the evaluation strategy/plan. Develops OUE Plans in support of R&D efforts. Archives all final approved plans in the OTB/SEB document library.

c. Reviews and approves all data collection forms, ensuring they are suitable to capture the required data elements from the test plan.

d. Develops the TI testing design of experiments and conducts TI test article validation at the TSIF when necessary.

e. Works with the support contractor to ensure resources are identified, properly trained, and available to start testing per approved timelines.

f. Conducts airport Federal Security Director (FSD) briefs and periodic coordination calls with local TSA to ensure test site readiness according to schedules.

g. Coordinates the completion of all required entrance criteria, ensuring a timely start to OT, OA, or OUE activity. Leads the Operational Test Readiness Review (OTRR) criteria tracking during the SET. Presents the OTRR to leadership.

(3) During OT/OA/OUE execution:

a. Leads the test execution and coordination of site activities.

b. Delivers the Start of Testing Notification (STN) upon completion of burn-in.

Alternatively, suggests extension of burn-in or escalates other issues as appropriate.

c. Monitors test progress against the test plan and delivers daily status reports to the SET. This requires working closely with the support contractor analyst to ensure the T&E Team is collecting data according to the test plan with an understanding of sample sizes collected to date, target sample size, and test duration remaining.

d. Escalates issues, as appropriate, to include test extension or termination recommendations.

e. Leads the presentation of data at the DAG, RMA scoring conference, and AA vulnerability scoring conference, and provides input on the scoring of data, to include any Test Incident Reports (TIRs).

T&E Guidebook Revision 6 9

(4) During OT/OA/OUE reporting:

a. Reviews and provides the System Evaluator with the dataset as authenticated in the DAG sessions.

b. Develops the Emerging Results Briefing (ERB) for OT events under DOT&E oversight.

c. Supports classified TI DAG sessions and reviews the TI Report.

d. Reviews and concurs with the OT QLR, OA Report, OUE Report, SER, and/or any other reporting artifact or memorandum.

e. Reviews and provides feedback to the DOT&E-developed LOA.

f. Upon PMO request, works with the System Evaluator to develop program/acquisition materials that contain the results of the OT/OA/OUE. Upon request, supports the program’s presentation of these results to leadership.

g. Upon PMO request, supports TIMs with vendors in which discussion of T&E results occur.

2.3 T&E Manager

While the T&E Manager is functionally aligned within the APM/T&E Division, they are not a representative of the ITA. In accordance with DHS Test and Evaluation Instruction 026-06-001, the T&E Manager acts as the focal point for T&E efforts. Specifically, the T&E Manager:

(1) Supports the Program Manager (PM) by providing TES and risk mitigation input during the development of an Acquisition Strategy.

(2) Coordinates T&E activities by providing insights to stakeholders concerning TES, T&E processes, T&E guidelines, planning, execution and reporting documentation, as well as the relationship to Capability Manager- and acquisition-related documentation required to support T&E efforts.

(3) Provides input to the PMO to support technically informed and objective judgement on T&E planning, execution, and reporting. Assists in the planning and coordination of resources to support the program’s T&E efforts.

(4) Schedules and facilitates discussions between PMO, T&E Division, TSL, and other stakeholders upon request to mitigate T&E issues due to schedule or resources conflicts or to address stakeholder requests for accelerated or other unique test services, as needed and upon request.

(5) Provides input to the program IMS, and tracks and communicates the T&E schedules for assigned acquisition programs. Coordinates with the PMO and the ITA to mitigate issues that result in a shift right to the schedule.

(6) Provides input into all requirements to ensure that they are testable, measurable, and achievable. Coordinates Functional Requirements Document (FRD) review with the ITA and ensures the PMO reviews and adjudicates ITA feedback.

T&E Guidebook Revision 6 10

(7) Chairs the T&E Integrated Product Team (IPT) as delegated by the PM.

(8) Collaborates with the System Evaluator to determine the TES and the test activities required to evaluate a system to support the PMO acquisition strategy.

(9) Provides input into the System Evaluator’s ESR briefing to the Executive Director, ITA.

(10) In support of the PM and in coordination with the ITA, develops the TEMP and TES briefing. Supports the PM and ITA in briefing the TES to DOT&E as needed.

(11) With the System Evaluator, develops the EF.

(12) Reviews and provides feedback on test briefings, plans, and reports such as the SEP, TECB, OTP, and SER to ensure alignment with the TEMP.

(13) Participates in and provides feedback at DAG meetings.

(14) Reviews and provides feedback to the DOT&E-developed LOA.

(15) Supports the PM in the development of program/acquisition materials that contain T&E results. Upon request, supports the program’s presentation of these results to leadership at ART meetings and ARBs.

(16) Upon PMO request, supports TIMs with vendors in which discussion of T&E results occur.

(17) Supports the DGCAs in the development and revision of the Certification Test Management Plans as per TSA CERT policy for PM signature.

(18) Reviews and provides input to the System Evaluator concerning the T&E Division Memorandums: (DTS, SCM, CERT Letter Concurrence memorandum).

(19) Supports the program procurement documents (e.g., Request for Proposals) to ensure T&E equities have been included.

2.4 Qualification Test Director

The QTD is a member of the ITA and is responsible for leading the planning, execution and reporting of QT activities. Specifically, the QTD:

1) Reviews all technical and functional requirements to ensure they are testable, measurable, and achievable.

2) Supports the T&E Manager and System Evaluator by providing input to EF development with a focus on the QT primary and secondary data source assignments for requirements verification.

3) Provides input as requested by the PMO or the T&E Manager for the PMO-managed IMS concerning QT test activities, timelines, and predecessors.

4) Provides QT inputs to the TEMP and EF as requested by the PM or T&E Manager.

5) For deployed systems, reviews and provides feedback to the CM Team on system change documentation as it relates to a potential Regression QT (RQT).

T&E Guidebook Revision 6 11

6) Coordinates with the System Evaluator and PMO to identify RQT test scope for change requests. Provides input as requested to the System Evaluator for the development of a

DTS.

7) Develops and distributes the test plans and reports for QT and RQT results.

8) Executes all QT and RQT test activities

2.5 Designated Government Certification Agent

(1) Reviews all CERT-related operational, technical, and functional requirements to ensure they are testable, measurable, and achievable.

(2) Supports the T&E Manager and System Evaluator by providing input to EF development with a focus on the CERT primary and secondary data source assignments for requirements verification.

(3) Provides input as requested by the PMO or the T&E Manager for the PMO-managed IMS concerning CERT activities, timelines, and predecessors.

(4) Provides CERT inputs to the TEMP as requested by the T&E Manager.

(5) For deployed systems, reviews and provides feedback to the CM Team on system change documentation as it relates to CERT.

(6) Provides input as requested to the System Evaluator during the development of the

DTS.

(7) Develops and distributes the CERT Test Plan, CERT QLR, CERT Emerging Results

Report, and CERT Final Report.

(8) Executes all Certification activities for Readiness Assessments, Certification Readiness

Tests and Certification Tests.

(9) Provides input, as requested, to the System Evaluator for the development of the CERT

Letter Concurrence Memorandum.

(10) Provides input into the threat working group to define the threat/threat set required to execute CERT against applicable requirements and detection standards.

(11) Develops the CERT Management Plan in collaboration with the PMO and T&E

Manager.

(12) Supports vendor TIMs upon request of the PM on areas pertinent to CERT and

Certification Readiness Testing.

(13) Advises on potential detection impacts with respect to changes in the detection standard or functional requirements associated with explosive and PI detection.

2.6 T&E Stakeholder Organizations

The following are TSA organizations that provide additional input and support to T&E efforts.

T&E Guidebook Revision 6 12

2.6.1 APM/PEO

The PEO is the organization within APM in which the PMOs are functionally aligned. The PMOs are segmented within Portfolios with an identified PM to oversee each acquisition program. The PMO is responsible for the overall management of the program to include schedule, budget, and technical capability against requirements.

2.6.2 APM/Capability and Acceptance Management Branch

The Capability and Acceptance Management Branch (CAMB) within APM is responsible for overseeing the capability acceptance process, which manages all donation activities of security equipment for TSA. CAMB provides guidelines for donors, ensures the donation activity has an appropriate financial analysis and business case, determines whether the donated equipment meets TSA guidelines, and ensures donated equipment receive satisfactory third party AT prior to operational use.

2.6.3 APM/Deployment and Sustainment Division

The Deployment and Sustainment Division (DSD) provides site characteristics and relevant historical data to inform the site selection process and supports the installation and integration of systems at operational sites as needed.

2.6.4 CAE Staff

The CAE staff provides acquisition assistance to the PMO and the ITA. The CAE staff guides the PMO through the acquisition process to successfully navigate all ADEs and coordinate TSA ADMs. The CAE staff, at the time of this guidebook revision, are organizationally aligned within the APM/Acquisition Advisory Division (AAD).

2.6.5 Contracting and Procurement

Contracting and Procurement (C&P), together with the PMO, serves as the liaison between TSA and the vendor. C&P is responsible for managing the overall procurement process, coordinating industry days, issuing contract awards, and the release of qualification documentation. C&P supports the PMO in the development and execution of the program acquisition strategy.

2.6.6 IT/IAD

IT personnel provide expertise in areas associated with IT and cybersecurity. IT/IAD, or alternate entity depending on the TES, provides general cybersecurity T&E input to the System Evaluator. This includes activities such as development of threat assessments, conducting Cooperative Vulnerability and Penetration Assessments (CVPAs) and Penetration Tests, results analysis, and development of reports. The ITA is responsible for the execution of the AA.

T&E Guidebook Revision 6 13

2.6.7 IT/Governance, Risk, and Compliance

The Governance, Risk, and Compliance (GRC) Division within IT, manages the overall risk and compliance determinations of TSA systems. As it relates to T&E, GRC conducts security controls assessments in support of authority to operate or risk acceptance memorandum decisions, which must be in place prior to a test activity of a TSA-connected system. Further, the GRC concurs with plan of action and milestone vulnerability closure requests upon review of closure evidence and supporting materials.

2.6.8 Intelligence and Analysis

Intelligence and Analysis provides the PMO, requirements sponsor, and T&E organizations with intelligence-based threat information (i.e., threat actors, vulnerabilities, adversary tendencies and preferences) in support of the system threat assessment and overarching TES.

2.6.9 Occupational Safety, Health, and Environment

Occupational Safety, Health, and Environment (OSHE) provides support to the T&E organizations by performing, or supporting, safety assessments for candidate TSE. OSHE provides findings, impact assessment, and recommendations related to issues discovered.

2.6.10 Requirements and Capabilities Analysis/Analysis and Engineering

Requirements and Capabilities Analysis (RCA)/Analysis and Engineering Directorate supports T&E by providing analysis, threat and risk assessments, requirements development, and acquisition documentation development such as the Mission Needs Statement (MNS), Concept of Operations (CONOPs), and Operational Requirements Document (ORD).

This Directorate also supports RCA's mission by managing and/or coordinating demonstrations of emerging technologies and support early identification of capabilities and limitations. When requested, the T&E Division may conduct one or more early test activities (e.g., OUEs, Data Collection Activities) in support of these initiatives.

2.6.11 RCA/Capabilities Management and Innovation

RCA/Capabilities Management and Innovation Directorate personnel serve as the capability managers (User Representatives) for acquisition programs. The User organization is responsible for approving and or concurring with acquisition documentation such as the MNS, CONOPs, ORD, and TEMP. The User organization provides the operator’s perspective throughout the T&E process, to include providing input to any observed T&E deficiencies, approves test articles used during testing, and provides critical inputs to the OT site selection process. The ITF leads technology exploration and R&D initiatives. When requested, the T&E Division may conduct one or more early test activities (e.g., OUEs, Data Collection Activities at the TSIF) in support of these initiatives.

T&E Guidebook Revision 6 14

2.6.12 Training and Development

Training and Development is responsible for coordinating with the vendor, airport TSA personnel, and the T&E IPT to prepare training curriculum and job aides prior to test execution.

Training and Development personnel review vendor documentation and TSA Standard Operating Procedures (SOPs), develop the curriculum, coordinate with the user organization and each test site to determine training needs, and administer the training.

T&E Guidebook Revision 6 15

3.0 T&E STRATEGY AND PLANNING

The following sections describe the procedures for the development of the TES and evaluation planning in support of Acquisition Program T&E. Efforts in support of non-acquisition activities (e.g., R&D initiatives) typically do not include the development of a TES and proceed directly to test event planning. These efforts include RCA/ITF demonstration projects. Additional information concerning these activities can be found in the TSAAM.

The PMO leads the development of the TES early in the acquisition process in parallel to requirement development and prior to ADE-2A. The TEMP describes the TES and is a key reference point for the System Evaluator in their development of the SEP. Figure 3-1 illustrates TSA TES and evaluation planning activities.

Figure 3-1: TES and Evaluation Planning Process Flow

3.1 T&E IPT

To ensure coordination of the acquisition program and T&E activities, the PMO forms a T&E IPT that includes representation from the user community, the ITA, and other acquisition stakeholders. The PM charters the T&E IPT and has the overall responsibility for coordination of T&E activities. The Program T&E Manager leads the T&E IPT upon delegation by the PM and is responsible for soliciting T&E IPT input to develop the TES and TEMP.

T&E Guidebook Revision 6 16

3.2 System Evaluation Team

The primary objective of the SET is to provide the coordination necessary on behalf of the ITA to execute T&E activities per the TEMP. The System Evaluator is the SET Chair and forms the SET as the TEMP nears completion or in parallel to acquisition documentation (e.g., requirements) development in cases where there is no established T&E IPT. The SET’s individual members should:

• Be empowered by their parent organizations to make decisions, and keep their organization informed of issues discussed by the SET.

• Plan, execute, and report test activity results, progress, and risks to the SET chair.

• Communicate and maintain awareness of the T&E schedule and milestones to ensure timely completion of T&E activities.

• Provide input, feedback, and concurrence/non-concurrence as required for all T&E documentation requiring SET approval.

• Provide input in support of site selection.

• Provide input and complete tasks as listed in the Qualification Test Readiness Review (QTRR) and OTRR.

SET participating organizations will typically consist of the PMO, SEB, OTB, DGCA, TTB, Cybersecurity Test Teams, DOT&E, and the User Representative. The System Evaluator will expand SET membership as needed to ensure engagement of all necessary parties.

3.3 Reviewing and Assessing Requirements

Working within the T&E IPT, the ITA provides early and continuous feedback to RCA and the User Representative on the testability and measurability of operational requirements, to include the Key Performance Parameters (KPPs) as documented in the ORD. Similarly, the T&E Manager works with the ITA to solicit FRD feedback prior to signature. The test organizations also comment on other requirements documents when appropriate such as detection standards and interface control documents, which specify functionality for system interoperability.

When reviewing operational and functional requirements, T&E Teams consider the following:

• Whether requirements are clearly defined, achievable, testable, and measurable.

• Available test capabilities and limitations as it relates to the evaluation of requirements.

• Major operational risks, changes, or impacts.

• Safety and security risks.

3.4 Use of Third Party Testing in a System Evaluation

The System Evaluator may consider external data sources (e.g., third party test data) in a system evaluation to reduce the testing performed by the ITA and/or add additional confidence to test results. The ITA’s decision to accept an external data source in its evaluation is separate from the

T&E Guidebook Revision 6 17

PMO’s decision on whether to accept a vendor as part of TSA’s qualification process.

Generally, the ITA first reviews vendor-provided data source information and test plans in the Qualification Verification Package (QVP) and then reviews test data from these data sources in the Qualification Data Package (QDP) as illustrated in Figure 3-2. The TSA Third Party Testing Framework defines this process in detail.

Figure 3-2: ITA Review of External Data

3.5 Developing Evaluation Criteria

The evaluation of Operational Effectiveness, Suitability, and Resilience is dependent on the resolution of User Representative-identified Critical Operational Issues (COIs). COIs are introduced in the ORD and represent key operational concerns, with standards of performance that if met, signify that the system is operationally ready to be fielded.

Upon review of the ORD, the System Evaluator may identify one or more Additional Issues (AIs). AIs are key operational concerns that have not been explicitly designated by the User Representative as a COI, however, are necessary for assessment per the judgement of the System Evaluator. Each COI/AI is associated with one or more Critical Operational Issue and Criteria (COICs) and/or AI criteria. COICs are the standards of performance (criteria) for each COI, developed by the System Evaluator, that must be answered to determine if the system is ready to enter full-rate production.

For each criterion, the System Evaluator derives supporting measures used to assess the system’s ability to satisfy the criterion and in turn, the COI/AI. The System Evaluator uses KPPs, when possible, as criteria that directly support the evaluation of a COI. Characteristics of good COICs/AI criteria are:

• Measurable and can be evaluated as met or not met.

T&E Guidebook Revision 6 18

• Avoid terms that could be misinterpreted during the analysis and/or evaluation.

• Associated with one or more measures that reflect the minimal system acceptable performance for entry into procurement (i.e., threshold values).

Figure 3-3 depicts examples of COICs (phrased as questions) and derived measures.

Figure 3-3: Example Criteria and Derived Measures

3.6 TES Development

For major programs (non-delegated systems), the PM, T&E Manager, and ITA jointly prepare a TES briefing in alignment with the program’s acquisition strategy. The System Evaluator is responsible for providing an ESR briefing to the Executive Director, ITA and Executive Director, PEO a minimum of 21 calendar days prior to the TES briefing to DOT&E. An ESR is not required for non-major, major program (post-FOC delivery), and major programs (delegated systems).

The ESR is a concise briefing summarizing the TES and emphasizing best practice T&E principles that enable speed to field and increased performance by discovering problems early.

Upon completion of the ESR and 60 days prior to submission of the TEMP, the PM and T&E Manager brief the TES to DOT&E for major programs (non-delegated systems) or to the CAE for non-major programs, if desired by the CAE. Upon request by the PM and/or T&E Manager, the System Evaluator and/or Test Directors will brief the details of the test events required to support the TES. It is critical for all relevant stakeholder groups (e.g., User Representative, T&E

T&E Guidebook Revision 6 19 organizations) be present for the TES briefing to answer any questions. The TES briefing provides the PMO and T&E stakeholders with input for incorporation during TEMP development. Within 10 calendar days of the TES briefing, the DOT&E will provide a TES Brief Memorandum to document recommendations and guidance for inclusion within the program’s

TEMP.

Figure 3-4 shows the hierarchical relationship between COIs, COICs, and measures.

Figure 3-4: Relationship of COIs, Criteria, and Measures

3.7 TEMP Development

The T&E Manager, in support of the PM and in coordination with the T&E IPT, develops a draft TEMP in parallel to the development of requirements, and obtains final approval from DOT&E 30 days prior to ADE-2A. The T&E Manager follows the guidance in DHS Instruction Guide 026-06-001-01, Test and Evaluation Master Plan (TEMP), dated October 03, 2018, when developing the TEMP. The CAE concurs with the TEMP for major programs (non-delegated systems) prior to routing to DOT&E for final approval. The CAE approves the TEMP for non-major programs.

The TEMP describes the test strategy as summarized in the TES briefing, includes an EF that identifies the data sources to be used to evaluate the operational and functional requirements, and the sequence of test activities necessary to complete the evaluation. The ITA provides inputs to the TEMP with a specific focus on the EF, Failure Definition and Scoring Criteria, and Integrated Test and Evaluation Activities. Figure 3-5 illustrates the general process and documentation hierarchy for a system, beginning with the ESR and culminating in the development of the SER.

T&E Guidebook Revision 6 20

Figure 3-5: TEMP to Reporting Hierarchy

The T&E Manager may update the TEMP, as needed, prior to ADE-2B and anytime changes to the program (e.g., Acquisition Strategy, program mission, key operational requirements) necessitate a change in the overarching TES. The TEMP revisions should receive the same endorsements and approvals as the initial version. If only the EF changes, the T&E Manager should communicate these changes to DOT&E and transmit the EF revision to all parties who previously received the approved TEMP. Under these circumstances, a TEMP re-approval is not necessary unless requested by DOT&E. The PM submits the initial TEMP to DOT&E for approval a minimum of 30 calendar days prior to ADE-2A.

3.7.1 Evaluation Framework

To ensure that the requirements decomposition is complete, the PMO develops a Requirements Traceability Matrix (RTM), which provides a method for tracking that all requirements in the ORD have been decomposed within the FRD. The RTM includes the assignment of a suggested verification method (inspection, demonstration, test, or analysis).

The System Evaluator uses the ORD, FRD, and RTM (if available) to develop a Requirements Crosswalk Matrix (RCM) and provides the RCM to the Program T&E Manager as an input to the formation of an EF. The development of the RCM portion of the EF begins with the System Evaluator correlating FRD requirements to ORD requirements (or confirming the PMO’s correlation of requirements if provided through the RTM), to measures, to COICs, and to COI.

T&E Guidebook Revision 6 21

This traceability exercise enables the System Evaluator to produce a consistent, fully-justified set of operational measures, in alignment with the COIs, as the…